Text Generation with Custom HuggingFace Model

This demo helps you learn about:

• Launching a pre trained custom text generation HuggingFace model in a Seldon Pipeline

• Sending a text input request to get a generated text prediction

The custom HuggingFace text generation model is based on the TinyStories-1M model in the HuggingFace hub.

Create a Seldon ML Pipeline

1. In the Overview page click Create new deployment.

2. Enter the deployment details as follows:

   • Name: hf-custom-tiny-stories

   • Namespace: seldon

3. Configure the default predictor as follows:

   • Runtime: HuggingFace

   • Model Project: default

4. Click Next for the remaining steps and click Launch.

When the deployment is launched successfully, the status of the deployment becomes Available.

1. Click the hf-custom-tiny-stories deployment that you created.

2. In the deployment Dashboard page , click Predict in the left pane.

3. In the Predict REST API dialog, click Enter JSON and paste the following text:

Try other or try a larger-scale model. You can find one in gs://seldon-models/scv2/samples/mlserver_1.6.0/huggingface-text-gen-custom-gpt2. However, you may need to request more memory.