Text Generation with Custom HuggingFace Model

This demo helps you learn about:

• Launching a pre trained custom text generation HuggingFace model in a Seldon Pipeline

• Sending a text input request to get a generated text prediction

The custom HuggingFace text generation model is based on the TinyStories-1M model in the HuggingFace hub.

Create a Seldon ML Pipeline

1. In the Overview page click Create new deployment.

2. Enter the deployment details as follows:

   • Name: hf-custom-tiny-stories

   • Namespace: seldon

   • Type: Seldon ML Pipeline

3. Configure the default predictor as follows:

   • Runtime: HuggingFace

   • Model Project: default

   • Model URI: gs://seldon-models/scv2/samples/mlserver_1.6.0/huggingface-text-gen-custom-tiny-stories

   • Storage Secret: (leave blank/none)

   

   Default predictor spec
4. Click Next for the remaining steps and click Launch.

When the deployment is launched successfully, the status of the deployment becomes Available.

Make Predictions

1. Click the hf-custom-tiny-stories deployment that you created.

2. In the deployment Dashboard page , click Predict in the left pane.

3. In the Predict REST API dialog, click Enter JSON and paste the following text:

   {
     "inputs": [{
       "name": "args",
       "shape": [1],
       "datatype": "BYTES",
       "data": ["The brown fox jumped"]
     }]
   }

4. Click Predict.

   

   A screenshot showing the Predict page with the textarea pre-populated

Next steps

Try other demos or try a larger-scale model. You can find one in gs://seldon-models/scv2/samples/mlserver_1.6.0/huggingface-text-gen-custom-gpt2. However, you may need to request more memory.