To run these demos install Seldon Enterprise Platform with the necessary permissions to create or update deployments.

Most demos utilize pre-built models, except for the Kubeflow demo, which includes steps to build a model and push it to MinIO.

Models can either be:

• Pushed to MinIO or other object stores for use with pre-packaged model servers.

• Packaged as Docker containers using language wrappers.

For detailed guidance on the building and hosting stages, refer to the .

Regardless of how you create the Seldon ML Pipelines, the Seldon Enterprise Platform should have visibility into the relevant namespaces. These demos utilize the Enterprise Platform UI for simplicity and convenience.

When ML models are deployed in production, sometimes even minor changes in a data distribution can adversely affect the performance of ML models. When the input data distribution shifts then prediction quality can drop. It is important to track this drift. This demo is based on the mixed-type tabular data drift detection method in the alibi detect project for tabular datasets.

Here we will:

• Launch a Seldon ML pipeline with the income classifier model.

• Setup a mixed-type tabular data drift detector for this particular model.

• Make a batch of predictions over time

• Track the drift metrics in the Monitoring dashboard.

This demo uses a model trained to predict high or low income based on .

Register a pre-trained income classifier SKLearn model.

1. In the Model Catalog page, click Register a new model:
2. In the Register New Model wizard, enter the following information, then click Register Model:

Edit the model metadata to update the prediction schema for the model. The prediction schema is a generic schema structure for machine learning model predictions. It is a definition of feature inputs and output targets from the model prediction. Learn more about the predictions schema at the open source repository. Use the income classifier model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata.

1. Click the model income-classifier that you registered.
2. Click Edit Metadata to update the Prediction schema field associated with the model using the contents of prediction schema income-classifier-prediction-schema.json.

Deploy the income classifier model from the catalog into an appropriate namespace

1. From the model catalog, under the Action dropdown list, select Deploy.
2. Enter the deployment details in the deployment creation wizard and click Next:

From the deployment overview page, select your deployment to enter the deployment dashboard. Inside the deployment dashboard, add a drift detector with by clicking the Add button within the DRIFT DETECTION widget.

Enter the following parameters in the modal popup which appears, to configure the detector:

• Detector Name: income-drift.

• Model URI: (For public google buckets, secret field is optional)

• Reply URL: (By default, the Reply URL is set as seldon-request-logger in the logger's default namespace. If you are using a custom installation, please change this parameter according to your installation.)

Then, click Create Detector to complete the setup.

As per the income classifier model, use the same model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata for drift detector.

1. Click on the vertical ellipses “⋮” icon for the drift detector you have just registered.
2. Click the Configure Metadata option to update the prediction schema associated with the model

3. Paste the downloaded income-classifier-prediction-schema.json, name the model

1. From the deployment dashboard, click on Batch Jobs. Run a batch prediction job using the Open Inference Protocol (OIP) payload format text predictions data file data.txt.

This file has 4000 individual data points and based on our drift detector configuration, drift will be detected for a batch every 200 points. The distribution of the data in the first half section is the same as the distribution of the reference data the drift detector was configured with and the second half section of the data should be different to observe drift.

1. Upload the data to a bucket store of your choice. This demo will use and store the data at bucket path minio://income-batch-data/data.txt . Do not forget to - we have it as minio-bucket-envvars here. Refer to the for an example of how this can be done via the minio browser.

2. Running a batch job with the configuration below. This runs an offline job that makes a prediction request for a batch of 200 rows in the file at minio://income-batch-data/data.txt every 5 seconds:

Under the Monitor section of your deployment navigation, on the Drift Detection Tab, you can see a timeline of drift detection metrics.

The drift dashboard showcases 2 types of metrics graphs:

1. P-value score over time

   1. Zoomed in view, focusing on features that have drifted, i.e. features that have a p-value score of less than the threshold.

   1. Zoomed out view, showing all features.

If you have you should see a notification about the drift

with further details present on the alerting log

To further analyse prediction data drift, you can also switch to the feature distribution tab to compare predictions to reference data distribution. See demo for setup details.

Upload the income classifier reference dataset drift-reference-v2.csv as the reference data to monitor data drift in terms of feature distributions.

Once reference data is available, you can compare the distributions of the prediction data to the reference data.

You can see when reference data is available by checking the button on the top left of the Distributions dashboard. If it is not clickable and displays Reference data available, then reference data is available.

For each feature, you can click on Toggle reference data to view reference data side by side.

We will see that the drifted data has lower education individuals that were not in the reference data.

If you experience issues with this demo, see the or sections.

Distributions monitoring provides an ability to view the statistics and distributions of features and predictions made by your model between any given time. This feature also enables you to draw comparisons between the model predictions for different feature combinations, cohorts and/or time slices. It is a vital aspect of model monitoring cycle to understand if the deployed model has the desired prediction characteristics during different times and for different cohorts.

This demo uses a model trained to predict high or low income based on demographic features from a 1996 US census.

In this demo, we will:

• Register an income classifier model with the relevant predictions schema

• Launch a Seldon ML pipeline with the income classifier model

• Run a Batch Job to send predictions using the model pipeline

• Observe the feature distributions of the live predictions

• Filter distributions by time or predictions and feature level filters

Under the production installation, the following must be installed:

• (required) - used for storage of live predictions and reference data

• (required) - used for storage of model metadata (which includes the prediction schema)

• (required for demo) - allows for batch jobs to be run

In addition to the prerequisites, this demo needs the request logger to connect to Seldon Enterprise Platform in order to fetch model level predictions schema. This requires . Also, this feature is supported with Open Inference Protocol (OIP) only. Support for json data, string data, bytes payload or multi-node graph use cases is not available yet.

Register a pre-trained income classifier SKLearn model.

1. In the Model Catalog page, click Register a new model:
2. In the Register New Model wizard, enter the following information, then click Register Model:

Edit the model metadata to update the prediction schema for the model. The prediction schema is a generic schema structure for machine learning model predictions. It is a definition of feature inputs and output targets from the model prediction. Learn more about the predictions schema at the open source repository. Use the income classifier model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata.

1. Click the model income-classifier that you registered.
2. Click Edit Metadata to update the Prediction schema field associated with the model using the contents of prediction schema income-classifier-prediction-schema.json.

Deploy the income classifier model from the catalog into an appropriate namespace

1. In the Model catalog, select Deploy of the Action dropdown.
2. Enter the deployment details in the deployment creation wizard and click Next:

   • Name

In order to observe the predictions and feature distributions, first we need to send some predictions to the model. To simulate such a use case, we are going to run a that will require an input file of predictions formatted according to the . To do so, we need to follow the steps below:

1. Download the predictions data file predictions.txt.

This is a dataset which contains 60 predictions. The first few lines of the input file 'predictions.txt' should show the following format:

1. Upload the data to a bucket store of your choice. This demo uses and stores the data at bucket path minio://predictions-data/predictions.txt. Refer to the for an example of how this can be done using the minio browser.

2. Open the Deployment Dashboard of your deployment by clicking on it in the Overview page.

This will create a new Batch Job that will use the predictions data, located at minio://predictions-data/predictions.txt, to send requests to the model. The output data will be stored at minio://predictions-data/output-data-{{workflow.name}}.txt. The workflow will send requests to the model every 5 seconds using the REST transport protocol, and those will be formatted according to the . We also specify the secret name minio-bucket-envvars which contains the credentials required for access to the MinIO bucket store and the predictions data.

Select the income classifier deployment and go to the monitor section to view the predictions and feature distributions.

Filter distributions by time or predictions and feature level filters to compare different cohorts and further analysis. For example let's look at the predictions for all individuals in the Age group 25-50 and also filter by their Marital Status as Married and Never-Married only and see how the average prediction frequency changes for this cohort.

Distributions parameters configuration allows you to configure your charts for further analysis. For example let's look at at the charts in the Age group and change the Histogram interval to 15 and Number of time buckets to 30 to see.

A useful feature for monitoring feature distributions is the ability to compare features against features from a reference dataset. Feature level comparisons allow users to easily gauge which features are actually drifting visually.

Here, we define reference data as a set of data where the distribution of the data is a useful representation of the expected live predictions. Typically, this would be a sampled subset of the training data used to create the inference model.

In this section of the demo, we will extend the distributions monitoring demo and:

• Show how reference data needs to be prepared before it can be inserted into Seldon Enterprise Platform

• Create a bucket and upload some reference data onto minio

• Trigger a retrieval job via the Seldon Enterprise Platform UI

Currently, there is a strict requirement for the types of data that can be inserted into Seldon Enterprise Platform that is dependent on the .

1. The number of columns in the reference data must match the number of expected columns from the prediction schema. This means that some feature types, (i.e. ONE_HOT and PROBA) may need to be split into dummy/indicator values. For example, a PROBA feature (e.g. the Iris Species form the ) might have 3 categories in the schema:

   However, the raw reference dataset represents this feature as a single column where the values are its categories. This is typically the case for output features where the reference data has the expected real output, while the model returns a probability distribution of the different possible outputs:

   Iris Species

Download the income classifier reference dataset income-reference-data.csv. Store it in a bucket of your choice.

In this demo, we will use and create a bucket called reference-data which we can upload the income-reference-data.csv file to.

In your deployment, navigate to the Monitor page and open the Distributions tab. An Add reference data button should be available.

This button opens a wizard where the bucket path and secret can be specified. For this demo, use the following values:

• Bucket Path: minio://reference-data/

• Bucket Secret: minio-bucket-envvars

After confirming, the job will start running, and the Add reference data button will change to a Retrieving data... status.

Give the job a few minutes to finish. Once finished, the Retrieving data... button will change to Reference data available. Now, the Toggle reference data toggle will become available to click for every feature, and you can view comparisons of the live prediction distributions against the reference data.

Use the filters to filter both live predictions and reference data.

This demo helps you learn about:

• Launching a pre trained custom text generation HuggingFace model in a Seldon Pipeline

• Sending a text input request to get a generated text prediction

The custom HuggingFace text generation model is based on the TinyStories-1M model in the HuggingFace hub.

Create a Seldon ML Pipeline

1. In the Overview page click Create new deployment.

2. Enter the deployment details as follows:

   • Name: hf-custom-tiny-stories

   • Namespace: seldon
3. Configure the default predictor as follows:

   • Runtime: HuggingFace

   • Model Project: default
4. Click Next for the remaining steps and click Launch.

When the deployment is launched successfully, the status of the deployment becomes Available.

1. Click the hf-custom-tiny-stories deployment that you created.

2. In the deployment Dashboard page , click Predict in the left pane.

3. In the Predict REST API dialog, click Enter JSON and paste the following text:

Try other or try a larger-scale model. You can find one in gs://seldon-models/scv2/samples/mlserver_1.6.0/huggingface-text-gen-custom-gpt2. However, you may need to request more memory.

• Install with Seldon Enterprise Platform.

• Set up the namespace with a service account for a production environment. For more information, see the .

This demo helps you learn about:

Understanding how complex models make predictions is crucial for ensuring transparency, building trust, and identifying potential biases. Model explainers provide insights into how features influence outcomes, aiding in debugging and refining models.

In this demonstration, you can learn about using method to explore model explanations. This includes identifying the segments of an input image that had the most influence on the prediction and analyzing the precision of the Anchor and Coverage metrics.

This demo helps you learn about:

• Launching an image classification pipeline

In a production environment, monitoring the data used for your machine learning model's inferences is essential, as data changes can significantly impact the performance of the model.

Using Alibi Detect's method for tabular datasets, this demo helps you to identify outliers in your inference data by:

• Launching an image classifier model trained on the . The data instances contain 32x32x3 pixels images that are classified into 10 classes such as truck, frog, cat, and others.

In this demo we will:

• Create a pipeline which can be used to classify tabular data

• Create an explainer that will offer insight into why a particular prediction was made for a given input

• Send a prediction request to the pipeline

• View the explanation

Register a pre-trained income classifier SKLearn model. See the section in the Drift Detection demo for detailed instructions.

Edit the model metadata to update the prediction schema for the model. See the section in the Drift Detection demo for detailed instructions.

1. Navigate to the Model Catalog page where the model was registered.

2. Find the model and under the Action dropdown list, select Deploy.

1. Enter the deployment details in the deployment creation wizard and click Next:

   • Name: income-classifier

   • Namespace: seldon

1. The default predictor should already be filled in from the model catalog. Click Next.

2. Click Next for the remaining steps, then click Launch.

3. If your deployment is launched successfully, it will show an Available

1. Click on the income-classifier pipeline created in the previous section to enter the deployment dashboard.

2. Inside the deployment dashboard, on the left navigation drawer, click on the Predict button.

3. On the Predict page, enter the following text:

1. Click the Predict button.

There are currently 2 explainers available for tabular data classification:

• Anchor Explainer

• Kernel SHAP Explainer

1. From the income-classifier deployment dashboard, click Add inside the Model Explanation card.

2. For step 1 of the Explainer Configuration Wizard, select Tabular then click Next.

1. Skip step 4

2. For step 5, set following details

3. Click Next for the remaining steps, then click Launch.

1. Navigate to the Requests page using the left navigation drawer.

2. Click on the View explanation button to generate explanations for the request.

Congratulations, you've created an explanation for the request! 🥳

Why not try our other ? Ready to dive in? Read our to learn more about how to use Enterprise Platform.

This demo helps you learn about:

• Launching a movie sentiment pipeline which takes text input

• Sending a request to get a sentiment prediction

• Creating an explainer for the model

• Sending the same request and then get an explanation

The explainer uses the to provide insight into why a particular classification was made by the model. We'll see patterns in input text that are most relevant to the prediction outcome.

1. In the Overview page, click Create new deployment.

2. Type the following deployment details and click Next:

   Parameter

   Value

1. Click the movie deployment that you created.

2. In the Deployment Dashboard, click Predict in the left pane.

3. In the Predict page, click Enter JSON and paste the following text:

1. In the Deployment Dashboard page for the deployment movie, click Add inside the MODEL EXPLANATION card.

2. In the Explainer Configuration Wizard, choose Text and click Next.

3. In the Explainer Types step, choose the Anchor

After sometime, the explainer should become available.

1. In the Deployment Dashboard for the deployment named movie, click Predict in the left pane.

2. In the Predict page, click Enter JSON and once again paste the following text and click Predict:

3. Click Explain to generate explanations for the request.

Try the other or read our to learn more about how to use Seldon Enterprise Platform.

This demo helps you learn about:

• Deploying a pretrained sklearn iris model

• Load testing the model

• Observing requests and metrics

• Deploying a canary XGBoost model

• Load testing the canary model

• Observing requests and metrics for both models

• Promoting the canary model

Iris is the genus of flower which contains three species, namely setosa, versicolor, and virginica. This demo is based on iris classification model based on flower properties like sepal length, sepal width, petal length, and petal width. The species are also the classes that are used for the classification.

1. In the Overview page, select Create new deployment.

2. In the Deployment Creation Wizard, enter the deployment details as follows:

   • Name: iris-classifier

Afer your deployment is launched successfully, the status of the deployment reads Available in the Overview page.

1. Click the deployment that you created. The Dashboard page for the deployment is displayed.

2. In the Requests Monitor section, click Start Load Test to start a load test.

3. In the Load Test Wizard configure the following details:

After you click Run Test it spawns a Kubernetes job that send continuous prediction requests for the specified seconds to the SKLearn model in the deployment.

After the load test has started, you can monitor the upcoming requests, their responses and in the Requests page the deployment. If this doesn't work, consult the docs section for debugging.

You can also see core metrics in the Dashboard page.

The next step is to create an XGBoost canary model.

1. In the Dashboard of the deployment and click Add Canary.

2. In the Canary Configuration Wizard, configure the default predictor as follows:

   • Runtime: XGBoost

When the canary model is launched successfully, the deployment remains in an Available status.

This creates a new canary deployment with the XGBoost model and 10% of the traffic is sent to this deployment.

This time, create a new load test with the canary model running and observe the requests and metrics for both models. You can use either the same Json payload from the previous or construct a new one with different values or number of predictions.

After the load test has started, you can monitor the upcoming requests, their responses and in the Requests page the deployment. If this doesn't work, consult the docs section for debugging.

In order to see the requests for the canary model, you need to select the iris-classifier-canary predictor and the related Node in the Node Selector filter of the Requests page, as shown on the screenshot. Make a note that the number of requests for the canary model and it is 10% of the total number of requests, as was specified in the canary deployment.

You can also see core metrics such as Live Requests, and Resource Monitor for both models in the Dashboard page.

After you evaluated the performace of the canary model you can promote the canary model as a main model.

1. In the Dashboard page of the deployment and click Promote Canary.

2. In the Promote Canary dialog, click Confirm to promote the canary model to the main model.

When the canary model is promoted successfully, the deployment remains in an Available status.