Distributions monitoring provides an ability to view the statistics and distributions of features and predictions made by your model between any given time. This feature also enables you to draw comparisons between the model predictions for different feature combinations, cohorts and/or time slices. It is a vital aspect of model monitoring cycle to understand if the deployed model has the desired prediction characteristics during different times and for different cohorts.
Register an income classifier model with the relevant predictions schema
Launch a Seldon Deployment with the income classifier model
Run a Batch Job to send predictions to the model deployment
Observe the feature distributions of the live predictions
Filter distributions by time or predictions and feature level filters
Under the production installation, the following must be installed:
(required) - used for storage of live predictions and reference data
(required) - used for storage of model metadata (which includes the prediction schema)
(required for demo) - allows for batch jobs to be run
In addition to the prerequisites, this demo needs the request logger to connect to Seldon Enterprise Platform in order to fetch model level predictions schema. This requires . Also, this feature is supported with Open Inference Protocol (OIP) only. Support for json data, string data, bytes payload or multi-node graph use cases is not available yet.
Register a pre-trained income classifier SKLearn model.
In the Model Catalog page, click Register a new model:
In the Register New Model wizard, enter the following information, then click Register Model:
Edit the model metadata to update the prediction schema for the model. The prediction schema is a generic schema structure for machine learning model predictions. It is a definition of feature inputs and output targets from the model prediction. Learn more about the predictions schema at the open source repository. Use the income classifier model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata.
Click the model income-classifier that you registered.
Click Edit Metadata to update the Prediction schema field associated with the model using the contents of prediction schema income-classifier-prediction-schema.json.
Deploy the income classifier model from the catalog into an appropriate namespace
In the Model catalog, select Deploy of the Action dropdown.
Enter the deployment details in the deployment creation wizard and click Next:
Name
Click Next for the remaining steps, then click Launch.
In order to observe the predictions and feature distributions, first we need to send some predictions to the model. To simulate such a use case, we are going to run a that will require an input file of predictions formatted according to the . To do so, we need to follow the steps below:
Download the predictions data file predictions.txt.
This is a dataset which contains 60 predictions. The first few lines of the input file 'predictions.txt' should show the following format:
Upload the data to a bucket store of your choice. This demo uses and stores the data at bucket path minio://predictions-data/predictions.txt. Refer to the for an example of how this can be done using the minio browser.
Open the Deployment Dashboard of your deployment by clicking on it in the Overview page.
This will create a new Batch Job that will use the predictions data, located at minio://predictions-data/predictions.txt, to send requests to the model. The output data will be stored at minio://predictions-data/output-data-{{workflow.name}}.txt. The workflow will send requests to the model every 5 seconds using the REST transport protocol, and those will be formatted according to the . We also specify the secret name minio-bucket-envvars which contains the credentials required for access to the MinIO bucket store and the predictions data.
Select the income classifier deployment and go to the monitor section to view the predictions and feature distributions.
Filter distributions by time or predictions and feature level filters to compare different cohorts and further analysis. For example let's look at the predictions for all individuals in the Age group 25-50 and also filter by their Marital Status as Married and Never-Married only and see how the average prediction frequency changes for this cohort.
Distributions parameters configuration allows you to configure your charts for further analysis. For example let's look at at the charts in the Age group and change the Histogram interval to 15 and Number of time buckets to 30 to see.
A useful feature for monitoring feature distributions is the ability to compare features against features from a reference dataset. Feature level comparisons allow users to easily gauge which features are actually drifting visually.
Here, we define reference data as a set of data where the distribution of the data is a useful representation of the expected live predictions. Typically, this would be a sampled subset of the training data used to create the inference model.
In this section of the demo, we will extend the distributions monitoring demo and:
Show how reference data needs to be prepared before it can be inserted into Seldon Enterprise Platform
Create a bucket and upload some reference data onto minio
Trigger a retrieval job via the Seldon Enterprise Platform UI
Currently, there is a strict requirement for the types of data that can be inserted into Seldon Enterprise Platform that is dependent on the .
The number of columns in the reference data must match the number of expected columns from the prediction schema. This means that some feature types, (i.e. ONE_HOT and PROBA) may need to be split into dummy/indicator values. For example, a PROBA feature (e.g. the Iris Species form the ) might have 3 categories in the schema:
However, the raw reference dataset represents this feature as a single column where the values are its categories. This is typically the case for output features where the reference data has the expected real output, while the model returns a probability distribution of the different possible outputs:
Download the income classifier reference dataset income-reference-data.csv. Store it in a bucket of your choice.
In this demo, we will use and create a bucket called reference-data which we can upload the income-reference-data.csv file to.
In your deployment, navigate to the Monitor page and open the Distributions tab. An Add reference data button should be available.
This button opens a wizard where the bucket path and secret can be specified. For this demo, use the following values:
Bucket Path: minio://reference-data/
Bucket Secret: minio-bucket-envvars
After confirming, the job will start running, and the Add reference data button will change to a Retrieving data... status.
Give the job a few minutes to finish. Once finished, the Retrieving data... button will change to Reference data available. Now, the Toggle reference data toggle will become available to click for every feature, and you can view comparisons of the live prediction distributions against the reference data.
Use the filters to filter both live predictions and reference data.
MinIO (optional) - allows for easy storage of and access to downloaded datasets
The predictor details should already be filled in from the model catalog except the Model Name field. Use income for the Model Name field.
Model Name: income
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Click the Batch Jobs in the left pane.
Create a Batch Job with the following details:
Toggle the comparison of live predictions and reference data feature distributions
Iris Species
Versicolor
Setosa
Virginica
Virginica
Setosa
The processed data should have the following format
Setosa
Versicolor
Virginica
0
1
0
1
0
The order of columns must match the prediction schema
Both "input" and "output" features must be in the reference dataset, where "input" features come before "output" features.
Input Data Location: minio://predictions-data/predictions.txt
Output Data Location: minio://predictions-data/output-data-{{workflow.name}}.txt
Number of Workers: 1
Number of Retries: 3
Batch Size: 1
Minimum Batch Wait Interval (sec): 5
Method: Predict
Transport Protocol: REST
Input Data Type: Open Inference Protocol (OIP)
Object Store Secret Name: minio-bucket-envvars
To run these demos install Seldon Enterprise Platform with the necessary permissions to create or update deployments.
Most demos utilize pre-built models, except for the Kubeflow demo, which includes .
Models can either be:
Pushed to MinIO or other object stores for use with pre-packaged model servers.
Packaged as Docker containers using language wrappers.
0
0
0
1
0
0
1
1
0
0
For detailed guidance on the building and hosting stages, refer to the .
Regardless of how you install your SeldonDeployments, the Seldon Enterprise Platform should have visibility into the relevant namespaces. These demos utilize the Enterprise Platform UI for simplicity and convenience.
When ML models are deployed in production, sometimes even minor changes in a data distribution can adversely affect the performance of ML models. When the input data distribution shifts then prediction quality can drop. It is important to track this drift. This demo is based on the mixed-type tabular data drift detection method in the alibi detect project for tabular datasets.
Here we will :
Launch an income classifier model based on demographic features from a 1996 US census. The data instances contain a person’s characteristics like age, marital status or education while the label represents whether the person makes more or less than $50k per year.
Setup a mixed-type tabular data drift detector for this particular model.
Make a batch of predictions over time.
Track the drift metrics in the Monitoring dashboard.
Register a pre-trained income classifier SKLearn model.
In the Model Catalog page, click Register a new model:
In the Register New Model wizard, enter the following information, then click Register Model:
Edit the model metadata to update the prediction schema for the model. The prediction schema is a generic schema structure for machine learning model predictions. It is a definition of feature inputs and output targets from the model prediction. Learn more about the predictions schema at the open source repository. Use the income classifier model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata.
Click the model income-classifier that you registered.
Click Edit Metadata to update the Prediction schema field associated with the model using the contents of prediction schema income-classifier-prediction-schema.json.
Deploy the income classifier model from the catalog into an appropriate namespace
In the Model catalog, select Deploy of the Action dropdown.
Enter the deployment details in the deployment creation wizard and click Next:
Name
Click Next for the remaining steps, then click Launch.
From the deployment overview page, select your deployment to enter the deployment dashboard. Inside the deployment dashboard, add a drift detector with by clicking the Add button within the Drift Detection widget.
Enter the following parameters in the modal popup which appears, to configure the detector:
Model Name: income-drift.
Storage URI: (For public Google Buckets, secret field is optional)
Reply URL: Leave as the default value, shown below, unless you are using a custom installation, please change this parameter according to your installation
Then, click CREATE DETECTOR to complete the setup.
As per the income classifier model, use the same model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata for drift detector.
Click on the vertical ellipses “⋮” icon for the drift detector you have just registered.
Click the Configure Metadata option to update the prediction schema associated with the model
Paste the downloaded income-classifier-prediction-schema.json, name the model
From the deployment dashboard, click on Batch Jobs. Run a batch prediction job using the Open Inference Protocol (OIP) payload format text predictions data file data.txt.
This file has 4000 individual data points and based on our drift detector configuration, drift will be detected for a batch every 200 points. The distribution of the data in the first half section is the same as the distribution of the reference data the drift detector was configured with and the second half section of the data should be different to observe drift.
Upload the data to a bucket store of your choice. This demo will use and store the data at bucket path minio://income-batch-data/data.txt . Do not forget to - we have it as minio-bucket-envvars here. Refer to the for an example of how this can be done via the minio browser.
Running a batch job with the configuration below. This runs an offline job that makes a prediction request for a batch of 200 rows in the file at minio://income-batch-data/data.txt every 5 seconds:
Under the Monitor section of your deployment navigation, on the Drift Detection Tab, you can see a timeline of drift detection metrics.
The drift dashboard showcases 2 types of metrics graphs:
P-value score over time
1a. Zoomed in view, focusing on features that have drifted, i.e. features that have a p-value score of less than the threshold.
2b. Zoomed out view, showing all features.
Note that for both drift metrics graphs, the starting batches do not drift and are marked by an empty O symbol, while the later batches do drift and are marked by a filled O symbol.
If you have you should see a notification about the drift
with further details present on the alerting log
To further analyse prediction data drift, you can also switch to the feature distribution tab to compare predictions to reference data distribution. See demo for setup details.
Upload the income classifier reference dataset drift-reference-v2.csv as the reference data to monitor data drift in terms of feature distributions.
Once reference data is available, you can compare the distributions of the prediction data to the reference data.
You can see when reference data is available by checking the button on the top left of the Distributions dashboard. If it is not clickable and displays Reference data available, then reference data is available.
For each feature, you can click on Toggle reference data to view reference data side by side.
We will see that the drifted data has lower education individuals that were not in the reference data.
If you experience issues with this demo, see the and also the or sections.
The predictor details should already be filled in from the model catalog except the Model Name field. Use income for the Model Name field.
Model Name: income
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Input Data Location: minio://income-batch-data/data.txt
Output Data Location: minio://income-batch-data/output-{{workflow.name}}.txt
Number of Workers: 1
Number of Retries: 3
Batch Size: 200
Minimum Batch Wait Interval (sec): 5
Method: Predict
Transport Protocol: REST
Input Data Type: Open Inference Protocol (OIP)
Storage Secret Name: minio-bucket-envvars
This demo requires Knative installation on the cluster as the drift detector will be installed as a kservice. See Knative installation instructions for necessary setup required.
Understanding how complex models make predictions is essential for promoting transparency, trust, and identifying potential biases. Model explainers offer insights into feature influence on outcomes, supporting debugging and model refinement.
We are going to demonstrate model explanations using Alibi Explain's Anchor Images method, taking note of the segments in an input image that influenced the prediction the most, as well as, observing the Anchor's Precision and Coverage metrics.
In this demo we will:
Launch an image classification deployment
Send prediction requests to the deployment
Create an explainer for the deployment
Generate explanations for previously sent prediction requests
The model used in this demo was trained to classify images based on the .
Click the Next button until the end and, finally, click Launch.
If your deployment is launched successfully, it will have Available status.
We will make a prediction request using the image of a frog from the cifar10 dataset. The image is a JSON file in the REST format of the .
Click on the cifar10-classifier deployment created in the previous section to enter the deployment dashboard.
Inside the deployment dashboard, click on the Predict button.
On the Predict page, click on Browse
From the cifar10-classifier deployment dashboard, click Add inside the Model Explanation card.
For step 1 of the Explainer Configuration Wizard, select the Image model data type and click Next.
For step 2, make sure Anchor is selected, then click Next.
For step 3, enter the following values in the Explainer URI tab:
Explainer URI:
Click the Next button until the end, and finally, click Launch.
After a short while, the explainer should become available.
Now that the explainer is available, we can make use of it to generate an explanation for the prediction request we made earlier.
Click on the cifar10-classifier deployment created in the previous sections to enter the deployment dashboard.
Navigate to the Requests page using the left navigation drawer, and you will see the request you made earlier.
Click on the View explanation button to generate an explanation for the request.
After a few seconds, the explanation should be generated and displayed on the page.
Congratulations, you've created an explanation for the request! 🥳
Why not try our other ? Ready to dive in? Read our to learn more about how to use Enterprise Platform.
Text Generation with Custom HuggingFace Model
In this demo we will:
Launch a pretrained custom text generation HuggingFace model in a Seldon Deployment
Send a text input request to get a generated text prediction
The custom HuggingFace text generation model is based on the TinyStories-1M model in the HuggingFace hub.
Create a Seldon Deployment
In the Overview page, click Create new deployment.
Enter the deployment details as follows, then click Next:
Parameter
Value
Configure the default predictor as follows, then click Next:
Parameter
Value
Skip to the end and click Launch.
When your deployment is launched successfully the status will read as Available.
Click the hf-custom-tiny-stories deployment that you created.
In the Deployment Dashboard page, click Predict in the left pane.
In the Predict page, click Enter JSON
Congratulations, you've successfully sent a prediction request using a custom HuggingFace model! 🥳
Why not try our other ? Or perhaps try running a larger-scale model? You can find one in s://seldon-models/scv2/samples/mlserver_1.6.0/huggingface-text-gen-custom-gpt2. However, you may need to request more memory!
Batch Prediction Jobs
MinIO should already be installed with Seldon Enterprise Platform. The MinIO browser should be exposed on /minio/ (note the trailing forward slash).
For trials, the credentials will by default be the same as the Enterprise Platform login, with MinIO using the email as its Access Key and the password as its Secret Key.
On a production cluster, the namespace needs to have been set up with a service account. This can be found under the .
This demo helps you learn about:
Outlier Detection
In a production environment, it is critical to monitor the data your machine learning model runs inference on, as changes in data can adversely affect the performance of ML models.
In this demo learn how to identify outliers in your inference data using Alibi Detect's method for tabular datasets.
In this demo, we will:
Launch an image classification deploying with a model, trained on the .
Namespace: seldon
Type: Seldon Deployment
Model URI: gs://seldon-models/triton/tf_cifar10
Storage Secret: (leave blank/none)
Model Name: cifar10
to select and upload the previously downloaded prediction file.
Click the Predict button, the prediction request should be successful and the response should be shown.
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
The seldon and seldon-gitops namespaces are installed by default, which may not always be available. Select a namespace which best describes your environment.
A secret may be required for private buckets.
Additional steps may be required for your specific model.
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Setup Input Data
Run a Batch Job
In the Resources (Optional) section, you can specify how much memory and CPU are allocated to the containers in this specific batch job workflow. If no values are set on this form, the default values specified in Helm values will be used. Refer to the for details.
After a short while, the detector should become available.
Now that the outlier detector available, we can showcase how to make use of it to identify outliers in the inference data. We are going to send 2 requests to the model, one with a normal image and another with a perturbed image to identify the outlier.
A frog image from the CIFAR-10 dataset in the Open Inference Protocol (OIP) format:
A perturbed image of the same frog in the Open Inference Protocol (OIP) format:
In the deployment dashboard of click Predict in the left pane..
Click Browse to upload the cifar10-frog-oip.json file.
Click Predict. The prediction request is processed and the response is displayed.
Click Remove to remove the uploaded file
Click Browse again to make a prediction with the perturbed image of the frog using the cifar10-frog-perturbed-oip.json file. The prediction request is processed and the response is displayed.
Navigate to the Requests page using the left navigation drawer to view the requests made to the model and their prediction responses. Outlier scores will be available on the right side of each instance.
Previously made prediction requests with their prediction responses and outlier scores
You can also highlight outlier prediction responses using the Highlight Outliers toggle on the top-right of the screen.
Highlighted outlier prediction requests with their prediction responses and outlier scores
Furthermore, you can filter the requests to show only the outliers by clicking on the Filter icon in the top right(next to the Page number) and selecting the Outliers option, toggling to show only outliers or not.
A highlighted outlier prediction after applying a filter to show only outliers
It is important to be able to monitor the outlier detection requests in real-time to ensure that the model is performing as expected and to take corrective actions when necessary.
Click Monitor in the left pane.
Select the Outlier Detection tab to view a timeline graph of outlier/inlier requests.
A timeline graph showing the first request classified as an inlier and the second as an outlier
Congratulations, you've successfully demonstrated how to identify outliers in your inference data using Alibi Detect's VAE outlier detection method! 🥳
Why not try our other demos? Ready to dive in? Read our operations guide to learn more about how to use Enterprise Platform.
This demo requires Knative installation on the cluster as the drift detector will be installed as a kservice. See for necessary setup required.
Create a Seldon Deployment
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Launching a movie sentiment pipeline which takes text input
Sending a request to get a sentiment prediction
Creating an explainer for the model
Sending the same request and then get an explanation
The explainer uses the to provide insight into why a particular classification was made by the model. We'll see patterns in input text that are most relevant to the prediction outcome.
In the Overview page, click Create new deployment.
Type the following deployment details and click Next:
Parameter
Value
Click Next for the remaining steps in the Deployment Creation Wizard and then click Launch.
Click the movie deployment that you created.
In the Deployment Dashboard, click Predict in the left pane.
In the Predict page, click Enter JSON and paste the following text:
In the Deployment Dashboard page for the deployment movie, click Add inside the MODEL EXPLANATION card.
In the Explainer Configuration Wizard, choose Text and click Next.
In the Explainer Types step, choose the Anchor
After sometime, the explainer should become available.
In the Deployment Dashboard for the deployment named movie, click Predict in the left pane.
In the Predict page, click Enter JSON and once again paste the following text and click Predict:
Click Explain to generate explanations for the request.
Try the other or read our to learn more about how to use Seldon Enterprise Platform.
Canary Promotion
In this demo, we will:
Deploy a pretrained SKLearn classification model based on the Iris dataset
Load test the model with prediction data
Observe the prediction requests and their responses
Click Next for the remaining steps without changing any fields, and click Launch.
{
"parameters": {
"content_type": "str"
},
"inputs": [{
"name": "text review",
"shape": [1],
"datatype": "BYTES",
"data": ["this film has bad actors"]
}]
}
{
"parameters": {
"content_type": "str"
},
"inputs": [{
"name": "text review",
"shape": [1],
"datatype": "BYTES",
"data": ["this film has bad actors"]
}]
}
Create a Seldon Deployment
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
The seldon and seldon-gitops namespaces are installed by default, which may not always be available. Select a namespace which best describes your environment.
A secret may be required for private buckets.
Additional steps may be required for your specific model.
Get Predictions
Add an Anchor Text Explainer
It is only possible to create one explainer for each deployment.
You can also enter a comment here for a gitops enabled namespace.
Get Explanation for one Request
Note: If the explainer is making use of parameters.content_type field, View explanation button from the Requests dashboard then it leads to the server error. The /explain API works as expected if parameters.content_type is included in the request input.
Observe and compare the prediction requests and metrics for both models
Promote the canary model
You can use SKLearn classification model based on the well-known Iris dataset. This dataset includes 150 samples of iris flowers, each with 4 features: sepal length, sepal width, petal length, and petal width, measured in centimeters. The samples are labeled by iris species—setosa, versicolor, and virginica—with an even distribution across these classes.
In the Overview page, click Create new deployment.
In the Deployment Creation Wizard, enter the following deployment details and click Next:
Name: iris-classifier
Namespace: seldon
Type: Seldon Deployment
Deployment details
Configure the default predictor as follows and click Next:
Runtime: Scikit Learn
Model URI: gs://seldon-models/scv2/samples/mlserver_1.6.0/iris-sklearn
Model Project: default
Storage Secret: (leave blank/none)
Model Name: iris
Default predictor spec
Click Next for the remaining steps, then click Launch.
When your deployment is launched successfully, the status reads Available.
Once the deployment is in an Available status, navigate to its Dashboard page by clicking on it.
In the Dashboard page, scroll down to find the Requests Monitor section and click Start a load test with the following details:
Connections(total): 1
Load Parameter: Duration(seconds)
Value: 120
JSON payload:
Load Test Wizard
This will create a Kubernetes Job that will be sending prediction requests for the specified seconds to the SKLearn model in the deployment.
After the load test has started, you can monitor the upcoming requests and their responses by navigating to the Requests page of the deployment.
Requests page showing prediction requests and their responses
You can also monitor live requests metrics resulting from the load test if you navigate back to the Dashboard page of the deployment, and scroll down to the Live Requests section. In this screenshot you can see the number of requests per second and the average latency of the model.
Live Requests section showing the number of requests per second and the average latency of the model
Furthermore, you can monitor the utilization metrics for the model in the Dashboard page of the deployment. Scroll down to the Resource Monitor section to see the CPU and memory utilization of the model.
Resource Monitor section showing the CPU and memory utilization of the model
The next step is to create an XGBoost canary model that shares a percentage of the traffic with the main model.
Navigate to the Dashboard of the deployment and click Add Canary.
In the Update Deployment Wizard, configure the default predictor as follows:
Runtime: XGBoost
Model URI: gs://seldon-models/xgboost/iris
Model Project: default
Storage Secret: (leave blank/none)
Canary Traffic Percentage: 10
Model Name: iris
Canary Deployment Wizard
Click on Next for the remaining steps, then click Launch.
While the canary model is being launched, the deployment status changes to an Updating state.
When the canary model is launched successfully, the deployment status becomes Available.
Create a new canary deployment with the XGBoost model and roughly 10% of the traffic is sent to it.
This time, we will create a new load test with the canary model running and observe the requests and metrics for both models. You can use either the same JSON payload from the previous load test or construct a new one with different values or number of predictions.
After the second load test has started, you can monitor the upcoming requests and their responses by navigating to the Requests page of the deployment. Since we have two models running, you can choose to filter the requests by the model name to see the requests and responses for each model. In order to see the requests and responses for the canary model, you can filter the requests by clicking on the reverse pyramid icon, then click on the Node Selector dropdown, and, finally, select the canary predictor.
Requests page showing prediction requests and their responses for the canary model
You can also monitor live requests metrics for the both models if you navigate back to the Dashboard page of the deployment, and scroll down to the Live Requests section. In this screenshot you can see the number of requests per second and the average latency for both models. As expected, the main model is receiving more requests than the canary model, so the number of requests per second for the main model is higher.
Live Requests section showing the number of requests per second and the average latency for both models
Furthermore, you can monitor the utilization metrics for both models in the Dashboard page of the deployment. Scroll down to the Resource Monitor section to see the CPU and memory utilization for both models.
Resource Monitor section showing the CPU and memory utilization of the model
Great! Now we have observed the requests and metrics for both models. If we are happy with how the canary model is performing, we can promote it to become the main model.
Navigate to the Dashboard of the deployment and click on the Promote Canary button.
In the Promote Canary dialog, click Confirm to promote the canary model to the main model.
If the canary model is promoted successfully, the deployment status will become Available.
The canary model has been successfully promoted to be the main model of the deployment
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Start Load Test
Observe the prediction requests and their responses
Observe the utilization metrics for the model
Deploy a Canary model
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
Note: The deployment status represents the overal status of the deployment, including the main and canary models.
Load test the canary model
Remember that roughly 10% of the traffic is sent to the canary model. If, however, the canary model is not available, all the traffic is sent to the main model.
Observe the prediction requests and their responses for both models
Observe the utilization metrics for both models
Promote the Canary model
Model Accuracy
Iris is the genus of flower which contains 3 species: setosa, versicolor, and virginica. This demo is based on iris classification model based on flower properties like sepal length, sepal width, petal length, and petal width. The species are also the classes that will be used for the classification. Here we will:
Set up a metrics server for the pre-existing iris classifier deployment
Send a request to get an iris classification
Tabular Explanations
In this demo we will:
Create a deployment which can be used to classify tabular data
Create an explainer that will offer insight into why a particular prediction was made for a given input
As we saw the prediction response was versicolor (the second class of iris). In numeric form the response is,
A feedback request consists of the returned value in a response (response.data on line 5) and a truth value (truth.data on line 12), for that data. For example, a true-positive feedback request looks like this:
We'll send several feedback requests to the metrics server, that we set-up earlier, for various scenarios.
Set-up the environment variable(s) for the request:
export CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
(Recommended) You may also need to set-up for the Authorization header:
Return to the predict page, and paste in the prediction, as you did above
Click the Copy as curl button, and copy the value that looks like Bearer <authorization header>
Set the environment variable:
Scenario 1: True-positive:
Request without Authorization header:
Request with Authorization header:
Response:
Scenario 2: False-positive:
The difference from the previous scenario is the in the truth value:
Request without Authorization header:
Request with Authorization header:
Response:
Having done a prediction, metrics will begin to become available.
Go to the monitor screen's Prediction Accuracy tab to view all the metrics.
Set the time range to view the metrics, using the "From Time" and "To Time" selectors.
You can see metrics like accuracy, precision, recall and specificity here. Notice the drop in accuracy metrics after the false feedback was received.
Now we will submit a feedback as a batch using the Batch Processor component.
We will use two files, each containing 10k feedback instances:
90% success rate feedback file
40% success rate feedback file
We need to upload the files to MinIO's data bucket. For details on interacting with MinIO UI please see Batch Demo.
Once the files are in the MinIO bucket, go to the Batch Jobs screen using either the navigation bar on the left side or with the button on the model dashboard. For each of feedback-input-90.txt and feedback-input-40.txt files, submit a batch job. Wait for the first to complete before proceeding to the next. Submit batch request using the following form values:
Input Data Location: minio://data/feedback-input-40.txt
Output Data Location: minio://data/output-data-40-{{workflow.name}}.txt
Number of Workers: 15
Number of Retries: 3
Batch Size: 1
Minimum Batch Wait Interval (sec): 0
Method: Feedback
Transport Protocol: REST
Input Data Type: Raw Data
Object Store Secret Name: minio-bucket-envvars
Resources: Leave defaults
Now go to the monitor view and observe how metrics value evolve over time.
This demo requires Knative installation on the cluster as the metrics server will be installed as a kservice. Also, the metrics server only works with classification models in this version.
These model metrics are only supported for pre-existing Seldon Core 1 deployments with the Seldon inference protocol. Seldon now recommends adopting the industry-standard Open Inference Protocol (OIP) in preference to the Seldon protocol. Seldon has a separate module to support the calculation of model performance metrics with the Open Inference Protocol. Please contact your customer success representative to learn more.
Setup Metrics Server
Make Predictions
Send Feedback
Monitor accuracy metrics on the Monitor Screen
Submit batch feedback using Batch Processor component
Edit the model metadata to update the prediction schema for the model. The prediction schema is a generic schema structure for machine learning model predictions. It is a definition of feature inputs and output targets from the model prediction. Learn more about the predictions schema at the ML Predictions Schema open source repository. Use the income classifier model predictions schema income-classifier-prediction-schema.json to edit and save the model level metadata.
Click the model income-classifier that you registered.
Select "income-classifier" model on the Model Catalog page
Click Edit Metadata to update the Prediction schema field associated with the model using the contents of prediction schema income-classifier-prediction-schema.json.
Model's metadata wizard
Click Save Metadata.
Deploy the income classifier model from the catalog into an appropriate namespace
In the Model catalog, select Deploy of the Action dropdown.
"Deploy" action in the dropdown on the Model Catalog page
Enter the deployment details in the deployment creation wizard and click Next:
Name: income-classifier
Namespace: seldon
Type: Seldon Deployment
The predictor details should already be filled in from the model catalog except the Model Name field. Use income for the Model Name field.
Model Name: income
Income classifier deployment predictor
Click Next for the remaining steps, then click Launch.
Click the income-classifier deployment that you created.
In the deployment dashboard, click Predict in the left pane.
In the Predictpage, click Enter JSON and paste the following request payload:
Click Predict.
The Seldon Enterprise Platform supports several explanation methods for prediction requests. In this demo, you can learn about using these two methods:
The Anchor Explainer can be used to generate explanations for tabular data, text, and image classification models. Whereas, the Kernel SHAP Explainer can be used to generate explanations for only tabular data classification models. You can use the tabular data classification capabilities of these methods because the income-classifier deployment is considered a tabular data classification model.
In the income-classifier deployment dashboard, click Add in the MODEL EXPLANATION card.
In the Model Data Type tab, select the Tabular model data type option and click Next.
In the Explainer Types tab, select the Anchor explainer type option and click Next.
In the Explainer URI tab, enter the following details:
Click Next for the remaining steps, then click Launch.
If your explainer is launched successfully, both the deployment and the explainer will show an Available status.
Once the explainer of choice is successfully launched, you can now generate explanations for the prediction request made earlier.
Navigate to the Requests page using the left navigation drawer.
Click on the View explanation button to generate explanations for the request.
The Anchor explanation reveals which features most contributed to the classification of 'Low Income'. Here, for around 87% of the training data, a 'Relationship' of 'Unmarried' correlates to Low Income. Having a 'Sex' of 'Female' also correlates but is a less strong indicator. At the bottom-most screenshot, we are shown other examples of Low Income data points. This helps to reveal the patterns that the model is relying on to make predictions.
The explanation page showing the original prediction request and the predicted class - Low Income
The explanation summary consisting of the key contributing features towards the predicted class, and their request input values
The Anchor metrics of the explanation: Precision and Coverage
Perturbed variation samples of the prediction request which comply with the prediction
The Kernel SHAP explanation, on the other hand, reveals the top 6 contributing features towards the predicted class of 'Low Income'. The features and their corresponding SHAP values shown in the purple bars indicate the most influential features towards the predicted class, whereas the blue bars indicate the features contributing the most towards the opposite class of 'High Income'. Here, the most influential 'positive' features are 'Marital Status' of 'Separated', followed by 'Sex' of 'Female, 'Race' of 'Black, and others. The most influential 'negative' features are 'Occupation' of 'White-Collar', 'Hours per week' of '60', 'Age' of '53', and others.
The explanation page showing the original prediction request and the predicted response
The explanation summary consisting of the key contributing features towards the predicted class, and the correspoding SHAP values
Congratulations, you've created an explanation for the request! 🥳
Why not try our other demos? Ready to dive in? Read our operations guide to learn more about how to use Enterprise Platform.
Note: This demo uses a model trained to predict high or low income based on .
Register an income classifier model
Configure predictions schema for classifier
Launch a Seldon Deployment
Make a prediction
Add an Explainer
Add an Anchor Explainer
Add a Kernel SHAP Explainer
Note: If in the previous step you added the Anchor explainer, and you would like to use the Kernel SHAP explainer instead, you can do so by removing the Anchor explainer first. You will need to click on the ellipsis inside the Model Explanation card, select the Delete button, and confirm the deletion of the Anchor explainer.
Explain a prediction
An explanation made using the Anchor Explainer
An explanation made using the Kernel SHAP Explainer
The Model Name is linked to the name described in the model-settings.json file, located in the Google Cloud Storage location. Changing the name in the JSON file would also require changing the Model Name, and vice versa.
