A tour of Seldon Enterprise Platform features.

• Deployment Dashboard

• Deployment Wizard

• Model Explanations

The Deployment Wizard guides you through setting up and configuring new deployments.

The Overview button on the top-left of your screen will take you to the list of current deployments, where the deployment wizard can be opened by clicking on the Create new deployment button on the top-right.

Deployment Details

In the Deployment Details page, you need to choose a Name and Namespace for your deployment. You also need to choose the deployment Type, which affects the following sections of this wizard.

Default Predictor

In the Default Predictor page, you will need to specify your model's Runtime and the Model URI where its artifacts are stored. The Runtime field is pre-filled with supported runtimes, and also includes options for Custom runtime configurations.

You can optionally provide a Storage Secret for private model artifacts. See for details.

You can also change the Model Project from the default project. Projects are a logical grouping of resources for.

Configuring Custom Models

In addition to pre-packaged runtimes, custom container images or custom inference artifacts can be configured in a similar manner.

Seldon Core v1

For deployments of type Seldon Deployment, you can choose the Custom runtime and specify the URI for your Docker image instead of a model artifact URI.

Seldon Core v2

For deployments of type Seldon ML Pipeline, you can select a custom inference server for your model by specifying Model Requirements as follows. Custom inference servers need to be set up manually, not through the Deployment Wizard.

See documentation for how to configure servers and models.

Optional Configurations

In the following pages, you can set optional configurations such as resource requirements, auto-scaling, and version comments for GitOps.

For deployments of type Seldon Deployment, you can additionally configure environment variables, inference request logging, and input/output transformers.

Deployment Summary

Before creating the deployment, you can review its Kubernetes manifest in the Launch Deployment page. Click on the LAUNCH button to proceed.

Rollout strategies for ML Models

The typical rollout strategy for kubernetes is a rolling update, where traffic is moved over from a live version to a new version when the new version is ready. For Machine Learning it is common to perform more complex rollouts where traffic is split between versions of a model in a customized way.

With a shadow traffic is duplicated so that it goes to both a main model and also a shadow model. The shadow is typically a newer version being tried out. The responses to requests that reach the end user are only from the main model, not the shadow. But the shadow requests can be logged and the shadow can be monitored. Within Seldon Enterprise Platform there are features to:

1. A wizard to add a shadow:

1. Visualizing metrics for both default and shadow models:

1. Promotion of the Shadow to be the main model.

1. GitOps integration for the whole process, so that all changes can be audited (see GitOps under Architecture)

2. Request logs include any request to the shadow, with the responses.

With a canary rollout traffic is split between a main model, which receives a majority of traffic, and a new version, which is given a fraction of traffic until it is decided whether to promote the model. The features for this can be seen in full in the Demos section of the documentation under Deploying, Load Testing and Canarying Seldon Core Servers. The key features in particular are:

1. A wizard to add a canary:

1. Visualizing metrics for both default and canary models:

1. Promotion of the Canary to be the main model.

1. Request logs include any requests that go to the canary, with the responses.

Monitor changes in real-world data distributions

When machine learning models are deployed in production, sometimes even minor changes in a data distribution can adversely affect the performance of ML models. When the input data distribution shifts then prediction quality can drop. It is important to track this kind of data drift. Drift detection in Seldon Enterprise Platform is powered by our open source library alibi-detect. Learn more about the data drift detection concepts on the .

Available Detection Methods

Understand why models give particular predictions

Model explanations allow you to understand why a machine learning model gave a particular predictions. Seldon Enterprise Platform allows you to use Black Box Model Explainers that can be used on models built using any technology.

Available Methods

Seldon Enterprise Platform supports a subset of the methods currently available in Alibi for Seldon Core deployments. Find documentation on and read about creating your

This guide walks you through the features available in the Deployment Dashboard. You can navigate to the dashboard for a specific deployment by clicking on it in the main Overview screen.

There is a menu on the left-hand side of the Deployment Dashboard, which allows you to navigate between various parts of the dashboard's functionality.

Dashboard

This top-level Dashboard page allows you to inspect and interact with your deployment.

On the top row, you can configure different monitoring and explainability components for your deployment. See and for details.

The deployment name and status are shown here, along with a logo indicating the deployment type (i.e. Seldon Deployment or Seldon ML Pipeline).

Clicking on the model name under the Pipeline Components/Deployment Components section will display a model metadata pane to the right.

Clicking on the View button on the right displays the Kubernetes manifest for your deployment, while the Delete button deletes the deployment altogether, including any monitoring and explainability components.

You can also perform actions on Canary and Shadow deployments. See for details.

If a canary deployment is present, traffic is split between this and the default deployment, with traffic percentages displayed on the top right of each deployment pane.

Request Monitoring

The Dashboard view also includes request monitoring panels for real-time request metrics.

Load Test

Within the Requests Monitor section, you can find the START A LOAD TEST button on the right. You can set the request JSON data here to start load testing, with the number of requests to send or the duration for which the requests are sent.

Note that it could take a little while to provision the load test.

Live Requests

The Live Requests section provides a summary of the traffic going through your deployment, including request rates, success/failure rates by status code, and average latency.

Resource Monitor

For deployments of type Seldon Deployment, there is a Resource Monitor section that reports metrics for CPU and memory usage with utilization rates and limits.

Predict

In the Predict page, you can make prediction requests by either providing a JSON object directly or uploading a file containing a JSON object. The response body is shown on the right, with the HTTP status code and response time.

In the lower left part of this page, you can click on the Copy as curl button to display commands to make the same request via a CLI.

Monitor

In the Monitor page, you can inspect various types of distribution trends and characteristics based on your inference requests. You can see the inference requests by clicking on the View Requests button, which takes you to the Requests page introduced in the next section.

Distributions

In the first tab, Distributions, you can see visualizations of feature distributions and statistics, as well as changes in these distributions over time. Clicking on the Add reference data button will open a wizard to import your reference data, which will be displayed alongside the distributions from the inference requests for comparison. Each row is collapsible with a click.

This functionality also enables you to draw comparisons for different combinations of features and time slices by choosing appropriate filters from the Filter button on the right.

See the following demos for a step-by-step guide for distributions monitoring.

Outlier Detection

In the second tab, Outlier Detection, you can see the timestamps and outlier/inlier groups for inference data. Red dots indicate outliers and blue dots indicate inliers.

See the following demos for a step-by-step guide for outlier detection using image classification models.

Drift Detection

In the third tab, Drift Detection, you can see the significance of changes in feature distributions over time. The Switch Metric slider will change the views between p-values and distance scores.

See the following demos for a step-by-step guide for outlier detection using tabular classification models.

Requests

In the Requests page, you can investigate individual inference requests and responses with explainers and outlier detectors. The results can be filtered by the menu on the right, where you can specify time ranges as well as request IDs.

If an outlier detector is configured, it will show an outlier score to the right. Likewise, if an explainer is configured, you can generate a model explanation from the View explanation button.

Resources

In the Resources page, you can inspect Kubernetes resources for Pods, Deployments, and Services, displaying relevant information such as replicas, namespaces, and cluster IPs.

The Resources page is only available for deployments of type Seldon Deployment.

Batch Jobs

In the Batch Jobs page, you can create batch jobs for existing data sources, e.g. from cloud storage. In the batch job creation wizard, you can specify input and output locations for your data, as well as batch size, number of parallel workers, and other relevant details.

Once the job is done, it will show up in this page with either Succeeded or Failed status.

Succeeded indicates that the steps specified in the batch jobs were executed without errors. Be sure to check the output data as the batch job can succeed even if the input file is empty, etc.

See the following demos for running batch jobs.

Audit Logs

If GitOps is enabled for the current namespace, the Audit Logs page will show the history of deployment manifest files, with differences, for actions such as adding new models and promoting canaries. Each action in the UI is registered as a commit, with a description in the Deployment History menu on the right.

Additionally, for deployments of type Seldon Deployment, you can restore the deployment to a particular state by clicking on the Restore State button.

The Audit Logs page is only available for deployments in GitOps namespaces. The namespace can be changed from the top right drop-down menu whilst viewing the Overview page.

Usage

In the Usage page, you can monitor various metrics such as model, CPU, memory, and container usage. These metrics and the time range can be selected from the drop-down menu on the right.

The Usage page is only available for deployments of type Seldon Deployment.

Seldon Enterprise Platform API

The REST API of Seldon Enterprise Platform lets you interact with your machine learning deployments programmatically. This allows you to build complex deployment pipelines to integrate Seldon Enterprise Platform with any upstream services.

If you already have access to a Seldon Enterprise Platform installation, you can visit the interactive API reference to learn more about the Seldon Enterprise Platform API and the endpoints that it exposes. The interactive API documentation can be accessed by clicking on your profile icon, and then clicking on API Docs. Alternatively, you can go directly to $ML_PLATFORM_HOST/seldon-deploy/swagger/, where $ML_PLATFORM_HOST needs to be replaced by the domain where your Seldon Enterprise Platform installation can be accessed.

Usage

The recommended way of interacting with the Seldon Enterprise Platform REST API is through its . However, you can also use plain cURL to send requests.

You can find some example usages below.

Authentication

All requests to the Seldon Enterprise Platform API must be authenticated. Therefore, before using the API you must obtain an authentication token. Note that the process to issue a new authentication token may change depending on your architecture and your OIDC provider.

Out of the box, the Python SDK supports a set of common authentication workflows. Each of these can be found under the seldon_deploy_sdk.auth package.

• OIDCAuthenticator: Allows you to authenticate against an OIDC-compatible provider, using the auth_code, client_credentials, and password flows.

• SessionAuthenticator: Allows you to authenticate against Dex, configured as an ingress-level authentication provider.

You can see some authentication examples below. Further details can be found in the .

Auth Code Flow

This is the recommended OIDC flow as it is the most secure for client applications like the Python SDK. It is suitable for interactive workflows where a human user is involved.

It is a multi-stage process:

• Users log in to their OIDC provider directly through a browser

• After successfully logging in, the user is redirected to the service they want to access, i.e. Seldon Enterprise Platform

• The user is shown a one-time access code

The code-token exchange can optionally use client credentials for an added layer of security.

For this example, we will assume that the authorization-code flow is supported and that the OIDC client is confidential.

In Keycloak, this can be done by setting Standard Flow Enabled, ensuring the client's Access Type is confidential, and checking the Valid Redirect URIs list includes /seldon-deploy/auth/callback through the client dashboard in the admin UI.

Client Credentials Flow

This should only be used in trusted environments as there is no user authentication involved. It may be suitable for automated pipelines such as for CI workflows.

This is a simple request-response flow:

• The client sends a request to its OIDC provider with its client ID and a secret known only to it and the identity provider

• The OIDC provider responds with a long-lived token that can be used for future requests

For this example, we will assume that the client credentials flow is supported and that the OIDC client is confidential.

In Keycloak, this can be done by setting Service Accounts Enabled and ensuring the client's Access Type is confidential through the client dashboard in the admin UI.

Password Grant Flow

This flow is generally not recommended as it requires a user's credentials to be exposed to a client application. Users may be tempted to store their username and password in an unencrypted file for convenience, and these credentials are at risk of being leaked through logging or because they are held unencrypted in the client.

This is a simple request-response flow:

• The user provides their username and password to the client

• The client sends a request to the OIDC provider with the user's credentials

• The OIDC provider responds with a long-lived token that can be used for future requests

The request can optionally use client credentials for an added layer of security.

For this example, we will assume that the password grant flow is supported and that the OIDC client is confidential.

In Keycloak, this can be done by setting Direct Access Grants Enabled and ensuring the client's Access Type is confidential through the client dashboard in the admin UI.

Session Authentication

This approach does not require explicitly configuring an endpoint for the authentication provider. Requests to Seldon Enterprise Platform are intercepted at the ingress to the cluster and authenticated using cookies.

This is a simple request-response flow:

• The user provides their username and password to the client

• The client sends a request to Seldon Enterprise Platform which is intercepted and redirected to the auth provider

• The client sends a request to this auth provider with the user's credentials

For this example, we will assume the use of Dex as the identity provider. This is the .

Versioning

The API endpoints are versioned to avoid clashes between different versions of the API. The current version of the API is v1alpha1, which means that breaking changes are still highly likely to happen. Once the current version graduates to stable, it will be renamed to v1.

Note that this versioning schema is similar to the one followed in Kubernetes.

API Reference (v1alpha1)