Deployment models for Seldon Enterprise Platform

When it comes to deploying Seldon Enterprise Platform for a production workload, there are different deployment models that can apply depending on your use case. For example, you can consider the following questions:

• Do you want to support multiple environments (e.g. production, staging, etc.)? Will these be split across multiple clusters? Or across different namespaces?

• Are you planning on supporting multiple tenants in your infrastructure? How will you manage permissions for each one of them?

This section will showcase some of the most common deployment models to set up Seldon Enterprise Platform in your cluster.

Multiple environments

When you are managing machine learning models, a common use case is to test them first on some kind of test environment, isolated from production workloads. These test environments are usually known as staging, qa, pre-prod, etc.

In some cases, these environments will form a deployment (or continuous delivery) pipeline, where each environment is considered more stable than the previous one:

Dev >... > Staging > Production

Seldon Enterprise Platform allows you to have multiple environments by separating them as namespaces or clusters.

Namespace environments

The official Kubernetes docs . Therefore, it makes sense to treat each separate environment as a separate namespace. Following the example above, that would mean creating a dev, staging and production namespaces.

By default, Seldon Enterprise Platform will treat each namespace as a separate environment. Therefore, this is the suggested approach.

Out of the box, this gives you easy control over:

• Where is each model deployed (on which environments).

• Who is .

• Visibility over all your models across every environment from a single instance of Seldon Enterprise Platform.

Cluster environments

When you are dealing with multiple environments, it makes sense to keep them completely isolated of each other. That way, any cluster-wide effects will only affect one of them.

Using separate Kubernetes clusters for each of your environments allows you to achieve a higher-level of separation than namespaces. However, this separation also comes with a few caveats:

• All your infrastructure resources will need to be duplicated and managed across every environment (e.g. auth, ingress, logging, metrics, ...).

• Operations spanning multiple environments become more complicated (e.g. promotion of models between different environments).

This approach is usually recommended to test new versions of cluster-wide components at the infrastructure level. For example, a new version of Seldon Enterprise Platform or a new ingress system. However, due to the caveats listed above, we don't suggest this approach to segregate different versions of your models.

Having said so, you can still leverage the GitOps paradigm to communicate between different clusters.

Seldon Enterprise Platform has built-in app analytics, which help to improve the product. These analytics focus on tracking certain events that get triggered from the UI, for example user login.

Disable app analytics

You can disable the app analytics through the enableAppAnalytics variable of the Seldon Enterprise Platform Helm chart. To disable it just add the following to your Helm variables:

Events tracked

User login attributes

Operating Seldon Enterprise Platform in your cluster

These pages go into detail on best practices for operators or admins managing a Seldon Enterprise Platform cluster.

• App Analytics

• Authorization

Deployment source control and syncing with Gitops

Seldon Enterprise Platform can recognise that certain namespaces are to be maintained using GitOps. It will look for the label seldon.gitops: enabled.

If the seldon.gitops label is not present or is disabled then new deployments and modifications in the namespace will be pushed directly to the Kubernetes cluster.

If the seldon.gitops label is present then Enterprise Platform will look for an annotation named git-repo with a git URI. It uses a service account token for accessing the repo which is stored in a Secret installed with Seldon Enterprise Platform using the Helm chart (along with username and email). Enterprise Platform will add metadata to any commits it makes, including recording which dex user took the action.

Seldon Enterprise Platform can display an Audit Log for each deployment where it reads back git commits and makes the changes and metadata visible:

If the user is permitted then the state can also be restored to a previous commit.

See the for more on how GitOps works.

Underlying kubernetes resources and debugging

Seldon Enterprise Platform allows a view of the underlying kubernetes resources created by the deployed models. This view displays the pods, deployments and services created for each component of the Seldon Deployment/Inference Service custom CRDs.

Also for the pods created for each model component, you can dive into the live container logs and debug relevant issues if any.

Summary metrics are available showing usage over time. These can be accessed from 'Usage' on the top-right menu.

These metrics differ from those shown for individual models on their pages. The summary metrics show multiple models for a namespace or cluster.

By default if the cluster-level is chosen then all models are counted for all namespaces. If you wish to restrict to namespaces visible to the user, set the FILTER_USAGE_MON_NAMESPACES environment variable to true.

Resources are shown broken down by model. The type of resource of interest can be chosen. The containers resource is shown with a comparison against what your license allows for.

Models are stacked on top of each other in the breakdown when there are multiple

Seldon Enterprise Platform can create and manage secrets for models or artifacts stored in private buckets, or custom container images stored in a private registry.

If OPA is enabled, you will need write permission to the relevant namespace in order to create or delete secrets in it.

Management Methods

Seldon Enterprise Platform supports the management of secrets through both its UI and its API.

Bucket Secrets

Bucket stores, also called blob or object stores, provide storage for arbitrary data files. These might be binary files for serialized models, JSON or other textual formats for model settings, or even inference requests for batch jobs or reference data. Examples of bucket stores include Amazon S3, Azure Blob Storage, and Google Cloud Storage (GCS).

Storage buckets can be publicly accessible, such as Seldon's bucket, or they can be private and protected with some form of secret or access credentials.

Bucket secrets are used in Seldon for accessing various types of data, such as model artifacts and inference data from private bucket stores. It is a that actually uses the secret to perform the download. Secrets need to be provided in a format that the storage initializer understands. The following sections discuss these uses of bucket secrets in more detail.

Seldon Core 1

The default storage initializer for Core 1 is , although custom initializers can be used instead.

You can easily configure Rclone-compatible bucket secrets for both S3 and GCS from the UI, or you can also create generic Rclone bucket secrets, which unlocks a wide variety of providers, such as Microsoft Azure Blob Storage, Dropbox, and many more.

Seldon Core 1 uses environment variables to pass configuration to the storage initializer for a model. As such, the keys in a Core 1 bucket secret must be valid environment variables for the initializer.

When creating a bucket secret, you will need to specify a Remote name which will be used to create a secret called <remote>-bucket-envvars. This remote name is your choice and determines how you will refer to your bucket provider in storage URIs, i.e. <remote>://<path to bucket object>.

Seldon Core 2

Like Seldon Core 1, both Rclone and custom storage initializers can be used.

When creating a bucket secret, you will need to specify a Remote name which will be used to create a secret called <remote>-bucket-params. This remote name is your choice and determines how you will refer to your bucket provider in storage URIs, i.e. <remote>://<path to bucket object>.

Inference Data

Inference data can be retrieved from and written to storage buckets via batch jobs and reference data uploads.

Batch jobs and reference data jobs in Seldon Enterprise Platform use the same approach as for bucket secrets. Namely, they use environment variables to pass configuration from bucket secrets to a storage initializer. The format is exactly the same as for Core 1 and the same secrets can be used.

The default storage initializer for batch and reference data jobs is Rclone.

Registry Secrets

Private container registries can be used when deploying a custom runtime model.

When creating a registry secret, you need to specify the Secret name, which is your choice, and the Account credentials.

The format of container registry credentials can be seen by running the following example. For further information on creating and interacting with registry secrets, please refer to the .

Using Secrets

Once the secrets have been created, you can use them in the Seldon Enterprise Platform UI, e.g. in the Deployment Wizard, and through the API.

For bucket secrets, you need to use the Remote name you chose as the URI prefix. For example, if you named your remote minio then your URI should start with minio://.

Integration Options

Seldon Enterprise Platform integrates with Single Sign-On (SSO) providers for authentication.

The SSO system may also host the identities, or it may integrate to an identity provider (IdP). The IdP is often a source of truth for users across the organisation.

User details can be used for . Filtering is either by user or by group.

In order for group-based filtering to be used, groups need to be available. These come from an IdP but not all IdPs support groups or support groups in integration to SSO.

Seldon Enterprise Platform has a built-in health check that aims to monitor the availability of its surrounding ecosystem as well as troubleshoot problems when they arise. This can be done as an intermediary step before checking the logs or contacting us.

Where to find the Health Check Monitor

You can find the health check by clicking on the user icon on the top right of the screen.

Then you can select Health Check

Single sign-on (SSO) is a property of identity and access management (IAM) that enables users to securely authenticate with multiple applications by logging in only once (with just one set of credentials). For SSO, Seldon Enterprise Platform relies on the OIDC protocol that handles authentication through JSON Web Tokens and a central identity provider to verify user identity.

A typical enterprise SSO flow for web based applications is as follows

As shown in the diagram, Seldon Enterprise Platform sso integration works if any enterprise application on domain1.com and a Seldon Enterprise Platform installation on domain2.com is connected to a common auth Server on domain3.com.

Currently, Seldon Enterprise Platform SSO can be configured in

Managing Kubernetes namespaces for Seldon Enterprise Platform

Namespaces are where your models are deployed. You can set up GitOps and access controls on a per-namespace basis.

For an introduction to namespaces in Kubernetes, please refer to the official documentation. The rest of this guide discusses namespaces in the context of Seldon Enterprise Platform.

Namespace Detection

Seldon Enterprise Platform requires namespaces to be labelled for use with ML deployments. Otherwise, it will not detect them and so cannot manage resources within them.

You can create a new namespace and make it visible to Enterprise Platform using the following commands:

This is the minimum needed to use a namespace.

GitOps

Seldon Enterprise Platform supports GitOps with ArgoCD to manage deployments to a namespace through git.

A namespace can be set up for GitOps with Seldon Enterprise Platform by providing suitable metadata. See for a detailed guide on how to achieve this.

Authorization Policies

Seldon Enterprise Platform supports both user-based and group-based authorization policies to control visibility and permissions for namespaces. See the and guides for more details.

Namespaces that the user has access to appear in a drop-down namespace selector in the top-right:

From there the user can search for or choose to drill into deployments:

Multi-Tenancy

In a multi-tenant setup, Enterprise Platform is not automatically provided with permissions to manage multiple namespaces. Instead, each namespace needs further permissions to be defined. Details on this can be found in the .

Special Namespaces

The seldon-logs namespace is used for infrastructure for inference request logging. It is a special namespace and its setup is covered in the . This namespace should not be used for ML deployments.

Seldon Enterprise Platform supports the storage and management of model metadata. This includes project name, artifact URI, ML runtime, and prediction schema, amongst other things.

Model metadata is dependent on Postgres. For more information on configuring Postgres with Seldon Enterprise Platform, see the installation instructions.

You can create and manage model metadata through the Enterprise Platform UI and API. Enterprise Platform will also watch for deployments in the Kubernetes cluster and sync this information into the metadata store. In this case, the Platform will only populate basic information, but this can then be enriched via the UI and API as with any other model metadata.

Queries

Seldon Enterprise Platform provides an advanced query language that allows you to make sophisticated search queries over your model metadata.

Simple fields can be compared using SQL-like comparison and partial match operators. Nested fields can be compared by first using index notation to extract simple fields. More complex expressions can be formed with the AND and OR operators and parentheses. Note that the AND and OR operators are treated as having equal precedence.

Field names and operators are not case-sensitive. Field values are case-sensitive, however, and should not be wrapped in quotes. The list of valid fields to query is:

Examples

API

Using the model metadata API, you can specify a string query the same way you would in the UI, e.g.

Concepts and configuration for authorization in Seldon Enterprise Platform

User Identity

Once a user has been authenticated, they have an identity in the form of an OIDC ID token. The token defines a user ID and groups to which the user belongs. This information can be used to authorize their actions.

The user ID is determined by a claim in the OIDC token. This defaults to preferred_username, but can be configured through the environment variable USERID_CLAIM_KEY in install-values.yaml. This can be used for both human users and service accounts. If it is not possible to share claim keys for both, it is possible to use the environment variable SA_ID_CLAIM_KEY for service accounts.

The groups are determined by a claim in the OIDC token. This defaults to groups, but can be configured through the environment variable GROUPS_CLAIM_KEY in install-values.yaml.

Permission Evaluation

Every call to the Seldon Enterprise Platform API is protected by authorization checks. This happens regardless of whether you are interacting with it via the UI, via the Python SDK, or directly.

Seldon Enterprise Platform operates on a restrictive model. This means that a user is assumed to have no permissions except those which they have been explicitly granted.

The effective list of permissions that a user has is the union of:

• Those that are granted to them by their user ID.

• Those that are granted to all users.

• Those that are granted to one or more of the groups to which the user belongs.

Resource Actions

There are three actions users can be granted for resources:

• Read for listing available resources and viewing particular ones.

• Write for creating, deleting, and modifying resources.

• Grant for granting other users permissions to specific resources.

These actions do not imply one another. For example, having write access does not also grant read access. Each action is handled completely independently in the OPA policies, and each grant must be specified explicitly.

Resource Scopes

Resources in Enterprise Platform are grouped into three categories:

• that belong to a specific Kubernetes namespace. This includes ML deployments and artifact secrets.

• , including models in the and deployments using these. Projects are a Seldon Enterprise Platform concept, but not a Kubernetes one.

• , such as authorization policies themselves!

Resources can sometimes be both project-scoped and namespace-scoped. For example, an ML deployment is namespace-scoped because it exists in Kubernetes, and is also project-scoped based on models it uses. When this happens, a user needs access to both the namespace and all projects associated with that deployment to be able to view or interact with it.

Projects

A project defines a logical grouping of related resources.

At present, models are the only resources that are explicitly associated with projects. Any model that does not explicitly declare its associated project will be implicitly assigned to the default project.

A project can have multiple models associated with it, but each model is only associated with a single project. Projects-to-models is a one-to-many relationship.

As deployments reference models, they are also protected by project-based policies. Deployments can reference multiple models, so can be associated with multiple different projects. A user must have suitable permissions for all projects associated with a deployment in order to interact with it.

Namespaces

A namespace defines a physical grouping of resources.

Any resources related to deployments will exist in a namespace, including deployed models, explainers, detectors, canaries, shadows, and artifact storage secrets.

A namespace can have many resources associated with it, but each resource is only associated with a single namespace. Namespaces-to-resources is a one-to-many relationship.

System

System resources are resources that relate to administrative concerns for the Seldon Enterprise Platform platform, instead of ML models and deployments.

Currently the only system-level resources that can be protected by access controls in the Seldon Enterprise Platform API are authorization policies themselves. As a result, the initial policies must be defined by someone with access to the Kubernetes cluster.

Associating Scopes With Kubernetes Resources

As namespaces are a native Kubernetes concept, there is a natural association between namespaced resources and their associated namespace.

In contrast, projects are not a native Kubernetes concept and so Seldon Enterprise Platform must record the association between a resource and its project somewhere. It does this via Kubernetes metadata: annotations and labels.

When creating deployments through the Seldon Enterprise Platform UI, you can use the pre-filled value or specify the appropriate project for your model. When creating deployments via the Seldon Enterprise Platform API, whether directly or with the Seldon Enterprise Platform SDK, you must provide the appropriate project metadata in the resource definition. In either case, when no project is explicitly specified then the default project will be used.

For Seldon Core v1, projects are defined via annotations at the predictor level. Each predictor in a deployment's spec.predictors has an annotations field that can contain annotations of the following form:

Authorization Methods

Currently, there are two ways of defining permissions on the various resources served by Seldon Enterprise Platform:

• (recommended)

• (deprecated)

OPA policies are the recommended authorization method for Seldon Enterprise Platform as they can be used to authorize all .

OPA Policies

Seldon Enterprise Platform can use Open Policy Agent (OPA) policies to determine if a user has access to a resource. is popular, flexible open-source technology for defining policies. You can enable it by following the .

Enable OPA

To enable the use of OPA policies for the namespace and system scopes, set the following Helm variables in install-values.yaml:

As OPA policies cannot be used with , this should be disabled as shown above. However, Seldon Enterprise Platform still requires the seldon.restricted label to be present on namespaces in order to . When using OPA policies, it does not matter whether the value of this label is true or false.

If you also want to enable OPA for projects, set the following configuration in install-values.yaml:

Policy Management APIs

Seldon Enterprise Platform has API endpoints for managing OPA policies. These can be found in the .

By default, these are enabled and access can be granted to them on a per-user or group basis. If you want to disable these, set the following Helm value:

Policy Schema

OPA policies are defined in a JSON document as resource-action pairs for users or groups. This JSON document exists in a ConfigMap, as discussed in the .

The structure of policies is as follows:

The top-level fields of the document are the grant types: role_grants and user_grants. These are mandatory fields and correspond to groups and user IDs respectively.

Each grant type defines a map from user/group name to a list of permissions. These user or group names are called grant targets.

Each permission is defined as a map, describing an action on a resource scope. For example this permission allows read access to resources in a namespace called seldon:

Granting Access For All Users

For user grants, there is a special grant target called the star user and denoted *. This special target can be used to provide policies for all users.

To see how this can be used, please check the .

Resource Names

Resources are identified by the as a prefix and a name.

For the system scope, there is only one possible value:

• system/iam

For the project scope, aside from the default project the names are determined by you:

• project/default

• project/foo

• ...

For the namespace scope, the names should correspond to Kubernetes namespaces for deploying ML models:

• namespace/seldon

• namespace/seldon-gitops

• ...

Seldon Enterprise Platform supports globbing of resource names with an asterisk (*). It is valid to use globbing anywhere within the resource name, but not for the resource scope.

For example, these would all be allowed:

• "resource": "namespace/*"

• "resource": "namespace/seldon-*"

• "resource": "namespace/*-gitops"

On the other hand, these would all be prohibited:

• "resource": "*"

• "resource": "*/*"

• "resource": "*/seldon"

Examples

Giving everyone access to resources in a scope is possible using the special "star" user, *. An example of giving all users access to the default project looks like this:

Giving a group access to resources in a scope is possible using role_grants. An example of giving the data-scientist group access to the prod namespace looks like this:

Giving a user access to resources in a scope is possible using user_grants. An example of giving the alice user access to the system/iam scope looks like this:

An example of a complete policy file is given below.

In this example, users belonging to the data_scientist group will be able to create, modify, and read models from the iris project, as well as see deployments in the seldon namespace. However, they do not have permission to create or modify deployments in that namespace.

Members of the ml_engineer group can create deployments from the iris project in the seldon namespace. They are also allowed to interact with other resources in the seldon namespace, such as storage secrets.

Members of the ops group can view all resources in namespaces with the seldon prefix (including seldon itself), and can see the full Model Catalog.

The * user grant specifies that all users have read and write permissions for the default project and default namespace.

Only the user alice has access to the alice namespace.

Policy Operations

Interacting With Policies

The policy file can be modified in various ways:

• Using kubectl

• Using Seldon's policy_client command-line utility

• Using the permission management endpoints of the Seldon Enterprise Platform API

You can interact with the ConfigMap directly using kubectl. If your policies are defined in a JSON file called policy.json, you can turn it into a Kubernetes manifest called policy.yaml using the following command:

This resulting policy.yaml file can be tracked under version control if you wish. You can overwrite the existing policies by running the following:

The policy_client command-line utility can be downloaded from the . You will need to download the correct binary for your operating system and architecture, e.g. Linux AMD x64. For full documentation of the tool you can run policy_client --help.

The permission management API is provided via the /iam/policy endpoints discussed in the .

Seldon Enterprise Platform will automatically reload the policies if it detects that they have changed. This is dependent on how the file is mounted and how quickly Kubernetes propagates changes. If you want to ensure that Seldon Enterprise Platform definitely has the latest changes, you can restart it using:

Adding Policies

To add a new policy you can run:

The arguments target_groups and target_users can take multiple entries.

Deleting Policies

Similarly to adding, to remove an existing policy you can run:

Limitations

Using a ConfigMap for the policy file is flexible and easy to setup. However, it comes with a limitation which is true for all Kubernetes resources - it is , since that is the underlying etcd value limit.

If your use case requires a bigger size for the policy file consider mounting the file from somewhere else like an object store (S3, minio), or a database.

We are working on improving support for such use cases, which should be available in a future release.

Migrating From Namespace Labels

To migrate from using namespace labels for defining access requirements to a namespace you can use the following command:

It will create a ConfigMap called seldon-deploy-policies in the seldon-system namespace if one is not present. The name and namespace are configurable by flags. It will then populate policies corresponding to the namespace labels in the cluster. The command will not delete any existing policies in the ConfigMap.

To confirm the change has been successful you can check the ConfigMap by running:

Namespace Labels

Seldon Enterprise Platform can use labels on Kubernetes namespaces to determine if a user has access to resources within that namespace. Note that namespace labels can only be used for authorizing access to namespaces, but not to projects.

To authorize a user or a group to access resources in a namespace, add labels in the following manner along with the associated permission.

As an example, setting permissions on a namespace for user alice, service account sd_api and group ml_engineer would involve adding labels as follows,

Authorization Outside Seldon Enterprise Platform

Seldon Enterprise Platform runs in a Kubernetes platform which contains several other components. Access to these components should also be controlled. How this is done can vary based on your requirements and pre-existing access controls.

Seldon Core

One of the main features of Seldon Enterprise Platform is the ability to create Seldon Core deployments and allow users to make requests to these deployments. For requests to be made in a secure manner, these endpoints need to be authorized appropriately.

Seldon Enterprise Platform can automatically synchronize Istio authorization policies based on the permissions set in its OPA policies. Please refer to the for enabling this functionality.

This functionality is controlled by the rbac.opa.istioPolicySyncInterval Helm variable. You can enable synchronization by supplying a strictly positive value compatible with Go's time.ParseDuration . Using any other value, such as a negative duration or leaving the Helm parameter empty, will disable this functionality. To use the suggested interval of 5 minutes, please set the following in install-values.yaml:

Alternatively, the access to these endpoints can be manually controlled via the Istio configuration as described in the .

Other Components

As part of the , other components can be installed in your environment. Please refer to the official documentation for each such technology for details on how to configure authorization for them.

Configuration Reference

The full list of Helm values for authorization is given below. These can be set in install-values.yaml. Please refer to the above documentation for how to configure each setting.

Some features in Seldon Enterprise Platform require access to external storage providers, for example, to run batch jobs.

A storage initializer mechanism similar to the one used for Prepackaged Model Servers is used by Seldon Enterprise Platform. By default, Seldon Enterprise Platform uses the Rclone-based storage initializer. Rclone offers compatibility with over 40 different cloud storage products and is therefore a sensible default.

However, custom storage initializers may be used by modifying the storageInitializer values in install-values.yaml.

Configuration

Rclone storage initializers are configured using environmental variables passed as secrets in Seldon Enterprise Platform. These are configured per namespace.

Each remote storage provider needs to be configured using environment variables that follow the following pattern:

Once the remote is configured, the modelUri that is compatible with rclone takes the form:

For example modelUri: minio:sklearn/iris, or modelUri: gs:seldon-models/cifar10. Rclone will remove the leading slashes for buckets so this is equivalent to minio://sklearn/iris or gs://seldon-models/cifar10.

Below you will find a few example configurations. For other storage providers, please consult the .

Please note that you need the labels secret-type: bucket and seldon-deploy: "true" on your secret for it to be correctly picked up by Enterprise Platform. Without these labels it will not show in the secrets dropdown in the deployment creation wizard.

MinIO

This is an example of how to set up secrets so that Rclone may access .

Reference:

Public GCS configuration

Reference: .

AWS S3 with access key and secret

Reference:

Example AWS S3 with IAM roles configuration

Reference:

GCP/GKE

Reference:

For GCP/GKE, you will need create a service-account key and save it as a local json file.

First make sure that you have a gcloud service account ([SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com) that has sufficient permissions to access the bucket with your models (i.e. Storage Object Admin). You can check this using the .

Next, generate keys locally using the gcloud tool. This will create your service-account key file at gcloud-application-credentials.json:

Use the content of the locally saved gcloud-application-credentials.json file to create the rclone secret:

Custom Storage Initializer

If for some reason you would like to use a different storage initializer for batch jobs, e.g. KFServing storage initializer, you can set this by modifying install-values.yaml:

The corresponding secret for would also need to be created: