New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Amazon managed Apache Kafka (MSK). You can control access to your Amazon MSK clusters using sign-in credentials that are stored and secured using AWS Secrets Manager. Storing user credentials in Secrets Manager reduces the overhead of cluster authentication such as auditing, updating, and rotating credentials. Secrets Manager also lets you share user credentials across clusters.

Setting up SASL/SCRAM authentication for an Amazon MSK cluster

To setup SASL/SCRAM in an Amazon MSK cluster, please follow the guide from Amazon's Official documentation.

Do not forget to also copy the bootstrap.servers which we will use it in our configuration later below for Seldon.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

kubectl create secret generic aws-msk-kafka-secret -n seldon-mesh --from-literal password="<MSK SASL Password>"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-aws-msk-kafka-sasl-scram.yaml.tmpl
kafka:
  bootstrap: <msk-bootstrap-server-endpoints>
  topics:
    replicationFactor: 3
    numPartitions: 4

security:
    kafka:
      protocol: SASL_SSL
      sasl:
        mechanism: SCRAM-SHA-512
        client:
          username: < username >
          secret: aws-msk-kafka-secret
          passwordPath: password
      ssl:
        client:
          secret:
          brokerValidationSecret:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• Please check Amazon MSK Troubleshooting documentation.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Installing Ansible

Provided Ansible playbooks and roles depends on Ansible Collection for performing kubectl and helm operations. Check Ansible [documentation] for further information.

To install Ansible and required collections

We have tested provided instructions on Python 3.8 - 3.11 with following version of Python libraries

and kubernetes.core collection in version 2.4.0.

Once installed you can use the following Playbooks that you will find in folder of Seldon Core 2 repository.

You also need to have installed CLI.

Installing Seldon Core 2 using Ansible

One-liner local kind install (from scratch)

This will create a Kind cluster and install ecosystem dependencies (kafka, prometheus, opentelemetry, jager) as well as all the seldon-specific components. The seldon components are installed using helm-charts from the current git checkout (../k8s/helm-charts/).

Internally this runs, in order, the following playbooks (described in more detail in the sections below):

• kind-cluster.yaml

• setup-ecosystem.yaml

• setup-seldon.yaml

You may pass any of the additonal variables which are configurable for those playbooks to seldon-all. See the Customizing Ansible Instalation section for details.

For example:

Running the playbooks individually, as described in the sections below, will give you more control over what gets run and when (for example, if you want to install into an existing k8s cluster).

Create Kind Cluster

It is recommended to first install Seldon Core 2 inside Kind cluster. This allow to test and trial the installation in isolated environment that is easy to remove.

Setup Ecosystem

Seldon runs by default in seldon-mesh namespace and a Jaeger pod and and OpenTelemetry collector are installed in the chosen namespace. Run the following:

The most common change will be to install in another namespace with:

Install Seldon Core 2

Run the following from the ansible/ folder:

If you have changed the namespace you wish to use you will need to run with:

Customizing Ansible Installation

Ecosystem configuration options

The ecosystem setup can be parametrized by providing extra Ansible variables, e.g. using -e flag to ansible-playbook command.

For example run the following from the ansible/ folder:

will only install Kafka when setting up the ecosystem.

Seldon Core 2 configuration options

Custom Seldon images and private registries

By default, the container images used in the install are the ones defined by the helm charts (referring to images publicly available on dockerhub).

If you need to customize the images (i.e pull from private registry, pull given tag), create a custom images config file following the example in playbooks/vars/set-custom-images.yaml and run with:

If, instead of pulling images from an external repository you want to build certain components locally, please read README.dev.md

Private registries

When using private registries, access needs to be authenticated (typically, via a service account key), and the k8s cluster will need to have access to a secret holding this key to be able to pull images.

The setup-seldon.yaml playbook will create the required k8s secrets inside the cluster if it is provided with an auth file in dockerconfigjson format. You provide the path to this file by cusomizing the custom_image_pull_secrets.dockerconfigjson variable and define the secret name via custom_image_pull_secrets.name in the custom images config file (the one passed to the playbook via -e @file).

By default, docker creates the dockerconfigjson auth file in ~/.docker/config.json after passing the service-account key to docker login.

The docker login command would look like this (key in json format):

or, for keys in base64 format:

Saving helm-chart customisations

Because the additional custom images config file (starting from the playbooks/vars/set-custom-images.yaml example) overrides values in the helm-charts available in the repo, there's also a playbook option of saving those overrides as a separate values file, which could be used if deploying manually via helm.

This is controlled via two variables:

You can either pass those within the custom images config file or directly when running the playbook. For example, for just saving the helm-chart overrides (without installing seldon components), you would run:

Please note that when deploying outside ansible via helm using this saved overrides file, and using private registries, you will have to manually create the service-account key secret with the same name as the one defined in your custom image config file under custom_image_pull_secrets.name.

Uninstall

To fully remove the Ansible installation delete the created Kind cluster

This will stop and delete the Kind cluster freeing all of the resources taken by the dev/trial installation. You may want to also remove cache resources used for the installation with

We provide several Helm charts.

• seldon-core-v2-crds : cluster wide install of custom resources.

• seldon-core-v2-setup : installation of the manager to manage resources in the namespace or clusterwide. This also installs default SeldonConfig and ServerConfig resources which allow Runtimes and Servers to be installed easily on demand.

• seldon-core-v2-runtime : this installs a SeldonRuntime custom resource which creates the core components in a namespace.

• seldon-core-v2-servers : this installs Server custom resources which provide example core servers to load models.

• seldon-core-v2-certs : a default set of certificates for TLS.

The Helm charts can be found within the k8s/helm-charts folder and they are published

Assuming you have installed any ecosystem components: Jaeger, Prometheus, Kafka as discussed you can follow the following steps.

Note that for Kafka follow the steps discussed

Add Seldon Core 2 Charts

Install the CRDs

Install the Seldon Core 2 Components

This will install the operator namespaced so it will only control resources in the provided namespace. To allow cluster wide usage add the --set controller.clusterwide=true, e.g.

Cluster wide operations will require ClusterRoles to be created so when deploying be aware your user will require the required permissions. With cluster wide operations you can create SeldonRuntimes in any namespace.

Install the default Seldon Core 2 Runtime

This will install the core components in your desired namespace.

Install example servers

To install some MLServer and Triton servers you can either create Server resources yourself or for initial testing you can use our example Helm chart seldon-core-v2-servers:

By default this will install 1 MLServer and 1 Triton in the desired namespace. This namespace should be the same namespace you installed a Seldon Core Runtime.

Uninstall

Remove any models, pipelines that are running.

Remove the runtime:

Remove the core components:

Remove the CRDs

Seldon will run with .

At present we support mTLS authentication to MSK which can be run from a Kubernetes cluster inside or outside Amazon. If running outside your MSK cluster must have a public endpoint.

Considerations

Public Access to MSK Cluster

If you running your Kubernetes cluster outside AWS you will need to create a .

You will need to setup Kafka ACLs for your user where the username is the CommonName of the certificate of the client and allow full topic access. For example, to add a user with CN=myname to have full operations using the kafka-acls script with :

You will also need to allow the connecting user to be able to perform admin tasks on the cluster so we can create topics on demand.

You will need to allow group access also.

Create TLS Kubernetes Secrets

Create a secret for the client certificate you created. If you followed the you will need to export your private key from the JKS keystore. The certificate and chain will be provided in PEM format when you get the certificate signed. You can use these to create a secret with:

• tls.key : PEM formatted private key

• tls.crt : PEM formatted certificate

• ca.crt : Certificate chain

To extract certificates from truststore do:

Add ca.crt to a secret.

Example Helm install

We provide a template you can extend in k8s/samples/values-aws-msk-kafka-mtls.yaml.tmpl:

Copy this and modify by adding your broker endpoints.

Troubleshooting

No messages are being produced to the topics

Set the kafka config map debug setting to "all". For Helm install you can set kafka.debug=all.

If you see an error from the producer in the Pipeline gateway complaining about not enough insync replicas then the replication factor Seldon is using is less than the cluster setting for min.insync.replicas which for a default AWS MSK cluster defaults to 2. Ensure this is equal to that of the cluster. This value can be set in the Helm chart with kafka.topics.replicationFactor.

Prerequisites

• Ensure that the version of the Kubernetes cluster is v1.27 or later. Seldon Core 2 supports Kubernetes versions 1.27, 1.28, 1.29, 1.30, and 1.31. You can create a cluster on your local computer for testing with .

• Install the ecosystem components using .

Table

Install

To install Seldon Core 2 from the , you can choose one of the following methods:

• (recommended for production systems)

• (recommended for testing, development, or trial)

The Kubernetes operator that is installed runs in namespaced mode so any resources you create need to be in the same namespace as you installed into.

Kustomize

Seldon recommends installing Seldon Core 2 using Helm or Ansible. If you prefer to use Kustomize, you can base your configuration on the raw YAML files you generate in the k8s/yaml folder. Alternatively, you can follow the steps in the k8s/Makefile, which demonstrates how to build the YAML files from the Kustomize bases.

Operations

Seldon can be run with secure control plane and data plane operations. There are three areas of concern:

The various communication points between services are shown in the diagram below:

Control Plane

TLS control plane activation is switched on and off via the environment variable: CONTROL_PLANE_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

Certificates will be loaded and used for the control plane gRPC services. The secrets or folders will be watched for updates (on certificate renewal) and automatically loaded again.

Helm Control Plane Install

When installing seldon-core-v2-setup you can set the secret names for your certificates. If using cert-manager example discussed below this would be as follows:

Kafka

Kafka secure activation is switched on and off via the environment variable: KAFKA_SECURITY_PROTOCOL whose values can be PLAINTEXT, SSL or SASL_SSL.

Examples are shown below:

Data Plane

TLS Data plane activation is switched on and off via the environment variable: ENVOY_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

When activated this ensures TLS is used to communicate to Envoy via the xDS server as well as using the SDS service to send cretificates to envoy to use for upstream and downstream networking. Downstream is the external access to Seldon and upstream is the path from Envoy to the model servers or pipeline gateway.

Helm Data Plane Install

When installing seldon-core-v2-setup you can set data plane operations to TLS as below. This assumes the secrets installed by the helm chart at the end of this section.

The above uses default secret names defined for the certificates installed. You can change the names of the required certificate secrets as shown in a longer configuration below (again using the default names for illustration).

For this we use the following updated Helm values (k8s/samples/values-tls-dataplane-example.yaml):

We use Envoy internally to direct traffic and in Envoy's terminology upstream is for internal model servers called from Envoy while the downstream server is the entrypoint server running in Envoy to receive grpc and REST calls. The above settings ensure mTLS for internal "upstream" traffic while provides a standard SSL non-mTLS entrpoint.

To use the above with the seldon CLI you would need a custom config file as follow:

We skip SSL Verify as these are internal self-signed certificates. For production use you would change this to the correct DNS name you are exposing the Seldon entrypoint.

Cetificate Providers

Helm

You can install Certificates into the desired namespace, here we use seldon-mesh as an example.

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use SASL security mechanism.

Create API Keys

In your Confluent Cloud environment create new API keys. The easiest way to obtain all required information is to head to Clients -> New client (choose e.g. Go) and generate new Kafka cluster API key from there.

This will generate for you:

• Key (we use it as username)

• Secret (we use it as password)

Do not forget to also copy the bootstrap.servers from the example config.

See Confluent Cloud in case of issues.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

Troubleshooting

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Azure Event Hub via Kafka protocol.

Prerequisites

To start you will need to have an Azure Event Hub Namespace. You can create one following Azure quickstart . Note that you do not need to create an Event Hub (topics) as Seldon Core 2 will require all the topics it needs automatically.

Create API Keys

To connect to Azure Event Hub provided Kafka API you need to obtain:

• Kafka Endpoint

• Connection String

You can obtain both using Azure Portal as documented .

The Connection String should be in format of

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration. The username should read $ConnectionString and this is not a variable for you to replace.

Troubleshooting

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

• Verify that you did not hit quotas for topics or partitions in your Event Hub namespace.

Cluster Setup

If you have installed Strimzi we have an example Helm chart to create a Kafka cluster for seldon and an associated user in kafka/strimzi folder. Ensure the tls is enabled with:

The Ansible setup-ecosystem playbook will also install Strimzi and include a mTLS endpoint. See .

mTLS Example

Create a Kafka User seldon in the namespace seldon was installed. This assumes Strimzi Kafka cluster is installed in the same namespace or is running with cluster wide permissions. Our Ansible scripts to setup the ecosystem will also create this user if tls is active.

If you don't have this user you can install it with in your desired namespace (here seldon-mesh):

Install seldon with the Strimzi certificate secrets using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

You can now go ahead and install a SeldonRuntime in your desired install namespace (here seldon-mesh), e.g.

New in Seldon Core 2.7.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use .

Configure Identity Provider in Confluent Cloud Console

In your Confluent Cloud Console go to and register your Identity Provider.

See Confluent Cloud for further details.

Configure Identity Pool

In your Confluent Cloud Console go to and add new identity pool to your newly registered Identity Provider.

See Confluent Cloud for further details.

Create Kubernetes Secret

Seldon Core 2 expects oauth credentials to be in form of K8s secret

You will need following information from Confluent Cloud:

• Cluster ID: Cluster Overview → Cluster Settings → General → Identification

• Identity Pool ID: Accounts & access → Identity providers → <specific provider details>

Client ID, client secret and token endpoint url should come from identity provider, e.g. Keycloak or Azure AD.

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Create a Strimzi Kafka cluster with SASL_SSL enabled. This can be done with our Ansible scripts by running the following from the ansible/ folder:

ansible-playbook playbooks/setup-ecosystem.yaml -e @../k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml -e strimzi_kafka_operator_feature_gates=""

The referenced SASL/SCRAM YAML file looks like the below:

# k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml
seldon_kafka_cluster_values:
  broker:
    tls:
      authentication:
        type: scram-sha-512

This will use the Strimzi Helm chart provided in Seldon Core 2. This will call the Strimzi cluster Helm chart provided by the project with overrides for the cluster authentication type and will also create a user seldon with password credentials in a Kubernetes Secret.

Install Seldon Core 2 with SASL settings using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-strimzi-kafka-sasl-scram.yaml
kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh.svc.cluster.local:9093

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret: seldon
        passwordPath: password
    ssl:
      client:
        brokerValidationSecret: seldon-cluster-ca-cert
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-strimzi-kafka-sasl-scram.yaml

Helm Settings

# k8s/helm-charts/seldon-core-v2-setup/values.yaml
security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
        keyPath: /tmp/certs/cps/tls.key
        crtPath: /tmp/certs/cps/tls.crt
        caPath: /tmp/certs/cps/ca.crt
        clientCaPath: /tmp/certs/cpc/ca.crt
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
        keyPath: /tmp/certs/cpc/tls.key
        crtPath: /tmp/certs/cpc/tls.crt
        caPath: /tmp/certs/cpc/ca.crt
        serverCaPath: /tmp/certs/cps/ca.crt
  kafka:
    protocol: PLAINTEXT
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret:
        passwordPath: password
    ssl:
      client:
        secret:
        brokerValidationSecret:
        keyPath: /tmp/certs/kafka/client/tls.key
        crtPath: /tmp/certs/kafka/client/tls.crt
        caPath: /tmp/certs/kafka/client/ca.crt
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt
        endpointIdentificationAlgorithm:
  envoy:
    protocol: PLAINTEXT
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
          keyPath: /tmp/certs/dus/tls.key
          crtPath: /tmp/certs/dus/tls.crt
          caPath: /tmp/certs/dus/ca.crt
          clientCaPath: /tmp/certs/duc/ca.crt
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
          keyPath: /tmp/certs/duc/tls.key
          crtPath: /tmp/certs/duc/tls.crt
          caPath: /tmp/certs/duc/ca.crt
          serverCaPath: /tmp/certs/dus/ca.crt
      downstream:
        server:
          secret: seldon-downstream-server
          clientValidationSecret:
          keyPath: /tmp/certs/dds/tls.key
          crtPath: /tmp/certs/dds/tls.crt
          caPath: /tmp/certs/dds/ca.crt
          clientCaPath: /tmp/certs/ddc/ca.crt
        client:
          mtls: false
          secret:
          serverValidationSecret: seldon-downstream-server
          keyPath: /tmp/certs/ddc/tls.key
          crtPath: /tmp/certs/ddc/tls.crt
          caPath: /tmp/certs/ddc/ca.crt
          serverCaPath: /tmp/certs/dds/ca.crt

# A list of image pull secrets
imagePullSecrets:

Environment variables

Kubernetes secrets and mounted files can be used to provide the certificates in PEM format. These are controlled by environment variables for server or client depending on the component:

Control Plane

For a server (scheduler):

For a client (agent, modelgateway, hodometer, CRD controller):

Kafka

Envoy

Envoy xDS server will use the control plane server and client certificates defined above.

Downstream server

Downstream client

Upstream server

Upstream client