New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Azure Event Hub via Kafka protocol.

Prerequisites

To start you will need to have an Azure Event Hub Namespace. You can create one following Azure quickstart docs. Note that you do not need to create an Event Hub (topics) as Seldon Core 2 will require all the topics it needs automatically.

Create API Keys

To connect to Azure Event Hub provided Kafka API you need to obtain:

• Kafka Endpoint

• Connection String

You can obtain both using Azure Portal as documented here.

The Connection String should be in format of

Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

kubectl create secret generic azure-kafka-secret -n seldon-mesh --from-literal password="Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-azure-event-hub-sasl.yaml.tmpl
kafka:
  bootstrap: <namespace>.servicebus.windows.net
  topics:
    replicationFactor: 3
    numPartitions: 4

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: $ConnectionString
        secret: azure-kafka-secret
    ssl:
      client:
        secret:
        brokerValidationSecret:

You may need to tweak replicationFactor and numPartitions to your cluster configuration. The username should read $ConnectionString and this is not a variable for you to replace.

Troubleshooting

• First check Azure Event Hub troubleshooting guide.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

• Verify that you did not hit quotas for topics or partitions in your Event Hub namespace.

Preparation

1. git clone https://github.com/SeldonIO/seldon-core --branch=v2

2. Build Seldon CLI

3. Install Docker Compose (or directly from GitHub release if not using Docker Desktop).

4. Install make. This will depend on your version of Linux, for example on Ubuntu run sudo apt-get install build-essential.

Deploy

From the project root run:

make deploy-local

This will run with latest images for the components.

Note: Triton and MLServer are large images at present (11G and 9G respectively) so will take time to download on first usage.

Run a particular version

To run a particular release set the environment variable CUSTOM_IMAGE_TAG to the desired version before running the command, e.g.:

export CUSTOM_IMAGE_TAG=0.2.0
make deploy-local

GPU support

To enable GPU on servers:

1. Make sure that nvidia-container-runtime is installed, follow link

2. Enable GPU: export GPU_ENABLED=1

Local Models

To deploy with a local folder available for loading models set the environment variable LOCAL_MODEL_FOLDER to the folder, e.g.:

export LOCAL_MODEL_FOLDER=/home/seldon/models
make deploy-local

This folder will be mounted at /mnt/models. You can then specify models as shown below:

# samples/models/sklearn-iris-local.yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "/mnt/models/iris"
  requirements:
  - sklearn

If you have set the local model folder as above then this would be looking at /home/seldon/models/iris.

Tracing

The default local install will provide Jaeger tracing at http://0.0.0.0:16686/search.

Metrics

The default local install will expose Grafana at http://localhost:3000.

Undeploy

From the project root run:

make undeploy-local

Prerequisites

• Ensure that the version of the Kubernetes cluster meets the requirement listed below. You can create a KinD cluster on your local computer for testing with Ansible.

• Install the ecosystem components using Ansible.

Core 2 Dependencies

Here is a list of components that Seldon Core 2 depends on, with minimum and maximum supported versions.

* These components are shipped as part of Seldon Core 2 docker images set, users should not install them separately but they need to be aware of the configuration options that are supported by these versions. ** Kafka is only required to operate Seldon Core 2 dataflow Pipelines. If not required then users should not install seldon-modelgateway, seldon-pipelinegateway, and seldon-dataflow-engine. *** Not hard limit on the maximum version to be used.

Install Ecosystem Components

You also need to install our ecosystem components. For this we provide directions for Ansible to install these.

Install

To install Seldon Core 2 from the source repository, you can choose one of the following methods:

• Helm (recommended for production systems)

• Ansible (recommended for testing, development, or trial)

The Kubernetes operator that is installed runs in namespaced mode so any resources you create need to be in the same namespace as you installed into.

Kustomize

Seldon recommends installing Seldon Core 2 using Helm or Ansible. If you prefer to use Kustomize, you can base your configuration on the raw YAML files you generate in the k8s/yaml folder. Alternatively, you can follow the steps in the k8s/Makefile, which demonstrates how to build the YAML files from the Kustomize bases.

Operations

• Security

Seldon Core can be installed either with Docker Compose or with Kubernetes:

• Install locally with Docker Compose

• Install onto a Kubernetes cluster

Once installed:

• Try the existing examples.

• Train and deploy your own model artifact.

Core Concepts

There are three core resources you will use:

• Models - for deploying single machine learning models, custom transformation logic, drift detectors and outliers detectors.

• Pipelines - for connecting together flows of data transformations between Models with a synchronous path and multiple asynchronous paths.

• Experiments - for testing new versions of models

By default the standard installation will deploy MLServer and Triton inference servers which provide support for a wide range of machine learning model artifacts including Tensorflow models, PyTorch models, SKlearn models, XGBoost models, ONNX models, TensorRT models, custom python models and many more. For advanced use, the creation of new inference servers is manged by two resources:

• Servers - for deploying sets of replicas of core inference servers (MLServer or Triton by default).

• ServerConfigs - for defining server configurations including custom servers.

API for Inference

Once deployed models can be called using the Seldon V2 inference protocol. This protocol created by Seldon, NVIDIA and the KServe projects is supported by MLServer and Triton inference servers amingst others and allows REST and gRPC calls to your model.

Your model is exposed via our internal Envoy gateway. If you wish to expose your models in Kubernetes outside the cluster you are free to use any Service Mesh or Ingress technology. Various examples are provided for service mesh integration.

Inference Metrics

Metrics are exposed for scraping by Prometheus. For Kubernetes we provide example instructions for using kube-prometheus.

Pipeline requirements

Pipelines are built upon Kafka streaming technology.

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Amazon managed Apache Kafka (MSK). You can control access to your Amazon MSK clusters using sign-in credentials that are stored and secured using AWS Secrets Manager. Storing user credentials in Secrets Manager reduces the overhead of cluster authentication such as auditing, updating, and rotating credentials. Secrets Manager also lets you share user credentials across clusters.

Setting up SASL/SCRAM authentication for an Amazon MSK cluster

To setup SASL/SCRAM in an Amazon MSK cluster, please follow the guide from Amazon's Official documentation.

Do not forget to also copy the bootstrap.servers which we will use it in our configuration later below for Seldon.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

kubectl create secret generic aws-msk-kafka-secret -n seldon-mesh --from-literal password="<MSK SASL Password>"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-aws-msk-kafka-sasl-scram.yaml.tmpl
kafka:
  bootstrap: <msk-bootstrap-server-endpoints>
  topics:
    replicationFactor: 3
    numPartitions: 4

security:
    kafka:
      protocol: SASL_SSL
      sasl:
        mechanism: SCRAM-SHA-512
        client:
          username: < username >
          secret: aws-msk-kafka-secret
          passwordPath: password
      ssl:
        client:
          secret:
          brokerValidationSecret:

You may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• Please check Amazon MSK Troubleshooting documentation.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Installing Ansible

Provided Ansible playbooks and roles depends on kubernetes.core Ansible Collection for performing kubectl and helm operations. Check Ansible [documentation] for further information.

To install Ansible and required collections

pip install ansible openshift kubernetes docker
ansible-galaxy collection install kubernetes.core

We have tested provided instructions on Python 3.8 - 3.11 with following version of Python libraries

and kubernetes.core collection in version 2.4.0.

Once installed you can use the following Playbooks that you will find in Ansible folder of Seldon Core 2 repository.

You also need to have installed kubectl CLI.

Installing Seldon Core 2 using Ansible

One-liner local kind install (from scratch)

If you simply want to install into a fresh local kind k8s cluster, the seldon-all playbook allows you to do so with a single command:

ansible-playbook playbooks/seldon-all.yaml

This will create a Kind cluster and install ecosystem dependencies (kafka, prometheus, opentelemetry, jager) as well as all the seldon-specific components. The seldon components are installed using helm-charts from the current git checkout (../k8s/helm-charts/).

Internally this runs, in order, the following playbooks (described in more detail in the sections below):

• kind-cluster.yaml

• setup-ecosystem.yaml

• setup-seldon.yaml

You may pass any of the additonal variables which are configurable for those playbooks to seldon-all. See the Customizing Ansible Instalation section for details.

For example:

ansible-playbook playbooks/seldon-all.yaml -e seldon_mesh_namespace=my-seldon-mesh -e install_prometheus=no -e @playbooks/vars/set-custom-images.yaml

Running the playbooks individually, as described in the sections below, will give you more control over what gets run and when (for example, if you want to install into an existing k8s cluster).

Create Kind Cluster

It is recommended to first install Seldon Core 2 inside Kind cluster. This allow to test and trial the installation in isolated environment that is easy to remove.

ansible-playbook playbooks/kind-cluster.yaml

Setup Ecosystem

Seldon runs by default in seldon-mesh namespace and a Jaeger pod and and OpenTelemetry collector are installed in the chosen namespace. Run the following:

ansible-playbook playbooks/setup-ecosystem.yaml

The most common change will be to install in another namespace with:

ansible-playbook playbooks/setup-ecosystem.yaml -e seldon_mesh_namespace=<mynamespace>

Install Seldon Core 2

Run the following from the ansible/ folder:

ansible-playbook playbooks/setup-seldon.yaml

If you have changed the namespace you wish to use you will need to run with:

ansible-playbook playbooks/setup-seldon.yaml -e seldon_mesh_namespace=<mynamespace>

Customizing Ansible Installation

Ecosystem configuration options

The ecosystem setup can be parametrized by providing extra Ansible variables, e.g. using -e flag to ansible-playbook command.

For example run the following from the ansible/ folder:

ansible-playbook playbooks/setup-ecosystem.yaml -e full_install=no -e install_kafka=yes

will only install Kafka when setting up the ecosystem.

Seldon Core 2 configuration options

Custom Seldon images and private registries

By default, the container images used in the install are the ones defined by the helm charts (referring to images publicly available on dockerhub).

If you need to customize the images (i.e pull from private registry, pull given tag), create a custom images config file following the example in playbooks/vars/set-custom-images.yaml and run with:

ansible-playbook playbooks/setup-seldon.yaml -e @<path-to-custom-images-config.yaml>

If, instead of pulling images from an external repository you want to build certain components locally, please read README.dev.md

Private registries

When using private registries, access needs to be authenticated (typically, via a service account key), and the k8s cluster will need to have access to a secret holding this key to be able to pull images.

The setup-seldon.yaml playbook will create the required k8s secrets inside the cluster if it is provided with an auth file in dockerconfigjson format. You provide the path to this file by cusomizing the custom_image_pull_secrets.dockerconfigjson variable and define the secret name via custom_image_pull_secrets.name in the custom images config file (the one passed to the playbook via -e @file).

By default, docker creates the dockerconfigjson auth file in ~/.docker/config.json after passing the service-account key to docker login.

The docker login command would look like this (key in json format):

cat registry-sa-key.json | docker login -u _json_key --password-stdin <registry-url>

or, for keys in base64 format:

cat registry-sa-key | docker login -u _json_key_base64 --password-stdin <registry-url>

Saving helm-chart customisations

Because the additional custom images config file (starting from the playbooks/vars/set-custom-images.yaml example) overrides values in the helm-charts available in the repo, there's also a playbook option of saving those overrides as a separate values file, which could be used if deploying manually via helm.

This is controlled via two variables:

You can either pass those within the custom images config file or directly when running the playbook. For example, for just saving the helm-chart overrides (without installing seldon components), you would run:

ansible-playbook playbooks/setup-seldon.yaml -e full_install=no -e save_helm_components_overrides=yes -e @<path-to-custom-images-config.yaml>

Please note that when deploying outside ansible via helm using this saved overrides file, and using private registries, you will have to manually create the service-account key secret with the same name as the one defined in your custom image config file under custom_image_pull_secrets.name.

Uninstall

To fully remove the Ansible installation delete the created Kind cluster

kind delete cluster --name seldon

This will stop and delete the Kind cluster freeing all of the resources taken by the dev/trial installation. You may want to also remove cache resources used for the installation with

rm -rf ~/.cache/seldon/

Seldon will run with AWS MSK.

At present we support mTLS authentication to MSK which can be run from a Kubernetes cluster inside or outside Amazon. If running outside your MSK cluster must have a public endpoint.

Considerations

Public Access to MSK Cluster

If you running your Kubernetes cluster outside AWS you will need to create a public accessible MSK cluster.

You will need to setup Kafka ACLs for your user where the username is the CommonName of the certificate of the client and allow full topic access. For example, to add a user with CN=myname to have full operations using the kafka-acls script with mTLS config setup as described in AWS MSK docs:

kafka-acls.sh --bootstrap-server <mTLS endpoint>  --add --allow-principal User:CN=myname --operation All --topic '*' --command-config client.properties

You will also need to allow the connecting user to be able to perform admin tasks on the cluster so we can create topics on demand.

kafka-acls.sh --bootstrap-server <nTLS endpoint>  --add --allow-principal User:CN=myname --operation All --cluster '*' --command-config client.properties

You will need to allow group access also.

kafka-acls.sh --bootstrap-server <mTKS endpoint>  --add --allow-principal User:CN=myname --operation All --group '*' --command-config client.properties

Create TLS Kubernetes Secrets

Create a secret for the client certificate you created. If you followed the AWS MSK mTLS guide you will need to export your private key from the JKS keystore. The certificate and chain will be provided in PEM format when you get the certificate signed. You can use these to create a secret with:

• tls.key : PEM formatted private key

• tls.crt : PEM formatted certificate

• ca.crt : Certificate chain

kubectl create secret generic aws-msk-client --from-file=./tls.key --from-file=./tls.crt --from-file=./ca.crt -n seldon-mesh

Create a secret for the broker certificate. If following the AWS MSK mTLS guide you will need to export the trusstore of Amazon into PEM format and save as ca.crt.

To extract certificates from truststore do:

keytool -importkeystore -srckeystore truststore.jks    -destkeystore truststore.p12    -srcstoretype jks    -deststoretype pkcs12
openssl pkcs12 -in truststore.p12  -nodes -out trust.pem
cat trust.pem | sed  -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt

Add ca.crt to a secret.

kubectl create secret generic aws-msk-broker-ca --from-file=./ca.crt -n seldon-mesh

Example Helm install

We provide a template you can extend in k8s/samples/values-aws-msk-kafka-mtls.yaml.tmpl:

# k8s/samples/values-aws-msk-kafka-mtls.yaml.tmpl
kafka:
  bootstrap: <MSK Broker Endpoints>

security:
  kafka:
    protocol: SSL
    ssl:
      client:
        secret: aws-msk-client
        brokerValidationSecret: aws-msk-broker-ca

Copy this and modify by adding your broker endpoints.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-aws-msk-kafka-mtls.yaml --set kafka.bootstrap=<your aws msk broker endpoints>

Troubleshooting

First check AWS MSK troubleshooting.

No messages are being produced to the topics

Set the kafka config map debug setting to "all". For Helm install you can set kafka.debug=all.

If you see an error from the producer in the Pipeline gateway complaining about not enough insync replicas then the replication factor Seldon is using is less than the cluster setting for min.insync.replicas which for a default AWS MSK cluster defaults to 2. Ensure this is equal to that of the cluster. This value can be set in the Helm chart with kafka.topics.replicationFactor.

New in Seldon Core 2.7.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use Oauth 2.0 security mechanism.

Configure Identity Provider in Confluent Cloud Console

In your Confluent Cloud Console go to Account & Access / Identity providers and register your Identity Provider.

See Confluent Cloud documentation for further details.

Configure Identity Pool

In your Confluent Cloud Console go to Account & Access / Identity providers and add new identity pool to your newly registered Identity Provider.

See Confluent Cloud documentation for further details.

Create Kubernetes Secret

Seldon Core 2 expects oauth credentials to be in form of K8s secret

apiVersion: v1
kind: Secret
metadata:
  name: confluent-kafka-oauth
  namespace: seldon-mesh
type: Opaque
stringData:
  method: OIDC
  client_id: <client id>
  client_secret: <client secret>
  token_endpoint_url: <token endpoint url>
  extensions: logicalCluster=<cluster id>,identityPoolId=<identity pool id>
  scope: ""

You need the following information from Confluent Cloud:

• Cluster ID: Cluster Overview → Cluster Settings → General → Identification

• Identity Pool ID: Accounts & access → Identity providers → <specific provider details>

Client ID, client secret and token endpoint url should come from identity provider, e.g. Keycloak or Azure AD.

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-confluent-kafka-oauth.yaml.tmpl
kafka:
  bootstrap: < Confluent Cloud Broker Endpoints >
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: OAUTHBEARER
      client:
          secret: confluent-kafka-oauth
    ssl:
      client:
        secret:
        brokerValidationSecret:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• First check Confluent Cloud documentation.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

We provide several Helm charts.

• seldon-core-v2-crds : cluster wide install of custom resources.

• seldon-core-v2-setup : installation of the manager to manage resources in the namespace or clusterwide. This also installs default SeldonConfig and ServerConfig resources which allow Runtimes and Servers to be installed easily on demand.

• seldon-core-v2-runtime : this installs a SeldonRuntime custom resource which creates the core components in a namespace.

• seldon-core-v2-servers : this installs Server custom resources which provide example core servers to load models.

• seldon-core-v2-certs : a default set of certificates for TLS.

The Helm charts can be found within the k8s/helm-charts folder and they are published here

Assuming you have installed any ecosystem components: Jaeger, Prometheus, Kafka as discussed here you can follow the following steps.

Note that for Kafka follow the steps discussed here

Add Seldon Core 2 Charts

helm repo add seldon-charts https://seldonio.github.io/helm-charts
helm repo update seldon-charts

Install the CRDs

helm install seldon-core-v2-crds  seldon-charts/seldon-core-v2-crds

Install the Seldon Core 2 Components

You can install into any namespace. For illustration we will use seldon-mesh. This will install the core manager which will handle the key resources used by Seldon including the SeldonRuntime and Server resources.

kubectl create namespace seldon-mesh

helm install seldon-core-v2  seldon-charts/seldon-core-v2-setup --namespace seldon-mesh

This will install the operator namespaced so it will only control resources in the provided namespace. To allow cluster wide usage add the --set controller.clusterwide=true, e.g.

helm install seldon-core-v2  seldon-charts/seldon-core-v2-setup --namespace seldon-mesh --set controller.clusterwide=true

Cluster wide operations will require ClusterRoles to be created so when deploying be aware your user will require the required permissions. With cluster wide operations you can create SeldonRuntimes in any namespace.

Install the default Seldon Core 2 Runtime

helm install seldon-v2-runtime seldon-charts/seldon-core-v2-runtime --namespace seldon-mesh

This will install the core components in your desired namespace.

Install example servers

To install some MLServer and Triton servers you can either create Server resources yourself or for initial testing you can use our example Helm chart seldon-core-v2-servers:

helm install seldon-v2-servers seldon-charts/seldon-core-v2-servers --namespace seldon-mesh

By default this will install 1 MLServer and 1 Triton in the desired namespace. This namespace should be the same namespace you installed a Seldon Core Runtime.

Uninstall

Remove any models, pipelines that are running.

Remove the runtime:

helm uninstall seldon-core-v2-runtime  --namespace seldon-mesh

Remove the core components:

helm uninstall seldon-core-v2  --namespace seldon-mesh

Remove the CRDs

helm uninstall seldon-core-v2-crds

Cluster Setup

If you have installed Strimzi we have an example Helm chart to create a Kafka cluster for seldon and an associated user in kafka/strimzi folder. Ensure the tls is enabled with:

broker:
  tls:
    enabled: true
    port: 9093
    listenerType: internal
    authentication:
      type: tls

The Ansible setup-ecosystem playbook will also install Strimzi and include a mTLS endpoint. See here.

mTLS Example

Create a Kafka User seldon in the namespace seldon was installed. This assumes Strimzi Kafka cluster is installed in the same namespace or is running with cluster wide permissions. Our Ansible scripts to setup the ecosystem will also create this user if tls is active.

# k8s/samples/strimzi-example-tls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: seldon
  labels:
    strimzi.io/cluster: seldon
spec:
  authentication:
    type: tls

If you don't have this user you can install it with in your desired namespace (here seldon-mesh):

kubectl create -f k8s/samples/strimzi-example-tls-user.yaml -n seldon-mesh

Install seldon with the Strimzi certificate secrets using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-strimzi-kafka-mtls.yaml
kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh.svc.cluster.local:9093

security:
  kafka:
    protocol: SSL
    ssl:
      client:
        secret: seldon
        brokerValidationSecret: seldon-cluster-ca-cert
        keyPath: /tmp/certs/kafka/client/user.key
        crtPath: /tmp/certs/kafka/client/user.crt
        caPath: /tmp/certs/kafka/client/ca.crt
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-strimzi-kafka-mtls.yaml

You can now go ahead and install a SeldonRuntime in your desired install namespace (here seldon-mesh), e.g.

helm install seldon-v2-runtime ../k8s/helm-charts/seldon-core-v2-runtime  -n seldon-mesh

Create a Strimzi Kafka cluster with SASL_SSL enabled. This can be done with our Ansible scripts by running the following from the ansible/ folder:

ansible-playbook playbooks/setup-ecosystem.yaml -e @../k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml -e strimzi_kafka_operator_feature_gates=""

The referenced SASL/SCRAM YAML file looks like the below:

# k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml
seldon_kafka_cluster_values:
  broker:
    tls:
      authentication:
        type: scram-sha-512

This will use the Strimzi Helm chart provided in Seldon Core 2. This will call the Strimzi cluster Helm chart provided by the project with overrides for the cluster authentication type and will also create a user seldon with password credentials in a Kubernetes Secret.

Install Seldon Core 2 with SASL settings using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-strimzi-kafka-sasl-scram.yaml
kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh.svc.cluster.local:9093

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret: seldon
        passwordPath: password
    ssl:
      client:
        brokerValidationSecret: seldon-cluster-ca-cert
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-strimzi-kafka-sasl-scram.yaml

Helm Settings

# k8s/helm-charts/seldon-core-v2-setup/values.yaml
security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
        keyPath: /tmp/certs/cps/tls.key
        crtPath: /tmp/certs/cps/tls.crt
        caPath: /tmp/certs/cps/ca.crt
        clientCaPath: /tmp/certs/cpc/ca.crt
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
        keyPath: /tmp/certs/cpc/tls.key
        crtPath: /tmp/certs/cpc/tls.crt
        caPath: /tmp/certs/cpc/ca.crt
        serverCaPath: /tmp/certs/cps/ca.crt
  kafka:
    protocol: PLAINTEXT
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret:
        passwordPath: password
    ssl:
      client:
        secret:
        brokerValidationSecret:
        keyPath: /tmp/certs/kafka/client/tls.key
        crtPath: /tmp/certs/kafka/client/tls.crt
        caPath: /tmp/certs/kafka/client/ca.crt
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt
        endpointIdentificationAlgorithm:
  envoy:
    protocol: PLAINTEXT
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
          keyPath: /tmp/certs/dus/tls.key
          crtPath: /tmp/certs/dus/tls.crt
          caPath: /tmp/certs/dus/ca.crt
          clientCaPath: /tmp/certs/duc/ca.crt
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
          keyPath: /tmp/certs/duc/tls.key
          crtPath: /tmp/certs/duc/tls.crt
          caPath: /tmp/certs/duc/ca.crt
          serverCaPath: /tmp/certs/dus/ca.crt
      downstream:
        server:
          secret: seldon-downstream-server
          clientValidationSecret:
          keyPath: /tmp/certs/dds/tls.key
          crtPath: /tmp/certs/dds/tls.crt
          caPath: /tmp/certs/dds/ca.crt
          clientCaPath: /tmp/certs/ddc/ca.crt
        client:
          mtls: false
          secret:
          serverValidationSecret: seldon-downstream-server
          keyPath: /tmp/certs/ddc/tls.key
          crtPath: /tmp/certs/ddc/tls.crt
          caPath: /tmp/certs/ddc/ca.crt
          serverCaPath: /tmp/certs/dds/ca.crt

# A list of image pull secrets
imagePullSecrets:

Environment variables

Kubernetes secrets and mounted files can be used to provide the certificates in PEM format. These are controlled by environment variables for server or client depending on the component:

Control Plane

For a server (scheduler):

For a client (agent, modelgateway, hodometer, CRD controller):

Kafka

Envoy

Envoy xDS server will use the control plane server and client certificates defined above.

Downstream server

Downstream client

Upstream server

Upstream client

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use SASL security mechanism.

Create API Keys

In your Confluent Cloud environment create new API keys. The easiest way to obtain all required information is to head to Clients -> New client (choose e.g. Go) and generate new Kafka cluster API key from there.

This will generate for you:

• Key (we use it as username)

• Secret (we use it as password)

Do not forget to also copy the bootstrap.servers from the example config.

See Confluent Cloud documentation in case of issues.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret

kubectl create secret generic confluent-kafka-sasl -n seldon-mesh --from-literal password="<Confluent Cloud API Secret>"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-confluent-kafka-sasl.yaml.tmpl
kafka:
  bootstrap: < Confluent Cloud Broker Endpoints >
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: < username >
        secret: confluent-kafka-sasl
    ssl:
      client:
        secret:
        brokerValidationSecret:

Troubleshooting

• First check Confluent Cloud documentation.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Seldon recommends managed Kafka for production installation. On this page we will demonstrate how you can integrate and configure your managed Kafka with Seldon Core 2.

Securing managed Kafka services

You can secure your Seldon Core 2 integration with managed Kafka services by setting up encryption and authenticaion.

Kafka Encryption (TLS)

In production settings, always set up TLS encryption with Kafka. This ensures that neither the credentials nor the payloads are transported in plaintext.

When TLS is enabled, the client needs to know the root CA certificate used to create the server’s certificate. This is used to validate the certificate sent back by the Kafka server.

1. Create a certificate named ca.crt that is encoded as a PEM certificate. It is important that the certificate is saved as ca.crt. Otherwise, Seldon Core 2 may not be able to find the certificate. Within the cluster, you can provide the server’s root CA certificate through a secret. For example, a secret named kafka-broker-tls with a certificate.

kubectl create secret generic kafka-broker-tls -n seldon --from-file ./ca.crt

Kafka Authentication

In production environments, Kafka clusters often require authentication, especially when using managed Kafka solutions. Therefore, when installing Seldon Core 2 components, it is crucial to provide the correct credentials for a secure connection to Kafka.

The type of authentication used with Kafka varies depending on the setup but typically includes one of the following:

• Simple Authentication and Security Layer (SASL): Requires a username and password.

• Mutual TLS (mTLS): Involves using SSL certificates as credentials.

• OAuth 2.0: Uses the client credential flow to acquire a JWT token.

These credentials are stored as Kubernetes secrets within the cluster. When setting up Seldon Core 2 you must create the appropriate secret in the correct format and update the components-values.yaml, and install-values files respectively.

```
kubectl create secret generic kafka-sasl-secret --from-literal password=<kafka-password> -n seldon
```

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: <kafka-username>      # TODO: Replace with your Kafka username
        secret: kafka-sasl-secret       # NOTE: Secret name from previous step
    ssl:
      client:
        secret:                                   # NOTE: Leave empty
        brokerValidationSecret: kafka-broker-tls  # NOTE: Optional

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: OAUTHBEARER
      client:
          secret: kafka-oauth                     # NOTE: Secret name from earlier step
    ssl:
      client:
        secret:                                   # NOTE: Leave empty
        brokerValidationSecret: kafka-broker-tls  # NOTE: Optional

kubectl create secret generic kafka-client-tls -n seldon \
  --from-file ./tls.crt \
  --from-file ./tls.key \
  --from-file ./ca.crt

security:
  kafka:
    protocol: SSL
    ssl:
      client:
        secret: kafka-client-tls                  # NOTE: Secret name from earlier step
        brokerValidationSecret: kafka-broker-tls  # NOTE: Optional

Example configurations for managed Kafka services

Here are some examples to create secrets for managed Kafka services such as Azure Event Hub, Confluent Cloud(SASL), Confluent Cloud(OAuth2.0).

kubectl create secret generic azure-kafka-secret --from-literal=<password>="Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX" -n seldon

kubectl create secret generic confluent-kafka-sasl --from-literal password="<password>" -n seldon

Configuring Seldon Core 2

To integrate Kafka with Seldon Core 2.

1. Update the initial configuration.

controller:
  clusterwide: true

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: <namespace>.servicebus.windows.net:9093
  topics:
    replicationFactor: 3
    numPartitions: 4    
security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: $ConnectionString
        secret: azure-kafka-secret
    ssl:
      client:
        secret:
        brokerValidationSecret:
        
opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

controller:
  clusterwide: true

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: <confluent-endpoints>
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: <username>
        secret: confluent-kafka-sasl
    ssl:
      client:
        secret:
        brokerValidationSecret:

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

controller:
  clusterwide: true

dataflow:
  resources:
    cpu: 500m

envoy:
  service:
    type: ClusterIP

kafka:
  bootstrap: <confluent-endpoints>
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: OAUTHBEARER
      client:
        secret: confluent-kafka-oauth
    ssl:
      client:
        secret:
        brokerValidationSecret:

opentelemetry:
  enable: false

scheduler:
  service:
    type: ClusterIP

serverConfig:
  mlserver:
    resources:
      cpu: 1
      memory: 2Gi

  triton:
    resources:
      cpu: 1
      memory: 2Gi

serviceGRPCPrefix: "http2-"

1. To enable Kafka Encryption (TLS) you need to reference the secret that you created in the security.kafka.ssl.client.secret field of the Helm chart values. The resulting set of values to include in components-values.yaml is similar to:

security:
  kafka:
    ssl:
      secret:
        brokerValidationSecret: kafka-broker-tls

1. Change to the directory that contains the components-values.yaml file and then install Seldon Core 2 operator in the namespace seldon-system.

 helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
 --version 2.8.5 \
 -f components-values.yaml \
 --namespace seldon-system \
 --install

Installing Seldon CLI

To install Seldon CLI using prebuild binaries or build them locally.

Usage

seldon --help

Seldon can be run with secure control plane and data plane operations. There are three areas of concern:

• Control Plane

• Kafka

• Data Plane

The various communication points between services are shown in the diagram below:

Control Plane

TLS control plane activation is switched on and off via the environment variable: CONTROL_PLANE_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

Certificates will be loaded and used for the control plane gRPC services. The secrets or folders will be watched for updates (on certificate renewal) and automatically loaded again.

Helm Control Plane Install

When installing seldon-core-v2-setup you can set the secret names for your certificates. If using cert-manager example discussed below this would be as follows:

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
      --set security.controlplane.protocol=SSL

Kafka

Kafka secure activation is switched on and off via the environment variable: KAFKA_SECURITY_PROTOCOL whose values can be PLAINTEXT, SSL or SASL_SSL.

Examples are shown below:

• mTLS Strimzi example

• mTLS AWS MSK example

• SASL PLAIN with Confluent Cloud example

• SASL PLAIN with Azure Event Hub example

• SASL SCRAM with Strimzi example

• SASL SCRAM with AWS MSK example

• SASL OAUTH with Confluent Cloud example

Data Plane

TLS Data plane activation is switched on and off via the environment variable: ENVOY_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

When activated this ensures TLS is used to communicate to Envoy via the xDS server as well as using the SDS service to send cretificates to envoy to use for upstream and downstream networking. Downstream is the external access to Seldon and upstream is the path from Envoy to the model servers or pipeline gateway.

Helm Data Plane Install

When installing seldon-core-v2-setup you can set data plane operations to TLS as below. This assumes the secrets installed by the helm chart at the end of this section.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh --set security.envoy.protocol=SSL

The above uses default secret names defined for the certificates installed. You can change the names of the required certificate secrets as shown in a longer configuration below (again using the default names for illustration).

For this we use the following updated Helm values (k8s/samples/values-tls-dataplane-example.yaml):

security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
  envoy:
    protocol: SSL
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
      downstream:
        client:
          serverValidationSecret: seldon-downstream-server
        server:
          secret: seldon-downstream-server

We use Envoy internally to direct traffic and in Envoy's terminology upstream is for internal model servers called from Envoy while the downstream server is the entrypoint server running in Envoy to receive grpc and REST calls. The above settings ensure mTLS for internal "upstream" traffic while provides a standard SSL non-mTLS entrpoint.

To use the above with the seldon CLI you would need a custom config file as follow:

{
    "dataplane": {
	"tls": true,
	"skipSSLVerify": true
    }
}

We skip SSL Verify as these are internal self-signed certificates. For production use you would change this to the correct DNS name you are exposing the Seldon entrypoint.

Cetificate Providers

The installer/cluster controller for Seldon needs to provide the certificates. As part of Seldon we provide an example set of certificate issuers and certificates using cert-manager.

Helm

You can install Certificates into the desired namespace, here we use seldon-mesh as an example.

helm install seldon-v2-certs k8s/helm-charts/seldon-core-v2-certs/ -n seldon-mesh

Seldon can be configured via various config files.

Kafka Configuration

We allow configuration of the Kafka integration. In general this configuration looks like:

https://github.com/SeldonIO/seldon-core/blob/v2/scheduler/config/kafka-internal.json

The top level keys are:

• topicPrefix : the prefix to add to kafka topics created by Seldon

• consumerGroupIdPrefix : the prefix to add to Kafka consumer group IDs created by Seldon

• bootstrap.servers : the global bootstrap kafka servers to use

• consumer : consumer settings

• producer : producer settings

• streams : KStreams settings

For topicPrefix you can use any acceptable kafka topic characters which are a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). We use . (dot) internally as topic naming separator so we would suggest you don't end your topic prefix with a dot for clarity. For illustration, an example topic could be seldon.default.model.mymodel.inputs where seldon is the topic prefix.

The consumerGroupIdPrefix will ensure that all consumer groups created have a given prefix.

Kubernetes

For Kubernetes this is controlled via a ConfigMap called seldon-kafka whose default values are defined in the SeldonConfig custom resource. For more details see the componets.yaml file

https://github.com/SeldonIO/seldon-core/blob/b52a715865e1b28bad72cabc74e73ae51d11191c/k8s/yaml/components.yaml#L1115-L1127

When the SeldonRuntime is installed in a namespace a configMap will be created with these settings for Kafka configuration.

To customize the settings you can add and modify the Kafka configuration via Helm, for example below is a custom Helm values file that add compression for producers:

To use this with the SeldonRuntime Helm chart:

helm install seldon-v2-runtime k8s/helm-charts/seldon-core-v2-runtime \
    --namespace seldon-mesh \
    --values k8s/samples/values-runtime-kafka-compression.yaml

Topic and consumer isolation

If you use a shared Kafka cluster with other applications you may want to isolate the topic names and consumer group IDs from other users of the cluster to ensure there is no name clash. For this we provide two settings:

• topicPrefix: set a prefix for all topics

• consumerGroupIdPrefix: set a prefix for all consumer groups

An example to set this in the configuration when using the helm installation is showm below for creating the default SeldonConfig:

helm upgrade --install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
    --set controller.clusterwide=true \
    --set kafka.topicPrefix=myorg \
    --set kafka.consumerGroupIdPrefix=myorg

You can find a worked example here.

You can create alternate SeldonConfigs with different values or override values for particular SeldonRuntime installs.

Tracing Configuration

We allow configuration of tracing. This file looks like:

The top level keys are:

• enable : whether to enable tracing

• otelExporterEndpoint : The host and port for the OTEL exporter

• otelExporterProtocol : The protocol for the OTEL exporter. Currently used for jvm-based components only (such as dataflow-engine), because opentelemetry-java-instrumentation requires a http(s) URI for the endpoint but defaults to http/protobuf as a protocol. Because of this, gRPC connections (over http) can only be set up by setting this option to grpc

• ratio : The ratio of requests to trace. Takes values between 0 and 1 inclusive.

Kubernetes

For Kubernetes this is controlled via a ConfigMap call seldon-tracing whose default value is shown below:

Note, this ConfigMap is created via our Helm charts and there is usually no need to modify it manually.

At present Java instrumentation (for the dataflow engine) is duplicated via separate keys.