Seldon can be run with secure control plane and data plane operations. There are three areas of concern:

• Control Plane

• Kafka

• Data Plane

The various communication points between services are shown in the diagram below:

Control Plane

TLS control plane activation is switched on and off via the environment variable: CONTROL_PLANE_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

Certificates will be loaded and used for the control plane gRPC services. The secrets or folders will be watched for updates (on certificate renewal) and automatically loaded again.

Helm Control Plane Install

When installing seldon-core-v2-setup you can set the secret names for your certificates. If using cert-manager example discussed below this would be as follows:

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
      --set security.controlplane.protocol=SSL

Kafka

Kafka secure activation is switched on and off via the environment variable: KAFKA_SECURITY_PROTOCOL whose values can be PLAINTEXT, SSL or SASL_SSL.

Examples are shown below:

• mTLS Strimzi example

• mTLS AWS MSK example

• SASL PLAIN with Confluent Cloud example

• SASL PLAIN with Azure Event Hub example

• SASL SCRAM with Strimzi example

• SASL SCRAM with AWS MSK example

• SASL OAUTH with Confluent Cloud example

Data Plane

TLS Data plane activation is switched on and off via the environment variable: ENVOY_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

When activated this ensures TLS is used to communicate to Envoy via the xDS server as well as using the SDS service to send cretificates to envoy to use for upstream and downstream networking. Downstream is the external access to Seldon and upstream is the path from Envoy to the model servers or pipeline gateway.

Helm Data Plane Install

When installing seldon-core-v2-setup you can set data plane operations to TLS as below. This assumes the secrets installed by the helm chart at the end of this section.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh --set security.envoy.protocol=SSL

The above uses default secret names defined for the certificates installed. You can change the names of the required certificate secrets as shown in a longer configuration below (again using the default names for illustration).

For this we use the following updated Helm values (k8s/samples/values-tls-dataplane-example.yaml):

security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
  envoy:
    protocol: SSL
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
      downstream:
        client:
          serverValidationSecret: seldon-downstream-server
        server:
          secret: seldon-downstream-server

We use Envoy internally to direct traffic and in Envoy's terminology upstream is for internal model servers called from Envoy while the downstream server is the entrypoint server running in Envoy to receive grpc and REST calls. The above settings ensure mTLS for internal "upstream" traffic while provides a standard SSL non-mTLS entrpoint.

To use the above with the seldon CLI you would need a custom config file as follow:

{
    "dataplane": {
	"tls": true,
	"skipSSLVerify": true
    }
}

We skip SSL Verify as these are internal self-signed certificates. For production use you would change this to the correct DNS name you are exposing the Seldon entrypoint.

Cetificate Providers

The installer/cluster controller for Seldon needs to provide the certificates. As part of Seldon we provide an example set of certificate issuers and certificates using cert-manager.

Helm

You can install Certificates into the desired namespace, here we use seldon-mesh as an example.

helm install seldon-v2-certs k8s/helm-charts/seldon-core-v2-certs/ -n seldon-mesh

New in Seldon Core 2.7.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use .

Configure Identity Provider in Confluent Cloud Console

In your Confluent Cloud Console go to and register your Identity Provider.

See Confluent Cloud for further details.

Configure Identity Pool

In your Confluent Cloud Console go to and add new identity pool to your newly registered Identity Provider.

See Confluent Cloud for further details.

Create Kubernetes Secret

Seldon Core 2 expects oauth credentials to be in form of K8s secret

You will need following information from Confluent Cloud:

• Cluster ID: Cluster Overview → Cluster Settings → General → Identification

• Identity Pool ID: Accounts & access → Identity providers → <specific provider details>

Client ID, client secret and token endpoint url should come from identity provider, e.g. Keycloak or Azure AD.

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-confluent-kafka-oauth.yaml.tmpl
kafka:
  bootstrap: < Confluent Cloud Broker Endpoints >
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: OAUTHBEARER
      client:
          secret: confluent-kafka-oauth
    ssl:
      client:
        secret:
        brokerValidationSecret:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Seldon will run with .

At present we support mTLS authentication to MSK which can be run from a Kubernetes cluster inside or outside Amazon. If running outside your MSK cluster must have a public endpoint.

Considerations

Public Access to MSK Cluster

If you running your Kubernetes cluster outside AWS you will need to create a .

You will need to setup Kafka ACLs for your user where the username is the CommonName of the certificate of the client and allow full topic access. For example, to add a user with CN=myname to have full operations using the kafka-acls script with :

You will also need to allow the connecting user to be able to perform admin tasks on the cluster so we can create topics on demand.

You will need to allow group access also.

Create TLS Kubernetes Secrets

Create a secret for the client certificate you created. If you followed the you will need to export your private key from the JKS keystore. The certificate and chain will be provided in PEM format when you get the certificate signed. You can use these to create a secret with:

• tls.key : PEM formatted private key

• tls.crt : PEM formatted certificate

• ca.crt : Certificate chain

kubectl create secret generic aws-msk-client --from-file=./tls.key --from-file=./tls.crt --from-file=./ca.crt -n seldon-mesh

To extract certificates from truststore do:

keytool -importkeystore -srckeystore truststore.jks    -destkeystore truststore.p12    -srcstoretype jks    -deststoretype pkcs12
openssl pkcs12 -in truststore.p12  -nodes -out trust.pem
cat trust.pem | sed  -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt

Add ca.crt to a secret.

kubectl create secret generic aws-msk-broker-ca --from-file=./ca.crt -n seldon-mesh

Example Helm install

We provide a template you can extend in k8s/samples/values-aws-msk-kafka-mtls.yaml.tmpl:

# k8s/samples/values-aws-msk-kafka-mtls.yaml.tmpl
kafka:
  bootstrap: <MSK Broker Endpoints>

security:
  kafka:
    protocol: SSL
    ssl:
      client:
        secret: aws-msk-client
        brokerValidationSecret: aws-msk-broker-ca

Copy this and modify by adding your broker endpoints.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-aws-msk-kafka-mtls.yaml --set kafka.bootstrap=<your aws msk broker endpoints>

Troubleshooting

No messages are being produced to the topics

Set the kafka config map debug setting to "all". For Helm install you can set kafka.debug=all.

If you see an error from the producer in the Pipeline gateway complaining about not enough insync replicas then the replication factor Seldon is using is less than the cluster setting for min.insync.replicas which for a default AWS MSK cluster defaults to 2. Ensure this is equal to that of the cluster. This value can be set in the Helm chart with kafka.topics.replicationFactor.

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Confluent Cloud managed Kafka. In this example we use SASL security mechanism.

Create API Keys

In your Confluent Cloud environment create new API keys. The easiest way to obtain all required information is to head to Clients -> New client (choose e.g. Go) and generate new Kafka cluster API key from there.

This will generate for you:

• Key (we use it as username)

• Secret (we use it as password)

Do not forget to also copy the bootstrap.servers from the example config.

See Confluent Cloud in case of issues.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-confluent-kafka-sasl.yaml.tmpl
kafka:
  bootstrap: < Confluent Cloud Broker Endpoints >
  topics:
    replicationFactor: 3
    numPartitions: 4
  consumer:
    messageMaxBytes: 8388608
  producer:
    messageMaxBytes: 8388608

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: < username >
        secret: confluent-kafka-sasl
    ssl:
      client:
        secret:
        brokerValidationSecret:

Troubleshooting

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Amazon managed Apache Kafka (MSK). You can control access to your Amazon MSK clusters using sign-in credentials that are stored and secured using AWS Secrets Manager. Storing user credentials in Secrets Manager reduces the overhead of cluster authentication such as auditing, updating, and rotating credentials. Secrets Manager also lets you share user credentials across clusters.

Setting up SASL/SCRAM authentication for an Amazon MSK cluster

To setup SASL/SCRAM in an Amazon MSK cluster, please follow the guide from Amazon's Official documentation.

Do not forget to also copy the bootstrap.servers which we will use it in our configuration later below for Seldon.

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

kubectl create secret generic aws-msk-kafka-secret -n seldon-mesh --from-literal password="<MSK SASL Password>"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-aws-msk-kafka-sasl-scram.yaml.tmpl
kafka:
  bootstrap: <msk-bootstrap-server-endpoints>
  topics:
    replicationFactor: 3
    numPartitions: 4

security:
    kafka:
      protocol: SASL_SSL
      sasl:
        mechanism: SCRAM-SHA-512
        client:
          username: < username >
          secret: aws-msk-kafka-secret
          passwordPath: password
      ssl:
        client:
          secret:
          brokerValidationSecret:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration.

Troubleshooting

• Please check Amazon MSK Troubleshooting documentation.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

Cluster Setup

If you have installed Strimzi we have an example Helm chart to create a Kafka cluster for seldon and an associated user in kafka/strimzi folder. Ensure the tls is enabled with:

broker:
  tls:
    enabled: true
    port: 9093
    listenerType: internal
    authentication:
      type: tls

The Ansible setup-ecosystem playbook will also install Strimzi and include a mTLS endpoint. See here.

mTLS Example

Create a Kafka User seldon in the namespace seldon was installed. This assumes Strimzi Kafka cluster is installed in the same namespace or is running with cluster wide permissions. Our Ansible scripts to setup the ecosystem will also create this user if tls is active.

# k8s/samples/strimzi-example-tls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: seldon
  labels:
    strimzi.io/cluster: seldon
spec:
  authentication:
    type: tls

If you don't have this user you can install it with in your desired namespace (here seldon-mesh):

kubectl create -f k8s/samples/strimzi-example-tls-user.yaml -n seldon-mesh

Install seldon with the Strimzi certificate secrets using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-strimzi-kafka-mtls.yaml
kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh.svc.cluster.local:9093

security:
  kafka:
    protocol: SSL
    ssl:
      client:
        secret: seldon
        brokerValidationSecret: seldon-cluster-ca-cert
        keyPath: /tmp/certs/kafka/client/user.key
        crtPath: /tmp/certs/kafka/client/user.crt
        caPath: /tmp/certs/kafka/client/ca.crt
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-strimzi-kafka-mtls.yaml

You can now go ahead and install a SeldonRuntime in your desired install namespace (here seldon-mesh), e.g.

helm install seldon-v2-runtime ../k8s/helm-charts/seldon-core-v2-runtime  -n seldon-mesh

Create a Strimzi Kafka cluster with SASL_SSL enabled. This can be done with our Ansible scripts by running the following from the ansible/ folder:

ansible-playbook playbooks/setup-ecosystem.yaml -e @../k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml -e strimzi_kafka_operator_feature_gates=""

The referenced SASL/SCRAM YAML file looks like the below:

# k8s/samples/ansible-strimzi-kafka-sasl-scram.yaml
seldon_kafka_cluster_values:
  broker:
    tls:
      authentication:
        type: scram-sha-512

This will use the Strimzi Helm chart provided in Seldon Core 2. This will call the Strimzi cluster Helm chart provided by the project with overrides for the cluster authentication type and will also create a user seldon with password credentials in a Kubernetes Secret.

Install Seldon Core 2 with SASL settings using a custom values file. This sets the secret created by Strimzi for the user created above (seldon) and targets the server certificate authority secret from the name of the cluster created on install of the Kafka cluster (seldon-cluster-ca-cert).

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-strimzi-kafka-sasl-scram.yaml
kafka:
  bootstrap: seldon-kafka-bootstrap.seldon-mesh.svc.cluster.local:9093

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret: seldon
        passwordPath: password
    ssl:
      client:
        brokerValidationSecret: seldon-cluster-ca-cert
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh -f k8s/samples/values-strimzi-kafka-sasl-scram.yaml

New in Seldon Core 2.5.0

Seldon Core 2 can integrate with Azure Event Hub via Kafka protocol.

Prerequisites

To start you will need to have an Azure Event Hub Namespace. You can create one following Azure quickstart docs. Note that you do not need to create an Event Hub (topics) as Seldon Core 2 will require all the topics it needs automatically.

Create API Keys

To connect to Azure Event Hub provided Kafka API you need to obtain:

• Kafka Endpoint

• Connection String

You can obtain both using Azure Portal as documented here.

The Connection String should be in format of

Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX

Create Kubernetes Secret

Seldon Core 2 expects password to be in form of K8s secret.

kubectl create secret generic azure-kafka-secret -n seldon-mesh --from-literal password="Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=XXXXXX;SharedAccessKey=XXXXXX"

Configure Seldon Core 2

Configure Seldon Core 2 by setting following Helm values:

# k8s/samples/values-azure-event-hub-sasl.yaml.tmpl
kafka:
  bootstrap: <namespace>.servicebus.windows.net
  topics:
    replicationFactor: 3
    numPartitions: 4

security:
  kafka:
    protocol: SASL_SSL
    sasl:
      mechanism: "PLAIN"
      client:
        username: $ConnectionString
        secret: azure-kafka-secret
    ssl:
      client:
        secret:
        brokerValidationSecret:

Note you may need to tweak replicationFactor and numPartitions to your cluster configuration. The username should read $ConnectionString and this is not a variable for you to replace.

Troubleshooting

• First check Azure Event Hub troubleshooting guide.

• Set the kafka config map debug setting to all. For Helm install you can set kafka.debug=all.

• Verify that you did not hit quotas for topics or partitions in your Event Hub namespace.

Helm Settings

# k8s/helm-charts/seldon-core-v2-setup/values.yaml
security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
        keyPath: /tmp/certs/cps/tls.key
        crtPath: /tmp/certs/cps/tls.crt
        caPath: /tmp/certs/cps/ca.crt
        clientCaPath: /tmp/certs/cpc/ca.crt
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
        keyPath: /tmp/certs/cpc/tls.key
        crtPath: /tmp/certs/cpc/tls.crt
        caPath: /tmp/certs/cpc/ca.crt
        serverCaPath: /tmp/certs/cps/ca.crt
  kafka:
    protocol: PLAINTEXT
    sasl:
      mechanism: SCRAM-SHA-512
      client:
        username: seldon
        secret:
        passwordPath: password
    ssl:
      client:
        secret:
        brokerValidationSecret:
        keyPath: /tmp/certs/kafka/client/tls.key
        crtPath: /tmp/certs/kafka/client/tls.crt
        caPath: /tmp/certs/kafka/client/ca.crt
        brokerCaPath: /tmp/certs/kafka/broker/ca.crt
        endpointIdentificationAlgorithm:
  envoy:
    protocol: PLAINTEXT
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
          keyPath: /tmp/certs/dus/tls.key
          crtPath: /tmp/certs/dus/tls.crt
          caPath: /tmp/certs/dus/ca.crt
          clientCaPath: /tmp/certs/duc/ca.crt
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
          keyPath: /tmp/certs/duc/tls.key
          crtPath: /tmp/certs/duc/tls.crt
          caPath: /tmp/certs/duc/ca.crt
          serverCaPath: /tmp/certs/dus/ca.crt
      downstream:
        server:
          secret: seldon-downstream-server
          clientValidationSecret:
          keyPath: /tmp/certs/dds/tls.key
          crtPath: /tmp/certs/dds/tls.crt
          caPath: /tmp/certs/dds/ca.crt
          clientCaPath: /tmp/certs/ddc/ca.crt
        client:
          mtls: false
          secret:
          serverValidationSecret: seldon-downstream-server
          keyPath: /tmp/certs/ddc/tls.key
          crtPath: /tmp/certs/ddc/tls.crt
          caPath: /tmp/certs/ddc/ca.crt
          serverCaPath: /tmp/certs/dds/ca.crt

# A list of image pull secrets
imagePullSecrets:

Environment variables

Kubernetes secrets and mounted files can be used to provide the certificates in PEM format. These are controlled by environment variables for server or client depending on the component:

Control Plane

For a server (scheduler):

For a client (agent, modelgateway, hodometer, CRD controller):

Kafka

Envoy

Envoy xDS server will use the control plane server and client certificates defined above.

Downstream server

Downstream client

Upstream server

Upstream client