Self-hosted Kafka

You can run Kafka in the same Kubernetes cluster that hosts the Seldon Enterprise Platform. We recommend using the Strimzi operator for Kafka installation and maintenance.

Note: These instructions help you quickly set up a Kafka cluster. For production grade installation consult Strimzi documentation or use one of managed solutions .

Integrating self-hosted Kafka with Seldon Core 2 includes these steps:

1. Install Kafka

2. Configure Seldon Core 2

Installing Kafka in a Kubernetes cluster

Strimzi provides a Kubernetes Operator to deploy and manage Kafka clusters. First, we need to install the Strimzi Operator in your Kubernetes cluster.

1. Create a namespace where you want to install Kafka. For example the name space kafka:

   kubectl create namespace kafka || echo "namespace kafka exists"

2. Install Strimzi.

   helm repo add strimzi https://strimzi.io/charts/
   helm repo update

3. Install Strimzi Operator.

   helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace kafka

   This deploys the Strimzi Operator in the kafka namespace.

4. Next, you need to create a Kafka cluster by saving the following YAML configuration to a file named kafka.yaml and applying it with kubectl apply -f kafka.yaml:

    apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    metadata:
      name: seldon
      namespace: kafka
    spec:
      kafka:
        replicas: 3
        version: 3.7.0
        config:
          auto.create.topics.enable: true
          default.replication.factor: 1
          inter.broker.protocol.version: 3.7
          min.insync.replicas: 1
          offsets.topic.replication.factor: 1
          transaction.state.log.min.isr: 1
          transaction.state.log.replication.factor: 1
        listeners:
        - name: plain
          port: 9092
          tls: false
          type: internal
        storage:
          type: ephemeral
      zookeeper:
        replicas: 1
        storage:
          type: ephemeral

   This will set up a Kafka cluster with version 3.7.0. Ensure that you have reviewed the supported versions of Kafka and updated the version in the kafka.yaml file as needed.

5. Check the status of the Kafka pods to ensure they are running properly:

   kubectl get pods -n kafka

   You should see multiple pods for Kafka, Zookeeper, and Strimzi operators running.

   NAME                                       READY   STATUS    RESTARTS   AGE
   seldon-kafka-0                             1/1     Running   0          2d1h
   seldon-kafka-1                             1/1     Running   0          2d1h
   seldon-kafka-2                             1/1     Running   0          2d1h
   seldon-zookeeper-0                         1/1     Running   0          2d1h
   strimzi-cluster-operator-58ff6ccf5-pqfjn   1/1     Running   0          2d1h

Configuring Seldon Core 2

To integrate Kafka with Seldon Core 2:

1. Update the configuration for Kafka to integrate with Seldon Core 2 Operator. For example, update the components-values.yaml file. Use your preferred text editor to update and save the file with the following content:

   controller:
     clusterwide: true
   
   dataflow:
     resources:
       cpu: 500m
   
   envoy:
     service:
       type: ClusterIP
   
   kafka:
     bootstrap: seldon-kafka-bootstrap.kafka:9092
     topicPrefix: seldon
     debug:
     consumer:
       autoOffsetReset: earliest
       sessionTimeoutMs: 6000
       topicMetadataRefreshIntervalMs: 1000
       topicMetadataPropagationMaxMs: 300000
       messageMaxBytes: 1000000000
     producer:
       lingerMs: 0
       messageMaxBytes: 1000000000
     topics:
       replicationFactor: 1
       numPartitions: 1
   
   opentelemetry:
     enable: false
   
   scheduler:
     service:
       type: ClusterIP
   
   serverConfig:
     mlserver:
       resources:
         cpu: 1
         memory: 2Gi
   
     triton:
       resources:
         cpu: 1
         memory: 2Gi
   
   serviceGRPCPrefix: "http2-"

2. Change to the directory that contains the components-values.yaml file and then install Seldon Core 2 operator in the namespace seldon-system.

    helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.5 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install