> For the complete documentation index, see [llms.txt](https://docs.seldon.ai/seldon-enterprise-platform/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seldon.ai/seldon-enterprise-platform/learning-environment/self-hosted-kafka.md).

# Self-hosted Kafka

You can run Kafka in the same Kubernetes cluster that hosts the Seldon Enterprise Platform. We recommend using the [Strimzi operator](https://strimzi.io/docs/operators/latest/deploying) for Kafka installation and maintenance.

{% hint style="info" %}
**Note**: These instructions help you quickly set up a Kafka cluster. For production grade installation consult [Strimzi documentation](https://strimzi.io/documentation/) or use one of [managed solutions](/seldon-enterprise-platform/production-environment.md#managed-kafka) .
{% endhint %}

Integrating self-hosted Kafka with Seldon Core 2 includes these steps:

1. [Install Kafka](#installing-kafka-in-a-kubernetes-cluster)
2. [Configure Seldon Core 2](#configuring-seldon-core-2)

## Installing Kafka in a Kubernetes cluster

Strimzi provides a Kubernetes Operator to deploy and manage Kafka clusters. First, we need to install the Strimzi Operator in your Kubernetes cluster.

1. Create a namespace where you want to install Kafka. For example the name space `kafka`:

   ```
   kubectl create namespace kafka || echo "namespace kafka exists"
   ```
2. Install Strimzi.

   ```
   helm repo add strimzi https://strimzi.io/charts/
   helm repo update
   ```
3. Install Strimzi Operator.

   ```
   helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace kafka
   ```

   This deploys the `Strimzi Operator` in the `kafka` namespace.
4. Next, you need to create a Kafka cluster by saving the following YAML configuration to a file named `kafka.yaml` and applying it with `kubectl apply -f kafka.yaml`:

   ```yaml
    apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    metadata:
      name: seldon
      namespace: kafka
    spec:
      kafka:
        replicas: 3
        version: 3.7.0
        config:
          auto.create.topics.enable: true
          default.replication.factor: 1
          inter.broker.protocol.version: 3.7
          min.insync.replicas: 1
          offsets.topic.replication.factor: 1
          transaction.state.log.min.isr: 1
          transaction.state.log.replication.factor: 1
        listeners:
        - name: plain
          port: 9092
          tls: false
          type: internal
        storage:
          type: ephemeral
      zookeeper:
        replicas: 1
        storage:
          type: ephemeral
   ```

   This will set up a Kafka cluster with version 3.7.0. Ensure that you have reviewed the [supported versions](https://strimzi.io/downloads/) of Kafka and updated the version in the `kafka.yaml` file as needed.
5. Check the status of the Kafka pods to ensure they are running properly:

   ```
   kubectl get pods -n kafka
   ```

   You should see multiple pods for Kafka, Zookeeper, and Strimzi operators running.

   ```
   NAME                                       READY   STATUS    RESTARTS   AGE
   seldon-kafka-0                             1/1     Running   0          2d1h
   seldon-kafka-1                             1/1     Running   0          2d1h
   seldon-kafka-2                             1/1     Running   0          2d1h
   seldon-zookeeper-0                         1/1     Running   0          2d1h
   strimzi-cluster-operator-58ff6ccf5-pqfjn   1/1     Running   0          2d1h
   ```

## Configuring Seldon Core 2

To integrate Kafka with Seldon Core 2:

1. Update the configuration for Kafka to integrate with Seldon Core 2 Operator. For example, update the `components-values.yaml` file. Use your preferred text editor to update and save the file with the following content:

   ```
   controller:
     clusterwide: true

   dataflow:
     resources:
       cpu: 500m

   envoy:
     service:
       type: ClusterIP

   kafka:
     bootstrap: seldon-kafka-bootstrap.kafka:9092
     topicPrefix: seldon
     debug:
     consumer:
       autoOffsetReset: earliest
       sessionTimeoutMs: 6000
       topicMetadataRefreshIntervalMs: 1000
       topicMetadataPropagationMaxMs: 300000
       messageMaxBytes: 1000000000
     producer:
       lingerMs: 0
       messageMaxBytes: 1000000000
     topics:
       replicationFactor: 1
       numPartitions: 1

   opentelemetry:
     enable: false

   scheduler:
     service:
       type: ClusterIP

   serverConfig:
     mlserver:
       resources:
         cpu: 1
         memory: 2Gi

     triton:
       resources:
         cpu: 1
         memory: 2Gi

   serviceGRPCPrefix: "http2-"
   ```
2. Change to the directory that contains the `components-values.yaml` file and then install Seldon Core 2 operator in the namespace `seldon-system`.

   ```bash
    helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.5 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install
   ```