# Self-hosted Kafka

You can run Kafka in the same Kubernetes cluster that hosts the Seldon Enterprise Platform. We recommend using the [Strimzi operator](https://strimzi.io/docs/operators/latest/deploying) for Kafka installation and maintenance.

{% hint style="info" %}
**Note**: These instructions help you quickly set up a Kafka cluster. For production grade installation consult [Strimzi documentation](https://strimzi.io/documentation/) or use one of [managed solutions](/seldon-enterprise-platform/production-environment.md#managed-kafka) .
{% endhint %}

Integrating self-hosted Kafka with Seldon Core 2 includes these steps:

1. [Install Kafka](#installing-kafka-in-a-kubernetes-cluster)
2. [Configure Seldon Core 2](#configuring-seldon-core-2)

## Installing Kafka in a Kubernetes cluster

Strimzi provides a Kubernetes Operator to deploy and manage Kafka clusters. First, we need to install the Strimzi Operator in your Kubernetes cluster.

1. Create a namespace where you want to install Kafka. For example the name space `kafka`:

   ```
   kubectl create namespace kafka || echo "namespace kafka exists"
   ```
2. Install Strimzi.

   ```
   helm repo add strimzi https://strimzi.io/charts/
   helm repo update
   ```
3. Install Strimzi Operator.

   ```
   helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace kafka
   ```

   This deploys the `Strimzi Operator` in the `kafka` namespace.
4. Next, you need to create a Kafka cluster by saving the following YAML configuration to a file named `kafka.yaml` and applying it with `kubectl apply -f kafka.yaml`:

   ```yaml
    apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    metadata:
      name: seldon
      namespace: kafka
    spec:
      kafka:
        replicas: 3
        version: 3.7.0
        config:
          auto.create.topics.enable: true
          default.replication.factor: 1
          inter.broker.protocol.version: 3.7
          min.insync.replicas: 1
          offsets.topic.replication.factor: 1
          transaction.state.log.min.isr: 1
          transaction.state.log.replication.factor: 1
        listeners:
        - name: plain
          port: 9092
          tls: false
          type: internal
        storage:
          type: ephemeral
      zookeeper:
        replicas: 1
        storage:
          type: ephemeral
   ```

   This will set up a Kafka cluster with version 3.7.0. Ensure that you have reviewed the [supported versions](https://strimzi.io/downloads/) of Kafka and updated the version in the `kafka.yaml` file as needed.
5. Check the status of the Kafka pods to ensure they are running properly:

   ```
   kubectl get pods -n kafka
   ```

   You should see multiple pods for Kafka, Zookeeper, and Strimzi operators running.

   ```
   NAME                                       READY   STATUS    RESTARTS   AGE
   seldon-kafka-0                             1/1     Running   0          2d1h
   seldon-kafka-1                             1/1     Running   0          2d1h
   seldon-kafka-2                             1/1     Running   0          2d1h
   seldon-zookeeper-0                         1/1     Running   0          2d1h
   strimzi-cluster-operator-58ff6ccf5-pqfjn   1/1     Running   0          2d1h
   ```

## Configuring Seldon Core 2

To integrate Kafka with Seldon Core 2:

1. Update the configuration for Kafka to integrate with Seldon Core 2 Operator. For example, update the `components-values.yaml` file. Use your preferred text editor to update and save the file with the following content:

   ```
   controller:
     clusterwide: true

   dataflow:
     resources:
       cpu: 500m

   envoy:
     service:
       type: ClusterIP

   kafka:
     bootstrap: seldon-kafka-bootstrap.kafka:9092
     topicPrefix: seldon
     debug:
     consumer:
       autoOffsetReset: earliest
       sessionTimeoutMs: 6000
       topicMetadataRefreshIntervalMs: 1000
       topicMetadataPropagationMaxMs: 300000
       messageMaxBytes: 1000000000
     producer:
       lingerMs: 0
       messageMaxBytes: 1000000000
     topics:
       replicationFactor: 1
       numPartitions: 1

   opentelemetry:
     enable: false

   scheduler:
     service:
       type: ClusterIP

   serverConfig:
     mlserver:
       resources:
         cpu: 1
         memory: 2Gi

     triton:
       resources:
         cpu: 1
         memory: 2Gi

   serviceGRPCPrefix: "http2-"
   ```
2. Change to the directory that contains the `components-values.yaml` file and then install Seldon Core 2 operator in the namespace `seldon-system`.

   ```bash
    helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.5 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install
   ```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.seldon.ai/seldon-enterprise-platform/learning-environment/self-hosted-kafka.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.