# Scalability of Pipelines

Core 2 supports full **horizontal scaling** for the **dataflow engine**, **model gateway**, and **pipeline gateway**. Each service automatically distributes pipelines or models across replicas using consistent hashing, so you don’t need to manually assign workloads.

This guide explains **how scaling works**, **what configuration controls it**, and **what happens when replicas or pipelines/models change**.

## 1. How scaling works (at a glance)

| Component            | What it scales with (max `maxShardCountMultiplier` = #partitions )              | Max replicas used                                             |
| -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **Dataflow engine**  | `#pipelines × maxShardCountMultiplier` (capped by configured replicas)          | `min(replicas, pipelines × partitions)`                       |
| **Model gateway**    | `#models × maxShardCountMultiplier` (capped by replicas and maxNumConsumers)    | `min(replicas, min(models, maxNumConsumers) × partitions)`    |
| **Pipeline gateway** | `#pipelines × maxShardCountMultiplier` (capped by replicas and maxNumConsumers) | `min(replicas, min(pipelines, maxNumConsumers) × partitions)` |

Each pipeline/model **is loaded only on a subset of replicas**, and **automatically rebalanced** when:

1. You **scale replicas up/down**
2. You **deploy or delete pipelines / models**

The configuration parameter determining the maximum number of component replicas on which a pipeline/model can be loaded is `maxShardCountMultiplier`. It can be set in `SeldonConfig` under `config.scalingConfig.pipelines.maxShardCountMultiplier`. For installs via helm, the value of this parameter defaults to `{{ .Values.kafka.topics.numPartitions }}`. In fact, the number of kafka partitions per topic is the maximum value that `maxShardCountMultiplier` should be set to. Increasing this value beyond the number of kafka partitions not only does not bring any additional performance benefits, but may actually lead to dropped requests due to the extra replicas receiving requests but managing no kafka partitions within their consumer groups.

This parameter may be changed during cluster operation, with the new value being propagated to all components over a \~1 minute interval. Changing this value will cause pipelines/models to be rebalanced across the `dataflow-engine`, `model-gateway`, and `pipeline-gateway` replicas and may lead to downtime depending on the configured kafka partition assignment strategy. If the used Kafka version supports cooperative rebalancing of consumer groups, then setting the partition assignment strategy to `cooperative-sticky` will ensure that rebalancing happens with minimal disruption. `dataflow-engine` uses a cooperative rebalancing strategy by default.

You **do not** need to manually assign work — it’s handled automatically.

## 2. Scaling the dataflow engine

**Dataflow engine** is responsible for **executing pipeline logic** and moving data between pipeline stages. Core 2 now supports running **multiple pipelines in parallel across multiple dataflow engine replicas**.

### 2.1. What controls scaling?

You control scaling using:

| Config                                                   | Location        | Purpose                                                                     |
| -------------------------------------------------------- | --------------- | --------------------------------------------------------------------------- |
| `spec.replicas`                                          | `SeldonRuntime` | Maximum number of dataflow engine instances                                 |
| `config.scalingConfig.pipelines.maxShardCountMultiplier` | `SeldonConfig`  | Determines max replication per pipeline (max possible: `#Kafka partitions`) |

### 2.2. How many replicas will actually be used?

Dataflow engine replicas are dynamically adjusted based on **number of pipelines deployed** and **Kafka partitions**. The final number of dataflow engine replicas is given by:

$\text{FinalReplicaCount} = \min(\text{spec.replicas},\ \text{pipelines} \times \text{partitions})$

**Example**

| Pipelines deployed | maxShardCountMultiplier | spec.replicas | Final dataflow replicas used      |
| ------------------ | ----------------------- | ------------- | --------------------------------- |
| 3                  | 4                       | 9             | `min(9, 3 x 4 = 12)` → 9 replicas |
| 2                  | 4                       | 9             | `min(9, 2 x 4 = 8)` → 8 replicas  |
| 1                  | 4                       | 9             | `min(9. 1 x 4 = 4)` → 4 replicas  |

**Note**: Unused replicas are automatically scaled down. As more pipelines are added, dataflow engine automatically scales up, capped by the maximum number of replicas.

### 2.3. How are pipeline assigned to replicas?

* Core 2 uses **consistent hashing** to distribute pipelines evenly across dataflow replicas. This ensures a **balanced workload**, but it does *not* guarantee a perfect one-to-one mapping.
  * Even if the number of replicas equals `pipelines × partitions`, small imbalances between the number of pipelines handled by each replica may exist. In practice, the distribution is **statistically uniform**.
* Each pipeline is **replicated across multiple dataflow engines** (up to number of Kafka partitions).
* When instances are added or removed, **pipelines are automatically rebalanced**.

**Note:** This process is handled internally by Core 2, so no manual intervention is needed.

### 2.4. Loading/unloading of the pipelines from dataflow engine

* Loading/unloading of the pipeline from the dataflow engine is performed when the pipeline CR is loaded/unloaded.
* The scheduler confirms whether the loading/unloading was performed successfully through the `Pipeline` status under the CR.

Rebalancing happens in the background — you don’t need to intervene.

**Note:** For pipelines, `Pipeline` ready status must be satisfied in order for the pipeline to be marked ready.

## 3. Scaling the model gateway

The **model gateway** is responsible for routing inference requests to models when used inside pipelines. Like the **dataflow engine**, it scales dynamically based on *how many models are deployed*.

### 3.1. What controls scaling?

| Config                                                   | Location                                              | Purpose                                                    |
| -------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------- |
| `spec.replcias`                                          | `SeldonRuntime`                                       | Maximum number of model gateway instances                  |
| `config.scalingConfig.pipelines.maxShardCountMultiplier` | `SeldonConfig`                                        | Determines max replication per model (`#Kafka partitions`) |
| `maxNumConsumers`                                        | `SeldonConfig` - model gateway enc var (default: 100) | Caps how many distinct consumer groups can exist           |

### 3.2. How many replicas will actually be used?

Model gateway replicas are dynamically adjusted based on **number of models deployed**, **Kafka partitions**, and **maxNumConsumers**. The final number of model gateway replicas is given by:

$\text{FinalReplicaCount} = \min(\text{spec.replicas},\ \min(\text{models}, \text{maxNumConsumers}) \times \text{partitions})$

**Example**

| Models Deployed | maxShardCountMultiplier | spec.replicas | maxNumConsumers | Final model gateway replicas                       |
| --------------- | ----------------------- | ------------- | --------------- | -------------------------------------------------- |
| 5               | 4                       | 20            | 100             | `min(20, min(5, 100) x 4 = 20) = 20` → 20 replicas |
| 1               | 4                       | 20            | 100             | `min(20, min(1, 100) x 4 = 4) = 4` → 4 replicas    |

**Note:** If you remove models, the model gateway automatically scales down, and if we add models, the model gateway automatically scales up, capped by the maximum number of replicas.

### 3.3. How are models assigned to replicas?

Model gateway doesn’t load every model on every replica but only on a subset of replicas. The same principle as for dataflow engine applies for model gateway (sharding through consistent hashing).

### 3.4. Loading/unloading of the models from model gateway

* Loading/unloading of the model from the model gateway is performed when the model CR is loaded/unloaded.
* The scheduler confirms whether the loading/unloading was performed successfully through the `ModelGw` status under the CR.

Rebalancing happens in the background — you don’t need to intervene.

**Note:** `ModelGw` status does not represent a condition for the model to be available. If the loading was successful on the dedicated servers, the model itself is ready for inference.

* The `ModelGw` status becomes relevant for pipelines, or whether the end user wants to perform inference via the async path (i.e., writing the requests in the model input topic and reading the responses from the model output topic from Kafka).
* In the context of pipelines, the `ModelReady` status becomes a conjunction on whether the model is available on servers and if the model has been loaded successfully on the model gateway.

## 4. Scaling the pipeline gateway

The **pipeline gateway** is responsible for writing the requests in the input topic of the pipeline, and wait for the response on the output topic. Like dataflow engine and model gateway, pipeline gateway can scale horizontally.

### 4.1. What Controls Scaling?

| Config                                                   | Location                                                 | Purpose                                                        |
| -------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------- |
| `spec.replcias`                                          | `SeldonRuntime`                                          | Maximum number of pipeline gateway instances                   |
| `config.scalingConfig.pipelines.maxShardCountMultiplier` | `SeldonConfig`                                           | Determines max replication per pipeline (`# Kafka partitions`) |
| `maxNumConsumers`                                        | `SeldonConfig` - Pipeline gateway enc var (default: 100) | Caps how many distinct consumer groups can exist               |

### 4.2. How many replicas will actually be used?

Pipeline gateway replicas are dynamically adjusted based on **number of pipelines deployed**, **Kafka partitions**, and **maxNumConsumers**. The final number of pipeline gateway replicas is given by:

$\text{FinalReplicaCount} = \min(\text{spec.replicas},\ \min(\text{pipelines}, \text{maxNumConsumers}) \times \text{partitions})$

**Example**

| Pipelines Deployed | maxShardCountMultiplier | spec.replicas | maxNumConsumers | Final pipeline gateway replicas                    |
| ------------------ | ----------------------- | ------------- | --------------- | -------------------------------------------------- |
| 8                  | 4                       | 10            | 100             | `min(10, min(8, 100) x 4 = 32) = 10` → 10 replicas |
| 2                  | 4                       | 10            | 100             | `min(10, min(2, 100) x 4 = 8) = 8` → 8 replicas    |
| 1                  | 4                       | 10            | 100             | `min(10, min(1, 100) x 4 = 4) = 4`→ 4 replicas     |

**Note:** Similarly to dataflow engine, pipeline gateway scales up and down as pipeline are added and removed.

### 4.3. How are pipeline assigned to replicas?

Pipeline gateway doesn’t load every pipeline on every replica but only on a subset of replicas. The same principle as for dataflow engine and model gateway applies for pipeline gateway (sharding through consistent hashing).

### 4.4. Loading/unloading of the pipelines from Pipeline Gateway

* Loading/unloading of the pipeline from the pipeline gateway is performed when the pipeline CR is loaded/unloaded.
* The scheduler confirms whether the loading/unloading was performed successfully through the `PipelineGw` status under the CR.

Analogous with the previous services, rebalancing happens in the background — you don’t need to intervene.

**Note:** For pipelines, `PipelineGw` ready status must be satisfied in order for the pipeline to be marked ready.