After the models are deployed, Core 2 enables the monitoring and experimentation on those systems in production. With support for a wide range of model types, and design patterns to build around those models, you can standardize ML deployment across a range of use-cases in the cloud or on-premise serving infrastructure of your choice.

Model Deployment

Seldon Core 2 orchestrates and scales machine learning components running as production-grade microservices. These components can be deployed locally or in enterprise-scale kubernetes clusters. The components of your ML system - such as models, processing steps, custom logic, or monitoring methods - are deployed as Models, leveraging serving solutions compatible with Core 2 such as MLServer, Alibi, LLM Module, or Triton Inference Server. These serving solutions package the required dependencies and standardize inference using the Open Inference Protocol. This ensures that, regardless of your model types and use-cases, all request and responses follow a unified format. After models are deployed, they can process REST or gRPC requests for real-time inference.

Complex Applications & Orchestration

Machine learning applications are increasingly complex. They've evolved from individual models deployed as services, to complex applications that can consist of multiple models, processing steps, custom logic, and asynchronous monitoring components. With Core you can build Pipelines that connect any of these components to make data-centric applications. Core 2 handles orchestration and scaling of the underlying components of such an application, and exposes the data streamed through the application in real time using Kafka.

This approach to MLOps, influenced by our position paper , enables real-time observability, insight, and control on the behavior, and performance of your ML systems.

Lastly, Core 2 provides Experiments as part of its orchestration capabilities, enabling users to implement routing logic such as A/B tests or Canary deployments to models or pipelines in production. After experiments are run, you can promote new models and/or pipelines, or launch new experiments, so that you can continuously improve the performance of your ML applications.

Resource Management

In Seldon Core 2 your models are deployed on inference servers, which are software that manage the packaging and execution of ML workloads. As part its design, Core 2 separates out Servers and Models as separate resources. This approach enables flexible allocation of models to servers aligning with the requirements of your models, and to the underlying infrastructure that you want your servers to run on. Core 2 also provides functionality to autoscale your models and servers up and down as needed based on your workload requirements or user-defined metrics.

With the modular design of Core 2, users are able to implement cutting-edge methods to minimize hardware costs:

• Multi-Model serving consolidates multiple models onto shared inference servers to optimize resource utilization and decrease the number of servers required.

• Over-commit allows you to provision more models than the available memory would normally allow by dynamically loading and unloading models from memory to disk based on demand.

End-to-End MLOps with Core 2

Core 2 demonstrates the power of a standardized, data-centric approach to MLOps at scale, ensuring that data observability and management are prioritized across every layer of machine learning operations. Furthermore, Core 2 seamlessly integrates into end-to-end MLOps workflows, from CI/CD, managing traffic with the service mesh of your choice, alerting, data visualization, or authentication and authorization.

This modular, flexible architecture not only supports diverse deployment patterns but also ensures compatibility with the latest AI innovations. By embedding data-centricity and adaptability into its foundation, Core 2 equips organizations to scale and improve their machine learning systems effectively, to capture value from increasingly complex AI systems.

Next Steps

• Explore our

• for updates or for answers to any questions

Seldon Core 2 is a Kubernetes-native framework for deploying and managing machine learning (ML) and Large Language Model (LLM) systems at scale. Its data-centric approach and modular architecture enable seamless handling of simple models to complex ML applications across on-premise, hybrid, and multi-cloud environments while ensuring flexibility, standardization, observability, and cost efficiency.

Flexibility: Real-time, your way

Seldon Core 2 offers a platform-agnostic, flexible framework for seamless deployment of different types of ML models across on-premise, cloud, and hybrid environments. Its adaptive architecture enables customizable applications, future-proofing MLOps or LLMOps by scaling deployments as data and applications evolve. The modular design enhances resource-efficiency, allowing dynamic scaling, component reuse, and optimized resource allocation. This ensures long-term scalability, operational efficiency, and adaptability to changing demands.

Standardization: Consistency across workflows

Seldon Core 2 enforces best practices for ML deployment, ensuring consistency, reliability, and efficiency across the entire lifecycle. By automating key deployment steps, it removes operational bottlenecks, enabling faster rollouts and allowing teams to focus on high-value tasks.

With a "learn once, deploy anywhere" approach, Seldon Core 2 standardizes model deployment across on-premise, cloud, and hybrid environments, reducing risk and improving productivity. Its unified execution framework supports conventional, foundational, and LLM models, streamlining deployment and enabling seamless scalability. It also enhances collaboration between MLOps Engineers, Data Scientists, and Software Engineers by providing a customizable framework that fosters knowledge sharing, innovation, and the adoption of new data science capabilities.

Enhanced Observability

Observability in Seldon Core 2 enables real-time monitoring, analysis, and performance tracking of ML systems, covering data pipelines, models, and deployment environments. Its customizable framework combines and monitoring, ensuring teams have the key metrics needed for maintenance and decision-making.

Seldon simplifies operational monitoring, allowing real-time ML or LLM deployments to expand across organizations while supporting complex, mission-critical use cases. A ensures all prediction data is auditable, maintaining explainability, compliance, and trust in AI-driven decisions.

Optimization: Modularity to maximize resource efficiency

Seldon Core 2 is built for scalability, efficiency, and cost-effective ML operations, enabling you to deploy only the necessary components while maintaining agility and high performance. Its modular architecture ensures that resources are optimized, infrastructure is consolidated, and deployments remain adaptable to evolving business needs.

Scaling for Efficiency: infrastructure dynamically based on demand, auto-scaling for real-time use cases while scaling to zero for on-demand workloads, preserving deployment state for seamless reactivation. By eliminating redundancy and optimizing deployments, it balances cost efficiency and performance, ensuring reliable inference at any scale.

Consolidated Serving Infrastructure: maximizes resource utilization with multi-model serving (MMS) and overcommit, reducing compute overhead while ensuring efficient, reliable inference.

Extendability & Modular Adaptation: integrates with LLMs, Alibi, and other modules, enabling on-demand ML expansion. Its modular design ensures scalable AI, maximizing value extraction, agility, and cost efficiency.

Reusability for Cost Optimization: provides predictable, fixed pricing, enabling cost-effective scaling and innovation while ensuring financial transparency and flexibility.

Next Steps

• Explore our other

• Learn about the

• for updates or for answers to any questions

In the context of machine learning and Seldon Core 2, concepts provide a framework for understanding key functionalities, architectures, and workflows within the system. Some of the key concepts in Seldon Core 2 are:

• Data-Centric MLOps

• Open Inference Protocol

• Components

Data-Centric MLOps

Data-centricity is an approach that puts the management, integrity, and flow of data at the core of machine learning deployment. Rather than focusing solely on models, this approach ensures that data quality, consistency, and adaptability drive successful ML operations. In Seldon Core 2, data-centricity is embedded in every stage of the inference workflow.

Why Data-Centricity Matters?

By adopting a data-centric approach, Seldon Core 2 enables:

• More reliable and high-quality predictions by ensuring clean, well-structured data.

• Scalable and future-proof ML deployments through standardized data management.

• Efficient monitoring and maintenance, reducing risks related to model drift and inconsistencies.

With data-centricity as a core principle, Seldon Core 2 ensures end-to-end control over ML workflows, enabling you to maximize model performance and reliability in production environments.

Open Inference Protocol in Seldon Core 2

The Open Inference Protocol (OIP) defines a standard way for inference servers and clients to communicate in Seldon Core 2. Its goal is to enable interoperability, flexibility, and consistency across different model-serving runtimes. It exposes the Health API, Metadata API, and the Inference API.

Some of the features of Open Inference Protocol includes:

• Transport Agnostic: Servers can implement HTTP/REST or gRPC protocols.

• Runtime Awareness: Use the protocolVersion field in your runtime YAML or consult the supported runtimes table to verify compatibility.

By adopting OIP, Seldon Core 2 promotes a consistent experience across a diverse set of model deployments.

Components in Seldon Core 2

Components are the building blocks of an inference graph, processing data at various stages of the ML inference pipeline. They provide reusable, standardized interfaces, making it easier to maintain and update workflows without disrupting the entire system. Components include ML models, data processors, routers, and supplementary services.

Types of Components

By modularizing inference workflows, components allow you to scale, experiment, and optimize ML deployments efficiently while ensuring data consistency and reliability.

Pipelines

In a model serving platform like Seldon Core 2, a pipeline refers to an orchestrated sequence of models and components that work together to serve more complex AI applications. These pipelines allow you to connect models, routers, transformers, and other inference components, with Kafka used to stream data between them. Each component in the pipeline is modular and independently managed—meaning it can be scaled, configured, and updated separately based on its specific input and output requirements.

This use of "pipeline" is distinct from how the term is used in MLOps for CI/CD pipelines, which automate workflows for building, testing, and deploying models. In contrast, Core 2 pipelines operate at runtime and focus on the live composition and orchestration of inference systems in production.

Servers

In Core 2, servers are responsible for hosting and serving machine learning models, handling inference requests, and ensuring scalability, efficiency, and observability in production. Core 2 supports multiple inference servers, including MLServer, and NVIDIA Triton Inference Server, enabling flexible and optimized model deployments.

MLServer: A lightweight, extensible inference server designed to work with multiple ML frameworks, including scikit-learn, XGBoost, TensorFlow, and PyTorch. It supports custom Python models and integrates well with MLOps workflows. It is built for *General-purpose model serving, custom model wrappers, multi-framework support.

Experiments

In Seldon Core 2, experiments enable controlled A/B testing, model comparisons, and performance evaluations by defining an HTTP traffic split between different models or inference pipelines. This allows organizations to test multiple versions of a model in production while managing risk and ensuring continuous improvements.

Some of the advantages of using Experiments:

• A/B Testing & Model comparison: Compare different models under real-world conditions without full deployment.

• Risk-Free model validation: Test a new model or pipeline in parallel without affecting live predictions.

• Performance monitoring: Assess latency, accuracy, and reliability before a full rollout.

• Continuous improvement: Make data-driven deployment decisions based on real-time model performance.

Seldon Core 2 uses a microservice architecture where each service has limited and well-defined responsibilities working together to orchestrate scalable and fault-tolerant ML serving and management. These components communicate internally using gRPC and they can be scaled independently. Seldon Core 2 services can be split into two categories:

• Control Plane services are responsible for managing the operations and configurations of your ML models and workflows. This includes functionality to instantiate new inference servers, load models, update new versions of models, configure model experiments and pipelines, and expose endpoints that may receive inference requests. The main control plane component is the Scheduler that is responsible for managing the loading and unloading of resources (models, pipelines, experiments) onto the respective components.

• Data Plane services are responsible for managing the flow of data between components or models. Core 2 supports REST and gRPC payloads that follow the Open Inference Protocol (OIP). The main data plane service is

We utilize Rclone to copy model artifacts from a storage location to the model servers. This allows users to take advantage of Rclones support for over 40 cloud storage backends including Amazon S3, Google Storage and many others.

For authorization needed for cloud storage when running on Kubernetes see here.

Seldon Core 2 can be installed in various setups to suit different stages of the development lifecycle. The most common modes include:

Local Environment

Ideal for development and testing purposes, a local setup allows for quick iteration and experimentation with minimal overhead. Common tools include:

• Docker Compose: Simplifies deployment by orchestrating Seldon Core components and dependencies in Docker containers. Suitable for environments without Kubernetes, providing a lightweight alternative.

• Kind (Kubernetes IN Docker): Runs a Kubernetes cluster inside Docker, offering a realistic testing environment. Ideal for experimenting with Kubernetes-native features.

Production Environment

Designed for high-availability and scalable deployments, a production setup ensures security, reliability, and resource efficiency. Typical tools and setups include:

• Managed Kubernetes Clusters: Platforms like GKE (Google Kubernetes Engine), EKS (Amazon Elastic Kubernetes Service), and AKS (Azure Kubernetes Service) provide managed Kubernetes solutions. Suitable for enterprises requiring scalability and cloud integration.

• On-Premises Kubernetes Clusters: For organizations with strict compliance or data sovereignty requirements. Can be deployed on platforms like OpenShift or custom Kubernetes setups.

By selecting the appropriate installation mode—whether it's Docker Compose for simplicity, Kind for local Kubernetes experimentation, or production-grade Kubernetes for scalability—you can effectively leverage Seldon Core 2 to meet your specific needs.

Helm Charts

For more information, see the published .

For the description of (some) values that can be configured for these charts, see this .

Seldon Core 2 Dependencies

Here is a list of components that Seldon Core 2 requires, along with the minimum and maximum supported versions:

Notes:

• Envoy and Rclone: These components are included as part of the Seldon Core 2 Docker images. You are not required to install them separately but must be aware of the configuration options supported by these versions.

• Kafka: Only required for operating Seldon Core 2 dataflow Pipelines. If not needed, you should avoid installing seldon-modelgateway, seldon-pipelinegateway, and seldon-dataflow-engine.

• Maximum Versions

Get started

For Kubernetes usage we provide a set of custom resources for interacting with Seldon.

• - for installing Seldon in a particular namespace.

• - for deploying sets of replicas of core inference servers (MLServer or Triton).

• - for deploying single machine learning models, custom transformation logic, drift detectors, outliers detectors and explainers.

Seldon Core provides native autoscaling features for both Models and Servers, enabling automatic scaling based on inference load. The diagram below depicts an autoscaling implementation that uses both Model and Server autoscaling features native to Seldon Core (i.e. this implementation doesn't leverage HPA for autoscaling, an approach we cover )

Model Autoscaling

Models can automatically scale their replicas based on load. Enable it by setting MinReplicas or MaxReplicas in your model spec. For more detail on this setup see

Seldon Core 2 runs with several control and dataplane components. The scaling of these resources is discussed below:

• Pipeline gateway: The pipeline gateway handles REST and gRPC synchronous requests to Pipelines. It is stateless and can be scaled based on traffic demand.

• Model gateway: This component pulls model requests from Kafka and sends them to inference servers. It can be scaled up to the partition factor of your Kafka topics. At present we set a uniform partition factor for all topics in one installation of Seldon.

• Dataflow engine: The dataflow engine runs KStream topologies to manage Pipelines. It can run as multiple replicas and the scheduler will balance Pipelines to run across it with a consistent hashing load balancer. Each Pipeline is managed up to the partition factor of Kafka (presently hardwired to one). We recommend using as many replicas of dataflow-engine as you have Kafka partitions in order to leverage the balanced distribution of inference traffic using hashing

• Scheduler: The scheduler manages the control plane operations. It is presently required to be one replica as it maintains internal state within a BadgerDB held on local persistent storage (stateful set in Kubernetes). Performance tests have shown this not to be a bottleneck at present.

• Kubernetes Controller: The Kubernetes controller manages resources updates on the cluster which it passes on to the Scheduler. It is by default one replica but has the ability to scale.

• Envoy: Envoy replicas get their state from the scheduler for routing information and can be scaled as needed.

You can install Seldon Core 2 on your local computer that is running a Kubernetes cluster using kind.

Seldon publishes the Helm charts that are required to install Seldon Core 2. For more information about the Helm charts and the related dependencies,see Helm charts and Dependencies.

Prerequisites

• Install a Kubernetes cluster that is running version 1.27 or later.

• Install , the Kubernetes command-line tool.

• Install , the package manager for Kubernetes or , the automation tool used for provisioning, configuration management, and application deployment.

Installing Seldon Core 2

Next Steps

If you installed Seldon Core 2 using Helm, you need to complete the installation of other components in the following order:

Multi-model Serving

Multi-model serving is an architecture pattern where one ML inference server hosts multiple models at the same time. This means that, within a single instance of the server, you can serve multiple models under different paths. This is a feature provided out of the box by Nvidia Triton and Seldon MLServer, currently the two inference servers that are integrated in Seldon Core 2.

This deployment pattern allows the system to handle a large number of deployed models letting them share hardware resources allocated to inference servers (e.g GPUs). For example if a single model inference server is deployed on a one GPU node, the underlying loaded models on this inference server instance are able to effectively share this GPU. This is contrast to a single model per server deployment pattern where only one model can use the allocated GPU.

Models provide the atomic building blocks of Seldon. They represents machine learning models, drift detectors, outlier detectors, explainers, feature transformations, and more complex routing models such as multi-armed bandits.

• Seldon can handle a wide range of

• Artifacts can be stored on any of the 40 or more cloud storage technologies as well as from local (mounted) folder as discussed .

The Model specification allows parameters to be passed to the loaded model to allow customization. For example:

This capability is only available for MLServer custom model runtimes. The named keys and values will be added to the model-settings.json file for the provided model in theparameters.extra Dict. MLServer models are able to read these values in their load method.

Example Parameterized Models

Seldon Core 2 provides robust tools for tracking the performance and health of machine learning models in production.

Monitoring

• Real-Time metrics: collects and displays real-time metrics from deployed models, such as response times, error rates, and resource usage.

In MLOps, system performance can be defined as how efficiently and effectively an ML model or application operates in a production environment. It is typically measured across several key dimensions, such as latency, throughput, scalability, and resource-efficiency. These factors are deeply connected: changes in configuration often result in tradeoffs, and should be carefully considered. Specifically, latency, throughput and resource usage can all impact each other, and the approach to optimizing system performance depends on the desired balance of these outcomes in order to ensure a positive end-user experience, while also minimising infrastructure costs.

High Level Approach

There are many different levers that can be considered to tune performance for ML systems deployed with Core 2, across infrastructure, models, inference execution, and the related configurations exposed by Core 2. When reasoning about the performance of an ML-based system deployed using Seldon, we recommend breaking down the problem by first understanding and tuning the performance of deployed Models, and then subsequently considering more complex Pipelines composed of those models (if applicable). For both models and pipelines, it is important to run tests to understand baseline performance characteristics, before making efforts to tune these through changes to models, infrastructure, or inference configurations. The recommended approach can be broken down as follows:

Given Seldon Core 2 is predominantly for serving ML in Kubernetes, it is possible to leverage HorizontalPodAutoscaler or to define scaling logic that automatically scales up and down Kubernetes resources. HPA targets Kubernetes or custom metrics to trigger scale-up or scale-down events for specified resources. Using HPA is recommnended if custom Scaling Metrics are required. These would be exposed using Prometheus, and or similar tools for explosing metrics to HPA. If these tools cause conflicts, that does not require exposing custom metrics is recommended.

Seldon Core 2 provides two main approaches to leveraging Kubernetes Horizontal Pod Autoscaler (HPA) for autoscaling. It is important to remember that since in Core 2 Models and Servers are separate, autoscaling of both Models and Servers, in a coordinated way, needs to be accounted for when implementing autoscaling. In order to implement either approach, metrics first need to be exposed - this is explained in the guide which explains the fundamental requirements and configuration needed to enable HPA-based scaling in Seldon Core 2.

This model allows a Pandas query to be run in the input to select rows. An example is shown below:

# samples/models/choice1.yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: choice-is-one
spec:
  storageUri: "gs://seldon-models/scv2/examples/pandasquery"
  requirements:
  - mlserver
  - python
  parameters:
  - name: query
    value: "choice == 1"

This invocation check filters for tensor A having value 1.

• The model also returns a tensor called status which indicates the operation run and whether it was a success. If no rows satisfy the query then just a status tensor output will be returned.

• For further details, see Pandas query

This model can be useful for conditional Pipelines. For example, you could have two invocations of this model:

and

By including these in a Pipeline as follows we can define conditional routes:

Here the mul10 model will be called if the choice-is-one model succeeds and the add10 model will be called if the choice-is-two model succeeds.

For more details, see the Pandas query .

Core 2 architecture is built around decoupling Model and Server CRs to allow for multi-model deployment, enabling multiple models to be loaded and served on one server replica or a single Pod. Multi-model serving allows for more efficient use of resources, see Multi Model Serving for more information.

This architecture requires that Core 2 handles scheduling of models onto server pods natively. In particular Core 2 implements different sorters and filters which are used to find the best Server that is able to host a given Model. This process we describe in the following section.

Scheduling Process

Overview

The scheduling process in Core 2 identifies a suitable candidate server for a given model through a series of steps. These steps involve sorting and filtering servers primarily based on the following criteria:

• Server has matching Capabilities with Model spec.requirements.

• Server has enough replicas to load the desired spec.replicas of the Model.

• Each replica of Server has enough available memory to load one replica of the model defined in spec.memory.

After a suitable candidate server is identified for a given model, Core 2 attempts to load the model onto it. If no matching server is found, the model is marked as ScheduleFailed.

This process is designed to be extensible, allowing for the addition of new filters in future versions to enhance scheduling decisions.

Partial Scheduling

Core 2 (from 2.9) is able to do partial scheduling of Models. Partial scheduling is defined as the loading of enough replicas of the model above spec.minReplicas and upto the number of available Server replicas. This allows the user a little bit more flexibility in serving traffic while optimising infrastructure provisioning.

To enable partial scheduling, spec.minReplicas needs to be defined as it provides Core 2 the minimum replicas of the model that is required for serving.

Partial scheduling does not have an explicit state; instead, the model is marked as ModelAvailable and is ready to serve traffic. The status of the Model CR can be inspected, where DESIRED REPLICAS and AVAILABLE REPLICAS provide insight into the number of replicas currently loaded in Core 2. Based on this information, the following logic are applied:

• Fully Scheduled: READY is True and DESIRED REPLICAS is equal to AVAILABLE REPLICAS (STATUS is ModelAvailable)

• Partially Scheduled: READY is TRUE and DESIRED REPLICAS is greater than AVAILABLE REPLICAS (STATUS is ModelAvailable)

• Not Scheduled: Ready is False (Status is ScheduleFailed)

Model Kafka topics

A in Seldon Core 2 represents the fundamental unit for serving a machine learning artifact within a running server instance.

If Kafka is installed in your cluster, Seldon Core automatically creates dedicated input and output topics for each model as it is loaded. These topics facilitate asynchronous messaging, enabling clients to send input messages and retrieve output responses independently and at a later time.

By default, when a model is unloaded, the associated Kafka topics are preserved. This supports use cases like auditing, but can also lead to increased Kafka resource usage and unnecessary costs for workloads that don't require persistent topics.

You can control this behavior by configuring the

Prerequisites

• Set up and connect to a Kubernetes cluster running version 1.27 or later. For instructions on connecting to your Kubernetes cluster, refer to the documentation provided by your cloud provider.

• Install

Workflows

Seldon inference is built from atomic Model components. Models as cover a wide range of artifacts including:

• Core machine learning models, e.g. a PyTorch model.

Before looking to make changes to improve latency or throughput of your models, it is important to undergo load testing to understand the existing performance characteristics of your model(s) when deployed onto the chosen inference server (MLServer or Triton). The goal of load testing should be to understand the performance behavior of deployed models in saturated (i.e at the maximum throughput a model replica can handle) and non-saturated regimes, then compare it with expected latency objectives.

The results here will also inform the setup of autoscaling parameters, with the target being to run each replica with some margin below the saturation throughput (say, by 10-20%) in order to ensure that latency does not degrade, and that there is sufficient capacity to absorb load for the time it takes for new inference server replicas (and model replicas) to become available.

An Experiment defines a traffic split between Models or Pipelines. This allows new versions of models and pipelines to be tested.

An experiment spec has three sections:

• candidates (required) : a set of candidate models to split traffic.

• default (optional) : an existing candidate who endpoint should be modified to split traffic as defined by the candidates.

Autoscaling in Seldon Core 2

Seldon Core 2 provides multiple approaches to scaling your machine learning deployments, allowing you to optimize resource utilization and handle varying workloads efficiently. In Core 2, we separate out Models and Servers, and Servers can have multiple Models loaded on them (Multi-Model Serving). Given this, setting up autoscaling requires defining the logic by which you want to scale your Models and then configuring the autoscaling of Servers such that they autoscale in a coordinated way. The following steps can be followed to set up autoscaling based on specific requirements:

1. Identify metrics that you want to scale Models on. There are a couple of different options here:

Explainers are Model resources with some extra settings. They allow a range of explainers from the Alibi-Explain library to be run on MLServer.

An example Anchors explainer definitions is shown below.

The key additions are:

• type: This must be one of the supported by the Alibi Explain runtime in MLServer.

• modelRef

Upgrading from 2.9 - 2.10

All CRD changes maintain backward compatibility with existing CRs. We introduce new Core 2 scaling configuration options in SeldonConfig (config.ScalingConfig.*), with a wider goal of centralising Core 2 configuration and allowing for configuration changes after the Core 2 cluster is deployed. To ensure a smooth transition, some of the configuration options will only take effect starting from the next releases, but end-users are encouraged to set them to the desired values before upgrading to the next release (2.11).

Upgrading when using helm is seamless, with existing helm values being used to fill in new configuration options. If not using helm, previous SeldonConfig CRs remain valid, but restrictive defaults will be used for the scaling configuration. One parameter in particular, maxShardCountMultiplier docs will need to be set in order to take advantage of the new pipeline scalability features. This parameter can be changed and the effects of its value will be propagated to all components that use the config.

For full release notes, see .

Upgrading from 2.8 - 2.9

Though there are no breaking changes between 2.8 and 2.9, there are some new functionalties offered that require changes to fields in our CRDs:

• In Core 2.9 you can now set minReplicas to enable of Models. This means that users will no longer have to wait for the full set of desired replicas before loading models onto servers (e.g. when scaling up).

• We've also added a spec.llm field to the Model CRD . The field is used by the PromptRuntime in Seldon's to reference a LLM model. Only one of spec.llm and spec.explainer should be set at a given time. This allows the deployment of multiple "models" acting as prompt generators for the same LLM.

• Due to the introduction of Server-autoscaling, it is important to understand what type of autoscaling you want to leverage, and how that can be configured. Below are configuratation that help set autoscaling behaviour. All options here have corresponding command-line arguments that can be passed to seldon-scheduler when not using helm as the install method. The following helm values can be set

Upgrading from 2.7 - 2.8

Core 2.8 introduces several new fields in our CRDs:

• statefulSetPersistentVolumeClaimRetentionPolicy enables users to configure the cleaning of PVC on their servers. This field is set to retain as default.

• Status.selector was introduced as a mandatory field for models in 2.8.4 and made optional in 2.8.5. This field enables autoscaling with HPA.

• PodSpec in the OverrideSpec

These added fields do not result in breaking changes, apart from 2.8.4 which required the setting of the Status.selector upon upgrading. This field was however changed to optional in the subsequent 2.8.5 release. Updating the CRDs (e.g. via helm) will enable users to benefit from the associated functionality.

Upgrading from 2.6 - 2.7

All pods provisioned through the operator i.e. SeldonRuntime and Server resources now have the label app.kubernetes.io/name for identifying the pods.

Previously, the labelling has been inconsistent across different versions of Seldon Core 2, with mixture of app and app.kubernetes.io/name used.

If using the Prometheus operator ("Kube Prometheus"), please apply the v2.7.0 manifests for Seldon Core 2 according to the .

Note that these manifests need to be adjusted to discover metrics endpoints based on the existing setup.

If previous pod monitors had namespaceSelector fields set, these should be copied over and applied to the new manifests.

If namespaces do not matter, cluster-wide metrics endpoint discovery can be setup by modifying thenamespaceSelector field in the pod monitors:

Upgrading from 2.5 - 2.6

Release 2.6 brings with it new custom resources SeldonConfig and SeldonRuntime, which provide a new way to install Seldon Core 2 in Kubernetes. Upgrading in the same namespace will cause downtime while the pods are being recreated. Alternatively users can have an external service mesh or other means to be used over multiple namespaces to bring up the system in a new namespace and redeploy models before switch traffic between them.

If the new 2.6 charts are used to upgrade in an existing namespace models will eventually be redeloyed but there will be service downtime as the core components are redeployed.

To confirm the successful installation of Seldon Core 2, Kafka, and the service mesh, deploy a sample model and perform an inference test. Follow these steps:

Deploy the Iris Model

1. Apply the following configuration to deploy the Iris model in the namespace seldon-mesh:

The output is:

1. Verify that the model is deployed in the namespace seldon-mesh.

When the model is deployed, the output is similar to:

Deploy a pipeline for the Iris Model

1. Apply the following configuration to deploy the Iris model in the namespace seldon-mesh:

The output is:

1. Verify that the pipeline is deployed in the namespace seldon-mesh.

When the pipeline is deployed, the output is similar to:

Perform an Inference test

1. Use curl to send a test inference request to the deployed model. Replace <INGRESS_IP> with your service mesh's ingress IP address. Ensure that:

• The Host header matches the expected virtual host configured in your service mesh.

• The Seldon-Model header specifies the correct model name.

The output is similar to:

1. Use curl to send a test inference request through the pipeline to the deployed model. Replace <INGRESS_IP> with your service mesh's ingress IP address. Ensure that:

• The Host header matches the expected virtual host configured in your service mesh.

• The Seldon-Model header specifies the correct pipeline name.

• To route inference requests to a pipeline endpoint, include the .pipeline suffix in the model name within the request header. This distinguishes the pipeline from a model that shares the same base name.

The output is similar to:

Seldon V2 Kubernetes Multi Version Artifact Examples

By default Seldon installs two server farms using MLServer and Triton with 1 replica each. Models are scheduled onto servers based on the server's resources and whether the capabilities of the server matches the requirements specified in the Model request. For example:

This model specifies the requirement sklearn

There is a default capabilities for each server as follows:

• MLServer

• Triton

This page describes how to implement autoscaling for both Models and Servers using Kubernetes HPA (Horizontal Pod Autoscaler) in a single-model serving setup. This approach is specifically designed for deployments where each Server hosts exactly one Model replica (1:1 mapping between Models and Servers).

Overview

In single-model serving deployments, you can use HPA to scale both Models and their associated Servers independently, while ensuring they scale in a coordinated manner. This is achieved by:

1. Setting up HPA for Models based on custom metrics (e.g., RPS)

2. Setting up matching HPA configurations for Servers

3. Ensuring both HPAs target the same metrics and scaling policies

Key Considerations

• Only custom metrics from Prometheus are supported. Native Kubernetes resource metrics such as CPU or memory are not. This limitation exists because of HPA's design: In order to prevent multiple HPA CRs from issuing conflicting scaling instructions, each HPA CR must exclusively control a set of pods which is disjoint from the pods controlled by other HPA CRs. In Seldon Core 2, CPU/memory metrics can be used to scale the number of Server replicas via HPA. However, this also means that the CPU/memory metrics from the same set of pods can no longer be used to scale the number of model replicas.

• No Multi-Model Serving - this approach requires a 1-1 mapping of Models and Servers, meaning that consolidation of multiple Models onto shared Servers (Multi-Model Serving) is not possible.

Implementation

In order to set up HPA Autoscaling for Models and Servers together, metrics need to be exposed in the same way as is explained in the tutorial. If metrics have not yet been exposed, follow that workflow until you are at the point where you will configure and apply the HPA manifests.

In this implementation, Models and Servers will be configured to autoscale with a separate HPA manifest targetting the same metric as Models. In order for the scaling metrics to be in sync, it's important to apply the HPA manifests simultaneously. It is important to keep both the scaling metric and any scaling policies the same across the two HPA manifests. This is to ensure that both the Models and the Servers are scaled up/down at approximately the same time. Small variations in the scale-up time are expected because each HPA samples the metrics independently, at regular intervals.

Here's an example configuration, utilizing the same infer_rps metric as was set up in the .

Scaling Behavior

In order to ensure similar scaling behaviour between Models and Servers, the number of minReplicas and maxReplicas, as well as any other configured scaling policies should be kept in sync across the HPA for the model and the server.

Each HPA CR has it's own timer on which it samples the specified custom metrics. This timer starts when the CR is created, with sampling of the metric being done at regular intervals (by default, 15 seconds). As a side effect of this, creating the Model HPA and the Server HPA (for a given model) at different times will mean that the scaling decisions on the two are taken at different times. Even when creating the two CRs together as part of the same manifest, there will usually be a small delay between the point where the Model and Server spec.replicas values are changed. Despite this delay, the two will converge to the same number when the decisions are taken based on the same metric (as in the previous examples).

Monitoring Scaling

When showing the HPA CR information via kubectl get, a column of the output will display the current metric value per replica and the target average value in the format [per replica metric value]/[target]. This information is updated in accordance to the sampling rate of each HPA resource. It is therefore expected to sometimes see different metric values for the Model and its corresponding Server.

Limitations

In this implementation, the scheduler itself does not create new Server replicas when the existing replicas are not sufficient for loading a Model's replicas (one Model replica per Server replica). Whenever a Model requests more replicas than available on any of the available Servers, its ModelReady condition transitions to Status: False with a ScheduleFailed message. However, any replicas of that Model that are already loaded at that point remain available for servicing inference load.

Best Practices

The following elements are important to take into account when setting the HPA policies.

• The speed with which new Server replicas can become available versus how many new replicas may HPA request in a given time:

  • The HPA scale-up policy should not be configured to request more replicas than can become available in the specified time. The following example reflects a confidence that 5 Server pods will become available within 90 seconds, with some safety margin. The default scale-up config, that also adds a percentage based policy (double the existing replicas within the set periodSeconds) is not recommended because of this.

  • Perhaps more importantly, there is no reason to scale faster than the time it takes for replicas to become available - this is the true maximum rate with which scaling up can happen anyway.

• The duration of transient load spikes which you might want to absorb within the existing per-replica RPS margins.

  • The previous example, at line 13, configures a scale-up stabilization window of one minute. It means that for all of the HPA recommended replicas in the last 60 second window (4 samples of the custom metric considering the default sampling rate), only the smallest will be applied.

  • Such stabilization windows should be set depending on typical load patterns in your cluster: not being too aggressive in reacting to increased load will allow you to achieve cost savings, but has the disadvantage of a delayed reaction if the load spike turns out to be sustained.

Text Generation Model

Load the model

kubectl apply -f ./models/hf-text-gen.yaml

model.mlops.seldon.io/text-gen created

seldon model load -f ./models/hf-text-gen.yaml

{}

Wait for the model to be ready

kubectl get model text-gen -n ${NAMESPACE} -o json | jq -r '.status.conditions[] | select(.message == "ModelAvailable") | .status'

True

seldon model status text-gen -w ModelAvailable | jq -M .

{}

Do a REST inference call

Unload the model

Custom Text Generation Model

Load the model

Unload the model

Custom Server with Capabilities

Outlier detection models are treated as any other Model. You can run any saved outlier detection model by adding the requirement alibi-detect.

An example outlier detection model from the CIFAR10 image classification example is shown below:

Examples

Inference artifacts referenced by Models can be stored in any of the storage backends supported by Rclone. This includes local filesystems, AWS S3, and Google Cloud Storage (GCS), among others. Configuration is provided out-of-the-box for public GCS buckets, which enables the use of Seldon-provided models like in the below example:

This configuration is provided by the Kubernetes Secret seldon-rclone-gs-public. It is made available to Servers as a preloaded secret. You can define and use your own storage configurations in exactly the same way.

Configuration Format

To define a new storage configuration, you need the following details:

• Remote name

• Remote type

• Provider parameters

A remote is what Rclone calls a storage location. The type defines what protocol Rclone should use to talk to this remote. A provider is a particular implementation for that storage type. Some storage types have multiple providers, such as s3 having AWS S3 itself, MinIO, Ceph, and so on.

The remote name is your choice. The prefix you use for models in spec.storageUri must be the same as this remote name.

The remote type is one of the values . For example, for AWS S3 it is s3 and for Dropbox it is dropbox.

The provider parameters depend entirely on the remote type and the specific provider you are using. Please check the Rclone documentation for the appropriate provider. Note that Rclone docs for storage types call the parameters properties and provide both config and env var formats--you need to use the config format. For example, the GCS parameter --gcs-client-id described should be used as client_id.

For reference, this format is described in the . Note that we do not support the use of opts discussed in that section.

Kubernetes Secrets

Kubernetes Secrets are used to store Rclone configurations, or storage secrets, for use by Servers. Each Secret should contain exactly one Rclone configuration.

A Server can use storage secrets in one of two ways:

• It can dynamically load a secret specified by a Model in its .spec.secretName

• It can use global configurations made available via

The name of a Secret is entirely your choice, as is the name of the data key in that Secret. All that matters is that there is a single data key and that its value is in the format described above.

Preloaded Secrets

Rather than Models always having to specify which secret to use, a Server can load storage secrets ahead of time. These can then be reused across many Models.

When using a preloaded secret, the Model definition should leave .spec.secretName empty. The protocol prefix in .spec.storageUri still needs to match the remote name specified by a storage secret.

The secrets to preload are named in a centralised ConfigMap called seldon-agent. This ConfigMap applies to all Servers managed by the same SeldonRuntime. By default this ConfigMap only includes seldon-rclone-gs-public, but can be extended with your own secrets as shown below:

The easiest way to change this is to update your SeldonRuntime.

• If your SeldonRuntime is configured using the seldon-core-v2-runtime Helm chart, the corresponding value is config.agentConfig.rclone.configSecrets. This can be used as shown below:

• Otherwise, if your SeldonRuntime is configured directly, you can add secrets by setting .spec.config.agentConfig.rclone.config_secrets. This can be used as follows:

Examples

We will show:

• Model inference to a Tensorflow model

  • REST and gRPC using seldon CLI, curl and grpcurl

• Pipeline inference

  • REST and gRPC using seldon CLI, curl and grpcurl

Tensorflow Model

Load the model.

Wait for the model to be ready.

Kind cluster setup

To run this example in Kind we need to start Kind with access to a local folder where are models are location. In this example it is a folder in /tmp and associate that with a path in the container.

To start a Kind cluster see, .

Create the local folder for models and copy an example iris sklearn model to it.

Create Server with PVC

Create a storage class and associated persistent colume referencing the /models folder where models are stored.

Now create a new Server based on the provided MLServer configuration but extend it with our PVC by adding this to the rclone container which will allow rclone to move models from this PVC onto the server.

We also add a new capability pvc to allow us to schedule models to this server that has the PVC.

SKLearn Model

Use a simple sklearn iris classification model with the added pvc requirement so that MLServer with the PVC is targeted during scheduling.

Do a gRPC inference call

Kafka is a component in the Seldon Core 2 ecosystem, that provides scalable, reliable, and flexible communication for machine learning deployments. It serves as a strong backbone for building complex inference pipelines, managing high-throughput asynchronous predictions, and seamlessly integrating with event-driven systems—key features needed for contemporary enterprise-grade ML platforms.

An inference request is a request sent to a machine learning model to make a prediction or inference based on input data. It is a core concept in deploying machine learning models in production, where models serve predictions to users or systems in real-time or batch mode.

To explore this feature of Seldon Core 2, you need to integrate with Kafka. Integrate Kafka through managed cloud services or by deploying it directly within a Kubernetes cluster.

• provides more information about the encrytion and authentication.

• provides the steps to configure some of the managed Kafka services.

Self-hosted Kafka

Seldon Core 2 requires Kafka to implement data-centric inference Pipelines. To install Kafka for testing purposed in your Kubernetes cluster, use . For more information, see

When deploying machine learning models in Kubernetes, you may need to control which infrastructure resources these models use. This is especially important in environments where certain workloads, such as resource-intensive models, should be isolated from others or where specific hardware such as GPUs, needs to be dedicated to particular tasks. Without fine-grained control over workload placement, models might end up running on suboptimal nodes, leading to inefficiencies or resource contention.

For example, you may want to:

• Isolate inference workloads from control plane components or other services to prevent resource contention.

• Ensure that GPU nodes are reserved exclusively for models that require hardware acceleration.

• Keep business-critical models on dedicated nodes to ensure performance and reliability.

• Run external dependencies like Kafka on separate nodes to avoid interference with inference workloads.

To solve these problems, Kubernetes provides mechanisms such as taints, tolerations, and nodeAffinity or nodeSelector to control resource allocation and workload scheduling.

are applied to nodes and to Pods to control which Pods can be scheduled on specific nodes within the Kubernetes cluster. Pods without a matching toleration for a node’s taint are not scheduled on that node. For instance, if a node has GPUs or other specialized hardware, you can prevent Pods that don’t need these resources from running on that node to avoid unnecessary resource usage.

When used together, taints and tolerations with nodeAffinity or nodeSelector can effectively allocate certain Pods to specific nodes, while preventing other Pods from being scheduled on those nodes.

In a Kubernetes cluster running Seldon Core 2, this involves two key configurations:

1. Configuring servers to run on specific nodes using mechanisms like taints, tolerations, and nodeAffinity or nodeSelector.

2. Configuring models so that they are scheduled and loaded on the appropriate servers.

This ensures that models are deployed on the optimal infrastructure and servers that meet their requirements.

Server configurations define how to create an inference server. By default one is provided for Seldon MLServer and one for NVIDIA Triton Inference Server. Both these servers support the Open Inference Protocol which is a requirement for all inference servers. They define how the Kubernetes ReplicaSet is defined which includes the Seldon Agent reverse proxy as well as an Rclone server for downloading artifacts for the server. The Kustomize ServerConfig for MlServer is shown below:

Local example settings.

Remote k8s cluster example settings - change as neeed for your needs.

Model Chain - Ready Check

We will check the readiness of the Pipeline after every change to model and pipeline.

This notebook will show how we can update running experiments.

Test change candidate for a model

We will use three SKlearn Iris classification models to illustrate experiment updates.

Load all models.

Let's call all three models individually first.

We will start an experiment to change the iris endpoint to split traffic with the iris2 model.

The SeldonConfig resource defines the core installation components installed by Seldon. If you wish to install Seldon, you can use the resource which allows easy overriding of some parts defined in this specification. In general, we advise core DevOps to use the default SeldonConfig or customize it for their usage. Individual installation of Seldon can then use the SeldonRuntime with a few overrides for special customisation needed in that namespace.

The specification contains core PodSpecs for each core component and a section for general configuration including the ConfigMaps that are created for the Agent (rclone defaults), Kafka and Tracing (open telemetry).

Some of these values can be overridden on a per namespace basis via the SeldonRuntime resource. Labels and annotations can also be set at the component level - these will be merged with the labels and annotations from the SeldonConfig resource in which they are defined and added to the component's corresponding Deployment, or StatefulSet.

Drift detection models are treated as any other Model. You can run any saved drift detection model by adding the requirement alibi-detect.

An example drift detection model from the CIFAR10 image classification example is shown below:

Usually you would run these models in an asynchronous part of a Pipeline, i.e. they are not connected to the output of the Pipeline which defines the synchronous path. For example, the CIFAR-10 image detection example uses a pipeline as shown below:

Note how the cifar10-drift model is not part of the path to the outputs. Drift alerts can be read from the Kafka topic of the model.

When tuning performance for pipelines, reducing the overhead of Core 2 components responsible for data-processing within a pipeline is another aspect to consider. In Core 2, four components influence this overhead:

• pipelinegateway which handles pipeline requests

• modelgateway which sends requests to model inference servers