Servers

Learn how to configure and manage inference servers in Seldon Core 2, including MLServer and Triton server farms, model scheduling, and capability management.

By default Seldon installs two server farms using MLServer and Triton with 1 replica each. Models are scheduled onto servers based on the server's resources and whether the capabilities of the server matches the requirements specified in the Model request. For example:

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.5.0/iris-sklearn"
  requirements:
  - sklearn
  memory: 100Ki

This model specifies the requirement sklearn

There is a default capabilities for each server as follows:

• MLServer

  - name: SELDON_SERVER_CAPABILITIES
    value: "mlserver,alibi-detect,alibi-explain,huggingface,lightgbm,mlflow,python,sklearn,spark-mlib,xgboost"

• Triton

  - name: SELDON_SERVER_CAPABILITIES
    value: "triton,dali,fil,onnx,openvino,python,pytorch,tensorflow,tensorrt"

Custom Capabilities

Servers can be defined with a capabilities field to indicate custom configurations (e.g. Python dependencies). For instance:

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-134
spec:
  serverConfig: mlserver
  capabilities:
  - mlserver-1.3.4
  podSpec:
    containers:
    - image: seldonio/mlserver:1.3.4
      name: mlserver

These capabilities override the ones from the serverConfig: mlserver. A model that takes advantage of this is shown below:

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "gs://seldon-models/mlserver/iris"
  requirements:
  - mlserver-1.3.4

This above model will be matched with the previous custom server mlserver-134.

Servers can also be set up with the extraCapabilities that add to existing capabilities from the referenced ServerConfig. For instance:

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-extra
spec:
  serverConfig: mlserver
  extraCapabilities:
  - extra

This server, mlserver-extra, inherits a default set of capabilities via serverConfig: mlserver. These defaults are discussed above. The extraCapabilities are appended to these to create a single list of capabilities for this server.

Models can then specify requirements to select a server that satisfies those requirements as follows.

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: extra-model-requirements
spec:
  storageUri: "gs://seldon-models/mlserver/iris"
  requirements:
  - extra

The capabilities field takes precedence over the extraCapabilities field.

For some examples see here.

Autoscaling of Servers

Within docker we don't support this but for Kubernetes see here