Prepackaged Model Servers

Seldon provides several prepacked servers you can use to deploy trained models:

• SKLearn Server

• XGBoost Server

• Tensorflow Serving

• MLflow Server

• Custom Servers

• Alibi Explainers

For these servers you only need the location of the saved model in a local filestore, Google bucket, S3 bucket, azure or minio. An example manifest with an sklearn server is shown below:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn-iris
spec:
  predictors:
    - name: default
      replicas: 1
      graph:
        name: classifier
        implementation: SKLEARN_SERVER
        modelUri: gs://seldon-models/v1.19.0-dev/sklearn/iris

By default only public models published to Google Cloud Storage will be accessible. See below notes on how to configure credentials for AWS S3, Minio and other storage solutions.

Init Containers

Seldon Core uses Init Containers to download model binaries for the prepackaged model servers. We use rclone-based storage initailizer for our Init Containers by defining

storageInitializer:
  image: seldonio/rclone-storage-initializer:1.19.0-dev

in our default helm values. See the Dockerfile for a detailed reference. You can overwrite this value to specify another default initContainer. See details on requirements bellow

Secrets are injected into the init containers as environmental variables from kubernetes secrets. The default secret name can be defined by setting following helm value

predictiveUnit:
  defaultEnvSecretRefName: ""

Customizing Init Containers

You can specify a custom initContainer image and default secret globally by overwriting the helm values specified in the previous section.

To illustrate how initContainers are used by the prepackaged model servers, consider a following Seldon Deployment with volumes, volumeMounts and initContainers equivalent to ones that would be injected by the Seldon Core Operator if this was prepackaged model server:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: sklearn-iris
spec:
  predictors:
  - name: default
    replicas: 1
    graph:
      name: classifier
      type: MODEL

    componentSpecs:
    - spec:
        volumes:
        - name: classifier-provision-location
          emptyDir: {}

        initContainers:
        - name: classifier-model-initializer
          image: seldonio/rclone-storage-initializer:1.19.0-dev
          imagePullPolicy: IfNotPresent
          args:
            - "s3://sklearn/iris"
            - "/mnt/models"

          volumeMounts:
          - mountPath: /mnt/models
            name: classifier-provision-location

          envFrom:
          - secretRef:
              name: seldon-init-container-secret

        containers:
        - name: classifier
          image: seldonio/sklearnserver:1.8.0-dev

          volumeMounts:
          - mountPath: /mnt/models
            name: classifier-provision-location
            readOnly: true

          env:
          - name: PREDICTIVE_UNIT_PARAMETERS
            value: '[{"name":"model_uri","value":"/mnt/models","type":"STRING"}]'

Key observations:

• Our prepackaged model will expect model binaries to be saved into /mnt/models path

• Default initContainers name is constructed from {predictiveUnitName}-model-initializer

• The entrypoint of the container must take two arguments:

  • First representing the models URI

  • Second the desired path where binary should be downloaded to

• If user would to provide their own initContainer which name matches the above pattern it would be used as provided

This is equivalent to the following sklearn-iris Seldon Deployment. As we can see using prepackaged model servers allow one to avoid defining boilerplate and make definition much cleaner:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: iris-sklearn
spec:
  predictors:
  - name: default
    replicas: 1
    graph:
      name: classifier
      implementation: SKLEARN_SERVER
      modelUri: s3://sklearn/iris
      storageInitializerImage: seldonio/rclone-storage-initializer:1.19.0-dev  # Specify custom image here
      envSecretRefName: seldon-init-container-secret                          # Specify custom secret here

Note that image and secret used by Storage Initializer can be customised per-deployment.

See our example that explains in details how init containers are used and how to write a custom one using rclone for cloud storage operations as an example.

Further Customisation for Prepackaged Model Servers

If you want to customize the resources for the server you can add a skeleton Container with the same name to your podSpecs, e.g.

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: sklearn-iris
spec:
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - name: classifier
          resources:
            requests:
              memory: 50Mi
    name: default
    replicas: 1
    graph:
      name: classifier
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/v1.19.0-dev/sklearn/iris

The image name and other details will be added when this is deployed automatically.

Next steps:

• Worked notebook

• SKLearn Server

• XGBoost Server

• Tensorflow Serving

• MLflow Server

• SKLearn Server with MinIO

You can also build and add your own custom inference servers, which can then be used in a similar way as the prepackaged ones.

If your use case does not fit for a reusable standard server then you can create your own component using our wrappers.

Handling Credentials

General notes

Rclone remotes can be configured using the environmental variables:

RCLONE_CONFIG_<remote name>_<config variable>: <config value>

Note: multiple remotes can be configured simultaneously.

Once the remote is configured the modelUri that is compatible with rclone takes form

modelUri: <remote>:<bucket name>

for example modelUri: s3:sklearn/iris.

Note: Rclone will remove the leading slashes for buckets so this is equivalent to s3://sklearn/iris.

Below you will find a few example configurations. For other cloud solutions, please, consult great documentation of the rclone project.

Example for public GCS configuration

Note: this is configured by default in the seldonio/rclone-storage-initializer image.

Reference: rclone documentation.

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_GS_TYPE: google cloud storage
  RCLONE_CONFIG_GS_ANONYMOUS: "true"

Example deployment

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: rclone-sklearn-gs
spec:
  predictors:
  - name: default
    replicas: 1
    graph:
      name: classifier
      implementation: SKLEARN_SERVER
      modelUri: gs:seldon-models/sklearn/iris
      envSecretRefName: seldon-rclone-secret

Example minio configuration

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: minio
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: minioadmin
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: minioadmin
  RCLONE_CONFIG_S3_ENDPOINT: http://minio.minio-system.svc.cluster.local:9000

Example AWS S3 with access key and secret

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "<your AWS_ACCESS_KEY_ID here>"
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "<your AWS_SECRET_ACCESS_KEY here>"

Example AWS S3 with IAM roles configuration

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: ""
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ""
  RCLONE_CONFIG_S3_ENV_AUTH: "true"

Example for GCP/GKE

Reference: rclone documentation

For GCP/GKE, you will need create a service-account key and have it as local json file. First make sure that you have [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com service account created in the gcloud console that have sufficient permissions to access the bucket with your models (i.e. Storage Object Admin).

Now, generate keys locally using the gcloud tool

gcloud iam service-accounts keys create gcloud-application-credentials.json --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com

Now using the content of locally saved gcloud-application-credentials.json file create a secret

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_GCS_TYPE: google cloud storage
  RCLONE_CONFIG_GCS_ANONYMOUS: "false"
  RCLONE_CONFIG_GCS_SERVICE_ACCOUNT_CREDENTIALS: '{"type":"service_account", ... <rest of gcloud-application-credentials.json>}'

Note: remote name is gcs here so urls would take form similar to gcs:<your bucket>. Tip: using cat gcloud-application-credentials.json | jq -c . can help to easily collapse credentials.json into one line.

Example Azure Blob with Account Key

Reference: rclone documentation

apiVersion: v1
kind: Secret
metadata:
  name: seldon-rclone-secret
type: Opaque
stringData:
  RCLONE_CONFIG_AZUREBLOB_TYPE: azureblob
  RCLONE_CONFIG_AZUREBLOB_ACCOUNT: ""
  RCLONE_CONFIG_AZUREBLOB_ENV_AUTH: "false"
  RCLONE_CONFIG_AZUREBLOB_KEY: ""

Note: remote name is azureblob here so urls would take form similar to azureblob:<container>/path/to/dir.

Directly from PVC

You are able to make models available directly from PVCs instead of object stores. This may be desirable if you have a lot of very large files and you want to avoid uploading/downloading, for example through NFS drives.

The way in which you are able to specify the PVC is using the modelUri with the following format below. One thing to take into consideration is the permissions in the files as the containers will have their respective runAsUser parameters.

...
    modelUri: pvc://<pvc-name>/<path>