Seldon Core 2

Seldon Core 2 provides a state of the art solution for machine learning inference.

Prerequisites

• Set up and connect to a Kubernetes cluster running version 1.23 or later. For instructions on connecting to your Kubernetes cluster, refer to the documentation provided by your cloud provider.

• Install kubectl, the Kubernetes command-line tool.

• Install Helm, the package manager for Kubernetes.

To use Seldon Core 2 in a production environment:

1. Create namespaces

2. Install Seldon Core 2

Creating Namespaces

• Create a namespace to contain the main components of Seldon. For example, create the namespace seldon-system:

  kubectl create ns seldon-system || echo "Namespace seldon-system already exists"

• Create a namespace to contain Kafka. For example, create the namespace kafka:

  kubectl create ns kafka || echo "Namespace kafka already exists"

• Create a namespace to contain the components related to request logging. For example, create the namespace seldon-logs:

  kubectl create ns seldon-logs || echo "Namespace seldon-logs already exists"

• Create a namespace that is accessible by Seldon Enterprise Platform, Seldon Core 2 runtime that defines core components required in each model, and Seldon Core 2 pre-configured servers to host the models. For example, create the namespace seldon:

  kubectl create ns seldon || echo "Namespace seldon already exists"

• Annotate the namespace seldon so that it is accessible in the Seldon Enterprise Platform UI:

  kubectl label ns seldon seldon.restricted=false --overwrite=true

Installing Seldon Core 2

1. Add and update the Helm charts seldon-charts to the repository.

   helm repo add seldon-charts https://seldonio.github.io/helm-charts/
   helm repo update seldon-charts

2. Install Custom resource definitions for Seldon Core 2.

   helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
   --version 2.8.5 \
   --namespace default \
   --install 

3. Create a YAML file to specify the initial configuration for Seldon Core 2 operator. For example, create the components-values.yaml file. Use your preferred text editor to create and save the file with the following content:

   controller:
     clusterwide: true
   
   dataflow:
     resources:
       cpu: 500m
   
   envoy:
     service:
       type: ClusterIP
   
   kafka:
     bootstrap: seldon-kafka-bootstrap.kafka:9092
     topics:
       numPartitions: 4
   
   opentelemetry:
     enable: false
   
   scheduler:
     service:
       type: ClusterIP
   
   serverConfig:
     mlserver:
       resources:
         cpu: 1
         memory: 2Gi
   
     triton:
       resources:
         cpu: 1
         memory: 2Gi
   
   serviceGRPCPrefix: "http2-"

   This configuration installs the Seldon Core 2 Operator to work across an entire Kubernetes cluster. If you wish to install the operator in a specific namespace instead, set clusterwide to false in the components-values.yaml file.

4. Change to the directory that contains the components-values.yaml file and then install Seldon Core 2 operator in the namespace seldon-system.

    helm upgrade seldon-core-v2-components seldon-charts/seldon-core-v2-setup \
    --version 2.8.5 \
    -f components-values.yaml \
    --namespace seldon-system \
    --install

5. Install Seldon Core 2 runtimes in the namespace seldon.

   helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
   --version 2.8.5 \
   --namespace seldon \
   --install

   One of the runtime components installed in this step is the Dataflow Engine. It requires a running Kafka instance to function properly. Ensure that you have a Kafka bootstrap server running at seldon-kafka-bootstrap.kafka:9092, as specified in the YAML file in step 3. If this is not the case, you can still proceed with the next step and resolve this issue in a later step, when you install Kafka.
6. Create a YAML file to specify the initial configuration for Seldon Core 2 servers. For example, create the servers-values.yaml file. Use your preferred text editor to create and save the file with the following content:

   mlserver:
    replicas: 1
   
   triton:
     replicas: 1

7. Change to the directory that contains the servers-values.yaml file and then install Seldon Core 2 servers in the namespace seldon.

    helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --version 2.8.5 \
    -f servers-values.yaml \
    --namespace seldon \
    --install

Additional Resources

• Seldon Core 2 Documentation
• GKE Documentation
• AWS Documentation
• Azure Documentation