SKLearn Iris Classifier

Note: Seldon has adopted the industry-standard Open Inference Protocol (OIP) and is no longer maintaining the Seldon and TensorFlow protocols. This transition allows for greater interoperability among various model serving runtimes, such as MLServer. To learn more about implementing OIP for model serving in Seldon Core 1, see MLServer.

We strongly encourage you to adopt the OIP, which provides seamless integration across diverse model serving runtimes, supports the development of versatile client and benchmarking tools, and ensures a high-performance, consistent, and unified inference experience.

In this notebook we create a new custom LIGHTGBM_SERVER prepackaged server with two versions:

• A Seldon protocol LightGBM model server

• A KfServing Open Inference protocol or V2 protocol version using MLServer for running lightgbm models

The Seldon model server is in defined in lightgbmserver folder.

Prerequisites

• A kubernetes cluster with kubectl configured

• curl

Setup Seldon Core

Use the setup notebook to Setup Cluster to setup Seldon Core with an ingress - either Ambassador or Istio.

Then port-forward to that ingress on localhost:8003 in a separate terminal either with:

• Ambassador: kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080

• Istio: kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080

!kubectl create namespace seldon

from IPython.core.magic import register_line_cell_magic


@register_line_cell_magic
def writetemplate(line, cell):
    with open(line, "w") as f:
        f.write(cell.format(**globals()))

VERSION = !cat ../../../version.txt
VERSION = VERSION[0]
VERSION

Training (can be skipped)

TRAIN_MODEL = False
if TRAIN_MODEL:
    import os

    import joblib
    import lightgbm as lgb
    from sklearn import datasets
    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split

    model_dir = "./artifacts"
    BST_FILE = "model.txt"

    iris = load_iris()
    y = iris["target"]
    X = iris["data"]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1)
    dtrain = lgb.Dataset(X_train, label=y_train)

    params = {"objective": "multiclass", "metric": "softmax", "num_class": 3}
    lgb_model = lgb.train(params=params, train_set=dtrain)
    model_file = os.path.join(model_dir, BST_FILE)
    lgb_model.save_model(model_file)

Update Seldon Core with Custom Model

%%writetemplate values.yaml
predictor_servers:
  MLFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/mlflowserver
  SKLEARN_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/sklearnserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  TENSORFLOW_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/tfserving-proxy
      tensorflow: 
        defaultImageVersion: 2.1.0
        image:  tensorflow/serving
  XGBOOST_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/xgboostserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  LIGHTGBM_SERVER:
    protocols:
      seldon:
        defaultImageVersion: "{VERSION}"
        image: seldonio/lighgbmserver
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver
  TRITON_SERVER:
    protocols:
      kfserving:
        defaultImageVersion: "21.08-py3"
        image: nvcr.io/nvidia/tritonserver
  TEMPO_SERVER:
    protocols:
      kfserving:
        defaultImageVersion: "0.3.2"
        image: seldonio/mlserver

!helm upgrade seldon-core  \
    ../../../helm-charts/seldon-core-operator \
    --namespace seldon-system \
    --values values.yaml \
    --set istio.enabled=true

DeployLightGBM Model with Seldon Protocol

!cat model_seldon_v1.yaml

Wait for new webhook certificates to be loaded

import time

time.sleep(60)

!kubectl create -f model_seldon_v1.yaml -n seldon

!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=iris -o jsonpath='{.items[0].metadata.name}' -n seldon) -n seldon

for i in range(60):
    state = !kubectl get sdep iris -n seldon -o jsonpath='{.status.state}'
    state = state[0]
    print(state)
    if state == "Available":
        break
    time.sleep(1)
assert state == "Available"

import json
X=!curl -s -d '{"data": {"ndarray":[[1.0, 2.0, 3.0, 4.0]]}}' \
   -X POST http://localhost:8003/seldon/seldon/iris/api/v1.0/predictions \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)

!kubectl delete -f model_seldon_v1.yaml

Deploy Model with KFserving Protocol

!cat model_seldon_v2.yaml

!kubectl create -f model_seldon_v2.yaml -n seldon

!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=iris -o jsonpath='{.items[0].metadata.name}' -n seldon) -n seldon

for i in range(60):
    state = !kubectl get sdep iris -n seldon -o jsonpath='{.status.state}'
    state = state[0]
    print(state)
    if state == "Available":
        break
    time.sleep(1)
assert state == "Available"

import json
X=!curl -s -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'\
   -X POST http://localhost:8003/seldon/seldon/iris/v2/models/infer \
   -H "Content-Type: application/json"
d=json.loads(X[0])
print(d)

!kubectl delete -f model_seldon_v2.yaml