1 of 1

Multi-Model Serving

MLServer has been built with Multi-Model Serving (MMS) in mind. This means that, within a single instance of MLServer, you can serve multiple models under different paths. This also includes multiple versions of the same model.

This notebook shows an example of how you can leverage MMS with MLServer.

Training

We will first start by training 2 different models:

Name

Framework

Source

Trained Model Path

Training our `mnist-svm` model

Training our `mushroom-xgboost` model

Serving

The next step will be serving both our models within the same MLServer instance. For that, we will just need to create a model-settings.json file local to each of our models and a server-wide settings.json. That is,

settings.json: holds the configuration of our server (e.g. ports, log level, etc.).
models/mnist-svm/model-settings.json: holds the configuration specific to our mnist-svm model (e.g. input type, runtime to use, etc.).

`settings.json`

`models/mnist-svm/model-settings.json`

`models/mushroom-xgboost/model-settings.json`

Start serving our model

Now that we have our config in-place, we can start the server by running mlserver start .. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.

Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal.

Testing

By this point, we should have both our models getting served by MLServer. To make sure that everything is working as expected, let's send a request from each test set.

For that, we can use the Python types that the mlserver package provides out of box, or we can build our request manually.

Testing our `mnist-svm` model

Testing our `mushroom-xgboost` model

Multi-Model Serving

This notebook shows an example of how you can leverage MMS with MLServer.

Training

We will first start by training 2 different models:

Name

Framework

Source

Trained Model Path

Training our `mnist-svm` model

Training our `mushroom-xgboost` model

Serving

settings.json: holds the configuration of our server (e.g. ports, log level, etc.).
models/mnist-svm/model-settings.json: holds the configuration specific to our mnist-svm model (e.g. input type, runtime to use, etc.).

`settings.json`

`models/mnist-svm/model-settings.json`

`models/mushroom-xgboost/model-settings.json`

Start serving our model

Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal.

Testing

By this point, we should have both our models getting served by MLServer. To make sure that everything is working as expected, let's send a request from each test set.

For that, we can use the Python types that the mlserver package provides out of box, or we can build our request manually.

Testing our `mnist-svm` model

Testing our `mushroom-xgboost` model

# Original source code and more details can be found in:
# https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html

# Import datasets, classifiers and performance metrics
from sklearn import datasets, svm, metrics
from sklearn.model_selection import train_test_split

# The digits dataset
digits = datasets.load_digits()

# To apply a classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))

# Create a classifier: a support vector classifier
classifier = svm.SVC(gamma=0.001)

# Split data into train and test subsets
X_train, X_test_digits, y_train, y_test_digits = train_test_split(
    data, digits.target, test_size=0.5, shuffle=False)

# We learn the digits on the first half of the digits
classifier.fit(X_train, y_train)

# Original code and extra details can be found in:
# https://xgboost.readthedocs.io/en/latest/get_started.html#python

import os
import xgboost as xgb
import requests

from urllib.parse import urlparse
from sklearn.datasets import load_svmlight_file


TRAIN_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.train'
TEST_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.test'


def _download_file(url: str) -> str:
    parsed = urlparse(url)
    file_name = os.path.basename(parsed.path)
    file_path = os.path.join(os.getcwd(), file_name)
    
    res = requests.get(url)
    
    with open(file_path, 'wb') as file:
        file.write(res.content)
    
    return file_path

train_dataset_path = _download_file(TRAIN_DATASET_URL)
test_dataset_path = _download_file(TEST_DATASET_URL)

# NOTE: Workaround to load SVMLight files from the XGBoost example
X_train, y_train = load_svmlight_file(train_dataset_path)
X_test_agar, y_test_agar = load_svmlight_file(test_dataset_path)
X_train = X_train.toarray()
X_test_agar = X_test_agar.toarray()

# read in data
dtrain = xgb.DMatrix(data=X_train, label=y_train)

# specify parameters via map
param = {'max_depth':2, 'eta':1, 'objective':'binary:logistic' }
num_round = 2
bst = xgb.train(param, dtrain, num_round)

bst

# Original source code and more details can be found in:
# https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html

# Import datasets, classifiers and performance metrics
from sklearn import datasets, svm, metrics
from sklearn.model_selection import train_test_split

# The digits dataset
digits = datasets.load_digits()

# To apply a classifier on this data, we need to flatten the image, to
# turn the data in a (samples, feature) matrix:
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))

# Create a classifier: a support vector classifier
classifier = svm.SVC(gamma=0.001)

# Split data into train and test subsets
X_train, X_test_digits, y_train, y_test_digits = train_test_split(
    data, digits.target, test_size=0.5, shuffle=False)

# We learn the digits on the first half of the digits
classifier.fit(X_train, y_train)

# Original code and extra details can be found in:
# https://xgboost.readthedocs.io/en/latest/get_started.html#python

import os
import xgboost as xgb
import requests

from urllib.parse import urlparse
from sklearn.datasets import load_svmlight_file


TRAIN_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.train'
TEST_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.test'


def _download_file(url: str) -> str:
    parsed = urlparse(url)
    file_name = os.path.basename(parsed.path)
    file_path = os.path.join(os.getcwd(), file_name)
    
    res = requests.get(url)
    
    with open(file_path, 'wb') as file:
        file.write(res.content)
    
    return file_path

train_dataset_path = _download_file(TRAIN_DATASET_URL)
test_dataset_path = _download_file(TEST_DATASET_URL)

# NOTE: Workaround to load SVMLight files from the XGBoost example
X_train, y_train = load_svmlight_file(train_dataset_path)
X_test_agar, y_test_agar = load_svmlight_file(test_dataset_path)
X_train = X_train.toarray()
X_test_agar = X_test_agar.toarray()

# read in data
dtrain = xgb.DMatrix(data=X_train, label=y_train)

# specify parameters via map
param = {'max_depth':2, 'eta':1, 'objective':'binary:logistic' }
num_round = 2
bst = xgb.train(param, dtrain, num_round)

bst

Multi-Model Serving

hashtagTraining

hashtagTraining our mnist-svm model

hashtagTraining our mushroom-xgboost model

hashtagServing

hashtagsettings.json

hashtagmodels/mnist-svm/model-settings.json

hashtagmodels/mushroom-xgboost/model-settings.json

hashtagStart serving our model

hashtagTesting

hashtagTesting our mnist-svm model

hashtagTesting our mushroom-xgboost model

Multi-Model Serving

hashtagTraining

hashtagTraining our mnist-svm model

hashtagTraining our mushroom-xgboost model

hashtagServing

hashtagsettings.json

hashtagmodels/mnist-svm/model-settings.json

hashtagmodels/mushroom-xgboost/model-settings.json

hashtagStart serving our model

hashtagTesting

hashtagTesting our mnist-svm model

hashtagTesting our mushroom-xgboost model

Training

Training our `mnist-svm` model

Training our `mushroom-xgboost` model

Serving

`settings.json`

`models/mnist-svm/model-settings.json`

`models/mushroom-xgboost/model-settings.json`

Start serving our model

Testing

Testing our `mnist-svm` model

Testing our `mushroom-xgboost` model

Training

Training our `mnist-svm` model

Training our `mushroom-xgboost` model

Serving

`settings.json`

`models/mnist-svm/model-settings.json`

`models/mushroom-xgboost/model-settings.json`

Start serving our model

Testing

Testing our `mnist-svm` model

Testing our `mushroom-xgboost` model