This package provides a MLServer runtime compatible with Spark MLlib.

Usage

You can install the runtime, alongside mlserver, as:

pip install mlserver mlserver-mllib

For further information on how to use MLServer with Spark MLlib, you can check out the MLServer repository.

This package provides a MLServer runtime compatible with .

Usage

You can install the runtime, alongside mlserver, as:

Out of the box, MLServer includes support for streaming data to your models. Streaming support is available for both the REST and gRPC servers.

REST Server

Streaming support for the REST server is limited only to server streaming. This means that the client sends a single request to the server, and the server responds with a stream of data.

The streaming endpoints are available for both the infer and generate

MLServer follows the Open Inference Protocol (previously known as the "V2 Protocol"). You can find the full OpenAPI spec for the Open Inference Protocol in the links below:

Swagger UI

On top of the OpenAPI spec above, MLServer also autogenerates a Swagger UI which can be used to interact dynamycally with the Open Inference Protocol.

The autogenerated Swagger UI can be accessed under the /v2/docs endpoint.

Model Swagger UI

Alongside the , MLServer will also autogenerate a Swagger UI tailored to individual models, showing the endpoints available for each one.

The model-specific autogenerated Swagger UI can be accessed under the following endpoints:

• /v2/models/{model_name}/docs

• /v2/models/{model_name}/versions/{model_version}/docs

MLServer is currently used as the core Python inference server in some of most popular Kubernetes-native serving frameworks, including Seldon Core and KServe (formerly known as KFServing). This allows MLServer users to leverage the usability and maturity of these frameworks to take their model deployments to the next level of their MLOps journey, ensuring that they are served in a robust and scalable infrastructure.

Out of the box, MLServer includes support to offload inference workloads to a pool of workers running in separate processes. This allows MLServer to scale out beyond the limitations of the Python interpreter. To learn more about why this can be beneficial, you can check the concurrency section below.

By default, MLServer will spin up a pool with only one worker process to run inference. All models will be loaded uniformly across the inference pool workers. To read more about advanced settings, please see the usage section below.

Concurrency in Python

The is a mutex lock that exists in most Python interpreters (e.g. CPython). Its main purpose is to lock Python’s execution so that it only runs on a single processor at the same time. This simplifies certain things to the interpreter. However, it also adds the limitation that a single Python process will never be able to leverage multiple cores.

When we think about MLServer's support for , this could lead to scenarios where a heavily-used model starves the other models running within the same MLServer instance. Similarly, even if we don’t take MMS into account, the GIL also makes it harder to scale inference for a single model.

To work around this limitation, MLServer offloads the model inference to a pool of workers, where each worker is a separate Python process (and thus has its own separate GIL). This means that we can get full access to the underlying hardware.

Overhead

Managing the Inter-Process Communication (IPC) between the main MLServer process and the inference pool workers brings in some overhead. Under the hood, MLServer uses the multiprocessing library to implement the distributed processing management, which has been shown to offer the smallest possible overhead when implementing these type of distributed strategies {cite}zhiFiberPlatformEfficient2020.

The extra overhead introduced by other libraries is usually brought in as a trade off in exchange of other advanced features for complex distributed processing scenarios. However, MLServer's use case is simple enough to not require any of these.

Despite the above, even though this overhead is minimised, this it can still be particularly noticeable for lightweight inference methods, where the extra IPC overhead can take a large percentage of the overall time. In these cases (which can only be assessed on a model-by-model basis), the user has the option to .

For regular models where inference can take a bit more time, this overhead is usually offset by the benefit of having multiple cores to compute inference on.

Usage

By default, MLServer will always create an inference pool with one single worker. The number of workers (i.e. the size of the inference pool) can be adjusted globally through the server-level parallel_workers setting.

parallel_workers

The parallel_workers field of the settings.json file (or alternatively, the MLSERVER_PARALLEL_WORKERS global environment variable) controls the size of MLServer's inference pool. The expected values are:

• N, where N > 0, will create a pool of N workers.

• 0, will disable the parallel inference feature. In other words, inference will happen within the main MLServer process.

inference_pool_gid

The inference_pool_gid field of the model-settings.json file (or alternatively, the MLSERVER_MODEL_INFERENCE_POOL_GID global environment variable) allows to load models on a dedicated inference pool based on the group ID (GID) to prevent starvation behavior.

Complementing the inference_pool_gid, if the autogenerate_inference_pool_gid field of the model-settings.json file (or alternatively, the MLSERVER_MODEL_AUTOGENERATE_INFERENCE_POOL_GID global environment variable) is set to True, a UUID is automatically generated, and a dedicated inference pool will load the given model. This option is useful if the user wants to load a single model on an dedicated inference pool without having to manage the GID themselves.

References

Jiale Zhi, Rui Wang, Jeff Clune, and Kenneth O. Stanley. Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods. arXiv:2003.11164 [cs, stat], March 2020. .

This package provides a MLServer runtime compatible with Scikit-Learn.

Usage

You can install the runtime, alongside mlserver, as:

For further information on how to use MLServer with Scikit-Learn, you can check out this .

MetricsServer

Methods

on_worker_stop()

start()

stop()

configure_metrics()

No description available.

log()

Logs a new set of metric values. Each kwarg of this method will be treated as a separate metric / value pair. If any of the metrics does not exist, a new one will be created with a default description.

register()

Registers a new metric with its description. If the metric already exists, it will just return the existing one.

Out-of-the-box, MLServer exposes a set of metrics that help you monitor your machine learning workloads in production. These include standard metrics like number of requests and latency.

On top of these, you can also register and track your own as part of your .

Default Metrics

By default, MLServer will expose metrics around inference requests (count and error rate) and the status of its internal requests queues. These internal queues are used for and .

MLServer is used as the in . This allows for a straightforward avenue to deploy your models into a scalable serving infrastructure backed by Kubernetes.

MLServer is used as the in . Therefore, it should be straightforward to deploy your models either by using one of the or by pointing to a .

This package provides a MLServer runtime compatible with XGBoost.

Usage

You can install the runtime, alongside mlserver, as:

For further information on how to use MLServer with XGBoost, you can check out this .

This package provides a MLServer runtime compatible with LightGBM.

Usage

You can install the runtime, alongside mlserver, as:

For further information on how to use MLServer with LightGBM, you can check out this .

There may be cases where the offered out-of-the-box by MLServer may not be enough, or where you may need extra custom functionality which is not included in MLServer (e.g. custom codecs). To cover these cases, MLServer lets you create custom runtimes very easily.

To learn more about how you can write custom runtimes with MLServer, check out the . Alternatively, you can also see this which walks through the process of writing a custom runtime.

This package provides a MLServer runtime compatible with Alibi-Explain.

Usage

You can install the runtime, alongside mlserver, as:

pip install mlserver mlserver-alibi-explain

This guide will help you get started creating machine learning microservices with MLServer in less than 30 minutes. Our use case will be to create a service that helps us compare the similarity between two documents. Think about whenever you are comparing a book, news article, blog post, or tutorial to read next, wouldn't it be great to have a way to compare with similar ones that you have already read and liked (without having to rely on a recommendation's system)? That's what we'll focus on this guide, on creating a document similarity service. 📜 + 📃 = 😎👌🔥

The code is showcased as if it were cells inside a notebook but you can run each of the steps inside Python files with minimal effort.

00 What is MLServer?

MLServer is an open-source Python library for building production-ready asynchronous APIs for machine learning models.

01 Dependencies

The first step is to install mlserver, the spacy library, and the spacy will need for our use case. We will also download the wikipedia-api library to test our use case with a few fun summaries.

If you've never heard of before, it is an open-source Python library for advanced natural language processing that excels at large-scale information extraction and retrieval tasks, among many others. The model we'll use is a pre-trained model on English text from the web. This model will help us get started with our use case faster than if we had to train a model from scratch for our use case.

Let's first install these libraries.

We will also need to download the language model separately once we have spaCy inside our virtual environment.

If you're going over this guide inside a notebook, don't forget to add an exclamation mark ! in front of the two commands above. If you are in VSCode, you can keep them as they are and change the cell type to bash.

02 Set Up

At its core, MLServer requires that users give it 3 things, a model-settings.json file with information about the model, an (optional) settings.json file with information related to the server you are about to set up, and a .py file with the load-predict recipe for your model (as shown in the picture above).

Let's create a directory for our model.

Before we create a service that allows us to compare the similarity between two documents, it is good practice to first test that our solution works first, especially if we're using a pre-trained model and/or a pipeline.

Now that we have our model loaded, let's look at the similarity of the abstracts of using the wikipedia-api Python library. The main requirement of the API is that we pass into the main class, Wikipedia(), a project name, an email and the language we want information to be returned in. After that, we can search the for the movie summaries we want by passing the title of the movie to the .page() method and accessing the summary of it with the .summary attribute.

Feel free to change the movies for other topics you might be interested in.

You can run the following lines inside a notebook or, conversely, add them to a app.py file.

If you created an app.py file with the code above, make sure you run python app.py from the terminal.

Now that we have our two summaries, let's compare them using spacy.

Notice that both summaries have information about the other movie, about "films" in general, and about the dates each aired on (which is the same). The reality is that, the model hasn't seen any of these movies so it might be generalizing to the context of each article, "movies," rather than their content, "dolls as humans and the atomic bomb."

You should, of course, play around with different pages and see if what you get back is coherent with what you would expect.

Time to create a machine learning API for our use-case. 😎

03 Building a Service

MLServer allows us to wrap machine learning models into APIs and build microservices with replicas of a single model, or different models all together.

To create a service with MLServer, we will define a class with two asynchronous functions, one that loads the model and another one to run inference (or predict) with. The former will load the spacy model we tested in the last section, and the latter will take in a list with the two documents we want to compare. Lastly, our function will return a numpy array with a single value, our similarity score. We'll write the file to our similarity_model directory and call it my_model.py.

Now that we have our model file ready to go, the last piece of the puzzle is to tell MLServer a bit of info about it. In particular, it wants (or needs) to know the name of the model and how to implement it. The former can be anything you want (and it will be part of the URL of your API), and the latter will follow the recipe of name_of_py_file_with_your_model.class_with_your_model.

Let's create the model-settings.json file MLServer is expecting inside our similarity_model directory and add the name and the implementation of our model to it.

Now that everything is in place, we can start serving predictions locally to test how things would play out for our future users. We'll initiate our server via the command line, and later on we'll see how to do the same via Python files. Here's where we are at right now in the process of developing microservices with MLServer.

As you can see in the image, our server will be initialized with three entry points, one for HTTP requests, another for gRPC, and another for the metrics. To learn more about the powerful metrics feature of MLServer, please visit the relevant docs page . To learn more about gRPC, please see this tutorial .

To start our service, open up a terminal and run the following command.

Note: If this is a fresh terminal, make sure you activate your environment before you run the command above. If you run the command above from your notebook (e.g. !mlserver start similarity_model/), you will have to send the request below from another notebook or terminal since the cell will continue to run until you turn it off.

04 Testing our Service

Time to become a client of our service and test it. For this, we'll set up the payload we'll send to our service and use the requests library to our request.

Please note that the request below uses the variables we created earlier with the summaries of Barbie and Oppenheimer. If you are sending this POST request from a fresh python file, make sure you move those lines of code above into your request file.

Let's decompose what just happened.

The URL for our service might seem a bit odd if you've never heard of the . This protocol is a set of specifications that allows machine learning models to be shared and deployed in a standardized way. This protocol enables the use of machine learning models on a variety of platforms and devices without requiring changes to the model or its code. The OIP is useful because it allows us to integrate machine learning into a wide range of applications in a standard way.

All URLs you create with MLServer will have the following structure.

This kind of protocol is a standard adopted by different companies like NVIDIA, Tensorflow Serving, KServe, and others, to keep everyone on the same page. If you think about driving cars globally, your country has to apply a standard for driving on a particular side of the road, and this ensures you and everyone else stays on the left (or the right depending on where you are at). Adopting this means that you won't have to wonder where the next driver is going to come out of when you are driving and are about to take a turn, instead, you can focus on getting to where you're going to without much worrying.

Let's describe what each of the components of our inference_request does.

• name: this maps one-to-one to the name of the parameter in your predict() function.

• shape: represents the shape of the elements in our data. In our case, it is a list with [2] strings.

To learn more about the OIP and how MLServer content types work, please have a looks at their .

05 Creating Model Replicas

Say you need to meet the demand of a high number of users and one model might not be enough, or is not using all of the resources of the virtual machine instance it was allocated to. What we can do in this case is to create multiple replicas of our model to increase the throughput of the requests that come in. This can be particularly useful at the peak times of our server. To do this, we need to tweak the settings of our server via the settings.json file. In it, we'll add the number of independent models we want to have to the parameter "parallel_workers": 3.

Let's stop our server, change the settings of it, start it again, and test it.

As you can see in the output of the terminal in the picture above, we now have 3 models running in parallel. The reason you might see 4 is because, by default, MLServer will print the name of the initialized model if it is one or more, and it will also print one for each of the replicas specified in the settings.

Let's get a few more to test our server. Get as creative as you'd like. 💡

Let's first test that the function works as intended.

Now let's map three POST requests at the same time.

We can also test it one by one.

06 Packaging our Service

For the last step of this guide, we are going to package our model and service into a docker image that we can reuse in another project or share with colleagues immediately. This step requires that we have docker installed and configured in our PCs, so if you need to set up docker, you can do so by following the instructions in the documentation .

The first step is to create a requirements.txt file with all of our dependencies and add it to the directory we've been using for our service (similarity_model).

The next step is to build a docker image with our model, its dependencies and our server. If you've never heard of docker images before, here's a short description.

A Docker image is a lightweight, standalone, and executable package that includes everything needed to run a piece of software, including code, libraries, dependencies, and settings. It's like a carry-on bag for your application, containing everything it needs to travel safely and run smoothly in different environments. Just as a carry-on bag allows you to bring your essentials with you on a trip, a Docker image enables you to transport your application and its requirements across various computing environments, ensuring consistent and reliable deployment.

MLServer has a convenient function that lets us create docker images with our services. Let's use it.

We can check that our image was successfully build not only by looking at the logs of the previous command but also with the docker images command.

Let's test that our image works as intended with the following command. Make sure you have closed your previous server by using CTRL + C in your terminal.

Now that you have a packaged and fully-functioning microservice with our model, we could deploy our container to a production serving platform like , or via different offerings available through the many cloud providers out there (e.g. AWS Lambda, Google Cloud Run, etc.). You could also run this image on KServe, a Kubernetes native tool for model serving, or anywhere else where you can bring your docker image with you.

To learn more about MLServer and the different ways in which you can use it, head over to the section or the . To learn about some of the deployment options available, head over to the docs .

To keep up to date with what we are up to at Seldon, make sure you join our .

Inference runtimes allow you to define how your model should be used within MLServer. You can think of them as the backend glue between MLServer and your machine learning framework of choice.

Out of the box, MLServer comes with a set of pre-packaged runtimes which let you interact with a subset of common ML frameworks. This allows you to start serving models saved in these frameworks straight away. To avoid bringing in dependencies for frameworks that you don't need to use, these runtimes are implemented as independent (and optional) Python packages. This mechanism also allows you to rollout your own custom runtimes very easily.

To pick which runtime you want to use for your model, you just need to make sure that the right package is installed, and then point to the correct runtime class in your model-settings.json file.

Included Inference Runtimes

Config

MLServer exposes a Python framework to build custom inference runtimes, define request/response types, plug codecs for payload conversion, and emit metrics. This page provides a high-level overview and links to the API docs.

• • Base class to implement custom inference runtimes.

This package provides a MLServer runtime compatible with CatBoost's CatboostClassifier.

Usage

You can install the runtime, alongside mlserver, as:

For further information on how to use MLServer with CatBoost, you can check out this .

This package provides a MLServer runtime compatible with HuggingFace Transformers.

Usage

You can install the runtime, alongside mlserver, as:

For further information on how to use MLServer with HuggingFace, you can check out this .

This package provides a MLServer runtime compatible with models.

Usage

You can install the mlserver-alibi-detect runtime, alongside mlserver, as:

For further information on how to use MLServer with Alibi-Detect, you can check out this

Out of the box, mlserver supports the deployment and serving of lightgbm models. By default, it will assume that these models have been serialised using the bst.save_model() method.

In this example, we will cover how we can train and serialise a simple model, to then serve it using mlserver.

Training

To test the LightGBM Server, first we need to generate a simple LightGBM model using Python.

Our model will be persisted as a file named iris-lightgbm.bst.

Serving

Now that we have trained and saved our model, the next step will be to serve it using mlserver. For that, we will need to create 2 configuration files:

• settings.json: holds the configuration of our server (e.g. ports, log level, etc.).

• model-settings.json: holds the configuration of our model (e.g. input type, runtime to use, etc.).

settings.json

model-settings.json

Start serving our model

Now that we have our config in-place, we can start the server by running mlserver start .. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.

Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal.

Send test inference request

We now have our model being served by mlserver. To make sure that everything is working as expected, let's send a request from our test set.

For that, we can use the Python types that mlserver provides out of box, or we can build our request manually.

As we can see above, the model predicted the probability for each class, and the probability of class 1 is the biggest, close to 0.99, which matches what's on the test set.

The MLServer package includes a mlserver CLI designed to help with common tasks in a model’s lifecycle. You can see a high-level outline at any time via:

mlserver

Command-line interface to manage MLServer models.

Options

• --version (Default: False) Show the version and exit.

build

Build a Docker image for a custom MLServer runtime.

Options

• -t, --tag <text>

• --no-cache (Default: False)

Arguments

• FOLDER Required argument

dockerfile

Generate a Dockerfile

Options

• -i, --include-dockerignore (Default: False)

Arguments

• FOLDER Required argument

infer

Deprecated: This experimental feature will be removed in future work. Execute batch inference requests against V2 inference server.

Deprecated: This experimental feature will be removed in future work.

Options

• --url, -u <text> (Default: localhost:8080; Env: MLSERVER_INFER_URL) URL of the MLServer to send inference requests to. Should not contain http or https.

• --model-name, -m

init

Generate a base project template

Options

• -t, --template <text> (Default: https://github.com/EthicalML/sml-security/)

start

Start serving a machine learning model with MLServer.

Arguments

• FOLDER Required argument

Abstract inference runtime which exposes the main interface to interact with ML models.

Methods

decode()

Helper to decode a request input into its corresponding high-level Python object. This method will find the most appropiate :doc:input codec </user-guide/content-type> based on the model's metadata and the input's content type. Otherwise, it will fall back to the codec specified in the default_codec kwarg.

decode_request()

Helper to decode an inference request into its corresponding high-level Python object. This method will find the most appropiate :doc:request codec </user-guide/content-type> based on the model's metadata and the requests's content type. Otherwise, it will fall back to the codec specified in the default_codec kwarg.

encode()

Helper to encode a high-level Python object into its corresponding response output. This method will find the most appropiate :doc:input codec </user-guide/content-type> based on the model's metadata, request output's content type or payload's type. Otherwise, it will fall back to the codec specified in the default_codec kwarg.

encode_response()

Helper to encode a high-level Python object into its corresponding inference response. This method will find the most appropiate :doc:request codec </user-guide/content-type> based on the payload's type. Otherwise, it will fall back to the codec specified in the default_codec kwarg.

load()

Method responsible for loading the model from a model artefact. This method will be called on each of the parallel workers (when :doc:parallel inference </user-guide/parallel-inference>) is enabled). Its return value will represent the model's readiness status. A return value of True will mean the model is ready.

This method can be overriden to implement your custom load logic.

metadata()

No description available.

predict()

Method responsible for running inference on the model.

This method can be overriden to implement your custom inference logic.

predict_stream()

Method responsible for running generation on the model, streaming a set of responses back to the client.

This method can be overriden to implement your custom inference logic.

unload()

Method responsible for unloading the model, freeing any resources (e.g. CPU memory, GPU memory, etc.). This method will be called on each of the parallel workers (when :doc:parallel inference </user-guide/parallel-inference>) is enabled). A return value of True will mean the model is now unloaded.

This method can be overriden to implement your custom unload logic.

MLServer supports loading and unloading models dynamically from a models repository. This allows you to enable and disable the models accessible by MLServer on demand. This extension builds on top of the support for Multi-Model Serving, letting you change at runtime which models is MLServer currently serving.

The API to manage the model repository is modelled after Triton's Model Repository extension to the V2 Dataplane and is thus fully compatible with it.

This notebook will walk you through an example using the Model Repository API.

Training

First of all, we will need to train some models. For that, we will re-use the models we trained previously in the . You can check the details on how they are trained following that notebook.

Serving

Next up, we will start our mlserver inference server. Note that, by default, this will load all our models.

List available models

Now that we've got our inference server up and running, and serving 2 different models, we can start using the Model Repository API. To get us started, we will first list all available models in the repository.

As we can, the repository lists 2 models (i.e. mushroom-xgboost and mnist-svm). Note that the state for both is set to READY. This means that both models are loaded, and thus ready for inference.

Unloading our mushroom-xgboost model

We will now try to unload one of the 2 models, mushroom-xgboost. This will unload the model from the inference server but will keep it available on our model repository.

If we now try to list the models available in our repository, we will see that the mushroom-xgboost model is flagged as UNAVAILABLE. This means that it's present in the repository but it's not loaded for inference.

Loading our mushroom-xgboost model back

We will now load our model back into our inference server.

If we now try to list the models again, we will see that our mushroom-xgboost is back again, ready for inference.

Config

Out of the box, mlserver supports the deployment and serving of xgboost models. By default, it will assume that these models have been serialised using the bst.save_model() method.

In this example, we will cover how we can train and serialise a simple model, to then serve it using mlserver.

Training

The first step will be to train a simple xgboost model. For that, we will use the .

Saving our trained model

To save our trained model, we will serialise it using bst.save_model() and the JSON format. This is the .

Our model will be persisted as a file named mushroom-xgboost.json.

Serving

Now that we have trained and saved our model, the next step will be to serve it using mlserver. For that, we will need to create 2 configuration files:

• settings.json: holds the configuration of our server (e.g. ports, log level, etc.).

• model-settings.json: holds the configuration of our model (e.g. input type, runtime to use, etc.).

settings.json

model-settings.json

Start serving our model

Now that we have our config in-place, we can start the server by running mlserver start .. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.

Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal.

Send test inference request

We now have our model being served by mlserver. To make sure that everything is working as expected, let's send a request from our test set.

For that, we can use the Python types that mlserver provides out of box, or we can build our request manually.

As we can see above, the model predicted the input as close to 0, which matches what's on the test set.

Out of the box, mlserver supports the deployment and serving of scikit-learn models. By default, it will assume that these models have been serialised using joblib.

In this example, we will cover how we can train and serialise a simple model, to then serve it using mlserver.

Training

The first step will be to train a simple scikit-learn model. For that, we will use the which trains an SVM model.

Saving our trained model

To save our trained model, we will serialise it using joblib. While this is not a perfect approach, it's currently the recommended method to persist models to disk in the .

Our model will be persisted as a file named mnist-svm.joblib

Serving

Now that we have trained and saved our model, the next step will be to serve it using mlserver. For that, we will need to create 2 configuration files:

• settings.json: holds the configuration of our server (e.g. ports, log level, etc.).

• model-settings.json: holds the configuration of our model (e.g. input type, runtime to use, etc.).

settings.json

model-settings.json

Start serving our model

Now that we have our config in-place, we can start the server by running mlserver start .. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.

Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal.

Send test inference request

We now have our model being served by mlserver. To make sure that everything is working as expected, let's send a request from our test set.

For that, we can use the Python types that mlserver provides out of box, or we can build our request manually.

As we can see above, the model predicted the input as the number 8, which matches what's on the test set.

There may be cases where the inference runtimes offered out-of-the-box by MLServer may not be enough, or where you may need extra custom functionality which is not included in MLServer (e.g. custom codecs). To cover these cases, MLServer lets you create custom runtimes very easily.

This page covers some of the bigger points that need to be taken into account when extending MLServer. You can also see this which walks through the process of writing a custom runtime.

Writing a custom inference runtime

MLServer is designed as an easy-to-extend framework, encouraging users to write their own custom runtimes easily. The starting point for this is the

It's not unusual that model runtimes require extra dependencies that are not direct dependencies of MLServer. This is the case when we want to use , but also when our model artifacts are the output of older versions of a toolkit (e.g. models trained with an older version of SKLearn).

In these cases, since these dependencies (or dependency versions) are not known in advance by MLServer, they won't be included in the default seldonio/mlserver Docker image. To cover these cases, the seldonio/mlserver Docker image allows you to load custom environments before starting the server itself.

This example will walk you through how to create and save an custom environment, so that it can be loaded in MLServer without any extra change to the seldonio/mlserver Docker image.

Machine learning models generally expect their inputs to be passed down as a particular Python type. Most commonly, this type ranges from "general purpose" NumPy arrays or Pandas DataFrames to more granular definitions, like datetime objects, Pillow images, etc. Unfortunately, the definition of the V2 Inference Protocol doesn't cover any of the specific use cases. This protocol can be thought of a wider "lower level" spec, which only defines what fields a payload should have.

To account for this gap, MLServer introduces support for content types, which offer a way to let MLServer know how it should "decode" V2-compatible payloads. When shaped in the right way, these payloads should "encode" all the information required to extract the higher level Python type that will be required for a model.

To illustrate the above, we can think of a Scikit-Learn pipeline, which takes in a Pandas DataFrame and returns a NumPy Array. Without the use of content types, the V2 payload itself would probably lack information about how this payload should be treated by MLServer Likewise, the Scikit-Learn pipeline wouldn't know how to treat a raw V2 payload. In this scenario, the use of content types allows us to specify information on what's the actual "higher level" information encoded within the V2 protocol payloads.

Usage

To let MLServer know that a particular payload must be decoded / encoded as a different Python data type (e.g. NumPy Array, Pandas DataFrame, etc.), you can specify it through the content_type field of the parameters section of your request.

As an example, we can consider the following dataframe, containing two columns: Age and First Name.

This table, could be specified in the V2 protocol as the following payload, where we declare that:

• The whole set of inputs should be decoded as a Pandas Dataframe (i.e. setting the content type as pd).

• The First Name column should be decoded as a UTF-8 string (i.e. setting the content type as str).

To learn more about the available content types and how to use them, you can see all the available ones in the section below.

Codecs

Under the hood, the conversion between content types is implemented using codecs. In the MLServer architecture, codecs are an abstraction which know how to encode and decode high-level Python types to and from the V2 Inference Protocol.

Depending on the high-level Python type, encoding / decoding operations may require access to multiple input or output heads. For example, a Pandas Dataframe would need to aggregate all of the input-/output-heads present in a V2 Inference Protocol response.

However, a Numpy array or a list of strings, could be encoded directly as an input head within a larger request.

To account for this, codecs can work at either the request- / response-level (known as request codecs), or the input- / output-level (known as input codecs). Each of these codecs, expose the following public interface, where Any represents a high-level Python datatype (e.g. a Pandas Dataframe, a Numpy Array, etc.):

• Request Codecs

  • encode_request() <mlserver.codecs.RequestCodec.encode_request>

  • decode_request() <mlserver.codecs.RequestCodec.decode_request>

Note that, these methods can also be used as helpers to encode requests and decode responses on the client side. This can help to abstract away from the user most of the details about the underlying structure of V2-compatible payloads.

For example, in the example above, we could use codecs to encode the DataFrame into a V2-compatible request simply as:

For a full end-to-end example on how content types and codecs work under the hood, feel free to check out this .

Converting to / from JSON

When using MLServer's request codecs, the output of encoding payloads will always be one of the classes within the mlserver.types package (i.e. InferenceRequest <mlserver.types.InferenceRequest> or InferenceResponse <mlserver.types.InferenceResponse>). Therefore, if you want to use them with requests (or other package outside of MLServer) you will need to convert them to a Python dict or a JSON string.

Luckily, these classes leverage under the hood. Therefore you can just call the .model_dump() or .model_dump_json() method to convert them. Likewise, to read them back from JSON, we can always pass the JSON fields as kwargs to the class' constructor (or use any of the available within Pydantic).

For example, if we want to send an inference request to model foo, we could do something along the following lines:

Support for NaN values

The NaN (Not a Number) value is used in Numpy and other scientific libraries to describe an invalid or missing value (e.g. a division by zero). In some scenarios, it may be desirable to let your models receive and / or output NaN values (e.g. these can be useful sometimes with GBTs, like XGBoost models). This is why MLServer supports encoding NaN values on your request / response payloads under some conditions.

In order to send / receive NaN values, you must ensure that:

• You are using the REST interface.

• The input / output entry containing NaN values uses either the FP16, FP32 or FP64 datatypes.

Assuming those conditions are satisfied, any null value within your tensor payload will be converted to NaN.

For example, if you take the following Numpy array:

We could encode it as:

Model Metadata

Content types can also be defined as part of the . This lets the user pre-configure what content types should a model use by default to decode / encode its requests / responses, without the need to specify it on each request.

For example, to configure the content type values of the , one could create a model-settings.json file like the one below:

It's important to keep in mind that content types passed explicitly as part of the request will always take precedence over the model's metadata. Therefore, we can leverage this to override the model's metadata when needed.

Available Content Types

Out of the box, MLServer supports the following list of content types. However, this can be extended through the use of 3rd-party or custom runtimes.

NumPy Array

The np content type will decode / encode V2 payloads to a NumPy Array, taking into account the following:

• The datatype field will be matched to the closest .

• The shape field will be used to reshape the flattened array expected by the V2 protocol into the expected tensor shape.