# MLServer An open source inference server for your machine learning models. [![video\_play\_icon](https://user-images.githubusercontent.com/10466106/151803854-75d17c32-541c-4eee-b589-d45b07ea486d.png)](https://www.youtube.com/watch?v=aZHe3z-8C_w) ## Overview MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with [KFServing's V2 Dataplane](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html) spec. Watch a quick video introducing the project [here](https://www.youtube.com/watch?v=aZHe3z-8C_w). * Multi-model serving, letting users run multiple models within the same process. * Ability to run [inference in parallel for vertical scaling](https://mlserver.readthedocs.io/en/latest/user-guide/parallel-inference.html) across multiple models through a pool of inference workers. * Support for [adaptive batching](https://mlserver.readthedocs.io/en/latest/user-guide/adaptive-batching.html), to group inference requests together on the fly. * Scalability with deployment in Kubernetes native frameworks, including [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#v2-kfserving-protocol) and [KServe (formerly known as KFServing)](https://kserve.github.io/website/modelserving/v1beta1/sklearn/v2/), where MLServer is the core Python inference server used to serve machine learning models. * Support for the standard [V2 Inference Protocol](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html) on both the gRPC and REST flavours, which has been standardised and adopted by various model serving frameworks. You can read more about the goals of this project on the [initial design document](https://docs.google.com/document/d/1C2uf4SaAtwLTlBCciOhvdiKQ2Eay4U72VxAD4bXe7iU/edit?usp=sharing). ## Usage You can install the `mlserver` package running: ```bash pip install mlserver ``` Note that to use any of the optional [inference runtimes](#inference-runtimes), you'll need to install the relevant package. For example, to serve a `scikit-learn` model, you would need to install the `mlserver-sklearn` package: ```bash pip install mlserver-sklearn ``` For further information on how to use MLServer, you can check any of the [available examples](#examples). ## Inference Runtimes Inference runtimes allow you to define how your model should be used within MLServer. You can think of them as the **backend glue** between MLServer and your machine learning framework of choice. You can read more about [inference runtimes in their documentation page](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/runtimes/index.md). Out of the box, MLServer comes with a set of pre-packaged runtimes which let you interact with a subset of common frameworks. This allows you to start serving models saved in these frameworks straight away. However, it's also possible to [**write custom runtimes**](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/runtimes/custom.md). Out of the box, MLServer provides support for: | Framework | Supported | Documentation | | ------------- | --------- | ------------------------------------------------------------------------------------------------------------------- | | Scikit-Learn | ✅ | [MLServer SKLearn](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/sklearn/README.md) | | XGBoost | ✅ | [MLServer XGBoost](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/xgboost/README.md) | | Spark MLlib | ✅ | [MLServer MLlib](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/mllib/README.md) | | LightGBM | ✅ | [MLServer LightGBM](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/lightgbm/README.md) | | CatBoost | ✅ | [MLServer CatBoost](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/catboost/README.md) | | Tempo | ✅ | [`github.com/SeldonIO/tempo`](https://github.com/SeldonIO/tempo) | | MLflow | ✅ | [MLServer MLflow](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/mlflow/README.md) | | Alibi-Detect | ✅ | [MLServer Alibi Detect](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/alibi-detect/README.md) | | Alibi-Explain | ✅ | [MLServer Alibi Explain](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/alibi-explain/README.md) | | HuggingFace | ✅ | [MLServer HuggingFace](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/runtimes/huggingface/README.md) | ## Supported Python Versions 🔴 Unsupported 🟠 Deprecated: To be removed in a future version 🟢 Supported 🔵 Untested | Python Version | Status | | -------------- | ------ | | 3.7 | 🔴 | | 3.8 | 🔴 | | 3.9 | 🟢 | | 3.10 | 🟢 | | 3.11 | 🟢 | | 3.12 | 🟢 | | 3.13 | 🔴 | ## Examples To see MLServer in action, check out [our full list of examples](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/index.md). You can find below a few selected examples showcasing how you can leverage MLServer to start serving your machine learning models. * [Serving a `scikit-learn` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/sklearn/README.md) * [Serving a `xgboost` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/xgboost/README.md) * [Serving a `lightgbm` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/lightgbm/README.md) * [Serving a `catboost` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/catboost/README.md) * [Serving a `tempo` pipeline](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/tempo/README.md) * [Serving a custom model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/custom/README.md) * [Serving an `alibi-detect` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/alibi-detect/README.md) * [Serving a `HuggingFace` model](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/huggingface/README.md) * [Multi-Model Serving with multiple frameworks](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/mms/README.md) * [Loading / unloading models from a model repository](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/examples/model-repository/README.md) ## Developer Guide ### Versioning Both the main `mlserver` package and the [inference runtimes packages](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/docs/runtimes/index.md) try to follow the same versioning schema. To bump the version across all of them, you can use the [`./hack/update-version.sh`](https://github.com/SeldonIO/MLServer/blob/master/docs-gb/hack/update-version.sh) script. We generally keep the version as a placeholder for an upcoming version. For example: ```bash ./hack/update-version.sh 0.2.0.dev1 ``` ### Testing To run all of the tests for MLServer and the runtimes, use: ```bash make test ``` To run run tests for a single file, use something like: ```bash tox -e py3 -- tests/batch_processing/test_rest.py ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.seldon.ai/mlserver/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.