Inference Server

Learn how to perform model inference in Seldon Core using REST and gRPC protocols, including request/response formats and client examples.

This section will discuss how to make inference calls against your Seldon models or pipelines.

You can make synchronous inference requests via REST or gRPC or asynchronous requests via Kafka topics. The content of your request should be an inference v2 protocol payload:

REST payloads will generally be in the JSON v2 protocol format.
gRPC and Kafka payloads must be in the Protobuf v2 protocol format.

Synchronous Requests

For making synchronous requests, the process will generally be:

Find the appropriate service endpoint (IP address and port) for accessing the installation of Seldon Core 2.
Determine the appropriate headers/metadata for the request.
Make requests via REST or gRPC.

Find the Seldon Service Endpoint

In the default Docker Compose setup, container ports are accessible from the host machine. This means you can use localhost or 0.0.0.0 as the hostname.

The default port for sending inference requests to the Seldon system is 9000. This is controlled by the ENVOY_DATA_PORT environment variable for Compose.

Putting this together, you can send inference requests to 0.0.0.0:9000.

In Kubernetes, Seldon creates a single Service called seldon-mesh in the namespace it is installed into. By default, this namespace is also called seldon-mesh.

If this Service is exposed via a load balancer, the appropriate address and port can be found via:

kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

If you are not using a LoadBalancer for the seldon-mesh Service, you can still send inference requests.

For development and testing purposes, you can port-forward the Service locally using the below. Inference requests can then be sent to localhost:8080.

kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80

If you are using a service mesh like Istio or Ambassador, you will need to use the IP address of the service mesh ingress and determine the appropriate port.

Make Inference Requests

Let us imagine making inference requests to a model called iris.

This iris model has the following schema, which can be set in a model-settings.json file for MLServer:

{
    "name": "iris",
    "implementation": "mlserver_sklearn.SKLearnModel",
    "inputs": [
        {
            "name": "predict",
            "datatype": "FP32",
            "shape": [-1, 4]
        }
    ],
    "outputs": [
        {
            "name": "predict",
            "datatype": "INT64",
            "shape": [-1, 1]
        }
    ],
    "parameters": {
        "version": "1"
    }
}

Examples are given below for some common tools for making requests.

An example seldon request might look like this:

seldon model infer iris \
        '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

The default inference mode is REST, but you can also send gRPC requests like this:

seldon model infer iris \
        --inference-mode grpc \
        '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}'

An example curl request might look like this:

curl -v http://0.0.0.0:9000/v2/models/iris/infer \
        -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

An example grpcurl request might look like this:

grpcurl \
	-d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
	-plaintext \
	-import-path apis \
	-proto apis/mlops/v2_dataplane/v2_dataplane.proto \
	0.0.0.0:9000 inference.GRPCInferenceService/ModelInfer

The above request was run from the project root folder allowing reference to the Protobuf manifests defined in the apis/ folder.

You can use the Python tritonclient package to send inference requests.

A short, self-contained example is:

import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(
    url="localhost:8080",
    verbose=False,
)

inputs = [httpclient.InferInput("predict", (1, 4), "FP64")]
inputs[0].set_data_from_numpy(
    np.array([[1, 2, 3, 4]]).astype("float64"),
    binary_data=False,
)

result = client.infer("iris", inputs)
print("result is:", result.as_numpy("predict"))

Note: For pipelines, a synchronous request is possible if the pipeline has an outputs section defined in its spec.

Request Routing

Seldon Routes

Seldon needs to determine where to route requests to, as models and pipelines might have the same name. There are two ways of doing this: header-based routing (preferred) and path-based routing.

Seldon can route requests to the correct endpoint via headers in HTTP calls, both for REST (HTTP/1.1) and gRPC (HTTP/2).

Use the Seldon-Model header as follows:

For models, use the model name as the value. For example, to send requests to a model named foo use the header Seldon-Model: foo.
For pipelines, use the pipeline name followed by .pipeline as the value. For example, to send requests to a pipeline named foo use the header Seldon-Model: foo.pipeline.

The seldon CLI is aware of these rules and can be used to easily send requests to your deployed resources. See the examples and the Seldon CLI docs for more information.

Extending our examples from above, the requests may look like the below when using header-based routing.

No changes are required as the seldon CLI already understands how to set the appropriate gRPC and REST headers.

:emphasize-lines: 4

curl -v http://0.0.0.0:9000/v2/models/iris/infer \
        -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}' \
        -H "Seldon-Model: iris"

Note the rpc-header flag in the penultimate line:

:emphasize-lines: 6

grpcurl \
	-d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
	-plaintext \
	-import-path apis \
	-proto apis/mlops/v2_dataplane/v2_dataplane.proto \
	-rpc-header seldon-model:iris \
	0.0.0.0:9000 inference.GRPCInferenceService/ModelInfer

Note the headers dictionary in the client.infer() call:

:emphasize-lines: 18

import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(
    url="localhost:8080",
    verbose=False,
)

inputs = [httpclient.InferInput("predict", (1, 4), "FP64")]
inputs[0].set_data_from_numpy(
    np.array([[1, 2, 3, 4]]).astype("float64"),
    binary_data=False,
)

result = client.infer(
    "iris",
    inputs,
    headers={"Seldon-Model": "iris"},
)
print("result is:", result.as_numpy("predict"))

Ingress Routes

If you are using an ingress controller to make inference requests with Seldon, you will need to configure the routing rules correctly.

There are many ways to do this, but custom path prefixes will not work with gRPC. This is because gRPC determines the path based on the Protobuf definition. Some gRPC implementations permit manipulating paths when sending requests, but this is by no means universal.

If you want to expose your inference endpoints via gRPC and REST in a consistent way, you should use virtual hosts, subdomains, or headers.

The downside of using only paths is that you cannot differentiate between different installations of Seldon Core 2 or between traffic to Seldon and any other inference endpoints you may have exposed via the same ingress.

You might want to use a mixture of these methods; the choice is yours.

Virtual hosts are a way of differentiating between logical services accessed via the same physical machine(s).

Virtual hosts are defined by the Host header for HTTP/1 and the :authority pseudo-header for HTTP/2. These represent the same thing, and the HTTP/2 specification defines how to translate these when converting between protocol versions.

Many tools and libraries treat these headers as special and have particular ways of handling them. Some common ones are given below:

The seldon CLI has an --authority flag which applies to both REST and gRPC inference calls.
curl accepts Host as a normal header.
grpcurl has an -authority flag.
In Go, the standard library's http.Request struct has a Host field and ignores attempts to set this value via headers.
In Python, the requests library accepts the host as a normal header.

Be sure to check the documentation for how to set this with your preferred tools and languages.

Subdomain names constitute a part of the overall host name. As such, specifying a subdomain name for requests will involve setting the appropriate host in the URI.

For example, you may expose inference services in the namespaces seldon-1 and seldon-2 as in the following snippets:

curl https://seldon-1.example.com/v2/models/iris/infer ...

seldon model infer --inference-host https://seldon-2.example.com/v2/models/iris/infer ...

Many popular ingresses support subdomain-based routing, including Istio and Nginx. Please refer to the documentation for your ingress of choice for further information.

It is possible to route on paths by using well-known path prefixes defined by the inference v2 protocol. For gRPC, the full path (or "method") for an inference call is:

/inference.GRPCInferenceService/ModelInfer

This corresponds to the package (inference), service (GRPCInferenceService), and RPC name (ModelInfer) in the Protobuf definition of the inference v2 protocol.

You could use an exact match or a regex like .*inference.* to match this path, for example.

Asynchronous Requests

The Seldon architecture uses Kafka and therefore asynchronous requests can be sent by pushing inference v2 protocol payloads to the appropriate topic. Topics have the following form:

seldon.<namespace>.<model|pipeline>.<name>.<inputs|outputs>

Note: If writing to a pipeline topic, you will need to include a Kafka header with the key pipeline and the value being the name of the pipeline.

Model Inference

For a local install if you have a model iris, you would be able to send a prediction request by pushing to the topic: seldon.default.model.iris.inputs. The response will appear on seldon.default.model.iris.outputs.

For a Kubernetes install in seldon-mesh if you have a model iris, you would be able to send a prediction request by pushing to the topic: seldon.seldon-mesh.model.iris.inputs. The response will appear on seldon.seldon-mesh.model.iris.outputs.

Pipeline Inference

For a local install if you have a pipeline mypipeline, you would be able to send a prediction request by pushing to the topic: seldon.default.pipeline.mypipeline.inputs. The response will appear on seldon.default.pipeline.mypipeline.outputs.

For a Kubernetes install in seldon-mesh if you have a pipeline mypipeline, you would be able to send a prediction request by pushing to the topic: seldon.seldon-mesh.pipeline.mypipeline.inputs. The response will appear on seldon.seldon-mesh.pipeline.mypipeline.outputs.

Pipeline Metadata

It may be useful to send metadata alongside your inference.

If using Kafka directly as described above, you can attach Kafka metadata to your request, which will be passed around the graph. When making synchronous requests to your pipeline with REST or gRPC you can also do this.

For REST requests add HTTP headers prefixed with X-
For gRPC requests add metadata with keys starting with X-

You can also do this with the Seldon CLI by setting headers with the --header argument (and also showing response headers with the --show-headers argument)

seldon pipeline infer --show-headers --header X-foo=bar tfsimples \
    '{"inputs":[{"name":"INPUT0","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]}]}'

For pipeline inference, the response also contains a x-pipeline-version header, indicating which version of pipeline it ran inference with.

Request IDs

For both model and pipeline requests the response will contain a x-request-id response header. For pipeline requests this can be used to inspect the pipeline steps via the CLI, e.g.:

seldon pipeline inspect tfsimples --request-id carjjolvqj3j2pfbut10 --offset 10

The --offset parameter specifies how many messages (from the latest) you want to search to find your request. If not specified the last request will be shown.

x-request-id will also appear in tracing spans.

If x-request-id is passed in by the caller then this will be used. It is the caller's responsibility to ensure it is unique.

The IDs generated are XIDs.

PreviousInference NextRun Inference

Last updated 6 months ago

Was this helpful?

hashtagSynchronous Requests

hashtagFind the Seldon Service Endpoint

hashtagMake Inference Requests

hashtagRequest Routing

hashtagSeldon Routes

hashtagIngress Routes

hashtagAsynchronous Requests

hashtagModel Inference

hashtagPipeline Inference

hashtagPipeline Metadata

hashtagRequest IDs