Edit

Istio

Learn how to integrate Seldon Core 2 with Istio service mesh for traffic management and ML model deployment, including gateway configuration and virtual service setup.

Istio provides a service mesh and ingress solution.

We will run through some examples as shown in the notebook service-meshes/istio/istio.ipynb in our repo.

Single Model

  • A Seldon Iris Model

  • An istio Gateway

  • An instio VirtualService to expose REST and gRPC

# service-meshes/istio/static/single-model.yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: seldon-mesh
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: iris-route
  namespace: seldon-mesh
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /v2
    name: iris-http
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris
  - match:
    - uri:
        prefix: /inference.GRPCInferenceService
    name: iris-grpc
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris

Traffic Split

  • Two Iris Models

  • An istio Gateway

  • An istio VirtualService with traffic split

Istio Notebook Examples

Assumes

  • You have installed istio as per their docs

  • You have exposed the ingressgateway as an external loadbalancer

tested with:

Istio Single Model Example

Traffic Split Two Models

PreviousAmbassadorNextTraefik

Last updated

Was this helpful?