Istio

Learn about installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform.

Istio implements the Kubernetes ingress resource to expose a service and make it accessible from outside the cluster. You can install Istio in either a self-hosted Kubernetes cluster or a managed Kubernetes service provided by a cloud provider that is running the Seldon Enterprise Platform.

Prerequisites

Install Seldon Enterprise Platform.
Ensure that you install a version of Istio that is compatible with your Kubernetes cluster version. For detailed information on supported versions, refer to the Istio Compatibility Matrix.

Installing Istio ingress controller

Installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform involves these tasks:

Install Istio

Download the Istio installation package for the version you want to use. In the following command replace <version> with the version of Istio that you downloaded:
```
curl -L https://istio.io/downloadIstio | sh -
cd istio-<version>
export PATH=$PWD/bin:$PATH
```
Install the Istio Custom Resource Definitions (CRDs) and Istio components in your cluster using the istioctl command line tool:
```
istioctl install --set profile=default -y
```
Create a namespace where you want to enable Istio automatic sidecar injection. For example in the namespace istio-system:
```
kubectl label namespace istio-system istio-injection=enabled
```

Install Istio Ingress Gateway

Verify that Istio Ingress Gateway is installed:
```
kubectl get svc istio-ingressgateway -n istio-system
```
This should return details of the Istio Ingress Gateway, including the external IP address.

Create a YAML file to specify Gateway resource in the istio-system namespace to expose your application. For example, create the istio-seldon-gateway.yaml file. Use your preferred text editor to create and save the file with the following content:

 apiVersion: networking.istio.io/v1alpha3
 kind: Gateway
 metadata:
   name: my-gateway
   namespace: istio-system
 spec:
   selector:
     istio: ingressgateway # Use Istio's default ingress gateway
   servers:
   - port:
       number: 80
       name: http
       protocol: HTTP
     hosts:
     - "*"

Change to the directory that contains istio-seldon-gateway.yaml file and apply the configuration:
```
kubectl apply -f istio-seldon-gateway.yaml
```
When the configuration is applied, you should see this:
```
gateway.networking.istio.io/seldon-gateway created
```

Find the IP address of the Seldon Enterprise Platform instance running with Istio:

ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/"

Make a note of the IP address that is displayed in the output.

Expose Seldon mesh service

It is important to expose seldon-service service to enable communication between deployed machine learning models and external clients or services. The Seldon Core 2 inference API is exposed through the seldon-mesh service in the seldon namespace. If you install Core 2 in multiple namespaces, you need to expose the seldon-mesh service in each of namespace.

Verify if the seldon-mesh service is running for example, in the namespace seldon.

kubectl get svc -n seldon

When the services are running you should see something similar to this:

mlserver-0               ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP                                                                  43m
seldon-mesh              LoadBalancer   34.118.225.130   34.90.213.15    80:32228/TCP,9003:31265/TCP                                                                 45m
seldon-pipelinegateway   ClusterIP      None             <none>          9010/TCP,9011/TCP                                                                           45m
seldon-scheduler         LoadBalancer   34.118.225.138   35.204.34.162   9002:32099/TCP,9004:32100/TCP,9044:30342/TCP,9005:30473/TCP,9055:32732/TCP,9008:32716/TCP   45m
triton-0                 ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP

Create a YAML file to create a VirtualService Seldon Core 2 seldon-mesh. For example, create the seldon-mesh-vs.yaml file. Use your preferred text editor to create and save the file with the following content:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: seldon-mesh
  namespace: seldon
spec:
  gateways:
    - istio-system/seldon-gateway
  hosts:
    - "*"
  http:
    - name: "data-plane-seldon"
      match:
        - authority:
            exact: "seldon.inference.seldon"
      route:
        - destination:
            host: "seldon-mesh.seldon.svc.cluster.local"
            port:
             number: 80
  - name: "control-plane-seldon"
    match:
      - authority:
          exact: "seldon.admin.seldon"
    route:
      - destination:
          host: "seldon-scheduler.seldon.svc.cluster.local"
          port:
            number: 9004

Create a virtual service to expose the seldon-mesh service.
```
kubectl apply -f seldon-mesh-vs.yaml
```
When the virtual service is created, you should see this:
```
virtualservice.networking.istio.io/seldon-mesh created
```

Install Seldon Enterprise Platform with Istio ingress controller

Update the configurations in the install-values.yaml file you created during the Seldon Enterprise installation. Replace <ip_address> with the IP address noted during the Istio Ingress Gateway installation in the following values and save the file:

ingressGateway:
  seldonIngressService: "istio-ingressgateway"
  ingressNamespace: "istio-system"

virtualService:
  create: true
  gateways:
    - istio-system/seldon-gateway

seldon:
  curlForm: |
    curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/api/v0.1/predictions \<br/>
    &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'
  tensorFlowCurlForm: |
    curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/v1/models/:predict \<br/>
    &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'

seldonCoreV2:
  curlForm: |
    curl -k https://<ip_address>/v2/models/{{ .ModelName }}/infer \<br/>
    &nbsp;&nbsp;-H "Host: {{ .Namespace }}.inference.seldon" \<br/>
    &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
    &nbsp;&nbsp;-H "Seldon-Model: {{ .ModelName }}.pipeline" \<br/>
    &nbsp;&nbsp;-d '{{ .Payload }}'
  enabled: true
  requestForm: '{{ .SeldonProtocol }}://seldon-mesh.{{ .Namespace }}.svc.cluster.local/v2/pipelines/{{
    .ModelName }}/infer'

Change to the directory that contains the install-values.yaml file and then upgrade the Seldon Enterprise Platform installation in the namespace seldon-system.
```
helm upgrade seldon-enterprise seldon-charts/seldon-deploy --namespace seldon-system  -f install-values.yaml --version 2.4.0 --install
```

Check the status of the installation seldon-enterprise-seldon-deploy.

kubectl rollout status deployment/seldon-enterprise-seldon-deploy -n seldon-system

When the installation is complete you should see this:

deployment "seldon-enterprise-seldon-deploy" successfully rolled out

Access Seldon Enterprise Platform.

Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.

export POD_NAME=$(kubectl get pods --namespace seldon-system -l "app.kubernetes.io/name=seldon-deploy,app.kubernetes.io/instance=seldon-enterprise" -o jsonpath="{.items[0].metadata.name}")

You can use port-forwarding to access your application locally.
```
kubectl port-forward $POD_NAME 8000:8000 --namespace seldon-system
```
Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/ to access Seldon Enterprise Platform.

Replace <ip_address> with the IP address noted during the Istio Ingress Gateway installation. 1. Open your browser and navigate to http://<ip_address>/seldon-deploy/ to access Seldon Enterprise Platform.

Optional: Enable HTTPS/TLS

To secure your Ingress with HTTPS, you can configure TLS settings in the Gateway resource using a certificate and key. This involves additional steps like creating Kubernetes secrets for your certificates.

Additional Resources

PreviousIngress Controller NextModel Catalog

Last updated 7 months ago

Was this helpful?