Istio

Learn about installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform.

Istio implements the Kubernetes ingress resource to expose a service and make it accessible from outside the cluster. You can install Istio in either a self-hosted Kubernetes cluster or a managed Kubernetes service provided by a cloud provider that is running the Seldon Enterprise Platform.

Prerequisites

• Install Seldon Enterprise Platform.

• Ensure that you install a version of Istio that is compatible with your Kubernetes cluster version. For detailed information on supported versions, refer to the Istio Compatibility Matrix.

Installing Istio ingress controller

Installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform involves these tasks:

1. Install Istio

2. Install Istio Ingress Gateway

3. Expose Seldon mesh service

4. Install Seldon Enterprise Platform with Istio ingress controller

Install Istio

1. Download the Istio installation package for the version you want to use. In the following command replace <version> with the version of Istio that you downloaded:

   curl -L https://istio.io/downloadIstio | sh -
   cd istio-<version>
   export PATH=$PWD/bin:$PATH

2. Install the Istio Custom Resource Definitions (CRDs) and Istio components in your cluster using the istioctl command line tool:

   istioctl install --set profile=default -y

3. Create a namespace where you want to enable Istio automatic sidecar injection. For example in the namespace istio-system:

   kubectl label namespace istio-system istio-injection=enabled

Install Istio Ingress Gateway

1. Verify that Istio Ingress Gateway is installed:

   kubectl get svc istio-ingressgateway -n istio-system

   This should return details of the Istio Ingress Gateway, including the external IP address.

2. Create a YAML file to specify Gateway resource in the istio-system namespace to expose your application. For example, create the istio-seldon-gateway.yaml file. Use your preferred text editor to create and save the file with the following content:

    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: my-gateway
      namespace: istio-system
    spec:
      selector:
        istio: ingressgateway # Use Istio's default ingress gateway
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:
        - "*"

3. Change to the directory that contains istio-seldon-gateway.yaml file and apply the configuration:

   kubectl apply -f istio-seldon-gateway.yaml

   When the configuration is applied, you should see this:

   gateway.networking.istio.io/seldon-gateway created

4. Find the IP address of the Seldon Enterprise Platform instance running with Istio:

   ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   
   echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/"
   

   Make a note of the IP address that is displayed in the output.

Expose Seldon mesh service

It is important to expose seldon-service service to enable communication between deployed machine learning models and external clients or services. The Seldon Core 2 inference API is exposed through the seldon-mesh service in the seldon namespace. If you install Core 2 in multiple namespaces, you need to expose the seldon-mesh service in each of namespace.

1. Verify if the seldon-mesh service is running for example, in the namespace seldon.

   kubectl get svc -n seldon

   When the services are running you should see something similar to this:

   mlserver-0               ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP                                                                  43m
   seldon-mesh              LoadBalancer   34.118.225.130   34.90.213.15    80:32228/TCP,9003:31265/TCP                                                                 45m
   seldon-pipelinegateway   ClusterIP      None             <none>          9010/TCP,9011/TCP                                                                           45m
   seldon-scheduler         LoadBalancer   34.118.225.138   35.204.34.162   9002:32099/TCP,9004:32100/TCP,9044:30342/TCP,9005:30473/TCP,9055:32732/TCP,9008:32716/TCP   45m
   triton-0                 ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP 

2. Create a YAML file to create a VirtualService Seldon Core 2 seldon-mesh. For example, create the seldon-mesh-vs.yaml file. Use your preferred text editor to create and save the file with the following content:

   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: seldon-mesh
     namespace: seldon
   spec:
     gateways:
       - istio-system/seldon-gateway
     hosts:
       - "*"
     http:
       - name: "data-plane-seldon"
         match:
           - authority:
               exact: "seldon.inference.seldon"
         route:
           - destination:
               host: "seldon-mesh.seldon.svc.cluster.local"
               port:
                number: 80
     - name: "control-plane-seldon"
       match:
         - authority:
             exact: "seldon.admin.seldon"
       route:
         - destination:
             host: "seldon-scheduler.seldon.svc.cluster.local"
             port:
               number: 9004

3. Create a virtual service to expose the seldon-mesh service.

   kubectl apply -f seldon-mesh-vs.yaml

   When the virtual service is created, you should see this:

   virtualservice.networking.istio.io/seldon-mesh created

Install Seldon Enterprise Platform with Istio ingress controller

1. Update the configurations in the install-values.yaml file you created during the Seldon Enterprise installation. Replace <ip_address> with the IP address noted during the Istio Ingress Gateway installation in the following values and save the file:

   ingressGateway:
     seldonIngressService: "istio-ingressgateway"
     ingressNamespace: "istio-system"
   
   virtualService:
     create: true
     gateways:
       - istio-system/seldon-gateway
   
   seldon:
     curlForm: |
       curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/api/v0.1/predictions \<br/>
       &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
       &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
       &nbsp;&nbsp;-d '{{ .Payload }}'
     tensorFlowCurlForm: |
       curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/v1/models/:predict \<br/>
       &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
       &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
       &nbsp;&nbsp;-d '{{ .Payload }}'
   
   seldonCoreV2:
     curlForm: |
       curl -k https://<ip_address>/v2/models/{{ .ModelName }}/infer \<br/>
       &nbsp;&nbsp;-H "Host: {{ .Namespace }}.inference.seldon" \<br/>
       &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
       &nbsp;&nbsp;-H "Seldon-Model: {{ .ModelName }}.pipeline" \<br/>
       &nbsp;&nbsp;-d '{{ .Payload }}'
     enabled: true
     requestForm: '{{ .SeldonProtocol }}://seldon-mesh.{{ .Namespace }}.svc.cluster.local/v2/pipelines/{{
       .ModelName }}/infer'
   

2. Change to the directory that contains the install-values.yaml file and then upgrade the Seldon Enterprise Platform installation in the namespace seldon-system.

   helm upgrade seldon-enterprise seldon-charts/seldon-deploy --namespace seldon-system  -f install-values.yaml --version 2.4.0 --install

3. Check the status of the installation seldon-enterprise-seldon-deploy.

   kubectl rollout status deployment/seldon-enterprise-seldon-deploy -n seldon-system

   When the installation is complete you should see this:

   deployment "seldon-enterprise-seldon-deploy" successfully rolled out

4. Access Seldon Enterprise Platform.

Port forwarding

1. Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.

   export POD_NAME=$(kubectl get pods --namespace seldon-system -l "app.kubernetes.io/name=seldon-deploy,app.kubernetes.io/instance=seldon-enterprise" -o jsonpath="{.items[0].metadata.name}")

2. You can use port-forwarding to access your application locally.

   kubectl port-forward $POD_NAME 8000:8000 --namespace seldon-system

3. Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/ to access Seldon Enterprise Platform.

Static IP address

Replace <ip_address> with the IP address noted during the Istio Ingress Gateway installation. 1. Open your browser and navigate to http://<ip_address>/seldon-deploy/ to access Seldon Enterprise Platform.

Optional: Enable HTTPS/TLS

To secure your Ingress with HTTPS, you can configure TLS settings in the Gateway resource using a certificate and key. This involves additional steps like creating Kubernetes secrets for your certificates.

Additional Resources

• Istio Documentation

• GKE Ingress Guide

• AWS Documentation