All pages
Powered by GitBook
1 of 2

Loading...

Loading...

Ingress Controller

Learn more about the Ingress controllers that Seldon Enterprise Platform supports.

An ingress controller functions as a reverse proxy and load balancer, implementing a Kubernetes Ingress. It adds an abstraction layer for traffic routing by receiving traffic from outside the Kubernetes platform and load balancing it to Pods running within the Kubernetes cluster.

Note: Seldon Enterprise Platform does not support NGINX as an alternative for Istio.

The Seldon Enterprise Platform supports Istio ingress controllers. Seldon Core 1 and Seldon Core 2 integrate most effectively with an Istio ingress controller in your Seldon Enterprise Platform ecosystem.

This table lists the features and the ingress controllers for Seldon Core 1 and Seldon Core 2.

Seldon Core 1
Seldon Core 2
  • Service mesh with advanced traffic management capabilities.

  • Required for effective use of Seldon Core 1.

  • Automatic configuration of ingress routes for Seldon Core 1 deployments.

  • Enables authorization of Seldon Core 1 deployments.

  • Manages traffic for rollout strategies, drift and outlier detection, and inference data logging for Seldon Core 1.

  • Supports both HTTP/1 and HTTP/2 (gRPC) traffic.

  • Provides TLS support for HTTPS and gRPCS.

  • Supports ingress routes for ML deployments.

  • The features of Seldon Core 1 does not function without the Istio ingress controller.

    Seldon Core 2 is by design service mesh-agnostic and as a result you can choose any ingress controller you want. However, If you choose a non-Istio ingress controller it may restrict you to the more basic or limited functionality of Seldon Core 2.

    Seldon Core uses Istio because it relies on a service mesh for traffic splitting and leverages Knative for drift and outlier detection, as well as inference event logging.

    Seldon Core 2 offers features such as multi-model serving and overcommit functionality and uses Kafka as a message queue for data-centric pipelines. These features work best with Istio ingress controller.

    Features of Istio ingress controller

    Istio

    Learn about installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform.

    Istio implements the Kubernetes ingress resource to expose a service and make it accessible from outside the cluster. You can install Istio in either a self-hosted Kubernetes cluster or a managed Kubernetes service provided by a cloud provider that is running the Seldon Enterprise Platform.

    Prerequisites

    • Install Seldon Enterprise Platform.

    • Ensure that you install a version of Istio that is compatible with your Kubernetes cluster version. For detailed information on supported versions, refer to the Istio Compatibility Matrix.

    Installing Istio ingress controller

    Installing Istio ingress controller in a Kubernetes cluster running Seldon Enterprise Platform involves these tasks:

    1. Download the Istio installation package for the version you want to use. In the following command replace <version> with the version of Istio that you downloaded:

    2. Install the Istio Custom Resource Definitions (CRDs) and Istio components in your cluster using the istioctl command line tool:

    3. Create a namespace where you want to enable Istio automatic sidecar injection. For example in the namespace istio-system

    1. Verify that Istio Ingress Gateway is installed:

      This should return details of the Istio Ingress Gateway, including the external IP address.

    2. Create a YAML file to specify Gateway resource in the istio-system namespace to expose your application. For example, create the istio-seldon-gateway.yaml file. Use your preferred text editor to create and save the file with the following content:

    It is important to expose seldon-service service to enable communication between deployed machine learning models and external clients or services. The Seldon Core 2 inference API is exposed through the seldon-mesh service in the seldon namespace. If you install Core 2 in multiple namespaces, you need to expose the seldon-mesh service in each of namespace.

    1. Verify if the seldon-mesh service is running for example, in the namespace seldon.

      When the services are running you should see something similar to this:

    2. Create a YAML file to create a VirtualService Seldon Core 2 seldon-mesh. For example, create the seldon-mesh-vs.yaml file. Use your preferred text editor to create and save the file with the following content:

    1. Update the configurations in the install-values.yaml file you created during the Seldon Enterprise installation. Replace <ip_address> with the IP address noted during the in the following values and save the file:

    2. Change to the directory that contains the install-values.yaml file and then upgrade the Seldon Enterprise Platform installation in the namespace seldon-system.

    1. Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.

    2. You can use port-forwarding to access your application locally.

    3. Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/ to access Seldon Enterprise Platform.

    To secure your Ingress with HTTPS, you can configure TLS settings in the Gateway resource using a certificate and key. This involves additional steps like creating Kubernetes secrets for your certificates.

    :
    Change to the directory that contains istio-seldon-gateway.yaml file and apply the configuration:

    When the configuration is applied, you should see this:

  • Find the IP address of the Seldon Enterprise Platform instance running with Istio:

    Make a note of the IP address that is displayed in the output.

  • Create a virtual service to expose the seldon-mesh service.

    When the virtual service is created, you should see this:

    Check the status of the installation seldon-enterprise-seldon-deploy.

    When the installation is complete you should see this:

  • Access Seldon Enterprise Platform.

  • Replace <ip_address> with the IP address noted during the Istio Ingress Gateway installation. 1. Open your browser and navigate to http://<ip_address>/seldon-deploy/ to access Seldon Enterprise Platform.

    Install Istio

    Install Istio Ingress Gateway

    Expose Seldon mesh service

    Install Seldon Enterprise Platform with Istio ingress controller

    Optional: Enable HTTPS/TLS

    Additional Resources

    Install Istio
    Install Istio Ingress Gateway
    Expose Seldon mesh service
    Install Seldon Enterprise Platform with Istio ingress controller
    Istio Ingress Gateway installation
    Istio Documentation
    GKE Ingress Guide
    AWS Documentation
    ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
    ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
    
    echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/"
    
    kubectl apply -f seldon-mesh-vs.yaml
    virtualservice.networking.istio.io/seldon-mesh created
    kubectl rollout status deployment/seldon-enterprise-seldon-deploy -n seldon-system
    deployment "seldon-enterprise-seldon-deploy" successfully rolled out
    curl -L https://istio.io/downloadIstio | sh -
    cd istio-<version>
    export PATH=$PWD/bin:$PATH
    istioctl install --set profile=default -y
    kubectl get svc istio-ingressgateway -n istio-system
     apiVersion: networking.istio.io/v1alpha3
     kind: Gateway
     metadata:
       name: my-gateway
       namespace: istio-system
     spec:
       selector:
         istio: ingressgateway # Use Istio's default ingress gateway
       servers:
       - port:
           number: 80
           name: http
           protocol: HTTP
         hosts:
         - "*"
    kubectl get svc -n seldon
    mlserver-0               ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP                                                                  43m
    seldon-mesh              LoadBalancer   34.118.225.130   34.90.213.15    80:32228/TCP,9003:31265/TCP                                                                 45m
    seldon-pipelinegateway   ClusterIP      None             <none>          9010/TCP,9011/TCP                                                                           45m
    seldon-scheduler         LoadBalancer   34.118.225.138   35.204.34.162   9002:32099/TCP,9004:32100/TCP,9044:30342/TCP,9005:30473/TCP,9055:32732/TCP,9008:32716/TCP   45m
    triton-0                 ClusterIP      None             <none>          9000/TCP,9500/TCP,9005/TCP 
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: seldon-mesh
      namespace: seldon
    spec:
      gateways:
        - istio-system/seldon-gateway
      hosts:
        - "*"
      http:
        - name: "data-plane-seldon"
          match:
            - authority:
                exact: "seldon.inference.seldon"
          route:
            - destination:
                host: "seldon-mesh.seldon.svc.cluster.local"
                port:
                 number: 80
      - name: "control-plane-seldon"
        match:
          - authority:
              exact: "seldon.admin.seldon"
        route:
          - destination:
              host: "seldon-scheduler.seldon.svc.cluster.local"
              port:
                number: 9004
    ingressGateway:
      seldonIngressService: "istio-ingressgateway"
      ingressNamespace: "istio-system"
    
    virtualService:
      create: true
      gateways:
        - istio-system/seldon-gateway
    
    seldon:
      curlForm: |
        curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/api/v0.1/predictions \<br/>
        &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
        &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
        &nbsp;&nbsp;-d '{{ .Payload }}'
      tensorFlowCurlForm: |
        curl -k https://<ip_address>/seldon/{{ .Namespace }}/{{ .ModelName }}/v1/models/:predict \<br/>
        &nbsp;&nbsp;-H "{{ .TokenHeader }}: {{ .Token }}" \<br/>
        &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
        &nbsp;&nbsp;-d '{{ .Payload }}'
    
    seldonCoreV2:
      curlForm: |
        curl -k https://<ip_address>/v2/models/{{ .ModelName }}/infer \<br/>
        &nbsp;&nbsp;-H "Host: {{ .Namespace }}.inference.seldon" \<br/>
        &nbsp;&nbsp;-H "Content-Type: application/json" \<br/>
        &nbsp;&nbsp;-H "Seldon-Model: {{ .ModelName }}.pipeline" \<br/>
        &nbsp;&nbsp;-d '{{ .Payload }}'
      enabled: true
      requestForm: '{{ .SeldonProtocol }}://seldon-mesh.{{ .Namespace }}.svc.cluster.local/v2/pipelines/{{
        .ModelName }}/infer'
    
    helm upgrade seldon-enterprise seldon-charts/seldon-deploy --namespace seldon-system  -f install-values.yaml --version 2.4.0 --install
    export POD_NAME=$(kubectl get pods --namespace seldon-system -l "app.kubernetes.io/name=seldon-deploy,app.kubernetes.io/instance=seldon-enterprise" -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward $POD_NAME 8000:8000 --namespace seldon-system
    kubectl label namespace istio-system istio-injection=enabled
    kubectl apply -f istio-seldon-gateway.yaml
    gateway.networking.istio.io/seldon-gateway created