Auto Scaling

Note: Knative requires Istio to be installed. This page is for users installing Knative for the first time.

If you already have Knative, but a newer version is required, please refer to the official documentation for details on upgrading an existing installation instead.

You still need to configure a broker for Seldon Enterprise Platform, regardless of whether you install or upgrade Knative components.

You may also wish to refer to the tests validating installations and the auto-scaling configuration discussed on this page.

This section walks through installation of Knative for Seldon Enterprise Platform.

Knative Eventing and Serving are used for request logging and for post-predict detector components such as outlier, drift, and metrics. For more details, see data events logging.

There are different ways to install Knative. If you have an existing installation, then apply any steps needed to customize it.\

Prerequisites

• Download the installation resources.

• Verify the cluster requirements.

• Install Seldon Enterprise Platform.

• Install Seldon Core 2.

Version requirements

Verify that you meet the version requirements per Knative's documentation for Kubernetes and for Istio for both Eventing and Serving. Here's a quick-reference table:

Knative version	Kubernetes version	Istio version
1.8.x	1.23+	1.15.x -- recommended
1.7.x	1.22+	1.14.x -- recommended
1.6.x	1.22+	1.14.x -- recommended
1.5.x	1.22+	1.13.x -- recommended
1.4.x	1.22+	1.13.x -- recommended
1.3.x	1.21+	1.12.x -- recommended
1.2.x	1.21+	1.12.x -- recommended
1.1.x	1.20+	1.9.x -- recommended
1.0.x	1.20+	1.9.x -- recommended

Install version v1.8.0 of Knative.

Install Knative Serving

Run the following shell commands, changing the version as required:

KNATIVE_SERVING_URL="https://github.com/knative/serving/releases/download"
NET_ISTIO_URL="https://github.com/knative/net-istio/releases/download"
SERVING_VERSION="knative-v1.8.0"

kubectl apply -f ${KNATIVE_SERVING_URL}/${SERVING_VERSION}/serving-crds.yaml
kubectl apply -f ${KNATIVE_SERVING_URL}/${SERVING_VERSION}/serving-core.yaml

kubectl apply -f ${NET_ISTIO_URL}/${SERVING_VERSION}/net-istio.yaml

If you are using Seldon Core Analytics for Prometheus, then for Knative metrics add these annotations:

kubectl annotate -n knative-serving service autoscaler prometheus.io/scrape=true
kubectl annotate -n knative-serving service autoscaler prometheus.io/port=9090

kubectl annotate -n knative-serving service activator-service prometheus.io/scrape=true
kubectl annotate -n knative-serving service activator-service prometheus.io/port=9090

Test Knative Serving

To check the installed version of Knative Serving:

kubectl get namespace knative-serving -o 'go-template={{index .metadata.labels "app.kubernetes.io/version"}}'

Check that the Knative components all have a STATUS of running.

kubectl get pods -n knative-serving

Example output:

NAME                                    READY   STATUS    RESTARTS   AGE
activator-7769fd4b7-qsgz8               1/1     Running   0          5m54s
autoscaler-7f8b5bc69d-ql9kb             1/1     Running   0          5m54s
controller-6568dfc665-bh5wq             1/1     Running   0          5m54s
domain-mapping-59566c8dd4-954wf         1/1     Running   0          5m54s
domainmapping-webhook-7d79fb547-sb9fz   1/1     Running   0          5m54s
net-istio-controller-bc5d4f964-vcl5w    1/1     Running   0          5m29s
net-istio-webhook-5f595d4f6d-f7p58      1/1     Running   0          5m29s
webhook-5ff4dfd6f8-xs65d                1/1     Running   0          5m53s

To verify the install, first create a file containing the below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
          env:
            - name: TARGET
              value: "Go Sample v1"

Next, start watching pods in the default namespace using a different terminal window:

kubectl get pod -n default -w

And apply the file from the first step and curl it with the below:

kubectl apply -f <filename>

kubectl run --quiet=true -it --rm curl --image=radial/busyboxplus:curl --restart=Never -- \
  curl -v -X GET "http://helloworld-go.default.svc.cluster.local"

You should get a successful response and see a pod come up in the default namespace. If you don't, then see the note below on private registries before looking at resources such as the Seldon and Knative Slack channels.

Clean up with:

kubectl delete -f <filename>

Knative Serving auto-scaling and bounds

You can configure upper and lower bounds to control autoscaling behavior with Knative services. Seldon Enterprise Platform configures the outlier, drift detectors, and metrics servers as Knative services. The scaling bounds are automatically set up upon deployment as revision annotations.

It's good practice to control the initial and maximum numbers of replicas that each revision should have, for performance and cost reasons. Knative will attempt to never have more than this number of replicas running or in the process of being created at any one point in time. In the current Seldon Enterprise Platform setup, it is most useful to configure these bounds for outlier detectors, since drift detectors and metrics servers are not auto-scalable.

Such max-scale limits can be set at a global level as per Knative documentation. For example, the following ConfigMap would set auto-scaling upper limits.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  max-scale: "3"
  max-scale-limit: "100"

Install Knative Eventing

Run the following shell commands, changing the version as required:

KNATIVE_EVENTING_URL=https://github.com/knative/eventing/releases/download
EVENTING_VERSION="knative-v1.8.0"

kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing-crds.yaml
kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/eventing-core.yaml

kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/in-memory-channel.yaml
kubectl apply -f ${KNATIVE_EVENTING_URL}/${EVENTING_VERSION}/mt-channel-broker.yaml

You can check the progress of the rollout using the below:

kubectl rollout status -n knative-eventing deployment/imc-controller

Or:

kubectl get namespace knative-eventing -o 'go-template={{index .metadata.labels "app.kubernetes.io/version"}}'

Configure Knative Eventing broker

Define a Knative Event broker to handle the logging by creating a file with the below:

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: default
  namespace: seldon-logs

And apply this:

kubectl apply -f <filename>

Test Knative Eventing

To check the installed version of Knative Eventing:

kubectl get namespace knative-eventing -o 'go-template={{index .metadata.labels "eventing.knative.dev/release"}}'

To test Knative Eventing it is easiest to have Seldon fully installed with request logging and a model running.

Make a prediction to a model following one of the Seldon Core demos. You should see entries under Requests.

If you see entries under requests, you are all good.

If you don't see entries under requests, first find the request logger pod in the seldon-logs namespace. Tail its logs (kubectl logs -n seldon-logs <pod-name> -f) and make a request again. Do you see output?

If this doesn't work then find the pod for the model in your SeldonDeployment. Tail the logs of the seldon-container-engine container and make a prediction again.

If the predictions aren't sending then it could be a problem with the broker URL (executor.requestLogger.defaultEndpoint in helm get values -n seldon-system seldon-core) or the broker (kubectl get broker -n seldon-logs).

If there are no requests and no obvious problems with the broker transmission, then it could be the trigger stage.

First try kubectl get trigger -n seldon-logs to check the trigger status.

If that looks healthy then we need to debug the Knative trigger process.

Do a kubectl apply -f to the default namespace on a file containing the below (or change references to default for a different namespace):

# event-display app deploment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-display
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: &labels
      app: event-display
  template:
    metadata:
      labels: *labels
    spec:
      containers:
        - name: helloworld-python
          image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/event_display
---
# Service that exposes event-display app.
# This will be the subscriber for the Trigger
kind: Service
apiVersion: v1
metadata:
  name: event-display
  namespace: default
spec:
  selector:
    app: event-display
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
# Trigger to send events to service above
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: event-display
  namespace: seldon-logs
spec:
  broker: default
  subscriber:
    uri: http://event-display.default:80

Now find the event-display pod and tail its logs (kubectl get pod -n default and kubectl logs -n default <pod_name>). Make a prediction to a model following one of the Seldon Core demos.

You should see something in the event-display logs -- even an event decoding error message is good.

To eliminate any Seldon components, we can send an event directly to the broker. There is an example in the Knative docs.

What we've done now corresponds to the Knative Eventing hello-world. Nothing at all in the event-display pod means Knative Eventing is not working.

Occasionally you see a RevisionMissing status on the ksvc and a ContainerCreating message on its Revision. If this happens check the Deployment, and if there are no issues then delete and try again.

Hopefully you've got things working before here. If not then check the pods in the knative-eventing namespace. If that doesn't help find the problem, then the Knative Slack and/or Seldon Slack can help with further debugging.

Enable Knative support in Seldon Enterprise Platform

To Enable v1 support in Seldon Enterprise Platform add or change a following variable in install-values.yaml file

seldon:
  knativeEnabled: true

Once you modify your install-values.yaml you need to apply it with

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f install-values.yaml \
    --namespace=seldon-system \
    --version 2.4.0 \
    --install

Knative with a private registry

By default, Knative assumes the image registries used for images will be public. You can follow the official documentation to use a private registry.