Installation

This section provides instructions on how to install the Model Performance Metrics module on a Kubernetes cluster. It is deployed using a Helm chart, which is stored in a private Google Artifact Registry, managed by Seldon. The Helm chart makes use of a Docker image, which is also stored in the same registry.

Prerequisites

This guide assumes you have received Seldon Artifact Registry access credentials in a JSON format. If this is not the case, please reach out to us.

Before installing the module, ensure you have the following:

• Access to the private Google Artifact Registry where the Helm chart and the Docker image are stored.

• Kubernetes cluster configured and running.

• Seldon Core 2 installed and running on the Kubernetes cluster.

• Kafka installed, configured with Seldon Core 2 and running on the Kubernetes cluster or reachable from it.

• PostgreSQL installed and running on the Kubernetes cluster reachable from it.

• Istio installed and running on the Kubernetes cluster (Optional). Only needed if you wish to make the module externally accessible from the cluster. Requires an Istio Gateway to be set up as well.

Components compatibility matrix

Component	Supported Versions	Importance
Kubernetes	minor versions of 1.27 or later	Required for deploying the Helm chart
Seldon Core 2	minor versions of 2.8 or later	Required for producing inference response events to Kafka
Kafka	minor versions of 3.3.1 or later	Required for Kafka Consumer, consuming inference responses produced by Seldon Core 2
PostgreSQL	version 10 - version 15	Required for storing metrics data, inference responses and more
Istio	minor versions of 1.17 or later	Optional for exposing the Model Performance Metrics module externally from the cluster

Required CLIs

Ensure you have the following CLIs installed on your local machine:

• kubectl

• docker

• Helm

• gcloud

Accessing the artifacts

Note

In the following examples, we assume you have received the credentials in a JSON format and stored them in a file named credentials.json.

Authenticate to the Google Artifact Registry

To be able to pull the Helm chart and the Docker image from the Google Artifact Registry, you need to authenticate with the Docker CLI first.

REGISTRY=europe-west2-docker.pkg.dev
cat credentials.json | docker login -u _json_key --password-stdin ${REGISTRY}

The result of this command should be a successful login to the Google Artifact Registry:

Login Succeeded

To pull the Docker image, you can use the following command:

docker pull europe-west2-docker.pkg.dev/seldon-registry/metrics-server/metrics-server:0.1.0

The result of this command should be a successful pull of the Docker image.

To be able to pull the Helm chart from the Google Artifact Registry, you can use the following command:

helm pull oci://europe-west2-docker.pkg.dev/seldon-registry/charts/metrics-server --version 0.1.0 --untar

This command will pull the Helm chart from the Google Artifact Registry and extract it in the current directory. You can then navigate to the directory and inspect the Helm chart.

Note

If you prefer to pull the Helm chart in a tar format, you can remove the --untar flag from the command.

Helm Chart Components

The Helm chart consists of the following components:

• Chart.yaml: Contains the metadata of the Helm chart.

• templates/: Contains the Kubernetes resources to be deployed on the cluster.

  • deployment.yaml: Contains the deployment configuration for the service.

  • service.yaml: Contains the service configuration for the service.

  • virtualservice.yaml: Contains the virtual service(Istio) configuration for the service (optional).

• values.yaml: Contains the default values for the Helm chart.

Configuring the Helm Chart

Helm Chart Values

After downloading the Helm chart, you can inspect the values.yaml file to see the default values for the Helm chart. There are values that must be set to install the Chart, values that must be set for the module to initialise successfully and values that are optional. The following table lists all the possible values, their description and their importance.

Chart

Helm Value Key	Description	Importance
appName	The name of the application. This value is used to label the resources created by the Helm chart.	Required to install chart
namespace	The namespace where the Deployment, the Service and, optionally, the VirtualService will be deployed.	Required to install chart
image	The Docker image to be used by the service. This value should be the full path to the image in the Google Artifact Registry, excluding the tag.	Required to install chart
imageTag	The Docker image version tag. We use Semantic Versioning for tagging the Docker image.	Required to install chart
imagePullSecretName	The name of the Kubernetes secret that contains the credentials to pull the Docker image from the Google Artifact Registry.	Required to install chart

General

Helm Value Key	Description	Importance
logLevel	The log level for the module. The value is case-insensitive and can be either one of: "disabled", "trace", "debug", "info", "warn", "error", "fatal", "panic". Defaults to "info".	Optional for the module

Virtual Service

Helm Value Key	Description	Importance
istioVirtualService.create	Whether or not to create an Istio VirtualService. Defaults to false.	Optional to create the VirtualService
istioVirtualService.gateway	The gateway name for the Istio VirtualService. Must be supplied if istioVirtualService.create is true. An example default value could be istio-system/seldon-gateway.	Optional to create the VirtualService
istioVirtualService.path	The prefix path which follows the external load balancer IP and before the Model Performance Metrics module's endpoints. An example value could be /metrics-server/.	Optional to create the VirtualService

Kafka Metadata

Helm Value Key	Description	Importance
kafka.metadata.retryMax	The total number of times to retry a metadata request when the cluster is in the middle of a leader election. Defaults to 3.	Optional for the module
kafka.metadata.retryBackoff	How long to wait for leader election to occur before retrying. Defaults to 250ms.	Optional for the module
kafka.metadata.refreshFrequency	How frequently to refresh the cluster metadata in the background. Defaults to 10s.	Optional for the module
kafka.metadata.full	Whether or not to maintain a full set of metadata for all topics, or just the minimal set that has been necessary so far. Defaults to true.	Optional for the module
kafka.metadata.allowAutoTopicCreation	Whether or not to allow auto-create topics in metadata refresh. For production environments, it is recommended to disable automatic topic creation. Defaults to true.	Optional for the module

Kafka Auth

Helm Value Key	Description	Importance
kafka.auth.sasl.enabled	Whether or not to enable SASL_SSL(with SCRAM) authentication. Defaults to false.	Optional to authenticate securely
kafka.auth.sasl.mechanism	The SASL mechanism to use. Accepted values are "SCRAM-SHA-512" and "SCRAM-SHA-256". Defaults to "SCRAM-SHA-512".	Optional to authenticate securely
kafka.auth.sasl.username	The username to use for SASL authentication. Defaults to "seldon".	Optional to authenticate securely
kafka.auth.sasl.passwordSecretName	The name of the secret containing the password for SASL authentication. Defaults to "kafka-client-auth".	Optional to authenticate securely
kafka.auth.sasl.passwordSecretKey	The key in the secret containing the password for SASL authentication. Defaults to "password".	Optional to authenticate securely
kafka.auth.sasl.caCertLocation	The location of the CA certificate file. Defaults to "/tmp/certs/kafka/broker/ca.crt".	Optional to authenticate securely
kafka.auth.sasl.caCertSecretName	The name of the secret containing the CA certificate. Defaults to "kafka-broker-auth".	Optional to authenticate securely
kafka.auth.sasl.caCertSecretKey	The key in the secret containing the CA certificate. Defaults to "ca.crt".	Optional to authenticate securely

Kafka Consumer

Helm Value Key	Description	Importance
kafka.consumer.bootstrapServer	The Kafka bootstrap server address, including a port. This value is used by the Kafka consumer to connect to the Kafka broker. Defaults to "seldon-kafka-bootstrap.kafka.svc.cluster.local:9092".	Required to run the module
kafka.consumer.topicPrefix	The prefix of the topics MPM will attempt to start consuming messages from. Defaults to seldon.	Optional for the module
kafka.consumer.pollInterval	How often the Kafka consumer within MPM will check if there are new topics matching a regex and start consuming messages from them. Defaults to 10s.	Optional for the module
kafka.consumer.offsets.autoCommitEnable	Whether or not to auto-commit updated offsets back to the broker. Defaults to true.	Optional for the module
kafka.consumer.offsets.autoCommitInterval	How frequently to commit updated offsets. Ineffective unless auto-commit is enabled. Defaults to 1s.	Optional for the module
kafka.consumer.offsets.initial	The initial offset to use if no offset was previously committed. Accepted values are "oldest" and "newest". Defaults to "newest".	Optional for the module

PostgreSQL

Helm Value Key	Description	Importance
postgres.host	The host of the PostgreSQL database, excluding the port number. Defaults to "localhost".	Required to run the module
postgres.port	The port of the PostgreSQL database. Defaults to 5432.	Required to run the module
postgres.databaseName	The name of the database to use. Defaults to "seldon_metrics".	Required to run the module
postgres.user	The username to use for authentication. Defaults to "seldon".	Required to run the module
postgres.secretName	The name of the secret containing the password. Defaults to "password".	Required to run the module
postgres.secretKey	The key in the secret containing the password. Defaults to "password".	Required to run the module
postgres.sslmode	Controls the encryption and verification of PostgreSQL connections. Defaults to "prefer". Available options can be found here, except verify-ca and verify-full.	Optional for the module

An example values.yaml file with the minimal values required to install the Helm chart is shown below:

appName: metrics-server
namespace: seldon-system
image: europe-west2-docker.pkg.dev/seldon-registry/metrics-server/metrics-server
imageTag: 0.1.0
imagePullSecretName: seldon-registry

kafka:
  consumer:
    bootstrapServer: "localhost:9092"

postgres:
  host: "localhost"
  port: 5432
  databaseName: "seldon_metrics"
  user: "seldon"
  secretName: "seldon-metrics-postgres"
  secretKey: "password"

Notes

• When kafka.consumer.bootstrapServer is provided and kafka.auth.sasl.enabled is set to false, the module will attempt to establish a connection to the Kafka broker using the PLAINTEXT protocol.

• When kafka.auth.sasl.enabled is set to true, the module will attempt to establish a connection to the Kafka broker using the SASL_SSL protocol. In such cases, the port of the Kafka bootstrap server could be different. For example, 9093 instead of 9092.

• The Kafka consumer only supports a single replicate of the module's Deployment definition which is reflected in its spec object: replica:1.

Creating a Kubernetes Secret (Optional)

After you have successfully pulled the Docker image and the Helm chart, and you have configured the Helm values to suit your environment, optionally you may have to create a Kubernetes secret so that the Helm Deployment can pull the Docker image from the Google Artifact Registry.

To create a Kubernetes secret with the credentials, you can use the following command:

NAMESPACE=seldon-system
REGISTRY=europe-west2-docker.pkg.dev
CREDENTIALS=$(cat credentials.json)
SECRET_NAME=seldon-registry

kubectl create secret docker-registry ${SECRET_NAME} \
	--docker-server="${REGISTRY}" \
	--docker-username="_json_key" \
	--docker-password="${CREDENTIALS}" \
	--dry-run=client -o yaml | kubectl apply -n ${NAMESPACE} -f -

You can verify the secret was created successfully by running:

kubectl get secret ${SECRET_NAME} -n ${NAMESPACE}

Installing the Helm Chart

After you have configured the Helm chart values and created the Kubernetes secret, you can install the Helm chart on the Kubernetes cluster. To do so, using the modified values.yaml, you can use the following command:

helm install metrics-server ./metrics-server

To uninstall the Helm chart, you can use the following command:

helm uninstall metrics-server

Ingress Controller (Optional)

An ingress controller functions as a reverse proxy and load balancer, implementing a Kubernetes Ingress. It adds an abstraction layer for traffic routing by receiving traffic from outside the Kubernetes platform and load balancing it to Pods running within the Kubernetes cluster.

The Model Performance Metrics will work with any service mesh or ingress controller, offering flexibility in your deployment setup. In this example, we are going to use Istio to expose the module and make it accessible from outside the Kubernetes cluster.

Install Istio Base Component

1. Add the Istio Helm charts repository and update it:

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

1. Create the istio-system namespace where Istio components are installed:

kubectl create namespace istio-system

1. Install the base component:

helm install istio-base istio/base -n istio-system

1. Install Istiod, the Istio control plane:

helm install istiod istio/istiod -n istio-system --wait

Install Istio Ingress Gateway deployment

1. Install the Istio Ingress Gateway deployment:

helm install istio-ingressgateway istio/gateway -n istio-system

1. Verify it is installed:

kubectl get svc istio-ingressgateway -n istio-system

This should return details of the Istio Ingress Gateway pod of type LoadBalancer, including the external IP address.

Notes

If your Istio Ingress Gateway pod shows EXTERNAL-IP: , it means that Kubernetes is still waiting to assign an external IP address to the Istio Ingress Gateway LoadBalancer service.

When using a cloud provider:

• If the cloud provider of choice supports automatic LoadBalancer provisioning, Kubernetes should create one and assign an external IP after a few minutes.

• If the cloud provider of choice does not support automatic LoadBalancer provisioning, either you have to manually reserve and assign an external IP or manually create one. This will depend on the cloud provider's documentation.

When using a local Kubernetes cluster:

• In a local Kubernetes cluster(e.g. Minikube, Kind), there is no external cloud provider to provision the IP automatically.

• You can install MetalLB to provide a load balancer implementation for bare metal Kubernetes clusters.

Install MetalLB in a local Kubernetes cluster (Optional)

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/config/manifests/metallb-native.yaml

This will install MetalLB in the metallb-system namespace.

• Determine the IPv4 subnet of the corresponding docker network(assuming the cluster is running in Docker with kind):

docker network inspect kind | jq -r '.[0].IPAM.Config[0].Subnet'

An example output could be 172.18.0.0/16

• Create a MetalLB configuration manifest, named metallb-config.yaml and replace x.y with the subnet values, in this case 172.18:

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: kind-pool
  namespace: metallb-system
spec:
  addresses:
  - x.y.255.1-x.y.255.255

---

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisement
  namespace: metallb-system

After both resources have been applied, MetalLB should be able to assign an external IP address to the Istio Ingress Gateway LoadBalancer service. To verify this, you can run:

kubectl get svc istio-ingressgateway -n istio-system

Install Istio Gateway

You will need to create a Kubernetes Gateway resource to expose the Model Performance Metrics module to the outside world. The Gateway resource will be used by the VirtualService resource to route traffic to the module.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

Install the Virtual Service for the Model Performance Metrics module

• You can do so by setting the following values in the values.yaml file:

istioVirtualService:
  create: true
  gateway: istio-system/seldon-gateway # referencing the name of the Gateway resource
  path: /metrics-server/ # the prefix path to the module's endpoints

You can then install the Helm chart with the modified values.yaml file:

helm install metrics-server ./metrics-server

• or by creating a VirtualService resource manually in a file called virtual-service.yaml:

You will have to update the corresponding values in the template using the aforementioned yaml, and then apply the VirtualService resource to the Kubernetes cluster:

kubectl apply -f virtual-service.yaml

Verify the installation

You can verify the Model Performance Metrics module is accessible from outside the Kubernetes cluster.

Obtain the external IP address of the Istio Ingress Gateway service:

kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Then, you can access the Model Performance Metrics module by navigating to http://<EXTERNAL_IP>/metrics-server/api/v1/pipeline/subscriptions in your browser, which should return an empty JSON list with 200 Status Code:

curl -v http://<EXTERNAL_IP>/metrics-server/api/v1/pipeline/subscriptions