1 of 1

Monitoring

Installing kube-prometheus-stack in the same Kubernetes cluster that hosts the Seldon Enterprise Platform.

kube-prometheus, also known as Prometheus Operator, is a popular open-source project that provides complete monitoring and alerting solutions for Kubernetes clusters. It combines tools and components to create a monitoring stack for Kubernetes environments.

Note: Always install Prometheus within the same Kubernetes cluster as the Seldon Enterprise Platform.

The Seldon Enterprise Platform, along with any deployed models, automatically exposes metrics to Prometheus. By default, certain alerting rules are pre-configured, and an alertmanager instance is included.

You can install kube-prometheus to monitor Seldon components, and ensure that the appropriate ServiceMonitors are in place for Seldon deployments. The analytics component is configured with the Prometheus integration. The monitoring for Seldon Enterprise Platform is based on the Prometheus Operator and the related PodMonitor and PrometheusRule resources.

Monitoring the model deployments in Seldon Enterprise Platform involves:

Prerequisites

Install .
Install .
Install

Installing kube-prometheus

Download the seldon-deploy-install.tar file that contains required installation resources. For example, to download the installation resources for version 2.4.0 of Seldon Enterprise Platform run the following:
Extract the contents of the seldon-deploy-install.tar file.
Create a namespace for the monitoring components of Seldon Enterprise Platform.

Configuring monitoring for Seldon Enterprise Platform

To configure monitoring create a dedicated PodMonitor and PrometheusRule resources. Copy the installation resource files from the seldon-deploy-install/reference-configuration/metrics/ directory to the current directory.
Apply the configurations to the Kubernetes cluster that is running the Seldon Enterprise Platform.
When the configuration is complete, you should see this:

Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.
You can use port-forwarding to access your application locally.
Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/ to access Seldon Enterprise Platform.

You may now be able to check the status of Seldon components in Prometheus:

Open your browser and navigate to http://127.0.0.1:9090/ to access Prometheus UI from outside the cluster.
Go to Status and select Targets.

The status of all the endpoints and the scrape details are displayed.

Monitoring

Installing kube-prometheus-stack in the same Kubernetes cluster that hosts the Seldon Enterprise Platform.

Note: Always install Prometheus within the same Kubernetes cluster as the Seldon Enterprise Platform.

Monitoring the model deployments in Seldon Enterprise Platform involves:

Prerequisites

Install .
Install .
Install

Installing kube-prometheus

Download the seldon-deploy-install.tar file that contains required installation resources. For example, to download the installation resources for version 2.4.0 of Seldon Enterprise Platform run the following:
Extract the contents of the seldon-deploy-install.tar file.
Create a namespace for the monitoring components of Seldon Enterprise Platform.

Configuring monitoring for Seldon Enterprise Platform

To configure monitoring create a dedicated PodMonitor and PrometheusRule resources. Copy the installation resource files from the seldon-deploy-install/reference-configuration/metrics/ directory to the current directory.
Apply the configurations to the Kubernetes cluster that is running the Seldon Enterprise Platform.
When the configuration is complete, you should see this:

Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.
You can use port-forwarding to access your application locally.
Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/ to access Seldon Enterprise Platform.

You may now be able to check the status of Seldon components in Prometheus:

Open your browser and navigate to http://127.0.0.1:9090/ to access Prometheus UI from outside the cluster.
Go to Status and select Targets.

The status of all the endpoints and the scrape details are displayed.

Create a YAML file to specify the initial configuration. For example, create the prometheus-values.yaml file. Use your preferred text editor to create and save the file with the following content:

fullnameOverride: seldon-monitoring
kube-state-metrics:
  extraArgs:
    metric-labels-allowlist: pods=[*]

Note: Make sure to include metric-labels-allowlist: pods=[*] in the Helm values file. If you are using your own Prometheus Operator installation, ensure that the pods labels, particularly app.kubernetes.io/managed-by=seldon-core, are part of the collected metrics. These labels are essential for calculating deployment usage rules.

Change to the directory that contains the prometheus-values file and run the following command to install version 9.5.12 of kube-prometheus.

helm upgrade --install prometheus kube-prometheus \
 --version 9.5.12 \
 --namespace seldon-monitoring \
 --values prometheus-values.yaml \
 --repo https://charts.bitnami.com/bitnami

When the installation is complete, you should see this:

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - alertmanager.resources
  - blackboxExporter.resources
  - operator.resources
  - prometheus.resources
  - prometheus.thanos.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Configure metrics collection by creating the following PodMonitor resources.

PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.8.5/prometheus/monitors

kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml
kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml

When the resources are created, you should see this:

podmonitor.monitoring.coreos.com/agent created
servicemonitor.monitoring.coreos.com/envoy created
podmonitor.monitoring.coreos.com/pipelinegateway created
podmonitor.monitoring.coreos.com/server created

Find the IP address of the Seldon Enterprise Platform instance running with Istio:

ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/"

Open your browser and navigate to http://$ISTIO_INGRESS/seldon-deploy/ to access Seldon Enterprise Platform. Where $ISTIO_INGRESS is the IP address of Seldon Enterprise Platform.

Monitoring

Prerequisites

Installing kube-prometheus

Configuring monitoring for Seldon Enterprise Platform

Next

Monitoring

Prerequisites

Installing kube-prometheus

Configuring monitoring for Seldon Enterprise Platform

Next