# Monitoring `kube-prometheus`, also known as Prometheus Operator, is a popular open-source project that provides complete monitoring and alerting solutions for Kubernetes clusters. It combines tools and components to create a monitoring stack for Kubernetes environments. {% hint style="info" %} **Note**: Always install Prometheus within the same Kubernetes cluster as the Seldon Enterprise Platform. {% endhint %} The Seldon Enterprise Platform, along with any deployed models, automatically exposes metrics to Prometheus. By default, certain alerting rules are pre-configured, and an alertmanager instance is included. You can install `kube-prometheus` to monitor Seldon components, and ensure that the appropriate `ServiceMonitors` are in place for Seldon deployments. The analytics component is configured with the Prometheus integration. The monitoring for Seldon Enterprise Platform is based on the Prometheus Operator and the related `PodMonitor` and `PrometheusRule` resources. Monitoring the model deployments in Seldon Enterprise Platform involves: 1. [Installing kube-prometheus](#installing-kube-prometheus) 2. [Configuring monitoring](#configuring-monitoring-for-seldon-enterprise-platform) ## Prerequisites 1. Install [Seldon Enterprise Platform](/seldon-enterprise-platform/production-environment/seldon-enterprise-platform.md). 2. Install [Ingress Controller](/seldon-enterprise-platform/production-environment/ingress-controller.md). 3. Install [Docker](https://docs.docker.com/engine/install/) ## Installing kube-prometheus 1. Download the `seldon-deploy-install.tar` file that contains required installation resources. For example, to download the installation resources for version `2.4.0` of Seldon Enterprise Platform run the following: ``` TAG=2.4.0 && \ docker create --name=tmp-sd-container seldonio/seldon-deploy-server:2.4.0 && \ docker cp tmp-sd-container:/seldon-deploy-dist/seldon-deploy-install.tar.gz . && \ docker rm -v tmp-sd-container ``` 2. Extract the contents of the `seldon-deploy-install.tar` file. ``` tar -xzf seldon-deploy-install.tar.gz ``` 3. Create a namespace for the monitoring components of Seldon Enterprise Platform. ``` kubectl create ns seldon-monitoring || echo "Namespace seldon-monitoring already exists" ``` 4. Create a YAML file to specify the initial configuration. For example, create the `prometheus-values.yaml` file. Use your preferred text editor to create and save the file with the following content: ```yaml fullnameOverride: seldon-monitoring kube-state-metrics: extraArgs: metric-labels-allowlist: pods=[*] ``` **Note**: Make sure to include `metric-labels-allowlist: pods=[*]` in the Helm values file. If you are using your own Prometheus Operator installation, ensure that the pods labels, particularly `app.kubernetes.io/managed-by=seldon-core`, are part of the collected metrics. These labels are essential for calculating deployment usage rules. 5. Change to the directory that contains the `prometheus-values` file and run the following command to install version `9.5.12` of `kube-prometheus`. ``` helm upgrade --install prometheus kube-prometheus \ --version 9.5.12 \ --namespace seldon-monitoring \ --values prometheus-values.yaml \ --repo https://charts.bitnami.com/bitnami ``` When the installation is complete, you should see this: ``` WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs: - alertmanager.resources - blackboxExporter.resources - operator.resources - prometheus.resources - prometheus.thanos.resources +info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ ``` 6. Check the status of the installation. ``` kubectl rollout status -n seldon-monitoring deployment/seldon-monitoring-operator ``` When the installation is complete, you should see this: ``` Waiting for deployment "seldon-monitoring-operator" rollout to finish: 0 of 1 updated replicas are available... deployment "seldon-monitoring-operator" successfully rolled out ``` ## Configuring monitoring for Seldon Enterprise Platform 1. To configure monitoring create a dedicated `PodMonitor` and `PrometheusRule` resources. Copy the installation resource files from the `seldon-deploy-install/reference-configuration/metrics/` directory to the current directory. ``` cp seldon-deploy-install/reference-configuration/metrics/seldon-monitor.yaml seldon-monitor.yaml cp seldon-deploy-install/reference-configuration/metrics/drift-monitor.yaml drift-monitor.yaml cp seldon-deploy-install/reference-configuration/metrics/deploy-monitor.yaml deploy-monitor.yaml cp seldon-deploy-install/reference-configuration/metrics/metrics-server-monitor.yaml metrics-server-monitor.yaml cp seldon-deploy-install/reference-configuration/metrics/deployment-usage-rules.yaml deployment-usage-rules.yaml ``` 2. Apply the configurations to the Kubernetes cluster that is running the Seldon Enterprise Platform. ``` kubectl apply -n seldon-monitoring -f seldon-monitor.yaml kubectl apply -n seldon-monitoring -f drift-monitor.yaml kubectl apply -n seldon-monitoring -f deploy-monitor.yaml kubectl apply -n seldon-monitoring -f metrics-server-monitor.yaml kubectl apply -f deployment-usage-rules.yaml -n seldon-monitoring ``` When the configuration is complete, you should see this: ``` podmonitor.monitoring.coreos.com/seldon-core created podmonitor.monitoring.coreos.com/seldon-drift-detector created podmonitor.monitoring.coreos.com/seldon-deploy created podmonitor.monitoring.coreos.com/metrics-server created prometheusrule.monitoring.coreos.com/seldon-deployment-usage-rules created ``` 3. You can access Prometheus from outside the cluster by running the following commands: ``` echo "Prometheus URL: http://127.0.0.1:9090/" kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-prometheus 9090:9090 ``` 4. You can access Alertmanager from outside the cluster by running the following commands: ``` echo "Alertmanager URL: http://127.0.0.1:9093/" kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-alertmanager 9093:9093 ``` 5. Add the following to your `install-values.yaml`file. ``` prometheus: seldon: namespaceMetricName: namespace activeModelsNamespaceMetricName: exported_namespace serviceMetricName: service url: http://seldon-monitoring-prometheus.seldon-monitoring:9090/api/v1/ env: ALERTMANAGER_URL: http://seldon-monitoring-alertmanager.seldon-monitoring:9093/api/v1/alerts ``` 6. Configure metrics collection by creating the following `PodMonitor` resources. ```bash PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.8.5/prometheus/monitors kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml ``` When the resources are created, you should see this: ```bash podmonitor.monitoring.coreos.com/agent created servicemonitor.monitoring.coreos.com/envoy created podmonitor.monitoring.coreos.com/pipelinegateway created podmonitor.monitoring.coreos.com/server created ``` 7. Change to the directory that contains the `install-values.yaml` file and then upgrade the Seldon Enterprise Platform installation in the namespace `seldon-system`. ``` helm upgrade seldon-enterprise seldon-charts/seldon-deploy --namespace seldon-system -f install-values.yaml --version 2.4.0 --install ``` 8. Check the status of the installation seldon-enterprise-seldon-deploy. ``` kubectl rollout status deployment/seldon-enterprise-seldon-deploy -n seldon-system ``` When the installation is complete you should see this: ``` deployment "seldon-enterprise-seldon-deploy" successfully rolled out ``` 9. Access Seldon Enterprise Platform. {% tabs %} {% tab title="Port forwarding" %} 1. Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as `$POD_NAME.` ``` export POD_NAME=$(kubectl get pods --namespace seldon-system -l "app.kubernetes.io/name=seldon-deploy,app.kubernetes.io/instance=seldon-enterprise" -o jsonpath="{.items[0].metadata.name}") ``` 2. You can use port-forwarding to access your application locally. ``` kubectl port-forward $POD_NAME 8000:8000 --namespace seldon-system ``` 3. Open your browser and navigate to `http://127.0.0.1:8000/seldon-deploy/` to access Seldon Enterprise Platform. {% endtab %} {% tab title="Static IP address" %} 1. Find the IP address of the Seldon Enterprise Platform instance running with Istio: ``` ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/" ``` 2. Open your browser and navigate to `http://$ISTIO_INGRESS/seldon-deploy/` to access Seldon Enterprise Platform. Where `$ISTIO_INGRESS` is the IP address of Seldon Enterprise Platform. {% endtab %} {% endtabs %} ## Next You may now be able to check the status of Seldon components in Prometheus: 1. Open your browser and navigate to `http://127.0.0.1:9090/` to access Prometheus UI from outside the cluster. 2. Go to **Status** and select **Targets**. The status of all the endpoints and the scrape details are displayed. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.seldon.ai/seldon-enterprise-platform/production-environment/observability-alerting/observability.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.