# Monitoring

`kube-prometheus`, also known as Prometheus Operator, is a popular open-source project that provides complete monitoring and alerting solutions for Kubernetes clusters. It combines tools and components to create a monitoring stack for Kubernetes environments.

{% hint style="info" %}
**Note**: Always install Prometheus within the same Kubernetes cluster as the Seldon Enterprise Platform.
{% endhint %}

The Seldon Enterprise Platform, along with any deployed models, automatically exposes metrics to Prometheus. By default, certain alerting rules are pre-configured, and an alertmanager instance is included.

You can install `kube-prometheus` to monitor Seldon components, and ensure that the appropriate `ServiceMonitors` are in place for Seldon deployments. The analytics component is configured with the Prometheus integration. The monitoring for Seldon Enterprise Platform is based on the Prometheus Operator and the related `PodMonitor` and `PrometheusRule` resources.

Monitoring the model deployments in Seldon Enterprise Platform involves:

1. [Installing kube-prometheus](#installing-kube-prometheus)
2. [Configuring monitoring](#configuring-monitoring-for-seldon-enterprise-platform)

## Prerequisites

1. Install [Seldon Enterprise Platform](https://docs.seldon.ai/seldon-enterprise-platform/production-environment/seldon-enterprise-platform).
2. Install [Ingress Controller](https://docs.seldon.ai/seldon-enterprise-platform/production-environment/ingress-controller).
3. Install [Docker](https://docs.docker.com/engine/install/)

## Installing kube-prometheus

1. Download the `seldon-deploy-install.tar` file that contains required installation resources. For example, to download the installation resources for version `2.4.0` of Seldon Enterprise Platform run the following:

   ```
   TAG=2.4.0 && \
    docker create --name=tmp-sd-container seldonio/seldon-deploy-server:2.4.0 && \
    docker cp tmp-sd-container:/seldon-deploy-dist/seldon-deploy-install.tar.gz . && \
    docker rm -v tmp-sd-container
   ```
2. Extract the contents of the `seldon-deploy-install.tar` file.

   ```
   tar -xzf seldon-deploy-install.tar.gz
   ```
3. Create a namespace for the monitoring components of Seldon Enterprise Platform.

   ```
   kubectl create ns seldon-monitoring || echo "Namespace seldon-monitoring already exists"
   ```
4. Create a YAML file to specify the initial configuration. For example, create the `prometheus-values.yaml` file. Use your preferred text editor to create and save the file with the following content:

   ```yaml
   fullnameOverride: seldon-monitoring
   kube-state-metrics:
     extraArgs:
       metric-labels-allowlist: pods=[*]
   ```

   **Note**: Make sure to include `metric-labels-allowlist: pods=[*]` in the Helm values file. If you are using your own Prometheus Operator installation, ensure that the pods labels, particularly `app.kubernetes.io/managed-by=seldon-core`, are part of the collected metrics. These labels are essential for calculating deployment usage rules.
5. Change to the directory that contains the `prometheus-values` file and run the following command to install version `9.5.12` of `kube-prometheus`.

   ```
   helm upgrade --install prometheus kube-prometheus \
    --version 9.5.12 \
    --namespace seldon-monitoring \
    --values prometheus-values.yaml \
    --repo https://charts.bitnami.com/bitnami
   ```

   When the installation is complete, you should see this:

   ```
   WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
     - alertmanager.resources
     - blackboxExporter.resources
     - operator.resources
     - prometheus.resources
     - prometheus.thanos.resources
   +info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

   ```
6. Check the status of the installation.

   ```
   kubectl rollout status -n seldon-monitoring deployment/seldon-monitoring-operator
   ```

   When the installation is complete, you should see this:

   ```
   Waiting for deployment "seldon-monitoring-operator" rollout to finish: 0 of 1 updated replicas are available...
   deployment "seldon-monitoring-operator" successfully rolled out
   ```

## Configuring monitoring for Seldon Enterprise Platform

1. To configure monitoring create a dedicated `PodMonitor` and `PrometheusRule` resources. Copy the installation resource files from the `seldon-deploy-install/reference-configuration/metrics/` directory to the current directory.

   ```
   cp seldon-deploy-install/reference-configuration/metrics/seldon-monitor.yaml seldon-monitor.yaml
   cp seldon-deploy-install/reference-configuration/metrics/drift-monitor.yaml drift-monitor.yaml
   cp seldon-deploy-install/reference-configuration/metrics/deploy-monitor.yaml deploy-monitor.yaml
   cp seldon-deploy-install/reference-configuration/metrics/metrics-server-monitor.yaml metrics-server-monitor.yaml
   cp seldon-deploy-install/reference-configuration/metrics/deployment-usage-rules.yaml deployment-usage-rules.yaml
   ```
2. Apply the configurations to the Kubernetes cluster that is running the Seldon Enterprise Platform.

   ```
   kubectl apply -n seldon-monitoring -f seldon-monitor.yaml
   kubectl apply -n seldon-monitoring -f drift-monitor.yaml
   kubectl apply -n seldon-monitoring -f deploy-monitor.yaml
   kubectl apply -n seldon-monitoring -f metrics-server-monitor.yaml
   kubectl apply -f deployment-usage-rules.yaml -n seldon-monitoring
   ```

   When the configuration is complete, you should see this:

   ```
   podmonitor.monitoring.coreos.com/seldon-core created
   podmonitor.monitoring.coreos.com/seldon-drift-detector created
   podmonitor.monitoring.coreos.com/seldon-deploy created
   podmonitor.monitoring.coreos.com/metrics-server created
   prometheusrule.monitoring.coreos.com/seldon-deployment-usage-rules created
   ```
3. You can access Prometheus from outside the cluster by running the following commands:

   ```
   echo "Prometheus URL: http://127.0.0.1:9090/"
   kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-prometheus 9090:9090
   ```
4. You can access Alertmanager from outside the cluster by running the following commands:

   ```
   echo "Alertmanager URL: http://127.0.0.1:9093/"
   kubectl port-forward --namespace seldon-monitoring svc/seldon-monitoring-alertmanager 9093:9093
   ```
5. Add the following to your `install-values.yaml`file.

   ```
   prometheus:
    seldon:
      namespaceMetricName: namespace
      activeModelsNamespaceMetricName: exported_namespace
      serviceMetricName: service
      url: http://seldon-monitoring-prometheus.seldon-monitoring:9090/api/v1/
   env:
     ALERTMANAGER_URL: http://seldon-monitoring-alertmanager.seldon-monitoring:9093/api/v1/alerts   
   ```
6. Configure metrics collection by creating the following `PodMonitor` resources.

   ```bash
   PODMONITOR_RESOURCE_LOCATION=https://raw.githubusercontent.com/SeldonIO/seldon-core/v2.8.5/prometheus/monitors

   kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/agent-podmonitor.yaml
   kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/envoy-servicemonitor.yaml
   kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/pipelinegateway-podmonitor.yaml
   kubectl apply -f ${PODMONITOR_RESOURCE_LOCATION}/server-podmonitor.yaml
   ```

   When the resources are created, you should see this:

   ```bash
   podmonitor.monitoring.coreos.com/agent created
   servicemonitor.monitoring.coreos.com/envoy created
   podmonitor.monitoring.coreos.com/pipelinegateway created
   podmonitor.monitoring.coreos.com/server created
   ```
7. Change to the directory that contains the `install-values.yaml` file and then upgrade the Seldon Enterprise Platform installation in the namespace `seldon-system`.

   ```
   helm upgrade seldon-enterprise seldon-charts/seldon-deploy --namespace seldon-system  -f install-values.yaml --version 2.4.0 --install
   ```
8. Check the status of the installation seldon-enterprise-seldon-deploy.

   ```
   kubectl rollout status deployment/seldon-enterprise-seldon-deploy -n seldon-system
   ```

   When the installation is complete you should see this:

   ```
   deployment "seldon-enterprise-seldon-deploy" successfully rolled out
   ```
9. Access Seldon Enterprise Platform.

{% tabs %}
{% tab title="Port forwarding" %}

1. Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as `$POD_NAME.`

   ```
   export POD_NAME=$(kubectl get pods --namespace seldon-system -l "app.kubernetes.io/name=seldon-deploy,app.kubernetes.io/instance=seldon-enterprise" -o jsonpath="{.items[0].metadata.name}")
   ```
2. You can use port-forwarding to access your application locally.

   ```
   kubectl port-forward $POD_NAME 8000:8000 --namespace seldon-system
   ```
3. Open your browser and navigate to `http://127.0.0.1:8000/seldon-deploy/` to access Seldon Enterprise Platform.
   {% endtab %}

{% tab title="Static IP address" %}

1. Find the IP address of the Seldon Enterprise Platform instance running with Istio:

   ```
   ISTIO_INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   ISTIO_INGRESS+=$(kubectl get svc -n istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

   echo "Seldon Enterprise Platform: http://$ISTIO_INGRESS/seldon-deploy/"

   ```
2. Open your browser and navigate to `http://$ISTIO_INGRESS/seldon-deploy/` to access Seldon Enterprise Platform. Where `$ISTIO_INGRESS` is the IP address of Seldon Enterprise Platform.
   {% endtab %}
   {% endtabs %}

## Next

You may now be able to check the status of Seldon components in Prometheus:

1. Open your browser and navigate to `http://127.0.0.1:9090/` to access Prometheus UI from outside the cluster.
2. Go to **Status** and select **Targets**.

The status of all the endpoints and the scrape details are displayed.