Alerting
Installing kube-prometheus-stack in the same Kubernetes cluster that hosts the Seldon Enterprise Platform.
Last updated
Was this helpful?
Installing kube-prometheus-stack in the same Kubernetes cluster that hosts the Seldon Enterprise Platform.
Last updated
Was this helpful?
The Seldon Enterprise Platform, along with any deployed models, automatically exposes metrics to Prometheus. By default, certain alerting rules are pre-configured, and an alertmanager instance is included.
You can configure Alertmanager to send alerts through email or Slack. It can also be integrated into an incident response tool. To receive alerts when using Seldon Enterprise Platform you need to:
Install .
Install .
Install
To configure default alerting rules, copy the installation resource files from the seldon-deploy-install/reference-configuration/metrics/
directory to the current directory. To configure custom alerts, see the custom alerts section.
Apply the configurations to the Kubernetes cluster that is running the Seldon Enterprise Platform.
When the configuration is complete, you should see this:
Create a YAML file to specify the initial configuration. For example, create the alertmanager.yaml
file. Use your preferred text editor to create and save the file with the following content:
For more information about configuring alerts during authetication, see Authentication alerts section.
Apply the Altermanager configurations in the Kubernetes cluster that is running Seldon Enterprise Platform:
When the configurations are applied, you should see this:
You can access Alertmanager from outside the cluster by running the following commands:
Access Seldon Enterprise Platform.
Get the Pod that is running Seldon Enterprise Platform in the cluster and save it as $POD_NAME.
You can use port-forwarding to access your application locally.
Open your browser and navigate to http://127.0.0.1:8000/seldon-deploy/
to access Seldon Enterprise Platform.
You can also define your own custom alerting rules in Prometheus.
Create a file called custom-alert.yaml
that contains your new rules. You can find some examples in the file user-alerts.yaml
file located in the seldon-deploy-install/reference-configuration/metrics/
folder.
Apply the alerts using:
You may now be able to check the alerts that you configured in Alertmanager:
Open your browser and navigate to http://127.0.0.1:9093/
to access Alertmanager UI from outside the cluster.
Go to Alerts and check if any alert rules that are listed in Prometheus are tiggered.
If any of the alert rules are triggered then those alerts are displayed.
If you are using App Level Authentication you need to add http_config
in the webhook_configs
section of alertmanager.yaml
. This needs a client that has been configured to access the . The token_url value may vary, depending on your OIDC provider.
If you are using a self-signed certificate on your OIDC provider then you need to set insecure_skip_verify
in the tls_config
of the oauth2
block. Alternatively, you can mount your CA certificate onto the Alertmanager instance to validate the server certificate using ca_file
. For more information see, the .
You can integrate the alerts that you configured in Seldon Enterprise Platform with various alert notification tools such as or .