# Usage Metrics

There are various interesting system metrics about how Seldon Core 2 is used. These metrics can be recorded **anonymously** and sent to Seldon by a lightweight, optional, stand-alone component called Hodometer.

When provided, these metrics are used to understand the adoption of Seldon Core 2 and how you interact with it. For example, knowing how many clusters Seldon Core 2 is running on, if it is used in Kubernetes or for local development, and how many users are benefitting from features such as multi-model serving.

## Architecture

![Hodometer architecture](/files/316lfkr2m8hIwTjChTDw)

Hodometer is not an integral part of Seldon Core 2, but rather an independent component which connects to the public APIs of the Seldon Core 2 scheduler. If deployed in Kubernetes, it requests some basic information from the Kubernetes API.

Recorded metrics are sent to Seldon and, optionally, to any [additional endpoints](#extra-publish-urls) you define.

## Privacy

Hodometer was explicitly designed with privacy of user information and transparency of implementation in mind.

It does not record any sensitive or identifying information. For example, it has no knowledge of IP addresses, model names, or user information. All information sent to Seldon is anonymised with a completely random cluster identifier.

Hodometer supports [different information levels](#metrics-levels), so you have full control over what metrics are provided to Seldon, if any.

For transparency, the implementation is fully open-source and designed to be easy to read. The full source code is available [here](https://github.com/seldonio/seldon-core/tree/v2/hodometer), with metrics defined in code [here](https://github.com/seldonio/seldon-core/tree/v2/hodometer/pkg/hodometer/metrics.go). See[below](#list-of-metrics) for an equivalent table of metrics.

## Performance

Metrics are collected as periodic snapshots a few times per day. They are lightweight to collect, coming mostly from the Seldon Core v2 scheduler, and are heavily aggregated. As such, they should have minimal impact on CPU, memory, and network consumption.

Hodometer does not store anything it records, so does not have any persistent storage. As a result, it should not be considered a replacement for tools like Prometheus.

## Configuration

### Metrics levels

Hodometer supports 3 different metrics levels:

| Level    | Description                                                                                                 |
| -------- | ----------------------------------------------------------------------------------------------------------- |
| Cluster  | Basic information about the Seldon Core v2 installation                                                     |
| Resource | High-level information about which Seldon Core v2 resources are used                                        |
| Feature  | More detailed information about how resources are used and whether or not certain feature flags are enabled |

Alternatively, usage metrics can be completely disabled.\
To do so, simply remove any existing deployment of Hodometer or disable it in the installation for your environment, discussed below.

### Options

The following environment variables control the behaviour of Hodometer, regardless of the environment it is installed in.

| Flag                 | Format                       | Example                                                     | Description                                                                      |
| -------------------- | ---------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `METRICS_LEVEL`      | string                       | feature                                                     | Level of detail for recorded metrics; one of `feature`, `resource`, or `cluster` |
| `EXTRA_PUBLISH_URLS` | comma-separated list of URLs | `http://<my-endpoint-1>:8000`,`http://<my-endpoint-2>:8000` | Additional endpoints to publish metrics to                                       |
| `SCHEDULER_HOST`     | string                       | seldon-scheduler                                            | Hostname for Seldon Core v2 scheduler                                            |
| `SCHEDULER_PORT`     | integer                      | 9004                                                        | Port for Seldon Core v2 scheduler                                                |
| `LOG_LEVEL`          | string                       | info                                                        | Level of detail for application logs                                             |

### Kubernetes

Hodometer is installed as a separate deployment, by default in the same namespace as the rest of the Seldon components.

**Helm**

If you install Seldon Core v2 by [Helm chart](https://github.com/SeldonIO/seldon-core/blob/v2/docs-gb/getting-started/kubernetes-installation/helm.md), there are\
values corresponding to the key environment variables discussed [above](#options). These Helm values and their\
equivalents are provided below:

| Helm value                   | Environment variable |
| ---------------------------- | -------------------- |
| `hodometer.metricsLevel`     | `METRICS_LEVEL`      |
| `hodometer.extraPublishUrls` | `EXTRA_PUBLISH_URLS` |
| `hodometer.logLevel`         | `LOG_LEVEL`          |

If you do not want usage metrics to be recorded, you can disable Hodometer via the `hodometer.disable` Helm\
value when installing the runtime Helm chart. The following command disables collection of usage metrics in\
fresh installations and also serves to remove Hodometer from an existing installation:

```bash
helm install seldon-v2-runtime k8s/helm-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh \
  --set hodometer.disable=true
```

{% hint style="info" %}
**Note**: It is a good practice to set Helm values in values file.\
These can be applied by using the `-f <filename>` switch when running Helm.
{% endhint %}

### Extra publish URLs

Hodometer can be instructed to publish metrics not only to Seldon, but also to any extra endpoints you specify.\
This is controlled by the `EXTRA_PUBLISH_URLS` environment variable, which expects a comma-separated list of\
HTTP-compatible URLs.

You might choose to use this for your own usage monitoring. For example, you could capture these metrics and\
expose them to Prometheus or another monitoring system using your own service.

Metrics are recorded in MixPanel-compatible format, which employs a highly flexible JSON schema.

For an example of how to define your own metrics listener, see the[`receiver` Go package](https://github.com/SeldonIO/seldon-core/tree/v2/hodometer/pkg/receiver) in the `hodometer` sub-project.

## List of metrics

| Metric name                | Level    | Format         | Notes                                                                |
| -------------------------- | -------- | -------------- | -------------------------------------------------------------------- |
| `cluster_id`               | cluster  | UUID           | A random identifier for this cluster for de-duplication              |
| `seldon_core_version`      | cluster  | Version number | E.g. 1.2.3                                                           |
| `is_global_installation`   | cluster  | Boolean        | Whether installation is global or namespaced                         |
| `is_kubernetes`            | cluster  | Boolean        | Whether or not the installation is in Kubernetes                     |
| `kubernetes_version`       | cluster  | Version number | Kubernetes server version, if inside Kubernetes                      |
| `node_count`               | cluster  | Integer        | Number of nodes in the cluster, if inside Kubernetes                 |
| `model_count`              | resource | Integer        | Number of `Model` resources                                          |
| `pipeline_count`           | resource | Integer        | Number of `Pipeline` resources                                       |
| `experiment_count`         | resource | Integer        | Number of `Experiment` resources                                     |
| `server_count`             | resource | Integer        | Number of `Server` resources                                         |
| `server_replica_count`     | resource | Integer        | Total number of `Server` resource replicas                           |
| `multimodel_enabled_count` | feature  | Integer        | Number of `Server` resources with multi-model serving enabled        |
| `overcommit_enabled_count` | feature  | Integer        | Number of `Server` resources with overcommitting enabled             |
| `gpu_enabled_count`        | feature  | Integer        | Number of `Server` resources with GPUs attached                      |
| `inference_server_name`    | feature  | String         | Name of inference server, e.g. MLServer or Triton                    |
| `server_cpu_cores_sum`     | feature  | Float          | Total of CPU limits across all `Server` resource replicas, in cores  |
| `server_memory_gb_sum`     | feature  | Float          | Total of memory limits across all `Server` resource replicas, in GiB |


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.seldon.ai/seldon-core-2/user-guide/operational-monitoring/usage.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.