Calculating Metrics

Metrics are computed over a user-specified time window. The specified time window is then split into time buckets, each of which having a duration equal to a user-specified interval.

Time Window

Specify the time window for the metrics calculation by providing a start datetime and an end datetime, using the ISO 8601 format (e.g., 2025-01-01T00:00:00.000Z).

The end datetime is open, meaning it is exclusive, and the same applies to the end datetimes within each of the time buckets.

Interval

The interval parameter determines the size of each bucket within the time window and must be specified as a duration string. A duration string consists of decimal numbers, optionally including fractions, followed by a unit suffix such as 10s for 10 seconds, 2h45m for 2 hours and 45 minutes, or 1.5h for 1.5 hour. Valid time units include us (or µs), ms, s, m, h.

Although ns (nanoseconds) is a valid time unit from the protobuf perspective, it is not meaningful in this context, as PostgreSQL database used for storing predictions and ground truth supports only us (microsecond) resolution.

Consider the following important points:

• The interval value must not exceed the duration of the time window.

• If the interval parameter is not provided, a default value of one bucket matching the entire time window is used.

Time Bucketing

The time window is divided into buckets of the specified interval size, starting from the time window's start datetime.

If a non-integer number of intervals fits within the time window, the last bucket will have the following start and end datetimes:

$$\left[ \texttt{time window start datetime} + \texttt{math.floor} \left( \frac{\texttt{time window duration}}{\texttt{interval}}\right) \times \texttt{interval}, \texttt{time window end datetime}\right)$$

In this case the final bucket will be smaller than the specified interval.

The maximum number of buckets is 100; therefore, the interval and the time window should be selected accordingly.

A data point is assigned to a bucket if the model's inference time falls within that bucket. Each bucket in the Metrics API response includes an endTime field.

If there are no data points within the time bucket:

• -1 will be returned for scalar metrics such as Accuracy, Recall, Precision, Specificity, F1, MSE, MAE, and RMSE.

• An empty array will be returned for non-scalar fields, such as the confusion matrix fields.