Ground Truth

Ground truth feedback consists of actual, real-world values that correspond to a model's predictions. Comparing ground truth to model predictions allows for the calculation of model performance metrics. For the Model Performance Metrics module, it is critical to ensure that the ground truth data represents the full distribution of predictions. This helps prevent biased evaluations and ensures that performance metrics accurately reflect real-world model behavior across different input scenarios.

Feedback must include:

• Model name,

• Pipeline name,

• namespace,

• The unique inference request ID for the corresponding prediction, and

• A feedback object containing the feedback type and ground truth value.

By providing all of this information, the module can accurately match the feedback with the inference response.

A model subscription with defined output feature metadata is required before providing ground truth feedback.

Inference Request ID

The inference request ID corresponds to the inference response ID and can be retrieved from the x-request-id header when making the inference request to the inference server.

For more information on retrieving inference request IDs, please refer to the Kafka messages processing section.

Consuming Kafka Events

Example

The example below illustrates how to provide ground truth feedback for the Iris classification model:

main.py

pipeline_name = "iris-pipeline"
model_name = "iris-model"
pipeline_namespace = "seldon"
request_id = "42a52235-f353-4ab5-98fa-a211e1d72c0a"

url = f'http://{CLUSTER_IP}/metrics-server/api/v1/pipeline/model/feedback'
data = {
    "modelName": model_name,
    "pipelineName": pipeline_name,
    "namespace": pipeline_namespace,
    "requestId": request_id,
    # the value can be one of [0, 1, 2], mapping to Setosa, Versicolor, or Virginica, respectively
    "feedback": {"classificationFeedback": {"value": 2}},
}
response = requests.post(url, json=data, verify=False)

If feedback is provided for a regression model, the feedback request should look as follows:

main.py

url = f'http://{CLUSTER_IP}/metrics-server/api/v1/pipeline/model/feedback'
data = {
    "modelName": model_name,
    "pipelineName": pipeline_name,
    "namespace": pipeline_namespace,
    "requestId": request_id,
    "feedback": {"regressionFeedback": {"value": 4}},
}
response = requests.post(url, json=data, verify=False)

By providing a new feedback value for a specific model name, pipeline name, namespace, and request ID, you can update the existing feedback.

You can access the API Swagger documentation by going to the [API Reference](../resources/api-reference.md#feedback-api) page.