# Model Accuracy

Iris is the genus of flower which contains 3 species: setosa, versicolor, and virginica. This demo is based on iris classification model based on flower properties like sepal length, sepal width, petal length, and petal width. The species are also the classes that will be used for the classification. Here we will:

* Set up a metrics server for the pre-existing iris classifier deployment
* Send a request to get an iris classification
* Send feedback requests to gather accuracy metrics

{% hint style="warning" %}
This demo requires Knative installation on the cluster as the metrics server will be installed as a kservice. Also, the metrics server only works with classification models in this version.
{% endhint %}

{% hint style="danger" %}
These model metrics are only supported for **pre-existing Seldon Core 1 deployments with the Seldon inference protocol**. Seldon now recommends adopting the industry-standard Open Inference Protocol (OIP) in preference to the Seldon protocol. Seldon has a separate module to support the calculation of model performance metrics with the Open Inference Protocol. Please contact your customer success representative to learn more.
{% endhint %}

## Setup Metrics Server

1. Use the pre-existing iris classifier deployment with the Seldon inference protocol
2. From the `Dashboard` page, click on `Add` in the Metrics Server panel:

   * *Detector Name*: `multiclass`
   * *Storage URI*: `adserver.cm_models.multiclass_numeric.MultiClassNumeric`
   * *Storage Secret*: Leave empty as we are using a public bucket
   * *Reply URL*: Leave as default value, `http://seldon-request-logger.seldon-logs`

   ![metrics server](/files/vlZ7iG9xUpRKf9ux2YFk)

## Make Predictions

Run a single prediction using the ndarray payload format. Make a couple of these requests at random using the predict tool in the UI.

1. Go to the Predict page.
2. Paste in the following JSON data for the prediction, and click the `Predict` button:

   ```json
   {
     "data": {
       "names": [
         "Sepal length",
         "Sepal width",
         "Petal length",
         "Petal Width"
       ],
       "ndarray": [
         [
           6.8,
           2.8,
           4.8,
           1.4
         ]
       ]
     }
   }
   ```
3. Inspect the response, to see that the class with the highest confidence is the second of the iris dataset, `versicolor`.

   ```json
   {
     "data": {
       "names": [
         "t:0",
         "t:1",
         "t:2"
       ],
       "ndarray": [
         [
            0.008074020139119268,
            0.7781601484223357,
            0.21376583143854502
         ]
       ]
     },
     "meta": {
       "requestPath": {
       "iris-accuracy-container": "seldonio/sklearnserver:1.18.2"
       }
     }
   }
   ```

## Send Feedback

As we saw the prediction response was `versicolor` (the second class of iris). In numeric form the response is,

```json
{
  "data": {
    "ndarray": [
      1
    ]
  }
}
```

A feedback request consists of the returned value in a response (`response.data` on line 5) and a truth value (`truth.data` on line 12), for that data. For example, a true-positive feedback request looks like this:

```json
{
  "response": {
    "data": {
      "ndarray": [
        1
      ]
    }
  },
  "truth": {
    "data": {
      "ndarray": [
        1
      ]
    }
  }
}
```

We'll send several feedback requests to the metrics server, that we set-up earlier, for various scenarios.

1. Set-up the environment variable(s) for the request:

   ```bash
   export CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   ```

   (Recommended) You may also need to set-up for the `Authorization` header:

   1. Return to the predict page, and paste in the prediction, as you did above
   2. Click the `Copy as curl` button, and copy the value that looks like `Bearer <authorization header>`

   ![Copying aithorization header](/files/xuGl8xiqYrYgqPTwUCyV)

   3. Set the environment variable:

   ```bash
   export AUTH_TOKEN="<paste the authorization value>"
   ```
2. Scenario 1: True-positive:

   Request without `Authorization` header:

   ```bash
   curl -k \
       -H "Content-Type: application/json" \
       https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
       -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'
   ```

   Request with `Authorization` header:

   ```bash
   curl -k \
       -H "Content-Type: application/json" \
       -H "Authorization: $AUTH_TOKEN" \
       https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
       -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'
   ```

   Response:

   ```json
   {
     "data": {
        "tensor": {
        "shape": [
            0
          ]
         }
       },
       "meta": {
        "requestPath": {
          "iris-accuracy-container": 1.18.2
       }
     }
   }
   ```
3. Scenario 2: False-positive:

   The difference from the previous scenario is the in the `truth` value:

   ```diff
   -   -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[1]}}}'
   +   -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'
   ```

   Request without `Authorization` header:

   ```bash
   curl -k \
       -H "Content-Type: application/json" \
       https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
       -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'
   ```

   Request with `Authorization` header:

   ```bash
   curl -k \
       -H "Content-Type: application/json" \
       -H "Authorization: $AUTH_TOKEN" \
       https://$CLUSTER_IP/seldon/seldon/iris-accuracy/api/v0.1/feedback \
       -d '{"response":{"data":{"ndarray":[1]}},"truth":{"data":{"ndarray":[0]}}}'
   ```

   Response:

   ```json
   {
     "data": {
        "tensor": {
        "shape": [
            0
          ]
         }
       },
       "meta": {
        "requestPath": {
          "iris-accuracy-container": 1.18.2
       }
     }
   }
   ```

## Monitor accuracy metrics on the Monitor Screen

Having done a prediction, metrics will begin to become available.

1. Go to the monitor screen's `Prediction Accuracy` tab to view all the metrics.
2. Set the time range to view the metrics, using the "From Time" and "To Time" selectors.
3. You can see metrics like accuracy, precision, recall and specificity here. Notice the drop in accuracy metrics after the false feedback was received.

## Submit batch feedback using Batch Processor component

Now we will submit a feedback as a batch using the Batch Processor component.

We will use two files, each containing 10k feedback instances:

* 90% success rate feedback file

{% file src="/files/pVmsCP6xE1YKLA1061Mp" %}

* 40% success rate feedback file

{% file src="/files/A12ogFgW3FkPctQ6Up2e" %}

1. We need to upload the files to MinIO's `data` bucket. For details on interacting with MinIO UI please see [Batch Demo](/seldon-enterprise-platform/demos/seldon-core-v1/batch-requests.md).
2. Once the files are in the MinIO bucket, go to the `Batch Jobs` screen using either the navigation bar on the left side or with the button on the model dashboard. For each of `feedback-input-90.txt` and `feedback-input-40.txt` files, submit a batch job. Wait for the first to complete before proceeding to the next. Submit batch request using the following form values:

   * *Input Data Location*: `minio://data/feedback-input-40.txt`
   * *Output Data Location*: `minio://data/output-data-40-{{workflow.name}}.txt`
   * *Number of Workers*: `15`
   * *Number of Retries*: `3`
   * *Batch Size*: `1`
   * *Minimum Batch Wait Interval (sec)*: `0`
   * *Method*: `Feedback`
   * *Transport Protocol*: `REST`
   * *Input Data Type*: `Raw Data`
   * *Object Store Secret Name*: `minio-bucket-envvars`
   * *Resources*: `Leave defaults`

   ![metrics server](/files/3psD8oBqisKhG3ji4zjo)
3. Now go to the monitor view and observe how metrics value evolve over time.

## Troubleshooting

If you experience issues with this demo, see the [troubleshooting docs](/seldon-enterprise-platform/help-and-support.md) and also the [Knative](/seldon-enterprise-platform/production-environment/request-logging.md) or [Elasticsearch](/seldon-enterprise-platform/production-environment/elasticsearch.md) sections.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.seldon.ai/seldon-enterprise-platform/demos/seldon-core-v1/model-accuracy-metrics.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.