# Request Logging ## Prerequisites * Download the [installation resources](/seldon-enterprise-platform/production-environment/observability-alerting/observability.md#installing-kube-prometheus). * Verify the [cluster requirements](/seldon-enterprise-platform/production-environment.md#cluster-requirements). * Install [Seldon Enterprise Platform](/seldon-enterprise-platform/production-environment/seldon-enterprise-platform.md). * Install [Seldon Core 2](/seldon-enterprise-platform/production-environment/seldon-core-2.md). ## Install Knative Eventing If you have not installed Istio as your ingress controller, you will not be able to use Knative Eventing and will need to configure Seldon Enterprise Platform accordingly. Knative Eventing technology is utilized to create an event driven architecture for ML data events logging and is required to log prediction requests for Core V1. See[ Knative section](/seldon-enterprise-platform/production-environment/knative.md) for installation steps. ## Install Elasticsearch Stack [Elasticsearch](https://www.elastic.co/elasticsearch/) is the main data storage for various ML data event payloads.\ Seldon Enterprise Platform does not support [OpenSearch](https://opensearch.org/docs/latest/) as an alternative to Elasticsearch. Fluentd is used for collecting application logs. Seldon uses an in-house developed component called [metronome](#metronome) for storing the event payloads and other data as Elasticsearch documents. ## Seldon Core and Enterprise Platform Configuration For Seldon Core, add the following options to your `core-values.yaml` file: ```yaml executor: requestLogger: defaultEndpoint: "http://broker-ingress.knative-eventing.svc.cluster.local/seldon-logs/default" ``` For Seldon Enterprise Platform, add the following options to your `install-value.yaml` file: ```yaml requestLogger: create: true ``` ## Metronome The above configuration for Seldon Enterprise Platform creates a component called Metronome, developed by Seldon. Metronome is a request logger for ML events. Metronome splits out batch payloads and enriches request payloads with additional metadata from the ML prediction schema. This component is also responsible for creating the correct database mappings, define the payload schema and storing inferred feature data. ### ML Metadata Metronome can optionally obtain a prediction schema from the Seldon Enterprise Platform metadata service, if enabled. This can be used to enrich prediction logs (add category name for categorical features and identify probabilistic features) for better logging and advanced monitoring. To enable this feature, it is necessary to ensure that the Seldon Enterprise Platform instance is configured with a suitable license. Then, follow the steps below: For more details see the [PostgreSQL setup](/seldon-enterprise-platform/production-environment/postgresql.md). 1. Have an OIDC provider configured that supports the client credentials grant flow and enable this on a client for Metronome to use. In some OIDC providers, including Keycloak, setting up a service account is required for the client credentials auth flow. The password grant flow can be used for cases in which a client credentials flow is not supported. 2. Create an auth secret for the Metronome configuration as below, setting the parameters as per your environment. ```bash kubectl create secret generic -n seldon-logs request-logger-auth \ --from-literal=OIDC_PROVIDER="${OIDC_PROVIDER}" \ --from-literal=CLIENT_ID="${CLIENT_ID}" \ --from-literal=CLIENT_SECRET="${CLIENT_SECRET}" \ --from-literal=OIDC_AUTH_METHOD="client_credentials" \ --from-literal=OIDC_SCOPES="${OIDC_SCOPES}" \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. Ensure `requestLogger.deployHost` and `requestLogger.authSecret` are set, in `install-values.yaml`. ```yaml requestLogger: authSecret: request-logger-auth ``` If you are not using [app-level auth](/seldon-enterprise-platform/production-environment/auth.md), because you are using ingress-level authentication, skip all the above steps except (1) and set the below in `install-values.yaml`: ```yaml requestLogger: authSecret: "" ``` ## Authentication on Elasticsearch The Seldon Enterprise Platform Helm values file has two options for connecting to a secured Elasticsearch database. One is token-based authentication, which you should use if you have an auth token. This is used for OpenShift cluster logging flavour of Elasticsearch/OpenSearch The other option is basic authentication. This requires creating a secret with the username and password for the Elasticsearch/OpenSearch database. These steps are described in the [Elasticsearch installation ](/seldon-enterprise-platform/production-environment/elasticsearch.md)documentation. For Enterprise Platform this would need secrets in the namespaces `seldon-logs` (containing Elasticsearch and Metronome) and `seldon-system` (containing the Plaform itself) as Enterprise Platform would need to speak to Elasticsearch using the secret. This could look like: ```bash ELASTIC_USER=replaceme ELASTIC_PASSWORD=replaceme kubectl create secret generic elastic-credentials -n seldon-logs \ --from-literal=username="${ELASTIC_USER}" \ --from-literal=password="${ELASTIC_PASSWORD}" \ --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic elastic-credentials -n seldon-system \ --from-literal=username="${ELASTIC_USER}" \ --from-literal=password="${ELASTIC_PASSWORD}" \ --dry-run=client -o yaml | kubectl apply -f - ``` ## Running without Knative ### Disable Knative Triggers If you run without Knative, for example because you use an ingress other than Istio, you need to disable Knative Triggers in your `install-values.yaml` file: ```yaml requestLogger: trigger: create: false ``` This will create a deployment of Metronome (Seldon request logger) that can still log payloads from Core v2 pipelines. ### Configure Seldon Core If you just run without Knative, but still use Istio and Seldon Core v1, you need to adjust the Seldon Core Helm installation to not use Knative Eventing for logging. You can do this by setting the following in your `core-values.yaml`: ```yaml executor: requestLogger: defaultEndpoint: "http://seldon-request-logger.seldon-logs.svc.cluster.local" ``` ## ML data payload parsing Metronome will try to infer the request and response payloads, parse and insert them into Elasticsearch. If metadata is available in the Seldon Enterprise Platform model catalogue, then this will be used to enrich the payloads. Metronome supports the Seldon v1, TensorFlow, and [Open Inference Protocol](https://github.com/kserve/open-inference-protocol/tree/main) payloads for the prediction and alibi-detect protocols for outlier/drift detection use cases. In general, Metronome: * Assumes the first dimension of tensor based data is the batch dimension and attempts to split the data into individual request/response entries before storing them in Elasticsearch. * Passes through seldon v1 `jsonData`, `strData` and `binData` unmodified. If you're sending image or text data, the default maximum payload of Metronome, which is 300 KB, may not be enough. To change this, you can add the following to your `install-values.yaml`: ```yaml requestLogger: env: MAX_PAYLOAD_BYTES: "300000" # 300 KB, change as needed ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.seldon.ai/seldon-enterprise-platform/production-environment/request-logging.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.