# ElasticSearch ## Elasticsearch Installation {% hint style="info" %} **Note**: Elasticsearch is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator's responsibility to administrate and manage the Elasticsearch instance used by Seldon. Seldon Enterprise Platform does not support [OpenSearch](https://opensearch.org/docs/latest/) as an alternative to Elasticsearch. {% endhint %} ### Initial Configuration Copy the default Fluentd config {% code overflow="wrap" %} ```bash cp ./seldon-deploy-install/reference-configuration/efk/values-fluentd.yaml fluentd-values.yaml ``` {% endcode %} As the starting `fluentd` configuration is crafted for Elasticsearch, by Open Distro, you need to modify the `elasticsearch` section in the `fluentd-values.yaml` file: ```yaml elasticsearch: hosts: ['elasticsearch-master.seldon-logs.svc.cluster.local'] logstash: enabled: true prefix: 'kubernetes_cluster' auth: enabled: false scheme: "http" sslVerify: false ``` #### Ensure Required Namespaces Exist We'll be installing in the `seldon-logs` namespace. We'll also set up some config in the `seldon-system` namespace. ```bash kubectl create namespace seldon-logs || echo "namespace seldon-logs exists" kubectl create namespace seldon-system || echo "namespace seldon-system exists" ``` ### Elasticsearch Elasticsearch can be installed using Elastic Cloud on Kubernetes (ECK). ECK can be installed using helm: ```bash helm repo add elastic https://helm.elastic.co helm repo update helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace --version=2.11.1 ``` Then, create an Elasticsearch instance called `seldon` in the `seldon-logs` namespace using the following scripts: ```bash cat << EOF > elasticsearch.yaml apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: seldon namespace: seldon-logs spec: nodeSets: - config: node.store.allow_mmap: false count: 3 name: default podTemplate: spec: containers: - name: elasticsearch resources: limits: memory: 2Gi requests: cpu: "1" memory: 2Gi volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: standard version: 7.17.18 EOF kubectl apply -f elasticsearch.yaml ``` {% hint style="info" %} **Note**: Currently, we guarantee compatibility with Elasticsearch 7.X. Compatibility with Elasticsearch 8.X is not guaranteed. {% endhint %} ### Authentication Security is managed by the ECK operator and cannot be disabled. The operator will create a secret with the credentials for the `elastic` user. We can use this password to provide credentials to other components that need to access Elasticsearch. To do this, we generate secrets in the `seldon-logs` (for the request logger) and `seldon-system` (for Seldon Enterprise Platform) namespaces from the `elastic` user's password: ```bash ELASTIC_USERNAME=$(echo -n elastic | base64) ELASTIC_PASSWORD=$(kubectl get secret seldon-es-elastic-user -n seldon-logs -o go-template='{{.data.elastic}}') cat << EOF > elastic-credentials.yaml apiVersion: v1 data: password: ${ELASTIC_PASSWORD} username: ${ELASTIC_USERNAME} kind: Secret metadata: name: elastic-credentials type: Opaque EOF kubectl apply -f elastic-credentials.yaml -n seldon-logs kubectl apply -f elastic-credentials.yaml -n seldon-system ``` ### Fluentd We need to modify the `fluentd-values.yaml` file to point to the Elasticsearch instance we just created, as well as set the appropriate credentials. Retrieve the `ELASTIC_PASSWORD`: ```bash export ELASTIC_PASSWORD=$(kubectl get secret elastic-credentials -n seldon-logs -o go-template='{{.data.password | base64decode}}') ``` Make a copy of the `values-fluentd.yaml` file: ```bash cp values-fluentd.yaml values-elasticsearch-fluentd.yaml ``` Update the following values in `values-elasticsearch-fluentd.yaml`: ```yaml elasticsearch: auth: user: "elastic" password: hosts: - seldon-es-http.seldon-logs.svc.cluster.local ``` We can then install Fluentd using helm: ```bash helm upgrade --install fluentd fluentd-elasticsearch \ --version 10.0.1 \ --namespace seldon-logs -f values-elasticsearch-fluentd.yaml \ --repo https://kokuwaio.github.io/helm-charts ``` ### Kibana (optional) Kibana is useful for creating visualizations and dashboards for Elasticsearch. It is not required for Seldon Enterprise Platform, however users may choose to install it for debugging purposes: As we are using ECK, we can install Kibana with the following script: ```bash cat << EOF > kibana.yaml apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: seldon namespace: seldon-logs spec: version: 7.17.18 count: 1 elasticsearchRef: name: seldon namespace: seldon-logs EOF kubectl apply -f kibana.yaml kubectl rollout status deployment/kibana-kibana -n seldon-logs ``` ### Configure Seldon Enterprise Platform The following Helm values need to be set in `install-values.yaml`. If you did not install Knative Eventing, `requestLogger.trigger.create` has to be set to `false`. ```yaml requestLogger: create: true elasticsearch: host: seldon-es-http.seldon-logs.svc.cluster.local port: "9200" protocol: http trigger: create: true # false if not using Knative elasticsearch: url: http://seldon-es-http.seldon-logs.svc.cluster.local:9200 basicAuth: true secret: name: "elastic-credentials" userKey: "username" passwordKey: "password" ``` As the Elasticsearch instance has authentication enabled, we set `elasticsearch.basicAuth` to `true`. We also provide the name of the secret containing the `elastic` user's credentials in the `elasticsearch.secret` section. #### Authorization The Seldon Enterprise Platform setup needs authorization on the Elasticsearch cluster to create, manage, and search indexes for prediction logging and other monitoring features. The following security privileges are mandatory for proper functioning of the current Seldon Enterprise Platform features. Read more about Elasticsearch security privileges [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-privileges.html). **Seldon Enterprise Platform user security privileges** | Elasticsearch Privileges | Privilege Level | Index-pattern(s) | | ------------------------ | --------------- | ---------------------- | | monitor | Cluster | NA | | index | Index | `inference-log-*` | | index | Index | `reference-log-*` | | index | Index | `drift-log-*` | | read | Index | `inference-log-*` | | read | Index | `reference-log-*` | | read | Index | `drift-log-*` | | read | Index | `kubernetes_cluster-*` | | read | Index | `*` | **Seldon Request Logger user security Privileges** | Elasticsearch Privileges | Privilege Level | Index-pattern(s) | | ------------------------ | --------------- | --------------------------------------------------- | | monitor | Cluster | NA | | create\_index | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | | index | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | | read | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | | write | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | | manage | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | | bulk | Index | `inference-log-*`, `reference-log-*`, `drift-log-*` | ## Configure EFK Ingress (Optional) ### Kibana It can be useful to access Kibana's UI without having to port-forward. To expose Kibana externally it needs to have its own path. This means that we need to modify our `kibana.yaml` to include an extra `spec.config` and `spec.http` section: ```yaml spec: config: server.basePath: /kibana http: tls: selfSignedCertificate: disabled: true ``` The base path is required as Kibana be default [runs behind a proxy that adds a random path component to its URL](https://www.elastic.co/guide/en/kibana/7.17/development-basepath.html) The `tls` section is required to disable the self-signed certificate that Kibana uses by default, so that we can use our own certificate. Then reapply the `kibana.yaml`: ```bash kubectl apply -f kibana.yaml ``` Next, configure either an Istio `VirtualService` .\ The following `VirtualService` for `Kiban`\`, created in `kibana-vs.yaml`, enables its ingress: ```yaml apiVersion: networking.istio.io/v1beta1 kind: VirtualService metadata: name: kibana namespace: seldon-logs spec: gateways: - istio-system/seldon-gateway hosts: - '*' http: - match: - uri: prefix: /kibana/ - uri: prefix: /kibana rewrite: uri: / route: - destination: host: seldon-kb-http port: number: 5601 ``` Apply the configuration using the command `kubectl apply -f kibana-vs.yaml`. You can access Kibana at `/kibana`. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.seldon.ai/seldon-enterprise-platform/production-environment/elasticsearch.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.