OpenSearch

Installation for OpenSearch, a fully open-source alternative to Elasticsearch.

OpenSearch Installation

OpenSearch is an external component outside of the main Seldon stack.
Therefore, it is the cluster administrator's responsibility to administrate and manage the OpenSearch instance used by Seldon.

Compatibility

The below table summarises the compatibility between different components in the Seldon ecosystem and different release lines of OpenSearch.

Component	Compatible with OpenSearch 1.x	Compatible with OpenSearch 2.x
Seldon Enterprise Platform 2.4.0	Yes	Yes
Metronome >=1.8.2	No	No
Metronome 1.1...1.8.1	Yes	Yes
Metronome 1.0	Yes	No
Fluentd	Yes	With a plugin

Initial Configuration

Copy default Fluentd and OpenSearch Helm config files (and edit if desired)

cp ./seldon-deploy-install/reference-configuration/efk/values-fluentd.yaml values-fluentd.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch.yaml values-opensearch.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch-dashboards.yaml values-opensearch-dashboards.yaml

Ensure Required Namespaces Exist

We'll be installing in the seldon-logs namespace. We'll also set up some config in the seldon-system namespace.

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"
kubectl create namespace seldon-system || echo "namespace seldon-system exists"

Authentication

Authentication is optional for OpenSearch, but enabled by default.

See the security plugin documentation for how to disable it.

Using authentication means components interacting with OpenSearch will need secrets, including Seldon Enterprise Platform. We'll need secrets in the seldon-logs namespace (for the request logger) and the seldon-system namespace (for Enterprise Platform).

Using the OpenSearch defaults, we can set:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

It is possible to set different credentials for the default user(s) in the internal user database. For further information on this, please refer to the Open Distro documentation, as the process is fundamentally the same; exact filesystem paths may differ.

Once updated, these credentials can be provided to Fluentd in its Helm values file.

If you have chosen to install OpenSearch Dashboards (equivalent to Kibana), you can configure the credentials to access OpenSearch using the Helm values.

Authorization

The Seldon Enterprise Platform setup needs authorization on the OpenSearch cluster to create, manage and search indexes for prediction logging and other monitoring features. Read more about OpenSearch action groups here.

The following permissions are mandatory for proper functioning of the current Seldon Enterprise Platform features.

• Seldon Enterprise Platform User Action Groups

Action Group	Permission Level	Index-pattern
cluster_monitor	Cluster	NA
read	Index	inference-log-*
read	Index	reference-log-*
read	Index	drift-log-*
search	Index	inference-log-*
search	Index	reference-log-*
search	Index	drift-log-*
search	Index	kubernetes_cluster-*
search	Index	*

• Seldon Request Logger User Action Groups

Action Group	Permission Level	Index-patterns
cluster_monitor	Cluster	NA
create_index	Index	inference-log-*, reference-log-*, drift-log-*
get	Index	inference-log-*, reference-log-*, drift-log-*
read	Index	inference-log-*, reference-log-*, drift-log-*
write	Index	inference-log-*, reference-log-*, drift-log-*
manage	Index	inference-log-*, reference-log-*, drift-log-*
bulk	Index	inference-log-*, reference-log-*, drift-log-*

The authorization can also be done with access control permissions. Read more about OpenSearch permissions here.

• Seldon Enterprise Platform User Permissions

Permission	Permission Level	Index-patterns
cluster:monitor/main	Cluster	NA
cluster:monitor/health	Cluster	NA
indices:admin/get	Index	inference-log-*, reference-log-*, drift-log-*
indices:data/read/search*	Index	inference-log-*, reference-log-*, drift-log-*, kubernetes_cluster-*, *

• Seldon Request Logger User Permissions

Permission	Permission Level	Index-patterns
cluster:monitor/main	Cluster	NA
indices:admin/get	Index	inference-log-*, reference-log-*, drift-log-*
indices:admin/create	Index	inference-log-*, reference-log-*, drift-log-*
indices:admin/mapping/put	Index	inference-log-*, reference-log-*, drift-log-*
indices:data/write/index	Index	inference-log-*, reference-log-*, drift-log-*
indices:data/write/update	Index	inference-log-*, reference-log-*, drift-log-*
indices:data/write/bulk	Index	inference-log-*, reference-log-*, drift-log-*
indices:data/write/bulk*	Index	inference-log-*, reference-log-*, drift-log-*

OpenSearch

Install OpenSearch using the following script:

helm upgrade --install opensearch \
  opensearch --repo https://opensearch-project.github.io/helm-charts/ \
  --version 1.11.1 \
  --namespace=seldon-logs \
  --values=values-opensearch.yaml

You can wait for it to come up with:

kubectl rollout status -n seldon-logs statefulset/opensearch-cluster-master

OpenSearch Dashboards

Note: OpenSearch Dashboards is not a required dependency for Seldon Enterprise Platform and as such is optional.

OpenSearch Dashboards is equivalent to Kibana for Elasticsearch.

Install it using the following:

helm upgrade --install opensearch-dashboards \
  opensearch-dashboards --repo https://opensearch-project.github.io/helm-charts/ \
  --version 1.5.1 \
  --namespace=seldon-logs \
  --values=values-opensearch-dashboards.yaml

Fluentd

Install Fluentd with this:

helm upgrade --install fluentd \
  fluentd-elasticsearch --repo https://kokuwaio.github.io/helm-charts \
  --version 10.0.1 \
  --namespace=seldon-logs \
  --values=values-fluentd.yaml

Note that if alternative creds are used then these need to be set in the Fluentd Helm values file.

Configure Seldon Enterprise Platform

Not all request logger versions are compatible with OpenSearch. Currently only metronome versions <=1.7.1 are compatible.

The following Helm values need to be set in install-values.yaml.

requestLogger:
  create: true
  image: seldonio/metronome:1.7.1 # cannot be higher than 1.7.1
  elasticsearch:
    host: opensearch-cluster-master.seldon-logs.svc.cluster.local
    port: "9200"
    protocol: https
  trigger:
    create: true # false if not using Knative

elasticsearch:
  basicAuth: true
  url: https://opensearch-cluster-master.seldon-logs.svc.cluster.local:9200

If you did not install Knative Eventing, requestLogger.trigger.create has to be set to false.

Verify Installation

You can access the OpenSearch cluster by port-forwarding the service and sending a basic API request:

kubectl port-forward -n seldon-logs svc/opensearch-cluster-master 9200
curl --insecure -X GET -u admin:admin https://localhost:9200

You should see output like the below:

{
  "name": "opensearch-cluster-master-0",
  "cluster_name": "opensearch-cluster",
  "cluster_uuid": "IAbgl0dqSoiSyMuyvXpcHw",
  "version": {
    "distribution": "opensearch",
    "number": "1.3.2",
    "build_type": "tar",
    "build_hash": "6febcf7b53ff189de767e460e905e9e5aeecc8cb",
    "build_date": "2022-05-04T03:58:48.328641Z",
    "build_snapshot": false,
    "lucene_version": "8.10.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "The OpenSearch Project: https://opensearch.org/"
}