OpenSearch

Installation for OpenSearch, a fully open-source alternative to Elasticsearch.

OpenSearch Installation

OpenSearch is an external component outside of the main Seldon stack.
Therefore, it is the cluster administrator's responsibility to administrate and manage the OpenSearch instance used by Seldon.

Compatibility

The below table summarises the compatibility between different components in the Seldon ecosystem and different release lines of OpenSearch.

Component
Compatible with OpenSearch 1.x
Compatible with OpenSearch 2.x

Seldon Enterprise Platform 2.4.0

Yes

Yes

Metronome >=1.8.2

No

No

Metronome 1.1...1.8.1

Yes

Yes

Metronome 1.0

Yes

No

Fluentd

Yes

With a plugin

Initial Configuration

Copy default Fluentd and OpenSearch Helm config files (and edit if desired)

cp ./seldon-deploy-install/reference-configuration/efk/values-fluentd.yaml values-fluentd.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch.yaml values-opensearch.yaml
cp ./seldon-deploy-install/reference-configuration/efk/values-opensearch-dashboards.yaml values-opensearch-dashboards.yaml

Ensure Required Namespaces Exist

We'll be installing in the seldon-logs namespace. We'll also set up some config in the seldon-system namespace.

kubectl create namespace seldon-logs || echo "namespace seldon-logs exists"
kubectl create namespace seldon-system || echo "namespace seldon-system exists"

Authentication

Authentication is optional for OpenSearch, but enabled by default.

See the security plugin documentation for how to disable it.

Using authentication means components interacting with OpenSearch will need secrets, including Seldon Enterprise Platform. We'll need secrets in the seldon-logs namespace (for the request logger) and the seldon-system namespace (for Enterprise Platform).

Using the OpenSearch defaults, we can set:

ELASTIC_USER=admin
ELASTIC_PASSWORD=admin

kubectl create secret generic elastic-credentials -n seldon-logs \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic elastic-credentials -n seldon-system \
  --from-literal=username="${ELASTIC_USER}" \
  --from-literal=password="${ELASTIC_PASSWORD}" \
  --dry-run=client -o yaml | kubectl apply -f -

It is possible to set different credentials for the default user(s) in the internal user database. For further information on this, please refer to the Open Distro documentation, as the process is fundamentally the same; exact filesystem paths may differ.

Once updated, these credentials can be provided to Fluentd in its Helm values file.

If you have chosen to install OpenSearch Dashboards (equivalent to Kibana), you can configure the credentials to access OpenSearch using the Helm values.

Authorization

The Seldon Enterprise Platform setup needs authorization on the OpenSearch cluster to create, manage and search indexes for prediction logging and other monitoring features. Read more about OpenSearch action groups here.

The following permissions are mandatory for proper functioning of the current Seldon Enterprise Platform features.

  • Seldon Enterprise Platform User Action Groups

Action Group
Permission Level
Index-pattern

cluster_monitor

Cluster

NA

read

Index

inference-log-*

read

Index

reference-log-*

read

Index

drift-log-*

search

Index

inference-log-*

search

Index

reference-log-*

search

Index

drift-log-*

search

Index

kubernetes_cluster-*

search

Index

*

  • Seldon Request Logger User Action Groups

Action Group
Permission Level
Index-patterns

cluster_monitor

Cluster

NA

create_index

Index

inference-log-*, reference-log-*, drift-log-*

get

Index

inference-log-*, reference-log-*, drift-log-*

read

Index

inference-log-*, reference-log-*, drift-log-*

write

Index

inference-log-*, reference-log-*, drift-log-*

manage

Index

inference-log-*, reference-log-*, drift-log-*

bulk

Index

inference-log-*, reference-log-*, drift-log-*

The authorization can also be done with access control permissions. Read more about OpenSearch permissions here.

  • Seldon Enterprise Platform User Permissions

Permission
Permission Level
Index-patterns

cluster:monitor/main

Cluster

NA

cluster:monitor/health

Cluster

NA

indices:admin/get

Index

inference-log-*, reference-log-*, drift-log-*

indices:data/read/search*

Index

inference-log-*, reference-log-*, drift-log-*, kubernetes_cluster-*, *

  • Seldon Request Logger User Permissions

Permission
Permission Level
Index-patterns

cluster:monitor/main

Cluster

NA

indices:admin/get

Index

inference-log-*, reference-log-*, drift-log-*

indices:admin/create

Index

inference-log-*, reference-log-*, drift-log-*

indices:admin/mapping/put

Index

inference-log-*, reference-log-*, drift-log-*

indices:data/write/index

Index

inference-log-*, reference-log-*, drift-log-*

indices:data/write/update

Index

inference-log-*, reference-log-*, drift-log-*

indices:data/write/bulk

Index

inference-log-*, reference-log-*, drift-log-*

indices:data/write/bulk*

Index

inference-log-*, reference-log-*, drift-log-*

OpenSearch

Install OpenSearch using the following script:

helm upgrade --install opensearch \
  opensearch --repo https://opensearch-project.github.io/helm-charts/ \
  --version 1.11.1 \
  --namespace=seldon-logs \
  --values=values-opensearch.yaml

You can wait for it to come up with:

kubectl rollout status -n seldon-logs statefulset/opensearch-cluster-master

OpenSearch Dashboards

Note: OpenSearch Dashboards is not a required dependency for Seldon Enterprise Platform and as such is optional.

OpenSearch Dashboards is equivalent to Kibana for Elasticsearch.

Install it using the following:

helm upgrade --install opensearch-dashboards \
  opensearch-dashboards --repo https://opensearch-project.github.io/helm-charts/ \
  --version 1.5.1 \
  --namespace=seldon-logs \
  --values=values-opensearch-dashboards.yaml

Fluentd

Install Fluentd with this:

helm upgrade --install fluentd \
  fluentd-elasticsearch --repo https://kokuwaio.github.io/helm-charts \
  --version 10.0.1 \
  --namespace=seldon-logs \
  --values=values-fluentd.yaml

Note that if alternative creds are used then these need to be set in the Fluentd Helm values file.

Configure Seldon Enterprise Platform

Not all request logger versions are compatible with OpenSearch. Currently only metronome versions <=1.7.1 are compatible.

The following Helm values need to be set in install-values.yaml.

requestLogger:
  create: true
  image: seldonio/metronome:1.7.1 # cannot be higher than 1.7.1
  elasticsearch:
    host: opensearch-cluster-master.seldon-logs.svc.cluster.local
    port: "9200"
    protocol: https
  trigger:
    create: true # false if not using Knative

elasticsearch:
  basicAuth: true
  url: https://opensearch-cluster-master.seldon-logs.svc.cluster.local:9200

If you did not install Knative Eventing, requestLogger.trigger.create has to be set to false.

Verify Installation

You can access the OpenSearch cluster by port-forwarding the service and sending a basic API request:

kubectl port-forward -n seldon-logs svc/opensearch-cluster-master 9200
curl --insecure -X GET -u admin:admin https://localhost:9200

You should see output like the below:

{
  "name": "opensearch-cluster-master-0",
  "cluster_name": "opensearch-cluster",
  "cluster_uuid": "IAbgl0dqSoiSyMuyvXpcHw",
  "version": {
    "distribution": "opensearch",
    "number": "1.3.2",
    "build_type": "tar",
    "build_hash": "6febcf7b53ff189de767e460e905e9e5aeecc8cb",
    "build_date": "2022-05-04T03:58:48.328641Z",
    "build_snapshot": false,
    "lucene_version": "8.10.1",
    "minimum_wire_compatibility_version": "6.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1"
  },
  "tagline": "The OpenSearch Project: https://opensearch.org/"
}
PreviousElasticSearchNextRequest Logging

Last updated