Upgrading to 1.5.0

Secrets Management

• Seldon Deploy 1.5.0 introduces the ability to manage secrets for registry and bucket access directly through the UI.

• The default Minio secret used in some installations has been renamed from seldon-rclone-secret to minio-bucket, and the URI remote has changed from s3:// to minio://

• Secrets fields are now dropdowns rather than free text fields. Secrets created through the Deploy interface or API will automatically appear, but to allow your existing secrets to show in the dropdown please label your Kubernetes secret resources with the following:

  seldon-deploy=true
  secret-type=<secret-type> # where <secret-type> is registry or bucket

ML Data Logging - Metronome

• Seldon Deploy 1.5 introduces a new request logger developed by Seldon called metronome to replace the old request logger. It is available as a public docker image at https://hub.docker.com/r/seldonio/metronome, where tag 1.0.1 is the default version being used by Seldon Deploy 1.5.

• Note that metronome is a required replacement for the old request logger as several new features introduced by Seldon Deploy 1.5 depend on it.

• If upgrading from an older version of Seldon Deploy to 1.5, please ensure that the following section has been updated to use metronome for request logging:

  requestLogger:
    create: true
    image: seldonio/metronome:1.0.1

Monitoring - Feature Distributions

• Seldon deploy 1.5 allows node level prediction data logging and feature distributions. Because of that, the Elasticsearch index pattern used by the feature distributions dashboard has been updated to include the node name/model id of the predictor, which is the pattern that the new metronome request logger uses.

• This means that data from previous versions (<v1.5.0) of Seldon Deploy using the old request logger), will no longer be found on the monitoring - feature distributions dashboard since the index pattern is different.

• This also applies to reference data.

• New requests that are stored in this new index using metronome will be available

• In order to re-use requests logged using the old python request logger, users may manually reindex the old requests on Elasticsearch:

  1. For example, given a Seldon Deployment with the following spec (this can be viewed on the Deployment Dashboard), the node name/model id of the model is income-container:

     predictors:
       - graph:
           implementation: SKLEARN_SERVER
           modelUri: 'gs://seldon-models/sklearn/income/model-0.23.2'
           name: income-container
         name: default

  2. Port forward the Elasticsearch port to localhost:9200:

     kubectl port-forward svc/elasticsearch-opendistro-es-client-service 9200:9200

     For convenience, we export the address to the variable ES_ADDR:

     export ES_ADDR=http://localhost:9200

  3. Get the names of the old and new index. The pattern of the old index should follow the pattern inference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint> while the new index would be inference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint>-<deployment node>. If the above Seldon Deployment is in the dev namespace, the old index would be inference-log-seldon-dev-income-default, whereas the new index would be inference-log-seldon-dev-income-default-income-container. The following command is useful:

     curl --location --request GET 'https://localhost:9200/_cat/indices/inference-log-*'

     We export and refer to the old and new indices as follows:

     export OLD_INDEX=inference-log-seldon-dev-income-default
     export NEW_INDEX=inference-log-seldon-dev-income-default-income-container

  4. Get the mapping of the old index. Store the mapping in a shell variable MAPPINGS as follows:

     MAPPINGS=$(curl --request GET "${ES_ADDR}/${OLD_INDEX}?pretty"  | jq ".\"${OLD_INDEX}\".mappings")

  5. Create the new index with the same mappings as the old index.

     curl --request PUT "${ES_ADDR}/${NEW_INDEX}" --header 'Content-Type: application/json' --data-raw "{\"mappings\": ${MAPPINGS}}"
     

  6. Reindex the old requests to the new index with the following command:

     curl --request POST "${ES_ADDR}/_reindex" \
       --headertomised your Deploy installation, please be aware that the following Helm values have changed since v1.4.0:

Name	Previous value	Current value
requestLogger.image	seldonio/seldon-request-logger:1.11.2	seldonio/metronome:1.0.1
alibidetect.image	seldonio/alibi-detect-server:1.11.2	seldonio/alibi-detect-server:1.13.1
batchjobs.processor.image	seldonio/seldon-core-s2i-python37:1.11.2	seldonio/seldon-core-s2i-python37:1.13.1
batchjobs.storageInitializer.image	seldonio/rclone-storage-initializer:1.11.2	seldonio/rclone-storage-initializer:1.13.1
seldon.v2ExplainForm	"http://{{ .ModelName }}-{{ .Predictor }}-explainer.{{ .Namespace }}:9000/v2/models/{{ .GraphModelName }}/explain"	"http://{{ .ModelName }}-{{ .Predictor }}-explainer.{{ .Namespace }}:9000/explain"
kfserving.enabled	true	false