All pages
Powered by GitBook
1 of 1

Loading...

Upgrading to 1.5.0

Secrets Management

  • Seldon Deploy 1.5.0 introduces the ability to manage secrets for registry and bucket access directly through the UI.

  • The default Minio secret used in some installations has been renamed from seldon-rclone-secret to minio-bucket, and the URI remote has changed from s3:// to minio://

  • Secrets fields are now dropdowns rather than free text fields. Secrets created through the Deploy interface or API will automatically appear, but to allow your existing secrets to show in the dropdown please label your Kubernetes secret resources with the following:

  • Seldon Deploy 1.5 introduces a new request logger developed by Seldon called metronome to replace the . It is available as a public docker image at , where tag 1.0.1 is the default version being used by Seldon Deploy 1.5.

  • Note that metronome is a required replacement for the old request logger as several new features introduced by Seldon Deploy 1.5 depend on it.

  • Seldon deploy 1.5 allows node level prediction data logging and feature distributions. Because of that, the Elasticsearch index pattern used by the feature distributions dashboard has been updated to include the node name/model id of the predictor, which is the pattern that the new metronome request logger uses.

  • This means that data from previous versions (<v1.5.0) of Seldon Deploy using the ), will no longer be found on the monitoring - feature distributions dashboard since the index pattern is different.

  • This also applies to reference data

Name
Previous value
Current value
If upgrading from an older version of Seldon Deploy to 1.5, please ensure that the following section has been updated to use metronome for request logging:
.
  • New requests that are stored in this new index using metronome will be available

  • In order to re-use requests logged using the old python request logger, users may manually reindex the old requests on Elasticsearch:

    1. For example, given a Seldon Deployment with the following spec (this can be viewed on the Deployment Dashboard), the node name/model id of the model is income-container:

      predictors:
        - graph:
            implementation: SKLEARN_SERVER
            modelUri: 'gs://seldon-models/sklearn/income/model-0.23.2'
            name: income-container
          name: default
    2. Port forward the Elasticsearch port to localhost:9200:

      kubectl port-forward svc/elasticsearch-opendistro-es-client-service 9200:9200

      For convenience, we export the address to the variable ES_ADDR:

    3. Get the names of the old and new index. The pattern of the old index should follow the pattern inference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint> while the new index would be inference-log-<serving engine>-<deployment namespace>-<deployment name>-<deployment endpoint>-<deployment node>. If the above Seldon Deployment is in the dev namespace, the old index would be inference-log-seldon-dev-income-default, whereas the new index would be inference-log-seldon-dev-income-default-income-container. The following command is useful:

      We export and refer to the old and new indices as follows:

    4. Get the mapping of the old index. Store the mapping in a shell variable MAPPINGS as follows:

    5. Create the new index with the same mappings as the old index.

    6. Reindex the old requests to the new index with the following command:

  • batchjobs.processor.image

    seldonio/seldon-core-s2i-python37:1.11.2

    seldonio/seldon-core-s2i-python37:1.13.1

    batchjobs.storageInitializer.image

    seldonio/rclone-storage-initializer:1.11.2

    seldonio/rclone-storage-initializer:1.13.1

    seldon.v2ExplainForm

    "http://{{ .ModelName }}-{{ .Predictor }}-explainer.{{ .Namespace }}:9000/v2/models/{{ .GraphModelName }}/explain"

    "http://{{ .ModelName }}-{{ .Predictor }}-explainer.{{ .Namespace }}:9000/explain"

    kfserving.enabled

    true

    false

    seldon-deploy=true
    secret-type=<secret-type> # where <secret-type> is registry or bucket

    requestLogger.image

    seldonio/seldon-request-logger:1.11.2

    seldonio/metronome:1.0.1

    alibidetect.image

    seldonio/alibi-detect-server:1.11.2

    seldonio/alibi-detect-server:1.13.1

    ML Data Logging - Metronome

    Monitoring - Feature Distributions

    old request logger
    https://hub.docker.com/r/seldonio/metronome
    old request logger
    requestLogger:
      create: true
      image: seldonio/metronome:1.0.1
    export ES_ADDR=http://localhost:9200
    curl --location --request GET 'https://localhost:9200/_cat/indices/inference-log-*'
    export OLD_INDEX=inference-log-seldon-dev-income-default
    export NEW_INDEX=inference-log-seldon-dev-income-default-income-container
    MAPPINGS=$(curl --request GET "${ES_ADDR}/${OLD_INDEX}?pretty"  | jq ".\"${OLD_INDEX}\".mappings")
    curl --request PUT "${ES_ADDR}/${NEW_INDEX}" --header 'Content-Type: application/json' --data-raw "{\"mappings\": ${MAPPINGS}}"
    
    curl --request POST "${ES_ADDR}/_reindex" \
      --headertomised your Deploy installation, please be aware that the following Helm values have changed since v1.4.0: