Upgrading to 2.2.0

Upgrading to Seldon Enterprise Platform from previous versions of Seldon Deploy (Seldon Enterprise Platform now).

Regular Installation Upgrade

Seldon Core

For Seldon Core v2, we now recommend using version minor versions of 2.7.0 or later. More information on this can be found in the Core v2 installation documentation.

When upgrading to 2.7.0 you can now choose to install the operator namespaced or cluster-wide. First of all, you'll have to install the upgraded CRD chart for 2.7.0. Afterwards, there are a couple different scenarios that can happen now (all examples below assume the Core v2 operator was installed in namespace seldon):

Upgrade from 2.5.0 to 2.7.0 namespaced operator

• add a controller.clusterwide: false flag to the Helm values setup chart

• upgrade the seldon-core-v2-setup chart to 2.7.0 in seldon namespace

• install the Seldon Runtime in seldon namespace

Upgrade from 2.5.0 to 2.7.0 cluster-wide operator in same setup chart as model

• add a controller.clusterwide: true flag to the Helm values setup chart

• upgrade the seldon-core-v2-setup chart to 2.7.0 in seldon namespace

• install Seldon Runtime in seldon namespace

• add Seldon Runtime to all other namespaces that host models

Upgrade from 2.5.0 to 2.7.0 with a cluster-wide operator in seldon-system namespace

• add a controller.clusterwide: true flag to the Helm values setup chart

• remove the previously installed seldon-core-v2-setup Helm chart in seldon namespace (i.e. the namespace that was used before)

• add the setup chart to seldon-system namespace with version 2.7.0

• install Seldon Runtime in seldon namespace and all other namespaces that host models

If you're using Seldon Servers Helm Chart, upgrade / install it like the Seldon Runtime in every namespace that host models. We recommend the third approach but note that in production this will quite possibly lead to timeouts of Seldon services for a short while. More information on the installation can be found in the Core v2 installation documentation as well.

Seldon Enterprise Platform Installation Upgrade

To upgrade Seldon Enterprise Platform itself, you should upgrade your Helm values and run the helm upgrade ... command as described here. In the following sections we describe the changes that you need to make to your Helm values and you can execute these with:

helm upgrade seldon-deploy seldon-charts/seldon-deploy \
    -f deploy-values.yaml \
    --namespace=seldon-system \
    --version 2.2.0 \
    --install

Helm values: Enterprise Platform

Within the deploy-values.yaml, which you can create or re-use from the installation, you should update the image to:

image:
  image: seldonio/seldon-deploy-server:2.2.0

This will update Seldon Enterprise Platform to version 2.2.0.

Helm values: Elasticsearch and Request Logging

If you've used OpenSearch for request logging and want to continue using it, note that only Metronome 1.7.1 and below is compatible with OpenSearch.

requestLogger:
  image: seldonio/metronome:1.7.1 # cannot be higher than 1.7.1

If you've used Elasticsearch for request logging, there is no need to modify your default helm values.

We recommend using the Elastic version of Elasticsearch for production use. Note that we only guarantee support for Elasticsearch 7.X.

Please refer to the Elasticsearch documentation for more details.

Helm values: Theme Customization

If you've used the Theme Customization feature before, remove the volume and volume-mounts associated with the custom theme from your Helm charts as this feature is no longer supported.

Helm values: Drift Detection

The default image used for drift detection has changed. If you modified this Helm value you may need to update it manually in order to enable alerting on drift.

alibidetect:
  image: seldonio/alibi-detect-server:1.17.1

Helm values: ClientGo Rate Limiter

You can now configure the QPS and the burst rate settings for the client-go's rate limiter. In this way, you can control the traffic sent from this library to the K8s api-server based on your load. Previously, the values being used were 5 and 10 respectively, as set as default in client-go. You can configure these Helm values as follows:

clientGo:
  rateLimiter:
    qps: 250
    burst: 250

You can find more information on the client-go rate limiter in the official documentation.

Update Monitoring: Core v2 labels

From version 2.7.0 Core v2 components provisioned through the SeldonRuntime operator will now have the label app.kubernetes.io/name for identifying the pods. Previously, the app label has been inconsistent across different versions of Seldon Core v2 (a mixture of app and app.kubernetes.io/name labels was used), so users should modify or configure their existing podmonitors accordingly. See the corresponding section in the Seldon Core v2 installation documentation on how to create / update the required monitors.

Update Monitoring: Drift Alerting

A new Pod Monitor resource has been added to collect metrics required for alerting on drift for Core v1 deployments. To add new PodMonitor copy it from reference-configuration, modify if required, and apply to your cluster as described in documentation here

cp seldon-deploy-install/reference-configuration/metrics/drift-monitor.yaml drift-monitor.yaml
kubectl apply -n seldon-monitoring -f drift-monitor.yaml

New Prometheus Rules for alerting on drift for both Core v1 and Core v2 have been added. To add new alerts, copy them from reference-configuration, modify if required, and apply to your cluster as described in the documentation here.

cp seldon-deploy-install/reference-configuration/metrics/drift-alerts.yaml drift-alerts.yaml
kubectl apply -n seldon-monitoring -f drift-alerts.yaml

OpenShift Installation Upgrade

Full documentation for installing Seldon Enterprise Platform on OpenShift clusters can be found here.

Helm Values

Unless specified otherwise here, upgrade Helm values as described in the notes for Regular Installation Upgrade.

Update Monitoring: Core v2 labels

As described in the notes for Regular Installation Upgrade labels on certain Core v2 components have changed. Make sure that you update your Pod Monitors using new resources as described in the documentation here.

Update Monitoring: Drift Alerting

As described in the notes for Regular Installation Upgrade new Pod Monitor and Alerting Rules are required to enable alerting on drift:

• seldon-podmonitor.yaml manifest has been extended to include PodMonitor named seldon-drift-detector, see documentation here

• drift-alerts.yaml manifest has been added containing required PrometheusRules, see documentation here