Disruption Budgets Example

Prerequisites

• A kubernetes cluster with kubectl configured

• pygmentize

Setup Seldon Core

Use the setup notebook to Setup Cluster with Ambassador Ingress.

!kubectl create namespace seldon

Error from server (AlreadyExists): namespaces "seldon" already exists

Create model with Pod Disruption Budget

To create a model with a Pod Disruption Budget, it is first important to understand how you would like your application to respond to voluntary disruptions. Depending on the type of disruption budgeting your application needs, you will either define either of the following:

• minAvailable which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod. minAvailable can be either an absolute number or a percentage.

• maxUnavailable which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage.

The full SeldonDeployment spec is shown below.

!pygmentize model_with_pdb.yaml

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  replicas: 2
  predictors:
  - componentSpecs:
    - pdbSpec:
        maxUnavailable: 2
      spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          imagePullPolicy: IfNotPresent
          name: classifier
          resources:
            requests:
              cpu: '0.5'
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example

!kubectl apply -f model_with_pdb.yaml -n seldon

seldondeployment.machinelearning.seldon.io/seldon-model created

!kubectl wait sdep/seldon-model \
  --for=condition=ready \
  --timeout=120s \
  -n seldon

seldondeployment.machinelearning.seldon.io/seldon-model condition met

Validate Disruption Budget Configuration

import json


def getPdbConfig():
    dp = !kubectl get pdb -n seldon seldon-model-example-0-classifier -o json
    dp = json.loads("".join(dp))
    return dp["spec"]["maxUnavailable"]


assert getPdbConfig() == 2

!kubectl get pods,deployments,pdb -n seldon

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/seldon-model-example-0-classifier-7964b5c9f8-89x9h   2/2     Running   0          29s
pod/seldon-model-example-0-classifier-7964b5c9f8-p5fg9   2/2     Running   0          29s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/seldon-model-example-0-classifier   2/2     2            2           29s

NAME                                                           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
poddisruptionbudget.policy/seldon-model-example-0-classifier   N/A             2                 2                     29s

Update Disruption Budget and Validate Change

Next, we'll update the maximum number of unavailable pods and check that the PDB is properly updated to match.

!pygmentize model_with_patched_pdb.yaml

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  replicas: 2
  predictors:
  - componentSpecs:
    - pdbSpec:
        maxUnavailable: 1
      spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          imagePullPolicy: IfNotPresent
          name: classifier
          resources:
            requests:
              cpu: '0.5'
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example

!kubectl apply -f model_with_patched_pdb.yaml -n seldon

seldondeployment.machinelearning.seldon.io/seldon-model configured

!kubectl wait sdep/seldon-model \
  --for=condition=ready \
  --timeout=120s \
  -n seldon

seldondeployment.machinelearning.seldon.io/seldon-model condition met

assert getPdbConfig() == 1

Clean Up

!kubectl get pods,deployments,pdb -n seldon

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/seldon-model-example-0-classifier-778dc959fd-dhts9   2/2     Running   0          33s
pod/seldon-model-example-0-classifier-778dc959fd-dpsnc   0/2     Running   0          9s
pod/seldon-model-example-0-classifier-7964b5c9f8-89x9h   2/2     Running   0          73s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/seldon-model-example-0-classifier   2/2     2            2           73s

NAME                                                           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
poddisruptionbudget.policy/seldon-model-example-0-classifier   N/A             1                 1                     73s

!kubectl delete -f model_with_patched_pdb.yaml -n seldon

seldondeployment.machinelearning.seldon.io "seldon-model" deleted