Checking Pipeline readiness

Local example settings.

%env INFER_REST_ENDPOINT=http://0.0.0.0:9000
%env INFER_GRPC_ENDPOINT=0.0.0.0:9000
%env SELDON_SCHEDULE_HOST=0.0.0.0:9004

env: INFER_REST_ENDPOINT=http://0.0.0.0:9000
env: INFER_GRPC_ENDPOINT=0.0.0.0:9000
env: SELDON_SCHEDULE_HOST=0.0.0.0:9004

Remote k8s cluster example settings - change as neeed for your needs.

#%env INFER_REST_ENDPOINT=http://172.19.255.1:80
#%env INFER_GRPC_ENDPOINT=172.19.255.1:80
#%env SELDON_SCHEDULE_HOST=172.19.255.2:9004

Model Chain - Ready Check

We will check the readiness of the Pipeline after every change to model and pipeline.

cat ./pipelines/tfsimples.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: tfsimples
spec:
  steps:
    - name: tfsimple1
    - name: tfsimple2
      inputs:
      - tfsimple1
      tensorMap:
        tfsimple1.outputs.OUTPUT0: INPUT0
        tfsimple1.outputs.OUTPUT1: INPUT1
  output:
    steps:
    - tfsimple2

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

ERROR:
  Code: Unimplemented
  Message:

kubectl

kubectl apply -f ./pipelines/tfsimples.yaml -n ${NAMESPACE}

pipeline.mlops.seldon.io/tfsimples created

kubectl wait --for condition=ready --timeout=300s pipeline tfsimples -n ${NAMESPACE}

seldon-cli

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

{

}

kubectl

kubectl apply -f ./models/tfsimple1.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model tfsimple1 -n ${NAMESPACE}

model.mlops.seldon.io/tfsimple1 created
model.mlops.seldon.io/tfsimple1 condition met

seldon-cli

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

{

}

kubectl

kubectl apply -f ./models/tfsimple2.yaml -n ${NAMESPACE}
kubectl wait --for condition=ready --timeout=300s model tfsimple2 -n ${NAMESPACE}

model.mlops.seldon.io/tfsimple2 created
model.mlops.seldon.io/tfsimple2 condition met

seldon-cli

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

{
  "ready": true
}

seldon pipeline status tfsimples | jq .versions[0].state.modelsReady

true

seldon pipeline unload tfsimples

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

ERROR:
  Code: Unimplemented
  Message:

Models will still be ready even though Pipeline terminated

seldon pipeline status tfsimples | jq .versions[0].state.modelsReady

true

seldon pipeline load -f ./pipelines/tfsimples.yaml
seldon pipeline status tfsimples -w PipelineReady

{"pipelineName":"tfsimples", "versions":[{"pipeline":{"name":"tfsimples", "uid":"ciepj5qi8ufs73flaiu0", "version":1, "steps":[{"name":"tfsimple1"}, {"name":"tfsimple2", "inputs":["tfsimple1.outputs"], "tensorMap":{"tfsimple1.outputs.OUTPUT0":"INPUT0", "tfsimple1.outputs.OUTPUT1":"INPUT1"}}], "output":{"steps":["tfsimple2.outputs"]}, "kubernetesMeta":{}}, "state":{"pipelineVersion":1, "status":"PipelineReady", "reason":"created pipeline", "lastChangeTimestamp":"2023-06-29T14:47:51.626155116Z", "modelsReady":true}}]}

curl -Ik ${INFER_REST_ENDPOINT}/v2/pipelines/tfsimples/ready

grpcurl -d '{"name":"tfsimples"}' \
    -plaintext \
    -import-path ../apis \
    -proto ../apis/mlops/v2_dataplane/v2_dataplane.proto \
    -rpc-header seldon-model:tfsimples.pipeline \
    ${INFER_GRPC_ENDPOINT} inference.GRPCInferenceService/ModelReady

{
  "ready": true
}

seldon pipeline status tfsimples | jq .versions[0].state.modelsReady

true

seldon model unload tfsimple1
seldon model unload tfsimple2

seldon pipeline status tfsimples | jq .versions[0].state.modelsReady

null

seldon pipeline unload tfsimples

Kubernetes Resource Example

Note: The Seldon CLI allows you to view information about underlying Seldon resources and make changes to them through the scheduler in non-Kubernetes environments. However, it cannot modify underlying manifests within a Kubernetes cluster. Therefore, using the Seldon CLI for control plane operations in a Kubernetes environment is not recommended. For more details, see Seldon CLI.

import os
os.environ["NAMESPACE"] = "seldon-mesh"

MESH_IP=!kubectl get svc seldon-mesh -n ${NAMESPACE} -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
MESH_IP=MESH_IP[0]
import os
os.environ['MESH_IP'] = MESH_IP
MESH_IP

'172.19.255.1'

kubectl create -f ./pipelines/tfsimples.yaml -n ${NAMESPACE}

pipeline.mlops.seldon.io/tfsimples created

kubectl wait --for condition=ready --timeout=1s pipeline --all -n ${NAMESPACE}

error: timed out waiting for the condition on pipelines/tfsimples

kubectl get pipeline tfsimples -o jsonpath='{.status.conditions[0]}' -n ${NAMESPACE}

{"lastTransitionTime":"2022-11-14T10:25:31Z","status":"False","type":"ModelsReady"}

kubectl create -f ./models/tfsimple1.yaml -n ${NAMESPACE}
kubectl create -f ./models/tfsimple2.yaml -n ${NAMESPACE}

model.mlops.seldon.io/tfsimple1 created
model.mlops.seldon.io/tfsimple2 created

kubectl wait --for condition=ready --timeout=300s pipeline --all -n ${NAMESPACE}

pipeline.mlops.seldon.io/tfsimples condition met

kubectl get pipeline tfsimples -o jsonpath='{.status.conditions[0]}' -n ${NAMESPACE}

{"lastTransitionTime":"2022-11-14T10:25:49Z","status":"True","type":"ModelsReady"}

kubectl delete -f ./models/tfsimple1.yaml -n ${NAMESPACE}
kubectl delete -f ./models/tfsimple2.yaml -n ${NAMESPACE}
kubectl delete -f ./pipelines/tfsimples.yaml -n ${NAMESPACE}

model.mlops.seldon.io "tfsimple1" deleted
model.mlops.seldon.io "tfsimple2" deleted
pipeline.mlops.seldon.io "tfsimples" deleted