Pipeline examples

This notebook illustrates a series of Pipelines showing of different ways of combining flows of data and conditional logic. We assume you have Seldon Core 2 running locally.

Models Used

gs://seldon-models/triton/simple an example Triton tensorflow model that takes 2 inputs INPUT0 and INPUT1 and adds them to produce OUTPUT0 and also subtracts INPUT1 from INPUT0 to produce OUTPUT1. See here for the original source code and license.
Other models can be found at https://github.com/SeldonIO/triton-python-examples

Model Chaining

Chain the output of one model into the next. Also shows chaning the tensor names via tensorMap to conform to the expected input tensor names of the second model.

cat ./models/tfsimple1.yaml
cat ./models/tfsimple2.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple1
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki
---  
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple2
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki

seldon model load -f ./models/tfsimple1.yaml
seldon model load -f ./models/tfsimple2.yaml

{}
{}

seldon model status tfsimple1 -w ModelAvailable | jq -M .
seldon model status tfsimple2 -w ModelAvailable | jq -M .

{}
{}

The pipeline below chains the output of tfsimple1 into tfsimple2. As these models have compatible shape and data type this can be done. However, the output tensor names from tfsimple1 need to be renamed to match the input tensor names for tfsimple2. We do this with the tensorMap feature.

The output of the Pipeline is the output from tfsimple2.

cat ./pipelines/tfsimples.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: tfsimples
spec:
  steps:
    - name: tfsimple1
    - name: tfsimple2
      inputs:
      - tfsimple1
      tensorMap:
        tfsimple1.outputs.OUTPUT0: INPUT0
        tfsimple1.outputs.OUTPUT1: INPUT1
  output:
    steps:
    - tfsimple2

seldon pipeline load -f ./pipelines/tfsimples.yaml

seldon pipeline status tfsimples -w PipelineReady | jq -M .

{
  "pipelineName": "tfsimples",
  "versions": [
    {
      "pipeline": {
        "name": "tfsimples",
        "uid": "ciep26qi8ufs73flaiqg",
        "version": 2,
        "steps": [
          {
            "name": "tfsimple1"
          },
          {
            "name": "tfsimple2",
            "inputs": [
              "tfsimple1.outputs"
            ],
            "tensorMap": {
              "tfsimple1.outputs.OUTPUT0": "INPUT0",
              "tfsimple1.outputs.OUTPUT1": "INPUT1"
            }
          }
        ],
        "output": {
          "steps": [
            "tfsimple2.outputs"
          ]
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 2,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:11:40.101677847Z",
        "modelsReady": true
      }
    }
  ]
}

seldon pipeline infer tfsimples \
    '{"inputs":[{"name":"INPUT0","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]}]}' | jq -M .

{
  "model_name": "",
  "outputs": [
    {
      "data": [
        2,
        4,
        6,
        8,
        10,
        12,
        14,
        16,
        18,
        20,
        22,
        24,
        26,
        28,
        30,
        32
      ],
      "name": "OUTPUT0",
      "shape": [
        1,
        16
      ],
      "datatype": "INT32"
    },
    {
      "data": [
        2,
        4,
        6,
        8,
        10,
        12,
        14,
        16,
        18,
        20,
        22,
        24,
        26,
        28,
        30,
        32
      ],
      "name": "OUTPUT1",
      "shape": [
        1,
        16
      ],
      "datatype": "INT32"
    }
  ]
}

seldon pipeline infer tfsimples --inference-mode grpc \
    '{"model_name":"simple","inputs":[{"name":"INPUT0","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT0",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          2,
          4,
          6,
          8,
          10,
          12,
          14,
          16,
          18,
          20,
          22,
          24,
          26,
          28,
          30,
          32
        ]
      }
    },
    {
      "name": "OUTPUT1",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          2,
          4,
          6,
          8,
          10,
          12,
          14,
          16,
          18,
          20,
          22,
          24,
          26,
          28,
          30,
          32
        ]
      }
    }
  ]
}

We use the Seldon CLI pipeline inspect feature to look at the data for all steps of the pipeline for the last data item passed through the pipeline (the default). This can be useful for debugging.

seldon pipeline inspect tfsimples

seldon.default.model.tfsimple1.inputs	ciep298fh5ss73dpdir0	{"inputs":[{"name":"INPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}, {"name":"INPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}]}
seldon.default.model.tfsimple1.outputs	ciep298fh5ss73dpdir0	{"modelName":"tfsimple1_1", "modelVersion":"1", "outputs":[{"name":"OUTPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {"name":"OUTPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}}]}
seldon.default.model.tfsimple2.inputs	ciep298fh5ss73dpdir0	{"inputs":[{"name":"INPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {"name":"INPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}}], "rawInputContents":["AgAAAAQAAAAGAAAACAAAAAoAAAAMAAAADgAAABAAAAASAAAAFAAAABYAAAAYAAAAGgAAABwAAAAeAAAAIAAAAA==", "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="]}
seldon.default.model.tfsimple2.outputs	ciep298fh5ss73dpdir0	{"modelName":"tfsimple2_1", "modelVersion":"1", "outputs":[{"name":"OUTPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {"name":"OUTPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}]}
seldon.default.pipeline.tfsimples.inputs	ciep298fh5ss73dpdir0	{"modelName":"tfsimples", "inputs":[{"name":"INPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}, {"name":"INPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]}}]}
seldon.default.pipeline.tfsimples.outputs	ciep298fh5ss73dpdir0	{"outputs":[{"name":"OUTPUT0", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}, {"name":"OUTPUT1", "datatype":"INT32", "shape":["1", "16"], "contents":{"intContents":[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]}}]}

Next, we look get the output as json and use the jq tool to get just one value.

seldon pipeline inspect tfsimples --format json | jq -M .topics[0].msgs[0].value

{
  "inputs": [
    {
      "name": "INPUT0",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16
        ]
      }
    },
    {
      "name": "INPUT1",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16
        ]
      }
    }
  ]
}

seldon pipeline unload tfsimples

seldon model unload tfsimple1
seldon model unload tfsimple2

Model Chaining from inputs

Chain the output of one model into the next. Shows using the input and outputs and combining.

cat ./models/tfsimple1.yaml
cat ./models/tfsimple2.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple1
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple2
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki

seldon model load -f ./models/tfsimple1.yaml
seldon model load -f ./models/tfsimple2.yaml

{}
{}

seldon model status tfsimple1 -w ModelAvailable | jq -M .
seldon model status tfsimple2 -w ModelAvailable | jq -M .

{}
{}

cat ./pipelines/tfsimples-input.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: tfsimples-input
spec:
  steps:
    - name: tfsimple1
    - name: tfsimple2
      inputs:
      - tfsimple1.inputs.INPUT0
      - tfsimple1.outputs.OUTPUT1
      tensorMap:
        tfsimple1.outputs.OUTPUT1: INPUT1
  output:
    steps:
    - tfsimple2

seldon pipeline load -f ./pipelines/tfsimples-input.yaml

seldon pipeline status tfsimples-input -w PipelineReady | jq -M .

{
  "pipelineName": "tfsimples-input",
  "versions": [
    {
      "pipeline": {
        "name": "tfsimples-input",
        "uid": "ciep2fii8ufs73flair0",
        "version": 1,
        "steps": [
          {
            "name": "tfsimple1"
          },
          {
            "name": "tfsimple2",
            "inputs": [
              "tfsimple1.inputs.INPUT0",
              "tfsimple1.outputs.OUTPUT1"
            ],
            "tensorMap": {
              "tfsimple1.outputs.OUTPUT1": "INPUT1"
            }
          }
        ],
        "output": {
          "steps": [
            "tfsimple2.outputs"
          ]
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:12:14.711416101Z",
        "modelsReady": true
      }
    }
  ]
}

seldon pipeline infer tfsimples-input \
    '{"inputs":[{"name":"INPUT0","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]}]}' | jq -M .

{
  "model_name": "",
  "outputs": [
    {
      "data": [
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        10,
        11,
        12,
        13,
        14,
        15,
        16
      ],
      "name": "OUTPUT0",
      "shape": [
        1,
        16
      ],
      "datatype": "INT32"
    },
    {
      "data": [
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        10,
        11,
        12,
        13,
        14,
        15,
        16
      ],
      "name": "OUTPUT1",
      "shape": [
        1,
        16
      ],
      "datatype": "INT32"
    }
  ]
}

seldon pipeline infer tfsimples-input --inference-mode grpc \
    '{"model_name":"simple","inputs":[{"name":"INPUT0","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT0",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16
        ]
      }
    },
    {
      "name": "OUTPUT1",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16
        ]
      }
    }
  ]
}

seldon pipeline unload tfsimples-input

seldon model unload tfsimple1
seldon model unload tfsimple2

Model Join

Join two flows of data from two models as input to a third model. This shows how individual flows of data can be combined.

cat ./models/tfsimple1.yaml
echo "---"
cat ./models/tfsimple2.yaml
echo "---"
cat ./models/tfsimple3.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple1
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple2
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: tfsimple3
spec:
  storageUri: "gs://seldon-models/triton/simple"
  requirements:
  - tensorflow
  memory: 100Ki

seldon model load -f ./models/tfsimple1.yaml
seldon model load -f ./models/tfsimple2.yaml
seldon model load -f ./models/tfsimple3.yaml

{}
{}
{}

seldon model status tfsimple1 -w ModelAvailable | jq -M .
seldon model status tfsimple2 -w ModelAvailable | jq -M .
seldon model status tfsimple3 -w ModelAvailable | jq -M .

{}
{}
{}

In the pipeline below for the input to tfsimple3 we join 1 output tensor each from the two previous models tfsimple1 and tfsimple2. We need to use the tensorMap feature to rename each output tensor to one of the expected input tensors for the tfsimple3 model.

cat ./pipelines/tfsimples-join.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: join
spec:
  steps:
    - name: tfsimple1
    - name: tfsimple2
    - name: tfsimple3
      inputs:
      - tfsimple1.outputs.OUTPUT0
      - tfsimple2.outputs.OUTPUT1
      tensorMap:
        tfsimple1.outputs.OUTPUT0: INPUT0
        tfsimple2.outputs.OUTPUT1: INPUT1
  output:
    steps:
    - tfsimple3

seldon pipeline load -f ./pipelines/tfsimples-join.yaml

seldon pipeline status join -w PipelineReady | jq -M .

{
  "pipelineName": "join",
  "versions": [
    {
      "pipeline": {
        "name": "join",
        "uid": "ciep2k2i8ufs73flairg",
        "version": 1,
        "steps": [
          {
            "name": "tfsimple1"
          },
          {
            "name": "tfsimple2"
          },
          {
            "name": "tfsimple3",
            "inputs": [
              "tfsimple1.outputs.OUTPUT0",
              "tfsimple2.outputs.OUTPUT1"
            ],
            "tensorMap": {
              "tfsimple1.outputs.OUTPUT0": "INPUT0",
              "tfsimple2.outputs.OUTPUT1": "INPUT1"
            }
          }
        ],
        "output": {
          "steps": [
            "tfsimple3.outputs"
          ]
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:12:32.938603415Z",
        "modelsReady": true
      }
    }
  ]
}

The outputs are the sequence "2,4,6..." which conforms to the logic of this model (addition and subtraction) when fed the output of the first two models.

seldon pipeline infer join --inference-mode grpc \
    '{"model_name":"simple","inputs":[{"name":"INPUT0","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","contents":{"int_contents":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]},"datatype":"INT32","shape":[1,16]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT0",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          2,
          4,
          6,
          8,
          10,
          12,
          14,
          16,
          18,
          20,
          22,
          24,
          26,
          28,
          30,
          32
        ]
      }
    },
    {
      "name": "OUTPUT1",
      "datatype": "INT32",
      "shape": [
        "1",
        "16"
      ],
      "contents": {
        "intContents": [
          2,
          4,
          6,
          8,
          10,
          12,
          14,
          16,
          18,
          20,
          22,
          24,
          26,
          28,
          30,
          32
        ]
      }
    }
  ]
}

seldon pipeline unload join

seldon model unload tfsimple1
seldon model unload tfsimple2
seldon model unload tfsimple3

Conditional

Shows conditional data flows - one of two models is run based on output tensors from first.

cat ./models/conditional.yaml
echo "---"
cat ./models/add10.yaml
echo "---"
cat ./models/mul10.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: conditional
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/conditional"
  requirements:
  - triton
  - python
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: add10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
  requirements:
  - triton
  - python
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: mul10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
  requirements:
  - triton
  - python

seldon model load -f ./models/conditional.yaml
seldon model load -f ./models/add10.yaml
seldon model load -f ./models/mul10.yaml

{}
{}
{}

seldon model status conditional -w ModelAvailable | jq -M .
seldon model status add10 -w ModelAvailable | jq -M .
seldon model status mul10 -w ModelAvailable | jq -M .

{}
{}
{}

Here we assume the conditional model can output two tensors OUTPUT0 and OUTPUT1 but only outputs the former if the CHOICE input tensor is set to 0 otherwise it outputs tensor OUTPUT1. By this means only one of the two downstream models will receive data and run. The output steps does an any join from both models and whichever data appears first will be sent as output to pipeline. As in this case only 1 of the two models add10 and mul10 runs we will receive their output.

cat ./pipelines/conditional.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: tfsimple-conditional
spec:
  steps:
  - name: conditional
  - name: mul10
    inputs:
    - conditional.outputs.OUTPUT0
    tensorMap:
      conditional.outputs.OUTPUT0: INPUT
  - name: add10
    inputs:
    - conditional.outputs.OUTPUT1
    tensorMap:
      conditional.outputs.OUTPUT1: INPUT
  output:
    steps:
    - mul10
    - add10
    stepsJoin: any

seldon pipeline load -f ./pipelines/conditional.yaml

seldon pipeline status tfsimple-conditional -w PipelineReady | jq -M .

{
  "pipelineName": "tfsimple-conditional",
  "versions": [
    {
      "pipeline": {
        "name": "tfsimple-conditional",
        "uid": "ciepga2i8ufs73flais0",
        "version": 1,
        "steps": [
          {
            "name": "add10",
            "inputs": [
              "conditional.outputs.OUTPUT1"
            ],
            "tensorMap": {
              "conditional.outputs.OUTPUT1": "INPUT"
            }
          },
          {
            "name": "conditional"
          },
          {
            "name": "mul10",
            "inputs": [
              "conditional.outputs.OUTPUT0"
            ],
            "tensorMap": {
              "conditional.outputs.OUTPUT0": "INPUT"
            }
          }
        ],
        "output": {
          "steps": [
            "mul10.outputs",
            "add10.outputs"
          ],
          "stepsJoin": "ANY"
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:41:45.133142725Z",
        "modelsReady": true
      }
    }
  ]
}

The mul10 model will run as the CHOICE tensor is set to 0.

seldon pipeline infer tfsimple-conditional --inference-mode grpc \
 '{"model_name":"conditional","inputs":[{"name":"CHOICE","contents":{"int_contents":[0]},"datatype":"INT32","shape":[1]},{"name":"INPUT0","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]},{"name":"INPUT1","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          10,
          20,
          30,
          40
        ]
      }
    }
  ]
}

The add10 model will run as the CHOICE tensor is not set to zero.

seldon pipeline infer tfsimple-conditional --inference-mode grpc \
 '{"model_name":"conditional","inputs":[{"name":"CHOICE","contents":{"int_contents":[1]},"datatype":"INT32","shape":[1]},{"name":"INPUT0","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]},{"name":"INPUT1","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          11,
          12,
          13,
          14
        ]
      }
    }
  ]
}

seldon pipeline unload tfsimple-conditional

seldon model unload conditional
seldon model unload add10
seldon model unload mul10

Pipeline Input Tensors

Access to indivudal tensors in pipeline inputs

cat ./models/mul10.yaml
echo "---"
cat ./models/add10.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: mul10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
  requirements:
  - triton
  - python
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: add10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
  requirements:
  - triton
  - python

seldon model load -f ./models/mul10.yaml
seldon model load -f ./models/add10.yaml

{}
{}

seldon model status mul10 -w ModelAvailable | jq -M .
seldon model status add10 -w ModelAvailable | jq -M .

{}
{}

This pipeline shows how we can access pipeline inputs INPUT0 and INPUT1 from different steps.

cat ./pipelines/pipeline-inputs.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: pipeline-inputs
spec:
  steps:
  - name: mul10
    inputs:
    - pipeline-inputs.inputs.INPUT0
    tensorMap:
      pipeline-inputs.inputs.INPUT0: INPUT
  - name: add10
    inputs:
    - pipeline-inputs.inputs.INPUT1
    tensorMap:
      pipeline-inputs.inputs.INPUT1: INPUT
  output:
    steps:
    - mul10
    - add10

seldon pipeline load -f ./pipelines/pipeline-inputs.yaml

seldon pipeline status pipeline-inputs -w PipelineReady | jq -M .

{
  "pipelineName": "pipeline-inputs",
  "versions": [
    {
      "pipeline": {
        "name": "pipeline-inputs",
        "uid": "ciepgeqi8ufs73flaisg",
        "version": 1,
        "steps": [
          {
            "name": "add10",
            "inputs": [
              "pipeline-inputs.inputs.INPUT1"
            ],
            "tensorMap": {
              "pipeline-inputs.inputs.INPUT1": "INPUT"
            }
          },
          {
            "name": "mul10",
            "inputs": [
              "pipeline-inputs.inputs.INPUT0"
            ],
            "tensorMap": {
              "pipeline-inputs.inputs.INPUT0": "INPUT"
            }
          }
        ],
        "output": {
          "steps": [
            "mul10.outputs",
            "add10.outputs"
          ]
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:42:04.202598715Z",
        "modelsReady": true
      }
    }
  ]
}

seldon pipeline infer pipeline-inputs --inference-mode grpc \
    '{"model_name":"pipeline","inputs":[{"name":"INPUT0","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]},{"name":"INPUT1","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          10,
          20,
          30,
          40
        ]
      }
    },
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          11,
          12,
          13,
          14
        ]
      }
    }
  ]
}

seldon pipeline unload pipeline-inputs

seldon model unload mul10
seldon model unload add10

Trigger Joins

Shows how joins can be used for triggers as well.

cat ./models/mul10.yaml
cat ./models/add10.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: mul10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/mul10"
  requirements:
  - triton
  - python
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: add10
spec:
  storageUri: "gs://seldon-models/scv2/samples/triton_23-03/add10"
  requirements:
  - triton
  - python

seldon model load -f ./models/mul10.yaml
seldon model load -f ./models/add10.yaml

{}
{}

seldon model status mul10 -w ModelAvailable | jq -M .
seldon model status add10 -w ModelAvailable | jq -M .

{}
{}

Here we required tensors names ok1 or ok2 to exist on pipeline inputs to run the mul10 model but require tensor ok3 to exist on pipeline inputs to run the add10 model. The logic on mul10 is handled by a trigger join of any meaning either of these input data can exist to satisfy the trigger join.

cat ./pipelines/trigger-joins.yaml

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: trigger-joins
spec:
  steps:
  - name: mul10
    inputs:
    - trigger-joins.inputs.INPUT
    triggers:
    - trigger-joins.inputs.ok1
    - trigger-joins.inputs.ok2
    triggersJoinType: any
  - name: add10
    inputs:
    - trigger-joins.inputs.INPUT
    triggers:
    - trigger-joins.inputs.ok3
  output:
    steps:
    - mul10
    - add10
    stepsJoin: any

seldon pipeline load -f ./pipelines/trigger-joins.yaml

seldon pipeline status trigger-joins -w PipelineReady | jq -M .

{
  "pipelineName": "trigger-joins",
  "versions": [
    {
      "pipeline": {
        "name": "trigger-joins",
        "uid": "ciepgkqi8ufs73flait0",
        "version": 1,
        "steps": [
          {
            "name": "add10",
            "inputs": [
              "trigger-joins.inputs.INPUT"
            ],
            "triggers": [
              "trigger-joins.inputs.ok3"
            ]
          },
          {
            "name": "mul10",
            "inputs": [
              "trigger-joins.inputs.INPUT"
            ],
            "triggers": [
              "trigger-joins.inputs.ok1",
              "trigger-joins.inputs.ok2"
            ],
            "triggersJoin": "ANY"
          }
        ],
        "output": {
          "steps": [
            "mul10.outputs",
            "add10.outputs"
          ],
          "stepsJoin": "ANY"
        },
        "kubernetesMeta": {}
      },
      "state": {
        "pipelineVersion": 1,
        "status": "PipelineReady",
        "reason": "created pipeline",
        "lastChangeTimestamp": "2023-06-29T14:42:27.595300698Z",
        "modelsReady": true
      }
    }
  ]
}

seldon pipeline infer trigger-joins --inference-mode grpc \
    '{"model_name":"pipeline","inputs":[{"name":"ok1","contents":{"fp32_contents":[1]},"datatype":"FP32","shape":[1]},{"name":"INPUT","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          10,
          20,
          30,
          40
        ]
      }
    }
  ]
}

seldon pipeline infer trigger-joins --inference-mode grpc \
    '{"model_name":"pipeline","inputs":[{"name":"ok3","contents":{"fp32_contents":[1]},"datatype":"FP32","shape":[1]},{"name":"INPUT","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[4]}]}' | jq -M .

{
  "outputs": [
    {
      "name": "OUTPUT",
      "datatype": "FP32",
      "shape": [
        "4"
      ],
      "contents": {
        "fp32Contents": [
          11,
          12,
          13,
          14
        ]
      }
    }
  ]
}

seldon pipeline unload trigger-joins

seldon model unload mul10
seldon model unload add10