Edit

Argo Workflows Benchmarking

In this notebook we will dive into how you can run bench marking with batch processing with Argo Workflows, Seldon Core and Vegeta.

Dependencies:

  • Seldon core installed as per the docs with Istio as an ingress

  • Argo Workfklows installed in cluster (and argo CLI for commands)

Setup

Install Seldon Core

Use the notebook to set-up Seldon Core with Ambassador or Istio Ingress.

Note: If running with KIND you need to make sure do follow these steps as workaround to the /.../docker.sock known issue.

Install Argo Workflows

You can follow the instructions from the official Argo Workflows Documentation.

Download the right CLi for your environment following the documentation (https://github.com/argoproj/argo-workflows/releases/tag/v3.0.8)

You also need to make sure that argo has permissions to create seldon deployments - for this you can just create a default-admin rolebinding as follows:

Set up the RBAC so the argo workflow is able to create seldon deployments.

Set up the configmap in order for it to work in KIND and other environments where Docker may not be thr main runtime (see https://github.com/argoproj/argo-workflows/issues/5243#issuecomment-792993742)

Create Benchmark Argo Workflow

In order to create a benchmark, we created a simple argo workflow template so you can leverage the power of the helm charts.

Before we dive into the contents of the full helm chart, let's first give it a try with some of the settings.

We will run a batch job that will set up a Seldon Deployment with 1 replicas and 4 cpus (with 100 max workers) to send requests.

Process the results

We can now print the results in a consumable format.

Deeper Analysis

Now that we have all the parameters, we can do a deeper analysis

mean
50th
90th
95th
99th
throughputAchieved
success
errors
name
replicas
...
apiType
requestsCpu
requestsMemory
limitsCpu
limitsMemory
benchmarkCpu
concurrency
duration
rate
disableOrchestrator

0

4.573302

4.018635

6.225710

7.480878

13.893386

218.518811

6559

0

sklearn-0

1

...

rest

2000Mi

100Mi

2000Mi

1000Mi

2

1

30s

0

true

1

4.565145

3.939032

6.785393

7.928704

13.315820

218.892806

6568

0

sklearn-1

1

...

rest

2000Mi

100Mi

2000Mi

1000Mi

2

1

30s

0

false

2

3.747319

3.212300

5.651600

6.858700

9.191800

258.595746

7757

1

sklearn-2

1

...

grpc

2000Mi

100Mi

2000Mi

1000Mi

2

1

30s

0

true

3

4.271879

3.855800

6.495800

7.353500

8.980500

226.930063

6807

1

sklearn-3

1

...

grpc

2000Mi

100Mi

2000Mi

1000Mi

2

1

30s

0

false

4 rows × 25 columns

PreviousTensorflow BenchmarkNextPython Serialization Cost Benchmark

Last updated

Was this helpful?