Batch Processing with Kubeflow Pipelines
In this notebook we will dive into how you can run batch processing with Kubeflow Pipelines and Seldon Core.
Dependencies:
Seldon core installed as per the docs with Istio Ingress
Kubeflow Pipelines installed (installation instructions in this notebook)
Kubeflow Pipelines Setup
Setup the pipeline in your current cluster:
We also install the Python Library so we can create our pipeline:
Add Batch Data
In order to run our batch job we will need to create some batch data that can be used to process.
This batch dataset will be pushed to a minio instance so it can be downloaded from Minio (which we need to install first)
Install Minio
Forward the Minio port so you can access it
You can do this by runnning the following command in your terminal:
Create some input for our model
We will create a file that will contain the inputs that will be sent to our model
Check the contents of the file
Upload the file to our minio
Create Kubeflow Pipeline
We are now able to create a kubeflow pipeline that will allow us to enter the batch parameters through the UI.
We will also be able to add extra steps that will download the data from a Minio client.
We use the pipeline syntax to create the kubeflow pipeline, as outlined below:
Trigger the creation
We will run the python file which triggers the creation of the pipeline that we can the upload on the UI:
Check the pipeline has been created:
Open the Kubeflow Pipelines UI
We can now open the UI by port forwarding the UI with the following command:
And we can open it locally in our browser via http://localhost:8000
Now we can follow the standard steps to create and deploy the kubeflow pipline
Last updated
Was this helpful?