Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Learn more about using Seldon CLI commands
Seldon provides a CLI for easy management and testing of model, experiment, and pipeline resources. The Seldon CLI allows you to view information about underlying Seldon resources and make changes to them through the scheduler in non-Kubernetes environments. However, it cannot modify underlying manifests within a Kubernetes cluster. Therefore, using the Seldon CLI for control plane operations in a Kubernetes environment is not recommended and is disabled by default.
While Seldon CLI control plane operations (e.g. load
and unload
) in a Kubernetes environment are not recommended, there are other use cases (e.g. inspecting kafka topics in a pipeline) that are enabled easily with Seldon CLI. It offers out-of-the box deserialisation of the kafka messages according to Open Inference Protocol (OIP), allowing to test these pipelines in these environments.
The following table provides more information about when and where to use these command line tools.
Primary purpose
Simplifies management for non-Kubernetes users, abstracting control plane operations like load, unload, status.
Kubernetes-native, manages resources via Kubernetes Custom Resources (CRs) like Deployments, Pods, etc.
Control Plane Operations
Executes operations such as load
and unload
models through scheduler gRPC endpoints, without interaction with Kubernetes. This is disabled by default and the user has to explicitly specify --force
flag as argument (or set envar SELDON_FORCE_CONTROL_PLANE=true
).
Interacts with Kubernetes, creating and managing CRs such as SeldonDeployments and other Kubernetes resources.
Data Plane Operations
Abstracts open inference protocol to issue infer
or inspect
requests for testing purposes. This is useful for example when inspecting intermediate kafka messages in a multi-step pipeline.
Used indirectly for data plane operations by exposing Kubernetes services and interacting with them.
Visibility of Resources
Resources created using Seldon CLI are internal to the scheduler and not visible as Kubernetes resources.
All resources that are created using kubectl
are visible and manageable within the Kubernetes environment.
Seldon CLI can be deployed as a Pod in the same namespace along side a Core 2 installation, which would allow users to have access to the different CLI commands with minimal setup (i.e. envars set appropriately).
seldonio/seldon-cli
Docker image has prepackaged Seldon CLI suitable for deployment in a Kubernetes cluster. Consider the following Kubernetes Pod manifest:
which can be deployed with
Notes:
The above manifest sets up envars with the correct ip/port for the scheduler control plane (SELDON_SCHEDULE_HOST
).
It mounts seldon-kafka
ConfigMap in /mnt/kafka/kafka.json
so that Seldon CLI reads the relevant details from it (specifically the kafka bootstrap servers ip/port).
It also sets Kafka TLS envars, which should be identical to seldon-modelgateway
deployment.
Service account seldon-scheduler
is required to access Kafka TLS secrets if any.
In this case users can run any Seldon CLI commands via kubectl exec
utility, For example:
In particular this mode is also useful for inspecting data in kafka topics that form a pipeline. This is enabled because all the relevant arguments are setup so that Seldon CLI can consume data from these topics using the same settings used in the corresponding Core 2 deployment in the same namespace.
check operator/config/cli
for a helper script and example manifest.
The CLI talks to 3 backend services on default endpoints:
The Seldon Core 2 Scheduler: default 0.0.0.0:9004
The Seldon Core inference endpoint: default 0.0.0.0:9000
The Seldon Kafka broker: default: 0.0.0.0:9092
These defaults will be correct when Seldon Core 2 is installed locally as per the docs. For Kubernetes, you will need to change these by defining environment variables for the following.
For a default install into the seldon-mesh
namespace if you have exposed the inference svc
as a loadbalancer you will find it at:
Use above IP at port 80:
For a default install into the seldon-mesh
namespace if you have exposed the scheduler svc as a loadbalancer you will find it at:
Use above IP at port 9004:
The Kafka broker will depend on how you have installed Kafka into your Kubernetes cluster. Find the broker IP and use:
You can create a config file to manage connections to running Seldon Core 2 installs. The settings will override any environment variable settings.
The definition is shown below:
An example below shows an example where we connect via TLS to the Seldon scheduler using our scheduler client certificate:
To manage config files and activate them you can use the CLI command seldon config
which has subcommands to list, add, remove, activate and decative configs.
For example:
pipeline inspect
)Additional Kafka configuration specifying Kafka broker and relevant consumer config can be specified as path to a json formatted file. This is currently only supported for inspecting data in topics that form a specific pipeline by passing --kafka-config-path
. For example:
For running with Kubernetes TLS connections on the control and/or data plane, certificates will need to be downloaded locally. We provide an example script which will download certificates from a Kubernetes secret and store them in a folder. It can be found in hack/download-k8s-certs.sh
and takes 2 or 3 arguments:
e.g.:
get status for experiment
get status for experiment
seldon experiment - manage experiments
get list of experiments
get list of experiments and whether they are active
seldon experiment - manage experiments
manage configs
Manage and activate configuration files for the CLI
seldon -
seldon config activate - activate config
seldon config add - add config
seldon config deactivate - deactivate config
seldon config list - list configs
seldon config remove - remove config
manage models
load and unload and get status for models
seldon -
seldon model infer - run inference on a model
seldon model list - get list of models
seldon model load - load a model
seldon model metadata - get model metadata
seldon model status - get status for model
seldon model unload - unload a model
seldon config - manage configs
seldon experiment - manage experiments
seldon model - manage models
seldon pipeline - manage pipelines
seldon server - manage servers
run inference on a model
call a model with a given input and get a prediction
seldon model - manage models
get list of models
get the list of all models with their status
seldon model - manage models
- manage pipelines
- manage pipelines