1 of 30

CLI

Learn more about using Seldon CLI commands

Seldon provides a CLI for easy management and testing of model, experiment, and pipeline resources. The Seldon CLI allows you to view information about underlying Seldon resources and make changes to them through the scheduler in non-Kubernetes environments. However, it cannot modify underlying manifests within a Kubernetes cluster. Therefore, using the Seldon CLI for control plane operations in a Kubernetes environment is not recommended.

The following table provides more information about when and where to use these command line tools.

Usage

Seldon CLI

kubectl

Primary purpose

Simplifies management for non-Kubernetes users, abstracting control plane operations like load, unload, status.

Kubernetes-native, manages resources via Kubernetes Custom Resources (CRs) like Deployments, Pods, etc.

Control Plane Operations

Executes operations such as load and unloadmodels through scheduler gRPC endpoints, without interaction with Kubernetes.

Interacts with Kubernetes, creating and managing CRs such as SeldonDeployments and other Kubernetes resources.

Data Plane Operations

Abstracts open inference protocol to issue infer or inspect requests for testing purposes.

Used indirectly for data plane operations by exposing Kubernetes services and interacting with them.

Visibility of Resources

Resources created using Seldon CLI are internal to the scheduler and not visible as Kubernetes resources.

All resources that are created using kubectl are visible and manageable within the Kubernetes environment.

Environment Variables and Services

The CLI talks to 3 backend services on default endpoints:

The Seldon Core 2 Scheduler: default 0.0.0.0:9004
The Seldon Core inference endpoint: default 0.0.0.0:9000
The Seldon Kafka broker: default: 0.0.0.0:9092

These defaults will be correct when Seldon Core 2 is installed locally as per the docs. For Kubernetes, you will need to change these by defining environment variables.

const (
	defaultInferHost     = "0.0.0.0:9000"
	defaultKafkaHost     = "0.0.0.0:9092"
	defaultSchedulerHost = "0.0.0.0:9004"
)

Kubernetes Usage

Inference Service

For a default install into the seldon-mesh namespace if you have exposed the inference svc as a loadbalancer you will find it at:

kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 80:

export SELDON_INFER_HOST=<ip>:80

Scheduler Service

For a default install into the seldon-mesh namespace if you have exposed the scheduler svc as a loadbalancer you will find it at:

kubectl get svc seldon-scheduler -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 9004:

export SELDON_SCHEDULE_HOST=<ip>:9004

Kafka Broker

The Kafka broker will depend on how you have installed Kafka into your Kubernetes cluster. Find the broker IP and use:

export SELDON_KAFKA_BROKER=<ip>:<port>

Config file

You can create a config file to manage connections to running Seldon Core 2 installs. The settings will override any environment variable settings.

The definition is shown below:

type SeldonCLIConfig struct {
	Dataplane    *Dataplane    `json:"dataplane,omitempty"`
	Controlplane *ControlPlane `json:"controlplane,omitempty"`
	Kafka        *KafkaConfig  `json:"kafka,omitempty"`
}

type Dataplane struct {
	InferHost     string `json:"inferHost,omitempty"`
	Tls           bool   `json:"tls,omitempty"`
	SkipSSLVerify bool   `json:"skipSSLVerify,omitempty"`
	KeyPath       string `json:"keyPath,omitempty"`
	CrtPath       string `json:"crtPath,omitempty"`
	CaPath        string `json:"caPath,omitempty"`
}

type ControlPlane struct {
	SchedulerHost string `json:"schedulerHost,omitempty"`
	Tls           bool   `json:"tls,omitempty"`
	KeyPath       string `json:"keyPath,omitempty"`
	CrtPath       string `json:"crtPath,omitempty"`
	CaPath        string `json:"caPath,omitempty"`
}

const (
	KafkaConfigProtocolSSL          = "ssl"
	KafkaConfigProtocolSASLSSL      = "sasl_ssl"
	KafkaConfigProtocolSASLPlaintxt = "sasl_plaintxt"
)

type KafkaConfig struct {
	Bootstrap    string `json:"bootstrap,omitempty"`
	Namespace    string `json:"namespace,omitempty"`
	Protocol     string `json:"protocol,omitempty"`
	KeyPath      string `json:"keyPath,omitempty"`
	CrtPath      string `json:"crtPath,omitempty"`
	CaPath       string `json:"caPath,omitempty"`
	SaslUsername string `json:"saslUsername,omitempty"`
	SaslPassword string `json:"saslPassword,omitempty"`
	TopicPrefix  string `json:"topicPrefix,omitempty"`
}

An example below shows an example where we connect via TLS to the Seldon scheduler using our scheduler client certificate:

{
  "controlplane": {
  	"schedulerHost": "seldon-scheduler.svc:9044",
  	"tls": true,
  	"keyPath": "/home/certs/seldon-scheduler-client/tls.key",
  	"crtPath": "/home/certs/seldon-scheduler-client/tls.crt",
  	"caPath": "/home/certs/seldon-scheduler-client/ca.crt"
  }
}

To manage config files and activate them you can use the CLI command seldon config which has subcommands to list, add, remove, activate and decative configs.

For example:

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json		*

$ seldon config deactivate kind-sasl

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config add gcp-scv2 ~/seldon/cli/gcp.json

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config activate gcp-scv2

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json	    		*
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config list kind-sasl
{
  "controlplane": {
    "schedulerHost": "172.19.255.2:9004"
  },
  "kafka": {
    "bootstrap": "172.19.255.3:9093",
    "caPath": "/home/work/gcp/scv2/certs/seldon-cluster-ca-cert/ca.crt"
  }
}

TLS Certificates for Local Use

For running with Kubernetes TLS connections on the control and/or data plane, certificates will need to be downloaded locally. We provide an example script which will download certificates from a Kubernetes secret and store them in a folder. It can be found in hack/download-k8s-certs.sh and takes 2 or 3 arguments:

./download-k8s-certs.sh <namespace> <secret> [<folder>]

e.g.:

./download-k8s-certs.sh seldon-mesh seldon-scheduler-client

Seldon

Options

Config

manage configs

Synopsis

Manage and activate configuration files for the CLI

seldon config <subcomand> [flags]

Options

  -h, --help   help for config

Config Activate

activate config

Synopsis

activate config

seldon config activate [flags]

Options

  -h, --help   help for activate

Config Deactivate

deactivate config

Synopsis

deactivate config

seldon config deactivate [flags]

Options

  -h, --help   help for deactivate

Config Add

add config

Synopsis

add config

seldon config add [flags]

Options

  -h, --help   help for add

Config List

list configs

Synopsis

list configs

seldon config list [flags]

Options

  -h, --help   help for list

Config Remove

remove config

Synopsis

remove config

seldon config remove [flags]

Options

  -h, --help   help for remove

get status for experiment

seldon experiment status <experimentName> [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for status
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")
  -r, --show-request            show request
  -o, --show-response           show response (default true)
  -w, --wait                    wait for experiment to be active

Experiment List

get list of experiments

Synopsis

get list of experiments and whether they are active

seldon experiment list [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for list
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")

get the status for a model

seldon model status <modelName> [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for status
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")
  -r, --show-request            show request
  -o, --show-response           show response (default true)
  -w, --wait string             model wait condition

Model Load

load a model

Synopsis

load a model

seldon model load [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -f, --file-path string        model manifest file (YAML)
  -h, --help                    help for load
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")
  -r, --show-request            show request
  -o, --show-response           show response (default true)

Model List

get list of models

Synopsis

get the list of all models with their status

seldon model list [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for list
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")

call a pipeline with a given input and get a prediction

seldon pipeline infer <pipelineName> (data) [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -f, --file-path string        inference payload file
      --header stringArray      add a header, e.g. key=value; use the flag multiple times to add more than one header
  -h, --help                    help for infer
      --inference-host string   seldon inference host (default "0.0.0.0:9000")
      --inference-mode string   inference mode (rest or grpc) (default "rest")
  -i, --iterations int          how many times to run inference (default 1)
  -t, --seconds int             number of secs to run inference
      --show-headers            show request and response headers
  -r, --show-request            show request
  -o, --show-response           show response (default true)
  -s, --sticky-session          use sticky session from last inference (only works with experiments)

Pipeline Unload

unload a pipeline

Synopsis

unload a pipeline

seldon pipeline unload <pipelineName> [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for unload
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")
  -r, --show-request            show request
  -o, --show-response           show response (default true)

Server

manage servers

Synopsis

get status for servers

seldon server <subcomand> [flags]

Options

  -h, --help   help for server

Server List

get list of servers

Synopsis

get the available servers, their replicas and loaded models

seldon server list [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for list
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")

Server Status

get status for server

Synopsis

get the status for a server

seldon server status [flags]

Options

      --authority string        authority (HTTP/2) or virtual host (HTTP/1)
  -h, --help                    help for status
      --scheduler-host string   seldon scheduler host (default "0.0.0.0:9004")
  -r, --show-request            show request
  -o, --show-response           show response (default true)

CLI

Learn more about using Seldon CLI commands

The following table provides more information about when and where to use these command line tools.

Usage

Seldon CLI

kubectl

Primary purpose

Simplifies management for non-Kubernetes users, abstracting control plane operations like load, unload, status.

Kubernetes-native, manages resources via Kubernetes Custom Resources (CRs) like Deployments, Pods, etc.

Control Plane Operations

Executes operations such as load and unloadmodels through scheduler gRPC endpoints, without interaction with Kubernetes.

Interacts with Kubernetes, creating and managing CRs such as SeldonDeployments and other Kubernetes resources.

Data Plane Operations

Abstracts open inference protocol to issue infer or inspect requests for testing purposes.

Used indirectly for data plane operations by exposing Kubernetes services and interacting with them.

Visibility of Resources

Resources created using Seldon CLI are internal to the scheduler and not visible as Kubernetes resources.

All resources that are created using kubectl are visible and manageable within the Kubernetes environment.

Environment Variables and Services

The CLI talks to 3 backend services on default endpoints:

The Seldon Core 2 Scheduler: default 0.0.0.0:9004
The Seldon Core inference endpoint: default 0.0.0.0:9000
The Seldon Kafka broker: default: 0.0.0.0:9092

These defaults will be correct when Seldon Core 2 is installed locally as per the docs. For Kubernetes, you will need to change these by defining environment variables.

const (
	defaultInferHost     = "0.0.0.0:9000"
	defaultKafkaHost     = "0.0.0.0:9092"
	defaultSchedulerHost = "0.0.0.0:9004"
)

Kubernetes Usage

Inference Service

For a default install into the seldon-mesh namespace if you have exposed the inference svc as a loadbalancer you will find it at:

kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 80:

export SELDON_INFER_HOST=<ip>:80

Scheduler Service

For a default install into the seldon-mesh namespace if you have exposed the scheduler svc as a loadbalancer you will find it at:

kubectl get svc seldon-scheduler -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 9004:

export SELDON_SCHEDULE_HOST=<ip>:9004

Kafka Broker

The Kafka broker will depend on how you have installed Kafka into your Kubernetes cluster. Find the broker IP and use:

export SELDON_KAFKA_BROKER=<ip>:<port>

Config file

You can create a config file to manage connections to running Seldon Core 2 installs. The settings will override any environment variable settings.

The definition is shown below:

type SeldonCLIConfig struct {
	Dataplane    *Dataplane    `json:"dataplane,omitempty"`
	Controlplane *ControlPlane `json:"controlplane,omitempty"`
	Kafka        *KafkaConfig  `json:"kafka,omitempty"`
}

type Dataplane struct {
	InferHost     string `json:"inferHost,omitempty"`
	Tls           bool   `json:"tls,omitempty"`
	SkipSSLVerify bool   `json:"skipSSLVerify,omitempty"`
	KeyPath       string `json:"keyPath,omitempty"`
	CrtPath       string `json:"crtPath,omitempty"`
	CaPath        string `json:"caPath,omitempty"`
}

type ControlPlane struct {
	SchedulerHost string `json:"schedulerHost,omitempty"`
	Tls           bool   `json:"tls,omitempty"`
	KeyPath       string `json:"keyPath,omitempty"`
	CrtPath       string `json:"crtPath,omitempty"`
	CaPath        string `json:"caPath,omitempty"`
}

const (
	KafkaConfigProtocolSSL          = "ssl"
	KafkaConfigProtocolSASLSSL      = "sasl_ssl"
	KafkaConfigProtocolSASLPlaintxt = "sasl_plaintxt"
)

type KafkaConfig struct {
	Bootstrap    string `json:"bootstrap,omitempty"`
	Namespace    string `json:"namespace,omitempty"`
	Protocol     string `json:"protocol,omitempty"`
	KeyPath      string `json:"keyPath,omitempty"`
	CrtPath      string `json:"crtPath,omitempty"`
	CaPath       string `json:"caPath,omitempty"`
	SaslUsername string `json:"saslUsername,omitempty"`
	SaslPassword string `json:"saslPassword,omitempty"`
	TopicPrefix  string `json:"topicPrefix,omitempty"`
}

An example below shows an example where we connect via TLS to the Seldon scheduler using our scheduler client certificate:

{
  "controlplane": {
  	"schedulerHost": "seldon-scheduler.svc:9044",
  	"tls": true,
  	"keyPath": "/home/certs/seldon-scheduler-client/tls.key",
  	"crtPath": "/home/certs/seldon-scheduler-client/tls.crt",
  	"caPath": "/home/certs/seldon-scheduler-client/ca.crt"
  }
}

To manage config files and activate them you can use the CLI command seldon config which has subcommands to list, add, remove, activate and decative configs.

For example:

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json		*

$ seldon config deactivate kind-sasl

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config add gcp-scv2 ~/seldon/cli/gcp.json

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config activate gcp-scv2

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json	    		*
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config list kind-sasl
{
  "controlplane": {
    "schedulerHost": "172.19.255.2:9004"
  },
  "kafka": {
    "bootstrap": "172.19.255.3:9093",
    "caPath": "/home/work/gcp/scv2/certs/seldon-cluster-ca-cert/ca.crt"
  }
}

TLS Certificates for Local Use

./download-k8s-certs.sh <namespace> <secret> [<folder>]

e.g.:

./download-k8s-certs.sh seldon-mesh seldon-scheduler-client

CLI

Environment Variables and Services

Kubernetes Usage

Inference Service

Scheduler Service

Kafka Broker

Config file

TLS Certificates for Local Use

Seldon

Options

SEE ALSO

Config

Synopsis

Options

SEE ALSO

Config Activate

Synopsis

Options

SEE ALSO

Config Deactivate

Synopsis

Options

SEE ALSO

Config Add

Synopsis

Options

SEE ALSO

Config List

Synopsis

Options

SEE ALSO

Config Remove

Synopsis

Options

SEE ALSO

Experiment

Synopsis

Options

SEE ALSO

Experiment Start

Synopsis

Options

SEE ALSO

Experiment Status

Synopsis

Options

SEE ALSO

Experiment List

Synopsis

Options

SEE ALSO

Experiment Stop

Synopsis

Options

SEE ALSO

Model

Synopsis

Options

SEE ALSO

Model Status

Synopsis

Options

SEE ALSO

Model Load

Synopsis

Options

SEE ALSO

Model List

Synopsis

Options

SEE ALSO

Model Infer

Synopsis

Options

SEE ALSO

Model Metadata

Synopsis

Options

SEE ALSO

Model Unload