# Argo Workflows

## Installing Argo Workflows

Install Argo Workflows in line with the [official instructions](https://argo-workflows.readthedocs.io/en/latest/quick-start/). At the time of writing these are:

```bash
kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.6/install.yaml
```

## Per Namespace Setup

If you intend to use batch jobs in a namespace, the following need to be configured:

1. [Service account and rolebinding for workflows](#service-accounts-and-role-bindings)
2. [Storage initializer secret for retrieving data](#storage-initializer-secret)

### Service Accounts and Role Bindings

A service account and rolebinding need to be created to allow the Seldon Enterprise Platform server to access and create Argo Workflows:

```bash
export NAMESPACE=seldon

cat << EOF > workflow-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - "*"
- apiGroups:
  - "apps"
  resources:
  - deployments
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - "*"
- apiGroups:
  - machinelearning.seldon.io
  resources:
  - "*"
  verbs:
  - "*"
EOF

kubectl apply -n ${NAMESPACE} -f workflow-role.yaml

kubectl create -n ${NAMESPACE} serviceaccount workflow

kubectl create rolebinding -n ${NAMESPACE} workflow --role=workflow --serviceaccount=${NAMESPACE}:workflow
```

### Storage Initializer Secret

Argo Workflows in Seldon Enterprise Platform use a [storage initializer mechanism](https://docs.seldon.ai/seldon-enterprise-platform/operations/storage-initializers) similar to one used for [Prepackaged Model Servers](https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html#init-containers) in order to access data from external data stores.

Secrets containing the [storage access credentials need to be created](https://docs.seldon.ai/seldon-enterprise-platform/operations/storage-initializers#configuration).

The format of these secrets depends on the storage initializer. By default, Seldon Enterprise Platform uses [the Rclone-based storage initializer](https://github.com/SeldonIO/seldon-core/tree/master/components/rclone-storage-initializer), which is specified in `install-values.yaml`:

```yaml
batchjobs:
  storageInitializer:
    image: seldonio/rclone-storage-initializer:1.18.2
```

## Running on GKE or inside Kind cluster

If running inside `kind` cluster or on GKE you must patch Argo's config. For example:

```bash
kubectl patch -n argo configmap workflow-controller-configmap --type merge \
    -p '{"data": {"config": "containerRuntimeExecutor: k8sapi"}}'
```

## Verification and Debugging

You can check the status of Argo Workflows by going to the Argo Workflows UI. First port-forward the web server:

```
kubectl port-forward -n argo svc/argo-server 2746
```

Then go to `https://localhost:2746/` in the browser.

If Argo Workflows is set up correctly then you should be able to run the [batch demo](https://docs.seldon.ai/seldon-enterprise-platform/demos/seldon-core-v1/batch-requests).

To see running jobs, you can use the Argo Workflows UI, or its CLI, if you install it. You can list jobs in the namespace with `argo list -n <namespace>`. An `argo get` tells you the pod names of the steps.

To see logs for a running job, go to the relevant pod. If you don't have the Argo Workflows CLI you can work out the pod name as there should be a pod in the namespace with a running status and a name similar to the model name.