> For the complete documentation index, see [llms.txt](https://docs.seldon.ai/seldon-enterprise-platform/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seldon.ai/seldon-enterprise-platform/operations/storage-initializers.md).

# Storage Initializers

{% hint style="info" %}
**Note**: Managing secrets directly in Kubernetes is still supported, but the recommended approach is to use Enterprise Platform's [secrets management tooling.](/seldon-enterprise-platform/operations/secrets-management.md)
{% endhint %}

Some features in Seldon Enterprise Platform require access to external storage providers, for example, to run batch jobs.

A storage initializer mechanism similar to the one used for [Prepackaged Model Servers](https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html#init-containers) is used by Seldon Enterprise Platform. By default, Seldon Enterprise Platform uses the [Rclone-based storage initializer](https://github.com/SeldonIO/seldon-core/tree/master/components/rclone-storage-initializer). [Rclone](https://rclone.org/) offers compatibility with over 40 different cloud storage products and is therefore a sensible default.

However, custom storage initializers may be used by modifying the `storageInitializer` values in `install-values.yaml`.

## Configuration

Rclone storage initializers are configured using environmental variables passed as **secrets** in Seldon Enterprise Platform. **These are configured per namespace**.

Each remote storage provider needs to be configured using environment variables that follow the following pattern:

```
RCLONE_CONFIG_<remote name>_<config variable>: <config value>
```

{% hint style="info" %}
**Note**: Multiple remotes can be configured simultaneously. For example, both `s3` (AWS S3) and `gs` (Google Cloud Storage) remotes can be configured for the same storage initializer.
{% endhint %}

Once the remote is configured, the `modelUri` that is compatible with `rclone` takes the form:

```
modelUri: <remote>:<bucket name>
```

For example `modelUri: minio:sklearn/iris`, or `modelUri: gs:seldon-models/cifar10`. **Rclone will remove the leading slashes for buckets** so this is equivalent to `minio://sklearn/iris` or `gs://seldon-models/cifar10`.

Below you will find a few example configurations. For other storage providers, please consult the [Rclone documentation](https://rclone.org/).

Please note that you need the labels `secret-type: bucket` and `seldon-deploy: "true"` on your secret for it to be correctly picked up by Enterprise Platform. Without these labels it will not show in the secrets dropdown in the deployment creation wizard.

### MinIO

This is an example of how to set up secrets so that Rclone may access [MinIO as configured in the production installation](/seldon-enterprise-platform/production-environment/minio.md).

Reference: [Rclone documentation](https://rclone.org/s3/#minio)

```bash
export MINIONAMESPACE=minio-system
export MINIOUSER=minioadmin
export MINIOPASSWORD=minioadmin
export NAMESPACE=seldon

cat << EOF > seldon-rclone-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: minio-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: minio
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: ${MINIOUSER}
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ${MINIOPASSWORD}
  RCLONE_CONFIG_S3_ENDPOINT: http://minio.${MINIONAMESPACE}.svc.cluster.local:9000
EOF

kubectl apply -n ${NAMESPACE} -f minio-bucket.yaml
```

### Public GCS configuration

{% hint style="info" %}
This is configured by default in the `seldonio/rclone-storage-initializer` image.
{% endhint %}

Reference: [Rclone documentation](https://rclone.org/googlecloudstorage/).

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: gs-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_GS_TYPE: google cloud storage
  RCLONE_CONFIG_GS_ANONYMOUS: "true"
```

### AWS S3 with access key and secret

Reference: [Rclone documentation](https://rclone.org/s3#aws-s3)

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: s3-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ENV_AUTH: "false"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "<your AWS_ACCESS_KEY_ID here>"
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "<your AWS_SECRET_ACCESS_KEY here>"
```

### Example AWS S3 with IAM roles configuration

Reference: [Rclone documentation](https://rclone.org/s3/#aws-s3)

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: s3-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_S3_TYPE: s3
  RCLONE_CONFIG_S3_PROVIDER: aws
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: ""
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: ""
  RCLONE_CONFIG_S3_ENV_AUTH: "true"
```

### GCP/GKE

Reference: [Rclone documentation](https://rclone.org/googlecloudstorage/)

For GCP/GKE, you will need create a service-account key and save it as a local `json` file.

First make sure that you have a gcloud service account (`[SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com`) that has sufficient permissions to access the bucket with your models (i.e. `Storage Object Admin`). You can check this using the [gcloud console](https://cloud.google.com/sdk/docs/install).

Next, generate `keys` locally using the `gcloud` tool. This will create your service-account key file at `gcloud-application-credentials.json`:

```bash
gcloud iam service-accounts keys create gcloud-application-credentials.json --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
```

Use the content of the locally saved `gcloud-application-credentials.json` file to create the rclone secret:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: gcs-bucket
  labels:
    secret-type: bucket
    seldon-deploy: "true"
type: Opaque
stringData:
  RCLONE_CONFIG_GCS_TYPE: google cloud storage
  RCLONE_CONFIG_GCS_ANONYMOUS: "false"
  RCLONE_CONFIG_GCS_SERVICE_ACCOUNT_CREDENTIALS: '{"type":"service_account", ... <rest of gcloud-application-credentials.json>}'
```

{% hint style="info" %}
The remote name is `gcs` here so urls would take form similar to `gcs:<your bucket>`.
{% endhint %}

### Custom Storage Initializer

If for some reason you would like to use a different storage initializer for batch jobs, e.g. KFServing storage initializer, you can set this by modifying `install-values.yaml`:

```yaml
batchjobs:
  storageInitializer:
    image: gcr.io/kfserving/storage-initializer:v0.4.0
```

The corresponding secret for[ MinIO as configured in the production installation](/seldon-enterprise-platform/production-environment/minio.md) would also need to be created:

```bash
export MINIOUSER=minioadmin
export MINIOPASSWORD=minioadmin
export NAMESPACE=seldon

cat << EOF > seldon-kfserving-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: seldon-kfserving-secret
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: ${MINIOUSER}
  AWS_SECRET_ACCESS_KEY: ${MINIOPASSWORD}
  AWS_ENDPOINT_URL: http://minio.minio-system.svc.cluster.local:9000
  USE_SSL: "false"
EOF

kubectl apply -n ${NAMESPACE} -f seldon-kfserving-secret.yaml
```