# GitOps

This page provides a detailed overview of how to set up the GitOps functionality for Seldon Enterprise Platform. Setting up GitOps is highly recommended.

Seldon Enterprise Platform leverages [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) to implement GitOps in your cluster. You can read more about Seldon Enterprise Platform's GitOps integration in the [architecture](/seldon-enterprise-platform/architecture.md) and [operation](/seldon-enterprise-platform/operations.md) sections.

This documentation page will walk you through the different steps required to set up ArgoCD and GitOps in your cluster. Note that this guide assumes a fresh cluster without ArgoCD, thus it must only be treated as a general reference. Feel free to adapt these steps to your particular infrastructure.

## Prerequisites

* Download the [installation resources](/seldon-enterprise-platform/production-environment/observability-alerting/observability.md#installing-kube-prometheus).
* Verify the [cluster requirements](/seldon-enterprise-platform/production-environment.md#cluster-requirements).
* Install [Seldon Enterprise Platform](/seldon-enterprise-platform/production-environment/seldon-enterprise-platform.md).
* Install [Seldon Core 2](/seldon-enterprise-platform/production-environment/seldon-core-2.md).

## ArgoCD Installation

Since this guide assumes that you currently don't have ArgoCD installed in your cluster, the first step will be on installing and configuring it. You can treat these steps as a quick way to get started with ArgoCD. Therefore, it is **highly encouraged** to check [ArgoCD's official documentation](https://argo-cd.readthedocs.io/en/stable/getting_started/) for further configuration.

To install and configure ArgoCD we will first install their official K8s manifest into our cluster. For version `v2.6.7` of ArgoCD, this can be done as follows:

```bash
ARGOCD_VERSION=v2.6.7

kubectl create namespace argocd || echo "namespace exist"
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/${ARGOCD_VERSION}/manifests/install.yaml

kubectl rollout status deployment/argocd-applicationset-controller -n argocd
kubectl rollout status deployment/argocd-repo-server -n argocd
kubectl rollout status deployment/argocd-server -n argocd
kubectl rollout status deployment/argocd-redis -n argocd
kubectl rollout status deployment/argocd-dex-server -n argocd
```

## ArgoCD Configuration

Now that we have ArgoCD installed in our cluster, the next step will involve configuring it so that:

* The admin user credentials get changed.
* SSL termination is disabled in ArgoCD.
* ArgoCD gets exposed under the `/argocd/` path of our load balancer.

Note that these steps must be treated just as a guideline. Therefore, you may need to adapt them to your infrastructure. For more information on ArgoCD's configuration, we suggest checking [ArgoCD's official documentation](https://argo-cd.readthedocs.io/en/stable/getting_started/).

### Admin user and password

Upon the initial installation of ArgoCD, it is recommended to change its user and password. This guideline will show how to change ArgoCD's default admin user as an example. However, it's also possible to hook [ArgoCD to a centralised OIDC provider](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/). This could be the same as [Seldon Enterprise Platform's OIDC provider](/seldon-enterprise-platform/production-environment/auth.md), providing a Single Sign-On across your entire stack.

The following command will assume that we want to set the admin credentials as `admin` // `$ARGOCDINITIALPASS`. ArgoCD requires the password to be [provided as a bcrypt hash](https://github.com/argoproj/argo-cd/blob/master/docs/faq.md#user-content-i-forgot-the-admin-password-how-do-i-reset-it). With this in mind, we can set those credentials as:

```bash
ARGOCDINITIALPASS=12341234
kubectl patch secret \
    -n argocd argocd-secret \
    -p '{"stringData": { "admin.password": "'$(htpasswd -bnBC 10 "" ${ARGOCDINITIALPASS} | tr -d ':\n')'"}}'
```

This command requires `htpasswd` that can be installed on Ubuntu, Debian and Mint systems with:

```bash
sudo apt install apache2-utils
```

### SSL Termination

As a general choice, we recommend to handle SSL termination at the ingress / load balancer level. This simplifies the components setup within the cluster.

Following this approach, we will need to disable the SSL termination in ArgoCD. Otherwise, ArgoCD will expect to receive SSL traffic by default. To do this, we can ask ArgoCD to run in `insecure` mode by running the command below:

```bash
kubectl patch deploy argocd-server \
    -n argocd \
    -p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--insecure"}]' \
    --type json
```

### Exposing ArgoCD through our ingress

There are different ways to access ArgoCD. However, the most convenient is usually to expose it through our ingress layer so that it can be accessed as a different subpath (`/argocd/`). This will make the ArgoCD API and UI reachable as `http(s)://<load-balancer-domain>/argocd/`.

This will need to be configured in a couple of places.

1. On one hand, we need to tell ArgoCD to expect its static assets to be served under the `/argocd/` path. This can be done by running the following command:

   ```bash
   kubectl patch deploy argocd-server \
       -n argocd \
       -p '[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--rootpath"}, {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "/argocd"}]' \
       --type json
   ```
2. Create a `VirtualService` that maps the `/argocd/` path to ArgoCD's service. To expose ArgoCD as a VirtualService, you can create the file `argocd-vs.yaml` file:

```yaml
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: argocd-server
     namespace: argocd
   spec:
     gateways:
     - istio-system/seldon-gateway
     hosts:
     - '*'
     http:
     - match:
       - uri:
           prefix: /argocd/
       route:
       - destination:
           host: argocd-server
           port:
             number: 80
```

3. Apply the configuration: `kubectl apply -f argocd-vs.yaml`

## Accessing ArgoCD

After configuring ArgoCD, we should now be able to reach ArgoCD under:

```
http(s)://<load-balancer-domain>/argocd/
```

In a standard setup, you should be able to obtain your load balancer domain by checking the Istio resources created in your cluster:

```bash
INGRESS=$(\
  kubectl get svc -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}'\
)
INGRESS+=$(\
  kubectl get svc -n istio-system istio-ingressgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'\
)

echo $INGRESS
```

Alternatively, if your ingress layer doesn't expose an external IP address or domain, it's also possible to *port-forward* a local port into ArgoCD. For example, you can access ArgoCD locally at `localhost:8080` or `localhost:8080/argocd/` if you defined the `rootpath` above by running:

```bash
kubectl port-forward -n argocd svc/argocd-server 8080:80
```

Once we know our load balancer domain, we can then access ArgoCD through its UI or API. For the latter, we will leverage ArgoCD's CLI, called `argocd`. Below, you can find instructions on how to set up and access both of them.

{% tabs %}
{% tab title="CLI" %}
ArgoCD ships with a CLI tool called `argocd`.

This tool allows you to interact with ArgoCD from the command line. The rest of this guide will provide command examples that can be used to configure ArgoCD using their \`argocd\` tool.

However, it's also possible to perform the same actions from the ArgoCD UI.

To use the \`argocd\` CLI, we will need to:

1\. Install the `argocd` CLI locally.

This can be achieved by running the following commands:

```
ARGOCD_VERSION=v2.6.7
   wget -q -O argocd "https://github.com/argoproj/argo-cd/releases/download/${ARGOCD_VERSION}/argocd-linux-amd64"
   chmod +x ./argocd
```

2\. Authenticate the CLI against our ArgoCD instance.

Assuming that our load balancer domain can be found under `$INGRESS`, it should be enough to run the command below.

```
./argocd login \
     "$INGRESS:80" \
     --username admin \
     --password ${ARGOCDINITIALPASS} \
     --insecure \
     --grpc-web-root-path /argocd
```

{% endtab %}

{% tab title="UI" %}
ArgoCD exposes a UI that can be used to configure your GitOps repository.

Assuming that you have followed the [ArgoCD setup](#argocd-configuration) guidelines, this UI should be accessible under the `/argocd/` path of your load balancer domain. After accessing the UI, you should be prompted with a login form asking for the admin credentials. You should be able to access this UI by using ArgoCD's user or password combination `admin`and `$ARGOCDINITIALPASS` by default, respectively. You can learn more about how to navigate ArgoCD's UI in the [documentation](https://argo-cd.readthedocs.io/en/stable/).
{% endtab %}
{% endtabs %}

## GitOps for logging components

The request logger is an infrastructure component that runs in the [centralised logging namespace](/seldon-enterprise-platform/production-environment/request-logging.md), `seldon-logs` by default. Alongside this, Triggers can also be created for detectors and these tied to **particular models**. Thus is good to back them through a GitOps environment.

With this goal in mind, it's encouraged to **create a system-level GitOps environment** to track these infrastructure resources. To do this, you can follow the [instructions to create a new GitOps environment](#gitops-environment-creation), using the `seldon-logs` namespace as the target.

{% hint style="info" %}
**Note**: If you've configured Seldon Enterprise Platform to use a different centralised namespace for logging, you'll need to point to that namespace instead. Once the logging GitOps environment is created, Seldon Enterprise Platform should be able to pick it up and use it without any further configuration.
{% endhint %}

## GitOps environment creation

Now that ArgoCD has been [installed and configured](#argocd-installation), you can proceed to create a new GitOps environment. This environment provides a Kubernetes namespace to deploy our machine learning models, which is backed by a GitOps repository.

This section walks you through how to create a new GitOps-tracked environment for a namespace with name `$namespace`. The GitOps repository which acts as the source of truth for this environment exists under `$GIT_REPO_URL`.

{% hint style="info" %}
**Note**: the **\`**`` $GIT_REPO_URL` ``should not contain the `http` or `https` protocol schema.
{% endhint %}

## Git Repository

The only requirements for the Git repository are:

* It contains a non-empty folder **named as the namespace** (`./$namespace`).
* It is accessible from within the cluster. Assuming you have already set the shell variables `$GIT_USER` and `$GIT_TOKEN`, this can be verified using this one-liner command:

  ```bash
  kubectl run --quiet=true -it --rm git-clone --image=radial/busyboxplus:git  --restart=Never -- git clone https://$GIT_USER:$GIT_TOKEN@$GIT_REPO_URL
  ```

If you are using a private repository and see an error the reads: `fatal: repository 'YOUR GIT REPOSITORY' not found`, it is likely your \`$GIT\_TOKEN\` has insufficient permissions.

For GitHub, your token needs the `repo` scope for full control of private repositories. For more information, see [generating a token and setting the right scopes](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens). For other providers, check the relevant docs for token-based repository access.

## ArgoCD Resources

In order for ArgoCD to keep track of our new environment, it's necessary to configure a [*Project*](https://argo-cd.readthedocs.io/en/stable/user-guide/projects/) and an [*Application*](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#applications). These are concepts native to ArgoCD. We suggest checking the [ArgoCD official documentation](https://argo-cd.readthedocs.io/en/stable/) for more information about them.

ArgoCD projects and applications can be created via the ArgoCD UI, `argocd` CLI tool and through CRDs. As the [declarative](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/) approach guarantees the best reproducibility, this is the approach that we recommend.

You can create an ArgoCD project by creating the following `AppProject` resource named `argocd-project.yaml`

```bash
cat << EOF > ./argocd-project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: seldon
  namespace: argocd
spec:
  description: Seldon Enterprise Platform Project
  sourceRepos:
  - https://${GIT_REPO_URL}
  destinations:
  - namespace: ${namespace}
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  roles:
  - name: seldon-admin
    policies:
    - p, proj:seldon:seldon-admin, applications, get, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, create, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, update, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, delete, seldon/*, allow
    - p, proj:seldon:seldon-admin, applications, sync, seldon/*, allow
EOF

kubectl apply -f argocd-project.yaml
```

Similarly create the ArgoCD `Application` resource:

```bash
ARGO_APP_NAME=seldon-gitops-"${namespace}"

cat << EOF > ./argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ${ARGO_APP_NAME}
  namespace: argocd
spec:
  project: seldon
  destination:
    namespace: ${namespace}
    server: https://kubernetes.default.svc
  source:
    directory:
      recurse: true
    path: ${namespace}
    repoURL: https://${GIT_REPO_URL}
  syncPolicy:
    automated: {}
EOF

kubectl apply -f argocd-application.yaml
```

## Kubernetes Model Namespace

To use a namespace with GitOps, you need to annotate it with the Git repository URL using key `git-repo` and enable it for GitOps access by adding the `seldon.gitops` label. As an example, to enable a namespace named `$namespace`, you could run the following commands:

```bash
kubectl create namespace $namespace || echo "namespace $namespace already exists"

kubectl label namespace $namespace seldon.restricted=false --overwrite=true
kubectl label namespace $namespace seldon.gitops=enabled --overwrite=true
kubectl annotate namespace $namespace git-repo="https://${GIT_REPO_URL}" --overwrite=true
```

{% hint style="info" %}
**Note**: If your ArgoCD application does not follow the `seldon-gitops-${namespace}`

naming convention, you can label the namespace accordingly:

`kubectl label namespace $namespace argocdapp=${ARGO_APP_NAME} --overwrite=true`
{% endhint %}

If you intend to use Batch jobs on the namespace then you'll need additional config. You can visit the [Argo Workflows section](/seldon-enterprise-platform/production-environment/argo-workflows.md) of the docs for more details.

## Configuring Git Credentials

The last required step is to provide the relevant Git credentials to access our GitOps repository. Note that these credentials need to be set at both the ArgoCD and Seldon Enterprise Platform levels.

### ArgoCD

You can configure your GitOps repository access in ArgoCD using either SSH or user / password credentials. These credentials can be provided using the `argocd` CLI tool, or directly through the ArgoCD UI.

{% tabs %}
{% tab title="SSH CLI" %}
Ensure that the `argocd` CLI tool is already [authenticated](#accessing-argocd), and that `${GIT_USER}` and `${GIT_TOKEN` represent GitOps repository user and password, run the following:

```
./argocd repo add "https://${GIT_REPO_URL}" \
    --username ${GIT_USER} \
    --password ${GIT_TOKEN} \
    --upsert
```

{% endtab %}

{% tab title="User / Password (CLI)" %}
Ensure that the `argocd` CLI tool is already [authenticated](#accessing-argocd), and that `$GIT_SSH_PATH` represents the path to an SSH private key with access to GitOps repository, run the following:

```
./argocd repo add "ssh://${GIT_REPO_URL}" \
    --ssh-private-key-path ${GIT_SSH_PATH} \
    --upsert
```

{% endtab %}

{% tab title="UI" %}
ArgoCD exposes a UI that can be used to configure your GitOps repository. After you have followed the [ArgoCD setup guidelines](#accessing-argocd), this UI should be accessible under the `/argocd/` path. You should be able to access this UI by using ArgoCD's user or password combination it is `admin` and `12341234` by default, respectively. For more details, see[how to configure repositories using ArgoCD's UI](https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/) .
{% endtab %}
{% endtabs %}

### Seldon Enterprise Platform

Seldon Enterprise Platform also requires access to the repository. This access is used to monitor the status of the current resources in the cluster, as well as to create new ones.

To configure our Git credentials in Seldon Enterprise Platform, follow these steps:

1. Create a Kubernetes secret containing our credentials, either as a SSH key or a User / Password combination. This secret can have any arbitrary name, but must live in the same namespace as Seldon Enterprise Platform.

{% tabs %}
{% tab title="SSH" %}
If the private key is present under `$GIT_SSH_PATH`, you can create the credentials secret using:

```
kubectl create secret generic git-creds -n seldon-system \
  --from-file=id_rsa=${GIT_SSH_PATH} \
  --from-file=known_hosts=${GIT_KNOWN_HOSTS_PATH} \
  --from-literal=passphrase="${GIT_SSHKEY_PASSPHRASE}" \
  --from-literal=username="${GIT_USER}" \
  --from-literal=email="${GIT_EMAIL}"  \
  --dry-run=client -o yaml | kubectl apply -f -
```

The \`passphrase\` field can be left empty if they SSH key doesn't have a passphrase.
{% endtab %}

{% tab title="User / Password" %}
Create the credentials secret using a User / Password combination or User / Personal Access Token using:

```
kubectl create secret generic git-creds -n seldon-system \
  --from-literal=username="${GIT_USER}" \
  --from-literal=token="${GIT_TOKEN}" \
  --from-literal=email="${GIT_EMAIL}"  \
  --dry-run=client -o yaml | kubectl apply -f -
```

{% endtab %}
{% endtabs %}

2. Update Seldon Enterprise Platform's configuration to point to the newly created secret. In particular, you need to modify the `gitops` section of the values of the Seldon Enterprise Platform Helm chart. Here, you need to set the `gitops.argocd.enabled` flag to `true`, and point the `gitops.git.secret` field to the right secret name `git-creds` .

```yaml
gitops:
  git:
    secret: git-creds
  argocd:
    enabled: true
```

## Webhooks Configuration (optional)

By default, the resource synchronisation against the GitOps repository happens on a poll basis. That is, Seldon Enterprise Platform checks the repo periodically for updates.

The main caveat of this approach is that there may be a small delay between taking an action and seeing it reflected in the cluster or UI. To work around this, you can configure a set of webhooks in the cluster that gets pinged everytime there is a change in the cluster.

{% hint style="info" %}
**Note:** This is an optional step, and its feasibility may depend on your networking infrastructure and your Git provider.\
If webhooks are not arriving at their destination it may be helpful to enable webhook history. It may be necessary to also select any `Skip certificate verification` option.
{% endhint %}

You can find these instructions on how to set up these webhooks for common Git providers.

{% tabs %}
{% tab title="GitHub" %}
1\. Configure the webhook that sends updates to the Seldon Enterprise Platform endpoint:

```
whIp=$(kubectl -n seldon-system get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
GIT_API='api.github.com'
_repo_without_suffix=${GIT_REPO_URL%.git}
REPONAME="${_repo_without_suffix##*/}"

curl -u ${GIT_USER}:${GIT_TOKEN} \
   -H "Content-Type: application/json" \
   -d '
    {
      "name": "web",
      "active": true,
      "events": ["*"],
      "config": {
        "url": "http://'"${whIp}"'/api/git-webhook",
        "content_type": "json"
      }
    }' \
   https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks

```

2. Set `$ARGOCDURL` and create the webhook that sends the updates to ArgoCD:

```
ARGOCDURL="$whIp/argocd"

curl -u ${GIT_USER}:${GIT_TOKEN} \
   -H "Content-Type: application/json" \
   -d '
    {
      "name": "web",
      "active": true,
      "events": ["*"],
      "config": {
        "url": "https://'"${ARGOCDURL}"'/api/webhook",
        "content_type": "json",
        "secret": "'"${ARGOCDINITIALPASS}"'",
        "insecure_ssl": "1"
      }
    }' \
   https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks

```

{% endtab %}

{% tab title="Bitbucket" %}
1\. Configure the webhook that sends updates to the Seldon Enterprise Platform endpoint:

```
whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
GIT_API='api.bitbucket.org'
_repo_without_suffix=${GIT_REPO_URL%.git}
REPONAME="${_repo_without_suffix##*/}"

curl -u ${GIT_USER}:${GIT_TOKEN} \
   -v -H "Content-Type: application/json" \
   -d '
    {
        "description": "web",
        "events": ["repo:push"],
        "url": "'"http://${whIp}/api/git-webhook"'",
        "active": true
    }' \
  https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks

```

2. Set `$ARGOCDURL` and create the webhook that sends the updates to ArgoCD:

```
ARGOCDURL="$whIp/argocd"

curl -u ${GIT_USER}:${GIT_TOKEN} \
   -v -H "Content-Type: application/json" \
   -d '
    {
        "description": "web",
        "events": ["repo:push"],
        "url": "'"https://${ARGOCDURL}/api/webhook"'",
        "active": true
    }' \
  https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks
```

{% endtab %}
{% endtabs %}

## Enabling a namespace for GitOps

After following the preceding steps, further namespaces can be enabled for GitOps by doing the following:

* Create the namespace, if it does not already exist
* [Add that namespace](#argocd-resources) to the appropriate `AppProject`
* [Create a new `Application` resource](#argocd-resources) to deploy into that namespace, if required
* [Annotate the namespace](#kubernetes-model-namespace) for Enterprise Platform to use it

Further context is provided in[ Namespace Setup](/seldon-enterprise-platform/operations/namespace-setup.md).

## Troubleshooting

You should see one application per gitops namespace when running the following:

```
kubectl get application -n argocd
```

You can check the status of an argocd app with `kubectl get application -n argocd seldon-gitops-<namespace_name> -o yaml`.

You can also use the ArgoCD UI to inspect applications. You can trigger a manual sync from there. If a particular application is missing or in error try running its setup again in order to debug.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.seldon.ai/seldon-enterprise-platform/production-environment/gitops.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.