Authorization

Concepts and configuration for authorization in Seldon Enterprise Platform

User Identity

Once a user has been authenticated, they have an identity in the form of an OIDC ID token. The token defines a user ID and groups to which the user belongs. This information can be used to authorize their actions.

The user ID is determined by a claim in the OIDC token. This defaults to preferred_username, but can be configured through the environment variable USERID_CLAIM_KEY in install-values.yaml. This can be used for both human users and service accounts. If it is not possible to share claim keys for both, it is possible to use the environment variable SA_ID_CLAIM_KEY for service accounts.

Note: the USERID_CLAIM_KEY is always used if present. SA_ID_CLAIM_KEY is used only if USERID_CLAIM_KEY does not exist in the token.

env:
  USERID_CLAIM_KEY: "preferred_username"
  SA_ID_CLAIM_KEY: "preferred_username"

The groups are determined by a claim in the OIDC token. This defaults to groups, but can be configured through the environment variable GROUPS_CLAIM_KEY in install-values.yaml.

env:
  GROUPS_CLAIM_KEY: "groups"

Claim is a technical term in the OpenID specification. It means a piece of information about a user, or "entity". You can read more about claims here.

Permission Evaluation

Every call to the Seldon Enterprise Platform API is protected by authorization checks. This happens regardless of whether you are interacting with it via the UI, via the Python SDK, or directly.

Seldon Enterprise Platform operates on a restrictive model. This means that a user is assumed to have no permissions except those which they have been explicitly granted.

The effective list of permissions that a user has is the union of:

• Those that are granted to them by their user ID.

• Those that are granted to all users.

• Those that are granted to one or more of the groups to which the user belongs.

Note: There is no way to define negative permissions, i.e. that a user is denied access to a particular resource.

Resource Actions

There are three actions users can be granted for resources:

• Read for listing available resources and viewing particular ones.

• Write for creating, deleting, and modifying resources.

• Grant for granting other users permissions to specific resources.

These actions do not imply one another. For example, having write access does not also grant read access. Each action is handled completely independently in the OPA policies, and each grant must be specified explicitly.

Resource Scopes

Resources in Enterprise Platform are grouped into three categories:

• Namespace-scoped resources that belong to a specific Kubernetes namespace. This includes ML deployments and artifact secrets.

• Project-scoped resources, including models in the Model Catalog and deployments using these. Projects are a Seldon Enterprise Platform concept, but not a Kubernetes one.

• System resources, such as authorization policies themselves!

Resources can sometimes be both project-scoped and namespace-scoped. For example, an ML deployment is namespace-scoped because it exists in Kubernetes, and is also project-scoped based on models it uses. When this happens, a user needs access to both the namespace and all projects associated with that deployment to be able to view or interact with it.

Projects

A project defines a logical grouping of related resources.

At present, models are the only resources that are explicitly associated with projects. Any model that does not explicitly declare its associated project will be implicitly assigned to the default project.

A project can have multiple models associated with it, but each model is only associated with a single project. Projects-to-models is a one-to-many relationship.

As deployments reference models, they are also protected by project-based policies. Deployments can reference multiple models, so can be associated with multiple different projects. A user must have suitable permissions for all projects associated with a deployment in order to interact with it.

Note: The term "model" here refers to inference models, explainers, and detectors.

Note: It is advised to have a policy such that the default project is accessible to everyone.

Namespaces

A namespace defines a physical grouping of resources.

Any resources related to deployments will exist in a namespace, including deployed models, explainers, detectors, canaries, shadows, and artifact storage secrets.

A namespace can have many resources associated with it, but each resource is only associated with a single namespace. Namespaces-to-resources is a one-to-many relationship.

System

System resources are resources that relate to administrative concerns for the Seldon Enterprise Platform platform, instead of ML models and deployments.

Currently the only system-level resources that can be protected by access controls in the Seldon Enterprise Platform API are authorization policies themselves. As a result, the initial policies must be defined by someone with access to the Kubernetes cluster.

Associating Scopes With Kubernetes Resources

As namespaces are a native Kubernetes concept, there is a natural association between namespaced resources and their associated namespace.

In contrast, projects are not a native Kubernetes concept and so Seldon Enterprise Platform must record the association between a resource and its project somewhere. It does this via Kubernetes metadata: annotations and labels.

When creating deployments through the Seldon Enterprise Platform UI, you can use the pre-filled value or specify the appropriate project for your model. When creating deployments via the Seldon Enterprise Platform API, whether directly or with the Seldon Enterprise Platform SDK, you must provide the appropriate project metadata in the resource definition. In either case, when no project is explicitly specified then the default project will be used.

For Seldon Core v1, projects are defined via annotations at the predictor level. Each predictor in a deployment's spec.predictors has an annotations field that can contain annotations of the following form:

"project.seldon.io/<name of component in graph>": "<name of project>"

Example - Seldon Core v1

Given below is a heavily simplified example of a Seldon Core v1 deployment with a single-model predictor.

The key things to note are the annotations field and the metadata.namespace field. The former corresponds to the mock-example-container model in the graph and associates it with project-a. The latter defines the namespace to be seldon-deployments.

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: mock-deployment
  namespace: seldon-deployments
spec:
  name: mock-example
  predictors:
  - name: default
    annotations:
      project.seldon.io/mock-example-container: project-a
    graph:
      name: mock-example-container
      type: MODEL
    componentSpecs:
    - spec:
        containers:
        - name: mock-example-container
          image: "seldonio/mock_classifier:1.5.0"

Example - Seldon Core v2

Given below is an example of a Seldon Core v2 Model.

The key things to note here are the metadata.annotations field, which associates the model with project-a, and the metadata.namespace field, which defines the namespace to be seldon-deployments.

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
  namespace: seldon-deployments
  annotations:
    seldon.io/project: project-a
spec:
  storageUri: "gs://seldon-models/mlserver/iris"
  requirements:
  - sklearn
  memory: 100Ki

Authorization Methods

Currently, there are two ways of defining permissions on the various resources served by Seldon Enterprise Platform:

• OPA policies (recommended)

• Namespace labels (deprecated)

OPA policies are the recommended authorization method for Seldon Enterprise Platform as they can be used to authorize all resource scopes.

OPA Policies

Seldon Enterprise Platform can use Open Policy Agent (OPA) policies to determine if a user has access to a resource. OPA is popular, flexible open-source technology for defining policies. You can enable it by following the installation guide.

Enable OPA

To enable the use of OPA policies for the namespace and system scopes, set the following Helm variables in install-values.yaml:

rbac:
  opa:
    enabled: true

  nsLabelsAuth:
    enabled: false

As OPA policies cannot be used with label-based authorization, this should be disabled as shown above. However, Seldon Enterprise Platform still requires the seldon.restricted label to be present on namespaces in order to detect them. When using OPA policies, it does not matter whether the value of this label is true or false.

If you also want to enable OPA for projects, set the following configuration in install-values.yaml:

rbac:
  opa:
    projectAuthEnabled: true

Policy Management APIs

Seldon Enterprise Platform has API endpoints for managing OPA policies. These can be found in the API documentation.

By default, these are enabled and access can be granted to them on a per-user or group basis. If you want to disable these, set the following Helm value:

rbac:
  opa:
    permissionManagementAPIDisabled: true

Policy Schema

OPA policies are defined in a JSON document as resource-action pairs for users or groups. This JSON document exists in a ConfigMap, as discussed in the installation guide.

The structure of policies is as follows:

{
  "role_grants": {
    "<group name>": [
      {
        "resource": "<resource>",
        "action":   "<action>"
      },
      // More permissions for this group
    ],
    // More groups
  },
  "user_grants": {
    "<user name>": [
      {
        "resource": "<resource>",
        "action":   "<action>"
      },
      // More permissions for this user
    ],
    // More users
  }
}

The top-level fields of the document are the grant types: role_grants and user_grants. These are mandatory fields and correspond to groups and user IDs respectively.

Each grant type defines a map from user/group name to a list of permissions. These user or group names are called grant targets.

Each permission is defined as a map, describing an action on a resource scope. For example this permission allows read access to resources in a namespace called seldon:

{
  "action": "read",
  "resource": "namespace/seldon"
}

Note: When you do not want to specify either user or role grants, use an empty JSON object as the value.

{
  "user_grants": {},
  "role_grants": {
    // ...
  }
}

{
  "user_grants": {
    // ...
  },
  "role_grants": {}
}

Granting Access For All Users

For user grants, there is a special grant target called the star user and denoted *. This special target can be used to provide policies for all users.

To see how this can be used, please check the examples.

Note: This special star user is not a glob but rather a well-known value that Seldon Enterprise Platform understands.

Seldon Enterprise Platform does not support globbing for user and group names. For example, a grant target like abc* would not be valid. If you need to define permissions for a collection of users, please use role grants.

Resource Names

Resources are identified by the scope as a prefix and a name.

For the system scope, there is only one possible value:

• system/iam

For the project scope, aside from the default project the names are determined by you:

• project/default

• project/foo

• ...

For the namespace scope, the names should correspond to Kubernetes namespaces for deploying ML models:

• namespace/seldon

• namespace/seldon-gitops

• ...

Seldon Enterprise Platform supports globbing of resource names with an asterisk (*). It is valid to use globbing anywhere within the resource name, but not for the resource scope.

For example, these would all be allowed:

• "resource": "namespace/*"

• "resource": "namespace/seldon-*"

• "resource": "namespace/*-gitops"

• "resource": "project/dev*"

On the other hand, these would all be prohibited:

• "resource": "*"

• "resource": "*/*"

• "resource": "*/seldon"

• "resource": "name*/seldon"

Note: For more details on the specifics of each resource type, please see the namespace and project sections.

Examples

Giving everyone access to resources in a scope is possible using the special "star" user, *. An example of giving all users access to the default project looks like this:

{
  "role_grants": {},
  "user_grants": {
    "*": [
      {
        "action": "read",
        "resource": "project/default"
      },
      {
        "action": "write",
        "resource": "project/default"
      }
    ]
  }
}

Giving a group access to resources in a scope is possible using role_grants. An example of giving the data-scientist group access to the prod namespace looks like this:

{
  "role_grants": {
    "data-scientist": [
      {
        "action": "read",
        "resource": "namespace/prod"
      },
      {
        "action": "write",
        "resource": "namespace/prod"
      }
    ]
  },
  "user_grants": {}
}

Giving a user access to resources in a scope is possible using user_grants. An example of giving the alice user access to the system/iam scope looks like this:

{
  "role_grants": {},
  "user_grants": {
    "alice": [
      {
        "action": "read",
        "resource": "system/iam"
      },
      {
        "action": "write",
        "resource": "system/iam"
      }
    ]
  }
}

An example of a complete policy file is given below.

In this example, users belonging to the data_scientist group will be able to create, modify, and read models from the iris project, as well as see deployments in the seldon namespace. However, they do not have permission to create or modify deployments in that namespace.

Members of the ml_engineer group can create deployments from the iris project in the seldon namespace. They are also allowed to interact with other resources in the seldon namespace, such as storage secrets.

Members of the ops group can view all resources in namespaces with the seldon prefix (including seldon itself), and can see the full Model Catalog.

The * user grant specifies that all users have read and write permissions for the default project and default namespace.

Only the user alice has access to the alice namespace.

{
  "role_grants": {
    "data_scientist": [
      {
        "action": "read",
        "resource": "project/iris"
      },
      {
        "action": "write",
        "resource": "project/iris"
      },
      {
        "action": "read",
        "resource": "namespace/seldon"
      }
    ],
    "ml_engineer": [
      {
        "action": "read",
        "resource": "project/iris"
      },
      {
        "action": "write",
        "resource": "project/iris"
      },
      {
        "action": "read",
        "resource": "namespace/seldon"
      },
      {
        "action": "write",
        "resource": "namespace/seldon"
      }
    ],
    "ops": [
      {
        "action": "read",
        "resource": "namespace/seldon*"
      },
      {
        "action": "read",
        "resource": "project/*"
      }
    ]
  },
  "user_grants": {
    "*": [
      {
        "action": "read",
        "resource": "project/default"
      },
      {
        "action": "write",
        "resource": "project/default"
      },
      {
        "action": "read",
        "resource": "namespace/default"
      },
      {
        "action": "write",
        "resource": "namespace/default"
      }
    ],
    "alice": [
      {
        "action": "read",
        "resource": "namespace/alice"
      },
      {
        "action": "write",
        "resource": "namespace/alice"
      }
    ]
  }
}

Policy Operations

Interacting With Policies

The policy file can be modified in various ways:

• Using kubectl

• Using Seldon's policy_client command-line utility

• Using the permission management endpoints of the Seldon Enterprise Platform API

You can interact with the ConfigMap directly using kubectl. If your policies are defined in a JSON file called policy.json, you can turn it into a Kubernetes manifest called policy.yaml using the following command:

kubectl -n seldon-system create configmap seldon-deploy-policies --from-file=data=policy.json --dry-run=client -o yaml > policy.yaml

This resulting policy.yaml file can be tracked under version control if you wish. You can overwrite the existing policies by running the following:

  kubectl apply -f policy.yaml

The policy_client command-line utility can be downloaded from the seldon-deploy-resources repository. You will need to download the correct binary for your operating system and architecture, e.g. Linux AMD x64. For full documentation of the tool you can run policy_client --help.

The permission management API is provided via the /iam/policy endpoints discussed in the API documentation.

Seldon Enterprise Platform will automatically reload the policies if it detects that they have changed. This is dependent on how the file is mounted and how quickly Kubernetes propagates changes. If you want to ensure that Seldon Enterprise Platform definitely has the latest changes, you can restart it using:

kubectl -n seldon-system rollout restart deployment seldon-deploy

Adding Policies

To add a new policy you can run:

policy_client addPolicy --resource=namespace/seldon --action=write --target_users=alice --target_groups=admin

The arguments target_groups and target_users can take multiple entries.

Deleting Policies

Similarly to adding, to remove an existing policy you can run:

policy_client removePolicy --resource=namespace/seldon --action=write --target_users=alice --target_groups=admin

Limitations

Using a ConfigMap for the policy file is flexible and easy to setup. However, it comes with a limitation which is true for all Kubernetes resources - it is limited to 1MB size, since that is the underlying etcd value limit.

If your use case requires a bigger size for the policy file consider mounting the file from somewhere else like an object store (S3, minio), or a database.

We are working on improving support for such use cases, which should be available in a future release.

Migrating From Namespace Labels

To migrate from using namespace labels for defining access requirements to a namespace you can use the following command:

policy_client sync

It will create a ConfigMap called seldon-deploy-policies in the seldon-system namespace if one is not present. The name and namespace are configurable by flags. It will then populate policies corresponding to the namespace labels in the cluster. The command will not delete any existing policies in the ConfigMap.

To confirm the change has been successful you can check the ConfigMap by running:

kubectl get cm -n seldon-system seldon-deploy-policies -o jsonpath='{.data.data}' | jq

Namespace Labels

This approach has been deprecated and will be removed in the future. For a more complete and efficient authorization solution, please use Open Policy Agent policies.

Label-based and OPA policy-based authorization cannot be used at the same time. Attempts to do so will result in only OPA policies taking effect.

Seldon Enterprise Platform can use labels on Kubernetes namespaces to determine if a user has access to resources within that namespace. Note that namespace labels can only be used for authorizing access to namespaces, but not to projects.

To authorize a user or a group to access resources in a namespace, add labels in the following manner along with the associated permission.

seldon.user.<user name>: "write"
seldon.group.<group name>: "read"

As an example, setting permissions on a namespace for user alice, service account sd_api and group ml_engineer would involve adding labels as follows,

seldon.user.alice: "write"
seldon.user.sd_api: "read"
seldon.group.ml_engineer: "read"

Note:Service accounts are also treated as users for authorization purposes and the service account user identity can be configured using a separate token claim if needed.

Authorization Outside Seldon Enterprise Platform

Seldon Enterprise Platform runs in a Kubernetes platform which contains several other components. Access to these components should also be controlled. How this is done can vary based on your requirements and pre-existing access controls.

Seldon Core

Note:This feature is only supported in Core v1 as of now and requires Istio to be installed in the cluster.

One of the main features of Seldon Enterprise Platform is the ability to create Seldon Core deployments and allow users to make requests to these deployments. For requests to be made in a secure manner, these endpoints need to be authorized appropriately.

Seldon Enterprise Platform can automatically synchronize Istio authorization policies based on the permissions set in its OPA policies. Please refer to the production installation guide for enabling this functionality.

This functionality is controlled by the rbac.opa.istioPolicySyncInterval Helm variable. You can enable synchronization by supplying a strictly positive value compatible with Go's time.ParseDuration function. Using any other value, such as a negative duration or leaving the Helm parameter empty, will disable this functionality. To use the suggested interval of 5 minutes, please set the following in install-values.yaml:

rbac:
  opa:
    istioPolicySyncInterval: "5m"

Alternatively, the access to these endpoints can be manually controlled via the Istio configuration as described in the Seldon Core documentation.

Seldon Enterprise Platform provides the ability to proxy requests to Seldon Core deployments via its API. If these requests go through Seldon Enterprise Platform, then they will already be authorized based on the user's permissions.

However, it is not recommended to do this except for testing and debugging purposes, as Seldon Enterprise Platform is not designed as a high-performance proxy and cannot be configured as one. For reasons of latency, scalability, and availability you should use dedicated networking infrastructure like an ingress controller or service mesh.

Other Components

As part of the production installation, other components can be installed in your environment. Please refer to the official documentation for each such technology for details on how to configure authorization for them.

Configuration Reference

The full list of Helm values for authorization is given below. These can be set in install-values.yaml. Please refer to the above documentation for how to configure each setting.

rbac:
  opa:
    enabled: true
    projectAuthEnabled: true
    permissionManagementAPIDisabled: false
    istioPolicySyncInterval: ""
    configMap: seldon-deploy-policies
  nsLabelsAuth:
    enabled: false