Logging and Log Level

Out of the box, your Seldon deployments will be pre-configured to a sane set of defaults when it comes to logging. These settings involve both the logging level and the structure of the log messages.

These settings can be changed on a per-component basis.

Log level

By default, all the components in your Seldon deployment will come out of the box with INFO as the default log level.

To change the log level you can use the SELDON_LOG_LEVEL environment variable. In general, this variable can be set to the following log levels (from more to less verbose):

• DEBUG

• INFO

• WARNING

• ERROR

Python inference servers

.. Note:: Setting the SELDON_LOG_LEVEL to WARNING and above in the Python wrapper will disable the server's access logs, which are considered INFO-level logs.

When using the Python wrapper (including the MLflow, SKLearn and XGBoost pre-package servers), you can control the log level using the SELDON_LOG_LEVEL environment variable. Note that the SELDON_LOG_LEVEL variable has to be set in the respective container within your inference graph.

For example, to set it in each container running with the python wrapper, you would do it as follows by adding the environment variable SELDON_LOG_LEVEL to the containers running images wrapped by the python wrapper:

"spec": {
  // ...
  "predictors": [
    {
      "componentSpecs": [
        {
          "spec": {
            "containers": [
                { 
                    "name": "mymodel",
                    "image": "x.y:123",
                    "env": [
                        {
                            "name": "SELDON_LOG_LEVEL",
                            "value": "DEBUG"
                        }
                    ]
                }
            ]
          }
        }
      ]
    }
  ]
  // ...
}

Once this has been set, it's possible to use the log in your wrapper code as follows:

import logging

log = logging.getLogger()
log.debug(...)

Log level in the service orchestrator

To change the log level in the service orchestrator, you can set the SELDON_LOG_LEVEL environment variable on the svcOrchSpec section of the SeldonDeployment CRD:

"spec": {
  // ...
  "predictors": [
    {
      "svcOrchSpec": {
          "env": [
              {
                  "name": "SELDON_LOG_LEVEL",
                  "value": "DEBUG"
              }
          ]
      }
    }
  ]
  // ...
}

Log format and sampling

By default, Seldon's service orchestrator and operator will serialise the log messages as JSON and will enable log sampling. This behaviour can be disabled by setting the SELDON_DEBUG variable to true. Note that this will enable "debug mode", which can also have other side effects.

For example, to change this on the service orchestrator, you would do:

"spec": {
  // ...
  "predictors": [
    {
      "svcOrchSpec": {
          "env": [
              {
                  "name": "SELDON_DEBUG",
                  "value": "true"
              }
          ]
      }
    }
  ]
  // ...
}