Overview

The Auto-Encoder (AE) outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The AE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.

Usage

Initialize

Parameters:

• threshold: threshold value above which the instance is flagged as an outlier.

• encoder_net: tf.keras.Sequential instance containing the encoder network. Example:

• decoder_net: tf.keras.Sequential instance containing the decoder network. Example:

• ae: instead of using a separate encoder and decoder, the AE can also be passed as a tf.keras.Model.

• data_type: can specify data type added to metadata. E.g. 'tabular' or 'image'.

Initialized outlier detector example:

We then need to train the outlier detector. The following parameters can be specified:

• X: training batch as a numpy array of preferably normal data.

• loss_fn: loss function used for training. Defaults to the Mean Squared Error loss.

• optimizer: optimizer used for training. Defaults to with learning rate 1e-3.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

We detect outliers by simply calling predict on a batch of instances X. Detection can be customized via the following parameters:

• outlier_type: either 'instance' or 'feature'. If the outlier type equals 'instance', the outlier score at the instance level will be used to classify the instance as an outlier or not. If 'feature' is selected, outlier detection happens at the feature level (e.g. by pixel in images).

• outlier_perc: percentage of the sorted (descending) feature level outlier scores. We might for instance want to flag an image as an outlier if at least 20% of the pixel values are on average above the threshold. In this case, we set outlier_perc to 20. The default value is 100 (using all the features).

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances or features are above the threshold and therefore outliers. If outlier_type equals 'instance', then the array is of shape (batch size,). If it equals 'feature', then the array is of shape (batch size, instance shape).

• feature_score: contains feature level scores if return_feature_score equals True.

Isolation Forest

Overview

Isolation forests (IF) are tree based models specifically used for outlier detection. The IF isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. The number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node. This path length, averaged over a forest of random trees, is a measure of normality and is used to define an anomaly score. Outliers can typically be isolated quicker, leading to shorter paths. The algorithm is suitable for low to medium dimensional tabular data.

Usage

Initialize

Parameters:

• threshold: threshold value for the outlier score above which the instance is flagged as an outlier.

• n_estimators: number of base estimators in the ensemble. Defaults to 100.

• max_samples: number of samples to draw from the training data to train each base estimator. If int, draw max_samples samples. If float, draw max_samples times number of features samples. If 'auto', max_samples = min(256, number of samples).

• max_features: number of features to draw from the training data to train each base estimator. If int, draw max_features features. If float, draw max_features times number of features features.

• bootstrap: whether to fit individual trees on random subsets of the training data, sampled with replacement.

• n_jobs: number of jobs to run in parallel for fit and predict.

• data_type: can specify data type added to metadata. E.g. 'tabular' or 'image'.

Initialized outlier detector example:

We then need to train the outlier detector. The following parameters can be specified:

• X: training batch as a numpy array.

• sample_weight: array with shape (batch size,) used to assign different weights to each instance during training. Defaults to None.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

We detect outliers by simply calling predict on a batch of instances X to compute the instance level outlier scores. We can also return the instance level outlier score by setting return_instance_score to True.

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances are above the threshold and therefore outlier instances. The array is of shape (batch size,).

• instance_score: contains instance level scores if return_instance_score equals True.

Mahalanobis Distance

Overview

The Mahalanobis online outlier detector aims to predict anomalies in tabular data. The algorithm calculates an outlier score, which is a measure of distance from the center of the features distribution (Mahalanobis distance). If this outlier score is higher than a user-defined threshold, the observation is flagged as an outlier. The algorithm is online, which means that it starts without knowledge about the distribution of the features and learns as requests arrive. Consequently you should expect the output to be bad at the start and to improve over time. The algorithm is suitable for low to medium dimensional tabular data.

The algorithm is also able to include categorical variables. The fit step first computes pairwise distances between the categories of each categorical variable. The pairwise distances are based on either the model predictions (MVDM method) or the context provided by the other variables in the dataset (ABDM method). For MVDM, we use the difference between the conditional model prediction probabilities of each category. This method is based on the Modified Value Difference Metric (MVDM) by Cost et al (1993). ABDM stands for Association-Based Distance Metric, a categorical distance measure introduced by Le et al (2005). ABDM infers context from the presence of other variables in the data and computes a dissimilarity measure based on the Kullback-Leibler divergence. Both methods can also be combined as ABDM-MVDM. We can then apply multidimensional scaling to project the pairwise distances into Euclidean space.

Usage

Initialize

Parameters:

• threshold: Mahalanobis distance threshold above which the instance is flagged as an outlier.

• n_components: number of principal components used.

• std_clip: feature-wise standard deviation used to clip the observations before updating the mean and covariance matrix.

Initialized outlier detector example:

We only need to fit the outlier detector if there are categorical variables present in the data. The following parameters can be specified:

• X: training batch as a numpy array.

• y: model class predictions or ground truth labels for X. Used for 'mvdm' and 'abdm-mvdm' pairwise distance metrics. Not needed for 'abdm'.

• d_type

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

Beware though that the outlier detector is stateful and every call to the score function will update the mean and covariance matrix, even when inferring the threshold.

We detect outliers by simply calling predict on a batch of instances X to compute the instance level Mahalanobis distances. We can also return the instance level outlier score by setting return_instance_score to True.

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances are above the threshold and therefore outlier instances. The array is of shape (batch size,).

• instance_score: contains instance level scores if return_instance_score equals True.

Variational Auto-Encoder

Overview

The Variational Auto-Encoder (VAE) outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised or semi-supervised training is desirable since labeled data is often scarce. The VAE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is either measured as the mean squared error (MSE) between the input and the reconstructed instance or as the probability that both the input and the reconstructed instance are generated by the same process. The algorithm is suitable for tabular and image data.

Usage

Initialize

Parameters:

• threshold: threshold value above which the instance is flagged as an outlier.

• score_type: scoring method used to detect outliers. Currently only the default 'mse' supported.

• latent_dim: latent dimension of the VAE.

• encoder_net: tf.keras.Sequential instance containing the encoder network. Example:

• decoder_net: tf.keras.Sequential instance containing the decoder network. Example:

• vae: instead of using a separate encoder and decoder, the VAE can also be passed as a tf.keras.Model.

• samples: number of samples drawn during detection for each instance to detect.

• beta: weight on the KL-divergence loss term following the $\beta$- framework. Default equals 1.

Initialized outlier detector example:

We then need to train the outlier detector. The following parameters can be specified:

• X: training batch as a numpy array of preferably normal data.

• loss_fn: loss function used for training. Defaults to the loss.

• optimizer: optimizer used for training. Defaults to with learning rate 1e-3.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

We detect outliers by simply calling predict on a batch of instances X. Detection can be customized via the following parameters:

• outlier_type: either 'instance' or 'feature'. If the outlier type equals 'instance', the outlier score at the instance level will be used to classify the instance as an outlier or not. If 'feature' is selected, outlier detection happens at the feature level (e.g. by pixel in images).

• outlier_perc: percentage of the sorted (descending) feature level outlier scores. We might for instance want to flag an image as an outlier if at least 20% of the pixel values are on average above the threshold. In this case, we set outlier_perc to 20. The default value is 100 (using all the features).

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances or features are above the threshold and therefore outliers. If outlier_type equals 'instance', then the array is of shape (batch size,). If it equals 'feature', then the array is of shape (batch size, instance shape).

• feature_score: contains feature level scores if return_feature_score equals True.

Likelihood Ratios for Outlier Detection

Overview

The outlier detector described by Ren et al. (2019) in Likelihood Ratios for Out-of-Distribution Detection uses the likelihood ratio (LLR) between 2 generative models as the outlier score. One model is trained on the original data while the other is trained on a perturbed version of the dataset. This is based on the observation that the log likelihood for an instance under a generative model can be heavily affected by population level background statistics. The second generative model is therefore trained to capture the background statistics still present in the perturbed data while the semantic features have been erased by the perturbations.

The perturbations are added using an independent and identical Bernoulli distribution with rate $\mu$ which substitutes a feature with one of the other possible feature values with equal probability. For images, this means for instance changing a pixel with a different pixel value randomly sampled within the $0$ to $255$ pixel range. The package also contains a PixelCNN++ implementation adapted from the official TensorFlow Probability version, and available as a standalone model in alibi_detect.models.tensorflow.pixelcnn.

Usage

Initialize

Parameters:

• threshold: outlier threshold value used for the negative likelihood ratio. Scores above the threshold are flagged as outliers.

• model: a generative model, either as a tf.keras.Model, TensorFlow Probability distribution or built-in PixelCNN++ model.

• model_background: optional separate model fit on the perturbed background data. If this is not specified, a copy of model will be used.

• log_prob: if the model does not have a log_prob function like e.g. a TensorFlow Probability distribution, a function needs to be passed that evaluates the log likelihood.

• sequential: flag whether the data is sequential or not. Used to create targets during training. Defaults to False.

• data_type: can specify data type added to metadata. E.g. 'tabular' or 'image'.

Initialized outlier detector example:

We then need to train the 2 generative models in sequence. The following parameters can be specified:

• X: training batch as a numpy array of preferably normal data.

• mutate_fn: function used to create the perturbations. Defaults to an independent and identical Bernoulli distribution with rate $\mu$

• mutate_fn_kwargs: kwargs for mutate_fn. For the default function, the mutation rate and feature range needs to be specified, e.g.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

We detect outliers by simply calling predict on a batch of instances X. Detection can be customized via the following parameters:

• outlier_type: either 'instance' or 'feature'. If the outlier type equals 'instance', the outlier score at the instance level will be used to classify the instance as an outlier or not. If 'feature' is selected, outlier detection happens at the feature level (e.g. by pixel in images).

• batch_size: batch size used for model prediction calls.

• return_feature_score

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances or features are above the threshold and therefore outliers. If outlier_type equals 'instance', then the array is of shape (batch size,). If it equals 'feature', then the array is of shape (batch size, instance shape).

• feature_score: contains feature level scores if return_feature_score equals True.

Sequence-to-Sequence (Seq2Seq)

Overview

The Sequence-to-Sequence (Seq2Seq) outlier detector consists of 2 main building blocks: an encoder and a decoder. The encoder consists of a Bidirectional LSTM which processes the input sequence and initializes the decoder. The LSTM decoder then makes sequential predictions for the output sequence. In our case, the decoder aims to reconstruct the input sequence. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.

Since even for normal data the reconstruction error can be state-dependent, we add an outlier threshold estimator network to the Seq2Seq model. This network takes in the hidden state of the decoder at each timestep and predicts the estimated reconstruction error for normal data. As a result, the outlier threshold is not static and becomes a function of the model state. This is similar to Park et al. (2017), but while they train the threshold estimator separately from the Seq2Seq model with a Support-Vector Regressor, we train a neural net regression network end-to-end with the Seq2Seq model.

The detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The Seq2Seq outlier detector is suitable for both univariate and multivariate time series.

Usage

Initialize

Parameters:

• n_features: number of features in the time series.

• seq_len: sequence length fed into the Seq2Seq model.

• threshold: threshold used for outlier detection. Can be a float or feature-wise array.

• latent_dim: latent dimension of the encoder and decoder.

• output_activation: activation used in the Dense output layer of the decoder.

• beta: weight on the threshold estimation mean-squared error (MSE) loss term.

Initialized outlier detector example:

We then need to train the outlier detector. The following parameters can be specified:

• X: univariate or multivariate time series array with preferably normal data used for training. Shape equals (batch, n_features) or (batch, seq_len, n_features).

• loss_fn: loss function used for training. Defaults to the MSE loss.

• optimizer: optimizer used for training. Defaults to with learning rate 1e-3.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold. We can either set the threshold over both features combined or determine a feature-wise threshold. Here we opt for the feature-wise threshold. This is for instance useful when different features have different variance or sensitivity to outliers. The snippet assumes there are about 5% outliers in the first feature and 10% in the second:

We detect outliers by simply calling predict on a batch of instances X. Detection can be customized via the following parameters:

• outlier_type: either 'instance' or 'feature'. If the outlier type equals 'instance', the outlier score at the instance level will be used to classify the instance as an outlier or not. If 'feature' is selected, outlier detection happens at the feature level. It is important to distinguish 2 use cases:

  • X has shape (batch, n_features):

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances or features are above the threshold and therefore outliers. If outlier_type equals 'instance', then the array is of shape (batch,). If it equals 'feature', then the array is of shape (batch, seq_len, n_features) or (batch, n_features), depending on the shape of X.

• feature_score: contains feature level scores if return_feature_score equals True.

Spectral Residual

Overview

The Spectral Residual outlier detector is based on the paper Time-Series Anomaly Detection Service at Microsoft and is suitable for unsupervised online anomaly detection in univariate time series data. The algorithm first computes the Fourier Transform of the original data. Then it computes the spectral residual of the log amplitude of the transformed signal before applying the Inverse Fourier Transform to map the sequence back from the frequency to the time domain. This sequence is called the saliency map. The anomaly score is then computed as the relative difference between the saliency map values and their moving averages. If the score is above a threshold, the value at a specific timestep is flagged as an outlier. For more details, please check out the paper.

Usage

Initialize

Parameters:

• threshold: Threshold used to classify outliers. Relative saliency map distance from the moving average.

• window_amp: Window used for the moving average in the spectral residual computation. The spectral residual is the difference between the log amplitude of the Fourier Transform and a convolution of the log amplitude over window_amp.

• window_local: Window used for the moving average in the outlier score computation. The outlier score computes the relative difference between the saliency map and a moving average of the saliency map over window_local timesteps.

• padding_amp_method: Padding method to be used prior to each convolution over log amplitude. Possible values: constant | replicate | reflect. Default value: replicate.

  • constant - padding with constant 0.
• padding_local_method: Padding method to be used prior to each convolution over saliency map. Possible values: constant | replicate | reflect. Default value: replicate.

  • constant - padding with constant 0.
• padding_amp_side: Whether to pad the amplitudes on both sides or only on one side. Possible values: bilateral | left | right.

• n_est_points: Number of estimated points padded to the end of the sequence.

• n_grad_points: Number of points used for the gradient estimation of the additional points padded to the end of the sequence. The paper sets this value to 5.

Initialized outlier detector example:

It is often hard to find a good threshold value. If we have a time series containing both normal and outlier data and we know approximately the percentage of normal data in the time series, we can infer a suitable threshold:

We detect outliers by simply calling predict on a time series X to compute the outlier scores and flag the anomalies. We can also return the instance (timestep) level outlier score by setting return_instance_score to True.

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances are above the threshold and therefore outlier instances. The array is of shape (timesteps,).

• instance_score: contains instance level scores if return_instance_score equals True.

(IF) are tree based models specifically used for outlier detection. The IF isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. The number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node. This path length, averaged over a forest of random trees, is a measure of normality and is used to define an anomaly score. Outliers can typically be isolated quicker, leading to shorter paths.

The outlier detector needs to detect computer network intrusions using TCP dump data for a local-area network (LAN) simulating a typical U.S. Air Force LAN. A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack.

There are 4 types of attacks in the dataset:

• DOS: denial-of-service, e.g. syn flood;

The Auto-Encoding Gaussian Mixture Model (AEGMM) Outlier Detector follows the paper. The encoder compresses the data while the reconstructed instances generated by the decoder are used to create additional features based on the reconstruction error between the input and the reconstructions. These features are combined with encodings and fed into a Gaussian Mixture Model (). The AEGMM outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised or semi-supervised training is desirable since labeled data is often scarce. The sample energy of the GMM can then be used to determine whether an instance is an outlier (high sample energy) or not (low sample energy). The algorithm is suitable for tabular and image data.

Parameters:

• threshold: threshold value for the sample energy above which the instance is flagged as an outlier.

The Prophet outlier detector uses the time series forecasting package explained in . The underlying Prophet model is a decomposable univariate time series model combining trend, seasonality and holiday effects. The model forecast also includes an uncertainty interval around the estimated trend component using the of the extrapolated model. Alternatively, full Bayesian inference can be done at the expense of increased compute. The upper and lower values of the uncertainty interval can then be used as outlier thresholds for each point in time. First, the distance from the observed value to the nearest uncertainty boundary (upper or lower) is computed. If the observation is within the boundaries, the outlier score equals the negative distance. As a result, the outlier score is the lowest when the observation equals the model prediction. If the observation is outside of the boundaries, the score equals the distance measure and the observation is flagged as an outlier. One of the main drawbacks of the method however is that you need to refit the model as new data comes in. This is undesirable for applications with high throughput and real-time detection.

Note

To use this detector, first install Prophet by running:

This will install Prophet, and its major dependency PyStan. PyStan is currently only . If this detector is to be used on a Windows system, it is recommended to manually install (and test) PyStan before running the command above.

The example uses a weather time series dataset recorded by the . The dataset contains 14 different features such as air temperature, atmospheric pressure, and humidity. These were collected every 10 minutes, beginning in 2003. Like the

Method

The Spectral Residual outlier detector is based on the paper Time-Series Anomaly Detection Service at Microsoft and is suitable for unsupervised online anomaly detection in univariate time series data. The algorithm first computes the Fourier Transform of the original data. Then it computes the spectral residual of the log amplitude of the transformed signal before applying the Inverse Fourier Transform to map the sequence back from the frequency to the time domain. This sequence is called the saliency map. The anomaly score is then computed as the relative difference between the saliency map values and their moving averages. If this score is above a threshold, the value at a specific timestep is flagged as an outlier. For more details, please check out the paper.

Dataset

We test the outlier detector on a synthetic dataset generated with the TimeSynth package. It allows you to generate a wide range of time series (e.g. pseudo-periodic, autoregressive or Gaussian Process generated signals) and noise types (white or red noise). It can be installed as follows:

!pip install git+https://github.com/TimeSynth/TimeSynth.git

Additionally, this notebook requires the seaborn package for visualization which can be installed via pip:

!pip install seaborn

import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, recall_score
import timesynth as ts

from alibi_detect.od import SpectralResidual
from alibi_detect.utils.perturbation import inject_outlier_ts
from alibi_detect.saving import save_detector, load_detector
from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_ts

Create univariate time series

Define number of sampled points and the type of simulated time series. We use TimeSynth to generate a sinusoidal signal with Gaussian noise.

We can inject noise in the time series via inject_outlier_ts. The noise can be regulated via the percentage of outliers (perc_outlier), the strength of the perturbation (n_std) and the minimum size of the noise perturbation (min_std):

Visualize part of the original and perturbed time series:

Perturbed data:

Note that for the local convolution we pad the signal internally only on the left, following the paper's recommendation.

The warning tells us that we need to set the outlier threshold. This can be done with the infer_threshold method. We need to pass a batch of instances and specify what percentage of those we consider to be normal via threshold_perc. Let's assume we have some data which we know contains around 10% outliers:

Let's infer the threshold:

Let's save the outlier detector with the updated threshold:

We can load the same detector via load_detector:

Predict outliers:

F1 score, accuracy, recall and confusion matrix:

Plot the outlier scores of the time series vs. the outlier threshold. :

Let's zoom in on a smaller time scale to have a clear picture:

Variational Auto-Encoding Gaussian Mixture Model

Overview

The Variational Auto-Encoding Gaussian Mixture Model (VAEGMM) Outlier Detector follows the Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection paper but with a VAE instead of a regular Auto-Encoder. The encoder compresses the data while the reconstructed instances generated by the decoder are used to create additional features based on the reconstruction error between the input and the reconstructions. These features are combined with encodings and fed into a Gaussian Mixture Model (GMM). The VAEGMM outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised or semi-supervised training is desirable since labeled data is often scarce. The sample energy of the GMM can then be used to determine whether an instance is an outlier (high sample energy) or not (low sample energy). The algorithm is suitable for tabular and image data.

Usage

Initialize

Parameters:

• threshold: threshold value for the sample energy above which the instance is flagged as an outlier.

• latent_dim: latent dimension of the VAE.

• n_gmm: number of components in the GMM.

• encoder_net: tf.keras.Sequential instance containing the encoder network. Example:

• decoder_net: tf.keras.Sequential instance containing the decoder network. Example:

• gmm_density_net: layers for the GMM network wrapped in a tf.keras.Sequential class. Example:

• vaegmm: instead of using a separate encoder, decoder and GMM density net, the VAEGMM can also be passed as a tf.keras.Model.

• samples: number of samples drawn during detection for each instance to detect.

• beta: weight on the KL-divergence loss term following the $\beta$- framework. Default equals 1.

Initialized outlier detector example:

We then need to train the outlier detector. The following parameters can be specified:

• X: training batch as a numpy array of preferably normal data.

• loss_fn: loss function used for training. Defaults to the custom VAEGMM loss which is a combination of the loss, sample energy of the GMM and a loss term penalizing small values on the diagonals of the covariance matrices in the GMM to avoid trivial solutions. It is important to balance the loss weights below so no single loss term dominates during the optimization.

• w_recon: weight on elbo loss term. Defaults to 1e-7.

It is often hard to find a good threshold value. If we have a batch of normal and outlier data and we know approximately the percentage of normal data in the batch, we can infer a suitable threshold:

We detect outliers by simply calling predict on a batch of instances X to compute the instance level sample energies. We can also return the instance level outlier score by setting return_instance_score to True.

The prediction takes the form of a dictionary with meta and data keys. meta contains the detector's metadata while data is also a dictionary which contains the actual predictions stored in the following keys:

• is_outlier: boolean whether instances are above the threshold and therefore outlier instances. The array is of shape (batch size,).

• instance_score: contains instance level scores if return_instance_score equals True.

The Variational Auto-Encoder () outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The VAE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is either measured as the mean squared error (MSE) between the input and the reconstructed instance or as the probability that both the input and the reconstructed instance are generated by the same process.

The outlier detector needs to detect computer network intrusions using TCP dump data for a local-area network (LAN) simulating a typical U.S. Air Force LAN. A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack.

There are 4 types of attacks in the dataset:

• DOS: denial-of-service, e.g. syn flood;

The Prophet outlier detector uses the time series forecasting package explained in . The underlying Prophet model is a decomposable univariate time series model combining trend, seasonality and holiday effects. The model forecast also includes an uncertainty interval around the estimated trend component using the of the extrapolated model. Alternatively, full Bayesian inference can be done at the expense of increased compute. The upper and lower values of the uncertainty interval can then be used as outlier thresholds for each point in time. First, the distance from the observed value to the nearest uncertainty boundary (upper or lower) is computed. If the observation is within the boundaries, the outlier score equals the negative distance. As a result, the outlier score is the lowest when the observation equals the model prediction. If the observation is outside of the boundaries, the score equals the distance measure and the observation is flagged as an outlier. One of the main drawbacks of the method however is that you need to refit the model as new data comes in. This is undesirable for applications with high throughput and real-time detection.

Note

To use this detector, first install Prophet by running:

This will install Prophet, and its major dependency PyStan. PyStan is currently only . If this detector is to be used on a Windows system, it is recommended to manually install (and test) PyStan before running the command above.

Method

The outlier detector described by Ren et al. (2019) in Likelihood Ratios for Out-of-Distribution Detection uses the likelihood ratio between 2 generative models as the outlier score. One model is trained on the original data while the other is trained on a perturbed version of the dataset. This is based on the observation that the likelihood score for an instance under a generative model can be heavily affected by population level background statistics. The second generative model is therefore trained to capture the background statistics still present in the perturbed data while the semantic features have been erased by the perturbations.

The perturbations are added using an independent and identical Bernoulli distribution with rate $\mu$ which substitutes a feature with one of the other possible feature values with equal probability. Each feature in the genome dataset can take 4 values (one of the ACGT nucleobases). This means that a perturbed feature is swapped with one of the other nucleobases. The generative model used in the example is a simple LSTM network.

Dataset

The bacteria genomics dataset for out-of-distribution detection was released as part of the Likelihood Ratios for Out-of-Distribution Detection paper. From the original TL;DR: The dataset contains genomic sequences of 250 base pairs from 10 in-distribution bacteria classes for training, 60 OOD bacteria classes for validation, and another 60 different OOD bacteria classes for test. There are respectively 1, 7 and again 7 million sequences in the training, validation and test sets. For detailed info on the dataset check the README.

This notebook requires the seaborn package for visualization which can be installed via pip:

X represents the genome sequences and y whether they are outliers ($1$) or not ($0$).

There are no outliers in the training set and a majority of outliers (compared to the training data) in the validation and test sets:

We need to define a generative model which models the genome sequences. We follow the paper and opt for a simple LSTM. Note that we don't actually need to define the model below if we simply load the pretrained detector later on:

We also need to define our loss function which we can utilize to evaluate the log-likelihood for the outlier detector:

We can again either fetch the pretrained detector from a or train one from scratch:

Let's compare the log likelihoods of the inliers vs. the outlier test set data under the semantic and background models. We randomly sample $100,000$ instances from both distributions since the full test set contains $7,000,000$ genomic sequences. The histograms show that the generative model does not distinguish well between inliers and outliers.

This is because of the background-effect which is in this case the GC-content in the genomic sequences. This effect is partially reduced when taking the likelihood ratio:

We follow the same procedure with the outlier detector. First we need to set an outlier threshold with infer_threshold. We need to pass a batch of instances and specify what percentage of those we consider to be normal via threshold_perc. Let's assume we have a small batch of data with roughly $30$% outliers but we don't know exactly which ones.

Let's save the outlier detector with updated threshold:

Let'spredict outliers on a sample of the test set:

F1 score, accuracy, precision, recall and confusion matrix:

We can also plot the ROC curve based on the instance level outlier scores:

Method

The Variational Auto-Encoder (VAE) outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The VAE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.

Dataset

The instances contain a person's characteristics like age, marital status or education while the label represents whether the person makes more or less than $50k per year. The dataset consists of a mixture of numerical and categorical features. It is originally not an outlier detection dataset so we will inject artificial outliers. It is fetched using the Alibi library, which can be installed with pip. We also use seaborn to visualize the data:

!pip install alibi seaborn

import os
import alibi
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, precision_score, recall_score
from sklearn.preprocessing import OneHotEncoder
import tensorflow as tf
tf.keras.backend.clear_session()
from tensorflow.keras.layers import Dense, InputLayer

from alibi_detect.od import OutlierVAE
from alibi_detect.utils.perturbation import inject_outlier_tabular
from alibi_detect.utils.fetching import fetch_detector
from alibi_detect.saving import save_detector, load_detector
from alibi_detect.utils.visualize import plot_instance_score

def set_seed(s=0):
    np.random.seed(s)
    tf.random.set_seed(s)

Load adult dataset

The fetch_adult function returns a Bunch object containing the features, the targets, the feature names and a mapping of the categories in each categorical variable.

adult = alibi.datasets.fetch_adult()
X, y = adult.data, adult.target
feature_names = adult.feature_names
category_map_tmp = adult.category_map

Shuffle data:

set_seed(0)
Xy_perm = np.random.permutation(np.c_[X, y])
X, y = Xy_perm[:,:-1], Xy_perm[:,-1]

Reorganize data so categorical features come first, remove some features and adjust feature_names and category_map accordingly:

Normalize numerical features or scale numerical between -1 and 1:

Fit OHE to categorical variables:

Combine numerical and categorical data:

Define train, validation (to find outlier threshold) and test set:

Inject outliers in the numerical features. First we need to know the features for each kind:

Now we can add outliers to the validation (or threshold) and test sets. For the numerical data, we need to specify the numerical columns (cols), the percentage of outliers (perc_outlier), the strength (n_std) and the minimum size of the perturbation (min_std). The outliers are distributed evenly across the numerical features:

Let's inspect an instance that was changed:

Same thing for the test set:

OHE to train, threshold and outlier sets:

The pretrained outlier and adversarial detectors used in the example notebooks can be found . You can use the built-in fetch_detector function which saves the pre-trained models in a local directory filepath and loads the detector. Alternatively, you can train a detector from scratch:

The warning tells us we still need to set the outlier threshold. This can be done with the infer_threshold method. We need to pass a batch of instances and specify what percentage of those we consider to be normal via threshold_perc.

Let’s save the outlier detector with updated threshold:

F1 score and confusion matrix:

Plot instance level outlier scores vs. the outlier threshold:

Method

The Sequence-to-Sequence (Seq2Seq) outlier detector consists of 2 main building blocks: an encoder and a decoder. The encoder consists of a Bidirectional LSTM which processes the input sequence and initializes the decoder. The LSTM decoder then makes sequential predictions for the output sequence. In our case, the decoder aims to reconstruct the input sequence. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.

Since even for normal data the reconstruction error can be state-dependent, we add an outlier threshold estimator network to the Seq2Seq model. This network takes in the hidden state of the decoder at each timestep and predicts the estimated reconstruction error for normal data. As a result, the outlier threshold is not static and becomes a function of the model state. This is similar to Park et al. (2017), but while they train the threshold estimator separately from the Seq2Seq model with a Support-Vector Regressor, we train a neural net regression network end-to-end with the Seq2Seq model.

The detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The Seq2Seq outlier detector is suitable for both univariate and multivariate time series.

Dataset

We test the outlier detector on a synthetic dataset generated with the TimeSynth package. It allows you to generate a wide range of time series (e.g. pseudo-periodic, autoregressive or Gaussian Process generated signals) and noise types (white or red noise). It can be installed as follows:

Additionally, this notebook requires the seaborn package for visualization which can be installed via pip:

Define number of sampled points and the type of simulated time series. We use to generate sinusoidal signals with noise.

Visualize:

We still need to set the outlier threshold. This can be done with the infer_threshold method. We need to pass a time series of instances and specify what percentage of those we consider to be normal via threshold_perc. First we create outliers by injecting noise in the time series via inject_outlier_ts. The noise can be regulated via the percentage of outliers (perc_outlier), the strength of the perturbation (n_std) and the minimum size of the noise perturbation (min_std). Let's assume we have some data which we know contains around 10% outliers in either of the features:

Visualize outlier data used to determine the threshold:

Let's infer the threshold. The inject_outlier_ts method distributes perturbations evenly across features. As a result, each feature contains about 5% outliers. We can either set the threshold over both features combined or determine a feature-wise threshold. Here we opt for the feature-wise threshold. This is for instance useful when different features have different variance or sensitivity to outliers. We also manually decrease the threshold a bit to increase the sensitivity of our detector:

Let's save the outlier detector with the updated threshold:

We can load the same detector via load_detector:

Generate the outliers to detect:

Predict outliers:

F1 score, accuracy, recall and confusion matrix:

Plot the feature-wise outlier scores of the time series for each timestep vs. the outlier threshold:

We can also plot the ROC curve using the instance level outlier scores:

Method

The Sequence-to-Sequence (Seq2Seq) outlier detector consists of 2 main building blocks: an encoder and a decoder. The encoder consists of a Bidirectional LSTM which processes the input sequence and initializes the decoder. The LSTM decoder then makes sequential predictions for the output sequence. In our case, the decoder aims to reconstruct the input sequence. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is measured as the mean squared error (MSE) between the input and the reconstructed instance.

Since even for normal data the reconstruction error can be state-dependent, we add an outlier threshold estimator network to the Seq2Seq model. This network takes in the hidden state of the decoder at each timestep and predicts the estimated reconstruction error for normal data. As a result, the outlier threshold is not static and becomes a function of the model state. This is similar to Park et al. (2017), but while they train the threshold estimator separately from the Seq2Seq model with a Support-Vector Regressor, we train a neural net regression network end-to-end with the Seq2Seq model.

The detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The Seq2Seq outlier detector is suitable for both univariate and multivariate time series.

Dataset

The outlier detector needs to spot anomalies in electrocardiograms (ECG's). The dataset contains 5000 ECG's, originally obtained from Physionet under the name BIDMC Congestive Heart Failure Database(chfdb), record chf07. The data has been pre-processed in 2 steps: first each heartbeat is extracted, and then each beat is made equal length via interpolation. The data is labeled and contains 5 classes. The first class which contains almost 60% of the observations is seen as normal while the others are outliers. The detector is trained on heartbeats from the first class and needs to flag the other classes as anomalies.

This notebook requires the seaborn package for visualization which can be installed via pip:

Flip train and test data because there are only 500 ECG's in the original training set and 4500 in the test set:

Since we treat the first class as the normal, inlier data and the rest of X_train as outliers, we need to adjust the training (inlier) data and the labels of the test set.

Some of the outliers in X_train are used in combination with some of the inlier instances to infer the threshold level:

Apply min-max scaling between 0 and 1 to the observations using the inlier data:

Reshape the observations to (batch size, sequence length, features) for the detector:

We can now visualize scaled instances from each class:

The pretrained outlier and adversarial detectors used in the example notebooks can be found . You can use the built-in fetch_detector function which saves the pre-trained models in a local directory filepath and loads the detector. Alternatively, you can train a detector from scratch:

Let's inspect how well the sequence-to-sequence model can predict the ECG's of the inlier and outlier classes. The predictions in the charts below are made on ECG's from the test set:

It is clear that the model can reconstruct the inlier class but struggles with the outliers.

If we trained a model from scratch, the warning thrown when we initialized the model tells us that we need to set the outlier threshold. This can be done with the infer_threshold method. We need to pass a time series of instances and specify what percentage of those we consider to be normal via threshold_perc, equal to the percentage of Class 1 in X_threshold. The outlier_perc parameter defines the percentage of features used to define the outlier threshold. In this example, the number of features considered per instance equals 140 (1 for each timestep). We set the outlier_perc at 95, which means that we will use the 95% features with highest reconstruction error, adjusted for by the threshold estimate.

Let's save the outlier detector with the updated threshold:

We can load the same detector via load_detector:

F1 score, accuracy, recall and confusion matrix:

We can also plot the ROC curve based on the instance level outlier scores:

Method

The Variational Auto-Encoder (VAE) outlier detector is first trained on a batch of unlabeled, but normal (inlier) data. Unsupervised training is desireable since labeled data is often scarce. The VAE detector tries to reconstruct the input it receives. If the input data cannot be reconstructed well, the reconstruction error is high and the data can be flagged as an outlier. The reconstruction error is either measured as the mean squared error (MSE) between the input and the reconstructed instance or as the probability that both the input and the reconstructed instance are generated by the same process.

Dataset

CIFAR10 consists of 60,000 32 by 32 RGB images equally distributed over 10 classes.

import os
import logging
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
tf.keras.backend.clear_session()
from tensorflow.keras.layers import Conv2D, Conv2DTranspose, Dense, Layer, Reshape, InputLayer
from tqdm import tqdm

from alibi_detect.models.tensorflow import elbo
from alibi_detect.od import OutlierVAE
from alibi_detect.utils.fetching import fetch_detector
from alibi_detect.utils.perturbation import apply_mask
from alibi_detect.saving import save_detector, load_detector
from alibi_detect.utils.visualize import plot_instance_score, plot_feature_outlier_image

logger = tf.get_logger()
logger.setLevel(logging.ERROR)

Load CIFAR10 data

train, test = tf.keras.datasets.cifar10.load_data()
X_train, y_train = train
X_test, y_test = test

X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255
print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

Load or define outlier detector

The pretrained outlier and adversarial detectors used in the example notebooks can be found here. You can use the built-in fetch_detector function which saves the pre-trained models in a local directory filepath and loads the detector. Alternatively, you can train a detector from scratch:

load_outlier_detector = True

filepath = 'my_path'  # change to directory where model is downloaded
detector_type = 'outlier'
dataset = 'cifar10'
detector_name = 'OutlierVAE'
filepath = os.path.join(filepath, detector_name)
if load_outlier_detector:  # load pretrained outlier detector
    od = fetch_detector(filepath, detector_type, dataset, detector_name)
else:  # define model, initialize, train and save outlier detector
    latent_dim = 1024
    
    encoder_net = tf.keras.Sequential(
      [
          InputLayer(input_shape=(32, 32, 3)),
          Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),
          Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),
          Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu)
      ])

    decoder_net = tf.keras.Sequential(
      [
          InputLayer(input_shape=(latent_dim,)),
          Dense(4*4*128),
          Reshape(target_shape=(4, 4, 128)),
          Conv2DTranspose(256, 4, strides=2, padding='same', activation=tf.nn.relu),
          Conv2DTranspose(64, 4, strides=2, padding='same', activation=tf.nn.relu),
          Conv2DTranspose(3, 4, strides=2, padding='same', activation='sigmoid')
      ])
    
    # initialize outlier detector
    od = OutlierVAE(threshold=.015,  # threshold for outlier score
                    score_type='mse',  # use MSE of reconstruction error for outlier detection
                    encoder_net=encoder_net,  # can also pass VAE model instead
                    decoder_net=decoder_net,  # of separate encoder and decoder
                    latent_dim=latent_dim,
                    samples=2)
    # train
    od.fit(X_train, 
           loss_fn=elbo,
           cov_elbo=dict(sim=.05),
           epochs=50,
           verbose=False)
    
    # save the trained outlier detector
    save_detector(od, filepath)