> For the complete documentation index, see [llms.txt](https://docs.seldon.ai/alibi-detect/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seldon.ai/alibi-detect/api-reference/utils/prediction.md).

# alibi\_detect.utils.prediction

## Functions

### `tokenize_transformer`

```python
tokenize_transformer(x: Union[list, numpy.ndarray], tokenizer: Callable, max_len: int, backend: str) -> dict
```

Batch tokenizer for transformer models.

| Name        | Type                         | Default | Description                                  |
| ----------- | ---------------------------- | ------- | -------------------------------------------- |
| `x`         | `Union[list, numpy.ndarray]` |         | Batch of instances.                          |
| `tokenizer` | `Callable`                   |         | Tokenizer for model.                         |
| `max_len`   | `int`                        |         | Max token length.                            |
| `backend`   | `str`                        |         | PyTorch ('pt') or TensorFlow ('tf') backend. |

**Returns**

* Type: `dict`