HuggingFace
Last updated
Was this helpful?
Last updated
Was this helpful?
This package provides a MLServer runtime compatible with HuggingFace Transformers.
You can install the runtime, alongside mlserver
, as:
For further information on how to use MLServer with HuggingFace, you can check out this .
The HuggingFace runtime will always decode the input request using its own built-in codec. Therefore, at the request level will be ignored. Note that this doesn't include annotations, which will be respected as usual.
The HuggingFace runtime exposes a couple extra parameters which can be used to customise how the runtime behaves. These settings can be added under the parameters.extra
section of your model-settings.json
file, e.g.
It is possible to load a local model into a HuggingFace pipeline by specifying the model artefact folder path in parameters.uri
in model-settings.json
.
Models in the HuggingFace hub can be loaded by specifying their name in parameters.extra.pretrained_model
in model-settings.json
.
You can find the full reference of the accepted extra settings for the HuggingFace runtime below: