OpenAI
The following demonstrates how to locally run an API Runtime instance to run inference with OpenAI models. It also illustrates the different ways it can be used.
To get up and running we need to pull the runtime Docker image. To pull the docker image, you must be authenticated. Check our installation tutorial to see how you can authenticate with Docker CLI.
docker pull \
europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-api:0.7.0Before we can start the runtime we need to create the model-settings.json which will tell it what model to run:
!cat models/openai-chat-completions/model-settings.json{
"name": "openai-chat-completions",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "openai",
"config": {
"model_id": "gpt-3.5-turbo",
"model_type": "chat.completions"
}
}
}
}In the above settings, the runtime config is specified in the parameters JSON field:
the choice of
provider_id- currently, the providers supported are"openai"and"gemini".we've chosen the
gpt-3.5-turbomodel and thechat.completionsAPI.
Starting the Runtime
Finally, to start the server run:
docker run -it --rm -p 8080:8080 \
-v ${PWD}/models:/models \
-e MLSERVER_MODEL_OPENAI_API_KEY=<your_openai_key> \
europe-west2-docker.pkg.dev/seldon-registry/llm/mlserver-llm-api:0.7.0 \
mlserver start /modelsSending Requests
To send our first request to the chat.completions endpoint that we are now serving via mlserver, we use the following:
import json
import pprint
import requests
inference_request = {
"inputs": [
{
"name": "role",
"shape": [2],
"datatype": "BYTES",
"data": ["system", "user"]
},
{
"name": "content",
"shape": [2],
"datatype": "BYTES",
"data": [
json.dumps(["You are a helpful assistant"]),
json.dumps(["Hello from MLServer!"]),
],
},
{
"name": "type",
"shape": [2],
"datatype": "BYTES",
"data": [
json.dumps(["text"]),
json.dumps(["text"]),
]
}
]
}
endpoint = "http://localhost:8080/v2/models/openai-chat-completions/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=4){'id': 'aa65a0ac-6aea-4284-9e55-7c74e299390e',
'model_name': 'openai-chat-completions',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['Hello! How can I assist you today?'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{\n'
' "id": "chatcmpl-AYD7ygNL8rqda0t5mG691k1aXpGLC",\n'
' "choices": [\n'
' {\n'
' "finish_reason": "stop",\n'
' "index": 0,\n'
' "logprobs": null,\n'
' "message": {\n'
' "content": "Hello! How can I assist you '
'today?",\n'
' "refusal": null,\n'
' "role": "assistant"\n'
' }\n'
' }\n'
' ],\n'
' "created": 1732716978,\n'
' "model": "gpt-3.5-turbo-0125",\n'
' "object": "chat.completion",\n'
' "system_fingerprint": null,\n'
' "usage": {\n'
' "completion_tokens": 9,\n'
' "prompt_tokens": 21,\n'
' "total_tokens": 30,\n'
' "completion_tokens_details": {\n'
' "reasoning_tokens": 0,\n'
' "audio_tokens": 0,\n'
' "accepted_prediction_tokens": 0,\n'
' "rejected_prediction_tokens": 0\n'
' },\n'
' "prompt_tokens_details": {\n'
' "cached_tokens": 0,\n'
' "audio_tokens": 0\n'
' }\n'
' }\n'
'}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}Note that we've sent three tensors: a "role", a "content", and a "type" tensor. The "role" tensors tell the model who is speaking. In this case, it includes a "system" role and a "user" role. The "system" role is used to dictate the context of the interaction and the "user" role indicates that the matching content is sent by a user. In the above the system content is: "You are a helpful assistant" and the user content is "Hello from MLServer". The "type" tensor indicates that the content we sent is a text.
The endpoint responds with its own "role", "content" and "type" tensors. Its "role" is given as "assistant" and the "content" it returns is "Hello! How can I assist you today?". As well as this the server returns the full response received from OpenAI via the "output_all" tensor.
Requests with Parameters
As well as this we can add parameters to the request, to specify the number of generations and temperature, etc. For a list of all available parameters see the OpenAI documentation for their API. The following sets the temperature, the maximum number of tokens to generate and the number of generations to return:
import pprint
import requests
inference_request = {
"inputs": [
{
"name": "role",
"shape": [1],
"datatype": "BYTES",
"data": ["user"]
},
{
"name": "content",
"shape": [1],
"datatype": "BYTES",
"data": ["Hello from MLServer!"],
},
{
"name": "type",
"shape": [1],
"datatype": "BYTES",
"data": ["text"]
}
],
"parameters": {
"llm_parameters": {
"temperature": 0.5,
"max_tokens": 100,
}
},
}
endpoint = "http://localhost:8080/v2/models/openai-chat-completions/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=4){'id': '6993f40a-46f3-4aca-ab0e-38b1d64b3c7b',
'model_name': 'openai-chat-completions',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['Hello! How can I assist you today?'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{\n'
' "id": "chatcmpl-AYD86FsU7OSnNFhPUk8SBL7GtJIRS",\n'
' "choices": [\n'
' {\n'
' "finish_reason": "stop",\n'
' "index": 0,\n'
' "logprobs": null,\n'
' "message": {\n'
' "content": "Hello! How can I assist you '
'today?",\n'
' "refusal": null,\n'
' "role": "assistant"\n'
' }\n'
' }\n'
' ],\n'
' "created": 1732716986,\n'
' "model": "gpt-3.5-turbo-0125",\n'
' "object": "chat.completion",\n'
' "system_fingerprint": null,\n'
' "usage": {\n'
' "completion_tokens": 9,\n'
' "prompt_tokens": 12,\n'
' "total_tokens": 21,\n'
' "completion_tokens_details": {\n'
' "reasoning_tokens": 0,\n'
' "audio_tokens": 0,\n'
' "accepted_prediction_tokens": 0,\n'
' "rejected_prediction_tokens": 0\n'
' },\n'
' "prompt_tokens_details": {\n'
' "cached_tokens": 0,\n'
' "audio_tokens": 0\n'
' }\n'
' }\n'
'}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}Note that if you are sending a single message, you must not encode the content and type as json.
Adding prompts
Prompting is a pivotal part of using large language models. It allows developers to embed messages sent by a user within other textual contexts that provide more information about the task you want the model to perform. For instance, we use the following prompt to give the model more context about the kind of question that the model is expected to answer:
!cat models/openai-prompt-chat-completions/math.jinjaTranslate a math problem into Python code that can be executed in Python 3 REPL. Use the output of running this code to answer the question.
Question: "Question with math problem."
```python
"Code that solves the problem and prints the solution"
```
```output
"Output of running the code"
```
Answer: "Answer"
Begin.
Question: What is 37593 * 67?
```python
print(37593 * 67)
```
```output
2518731
```
Answer: 2518731
Question: {{ question[0] }}In the above the content sent by the user can be inserted in the {question} variable. A developer should specify what content to insert thereby giving that tensor the name "question". To start with we need to create a new model-settings.json file. This will be the same as the previous one but in addition it specifies the prompt tempalte to be used through prompt_utils settings.
!cat models/openai-prompt-chat-completions/model-settings.json{
"name": "openai-prompt-chat-completions",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "openai",
"config": {
"model_id": "gpt-3.5-turbo",
"model_type": "chat.completions"
},
"prompt_utils": {
"prompt_options": {
"uri": "math.jinja"
}
}
}
}
}We can test this using the following:
import pprint
import requests
inference_request = {
"inputs": [
{
"name": "role",
"shape": [1],
"datatype": "BYTES",
"data": ["user"],
},
{
"name": "question",
"shape": [1],
"datatype": "BYTES",
"data": ["What is 100 * 2?"],
},
]
}
endpoint = "http://localhost:8080/v2/models/openai-prompt-chat-completions/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=4)
{'id': '396ef236-7ccb-4d57-aa1c-b44c0fc1302d',
'model_name': 'openai-prompt-chat-completions',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['```python\n'
'print(100 * 2)\n'
'```\n'
'```output\n'
'200\n'
'```\n'
'Answer: 200'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{\n'
' "id": "chatcmpl-AYD8UsZUETVwrZeaQJ19Y4uUpojjz",\n'
' "choices": [\n'
' {\n'
' "finish_reason": "stop",\n'
' "index": 0,\n'
' "logprobs": null,\n'
' "message": {\n'
' "content": "```python\\nprint(100 * '
'2)\\n```\\n```output\\n200\\n```\\nAnswer: 200",\n'
' "refusal": null,\n'
' "role": "assistant"\n'
' }\n'
' }\n'
' ],\n'
' "created": 1732717010,\n'
' "model": "gpt-3.5-turbo-0125",\n'
' "object": "chat.completion",\n'
' "system_fingerprint": null,\n'
' "usage": {\n'
' "completion_tokens": 23,\n'
' "prompt_tokens": 129,\n'
' "total_tokens": 152,\n'
' "completion_tokens_details": {\n'
' "reasoning_tokens": 0,\n'
' "audio_tokens": 0,\n'
' "accepted_prediction_tokens": 0,\n'
' "rejected_prediction_tokens": 0\n'
' },\n'
' "prompt_tokens_details": {\n'
' "cached_tokens": 0,\n'
' "audio_tokens": 0\n'
' }\n'
' }\n'
'}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}Note that we sent a single tensor named "question", which was the inserted content.
Embedding text
The OpenAI runtime also provides an embedding API that allows you to encode a body of text into a high-dimensional vector representation. Developers can then use this to perform vector searches over similarly encoded text. Here is an example of a model-settings.json:
!cat models/openai-embeddings/model-settings.json{
"name": "openai-embeddings",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "openai",
"config": {
"model_id": "text-embedding-ada-002",
"model_type": "embeddings"
}
}
}
}import pprint
import requests
inference_request = {
"inputs": [
{
"name": "input",
"shape": [2],
"datatype": "BYTES",
"data": [
"At the British Academy Film Awards, Oppenheimer wins Best Film and six other awards, including Best Actor for Cillian Murphy",
"In American football, the Kansas City Chiefs defeat the San Francisco 49ers to win the Super Bowl.",
],
},
]
}
endpoint = "http://localhost:8080/v2/models/openai-embeddings/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=3)
print('\nShape of vectors', response.json()["outputs"][0]["shape"]){'id': 'e747c598-bf96-455e-a018-6762560f3bfd',
'model_name': 'openai-embeddings',
'outputs': [{'data': [...],
'datatype': 'FP64',
'name': 'embedding',
'parameters': {...},
'shape': [...]},
{'data': [...],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {...},
'shape': [...]}],
'parameters': {}}
Shape of vectors [2, 1536]The above output is limited because the embedding vectors are quite long, you can see that there are two and each is length 1536. These can be used to search similar vectorized text corpora for similar content.
Image Generation
The OpenAI runtime also allows access to the Dall-E image generations endpoint, the model-settings.json is similar to the previous cases.
!cat models/openai-image-generations/model-settings.json{
"name": "openai-image-generations",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "openai",
"config": {
"model_type": "images.generations"
}
}
}
}When calling this endpoint we send a prompt tensor. Let's try and depict something nice and cheery!
import pprint
import requests
inference_request = {
"inputs": [
{
"name": "prompt",
"shape": [1],
"datatype": "BYTES",
"data": ["A tropical paradise"],
},
]
}
endpoint = "http://localhost:8080/v2/models/openai-image-generations/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=4){'id': '251e104b-ca3d-4a6a-b53d-dc902be24dfc',
'model_name': 'openai-image-generations',
'outputs': [{'data': ['https://oaidalleapiprodscus.blob.core.windows.net/private/org-RpvLIXbLFPC4TppFUFheeITu/user-KNvzxivr23yj7hkdCOKiXwVT/img-RMzanu241DxCfRUaZGE9PtbI.png?st=2024-11-27T13%3A19%3A28Z&se=2024-11-27T15%3A19%3A28Z&sp=r&sv=2024-08-04&sr=b&rscd=inline&rsct=image/png&skoid=d505667d-d6c1-4a0a-bac7-5c84a87759f8&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-11-27T00%3A20%3A13Z&ske=2024-11-28T00%3A20%3A13Z&sks=b&skv=2024-08-04&sig=ShKcwDQ8P%2B6xarlK0J1MpEpl%2Bl1uEap/UEDvAWpW19U%3D'],
'datatype': 'BYTES',
'name': 'image',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{\n'
' "created": 1732717168,\n'
' "data": [\n'
' {\n'
' "url": '
'"https://oaidalleapiprodscus.blob.core.windows.net/private/org-RpvLIXbLFPC4TppFUFheeITu/user-KNvzxivr23yj7hkdCOKiXwVT/img-RMzanu241DxCfRUaZGE9PtbI.png?st=2024-11-27T13%3A19%3A28Z&se=2024-11-27T15%3A19%3A28Z&sp=r&sv=2024-08-04&sr=b&rscd=inline&rsct=image/png&skoid=d505667d-d6c1-4a0a-bac7-5c84a87759f8&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-11-27T00%3A20%3A13Z&ske=2024-11-28T00%3A20%3A13Z&sks=b&skv=2024-08-04&sig=ShKcwDQ8P%2B6xarlK0J1MpEpl%2Bl1uEap/UEDvAWpW19U%3D"\n'
' }\n'
' ]\n'
'}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}The images are returned as a URL to their location that can be used to download them.
import os
import urllib.request
if not os.path.exists("images"):
os.mkdir("images")
url = response.json()['outputs'][0]['data'][0]
urllib.request.urlretrieve(url, "images/generated-image.jpeg");
Azure Deployments
Our runtime can also be set up to integrate with OpenAI deployments on Azure. To set up such an Azure OpenAI deployment see here.
The following is an example model-settings.json file.
!cat models/azure-chat-completions/model-settings.json{
"name": "azure-openai-chat-model",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "openai",
"config": {
"api_type": "azure",
"azure_endpoint": "https://mlinfra-openai-test.openai.azure.com/",
"model_id": "mlinfra-gpt-35-turbo-test",
"api_version": "2024-08-01-preview",
"model_type": "chat.completions"
}
}
}
}In the above YAML definition
api_typeis "azure" or "azure_ad"azure_endpointis the deployment endpointmodel_idis deployment nameapi_versionis "2024-08-01-preview" and could change in the future
You can call the models in azure as before - everything else stays the same.
Deploying on Seldon Core 2
We will now demostrate how do deploy the chat completions model on Seldon Core 2. All the other models can be deployed with the same steps.
While the runtime image can be used as a stand alone server in most cases you'll want to deploy it as part of a Kubernetes cluster. This section assumes the user has a Kubernetes cluster running with Seldon Core 2 installed in the seldon namespace. In order to start serving OpenAI models you will have to first create a secret for the OpenAI API key and deploy the API Runtime server. Please check our installation tutorial to see how you can do so.
To deploy the chat completions model, we will need to create the associated manifest file.
!cat manifests/openai-chat-completions.yamlapiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
name: openai-chat-completions
spec:
storageUri: "gs://seldon-models/llm-runtimes-settings/examples/api/openai/models/openai_chat_completions"
requirements:
- openaiTo load the model in Seldon Core 2, run:
!kubectl apply -f manifests/openai-chat-completions.yaml -n seldon
!kubectl wait --for condition=ready --timeout=600s model --all -n seldonmodel.mlops.seldon.io/openai-chat-completions created
model.mlops.seldon.io/openai-chat-completions condition metBefore sending the actual request, we need to get the mesh ip. The following util function will help you retrieve the correct ip:
import subprocess
def get_mesh_ip():
cmd = f"kubectl get svc seldon-mesh -n seldon -o jsonpath='{{.status.loadBalancer.ingress[0].ip}}'"
return subprocess.check_output(cmd, shell=True).decode('utf-8')As before, we can now send a request to the model:
import json
import pprint
import requests
inference_request = {
"inputs": [
{
"name": "role",
"shape": [2],
"datatype": "BYTES",
"data": ["system", "user"]
},
{
"name": "content",
"shape": [2],
"datatype": "BYTES",
"data": [
json.dumps(["You are a helpful assistant"]),
json.dumps(["Hello from MLServer!"]),
],
},
{
"name": "type",
"shape": [2],
"datatype": "BYTES",
"data": [
json.dumps(["text"]),
json.dumps(["text"]),
]
}
]
}
endpoint = f"http://{get_mesh_ip()}/v2/models/openai-chat-completions/infer"
response = requests.post(endpoint, json=inference_request)
pprint.pprint(response.json(), depth=4){'id': 'd5325b77-ad4d-4389-8add-f5728b32a493',
'model_name': 'openai-chat-completions_1',
'model_version': '1',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['Hello! How can I assist you today?'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{\n'
' "id": "chatcmpl-AYDDh6QqLGR94agCusfrFDuOvunw5",\n'
' "choices": [\n'
' {\n'
' "finish_reason": "stop",\n'
' "index": 0,\n'
' "logprobs": null,\n'
' "message": {\n'
' "content": "Hello! How can I assist you '
'today?",\n'
' "refusal": null,\n'
' "role": "assistant"\n'
' }\n'
' }\n'
' ],\n'
' "created": 1732717333,\n'
' "model": "gpt-3.5-turbo-0125",\n'
' "object": "chat.completion",\n'
' "system_fingerprint": null,\n'
' "usage": {\n'
' "completion_tokens": 9,\n'
' "prompt_tokens": 21,\n'
' "total_tokens": 30,\n'
' "completion_tokens_details": {\n'
' "reasoning_tokens": 0,\n'
' "audio_tokens": 0,\n'
' "accepted_prediction_tokens": 0,\n'
' "rejected_prediction_tokens": 0\n'
' },\n'
' "prompt_tokens_details": {\n'
' "cached_tokens": 0,\n'
' "audio_tokens": 0\n'
' }\n'
' }\n'
'}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}You now have a deployed model in Seldon Core 2, ready and available for requests! To unload the model, you can run the following command:
!kubectl delete -f manifests/openai-chat-completions.yaml -n seldonmodel.mlops.seldon.io "openai-chat-completions" deletedLast updated
Was this helpful?