The following demonstrates how to locally run an API Runtime instance to run inference with Gemini models. It also illustrates the different ways it can be used.
This example only showcases the API Runtime as a stand-alone component, for a more integrated example check out the chatbot demo.
To get up and running we need to pull the runtime Docker image. To pull the docker image, you must be authenticated. Check our installation tutorial to see how you can authenticate with Docker CLI.
{
"name": "gemini-chat-completions",
"implementation": "mlserver_llm_api.LLMRuntime",
"parameters": {
"extra": {
"provider_id": "gemini",
"config": {
"model_id": "gemini-1.5-flash",
"model_type": "chat.completions",
"llm_parameters": {
"generation_config": {
"temperature": 0.7
},
"safety_settings": {
"HARASSMENT": "BLOCK_LOW_AND_ABOVE"
},
"system_instruction": "You are a cat, your name is Neko"
}
}
}
}
}
In the above settings, the runtime config is specified in the parameters JSON field:
the choice of "provider_id" - currently, the providers supported are "openai" and "gemini".
we've chosen the gemini-1.5-flash model and the chat.completions API.
Starting the Runtime
Finally, to start the server run:
Sending Requests
To send our first request to the chat.completions endpoint that we are now serving via MLServer, we use the following:
Note that we've sent three tensors: a "role", a "content", and a "type" tensor. The "role" tensors tell the model who is speaking. In this case, it includes a "system" role and a "user" role. The "system" role is used to dictate the context of the interaction and the "user" role indicates that the matching content is sent by a user. In the above the system content is: "You are a helpful assistant" and the user content is "Hello from MLServer". The "type" tensor indicates that the content we sent is a text.
The endpoint responds with its own "role", "content" and "type" tensors. Its "role" is given as "assistant" and the "content" it returns is "Hello! How can I assist you today?". As well as this the server returns the full response received from OpenAI via the "output_all" tensor.
Requests with Parameters
We can also add parameters to a given request, to specify sampling arguments. For a list of all available parameters see the Gemini documentation for their API. The following sets the top_k tokens to sample and the maximum number of tokens to generate:
Note that if you are sending a single message, you must not encode the content and type as json.
Adding prompts
Prompting is a pivotal part of using large language models. It allows developers to embed messages sent by a user within other textual contexts that provide more information about the task you want the model to perform. For instance, we use the following prompt to give the model more context about the kind of question that the model is expected to answer:
In the above the content sent by the user can be inserted in the {question} variable. A developer should specify what content to insert thereby giving that tensor the name "question". To start with we need to create a new model-settings.json file. This will be the same as the previous one but in addition it specifies the prompt tempalte to be used through prompt_utils settings.
We can test this using the following (Note that we send a single tensor named "question", this is the content that will be inserted.):
Multi-modal inputs
Gemini API allows the specific mime types and converting the either base64 string-encoded media (in case of REST) or raw bytes (in case of gRPC) to the expected message format. The supported mime types are described in the Gemini API docs (vision and audio):
Images
The following code defines two utility function used to serialize the image for REST and gRPC.
Use any jpg/jpeg image of your choice and replace "assets/grand-canyon.jpg" with the path of your image. For this tutorial we will use the following image (source here).
Audio
We download a mp3 audio sample to be used in this example.
The following code defines an utility function used to serialize the audio file for REST and gRPC.
Deploying on Seldon Core 2
We will now demostrate how do deploy the chat completions model on Seldon Core 2. All the other models can be deployed with the same steps.
While the runtime image can be used as a stand alone server in most cases you'll want to deploy it as part of a Kubernetes cluster. This section assumes the user has a Kubernetes cluster running with Seldon Core 2 installed in the seldon namespace. In order start serving Gemini models you will have to first create a secret for the Gemini API key and deploy the API Runtime server. Please check our installation tutorial to see how you can do so.
To deploy the chat completions model, we will need to create the associated manifest file.
To load the model in Seldon Core 2, run:
Before sending the actual request, we need to get the mesh ip. The following util function will help you retrieve the correct ip:
As before, we can now send a request to the model:
You now have a deployed model in Seldon Core 2, ready and available for requests! To unload the model, you can run the following command.
Qt: At a zoo, each adult ticket costs A and children under 5 can enter for free. If a family of B adults and C children under 5 visit the zoo, what is the total cost for the family to enter?
Mapping: {A: 12, B: 4, C: 2}
Answer:
```python
def zoo_cost(A=12, B=4, C=2):
return A * B
```
Following the example above, write a python function that returns the answer to the following question:
Qt: At a store, shoes cost A per pair and socks cost B per pair. If a customer buys C pairs of shoes and D pairs of socks, what is the total cost of the purchase?
Mapping: {A: {{ A[0] }}, B: {{ B[0] }}, C: {{ C[0] }}, D: {{ D[0]}} }
{'id': 'f63005e0-3b16-4930-9117-2f2e49b58dd4',
'model_name': 'gemini-prompt-chat-completions',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['```python\n'
'def store_cost(A=10, B=5, C=3, D=2): # Added a default '
'value for B. The prompt is ambiguous about what to do '
'if B is missing.\n'
' """Calculates the total cost of shoes and socks.\n'
'\n'
' Args:\n'
' A: Cost of a pair of shoes.\n'
' B: Cost of a pair of socks.\n'
' C: Number of pairs of shoes.\n'
' D: Number of pairs of socks.\n'
'\n'
' Returns:\n'
' The total cost of the purchase.\n'
' """\n'
' return (A * C) + (B * D)\n'
'\n'
'```'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{"candidates": [{"content": {"parts": [{"text": '
'"```python\\ndef store_cost(A=10, B=5, C=3, D=2): # '
'Added a default value for B. The prompt is ambiguous '
'about what to do if B is missing.\\n '
'\\"\\"\\"Calculates the total cost of shoes and '
'socks.\\n\\n Args:\\n A: Cost of a pair of '
'shoes.\\n B: Cost of a pair of socks.\\n C: '
'Number of pairs of shoes.\\n D: Number of pairs of '
'socks.\\n\\n Returns:\\n The total cost of the '
'purchase.\\n \\"\\"\\"\\n return (A * C) + (B * '
'D)\\n\\n```"}], "role": "model"}, "finish_reason": 1, '
'"safety_ratings": [], "token_count": 0, '
'"grounding_attributions": []}], "usage_metadata": '
'{"prompt_token_count": 187, "candidates_token_count": '
'138, "total_token_count": 325, '
'"cached_content_token_count": 0}}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}
{'id': '552a633f-032e-4ee5-bffd-0ad94a255c89',
'model_name': 'gemini-chat-completions',
'outputs': [{'data': ['assistant'],
'datatype': 'BYTES',
'name': 'role',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['Mrow! From my perch high on this reddish-orange rock, '
"I see a HUGE river snaking down below. It's a deep, "
'dark blue, almost purple in the shadows. The river '
'curls around a giant, reddish-brown rock formation, '
"making a perfect horseshoe shape. It's so big! The "
'walls of the canyon are all the same rusty color, '
'towering all around. The sky is a pretty sunset '
"orange and pink. There's even a little green plant "
"right beside me, though it's far too prickly to nap "
'on. A very interesting view for a discerning feline '
'such as myself. Purrfect!\n'],
'datatype': 'BYTES',
'name': 'content',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['text'],
'datatype': 'BYTES',
'name': 'type',
'parameters': {'content_type': 'str'},
'shape': [1, 1]},
{'data': ['{"candidates": [{"content": {"parts": [{"text": '
'"Mrow! From my perch high on this reddish-orange '
"rock, I see a HUGE river snaking down below. It's a "
'deep, dark blue, almost purple in the shadows. The '
'river curls around a giant, reddish-brown rock '
"formation, making a perfect horseshoe shape. It's so "
'big! The walls of the canyon are all the same rusty '
'color, towering all around. The sky is a pretty '
"sunset orange and pink. There's even a little green "
"plant right beside me, though it's far too prickly to "
'nap on. A very interesting view for a discerning '
'feline such as myself. Purrfect!\\n"}], "role": '
'"model"}, "finish_reason": 1, "safety_ratings": '
'[{"category": 8, "probability": 1, "blocked": false}, '
'{"category": 10, "probability": 1, "blocked": false}, '
'{"category": 7, "probability": 1, "blocked": false}, '
'{"category": 9, "probability": 1, "blocked": false}], '
'"token_count": 0, "grounding_attributions": []}], '
'"usage_metadata": {"prompt_token_count": 274, '
'"candidates_token_count": 139, "total_token_count": '
'413, "cached_content_token_count": 0}}'],
'datatype': 'BYTES',
'name': 'output_all',
'parameters': {'content_type': 'str'},
'shape': [1, 1]}],
'parameters': {}}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 156k 100 156k 0 0 284k 0 --:--:-- --:--:-- --:--:-- 284k
