# Changelog ## [0.7.0](https://github.com/SeldonIO/llm-runtimes/releases/tag/0.7.0) - 2025-06-06 ## Features * Added support of agentic workflows via OpenAI API (e.g. tools, planning, reflection patterns). ## What's Changed * Update CHANGELOG by [@github-actions](https://github.com/github-actions) in [#672](https://github.com/SeldonIO/llm-runtimes/pull/672) * Bumped version to 0.6.0 in docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#673](https://github.com/SeldonIO/llm-runtimes/pull/673) * Bumped version to 0.7.0.dev1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#674](https://github.com/SeldonIO/llm-runtimes/pull/674) * Improved rag example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#675](https://github.com/SeldonIO/llm-runtimes/pull/675) * build(deps-dev): Bump pytest-asyncio from 0.25.3 to 0.26.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#677](https://github.com/SeldonIO/llm-runtimes/pull/677) * build(deps-dev): Bump syrupy from 4.9.0 to 4.9.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#676](https://github.com/SeldonIO/llm-runtimes/pull/676) * Updated changelog for 0.5.0 and 0.6.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#681](https://github.com/SeldonIO/llm-runtimes/pull/681) * Add files via upload by [@paulb-seldon](https://github.com/paulb-seldon) in [#683](https://github.com/SeldonIO/llm-runtimes/pull/683) * Fixed async operation on the agentic workflow pipeline by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#684](https://github.com/SeldonIO/llm-runtimes/pull/684) * Fix database sync for chatbot and agentic workflow by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#686](https://github.com/SeldonIO/llm-runtimes/pull/686) * Included support for agents implemented with cyclic pipelines by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#685](https://github.com/SeldonIO/llm-runtimes/pull/685) * Fixed tool args for openai and gemini by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#688](https://github.com/SeldonIO/llm-runtimes/pull/688) * build(deps-dev): Bump wheel from 0.44.0 to 0.46.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#682](https://github.com/SeldonIO/llm-runtimes/pull/682) * build(deps-dev): Bump flake8 from 7.1.2 to 7.2.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#680](https://github.com/SeldonIO/llm-runtimes/pull/680) * build(deps-dev): Bump types-requests from 2.32.0.20250306 to 2.32.0.20250328 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#679](https://github.com/SeldonIO/llm-runtimes/pull/679) * Planning agent pattern example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#687](https://github.com/SeldonIO/llm-runtimes/pull/687) * add agents docs changes by [@paulb-seldon](https://github.com/paulb-seldon) in [#689](https://github.com/SeldonIO/llm-runtimes/pull/689) * FIx introduction links by [@paulb-seldon](https://github.com/paulb-seldon) in [#690](https://github.com/SeldonIO/llm-runtimes/pull/690) * Change email to reach out to us by [@paulb-seldon](https://github.com/paulb-seldon) in [#691](https://github.com/SeldonIO/llm-runtimes/pull/691) * Bumped MLServer to 1.7.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#693](https://github.com/SeldonIO/llm-runtimes/pull/693) * Bumped st<4.2.0 and included upper bound for optimum by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#694](https://github.com/SeldonIO/llm-runtimes/pull/694) * build(deps-dev): Bump types-requests from 2.32.0.20250328 to 2.32.0.20250515 in /prompt-utils by [@dependabot](https://github.com/dependabot) in [#695](https://github.com/SeldonIO/llm-runtimes/pull/695) * Set release large runner for local runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#696](https://github.com/SeldonIO/llm-runtimes/pull/696) * Bump MLServer to 1.7.1.rc1 and LLMIS to 0.3.3 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#702](https://github.com/SeldonIO/llm-runtimes/pull/702) * ci: Merge change for release 0.7.0 ([#702](https://github.com/SeldonIO/llm-runtimes/issues/702)) by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#703](https://github.com/SeldonIO/llm-runtimes/pull/703) * Fixed docker file for local runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#704](https://github.com/SeldonIO/llm-runtimes/pull/704) * Fix docs for 0.7.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#705](https://github.com/SeldonIO/llm-runtimes/pull/705) * Bumped MLServer to 1.7.1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#706](https://github.com/SeldonIO/llm-runtimes/pull/706) * ci: Merge change for release 0.7.0 \[2] by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in [#707](https://github.com/SeldonIO/llm-runtimes/pull/707) **Full Changelog**: ## [0.6.0](https://github.com/SeldonIO/llm-runtimes/releases/tag/0.6.0) - Mar 13, 2025 ## Main Features * New runtimes: * Sentence Transformers for text embeddings * Prompt Runtime - allows to reference the same model deployed locally with different prompts * Support for conditional routing through Core 2 pipelines - this feature allows conditional data flows based on the output of the LLM enabling support for agentic workflows ## What's Changed * Replaced local images with the ones from artifacts registry by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Remove redundant server configs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Update Models Overview by [@paulb-seldon](https://github.com/paulb-seldon) in * Quickstart example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed models and pipelines naming by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included upperbound on httpx by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Clean docs by [@paulb-seldon](https://github.com/paulb-seldon) in * Remove PVC from docs examples by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated images tag to 0.5.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed installation test image path by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Cat the pipeline definition before deploying it by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed typo in azure docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed name of vector db env prefix by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Sentence transformers support by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Simplify content when content is a single text message by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Refactored extra settings init by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Passed database as string in get\_db\_interface by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed memory psql db url by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Prompt runtime for local llms by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Docs for local-embedding runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Renamed url to infer\_uri in PromptRuntime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included output postprocessing for conditional routing by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Wrote prompt runtime docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Wrote docs for routing by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Renamed routing to agents by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump black from 24.10.0 to 25.1.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump isort from 5.13.2 to 6.0.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.23.0 to 4.24.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump syrupy from 4.7.2 to 4.8.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.13.0 to 1.14.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-asyncio from 0.24.0 to 0.25.3 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * Implemented routing for prompt runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included installation steps for prompt runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Guardrails docs example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bumped mlserver to 1.7.0.rc1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * bump LLMIS to 0.3.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Workflow for prompt-utils runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Workflow for prompt-utils runtime (#652) by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed capabilities for prompt utils server by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Docs for http sse streaming by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bumped LLMIS to 0.3.1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump flake8 from 7.1.1 to 7.1.2 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump isort from 6.0.0 to 6.0.1 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 8.3.3 to 8.3.5 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20241016 to 2.32.0.20250306 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.24.1 to 4.24.2 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump syrupy from 4.8.1 to 4.9.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * ci: Merge change for release 0.6.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fix docs for 0.6.0.rc3 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fix docs for 0.6.0.rc3 (#668) by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bump mlserver to 1.7.0.rc2 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bump mlserver to 1.7.0.rc2 (#670) by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in **Full Changelog**: ## [0.5.0](https://github.com/SeldonIO/llm-runtimes/releases/tag/0.5.0) - Dec 4, 2024 ## Main Features * New runtimes: * Gemini * VectorDB with support for [Qdrant](https://qdrant.tech/) and [PGVector](https://github.com/pgvector/pgvector) * Streaming support for OpenAI and Gemini * Standardized IO accross all runtimes (i.e. prompting) * RAG support for the Local Runtime ## What's Changed * Update CHANGELOG by [@github-actions](https://github.com/github-actions) in * Bump openai 1.35.3 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included streaming support for OpenAI by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump tox from 4.15.1 to 4.16.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.15.1 to 4.16.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.15.1 to 4.16.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): Bump orjson from 3.10.4 to 3.10.6 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.0 to 1.10.1 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.0 to 1.10.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.0 to 1.10.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240602 to 2.32.0.20240622 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240602 to 2.32.0.20240622 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240602 to 2.32.0.20240622 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.0.0 to 7.1.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.0.0 to 7.1.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.0.0 to 7.1.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps): Bump urllib3 from 2.2.1 to 2.2.2 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps): Bump urllib3 from 2.2.1 to 2.2.2 in /local/requirements by [@dependabot](https://github.com/dependabot) in * Add a `DISABLE_CYTHON` environment variable for development purposes by @jklaise in * Do not exclude packages when installing in dev mode without Cython by @jklaise in * added the create the servers pieces to the documentation by @joshsgoldstein in * Feat/gemini api runtime by @jklaise in * Migration to GitBook and General Doc Improvements by @ramonpzg in * Update dependabot.yml to remove docs scan by [@sakoush](https://github.com/sakoush) in * fix: CI failing because because of a change of permission by[@sakoush](https://github.com/sakoush) in * fix: remove docs in version script by [@sakoush](https://github.com/sakoush) in * bump versions from `0.3.0` to `0.5.0.dev` by @mauicv in * Fix/gemini grpc bytes decoding by @jklaise in * Add types tensor to Gemini output by @jklaise in * fix: add security workflow for llm-runtimes by [@sakoush](https://github.com/sakoush) in * fix: move workflow to workflows directory by [@sakoush](https://github.com/sakoush) in * fix: use one big security scan task by [@sakoush](https://github.com/sakoush) in * fix: add build base to local docker image build step by [@sakoush](https://github.com/sakoush) in * fix: Update security.yml to exclude our runtimes from 3rd party deps by [@sakoush](https://github.com/sakoush) in * Feat/gemini multi modal input by @jklaise in * Streaming support for Gemini by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fix setting stream flag for other tasks than completion and chat by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump black from 24.4.2 to 24.8.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.4.2 to 24.8.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.4.2 to 24.8.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.1.0 to 7.1.1 in /api by [@dependabot](https://github.com/dependabot) in * Included VectorDB runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump flake8 from 7.1.0 to 7.1.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump wheel from 0.43.0 to 0.44.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.1.0 to 7.1.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): Bump orjson from 3.10.6 to 3.10.7 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.1 to 1.11.2 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump wheel from 0.43.0 to 0.44.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.4.2 to 24.8.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump flake8 from 7.1.0 to 7.1.1 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.16.0 to 4.18.1 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.16.0 to 4.18.1 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240622 to 2.32.0.20240907 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps): Update google-generativeai requirement from ==0.7.\* to >=0.7,<0.9 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.16.0 to 4.18.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-postgresql from 6.0.0 to 6.1.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps): Bump sqlalchemy from 2.0.31 to 2.0.34 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump wheel from 0.43.0 to 0.44.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240622 to 2.32.0.20240907 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240622 to 2.32.0.20240907 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.1 to 1.11.2 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.1 to 1.11.2 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump wheel from 0.43.0 to 0.44.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.16.0 to 4.18.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240622 to 2.32.0.20240907 in /api by [@dependabot](https://github.com/dependabot) in * Update security.yml to include vector-db for code scan by [@sakoush](https://github.com/sakoush) in * build(deps-dev): Bump syrupy from 4.6.1 to 4.7.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.1 to 1.11.2 in /local by [@dependabot](https://github.com/dependabot) in * Bumped MLServer to 1.6.1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bumped llmis to 0.2.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included missing files for qdrant by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included pgvector support by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump tox from 4.18.1 to 4.19.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.18.1 to 4.19.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.18.1 to 4.19.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.18.1 to 4.19.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240907 to 2.32.0.20240914 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240907 to 2.32.0.20240914 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240907 to 2.32.0.20240914 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240907 to 2.32.0.20240914 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): Bump sqlalchemy from 2.0.34 to 2.0.35 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 7.3.1 to 8.3.3 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 7.3.1 to 8.3.3 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 7.3.1 to 8.3.3 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-asyncio from 0.23.7 to 0.24.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-asyncio from 0.23.7 to 0.24.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-asyncio from 0.23.7 to 0.24.0 in /local by [@dependabot](https://github.com/dependabot) in * Prototype/add prompt and preprocessing by @mauicv in * Add gitbook docs to master branch by [@sakoush](https://github.com/sakoush) in * Integrate 461 in memory by @mauicv in * Moved prompt-utils in requirements.txt by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Integrate #461 in API runtime by @mauicv in * Prototype integrate 461 in local by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Prototype integrate 461 in vector db by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump pytest from 7.3.1 to 8.3.3 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-asyncio from 0.23.7 to 0.24.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-cases from 3.8.5 to 3.8.6 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-cases from 3.8.5 to 3.8.6 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-cases from 3.8.5 to 3.8.6 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-cases from 3.8.5 to 3.8.6 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.19.0 to 4.21.2 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest-cases from 3.8.5 to 3.8.6 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.19.0 to 4.21.2 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240914 to 2.32.0.20241016 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump syrupy from 4.7.1 to 4.7.2 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240907 to 2.32.0.20241016 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * Rag support for local runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump tox from 4.19.0 to 4.21.2 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.8.0 to 24.10.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.8.0 to 24.10.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.19.0 to 4.23.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * Fix memory system prompt by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Runtime settings consistency by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump black from 24.8.0 to 24.10.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump syrupy from 4.7.1 to 4.7.2 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump black from 24.8.0 to 24.10.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps): Update qdrant-client requirement from <1.12.0 to <1.13.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.11.2 to 1.12.0 in /vector-db by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.11.2 to 1.12.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.11.2 to 1.12.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.18.1 to 4.23.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump mypy from 1.10.1 to 1.13.0 in /prompt-utils by [@dependabot](https://github.com/dependabot) in * Default model type initialization in prompt utils by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Renamed mlserver-llm-openai to mlserver-llm-api by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fix openai setting config by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated openai docs with IO by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * OpenAI function calling docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO gemini docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Refactored the azure docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO memory docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO vector-db docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO local docs - chat model by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO local mms docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO local chat template by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * fix(docs): pull changes from docs-master by [@sakoush](https://github.com/sakoush) in * IO local quantization docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * IO chat bot by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Fixed tensor names for vector-db example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated azure model settings and image by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Improved local server manifest by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Included wait ready for models, pipelines, and deployments by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated and added docs refs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated installation docs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Removed k8s example by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Docs for monitoring by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Updated runtimes docs intro by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Docs restructure by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * ci: Merge change for release 0.5.0 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in **Full Changelog**: ## [0.4.0](https://github.com/SeldonIO/llm-runtimes/releases/tag/0.4.0) - 27 Jun 2024 ## Main Features * Code transpilation using `cython`, which allows code obfuscation * Output streaming support for local runtime, which allows faster time to first token back to the user * SQL backend for memory runtime, which allows the use of dbs such as postgres or mysql to store chat messages * Support for MMS, which allows the use of multiple models on the same GPU (limited support) ## What's Changed * build(ci): Use a correct example by [@jesse-c](https://github.com/jesse-c) in * docs: Remove link to GitHub repository by [@jesse-c](https://github.com/jesse-c) in * build: Add license by [@jesse-c](https://github.com/jesse-c) in * feat(local): Bump LLMIS to `0.0.1.rc2` by [@jesse-c](https://github.com/jesse-c) in * build(deps): bump idna from 3.6 to 3.7 in /docs by [@dependabot](https://github.com/dependabot) in * docs: Correct link to memory examples by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump black from 24.3.0 to 24.4.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.3.0 to 24.4.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.3.0 to 24.4.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.3.0 to 24.4.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): bump sphinx-autobuild from 2024.2.4 to 2024.4.13 in /docs by [@dependabot](https://github.com/dependabot) in * Remove duplicate docs examples by [@mauicv](https://github.com/mauicv) in * docs(local): Add Core 2 model examples by [@jesse-c](https://github.com/jesse-c) in * docs: Add changelog by [@github-actions](https://github.com/github-actions) in * release: placeholder for promotion action by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * build(deps): bump sphinx-autobuild from 2024.4.13 to 2024.4.16 in /docs by [@dependabot](https://github.com/dependabot) in * release: promote images from dev to production registry by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * feat(docs): Add e2e example by [@mauicv](https://github.com/mauicv) in * Bugfix/add docs pr changes by [@mauicv](https://github.com/mauicv) in * build(deps): bump notebook from 7.1.2 to 7.1.3 in /docs by [@dependabot](https://github.com/dependabot) in * fix(local): Remove dupe `kwargs` setting by [@jesse-c](https://github.com/jesse-c) in * Fix links and MakeFile by [@mauicv](https://github.com/mauicv) in * Add minor note on MinIO to chatbot example by [@mauicv](https://github.com/mauicv) in * Update the image in the chatbot example doc by [@mauicv](https://github.com/mauicv) in * build(deps): bump myst-parser from 2.0.0 to 3.0.0 in /docs by [@dependabot](https://github.com/dependabot) in * fix(deps): Use LLMIS 0.0.1 by [@sakoush](https://github.com/sakoush) in * docs: Add installation by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump black from 24.4.0 to 24.4.1 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.9.0 to 1.10.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.4.0 to 24.4.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.4.0 to 24.4.2 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.9.0 to 1.10.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.9.0 to 1.10.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.9.0 to 1.10.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): update torch requirement from <2.3.0,>=2.0.1 to >=2.0.1,<2.4.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.4.0 to 24.4.2 in /api by [@dependabot](https://github.com/dependabot) in * Update CHANGELOG by [@github-actions](https://github.com/github-actions) in * Bump version to 0.3.0 by [@jesse-c](https://github.com/jesse-c) in * docs(api): Fix header level by [@jesse-c](https://github.com/jesse-c) in * Add section at end of e2e example for changing api to local rt by [@mauicv](https://github.com/mauicv) in * Wholesale docs edits pb by [@paulb-seldon](https://github.com/paulb-seldon) in * run CI tests only for version of Python that is used within the Docker image by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * deprecate and archive deepspeed by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * fix: Remove references to removed runtime by [@jesse-c](https://github.com/jesse-c) in * Update API examples for docker access and kubernetes deployment by [@mauicv](https://github.com/mauicv) in * Included local runtime limitations section. by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * use {{ current-version }} in example manifest by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * Feature/add prompt note by [@mauicv](https://github.com/mauicv) in * Docs example for quantization and tensor parallelism by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Add local model loading from settings relative path by [@mauicv](https://github.com/mauicv) in * docs: Update docs for azure openai by [@sakoush](https://github.com/sakoush) in * build(deps): bump notebook from 7.1.3 to 7.2.0 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps): bump jinja2 from 3.1.3 to 3.1.4 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps): bump myst-parser from 3.0.0 to 3.0.1 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.1.1 to 8.2.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.1.1 to 8.2.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.1.1 to 8.2.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.14.2 to 4.15.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.14.2 to 4.15.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 24.4.1 to 24.4.2 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.14.2 to 4.15.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps): bump requests from 2.31.0 to 2.32.0 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.23.6 to 0.23.7 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.23.6 to 0.23.7 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.23.6 to 0.23.7 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.2.0 to 8.2.1 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.2.0 to 8.2.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 8.2.0 to 8.2.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.31.0.20240406 to 2.32.0.20240521 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.31.0.20240406 to 2.32.0.20240521 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.31.0.20240406 to 2.32.0.20240521 in /local by [@dependabot](https://github.com/dependabot) in * feat(memory): Add postgres backend by [@mauicv](https://github.com/mauicv) in * Included documentation and examples for mms by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump types-requests from 2.32.0.20240521 to 2.32.0.20240523 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240521 to 2.32.0.20240523 in /api by [@dependabot](https://github.com/dependabot) in * build(deps): Bump sphinx-design from 0.5.0 to 0.6.0 in /docs by [@dependabot](https://github.com/dependabot) in * Pin open telemetry sdk package to fix ci by [@mauicv](https://github.com/mauicv) in * Ensure db portability in memory rt by [@mauicv](https://github.com/mauicv) in * build(deps-dev): Bump types-requests from 2.32.0.20240523 to 2.32.0.20240602 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240521 to 2.32.0.20240602 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump types-requests from 2.32.0.20240523 to 2.32.0.20240602 in /memory by [@dependabot](https://github.com/dependabot) in * Update incorrect prompt settings in model-settings.json cell by [@mauicv](https://github.com/mauicv) in * build(deps): Bump opentelemetry-sdk from 1.24.0 to 1.25.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps): Bump opentelemetry-sdk from 1.24.0 to 1.25.0 in /memory by [@dependabot](https://github.com/dependabot) in * Revert "build(deps): Bump opentelemetry-sdk from 1.24.0 to 1.25.0 in /memory" by [@jesse-c](https://github.com/jesse-c) in * Revert "build(deps): Bump opentelemetry-sdk from 1.24.0 to 1.25.0 in /api" by [@jesse-c](https://github.com/jesse-c) in * Remove pinned opentelemetry dep for memory, api and local rt by [@mauicv](https://github.com/mauicv) in * build(deps-dev): Bump pytest from 8.2.1 to 8.2.2 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 8.2.1 to 8.2.2 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump pytest from 8.2.1 to 8.2.2 in /local by [@dependabot](https://github.com/dependabot) in * SQL backend docs by [@mauicv](https://github.com/mauicv) in * docs(local): Add output for example by [@jesse-c](https://github.com/jesse-c) in * docs(local): Fix dupe `image` field by [@jesse-c](https://github.com/jesse-c) in * Migrate to pydantic v2 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Bumped mlserver to 1.6.0.dev2 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Cython support for llm-runtimes by [@sakoush](https://github.com/sakoush) in * Streaming support for local runtime by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * build(deps-dev): Bump tox from 4.15.0 to 4.15.1 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.15.0 to 4.15.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump tox from 4.15.0 to 4.15.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps): Bump orjson from 3.10.3 to 3.10.4 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps): Bump tornado from 6.4 to 6.4.1 in /docs by [@dependabot](https://github.com/dependabot) in * build(deps): Bump notebook from 7.2.0 to 7.2.1 in /docs by [@dependabot](https://github.com/dependabot) in * Rename db option by [@mauicv](https://github.com/mauicv) in * Bumped llmis to 0.1.1 by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * use mlserver 1.6.0.rc1 by [@sakoush](https://github.com/sakoush) in * build(deps): Bump sqlalchemy from 2.0.30 to 2.0.31 in /memory by [@dependabot](https://github.com/dependabot) in * ci: Merge changes from master to release branch by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * Update tests.yml by [@sakoush](https://github.com/sakoush) in * ci: Merge changes from master to release branch by [@sakoush](https://github.com/sakoush) in * use mlserver 1.6.0 by [@sakoush](https://github.com/sakoush) in * ci: Merge changes from master to release branch by [@sakoush](https://github.com/sakoush) in ## New Contributors * [@paulb-seldon](https://github.com/paulb-seldon) made their first contribution in **Full Changelog**: [Changes](https://github.com/SeldonIO/llm-runtimes/compare/0.3.0...0.4.0) ## [0.3.0](https://github.com/SeldonIO/llm-runtimes/releases/tag/0.3.0) - 26 Apr 2024 ## Overview **Content** This is the initial release of Seldon’s LLM Module. The LLM Module provides a set of MLServer runtimes to enable serving and deployment of large language (and other GenAI) model applications using Seldon Core v2 models and pipelines. The initial set of runtimes includes an API gateway to access LLM solutions hosted by 3rd parties, a Local self-hosted for on-prem LLM deployment, and Conversational Memory to build stateful LLM applications. **API** There are many exciting companies who provide access to LLMs as a service. They all vary in their APIs. This runtime provides a unified way to access them, starting with OpenAI. This requires egress access to OpenAI’s endpoints. **Local** You can use both open foundational or fine-tuned models, such as those from Mistral and Meta, or your own custom models, trained from scratch. There’s many different performance characteristics of the different ways to run these models. We provide a unified way to access leading backends, including Transformers, vLLM, and DeepSpeed. **Conversational Memory** This facilitates the building of stateful chatbots that save conversational history. With this, you’re able to memorise (aka store) conversations long-term. This means that the context is kept and is able to be used by the different models through the **API** and **Local** runtimes. We’ve carefully benchmarked each runtime to ensure that there is minimal overhead, and in some circumstances, have made our own improvements on top of the supported backends. ## What's Changed * Add tox tests and github workflow by [@sakoush](https://github.com/sakoush) in * Updates to allow runtime to be built for MLServer by [@sakoush](https://github.com/sakoush) in * Refactor openai runtime directory name by [@sakoush](https://github.com/sakoush) in * Deepspeed runtime by [@sakoush](https://github.com/sakoush) in * Initial commit for Microsoft Guidance Runtime by [@ukclivecox](https://github.com/ukclivecox) in * Update workflow to add linting for Guidance by [@sakoush](https://github.com/sakoush) in * Update notebook and readme by [@sakoush](https://github.com/sakoush) in * Update notebook (2) by [@sakoush](https://github.com/sakoush) in * Add mms notebook for deepspeed by [@sakoush](https://github.com/sakoush) in * Change title of MMS notebook by [@sakoush](https://github.com/sakoush) in * Update deepspeed mii to 0.0.6 by [@sakoush](https://github.com/sakoush) in * Return root error if failing to call openai endpoints by [@sakoush](https://github.com/sakoush) in * Add MS azure openai support by [@sakoush](https://github.com/sakoush) in * Update packages (mlserver 1.4.rc6 and mii 0.0.8) by [@sakoush](https://github.com/sakoush) in * Add parsing json string for HF params by [@sakoush](https://github.com/sakoush) in * Add extra args in inference config to be processed by [@sakoush](https://github.com/sakoush) in * Remove reference to apache2 in codebase by [@sakoush](https://github.com/sakoush) in * Openai images generation model fix + other small improvements by [@sakoush](https://github.com/sakoush) in * CI for building docker images by [@sakoush](https://github.com/sakoush) in * Add ability to specify version for deepspeed image by [@sakoush](https://github.com/sakoush) in * Fix bytes payload in openai runtime by [@sakoush](https://github.com/sakoush) in * upgrade deepspeed mii to `0.1.0` by [@sakoush](https://github.com/sakoush) in * Remove guidance by [@sakoush](https://github.com/sakoush) in * Release workflow by [@sakoush](https://github.com/sakoush) in * CI fix using envar by [@sakoush](https://github.com/sakoush) in * Adjust envar CI by [@sakoush](https://github.com/sakoush) in * CI fix: add env to name for action by [@sakoush](https://github.com/sakoush) in * Feature/add periodic ci and dependabot by [@mauicv](https://github.com/mauicv) in * Bump types-requests from 2.28.11.5 to 2.31.0.20240125 in /llm-api by [@dependabot](https://github.com/dependabot) in * Bump types-requests from 2.28.11.5 to 2.31.0.20240125 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump pytest-mock from 3.10.0 to 3.12.0 in /llm-api by [@dependabot](https://github.com/dependabot) in * Bump pytest-cases from 3.6.14 to 3.8.2 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump pytest-asyncio from 0.21.0 to 0.23.3 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump deepspeed-mii from 0.1.0 to 0.2.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump mypy from 1.2.0 to 1.8.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump flake8 from 6.0.0 to 7.0.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump flake8 from 6.0.0 to 7.0.0 in /llm-api by [@dependabot](https://github.com/dependabot) in * Update dependabot.yml to include dockerfiles by [@sakoush](https://github.com/sakoush) in * Bump pytest-asyncio from 0.23.3 to 0.23.4 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Update torch requirement from \~=2.0.1 to >=2.0.1,<2.2.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump pytest from 7.3.1 to 8.0.0 in /llm-api by [@dependabot](https://github.com/dependabot) in * Bump pytest-mock from 3.10.0 to 3.12.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump mypy from 1.2.0 to 1.8.0 in /llm-api by [@dependabot](https://github.com/dependabot) in * Bump pytest-cases from 3.6.14 to 3.8.2 in /llm-api by [@dependabot](https://github.com/dependabot) in * docs(readme): Correct spelling mistake by [@jesse-c](https://github.com/jesse-c) in * \[File System] Conversational Buffer Memory Runtime by [@mauicv](https://github.com/mauicv) in * Compatibility changes to openai-runtime for memory runtime requirements by [@mauicv](https://github.com/mauicv) in * fix: Use consistent package naming by [@jesse-c](https://github.com/jesse-c) in * fix(build): Missed folder name for prefix change by [@jesse-c](https://github.com/jesse-c) in * Bump pytest-asyncio from 0.23.4 to 0.23.5 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Add pydantic validation for api base runtime by [@mauicv](https://github.com/mauicv) in * build(ci): Migrate from Node 16 actions to at least Node 20 by [@jesse-c](https://github.com/jesse-c) in * build(ci): Correct disk usage maximising values by [@jesse-c](https://github.com/jesse-c) in * Bump types-requests from 2.31.0.20240125 to 2.31.0.20240218 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump black from 23.3.0 to 24.2.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump pytest from 7.3.1 to 8.0.1 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Update torch requirement from <2.2.0,>=2.0.1 to >=2.0.1,<2.3.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump deepspeed-mii from 0.2.0 to 0.2.2 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(dependabot): Correct the config and update DeepSeed README by [@jesse-c](https://github.com/jesse-c) in * fix(api): Use latest formatting by [@jesse-c](https://github.com/jesse-c) in * Bump pytest-asyncio from 0.21.0 to 0.23.5 in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest from 8.0.0 to 8.0.1 in /api by [@dependabot](https://github.com/dependabot) in * Bump types-requests from 2.31.0.20240125 to 2.31.0.20240218 in /api by [@dependabot](https://github.com/dependabot) in * Remove depreciated edits endpoint by [@mauicv](https://github.com/mauicv) in * Fix incorrect window size on bulk upload of messages by [@mauicv](https://github.com/mauicv) in * Bump seldonio/mlserver from 1.4.0.rc5-slim to 1.4.0-slim in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump seldonio/mlserver from 1.4.0.rc5-slim to 1.4.0-slim in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest from 8.0.1 to 8.0.2 in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest from 8.0.1 to 8.0.2 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build: Bump to latest MLServer release by [@jesse-c](https://github.com/jesse-c) in * Bump seldonio/mlserver from 1.4.0-slim to 1.5.0-slim in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump seldonio/mlserver from 1.4.0-slim to 1.5.0-slim in /api by [@dependabot](https://github.com/dependabot) in * Add memory docs by [@mauicv](https://github.com/mauicv) in * Bump opentelemetry-instrumentation from 0.39b0 to 0.41b0 in /docs by [@dependabot](https://github.com/dependabot) in * Bump starlette from 0.27.0 to 0.36.2 in /docs by [@dependabot](https://github.com/dependabot) in * Bump pytest-asyncio from 0.23.5 to 0.23.5.post1 in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest-cases from 3.8.2 to 3.8.3 in /api by [@dependabot](https://github.com/dependabot) in * Bump mypy from 1.8.0 to 1.9.0 in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest-cases from 3.8.2 to 3.8.3 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump pytest-asyncio from 0.23.5 to 0.23.5.post1 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump deepspeed-mii from 0.2.2 to 0.2.3 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump fastapi from 0.97.0 to 0.109.1 in /docs by [@dependabot](https://github.com/dependabot) in * Bump mypy from 1.8.0 to 1.9.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * feat(local): Initial support by [@jesse-c](https://github.com/jesse-c) in * feat(memory):Refactor filesys interface by [@mauicv](https://github.com/mauicv) in * feat(docs): Add OpenAI runtime docs by [@mauicv](https://github.com/mauicv) in * feat(local): Add prompting by [@jesse-c](https://github.com/jesse-c) in * feat(local): Add text generation by [@jesse-c](https://github.com/jesse-c) in * fix(local): Remove uneeded files by [@jesse-c](https://github.com/jesse-c) in * build(ci): Fix testing by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump black from 24.2.0 to 24.3.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * fix(local): Correctly pass parameters and check for lengths by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump black from 24.2.0 to 24.3.0 in /api by [@dependabot](https://github.com/dependabot) in * Bump types-requests from 2.31.0.20240218 to 2.31.0.20240311 in /deepspeed by [@dependabot](https://github.com/dependabot) in * Bump types-requests from 2.31.0.20240218 to 2.31.0.20240311 in /api by [@dependabot](https://github.com/dependabot) in * refactor(local): Support choosing LLMIS runtimes by [@jesse-c](https://github.com/jesse-c) in * refactor(local): Use async generation by [@jesse-c](https://github.com/jesse-c) in * build(ci): Add missing images by [@jesse-c](https://github.com/jesse-c) in * fix(deps): Add dependabot workflow + upgrade mlserver to 1.5 by [@sakoush](https://github.com/sakoush) in * build(deps-dev): bump tox from 4.4.12 to 4.14.1 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.8.0 to 1.9.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): Bump to latest for local by [@jesse-c](https://github.com/jesse-c) in * feat(local): Add vLLM and DeepSpeed examples and tests by [@jesse-c](https://github.com/jesse-c) in * refactor(local): Don't require the model ready to unload by [@jesse-c](https://github.com/jesse-c) in * fix(local): Remove unused function stub by [@jesse-c](https://github.com/jesse-c) in * docs: Correct install target by [@jesse-c](https://github.com/jesse-c) in * docs(local): Update README with newer examples by [@jesse-c](https://github.com/jesse-c) in * fix(local): Pin LLMIS commit by [@jesse-c](https://github.com/jesse-c) in * refactor(local): Share response collection by [@jesse-c](https://github.com/jesse-c) in * refactor(local): Return correct error types by [@jesse-c](https://github.com/jesse-c) in * refactor(local): Appropriately log for debugging by [@jesse-c](https://github.com/jesse-c) in * feat(local): relative prompt path in local rt model-settings by [@mauicv](https://github.com/mauicv) in * Rename memory image by [@mauicv](https://github.com/mauicv) in * fix(ci): fix GH worker space issue by [@sakoush](https://github.com/sakoush) in * fix(build): fix failing build for local runtime by [@RafalSkolasinski](https://github.com/RafalSkolasinski) in * fix: upgrade image to cuda 12.1 by [@sakoush](https://github.com/sakoush) in * build(deps-dev): bump pytest-mock from 3.12.0 to 3.14.0 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.14.1 to 4.14.2 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump mypy from 1.2.0 to 1.9.0 in /memory by [@dependabot](https://github.com/dependabot) in * Bump pytest from 8.0.2 to 8.1.1 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.4.12 to 4.14.2 in /api by [@dependabot](https://github.com/dependabot) in * Bump pytest from 8.0.2 to 8.1.1 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.23.5.post1 to 0.23.6 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.6.14 to 3.8.4 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.23.5.post1 to 0.23.6 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump black from 23.3.0 to 24.3.0 in /memory/requirements by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.8.3 to 3.8.4 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-asyncio from 0.21.0 to 0.23.6 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-mock from 3.12.0 to 3.14.0 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest from 7.3.1 to 8.1.1 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-mock from 3.12.0 to 3.14.0 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump flake8 from 6.0.0 to 7.0.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.8.3 to 3.8.4 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.4.12 to 4.14.2 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.28.11.5 to 2.31.0.20240311 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump tox from 4.4.12 to 4.14.2 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-mock from 3.10.0 to 3.14.0 in /memory by [@dependabot](https://github.com/dependabot) in * build(local): Remove Pytorch dev dependency by [@jesse-c](https://github.com/jesse-c) in * build(docs): Add Dependabot by [@jesse-c](https://github.com/jesse-c) in * docs: New structure and Local content by [@jesse-c](https://github.com/jesse-c) in * build(deps): bump notebook from 7.1.1 to 7.1.2 in /docs by [@dependabot](https://github.com/dependabot) in * build: Use wheel for distribution by [@jesse-c](https://github.com/jesse-c) in * build(docs): Ignore common virtualenv directories by [@jesse-c](https://github.com/jesse-c) in * docs: Continue restructure and add reference for Local by [@jesse-c](https://github.com/jesse-c) in * docs(local): Add inference requests and backends' model settings by [@jesse-c](https://github.com/jesse-c) in * docs: Add analytics, fix quoted sidebar, and add versions by [@jesse-c](https://github.com/jesse-c) in * fix(local): Allow Pydantic models' validators reuse by [@jesse-c](https://github.com/jesse-c) in * docs: Add ReadTheDocs config by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump types-requests from 2.31.0.20240311 to 2.31.0.20240403 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240311 to 2.31.0.20240403 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240311 to 2.31.0.20240403 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240311 to 2.31.0.20240403 in /memory by [@dependabot](https://github.com/dependabot) in * refactor: Align versioning by [@jesse-c](https://github.com/jesse-c) in * fix: upgrade to cuda 12.1.1 by [@sakoush](https://github.com/sakoush) in * OpenAI api default generation kwargs by [@RobertSamoilescu](https://github.com/RobertSamoilescu) in * fix(local): Don't delete `parameters.extra` by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump pytest-cases from 3.8.4 to 3.8.5 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.8.4 to 3.8.5 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.8.4 to 3.8.5 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump pytest-cases from 3.8.4 to 3.8.5 in /deepspeed by [@dependabot](https://github.com/dependabot) in * feat(local): Allow `kwargs` as serialised JSON string by [@jesse-c](https://github.com/jesse-c) in * test(local): Use parameterisation by [@jesse-c](https://github.com/jesse-c) in * feat(api): Allow `llm_parameters` as serialised JSON string by [@jesse-c](https://github.com/jesse-c) in * build(local): Bump LLMIS version to `0.0.1rc1` by [@jesse-c](https://github.com/jesse-c) in * build(ci): Align version check by [@jesse-c](https://github.com/jesse-c) in * build(ci): Improve space saving when build images by [@jesse-c](https://github.com/jesse-c) in * build(ci): Use correct names and SSH agent by [@jesse-c](https://github.com/jesse-c) in * build(deps-dev): bump types-requests from 2.31.0.20240403 to 2.31.0.20240406 in /memory by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240403 to 2.31.0.20240406 in /api by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240403 to 2.31.0.20240406 in /local by [@dependabot](https://github.com/dependabot) in * build(deps-dev): bump types-requests from 2.31.0.20240403 to 2.31.0.20240406 in /deepspeed by [@dependabot](https://github.com/dependabot) in * build(ci): Migrate from Node 16 by [@jesse-c](https://github.com/jesse-c) in **Full Changelog**: [Changes](https://github.com/SeldonIO/llm-runtimes/tree/0.3.0) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.seldon.ai/llm-module/resources/changelog.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.