llama.cpp/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.2.75]

• feat: Update llama.cpp to ggerganov/llama.cpp@13ad16af12
• fix: segfault for models without eos / bos tokens by @abetlen in d99a6ba607
• feat: add MinTokensLogitProcessor and min_tokens argument to server by @twaka in #1333
• misc: Remove unnecessary metadata lookups by @CISC in #1448

[0.2.74]

• feat: Update llama.cpp to ggerganov/llama.cpp@b228aba91a
• fix: Enable CUDA backend for llava by @abetlen in 7f59856fa6
• docs: Fix typo in README.md by @yupbank in #1444

[0.2.73]

• feat: Update llama.cpp to ggerganov/llama.cpp@25c6e82e7a
• fix: Clear kv cache at beginning of image chat formats to avoid bug when image is evaluated first by @abetlen in ac55d0a175

[0.2.72]

• fix(security): Remote Code Execution by Server-Side Template Injection in Model Metadata by @retr0reg in b454f40a9a
• fix(security): Update remaining jinja chat templates to use immutable sandbox by @CISC in #1441

[0.2.71]

• feat: Update llama.cpp to ggerganov/llama.cpp@911b3900dd
• fix: Make leading bos_token optional for image chat formats, fix nanollava system message by @abetlen in 77122638b4
• fix: free last image embed in llava chat handler by @abetlen in 3757328b70

[0.2.70]

• feat: Update llama.cpp to ggerganov/llama.cpp@c0e6fbf8c3
• feat: fill-in-middle support by @CISC in #1386
• fix: adding missing args in create_completion for functionary chat handler by @skalade in #1430
• docs: update README.md @eltociear in #1432
• fix: chat_format log where auto-detected format prints None by @balvisio in #1434
• feat(server): Add support for setting root_path by @abetlen in 0318702cdc
• feat(ci): Add docker checks and check deps more frequently by @Smartappli in #1426
• fix: detokenization case where first token does not start with a leading space by @noamgat in #1375
• feat: Implement streaming for Functionary v2 + Bug fixes by @jeffrey-fong in #1419
• fix: Use memmove to copy str_value kv_override by @abetlen in 9f7a85571a
• feat(server): Remove temperature bounds checks for server by @abetlen in 0a454bebe6
• fix(server): Propagate flash_attn to model load by @dthuerck in #1424

[0.2.69]

• feat: Update llama.cpp to ggerganov/llama.cpp@6ecf3189e0
• feat: Add llama-3-vision-alpha chat format by @abetlen in 31b1d95a6c
• fix: Change default verbose value of verbose in image chat format handlers to True to match Llama by @abetlen in 4f01c452b6
• fix: Suppress all logs when verbose=False, use hardcoded fileno's to work in colab notebooks by @abetlen in f116175a5a
• fix: UTF-8 handling with grammars by @jsoma in #1415

[0.2.68]

• feat: Update llama.cpp to ggerganov/llama.cpp@77e15bec62
• feat: Add option to enable flash_attn to Lllama params and ModelSettings by @abetlen in 22d77eefd2
• fix(ci): Fix build-and-release.yaml by @Smartappli in #1413

[0.2.67]

• fix: Ensure image renders before text in chat formats regardless of message content order by @abetlen in 3489ef09d3
• fix(ci): Fix bug in use of upload-artifact failing to merge multiple artifacts into a single release by @abetlen in d03f15bb73

[0.2.66]

• feat: Update llama.cpp to ggerganov/llama.cpp@8843a98c2b
• feat: Generic Chat Formats, Tool Calling, and Huggingface Pull Support for Multimodal Models (Obsidian, LLaVA1.6, Moondream) by @abetlen in #1147
• ci(fix): Workflow actions updates and fix arm64 wheels not included in release by @Smartappli in #1392
• ci: Add support for pre-built cuda 12.4.1 wheels by @Smartappli in #1388
• feat: Add support for str type kv_overrides by @abetlen in a411612b38
• fix: Functionary bug fixes by @jeffrey-fong in #1385
• examples: fix quantize example by @iyubondyrev in #1387
• ci: Update dependabot.yml by @Smartappli in #1391

[0.2.65]

• feat: Update llama.cpp to ggerganov/llama.cpp@46e12c4692
• feat: Allow for possibly non-pooled embeddings by @iamlemec in #1380

[0.2.64]

• feat: Update llama.cpp to ggerganov/llama.cpp@4e96a812b3
• feat: Add llama-3 chat format by @andreabak in #1371
• feat: Use new llama_token_is_eog in create_completions by @abetlen in d40a250ef3
• feat(server): Provide ability to dynamically allocate all threads if desired using -1 by @sean-bailey in #1364
• ci: Build arm64 wheels by @gaby in 611781f531
• fix: Update scikit-build-core build dependency avoid bug in 0.9.1 by @evelkey in #1370

[0.2.63]

• feat: Update llama.cpp to ggerganov/llama.cpp@0e4802b2ec
• feat: Add stopping_criteria to ChatFormatter, allow stopping on arbitrary token ids, fixes llama3 instruct by @abetlen in cc81afebf0

[0.2.62]

• feat: Update llama.cpp to ggerganov/llama.cpp@3b8f1ec4b1
• feat: update grammar schema converter to match llama.cpp by @themrzmaster in #1353
• feat: add disable_ping_events flag by @khimaros in #1257
• feat: Make saved state more compact on-disk by @tc-wolf in #1296
• feat: Use all available CPUs for batch processing by @ddh0 in #1345

[0.2.61]

• feat: Update llama.cpp to ggerganov/llama.cpp@ba5e134e07
• fix: pass correct type to chat handlers for chat completion logprobs by @abetlen in bb65b4d764
• feat: Add support for yaml based server configs by @abetlen in 060bfa64d5
• feat: Add typechecking for ctypes structure attributes by @abetlen in 1347e1d050

[0.2.60]

• feat: Update llama.cpp to ggerganov/llama.cpp@75cd4c7729
• fix: Always embed metal library by @abetlen in b3bfea6dbf
• fix: missing logprobs in response, incorrect response type for functionary by @abetlen in 1ae3abbcc3
• fix(docs): incorrect tool_choice example by @CISC in #1330

[0.2.59]

• feat: Update llama.cpp to ggerganov/llama.cpp@ba0c7c70ab
• feat: Binary wheels for CPU, CUDA (12.1 - 12.3), Metal by @abetlen, @jllllll, and @oobabooga in #1247
• fix: segfault when logits_all=False by @abetlen in 8649d7671b
• fix: last tokens passing to sample_repetition_penalties function by @ymikhailov in #1295

[0.2.58]

• feat: Update llama.cpp to ggerganov/llama.cpp@ba0c7c70ab
• feat: add support for KV cache quantization options by @Limour-dev in #1307
• feat: Add logprobs support to chat completions by @windspirit95 in #1311
• fix: set LLAMA_METAL_EMBED_LIBRARY=on on MacOS arm64 by @bretello in #1289
• feat: Add tools/functions variables to Jinja2ChatFormatter, add function response formatting for all simple chat formats by @CISC in #1273
• fix: Changed local API doc references to hosted by by @lawfordp2017 in #1317

[0.2.57]

• feat: Update llama.cpp to ggerganov/llama.cpp@ac9ee6a4ad
• fix: set default embedding pooling type to unspecified by @abetlen in 4084aabe86
• fix: Fix and optimize functionary chat handler by @jeffrey-fong in #1282
• fix: json mode for basic chat formats by @abetlen in 20e6815252

[0.2.56]

• feat: Update llama.cpp to ggerganov/llama.cpp@c2101a2e90
• feat(server): Add endpoints for tokenize, detokenize and count tokens by @felipelo in #1136
• feat: Switch embed to llama_get_embeddings_seq by @iamlemec in #1263
• fix: Fixed json strings grammar by blacklisting character control set by @ExtReMLapin in d02a9cf16f
• fix: Check for existence of clip model path by @kejcao in #1264

[0.2.55]

• feat: Update llama.cpp to ggerganov/llama.cpp@9731134296
• docs: fix small typo in README: 'model know how' -> 'model knows how' by @boegel in #1244

[0.2.54]

• feat: Update llama.cpp to ggerganov/llama.cpp@cb49e0f8c9
• docs: fix typo in README.md embeddings example by @iamlemec in #1232

[0.2.53]

• feat: Update llama.cpp to ggerganov/llama.cpp@cb49e0f8c9
• fix: eos/bos_token set correctly for Jinja2ChatFormatter and automatic chat formatter by @CISC in #1230

[0.2.52]

• feat: Update llama.cpp to ggerganov/llama.cpp@a33e6a0d2a
• fix: Llava15ChatHandler (this function takes at least 4 arguments) by @abetlen in 8383a9e562

[0.2.51]

• feat: Update llama.cpp to ggerganov/llama.cpp@c393733988
• fix: Restore type hints for low-level api by @abetlen in 19234aa0db

[0.2.50]

• docs: Update Functionary OpenAI Server Readme by @jeffrey-fong in #1193
• fix: LlamaHFTokenizer now receives pre_tokens by @abetlen in 47bad30dd7

[0.2.49]

• fix: module 'llama_cpp.llama_cpp' has no attribute 'c_uint8' in Llama.save_state by @abetlen in db776a885c
• feat: Auto detect Mixtral's slightly different format by @lukestanley in #1214

[0.2.48]

• feat: Update llama.cpp to ggerganov/llama.cpp@15499eb942
• feat: Add Google's Gemma formatting via chat_format="gemma" by @alvarobartt in #1210
• feat: support minItems/maxItems in JSON grammar converter by @nopperl in 3921e10770
• fix: Update from_pretrained defaults to match hf_hub_download and pull to local cache folder by @abetlen in e6d6260a91
• fix: Raise exceptions when llama model or context fails to load by @abetlen in dd22010e85
• docs: Update README.md to fix pip install llama cpp server by @audip in #1187

[0.2.47]

• feat: Update llama.cpp to ggerganov/llama.cpp@973053d8b0

[0.2.46]

• feat: Update llama.cpp to ggerganov/llama.cpp@ba2135ccae
• feat: Pull models directly from huggingface by @abetlen in #1206
• feat(low-level-api): Improve API static type-safety and performance. Low level api functions are positional args only now. by @abetlen in #1205

[0.2.45]

• feat: Update llama.cpp to ggerganov/llama.cpp@89febfed93

[0.2.44]

• feat: Update llama.cpp to ggerganov/llama.cpp@4524290e87
• fix: create_embedding broken response for input type str by @abetlen in 0ce66bc080
• fix: Use '\n' seperator for EventSourceResponse by @khimaros in #1188
• fix: Incorporate embedding pooling layer fixes by @iamlemec in #1194

[0.2.43]

• feat: Update llama.cpp to ggerganov/llama.cpp@8084d55440
• feat: Support batch embeddings by @iamlemec in #1186
• fix: submodule kompute is not included in sdist by @abetlen in 7dbbfdecad
• fix: fix: Update openbuddy prompt format by @abetlen in 07a783779a

[0.2.42]

• feat: Update llama.cpp to ggerganov/llama.cpp@ea9c8e1143
• fix: sample idx off-by-one error for logit_processors by @lapp0 in #1179
• fix: chat formatting bugs in chatml-function-calling by @abetlen in 4b0e3320bd and 68fb71b6a2

[0.2.41]

• feat: Update llama.cpp to ggerganov/llama.cpp@895407f31b
• fix: Don't change order of json schema object properties in generated grammar unless prop_order is passed by @abetlen in d1822fed6b

[0.2.40]

• feat: Update llama.cpp to ggerganov/llama.cpp@3bdc4cd0f5
• feat: Generic chatml Function Calling using chat_format="chatml-function-calling"` by @abetlen in #957
• fix: Circular dependancy preventing early Llama object free by @notwa in #1176
• docs: Set the correct command for compiling with syscl support by @akarshanbiswas in #1172
• feat: use gpu backend for clip if available by @iamlemec in #1175

[0.2.39]

• feat: Update llama.cpp to ggerganov/llama.cpp@b08f22c882
• fix: Fix destructor logging bugs by using llama_log_callback to avoid suppress_stdout_stderr by @abetlen in 59760c85ed

[0.2.38]

• feat: Update llama.cpp to ggerganov/llama.cpp@1cfb5372cf
• feat: Add speculative decoding by @abetlen in #1120
• fix: Pass raise_exception and add_generation_prompt to jinja2 chat template by @abetlen in 078cca0361

[0.2.37]

• feat: Update llama.cpp to ggerganov/llama.cpp@fea4fd4ba7
• feat: Automatically set chat format from gguf by @abetlen in #1110

[0.2.36]

• feat: Update llama.cpp to ggerganov/llama.cpp@2aed77eb06
• feat: Add mistral instruct chat format as "mistral-instruct" by @Rafaelblsilva in #799

[0.2.35]

• feat: Update llama.cpp to ggerganov/llama.cpp@d2f650cb5b

[0.2.34]

• feat: Update llama.cpp to ggerganov/llama.cpp@6db2b41a76
• feat: Add json schema mode by @abetlen in #1122

[0.2.33]

• feat: Update llama.cpp to ggerganov/llama.cpp@faa3526a1e
• feat(server): include llama-cpp-python version in openapi spec by @abetlen in cde7514c3d
• fix: use both eos and bos tokens as stop sequences for hf-tokenizer-config chat format. by @abetlen in 5b982d0f8c
• fix: GGUF metadata KV overrides, re #1011 by @phiharri in #1116
• fix: llama_log_set should be able to accept null pointer by @abetlen in c970d41a85

[0.2.32]

• feat: Update llama.cpp to ggerganov/llama.cpp@504dc37be8
• fix: from_json_schema oneof/anyof bug by @jndiogo in d3f5528ca8
• fix: pass chat handler not chat formatter for huggingface autotokenizer and tokenizer_config formats by @abetlen in 24f39454e9
• feat: Add add_generation_prompt option for jinja2chatformatter by @abetlen in 7f3209b1eb
• feat: Add Jinja2ChatFormatter by @abetlen in be09318c26
• feat: Expose gguf model metadata in metadata property by @abetlen in 5a34c57e54

[0.2.31]

• feat: Update llama.cpp to ggerganov/llama.cpp@a5cacb22b2
• fix: Mirostat sampling now passes correct type to ctypes and tracks state during generation by @abetlen in 3babe3512c
• fix: Python3.8 support in server by @abetlen in 141293a75b

[0.2.30]

• feat: Update llama.cpp to ggerganov/llama.cpp@57e2a7a52a
• feat(server): Add ability to load chat format from huggingface autotokenizer or tokenizer_config.json files by @abetlen in b8fc1c7d83
• feat: Integration of Jinja2 Templating for chat formats by @teleprint-me in #875
• fix: Offload KQV by default by @abetlen in 48c3b77e6f
• fix: Support Accept text/event-stream in chat and completion endpoints, resolves #1083 by @aniljava in #1088
• fix(cli): allow passing n_ctx=0 to openAI API server args to use model n_ctx_train field per #1015 by @K-Mistele in #1093

[0.2.29]

• feat: Update llama.cpp to ggerganov/llama.cpp@4483396751
• feat: Add split_mode option by @abetlen in 84615adbc6
• feat: Implement GGUF metadata KV overrides by @phiharri in #1011
• fix: Avoid "LookupError: unknown encoding: ascii" when open() called in a destructor by @yieldthought in #1012
• fix: Fix low_level_api_chat_cpp example to match current API by @aniljava in #1086
• fix: Fix Pydantic model parsing by @DeNeutoy in #1087

[0.2.28]

• feat: Update llama.cpp to ggerganov/llama.cpp@6efb8eb30e
• feat: Add ability to pass in penalize_nl param by @shankinson in #1068
• fix: print_grammar to stderr by @turian in #1052

[0.2.27]

• feat: Update llama.cpp to ggerganov/llama.cpp@b3a7c20b5c
• feat: Add saiga chat format by @femoiseev in #1050
• feat: Added chatglm3 chat format by @xaviviro in #1059
• fix: Correct typo in README.md by @qeleb in (#1058)

[0.2.26]

• feat: Update llama.cpp to ggerganov/llama.cpp@f6793491b5

[0.2.25]

• feat(server): Multi model support by @D4ve-R in #931
• feat(server): Support none defaulting to infinity for completions by @swg in #111
• feat(server): Implement openai api compatible authentication by @docmeth2 in #1010
• fix: text_offset of multi-token characters by @twaka in #1037
• fix: ctypes bindings for kv override by @phiharri in #1011
• fix: ctypes definitions of llama_kv_cache_view_update and llama_kv_cache_view_free. by @e-c-d in #1028

[0.2.24]

• feat: Update llama.cpp to ggerganov/llama.cpp@0e18b2e7d0
• feat: Add offload_kqv option to llama and server by @abetlen in 095c650006
• feat: n_ctx=0 now uses the n_ctx_train of the model by @DanieleMorotti in #1015
• feat: logits_to_logprobs supports both 2-D and 3-D logits arrays by @kddubey in #1002
• fix: Remove f16_kv, add offload_kqv fields in low level and llama apis by @brandonrobertz in #1019
• perf: Don't convert logprobs arrays to lists by @kddubey in #1021
• docs: Fix README.md functionary demo typo by @evelynmitchell in #996
• examples: Update low_level_api_llama_cpp.py to match current API by @jsoma in #1023

[0.2.23]

• Update llama.cpp to ggerganov/llama.cpp@948ff137ec
• Add qwen chat format by @yhfgyyf in #1005
• Add support for running the server with SSL by @rgerganov in #994
• Replace logits_to_logprobs implementation with numpy equivalent to llama.cpp by @player1537 in #991
• Fix UnsupportedOperation: fileno in suppress_stdout_stderr by @zocainViken in #961
• Add Pygmalion chat format by @chiensen in #986
• README.md multimodal params fix by @zocainViken in #967
• Fix minor typo in README by @aniketmaurya in #958

[0.2.22]

• Update llama.cpp to ggerganov/llama.cpp@8a7b2fa528
• Fix conflict with transformers library by kddubey in #952

[0.2.21]

• Update llama.cpp to ggerganov/llama.cpp@64e64aa255
• Make building llava optional by setting CMAKE_ARGS="-DLLAVA_BUILD=OFF" and using LLAVA_CPP_LIB to specify alternative path to shared library by @abetlen in e3941d9c67

[0.2.20]

• Update llama.cpp to ggerganov/llama.cpp@b38a16dfcf
• Add zephyr chat format by @fakerybakery in #938
• Add baichuan chat format by @caiyesd in #938
• Add baichuan-2 chat format by @caiyesd in #936
• Improve documentation for server chat formats by @jooray in #934
• Fix typo in README by @antonvice in 940
• Fix typo in the Open Orca chat format by @gardner in #947

[0.2.19]

• Update llama.cpp to ggerganov/llama.cpp@0b871f1a04
• Fix #569: stop parameter in chat completion api should accept str by @abetlen in 128dc4731f
• Document server host and port parameters by @jamesbraza in #768
• Do not set grammar to None when initializing LlamaGrammar by @mthuurne in #834
• Add mistrallite, intel, and openchat formats by @fakerybakery in #927
• Add support for min_p parameter by @tk-master in #921
• Fix #929: tokenizer adding leading space when generating from empty prompt by @abetlen in a34d480141
• Fix low level api example by @zocainViken in #925
• Fix missing package in openblas docker image by @ZisisTsatsas in #920

[0.2.18]

• Update llama.cpp to ggerganov/llama.cpp@6bb4908a17

[0.2.17]

• Update llama.cpp to ggerganov/llama.cpp@df9d1293de
• Hotfix: Set CUDA_ARCHITECTURES=OFF for llava_shared target on Windows by @abetlen in 4388f33414

[0.2.16]

• Update llama.cpp to ggerganov/llama.cp@a75fa576ab
• Add set_seed to Llama class by @abetlen in fd41ed3a90
• Fix server doc arguments by @kjunggithub in #892
• Fix response_format handler in llava chat handler by @abetlen in b62c449839
• Fix default max_tokens, chat completion is now unlimited (to context length) and completion is 16 tokens to match OpenAI defaults by @abetlen in e7962d2c73
• Fix json_schema_to_gbnf helper so that it takes a json schema string as input instead by @abetlen in faeae181b1
• Add support for $ref and $def in json_schema_to_gbnf to handle more complex function schemas by @abetlen in 770df34436
• Update functionary chat handler for new OpenAI api by abetlen in 1b376c62b7
• Fix add default stop sequence to chatml chat format by @abetlen in b84d76a844
• Fix sampling bug when logits_all=False by @abetlen in 6f0b0b1b84

[0.2.15]

• Update llama.cpp to ggerganov/llama.cpp@0a7c980b6f
• Add support for Llava1.5 multimodal models by @damian0815 and @abetlen in #821
• Update OpenAI API compatibility to match dev day update by @abetlen in #821
• Add seed parameter to completion and chat_completion functions of Llama class by @abetlen in 86aeb9f3a1
• Add JSON mode support to constrain chat completion to JSON objects by @abetlen in b30b9c338b

[0.2.14]

• Update llama.cpp to ggerganov/llama.cpp@f0b30ef7dc
• Add support for Huggingface Autotokenizer Chat Formats by @bioshazard and @abetlen in #790 and bbffdaebaa
• Fix llama-2 chat format by @earonesty in #869
• Add support for functionary chat format by @abetlen in #784
• Migrate inference from deprecated llama_evalAPI to llama_batch and llama_decode by @abetlen in #795

[0.2.13]

• Update llama.cpp to ggerganov/llama.cpp@51b2fc11f7
• Fix name 'open' is not defined exception when deleting model by @abetlen in 011b95d7f3
• Fix tokenization of special characters by @antoine-lizee in #850

[0.2.12]

• Update llama.cpp to ggerganov/llama.cpp@50337961a6
• Fix missing n_seq_id in llama_batch by @NickAlgra in #842
• Fix for shared libraries on Windows that start with lib prefix by @sujeendran in #848
• Fix exception raised in __del__ when freeing models by @cebtenzzre in #846
• Performance improvement for logit bias by @zolastro in #851
• Fix suffix check arbitrary code execution bug by @mtasic85 in #854
• Fix typo in function_call parameter in llama_types.py by @akatora28 in #849
• Fix streaming not returning finish_reason by @gmcgoldr in #798
• Fix n_gpu_layers check to allow values less than 1 for server by @hxy9243 in #826
• Supppress stdout and stderr when freeing model by @paschembri in #803
• Fix llama2 chat format by @delock in #808
• Add validation for tensor_split size by @eric1932 #820
• Print stack trace on server error by @abetlen in d6a130a052
• Update docs for gguf by @johnccshen in #783
• Add chatml chat format by @abetlen in 305482bd41

[0.2.11]

• Fix bug in llama_model_params object has no attribute logits_all by @abetlen in d696251fbe

[0.2.10]

• Fix bug 'llama_model_params' object has no attribute 'embedding' by @abetlen in 42bb721d64

[0.2.9]

• Fix critical bug in pip installation of v0.2.8 due to .git directory in ac853e01e1

[0.2.8]

• Update llama.cpp to ggerganov/llama.cpp@40e07a60f9
• Add configurable chat formats by @abetlen in #711
• Fix rope scaling bug by @Josh-XT in #767
• Fix missing numa parameter in server by @abetlen in d9bce17794

[0.2.7]

• Update llama.cpp to ggerganov/llama.cpp@a98b1633d5
• Install required runtime dlls to package directory on windows by @abetlen in 8d75016549
• Add openai-processing-ms to server response header by @Tradunsky in #748
• Bump minimum version of scikit-build-core to 0.5.1 to fix msvc cmake issue by @abetlen in 1ed0f3ebe1
• Update llama_types.py to better match the openai api, old names are aliased to new ones by @abetlen in dbca136fea

[0.2.6]

• Update llama.cpp to 80291a1d02a07f7f66666fb576c5b1e75aa48b46

[0.2.5]

• Fix docker images missing starlette-context dependency by @abetlen in 2291798900
• Fix loading dll in Windows Isolation Containers by @abetlen in 8474665625
• Fix build issue on m1 macs by @abetlen in dbd3a6d1ed
• Update docs to gguf and add hw acceleration docs for server by @jasonacox in #688

[0.2.4]

• Add NUMA support. NOTE low level api users must call llama_backend_init at the start of their programs by abetlen in f4090a0bb2
• Fix tensor_split server cli argument by @abetlen in c4c440ba2d
• Made all Llama init parameters into keyword-only parameters by @abetlen in c8f9b8a734
• Added server params for low_vram, main_gpu, lora_base, and lora_path by @abetlen in 2920c4bf7e
• Removed server params for rms_norm_eps and n_gqa by @abetlen in 2920c4bf7e
• Fix boolean cli options by @abetlen in c999325e8e and 0449d29b9f
• Silence Pydantic Settings warnings about model_alias setting by @earonesty in #705

[0.2.3]

• Update llama.cpp to ggerganov/llama.cpp@71ca2fad7d
• Add X-Request-ID request header for mirroring custom IDs by @devrimcavusoglu in #703
• Add pyproject extra for scikit-build-core to ensure compatible pathspec version by @abetlen in 6cfc54284b
• Fix issue with Literal and Optional cli arguments not working by @abetlen in #702

[0.2.2]

• Fix bug in pip install of v0.2.1 due to scikit-build-core removing all .metal files in the source distribution (see #701)

[0.2.1]

• Fix bug in pip install of v0.2.0 due to .git folder being included in the source distribution (see #701)

[0.2.0]

• Migrated to scikit-build-core build system by @abetlen in #499
• Use numpy views for LogitsProcessor and StoppingCriteria instead of python lists by @abetlen in #499
• Drop support for end-of-life Python3.7 by @abetlen in #499
• Convert low level llama.cpp constants to use basic python types instead of ctypes types by @abetlen in #499

[0.1.85]

• Add llama_cpp.__version__ attribute by @janvdp in #684
• Fix low level api examples by @jbochi in #680

[0.1.84]

• Update llama.cpp

[0.1.83]

• Update llama.cpp

[0.1.82]

• Update llama.cpp

[0.1.81]

• Update llama.cpp

[0.1.80]

• Update llama.cpp

[0.1.79]

• GGUF Support (breaking change requiring new model format)

[0.1.78]

• Grammar based sampling via LlamaGrammar which can be passed to completions
• Make n_gpu_layers == -1 offload all layers

[0.1.77]

• (llama.cpp) Update llama.cpp add support for LLaMa 2 70B
• (server) Add temporary n_gqa and rms_norm_eps parameters required for LLaMa 2 70B

[0.1.76]

• (llama.cpp) Update llama.cpp add support for LLaMa 2 70B

[0.1.75]

• Update llama.cpp

[0.1.74]

• (server) OpenAI style error responses

[0.1.73]

• (server) Add rope parameters to server settings

[0.1.72]

• (llama.cpp) Update llama.cpp added custom_rope for extended context lengths

[0.1.71]

• (llama.cpp) Update llama.cpp

• (server) Fix several pydantic v2 migration bugs

[0.1.70]

• (Llama.create_completion) Revert change so that max_tokens is not truncated to context_size in create_completion
• (server) Fixed changed settings field names from pydantic v2 migration

[0.1.69]

• (server) Streaming requests can are now interrupted pre-maturely when a concurrent request is made. Can be controlled with the interrupt_requests setting.
• (server) Moved to fastapi v0.100.0 and pydantic v2
• (docker) Added a new "simple" image that builds llama.cpp from source when started.
• (server) performance improvements by avoiding unnecessary memory allocations during sampling

[0.1.68]

• (llama.cpp) Update llama.cpp

[0.1.67]

• Fix performance bug in Llama model by pre-allocating memory tokens and logits.
• Fix bug in Llama model where the model was not free'd after use.

[0.1.66]

• (llama.cpp) New model API

• Performance issue during eval caused by looped np.concatenate call

• State pickling issue when saving cache to disk

[0.1.65]

• (llama.cpp) Fix struct misalignment bug

[0.1.64]

• (llama.cpp) Update llama.cpp
• Fix docs for seed. Set -1 for random.

[0.1.63]

• (llama.cpp) Add full gpu utilisation in CUDA
• (llama.cpp) Add get_vocab
• (llama.cpp) Add low_vram parameter
• (server) Add logit_bias parameter

[0.1.62]

• Metal support working
• Cache re-enabled

[0.1.61]

• Fix broken pip installation

[0.1.60]

NOTE: This release was deleted due to a bug with the packaging system that caused pip installations to fail.

• Truncate max_tokens in create_completion so requested tokens doesn't exceed context size.
• Temporarily disable cache for completion requests

[v0.1.59]

• (llama.cpp) k-quants support
• (server) mirostat sampling parameters to server
• Support both .so and .dylib for libllama on MacOS

[v0.1.58]

• (llama.cpp) Metal Silicon support

[v0.1.57]

• (llama.cpp) OpenLlama 3B support

[v0.1.56]

• (misc) Added first version of the changelog
• (server) Use async routes
• (python-api) Use numpy for internal buffers to reduce memory usage and improve performance.
• (python-api) Performance bug in stop sequence check slowing down streaming.