baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
1434 commits 1 branch 0 tags 2.1 MiB
Commit graph

139 commits

Author SHA1 Message Date
Andrei
fb762a6041
Add speculative decoding (#1120) 
* Add draft model param to llama class, implement basic prompt lookup decoding draft model

* Use samplingcontext for sampling

* Use 1d array

* Use draft model for sampling

* Fix dumb mistake

* Allow for later extensions to the LlamaDraftModel api

* Cleanup

* Adaptive candidate prediction

* Update implementation to match hf transformers

* Tuning

* Fix bug where last token was not used for ngram prediction

* Remove heuristic for num_pred_tokens (no benefit)

* fix: n_candidates bug.

* Add draft_model_num_pred_tokens server setting

* Cleanup

* Update README
 2024-01-31 14:08:14 -05:00
Andrei
da003d8768
Automatically set chat format from gguf (#1110) 
* Use jinja formatter to load chat format from gguf

* Fix off-by-one error in metadata loader

* Implement chat format auto-detection
 2024-01-29 14:22:23 -05:00
Andrei Betlen
cde7514c3d feat(server): include llama-cpp-python version in openapi spec 2024-01-25 11:23:18 -05:00
Andrei Betlen
24f39454e9 fix: pass chat handler not chat formatter for huggingface autotokenizer and tokenizer_config formats. 2024-01-21 18:38:04 -05:00
Andrei Betlen
141293a75b Fix python3.8 support 2024-01-19 08:17:49 -05:00
Andrei Betlen
b8fc1c7d83 feat: Add ability to load chat format from huggingface autotokenizer or tokenizer_config.json files. 2024-01-18 21:21:37 -05:00
Andrei Betlen
48c3b77e6f Offload KQV by default 2024-01-18 11:08:57 -05:00
Kyle Mistele
9c36688b33
fix(cli): allow passing n_ctx=0 to openAI API server args to use model n_ctx_train field per #1015 (#1093) 2024-01-16 18:54:06 -05:00
anil
cfb7da98ed
Support Accept text/event-stream in chat and completion endpoints, resolves #1083 (#1088) 
Co-authored-by: Anil Pathak <anil@heyday.com>
Co-authored-by: Andrei Betlen <abetlen@gmail.com>
 2024-01-16 12:52:52 -05:00
Andrei Betlen
84615adbc6 Add split_mode option. Closes #1085 2024-01-15 12:49:20 -05:00
Phil H
76aafa6149
Implement GGUF metadata KV overrides (#1011) 
* Implement GGUF metadata overrides

* whitespace fix

* Fix kv overrides.

* Fix pointer and pickle

* Match llama.cpp kv_overrides cli argument

---------

Co-authored-by: Andrei <abetlen@gmail.com>
 2024-01-15 12:29:29 -05:00
Andrei Betlen
522aecb868 docs: add server config docs 2023-12-22 14:37:24 -05:00
swg
4b01a873ef
server: Support none defaulting to infinity for completions (#111) 
* Support defaulting to infinity or -1 for chat completions

* Check if completion_tokens is none in error handler.

* fix: max_tokens in create completion should match openai spec

* Fix __call__

---------

Co-authored-by: Andrei Betlen <abetlen@gmail.com>
 2023-12-22 14:05:13 -05:00
Dave
12b7f2f4e9
[Feat] Multi model support (#931) 
* Update Llama class to handle chat_format & caching

* Add settings.py

* Add util.py & update __main__.py

* multimodel

* update settings.py

* cleanup

* delete util.py

* Fix /v1/models endpoint

* MultiLlama now iterable, app check-alive on "/"

* instant model init if file is given

* backward compability

* revert model param mandatory

* fix error

* handle individual model config json

* refactor

* revert chathandler/clip_model changes

* handle chat_handler in MulitLlama()

* split settings into server/llama

* reduce global vars

* Update LlamaProxy to handle config files

* Add free method to LlamaProxy

* update arg parsers & install server alias

* refactor cache settings

* change server executable name

* better var name

* whitespace

* Revert "whitespace"

This reverts commit bc5cf51c64a95bfc9926e1bc58166059711a1cd8.

* remove exe_name

* Fix merge bugs

* Fix type annotations

* Fix type annotations

* Fix uvicorn app factory

* Fix settings

* Refactor server

* Remove formatting fix

* Format

* Use default model if not found in model settings

* Fix

* Cleanup

* Fix

* Fix

* Remove unnused CommandLineSettings

* Cleanup

* Support default name for copilot-codex models

---------

Co-authored-by: Andrei Betlen <abetlen@gmail.com>
 2023-12-22 05:51:25 -05:00
docmeth02
33cc623346
Implement openai api compatible authentication (#1010) 2023-12-21 13:44:49 -05:00
Andrei Betlen
095c650006 Add offload_kqv option to llama and server 2023-12-18 15:36:09 -05:00
Brandon Roberts
62944df142
Bugfix: Remove f16_kv, add offload_kqv field (#1019) 
F16_KV appears to have been removed here: af99c6fbfc

This addresses two issues:

 - #995 which just requests to add the KV cache offloading param
 - #1006 a NULL ptr exception when using the embeddings (introduced by
   leaving f16_kv in the fields struct)
 2023-12-18 14:27:11 -05:00
Radoslav Gerganov
8e44a32075
Add support for running the server with SSL (#994) 2023-12-11 20:47:11 -05:00
Andrei Betlen
1a7bf2037b docs: Update openapi endpoint names 2023-11-24 03:39:29 -05:00
Andrei Betlen
128dc4731f Fix #569 2023-11-21 04:39:05 -05:00
Andrei Betlen
7a3f87846b Format 2023-11-21 04:02:20 -05:00
Andrei Betlen
07e47f55ba Add support for logit_bias outside of server api. Closes #827 2023-11-21 03:59:46 -05:00
TK-Master
b8438f70b5
Added support for min_p (#921) 
* Added support for min_p

My small contribution to this great project.

Ref: https://github.com/ggerganov/llama.cpp/pull/3841

Closes: https://github.com/abetlen/llama-cpp-python/issues/911

* Fix for negative temp (sample_softmax)
 2023-11-20 23:21:33 -05:00
Andrei Betlen
e7962d2c73 Fix: default max_tokens matches openai api (16 for completion, max length for chat completion) 2023-11-10 02:49:27 -05:00
Andrei Betlen
ca4cb88351 Fix destructor NoneType is not callable error 2023-11-08 11:05:45 -05:00
Andrei Betlen
b30b9c338b Add JSON mode support. Closes #881 2023-11-08 00:07:16 -05:00
Andrei Betlen
86aeb9f3a1 Add seed parameter support for completion and chat_completion requests. Closes #884 2023-11-07 23:37:28 -05:00
Damian Stewart
aab74f0b2b
Multimodal Support (Llava 1.5) (#821) 
* llava v1.5 integration

* Point llama.cpp to fork

* Add llava shared library target

* Fix type

* Update llama.cpp

* Add llava api

* Revert changes to llama and llama_cpp

* Update llava example

* Add types for new gpt-4-vision-preview api

* Fix typo

* Update llama.cpp

* Update llama_types to match OpenAI v1 API

* Update ChatCompletionFunction type

* Reorder request parameters

* More API type fixes

* Even More Type Updates

* Add parameter for custom chat_handler to Llama class

* Fix circular import

* Convert to absolute imports

* Fix

* Fix pydantic Jsontype bug

* Accept list of prompt tokens in create_completion

* Add llava1.5 chat handler

* Add Multimodal notebook

* Clean up examples

* Add server docs

---------

Co-authored-by: Andrei Betlen <abetlen@gmail.com>
 2023-11-07 22:48:51 -05:00
Andrei Betlen
df9362eeea Update llama.cpp 2023-11-03 11:34:50 -04:00
Andrei
3af7b21ff1
Add functionary support (#784) 
* Add common grammars and json-schema-to-grammar utility function from llama.cpp

* Pass functions to format function

* Add basic functionary formatting

* Add LlamaChatHandler for more complex chat use cases

* Add function calling example notebook

* Add support for regular chat completions alongside function calling
 2023-11-03 02:12:14 -04:00
Andrei Betlen
fa83cc5f9c Update llama.cpp 
Fix build examples

Exclude examples directory

Revert cmake changes

Try actions/checkout@v4

Try to update submodules

Revert

Update llama.cpp

Fix build examples

Exclude examples directory

Revert cmake changes

Try actions/checkout@v4

Try to update submodules

Revert
 2023-11-02 14:28:15 -04:00
Antoine Lizee
4d4e0f11e2 fix: tokenization of special characters: (#850) 
It should behave like llama.cpp, where most out of the box usages
treat special characters accordingly
 2023-11-02 14:28:14 -04:00
David Ponce
3fc9147218
Iterate over tokens that should be biased rather than the entire vocabulary. (#851) 2023-11-01 18:53:47 -04:00
Daniel Thuerck
5f8f369d1b
Pass-Through grammar parameter in web server. (#855) Closes #778 2023-11-01 18:51:12 -04:00
Xiaoyu Kevin Hu
a315128d66
update value check for n_gpu_layers field (#826) 2023-10-18 18:25:25 -04:00
Andrei Betlen
d6a130a052 Print traceback on server error 2023-10-10 15:56:04 -04:00
Andrei Betlen
5ef5280ef9 Log server exceptions to stdout 2023-09-30 19:13:36 -04:00
Andrei Betlen
d9bce17794 Update server params 2023-09-29 19:59:12 -04:00
Viacheslav/Slava Tradunsky
3d5e5b1c04
Adds openai-processing-ms response header (#748) 2023-09-25 13:55:58 -04:00
Andrei Betlen
b047b3034e Remove confusing helpstring from server cli args. Closes #719 2023-09-15 14:09:43 -04:00
Andrei Betlen
0449d29b9f Fix boolean env vars and cli arguments 2023-09-13 23:09:57 -04:00
earonesty
58a6e42cc0
Update app.py (#705) 2023-09-13 23:01:34 -04:00
Andrei Betlen
f4090a0bb2 Add numa support, low level api users must now explicitly call llama_backend_init at the start of their programs. 2023-09-13 23:00:43 -04:00
Andrei Betlen
c999325e8e Fix boolean cli flags 2023-09-13 22:56:10 -04:00
Andrei Betlen
4daf77e546 Format 2023-09-13 21:23:23 -04:00
Andrei Betlen
2920c4bf7e Update server params. Added lora_base, lora_path, low_vram, and main_gpu. Removed rms_norm_eps and n_gqa (deprecated in llama.cpp) 2023-09-13 21:23:13 -04:00
Andrei Betlen
c4c440ba2d Fix tensor_split cli option 2023-09-13 20:00:42 -04:00
Andrei Betlen
759405c84b Fix issue with Literal and Optional cli arguments not working. Closes #702 2023-09-13 18:06:12 -04:00
Devrim
da9df78db0
Add X-Request-ID request header for mirroring custom IDs. (#703) 2023-09-13 16:18:31 -04:00
Andrei Betlen
1910793f56 Merge branch 'main' into v0.2-wip 2023-09-12 16:43:32 -04:00