Andrei fb762a6041 Add speculative decoding (#1120) ... * Add draft model param to llama class, implement basic prompt lookup decoding draft model * Use samplingcontext for sampling * Use 1d array * Use draft model for sampling * Fix dumb mistake * Allow for later extensions to the LlamaDraftModel api * Cleanup * Adaptive candidate prediction * Update implementation to match hf transformers * Tuning * Fix bug where last token was not used for ngram prediction * Remove heuristic for num_pred_tokens (no benefit) * fix: n_candidates bug. * Add draft_model_num_pred_tokens server setting * Cleanup * Update README		2024-01-31 14:08:14 -05:00
..
server	Add speculative decoding (#1120)	2024-01-31 14:08:14 -05:00
__init__.py	Bump version	2024-01-30 12:27:27 -05:00
_internals.py	Automatically set chat format from gguf (#1110)	2024-01-29 14:22:23 -05:00
_utils.py	feat: Add ability to load chat format from huggingface autotokenizer or tokenizer_config.json files.	2024-01-18 21:21:37 -05:00
llama.py	Add speculative decoding (#1120)	2024-01-31 14:08:14 -05:00
llama_cache.py	Move cache classes to llama_cache submodule.	2024-01-17 09:09:12 -05:00
llama_chat_format.py	fix: Pass raise_exception and add_generation_prompt to jinja2 chat template	2024-01-31 08:42:21 -05:00
llama_cpp.py	Update llama.cpp	2024-01-31 10:41:42 -05:00
llama_grammar.py	fix: from_json_schema oneof/anyof bug. Closes #1097	2024-01-21 19:06:53 -05:00
llama_speculative.py	Add speculative decoding (#1120)	2024-01-31 14:08:14 -05:00
llama_types.py	Add json schema mode (#1122)	2024-01-27 16:52:18 -05:00
llava_cpp.py	Make building llava optional	2023-11-28 04:55:21 -05:00
py.typed	Add py.typed	2023-08-11 09:58:48 +02:00