Andrei fb762a6041 Add speculative decoding (#1120) ... * Add draft model param to llama class, implement basic prompt lookup decoding draft model * Use samplingcontext for sampling * Use 1d array * Use draft model for sampling * Fix dumb mistake * Allow for later extensions to the LlamaDraftModel api * Cleanup * Adaptive candidate prediction * Update implementation to match hf transformers * Tuning * Fix bug where last token was not used for ngram prediction * Remove heuristic for num_pred_tokens (no benefit) * fix: n_candidates bug. * Add draft_model_num_pred_tokens server setting * Cleanup * Update README		2024-01-31 14:08:14 -05:00
..
test_grammar.py	fix: from_json_schema oneof/anyof bug. Closes #1097	2024-01-21 19:06:53 -05:00
test_llama.py	Fix logits_to_logprobs for 2-D and 3-D logits (#1002)	2023-12-16 18:59:26 -05:00
test_llama_chat_format.py	Add chat format test.	2024-01-29 00:59:01 -05:00
test_llama_speculative.py	Add speculative decoding (#1120)	2024-01-31 14:08:14 -05:00