027f7bc678
* Templates sometimes have BOS in them, remove duplicate * tokenize chat format prompts before completion This is to ensure that we don't duplicate any special tokens. Hopefully I amended the existing formats correctly? * updated comment * corrected a few * add some missing internals * proper bos/eos detection * just let tokenizer do the job * typo-- * align test with new response * changed to a warning * move to another PR * Use python warnings module --------- Co-authored-by: Andrei Betlen <abetlen@gmail.com> |
||
---|---|---|
.. | ||
server | ||
__init__.py | ||
_internals.py | ||
_logger.py | ||
_utils.py | ||
llama.py | ||
llama_cache.py | ||
llama_chat_format.py | ||
llama_cpp.py | ||
llama_grammar.py | ||
llama_speculative.py | ||
llama_tokenizer.py | ||
llama_types.py | ||
llava_cpp.py | ||
py.typed |