dbcf64cf07
* Support SPM infill * typo-- * one less layer of parenthesis necessary * new required internals * manually add bos/eos if model requires it * add bos even when unknown This is identical behaviour to llama.cpp I guess any model that doesn't use BOS is recent enough to have the add_bos_token metadata. * don't add bos/eos on non-infill pre-tokenized prompt * add tokenizer hack to remove leading space in suffix * I keep forgetting metadata are strings * check if bos exists * add example * add cls/sep instead of bos/eos for WPM vocab * simplify * color-code filtered suffix --------- Co-authored-by: Andrei Betlen <abetlen@gmail.com> |
||
|---|---|---|
| .. | ||
| server | ||
| __init__.py | ||
| _internals.py | ||
| _logger.py | ||
| _utils.py | ||
| llama.py | ||
| llama_cache.py | ||
| llama_chat_format.py | ||
| llama_cpp.py | ||
| llama_grammar.py | ||
| llama_speculative.py | ||
| llama_tokenizer.py | ||
| llama_types.py | ||
| llava_cpp.py | ||
| py.typed | ||