llama.cpp/llama_cpp
Sigbjørn Skjæret 027f7bc678
fix: Avoid duplicate special tokens in chat formats ()
* Templates sometimes have BOS in them, remove duplicate

* tokenize chat format prompts before completion

This is to ensure that we don't duplicate any special tokens.

Hopefully I amended the existing formats correctly?

* updated comment

* corrected a few

* add some missing internals

* proper bos/eos detection

* just let tokenizer do the job

* typo--

* align test with new response

* changed to a warning

* move to another PR

* Use python warnings module

---------

Co-authored-by: Andrei Betlen <abetlen@gmail.com>
2024-06-04 10:15:41 -04:00
..
server fix: fix string value kv_overrides. Closes 2024-05-29 02:02:22 -04:00
__init__.py chore: Bump version 2024-06-04 00:49:24 -04:00
_internals.py fix: Avoid duplicate special tokens in chat formats () 2024-06-04 10:15:41 -04:00
_logger.py fix: Use llama_log_callback to avoid suppress_stdout_stderr 2024-02-05 21:52:12 -05:00
_utils.py fix: Suppress all logs when verbose=False, use hardcoded fileno's to work in colab notebooks. Closes Closes 2024-04-30 15:45:34 -04:00
llama.py fix: Avoid duplicate special tokens in chat formats () 2024-06-04 10:15:41 -04:00
llama_cache.py Move cache classes to llama_cache submodule. 2024-01-17 09:09:12 -05:00
llama_chat_format.py fix: Avoid duplicate special tokens in chat formats () 2024-06-04 10:15:41 -04:00
llama_cpp.py feat: Update llama.cpp 2024-06-04 00:35:47 -04:00
llama_grammar.py fix: UTF-8 handling with grammars () 2024-04-30 14:33:23 -04:00
llama_speculative.py Add speculative decoding () 2024-01-31 14:08:14 -05:00
llama_tokenizer.py fix: LlamaHFTokenizer now receives pre_tokens 2024-02-23 12:23:24 -05:00
llama_types.py feat: Allow for possibly non-pooled embeddings () 2024-04-25 21:32:44 -04:00
llava_cpp.py feat: Generic Chat Formats, Tool Calling, and Huggingface Pull Support for Multimodal Models (Obsidian, LLaVA1.6, Moondream) () 2024-04-30 01:35:38 -04:00
py.typed Add py.typed 2023-08-11 09:58:48 +02:00