baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

docs: Update Functionary OpenAI Server Readme (#1193)

Browse source 
* update functionary parts in server readme

* add write-up about hf tokenizer
This commit is contained in:
Jeffrey Fong 2024-02-24 01:24:10 +08:00 committed by GitHub
parent 47bad30dd7
commit bce6dc0ac2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 6 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
docs/server.md
View file

@ -76,12 +76,14 @@ Function calling is completely compatible with the OpenAI function calling API a
You'll first need to download one of the available function calling models in GGUF format:
- [functionary-7b-v1](https://huggingface.co/abetlen/functionary-7b-v1-GGUF)
- [functionary](https://huggingface.co/meetkai)
Then when you run the server you'll need to also specify the `functionary` chat_format
Then when you run the server you'll need to also specify either `functionary-v1` or `functionary-v2` chat_format.
Note that since functionary requires a HF Tokenizer due to discrepancies between llama.cpp and HuggingFace's tokenizers as mentioned [here](https://github.com/abetlen/llama-cpp-python/blob/main?tab=readme-ov-file#function-calling), you will need to pass in the path to the tokenizer too. The tokenizer files are already included in the respective HF repositories hosting the gguf files.
```bash
python3 -m llama_cpp.server --model <model_path> --chat_format functionary
python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>
```
Check out this [example notebook](https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Functions.ipynb) for a walkthrough of some interesting use cases for function calling.

2
examples/notebooks/Functions.ipynb
View file

@ -9,7 +9,7 @@
"The OpenAI compatbile web server in `llama-cpp-python` supports function calling.\n",
"\n",
"Function calling allows API clients to specify a schema that gives the model a format it should respond in.\n",
"Function calling in `llama-cpp-python` works by combining models pretrained for function calling such as [`functionary`](https://huggingface.co/abetlen/functionary-7b-v1-GGUF) with constrained sampling to produce a response that is compatible with the schema.\n",
"Function calling in `llama-cpp-python` works by combining models pretrained for function calling such as [`functionary`](https://huggingface.co/meetkai) with constrained sampling to produce a response that is compatible with the schema.\n",
"\n",
"Note however that this improves but does not guarantee that the response will be compatible with the schema.\n",
"\n",