baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Update server docs

Browse source
This commit is contained in:
Andrei Betlen 2023-11-08 00:52:13 -05:00
parent 598780fde8
commit 9ae9c86be0
1 changed files with 17 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
docs/server.md
View file

@ -34,15 +34,29 @@ NOTE: All server options are also available as environment variables. For exampl
## Guides ## Guides
### Multi-modal Models ### Function Calling
`llama-cpp-python` supports structured function calling based on a JSON schema.
You'll first need to download one of the available function calling models in GGUF format:
- [functionary-7b-v1](https://huggingface.co/abetlen/functionary-7b-v1-GGUF)
Then when you run the server you'll need to also specify the `functionary-7b-v1` chat_format
```bash
python3 -m llama_cpp.server --model <model_path> --chat-format functionary
```
### Multimodal Models
`llama-cpp-python` supports the llava1.5 family of multi-modal models which allow the language model to `llama-cpp-python` supports the llava1.5 family of multi-modal models which allow the language model to
read information from both text and images. read information from both text and images.
You'll first need to download one of the available multi-modal models in GGUF format: You'll first need to download one of the available multi-modal models in GGUF format:
- [llava1.5 7b](https://huggingface.co/mys/ggml_llava-v1.5-7b) - [llava-v1.5-7b](https://huggingface.co/mys/ggml_llava-v1.5-7b)
- [llava1.5 13b](https://huggingface.co/mys/ggml_llava-v1.5-13b) - [llava-v1.5-13b](https://huggingface.co/mys/ggml_llava-v1.5-13b)
Then when you run the server you'll need to also specify the path to the clip model used for image embedding and the `llava-1-5` chat_format Then when you run the server you'll need to also specify the path to the clip model used for image embedding and the `llava-1-5` chat_format