llama.cpp/docker/README.md

41 lines
1.6 KiB
Markdown
Raw Normal View History

2023-05-23 19:26:40 +00:00
# Get model from Hugging Face
`python3 ./hug_model.py`
2023-05-23 19:36:21 +00:00
You should now have a model in the current directory and `model.bin` symlinked to it for the subsequent Docker build and copy step. e.g.
2023-05-23 19:26:40 +00:00
```
docker $ ls -lh *.bin
2023-05-23 19:36:21 +00:00
-rw-rw-r-- 1 user user 4.8G May 23 18:30 <downloaded-model-file>.q5_1.bin
2023-05-23 19:26:40 +00:00
lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> <downloaded-model-file>.q5_1.bin
```
2023-05-23 19:36:21 +00:00
**Note #1:** Make sure you have enough disk space to d/l the model. As the model is then copied into the image you will need at least
2023-05-23 19:26:40 +00:00
**TWICE** as much disk space as the size of the model:
| Model | Quantized size |
|------:|----------------:|
| 7B | 5 GB |
| 13B | 10 GB |
| 30B | 25 GB |
| 65B | 50 GB |
2023-05-23 19:36:21 +00:00
**Note #2:** If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...`
2023-05-23 19:26:40 +00:00
2023-05-23 19:36:21 +00:00
# Install Docker Server
**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this README with a PR!
[Install Docker Engine](https://docs.docker.com/engine/install)
# Use OpenBLAS
No NVidia GPU, defaults to `python:3-slim-bullseye` Docker base image and OpenBlAS:
2023-05-23 19:26:40 +00:00
## Build:
`docker build --build-arg -t openblas .`
## Run:
`docker run --cap-add SYS_RESOURCE -t openblas`
# Use CuBLAS
2023-05-23 19:36:21 +00:00
Requires NVidia GPU and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
2023-05-23 19:26:40 +00:00
## Build:
`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t opencuda .`
## Run:
`docker run --cap-add SYS_RESOURCE -t cublas`