diff --git a/docker/README.md b/docker/README.md index 053d311..9ffd3b1 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,13 +1,13 @@ -# Install Docker Server - -**Note #1:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this `README.md` with a PR! +### Install Docker Server +> [!IMPORTANT] +> This was tested with Docker running on Linux.
If you can get it working on Windows or MacOS, please update this `README.md` with a PR!
[Install Docker Engine](https://docs.docker.com/engine/install) -**Note #2:** NVidia GPU CuBLAS support requires a NVidia GPU with sufficient VRAM (approximately as much as the size in the table below) and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)) -# Simple Dockerfiles for building the llama-cpp-python server with external model bin files -## openblas_simple - a simple Dockerfile for non-GPU OpenBLAS, where the model is located outside the Docker image +## Simple Dockerfiles for building the llama-cpp-python server with external model bin files +### openblas_simple +A simple Dockerfile for non-GPU OpenBLAS, where the model is located outside the Docker image: ``` cd ./openblas_simple docker build -t openblas_simple . @@ -15,23 +15,30 @@ docker run -e USE_MLOCK=0 -e MODEL=/var/model/ -v : ``` where `/` is the full path to the model file on the Docker host system. -## cuda_simple - a simple Dockerfile for CUDA accelerated CuBLAS, where the model is located outside the Docker image +### cuda_simple +> [!WARNING] +> Nvidia GPU CuBLAS support requires an Nvidia GPU with sufficient VRAM (approximately as much as the size in the table below) and Docker Nvidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
+ +A simple Dockerfile for CUDA-accelerated CuBLAS, where the model is located outside the Docker image: + ``` cd ./cuda_simple docker build -t cuda_simple . -docker run -e USE_MLOCK=0 -e MODEL=/var/model/ -v :/var/model -t cuda_simple +docker run --gpus=all --cap-add SYS_RESOURCE -e USE_MLOCK=0 -e MODEL=/var/model/ -v :/var/model -t cuda_simple ``` where `/` is the full path to the model file on the Docker host system. -# "Open-Llama-in-a-box" -## Download an Apache V2.0 licensed 3B paramter Open Llama model and install into a Docker image that runs an OpenBLAS-enabled llama-cpp-python server +-------------------------------------------------------------------------- + +### "Open-Llama-in-a-box" +Download an Apache V2.0 licensed 3B params Open LLaMA model and install into a Docker image that runs an OpenBLAS-enabled llama-cpp-python server: ``` $ cd ./open_llama ./build.sh ./start.sh ``` -# Manually choose your own Llama model from Hugging Face +### Manually choose your own Llama model from Hugging Face `python3 ./hug_model.py -a TheBloke -t llama` You should now have a model in the current directory and `model.bin` symlinked to it for the subsequent Docker build and copy step. e.g. ``` @@ -39,8 +46,10 @@ docker $ ls -lh *.bin -rw-rw-r-- 1 user user 4.8G May 23 18:30 q5_1.bin lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> q5_1.bin ``` -**Note #1:** Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least -**TWICE** as much disk space as the size of the model: + +> [!NOTE] +> Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least +**TWICE** as much disk space as the size of the model:
| Model | Quantized size | |------:|----------------:| @@ -50,17 +59,6 @@ lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> q5_ | 33B | 25 GB | | 65B | 50 GB | -**Note #2:** If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...` -## Use OpenBLAS -Use if you don't have a NVidia GPU. Defaults to `python:3-slim-bullseye` Docker base image and OpenBLAS: -### Build: -`docker build -t openblas .` -### Run: -`docker run --cap-add SYS_RESOURCE -t openblas` - -## Use CuBLAS -### Build: -`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t cublas .` -### Run: -`docker run --cap-add SYS_RESOURCE -t cublas` +> [!NOTE] +> If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...`