diff --git a/docker/README.md b/docker/README.md index 3a538af..100bcbd 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,3 +1,9 @@ +# Dockerfiles for building the llama-cpp-python server +- `Dockerfile.openblas_simple` - a simple Dockerfile for non-GPU OpenBLAS +- `Dockerfile.cuda_simple` - a simple Dockerfile for CUDA accelerated CuBLAS +- `hug_model.py` - a Python utility for interactively choosing and downloading the latest `5_1` quantized models from [huggingface.co/TheBloke]( https://huggingface.co/TheBloke) +- `Dockerfile` - a single OpenBLAS and CuBLAS combined Dockerfile that automatically installs a previously downloaded model `model.bin` + # Get model from Hugging Face `python3 ./hug_model.py` @@ -7,7 +13,7 @@ docker $ ls -lh *.bin -rw-rw-r-- 1 user user 4.8G May 23 18:30 .q5_1.bin lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> .q5_1.bin ``` -**Note #1:** Make sure you have enough disk space to d/l the model. As the model is then copied into the image you will need at least +**Note #1:** Make sure you have enough disk space to download the model. As the model is then copied into the image you will need at least **TWICE** as much disk space as the size of the model: | Model | Quantized size | @@ -21,20 +27,20 @@ lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> .q5 # Install Docker Server -**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this README with a PR! +**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this `README.md` with a PR! [Install Docker Engine](https://docs.docker.com/engine/install) # Use OpenBLAS -No NVidia GPU, defaults to `python:3-slim-bullseye` Docker base image and OpenBlAS: +Use if you don't have a NVidia GPU. Defaults to `python:3-slim-bullseye` Docker base image and OpenBLAS: ## Build: `docker build --build-arg -t openblas .` ## Run: `docker run --cap-add SYS_RESOURCE -t openblas` # Use CuBLAS -Requires NVidia GPU and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)) +Requires a NVidia GPU with sufficient VRAM (approximately as much as the size above) and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)) ## Build: -`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t opencuda .` +`docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t cublas .` ## Run: `docker run --cap-add SYS_RESOURCE -t cublas`