From 70f629a72fe1dae576988a8107f683c66c887d7f Mon Sep 17 00:00:00 2001 From: Gary Mulder Date: Tue, 23 May 2023 20:36:21 +0100 Subject: [PATCH] Update README.md --- docker/README.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docker/README.md b/docker/README.md index 445f264..3a538af 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,13 +1,13 @@ # Get model from Hugging Face `python3 ./hug_model.py` -You should now have a model in the current directory and model.bin symlinked to it for the subsequent Docker build and copy step. e.g. +You should now have a model in the current directory and `model.bin` symlinked to it for the subsequent Docker build and copy step. e.g. ``` docker $ ls -lh *.bin --rw-rw-r-- 1 user user 4.8G May 23 18:30 llama-7b.ggmlv3.q5_1.bin +-rw-rw-r-- 1 user user 4.8G May 23 18:30 .q5_1.bin lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> .q5_1.bin ``` -- Note #1: Make sure you have enough disk space to d/l the model. As the model is then copied into the image you will need at least +**Note #1:** Make sure you have enough disk space to d/l the model. As the model is then copied into the image you will need at least **TWICE** as much disk space as the size of the model: | Model | Quantized size | @@ -17,16 +17,23 @@ lrwxrwxrwx 1 user user 24 May 23 18:30 model.bin -> .q5 | 30B | 25 GB | | 65B | 50 GB | -- Note #2: If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...` +**Note #2:** If you want to pass or tune additional parameters, customise `./start_server.sh` before running `docker build ...` -# Use OpenBLAS (No NVidia GPU, defaults to `python:3-slim-bullseye` Docker base image) +# Install Docker Server + +**Note #3:** This was tested with Docker running on Linux. If you can get it working on Windows or MacOS, please update this README with a PR! + +[Install Docker Engine](https://docs.docker.com/engine/install) + +# Use OpenBLAS +No NVidia GPU, defaults to `python:3-slim-bullseye` Docker base image and OpenBlAS: ## Build: `docker build --build-arg -t openblas .` ## Run: `docker run --cap-add SYS_RESOURCE -t openblas` # Use CuBLAS -Requires NVidia GPU and Docker NVidia support (see https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) +Requires NVidia GPU and Docker NVidia support (see [container-toolkit/install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)) ## Build: `docker build --build-arg IMAGE=nvidia/cuda:12.1.1-devel-ubuntu22.04 -t opencuda .` ## Run: