baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Fixed Cuda Dockerfile

Browse source 
Previously models produced garbage output when running on GPU with layers offloaded.

Similar to related fix on another repo: 331326a0e3
This commit is contained in:
Pradyumna Singh Rathore 2023-08-10 20:41:34 +05:30 committed by GitHub
parent 66fb0345e8
commit d010ea70d2
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 14 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
docker/cuda_simple/Dockerfile
View file

@ -4,13 +4,24 @@ FROM nvidia/cuda:${CUDA_IMAGE}
# We need to set the host to 0.0.0.0 to allow outside access # We need to set the host to 0.0.0.0 to allow outside access
ENV HOST 0.0.0.0 ENV HOST 0.0.0.0
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y git build-essential \
python3 python3-pip gcc wget \
ocl-icd-opencl-dev opencl-headers clinfo \
libclblast-dev libopenblas-dev \
&& mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
COPY . . COPY . .
# Install the package # setting build related env vars
RUN apt update && apt install -y python3 python3-pip ENV CUDA_DOCKER_ARCH=all
ENV LLAMA_CUBLAS=1
# Install depencencies
RUN python3 -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette pydantic-settings RUN python3 -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi uvicorn sse-starlette pydantic-settings
RUN LLAMA_CUBLAS=1 pip install llama-cpp-python # Install llama-cpp-python (build with cuda)
RUN CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python
# Run the server # Run the server
CMD python3 -m llama_cpp.server CMD python3 -m llama_cpp.server