From b53229a2eda7174c98c860f26f6afce583139031 Mon Sep 17 00:00:00 2001 From: Daniel Hiltgen Date: Mon, 11 Mar 2024 16:54:38 -0700 Subject: [PATCH] Add docs explaining GPU selection env vars --- docs/faq.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/faq.md b/docs/faq.md index 805f3fa4..1362d783 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -193,3 +193,13 @@ To unload the model and free up memory use: ```shell curl http://localhost:11434/api/generate -d '{"model": "llama2", "keep_alive": 0}' ``` + +## Controlling which GPUs to use + +By default, on Linux and Windows, Ollama will attempt to use Nvidia GPUs, or +Radeon GPUs, and will use all the GPUs it can find. You can limit which GPUs +will be utilized by setting the environment variable `CUDA_VISIBLE_DEVICES` for +NVIDIA cards, or `HIP_VISIBLE_DEVICES` for Radeon GPUs to a comma delimited list +of GPU IDs. You can see the list of devices with GPU tools such as `nvidia-smi` or +`rocminfo`. You can set to an invalid GPU ID (e.g., "-1") to bypass the GPU and +fallback to CPU. \ No newline at end of file