From 40b22909dc55806cd546a225ebbc9d790165e0da Mon Sep 17 00:00:00 2001
From: Jason Cox <jason@jasonacox.com>
Date: Thu, 14 Sep 2023 11:48:21 -0700
Subject: [PATCH] Update examples from ggml to gguf and add hw-accel note for
 Web Server (#688)

* Examples from ggml to gguf

* Use gguf file extension

Update examples to use filenames with gguf extension (e.g. llama-model.gguf).

---------

Co-authored-by: Andrei <abetlen@gmail.com>
---
 README.md | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

diff --git a/README.md b/README.md
index 07accb6..1881af6 100644
--- a/README.md
+++ b/README.md
@@ -106,14 +106,14 @@ Below is a short example demonstrating how to use the high-level API to generate
 
 ```python
 >>> from llama_cpp import Llama
->>> llm = Llama(model_path="./models/7B/ggml-model.bin")
+>>> llm = Llama(model_path="./models/7B/llama-model.gguf")
 >>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
 >>> print(output)
 {
   "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "object": "text_completion",
   "created": 1679561337,
-  "model": "./models/7B/ggml-model.bin",
+  "model": "./models/7B/llama-model.gguf",
   "choices": [
     {
       "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
@@ -136,7 +136,7 @@ The context window of the Llama models determines the maximum number of tokens t
 For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object:
 
 ```python
-llm = Llama(model_path="./models/7B/ggml-model.bin", n_ctx=2048)
+llm = Llama(model_path="./models/7B/llama-model.gguf", n_ctx=2048)
 ```
 
 ### Loading llama-2 70b
@@ -144,7 +144,7 @@ llm = Llama(model_path="./models/7B/ggml-model.bin", n_ctx=2048)
 Llama2 70b must set the `n_gqa` parameter (grouped-query attention factor) to 8 when loading:
 
 ```python
-llm = Llama(model_path="./models/70B/ggml-model.bin", n_gqa=8)
+llm = Llama(model_path="./models/70B/llama-model.gguf", n_gqa=8)
 ```
 
 ## Web Server
@@ -156,17 +156,24 @@ To install the server package and get started:
 
 ```bash
 pip install llama-cpp-python[server]
-python3 -m llama_cpp.server --model models/7B/ggml-model.bin
+python3 -m llama_cpp.server --model models/7B/llama-model.gguf
+```
+Similar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:
+
+```bash
+CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python[server]
+python3 -m llama_cpp.server --model models/7B/llama-model.gguf --n_gpu_layers 35
 ```
 
 Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) to see the OpenAPI documentation.
 
+
 ## Docker image
 
 A Docker image is available on [GHCR](https://ghcr.io/abetlen/llama-cpp-python). To run the server:
 
 ```bash
-docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/ggml-model-name.bin ghcr.io/abetlen/llama-cpp-python:latest
+docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf ghcr.io/abetlen/llama-cpp-python:latest
 ```
 [Docker on termux (requires root)](https://gist.github.com/FreddieOliveira/efe850df7ff3951cb62d74bd770dce27) is currently the only known way to run this on phones, see [termux support issue](https://github.com/abetlen/llama-cpp-python/issues/389) 
 
@@ -183,7 +190,7 @@ Below is a short example demonstrating how to use the low-level API to tokenize
 >>> llama_cpp.llama_backend_init(numa=False) # Must be called once at the start of each program
 >>> params = llama_cpp.llama_context_default_params()
 # use bytes for char * params
->>> model = llama_cpp.llama_load_model_from_file(b"./models/7b/ggml-model.bin", params)
+>>> model = llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf", params)
 >>> ctx = llama_cpp.llama_new_context_with_model(model, params)
 >>> max_tokens = params.n_ctx
 # use ctypes arrays for array params