No description
Find a file
Lucas Doyle 7e03fdd766 poetry: add poetry.toml, configure to install in a virtualenv
Its cleaner to keep the virtualenv in the project directory, and
poetry makes it easy to do that via this config.
2023-05-09 16:03:19 -07:00
.github Update issue templates 2023-05-06 22:18:31 -04:00
docs Update docs 2023-04-24 19:56:57 -04:00
examples Merge pull request #126 from Stonelinks/deprecate-example-server 2023-05-08 19:29:04 -04:00
llama_cpp Implement sampling as in llama.cpp main example 2023-05-08 21:21:25 -04:00
tests Fix 2023-05-01 22:41:54 -04:00
vendor Update llama.cpp 2023-05-07 00:12:47 -04:00
.dockerignore Add dockerignore 2023-05-02 00:55:34 -04:00
.gitignore Ignore ./idea folder 2023-04-05 18:23:17 -04:00
.gitmodules Add llama.cpp to vendor folder 2023-03-23 05:37:26 -04:00
CMakeLists.txt Add FORCE_CMAKE option 2023-04-25 01:36:37 -04:00
Dockerfile Slim-Bullseye based docker image 2023-05-04 21:03:19 +02:00
Dockerfile.cuda Add docker cuda image. Closes #143 2023-05-03 10:29:05 -04:00
LICENSE.md Initial commit 2023-03-23 05:33:06 -04:00
mkdocs.yml Add search to mkdocs 2023-03-31 00:01:53 -04:00
poetry.lock pyproject.toml: add skbuild to dev dependencies 2023-05-09 16:03:13 -07:00
poetry.toml poetry: add poetry.toml, configure to install in a virtualenv 2023-05-09 16:03:19 -07:00
pyproject.toml pyproject.toml: add skbuild to dev dependencies 2023-05-09 16:03:13 -07:00
README.md Update README 2023-05-07 05:20:04 -04:00
setup.py Bump version 2023-05-08 18:49:37 -04:00

🦙 Python Bindings for llama.cpp

Documentation Tests PyPI PyPI - Python Version PyPI - License PyPI - Downloads

Simple Python bindings for @ggerganov's llama.cpp library. This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility

Install from PyPI (requires a c compiler):

pip install llama-cpp-python

The above command will attempt to install the package and build build llama.cpp from source. This is the recommended installation method as it ensures that llama.cpp is built with the available optimizations for your system.

Installation with OpenBLAS / cuBLAS / CLBlast

llama.cpp supports multiple BLAS backends for faster processing. Use the FORCE_CMAKE=1 environment variable to force the use of cmake and install the pip package for the desired BLAS backend.

To install with OpenBLAS, set the LLAMA_OPENBLAS=1 environment variable before installing:

LLAMA_OPENBLAS=1 FORCE_CMAKE=1 pip install llama-cpp-python

To install with cuBLAS, set the LLAMA_CUBLAS=1 environment variable before installing:

LLAMA_CUBLAS=1 FORCE_CMAKE=1 pip install llama-cpp-python

To install with CLBlast, set the LLAMA_CLBLAST=1 environment variable before installing:

LLAMA_CLBLAST=1 FORCE_CMAKE=1 pip install llama-cpp-python

High-level API

The high-level API provides a simple managed interface through the Llama class.

Below is a short example demonstrating how to use the high-level API to generate text:

>>> from llama_cpp import Llama
>>> llm = Llama(model_path="./models/7B/ggml-model.bin")
>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
>>> print(output)
{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "./models/7B/ggml-model.bin",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}

Web Server

llama-cpp-python offers a web server which aims to act as a drop-in replacement for the OpenAI API. This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).

To install the server package and get started:

pip install llama-cpp-python[server]
python3 -m llama_cpp.server --model models/7B/ggml-model.bin

Navigate to http://localhost:8000/docs to see the OpenAPI documentation.

Docker image

A Docker image is available on GHCR. To run the server:

docker run --rm -it -p8000:8000 -v /path/to/models:/models -eMODEL=/models/ggml-model-name.bin ghcr.io/abetlen/llama-cpp-python:latest

Low-level API

The low-level API is a direct ctypes binding to the C API provided by llama.cpp. The entire lowe-level API can be found in llama_cpp/llama_cpp.py and directly mirrors the C API in llama.h.

Below is a short example demonstrating how to use the low-level API to tokenize a prompt:

>>> import llama_cpp
>>> import ctypes
>>> params = llama_cpp.llama_context_default_params()
# use bytes for char * params
>>> ctx = llama_cpp.llama_init_from_file(b"./models/7b/ggml-model.bin", params)
>>> max_tokens = params.n_ctx
# use ctypes arrays for array params
>>> tokens = (llama_cppp.llama_token * int(max_tokens))()
>>> n_tokens = llama_cpp.llama_tokenize(ctx, b"Q: Name the planets in the solar system? A: ", tokens, max_tokens, add_bos=llama_cpp.c_bool(True))
>>> llama_cpp.llama_free(ctx)

Check out the examples folder for more examples of using the low-level API.

Documentation

Documentation is available at https://abetlen.github.io/llama-cpp-python. If you find any issues with the documentation, please open an issue or submit a PR.

Development

This package is under active development and I welcome any contributions.

To get started, clone the repository and install the package in development mode:

git clone --recurse-submodules git@github.com:abetlen/llama-cpp-python.git
# Will need to be re-run any time vendor/llama.cpp is updated
python3 setup.py develop

How does this compare to other Python bindings of llama.cpp?

I originally wrote this package for my own use with two goals in mind:

  • Provide a simple process to install llama.cpp and access the full C API in llama.h from Python
  • Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use llama.cpp

Any contributions and changes to this package will be made with these goals in mind.

License

This project is licensed under the terms of the MIT license.