No description
Find a file
2023-04-07 13:32:19 +02:00
.github/workflows Update workflow name and add badge to README 2023-04-05 04:41:24 -04:00
docs Add chat completion method to docs 2023-04-03 20:14:03 -04:00
examples More interoperability to the original llama.cpp, and arguments now work 2023-04-07 13:32:19 +02:00
llama_cpp Make Llama instance pickleable. Closes #27 2023-04-05 06:52:17 -04:00
tests Make Llama instance pickleable. Closes #27 2023-04-05 06:52:17 -04:00
vendor Update llama.cpp 2023-04-03 14:49:07 -04:00
.gitignore Updated package to build with skbuild 2023-03-23 13:54:14 -04:00
.gitmodules Add llama.cpp to vendor folder 2023-03-23 05:37:26 -04:00
CMakeLists.txt Update llama.cpp and shared library build process 2023-03-23 17:01:06 -04:00
LICENSE.md Initial commit 2023-03-23 05:33:06 -04:00
mkdocs.yml Add search to mkdocs 2023-03-31 00:01:53 -04:00
poetry.lock Add basic tests. Closes #24 2023-04-05 03:23:15 -04:00
pyproject.toml Bump version 2023-04-05 06:53:43 -04:00
README.md Update workflow name and add badge to README 2023-04-05 04:41:24 -04:00
setup.py Bump version 2023-04-05 06:53:43 -04:00

🦙 Python Bindings for llama.cpp

Documentation Tests PyPI PyPI - Python Version PyPI - License PyPI - Downloads

Simple Python bindings for @ggerganov's llama.cpp library. This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility

Installation

Install from PyPI:

pip install llama-cpp-python

Usage

>>> from llama_cpp import Llama
>>> llm = Llama(model_path="models/7B/...")
>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
>>> print(output)
{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "models/7B/...",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}

Documentation

Documentation is available at https://abetlen.github.io/llama-cpp-python. If you find any issues with the documentation, please open an issue or submit a PR.

Development

This package is under active development and I welcome any contributions.

To get started, clone the repository and install the package in development mode:

git clone git@github.com:abetlen/llama-cpp-python.git
git submodule update --init --recursive
# Will need to be re-run any time vendor/llama.cpp is updated
python3 setup.py develop

How does this compare to other Python bindings of llama.cpp?

I originally wrote this package for my own use with two goals in mind:

  • Provide a simple process to install llama.cpp and access the full C API in llama.h from Python
  • Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use llama.cpp

Any contributions and changes to this package will be made with these goals in mind.

License

This project is licensed under the terms of the MIT license.