82 lines
2.7 KiB
Markdown
82 lines
2.7 KiB
Markdown
# 🦙 Python Bindings for `llama.cpp`
|
|
|
|
[](https://abetlen.github.io/llama-cpp-python)
|
|
[](https://pypi.org/project/llama-cpp-python/)
|
|
[](https://pypi.org/project/llama-cpp-python/)
|
|
[](https://pypi.org/project/llama-cpp-python/)
|
|
[](https://pypi.org/project/llama-cpp-python/)
|
|
|
|
Simple Python bindings for **@ggerganov's** [`llama.cpp`](https://github.com/ggerganov/llama.cpp) library.
|
|
This package provides:
|
|
|
|
- Low-level access to C API via `ctypes` interface.
|
|
- High-level Python API for text completion
|
|
- OpenAI-like API
|
|
- LangChain compatibility
|
|
|
|
# Installation
|
|
|
|
Install from PyPI:
|
|
|
|
```bash
|
|
pip install llama-cpp-python
|
|
```
|
|
|
|
# Usage
|
|
|
|
```python
|
|
>>> from llama_cpp import Llama
|
|
>>> llm = Llama(model_path="models/7B/...")
|
|
>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
|
|
>>> print(output)
|
|
{
|
|
"id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
|
|
"object": "text_completion",
|
|
"created": 1679561337,
|
|
"model": "models/7B/...",
|
|
"choices": [
|
|
{
|
|
"text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
|
|
"index": 0,
|
|
"logprobs": None,
|
|
"finish_reason": "stop"
|
|
}
|
|
],
|
|
"usage": {
|
|
"prompt_tokens": 14,
|
|
"completion_tokens": 28,
|
|
"total_tokens": 42
|
|
}
|
|
}
|
|
```
|
|
|
|
# Documentation
|
|
|
|
Documentation is available at [https://abetlen.github.io/llama-cpp-python](https://abetlen.github.io/llama-cpp-python).
|
|
If you find any issues with the documentation, please open an issue or submit a PR.
|
|
|
|
# Development
|
|
|
|
This package is under active development and I welcome any contributions.
|
|
|
|
To get started, clone the repository and install the package in development mode:
|
|
|
|
```bash
|
|
git clone git@github.com:abetlen/llama-cpp-python.git
|
|
git submodule update --init --recursive
|
|
# Will need to be re-run any time vendor/llama.cpp is updated
|
|
python3 setup.py develop
|
|
```
|
|
|
|
# How does this compare to other Python bindings of `llama.cpp`?
|
|
|
|
I wrote this package for my own use, I had two goals in mind:
|
|
|
|
- Provide a simple process to install `llama.cpp` and access the full C API in `llama.h` from Python
|
|
- Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use `llama.cpp`
|
|
|
|
Any contributions and changes to this package will be made with these goals in mind.
|
|
|
|
# License
|
|
|
|
This project is licensed under the terms of the MIT license.
|