No description
Find a file
2023-03-31 00:00:43 -04:00
docs Add new Llama methods to docs 2023-03-28 05:04:15 -04:00
examples Add support to get embeddings from high-level api. Closes #4 2023-03-28 04:59:54 -04:00
llama_cpp Formatting 2023-03-31 00:00:27 -04:00
vendor Update llama.cpp 2023-03-31 00:00:43 -04:00
.gitignore Updated package to build with skbuild 2023-03-23 13:54:14 -04:00
.gitmodules Add llama.cpp to vendor folder 2023-03-23 05:37:26 -04:00
CMakeLists.txt Update llama.cpp and shared library build process 2023-03-23 17:01:06 -04:00
LICENSE.md Initial commit 2023-03-23 05:33:06 -04:00
mkdocs.yml Update docs 2023-03-24 19:02:36 -04:00
poetry.lock Add typing-extensions dependency 2023-03-30 06:43:31 -04:00
pyproject.toml Add typing-extensions dependency 2023-03-30 06:43:31 -04:00
README.md Add docs link 2023-03-27 18:30:12 -04:00
setup.py Add minimum python version, typing-extensions dependency, and long description for PyPI 2023-03-30 06:42:54 -04:00

🦙 Python Bindings for llama.cpp

Documentation PyPI PyPI - Python Version PyPI - License PyPI - Downloads

Simple Python bindings for @ggerganov's llama.cpp library. This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility

Installation

Install from PyPI:

pip install llama-cpp-python

Usage

>>> from llama_cpp import Llama
>>> llm = Llama(model_path="models/7B/...")
>>> output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
>>> print(output)
{
  "id": "cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "object": "text_completion",
  "created": 1679561337,
  "model": "models/7B/...",
  "choices": [
    {
      "text": "Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.",
      "index": 0,
      "logprobs": None,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 28,
    "total_tokens": 42
  }
}

License

This project is licensed under the terms of the MIT license.