8da7bef05f
In some cases we may want multiple variants for a given GPU type or CPU. This adds logic to have an optional Variant which we can use to select an optimal library, but also allows us to try multiple variants in case some fail to load. This can be useful for scenarios such as ROCm v5 vs v6 incompatibility or potentially CPU features.
18 lines
No EOL
874 B
Markdown
Vendored
18 lines
No EOL
874 B
Markdown
Vendored
# Extern C Server
|
|
|
|
This directory contains a thin facade we layer on top of the Llama.cpp server to
|
|
expose `extern C` interfaces to access the functionality through direct API
|
|
calls in-process. The llama.cpp code uses compile time macros to configure GPU
|
|
type along with other settings. During the `go generate ./...` execution, the
|
|
build will generate one or more copies of the llama.cpp `extern C` server based
|
|
on what GPU libraries are detected to support multiple GPU types as well as CPU
|
|
only support. The Ollama go build then embeds these different servers to support
|
|
different GPUs and settings at runtime.
|
|
|
|
If you are making changes to the code in this directory, make sure to disable
|
|
caching during your go build to ensure you pick up your changes. A typical
|
|
iteration cycle from the top of the source tree looks like:
|
|
|
|
```
|
|
go generate ./... && go build -a .
|
|
``` |