712e99d477
This will no longer error if built with regular gcc on windows. To help triage issues that may come in related to different compilers, the runner now reports the compier used by cgo. |
||
|---|---|---|
| .. | ||
| cache.go | ||
| cache_test.go | ||
| README.md | ||
| requirements.go | ||
| runner.go | ||
| stop.go | ||
| stop_test.go | ||
runner
Note: this is a work in progress
A minimial runner for loading a model and running inference via a http web server.
./runner -model <model binary>
Completion
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "hi"}' http://localhost:8080/completion
Embeddings
curl -X POST -H "Content-Type: application/json" -d '{"prompt": "turn me into an embedding"}' http://localhost:8080/embeddings