baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Set n_batch to the default value of 8. I think this is leftover from when n_ctx was missing and n_batch was 2048.

Browse source
This commit is contained in:
MillionthOdin16 2023-04-05 17:44:53 -04:00
parent 1e90597983
commit 76a82babef
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
llama_cpp/server/__main__.py
View file

@ -27,7 +27,7 @@ from sse_starlette.sse import EventSourceResponse
class Settings(BaseSettings):
model: str
n_ctx: int = 2048
n_batch: int = 2048
n_batch: int = 8
n_threads: int = os.cpu_count() or 1
f16_kv: bool = True
use_mlock: bool = True