baalajimaestro/ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

runner: Set windows above normal priority (#6905)

Browse source 
When running the subprocess as a background service windows may
throttle, which can lead to thrashing and very poor token rate.
This commit is contained in:
Daniel Hiltgen 2024-09-21 16:54:49 -07:00 committed by GitHub
parent 6c2eb73a70
commit dbba73469d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 8 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
llm/llm_windows.go
View file

@ -4,7 +4,10 @@ import (
"syscall" "syscall"
) )
const CREATE_DEFAULT_ERROR_MODE = 0x04000000 const (
CREATE_DEFAULT_ERROR_MODE = 0x04000000
ABOVE_NORMAL_PRIORITY_CLASS = 0x00008000
)
var LlamaServerSysProcAttr = &syscall.SysProcAttr{ var LlamaServerSysProcAttr = &syscall.SysProcAttr{
// Wire up the default error handling logic If for some reason a DLL is // Wire up the default error handling logic If for some reason a DLL is
@ -12,5 +15,8 @@ var LlamaServerSysProcAttr = &syscall.SysProcAttr{
// the user can either fix their PATH, or report a bug. Without this // the user can either fix their PATH, or report a bug. Without this
// setting, the process exits immediately with a generic exit status but no // setting, the process exits immediately with a generic exit status but no
// way to (easily) figure out what the actual missing DLL was. // way to (easily) figure out what the actual missing DLL was.
CreationFlags: CREATE_DEFAULT_ERROR_MODE, //
// Setting Above Normal priority class ensures when running as a "background service"
// with "programs" given best priority, we aren't starved of cpu cycles
CreationFlags: CREATE_DEFAULT_ERROR_MODE | ABOVE_NORMAL_PRIORITY_CLASS,
} }