baalajimaestro/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

llama_cpp server: mark model as required

Browse source 
`model` is ignored, but currently marked "optional"... on the one hand could mark "required" to make it explicit in case the server supports multiple llama's at the same time, but also could delete it since its ignored. decision: mark it required for the sake of openai api compatibility.

I think out of all parameters, `model` is probably the most important one for people to keep using even if its ignored for now.
This commit is contained in:
Lucas Doyle 2023-04-29 00:47:35 -07:00
parent 53c0129eb6
commit e40fcb0575
1 changed files with 12 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
llama_cpp/server/app.py
View file

@ -66,6 +66,10 @@ def get_llama():
with llama_lock: with llama_lock:
yield llama yield llama
model_field = Field(
description="The model to use for generating completions."
)
class CreateCompletionRequest(BaseModel): class CreateCompletionRequest(BaseModel):
prompt: Union[str, List[str]] prompt: Union[str, List[str]]
suffix: Optional[str] = Field(None) suffix: Optional[str] = Field(None)
@ -76,8 +80,9 @@ class CreateCompletionRequest(BaseModel):
stop: Optional[List[str]] = [] stop: Optional[List[str]] = []
stream: bool = False stream: bool = False
# ignored or currently unsupported # ignored, but marked as required for the sake of compatibility with openai's api
model: Optional[str] = Field(None) model: str = model_field
n: Optional[int] = 1 n: Optional[int] = 1
logprobs: Optional[int] = Field(None) logprobs: Optional[int] = Field(None)
presence_penalty: Optional[float] = 0 presence_penalty: Optional[float] = 0
@ -133,7 +138,8 @@ def create_completion(
class CreateEmbeddingRequest(BaseModel): class CreateEmbeddingRequest(BaseModel):
model: Optional[str] # ignored, but marked as required for the sake of compatibility with openai's api
model: str = model_field
input: str input: str
user: Optional[str] user: Optional[str]
@ -173,8 +179,9 @@ class CreateChatCompletionRequest(BaseModel):
stop: Optional[List[str]] = [] stop: Optional[List[str]] = []
max_tokens: int = 128 max_tokens: int = 128
# ignored or currently unsupported # ignored, but marked as required for the sake of compatibility with openai's api
model: Optional[str] = Field(None) model: str = model_field
n: Optional[int] = 1 n: Optional[int] = 1
presence_penalty: Optional[float] = 0 presence_penalty: Optional[float] = 0
frequency_penalty: Optional[float] = 0 frequency_penalty: Optional[float] = 0