baalajimaestro/ollama
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
ollama/llm/dyn_ext_server.h

74 lines
3.2 KiB
C
Raw Normal View History

Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
#include <stdlib.h>
Refactor how we augment llama.cpp This changes the model for llama.cpp inclusion so we're not applying a patch, but instead have the C++ code directly in the ollama tree, which should make it easier to refine and update over time.
2023-12-22 17:51:53 +00:00
#include "ext_server.h"
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
#ifdef __cplusplus
extern "C" {
#endif
struct dynamic_llama_server {
void *handle;
void (*llama_server_init)(ext_server_params_t *sparams,
ext_server_resp_t *err);
void (*llama_server_start)();
void (*llama_server_stop)();
void (*llama_server_completion)(const char *json_req,
ext_server_resp_t *resp);
void (*llama_server_completion_next_result)(const int task_id,
ext_server_task_result_t *result);
void (*llama_server_completion_cancel)(const int task_id,
ext_server_resp_t *err);
void (*llama_server_release_task_result)(ext_server_task_result_t *result);
void (*llama_server_tokenize)(const char *json_req, char **json_resp,
ext_server_resp_t *err);
void (*llama_server_detokenize)(const char *json_req, char **json_resp,
ext_server_resp_t *err);
void (*llama_server_embedding)(const char *json_req, char **json_resp,
ext_server_resp_t *err);
void (*llama_server_release_json_resp)(char **json_resp);
};
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_init(const char *libPath, struct dynamic_llama_server *s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
ext_server_resp_t *err);
// No good way to call C function pointers from Go so inline the indirection
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_init(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
ext_server_params_t *sparams,
ext_server_resp_t *err);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_start(struct dynamic_llama_server s);
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_stop(struct dynamic_llama_server s);
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_completion(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
const char *json_req,
ext_server_resp_t *resp);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_completion_next_result(
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
struct dynamic_llama_server s, const int task_id,
ext_server_task_result_t *result);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_completion_cancel(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
const int task_id,
ext_server_resp_t *err);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_release_task_result(
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
struct dynamic_llama_server s, ext_server_task_result_t *result);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_tokenize(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
const char *json_req, char **json_resp,
ext_server_resp_t *err);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_detokenize(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
const char *json_req,
char **json_resp,
ext_server_resp_t *err);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_embedding(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
const char *json_req, char **json_resp,
ext_server_resp_t *err);
Always dynamically load the llm server library This switches darwin to dynamic loading, and refactors the code now that no static linking of the library is used on any platform
2024-01-10 04:29:58 +00:00
void dyn_llama_server_release_json_resp(struct dynamic_llama_server s,
Revamp the dynamic library shim This switches the default llama.cpp to be CPU based, and builds the GPU variants as dynamically loaded libraries which we can select at runtime. This also bumps the ROCm library to version 6 given 5.7 builds don't work on the latest ROCm library that just shipped.
2023-12-20 18:36:01 +00:00
char **json_resp);
#ifdef __cplusplus
}
#endif
Reference in a new issue Copy permalink