96efd9052f
* Re-introduce the llama package This PR brings back the llama package, making it possible to call llama.cpp and ggml APIs from Go directly via CGo. This has a few advantages: - C APIs can be called directly from Go without needing to use the previous "server" REST API - On macOS and for CPU builds on Linux and Windows, Ollama can be built without a go generate ./... step, making it easy to get up and running to hack on parts of Ollama that don't require fast inference - Faster build times for AVX,AVX2,CUDA and ROCM (a full build of all runners takes <5 min on a fast CPU) - No git submodule making it easier to clone and build from source This is a big PR, but much of it is vendor code except for: - llama.go CGo bindings - example/: a simple example of running inference - runner/: a subprocess server designed to replace the llm/ext_server package - Makefile an as minimal as possible Makefile to build the runner package for different targets (cpu, avx, avx2, cuda, rocm) Co-authored-by: Jesse Gross <jesse@ollama.com> Co-authored-by: Daniel Hiltgen <daniel@ollama.com> * cache: Clear old KV cache entries when evicting a slot When forking a cache entry, if no empty slots are available we evict the least recently used one and copy over the KV entries from the closest match. However, this copy does not overwrite existing values but only adds new ones. Therefore, we need to clear the old slot first. This change fixes two issues: - The KV cache fills up and runs out of space even though we think we are managing it correctly - Performance gets worse over time as we use new cache entries that are not hot in the processor caches * doc: explain golang objc linker warning (#6830) * llama: gather transitive dependencies for rocm for dist packaging (#6848) * Refine go server makefiles to be more DRY (#6924) This breaks up the monolithic Makefile for the Go based runners into a set of utility files as well as recursive Makefiles for the runners. Files starting with the name "Makefile" are buildable, while files that end with ".make" are utilities to include in other Makefiles. This reduces the amount of nearly identical targets and helps set a pattern for future community contributions for new GPU runner architectures. When we are ready to switch over to the Go runners, these files should move to the top of the repo, and we should add targets for the main CLI, as well as a helper "install" (put all the built binaries on the local system in a runnable state) and "dist" target (generate the various tar/zip files for distribution) for local developer use. * llama: don't create extraneous directories (#6988) * llama: Exercise the new build in CI (#6989) Wire up some basic sanity testing in CI for the Go runner. GPU runners are not covered yet. * llama: Refine developer docs for Go server (#6842) This enhances the documentation for development focusing on the new Go server. After we complete the transition further doc refinements can remove the "transition" discussion. * runner.go: Allocate batches for all sequences during init We should tell the model that we could have full batches for all sequences. We already do this when we allocate the batches but it was missed during initialization. * llama.go: Don't return nil from Tokenize on zero length input Potentially receiving nil in a non-error condition is surprising to most callers - it's better to return an empty slice. * runner.go: Remove stop tokens from cache If the last token is EOG then we don't return this and it isn't present in the cache (because it was never submitted to Decode). This works well for extending the cache entry with a new sequence. However, for multi-token stop sequences, we won't return any of the tokens but all but the last one will be in the cache. This means when the conversation continues the cache will contain tokens that don't overlap with the new prompt. This works (we will pick up the portion where there is overlap) but it causes unnecessary cache thrashing because we will fork the original cache entry as it is not a perfect match. By trimming the cache to the tokens that we actually return this issue can be avoided. * runner.go: Simplify flushing of pending tokens * runner.go: Update TODOs * runner.go: Don't panic when processing sequences If there is an error processing a sequence, we should return a clean HTTP error back to Ollama rather than panicing. This will make us more resilient to transient failures. Panics can still occur during startup as there is no way to serve requests if that fails. Co-authored-by: jmorganca <jmorganca@gmail.com> * runner.go: More accurately capture timings Currently prompt processing time doesn't capture the that it takes to tokenize the input, only decoding time. We should capture the full process to more accurately reflect reality. This is especially true once we start processing images where the initial processing can take significant time. This is also more consistent with the existing C++ runner. * runner.go: Support for vision models In addition to bringing feature parity with the C++ runner, this also incorporates several improvements: - Cache prompting works with images, avoiding the need to re-decode embeddings for every message in a conversation - Parallelism is supported, avoiding the need to restrict to one sequence at a time. (Though for now Ollama will not schedule them while we might need to fall back to the old runner.) Co-authored-by: jmorganca <jmorganca@gmail.com> * runner.go: Move Unicode checking code and add tests * runner.go: Export external cache members Runner and cache are in the same package so the change doesn't affect anything but it is more internally consistent. * runner.go: Image embedding cache Generating embeddings from images can take significant time (on my machine between 100ms and 8s depending on the model). Although we already cache the result of decoding these images, the embeddings need to be regenerated every time. This is not necessary if we get the same image over and over again, for example, during a conversation. This currently uses a very small cache with a very simple algorithm but it is easy to improve as is warranted. * llama: catch up on patches Carry forward solar-pro and cli-unicode patches * runner.go: Don't re-allocate memory for every batch We can reuse memory allocated from batch to batch since batch size is fixed. This both saves the cost of reallocation as well keeps the cache lines hot. This results in a roughly 1% performance improvement for token generation with Nvidia GPUs on Linux. * runner.go: Default to classic input cache policy The input cache as part of the go runner implemented a cache policy that aims to maximize hit rate in both single and multi- user scenarios. When there is a cache hit, the response is very fast. However, performance is actually slower when there is an input cache miss due to worse GPU VRAM locality. This means that performance is generally better overall for multi-user scenarios (better input cache hit rate, locality was relatively poor already). But worse for single users (input cache hit rate is about the same, locality is now worse). This defaults the policy back to the old one to avoid a regression but keeps the new one available through an environment variable OLLAMA_MULTIUSER_CACHE. This is left undocumented as the goal is to improve this in the future to get the best of both worlds without user configuration. For inputs that result in cache misses, on Nvidia/Linux this change improves performance by 31% for prompt processing and 13% for token generation. * runner.go: Increase size of response channel Generally the CPU can easily keep up with handling reponses that are generated but there's no reason not to let generation continue and handle things in larger batches if needed. * llama: Add CI to verify all vendored changes have patches (#7066) Make sure we don't accidentally merge changes in the vendored code that aren't also reflected in the patches. * llama: adjust clip patch for mingw utf-16 (#7065) * llama: adjust clip patch for mingw utf-16 * llama: ensure static linking of runtime libs Avoid runtime dependencies on non-standard libraries * runner.go: Enable llamafile (all platforms) and BLAS (Mac OS) These are two features that are shown on llama.cpp's system info that are currently different between the two runners. On my test systems the performance difference is very small to negligible but it is probably still good to equalize the features. * llm: Don't add BOS/EOS for tokenize requests This is consistent with what server.cpp currently does. It affects things like token processing counts for embedding requests. * runner.go: Don't cache prompts for embeddings Our integration with server.cpp implicitly disables prompt caching because it is not part of the JSON object being parsed, this makes the Go runner behavior similarly. Prompt caching has been seen to affect the results of text completions on certain hardware. The results are not wrong either way but they are non-deterministic. However, embeddings seem to be affected even on hardware that does not show this behavior for completions. For now, it is best to maintain consistency with the existing behavior. * runner.go: Adjust debug log levels Add system info printed at startup and quiet down noisier logging. * llama: fix compiler flag differences (#7082) Adjust the flags for the new Go server to more closely match the generate flow * llama: refine developer docs (#7121) * llama: doc and example clean up (#7122) * llama: doc and example clean up * llama: Move new dockerfile into llama dir Temporary home until we fully transition to the Go server * llama: runner doc cleanup * llama.go: Add description for Tokenize error case --------- Co-authored-by: Jesse Gross <jesse@ollama.com> Co-authored-by: Daniel Hiltgen <daniel@ollama.com> Co-authored-by: Daniel Hiltgen <dhiltgen@users.noreply.github.com>
750 lines
25 KiB
C++
750 lines
25 KiB
C++
/**
|
|
* llama.cpp - commit 8962422b1c6f9b8b15f5aeaea42600bcc2d44177 - do not edit this file
|
|
*
|
|
* MIT License
|
|
*
|
|
* Copyright (c) 2023-2024 The ggml authors
|
|
*
|
|
* Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
* of this software and associated documentation files (the "Software"), to deal
|
|
* in the Software without restriction, including without limitation the rights
|
|
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
* copies of the Software, and to permit persons to whom the Software is
|
|
* furnished to do so, subject to the following conditions:
|
|
*
|
|
* The above copyright notice and this permission notice shall be included in all
|
|
* copies or substantial portions of the Software.
|
|
*
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
* SOFTWARE.
|
|
*/
|
|
|
|
#pragma once
|
|
|
|
#include <chrono>
|
|
#include <cstring>
|
|
#include <sstream>
|
|
#include <iostream>
|
|
#include <thread>
|
|
#include <vector>
|
|
#include <algorithm>
|
|
#include <cinttypes>
|
|
|
|
// --------------------------------
|
|
//
|
|
// Basic usage:
|
|
//
|
|
// --------
|
|
//
|
|
// The LOG() and LOG_TEE() macros are ready to go by default
|
|
// they do not require any initialization.
|
|
//
|
|
// LOGLN() and LOG_TEELN() are variants which automatically
|
|
// include \n character at the end of the log string.
|
|
//
|
|
// LOG() behaves exactly like printf, by default writing to a logfile.
|
|
// LOG_TEE() additionally, prints to the screen too ( mimics Unix tee command ).
|
|
//
|
|
// Default logfile is named
|
|
// "llama.<threadID>.log"
|
|
// Default LOG_TEE() secondary output target is
|
|
// stderr
|
|
//
|
|
// Logs can be dynamically disabled or enabled using functions:
|
|
// log_disable()
|
|
// and
|
|
// log_enable()
|
|
//
|
|
// A log target can be changed with:
|
|
// log_set_target( string )
|
|
// creating and opening, or re-opening a file by string filename
|
|
// or
|
|
// log_set_target( FILE* )
|
|
// allowing to point at stderr, stdout, or any valid FILE* file handler.
|
|
//
|
|
// --------
|
|
//
|
|
// End of Basic usage.
|
|
//
|
|
// --------------------------------
|
|
|
|
// Specifies a log target.
|
|
// default uses log_handler() with "llama.log" log file
|
|
// this can be changed, by defining LOG_TARGET
|
|
// like so:
|
|
//
|
|
// #define LOG_TARGET (a valid FILE*)
|
|
// #include "log.h"
|
|
//
|
|
// or it can be simply redirected to stdout or stderr
|
|
// like so:
|
|
//
|
|
// #define LOG_TARGET stderr
|
|
// #include "log.h"
|
|
//
|
|
// The log target can also be redirected to a different function
|
|
// like so:
|
|
//
|
|
// #define LOG_TARGET log_handler_different()
|
|
// #include "log.h"
|
|
//
|
|
// FILE* log_handler_different()
|
|
// {
|
|
// return stderr;
|
|
// }
|
|
//
|
|
// or:
|
|
//
|
|
// #define LOG_TARGET log_handler_another_one("somelog.log")
|
|
// #include "log.h"
|
|
//
|
|
// FILE* log_handler_another_one(char*filename)
|
|
// {
|
|
// static FILE* logfile = nullptr;
|
|
// (...)
|
|
// if( !logfile )
|
|
// {
|
|
// fopen(...)
|
|
// }
|
|
// (...)
|
|
// return logfile
|
|
// }
|
|
//
|
|
#ifndef LOG_TARGET
|
|
#define LOG_TARGET log_handler()
|
|
#endif
|
|
|
|
#ifndef LOG_TEE_TARGET
|
|
#define LOG_TEE_TARGET stderr
|
|
#endif
|
|
|
|
// Utility for synchronizing log configuration state
|
|
// since std::optional was introduced only in c++17
|
|
enum LogTriState
|
|
{
|
|
LogTriStateSame,
|
|
LogTriStateFalse,
|
|
LogTriStateTrue
|
|
};
|
|
|
|
// Utility to obtain "pid" like unique process id and use it when creating log files.
|
|
inline std::string log_get_pid()
|
|
{
|
|
static std::string pid;
|
|
if (pid.empty())
|
|
{
|
|
// std::this_thread::get_id() is the most portable way of obtaining a "process id"
|
|
// it's not the same as "pid" but is unique enough to solve multiple instances
|
|
// trying to write to the same log.
|
|
std::stringstream ss;
|
|
ss << std::this_thread::get_id();
|
|
pid = ss.str();
|
|
}
|
|
|
|
return pid;
|
|
}
|
|
|
|
// Utility function for generating log file names with unique id based on thread id.
|
|
// invocation with log_filename_generator( "llama", "log" ) creates a string "llama.<number>.log"
|
|
// where the number is a runtime id of the current thread.
|
|
|
|
#define log_filename_generator(log_file_basename, log_file_extension) log_filename_generator_impl(LogTriStateSame, log_file_basename, log_file_extension)
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline std::string log_filename_generator_impl(LogTriState multilog, const std::string & log_file_basename, const std::string & log_file_extension)
|
|
{
|
|
static bool _multilog = false;
|
|
|
|
if (multilog != LogTriStateSame)
|
|
{
|
|
_multilog = multilog == LogTriStateTrue;
|
|
}
|
|
|
|
std::stringstream buf;
|
|
|
|
buf << log_file_basename;
|
|
if (_multilog)
|
|
{
|
|
buf << ".";
|
|
buf << log_get_pid();
|
|
}
|
|
buf << ".";
|
|
buf << log_file_extension;
|
|
|
|
return buf.str();
|
|
}
|
|
|
|
#ifndef LOG_DEFAULT_FILE_NAME
|
|
#define LOG_DEFAULT_FILE_NAME log_filename_generator("llama", "log")
|
|
#endif
|
|
|
|
// Utility for turning #define values into string literals
|
|
// so we can have a define for stderr and
|
|
// we can print "stderr" instead of literal stderr, etc.
|
|
#define LOG_STRINGIZE1(s) #s
|
|
#define LOG_STRINGIZE(s) LOG_STRINGIZE1(s)
|
|
|
|
#define LOG_TEE_TARGET_STRING LOG_STRINGIZE(LOG_TEE_TARGET)
|
|
|
|
// Allows disabling timestamps.
|
|
// in order to disable, define LOG_NO_TIMESTAMPS
|
|
// like so:
|
|
//
|
|
// #define LOG_NO_TIMESTAMPS
|
|
// #include "log.h"
|
|
//
|
|
#ifndef LOG_NO_TIMESTAMPS
|
|
#ifndef _MSC_VER
|
|
#define LOG_TIMESTAMP_FMT "[%" PRIu64 "] "
|
|
#define LOG_TIMESTAMP_VAL , (std::chrono::duration_cast<std::chrono::duration<std::uint64_t>>(std::chrono::system_clock::now().time_since_epoch())).count()
|
|
#else
|
|
#define LOG_TIMESTAMP_FMT "[%" PRIu64 "] "
|
|
#define LOG_TIMESTAMP_VAL , (std::chrono::duration_cast<std::chrono::duration<std::uint64_t>>(std::chrono::system_clock::now().time_since_epoch())).count()
|
|
#endif
|
|
#else
|
|
#define LOG_TIMESTAMP_FMT "%s"
|
|
#define LOG_TIMESTAMP_VAL ,""
|
|
#endif
|
|
|
|
#ifdef LOG_TEE_TIMESTAMPS
|
|
#ifndef _MSC_VER
|
|
#define LOG_TEE_TIMESTAMP_FMT "[%" PRIu64 "] "
|
|
#define LOG_TEE_TIMESTAMP_VAL , (std::chrono::duration_cast<std::chrono::duration<std::uint64_t>>(std::chrono::system_clock::now().time_since_epoch())).count()
|
|
#else
|
|
#define LOG_TEE_TIMESTAMP_FMT "[%" PRIu64 "] "
|
|
#define LOG_TEE_TIMESTAMP_VAL , (std::chrono::duration_cast<std::chrono::duration<std::uint64_t>>(std::chrono::system_clock::now().time_since_epoch())).count()
|
|
#endif
|
|
#else
|
|
#define LOG_TEE_TIMESTAMP_FMT "%s"
|
|
#define LOG_TEE_TIMESTAMP_VAL ,""
|
|
#endif
|
|
|
|
// Allows disabling file/line/function prefix
|
|
// in order to disable, define LOG_NO_FILE_LINE_FUNCTION
|
|
// like so:
|
|
//
|
|
// #define LOG_NO_FILE_LINE_FUNCTION
|
|
// #include "log.h"
|
|
//
|
|
#ifndef LOG_NO_FILE_LINE_FUNCTION
|
|
#ifndef _MSC_VER
|
|
#define LOG_FLF_FMT "[%24s:%5d][%24s] "
|
|
#define LOG_FLF_VAL , __FILE__, __LINE__, __FUNCTION__
|
|
#else
|
|
#define LOG_FLF_FMT "[%24s:%5ld][%24s] "
|
|
#define LOG_FLF_VAL , __FILE__, (long)__LINE__, __FUNCTION__
|
|
#endif
|
|
#else
|
|
#define LOG_FLF_FMT "%s"
|
|
#define LOG_FLF_VAL ,""
|
|
#endif
|
|
|
|
#ifdef LOG_TEE_FILE_LINE_FUNCTION
|
|
#ifndef _MSC_VER
|
|
#define LOG_TEE_FLF_FMT "[%24s:%5d][%24s] "
|
|
#define LOG_TEE_FLF_VAL , __FILE__, __LINE__, __FUNCTION__
|
|
#else
|
|
#define LOG_TEE_FLF_FMT "[%24s:%5ld][%24s] "
|
|
#define LOG_TEE_FLF_VAL , __FILE__, (long)__LINE__, __FUNCTION__
|
|
#endif
|
|
#else
|
|
#define LOG_TEE_FLF_FMT "%s"
|
|
#define LOG_TEE_FLF_VAL ,""
|
|
#endif
|
|
|
|
// INTERNAL, DO NOT USE
|
|
// USE LOG() INSTEAD
|
|
//
|
|
#if !defined(_MSC_VER) || defined(__INTEL_LLVM_COMPILER) || defined(__clang__)
|
|
#define LOG_IMPL(str, ...) \
|
|
do { \
|
|
if (LOG_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TARGET, LOG_TIMESTAMP_FMT LOG_FLF_FMT str "%s" LOG_TIMESTAMP_VAL LOG_FLF_VAL, __VA_ARGS__); \
|
|
fflush(LOG_TARGET); \
|
|
} \
|
|
} while (0)
|
|
#else
|
|
#define LOG_IMPL(str, ...) \
|
|
do { \
|
|
if (LOG_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TARGET, LOG_TIMESTAMP_FMT LOG_FLF_FMT str "%s" LOG_TIMESTAMP_VAL LOG_FLF_VAL "", ##__VA_ARGS__); \
|
|
fflush(LOG_TARGET); \
|
|
} \
|
|
} while (0)
|
|
#endif
|
|
|
|
// INTERNAL, DO NOT USE
|
|
// USE LOG_TEE() INSTEAD
|
|
//
|
|
#if !defined(_MSC_VER) || defined(__INTEL_LLVM_COMPILER) || defined(__clang__)
|
|
#define LOG_TEE_IMPL(str, ...) \
|
|
do { \
|
|
if (LOG_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TARGET, LOG_TIMESTAMP_FMT LOG_FLF_FMT str "%s" LOG_TIMESTAMP_VAL LOG_FLF_VAL, __VA_ARGS__); \
|
|
fflush(LOG_TARGET); \
|
|
} \
|
|
if (LOG_TARGET != nullptr && LOG_TARGET != stdout && LOG_TARGET != stderr && LOG_TEE_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TEE_TARGET, LOG_TEE_TIMESTAMP_FMT LOG_TEE_FLF_FMT str "%s" LOG_TEE_TIMESTAMP_VAL LOG_TEE_FLF_VAL, __VA_ARGS__); \
|
|
fflush(LOG_TEE_TARGET); \
|
|
} \
|
|
} while (0)
|
|
#else
|
|
#define LOG_TEE_IMPL(str, ...) \
|
|
do { \
|
|
if (LOG_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TARGET, LOG_TIMESTAMP_FMT LOG_FLF_FMT str "%s" LOG_TIMESTAMP_VAL LOG_FLF_VAL "", ##__VA_ARGS__); \
|
|
fflush(LOG_TARGET); \
|
|
} \
|
|
if (LOG_TARGET != nullptr && LOG_TARGET != stdout && LOG_TARGET != stderr && LOG_TEE_TARGET != nullptr) \
|
|
{ \
|
|
fprintf(LOG_TEE_TARGET, LOG_TEE_TIMESTAMP_FMT LOG_TEE_FLF_FMT str "%s" LOG_TEE_TIMESTAMP_VAL LOG_TEE_FLF_VAL "", ##__VA_ARGS__); \
|
|
fflush(LOG_TEE_TARGET); \
|
|
} \
|
|
} while (0)
|
|
#endif
|
|
|
|
// The '\0' as a last argument, is a trick to bypass the silly
|
|
// "warning: ISO C++11 requires at least one argument for the "..." in a variadic macro"
|
|
// so we can have a single macro which can be called just like printf.
|
|
|
|
// Main LOG macro.
|
|
// behaves like printf, and supports arguments the exact same way.
|
|
//
|
|
#if !defined(_MSC_VER) || defined(__clang__)
|
|
#define LOG(...) LOG_IMPL(__VA_ARGS__, "")
|
|
#else
|
|
#define LOG(str, ...) LOG_IMPL("%s" str, "", ##__VA_ARGS__, "")
|
|
#endif
|
|
|
|
// Main TEE macro.
|
|
// does the same as LOG
|
|
// and
|
|
// simultaneously writes stderr.
|
|
//
|
|
// Secondary target can be changed just like LOG_TARGET
|
|
// by defining LOG_TEE_TARGET
|
|
//
|
|
#if !defined(_MSC_VER) || defined(__clang__)
|
|
#define LOG_TEE(...) LOG_TEE_IMPL(__VA_ARGS__, "")
|
|
#else
|
|
#define LOG_TEE(str, ...) LOG_TEE_IMPL("%s" str, "", ##__VA_ARGS__, "")
|
|
#endif
|
|
|
|
// LOG macro variants with auto endline.
|
|
#if !defined(_MSC_VER) || defined(__clang__)
|
|
#define LOGLN(...) LOG_IMPL(__VA_ARGS__, "\n")
|
|
#define LOG_TEELN(...) LOG_TEE_IMPL(__VA_ARGS__, "\n")
|
|
#else
|
|
#define LOGLN(str, ...) LOG_IMPL("%s" str, "", ##__VA_ARGS__, "\n")
|
|
#define LOG_TEELN(str, ...) LOG_TEE_IMPL("%s" str, "", ##__VA_ARGS__, "\n")
|
|
#endif
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_handler1_impl(bool change = false, LogTriState append = LogTriStateSame, LogTriState disable = LogTriStateSame, const std::string & filename = LOG_DEFAULT_FILE_NAME, FILE *target = nullptr)
|
|
{
|
|
static bool _initialized = false;
|
|
static bool _append = false;
|
|
static bool _disabled = filename.empty() && target == nullptr;
|
|
static std::string log_current_filename{filename};
|
|
static FILE *log_current_target{target};
|
|
static FILE *logfile = nullptr;
|
|
|
|
if (change)
|
|
{
|
|
if (append != LogTriStateSame)
|
|
{
|
|
_append = append == LogTriStateTrue;
|
|
return logfile;
|
|
}
|
|
|
|
if (disable == LogTriStateTrue)
|
|
{
|
|
// Disable primary target
|
|
_disabled = true;
|
|
}
|
|
// If previously disabled, only enable, and keep previous target
|
|
else if (disable == LogTriStateFalse)
|
|
{
|
|
_disabled = false;
|
|
}
|
|
// Otherwise, process the arguments
|
|
else if (log_current_filename != filename || log_current_target != target)
|
|
{
|
|
_initialized = false;
|
|
}
|
|
}
|
|
|
|
if (_disabled)
|
|
{
|
|
// Log is disabled
|
|
return nullptr;
|
|
}
|
|
|
|
if (_initialized)
|
|
{
|
|
// with fallback in case something went wrong
|
|
return logfile ? logfile : stderr;
|
|
}
|
|
|
|
// do the (re)initialization
|
|
if (target != nullptr)
|
|
{
|
|
if (logfile != nullptr && logfile != stdout && logfile != stderr)
|
|
{
|
|
fclose(logfile);
|
|
}
|
|
|
|
log_current_filename = LOG_DEFAULT_FILE_NAME;
|
|
log_current_target = target;
|
|
|
|
logfile = target;
|
|
}
|
|
else
|
|
{
|
|
if (log_current_filename != filename)
|
|
{
|
|
if (logfile != nullptr && logfile != stdout && logfile != stderr)
|
|
{
|
|
fclose(logfile);
|
|
}
|
|
}
|
|
|
|
logfile = fopen(filename.c_str(), _append ? "a" : "w");
|
|
}
|
|
|
|
if (!logfile)
|
|
{
|
|
// Verify whether the file was opened, otherwise fallback to stderr
|
|
logfile = stderr;
|
|
|
|
fprintf(stderr, "Failed to open logfile '%s' with error '%s'\n", filename.c_str(), std::strerror(errno));
|
|
fflush(stderr);
|
|
|
|
// At this point we let the init flag be to true below, and let the target fallback to stderr
|
|
// otherwise we would repeatedly fopen() which was already unsuccessful
|
|
}
|
|
|
|
_initialized = true;
|
|
|
|
return logfile ? logfile : stderr;
|
|
}
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_handler2_impl(bool change = false, LogTriState append = LogTriStateSame, LogTriState disable = LogTriStateSame, FILE *target = nullptr, const std::string & filename = LOG_DEFAULT_FILE_NAME)
|
|
{
|
|
return log_handler1_impl(change, append, disable, filename, target);
|
|
}
|
|
|
|
// Disables logs entirely at runtime.
|
|
// Makes LOG() and LOG_TEE() produce no output,
|
|
// until enabled back.
|
|
#define log_disable() log_disable_impl()
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_disable_impl()
|
|
{
|
|
return log_handler1_impl(true, LogTriStateSame, LogTriStateTrue);
|
|
}
|
|
|
|
// Enables logs at runtime.
|
|
#define log_enable() log_enable_impl()
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_enable_impl()
|
|
{
|
|
return log_handler1_impl(true, LogTriStateSame, LogTriStateFalse);
|
|
}
|
|
|
|
// Sets target fir logs, either by a file name or FILE* pointer (stdout, stderr, or any valid FILE*)
|
|
#define log_set_target(target) log_set_target_impl(target)
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_set_target_impl(const std::string & filename) { return log_handler1_impl(true, LogTriStateSame, LogTriStateSame, filename); }
|
|
inline FILE *log_set_target_impl(FILE *target) { return log_handler2_impl(true, LogTriStateSame, LogTriStateSame, target); }
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_handler() { return log_handler1_impl(); }
|
|
|
|
// Enable or disable creating separate log files for each run.
|
|
// can ONLY be invoked BEFORE first log use.
|
|
#define log_multilog(enable) log_filename_generator_impl((enable) ? LogTriStateTrue : LogTriStateFalse, "", "")
|
|
// Enable or disable append mode for log file.
|
|
// can ONLY be invoked BEFORE first log use.
|
|
#define log_append(enable) log_append_impl(enable)
|
|
// INTERNAL, DO NOT USE
|
|
inline FILE *log_append_impl(bool enable)
|
|
{
|
|
return log_handler1_impl(true, enable ? LogTriStateTrue : LogTriStateFalse, LogTriStateSame);
|
|
}
|
|
|
|
inline void log_test()
|
|
{
|
|
log_disable();
|
|
LOG("01 Hello World to nobody, because logs are disabled!\n");
|
|
log_enable();
|
|
LOG("02 Hello World to default output, which is \"%s\" ( Yaaay, arguments! )!\n", LOG_STRINGIZE(LOG_TARGET));
|
|
LOG_TEE("03 Hello World to **both** default output and " LOG_TEE_TARGET_STRING "!\n");
|
|
log_set_target(stderr);
|
|
LOG("04 Hello World to stderr!\n");
|
|
LOG_TEE("05 Hello World TEE with double printing to stderr prevented!\n");
|
|
log_set_target(LOG_DEFAULT_FILE_NAME);
|
|
LOG("06 Hello World to default log file!\n");
|
|
log_set_target(stdout);
|
|
LOG("07 Hello World to stdout!\n");
|
|
log_set_target(LOG_DEFAULT_FILE_NAME);
|
|
LOG("08 Hello World to default log file again!\n");
|
|
log_disable();
|
|
LOG("09 Hello World _1_ into the void!\n");
|
|
log_enable();
|
|
LOG("10 Hello World back from the void ( you should not see _1_ in the log or the output )!\n");
|
|
log_disable();
|
|
log_set_target("llama.anotherlog.log");
|
|
LOG("11 Hello World _2_ to nobody, new target was selected but logs are still disabled!\n");
|
|
log_enable();
|
|
LOG("12 Hello World this time in a new file ( you should not see _2_ in the log or the output )?\n");
|
|
log_set_target("llama.yetanotherlog.log");
|
|
LOG("13 Hello World this time in yet new file?\n");
|
|
log_set_target(log_filename_generator("llama_autonamed", "log"));
|
|
LOG("14 Hello World in log with generated filename!\n");
|
|
#ifdef _MSC_VER
|
|
LOG_TEE("15 Hello msvc TEE without arguments\n");
|
|
LOG_TEE("16 Hello msvc TEE with (%d)(%s) arguments\n", 1, "test");
|
|
LOG_TEELN("17 Hello msvc TEELN without arguments\n");
|
|
LOG_TEELN("18 Hello msvc TEELN with (%d)(%s) arguments\n", 1, "test");
|
|
LOG("19 Hello msvc LOG without arguments\n");
|
|
LOG("20 Hello msvc LOG with (%d)(%s) arguments\n", 1, "test");
|
|
LOGLN("21 Hello msvc LOGLN without arguments\n");
|
|
LOGLN("22 Hello msvc LOGLN with (%d)(%s) arguments\n", 1, "test");
|
|
#endif
|
|
}
|
|
|
|
inline bool log_param_single_parse(const std::string & param)
|
|
{
|
|
if ( param == "--log-test")
|
|
{
|
|
log_test();
|
|
return true;
|
|
}
|
|
|
|
if ( param == "--log-disable")
|
|
{
|
|
log_disable();
|
|
return true;
|
|
}
|
|
|
|
if ( param == "--log-enable")
|
|
{
|
|
log_enable();
|
|
return true;
|
|
}
|
|
|
|
if (param == "--log-new")
|
|
{
|
|
log_multilog(true);
|
|
return true;
|
|
}
|
|
|
|
if (param == "--log-append")
|
|
{
|
|
log_append(true);
|
|
return true;
|
|
}
|
|
|
|
return false;
|
|
}
|
|
|
|
inline bool log_param_pair_parse(bool check_but_dont_parse, const std::string & param, const std::string & next = std::string())
|
|
{
|
|
if ( param == "--log-file")
|
|
{
|
|
if (!check_but_dont_parse)
|
|
{
|
|
log_set_target(log_filename_generator(next.empty() ? "unnamed" : next, "log"));
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
return false;
|
|
}
|
|
|
|
inline void log_print_usage()
|
|
{
|
|
printf("log options:\n");
|
|
/* format
|
|
printf(" -h, --help show this help message and exit\n");*/
|
|
/* spacing
|
|
printf("__-param----------------Description\n");*/
|
|
printf(" --log-test Run simple logging test\n");
|
|
printf(" --log-disable Disable trace logs\n");
|
|
printf(" --log-enable Enable trace logs\n");
|
|
printf(" --log-file Specify a log filename (without extension)\n");
|
|
printf(" --log-new Create a separate new log file on start. "
|
|
"Each log file will have unique name: \"<name>.<ID>.log\"\n");
|
|
printf(" --log-append Don't truncate the old log file.\n");
|
|
printf("\n");
|
|
}
|
|
|
|
#define log_dump_cmdline(argc, argv) log_dump_cmdline_impl(argc, argv)
|
|
|
|
// INTERNAL, DO NOT USE
|
|
inline void log_dump_cmdline_impl(int argc, char **argv)
|
|
{
|
|
std::stringstream buf;
|
|
for (int i = 0; i < argc; ++i)
|
|
{
|
|
if (std::string(argv[i]).find(' ') != std::string::npos)
|
|
{
|
|
buf << " \"" << argv[i] <<"\"";
|
|
}
|
|
else
|
|
{
|
|
buf << " " << argv[i];
|
|
}
|
|
}
|
|
LOGLN("Cmd:%s", buf.str().c_str());
|
|
}
|
|
|
|
#define log_tostr(var) log_var_to_string_impl(var).c_str()
|
|
|
|
inline std::string log_var_to_string_impl(bool var)
|
|
{
|
|
return var ? "true" : "false";
|
|
}
|
|
|
|
inline std::string log_var_to_string_impl(std::string var)
|
|
{
|
|
return var;
|
|
}
|
|
|
|
inline std::string log_var_to_string_impl(const std::vector<int> & var)
|
|
{
|
|
std::stringstream buf;
|
|
buf << "[ ";
|
|
bool first = true;
|
|
for (auto e : var)
|
|
{
|
|
if (first)
|
|
{
|
|
first = false;
|
|
}
|
|
else
|
|
{
|
|
buf << ", ";
|
|
}
|
|
buf << std::to_string(e);
|
|
}
|
|
buf << " ]";
|
|
|
|
return buf.str();
|
|
}
|
|
|
|
template <typename C, typename T>
|
|
inline std::string LOG_TOKENS_TOSTR_PRETTY(const C & ctx, const T & tokens)
|
|
{
|
|
std::stringstream buf;
|
|
buf << "[ ";
|
|
|
|
bool first = true;
|
|
for (const auto & token : tokens)
|
|
{
|
|
if (!first) {
|
|
buf << ", ";
|
|
} else {
|
|
first = false;
|
|
}
|
|
|
|
auto detokenized = llama_token_to_piece(ctx, token);
|
|
|
|
detokenized.erase(
|
|
std::remove_if(
|
|
detokenized.begin(),
|
|
detokenized.end(),
|
|
[](const unsigned char c) { return !std::isprint(c); }),
|
|
detokenized.end());
|
|
|
|
buf
|
|
<< "'" << detokenized << "'"
|
|
<< ":" << std::to_string(token);
|
|
}
|
|
buf << " ]";
|
|
|
|
return buf.str();
|
|
}
|
|
|
|
template <typename C, typename B>
|
|
inline std::string LOG_BATCH_TOSTR_PRETTY(const C & ctx, const B & batch)
|
|
{
|
|
std::stringstream buf;
|
|
buf << "[ ";
|
|
|
|
bool first = true;
|
|
for (int i = 0; i < batch.n_tokens; ++i)
|
|
{
|
|
if (!first) {
|
|
buf << ", ";
|
|
} else {
|
|
first = false;
|
|
}
|
|
|
|
auto detokenized = llama_token_to_piece(ctx, batch.token[i]);
|
|
|
|
detokenized.erase(
|
|
std::remove_if(
|
|
detokenized.begin(),
|
|
detokenized.end(),
|
|
[](const unsigned char c) { return !std::isprint(c); }),
|
|
detokenized.end());
|
|
|
|
buf
|
|
<< "\n" << std::to_string(i)
|
|
<< ":token '" << detokenized << "'"
|
|
<< ":pos " << std::to_string(batch.pos[i])
|
|
<< ":n_seq_id " << std::to_string(batch.n_seq_id[i])
|
|
<< ":seq_id " << std::to_string(batch.seq_id[i][0])
|
|
<< ":logits " << std::to_string(batch.logits[i]);
|
|
}
|
|
buf << " ]";
|
|
|
|
return buf.str();
|
|
}
|
|
|
|
#ifdef LOG_DISABLE_LOGS
|
|
|
|
#undef LOG
|
|
#define LOG(...) // dummy stub
|
|
#undef LOGLN
|
|
#define LOGLN(...) // dummy stub
|
|
|
|
#undef LOG_TEE
|
|
#define LOG_TEE(...) fprintf(stderr, __VA_ARGS__) // convert to normal fprintf
|
|
|
|
#undef LOG_TEELN
|
|
#define LOG_TEELN(...) fprintf(stderr, __VA_ARGS__) // convert to normal fprintf
|
|
|
|
#undef LOG_DISABLE
|
|
#define LOG_DISABLE() // dummy stub
|
|
|
|
#undef LOG_ENABLE
|
|
#define LOG_ENABLE() // dummy stub
|
|
|
|
#undef LOG_ENABLE
|
|
#define LOG_ENABLE() // dummy stub
|
|
|
|
#undef LOG_SET_TARGET
|
|
#define LOG_SET_TARGET(...) // dummy stub
|
|
|
|
#undef LOG_DUMP_CMDLINE
|
|
#define LOG_DUMP_CMDLINE(...) // dummy stub
|
|
|
|
#endif // LOG_DISABLE_LOGS
|