Update quill v7.3.0

[jellehierck]

The Quill version used in cactus_rt currently is quite outdated and not documented well at all. This made it hard to set up Quill as logging service for my own application which already uses cactus_rt. You already want to update Quill (#126) and because I love cactus_rt and I came to love Quill in the pas few weeks, I decided to put in the work.

Also, it would probably make logging data to a file on the hot path much simpler compared to creating a custom DataLogger thread as in the message passing example. Don't get me wrong, the message passing example is very cool! But does require me to write much of the data parsing myself. I much rather use Quill instead.

In addition, it is also possible to log backtraces (documentation), which is also an open issue for cactus_rt (#93), although I did not investigate this much yet.

Finally, I believe that #72 can be fixed with the quill::CsvWriter which will work in v7.4.0 (there is currently a bug in 7.3.0 but more on that later).

Overview of changes in behaviour

	Old (v3.3.2)	New (v7.3.0)
Use Bounded Queue	A compile flag is set in CMakeLists.txt: target_compile_definitions(cactus_rt PUBLIC QUILL_USE_BOUNDED_QUEUE).	Custom Frontend Options are specified in include/cactus_rt/logging.h, which are passed to create a custom cactus_rt::logging::Frontend and cactus_rt::logging::Logger. These custom cactus_rt::logging::Frontend and cactus_rt::logging::Logger types must be used throughout the application instead of the default quill::Frontend and quill::Logger (see the documentation).
Backend thread configuration	Quill backend thread is configured using quill::Config. The default options provided by Quill are used if none are set by the user.	Quill backend thread is configured with quill::BackendOptions. In include/cactus_rt/logging.h, the function cactus_rt::logging::DefaultBackendOptions() is used to set default options. We only change log_timestamp_ordering_grace_period to disable timestamp ordering (documentation) Note: because of a bug in v7.3.0 we do not actually set the one relevant option, but this will be fixed in v7.4.0.
Custom logger instantiation	Custom handlers are defined in the AppConfig::logger_config.default_handlers and applied to all threads in the cactus_rt::App. If no handlers are specified, a default logger with console handler is created instead.	Custom loggers, with one or many sinks and a formatter, must be configured before the thread is created. The custom logger can then be passed to a thread with ThreadConfig::logger_config.logger_name. If the logger_name option is an empty string, a default logger and console handler is created instead.
Default log f ormat and console sink (previous handler)	If no handlers were set in the AppConfig::logger_config.default_handlers, a default log format and default console handler were created in cactus_rt/app.h and applied to all loggers in all threads automatically.	The log format needs to be set per logger, and it needs to be set when the logger is created. The functioncactus_rt::logging::DefaultPatternFormatterOptions() (cactus_rt/logging.h) can be used to get the default format. The function cactus_rt::logging::DefaultConsoleSink() can be used to get a default ConsoleSink. The cactus_rt::logging::DefaultLogger("logger_name") uses the default formatter options.
Formatting STL library types	No additional action needed.	Requires including relevant headers from quill/std/* (e.g. #include "quill/std/Chrono.h") in the file where it is used. Example: LOG_INFO(logger, "Time: {}", std::chrono::seconds(3). The Quill Cheat Sheet contains very useful information for logging various non-primitive types.
Flushing loggers on cleanup	All Quill loggers are flushed when the cactus_rt::App destructor is called.	The logger associated with a thread is flushed when the cactus_rt::Thread destructor is called (and also when the cactus_rt::tracing::TraceAggregator destructor is called). When the cactus_rt::App destructor is called, the quill::Backend is stopped (but all associated threads should have flushed by then).
Including the logging macros	Done implicitly with #include "quill/Quill.h" which is called in other cactus_rt header files, and therefore also when a user includes #include "cactus_rt/rt.h".	Done explicitly in include/cactus_rt/logging.h, and therefore also when a user includes #include "cactus_rt/rt.h". However, in the source and header files of cactus_rt itself, #include "quill/LogMacros.h" is included explicitly.

All files were updated to match these changes. This also includes small updates such as #include statements, variable name changes, and so that the custom cactus_rt::logging::Frontend and cactus_rt::logging::Logger types are used.

Creating custom loggers

There are some important things to keep in mind when creating custom loggers.

• To use the BoundedDropping queue in loggers, a custom cactus_rt::logging::Frontend and cactus_rt::logging::Logger are defined inside include/cactus_rt/logging.h.
  • This means that creating a logger looks like this: cactus_rt::logging::Logger* logger = cactus_rt::logging::Frontend::create_or_get_logger(...);
  • These must be defined at compile time as per the documentation. Therefore, it is important that only cactus_rt::logging::Frontend and cactus_rt::logging::Logger are used throughout the application and never the default quill::Frontend and quill::Logger. Quill tries to throw an error if you do mix a default and custom Frontend, but does not catch all cases.
• When creating a custom logger, you can only set the formatter and one or more sinks when the logger is instantiated. They cannot be set or changed later. So, you must create custom quill::PatternFormatterOptions and quill::*Sink(s) before creating the logger.
  • Remember, creating a sink is done with the custom Frontend, e.g. auto console_sink = cactus_rt::logging::Frontend<quill::ConsoleSink>("sink_name");
• After creation, the logger name is passed to the cactus_rt::ThreadConfig::logger_config.logger_nameto use as the thread logger.
  • If no cactus_rt::ThreadConfig::logger_config.logger_name is set, a default logger is made using the thread's name (cactus_rt::logging::DefaultLogger(thread->Name())).
  • If the passed cactus_rt::ThreadConfig::logger_config.logger_name does not correspond to an existing logger yet, a default logger is made using the passed name (cactus_rt::logging::DefaultLogger(config_.logger_config.logger_name)).

An example of a custom thread logger with multiple sinks and a custom formatter pattern is given in examples/logging_example.

What is left to do

When updating Quill to v7.3.0, I encountered some bugs in Quill which were reported and fixed already. These changes will be part of release 7.4.0, which will probably be released no later than the beginning of November (odygrd/quill#609 (comment)).

• It is not possible to set the quill::BackendOptions::log_timestamp_ordering_grace_period to 0 since this causes an assertion error in Debug builds (Setting BackendOptions log_timestamp_ordering_grace_period to 0 milliseconds causes assertion error in Debug build odygrd/quill#605). This could result in slight delays in the logs on the hot path, which is obviously bad for real-time programming. For now, setting the grace period to 1 ns is the best option.
• It is not possible to create a quill::CsvWriter object with our custom cactus_rt::logging::FrontendOptions, i.e. which use the BoundedDropping queue type. This will result in a compilation error (Creating a CsvWriter with custom FrontendOptions gives compile error, and CsvWriter with default FrontendOptions but custom Logger gives runtime assertion violation odygrd/quill#609). Therefore, the CsvWriter is currently unusable until Quill v7.4.0 comes around (or we need to pull from the rolling master branch instead...)

[jellehierck]

These changes are quite big so I expect that some parts may not align with how you see cactus_rt. Let me know if that is the case and we can work out a working solution!