diff --git a/.github/workflows/ci-tests.yaml b/.github/workflows/ci-tests.yaml index fdbe5e5f2..2ae1d673f 100644 --- a/.github/workflows/ci-tests.yaml +++ b/.github/workflows/ci-tests.yaml @@ -68,6 +68,7 @@ jobs: --name capio-docker \ alphaunito/capio:latest \ sh -c 'jq -n --arg pwd "$PWD" "$1" > test_config.json && \ + printf "[capiocl]\nworkflow_name = \"CAPIO\"\nconfig_path = \"test_config.json\"\n[capio]\ndirectory = \"$PWD\"\n" > test_config.toml && \ LD_PRELOAD=libcapio_posix.so capio_syscall_unit_tests \ --gtest_break_on_failure \ --gtest_print_time=1' sh \ @@ -186,8 +187,11 @@ jobs: --arg pwd $(pwd) \ '{name: "CAPIO", IO_Graph: [], exclude: [$pwd + "/aNonExistingFile", $pwd, $pwd + "/"]}' \ > test_config.json + + printf '[capiocl]\nworkflow_name = "CAPIO"\nconfig_path = "test_config.json"\n[capio]\ndirectory = "%s"\n' \ + "$(pwd)" > test_config.toml - export CONFIG_PATH=$(realpath test_config.json) + export CONFIG_PATH=$(realpath test_config.toml) export LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH}" diff --git a/CMakeLists.txt b/CMakeLists.txt index 21536928f..5ad83f34b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -83,7 +83,7 @@ ENDIF () FetchContent_Declare( capio_cl GIT_REPOSITORY https://github.com/High-Performance-IO/CAPIO-CL.git - GIT_TAG v1.5.2 + GIT_TAG v2.0.1 ) FetchContent_Declare( diff --git a/Dockerfile b/Dockerfile index 653a792cd..8b7a03a73 100644 --- a/Dockerfile +++ b/Dockerfile @@ -109,7 +109,6 @@ COPY --from=builder \ # Pkgconfig COPY --from=builder \ - "/usr/local/lib/pkgconfig/args.pc" \ "/usr/local/lib/pkgconfig/gmock.p[c]" \ "/usr/local/lib/pkgconfig/gmock_main.p[c]" \ "/usr/local/lib/pkgconfig/gtest.p[c]" \ diff --git a/README.md b/README.md index 6f74ed72e..06d97914d 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,6 @@ The following dependencies are automatically fetched during the CMake configurat - [CALF](https://github.com/High-Performance-IO/CALF) provides logging and CLI output. - [alpha-unito/syscall_intercept](https://github.com/alpha-unito/syscall_intercept) intercepts system calls (forked from `pmem/syscall_intercept`). - [ParaGroup/MTCL](https://github.com/ParaGroup/MTCL) provides dynamic, multi-backend communication between CAPIO server instances. -- [Taywee/args](https://github.com/Taywee/args) parses server command-line arguments. ### Compile capio @@ -50,81 +49,89 @@ It is also possible to enable log in CAPIO, by defining `-DCAPIO_LOG=TRUE`. Good news! You don't need to modify your code to benefit from the features of CAPIO. You have only to do three steps ( the first is optional). -1) Write a configuration file for injecting streaming capabilities to your workflow +1. Generate and edit the TOML configuration with `capio_server --genconf`, or use built-in defaults with `capio_server --defconf`. -2) Launch the CAPIO daemons with MPI passing the (eventual) configuration file as argument on the machines in which you - want to execute your program (one daemon for each node). If you desire to specify a custom folder - for capio, set `CAPIO_DIR` as a environment variable. +2. Launch the CAPIO daemons with MPI, passing the TOML configuration file on the machines in which you + want to execute your program (one daemon for each node). Set `capio.directory` in that file to + choose the managed root directory. ```bash - [CAPIO_DIR=your_capiodir] [mpiexec -N 1 --hostfile your_hostfile] capio_server -c conf.json + [mpiexec -N 1 --hostfile your_hostfile] capio_server default.toml ``` -3) Launch your programs preloading the CAPIO shared library like this: +3. Launch your programs preloading the CAPIO shared library like this: ```bash CAPIO_DIR=your_capiodir \ - CAPIO_WORKFLOW_NAME=wfname \ + CAPIO_WORKFLOW_NAME=wfname \ CAPIO_APP_NAME=appname \ - LD_PRELOAD=libcapio_posix.so \ + LD_PRELOAD=libcapio_posix.so \ ./your_app ``` > [!WARNING] > `CAPIO_DIR` must be specified when launching a program with the CAPIO library. if `CAPIO_DIR` is not specified, CAPIO -> will not intercept syscalls. - -> [!NOTE] -> If `CAPIO_DIR` is not specified when launching `capio_server`, it defaults to the server's current working directory. +> will not intercept syscalls. Its value must match `capio.directory` in the server TOML configuration. ### Server backend options -Use `--backend-options` to configure the MTCL backend. Its format is -`PROTO:PORT@POLL_INTERVAL_US`; for example: +Configure the backend in TOML. For example: -```bash -capio_server --no-config --backend mtcl --backend-options TCP:7600@1000000 +```toml +[capio.backend] +type = "mtcl" + +[capio.backend.mtcl] +proto = "TCP" +listen_address = "0.0.0.0" +port = 7600 +poll_interval_us = 1000000 ``` -If omitted, MTCL defaults to `TCP:7600@1000000`. See the +If omitted, CAPIO uses the `none` backend. See the [MTCL repository](https://github.com/ParaGroup/MTCL) for supported communication protocols and their requirements. -### Available environment variables +### Server configuration -CAPIO can be controlled through the usage of environment variables. The available variables are listed below: +The server reads runtime settings only from TOML. `capio_server --genconf` writes a documented `default.toml`, while +`capio_server --defconf` starts directly with the same built-in defaults. Important server settings include: -#### Global environment variable +```toml +[capio] +directory = "." -- `CAPIO_DIR` This environment variable tells to both server and application the mount point of capio; -- `CAPIO_LOG_LEVEL` this environment tells both server and application the log level to use. This variable works only - if `-DCAPIO_LOG=TRUE` was specified during cmake phase; -- `CAPIO_LOG_PREFIX` This environment variable is defined only for capio_posix applications and specifies the prefix of - the logfile name to which capio will log to. The default value is `posix_thread_`, which means that capio will log by - default to a set of files called `posix_thread_*.log`. An equivalent behaviour can be set on the capio server using - the `-l` option; -- `CAPIO_LOG_DIR` This environment variable is defined only for capio_posix applications and specifies the directory - name to which capio will be created. If this variable is not defined, capio will log by default to `capio_logs`. An - equivalent behaviour can be set on the capio server using the `-d` option; -- `CAPIO_CACHE_LINES`: This environment variable controls how many lines of cache are presents between posix and server - applications. defaults to 10 lines; -- `CAPIO_CACHE_LINE_SIZE`: This environment variable controls the size of a single cache line. defaults to 256KB; +[capio.storage] +file_initial_size = 4294967296 +prefetch_data_size = 0 -#### Server only environment variable +[capio.cache] +lines = 10 +line_size = 262144 +``` + +### POSIX environment variables + +Preloaded applications remain separate processes and use these environment variables to connect to the configured +server: -- `CAPIO_FILE_INIT_SIZE`: This environment variable defines the default size of pre allocated memory for a new file - handled by capio. Defaults to 4MB. Bigger sizes will reduce the overhead of malloc but will fill faster node memory. - Value has to be expressed in bytes; -- `CAPIO_PREFETCH_DATA_SIZE`: If this variable is set, then data transfers between nodes will be always, at least of the - given value in bytes; +#### Logging + +- `CAPIO_LOG_LEVEL` controls the application log level. This variable works only + if `-DCAPIO_LOG=TRUE` was specified during cmake phase; +- `CAPIO_LOG_PREFIX` specifies the prefix of + the logfile name to which capio will log to. The default value is `posix_thread_`, which means that capio will log by + default to a set of files called `posix_thread_*.log`; +- `CAPIO_LOG_DIR` specifies the log directory. It defaults to `capio_logs`. -#### Posix only environment variable +#### Runtime > [!WARNING] > The following variables are mandatory. If not provided to a posix, application, CAPIO will not be able to correctly -> handle the -> application, according to the specifications given from the json configuration file! +> handle the application according to the workflow configuration. -- `CAPIO_WORKFLOW_NAME`: This environment variable is used to define the scope of a workflow for a given step. Needs to - be the same one as the field `"name"` inside the json configuration file; +- `CAPIO_DIR`: must match `capio.directory` in the server TOML configuration; +- `CAPIO_WORKFLOW_NAME`: must match `capiocl.workflow_name` in the server TOML configuration; - `CAPIO_APP_NAME`: This environment variable defines the app name within a workflow for a given step; +- `CAPIO_CACHE_LINES`: must match `capio.cache.lines`; defaults to 10; +- `CAPIO_CACHE_LINE_SIZE`: must match `capio.cache.line_size`; defaults to 256 KiB. ## How to inject streaming capabilities into your workflow diff --git a/capio/common/shm.hpp b/capio/common/shm.hpp index 934337c4b..5bbe0ffff 100644 --- a/capio/common/shm.hpp +++ b/capio/common/shm.hpp @@ -9,6 +9,8 @@ #include #include +#include "common/syscall.hpp" + #ifdef __CAPIO_POSIX #include "calf/SyscallLogger.h" #else diff --git a/capio/server/CMakeLists.txt b/capio/server/CMakeLists.txt index 45f717a5e..9b3e57ca3 100644 --- a/capio/server/CMakeLists.txt +++ b/capio/server/CMakeLists.txt @@ -14,15 +14,12 @@ endif () # External projects ##################################### FetchContent_Declare( - args - GIT_REPOSITORY https://github.com/Taywee/args.git - GIT_TAG 6.4.7 + tomlplusplus + GIT_REPOSITORY https://github.com/marzer/tomlplusplus.git + GIT_TAG v3.4.0 ) -set(ARGS_BUILD_EXAMPLE OFF CACHE INTERNAL "") -set(ARGS_BUILD_UNITTESTS OFF CACHE INTERNAL "") - -FetchContent_MakeAvailable(args capio_cl mtcl) +FetchContent_MakeAvailable(capio_cl mtcl tomlplusplus) ##################################### @@ -51,9 +48,13 @@ target_sources(${TARGET_NAME} PRIVATE target_include_directories(${TARGET_NAME} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include ${capio_cl_SOURCE_DIR} + ${TOMLPLUSPLUS_SOURCE_DIR}/include ) target_include_directories(${TARGET_NAME} SYSTEM PRIVATE ${mtcl_SOURCE_DIR}/include) + +target_link_libraries(${TARGET_NAME} PRIVATE tomlplusplus::tomlplusplus) + ##################################### # set CALF logger component name ##################################### @@ -64,7 +65,7 @@ calf_set_default_log_dir(${TARGET_NAME} "./capio_logs") ##################################### # Link libraries ##################################### -target_link_libraries(${TARGET_NAME} PRIVATE pthread rt stdc++fs libcapio_cl args) +target_link_libraries(${TARGET_NAME} PRIVATE pthread rt stdc++fs libcapio_cl) if (MPI_CXX_FOUND) target_compile_definitions(${TARGET_NAME} PRIVATE CAPIO_HAS_MPI) target_link_libraries(${TARGET_NAME} PRIVATE MPI::MPI_CXX) diff --git a/capio/server/capio_server.cpp b/capio/server/capio_server.cpp index 7e9778a15..9fe7358ce 100644 --- a/capio/server/capio_server.cpp +++ b/capio/server/capio_server.cpp @@ -23,14 +23,12 @@ #include "utils/capiocl_adapter.hpp" #include "client-manager/client_manager.hpp" -#include "common/env.hpp" #include "common/requests.hpp" #include "common/semaphore.hpp" #include "remote/backend.hpp" #include "remote/discovery.hpp" #include "storage/capio_file.hpp" #include "utils/common.hpp" -#include "utils/env.hpp" #include "utils/types.hpp" ClientManager *client_manager; @@ -39,8 +37,8 @@ Backend *backend; DiscoveryService *discovery_service; #include "handlers.hpp" -#include "utils/cli_parser.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" #include "utils/signals.hpp" #include "remote/listener.hpp" @@ -94,7 +92,7 @@ static constexpr std::array build_request_handle setup_signal_handlers(); backend->handshake_servers(); - storage_manager->addDirectory(getpid(), get_capio_dir()); + storage_manager->addDirectory(getpid(), get_server_capio_dir()); internal_server_sem.unlock(); @@ -115,30 +113,19 @@ int main(int argc, char **argv) { Semaphore internal_server_sem(0); + const auto configuration = parse_cli(argc, argv); + for (const auto line : CAPIO_LOG_SERVER_BANNER) { CALF_PRINT("%s", line); } + configure_server_runtime( + configuration.capio_dir, configuration.cache_lines, configuration.cache_line_size, + configuration.capio_file_default_init_size, configuration.capio_prefetch_data_size); - const auto configuration = parseCLI(argc, argv); - - if (configuration.capio_cl_dynamic_config) { - capio_cl_engine = new capiocl::engine::Engine(); - capio_cl_engine->startApiServer(); - } else if (!configuration.capio_cl_config_path.empty()) { - capio_cl_engine = capiocl::parser::Parser::parse(configuration.capio_cl_config_path, - configuration.capio_cl_resolve_path, - configuration.store_all_in_memory); - } else { - capio_cl_engine = new capiocl::engine::Engine(); - capio_cl_engine->setWorkflowName(get_capio_workflow_name()); - } + capio_cl_engine = capiocl::parser::Parser::parse(configuration.capio_cl_config); UPDATE_CALF_WORKFLOW_NAME(capio_cl_engine->getWorkflowName()); - if (configuration.store_all_in_memory) { - capio_cl_engine->setAllStoreInMemory(); - } - capio_cl_engine->print(); discovery_service = select_discovery_service(configuration); diff --git a/capio/server/include/handlers/getdents.hpp b/capio/server/include/handlers/getdents.hpp index 20f8299bf..fc6bc2e2c 100644 --- a/capio/server/include/handlers/getdents.hpp +++ b/capio/server/include/handlers/getdents.hpp @@ -3,11 +3,10 @@ #include -#include "posix/utils/env.hpp" - #include "remote/backend.hpp" #include "remote/requests.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" extern StorageManager *storage_manager; extern Backend *backend; @@ -41,7 +40,7 @@ inline void handle_getdents(int tid, int fd, long int count) { const std::string &app_name = client_manager->getAppName(tid); const std::filesystem::path &path_to_check = storage_manager->getPath(tid, fd); - const std::filesystem::path &capio_dir = get_capio_dir(); + const std::filesystem::path &capio_dir = get_server_capio_dir(); bool is_prod = CapioCLEngine::get().isProducer(path_to_check, app_name); auto file_location_opt = get_file_location_opt(path_to_check); diff --git a/capio/server/include/handlers/read.hpp b/capio/server/include/handlers/read.hpp index e05ffc63e..213039572 100644 --- a/capio/server/include/handlers/read.hpp +++ b/capio/server/include/handlers/read.hpp @@ -8,6 +8,7 @@ #include "remote/requests.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" extern Backend *backend; @@ -135,7 +136,7 @@ inline void handle_read(int tid, int fd, off64_t count) { START_LOG(gettid(), "call(tid=%d, fd=%d, count=%ld)", tid, fd, count); const std::filesystem::path &path = storage_manager->getPath(tid, fd); - const std::filesystem::path &capio_dir = get_capio_dir(); + const std::filesystem::path &capio_dir = get_server_capio_dir(); const std::string &app_name = client_manager->getAppName(tid); bool is_prod = CapioCLEngine::get().isProducer(path, app_name) || client_manager->isProducer(tid, path); diff --git a/capio/server/include/handlers/stat.hpp b/capio/server/include/handlers/stat.hpp index e7e253061..683406e59 100644 --- a/capio/server/include/handlers/stat.hpp +++ b/capio/server/include/handlers/stat.hpp @@ -9,6 +9,7 @@ #include "remote/requests.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" #include "utils/types.hpp" #include "client-manager/client_manager.hpp" @@ -67,7 +68,7 @@ inline void reply_stat(int tid, const std::filesystem::path &path) { CapioFile &c_file = (c_file_opt) ? c_file_opt->get() : storage_manager->add(path, false, get_file_initial_size()); LOG("Obtained capio file. ready to reply to client"); - const std::filesystem::path &capio_dir = get_capio_dir(); + const std::filesystem::path &capio_dir = get_server_capio_dir(); LOG("Obtained capio_dir"); if (!file_location_opt) { LOG("File is now present from remote node. retrieving file again."); diff --git a/capio/server/include/remote/discovery.hpp b/capio/server/include/remote/discovery.hpp index d47a889ca..c416b3e40 100644 --- a/capio/server/include/remote/discovery.hpp +++ b/capio/server/include/remote/discovery.hpp @@ -6,7 +6,7 @@ #include #include "common/constants.hpp" -#include "utils/cli_parser.hpp" +#include "utils/runtime_configuration.hpp" #include "utils/shm_canary.hpp" /** diff --git a/capio/server/include/remote/listener.hpp b/capio/server/include/remote/listener.hpp index b19e4c0cc..987e36f4a 100644 --- a/capio/server/include/remote/listener.hpp +++ b/capio/server/include/remote/listener.hpp @@ -9,7 +9,7 @@ #include "remote/backend/include.hpp" #include "remote/handlers/read.hpp" #include "remote/handlers/stat.hpp" -#include "utils/cli_parser.hpp" +#include "utils/runtime_configuration.hpp" #include #include diff --git a/capio/server/include/utils/cli_parser.hpp b/capio/server/include/utils/cli_parser.hpp deleted file mode 100644 index f6091d141..000000000 --- a/capio/server/include/utils/cli_parser.hpp +++ /dev/null @@ -1,20 +0,0 @@ -#ifndef CAPIO_CLI_PARSER_HPP -#define CAPIO_CLI_PARSER_HPP -#include - -struct CapioParsedConfig { - std::string backend_name; - std::string discovery_interface; - std::string mcast_addr; - unsigned int mcast_port; - std::string token_directory; - std::string backend_options; - std::string capio_cl_config_path; - std::string capio_cl_resolve_path; - bool capio_cl_dynamic_config = false; - bool store_all_in_memory = false; -}; - -CapioParsedConfig parseCLI(int argc, char **argv); - -#endif // CAPIO_CLI_PARSER_HPP diff --git a/capio/server/include/utils/env.hpp b/capio/server/include/utils/env.hpp deleted file mode 100644 index 18b9588d0..000000000 --- a/capio/server/include/utils/env.hpp +++ /dev/null @@ -1,41 +0,0 @@ -#ifndef CAPIO_SERVER_UTILS_ENV_HPP -#define CAPIO_SERVER_UTILS_ENV_HPP - -#include - -#include "calf/StlLogger.h" -#include "common/constants.hpp" - -off64_t get_file_initial_size() { - START_LOG(gettid(), "call()"); - static off64_t file_initial_size = 0; - if (file_initial_size == 0) { - char *val; - val = std::getenv("CAPIO_FILE_INIT_SIZE"); - if (val != nullptr) { - file_initial_size = std::strtol(val, nullptr, 10); - } else { - file_initial_size = CAPIO_DEFAULT_FILE_INITIAL_SIZE; - } - } - - return file_initial_size; -} - -off64_t get_prefetch_data_size() { - START_LOG(gettid(), "call()"); - static off64_t prefetch_data_size = -1; - if (prefetch_data_size == -1) { - char *val; - val = std::getenv("CAPIO_PREFETCH_DATA_SIZE"); - if (val != nullptr) { - prefetch_data_size = std::strtol(val, nullptr, 10); - } else { - prefetch_data_size = 0; - } - } - - return prefetch_data_size; -} - -#endif // CAPIO_SERVER_UTILS_ENV_HPP diff --git a/capio/server/include/utils/runtime_configuration.hpp b/capio/server/include/utils/runtime_configuration.hpp new file mode 100644 index 000000000..6ddcaca50 --- /dev/null +++ b/capio/server/include/utils/runtime_configuration.hpp @@ -0,0 +1,56 @@ +#ifndef CAPIO_RUNTIME_CONFIGURATION_HPP +#define CAPIO_RUNTIME_CONFIGURATION_HPP +#include "capiocl/configuration.h" +#include "common/constants.hpp" + +#include +#include +#include +#include + +using RuntimeConfigMap = std::unordered_map; + +struct CapioParsedConfig { + std::filesystem::path capio_dir; + std::string backend_name; + std::string discovery_interface; + std::string mcast_addr; + unsigned int mcast_port = CAPIO_MCAST_ADV_DEFAULT_PORT; + std::string token_directory; + std::string backend_options; + capiocl::configuration::CapioClConfiguration capio_cl_config; + bool continue_on_error = false; + long cache_lines = CAPIO_CACHE_LINES_DEFAULT; + long cache_line_size = CAPIO_CACHE_LINE_SIZE_DEFAULT; + off64_t capio_file_default_init_size = CAPIO_DEFAULT_FILE_INITIAL_SIZE; + off64_t capio_prefetch_data_size = 0; +}; + +CapioParsedConfig parse_config(const std::filesystem::path &path); +CapioParsedConfig default_config(); +CapioParsedConfig parse_cli(int argc, char **argv); +void write_default_config(std::ostream &output); + +inline std::filesystem::path server_capio_dir = std::filesystem::current_path(); +inline long server_cache_lines = CAPIO_CACHE_LINES_DEFAULT; +inline long server_cache_line_size = CAPIO_CACHE_LINE_SIZE_DEFAULT; +inline off64_t server_file_initial_size = CAPIO_DEFAULT_FILE_INITIAL_SIZE; +inline off64_t server_prefetch_data_size = 0; + +inline void configure_server_runtime(const std::filesystem::path &capio_dir, long cache_lines, + long cache_line_size, off64_t file_initial_size, + off64_t prefetch_data_size) { + server_capio_dir = capio_dir; + server_cache_lines = cache_lines; + server_cache_line_size = cache_line_size; + server_file_initial_size = file_initial_size; + server_prefetch_data_size = prefetch_data_size; +} + +inline const std::filesystem::path &get_server_capio_dir() { return server_capio_dir; } +inline long get_server_cache_lines() { return server_cache_lines; } +inline long get_server_cache_line_size() { return server_cache_line_size; } +inline off64_t get_file_initial_size() { return server_file_initial_size; } +inline off64_t get_prefetch_data_size() { return server_prefetch_data_size; } + +#endif // CAPIO_RUNTIME_CONFIGURATION_HPP diff --git a/capio/server/src/cli_parser.cpp b/capio/server/src/cli_parser.cpp deleted file mode 100644 index 018012f14..000000000 --- a/capio/server/src/cli_parser.cpp +++ /dev/null @@ -1,149 +0,0 @@ -#include "utils/cli_parser.hpp" - -#include "calf/StdOutLogger.h" -#include "calf/StlLogger.h" - -#include "common/constants.hpp" -#include "utils/common.hpp" - -#include - -CapioParsedConfig parseCLI(int argc, char **argv) { - CapioParsedConfig capio_config; - - args::ArgumentParser parser(CAPIO_SERVER_ARG_PARSER_PRE, CAPIO_SERVER_ARG_PARSER_EPILOGUE); - parser.LongSeparator(" "); - parser.LongPrefix("--"); - parser.ShortPrefix("-"); - - args::Group arguments(parser, "Arguments"); - args::HelpFlag help(arguments, "help", "Display this help menu", {'h', "help"}); - args::ValueFlag resolve_prefix(arguments, "resolve-prefix", - CAPIO_SERVER_ARG_PARSER_RESOLVE_PREFIX_OPT_HELP, - {'r', "resolve-prefix"}); - - args::ValueFlag config(arguments, "filename", - CAPIO_SERVER_ARG_PARSER_CONFIG_OPT_HELP, {'c', "config"}); - args::Flag noConfigFile(arguments, "no-config", - CAPIO_SERVER_ARG_PARSER_CONFIG_NO_CONF_FILE_HELP, {"no-config"}); - args::ValueFlag backend_flag( - arguments, "backend", CAPIO_SERVER_ARG_PARSER_CONFIG_BACKEND_HELP, {'b', "backend"}); - args::ValueFlag discovery_flag( - arguments, "discovery", CAPIO_SERVER_ARG_PARSER_DISCOVERY_HELP, {"discovery"}); - args::ValueFlag mcast_addr_flag( - arguments, "address", CAPIO_SERVER_ARG_PARSER_MCAST_ADDR_HELP, {"mcast-addr"}); - args::ValueFlag mcast_port_flag( - arguments, "port", CAPIO_SERVER_ARG_PARSER_MCAST_PORT_HELP, {"mcast-port"}); - args::ValueFlag token_directory_flag( - arguments, "directory", CAPIO_SERVER_ARG_PARSER_TOKEN_DIRECTORY_HELP, {"token-directory"}); - args::ValueFlag backend_options_flag( - arguments, "options", CAPIO_SERVER_ARG_PARSER_BACKEND_OPTIONS_HELP, {"backend-options"}); - - args::Flag continueOnErrorFlag(arguments, "continue-on-error", - CAPIO_SERVER_ARG_PARSER_CONFIG_NCONTINUE_ON_ERROR_HELP, - {"continue-on-error"}); - args::Flag mem_only_flag(arguments, "mem-only", - CAPIO_SERVER_ARG_PARSER_STORE_ALL_IN_MEMORY_OPT_HELP, {"mem-only"}); - - try { - parser.ParseCLI(argc, argv); - } catch (args::Help &) { - std::cout << parser; - exit(EXIT_SUCCESS); - } catch (args::ParseError &e) { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, "%s", e.what()); - std::cerr << parser; - exit(EXIT_FAILURE); - } catch (args::ValidationError &e) { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, "%s", e.what()); - std::cerr << parser; - exit(EXIT_FAILURE); - } - - capio_config.discovery_interface = - discovery_flag ? args::get(discovery_flag) : CAPIO_MCAST_PROTO_FLAG; - if (capio_config.discovery_interface != CAPIO_MCAST_PROTO_FLAG && - capio_config.discovery_interface != CAPIO_FS_PROTO_FLAG) { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, - "Invalid discovery interface: %s. Expected mcast or fs.", - capio_config.discovery_interface.c_str()); - exit(EXIT_FAILURE); - } - capio_config.mcast_addr = - mcast_addr_flag ? args::get(mcast_addr_flag) : CAPIO_MCAST_ADV_DEFAULT_ADDR; - capio_config.mcast_port = - mcast_port_flag ? args::get(mcast_port_flag) : CAPIO_MCAST_ADV_DEFAULT_PORT; - capio_config.token_directory = - token_directory_flag ? args::get(token_directory_flag) : ".capio_tokens/"; - if (backend_options_flag) { - capio_config.backend_options = args::get(backend_options_flag); - } - - if (continueOnErrorFlag) { -#ifdef CAPIO_LOG - continue_on_error = true; - for (const auto line : CAPIO_LOG_SERVER_CLI_CONT_ON_ERR_WARNING) { - CALF_PRINT("%s", line); - } - -#else - CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, - "--continue-on-error flag given, but logger is not compiled into " - "CAPIO. Flag is ignored."); -#endif - } - -#ifdef CAPIO_LOG - for (const auto line : CAPIO_LOG_SERVER_CLI_LOGGING_ENABLED_WARNING) { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, "%s", line); - } - - auto log = new Logger(__func__, __FILE__, __LINE__, gettid(), "Created new log file"); - CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, "started logging to logfile %s", - log->getLogFileName().c_str()); -#endif - - if (mem_only_flag) { - capio_config.store_all_in_memory = args::get(mem_only_flag); - } - - if (config) { - - capio_config.capio_cl_config_path = args::get(config); - - if (std::string token = args::get(config); token == "dynamic") { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, - "Starting CAPIO-CL engine with dynamic configuration"); - capio_config.capio_cl_dynamic_config = true; - - } else { - std::filesystem::path resolve_path = ""; - - if (resolve_prefix) { - capio_config.capio_cl_resolve_path = args::get(resolve_prefix); - } - } - - } else if (noConfigFile) { - CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, "skipping config file parsing."); - CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, - "Obtained from environment variable current workflow name: %s", - get_capio_workflow_name().c_str()); - } else { - CALF_PRINT_COLOR( - CALF_CLI_LEVEL_ERROR, - "Error: no config file provided. To skip config file use --no-config option!"); -#ifdef CAPIO_LOG - log->log("no config file provided, and --no-config not provided"); -#endif - exit(EXIT_FAILURE); - } - - CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, "CAPIO_DIR=%s", get_capio_dir().c_str()); - // Backend selection phase - if (backend_flag) { - capio_config.backend_name = args::get(backend_flag); - } - - return capio_config; -} diff --git a/capio/server/src/client_manager.cpp b/capio/server/src/client_manager.cpp index 34b30001f..0f3d6ba07 100644 --- a/capio/server/src/client_manager.cpp +++ b/capio/server/src/client_manager.cpp @@ -8,12 +8,15 @@ #include "common/queue.hpp" #include "utils/capiocl_adapter.hpp" #include "utils/common.hpp" +#include "utils/runtime_configuration.hpp" ClientManager::ClientDataBuffers::ClientDataBuffers(const std::string &clientToServerName, const std::string &serverToClientName, const std::string &wf_name) - : ClientToServer(clientToServerName, get_cache_lines(), get_cache_line_size(), wf_name), - ServerToClient(serverToClientName, get_cache_lines(), get_cache_line_size(), wf_name) {} + : ClientToServer(clientToServerName, get_server_cache_lines(), get_server_cache_line_size(), + wf_name), + ServerToClient(serverToClientName, get_server_cache_lines(), get_server_cache_line_size(), + wf_name) {} ClientManager::ClientManager() : requests{SHM_COMM_CHAN_NAME, CAPIO_REQ_BUFF_CNT, CAPIO_REQ_MAX_SIZE, @@ -191,4 +194,4 @@ int ClientManager::readNextRequest(char *str) { return -1; } return code; -} \ No newline at end of file +} diff --git a/capio/server/src/runtime_configuration.cpp b/capio/server/src/runtime_configuration.cpp new file mode 100644 index 000000000..34db369fc --- /dev/null +++ b/capio/server/src/runtime_configuration.cpp @@ -0,0 +1,283 @@ +#include "utils/runtime_configuration.hpp" + +#include "calf/StdOutLogger.h" +#include "calf/StlLogger.h" + +#include "utils/common.hpp" + +#include "toml++/toml.hpp" + +#include +#include +#include +#include +#include +#include + +namespace { + +void flatten(const toml::table &table, RuntimeConfigMap &config, const std::string &prefix = "") { + for (const auto &[key, value] : table) { + const auto full_key = + prefix.empty() ? std::string{key.str()} : prefix + "." + std::string{key.str()}; + if (const auto *nested = value.as_table()) { + flatten(*nested, config, full_key); + } else if (value.is_string()) { + config[full_key] = value.as_string()->get(); + } else if (value.is_boolean()) { + config[full_key] = value.as_boolean()->get() ? "true" : "false"; + } else if (value.is_integer()) { + config[full_key] = std::to_string(value.as_integer()->get()); + } else { + throw std::runtime_error("unsupported TOML value for '" + full_key + "'"); + } + } +} + +std::string get(const RuntimeConfigMap &config, const std::string &key, + const std::string &fallback = "") { + const auto value = config.find(key); + return value == config.end() ? fallback : value->second; +} + +bool get_bool(const RuntimeConfigMap &config, const std::string &key, bool fallback = false) { + const auto value = get(config, key, fallback ? "true" : "false"); + if (value == "true") { + return true; + } + if (value == "false") { + return false; + } + throw std::runtime_error("invalid boolean for '" + key + "'"); +} + +long long get_integer(const RuntimeConfigMap &config, const std::string &key, long long fallback) { + const auto value = config.find(key); + return value == config.end() ? fallback : std::stoll(value->second); +} + +void write_comment(std::ostream &output, std::string_view comment) { + constexpr size_t max_content_length = 118; // Account for the "# " prefix. + while (!comment.empty()) { + const auto newline = comment.find('\n'); + auto line = comment.substr(0, newline); + while (line.size() > max_content_length) { + auto split = line.rfind(' ', max_content_length); + if (split == std::string_view::npos || split == 0) { + split = max_content_length; + } + output << "# " << line.substr(0, split) << '\n'; + line.remove_prefix(split); + while (!line.empty() && line.front() == ' ') { + line.remove_prefix(1); + } + } + while (!line.empty() && line.back() == ' ') { + line.remove_suffix(1); + } + output << "# " << line << '\n'; + if (newline == std::string_view::npos) { + break; + } + comment.remove_prefix(newline + 1); + } +} + +void write_usage(std::ostream &output) { + output << "Usage:\n" + " capio_server Start with a TOML configuration file\n" + " capio_server --defconf Start with built-in defaults\n" + " capio_server --genconf Write built-in defaults to ./default.toml\n"; +} + +} // namespace + +void write_default_config(std::ostream &output) { + output << "[capiocl]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_CONFIG_NO_CONF_FILE_HELP); + output << "workflow_name = \"CAPIO\"\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_CONFIG_OPT_HELP); + output << "config_path = \"\"\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_RESOLVE_PREFIX_OPT_HELP); + output << "resolve_path = \"\"\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_STORE_ALL_IN_MEMORY_OPT_HELP); + output << "store_all_in_memory = false\n\n" + "[capiocl.monitor.mcast]\n" + "enabled = true\n" + "delay_ms = 300\n" + "commit = { ip = \"224.224.224.1\", port = 12345 }\n" + "homenode = { ip = \"224.224.224.2\", port = 12345 }\n\n" + "[capiocl.monitor.filesystem]\n" + "enabled = true\n\n" + "[capiocl.dynamic_api]\n" + "enabled = false\n" + "ip = \"224.224.224.3\"\n" + "port = 11223\n\n" + "[capio]\n"; + output << "# Root directory managed by CAPIO.\n" + "directory = \".\"\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_CONFIG_NCONTINUE_ON_ERROR_HELP); + output << "continue_on_error = false\n\n" + "[capio.storage]\n" + "file_initial_size = " + << CAPIO_DEFAULT_FILE_INITIAL_SIZE + << "\n" + "prefetch_data_size = 0\n\n" + "[capio.cache]\n" + "lines = " + << CAPIO_CACHE_LINES_DEFAULT << "\nline_size = " << CAPIO_CACHE_LINE_SIZE_DEFAULT + << "\n\n" + "[capio.discovery_service]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_DISCOVERY_HELP); + output << "type = \"mcast\"\n" + "interval_ms = 1000\n\n" + "[capio.discovery_service.mcast]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_MCAST_ADDR_HELP); + output << "addr = \"" << CAPIO_MCAST_ADV_DEFAULT_ADDR << "\"\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_MCAST_PORT_HELP); + output << "port = " << CAPIO_MCAST_ADV_DEFAULT_PORT + << "\n\n" + "[capio.discovery_service.fs]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_TOKEN_DIRECTORY_HELP); + output << "token_directory = \".capio_tokens/\"\n\n" + "[capio.backend]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_CONFIG_BACKEND_HELP); + output << "type = \"none\"\n\n" + "[capio.backend.mtcl]\n"; + write_comment(output, CAPIO_SERVER_ARG_PARSER_BACKEND_OPTIONS_HELP); + output << "proto = \"" << CAPIO_MTCL_DEFAULT_PROTOCOL + << "\"\n" + "listen_address = \"0.0.0.0\"\n" + "port = " + << CAPIO_MTCL_DEFAULT_PORT << "\npoll_interval_us = " << CAPIO_MTCL_DEFAULT_POLL_INTERVAL + << '\n'; +} + +static CapioParsedConfig parse_table(const toml::table &table) { + RuntimeConfigMap flattened; + flatten(table, flattened); + + CapioParsedConfig result; + RuntimeConfigMap capio_cl_config; + constexpr char capiocl_prefix[] = "capiocl."; + for (const auto &[key, value] : flattened) { + if (key.compare(0, sizeof(capiocl_prefix) - 1, capiocl_prefix) == 0) { + capio_cl_config.emplace(key, value); + } + } + result.capio_cl_config = + capiocl::configuration::CapioClConfiguration(std::move(capio_cl_config)); + + const auto capio_dir = get(flattened, "capio.directory", "."); + if (!std::filesystem::is_directory(capio_dir)) { + throw std::runtime_error("capio.directory must be an existing directory"); + } + result.capio_dir = std::filesystem::canonical(capio_dir); + + result.discovery_interface = + get(flattened, "capio.discovery_service.type", CAPIO_MCAST_PROTO_FLAG); + if (result.discovery_interface != CAPIO_MCAST_PROTO_FLAG && + result.discovery_interface != CAPIO_FS_PROTO_FLAG) { + throw std::runtime_error("capio.discovery_service.type must be 'mcast' or 'fs'"); + } + result.mcast_addr = + get(flattened, "capio.discovery_service.mcast.addr", CAPIO_MCAST_ADV_DEFAULT_ADDR); + result.mcast_port = static_cast( + get_integer(flattened, "capio.discovery_service.mcast.port", CAPIO_MCAST_ADV_DEFAULT_PORT)); + result.token_directory = + get(flattened, "capio.discovery_service.fs.token_directory", ".capio_tokens/"); + + result.backend_name = get(flattened, "capio.backend.type", "none"); + const auto protocol = get(flattened, "capio.backend.mtcl.proto", CAPIO_MTCL_DEFAULT_PROTOCOL); + const auto port = get(flattened, "capio.backend.mtcl.port", CAPIO_MTCL_DEFAULT_PORT); + const auto poll_interval = get_integer(flattened, "capio.backend.mtcl.poll_interval_us", + CAPIO_MTCL_DEFAULT_POLL_INTERVAL); + result.backend_options = protocol + ":" + port + "@" + std::to_string(poll_interval); + + result.capio_file_default_init_size = + get_integer(flattened, "capio.storage.file_initial_size", CAPIO_DEFAULT_FILE_INITIAL_SIZE); + result.capio_prefetch_data_size = get_integer(flattened, "capio.storage.prefetch_data_size", 0); + result.cache_lines = get_integer(flattened, "capio.cache.lines", CAPIO_CACHE_LINES_DEFAULT); + result.cache_line_size = + get_integer(flattened, "capio.cache.line_size", CAPIO_CACHE_LINE_SIZE_DEFAULT); + if (result.cache_lines <= 0 || result.cache_line_size <= 0 || + result.capio_file_default_init_size <= 0 || result.capio_prefetch_data_size < 0) { + throw std::runtime_error("CAPIO storage and cache sizes must be positive"); + } + result.continue_on_error = get_bool(flattened, "capio.continue_on_error"); + return result; +} + +CapioParsedConfig parse_config(const std::filesystem::path &path) { + return parse_table(toml::parse_file(path.string())); +} + +CapioParsedConfig default_config() { + std::ostringstream output; + write_default_config(output); + return parse_table(toml::parse(output.str())); +} + +CapioParsedConfig parse_cli(int argc, char **argv) { + if (argc == 2 && std::string{argv[1]} == "--genconf") { + std::ofstream output("default.toml"); + if (!output) { + CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, "Failed to create default.toml"); + std::exit(EXIT_FAILURE); + } + write_default_config(output); + output.close(); + if (!output) { + CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, "Failed to write default.toml"); + std::exit(EXIT_FAILURE); + } + CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, "Generated default.toml"); + std::exit(EXIT_SUCCESS); + } + if (argc != 2 || std::string{argv[1]} == "--help") { + if (argc == 1) { + std::cerr << "No configuration provided. Choose a TOML file or a default mode.\n\n"; + } else if (argc != 2) { + std::cerr << "Expected exactly one argument.\n\n"; + } + write_usage(argc == 2 ? std::cout : std::cerr); + std::exit(argc == 2 ? EXIT_SUCCESS : EXIT_FAILURE); + } + + CapioParsedConfig config; + if (std::string{argv[1]} == "--defconf") { + config = default_config(); + } else { + try { + config = parse_config(argv[1]); + } catch (const std::exception &error) { + CALF_PRINT_COLOR(CALF_CLI_LEVEL_ERROR, "Failed to load configuration '%s': %s", argv[1], + error.what()); + std::exit(EXIT_FAILURE); + } + } + + if (config.continue_on_error) { +#ifdef CAPIO_LOG + continue_on_error = true; + for (const auto line : CAPIO_LOG_SERVER_CLI_CONT_ON_ERR_WARNING) { + CALF_PRINT("%s", line); + } +#else + CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, + "capio.continue_on_error is enabled, but logging is not compiled in"); +#endif + } + +#ifdef CAPIO_LOG + for (const auto line : CAPIO_LOG_SERVER_CLI_LOGGING_ENABLED_WARNING) { + CALF_PRINT_COLOR(CALF_CLI_LEVEL_WARNING, "%s", line); + } + auto log = new Logger(__func__, __FILE__, __LINE__, gettid(), "Created new log file"); + CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, "started logging to logfile %s", + log->getLogFileName().c_str()); +#endif + + CALF_PRINT_COLOR(CALF_CLI_LEVEL_INFO, "CAPIO directory=%s", config.capio_dir.c_str()); + return config; +} diff --git a/capio/server/src/shm_canary.cpp b/capio/server/src/shm_canary.cpp index 1e7a47ec8..12a7dafad 100644 --- a/capio/server/src/shm_canary.cpp +++ b/capio/server/src/shm_canary.cpp @@ -4,15 +4,15 @@ #include "calf/StlLogger.h" #include "common/constants.hpp" -#include "common/env.hpp" #include "common/shm.hpp" +#include "common/syscall.hpp" #include "utils/common.hpp" CapioShmCanary::CapioShmCanary(const std::string &capio_workflow_name) : _canary_name(capio_workflow_name) { START_LOG(capio_syscall(SYS_gettid), "call(capio_workflow_name: %s)", _canary_name.data()); if (_canary_name.empty()) { - _canary_name = get_capio_workflow_name(); + _canary_name = CAPIO_DEFAULT_WORKFLOW_NAME; } _shm_id = shm_open(_canary_name.data(), O_CREAT | O_EXCL, S_IRUSR | S_IWUSR); if (_shm_id == -1) { diff --git a/capio/server/src/storage_manager.cpp b/capio/server/src/storage_manager.cpp index 11172265a..c5aa2ba83 100644 --- a/capio/server/src/storage_manager.cpp +++ b/capio/server/src/storage_manager.cpp @@ -13,6 +13,7 @@ #include "utils/capiocl_adapter.hpp" #include "utils/common.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" #include "utils/shared_mutex.hpp" #include "utils/types.hpp" @@ -311,7 +312,7 @@ off64_t StorageManager::addDirectory(const pid_t tid, const std::filesystem::pat if (c_file.isFirstWrite()) { c_file.registerFirstWrite(); // TODO: it works only if there is one prod per file - if (is_capio_dir(path)) { + if (get_server_capio_dir() == path) { add_file_location(path, backend->get_node_name().c_str(), -1); } else { write_file_location(path); diff --git a/capio/tests/integration/docker-compose.yml b/capio/tests/integration/docker-compose.yml index 7eba3b1e9..97e438501 100644 --- a/capio/tests/integration/docker-compose.yml +++ b/capio/tests/integration/docker-compose.yml @@ -3,10 +3,9 @@ x-test-service: &test-service working_dir: /shared volumes: - shared_data:/shared - - discovery_tokens:/tokens - test_status:/status - coverage_reports:/coverage - - ./config.json:/config.json:ro + - ./config.json:/shared/config.json:ro expose: - "1234" networks: @@ -16,7 +15,8 @@ x-test-service: &test-service - -c - | set -e - CAPIO_DIR=/shared capio_server -c /config.json -b mtcl --discovery fs --token-directory /tokens & + printf '[capiocl]\nworkflow_name = "CAPIO"\nconfig_path = "config.json"\n[capio]\ndirectory = "%s"\n[capio.discovery_service]\ntype = "fs"\n[capio.discovery_service.fs]\ntoken_directory = ".capio_tokens/"\n[capio.backend]\ntype = "mtcl"\n' "$$PWD" > /tmp/capio.toml + capio_server /tmp/capio.toml & server_pid=$$! cleanup() { status=$$? @@ -86,7 +86,6 @@ services: volumes: shared_data: - discovery_tokens: test_status: coverage_reports: diff --git a/capio/tests/unit/server/CMakeLists.txt b/capio/tests/unit/server/CMakeLists.txt index 9a4f1246f..fb4b5bfaa 100644 --- a/capio/tests/unit/server/CMakeLists.txt +++ b/capio/tests/unit/server/CMakeLists.txt @@ -3,7 +3,7 @@ ##################################### set(TARGET_NAME capio_server_unit_tests) -FetchContent_MakeAvailable(capio_cl mtcl) +FetchContent_MakeAvailable(capio_cl mtcl tomlplusplus) set(TARGET_INCLUDE_FOLDER "${PROJECT_SOURCE_DIR}/capio/server") @@ -37,6 +37,7 @@ target_sources(${TARGET_NAME} PRIVATE target_include_directories(${TARGET_NAME} PRIVATE "${TARGET_INCLUDE_FOLDER}/include" ${capio_cl_SOURCE_DIR} + ${TOMLPLUSPLUS_SOURCE_DIR}/include ) target_include_directories(${TARGET_NAME} SYSTEM PRIVATE ${mtcl_SOURCE_DIR}/include) @@ -50,7 +51,8 @@ calf_set_default_log_dir(${TARGET_NAME} "./capio_logs") ##################################### # Link libraries ##################################### -target_link_libraries(${TARGET_NAME} PRIVATE GTest::gtest_main rt libcapio_cl args) +target_link_libraries(${TARGET_NAME} PRIVATE GTest::gtest_main rt libcapio_cl + tomlplusplus::tomlplusplus) if (MPI_CXX_FOUND) target_compile_definitions(${TARGET_NAME} PRIVATE CAPIO_HAS_MPI) target_link_libraries(${TARGET_NAME} PRIVATE MPI::MPI_CXX) diff --git a/capio/tests/unit/server/src/cli_parser.cpp b/capio/tests/unit/server/src/cli_parser.cpp deleted file mode 100644 index f01f0b0c7..000000000 --- a/capio/tests/unit/server/src/cli_parser.cpp +++ /dev/null @@ -1,51 +0,0 @@ -#include - -#include "common/constants.hpp" -#include "utils/cli_parser.hpp" - -namespace { - -CapioParsedConfig parse(std::vector arguments) { - std::vector argv; - argv.reserve(arguments.size()); - for (auto &argument : arguments) { - argv.push_back(argument.data()); - } - return parseCLI(static_cast(argv.size()), argv.data()); -} - -} // namespace - -TEST(CliParserTest, UsesDefaultsWithNoConfig) { - const auto config = parse({"capio_server", "--no-config"}); - - EXPECT_EQ(config.discovery_interface, CAPIO_MCAST_PROTO_FLAG); - EXPECT_EQ(config.mcast_addr, CAPIO_MCAST_ADV_DEFAULT_ADDR); - EXPECT_EQ(config.mcast_port, CAPIO_MCAST_ADV_DEFAULT_PORT); - EXPECT_EQ(config.token_directory, ".capio_tokens/"); - EXPECT_TRUE(config.backend_name.empty()); - EXPECT_FALSE(config.store_all_in_memory); -} - -TEST(CliParserTest, ParsesExplicitOptions) { - const auto config = parse({"capio_server", "--config", "config.json", "--resolve-prefix", - "/prefix", "--backend", "mtcl", "--backend-options", "TCP:7601@50", - "--discovery", "fs", "--mcast-addr", "239.1.2.3", "--mcast-port", - "12345", "--token-directory", "/tokens", "--mem-only"}); - - EXPECT_EQ(config.capio_cl_config_path, "config.json"); - EXPECT_EQ(config.capio_cl_resolve_path, "/prefix"); - EXPECT_EQ(config.backend_name, "mtcl"); - EXPECT_EQ(config.backend_options, "TCP:7601@50"); - EXPECT_EQ(config.discovery_interface, CAPIO_FS_PROTO_FLAG); - EXPECT_EQ(config.mcast_addr, "239.1.2.3"); - EXPECT_EQ(config.mcast_port, 12345); - EXPECT_EQ(config.token_directory, "/tokens"); - EXPECT_TRUE(config.store_all_in_memory); -} - -TEST(CliParserTest, RecognizesDynamicConfig) { - const auto config = parse({"capio_server", "--config", "dynamic"}); - - EXPECT_TRUE(config.capio_cl_dynamic_config); -} diff --git a/capio/tests/unit/server/src/main.cpp b/capio/tests/unit/server/src/main.cpp index c133c092f..173962702 100644 --- a/capio/tests/unit/server/src/main.cpp +++ b/capio/tests/unit/server/src/main.cpp @@ -3,11 +3,12 @@ #include "capiocl.hpp" #include "capiocl/engine.h" #include "client-manager/client_manager.hpp" -#include "common/env.hpp" +#include "common/constants.hpp" #include "remote/discovery.hpp" #include "storage/manager.hpp" #include "utils/capiocl_adapter.hpp" #include "utils/location.hpp" +#include "utils/runtime_configuration.hpp" capiocl::engine::Engine *capio_cl_engine = nullptr; StorageManager *storage_manager = nullptr; @@ -28,8 +29,10 @@ class ServerUnitTestEnvironment : public testing::Environment { explicit ServerUnitTestEnvironment() = default; void SetUp() override { + configure_server_runtime("/tmp", CAPIO_CACHE_LINES_DEFAULT, CAPIO_CACHE_LINE_SIZE_DEFAULT, + CAPIO_DEFAULT_FILE_INITIAL_SIZE, 0); capio_cl_engine = new capiocl::engine::Engine(false); - capio_cl_engine->setWorkflowName(get_capio_workflow_name()); + capio_cl_engine->setWorkflowName(CAPIO_DEFAULT_WORKFLOW_NAME); client_manager = new ClientManager(); storage_manager = new StorageManager(); discovery_service = new DiscoveryService(std::make_unique()); diff --git a/capio/tests/unit/server/src/runtime_configuration.cpp b/capio/tests/unit/server/src/runtime_configuration.cpp new file mode 100644 index 000000000..6613714e4 --- /dev/null +++ b/capio/tests/unit/server/src/runtime_configuration.cpp @@ -0,0 +1,104 @@ +#include + +#include "common/constants.hpp" +#include "utils/runtime_configuration.hpp" + +#include +#include +#include + +TEST(RuntimeConfigurationTest, ParsesCapioAndExtractsCapioClConfiguration) { + const auto path = std::filesystem::temp_directory_path() / "capio-runtime-config-test.toml"; + { + std::ofstream config(path); + config << R"( +[capiocl] +config_path = "workflow.json" +resolve_path = "prefix" +store_all_in_memory = true +[capiocl.monitor.mcast] +enabled = true +[capiocl.dynamic_api] +enabled = true + +[capio] +directory = "." +continue_on_error = true +[capio.storage] +file_initial_size = 4096 +prefetch_data_size = 1024 +[capio.cache] +lines = 20 +line_size = 8192 +[capio.discovery_service] +type = "fs" +[capio.discovery_service.mcast] +addr = "239.1.2.3" +port = 12345 +[capio.discovery_service.fs] +token_directory = ".tokens" +[capio.backend] +type = "mtcl" +[capio.backend.mtcl] +proto = "TCP" +port = 7601 +poll_interval_us = 50 +)"; + } + + const auto config = parse_config(path); + std::filesystem::remove(path); + + std::string config_path, resolve_path, dynamic_enabled, store_all_in_memory, multicast_enabled; + config.capio_cl_config.getParameter("capiocl.config_path", &config_path, ""); + config.capio_cl_config.getParameter("capiocl.resolve_path", &resolve_path, ""); + config.capio_cl_config.getParameter("capiocl.dynamic_api.enabled", &dynamic_enabled, "false"); + config.capio_cl_config.getParameter("capiocl.store_all_in_memory", &store_all_in_memory, + "false"); + config.capio_cl_config.getParameter("capiocl.monitor.mcast.enabled", &multicast_enabled, + "false"); + EXPECT_EQ(config_path, "workflow.json"); + EXPECT_EQ(resolve_path, "prefix"); + EXPECT_EQ(dynamic_enabled, "true"); + EXPECT_EQ(store_all_in_memory, "true"); + EXPECT_EQ(multicast_enabled, "true"); + EXPECT_EQ(config.discovery_interface, CAPIO_FS_PROTO_FLAG); + EXPECT_EQ(config.mcast_addr, "239.1.2.3"); + EXPECT_EQ(config.mcast_port, 12345); + EXPECT_EQ(config.token_directory, ".tokens"); + EXPECT_EQ(config.backend_name, "mtcl"); + EXPECT_EQ(config.backend_options, "TCP:7601@50"); + EXPECT_EQ(config.capio_dir, std::filesystem::current_path()); + EXPECT_EQ(config.cache_lines, 20); + EXPECT_EQ(config.cache_line_size, 8192); + EXPECT_EQ(config.capio_file_default_init_size, 4096); + EXPECT_EQ(config.capio_prefetch_data_size, 1024); + EXPECT_TRUE(config.continue_on_error); +} + +TEST(RuntimeConfigurationTest, GeneratesDocumentedDefaultConfiguration) { + std::ostringstream output; + write_default_config(output); + + EXPECT_NE(output.str().find(CAPIO_SERVER_ARG_PARSER_DISCOVERY_HELP), std::string::npos); + EXPECT_NE(output.str().find("# \t> mtcl"), std::string::npos); + std::istringstream lines(output.str()); + for (std::string line; std::getline(lines, line);) { + EXPECT_LE(line.size(), 120) << line; + } + + const auto path = std::filesystem::temp_directory_path() / "capio-default-config-test.toml"; + { + std::ofstream config(path); + config << output.str(); + } + const auto config = parse_config(path); + std::filesystem::remove(path); + + EXPECT_EQ(config.backend_name, "none"); + EXPECT_EQ(config.discovery_interface, CAPIO_MCAST_PROTO_FLAG); + + const auto defaults = default_config(); + EXPECT_EQ(defaults.backend_name, config.backend_name); + EXPECT_EQ(defaults.discovery_interface, config.discovery_interface); +} diff --git a/capio/tests/unit/server/src/storage_manager.cpp b/capio/tests/unit/server/src/storage_manager.cpp index 15bce6647..c1c136cd5 100644 --- a/capio/tests/unit/server/src/storage_manager.cpp +++ b/capio/tests/unit/server/src/storage_manager.cpp @@ -72,16 +72,11 @@ TEST_F(StorageManagerTestEnvironment, testInitDirectory) { } TEST_F(StorageManagerTestEnvironment, testAddDirectoryFailure) { - char *old_capio_dir = getenv("CAPIO_DIR"); - setenv("CAPIO_DIR", "/", 1); open_files_location(); storage_manager->add("/tmp", true, 0); EXPECT_EQ(storage_manager->addDirectory(1, "/tmp/newDirectoryFail"), 0); EXPECT_EQ(storage_manager->addDirectory(1, "/tmp/newDirectoryFail"), 1); - if (old_capio_dir != nullptr) { - setenv("CAPIO_DIR", old_capio_dir, 1); - } } TEST_F(StorageManagerTestEnvironment, testRemameFile) { diff --git a/capio/tests/unit/syscall/src/main.cpp b/capio/tests/unit/syscall/src/main.cpp index b3abf1939..66688bf9f 100644 --- a/capio/tests/unit/syscall/src/main.cpp +++ b/capio/tests/unit/syscall/src/main.cpp @@ -4,7 +4,7 @@ #include "common/syscall.hpp" char **build_args() { - char **args = (char **) malloc(4 * sizeof(uintptr_t)); + char **args = (char **) malloc(3 * sizeof(uintptr_t)); char const *command = std::getenv("CAPIO_SERVER_PATH"); if (command == nullptr) { @@ -13,13 +13,12 @@ char **build_args() { char const *path = std::getenv("CONFIG_PATH"); if (path == nullptr) { - path = "test_config.json"; + path = "test_config.toml"; } args[0] = strdup(command); - args[1] = strdup("-c"); - args[2] = strdup(path); - args[3] = (char *) nullptr; + args[1] = strdup(path); + args[2] = (char *) nullptr; return args; } diff --git a/examples/docker/README.md b/examples/docker/README.md index 5b54f983b..f7328d146 100644 --- a/examples/docker/README.md +++ b/examples/docker/README.md @@ -50,6 +50,10 @@ The last preliminary step is to create the `CAPIO_DIR` on every node. For this e ```bash docker compose -p example exec --user capio --index 1 capio \ mpirun -N 1 --hostfile /home/capio/hostfile mkdir -p /tmp/capio + +docker compose -p example exec --user capio --index 1 capio \ + sh -c 'printf "[capiocl]\nworkflow_name = \"CAPIO\"\n[capio]\ndirectory = \"/tmp/capio\"\n" \ + > /home/capio/server/capio.toml' ``` Note that the command requires MPI to execute one process per node using the `-N 1` option of the `mpirun` command, and specifies the nodes' hostnames through the `hostfile` you just generated. @@ -60,19 +64,17 @@ Finally, the CAPIO server can be started in background using the following comma docker compose -p example exec \ --detach \ --index 1 \ - --env CAPIO_DIR=/tmp/capio \ --user capio \ --workdir /home/capio/server \ capio \ mpirun \ -N 1 \ --hostfile /home/capio/hostfile \ - -x CAPIO_DIR \ - sh -c ' capio_server --no-config \ + sh -c 'capio_server /home/capio/server/capio.toml \ > server_${OMPI_COMM_WORLD_RANK}.out 2>&1' ``` -Let's examine some of the options introduced in the previous command. The `--detach` option allows to run a command in background. The `--env` option adds environment variables to the target container instance, in this case the one called `example-capio-1`. The `-x` option of the `mpirun` command propagates the specified list of environment variables to all nodes it targets. Finally, the `--workdir` option specifies that the CAPIO server should use the shared `/home/capio/server` directory as the working directory, to store logs and other configuration files. +Let's examine some of the options introduced in the previous command. The `--detach` option allows running a command in background. The TOML file supplies the server runtime configuration, including its managed directory. The `--workdir` option specifies that the CAPIO server should use the shared `/home/capio/server` directory to store logs and other files. ### Start the CAPIO application @@ -124,4 +126,3 @@ If the command succeeds, it should print something like this ✔ Container example-capio-1 Removed 0.4s ✔ Network example_capionet Removed 0.3s ``` -