Extended configuration options

README.md

@@ -158,6 +158,108 @@ This assumes that all build dependencies not fetched by cmake are available.
 
 ---
 
+## Runtime TOML Configuration
+
+Runtime behavior is configured with a TOML file loaded into `CapioClConfiguration`. This is separate from the JSON
+coordination-language document: the TOML file selects the JSON document, monitor backends, metadata storage, and dynamic
+API settings.
+
+### Complete example
+
+```toml
+# Workflow and optional JSON coordination-language document
+[capiocl]
+workflow_name = "my-workflow"
+config_path = "workflow.json"
+resolve_path = "/data/run-42"
+store_all_in_memory = false
+
+[capiocl.dynamic_api]
+enabled = false
+ip = "224.224.224.3"
+port = 11223
+
+[capiocl.monitor.filesystem]
+enabled = true
+metadata_dir = "/shared/trusted/run-42"
+
+[capiocl.monitor.mcast]
+enabled = false
+delay_ms = 300
+
+[capiocl.monitor.mcast.commit]
+ip = "224.224.224.1"
+port = 12345
+
+[capiocl.monitor.mcast.homenode]
+ip = "224.224.224.2"
+port = 12345
+```
+
+### Options
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| `capiocl.workflow_name` | string | JSON `name`, or `CAPIO` without JSON | Overrides the workflow name. |
+| `capiocl.config_path` | string/path | empty | JSON CAPIO-CL document to parse. An empty value creates a runtime-only engine. |
+| `capiocl.resolve_path` | string/path | empty | Prefix applied to relative paths in the JSON document. |
+| `capiocl.store_all_in_memory` | boolean | `false` | Marks every parsed data path for in-memory storage. |
+| `capiocl.dynamic_api.enabled` | boolean | `false` | Starts the dynamic configuration API. |
+| `capiocl.dynamic_api.ip` | string | `224.224.224.3` | Multicast address used by the dynamic API. |
+| `capiocl.dynamic_api.port` | integer | `11223` | UDP port used by the dynamic API. |
+| `capiocl.monitor.filesystem.enabled` | boolean | see below | Enables filesystem commit and home-node tokens. |
+| `capiocl.monitor.filesystem.metadata_dir` | string/path | empty | Trusted metadata root for persistent counted `ON_CLOSE` state. |
+| `capiocl.monitor.mcast.enabled` | boolean | see below | Enables multicast commit and home-node propagation. |
+| `capiocl.monitor.mcast.delay_ms` | integer | `300` | Delay before multicast status operations, in milliseconds. |
+| `capiocl.monitor.mcast.commit.ip` | string | `224.224.224.1` | Multicast group for commit state. |
+| `capiocl.monitor.mcast.commit.port` | integer | `12345` | UDP port for commit state. |
+| `capiocl.monitor.mcast.homenode.ip` | string | `224.224.224.2` | Multicast group for home-node state. |
+| `capiocl.monitor.mcast.homenode.port` | integer | `12345` | UDP port for home-node state. |
+
+Every CAPIO-CL option is under the top-level `capiocl` table. Other top-level tables may coexist in the same TOML file
+and are ignored by CAPIO-CL. When parsing a user-provided configuration, omitted
+`capiocl.monitor.filesystem.enabled` and `capiocl.monitor.mcast.enabled` values are `false`. `Engine()` and
+`CapioClConfiguration.loadDefaults()` use the built-in configuration, which enables both monitors. Set both values
+explicitly in deployed TOML files to avoid ambiguity.
+
+TOML booleans must be unquoted `true` or `false`, and ports/delays must be integers. Relative `capiocl.config_path`,
+`capiocl.resolve_path`, and `capiocl.monitor.filesystem.metadata_dir` values are interpreted from the process working
+directory. Unknown keys are retained but ignored by CAPIO-CL.
+
+### Filesystem metadata
+
+`capiocl.monitor.filesystem.metadata_dir` is required only when an `ON_CLOSE` rule commits after more than one close. It
+must name a trusted, non-attacker-writable directory unique to the workflow run. CAPIO-CL creates and owns a `capiocl`
+subdirectory beneath it. Multi-process and multi-node producers must share that directory through storage providing
+coherent atomic exclusive file creation, rename, and unlink. Ordinary commit and home-node token operations do not
+require this option.
+
+### Loading configuration
+
+```cpp
+#include "capiocl/configuration.h"
+#include "capiocl/parser.h"
+
+using capiocl::configuration::CapioClConfiguration;
+
+CapioClConfiguration config;
+config.load("runtime.toml");
+std::unique_ptr<capiocl::engine::Engine> engine(capiocl::parser::Parser::parse(config));
+```
+
+```python
+import py_capio_cl
+
+config = py_capio_cl.CapioClConfiguration()
+config.load("runtime.toml")
+engine = py_capio_cl.Parser.parse(config)
+```
+
+Configuration can also be constructed from a string map/dictionary. Values in that form must use their flattened keys
+and string representations, for example `{"capiocl.monitor.filesystem.enabled": "true"}`.
+
+---
+
 ## API Snapshot
 
 A simplified example of CAPIO-CL usage in C++:
@@ -210,4 +312,3 @@ Serializer.dump(engine, "my_workflow", "my_workflow.json")
 | Name                            | Role     | Contact                                                                                                          |
 |---------------------------------|----------|------------------------------------------------------------------------------------------------------------------|
 | **Alberto Riccardo Martinelli** | Designer | [email](mailto:albertoriccardo.martinelli@unito.it) \| [Homepage](https://alpha.di.unito.it/alberto-martinelli/) |
-

bindings/python_bindings.cpp

@@ -6,6 +6,7 @@
 #include <string>
 
 #include "capiocl.hpp"
+#include "capiocl/configuration.h"
 #include "capiocl/engine.h"
 #include "capiocl/monitor.h"
 #include "capiocl/parser.h"
@@ -18,6 +19,8 @@ PYBIND11_MODULE(_py_capio_cl, m) {
         "CAPIO-CL: Cross Application Programmable I/O - Coordination Language python bindings.";
 
     py::register_exception<capiocl::parser::ParserException>(m, "ParserException");
+    py::register_exception<capiocl::configuration::CapioClConfigurationException>(
+        m, "CapioClConfigurationException");
     py::register_exception<capiocl::serializer::SerializerException>(m, "SerializerException");
     py::register_exception<capiocl::monitor::MonitorException>(m, "MonitorException");
 
@@ -39,9 +42,16 @@ PYBIND11_MODULE(_py_capio_cl, m) {
     py::module_ VERSION = m.def_submodule("VERSION", "CAPIO-CL version");
     VERSION.attr("V1")  = py::str(capiocl::CAPIO_CL_VERSION::V1);
 
+    py::class_<capiocl::configuration::CapioClConfiguration>(m, "CapioClConfiguration")
+        .def(py::init<>())
+        .def(py::init<std::unordered_map<std::string, std::string>>(), py::arg("values"))
+        .def("load", &capiocl::configuration::CapioClConfiguration::load, py::arg("path"))
+        .def("loadDefaults", &capiocl::configuration::CapioClConfiguration::loadDefaults);
+
     py::class_<capiocl::engine::Engine>(
         m, "Engine", "The main CAPIO-CL engine for managing data communication and I/O operations.")
         .def(py::init<>())
+        .def(py::init<const capiocl::configuration::CapioClConfiguration &>(), py::arg("config"))
         .def("newFile", &capiocl::engine::Engine::newFile, py::arg("filename"))
         .def("print", &capiocl::engine::Engine::print)
         .def("contains", &capiocl::engine::Engine::contains, py::arg("path"))
@@ -119,8 +129,10 @@ PYBIND11_MODULE(_py_capio_cl, m) {
         .def(py::self == py::self);
 
     py::class_<capiocl::parser::Parser>(m, "Parser", "The CAPIO-CL Parser component.")
-        .def_static("parse", &capiocl::parser::Parser::parse, py::arg("source"),
-                    py::arg("resolve_prefix") = "", py::arg("store_only_in_memory") = false)
+        .def_static("parse",
+                    py::overload_cast<const capiocl::configuration::CapioClConfiguration &>(
+                        &capiocl::parser::Parser::parse),
+                    py::arg("config"))
         .def("__str__",
              [](const capiocl::parser::Parser &e) {
                  return "<Parser repr at " + std::to_string(reinterpret_cast<uintptr_t>(&e)) + ">";

capiocl/configuration.h

@@ -29,6 +29,8 @@ struct capiocl::configuration::defaults {
     static ConfigurationEntry DEFAULT_MONITOR_HOMENODE_PORT;
     /// @brief Enable File system monitor by default
     static ConfigurationEntry DEFAULT_MONITOR_FS_ENABLED;
+    /// @brief Directory used for persistent filesystem monitor metadata
+    static ConfigurationEntry DEFAULT_MONITOR_FS_METADATA_DIR;
     /// @brief IP multicast address for receiving and sending changes in the CapioCL configuration
     static ConfigurationEntry DEFAULT_API_MULTICAST_IP;
     /// @brief IP multicast port for receiving and sending changes in the CapioCL configuration
@@ -55,7 +57,10 @@ class capiocl::configuration::CapioClConfiguration {
     void set(const ConfigurationEntry &entry);
 
   public:
-    ~CapioClConfiguration() = default;
+    CapioClConfiguration() = default;
+
+    /// @brief Build a configuration from an external generated unordered map of key values
+    explicit CapioClConfiguration(std::unordered_map<std::string, std::string> config);
 
     /**
      * Load a configuration from a TOML file
@@ -106,4 +111,4 @@ class capiocl::configuration::CapioClConfigurationException final : public std::
     [[nodiscard]] const char *what() const noexcept override { return message.c_str(); }
 };
 
-#endif
+#endif

capiocl/engine.h

@@ -92,6 +92,9 @@ class Engine final {
     ///@brief Configuration imported from CAPIO-CL config TOML file
     configuration::CapioClConfiguration configuration;
 
+    /// @brief Configure the monitor backend from the currently loaded configuration
+    void configureMonitorBackends();
+
     /// @brief Monitor instance to check runtime information of CAPIO-CL files
     monitor::Monitor monitor;
 
@@ -432,9 +435,9 @@ class Engine final {
 
     /**
      * @brief Record one qualified logical close for an ON_CLOSE path.
-     * Filesystem-backed counts greater than one require CAPIO_METADATA_DIR to identify a trusted,
-     * unique-per-run metadata directory with coherent atomic exclusive create/unlink. Multicast
-     * counts are volatile and eventual.
+     * Filesystem-backed counts greater than one require
+     * capiocl.monitor.filesystem.metadata_dir to identify a trusted, unique-per-run directory with
+     * coherent atomic exclusive create/unlink. Multicast counts are volatile and eventual.
      * @return true when the path is effectively committed after this call.
      */
     bool increaseCloseCount(const std::filesystem::path &path) const;

capiocl/monitor.h

@@ -261,6 +261,7 @@ class MulticastMonitor final : public MonitorInterface {
 class FileSystemMonitor final : public MonitorInterface {
 
     typedef enum { COMMIT, HOME_NODE } CAPIO_CL_COMMIT_TOKEN_TYPES;
+    std::filesystem::path metadata_root;
 
     /**
      * @brief Compute the token filename used to represent the commit state of the given file.
@@ -299,6 +300,12 @@ class FileSystemMonitor final : public MonitorInterface {
      */
     FileSystemMonitor();
 
+    /**
+     * @brief Construct a filesystem-based monitor using runtime configuration.
+     * @param config Configuration containing the filesystem metadata directory.
+     */
+    explicit FileSystemMonitor(const configuration::CapioClConfiguration &config);
+
     /**
      * @brief Destructor for FileSystemMonitor.
      */

capiocl/parser.h

@@ -37,23 +37,29 @@ class Parser final {
          * @param resolve_prefix Prefix to prepend to path if found to be relative
          * @param store_only_in_memory Flag to set to returned instance of Engine if required to
          * store all files in memory
+         * @param config An externally provided configuration to be used within the CAPIO-CL engine
+         * instance that will be started up
          * @return Parsed Engine.
          */
         static engine::Engine *parse_v1(const std::filesystem::path &source,
                                         const std::filesystem::path &resolve_prefix,
-                                        bool store_only_in_memory);
+                                        bool store_only_in_memory,
+                                        const configuration::CapioClConfiguration *config);
 
         /**
          * Parser for the V1.1 Specification of the CAPIO-CL language
          * @param source Path of CAPIO-CL configuration file
          * @param resolve_prefix Prefix to prepend to path if found to be relative
          * @param store_only_in_memory Flag to set to returned instance of Engine if required to
          * store all files in memory
+         * @param config An externally provided configuration to be used within the
+         * CAPIO-CL engine instance that will be started up
          * @return Parsed Engine.
          */
         static engine::Engine *parse_v1_1(const std::filesystem::path &source,
                                           const std::filesystem::path &resolve_prefix,
-                                          bool store_only_in_memory);
+                                          bool store_only_in_memory,
+                                          const configuration::CapioClConfiguration *config);
     };
 
     /**
@@ -74,26 +80,16 @@ class Parser final {
                                          const std::filesystem::path &prefix);
 
   public:
+    /** Build an engine using only CAPIO-CL runtime configuration parameters. */
+    static engine::Engine *parse(const configuration::CapioClConfiguration &config);
+
     /**
      * Validate a CAPIO-CL configuration file according to the JSON schema of the language
      * @param doc The loaded CAPIO-CL configuration file
      * @param str_schema Raw JSON schema to use
      */
     static void validate_json(const jsoncons::json &doc, const char *str_schema);
-
-    /**
-     * @brief Perform the parsing of the capio_server configuration file
-     *
-     * @param source Input CAPIO-CL Json configuration File
-     * @param resolve_prefix If paths are found to be relative, they are appended to this path
-     * @param store_only_in_memory Set to true to set all files to be stored in memory
-     * @return Engine instance with the information provided by  the config file
-     * @throw ParserException
-     */
-    static engine::Engine *parse(const std::filesystem::path &source,
-                                 const std::filesystem::path &resolve_prefix = "",
-                                 bool store_only_in_memory                   = false);
 };
 } // namespace capiocl::parser
 
-#endif // CAPIO_CL_PARSER_H
+#endif // CAPIO_CL_PARSER_H

doxygen/configuration.md

@@ -14,19 +14,19 @@ If no configuration file is provided, CAPIO-CL falls back to its built-in defaul
 
 ## TOML Configuration Structure
 
-The configuration file uses a structured namespace under the top-level table `monitor.mcast`.
+All CAPIO-CL options use the top-level `capiocl` table, so unrelated application settings can coexist in the same file.
 
 Available configuration parameters:
 
 | Key                           | Type    | Default         | Description                                                                                                                        |
 |-------------------------------|---------|-----------------|------------------------------------------------------------------------------------------------------------------------------------|
-| `monitor.filesystem.enabled`  | boolean | `false`         | Enable FileSystem commit monitor                                                                                                   |
-| `monitor.mcast.enabled`       | boolean | `false`         | Enable Multicast commit monitor                                                                                                    |
-| `monitor.mcast.commit.ip`     | string  | `224.224.224.1` | Multicast IP address used for commit messages                                                                                      |
-| `monitor.mcast.commit.port`   | integer | `12345`         | UDP port for commit messages                                                                                                       |
-| `monitor.mcast.delay_ms`      | integer | `300`           | Artificial delay (in milliseconds) inserted before sending multicast messages. Useful for debugging or simulating slower networks. |
-| `monitor.mcast.homenode.ip`   | string  | `224.224.224.2` | IP address of the home node for monitoring operations                                                                              |
-| `monitor.mcast.homenode.port` | integer | `12345`         | Port associated with the home node monitoring endpoint                                                                             |
+| `capiocl.monitor.filesystem.enabled`  | boolean | `false`         | Enable FileSystem commit monitor                                                                                                   |
+| `capiocl.monitor.mcast.enabled`       | boolean | `false`         | Enable Multicast commit monitor                                                                                                    |
+| `capiocl.monitor.mcast.commit.ip`     | string  | `224.224.224.1` | Multicast IP address used for commit messages                                                                                      |
+| `capiocl.monitor.mcast.commit.port`   | integer | `12345`         | UDP port for commit messages                                                                                                       |
+| `capiocl.monitor.mcast.delay_ms`      | integer | `300`           | Artificial delay (in milliseconds) inserted before sending multicast messages. Useful for debugging or simulating slower networks. |
+| `capiocl.monitor.mcast.homenode.ip`   | string  | `224.224.224.2` | IP address of the home node for monitoring operations                                                                              |
+| `capiocl.monitor.mcast.homenode.port` | integer | `12345`         | Port associated with the home node monitoring endpoint                                                                             |
 
 ---
 
@@ -36,9 +36,10 @@ Below is a complete example of a `config.toml` file:
 
     # Example CAPIO-CL TOML configuration
 
-    monitor.filesystem.enabled = true    
+    [capiocl.monitor.filesystem]
+    enabled = true
 
-    [monitor.mcast]
+    [capiocl.monitor.mcast]
     enabled = true
 
     # Multicast settings for commit messages
@@ -56,12 +57,12 @@ Below is a complete example of a `config.toml` file:
 
 ## How CAPIO-CL Uses These Settings
 
-### `commit.ip` and `commit.port`
+### `capiocl.monitor.mcast.commit.ip` and `capiocl.monitor.mcast.commit.port`
 
 These fields define where CAPIO-CL sends **commit broadcast messages**.  
 Commit messages are used for consistency coordination across distributed nodes.
 
-### `delay_ms`
+### `capiocl.monitor.mcast.delay_ms`
 
 A small configurable delay may help with:
 
@@ -71,7 +72,7 @@ A small configurable delay may help with:
 
 A value of `0` means no delay.
 
-### `homenode.ip` and `homenode.port`
+### `capiocl.monitor.mcast.homenode.ip` and `capiocl.monitor.mcast.homenode.port`
 
 These define the **central monitoring endpoint** (the “home node”).  
 CAPIO-CL uses this endpoint to coordinate monitoring metadata and cluster-wide communication.