diff --git a/.config/typos.toml b/.config/typos.toml index 3607c85..bc5706f 100644 --- a/.config/typos.toml +++ b/.config/typos.toml @@ -1,5 +1,5 @@ [files] -extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg"] +extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg", "*.js"] [default.extend-words] ratatui = "ratatui" diff --git a/_quarto.yml b/_quarto.yml index 8ad1ec7..6e64700 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -50,6 +50,7 @@ website: contents: - docs/design/interface/inputs.qmd - docs/design/interface/outputs.qmd + - docs/design/interface/request.qmd - docs/design/interface/cli.qmd - docs/design/interface/web.qmd - docs/design/interface/rust.qmd diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index fb47b48..fc4e86b 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -170,11 +170,29 @@ flowchart LR ## {{< var planned >}} `seedcase-propagate check` The `check` command confirms that the `request.yaml` file is correctly -structured and that it matches the content of the source metadata file. While -internally `build`, `subset`, and `create-request` run this check when -processing or creating the `request.yaml` file, we also expose it so the user -can manually check the request file (e.g. if included within an automated -pipeline). It has the following signature with two positional arguments: +structured and that it matches the content of the source metadata file. More +precisely, the following checks are made: + +- Required sections and keys are present. +- The referenced data package has the same id and name as the source metadata + file. +- The referenced resources exist in the data package. +- The requested columns exist in the selected resources. +- Row filters reference existing columns. +- Row filters use supported SQL-like syntax. +- Only allowed SQL operations and shorthands are used. + +While most of these check are likely easier to test directly in the YAML file, +the subset queries filters can also be converted to SQL and checked with +existing SQL parsers or checkers where possible. This avoids implementing a +custom expression parser for the request format. + +Internally `build`, `subset`, and `create-request` run these checks when +processing or creating the `request.yaml` file. We also expose it as a command +so the user can check the request file independently (e.g. if included within an +automated pipeline). + +`check` has the following signature with two positional arguments: ```bash {filename="Terminal"} seedcase-propagate check [REQUEST] [SOURCE] diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd new file mode 100644 index 0000000..87c76c0 --- /dev/null +++ b/docs/design/interface/requests.qmd @@ -0,0 +1,312 @@ +--- +title: "Request file" +--- + +The request file, `request.yaml`, contains the instructions for requesting a +subset of data from a data package. The file is created by a Requester through +the web application, terminal interface, or even manually (though not +recommended). The file is used by an Owner to check and generate the requested +data subset. You can +[read more about the Requester and Owner user types in the Design overview page](/docs/design/interface/inputs.qmd) + +The request file is intended to be both human-readable and machine-readable. It +should be simple enough for a Requester to understand and, when needed, edit +manually. At the same time, it should be structured enough that Propagate can +check it against the data package metadata and convert it into an SQL query for +generating the requested data subset. + +## Format + +The request file uses YAML. YAML was chosen because it is more readable for +manual editing than JSON and less verbose than TOML for lists of columns and row +filters. + +## Structure + +The request file contains the following top-level sections: + +- `request`: Information about this specific request. +- `requester`: Information about the person requesting the data. +- `project`: Information about the research project that requires the data. +- `data-package`: Information about the data package the request applies to. +- `subsets`: The requested data subsets, with one item per resource. + +Each of these is elaborated on below: + +### Request + +The `request` includes information about the request itself, such as when and +why it was created. This helps an Owner organize and track requests, which is +especially beneficial for data packages that receive large volumes of requests. + +```{.yaml} +request: + date-modified: DATE + date-created: DATE + motivation: TEXT + iteration: NUMBER +``` + +- `date-modified`: When the request was last modified. This can act as an ID + which facilitates referring to a specific iteration of a request, e.g. in + dialog between the Owner and Requester. Precision to the second should be + sufficient. + - Note that this field is automatically populated when the request is saved + via a user interface (be it the web app or the TUI). If the request is + edited manually, it must be updated manually. + - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00` + +- `date-created`: When the first iteration of the request was created. + +- `motivation`: Explanation for why the requester's project needs access to this + data subset. This is used by the Owner to check whether the requested data is + relevant to the project. The motivation is particularly needed for human + health data since there are legal requirements (e.g. GDPR) to declare for what + purposes the data is used. +- `iteration`: The iteration of requests from this project for this data + package. + + +### Requester + +The `requester` section describes who is requesting the data. This person +doesn't strictly have to be the one working with the data, but they should serve +as the main point of contact for the Owner. + + +```{.yaml} +requester: + name: TEXT + email: TEXT +``` + +- `name`: The name of the requester. +- `email`: The email of the requester. + +### Project + +The `project` section contains information about the research project that +requires the data. A project is defined as a body of work that result in a +research output, e.g. a paper, report, website, dashboard, or analysis. This +could be a part of a PhD student's thesis or a collaboration research project +for a group of researchers. This field is necessary to ensure that the data +requested is aligned with what it is intended to be used for, which can a legal + +requirement (e.g. in GDPR). + +- `name`: A short machine-readable project name, using lowercase letters and + hyphens where possible, e.g. `my-research-project`. This is mainly used for + identification purposes when managing several requests. +- `title`: A longer human-readable project title, e.g. `My research project`. + This could be used by the Owner to publicly display the data package requests. +- `description`: A summary of the project, including details like + background, rationale, basic methods, and potential research questions. This + information, combined with the `motivation`, is mainly to help the Owner + ensure that the data requested matches what it will be used for. + +### Data package + +The `data-package` section identifies the data package the request applies to. +This information will be automatically populated from the data package metadata +when creating a request via the web application or the TUI. + +```{.yaml} +data-package: + id: TEXT + name: TEXT + version: X.Y.Z +``` + +- `id`: A unique identifier of the data package. +- `name`: A human readable name to refer to. It is lowercased and doesn't + include any spaces. E.g., `my-data-package`. +- `version`: Helps the Owner detect which version of the data package the + request is for. + +### Subsets + +The `subsets` section describes which data subsets the Requester would like to create. + +```{.yaml} +subsets: + - resource: RESOURCE-NAME + columns: + - COLUMN-NAME + - COLUMN-NAME + - COLUMN-NAME + rows: + - "ROW-CONDITION" # As SQL + # Alternatively, this more structured format, see comparison below + - column: COLUMN-NAME + operator: "OPERATOR" # E.g. ">" + value: VALUE-TO-COMPARE-AGAINST + - resource: RESOURCE-NAME + columns: + - COLUMN-NAME +``` + + + +- `resource`: Specifies the data resource to create the subset from. Multiple + subsets can be specified in the same request by repeating the `resource` key. + This can be done either with the same resource multiple times or with + different resources for each. +- `columns`: The columns to include. Each list item must correspond to an exact + column name. SQL-like shorthands are not supported in order to keep the + request file simple and to avoid parsing unsafe queries of the data (e.g. via + SQL injection). This can not be left empty / unspecified; at least one column + must be included in the `columns` field. +- `rows`: Describes which rows to include in the subset. Can reference columns + that are not a part of the subset. If the `rows` key is empty or not included, + all rows will be selected. + +There are currently two candidates for what this might look like, see below. + +#### Row condition alternatives + +##### Alternative 1 + +- A list of SQL filter conditions that describe which rows to include in the + subset. +- Multiple row filters for the same resource are combined with `AND` logic. For + example, writing + + ```{.yaml} + rows: + - "age > 50" + - "sex = 'Man'" + ``` + + is equivalent to writing + + ```{.yaml} + rows: + - "age > 50 AND sex = 'Man'" + ``` + + ::: callout-note + Since YAML has special syntax for some characters, SQL-like row filters should + be quoted when written manually. This avoids ambiguity around characters such as + `>`, `:`, `#`, `*`, `&`, and `!`. + + ```{.yaml} + rows: + - "col1 > 50" + ``` + + Instead of: + + ```{.yaml} + rows: + - col1 > 50 + ``` + + The unquoted form will often parse correctly, but the quoted form is safer and + should be used in documentation and generated files. + ::: + +- The initial supported row filter syntax includes the following operators: + - Equality: `=` + - E.g. "age = 18" + - Comparisons: `>`, `>=`, `<`, `<=` + - E.g. `"age >= 18"` + - Membership: `IN` + - E.g. `"country IN ('Denmark', 'Sweden')"` + - Ranges: `BETWEEN` + - E.g. `"visit_date BETWEEN '2020-01-01' AND '2020-12-31'"` + - Missing values: `IS NULL`, `IS NOT NULL` + - E.g. `"end_date IS NULL"` + - Negation: `NOT` + - E.g. `NOT =`, `NOT IN`, `NOT BETWEEN` +- This alternative is more closely mapped to how the SQL syntax will look, and + the options from the interface would need to be combined into the SQL row + filter syntax. +- An advantage of this syntax is that it is more flexible. This might mainly + matter when editing manually as we allow more expressiveness here. +- A drawback could be that requesters write things we don't want to allow and + that we would need to filter out at the SQL level or by parsing the string + which is likely more error prone than a more structured format. + +#### Alternative 2 + +- A more structured key-value mapping for the different components of the filter + condition. + +```{.yaml} +rows: + - column: col1 + operator: ">" + value: 50 + - column: col2 + operator: "in" + value: + - dog + - cat + - column: col3 + operator: between + value: + - 2012-01-01 + - 2012-12-31 +``` + +- This alternative is more closely mapped to the interface, e.g. the dropdown + for the operator corresponds to the "operator" key here and the condition + groups in the interface are also more similar to how they are grouped here + with "any". +- An advantage of this syntax is that it is more structured, so it would be + easier to check that e.g. only allowed operators are used if someone manually + edits the file. +- If we want to allow more flexibility when the file is edited manually, it + could be harder with this more strict spec, but it doesn't sound like that is + our intended use case currently. + +### A complete example + +A complete request file might look like this. + +```{.yaml} +request: + date-modified: 2026-07-06T01:45:34-12:00 + motivation: | + We would like access to metabolic and block variables to evaluate + our hypothesis regarding ... + date-created: 2026-07-04T01:44:34-12:00 + iteration: 2 + +project: + name: + title: Metabolic costs + description: | + Our project investigates the gas exchange during metabolism + with the aim to determine ... + +data-package: + id: 8dbef2a2-0354-4996-bc9e-71a6c79aec7d + name: example-seed-beetle + version: 0.5.1 + +subsets: + # TODO Showing both row syntaxes for now + - resource: metabolic-rate + columns: + - o2_consumed + - co2_produced + rows: + - "o2_consumed > 50" + - resource: metabolic-rate + columns: + - strain + - activity + rows: + - column: block + operator: "=" + value: block1 +``` + +### Non-goals + +Propagate is not intended to be a general-purpose query engine. The initial +request format should not support joins, grouping, arbitrary functions, random +sampling, or row selection by position. We also don't support shorthands such as `*` since that would make it less +explicit which columns are used from just looking at the request file without +also seeing the data package metadata.