docs: 📝 add design docs for the request file

.config/typos.toml

@@ -1,5 +1,5 @@
 [files]
-extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg"]
+extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg", "*.js"]
 
 [default.extend-words]
 ratatui = "ratatui"

_quarto.yml

@@ -50,6 +50,7 @@ website:
           contents:
             - docs/design/interface/inputs.qmd
             - docs/design/interface/outputs.qmd
+            - docs/design/interface/request.qmd
             - docs/design/interface/cli.qmd
             - docs/design/interface/web.qmd
             - docs/design/interface/rust.qmd

docs/design/interface/cli.qmd

@@ -170,11 +170,29 @@ flowchart LR
 ## {{< var planned >}} `seedcase-propagate check`
 
 The `check` command confirms that the `request.yaml` file is correctly
-structured and that it matches the content of the source metadata file. While
-internally `build`, `subset`, and `create-request` run this check when
-processing or creating the `request.yaml` file, we also expose it so the user
-can manually check the request file (e.g. if included within an automated
-pipeline). It has the following signature with two positional arguments:
+structured and that it matches the content of the source metadata file. More
+precisely, the following checks are made:
+
+- Required sections and keys are present.
+- The referenced data package has the same id and name as the source metadata
+  file.
+- The referenced resources exist in the data package.
+- The requested columns exist in the selected resources.
+- Row filters reference existing columns.
+- Row filters use supported SQL-like syntax.
+- Only allowed SQL operations and shorthands are used.
+
+While most of these check are likely easier to test directly in the YAML file,
+the subset queries filters can also be converted to SQL and checked with
+existing SQL parsers or checkers where possible. This avoids implementing a
+custom expression parser for the request format.
+
+Internally `build`, `subset`, and `create-request` run these checks when
+processing or creating the `request.yaml` file. We also expose it as a command
+so the user can check the request file independently (e.g. if included within an
+automated pipeline).
+
+`check` has the following signature with two positional arguments:
 
 ```bash {filename="Terminal"}
 seedcase-propagate check [REQUEST] [SOURCE]

docs/design/interface/requests.qmd

@@ -0,0 +1,312 @@
+---
+title: "Request file"
+---
+
+The request file, `request.yaml`, contains the instructions for requesting a
+subset of data from a data package. The file is created by a Requester through
+the web application, terminal interface, or even manually (though not
+recommended). The file is used by an Owner to check and generate the requested
+data subset. You can
+[read more about the Requester and Owner user types in the Design overview page](/docs/design/interface/inputs.qmd)
+
+The request file is intended to be both human-readable and machine-readable. It
+should be simple enough for a Requester to understand and, when needed, edit
+manually. At the same time, it should be structured enough that Propagate can
+check it against the data package metadata and convert it into an SQL query for
+generating the requested data subset.
+
+## Format
+
+The request file uses YAML. YAML was chosen because it is more readable for
+manual editing than JSON and less verbose than TOML for lists of columns and row
+filters.
+
+## Structure
+
+The request file contains the following top-level sections:
+
+- `request`: Information about this specific request.
+- `requester`: Information about the person requesting the data.
+- `project`: Information about the research project that requires the data.
+- `data-package`: Information about the data package the request applies to.
+- `subsets`: The requested data subsets, with one item per resource.
+
+Each of these is elaborated on below:
+
+### Request
+
+The `request` includes information about the request itself, such as when and
+why it was created. This helps an Owner organize and track requests, which is
+especially beneficial for data packages that receive large volumes of requests.
+
+```{.yaml}
+request:
+  date-modified: DATE
+  date-created: DATE
+  motivation: TEXT
+  iteration: NUMBER
+```
+
+- `date-modified`: When the request was last modified. This can act as an ID
+  which facilitates referring to a specific iteration of a request, e.g. in
+  dialog between the Owner and Requester. Precision to the second should be
+  sufficient.
+  - Note that this field is automatically populated when the request is saved
+    via a user interface (be it the web app or the TUI). If the request is
+    edited manually, it must be updated manually.
+  - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00`
+<!-- TODO this is this the most suitable date format? It seems like it makes sense to include timezone info so it can be easily interpreted by both sides if they are in different timezones -->
+- `date-created`: When the first iteration of the request was created.
+<!-- TODO The creation date would only be correctly populated if the requester uploads their old request in the interface when they need to make a modification. If they rewrite the request from scratch, they could (mistakenly) put an incorrect date here, but that's fine I think. -->
+- `motivation`: Explanation for why the requester's project needs access to this
+  data subset. This is used by the Owner to check whether the requested data is
+  relevant to the project. The motivation is particularly needed for human
+  health data since there are legal requirements (e.g. GDPR) to declare for what
+  purposes the data is used.
+- `iteration`: The iteration of requests from this project for this data
+  package.
+<!-- TODO This is a "maybe" since it seems a bit less reliable. It could be automatically incremented if a file is loaded into the web-interface, but there is not way to know if that file was a saved version that was never sent (ie the iteration should not be updated) or if it was sent and is now being modified for sending again (ie the iteration should be updated). It could be manually filled in of course, but we should think about who this is mainly useful for; if it is the owner then they could run something on their end to find out how many requests they have; if it is someone viewing the data package then it's more useful to try to have it in the file itself. -->
+
+### Requester
+
+The `requester` section describes who is requesting the data. This person
+doesn't strictly have to be the one working with the data, but they should serve
+as the main point of contact for the Owner.
+
+
+```{.yaml}
+requester:
+  name: TEXT
+  email: TEXT
+```
+
+- `name`: The name of the requester.
+- `email`: The email of the requester.
+
+### Project
+
+The `project` section contains information about the research project that
+requires the data. A project is defined as a body of work that result in a
+research output, e.g. a paper, report, website, dashboard, or analysis. This
+could be a part of a PhD student's thesis or a collaboration research project
+for a group of researchers. This field is necessary to ensure that the data
+requested is aligned with what it is intended to be used for, which can a legal
+
+requirement (e.g. in GDPR).
+
+- `name`: A short machine-readable project name, using lowercase letters and
+  hyphens where possible, e.g. `my-research-project`. This is mainly used for
+  identification purposes when managing several requests.
+- `title`: A longer human-readable project title, e.g. `My research project`.
+  This could be used by the Owner to publicly display the data package requests.
+- `description`: A summary of the project, including details like
+  background, rationale, basic methods, and potential research questions. This
+  information, combined with the `motivation`, is mainly to help the Owner
+  ensure that the data requested matches what it will be used for.
+
+### Data package
+
+The `data-package` section identifies the data package the request applies to.
+This information will be automatically populated from the data package metadata
+when creating a request via the web application or the TUI.
+
+```{.yaml}
+data-package:
+  id: TEXT
+  name: TEXT
+  version: X.Y.Z
+```
+
+- `id`: A unique identifier of the data package.
+- `name`: A human readable name to refer to. It is lowercased and doesn't
+  include any spaces. E.g., `my-data-package`.
+- `version`: Helps the Owner detect which version of the data package the
+  request is for.
+
+### Subsets
+
+The `subsets` section describes which data subsets the Requester would like to create.
+
+```{.yaml}
+subsets:
+  - resource: RESOURCE-NAME
+    columns:
+      - COLUMN-NAME
+      - COLUMN-NAME
+      - COLUMN-NAME
+    rows:
+      - "ROW-CONDITION" # As SQL
+      # Alternatively, this more structured format, see comparison below
+      - column: COLUMN-NAME
+        operator: "OPERATOR" # E.g. ">"
+        value: VALUE-TO-COMPARE-AGAINST
+  - resource: RESOURCE-NAME
+    columns:
+      - COLUMN-NAME
+```
+
+<!-- TODO When editing the file directly, do we require the user to enter row conditions (and maybe all other values too) as strings? It seems like we have to in order not to have yaml parsing potentially break if they do something illegal. -->
+
+- `resource`: Specifies the data resource to create the subset from. Multiple
+  subsets can be specified in the same request by repeating the `resource` key.
+  This can be done either with the same resource multiple times or with
+  different resources for each.
+- `columns`: The columns to include. Each list item must correspond to an exact
+  column name. SQL-like shorthands are not supported in order to keep the
+  request file simple and to avoid parsing unsafe queries of the data (e.g. via
+  SQL injection). This can not be left empty / unspecified; at least one column
+  must be included in the `columns` field.
+- `rows`: Describes which rows to include in the subset. Can reference columns
+  that are not a part of the subset. If the `rows` key is empty or not included,
+  all rows will be selected.
+
+There are currently two candidates for what this might look like, see below.
+
+#### Row condition alternatives
+
+##### Alternative 1
+
+- A list of SQL filter conditions that describe which rows to include in the
+  subset.
+- Multiple row filters for the same resource are combined with `AND` logic. For
+  example, writing
+
+  ```{.yaml}
+  rows:
+    - "age > 50"
+    - "sex = 'Man'"
+  ```
+
+  is equivalent to writing
+
+  ```{.yaml}
+  rows:
+    - "age > 50 AND sex = 'Man'"
+  ```
+
+  ::: callout-note
+  Since YAML has special syntax for some characters, SQL-like row filters should
+  be quoted when written manually. This avoids ambiguity around characters such as
+  `>`, `:`, `#`, `*`, `&`, and `!`.
+
+  ```{.yaml}
+  rows:
+    - "col1 > 50"
+  ```
+
+  Instead of:
+
+  ```{.yaml}
+  rows:
+    - col1 > 50
+  ```
+
+  The unquoted form will often parse correctly, but the quoted form is safer and
+  should be used in documentation and generated files.
+  :::
+
+- The initial supported row filter syntax includes the following operators:
+  - Equality: `=`
+    - E.g. "age = 18"
+  - Comparisons: `>`, `>=`, `<`, `<=`
+    - E.g. `"age >= 18"`
+  - Membership: `IN`
+    - E.g. `"country IN ('Denmark', 'Sweden')"`
+  - Ranges: `BETWEEN`
+    - E.g. `"visit_date BETWEEN '2020-01-01' AND '2020-12-31'"`
+  - Missing values: `IS NULL`, `IS NOT NULL`
+    - E.g. `"end_date IS NULL"`
+  - Negation: `NOT`
+    - E.g. `NOT =`, `NOT IN`, `NOT BETWEEN`
+- This alternative is more closely mapped to how the SQL syntax will look, and
+  the options from the interface would need to be combined into the SQL row
+  filter syntax.
+- An advantage of this syntax is that it is more flexible. This might mainly
+  matter when editing manually as we allow more expressiveness here.
+- A drawback could be that requesters write things we don't want to allow and
+  that we would need to filter out at the SQL level or by parsing the string
+  which is likely more error prone than a more structured format.
+
+#### Alternative 2
+
+- A more structured key-value mapping for the different components of the filter
+  condition.
+
+```{.yaml}
+rows:
+  - column: col1
+    operator: ">"
+    value: 50
+  - column: col2
+    operator: "in"
+    value:
+      - dog
+      - cat
+  - column: col3
+    operator: between
+    value:
+      - 2012-01-01
+      - 2012-12-31
+```
+
+- This alternative is more closely mapped to the interface, e.g. the dropdown
+  for the operator corresponds to the "operator" key here and the condition
+  groups in the interface are also more similar to how they are grouped here
+  with "any".
+- An advantage of this syntax is that it is more structured, so it would be
+  easier to check that e.g. only allowed operators are used if someone manually
+  edits the file.
+- If we want to allow more flexibility when the file is edited manually, it
+  could be harder with this more strict spec, but it doesn't sound like that is
+  our intended use case currently.
+
+### A complete example
+
+A complete request file might look like this.
+
+```{.yaml}
+request:
+  date-modified: 2026-07-06T01:45:34-12:00
+  motivation: |
+    We would like access to metabolic and block variables to evaluate
+    our hypothesis regarding ...
+  date-created: 2026-07-04T01:44:34-12:00
+  iteration: 2
+
+project:
+  name:
+  title: Metabolic costs
+  description: |
+    Our project investigates the gas exchange during metabolism
+    with the aim to determine ...
+
+data-package:
+  id: 8dbef2a2-0354-4996-bc9e-71a6c79aec7d
+  name: example-seed-beetle
+  version: 0.5.1
+
+subsets:
+  # TODO Showing both row syntaxes for now
+  - resource: metabolic-rate
+    columns:
+      - o2_consumed
+      - co2_produced
+    rows:
+      - "o2_consumed > 50"
+  - resource: metabolic-rate
+    columns:
+      - strain
+      - activity
+    rows:
+      - column: block
+        operator: "="
+        value: block1
+```
+
+### Non-goals
+
+Propagate is not intended to be a general-purpose query engine. The initial
+request format should not support joins, grouping, arbitrary functions, random
+sampling, or row selection by position. We also don't support shorthands such as `*` since that would make it less
+explicit which columns are used from just looking at the request file without
+also seeing the data package metadata.