From 8667f4d4659eaf71f9e74171211e602b9991949b Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 10:23:36 +0200 Subject: [PATCH 01/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20Add=20request=20fi?= =?UTF-8?q?le=20design=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- _quarto.yml | 1 + docs/design/interface/request-file.qmd | 256 +++++++++++++++++++++++++ 2 files changed, 257 insertions(+) create mode 100644 docs/design/interface/request-file.qmd diff --git a/_quarto.yml b/_quarto.yml index 76644a6..6a28f72 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -47,6 +47,7 @@ website: contents: - docs/design/interface/inputs.qmd - docs/design/interface/outputs.qmd + - docs/design/interface/request-file.qmd - docs/design/interface/cli.qmd - docs/design/interface/web.qmd - docs/design/interface/rust.qmd diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd new file mode 100644 index 0000000..0dae182 --- /dev/null +++ b/docs/design/interface/request-file.qmd @@ -0,0 +1,256 @@ +--- +title: "Request File" +--- + +The request file, `request.yaml`, contains the instructions for requesting a +subset of data from a data package. The file is created by a Requester through +the web application and is used by an Owner to check and generate the requested +data subset. + +The request file is intended to be both human-readable and machine-readable. It +should be simple enough for a Requester to understand and, when needed, edit +manually. At the same time, it should be structured enough that Propagate can +check it against the data package metadata and convert it into an SQL query for +generating the requested data subset. + +## Format + +The request file uses YAML. YAML was chosen because it is more readable for +manual editing than JSON and less verbose than TOML for lists of columns and row +filters. + +::: callout-note +Since YAML has special syntax for some characters, SQL-like row filters should +be quoted when written manually. This avoids ambiguity around characters such as +`>`, `:`, `#`, `*`, `&`, and `!`. + +```{.yaml} +rows: + - "col1 > 50" +``` + +Instead of: + +```{.yaml} +rows: + - col1 > 50 +``` + +The unquoted form will often parse correctly, but the quoted form is safer and +should be used in documentation and generated files. +::: + +## Structure + +The request file contains four top-level sections: + +- `request`: Information about this specific request. +- `requester`: Information about the person and project requesting the data. +- `data-package`: Information about the data package the request applies to. +- `subsets`: The requested subsets, with one item per resource. + +Each of these is elaborated on below: + +### A complete example + +A complete request file might look like this. + +```{.yaml} +request: + date-modified: # When the request was last modified, this can act as an ID which facilitates referring to a specific iteration of the request, e.g. in dialog between the owner and requester. Precision to the minute or second should be sufficient + motivation: # Why does your project need access to this data subset + # The following two fields are "maybes" since the would have to be updated manually which might make it hard to guarantee that they are filled in (the forms pages could of course require them, but there is no guarantee that they are correctly filled in). + date-created: # When the project first requested data from this data package + iteration: 1 # This iteration of requests from this project + +project: + name: NAME # The name is a sort of project ID, e.g. my-project-name + title: TITLE # More readable than the name + description: # General brief project description with a connection to the `motivation` above. + +data-package: + id: ID + name: NAME + version: X.Y.Z + +subsets: + - resource: resource1 + columns: + - col1 + rows: + - "col1 > 50" + - resource: resource2 + columns: + - col1 + - col2 + rows: + - "col2 = 'luke'" + - "col1 BETWEEN 10 AND 20" +``` + +More information about each of these sections can be found under the respective +heading below. + +### Request + +The `request` section describes this specific request. + +```yaml +request: + date-modified: DATE + date-created: DATE + motivation: TEXT + iteration: NUMBER +``` + +- `date-modified`: When the request was last modified. This can act as an ID + which facilitates referring to a specific iteration of a request, e.g. in + dialog between the owner and requester. Precision to the second should be + sufficient. + - Note that this field should not be updated via the user interface, it should + be automatically populated when the request is saved. + - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00` + +- `date-created`: When the first iteration of the request was created. + +- `motivation`: Text explaining why the requester's project needs access to this data subset. +- `iteration`: The iteration of requests from this project for this data package. + + +### Requester + +The `requester` section describes who is requesting the data and the project the +data will be used for. + +```yaml +requester: + name: TEXT + email: TEXT + project: + name: TEXT + title: TEXT +``` + +- `name`: The name of the requester. +- `email`: The email of the requester. +- `project.name`: A short machine-readable project name, using lowercase letters + and hyphens where possible, e.g. `my-research-project`. +- `project.title`: A longer human-readable project title, e.g. + `My research project`. + +### Data Package + +The `data-package` section identifies the data package the request applies to. + +```yaml +data-package: + id: TEXT + name: TEXT + version: X.Y.Z +``` + +This information will be automatically populated from the data package metadata +when creating a request via the web application. Including this information +helps Owners check that the request was created for the intended data package +and version. + +### Subsets + +The `subsets` section describes which data subsets the requester would like to create. + +```{.yaml} +subsets: + - resource: RESOURCE-NAME + columns: + - COLUMN-NAME + - COLUMN-NAME + - COLUMN-NAME + rows: + - "ROW-CONDITION" # As valid SQL + # Alternatively, this more structured format, see comparison below + - column: COLUMN-NAME + operator: "OPERATOR" # E.g. ">" + value: VALUE-TO-COMPARE-AGAINST + - resource: RESOURCE-NAME + columns: + - COLUMN-NAME +``` + + + +- `resource`: Specifies the data resource to create the subset from. Multiple + subsets can be specified in the same request by repeating the `resource` key, + either with the same resource multiple times or different ones. +- `columns`: A list of columns to include. Features such as selecting a range of + columns are handled via the interface and expanded to explicit column lists in + the request file. +- `rows`: Describes which rows to include in the subset. + - There are currently two candidates for what this might look like, see below. + +#### Row condition alternatives + +Propagate is not intended to be a general-purpose query engine. The initial +request format should not support joins, grouping, arbitrary functions, random +sampling, or row selection by position. + +##### Alternative 1 + +- A list of SQL filter conditions that describe which rows to include in the + subset. +- Multiple row filters for the same resource are combined with `AND` logic. For + example, writing + + ```{.yaml} + rows: + - "age > 50" + - "sex = 'Man'" + ``` + + is equivalent to writing + + ```{.yaml} + rows: + - "age > 50 AND sex = 'Man'" + ``` + +- The initial supported row filter syntax includes the following operators: + - Equality and inequality: `=`, `!=` + - Comparisons: `>`, `>=`, `<`, `<=` + - E.g. `"age >= 18"` + - Membership: `IN` + - E.g. `"country IN ('Denmark', 'Sweden')"` + - Ranges: `BETWEEN` + - E.g. `"visit_date BETWEEN '2020-01-01' AND '2020-12-31'"` + - Missing values: `IS NULL`, `IS NOT NULL` + - E.g. `"end_date IS NULL"` + +- An advantage of this syntax is that it is more flexible. This might mainly + matter when editing manually and we allow more expressiveness here than, which + doesn't seem to be a common use case based on +- A drawback could be that requesters write things we don't want to allow and + that we would need to filter out at the SQL level or by parsing the string + which is likely more error prone than a more structured format. + +#### Alternative 2 + + + + + + + + +## Checking + +Propagate should check the request file before generating a data subset. The +checks should include that: + +- Required sections and keys are present. +- The referenced data package and resources exist. +- The requested columns exist in the selected resources. +- Row filters reference existing columns. +- Row filters use supported SQL-like syntax. + +The row filters can be converted to SQL and checked with existing SQL parsing or +validation tools where possible. This avoids implementing a custom expression +parser for the request format. From 20e300711eca7336958aaa709c4fc1806c91c4f7 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 11:08:04 +0200 Subject: [PATCH 02/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20move=20and=20elabo?= =?UTF-8?q?rate=20on=20the=20complete=20example?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/request-file.qmd | 89 ++++++++++++++------------ 1 file changed, 47 insertions(+), 42 deletions(-) diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index 0dae182..71418d9 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -51,46 +51,6 @@ The request file contains four top-level sections: Each of these is elaborated on below: -### A complete example - -A complete request file might look like this. - -```{.yaml} -request: - date-modified: # When the request was last modified, this can act as an ID which facilitates referring to a specific iteration of the request, e.g. in dialog between the owner and requester. Precision to the minute or second should be sufficient - motivation: # Why does your project need access to this data subset - # The following two fields are "maybes" since the would have to be updated manually which might make it hard to guarantee that they are filled in (the forms pages could of course require them, but there is no guarantee that they are correctly filled in). - date-created: # When the project first requested data from this data package - iteration: 1 # This iteration of requests from this project - -project: - name: NAME # The name is a sort of project ID, e.g. my-project-name - title: TITLE # More readable than the name - description: # General brief project description with a connection to the `motivation` above. - -data-package: - id: ID - name: NAME - version: X.Y.Z - -subsets: - - resource: resource1 - columns: - - col1 - rows: - - "col1 > 50" - - resource: resource2 - columns: - - col1 - - col2 - rows: - - "col2 = 'luke'" - - "col1 BETWEEN 10 AND 20" -``` - -More information about each of these sections can be found under the respective -heading below. - ### Request The `request` section describes this specific request. @@ -184,7 +144,8 @@ subsets: - `columns`: A list of columns to include. Features such as selecting a range of columns are handled via the interface and expanded to explicit column lists in the request file. -- `rows`: Describes which rows to include in the subset. +- `rows`: Describes which rows to include in the subset. Can reference columns + that are not part of the subset. - There are currently two candidates for what this might look like, see below. #### Row condition alternatives @@ -214,7 +175,8 @@ sampling, or row selection by position. ``` - The initial supported row filter syntax includes the following operators: - - Equality and inequality: `=`, `!=` + - Equality and inequality: `=`, `!=` - Comparisons: `>`, `>=`, `<`, `<=` - E.g. `"age >= 18"` - Membership: `IN` @@ -254,3 +216,46 @@ checks should include that: The row filters can be converted to SQL and checked with existing SQL parsing or validation tools where possible. This avoids implementing a custom expression parser for the request format. + +### A complete example + +A complete request file might look like this. + +```{.yaml} +request: + date-modified: 2026-07-06T01:45:34-12:00 + motivation: | + We would like access to metabolic and block variables to evaluate + our hypothesis regarding ... + date-created: 2026-07-04T01:44:34-12:00 + iteration: 2 + +project: + name: + title: Metabolic costs + description: | + Our project investigates the gas exchange during metabolism + with the aim to determine ... + +data-package: + id: 8dbef2a2-0354-4996-bc9e-71a6c79aec7d + name: example-seed-beetle + version: 0.5.1 + +subsets: + # Showing both row syntaxes for now + - resource: metabolic-rate + columns: + - o2_consumed + - co2_produced + rows: + - "o2_consumed > 50" + - resource: metabolic-rate + columns: + - strain + - activity + rows: + - column: block + operator: "=" + value: block1 +``` From 458981dbb0a296271e1311d355d8b8341c39a551 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 11:43:01 +0200 Subject: [PATCH 03/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20add=20alternative?= =?UTF-8?q?=20filter=20syntax?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/request-file.qmd | 39 ++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index 71418d9..ee364f8 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -148,6 +148,10 @@ subsets: that are not part of the subset. - There are currently two candidates for what this might look like, see below. + + + + #### Row condition alternatives Propagate is not intended to be a general-purpose query engine. The initial @@ -186,6 +190,9 @@ sampling, or row selection by position. - Missing values: `IS NULL`, `IS NOT NULL` - E.g. `"end_date IS NULL"` +- This alternative is more closely mapped to how the SQL syntax will look, and + the options from the interface would need to be combined into the sql row + filter syntax. - An advantage of this syntax is that it is more flexible. This might mainly matter when editing manually and we allow more expressiveness here than, which doesn't seem to be a common use case based on @@ -195,12 +202,38 @@ sampling, or row selection by position. #### Alternative 2 +- A more structure key value mapping for the different components of the filter + condition. +```{.yaml} +rows: + - column: col1 + operator: ">" + value: 50 + - column: col2 + operator: "in" + value: + - dog + - cat + - column: col3 + operator: between + value: + - 2012-01-01 + - 2012-12-31 +``` + +- This alternative is more closely mapped to the interface, e.g. the dropdown + for the operator coresponds to the "operator" key here and the condition + groups in the interface are also more simlar to how they are grouped here with + "any". +- An advantage of this syntax is that it is more structured, so it would be + easier to check that e.g. only allowed operators are used if someone manually + edits the file. +- If we want to allow more flexibility when the file is edited manually, it + could be harder with this more strict spec, but it doesn't sound like that is + our intended use case currently. - - - ## Checking From 10df2b08727a0a26e7385a4da571a5560288b06d Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 11:51:00 +0200 Subject: [PATCH 04/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20add=20non-goals=20?= =?UTF-8?q?and=20move=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/request-file.qmd | 61 ++++++++++++++------------ 1 file changed, 32 insertions(+), 29 deletions(-) diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index ee364f8..fd3a686 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -19,27 +19,6 @@ The request file uses YAML. YAML was chosen because it is more readable for manual editing than JSON and less verbose than TOML for lists of columns and row filters. -::: callout-note -Since YAML has special syntax for some characters, SQL-like row filters should -be quoted when written manually. This avoids ambiguity around characters such as -`>`, `:`, `#`, `*`, `&`, and `!`. - -```{.yaml} -rows: - - "col1 > 50" -``` - -Instead of: - -```{.yaml} -rows: - - col1 > 50 -``` - -The unquoted form will often parse correctly, but the quoted form is safer and -should be used in documentation and generated files. -::: - ## Structure The request file contains four top-level sections: @@ -154,10 +133,6 @@ subsets: #### Row condition alternatives -Propagate is not intended to be a general-purpose query engine. The initial -request format should not support joins, grouping, arbitrary functions, random -sampling, or row selection by position. - ##### Alternative 1 - A list of SQL filter conditions that describe which rows to include in the @@ -178,6 +153,27 @@ sampling, or row selection by position. - "age > 50 AND sex = 'Man'" ``` + ::: callout-note + Since YAML has special syntax for some characters, SQL-like row filters should + be quoted when written manually. This avoids ambiguity around characters such as + `>`, `:`, `#`, `*`, `&`, and `!`. + + ```{.yaml} + rows: + - "col1 > 50" + ``` + + Instead of: + + ```{.yaml} + rows: + - col1 > 50 + ``` + + The unquoted form will often parse correctly, but the quoted form is safer and + should be used in documentation and generated files. + ::: + - The initial supported row filter syntax includes the following operators: - Equality and inequality: `=`, `!=` @@ -189,7 +185,6 @@ sampling, or row selection by position. - E.g. `"visit_date BETWEEN '2020-01-01' AND '2020-12-31'"` - Missing values: `IS NULL`, `IS NOT NULL` - E.g. `"end_date IS NULL"` - - This alternative is more closely mapped to how the SQL syntax will look, and the options from the interface would need to be combined into the sql row filter syntax. @@ -233,8 +228,6 @@ rows: could be harder with this more strict spec, but it doesn't sound like that is our intended use case currently. - - ## Checking Propagate should check the request file before generating a data subset. The @@ -276,7 +269,7 @@ data-package: version: 0.5.1 subsets: - # Showing both row syntaxes for now + # TODO Showing both row syntaxes for now - resource: metabolic-rate columns: - o2_consumed @@ -292,3 +285,13 @@ subsets: operator: "=" value: block1 ``` + +### Non-goals + +- Propagate is not intended to be a general-purpose query engine. The initial + request format should not support joins, grouping, arbitrary functions, random + sampling, or row selection by position. +- We also don't support shorthands such as `*` since that would make it less + explicit which columns are used from just looking at the request file without + also seeing the data package metadata. + From c66421ae85d124c3e57da531e2a7dabd0c961bf7 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 12:04:45 +0200 Subject: [PATCH 05/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20move=20check=20sec?= =?UTF-8?q?tion=20to=20cli=20check=20command?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/cli.qmd | 27 +++++++++++++++++++++----- docs/design/interface/request-file.qmd | 15 -------------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index fb47b48..7d73af5 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -170,11 +170,28 @@ flowchart LR ## {{< var planned >}} `seedcase-propagate check` The `check` command confirms that the `request.yaml` file is correctly -structured and that it matches the content of the source metadata file. While -internally `build`, `subset`, and `create-request` run this check when -processing or creating the `request.yaml` file, we also expose it so the user -can manually check the request file (e.g. if included within an automated -pipeline). It has the following signature with two positional arguments: +structured and that the matches the content of the source metadata file. More +precisely, the following checks are made: + +- Required sections and keys are present. +- The referenced data package contains the correct id and name. +- The referencd resources exist in the data package. +- The requested columns exist in the selected resources. +- Row filters reference existing columns. +- Row filters use supported SQL-like syntax. +- Only allowed SQL operations and shorthands are used. + +While most of these check are likely easier to test directly in the yaml file, +the subset queries filters can also be converted to SQL and checked with +existing SQL parsing or validation tools where possible. This avoids +implementing a custom expression parser for the request format. + +Internally `build`, `subset`, and `create-request` run this check when +processing or creating the `request.yaml` file. We also expose it as a command +so the user can manually check the request file (e.g. if included within an +automated pipeline). + +`check` has the following signature with two positional arguments: ```bash {filename="Terminal"} seedcase-propagate check [REQUEST] [SOURCE] diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index fd3a686..bf3bd90 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -228,21 +228,6 @@ rows: could be harder with this more strict spec, but it doesn't sound like that is our intended use case currently. -## Checking - -Propagate should check the request file before generating a data subset. The -checks should include that: - -- Required sections and keys are present. -- The referenced data package and resources exist. -- The requested columns exist in the selected resources. -- Row filters reference existing columns. -- Row filters use supported SQL-like syntax. - -The row filters can be converted to SQL and checked with existing SQL parsing or -validation tools where possible. This avoids implementing a custom expression -parser for the request format. - ### A complete example A complete request file might look like this. From aa3bbe0bb276c73149fd7be3477c2378c573b7c6 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 12:11:32 +0200 Subject: [PATCH 06/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20fix=20typos=20and?= =?UTF-8?q?=20checking?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .config/typos.toml | 2 +- docs/design/interface/cli.qmd | 2 +- docs/design/interface/request-file.qmd | 11 +++++------ 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/.config/typos.toml b/.config/typos.toml index 3607c85..bc5706f 100644 --- a/.config/typos.toml +++ b/.config/typos.toml @@ -1,5 +1,5 @@ [files] -extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg"] +extend-exclude = ["*.html", "*.css", ".quarto/*", "_site/*", "*.svg", "*.js"] [default.extend-words] ratatui = "ratatui" diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index 7d73af5..6423d27 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -175,7 +175,7 @@ precisely, the following checks are made: - Required sections and keys are present. - The referenced data package contains the correct id and name. -- The referencd resources exist in the data package. +- The referenced resources exist in the data package. - The requested columns exist in the selected resources. - Row filters reference existing columns. - Row filters use supported SQL-like syntax. diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index bf3bd90..ec92dcd 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -51,7 +51,7 @@ request: - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00` - `date-created`: When the first iteration of the request was created. - + - `motivation`: Text explaining why the requester's project needs access to this data subset. - `iteration`: The iteration of requests from this project for this data package. @@ -115,7 +115,7 @@ subsets: - COLUMN-NAME ``` - + - `resource`: Specifies the data resource to create the subset from. Multiple subsets can be specified in the same request by repeating the `resource` key, @@ -218,9 +218,9 @@ rows: ``` - This alternative is more closely mapped to the interface, e.g. the dropdown - for the operator coresponds to the "operator" key here and the condition - groups in the interface are also more simlar to how they are grouped here with - "any". + for the operator corresponds to the "operator" key here and the condition + groups in the interface are also more similar to how they are grouped here + with "any". - An advantage of this syntax is that it is more structured, so it would be easier to check that e.g. only allowed operators are used if someone manually edits the file. @@ -279,4 +279,3 @@ subsets: - We also don't support shorthands such as `*` since that would make it less explicit which columns are used from just looking at the request file without also seeing the data package metadata. - From cda4e483df00d6aa1397e060ffba2f6cf9fa4f33 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 12:15:12 +0200 Subject: [PATCH 07/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20fix=20yaml=20code?= =?UTF-8?q?=20block=20syntax?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/request-file.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/request-file.qmd index ec92dcd..29d9376 100644 --- a/docs/design/interface/request-file.qmd +++ b/docs/design/interface/request-file.qmd @@ -34,7 +34,7 @@ Each of these is elaborated on below: The `request` section describes this specific request. -```yaml +```{.yaml} request: date-modified: DATE date-created: DATE @@ -61,7 +61,7 @@ request: The `requester` section describes who is requesting the data and the project the data will be used for. -```yaml +```{.yaml} requester: name: TEXT email: TEXT @@ -81,7 +81,7 @@ requester: The `data-package` section identifies the data package the request applies to. -```yaml +```{.yaml} data-package: id: TEXT name: TEXT From 9adf2454aea73d7b61b4eaf35a94bad61f9badfa Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 7 Jul 2026 12:17:09 +0200 Subject: [PATCH 08/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20rename=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- _quarto.yml | 2 +- docs/design/interface/{request-file.qmd => requests.qmd} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename docs/design/interface/{request-file.qmd => requests.qmd} (100%) diff --git a/_quarto.yml b/_quarto.yml index 6a28f72..abb8cfd 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -47,7 +47,7 @@ website: contents: - docs/design/interface/inputs.qmd - docs/design/interface/outputs.qmd - - docs/design/interface/request-file.qmd + - docs/design/interface/request.qmd - docs/design/interface/cli.qmd - docs/design/interface/web.qmd - docs/design/interface/rust.qmd diff --git a/docs/design/interface/request-file.qmd b/docs/design/interface/requests.qmd similarity index 100% rename from docs/design/interface/request-file.qmd rename to docs/design/interface/requests.qmd From 2e9344915b74db8c0beede1bf8c4012fd8a1212f Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Thu, 9 Jul 2026 16:51:16 +0200 Subject: [PATCH 09/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/cli.qmd | 4 ++-- docs/design/interface/requests.qmd | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index 6423d27..79fef9b 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -181,9 +181,9 @@ precisely, the following checks are made: - Row filters use supported SQL-like syntax. - Only allowed SQL operations and shorthands are used. -While most of these check are likely easier to test directly in the yaml file, +While most of these check are likely easier to test directly in the YAML file, the subset queries filters can also be converted to SQL and checked with -existing SQL parsing or validation tools where possible. This avoids +existing SQL parsers or checkers where possible. This avoids implementing a custom expression parser for the request format. Internally `build`, `subset`, and `create-request` run this check when diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 29d9376..d8e9e63 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -44,7 +44,7 @@ request: - `date-modified`: When the request was last modified. This can act as an ID which facilitates referring to a specific iteration of a request, e.g. in - dialog between the owner and requester. Precision to the second should be + dialog between the Owner and Requester. Precision to the second should be sufficient. - Note that this field should not be updated via the user interface, it should be automatically populated when the request is saved. @@ -105,7 +105,7 @@ subsets: - COLUMN-NAME - COLUMN-NAME rows: - - "ROW-CONDITION" # As valid SQL + - "ROW-CONDITION" # As SQL # Alternatively, this more structured format, see comparison below - column: COLUMN-NAME operator: "OPERATOR" # E.g. ">" From cdc524a458c4a3ffe7e138ad691aebeeeadc2e03 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Thu, 9 Jul 2026 18:19:27 +0200 Subject: [PATCH 10/25] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com> Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com> --- docs/design/interface/cli.qmd | 6 +++--- docs/design/interface/requests.qmd | 24 ++++++++++++------------ 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index 79fef9b..6a10f68 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -170,7 +170,7 @@ flowchart LR ## {{< var planned >}} `seedcase-propagate check` The `check` command confirms that the `request.yaml` file is correctly -structured and that the matches the content of the source metadata file. More +structured and that it matches the content of the source metadata file. More precisely, the following checks are made: - Required sections and keys are present. @@ -186,9 +186,9 @@ the subset queries filters can also be converted to SQL and checked with existing SQL parsers or checkers where possible. This avoids implementing a custom expression parser for the request format. -Internally `build`, `subset`, and `create-request` run this check when +Internally `build`, `subset`, and `create-request` run these checks when processing or creating the `request.yaml` file. We also expose it as a command -so the user can manually check the request file (e.g. if included within an +so the user can check the request file independently (e.g. if included within an automated pipeline). `check` has the following signature with two positional arguments: diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index d8e9e63..f1ca522 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -1,5 +1,5 @@ --- -title: "Request File" +title: "Request file" --- The request file, `request.yaml`, contains the instructions for requesting a @@ -21,7 +21,7 @@ filters. ## Structure -The request file contains four top-level sections: +The request file contains the following top-level sections: - `request`: Information about this specific request. - `requester`: Information about the person and project requesting the data. @@ -95,7 +95,7 @@ and version. ### Subsets -The `subsets` section describes which data subsets the requester would like to create. +The `subsets` section describes which data subsets the Requester would like to create. ```{.yaml} subsets: @@ -177,6 +177,7 @@ subsets: - The initial supported row filter syntax includes the following operators: - Equality and inequality: `=`, `!=` + - E.g. "age = 18" or "NOT age > 18" - Comparisons: `>`, `>=`, `<`, `<=` - E.g. `"age >= 18"` - Membership: `IN` @@ -186,10 +187,10 @@ subsets: - Missing values: `IS NULL`, `IS NOT NULL` - E.g. `"end_date IS NULL"` - This alternative is more closely mapped to how the SQL syntax will look, and - the options from the interface would need to be combined into the sql row + the options from the interface would need to be combined into the SQL row filter syntax. - An advantage of this syntax is that it is more flexible. This might mainly - matter when editing manually and we allow more expressiveness here than, which + matter when editing manually and we allow more expressiveness here, which doesn't seem to be a common use case based on - A drawback could be that requesters write things we don't want to allow and that we would need to filter out at the SQL level or by parsing the string @@ -197,7 +198,7 @@ subsets: #### Alternative 2 -- A more structure key value mapping for the different components of the filter +- A more structured key-value mapping for the different components of the filter condition. ```{.yaml} @@ -273,9 +274,8 @@ subsets: ### Non-goals -- Propagate is not intended to be a general-purpose query engine. The initial - request format should not support joins, grouping, arbitrary functions, random - sampling, or row selection by position. -- We also don't support shorthands such as `*` since that would make it less - explicit which columns are used from just looking at the request file without - also seeing the data package metadata. +Propagate is not intended to be a general-purpose query engine. The initial +request format should not support joins, grouping, arbitrary functions, random +sampling, or row selection by position. We also don't support shorthands such as `*` since that would make it less +explicit which columns are used from just looking at the request file without +also seeing the data package metadata. From 93474535b4daccc104909b646eaf0d2577d96c8e Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Thu, 9 Jul 2026 18:44:37 +0200 Subject: [PATCH 11/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/requests.qmd | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index f1ca522..8ab8a8d 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -53,6 +53,11 @@ request: - `date-created`: When the first iteration of the request was created. - `motivation`: Text explaining why the requester's project needs access to this data subset. + This is particularly needed for human health data, as how or why data is used + and for what purposes is a legal requirement (e.g. GDPR). This, combined with + the `project` details, is used by the Owner to check whether the requested + data matches whether it is actually relevant to the project's and Requester's + needs. - `iteration`: The iteration of requests from this project for this data package. From b3da1c0ec50e7b17f408c564d0992a96b388d26e Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Fri, 10 Jul 2026 14:47:38 +0200 Subject: [PATCH 12/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/requests.qmd | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 8ab8a8d..a18da1a 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -70,13 +70,19 @@ data will be used for. requester: name: TEXT email: TEXT - project: - name: TEXT - title: TEXT ``` - `name`: The name of the requester. - `email`: The email of the requester. + +### Project + +The `project` section contains information about the project that requires +the data. A project is any set of tasks and work that result in a research +output, e.g. a paper, report, website, dashboard, or analysis. +This would be, e.g. a part of a PhD student's thesis or a research +project for a group a researchers. + - `project.name`: A short machine-readable project name, using lowercase letters and hyphens where possible, e.g. `my-research-project`. - `project.title`: A longer human-readable project title, e.g. From aae4b26940b9261be3eb25cbee9e9780370b2921 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Fri, 10 Jul 2026 14:51:47 +0200 Subject: [PATCH 13/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/requests.qmd | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index a18da1a..c0d5080 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -83,11 +83,16 @@ output, e.g. a paper, report, website, dashboard, or analysis. This would be, e.g. a part of a PhD student's thesis or a research project for a group a researchers. -- `project.name`: A short machine-readable project name, using lowercase letters - and hyphens where possible, e.g. `my-research-project`. -- `project.title`: A longer human-readable project title, e.g. - `My research project`. - +- `name`: A short machine-readable project name, using lowercase letters + and hyphens where possible, e.g. `my-research-project`. This is mainly used + for identification purposes when managing several requests. +- `title`: A longer human-readable project title, e.g. + `My research project`. This could be used by the Owner to publicly display all + the requests that people are making to the data package. +- `description`: This is a summary of the project, including details like + background, rationale, basic methods, and potential research questions. + This information, combined with the `motivation`, is mainly to help the + Owner ensure that the data requested matches what it will be used for. ### Data Package The `data-package` section identifies the data package the request applies to. From da8fa3f2ab37d8c66548afb8d5ff5afd2315ae92 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Thu, 9 Jul 2026 18:45:01 +0200 Subject: [PATCH 14/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20clarify=20language?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/cli.qmd | 7 ++++--- docs/design/interface/requests.qmd | 10 +++++++--- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index 6a10f68..238ce76 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -174,7 +174,8 @@ structured and that it matches the content of the source metadata file. More precisely, the following checks are made: - Required sections and keys are present. -- The referenced data package contains the correct id and name. +- The referenced data package has the same id and name of the source metadata + file. - The referenced resources exist in the data package. - The requested columns exist in the selected resources. - Row filters reference existing columns. @@ -183,8 +184,8 @@ precisely, the following checks are made: While most of these check are likely easier to test directly in the YAML file, the subset queries filters can also be converted to SQL and checked with -existing SQL parsers or checkers where possible. This avoids -implementing a custom expression parser for the request format. +existing SQL parsers or checkers where possible. This avoids implementing a +custom expression parser for the request format. Internally `build`, `subset`, and `create-request` run these checks when processing or creating the `request.yaml` file. We also expose it as a command diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index c0d5080..b3d188b 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -24,15 +24,19 @@ filters. The request file contains the following top-level sections: - `request`: Information about this specific request. -- `requester`: Information about the person and project requesting the data. +- `requester`: Information about the person requesting the data. +- `project`: Information about the research project that requires the data. - `data-package`: Information about the data package the request applies to. -- `subsets`: The requested subsets, with one item per resource. +- `subsets`: The requested data subsets, with one item per resource. Each of these is elaborated on below: ### Request -The `request` section describes this specific request. +The `request` includes information about the request itself, such as when and +why it was created. This can help an owner organize and track requests, which +would primarily be beneficial for data packages that receive large volumes of +requests. ```{.yaml} request: From e613d934c67b37933a2e3491526d9f3e01b4d461 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Fri, 10 Jul 2026 14:47:28 +0200 Subject: [PATCH 15/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20improve=20owner=20?= =?UTF-8?q?wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index b3d188b..7c7c999 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -56,19 +56,21 @@ request: - `date-created`: When the first iteration of the request was created. -- `motivation`: Text explaining why the requester's project needs access to this data subset. - This is particularly needed for human health data, as how or why data is used - and for what purposes is a legal requirement (e.g. GDPR). This, combined with - the `project` details, is used by the Owner to check whether the requested - data matches whether it is actually relevant to the project's and Requester's - needs. -- `iteration`: The iteration of requests from this project for this data package. +- `motivation`: Explanation for why requester's project needs access to this + data subset. This is used by the Owner to check whether the requested data is + relevant to the project. The motivation is particularly needed for human + health data since there are legal requirements (e.g. GDPR) to declare for what + purposes such data is used. +- `iteration`: The iteration of requests from this project for this data + package. ### Requester -The `requester` section describes who is requesting the data and the project the -data will be used for. +The `requester` section describes who is requesting the data. This person +doesn't strictly have to be the one working with the data, but they should serve +as the main point of contact for the Owner. + ```{.yaml} requester: From 9fe1cc3192aa5f78aa092d232a4d3d546f83e8b7 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Fri, 10 Jul 2026 14:51:42 +0200 Subject: [PATCH 16/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20improve=20proj=20d?= =?UTF-8?q?esc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 33 ++++++++++++++++-------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 7c7c999..4dc2484 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -83,22 +83,25 @@ requester: ### Project -The `project` section contains information about the project that requires -the data. A project is any set of tasks and work that result in a research -output, e.g. a paper, report, website, dashboard, or analysis. -This would be, e.g. a part of a PhD student's thesis or a research -project for a group a researchers. - -- `name`: A short machine-readable project name, using lowercase letters - and hyphens where possible, e.g. `my-research-project`. This is mainly used - for identification purposes when managing several requests. -- `title`: A longer human-readable project title, e.g. - `My research project`. This could be used by the Owner to publicly display all - the requests that people are making to the data package. +The `project` section contains information about the research project that +requires the data. A project is defined as a body of work that result in a +research output, e.g. a paper, report, website, dashboard, or analysis. This +could be a part of a PhD student's thesis or a collaboration research project +for a group a researchers. This field is necessary to ensure that the data +requested is aligned with what it is intended to be used for, which is a legal +requirement (e.g. in GDPR). + +- `name`: A short machine-readable project name, using lowercase letters and + hyphens where possible, e.g. `my-research-project`. This is mainly used for + identification purposes when managing several requests. +- `title`: A longer human-readable project title, e.g. `My research project`. + This could be used by the Owner to publicly display all the requests that + people are making to the data package. - `description`: This is a summary of the project, including details like - background, rationale, basic methods, and potential research questions. - This information, combined with the `motivation`, is mainly to help the - Owner ensure that the data requested matches what it will be used for. + background, rationale, basic methods, and potential research questions. This + information, combined with the `motivation`, is mainly to help the Owner + ensure that the data requested matches what it will be used for. + ### Data Package The `data-package` section identifies the data package the request applies to. From f0bdb36f52478dc9b0ee7ec7f21a7bfe642538e8 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Fri, 10 Jul 2026 15:07:29 +0200 Subject: [PATCH 17/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/requests.qmd | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 4dc2484..de8bef7 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -102,9 +102,13 @@ requirement (e.g. in GDPR). information, combined with the `motivation`, is mainly to help the Owner ensure that the data requested matches what it will be used for. -### Data Package +### Data package The `data-package` section identifies the data package the request applies to. +The main purpose of this section is to keep track of the version of the data +package the request was made for. That way, the Owner can determine if a +given request needs an update to the data depending on which version of the +data it uses. ```{.yaml} data-package: From 9d43ec6866798e3225eae09253fff449d13303a0 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Mon, 13 Jul 2026 11:49:33 +0200 Subject: [PATCH 18/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20improve=20wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 32 +++++++++++++++++------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index de8bef7..5f2d676 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -105,10 +105,8 @@ requirement (e.g. in GDPR). ### Data package The `data-package` section identifies the data package the request applies to. -The main purpose of this section is to keep track of the version of the data -package the request was made for. That way, the Owner can determine if a -given request needs an update to the data depending on which version of the -data it uses. +This information will be automatically populated from the data package metadata +when creating a request via the web application. ```{.yaml} data-package: @@ -117,10 +115,12 @@ data-package: version: X.Y.Z ``` -This information will be automatically populated from the data package metadata -when creating a request via the web application. Including this information -helps Owners check that the request was created for the intended data package -and version. +- `id`: The unique identifier of the data package helps identify where data is + requested from. +- `name`: The name of the data package gives a human readable name to refer to. +- `version`: The version of the data package helps the owner the Owner detect if + a request was made towards an outdated version of the data package and + therefore needs to be updated. ### Subsets @@ -147,13 +147,17 @@ subsets: - `resource`: Specifies the data resource to create the subset from. Multiple - subsets can be specified in the same request by repeating the `resource` key, - either with the same resource multiple times or different ones. -- `columns`: A list of columns to include. Features such as selecting a range of - columns are handled via the interface and expanded to explicit column lists in - the request file. + subsets can be specified in the same request by repeating the `resource` key. + This can be done either with the same resource multiple times or with + different resources for each. +- `columns`: The columns to include. Each list item must correspond to an exact + column name and SQL-like shorthands are not supported in order to keep the + request file simple and to avoid parsing unsafe queries of the data (e.g. via + SQL injection). This can not be left empty / unspecified; at least one column + must be included in the `columns` field. - `rows`: Describes which rows to include in the subset. Can reference columns - that are not part of the subset. + that are not part of the subset. If the rows key is empty or not included, all + rows will be selected. - There are currently two candidates for what this might look like, see below. From eda9c812453214d2277a0c21d0e9761d9781df55 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Mon, 13 Jul 2026 11:50:12 +0200 Subject: [PATCH 19/25] Apply suggestions from code review Co-authored-by: Luke W. Johnston --- docs/design/interface/requests.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 5f2d676..56738fe 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -4,7 +4,7 @@ title: "Request file" The request file, `request.yaml`, contains the instructions for requesting a subset of data from a data package. The file is created by a Requester through -the web application and is used by an Owner to check and generate the requested +the web application, terminal interface, or even manually (though not recommended). The file is used by an Owner to check and generate the requested data subset. The request file is intended to be both human-readable and machine-readable. It From ac34861165502aacc2e58e5c6beb42e666c5f9bc Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Mon, 13 Jul 2026 12:00:02 +0200 Subject: [PATCH 20/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20include=20more=20a?= =?UTF-8?q?tbout=20NOT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 56738fe..b16bfec 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -208,9 +208,8 @@ subsets: ::: - The initial supported row filter syntax includes the following operators: - - Equality and inequality: `=`, `!=` - - E.g. "age = 18" or "NOT age > 18" + - Equality: `=` + - E.g. "age = 18" - Comparisons: `>`, `>=`, `<`, `<=` - E.g. `"age >= 18"` - Membership: `IN` @@ -219,6 +218,8 @@ subsets: - E.g. `"visit_date BETWEEN '2020-01-01' AND '2020-12-31'"` - Missing values: `IS NULL`, `IS NOT NULL` - E.g. `"end_date IS NULL"` + - Negation: `NOT` + - E.g. `NOT =`, `NOT IN`, `NOT BETWEEN` - This alternative is more closely mapped to how the SQL syntax will look, and the options from the interface would need to be combined into the SQL row filter syntax. From 2ecec384d8310067b88348e28e252d4e56c74560 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 14 Jul 2026 15:42:03 +0200 Subject: [PATCH 21/25] Apply suggestions from code review Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com> --- docs/design/interface/cli.qmd | 2 +- docs/design/interface/requests.qmd | 9 ++++----- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/design/interface/cli.qmd b/docs/design/interface/cli.qmd index 238ce76..fc4e86b 100644 --- a/docs/design/interface/cli.qmd +++ b/docs/design/interface/cli.qmd @@ -174,7 +174,7 @@ structured and that it matches the content of the source metadata file. More precisely, the following checks are made: - Required sections and keys are present. -- The referenced data package has the same id and name of the source metadata +- The referenced data package has the same id and name as the source metadata file. - The referenced resources exist in the data package. - The requested columns exist in the selected resources. diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index b16bfec..9436fa9 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -56,7 +56,7 @@ request: - `date-created`: When the first iteration of the request was created. -- `motivation`: Explanation for why requester's project needs access to this +- `motivation`: Explanation for why the requester's project needs access to this data subset. This is used by the Owner to check whether the requested data is relevant to the project. The motivation is particularly needed for human health data since there are legal requirements (e.g. GDPR) to declare for what @@ -87,7 +87,7 @@ The `project` section contains information about the research project that requires the data. A project is defined as a body of work that result in a research output, e.g. a paper, report, website, dashboard, or analysis. This could be a part of a PhD student's thesis or a collaboration research project -for a group a researchers. This field is necessary to ensure that the data +for a group of researchers. This field is necessary to ensure that the data requested is aligned with what it is intended to be used for, which is a legal requirement (e.g. in GDPR). @@ -118,7 +118,7 @@ data-package: - `id`: The unique identifier of the data package helps identify where data is requested from. - `name`: The name of the data package gives a human readable name to refer to. -- `version`: The version of the data package helps the owner the Owner detect if +- `version`: The version of the data package helps the Owner detect if a request was made towards an outdated version of the data package and therefore needs to be updated. @@ -224,8 +224,7 @@ subsets: the options from the interface would need to be combined into the SQL row filter syntax. - An advantage of this syntax is that it is more flexible. This might mainly - matter when editing manually and we allow more expressiveness here, which - doesn't seem to be a common use case based on + matter when editing manually as we allow more expressiveness here. - A drawback could be that requesters write things we don't want to allow and that we would need to filter out at the SQL level or by parsing the string which is likely more error prone than a more structured format. From f6941a643793b4808752b969398c8bd292433e3b Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Tue, 14 Jul 2026 16:23:17 +0200 Subject: [PATCH 22/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20Add=20link=20to=20?= =?UTF-8?q?user=20types?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 9436fa9..4c60f71 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -4,8 +4,10 @@ title: "Request file" The request file, `request.yaml`, contains the instructions for requesting a subset of data from a data package. The file is created by a Requester through -the web application, terminal interface, or even manually (though not recommended). The file is used by an Owner to check and generate the requested -data subset. +the web application, terminal interface, or even manually (though not +recommended). The file is used by an Owner to check and generate the requested +data subset. You can +[read more about the Requester and Owner user types in the Design overview page](/docs/design/interface/inputs.qmd) The request file is intended to be both human-readable and machine-readable. It should be simple enough for a Requester to understand and, when needed, edit From d0076255a28be15a9d55e2ba0702001718b5203c Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Wed, 15 Jul 2026 06:57:15 +0200 Subject: [PATCH 23/25] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com> --- docs/design/interface/requests.qmd | 35 ++++++++++++++++-------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 4c60f71..00bcc63 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -36,8 +36,8 @@ Each of these is elaborated on below: ### Request The `request` includes information about the request itself, such as when and -why it was created. This can help an owner organize and track requests, which -would primarily be beneficial for data packages that receive large volumes of +why it was created. This helps an Owner organize and track requests, which +is especially beneficial for data packages that receive large volumes of requests. ```{.yaml} @@ -52,8 +52,8 @@ request: which facilitates referring to a specific iteration of a request, e.g. in dialog between the Owner and Requester. Precision to the second should be sufficient. - - Note that this field should not be updated via the user interface, it should - be automatically populated when the request is saved. + - Note that this field is + automatically populated when the request is saved via a user interface (be it the web app or the TUI). If the request is edited manually, it must be updated manually. - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00` - `date-created`: When the first iteration of the request was created. @@ -62,7 +62,7 @@ request: data subset. This is used by the Owner to check whether the requested data is relevant to the project. The motivation is particularly needed for human health data since there are legal requirements (e.g. GDPR) to declare for what - purposes such data is used. + purposes the data is used. - `iteration`: The iteration of requests from this project for this data package. @@ -90,16 +90,16 @@ requires the data. A project is defined as a body of work that result in a research output, e.g. a paper, report, website, dashboard, or analysis. This could be a part of a PhD student's thesis or a collaboration research project for a group of researchers. This field is necessary to ensure that the data -requested is aligned with what it is intended to be used for, which is a legal +requested is aligned with what it is intended to be used for, which can a legal + requirement (e.g. in GDPR). - `name`: A short machine-readable project name, using lowercase letters and hyphens where possible, e.g. `my-research-project`. This is mainly used for identification purposes when managing several requests. - `title`: A longer human-readable project title, e.g. `My research project`. - This could be used by the Owner to publicly display all the requests that - people are making to the data package. -- `description`: This is a summary of the project, including details like + This could be used by the Owner to publicly display the data package requests. +- `description`: A summary of the project, including details like background, rationale, basic methods, and potential research questions. This information, combined with the `motivation`, is mainly to help the Owner ensure that the data requested matches what it will be used for. @@ -108,7 +108,8 @@ requirement (e.g. in GDPR). The `data-package` section identifies the data package the request applies to. This information will be automatically populated from the data package metadata -when creating a request via the web application. +This information will be automatically populated from the data package metadata +when creating a request via the web application or the TUI. ```{.yaml} data-package: @@ -118,11 +119,12 @@ data-package: ``` - `id`: The unique identifier of the data package helps identify where data is - requested from. -- `name`: The name of the data package gives a human readable name to refer to. +- `id`: The unique identifier of the data package. +- `name`: The name of the data package gives a human readable name to refer to. It is lowercased and doesn't include any spaces. E.g., `my-data-package`. - `version`: The version of the data package helps the Owner detect if a request was made towards an outdated version of the data package and - therefore needs to be updated. +- `version`: The version of the data package helps the Owner detect which + version of the data package the request is for. ### Subsets @@ -153,14 +155,15 @@ subsets: This can be done either with the same resource multiple times or with different resources for each. - `columns`: The columns to include. Each list item must correspond to an exact - column name and SQL-like shorthands are not supported in order to keep the + column name. SQL-like shorthands are not supported in order to keep the request file simple and to avoid parsing unsafe queries of the data (e.g. via SQL injection). This can not be left empty / unspecified; at least one column must be included in the `columns` field. - `rows`: Describes which rows to include in the subset. Can reference columns - that are not part of the subset. If the rows key is empty or not included, all + that are not a part of the subset. If the `rows` key is empty or not included, all rows will be selected. - - There are currently two candidates for what this might look like, see below. + +There are currently two candidates for what this might look like, see below. From ff41b617c4d83d2d0963be8cfcecad51f2a8021b Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Wed, 15 Jul 2026 07:01:59 +0200 Subject: [PATCH 24/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20Improve=20wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 00bcc63..466dc13 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -36,9 +36,8 @@ Each of these is elaborated on below: ### Request The `request` includes information about the request itself, such as when and -why it was created. This helps an Owner organize and track requests, which -is especially beneficial for data packages that receive large volumes of -requests. +why it was created. This helps an Owner organize and track requests, which is +especially beneficial for data packages that receive large volumes of requests. ```{.yaml} request: @@ -52,8 +51,9 @@ request: which facilitates referring to a specific iteration of a request, e.g. in dialog between the Owner and Requester. Precision to the second should be sufficient. - - Note that this field is - automatically populated when the request is saved via a user interface (be it the web app or the TUI). If the request is edited manually, it must be updated manually. + - Note that this field is automatically populated when the request is saved + via a user interface (be it the web app or the TUI). If the request is + edited manually, it must be updated manually. - The format should be ISO 8601, e.g. `2026-07-06T01:45:34-12:00` - `date-created`: When the first iteration of the request was created. @@ -108,7 +108,6 @@ requirement (e.g. in GDPR). The `data-package` section identifies the data package the request applies to. This information will be automatically populated from the data package metadata -This information will be automatically populated from the data package metadata when creating a request via the web application or the TUI. ```{.yaml} @@ -118,13 +117,11 @@ data-package: version: X.Y.Z ``` -- `id`: The unique identifier of the data package helps identify where data is -- `id`: The unique identifier of the data package. -- `name`: The name of the data package gives a human readable name to refer to. It is lowercased and doesn't include any spaces. E.g., `my-data-package`. -- `version`: The version of the data package helps the Owner detect if - a request was made towards an outdated version of the data package and -- `version`: The version of the data package helps the Owner detect which - version of the data package the request is for. +- `id`: A unique identifier of the data package. +- `name`: A human readable name to refer to. It is lowercased and doesn't + include any spaces. E.g., `my-data-package`. +- `version`: Helps the Owner detect which version of the data package the + request is for. ### Subsets @@ -160,8 +157,8 @@ subsets: SQL injection). This can not be left empty / unspecified; at least one column must be included in the `columns` field. - `rows`: Describes which rows to include in the subset. Can reference columns - that are not a part of the subset. If the `rows` key is empty or not included, all - rows will be selected. + that are not a part of the subset. If the `rows` key is empty or not included, + all rows will be selected. There are currently two candidates for what this might look like, see below. From c4df5161ef77b0b3c6989f3c52b0d2e820a2cc04 Mon Sep 17 00:00:00 2001 From: Joel Ostblom Date: Wed, 15 Jul 2026 07:13:54 +0200 Subject: [PATCH 25/25] =?UTF-8?q?docs:=20=F0=9F=93=9D=20remove=20comment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/interface/requests.qmd | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/design/interface/requests.qmd b/docs/design/interface/requests.qmd index 466dc13..87c76c0 100644 --- a/docs/design/interface/requests.qmd +++ b/docs/design/interface/requests.qmd @@ -162,10 +162,6 @@ subsets: There are currently two candidates for what this might look like, see below. - - - - #### Row condition alternatives ##### Alternative 1