Issue 1460 export results as parquet format

[samuelafolabi]

Fixes #1460

This PR adds a new parameter save_dashboard_parquet to GGIR that allows users to export a consolidated, web dashboard-ready Parquet file from the GGIR output CSVs.

• When save_dashboard_parquet = TRUE, GGIR writes results/ggir_results.parquet after all requested parts have completed
• Attaches Parquet key-value metadata recording the variable dictionary, activity threshold configuration, and accelerometer metric used
• Cleans all column names to be SQL-friendly (lowercase, no special characters) for direct use with DuckDB or similar query engines
• Find web dashboard repo and live link here- https://github.com/samuelafolabi/ggir-web-dashboard

Checklist before merging:

• Existing tests still work (check by running the test suite, e.g. from RStudio).
• Added tests (if you added functionality) or fixed existing test (if you fixed a bug).
• Clean code has been attempted, e.g. intuitive object names and no code redundancy.
• Documentation updated:
  • Function documentation
  • Chapter vignettes for GitHub IO
  • Vignettes for CRAN
• Corresponding issue tagged in PR message. If no issue exist, please create an issue and tag it.
• Updated release notes in inst/NEWS.Rd with a user-readable summary. Please, include references to relevant issues or PR discussions.
• If you think you made a significant contribution, add your name to the contributors lists in the DESCRIPTION, zenodo.json, and inst/CITATION files.
• GGIR parameters were added/removed. If yes, please also complete checklist below.
  If NEW GGIR parameter(s) were added then these NEW parameter(s) are:
  • documented in man/GGIR.Rd
  • included with a default in R/load_params.R
  • included with value class check in R/check_params.R
  • included in table of vignettes/GGIRParameters.Rmd with references to the GGIR parts the parameter is used in.
  • mentioned in NEWS.Rd as NEW parameter

[vincentvanhees]

Thanks @samuelafolabi looks good, I will review it tomorrow.
I just made some updates to the main branch, could you update your branch with those changes? I do not expect them to interfere with your work, but it may be good to have it all up to date.

[vincentvanhees]

Code looks great Samuel, only a few comments below. My main concern is an error message I get when I test it on one of my files (Axivity cwa).

I am only using GGIR defaults parameter values and save_dashboard_parquet = TRUE.

GGIR(
  datadir = datadir,
  outputdir = outputdir,
  overwrite = TRUE,
  mode = 1:5,
  do.report = c(2, 4, 5),
  save_dashboard_parquet  = TRUE
)

which results in error

Error: Invalid: Can only convert data frames to Struct type

I think the error originates around line 174 in write_parquet.R, where the nested epoch-level time series are attached as a list-column. From that point onward arrow::arrow_table(consolidated) is no longer possible.

[samuelafolabi]

Code looks great Samuel, only a few comments below. My main concern is an error message I get when I test it on one of my files (Axivity cwa).

I am only using GGIR defaults parameter values and save_dashboard_parquet = TRUE.

GGIR(
  datadir = datadir,
  outputdir = outputdir,
  overwrite = TRUE,
  mode = 1:5,
  do.report = c(2, 4, 5),
  save_dashboard_parquet  = TRUE
)

which results in error

Error: Invalid: Can only convert data frames to Struct type

I think the error originates around line 174 in write_parquet.R, where the nested epoch-level time series are attached as a list-column. From that point onward arrow::arrow_table(consolidated) is no longer possible.

@vincentvanhees Thank you for reveiwing.
I'll check this out and fix as soon as I can. I was only able to test with the test file I created using the create_test_file function before I made a PR.

Is it possible you share some files with me so I could rigorously test before tagging you to the PR again?

Thanks once again.

[samuelafolabi]

Thanks @samuelafolabi looks good, I will review it tomorrow. I just made some updates to the main branch, could you update your branch with those changes? I do not expect them to interfere with your work, but it may be good to have it all up to date.

Yea, thanks for the heads-up. I saw the recent commits. I'll pull them in before I make these fixes.

[vincentvanhees]

Is it possible you share some files

No problem, I have sent you an email with the link.

[samuelafolabi]

90fbb43#diff-d2e3eb08f9cb1813199166ff82920b03c3c7759ad946fa9f43bd7cdcc06bca94

@vincentvanhees

[vincentvanhees]

Thanks, all looks better now. A few more comments.

Please also update the NEWS.md file in the root of the repository. As you will see there is a block for GGIR version 3.3-? at the top of that file to reflect the upcoming release. Please add a line to this block. For example:

- Functionality added to save all key output to one parquet file per person. #1460

where the #1460 is my way of tracking the GitHub item an update corresponded to.

[vincentvanhees]

"Related to #1460" in the first message should probably be "Fixes #1460" because that will make sure the issue is auto-closed once the PR is merged.

[samuelafolabi]

"Related to #1460" in the first message should probably be "Fixes #1460" because that will make sure the issue is auto-closed once the PR is merged.

Oh yes! I had plans of reading the comment section of the issue over again after the PR is merged. I assumed it would help reinforce my understanding of things and avoid making similar mistakes in future. Then the issue could be manually closed after that. @vincentvanhees

[samuelafolabi]

@vincentvanhees I have made the fixes required. Should have done it earlier but I have been under the weather. Kindly review. Please let me know if there are any other things needed to be done.

I also noticed you just made a release some hours ago. I pulled the latest changes in from main and created a new section in the NEWS.md file titled "# CHANGES IN GGIR VERSION 3.3-", since my new updates could only possibly get pushed with the next release.

https://github.com/samuelafolabi/GGIR/blob/310c7267fb359e4318616902a50683f74dbbb6ab/NEWS.md?plain=1#L1-L18

[vincentvanhees]

Thanks @samuelafolabi, as you can see the checks failed. Let me know if you need any help fixing this:

No problem regarding your speed of work, I have also been quite busy myself. In terms of next steps: How is the dashboard progressing? Are you planning to make it public? We can also email about this.

[samuelafolabi]

Ah! Thank you, I see it. I think it is something I can fix. I'll let you know if otherwise.

The dashboard is progressing well; and yes, I am making it public. I'll send you a progress report ASAP.

[samuelafolabi]

Hello @vincentvanhees. Fixed it. I reproduced the main GitHub Actions checks locally.

I was only left with this warning and a note (Find attached image) after fixing. I guess it should pass now.

[vincentvanhees]

Looks great @samuelafolabi
Thanks for adapting the code to store the parquet files in a separate directory.