Model test continuation (#130)

* fix(saxs): harden experimental data loading on Windows

Incorporate the safe parts of PR #125 onto the up-to-date continuation branch. Preserve comment-header column detection for metadata-heavy experimental files and constrain the experimental data header dialog so long Windows paths do not oversized-crash the UI.

Co-authored-by: KWWyatt <117381914+KWWyatt@users.noreply.github.com>

* feat(bondanalysis): expand analysis outputs and controls

Add dihedral and coordination-number analysis support, richer GDS-oriented statistics, updated plotting/export behavior, and focused UI/CLI coverage for the bond analysis workflow.

* feat(clusterdynamics): improve plotting and project startup

Add project-aware defaults, richer trajectory plotting/export behavior, startup progress handling, and updated cluster dynamics prediction workflows with focused tests and docs.

* feat(saxs): add project tooling and EXAFS workflows

Add project layout/memory support, Direct Frame SAXS, EXAFS GDS mapping workflows, shared plotting/editor improvements, DREAM/project UI refinements, and tests for the updated SAXSShell tool surface.

* docs: refresh SAXSShell user guide navigation

Update general user-guide pages and navigation for the current project workflow, cluster dynamics organization, output locations, and tool behavior without including the spectroscopy mapping or geometric model reference files.

---------

Co-authored-by: KWWyatt <117381914+KWWyatt@users.noreply.github.com>

Author: kewh5868

.gitignore

@@ -84,6 +84,7 @@ site/
 tests/born_vs_debye_backend_debug_*/
 tests/contrast_fft_backend_debug_*/
 tests/edm_contiguous_mode_report/
+tests/direct_frame_saxs_beta/
 tests/representativefinder_performance/output_results/
 tests/smearing_analysis_*/
 tests/CsPbI3_SAXS_fit/
@@ -92,6 +93,10 @@ tests/CsPbI3_SAXS_fit.zip
 # Generated reports
 docs/reports/
 
+# Generated direct-frame SAXS beta outputs
+direct_frame_saxs_runs/
+examples/direct_frame_saxs_runs/
+
 # PyBuilder
 target/
 

README.rst

@@ -607,15 +607,16 @@ new, resumed, or already complete.
 ``bondanalysis``
 ~~~~~~~~~~~~~~~~
 
-The ``bondanalysis`` application is designed for bond-pair and
-angle-triplet distribution analysis on the stoichiometry-level cluster
-folders produced by ``clusters``. A typical workflow is:
+The ``bondanalysis`` application is designed for bond-pair,
+angle-triplet, dihedral-quartet, and coordination-number distribution
+analysis on the stoichiometry-level cluster folders produced by
+``clusters``. A typical workflow is:
 
 1. Use ``mdtrajectory`` to export frames.
 2. Use ``clusters`` to sort those frames into stoichiometry folders.
 3. Open ``bondanalysis`` on the cluster-output folder.
-4. Choose the bond pairs and angle triplets to measure, along with
-   their cutoff distances.
+4. Choose the bond pairs, angle triplets, dihedral quartets, and
+   coordination rules to measure, along with their cutoff distances.
 5. Run the analysis to write CSVs, histograms, comparison plots, and a
    JSON manifest into a sibling ``bondanalysis_*`` folder.
 
@@ -632,10 +633,10 @@ To launch the Qt application from the repository root ::
 Bondanalysis UI Use
 -------------------
 
-The Qt interface focuses only on bond-pair and angle-distribution
-analysis. The legacy displacement-analysis tooling is intentionally not
-part of the new window and should be treated as deprecated until it is
-updated.
+The Qt interface focuses on bond-pair, angle, dihedral, and
+coordination-distribution analysis. The legacy displacement-analysis
+tooling is intentionally not part of the new window and should be
+treated as deprecated until it is updated.
 
 Launch the UI ::
 
@@ -661,7 +662,12 @@ Inside the window:
    distance cutoff in angstrom.
 6. Add one or more angle-triplet rows using the vertex atom, the two
    arm atoms, and the two vertex-arm cutoffs in angstrom.
-7. Click ``Analyze Bond Pairs and Angle Distributions``.
+7. Add one or more dihedral-quartet rows using four atoms and the
+   three adjacent-pair cutoffs in angstrom. For Pb-coordinated DMF,
+   ``Pb-O-C-N`` extracts the signed torsion needed for GDS constraint
+   setup.
+8. Click ``Analyze Bond, Angle, Dihedral, and Coordination
+   Distributions``.
 
 Each run writes:
 
@@ -671,8 +677,8 @@ Each run writes:
    types
    under ``all_clusters/``.
 3. Overlay comparison CSVs, NPYs, and PNG plots under ``comparisons/``.
-4. A ``bondanalysis_manifest.json`` file describing the run inputs and
-   outputs.
+4. A ``bondanalysis_results_index.json`` file describing the run inputs,
+   outputs, and GDS variable registry.
 
 Bondanalysis Terminal Use
 -------------------------
@@ -681,12 +687,24 @@ Inspect a clusters directory before running analysis ::
 
         PYTHONPATH=src conda run --no-capture-output -n saxshell-py312 python -m saxshell.bondanalysis inspect clusters_splitxyz0001
 
-Run bond-pair and angle analysis headlessly on every cluster type ::
+Run bond-pair, angle, and dihedral analysis headlessly on every cluster
+type ::
 
         PYTHONPATH=src conda run --no-capture-output -n saxshell-py312 python -m saxshell.bondanalysis run clusters_splitxyz0001 \
             --bond-pair Pb:I:3.50 \
             --bond-pair Pb:O:3.20 \
-            --angle-triplet Pb:I:I:3.50:3.50
+            --angle-triplet Pb:I:I:3.50:3.50 \
+            --dihedral Pb:O:C:N:3.20:2.20:2.20
+
+Each histogram CSV includes ordinary distribution statistics plus GDS-oriented
+metadata. Bond-distance histograms add ``gds_center_angstrom``,
+``gds_sigma_angstrom``, and ``gds_sigma2_angstrom_squared``. Angle and
+dihedral histograms add degree and radian forms, including
+``gds_center_degrees``, ``gds_sigma_degrees``, ``gds_center_radians``,
+``gds_sigma_radians``, and ``gds_variance_radians_squared``. The metadata also
+includes ``gds_*_variable`` names and paste-ready Artemis ``set`` rows, and
+the run's ``bondanalysis_results_index.json`` registers those variables across
+the saved cluster and aggregate distributions.
 
 Restrict the run to selected stoichiometry folders and choose an
 explicit output directory ::
@@ -710,6 +728,7 @@ inspection and execution steps used by the UI and terminal interfaces.
             AngleTripletDefinition,
             BondAnalysisWorkflow,
             BondPairDefinition,
+            DihedralQuartetDefinition,
         )
 
         workflow = BondAnalysisWorkflow(
@@ -721,6 +740,11 @@ inspection and execution steps used by the UI and terminal interfaces.
             angle_triplets=[
                 AngleTripletDefinition("Pb", "I", "I", 3.50, 3.50),
             ],
+            dihedral_quartets=[
+                DihedralQuartetDefinition(
+                    "Pb", "O", "C", "N", 3.20, 2.20, 2.20
+                ),
+            ],
         )
         summary = workflow.inspect()
         result = workflow.run()

docs/getting-started/project-setup.md

@@ -108,7 +108,7 @@ The **Component build mode** dropdown controls what happens when you click
 Representative structures are optional project-backed files that compatible
 Debye, Born, FFT, and RMCSetup workflows can use instead of average cluster
 folders. Use **Tools > Structure Analysis > Open Representative Structures** for
-the full interactive analysis UI, or use **Tools > CLI Setup > Open
+the full interactive analysis UI, or use **Tools > (beta) > CLI Setup > Open
 Representative CLI Setup (Beta)** to save
 `representative_structure_cli_run.json` and run the same backend from the
 source checkout:
@@ -117,8 +117,8 @@ source checkout:
 PYTHONPATH=src conda run --no-capture-output -n saxshell-py312 python -m saxshell.representativefinder run /path/to/project
 ```
 
-The same **Tools > CLI Setup** menu can prepare project-local run files for
-XYZ-to-PDB conversion, cluster extraction, cluster dynamics, and cluster
+The same **Tools > (beta) > CLI Setup** menu can prepare project-local run files
+for XYZ-to-PDB conversion, cluster extraction, cluster dynamics, and cluster
 dynamics ML. Those run files let you run long jobs from a terminal or batch
 several prepared project folders while keeping outputs linked back to their
 projects.

docs/user-guide/bond-analysis.md

@@ -1,8 +1,8 @@
 # Bond Analysis
 
 The **Bond Analysis** tool is SAXSShell's structure-analysis application for
-measuring bond-length and angle distributions from stoichiometry-sorted cluster
-folders.
+measuring bond-length, angle, dihedral, and coordination distributions from
+stoichiometry-sorted cluster folders.
 
 ## Launching the application
 
@@ -22,25 +22,80 @@ The current UI supports:
 - limiting the run to checked stoichiometry labels
 - defining bond-pair cutoffs directly in a table
 - defining angle triplets directly in a table
+- defining signed dihedral quartets directly in a table
 - loading built-in presets and saving custom presets for later reuse
 - reopening an existing bond-analysis output folder and browsing its saved
   distributions
 
 The right side of the window focuses on the computed distributions. You can
-refresh a results directory, select one or more saved bond-pair or angle
-entries, and open them in a dedicated plot window. Matching items from multiple
-cluster types can be overlaid together for comparison.
+refresh a results directory, select one or more saved bond-pair, angle,
+dihedral, or coordination entries, and open them in a dedicated plot window.
+Matching items from multiple cluster types can be overlaid together for
+comparison.
 
 ## Typical workflow
 
 1. Start from the project's sorted clusters folder.
 2. Confirm or choose the bond-analysis output directory.
 3. Refresh the detected cluster types and clear any stoichiometries you do not
    want to include.
-4. Load a preset or define the bond pairs and angle triplets manually.
+4. Load a preset or define the bond pairs, angle triplets, and dihedral
+   quartets manually.
 5. Run the calculation and inspect the saved distributions from the results
    browser.
 
+Dihedral quartets use adjacent-pair cutoffs: `ATOM1-ATOM2`, `ATOM2-ATOM3`,
+and `ATOM3-ATOM4` must each be within their requested cutoff. The reported
+values are signed degrees in `[-180, 180]`. The calculation projects the two
+outer bonds onto the plane perpendicular to the middle bond and uses `atan2` to
+keep the torsion sign. `-180` and `+180` are the same anti-aligned planar
+torsion at the wrap boundary, while `0` is the aligned planar torsion and is
+not equivalent to `180`.
+
+When plotting dihedral distributions, the saved values are not modified, but
+the plot display is recentered when helpful. The display center snaps to the
+nearest clean multiple of 90 degrees (`0`, `+90`, `-90`, or `180`) based on the
+circular center, so a wrapped population near `-180` / `+180` appears as one
+continuous peak centered near `180` instead of being split across both plot
+edges. Even when the display is internally recentered, tick labels and summary
+values remain in the signed `[-180, 180]` convention. Dihedral plots also
+provide a plot-style toggle between a normal histogram and a radial histogram.
+The radial view draws all dihedral values on one circular axis, which is useful
+for combined nearly degenerate terminal distributions such as `O-C-N-C` where
+one branch may populate `0` and another may populate the `-180` / `+180`
+boundary.
+
+Each histogram CSV includes ordinary distribution statistics plus
+GDS-oriented metadata. Bond-distance histograms add `gds_center_angstrom`,
+`gds_sigma_angstrom`, and `gds_sigma2_angstrom_squared`. Angle and dihedral
+histograms add `gds_center_degrees`, `gds_sigma_degrees`,
+`gds_center_radians`, `gds_sigma_radians`, and
+`gds_variance_radians_squared`. The same metadata also includes
+`gds_*_variable` names and paste-ready Artemis `set` rows. The run's
+`bondanalysis_results_index.json` registers those variables across the saved
+cluster and aggregate distributions. Dihedral GDS centers and widths use
+circular statistics, so a population split across the `-180` / `+180` histogram
+edge is treated as one wrapped distribution instead of being averaged toward
+`0`.
+
+## EXAFS GDS handoff
+
+After representative structures and bondanalysis are complete, open
+`Tools > Structure Analysis > Open EXAFS GDS Mapping` from the main SAXS UI.
+The mapping window loads the project representative-structure metadata and
+completed bondanalysis result folders, then lets you choose a stoichiometry
+representative, inspect labeled 3D and 2D absorber-scatterer path diagrams,
+select bond, angle, and dihedral registry variables, preview the generated GDS,
+and write a validated Artemis setup file.
+
+The bondanalysis variables are imported as GDS-ready statistical `set` rows.
+They provide the distribution centers and sigmas that can anchor the GDS
+constraint setup while the EXAFS mapping tool builds the path/template side
+from the selected representative structure. Hydrogen-family atoms are excluded
+from the mapping diagrams and generated EXAFS paths. The absorber defaults to
+Pb when Pb is present, and can be changed to another non-hydrogen coordination
+center in the mapping window.
+
 ## Related pages
 
 - [MD Extraction and Cluster Preparation](cluster-extraction.md)

docs/user-guide/cluster-dynamics-ml.md

@@ -1,4 +1,4 @@
-# Cluster Dynamics ML
+# Cluster Dynamics
 
 `clusterdynamicsml` is the predictive companion to `clusterdynamics`. It can be
 launched directly from the source checkout or from the main SAXSShell UI. It
@@ -15,7 +15,7 @@ that includes both observed and predicted structures.
 
 ## What the application does
 
-At a high level, Cluster Dynamics ML answers this question:
+At a high level, Cluster Dynamics answers this question:
 
 > Given the smaller clusters that are observed in a trajectory and the
 > reference structures available for those smaller clusters, what larger
@@ -28,7 +28,7 @@ geometry rules.
 
 ## Inputs
 
-Cluster Dynamics ML expects all of the inputs required by `clusterdynamics`,
+Cluster Dynamics expects all of the inputs required by `clusterdynamics`,
 plus a structure library for the observed smaller clusters.
 
 ### Required inputs
@@ -58,14 +58,14 @@ plus a structure library for the observed smaller clusters.
 6. Set the predicted share threshold used to prune tiny candidate populations.
 7. Optionally load experimental SAXS data.
 8. Run **Analyze and Predict Larger Clusters**.
-9. Review the `Summary`, `Lifetimes`, `Debye-Waller`, `Histograms`, and
+9. Review the `Summary`, `Lifetimes`, `Debye-Waller (beta)`, `Histograms`, and
    `SAXS` tabs.
 10. Save the dataset, CSV exports, or a detailed PowerPoint report if needed.
 
 ## Project-Backed CLI Setup
 
-For background prediction runs, use **Tools > CLI Setup > Open Cluster Dynamics
-ML CLI Setup (Beta)**. The setup window writes
+For background prediction runs, use **Tools > (beta) > CLI Setup > Open Cluster
+Dynamics ML CLI Setup (Beta)**. The setup window writes
 `cluster_dynamics_ml_cli_run.json` in the active project folder and shows the
 terminal commands to run later:
 
@@ -305,7 +305,7 @@ definition.
 
 ## Debye scattering with pairwise Debye-Waller damping
 
-Cluster Dynamics ML now distinguishes between two SAXS-component cases:
+Cluster Dynamics now distinguishes between two SAXS-component cases:
 
 - `Averaged component`
   When a SAXS component is already averaged over many structure files, the
@@ -347,7 +347,7 @@ $\sin(q r_{ij}) / (q r_{ij})$.
 
 ### Debye-Waller-extended single-structure equation
 
-For a single representative structure, Cluster Dynamics ML uses a pairwise
+For a single representative structure, Cluster Dynamics uses a pairwise
 Debye-Waller damping factor on the off-diagonal pair contributions:
 
 $$
@@ -375,7 +375,7 @@ interference terms between distinct atoms are attenuated.
 
 ### Relation between $\sigma$ and $B$
 
-Cluster Dynamics ML reports both the Gaussian displacement width
+Cluster Dynamics reports both the Gaussian displacement width
 $\sigma_{\alpha\beta}$ and the equivalent Debye-Waller $B$ coefficient:
 
 $$
@@ -399,7 +399,7 @@ main-model refinement.
 
 ## How Debye-Waller coefficients are estimated
 
-Cluster Dynamics ML estimates pairwise disorder from the observed structure
+Cluster Dynamics estimates pairwise disorder from the observed structure
 ensembles before it predicts values for the larger clusters.
 
 ### Observed-cluster ensemble estimate
@@ -471,8 +471,8 @@ measure a spread.
 ### Predicted-cluster estimate
 
 The larger predicted clusters do not have their own ensembles yet, so Cluster
-Dynamics ML fits a separate weighted ridge-regression model for each element
-pair type using the observed $\sigma_{L,\alpha\beta}$ values as the training
+Dynamics fits a separate weighted ridge-regression model for each element pair
+type using the observed $\sigma_{L,\alpha\beta}$ values as the training
 targets.
 
 For a candidate feature vector $\mathbf{x}$, the predicted disorder value is:
@@ -493,7 +493,7 @@ $$
 $$
 
 The feature vector is the same one already used for the other Cluster Dynamics
-ML properties:
+properties:
 
 $$
 \mathbf{x} =
@@ -532,11 +532,11 @@ single extrapolated candidate dominate the whole distribution.
 
 ## Outputs
 
-Cluster Dynamics ML can produce:
+Cluster Dynamics can produce:
 
 - the standard time-binned colormap from `clusterdynamics`
 - a combined lifetime table containing observed and predicted rows
-- a `Debye-Waller` table listing the resolved \(\sigma\) and \(B\) values for
+- a `Debye-Waller (beta)` table listing the resolved \(\sigma\) and \(B\) values for
   each observed and predicted element pair
 - histogram views for observed-only and observed-plus-predicted populations
 - SAXS traces for observed-only and observed-plus-predicted models
@@ -622,7 +622,7 @@ model. Related model families include:
 
 ## Why the current algorithm was chosen
 
-Cluster Dynamics ML is trying to extrapolate from a small observed size series
+Cluster Dynamics is trying to extrapolate from a small observed size series
 to larger unobserved clusters. In that setting, a simple regularized model has
 practical advantages:
 
@@ -637,7 +637,7 @@ ridge-style regression plus geometry rules over a higher-capacity learned model.
 
 ## TODO
 
-The current Debye-Waller workflow is intentionally scoped to Cluster Dynamics ML
+The current Debye-Waller workflow is intentionally scoped to Cluster Dynamics
 result inspection and to the single-structure component traces built inside that
 tool. A later extension may expose these pairwise \(B\) or \(\sigma\)
 coefficients to the main SAXS prefit and DREAM refinement templates as optional

docs/user-guide/cluster-dynamics.md

@@ -1,4 +1,4 @@
-# Cluster Dynamics
+# Cluster Dynamics Time-Binned
 
 `clusterdynamics` is the time-resolved companion to `clusters`. It analyzes the
 same extracted XYZ or PDB frame folders from `mdtrajectory`, reuses the same
@@ -80,8 +80,8 @@ directory automatically.
 
 ## Project-Backed CLI Setup
 
-For longer cluster-dynamics runs, use **Tools > CLI Setup > Open Cluster
-Dynamics CLI Setup (Beta)**. The setup window writes
+For longer cluster-dynamics runs, use **Tools > (beta) > CLI Setup > Open
+Cluster Dynamics CLI Setup (Beta)**. The setup window writes
 `cluster_dynamics_cli_run.json` in the active project folder and shows the
 terminal commands to run later:
 
@@ -214,6 +214,6 @@ time basis explicit so the heatmap and lifetime interpretation stay transparent.
 ## Related pages
 
 - [Cluster Extraction](cluster-extraction.md)
-- [Cluster Dynamics ML](cluster-dynamics-ml.md)
+- [Cluster Dynamics](cluster-dynamics-ml.md)
 - [Project Configuration](project-configuration.md)
 - [Results and Export](results-and-export.md)