Skip to content

15 design of experiments: planning, factorial and space-filling notebooks - #12

Open
yamierick wants to merge 6 commits into
mainfrom
doe/factorial-and-space-filling
Open

yamierick wants to merge 6 commits into
mainfrom
doe/factorial-and-space-filling

Conversation

@yamierick

Copy link
Copy Markdown
Contributor

What

A new subject folder, notebooks/15_design_of_experiments/, with three notebooks on one synthetic five-factor reactor, plus the orteach.doe package module and its tests.

Notebook For Seed
planning_an_experiment The basics, for anyone planning a study. Six steps in order: noise, run count and replicates, the reduction ladder, randomizing and blocking, what existing runs can estimate, which runs to add. Each step has a short section of theory. seed_planning.py
factorial_designs The deep dive behind planning steps 2, 3 and 6: effects, half-normal plot, fractions and aliasing, center points. seed_factorial.py
space_filling_designs Computer experiments only: Latin hypercubes, maximin. seed_space_filling.py

The folder README gives the reading order and says where to stop.

Key numbers (from outputs)

  • Detecting a 2-point effect at 80% power needs about 28 runs in total. The recommended 2^5 + 4 center points (36 runs) gives an exact power of 0.810, and a 4,000-rep simulation gives 0.809.
  • Power along the reduction ladder: 729 runs 1.000, 100 runs 0.999, 36 runs 0.810, half fraction + 4 centers 0.315, quarter + 4 centers 0.231.
  • Runs taken one factor at a time can estimate 6 of 16 terms, with no error df. D-optimal augmentation keeps those six runs and adds 16 to reach 17 estimable terms.
  • Fold-over cuts the quarter fraction's false stirring effect from +6.06 to +0.13.

Data

The instance is invented and generated from a printed seed. No student, lab or sponsor data is included, and no private material is committed.

Checks

  • pytest: 251 passed, 36 of them in test_doe.py
  • tools/check_notebooks.py: 25 notebooks, 0 failing
  • tools/check_seeds.py: 19 seeds, 0 failures
  • Guards were tested by breaking them on purpose:
    • A wrong effect_se fails two tests and the planning agreement cell.
    • A >= tie-break in augment fails only the agreement cell. pytest does not catch it.

Not done

  • Colab: the notebooks clone REF = v1.0.1, which predates orteach.doe, so they run locally but not in Colab until a v1.1.0 release:
    • bump paths.TAG, the pyproject version and REF in the seeds and notebooks;
    • re-execute the notebooks, then tag.
  • Response-surface / steepest-ascent and degradation-study notebooks are named in the README as not yet written.

🤖 Generated with Claude Code

yamierick and others added 6 commits November 19, 2026 08:28
…ooks

New subject folder, three notebooks on one synthetic five-factor reactor
(data/raw/doe_reactor_runs.csv, generated by orteach.doe from a printed seed
and a stated true model). No student or lab data; the instance is invented.

Source: written new for this library, prompted by recurring dissertation-
committee questions from students in other departments (how many runs, how
many replicates, how to cut a design down, what to do with runs already
taken). No term folder was migrated, so there is no prior defect to record.

What each notebook does:

- planning_an_experiment (seed_planning.py): six steps in the order a study
  is planned, each with one piece of theory.
  1. Noise: replicates vs repeated measures; chi-square interval for sigma
     and a one-sided bound as the planning value.
  2. Run count: SE = 2 sigma / sqrt(N), N ~ 31 (sigma/Delta)^2, hidden
     replication, replicates by number of factors; exact noncentral-t power
     (0.810 for the 36-run design) checked by simulation (0.809).
  3. Reduction ladder 3^5x3 -> 2^5x3+4 -> 2^5+4 -> half -> quarter, with the
     power each rung keeps; why center points add df but not precision.
  4. Randomize and block: drift as a fake effect (bias = drift x the effect's
     correlation with run order); blocking on ABCDE leaves main effects and
     2FIs unchanged.
  5. Runs already taken: OFAT rank 6, no error df, variance 2 sigma^2 vs
     4 sigma^2 / N.
  6. Runs to add: greedy D-optimal augmentation with a block column,
     D-efficiency, fold-over (stirring's +6.06 alias falls to +0.13), axial
     points named.
  Closes with an answer key (true sigma 1.5: planned power 0.81, actual 0.95).

- factorial_designs (seed_factorial.py): effects by hand, half-normal plot and
  Lenth margins, half and quarter fractions with alias algebra drawn on the
  cube, center-point curvature test. Now signposts which planning step or
  ladder rung each section explains.

- space_filling_designs (seed_space_filling.py): Latin hypercubes in 2-D and
  3-D, maximin, why run count does not grow with factors. Marked as for
  computer experiments only.

Package: orteach.doe (model_matrix, effect_se, sigma_interval, sigma_bound,
power, runs_needed, augment, d_efficiency, log_det, plus the factorial and
LHS functions). AUGMENT_RIDGE is a regularizer, not a comparison tolerance.

Tests: tests/test_doe.py, 36 tests, including a coverage simulation for the
sigma interval, t-power vs simulation, t-power -> normal as df grows, OFAT
rank and variance, augmentation rank and no repeats, D-efficiency of full and
half fractions, and a concavity bound on the ridge.

Guards checked by breaking them: an effect_se using len(x) fails two tests
and the planning agreement cell; a >= tie-break in augment fails only the
planning agreement cell (the augmentation order), not pytest.

Checks: pytest 251 passed; check_notebooks 25 notebooks, 0 failing;
check_seeds 19 seeds, 0 failures.

Not done: Colab clones REF v1.0.1, which does not contain orteach.doe, so
these notebooks run locally but not in Colab until a v1.1.0 release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source: Jones's review of PR #12. The notebook used regression and showed
confounding (drift, aliasing) but never said why a regression line through
runs nobody designed is not an experiment.

Adds a second theory section to step 5, "a regression line is not an
experiment":

- Omitted-variable bias, b_A = beta_A + beta_B * delta, as "correlation is not
  causation" written as a formula, and the three ways to get delta = 0:
  measure and include (costs 1/sqrt(1 - r^2) in SE), set the factors
  orthogonally, randomize.
- 40 logged runs simulated from the reactor's true model (seed 1119), with
  catalyst tied to temperature (r = 0.85). Temperature-only slope 9.34,
  95% CI 8.39 to 10.29, misses the true 6; the formula reproduces it exactly;
  adding catalyst gives 6.29 with SE inflated 1.92x; the designed 2^5 gives
  5.847 with or without catalyst.
- Figure: logged runs coloured by catalyst with fitted and true slopes, and
  the four coefficient intervals against the truth.
- What to do with undesigned runs, tied to step 6 (det(M'M) ~ 1 - r^2 for two
  columns, so D-optimal augmentation decorrelates them).

Also: intro table and "In short" step 5, an RHO exercise, the agreement cell
now compares the four coefficients against doe.coefficients (30 numbers,
agree to 2.2e-16), folder README theory list.

Checks: pytest 251 passed; check_notebooks 25, 0 failing; check_seeds 19,
0 failures; ruff clean. Earlier outputs unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source: Jones's review of PR #12. Studies often measure a vector of
responses per run; the notebook handled one response only.

Adds "More than one response" at the end of step 2:

- Theory: Y = MB + E, B-hat = (M'M)^-1 M'Y. M depends only on the settings,
  so the same runs serve every response, and the joint fit equals separate
  fits column by column (checked: gap 1.4e-14).
- Purity, a second response simulated on the same 36 runs from a model stated
  in the cell (seed 2612, sigma 0.8, Delta 1). Sizing per response: purity's
  sigma/Delta 1.364 against yield's 0.945, 58.4 runs against 28.1; the 36-run
  design gives purity 0.504 power. So the second response sets the size.
- False alarms: 32 tests, 1.6 expected by chance. Purity flags BE and CE at
  5%, not in its true model; Bonferroni keeps exactly A, D, AD, and also drops
  yield's real curvature, so the prose recommends a primary response.
- Figure: t statistics for both responses from one model matrix, with the 5%
  and Bonferroni lines.
- Pointer: readings of one unit over time are repeated measures; two-stage
  curve parameters become the columns of Y. Degradation notebook not written.

Seed choice: 2611 gave a purity pilot s of 0.205 against a true 0.8, which
made purity look easy and hid the point; 2612 gives 0.789. The exercise asks
the reader to vary SEED_P and see how often four center runs mislead.

Also: intro table and "In short" step 2, a DELTA_P/SEED_P exercise, folder
README theory list; agreement cell now 48 numbers (purity sigma plan, runs
needed, power, 15 coefficients vs doe.coefficients), agree to 7.1e-15.

Checks: pytest 251 passed; check_notebooks 25, 0 failing; check_seeds 19,
0 failures; ruff clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The abstract still counted the notebooks before 02 and 15 were added.
README and CLAUDE.md already say twenty-five.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source: Jones's review of PR #12. The plain analysis script written for a
student's study was much easier to follow than this notebook's code, which
used terse names (M36, tc36, B_joint, Xo, sig_lo), packed several ideas into
each line, and had few comments.

Every code cell in seed_planning.py now uses plain names (design36, t_crit36,
coef_both, X_ofat, sigma_low), one idea per block, and a comment on each step.
Specific changes:

- The reduction ladder is a list of dicts with named fields (runs, sum_x2,
  terms, se, df, power) instead of positional tuples read as r[3], r[4].
- Term names and their column indices are built once (terms, term_index) and
  reused, instead of re-deriving "ABCDE".index(ch) in five cells.
- The fold-over mirror rows come from one readable comprehension instead of a
  32x8 boolean matrix and argmax.
- The augmentation loop names its pieces (current, M_trial, best_logdet) and
  says why ties go to the first candidate.
- Plot calls are one per line instead of chained with semicolons.

What the notebook computes is unchanged, and this is checked: all 27 text
outputs (every printed number, table and Figure line) are byte-identical to
the previous execution, so no prose number moved. The exercise that named
words2[:5] now names terms[:5].

Checks: pytest 251 passed; check_notebooks 25, 0 failing; check_seeds 19,
0 failures; ruff clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Source: Jones's review of PR #12, following the same rewrite of the planning
notebook (ec09fbf). The other two DOE notebooks used the same terse style.

factorial_designs (seed_factorial.py):
- Plain names: Xh/yh -> X_half/y_half, Xq/yq -> X_quarter/y_quarter,
  n_f/n_c -> n_corner/n_center, q -> hn_quantile, hi/lo -> mean_high/mean_low,
  corner -> is_corner, C -> corr.
- The interaction cell names its four run groups (A_high, B_low, ...) instead
  of repeating the boolean conditions inline.
- Lenth's PSE, ME and SME each carry a comment saying what they are.
- Comments on every step; one plot call per line instead of semicolon chains.

space_filling_designs (seed_space_filling.py):
- Comments on the construction, the distance computation, the maximin search
  and the sampling study; the repeated pair index is named once (pair).
- One plot call per line, including the 3-D axes, instead of semicolon chains.
- Every random-number call is left exactly where it was, so the drawn designs
  cannot change.

What the notebooks compute is unchanged, and this is checked against the
previous execution: factorial's 23 text outputs are byte-identical;
space-filling's printed text is identical once Jupyter's stream chunking is
ignored (one cell's print now arrives as one chunk instead of two).

Checks: pytest 251 passed; check_notebooks 25, 0 failing; check_seeds 19,
0 failures; ruff clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant