Stage 4, Phase 2 — Applying the Pipeline to All Datasets (4.2-Per_Dataset_Pipeline)¶

Per 4-Projet/index.html, Section 2 / Task 2: wrap the physics + ML pipeline into a single reusable function and apply it to all 6 {rgs000,rgs180}x{0,m4,p4} calibration configurations, logging results to a new MLflow experiment ("per_dataset").

Continuing an open question from Phase 1. 4.1-ML_Comparison.ipynb compared five ML model families for residual correction on rgs000_0 alone, both per-axis (MLP(y), MLP(z) independently) and jointly (MLP(y,z), one model for both axes). It dropped SVM (in favor of this project's planned NN-based direction, not a clean numeric win) but explicitly left the independent-vs-joint MLP choice open — the margin was small (~6%) on rgs000_0's own test set and reversed sign on the rgs180_0 generalization check. Phase 1's Recap says this comparison "continues in Phase 2". This notebook carries both MLP variants through all 6 datasets and looks for a consistent winner across independent fits, rather than one dataset's noise-level margin.

Design decisions carried over from planning (see 4-Projet/TODO.md/the project plan):

• Physical parameters: loaded from the frozen Stage 3 fits (models/stage3_fit_<cfg>.toml, all 6 already exist) — not refit here. Same pattern 4.1-ML_Comparison.ipynb used for rgs000_0.
• ML correction: the winning MLP hyperparameters per axis/joint are read back from Phase 1's residual_correction MLflow experiment (no pre-selected "best" run was saved there — every swept combination is logged, so this reads the run history directly), then retrained fresh on each dataset's own train split — not the literal rgs000_0-trained model object (that cross-dataset generalization question is what Phase 1 Section 7 already tested once, separately).

1. Fetch Phase 1's Candidate MLP Hyperparameters¶

No "best" model was saved to MLflow in Phase 1 (see its Recap) — every swept hyperparameter combination is logged in the residual_correction experiment, both independent-per-axis runs (axis="y"/"z", Phase 1 Section 4) and joint runs (axis="joint", Phase 1 Section 4b). Reading the winning combination back from that run history (rather than re-typing the numbers 4.1-ML_Comparison.ipynb printed) keeps this notebook traceable to Phase 1's tracked runs, the same way Phase 1 itself picked best_by_axis and joint_best.

2. Reusable Per-Dataset Pipeline Function¶

run_pipeline(cfg) does the physics step once (load the frozen Stage 3 fit, compute residuals, group-aware 80/20 split) and then trains both ML candidates from it:

• Candidate A — independent: two separate MLPRegressors (mlp_y_params, mlp_z_params), one per axis.
• Candidate B — joint: one MLPRegressor (mlp_joint_params) predicting (r_y, r_z) together.

Both are fit fresh on this dataset's own train split (not the rgs000_0-trained object). The deciding metric throughout is the combined point-distance mean(hypot(r_y, r_z)) — the same metric 4.1-ML_Comparison.ipynb Section 4 established as "the mm-scale number that actually matters", not per-axis RMSE alone.

3. MLflow Experiment: per_dataset¶

Two runs per dataset (one per candidate), same one-run-per-combination granularity Phase 1 used — not one row per dataset hiding which approach won.

4. Run Across All 6 Configs¶

5. Parameter Consistency Analysis¶

Physical parameters don't depend on the ML candidate, so this uses one row per dataset (either approach's copy — they're identical). Per the physics-model-calibration skill, checking which parameters look consistent across datasets identifies candidates for Phase 3's "shared" treatment.

Reading the consistency results¶

• offset_y_mm: consistent across all 6 datasets — mean −0.524 mm, std 0.041 mm (~8% relative spread), no grouping by grism identity or tilt. A good candidate for a shared parameter in Phase 3.
• rho: close-but-not-identical between the two grism instances, as Stage 3 already found — within-group std (~0.025 for rgs000, ~0.024 for rgs180) is actually about the same size as the across-group mean difference (0.0097). Confirms instance-specific, identifiable: not a single shared value, but not dataset-specific either — one value per physical grism.
• offset_z_mm — surprising, not consistent. Std (0.796 mm) is ~20x larger than offset_y_mm's, and it isn't scattered randomly: every rgs000_* value is positive (0.31–1.24 mm) and every rgs180_* value is negative (−0.02 to −0.95 mm) — a clean split by grism identity, not detector drift. This lines up with 3-Intro_ML.ipynb's own degeneracy discussion: flipping tilt_deg by 180° negates the dispersion-direction unit vector in Grism.forward, and detector.offset_z absorbs whatever constant term is left over — differently, depending on which side of that 180° flip a given grism sits on. So offset_z_mm, as currently parameterized, is not purely "single detector, one registration constant" the way offset_y_mm is — it's carrying a mounting-orientation-dependent sign. Flagged for Phase 3, not resolved here: naively sharing offset_z_mm across all 6 datasets would be physically wrong; either it stays dataset- (or at least grism-instance-)specific, or Phase 3's model needs an explicit sign/parity term tied to grism identity before offset_z itself can be shared.
• tilt_deg: as expected, tracks the commanded GWA angle. Cross-check against Stage 3's rgs180 = rgs000 + 180° finding: rgs000_0→rgs180_0 differ by 180.10°, rgs000_m4→rgs180_m4 by 180.05°, rgs000_p4→ rgs180_p4 by 179.96° — all within ~0.1° of exactly 180°, confirming the frozen Stage 3 TOMLs were loaded correctly here.

6. RMSE-Across-Datasets Comparison¶

Physical explanation for the hardest-to-fit configuration¶

Physical-only combined distance, all 6 configs (mm): rgs000_0 0.308, rgs000_m4 0.293, rgs000_p4 0.349, rgs180_0 0.328, rgs180_m4 0.350, rgs180_p4 0.313.

No clean split by grism identity: rgs000_p4 (0.349) is nearly as hard as the overall hardest rgs180_m4 (0.350), and harder than rgs180_p4 (0.313) — so "rgs180 fits worse than rgs000" doesn't hold here. The pattern that does hold: the two ±4° configs at the extremes are both tilted configs (rgs000_m4 easiest, rgs180_m4/rgs000_p4 hardest), while both 0°-tilt configs (rgs000_0, rgs180_0) sit in a narrower, middling band (0.308–0.328 mm). Tilted configurations show more spread in fit quality (both the best and the two worst results are tilted configs) than the nominal 0° position. A plausible physical reading: 0° is likely the GWA's most repeatable mechanical stop (its nominal/rest position), while ±4° commanded positions depend on the grism-wheel actuator's positioning repeatability — small mechanical backlash or angle-readout error at a non-rest position would show up as extra fit residual that varies from one ±4° acquisition to the next, rather than as a systematic per-grism effect. The overall spread across all 6 configs is modest either way (0.293–0.350 mm, ~20% relative) — no configuration is a dramatic outlier.

7. Independent vs. Joint MLP: Cross-Dataset Verdict¶

The comparison Phase 1 left open, now run 6x instead of once. Phase 1's own single-dataset margin was small (~6%) and flipped sign on the rgs180_0 generalization check — the question here is whether a consistent winner emerges across 6 independent fits, or the same noise-level pattern persists.

Residual structure on the config with the largest divergence¶

Predicted-vs-actual for both candidates on the dataset where they disagree most (worst_divergence_cfg above) — not all 6 datasets, to keep this notebook proportionate; the aggregate table above is the primary evidence.

Verdict¶

Tied, 3 datasets to 3. independent (MLP(y)+MLP(z)) wins on rgs000_0, rgs180_0, rgs180_p4; joint (MLP(y,z)) wins on rgs000_m4, rgs000_p4, rgs180_m4. Mean margin is +0.80% in independent's favor (i.e. essentially zero), and the per-dataset margins (−7.0% to +5.9%) show no consistent sign by grism identity or by tilt direction — the winner alternates dataset to dataset with no visible driver. This is the same noise-level pattern Phase 1 found on a single dataset (test-set margin flipped sign on the rgs180_0 generalization check); running it 6 more times did not resolve it into a consistent winner — if anything it confirms the two approaches really are comparably accurate on this data, not that either notebook's comparison was underpowered.

The residual-structure check on rgs180_m4 (the largest divergence, −7.0%, favoring joint) shows both candidates tracking the actual residuals about equally well on both axes — no visible qualitative difference explaining the numeric margin.

Per the verdict rule: with no consistent-sign winner across 6 independent fits, this stays undecided on accuracy grounds. Falling back to the hybrid-residual-modeling skill's simplicity-vs-accuracy default — prefer the simpler option unless the more complex one clearly earns its complexity — MLP(y)+MLP(z) (independent) is carried forward as the default for Phase 3: two smaller, independently tunable per-axis models are simpler to reason about and diagnose than one shared-weight network predicting both axes, and nothing in this notebook's 6-dataset comparison shows the joint model earning its added complexity. This is a documented tie-breaker, not a claim that independent is more accurate — Phase 3 or later work is free to revisit if new evidence (e.g. a case where the two axes' residuals are known to be correlated) favors joint modeling.

8. Recap¶

• Reusable pipeline (run_pipeline(cfg)) built and run across all 6 {rgs000,rgs180}×{0,m4,p4} configs. Physical parameters loaded from the frozen Stage 3 fits (models/stage3_fit_<cfg>.toml) — row-count assertion passed for all 6, and the tilt_deg cross-check reconfirmed rgs180 ≈ rgs000 + 180° (within ~0.1°), so the reused fits loaded correctly.
• MLflow "per_dataset" experiment: 12 runs logged (one independent + one joint per dataset), each with the fitted physical parameters, MLP hyperparameters used, and physical/hybrid test metrics.
• Parameter consistency (Section 5): offset_y_mm behaves as a good shared-parameter candidate (~8% relative spread, no grouping); rho is instance-specific but identifiable, close between rgs000/rgs180 as Stage 3 predicted. offset_z_mm was not consistent — it splits cleanly by grism identity (positive for rgs000_*, negative for rgs180_*) rather than clustering around one shared value, traced to the same tilt+180° degeneracy 3-Intro_ML.ipynb flagged. This is new, concrete input for Phase 3's shared-vs-specific classification, not something Stage 3's qualitative discussion alone showed.
• RMSE across datasets (Section 6): physical-only fit quality stays in a fairly narrow band (0.293–0.350 mm combined distance) across all 6 configs; the two extremes are both ±4°-tilt configurations, not a rgs000-vs-rgs180 split — read as GWA positioning repeatability being less consistent away from the 0° nominal stop, not a grism-specific effect. Hybrid ML correction (either candidate) improves 77–83% over physical-only, consistently across all 6 datasets.
• Independent (MLP(y)+MLP(z)) vs. joint (MLP(y,z)) (Section 7): the question Phase 1 left open stayed open — a 3-3 tie across the 6 datasets, mean margin ≈0.8%, no consistent-sign pattern. Per the documented tie-breaker (simplicity-vs-accuracy default from the hybrid-residual-modeling skill), MLP(y)+MLP(z) (independent) is carried forward as the working choice — not because it measured more accurate, but because nothing here shows the joint model earning its extra complexity.
• Stops here per this phase's scope. Phase 3 (4-Projet/index.html Section 3, "Multi-Dataset Fitting" — a combined cost function jointly optimizing shared parameters across all 6 datasets, then re-optimizing dataset-specific ones, then adding ML correction) is out of scope for this notebook and gets its own notebook once started. The offset_z_mm inconsistency finding above is directly relevant to how that phase should classify parameters as shared vs. dataset-specific.