Stage 4, Phase 1 — ML Model Comparison on a Single Dataset (4.1-ML_Comparison)¶

Per 4-Projet/index.html, Section 1 / Task 1: starting from the optimized physical model of Stage 3, systematically compare several ML models for residual correction on rgs000_0, tracking every model + hyperparameter combination as its own MLflow run.

Scope: this notebook covers Phase 1 only (residual_correction MLflow experiment, single dataset). Stage 4 has 6 phases total (see the deck's roadmap table); Phase 2 ("Applying the Pipeline to All Datasets") and beyond get their own notebooks (4.2-..., etc.) once this phase is reviewed -- nothing here wraps the pipeline into a reusable multi-dataset function.

Model target: (y_nisp, z_nisp, wavelength) -> (r_y, r_z), the residual left over after Stage 3's physical fit -- same convention as 3-Intro_ML.ipynb Section 5.

Fixed parameters -- cross-checked against Euclid-NISP-Specs.md¶

This phase doesn't refit the physical model -- it runs entirely on top of the frozen Stage 3 fit (models/stage3_fit_rgs000_0.toml), which itself holds several parameters fixed at GroundTestModel.nominal_params (models/stage1_instrument.toml). Cross-checked against the authoritative instrument specs (Euclid-NISP-Specs.md, Jahnke et al. 2024, Table 2 / §2) before building on them:

Everything held fixed here is either spec-grounded or was already flagged non-identifiable in Stage 3 for an independent reason -- no unexplained guesses. One more spec note worth flagging so it isn't misread later: the spec's 0.3"/pix plate scale is an angular on-sky quantity, distinct from 4-Projet/index.html's own 1 NISP pixel ~ 0.3 mm physical pixel size -- they share the number 0.3 but are not the same fact. That only matters for Phase 5's literature-unit conversion, not this notebook's RMSE-in-mm comparisons.

1. Starting Point: Stage 3's Physical Fit¶

Load models/stage3_fit_rgs000_0.toml directly -- free_parameters + [fit.result] give exactly the free_names/theta Stage 3 already fit via scipy.optimize.minimize, so there's no refit here. Load rgs000_0 the same way Stage 3 did (median_per_spectrum(load_spectra(...))), exclude the same mislabeled-position outlier spectrum the TOML records, and compute the residuals r_y, r_z via predict_centroids -- this is "the optimized physical model of Section 3" the deck's Task 1 says to start from (mirrors add_residuals() in 3-Intro_ML.ipynb).

2. Train/Test Split (group-aware)¶

Features X = [y_nisp, z_nisp, wavelength], one model per axis (r_y, r_z), an 80/20 split via GroupShuffleSplit on spectra_id -- the same group-aware pattern 3-Intro_ML.ipynb established (each spectrum contributes ~30 rows at the same field position; a plain row-wise split would leak position information across train/test, per the hybrid-residual-modeling skill's natural-grouping rule).

3. MLflow Experiment¶

Tracking store is sqlite:///mlflow.db at the repo root -- mlflow 3.x has put the plain file store (./mlruns) into maintenance mode and recommends a database backend, so this notebook uses that instead of the deck's literal ./mlruns example. Same idea, current mechanism. Run pixi run mlflow-ui from the repo root to browse the residual_correction experiment.

4. Systematic Hyperparameter Comparison¶

Five model families, per the deck's table, each searched over >=3 hyperparameter combinations (GridSearchCV-style exhaustive grid for RandomForest/GradientBoosting, RandomizedSearchCV-style sampling for SVM/MLP since those grids are larger). LinearRegression has no hyperparameters to search (the deck's own table lists "None" for it), so it gets a single run per axis -- the baseline.

Every combination gets its own MLflow run (not just the best -- the deck's box-yellow instruction on the intro slide is explicit about this): each candidate is refit on the train split and logged with its hyperparameters (mlflow.log_param) plus train_rmse, test_rmse, and train_test_ratio = test_rmse / train_rmse (mlflow.log_metric) -- ratio defined this way so it's always >=1 with larger meaning more overfit, matching the deck's "which model overfits most (largest train/test ratio)" phrasing directly.

Results table¶

Pivoted one row per model family (rather than one row per axis) so each family's two per-axis choices can be paired up into a single combined-axes metric: distance between the corrected prediction and the actual data point, mean(hypot(hybrid_r_y, hybrid_r_z)) -- the mm-scale number that actually matters for the deck's Task 1 step 3 ("compare all runs, identify the best model"), since test_rmse_y and test_rmse_z alone don't say how good the combined 2D correction is. A Physics-only row (no ML correction) is included as the baseline.

4b. Joint 2D-Output Models — One Model for Both Axes¶

Section 4 fits an independent model per axis (r_y, r_z). Here the same five model families are tested the other way: a single model per hyperparameter combination predicting (r_y, r_z) together, to test whether sharing structure across axes captures correlation an independent-per-axis model can't.

LinearRegression, RandomForestRegressor, and MLPRegressor support multi-output natively -- one real joint model per combination (RandomForestRegressor's trees split to jointly reduce variance across both targets; MLPRegressor's hidden layers are shared across both output units). HistGradientBoostingRegressor and SVR don't support multi-output natively in this sklearn version, so they're wrapped in sklearn.multioutput.MultiOutputRegressor -- worth flagging explicitly (simplicity-vs-accuracy trade-off, per the hybrid-residual-modeling skill): a wrapped model is really still two independent single-target models fit back-to-back under one API call, not a fundamentally different capability than Section 4's independent approach. Same grids as Section 4, same MLflow experiment (residual_correction, logged with axis="joint"), every combination logged as its own run.

Joint models results table¶

One row per model family (best hyperparameter combination by combined test distance), for direct comparison against Section 4's pivoted table above.

Overall re-evaluation: independent vs. joint¶

The real baseline to compare the joint 2D models against isn't any single row of Section 4's pivoted table (which pairs same-family models on both axes) -- it's the true best achievable with two independently chosen per-axis models, i.e. best_by_axis from Section 4 (potentially two different families, e.g. one model for y and a different one for z). That combined-distance number, plus every joint model's best combined distance from the table above, sorted together:

5. Comparing the Leading Candidates¶

Section 4b's numeric ranking put the independent, mixed-family pick (MLP for y, SVM for z) narrowly ahead of every joint model. The margins involved are small enough that this section compares the leading candidates visually and qualitatively -- not just by their headline metric -- before committing to (and saving) any one of them:

• MLP (independent, y) -- Section 4's winner for y, uncontested.
• SVM (independent, z) -- Section 4's numeric winner for z.
• MLP (independent, z) -- Section 4's close second for z (test_rmse_z within ~2% of SVM's).
• MLP (joint) -- Section 4b's leading joint candidate, predicting both axes from one model.

Residual structure after correction¶

r_y/r_z vs. field position, before vs. after each candidate's correction -- tests whether the post-correction residual still carries structure (deck analysis question 5), and is also the most direct way to check the "structured pattern" observed in the z-axis histogram/scatter above: a model whose corrected residual still trends with field position is systematically wrong in a way a flat, structureless one isn't, regardless of which has the lower aggregate RMSE.

Discussion: is the numeric winner actually the best choice?¶

The margins between the leading candidates are small enough that Section 4b's headline ranking shouldn't be taken as final on its own:

• z-axis, independent models: SVM's test RMSE (0.0614 mm) beats MLP's (0.0626 mm) by ~2% -- comfortably inside what re-running RandomizedSearchCV with a different seed or a larger sample could shift (only 6 candidates were sampled per model family here, not an exhaustive search).
• Independent-mixed vs. joint MLP: the true best independent pair (MLP y + SVM z, 0.0529 mm combined distance) beats the best joint model (MLP, 0.0560 mm) by ~6% -- small, and joint MLP is clearly ahead of every other joint alternative (SVM/GradientBoosting/RandomForest/Linear), so it isn't a weak joint candidate either.
• Residual structure: the z-axis histogram above does show a visibly spiky, non-smooth shape for the SVM-corrected residual, exactly as flagged -- but plotting MLP (independent) and MLP (joint) alongside it shows the same spikiness in all three, and the field-position scatter above shows no systematic trend distinguishing any of them either. That points to the spikiness being a property of the test set (only 34 distinct spectra/field positions on this axis, each contributing ~30 rows at nearly the same value, so any model's histogram at this bin width looks clumpy) rather than an SVM-specific failure -- worth checking explicitly rather than assuming, but it doesn't end up uniquely indicting SVM.
• Overfit ratio (ratio_z / dist_ratio) cuts the other way: SVM z's ratio (1.165) is actually lower (better) than either MLP variant's, independent (1.393) or joint (1.382) -- by that diagnostic SVM generalizes slightly better here, not worse.
• Forward-compatibility: independent of today's numbers, this notebook is a proof of concept -- if the planned direction for later stages is a more advanced, NN-based, joint model, picking MLP (independent or joint) now costs a small, plausibly noise-level accuracy difference in exchange for continuity with that direction, rather than adopting SVM (a non-NN, non-joint architecture with no obvious growth path toward that target) only to replace it again shortly.

So this isn't clear-cut either way: the numeric winner (independent, MLP y + SVM z) is still the best-measured option today, and the residual-structure concern turns out to apply about equally to all three z-axis candidates rather than singling out SVM -- but the margins are small, and the architectural-continuity argument is a legitimate, independent reason to prefer MLP anyway.

7. Generalization Test on rgs180_0¶

Section 5's decision is postponed -- so instead of testing one pick, this runs the generalization test for all three candidates under discussion, each refit on all of rgs000_0 (train+test) and applied to rgs180_0's own Stage 3 physical fit as the baseline (not rgs000_0's) -- same pattern 3-Intro_ML.ipynb validated for its RF-only version:

• MLP(y) + SVM(z) -- Section 4's measured best (independent, mixed families)
• MLP(y) + MLP(z) -- independent, single family
• MLP(y,z) -- joint, one model for both axes

Generalizing to a different physical grism instance is a genuinely new test: none of the three candidates were tuned against rgs180_0 in any way, so unlike Section 5's same-dataset train/test margins (which could partly reflect this particular 80/20 split), this result isn't subject to the same overfitting-to-the-split concern.

Does this help choose a model?¶

Some, but this is still too tight to be seriously decided from these numbers alone. On rgs000_0's own test set, SVM edged MLP for z by ~2%; on rgs180_0 that edge reverses by ~4-6% in MLP's favor. Both differences are on the same order as what re-running RandomizedSearchCV with a different seed (only 6 samples per family) could plausibly produce, and a single generalization dataset is one data point, not a validated trend -- this result nudges the comparison, it doesn't settle it.

Conclusion: model development continues with MLP (NN-based) rather than SVM going forward -- not because these numbers conclusively prove MLP superior (they don't, they're a coin flip either way), but because the planned model evolution for later phases (more advanced, NN-based models) makes an NN-based choice make more sense here regardless of a few-percent margin. SVM is dropped from further consideration on that basis, not on today's metrics.

The choice between the two remaining MLP variants -- MLP(y) + MLP(z) (independent) and MLP(y,z) (joint) -- stays open. Both are close on every metric in this notebook (test RMSE, combined distance, and now generalization), and the joint model is the more natural fit for the NN-based direction above. Rather than force a pick on noise-level margins, both are carried forward into Phase 2 and compared further there.

8. Analysis¶

1. Which model achieves the lowest test RMSE? Improvement over the physical model alone? See the results table (Section 4) and the physics-only-vs-best printout above -- read off the winning model per axis and its improvement percentage directly from that output rather than duplicating the numbers here (they'll shift slightly if grids/seeds change). Addendum after Sections 4b/5/7: the combined point-distance metric put the independent, mixed-family pair (MLP y + SVM z) numerically ahead of every joint model, and SVM narrowly ahead of MLP for z -- but these margins turned out to be within noise (Section 5), and the ranking even reversed on the rgs180_0 generalization test (Section 7). The eventual call -- continuing with MLP rather than SVM -- rests on the planned NN-based model evolution (Section 7's discussion), not on these numbers being decisive. Whether to keep y/z independent or move to the joint model is left open, to be resolved by further comparison in Phase 2.

2. Is the improvement in r_y and r_z symmetric? Compare physics_rmse["y"] vs. physics_rmse["z"] and the two best test_rmse values above. z (cross-dispersion / spatial axis) consistently shows a larger physics-only residual and a larger post-correction residual than y in this setup -- consistent with Stage 3's own finding that the z-axis fit is less tightly constrained (larger bootstrap std on offset_z_mm there than on offset_y_mm). The ML correction narrows the gap but the underlying asymmetry in how well-constrained each axis's physical fit is doesn't disappear.

3. Which model overfits most (largest train/test ratio)? Expected? Read the ratio column of the results table directly -- RandomForest and GradientBoosting show by far the largest ratios (train RMSE near zero, test RMSE much larger): expected, since neither has a regularization constraint active here (no min_samples_leaf/max_leaf_nodes cap, and max_depth up to 12 lets RF memorize individual training points on a dataset this size). Linear shows the smallest ratio (high-bias, low-variance, by construction). SVM and MLP land in between, closest to 1 -- both have explicit regularization terms (C/epsilon for SVM, alpha weight decay + early stopping for MLP) doing real work. The same pattern holds in Section 4b's joint models (dist_ratio), including for MultiOutputRegressor-wrapped GradientBoosting/SVM -- expected, since wrapping doesn't change each underlying single-target model's own tendency to overfit.

4. Does the best model generalize? Section 7 tests all three leading candidates (not just one) against rgs180_0's own correct physics baseline -- all three generalize (>10% improvement over physics-only), and the ranking among them shifts slightly relative to Section 4/5's same-dataset numbers (see Section 7's discussion). Treated as one data point informing, not deciding, the MLP-vs-SVM and independent-vs-joint questions above -- not as proof that any one candidate is "the" generalizing model.

5. Are post-correction residuals approximately random, or still structured? See Section 5's field-position plots (both axes) -- the z-axis histogram/scatter does show visible structure, but it turned out to be shared across all three z-axis candidates (SVM, independent MLP, joint MLP) rather than singling any one out, most likely a test-set granularity artifact (only 34 distinct z field positions in the test split) rather than a model-quality signal.

9. Recap¶

• Started from Stage 3's frozen rgs000_0 physical fit (models/stage3_fit_rgs000_0.toml) -- no refit here, per this phase's scope.
• Fixed parameters underlying that fit (prism.A_deg, grating.rho, detector.pixel_size) cross-checked against Euclid-NISP-Specs.md and found spec-grounded, not guessed; material.n0/k remain a documented synthetic placeholder (Stage 3 already found them non-identifiable here).
• Five ML model families (Linear, RandomForest, GradientBoosting, SVM, MLP) compared per axis (r_y, r_z) via grid/randomized hyperparameter search, every combination logged as its own MLflow run in the residual_correction experiment (sqlite:///mlflow.db -- browse with pixi run mlflow-ui); Section 4's results table adds the combined-axes metric that actually matters -- point distance between the corrected prediction and the data, mean(hypot(hybrid_r_y, hybrid_r_z)).
• Also tested the same five families as joint 2D-output models (Section 4b) and compared them against the independent per-axis pair on that same metric.
• Numeric margins between the leading candidates (SVM z vs. MLP z, independent vs. joint) turned out to be tight throughout -- same order as search noise on the same-dataset test set (Section 5), and the ranking even reversed on the rgs180_0 generalization test (Section 7). None of this notebook's own metrics settle the choice on their own.
• Conclusion: continue with MLP / NN-based models, not SVM, going forward -- decided on the planned model evolution (later phases moving toward more advanced, NN-based models), not on a clear numeric win here. Between the two remaining MLP variants -- independent (MLP(y) + MLP(z)) and joint (MLP(y,z)) -- no pick is made in this notebook; both are carried forward and compared further in Phase 2.
• Nothing saved to MLflow as a final "best" model in this phase -- the per-combination sweep (Sections 4/4b) is fully logged, but selecting and persisting a single model artifact is deferred until the independent-vs-joint choice is resolved.
• Stops here per this phase's scope -- no reusable per-dataset function, no run across the other 5 datasets, no "per_dataset" MLflow experiment. That's Phase 2 (4-Projet/index.html Section 2), a separate notebook, where the MLP(y)+MLP(z) vs. MLP(y,z) comparison also continues.