Stage 5, Phase 2 — Field-Dependent Parameters (5.2-Field_Dependent_Parameters)¶

Per CLAUDE.md's Stage 5 Phase 2 plan, motivated by Stage 4 Phase 5's root-cause finding: the reference paper (arXiv 2506.08378, Eq. 2-3) fits trace shift as a 2D-Chebyshev function of field position (y0, z0), while this project's offset_y_mm/offset_z_mm are per-dataset constants — so Stage 4's MLP residual has been standing in, data-inefficiently, for a missing spatial term.

Two steps so far:

• Step 1 (Sections 1-4), done: diagnostic only — reload Stage 4 Phase 3's frozen per-dataset fits, check whether their residuals show real structure across field position on this project's own data (not just by analogy to the reference paper). No new model fit.
• Step 2 (Sections 5-8), this notebook's addition: NN architecture comparison for the field-dependent term identified in Step 1 — input encoding, output structure, and shared-vs-per-dataset network granularity, compared systematically per the model-comparison-report skill and in the spirit of Stage 4 Phase 1 (every candidate logged, not just the winner). Still no promotion to dispcraft/ and no refit of the actual per-dataset tier — this settles the architecture question the next step (training the real field-dependent tier into the calibration) will build on.

1. Reload Stage 4 Phase 3's Frozen Per-Dataset Fits¶

Each models/joint_specific_fit_<cfg>.toml already records everything needed to reproduce that dataset's frozen prediction exactly: fit.fixed (Tier 1+2 values: coll_f, cam_f, A_deg, rho, material_n0, material_k) and fit.result (Tier 3: tilt_deg, offset_y_mm, offset_z_mm, plus bootstrap-std entries to drop). Merged, these give the full parameter dict for dispcraft.calibration.predict_centroids with free_names=[] — no refitting, just replaying the frozen model.

2. Residuals vs. Field Position¶

One row per dataset, r_y and r_z scatter over the (y_nisp, z_nisp) field plane, colored by residual value. A per-dataset constant offset_y_mm/ offset_z_mm predicts flat, unstructured (noise-only) color across the field if the physical model's assumption holds; a smooth gradient or trend across (y0, z0) is exactly what a field-dependent offset term would be needed to absorb.

3. Quantifying the Structure¶

Visual gradients can be misleading, so fit the simplest possible field-dependent term — a plain linear model r_y ~ y0 + z0 and r_z ~ y0 + z0 — per dataset, per axis, and report the variance it explains (R²) plus how much it reduces RMSE relative to the frozen constant-offset residual. This is a lower bound: any NN considered in the next step should beat a plain linear field term, not just beat the constant-offset baseline. A near-zero R² here would mean the residual is dominated by noise, not smooth field structure, and the field- dependent-parameter direction would need to be reconsidered for that dataset/axis.

4. Findings¶

The motivation holds — this is real field structure, not noise. A plain linear r ~ y0 + z0 term explains 29-60% of residual variance (R²) and cuts RMS by 15-37%, consistently across all 6 datasets, both axes. That's a lower bound: the scatter plots in Section 2 show more than a linear gradient —

• r_y is close to a clean left-right gradient (function of y0 only) in every dataset, sign and shape consistent across all 6 — looks like a shared, close-to-linear effect.
• r_z is mostly a top-bottom gradient (function of z0), but with a noticeably stronger, non-linear excursion concentrated in one field corner — top-left for rgs000_*, bottom-left for rgs180_*. That corner-locked asymmetry is the kind of thing a plain low-order polynomial underfits and a coordinate NN (or a higher-order Chebyshev term, per the reference paper) is suited to — worth keeping in mind when picking the Section 2/next-step architecture, and worth checking whether it lines up with the tilt+180° grism-mounting geometry (same asymmetry Stage 4 Phase 2 traced offset_z_mm's sign split to).

This confirms Stage 4 Phase 5's hypothesis with this project's own data, not just by analogy to the reference paper, and gives a concrete target shape (mostly-linear y0 term for r_y; z0 term plus a corner feature for r_z) to sanity-check candidate NN architectures against in the next step.

5. Step 2 Setup — One Point per Field Position¶

Section 2's residuals are one row per (spectrum, wavelength) — e.g. rgs000_0 has 5328 rows but only 170 distinct spectra_id field positions. offset_y_mm(y0,z0)/offset_z_mm(y0,z0) is a geometric term, not chromatic, so before fitting anything it's worth checking whether the residual actually varies within a spectrum (across wavelength) or only between spectra (across field position) — if it's the latter, the ~30 wavelength rows per spectrum are correlated repeats of one field-position measurement, not independent samples, and training on raw rows would silently overweight whichever field positions happen to have more wavelength samples while doing nothing for the architecture question itself.

within_spectrum_std = mean of each spectrum's own row-to-row std (wavelength variation); between_spectrum_std = std of each spectrum's mean residual (field-position variation).

6. Architecture Grid — Input Encoding × Output Structure (rgs000_0)¶

With ~150-200 independent field-position samples per dataset (Section 5), a single train/test split would leave a noisy ~30-sample test set — too small to compare architectures reliably. Using KFold (5-fold, no grouping needed: each row is already one field position, so there's no leakage to guard against per the hybrid-residual-modeling skill's grouping rule) instead of Stage 4 Phase 1's single GroupShuffleSplit, on rgs000_0 as the representative single dataset (same scope choice Phase 1 made).

Reusing dispcraft.ml's ResidualMLP/LitResidualRegressor machinery from Stage 5 Phase 1 unchanged — it already supports arbitrary input/output dimensions, so the only new code is the input encoding.

Two encoding × output-structure axes, per CLAUDE.md's open question:

• n_bands: 0 = plain standardized (y0,z0); 4 = that plus 4 bands of sin/cos(k*pi*u) Fourier features on each coordinate (u = y0 or z0, standardized by FIELD_SCALE_MM into ~[-1,1]) — a positional encoding for a smooth low-frequency field, per the phase plan.
• output_mode: independent (one single-output MLP per axis, Stage 4 Phase 2's simplicity-grounds default) vs joint (one two-output MLP).

Plus a small hidden_layer_sizes/alpha grid, weighted toward more regularization than Stage 4 used, given the sparse per-dataset field sampling this phase's plan explicitly flags as an overfitting risk. LinearRegression (Section 3) and the raw pre-model residual (n_bands-free, "zero field model", i.e. Stage 4's constant-offset assumption) are carried along as baselines — the hybrid-residual-modeling skill's rule to always compare against the simpler alternatives, not just rank NN candidates against each other.

7. Sharing Granularity — Per-Dataset vs. Per-Instance Pooled Network¶

The remaining open axis from CLAUDE.md's plan: should the field-dependent network be fit per-dataset (matching the current per-dataset offset_y_mm/offset_z_mm scalars it replaces) or shared per grism instance (pooling {rgs000_0, rgs000_m4, rgs000_p4} / {rgs180_0, rgs180_m4, rgs180_p4}), the same shared/specific question Stage 4 Phase 3 asked of rho/A_deg? The three same-instance configs turn out to sample almost the same field grid (138/143 exact-position matches out of 144-169 per config) — each is a different GWA tilt at (nearly) the same physical field points, so pooling gives 3 noisy repeats per position rather than denser field coverage. Whether that's still worth it depends on whether the field-dependent term is truly instrument-level (shared distortion from the collimator/camera optics, constant across tilt) or secretly tilt-specific (in which case pooling averages away real signal).

Tested directly: for each instrument, take the position-level intersection across its 3 configs, 5-fold KFold over positions (not rows), and for each fold compare a per-dataset network (trained on that config's own training-fold positions only) against a pooled network (trained on all 3 configs' training-fold positions/residuals together) — both evaluated on each config's own held-out positions. Uses the winning (n_bands, hidden_layer_sizes, alpha, output_mode) combination from Section 6.

8. Findings¶

Aggregation (Section 5) is justified. Within-spectrum (wavelength) std is 0.006-0.019 mm; between-spectrum (field-position) std is 0.19-0.36 mm — a 20-30x gap, confirming r_y/r_z are overwhelmingly a function of field position, not wavelength, and that the ~144-201 field positions per dataset (not the ~4500-6200 rows) are the real sample size this phase is working with.

Architecture grid (Section 6). Every NN candidate demolishes both baselines: best NN CV-test RMSE 0.052 mm vs. baseline_linear 0.204 mm vs. baseline_zero (Stage 4's constant-offset assumption) 0.247 mm — a ~75% reduction beyond the linear floor Step 1 already established, quantitatively confirming Step 1's read that a non-linear (corner) feature is there for an NN to earn its complexity on.

• Output structure decided clearly, unlike Stage 4: all 4 top-ranked candidates are joint (2-output) models; the best independent candidate (rank 7, 0.070 mm) is beaten by the worst joint one in the top 6. Stage 4 Phases 1-3 found independent-vs-joint a near-wash for the wavelength-based MLP residual; here it isn't — plausible explanation: r_y and r_z share the same 2D input and the corner feature likely has one shared physical cause (Step 1 flagged the same tilt+180° geometry offset_z_mm's sign split traces to), so a joint model can exploit that cross-axis correlation where two independent single-output models can't.
• Fourier encoding didn't help: the best n_bands=4 candidate (0.064 mm) is beaten by the best n_bands=0 (plain standardized coordinates, 0.052 mm) at comparable hyperparameters. With only a 2D input, an MLP's own hidden-layer non-linearity already reproduces the corner feature; adding explicit high-frequency features just adds capacity this ~136-point training set doesn't need.
• Regularization: the winning combination uses alpha=1e-3 (light), not 1e-1 — its train/test RMSE ratio (0.043/0.052 ≈ 1.2) shows only mild overfitting in practice. The phase plan's concern about the sparse per-dataset sampling was reasonable caution, but for a plain 2D-input MLP of this size it didn't bite as hard as expected — worth noting honestly rather than forcing heavier regularization the data didn't ask for.
• Winning architecture: standardized (y0,z0) input (no Fourier bands), joint 2-output MLP, hidden_layer_sizes=(32,16), alpha=1e-3.

Sharing granularity (Section 7): per-dataset wins, decisively. Per-dataset beats pooled-instance in 5 of 6 configs — decisively for rgs180_* (e.g. rgs180_0: 0.185 mm vs. 0.269 mm pooled; rgs180_m4: 0.060 vs. 0.093 mm) and narrowly for rgs000_0/rgs000_m4; only rgs000_p4 marginally favors pooling (0.068 vs. 0.067 mm). Since the 3 same-instance configs sample almost the same field grid (138/143 shared positions), pooling here only tests "does averaging 3 tilt measurements at the same point help," and the answer is no: the field-dependent term is tilt (dataset)-specific, not a shared instrument-level distortion — it should stay a Tier-3 (per-dataset) parameter, like the offset_y_mm/offset_z_mm scalars it replaces, not promoted to Tier 2 (per-grism-instance) alongside rho/A_deg. This directly answers the open question in CLAUDE.md's phase plan.

Caveat: this test doesn't rule out that a genuinely denser field grid (e.g. exploiting the 180° mounting relationship between rgs000/rgs180) could still help — out of scope here, since the available same-instance configs don't add spatial coverage, only repeated measurements.

Caveat: rgs180_0 is markedly harder to fit than the other 5 configs (0.18 mm vs. 0.05-0.09 mm test RMSE for either model) — consistent with it having the largest rms_before/rms_after_linear in Step 1's own Section 3 table, so likely a genuinely harder dataset rather than an artifact of this comparison; noted, not over-interpreted.

Recommendation for the next step (training the actual field-dependent tier into the calibration, replacing the offset_y_mm/offset_z_mm scalars): a per-dataset joint 2-output MLP on standardized (y0,z0), hidden_layer_sizes=(32,16), alpha≈1e-3 — no Fourier encoding, no per-instance sharing. Still needs the identifiability check CLAUDE.md flags before refitting (the current scalar offsets are this network's zero-order term and must be replaced, not added to).

9. Step 3 — Training the Field-Dependent Tier into the Calibration¶

Per CLAUDE.md's plan and Step 2's result: offset_y_mm/offset_z_mm replace, not add to, the winning architecture (per-dataset joint MLP, standardized (y0,z0), hidden_layer_sizes=(32,16), alpha=1e-3) — fixed at 0.0 in the physical model, with the NN supplying the entire position-dependent detector-registration correction (mean + spatial shape) that the two scalars used to approximate as a single constant. tilt_deg stays a genuine free physical scalar.

Why "replace, not add" matters here: if the scalar offsets stayed free and the NN kept an unconstrained output bias, the two would be perfectly degenerate — any constant could be split between them arbitrarily with no change in prediction. Fixing the offsets at exactly 0.0 removes that degree of freedom outright, so the NN's own bias is the only thing that can supply the constant term (this is different from, and safer than, Sections 5-7's approach, which trained the NN on the residual of an already-fit frozen offset — sequential, not simultaneous, so no such ambiguity was possible there).

A second, subtler risk: tilt_deg (a rotation) and the field-dependent term's near-linear-in-y0 component (Step 1, Section 4) could partially trade off against each other if fit simultaneously from a cold start. Mirrors Stage 4 Phase 3's alternating tiered fit to avoid this: warm-start tilt_deg at its frozen joint_specific_fit_<cfg>.toml value, then alternate

1. train the NN on the residual of the current tilt_deg (offset fixed at 0.0), aggregated to one point per field position as in Section 5;
2. re-fit tilt_deg (Nelder-Mead, dispcraft.calibration's pattern) with that NN's correction now added to the physical prediction,

for a few passes, tracking tilt_deg's trajectory — if it settles near its frozen value rather than drifting, that's evidence against the NN quietly absorbing rotation. coll_f/cam_f/A_deg/rho/material_n0/material_k stay at their frozen Tier 1+2 values throughout, unchanged from joint_specific_fit_<cfg>.toml's fit.fixed.

Promoted: the alternating-fit machinery below (fit_field_dependent_tier/ predict_centroids_field/standardize_field/aggregate_by_position) now lives in dispcraft/field_calibration.py, tests in tests/test_field_calibration.py (including an end-to-end synthetic-recovery test that would fail loudly if the sign bug below were reintroduced). This notebook cell is now a thin driver over that module, per this project's notebook-to-library workflow; the identifiability-check interpretation (comparing tilt_deg/nn_mean against the frozen fit) stays notebook-local -- it's specific to reviewing this result, not reusable machinery.

10. Residuals After the Field-Dependent Fit¶

Same layout as Section 2, now for the field-dependent fit's residual (data - physical_pred(tilt_deg_final, offset=0) - NN(y0,z0)). If the field term genuinely absorbed the structure Step 1 found, this should look like unstructured noise at a much smaller color scale than Section 2 — not just a lower aggregate RMSE.

11. Findings¶

The bug worth flagging: the first run of this section trained the NN on the residual r = pred(offset=0) - data directly and added that same-signed prediction back into the cost -- doubling the error instead of cancelling it (tilt_deg then ran away by 6-8° trying to compensate, and RMSE got worse than the frozen fit, 1.2-2.5 mm vs. 0.35-0.41 mm). The NN needs to predict the correction (-r, i.e. what the old scalar offset supplied), not the residual itself. Fixed by negating the aggregated target before training; caught by comparing to the frozen baseline before trusting the result, not by the code raising an error -- a plain regression on the wrong sign trains and converges just as happily as the right one.

With the fix, the field-dependent tier is a clear, decisive win. RMSE dropped from 0.34-0.41 mm (frozen constant-offset Tier 3) to 0.06-0.10 mm — a 76-83% reduction across all 6 datasets, matching the magnitude Section 6's architecture-only CV predicted (0.05-0.09 mm range) now that it's actually wired into the calibration rather than just cross-validated on a static residual.

Both identifiability checks pass:

• tilt_deg didn't drift. Final vs. frozen delta is 0.002-0.033° across all 6 configs, in every case smaller than that dataset's own tilt_deg_bootstrap_std (0.060-0.071°) — the rotation and the field-dependent term's near-linear component are not trading off against each other; tilt_deg settles back to (within-uncertainty) its previously-established physical value once the NN supplies the spatial correction the alternation's first pass needs. The alternation itself converges smoothly (e.g. rgs000_0's per-iteration tilt_deg delta: 0.0049° → 0.0007°, monotonic and shrinking, not oscillating) though it doesn't quite reach the 1e-4° tolerance within N_ITER=5 — the residual per-iteration jitter is attributable to each pass retraining the NN from scratch (stochastic init), and is 1-2 orders of magnitude below the parameter's own bootstrap uncertainty, so not physically meaningful.
• The NN's zero-order term recovered the old scalar offsets almost exactly: e.g. rgs000_0's nn_mean_y=-0.551 vs. frozen offset_y_mm=-0.549; nn_mean_z=0.7355 vs. frozen offset_z_mm=0.7357 — matching to 3 significant figures across all 6 datasets, both axes. Confirms the "replace, not add" design is doing exactly the job intended: the NN's bias term picked up the same constant the old scalar did, on top of which it now also supplies the spatial shape Step 1 found and Section 6 validated the architecture for.

Residual structure is genuinely gone, not just smaller in RMS: refitting a plain linear r ~ y0 + z0 to the final per-row residual (Section 10) gives R²=0.001-0.004 on every dataset/axis — down from 29-60% before the field-dependent term (Step 1, Section 4). The field term isn't just shrinking the residual, it's absorbing the specific structure it was built to absorb.

Caveat: per-row residuals still have a handful of outliers well above the aggregate RMSE (max |r_z| up to 0.49 mm for rgs000_p4, 0.38 mm for rgs180_0, vs. their 0.06-0.10 mm RMSE) — consistent with Step 1 flagging rgs180_0 in particular as a harder dataset, and worth a closer look (e.g. specific field corners or spectra) before this tier is treated as final, but not investigated further here.

Status: still entirely notebook-local, per this project's promote-on- confirmation workflow — no changes to dispcraft/calibration.py or new models/*.toml. tilt_deg here is a per-dataset scipy point estimate, not yet bootstrapped for uncertainty the way Stage 4's frozen fits were; the NN itself isn't persisted (matching Stage 4/5 Phase 1's convention of not checkpointing residual-correction models to disk). Promoting this tier (TOML recording, bootstrap uncertainty, and updating dispcraft/calibration.py's predict_centroids to accept a field-dependent correction term) is future work, pending review of this result.

Promoted: fit_field_dependent_tier/predict_centroids_field/standardize_field/aggregate_by_position now live in dispcraft/field_calibration.py (tests in tests/test_field_calibration.py, including an end-to-end synthetic-recovery test guarding against the sign bug above). dispcraft/calibration.py itself is unchanged -- the new module composes its predict_centroids with dispcraft.ml's residual-MLP machinery rather than extending either. Bootstrap uncertainty for tilt_deg and TOML recording of the final per-dataset fits are still open (this promotion covers the fitting/prediction logic, not a frozen, reviewed result set).

12. Step 4 — Re-examining Held-Out Generalization¶

notebooks/6-Status_Report.ipynb (Sections 4-5) refit this same field-dependent tier on a train split only and evaluated on a held-out test split (GroupShuffleSplit, grouped by spectra_id, random_state=42 — the convention used everywhere else in this project) and found a very different story than Step 3's in-sample number: the tier barely beats, and is largely redundant with, Stage 4's existing generic (y,z,wavelength) MLP residual (held-out combined RMSE: field tier alone 0.090mm vs. ML alone 0.077mm, mean over 6 datasets; layering both gives only 0.077→0.077mm). Status_Report .md §9's top recommendation asks to re-examine this before recommending the field tier for production, via two concrete levers — a smaller/more- regularized architecture, and more field positions (pooling datasets that share a grism instance, which Section 7 above tested but only under in-sample position-level CV, not this held-out framing) — plus an open question of whether Section 6's own CV estimate was itself optimistic.

Four sub-steps, continuing this notebook (Stage 5 Phase 2 is still open in CLAUDE.md, not a new stage):

• 4.0: quantify the gap between Section 6's in-sample CV estimate, Section 9's full-dataset in-sample fit, and a genuinely fair train/test held-out fit, for the current winning architecture.
• 4.1: search smaller/more-regularized architectures under the fair held-out protocol (select via train-only CV, evaluate once on test).
• 4.2: re-test per-instance pooling under the fair protocol (Section 7 only tested in-sample position-level CV).
• 4.3: consolidate into one comparison table and state the conclusion.

No dispcraft/ changes in this step — fit_field_dependent_tier already accepts mlp_params, so every architecture variant below reuses it unchanged; pooling reuses the same primitives (aggregate_by_position, standardize_field, dispcraft.ml.train_residual_mlp/predict) it's built from.

4.0 Fair Held-Out Split, and a Diagnostic on Section 6's CV¶

held_out_split mirrors 6-Status_Report.ipynb's test_split exactly (GroupShuffleSplit, test_size=0.2, grouped by spectra_id, random_state=42) so results here are directly comparable to that notebook.

For each config: fit the current winning tier (hidden_layer_sizes=(32,16), alpha=1e-3, from Section 8) on the train split only via fc.fit_field_dependent_tier, and evaluate on the held-out test split — the same "fair" protocol Section 6 of 6-Status_Report.ipynb used. Comparing this against two numbers already sitting in this notebook's own namespace:

• Section 9's tier3_summary — full-dataset (in-sample) fit, all 6 configs.
• Section 6's arch_results — 5-fold position-level CV on rgs000_0 only, computed from residuals of a physical fit that already used all the data (including each CV fold's own test positions) to fit tilt_deg/the old scalar offsets before the NN ever saw a residual. That's the concrete, checkable version of "was the CV estimate itself optimistic": the physical fit's own least-squares optimum already incorporates each test fold's positions, so the residual those folds see is smaller than it would be under a genuinely train-only physical fit — before the NN contributes anything.

Sanity check: this cell's per-dataset rmse_y_mm/rmse_z_mm mean to 0.036/0.083mm (combined 0.090mm) across the 6 configs — matching 6-Status_Report.ipynb's reported held-out numbers to 3 decimal places, confirming this harness reproduces that notebook's result rather than a subtly different protocol.

Section 6's CV was indeed optimistic — confirmed, not just suspected. For rgs000_0, Section 6's in-sample CV-test RMSE was 0.052mm; the fair held-out RMSE for the identical architecture is 0.079mm — 52% higher. The mechanism is exactly the one flagged above: Section 6's residual target came from a physical fit (tilt_deg, old scalar offsets) that had already seen every CV fold's positions when it was fit, so the "test fold" residual was never a genuine out-of-sample number even before the NN was involved. This means part (not all) of §7.1's in-sample-vs-held-out gap is an artifact of comparing two different things (CV-on-leaked-residual vs. genuinely-held-out), not purely the architecture generalizing badly.

But a real generalization gap remains even accounting for that. Fair held-out RMSE (mean 0.090mm) is still 1.2-1.6x worse than the full-dataset in-sample fit (mean 0.073mm) for 5 of 6 configs — rgs180_0 is the outlier, slightly better held-out (0.088 vs. 0.098mm in-sample), plausibly noise given it's already flagged (Section 8) as the hardest, highest-variance dataset. So there are two distinct effects stacked in §7.1's original finding: (1) Section 6's CV being optimistic (now quantified above), and (2) genuine architecture overfitting to the sparse per-dataset field sampling once evaluated fairly. Step 4.1 targets effect (2) directly.

4.1 Smaller/More-Regularized Architecture, Selected Fairly¶

Grid: hidden_layer_sizes ∈ {(8,), (16,), (16,8)} × alpha ∈ {1e-2, 3e-2, 1e-1, 3e-1} — smaller and more regularized than the current (32,16)/alpha=1e-3 — plus the current default itself, plus three iso-capacity variants ((32,16) at alpha ∈ {1e-2, 3e-2, 1e-1}) that keep the same hidden-layer capacity and vary only regularization strength, to separate "smaller network" from "more regularized" (the grid above confounds the two). Selected via a train-only 3-fold KFold (position-level, same style as Section 6) on rgs000_0's train split alone, at a fixed tilt_deg (no alternation, matching Section 6's own convention — the alternation is a secondary effect, confirmed in Section 11), so the selection never touches the test split. The winner is then evaluated the fair way (fc.fit_field_dependent_tier, alternating tilt_deg, trained on the full train split, evaluated on the true held-out test split) across all 6 configs, reusing field_tier_heldout_rmse/train_test from §4.0.

No smaller/more-regularized architecture beats the current default — if anything, shrinking capacity makes held-out performance measurably worse.

• The inner CV does favor smaller networks, but the effect is capacity, not regularization: among the iso-capacity (32,16) variants, held-out proxy RMSE gets monotonically worse as alpha increases (0.129 → 0.131 → 0.133 → 0.137mm for alpha 1e-3 → 1e-2 → 3e-2 → 1e-1) — more regularization at fixed capacity never helps on this inner-CV surface. Only shrinking hidden_layer_sizes to (8,) improves the inner-CV number (down to 0.088-0.094mm), and the winner is (8,), alpha=0.03.
• That selection does not transfer. Evaluated the fair way (alternating tilt_deg fit, full train split, genuine held-out test) across all 6 configs, the selected (8,)/alpha=0.03 architecture is decisively worse than the current (32,16)/alpha=1e-3 default: mean combined RMSE 0.162mm vs. 0.090mm — roughly 80% worse, not better. The damage is concentrated in the two ±4°-tilt configs used for architecture selection's blind spot: rgs000_m4 (0.257 vs. 0.073mm, +252%) and rgs180_m4 (0.251 vs. 0.109mm, +131%); the untilted rgs000_0/rgs180_0 and the other tilt configs are hurt too, but less severely (9-61% worse).
• Plausible cause: architecture selection here (per this project's own established convention, Section 6/Stage 4 Phase 1) used rgs000_0 — an untilted config — as the single representative dataset. Section 1/8 already found the field-dependent term's non-linear "corner" feature is present in every config but the tilt configs show it more strongly. An 8-unit single-hidden-layer network may simply lack the capacity to represent that corner feature once evaluated on the configs where it matters most, even though it looked like a clean win on the (relatively easier) dataset used to pick it. tilt_deg itself doesn't run away for either config (checked directly: final vs. initial delta is 0.001-0.098°, within or barely outside each config's own bootstrap uncertainty of 0.06-0.07° — not the sign-bug-style blow-up Section 11 guards against) — the damage is from the NN itself underfitting, not a physical-parameter trade-off.

Conclusion: this rules out "architecture too large / under-regularized" as the explanation for §7.1's held-out gap — the opposite lever (more capacity or the current default) is what held-out data actually rewards. The current (32,16)/alpha=1e-3 default stays the best architecture found so far under genuine held-out evaluation.

Caveat: only one global architecture (selected from rgs000_0, this project's standing convention for representative-dataset selection) was tested per candidate — a per-config-tuned architecture, allowing the harder tilt configs more capacity specifically, wasn't tried and might do better; flagged as a possible follow-up, out of scope here.

4.2 Per-Instance Pooling, Evaluated Fairly¶

Section 7 tested pooling ({rgs000_0,rgs000_m4,rgs000_p4} / {rgs180_0,rgs180_m4,rgs180_p4}) but only under in-sample position-level CV — every position it trained and evaluated on came from the full, already-fit dataset. Retested here under the fair held-out protocol: for each grism instance, fix tilt_deg per config at its §4.0 fair-fit value (the current (32,16)/alpha=1e-3 architecture — confirmed in §4.1 as the best found), compute each config's train-split correction target at that fixed tilt_deg (no alternation — isolates the sharing-granularity question alone, same choice Section 7 made), then compare a per-dataset NN (trained on one config's own train positions) against a pooled NN (trained on all 3 configs' train positions concatenated) — both evaluated on each config's own held-out test split (row-level, not the position intersection trick Section 7 used, since here every config keeps its own distinct train/test spectra).

Pooling is decisively worse under the fair protocol — more so than Section 7's in-sample test suggested.

Per-dataset vs. pooled-instance combined held-out RMSE: only rgs180_0 marginally favors pooling (0.088 vs. 0.088mm, +0.6%); every other config is severely worse pooled — rgs000_m4 0.074→0.455mm (-518%), rgs000_p4 0.097→0.469mm (-382%), rgs180_m4 0.109→0.442mm (-306%), rgs180_p4 0.097→0.495mm (-410%), rgs000_0 0.087→0.094mm (-9%). This is a much larger effect than Section 7's in-sample position-level CV found (there, pooling was only 30-50% worse).

Confirmed as a genuine underfit, not a held-out artifact: checked the pooled network's own training RMSE (not held-out) directly — 0.248mm combined vs. 0.038-0.054mm per-dataset — the pooled network can't even fit the training data of all 3 configs at once. The reason is visible in the raw targets: at (nearly) the same field positions, the correction needed shifts by tilt — e.g. rgs000 instance's r_z train range is 0.13-1.59mm (_0) vs. 0.58-2.04mm (_m4) vs. -0.31-1.16mm (_p4), a baseline shift of ~0.4-0.9mm between tilts, an order of magnitude larger than the field-dependent shape itself. (y0,z0) alone doesn't encode which tilt a position's measurement came from, so a single pooled function is being asked to reproduce 3 substantially different surfaces from the same input — not merely "3 noisy repeats of one surface" (Section 7's framing). This is a stronger, more direct version of the same conclusion Section 8 already reached ("field-dependent term is tilt-specific, not shared") — Section 7's exact-position-averaging test softened the effect (it trains on one shared position array repeated 3 times, nudging the fit toward each position's cross-tilt mean rather than genuinely conflating 3 offset-shifted surfaces); this test doesn't have that cushion, since each config's own train-split positions and targets are pooled directly.

Conclusion: pooling does not help under genuine held-out evaluation either — it is decisively, and for most configs catastrophically, worse than per-dataset. §9's second lever is also ruled out.

4.3 Consolidated Comparison and Conclusion¶

Extends 6-Status_Report.ipynb's progression-table style with this notebook's own held-out split (train_test, same protocol, computed fresh here for self-containment): physical (joint fit alone), hybrid (joint + the existing generic wavelength-based ML residual, Stage 4's default, no field tier), field tier alone (current architecture), field tier + ML layered on top, plus this section's two negative results (regularized architecture, pooled instance) for direct comparison against the same baseline.