dispcraft.field_calibration

field_calibration

Field-dependent Tier-3 fit: physics (tilt_deg) + ML (spatial offset).

Stage 4 Phase 5 found the physical model's per-dataset offset_y_mm/ offset_z_mm -- constants -- can't represent the reference paper's position-dependent trace shift; Stage 5 Phase 2 confirmed real field structure in this project's own residuals (notebooks/5.2- Field_Dependent_Parameters.ipynb, Step 1), picked an architecture for it (Step 2: a per-dataset joint 2-output MLP on standardized (y0,z0), DEFAULT_FIELD_MLP_PARAMS), and validated fitting it into the calibration (Step 3, this module).

offset_y_mm/offset_z_mm are replaced, not added to: fixed at 0.0, with the NN supplying the entire position-dependent correction (its bias term recovers the old scalars' role; Step 3 confirmed this to 3 significant figures). Keeping both free at once would be perfectly degenerate -- any constant could be split between the scalar and the NN's bias with no change in prediction. fit_field_dependent_tier enforces this by rejecting offset_y_mm/offset_z_mm/tilt_deg in fixed_tier12.

tilt_deg is re-fit alternately against the NN (mirrors notebooks/4.3-Multi_Dataset_Fitting.ipynb's alternating tiered pattern): train the NN on the residual of the current tilt_deg, then re-optimize tilt_deg (Nelder-Mead) with that NN's correction included, warm-started at a prior tilt_deg estimate to avoid a cold-start trade-off between rotation and the field term's near-linear-in-y0 component. Step 3 confirmed this doesn't happen in practice: tilt_deg moves by less than its own bootstrap uncertainty once the NN is in the loop.

The NN's training target is the correction needed to cancel the physical residual (-r, i.e. data - pred(offset=0)), not the residual itself (r = pred(offset=0) - data, this project's residual sign convention elsewhere, e.g. dispcraft.calibration.cost) -- get this backwards and the correction doubles the error instead of removing it. This is a real bug Step 3 caught (tilt_deg ran away several degrees, RMSE got worse than the frozen constant-offset fit) before comparing against the frozen baseline; tests/test_field_calibration.py guards against reintroducing it via an end-to-end synthetic-recovery check.

aggregate_by_position 

aggregate_by_position(df, value_cols=('r_y', 'r_z'))

Collapse repeated per-wavelength rows to one row per field position (spectra_id), averaging value_cols (and y_nisp/z_nisp).

The field-dependent term is a geometric, not chromatic, effect: Stage 5 Phase 2 Step 2 found within-spectrum (wavelength) std 20-30x smaller than between-spectrum (field-position) std, i.e. the ~30 wavelength rows per spectrum are correlated repeats of one field-position measurement, not independent samples. Fitting the NN on raw rows would silently overweight field positions with more wavelength samples.

Source code in dispcraft/field_calibration.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
def aggregate_by_position(df, value_cols=("r_y", "r_z")):
    """Collapse repeated per-wavelength rows to one row per field position
    (`spectra_id`), averaging `value_cols` (and `y_nisp`/`z_nisp`).

    The field-dependent term is a geometric, not chromatic, effect: Stage 5
    Phase 2 Step 2 found within-spectrum (wavelength) std 20-30x smaller
    than between-spectrum (field-position) std, i.e. the ~30 wavelength rows
    per spectrum are correlated repeats of one field-position measurement,
    not independent samples. Fitting the NN on raw rows would silently
    overweight field positions with more wavelength samples.
    """
    agg = {"y_nisp": "mean", "z_nisp": "mean", **{c: "mean" for c in value_cols}}
    return df.groupby("spectra_id").agg(agg).reset_index()

fit_field_dependent_tier 

fit_field_dependent_tier(df, model, fixed_tier12, tilt_deg_init, mlp_params=None, n_iter=5, tol_deg=0.0001)

Alternating fit of tilt_deg (physical, Nelder-Mead) and a field-dependent NN correction replacing offset_y_mm/offset_z_mm.

Parameters:
  • df (DataFrame with `y_nisp`, `z_nisp`, `wavelength`, `cent_y`, `cent_z`,) –

    spectra_id columns (e.g. dispcraft.measurement.median_per_spectrum's output, outliers already excluded)

  • model (GroundTestModel) –
  • fixed_tier12 (dict -- Tier 1+2 physical params (`coll_f`, `cam_f`,) –

    A_deg, rho, material_n0, material_k); must not include offset_y_mm/offset_z_mm/tilt_deg (see module docstring)

  • tilt_deg_init (float -- warm start, e.g. a prior frozen Tier-3 fit's value) –
  • mlp_params (dict, default: `DEFAULT_FIELD_MLP_PARAMS` ) –
  • n_iter (int -- max alternation passes, default: 5 ) –
  • tol_deg (float -- stop early once `tilt_deg` moves less than this between passes, default: 0.0001 ) –
Returns:
  • tilt_deg( float ) –
  • nn_model( dispcraft.ml.LitResidualRegressor -- predicts the field ) –

    correction from standardize_field(y0, z0)

  • history( list of dict, one per alternation pass (`iter`, `tilt_deg`, `delta_deg`, `cost_mm2`) ) –
Source code in dispcraft/field_calibration.py 
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
def fit_field_dependent_tier(df, model, fixed_tier12, tilt_deg_init, mlp_params=None, n_iter=5, tol_deg=1e-4):
    """Alternating fit of `tilt_deg` (physical, Nelder-Mead) and a
    field-dependent NN correction replacing `offset_y_mm`/`offset_z_mm`.

    Parameters
    ----------
    df : DataFrame with `y_nisp`, `z_nisp`, `wavelength`, `cent_y`, `cent_z`,
        `spectra_id` columns (e.g. `dispcraft.measurement.median_per_spectrum`'s
        output, outliers already excluded)
    model : dispcraft.calibration.GroundTestModel
    fixed_tier12 : dict -- Tier 1+2 physical params (`coll_f`, `cam_f`,
        `A_deg`, `rho`, `material_n0`, `material_k`); must not include
        `offset_y_mm`/`offset_z_mm`/`tilt_deg` (see module docstring)
    tilt_deg_init : float -- warm start, e.g. a prior frozen Tier-3 fit's value
    mlp_params : dict, default `DEFAULT_FIELD_MLP_PARAMS`
    n_iter : int -- max alternation passes
    tol_deg : float -- stop early once `tilt_deg` moves less than this between passes

    Returns
    -------
    tilt_deg : float
    nn_model : dispcraft.ml.LitResidualRegressor -- predicts the field
        correction from `standardize_field(y0, z0)`
    history : list of dict, one per alternation pass (`iter`, `tilt_deg`, `delta_deg`, `cost_mm2`)
    """
    _check_fixed_tier12(fixed_tier12)
    mlp_params = mlp_params or DEFAULT_FIELD_MLP_PARAMS
    fixed0 = {**fixed_tier12, "offset_y_mm": 0.0, "offset_z_mm": 0.0}

    tilt_deg = tilt_deg_init
    nn_model = None
    history = []
    for it in range(n_iter):
        r_y, r_z = _row_residual(df, tilt_deg, model, fixed_tier12)
        # NN target is the *correction* needed to cancel the residual (-r),
        # not the residual itself -- see module docstring.
        d = df.assign(r_y=-r_y, r_z=-r_z)
        field = aggregate_by_position(d)
        X_field = standardize_field(field["y_nisp"].values, field["z_nisp"].values)
        y_field = field[["r_y", "r_z"]].values
        nn_model = train_residual_mlp(X_field, y_field, mlp_params, n_outputs=2)

        res = minimize(_cost_tilt_with_correction, [tilt_deg], args=(df, model, fixed0, nn_model),
                        method="Nelder-Mead", options=_MINIMIZE_OPTIONS)
        tilt_deg_new = float(res.x[0])
        delta = abs(tilt_deg_new - tilt_deg)
        history.append({"iter": it, "tilt_deg": tilt_deg_new, "delta_deg": delta, "cost_mm2": float(res.fun)})
        tilt_deg = tilt_deg_new
        if delta < tol_deg:
            break
    return tilt_deg, nn_model, history

predict_centroids_field 

predict_centroids_field(y_nisp_mm, z_nisp_mm, wavelength_nm, tilt_deg, model, fixed_tier12, nn_model)

Predict centroids from the physical model (offset fixed at 0) plus a fit_field_dependent_tier-trained NN's field-dependent correction.

Returns:
  • (2, N) array -- [cent_y, cent_z], in mm (same shape as `predict_centroids`) –
Source code in dispcraft/field_calibration.py 
153
154
155
156
157
158
159
160
161
162
163
164
165
def predict_centroids_field(y_nisp_mm, z_nisp_mm, wavelength_nm, tilt_deg, model, fixed_tier12, nn_model):
    """Predict centroids from the physical model (offset fixed at 0) plus a
    `fit_field_dependent_tier`-trained NN's field-dependent correction.

    Returns
    -------
    (2, N) array -- [cent_y, cent_z], in mm (same shape as `predict_centroids`)
    """
    _check_fixed_tier12(fixed_tier12)
    fixed0 = {**fixed_tier12, "offset_y_mm": 0.0, "offset_z_mm": 0.0}
    pred = predict_centroids(y_nisp_mm, z_nisp_mm, wavelength_nm, [tilt_deg], ["tilt_deg"], model, fixed=fixed0)
    corr = ml_predict(nn_model, standardize_field(np.asarray(y_nisp_mm), np.asarray(z_nisp_mm)))
    return np.stack([pred[0] + corr[:, 0], pred[1] + corr[:, 1]])

standardize_field 

standardize_field(y_nisp_mm, z_nisp_mm, scale=FIELD_SCALE_MM)

(y0,z0) in mm -> (N,2) standardized array, the field-dependent NN's input.

Source code in dispcraft/field_calibration.py 
58
59
60
def standardize_field(y_nisp_mm, z_nisp_mm, scale=FIELD_SCALE_MM):
    """(y0,z0) in mm -> (N,2) standardized array, the field-dependent NN's input."""
    return np.stack([np.asarray(y_nisp_mm) / scale, np.asarray(z_nisp_mm) / scale], axis=1)