NISP grism spectral-trace calibration model — implementation spec¶

Source: Euclid Collaboration: W. Gillard et al., "The NISP spectroscopy channel, on ground performance and calibration" (2506.08378v2). Section 5 (model + fitting procedure), Appendix A (coefficient tables).

Goal of this doc: give Claude Code everything needed to reimplement the on-ground NISP grism dispersion-law model — no need to re-derive anything from the paper text.

1. Model definition and expression¶

1.1 Physical picture¶

Each NISP grism produces, for a point source at focal-plane reference position (y0, z0) (the 0th-order / undeflected position), a 1st-order spectral trace: a curve (y(λ), z(λ)) on the focal plane parameterized by wavelength λ. The model describes this trace, and how its shape changes across the field of view (FoV).

Axes convention: y = cross-dispersion direction, z = dispersion direction (both parallel to RGS000's directions, used as the common reference frame for all grisms/orientations).

There are two nested models:

• Eq. (2) — per-position trace model: maps wavelength → focal-plane displacement, for one fixed reference position (y0, z0).
• Eq. (3) — spatial model: describes how the Eq. (2) coefficients themselves vary with (y0, z0) across the FoV.

Composing them gives a single function f(λ, y0, z0) → (y, z) valid over the whole FoV. Eq. (5) is just Eq. (2) rewritten in a regular (non-Chebyshev) polynomial basis, useful for physical interpretation (not required for the forward model itself, but easy to support since the paper tabulates it directly for the FoV-averaged case).

1.2 Eq. (2) — spectral trace model (per reference position)¶

y - y0 = Σ_{i=0}^{m-1} C_y_i · T_i(λ')
z - z0 = Σ_{j=0}^{n-1} C_z_j · T_j(λ')

λ' = (λ - 0.5(λmax + λmin)) / (0.5(λmax - λmin))

• (y, z): focal-plane position [mm] of the spectral feature at wavelength λ.
• (y0, z0): reference position [mm] = the 0th-order position for that spectrogram.
• T_i(x): Chebyshev polynomial of the first kind, degree i, standard recurrence T_0=1, T_1=x, T_{k+1}=2x·T_k - T_{k-1}.
• λ': wavelength normalized to [-1, 1] over [λmin, λmax] (the grism's passband, Table 2 of the instrument paper / FWHM bandpass column below).
• C_y_i, C_z_j: Chebyshev coefficients — these are themselves functions of (y0, z0), given by Eq. (3) below. m, n are the polynomial orders used per grism (Sect. 2.2).

1.3 Eq. (3) — spatial dependence of the trace coefficients¶

C_κ_i(y0, z0) = Σ_{k=0}^{m'-1} Σ_{l=0}^{n'-1} a_{kl} · T_k(z0') · T_l(y0')

(paper uses indices i,j for both the outer sum in Eq.3 and the coefficient index C_κ_i — in this spec I use k,l for the Eq.3 double sum to avoid clashing with the i index of Eq.2; they are the same object as in the paper, just renamed for clarity.)

• κ ∈ {y, z}; i runs over the same index as in Eq. (2) (i.e., you evaluate C_y_0, C_y_1, C_y_2, ... and C_z_0, C_z_1, C_z_2, C_z_3, ... each with its own 2D Chebyshev expansion a_{kl}).
• T_k, T_l: Chebyshev polynomials of the first kind again, now over normalized focal-plane coordinates.
• a_{kl}: the tabulated calibration parameters (Appendix A tables) — this is the actual fitted content you need to hardcode/load.

Normalization (Eq. 4), applies to both y0 and z0:

κ' = (κ - 0.5(κmax + κmin)) / (0.5(κmax - κmin))

with [κmin, κmax] = [-85, +85] mm for both y and z axes (fixed FoV half-size, same for every grism).

1.4 Eq. (5) — regular-polynomial equivalent (FoV-averaged only, optional)¶

Py = y - y0 = Σ_{i=0}^{m-1} w_y_i · λ^i
Pz = z - z0 = Σ_{j=0}^{n-1} w_z_j · λ^j

• Same trace model as Eq. (2), but expressed as ordinary powers of λ (not normalized λ', not Chebyshev) instead of Chebyshev polynomials in λ'.
• w_κ_i(y0,z0) are obtainable from C_κ_i by the standard Chebyshev→power-series basis transform (linear, exact); the paper only tabulates the FoV-averaged w values (Table A.1) — it does not tabulate a spatial model for w, only for C (Eq. 3). So Eq. (5) is only directly usable with the averaged coefficients; for a position-dependent model you must go through Eq. (2)+(3) and, if you want w, convert numerically per-position.
• Physical meaning of w (paper, Sect. 6): w_κ_0 = distance between 0th and 1st order at the trace's reference wavelength; w_κ_1 = local dispersion dλ/dκ; higher orders = trace curvature / nonlinearity from field distortion.

1.5 Practical evaluation order (forward model)¶

Given a target grism, a wavelength λ, and a focal-plane reference position (y0, z0):

1. Normalize (y0, z0) → (y0', z0') via Eq. (4), range [-85, 85] mm.
2. For each needed coefficient C_y_i (i = 0..m-1) and C_z_j (j = 0..n-1): evaluate Eq. (3) using that coefficient's a_kl table → scalar value at this position.
3. Normalize λ → λ' via Eq. (2)'s own normalization, range [λmin, λmax] = grism passband.
4. Evaluate Eq. (2) with the C_y_i(y0,z0), C_z_j(y0,z0) from step 2 → (y - y0, z - z0).
5. Add back (y0, z0) → absolute focal-plane trace position at wavelength λ.

This is the function you need for e.g. simulating/predicting a spectral trace anywhere on the NISP FoV, for any of the calibrated grism configurations.

2. Model parameters from Appendix A¶

2.1 What's tabulated, and for which grism configurations¶

Ground calibration was performed independently for 7 grism "configurations" (treating tilted positions as distinct grisms):

• BGS000 (blue grism, nominal)
• RGS000 (red grism, nominal), RGS000-4°, RGS000+4°
• RGS180 (red grism, nominal), RGS180-4°, RGS180+4°
• RGS270 — ground data exists but no calibration coefficients are given/fitted (grism physically mis-mounted 180°, tilted focal plane along its dispersion direction puts it out of focus outside its central wavelength; excluded from Table A.1 and has no A.x table).

Polynomial orders used (Sect. 5.1, text): Chebyshev expansion in Eq. (2) up to 3rd order in cross-dispersion (y, i.e. m=4 terms, i=0..3) and up to 4th order in dispersion (z, i.e. n=5 terms, j=0..4) for the red grisms; for the blue grism BGS000 both directions limited to 3rd order (m=n=4, but tables actually show only up to i=2/j=2 for Cy and Cy for BGS000 in Table A.2, i.e. effectively 2nd order was populated — use the a_kl table's actual populated entries as ground truth over the text statement).

FoV limits for Eq. (4): [y0min,y0max] = [z0min,z0max] = [-85, +85] mm, same for all grisms.

2.2 Table A.1 — FoV-averaged coefficients (both bases), used for Eq. (2) with fixed¶

(non-spatial) coefficients, or for Eq. (5)

All values as printed (Chebyshev ⟨C_κ_i⟩ and regular-polynomial ⟨w_κ_i⟩, identical numerically per the paper's convention here — i.e. paper lists same numbers under both headers for these averaged rows):

Units: Cy0, Cz0 in mm; Cy1, Cz1 in mm (dimensionless coefficient of T_1(λ'), since λ' is dimensionless — the physical dispersion scale is folded into the Chebyshev-to-power-series conversion, see w values which carry mm/nm units); higher orders likewise dimensionless in the Chebyshev basis. Regular-basis units (identical values here, but conceptually): w_κ_0 [mm], w_κ_1 [mm/nm], w_κ_2 [mm/nm²], w_κ_3 [mm/nm³].

Sanity check embedded in the paper: w_y_1 (cross-dispersion) is ~2 orders of magnitude smaller than w_z_1 (dispersion) for nominal grism positions — confirms dispersion is essentially 1D along z at nominal tilt, and only becomes comparably strong in y for the ±4° tilted configurations (there w_y_1 is only ~1 order of magnitude below w_z_1).

2.3 Tables A.2–A.8 — spatial coefficients a_kl for Eq. (3), per grism¶

Each table gives a_kl (rows = kl index pairs a00, a01, a02, a03, a10, a11, ... a33) for the columns Cy0, Cy1, Cy2, [Cy3], Cz0, Cz1, Cz2, Cz3. A — entry means that Chebyshev term was not fitted for that coefficient (expansion truncated at lower order) and should be treated as 0, not "missing/unknown."

Row index convention: a_kl multiplies T_k(z0') · T_l(y0') (k = z-order, l = y-order) per Eq. (3) as printed. Double-check this against the paper before wiring signs/axes if it matters for your use case — the paper's own Eq. (3) writes T_i(z')T_j(y') with i associated to the Cκ_i degree in λ (from Eq. 2) — the (k,l) pair here is an independent set of indices for the spatial expansion of that one fixed Cκ_i. I.e., for each Cy_i or Cz_j (i,j from Eq. 2), there is one full a_kl(z0', y0') 2D-Chebyshev surface, given by one column of the table below.

Table A.2 — BGS000

Table A.3 — RGS000-4°

Table A.4 — RGS000

Table A.5 — RGS000+4°

Table A.6 — RGS180-4°

Table A.7 — RGS180

Table A.8 — RGS180+4°

RGS270: no a_kl table exists — do not implement a trace model for this grism from this paper; if needed, flag as unsupported.

2.4 Auxiliary numbers needed to fully instantiate the model¶

• Grism passbands [λmin, λmax] for Eq. (2)'s λ' normalization (from NISP instrument paper, Jahnke et al. 2024, Table 2 — reuse those values here since this paper does not repeat them): red grisms (RGS000/180, all tilts) 1206–1892 nm; blue grism (BGS000) 926–1366 nm.
• FoV normalization for Eq. (4): [-85, +85] mm on both axes, all grisms.
• Fit-quality diagnostics (not needed for forward evaluation, useful for validation/tests): average reduced χ² ≈ 0.92 ± 0.03 over all configurations; predicted vs. measured Argon line positions agree to ≈ 0.5 pixel; 0th-order centroid uncertainty ≈ 0.05 px (cross-dispersion), ≈ 0.12 px (dispersion), from χ²-profiling.

3. How to use it — implementation notes¶

3.1 What to build¶

A function (or small class) per grism configuration:

predict_trace(lam_nm, y0_mm, z0_mm, grism="RGS000") -> (y_mm, z_mm)

Internally: 1. Look up [λmin, λmax] for that grism (Sect 2.4). 2. Look up polynomial orders (m, n) actually populated in that grism's a_kl table (i.e. read the table shape — don't assume fixed 4×5 orders; BGS000 truncates lower). 3. For κ in {y, z}, for each order index (0..m-1 or 0..n-1): evaluate the Eq. (3) double Chebyshev sum in normalized (z0', y0') using that coefficient's a_kl slice → gives C_κ_i(y0, z0) as a plain float. 4. Plug the resulting {C_y_i}, {C_z_j} into Eq. (2) at the requested λ, normalized the same way, to get (y-y0, z-z0). 5. Return (y0 + Δy, z0 + Δz).

Recommend implementing Chebyshev evaluation with numpy.polynomial.chebyshev.chebval (1D) called twice per coefficient (once over z0', once over y0', or via chebval2d if available) rather than hand-rolling recursion — reduces bug surface. Same for the outer Eq. (2) evaluation in λ'.

3.2 Inverse problem (if needed downstream)¶

The paper's own calibration direction is: measure (y, z) of known spectral features (Fabry–Pérot peaks, Argon lines) at many (y0, z0) → fit a_kl. If the implementation needs the inverse (given observed (y, z) at a known/estimated (y0, z0), solve for λ), that requires root-finding on Eq. (2) (1D in λ, e.g. Brent's method) since Eq. (2) is not analytically invertible in Chebyshev form for m,n > 2. This is not derived in the paper — treat as a numerical add-on, not a documented model.

3.3 Validity / limitations to encode as guardrails, not silently ignore¶

• Valid λ range is the grism passband only ([λmin, λmax] above) — extrapolation outside is unconstrained by data (Chebyshev fits diverge/behave poorly outside their fit interval).
• Valid (y0, z0) range is [-85, 85] mm on both axes (FoV limits) — same caveat for Eq. (3).
• RGS270 has no fitted model in this reference (exclude/flag).
• These are ground/lab-derived coefficients (telescope simulator, not real telescope); the paper explicitly notes the true telescope PSF/optics were not in the light path during this calibration, so in-flight dispersion may differ somewhat (see actual instrument paper, Table 5, for in-flight PSF numbers, not trace numbers — no in-flight trace-recalibration numbers are in this document).
• Each grism config (nominal / −4° / +4°) is calibrated as an independent model, not a continuous function of tilt angle — don't interpolate between them without independent justification; the paper treats them as unrelated grisms for calibration purposes.

4. Model accuracy — benchmarks for comparison with new models¶

This section collects every quantitative accuracy/precision number reported for the Eq.(2)+(3) trace model and its associated measurement chain, so a new model can be benchmarked on the same metrics and against the same baseline numbers. Distinguish three different things that are easy to conflate: (a) input measurement precision (noise floor on the data used to fit the model), (b) the trace-model's own fit/reconstruction accuracy, and (c) downstream science-performance numbers (resolving power) that depend on the trace model plus the PSF.

4.1 Input measurement precision (noise floor — sets a lower bound on any model's achievable accuracy)¶

• 0th-order centroid position error (1σ, from χ²-profiling of the template fit): mean 0.051 ± 0.006 px (std 0.15 px) in cross-dispersion direction; mean 0.12 ± 0.02 px (std 0.4 px) in dispersion direction. → cross-dispersion positioning is intrinsically ~2.5x more precise than dispersion-direction positioning, because the 0th-order profile is much narrower in cross-dispersion.
• Fabry–Pérot / 1st-order peak centroiding accuracy: ≃ 1/10 px (from PSF-profile fitting).
• Detector metrology (focal-plane position of each detector, used to convert pixel→mm): room-temperature precision ≃ 1 px (≃ 18 µm); at operating temperature this relies on a thermal model (not a direct cold measurement) — an acknowledged, uncorrected systematic. Effect: spectra whose 0th and 1st order fall on two different detectors show a local offset in the fitted C_y_0/C_z_0 coefficients (bias, not noise) attributed to this metrology uncertainty; checked to keep calibration parameters within 3σ whether or not those spectra are included in the fit.
• PSF-model choice barely matters: three independent PSF profiles (asymmetric-Gaussian, dual-Gaussian, Moffat) give centroid positions consistent to std < 0.1 px, mean offset (7–10)×10⁻³ px between any pair of models. Reduced χ² medians: asym-Gaussian ≈1.04, dual-Gaussian ≈0.97, Moffat ≈1.02 — statistically indistinguishable, no preferred model.
• Data-quality cuts applied before fitting (relevant if reproducing the reference numbers): reject PSFs with reduced χ² > 5 (~5% of PSFs excluded, mostly hot/bad-pixel or cosmic-ray contamination); reject any PSF with unphysical peak signal > 10 000 e⁻/s.

4.2 Trace-model (Eq. 2 + Eq. 3) fit quality — the main baseline to beat¶

• Fit goodness-of-fit: recursive χ² fit (outliers >5σ from trace rejected each iteration until convergence). Chebyshev expansion order: up to 3rd order cross-dispersion / 4th order dispersion for red grisms; 3rd/3rd for blue grism (chosen so that reduced χ² ≈ 1). Resulting averaged reduced χ² ≈ 0.92 ± 0.03 across configurations (skewed toward lower values — a few 0th-order template-fit inaccuracies propagate into some 1st-order positions and inflate a subset of position-error estimates, biasing reduced χ² down for those points).
• Independent validation (held-out data): dispersion law fitted on Fabry–Pérot data, validated against separately-acquired Argon-lamp spectra (16 spectrograms per grism, at detector-centre pointings) → predicted vs. measured Argon line position agreement ≃ 0.5 px. This is the most direct "generalization error" number in the paper and the most natural target for a like-for-like comparison with a new model evaluated on the same held-out set.
• In-sample reconstruction error (predicted vs. measured Fabry–Pérot line positions, same data used for the fit, via Eq. 2+3): < 7×10⁻⁴ mm (< 0.04 px) cross-dispersion, < 1×10⁻³ mm (< 0.06 px) dispersion direction, for every grism configuration (Fig. 12 histograms, BGS000 / RGS000±tilts / RGS180±tilts). → Note the ordering: in-sample residual (0.04–0.06 px) is smaller than the held-out Argon residual (~0.5 px), which is itself smaller than the raw 0th-order centroid noise floor (~0.05–0.4 px) — the ~0.5 px Argon number is dominated by Argon-specific systematics (see below), not by the trace model's intrinsic fit quality.
• Known systematic not corrected in this reference model: cross-detector 0th/1st-order metrology bias described in §4.1 — a new model with an independently-calibrated focal-plane metrology (e.g. from cold data) could plausibly beat the reference on this specific term.
• Argon validation caveat: Argon spectrograms were affected by persistence-charge contamination from the preceding Fabry–Pérot exposure at the same pointing — the ~0.5 px validation number may be partly limited by this instrumental artifact rather than by the trace model itself; keep this in mind if a new model's validation set doesn't share this specific systematic (i.e. a new model beating 0.5 px on clean data isn't necessarily "better," it may just avoid a systematic the reference validation happened to have).

4.3 Downstream science-performance numbers (model + PSF combined)¶

• Resolving power R (Eq. 6, R = λ/(2 Δλ σ_e), for a 0.5″ FWHM source), averaged over FoV:
• Blue grism (BGS000): R ≃ 440 at 900 nm → R ≃ 690 at 1300 nm.
• Red grisms (RGS000/RGS180): R ≃ 550 at 1300 nm → R ≃ 740 at 1800 nm.
• Requirement: R ≥ 260 (blue), R ≥ 380 (red) — measured values exceed requirement by a comfortable margin (~1.7x at band edges).
• Image quality (EE50/EE80) vs. requirement: measured EE50 ≃ 0.6× the requirement (< 0.3″ at 1500 nm) — i.e. NISP grisms deliver PSFs ~40% tighter than the minimum spec. No numerical EE50/EE80 values tabulated beyond Table 5 of the instrument paper (already captured in the earlier extraction) — this paper's Fig. 6/7 give the FoV-averaged trend and the null result for ±4° tilt (no measurable EE50/EE80 change under tilt).
• Redshift error implied by R (from NISP instrument paper, consistent with these R values): σ(z) < 0.001(1+z) for Hα emission-line galaxies of 0.5″ FWHM.
• Payload-module-level check (with real Euclid mirrors + dichroic in the beam): confirmed no significant modification to dispersion, resolving power, or optical performance relative to the NISP-alone ground tests above — i.e. the grism-level numbers above should carry over to the as-integrated instrument without a large correction factor (this is a qualitative confirmation, not a new quantitative bound).

4.4 Suggested comparison protocol for new (ML-based) models¶

To make an apples-to-apples comparison against this reference model: 1. Match the metric: report position residuals in the same units the paper uses — mm on the focal plane and pixels (18 µm/px) and separately for cross-dispersion (y) vs. dispersion (z) direction; these two directions have very different noise floors and error budgets and averaging them together hides which axis a new model actually improves. 2. Match the split: report both an in-sample number (fit residual) and a held-out number (trained on Fabry–Pérot-like data, validated on an independent line set) — the reference model's two numbers differ by >10x (0.04–0.06 px vs. 0.5 px), so a single aggregate number is not comparable to either of the reference's reported figures. 3. Report against the noise floor, not zero: the ~0.05–01.4 px input centroiding precision (§4.1) is a hard floor below which "improvement" is not physically meaningful without also improving the underlying centroiding method — a new trace model achieving sub-noise-floor residuals on the same input centroids would indicate overfitting, not genuine improvement. 4. State the FoV/λ coverage: reference numbers are FoV-averaged (Table A.1-based) or pooled across the ±85 mm / grism-passband domain (Fig. 12 histograms) — if a new model is evaluated only at FoV centre or on a narrower λ range, note this explicitly since the reference model's residuals are known to vary smoothly across the FoV (radial dispersion gradient, Fig. 14) and could look artificially better/worse on a subset. 5. If comparing resolving power rather than raw trace-position accuracy, remember R depends on both the trace model (via Δλ = dλ/dz) and the PSF model (via FWHM) — an improvement in trace-model accuracy alone will only move R if it changes the effective dispersion estimate, not just the residual scatter.

5. Suggested test cases for the implementation¶

• At the FoV centre (y0,z0)=(0,0), Eq. (3) should reduce (up to numerical fit residuals) to something close to the Table A.1 averaged values — use as a smoke test (not exact equality, since Table A.1 is itself a separate average fit, not the exact value of Eq. 3 at (0,0), but should be the same order of magnitude and sign).
• w_y_1 should be ~100x smaller than w_z_1 for nominal (non-tilted) grisms; ~10x smaller for ±4° tilted grisms — regression check on the physical sanity of any basis-conversion code if Eq. (5)/regular-polynomial support is implemented.