# GAN-EC: reproducibility of the operational networks

*Status: 9 October 2026. Covers the network pair that has run the live product since the 7 October 2026 00 UTC
cycle: **rR1et4** (seed A) and **rR2t4** (seed B).*

## In short

| Question | Answer |
|---|---|
| Can the networks be retrained bit for bit? | **No.** GPU training is not deterministic here (TF32 matrix math and cuDNN kernels), and seed B's main run was interrupted and resumed once (below). |
| Does the same recipe give equivalent networks? | **Yes, for the scores we track.** The two seeds started from different random states and took different paths, yet their test-year scores agree within a few percent (table below). |
| What varies most between retrainings? | **The heaviest rain.** At single 4 km pixels, seed A makes 100 mm in 3 h 0.63 times as often as SAAG and seed B 0.98 times. Starting the finishing stage after a long main run (90,000 steps) left 100 mm at 0.5–0.65 of SAAG; starting it after 32,000 steps brought it near 1. |
| Are the live forecasts reproducible from the saved networks? | **Yes, up to floating-point rounding.** Noise seeds are fixed per member, so the same checkpoint and the same global-model input give the same ensemble on the same hardware. Runs on the CPU fallback differ from GPU runs only in rounding. |
| Can someone outside our group reproduce it? | **Not yet without help.** The code is in a git repository without a public remote, and the training data (SAAG) and intermediate stores sit on the University of Illinois *keeling* cluster. See "Gaps". |

## Seed-to-seed agreement on the test years

Test periods never seen in training: all of 2017 (coastal El Niño, GEFSv12 reforecast inputs) and April–May 2021
(operational GFS inputs). 16 members per case; CRPS in the units of the variable; lower is better.

| | Seed A (rR1et4) | Seed B (rR2t4) |
|---|---|---|
| CRPS, 2 m temperature, 2017 / 2021 | 0.640 / 0.636 K | 0.654 / 0.630 K |
| CRPS, 2 m humidity, 2017 / 2021 | 0.523 / 0.507 g/kg | 0.523 / 0.501 g/kg |
| CRPS, 3-h precipitation, 2017 / 2021 | 0.921 / 1.048 mm | 0.925 / 1.053 mm |
| Spread / error, 3-h precipitation, 2017 / 2021 | 0.95 / 0.96 | 1.04 / 1.05 |
| Fractions skill score, 10 mm in 3 h, at 4 / 68 / 260 km | 0.068 / 0.193 / 0.411 | 0.070 / 0.195 / 0.398 |

The main difference is spread: seed A is slightly under-dispersed and seed B slightly over-dispersed, so the pair
used together is close to 1.

## Retraining seed B cleanly (8 October 2026)

Seed B was retrained in one uninterrupted run with the seed A recipe and `--seed 1`
(`slurm/train_pair_member.sbatch rR2r 1`: main run `rR2r`, finishing stage `rR2rt4`, best.pt at step 3,000, sha256
`c6d651e803900ae3e907379bd42dc45574911dedac0d48c53977734a8c71f990`).

| | Seed A (live) | Seed B (live) | Seed B, clean rerun |
|---|---|---|---|
| CRPS, 3-h precipitation, 2017 / 2021 | 0.921 / 1.048 mm | 0.925 / 1.053 mm | 0.921 / 1.047 mm |
| CRPS, 2 m temperature, 2017 / 2021 | 0.640 / 0.636 K | 0.654 / 0.630 K | 0.647 / 0.628 K |
| Spread / error, 3-h precipitation, 2017 / 2021 | 0.95 / 0.96 | 1.04 / 1.05 | 0.94 / 0.91 |
| 100 mm in 3 h at single pixels, ratio to SAAG | 0.63 | 0.98 | 0.49 |
| 100 mm in 3 h within 20 km, ratio to SAAG | 0.89 | 0.99 | 0.71 |

The recipe reproduces the skill scores; it does **not** reproduce the heaviest rain. The clean rerun makes 100 mm
less often than SAAG, like seed A, so the live seed B's near-SAAG frequency of 100 mm is an outcome of its
particular (interrupted) training path rather than of the recipe. The live pair is unchanged.

**Replaying the live seed B's exact path** (9 October 2026; `slurm/replay_seedB.sbatch rR2p`: 26,000 steps with
rR2's settings, a 6,000-step restart with a fresh optimizer as rR2b did, then the finishing stage; best.pt sha256
`bc37dbc3e1f3e71f24a7b3231d729cb5e230bc86ebd91f9682ef804820a6e50f`) gave the same skill (precipitation CRPS 0.924 /
1.049 mm, spread/error 0.98 / 0.96) but 100 mm in 3 h at 0.72 of SAAG's frequency at single pixels and 0.88 within
20 km, against 0.98 and 0.99 for the live seed B. The same commands on the same hardware, differing only by GPU
rounding, therefore move the heaviest-rain frequency by a quarter. Across the four networks trained with this recipe
it ranges from 0.49 to 0.98 at single pixels and 0.71 to 0.99 within 20 km, while the skill scores agree within
about 1 %. The live seed B's checkpoint is the reproducible artifact; its heavy-rain behavior is not reproducible by
retraining.

## Fingerprint of what is running

| Item | Value |
|---|---|
| Seed A checkpoint | `runs/rR1et4/best.pt` (step 3,000 of the finishing stage), sha256 `78c4db1c1d42836a853a3944478b7a726fc752908fc20cdba62111fff4adf323` |
| Seed B checkpoint | `runs/rR2t4/best.pt` (step 3,000 of the finishing stage), sha256 `16ae2a3ba33c544ce6cdd6c0de63257b20194a63625ac8818cfce82c3dd7b63b` |
| Training code | commit `3c1b508` (3 October 2026); `train.py`, `models.py`, `dataset.py` and `shards.py` are unchanged since |
| Environment | Python 3.12.14, PyTorch 2.6.0 (CUDA 12.4, cuDNN 9.1), NumPy 2.5.3, xarray 2026.7.0, zarr 3.3.0, pandas 3.0.6 (conda env `gan-ec`) |
| Hardware | one NVIDIA L40S (48 GB) per training run |
| CHIRPS correction | `biascorr/qm_oper_rR1et4-rR2t4.nc`, sha256 `295417ab63c9ad8a45c2e422fc52cce2990a87fb52d253e51181a03de507cf3c` |
| Risk calibration | `risk_calib/risk_calibration.csv` (= `risk_calibration_rR1et4-rR2t4.csv`), sha256 `be6a2885a0cfec7efcc86901bfe2dbc2b129fd3e288f4313893d6aed3a303bab` |
| Multi-model weights | `multi/weights_chirps.json`, sha256 `4050a8b2ce42a214f4ec686117e50b06dbe9e00500def66b3d277e0315e52027` |

Paths are relative to `$WORK` = `/data/scratch/a/snesbitt/gan-ec`. Each checkpoint also stores its full argument
set (`args`) and input count, and `runs/<run>/args.json` repeats the arguments.

## Data

| Data | Role | Source |
|---|---|---|
| SAAG WRF, 4 km, hourly, 2000–2021 | training target ("truth") | South America Affinity Group convection-permitting simulations (Dominguez et al. 2024), read from the copies on keeling (`/data/francina/{d,e,f,g}/SAAG`); subset to a 456 × 456 box over NW South America (`data/extract_saag.py` → `$WORK/saag/saag_YYYYMM.zarr`, 618 GB) |
| GEFSv12 reforecast, control member, 0.25° | coarse inputs, 2000 to May 2019 | NOAA, public on AWS (`data/fetch_gefs_reforecast.py`) |
| Operational GFS, 0.25° | coarse inputs, June 2019 to 2021 | NCAR GDEX ds084.1 (`data/fetch_gfs_ncar.py`) |
| Combined coarse store | all inputs on the 24 km grid | `data/build_training.py` → `$WORK/lr_all.zarr` (27 GB) |
| 4 km static fields | terrain, land mask | from SAAG → `$WORK/hr.zarr` |

Split (`config.py`, `gan/shards.py`): test = all of 2017, April–May 2021 and September–October 2021; validation =
the last 6 days of every month; training = the rest. Months are streamed in shards of 24 months, 4,000 steps per
shard.

## Training recipe

Both seeds use identical settings except `--seed` (0 for A, 1 for B) and the length of the main run. Every
setting below is the saved value, not a default.

**Stage 1, main run** (relativistic GAN with R1/R2 penalties, adversarial term on after 5,000 steps, heavy-rain
terms throughout):

```
python gan/train.py --run rR1e --steps 32000 --pretrain 5000 --batch 16 --patch 16 --m 4 --n_critic 2 \
  --w_adv 0.005 --w_gp 10.0 --gan_loss rpgan --gamma 1.0 --d_beta1 0.0 --d_beta2 0.99 --d_warmup 0 \
  --lr_g 0.0001 --lr_d 0.0001 --val_every 2000 --log_every 500 --seed 0 --w_pcoarse 0.2 \
  --p_transform log1p --ema 0.999 --data all --shard_months 24 --shard_steps 4000 --val_months 12 \
  --gpu_data 1 --eps 0.0 --w_tail 20.0 --w_nb 4.0 --eps_tail 0.05 --eps_tail_hi 0.5 --eps_tail_split 75.0 \
  --eps_tail_split2 150.0 --tail_width 0.1 --soft_cap 1 --up resize --w_T 1.0 --w_Q 1.0
```

**Stage 2, finishing** (half the generator learning rate; the heavy-rain term's weight above 75 mm drops from 0.5
to 0.05 so it no longer suppresses isolated heavy members):

```
python gan/train.py --run rR1et4 --init $WORK/runs/rR1e/last.pt --steps 4000 --pretrain 0 --lr_g 5e-05 \
  --val_every 1000 --eps_tail_hi 0.05   [all other settings as stage 1]
```

**Checkpoint selection.** Validation every 1,000 (stage 2) or 2,000 (stage 1) steps. `best.pt` is the lowest
validation CRPS once the adversarial term is on, among checkpoints whose 100 and 150 mm exceedance frequencies are
within a factor of 2 of SAAG's. Both live networks are `best.pt` of stage 2, at step 3,000.

**How each seed actually got there:**

| | Main run | Finishing stage starts from | Live checkpoint |
|---|---|---|---|
| Seed A | `rR1e`: 32,000 steps in one run (2.4 h) | `rR1e/last.pt`, step 32,000 | `rR1et4/best.pt`, step 3,000 |
| Seed B | `rR2`: stopped by an out-of-memory error at a shard swap after step 28,000 (last saved step 26,000; the bug was fixed in `3c1b508`); resumed as `rR2b` from `rR2/last.pt` | `rR2b/best.pt`, 6,000 steps after the resume (32,000 in total) | `rR2t4/best.pt`, step 3,000 |

Seed B's resume restarted the optimizer state and the shard sequence, which is why its path cannot be replayed
exactly. Retraining it as seed A was trained (one uninterrupted 32,000-step run with `--seed 1`) is the
reproducible equivalent. The seed A chain is the script `$WORK/tmp/seedA_test.sh` (one L40S job, about 4.8 h with
the evaluations it also runs).

## After training: the calibrations the live product depends on

Each step is deterministic given the checkpoints and the stores, apart from GPU rounding in the hindcasts.

| Step | Command | Output |
|---|---|---|
| Risk calibration (member count → probability, isotonic, fit on 2017 against SAAG) | `calibrate_risk.py counts --run rR1et4 --m 16`; same for `rR2t4`; `calibrate_risk.py fit --runs rR1et4 rR2t4 --suffix _rR1et4-rR2t4` | `risk_calib/risk_calibration_rR1et4-rR2t4.csv`, installed as `risk_calibration.csv` |
| CHIRPS correction (quantile mapping of daily totals) | `bias_correct.py hindcast_oper --runs rR1et4 rR2t4 --m 4`; `bias_correct.py fit --source oper_rR1et4-rR2t4` | `biascorr/qm_oper_rR1et4-rR2t4.nc` |
| Multi-driver hindcast (six drivers, from each driver's archive start, November 2024 for most, to September 2026) | `hindcast_multi.py <driver> --runs rR1et4 rR2t4 --chunk 8` for gfs, gefs, aigefs, hres, aifs, ifs | `multi/<driver>_24km.zarr`, `multi/<driver>_nb.zarr` |
| Multi-model weights and daily calibration against CHIRPS v3 | `chirps_daily_cal.py --runs rR1et4-rR2t4` | `multi/weights_chirps.json`, `multi/chirps_daily_calibration.csv` |

## Checking a retrained network

```
python gan/evaluate.py --run <run> --m 16          # test-year CRPS, spread/error, FSS -> runs/<run>/eval/
python gan/structure_eval.py --runs <run>:best.pt --windows 240 --m 4   # heavy-rain frequencies, grid pattern, rain areas
```

A retrained network should fall within the seed A–seed B range of the table above. Check the heavy-rain
frequencies separately: they are the least reproducible quantity.

## Live forecasts

`realtime/cycle_driver.sbatch` runs `gan/demo.py` (GFS) or `gan/ifs_ens.py` (other drivers) with
`--runs rR1et4 rR2t4 --noise_rho 0.6` and the CHIRPS correction above. Member *i* draws its noise from seed
`noise_seed × 1000 + i` (default `noise_seed` 0), evolving as an AR(1) process from window to window, so a cycle
can be regenerated from the saved checkpoint and the archived global-model input. Products, calibration and the
multi-model pool follow in `realtime/cycle_update.sbatch` (`risk_product.py`, `dashboard.py`). The RRI rainfall files
(`hydro_members.py`) were retired on 20 October 2026.

## Gaps, and what would close them

- **Code is not public.** The repository has no public remote. Archiving it with a DOI (with the checkpoints and
  calibration files above) would make the product citable and reproducible outside the group.
- **The heaviest rain is not reproducible by retraining**, even along the same path (above). The live checkpoints,
  with their hashes, are what must be archived.
- **Seed B's path** can be replayed from commands (`slurm/replay_seedB.sbatch`). The recipe is also a repository script
  (`slurm/train_pair_member.sbatch`, either seed), and a clean seed B reproduces the skill scores but not the
  heaviest rain (above). Heavy-rain frequency depends on the training path, which a generator trained without an
  adversary (diffusion or flow matching) may make less sensitive.
- **The original seed A chain** is kept as `slurm/history/seedA_test_2026-10-04.sh`.
- **Bit-for-bit reruns would need** `torch.use_deterministic_algorithms(True)` and TF32 off, at a cost in speed.
  The scores above suggest this is not needed to reproduce the product's skill.
- **Data access.** SAAG and the intermediate stores (SAAG subsets 618 GB, coarse inputs 27 GB) are on keeling.
  Everyone on keeling can now read `$WORK`; outside users would need a data release.

*Related: the methodology document describes what the system does; this page covers how to rebuild the networks
it runs. Source of this page: `docs/REPRODUCIBILITY.md` in the GAN-EC repository.*