Panel pipeline: full plumbing for per-year snapshots (closes #345)

[vahid-ahmadi]

Summary

Turns the single-year cross-sectional pipeline into a full dynamic panel microsimulation: real ONS demographic rates, real UKHLS-estimated within-person transitions, and a one-call composer that ages the UK population forward year-by-year. Closes #345.

No change to the default build. Every new behaviour is opt-in via create_yearly_snapshots(mode="full_panel") or by calling advance_year() directly.

The PR grew into four layers:

1. Panel-ID plumbing (steps 1-2 of Generate panel data across years (synthetic panel on FRS 2023-24 base) #345): ID contract + yearly-snapshot helper.
2. Dynamics: demographic ageing + the five life-cycle transitions (marriage, separation, children leaving home, migration, within-person income/employment).
3. Real data: ONS mortality / fertility / marriage / divorce / leaving-home / migration statistics, and UKHLS panel microdata (Waves 1-15) for within-person transitions.
4. Composition: advance_year and a mode="full_panel" on create_yearly_snapshots to chain it all together deterministically under one seed.

---

What's in the pipeline

Every component below ships as opt-in machinery — callers decide which rates to use and when to run transitions.

Panel identity and snapshots

• utils/panel_ids.py: PANEL_ID_COLUMNS, get_panel_ids, assert_panel_id_consistency, classify_panel_ids, PanelIDTransition. Documents household_id / benunit_id / person_id as the panel keys.
• datasets/yearly_snapshots.py: create_yearly_snapshots(base, years, output_dir, *, mode, ...).
  • mode="uprate_only" (default): uprate each target year, IDs preserved byte-for-byte.
  • mode="full_panel": roll the population forward year-by-year via advance_year; IDs evolve (deaths / births / migration).
• storage/upload_yearly_snapshots.py: parallel uploader to the existing private HF destination. Destination constants are not exposed as arguments — redirecting the upload requires a code edit reviewed under CLAUDE.md data-protection rules.

Demographic ageing

• utils/demographic_ageing.py: age_dataset(base, years, *, mortality_rates, fertility_rates, seed).
• Mortality removes person rows, fertility appends newborns attached to the mother's benunit + household, age increments.
• Accepts either age-only Mapping[int, float] or sex-specific Mapping[str, Mapping[int, float]] for mortality.

Household-composition transitions

All in utils/household_transitions.py:

• apply_marriages: pair unmarried adults within region, closest-age match, merge benunits (weights folded).
• apply_separations: split two-adult benunits; children default to the mother.
• apply_children_leaving_home: move adult dependents (both FRS shapes — dependent young adults on parents' benunit, and separate benunits sharing the parental household) into fresh benunit + household.
• apply_migration: Poisson-distributed net inflow / outflow per age, inflow cloned from same-age donors, outflow removed cleanly.

Labour-market transitions

• apply_employment_transitions(dataset, *, ukhls_rates=None, ...):
  • Rule-based (default): retirement at SPA → CPI-plus wage drift → job loss / gain.
  • UKHLS-driven (opt-in): classifies each working-age person into a four-state label and draws the next state from the empirical age-band × sex transition matrix estimated from UKHLS.
• apply_income_decile_transitions: ranks workers into deciles within (age_band, sex), draws a destination decile from the UKHLS matrix, rescales income by the ratio of target/origin decile medians.

Year-aware calibration plumbing

• targets/build_loss_matrix.py: resolve_target_value(target, year, *, tolerance=3), constant YEAR_FALLBACK_TOLERANCE = 3.
• Two real bug fixes picked up along the way:
  • local_authorities/loss.py was reading household_weight at a hard-coded 2025.
  • constituencies/loss.py was reading dataset.time_period instead of the passed time_period argument in two places.
    Both now honour the argument.
• docs/targets_coverage.md: year coverage across every target category plus the remaining data-sourcing gaps.

Cross-year smoothness

• utils/calibrate.py: compute_log_weight_smoothness_penalty, plus two keyword-only kwargs on calibrate_local_areas (prior_weights, smoothness_penalty). When both are supplied, a log-space L2 penalty pulls the current year's weights towards the prior year's. Off by default — pre-step-5 behaviour is reproduced bit-for-bit.

Top-level composer

• utils/advance_year.py: advance_year(dataset, *, target_year, seed, ...). Chains migration → separation → leaving-home → marriage → employment transitions → income-decile transitions → mortality → fertility → age increment → uprating. One seeded RNG so the whole year is reproducible.

---

Real-data backing

All seven rate sources are either public ONS data (committed alongside the existing ONS xlsx fixtures in storage/) or aggregated UKHLS derivatives (cell-suppression ≥ 10 applied, raw microdata never committed).

Transition	Source	Location on branch
Mortality	ONS National Life Tables UK (1980–2024, age × sex × qx)	targets/sources/ons_mortality.py + storage/ons_national_life_tables.xlsx
Fertility	ONS Births in England and Wales: registrations Table 10	targets/sources/ons_fertility.py + storage/ons_asfr.xlsx
Marriage	ONS Marriage Statistics England & Wales	utils/household_transitions.py
Separation	ONS Divorce Statistics England & Wales	utils/household_transitions.py
Children leaving home	ONS LFS Young adults living with parents	utils/household_transitions.py
Migration	ONS Long-Term International Migration	utils/household_transitions.py
Within-person income / employment	UKHLS Waves 1–15 (SN 6614), 601,795 person-wave obs	datasets/ukhls.py + utils/ukhls_transitions.py + committed aggregated CSVs

UKHLS pipeline detail

• datasets/ukhls.py reads the UKDS-licensed .dta files from storage/ukhls/ (git-ignored; raw microdata never leaves the machine).
• utils/ukhls_transitions.py pairs consecutive waves on pidp (cross-wave person key) and estimates:
  • P(state_{t+1} | state_t, age_band, sex) — four-state (IN_WORK / UNEMPLOYED / RETIRED / INACTIVE)
  • P(decile_{t+1} | decile_t, age_band, sex) — within-wave income deciles
• Cells with fewer than 10 observations are dropped (ONS Safe Setting convention). Probabilities re-normalised within surviving (age_band, sex, state_from) row groups.
• Aggregated outputs committed to storage/:
  • ukhls_employment_state_transitions.csv (≈16 KB)
  • ukhls_income_decile_transitions.csv (≈94 KB)
• Spot-check validation vs published statistics:
  • MALE 25–29 IN_WORK → IN_WORK = 95.6 % (matches ONS LFS 2-quarter flow rate).
  • MALE 25–29 UNEMPLOYED → IN_WORK = 36.5 % (consistent with ONS published ~35–40 %).
  • Avg probability of staying in the same income decile year-on-year = 39.9 % (consistent with IFS Living Standards mobility estimates).

Data-protection safeguards

• **/*.dta and storage/ukhls/ are gitignored (both directory and symlink forms).
• Only aggregated, cell-suppressed transition tables are committed.
• storage/upload_yearly_snapshots.py locks the private HF repo + GCS bucket constants at function level; redirecting the upload requires a reviewable code edit.

---

What's not in this PR (explicit non-goals)

• create_datasets.py:main() refactor to invoke create_yearly_snapshots automatically — kept as a follow-up so the change-of-default conversation happens in its own PR.
• Per-year calibration against year-specific targets — plumbing is here (resolve_target_value, smoothness penalty), but the full year-loop wiring belongs in a separate step.
• Smoothness-penalty coefficient tuning — empirical question best answered against full loss matrices, deferred.
• policyengine-uk changes — runtime-uprating skip flags, fixture migration — documented in docs/panel_downstream.md, left to the sibling repo.
• Same-sex marriage, cohabitation, second-marriage modelling — deferred.
• Admin-linked data (DWP Safe Access UKHLS variants) — require Secure Access, not a practical next step.
• End-to-end smoke test on the real enhanced_frs_2023_24.h5 — 176 in-tree tests cover every unit and a composed advance_year run, but the full-panel dry-run through create_yearly_snapshots is a follow-up I'll attach before moving this out of Draft.

Test plan

176 new tests across the stack, all green locally; no real FRS or UKHLS microdata touched in CI (every test uses in-memory synthetic fixtures or committed aggregate CSVs).

test_advance_year.py                          7
test_age_dataset_sex_specific_mortality.py    5
test_apply_children_leaving_home.py           8
test_apply_employment_transitions.py         14
test_apply_income_decile_transitions.py       7
test_apply_marriages.py                       9
test_apply_migration.py                       9
test_apply_separations.py                     8
test_calibrate_smoothness_integration.py      5
test_conftest_fixtures.py                     7
test_demographic_ageing.py                   23
test_ons_fertility.py                         9
test_ons_mortality.py                         7
test_panel_ids.py                            10
test_resolve_target_value.py                 12
test_smoothness_penalty.py                   10
test_ukhls_transitions.py                     9
test_upload_yearly_snapshots.py               7
test_yearly_snapshots.py                     10

Notable coverage

• Upload safety: the private HF repo, HF repo type and GCS bucket are locked by test; the function signature is asserted to only allow years and storage_folder as kwargs.
• No-partial-upload invariant: a missing file aborts the upload call before any network traffic.
• Smoothness regulariser: gradient masking on zero-prior entries, log-space symmetry, large-penalty integration test that proves pull-towards-prior.
• Panel-ID contract: survivors, deaths, and births classified correctly across a single age_dataset run.
• Disclosure control: UKHLS transition tests build a tiny cohort and confirm every low-count cell is suppressed before the CSV is written.
• UKHLS integration: hermetic synthetic panel recovers planted transition probabilities within Monte-Carlo noise, plus a @pytest.mark.skipif path that exercises the committed aggregate CSVs when present.
• Determinism: every stochastic transition has a same-seed → same-output test.

Running a one-year panel roll

from policyengine_uk_data.utils.advance_year import advance_year
from policyengine_uk_data.utils.ukhls_transitions import (
    load_employment_transitions, load_income_decile_transitions,
)
from policyengine_uk_data.targets.sources.ons_mortality import get_mortality_rates
from policyengine_uk_data.targets.sources.ons_fertility import get_fertility_rates

out = advance_year(
    base,
    target_year=2024,
    seed=0,
    mortality_rates=get_mortality_rates(2024),
    fertility_rates=get_fertility_rates(2024),
    ukhls_employment_rates=load_employment_transitions(),
    ukhls_decile_rates=load_income_decile_transitions(),
)

Tracks and closes: #345.

[vahid-ahmadi]

Follow-up PRs (explicitly not this one)

5. Wire create_yearly_snapshots(mode="full_panel") into create_datasets.py:main() — the default build starts producing enhanced_frs_<year>.h5 for every year in a declared range. Kept separate per the PR body so the change-of-default conversation happens in its own PR.

6. Per-year calibration loop with smoothness penalty on — run calibrate_local_areas on each produced snapshot, passing the prior year's final weights as prior_weights and a tuned smoothness_penalty coefficient. Plumbing is here; the orchestration is not.

7. Tune the smoothness-penalty coefficient — small sweep on real loss matrices (the empirical piece the PR body explicitly defers).

8. Validation harness — script that compares panel-rolled output year-on-year aggregates against DWP Income Dynamics published transition tables + IFS income mobility figures. Signals when any transition rate is drifting from published statistics.

9. policyengine-uk sibling-repo changes — runtime-uprating skip options and fixture migration, per docs/panel_downstream.md.

10. LFS 5-quarter rotation panel as a cross-check source — UKHLS covers the long horizon; LFS gives a richer short-horizon view of employment-state transitions. Useful as an independent validator of the employment-state matrix in this PR.