PolicyEngine-UK 2.92.2__tar.gz → 2.93.1__tar.gz

{policyengine_uk-2.92.2 → policyengine_uk-2.93.1}/CHANGELOG.md

@@ -1,3 +1,19 @@
+## [2.93.1] - 2026-08-28
+
+### Fixed
+
+- Place earnings quintiles against thresholds taken from the observed earnings distribution of working adults, rather than ranking the whole population on actual earnings. Ranking everyone, children included, left the bottom two quintiles entirely without earners, placed every potential labour-market entrant at the steep end of the OBR Table A1 elasticities, and left those quintiles with no employed donors — so `impute_wages_for_nonworkers` returned a wage of zero for them, which silently bars entry into employment. Wage donors are now grouped by sex and age band, which also breaks the circular dependency between the two functions.
+
+  Non-workers are placed on the quintile table at full-time-equivalent income. This is an assumption beyond the cited sources, and it is material: unscaled 18.8-hour placement moves entrants to quintile 1 and roughly doubles the elasticity they draw. Documented in the module with the bound and how to reproduce it.
+
+
+## [2.93.0] - 2026-08-28
+
+### Added
+
+- - Add `tax_free_childcare_spend_routed_share`, a per-child input for the share of childcare spending paid through a Tax-Free Childcare account. Only routed spending attracts the top-up, and `childcare_expenses` is annual. The share is measured across the eligible period, not the whole year, because `tax_free_childcare` already prorates annual spending by the eligible fraction and a whole-year share would discount the same months twice. Defaults to 1 — a neutral all-spend-routed assumption rather than a statutory requirement — and is clipped to 0-1, so no household calculation changes. The dataset build may supply an HMRC-derived account-activity duration proxy rather than an observed routed-expenditure share; see policyengine-uk-data.
+
+
 ## [2.92.2] - 2026-08-28
 
 ### Fixed

{policyengine_uk-2.92.2 → policyengine_uk-2.93.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: policyengine-uk
-Version: 2.92.2
+Version: 2.93.1
 Summary: PolicyEngine tax and benefit system for the UK.
 Project-URL: Homepage, https://github.com/PolicyEngine/policyengine-uk
 Project-URL: Repository, https://github.com/PolicyEngine/policyengine-uk

{policyengine_uk-2.92.2 → policyengine_uk-2.93.1}/policyengine_uk/dynamics/participation.py

@@ -10,9 +10,14 @@ Reference: https://obr.uk/docs/dlm_uploads/NICS-Cut-Impact-on-Labour-Supply-Note
 import numpy as np
 import pandas as pd
 from policyengine_uk import Simulation
-from policyengine_uk.model_api import WEEKS_IN_YEAR
 import warnings
 
+from policyengine_uk.model_api import WEEKS_IN_YEAR
+
+# Weekly hours treated as full time when putting non-workers on a comparable
+# footing with observed earnings for quintile placement.
+FULL_TIME_WEEKLY_HOURS = 37.5
+
 
 def calculate_participation_elasticities(
     sim: Simulation,
@@ -223,10 +228,22 @@ def impute_wages_for_nonworkers(
     year: int = 2025,
     hours_for_new_entrants: float = 18.8,
 ) -> np.ndarray:
-    """Impute wages for non-workers based on their elasticity group.
+    """Impute annual earnings for non-workers entering employment.
+
+    Assumes a non-worker would work ``hours_for_new_entrants`` hours a week at
+    the median hourly wage of workers of the same sex and age band.
+
+    Donors are grouped by sex and age band rather than by elasticity group.
+    The elasticity group is keyed off :func:`calculate_earnings_quintile`, and
+    quintiles of *potential* earnings cannot be computed until non-workers have
+    an imputed wage — so grouping donors that way is circular. In practice it
+    left the lowest quintiles with no employed donors at all and their
+    non-workers with an imputed wage of zero, and a zero imputed wage silently
+    bars someone from entering employment in
+    :func:`apply_participation_responses`, whatever the reform.
 
-    Assumes non-workers would work 18.8 hours per week at the average wage
-    for their specific elasticity group.
+    The median is used rather than the mean because a small share of workers
+    have implausibly low implied hourly wages.
 
     Args:
         sim: PolicyEngine simulation object
@@ -237,30 +254,58 @@ def impute_wages_for_nonworkers(
         Array of imputed annual employment income for non-workers
     """
     employment_income, hourly_wages, working_mask = hourly_wage(sim, year)
-
-    # Get elasticity groups for wage calculation
-    earnings_quintile = calculate_earnings_quintile(
-        sim, year, hours_for_new_entrants, random_seed=42
+    age = np.asarray(sim.calculate("age", year), dtype=float)
+    gender = np.asarray(sim.calculate("gender", year))
+    weights = np.asarray(
+        sim.calculate("household_weight", year, map_to="person"), dtype=float
     )
-    elasticities = calculate_participation_elasticities(sim, earnings_quintile)
 
-    # Create elasticity bins for grouping
-    unique_elasticities = np.unique(elasticities[elasticities > 0])
+    age_band = np.clip(age // 10 * 10, 10, 60)
+    overall_median = weighted_median(hourly_wages[working_mask], weights[working_mask])
 
-    # Calculate average wage by elasticity group
     imputed_wages = np.zeros_like(employment_income, dtype=float)
-
-    for elasticity_val in unique_elasticities:
-        elasticity_mask = (elasticities == elasticity_val) & working_mask
-        if elasticity_mask.any():
-            avg_hourly_wage = np.mean(hourly_wages[elasticity_mask])
-            # Apply to all non-workers in this elasticity group
-            nonworker_mask = (elasticities == elasticity_val) & ~working_mask
-            imputed_wages[nonworker_mask] = (
-                avg_hourly_wage * hours_for_new_entrants * 52
-            )
-
-    return imputed_wages
+    nonworker_mask = ~working_mask
+    for sex in np.unique(gender):
+        for band in np.unique(age_band):
+            group = (gender == sex) & (age_band == band)
+            donors = group & working_mask
+            median_wage = weighted_median(hourly_wages[donors], weights[donors])
+            if median_wage <= 0:
+                # No donors, or donors carrying no weight. Either way the
+                # group median is unusable and the overall one stands in; a
+                # zero here would silently bar entry into employment.
+                median_wage = overall_median
+            imputed_wages[group & nonworker_mask] = median_wage
+
+    return imputed_wages * hours_for_new_entrants * WEEKS_IN_YEAR
+
+
+def weighted_median(values: np.ndarray, weights: np.ndarray) -> float:
+    """Weighted median of ``values``.
+
+    Falls back to the unweighted median where the group carries no weight, so
+    that a zero-weight donor group still yields a wage. Returning zero there
+    would silently bar the people it is imputed for from entering employment,
+    since :func:`apply_participation_responses` gates entry on a positive
+    imputed wage. Only a genuinely empty group returns zero.
+    """
+    values = np.asarray(values, dtype=float)
+    weights = np.asarray(weights, dtype=float)
+    if values.shape != weights.shape:
+        raise ValueError(
+            f"weighted_median got {values.shape} values and {weights.shape} weights"
+        )
+    # np.argsort sorts NaN last, so a single NaN would otherwise be returned as
+    # the group maximum rather than flagged. Drop non-finite pairs instead.
+    usable = np.isfinite(values) & np.isfinite(weights)
+    values, weights = values[usable], weights[usable]
+    if not len(values):
+        return 0.0
+    order = np.argsort(values)
+    cumulative = np.cumsum(weights[order])
+    if cumulative[-1] <= 0:
+        return float(np.median(values))
+    return float(values[order][np.searchsorted(cumulative, cumulative[-1] / 2)])
 
 
 def calculate_gain_to_work(
@@ -324,11 +369,16 @@ def calculate_gain_to_work(
                 "household_net_income", year, map_to="person"
             )[is_adult_i]
 
-            # Calculate in-work income (use imputed wages if applicable)
+            # Calculate in-work income (use imputed wages if applicable).
+            # employment_income is stored as float32; imputed wages are float64,
+            # and assigning them raises under pandas 3 rather than down-casting.
+            # set_input down-casts on storage anyway, so float32 is already the
+            # model's precision and casting here just moves the truncation
+            # earlier (max relative error about 5e-08).
             temp_employment_in = employment_income.copy()
             temp_employment_in[is_adult_i] = employment_income_with_imputation[
                 is_adult_i
-            ]
+            ].astype(temp_employment_in.dtype)
 
             sim.reset_calculations()
             sim.set_input("employment_income", year, temp_employment_in)
@@ -352,42 +402,154 @@ def calculate_gain_to_work(
     )
 
 
+def weighted_quantiles(
+    values: np.ndarray, weights: np.ndarray, quantiles: list[float]
+) -> np.ndarray:
+    """Weighted quantiles of ``values``, for use as distribution thresholds."""
+    if not len(values):
+        return np.zeros(len(quantiles))
+    # Aggregate weight by *distinct* value before building the CDF. Sorting
+    # rows and taking each row's own weight midpoint splits a tied value into
+    # as many positions as it has rows, and `np.argsort` orders those rows by
+    # input position — so two equal-earning workers carrying unequal weights
+    # produced different thresholds depending only on which came first in the
+    # file, and could land either side of a boundary. One value must have one
+    # position on the distribution however many rows carry it.
+    sorted_values, inverse = np.unique(values, return_inverse=True)
+    sorted_weights = np.bincount(inverse, weights=weights, minlength=len(sorted_values))
+    cumulative = np.cumsum(sorted_weights)
+    total = cumulative[-1]
+    if not np.isfinite(total) or total <= 0:
+        # No usable weights. Returning zeros would put every positive value in
+        # the top quintile — the lowest-elasticity end of the OBR table — so
+        # fall back to unweighted quantiles of the values themselves.
+        return np.quantile(values, quantiles)
+    # Midpoint of each value's total weight, so a heavy point mass does not
+    # pull a threshold to one of its edges.
+    position = (cumulative - sorted_weights / 2) / total
+    return np.interp(quantiles, position, sorted_values)
+
+
 def calculate_earnings_quintile(
     sim: Simulation,
     year: int = 2025,
     hours_for_new_entrants: float = 18.8,
     random_seed: int = 42,
 ) -> np.ndarray:
-    """Calculate earnings quintile for each person based on potential earnings.
-
-    For workers, uses actual earnings. For non-workers, uses imputed potential earnings.
+    """Calculate earnings quintile for each adult based on potential earnings.
+
+    Thresholds are the weighted quintile boundaries of the **observed earnings
+    distribution of working adults**. Everyone is then placed against those
+    thresholds: workers on their actual earnings, non-workers on the potential
+    earnings imputed by :func:`impute_wages_for_nonworkers`.
+
+    Two properties matter for the OBR Table A1 elasticities this feeds.
+
+    * **The distribution is of adult earnings.** Ranking every person in the
+      dataset put roughly a quarter of each lower quintile into children, and
+      left the bottom two quintiles entirely without earners — so the
+      elasticity table was indexed by a distribution that contained no wages
+      at its lower end.
+    * **Placement is by value against a fixed threshold, not by rank within a
+      pooled ranking.** Every non-worker in a donor group shares one imputed
+      wage, so potential earnings carry large point masses. Ranking a pooled
+      distribution splits such a mass across a quintile boundary, and which
+      member of the mass falls on which side then depends on the order of rows
+      in the dataset — making elasticities and stochastic responses vary with
+      serialisation. Thresholds avoid this: a tied mass lands wholly in the
+      quintile its value belongs to.
+
+    Because non-workers are placed rather than ranked, quintiles are not equal
+    shares of the adult population. That is intended — they are quintiles of
+    the earnings distribution, which is what Table A1 is indexed on.
 
     Args:
         sim: PolicyEngine simulation object
         year: Year for calculation
         hours_for_new_entrants: Weekly hours assumed for new entrants
-        random_seed: Seed for random number generation
+        random_seed: Unused. Placement is deterministic; retained so existing
+            callers do not break.
 
     Returns:
-        Array of quintiles (1-5) for each person
+        Array of quintiles (1-5) for each person. Non-adults are assigned 1;
+        they are excluded from labour supply responses elsewhere.
     """
-    employment_income = sim.calculate("employment_income", year)
-
-    # Calculate quintiles
-    # Use pandas qcut for equal-sized bins
-    # Add random noise to avoid ties in quintile calculation
-    rng = np.random.RandomState(random_seed)
-    quintiles = pd.qcut(
-        employment_income + rng.random(employment_income.shape),
-        q=5,
-        labels=[1, 2, 3, 4, 5],
-        duplicates="drop",
+    employment_income = np.asarray(
+        sim.calculate("employment_income", year), dtype=float
+    )
+    adult_index = np.asarray(sim.calculate("adult_index", year), dtype=float)
+    weights = np.asarray(
+        sim.calculate("household_weight", year, map_to="person"), dtype=float
+    )
+    # Imputed earnings are for an entrant working `hours_for_new_entrants` a
+    # week, but the thresholds come from observed earnings at whatever hours
+    # workers actually do — mostly full time. Placing a part-time figure against
+    # a mostly-full-time distribution is not a like-for-like comparison, and it
+    # pushes every non-worker down by roughly the hours ratio: at 18.8 hours the
+    # imputed values span a single threshold interval. Scaling to full-time
+    # equivalent for placement — the entrant's actual earnings stay at the
+    # assumed hours — moves them off that single interval.
+    #
+    # It does not spread them across the table. Because every non-worker
+    # inherits one of about a dozen sex-and-age-band medians, the placement is
+    # still coarse: on enhanced_frs_2024_25 they occupy quintiles 2 to 4 only, with
+    # no entrant placed at either end, so the Table A1 gradient is only
+    # partially exercised for exactly the
+    # population whose entry this module models. Fixing that needs a finer
+    # imputation, not a different placement rule.
+    #
+    # This scaling is an assumption, not something the sources state. The OBR
+    # note assumes entrants work 18.8 hours a week and labels Table A1 by
+    # position in the earnings distribution, without saying which basis places
+    # a non-worker on it; Adam and Phillips (2013) do not allocate elasticities
+    # to non-workers at all. The choice is material and runs one way: placing
+    # entrants at their unscaled 18.8-hour income moves them down the table to
+    # quintile 1, roughly *doubling* the elasticity each demographic row draws
+    # and so the modelled entry response. Callers wanting that bound can pass
+    # `hours_for_new_entrants=FULL_TIME_WEEKLY_HOURS` to disable the scaling.
+    # Neither end is validated against an observed entry rate — see #1836.
+    imputed = impute_wages_for_nonworkers(sim, year, hours_for_new_entrants)
+    full_time_equivalent = imputed * (FULL_TIME_WEEKLY_HOURS / hours_for_new_entrants)
+    potential = np.where(employment_income > 0, employment_income, full_time_equivalent)
+
+    adults = adult_index > 0
+    workers = adults & (employment_income > 0)
+    thresholds = weighted_quantiles(
+        employment_income[workers], weights[workers], [0.2, 0.4, 0.6, 0.8]
     )
 
-    if quintiles is not None:
-        return quintiles.astype(int)
-    else:
-        return np.ones(employment_income.shape, dtype=int)
+    quintile = np.ones(len(potential), dtype=int)
+    if len(np.unique(thresholds)) == 1:
+        # No dispersion to place anyone against — a population with no workers
+        # leaves every threshold at zero, and `searchsorted(side="right")` would
+        # then put every adult in the top quintile, the end of the OBR table
+        # with the *smallest* elasticities. That is the opposite of what a
+        # population of non-workers implies. With no earnings distribution to
+        # read, spread adults evenly rather than assert one.
+        index = np.flatnonzero(adults)
+        if len(index):
+            adult_potential = potential[index]
+            adult_weights = weights[index]
+            total = adult_weights.sum()
+            if total > 0:
+                # Place each *distinct* potential-earnings value at the midpoint
+                # of its own block of weight, not each row at the midpoint of
+                # its own weight. Spreading row by row would sort ties in input
+                # order, so two identical records would land in different
+                # quintiles purely because of serialisation — the invariant this
+                # function documents. Blocking on the value makes placement a
+                # function of the data alone.
+                values, inverse = np.unique(adult_potential, return_inverse=True)
+                block = np.bincount(inverse, weights=adult_weights)
+                below = np.concatenate([[0.0], np.cumsum(block)[:-1]])
+                position = (below + block / 2) / total
+                quintile[index] = np.clip((position[inverse] * 5).astype(int) + 1, 1, 5)
+            else:
+                quintile[index] = 3
+        return quintile
+
+    quintile[adults] = np.searchsorted(thresholds, potential[adults], side="right") + 1
+    return np.clip(quintile, 1, 5)
 
 
 def apply_participation_responses(
@@ -547,7 +709,10 @@ def apply_participation_responses(
             0, participation_change[i]
         )  # Only consider positive changes
         if np.random.random() < entry_probability and imputed_wages[i] > 0:
-            new_employment_income[i] = imputed_wages[i]
+            # Cast for the same reason as the in-work assignment above.
+            new_employment_income[i] = new_employment_income.dtype.type(
+                imputed_wages[i]
+            )
             participation_response[i] = True  # Entered work
 
     # Update simulation with new employment incomes

{policyengine_uk-2.92.2 → policyengine_uk-2.93.1}/policyengine_uk/tests/policy/baseline/gov/hmrc/tax_free_childcare/tax_free_childcare.yaml

@@ -165,3 +165,124 @@
     childcare_expenses: 8_000
   output:
     tax_free_childcare: 0
+
+# Only childcare spending routed through the account attracts a top-up. The
+# share defaults to 1 — a neutral all-spend-routed assumption — so a household
+# calculation returns the statutory amount. The dataset build sets it from
+# observed account use; see policyengine-uk-data.
+- name: Routing share defaults to the full statutory top-up
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    is_disabled_for_benefits: false
+    childcare_expenses: 2_000
+  output:
+    # 2_000 * 0.2
+    tax_free_childcare: 400
+
+- name: Routing half of childcare spending halves the top-up
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_spend_routed_share: 0.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 2_000
+  output:
+    # 2_000 * 0.5 * 0.2
+    tax_free_childcare: 200
+
+- name: Routing share does not lift a capped award
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_spend_routed_share: 0.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 40_000
+  output:
+    # 40_000 * 0.5 * 0.2 = 4_000, still above the £2,000 cap
+    tax_free_childcare: 2_000
+
+- name: Routing no spending through the account gives no top-up
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_spend_routed_share: 0
+    is_disabled_for_benefits: false
+    childcare_expenses: 8_000
+  output:
+    tax_free_childcare: 0
+
+- name: A routing share above one is clipped to the statutory amount
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_spend_routed_share: 1.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 2_000
+  output:
+    tax_free_childcare: 400
+
+- name: A negative routing share is clipped to zero
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_spend_routed_share: -0.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 2_000
+  output:
+    tax_free_childcare: 0
+
+- name: Siblings can route different shares
+  period: 2025
+  input:
+    people:
+      child_1:
+        tax_free_childcare_qualifying_child: true
+        tax_free_childcare_spend_routed_share: 1
+        childcare_expenses: 2_000
+      child_2:
+        tax_free_childcare_qualifying_child: true
+        tax_free_childcare_spend_routed_share: 0.5
+        childcare_expenses: 2_000
+    benunits:
+      benunit:
+        members: [child_1, child_2]
+        tax_free_childcare_eligible: true
+  output:
+    tax_free_childcare: [400, 200]
+
+# The routed share is measured across the eligible period, not the whole year.
+# tax_free_childcare already prorates annual spend by the eligible fraction, so
+# a whole-year share would discount the same months twice.
+- name: Part-year eligibility and routed share compose without double-discounting
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_eligible_declaration_periods: 2
+    tax_free_childcare_spend_routed_share: 0.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 12_000
+  output:
+    # Eligible half the year, routing half of that period's spend: the top-up
+    # base is 12_000 * 0.5 * 0.5 = 3_000, and 20% of that is 600. Under the
+    # cap, which the two eligible periods halve to 1_000.
+    tax_free_childcare: 600
+
+- name: Full-year eligibility leaves the routed share as the only scaler
+  period: 2025
+  input:
+    tax_free_childcare_eligible: true
+    tax_free_childcare_qualifying_child: true
+    tax_free_childcare_eligible_declaration_periods: 4
+    tax_free_childcare_spend_routed_share: 0.5
+    is_disabled_for_benefits: false
+    childcare_expenses: 12_000
+  output:
+    tax_free_childcare: 1_200