ltc-code 0.2.31__tar.gz → 0.2.33__tar.gz

ltc_code-0.2.33/PKG-INFO

@@ -0,0 +1,32 @@
+Metadata-Version: 2.3
+Name: ltc-code
+Version: 0.2.33
+Summary: Add your description here
+Requires-Dist: fastexcel>=0.16,<0.20
+Requires-Dist: polars>=1.36.1,<1.42
+Requires-Dist: polars-readstat>=0.20.2
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+
+
+### College scatter plots (0.2.32)
+
+`from ltc_code.plot_colleges import plot_colleges` provides an OI-themed
+scatter with explicit Polars `x`/`y` aggregations, college grouping, optional
+filters and categorical color groups. Plotting additionally requires
+`oi-tools`, `plotnine`, and `pyarrow` in the analysis environment; they are
+not required to run the NSC build. See [NSC extensions](src/ltc_code/nsc/EXTENSIONS.md)
+for attendance definitions, coarsening names, and plotting examples.
+
+
+### Paper exhibits (0.2.33)
+
+The complete standalone LaTeX PDF workflow is bundled under
+`src/ltc_code/paper` (installed as `ltc_code/paper`). It includes the
+fixed disclosed and replaceable pending Excel workbooks, Python scripts,
+LaTeX templates, and table/figure exclusion controls. See
+[paper instructions](src/ltc_code/paper/README.md). Copy that whole directory
+to a writable working location and run `bash run_latex.sh`. The paper
+workflow uses its own pinned Python dependencies and requires LaTeX;
+the existing package dependencies are unchanged.

ltc_code-0.2.33/README.md

@@ -0,0 +1,22 @@
+
+
+### College scatter plots (0.2.32)
+
+`from ltc_code.plot_colleges import plot_colleges` provides an OI-themed
+scatter with explicit Polars `x`/`y` aggregations, college grouping, optional
+filters and categorical color groups. Plotting additionally requires
+`oi-tools`, `plotnine`, and `pyarrow` in the analysis environment; they are
+not required to run the NSC build. See [NSC extensions](src/ltc_code/nsc/EXTENSIONS.md)
+for attendance definitions, coarsening names, and plotting examples.
+
+
+### Paper exhibits (0.2.33)
+
+The complete standalone LaTeX PDF workflow is bundled under
+`src/ltc_code/paper` (installed as `ltc_code/paper`). It includes the
+fixed disclosed and replaceable pending Excel workbooks, Python scripts,
+LaTeX templates, and table/figure exclusion controls. See
+[paper instructions](src/ltc_code/paper/README.md). Copy that whole directory
+to a writable working location and run `bash run_latex.sh`. The paper
+workflow uses its own pinned Python dependencies and requires LaTeX;
+the existing package dependencies are unchanged.

{ltc_code-0.2.31 → ltc_code-0.2.33}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ltc-code"
-version = "0.2.31"
+version = "0.2.33"
 description = "Add your description here"
 readme = "README.md"
 requires-python = ">=3.9"

ltc_code-0.2.33/src/ltc_code/nsc/EXTENSIONS.md

@@ -0,0 +1,102 @@
+# NSC attendance and coarsening changes in the sandbox
+
+The package build is `ltc_code/nsc/build_nsc_outcomes_new.py`.
+Import the scatter helper with `from ltc_code.plot_colleges import plot_colleges`.
+
+Version 0.2.32 changes `coarse_t12`, `coarse_t34`, and `coarse_t58` to the
+overall four-year benchmark; their former within-group calculations are
+retained as `coarse_t12_pool`, `coarse_t34_pool`, and `coarse_t58_pool`.
+
+## Two attendance measures
+
+For each age 18–26 and each grouping `any`, `4yr`, `2yr`:
+
+| Example | Counts |
+| --- | --- |
+| `att_any_1098_20` | F, Q, H, plus unknown statuses under the existing policy |
+| `att_any_1098_all_20` | The same, plus L (less than half-time) |
+
+Both versions exclude W (withdrawn), A (leave), D (deceased), and other
+non-enrollment statuses. The existing handling of generic part-time text is
+retained. Q now qualifies, and written-out "less than half-time" is classified
+before matching "half-time". Adjacent spells with different eligibility are
+kept separate rather than assigning the highest status to the entire period.
+
+The original names are retained. There are no duplicate `_fulltime_` columns.
+
+Both measures use the available tax years 2011–2014 and 2016–2022. Observable
+years without counted attendance get zero; unavailable years remain missing.
+Four-year attendance takes priority separately within each version. These
+measures retain the existing NSC matching, record deduplication, and weekly
+institution selection; they cannot recover records missing from the source.
+
+## Coarsening: preserved and additional variables
+
+Names without `_pool` use the overall four-year benchmark only for the named
+tiers. Names ending in `_pool` use each named group's own pooled rate.
+All names below start with `adj_cmp_rate_4yr_coarse_`:
+
+| Suffix | Calculation |
+| --- | --- |
+| `t12` | Replace only tiers 1–2 with the overall four-year average |
+| `t34` | Replace only tiers 3–4 with the overall four-year average |
+| `t58` | Replace only tiers 5–8 with the overall four-year average |
+| `t12_pool` | Replace tiers 1–2 with their own pooled average |
+| `t34_pool` | Replace tiers 3–4 with their own pooled average |
+| `t58_pool` | Replace tiers 5–8 with their own pooled average |
+| `t12_t34_pool` | Pool 1–2 and, separately, 3–4 |
+| `t14_t58_pool` | Pool 1–4 and, separately, 5–8 |
+
+Every average uses total IPEDS completers divided by the total adjusted
+graduation cohort in the relevant group. Non-target colleges keep the detailed
+four-year prediction. Existing zero and missing-value rules are preserved.
+The separate `adj_cmp_rate_coarse_4yr` still replaces ALL four-year starters'
+rates with the common four-year average.
+
+## Simple college scatter
+
+```python
+plot = plot_colleges(
+    df,
+    x=pl.col("sid_cepr").drop_nulls().n_unique(),
+    y=pl.col("completion_rate_150pct_firstinst").drop_nulls().first() * 100,
+    group_by="unitid_firstinst",
+    filters={"enrolled": 1, "tier_firstinst": [3, 4]},
+    title="First colleges of charter enrollers: tiers 3–4",
+)
+plots.append(plot)
+```
+
+Replace `enrolled` with your charter-enrollment column. Omit `filters` for everyone,
+or supply just one dictionary entry. Polars expressions and lists of expressions
+also work for more general conditions. Filters apply before the x/y aggregation.
+The title and axis labels are customizable. x and y are Polars aggregation
+expressions; their choice determines what each college's dot represents.
+
+`unitid_firstinst` is the matched IPEDS ID of the first qualifying institution
+within the by-Y4 attendance window. `completion_rate_150pct_firstinst` is that
+institution's raw graduation rate. This example describes first colleges, not
+every college ever attended or the institution awarding the degree. Students
+without a matched first-college ID and colleges without a rate do not appear.
+Use a consistent institutional rate per college: `.first()` selects that rate,
+not an average of students' predicted completion outcomes.
+
+The helper returns a plotnine plot; `return_data=True` returns `(plot, data)`.
+It does not save files. Call `.save(...)` on the returned plot to export it.
+
+### OI theme and color groups
+
+The scatter uses `oi_tools.figures.theme_oi()` (the same theme as `plot_bars`),
+with no grid lines and clear black axes. The analysis environment needs
+`oi-tools`, `plotnine`, and `pyarrow`.
+
+Add `color="enrolled"` to draw separate dots for each college's enrollers and
+non-enrollers. The function aggregates x and y separately within each
+college/color group, after applying filters. Numeric codes become discrete
+legend labels. For descriptive labels, pass a column containing strings such
+as "Enrollers" and "Non-enrollers". Missing color values are labeled "Missing".
+The OI palette supplies the colors. Omit `color` for one dot per college.
+
+Because the institutional graduation rate is the same for both groups, their
+dots have the same y coordinate; x differs with the number of unique students.
+If both counts are identical, their dots overlap.

{ltc_code-0.2.31 → ltc_code-0.2.33}/src/ltc_code/nsc/build_nsc_outcomes_new.py

@@ -914,7 +914,7 @@ nsc = nsc.with_columns(
 ###########################################################
 
 # Sarah counts half-time or more as an attended enrollment spell.
-enrollment_status = pl.col("enrollment").str.to_uppercase().fill_null("")
+enrollment_status = pl.col("enrollment").str.to_uppercase().str.strip_chars().fill_null("")
 
 enroll = (
     nsc.filter(pl.col("graduated") == "N")
@@ -935,16 +935,19 @@ enroll = (
         .then(5)
         .when(enrollment_status.str.contains("PART", literal=False))
         .then(4)
+        .when(enrollment_status.str.contains("LESS|\\bL\\b", literal=False))
+        .then(1)
         .when(enrollment_status.str.contains("HALF|\\bH\\b", literal=False))
         .then(3)
         .when(enrollment_status.str.contains("QUARTER|\\bQ\\b", literal=False))
-        .then(2)
-        .when(enrollment_status.str.contains("LESS|\\bL\\b", literal=False))
-        .then(1)
+        .then(4)  # Three-quarter time qualifies as at least half-time.
         .otherwise(0)
         .alias("_enrollment_score")
     )
-    .with_columns((pl.col("_enrollment_score") >= 3).cast(pl.Int8).alias("_term_att"))
+    .with_columns(
+        (pl.col("_enrollment_score") >= 3).cast(pl.Int8).alias("_term_att"),
+        (pl.col("_enrollment_score") >= 1).cast(pl.Int8).alias("_term_att_all"),
+    )
     .with_columns(
         ((pl.col("term_start_date").cast(pl.Int64) // 7).cast(pl.Int64)).alias(
             "week_start"
@@ -1035,6 +1038,7 @@ enroll_weeks = (
         "week_start",
         "week_end",
         "_term_att",
+        "_term_att_all",
         "_enrollment_score",
         "_enrollment_stem_sarah",
         "_enrollment_stem_dhs",
@@ -1068,6 +1072,8 @@ enroll_weeks = (
             (pl.col("week") != pl.col("week").shift(1) + 1)
             | (pl.col("sid_cepr") != pl.col("sid_cepr").shift(1))
             | (pl.col("ID_FSC") != pl.col("ID_FSC").shift(1))
+            | (pl.col("_term_att") != pl.col("_term_att").shift(1))
+            | (pl.col("_term_att_all") != pl.col("_term_att_all").shift(1))
             | (
                 pl.col("_enrollment_stem_sarah")
                 != pl.col("_enrollment_stem_sarah").shift(1)
@@ -1102,6 +1108,7 @@ enroll = (
         pl.col("k_mean").drop_nulls().first(),
         pl.col("completion_rate_150pct_ip").drop_nulls().first(),
         pl.col("_term_att").max(),
+        pl.col("_term_att_all").max(),
         pl.col("_enrollment_stem_sarah").max(),
         pl.col("_enrollment_stem_dhs").max(),
         pl.col("week").min().alias("week_start"),
@@ -1216,10 +1223,11 @@ for age in AGE_ATTENDANCE_RANGE:
         (
             (pl.col("term_start_date") <= pl.col("_age_window_end"))
             & (pl.col("term_end_date") >= pl.col("_age_window_start"))
-            & (pl.col("_term_att") == 1)
         )
         .cast(pl.Int8)
-        .alias("_overlaps")
+        .alias("_overlaps_all")
+    ).with_columns(
+        (pl.col("_overlaps_all") * pl.col("_term_att")).alias("_overlaps")
     )
 
     enrollment_pieces.append(
@@ -1236,6 +1244,19 @@ for age in AGE_ATTENDANCE_RANGE:
                 .alias(f"att_{college_type}_{age}")
                 for college_type in AGE_ATTENDANCE_TYPES
             ]
+            + [
+                (
+                    pl.col("_overlaps_all")
+                    * pl.col("_term_att_all")
+                    * (
+                        pl.lit(1) if college_type == "any"
+                        else pl.col(f"college_{college_type}").fill_null(0).cast(pl.Int8)
+                    )
+                )
+                .max()
+                .alias(f"att_{college_type}_all_{age}")
+                for college_type in ["any", "4yr", "2yr"]
+            ]
             + [
                 expression
                 for definition in STEM_DEFINITIONS
@@ -1942,7 +1963,7 @@ ipeds_tier_completion = ipeds_completion_with_sector.join(
 ).filter(pl.col("college_sector") == "4yr")
 
 # Coarsen one bucket at a time, always starting from the detailed prediction.
-tier_buckets = {"t12": [1, 2], "t34": [3, 4], "t58": [5, 6, 7, 8]}
+tier_buckets = {"t12_pool": [1, 2], "t34_pool": [3, 4], "t58_pool": [5, 6, 7, 8]}
 for label, tiers in tier_buckets.items():
     bucket_rate = (
         ipeds_tier_completion.filter(pl.col("tier").is_in(tiers))
@@ -1964,18 +1985,66 @@ for label, tiers in tier_buckets.items():
         .alias(f"adj_cmp_rate_4yr_coarse_{label}")
     )
 
-# Match the available 1098-T calendar years, including the missing 2015 year.
-tax_years = list(range(2011, 2015)) + list(range(2016, 2023))
-nsc_outcomes = nsc_outcomes.with_columns(
-    [
-        pl.when((pl.col("cohort_lottery") + age).is_in(tax_years))
-        .then(pl.col(f"att_any_{age}"))
-        .otherwise(None)
-        .alias(f"att_any_1098_{age}")
-        for age in AGE_ATTENDANCE_RANGE
-    ]
+# Additional coarsening scenarios. Each listed group gets its own pooled
+# IPEDS rate; institutions outside those groups keep their detailed prediction.
+tier_coarsening_scenarios = {
+    "t12_t34_pool": [[1, 2], [3, 4]],
+    "t14_t58_pool": [[1, 2, 3, 4], [5, 6, 7, 8]],
+}
+for label, groups in tier_coarsening_scenarios.items():
+    prediction = pl.col("adj_cmp_rate_4yr")
+    for tiers in groups:
+        group_rate = (
+            ipeds_tier_completion.filter(pl.col("tier").is_in(tiers))
+            .select(pl.col("completers_150pct_ip").sum() / pl.col("cohort_adj_150pct_ip").sum())
+            .item()
+        )
+        if group_rate is None or not 0 <= group_rate <= 1:
+            raise ValueError(f"Invalid pooled completion rate for tiers {tiers}: {group_rate}")
+        prediction = pl.when(
+            (pl.col("college_years_firstinst") == 4)
+            & pl.col("tier_firstinst").is_in(tiers)
+            & (pl.col("att_any_byY4") == 1)
+            & pl.col("adj_cmp_rate_4yr").is_not_null()
+        ).then(group_rate).otherwise(prediction)
+    nsc_outcomes = nsc_outcomes.with_columns(
+        prediction.alias(f"adj_cmp_rate_4yr_coarse_{label}")
+    )
 
-)
+# Replace only the named tiers with the common overall four-year average.
+# The _pool versions above instead use each group's own IPEDS average.
+benchmark_tier_buckets = {"t12": [1, 2], "t34": [3, 4], "t58": [5, 6, 7, 8]}
+for label, tiers in benchmark_tier_buckets.items():
+    nsc_outcomes = nsc_outcomes.with_columns(
+        pl.when(
+            (pl.col("college_years_firstinst") == 4)
+            & pl.col("tier_firstinst").is_in(tiers)
+            & (pl.col("att_any_byY4") == 1)
+            & pl.col("adj_cmp_rate_4yr").is_not_null()
+        ).then(cmp_rate_4yr_coarse).otherwise(pl.col("adj_cmp_rate_4yr"))
+        .alias(f"adj_cmp_rate_4yr_coarse_{label}")
+    )
+
+# Match available 1098-T calendar years, including the missing 2015 year.
+# Existing names retain AT LEAST HALF-TIME (including Q), with the existing
+# unknown-status policy. "all" adds less-than-half-time enrollment, excluding W/A/D and
+# other non-enrollment statuses. Unknown statuses count in both versions.
+tax_years = list(range(2011, 2015)) + list(range(2016, 2023))
+tax_attendance_columns = []
+for college_type in ["any", "4yr", "2yr"]:
+    for age in AGE_ATTENDANCE_RANGE:
+        for label, source in [
+            ("", f"att_{college_type}_{age}"),
+            ("_all", f"att_{college_type}_all_{age}"),
+        ]:
+            column = f"att_{college_type}_1098{label}_{age}"
+            tax_attendance_columns.append(column)
+            nsc_outcomes = nsc_outcomes.with_columns(
+                pl.when((pl.col("cohort_lottery") + age).is_in(tax_years))
+                .then(pl.col(source))
+                .otherwise(None)
+                .alias(column)
+            )
 
 
 ###########################################################
@@ -2066,6 +2135,8 @@ keep_columns = [
     "adj_cmp_rate_coarse_2yr",
     "adj_cmp_rate_coarse_4yr",
     *[f"adj_cmp_rate_4yr_coarse_{label}" for label in tier_buckets],
+    *[f"adj_cmp_rate_4yr_coarse_{label}" for label in tier_coarsening_scenarios],
+    *[f"adj_cmp_rate_4yr_coarse_{label}" for label in benchmark_tier_buckets],
     "adj_cmp_rate_coarse",
     "adj_cmp_rate_coarse_sample",
     "adj_cmp_rate_coarsen_4yr",
@@ -2076,7 +2147,7 @@ keep_columns = [
     "college_years_firstinst",
     "tier_firstinst",
     "completion_rate_150pct_firstinst",
-] + outcome_columns + stem_outcome_columns + selectivity_outcome_columns
+] + outcome_columns + stem_outcome_columns + selectivity_outcome_columns + tax_attendance_columns
 keep_columns = list(dict.fromkeys(keep_columns))
 
 nsc_outcomes_final = nsc_outcomes.select(keep_columns)

{ltc_code-0.2.31 → ltc_code-0.2.33}/src/ltc_code/nsc/raw/directory.dta

@@ -1,4 +1,4 @@
 [diffend] Oversized file quarantined before diffing.
-name: ltc_code-0.2.31/src/ltc_code/nsc/raw/directory.dta
+name: ltc_code-0.2.33/src/ltc_code/nsc/raw/directory.dta
 size: 256195841 bytes
 sha256: 9fe2a62ad77a3a17f5c8f45df08cc2e8daeb46425eeb062a576353aa2bd463d1

ltc_code-0.2.33/src/ltc_code/paper/README.md

@@ -0,0 +1,114 @@
+# Charter exhibits: two Excel inputs to the original LaTeX PDF
+
+The build reads two independent workbooks. It produces the approved 19-page PDF, with the same tables, red/black numbers, figures, captions and notes.
+
+## Run in Census
+
+1. Keep `data/charter_exhibits/disclosed_results.xlsx` fixed.
+2. Replace `data/charter_exhibits/pending_results.xlsx` with your actual pending results, retaining its sheets, headers and record keys. You can instead change `PENDING_INPUT` in `build_latex.py` to your new file path.
+3. Set `synthetic` to `false` in the pending workbook's `metadata` tab after replacing its synthetic values and chart inputs. Keep `role=pending` and `schema_version=2`. This only removes the synthetic footer; **it never switches off the fixed workbook**.
+4. Run `bash run_latex.sh` in the extracted portable folder. Output: `output/charter_exhibits_latex.pdf`.
+
+In the sandbox, run `bash code/explore_codex/charter_exhibits/run_latex.sh` from the sandbox root. Input files are under `outputs/charter_exhibits/inputs/` and the PDF is under `outputs/charter_exhibits/`.
+
+Python dependencies are listed in `latex_requirements.txt`; the runner uses uv. LaTeX requires `latexmk` and `pdflatex` (TeX Live or MacTeX). With dependencies already installed, `python build_latex.py` also runs the build. No network, live Overleaf connection, or live charter repository is needed. The sandbox charter repository is not modified.
+
+## Exclude tables or figures
+
+Edit the two lists near the top of `build_latex.py`. They default to empty, so the complete deck still builds:
+
+```python
+EXCLUDE_TABLES = []
+EXCLUDE_FIGURES = []
+```
+
+For example:
+
+```python
+EXCLUDE_TABLES = ["Table A.4", "Table A.6"]
+EXCLUDE_FIGURES = ["Figure A.1"]
+```
+
+You may use either the printed label or its identifier below. Exclusions skip the complete exhibit, its generation, and inputs used only by that exhibit. Those unused rows or whole tabs may be absent from the workbooks. Shared inputs remain required by any included exhibit. Both workbooks and their `metadata` tabs remain required.
+
+Remaining tables and figures retain their original numbers (e.g., Table 4 stays Table 4); page numbers run consecutively. References in the unchanged draft notes retain the original exhibit numbers, even when referring to an omitted exhibit. Clearing both lists restores everything. Invalid names and excluding every exhibit produce a clear error. The audit JSON records the included and excluded exhibits. These switches control whole tables/figures, not sample columns within an exhibit.
+
+| Printed label | Identifier |
+| --- | --- |
+| Table 1 | `summary_statistics_table` |
+| Table 2 | `balance_table` |
+| Table 3 | `age25_earnings_table` |
+| Table 4 | `earnings_heterogeneity_table` |
+| Table 5 | `college_enrollment_completion_table` |
+| Table 6 | `college_test_score_mediation_table` |
+| Table 7 | `other_mediators_table` |
+| Figure A.1 | `appendix-credo-distributions` |
+| Figure A.2 | `appendix-college-by-age-placeholder` |
+| Figure A.3 | `appendix-cmo-leave-one-out` |
+| Figure A.4 | `appendix-earnings-by-age-firm-chars` |
+| Table A.1 | `earnings_robustness_table` |
+| Table A.2 | `earnings_threshold_table` |
+| Table A.3 | `college_data_comparison_table` |
+| Table A.4 | `college_heterogeneity_table` |
+| Table A.5 | `college_mediation_additional_specs_table` |
+| Table A.6 | `lottery_sample_selection_table` |
+| Table A.7 | `cmo_details_table` |
+| Table A.8 | `intervention_calculation_table` |
+
+## Fixed disclosed workbook
+
+`disclosed_results.xlsx` contains only existing manuscript values from synced Overleaf commit `6ebdd8c1a84bcfd72cabf3c85a6b2a986cc5c416`, using numeric Excel cells rather than display strings for estimates.
+
+| Tab | Input columns / purpose |
+| --- | --- |
+| `summary_stats` | `sample, variable, mean, sd, count, unit` |
+| `TOT` | `sample, depvar, spec, coef, se, ccm, N_ppl, unit, coef_stars` |
+| `ITT`, `first_stage` | Regression keys, `coef, se, control_mean, N_ppl, unit, coef_stars` |
+| `OLS` | Regression keys, `coef, se, comparison_mean, N_ppl, unit, coef_stars` |
+| `balance` | `sample, depvar, coef, se, control_mean, N_ppl, unit, coef_stars` |
+| `mediation_conversions` | `depvar, earnings_gain, unit`; exact conversion values on the manuscript's existing scale |
+| `interventions` | `program, estimated_effect, intervention, notes, years, unit`; source effects, durations and calculation notes |
+| `metadata` | Input role, version and source snapshot |
+
+Unreported sample sizes are blank. The summary table's `variable=n_ppl` rows store reported total people in `count`; no source sample sizes were invented or copied from synthetic data. Statistical stars supplied by the manuscript are retained. The previously corrected first-stage/ITT SE pairing is retained.
+
+## Pending workbook
+
+`pending_results.xlsx` contains the red table entries and synthetic chart data. It currently reproduces the approved demonstration. Replace its data with your actual estimates. No old synthetic workbook or hidden numerical fallback is used.
+
+- `summary_stats`, `TOT`, `OLS`, `balance`: the same keyed estimate format as the fixed file, for pending fields only.
+- `predictions`: `sample, depvar, spec, coef, predicted_gain, unit`. Supply precomputed lognormal threshold effects in `coef` and mediation predicted earnings in `predicted_gain`. These are direct inputs, not calculations from the demonstration's old synthetic conversion factors. The sample workbook preserves the exact red predictions you approved.
+- `sample_selection`: `step, applications, N_ppl`.
+- `cmo_details`: `cmo, geographic_area, grades, lottery_years`. Grade and year ranges are text.
+- `credo`: one CMO per row, with `math_effect`, `reading_effect`, `in_study` (0 or 1). Effects are in standard deviations.
+- `TOT_by_cmo`: network regression estimates, SEs and CCMs; supplies the leave-one-network-out plot.
+- `figure_reference`: the separately supplied pooled estimate used for the leave-one-network-out reference line. This preserves the approved synthetic chart independently of the fixed earnings table. Replace it with the reference estimate appropriate to your real chart.
+- Age profiles remain in `TOT`, with `sample=age_profile` and `spec=observed` or `ein_fes`.
+- `metadata`: keep `role=pending`, `schema_version=2`; update `synthetic` and `source` for real inputs.
+
+The pending file is a template for the fields that remain to be filled. **A blank field alongside a populated field can belong to the fixed workbook.** For example, a subgroup row may contain only its pending `ccm`, while `coef` and `se` are fixed in the disclosed file. Leave those fixed fields blank in the pending file. The loader rejects conflicting nonblank fields instead of silently overwriting them. It also rejects duplicate keys and missing required rows or columns. Rows can be sorted freely.
+
+## Units, missingness and stars
+
+- `unit=proportion`: store probabilities and probability effects on the 0-1 scale. For example, `0.056` displays as `5.6` percentage points. Means, SEs and CCMs use the same scale.
+- `unit=usd`: unscaled dollars. `percentile`: rank points. `years` and `year`: years. `count`: whole people or applications.
+- Threshold prediction inputs are percentage points (`3.3` means 3.3 pp); mediation prediction inputs are dollars (`834` means $834). The fixed mediation conversions retain the manuscript's exact source scale.
+- Keep unknown numerical results empty. Zero is a real result and displays as zero. Missing values display as dashes.
+- `coef_stars` accepts `*`, `**`, `***`, `none`, or `auto`. The supplied demo keeps the exact approved stars. When replacing estimates, supply your corresponding stars or set `auto` to calculate normal-approximation stars from `coef/se`. Table 6 deliberately displays its mediator coefficients without stars, as before.
+- `N_ppl` and summary counts are retained as inputs; they appear only where the original paper layout includes them. Blank fixed Ns do not borrow pending counts.
+
+## Code and verification
+
+- `build_latex.py`: two input paths, LaTeX document wrapper and compilation.
+- `split_inputs.py`: keyed input validation for included exhibits, fixed/pending field separation, formatting and table rendering.
+- `exhibit_selection.py`: exclusion names, source-block selection, original numbering and omitted-exhibit references.
+- `table_bindings.json`: stable sample/outcome/specification-to-table-cell mappings and formatting; no fitted numerical estimates.
+- `latex_figures.py`: workbook-driven charts, with the previously approved composition.
+- `latex_source/`: original manuscript LaTeX templates, captions and draft notes.
+- `verify_latex.py`: verifies row-order independence, rejects duplicate/overlapping inputs, and simulates a real pending replacement while checking all fixed cells remain unchanged.
+
+`verify_exclusions.py` additionally tests missing excluded input sheets, shared dependencies, table-only and figure-only decks, original numbering, and restoring the full deck.
+
+The build writes an audit JSON with both input paths and SHA-256 hashes, consumed keys, and layout hashes. The refactor was checked against the approved PDF: all 19 pages have identical text and the chart PNGs are byte-identical. Replacement testing leaves the fixed workbook byte-identical.
+
+Only the two XLSX files and templates are needed at runtime. The earlier single-workbook `overleaf_values` mechanism and prototype synthetic calculation scripts are obsolete and excluded from the portable package. Existing draft caption/figure inconsistencies remain as approved; this refactor changes input ownership and code structure, not the paper's substantive content.

ltc_code-0.2.33/src/ltc_code/paper/build_latex.py

@@ -0,0 +1,125 @@
+# --- Import necessary packages ---
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+sys.path.append('/Users/williamratnavale/Projects/codex_sandbox/code')
+from config import HERE, OUTPUT_DIR
+os.environ.setdefault('MPLCONFIGDIR',str(OUTPUT_DIR/'.matplotlib'))
+from split_inputs import SplitInputs, build_tables
+from latex_figures import make_figures
+from pypdf import PdfReader
+from exhibit_selection import select_exhibits
+
+###########################################################
+# Define paths: this build never reads or writes the live charter repository
+###########################################################
+
+LOCAL_INPUTS=HERE/'data/charter_exhibits'
+INPUT_DIR=LOCAL_INPUTS if (LOCAL_INPUTS/'disclosed_results.xlsx').exists() else OUTPUT_DIR/'inputs'
+# Keep DISCLOSED_INPUT fixed. Replace PENDING_INPUT with your pending actual results.
+DISCLOSED_INPUT=INPUT_DIR/'disclosed_results.xlsx'
+PENDING_INPUT=INPUT_DIR/'pending_results.xlsx'
+LATEX_OUTPUT=(HERE/'output' if INPUT_DIR==LOCAL_INPUTS else OUTPUT_DIR)/'charter_exhibits_latex.pdf'
+SOURCE=HERE/'latex_source'
+BUILD=LATEX_OUTPUT.parent/'latex_build'
+
+# Toggle whole exhibits here. Empty lists include everything.
+# Examples: EXCLUDE_TABLES = ['Table A.4', 'lottery_sample_selection_table']
+#           EXCLUDE_FIGURES = ['Figure A.1', 'appendix-cmo-leave-one-out']
+# All accepted labels/identifiers are listed in README_LATEX.md (README.md in ZIP).
+EXCLUDE_TABLES = []
+EXCLUDE_FIGURES = []
+
+
+def build_latex(pending_path=PENDING_INPUT, output_path=LATEX_OUTPUT, build_dir=BUILD, disclosed_path=DISCLOSED_INPUT,
+                exclude_tables=None, exclude_figures=None):
+    main,appendix,included,excluded,references=select_exhibits(
+        SOURCE, EXCLUDE_TABLES if exclude_tables is None else exclude_tables,
+        EXCLUDE_FIGURES if exclude_figures is None else exclude_figures)
+    tables=[e['name'] for e in included if e['kind']=='table']
+    figures=[e['name'] for e in included if e['kind']=='figure']
+    data=SplitInputs(disclosed_path,pending_path,selected_tables=tables,selected_figures=figures)
+    build_dir=Path(build_dir);output_path=Path(output_path)
+    build_dir.mkdir(parents=True,exist_ok=True)
+    shutil.copytree(SOURCE/'paper',build_dir/'paper',dirs_exist_ok=True)
+    (build_dir/'outputs/scalars').mkdir(parents=True,exist_ok=True)
+    # The exhibit-only document does not consume paper prose scalars.
+    (build_dir/'outputs/scalars/scalars.sty').write_text('% No prose scalars are used by these exhibits.\n')
+    for folder in ['tables','figures']:
+        shutil.rmtree(build_dir/'outputs'/folder,ignore_errors=True)
+    derived=build_tables(data,SOURCE,build_dir)
+    make_figures(data,build_dir/'outputs/figures',selected_figures=figures)
+    (build_dir/'paper/main-tables.tex').write_text(main)
+    (build_dir/'paper/appendix.tex').write_text(appendix)
+    # Preserve the original captions, notes, float/page geometry and appendix numbering.
+    # Reference-only labels resolve references to the preceding (excluded) figures.
+    wrapper=r'''\documentclass[11pt,english]{article}
+\makeatletter
+\def\input@path{{paper/}{outputs/scalars/}}
+\makeatother
+\input{preamble}
+\makeatletter
+\AtBeginDocument{%
+\newlabel{fig:earnings-early-adulthood}{{1}{1}{}{figure.1}{}}%
+\newlabel{fig:earnings-distribution}{{2}{1}{}{figure.2}{}}%
+\newlabel{fig:childhood-program-earnings}{{4}{1}{}{figure.4}{}}%
+\bibcite{chetty2011star}{{2011}{Chetty et~al.}{{Chetty et~al.}}{}}%
+\bibcite{credoCMO}{{2017}{CREDO}{{CREDO}}{}}%
+}
+\makeatother
+\begin{document}
+\input{main-tables}
+\begin{appendices}
+\input{appendix}
+\end{appendices}
+\end{document}
+'''
+    if references:
+        wrapper=wrapper.replace(r'\AtBeginDocument{%',r'\AtBeginDocument{%'+'\n'+references)
+    if not any(e['section']=='appendix' for e in included):
+        wrapper=wrapper.replace('\\begin{appendices}\n\\input{appendix}\n\\end{appendices}', '')
+    if data.metadata['synthetic'].lower()=='true':
+        wrapper=wrapper.replace(r'\begin{document}',r'''\AddToShipoutPictureFG{\AtPageLowerLeft{\put(72,18){\makebox(0,0)[l]{\sffamily\fontsize{7}{8}\selectfont Synthetic inputs -- layout reproduced from manuscript}}}}
+\begin{document}''')
+    mixed=data.metadata.get('synthetic','').lower()=='true'
+    if mixed:
+        wrapper=wrapper.replace('Synthetic inputs -- layout reproduced from manuscript',
+            'Black table values: manuscript; red: synthetic placeholders. Plots: synthetic.')
+    (build_dir/'main.tex').write_text(wrapper)
+    engine=shutil.which('latexmk') or '/Library/TeX/texbin/latexmk'
+    command=[engine,'-pdf','-interaction=nonstopmode','-halt-on-error','-file-line-error','main.tex']
+    env=os.environ.copy();env['PATH']='/Library/TeX/texbin:'+env.get('PATH','')
+    result=subprocess.run(command,cwd=build_dir,env=env,capture_output=True,text=True)
+    (build_dir/'compile-output.txt').write_text(result.stdout+'\n'+result.stderr)
+    if result.returncode:
+        raise RuntimeError('LaTeX compilation failed:\n'+(result.stdout+result.stderr)[-4500:])
+    pdf=PdfReader(build_dir/'main.pdf')
+    text='\n'.join(p.extract_text() for p in pdf.pages)
+    if len(pdf.pages)!=len(included):
+        raise AssertionError(f'Expected {len(included)} exhibit pages, got {len(pdf.pages)}')
+    for page,exhibit in zip(pdf.pages,included):
+        if exhibit['title'].upper() not in page.extract_text():
+            raise AssertionError(f'Missing or renumbered exhibit: {exhibit["title"]}')
+    output_path.parent.mkdir(parents=True,exist_ok=True)
+    shutil.copyfile(build_dir/'main.pdf',output_path)
+    audit={'pages':len(pdf.pages),'input_sha256':{role:hashlib.sha256(path.read_bytes()).hexdigest() for role,path in data.paths.items()},
+           'source_hashes':{str(p.relative_to(SOURCE)):hashlib.sha256(p.read_bytes()).hexdigest() for p in SOURCE.rglob('*.tex')},
+           'derived':derived,'used_keys':[list(k) for k in sorted(data.used)],
+           'input_paths':{role:str(path) for role,path in data.paths.items()},
+           'binding_sha256':hashlib.sha256((HERE/'table_bindings.json').read_bytes()).hexdigest(),
+           'derived_scope':'Fixed disclosed fields and separately supplied pending results',
+           'included_exhibits':[e['title'] for e in included],
+           'excluded_exhibits':[e['title'] for e in excluded]}
+    output_path.with_suffix('.audit.json').write_text(json.dumps(audit,indent=2)+'\n')
+    print(f'Created {output_path}: {len(pdf.pages)} pages using source LaTeX layouts')
+    return audit
+
+
+if __name__=='__main__':
+    build_latex()

ltc_code-0.2.33/src/ltc_code/paper/config.py

@@ -0,0 +1,27 @@
+# --- Import necessary packages ---
+import sys
+from pathlib import Path
+
+sys.path.append("/Users/williamratnavale/Projects/codex_sandbox/code")
+try:
+    from paths import CODEX_SYNTHETIC, OUTPUTS
+except ModuleNotFoundError:
+    # A copied folder also works on the inside without the sandbox paths module.
+    CODEX_SYNTHETIC = Path(__file__).resolve().parent / "data"
+    OUTPUTS = Path(__file__).resolve().parent / "output"
+
+###########################################################
+# Define necessary paths
+###########################################################
+
+HERE = Path(__file__).resolve().parent
+# Replace this one path with your real workbook. The renderer never generates data.
+BUNDLED_INPUT = HERE / "data" / "charter_exhibits" / "synthetic_estimates.xlsx"
+INPUT_XLSX = BUNDLED_INPUT if BUNDLED_INPUT.exists() else CODEX_SYNTHETIC / "charter_exhibits" / "synthetic_estimates.xlsx"
+OUTPUT_DIR = HERE / "output" if BUNDLED_INPUT.exists() else OUTPUTS / "charter_exhibits"
+OUTPUT_PDF = OUTPUT_DIR / "charter_exhibits.pdf"
+SOURCE_SNAPSHOT = HERE / "source_snapshot.json"
+
+# Blank values require an explicit status, rather than becoming zero.
+STATUSES = {"available", "unavailable", "not_applicable"}
+PAGE_SIZE = (960, 540)