ltc-code 0.2.28__tar.gz → 0.2.30__tar.gz

{ltc_code-0.2.28 → ltc_code-0.2.30}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: ltc-code
-Version: 0.2.28
+Version: 0.2.30
 Summary: Add your description here
 Requires-Dist: fastexcel>=0.16,<0.20
 Requires-Dist: polars>=1.36.1,<1.42

{ltc_code-0.2.28 → ltc_code-0.2.30}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ltc-code"
-version = "0.2.28"
+version = "0.2.30"
 description = "Add your description here"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -17,3 +17,7 @@ ltc-summary-stats = "ltc_code.make_summary_stats_table:main"
 [build-system]
 requires = ["uv_build>=0.11.16,<0.12.0"]
 build-backend = "uv_build"
+
+[tool.uv.build-backend]
+source-exclude = ["**/.DS_Store", "**/__pycache__/**", "**/*.pyc", "src/ltc_code/nsc/input/**", "src/ltc_code/nsc/output/**"]
+wheel-exclude = ["**/.DS_Store", "**/__pycache__/**", "**/*.pyc", "ltc_code/nsc/input/**", "ltc_code/nsc/output/**"]

ltc_code-0.2.30/src/ltc_code/nsc/NEW_CROSSWALK.md

@@ -0,0 +1,26 @@
+# New NSC crosswalk build
+
+Run the new variant explicitly with `python -m ltc_code.nsc.build_nsc_outcomes_new` after providing the student input files. The original build remains available. The new variant writes to `nsc/output/new_crosswalk/` to keep its outputs separate.
+
+The official April 2023 NSC crosswalk is primary. Append mapped legacy codes only when their full lookup key is absent from the official file. Do not fill the 108 present-but-blank official keys using Sarah's mappings. Existing 18 sole-rate candidate choices and 24 Calhoun branch assignments are explicit and documented in the code. Preserve full branch codes and deterministic legacy selection.
+
+Institution type uses IPEDS 2013, then the latest nonmissing directory classification, then three inline exceptions formerly loaded from the manual spreadsheet. Less-than-two-year labels are parsed before broader two-year labels. Treat Missing/not reported as null before choosing the latest directory record. The existing two-year-or-less outcome grouping is retained. The 5–8 tier bucket lookup counts Nevada State once despite its tier 5/7 aliases.
+
+All student outcome formulas, eligibility windows, completion-rate imputation, and the existing within-bucket coarse-rate exercise remain unchanged. The change affects their institutional inputs. No annual enrollment-year classification is introduced.
+
+Expected counts with the audited files: 24,166 keys; 21,315 with UNITID; 21,204 matching a UNITID in the 2013/directory files (99.479%); 7,867 matched institutions with no missing type. Distinct institution types: 3,329 four-year, 2,440 two-year, 2,098 less-than-two-year. These are reference-table counts, not secure-sample match rates or completeness of completion rates.
+
+Required files in `nsc/raw/`:
+- NSC_SCHOOL_CODE_TO_IPEDS_UNIT_ID_XWALK_APR-2023.xlsx (official workbook, NSC_to_IPEDS_UNIT_ID sheet)
+- college_crosswalk.xls (legacy fallback)
+- IPEDS_IC_2013.csv
+- directory.dta
+- CREDENTIAL_LEVEL_LOOKUP_TABLE.xlsx
+- ipeds_data.dta
+- chetty/mrc_table2.dta and chetty/mrc_table11.dta
+
+The old IPEDS_IC_manual.xlsx is not needed by this variant. Student inputs remain input/apps.csv, input/nsc_records_old.dta, and input/nsc_records_new.csv. Do not supply synthetic records for actual analysis. Institutional support files are bundled in the locally built wheel and source distribution but are Git-ignored. GitHub Actions publishes these verified distributions from release assets; it does not rebuild from a checkout that lacks the data. Student inputs are never bundled.
+
+Intermediate diagnostic CSVs and printed audit tables are not generated. Only the full and selected final outcome files are written, each in Parquet and CSV format. The 99.48% matched subset is an audit denominator, not a filter that drops student records from the build.
+
+Version 0.2.30 adds final audit exports and selectivity/BA sensitivity outcomes to both builds; see [OUTCOMES.md](OUTCOMES.md) for names and definitions.

ltc_code-0.2.30/src/ltc_code/nsc/OUTCOMES.md

@@ -0,0 +1,59 @@
+# NSC outcomes added in 0.2.30
+
+Both `build_nsc_outcomes.py` and `build_nsc_outcomes_new.py` provide these fields.
+The new-crosswalk variant remains an explicit separate build.
+
+## Final institution audit fields
+
+`nsc_outcomes_final` retains `ID_FSC_firstinst`, `college_name_firstinst`,
+`unitid_firstinst`, `tier_firstinst`, `college_years_firstinst`, and
+`completion_rate_150pct_firstinst`. Institution years are 4 (four-year),
+2 (two-year), 1 (less than two-year), or null if unknown.
+
+The first institution remains the earliest retained enrollment spell starting
+on or after July 1 of the year the student turns 18, with the existing tie rule.
+It need not meet the attendance-status rule or fall within the by-year-four
+window. Attendance indicators describe qualifying enrollment in their window;
+they do not describe the first institution. A transfer can therefore have a
+two-year first institution and four-year attendance. Prediction formulas and
+first-institution selection have not changed.
+
+The build writes the selected table to `nsc_outcomes_final.parquet` and `.csv`
+alongside the existing full `nsc_outcomes.parquet` and `.csv` outputs. These
+are the only four files written; intermediate diagnostic exports and printed
+audit tables have been removed.
+
+## Selectivity outcomes
+
+Selective means tiers 3–4. The existing `elite` name means highly selective,
+tiers 1–2. These institution groups do not overlap. A student can attend or
+earn degrees at institutions in both groups, so their student indicators
+are not forced to be mutually exclusive.
+
+| Example | Meaning |
+| --- | --- |
+| `att_selective_byY4` | Qualifying attendance at a tier 3–4 institution by year 4 |
+| `att_elite_byY4` | Qualifying attendance at a tier 1–2 institution by year 4 |
+| `cmp_selective_byY8` | AA or BA from a tier 3–4 institution by year 8 |
+| `cmp_elite_byY8` | Existing AA-or-BA measure for tiers 1–2 |
+| `cmp_BA_selective_byY8` | BA from a tier 3–4 institution by year 8 |
+| `cmp_BA_elite_byY8` | BA from a tier 1–2 institution by year 8 |
+| `cmp_BA_noimpute_byY8` | Any BA without institution-type credential imputation |
+| `cmp_BA_selective_noimpute_byY8` | Same sensitivity definition for tiers 3–4 |
+| `cmp_BA_elite_noimpute_byY8` | Same sensitivity definition for tiers 1–2 |
+
+Attendance is available for `inY1`–`inY8`, `byY1`–`byY8`, their `fall` and
+`spring` variants, and calendar ages 18–26 (e.g. `att_selective_20`). Completion
+is available for `byY1`–`byY8`. All these selectivity and sensitivity fields are
+retained in the selected final table. They follow the existing observation
+windows and zero/null rules. Unknown institutional tiers do not count as
+selective. Completion uses the degree-granting institution's tier, not the
+first institution's tier.
+
+The default BA measures retain the existing institution-type imputation.
+The `noimpute` versions use credential-lookup/title evidence before that step:
+an unknown award at a four-year institution alone does not count as a BA.
+Students remain in the sample, and another identified BA can still qualify.
+This does not change institutional IPEDS rates or coarse predictions.
+
+Verification: run `uv run python tests/verify_nsc_outcomes.py` from the repository root with the local institutional support files present. This creates temporary synthetic records and does not require secured student inputs.

{ltc_code-0.2.28 → ltc_code-0.2.30}/src/ltc_code/nsc/build_nsc_outcomes.py

@@ -93,8 +93,8 @@ RANDOM_SEED = 3852804
 SARAH_STEM_CIP_FAMILIES = {"11", "14", "15", "26", "27", "40", "41"}
 DHS_STEM_CIP_FAMILIES = {"14", "26", "27", "40"}
 STEM_DEFINITIONS = ["sarah", "dhs"]
-COLLEGE_TYPES = ["any", "4yr", "2yr", "elite"]
-AGE_ATTENDANCE_TYPES = ["any", "4yr", "2yr"]
+COLLEGE_TYPES = ["any", "4yr", "2yr", "elite", "selective"]
+AGE_ATTENDANCE_TYPES = ["any", "4yr", "2yr", "elite", "selective"]
 AGE_ATTENDANCE_RANGE = range(18, 27)
 
 # These ordered rules exactly reproduce Sarah's Stata replacements. Every rule
@@ -573,7 +573,9 @@ college_ref = (
     .with_columns(
         (pl.col("college_years") == 4).cast(pl.Int8).alias("college_4yr"),
         pl.col("college_years").is_in([1, 2]).cast(pl.Int8).alias("college_2yr"),
+        # Institution groups: highly selective (elite) is tiers 1–2; selective is 3–4.
         pl.col("tier").is_in([1, 2]).cast(pl.Int8).alias("college_elite"),
+        pl.col("tier").is_in([3, 4]).cast(pl.Int8).alias("college_selective"),
     )
     .with_columns(
         (pl.col("college_4yr") + pl.col("college_2yr") > 0)
@@ -592,6 +594,7 @@ college_ref = (
         "college_2yr",
         "tier",
         "college_elite",
+        "college_selective",
         "k_mean",
         "completion_rate_150pct_ip",
     )
@@ -1061,6 +1064,7 @@ enroll_weeks = (
         "college_2yr",
         "tier",
         "college_elite",
+        "college_selective",
         "k_mean",
         "completion_rate_150pct_ip",
         "term_start_date",
@@ -1131,6 +1135,7 @@ enroll = (
         pl.col("college_2yr").max(),
         pl.col("tier").drop_nulls().min(),
         pl.col("college_elite").max(),
+        pl.col("college_selective").max(),
         pl.col("k_mean").drop_nulls().first(),
         pl.col("completion_rate_150pct_ip").drop_nulls().first(),
         pl.col("_term_att").max(),
@@ -1453,6 +1458,8 @@ degrees = (
         .otherwise(pl.lit(None))
         .alias("_degree")
     )
+    # Preserve credential/title evidence before filling unknown awards by sector.
+    .with_columns(pl.col("_degree").alias("_degree_no_type_fill"))
     .with_columns(
         pl.when(pl.col("_degree").is_null() & (pl.col("college_years") == 4))
         .then(pl.lit("BA"))
@@ -1504,6 +1511,12 @@ for year in range(1, N_YEARS_OUT + 1):
             (pl.col("_by_end") * (pl.col("_degree") == "BA").cast(pl.Int8))
             .max()
             .alias(f"cmp_BA_byY{year}"),
+            (
+                pl.col("_by_end")
+                * (pl.col("_degree_no_type_fill") == "BA").fill_null(False).cast(pl.Int8)
+            )
+            .max()
+            .alias(f"cmp_BA_noimpute_byY{year}"),
             (pl.col("_by_end") * pl.col("_degree").is_in(["AA", "BA"]).cast(pl.Int8))
             .max()
             .alias(f"cmp_any_byY{year}"),
@@ -1514,6 +1527,28 @@ for year in range(1, N_YEARS_OUT + 1):
             )
             .max()
             .alias(f"cmp_elite_byY{year}"),
+            (
+                pl.col("_by_end")
+                * pl.col("_degree").is_in(["AA", "BA"]).cast(pl.Int8)
+                * pl.col("college_selective").fill_null(0).cast(pl.Int8)
+            )
+            .max()
+            .alias(f"cmp_selective_byY{year}"),
+            # BA-only outcomes support comparisons with any-BA completion.
+            *[
+                (
+                    pl.col("_by_end")
+                    * (pl.col(degree_column) == "BA").fill_null(False).cast(pl.Int8)
+                    * pl.col(f"college_{college_type}").fill_null(0).cast(pl.Int8)
+                )
+                .max()
+                .alias(f"cmp_BA_{college_type}{suffix}_byY{year}")
+                for college_type in ["selective", "elite"]
+                for degree_column, suffix in [
+                    ("_degree", ""),
+                    ("_degree_no_type_fill", "_noimpute"),
+                ]
+            ],
             *[
                 expression
                 for definition in STEM_DEFINITIONS
@@ -1898,8 +1933,14 @@ nsc_outcomes = nsc_outcomes.with_columns(
     .alias("adj_cmp_rate_coarse_2yr"),
 )
 
-# Give each IPEDS institution one tier, excluding missing crosswalk tiers.
-ipeds_tiers = college_ref.select("unitid", "tier").drop_nulls().unique()
+# Only tiers 1-8 enter these four-year bucket averages. Other-tier branch
+# mappings can share a UNITID and must not duplicate the institution's cohort.
+ipeds_tiers = (
+    college_ref.select("unitid", "tier")
+    .drop_nulls()
+    .filter(pl.col("tier").is_in(range(1, 9)))
+    .unique()
+)
 ipeds_tier_completion = ipeds_completion_with_sector.join(
     ipeds_tiers, on="unitid", how="left", validate="1:1"
 ).filter(pl.col("college_sector") == "4yr")
@@ -1926,7 +1967,6 @@ for label, tiers in tier_buckets.items():
         .otherwise(pl.col("adj_cmp_rate_4yr"))
         .alias(f"adj_cmp_rate_4yr_coarse_{label}")
     )
-    print(f"Four-year IPEDS-cohort-weighted completion rate, {label}: {bucket_rate}")
 
 # Match the available 1098-T calendar years, including the missing 2015 year.
 tax_years = list(range(2011, 2015)) + list(range(2016, 2023))
@@ -2008,6 +2048,16 @@ nsc_outcomes = nsc_outcomes.with_columns(
     .alias("recovered_nsc_outcome")
 )
 
+# Export every selectivity window, plus the BA sensitivity outcomes. Keep the
+# existing recovered-outcome definition above independent of this export list.
+selectivity_outcome_columns = [
+    column for column in nsc_outcomes.columns
+    if column.startswith((
+        "att_selective_", "att_elite_", "cmp_selective_", "cmp_elite_",
+        "cmp_BA_selective_", "cmp_BA_elite_", "cmp_BA_noimpute_",
+    ))
+]
+
 keep_columns = [
     "sid_cepr",
     "k_mean",
@@ -2026,140 +2076,36 @@ keep_columns = [
     "adj_cmp_rate_coarsen_2yr",
     "ID_FSC_firstinst",
     "college_name_firstinst",
+    "unitid_firstinst",
+    "college_years_firstinst",
     "tier_firstinst",
     "completion_rate_150pct_firstinst",
-] + outcome_columns + stem_outcome_columns
+] + outcome_columns + stem_outcome_columns + selectivity_outcome_columns
+keep_columns = list(dict.fromkeys(keep_columns))
 
 nsc_outcomes_final = nsc_outcomes.select(keep_columns)
 
+# Retain the existing completeness check without generating audit tables.
+if nsc_outcomes.filter(
+    pl.col("att_any_byY4").is_not_null() & pl.col("k_mean_coarse").is_null()
+).height > 0:
+    raise ValueError("k_mean_coarse is unexpectedly missing for observable students.")
+
 OUTCOMES.parent.mkdir(parents=True, exist_ok=True)
 nsc_outcomes.write_parquet(OUTCOMES)
 nsc_outcomes.with_columns(
     pl.all().exclude(pl.Date).cast(pl.String, strict=False)
 ).write_csv(OUTCOMES.with_suffix(".csv"))
 
+# Save the selected analysis/audit table as well as the existing full output.
+final_path = OUTCOMES.with_name(f"{OUTCOMES.stem}_final.parquet")
+nsc_outcomes_final.write_parquet(final_path)
+nsc_outcomes_final.with_columns(
+    pl.all().exclude(pl.Date).cast(pl.String, strict=False)
+).write_csv(final_path.with_suffix(".csv"))
+
+print(f"Wrote {final_path}")
+print(f"Wrote {final_path.with_suffix('.csv')}")
 print(f"Wrote {OUTCOMES}")
 print(f"Wrote {OUTCOMES.with_suffix('.csv')}")
 print(f"Rows: {nsc_outcomes.height}, columns: {len(nsc_outcomes.columns)}")
-
-first_college_coverage = (
-    nsc_outcomes.filter(pl.col("ID_FSC_firstinst").is_not_null())
-    .group_by("ID_FSC_firstinst")
-    .agg(
-        pl.col("k_mean_firstinst").is_not_null().any().alias("has_k_mean"),
-        pl.col("completion_rate_150pct_firstinst")
-        .is_not_null()
-        .any()
-        .alias("has_completion_rate"),
-    )
-)
-print(
-    "First-college coverage: "
-    f"{first_college_coverage['has_k_mean'].sum()}/"
-    f"{first_college_coverage.height} with college-specific k_mean; "
-    f"{first_college_coverage['has_completion_rate'].sum()}/"
-    f"{first_college_coverage.height} with IPEDS completion rate"
-)
-print(
-    "National student-weighted coarse values: "
-    f"k_mean 4yr={k_mean_4yr_coarse:.2f}, "
-    f"k_mean 2yr-or-less={k_mean_2yr_coarse:.2f}, "
-    f"completion 4yr={cmp_rate_4yr_coarse:.4f}, "
-    f"completion 2yr-or-less={cmp_rate_2yr_coarse:.4f}; "
-    "observed by-Y4 nonattender completion=0"
-)
-print(
-    "Sample-institution IPEDS-cohort-weighted coarse values: "
-    f"completion 4yr={cmp_rate_4yr_coarse_sample:.4f} "
-    f"({sample_4yr_institutions} institutions), "
-    f"completion 2yr-or-less={cmp_rate_2yr_coarse_sample:.4f} "
-    f"({sample_2yr_institutions} institutions)"
-)
-
-observable_y4 = nsc_outcomes.filter(pl.col("att_any_byY4").is_not_null())
-coarse_audit = observable_y4.select(
-    pl.len().alias("students"),
-    pl.col("k_mean_coarse").null_count().alias("k_mean_coarse_missing"),
-    pl.col("k_mean_coarse").min().alias("k_mean_coarse_min"),
-    pl.col("k_mean_coarse").max().alias("k_mean_coarse_max"),
-    pl.col("adj_cmp_rate_coarse")
-    .null_count()
-    .alias("adj_cmp_rate_coarse_missing"),
-    pl.col("adj_cmp_rate_coarse").min().alias("adj_cmp_rate_coarse_min"),
-    pl.col("adj_cmp_rate_coarse").max().alias("adj_cmp_rate_coarse_max"),
-    pl.col("adj_cmp_rate_coarse_sample")
-    .null_count()
-    .alias("adj_cmp_rate_coarse_sample_missing"),
-    pl.col("adj_cmp_rate_coarse_sample")
-    .min()
-    .alias("adj_cmp_rate_coarse_sample_min"),
-    pl.col("adj_cmp_rate_coarse_sample")
-    .max()
-    .alias("adj_cmp_rate_coarse_sample_max"),
-)
-
-if coarse_audit.item(0, "k_mean_coarse_missing") > 0:
-    raise ValueError("k_mean_coarse is unexpectedly missing for observable students.")
-# Attendees without a classified first college can have missing coarse rates.
-# Report those counts below rather than treating them as a build failure.
-print("Coarse outcome audit:")
-print(coarse_audit)
-
-attendee_completion_audit = (
-    observable_y4.filter(pl.col("att_any_byY4") == 1)
-    .select(
-        pl.len().alias("by_y4_attendees"),
-        pl.col("completion_rate_150pct_firstinst")
-        .is_null()
-        .sum()
-        .alias("missing_first_institution_rate"),
-        (
-            pl.col("completion_rate_150pct_firstinst").is_null()
-            & pl.col("completion_rate_150pct_ip").is_not_null()
-        )
-        .sum()
-        .alias("filled_by_tier_median"),
-        pl.col("adj_cmp_rate")
-        .is_null()
-        .sum()
-        .alias("missing_after_tier_median"),
-        pl.col("adj_cmp_rate")
-        .is_null()
-        .mean()
-        .alias("missing_after_tier_median_share"),
-        (
-            pl.col("adj_cmp_rate").is_null()
-            & pl.col("tier_firstinst").is_null()
-        )
-        .sum()
-        .alias("missing_after_tier_median_no_tier"),
-        (
-            pl.col("adj_cmp_rate").is_null()
-            & pl.col("tier_firstinst").is_not_null()
-        )
-        .sum()
-        .alias("missing_after_tier_median_with_tier"),
-    )
-)
-print("By-Y4 attendee completion-rate audit:")
-print(attendee_completion_audit)
-print(
-    nsc_outcomes.select("sid_cepr", "cohort_lottery", "recovered_nsc_outcome").sort(
-        "sid_cepr"
-    )
-)
-
-# Nonmissing year-eight completion identifies students with the full Y8 window.
-year8_attendance_audit = (
-    nsc_outcomes.filter(pl.col("cmp_BA_byY8").is_not_null())
-    .select(
-        pl.len().alias("N_year8_sample"),
-        (pl.col("att_any_byY4") == 0).sum().alias("N_no_attendance_byY4"),
-        (
-            (pl.col("att_any_byY4") == 0)
-            & (pl.col("cmp_BA_byY8") == 1)
-        ).sum().alias("N_no_attendance_byY4_BA_byY8"),
-    )
-)
-print("Year-eight sample attendance and late BA completion check:")
-print(year8_attendance_audit)