ltc-code 0.2.31__tar.gz → 0.2.32__tar.gz

ltc_code-0.2.32/PKG-INFO

@@ -0,0 +1,20 @@
+Metadata-Version: 2.3
+Name: ltc-code
+Version: 0.2.32
+Summary: Add your description here
+Requires-Dist: fastexcel>=0.16,<0.20
+Requires-Dist: polars>=1.36.1,<1.42
+Requires-Dist: polars-readstat>=0.20.2
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+
+
+### College scatter plots (0.2.32)
+
+`from ltc_code.plot_colleges import plot_colleges` provides an OI-themed
+scatter with explicit Polars `x`/`y` aggregations, college grouping, optional
+filters and categorical color groups. Plotting additionally requires
+`oi-tools`, `plotnine`, and `pyarrow` in the analysis environment; they are
+not required to run the NSC build. See [NSC extensions](src/ltc_code/nsc/EXTENSIONS.md)
+for attendance definitions, coarsening names, and plotting examples.

ltc_code-0.2.32/README.md

@@ -0,0 +1,10 @@
+
+
+### College scatter plots (0.2.32)
+
+`from ltc_code.plot_colleges import plot_colleges` provides an OI-themed
+scatter with explicit Polars `x`/`y` aggregations, college grouping, optional
+filters and categorical color groups. Plotting additionally requires
+`oi-tools`, `plotnine`, and `pyarrow` in the analysis environment; they are
+not required to run the NSC build. See [NSC extensions](src/ltc_code/nsc/EXTENSIONS.md)
+for attendance definitions, coarsening names, and plotting examples.

{ltc_code-0.2.31 → ltc_code-0.2.32}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ltc-code"
-version = "0.2.31"
+version = "0.2.32"
 description = "Add your description here"
 readme = "README.md"
 requires-python = ">=3.9"

ltc_code-0.2.32/src/ltc_code/nsc/EXTENSIONS.md

@@ -0,0 +1,102 @@
+# NSC attendance and coarsening changes in the sandbox
+
+The package build is `ltc_code/nsc/build_nsc_outcomes_new.py`.
+Import the scatter helper with `from ltc_code.plot_colleges import plot_colleges`.
+
+Version 0.2.32 changes `coarse_t12`, `coarse_t34`, and `coarse_t58` to the
+overall four-year benchmark; their former within-group calculations are
+retained as `coarse_t12_pool`, `coarse_t34_pool`, and `coarse_t58_pool`.
+
+## Two attendance measures
+
+For each age 18–26 and each grouping `any`, `4yr`, `2yr`:
+
+| Example | Counts |
+| --- | --- |
+| `att_any_1098_20` | F, Q, H, plus unknown statuses under the existing policy |
+| `att_any_1098_all_20` | The same, plus L (less than half-time) |
+
+Both versions exclude W (withdrawn), A (leave), D (deceased), and other
+non-enrollment statuses. The existing handling of generic part-time text is
+retained. Q now qualifies, and written-out "less than half-time" is classified
+before matching "half-time". Adjacent spells with different eligibility are
+kept separate rather than assigning the highest status to the entire period.
+
+The original names are retained. There are no duplicate `_fulltime_` columns.
+
+Both measures use the available tax years 2011–2014 and 2016–2022. Observable
+years without counted attendance get zero; unavailable years remain missing.
+Four-year attendance takes priority separately within each version. These
+measures retain the existing NSC matching, record deduplication, and weekly
+institution selection; they cannot recover records missing from the source.
+
+## Coarsening: preserved and additional variables
+
+Names without `_pool` use the overall four-year benchmark only for the named
+tiers. Names ending in `_pool` use each named group's own pooled rate.
+All names below start with `adj_cmp_rate_4yr_coarse_`:
+
+| Suffix | Calculation |
+| --- | --- |
+| `t12` | Replace only tiers 1–2 with the overall four-year average |
+| `t34` | Replace only tiers 3–4 with the overall four-year average |
+| `t58` | Replace only tiers 5–8 with the overall four-year average |
+| `t12_pool` | Replace tiers 1–2 with their own pooled average |
+| `t34_pool` | Replace tiers 3–4 with their own pooled average |
+| `t58_pool` | Replace tiers 5–8 with their own pooled average |
+| `t12_t34_pool` | Pool 1–2 and, separately, 3–4 |
+| `t14_t58_pool` | Pool 1–4 and, separately, 5–8 |
+
+Every average uses total IPEDS completers divided by the total adjusted
+graduation cohort in the relevant group. Non-target colleges keep the detailed
+four-year prediction. Existing zero and missing-value rules are preserved.
+The separate `adj_cmp_rate_coarse_4yr` still replaces ALL four-year starters'
+rates with the common four-year average.
+
+## Simple college scatter
+
+```python
+plot = plot_colleges(
+    df,
+    x=pl.col("sid_cepr").drop_nulls().n_unique(),
+    y=pl.col("completion_rate_150pct_firstinst").drop_nulls().first() * 100,
+    group_by="unitid_firstinst",
+    filters={"enrolled": 1, "tier_firstinst": [3, 4]},
+    title="First colleges of charter enrollers: tiers 3–4",
+)
+plots.append(plot)
+```
+
+Replace `enrolled` with your charter-enrollment column. Omit `filters` for everyone,
+or supply just one dictionary entry. Polars expressions and lists of expressions
+also work for more general conditions. Filters apply before the x/y aggregation.
+The title and axis labels are customizable. x and y are Polars aggregation
+expressions; their choice determines what each college's dot represents.
+
+`unitid_firstinst` is the matched IPEDS ID of the first qualifying institution
+within the by-Y4 attendance window. `completion_rate_150pct_firstinst` is that
+institution's raw graduation rate. This example describes first colleges, not
+every college ever attended or the institution awarding the degree. Students
+without a matched first-college ID and colleges without a rate do not appear.
+Use a consistent institutional rate per college: `.first()` selects that rate,
+not an average of students' predicted completion outcomes.
+
+The helper returns a plotnine plot; `return_data=True` returns `(plot, data)`.
+It does not save files. Call `.save(...)` on the returned plot to export it.
+
+### OI theme and color groups
+
+The scatter uses `oi_tools.figures.theme_oi()` (the same theme as `plot_bars`),
+with no grid lines and clear black axes. The analysis environment needs
+`oi-tools`, `plotnine`, and `pyarrow`.
+
+Add `color="enrolled"` to draw separate dots for each college's enrollers and
+non-enrollers. The function aggregates x and y separately within each
+college/color group, after applying filters. Numeric codes become discrete
+legend labels. For descriptive labels, pass a column containing strings such
+as "Enrollers" and "Non-enrollers". Missing color values are labeled "Missing".
+The OI palette supplies the colors. Omit `color` for one dot per college.
+
+Because the institutional graduation rate is the same for both groups, their
+dots have the same y coordinate; x differs with the number of unique students.
+If both counts are identical, their dots overlap.

{ltc_code-0.2.31 → ltc_code-0.2.32}/src/ltc_code/nsc/build_nsc_outcomes_new.py

@@ -914,7 +914,7 @@ nsc = nsc.with_columns(
 ###########################################################
 
 # Sarah counts half-time or more as an attended enrollment spell.
-enrollment_status = pl.col("enrollment").str.to_uppercase().fill_null("")
+enrollment_status = pl.col("enrollment").str.to_uppercase().str.strip_chars().fill_null("")
 
 enroll = (
     nsc.filter(pl.col("graduated") == "N")
@@ -935,16 +935,19 @@ enroll = (
         .then(5)
         .when(enrollment_status.str.contains("PART", literal=False))
         .then(4)
+        .when(enrollment_status.str.contains("LESS|\\bL\\b", literal=False))
+        .then(1)
         .when(enrollment_status.str.contains("HALF|\\bH\\b", literal=False))
         .then(3)
         .when(enrollment_status.str.contains("QUARTER|\\bQ\\b", literal=False))
-        .then(2)
-        .when(enrollment_status.str.contains("LESS|\\bL\\b", literal=False))
-        .then(1)
+        .then(4)  # Three-quarter time qualifies as at least half-time.
         .otherwise(0)
         .alias("_enrollment_score")
     )
-    .with_columns((pl.col("_enrollment_score") >= 3).cast(pl.Int8).alias("_term_att"))
+    .with_columns(
+        (pl.col("_enrollment_score") >= 3).cast(pl.Int8).alias("_term_att"),
+        (pl.col("_enrollment_score") >= 1).cast(pl.Int8).alias("_term_att_all"),
+    )
     .with_columns(
         ((pl.col("term_start_date").cast(pl.Int64) // 7).cast(pl.Int64)).alias(
             "week_start"
@@ -1035,6 +1038,7 @@ enroll_weeks = (
         "week_start",
         "week_end",
         "_term_att",
+        "_term_att_all",
         "_enrollment_score",
         "_enrollment_stem_sarah",
         "_enrollment_stem_dhs",
@@ -1068,6 +1072,8 @@ enroll_weeks = (
             (pl.col("week") != pl.col("week").shift(1) + 1)
             | (pl.col("sid_cepr") != pl.col("sid_cepr").shift(1))
             | (pl.col("ID_FSC") != pl.col("ID_FSC").shift(1))
+            | (pl.col("_term_att") != pl.col("_term_att").shift(1))
+            | (pl.col("_term_att_all") != pl.col("_term_att_all").shift(1))
             | (
                 pl.col("_enrollment_stem_sarah")
                 != pl.col("_enrollment_stem_sarah").shift(1)
@@ -1102,6 +1108,7 @@ enroll = (
         pl.col("k_mean").drop_nulls().first(),
         pl.col("completion_rate_150pct_ip").drop_nulls().first(),
         pl.col("_term_att").max(),
+        pl.col("_term_att_all").max(),
         pl.col("_enrollment_stem_sarah").max(),
         pl.col("_enrollment_stem_dhs").max(),
         pl.col("week").min().alias("week_start"),
@@ -1216,10 +1223,11 @@ for age in AGE_ATTENDANCE_RANGE:
         (
             (pl.col("term_start_date") <= pl.col("_age_window_end"))
             & (pl.col("term_end_date") >= pl.col("_age_window_start"))
-            & (pl.col("_term_att") == 1)
         )
         .cast(pl.Int8)
-        .alias("_overlaps")
+        .alias("_overlaps_all")
+    ).with_columns(
+        (pl.col("_overlaps_all") * pl.col("_term_att")).alias("_overlaps")
     )
 
     enrollment_pieces.append(
@@ -1236,6 +1244,19 @@ for age in AGE_ATTENDANCE_RANGE:
                 .alias(f"att_{college_type}_{age}")
                 for college_type in AGE_ATTENDANCE_TYPES
             ]
+            + [
+                (
+                    pl.col("_overlaps_all")
+                    * pl.col("_term_att_all")
+                    * (
+                        pl.lit(1) if college_type == "any"
+                        else pl.col(f"college_{college_type}").fill_null(0).cast(pl.Int8)
+                    )
+                )
+                .max()
+                .alias(f"att_{college_type}_all_{age}")
+                for college_type in ["any", "4yr", "2yr"]
+            ]
             + [
                 expression
                 for definition in STEM_DEFINITIONS
@@ -1942,7 +1963,7 @@ ipeds_tier_completion = ipeds_completion_with_sector.join(
 ).filter(pl.col("college_sector") == "4yr")
 
 # Coarsen one bucket at a time, always starting from the detailed prediction.
-tier_buckets = {"t12": [1, 2], "t34": [3, 4], "t58": [5, 6, 7, 8]}
+tier_buckets = {"t12_pool": [1, 2], "t34_pool": [3, 4], "t58_pool": [5, 6, 7, 8]}
 for label, tiers in tier_buckets.items():
     bucket_rate = (
         ipeds_tier_completion.filter(pl.col("tier").is_in(tiers))
@@ -1964,18 +1985,66 @@ for label, tiers in tier_buckets.items():
         .alias(f"adj_cmp_rate_4yr_coarse_{label}")
     )
 
-# Match the available 1098-T calendar years, including the missing 2015 year.
-tax_years = list(range(2011, 2015)) + list(range(2016, 2023))
-nsc_outcomes = nsc_outcomes.with_columns(
-    [
-        pl.when((pl.col("cohort_lottery") + age).is_in(tax_years))
-        .then(pl.col(f"att_any_{age}"))
-        .otherwise(None)
-        .alias(f"att_any_1098_{age}")
-        for age in AGE_ATTENDANCE_RANGE
-    ]
+# Additional coarsening scenarios. Each listed group gets its own pooled
+# IPEDS rate; institutions outside those groups keep their detailed prediction.
+tier_coarsening_scenarios = {
+    "t12_t34_pool": [[1, 2], [3, 4]],
+    "t14_t58_pool": [[1, 2, 3, 4], [5, 6, 7, 8]],
+}
+for label, groups in tier_coarsening_scenarios.items():
+    prediction = pl.col("adj_cmp_rate_4yr")
+    for tiers in groups:
+        group_rate = (
+            ipeds_tier_completion.filter(pl.col("tier").is_in(tiers))
+            .select(pl.col("completers_150pct_ip").sum() / pl.col("cohort_adj_150pct_ip").sum())
+            .item()
+        )
+        if group_rate is None or not 0 <= group_rate <= 1:
+            raise ValueError(f"Invalid pooled completion rate for tiers {tiers}: {group_rate}")
+        prediction = pl.when(
+            (pl.col("college_years_firstinst") == 4)
+            & pl.col("tier_firstinst").is_in(tiers)
+            & (pl.col("att_any_byY4") == 1)
+            & pl.col("adj_cmp_rate_4yr").is_not_null()
+        ).then(group_rate).otherwise(prediction)
+    nsc_outcomes = nsc_outcomes.with_columns(
+        prediction.alias(f"adj_cmp_rate_4yr_coarse_{label}")
+    )
 
-)
+# Replace only the named tiers with the common overall four-year average.
+# The _pool versions above instead use each group's own IPEDS average.
+benchmark_tier_buckets = {"t12": [1, 2], "t34": [3, 4], "t58": [5, 6, 7, 8]}
+for label, tiers in benchmark_tier_buckets.items():
+    nsc_outcomes = nsc_outcomes.with_columns(
+        pl.when(
+            (pl.col("college_years_firstinst") == 4)
+            & pl.col("tier_firstinst").is_in(tiers)
+            & (pl.col("att_any_byY4") == 1)
+            & pl.col("adj_cmp_rate_4yr").is_not_null()
+        ).then(cmp_rate_4yr_coarse).otherwise(pl.col("adj_cmp_rate_4yr"))
+        .alias(f"adj_cmp_rate_4yr_coarse_{label}")
+    )
+
+# Match available 1098-T calendar years, including the missing 2015 year.
+# Existing names retain AT LEAST HALF-TIME (including Q), with the existing
+# unknown-status policy. "all" adds less-than-half-time enrollment, excluding W/A/D and
+# other non-enrollment statuses. Unknown statuses count in both versions.
+tax_years = list(range(2011, 2015)) + list(range(2016, 2023))
+tax_attendance_columns = []
+for college_type in ["any", "4yr", "2yr"]:
+    for age in AGE_ATTENDANCE_RANGE:
+        for label, source in [
+            ("", f"att_{college_type}_{age}"),
+            ("_all", f"att_{college_type}_all_{age}"),
+        ]:
+            column = f"att_{college_type}_1098{label}_{age}"
+            tax_attendance_columns.append(column)
+            nsc_outcomes = nsc_outcomes.with_columns(
+                pl.when((pl.col("cohort_lottery") + age).is_in(tax_years))
+                .then(pl.col(source))
+                .otherwise(None)
+                .alias(column)
+            )
 
 
 ###########################################################
@@ -2066,6 +2135,8 @@ keep_columns = [
     "adj_cmp_rate_coarse_2yr",
     "adj_cmp_rate_coarse_4yr",
     *[f"adj_cmp_rate_4yr_coarse_{label}" for label in tier_buckets],
+    *[f"adj_cmp_rate_4yr_coarse_{label}" for label in tier_coarsening_scenarios],
+    *[f"adj_cmp_rate_4yr_coarse_{label}" for label in benchmark_tier_buckets],
     "adj_cmp_rate_coarse",
     "adj_cmp_rate_coarse_sample",
     "adj_cmp_rate_coarsen_4yr",
@@ -2076,7 +2147,7 @@ keep_columns = [
     "college_years_firstinst",
     "tier_firstinst",
     "completion_rate_150pct_firstinst",
-] + outcome_columns + stem_outcome_columns + selectivity_outcome_columns
+] + outcome_columns + stem_outcome_columns + selectivity_outcome_columns + tax_attendance_columns
 keep_columns = list(dict.fromkeys(keep_columns))
 
 nsc_outcomes_final = nsc_outcomes.select(keep_columns)

ltc_code-0.2.32/src/ltc_code/plot_colleges.py

@@ -0,0 +1,74 @@
+# --- Import necessary packages ---
+import polars as pl
+
+
+def plot_colleges(
+    df: pl.DataFrame,
+    *,
+    x: pl.Expr,
+    y: pl.Expr,
+    group_by: str,
+    filters=None,
+    color: str = None,
+    title: str = "College graduation rates",
+    x_label: str = "Unique students",
+    y_label: str = "150% graduation rate (%)",
+    return_data: bool = False,
+):
+    """Filter rows and aggregate x/y by college, optionally split by color.
+
+    x and y are Polars aggregation expressions, e.g. col('sid_cepr').n_unique()
+    and col('rate').drop_nulls().first() * 100. Supply a raw institution rate
+    with a consistent value per college, not a student-level prediction.
+    filters accepts a dictionary such as {'enrolled': 1, 'tier': [3, 4]},
+    a Polars expression, or a list of expressions. Filtering precedes counting.
+    color names a categorical column: each college/color group gets its own
+    dot and its own x/y aggregation. Numeric group codes are treated as labels.
+    Missing college IDs and missing/nonfinite x or y are omitted. No files
+    are saved unless the caller explicitly saves the returned plot.
+    """
+    from plotnine import aes, element_blank, element_line, geom_point, ggplot, labs, theme
+    from oi_tools.figures import OIColors, scale_color_oi, theme_oi
+
+    if isinstance(filters, dict):
+        filters = [
+            pl.col(column).is_in(value) if isinstance(value, (list, tuple, set))
+            else pl.col(column).is_null() if value is None
+            else pl.col(column) == value
+            for column, value in filters.items()
+        ]
+    if filters is not None:
+        df = df.filter(*filters) if isinstance(filters, (list, tuple)) else df.filter(filters)
+    groups = list(dict.fromkeys([group_by] + ([color] if color else [])))
+    colleges = (
+        df.drop_nulls(group_by)
+        .group_by(groups)
+        .agg(x.alias('_x'), y.alias('_y'))
+        .filter(pl.col('_x').is_finite() & pl.col('_y').is_finite())
+        .sort(groups)
+    )
+    if colleges.is_empty():
+        raise ValueError('No colleges with both x and y remain after filtering.')
+    mapping = aes(x='_x', y='_y')
+    points = geom_point(alpha=0.8, size=2.5, color=OIColors.BLUE)
+    if color is not None:
+        colleges = colleges.with_columns(
+            pl.col(color).cast(pl.String).fill_null('Missing').alias('_color')
+        )
+        mapping = aes(x='_x', y='_y', color='_color')
+        points = geom_point(alpha=0.8, size=2.5)
+    plot = (
+        ggplot(colleges, mapping)
+        + points
+        + theme_oi()
+        + labs(title=title, x=x_label, y=y_label,
+               caption=f'{colleges[group_by].n_unique():,} colleges with observed values')
+        + theme(
+            panel_grid=element_blank(),
+            axis_line=element_line(color='black', size=0.7),
+            figure_size=(7, 5),
+        )
+    )
+    if color is not None:
+        plot += scale_color_oi()
+    return (plot, colleges) if return_data else plot

ltc_code-0.2.31/PKG-INFO

@@ -1,10 +0,0 @@
-Metadata-Version: 2.3
-Name: ltc-code
-Version: 0.2.31
-Summary: Add your description here
-Requires-Dist: fastexcel>=0.16,<0.20
-Requires-Dist: polars>=1.36.1,<1.42
-Requires-Dist: polars-readstat>=0.20.2
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-