robustkit 0.4.0__tar.gz → 0.5.0__tar.gz

{robustkit-0.4.0/robustkit.egg-info → robustkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: robustkit
-Version: 0.4.0
+Version: 0.5.0
 Summary: Practical tools for robust analysis of a single continuous relationship: trend fitting, stability checks, influence diagnostics, and bootstrap uncertainty.
 Author: Mikael Lundqvist
 License: MIT License
@@ -60,8 +60,11 @@ consistency checks), `robustkit.segmentation` (hierarchical grouping,
 per-segment analysis), `robustkit.information` (mutual-information
 feature ranking, quadrant classification, pairwise redundancy/synergy
 scoring), `robustkit.benchmark` (global-trend segment comparison,
-Robustness Map), and `robustkit.report` (analyst vs. publisher views,
-dispersion measures) are stable and tested.
+Robustness Map), `robustkit.report` (analyst vs. publisher views,
+dispersion measures), and `robustkit.quantiles` (generic JSON-stat
+loading, published-quantile-trend visualization, and lognormal-
+calibrated reconstruction of individual-level data from aggregated
+summaries) are stable and tested.
 
 **Note on `information_efficiency`:** values can exceed 1.0 for
 continuous features. `mutual_information` is estimated on the
@@ -111,6 +114,32 @@ ci = bca_bootstrap_ci(x, y, statistic_fn=lambda x_, y_: np.median(y_))
 
 See `examples/quickstart_tutorial.py` for a complete, runnable walkthrough.
 
+## Trend growth rate and goodness of fit
+
+```python
+from robustkit import trend_derivative, goodness_of_fit, compare_polynomial_degrees
+
+fit = fit_huber_trend(df["age"], df["salary"])
+
+# Rate of change of the trend itself (e.g. "salary growth per year of
+# age"), not just its level
+rates = trend_derivative(fit, x=[30, 40, 50])
+
+# How well does this fit actually explain the variation in y?
+goodness_of_fit(df["age"], df["salary"], degree=2)
+
+# Don't assume a quadratic trend is always the right choice -- check
+# empirically whether a higher degree captures meaningfully more
+compare_polynomial_degrees(df["age"], df["salary"], degrees=(1, 2, 3, 4))
+```
+
+**Note:** x is standardized internally before building polynomial
+features (both here and throughout `robustkit.core`), since raw
+polynomial features become numerically unstable at higher degrees for
+realistic x scales (e.g. age^5 vastly outscales age^1). This is
+transparent to callers -- `predict_trend` and `trend_derivative` still
+take and return values in the original x scale.
+
 ## Segmentation
 
 Run any `robustkit.core` analysis independently across subgroups of a
@@ -246,6 +275,127 @@ standalone for tabular reporting; `dispersion_by_bin(x, y, n_bins=10)`
 computes both across bins of a continuous x, e.g. to check whether
 dispersion (inequality) grows with age.
 
+## Loading published quantile tables (SCB / JSON-stat)
+
+Some statistics agencies (e.g. Statistics Sweden, SCB) publish
+quantiles (Q1/median/Q3) directly, with no individual-level data
+available at all. `robustkit.quantiles` loads these tables generically
+via JSON-stat, a standardized dimensional-data format used by SCB and
+other national statistics agencies -- avoiding the fragility of
+parsing metadata out of column-name strings in a wide CSV export.
+
+```python
+from robustkit import load_scb_json_stat, plot_quantile_trend, quantile_trend_dispersion
+
+df = load_scb_json_stat("some_scb_table.json")
+
+# A real SCB quirk this loader does NOT try to guess automatically:
+# category labels can change meaning over time (e.g. Sweden's oldest
+# working-age bracket was labeled "65-66 år" through 2022 and
+# "65-68 år" from 2023, following a pension-age reform). Merge such
+# cases explicitly:
+df = load_scb_json_stat(
+    "some_scb_table.json",
+    rename_categories={"ålder": {"65–68 år": "65–66 år"}},
+)
+
+# Once reshaped to a wide table with q1/median/q3 columns:
+plot_quantile_trend(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+quantile_trend_dispersion(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+```
+
+This is the "quantiles are already given" case. A complementary case
+-- reconstructing approximate individual-level data from aggregated
+group means, for when only summary statistics (not quantiles) are
+available -- is planned as a follow-up (`robustkit.quantiles.reconstruct`).
+
+See `examples/quantiles_tutorial.py` for a complete walkthrough.
+
+## Reconstructing individual-level data from aggregated summaries
+
+For the complementary case -- only aggregated group summaries (n,
+Q1, median, Q3) are available, not the quantile trend itself as the
+final answer, and you want to run `robustkit.core` analyses as if
+individual data existed:
+
+```python
+from robustkit import expand_aggregated_table, check_reconstruction_quality, fit_huber_trend
+
+# One row per group (e.g. year), with n/q1/median/q3 columns
+synthetic = expand_aggregated_table(
+    summary_df, n_col="n", q1_col="q1", median_col="median", q3_col="q3",
+    group_cols=["year"], value_name="salary",
+)
+
+# Now usable exactly like real individual-level data:
+fit = fit_huber_trend(synthetic["year"], synthetic["salary"])
+```
+
+Method: a lognormal distribution is calibrated (via the IQR) to match
+each group's reported Q1/median/Q3, then `n` synthetic values are
+drawn from it. Validated end-to-end against real published SCB salary
+data: a Huber trend fitted on reconstructed pseudo-individual data
+tracked the true published median trend within 2% across 12 years.
+
+**Note on what this recovers:** because a Huber (or Tukey) fit on
+right-skewed reconstructed data tracks something close to the
+*median* trend it was calibrated against -- not the arithmetic mean --
+this is consistent with, not a limitation of, the reconstruction
+method. To target the mean instead, fit on `log(value)` and
+exponentiate predictions back, which approximates the geometric mean.
+
+Always check `check_reconstruction_quality()` before trusting a
+reconstruction: real Q1/median/Q3 triples aren't always perfectly
+consistent with a pure lognormal shape.
+
+**Warning -- unbounded tail at large n:** a lognormal has no natural
+upper limit, and its expected maximum grows with n. Reconstructing at
+the TRUE group size from a national table (SCB salary tables can
+report n in the hundreds of thousands to millions) can produce
+implausibly extreme tail values -- real salaries have practical
+ceilings a pure lognormal doesn't know about. This package's own
+examples and tests deliberately scale n down to a few thousand for
+demonstration; calibration quality (matching Q1/median/Q3) doesn't
+depend on reproducing the true population size, but tail plausibility
+does. No clipping is applied automatically.
+
+### When only a mean is available (no quantiles at all)
+
+Some tables (e.g. SCB's age-breakdown salary tables) report only a
+mean per group, with no spread information. Two deliberately separate
+methods are provided, each making a different explicit assumption --
+compare them rather than silently picking one:
+
+```python
+from robustkit import (
+    expand_aggregated_table_flat, expand_aggregated_table_borrowed_dispersion,
+    compare_reconstruction_methods,
+)
+
+# Method 1: repeat the mean n times -- zero within-group spread.
+# Recovers between-group regression coefficients reasonably well
+# (validated in the original technique this is based on) but
+# understates individual-level variation.
+flat = expand_aggregated_table_flat(df, n_col="n", mean_col="mean_salary", group_cols=["age"])
+
+# Method 2: borrow a dispersion_ratio from a DIFFERENT table that does
+# report quantiles, and use it to imply an approximate spread around
+# the mean. Stacks two assumptions (mean-as-median, and that the
+# borrowed ratio transfers to this population) -- illustrative, not a
+# substitute for genuine quantile data for this specific table.
+borrowed = expand_aggregated_table_borrowed_dispersion(
+    df, n_col="n", mean_col="mean_salary", dispersion_ratio=0.45, group_cols=["age"],
+)
+
+# Compare both for a single group directly:
+compare_reconstruction_methods(n=2000, mean=52200, dispersion_ratio=0.45)
+```
+
+Both methods are documented with their specific assumptions rather
+than presented as equally valid defaults -- being explicit about which
+assumption was made lets the analyst judge how much a conclusion
+depends on it, rather than presenting an assumption as a measurement.
+
 ## Feature pairing (information)
 
 Beyond ranking single features, evaluate *pairs* of features together:

{robustkit-0.4.0 → robustkit-0.5.0}/README.md

@@ -20,8 +20,11 @@ consistency checks), `robustkit.segmentation` (hierarchical grouping,
 per-segment analysis), `robustkit.information` (mutual-information
 feature ranking, quadrant classification, pairwise redundancy/synergy
 scoring), `robustkit.benchmark` (global-trend segment comparison,
-Robustness Map), and `robustkit.report` (analyst vs. publisher views,
-dispersion measures) are stable and tested.
+Robustness Map), `robustkit.report` (analyst vs. publisher views,
+dispersion measures), and `robustkit.quantiles` (generic JSON-stat
+loading, published-quantile-trend visualization, and lognormal-
+calibrated reconstruction of individual-level data from aggregated
+summaries) are stable and tested.
 
 **Note on `information_efficiency`:** values can exceed 1.0 for
 continuous features. `mutual_information` is estimated on the
@@ -71,6 +74,32 @@ ci = bca_bootstrap_ci(x, y, statistic_fn=lambda x_, y_: np.median(y_))
 
 See `examples/quickstart_tutorial.py` for a complete, runnable walkthrough.
 
+## Trend growth rate and goodness of fit
+
+```python
+from robustkit import trend_derivative, goodness_of_fit, compare_polynomial_degrees
+
+fit = fit_huber_trend(df["age"], df["salary"])
+
+# Rate of change of the trend itself (e.g. "salary growth per year of
+# age"), not just its level
+rates = trend_derivative(fit, x=[30, 40, 50])
+
+# How well does this fit actually explain the variation in y?
+goodness_of_fit(df["age"], df["salary"], degree=2)
+
+# Don't assume a quadratic trend is always the right choice -- check
+# empirically whether a higher degree captures meaningfully more
+compare_polynomial_degrees(df["age"], df["salary"], degrees=(1, 2, 3, 4))
+```
+
+**Note:** x is standardized internally before building polynomial
+features (both here and throughout `robustkit.core`), since raw
+polynomial features become numerically unstable at higher degrees for
+realistic x scales (e.g. age^5 vastly outscales age^1). This is
+transparent to callers -- `predict_trend` and `trend_derivative` still
+take and return values in the original x scale.
+
 ## Segmentation
 
 Run any `robustkit.core` analysis independently across subgroups of a
@@ -206,6 +235,127 @@ standalone for tabular reporting; `dispersion_by_bin(x, y, n_bins=10)`
 computes both across bins of a continuous x, e.g. to check whether
 dispersion (inequality) grows with age.
 
+## Loading published quantile tables (SCB / JSON-stat)
+
+Some statistics agencies (e.g. Statistics Sweden, SCB) publish
+quantiles (Q1/median/Q3) directly, with no individual-level data
+available at all. `robustkit.quantiles` loads these tables generically
+via JSON-stat, a standardized dimensional-data format used by SCB and
+other national statistics agencies -- avoiding the fragility of
+parsing metadata out of column-name strings in a wide CSV export.
+
+```python
+from robustkit import load_scb_json_stat, plot_quantile_trend, quantile_trend_dispersion
+
+df = load_scb_json_stat("some_scb_table.json")
+
+# A real SCB quirk this loader does NOT try to guess automatically:
+# category labels can change meaning over time (e.g. Sweden's oldest
+# working-age bracket was labeled "65-66 år" through 2022 and
+# "65-68 år" from 2023, following a pension-age reform). Merge such
+# cases explicitly:
+df = load_scb_json_stat(
+    "some_scb_table.json",
+    rename_categories={"ålder": {"65–68 år": "65–66 år"}},
+)
+
+# Once reshaped to a wide table with q1/median/q3 columns:
+plot_quantile_trend(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+quantile_trend_dispersion(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+```
+
+This is the "quantiles are already given" case. A complementary case
+-- reconstructing approximate individual-level data from aggregated
+group means, for when only summary statistics (not quantiles) are
+available -- is planned as a follow-up (`robustkit.quantiles.reconstruct`).
+
+See `examples/quantiles_tutorial.py` for a complete walkthrough.
+
+## Reconstructing individual-level data from aggregated summaries
+
+For the complementary case -- only aggregated group summaries (n,
+Q1, median, Q3) are available, not the quantile trend itself as the
+final answer, and you want to run `robustkit.core` analyses as if
+individual data existed:
+
+```python
+from robustkit import expand_aggregated_table, check_reconstruction_quality, fit_huber_trend
+
+# One row per group (e.g. year), with n/q1/median/q3 columns
+synthetic = expand_aggregated_table(
+    summary_df, n_col="n", q1_col="q1", median_col="median", q3_col="q3",
+    group_cols=["year"], value_name="salary",
+)
+
+# Now usable exactly like real individual-level data:
+fit = fit_huber_trend(synthetic["year"], synthetic["salary"])
+```
+
+Method: a lognormal distribution is calibrated (via the IQR) to match
+each group's reported Q1/median/Q3, then `n` synthetic values are
+drawn from it. Validated end-to-end against real published SCB salary
+data: a Huber trend fitted on reconstructed pseudo-individual data
+tracked the true published median trend within 2% across 12 years.
+
+**Note on what this recovers:** because a Huber (or Tukey) fit on
+right-skewed reconstructed data tracks something close to the
+*median* trend it was calibrated against -- not the arithmetic mean --
+this is consistent with, not a limitation of, the reconstruction
+method. To target the mean instead, fit on `log(value)` and
+exponentiate predictions back, which approximates the geometric mean.
+
+Always check `check_reconstruction_quality()` before trusting a
+reconstruction: real Q1/median/Q3 triples aren't always perfectly
+consistent with a pure lognormal shape.
+
+**Warning -- unbounded tail at large n:** a lognormal has no natural
+upper limit, and its expected maximum grows with n. Reconstructing at
+the TRUE group size from a national table (SCB salary tables can
+report n in the hundreds of thousands to millions) can produce
+implausibly extreme tail values -- real salaries have practical
+ceilings a pure lognormal doesn't know about. This package's own
+examples and tests deliberately scale n down to a few thousand for
+demonstration; calibration quality (matching Q1/median/Q3) doesn't
+depend on reproducing the true population size, but tail plausibility
+does. No clipping is applied automatically.
+
+### When only a mean is available (no quantiles at all)
+
+Some tables (e.g. SCB's age-breakdown salary tables) report only a
+mean per group, with no spread information. Two deliberately separate
+methods are provided, each making a different explicit assumption --
+compare them rather than silently picking one:
+
+```python
+from robustkit import (
+    expand_aggregated_table_flat, expand_aggregated_table_borrowed_dispersion,
+    compare_reconstruction_methods,
+)
+
+# Method 1: repeat the mean n times -- zero within-group spread.
+# Recovers between-group regression coefficients reasonably well
+# (validated in the original technique this is based on) but
+# understates individual-level variation.
+flat = expand_aggregated_table_flat(df, n_col="n", mean_col="mean_salary", group_cols=["age"])
+
+# Method 2: borrow a dispersion_ratio from a DIFFERENT table that does
+# report quantiles, and use it to imply an approximate spread around
+# the mean. Stacks two assumptions (mean-as-median, and that the
+# borrowed ratio transfers to this population) -- illustrative, not a
+# substitute for genuine quantile data for this specific table.
+borrowed = expand_aggregated_table_borrowed_dispersion(
+    df, n_col="n", mean_col="mean_salary", dispersion_ratio=0.45, group_cols=["age"],
+)
+
+# Compare both for a single group directly:
+compare_reconstruction_methods(n=2000, mean=52200, dispersion_ratio=0.45)
+```
+
+Both methods are documented with their specific assumptions rather
+than presented as equally valid defaults -- being explicit about which
+assumption was made lets the analyst judge how much a conclusion
+depends on it, rather than presenting an assumption as a measurement.
+
 ## Feature pairing (information)
 
 Beyond ranking single features, evaluate *pairs* of features together:

{robustkit-0.4.0 → robustkit-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "robustkit"
-version = "0.4.0"
+version = "0.5.0"
 description = "Practical tools for robust analysis of a single continuous relationship: trend fitting, stability checks, influence diagnostics, and bootstrap uncertainty."
 readme = "README.md"
 license = { file = "LICENSE" }

{robustkit-0.4.0 → robustkit-0.5.0}/robustkit/__init__.py

@@ -13,11 +13,12 @@ Modules:
     robustkit.information    -- mutual-information-based feature ranking
 """
 
-from .core.trend import fit_huber_trend, fit_tukey_trend, fit_ols_trend, predict_trend
+from .core.trend import fit_huber_trend, fit_tukey_trend, fit_ols_trend, predict_trend, trend_derivative
 from .core.stability import model_stability_pct
 from .core.diagnostics import cooks_diagnostic, cook_impact
 from .core.uncertainty import bootstrap_band, bca_bootstrap_ci
 from .core.consistency import check_row_integrity, compare_row_sets
+from .core.goodness_of_fit import goodness_of_fit, compare_polynomial_degrees
 from .segmentation.hierarchy import hierarchical_segment, segment_sizes
 from .segmentation.apply import apply_by_segment
 from .information.entropy import entropy
@@ -34,12 +35,21 @@ from .benchmark.robustness_map import feature_robustness_report, plot_feature_ro
 from .report.dispersion import iqr, dispersion_ratio, dispersion_by_bin
 from .report.visualize_analyst import plot_analyst_view
 from .report.visualize_publisher import plot_publisher_view
+from .quantiles.io import load_scb_json_stat
+from .quantiles.trend import prepare_quantile_trend, plot_quantile_trend, quantile_trend_dispersion
+from .quantiles.reconstruct import (
+    expand_aggregated_group, expand_aggregated_table, check_reconstruction_quality,
+    expand_aggregated_group_flat, expand_aggregated_group_borrowed_dispersion,
+    expand_aggregated_table_flat, expand_aggregated_table_borrowed_dispersion,
+    compare_reconstruction_methods,
+)
 
 __all__ = [
     "fit_huber_trend",
     "fit_tukey_trend",
     "fit_ols_trend",
     "predict_trend",
+    "trend_derivative",
     "model_stability_pct",
     "cooks_diagnostic",
     "cook_impact",
@@ -47,6 +57,8 @@ __all__ = [
     "bca_bootstrap_ci",
     "check_row_integrity",
     "compare_row_sets",
+    "goodness_of_fit",
+    "compare_polynomial_degrees",
     "hierarchical_segment",
     "segment_sizes",
     "apply_by_segment",
@@ -74,6 +86,18 @@ __all__ = [
     "dispersion_by_bin",
     "plot_analyst_view",
     "plot_publisher_view",
+    "load_scb_json_stat",
+    "prepare_quantile_trend",
+    "plot_quantile_trend",
+    "quantile_trend_dispersion",
+    "expand_aggregated_group",
+    "expand_aggregated_table",
+    "check_reconstruction_quality",
+    "expand_aggregated_group_flat",
+    "expand_aggregated_group_borrowed_dispersion",
+    "expand_aggregated_table_flat",
+    "expand_aggregated_table_borrowed_dispersion",
+    "compare_reconstruction_methods",
 ]
 
 __version__ = "0.0.1"

{robustkit-0.4.0 → robustkit-0.5.0}/robustkit/benchmark/robustness_map.py

@@ -50,8 +50,26 @@ def feature_robustness_report(df, target, features=None, degree=2,
                                     barely matters)
         fragile                 -- high stability spread AND high Cook
                                     impact (the least trustworthy)
+
+    Requires at least 2 features. Quadrant assignment is threshold-
+    based (median by default, see robustkit.classify_quadrants) --
+    with a single feature, that feature is trivially "at or above its
+    own median" on both axes, so it would always be classified
+    "fragile" regardless of its actual stability/impact values. This
+    is a property of median-based thresholding with n=1, not a
+    meaningful result, so it is rejected explicitly here rather than
+    silently returning a misleading label.
     """
     features = features or [c for c in df.columns if c != target]
+    if len(features) < 2:
+        raise ValueError(
+            f"feature_robustness_report requires at least 2 features to classify "
+            f"meaningfully (quadrant thresholds are computed across the candidate "
+            f"features); got {len(features)}: {features}. With a single feature, "
+            f"median-based thresholds are degenerate and always classify it as "
+            f"'fragile' regardless of its actual values."
+        )
+
     y = df[target].to_numpy(dtype=float)
 
     rows = []

{robustkit-0.4.0 → robustkit-0.5.0}/robustkit/core/diagnostics.py

@@ -14,7 +14,7 @@ dependency.
 
 import numpy as np
 
-from .trend import _design_matrix, fit_huber_trend, predict_trend
+from .trend import _fit_design_matrix, fit_huber_trend, predict_trend
 
 
 def cooks_diagnostic(x, y, degree=2):
@@ -27,7 +27,7 @@ def cooks_diagnostic(x, y, degree=2):
     y = np.asarray(y, dtype=float)
     n = len(y)
 
-    X, _ = _design_matrix(x, degree)
+    X, _, _ = _fit_design_matrix(x, degree)
     X_design = np.column_stack([np.ones(n), X])
     p = X_design.shape[1]
 

robustkit-0.5.0/robustkit/core/goodness_of_fit.py

@@ -0,0 +1,77 @@
+"""
+Measures of how well a fitted trend explains the variation in y, and a
+diagnostic for choosing the polynomial degree empirically rather than
+assuming a fixed degree fits every trend equally well.
+
+Note on R² across methods: for Huber and Tukey fits, the R² computed
+here is a descriptive "variance explained" measure derived from the
+fit's predictions -- it is NOT the objective those methods actually
+minimize (Huber/Tukey minimize a robust loss, not squared error). It
+remains a useful, comparable summary of how much of y's variation the
+fitted curve captures, consistently computed the same way across
+methods and across polynomial degrees, which is what makes it useful
+for comparison even though it isn't each method's "native" fit
+statistic.
+"""
+
+import numpy as np
+import pandas as pd
+
+from .trend import fit_huber_trend, fit_tukey_trend, fit_ols_trend, predict_trend
+
+_FIT_FUNCTIONS = {
+    "huber": fit_huber_trend,
+    "tukey": fit_tukey_trend,
+    "ols": fit_ols_trend,
+}
+
+
+def goodness_of_fit(x, y, degree=2, method="huber"):
+    """
+    Fit a trend and report how well it explains the variation in y.
+
+    Returns r_squared (1 - var(residuals)/var(y)), plus RMSE and MAE
+    of residuals -- reported alongside R² since a single R²-style
+    number can hide whether errors are dominated by a few large misses
+    or spread evenly across observations.
+    """
+    x = np.asarray(x, dtype=float)
+    y = np.asarray(y, dtype=float)
+
+    fit_fn = _FIT_FUNCTIONS[method]
+    fit = fit_fn(x, y, degree=degree)
+    y_pred = predict_trend(fit, x)
+
+    residuals = y - y_pred
+    var_y = np.var(y)
+    r_squared = 1 - np.var(residuals) / var_y if var_y > 0 else 0.0
+
+    return {
+        "method": method,
+        "degree": degree,
+        "r_squared": float(r_squared),
+        "rmse": float(np.sqrt(np.mean(residuals ** 2))),
+        "mae": float(np.mean(np.abs(residuals))),
+    }
+
+
+def compare_polynomial_degrees(x, y, degrees=(1, 2, 3, 4), method="huber"):
+    """
+    Fit the same (x, y) data at several polynomial degrees and report
+    goodness_of_fit for each, so the right degree can be chosen
+    empirically rather than assumed.
+
+    A quadratic trend (the package default) is a reasonable starting
+    point for many relationships, but not all -- some trends are more
+    volatile and need a higher degree to be captured honestly, while
+    an unnecessarily high degree risks fitting noise rather than a
+    real pattern. The r_squared_gain column shows how much each extra
+    degree actually buys you: a large gain going from degree 2 to 3
+    suggests the quadratic default is too restrictive for this trend;
+    a negligible gain suggests the extra complexity isn't earning its
+    keep.
+    """
+    rows = [goodness_of_fit(x, y, degree=d, method=method) for d in degrees]
+    df = pd.DataFrame(rows)
+    df["r_squared_gain"] = df["r_squared"].diff()
+    return df