robustkit 0.5.0__tar.gz → 0.7.0__tar.gz

robustkit-0.7.0/PKG-INFO

@@ -0,0 +1,1306 @@
+Metadata-Version: 2.4
+Name: robustkit
+Version: 0.7.0
+Summary: Practical tools for robust analysis of a single continuous relationship: trend fitting, stability checks, influence diagnostics, and bootstrap uncertainty.
+Author: Mikael Lundqvist
+License: MIT License
+        
+        Copyright (c) 2026 Mikael Lundqvist
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: scipy>=1.10
+Requires-Dist: matplotlib>=3.7
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Provides-Extra: excel
+Requires-Dist: openpyxl>=3.1; extra == "excel"
+Dynamic: license-file
+
+# robustkit
+
+Practical tools for robust analysis of a single continuous relationship:
+y as a function of one continuous x.
+
+The guiding idea: **a conclusion that survives multiple fitting methods
+is more trustworthy than one that only holds under a single model.**
+`robustkit` makes it easy to compare Huber, Tukey biweight, and OLS
+fits side by side, identify and quantify the influence of individual
+observations, and get honest, bias-corrected uncertainty estimates.
+
+## Status
+
+`robustkit.core` (trend fitting, stability, diagnostics, uncertainty,
+consistency checks), `robustkit.segmentation` (hierarchical grouping,
+per-segment analysis), `robustkit.information` (mutual-information
+feature ranking, quadrant classification, pairwise redundancy/synergy
+scoring), `robustkit.benchmark` (global-trend segment comparison,
+model-agnostic custom benchmarks, residual/deviation reporting, Excel
+export, Robustness Map), `robustkit.report` (analyst vs. publisher
+views, dispersion measures, combined Huber+IQR view),
+`robustkit.quantiles` (generic JSON-stat loading, published-quantile-
+trend visualization, and lognormal-calibrated reconstruction of
+individual-level data from aggregated summaries), and
+`robustkit.segment_awareness` (automatic hierarchical segmentation +
+analysis, no manual hierarchy construction required) are stable and
+tested.
+
+**What is new in 0.7.0:** see `CHANGELOG.md`. In short: a `reference`
+parameter for the outlier reports (`"model"`, `"curve"`, `"benchmark_curve"`),
+dual-reference outlier classification, benchmark screening and
+goodness-of-fit, pointwise stability, per-segment Huber+IQR export,
+several drilldown and quality reports, and `jackknife_cap` for faster
+bootstrap intervals on large segments. Calls written for 0.6.0 keep
+working.
+
+**Recent fixes from real-dataset validation:**
+- `rank_features`/`quadrant_report`/`rank_communicative_pairs` no
+  longer crash on pandas `Categorical` columns containing missing
+  values (found via OpenML's Boston Housing dataset).
+- `bootstrap_band` (and `plot_analyst_view`, which uses it) now
+  defaults to `n_boot="auto"`, scaling iterations down for large
+  datasets since each iteration refits a full Huber model -- found to
+  become impractically slow at n_boot=200 on a ~54,000-row dataset.
+  Pass an explicit integer to opt out and always use exactly that many
+  iterations.
+
+**Note on `information_efficiency`:** values can exceed 1.0 for
+continuous features. `mutual_information` is estimated on the
+full-resolution continuous values, while `entropy_bits` is computed on
+a binned version of the same feature (since `entropy()` expects
+categorical input). Binning discards information, so `entropy_bits` is
+a lower bound on the feature's true entropy -- an efficiency above 1.0
+signals that the feature carries more usable information than a coarse
+categorical summary of it would capture. This is expected behavior,
+not a bug.
+
+## Installation
+
+```bash
+git clone https://github.com/<your-username>/robustkit.git
+cd robustkit
+pip install -e ".[dev]"
+```
+
+## Quickstart
+
+```python
+import numpy as np
+from robustkit import (
+    fit_huber_trend, fit_tukey_trend, predict_trend,
+    model_stability_pct, cooks_diagnostic, cook_impact,
+    bootstrap_band, bca_bootstrap_ci,
+)
+
+# x: a single continuous predictor, y: a single continuous outcome
+x = np.random.default_rng(0).uniform(20, 60, 200)
+y = 1000 + 50 * x - 0.4 * x**2 + np.random.default_rng(1).normal(0, 500, 200)
+
+fit = fit_huber_trend(x, y, degree=2)
+y_pred = predict_trend(fit, x_new=[30, 40, 50])
+
+stability = model_stability_pct(x, y)
+print("Median % spread between Huber/Tukey/OLS:", stability["median_pct_diff"])
+
+diag = cooks_diagnostic(x, y)
+impact = cook_impact(x, y, diag["flagged_indices"])
+print("Median % change in curve if flagged points removed:", impact["median_pct_change"])
+
+band = bootstrap_band(x, y)
+ci = bca_bootstrap_ci(x, y, statistic_fn=lambda x_, y_: np.median(y_))
+```
+
+See `examples/quickstart_tutorial.py` for a complete, runnable walkthrough.
+
+## Where along x is the trend weak?
+
+`model_stability_pct` summarises the disagreement between Huber, Tukey and
+OLS over the whole x range, and `bootstrap_band` gives the sampling
+uncertainty of the Huber curve. `pointwise_stability_report` puts the two
+side by side at every grid point, together with the number of observations
+near that point, so that a thin region (say, few young employees) can be told
+apart from a region where the fitting method matters:
+
+```python
+from robustkit import pointwise_stability_report, plot_pointwise_stability
+
+report = pointwise_stability_report(x, y, n_boot=300)
+report[["x", "method_spread_pct", "band_width_pct", "local_n", "dominant"]]
+# dominant == "sampling": the bootstrap band is wider than the methods' spread
+# dominant == "method":   the methods differ by more than the band
+
+plot_pointwise_stability(x, y, report=report)   # curves, band and a rug of observations
+```
+
+`window_frac` (default 0.1) sets the width of the window used for `local_n`,
+as a fraction of the x range. The three curves, the band and `pct_spread` are
+the same numbers `model_stability_pct` and `bootstrap_band` return; nothing new
+is fitted. Requires statsmodels (Tukey), like `model_stability_pct`.
+
+## Trend growth rate and goodness of fit
+
+```python
+from robustkit import trend_derivative, goodness_of_fit, compare_polynomial_degrees
+
+fit = fit_huber_trend(df["age"], df["salary"])
+
+# Rate of change of the trend itself (e.g. "salary growth per year of
+# age"), not just its level
+rates = trend_derivative(fit, x=[30, 40, 50])
+
+# How well does this fit actually explain the variation in y?
+goodness_of_fit(df["age"], df["salary"], degree=2)
+
+# Don't assume a quadratic trend is always the right choice -- check
+# empirically whether a higher degree captures meaningfully more
+compare_polynomial_degrees(df["age"], df["salary"], degrees=(1, 2, 3, 4))
+```
+
+**Note:** x is standardized internally before building polynomial
+features (both here and throughout `robustkit.core`), since raw
+polynomial features become numerically unstable at higher degrees for
+realistic x scales (e.g. age^5 vastly outscales age^1). This is
+transparent to callers -- `predict_trend` and `trend_derivative` still
+take and return values in the original x scale.
+
+## Segmentation
+
+Run any `robustkit.core` analysis independently across subgroups of a
+larger dataset, with automatic fallback to coarser groupings when a
+finer one is too small to analyze reliably:
+
+```python
+from robustkit import hierarchical_segment, apply_by_segment, model_stability_pct
+
+hierarchy = [["department", "level", "status"], ["level", "status"], ["status"]]
+segmented = hierarchical_segment(df, hierarchy, min_size=20)
+
+report = apply_by_segment(
+    segmented, segment_col="segment_id", x_col="age", y_col="value",
+    analysis_fn=model_stability_pct,
+)
+```
+
+`apply_by_segment` works with any function shaped like
+`analysis_fn(x, y, **kwargs) -> dict` -- built-in ones
+(`model_stability_pct`, `cook_impact`, `bca_bootstrap_ci`, ...) or your
+own. Only scalar values in the returned dict end up in the report
+table; segments below `min_points` are skipped rather than causing an
+error.
+
+**Sanity-checking a segmentation before trusting it:**
+`segment_consistency_report` runs a small battery of checks per
+segment -- does it meet the recommended minimum size, and does fitting
+a Huber trend on it use every row (robust methods don't need outliers
+pre-removed, so a silently dropped row usually means a missing x/y
+value slipped through, not intentional filtering):
+
+```python
+from robustkit import segment_consistency_report
+
+segment_consistency_report(df, segment_col="department", x_col="age", y_col="salary", min_size=20)
+#   segment  n_total  n_valid_xy  n_dropped_missing_xy  size_ok  fit_ok  fit_error
+```
+
+## Feature ranking (information)
+
+Rank features by mutual information with a target, normalized by each
+feature's own entropy, and classify them into four quadrants:
+
+```python
+from robustkit import rank_features, quadrant_report, plot_feature_space
+
+ranking = rank_features(df, target="value")
+report = quadrant_report(df, target="value")   # adds a `quadrant` column
+plot_feature_space(df, target="value")          # same quadrants, visualized
+```
+
+`quadrant_report` and `plot_feature_space` always agree on quadrant
+assignment -- both route through the same thresholding logic.
+
+**Caveat:** default thresholds are the *median* mutual information /
+efficiency across the ranked features. With only a handful of
+features, this can put a genuinely weak feature in the same "high"
+half as a strong one, since roughly half of any list sits above its
+own median regardless of how large the actual gap is. Median
+thresholding becomes meaningful with a reasonably large feature set;
+for a handful of candidates, read the raw `mutual_information` /
+`information_efficiency` values directly rather than relying on the
+quadrant label alone.
+
+See `examples/information_tutorial.py` for a complete walkthrough.
+
+## Benchmarking against a global trend
+
+Compare each segment's observed outcome against what a benchmark model
+predicts, with bootstrap uncertainty on the difference -- answers
+"which groups deviate from the overall trend, and by how much?" rather
+than "how does the trend look overall?":
+
+```python
+from robustkit import segment_position_report
+
+# Default: a single global Huber trend on one continuous x
+report = segment_position_report(
+    df, segment_col="department", x_col="age", y_col="salary",
+)
+#   segment    n  observed_median  expected_median  difference  ci_lower  ci_upper  ci_available
+#   Finance  176        48339.70         47799.82      539.88    202.15   1031.01          True
+#        HR  174        45718.84         46647.39     -928.55  -1293.26   -580.36          True
+#        IT  250        47226.50         47126.91       99.59   -117.56    510.81          True
+```
+
+A segment's confidence interval crossing zero means no clear deviation
+from the benchmark; HR and Finance above don't cross zero, IT does.
+
+**Custom, model-agnostic benchmarks:** the default single-column Huber
+trend can be replaced with any richer model -- e.g. one using age,
+age-squared, job level, overtime status, and a reference cluster
+together, rather than a single x. Provide any object exposing
+`predict(dataframe) -> array`:
+
+```python
+report = segment_position_report(
+    df, segment_col="department", y_col="salary", benchmark_fit=my_richer_model,
+)
+```
+
+`segment_position_report` never inspects what the model uses
+internally -- it only calls `predict()`.
+
+**A pitfall worth knowing about if your custom model's `predict()`
+uses `pd.get_dummies()`:** `segment_position_report` and
+`plot_segment_vs_benchmark` both call `predict()` on ONE segment at a
+time -- data that, by construction, has only a single value in each
+of its defining categorical columns. Calling `pd.get_dummies()` fresh
+on data like that generates a dummy column only for categories
+actually present in that call, which for a single, fully-specified
+segment is often none at all; a `reindex(columns=..., fill_value=0)`
+step then silently fills in `0` where the correct value was `1`,
+corrupting every per-segment prediction without raising an error.
+Verified directly: an under-specified benchmark's segment-level
+finding came out with the wrong sign and a wildly different magnitude
+until this was fixed. The fix is to fit a `sklearn.OneHotEncoder` (or
+equivalent) ONCE during training and reuse that SAME fitted encoder
+inside `predict()`, rather than re-deriving dummy columns from
+whatever data `predict()` happens to receive -- a fitted encoder
+always knows the full set of training-time categories, regardless of
+how homogeneous a later single call's data is. (`mad_outlier_report`
+and `dual_reference_outlier_report` are NOT affected by this, since
+both call `predict()` once on the full, heterogeneous population
+rather than per segment.)
+
+**Small segments:** groups with fewer than `MIN_POINTS_FOR_CI` (default
+20) observations still get `observed_median` / `expected_median` /
+`difference`, but `ci_lower` / `ci_upper` are `NaN` and
+`ci_available` is `False` -- a BCa bootstrap confidence interval (which
+relies on a jackknife step) is not attempted for populations that
+small, since it can fail outright or become statistically meaningless.
+For segments at or above the threshold, the interval is a full BCa
+(bias-corrected and accelerated) bootstrap interval, via the same
+`bca_bootstrap_ci_by_index` primitive used elsewhere in the package --
+not a plain percentile bootstrap. That primitive's jackknife step
+(needed for the acceleration term) is inherently O(n) regardless of
+`n_boot`, which becomes the dominant cost on large real segments --
+verified directly on a ~185,000-row segment (minutes, unusable) --
+so above `jackknife_cap` observations (1000 by default), a random
+subsample of that many positions is left out one at a time instead of
+every single one. Confidence interval bounds from the capped and
+uncapped versions were verified to differ by roughly 0.000005 on that
+same real segment, while cutting its runtime to about 3 seconds; pass
+`jackknife_cap=None` to any function built on
+`bca_bootstrap_ci_by_index` (including `segment_contribution_report`
+and `segment_benchmark_drilldown_report`) to always use the full,
+uncapped jackknife instead.
+
+`plot_segment_vs_benchmark` is the visual counterpart to
+`segment_position_report`: a single summary difference (e.g. "ITS is
+$988 below benchmark") can arise from several different underlying
+patterns that look identical in a table -- the whole segment sitting
+a bit below throughout, only its younger members sitting far below,
+only its older members, or two opposing effects that happen to cancel
+out in the median. This plots, for one or more segments, the actual
+observations, each segment's own OBSERVED trend (via LOWESS -- a
+flexible, assumption-free local smoother, deliberately not a
+Huber/Tukey/OLS fit at a fixed degree, since the point here is a
+segment's raw shape without imposing one), and the benchmark's
+expected trend for that segment, letting you see exactly where
+across `x_col` a segment's divergence actually occurs:
+
+```python
+from robustkit import plot_segment_vs_benchmark
+
+plot_segment_vs_benchmark(
+    df, segment_col="JobFamily", segment_values=["ENG", "ITS"],
+    y_col="salary", x_col="age", benchmark_fit=model_c,
+)
+```
+
+Works with a custom, multivariate `benchmark_fit` (any
+`predict(dataframe)` object) the same way `segment_position_report`
+does -- other columns the model needs are held constant per segment
+(at that segment's own value) when building the benchmark curve, so a
+segment like `"ENG_P2_Yes"` gets its benchmark line computed at
+`P_niva="P2", OT="Yes"` throughout, not averaged across values that
+don't apply to it. Requires `statsmodels` (imported lazily, same as
+`fit_tukey_trend`) -- this is a diagnostic plot only, not a source of
+numbers other functions build on, which is why LOWESS is used here
+specifically rather than elsewhere in the package.
+
+`benchmark_goodness_of_fit` evaluates an ALREADY-FITTED benchmark's
+overall explanatory power -- r_squared, rmse, mae -- against the
+actual values in y_col:
+
+```python
+from robustkit import benchmark_goodness_of_fit
+
+benchmark_goodness_of_fit(df, y_col="salary", benchmark_fit=model_c, x_col="age")
+# {"r_squared": 0.997, "rmse": 895, "mae": 712}
+```
+
+This is a different question from screening, below: even a screen
+that finds nothing (every candidate's residual mutual information
+near zero) says nothing about how much the benchmark explains
+overall -- a benchmark can be "complete", in the sense that no
+obvious structure is left to add, while still explaining relatively
+little of `y_col`'s variance if the underlying noise is simply large.
+Unlike `core.goodness_of_fit.goodness_of_fit`, which fits its own
+trend internally and therefore only evaluates a single-column
+robustkit trend, this function evaluates a benchmark that's already
+been fit elsewhere -- a robustkit trend-fit dict, or any custom,
+potentially multivariate `predict(dataframe)` object -- and never
+re-fits anything itself. Verified directly: an intentionally
+under-specified (age-only, linear) benchmark scored r_squared=0.874;
+adding the true quadratic term and the two missing categorical
+drivers raised it to r_squared=0.997 on the same data.
+
+`benchmark_screening_report` screens a fitted benchmark for likely-missing
+structure -- answering "is this benchmark good enough, or is there
+something it's still missing?" -- without ever building or fitting a
+new benchmark model automatically. Its `result["fit"]` entry is exactly
+`benchmark_goodness_of_fit`'s output for the same benchmark, so the two
+questions -- "how much does it explain overall" and "what specific
+structure might it still be missing" -- are always available side by
+side, numerically identical to calling either function directly:
+
+```python
+from robustkit import benchmark_screening_report
+
+result = benchmark_screening_report(
+    df, y_col="salary", benchmark_fit=model_c, x_col="age",
+    candidate_cols=["JobFamily", "level"],
+)
+result["fit"]            # how much does the benchmark explain overall?
+result["polynomial"]     # does x_col's current degree capture its shape?
+result["interactions"]   # is there missing categorical structure or an interaction?
+```
+
+Two genuinely different questions, deliberately answered with two
+different tools rather than one unified ranking, because mutual
+information cannot distinguish them. `result["polynomial"]` re-runs
+`compare_polynomial_degrees` (Chapter 5) on the raw `(x_col, y_col)`
+relationship directly -- checking polynomial degree this way, rather
+than via mutual information, matters because MI is invariant to
+monotonic reparametrization: on a domain where `x_col` is always
+positive, `x_col`, `x_col**2`, and `x_col**3` are all monotonic
+transforms of each other, so MI-based screening would report them as
+carrying nearly identical information regardless of which degree
+actually fits -- verified directly, within about 2% of each other on
+a deliberately under-fit benchmark that clearly needed degree 2, not
+1. `result["interactions"]` instead fits the benchmark's own residual
+as the target of `rank_features` -- reliable here specifically because
+a genuinely missing categorical variable, or a combination of two
+(`"JobFamily x level"`, generated automatically for every pairwise
+combination of `candidate_cols` unless `interaction_pairs` is given
+explicitly), is not a reparametrization of anything already in the
+model, so MI detects it directly. Sorted by `information_efficiency`,
+not raw `mutual_information` -- verified directly: an interaction term
+can carry MORE raw mutual information than a simpler candidate while
+still ranking lower on efficiency, the same high-cardinality trap
+Chapter 8's `zip_code` example demonstrated. Every row is a candidate
+worth investigating further, not a confirmed missing term -- automatic
+interaction generation grows combinatorially and carries real
+multiple-testing risk, especially with many candidate columns or a
+modest sample size.
+
+## Reporting: residuals, individual deviations, batch runs, and Excel export
+
+Four functions built on the same benchmark contract as
+`segment_position_report`, for turning a benchmark into something a
+non-technical audience (or a spreadsheet) can use directly:
+
+```python
+from robustkit import (
+    residual_summary, deviation_report,
+    benchmark_report_suite, export_benchmark_excel,
+)
+
+# Per-segment residual SHAPE (not just the median difference):
+residual_summary(df, segment_col="job_family", y_col="salary", x_col="age")
+#   segment    n  median_residual  mad_residual  p10_residual  p90_residual
+
+# Individuals furthest from the benchmark, sorted by residual --
+# material for a conversation, not an automatic flag:
+deviation_report(
+    df, y_col="salary", x_col="age", top_n=50, id_cols=["employee_id"],
+)
+#   employee_id     actual   expected  residual
+
+# direction="negative" (default, furthest below), "positive" (furthest
+# above), or "two_sided" (largest |residual| either direction).
+
+# Run the same benchmark across several grouping columns at once,
+# reusing ONE fitted benchmark so results are directly comparable:
+reports = benchmark_report_suite(
+    df, group_columns=["gender", "job_family", "location"], y_col="salary", x_col="age",
+)
+# -> {"gender": DataFrame, "job_family": DataFrame, "location": DataFrame}
+
+# Every report as its own sheet in one workbook:
+export_benchmark_excel(reports, "salary_report.xlsx")
+```
+
+All four accept the same `benchmark_fit` / `x_col` contract as
+`segment_position_report` (default single-column Huber trend, or any
+custom model exposing `predict(dataframe)`).
+
+**Dependency-free Excel export:** `export_benchmark_excel` requires
+`openpyxl` (an optional dependency). In an offline/air-gapped
+environment where installing it isn't possible, use
+`export_benchmark_excel_no_deps` instead -- identical interface,
+implemented with only the Python standard library (writes valid
+`.xlsx` files via `zipfile` and OOXML templating directly, no
+third-party package required):
+
+```python
+from robustkit import export_benchmark_excel_no_deps
+
+export_benchmark_excel_no_deps(reports, "salary_report.xlsx")
+```
+
+Prefer `export_benchmark_excel` when `openpyxl` is available -- it's a
+more complete, better-tested implementation of the Excel format.
+
+## Robustness Map
+
+Classify features by how much a conclusion about their relationship
+with the target depends on (a) fitting method choice and (b) specific
+influential observations -- two genuinely different failure modes that
+a single diagnostic can miss:
+
+```python
+from robustkit import feature_robustness_report, plot_feature_robustness
+
+report = feature_robustness_report(df, target="value")
+#   feature  stability_pct  cook_impact_pct    quadrant
+#      CRIM          8.9             17.1     fragile
+#       AGE         16.8             15.5     fragile
+#        RM          4.9              0.1     robust
+#       TAX         22.2              1.4     structural_sensitivity
+
+plot_feature_robustness(report=report)
+```
+
+Four quadrants: **robust** (low spread, low impact), **structural
+sensitivity** (sensitive to fitting method, not to specific points),
+**data sensitive** (a few points drive the conclusion, method choice
+barely matters), **fragile** (both -- least trustworthy).
+
+`quadrant_report`/`plot_feature_space` (information) and
+`feature_robustness_report`/`plot_feature_robustness` (benchmark) both
+route through the same shared classifier, `robustkit.classify_quadrants`
+-- any future quadrant-based analysis in this package will too.
+
+## Analyst view vs. publisher view
+
+Two visualizations that look superficially similar but answer
+genuinely different questions:
+
+```python
+from robustkit import plot_analyst_view, plot_publisher_view, dispersion_ratio, iqr
+
+# "How confident are we in the trend estimate?" -- a bootstrap
+# confidence band that SHRINKS as sample size grows.
+plot_analyst_view(df["age"], df["salary"])
+
+# "How spread out are actual values in the population?" -- a median +
+# IQR band that does NOT shrink with more data, since it reflects
+# real dispersion, not estimation uncertainty. show_points defaults to
+# False, since this view is meant for publishing potentially sensitive
+# data (e.g. individual salaries) without exposing raw points.
+plot_publisher_view(df["age"], df["salary"])
+```
+
+This distinction matters in practice: with 20x more data (same
+underlying distribution), the analyst view's confidence band roughly
+halves in width, while the publisher view's IQR band stays essentially
+unchanged -- confirmed by the package's own test suite.
+
+Both accept an optional `title=None` to override the default title
+(e.g. `plot_analyst_view(x, y, title="Q3 salary review")`), as does
+`plot_quantile_trend`.
+
+`dispersion_ratio(y)` -- (Q3-Q1)/median -- and `iqr(y)` are available
+standalone for tabular reporting; `dispersion_by_bin(x, y, n_bins=10)`
+computes both across bins of a continuous x, e.g. to check whether
+dispersion (inequality) grows with age.
+
+## Segment awareness: automatic hierarchical grouping + analysis
+
+`segment_stability_report` and `segment_benchmark_report` build the
+hierarchical segmentation automatically from a flat, most-specific-
+first list of columns, then run an existing analysis within the
+result -- no separate `hierarchical_segment(...)` + `apply_by_segment(...)`
+preparation step required:
+
+```python
+from robustkit import segment_stability_report, segment_benchmark_report
+
+# hierarchy built automatically: [JobFamily, Level, OT] -> [Level, OT] -> [OT] -> ALL
+report = segment_stability_report(
+    df, x_col="age", y_col="salary",
+    segment_cols=["JobFamily", "Level", "OT"], min_size=20,
+)
+
+report = segment_benchmark_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"],
+    x_col="age", min_size=20,  # or benchmark_fit=my_custom_model
+)
+```
+
+Both add a `segment_level` column showing which tier of the hierarchy
+each reported segment actually landed on (0 = finest), so a fallback
+to a coarser grouping is visible rather than silent. These are pure
+convenience wrappers -- identical results to building the hierarchy
+by hand with `hierarchical_segment` and calling `apply_by_segment` /
+`segment_position_report` directly.
+
+`segment_quality_report` combines several independent robustness
+measures into a single row per segment -- model stability, Cook's
+impact, MdAPE, and IQR of residuals -- so one table shows how
+trustworthy each segment's conclusion is, rather than cross-
+referencing several separate function calls by hand:
+
+```python
+from robustkit import segment_quality_report
+
+segment_quality_report(
+    df, x_col="age", y_col="salary", segment_cols=["JobFamily", "Level", "OT"], min_size=20,
+)
+#   segment  n  skipped  median_pct_diff  max_pct_diff  n_flagged  cook_impact_pct  mdape  iqr_resid  segment_level
+```
+
+Each measure is computed and error-handled independently -- if
+model stability fails (e.g. Tukey needs statsmodels, which may not be
+installed), Cook's impact, MdAPE, and IQR are still reported for that
+segment rather than the whole row failing.
+
+`mad_outlier_report` flags individuals whose residual is an outlier
+relative to their OWN segment's typical spread (MAD), not the whole
+population -- built on the same automatic hierarchical segmentation as
+above, so even someone in a small segment is compared against a
+sensibly-sized reference group rather than an irrelevant one:
+
+```python
+from robustkit import mad_outlier_report
+
+report = mad_outlier_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age",
+    min_size=20, k=3.0, direction="negative", id_cols=["employee_id"],
+)
+#   employee_id     actual   expected   residual  residual_pct    segment  segment_level  segment_mad  threshold  flagged
+```
+
+`direction`: `"negative"` (default -- flag underperformance relative
+to the benchmark), `"positive"`, or `"two_sided"`. Flagging compares
+each residual to `k` MADs from its *own segment's* median residual
+(not literally zero), so a segment the benchmark is systematically
+biased for doesn't get every member flagged just for that bias.
+Segments with zero MAD (a degenerate case, usually a tiny segment
+where every residual happens to match) are treated as having an
+infinite threshold rather than flagging everyone in them.
+
+`reference`: `"model"` (default) flags against `benchmark_fit` (or, if
+none is given, a single Huber trend fit once across the whole
+population). `"curve"` instead fits a fresh Huber trend WITHIN each
+segment, on that segment's own data alone, and flags against THAT --
+no benchmark model involved at all. Use `"curve"` when the flagging
+itself, not just a chart, should match "how does this person compare
+to their immediate segment-mates" -- e.g. a report for someone who
+shouldn't need the underlying benchmark model explained to trust the
+numbers. `"benchmark_curve"` is a third option in between: like
+`"model"`, it starts from `benchmark_fit`'s raw predictions, but then,
+WITHIN each segment, fits a Huber trend to those predictions (not to
+the actual y values, as `"curve"` does) and flags against that smoothed
+curve -- the exact reference `export_outlier_pdf`'s `mode="benchmark"`
+draws, so pairing the two gives full consistency without discarding
+the benchmark model's other covariates the way `"curve"` does.
+`x_col` is required for all three, but `"curve"` and `"benchmark_curve"`
+need it even when a custom `benchmark_fit` would otherwise have made it
+optional. `mad_outlier_drilldown_report` takes the same `reference`
+parameter, with identical semantics.
+
+`export_outlier_pdf` renders one chart per segment -- built from the
+same segmentation and flagging as `mad_outlier_report` -- as a
+one-page-per-segment PDF, for visual verification alongside the
+numeric report:
+
+```python
+from robustkit import export_outlier_pdf
+
+export_outlier_pdf(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age",
+    path="outliers.pdf", min_size=20, k=3.0, id_cols=["employee_id"],
+    annotate_flagged_with="employee_id",  # optional: label flagged points directly
+)
+```
+
+`mode="benchmark"` (default) plots a Huber curve fit to the benchmark's
+own predicted values, not the raw values connected point-to-point --
+for a multivariate benchmark model (depending on more than `x_col`),
+raw expected values aren't a pure function of `x_col` within a
+segment, so connecting them can zig-zag sharply even for a perfectly
+sensible model. The Huber-fit curve smooths over that without becoming
+a different reference: it's still derived entirely from the
+benchmark's own predictions. `mode="local"` instead fits a fresh Huber
+trend within each segment on the actual observed values alone -- often
+easier to read for "where does this person sit relative to their
+immediate colleagues?"
+
+`mode` (what's drawn) and `reference` (what flagging actually uses)
+are independent choices -- `export_outlier_pdf` takes the same
+`reference="model"/"curve"/"benchmark_curve"` parameter as
+`mad_outlier_report`, and always computes flagging exactly the way
+that report would with matching arguments, so the chart and the
+numeric report never disagree. Pairing `mode="local"` with
+`reference="curve"`, or `mode="benchmark"` with
+`reference="benchmark_curve"`, gives a chart where the drawn curve and
+the flagging rule are the *same thing* -- no benchmark model to
+explain at all in the first case, or the full benchmark model's
+covariates preserved (just smoothed for display) in the second --
+useful for charts handed to someone who should just be able to look
+and trust it. Other combinations -- e.g. `mode="benchmark"` with
+`reference="curve"` -- are allowed too (draw the benchmark's smoothed
+curve for context, but flag against each segment's own local trend);
+the info box always states plainly which curve is drawn and which
+reference the flagging rule actually used, so no combination is
+ambiguous on the page itself.
+
+`segment_mode="exclusive"` (default) vs. `segment_mode="drilldown"`
+offers the same choice as `export_huber_iqr_pdf`: one page per
+individual's single most-specific segment, or a page for every
+qualifying group at every hierarchy level independently (with the
+expected overlap -- the same individual can appear on multiple pages).
+
+Each page plots every observation in that segment, the reference
+curve, flagged outliers marked distinctly, and an info box (n, flagged
+count/percentage, the segment's MAD, the flagging rule, and which
+curve/reference is in play) so a reviewer can see *why* points are
+flagged directly from the chart, without cross-referencing the numeric
+report.
+Segments with fewer than `min_points_to_plot` (default 5) observations
+are skipped in the PDF -- a chart with a handful of points isn't
+meaningfully verifiable -- but still appear in `mad_outlier_report`'s
+numeric output. The idea: a numeric flag and a visual confirmation are
+two independent checks, and agreement between them is stronger
+evidence than either alone.
+
+### Drilldown reports: every hierarchy level at once, without exclusive assignment
+
+`segment_stability_report`, `segment_benchmark_report`, and
+`mad_outlier_report` each assign every individual to exactly ONE
+segment (their most specific grouping meeting `min_size`). That
+answers "what is the single most relevant reference population for
+THIS individual?"
+
+`segment_benchmark_drilldown_report` and `mad_outlier_drilldown_report`
+answer a different question -- "what does every granularity level look
+like on its own?" -- by reporting EVERY level of the hierarchy
+independently, without exclusive assignment. The same individual can
+appear in multiple rows (e.g. once in a `JobFamily x Level x OT` row,
+and again in the broader `Level x OT` row), whenever both groupings
+independently meet `min_size`:
+
+```python
+from robustkit import (
+    segment_benchmark_drilldown_report, mad_outlier_drilldown_report,
+    segment_stability_drilldown_report, outlier_drilldown_summary,
+)
+
+segment_benchmark_drilldown_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age", min_size=20,
+)
+mad_outlier_drilldown_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age", k=3.0,
+)
+segment_stability_drilldown_report(
+    df, x_col="age", y_col="salary", segment_cols=["JobFamily", "Level", "OT"], min_size=20,
+)
+```
+
+Both add a `segment_level` column, and the sum of `n` across rows will
+exceed the population size -- that's the expected signature of
+overlap, not a bug. Use the exclusive functions when you need to route
+each individual to one home; use the drilldown functions when you want
+to see every level side by side.
+
+**A conclusion that survives multiple reference populations is more
+trustworthy than one that only holds under a single, narrow
+definition of "expected".** `outlier_drilldown_summary` counts how
+many hierarchy levels each individual was flagged on:
+
+```python
+drilldown = mad_outlier_drilldown_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age",
+    k=3.0, id_cols=["employee_id"],
+)
+outlier_drilldown_summary(drilldown, id_col="employee_id")
+#   employee_id  outlier_levels          levels
+```
+
+An individual flagged at every level (finest grouping AND every
+broader fallback) is a stronger candidate than one that only appears
+as an outlier under one specific, narrow segmentation -- a Cook's-
+distance-style robustness check, applied to the choice of reference
+population rather than to individual data points.
+
+### Reference Population Sensitivity: a fourth robustness axis
+
+`robustkit` already checks robustness against fitting method
+(Huber/Tukey/OLS, `core`), individual observations (Cook's impact,
+`core.diagnostics`), and segment granularity (the drilldown functions
+above). `dual_reference_outlier_report` adds a fourth axis: robustness
+against WHICH reference population an individual is compared to.
+
+```python
+from robustkit import dual_reference_outlier_report
+
+report = dual_reference_outlier_report(
+    df, y_col="salary", x_col="age", segment_cols=["JobFamily", "Level", "OT"],
+    min_size=20, k=3.0, id_cols=["employee_id"],
+)
+#   employee_id  ...  local_flagged  global_flagged  reference_type
+```
+
+Each individual's residual is computed against two different
+references simultaneously:
+- **local** -- their own segment's Huber trend, fit fresh on just that
+  segment. Answers "how does this person deviate from their immediate
+  colleagues?"
+- **global** -- the benchmark model (default single-column Huber, or
+  a custom multi-column model). Answers "how does this person deviate
+  from the organization's expected structure?"
+
+`reference_type` classifies each individual: **"A"** (outlier under
+both -- the strongest candidates, low regardless of how "expected" is
+defined), **"B"** (outlier locally only -- often a tightly-clustered
+segment where a modest dip stands out among close peers but not
+against the wider population), **"C"** (outlier globally only --
+often a whole segment trending low while each individual is typical
+within it), or **"D"** (neither -- the normal case).
+
+**A subtlety that matters for what "global" actually means:** the
+global reference's MAD/median is computed across the ENTIRE
+population, not re-centered per segment. If it were centered per
+segment, a segment uniformly shifted below the benchmark would have
+that shift silently absorbed by the per-segment median subtraction --
+nobody in it would ever register as a global outlier, no matter how
+far the whole segment sits from the benchmark, since everyone would
+share roughly the same "typical" residual for their segment. This was
+found by testing, not designed in from the start: an earlier version
+reused `mad_outlier_report`'s per-segment MAD for the "global" side
+and it could never produce a genuine Type C, because per-segment
+recentering makes "global" behave like a second "local" view. It's
+also worth knowing that if segment-level shifts are large relative to
+individual noise AND affect a large share of the population, the
+population-wide MAD naturally inflates to accommodate that variation
+-- so a single shifted segment competing against several
+similarly-large ones may not clear the threshold either; Type C shows
+up most clearly when one segment's shift is a real minority pattern
+against an otherwise-homogeneous majority, not when most segments
+already differ substantially from each other.
+
+`dual_reference_outlier_drilldown_report` extends the same A/B/C/D
+classification to every hierarchy level at once, without exclusive
+assignment -- the drilldown counterpart, same overlap semantics as
+the other drilldown functions:
+
+```python
+from robustkit import dual_reference_outlier_drilldown_report
+
+dual_reference_outlier_drilldown_report(
+    df, y_col="salary", x_col="age", segment_cols=["JobFamily", "Level", "OT"], k=3.0,
+)
+#   ...  segment  segment_level  reference_type
+```
+
+The GLOBAL reference is still computed once across the whole
+population (not per level) for the same reason as above; only the
+LOCAL reference is refit fresh within each group at each level.
+
+### Where does the effect come from: segment_contribution_report
+
+`segment_benchmark_report`/`-drilldown_report` answer "how far from
+benchmark is this segment?" `segment_contribution_report` answers a
+different question: **where in the hierarchy does that gap actually
+arise?**
+
+```python
+from robustkit import segment_contribution_report
+
+segment_contribution_report(
+    df, y_col="salary", segment_cols=["JobFamily", "Level", "OT"], x_col="age", min_size=20,
+)
+#   segment  segment_level  ...  difference  parent_segment  contribution
+```
+
+Each segment's `contribution` is `child_difference - parent_difference`
+-- the part of its benchmark gap NOT already explained by its broader,
+coarser parent. A JobFamily within a Level+OT combination might show a
+large `difference` from benchmark simply because that whole Level+OT
+combination runs high or low (a STRUCTURAL effect, already visible one
+level up); `contribution` isolates whatever remains once that broader
+pattern is subtracted out -- the LOCAL effect specific to this finer
+grouping. Segments at the coarsest hierarchy level have no parent
+within `segment_cols`, so `contribution` is `NaN` and `parent_segment`
+is `None` for them.
+
+## Combined model + spread view
+
+`plot_analyst_view` and `plot_publisher_view` each show one thing --
+estimation uncertainty, or population spread -- deliberately kept
+separate. `plot_huber_iqr` shows both together: one or more trend
+curves overlaid with median + IQR error bars, plus an optional
+residual-quality box, matching the combined model-and-spread diagram
+style common in salary/wage analysis reporting:
+
+```python
+from robustkit import plot_huber_iqr
+
+result = plot_huber_iqr(df["age"], df["salary"], degree=2, bins=15)
+# result["grid"], result["huber"], result["binned"]
+```
+
+`show_points` defaults to `False`, consistent with `plot_publisher_view`.
+
+**Multiple curves, bootstrap bands, and full style control:**
+
+```python
+plot_huber_iqr(
+    df["age"], df["salary"],
+    methods=("huber", "tukey", "ols", "median_ensemble"),  # overlay all four
+    show_bootstrap_band=True, bootstrap_levels=(95, 50),    # nested confidence bands
+    cap_style="manual",         # hand-drawn boxplot-style Q1/Q3 "hats" instead of matplotlib's default caps
+    residual_box_metric="mdape",  # MdAPE + IQR(resid) instead of R^2/MAE/RMSE
+    ylim="dynamic",              # y-limits set from the data (min*0.95, max*1.05)
+    style={
+        "huber_line": {"color": "red", "linewidth": 2, "linestyle": "-", "label": "Huber poly(2)"},
+        "iqr_color": "black", "cap_width": 0.15,
+    },
+)
+```
+
+`methods` selects which trend curve(s) to draw (`"tukey"` and
+`"median_ensemble"` -- the pointwise median of Huber/Tukey/OLS --
+require statsmodels). `style` overrides individual colors, line
+widths, and other visual details without needing to touch anything
+else; every new parameter here defaults to the original, simpler
+single-Huber-curve appearance, so existing calls are unaffected.
+
+**Two grouping strategies:** `grouping="bin"` (default) uses quantile-
+based binning for stable estimates even in small populations.
+`grouping="unique"` instead groups by each EXACT x value (e.g. every
+individual age in years) -- matching a workbook-style `groupby(x)`
+aggregation -- and, when a given x value has fewer than
+`min_n_for_iqr` (default 5) observations, omits its IQR error bar
+entirely rather than showing an unreliable one:
+
+**Caveat, found via validation against a real dataset:** `grouping="unique"`
+only makes sense for x values with natural repetition (e.g. integer
+ages) -- for a genuinely continuous, high-precision measurement (e.g.
+carat weight to several decimal places), nearly every x value is
+unique, so almost nothing meets `min_n_for_iqr` and the result shows
+no IQR bars at all. Use `grouping="bin"` (the default) for
+high-precision continuous x; reserve `grouping="unique"` for x values
+that naturally repeat.
+
+```python
+plot_huber_iqr(
+    df["age"], df["salary"], grouping="unique", min_n_for_iqr=5,
+)
+```
+
+The absence of an error bar at a given age is itself information --
+it signals the sample at that exact value is too small to say
+anything about spread, not just a plotting simplification.
+
+**Per-segment PDF export:** `export_huber_iqr_pdf` renders one
+`plot_huber_iqr` chart per segment -- built from the same automatic
+hierarchical segmentation as the rest of `segment_awareness` -- as a
+one-page-per-segment PDF, suitable for sharing with the people each
+chart describes:
+
+```python
+from robustkit import export_huber_iqr_pdf
+
+export_huber_iqr_pdf(
+    df, x_col="age", y_col="salary", segment_cols=["JobFamily", "Level", "OT"],
+    path="segments.pdf", min_size=20,
+)
+```
+
+Domain-specific segment naming (e.g. translating an internal code like
+`"ENG_P3_Yes"` into a readable label such as "Engineers, level 3,
+overtime-eligible") stays entirely outside the package via an optional
+`title_fn(segment_name, info) -> str` callback, where `info` is
+`{"n", "segment_level"}`:
+
+```python
+def pretty_title(segment_code, info):
+    return f"{my_own_translation(segment_code)} (n={info['n']})"
+
+export_huber_iqr_pdf(..., title_fn=pretty_title)
+```
+
+`export_outlier_pdf` accepts the same `title_fn(segment_name, info)`
+callback for symmetry, with `info` there being `{"n", "n_flagged",
+"pct_flagged", "k", "segment_mad", "segment_level"}` -- useful if
+outlier review PDFs also need domain-specific titles, even though for
+many teams that PDF stays internal and doesn't need one. Both default
+to a generic title when `title_fn` is omitted, so existing calls are
+unaffected.
+
+**`mode="exclusive"` (default) vs. `mode="drilldown"`**, matching the
+same distinction as the report functions above: `"exclusive"` assigns
+each individual to exactly one, most-specific segment -- one page per
+final segment, no overlap. `"drilldown"` instead renders a page for
+EVERY qualifying group at EVERY hierarchy level independently -- e.g.
+a page for `"ENG_P3_Yes"` AND a separate page for the broader
+`"P3_Yes"`, which includes those same individuals alongside everyone
+else at that level. The same individual can appear on multiple pages
+in drilldown mode -- that's the expected signature of overlap, same as
+the drilldown reports:
+
+```python
+export_huber_iqr_pdf(..., mode="drilldown")
+```
+
+All of `plot_huber_iqr`'s style parameters (`methods`, `show_points`,
+`show_bootstrap_band`, `cap_style`, `residual_box_metric`, `ylim`,
+`style`, `show_undersized_points`, ...) pass straight through `export_huber_iqr_pdf` to each
+segment's page -- including `show_points`, which switches between
+"publisher" charts (the default, `show_points=False`, no raw scatter)
+and "analyst" charts (`show_points=True`, individual observations
+visible) without needing to fall back to a manual per-segment loop.
+
+**`export_huber_iqr_images`** is the same function in every respect
+except where the output goes: one image file per segment in a
+directory, instead of one combined PDF. Same segment_cols, same
+`mode`, same `title_fn`, same style parameters -- swap the function
+name and `path` for `output_dir` to get individual files instead of
+pages:
+
+```python
+from robustkit import export_huber_iqr_images
+
+export_huber_iqr_images(
+    df, x_col="age", y_col="salary", segment_cols=["JobFamily", "Level", "OT"],
+    output_dir="salary_plots", mode="drilldown",
+)
+```
+
+Useful whenever per-segment charts get consumed individually --
+dropped into slide decks one at a time, browsed in a file explorer,
+or fed into some other pipeline expecting separate image files --
+rather than as a single document. Filenames are derived from each
+segment's id or label, sanitized for the filesystem; a numeric suffix
+is appended if two segments would otherwise collide. Returns the list
+of saved file paths, in the order rendered.
+
+## Loading published quantile tables (SCB / JSON-stat)
+
+Some statistics agencies (e.g. Statistics Sweden, SCB) publish
+quantiles (Q1/median/Q3) directly, with no individual-level data
+available at all. `robustkit.quantiles` loads these tables generically
+via JSON-stat, a standardized dimensional-data format used by SCB and
+other national statistics agencies -- avoiding the fragility of
+parsing metadata out of column-name strings in a wide CSV export.
+
+```python
+from robustkit import load_json_stat, plot_quantile_trend, quantile_trend_dispersion
+
+df = load_json_stat("some_scb_table.json")
+
+# A real SCB quirk this loader does NOT try to guess automatically:
+# category labels can change meaning over time (e.g. Sweden's oldest
+# working-age bracket was labeled "65-66 år" through 2022 and
+# "65-68 år" from 2023, following a pension-age reform). Merge such
+# cases explicitly:
+df = load_json_stat(
+    "some_scb_table.json",
+    rename_categories={"ålder": {"65–68 år": "65–66 år"}},
+)
+
+# Once reshaped to a wide table with q1/median/q3 columns:
+plot_quantile_trend(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+quantile_trend_dispersion(wide_df, x_col="år", q1_col="q1", median_col="median", q3_col="q3")
+```
+
+This is the "quantiles are already given" case. A complementary case
+-- reconstructing approximate individual-level data from aggregated
+group means, for when only summary statistics (not quantiles) are
+available -- is planned as a follow-up (`robustkit.quantiles.reconstruct`).
+
+See `examples/quantiles_tutorial.py` for a complete walkthrough.
+
+## Reconstructing individual-level data from aggregated summaries
+
+For the complementary case -- only aggregated group summaries (n,
+Q1, median, Q3) are available, not the quantile trend itself as the
+final answer, and you want to run `robustkit.core` analyses as if
+individual data existed:
+
+```python
+from robustkit import expand_aggregated_table, check_reconstruction_quality, fit_huber_trend
+
+# One row per group (e.g. year), with n/q1/median/q3 columns
+synthetic = expand_aggregated_table(
+    summary_df, n_col="n", q1_col="q1", median_col="median", q3_col="q3",
+    group_cols=["year"], value_name="salary",
+)
+
+# Now usable exactly like real individual-level data:
+fit = fit_huber_trend(synthetic["year"], synthetic["salary"])
+```
+
+Method: a lognormal distribution is calibrated (via the IQR) to match
+each group's reported Q1/median/Q3, then `n` synthetic values are
+drawn from it. Validated end-to-end against real published SCB salary
+data: a Huber trend fitted on reconstructed pseudo-individual data
+tracked the true published median trend within 2% across 12 years.
+
+**Note on what this recovers:** because a Huber (or Tukey) fit on
+right-skewed reconstructed data tracks something close to the
+*median* trend it was calibrated against -- not the arithmetic mean --
+this is consistent with, not a limitation of, the reconstruction
+method. To target the mean instead, fit on `log(value)` and
+exponentiate predictions back, which approximates the geometric mean.
+
+Always check `check_reconstruction_quality()` before trusting a
+reconstruction: real Q1/median/Q3 triples aren't always perfectly
+consistent with a pure lognormal shape.
+
+**Warning -- unbounded tail at large n:** a lognormal has no natural
+upper limit, and its expected maximum grows with n. Reconstructing at
+the TRUE group size from a national table (SCB salary tables can
+report n in the hundreds of thousands to millions) can produce
+implausibly extreme tail values -- real salaries have practical
+ceilings a pure lognormal doesn't know about. This package's own
+examples and tests deliberately scale n down to a few thousand for
+demonstration; calibration quality (matching Q1/median/Q3) doesn't
+depend on reproducing the true population size, but tail plausibility
+does. No clipping is applied automatically.
+
+### When only a mean is available (no quantiles at all)
+
+Some tables (e.g. SCB's age-breakdown salary tables) report only a
+mean per group, with no spread information. Two deliberately separate
+methods are provided, each making a different explicit assumption --
+compare them rather than silently picking one:
+
+```python
+from robustkit import (
+    expand_aggregated_table_flat, expand_aggregated_table_borrowed_dispersion,
+    compare_reconstruction_methods,
+)
+
+# Method 1: repeat the mean n times -- zero within-group spread.
+# Equivalent to weighting each group by n (see "What the flat method
+# actually is", below); recovers between-group regression coefficients
+# (validated in the original technique this is based on) but
+# understates individual-level variation.
+flat = expand_aggregated_table_flat(df, n_col="n", mean_col="mean_salary", group_cols=["age"])
+
+# Method 2: borrow a dispersion_ratio from a DIFFERENT table that does
+# report quantiles, and use it to imply an approximate spread around
+# the given value, which becomes the reconstruction's MEDIAN (see
+# "What the borrowed-dispersion method does with `mean_col`", below).
+# Stacks two assumptions (mean-as-median, and that the borrowed ratio
+# transfers to this population) -- illustrative, not a substitute for
+# genuine quantile data for this specific table.
+borrowed = expand_aggregated_table_borrowed_dispersion(
+    df, n_col="n", mean_col="mean_salary", dispersion_ratio=0.45, group_cols=["age"],
+)
+
+# Compare both for a single group directly:
+compare_reconstruction_methods(n=2000, mean=52200, dispersion_ratio=0.45)
+```
+
+**What the flat method actually is.** Repeating a group's number *n*
+times is exactly the same as weighting that group by *n*. For an
+ordinary least-squares fit whose regressors are constant within each
+group (an age band, say), a fit on the reconstructed rows is identical
+to a fit on the group values weighted by *n* -- and therefore to a fit
+on the original individuals; in a synthetic check, a slope of 1.948
+against 1.950. For a Huber fit it matches a Huber fit on the group
+values weighted by *n* (2.029 against 2.029). robustkit's `fit_*_trend`
+functions take no weights, so this is how to obtain *n*-weighting here.
+Treat it as a workaround rather than a feature: it adds no information
+beyond the group values and sizes. Two practical notes:
+
+- It answers questions about whatever the table's number is. A table of
+  means supports questions about the mean; a table of medians,
+  questions about the typical value. Asking the other question gives
+  large errors -- in the synthetic check, +75% (a means table asked
+  about the typical value) and -37% (a medians table asked about the
+  mean). The `mean_col` argument accepts a median if that is what the
+  table gives.
+- Only the relative group sizes matter, so if the true *n* is very large
+  (national tables can report millions per group), scale every *n* down
+  by a common factor before reconstructing. Scaling by 0.1 changed a
+  fitted slope only in the fourth decimal in the same check; scaling by
+  0.01, which rounds the smallest groups to a few rows, moved it by
+  under half a percent. This applies to the flat method only -- in
+  `expand_aggregated_table` and the borrowed-dispersion reconstruction
+  the number of draws changes the tail, not just the weight.
+
+**What the borrowed-dispersion method does with `mean_col`.** The
+reconstruction's *median* is set to that value, whatever the column is
+called. If the table publishes a median, pass it: the first assumption
+above (mean-as-median) then does not apply, and the reconstruction
+reproduces the median (a published 10.4 came back as 10.3 in a
+synthetic check). If the table publishes a mean, two things follow:
+
+- The reconstruction does not reproduce the mean it was given. A
+  lognormal's mean lies above its median, so the reconstructed mean is
+  about `mean * exp(sigma**2 / 2)`, with
+  `sigma = ln((1 + r/2) / (1 - r/2)) / (2 * 0.6745)` for a borrowed
+  ratio `r`. For `r` = 0.4, 0.8 and 1.2 that is roughly 5%, 22% and 70%
+  above the published mean (15.0 published, 18.3 reconstructed at
+  `r` = 0.8 in the synthetic check).
+- How far a mean sits above its median depends on the spread, so the
+  error this stand-in introduces differs between groups whose spread
+  differs (in the same check the mean was 18% above the median in the
+  youngest age band and 45% above it in the oldest).
+
+Both methods are documented with their specific assumptions rather
+than presented as equally valid defaults -- being explicit about which
+assumption was made lets the analyst judge how much a conclusion
+depends on it, rather than presenting an assumption as a measurement.
+
+## Feature pairing (information)
+
+Beyond ranking single features, evaluate *pairs* of features together:
+how redundant are they with each other, and does knowing one reveal
+additional predictive value in the other (synergy, e.g. an interaction
+effect)?
+
+```python
+from robustkit import (
+    conditional_mutual_information, communication_score,
+    rank_by_communication, pair_redundancy, pair_synergy,
+    rank_communicative_pairs,
+)
+
+# How communicable is a single feature -- not just predictive, but
+# suitable for a clear chart/table (adequate group sizes, homogeneous
+# groups, few enough categories to show at once)?
+comm_ranking = rank_by_communication(df, target="value")
+
+# How much does region's relevance to the target change once
+# department is already known?
+synergy = pair_synergy(df, feature_1="department", feature_2="region", target="value")
+
+# Rank every candidate pair by combined relevance, penalizing
+# redundant pairs and rewarding genuine synergy
+pairs = rank_communicative_pairs(df, target="value")
+```
+
+All mutual-information-based quantities in this module (`rank_features`,
+`conditional_mutual_information`, `pair_redundancy`, `pair_synergy`,
+`communication_score`) are expressed in **bits**, consistent with
+`entropy()` -- internally, scikit-learn's MI estimators return nats
+and are converted before being used anywhere in this package.
+
+## Design principles
+
+- **One continuous x, one continuous y** at the core. This keeps every
+  function's output visually and numerically interpretable (a curve
+  you can plot, a band you can read).
+- **Diagnosis and action are separate steps.** `cooks_diagnostic`
+  flags candidates; `cook_impact` tells you whether removing them
+  actually changes anything.
+- **OLS is a reference point, not the enemy.** Comparing robust fits
+  against OLS is how you know whether robustness mattered at all.
+
+## Naming conventions
+
+A few parameter/column names look similar across the package but mean
+different things -- documented here explicitly so the difference reads
+as intentional, not as an inconsistency to "fix":
+
+- **`residual` vs. `difference`:** `residual` is an INDIVIDUAL-level
+  quantity (`actual - expected` for one row) -- used by
+  `mad_outlier_report`, `mad_outlier_drilldown_report`, and
+  `deviation_report`. `difference` is a SEGMENT/GROUP-level quantity
+  (typically the median residual within a group) -- used by
+  `segment_position_report`, `segment_benchmark_report`,
+  `segment_benchmark_drilldown_report`, and `benchmark_report_suite`.
+- **`target` vs. `y_col`:** `robustkit.information` uses `target` for
+  the column being explained, since it works with arbitrary features
+  (not necessarily a continuous regression outcome).
+  `robustkit.benchmark`, `robustkit.segment_awareness`, and
+  `robustkit.quantiles` use `y_col`, since they specifically model a
+  continuous `y` as a function of `x`.
+- **`segment_cols` vs. `group_columns`:** `segment_cols` (throughout
+  `robustkit.segment_awareness`) is an ORDERED, most-specific-first
+  list used to build a fallback HIERARCHY (see `hierarchical_segment`).
+  `group_columns` (`benchmark_report_suite`) is a FLAT list of
+  independent groupings, run separately with no hierarchy or fallback
+  between them. Different structure, different name on purpose.
+- **`min_size` vs. `min_points` vs. `min_stratum_size` vs.
+  `min_group_size`:** all mean "minimum group size," but at different
+  stages: `min_size` (`hierarchical_segment` and everything built on
+  it) gates whether a hierarchy LEVEL gets created at all;
+  `min_points` (`apply_by_segment`) gates whether an already-built
+  segment gets ANALYZED; `min_stratum_size`
+  (`conditional_mutual_information`) and `min_group_size`
+  (`communication_score`, `rank_by_communication`) are specific to
+  those `robustkit.information` calculations. Kept separate rather
+  than unified to one name, since collapsing them would obscure which
+  stage of a pipeline each threshold actually applies to.
+
+## License
+
+MIT -- see [LICENSE](LICENSE).