hdfe-stream 0.2.2__tar.gz → 0.4.0__tar.gz

hdfe_stream-0.4.0/CHANGELOG.md

@@ -0,0 +1,290 @@
+# Changelog
+
+## 0.4.0
+
+New features.
+
+- **`HDFEResult.predict()`** predicts on new data (a DataFrame, a LazyFrame, or
+  a Parquet path or glob) or on the fit's source, lazily, so the data can be
+  larger than memory. It returns the data with a prediction column: the linear
+  predictor, or with `type="response"` a GLM's mean. Rows with a fixed-effect
+  level the fit did not estimate get null. It supports `i()` covariates,
+  interacted fixed effects, varying slopes, and IV models.
+- **`HDFEResult.confint()`**: confidence intervals, pointwise or, with
+  `joint=True`, simultaneous over the selected coefficients, as pyfixest's.
+- **`HDFEResult.wald_test()`**: Wald tests of R beta = q, or that the
+  coefficients selected by `keep`/`drop` are zero, under the current vcov, with
+  F or chi-squared p-values. F is reported by default for any OLS hypothesis
+  (chi-squared for GLMs), with `df_t` denominator degrees of freedom, so a
+  one-coefficient test has `tidy()`'s p-value.
+- **`bonferroni()` and `holm()`**: multiple-testing corrections of one or
+  more coefficients' p-values across models (results, `HDFEMulti`, or lists
+  of them). Holm's step-down adjustment controls the familywise error rate as
+  Bonferroni's does, and never rejects less often.
+- **The wild cluster bootstrap: `bootcluster=`, `HDFEResult.wildboottest()`,
+  `rwolf()` and `wyoung()`.** With `bootcluster=`, the fit keeps, for each
+  cluster, the sums the bootstrap needs (about G x k^2 numbers, collected in the
+  residual pass it makes anyway; nothing without the option). After the fit,
+  `wildboottest()` runs the restricted or unrestricted wild cluster bootstrap of
+  any coefficient from those sums, without reading the data, and `rwolf()` and
+  `wyoung()` give Romano-Wolf and Westfall-Young adjusted p-values across models
+  on the same draws. One-way, OLS only, and every fixed effect must be nested
+  within the clusters (a fixed effect with few levels, such as year, can go in
+  as covariates instead).
+
+Fixes.
+
+- `cluster="firm_id"`, a single string, was read as a list of characters; it
+  now means one clustering, as documented.
+
+Performance.
+
+- **Cells are collapsed from sorted rows, without a hash aggregation.** Pass 0
+  now sorts each worker's rows by the other fixed effects' levels too, so that
+  every cell (one combination of levels) is a run of consecutive rows, and pass
+  1 collapses the runs as it reads the sorted rows, in a multithreaded kernel.
+  This replaces Polars' group_by over each slice of rows, and the cell table it
+  wrote to disk and read back. Where nearly every row is a cell of its own
+  (e.g., worker, firm and year effects on a worker-year panel), pass 0 sorts by
+  the streamed dimension alone. The new `collapse_cells` option (`"auto"`, the
+  default; True; False) overrides that choice, and
+  `diagnostics["cells_collapsed"]` reports it. Together with the faster
+  iterations below, this reduced compute time by 5-31% on large AKM
+  configurations, and peak disk by up to 40%, with the same or lower memory.
+- **Faster conjugate-gradient iterations.** The vector work of each iteration
+  (dot products, norms, and the updates of the solution, residual and search
+  direction) runs as fused, multithreaded kernels instead of single-threaded
+  numpy operations.
+- **Faster IRLS steps for Poisson, logit and probit.** Each step's read of the
+  rows (every row's linear predictor, the family's mean, weight and working
+  response, the deviance and log-likelihood, and the sums per cell) runs as one
+  fused, multithreaded kernel, instead of a dozen single-threaded numpy
+  operations over each chunk. On large regression benchmarks, compute time
+  declined by 11-32%.
+- **Pass 1 builds the Jacobi diagonal and step 2's right-hand side** chunk by
+  chunk while it has the cells in memory, instead of reading the identifying
+  cells back for each, when the variables are solved in one block (up to
+  `rhs_block` of them). Each of those reads is a pass over data as large as
+  the cell table, which matters most where the scratch disk is slow.
+- **A memory budget for per-thread accumulators.** `stream_cg` and the
+  leave-out passes give each thread its own level-sized accumulator, which
+  with many threads and many levels could take more memory than the rest of
+  the fit (64 threads, 30 million firm-years and 8 columns: about 120 GB,
+  allocated afresh every iteration). They now get one copy per thread only
+  while all of them fit in half of `max_s_gb`, with fewer threads on those
+  passes when that binds, and `stream_cg` allocates its copies once per
+  solve. Lowering `max_s_gb` tightens it.
+
+## 0.3.0
+
+New features.
+
+- **`connected_components`** finds the connected components of the graph of
+  two fixed-effect dimensions (either may be an interaction, `firm_id^year`)
+  by union-find over the rows as they stream, holding level-sized arrays
+  only. It returns a table per dimension giving each level its component,
+  numbered by size so that component 0 is the largest (ties go to the
+  component with the smallest level of the first dimension), and a table of
+  component sizes. `largest()` filters the data to the largest connected
+  set and `label()` adds each row's component, both lazily (on the data the
+  components were found in, or on other data passed to them). `min_degree`
+  removes levels with fewer distinct partners in the other dimension,
+  repeating until none is left, before the components are found: `(2, 1)`
+  keeps only movers, `2` the 2-core. `leave_one_out=True` prunes to the
+  leave-one-out connected set of KSS estimation instead. `rank_by` says
+  whether "largest" counts rows (the default) or the levels of one dimension.
+- **`connected=`, `min_degree=` and `connected_fe=`** in `feols_stream`,
+  `fepois_stream`, `feglm_stream` and `StreamingHDFE` restrict the fit to a
+  pair of fixed effects (`connected_fe`, by default the first two listed),
+  among the rows with no missing values and before singletons are dropped:
+  `min_degree` prunes levels with too few distinct partners, then
+  `connected="largest"` keeps the largest connected set of what is left, or
+  `connected="leave_one_out"` the leave-one-out connected set, as
+  `connected_components` does. `sample()` reports the rows they drop as
+  `"min_degree"`, `"disconnected"` and `"leave_one_out"`, and
+  `diagnostics["connected"]` and the summary count them.
+- **Python 3.15** is supported and tested in CI. pyfixest, which the
+  `formula` extra needs, has no 3.15 wheels yet (0.60.0), and building it
+  from source needs Rust and `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1`; the
+  rest of the package installs from wheels.
+
+Changes that can affect existing code.
+
+- **`leave_one_out_connected` and `LeaveOutSet` are removed.** Use
+  `connected_components(data, "worker_id + firm_id", leave_one_out=True)`,
+  whose `.largest()` is the pruned panel, or `connected="leave_one_out"`
+  in a fit. The set is the same: on simulated panels with many components,
+  workers observed once and several rounds of cut workers, the workers and
+  firms kept are identical. It is found 5-6 times faster (38s against 218s on
+  150 million rows), and a null worker or firm id no longer makes it fail.
+- **`leave_out_kss` prunes in the fit**, with `connected="leave_one_out"`, so
+  the leave-one-out connected set is that of the rows the model can use:
+  before, rows with a missing outcome or covariate were part of the network
+  being pruned. `lo.pruning` is now a dict of the pruning's counts (the
+  fit's `diagnostics["connected"]["leave_one_out"]`), and the rows it dropped
+  are those `lo.fit.sample()` labels `"leave_one_out"`.
+- **The `component` column of `fixef()` is numbered by size**, so component
+  0 is the largest (by rows in the estimation sample), as in
+  `connected_components`. Ties go to the component with the smallest level of
+  that dimension. It was numbered by each component's smallest level.
+- **Removed `hdfe_stream.iter_group_chunks` and `StreamingHDFE.from_formula`.**
+  The first read only the fit's internal row files, sorted by its internal
+  group code, and stays internal; the second duplicated `feols_stream`.
+
+Fixes.
+
+- **The leave-out operator's null space is right with several connected
+  components.** `apply_inverse` built one null vector for every number up to
+  the largest component label, which with several components left gaps, and
+  each gap added a spurious vector. Leave-out estimation runs on a connected
+  set and was not affected.
+
+Documentation.
+
+- **A documentation site**, at
+  [leetucker.github.io/hdfe-stream](https://leetucker.github.io/hdfe-stream/),
+  built with Sphinx from `docs/` and published by a new GitHub Actions
+  workflow. It has a page for each example, a section on leave-out variance
+  components (bringing together `docs/kss.md`, the methodological differences
+  and a derivation of the correction), a page for each benchmark, a function
+  reference built from the docstrings, and this changelog. `docs/kss.md`,
+  `docs/glm.md` and `docs/kss_methodological_differences.md` moved into it,
+  and the README is shorter, pointing to it. The public docstrings are now
+  numpydoc, documenting every option, including `tol`, `maxiter`,
+  `row_group_size` and `log_level` of `StreamingHDFE`, and the attributes of
+  `HDFEResult` and the other result classes. `pip install
+  "hdfe-stream[docs]"` installs what building it needs.
+- **New example, `connected_sets.py`**, and additions to the others:
+  `sample()` and joining the residuals back on `row_id`, `fe_dof`, and CRV3
+  (`formulas.py`); `summary_json` (`quickstart.py`); `keep=` and a
+  decomposition by group (`akm_variance.py`); and `StreamingGLM` (`glm.py`).
+
+## 0.2.2
+
+- **Polars 2.0 support.** `fit.sample()` returns the source's rows in the source's
+  order again. Polars 2.0 collects with the streaming engine by default, whose
+  joins do not keep row order unless asked, so the rows came back shuffled.
+  Under Polars 2.0, repeated fits of the same model can differ in the last
+  digit or so (around 1e-16 relative): its streaming engine adds floating-point
+  sums in an order that depends on thread scheduling. Under Polars 1.x they
+  are identical bit for bit.
+- **Dropping singletons no longer inflates memory with a wide design.** The
+  dropped levels were anti-joined after the design was evaluated, so the
+  streaming join buffered every design column. The rows are now dropped from
+  the source columns before the design is evaluated.
+- **Finding singletons takes fewer scans.** The search used to stop only
+  after a full round over the fixed effects found nothing. It now stops once
+  every fixed effect has been checked since the last drop, since a dimension's
+  own drop cannot leave a singleton in it: four scans instead of six for a
+  typical three-way model, about 40% less time spent on singletons, and the
+  same memory. The rows dropped are the same.
+- **Sorting the buckets and building the cell table takes less memory under
+  Polars 2.0.** Each bucket is sorted with Polars' in-memory engine and then
+  written, rather than sunk through the streaming engine, whose sort peaked
+  about 1.5x higher. Similarly, Pass 1 grouped each bucket with a streaming
+  `group_by`, which under Polars 2.0 gives every thread its own hash table: with
+  as many cells as rows, as in an AKM panel, the fit's peak more than doubled.
+  The rows are sorted by fe[0] group, so each bucket is now grouped a slice of
+  whole groups at a time and the cells appended in order. This reduces memory
+  use in AKM specifications and leaves wide designs unchanged or a little
+  better.
+- **The explicit solver builds its matrix faster and in less memory.** The
+  reduced matrix S is now written directly in compressed form, one row at a
+  time, instead of being assembled from chunks of coordinate triples that
+  were merged with scipy. Each row's size is counted first, so S is allocated
+  once and nothing larger than S is held while it is built; `max_s_gb` is
+  checked before S is allocated rather than partway through. Estimates are
+  unchanged up to rounding.
+- **Removed `triple_budget` and `dense_max_levels`** from `StreamingHDFE`. They
+  tuned the old way of building S and have no counterpart in the new one.
+- **Importing hdfe_stream costs a little less memory.** p-values, critical
+  values and the weak-identification interval use `scipy.special`'s
+  distribution functions instead of `scipy.stats`, which is no longer
+  imported. The values are the same.
+- **Empty run directories no longer pile up on NFS.** A file deleted while
+  still open or memory-mapped becomes a hidden .nfs file on NFS, so its run
+  directory could not be removed and was left empty once the process exited.
+  A failed removal is now retried after collecting garbage on every platform
+  (it was Windows only); the leave-out functions release their memory maps as
+  soon as they are done (`reload_intermediates()` maps them back in); and
+  empty run directories more than a minute old are removed by every new fit
+  in the same working directory and by `hdfe_stream.cleanup()`.
+- **Slightly less verbose default logging.** CG iteration messages are now
+  logged at DEBUG level and therefore not printed by default. Lower your log
+  level to obtain this information.
+
+
+## 0.2.1
+
+- **`fe_dof="exact"` no longer depends on the order of the fixed effects.**
+  With three or more fixed effects it counted the connected components of the
+  streamed dimension and whichever came next in the formula, so reordering the
+  formula could change the standard errors. It now counts the components of
+  every pair of dimensions and uses the pair with the most, which is never
+  more than the true number of redundant levels. With two fixed effects
+  nothing changes, except in the clustered case below.
+- **Clustered standard errors with fixed effects nested in the clusters.**
+  Under `fe_dof="exact"`, the components of a pair of fixed effects that are
+  both nested within the clusters were subtracted from K, although those
+  fixed effects' levels are already dropped from it. K could then fall below
+  the number of covariates and the standard errors were too small. Such a pair
+  is no longer used, and with no other pair K is pyfixest's.
+- **`fit.sample()` labels every singleton.** With three or more fixed effects
+  that each had singleton levels, rows dropped only for the third or a later
+  dimension came back with `in_sample=True`, so the counts disagreed with
+  `n_obs` and the summary's singleton count. The fit itself was right. GLM
+  `"separation"` labels had the same fault.
+- **A NaN in a fixed-effect or cluster column is missing**, as a null is,
+  rather than a level of its own. For fixed effects this matches pyfixest;
+  for clusters, where pyfixest raises an error, the row is dropped as it is
+  for a null cluster. `sample()` reports these rows as `"missing"`.
+- **Polars 1.41 is excluded** from the supported versions. Its streaming join
+  panics (`called Option::unwrap() on a None value`) once a frame has about
+  130 columns, which a model with more than about 120 covariates reaches in
+  pass 0. Polars 1.40 and earlier, and 1.42 and later, are unaffected.
+- **A panic inside Polars is reported as a `RuntimeError`** that names the
+  Polars version, says it is a Polars bug, and suggests a fix, instead of a
+  bare `PanicException`.
+
+## 0.2.0
+
+Changes to defaults, so results can differ from 0.1.0 without any change to
+your code.
+
+- **Default `vcov` is now `"iid"`**, with or without fixed effects, matching
+  pyfixest 0.60. Before, a model with fixed effects defaulted to CRV1 clustered
+  on the first fixed effect. Pass `vcov={"CRV1": "worker_id"}` for the old
+  behavior.
+- **Singleton observations are dropped by default** (`fixef_rm="singleton"`),
+  as in pyfixest, repeating until none is left. `fixef_rm="none"` keeps them.
+  `leave_out_kss` fits with `fixef_rm="none"`, since its pruning defines the
+  sample.
+
+New features.
+
+- **`row_id` in `resid()`**: each row carries its position in the source before
+  any row was dropped, so residuals join back onto the source (`row_id=` renames
+  the column).
+- **`fit.sample()`** returns the source, lazily, with `row_id`, `in_sample` and
+  `dropped_because` (`"missing"`, `"singleton"` or `"separation"`).
+  `multi.sample()` returns `{formula: frame}`.
+- **Polars `DataFrame` inputs** are accepted wherever a Parquet path or
+  `LazyFrame` was.
+- **Reserved column prefix `__hdfe_`.** All of the estimator's internal columns
+  now use it, and a source with a column starting with it is refused. This fixes
+  fits failing when a user column was named `w` (with weights), `v0`, `c1`,
+  `gcode` and similar.
+
+## 0.1.0
+
+First release.
+
+- `feols_stream`: out-of-core OLS with several high-dimensional fixed effects,
+  varying slopes, weights, IV, and CRV1/CRV3 and other variance estimators.
+- `fepois_stream` and `feglm_stream`: Poisson, logit and probit by iteratively
+  reweighted least squares over the same passes.
+- AKM variance decomposition and leave-out (KSS) bias correction with
+  standard errors and weak-identification diagnostics.
+- Reporting through pyfixest (`to_pyfixest()`, `etable`).
+- `summary_json()` and `summary_dict()`: the model-level information of
+  `summary()` as JSON or a dict, for saving to disk.

{hdfe_stream-0.2.2 → hdfe_stream-0.4.0}/CITATION.cff

@@ -7,8 +7,8 @@ authors:
   - family-names: Tucker
     given-names: Lee
     email: lee.c.tucker@census.gov
-version: 0.2.2
-date-released: 2026-10-08
+version: 0.4.0
+date-released: 2026-10-16
 license: MIT
 repository-code: "https://github.com/leetucker/hdfe-stream"
 url: "https://github.com/leetucker/hdfe-stream"

{hdfe_stream-0.2.2 → hdfe_stream-0.4.0}/MANIFEST.in

@@ -1,6 +1,7 @@
 include README.md CHANGELOG.md CITATION.cff LICENSE.md
 graft tests
 graft docs
+prune docs/_build
 prune benchmarks
 prune prototypes
 prune examples

hdfe_stream-0.4.0/PKG-INFO

@@ -0,0 +1,242 @@
+Metadata-Version: 2.4
+Name: hdfe-stream
+Version: 0.4.0
+Summary: Out-of-core regression with several high-dimensional fixed effects: OLS, Poisson, logit and probit, with AKM/KSS variance decompositions
+Author: Lee Tucker
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/leetucker/hdfe-stream
+Project-URL: Source, https://github.com/leetucker/hdfe-stream
+Project-URL: Issues, https://github.com/leetucker/hdfe-stream/issues
+Project-URL: Documentation, https://leetucker.github.io/hdfe-stream/
+Project-URL: Changelog, https://github.com/leetucker/hdfe-stream/blob/main/CHANGELOG.md
+Keywords: econometrics,fixed effects,high-dimensional,out-of-core,AKM,KSS,pyfixest
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Scientific/Engineering
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: polars!=1.41.*,>=1.37
+Requires-Dist: numpy>=1.24
+Requires-Dist: numba>=0.59
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: scipy>=1.11
+Provides-Extra: within
+Requires-Dist: within-py>=0.3.0; extra == "within"
+Provides-Extra: amg
+Requires-Dist: pyamg>=5.1; extra == "amg"
+Provides-Extra: formula
+Requires-Dist: pyfixest<0.61.0,>=0.60.0; extra == "formula"
+Requires-Dist: formulaic>=1.0; extra == "formula"
+Requires-Dist: pandas>=2.0; extra == "formula"
+Provides-Extra: all
+Requires-Dist: hdfe-stream[amg,formula,within]; extra == "all"
+Provides-Extra: benchmark
+Requires-Dist: hdfe-stream[formula]; extra == "benchmark"
+Requires-Dist: matplotlib>=3.7; extra == "benchmark"
+Provides-Extra: docs
+Requires-Dist: sphinx>=8.0; extra == "docs"
+Requires-Dist: myst-parser>=4.0; extra == "docs"
+Requires-Dist: furo>=2024.8.6; extra == "docs"
+Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
+Requires-Dist: sphinx-design>=0.6; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: hdfe-stream[all]; extra == "test"
+Requires-Dist: wildboottest>=0.3.2; extra == "test"
+Dynamic: license-file
+
+# hdfe-stream
+
+Regression with several high-dimensional fixed effects, on data that does not
+fit in memory: linear (`feols_stream`), and Poisson, logit and probit
+(`fepois_stream`, `feglm_stream`), with the Kline–Saggio–Sølvsten bias
+correction for AKM variance decompositions (`leave_out_kss`).
+
+```python
+from hdfe_stream import feols_stream
+
+fit = feols_stream(
+    "log_earn ~ age_squared + age_cubed | worker_id + firm_id + year",
+    "data/*.parquet",              # a Parquet path or glob, never fully loaded
+    workdir="scratch",             # where intermediates go: disk, not memory
+    vcov={"CRV1": "worker_id"},
+)
+fit.summary()
+```
+
+Formulas follow [pyfixest](https://github.com/py-econometrics/pyfixest)/fixest
+syntax and are parsed by pyfixest's own parser, so the syntax, the coefficient
+names and the results all match. The difference is where the work happens: rows
+are streamed off disk in batches and never held as a design matrix.
+
+The data can be a Parquet path or glob, a Polars LazyFrame, or a Polars
+DataFrame. Using file paths or LazyFrames keeps memory usage to a minimum, but
+using DataFrames works well for smaller data and for trying things out.
+
+**Documentation:
+[leetucker.github.io/hdfe-stream](https://leetucker.github.io/hdfe-stream/)**,
+with examples, the KSS bias correction, benchmarks and the function reference.
+
+## Features
+
+- [Any number of fixed effects](https://leetucker.github.io/hdfe-stream/examples/formulas.html#fixed-effects),
+  including [one or none](https://leetucker.github.io/hdfe-stream/examples/fewer_fixed_effects.html),
+  and interactions (`firm_id^year`)
+- [pyfixest formula syntax](https://leetucker.github.io/hdfe-stream/examples/formulas.html#covariates):
+  transformations, categoricals, interactions, event studies
+- [Several models in one call](https://leetucker.github.io/hdfe-stream/examples/formulas.html#several-models-at-once),
+  sharing passes over the data
+- [Standard errors](https://leetucker.github.io/hdfe-stream/examples/formulas.html#standard-errors):
+  iid, robust, multi-way clustered, and one-way CRV3
+- [Weighted least squares and 2SLS](https://leetucker.github.io/hdfe-stream/examples/weights_and_iv.html)
+- [Poisson, logit and probit](https://leetucker.github.io/hdfe-stream/examples/glm.html)
+- [Varying slopes in fixed effects](https://leetucker.github.io/hdfe-stream/examples/varying_slopes.html)
+  (`worker_id[t]`), on the streamed dimension only
+- [Wide designs](https://leetucker.github.io/hdfe-stream/examples/wide_designs.html),
+  with hundreds of covariates
+- [Connected components](https://leetucker.github.io/hdfe-stream/examples/connected_sets.html),
+  the largest connected set, and `min_degree` pruning
+- [AKM variance decompositions](https://leetucker.github.io/hdfe-stream/examples/akm_variance.html),
+  in a streaming pass
+- [KSS leave-out variance components](https://leetucker.github.io/hdfe-stream/kss/index.html),
+  with standard errors and weak-identification diagnostics
+- [The sample, row by row](https://leetucker.github.io/hdfe-stream/examples/formulas.html#missing-values-and-the-sample):
+  which rows were dropped and why, and residuals joined back to the source
+- [Regression tables and plots](https://leetucker.github.io/hdfe-stream/examples/reporting.html)
+  through pyfixest
+- [Memory and disk controls](https://leetucker.github.io/hdfe-stream/examples/out_of_core.html),
+  a choice of solvers, and logging
+- [Simulated panels](https://leetucker.github.io/hdfe-stream/examples/index.html#the-simulated-data)
+  with known effects, for testing
+
+## When to use this, and when not to
+
+**Use pyfixest.** It is the better tool whenever your data fits in memory:
+mature, broader in scope, more inference options, and no scratch directory to
+think about. If a `pandas` DataFrame of your data fits comfortably in RAM, stop
+reading.
+
+**Use this** when it doesn't. The case it was built for is a matched
+employer–employee panel — tens of millions of rows, millions of worker effects,
+an AKM variance decomposition — on a machine or a secure enclave where the data
+is larger than the memory you are allowed. Concretely, reach for it when:
+
+- the data does not fit in memory, or fits so tightly that nothing else does;
+- you have one very high-cardinality dimension (workers, patients, students)
+  whose groups each touch only a few levels of the others;
+- you want **worker-specific slopes** as well as intercepts (`worker_id[t]`),
+  which multiplies the parameter count by the number of slopes;
+- the design is **wide**: hundreds of covariates, such as a categorical
+  expanded into fine indicators or many interactions. An in-memory library
+  needs rows × covariates in memory before it starts; this needs one batch of
+  rows × covariates at a time, with any number of fixed effects, including one
+  or none (see [many covariates](https://leetucker.github.io/hdfe-stream/benchmarks/covariates.html));
+- you need a hard, predictable memory ceiling — a shared cluster, or a job that
+  must not be the one that gets killed.
+
+You are trading memory for disk, and you need scratch space: about five times
+the size of your Parquet input while the fit runs, and only the result files
+once it finishes. See [performance](#performance) for what that buys. Point
+`workdir=` at it, or set the `HDFE_STREAM_WORKDIR` environment variable once
+(for example to a cluster's scratch filesystem) and leave `workdir` out; a
+`workdir` passed to a call takes precedence, and with neither, the system
+temporary directory is used.
+
+## How it works, in one paragraph
+
+One fixed-effect dimension — normally the largest — is **streamed**: it
+is never represented as a vector in memory. Every other dimension is held as a
+vector sized by its number of *levels*, not rows. Rows are read in batches,
+hash-partitioned into buckets by the streamed dimension, sorted one bucket at a
+time, and reduced to a table of cells (one per combination of fixed-effect
+levels). The reduced normal equations for the non-streamed dimensions are then
+solved by conjugate gradient, and the streamed effects are recovered group by
+group in a final pass that writes residuals and fixed effects straight to
+Parquet. So memory scales with firms × years, not with workers × rows, and that
+asymmetry is the whole design. The covariates travel with the rows: what is
+held at once is one bucket or batch of rows × covariates, never all of them,
+which is why a wide design fits where an in-memory one does not. Full detail is in
+[`hdfe_stream/__init__.py`](https://github.com/leetucker/hdfe-stream/blob/main/hdfe_stream/__init__.py).
+
+## Install
+
+```bash
+pip install hdfe-stream[formula]     # formula syntax needs pyfixest
+```
+
+`polars`, `numpy`, `numba`, `pyarrow` and `scipy` are required. The extras:
+
+| extra | what it adds |
+|---|---|
+| `formula` | `feols_stream` formula syntax and pyfixest reporting (pyfixest, formulaic, pandas) |
+| `within` | the `solver="within"` backend |
+| `amg` | algebraic multigrid preconditioning (`precond="amg"`) for hard geometries |
+| `all` | all of the above |
+| `test` | the test suite |
+| `docs` | building the documentation (Sphinx) |
+
+On Python 3.15, pyfixest 0.60 (for the `formula` extra) has no wheels yet. It
+builds from source with a Rust toolchain and
+`PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` set; everything else installs from
+wheels.
+
+Without `formula` you still get `StreamingHDFE`, the lower-level interface that
+takes column names or Polars expressions instead of a formula string.
+
+## Performance
+
+The standard three-way AKM regression,
+`log_earn ~ age_squared + age_cubed | worker_id + firm_id + year`, clustered by
+worker, on a simulated panel of 5 million workers and 42.5 million rows,
+against [pyfixest](https://github.com/py-econometrics/pyfixest) and
+[xhdfe](https://github.com/reisportela/xhdfe-xfe). Every configuration agrees on
+the coefficients to within 3e-10.
+
+| configuration | wall time | peak memory | peak disk |
+|---|---:|---:|---:|
+| pyfixest (MAP, its default) | 1,960 s | 20.7 GB | — |
+| pyfixest (LSMR) | 138 s | 25.3 GB | — |
+| xhdfe | 238 s | 19.0 GB | — |
+| hdfe_stream (`explicit`, its default) | 52 s | 4.1 GB | 4.1 GB |
+| hdfe_stream (`stream_cg`) | 54 s | 4.0 GB | 4.1 GB |
+| **hdfe_stream (`stream_cg`, low memory)** | **60 s** | **3.2 GB** | 4.3 GB |
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/leetucker/hdfe-stream/main/docs/figures/akm_memory.dark.svg">
+  <img alt="AKM regression: peak memory against the number of workers, one line per configuration, log-log" src="https://raw.githubusercontent.com/leetucker/hdfe-stream/main/docs/figures/akm_memory.light.svg">
+</picture>
+
+On small data the in-memory libraries use less memory: an hdfe_stream process
+costs about 0.9 GB before it has done anything. The
+[benchmarks](https://leetucker.github.io/hdfe-stream/benchmarks/index.html) cover every size from 212,000 rows up,
+designs with up to 503 covariates, Poisson and logit, and the KSS
+decomposition, with how each was measured.
+
+## Documentation
+
+- [**Examples**](https://leetucker.github.io/hdfe-stream/examples/index.html): one page per feature, on simulated
+  data that ships with the package: formulas and standard errors, weights and
+  IV, Poisson and logit, wide designs, varying slopes, connected sets, running
+  out of core, and reporting.
+- [**Leave-out variance components**](https://leetucker.github.io/hdfe-stream/kss/index.html): the
+  Kline–Saggio–Sølvsten bias correction for the AKM decomposition, its standard
+  errors and weak-identification diagnostic, and every way it differs from the
+  reference implementations.
+- [**Benchmarks**](https://leetucker.github.io/hdfe-stream/benchmarks/index.html): time, memory and disk against
+  pyfixest and xhdfe.
+- [**Function reference**](https://leetucker.github.io/hdfe-stream/reference/index.html): every option.
+- [**Limitations**](https://leetucker.github.io/hdfe-stream/index.html#limitations): what it does not do.
+- [**Changelog**](https://github.com/leetucker/hdfe-stream/blob/main/CHANGELOG.md).
+
+The scripts behind the examples are in [`examples/`](https://github.com/leetucker/hdfe-stream/tree/main/examples), and the
+benchmarks in [`benchmarks/`](https://github.com/leetucker/hdfe-stream/tree/main/benchmarks).