hdfe-stream 0.2.0__tar.gz → 0.2.2__tar.gz

hdfe_stream-0.2.2/CHANGELOG.md

@@ -0,0 +1,136 @@
+# Changelog
+
+## 0.2.2
+
+- **Polars 2.0 support.** `fit.sample()` returns the source's rows in the source's
+  order again. Polars 2.0 collects with the streaming engine by default, whose
+  joins do not keep row order unless asked, so the rows came back shuffled.
+  Under Polars 2.0, repeated fits of the same model can differ in the last
+  digit or so (around 1e-16 relative): its streaming engine adds floating-point
+  sums in an order that depends on thread scheduling. Under Polars 1.x they
+  are identical bit for bit.
+- **Dropping singletons no longer inflates memory with a wide design.** The
+  dropped levels were anti-joined after the design was evaluated, so the
+  streaming join buffered every design column. The rows are now dropped from
+  the source columns before the design is evaluated.
+- **Finding singletons takes fewer scans.** The search used to stop only
+  after a full round over the fixed effects found nothing. It now stops once
+  every fixed effect has been checked since the last drop, since a dimension's
+  own drop cannot leave a singleton in it: four scans instead of six for a
+  typical three-way model, about 40% less time spent on singletons, and the
+  same memory. The rows dropped are the same.
+- **Sorting the buckets and building the cell table takes less memory under
+  Polars 2.0.** Each bucket is sorted with Polars' in-memory engine and then
+  written, rather than sunk through the streaming engine, whose sort peaked
+  about 1.5x higher. Similarly, Pass 1 grouped each bucket with a streaming
+  `group_by`, which under Polars 2.0 gives every thread its own hash table: with
+  as many cells as rows, as in an AKM panel, the fit's peak more than doubled.
+  The rows are sorted by fe[0] group, so each bucket is now grouped a slice of
+  whole groups at a time and the cells appended in order. This reduces memory
+  use in AKM specifications and leaves wide designs unchanged or a little
+  better.
+- **The explicit solver builds its matrix faster and in less memory.** The
+  reduced matrix S is now written directly in compressed form, one row at a
+  time, instead of being assembled from chunks of coordinate triples that
+  were merged with scipy. Each row's size is counted first, so S is allocated
+  once and nothing larger than S is held while it is built; `max_s_gb` is
+  checked before S is allocated rather than partway through. Estimates are
+  unchanged up to rounding.
+- **Removed `triple_budget` and `dense_max_levels`** from `StreamingHDFE`. They
+  tuned the old way of building S and have no counterpart in the new one.
+- **Per-iteration CG messages are logged at DEBUG.** With `verbose=True`, the
+  "CG iter N: max rel. residual" lines go to the logger at DEBUG, whatever
+  `log_level` is, so they appear only when the logger is enabled for DEBUG.
+  Without a logger they are no longer printed. Other progress messages are
+  unchanged.
+- **Importing hdfe_stream costs a little less memory.** p-values, critical
+  values and the weak-identification interval use `scipy.special`'s
+  distribution functions instead of `scipy.stats`, which is no longer
+  imported. The values are the same.
+- **Empty run directories no longer pile up on NFS.** A file deleted while
+  still open or memory-mapped becomes a hidden .nfs file on NFS, so its run
+  directory could not be removed and was left empty once the process exited.
+  A failed removal is now retried after collecting garbage on every platform
+  (it was Windows only); the leave-out functions release their memory maps as
+  soon as they are done (`reload_intermediates()` maps them back in); and
+  empty run directories more than a minute old are removed by every new fit
+  in the same working directory and by `hdfe_stream.cleanup()`.
+- **Slightly less verbose default logging.** CG iteration messages are now
+  logged at DEBUG level and therefore not printed by default. Lower your log
+  level to obtain this information.
+
+
+## 0.2.1
+
+- **`fe_dof="exact"` no longer depends on the order of the fixed effects.**
+  With three or more fixed effects it counted the connected components of the
+  streamed dimension and whichever came next in the formula, so reordering the
+  formula could change the standard errors. It now counts the components of
+  every pair of dimensions and uses the pair with the most, which is never
+  more than the true number of redundant levels. With two fixed effects
+  nothing changes, except in the clustered case below.
+- **Clustered standard errors with fixed effects nested in the clusters.**
+  Under `fe_dof="exact"`, the components of a pair of fixed effects that are
+  both nested within the clusters were subtracted from K, although those
+  fixed effects' levels are already dropped from it. K could then fall below
+  the number of covariates and the standard errors were too small. Such a pair
+  is no longer used, and with no other pair K is pyfixest's.
+- **`fit.sample()` labels every singleton.** With three or more fixed effects
+  that each had singleton levels, rows dropped only for the third or a later
+  dimension came back with `in_sample=True`, so the counts disagreed with
+  `n_obs` and the summary's singleton count. The fit itself was right. GLM
+  `"separation"` labels had the same fault.
+- **A NaN in a fixed-effect or cluster column is missing**, as a null is,
+  rather than a level of its own. For fixed effects this matches pyfixest;
+  for clusters, where pyfixest raises an error, the row is dropped as it is
+  for a null cluster. `sample()` reports these rows as `"missing"`.
+- **Polars 1.41 is excluded** from the supported versions. Its streaming join
+  panics (`called Option::unwrap() on a None value`) once a frame has about
+  130 columns, which a model with more than about 120 covariates reaches in
+  pass 0. Polars 1.40 and earlier, and 1.42 and later, are unaffected.
+- **A panic inside Polars is reported as a `RuntimeError`** that names the
+  Polars version, says it is a Polars bug, and suggests a fix, instead of a
+  bare `PanicException`.
+
+## 0.2.0
+
+Changes to defaults, so results can differ from 0.1.0 without any change to
+your code.
+
+- **Default `vcov` is now `"iid"`**, with or without fixed effects, matching
+  pyfixest 0.60. Before, a model with fixed effects defaulted to CRV1 clustered
+  on the first fixed effect. Pass `vcov={"CRV1": "worker_id"}` for the old
+  behavior.
+- **Singleton observations are dropped by default** (`fixef_rm="singleton"`),
+  as in pyfixest, repeating until none is left. `fixef_rm="none"` keeps them.
+  `leave_out_kss` fits with `fixef_rm="none"`, since its pruning defines the
+  sample.
+
+New features.
+
+- **`row_id` in `resid()`**: each row carries its position in the source before
+  any row was dropped, so residuals join back onto the source (`row_id=` renames
+  the column).
+- **`fit.sample()`** returns the source, lazily, with `row_id`, `in_sample` and
+  `dropped_because` (`"missing"`, `"singleton"` or `"separation"`).
+  `multi.sample()` returns `{formula: frame}`.
+- **Polars `DataFrame` inputs** are accepted wherever a Parquet path or
+  `LazyFrame` was.
+- **Reserved column prefix `__hdfe_`.** All of the estimator's internal columns
+  now use it, and a source with a column starting with it is refused. This fixes
+  fits failing when a user column was named `w` (with weights), `v0`, `c1`,
+  `gcode` and similar.
+
+## 0.1.0
+
+First release.
+
+- `feols_stream`: out-of-core OLS with several high-dimensional fixed effects,
+  varying slopes, weights, IV, and CRV1/CRV3 and other variance estimators.
+- `fepois_stream` and `feglm_stream`: Poisson, logit and probit by iteratively
+  reweighted least squares over the same passes.
+- AKM variance decomposition and leave-out (KSS) bias correction with
+  standard errors and weak-identification diagnostics.
+- Reporting through pyfixest (`to_pyfixest()`, `etable`).
+- `summary_json()` and `summary_dict()`: the model-level information of
+  `summary()` as JSON or a dict, for saving to disk.

{hdfe_stream-0.2.0 → hdfe_stream-0.2.2}/CITATION.cff

@@ -7,8 +7,8 @@ authors:
   - family-names: Tucker
     given-names: Lee
     email: lee.c.tucker@census.gov
-version: 0.2.0
-date-released: 2026-10-01
+version: 0.2.2
+date-released: 2026-10-08
 license: MIT
 repository-code: "https://github.com/leetucker/hdfe-stream"
 url: "https://github.com/leetucker/hdfe-stream"

{hdfe_stream-0.2.0 → hdfe_stream-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hdfe-stream
-Version: 0.2.0
+Version: 0.2.2
 Summary: Out-of-core regression with several high-dimensional fixed effects: OLS, Poisson, logit and probit, with AKM/KSS variance decompositions
 Author: Lee Tucker
 License-Expression: MIT
@@ -24,7 +24,7 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.md
-Requires-Dist: polars>=1.37
+Requires-Dist: polars!=1.41.*,>=1.37
 Requires-Dist: numpy>=1.24
 Requires-Dist: numba>=0.59
 Requires-Dist: pyarrow>=14.0
@@ -144,19 +144,24 @@ takes column names or Polars expressions instead of a formula string.
 
 ## Features
 
-**Fixed effects.** Any number of dimensions, including one (a within
-regression, `y ~ x | worker_id`) and none (OLS with an intercept, `y ~ x`).
-Interactions with `^`
-(`firm_id^year`). Varying slopes on the streamed dimension in fixest syntax
-(`worker_id[t]`, `worker_id[t, t2]`). Connected components of the
-worker–firm graph are computed and reported, and the degrees-of-freedom
-correction counts the fixed-effect parameters that are actually identified
-(`fe_dof="exact"`) or follows pyfixest's convention (`fe_dof="pyfixest"`).
-Singleton observations, those alone in their level of some fixed effect, are
-dropped as in pyfixest, repeating until none is left (`fixef_rm="singleton"`,
-the default; `fixef_rm="none"` keeps them). Finding them costs a few extra
-passes over the fixed-effect columns and memory for the singleton levels, not
-for the rows.
+**Fixed effects.** Any number of dimensions, including one (a within regression,
+`y ~ x | worker_id`) and none (OLS with an intercept, `y ~ x`). Interactions
+with `^` (`firm_id^year`). Varying slopes on the streamed dimension in fixest
+syntax (`worker_id[t]`, `worker_id[t, t2]`). Connected components of the
+worker–firm graph are computed and reported. The degrees-of-freedom correction
+either removes the redundant fixed-effect levels it can identify
+(`fe_dof="exact"`, the default) or follows pyfixest's convention
+(`fe_dof="pyfixest"`). `"exact"` counts the connected components of each pair of
+fixed-effect dimensions and uses the pair with the most, plus one redundant
+level for each further dimension. With two fixed effects this is the exact
+number of identified parameters. With more it is an upper bound, so standard
+errors are conservative. For clustered standard errors, fixed effects nested
+within the clusters are dropped from the count as in pyfixest, and a pair of two
+nested fixed effects is not used. Singleton observations, those alone in their
+level of some fixed effect, are dropped as in pyfixest, repeating until none is
+left (`fixef_rm="singleton"`, the default; `fixef_rm="none"` keeps them).
+Finding them costs a few extra passes over the fixed-effect columns and memory
+for the singleton levels, not for the rows.
 
 **Covariates.** Transformations (`I(age**2)`, `log(x)`), categoricals (`C(x)`,
 string columns), interactions (`:`, `*`), event-study terms
@@ -237,7 +242,7 @@ demeaners, and with [xhdfe](https://github.com/reisportela/xhdfe-xfe), on its
 CPU backend. Sizes run from 25,000 to 5,000,000 workers, with firms = workers /
 15 at every size and about 8.5 rows per worker, so the largest panel has 42.5
 million rows. Every configuration agrees with every other on the coefficients
-to within 4e-10, so these are measurements of equally good answers. Reproduce
+to within 3e-10, so these are measurements of equally good answers. Reproduce
 with:
 
 ```bash
@@ -273,46 +278,48 @@ At 5 million workers:
 
 | configuration | wall time | peak memory | peak disk |
 |---|---:|---:|---:|
-| pyfixest (MAP, its default) | 888 s | 20.7 GB | — |
-| pyfixest (LSMR) | 150 s | 25.8 GB | — |
-| xhdfe | 126 s | 19.1 GB | — |
-| hdfe_stream (`stream_cg`) | 40 s | 5.4 GB | 3.9 GB |
-| **hdfe_stream (`stream_cg`, low memory)** | **48 s** | **3.7 GB** | 4.0 GB |
+| pyfixest (MAP, its default) | 1,960 s | 20.7 GB | — |
+| pyfixest (LSMR) | 138 s | 25.3 GB | — |
+| xhdfe | 238 s | 19.0 GB | — |
+| hdfe_stream (`explicit`, its default) | 52 s | 4.1 GB | 4.1 GB |
+| hdfe_stream (`stream_cg`) | 54 s | 4.0 GB | 4.1 GB |
+| **hdfe_stream (`stream_cg`, low memory)** | **60 s** | **3.2 GB** | 4.3 GB |
 
 ### Reading the figures
 
 **Memory is the point.** At 5 million workers the in-memory libraries peak at
-19–26 GB: pyfixest's LSMR run needed almost all of this machine's 26 GB.
-hdfe_stream peaks at 5.4 GB with its defaults and 3.7 GB with the batch and
+19–25 GB: pyfixest's LSMR run needed almost all of this machine's 26 GB.
+hdfe_stream peaks at 4.1 GB with its defaults and 3.2 GB with the batch and
 bucket sizes turned down ("low memory": `rows_per_bucket=250_000`,
-`batch_rows=200_000`), for about 20% more time and the same answer. What it
-spends instead is disk: about 90 bytes per row, 4 GB at 42.5 million rows, freed
+`batch_rows=200_000`), for about 16% more time and the same answer. What it
+spends instead is disk: about 97 bytes per row, 4.1 GB at 42.5 million rows, freed
 when the fit finishes. The in-memory libraries' peak grows in proportion to the
-data. hdfe_stream's low-memory setting grows far more slowly: 0.8 GB at 25,000
-workers, 1.3 GB at a million, 3.7 GB at five million.
+data. hdfe_stream's low-memory setting grows far more slowly: 0.9 GB at 25,000
+workers, 1.4 GB at a million, 3.2 GB at five million.
 
-**It is not slower for it.** From about 400,000 workers up, hdfe_stream is the
-fastest configuration measured. At 5 million workers it takes 40 s against
-xhdfe's 126 s and pyfixest's 150 s (LSMR) or 888 s (MAP). With AKM panel data,
+**It is not slower for it.** From about a million workers up, hdfe_stream is
+the fastest configuration measured, and at 400,000 it is within half a second
+of xhdfe (4.1 s against 3.7 s). At 5 million workers it takes 52 s against
+pyfixest's 138 s (LSMR) or 1,960 s (MAP) and xhdfe's 238 s. With AKM panel data,
 reducing rows to worker-firm cells before solving more than pays for the disk
 traffic. The simulated panel has about 8.5 rows per worker and one cell per
 row, which suits this approach; a specification where the cell table is no
 smaller than the data, and the fixed effects are less local, may do worse.
 
 **Below about 400,000 workers, the in-memory libraries use less memory.**
-Running any hdfe_stream fit costs about 0.8 GB, most of it importing polars,
+Running any hdfe_stream fit costs about 0.9 GB, most of it importing polars,
 numba and scipy and starting Polars' streaming engine and numba's thread pools.
-On small data that floor dominates: at 25,000 workers xhdfe peaks at 0.27 GB and
+On small data that floor dominates: at 25,000 workers xhdfe peaks at 0.29 GB and
 pyfixest at about 0.5 GB. This is the concrete version of "use pyfixest (or
 xhdfe) if your data fits".
 
 **The default pyfixest demeaner is not the one to compare against.**
-`MapDemeaner` (alternating projections) is about 6x slower than `LsmrDemeaner`
-from 400,000 workers up, since many worker effects are the case alternating
+`MapDemeaner` (alternating projections) is 6 to 14 times slower than
+`LsmrDemeaner` from 400,000 workers up, since many worker effects are the case alternating
 projections struggles with. If you are comparing, compare against LSMR.
 
-**hdfe_stream's `within` solver** uses more memory at scale: 14.7 GB at 5
-million workers, where `stream_cg` and `explicit` stay near 5.4 GB. Prefer
+**hdfe_stream's `within` solver** uses more memory at scale: 14.9 GB at 5
+million workers, where `stream_cg` and `explicit` stay near 4 GB. Prefer
 those when memory is the constraint.
 
 ### Many covariates
@@ -349,17 +356,17 @@ Wall time and peak memory ("—": ran out of this machine's 26 GB):
 
 | covariates | pyfixest (MAP) | pyfixest (LSMR) | xhdfe | hdfe_stream | hdfe_stream, sized |
 |---:|---:|---:|---:|---:|---:|
-| 8 | 219 s, 5.7 GB | 17 s, 7.3 GB | 18 s, 9.0 GB | 7 s, 2.8 GB | 7 s, 2.9 GB |
-| 41 | 522 s, 13.1 GB | 56 s, 17.3 GB | 55 s, 18.1 GB | 21 s, 7.0 GB | 20 s, 3.7 GB |
-| 83 | 1,026 s, 24.8 GB | 189 s, 25.4 GB | — | 71 s, 12.5 GB | 56 s, 4.5 GB |
-| 167 | — | — | — | 114 s, 23.6 GB | 112 s, 5.1 GB |
-| 503 | — | — | — | — | **351 s, 8.6 GB** |
+| 8 | 132 s, 5.7 GB | 16 s, 7.3 GB | 15 s, 9.1 GB | 7 s, 3.2 GB | 7 s, 3.0 GB |
+| 41 | 520 s, 13.8 GB | 62 s, 18.2 GB | 60 s, 18.2 GB | 22 s, 6.9 GB | 20 s, 3.7 GB |
+| 83 | 976 s, 25.2 GB | 156 s, 25.8 GB | 125 s, 26.0 GB | 58 s, 12.4 GB | 44 s, 4.9 GB |
+| 167 | — | — | — | 127 s, 23.0 GB | 81 s, 5.1 GB |
+| 503 | — | — | — | — | **329 s, 8.6 GB** |
 
 **The in-memory libraries run out first.** Their peak grows with rows ×
-covariates. At 8.5 million rows pyfixest was within 1 GB of this machine's 26 GB
-at 83 indicators and out of memory at 167; xhdfe was already out of memory at
-83. hdfe_stream sized to the
-design goes from 2.9 GB at 8 indicators to 5.1 GB at 167 and 8.6 GB at 503. Of
+covariates. At 8.5 million rows all three were within about 1 GB of this
+machine's 26 GB at 83 indicators, and out of memory at 167. hdfe_stream sized
+to the design goes from 3.0 GB at 8 indicators to 5.1 GB at 167 and 8.6 GB at
+503. Of
 that last figure, about 4.9 GB is the memory-mapped cell table: file pages the
 kernel can drop under pressure. The memory the fit itself allocated peaked at
 4.6 GB (a separate profile, reading the process's anonymous and file-backed
@@ -367,7 +374,7 @@ pages apart).
 
 **Size the batches to the design.** At its defaults hdfe_stream holds a whole
 bucket of up to 20 million rows, with every covariate, while it sorts; that is
-fine for a handful of covariates and not for hundreds (23.6 GB at 167, out of
+fine for a handful of covariates and not for hundreds (23.0 GB at 167, out of
 memory at 503). Every step that holds rows holds all the covariates, so scale
 `rows_per_bucket`, `batch_rows` and `row_group_size` down with the width. The
 benchmark aims at about 1 GB of design per bucket and 250 MB per batch and row
@@ -382,16 +389,16 @@ fit = feols_stream(fml, data, rows_per_bucket=min(20_000_000, int(1e9 // per_row
 
 **Time grows with the square of the width.** The cross-products cost k × k per
 row; they are formed chunk by chunk as BLAS matrix products, and 503 indicators
-take 6 minutes where 167 take 2. hdfe_stream is the fastest configuration at
+take 5½ minutes where 167 take under 1½. hdfe_stream is the fastest configuration at
 every width measured.
 
 **Timings on this machine are noisy where memory is tight.** Each point is one
 run. Repeated runs of the default configuration at 83 indicators, which holds
-12.5 GB, took between 53 and 83 s; the sized configuration varied by about 15%.
+12.4 GB, took between 53 and 83 s; the sized configuration varied by about 15%.
 The differences between libraries above are far larger than that.
 
 **Disk grows with the width too**, since the working copy of the rows carries
-every covariate: 0.3 GB at 8 indicators, 6 GB at 503.
+every covariate: 0.3 GB at 8 indicators, 5.5 GB at 503.
 
 **How the design is built matters.** A formula term such as `i(age_bin)` is
 carried through the partitioning as the one column it is computed from, and the
@@ -416,12 +423,13 @@ python benchmarks/akm_benchmark.py --memory-sweep 1000000
 
 | `rows_per_bucket` | `batch_rows` | wall time | peak memory | peak disk |
 |---:|---:|---:|---:|---:|
-| 20,000,000 (default) | 2,000,000 | 7.1 s | 3,861 MB | 779 MB |
-| 1,000,000 | 500,000 | 7.4 s | 1,560 MB | 781 MB |
-| 250,000 | 200,000 | 7.8 s | 1,343 MB | 795 MB |
-| 100,000 | 100,000 | 9.3 s | 1,280 MB | 789 MB |
+| 20,000,000 (default) | 2,000,000 | 11.2 s | 2,341 MB | 818 MB |
+| 1,000,000 | 500,000 | 11.0 s | 1,849 MB | 821 MB |
+| 250,000 | 200,000 | 11.6 s | 1,384 MB | 863 MB |
+| 100,000 | 100,000 | 11.6 s | 1,555 MB | 876 MB |
 
-Most of the saving comes from the first step down. The defaults are tuned for a
+The saving comes from the first two steps down; below that, the peak rises
+again. The defaults are tuned for a
 machine with room to spare; if memory is the binding constraint, set
 `rows_per_bucket` to something near what you can afford and leave the rest
 alone (with many covariates, scale `batch_rows` and `row_group_size` down too;
@@ -434,8 +442,10 @@ Leave-out estimation has a memory knob of its own, `scratch_mb`; see
 [docs/kss.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md#performance).
 
 Measured on an Intel Core Ultra 7 258V, 8 cores, 26 GB RAM, Linux (WSL2);
-Python 3.13.5, polars 1.44.2, numpy 2.5.3, numba 0.67.0, scipy 1.18.1,
-pyfixest 0.60.0, xhdfe 2.28.0 (CPU backend). The exact versions are in
+Python 3.14.8, polars 2.0.0, numpy 2.5.3, numba 0.68.0, scipy 1.18.1,
+pyfixest 0.60.0, xhdfe 2.28.1 (CPU backend). In the regression benchmarks
+hdfe_stream was measured on 2026-10-07 and pyfixest and xhdfe on 2026-10-06;
+the GLM and KSS benchmarks were measured on 2026-10-07 and 2026-10-08. The exact versions are in
 [benchmarks/results/machine.json](https://github.com/leetucker/hdfe-stream/blob/main/benchmarks/results/machine.json). Numba
 kernels are compiled on first use and cached to disk; the benchmark runs a
 warm-up fit so compilation is not charged to any configuration.
@@ -543,7 +553,7 @@ errors, and KSS's weak-identification diagnostic and q = 1 interval.
 **[docs/kss.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md)** covers the options, the standard errors and their
 measured coverage, weak identification, validation, reproducibility, and
 [performance against xhdfe](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md#performance): at 5 million workers,
-5.3 GB of memory against xhdfe's 20 GB, with standard errors in 46 minutes
+4.8 GB of memory against xhdfe's 20 GB, with standard errors in 49 minutes
 where xhdfe's did not finish in three hours.
 **[docs/kss_methodological_differences.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss_methodological_differences.md)**
 lists every way this implementation differs from LeaveOutTwoWay,

{hdfe_stream-0.2.0 → hdfe_stream-0.2.2}/README.md

@@ -95,19 +95,24 @@ takes column names or Polars expressions instead of a formula string.
 
 ## Features
 
-**Fixed effects.** Any number of dimensions, including one (a within
-regression, `y ~ x | worker_id`) and none (OLS with an intercept, `y ~ x`).
-Interactions with `^`
-(`firm_id^year`). Varying slopes on the streamed dimension in fixest syntax
-(`worker_id[t]`, `worker_id[t, t2]`). Connected components of the
-worker–firm graph are computed and reported, and the degrees-of-freedom
-correction counts the fixed-effect parameters that are actually identified
-(`fe_dof="exact"`) or follows pyfixest's convention (`fe_dof="pyfixest"`).
-Singleton observations, those alone in their level of some fixed effect, are
-dropped as in pyfixest, repeating until none is left (`fixef_rm="singleton"`,
-the default; `fixef_rm="none"` keeps them). Finding them costs a few extra
-passes over the fixed-effect columns and memory for the singleton levels, not
-for the rows.
+**Fixed effects.** Any number of dimensions, including one (a within regression,
+`y ~ x | worker_id`) and none (OLS with an intercept, `y ~ x`). Interactions
+with `^` (`firm_id^year`). Varying slopes on the streamed dimension in fixest
+syntax (`worker_id[t]`, `worker_id[t, t2]`). Connected components of the
+worker–firm graph are computed and reported. The degrees-of-freedom correction
+either removes the redundant fixed-effect levels it can identify
+(`fe_dof="exact"`, the default) or follows pyfixest's convention
+(`fe_dof="pyfixest"`). `"exact"` counts the connected components of each pair of
+fixed-effect dimensions and uses the pair with the most, plus one redundant
+level for each further dimension. With two fixed effects this is the exact
+number of identified parameters. With more it is an upper bound, so standard
+errors are conservative. For clustered standard errors, fixed effects nested
+within the clusters are dropped from the count as in pyfixest, and a pair of two
+nested fixed effects is not used. Singleton observations, those alone in their
+level of some fixed effect, are dropped as in pyfixest, repeating until none is
+left (`fixef_rm="singleton"`, the default; `fixef_rm="none"` keeps them).
+Finding them costs a few extra passes over the fixed-effect columns and memory
+for the singleton levels, not for the rows.
 
 **Covariates.** Transformations (`I(age**2)`, `log(x)`), categoricals (`C(x)`,
 string columns), interactions (`:`, `*`), event-study terms
@@ -188,7 +193,7 @@ demeaners, and with [xhdfe](https://github.com/reisportela/xhdfe-xfe), on its
 CPU backend. Sizes run from 25,000 to 5,000,000 workers, with firms = workers /
 15 at every size and about 8.5 rows per worker, so the largest panel has 42.5
 million rows. Every configuration agrees with every other on the coefficients
-to within 4e-10, so these are measurements of equally good answers. Reproduce
+to within 3e-10, so these are measurements of equally good answers. Reproduce
 with:
 
 ```bash
@@ -224,46 +229,48 @@ At 5 million workers:
 
 | configuration | wall time | peak memory | peak disk |
 |---|---:|---:|---:|
-| pyfixest (MAP, its default) | 888 s | 20.7 GB | — |
-| pyfixest (LSMR) | 150 s | 25.8 GB | — |
-| xhdfe | 126 s | 19.1 GB | — |
-| hdfe_stream (`stream_cg`) | 40 s | 5.4 GB | 3.9 GB |
-| **hdfe_stream (`stream_cg`, low memory)** | **48 s** | **3.7 GB** | 4.0 GB |
+| pyfixest (MAP, its default) | 1,960 s | 20.7 GB | — |
+| pyfixest (LSMR) | 138 s | 25.3 GB | — |
+| xhdfe | 238 s | 19.0 GB | — |
+| hdfe_stream (`explicit`, its default) | 52 s | 4.1 GB | 4.1 GB |
+| hdfe_stream (`stream_cg`) | 54 s | 4.0 GB | 4.1 GB |
+| **hdfe_stream (`stream_cg`, low memory)** | **60 s** | **3.2 GB** | 4.3 GB |
 
 ### Reading the figures
 
 **Memory is the point.** At 5 million workers the in-memory libraries peak at
-19–26 GB: pyfixest's LSMR run needed almost all of this machine's 26 GB.
-hdfe_stream peaks at 5.4 GB with its defaults and 3.7 GB with the batch and
+19–25 GB: pyfixest's LSMR run needed almost all of this machine's 26 GB.
+hdfe_stream peaks at 4.1 GB with its defaults and 3.2 GB with the batch and
 bucket sizes turned down ("low memory": `rows_per_bucket=250_000`,
-`batch_rows=200_000`), for about 20% more time and the same answer. What it
-spends instead is disk: about 90 bytes per row, 4 GB at 42.5 million rows, freed
+`batch_rows=200_000`), for about 16% more time and the same answer. What it
+spends instead is disk: about 97 bytes per row, 4.1 GB at 42.5 million rows, freed
 when the fit finishes. The in-memory libraries' peak grows in proportion to the
-data. hdfe_stream's low-memory setting grows far more slowly: 0.8 GB at 25,000
-workers, 1.3 GB at a million, 3.7 GB at five million.
+data. hdfe_stream's low-memory setting grows far more slowly: 0.9 GB at 25,000
+workers, 1.4 GB at a million, 3.2 GB at five million.
 
-**It is not slower for it.** From about 400,000 workers up, hdfe_stream is the
-fastest configuration measured. At 5 million workers it takes 40 s against
-xhdfe's 126 s and pyfixest's 150 s (LSMR) or 888 s (MAP). With AKM panel data,
+**It is not slower for it.** From about a million workers up, hdfe_stream is
+the fastest configuration measured, and at 400,000 it is within half a second
+of xhdfe (4.1 s against 3.7 s). At 5 million workers it takes 52 s against
+pyfixest's 138 s (LSMR) or 1,960 s (MAP) and xhdfe's 238 s. With AKM panel data,
 reducing rows to worker-firm cells before solving more than pays for the disk
 traffic. The simulated panel has about 8.5 rows per worker and one cell per
 row, which suits this approach; a specification where the cell table is no
 smaller than the data, and the fixed effects are less local, may do worse.
 
 **Below about 400,000 workers, the in-memory libraries use less memory.**
-Running any hdfe_stream fit costs about 0.8 GB, most of it importing polars,
+Running any hdfe_stream fit costs about 0.9 GB, most of it importing polars,
 numba and scipy and starting Polars' streaming engine and numba's thread pools.
-On small data that floor dominates: at 25,000 workers xhdfe peaks at 0.27 GB and
+On small data that floor dominates: at 25,000 workers xhdfe peaks at 0.29 GB and
 pyfixest at about 0.5 GB. This is the concrete version of "use pyfixest (or
 xhdfe) if your data fits".
 
 **The default pyfixest demeaner is not the one to compare against.**
-`MapDemeaner` (alternating projections) is about 6x slower than `LsmrDemeaner`
-from 400,000 workers up, since many worker effects are the case alternating
+`MapDemeaner` (alternating projections) is 6 to 14 times slower than
+`LsmrDemeaner` from 400,000 workers up, since many worker effects are the case alternating
 projections struggles with. If you are comparing, compare against LSMR.
 
-**hdfe_stream's `within` solver** uses more memory at scale: 14.7 GB at 5
-million workers, where `stream_cg` and `explicit` stay near 5.4 GB. Prefer
+**hdfe_stream's `within` solver** uses more memory at scale: 14.9 GB at 5
+million workers, where `stream_cg` and `explicit` stay near 4 GB. Prefer
 those when memory is the constraint.
 
 ### Many covariates
@@ -300,17 +307,17 @@ Wall time and peak memory ("—": ran out of this machine's 26 GB):
 
 | covariates | pyfixest (MAP) | pyfixest (LSMR) | xhdfe | hdfe_stream | hdfe_stream, sized |
 |---:|---:|---:|---:|---:|---:|
-| 8 | 219 s, 5.7 GB | 17 s, 7.3 GB | 18 s, 9.0 GB | 7 s, 2.8 GB | 7 s, 2.9 GB |
-| 41 | 522 s, 13.1 GB | 56 s, 17.3 GB | 55 s, 18.1 GB | 21 s, 7.0 GB | 20 s, 3.7 GB |
-| 83 | 1,026 s, 24.8 GB | 189 s, 25.4 GB | — | 71 s, 12.5 GB | 56 s, 4.5 GB |
-| 167 | — | — | — | 114 s, 23.6 GB | 112 s, 5.1 GB |
-| 503 | — | — | — | — | **351 s, 8.6 GB** |
+| 8 | 132 s, 5.7 GB | 16 s, 7.3 GB | 15 s, 9.1 GB | 7 s, 3.2 GB | 7 s, 3.0 GB |
+| 41 | 520 s, 13.8 GB | 62 s, 18.2 GB | 60 s, 18.2 GB | 22 s, 6.9 GB | 20 s, 3.7 GB |
+| 83 | 976 s, 25.2 GB | 156 s, 25.8 GB | 125 s, 26.0 GB | 58 s, 12.4 GB | 44 s, 4.9 GB |
+| 167 | — | — | — | 127 s, 23.0 GB | 81 s, 5.1 GB |
+| 503 | — | — | — | — | **329 s, 8.6 GB** |
 
 **The in-memory libraries run out first.** Their peak grows with rows ×
-covariates. At 8.5 million rows pyfixest was within 1 GB of this machine's 26 GB
-at 83 indicators and out of memory at 167; xhdfe was already out of memory at
-83. hdfe_stream sized to the
-design goes from 2.9 GB at 8 indicators to 5.1 GB at 167 and 8.6 GB at 503. Of
+covariates. At 8.5 million rows all three were within about 1 GB of this
+machine's 26 GB at 83 indicators, and out of memory at 167. hdfe_stream sized
+to the design goes from 3.0 GB at 8 indicators to 5.1 GB at 167 and 8.6 GB at
+503. Of
 that last figure, about 4.9 GB is the memory-mapped cell table: file pages the
 kernel can drop under pressure. The memory the fit itself allocated peaked at
 4.6 GB (a separate profile, reading the process's anonymous and file-backed
@@ -318,7 +325,7 @@ pages apart).
 
 **Size the batches to the design.** At its defaults hdfe_stream holds a whole
 bucket of up to 20 million rows, with every covariate, while it sorts; that is
-fine for a handful of covariates and not for hundreds (23.6 GB at 167, out of
+fine for a handful of covariates and not for hundreds (23.0 GB at 167, out of
 memory at 503). Every step that holds rows holds all the covariates, so scale
 `rows_per_bucket`, `batch_rows` and `row_group_size` down with the width. The
 benchmark aims at about 1 GB of design per bucket and 250 MB per batch and row
@@ -333,16 +340,16 @@ fit = feols_stream(fml, data, rows_per_bucket=min(20_000_000, int(1e9 // per_row
 
 **Time grows with the square of the width.** The cross-products cost k × k per
 row; they are formed chunk by chunk as BLAS matrix products, and 503 indicators
-take 6 minutes where 167 take 2. hdfe_stream is the fastest configuration at
+take 5½ minutes where 167 take under 1½. hdfe_stream is the fastest configuration at
 every width measured.
 
 **Timings on this machine are noisy where memory is tight.** Each point is one
 run. Repeated runs of the default configuration at 83 indicators, which holds
-12.5 GB, took between 53 and 83 s; the sized configuration varied by about 15%.
+12.4 GB, took between 53 and 83 s; the sized configuration varied by about 15%.
 The differences between libraries above are far larger than that.
 
 **Disk grows with the width too**, since the working copy of the rows carries
-every covariate: 0.3 GB at 8 indicators, 6 GB at 503.
+every covariate: 0.3 GB at 8 indicators, 5.5 GB at 503.
 
 **How the design is built matters.** A formula term such as `i(age_bin)` is
 carried through the partitioning as the one column it is computed from, and the
@@ -367,12 +374,13 @@ python benchmarks/akm_benchmark.py --memory-sweep 1000000
 
 | `rows_per_bucket` | `batch_rows` | wall time | peak memory | peak disk |
 |---:|---:|---:|---:|---:|
-| 20,000,000 (default) | 2,000,000 | 7.1 s | 3,861 MB | 779 MB |
-| 1,000,000 | 500,000 | 7.4 s | 1,560 MB | 781 MB |
-| 250,000 | 200,000 | 7.8 s | 1,343 MB | 795 MB |
-| 100,000 | 100,000 | 9.3 s | 1,280 MB | 789 MB |
+| 20,000,000 (default) | 2,000,000 | 11.2 s | 2,341 MB | 818 MB |
+| 1,000,000 | 500,000 | 11.0 s | 1,849 MB | 821 MB |
+| 250,000 | 200,000 | 11.6 s | 1,384 MB | 863 MB |
+| 100,000 | 100,000 | 11.6 s | 1,555 MB | 876 MB |
 
-Most of the saving comes from the first step down. The defaults are tuned for a
+The saving comes from the first two steps down; below that, the peak rises
+again. The defaults are tuned for a
 machine with room to spare; if memory is the binding constraint, set
 `rows_per_bucket` to something near what you can afford and leave the rest
 alone (with many covariates, scale `batch_rows` and `row_group_size` down too;
@@ -385,8 +393,10 @@ Leave-out estimation has a memory knob of its own, `scratch_mb`; see
 [docs/kss.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md#performance).
 
 Measured on an Intel Core Ultra 7 258V, 8 cores, 26 GB RAM, Linux (WSL2);
-Python 3.13.5, polars 1.44.2, numpy 2.5.3, numba 0.67.0, scipy 1.18.1,
-pyfixest 0.60.0, xhdfe 2.28.0 (CPU backend). The exact versions are in
+Python 3.14.8, polars 2.0.0, numpy 2.5.3, numba 0.68.0, scipy 1.18.1,
+pyfixest 0.60.0, xhdfe 2.28.1 (CPU backend). In the regression benchmarks
+hdfe_stream was measured on 2026-10-07 and pyfixest and xhdfe on 2026-10-06;
+the GLM and KSS benchmarks were measured on 2026-10-07 and 2026-10-08. The exact versions are in
 [benchmarks/results/machine.json](https://github.com/leetucker/hdfe-stream/blob/main/benchmarks/results/machine.json). Numba
 kernels are compiled on first use and cached to disk; the benchmark runs a
 warm-up fit so compilation is not charged to any configuration.
@@ -494,7 +504,7 @@ errors, and KSS's weak-identification diagnostic and q = 1 interval.
 **[docs/kss.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md)** covers the options, the standard errors and their
 measured coverage, weak identification, validation, reproducibility, and
 [performance against xhdfe](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss.md#performance): at 5 million workers,
-5.3 GB of memory against xhdfe's 20 GB, with standard errors in 46 minutes
+4.8 GB of memory against xhdfe's 20 GB, with standard errors in 49 minutes
 where xhdfe's did not finish in three hours.
 **[docs/kss_methodological_differences.md](https://github.com/leetucker/hdfe-stream/blob/main/docs/kss_methodological_differences.md)**
 lists every way this implementation differs from LeaveOutTwoWay,