dfeols 0.7.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. dfeols-0.7.0/CHANGELOG.md +191 -0
  2. dfeols-0.7.0/CITATION.cff +20 -0
  3. dfeols-0.7.0/CONTRIBUTING.md +69 -0
  4. dfeols-0.7.0/HOWTO.md +270 -0
  5. dfeols-0.7.0/LICENSE +674 -0
  6. dfeols-0.7.0/MANIFEST.in +6 -0
  7. dfeols-0.7.0/PKG-INFO +315 -0
  8. dfeols-0.7.0/README.md +264 -0
  9. dfeols-0.7.0/ci/min-versions.txt +12 -0
  10. dfeols-0.7.0/docs/ARCHITECTURE.md +153 -0
  11. dfeols-0.7.0/examples/profiles/oscer.toml +28 -0
  12. dfeols-0.7.0/examples/profiles/templates.toml +50 -0
  13. dfeols-0.7.0/examples/run_fit.py +51 -0
  14. dfeols-0.7.0/pyproject.toml +111 -0
  15. dfeols-0.7.0/setup.cfg +4 -0
  16. dfeols-0.7.0/src/dfeols/__init__.py +34 -0
  17. dfeols-0.7.0/src/dfeols/bench.py +543 -0
  18. dfeols-0.7.0/src/dfeols/cli.py +314 -0
  19. dfeols-0.7.0/src/dfeols/cluster.py +318 -0
  20. dfeols-0.7.0/src/dfeols/demean.py +181 -0
  21. dfeols-0.7.0/src/dfeols/estimate.py +536 -0
  22. dfeols-0.7.0/src/dfeols/executors.py +141 -0
  23. dfeols-0.7.0/src/dfeols/formula.py +190 -0
  24. dfeols-0.7.0/src/dfeols/linalg.py +41 -0
  25. dfeols-0.7.0/src/dfeols/lock.py +136 -0
  26. dfeols-0.7.0/src/dfeols/prep.py +348 -0
  27. dfeols-0.7.0/src/dfeols/prep_dist.py +849 -0
  28. dfeols-0.7.0/src/dfeols/progress.py +263 -0
  29. dfeols-0.7.0/src/dfeols/results.py +108 -0
  30. dfeols-0.7.0/src/dfeols/site.py +209 -0
  31. dfeols-0.7.0/src/dfeols/synth/__init__.py +209 -0
  32. dfeols-0.7.0/src/dfeols/synth/__main__.py +5 -0
  33. dfeols-0.7.0/src/dfeols/synth/influencer.py +250 -0
  34. dfeols-0.7.0/src/dfeols/synth/panel.py +210 -0
  35. dfeols-0.7.0/src/dfeols/validate.py +59 -0
  36. dfeols-0.7.0/src/dfeols/vcov.py +40 -0
  37. dfeols-0.7.0/src/dfeols/worker.py +426 -0
  38. dfeols-0.7.0/src/dfeols.egg-info/PKG-INFO +315 -0
  39. dfeols-0.7.0/src/dfeols.egg-info/SOURCES.txt +52 -0
  40. dfeols-0.7.0/src/dfeols.egg-info/dependency_links.txt +1 -0
  41. dfeols-0.7.0/src/dfeols.egg-info/entry_points.txt +2 -0
  42. dfeols-0.7.0/src/dfeols.egg-info/requires.txt +26 -0
  43. dfeols-0.7.0/src/dfeols.egg-info/top_level.txt +1 -0
  44. dfeols-0.7.0/tests/conftest.py +114 -0
  45. dfeols-0.7.0/tests/simdata.py +56 -0
  46. dfeols-0.7.0/tests/test_cli.py +20 -0
  47. dfeols-0.7.0/tests/test_demean.py +42 -0
  48. dfeols-0.7.0/tests/test_influencer.py +274 -0
  49. dfeols-0.7.0/tests/test_memory_check.py +28 -0
  50. dfeols-0.7.0/tests/test_panel.py +159 -0
  51. dfeols-0.7.0/tests/test_prep_distributed.py +485 -0
  52. dfeols-0.7.0/tests/test_prep_lock.py +102 -0
  53. dfeols-0.7.0/tests/test_progress.py +61 -0
  54. dfeols-0.7.0/tests/test_vs_pyfixest.py +290 -0
@@ -0,0 +1,191 @@
1
+ # Changelog
2
+
3
+ All notable changes to dfeols are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
5
+ [semantic versioning](https://semver.org/) (before 1.0, minor versions may
6
+ break the API).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.7.0] - 2026-10-22
11
+
12
+ First public release.
13
+
14
+ ### Added
15
+ - `dfeols --version`.
16
+ - CI on GitHub Actions: Linux and macOS, Python 3.10–3.14, a Dask job, a
17
+ minimum-versions job and a nightly job against the newest polars and
18
+ pyfixest.
19
+
20
+ ### Changed
21
+ - Source moved to a `src/` layout; `examples/` and `tests/` are not installed.
22
+ - Licensed under GPL-3.0-or-later.
23
+ - Dependency ranges are now tested at both ends: `polars>=1.37,<3`,
24
+ `pyfixest>=0.50.1,<1`, `numpy>=1.25`; Python 3.10–3.14.
25
+ - Dense FE codes are `int64` (were `uint64`), which fixes `np.bincount`
26
+ failures under NumPy 1.x.
27
+ - pyfixest comparisons (tests, `dfeols.bench` baseline) use the typed
28
+ `demeaner=MapDemeaner(...)` argument on pyfixest ≥ 0.60 and fall back to
29
+ `fixef_tol`/`fixef_maxiter` on older releases
30
+ (`dfeols.formula.pyfixest_tol_kwargs`).
31
+ - pyfixest's compiled demeaner (a private API) is imported defensively; if
32
+ its location or signature changes, dfeols logs a warning and uses the NumPy
33
+ fallback. A test fails if the compiled path is not found, so the nightly CI
34
+ job catches upstream changes.
35
+ - The ETA bootstrap no longer shares one random generator across calls.
36
+ - Code formatted and linted with ruff.
37
+
38
+ ### Fixed
39
+ These include the fixes from the 0.6.2 snapshot (below), merged into the new
40
+ layout, plus:
41
+ - The coordinator merges per-file scan results as they arrive instead of
42
+ holding them until the scan ends. At 1e10 rows it kept 5,000 copies of the
43
+ group × time level set and its nesting-check tables (well over 100 GB) and
44
+ was killed; merged, these stay near the number of distinct keys.
45
+ - `fit` checks the largest shard's memory estimate against the profile's
46
+ worker memory: it warns above it and refuses above 1.5×, instead of
47
+ letting the scheduler kill worker after worker on the same shard.
48
+ - prep refuses to write a shard larger than twice the target plus the
49
+ largest partition group (only a bug can produce one), and warns when one
50
+ partition group is more than 4× the target.
51
+ - A `shard_dir` left by an interrupted prep (no `dfeols_meta.json`) is
52
+ cleared before prepping again, so its stale pieces cannot join the new
53
+ dataset.
54
+ - `fit` reads the levels of `i()` variables with executor tasks and caches
55
+ them in `<shard_dir>/_levels/`; it read the column from every shard on
56
+ the coordinator, 55 minutes before round 1 at 1e10 rows.
57
+ - The prep scan runs over batches of files (about 4 tasks per worker); each
58
+ task reduces its level sets and nesting checks before returning, so the
59
+ coordinator receives one copy per batch instead of one per file. At 1e10
60
+ rows the per-file results made the by-unit scan coordinator-bound
61
+ (50 minutes, against 16 for the by-group scan).
62
+ - Prep stages are no longer given an ETA projected from the scan rate (the
63
+ split stage showed "8866M rows/s, ETA 0:00:01" while its 200M-row tasks ran
64
+ for ~20 minutes); the round-1-to-round-2 prior now applies to fits only.
65
+ - The ETA no longer crashes with `OverflowError` when no workers are
66
+ connected (an infinite ETA is reported as unknown).
67
+ - Prep tasks counted their output by listing the folder they had written
68
+ into. On Lustre, listing a folder that 40 workers are writing into can
69
+ return an entry twice, so a bucket file was counted twice per repeated
70
+ entry: "encoded 10,002,000,000 rows, expected 10,000,000,000" at 1e10, with
71
+ every file on disk correct (10 entries of 200,000 rows; the earlier +2.8M
72
+ and +400k were the same). Tasks now count exactly the files they handed to
73
+ Polars, and the input file list is deduplicated.
74
+ - Every encode and split task checks that it wrote exactly the rows the scan
75
+ counted for its input; on a mismatch it logs which shards or buckets carry
76
+ the difference, moves its files to `<shard_dir>/_rejected/` (the first 5
77
+ attempts) and reruns, up to 3 attempts, and otherwise fails naming the task.
78
+ At 1e10 rows on OSCER, 4 of 1,000 encode attempts wrote 200k–800k extra
79
+ rows (whole multiples of one bucket's share) and were correct when rerun.
80
+ - Dask task keys include a per-run id; every prep used the same keys
81
+ (`dfeols-encode-prep-17`), so a second prep in the same job could be handed
82
+ a result Dask still held for the first.
83
+ - Folder locks also take over immediately when their holder ran on this
84
+ host and is gone, and a release never removes another holder's lock.
85
+
86
+ ### Removed
87
+ - `dfeols.slurm` (deprecated shim). Use `dfeols.executor(profile)` with a
88
+ site profile (`dfeols config init`).
89
+ - `benchmarks/bench.py` stub; use `python -m dfeols.bench`.
90
+
91
+ ## [0.6.2]
92
+
93
+ Fixes from the first 1e10 attempts on OSCER (a snapshot, not published).
94
+
95
+ ### Fixed
96
+ - Range-based shard assignment (more than `range_groups` partition groups,
97
+ new in 0.6.0) used polars' `UInt32` row counts, which wrap at 2**32: at
98
+ 2e9 rows it produced 2 shards instead of 1,000 (at 1e9 rows with 10M units,
99
+ 4 shards). Counts are Int64 and the bins and boundaries are checked to be
100
+ monotone. **Shard sets prepared with range assignment before 0.6.2 must be
101
+ rebuilt**; `dfeols bench` detects them (no `prep_version` in their
102
+ metadata) and re-preps them.
103
+ - A task that ran twice at once (a rerun overlapping its first attempt, or
104
+ another job writing the same folder) could be counted twice ("encoded
105
+ 2,000,400,000 rows, expected 2,000,000,000"). Every task attempt writes
106
+ uniquely named files and reports them; the coordinator keeps one attempt
107
+ per task, ignores duplicate results, sweeps the other files, and gives the
108
+ split and merge steps explicit file lists. Merge inputs are deleted by the
109
+ coordinator once the merge is accepted.
110
+ - Two jobs could write the same folder. Shard folders (during prep) and
111
+ benchmark data sets (during generation and prep) are locked with a sibling
112
+ `<folder>.lock` (O_EXCL, refreshed every 30 s, stale after 5 min); a
113
+ second job fails at once with `dfeols.LockedError` naming the holder.
114
+ - Overwrites rename the old folder before deleting it, so a straggling
115
+ writer cannot make the delete fail ("Directory not empty: '_buckets'").
116
+
117
+ ### Added
118
+ - `prep_version` in `dfeols_meta.json`.
119
+
120
+ ## [0.6.1]
121
+
122
+ ### Added
123
+ - `dfeols bench`: submits a benchmark as a coordinator job with one
124
+ `--profile`; the job's profile is passed through to the benchmark
125
+ (`$DFEOLS_PROFILE`).
126
+
127
+ ### Changed
128
+ - `dfeols submit` and `dfeols bench` print the profile and module lines used.
129
+ - `dfeols config init --force` backs up the existing config first.
130
+ - OSCER example coordinator raised to 16 cores / 64 GB (data generation runs
131
+ on the coordinator).
132
+
133
+ ## [0.6.0]
134
+
135
+ ### Added
136
+ - Hash ids for nested FEs with more than 2e6 levels (collisions checked
137
+ exactly, with a fallback to dense codes) and range-based shard assignment
138
+ for more than 5e6 partition groups, for 1e10-row datasets.
139
+ - Two-level shuffle in distributed prep when there would be more than 5e4
140
+ piece files.
141
+ - Stress presets `units100m_t100` (1e10 rows) and `skewed_groups`.
142
+ - Round-2 ETA prior.
143
+
144
+ ### Fixed
145
+ - Nested-in-cluster detection with signed ids.
146
+ - Worker memory estimate.
147
+
148
+ ## [0.5.1]
149
+
150
+ ### Changed
151
+ - Flat piece files; shards are merged only when fragmented.
152
+ - Adaptive task size in distributed prep.
153
+ - The coordinator runs read-only scan tasks itself while no workers are
154
+ connected.
155
+
156
+ ## [0.5.0]
157
+
158
+ ### Added
159
+ - Distributed prep on workers (scan → reduce → encode → merge/split).
160
+ - Polars spill directory and memory pinning to the job allocation.
161
+ - Collision-free data folders.
162
+
163
+ ## [0.4.0]
164
+
165
+ ### Added
166
+ - Sparse crossing FEs: with one local FE, crossing-FE blocks of X'WX come
167
+ from per-level sums and no dummy columns are built.
168
+ - Billion-row benchmark presets.
169
+ - Lazy worker scale-down (`idle_minutes`).
170
+ - Append-only results layout and `dfeols results`.
171
+
172
+ ## [0.3.0]
173
+
174
+ ### Added
175
+ - General HDFE with a generic unit × time panel DGP.
176
+ - Scheduler-agnostic TOML site profiles (Slurm, PBS, SGE, LSF, local).
177
+ - Exact two-way demeaning (Schur complement per connected component).
178
+
179
+ ## [0.2.0]
180
+
181
+ ### Added
182
+ - Synthetic DGP with known parameters, validity suite, benchmark runner.
183
+
184
+ ## [0.1.0]
185
+
186
+ ### Added
187
+ - Two-round map/reduce fit, Serial and Dask executors, ETA, agreement with
188
+ pyfixest.
189
+
190
+ [Unreleased]: https://github.com/jemather/dfeols/compare/v0.7.0...HEAD
191
+ [0.7.0]: https://github.com/jemather/dfeols/releases/tag/v0.7.0
@@ -0,0 +1,20 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use dfeols in your research, please cite it as below."
3
+ title: "dfeols: sharded, elastic high-dimensional fixed-effects OLS for HPC clusters"
4
+ type: software
5
+ authors:
6
+ - family-names: Matherly
7
+ given-names: Ted
8
+ affiliation: University of Oklahoma
9
+ orcid: "https://orcid.org/0000-0002-8628-1004"
10
+ version: 0.7.0
11
+ date-released: 2026-10-22
12
+ license: GPL-3.0-or-later
13
+ repository-code: "https://github.com/jemather/dfeols"
14
+ keywords:
15
+ - econometrics
16
+ - fixed effects
17
+ - high-dimensional fixed effects
18
+ - panel data
19
+ - high-performance computing
20
+ - Slurm
@@ -0,0 +1,69 @@
1
+ # Contributing to dfeols
2
+
3
+ Thanks for your interest. Bug reports, site profiles for new clusters, and
4
+ pull requests are all welcome.
5
+
6
+ ## Reporting a problem
7
+
8
+ Open an issue with:
9
+
10
+ - `dfeols --version`, `python --version`, and the versions of polars,
11
+ pyfixest and (if used) dask / dask-jobqueue;
12
+ - the scheduler (Slurm, PBS, …) and the output of `dfeols config show`
13
+ with account names removed;
14
+ - for wrong results: the formula, `dfeols info <shard_dir>`, and if possible
15
+ a small synthetic reproduction (`python -m dfeols.synth`);
16
+ - for crashes on a cluster: the coordinator log and one worker log.
17
+
18
+ ## Development setup
19
+
20
+ ```bash
21
+ git clone https://github.com/jemather/dfeols.git
22
+ cd dfeols
23
+ python -m venv .venv && source .venv/bin/activate
24
+ pip install -e ".[dev]"
25
+ pre-commit install # optional: runs ruff on each commit
26
+ ```
27
+
28
+ ## Checks
29
+
30
+ ```bash
31
+ ruff check . && ruff format --check .
32
+ pytest # ~2 minutes on 2 cores, 0.5M-row synthetic panels
33
+ pytest --executor dask:2 # the same fits on a local Dask cluster
34
+ pytest -m slow # Monte Carlo coverage (long)
35
+ mypy # advisory for now
36
+ ```
37
+
38
+ Larger test data: `pytest --rows 5e6 --data-dir /scratch/$USER/dfeols-tests`.
39
+
40
+ ## What a change needs
41
+
42
+ - **Correctness first.** Anything that touches estimation must keep agreement
43
+ with pyfixest (`tests/test_vs_pyfixest.py`) and recovery on the synthetic
44
+ DGPs (`tests/test_panel.py`, `tests/test_influencer.py`). New model
45
+ features need a pyfixest/fixest comparison test.
46
+ - **Invariance.** Results must not depend on the partition key, the number
47
+ of shards, or the executor; add to the invariance tests where relevant.
48
+ - **No site-specific code.** Queues, modules, paths and accounts belong in
49
+ site profiles (`examples/profiles/`), never in the package.
50
+ - Update `CHANGELOG.md` under *Unreleased*.
51
+
52
+ ## New site profiles
53
+
54
+ If you run dfeols on a new cluster, a tested profile in
55
+ `examples/profiles/` (with the site name, scheduler and date tested in a
56
+ comment) is a very useful contribution. PBS, SGE and LSF are currently
57
+ untested on real systems.
58
+
59
+ ## Releasing (maintainers)
60
+
61
+ 1. Update the version in `src/dfeols/__init__.py`, `CITATION.cff` and the
62
+ changelog.
63
+ 2. Tag `vX.Y.Z` and push the tag. The release workflow builds the sdist and
64
+ wheel, publishes to TestPyPI, then to PyPI (trusted publishing).
65
+
66
+ ## License
67
+
68
+ By contributing you agree that your contributions are licensed under the
69
+ project's license, GPL-3.0-or-later.
dfeols-0.7.0/HOWTO.md ADDED
@@ -0,0 +1,270 @@
1
+ # dfeols: how to use it with a real dataset
2
+
3
+ This is the short version. The README covers the design; `python -m dfeols.bench --help`,
4
+ `python -m dfeols.synth --help` and `dfeols --help` cover the tools.
5
+
6
+ ## 1. One-time setup on a machine
7
+
8
+ ```bash
9
+ python -m venv ~/envs/dfeols && source ~/envs/dfeols/bin/activate
10
+ pip install "dfeols[dask]" # or: pip install "dfeols[dask] @ git+https://github.com/jemather/dfeols"
11
+ dfeols config init # writes ~/.config/dfeols/config.toml
12
+ ```
13
+
14
+ Create the venv on a filesystem the compute nodes can see (usually your home directory), with
15
+ the same Python module the profile's `prologue` loads.
16
+
17
+ Edit the profile in that file:
18
+ - `scheduler`, `queue`, `account`
19
+ - `prologue`: the module loads and venv activation every job runs
20
+ - `scratch`: a shared filesystem every node can see
21
+ - `[worker]` cores, memory, walltime and `max_workers`
22
+
23
+ Smaller, shorter worker jobs start sooner on a busy queue. Workers are released only after
24
+ `idle_minutes` (default 15) with nothing to do, so a pause between fits doesn't hand back
25
+ workers that waited in the queue. `examples/profiles/oscer.toml`
26
+ is a tested Slurm profile; `examples/profiles/templates.toml` covers PBS, SGE, LSF and local.
27
+
28
+ Set `default = "<name>"` at the top of the file so commands use your profile without
29
+ `--profile`. `dfeols config init --force` replaces the file with the template (it saves a backup
30
+ first). Every `submit`/`bench` prints the profile and module lines it used.
31
+
32
+ Then check:
33
+
34
+ ```bash
35
+ dfeols check # one worker starts through the scheduler and runs a task
36
+ ```
37
+
38
+ If it fails:
39
+ - jobs pending: queue limits
40
+ - jobs exit at once: check the worker logs in `<scratch>/logs`
41
+ - "Timed out trying to connect": set `interface` in the profile
42
+
43
+ ## 2. Prep the data once
44
+
45
+ Prep encodes the FE and cluster variables, checks nesting and writes Parquet shards.
46
+ Every model on that dataset reuses the shards.
47
+
48
+ ```python
49
+ import dfeols
50
+ prof = dfeols.profile()
51
+ ex = dfeols.executor(prof)
52
+ ds = dfeols.prep("/data/panel/*.parquet", # Parquet files (or a frame / CSV without executor)
53
+ shard_dir=prof.scratch_dir("shards", "panel"),
54
+ executor=ex, # prep on the workers
55
+ partition="unit", # groups never split across shards
56
+ fixed_effects=["unit", "time", "unit^quarter"], # every FE any model uses
57
+ clusters=["unit", "state"], # every cluster variable any model uses
58
+ target_rows=2_000_000) # rows per shard
59
+ ```
60
+
61
+ **Choose the partition key** so that the most FEs are nested in it.
62
+ - Unit × time: partition by unit. The unit FE is then handled inside each shard, and time
63
+ crosses shards.
64
+ - `unit + group^time` with units nested in groups: partition by group, so both are local.
65
+ - Run `dfeols info <shard_dir>` to see how each FE was classified.
66
+
67
+ Crossing FEs need up to `max_dummy_levels` (2000) levels; high-cardinality crossing FEs are
68
+ not supported yet.
69
+
70
+ **Shard size:** worker memory peaks at roughly 3.5 × rows × (outcomes + regressors) × 8 bytes.
71
+ On OSCER, 2M-row shards peaked at about 15 GB with 74 regressors. Aim for at least 5 shards per
72
+ worker so the load balances and the ETA has data.
73
+
74
+ **Run prep on the workers** with `executor=ex`. This needs Parquet input; split a very large
75
+ single file into many, which also parallelizes. Prep then runs as map/reduce rounds:
76
+
77
+ 1. scan each file for its FE levels and group sizes; while no workers have connected yet, the
78
+ coordinator scans files itself
79
+ 2. build the code tables on the coordinator
80
+ 3. encode and bucket rows on the workers, in about 4 tasks per worker so the last wave is
81
+ short; each task writes one flat piece file per shard it touches
82
+ 4. merge only shards that ended up in more than 8 pieces (when partition groups are spread
83
+ across many input files); other shards keep their pieces as is
84
+
85
+ The coordinator holds only the code tables: 0.3 GB for 30M rows in testing, and roughly
86
+ 8 bytes per distinct FE level (10M units ≈ 80 MB). Without `executor`, prep streams on the
87
+ coordinator alone. That's fine up to a few hundred million rows if the coordinator has the
88
+ memory.
89
+
90
+ At very large scale prep switches strategy on its own (each threshold can be set as a `prep()`
91
+ keyword):
92
+
93
+ | Situation | What prep does | Threshold |
94
+ |---|---|---|
95
+ | a nested FE or cluster with millions of levels (e.g. 100M units) | 64-bit hash ids instead of a code table join; collisions are checked exactly, with a fallback to codes | `hash_levels=2e6` |
96
+ | millions of partition groups | shards are contiguous ranges of the partition key (or its hash for multi-column keys), so tasks need only the boundaries | `range_groups=5e6` |
97
+ | partition groups scattered across input files, so one piece per (task, shard) would mean too many files | two-level shuffle: tasks write to buckets, then each bucket is split into its shards | `max_files=5e4` |
98
+
99
+ Every job script sets two Polars variables:
100
+ - `POLARS_TEMP_DIR=<scratch>/tmp/polars`, so spill files go to scratch instead of the node's
101
+ small `/var/tmp`
102
+ - `POLARS_OVERRIDE_TOTAL_MEMORY_MB` = the job's memory, so Polars budgets for the job's limit
103
+ rather than the whole node
104
+
105
+ ## 3. Validate the specification before the full run
106
+
107
+ **Compare against a trusted tool on a subsample.** Prep a subset of whole units and fit it with
108
+ dfeols and with your Stata/pyfixest code. Results should agree to about 1e-8.
109
+
110
+ ```python
111
+ sub = pl.scan_parquet(src).filter(pl.col("unit") % 50 == 0).collect()
112
+ ds_small = dfeols.prep(sub, shard_dir=..., partition="unit", ...)
113
+ ```
114
+
115
+ When comparing to `reghdfe`/pyfixest with two FEs, use a tight tolerance (1e-10). At looser
116
+ defaults they can stop early.
117
+
118
+ **Check recovery on synthetic data with the same structure.** Generate a panel with known
119
+ parameters and confirm the spec recovers them with valid SEs:
120
+
121
+ ```bash
122
+ python -m dfeols.bench --rows 1e7 --set T=104 --set n_groups=300 --profile <name>
123
+ ```
124
+
125
+ ## 4. Write the run script
126
+
127
+ Copy `examples/run_fit.py`, set `SOURCE`, `PREP` and `SPECS`:
128
+
129
+ ```python
130
+ SPECS = {
131
+ "main": dict(formula="y ~ x1 + x2:i(size) | unit + time", cluster="state"),
132
+ "alt": dict(formula="y + y2 ~ x1 + x2 | unit + group^time", cluster="unit", weights="w",
133
+ vcov=["CRV1", "hetero"]),
134
+ }
135
+ ```
136
+
137
+ **Formula syntax:**
138
+
139
+ | Term | Meaning | Stata |
140
+ |---|---|---|
141
+ | `x` | continuous variable | `x` |
142
+ | `a:b` | product | `c.a#c.b` |
143
+ | `i(c)` | dummies, first level dropped | `i.c` |
144
+ | `x:i(c)` | x times every level of c | `c.x#ibn.c` |
145
+ | `x:i(c, ref=0)` | same, level 0 dropped | `c.x#i.c` |
146
+ | `\| fe1 + fe2^fe3` | absorbed FEs | `absorb(fe1 fe2#fe3)` |
147
+
148
+ **Other options:**
149
+ - `vcov`: `"iid"`, `"hetero"` or `"CRV1"`; a list computes several in one pass.
150
+ - `weights`: analytic weights.
151
+ - Multiple outcomes share one pass and a common sample.
152
+
153
+ ## 5. Submit and monitor
154
+
155
+ ```bash
156
+ dfeols submit run_fit.py --profile <name> # coordinator as a batch job (default profile if omitted)
157
+ dfeols status <scratch>/runs --watch 60 # shards done, workers, rows/s, ETA
158
+ ```
159
+
160
+ Always submit the coordinator as a batch job. In an interactive allocation, the run dies when
161
+ the allocation expires.
162
+
163
+ **The ETA** is based on the workers connected right now. It drops as the queue starts more
164
+ workers. The "if all workers start" figure is the lower bound.
165
+
166
+ ## 6. Where everything goes
167
+
168
+ Everything lives under the profile's `scratch` directory, in fixed places:
169
+
170
+ | Path | Contents |
171
+ |---|---|
172
+ | `jobs/` | coordinator job scripts and their output (`dfeols submit` prints the path) |
173
+ | `logs/` | worker logs |
174
+ | `runs/<label>/` | your own fits (from `run_fit.py`): result.json, coefs.csv, status.json |
175
+ | `bench/raw_<tag>/`, `bench/shards_*` | benchmark data and shards, reused across runs |
176
+ | `bench/runs/<run_id>/<tag>/<spec>/` | one benchmark fit; never overwritten |
177
+ | `bench/results/results.csv` | every benchmark fit ever run, newest first |
178
+ | `bench/results/<run_id>.md` | summary of one benchmark run; `latest.md` is the most recent |
179
+
180
+ `<run_id>` is the start time plus the scheduler job id, e.g. `20261021-171800-35355433`.
181
+ `dfeols results` prints the benchmark table from any login node, and
182
+ `dfeols results --run <run_id>` shows one run. The table is rebuilt from `runs/` each time,
183
+ so it can't lose earlier fits. Fits from before this layout are picked up too, and their
184
+ `run_id` ends in `-legacy`.
185
+
186
+ ## 7. Results
187
+
188
+ Each spec writes to `<scratch>/runs/<label>/`:
189
+
190
+ - `result.json`: coefficients, full vcov for every requested type, N, singletons dropped,
191
+ cluster and FE level counts, collinear terms, timings. Load it with
192
+ `dfeols.Result.load(path)`, then use `.tidy()`, `.summary()`, `.coef()` and `.vcov()`.
193
+ - `coefs.csv`: the tidy table.
194
+ - `status.json` and `history.jsonl`: progress and ETA history.
195
+
196
+ ## 8. What is fast, and what is not
197
+
198
+ | Model structure | How it's handled | Relative cost |
199
+ |---|---|---|
200
+ | 1 FE local to the shard | exact, one pass | fastest |
201
+ | 1 local FE + crossing FEs (unit + time) | crossing FEs via per-level sums, no dummy columns | fast; T up to 2000 |
202
+ | 2 local FEs (unit + group^time, spell + account×week) | exact Schur-complement solve per connected group | fast |
203
+ | 2 local FEs + crossing FEs | crossing FEs as dense dummies | memory grows with the levels |
204
+ | 3+ local FEs | iterated (alternating projections) | slowest; demeaning dominates |
205
+ | crossing FE with > 2000 levels | not supported yet | — |
206
+
207
+ ## Scaling to a billion rows and beyond
208
+
209
+ The two presets generate balanced panels of 1e9 rows with known parameters. Each fits
210
+ `unit + time` (partitioned by unit, time crossing) and `unit + group^time` (partitioned by
211
+ group):
212
+
213
+ ```bash
214
+ # run 1e8 first to check timing and memory, then the full size
215
+ dfeols bench --profile <name> --preset units10m_t100 --rows 1e8
216
+ dfeols bench --profile <name> --preset units10m_t100
217
+ dfeols bench --profile <name> --preset units1m_t1000
218
+ ```
219
+
220
+ One job per data set at a time: a benchmark locks its data set (`<scratch>/bench/locks/`) while it
221
+ generates and preps, and a second job for the same preset and size stops at once with an error
222
+ naming the job that holds it. Different presets can run side by side. A lock left by a job that
223
+ died is taken over after 5 minutes. Shard sets built by versions before 0.6.2 with range-based
224
+ shards are rebuilt automatically ("re-preparing …").
225
+
226
+ Stress tests:
227
+
228
+ ```bash
229
+ # 1e10 rows, 100M units, 5,000 shards per set: hash ids, range shards, two-level shuffle
230
+ dfeols bench --profile <name> --preset units100m_t100 --rows 2e9 # step up first
231
+ dfeols bench --profile <name> --preset units100m_t100
232
+ # skewed group sizes: the largest group (~25M rows) becomes one oversized shard
233
+ dfeols bench --profile <name> --preset skewed_groups
234
+ # all of the 1e10 code paths on a tiny panel, on this machine
235
+ python -m dfeols.bench --preset units100m_t100 --rows 2e6 --profile local \
236
+ --prep-opt hash_levels=0 --prep-opt range_groups=0 --prep-opt max_files=0
237
+ ```
238
+
239
+ For the 1e10 preset:
240
+ - **Scratch:** about 1 TB (raw ≈ 370 GB plus two shard sets).
241
+ - **Coordinator:** 64 GB. It holds the 100M-unit level set and group counts during the reduce
242
+ step.
243
+ - **Workers:** generation runs on the workers for this preset.
244
+ - **Throughput:** at the 1e9 rates, expect roughly 30 min of fitting per spec on 40 workers.
245
+
246
+ Budget for each 1e9-row preset:
247
+
248
+ - **Scratch:** about 37 GB of raw data, plus about 40 GB per shard set. The presets need two
249
+ shard sets (by unit and by group), and prep briefly needs another copy as staging, so plan
250
+ on about 160 GB.
251
+ - **Coordinator:** it generates the data itself (while the first worker jobs wait in the queue;
252
+ `--gen-on-workers` moves generation to the workers). Prep then runs on the workers. Give it
253
+ 8–16 cores and 32 GB, e.g. a `[profiles.<name>.coordinator]` with `cores = 16`,
254
+ `memory = "32GB"`. Most of that is for generation, not prep.
255
+ - **Data folders** are named by DGP, size and non-default parameters, for example
256
+ `raw_panel_1e09_T100_bal1_G500`, so the two presets never collide.
257
+ - **Measured on OSCER at 1e9 rows** (both presets, up to 40 four-core workers, about 36 on
258
+ average; distributed prep):
259
+
260
+ | | 10M units × T=100 | 1M units × T=1000 |
261
+ |---|---|---|
262
+ | generate (coordinator) | 48 s | 337 s |
263
+ | prep by unit / by group | 700 s / 543 s | 524 s / 1300 s |
264
+ | fit `unit + time` | 178 s | 91 s |
265
+ | fit `unit + group^time` | 158 s | 157 s |
266
+ | peak worker memory | 3.4 GB | 2.8 GB |
267
+ | recovers the truth | yes (max \|z\| 1.09, 2.77) | yes (max \|z\| 0.50, 1.36) |
268
+
269
+ End to end: about 27 min for T=100 and 40 min for T=1000. Prep is most of it, and part of
270
+ that is queue wait for the first workers.