dfeols 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dfeols-0.7.0/CHANGELOG.md +191 -0
- dfeols-0.7.0/CITATION.cff +20 -0
- dfeols-0.7.0/CONTRIBUTING.md +69 -0
- dfeols-0.7.0/HOWTO.md +270 -0
- dfeols-0.7.0/LICENSE +674 -0
- dfeols-0.7.0/MANIFEST.in +6 -0
- dfeols-0.7.0/PKG-INFO +315 -0
- dfeols-0.7.0/README.md +264 -0
- dfeols-0.7.0/ci/min-versions.txt +12 -0
- dfeols-0.7.0/docs/ARCHITECTURE.md +153 -0
- dfeols-0.7.0/examples/profiles/oscer.toml +28 -0
- dfeols-0.7.0/examples/profiles/templates.toml +50 -0
- dfeols-0.7.0/examples/run_fit.py +51 -0
- dfeols-0.7.0/pyproject.toml +111 -0
- dfeols-0.7.0/setup.cfg +4 -0
- dfeols-0.7.0/src/dfeols/__init__.py +34 -0
- dfeols-0.7.0/src/dfeols/bench.py +543 -0
- dfeols-0.7.0/src/dfeols/cli.py +314 -0
- dfeols-0.7.0/src/dfeols/cluster.py +318 -0
- dfeols-0.7.0/src/dfeols/demean.py +181 -0
- dfeols-0.7.0/src/dfeols/estimate.py +536 -0
- dfeols-0.7.0/src/dfeols/executors.py +141 -0
- dfeols-0.7.0/src/dfeols/formula.py +190 -0
- dfeols-0.7.0/src/dfeols/linalg.py +41 -0
- dfeols-0.7.0/src/dfeols/lock.py +136 -0
- dfeols-0.7.0/src/dfeols/prep.py +348 -0
- dfeols-0.7.0/src/dfeols/prep_dist.py +849 -0
- dfeols-0.7.0/src/dfeols/progress.py +263 -0
- dfeols-0.7.0/src/dfeols/results.py +108 -0
- dfeols-0.7.0/src/dfeols/site.py +209 -0
- dfeols-0.7.0/src/dfeols/synth/__init__.py +209 -0
- dfeols-0.7.0/src/dfeols/synth/__main__.py +5 -0
- dfeols-0.7.0/src/dfeols/synth/influencer.py +250 -0
- dfeols-0.7.0/src/dfeols/synth/panel.py +210 -0
- dfeols-0.7.0/src/dfeols/validate.py +59 -0
- dfeols-0.7.0/src/dfeols/vcov.py +40 -0
- dfeols-0.7.0/src/dfeols/worker.py +426 -0
- dfeols-0.7.0/src/dfeols.egg-info/PKG-INFO +315 -0
- dfeols-0.7.0/src/dfeols.egg-info/SOURCES.txt +52 -0
- dfeols-0.7.0/src/dfeols.egg-info/dependency_links.txt +1 -0
- dfeols-0.7.0/src/dfeols.egg-info/entry_points.txt +2 -0
- dfeols-0.7.0/src/dfeols.egg-info/requires.txt +26 -0
- dfeols-0.7.0/src/dfeols.egg-info/top_level.txt +1 -0
- dfeols-0.7.0/tests/conftest.py +114 -0
- dfeols-0.7.0/tests/simdata.py +56 -0
- dfeols-0.7.0/tests/test_cli.py +20 -0
- dfeols-0.7.0/tests/test_demean.py +42 -0
- dfeols-0.7.0/tests/test_influencer.py +274 -0
- dfeols-0.7.0/tests/test_memory_check.py +28 -0
- dfeols-0.7.0/tests/test_panel.py +159 -0
- dfeols-0.7.0/tests/test_prep_distributed.py +485 -0
- dfeols-0.7.0/tests/test_prep_lock.py +102 -0
- dfeols-0.7.0/tests/test_progress.py +61 -0
- dfeols-0.7.0/tests/test_vs_pyfixest.py +290 -0
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to dfeols are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
|
|
5
|
+
[semantic versioning](https://semver.org/) (before 1.0, minor versions may
|
|
6
|
+
break the API).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.7.0] - 2026-10-22
|
|
11
|
+
|
|
12
|
+
First public release.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
- `dfeols --version`.
|
|
16
|
+
- CI on GitHub Actions: Linux and macOS, Python 3.10–3.14, a Dask job, a
|
|
17
|
+
minimum-versions job and a nightly job against the newest polars and
|
|
18
|
+
pyfixest.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
- Source moved to a `src/` layout; `examples/` and `tests/` are not installed.
|
|
22
|
+
- Licensed under GPL-3.0-or-later.
|
|
23
|
+
- Dependency ranges are now tested at both ends: `polars>=1.37,<3`,
|
|
24
|
+
`pyfixest>=0.50.1,<1`, `numpy>=1.25`; Python 3.10–3.14.
|
|
25
|
+
- Dense FE codes are `int64` (were `uint64`), which fixes `np.bincount`
|
|
26
|
+
failures under NumPy 1.x.
|
|
27
|
+
- pyfixest comparisons (tests, `dfeols.bench` baseline) use the typed
|
|
28
|
+
`demeaner=MapDemeaner(...)` argument on pyfixest ≥ 0.60 and fall back to
|
|
29
|
+
`fixef_tol`/`fixef_maxiter` on older releases
|
|
30
|
+
(`dfeols.formula.pyfixest_tol_kwargs`).
|
|
31
|
+
- pyfixest's compiled demeaner (a private API) is imported defensively; if
|
|
32
|
+
its location or signature changes, dfeols logs a warning and uses the NumPy
|
|
33
|
+
fallback. A test fails if the compiled path is not found, so the nightly CI
|
|
34
|
+
job catches upstream changes.
|
|
35
|
+
- The ETA bootstrap no longer shares one random generator across calls.
|
|
36
|
+
- Code formatted and linted with ruff.
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
These include the fixes from the 0.6.2 snapshot (below), merged into the new
|
|
40
|
+
layout, plus:
|
|
41
|
+
- The coordinator merges per-file scan results as they arrive instead of
|
|
42
|
+
holding them until the scan ends. At 1e10 rows it kept 5,000 copies of the
|
|
43
|
+
group × time level set and its nesting-check tables (well over 100 GB) and
|
|
44
|
+
was killed; merged, these stay near the number of distinct keys.
|
|
45
|
+
- `fit` checks the largest shard's memory estimate against the profile's
|
|
46
|
+
worker memory: it warns above it and refuses above 1.5×, instead of
|
|
47
|
+
letting the scheduler kill worker after worker on the same shard.
|
|
48
|
+
- prep refuses to write a shard larger than twice the target plus the
|
|
49
|
+
largest partition group (only a bug can produce one), and warns when one
|
|
50
|
+
partition group is more than 4× the target.
|
|
51
|
+
- A `shard_dir` left by an interrupted prep (no `dfeols_meta.json`) is
|
|
52
|
+
cleared before prepping again, so its stale pieces cannot join the new
|
|
53
|
+
dataset.
|
|
54
|
+
- `fit` reads the levels of `i()` variables with executor tasks and caches
|
|
55
|
+
them in `<shard_dir>/_levels/`; it read the column from every shard on
|
|
56
|
+
the coordinator, 55 minutes before round 1 at 1e10 rows.
|
|
57
|
+
- The prep scan runs over batches of files (about 4 tasks per worker); each
|
|
58
|
+
task reduces its level sets and nesting checks before returning, so the
|
|
59
|
+
coordinator receives one copy per batch instead of one per file. At 1e10
|
|
60
|
+
rows the per-file results made the by-unit scan coordinator-bound
|
|
61
|
+
(50 minutes, against 16 for the by-group scan).
|
|
62
|
+
- Prep stages are no longer given an ETA projected from the scan rate (the
|
|
63
|
+
split stage showed "8866M rows/s, ETA 0:00:01" while its 200M-row tasks ran
|
|
64
|
+
for ~20 minutes); the round-1-to-round-2 prior now applies to fits only.
|
|
65
|
+
- The ETA no longer crashes with `OverflowError` when no workers are
|
|
66
|
+
connected (an infinite ETA is reported as unknown).
|
|
67
|
+
- Prep tasks counted their output by listing the folder they had written
|
|
68
|
+
into. On Lustre, listing a folder that 40 workers are writing into can
|
|
69
|
+
return an entry twice, so a bucket file was counted twice per repeated
|
|
70
|
+
entry: "encoded 10,002,000,000 rows, expected 10,000,000,000" at 1e10, with
|
|
71
|
+
every file on disk correct (10 entries of 200,000 rows; the earlier +2.8M
|
|
72
|
+
and +400k were the same). Tasks now count exactly the files they handed to
|
|
73
|
+
Polars, and the input file list is deduplicated.
|
|
74
|
+
- Every encode and split task checks that it wrote exactly the rows the scan
|
|
75
|
+
counted for its input; on a mismatch it logs which shards or buckets carry
|
|
76
|
+
the difference, moves its files to `<shard_dir>/_rejected/` (the first 5
|
|
77
|
+
attempts) and reruns, up to 3 attempts, and otherwise fails naming the task.
|
|
78
|
+
At 1e10 rows on OSCER, 4 of 1,000 encode attempts wrote 200k–800k extra
|
|
79
|
+
rows (whole multiples of one bucket's share) and were correct when rerun.
|
|
80
|
+
- Dask task keys include a per-run id; every prep used the same keys
|
|
81
|
+
(`dfeols-encode-prep-17`), so a second prep in the same job could be handed
|
|
82
|
+
a result Dask still held for the first.
|
|
83
|
+
- Folder locks also take over immediately when their holder ran on this
|
|
84
|
+
host and is gone, and a release never removes another holder's lock.
|
|
85
|
+
|
|
86
|
+
### Removed
|
|
87
|
+
- `dfeols.slurm` (deprecated shim). Use `dfeols.executor(profile)` with a
|
|
88
|
+
site profile (`dfeols config init`).
|
|
89
|
+
- `benchmarks/bench.py` stub; use `python -m dfeols.bench`.
|
|
90
|
+
|
|
91
|
+
## [0.6.2]
|
|
92
|
+
|
|
93
|
+
Fixes from the first 1e10 attempts on OSCER (a snapshot, not published).
|
|
94
|
+
|
|
95
|
+
### Fixed
|
|
96
|
+
- Range-based shard assignment (more than `range_groups` partition groups,
|
|
97
|
+
new in 0.6.0) used polars' `UInt32` row counts, which wrap at 2**32: at
|
|
98
|
+
2e9 rows it produced 2 shards instead of 1,000 (at 1e9 rows with 10M units,
|
|
99
|
+
4 shards). Counts are Int64 and the bins and boundaries are checked to be
|
|
100
|
+
monotone. **Shard sets prepared with range assignment before 0.6.2 must be
|
|
101
|
+
rebuilt**; `dfeols bench` detects them (no `prep_version` in their
|
|
102
|
+
metadata) and re-preps them.
|
|
103
|
+
- A task that ran twice at once (a rerun overlapping its first attempt, or
|
|
104
|
+
another job writing the same folder) could be counted twice ("encoded
|
|
105
|
+
2,000,400,000 rows, expected 2,000,000,000"). Every task attempt writes
|
|
106
|
+
uniquely named files and reports them; the coordinator keeps one attempt
|
|
107
|
+
per task, ignores duplicate results, sweeps the other files, and gives the
|
|
108
|
+
split and merge steps explicit file lists. Merge inputs are deleted by the
|
|
109
|
+
coordinator once the merge is accepted.
|
|
110
|
+
- Two jobs could write the same folder. Shard folders (during prep) and
|
|
111
|
+
benchmark data sets (during generation and prep) are locked with a sibling
|
|
112
|
+
`<folder>.lock` (O_EXCL, refreshed every 30 s, stale after 5 min); a
|
|
113
|
+
second job fails at once with `dfeols.LockedError` naming the holder.
|
|
114
|
+
- Overwrites rename the old folder before deleting it, so a straggling
|
|
115
|
+
writer cannot make the delete fail ("Directory not empty: '_buckets'").
|
|
116
|
+
|
|
117
|
+
### Added
|
|
118
|
+
- `prep_version` in `dfeols_meta.json`.
|
|
119
|
+
|
|
120
|
+
## [0.6.1]
|
|
121
|
+
|
|
122
|
+
### Added
|
|
123
|
+
- `dfeols bench`: submits a benchmark as a coordinator job with one
|
|
124
|
+
`--profile`; the job's profile is passed through to the benchmark
|
|
125
|
+
(`$DFEOLS_PROFILE`).
|
|
126
|
+
|
|
127
|
+
### Changed
|
|
128
|
+
- `dfeols submit` and `dfeols bench` print the profile and module lines used.
|
|
129
|
+
- `dfeols config init --force` backs up the existing config first.
|
|
130
|
+
- OSCER example coordinator raised to 16 cores / 64 GB (data generation runs
|
|
131
|
+
on the coordinator).
|
|
132
|
+
|
|
133
|
+
## [0.6.0]
|
|
134
|
+
|
|
135
|
+
### Added
|
|
136
|
+
- Hash ids for nested FEs with more than 2e6 levels (collisions checked
|
|
137
|
+
exactly, with a fallback to dense codes) and range-based shard assignment
|
|
138
|
+
for more than 5e6 partition groups, for 1e10-row datasets.
|
|
139
|
+
- Two-level shuffle in distributed prep when there would be more than 5e4
|
|
140
|
+
piece files.
|
|
141
|
+
- Stress presets `units100m_t100` (1e10 rows) and `skewed_groups`.
|
|
142
|
+
- Round-2 ETA prior.
|
|
143
|
+
|
|
144
|
+
### Fixed
|
|
145
|
+
- Nested-in-cluster detection with signed ids.
|
|
146
|
+
- Worker memory estimate.
|
|
147
|
+
|
|
148
|
+
## [0.5.1]
|
|
149
|
+
|
|
150
|
+
### Changed
|
|
151
|
+
- Flat piece files; shards are merged only when fragmented.
|
|
152
|
+
- Adaptive task size in distributed prep.
|
|
153
|
+
- The coordinator runs read-only scan tasks itself while no workers are
|
|
154
|
+
connected.
|
|
155
|
+
|
|
156
|
+
## [0.5.0]
|
|
157
|
+
|
|
158
|
+
### Added
|
|
159
|
+
- Distributed prep on workers (scan → reduce → encode → merge/split).
|
|
160
|
+
- Polars spill directory and memory pinning to the job allocation.
|
|
161
|
+
- Collision-free data folders.
|
|
162
|
+
|
|
163
|
+
## [0.4.0]
|
|
164
|
+
|
|
165
|
+
### Added
|
|
166
|
+
- Sparse crossing FEs: with one local FE, crossing-FE blocks of X'WX come
|
|
167
|
+
from per-level sums and no dummy columns are built.
|
|
168
|
+
- Billion-row benchmark presets.
|
|
169
|
+
- Lazy worker scale-down (`idle_minutes`).
|
|
170
|
+
- Append-only results layout and `dfeols results`.
|
|
171
|
+
|
|
172
|
+
## [0.3.0]
|
|
173
|
+
|
|
174
|
+
### Added
|
|
175
|
+
- General HDFE with a generic unit × time panel DGP.
|
|
176
|
+
- Scheduler-agnostic TOML site profiles (Slurm, PBS, SGE, LSF, local).
|
|
177
|
+
- Exact two-way demeaning (Schur complement per connected component).
|
|
178
|
+
|
|
179
|
+
## [0.2.0]
|
|
180
|
+
|
|
181
|
+
### Added
|
|
182
|
+
- Synthetic DGP with known parameters, validity suite, benchmark runner.
|
|
183
|
+
|
|
184
|
+
## [0.1.0]
|
|
185
|
+
|
|
186
|
+
### Added
|
|
187
|
+
- Two-round map/reduce fit, Serial and Dask executors, ETA, agreement with
|
|
188
|
+
pyfixest.
|
|
189
|
+
|
|
190
|
+
[Unreleased]: https://github.com/jemather/dfeols/compare/v0.7.0...HEAD
|
|
191
|
+
[0.7.0]: https://github.com/jemather/dfeols/releases/tag/v0.7.0
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use dfeols in your research, please cite it as below."
|
|
3
|
+
title: "dfeols: sharded, elastic high-dimensional fixed-effects OLS for HPC clusters"
|
|
4
|
+
type: software
|
|
5
|
+
authors:
|
|
6
|
+
- family-names: Matherly
|
|
7
|
+
given-names: Ted
|
|
8
|
+
affiliation: University of Oklahoma
|
|
9
|
+
orcid: "https://orcid.org/0000-0002-8628-1004"
|
|
10
|
+
version: 0.7.0
|
|
11
|
+
date-released: 2026-10-22
|
|
12
|
+
license: GPL-3.0-or-later
|
|
13
|
+
repository-code: "https://github.com/jemather/dfeols"
|
|
14
|
+
keywords:
|
|
15
|
+
- econometrics
|
|
16
|
+
- fixed effects
|
|
17
|
+
- high-dimensional fixed effects
|
|
18
|
+
- panel data
|
|
19
|
+
- high-performance computing
|
|
20
|
+
- Slurm
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Contributing to dfeols
|
|
2
|
+
|
|
3
|
+
Thanks for your interest. Bug reports, site profiles for new clusters, and
|
|
4
|
+
pull requests are all welcome.
|
|
5
|
+
|
|
6
|
+
## Reporting a problem
|
|
7
|
+
|
|
8
|
+
Open an issue with:
|
|
9
|
+
|
|
10
|
+
- `dfeols --version`, `python --version`, and the versions of polars,
|
|
11
|
+
pyfixest and (if used) dask / dask-jobqueue;
|
|
12
|
+
- the scheduler (Slurm, PBS, …) and the output of `dfeols config show`
|
|
13
|
+
with account names removed;
|
|
14
|
+
- for wrong results: the formula, `dfeols info <shard_dir>`, and if possible
|
|
15
|
+
a small synthetic reproduction (`python -m dfeols.synth`);
|
|
16
|
+
- for crashes on a cluster: the coordinator log and one worker log.
|
|
17
|
+
|
|
18
|
+
## Development setup
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
git clone https://github.com/jemather/dfeols.git
|
|
22
|
+
cd dfeols
|
|
23
|
+
python -m venv .venv && source .venv/bin/activate
|
|
24
|
+
pip install -e ".[dev]"
|
|
25
|
+
pre-commit install # optional: runs ruff on each commit
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Checks
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
ruff check . && ruff format --check .
|
|
32
|
+
pytest # ~2 minutes on 2 cores, 0.5M-row synthetic panels
|
|
33
|
+
pytest --executor dask:2 # the same fits on a local Dask cluster
|
|
34
|
+
pytest -m slow # Monte Carlo coverage (long)
|
|
35
|
+
mypy # advisory for now
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Larger test data: `pytest --rows 5e6 --data-dir /scratch/$USER/dfeols-tests`.
|
|
39
|
+
|
|
40
|
+
## What a change needs
|
|
41
|
+
|
|
42
|
+
- **Correctness first.** Anything that touches estimation must keep agreement
|
|
43
|
+
with pyfixest (`tests/test_vs_pyfixest.py`) and recovery on the synthetic
|
|
44
|
+
DGPs (`tests/test_panel.py`, `tests/test_influencer.py`). New model
|
|
45
|
+
features need a pyfixest/fixest comparison test.
|
|
46
|
+
- **Invariance.** Results must not depend on the partition key, the number
|
|
47
|
+
of shards, or the executor; add to the invariance tests where relevant.
|
|
48
|
+
- **No site-specific code.** Queues, modules, paths and accounts belong in
|
|
49
|
+
site profiles (`examples/profiles/`), never in the package.
|
|
50
|
+
- Update `CHANGELOG.md` under *Unreleased*.
|
|
51
|
+
|
|
52
|
+
## New site profiles
|
|
53
|
+
|
|
54
|
+
If you run dfeols on a new cluster, a tested profile in
|
|
55
|
+
`examples/profiles/` (with the site name, scheduler and date tested in a
|
|
56
|
+
comment) is a very useful contribution. PBS, SGE and LSF are currently
|
|
57
|
+
untested on real systems.
|
|
58
|
+
|
|
59
|
+
## Releasing (maintainers)
|
|
60
|
+
|
|
61
|
+
1. Update the version in `src/dfeols/__init__.py`, `CITATION.cff` and the
|
|
62
|
+
changelog.
|
|
63
|
+
2. Tag `vX.Y.Z` and push the tag. The release workflow builds the sdist and
|
|
64
|
+
wheel, publishes to TestPyPI, then to PyPI (trusted publishing).
|
|
65
|
+
|
|
66
|
+
## License
|
|
67
|
+
|
|
68
|
+
By contributing you agree that your contributions are licensed under the
|
|
69
|
+
project's license, GPL-3.0-or-later.
|
dfeols-0.7.0/HOWTO.md
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# dfeols: how to use it with a real dataset
|
|
2
|
+
|
|
3
|
+
This is the short version. The README covers the design; `python -m dfeols.bench --help`,
|
|
4
|
+
`python -m dfeols.synth --help` and `dfeols --help` cover the tools.
|
|
5
|
+
|
|
6
|
+
## 1. One-time setup on a machine
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
python -m venv ~/envs/dfeols && source ~/envs/dfeols/bin/activate
|
|
10
|
+
pip install "dfeols[dask]" # or: pip install "dfeols[dask] @ git+https://github.com/jemather/dfeols"
|
|
11
|
+
dfeols config init # writes ~/.config/dfeols/config.toml
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Create the venv on a filesystem the compute nodes can see (usually your home directory), with
|
|
15
|
+
the same Python module the profile's `prologue` loads.
|
|
16
|
+
|
|
17
|
+
Edit the profile in that file:
|
|
18
|
+
- `scheduler`, `queue`, `account`
|
|
19
|
+
- `prologue`: the module loads and venv activation every job runs
|
|
20
|
+
- `scratch`: a shared filesystem every node can see
|
|
21
|
+
- `[worker]` cores, memory, walltime and `max_workers`
|
|
22
|
+
|
|
23
|
+
Smaller, shorter worker jobs start sooner on a busy queue. Workers are released only after
|
|
24
|
+
`idle_minutes` (default 15) with nothing to do, so a pause between fits doesn't hand back
|
|
25
|
+
workers that waited in the queue. `examples/profiles/oscer.toml`
|
|
26
|
+
is a tested Slurm profile; `examples/profiles/templates.toml` covers PBS, SGE, LSF and local.
|
|
27
|
+
|
|
28
|
+
Set `default = "<name>"` at the top of the file so commands use your profile without
|
|
29
|
+
`--profile`. `dfeols config init --force` replaces the file with the template (it saves a backup
|
|
30
|
+
first). Every `submit`/`bench` prints the profile and module lines it used.
|
|
31
|
+
|
|
32
|
+
Then check:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
dfeols check # one worker starts through the scheduler and runs a task
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
If it fails:
|
|
39
|
+
- jobs pending: queue limits
|
|
40
|
+
- jobs exit at once: check the worker logs in `<scratch>/logs`
|
|
41
|
+
- "Timed out trying to connect": set `interface` in the profile
|
|
42
|
+
|
|
43
|
+
## 2. Prep the data once
|
|
44
|
+
|
|
45
|
+
Prep encodes the FE and cluster variables, checks nesting and writes Parquet shards.
|
|
46
|
+
Every model on that dataset reuses the shards.
|
|
47
|
+
|
|
48
|
+
```python
|
|
49
|
+
import dfeols
|
|
50
|
+
prof = dfeols.profile()
|
|
51
|
+
ex = dfeols.executor(prof)
|
|
52
|
+
ds = dfeols.prep("/data/panel/*.parquet", # Parquet files (or a frame / CSV without executor)
|
|
53
|
+
shard_dir=prof.scratch_dir("shards", "panel"),
|
|
54
|
+
executor=ex, # prep on the workers
|
|
55
|
+
partition="unit", # groups never split across shards
|
|
56
|
+
fixed_effects=["unit", "time", "unit^quarter"], # every FE any model uses
|
|
57
|
+
clusters=["unit", "state"], # every cluster variable any model uses
|
|
58
|
+
target_rows=2_000_000) # rows per shard
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
**Choose the partition key** so that the most FEs are nested in it.
|
|
62
|
+
- Unit × time: partition by unit. The unit FE is then handled inside each shard, and time
|
|
63
|
+
crosses shards.
|
|
64
|
+
- `unit + group^time` with units nested in groups: partition by group, so both are local.
|
|
65
|
+
- Run `dfeols info <shard_dir>` to see how each FE was classified.
|
|
66
|
+
|
|
67
|
+
Crossing FEs need up to `max_dummy_levels` (2000) levels; high-cardinality crossing FEs are
|
|
68
|
+
not supported yet.
|
|
69
|
+
|
|
70
|
+
**Shard size:** worker memory peaks at roughly 3.5 × rows × (outcomes + regressors) × 8 bytes.
|
|
71
|
+
On OSCER, 2M-row shards peaked at about 15 GB with 74 regressors. Aim for at least 5 shards per
|
|
72
|
+
worker so the load balances and the ETA has data.
|
|
73
|
+
|
|
74
|
+
**Run prep on the workers** with `executor=ex`. This needs Parquet input; split a very large
|
|
75
|
+
single file into many, which also parallelizes. Prep then runs as map/reduce rounds:
|
|
76
|
+
|
|
77
|
+
1. scan each file for its FE levels and group sizes; while no workers have connected yet, the
|
|
78
|
+
coordinator scans files itself
|
|
79
|
+
2. build the code tables on the coordinator
|
|
80
|
+
3. encode and bucket rows on the workers, in about 4 tasks per worker so the last wave is
|
|
81
|
+
short; each task writes one flat piece file per shard it touches
|
|
82
|
+
4. merge only shards that ended up in more than 8 pieces (when partition groups are spread
|
|
83
|
+
across many input files); other shards keep their pieces as is
|
|
84
|
+
|
|
85
|
+
The coordinator holds only the code tables: 0.3 GB for 30M rows in testing, and roughly
|
|
86
|
+
8 bytes per distinct FE level (10M units ≈ 80 MB). Without `executor`, prep streams on the
|
|
87
|
+
coordinator alone. That's fine up to a few hundred million rows if the coordinator has the
|
|
88
|
+
memory.
|
|
89
|
+
|
|
90
|
+
At very large scale prep switches strategy on its own (each threshold can be set as a `prep()`
|
|
91
|
+
keyword):
|
|
92
|
+
|
|
93
|
+
| Situation | What prep does | Threshold |
|
|
94
|
+
|---|---|---|
|
|
95
|
+
| a nested FE or cluster with millions of levels (e.g. 100M units) | 64-bit hash ids instead of a code table join; collisions are checked exactly, with a fallback to codes | `hash_levels=2e6` |
|
|
96
|
+
| millions of partition groups | shards are contiguous ranges of the partition key (or its hash for multi-column keys), so tasks need only the boundaries | `range_groups=5e6` |
|
|
97
|
+
| partition groups scattered across input files, so one piece per (task, shard) would mean too many files | two-level shuffle: tasks write to buckets, then each bucket is split into its shards | `max_files=5e4` |
|
|
98
|
+
|
|
99
|
+
Every job script sets two Polars variables:
|
|
100
|
+
- `POLARS_TEMP_DIR=<scratch>/tmp/polars`, so spill files go to scratch instead of the node's
|
|
101
|
+
small `/var/tmp`
|
|
102
|
+
- `POLARS_OVERRIDE_TOTAL_MEMORY_MB` = the job's memory, so Polars budgets for the job's limit
|
|
103
|
+
rather than the whole node
|
|
104
|
+
|
|
105
|
+
## 3. Validate the specification before the full run
|
|
106
|
+
|
|
107
|
+
**Compare against a trusted tool on a subsample.** Prep a subset of whole units and fit it with
|
|
108
|
+
dfeols and with your Stata/pyfixest code. Results should agree to about 1e-8.
|
|
109
|
+
|
|
110
|
+
```python
|
|
111
|
+
sub = pl.scan_parquet(src).filter(pl.col("unit") % 50 == 0).collect()
|
|
112
|
+
ds_small = dfeols.prep(sub, shard_dir=..., partition="unit", ...)
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
When comparing to `reghdfe`/pyfixest with two FEs, use a tight tolerance (1e-10). At looser
|
|
116
|
+
defaults they can stop early.
|
|
117
|
+
|
|
118
|
+
**Check recovery on synthetic data with the same structure.** Generate a panel with known
|
|
119
|
+
parameters and confirm the spec recovers them with valid SEs:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
python -m dfeols.bench --rows 1e7 --set T=104 --set n_groups=300 --profile <name>
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
## 4. Write the run script
|
|
126
|
+
|
|
127
|
+
Copy `examples/run_fit.py`, set `SOURCE`, `PREP` and `SPECS`:
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
SPECS = {
|
|
131
|
+
"main": dict(formula="y ~ x1 + x2:i(size) | unit + time", cluster="state"),
|
|
132
|
+
"alt": dict(formula="y + y2 ~ x1 + x2 | unit + group^time", cluster="unit", weights="w",
|
|
133
|
+
vcov=["CRV1", "hetero"]),
|
|
134
|
+
}
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
**Formula syntax:**
|
|
138
|
+
|
|
139
|
+
| Term | Meaning | Stata |
|
|
140
|
+
|---|---|---|
|
|
141
|
+
| `x` | continuous variable | `x` |
|
|
142
|
+
| `a:b` | product | `c.a#c.b` |
|
|
143
|
+
| `i(c)` | dummies, first level dropped | `i.c` |
|
|
144
|
+
| `x:i(c)` | x times every level of c | `c.x#ibn.c` |
|
|
145
|
+
| `x:i(c, ref=0)` | same, level 0 dropped | `c.x#i.c` |
|
|
146
|
+
| `\| fe1 + fe2^fe3` | absorbed FEs | `absorb(fe1 fe2#fe3)` |
|
|
147
|
+
|
|
148
|
+
**Other options:**
|
|
149
|
+
- `vcov`: `"iid"`, `"hetero"` or `"CRV1"`; a list computes several in one pass.
|
|
150
|
+
- `weights`: analytic weights.
|
|
151
|
+
- Multiple outcomes share one pass and a common sample.
|
|
152
|
+
|
|
153
|
+
## 5. Submit and monitor
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
dfeols submit run_fit.py --profile <name> # coordinator as a batch job (default profile if omitted)
|
|
157
|
+
dfeols status <scratch>/runs --watch 60 # shards done, workers, rows/s, ETA
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Always submit the coordinator as a batch job. In an interactive allocation, the run dies when
|
|
161
|
+
the allocation expires.
|
|
162
|
+
|
|
163
|
+
**The ETA** is based on the workers connected right now. It drops as the queue starts more
|
|
164
|
+
workers. The "if all workers start" figure is the lower bound.
|
|
165
|
+
|
|
166
|
+
## 6. Where everything goes
|
|
167
|
+
|
|
168
|
+
Everything lives under the profile's `scratch` directory, in fixed places:
|
|
169
|
+
|
|
170
|
+
| Path | Contents |
|
|
171
|
+
|---|---|
|
|
172
|
+
| `jobs/` | coordinator job scripts and their output (`dfeols submit` prints the path) |
|
|
173
|
+
| `logs/` | worker logs |
|
|
174
|
+
| `runs/<label>/` | your own fits (from `run_fit.py`): result.json, coefs.csv, status.json |
|
|
175
|
+
| `bench/raw_<tag>/`, `bench/shards_*` | benchmark data and shards, reused across runs |
|
|
176
|
+
| `bench/runs/<run_id>/<tag>/<spec>/` | one benchmark fit; never overwritten |
|
|
177
|
+
| `bench/results/results.csv` | every benchmark fit ever run, newest first |
|
|
178
|
+
| `bench/results/<run_id>.md` | summary of one benchmark run; `latest.md` is the most recent |
|
|
179
|
+
|
|
180
|
+
`<run_id>` is the start time plus the scheduler job id, e.g. `20261021-171800-35355433`.
|
|
181
|
+
`dfeols results` prints the benchmark table from any login node, and
|
|
182
|
+
`dfeols results --run <run_id>` shows one run. The table is rebuilt from `runs/` each time,
|
|
183
|
+
so it can't lose earlier fits. Fits from before this layout are picked up too, and their
|
|
184
|
+
`run_id` ends in `-legacy`.
|
|
185
|
+
|
|
186
|
+
## 7. Results
|
|
187
|
+
|
|
188
|
+
Each spec writes to `<scratch>/runs/<label>/`:
|
|
189
|
+
|
|
190
|
+
- `result.json`: coefficients, full vcov for every requested type, N, singletons dropped,
|
|
191
|
+
cluster and FE level counts, collinear terms, timings. Load it with
|
|
192
|
+
`dfeols.Result.load(path)`, then use `.tidy()`, `.summary()`, `.coef()` and `.vcov()`.
|
|
193
|
+
- `coefs.csv`: the tidy table.
|
|
194
|
+
- `status.json` and `history.jsonl`: progress and ETA history.
|
|
195
|
+
|
|
196
|
+
## 8. What is fast, and what is not
|
|
197
|
+
|
|
198
|
+
| Model structure | How it's handled | Relative cost |
|
|
199
|
+
|---|---|---|
|
|
200
|
+
| 1 FE local to the shard | exact, one pass | fastest |
|
|
201
|
+
| 1 local FE + crossing FEs (unit + time) | crossing FEs via per-level sums, no dummy columns | fast; T up to 2000 |
|
|
202
|
+
| 2 local FEs (unit + group^time, spell + account×week) | exact Schur-complement solve per connected group | fast |
|
|
203
|
+
| 2 local FEs + crossing FEs | crossing FEs as dense dummies | memory grows with the levels |
|
|
204
|
+
| 3+ local FEs | iterated (alternating projections) | slowest; demeaning dominates |
|
|
205
|
+
| crossing FE with > 2000 levels | not supported yet | — |
|
|
206
|
+
|
|
207
|
+
## Scaling to a billion rows and beyond
|
|
208
|
+
|
|
209
|
+
The two presets generate balanced panels of 1e9 rows with known parameters. Each fits
|
|
210
|
+
`unit + time` (partitioned by unit, time crossing) and `unit + group^time` (partitioned by
|
|
211
|
+
group):
|
|
212
|
+
|
|
213
|
+
```bash
|
|
214
|
+
# run 1e8 first to check timing and memory, then the full size
|
|
215
|
+
dfeols bench --profile <name> --preset units10m_t100 --rows 1e8
|
|
216
|
+
dfeols bench --profile <name> --preset units10m_t100
|
|
217
|
+
dfeols bench --profile <name> --preset units1m_t1000
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
One job per data set at a time: a benchmark locks its data set (`<scratch>/bench/locks/`) while it
|
|
221
|
+
generates and preps, and a second job for the same preset and size stops at once with an error
|
|
222
|
+
naming the job that holds it. Different presets can run side by side. A lock left by a job that
|
|
223
|
+
died is taken over after 5 minutes. Shard sets built by versions before 0.6.2 with range-based
|
|
224
|
+
shards are rebuilt automatically ("re-preparing …").
|
|
225
|
+
|
|
226
|
+
Stress tests:
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
# 1e10 rows, 100M units, 5,000 shards per set: hash ids, range shards, two-level shuffle
|
|
230
|
+
dfeols bench --profile <name> --preset units100m_t100 --rows 2e9 # step up first
|
|
231
|
+
dfeols bench --profile <name> --preset units100m_t100
|
|
232
|
+
# skewed group sizes: the largest group (~25M rows) becomes one oversized shard
|
|
233
|
+
dfeols bench --profile <name> --preset skewed_groups
|
|
234
|
+
# all of the 1e10 code paths on a tiny panel, on this machine
|
|
235
|
+
python -m dfeols.bench --preset units100m_t100 --rows 2e6 --profile local \
|
|
236
|
+
--prep-opt hash_levels=0 --prep-opt range_groups=0 --prep-opt max_files=0
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
For the 1e10 preset:
|
|
240
|
+
- **Scratch:** about 1 TB (raw ≈ 370 GB plus two shard sets).
|
|
241
|
+
- **Coordinator:** 64 GB. It holds the 100M-unit level set and group counts during the reduce
|
|
242
|
+
step.
|
|
243
|
+
- **Workers:** generation runs on the workers for this preset.
|
|
244
|
+
- **Throughput:** at the 1e9 rates, expect roughly 30 min of fitting per spec on 40 workers.
|
|
245
|
+
|
|
246
|
+
Budget for each 1e9-row preset:
|
|
247
|
+
|
|
248
|
+
- **Scratch:** about 37 GB of raw data, plus about 40 GB per shard set. The presets need two
|
|
249
|
+
shard sets (by unit and by group), and prep briefly needs another copy as staging, so plan
|
|
250
|
+
on about 160 GB.
|
|
251
|
+
- **Coordinator:** it generates the data itself (while the first worker jobs wait in the queue;
|
|
252
|
+
`--gen-on-workers` moves generation to the workers). Prep then runs on the workers. Give it
|
|
253
|
+
8–16 cores and 32 GB, e.g. a `[profiles.<name>.coordinator]` with `cores = 16`,
|
|
254
|
+
`memory = "32GB"`. Most of that is for generation, not prep.
|
|
255
|
+
- **Data folders** are named by DGP, size and non-default parameters, for example
|
|
256
|
+
`raw_panel_1e09_T100_bal1_G500`, so the two presets never collide.
|
|
257
|
+
- **Measured on OSCER at 1e9 rows** (both presets, up to 40 four-core workers, about 36 on
|
|
258
|
+
average; distributed prep):
|
|
259
|
+
|
|
260
|
+
| | 10M units × T=100 | 1M units × T=1000 |
|
|
261
|
+
|---|---|---|
|
|
262
|
+
| generate (coordinator) | 48 s | 337 s |
|
|
263
|
+
| prep by unit / by group | 700 s / 543 s | 524 s / 1300 s |
|
|
264
|
+
| fit `unit + time` | 178 s | 91 s |
|
|
265
|
+
| fit `unit + group^time` | 158 s | 157 s |
|
|
266
|
+
| peak worker memory | 3.4 GB | 2.8 GB |
|
|
267
|
+
| recovers the truth | yes (max \|z\| 1.09, 2.77) | yes (max \|z\| 0.50, 1.36) |
|
|
268
|
+
|
|
269
|
+
End to end: about 27 min for T=100 and 40 min for T=1000. Prep is most of it, and part of
|
|
270
|
+
that is queue wait for the first workers.
|