dpd-stream 0.1.0__tar.gz

dpd_stream-0.1.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+## 0.1.0
+
+The first version.
+
+- **Estimators.** Arellano–Bond difference GMM and Blundell–Bond system GMM;
+  one-step, two-step or iterated; first differences or forward orthogonal
+  deviations. Where conventions differ, the design follows Stata's xtabond2
+  (3.7.2), the reference for the instrument layout, the generalized inverse
+  (including `invsym()`'s treatment of negative pivots), the weights'
+  normalizations and the tests.
+- **Instruments.** Lag limits, `collapse`, `eq`, `split`, `passthru`, backward
+  orthogonal deviations, IV-style `mz`, `h()` and `pca`, written in the
+  formula (`gmm()`, `iv()`) or passed as `Gmm` / `Iv` objects.
+- **Out of core.** Polars reads the data once, from a Parquet file or glob or
+  a Lazy- or DataFrame, into blocks of whole panels. Three passes over the
+  blocks, in numba, give two-step estimates with Windmeijer's correction and
+  the Arellano–Bond tests; everything else comes from the sums. No instrument
+  matrix is ever built.
+- **Inference.** Windmeijer's (2005) correction; one- and multi-way
+  clustering; `small`; aweights, pweights and fweights; the Arellano–Bond
+  AR(m), Sargan, Hansen and difference-in-Hansen tests; Andrews and Lu's
+  MMSC; delta-method long-run effects (`long_run`).
+- **Identification tests.** Kleibergen–Paap rk LM and Wald and
+  Sanderson–Windmeijer tests (Windmeijer 2025), robust to any clustering,
+  with weights, PCA and absorbed effects; first-stage partial and Shea
+  partial R², the Anderson–Rubin test (and, with one endogenous regressor, its
+  confidence set) and Montiel Olea and Pflueger's effective F. They are
+  ivreg2's forms, not the LIML-based ones of Stata's `underid`, so the digits
+  differ (see the identification page). The algebra is hdfe-stream's
+  `core.ivtests`.
+- **Ahn–Schmidt nonlinear moments** (`nl="noserial"` or `"iid"`), with
+  xtdpdgmm's first step, one-step fits and Windmeijer's correction in its
+  general form. The nonlinear fits' standard errors are the exact ones for the
+  moments and differ by a few percent from xtdpdgmm's (see the nonlinear
+  page).
+- **Absorbed effects.** Time effects (`| year`) and, in difference GMM,
+  period-by-group effects (`| region^year`), with the estimates, variances and
+  tests of their dummies, under first differences or orthogonal deviations,
+  weights, any clustering and iterated GMM.
+- **`DPDResult`.** A hdfe-stream `RegressionResult`: `summary()`, `tidy()`,
+  `confint()`, `wald_test()`, `etable`, `summary_json()`, `with_vcov()`, the
+  one-step results of a two-step fit (`onestep`), and lazy `predict()`,
+  `resid()` and `sample()` over the source.
+- **`dpd_stream.simulate`.** `simulate_dynamic` and `truth_dynamic`: dynamic
+  panels with predetermined, endogenous or exogenous regressors, stationary or
+  not, unbalanced, with gaps, clusters and weights.
+- **Tested against** a dense implementation of the algebra, Roodman's (2009)
+  published output, pydynpd 0.2.2 and plm 2.6.7's `pgmm` (on a simulated panel
+  shipped with the tests; Arellano and Bond's data are not distributed), and
+  Stata's xtabond2 and xtdpdgmm on about fifty specifications, whose bundle
+  is in `benchmarks/external/stata`.

dpd_stream-0.1.0/CITATION.cff

@@ -0,0 +1,20 @@
+cff-version: 1.2.0
+message: "If you use this software in your research, please cite it as below."
+type: software
+title: "dpd-stream"
+abstract: "Out-of-core difference and system GMM for dynamic panels (Arellano-Bond, Blundell-Bond). One-step, two-step or iterated estimation with flexible instrument sets, Windmeijer-corrected and clustered inference, specification and identification tests, Ahn-Schmidt nonlinear moments and absorbed fixed effects, computed in streamed passes so that panel data far larger than memory can be used. Conventions follow Stata's xtabond2; results are tested against it, xtdpdgmm and other implementations."
+authors:
+  - family-names: Tucker
+    given-names: Lee
+    email: lee.c.tucker@census.gov
+version: 0.1.0
+license: MIT
+repository-code: "https://github.com/leetucker/dpd-stream"
+url: "https://github.com/leetucker/dpd-stream"
+keywords:
+  - econometrics
+  - dynamic panel data
+  - GMM
+  - Arellano-Bond
+  - Blundell-Bond
+  - out-of-core

dpd_stream-0.1.0/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lee Tucker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dpd_stream-0.1.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include LICENSE.md README.md CHANGELOG.md CITATION.cff
+include dpd_stream/py.typed

dpd_stream-0.1.0/PKG-INFO

@@ -0,0 +1,155 @@
+Metadata-Version: 2.4
+Name: dpd-stream
+Version: 0.1.0
+Summary: Out-of-core dynamic panel GMM: Arellano-Bond difference GMM and Blundell-Bond system GMM, by streamed passes
+Author: Lee Tucker
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/leetucker/dpd-stream
+Project-URL: Source, https://github.com/leetucker/dpd-stream
+Project-URL: Issues, https://github.com/leetucker/dpd-stream/issues
+Project-URL: Documentation, https://leetucker.github.io/dpd-stream/
+Project-URL: Changelog, https://github.com/leetucker/dpd-stream/blob/main/CHANGELOG.md
+Keywords: econometrics,dynamic panel,GMM,Arellano-Bond,Blundell-Bond,system GMM,xtabond2,out-of-core,panel data
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: hdfe-stream[formula]>=0.6.2
+Requires-Dist: polars!=1.41.*,>=1.37
+Requires-Dist: numpy>=1.24
+Requires-Dist: numba>=0.59
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: scipy>=1.11
+Requires-Dist: threadpoolctl>=3.1
+Provides-Extra: benchmark
+Requires-Dist: matplotlib>=3.7; extra == "benchmark"
+Requires-Dist: pandas>=2.0; extra == "benchmark"
+Provides-Extra: docs
+Requires-Dist: sphinx>=8.0; extra == "docs"
+Requires-Dist: myst-parser>=4.0; extra == "docs"
+Requires-Dist: furo>=2024.8.6; extra == "docs"
+Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
+Requires-Dist: sphinx-design>=0.6; extra == "docs"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: pandas>=2.0; extra == "test"
+Dynamic: license-file
+
+# dpd-stream
+
+Arellano–Bond (difference) and Blundell–Bond (system) GMM for dynamic panels,
+out of core. One-step, two-step or iterated estimation with flexible
+instrument sets, Windmeijer-corrected and clustered inference, specification
+and identification tests, Ahn–Schmidt nonlinear moments and absorbed fixed
+effects. It works on data far larger than memory, and is much faster than the
+existing implementations on large panels. Where conventions differ, it follows
+Stata's xtabond2, and its results are tested against xtabond2, xtdpdgmm and
+other implementations.
+
+```python
+from dpd_stream import dpd_stream
+
+fit = dpd_stream("n ~ l(n, 1) + w + k | year | gmm(n, 2:4) + gmm(w + k, 1:3)",
+                 "firms/*.parquet", panel=("firm", "year"), system=True)
+fit.summary()
+```
+
+## What it does
+
+**Estimators**
+- Difference or system GMM: one-step, two-step (Windmeijer-corrected) or
+  iterated.
+- First differences or forward orthogonal deviations.
+
+**Instruments**
+- Lag limits, `collapse`, `eq`, `split`, `passthru`, `orthogonal`, `mz`,
+  `h()`, `pca`.
+
+**Tests and inference**
+- Arellano–Bond AR(m), Sargan, Hansen, difference-in-Hansen.
+- Clustering, one- and multi-way.
+- `small`-sample statistics.
+- Weights.
+
+**Also**
+- Iterated GMM.
+- MMSC model selection.
+- Long-run effects.
+- Kleibergen–Paap and Sanderson–Windmeijer identification tests (ivreg2's
+  forms; they differ in the digits from Stata's `underid`).
+- Ahn–Schmidt nonlinear moments (xtdpdgmm's `nl()`).
+- Absorbed time effects, and absorbed period-by-group effects in difference
+  GMM.
+
+**Interface**
+- pyfixest-style formulas with fixest's `l()`/`f()`/`d()` lag operators.
+- Results with hdfe-stream's reporting: `summary()`, `tidy()`, `etable`,
+  `wald_test()`, `summary_json()`.
+
+## Why it is fast
+
+Everything linear GMM needs is a sum over panels: Z′X, Z′y, Σ Zᵢ′HZᵢ, and
+Σ gᵢgᵢ′ with gᵢ = Zᵢ′eᵢ.
+
+- **One read of the source.** The data are read once into blocks of whole
+  panels, sorted by time.
+- **No instrument matrix.** Each pass is a numba kernel that builds every
+  panel's rows and its sparse instruments and adds them into those sums. No
+  instrument matrix is ever built.
+- **Three passes in all** give two-step estimates with Windmeijer's
+  correction and the Arellano–Bond tests.
+
+The other implementations instead hold every panel's dense instrument
+matrix: xtabond2, xtdpdgmm, plm, pdynmc and pydynpd.
+
+## Performance
+
+Two-step GMM with Windmeijer-corrected standard errors on simulated panels
+of 8 periods, on an EC2 spot instance (16 vCPUs, 32 GiB):
+
+| Panels (rows) | | dpd-stream | pydynpd | plm `pgmm` | pdynmc |
+|---|---|---|---|---|---|
+| 100,000 (800,000) | difference GMM | 0.6 s, 1.3 GB | 15 s, 1.0 GB | 72 s, 5.2 GB | > 1,800 s |
+| 1,000,000 (8 million) | difference GMM | 5.7 s, 1.6 GB | 158 s, 7.7 GB | out of memory | — |
+| 1,000,000 (8 million) | system GMM | 5.3 s, 1.6 GB | 159 s, 8.2 GB | 799 s, 26 GB | — |
+| 10,000,000 (80 million) | system GMM | 80 s, 9.7 GB | out of memory | out of memory | — |
+
+Estimates agree to about 1e-13 (system GMM: with pydynpd; plm and pdynmc enter
+the year effects differently). With uncollapsed instruments over 20 periods
+(5,000 panels), dpd-stream takes 0.24 s; pydynpd, plm and pdynmc take 24, 47
+and 441 s, and at 30 periods plm and pdynmc run out of memory. systemgmmkit,
+which builds collapsed instruments only, takes 500 s at 100,000 panels. See the
+benchmarks in the documentation.
+
+## Install
+
+```bash
+pip install dpd-stream
+```
+
+## Documentation
+
+The documentation is at https://leetucker.github.io/dpd-stream/. Build it
+locally with:
+
+```bash
+pip install -e ".[docs]"
+sphinx-build -W --keep-going -b html docs docs/_build/html
+```
+
+`examples/` has runnable scripts, and `benchmarks/` the performance
+comparisons (see `benchmarks/README.md`).
+
+## License
+
+MIT.

dpd_stream-0.1.0/README.md

@@ -0,0 +1,108 @@
+# dpd-stream
+
+Arellano–Bond (difference) and Blundell–Bond (system) GMM for dynamic panels,
+out of core. One-step, two-step or iterated estimation with flexible
+instrument sets, Windmeijer-corrected and clustered inference, specification
+and identification tests, Ahn–Schmidt nonlinear moments and absorbed fixed
+effects. It works on data far larger than memory, and is much faster than the
+existing implementations on large panels. Where conventions differ, it follows
+Stata's xtabond2, and its results are tested against xtabond2, xtdpdgmm and
+other implementations.
+
+```python
+from dpd_stream import dpd_stream
+
+fit = dpd_stream("n ~ l(n, 1) + w + k | year | gmm(n, 2:4) + gmm(w + k, 1:3)",
+                 "firms/*.parquet", panel=("firm", "year"), system=True)
+fit.summary()
+```
+
+## What it does
+
+**Estimators**
+- Difference or system GMM: one-step, two-step (Windmeijer-corrected) or
+  iterated.
+- First differences or forward orthogonal deviations.
+
+**Instruments**
+- Lag limits, `collapse`, `eq`, `split`, `passthru`, `orthogonal`, `mz`,
+  `h()`, `pca`.
+
+**Tests and inference**
+- Arellano–Bond AR(m), Sargan, Hansen, difference-in-Hansen.
+- Clustering, one- and multi-way.
+- `small`-sample statistics.
+- Weights.
+
+**Also**
+- Iterated GMM.
+- MMSC model selection.
+- Long-run effects.
+- Kleibergen–Paap and Sanderson–Windmeijer identification tests (ivreg2's
+  forms; they differ in the digits from Stata's `underid`).
+- Ahn–Schmidt nonlinear moments (xtdpdgmm's `nl()`).
+- Absorbed time effects, and absorbed period-by-group effects in difference
+  GMM.
+
+**Interface**
+- pyfixest-style formulas with fixest's `l()`/`f()`/`d()` lag operators.
+- Results with hdfe-stream's reporting: `summary()`, `tidy()`, `etable`,
+  `wald_test()`, `summary_json()`.
+
+## Why it is fast
+
+Everything linear GMM needs is a sum over panels: Z′X, Z′y, Σ Zᵢ′HZᵢ, and
+Σ gᵢgᵢ′ with gᵢ = Zᵢ′eᵢ.
+
+- **One read of the source.** The data are read once into blocks of whole
+  panels, sorted by time.
+- **No instrument matrix.** Each pass is a numba kernel that builds every
+  panel's rows and its sparse instruments and adds them into those sums. No
+  instrument matrix is ever built.
+- **Three passes in all** give two-step estimates with Windmeijer's
+  correction and the Arellano–Bond tests.
+
+The other implementations instead hold every panel's dense instrument
+matrix: xtabond2, xtdpdgmm, plm, pdynmc and pydynpd.
+
+## Performance
+
+Two-step GMM with Windmeijer-corrected standard errors on simulated panels
+of 8 periods, on an EC2 spot instance (16 vCPUs, 32 GiB):
+
+| Panels (rows) | | dpd-stream | pydynpd | plm `pgmm` | pdynmc |
+|---|---|---|---|---|---|
+| 100,000 (800,000) | difference GMM | 0.6 s, 1.3 GB | 15 s, 1.0 GB | 72 s, 5.2 GB | > 1,800 s |
+| 1,000,000 (8 million) | difference GMM | 5.7 s, 1.6 GB | 158 s, 7.7 GB | out of memory | — |
+| 1,000,000 (8 million) | system GMM | 5.3 s, 1.6 GB | 159 s, 8.2 GB | 799 s, 26 GB | — |
+| 10,000,000 (80 million) | system GMM | 80 s, 9.7 GB | out of memory | out of memory | — |
+
+Estimates agree to about 1e-13 (system GMM: with pydynpd; plm and pdynmc enter
+the year effects differently). With uncollapsed instruments over 20 periods
+(5,000 panels), dpd-stream takes 0.24 s; pydynpd, plm and pdynmc take 24, 47
+and 441 s, and at 30 periods plm and pdynmc run out of memory. systemgmmkit,
+which builds collapsed instruments only, takes 500 s at 100,000 panels. See the
+benchmarks in the documentation.
+
+## Install
+
+```bash
+pip install dpd-stream
+```
+
+## Documentation
+
+The documentation is at https://leetucker.github.io/dpd-stream/. Build it
+locally with:
+
+```bash
+pip install -e ".[docs]"
+sphinx-build -W --keep-going -b html docs docs/_build/html
+```
+
+`examples/` has runnable scripts, and `benchmarks/` the performance
+comparisons (see `benchmarks/README.md`).
+
+## License
+
+MIT.

dpd_stream-0.1.0/dpd_stream/__init__.py

@@ -0,0 +1,56 @@
+"""dpd-stream: Arellano-Bond (difference) and Blundell-Bond (system) GMM
+for dynamic panels, out of core. Names and conventions follow Stata's
+xtabond2 where they overlap.
+
+The data are read once (pass 0) into blocks of whole panels sorted by
+time; every later pass reads the blocks, builds each panel's rows and
+instruments in a numba kernel, and accumulates the few matrices the
+estimator needs (j x j and j x K, j the number of instruments). Nothing
+the size of the data is held in memory, and no instrument matrix is
+built.
+
+Layout
+    api.py          dpd_stream()
+    _frontend.py    formula and instrument parsing; Gmm, Iv
+    layout.py       the instrument matrix's columns (xtabond2's order)
+    store.py        pass 0: the source -> panel-sorted blocks
+    kernels/        numba: a panel's rows and instruments; the passes
+    estimator.py    the fit: the passes and the estimates
+    _passes.py, _absorb.py, _idtests.py, _nl.py
+                    parts of the fit (mixins of estimator.DPD): the residual
+                    and Windmeijer passes; absorbed effects; identification
+                    tests; Ahn-Schmidt moments
+    inference.py    the algebra after the passes: generalized inverse,
+                    Windmeijer, Arellano-Bond, Sargan/Hansen, MMSC
+    results.py      DPDResult, DPDMulti
+    simulate.py     simulate_dynamic(), truth_dynamic()
+"""
+
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _version
+
+from hdfe_stream.core.multcomp import bonferroni, holm
+from hdfe_stream.core.reporting import etable
+from hdfe_stream.core.workspace import cleanup
+
+from ._frontend import Gmm, Iv
+from .api import dpd_stream
+from .results import DPDMulti, DPDResult
+
+try:
+    __version__ = _version("dpd-stream")
+except PackageNotFoundError:  # pragma: no cover
+    __version__ = "0.1.0"
+
+__all__ = [
+    "DPDMulti",
+    "DPDResult",
+    "Gmm",
+    "Iv",
+    "__version__",
+    "bonferroni",
+    "cleanup",
+    "dpd_stream",
+    "etable",
+    "holm",
+]