westata 0.0.1__tar.gz

westata-0.0.1/CHANGELOG.md

@@ -0,0 +1,101 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
+adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) — with
+the caveat that `0.0.x` explicitly carries **no stability promise**.
+
+---
+
+## [Unreleased]
+
+Nothing yet.
+
+---
+
+## [0.0.1] — first release
+
+The first tagged version. Everything below is the initial state of the library,
+not a diff against an earlier release.
+
+### What this library is
+
+`westata` reimplements a set of Stata estimation commands with a
+statsmodels-style API, and checks each one against a real Stata installation.
+The point is not "similar numbers" — it is a measured, reproducible claim about
+which commands agree and how closely.
+
+Verification: **StataNow 19.5 (MP)** via `pystata`. Captures are committed, so
+the comparisons replay without a licence.
+
+### Coverage
+
+| | |
+|---|---|
+| Registry entries | 87 |
+| Tests | 1434 |
+| Independent comparison points | 386 (against live Stata) |
+| Evidence level E3 (machine precision) | 52 |
+| Evidence level E2 (bounded, measured) | 26 |
+| Evidence level E1 (deviates or uncovered) | 9 |
+
+Domains: linear regression and IV, discrete choice and count models, panel
+data, limited-dependent and quantile models, time series, systems of equations
+and causal inference, and post-estimation.
+
+### Known limitations — READ THIS BEFORE RELYING ON A NUMBER
+
+Nine registry entries measurably deviate on at least one invocation. Each is
+listed with its worst measured relative difference and the reason, because a
+number without a reason is not actionable.
+
+| Command | Worst measured rel. diff. | Reason |
+|---|---|---|
+| `xtgls` | 3.4e+00 | **Only when the cross-equation Sigma is rank-deficient** (more equations than periods). No unique generalised inverse exists; Stata warns here too. Well-posed systems are at machine precision. |
+| `teffects nnmatch` | 7.1e-02 | **Standard error only** — the point estimate is exact (3.6e-15). Relies on compiled Mata internally. |
+| `frontier` | 4.3e-02 | `distribution(exponential)` only, where `sigma_u` is nearly unidentified on the test data. The likelihood at Stata's estimates matches exactly. |
+| `bsqreg` | 3.0e-02 | Bootstrap VCE. Coefficients are exact (3.4e-16); a resampling estimator has not converged at any finite replicate count. |
+| `permute` | 1.2e-02 | Stata's shuffle order is not reproducible from its output. |
+| `arima` | 5.5e-03 | Optimiser stopping point; our likelihood is *higher* than Stata's. |
+| `rreg` | 1.0e-03 | Final biweight case weights. |
+| `xtlogit, pa` | 1.1e-03 | GEE working-correlation handling. |
+| `heckman` | 1.9e-05 | Two-step standard errors. The ML path is a separate and exact estimate. |
+
+**One capability is absent rather than approximate:** `teffects nnmatch,
+biasadj()` raises `UnsupportedError`.
+
+Some of these gaps are provably irreducible rather than unfinished:
+
+- `xtgls` with a rank-deficient Sigma has no unique answer to converge to.
+- `teffects nnmatch` and parts of `qreg` call compiled Stata bytecode with no
+  readable source.
+- `arima`, the count family and `frontier` stop where Stata stops; matching
+  Stata's stopping point would mean reporting a *worse* optimum than the one
+  found.
+- A bootstrap VCE cannot match pointwise at any finite replicate count.
+
+### Dependencies
+
+- Required: `numpy`, `pandas`, `scipy`, `statsmodels`, `patsy`, `pyfixest`
+  (HDFE), `linearmodels` (IV and panel), `pyyaml`.
+  The last three are **not optional** — the package does not import without
+  them. An earlier build declared them nowhere and installed cleanly while
+  raising `ModuleNotFoundError` on import.
+- Optional: `arch` for GARCH and the Phillips–Perron test. Install with
+  `pip install "westata[arch]"`. Without it those commands raise
+  `UnsupportedError` naming the install command, and their tests skip rather
+  than fail.
+
+Python 3.10 or newer.
+
+### Notes on this release
+
+- Version `0.0.1` is a placeholder in the SemVer sense: the API may change
+  without notice. It exists so the packaging pipeline can be exercised
+  end to end.
+- The evidence levels are the project's central artefact. They are stored in
+  `tests/conformance/registry.yml`, graded by an independent sweep in `retest/`,
+  and a registry level that the measurement does not support causes the sweep to
+  fail. A false claim is treated as a defect.
+- `retest/` requires a Stata licence and does not run in CI. CI runs the test
+  suite, which replays committed captures and needs no licence.

westata-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 westata contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

westata-0.0.1/MANIFEST.in

@@ -0,0 +1,9 @@
+include CHANGELOG.md
+include LICENSE
+include README.md
+include README.zh-CN.md
+include docs/GUIDE.md
+include docs/GUIDE.zh-CN.md
+include docs/comparison.json
+include docs/data/*.csv
+recursive-include docs *.py

westata-0.0.1/PKG-INFO

@@ -0,0 +1,354 @@
+Metadata-Version: 2.4
+Name: westata
+Version: 0.0.1
+Summary: Stata-conformant econometrics with a statsmodels-style API
+License: MIT License
+        
+        Copyright (c) 2026 westata contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: scipy>=1.10
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: patsy>=0.5
+Requires-Dist: pyfixest>=0.30
+Requires-Dist: linearmodels>=6.0
+Requires-Dist: pyyaml>=6
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Provides-Extra: arch
+Requires-Dist: arch>=6.0; extra == "arch"
+Dynamic: license-file
+
+# westata
+
+Stata-conformant econometrics with a statsmodels-style API.
+
+> **Read this before relying on a number.** `westata` is a personal research
+> library at version 0.0.1. 52 of its 87 commands match Stata to machine
+> precision, but **9 do not**, and one feature is missing. The measured
+> deviation for each is listed below. They are graded, not hidden, and the
+> grading is reproducible from this repository.
+>
+> | Command | Worst measured relative difference | Why |
+> |---|---|---|
+> | `xtgls` | 3.4e+00 | **Only when the cross-equation Sigma is rank-deficient** (more equations than periods). No unique generalised inverse exists; Stata warns here too. On well-posed systems this is machine precision. |
+> | `teffects nnmatch` | 7.1e-02 | **Standard error only** — the point estimate is exact (3.6e-15). Uses compiled Mata internally. |
+> | `frontier` | 4.3e-02 | `distribution(exponential)` only, where `sigma_u` is nearly unidentified on the test data. The likelihood at Stata's estimates matches exactly. |
+> | `bsqreg` | 3.0e-02 | Bootstrap VCE. Coefficients are exact (3.4e-16); a resampling estimator has not converged at any finite replicate count. |
+> | `permute` | 1.2e-02 | Stata's shuffle order for the permutation draws is not reproducible from its output. |
+> | `arima` | 5.5e-03 | Optimiser stopping point; our likelihood is *higher* than Stata's. |
+> | `rreg` | 1.0e-03 | Final biweight case weights. |
+> | `xtlogit, pa` | 1.1e-03 | GEE working-correlation handling. |
+> | `heckman` | 1.9e-05 | Two-step standard errors. The ML path added later is a separate, exact estimate. |
+>
+> One capability is **absent rather than approximate**: `teffects nnmatch,
+> biasadj()` raises `UnsupportedError`.
+>
+> Each figure is the worst case across the invocations an independent
+> 386-point sweep exercised against live Stata. Most of the library is far
+> better than these bounds — 250 of 386 comparison points are at machine
+> precision — but these are the numbers that would bite you, so they come
+> first.
+
+`westata` is a personal research library that answers one question: *if I run
+this in Python, do I get the same numbers Stata would print?*
+
+- **API follows statsmodels** — `Model.from_formula(...).fit()`, `params`,
+  `bse`, `pvalues`, `conf_int()`, `cov_params()`. Results are duck-typed, so
+  existing statsmodels-style downstream code keeps working.
+- **Numerics follow Stata** — small-sample corrections, weight semantics,
+  factor-variable coding, degrees of freedom, and degrees-of-freedom splits are
+  implemented per the corresponding Stata manual entry.
+- **Claims are auditable** — every command carries an evidence level in
+  `tests/conformance/registry.yml`. Nothing claims more than it can show.
+
+## Status
+
+| | |
+|---|---|
+| Tests | **1448 passing** |
+| Commands covered | 87 registry entries |
+| Evidence | **52 × E3**, 26 × E2, 9 × E1 |
+| Independent sweep | 386 comparison points against live Stata (`retest/`) |
+
+Verified against **StataNow 19.5 (MP)** through `pystata`; the captures are
+committed so the comparison replays without a licence.
+
+## Install
+
+`westata` is **not on PyPI yet**. Install it from the repository:
+
+```bash
+git clone https://github.com/bigchill-z/westata.git
+cd westata
+python -m venv .venv
+.venv\Scripts\python.exe -m pip install -e ".[test]"
+```
+
+On Linux or macOS the last line is `./.venv/bin/python -m pip install -e ".[test]"`.
+
+Requires Python ≥ 3.10. `numpy`, `pandas`, `scipy`, `statsmodels`, `patsy`,
+`pyfixest` (HDFE), `linearmodels` (IV, panel) and `pyyaml` come in
+automatically — the package does not import without the last three.
+
+`arch` is optional and only needed for GARCH, EGARCH and the Phillips–Perron
+test. Install it alongside the package:
+
+```bash
+.venv\Scripts\python.exe -m pip install -e ".[test,arch]"
+```
+
+Without it those commands raise an `UnsupportedError` naming the missing extra,
+and their tests skip rather than fail. Once the package is on PyPI, the
+equivalent commands will be `pip install westata` and `pip install
+"westata[arch]"`.
+
+## Quick start
+
+Every line below is executed as written against `tests/data/auto.dta`.
+
+```python
+import pandas as pd
+import westata as ws
+
+df = pd.read_stata("tests/data/auto.dta")
+
+# OLS with a factor variable and clustered standard errors (regress)
+res = ws.OLS.from_formula("mpg ~ weight + C(foreign)", df).fit(
+    cov_type="cluster", cov_kwds={"groups": df["foreign"]},
+)
+print(res.summary(style="stata"))          # Stata layout
+```
+
+Coefficient names keep the formula's spelling, so a factor level appears as
+`C(foreign)[T.Foreign]`. Use that exact string:
+
+```python
+res.test("weight")                                  # single restriction
+res.test("weight", "C(foreign)[T.Foreign]")         # joint test of both
+res.test("weight = 0")                              # same, as an equation
+res.lincom("weight + 1000*C(foreign)[T.Foreign]")   # linear combination
+```
+
+`test` takes one specification per argument; a comma inside a single string is not a separator.
+
+Other models, on the same data:
+
+```python
+# Quantile regression (qreg) — median
+ws.QuantileRegression.from_formula("mpg ~ weight + C(foreign)", df).fit()
+
+# Robust regression (rreg)
+ws.RobustRegression.from_formula("mpg ~ weight", df).fit()
+
+# Systems of equations (sureg): one list entry per equation
+ws.Sureg.from_formula(["price ~ weight + mpg", "mpg ~ weight + foreign"], df).fit()
+ws.Sureg.from_formula(
+    ["price ~ weight + mpg", "mpg ~ weight + foreign"], df
+).fit(cov_type="robust")                     # robust VCE
+
+# Censored regression (tobit). `lower` is a from_formula argument, not a fit one.
+# `rep78` has missing values, which are dropped.
+ws.Tobit.from_formula("rep78 ~ weight + mpg", df, lower=0).fit()
+```
+
+A publication-style table:
+
+```python
+base = ws.OLS.from_formula("mpg ~ weight", df).fit()
+full = ws.OLS.from_formula("mpg ~ weight + C(foreign)", df).fit()
+print(ws.etable([base, full], model_names=["Base", "Full"]))
+```
+
+## Command coverage
+
+| Family | westata | Stata |
+|---|---|---|
+| Linear | `OLS` | `regress` (4 weight types, `i.` factors) |
+| IV | `IV2SLS` / `IVLIML` / `IVGMM` | `ivregress 2sls/liml/gmm` |
+| HDFE | `FEOLS` | `reghdfe` / `areg` |
+| Panel | `PanelOLS` / `RandomEffects` / `BetweenOLS` / `FirstDifferenceOLS` | `xtreg fe/re/be/fd` |
+| Binary | `Logit` / `Probit` | `logit` / `probit` |
+| Count | `Poisson` / `NegativeBinomial` | `poisson` / `nbreg` |
+| Ordered | `OrderedLogit` / `OrderedProbit` | `ologit` / `oprobit` |
+| Censored | `Tobit` (self-implemented) | `tobit` |
+| Time series | `ARIMA` / `VAR` / `VECM` / `GARCH` / `unitroot` | `arima` / `var` / `vec` / `arch` / `dfuller` |
+| Postestimation | `test` / `lincom` / `margins` / `predict` | same names |
+| Reporting | `etable` | `esttab` |
+
+## Weight semantics
+
+Following Stata `[U] 20.24`:
+
+| Type | Point estimate | Variance | Effective N |
+|---|---|---|---|
+| `fweight` | literal replication | conventional | `sum(w)` |
+| `aweight` | normalized to `sum(w) = N` | conventional | `N` |
+| `pweight` | same as aweight | **robust (HC1)** | `N` |
+| `iweight` | not normalized | conventional | `sum(w)` |
+
+`weight_type` is mandatory whenever weights are supplied — westata never
+guesses, because guessing silently changes the VCE.
+
+## Evidence levels
+
+| Level | Meaning | Count |
+|---|---|---|
+| E3 | Agrees with full-precision `e(b)`/`e(V)` from real Stata, on every invocation the independent sweep exercised | 52 |
+| E2 | Approximate, or exact only on a subset of invocations: real but bounded differences | 26 |
+| E1 | Materially differs on at least one measured invocation, or not covered by the sweep | 9 |
+
+**These levels come from the independent 386-point sweep, not from the library's own
+tests.** That distinction matters: the library's tests passed at 1247 while the sweep
+found 24 defects, because a test written from the same understanding as the code cannot
+detect a misunderstanding shared by both.
+
+Measured relative agreement with StataNow 19.5:
+
+| Command | Parameters | Standard errors |
+|---|---|---|
+| `regress` | 5.1e-15 | 6.8e-16 |
+| `regress, vce(robust)` | 5.1e-15 | 4.6e-15 |
+| `regress, cluster` | 5.1e-15 | 7.1e-15 |
+| `regress, i.factor` | 3.7e-14 | 5.1e-16 |
+| `ivregress 2sls` | 7.8e-15 | 7.5e-15 |
+| `ivregress liml` | 3.7e-16 | 3.7e-16 |
+| `xtreg, fe` | 1.8e-16 | 1.9e-16 |
+| `xtreg, re` | 6.8e-16 | 1.0e-15 |
+| `xtreg, be` | 3.6e-16 | 1.7e-16 |
+| `xtreg, fd` | not comparable (command removed) | — |
+| `areg` | 4.6e-16 | 1.8e-16 |
+| `reghdfe` (1 FE) | 4.5e-16 | 3.3e-16 |
+| `reghdfe` (2 FE) | 4.4e-16 | 1.9e-16 |
+| `reghdfe` (2 FE, cluster) | 4.4e-16 | 2.0e-16 |
+| `var` | 2.9e-15 | — |
+| `poisson` | 3.9e-13 | 5.8e-14 |
+| `logit` | ~1e-7 | ~1e-7 |
+| `probit` | 1.5e-7 | — |
+| `tobit` | 8.5e-8 | 6.4e-7 |
+| `margins, dydx` | 1.6e-7 | 5.2e-6 |
+| `ologit` / `oprobit` | ~2e-5 | — |
+| `nbreg` | 1.7e-5 | 9.1e-7 |
+| `arima` | 4.8e-5 | 8.2e-5 |
+| `predict` (OLS xb) | 1.6e-15 | — |
+| `predict` (logit pr) | 3.1e-11 | — |
+| `predict` (tobit e/ystar) | 5.8e-10 | — |
+| `etable` vs `esttab` | 0.0 | — |
+| `vecrank` (all lag orders) | 0.0 (4 dp) | — |
+| `dfuller` | 2.3e-4 | — |
+| `ivregress gmm` | 4.9e-16 | 6.9e-18 |
+| `arch` | 1.4e-04 | z values not comparable |
+
+The linear, panel-linear, multi-equation and complex-survey families agree to
+machine precision (1e-15). **Non-linear maximum-likelihood models are where the
+differences live**, and almost always in the second-order quantities rather than the
+coefficients: an exact point estimate with a standard error that is 5% off still gives a
+wrong z, a wrong p, and a wrong confidence interval.
+
+What was measured wrong, and then fixed:
+
+- `ologit`/`oprobit` `e(V)`: a numpy view alias let `exp()` act on the cut points, and the
+  slope-by-cut block was never transformed — `e(V)` off by 20x, all cut-point z and CI wrong.
+- `nl` covariance: `pinv` truncation made standard errors wrong by **6–7 orders of
+  magnitude**, so every t statistic was inflated.
+- `xtreg, fe vce(robust)` was mapped to HC1; Stata defines it as `vce(cluster panelvar)`
+  (verified: the two are identical in Stata, `max|V_robust − V_cluster| = 0`). The constant's
+  SE was 18x off.
+- `tobit` `se(/sigma)` lacked the delta-method Jacobian; `margins` used a fixed numerical
+  step; `podiscrete` VCEs silently accepted and ignored `vce(robust)`.
+
+What remains approximate is listed per command in
+[`docs/USING_WITH_STATA.md`](docs/USING_WITH_STATA.md), which is organised around the
+question a user actually has: **can I quote this number as Stata's?**
+
+Two commands could not be compared at all because StataNow 19 removed them:
+`xtreg, fd` (r(198)) and `kpss` (r(199)). Their westata counterparts are
+validated by other means.
+
+## Design notes worth knowing
+
+Things that bit during development, documented so they do not bite again:
+
+- **`df_resid` vs `e(df_r)` are different quantities.** Under clustering, t
+  tests use `G − 1` while Stata's `e(df_r)` stays `N − K`. Conflating them
+  silently gives wrong p-values.
+- **Frequency weights must equal the expanded-data fit.** Stata's own stated
+  invariant. Getting the robust meat's weight form wrong inflates fweighted
+  standard errors by up to 51%.
+- **`margins` differentiates the response, not the linear index.** For a logit,
+  `mean(p(1-p))·b` — differentiating `xb` returns the coefficient, which was
+  4.38× too large in the first implementation.
+- **`PanelOLS.from_formula` ignores `entity_effects`.** Passing it through
+  `fit()` silently produces pooled OLS. The model must be constructed directly.
+- **linearmodels' clustered VCE needs `auto_df=False, group_debias=True`** to
+  reproduce Stata's `G/(G-1)·(N-1)/(N-K)`; the default is ~11% off.
+- **`ivregress` reports z / Wald chi2**, not t / F. OLS and panel use t.
+
+## Layout
+
+```
+src/westata/
+  data/            weight normalization, panel/time-series metadata
+  formula/         patsy design matrices, C(x, base=)
+  vce/             HC0-HC3, CR1, ssc
+  estimators/      linear, iv, hdfe, panel, discrete, tobit, tseries
+  results/         StataResults, E(RETURN), Stata/statsmodels summaries
+  postestimation/  test, lincom, margins, predict
+  report/          etable, LaTeX/Markdown renderers
+  conformance/     evidence registry, tolerances, invariants
+tests/conformance/ registry.yml + published-output fixtures
+docs/              design spec and implementation plans
+```
+
+## Documentation
+
+- **Comparing westata with Stata**: [`docs/USING_WITH_STATA.md`](docs/USING_WITH_STATA.md) —
+  which quantities are interchangeable and which are not, for the four E2 commands.
+- Evidence registry: [`tests/conformance/registry.yml`](tests/conformance/registry.yml) —
+  per-command level, measured gap, and reasoning.
+- Known gaps: [`docs/GAP_REGISTER.md`](docs/GAP_REGISTER.md) — what is deliberately
+  not implemented, and why.
+- **Independent sweep**: [retest/README.md](retest/README.md) — the 386-point
+  check against live Stata, and how to re-run it. This is where the evidence
+  levels come from.
+- Design spec: [`docs/superpowers/specs/2026-10-22-westata-design.md`](docs/superpowers/specs/2026-10-22-westata-design.md)
+
+## Known gaps
+
+Not implemented, and they raise instead of approximating: Mata, the graphics
+system, data management, SEM/GSEM, survival analysis, Bayesian estimation, `mi`,
+`xtabond2`, `csdid`, `rdrobust`, `synth`, general `gmm`, `nlsur`, multi-way
+clustering (CGM), `vecrank`'s fixed-N exact critical values, and Stata's
+arbitrary `r()` statistics outside the `[U] exp_list` grammar.
+
+`docs/GAP_REGISTER.md` lists every one with its reason. Those are boundary
+statements, not a to-do list.