weightpipe 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- weightpipe-0.1.0/LICENSE +21 -0
- weightpipe-0.1.0/PKG-INFO +420 -0
- weightpipe-0.1.0/README.md +366 -0
- weightpipe-0.1.0/pyproject.toml +161 -0
- weightpipe-0.1.0/setup.cfg +4 -0
- weightpipe-0.1.0/tests/test_balance.py +103 -0
- weightpipe-0.1.0/tests/test_bootstrap.py +91 -0
- weightpipe-0.1.0/tests/test_cascade_parity.py +213 -0
- weightpipe-0.1.0/tests/test_cluster_bounds_ridge_trim.py +221 -0
- weightpipe-0.1.0/tests/test_design_estimate.py +172 -0
- weightpipe-0.1.0/tests/test_estimate_domains.py +93 -0
- weightpipe-0.1.0/tests/test_glm.py +116 -0
- weightpipe-0.1.0/tests/test_gold_csv.py +48 -0
- weightpipe-0.1.0/tests/test_gold_r_packages.py +425 -0
- weightpipe-0.1.0/tests/test_gold_sampler.py +139 -0
- weightpipe-0.1.0/tests/test_gold_svy.py +92 -0
- weightpipe-0.1.0/tests/test_linearization.py +161 -0
- weightpipe-0.1.0/tests/test_margins.py +90 -0
- weightpipe-0.1.0/tests/test_ml_calibrate.py +78 -0
- weightpipe-0.1.0/tests/test_package_smoke.py +32 -0
- weightpipe-0.1.0/tests/test_pipeline.py +113 -0
- weightpipe-0.1.0/tests/test_planning.py +98 -0
- weightpipe-0.1.0/tests/test_propensity_engines.py +242 -0
- weightpipe-0.1.0/tests/test_raking_recovery.py +69 -0
- weightpipe-0.1.0/tests/test_ratio_median.py +58 -0
- weightpipe-0.1.0/tests/test_steps_analytical.py +156 -0
- weightpipe-0.1.0/weightpipe/__init__.py +83 -0
- weightpipe-0.1.0/weightpipe/_logging.py +50 -0
- weightpipe-0.1.0/weightpipe/design.py +233 -0
- weightpipe-0.1.0/weightpipe/diagnostics/__init__.py +52 -0
- weightpipe-0.1.0/weightpipe/diagnostics/balance.py +354 -0
- weightpipe-0.1.0/weightpipe/diagnostics/margins.py +267 -0
- weightpipe-0.1.0/weightpipe/estimands.py +85 -0
- weightpipe-0.1.0/weightpipe/estimate.py +523 -0
- weightpipe-0.1.0/weightpipe/frame.py +126 -0
- weightpipe-0.1.0/weightpipe/methods/__init__.py +55 -0
- weightpipe-0.1.0/weightpipe/methods/calibrate_assist.py +184 -0
- weightpipe-0.1.0/weightpipe/methods/cluster_utils.py +52 -0
- weightpipe-0.1.0/weightpipe/methods/design_matrix.py +88 -0
- weightpipe-0.1.0/weightpipe/methods/eligibility.py +155 -0
- weightpipe-0.1.0/weightpipe/methods/glm.py +226 -0
- weightpipe-0.1.0/weightpipe/methods/linear.py +259 -0
- weightpipe-0.1.0/weightpipe/methods/ml_calibrate.py +235 -0
- weightpipe-0.1.0/weightpipe/methods/nonresponse.py +459 -0
- weightpipe-0.1.0/weightpipe/methods/poststrat.py +100 -0
- weightpipe-0.1.0/weightpipe/methods/raking.py +222 -0
- weightpipe-0.1.0/weightpipe/methods/trim.py +323 -0
- weightpipe-0.1.0/weightpipe/pipeline.py +230 -0
- weightpipe-0.1.0/weightpipe/planning.py +238 -0
- weightpipe-0.1.0/weightpipe/recipe.py +376 -0
- weightpipe-0.1.0/weightpipe/replicates/__init__.py +452 -0
- weightpipe-0.1.0/weightpipe/replicates/linearization.py +188 -0
- weightpipe-0.1.0/weightpipe/result.py +63 -0
- weightpipe-0.1.0/weightpipe/steps/__init__.py +449 -0
- weightpipe-0.1.0/weightpipe/steps/base.py +55 -0
- weightpipe-0.1.0/weightpipe.egg-info/PKG-INFO +420 -0
- weightpipe-0.1.0/weightpipe.egg-info/SOURCES.txt +58 -0
- weightpipe-0.1.0/weightpipe.egg-info/dependency_links.txt +1 -0
- weightpipe-0.1.0/weightpipe.egg-info/requires.txt +15 -0
- weightpipe-0.1.0/weightpipe.egg-info/top_level.txt +1 -0
weightpipe-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sebastian Daza
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,420 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: weightpipe
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Declarative survey weighting recipes with recipe-aware replicate weights
|
|
5
|
+
Author-email: Sebastian Daza <sebastian.daza@gmail.com>
|
|
6
|
+
License: MIT License
|
|
7
|
+
|
|
8
|
+
Copyright (c) 2026 Sebastian Daza
|
|
9
|
+
|
|
10
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
11
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
12
|
+
in the Software without restriction, including without limitation the rights
|
|
13
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
14
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
15
|
+
furnished to do so, subject to the following conditions:
|
|
16
|
+
|
|
17
|
+
The above copyright notice and this permission notice shall be included in all
|
|
18
|
+
copies or substantial portions of the Software.
|
|
19
|
+
|
|
20
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
21
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
22
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
23
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
24
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
25
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
26
|
+
SOFTWARE.
|
|
27
|
+
|
|
28
|
+
Project-URL: Homepage, https://github.com/sdaza/weightpipe
|
|
29
|
+
Project-URL: Repository, https://github.com/sdaza/weightpipe
|
|
30
|
+
Project-URL: Issues, https://github.com/sdaza/weightpipe/issues
|
|
31
|
+
Project-URL: Changelog, https://github.com/sdaza/weightpipe/blob/main/CHANGELOG.md
|
|
32
|
+
Classifier: Programming Language :: Python :: 3
|
|
33
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
34
|
+
Classifier: Operating System :: OS Independent
|
|
35
|
+
Classifier: Intended Audience :: Science/Research
|
|
36
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
37
|
+
Classifier: Development Status :: 3 - Alpha
|
|
38
|
+
Requires-Python: >=3.11
|
|
39
|
+
Description-Content-Type: text/markdown
|
|
40
|
+
License-File: LICENSE
|
|
41
|
+
Requires-Dist: numpy>=1.26.0
|
|
42
|
+
Requires-Dist: pandas>=2.2.0
|
|
43
|
+
Requires-Dist: pyyaml>=6.0.2
|
|
44
|
+
Requires-Dist: scipy>=1.11.0
|
|
45
|
+
Requires-Dist: scikit-learn>=1.5.2
|
|
46
|
+
Provides-Extra: plot
|
|
47
|
+
Requires-Dist: matplotlib>=3.8.0; extra == "plot"
|
|
48
|
+
Requires-Dist: seaborn>=0.13.2; extra == "plot"
|
|
49
|
+
Provides-Extra: r-gold
|
|
50
|
+
Requires-Dist: rpy2>=3.5.16; extra == "r-gold"
|
|
51
|
+
Provides-Extra: gold
|
|
52
|
+
Requires-Dist: svy>=0.25.0; extra == "gold"
|
|
53
|
+
Dynamic: license-file
|
|
54
|
+
|
|
55
|
+
# weightpipe
|
|
56
|
+
|
|
57
|
+
[](https://github.com/sdaza/weightpipe/actions/workflows/ci.yml)
|
|
58
|
+
[](https://www.python.org/downloads/)
|
|
59
|
+
[](https://opensource.org/licenses/MIT)
|
|
60
|
+
|
|
61
|
+
Survey weighting in Python: define a sampling design, apply a weighting recipe, then estimate means, totals, proportions, ratios, medians, and design-based GLMs with bootstrap, jackknife, or linearized standard errors.
|
|
62
|
+
|
|
63
|
+
The point is one API for the whole path — design, eligibility, nonresponse, calibration, trim, estimates, and sample-size planning — instead of stitching specialized raking, weighting, and variance packages together.
|
|
64
|
+
|
|
65
|
+
## Install
|
|
66
|
+
|
|
67
|
+
Install from GitHub (latest `main`):
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pip install "git+https://github.com/sdaza/weightpipe.git"
|
|
71
|
+
# or
|
|
72
|
+
uv add "git+https://github.com/sdaza/weightpipe.git"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Requires Python 3.11+. Not published on PyPI yet.
|
|
76
|
+
|
|
77
|
+
## Quickstart
|
|
78
|
+
|
|
79
|
+
`WeightPipe` is the single entry point: describe the sample, chain any adjustments, then read weights or estimates. Weights are computed on first use, so estimating with no adjustment steps works too.
|
|
80
|
+
|
|
81
|
+
```python
|
|
82
|
+
from weightpipe import WeightPipe, population_totals
|
|
83
|
+
|
|
84
|
+
totals = population_totals(pop, "~ region + sex + age")
|
|
85
|
+
|
|
86
|
+
pipe = (
|
|
87
|
+
WeightPipe(df, weight="pw", psu="psu", strata="stratum")
|
|
88
|
+
.unknown_eligibility(unknown="unknown", by=["region"], cluster="hh")
|
|
89
|
+
.drop_ineligible(ineligible="ineligible")
|
|
90
|
+
.nonresponse(
|
|
91
|
+
respondent="responded",
|
|
92
|
+
method="propensity",
|
|
93
|
+
engine="logit", # or "gbm" / "forest"
|
|
94
|
+
formula="~ region + sex",
|
|
95
|
+
num_classes=5,
|
|
96
|
+
cluster="hh",
|
|
97
|
+
)
|
|
98
|
+
.calibrate(
|
|
99
|
+
method="linear",
|
|
100
|
+
formula="~ region + sex + age",
|
|
101
|
+
totals=totals,
|
|
102
|
+
# engine="forest", population=pop, # tree-embedding GREG (needs population microdata)
|
|
103
|
+
# assist="propensity", # add p̂ to linear calibration
|
|
104
|
+
# assist="propensity_class", # rake/calibrate while keeping class mass
|
|
105
|
+
)
|
|
106
|
+
.trim(max_ratio=4.0, reference="median", redistribute=True)
|
|
107
|
+
)
|
|
108
|
+
|
|
109
|
+
pipe.weights # final weights
|
|
110
|
+
pipe.collect_weights(keep_intermediate=True) # weights and per-step factors
|
|
111
|
+
pipe.diagnostics # per-step diagnostics and alerts
|
|
112
|
+
|
|
113
|
+
pipe.estimate.mean(["income", "food_share"], by="urban_rural")
|
|
114
|
+
pipe.estimate.ratio(["food", "miles"], ["income", "trips"], by="urban_rural")
|
|
115
|
+
pipe.estimate.median("y", variance="bootstrap", replicates=200, seed=1)
|
|
116
|
+
pipe.estimate.glm("employed ~ region + age", family="binomial", variance="linearization")
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Design weights alone are enough to estimate:
|
|
120
|
+
|
|
121
|
+
```python
|
|
122
|
+
WeightPipe(df, N=10_000).estimate.mean("y", variance="bootstrap", seed=1)
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Each step returns a new pipe, so you can branch from a common base and compare.
|
|
126
|
+
|
|
127
|
+
## Sampling designs
|
|
128
|
+
|
|
129
|
+
Pass the sampling inputs; the design type is inferred automatically.
|
|
130
|
+
|
|
131
|
+
```python
|
|
132
|
+
WeightPipe(df) # no weight → base_weight=1 (logged)
|
|
133
|
+
WeightPipe(df, N=10_000) # SRS: w = N / n
|
|
134
|
+
WeightPipe(df, strata="region", N_h={"North": 5000, ...}) # stratified SRS: w = N_h / n_h
|
|
135
|
+
WeightPipe(df, weight="pw", psu="psu") # cluster
|
|
136
|
+
WeightPipe(df, weight="pw", psu="psu", strata="stratum") # stratified cluster
|
|
137
|
+
WeightPipe(df, probabilities=["p1", "p2"], psu="psu") # multi-stage: w = 1 / (p1 * p2)
|
|
138
|
+
WeightPipe(df, stage_weights=["w1", "w2"], psu="psu") # multi-stage: w = w1 * w2
|
|
139
|
+
WeightPipe(df, weight="pw") # existing design weights
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Check what was inferred with `pipe.kind`.
|
|
143
|
+
|
|
144
|
+
### Multi-stage clusters (one `psu`)
|
|
145
|
+
|
|
146
|
+
You can **draw** several nested stages in the field. You still name **one** `psu` for variance: the first-stage unit (the ultimate cluster). Later stages go into the weight, not into extra PSU columns.
|
|
147
|
+
|
|
148
|
+
Example: sample **schools**, then **classes** inside those schools, then **students** inside those classes. That is a three-stage cluster sample.
|
|
149
|
+
|
|
150
|
+
| Stage | Unit | What it is for |
|
|
151
|
+
|-------|------|----------------|
|
|
152
|
+
| 1 | School | PSU — independent draws; use this in `psu=` |
|
|
153
|
+
| 2 | Class | Nested in school — inclusion goes in the weight |
|
|
154
|
+
| 3 | Student | Row in the data — inclusion goes in the weight |
|
|
155
|
+
|
|
156
|
+
```python
|
|
157
|
+
# w = 1 / (p_school * p_class * p_student); SEs resample schools
|
|
158
|
+
WeightPipe(
|
|
159
|
+
df,
|
|
160
|
+
probabilities=["p_school", "p_class", "p_student"],
|
|
161
|
+
psu="school",
|
|
162
|
+
# strata="district", # if schools were drawn within strata
|
|
163
|
+
)
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
If you already have per-stage weights (`w_k = 1/π_k`), use `stage_weights=["w_school", "w_class", "w_student"]` with the same `psu="school"`. If the product is already in one column, `WeightPipe(df, weight="pw", psu="school")` is enough for variance (`kind` is then `"cluster"` rather than `"multistage"`).
|
|
167
|
+
|
|
168
|
+
**Why `psu` is school, not class.** Classes are selected independently *inside a sampled school*, not from a national list of all classes. A class can appear only if its school was selected first; two classes in the same school share that school draw. Bootstrap and jackknife therefore resample **whole schools** (every selected class and student in that school moves together). Setting `psu` to class would treat classes as independent first-stage units and understate SEs; setting it to student would treat the sample like an SRS of students.
|
|
169
|
+
|
|
170
|
+
`strata=` is separate (for example district). That is the groups *within which* schools were drawn, not a second PSU.
|
|
171
|
+
|
|
172
|
+
If you take every student in the selected classes, stage 3 has π = 1 and the sample is two-stage (schools → classes). `psu` is still `"school"`.
|
|
173
|
+
|
|
174
|
+
## Weighting steps
|
|
175
|
+
|
|
176
|
+
Chain adjustments on a pipe, in order:
|
|
177
|
+
|
|
178
|
+
| Step | What it does |
|
|
179
|
+
|------|----------------|
|
|
180
|
+
| `unknown_eligibility` | Redistribute unknown eligibility within cells; optional household `cluster=` |
|
|
181
|
+
| `drop_ineligible` | Set ineligible units to weight 0 |
|
|
182
|
+
| `nonresponse` | Weighting-class or propensity (`engine="logit"`, `"gbm"`, or `"forest"`); optional `cluster=` |
|
|
183
|
+
| `calibrate` | Raking, post-stratification, or linear/GREG; `engine="forest"`/`"gbm"` for tree-embedding GREG (`population=` required); optional `assist=` (`propensity_class` auto-converts `proportions=` → absolute `margins` before attaching class totals) |
|
|
184
|
+
| `trim` | Cap extreme weights by ratio to median/base/value |
|
|
185
|
+
| `trim_weights` | Automatic Tukey or Potter trimming |
|
|
186
|
+
|
|
187
|
+
After calibrate, step diagnostics include a tidy `margin_table` (target vs achieved). You can also check weighted category margins anytime:
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
pipe.margins("sex") # current weighted totals / proportions
|
|
191
|
+
pipe.margins(targets="calibrate") # reuse last calibrate targets
|
|
192
|
+
pipe.margins("sex", proportions={"sex": {"M": 0.5, "F": 0.5}})
|
|
193
|
+
pipe.diagnostics["steps"]["calibrate"]["margin_table"]
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Covariate **balance** compares base vs final weights to population targets via standardized mean differences (SMD), for continuous and categorical covariates:
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
report = pipe.balance(
|
|
200
|
+
["age", "sex", "region"],
|
|
201
|
+
means={"age": 40.0},
|
|
202
|
+
proportions={"sex": {"M": 0.5, "F": 0.5}, "region": {"N": 0.5, "S": 0.5}},
|
|
203
|
+
)
|
|
204
|
+
report.table # before / after / target / smd_* / balanced
|
|
205
|
+
report.summary # max |SMD|, n_imbalanced, ESS before/after
|
|
206
|
+
# or: pipe.balance(["age", "sex"], target=pop_df, target_weight="N")
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
## Estimation
|
|
210
|
+
|
|
211
|
+
Pass one variable or a list. `by=` splits into domains (same idea as R `svyby` or svy's `sample.estimation.mean(..., by=...)`). Replicate weights are built once and reused.
|
|
212
|
+
|
|
213
|
+
```python
|
|
214
|
+
pipe.estimate.mean(["income", "food_share"], by="urban_rural")
|
|
215
|
+
pipe.estimate.mean("y", variance="jackknife")
|
|
216
|
+
pipe.estimate.proportion("employed", by="region", variance="bootstrap", replicates=200, seed=1)
|
|
217
|
+
pipe.estimate.total("y", variance="linearization")
|
|
218
|
+
pipe.estimate.ratio("y", "x", variance="jackknife")
|
|
219
|
+
pipe.estimate.ratio(["food", "housing"], "income") # one denominator
|
|
220
|
+
pipe.estimate.ratio(["food", "miles"], ["income", "trips"], by="urban_rural") # paired
|
|
221
|
+
pipe.estimate.median("y", variance="jackknife")
|
|
222
|
+
|
|
223
|
+
# still valid
|
|
224
|
+
pipe.estimate("y", estimand="mean", variance="jackknife")
|
|
225
|
+
pipe.estimation.mean("y") # alias of estimate
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
The result is one row per variable × domain, with `estimate`, `se`, `cv`, and a confidence interval.
|
|
229
|
+
|
|
230
|
+
Supported estimands: `mean`, `total`, `proportion`, `ratio`, `median`.
|
|
231
|
+
Variance: `bootstrap` (Rao–Wu; default) and `jackknife` (delete-a-PSU) re-run the recipe on each replicate so SEs include estimated weights. `linearization` is the fast ultimate-cluster Taylor SE that treats the fitted weights as fixed (survey-style). Median has no linearized SE.
|
|
232
|
+
|
|
233
|
+
### Design-based GLM
|
|
234
|
+
|
|
235
|
+
`pipe.estimate.glm` is `survey::svyglm`-style regression: survey-weighted IRLS for the coefficients, then design-based SEs. It is not a `statsmodels` / sklearn wrapper with weights passed through.
|
|
236
|
+
|
|
237
|
+
```python
|
|
238
|
+
pipe.estimate.glm("employed ~ region + age", family="binomial", variance="linearization")
|
|
239
|
+
pipe.estimate.glm("y ~ region", family="gaussian", variance="jackknife")
|
|
240
|
+
pipe.estimate.glm("count ~ age", family="poisson")
|
|
241
|
+
pipe.estimate.glm("y ~ 1", family="gaussian", variance="linearization") # intercept = Hájek mean
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
| `family=` | Link | Outcome |
|
|
245
|
+
|-----------|------|---------|
|
|
246
|
+
| `gaussian` (`normal`) | identity | continuous |
|
|
247
|
+
| `binomial` (`logit`, `logistic`, `quasibinomial`) | logit | 0/1 |
|
|
248
|
+
| `poisson` (`log`, `quasipoisson`) | log | counts |
|
|
249
|
+
|
|
250
|
+
The result is one row per coefficient (`term`), with the same `estimate` / `se` / `cv` / CI columns as other estimands. Categorical predictors use sorted levels and drop the first (R treatment contrasts with alphabetical levels). An intercept-only gaussian matches `pipe.estimate.mean`; an intercept-only binomial is `logit` of the weighted proportion, not `svymean`.
|
|
251
|
+
|
|
252
|
+
`variance=` is the same as for means: `linearization` is the Binder sandwich (ultimate-cluster, weights fixed); `bootstrap` / `jackknife` re-fit β on recipe-aware replicate weights. Gold matches `survey::svyglm`. There is no `by=` on GLM, and no tabs / Rao–Scott.
|
|
253
|
+
|
|
254
|
+
## Sample-size planning
|
|
255
|
+
|
|
256
|
+
Plan sample sizes before fieldwork, then build a pipe after data collection:
|
|
257
|
+
|
|
258
|
+
```python
|
|
259
|
+
from weightpipe import (
|
|
260
|
+
WeightPipe,
|
|
261
|
+
allocate_strata,
|
|
262
|
+
allocation_table,
|
|
263
|
+
margin_of_error,
|
|
264
|
+
sample_size,
|
|
265
|
+
stratified_margin_of_error,
|
|
266
|
+
)
|
|
267
|
+
|
|
268
|
+
sample_size(0.05) # n for ±5 percentage points at 95% confidence
|
|
269
|
+
margin_of_error(384)
|
|
270
|
+
|
|
271
|
+
populations = {"North": 5_000, "South": 15_000}
|
|
272
|
+
plan = allocation_table(populations, sample=400, method="mixed")
|
|
273
|
+
stratified_margin_of_error(plan["sample"], population=plan["population"])
|
|
274
|
+
|
|
275
|
+
# After drawing the planned cases:
|
|
276
|
+
pipe = WeightPipe(df, strata="region", N_h=populations)
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
Allocation methods: `mixed`, `root`, `neyman`, `stdev`, and `error`.
|
|
280
|
+
|
|
281
|
+
## Logging
|
|
282
|
+
|
|
283
|
+
weightpipe stays silent unless you ask for messages. Turn them on to see notes such as a missing design weight:
|
|
284
|
+
|
|
285
|
+
```python
|
|
286
|
+
import weightpipe as wp
|
|
287
|
+
|
|
288
|
+
wp.setup_logging("INFO") # 2026-08-17 12:03:13 No design weight provided; ...
|
|
289
|
+
wp.set_log_level("WARNING") # quieter
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
Messages go to `stderr` through the `weightpipe` logger, so your own logging config keeps working if you'd rather configure it yourself.
|
|
293
|
+
|
|
294
|
+
## Lower-level API
|
|
295
|
+
|
|
296
|
+
`WeightPipe` wraps two objects you can also use directly: `Design` (sampling inputs and base weights) and `Recipe` (the adjustment steps, run with `prep()`). Reach for them when you want to hold a design or an unfitted recipe on its own, for example to build replicate weights by hand.
|
|
297
|
+
|
|
298
|
+
```python
|
|
299
|
+
from weightpipe import Design, Recipe, estimate, estimate_glm
|
|
300
|
+
|
|
301
|
+
design = Design(df, weight="pw", psu="psu", strata="stratum")
|
|
302
|
+
recipe = Recipe.from_design(design).step_calibrate(method="raking", proportions=props)
|
|
303
|
+
fitted = recipe.prep()
|
|
304
|
+
estimate(recipe, ["y", "x"], by="region", fitted=fitted, variance="jackknife")
|
|
305
|
+
estimate_glm(recipe, "employed ~ region", family="binomial", fitted=fitted, variance="linearization")
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
## Examples
|
|
309
|
+
|
|
310
|
+
Interactive scripts (run cell-by-cell or with Python):
|
|
311
|
+
|
|
312
|
+
- [`examples/01_minimal_recipe.py`](examples/01_minimal_recipe.py) — design weights and bootstrap estimates
|
|
313
|
+
- [`examples/02_nonresponse_raking.py`](examples/02_nonresponse_raking.py) — NR + raking
|
|
314
|
+
- [`examples/03_designs_estimate.py`](examples/03_designs_estimate.py) — SRS / stratified / cluster / multi-stage; `estimate.mean(..., by=)`; design-based GLM
|
|
315
|
+
- [`examples/04_cascade_parity.py`](examples/04_cascade_parity.py) — full cascade + jackknife
|
|
316
|
+
- [`examples/05_balance.py`](examples/05_balance.py) — covariate balance (SMD before/after)
|
|
317
|
+
|
|
318
|
+
## Comparisons
|
|
319
|
+
|
|
320
|
+
weightpipe is meant to **integrate** the steps you usually assemble from several tools, and to make that cascade **easier** in Python: one `WeightPipe`, in order, with diagnostics and estimates at the end.
|
|
321
|
+
|
|
322
|
+
Specialized packages still do one slice well.
|
|
323
|
+
|
|
324
|
+
**Raking / IPF only.** [Weightipy](https://pypi.org/project/weightipy/0.4.2/) 0.4.2 is a fast RIM engine (dicts or census tables, nested/segmented RIM, Kish efficiency). [ipfn](https://pypi.org/project/ipfn/) is a small Python IPF helper. R [`anesrake`](https://CRAN.R-project.org/package=anesrake) is ANES-style raking. [Quantipy3](https://github.com/Quantipy/quantipy3) is the market-research stack Weightipy was forked from.
|
|
325
|
+
|
|
326
|
+
**Weighting recipes.** R [`weightflow`](https://CRAN.R-project.org/package=weightflow) is the closest recipe-style analogue (eligibility → NR → calibrate → trim, plus recipe-aware bootstrap/jackknife). R [`icarus`](https://CRAN.R-project.org/package=icarus) and [`ReGenesees`](https://github.com/DiegoZardetto/ReGenesees) are calibration-focused (raking, linear/GREG, official-statistics workflows).
|
|
327
|
+
|
|
328
|
+
**Design-based analysis.** The current Python peer is [svy](https://svylab.com/svy) ([docs](https://svylab.com/docs/svy/)): one `svy.Design` + `svy.Sample`, then `.estimation`, `.weighting`, `.glm`, tabs, and BRR/SDR. It is the successor of archived [samplics](https://pypi.org/project/samplics/) (same author) and is validated against R [`survey`](https://CRAN.R-project.org/package=survey). In R, `survey` is still the usual toolkit once weights exist (`svydesign`, `svymean`, `svyby`, `svyglm`, `calibrate`, `rake`, `as.svrepdesign`); [`srvyr`](https://CRAN.R-project.org/package=srvyr) is a dplyr front end on it.
|
|
329
|
+
|
|
330
|
+
weightpipe is the recipe-first path: eligibility → NR → calibrate → trim on one `WeightPipe`, then the same estimate shape (`pipe.estimate.mean(["income", "food_share"], by="urban_rural")`, `pipe.estimate.glm("y ~ x", family="binomial")`). Use svy or R `survey` when you need sample *selection*, categorical tests, BRR/SDR, or small-area estimation ([`svy-sae`](https://svylab.com/svy)).
|
|
331
|
+
|
|
332
|
+
**Planning and selection.** [svy](https://svylab.com/svy) draws probability samples (SRS, systematic, PPS, multi-stage) and plans sizes. R [`sampler`](https://github.com/sdaza/sampler) is weightpipe's planning gold. R [`PracTools`](https://CRAN.R-project.org/package=PracTools), [`surveyplanning`](https://CRAN.R-project.org/package=surveyplanning), and [`sampling`](https://CRAN.R-project.org/package=sampling) (Tillé) are other R kits. weightpipe plans sizes, then you attach the collected microdata — it does not draw the field sample.
|
|
333
|
+
|
|
334
|
+
**Diagnostics.** Meta [`balance`](https://github.com/facebookresearch/balance) is covariate balance (SMD / Love plots), not a weighting recipe. weightpipe's `balance()` is the SMD before/after check; `margins()` checks calibration targets.
|
|
335
|
+
|
|
336
|
+
Marks mean the package has a first-class API for that row. "via `survey`" means weightflow builds the weights (or replicate weights) and you estimate in `survey` / `srvyr`.
|
|
337
|
+
|
|
338
|
+
| | weightpipe | svy | Weightipy 0.4.2 | R `survey` | R `weightflow` | R `sampler` |
|
|
339
|
+
|--|:----------:|:---:|:---------------:|:----------:|:--------------:|:-----------:|
|
|
340
|
+
| Language | Python | Python | Python | R | R | R |
|
|
341
|
+
| Design object (strata / PSU / weights) | ✓ | ✓ | | ✓ | base weights¹ | |
|
|
342
|
+
| Multi-stage inclusion weights | ✓ | ✓ | | ✓ | | |
|
|
343
|
+
| Sample selection (draw PPS / SRS) | | ✓ | | | | |
|
|
344
|
+
| Unknown eligibility / drop ineligible | ✓ | | | | ✓ | |
|
|
345
|
+
| Nonresponse | class + propensity | class | | | class + propensity | |
|
|
346
|
+
| Raking (RIM / IPF) | ✓ | ✓ | ✓ | ✓ | ✓ | |
|
|
347
|
+
| Nested / segmented RIM | | | ✓ | | | |
|
|
348
|
+
| Post-stratification | ✓ | ✓ | | ✓ | ✓ | |
|
|
349
|
+
| Linear / GREG calibration | ✓ | ✓ | | ✓ | ✓ | |
|
|
350
|
+
| Trim | ✓ | ✓ | | ✓ | ✓ | |
|
|
351
|
+
| One chained recipe | ✓ | fluent steps | raking only | | ✓ | |
|
|
352
|
+
| Recipe-aware replicate weights | ✓ | ✓ | | | ✓ | |
|
|
353
|
+
| Taylor linearization SE | ✓ | ✓ | | ✓ | via `survey` | |
|
|
354
|
+
| Bootstrap / jackknife SE | ✓ | ✓ | | ✓ | ✓ | |
|
|
355
|
+
| BRR / SDR | | ✓ | | ✓ | | |
|
|
356
|
+
| Domain estimates (`by=`) | ✓ | ✓ | | ✓ | via `survey` | |
|
|
357
|
+
| Several variables at once | ✓ | ✓ | | ✓ | via `survey` | |
|
|
358
|
+
| Design-based GLM | ✓ | ✓ | | ✓ | via `survey` | |
|
|
359
|
+
| Tabs / Rao–Scott | | ✓ | | ✓ | via `survey` | |
|
|
360
|
+
| Covariate balance (SMD) | ✓ | | | | | |
|
|
361
|
+
| Sample-size planning / allocation | ✓ | ✓ | | | | ✓ |
|
|
362
|
+
| Small-area estimation | | svy-sae | | | | |
|
|
363
|
+
|
|
364
|
+
¹ weightflow takes design weights you already computed (`weighting_spec(..., base_weights=)`). It uses strata/PSU when it resamples replicates. It does not infer SRS / stratified / cluster from `N` / `N_h` the way `WeightPipe` / `svy.Design` / `survey::svydesign` do.
|
|
365
|
+
|
|
366
|
+
Weightipy stays a focused raking library. weightpipe uses the same class of iterative raking as one `calibrate(method="raking")` step, then continues with eligibility, nonresponse, GREG, trim, balance, estimates (including design-based GLM), and planning. [svy](https://svylab.com/svy) and R `survey` remain broader for sample selection, categorical tests, tabs, BRR/SDR, and SAE.
|
|
367
|
+
|
|
368
|
+
### Numerical gold
|
|
369
|
+
|
|
370
|
+
Shared methods are checked on the same toy frames against frozen CSVs in [`tests/gold/`](tests/gold/) (CI) and optionally live R / [svy](https://svylab.com/svy). These are **correctness** checks, not runtime speed benchmarks.
|
|
371
|
+
|
|
372
|
+
| Method | R `survey` | R `weightflow` | svy² | R `sampler` |
|
|
373
|
+
|--------|:----------:|:--------------:|:----:|:-----------:|
|
|
374
|
+
| Unknown eligibility | | ✓ | | |
|
|
375
|
+
| Drop ineligible | | ✓ | | |
|
|
376
|
+
| Weighting-class NR | | ✓ | ✓ | |
|
|
377
|
+
| Raking | ✓ | ✓ | ✓ | |
|
|
378
|
+
| Post-stratification | ✓ | ✓ | ✓ | |
|
|
379
|
+
| Linear / GREG | ✓ | ✓ | ✓ | |
|
|
380
|
+
| Trim (value cap, no redistribute) | | ✓ | | |
|
|
381
|
+
| NR → raking cascade | | ✓ | | |
|
|
382
|
+
| Full cascade (eligibility → trim) | | ✓ | | |
|
|
383
|
+
| Mean, total, ratio, median | ✓ | | | |
|
|
384
|
+
| Design-based GLM | ✓ | | | |
|
|
385
|
+
| Sample size / allocation / MOE | | | | ✓ |
|
|
386
|
+
|
|
387
|
+
² Frozen `*_svy.csv` plus live checks from `uv sync --extra gold`.
|
|
388
|
+
|
|
389
|
+
Tolerances are tight (`1e-12`–`1e-6` depending on the solver). Median vs `survey::svyquantile` may differ by one unique *y* value because quantile definitions can differ. Design-based GLM coefficients and linearized SEs match `survey::svyglm` (gaussian / quasibinomial / quasipoisson). Logit propensity uses the same `1/p` adjustment as weightflow, but sklearn vs R `glm` coefficients are not bit-matched.
|
|
390
|
+
|
|
391
|
+
## Gold testing
|
|
392
|
+
|
|
393
|
+
Gold tests compare weightpipe to frozen reference outputs (and optionally live svy / R) on the same toy inputs. CSVs live in [`tests/gold/`](tests/gold/). Python weighting gold is [svy](https://svylab.com/svy); estimands and GLM gold are R `survey`.
|
|
394
|
+
|
|
395
|
+
**Run the gold suite** (same idea as CI):
|
|
396
|
+
|
|
397
|
+
```bash
|
|
398
|
+
uv sync --all-groups --extra gold --locked
|
|
399
|
+
uv run --extra gold pytest -m gold -q
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
Or run everything (unit tests + gold):
|
|
403
|
+
|
|
404
|
+
```bash
|
|
405
|
+
uv run --extra gold pytest -q
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
Frozen CSV checks always run when the files are present. Live svy checks need the `gold` extra. Live R checks need R packages and optionally `uv sync --extra r-gold`.
|
|
409
|
+
|
|
410
|
+
**Regenerate frozen CSVs** (local only — not CI). Do this when a reference tool or gold scenario intentionally changes, then commit the updated files under `tests/gold/`:
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
uv run --extra gold python tests/gold/generate_svy_gold.py
|
|
414
|
+
Rscript tests/gold/generate_r_gold.R # needs R packages survey + weightflow
|
|
415
|
+
Rscript tests/gold/generate_sampler_gold.R # needs R package sampler (or SAMPLER_R_DIR)
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
## License
|
|
419
|
+
|
|
420
|
+
MIT
|