mimic-tabular 0.1.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. mimic_tabular-0.1.3/LICENSE +21 -0
  2. mimic_tabular-0.1.3/PKG-INFO +416 -0
  3. mimic_tabular-0.1.3/README.md +376 -0
  4. mimic_tabular-0.1.3/pyproject.toml +70 -0
  5. mimic_tabular-0.1.3/setup.cfg +4 -0
  6. mimic_tabular-0.1.3/src/mimic/__init__.py +34 -0
  7. mimic_tabular-0.1.3/src/mimic/_torch_utils.py +23 -0
  8. mimic_tabular-0.1.3/src/mimic/_version.py +3 -0
  9. mimic_tabular-0.1.3/src/mimic/cli.py +164 -0
  10. mimic_tabular-0.1.3/src/mimic/datasets.py +259 -0
  11. mimic_tabular-0.1.3/src/mimic/decoders.py +912 -0
  12. mimic_tabular-0.1.3/src/mimic/diagnostics.py +304 -0
  13. mimic_tabular-0.1.3/src/mimic/encoders.py +685 -0
  14. mimic_tabular-0.1.3/src/mimic/iterated.py +373 -0
  15. mimic_tabular-0.1.3/src/mimic/mimic.py +2156 -0
  16. mimic_tabular-0.1.3/src/mimic/policies.py +28 -0
  17. mimic_tabular-0.1.3/src/mimic_experiments/__init__.py +2 -0
  18. mimic_tabular-0.1.3/src/mimic_experiments/q1_smote_roc.py +1016 -0
  19. mimic_tabular-0.1.3/src/mimic_experiments/q2_adasyn.py +482 -0
  20. mimic_tabular-0.1.3/src/mimic_experiments/q3_geometric_smote.py +669 -0
  21. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/PKG-INFO +416 -0
  22. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/SOURCES.txt +35 -0
  23. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/dependency_links.txt +1 -0
  24. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/entry_points.txt +2 -0
  25. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/requires.txt +15 -0
  26. mimic_tabular-0.1.3/src/mimic_tabular.egg-info/top_level.txt +3 -0
  27. mimic_tabular-0.1.3/src/mimic_vision/__init__.py +53 -0
  28. mimic_tabular-0.1.3/src/mimic_vision/datasets.py +395 -0
  29. mimic_tabular-0.1.3/src/mimic_vision/groups.py +34 -0
  30. mimic_tabular-0.1.3/src/mimic_vision/synthesis.py +203 -0
  31. mimic_tabular-0.1.3/src/mimic_vision/visualization.py +359 -0
  32. mimic_tabular-0.1.3/tests/test_datasets.py +58 -0
  33. mimic_tabular-0.1.3/tests/test_mimic.py +1982 -0
  34. mimic_tabular-0.1.3/tests/test_q1_smote_roc.py +408 -0
  35. mimic_tabular-0.1.3/tests/test_q2_adasyn.py +203 -0
  36. mimic_tabular-0.1.3/tests/test_q3_geometric_smote.py +236 -0
  37. mimic_tabular-0.1.3/tests/test_shared_resnet.py +172 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Fabrizio Costa
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,416 @@
1
+ Metadata-Version: 2.4
2
+ Name: mimic-tabular
3
+ Version: 0.1.3
4
+ Summary: MIMIC Tabular: modular inference for missingness, inconsistency, and creation.
5
+ Author: Fabrizio Costa
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/fabriziocosta/MIMIC
8
+ Project-URL: Documentation, https://mimic-tabular.readthedocs.io/en/latest/
9
+ Project-URL: Repository, https://github.com/fabriziocosta/MIMIC
10
+ Project-URL: Issues, https://github.com/fabriziocosta/MIMIC/issues
11
+ Project-URL: Changelog, https://github.com/fabriziocosta/MIMIC/blob/master/CHANGELOG.md
12
+ Project-URL: PyPI, https://pypi.org/project/mimic-tabular/
13
+ Keywords: tabular data,missing data,imputation,synthetic data,machine learning
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Intended Audience :: Science/Research
16
+ Classifier: Intended Audience :: Developers
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
20
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
21
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
22
+ Requires-Python: >=3.10
23
+ Description-Content-Type: text/markdown
24
+ License-File: LICENSE
25
+ Requires-Dist: joblib
26
+ Requires-Dist: numpy
27
+ Requires-Dist: pandas
28
+ Requires-Dist: scipy
29
+ Requires-Dist: scikit-learn
30
+ Requires-Dist: matplotlib
31
+ Requires-Dist: torch
32
+ Requires-Dist: tqdm
33
+ Provides-Extra: dev
34
+ Requires-Dist: pytest; extra == "dev"
35
+ Requires-Dist: ipykernel; extra == "dev"
36
+ Requires-Dist: jupyter; extra == "dev"
37
+ Requires-Dist: seaborn; extra == "dev"
38
+ Requires-Dist: scikit-posthocs; extra == "dev"
39
+ Dynamic: license-file
40
+
41
+ # MIMIC Tabular
42
+
43
+ [![Read the Docs](https://img.shields.io/readthedocs/mimic-tabular/latest)](https://mimic-tabular.readthedocs.io/en/latest/)
44
+ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.24445491.svg)](https://doi.org/10.5281/zenodo.24445490)
45
+
46
+ MIMIC stands for **Modular Inference for Missingness, Inconsistency, and
47
+ Creation**. It is a modular framework for working with mixed tabular data. It treats
48
+ imputation, consistency checking, supervised prediction, uncertainty estimation,
49
+ and synthetic data generation as variations of the same problem: predicting one
50
+ feature from the rest of the row.
51
+
52
+ ## Start here
53
+
54
+ - **Documentation:** Read the [MIMIC Tabular docs](https://mimic-tabular.readthedocs.io/en/latest/).
55
+ - **New to MIMIC?** Follow the [Getting started guide](docs/GETTING_STARTED.md)
56
+ for installation, a first generation run, and the main estimator methods.
57
+ - **Contributing?** Use the [Development guide](docs/DEVELOPMENT.md) for the
58
+ environment setup, code map, tests, and release pointers.
59
+ - **Looking for design details?** Read the [white paper](docs/WHITEPAPER.md)
60
+ or the [implementation notes](docs/IMPLEMENTATION.md).
61
+ - **Working with images?** See [MIMIC for Images](docs/IMAGE_WORKFLOWS.md) and
62
+ the [`image_workflows/` notebooks](image_workflows/notebooks/).
63
+
64
+ This README is the project overview and API tour. The onboarding guides provide
65
+ step-by-step paths for using and developing MIMIC.
66
+
67
+ The project is designed around a scikit-learn-style estimator with replaceable
68
+ encoders, decoders, and generation policies. The current implementation includes
69
+ tree-based, neural, linear, stochastic, and identity-style components so the same
70
+ interface can be used for practical modelling, diagnostics, and baselines.
71
+
72
+ ## What MIMIC Does
73
+
74
+ - Fills missing values in numerical and categorical columns.
75
+ - Estimates confidence and uncertainty for feature-wise predictions.
76
+ - Treats labels as ordinary columns, allowing supervised prediction through the
77
+ same imputation interface.
78
+ - Generates synthetic tabular rows from learned representations or baseline
79
+ feature-space policies.
80
+ - Keeps generation traceable by exposing the steps used to create synthetic
81
+ values.
82
+
83
+ ## Core Idea
84
+
85
+ For each modelled column, MIMIC learns how to predict that column from the other
86
+ columns. These feature-wise models produce latent representations, predictions,
87
+ and uncertainty diagnostics. Because every column is handled through the same
88
+ abstraction, the framework can support missing-value repair, label prediction,
89
+ outlier-style inconsistency checks, and synthetic generation without separate
90
+ task-specific pipelines.
91
+
92
+ ## Quick Example
93
+
94
+ ```python
95
+ from mimic import mimic_data
96
+
97
+ synthetic = mimic_data(df, mode="factorised", capacity=0.25)
98
+ ```
99
+
100
+ `mimic_data(df, mode="factorised", capacity=0.25)` fits a default `MIMIC` model internally and returns the same number
101
+ of synthetic rows. It is the simplest interface when you only need a sampled
102
+ dataframe.
103
+
104
+ The manuscript datasets can be loaded with matching MIMIC column roles in one
105
+ call:
106
+
107
+ ```python
108
+ from mimic import load_paper_dataset, mimic_data
109
+
110
+ df, columns = load_paper_dataset("adult", n_rows=1000, random_state=0)
111
+ synthetic = mimic_data(df, columns=columns)
112
+ ```
113
+
114
+ Available keys are `adult`, `bank_marketing`, and `default_credit`. The returned
115
+ `columns` mapping contains `ignore`, `regression`, and `classification` lists and
116
+ can be passed directly to `MIMIC` or `mimic_data`.
117
+
118
+ Nearest-neighbor ambiguity filtering is available as an opt-in attribution-risk
119
+ reduction step. It rejects generated embeddings that are too easily associated
120
+ with one training row or generation pair; it is not a formal differential
121
+ privacy guarantee.
122
+
123
+ ```python
124
+ synthetic = mimic_data(df, privacy_filter=True)
125
+ ```
126
+
127
+ Fitted models can be saved and loaded locally so generation can be repeated
128
+ without retraining:
129
+
130
+ ```python
131
+ synthetic = mimic_data(df, save_model="adult_mimic.joblib")
132
+ more_synthetic = mimic_data(df, load_model="adult_mimic.joblib", n_samples=1000)
133
+ ```
134
+
135
+ The estimator API exposes the same persistence path:
136
+
137
+ ```python
138
+ model = MIMIC(mode="factorised", random_state=0).fit(df)
139
+ model.save("adult_mimic.joblib")
140
+
141
+ loaded = MIMIC.load("adult_mimic.joblib")
142
+ synthetic = loaded.sample(1000)
143
+ ```
144
+
145
+ The same one-shot workflow is available from the command line:
146
+
147
+ ```bash
148
+ mimic-data data.csv
149
+ ```
150
+
151
+ This writes `data_mimic.csv` beside the input file. CSV, Parquet, and Excel
152
+ inputs are supported. Optional flags include `--columns`, `--mode`,
153
+ `--capacity`, `--save-model`, and `--load-model`; run `mimic-data --help` for
154
+ details.
155
+
156
+ Use the estimator directly when you want imputation, confidence diagnostics,
157
+ traceability, or reusable fitted state:
158
+
159
+ ```python
160
+ from mimic import MIMIC
161
+
162
+ model = MIMIC(
163
+ columns="auto",
164
+ random_state=0,
165
+ )
166
+
167
+ model.fit(df)
168
+
169
+ imputed = model.impute(df)
170
+ confidence = model.confidence(df)
171
+ synthetic = model.sample(100)
172
+ ```
173
+
174
+ Embeddings can also be stacked with `IteratedMIMIC`. The first MIMIC model fits
175
+ the original row space, later models fit the previous level's embedding table,
176
+ and `inverse_transform` decodes back through the stack. Later levels preserve
177
+ each feature embedding as one vector block, so a higher level predicts a whole
178
+ feature-embedding vector at once rather than one scalar dimension at a time:
179
+
180
+ ```python
181
+ from mimic import IteratedMIMIC
182
+
183
+ stack = IteratedMIMIC(n_steps=2, random_state=0).fit(df)
184
+ top_embedding = stack.transform(df)
185
+ reconstructed = stack.inverse_transform(top_embedding)
186
+ ```
187
+
188
+ `columns="auto"` infers roles using simple heuristics: ID-like or near-unique columns are ignored, small integer-valued numeric columns are classification, other numeric columns are regression, and non-numeric columns are classification. For production use, pass an explicit role mapping such as `columns={"ignore": [...], "regression": [...], "classification": [...]}`.
189
+
190
+ Embedding plots can focus on selected feature embeddings while colouring by raw
191
+ dataframe values:
192
+
193
+ ```python
194
+ fig, axes = model.plot(df, embedding_columns=["age", "income"], color_by="segment")
195
+ ```
196
+
197
+ Synthetic-data diagnostics include numeric pairwise plots and categorical
198
+ distribution comparisons:
199
+
200
+ ```python
201
+ from mimic.diagnostics import categorical_feature_report, pairwise_feature_plot
202
+
203
+ pairwise_feature_plot(
204
+ heldout,
205
+ synthetic,
206
+ features=["age", "hours-per-week", "capital-gain", "capital-loss"],
207
+ log1p_features=["capital-gain", "capital-loss"],
208
+ )
209
+ category_report, category_summary = categorical_feature_report(
210
+ heldout,
211
+ synthetic,
212
+ features=["workclass", "education", "income"],
213
+ )
214
+ ```
215
+
216
+ `MIMIC` also provides simplified `mode` and `capacity` presets. The default
217
+ `mode="factorised", capacity=0.25` uses neural components, factorised probabilistic decoding, `n_bootstrap=2`, and a
218
+ mutual-neighbour displacement generation policy:
219
+
220
+ ```python
221
+ GenerationPolicy(
222
+ method="displacement",
223
+ neighbour_mode="mutual",
224
+ n_neighbors=5,
225
+ lambda_range=(0.25, 0.75),
226
+ )
227
+ ```
228
+
229
+ ### `mode="identity"` / `mode=0`
230
+
231
+ Uses the identity encoder and identity decoder. Generation operates directly in
232
+ the preprocessed feature space, so this mode is useful as a transparent baseline
233
+ for checking what MIMIC adds beyond SMOTE-style interpolation or displacement.
234
+
235
+ ### `mode="direct"` / `mode=1`
236
+
237
+ Uses the neural encoder preset and deterministic decoding. Synthetic embeddings
238
+ are decoded directly into feature values without stochastic conditional
239
+ resampling, making outputs easier to reproduce and traces simpler.
240
+
241
+ ### `mode="factorised"` / `mode=2`
242
+
243
+ Uses the neural encoder preset with probabilistic factorised decoding. After an
244
+ initial deterministic decode, feature values are sampled from feature-wise
245
+ conditional decoders given the rest of the embedding. This is the default mode
246
+ because it keeps generation stochastic while preserving the modular
247
+ feature-wise design.
248
+
249
+ Numerical columns can optionally share one feature-conditioned residual
250
+ network. Each target receives a learned feature identity and optional coordinate
251
+ descriptors, while FiLM layers condition the residual blocks. The target value
252
+ is masked before encoding, so each returned feature block remains conditional
253
+ on the rest of the row:
254
+
255
+ ```python
256
+ from mimic import MIMIC, SharedFeatureGroup, SharedResNetEncoder
257
+
258
+ pixels = SharedFeatureGroup(
259
+ columns=pixel_columns,
260
+ coordinates=pixel_coordinates, # shape: (len(pixel_columns), coordinate_dim)
261
+ encoder=SharedResNetEncoder(embedding_dim=32),
262
+ target_chunk_size=64,
263
+ )
264
+ model = MIMIC(
265
+ columns={"regression": pixel_columns, "classification": [], "ignore": []},
266
+ shared_feature_groups={"pixels": pixels},
267
+ )
268
+ ```
269
+
270
+ Shared groups currently support numerical regression columns. Ungrouped and
271
+ categorical columns continue to use independent feature modules. Coordinates
272
+ must be finite and supplied in the same order as `columns`; image helpers
273
+ normalize spatial coordinates to `[0, 1]`.
274
+
275
+ ### `mode="joint"` / `mode=3`
276
+
277
+ Uses the neural encoder preset with deterministic joint decoding. A shared
278
+ neural row decoder predicts all modelled columns from concatenated conditional
279
+ evidence. This mode must be requested explicitly and currently requires the
280
+ neural conditional sampler.
281
+
282
+ `capacity` is a number from `0` to `1` that scales preset hyperparameters:
283
+ embedding dimension, hidden dimension, layer count, epochs, patience, batch
284
+ size, MDN components, bootstrap count, dropout, learning rate, and weight decay.
285
+ `capacity=0` is the smallest useful preset; `capacity=1` is the largest preset.
286
+ Learning rate is scaled downward in log space as capacity increases.
287
+
288
+ Explicit `encoder`, `decoder`, `policy`, `n_bootstrap`, or
289
+ `generation_decode_mode` arguments override the preset where supplied.
290
+
291
+ Independent columns use one encoder-decoder member per feature. Columns declared
292
+ in a shared numerical group instead use one full-data encoder for the group and
293
+ retain target-specific decoders. Bootstrap members are kept separate and are used
294
+ for confidence diagnostics and out-of-bag calibration, not for defining the
295
+ neighbour-search geometry used by `sample()`. Set `bootstrap=False` for
296
+ generation-only runs to skip fitting bootstrap members while still fitting the
297
+ full-data generation member:
298
+
299
+ ```python
300
+ model = MIMIC(mode="factorised", capacity=1.0, bootstrap=False, random_state=0)
301
+ synthetic = model.fit(df).sample(1000)
302
+ ```
303
+
304
+ Parallelism has two layers. `n_jobs` is passed to compatible underlying
305
+ estimators such as random forests, while `feature_n_jobs` parallelizes MIMIC's
306
+ independent feature-module fitting across target columns. Shared groups train as
307
+ single group-level models and are not split by `feature_n_jobs`. Keep one outer
308
+ layer high at a time: for example, when cross-validation folds are already parallel,
309
+ leave `feature_n_jobs=1`; when fitting one model at a time, increase
310
+ `feature_n_jobs` to train target features concurrently.
311
+
312
+ Set `verbose=True` to print constructor hyperparameters immediately and fitted
313
+ data/embedding sizes during `fit`.
314
+
315
+ `show_progress=True` is the default. It displays an overall `MIMIC fit` bar and,
316
+ for shared numerical groups, an epoch bar with validation loss. Set
317
+ `show_progress=False` for quiet batch jobs, tests, or redirected logs.
318
+
319
+ Calibration is opt-in. When enabled, MIMIC uses out-of-bag bootstrap predictions
320
+ from `fit()` to calibrate confidence outputs. Calibration therefore requires
321
+ bootstrap members; leave `bootstrap=True` when using these diagnostics:
322
+
323
+ ```python
324
+ model = MIMIC(
325
+ classification_calibration="temperature", # or "isotonic"
326
+ regression_calibration="conformal",
327
+ )
328
+ model.fit(df)
329
+ confidence = model.confidence(df)
330
+ calibration = model.calibration_report()
331
+ ```
332
+
333
+ Classification calibration adjusts reported probabilities. Regression conformal
334
+ calibration adds interval columns such as `lower_90` and `upper_90` to
335
+ `confidence()`.
336
+
337
+ Synthetic generation can choose how generated embeddings are decoded back to
338
+ rows with `generation_decode_mode`:
339
+
340
+ - `"auto"` preserves the default behaviour, using direct deterministic decoding
341
+ for ordinary decoders and factorised stochastic decoding for conditional
342
+ samplers.
343
+ - `"direct"` uses feature-wise point predictions from the generated embedding.
344
+ - `"factorised"` samples each non-conditioned feature from a fitted conditional
345
+ sampler such as `ForestConditionalSampler` or `NeuralConditionalSampler`.
346
+ - `"joint"` uses `NeuralConditionalSampler` evidence from all feature-wise
347
+ conditionals and a deterministic neural row decoder. It must be requested
348
+ explicitly and trains on complete modelled rows.
349
+
350
+ ## Installation
351
+
352
+ Install the published package:
353
+
354
+ ```bash
355
+ python -m pip install mimic-tabular
356
+ ```
357
+
358
+ The distribution is named `mimic-tabular`; its Python import package remains
359
+ `mimic`. To work from a source checkout, install it in editable mode from the
360
+ repository root:
361
+
362
+ ```bash
363
+ python -m pip install -e .
364
+ ```
365
+
366
+ For notebook and test dependencies:
367
+
368
+ ```bash
369
+ python -m pip install -e ".[dev]"
370
+ ```
371
+
372
+ ## Project Structure
373
+
374
+ | Path | Purpose |
375
+ | --- | --- |
376
+ | `src/mimic/` | Main installable tabular modeling package. |
377
+ | `src/mimic_vision/` | Image dataset and synthesis helpers used by the image workflows. |
378
+ | `src/mimic_experiments/` | Small experiment entry points distributed with the package. |
379
+ | `tests/` | Tests for the installable packages. |
380
+ | `notebooks/` | Core package tutorials and usage examples. |
381
+ | `image_workflows/` | Image-specific notebooks and their workspace guide. |
382
+ | `docs/` | User, contributor, design, and release documentation. |
383
+
384
+ See [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) for setup and contribution
385
+ guidance. Downloaded datasets and local run data belong under the ignored
386
+ root-level `data/` directory. The built documentation site is generated under
387
+ `site/` and is not source content.
388
+
389
+ The documentation guides are [Getting started](docs/GETTING_STARTED.md),
390
+ [Development](docs/DEVELOPMENT.md), [Implementation](docs/IMPLEMENTATION.md),
391
+ [Research white paper](docs/WHITEPAPER.md), [MIMIC for Images](docs/IMAGE_WORKFLOWS.md),
392
+ and [Versioning](docs/VERSIONING.md).
393
+
394
+ ## Development
395
+
396
+ Run the test suite with:
397
+
398
+ ```bash
399
+ pytest
400
+ ```
401
+
402
+ The notebooks provide examples for basic usage, classification and regression,
403
+ generation, oversampling, traceability, and alternative encoder/decoder choices.
404
+
405
+ ## Versioning
406
+
407
+ The package follows Semantic Versioning while its API is evolving. See
408
+ [`docs/VERSIONING.md`](docs/VERSIONING.md) for the release process and
409
+ [`CHANGELOG.md`](CHANGELOG.md) for release notes. The current version is
410
+ available as `mimic.__version__` and with `mimic-data --version`.
411
+
412
+ ## Status
413
+
414
+ MIMIC is an experimental research-oriented codebase. The public API is compact,
415
+ but individual modelling components are still intended to be interchangeable as
416
+ the framework evolves.