stagewatch 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. stagewatch-0.2.0/CHANGELOG.md +46 -0
  2. stagewatch-0.2.0/CITATION.cff +11 -0
  3. stagewatch-0.2.0/CONTRIBUTING.md +23 -0
  4. stagewatch-0.2.0/INSTALL.md +95 -0
  5. stagewatch-0.2.0/LICENSE +21 -0
  6. stagewatch-0.2.0/MANIFEST.in +6 -0
  7. stagewatch-0.2.0/PKG-INFO +373 -0
  8. stagewatch-0.2.0/README.md +327 -0
  9. stagewatch-0.2.0/RELEASE_MANIFEST.txt +29 -0
  10. stagewatch-0.2.0/RELEASE_NOTES.md +35 -0
  11. stagewatch-0.2.0/SECURITY.md +7 -0
  12. stagewatch-0.2.0/docs/API.md +61 -0
  13. stagewatch-0.2.0/docs/DETECTION.md +42 -0
  14. stagewatch-0.2.0/docs/FEATURES.md +30 -0
  15. stagewatch-0.2.0/docs/PUBLISHING.md +72 -0
  16. stagewatch-0.2.0/docs/TESTING.md +41 -0
  17. stagewatch-0.2.0/docs/VALIDATION.md +51 -0
  18. stagewatch-0.2.0/examples/basic_pipeline.py +25 -0
  19. stagewatch-0.2.0/pyproject.toml +73 -0
  20. stagewatch-0.2.0/run_all_tests.py +78 -0
  21. stagewatch-0.2.0/run_three_pipeline_validation.py +645 -0
  22. stagewatch-0.2.0/scripts/release_preflight.py +141 -0
  23. stagewatch-0.2.0/setup.cfg +4 -0
  24. stagewatch-0.2.0/src/stagewatch/__init__.py +11 -0
  25. stagewatch-0.2.0/src/stagewatch/__main__.py +3 -0
  26. stagewatch-0.2.0/src/stagewatch/cli.py +130 -0
  27. stagewatch-0.2.0/src/stagewatch/history.py +93 -0
  28. stagewatch-0.2.0/src/stagewatch/inspection.py +355 -0
  29. stagewatch-0.2.0/src/stagewatch/integrations/__init__.py +1 -0
  30. stagewatch-0.2.0/src/stagewatch/integrations/numpy.py +13 -0
  31. stagewatch-0.2.0/src/stagewatch/integrations/pandas.py +13 -0
  32. stagewatch-0.2.0/src/stagewatch/integrations/torch.py +13 -0
  33. stagewatch-0.2.0/src/stagewatch/models.py +117 -0
  34. stagewatch-0.2.0/src/stagewatch/pipeline.py +1016 -0
  35. stagewatch-0.2.0/src/stagewatch/py.typed +0 -0
  36. stagewatch-0.2.0/src/stagewatch/reporters/__init__.py +6 -0
  37. stagewatch-0.2.0/src/stagewatch/reporters/html_report.py +63 -0
  38. stagewatch-0.2.0/src/stagewatch/reporters/json_report.py +9 -0
  39. stagewatch-0.2.0/src/stagewatch/reporters/terminal.py +38 -0
  40. stagewatch-0.2.0/src/stagewatch.egg-info/PKG-INFO +373 -0
  41. stagewatch-0.2.0/src/stagewatch.egg-info/SOURCES.txt +45 -0
  42. stagewatch-0.2.0/src/stagewatch.egg-info/dependency_links.txt +1 -0
  43. stagewatch-0.2.0/src/stagewatch.egg-info/entry_points.txt +2 -0
  44. stagewatch-0.2.0/src/stagewatch.egg-info/requires.txt +20 -0
  45. stagewatch-0.2.0/src/stagewatch.egg-info/top_level.txt +1 -0
  46. stagewatch-0.2.0/tests/test_stagewatch.py +419 -0
  47. stagewatch-0.2.0/tests/test_stagewatch_020.py +407 -0
@@ -0,0 +1,46 @@
1
+ # Changelog
2
+
3
+ All notable StageWatch changes are documented here.
4
+
5
+ ## 0.2.0 - 2026-11-18
6
+
7
+ ### Added
8
+ - Rectangular shape inference for built-in nested lists and tuples.
9
+ - `Pipeline.expect_shape()` contracts with `None` wildcard dimensions, for input or output.
10
+ - Tukey-IQR outlier summaries and configurable outlier findings.
11
+ - Numerical distribution-shift analysis using recent successful run baselines.
12
+ - Per-numeric-column outlier and distribution summaries for Pandas DataFrames/Series.
13
+ - `Pipeline.expect_outlier_rate()` validation helper; non-numeric input now fails the measurable-rate contract rather than silently passing.
14
+ - Median-based performance regression baselines with a minimum sample count and minimum timing floor.
15
+ - Root-cause-oriented schema/input-change findings and downstream informational propagation findings.
16
+ - Deduplication of continuing anomaly counts, including re-detection after an anomaly has been cleared.
17
+ - Run-level insights in JSON and HTML reports; run status now reflects warning findings.
18
+ - Additional regression tests and a refreshed three-pipeline validation script.
19
+
20
+ ### Changed
21
+ - Expected variable shape/footprint changes can be declared and are no longer automatically treated as schema regressions.
22
+ - Cross-run shape/size changes in sequential stages are aggregated to reduce cascaded warning noise.
23
+ - Performance comparisons ignore timings below the configured floor and use a median baseline for regression alerts.
24
+ - Pandas numeric statistics are calculated per column for outlier and drift analysis.
25
+ - RunRecord deserialization remains compatible with 0.1.x records that do not contain pipeline insights.
26
+
27
+ ### Compatibility
28
+ - Python 3.10+.
29
+ - Core runtime dependencies remain empty.
30
+ - NumPy, Pandas and PyTorch remain optional integrations.
31
+ - Existing decorator, context-manager, manual recording, history, comparison, reporting and CLI interfaces are retained.
32
+
33
+ ### Limitations
34
+ - Root-cause attribution is a sequential-pipeline heuristic, not proof of causality.
35
+ - IQR outlier and distribution-shift checks are statistical signals and may not suit every distribution.
36
+ - Async, concurrent and branched pipeline orchestration is not a first-class supported API.
37
+
38
+ ## 0.1.1 - 2026-11-18
39
+
40
+ - Fixed SQLite connection lifecycle for Windows temporary-directory cleanup.
41
+ - Made CLI output resilient to legacy Windows console encodings.
42
+ - Added regression coverage for SQLite handle cleanup and unencodable stage names.
43
+
44
+ ## 0.1.0 - 2026-11-18
45
+
46
+ - Initial public alpha implementation of pipeline execution tracking, inspection, validation, history, comparison, reports, optional integrations and CLI.
@@ -0,0 +1,11 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use StageWatch in research or software, please cite the repository release."
3
+ title: "StageWatch: The Python Pipeline Inspector"
4
+ version: 0.2.0
5
+ repository-code: "https://github.com/harishraagav/StageWatch"
6
+ date-released: 2026-11-18
7
+ type: software
8
+ authors:
9
+ - family-names: "K J"
10
+ given-names: "Harish Raagav"
11
+ license: MIT
@@ -0,0 +1,23 @@
1
+ # Contributing
2
+
3
+ Thanks for considering a contribution to StageWatch.
4
+
5
+ ## Before opening a pull request
6
+
7
+ 1. Open an issue describing the problem or proposed change.
8
+ 2. Keep core runtime dependencies at zero unless there is a compelling reason.
9
+ 3. Add or update tests for every behaviour change.
10
+ 4. Update README/docs and CHANGELOG where appropriate.
11
+ 5. Run `python run_all_tests.py`, `pytest` (when installed), and `ruff check src tests` (when installed).
12
+
13
+ ## Design principles
14
+
15
+ - Local-first and offline by default.
16
+ - Never persist raw inputs or outputs by default.
17
+ - Keep optional framework integrations optional.
18
+ - Prefer stable, documented Python APIs over clever implicit behaviour.
19
+ - Treat inspection as best-effort and bounded; do not unexpectedly consume iterators or synchronize GPUs.
20
+
21
+ ## Pull requests
22
+
23
+ Use focused changes, explain the motivation, include tests, and document known limitations. Do not include secrets, real customer data, large generated artifacts, or personal information in fixtures.
@@ -0,0 +1,95 @@
1
+ # Install and test StageWatch 0.2.0a1
2
+
3
+ StageWatch 0.2.0a1 is an alpha release bundle for local evaluation. The core package has no mandatory runtime dependencies. NumPy, Pandas and PyTorch integrations are optional.
4
+
5
+ ## Windows (PowerShell or Command Prompt)
6
+
7
+ 1. Extract `StageWatch_0.2.0a1_release_bundle.zip`.
8
+ 2. Open a terminal in the extracted `StageWatch-0.2.0a1` folder (the folder containing `pyproject.toml`).
9
+ 3. Create and activate a virtual environment:
10
+
11
+ ```powershell
12
+ py -m venv .venv
13
+ .\.venv\Scripts\Activate.ps1
14
+ ```
15
+
16
+ In Command Prompt, use `\.venv\Scripts\activate.bat` instead.
17
+
18
+ 4. Install the included wheel:
19
+
20
+ ```powershell
21
+ python -m pip install --upgrade --force-reinstall .\dist\stagewatch-0.2.0a1-py3-none-any.whl
22
+ ```
23
+
24
+ 5. Confirm the installed version and location:
25
+
26
+ ```powershell
27
+ python -c "import stagewatch; print(stagewatch.__version__); print(stagewatch.__file__)"
28
+ ```
29
+
30
+ 6. Run the complete automated suite:
31
+
32
+ ```powershell
33
+ python run_all_tests.py
34
+ ```
35
+
36
+ 7. Run the three realistic synthetic pipeline scenarios:
37
+
38
+ ```powershell
39
+ python run_three_pipeline_validation.py
40
+ ```
41
+
42
+ 8. Try the included basic example:
43
+
44
+ ```powershell
45
+ python examples\basic_pipeline.py
46
+ ```
47
+
48
+ The two test scripts use the source tree in this release folder, so they test the files you extracted rather than silently using an older installed copy. The three-pipeline validation script uses local synthetic data; it does not download datasets or call external APIs.
49
+
50
+ ## macOS / Linux
51
+
52
+ ```bash
53
+ python3 -m venv .venv
54
+ source .venv/bin/activate
55
+ python -m pip install --upgrade --force-reinstall ./dist/stagewatch-0.2.0a1-py3-none-any.whl
56
+ python -c "import stagewatch; print(stagewatch.__version__); print(stagewatch.__file__)"
57
+ python run_all_tests.py
58
+ python run_three_pipeline_validation.py
59
+ python examples/basic_pipeline.py
60
+ ```
61
+
62
+ ## Install optional integrations
63
+
64
+ Install only the integrations you need:
65
+
66
+ ```bash
67
+ python -m pip install --pre "stagewatch[numpy]"
68
+ python -m pip install --pre "stagewatch[pandas]"
69
+ python -m pip install --pre "stagewatch[torch]"
70
+ python -m pip install --pre "stagewatch[all]"
71
+ ```
72
+
73
+ In the release source folder, install optional extras from source with `python -m pip install -e ".[numpy,pandas]"` (or add `torch`). The all-in-one test runner automatically runs optional integration tests when the relevant libraries are installed and reports a skip when an optional library is absent.
74
+
75
+ ## Build from source
76
+
77
+ With the build extra installed:
78
+
79
+ ```bash
80
+ python -m pip install -e ".[dev]"
81
+ python -m build
82
+ python -m twine check dist/*
83
+ ```
84
+
85
+ ## Test outputs
86
+
87
+ `run_all_tests.py` creates `test_output/` with terminal, JSON and HTML demo reports plus a local SQLite history database.
88
+
89
+ `run_three_pipeline_validation.py` creates timestamped reports, comparisons, summaries and SQLite history under `test_output/three_pipeline_validation/` for:
90
+
91
+ - RGB image processing;
92
+ - NumPy/PyTorch-style ML inference (prefers PyTorch, then NumPy, then pure Python);
93
+ - Pandas-style data cleaning (uses Pandas when installed, otherwise pure Python).
94
+
95
+ These are synthetic integration scenarios, not production workload benchmarks. Warnings are expected where the script deliberately introduces outliers or statistical changes.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Harish Raagav K J
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,6 @@
1
+ include README.md LICENSE CHANGELOG.md CONTRIBUTING.md SECURITY.md INSTALL.md CITATION.cff RELEASE_NOTES.md RELEASE_MANIFEST.txt
2
+ include run_all_tests.py run_three_pipeline_validation.py
3
+ recursive-include docs *.md
4
+ recursive-include examples *.py
5
+ recursive-include tests *.py
6
+ recursive-include scripts *.py
@@ -0,0 +1,373 @@
1
+ Metadata-Version: 2.4
2
+ Name: stagewatch
3
+ Version: 0.2.0
4
+ Summary: Local-first pipeline observability with anomaly detection, schema contracts and root-cause insights.
5
+ Author: Harish Raagav K J
6
+ Maintainer: Harish Raagav K J
7
+ License-Expression: MIT
8
+ Project-URL: Homepage, https://github.com/harishraagav/StageWatch
9
+ Project-URL: Repository, https://github.com/harishraagav/StageWatch
10
+ Project-URL: Issues, https://github.com/harishraagav/StageWatch/issues
11
+ Project-URL: Changelog, https://github.com/harishraagav/StageWatch/blob/main/CHANGELOG.md
12
+ Project-URL: Documentation, https://github.com/harishraagav/StageWatch/tree/main/docs
13
+ Keywords: python,pipeline,observability,profiling,data-quality,debugging,ml,etl
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Topic :: Software Development :: Debuggers
17
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
18
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
19
+ Classifier: Operating System :: OS Independent
20
+ Classifier: Programming Language :: Python :: 3
21
+ Classifier: Programming Language :: Python :: 3.10
22
+ Classifier: Programming Language :: Python :: 3.11
23
+ Classifier: Programming Language :: Python :: 3.12
24
+ Classifier: Programming Language :: Python :: 3.13
25
+ Classifier: Programming Language :: Python :: 3.14
26
+ Classifier: Typing :: Typed
27
+ Requires-Python: >=3.10
28
+ Description-Content-Type: text/markdown
29
+ License-File: LICENSE
30
+ Provides-Extra: numpy
31
+ Requires-Dist: numpy>=1.23; extra == "numpy"
32
+ Provides-Extra: pandas
33
+ Requires-Dist: pandas>=1.5; extra == "pandas"
34
+ Provides-Extra: torch
35
+ Requires-Dist: torch>=2.0; extra == "torch"
36
+ Provides-Extra: all
37
+ Requires-Dist: numpy>=1.23; extra == "all"
38
+ Requires-Dist: pandas>=1.5; extra == "all"
39
+ Requires-Dist: torch>=2.0; extra == "all"
40
+ Provides-Extra: dev
41
+ Requires-Dist: pytest>=7.4; extra == "dev"
42
+ Requires-Dist: ruff>=0.5; extra == "dev"
43
+ Requires-Dist: build>=1.2; extra == "dev"
44
+ Requires-Dist: twine>=5.1; extra == "dev"
45
+ Dynamic: license-file
46
+
47
+ # StageWatch
48
+
49
+ [![Tests](https://github.com/harishraagav/StageWatch/actions/workflows/tests.yml/badge.svg)](https://github.com/harishraagav/StageWatch/actions/workflows/tests.yml)
50
+ [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/harishraagav/StageWatch/blob/main/LICENSE)
51
+ [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
52
+
53
+ **Repository:** https://github.com/harishraagav/StageWatch
54
+ **Issues and feature requests:** https://github.com/harishraagav/StageWatch/issues
55
+
56
+ **The Python Pipeline Inspector.** Understand every stage of a Python data pipeline, detect suspicious numerical and structural changes, compare executions, and generate local reports.
57
+
58
+ **Author, developer and maintainer:** Harish Raagav K J
59
+
60
+ > **Current release: 0.2.0 — Public release.** This release adds dynamic shape contracts, IQR outlier detection, distribution-shift checks, robust performance baselines, upstream-change aggregation, and root-cause-oriented reporting. It is suitable for evaluation and integration testing; it is not a substitute for production monitoring or a correctness proof.
61
+
62
+ StageWatch is a local-first observability toolkit for sequential Python functions and data-processing pipelines. The core uses the Python standard library only. NumPy, Pandas and PyTorch support are optional extras.
63
+
64
+ ## Why StageWatch?
65
+
66
+ When a pipeline's final result looks wrong, developers often inspect each function manually. StageWatch records stage execution, bounded input/output summaries, timings, validation findings and exceptions. It then compares current behaviour with recent runs to help locate the earliest observable change.
67
+
68
+ ```text
69
+ load_rgb_image SUCCESS 0.42 ms
70
+ resize_image SUCCESS 0.18 ms
71
+ normalize_image WARNING 0.61 ms 3 NaN values detected
72
+ model_inference SUCCESS 18.42 ms
73
+ export SUCCESS 1.21 ms
74
+ ```
75
+
76
+ ## Highlights
77
+
78
+ - **Stage execution tracking:** order, run IDs, timestamps, duration, status and exception capture.
79
+ - **Safe data inspection:** types, dimensions, dtype, footprint estimates, missing values and numerical summaries without retaining raw inputs/outputs.
80
+ - **Numerical anomaly checks:** NaN, infinity, `None`, missing values, empty values and Tukey-IQR statistical outliers.
81
+ - **Distribution shift:** compare output means and spreads against a median baseline from recent runs; Pandas numeric columns are checked independently.
82
+ - **Schema contracts:** declare expected shapes with wildcard dimensions such as `(None, 16)` for variable batch sizes.
83
+ - **Upstream-change aggregation:** report the earliest observable structural change and mark likely downstream propagation as informational instead of repeating the same warning at every stage.
84
+ - **Performance profiling:** per-stage and run duration, plus regression alerts using a median baseline and configurable sample/duration floors.
85
+ - **Run comparison and local history:** compare recent executions; retain history in SQLite or memory only.
86
+ - **Reports:** terminal, JSON and standalone HTML with pipeline insights, findings and run history.
87
+ - **Custom checks:** predicate checks, numeric range checks, shape contracts and outlier-rate contracts.
88
+ - **Multiple APIs:** decorators, context managers and manual recording.
89
+ - **CLI:** inspect history, render reports and compare run IDs.
90
+ - **Local-first privacy:** no telemetry, network calls, API keys or hosted service. Raw inputs and outputs are not persisted by built-in recording paths.
91
+
92
+ ## Install
93
+
94
+ **Release status:** StageWatch 0.2.0 is the regular public release. Install it from PyPI with:
95
+
96
+ ```bash
97
+ python -m pip install stagewatch
98
+ ```
99
+
100
+ To install the included wheel from this release bundle before publication:
101
+
102
+ ```powershell
103
+ py -m pip install --upgrade --force-reinstall .\dist\stagewatch-0.2.0-py3-none-any.whl
104
+ ```
105
+
106
+ Optional integrations from this extracted source tree:
107
+
108
+ ```bash
109
+ python -m pip install ".[numpy]"
110
+ python -m pip install ".[pandas]"
111
+ python -m pip install ".[torch]"
112
+ python -m pip install ".[all]"
113
+ ```
114
+
115
+ After PyPI publication, install optional extras with `python -m pip install "stagewatch[numpy]"` (or `pandas`, `torch`, or `all`).
116
+
117
+ Core dependencies: **none**. Python: **3.10+**.
118
+
119
+ ## Quick start
120
+
121
+ ```python
122
+ from stagewatch import Pipeline, stage
123
+
124
+ pipeline = Pipeline("image-preprocessing")
125
+
126
+ @stage(pipeline)
127
+ def resize(image):
128
+ return image[::2, ::2, :]
129
+
130
+ @stage(pipeline)
131
+ def normalize(image):
132
+ return image.astype("float32") / 255.0
133
+
134
+ # Accept variable image height/width, but require three channels.
135
+ pipeline.expect_shape("resize", (None, None, 3))
136
+ pipeline.expect_shape("normalize", (None, None, 3))
137
+ pipeline.expect_range("normalize", minimum=0.0, maximum=1.0)
138
+
139
+ result = pipeline.run(image_array)
140
+ print(pipeline.report())
141
+ pipeline.report(format="html", path="artifacts/pipeline-report.html")
142
+ ```
143
+
144
+ `Pipeline.run(data)` passes each registered stage's output to the next stage, in registration order. The first positional argument of each decorated stage is inspected as its input.
145
+
146
+ ## Dynamic shape contracts
147
+
148
+ A `None` dimension is a wildcard. It explicitly declares that dimension as variable and avoids expected cross-run schema/footprint warnings for that contract.
149
+
150
+ ```python
151
+ from stagewatch import Pipeline, stage
152
+
153
+ pipeline = Pipeline("batch-inference")
154
+ pipeline.expect_shape("prepare", (None, 16))
155
+ pipeline.expect_shape("predict", (None, 4))
156
+
157
+ @stage(pipeline, "prepare")
158
+ def prepare(batch):
159
+ return batch
160
+
161
+ @stage(pipeline, "predict")
162
+ def predict(features):
163
+ return model(features)
164
+
165
+ pipeline.run(batch_of_32)
166
+ pipeline.run(batch_of_128) # batch-size change is allowed by the contracts
167
+ ```
168
+
169
+ Contracts can target `input` or `output`:
170
+
171
+ ```python
172
+ pipeline.expect_shape("prepare", (None, 16), target="input")
173
+ ```
174
+
175
+ A mismatch in a fixed dimension generates `shape_contract_violation`. For regular nested Python lists/tuples, StageWatch infers rectangular shapes. Ragged nested sequences do not receive a rectangular shape contract summary.
176
+
177
+ ## Anomaly and distribution settings
178
+
179
+ By default StageWatch:
180
+
181
+ - detects NaN, infinity, missing/None and empty values;
182
+ - applies Tukey's IQR rule with multiplier `1.5`;
183
+ - reports statistical outliers when at least 3 values and at least 1% of scanned numeric values are outliers;
184
+ - compares numeric output distribution against at least 3 recent successful observations per stage;
185
+ - checks distribution mean shift in baseline standard deviations and material spread changes.
186
+
187
+ Configure these choices:
188
+
189
+ ```python
190
+ pipeline = Pipeline(
191
+ "quality-check",
192
+ detect_outliers=True,
193
+ outlier_iqr_multiplier=1.5,
194
+ outlier_min_count=3,
195
+ outlier_min_ratio=0.01,
196
+ distribution_shift_threshold=3.0,
197
+ distribution_min_samples=3,
198
+ )
199
+ ```
200
+
201
+ Use the built-in outlier-rate contract:
202
+
203
+ ```python
204
+ pipeline.expect_outlier_rate(
205
+ "validate_features",
206
+ maximum_ratio=0.02,
207
+ target="output",
208
+ severity="warning",
209
+ )
210
+ ```
211
+
212
+ Outlier and distribution-shift findings are statistical signals, not proof of bad data. They can be inappropriate for naturally heavy-tailed, multimodal or mixed-scale data. For DataFrames, StageWatch calculates IQR outliers and distribution shifts independently for each numeric column.
213
+
214
+ ## Custom validation
215
+
216
+ ```python
217
+ pipeline.expect_range("normalize", minimum=0.0, maximum=1.0)
218
+ pipeline.expect_shape("infer", (None, 4))
219
+ pipeline.expect_outlier_rate("infer", maximum_ratio=0.01)
220
+
221
+ pipeline.add_check(
222
+ "validate_predictions",
223
+ lambda values: values is not None and len(values) > 0,
224
+ message="Predictions must not be empty",
225
+ severity="error",
226
+ )
227
+ ```
228
+
229
+ A custom check passes when it returns `True` or `None`. Returning `False` or a string creates a finding. A validator exception creates a `validator_error` finding and does not replace the pipeline's own exception handling.
230
+
231
+ ## Performance baselines
232
+
233
+ StageWatch uses recent successful runs and their **median** stage duration rather than treating one previous timing as a reliable baseline.
234
+
235
+ ```python
236
+ pipeline = Pipeline(
237
+ "training-step",
238
+ performance_window=7,
239
+ performance_min_samples=3,
240
+ performance_min_duration_ms=5.0,
241
+ slowdown_threshold=1.0,
242
+ )
243
+ ```
244
+
245
+ With these settings, regression checks wait for at least three baseline observations, ignore stage timings below 5 ms, and warn when the current stage is at least 100% slower than the median. Use `slowdown_threshold=None` to disable these checks. Very small timings are noisy and should not be treated as stable performance benchmarks.
246
+
247
+ ## Root-cause-oriented findings
248
+
249
+ When a stage's output first changes while its input structure remains stable, StageWatch marks that as the earliest observable root stage. Later shape/size changes are grouped as `upstream_change_propagated` informational findings rather than repeated warnings.
250
+
251
+ This is a **heuristic for sequential pipelines**, not a causal debugger. It cannot prove why a function's behaviour changed, and independent changes in concurrent, branched or externally stateful workflows may require custom checks and manual investigation.
252
+
253
+ ## Context-manager API
254
+
255
+ ```python
256
+ pipeline = Pipeline("etl-job")
257
+ source = {"records": [1, 2, 3]}
258
+
259
+ with pipeline.stage("transform", input_data=source) as tracked:
260
+ result = {"records": [x * 10 for x in source["records"]]}
261
+ tracked.set_output(result)
262
+
263
+ print(pipeline.report())
264
+ ```
265
+
266
+ If an exception escapes the context, StageWatch records the failed stage and allows the original exception to propagate.
267
+
268
+ ## Manual recording API
269
+
270
+ ```python
271
+ pipeline.record_stage("clean", raw_input, clean_output, duration_ms=2.4)
272
+ ```
273
+
274
+ ## Custom inspectors
275
+
276
+ ```python
277
+ from stagewatch import register_inspector
278
+
279
+ class Image:
280
+ def __init__(self, width, height):
281
+ self.width, self.height = width, height
282
+
283
+ register_inspector(
284
+ lambda value: isinstance(value, Image),
285
+ lambda image, byte_limit: {
286
+ "shape": [image.height, image.width],
287
+ "estimated_size_bytes": image.width * image.height * 3,
288
+ },
289
+ )
290
+ ```
291
+
292
+ ## Reports
293
+
294
+ ```python
295
+ print(pipeline.report(format="terminal"))
296
+ pipeline.report(format="json", path="artifacts/run.json")
297
+ pipeline.report(format="html", path="artifacts/run.html")
298
+ ```
299
+
300
+ - `terminal` returns readable text unless a path is provided.
301
+ - `json` returns JSON text unless a path is provided.
302
+ - `html` writes a standalone HTML report and returns its path.
303
+
304
+ ## CLI
305
+
306
+ ```bash
307
+ stagewatch --version
308
+ stagewatch history
309
+ stagewatch history --db .stagewatch/my-pipeline.sqlite3 --limit 10
310
+ stagewatch report --format terminal
311
+ stagewatch report --format json --output run.json
312
+ stagewatch report --format html --output report.html
313
+ stagewatch report --db .stagewatch/my-pipeline.sqlite3 --run-id RUN_ID
314
+ stagewatch compare RUN_ID_OLD RUN_ID_NEW --db .stagewatch/my-pipeline.sqlite3
315
+ ```
316
+
317
+ Without `--db`, the CLI uses the only `*.sqlite3` database it finds in `.stagewatch/`. It requests an explicit path if zero or multiple databases are found.
318
+
319
+ ## Local configuration
320
+
321
+ ```python
322
+ pipeline = Pipeline(
323
+ "my-pipeline",
324
+ storage_dir=".stagewatch", # SQLite directory; None disables persistence
325
+ history=True,
326
+ max_history=1000,
327
+ deep_scan_limit_bytes=1_000_000,
328
+ inspect_gpu=False,
329
+ size_change_threshold=1.0, # 100% footprint change; None disables
330
+ slowdown_threshold=1.0, # 100% slower than baseline median; None disables
331
+ performance_window=5,
332
+ performance_min_samples=3,
333
+ performance_min_duration_ms=5.0,
334
+ performance_comparison_threshold=0.10,
335
+ detect_outliers=True,
336
+ outlier_iqr_multiplier=1.5,
337
+ outlier_min_count=3,
338
+ outlier_min_ratio=0.01,
339
+ distribution_shift_threshold=3.0, # None disables
340
+ distribution_min_samples=3,
341
+ aggregate_upstream_changes=True,
342
+ )
343
+ ```
344
+
345
+ Use `storage_dir=None` or `history=False` to keep history in memory only. `deep_scan_limit_bytes` bounds deep inspection for supported third-party arrays/tables/tensors. GPU tensor inspection is disabled by default because reading a device tensor can synchronize execution.
346
+
347
+ ## Privacy and operational notes
348
+
349
+ - StageWatch does not send telemetry, make network requests or call external services.
350
+ - Built-in recording persists summaries and metadata, not raw input/output payloads.
351
+ - Run metadata is persisted as supplied. Do not put secrets, personal data or raw payloads in `metadata`.
352
+ - Exception messages can contain application data; inspect them before sharing reports or databases.
353
+ - Python object footprint estimates are approximate, not process RSS or exact allocator measurements.
354
+ - Inspections are best-effort and bounded; custom classes need a registered inspector for meaningful summaries.
355
+ - The package is synchronous and designed for sequential pipelines. Async/concurrent/branched orchestration is not a first-class supported API.
356
+
357
+ ## Development and testing
358
+
359
+ From the extracted repository root:
360
+
361
+ ```bash
362
+ python -m pip install -e ".[dev,numpy,pandas]"
363
+ python run_all_tests.py
364
+ python run_three_pipeline_validation.py
365
+ ```
366
+
367
+ The first command installs the editable package plus development and optional test dependencies. The test runner uses `unittest` and creates sample reports under `test_output/`. The three-pipeline script uses deterministic synthetic data only; it makes no network requests.
368
+
369
+ See [INSTALL.md](INSTALL.md), [docs/API.md](docs/API.md), [docs/DETECTION.md](docs/DETECTION.md), [docs/TESTING.md](docs/TESTING.md), and [docs/PUBLISHING.md](docs/PUBLISHING.md).
370
+
371
+ ## License
372
+
373
+ MIT. See [LICENSE](LICENSE).