quarantine-py 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- quarantine_py-0.1.0/.gitignore +28 -0
- quarantine_py-0.1.0/CHANGELOG.md +53 -0
- quarantine_py-0.1.0/CONTRIBUTING.md +49 -0
- quarantine_py-0.1.0/LICENSE +21 -0
- quarantine_py-0.1.0/Makefile +38 -0
- quarantine_py-0.1.0/PKG-INFO +448 -0
- quarantine_py-0.1.0/README.md +393 -0
- quarantine_py-0.1.0/docs/api.md +223 -0
- quarantine_py-0.1.0/docs/cli.md +214 -0
- quarantine_py-0.1.0/docs/faq.md +117 -0
- quarantine_py-0.1.0/docs/index.md +61 -0
- quarantine_py-0.1.0/docs/installation.md +145 -0
- quarantine_py-0.1.0/docs/on-disk-format.md +141 -0
- quarantine_py-0.1.0/docs/troubleshooting.md +179 -0
- quarantine_py-0.1.0/docs/usage.md +441 -0
- quarantine_py-0.1.0/pyproject.toml +191 -0
- quarantine_py-0.1.0/src/quarantine/__init__.py +75 -0
- quarantine_py-0.1.0/src/quarantine/__main__.py +8 -0
- quarantine_py-0.1.0/src/quarantine/_version.py +5 -0
- quarantine_py-0.1.0/src/quarantine/api.py +276 -0
- quarantine_py-0.1.0/src/quarantine/cli.py +503 -0
- quarantine_py-0.1.0/src/quarantine/core.py +606 -0
- quarantine_py-0.1.0/src/quarantine/errors.py +57 -0
- quarantine_py-0.1.0/src/quarantine/fingerprint.py +54 -0
- quarantine_py-0.1.0/src/quarantine/py.typed +0 -0
- quarantine_py-0.1.0/src/quarantine/record.py +179 -0
- quarantine_py-0.1.0/src/quarantine/redact.py +184 -0
- quarantine_py-0.1.0/src/quarantine/reporting.py +168 -0
- quarantine_py-0.1.0/src/quarantine/resolve.py +160 -0
- quarantine_py-0.1.0/src/quarantine/sentinels.py +58 -0
- quarantine_py-0.1.0/src/quarantine/serialize.py +177 -0
- quarantine_py-0.1.0/src/quarantine/store.py +484 -0
- quarantine_py-0.1.0/tests/__init__.py +1 -0
- quarantine_py-0.1.0/tests/conftest.py +60 -0
- quarantine_py-0.1.0/tests/test_api.py +144 -0
- quarantine_py-0.1.0/tests/test_async.py +79 -0
- quarantine_py-0.1.0/tests/test_cli.py +384 -0
- quarantine_py-0.1.0/tests/test_decorator.py +268 -0
- quarantine_py-0.1.0/tests/test_dedup.py +112 -0
- quarantine_py-0.1.0/tests/test_docs.py +148 -0
- quarantine_py-0.1.0/tests/test_limits.py +140 -0
- quarantine_py-0.1.0/tests/test_package.py +189 -0
- quarantine_py-0.1.0/tests/test_processes.py +100 -0
- quarantine_py-0.1.0/tests/test_redact.py +166 -0
- quarantine_py-0.1.0/tests/test_reporting.py +182 -0
- quarantine_py-0.1.0/tests/test_resolve.py +123 -0
- quarantine_py-0.1.0/tests/test_retry.py +240 -0
- quarantine_py-0.1.0/tests/test_serialize.py +148 -0
- quarantine_py-0.1.0/tests/test_shield.py +83 -0
- quarantine_py-0.1.0/tests/test_store.py +324 -0
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
build/
|
|
6
|
+
dist/
|
|
7
|
+
.eggs/
|
|
8
|
+
|
|
9
|
+
# Virtualenvs
|
|
10
|
+
.venv/
|
|
11
|
+
venv/
|
|
12
|
+
|
|
13
|
+
# Tooling caches
|
|
14
|
+
.pytest_cache/
|
|
15
|
+
.mypy_cache/
|
|
16
|
+
.ruff_cache/
|
|
17
|
+
.coverage
|
|
18
|
+
.coverage.*
|
|
19
|
+
coverage.xml
|
|
20
|
+
htmlcov/
|
|
21
|
+
|
|
22
|
+
# quarantine's own sick bay (never commit failed inputs - they may hold real data)
|
|
23
|
+
.quarantine/
|
|
24
|
+
|
|
25
|
+
# Editors / OS
|
|
26
|
+
.idea/
|
|
27
|
+
.vscode/
|
|
28
|
+
.DS_Store
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
|
|
5
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [0.1.0] - 2026-08-22
|
|
8
|
+
|
|
9
|
+
First release.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `@quarantine` decorator (bare or configured) for sync **and** `async def`
|
|
14
|
+
functions: exceptions are written to a `.quarantine/` folder and the caller
|
|
15
|
+
keeps going instead of crashing.
|
|
16
|
+
- `shield(items, using=...)` / `ashield(...)` for protecting a loop without
|
|
17
|
+
decorating anything.
|
|
18
|
+
- `Quarantine` class - the explicit, injectable form of the decorator, with
|
|
19
|
+
`wrap()`, `call()`, `acall()`, `records()`, `retry()`, `clear()` and `stats`.
|
|
20
|
+
- On-disk record format: `input.pkl` / `input.json`, `input.txt`,
|
|
21
|
+
`traceback.txt`, `meta.json` per record, plus a rebuildable `index.json`.
|
|
22
|
+
Records are built in a temp directory and atomically renamed into place, so a
|
|
23
|
+
crash mid-save can never corrupt the folder.
|
|
24
|
+
- Serialization fallback chain: pickle -> JSON -> `repr`, so *something*
|
|
25
|
+
readable is always saved.
|
|
26
|
+
- `redact=[...]` scrubbing (case-insensitive, glob-aware, recursive, never
|
|
27
|
+
mutates the caller's object) applied *before* anything touches the disk.
|
|
28
|
+
- Consecutive-failure circuit breaker (`halt_after`, default 50) raising
|
|
29
|
+
`SystemicFailure`, and a disk cap (`max_items`, default 10 000) raising
|
|
30
|
+
`QuarantineFull`.
|
|
31
|
+
- Deduplication: an item already in quarantine is skipped on a rerun
|
|
32
|
+
(`skip_known_bad`), which also keeps the folder free of duplicates.
|
|
33
|
+
- `on_quarantine=` hook for alerting, and an end-of-run summary line printed to
|
|
34
|
+
stderr (`report=`), with an ASCII fallback for non-UTF-8 consoles.
|
|
35
|
+
- CLI: `quarantine list`, `show`, `retry`, `debug`, `clear`, `stats`,
|
|
36
|
+
`reindex`, each with `--json` where it makes sense.
|
|
37
|
+
- `retry --import job.py` / `debug --import job.py`, so functions defined in a
|
|
38
|
+
script (module name `__main__`, which a later process cannot import) can
|
|
39
|
+
still be replayed. Records store the defining file for exactly this, and the
|
|
40
|
+
error message names the flag when it is needed.
|
|
41
|
+
- Console-safe output: every glyph falls back to ASCII when the stream cannot
|
|
42
|
+
encode it, checked *before* writing. `sys.stderr` uses `backslashreplace`, so
|
|
43
|
+
waiting for a `UnicodeEncodeError` would print an escape sequence rather than
|
|
44
|
+
a readable `OK`.
|
|
45
|
+
- Transient `PermissionError` on the commit rename - a scanner or sync client
|
|
46
|
+
holding a brand-new folder open on Windows - is retried with backoff instead
|
|
47
|
+
of costing a record.
|
|
48
|
+
- Documentation: `docs/` covering installation, usage, the CLI, the API, the
|
|
49
|
+
on-disk format, troubleshooting and an FAQ, with tests that fail if the docs
|
|
50
|
+
drift from the code.
|
|
51
|
+
- Full type annotations and a `py.typed` marker.
|
|
52
|
+
|
|
53
|
+
[0.1.0]: https://github.com/quarantine-py/quarantine/releases/tag/v0.1.0
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for looking. This is a small library with a deliberately small surface:
|
|
4
|
+
if a change makes the common case (`@quarantine` on one function) harder to
|
|
5
|
+
understand, it probably belongs in a different package.
|
|
6
|
+
|
|
7
|
+
## Getting set up
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
python -m venv .venv
|
|
11
|
+
. .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
12
|
+
pip install -e ".[dev]"
|
|
13
|
+
pre-commit install # optional, but it runs what CI runs
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## The checks
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
make check # or, without make:
|
|
20
|
+
ruff check . && ruff format --check .
|
|
21
|
+
mypy
|
|
22
|
+
pytest --cov --cov-report=term-missing
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
All three must pass, on Python 3.10 through 3.13, on Linux, macOS and Windows.
|
|
26
|
+
Coverage must stay at or above 90%.
|
|
27
|
+
|
|
28
|
+
## Ground rules for changes
|
|
29
|
+
|
|
30
|
+
- **Never lose a failure.** Any code path that catches an exception must either
|
|
31
|
+
write a record or re-raise. Silently dropping an item is the one bug this
|
|
32
|
+
library cannot have.
|
|
33
|
+
- **The folder is the API.** `.quarantine/` is plain files that people read,
|
|
34
|
+
`grep` and delete by hand. Record directories must stay self-describing, and
|
|
35
|
+
`index.json` must stay a rebuildable cache - never the source of truth.
|
|
36
|
+
- **Writes stay atomic.** Build in a temp directory, rename into place. If you
|
|
37
|
+
add a file to a record, add it before the rename.
|
|
38
|
+
- **Redact before writing, not after.** `redact_call()` runs before anything
|
|
39
|
+
reaches the disk, and never mutates the caller's object.
|
|
40
|
+
- **No runtime dependencies.** `pip install quarantine-py` should stay a
|
|
41
|
+
no-questions-asked install in someone else's messy environment.
|
|
42
|
+
- **Tests describe behaviour**, not implementation: `test_loop_survives_a_bad_item`,
|
|
43
|
+
not `test_wrapper_calls_store_add`.
|
|
44
|
+
|
|
45
|
+
## Releasing
|
|
46
|
+
|
|
47
|
+
1. Update `CHANGELOG.md` and `__version__` in `src/quarantine/_version.py`.
|
|
48
|
+
2. `make build` and check the artefacts in `dist/`.
|
|
49
|
+
3. Tag `v<version>`; CI builds and validates the wheel.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 quarantine contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Developer shortcuts. Everything here is also what CI runs.
|
|
2
|
+
.DEFAULT_GOAL := help
|
|
3
|
+
PYTHON ?= python
|
|
4
|
+
|
|
5
|
+
.PHONY: help install lint format typecheck test cover check build clean
|
|
6
|
+
|
|
7
|
+
help: ## Show this help
|
|
8
|
+
@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " %-12s %s\n", $$1, $$2}'
|
|
9
|
+
|
|
10
|
+
install: ## Install the package and dev tools into the current environment
|
|
11
|
+
$(PYTHON) -m pip install -e ".[dev]"
|
|
12
|
+
|
|
13
|
+
lint: ## Ruff lint + format check
|
|
14
|
+
ruff check .
|
|
15
|
+
ruff format --check .
|
|
16
|
+
|
|
17
|
+
format: ## Apply ruff fixes and formatting
|
|
18
|
+
ruff check . --fix
|
|
19
|
+
ruff format .
|
|
20
|
+
|
|
21
|
+
typecheck: ## mypy (strict)
|
|
22
|
+
mypy
|
|
23
|
+
|
|
24
|
+
test: ## Run the test suite
|
|
25
|
+
pytest
|
|
26
|
+
|
|
27
|
+
cover: ## Run the test suite with coverage
|
|
28
|
+
pytest --cov --cov-report=term-missing
|
|
29
|
+
|
|
30
|
+
check: lint typecheck cover ## Everything CI checks
|
|
31
|
+
|
|
32
|
+
build: ## Build sdist + wheel and validate the metadata
|
|
33
|
+
$(PYTHON) -m build
|
|
34
|
+
$(PYTHON) -m twine check --strict dist/*
|
|
35
|
+
|
|
36
|
+
clean: ## Remove build and cache artefacts
|
|
37
|
+
rm -rf dist build .pytest_cache .mypy_cache .ruff_cache htmlcov .coverage .coverage.*
|
|
38
|
+
find . -name __pycache__ -type d -prune -exec rm -rf {} +
|
|
@@ -0,0 +1,448 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: quarantine-py
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: When one bad item crashes your loop of 10,000 - don't crash. Set it aside, keep going, fix it later.
|
|
5
|
+
Project-URL: Homepage, https://github.com/quarantine-py/quarantine
|
|
6
|
+
Project-URL: Source, https://github.com/quarantine-py/quarantine
|
|
7
|
+
Project-URL: Issues, https://github.com/quarantine-py/quarantine/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/quarantine-py/quarantine/blob/main/CHANGELOG.md
|
|
9
|
+
Author: quarantine contributors
|
|
10
|
+
License: MIT License
|
|
11
|
+
|
|
12
|
+
Copyright (c) 2026 quarantine contributors
|
|
13
|
+
|
|
14
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
15
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
16
|
+
in the Software without restriction, including without limitation the rights
|
|
17
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
18
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
19
|
+
furnished to do so, subject to the following conditions:
|
|
20
|
+
|
|
21
|
+
The above copyright notice and this permission notice shall be included in all
|
|
22
|
+
copies or substantial portions of the Software.
|
|
23
|
+
|
|
24
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
25
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
26
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
27
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
28
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
29
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
30
|
+
SOFTWARE.
|
|
31
|
+
License-File: LICENSE
|
|
32
|
+
Keywords: batch,dead-letter-queue,debugging,error-handling,etl,resilience,retry
|
|
33
|
+
Classifier: Development Status :: 4 - Beta
|
|
34
|
+
Classifier: Environment :: Console
|
|
35
|
+
Classifier: Intended Audience :: Developers
|
|
36
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
37
|
+
Classifier: Operating System :: OS Independent
|
|
38
|
+
Classifier: Programming Language :: Python :: 3
|
|
39
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
40
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
41
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
42
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
43
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
44
|
+
Classifier: Topic :: Software Development :: Debuggers
|
|
45
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
46
|
+
Classifier: Typing :: Typed
|
|
47
|
+
Requires-Python: >=3.10
|
|
48
|
+
Provides-Extra: dev
|
|
49
|
+
Requires-Dist: mypy>=1.11; extra == 'dev'
|
|
50
|
+
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
|
|
51
|
+
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
|
|
52
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
53
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
54
|
+
Description-Content-Type: text/markdown
|
|
55
|
+
|
|
56
|
+
# 🏥 quarantine
|
|
57
|
+
|
|
58
|
+
**When one bad item crashes your loop of 10,000 — don't crash. Set it aside, keep going, fix it later.**
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
pip install quarantine-py
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
```python
|
|
65
|
+
from quarantine import quarantine
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
@quarantine
|
|
69
|
+
def process(item): ... # your normal code, unchanged
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
for item in items:
|
|
73
|
+
process(item)
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
That's it. Bad items no longer kill your job. They get saved to a `.quarantine/` folder — with their full error and the exact input that caused it — and your loop keeps running.
|
|
77
|
+
|
|
78
|
+
At the end:
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
✓ 9,996 processed · ✗ 4 quarantined → .quarantine/ (run `quarantine retry` after fixing)
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## The problem (explained like you're new to this)
|
|
87
|
+
|
|
88
|
+
Imagine you're processing 10,000 records — rows from a CSV, URLs to scrape, images to resize. You write a loop:
|
|
89
|
+
|
|
90
|
+
```python
|
|
91
|
+
for item in items:
|
|
92
|
+
process(item)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
You run it. It works... until **item #5,247**, which is malformed in some way you didn't expect. Your script crashes. You lost 5,246 items of finished work and 3 hours.
|
|
96
|
+
|
|
97
|
+
So you fix the bug for that one weird item and rerun from the start. Three hours later it crashes again — at item #7,913, for a *different* reason.
|
|
98
|
+
|
|
99
|
+
This cycle is one of the most common, most painful experiences in programming. `quarantine` ends it.
|
|
100
|
+
|
|
101
|
+
## "Can't I just use try/except?"
|
|
102
|
+
|
|
103
|
+
Yes — try/except is exactly what quarantine uses under the hood. But here's the try/except version once you make it actually safe:
|
|
104
|
+
|
|
105
|
+
```python
|
|
106
|
+
failed = []
|
|
107
|
+
for item in items:
|
|
108
|
+
try:
|
|
109
|
+
process(item)
|
|
110
|
+
except Exception as e:
|
|
111
|
+
failed.append(item) # ❌ lost forever if the script dies later
|
|
112
|
+
print(f"failed: {e}") # ❌ traceback gone — good luck debugging tomorrow
|
|
113
|
+
# ❌ how do I re-run JUST these failures after I fix the bug?
|
|
114
|
+
# ❌ what if 500 fail in a row because the API is down — keep going?!
|
|
115
|
+
# ❌ how do I save a weird object (DataFrame row? bytes?) to look at later?
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Every ❌ is a real problem you'd have to solve yourself, in every script, forever. quarantine solves them once:
|
|
119
|
+
|
|
120
|
+
| | `try/except` by hand | `@quarantine` |
|
|
121
|
+
|---|---|---|
|
|
122
|
+
| Loop survives bad items | ✅ | ✅ |
|
|
123
|
+
| Failures survive a crash/restart | ❌ in RAM, gone | ✅ saved to disk instantly |
|
|
124
|
+
| Full traceback kept for later | ❌ usually just printed | ✅ stored with the item |
|
|
125
|
+
| The exact bad input saved | ❌ you'd have to serialize it | ✅ automatic |
|
|
126
|
+
| Re-run *only* the failures | ❌ build it yourself | ✅ `quarantine retry` |
|
|
127
|
+
| Debug with the real bad input | ❌ archaeology in logs | ✅ `quarantine debug 2` |
|
|
128
|
+
| Detects "everything is failing, stop" | ❌ | ✅ halts on failure streaks |
|
|
129
|
+
| Skips already-known-bad items on rerun | ❌ | ✅ |
|
|
130
|
+
|
|
131
|
+
**quarantine is not a replacement for try/except. It's the 200 lines of bookkeeping you'd have to write around it — done correctly, once.**
|
|
132
|
+
|
|
133
|
+
## The idea in one picture
|
|
134
|
+
|
|
135
|
+
Hospitals don't shut down when one patient has an infection. They **quarantine** the patient, treat everyone else, and come back with the right medicine.
|
|
136
|
+
|
|
137
|
+
```
|
|
138
|
+
items ──▶ process() ──▶ ✓ done
|
|
139
|
+
│
|
|
140
|
+
✗ raises an exception
|
|
141
|
+
▼
|
|
142
|
+
.quarantine/ ◀── the "sick bay" folder
|
|
143
|
+
├── item + its data
|
|
144
|
+
├── the full error traceback
|
|
145
|
+
└── when/why it failed
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Your job finishes. The sick items wait for you, with their full medical charts.
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
## Installation
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
pip install quarantine-py
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
That is the whole install story. Some alternatives, if you prefer:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
uv add quarantine-py # uv projects
|
|
162
|
+
uv pip install quarantine-py # uv, without a project
|
|
163
|
+
python -m pip install --user quarantine
|
|
164
|
+
pip install git+https://github.com/quarantine-py/quarantine # unreleased main
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
**Requirements**
|
|
168
|
+
|
|
169
|
+
| | |
|
|
170
|
+
|---|---|
|
|
171
|
+
| Python | 3.10 or newer (CPython; tested on 3.10 - 3.13) |
|
|
172
|
+
| Runtime dependencies | none - it is standard library only |
|
|
173
|
+
| Operating systems | Linux, macOS, Windows (tested on all three in CI) |
|
|
174
|
+
|
|
175
|
+
Installing also puts a `quarantine` command on your `PATH`. Check both halves:
|
|
176
|
+
|
|
177
|
+
```bash
|
|
178
|
+
$ quarantine --version
|
|
179
|
+
quarantine 0.1.0
|
|
180
|
+
$ python -c "import quarantine; print(quarantine.__version__)"
|
|
181
|
+
0.1.0
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
If the command is not found (a common `--user` install wrinkle), the module
|
|
185
|
+
form always works and takes the same arguments:
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
python -m quarantine list
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Nothing needs configuring. The first time a call fails, `.quarantine/` appears
|
|
192
|
+
next to wherever you started Python. Add it to your `.gitignore` -
|
|
193
|
+
**quarantined inputs are real data, and real data does not belong in git.**
|
|
194
|
+
|
|
195
|
+
```gitignore
|
|
196
|
+
.quarantine/
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Usage
|
|
202
|
+
|
|
203
|
+
### 1. Basic — decorate and forget
|
|
204
|
+
|
|
205
|
+
```python
|
|
206
|
+
from quarantine import quarantine
|
|
207
|
+
|
|
208
|
+
|
|
209
|
+
@quarantine
|
|
210
|
+
def process(row):
|
|
211
|
+
price = float(row["price"]) # crashes on "N/A"? quarantined.
|
|
212
|
+
save_to_db(row["id"], price)
|
|
213
|
+
|
|
214
|
+
|
|
215
|
+
for row in rows:
|
|
216
|
+
process(row)
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### 2. See what got quarantined
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
$ quarantine list
|
|
223
|
+
# when function error input preview
|
|
224
|
+
1 09:14:02 process ValueError: could not convert {'id': 8812, 'price': 'N/A', ...}
|
|
225
|
+
2 09:31:44 process KeyError: 'price' {'id': 9107, ...}
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
### 3. Fix your code, then retry only the failures
|
|
229
|
+
|
|
230
|
+
```bash
|
|
231
|
+
$ quarantine retry
|
|
232
|
+
✓ 3 recovered · ✗ 1 still failing (kept in quarantine)
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
No rerunning the 9,996 items that already worked.
|
|
236
|
+
|
|
237
|
+
*(Note: If your function lives in a standalone script rather than an installed package, use `-i` to tell quarantine where to import it from: `quarantine retry -i my_script.py`)*
|
|
238
|
+
|
|
239
|
+
### 4. Debug with the actual bad input
|
|
240
|
+
|
|
241
|
+
```bash
|
|
242
|
+
$ quarantine debug 2
|
|
243
|
+
# opens a Python debugger with `item` set to the exact input that failed
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
*(You can use `-i` here too: `quarantine debug 2 -i my_script.py`)*
|
|
247
|
+
|
|
248
|
+
The single biggest time-saver: you never have to *reproduce* the bug. The bug's exact input is sitting on disk.
|
|
249
|
+
|
|
250
|
+
### 5. Safety valve — when it's not the data's fault
|
|
251
|
+
|
|
252
|
+
If 50 items fail **in a row**, that's not bad data — that's your database being down. Quarantining 10,000 items would be silly. quarantine halts instead:
|
|
253
|
+
|
|
254
|
+
```
|
|
255
|
+
✋ 50 consecutive failures — this looks systemic, not bad data. Halting.
|
|
256
|
+
Last error: ConnectionError: db.internal:5432 refused
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Tune it: `@quarantine(halt_after=100)`.
|
|
260
|
+
|
|
261
|
+
### 6. Options (all optional)
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
@quarantine(
|
|
265
|
+
dir=".quarantine", # where the sick bay lives
|
|
266
|
+
only=(ValueError, KeyError), # only quarantine these; others still crash
|
|
267
|
+
halt_after=50, # consecutive-failure circuit breaker
|
|
268
|
+
max_items=10_000, # cap disk usage
|
|
269
|
+
redact=["api_key", "password"], # scrub these fields before saving inputs
|
|
270
|
+
on_quarantine=my_alert_fn, # e.g., send a Slack ping
|
|
271
|
+
)
|
|
272
|
+
def process(item): ...
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
### 7. Works on loops too, without a decorator
|
|
276
|
+
|
|
277
|
+
```python
|
|
278
|
+
from quarantine import shield
|
|
279
|
+
|
|
280
|
+
for item in shield(items, using=process):
|
|
281
|
+
...
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
---
|
|
285
|
+
|
|
286
|
+
## What's in the `.quarantine/` folder?
|
|
287
|
+
|
|
288
|
+
Plain files. No database, no magic — you can inspect everything yourself:
|
|
289
|
+
|
|
290
|
+
```
|
|
291
|
+
.quarantine/
|
|
292
|
+
├── 0001/
|
|
293
|
+
│ ├── input.pkl # the exact item (pickle, JSON fallback for simple data)
|
|
294
|
+
│ ├── input.txt # human-readable repr, so you can just *look* at it
|
|
295
|
+
│ ├── traceback.txt # full error, exactly as it would have printed
|
|
296
|
+
│ └── meta.json # function name, timestamp, attempt count, python/pkg versions
|
|
297
|
+
└── index.json
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
Design rules:
|
|
301
|
+
- **Atomic writes** — a crash mid-save never corrupts the folder.
|
|
302
|
+
- **Redaction before disk** — fields you mark as secret never touch the filesystem.
|
|
303
|
+
- **Dedup on rerun** — an item already in quarantine is skipped (no log spam), unless you `quarantine retry` it.
|
|
304
|
+
- **Serialization fallbacks** — pickle → JSON → repr. Something readable is *always* saved, even for exotic objects.
|
|
305
|
+
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
---
|
|
309
|
+
|
|
310
|
+
## Command-line reference
|
|
311
|
+
|
|
312
|
+
Every command takes `-d/--dir PATH` (default: `$QUARANTINE_DIR`, else
|
|
313
|
+
`./.quarantine`), and `list`, `show`, `retry` and `stats` all take `--json` so
|
|
314
|
+
you can pipe them somewhere useful.
|
|
315
|
+
|
|
316
|
+
| Command | What it does |
|
|
317
|
+
|---|---|
|
|
318
|
+
| `quarantine list` | Table of everything quarantined. `ls` works too. `-f/--function NAME`, `-n/--limit N`. |
|
|
319
|
+
| `quarantine show ID [ID...]` | One record in full: metadata, the input, the whole traceback. |
|
|
320
|
+
| `quarantine retry [ID...]` | Re-run records; delete the ones that now succeed. `-f/--function NAME`, `--dry-run`, `-i/--import FILE.py` (for functions that live in a script). |
|
|
321
|
+
| `quarantine debug ID` | Re-run one record and drop you into `pdb` **on the frame that raised**. `-p/--print` to just dump it, `--no-post-mortem` to skip re-running and get the input in scope, `-i/--import FILE.py` as above. |
|
|
322
|
+
| `quarantine clear [ID...]` | Delete records. With no ids it clears everything and asks first; `-y/--yes` skips the prompt. `rm` works too. |
|
|
323
|
+
| `quarantine stats` | Counts by function and by error type, plus how much disk the folder is using. |
|
|
324
|
+
| `quarantine reindex` | Rebuild `index.json` from the record folders and sweep up leftover temp files from a hard crash. |
|
|
325
|
+
|
|
326
|
+
Exit codes, for scripts and CI:
|
|
327
|
+
|
|
328
|
+
| Code | Meaning |
|
|
329
|
+
|---|---|
|
|
330
|
+
| `0` | Everything you asked for succeeded. |
|
|
331
|
+
| `1` | The command ran, but something is still wrong - a retry failed again, or a record could not be replayed. |
|
|
332
|
+
| `2` | Bad usage, or the folder could not be read. |
|
|
333
|
+
|
|
334
|
+
```bash
|
|
335
|
+
# fail a nightly job if anything is still sitting in quarantine
|
|
336
|
+
quarantine retry || echo "still broken - look at: $(quarantine list -n 3)"
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
## Python API reference
|
|
340
|
+
|
|
341
|
+
```python
|
|
342
|
+
from quarantine import quarantine, shield, Quarantine, QUARANTINED, records, retry
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
**Decorating and looping**
|
|
346
|
+
|
|
347
|
+
| | |
|
|
348
|
+
|---|---|
|
|
349
|
+
| `@quarantine` / `@quarantine(...)` | Wrap one function. Options are listed under [Options](#6-options-all-optional). Works on `async def`. |
|
|
350
|
+
| `shield(items, using=fn, **options)` | Iterator yielding only the results that worked. |
|
|
351
|
+
| `ashield(items, using=fn, **options)` | Same, for `async def` work and/or async iterables. |
|
|
352
|
+
|
|
353
|
+
**Return values.** A quarantined call returns the `QUARANTINED` sentinel; an
|
|
354
|
+
input recognised as already-bad returns `SKIPPED`. Both are falsy, so
|
|
355
|
+
`if process(item):` does the sensible thing. Use `is_quarantined(result)` /
|
|
356
|
+
`is_skipped(result)` when you want to be explicit.
|
|
357
|
+
|
|
358
|
+
**The explicit object**, when you would rather pass something around than rely
|
|
359
|
+
on a decorator:
|
|
360
|
+
|
|
361
|
+
```python
|
|
362
|
+
from quarantine import Quarantine
|
|
363
|
+
|
|
364
|
+
q = Quarantine("build/bad-rows", halt_after=10, redact=["api_key"])
|
|
365
|
+
|
|
366
|
+
safe = q.wrap(process) # same as the decorator
|
|
367
|
+
q.call(process, item) # one-off call, same protection
|
|
368
|
+
await q.acall(fetch, url) # async one-off
|
|
369
|
+
q.records() # list[Record], oldest first
|
|
370
|
+
q.retry() # -> RetryResult(recovered, still_failing, unretryable)
|
|
371
|
+
await q.aretry() # for records from async functions
|
|
372
|
+
q.clear() # empty the folder
|
|
373
|
+
q.stats # Stats(processed, quarantined, skipped, recovered)
|
|
374
|
+
q.summary_line() # the one-line report, or None
|
|
375
|
+
len(q), list(q) # how many records; iterate them
|
|
376
|
+
```
|
|
377
|
+
|
|
378
|
+
**Module-level shortcuts** operate on the default folder (or `dir=`):
|
|
379
|
+
`records()`, `retry()`, `aretry()`, `clear()`, `summary()`.
|
|
380
|
+
|
|
381
|
+
**A `Record`** is what you get back from `records()`:
|
|
382
|
+
|
|
383
|
+
```python
|
|
384
|
+
record = records()[0]
|
|
385
|
+
record.id # 1
|
|
386
|
+
record.function # "process" (qualified name in .qualified_name)
|
|
387
|
+
record.error_type # "ValueError"
|
|
388
|
+
record.summary # "ValueError: could not convert string to float: 'N/A'"
|
|
389
|
+
record.attempts # 2, after one retry
|
|
390
|
+
record.redacted # ["api_key"] - what was scrubbed
|
|
391
|
+
record.path # Path(".quarantine/0001")
|
|
392
|
+
record.traceback_text() # the stored traceback
|
|
393
|
+
record.load_call() # Call(args=({...},), kwargs={}) - the original input
|
|
394
|
+
record.load_call().item # the item itself
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
**Exceptions** (all subclass `QuarantineError`, and none of them are ever
|
|
398
|
+
quarantined themselves):
|
|
399
|
+
|
|
400
|
+
| | |
|
|
401
|
+
|---|---|
|
|
402
|
+
| `SystemicFailure` | The `halt_after` circuit breaker tripped. `.count`, `.last_error`. |
|
|
403
|
+
| `QuarantineFull` | `max_items` reached. Nothing is dropped silently - this is raised instead, chained from the original error. |
|
|
404
|
+
| `StorageError` | The folder could not be read or written. |
|
|
405
|
+
| `ResolutionError` | A retry could not import the function a record came from. |
|
|
406
|
+
|
|
407
|
+
**Environment**
|
|
408
|
+
|
|
409
|
+
| | |
|
|
410
|
+
|---|---|
|
|
411
|
+
| `QUARANTINE_DIR` | Default folder for both the library and the CLI. |
|
|
412
|
+
|
|
413
|
+
## Documentation
|
|
414
|
+
|
|
415
|
+
| | |
|
|
416
|
+
|---|---|
|
|
417
|
+
| [docs/installation.md](docs/installation.md) | Installing, verifying, upgrading, uninstalling. |
|
|
418
|
+
| [docs/usage.md](docs/usage.md) | The full guide: options, async, threads, retry loops, alerting, recipes. |
|
|
419
|
+
| [docs/cli.md](docs/cli.md) | Every command, flag and exit code, with output samples. |
|
|
420
|
+
| [docs/api.md](docs/api.md) | Complete Python API reference. |
|
|
421
|
+
| [docs/on-disk-format.md](docs/on-disk-format.md) | What is in `.quarantine/`, and the guarantees about it. |
|
|
422
|
+
| [docs/troubleshooting.md](docs/troubleshooting.md) | "It skipped my item", "retry says it cannot import", and friends. |
|
|
423
|
+
| [docs/faq.md](docs/faq.md) | Longer answers to the questions below. |
|
|
424
|
+
| [CHANGELOG.md](CHANGELOG.md) | What changed, and when. |
|
|
425
|
+
| [CONTRIBUTING.md](CONTRIBUTING.md) | Setup, the checks, and the ground rules for changes. |
|
|
426
|
+
|
|
427
|
+
## When NOT to use quarantine
|
|
428
|
+
|
|
429
|
+
Honesty section:
|
|
430
|
+
|
|
431
|
+
- **You want the crash.** In a bank transfer pipeline, stopping on the first error might be correct. Silently continuing is a choice — make it deliberately.
|
|
432
|
+
- **You're already on Celery/Kafka/Airflow.** Those have dead-letter queues; use them. quarantine is for the 95% of scripts that will never justify that machinery.
|
|
433
|
+
- **Failures are expected and normal** (e.g., "404 means skip"). Handle those with a normal `if`/`except` — quarantine is for *unexpected* failures you'll want to investigate.
|
|
434
|
+
|
|
435
|
+
## FAQ
|
|
436
|
+
|
|
437
|
+
**Is this just a dead-letter queue?**
|
|
438
|
+
Yes — that's exactly the pattern, ported from message-queue infrastructure to a plain Python for-loop. No broker, no server, one decorator.
|
|
439
|
+
|
|
440
|
+
**Async?**
|
|
441
|
+
`@quarantine` works on `async def` too.
|
|
442
|
+
|
|
443
|
+
**Threads/processes?**
|
|
444
|
+
Yes — writes are atomic and the folder is append-only per item.
|
|
445
|
+
|
|
446
|
+
---
|
|
447
|
+
|
|
448
|
+
*Made for everyone whose overnight job died at item 5,247.*
|