pra-effect 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pra_effect-1.0.0/LICENSE +21 -0
- pra_effect-1.0.0/PKG-INFO +216 -0
- pra_effect-1.0.0/README.md +190 -0
- pra_effect-1.0.0/pra_effect.egg-info/PKG-INFO +216 -0
- pra_effect-1.0.0/pra_effect.egg-info/SOURCES.txt +9 -0
- pra_effect-1.0.0/pra_effect.egg-info/dependency_links.txt +1 -0
- pra_effect-1.0.0/pra_effect.egg-info/entry_points.txt +2 -0
- pra_effect-1.0.0/pra_effect.egg-info/top_level.txt +1 -0
- pra_effect-1.0.0/pra_effect.py +1342 -0
- pra_effect-1.0.0/pyproject.toml +38 -0
- pra_effect-1.0.0/setup.cfg +4 -0
pra_effect-1.0.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Genesis410
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pra-effect
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Zero-dependency local CLI for parsing, deduplicating, and fingerprinting Python & Node.js error stack traces.
|
|
5
|
+
Author: Genesis410
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Genesis410/pra-effect
|
|
8
|
+
Project-URL: Repository, https://github.com/Genesis410/pra-effect.git
|
|
9
|
+
Project-URL: Issues, https://github.com/Genesis410/pra-effect/issues
|
|
10
|
+
Keywords: stacktrace,deduplication,cli,incident-management,logging,developer-tools
|
|
11
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Software Development :: Debuggers
|
|
21
|
+
Classifier: Topic :: System :: Logging
|
|
22
|
+
Requires-Python: >=3.9
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Dynamic: license-file
|
|
26
|
+
|
|
27
|
+
# Pra-Effect
|
|
28
|
+
|
|
29
|
+
> Zero-dependency local CLI for parsing, deduplicating, and fingerprinting Python & Node.js error stack traces with low-memory streaming ingestion ($O(1)$ reader, $O(G)$ unique-group memory), ReDoS resilience, SQLite persistence, and standalone HTML reporting.
|
|
30
|
+
|
|
31
|
+
[](https://www.python.org/)
|
|
32
|
+
[](LICENSE)
|
|
33
|
+
[]()
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Highlights
|
|
38
|
+
|
|
39
|
+
- **Zero Third-Party Dependencies:** Pure Python standard library (`argparse`, `hashlib`, `html`, `re`, `sqlite3`, `pathlib`).
|
|
40
|
+
- **Resilient Fingerprinting (`fingerprint_v1`):** Normalizes dynamic memory addresses (`0x7f...`), variable line numbers, timestamps, UUIDs, and ephemeral ports so identical incidents aggregate into one canonical hash.
|
|
41
|
+
- **Dual Runtime Support:** Natively parses Python tracebacks (standard, chained exceptions, multiline messages) and Node.js call stacks (V8 engine).
|
|
42
|
+
- **Robust Input Handling:**
|
|
43
|
+
- Hard line truncation at 4,096 characters to prevent catastrophic regex backtracking (ReDoS).
|
|
44
|
+
- Interleaving tolerance up to 2 lines of interleaved non-trace log noise.
|
|
45
|
+
- Bounded message collection: log lines (`INFO ...`, timestamps) never leak into an error message.
|
|
46
|
+
- Chained Python exceptions collapse to a single root incident (causes kept as diagnostic context, never split the fingerprint).
|
|
47
|
+
- **Streaming Architecture & Performance:**
|
|
48
|
+
- Input parsed line-by-line through state machines — reader memory footprint is $O(1)$ (0.0047 MB tracemalloc for pure stream ingestion).
|
|
49
|
+
- Total memory scales with the number of *unique incident groups* $O(G)$, not total input byte size ($G$ distinct error patterns retained in memory).
|
|
50
|
+
- Benchmark (Python 3.14 on Windows, synthetic multi-group logs):
|
|
51
|
+
- *Typical log stream (28.2 MB, ~72k traces, ~6.8k unique groups):* ~6.1 MB Python heap, ~36.5 MB Peak RSS (GATE-MEM-01B PASS).
|
|
52
|
+
- *Adversarial high-cardinality stream (21.4 MB, ~54k traces, ~44.7k unique groups):* ~39.8 MB Python heap, ~106 MB Peak RSS (characterized in DEC-017).
|
|
53
|
+
- Optional `--max-input-mb` backstop rejects oversized raw log files (default 100 MB, 0 disables).
|
|
54
|
+
- **Local Persistence & Export:**
|
|
55
|
+
- WAL-mode SQLite storage for thread-safe concurrent writes and query performance.
|
|
56
|
+
- Self-contained, single-file HTML reports with zero external CDN assets.
|
|
57
|
+
- **Semantic Exit Codes:** POSIX standard compliance (`0` = success/analysis complete, `1` = fatal I/O, `2` = invalid CLI invocation).
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Installation
|
|
62
|
+
|
|
63
|
+
### From Source (Editable / Tool Mode)
|
|
64
|
+
```bash
|
|
65
|
+
git clone https://github.com/Genesis410/pra-effect.git
|
|
66
|
+
cd pra-effect
|
|
67
|
+
pip install -e .
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Standalone (Zero Install)
|
|
71
|
+
You can directly invoke the script without installation:
|
|
72
|
+
```bash
|
|
73
|
+
python pra_effect.py --help
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## CLI Usage
|
|
79
|
+
|
|
80
|
+
Pra-Effect provides three primary commands: `ingest`, `scan`, and `history`.
|
|
81
|
+
|
|
82
|
+
### 1. Ingest via STDIN (Pipe Mode)
|
|
83
|
+
Pipe stdout or stderr from test runners, logs, or services directly into Pra-Effect:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# Direct pipe from a failing script
|
|
87
|
+
python app.py 2>&1 | pra-effect ingest
|
|
88
|
+
|
|
89
|
+
# Save run to SQLite and export an HTML dashboard
|
|
90
|
+
pytest 2>&1 | pra-effect ingest --save --html report.html
|
|
91
|
+
|
|
92
|
+
# Check installed version (must match pyproject + git tag)
|
|
93
|
+
pra-effect --version
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### 2. Scan Existing Log Files
|
|
97
|
+
Analyze log dumps or application error logs on disk — a single file or a whole directory:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# Explicit file (must be readable, otherwise exit 1)
|
|
101
|
+
pra-effect scan /var/log/application.log --save --html report.html
|
|
102
|
+
|
|
103
|
+
# Directory scan: default pattern *.log,*.txt, comma-separated globs supported
|
|
104
|
+
pra-effect scan /var/log --pattern "*.log,*.txt,*.out"
|
|
105
|
+
|
|
106
|
+
# Files that cannot be read (binary/unreadable) are skipped with a warning on stderr.
|
|
107
|
+
# Exit 1 if zero readable files matched — never a silent "0 errors found".
|
|
108
|
+
|
|
109
|
+
# Input-size backstop (default 100 MB, streaming already bounds the pipeline; 0 disables)
|
|
110
|
+
pra-effect scan /var/log/huge --max-input-mb 250
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### 3. Historical Incident Aggregation
|
|
114
|
+
Query grouped incident metrics stored in your local SQLite database:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
pra-effect history
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Example Output:
|
|
121
|
+
```text
|
|
122
|
+
=== Pra-Effect Historical Incident Groups ===
|
|
123
|
+
Fingerprint: f9e160e1d88bbca1 (v1)
|
|
124
|
+
Runtime: python
|
|
125
|
+
Error Type: ValueError
|
|
126
|
+
Message: bad <script>alert(1)</script>
|
|
127
|
+
Occurrences: 3 total across 2 runs
|
|
128
|
+
First Seen: 2026-10-24 16:30:00 UTC
|
|
129
|
+
Last Seen: 2026-10-24 16:45:12 UTC
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Architecture & Data Flow
|
|
135
|
+
|
|
136
|
+
```text
|
|
137
|
+
Log Stream (STDIN / File)
|
|
138
|
+
│
|
|
139
|
+
▼
|
|
140
|
+
Pre-check Sanitizer (4,096 char cutoff, ReDoS guard)
|
|
141
|
+
│
|
|
142
|
+
▼
|
|
143
|
+
Grammar Parsers (Python Tracebacks / Node.js V8 Stacks)
|
|
144
|
+
│
|
|
145
|
+
▼
|
|
146
|
+
Message Normalizer (UUID / hex / ID collapse — not PII redaction)
|
|
147
|
+
│
|
|
148
|
+
▼
|
|
149
|
+
Fingerprint Engine (SHA-256 top-frame + message -> 16-hex ID)
|
|
150
|
+
│
|
|
151
|
+
├─────────────────────────┬─────────────────────────┐
|
|
152
|
+
▼ ▼ ▼
|
|
153
|
+
Terminal ANSI Output SQLite Storage (WAL) HTML Dashboard
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
|
|
158
|
+
## Database & Schema
|
|
159
|
+
|
|
160
|
+
When `--save` is passed, runs are stored in `./.pra-effect/traces.db` (override via `--db <path>`):
|
|
161
|
+
|
|
162
|
+
- `runs`: Metadata per execution (duration, line counts, unique group count).
|
|
163
|
+
- `incidents`: Normalized error incidents linked by foreign keys, tracking `is_truncated`, occurrence counts, and project frames.
|
|
164
|
+
- `v_historical_incidents`: High-performance view aggregating cross-run recurrence.
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Testing
|
|
169
|
+
|
|
170
|
+
Run the included verification suite (standard `unittest` discovery, works from any directory):
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
python -m unittest discover -s . -p "test_*.py"
|
|
174
|
+
# or
|
|
175
|
+
python test_pra_effect.py
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Memory benchmarks for release gates (`GATE-MEM-01A` streaming pipeline & `GATE-MEM-01B` realistic workload):
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
python bench_memory.py # exit 0 = PASS (<= 50 MB Peak RSS), 1 = gate exceeded, 2 = RSS NOT_RUN
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Benchmark & Performance Gates:
|
|
185
|
+
- **GATE-MEM-01A (Pipeline Ingestion Stream)**: Peak RSS $\le$ 35 MB, tracemalloc $<$ 1 MB. Empirically measured **24.36 MB Peak RSS** and **0.0047 MB tracemalloc** across 54,000+ tracebacks ($O(1)$ streaming ingestion).
|
|
186
|
+
- **GATE-MEM-01B (Realistic Workload End-to-End)**: Peak RSS $\le$ 50 MB on files $\ge$ 25 MB with realistic deduplication ($\le$ 10,000 unique groups). Empirically measured **36.50 MB Peak RSS** and **6.12 MB tracemalloc** on 28.2 MB / 72,316 tracebacks.
|
|
187
|
+
- **High-Cardinality Boundary**: In-memory grouping memory scales linearly $O(G)$ with unique error groups (~963 bytes per group with complete `representative_trace`). Peak RSS 50 MB is not guaranteed for adversarial logs where every single error is unique. `--max-input-mb` (default 100 MB) provides raw input byte capping.
|
|
188
|
+
|
|
189
|
+
Tests cover:
|
|
190
|
+
- Python & Node.js grouping and line drift resilience.
|
|
191
|
+
- Message normalization and hex address masking.
|
|
192
|
+
- ReDoS guard and bounded multiline collection.
|
|
193
|
+
- Interleaved non-trace log line tolerance.
|
|
194
|
+
- SQLite WAL mode and concurrent transaction safety.
|
|
195
|
+
- Single-file HTML generation with full XSS escaping on every rendered field, plus a PII notice (normalization is not redaction).
|
|
196
|
+
- Chained exception collapse, bounded message collector, directory scan `--pattern` with skip/exit semantics.
|
|
197
|
+
- Fingerprint version isolation in history queries (`FINGERPRINT_SPEC.md`).
|
|
198
|
+
- CLI exit codes (0, 1, 2).
|
|
199
|
+
- Streaming pipeline memory regression ($O(1)$ constant memory ingestion).
|
|
200
|
+
- Raw input file cap backstop (`--max-input-mb`).
|
|
201
|
+
- Golden CLI output parity (byte-identical across refactors).
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
## License
|
|
206
|
+
|
|
207
|
+
MIT License. See [LICENSE](LICENSE) for details.
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## Release & Changelog
|
|
212
|
+
|
|
213
|
+
See [CHANGELOG.md](CHANGELOG.md) for the version history. Current release: `1.0.0`.
|
|
214
|
+
|
|
215
|
+
Version consistency across the package (`pyproject.toml`), the CLI (`--version`), and
|
|
216
|
+
the git tag is enforced by `test_release_version_consistency`.
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
# Pra-Effect
|
|
2
|
+
|
|
3
|
+
> Zero-dependency local CLI for parsing, deduplicating, and fingerprinting Python & Node.js error stack traces with low-memory streaming ingestion ($O(1)$ reader, $O(G)$ unique-group memory), ReDoS resilience, SQLite persistence, and standalone HTML reporting.
|
|
4
|
+
|
|
5
|
+
[](https://www.python.org/)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
[]()
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Highlights
|
|
12
|
+
|
|
13
|
+
- **Zero Third-Party Dependencies:** Pure Python standard library (`argparse`, `hashlib`, `html`, `re`, `sqlite3`, `pathlib`).
|
|
14
|
+
- **Resilient Fingerprinting (`fingerprint_v1`):** Normalizes dynamic memory addresses (`0x7f...`), variable line numbers, timestamps, UUIDs, and ephemeral ports so identical incidents aggregate into one canonical hash.
|
|
15
|
+
- **Dual Runtime Support:** Natively parses Python tracebacks (standard, chained exceptions, multiline messages) and Node.js call stacks (V8 engine).
|
|
16
|
+
- **Robust Input Handling:**
|
|
17
|
+
- Hard line truncation at 4,096 characters to prevent catastrophic regex backtracking (ReDoS).
|
|
18
|
+
- Interleaving tolerance up to 2 lines of interleaved non-trace log noise.
|
|
19
|
+
- Bounded message collection: log lines (`INFO ...`, timestamps) never leak into an error message.
|
|
20
|
+
- Chained Python exceptions collapse to a single root incident (causes kept as diagnostic context, never split the fingerprint).
|
|
21
|
+
- **Streaming Architecture & Performance:**
|
|
22
|
+
- Input parsed line-by-line through state machines — reader memory footprint is $O(1)$ (0.0047 MB tracemalloc for pure stream ingestion).
|
|
23
|
+
- Total memory scales with the number of *unique incident groups* $O(G)$, not total input byte size ($G$ distinct error patterns retained in memory).
|
|
24
|
+
- Benchmark (Python 3.14 on Windows, synthetic multi-group logs):
|
|
25
|
+
- *Typical log stream (28.2 MB, ~72k traces, ~6.8k unique groups):* ~6.1 MB Python heap, ~36.5 MB Peak RSS (GATE-MEM-01B PASS).
|
|
26
|
+
- *Adversarial high-cardinality stream (21.4 MB, ~54k traces, ~44.7k unique groups):* ~39.8 MB Python heap, ~106 MB Peak RSS (characterized in DEC-017).
|
|
27
|
+
- Optional `--max-input-mb` backstop rejects oversized raw log files (default 100 MB, 0 disables).
|
|
28
|
+
- **Local Persistence & Export:**
|
|
29
|
+
- WAL-mode SQLite storage for thread-safe concurrent writes and query performance.
|
|
30
|
+
- Self-contained, single-file HTML reports with zero external CDN assets.
|
|
31
|
+
- **Semantic Exit Codes:** POSIX standard compliance (`0` = success/analysis complete, `1` = fatal I/O, `2` = invalid CLI invocation).
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Installation
|
|
36
|
+
|
|
37
|
+
### From Source (Editable / Tool Mode)
|
|
38
|
+
```bash
|
|
39
|
+
git clone https://github.com/Genesis410/pra-effect.git
|
|
40
|
+
cd pra-effect
|
|
41
|
+
pip install -e .
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
### Standalone (Zero Install)
|
|
45
|
+
You can directly invoke the script without installation:
|
|
46
|
+
```bash
|
|
47
|
+
python pra_effect.py --help
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## CLI Usage
|
|
53
|
+
|
|
54
|
+
Pra-Effect provides three primary commands: `ingest`, `scan`, and `history`.
|
|
55
|
+
|
|
56
|
+
### 1. Ingest via STDIN (Pipe Mode)
|
|
57
|
+
Pipe stdout or stderr from test runners, logs, or services directly into Pra-Effect:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
# Direct pipe from a failing script
|
|
61
|
+
python app.py 2>&1 | pra-effect ingest
|
|
62
|
+
|
|
63
|
+
# Save run to SQLite and export an HTML dashboard
|
|
64
|
+
pytest 2>&1 | pra-effect ingest --save --html report.html
|
|
65
|
+
|
|
66
|
+
# Check installed version (must match pyproject + git tag)
|
|
67
|
+
pra-effect --version
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### 2. Scan Existing Log Files
|
|
71
|
+
Analyze log dumps or application error logs on disk — a single file or a whole directory:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
# Explicit file (must be readable, otherwise exit 1)
|
|
75
|
+
pra-effect scan /var/log/application.log --save --html report.html
|
|
76
|
+
|
|
77
|
+
# Directory scan: default pattern *.log,*.txt, comma-separated globs supported
|
|
78
|
+
pra-effect scan /var/log --pattern "*.log,*.txt,*.out"
|
|
79
|
+
|
|
80
|
+
# Files that cannot be read (binary/unreadable) are skipped with a warning on stderr.
|
|
81
|
+
# Exit 1 if zero readable files matched — never a silent "0 errors found".
|
|
82
|
+
|
|
83
|
+
# Input-size backstop (default 100 MB, streaming already bounds the pipeline; 0 disables)
|
|
84
|
+
pra-effect scan /var/log/huge --max-input-mb 250
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
### 3. Historical Incident Aggregation
|
|
88
|
+
Query grouped incident metrics stored in your local SQLite database:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
pra-effect history
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Example Output:
|
|
95
|
+
```text
|
|
96
|
+
=== Pra-Effect Historical Incident Groups ===
|
|
97
|
+
Fingerprint: f9e160e1d88bbca1 (v1)
|
|
98
|
+
Runtime: python
|
|
99
|
+
Error Type: ValueError
|
|
100
|
+
Message: bad <script>alert(1)</script>
|
|
101
|
+
Occurrences: 3 total across 2 runs
|
|
102
|
+
First Seen: 2026-10-24 16:30:00 UTC
|
|
103
|
+
Last Seen: 2026-10-24 16:45:12 UTC
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## Architecture & Data Flow
|
|
109
|
+
|
|
110
|
+
```text
|
|
111
|
+
Log Stream (STDIN / File)
|
|
112
|
+
│
|
|
113
|
+
▼
|
|
114
|
+
Pre-check Sanitizer (4,096 char cutoff, ReDoS guard)
|
|
115
|
+
│
|
|
116
|
+
▼
|
|
117
|
+
Grammar Parsers (Python Tracebacks / Node.js V8 Stacks)
|
|
118
|
+
│
|
|
119
|
+
▼
|
|
120
|
+
Message Normalizer (UUID / hex / ID collapse — not PII redaction)
|
|
121
|
+
│
|
|
122
|
+
▼
|
|
123
|
+
Fingerprint Engine (SHA-256 top-frame + message -> 16-hex ID)
|
|
124
|
+
│
|
|
125
|
+
├─────────────────────────┬─────────────────────────┐
|
|
126
|
+
▼ ▼ ▼
|
|
127
|
+
Terminal ANSI Output SQLite Storage (WAL) HTML Dashboard
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Database & Schema
|
|
133
|
+
|
|
134
|
+
When `--save` is passed, runs are stored in `./.pra-effect/traces.db` (override via `--db <path>`):
|
|
135
|
+
|
|
136
|
+
- `runs`: Metadata per execution (duration, line counts, unique group count).
|
|
137
|
+
- `incidents`: Normalized error incidents linked by foreign keys, tracking `is_truncated`, occurrence counts, and project frames.
|
|
138
|
+
- `v_historical_incidents`: High-performance view aggregating cross-run recurrence.
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## Testing
|
|
143
|
+
|
|
144
|
+
Run the included verification suite (standard `unittest` discovery, works from any directory):
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
python -m unittest discover -s . -p "test_*.py"
|
|
148
|
+
# or
|
|
149
|
+
python test_pra_effect.py
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Memory benchmarks for release gates (`GATE-MEM-01A` streaming pipeline & `GATE-MEM-01B` realistic workload):
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
python bench_memory.py # exit 0 = PASS (<= 50 MB Peak RSS), 1 = gate exceeded, 2 = RSS NOT_RUN
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Benchmark & Performance Gates:
|
|
159
|
+
- **GATE-MEM-01A (Pipeline Ingestion Stream)**: Peak RSS $\le$ 35 MB, tracemalloc $<$ 1 MB. Empirically measured **24.36 MB Peak RSS** and **0.0047 MB tracemalloc** across 54,000+ tracebacks ($O(1)$ streaming ingestion).
|
|
160
|
+
- **GATE-MEM-01B (Realistic Workload End-to-End)**: Peak RSS $\le$ 50 MB on files $\ge$ 25 MB with realistic deduplication ($\le$ 10,000 unique groups). Empirically measured **36.50 MB Peak RSS** and **6.12 MB tracemalloc** on 28.2 MB / 72,316 tracebacks.
|
|
161
|
+
- **High-Cardinality Boundary**: In-memory grouping memory scales linearly $O(G)$ with unique error groups (~963 bytes per group with complete `representative_trace`). Peak RSS 50 MB is not guaranteed for adversarial logs where every single error is unique. `--max-input-mb` (default 100 MB) provides raw input byte capping.
|
|
162
|
+
|
|
163
|
+
Tests cover:
|
|
164
|
+
- Python & Node.js grouping and line drift resilience.
|
|
165
|
+
- Message normalization and hex address masking.
|
|
166
|
+
- ReDoS guard and bounded multiline collection.
|
|
167
|
+
- Interleaved non-trace log line tolerance.
|
|
168
|
+
- SQLite WAL mode and concurrent transaction safety.
|
|
169
|
+
- Single-file HTML generation with full XSS escaping on every rendered field, plus a PII notice (normalization is not redaction).
|
|
170
|
+
- Chained exception collapse, bounded message collector, directory scan `--pattern` with skip/exit semantics.
|
|
171
|
+
- Fingerprint version isolation in history queries (`FINGERPRINT_SPEC.md`).
|
|
172
|
+
- CLI exit codes (0, 1, 2).
|
|
173
|
+
- Streaming pipeline memory regression ($O(1)$ constant memory ingestion).
|
|
174
|
+
- Raw input file cap backstop (`--max-input-mb`).
|
|
175
|
+
- Golden CLI output parity (byte-identical across refactors).
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## License
|
|
180
|
+
|
|
181
|
+
MIT License. See [LICENSE](LICENSE) for details.
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Release & Changelog
|
|
186
|
+
|
|
187
|
+
See [CHANGELOG.md](CHANGELOG.md) for the version history. Current release: `1.0.0`.
|
|
188
|
+
|
|
189
|
+
Version consistency across the package (`pyproject.toml`), the CLI (`--version`), and
|
|
190
|
+
the git tag is enforced by `test_release_version_consistency`.
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pra-effect
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: Zero-dependency local CLI for parsing, deduplicating, and fingerprinting Python & Node.js error stack traces.
|
|
5
|
+
Author: Genesis410
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Genesis410/pra-effect
|
|
8
|
+
Project-URL: Repository, https://github.com/Genesis410/pra-effect.git
|
|
9
|
+
Project-URL: Issues, https://github.com/Genesis410/pra-effect/issues
|
|
10
|
+
Keywords: stacktrace,deduplication,cli,incident-management,logging,developer-tools
|
|
11
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Software Development :: Debuggers
|
|
21
|
+
Classifier: Topic :: System :: Logging
|
|
22
|
+
Requires-Python: >=3.9
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Dynamic: license-file
|
|
26
|
+
|
|
27
|
+
# Pra-Effect
|
|
28
|
+
|
|
29
|
+
> Zero-dependency local CLI for parsing, deduplicating, and fingerprinting Python & Node.js error stack traces with low-memory streaming ingestion ($O(1)$ reader, $O(G)$ unique-group memory), ReDoS resilience, SQLite persistence, and standalone HTML reporting.
|
|
30
|
+
|
|
31
|
+
[](https://www.python.org/)
|
|
32
|
+
[](LICENSE)
|
|
33
|
+
[]()
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Highlights
|
|
38
|
+
|
|
39
|
+
- **Zero Third-Party Dependencies:** Pure Python standard library (`argparse`, `hashlib`, `html`, `re`, `sqlite3`, `pathlib`).
|
|
40
|
+
- **Resilient Fingerprinting (`fingerprint_v1`):** Normalizes dynamic memory addresses (`0x7f...`), variable line numbers, timestamps, UUIDs, and ephemeral ports so identical incidents aggregate into one canonical hash.
|
|
41
|
+
- **Dual Runtime Support:** Natively parses Python tracebacks (standard, chained exceptions, multiline messages) and Node.js call stacks (V8 engine).
|
|
42
|
+
- **Robust Input Handling:**
|
|
43
|
+
- Hard line truncation at 4,096 characters to prevent catastrophic regex backtracking (ReDoS).
|
|
44
|
+
- Interleaving tolerance up to 2 lines of interleaved non-trace log noise.
|
|
45
|
+
- Bounded message collection: log lines (`INFO ...`, timestamps) never leak into an error message.
|
|
46
|
+
- Chained Python exceptions collapse to a single root incident (causes kept as diagnostic context, never split the fingerprint).
|
|
47
|
+
- **Streaming Architecture & Performance:**
|
|
48
|
+
- Input parsed line-by-line through state machines — reader memory footprint is $O(1)$ (0.0047 MB tracemalloc for pure stream ingestion).
|
|
49
|
+
- Total memory scales with the number of *unique incident groups* $O(G)$, not total input byte size ($G$ distinct error patterns retained in memory).
|
|
50
|
+
- Benchmark (Python 3.14 on Windows, synthetic multi-group logs):
|
|
51
|
+
- *Typical log stream (28.2 MB, ~72k traces, ~6.8k unique groups):* ~6.1 MB Python heap, ~36.5 MB Peak RSS (GATE-MEM-01B PASS).
|
|
52
|
+
- *Adversarial high-cardinality stream (21.4 MB, ~54k traces, ~44.7k unique groups):* ~39.8 MB Python heap, ~106 MB Peak RSS (characterized in DEC-017).
|
|
53
|
+
- Optional `--max-input-mb` backstop rejects oversized raw log files (default 100 MB, 0 disables).
|
|
54
|
+
- **Local Persistence & Export:**
|
|
55
|
+
- WAL-mode SQLite storage for thread-safe concurrent writes and query performance.
|
|
56
|
+
- Self-contained, single-file HTML reports with zero external CDN assets.
|
|
57
|
+
- **Semantic Exit Codes:** POSIX standard compliance (`0` = success/analysis complete, `1` = fatal I/O, `2` = invalid CLI invocation).
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Installation
|
|
62
|
+
|
|
63
|
+
### From Source (Editable / Tool Mode)
|
|
64
|
+
```bash
|
|
65
|
+
git clone https://github.com/Genesis410/pra-effect.git
|
|
66
|
+
cd pra-effect
|
|
67
|
+
pip install -e .
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Standalone (Zero Install)
|
|
71
|
+
You can directly invoke the script without installation:
|
|
72
|
+
```bash
|
|
73
|
+
python pra_effect.py --help
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## CLI Usage
|
|
79
|
+
|
|
80
|
+
Pra-Effect provides three primary commands: `ingest`, `scan`, and `history`.
|
|
81
|
+
|
|
82
|
+
### 1. Ingest via STDIN (Pipe Mode)
|
|
83
|
+
Pipe stdout or stderr from test runners, logs, or services directly into Pra-Effect:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# Direct pipe from a failing script
|
|
87
|
+
python app.py 2>&1 | pra-effect ingest
|
|
88
|
+
|
|
89
|
+
# Save run to SQLite and export an HTML dashboard
|
|
90
|
+
pytest 2>&1 | pra-effect ingest --save --html report.html
|
|
91
|
+
|
|
92
|
+
# Check installed version (must match pyproject + git tag)
|
|
93
|
+
pra-effect --version
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### 2. Scan Existing Log Files
|
|
97
|
+
Analyze log dumps or application error logs on disk — a single file or a whole directory:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# Explicit file (must be readable, otherwise exit 1)
|
|
101
|
+
pra-effect scan /var/log/application.log --save --html report.html
|
|
102
|
+
|
|
103
|
+
# Directory scan: default pattern *.log,*.txt, comma-separated globs supported
|
|
104
|
+
pra-effect scan /var/log --pattern "*.log,*.txt,*.out"
|
|
105
|
+
|
|
106
|
+
# Files that cannot be read (binary/unreadable) are skipped with a warning on stderr.
|
|
107
|
+
# Exit 1 if zero readable files matched — never a silent "0 errors found".
|
|
108
|
+
|
|
109
|
+
# Input-size backstop (default 100 MB, streaming already bounds the pipeline; 0 disables)
|
|
110
|
+
pra-effect scan /var/log/huge --max-input-mb 250
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### 3. Historical Incident Aggregation
|
|
114
|
+
Query grouped incident metrics stored in your local SQLite database:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
pra-effect history
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Example Output:
|
|
121
|
+
```text
|
|
122
|
+
=== Pra-Effect Historical Incident Groups ===
|
|
123
|
+
Fingerprint: f9e160e1d88bbca1 (v1)
|
|
124
|
+
Runtime: python
|
|
125
|
+
Error Type: ValueError
|
|
126
|
+
Message: bad <script>alert(1)</script>
|
|
127
|
+
Occurrences: 3 total across 2 runs
|
|
128
|
+
First Seen: 2026-10-24 16:30:00 UTC
|
|
129
|
+
Last Seen: 2026-10-24 16:45:12 UTC
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Architecture & Data Flow
|
|
135
|
+
|
|
136
|
+
```text
|
|
137
|
+
Log Stream (STDIN / File)
|
|
138
|
+
│
|
|
139
|
+
▼
|
|
140
|
+
Pre-check Sanitizer (4,096 char cutoff, ReDoS guard)
|
|
141
|
+
│
|
|
142
|
+
▼
|
|
143
|
+
Grammar Parsers (Python Tracebacks / Node.js V8 Stacks)
|
|
144
|
+
│
|
|
145
|
+
▼
|
|
146
|
+
Message Normalizer (UUID / hex / ID collapse — not PII redaction)
|
|
147
|
+
│
|
|
148
|
+
▼
|
|
149
|
+
Fingerprint Engine (SHA-256 top-frame + message -> 16-hex ID)
|
|
150
|
+
│
|
|
151
|
+
├─────────────────────────┬─────────────────────────┐
|
|
152
|
+
▼ ▼ ▼
|
|
153
|
+
Terminal ANSI Output SQLite Storage (WAL) HTML Dashboard
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
---
|
|
157
|
+
|
|
158
|
+
## Database & Schema
|
|
159
|
+
|
|
160
|
+
When `--save` is passed, runs are stored in `./.pra-effect/traces.db` (override via `--db <path>`):
|
|
161
|
+
|
|
162
|
+
- `runs`: Metadata per execution (duration, line counts, unique group count).
|
|
163
|
+
- `incidents`: Normalized error incidents linked by foreign keys, tracking `is_truncated`, occurrence counts, and project frames.
|
|
164
|
+
- `v_historical_incidents`: High-performance view aggregating cross-run recurrence.
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Testing
|
|
169
|
+
|
|
170
|
+
Run the included verification suite (standard `unittest` discovery, works from any directory):
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
python -m unittest discover -s . -p "test_*.py"
|
|
174
|
+
# or
|
|
175
|
+
python test_pra_effect.py
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Memory benchmarks for release gates (`GATE-MEM-01A` streaming pipeline & `GATE-MEM-01B` realistic workload):
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
python bench_memory.py # exit 0 = PASS (<= 50 MB Peak RSS), 1 = gate exceeded, 2 = RSS NOT_RUN
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Benchmark & Performance Gates:
|
|
185
|
+
- **GATE-MEM-01A (Pipeline Ingestion Stream)**: Peak RSS $\le$ 35 MB, tracemalloc $<$ 1 MB. Empirically measured **24.36 MB Peak RSS** and **0.0047 MB tracemalloc** across 54,000+ tracebacks ($O(1)$ streaming ingestion).
|
|
186
|
+
- **GATE-MEM-01B (Realistic Workload End-to-End)**: Peak RSS $\le$ 50 MB on files $\ge$ 25 MB with realistic deduplication ($\le$ 10,000 unique groups). Empirically measured **36.50 MB Peak RSS** and **6.12 MB tracemalloc** on 28.2 MB / 72,316 tracebacks.
|
|
187
|
+
- **High-Cardinality Boundary**: In-memory grouping memory scales linearly $O(G)$ with unique error groups (~963 bytes per group with complete `representative_trace`). Peak RSS 50 MB is not guaranteed for adversarial logs where every single error is unique. `--max-input-mb` (default 100 MB) provides raw input byte capping.
|
|
188
|
+
|
|
189
|
+
Tests cover:
|
|
190
|
+
- Python & Node.js grouping and line drift resilience.
|
|
191
|
+
- Message normalization and hex address masking.
|
|
192
|
+
- ReDoS guard and bounded multiline collection.
|
|
193
|
+
- Interleaved non-trace log line tolerance.
|
|
194
|
+
- SQLite WAL mode and concurrent transaction safety.
|
|
195
|
+
- Single-file HTML generation with full XSS escaping on every rendered field, plus a PII notice (normalization is not redaction).
|
|
196
|
+
- Chained exception collapse, bounded message collector, directory scan `--pattern` with skip/exit semantics.
|
|
197
|
+
- Fingerprint version isolation in history queries (`FINGERPRINT_SPEC.md`).
|
|
198
|
+
- CLI exit codes (0, 1, 2).
|
|
199
|
+
- Streaming pipeline memory regression ($O(1)$ constant memory ingestion).
|
|
200
|
+
- Raw input file cap backstop (`--max-input-mb`).
|
|
201
|
+
- Golden CLI output parity (byte-identical across refactors).
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
## License
|
|
206
|
+
|
|
207
|
+
MIT License. See [LICENSE](LICENSE) for details.
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## Release & Changelog
|
|
212
|
+
|
|
213
|
+
See [CHANGELOG.md](CHANGELOG.md) for the version history. Current release: `1.0.0`.
|
|
214
|
+
|
|
215
|
+
Version consistency across the package (`pyproject.toml`), the CLI (`--version`), and
|
|
216
|
+
the git tag is enforced by `test_release_version_consistency`.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
pra_effect
|