trace2rules 0.1.2__tar.gz → 0.1.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- trace2rules-0.1.3/PKG-INFO +312 -0
- trace2rules-0.1.3/README.md +283 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/pyproject.toml +1 -1
- {trace2rules-0.1.2 → trace2rules-0.1.3}/pyproject.toml.orig +1 -1
- trace2rules-0.1.3/src/trace2rules/agents.py +771 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/cli.py +396 -52
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/cluster.py +11 -2
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/db.py +30 -3
- trace2rules-0.1.3/src/trace2rules/detectors.py +899 -0
- trace2rules-0.1.3/src/trace2rules/evaluate.py +64 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/ingest.py +472 -166
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/models.py +4 -0
- trace2rules-0.1.3/src/trace2rules/reconcile.py +127 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/sources/agent_trajectory.py +121 -79
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/sources/opencode.py +27 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/sources/swe_chat.py +40 -3
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/synthesize.py +268 -108
- trace2rules-0.1.2/PKG-INFO +0 -329
- trace2rules-0.1.2/README.md +0 -300
- trace2rules-0.1.2/src/trace2rules/detectors.py +0 -185
- {trace2rules-0.1.2 → trace2rules-0.1.3}/LICENSE +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/__init__.py +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/labels.py +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/patterns.py +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/redact.py +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/sources/__init__.py +0 -0
- {trace2rules-0.1.2 → trace2rules-0.1.3}/src/trace2rules/sources/common.py +0 -0
|
@@ -0,0 +1,312 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: trace2rules
|
|
3
|
+
Version: 0.1.3
|
|
4
|
+
Summary: Mine coding-agent trajectories into evidence-backed AGENTS.md rules
|
|
5
|
+
Keywords: agents,coding-agents,agents.md,trajectories,duckdb
|
|
6
|
+
Author: Rakha Kanz Kautsar
|
|
7
|
+
Author-email: Rakha Kanz Kautsar <rkkautsar@gmail.com>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
15
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
16
|
+
Requires-Dist: duckdb>=2.0.0
|
|
17
|
+
Requires-Dist: openai>=3.51.0
|
|
18
|
+
Requires-Dist: pydantic>=2.14.3
|
|
19
|
+
Requires-Dist: python-dotenv>=1.2.4
|
|
20
|
+
Requires-Dist: rich>=15.0.0
|
|
21
|
+
Requires-Dist: scikit-learn>=1.9.1
|
|
22
|
+
Requires-Dist: typer>=0.29.1
|
|
23
|
+
Requires-Dist: agent-trajectory>=0.4.4
|
|
24
|
+
Requires-Python: >=3.12
|
|
25
|
+
Project-URL: Homepage, https://github.com/rkkautsar/trace2rules
|
|
26
|
+
Project-URL: Repository, https://github.com/rkkautsar/trace2rules
|
|
27
|
+
Project-URL: Issues, https://github.com/rkkautsar/trace2rules/issues
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# trace2rules
|
|
31
|
+
|
|
32
|
+
Mine and audit coding-agent sessions into evidence-backed `AGENTS.md` rules.
|
|
33
|
+
|
|
34
|
+
`trace2rules` is a local-first CLI. It ingests agent trajectories, mines recurring friction, audits existing `AGENTS.md`/`CLAUDE.md` rules against real sessions, and identifies whether each rule is observed, compliant, violated, unobserved, or already enforced by CI/pre-commit. New rule proposals are reconciled against existing rules instead of blindly duplicated.
|
|
35
|
+
|
|
36
|
+
Status: beta. The output is a proposal, not an automatically applied `AGENTS.md`.
|
|
37
|
+
|
|
38
|
+
## Example finding
|
|
39
|
+
|
|
40
|
+
On SWC (Rust), auditing the repository’s real `AGENTS.md` against 188 local sessions showed the gap between written rules and actual behavior:
|
|
41
|
+
|
|
42
|
+
| Rule | Observed | Applicable | Compliant | Violations | Enforcement |
|
|
43
|
+
|---|---:|---:|---:|---:|---|
|
|
44
|
+
| `Prefer fixture tests over inline (#[test]) tests` | 5 | 188 | 1 | 187 | missing |
|
|
45
|
+
| `Use gh CLI for github.com data` | 87 | 188 | 0 | 0 | configured in `.github/workflows/CI.yml` |
|
|
46
|
+
|
|
47
|
+
The first rule is a soft instruction: nearly every session was applicable, but only five were observed exercising it. The second is already backed by CI. That distinction is what `audit` exposes.
|
|
48
|
+
|
|
49
|
+
The same sessions produce suggested rules:
|
|
50
|
+
|
|
51
|
+
**`swc-project/swc`**
|
|
52
|
+
|
|
53
|
+
- Build and validate the touched crate offline with the pinned toolchain, e.g. `RUSTUP_TOOLCHAIN=1.84.1 cargo test -p swc_ecma_codegen --offline`; bare `cargo` attempts to download the nightly channel and fails in this sandbox.
|
|
54
|
+
- In large sources such as crates/swc_ecma_codegen/src/lib.rs and crates/swc_ecma_minifier/src/compress/optimize/{mod,sequences}.rs, locate symbols with `grep -n` and read `sed -n` line ranges instead of whole-file reads or repeated full re-reads.
|
|
55
|
+
|
|
56
|
+
### Local examples
|
|
57
|
+
|
|
58
|
+
**`rkkautsar/trace2rules`**
|
|
59
|
+
|
|
60
|
+
- Treat `src/trace2rules/` as a linear pipeline: `ingest.py` normalizes calls, `detectors.py` emits findings, `cluster.py` groups failures, `synthesize.py` renders rules, and `cli.py` wires commands. Change the owning module and its `tests/test_<module>.py`, not every stage.
|
|
61
|
+
- Find the symbol with `rg -n '^(def|class) ' src/trace2rules/<module>.py` and read only that range; avoid `cat -n` or `nl -ba` on all of `synthesize.py`, `cli.py`, `detectors.py`, or `cluster.py`.
|
|
62
|
+
- Run `uv run pytest -q` from the repository root after edits; narrow one failing CLI case with `uv run pytest -q tests/test_cli.py -k <pattern>`.
|
|
63
|
+
|
|
64
|
+
**`rkkautsar/louibot`**
|
|
65
|
+
|
|
66
|
+
- Verify changes from the repo root with `npx tsc --noEmit && npm test`, then `npm run build`.
|
|
67
|
+
- Keep a section map of `src/agent.ts` (prompts, compaction, validation) in AGENTS.md; grep the symbol, then read only that bounded range.
|
|
68
|
+
- Run vendored Mini-SWE-Agent code in `.louibot/benchmarks/*/miniswe-agent-env` with `python3.10`; `python3` 3.15 raises `ImportError: cannot import name 'ArgsKwargs' from 'pydantic_core._pydantic_core'`.
|
|
69
|
+
|
|
70
|
+
**`rkkautsar/personal-website`**
|
|
71
|
+
|
|
72
|
+
- After editing `scripts/generate-blog.mjs` or `scripts/og-image.mjs`, regenerate with `node scripts/generate-blog.mjs` and confirm the affected `dist/blog/<slug>/index.html` contains the expected `<meta property="og:image">`/`og:image:width` tags before reporting done.
|
|
73
|
+
- Read `scripts/generate-blog.mjs` and `scripts/og-image.mjs` with targeted line ranges (`sed -n '1,40p'`) around the function under change instead of dumping the whole file in one call.
|
|
74
|
+
|
|
75
|
+
Rules are capped at 5 rules, 60 lines, and 2,500 characters per repo.
|
|
76
|
+
|
|
77
|
+
## Quick start: audit existing rules
|
|
78
|
+
|
|
79
|
+
Run from the target checkout. `AGENTS.md` and `CLAUDE.md` are auto-detected.
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
Install trace2rules if missing (`uv tool install trace2rules`). Audit this repo's AGENTS.md and CLAUDE.md against my local sessions with `trace2rules audit`. Report unobserved, violated, and unenforced rules, and propose existing CI/pre-commit checks without duplicating configured enforcement.
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Manual:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
trace2rules audit --prompt > audit.txt
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Then give `audit.txt` to your agent.
|
|
92
|
+
|
|
93
|
+
## Quick start: your own sessions
|
|
94
|
+
|
|
95
|
+
Ask your coding agent to mine, report, and synthesize:
|
|
96
|
+
|
|
97
|
+
```text
|
|
98
|
+
Install trace2rules if missing (`uv tool install trace2rules`). Run `trace2rules ingest` for my local agent sessions, then `trace2rules mine`, then `trace2rules report --prompt`. Read that report and synthesize repo-specific AGENTS.md rules yourself. Keep only rules supported by the evidence.
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Or run the evidence step manually:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
uv tool install trace2rules
|
|
105
|
+
trace2rules ingest
|
|
106
|
+
trace2rules mine
|
|
107
|
+
trace2rules report --prompt > evidence.txt
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Then give `evidence.txt` to your agent to synthesize the rules. `trace2rules synthesize` is the optional built-in LLM fallback.
|
|
111
|
+
|
|
112
|
+
To write one prompt per detected repo instead of one combined stream:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
trace2rules report --prompt --out-dir reports/
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
`--source` is repeatable and defaults to your home directory. It scans all standard local stores: `~/.claude/projects`, `~/.codex/sessions`, `~/.pi/agent/sessions`, `~/.omp/agent/sessions`, and `~/.local/share/opencode`. The default cap is 200 sessions per source and selects the most recently modified files; use `--limit 0` for all.
|
|
119
|
+
|
|
120
|
+
For local sessions, the repo label comes from the git remote (`owner/name`) when available, falling back to the working-directory basename. `report --prompt` emits one prompt per detected repo. Run `trace2rules mine` to see labels.
|
|
121
|
+
|
|
122
|
+
Agent synthesis from the report is the default workflow. `trace2rules synthesize` is optional; add `--offline` to force its deterministic rules.
|
|
123
|
+
|
|
124
|
+
`ingest` writes to `$TMPDIR/trace2rules/trace2rules.duckdb` by default. Set `TRACE2RULES_DB` to use a persistent path.
|
|
125
|
+
|
|
126
|
+
## Audit existing rules
|
|
127
|
+
|
|
128
|
+
`audit` parses `AGENTS.md` and `CLAUDE.md`, checks each rule against real sessions, and finds existing enforcement:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
trace2rules audit
|
|
132
|
+
trace2rules audit --commit HEAD --json
|
|
133
|
+
trace2rules audit --prompt
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Real sqlglot example:
|
|
137
|
+
|
|
138
|
+
```text
|
|
139
|
+
Make sure all tests added to tests/dialects/*.py actually run...
|
|
140
|
+
observed: 723 applicable: 1000 compliant: 705
|
|
141
|
+
violations: 295 not observed: 277
|
|
142
|
+
scope: sqlglot/dialects/snowflake.py
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
```text
|
|
146
|
+
Run `make check`
|
|
147
|
+
status: configured
|
|
148
|
+
source: .github/workflows/package-test.yml
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
`available` means a Makefile/package target exists but is not wired into CI. `missing` means no mechanism was found. `--prompt` emits a self-contained review prompt for a fresh-context agent.
|
|
152
|
+
|
|
153
|
+
`trace2rules synthesize --agents AGENTS.md` reconciles proposed rules against the audit: duplicates are dropped, configured enforcement is reused, and soft rules are upgraded with concrete enforcement. When run from the target checkout, `--agents` is optional.
|
|
154
|
+
|
|
155
|
+
## Synthesis
|
|
156
|
+
|
|
157
|
+
Synthesis is LLM-first. `trace2rules synthesize` sends redacted gated findings for each repo to an OpenAI-compatible chat model and asks for structured rules. Common API keys, bearer tokens, URL credentials, query tokens, password assignments, and private keys are redacted before external calls; redaction is best-effort, not a guarantee.
|
|
158
|
+
|
|
159
|
+
Provider selection:
|
|
160
|
+
|
|
161
|
+
1. `TRACE2RULES_LLM_API_KEY` — API key for a custom OpenAI-compatible endpoint.
|
|
162
|
+
2. `GEMINI_API_KEY` or `GOOGLE_API_KEY` — Gemini OpenAI-compatible endpoint.
|
|
163
|
+
3. `OPENAI_API_KEY` — OpenAI.
|
|
164
|
+
|
|
165
|
+
If no key is configured, or the call fails, synthesis falls back to deterministic rules and continues. `--offline` forces the deterministic path without a network call.
|
|
166
|
+
|
|
167
|
+
`TRACE2RULES_BASE_URL` makes any OpenAI-compatible endpoint usable, including a LiteLLM proxy.
|
|
168
|
+
|
|
169
|
+
## Supported inputs
|
|
170
|
+
|
|
171
|
+
| Format | What it reads | Notes |
|
|
172
|
+
|---|---|---|
|
|
173
|
+
| `swe-rebench` | `nebius/SWE-rebench-openhands-trajectories` parquet | Opt-in; cached locally from Hugging Face |
|
|
174
|
+
| `agent-trajectory` | Local Claude Code, Codex, Pi, OMP, OpenCode SQLite/JSON | Default; Node.js 20+ required for non-OpenCode sources |
|
|
175
|
+
|
|
176
|
+
Datasets are not included in this repository.
|
|
177
|
+
|
|
178
|
+
## Install
|
|
179
|
+
|
|
180
|
+
Use the published package:
|
|
181
|
+
|
|
182
|
+
```bash
|
|
183
|
+
uv tool install trace2rules
|
|
184
|
+
# or
|
|
185
|
+
pip install trace2rules
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
For development from a clone:
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
uv sync
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
`agent-trajectory` calls its bundled JS bridge through the system Node runtime for Claude Code, Codex, Pi, and OMP. Install Node.js 20 or newer; OpenCode SQLite ingestion is pure Python.
|
|
195
|
+
|
|
196
|
+
## Local agent sessions
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
uv run trace2rules ingest --limit 0
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
By default, it scans the standard stores:
|
|
203
|
+
|
|
204
|
+
- `~/.claude/projects`
|
|
205
|
+
- `~/.codex/sessions`
|
|
206
|
+
- `~/.pi/agent/sessions`
|
|
207
|
+
- `~/.omp/agent/sessions`
|
|
208
|
+
- `~/.local/share/opencode`
|
|
209
|
+
|
|
210
|
+
OpenCode is read directly from its SQLite database; JSON exports are still accepted:
|
|
211
|
+
|
|
212
|
+
```bash
|
|
213
|
+
uv run trace2rules ingest --format agent-trajectory --source ~/.local/share/opencode
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
The repository label is inferred from the git remote or working-directory basename.
|
|
217
|
+
|
|
218
|
+
## Commands
|
|
219
|
+
|
|
220
|
+
| Command | Purpose |
|
|
221
|
+
|---|---|
|
|
222
|
+
| `ingest` | Load sessions and tool events into DuckDB |
|
|
223
|
+
| `mine` | Show friction clusters, hotspots, recovery loops, and context blasts |
|
|
224
|
+
| `report` | Emit all findings and patterns as evidence-rich JSON for an agent |
|
|
225
|
+
| `doctor` | Check install, Node, credentials, source stores, and database schema |
|
|
226
|
+
| `patterns` | Mine structural multi-step friction |
|
|
227
|
+
| `exact` | Show exact recurring sequences with evidence |
|
|
228
|
+
| `synthesize` | Print per-repo rules to stdout |
|
|
229
|
+
| `audit` | Audit existing `AGENTS.md` rules against sessions and suggest enforcement |
|
|
230
|
+
|
|
231
|
+
Useful options:
|
|
232
|
+
|
|
233
|
+
```bash
|
|
234
|
+
trace2rules ingest # ~, all detected local stores
|
|
235
|
+
trace2rules ingest --format swe-rebench # opt into the public dataset
|
|
236
|
+
trace2rules ingest --source PATH --limit 200
|
|
237
|
+
trace2rules synthesize
|
|
238
|
+
trace2rules synthesize --offline
|
|
239
|
+
trace2rules synthesize --max-rules 10
|
|
240
|
+
trace2rules synthesize
|
|
241
|
+
trace2rules audit --agents AGENTS.md
|
|
242
|
+
trace2rules audit --agents AGENTS.md --commit HEAD --json
|
|
243
|
+
trace2rules audit --agents AGENTS.md --prompt
|
|
244
|
+
trace2rules report --prompt --agents AGENTS.md
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
## Agentic workflow
|
|
248
|
+
|
|
249
|
+
The built-in synthesizer is optional. An agent can consume the evidence directly:
|
|
250
|
+
|
|
251
|
+
```bash
|
|
252
|
+
trace2rules ingest
|
|
253
|
+
trace2rules report > /tmp/trace2rules-report.json
|
|
254
|
+
trace2rules report --prompt > /tmp/trace2rules-prompt.txt
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
`report` emits JSON with repo/session/instance counts, findings, exact sequences, and patterns. For the main agent workflow, `report --prompt` emits the evidence prompt directly; feed that prompt to a fresh agent and let it write the rules. The prompt includes findings, support, windows, and source references, and excludes orientation/hotspot noise. Each finding carries `sessions`, `tokens`, `evidence`, and detector-specific fields such as `lift`, `rework`, `setup_tax`, `cost`, and `paired_instances`.
|
|
258
|
+
|
|
259
|
+
The calling agent can then write `AGENTS.md` rules from that evidence without invoking `trace2rules synthesize` or configuring an LLM key.
|
|
260
|
+
|
|
261
|
+
Rules go to stdout; progress and findings go to stderr.
|
|
262
|
+
|
|
263
|
+
Synthesis is LLM-first. `--offline` skips the LLM and uses deterministic rules.
|
|
264
|
+
|
|
265
|
+
## Configuration
|
|
266
|
+
|
|
267
|
+
| Variable | Purpose |
|
|
268
|
+
|---|---|
|
|
269
|
+
| `GEMINI_API_KEY` | Gemini synthesis provider |
|
|
270
|
+
| `GOOGLE_API_KEY` | Alternate Gemini key |
|
|
271
|
+
| `OPENAI_API_KEY` | OpenAI synthesis provider |
|
|
272
|
+
| `TRACE2RULES_LLM_API_KEY` | API key for a custom OpenAI-compatible endpoint |
|
|
273
|
+
| `TRACE2RULES_BASE_URL` | OpenAI-compatible base URL (including LiteLLM) |
|
|
274
|
+
| `TRACE2RULES_MODEL` | Synthesis model, e.g. `gemini-4-hydrogen` |
|
|
275
|
+
| `TRACE2RULES_DECISION_MODEL` | Labeling backend: `clef-flash` (default), `clef`, `jev`, `gemini-decision-1`, `gpt-6-luna` |
|
|
276
|
+
| `TRACE2RULES_CACHE_DIR` | Local parquet cache; defaults to `~/.cache/trace2rules` |
|
|
277
|
+
| `HF_TOKEN` | Used by `hf download` for gated datasets; not read by trace2rules |
|
|
278
|
+
|
|
279
|
+
`TRACE2RULES_BASE_URL` also works with an OpenAI-compatible LiteLLM proxy.
|
|
280
|
+
|
|
281
|
+
## How it works
|
|
282
|
+
|
|
283
|
+
1. **Ingest** normalizes tool calls into DuckDB (`sessions`, `tool_events`).
|
|
284
|
+
2. **Detectors** find high-coverage setup failures, repeated exception signatures, read-only hotspots, and large output calls.
|
|
285
|
+
3. **Exact n-gram mining** groups recurring 3–5-step tool sequences with lift and paired-rework evidence. Windows never cross a compaction boundary; nested sequences are deduplicated before selection.
|
|
286
|
+
5. **Synthesis** is LLM-first. It selects gated findings and asks the configured model for structured rules; `--offline` forces deterministic rules.
|
|
287
|
+
|
|
288
|
+
Generated verifier scripts and low-support task-specific sequences are filtered before synthesis.
|
|
289
|
+
|
|
290
|
+
## Data and privacy
|
|
291
|
+
|
|
292
|
+
- Session data stays in your local DuckDB database.
|
|
293
|
+
- `--offline` performs no LLM calls.
|
|
294
|
+
- LLM synthesis sends mined findings, not the full raw session database. Findings can still contain commands, paths, and repository names; review before using an external provider.
|
|
295
|
+
- `data/` is gitignored.
|
|
296
|
+
|
|
297
|
+
## Known limits
|
|
298
|
+
|
|
299
|
+
- Detectors are heuristics. Review every proposed rule.
|
|
300
|
+
- Local-session formats change frequently. `agent-trajectory` is the compatibility layer; pin its version.
|
|
301
|
+
|
|
302
|
+
## Development
|
|
303
|
+
|
|
304
|
+
```bash
|
|
305
|
+
uv sync
|
|
306
|
+
uv run pytest
|
|
307
|
+
uv build
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
## License
|
|
311
|
+
|
|
312
|
+
MIT. See `LICENSE`.
|
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
# trace2rules
|
|
2
|
+
|
|
3
|
+
Mine and audit coding-agent sessions into evidence-backed `AGENTS.md` rules.
|
|
4
|
+
|
|
5
|
+
`trace2rules` is a local-first CLI. It ingests agent trajectories, mines recurring friction, audits existing `AGENTS.md`/`CLAUDE.md` rules against real sessions, and identifies whether each rule is observed, compliant, violated, unobserved, or already enforced by CI/pre-commit. New rule proposals are reconciled against existing rules instead of blindly duplicated.
|
|
6
|
+
|
|
7
|
+
Status: beta. The output is a proposal, not an automatically applied `AGENTS.md`.
|
|
8
|
+
|
|
9
|
+
## Example finding
|
|
10
|
+
|
|
11
|
+
On SWC (Rust), auditing the repository’s real `AGENTS.md` against 188 local sessions showed the gap between written rules and actual behavior:
|
|
12
|
+
|
|
13
|
+
| Rule | Observed | Applicable | Compliant | Violations | Enforcement |
|
|
14
|
+
|---|---:|---:|---:|---:|---|
|
|
15
|
+
| `Prefer fixture tests over inline (#[test]) tests` | 5 | 188 | 1 | 187 | missing |
|
|
16
|
+
| `Use gh CLI for github.com data` | 87 | 188 | 0 | 0 | configured in `.github/workflows/CI.yml` |
|
|
17
|
+
|
|
18
|
+
The first rule is a soft instruction: nearly every session was applicable, but only five were observed exercising it. The second is already backed by CI. That distinction is what `audit` exposes.
|
|
19
|
+
|
|
20
|
+
The same sessions produce suggested rules:
|
|
21
|
+
|
|
22
|
+
**`swc-project/swc`**
|
|
23
|
+
|
|
24
|
+
- Build and validate the touched crate offline with the pinned toolchain, e.g. `RUSTUP_TOOLCHAIN=1.84.1 cargo test -p swc_ecma_codegen --offline`; bare `cargo` attempts to download the nightly channel and fails in this sandbox.
|
|
25
|
+
- In large sources such as crates/swc_ecma_codegen/src/lib.rs and crates/swc_ecma_minifier/src/compress/optimize/{mod,sequences}.rs, locate symbols with `grep -n` and read `sed -n` line ranges instead of whole-file reads or repeated full re-reads.
|
|
26
|
+
|
|
27
|
+
### Local examples
|
|
28
|
+
|
|
29
|
+
**`rkkautsar/trace2rules`**
|
|
30
|
+
|
|
31
|
+
- Treat `src/trace2rules/` as a linear pipeline: `ingest.py` normalizes calls, `detectors.py` emits findings, `cluster.py` groups failures, `synthesize.py` renders rules, and `cli.py` wires commands. Change the owning module and its `tests/test_<module>.py`, not every stage.
|
|
32
|
+
- Find the symbol with `rg -n '^(def|class) ' src/trace2rules/<module>.py` and read only that range; avoid `cat -n` or `nl -ba` on all of `synthesize.py`, `cli.py`, `detectors.py`, or `cluster.py`.
|
|
33
|
+
- Run `uv run pytest -q` from the repository root after edits; narrow one failing CLI case with `uv run pytest -q tests/test_cli.py -k <pattern>`.
|
|
34
|
+
|
|
35
|
+
**`rkkautsar/louibot`**
|
|
36
|
+
|
|
37
|
+
- Verify changes from the repo root with `npx tsc --noEmit && npm test`, then `npm run build`.
|
|
38
|
+
- Keep a section map of `src/agent.ts` (prompts, compaction, validation) in AGENTS.md; grep the symbol, then read only that bounded range.
|
|
39
|
+
- Run vendored Mini-SWE-Agent code in `.louibot/benchmarks/*/miniswe-agent-env` with `python3.10`; `python3` 3.15 raises `ImportError: cannot import name 'ArgsKwargs' from 'pydantic_core._pydantic_core'`.
|
|
40
|
+
|
|
41
|
+
**`rkkautsar/personal-website`**
|
|
42
|
+
|
|
43
|
+
- After editing `scripts/generate-blog.mjs` or `scripts/og-image.mjs`, regenerate with `node scripts/generate-blog.mjs` and confirm the affected `dist/blog/<slug>/index.html` contains the expected `<meta property="og:image">`/`og:image:width` tags before reporting done.
|
|
44
|
+
- Read `scripts/generate-blog.mjs` and `scripts/og-image.mjs` with targeted line ranges (`sed -n '1,40p'`) around the function under change instead of dumping the whole file in one call.
|
|
45
|
+
|
|
46
|
+
Rules are capped at 5 rules, 60 lines, and 2,500 characters per repo.
|
|
47
|
+
|
|
48
|
+
## Quick start: audit existing rules
|
|
49
|
+
|
|
50
|
+
Run from the target checkout. `AGENTS.md` and `CLAUDE.md` are auto-detected.
|
|
51
|
+
|
|
52
|
+
```text
|
|
53
|
+
Install trace2rules if missing (`uv tool install trace2rules`). Audit this repo's AGENTS.md and CLAUDE.md against my local sessions with `trace2rules audit`. Report unobserved, violated, and unenforced rules, and propose existing CI/pre-commit checks without duplicating configured enforcement.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Manual:
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
trace2rules audit --prompt > audit.txt
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Then give `audit.txt` to your agent.
|
|
63
|
+
|
|
64
|
+
## Quick start: your own sessions
|
|
65
|
+
|
|
66
|
+
Ask your coding agent to mine, report, and synthesize:
|
|
67
|
+
|
|
68
|
+
```text
|
|
69
|
+
Install trace2rules if missing (`uv tool install trace2rules`). Run `trace2rules ingest` for my local agent sessions, then `trace2rules mine`, then `trace2rules report --prompt`. Read that report and synthesize repo-specific AGENTS.md rules yourself. Keep only rules supported by the evidence.
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Or run the evidence step manually:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
uv tool install trace2rules
|
|
76
|
+
trace2rules ingest
|
|
77
|
+
trace2rules mine
|
|
78
|
+
trace2rules report --prompt > evidence.txt
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Then give `evidence.txt` to your agent to synthesize the rules. `trace2rules synthesize` is the optional built-in LLM fallback.
|
|
82
|
+
|
|
83
|
+
To write one prompt per detected repo instead of one combined stream:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
trace2rules report --prompt --out-dir reports/
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`--source` is repeatable and defaults to your home directory. It scans all standard local stores: `~/.claude/projects`, `~/.codex/sessions`, `~/.pi/agent/sessions`, `~/.omp/agent/sessions`, and `~/.local/share/opencode`. The default cap is 200 sessions per source and selects the most recently modified files; use `--limit 0` for all.
|
|
90
|
+
|
|
91
|
+
For local sessions, the repo label comes from the git remote (`owner/name`) when available, falling back to the working-directory basename. `report --prompt` emits one prompt per detected repo. Run `trace2rules mine` to see labels.
|
|
92
|
+
|
|
93
|
+
Agent synthesis from the report is the default workflow. `trace2rules synthesize` is optional; add `--offline` to force its deterministic rules.
|
|
94
|
+
|
|
95
|
+
`ingest` writes to `$TMPDIR/trace2rules/trace2rules.duckdb` by default. Set `TRACE2RULES_DB` to use a persistent path.
|
|
96
|
+
|
|
97
|
+
## Audit existing rules
|
|
98
|
+
|
|
99
|
+
`audit` parses `AGENTS.md` and `CLAUDE.md`, checks each rule against real sessions, and finds existing enforcement:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
trace2rules audit
|
|
103
|
+
trace2rules audit --commit HEAD --json
|
|
104
|
+
trace2rules audit --prompt
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Real sqlglot example:
|
|
108
|
+
|
|
109
|
+
```text
|
|
110
|
+
Make sure all tests added to tests/dialects/*.py actually run...
|
|
111
|
+
observed: 723 applicable: 1000 compliant: 705
|
|
112
|
+
violations: 295 not observed: 277
|
|
113
|
+
scope: sqlglot/dialects/snowflake.py
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
```text
|
|
117
|
+
Run `make check`
|
|
118
|
+
status: configured
|
|
119
|
+
source: .github/workflows/package-test.yml
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
`available` means a Makefile/package target exists but is not wired into CI. `missing` means no mechanism was found. `--prompt` emits a self-contained review prompt for a fresh-context agent.
|
|
123
|
+
|
|
124
|
+
`trace2rules synthesize --agents AGENTS.md` reconciles proposed rules against the audit: duplicates are dropped, configured enforcement is reused, and soft rules are upgraded with concrete enforcement. When run from the target checkout, `--agents` is optional.
|
|
125
|
+
|
|
126
|
+
## Synthesis
|
|
127
|
+
|
|
128
|
+
Synthesis is LLM-first. `trace2rules synthesize` sends redacted gated findings for each repo to an OpenAI-compatible chat model and asks for structured rules. Common API keys, bearer tokens, URL credentials, query tokens, password assignments, and private keys are redacted before external calls; redaction is best-effort, not a guarantee.
|
|
129
|
+
|
|
130
|
+
Provider selection:
|
|
131
|
+
|
|
132
|
+
1. `TRACE2RULES_LLM_API_KEY` — API key for a custom OpenAI-compatible endpoint.
|
|
133
|
+
2. `GEMINI_API_KEY` or `GOOGLE_API_KEY` — Gemini OpenAI-compatible endpoint.
|
|
134
|
+
3. `OPENAI_API_KEY` — OpenAI.
|
|
135
|
+
|
|
136
|
+
If no key is configured, or the call fails, synthesis falls back to deterministic rules and continues. `--offline` forces the deterministic path without a network call.
|
|
137
|
+
|
|
138
|
+
`TRACE2RULES_BASE_URL` makes any OpenAI-compatible endpoint usable, including a LiteLLM proxy.
|
|
139
|
+
|
|
140
|
+
## Supported inputs
|
|
141
|
+
|
|
142
|
+
| Format | What it reads | Notes |
|
|
143
|
+
|---|---|---|
|
|
144
|
+
| `swe-rebench` | `nebius/SWE-rebench-openhands-trajectories` parquet | Opt-in; cached locally from Hugging Face |
|
|
145
|
+
| `agent-trajectory` | Local Claude Code, Codex, Pi, OMP, OpenCode SQLite/JSON | Default; Node.js 20+ required for non-OpenCode sources |
|
|
146
|
+
|
|
147
|
+
Datasets are not included in this repository.
|
|
148
|
+
|
|
149
|
+
## Install
|
|
150
|
+
|
|
151
|
+
Use the published package:
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
uv tool install trace2rules
|
|
155
|
+
# or
|
|
156
|
+
pip install trace2rules
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
For development from a clone:
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
uv sync
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
`agent-trajectory` calls its bundled JS bridge through the system Node runtime for Claude Code, Codex, Pi, and OMP. Install Node.js 20 or newer; OpenCode SQLite ingestion is pure Python.
|
|
166
|
+
|
|
167
|
+
## Local agent sessions
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
uv run trace2rules ingest --limit 0
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
By default, it scans the standard stores:
|
|
174
|
+
|
|
175
|
+
- `~/.claude/projects`
|
|
176
|
+
- `~/.codex/sessions`
|
|
177
|
+
- `~/.pi/agent/sessions`
|
|
178
|
+
- `~/.omp/agent/sessions`
|
|
179
|
+
- `~/.local/share/opencode`
|
|
180
|
+
|
|
181
|
+
OpenCode is read directly from its SQLite database; JSON exports are still accepted:
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
uv run trace2rules ingest --format agent-trajectory --source ~/.local/share/opencode
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
The repository label is inferred from the git remote or working-directory basename.
|
|
188
|
+
|
|
189
|
+
## Commands
|
|
190
|
+
|
|
191
|
+
| Command | Purpose |
|
|
192
|
+
|---|---|
|
|
193
|
+
| `ingest` | Load sessions and tool events into DuckDB |
|
|
194
|
+
| `mine` | Show friction clusters, hotspots, recovery loops, and context blasts |
|
|
195
|
+
| `report` | Emit all findings and patterns as evidence-rich JSON for an agent |
|
|
196
|
+
| `doctor` | Check install, Node, credentials, source stores, and database schema |
|
|
197
|
+
| `patterns` | Mine structural multi-step friction |
|
|
198
|
+
| `exact` | Show exact recurring sequences with evidence |
|
|
199
|
+
| `synthesize` | Print per-repo rules to stdout |
|
|
200
|
+
| `audit` | Audit existing `AGENTS.md` rules against sessions and suggest enforcement |
|
|
201
|
+
|
|
202
|
+
Useful options:
|
|
203
|
+
|
|
204
|
+
```bash
|
|
205
|
+
trace2rules ingest # ~, all detected local stores
|
|
206
|
+
trace2rules ingest --format swe-rebench # opt into the public dataset
|
|
207
|
+
trace2rules ingest --source PATH --limit 200
|
|
208
|
+
trace2rules synthesize
|
|
209
|
+
trace2rules synthesize --offline
|
|
210
|
+
trace2rules synthesize --max-rules 10
|
|
211
|
+
trace2rules synthesize
|
|
212
|
+
trace2rules audit --agents AGENTS.md
|
|
213
|
+
trace2rules audit --agents AGENTS.md --commit HEAD --json
|
|
214
|
+
trace2rules audit --agents AGENTS.md --prompt
|
|
215
|
+
trace2rules report --prompt --agents AGENTS.md
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
## Agentic workflow
|
|
219
|
+
|
|
220
|
+
The built-in synthesizer is optional. An agent can consume the evidence directly:
|
|
221
|
+
|
|
222
|
+
```bash
|
|
223
|
+
trace2rules ingest
|
|
224
|
+
trace2rules report > /tmp/trace2rules-report.json
|
|
225
|
+
trace2rules report --prompt > /tmp/trace2rules-prompt.txt
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
`report` emits JSON with repo/session/instance counts, findings, exact sequences, and patterns. For the main agent workflow, `report --prompt` emits the evidence prompt directly; feed that prompt to a fresh agent and let it write the rules. The prompt includes findings, support, windows, and source references, and excludes orientation/hotspot noise. Each finding carries `sessions`, `tokens`, `evidence`, and detector-specific fields such as `lift`, `rework`, `setup_tax`, `cost`, and `paired_instances`.
|
|
229
|
+
|
|
230
|
+
The calling agent can then write `AGENTS.md` rules from that evidence without invoking `trace2rules synthesize` or configuring an LLM key.
|
|
231
|
+
|
|
232
|
+
Rules go to stdout; progress and findings go to stderr.
|
|
233
|
+
|
|
234
|
+
Synthesis is LLM-first. `--offline` skips the LLM and uses deterministic rules.
|
|
235
|
+
|
|
236
|
+
## Configuration
|
|
237
|
+
|
|
238
|
+
| Variable | Purpose |
|
|
239
|
+
|---|---|
|
|
240
|
+
| `GEMINI_API_KEY` | Gemini synthesis provider |
|
|
241
|
+
| `GOOGLE_API_KEY` | Alternate Gemini key |
|
|
242
|
+
| `OPENAI_API_KEY` | OpenAI synthesis provider |
|
|
243
|
+
| `TRACE2RULES_LLM_API_KEY` | API key for a custom OpenAI-compatible endpoint |
|
|
244
|
+
| `TRACE2RULES_BASE_URL` | OpenAI-compatible base URL (including LiteLLM) |
|
|
245
|
+
| `TRACE2RULES_MODEL` | Synthesis model, e.g. `gemini-4-hydrogen` |
|
|
246
|
+
| `TRACE2RULES_DECISION_MODEL` | Labeling backend: `clef-flash` (default), `clef`, `jev`, `gemini-decision-1`, `gpt-6-luna` |
|
|
247
|
+
| `TRACE2RULES_CACHE_DIR` | Local parquet cache; defaults to `~/.cache/trace2rules` |
|
|
248
|
+
| `HF_TOKEN` | Used by `hf download` for gated datasets; not read by trace2rules |
|
|
249
|
+
|
|
250
|
+
`TRACE2RULES_BASE_URL` also works with an OpenAI-compatible LiteLLM proxy.
|
|
251
|
+
|
|
252
|
+
## How it works
|
|
253
|
+
|
|
254
|
+
1. **Ingest** normalizes tool calls into DuckDB (`sessions`, `tool_events`).
|
|
255
|
+
2. **Detectors** find high-coverage setup failures, repeated exception signatures, read-only hotspots, and large output calls.
|
|
256
|
+
3. **Exact n-gram mining** groups recurring 3–5-step tool sequences with lift and paired-rework evidence. Windows never cross a compaction boundary; nested sequences are deduplicated before selection.
|
|
257
|
+
5. **Synthesis** is LLM-first. It selects gated findings and asks the configured model for structured rules; `--offline` forces deterministic rules.
|
|
258
|
+
|
|
259
|
+
Generated verifier scripts and low-support task-specific sequences are filtered before synthesis.
|
|
260
|
+
|
|
261
|
+
## Data and privacy
|
|
262
|
+
|
|
263
|
+
- Session data stays in your local DuckDB database.
|
|
264
|
+
- `--offline` performs no LLM calls.
|
|
265
|
+
- LLM synthesis sends mined findings, not the full raw session database. Findings can still contain commands, paths, and repository names; review before using an external provider.
|
|
266
|
+
- `data/` is gitignored.
|
|
267
|
+
|
|
268
|
+
## Known limits
|
|
269
|
+
|
|
270
|
+
- Detectors are heuristics. Review every proposed rule.
|
|
271
|
+
- Local-session formats change frequently. `agent-trajectory` is the compatibility layer; pin its version.
|
|
272
|
+
|
|
273
|
+
## Development
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
uv sync
|
|
277
|
+
uv run pytest
|
|
278
|
+
uv build
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
## License
|
|
282
|
+
|
|
283
|
+
MIT. See `LICENSE`.
|