ithrl 0.3.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {ithrl-0.3.0 → ithrl-0.4.0}/.gitignore +1 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/CHANGELOG.md +12 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/CONTRIBUTING.md +123 -46
- {ithrl-0.3.0 → ithrl-0.4.0}/PKG-INFO +35 -20
- {ithrl-0.3.0 → ithrl-0.4.0}/README.md +34 -19
- ithrl-0.4.0/ROADMAP.md +204 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/architecture.md +114 -111
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/cli.md +75 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/decisions/0003-strict-fresh-baselines.md +4 -3
- ithrl-0.4.0/docs/fuzzing.md +185 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/index.md +9 -5
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/measurements.md +55 -62
- ithrl-0.4.0/docs/native-runs.md +140 -0
- ithrl-0.4.0/docs/reduction.md +259 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/research/context-fuzzing.md +36 -41
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/research/feasibility.md +7 -7
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/research/index.md +1 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/research/ordinary-session-investigation.md +18 -25
- ithrl-0.4.0/docs/research/passive-compaction-investigation.md +91 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/research/reduction-design.md +49 -52
- ithrl-0.4.0/docs/results.md +304 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/troubleshooting.md +69 -51
- {ithrl-0.3.0 → ithrl-0.4.0}/pyproject.toml +10 -2
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/codex/decisions.py +0 -2
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/codex/parser.py +0 -2
- ithrl-0.4.0/src/ithrl/adapters/codex/reacquisition.py +332 -0
- ithrl-0.4.0/src/ithrl/adapters/copilot/reacquisition.py +293 -0
- ithrl-0.4.0/src/ithrl/adapters/reacquisition.py +56 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/registry.py +20 -1
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/evidence.py +45 -8
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/source_storage.py +10 -17
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/__init__.py +40 -1
- ithrl-0.4.0/src/ithrl/cli/inspect.py +339 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/native.py +28 -9
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/output.py +6 -4
- ithrl-0.4.0/src/ithrl/cli/paths.py +51 -0
- ithrl-0.4.0/src/ithrl/cli/preparation.py +265 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/sampled.py +12 -11
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/sampled_reduction.py +44 -78
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/context.py +34 -12
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/execution/budget.py +136 -1
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/generation.py +2 -1
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/mutation.py +3 -7
- ithrl-0.4.0/src/ithrl/preparation.py +255 -0
- ithrl-0.4.0/src/ithrl/reacquisition.py +313 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/reduce/search.py +11 -6
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/models.py +4 -11
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/run_state.py +40 -8
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/runtime_configuration.py +2 -4
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/isolation.py +6 -9
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/sampling.py +15 -5
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/statistics.py +0 -3
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/workflow.py +4 -2
- ithrl-0.3.0/ROADMAP.md +0 -142
- ithrl-0.3.0/docs/native-runs.md +0 -515
- ithrl-0.3.0/docs/results.md +0 -251
- ithrl-0.3.0/src/ithrl/cli/inspect.py +0 -179
- ithrl-0.3.0/src/ithrl/cli/paths.py +0 -24
- {ithrl-0.3.0 → ithrl-0.4.0}/LICENSE +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/branding.css +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-favicon.svg +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-icon.svg +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-logo-dark.svg +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-logo-light.svg +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-logo.png +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/branding/ithrl-social.png +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/decisions/0001-native-evidence-and-controlled-runners.md +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/decisions/0002-fixed-samples-and-durable-dispatch.md +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/decisions/0004-single-finding-fuzz-confirmation.md +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/decisions/index.md +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/docs/third-party-notices.txt +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/examples/fuzz/campaign.toml +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/examples/fuzz/codex.jsonl +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/examples/fuzz/copilot.jsonl +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/__main__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/_jsonl.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/codex/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/codex/context.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/codex/discovery.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/copilot/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/copilot/context.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/copilot/decisions.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/copilot/discovery.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/copilot/parser.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/discovery.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/adapters/errors.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/behavior.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/calibration.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/filesystem.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/source.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/source_context.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/case/storage.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/extract.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/fuzz.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/fuzz_output.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/fuzz_run.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/fuzz_session.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/progress.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/sampled_output.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/sampled_session.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/cli/shrink.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/decisions.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/examples/synthetic-trace.json +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/execution/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/execution/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/execution/preflight.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/experiment.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/extraction.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/campaign.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/configuration.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/deterministic.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/progress.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/sampling.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/simplify.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/fuzz/statistics.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/inspection.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/measurement.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/observations.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/path_safety.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/reduce/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/reduce/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/reduce/engine.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/inspection.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/storage.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/summary.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/results/validation.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/capability.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/configuration.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/isolation.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/materialize.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/policy.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/protocol.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/protocol_contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/relay.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/runner.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/schema.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/transcript.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/codex/worker.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/capability.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/configuration.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/materialize.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/policy.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/protocol/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/protocol/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/protocol/decoding.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/protocol/parser.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/protocol/state.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/runner.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/sdk_runtime.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/copilot/worker.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/evidence.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/fake.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/registry.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/__init__.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/certificates.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/contracts.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/output.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/probes.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/process.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/programs.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/protection.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/runtime/state.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/runner/validation.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/sampled_run_metadata.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/synthetic.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/text_sections.py +0 -0
- {ithrl-0.3.0 → ithrl-0.4.0}/src/ithrl/timestamps.py +0 -0
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
|
+
## v0.4.0 (2026-11-14)
|
|
4
|
+
|
|
5
|
+
### Features
|
|
6
|
+
|
|
7
|
+
- Inspect recorded operations across compaction ([#69](https://github.com/eberkut/ithrl/pull/69), [`3594d32`](https://github.com/eberkut/ithrl/commit/3594d3212b6dbc92276223f373908529e95a5e9e))
|
|
8
|
+
- Prepare native reduction experiments offline ([#66](https://github.com/eberkut/ithrl/pull/66), [`13c9626`](https://github.com/eberkut/ithrl/commit/13c96260335982a3a3ce52ea9c73b327f649efce))
|
|
9
|
+
- Qualify passive compaction inspection ([#70](https://github.com/eberkut/ithrl/pull/70), [`3b43a1b`](https://github.com/eberkut/ithrl/commit/3b43a1b8ef0c02bfd0182650460e4efed1c13076))
|
|
10
|
+
|
|
11
|
+
### Performance Improvements
|
|
12
|
+
|
|
13
|
+
- Reduce workflow overhead and track performance ([#71](https://github.com/eberkut/ithrl/pull/71), [`ae0c8cd`](https://github.com/eberkut/ithrl/commit/ae0c8cd238eff742ec51076a1054007580c9c0db))
|
|
14
|
+
|
|
3
15
|
## v0.3.0 (2026-11-10)
|
|
4
16
|
|
|
5
17
|
### Features
|
|
@@ -72,19 +72,74 @@ Install the local hooks with `uv run pre-commit install`; run them on the whole
|
|
|
72
72
|
`uv run pre-commit run --all-files`. Ruff and rumdl format files in place. CI checks formatting
|
|
73
73
|
without modifying files.
|
|
74
74
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
The required `Python tests` check summarizes the test matrix. It accepts a skipped matrix only
|
|
83
|
-
after successful Markdown-only classification. Failed classification, missing decisions, test
|
|
84
|
-
failures and unexpected skips fail this check.
|
|
75
|
+
Markdown-only PRs skip Python linting, typing, unit tests and dependency audits. They still run
|
|
76
|
+
Markdown, documentation, strict site, workflow, dependency-review and package checks. Mixed changes,
|
|
77
|
+
unknown paths, configuration changes or 3,000 or more changed files require full validation;
|
|
78
|
+
classification includes deleted files and both sides of renames. Failed detection fails Workflows.
|
|
79
|
+
Main, scheduled, manual and release runs always validate fully. The required `Python tests` summary
|
|
80
|
+
accepts a skipped matrix only after successful Markdown-only classification; missing decisions,
|
|
81
|
+
failures and unexpected skips fail it.
|
|
85
82
|
|
|
86
83
|
The logo sources, usage rules and regeneration commands are in [branding](docs/branding/README.md).
|
|
87
84
|
|
|
85
|
+
## Performance and maintainability
|
|
86
|
+
|
|
87
|
+
Measure the affected workflow before optimizing it. Keep sanitized input generation and setup
|
|
88
|
+
outside timings; record the revision, locked environment, input size, repeats and noise. Measure
|
|
89
|
+
allocations separately from elapsed time. Include index/cache construction and retained memory,
|
|
90
|
+
then repeat the same workload after the change. Provider latency and local implementation overhead
|
|
91
|
+
are separate measurements.
|
|
92
|
+
|
|
93
|
+
- Choose collections for their operations: a set for repeated membership, a dictionary for keyed
|
|
94
|
+
lookup, and a sequence for authoritative order. For example, fuzz proposal membership can use a
|
|
95
|
+
local set while seeded traversal retains the original sequence. Measure scaling near admitted
|
|
96
|
+
limits as well as small inputs; Radon scores alone do not justify a refactor.
|
|
97
|
+
- Keep reusable derived facts with the owner of their immutable inputs. Define copy/invalidation
|
|
98
|
+
behavior and memory bounds before adding a cache. A frozen model can still copy stale derived
|
|
99
|
+
state. Do not expose mutable internal indexes or borrow mutable native records across owners.
|
|
100
|
+
- Bound both output and examined work. Count rejected and duplicate proposals where the work cap
|
|
101
|
+
requires it. Avoid repeated whole-input scans when the required operation is a keyed lookup.
|
|
102
|
+
Release large temporary buffers before processing the next item.
|
|
103
|
+
- Batch fresh persistence reads where their consistency contract permits it. Preserve schema,
|
|
104
|
+
identity, chronology and corruption checks, locks, durable dispatch and accounting. Persisted
|
|
105
|
+
files and external state need fresh validation; an in-memory cache is not their authority. Reuse
|
|
106
|
+
validated facts within one synchronous operation; refresh after callbacks, waits or execution.
|
|
107
|
+
- Test configurable limits with small inputs at the exact boundary and one beyond; assert policy
|
|
108
|
+
defaults separately when needed. Use sparse files for size-only filesystem checks. Share completed
|
|
109
|
+
immutable fixture data, not mutable cases or stores. Retain real process, SQLite and distribution
|
|
110
|
+
checks when they establish lifecycle, recovery or artifact behavior.
|
|
111
|
+
- Share an existing mechanism only when its ownership, errors and side effects fit. Require a
|
|
112
|
+
concrete maintenance benefit from abstractions; keep different harness and statistical rules
|
|
113
|
+
explicit. Compare behavior, failure paths and recovery as well as performance.
|
|
114
|
+
|
|
115
|
+
### Benchmarks
|
|
116
|
+
|
|
117
|
+
The permanent [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/latest/) workloads
|
|
118
|
+
live in `tests/benchmarks/`. They use sanitized inputs and no native harness or provider.
|
|
119
|
+
Ordinary test runs skip them. From the repository root, save a baseline before changing code:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
uv run pytest tests/benchmarks --benchmark-only --no-cov --benchmark-save=before
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
After the change, compare against that saved run:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
uv run pytest tests/benchmarks --benchmark-only --no-cov --benchmark-compare=0001 --benchmark-save=after
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Replace `0001` with the baseline ID printed by the first command. The plugin keeps saved runs
|
|
132
|
+
under the ignored `.benchmarks/` directory. Compare identical workloads on the same interpreter
|
|
133
|
+
and machine with low competing load. Check the distributions and memory tradeoffs; do not infer
|
|
134
|
+
a regression from a small noisy difference. Keep setup outside the measured call and assert
|
|
135
|
+
outcomes. Stateful workloads need equivalent starting state for every round.
|
|
136
|
+
|
|
137
|
+
CI runs the benchmarks on Python 3.14 in the existing Quality job. Non-PR runs retain the generated
|
|
138
|
+
JSON as `benchmarks-python-3.14` artifacts for 90 days. Download comparable runs and inspect them
|
|
139
|
+
with `uv run pytest-benchmark compare FIRST.json SECOND.json`. There is no elapsed-time failure
|
|
140
|
+
threshold; benchmark assertions still fail on incorrect behavior. Saved JSON contains machine and
|
|
141
|
+
commit metadata; raw profiles and private-workflow inputs do not belong in these artifacts.
|
|
142
|
+
|
|
88
143
|
## Workflow and dependency checks
|
|
89
144
|
|
|
90
145
|
For GitHub Actions or Dependabot changes, use the jactionlint release pinned in
|
|
@@ -106,14 +161,11 @@ uv export --locked --all-groups --all-extras --no-emit-project --format requirem
|
|
|
106
161
|
uv run --locked --group audit pip-audit --strict --disable-pip --require-hashes -r /dev/stdin
|
|
107
162
|
```
|
|
108
163
|
|
|
109
|
-
The
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
For dependency changes, run the lockfile audit above on each supported Python version. GitHub's
|
|
115
|
-
dependency graph may omit packages from `uv.lock`, so Dependabot and Dependency Review cannot
|
|
116
|
-
replace that audit.
|
|
164
|
+
The audit sends locked package names and versions to PyPI's vulnerability service. It installs
|
|
165
|
+
nothing, resolves no alternatives and applies no fixes. It covers locked versions, not entire
|
|
166
|
+
dependency ranges. For dependency changes, run it on each supported Python version to evaluate
|
|
167
|
+
markers. Dependabot and Dependency Review cannot replace it: GitHub's dependency graph may omit
|
|
168
|
+
lockfile packages.
|
|
117
169
|
|
|
118
170
|
The PR dependency-review job rejects newly introduced known vulnerabilities at any severity across
|
|
119
171
|
runtime, development and unknown scopes. It uses a read-only token and does not check out or execute
|
|
@@ -138,13 +190,41 @@ Keep installation and first-use instructions in README, and detailed user guides
|
|
|
138
190
|
`docs/`. Support, security and conduct policies live at the repository root. Keep private sessions,
|
|
139
191
|
cases and credentials out of documentation and generated output.
|
|
140
192
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
and
|
|
193
|
+
### Writing and review
|
|
194
|
+
|
|
195
|
+
Start with the reader's task. Use [Diátaxis](https://diataxis.fr/start-here/) to distinguish a
|
|
196
|
+
guided learning example, a task-specific how-to, reference facts and explanation. A README can
|
|
197
|
+
combine orientation and first use; do not split pages just to fill those categories.
|
|
198
|
+
|
|
199
|
+
- Put prerequisites before commands, conditions before dependent actions and the expected result
|
|
200
|
+
after a step that needs verification. Use numbered steps for a sequence and tables for
|
|
201
|
+
comparisons.
|
|
202
|
+
- State what the reader can do or conclude. Prefer concrete verbs and established terms; remove
|
|
203
|
+
filler, promotional claims, rhetorical contrasts and sentences that repeat their heading.
|
|
204
|
+
For example, replace “It is important to note that execution requires explicit acknowledgement”
|
|
205
|
+
with “After reviewing the plan, pass `--acknowledge-model-execution` to allow provider calls.”
|
|
206
|
+
- Give each topic one owner. Keep options in the generated CLI reference, procedures in user guides,
|
|
207
|
+
measurements in `docs/measurements.md`, rationale in decisions and methods in research. Link to
|
|
208
|
+
detail instead of repeating it. Keep essential safety conditions beside the affected action.
|
|
209
|
+
- Shorten without widening claims. Preserve tested profiles, statistical conditions, costs,
|
|
210
|
+
uncertainty, failed outcomes and historical context. “The screen passed” does not mean
|
|
211
|
+
“preservation is confirmed.” Check commands against the current CLI and qualify future work.
|
|
212
|
+
- Follow [W3C's writing guidance](https://www.w3.org/WAI/tips/writing/): use descriptive link text,
|
|
213
|
+
ordered headings and useful image alternatives. Do not rely on color alone. Keep code blocks
|
|
214
|
+
executable or identify placeholders before the command.
|
|
215
|
+
|
|
216
|
+
Before review, check: can the intended reader find the task, run its steps and interpret the
|
|
217
|
+
result? Does each paragraph add information? Are terms, claims, links and generated content current?
|
|
218
|
+
Check the [documentation commands](#building-and-checking-documentation) below after edits.
|
|
219
|
+
|
|
220
|
+
For additional guidance, consult the
|
|
221
|
+
[Google developer documentation style guide](https://developers.google.com/style) and
|
|
222
|
+
[Microsoft Writing Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/).
|
|
223
|
+
Follow this project's conventions where they differ.
|
|
224
|
+
Use judgment rather than word-count targets; explanations and research need enough detail to support
|
|
225
|
+
their conclusions.
|
|
226
|
+
|
|
227
|
+
### Documentation ownership
|
|
148
228
|
|
|
149
229
|
[ROADMAP.md](ROADMAP.md) describes future direction. GitHub issues own actionable scope,
|
|
150
230
|
dependencies, acceptance criteria and status; release milestones group scope and the public
|
|
@@ -162,12 +242,12 @@ Before evaluating a new study, declare its tested domain, criteria, seeds, budge
|
|
|
162
242
|
claims. Report costs across complete workflows, including failed and incomplete work, and document
|
|
163
243
|
the alternatives and limitations needed to interpret the results.
|
|
164
244
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
245
|
+
### Building and checking documentation
|
|
246
|
+
|
|
247
|
+
The `docs` group pins MkDocs and the shadcn theme. Search runs in the browser; the site bundles
|
|
248
|
+
fonts and scripts. GitHub star-count requests and raw Markdown copying are disabled. When changing
|
|
249
|
+
the theme or extensions, review the asset allowlist in `scripts/docs_theme.py`, the
|
|
250
|
+
[third-party notices](docs/third-party-notices.txt) and the rendered site. To preview:
|
|
171
251
|
|
|
172
252
|
```bash
|
|
173
253
|
uv sync --locked --all-groups --all-extras
|
|
@@ -303,10 +383,8 @@ CodeRabbit's optional planning and code-generation features are configured in
|
|
|
303
383
|
|
|
304
384
|
## Preparing a release
|
|
305
385
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
For every feature, aim for parity across supported harnesses. Any exception must be clearly
|
|
309
|
-
justified and documented before release.
|
|
386
|
+
Maintainers need repository and package-publishing access. Before release, justify and document
|
|
387
|
+
any feature-parity exception between supported harnesses.
|
|
310
388
|
|
|
311
389
|
Python Semantic Release, pinned in the `release` dependency group, calculates versions from
|
|
312
390
|
Conventional Commits. `pyproject.toml` is the version source; installed CLI and Codex client
|
|
@@ -315,9 +393,8 @@ history, omitting routine build, CI, documentation, style and test commits. Its
|
|
|
315
393
|
[template](.github/changelog/CHANGELOG.md.j2) keeps lists compact, with one entry per source line.
|
|
316
394
|
Change the template to adjust formatting; do not edit generated entries by hand.
|
|
317
395
|
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
publication workflow builds it. The publication workflow uses that committed version.
|
|
396
|
+
Review and merge generated version/notes changes through a PR. CI tests the prepared version;
|
|
397
|
+
publication builds that committed version.
|
|
321
398
|
|
|
322
399
|
Use `main` with complete history and all existing release tags available. Preview the next version
|
|
323
400
|
without changing files, the index, commits, tags or remote state:
|
|
@@ -402,11 +479,11 @@ gh workflow run release.yml --repo eberkut/ithrl --ref main \
|
|
|
402
479
|
6. Verify the published versions, hashes, attestations and installed example. A successful upload
|
|
403
480
|
alone does not complete release acceptance.
|
|
404
481
|
|
|
405
|
-
Complete review within 30
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
`skip-existing`.
|
|
482
|
+
Complete review within the 30-day artifact retention and GitHub workflow lifetime, including
|
|
483
|
+
approval waits. Start one release at a time: a new queued run may replace an older pending run,
|
|
484
|
+
though the active run is preserved. Do not rerun the workflow to promote a completed rehearsal or
|
|
485
|
+
cancel during upload.
|
|
486
|
+
|
|
487
|
+
After failure or partial publication, inspect the index and release before recovery. Preserve
|
|
488
|
+
the original files and hashes; uploading identical missing files requires explicit approval.
|
|
489
|
+
Never overwrite a published version with rebuilt files or suppress conflicts with `skip-existing`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: ithrl
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: Find smaller agent contexts that preserve a selected model behavior
|
|
5
5
|
Project-URL: Source, https://github.com/eberkut/ithrl
|
|
6
6
|
Project-URL: Issues, https://github.com/eberkut/ithrl/issues
|
|
@@ -95,7 +95,15 @@ Use a new directory to repeat the example; extraction refuses to overwrite an ex
|
|
|
95
95
|
## Commands
|
|
96
96
|
|
|
97
97
|
- `ithrl extract` browses saved sessions and creates a case from a selected decision.
|
|
98
|
+
- `ithrl prepare CASE` generates a private reduction configuration from a preflight receipt and
|
|
99
|
+
statistical settings. Follow the
|
|
100
|
+
[preparation procedure](https://github.com/eberkut/ithrl/blob/main/docs/reduction.md).
|
|
98
101
|
- `ithrl shrink CASE` searches for context removals that preserve the selected behavior.
|
|
102
|
+
- `ithrl inspect sections CASE` lists editable text fields and paragraph boundaries without
|
|
103
|
+
displaying text or requiring a reduction result.
|
|
104
|
+
- `ithrl inspect reacquire SESSION` compares recorded operations and captured results across
|
|
105
|
+
compaction. See
|
|
106
|
+
[passive inspection](https://github.com/eberkut/ithrl/blob/main/docs/results.md#inspect-recorded-compaction).
|
|
99
107
|
- `ithrl inspect retained CASE --result RESULT` explains retained units and recorded removal trials
|
|
100
108
|
from a saved reduction without model access. See
|
|
101
109
|
[results and privacy](https://github.com/eberkut/ithrl/blob/main/docs/results.md#inspect-retained-context).
|
|
@@ -115,33 +123,40 @@ Returned actions are observed and denied. Cases contain copied session content;
|
|
|
115
123
|
|
|
116
124
|
## Supported inputs
|
|
117
125
|
|
|
118
|
-
| Input | Extracted decisions | Reduction | Fuzzing |
|
|
119
|
-
| --- | --- | --- | --- |
|
|
120
|
-
| Bundled synthetic example | Synthetic tool request | Deterministic local runner | Local fake runner |
|
|
121
|
-
| Codex CLI rollouts | Final responses | Experimental native runner; [paragraph-removal measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#ordinary-session-paragraph-removal) | Experimental prior-user text appends |
|
|
122
|
-
| Copilot CLI session events | Final responses and shell requests | Experimental final-response paragraph removal and denied `bash` runner | Experimental prior-user text appends for final-response targets |
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
126
|
+
| Input | Extracted decisions | Reduction | Fuzzing | Passive inspection |
|
|
127
|
+
| --- | --- | --- | --- | --- |
|
|
128
|
+
| Bundled synthetic example | Synthetic tool request | Deterministic local runner | Local fake runner | — |
|
|
129
|
+
| Codex CLI rollouts | Final responses | Experimental native runner; [paragraph-removal measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#ordinary-session-paragraph-removal) | Experimental prior-user text appends | Exact recorded-operation and captured-result comparisons across compaction |
|
|
130
|
+
| Copilot CLI session events | Final responses and shell requests | Experimental final-response paragraph removal and denied `bash` runner | Experimental prior-user text appends for final-response targets | Exact recorded-operation and captured-result comparisons across compaction |
|
|
131
|
+
|
|
132
|
+
Passive inspection reads saved evidence without a harness or model access. The
|
|
133
|
+
[worked investigation](https://github.com/eberkut/ithrl/blob/main/docs/research/passive-compaction-investigation.md)
|
|
134
|
+
records the qualified native profiles, ordinary examples and interpretation limits.
|
|
135
|
+
|
|
136
|
+
Native fuzzing appends text to eligible prior user input while preserving the fixed triggering
|
|
137
|
+
input. The
|
|
138
|
+
[fuzzing walkthrough](https://github.com/eberkut/ithrl/blob/main/docs/fuzzing.md)
|
|
139
|
+
covers a local example and native execution.
|
|
140
|
+
|
|
141
|
+
Ordinary Codex and Copilot paragraph-removal experiments passed fresh confirmation at a
|
|
142
|
+
20-percentage-point preservation margin. See the
|
|
143
|
+
[measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#saved-session-experiments)
|
|
144
|
+
for completed and unsuccessful outcomes, costs and qualification limits, and the
|
|
132
145
|
[worked investigation](https://github.com/eberkut/ithrl/blob/main/docs/research/ordinary-session-investigation.md)
|
|
133
|
-
|
|
134
|
-
The measurements distinguish private runtime qualification from integrated support and report
|
|
135
|
-
resource usage and unsuccessful outcomes alongside completed runs. Check
|
|
146
|
+
for a developer's interpretation. Check
|
|
136
147
|
[native compatibility](https://github.com/eberkut/ithrl/blob/main/docs/native-runs.md#compatibility-and-measurements)
|
|
137
|
-
|
|
148
|
+
before using an installed harness.
|
|
138
149
|
|
|
139
150
|
## Documentation
|
|
140
151
|
|
|
141
152
|
Read the [documentation site](https://eberkut.github.io/ithrl/) or browse the guides below.
|
|
142
153
|
|
|
143
154
|
- [Native runs](https://github.com/eberkut/ithrl/blob/main/docs/native-runs.md): session selection,
|
|
144
|
-
|
|
155
|
+
shared prerequisites and compatibility.
|
|
156
|
+
- [Native reduction](https://github.com/eberkut/ithrl/blob/main/docs/reduction.md): prepare,
|
|
157
|
+
inspect and execute a context-removal experiment.
|
|
158
|
+
- [Context fuzzing](https://github.com/eberkut/ithrl/blob/main/docs/fuzzing.md): try the local
|
|
159
|
+
example and run a native text-append campaign.
|
|
145
160
|
- [Results and privacy](https://github.com/eberkut/ithrl/blob/main/docs/results.md): saved files,
|
|
146
161
|
result interpretation, budgets, resume and sensitive data.
|
|
147
162
|
- [Contributing](https://github.com/eberkut/ithrl/blob/main/CONTRIBUTING.md): issue-first
|
|
@@ -62,7 +62,15 @@ Use a new directory to repeat the example; extraction refuses to overwrite an ex
|
|
|
62
62
|
## Commands
|
|
63
63
|
|
|
64
64
|
- `ithrl extract` browses saved sessions and creates a case from a selected decision.
|
|
65
|
+
- `ithrl prepare CASE` generates a private reduction configuration from a preflight receipt and
|
|
66
|
+
statistical settings. Follow the
|
|
67
|
+
[preparation procedure](https://github.com/eberkut/ithrl/blob/main/docs/reduction.md).
|
|
65
68
|
- `ithrl shrink CASE` searches for context removals that preserve the selected behavior.
|
|
69
|
+
- `ithrl inspect sections CASE` lists editable text fields and paragraph boundaries without
|
|
70
|
+
displaying text or requiring a reduction result.
|
|
71
|
+
- `ithrl inspect reacquire SESSION` compares recorded operations and captured results across
|
|
72
|
+
compaction. See
|
|
73
|
+
[passive inspection](https://github.com/eberkut/ithrl/blob/main/docs/results.md#inspect-recorded-compaction).
|
|
66
74
|
- `ithrl inspect retained CASE --result RESULT` explains retained units and recorded removal trials
|
|
67
75
|
from a saved reduction without model access. See
|
|
68
76
|
[results and privacy](https://github.com/eberkut/ithrl/blob/main/docs/results.md#inspect-retained-context).
|
|
@@ -82,33 +90,40 @@ Returned actions are observed and denied. Cases contain copied session content;
|
|
|
82
90
|
|
|
83
91
|
## Supported inputs
|
|
84
92
|
|
|
85
|
-
| Input | Extracted decisions | Reduction | Fuzzing |
|
|
86
|
-
| --- | --- | --- | --- |
|
|
87
|
-
| Bundled synthetic example | Synthetic tool request | Deterministic local runner | Local fake runner |
|
|
88
|
-
| Codex CLI rollouts | Final responses | Experimental native runner; [paragraph-removal measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#ordinary-session-paragraph-removal) | Experimental prior-user text appends |
|
|
89
|
-
| Copilot CLI session events | Final responses and shell requests | Experimental final-response paragraph removal and denied `bash` runner | Experimental prior-user text appends for final-response targets |
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
93
|
+
| Input | Extracted decisions | Reduction | Fuzzing | Passive inspection |
|
|
94
|
+
| --- | --- | --- | --- | --- |
|
|
95
|
+
| Bundled synthetic example | Synthetic tool request | Deterministic local runner | Local fake runner | — |
|
|
96
|
+
| Codex CLI rollouts | Final responses | Experimental native runner; [paragraph-removal measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#ordinary-session-paragraph-removal) | Experimental prior-user text appends | Exact recorded-operation and captured-result comparisons across compaction |
|
|
97
|
+
| Copilot CLI session events | Final responses and shell requests | Experimental final-response paragraph removal and denied `bash` runner | Experimental prior-user text appends for final-response targets | Exact recorded-operation and captured-result comparisons across compaction |
|
|
98
|
+
|
|
99
|
+
Passive inspection reads saved evidence without a harness or model access. The
|
|
100
|
+
[worked investigation](https://github.com/eberkut/ithrl/blob/main/docs/research/passive-compaction-investigation.md)
|
|
101
|
+
records the qualified native profiles, ordinary examples and interpretation limits.
|
|
102
|
+
|
|
103
|
+
Native fuzzing appends text to eligible prior user input while preserving the fixed triggering
|
|
104
|
+
input. The
|
|
105
|
+
[fuzzing walkthrough](https://github.com/eberkut/ithrl/blob/main/docs/fuzzing.md)
|
|
106
|
+
covers a local example and native execution.
|
|
107
|
+
|
|
108
|
+
Ordinary Codex and Copilot paragraph-removal experiments passed fresh confirmation at a
|
|
109
|
+
20-percentage-point preservation margin. See the
|
|
110
|
+
[measurements](https://github.com/eberkut/ithrl/blob/main/docs/measurements.md#saved-session-experiments)
|
|
111
|
+
for completed and unsuccessful outcomes, costs and qualification limits, and the
|
|
99
112
|
[worked investigation](https://github.com/eberkut/ithrl/blob/main/docs/research/ordinary-session-investigation.md)
|
|
100
|
-
|
|
101
|
-
The measurements distinguish private runtime qualification from integrated support and report
|
|
102
|
-
resource usage and unsuccessful outcomes alongside completed runs. Check
|
|
113
|
+
for a developer's interpretation. Check
|
|
103
114
|
[native compatibility](https://github.com/eberkut/ithrl/blob/main/docs/native-runs.md#compatibility-and-measurements)
|
|
104
|
-
|
|
115
|
+
before using an installed harness.
|
|
105
116
|
|
|
106
117
|
## Documentation
|
|
107
118
|
|
|
108
119
|
Read the [documentation site](https://eberkut.github.io/ithrl/) or browse the guides below.
|
|
109
120
|
|
|
110
121
|
- [Native runs](https://github.com/eberkut/ithrl/blob/main/docs/native-runs.md): session selection,
|
|
111
|
-
|
|
122
|
+
shared prerequisites and compatibility.
|
|
123
|
+
- [Native reduction](https://github.com/eberkut/ithrl/blob/main/docs/reduction.md): prepare,
|
|
124
|
+
inspect and execute a context-removal experiment.
|
|
125
|
+
- [Context fuzzing](https://github.com/eberkut/ithrl/blob/main/docs/fuzzing.md): try the local
|
|
126
|
+
example and run a native text-append campaign.
|
|
112
127
|
- [Results and privacy](https://github.com/eberkut/ithrl/blob/main/docs/results.md): saved files,
|
|
113
128
|
result interpretation, budgets, resume and sensitive data.
|
|
114
129
|
- [Contributing](https://github.com/eberkut/ithrl/blob/main/CONTRIBUTING.md): issue-first
|
ithrl-0.4.0/ROADMAP.md
ADDED
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
# Roadmap
|
|
2
|
+
|
|
3
|
+
`ithrl` tests which context preserves a selected agent decision and which changes alter it.
|
|
4
|
+
Development priorities depend on useful results, reliable measurement and measured execution cost.
|
|
5
|
+
|
|
6
|
+
The [supported-inputs table](README.md#supported-inputs) describes current public capabilities.
|
|
7
|
+
The roadmap records completed work and future direction. Future version labels establish an
|
|
8
|
+
intended sequence, with scope and release dates decided from evidence.
|
|
9
|
+
|
|
10
|
+
## Harness priorities
|
|
11
|
+
|
|
12
|
+
Codex and GitHub Copilot are the primary harness integrations. Closing feature gaps between them
|
|
13
|
+
is the current priority.
|
|
14
|
+
|
|
15
|
+
## Completed: v0.2 context fuzzing
|
|
16
|
+
|
|
17
|
+
The `ithrl fuzz` command explores bounded context changes, simplifies interventions and freshly
|
|
18
|
+
confirms **at most one finding per campaign**: a declared decrease in the target's match probability
|
|
19
|
+
among valid observations. This is conditional experimental evidence, not proof of a model defect,
|
|
20
|
+
unique cause or semantic equivalence.
|
|
21
|
+
|
|
22
|
+
Codex and Copilot prior-user append workflows have completed synthetic qualification. Their
|
|
23
|
+
[measurements](docs/measurements.md#native-context-fuzzing) cover workloads, results and costs;
|
|
24
|
+
[fuzzing research](docs/research/context-fuzzing.md) owns the method and assumptions.
|
|
25
|
+
[Practical settings](https://github.com/eberkut/ithrl/issues/10) and
|
|
26
|
+
[Copilot qualification](https://github.com/eberkut/ithrl/issues/29) record acceptance evidence.
|
|
27
|
+
|
|
28
|
+
See the [fuzzing capability issue](https://github.com/eberkut/ithrl/issues/5) and
|
|
29
|
+
[`v0.2.0` milestone](https://github.com/eberkut/ithrl/milestone/1) for scope and acceptance
|
|
30
|
+
criteria.
|
|
31
|
+
|
|
32
|
+
[Version 0.2.0](https://github.com/eberkut/ithrl/releases/tag/v0.2.0) is published, with
|
|
33
|
+
[release preparation and publication](https://github.com/eberkut/ithrl/issues/11) complete.
|
|
34
|
+
Broader mutation families and multiple-finding campaigns need their own evidence and prospective
|
|
35
|
+
error allocation.
|
|
36
|
+
|
|
37
|
+
## Completed: v0.3 deeper reduction
|
|
38
|
+
|
|
39
|
+
Provider-free retained-context inspection and finer removal within supported prior-user text are
|
|
40
|
+
implemented, with Codex and Copilot final-response workflows. Ordinary saved-session qualification
|
|
41
|
+
and a [worked investigation](docs/research/ordinary-session-investigation.md) are complete; the
|
|
42
|
+
[measurements](docs/measurements.md#ordinary-session-paragraph-removal) record outcomes, costs and
|
|
43
|
+
limits. [Version 0.3.0](https://github.com/eberkut/ithrl/releases/tag/v0.3.0) is published.
|
|
44
|
+
[Issue #12](https://github.com/eberkut/ithrl/issues/12) records the milestone's scope, acceptance
|
|
45
|
+
and release evidence.
|
|
46
|
+
|
|
47
|
+
Ablation, alternative reproducers, interaction analysis and statistically confirmed single-removal
|
|
48
|
+
minimality remain deferred beyond v0.3.
|
|
49
|
+
|
|
50
|
+
## Awaiting release: v0.4 preparation and passive compaction inspection
|
|
51
|
+
|
|
52
|
+
[Experiment preparation](https://github.com/eberkut/ithrl/issues/48) is implemented in
|
|
53
|
+
[PR #66](https://github.com/eberkut/ithrl/pull/66), keeping statistical choices, source identities
|
|
54
|
+
and execution limits explicit. It is merged on main and awaits release in
|
|
55
|
+
[v0.4.0](https://github.com/eberkut/ithrl/milestone/3).
|
|
56
|
+
|
|
57
|
+
Passive compaction inspection for ordinary Codex and Copilot evidence is implemented under #15,
|
|
58
|
+
with implementation in #64 and qualification in #65. Its
|
|
59
|
+
[provider-free feasibility assessment](https://github.com/eberkut/ithrl/issues/15#issuecomment-6782516915)
|
|
60
|
+
led to the current scope. The command has completed
|
|
61
|
+
[ordinary-session qualification](docs/research/passive-compaction-investigation.md) through the
|
|
62
|
+
CLI and both installed-artifact origins. It reports repeated recorded
|
|
63
|
+
operations and equal or changed captured results without inferring avoidable work, causal effects or
|
|
64
|
+
savings.
|
|
65
|
+
|
|
66
|
+
The project-wide documentation review is complete in
|
|
67
|
+
[PR #72](https://github.com/eberkut/ithrl/pull/72). All assigned v0.4 issues are complete on main;
|
|
68
|
+
release preparation and publication remain. The investigations below have no release assignment and
|
|
69
|
+
do not expand v0.4 scope.
|
|
70
|
+
|
|
71
|
+
| Scope | Issue | Expected outcome |
|
|
72
|
+
| --- | --- | --- |
|
|
73
|
+
| Completed on main; awaiting v0.4 | [#48: experiment preparation](https://github.com/eberkut/ithrl/issues/48) | Generate and inspect private native reduction configurations without copying target/oracle blocks or digests by hand |
|
|
74
|
+
| Completed on main; awaiting v0.4 | [#15: passive compaction inspection](https://github.com/eberkut/ithrl/issues/15) | Useful provider-free reports for both harnesses, with explicit operation/result relations and interpretation limits |
|
|
75
|
+
| Completed on main; awaiting v0.4 | [#64: implement passive inspection](https://github.com/eberkut/ithrl/issues/64) | Bounded source projections, matching/report contracts and `inspect reacquire`; experimental reconstruction support is unchanged |
|
|
76
|
+
| Qualification complete; awaiting v0.4 | [#65: qualify passive inspection](https://github.com/eberkut/ithrl/issues/65) | Ordinary sources, counterexamples and both installed-artifact origins validated; worked interpretation recorded |
|
|
77
|
+
| Completed on main; awaiting v0.4 | [#67: performance and simplification](https://github.com/eberkut/ithrl/issues/67) | Project-wide performance fixes, permanent local-workload benchmarks and durable coding guidance, merged in [PR #71](https://github.com/eberkut/ithrl/pull/71) |
|
|
78
|
+
| Completed on main; awaiting v0.4 | [#68: documentation and writing guidance](https://github.com/eberkut/ithrl/issues/68) | Project-wide documentation corrections and concise writing guidance, merged in [PR #72](https://github.com/eberkut/ithrl/pull/72) |
|
|
79
|
+
| Research; no release assigned | [#16: broader workloads and costs](https://github.com/eberkut/ithrl/issues/16) | Measure source eligibility, useful developer conclusions, reliability and full costs, including unsuccessful sources and runs |
|
|
80
|
+
| Candidate; no release assigned | [#13: comparative debugging](https://github.com/eberkut/ithrl/issues/13) | A comparability contract, paired workloads, interaction example and bounded validation proposal; decide whether to implement or defer |
|
|
81
|
+
| Candidate; no release assigned | [#73: sampled inspection metadata](https://github.com/eberkut/ithrl/issues/73) | Persist reconstruction and policy metadata bound to plan identity for offline reports, distinguishing declarations from recorded attempt evidence |
|
|
82
|
+
| Assessment; no release assigned | [#74: tool-output interventions](https://github.com/eberkut/ithrl/issues/74) | Assess native output visibility, source bindings and a useful localized intervention before proposing a new edit domain |
|
|
83
|
+
| Deferred; no release assigned | [#14: additional harnesses](https://github.com/eberkut/ithrl/issues/14) | Assess Claude Code, OpenCode and Pi before choosing an integration order and qualification scope |
|
|
84
|
+
| Deferred; no release assigned | [#18: portable archives and privacy](https://github.com/eberkut/ithrl/issues/18) | Define a useful sharing workflow, archive format and deliberate privacy review before implementation |
|
|
85
|
+
| Research; no release assigned | [#17: baseline sharing](https://github.com/eberkut/ithrl/issues/17) | Reconsider reuse only if useful complete-workflow savings justify the statistical and recovery complexity |
|
|
86
|
+
| Deferred research; no release assigned | [#61: ablation](https://github.com/eberkut/ithrl/issues/61) | Define retained-reproducer and full-original ablation, with distinct baselines and justified claims |
|
|
87
|
+
| Deferred research; no release assigned | [#62: alternative reproducers and interactions](https://github.com/eberkut/ithrl/issues/62) | Assess useful differences between preserving contexts, redundancy and joint effects, with bounded discovery and confirmation |
|
|
88
|
+
| Deferred research; no release assigned | [#63: confirmed single-removal minimality](https://github.com/eberkut/ithrl/issues/63) | Define the neighbor family, statistical claim, error allocation and complete comparison budget |
|
|
89
|
+
|
|
90
|
+
## Next investigations
|
|
91
|
+
|
|
92
|
+
Start with #16's provider-free workload preparation for both harnesses. Freeze debugging questions
|
|
93
|
+
and selection criteria before choosing successful examples; retain exclusions and unsuccessful
|
|
94
|
+
investigations.
|
|
95
|
+
[ctx](https://github.com/ctxrs/ctx/blob/33ba4645b620fd89d138c6db1fabf33096a0ee16/docs/search.md) may
|
|
96
|
+
help locate sessions, but ithrl must independently snapshot and validate native evidence. Measure
|
|
97
|
+
useful developer conclusions alongside reduction, reliability and cost.
|
|
98
|
+
|
|
99
|
+
Use the initial cohort to inform #13's good/bad-case comparison design and #74's tool-output
|
|
100
|
+
assessment; neither must wait for the entire study. #74 must distinguish original output,
|
|
101
|
+
delivered representation and saved evidence before proposing edits. #73's reporting improvement
|
|
102
|
+
is independent. Execution and release scope remain separate decisions in the owning issues.
|
|
103
|
+
|
|
104
|
+
The review of
|
|
105
|
+
[SkillOpt's training method](https://github.com/microsoft/SkillOpt/blob/45f65ca04f5f6713bca7b9797b495696ba812751/docs/guide/training-loop.md)
|
|
106
|
+
and
|
|
107
|
+
[Sift's tool-output benchmarks](https://github.com/ctxrs/sift/blob/c22fb1b3747352dba8f8323c057d60e7afd92bf5/benchmarks/results/2026-09-21-release-v0.4.0/README.md)
|
|
108
|
+
motivated better candidate selection and tool-output investigation. Their task-performance and
|
|
109
|
+
output-size results do not establish ithrl's debugging usefulness. No search engine or skill
|
|
110
|
+
optimizer is planned. Fuzzing already accepts supplied `stale_text`; #16 should establish a need for
|
|
111
|
+
distinct proposals tied to specific fields before creating further work. Passive inspection retains
|
|
112
|
+
its qualified exact-comparison contract; decoded or normalized matching needs a separate policy.
|
|
113
|
+
|
|
114
|
+
## Release history and current scope
|
|
115
|
+
|
|
116
|
+
### Deeper reduction
|
|
117
|
+
|
|
118
|
+
Reduction within a message must preserve its native structure and protocol dependencies. Retained
|
|
119
|
+
context inspection should distinguish protected or frozen material, dependency constraints,
|
|
120
|
+
rejected removal trials, incomplete evidence and untested work. Trial explanations must identify
|
|
121
|
+
the tested context; a rejected sampled screen does not establish behavioral necessity.
|
|
122
|
+
|
|
123
|
+
[Ablation](https://github.com/eberkut/ithrl/issues/61) of a retained reproducer and of the full
|
|
124
|
+
original answers different questions and must report its baseline.
|
|
125
|
+
[Alternative reproducer and interaction research](https://github.com/eberkut/ithrl/issues/62) can
|
|
126
|
+
expose several preserving contexts and joint effects; none is automatically a unique explanation.
|
|
127
|
+
Deterministic single-removal minimality and
|
|
128
|
+
[statistically confirmed minimality](https://github.com/eberkut/ithrl/issues/63) are distinct
|
|
129
|
+
claims. The latter requires additional complete comparisons, error allocation and a budget that
|
|
130
|
+
covers all advertised checks.
|
|
131
|
+
|
|
132
|
+
### Comparative debugging and another integration
|
|
133
|
+
|
|
134
|
+
`compare GOOD_CASE BAD_CASE` should search supported context differences with dependency-valid
|
|
135
|
+
interventions. It must establish comparability of target, oracle, model/provider, request surface,
|
|
136
|
+
environment and effective runner policy before attributing a regression to context. Differences in
|
|
137
|
+
model, harness or execution environment remain explicit experimental factors.
|
|
138
|
+
|
|
139
|
+
Claude Code, OpenCode and Pi are planned additional integrations. Claude Code and OpenCode have
|
|
140
|
+
promising portability evidence in the [feasibility study](docs/research/feasibility.md); Pi needs an
|
|
141
|
+
initial applicability assessment. Each requires an adapter, faithful reconstruction, isolation,
|
|
142
|
+
calibration and a useful live workflow before becoming supported. Their order and release scope
|
|
143
|
+
remain open; [the integration issue](https://github.com/eberkut/ithrl/issues/14) tracks assessment,
|
|
144
|
+
prioritization and implementation. Add them incrementally while maintaining Codex and Copilot
|
|
145
|
+
parity.
|
|
146
|
+
|
|
147
|
+
### Passive reacquisition analysis
|
|
148
|
+
|
|
149
|
+
The `inspect reacquire` implementation compares recorded operations and captured results
|
|
150
|
+
across explicit compaction boundaries in one saved Codex or Copilot session. It starts with exact
|
|
151
|
+
comparisons, keeps content and display scopes separate, and reports unknown state and completeness.
|
|
152
|
+
Default output excludes private paths, arguments and text; detail requires bounded opt-in. It
|
|
153
|
+
starts no harness, opens no referenced resources and makes no provider calls.
|
|
154
|
+
|
|
155
|
+
Resume, fork, handoff and cross-harness matching remain later directions. Any normalized or
|
|
156
|
+
heuristic matcher needs its own policy and qualification. Repeated operations or equal captured
|
|
157
|
+
text do not prove avoidable retrieval, forgetting or effects caused by compaction. Passive
|
|
158
|
+
inspection leaves active Codex compaction reconstruction unsupported; implementing agent handoffs
|
|
159
|
+
is separate work.
|
|
160
|
+
|
|
161
|
+
## Further capability candidates
|
|
162
|
+
|
|
163
|
+
- Further mutation operations: supported whitespace changes, independent reordering and adding
|
|
164
|
+
stale or irrelevant observations outside prior-user text. State whether each tests expected
|
|
165
|
+
invariance or introduces intentional stress; a mutator name cannot prove semantic equivalence.
|
|
166
|
+
- Optional `record` adapters for future sessions when ordinary saved evidence is insufficient.
|
|
167
|
+
Prior instrumentation must remain unnecessary for supported post-hoc extraction.
|
|
168
|
+
- Trajectory runners where faithful execution seams and safe action policies exist. The current
|
|
169
|
+
single-decision experiment does not claim to reproduce a complete agent trajectory.
|
|
170
|
+
- Custom command/Python runners and additional provider integrations with the same observation,
|
|
171
|
+
safety, identity and accounting contracts.
|
|
172
|
+
- [Portable export/import and deliberate redaction](https://github.com/eberkut/ithrl/issues/18).
|
|
173
|
+
Exported experiments must state
|
|
174
|
+
external state requirements and omitted sensitive evidence; portable evidence does not require
|
|
175
|
+
a universal agent-session format.
|
|
176
|
+
- Optional editor/agent integrations and handoff analysis that compose the CLI rather than
|
|
177
|
+
introducing separate experimental semantics.
|
|
178
|
+
|
|
179
|
+
These candidates have no assigned release.
|
|
180
|
+
[Broader native workloads and costs](https://github.com/eberkut/ithrl/issues/16) and
|
|
181
|
+
[baseline-sharing strategies](https://github.com/eberkut/ithrl/issues/17) remain research
|
|
182
|
+
opportunities. Synthetic qualification covers a planted approximately 10k workload;
|
|
183
|
+
[saved-session experiments](docs/measurements.md#saved-session-experiments) add selected larger
|
|
184
|
+
contexts, including incomplete and inconclusive outcomes. Neither establishes a general 100k
|
|
185
|
+
workload envelope. Qualification at that scale remains an optional expansion with a separate
|
|
186
|
+
cost proposal.
|
|
187
|
+
|
|
188
|
+
## Planning and release policy
|
|
189
|
+
|
|
190
|
+
GitHub issues track implementation scope, acceptance criteria and dependencies. Release milestones
|
|
191
|
+
group this work, and the [public Project](https://github.com/users/eberkut/projects/1/views/4) shows
|
|
192
|
+
status. See [decision records](docs/decisions/index.md) for rationale,
|
|
193
|
+
[research](docs/research/index.md) for methods and evidence, and
|
|
194
|
+
[architecture](docs/architecture.md) for the current design.
|
|
195
|
+
|
|
196
|
+
Keep each issue's problem, rationale, evidence, scope and stop point self-contained. Roadmap entries
|
|
197
|
+
explain direction and link to those owners; neither should depend on chat history or local plan
|
|
198
|
+
files.
|
|
199
|
+
|
|
200
|
+
Before 1.0, project-owned schema tags remain at `1` while formats evolve directly. Breaking changes
|
|
201
|
+
are acceptable; current-format validation and evidence identity remain strict. A future stable
|
|
202
|
+
release needs stable case/runner/oracle contracts, useful adapters, documented statistics,
|
|
203
|
+
portable artifacts, reviewed export privacy, migrations and practical ordinary-case cost evidence.
|
|
204
|
+
There is no fixed 1.0 feature list or release date.
|