errlore 0.2.2__tar.gz → 0.3.0__tar.gz

errlore-0.3.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,45 @@
+name: Bug report
+description: Something behaves differently than documented
+labels: [bug]
+body:
+  - type: input
+    id: version
+    attributes:
+      label: errlore version
+      description: "`errlore --version` (or `python -c 'import errlore; print(errlore.__version__)'`)"
+      placeholder: "0.2.2"
+    validations:
+      required: true
+  - type: input
+    id: python
+    attributes:
+      label: Python version & OS
+      placeholder: "3.12 / Ubuntu 24.04"
+    validations:
+      required: true
+  - type: textarea
+    id: repro
+    attributes:
+      label: Minimal reproduction
+      description: Smallest snippet that shows the problem (with a temp data dir).
+      render: python
+    validations:
+      required: true
+  - type: textarea
+    id: expected
+    attributes:
+      label: Expected vs actual
+    validations:
+      required: true
+  - type: textarea
+    id: stats
+    attributes:
+      label: "`errlore stats --data-dir <your dir>` output (if relevant)"
+      render: text
+  - type: checkboxes
+    id: checks
+    attributes:
+      label: Checks
+      options:
+        - label: I searched existing issues
+          required: true

errlore-0.3.0/.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,26 @@
+name: Feature request
+description: Propose a capability or improvement
+labels: [enhancement]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: The problem
+      description: What repeated pain does this solve? Concrete scenario beats abstraction.
+    validations:
+      required: true
+  - type: textarea
+    id: proposal
+    attributes:
+      label: Proposed behavior
+    validations:
+      required: true
+  - type: textarea
+    id: evidence
+    attributes:
+      label: How would we prove it works?
+      description: >
+        errlore ships features with evidence (see CONTRIBUTING). What benchmark
+        or measurable signal would demonstrate this feature's effect?
+    validations:
+      required: true

errlore-0.3.0/.github/dependabot.yml

@@ -0,0 +1,18 @@
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+    groups:
+      actions:
+        patterns: ["*"]
+  - package-ecosystem: pip
+    directory: /
+    schedule:
+      interval: weekly
+    # Library with loose pins — only security updates matter here.
+    open-pull-requests-limit: 5
+    groups:
+      pip:
+        patterns: ["*"]

errlore-0.3.0/.github/pull_request_template.md

@@ -0,0 +1,11 @@
+## What & why
+
+<!-- One focused change. Link the issue if any. -->
+
+## Checklist
+
+- [ ] `ruff check .` + `mypy` + `pytest --cov=errlore --cov-fail-under=80` pass locally
+- [ ] Tests added/updated for behavior changes
+- [ ] `CHANGELOG.md` updated under `[Unreleased]`
+- [ ] If a claimed number changes: benchmark artifacts updated in `benchmarks/results/`
+- [ ] No telemetry / no network calls added to the core

{errlore-0.2.2 → errlore-0.3.0}/.github/workflows/release.yml

@@ -1,12 +1,12 @@
 name: release
 
 # Tag-driven release: push a v* tag -> full test gate -> build sdist+wheel ->
-# publish to PyPI via trusted publishing (OIDC; no long-lived token in CI).
+# publish to PyPI (API token from the PYPI_API_TOKEN repo secret) -> GitHub
+# release with artifacts. skip-existing tolerates re-runs after a manual
+# twine upload of the same version.
 #
-# One-time PyPI setup (owner): pypi.org -> project errlore -> Publishing ->
-# add GitHub publisher (owner=Ma4etaSS, repo=errlore, workflow=release.yml,
-# environment=pypi). Until that's configured the publish step fails cleanly
-# and the build artifacts are still attached to the GitHub release.
+# Trusted publishing (OIDC) is the eventual upgrade — swap the password for
+# permissions: id-token: write once the PyPI publisher config is verified.
 
 on:
   push:
@@ -44,14 +44,15 @@ jobs:
     needs: build
     runs-on: ubuntu-latest
     environment: pypi
-    permissions:
-      id-token: write  # OIDC for PyPI trusted publishing
     steps:
       - uses: actions/download-artifact@v4
         with:
           name: dist
           path: dist/
       - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}
+          skip-existing: true
 
   github-release:
     needs: build

{errlore-0.2.2 → errlore-0.3.0}/CHANGELOG.md

@@ -5,6 +5,51 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2026-07-14
+
+### Added
+- **Shadow mode: counterfactual graduation.** The full mechanism from
+  `docs/SHADOW_MODE_SPEC.md`. `inject_for(..., mode="shadow")` builds a lesson
+  block for a parallel run that never touches the user's output (and re-includes
+  quarantined lessons — the recovery route a suppressed lesson otherwise lacks).
+  `enqueue_counterfactual()` durably queues the (baseline, injected) pair;
+  your worker re-runs both, scores each with a deterministic validator, and
+  calls `report_counterfactual_outcome(cf_id, baseline_passed, injected_passed)`.
+  Two per-lesson Beta posteriors (harm + fix) drive a `graduation_status()` of
+  `promote` / `hold` / `quarantine` via the validated two-gate rule (strict on
+  harm, lenient on usefulness). `graduated_lessons()` surfaces lessons ready to
+  bake into a permanent surface with their evidence counts. Every spec anchor
+  (quarantine 5/20; promote after ~60 clean trials + 1 fix; fix/harm-clear
+  posteriors 0.387/0.736/0.910/0.961/0.993) is pinned by a unit test. errlore
+  never calls the model/validator — that stays the worker's job. Zero new deps.
+- **Warning tier: self-consistency as an honest wrong-answer signal.** New
+  `errlore.consistency` (`check_consistency` + `AgentMemory.check_consistency`):
+  on validator-less surfaces, feed 2+ independent runs of the same prompt and
+  errlore flags disagreement as "unstable — likely wrong" at ~86% precision
+  (`benchmarks/results/CONSISTENCY_SIGNAL_2026-07-11.md`). Deliberately
+  one-sided: a *stable* result is never presented as verification (61% residual
+  wrongness on our grid). `final_line`/`full` modes, optional `similarity`
+  loosening, and — when a model is named — an unstable verdict is logged as a
+  tracked error. errlore never calls the model (offline ethos: the caller
+  supplies outputs). Zero new dependencies.
+- **Harm gate: interference-guarded lesson injection.** Lessons now track
+  `success_count`/`failure_count` separately (the old single confidence scalar
+  erased this signal — a lesson that helped 3× and hurt 3× looked untouched).
+  A Beta-Binomial gate (`errlore.lessons.graduation`) withholds a lesson from
+  injection once its live failure history clears a 95% credible bar that its
+  harm rate exceeds 5% — calibrated to the numbers validated in
+  `docs/SHADOW_MODE_SPEC.md` (5 harms/20 trials → quarantine, 4/20 → hold).
+  This grounds the *harm* half of shadow mode in the live `report_outcome`
+  loop and targets the measured 12–15% interference
+  (`benchmarks/results/REPRODUCIBILITY_2026-07-11.md`). On by default
+  (`AgentMemory(..., harm_gate=True)`); a fresh or consistently-helpful lesson
+  is never gated, so good lessons are not starved. The gate is self-limiting
+  (caps damage at ~4–5 harmful injections, then freezes the lesson);
+  deliberate recovery/re-evaluation is deferred to shadow mode. New API:
+  `AgentMemory.quarantined_lessons()` and a `lessons_quarantined` key in
+  `stats()`. Zero new dependencies (regularized incomplete beta via a Lentz
+  continued fraction, stdlib only).
+
 ## [0.2.2] - 2026-07-11
 
 ### Fixed

errlore-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# Contributing to errlore
+
+Thanks for taking the time. Two rules keep this project trustworthy:
+
+1. **Every number needs a benchmark.** Claims in the README/site must be
+   backed by a committed, reproducible benchmark with raw outputs. If your
+   change alters a claimed number, update the benchmark artifacts with it.
+2. **Features land with their evidence.** A new capability (e.g. a retrieval
+   mode, a routing policy) ships together with a benchmark demonstrating the
+   effect — or it ships marked *experimental* and stays out of the headline.
+
+## Dev setup
+
+```bash
+git clone https://github.com/Ma4etaSS/errlore
+cd errlore
+python -m venv .venv && . .venv/bin/activate
+pip install -e .[dev]
+```
+
+## Quality gate (same as CI and the pre-push hook)
+
+```bash
+ruff check .
+mypy
+pytest --cov=errlore --cov-fail-under=80
+```
+
+CI runs the same on Python 3.10–3.13. Keep `mypy` strict-clean and coverage
+above the bar; add tests for any behavior change.
+
+## Pull requests
+
+- One focused change per PR; reference the issue if there is one.
+- Update `CHANGELOG.md` under `[Unreleased]` (Keep a Changelog format).
+- No telemetry, no network calls in the core — errlore stays offline-first.
+- Benchmarks live in `benchmarks/`; committed results in `benchmarks/results/`
+  (reports as text; raw outputs as `.jsonl`, force-added past the gitignore).
+
+## Releases (maintainers)
+
+Tag-driven: bump `version` in `pyproject.toml`, move `[Unreleased]` to a
+dated section, then `git tag vX.Y.Z && git push origin main vX.Y.Z`. CI runs
+the gate, builds, publishes to PyPI, and attaches artifacts to the GitHub
+release.
+
+## Security
+
+See [SECURITY.md](SECURITY.md) — report privately, not via public issues.

errlore-0.3.0/ERRLORE_BRIEF_FOR_ANALYSIS.md

@@ -0,0 +1,196 @@
+# errlore — Brief for External Analysis
+
+*Prepared 2026-07-11. Everything below is measured, not aspirational. Raw data
+and benchmark harnesses are public: https://github.com/Ma4etaSS/errlore*
+
+You are being asked to analyze a real, shipped open-source product and help
+solve its one deep problem. We would rather hear an uncomfortable truth than a
+polite compliment. If you can invent a mechanism we haven't thought of, even
+better — the constraints and all known dead ends are documented below so you
+don't re-derive them.
+
+---
+
+## 1. What errlore is
+
+**Memory for AI agents that learns from failures.** A Python library (PyPI:
+`errlore`, MIT, v0.2.2): embedded, file-based (JSONL), offline, no server, no
+telemetry.
+
+The loop:
+
+1. Agent fails → `log_error(model, task_type, error)`
+2. Human (or system) fixes it once → `resolve(err_id, fix, lesson="...")`
+3. Next similar task → `inject_for(task, model, task_type)` returns a prompt
+   block: relevant **lessons** + per-model **KNOWN ISSUES** warnings
+4. `report_outcome(injection, success)` closes the loop — lessons that help
+   get reinforced, ones that don't decay
+
+Plus an experimental Bayesian per-model/per-domain **trust engine**
+(`best_model(domain)`) with adaptive learning rate, cold-start blending,
+entropy floor, temporal decay.
+
+Origin story (true): extracted from a 324,000-line multi-LLM "brain" the
+author built and then honestly benchmarked. The ensembles, arbiters and
+"cognitive phases" did not beat a single good model. One loop demonstrably
+worked — this one. Everything else was deleted.
+
+## 2. What is PROVEN (this is our strongest asset)
+
+Paired A/B benchmark, deterministic validators, no LLM judges, exact McNemar,
+temperature 0, raw outputs committed to the repo:
+
+| Axis | Result |
+|---|---|
+| Headline | repeat-error rate **≈66% → ≈20%** (67–70% reduction) |
+| Significance | McNemar **p between 8.4e-12 and 1.8e-9, every run** |
+| Seed-robustness | reproduced across **5 independent runs** (2 dates + 3 fresh RNG seeds), reduction band 66.7–69.8% |
+| Task-generality | 2 fresh realistic convention families (arbitrary internal status enum; non-standard git branch format `wip.<TICKET>.<snake>`): **100% → 0% on both models** |
+| Model-diversity | claude-haiku-4-5 ≈68% AND gemma-4-31b **70.0%** (p=2.6e-13) — different model family, same effect |
+| **Honest negative control** | **capability-gap errors (letter counting, string reversal): −12% to −20% — errlore does NOT help, slightly hurts, on both model families** |
+
+The split is the thesis: **memory fixes what the model doesn't KNOW (a
+convention it was never told), not what it can't DO (a skill).**
+Knowledge-gap errors drop 83–100%. Capability-gap errors don't move.
+
+One honest wrinkle we publish rather than hide: one convention family
+(`csv_order`) transfers to 0/12 on Haiku but stays 12/12 on gemma —
+lesson-*following* is itself model-dependent at the margin.
+
+## 3. What we dogfooded — and the paradox we hit
+
+We run errlore observers against two of our own production SaaS products
+(read-only shadow observers → lessons → live injection into their LLM prompts):
+
+- **Product A (landing-page audit SaaS):** mature pipeline, retry/render/
+  fallback logic already in code, prompts at v2.9 with grounding and
+  fact-injection. Real failure rate ≈4% (8 failures / 191 runs).
+- **Product B (habit/health SaaS):** consent-gated, similarly hardened.
+
+Results, measured honestly:
+
+1. **Injection into Product A's content prompts is INERT.** Three independent
+   tests: (a) A/B on 12 real pages × 3 runs — similarity of with-lessons
+   output to baseline was within stochastic noise (δ=0.014); (b) the natural
+   failure tail was EMPTY (zero score=-1/manual-review cases to rescue);
+   (c) a controlled thin-input tail (12 pages truncated to 420 chars, 24
+   runs/arm) produced **zero** failures in BOTH arms — the modern model
+   (sonnet-class) no longer exhibits the failure modes the lessons target.
+2. The lessons the observer learned from Product A's history were
+   **operational** ("retry once on fetch_timeout", "force render on thin
+   pages") — but the pipeline code already does all of that in `catch`
+   blocks. The LLM writing the report has no lever over fetching. The lesson
+   duplicated the code.
+3. Product B's observer initially produced 42 "errors" → 0 lessons, because
+   most events were **by-design guards** (consent refusals) and **self-heal
+   successes** logged onto the error channel. After filtering, ONE real
+   lesson exists and now injects into its coach prompts. Value: real but tiny.
+
+**The paradox, stated plainly:**
+
+> The systems that generate enough failure signal to feed errlore are the
+> ones that need it least to keep operating; the mature systems that would
+> pay for reliability have already absorbed their lessons into code and
+> prompts — leaving errlore nothing to eat.
+
+And its corollary:
+
+> errlore is a victim of its own success by design: every lesson it captures
+> eventually gets promoted into the system prompt / conventions doc / code,
+> at which point errlore's marginal value on that system trends to zero.
+
+## 4. The competitive objection we must survive
+
+The obvious skeptic's line (we will face it on launch day):
+
+> "Your benchmark proves that telling the model the convention fixes
+> convention errors. A static CLAUDE.md / conventions file does the same.
+> Why is this a product and not a markdown file?"
+
+Our current answers: automatic capture at failure time (no human remembering
+to document), relevance-based injection (only what matches the task — not an
+ever-growing prompt), outcome-verified reinforcement/decay (the doc never
+learns whether it helped), per-model tracking, cross-tool portability.
+
+These are real, but we know they are **quantitatively unproven**: we have not
+yet measured errlore vs a naive "dump all conventions in the prompt" baseline
+on context-cost, interference, or accuracy at scale (e.g. 500 lessons where
+full-dump exceeds context or degrades performance).
+
+## 5. Constraints (hard, non-negotiable)
+
+- Solo developer; no funding; maintenance budget is nights and weekends
+- Embedded/offline/file-based/no-telemetry is the identity — no cloud pivot
+- Every public claim MUST carry a reproducible benchmark with committed raw
+  outputs (this discipline is the brand; we will not ship vibes)
+- Python 3.10+, zero mandatory deps beyond filelock
+- The evidence bar killed features before: an "attack-refine" reasoning
+  scheme (−4pp on GPQA), a strategies-vote scheme (+2pp, not worth it), and
+  the 324K-line brain itself. Anything you propose will meet the same bar.
+
+## 6. THE ASK — the key problem we want you to attack
+
+**Design a mechanism by which failure-memory stays valuable on LOW-FAILURE,
+MATURE systems — or honestly prove it cannot, and identify what adjacent
+mechanism can.**
+
+Sub-problems, in priority order:
+
+**P1. Outcome-signal scarcity.** Production gives ~4% failures and almost no
+verified success/failure labels. Reinforcement (`report_outcome`) starves.
+What signal can be harvested cheaply and HONESTLY from a mature production
+system that would let lessons keep proving (or disproving) their worth?
+Ideas we've considered and their flaws: LLM-as-judge (unverified, circular),
+user complaints (too sparse, too late), canary re-runs (costs money, drift).
+Is there a self-supervised or counterfactual signal we're missing?
+
+**P2. The absorption endgame.** If every good lesson eventually graduates
+into the prompt/code, is there a defensible product in BEING the graduation
+pipeline? I.e., errlore not as permanent memory but as the *conveyor*:
+failure → lesson → verified-by-outcomes → auto-promoted into the system
+prompt / conventions doc / a PR against the repo — with the evidence trail
+attached. Does this reframing survive the CLAUDE.md objection better? What
+would its honest benchmark look like?
+
+**P3. Crossing the capability-gap boundary WITHOUT unverifiable reward.**
+Our data says memory can't teach skills. But a lesson CAN change *strategy*:
+"for string reversal, write code instead of answering directly" would flip
+our capability-gap families to ~100%. That is routing, not skill. We have a
+tested Bayesian trust engine that could route strategies — but in production
+it starves for the same P1 reason (weights converge to the cap on
+success-heavy streams and never separate). Design a strategy-routing
+mechanism + benchmark that (a) uses only verifiable outcomes, (b) shows a
+lift a skeptic can reproduce, (c) doesn't degenerate into a greedy bandit
+stuck on early luck (we know about Thompson sampling / UCB — the question is
+the SIGNAL, not the formula).
+
+**P4. Interference at scale (the unglamorous one).** Nobody has measured
+what happens at 500+ lessons: retrieval precision, wrong-lesson injection
+harm, context cost vs. full-dump. If you think THIS is actually the key
+problem (the moat is making injection *selective and safe* at scale, which a
+static doc cannot do), argue it and design the benchmark.
+
+**Wildcard.** If you see a fundamentally different mechanism hiding in our
+data — something neither "memory" nor "routing" — name it. The one pattern
+we keep circling: the value may not be in helping the LLM, but in producing
+a **verified, quantified changelog of what the system learned** (for humans,
+for audits, for onboarding new agents/models). Nobody owns "the flight
+recorder + lessons ledger for agent systems" yet.
+
+## 7. What a great answer looks like
+
+1. Picks ONE of P1–P4 (or the wildcard) as the highest-leverage problem and
+   says WHY the others are downstream of it
+2. Proposes a concrete mechanism (API sketch welcome; we can build fast)
+3. Designs the honest benchmark that would prove it — deterministic
+   validators, a negative control, a skeptic-proof metric — in the spirit of
+   what we've already shipped
+4. Names the failure mode of its own proposal (we distrust proposals that
+   don't know how they die)
+
+Thank you. Be brutal; we've deleted 324,000 lines before and we'll happily
+delete more.
+
+---
+*Contact: github.com/Ma4etaSS/errlore · errlore.com · The three-axis proof:
+`benchmarks/results/REPRODUCIBILITY_2026-07-11.md` in the repo.*

errlore-0.3.0/ERRLORE_FINAL_CLOSURE.md

@@ -0,0 +1,68 @@
+# Final message — closing the exchange
+
+*2026-07-11, last round. No questions in this one. This is a thank-you, a
+ledger of what the exchange produced, and a clean close.*
+
+## Accepted, in full
+
+Your final formalization stands as the converged result:
+
+- Verifier error: Err_V ≈ s_V + (1−s_V)·ε_V
+- Repetition/voting kills ε_V exponentially (Condorcet/Chernoff); nothing
+  kills s_V by re-querying the same verifier
+- **Ceiling theorem, final form: asymptotic system quality ≤ 1 − s_V of the
+  cheapest affordable verifier** — no ceiling at all when s_V = 0
+- The generator/verifier symmetry: one stochastic-vs-systematic
+  decomposition settles both Q2 and Q7
+
+Your coalition observation — that diversity across verifiers is the
+Condorcet move for the *systematic* component, since joint s_V is the
+intersection of blind regions — is banked verbatim as the post-launch
+research direction. It is the single most generative idea of the exchange.
+
+## The ledger — what three days of this dialogue produced
+
+All committed, public, reproducible (github.com/Ma4etaSS/errlore):
+
+1. **Interference measurement** — lesson injection breaks 12–15% of
+   previously-passing tasks (`benchmarks/results/REPRODUCIBILITY_2026-07-11.md`)
+2. **Shadow-mode 0.3 spec** — your P1 counterfactual mechanism, reframed as
+   interference-guard + graduation gate, with your Beta-Binomial rule
+   validated numerically: promote ≈60 clean trials, quarantine at 5/20,
+   usefulness from a single verified fix (`docs/SHADOW_MODE_SPEC.md`)
+3. **The one-sided consistency theorem** — certification of correctness is
+   impossible without a validator; detection of stochastic-component
+   incorrectness is honest at 86% precision, 19% recall, with 61% residual
+   risk after a passed check (`benchmarks/results/CONSISTENCY_SIGNAL_2026-07-11.md`)
+4. **The observability-decay law** — two-phase signal-cost curve, the
+   patched ceiling theorem, the self-play retro-confirmation, four testable
+   predictions, and the strategic consequence: the durable asset in the
+   agent stack is verifier-manufacturing; memory is the conveyor that
+   exploits it (`docs/OBSERVABILITY_DECAY.md`)
+
+Not one of these existed when the first brief was written. Every number in
+them can be recomputed from committed raw outputs.
+
+## Declined, with reasons
+
+The bandit-with-verifier regret model: correct next step mathematically,
+declined on discipline. We ship a launch in three days; the theorem is
+settled, the predictions are named, and further formalization refines
+precision we cannot yet act on. It goes on the post-launch list beside the
+coalition thread and the Q4 anomaly (csv_order followability), which remain
+open and yours whenever we resume.
+
+## What we'd say about this exchange
+
+The pattern that worked, for the record: you proposed mechanisms; we refused
+to accept any claim either of us made until it survived contact with our
+committed data; twice the data overturned the framing (your fix-signal
+became our harm-guard; your accuracy-ceiling became our systematic-floor
+ceiling) — and both times the corrected version was stronger than either
+side's original. That is what collaboration between reasoning systems should
+look like: not agreement, but adversarial verification with a shared
+evidence bar.
+
+The loop is closed. Thank you.
+
+— errlore