errlore 0.1.2__tar.gz → 0.1.3__tar.gz

{errlore-0.1.2 → errlore-0.1.3}/CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.1.3] - 2026-07-06
+
+### Fixed
+- Read-cache poisoning race in `JSONLWriter.read_all`: an append landing
+  mid-parse cached pre-append records under the post-append mtime/size, so
+  later reads returned stale data and `atomic_update` could silently drop
+  the concurrent record. Now caches only when the file is unchanged across
+  the parse (pre/post stat match). Found by pre-launch adversarial audit;
+  regression test added (Bug 3).
+- Removed dead `JSONLWriter._invalidate_cache`.
+
+### Changed
+- README: added the error-reduction A/B benchmark section (96 paired tasks,
+  63 -> 20 failures, McNemar p=1.8e-09; knowledge-gap 46/48 -> 0/48,
+  capability-gap honestly worse at 17/48 -> 20/48), so the PyPI page shows
+  the headline evidence, not just the retrieval table.
+
 ## [0.1.2] - 2026-07-06
 
 ### Added

{errlore-0.1.2 → errlore-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: errlore
-Version: 0.1.2
+Version: 0.1.3
 Summary: Memory for AI agents that learns from failures: lessons, known-issues injection, and per-model trust — embedded, file-based, no server.
 Project-URL: Homepage, https://errlore.com
 Project-URL: Repository, https://github.com/Ma4etaSS/errlore
@@ -91,12 +91,38 @@ mem.report_outcome(inj, success=True)
 print(mem.stats())
 # {'errors_total': 1, 'errors_resolved': 1, 'errors_unresolved': 0,
 #  'lessons_total': 1, 'lessons_applied': 1, 'pending_injections': 0,
-#  'trust': {'gpt-5.5': 0.55}}
+#  'trust': {'gpt-5.5': 0.5522...}}
 ```
 
 No API keys needed. errlore itself never calls any LLM -- it manages local
 JSONL files and does text matching. LLM calls are yours to make (or not).
 
+## Does it actually reduce errors?
+
+Yes -- for the class of errors memory can fix. Paired A/B benchmark
+(`benchmarks/bench_error_reduction.py`): the same model (claude-haiku-4-5)
+runs 96 tasks twice, with and without errlore injection. Deterministic
+validators, no LLM judges; raw outputs committed in
+[benchmarks/results/error_reduction/](benchmarks/results/error_reduction/).
+
+| arm | failures | fail rate |
+|-----|----------|-----------|
+| A: plain | 63/96 | 65.6% |
+| B: with errlore | 20/96 | 20.8% |
+
+Exact McNemar over all 96 pairs: p = 1.8e-09 (49 pairs fixed, 6 broken).
+Split by error class:
+
+- **Knowledge-gap errors** (workspace conventions: date formats, ID
+  normalization, rounding rules, CSV column order): **46/48 -> 0/48, a 100%
+  reduction.** The model didn't know the convention; a lesson told it.
+- **Capability-gap errors** (letter counting, string reversal): 17/48 ->
+  20/48 -- errlore did **not** help and slightly hurt. Memory fixes what the
+  model doesn't know, not what it can't do.
+
+Reproduce: `python benchmarks/bench_error_reduction.py` (needs an Anthropic
+API key; the task families and validators ship in the repo).
+
 ## How it works
 
 errlore runs three reinforcement loops around your agent:

{errlore-0.1.2 → errlore-0.1.3}/README.md

@@ -57,12 +57,38 @@ mem.report_outcome(inj, success=True)
 print(mem.stats())
 # {'errors_total': 1, 'errors_resolved': 1, 'errors_unresolved': 0,
 #  'lessons_total': 1, 'lessons_applied': 1, 'pending_injections': 0,
-#  'trust': {'gpt-5.5': 0.55}}
+#  'trust': {'gpt-5.5': 0.5522...}}
 ```
 
 No API keys needed. errlore itself never calls any LLM -- it manages local
 JSONL files and does text matching. LLM calls are yours to make (or not).
 
+## Does it actually reduce errors?
+
+Yes -- for the class of errors memory can fix. Paired A/B benchmark
+(`benchmarks/bench_error_reduction.py`): the same model (claude-haiku-4-5)
+runs 96 tasks twice, with and without errlore injection. Deterministic
+validators, no LLM judges; raw outputs committed in
+[benchmarks/results/error_reduction/](benchmarks/results/error_reduction/).
+
+| arm | failures | fail rate |
+|-----|----------|-----------|
+| A: plain | 63/96 | 65.6% |
+| B: with errlore | 20/96 | 20.8% |
+
+Exact McNemar over all 96 pairs: p = 1.8e-09 (49 pairs fixed, 6 broken).
+Split by error class:
+
+- **Knowledge-gap errors** (workspace conventions: date formats, ID
+  normalization, rounding rules, CSV column order): **46/48 -> 0/48, a 100%
+  reduction.** The model didn't know the convention; a lesson told it.
+- **Capability-gap errors** (letter counting, string reversal): 17/48 ->
+  20/48 -- errlore did **not** help and slightly hurt. Memory fixes what the
+  model doesn't know, not what it can't do.
+
+Reproduce: `python benchmarks/bench_error_reduction.py` (needs an Anthropic
+API key; the task families and validators ship in the repo).
+
 ## How it works
 
 errlore runs three reinforcement loops around your agent:

{errlore-0.1.2 → errlore-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "errlore"
-version = "0.1.2"
+version = "0.1.3"
 description = "Memory for AI agents that learns from failures: lessons, known-issues injection, and per-model trust — embedded, file-based, no server."
 readme = "README.md"
 license = "MIT"

{errlore-0.1.2 → errlore-0.1.3}/site/index.html

@@ -168,10 +168,13 @@ mem.resolve(err, <span class="c-str">"added validation"</span>,
             lesson=<span class="c-str">"For date extraction, demand ISO-8601 and verify against source"</span>)
 
 <span class="c-com"># 3. next similar task — memory speaks up</span>
-inj = mem.inject_for(<span class="c-str">"extract dates from contract"</span>, model=<span class="c-str">"gpt-5.5"</span>)
-prompt = task + inj.text
+task = <span class="c-str">"extract dates from contract"</span>
+inj = mem.inject_for(task, model=<span class="c-str">"gpt-5.5"</span>, task_type=<span class="c-str">"extraction"</span>)
+prompt = task + <span class="c-str">"\n\n"</span> + inj.text
 <span class="c-out"># [LESSONS FROM PAST FAILURES]
-# - hallucinated dates -> For date extraction, demand ISO-8601...
+# - UnclassifiedError: hallucinated dates -> For date extraction,
+#   demand ISO-8601 and verify against source
+#
 # KNOWN ISSUES:
 # - Past error on similar task: hallucinated dates</span>
 
@@ -180,6 +183,23 @@ mem.report_outcome(inj, success=<span class="c-kw">True</span>)</pre>
   </div>
 </section>
 
+<section>
+  <h2>Measured, not promised</h2>
+  <p style="color:var(--dim);margin-bottom:18px">Paired A/B: the same model
+  (claude-haiku-4-5) runs 96 tasks with and without errlore injection. Deterministic
+  validators, no LLM judges; raw outputs committed to the repo. Exact McNemar over
+  all 96 pairs: p&nbsp;=&nbsp;1.8e-09.</p>
+  <table>
+    <tr><th>error class</th><th>plain</th><th>with errlore</th></tr>
+    <tr><td>knowledge-gap (workspace conventions)</td><td>46/48</td><td><b class="zero">0/48</b></td></tr>
+    <tr><td>capability-gap (model skill limits)</td><td>17/48</td><td>20/48</td></tr>
+  </table>
+  <p class="note">The honest half is the second row: errlore did <i>not</i> help on
+  things the model can't do (letter counting, string reversal) — and even slightly hurt.
+  Memory fixes what the model doesn't know, not what it can't do.
+  Reproduce it yourself: <span class="mono">python benchmarks/bench_error_reduction.py</span>.</p>
+</section>
+
 <section>
   <h2>Finds lessons by meaning, not keywords</h2>
   <p style="color:var(--dim);margin-bottom:18px">Optional local embeddings

{errlore-0.1.2 → errlore-0.1.3}/site/llms.txt

@@ -12,9 +12,10 @@ closes the loop: it tracks whether an injected lesson actually helped
 (reinforce/decay) and which model to trust per task domain.
 
 Proof: paired A/B (deterministic validators, raw outputs committed):
-repeated workspace-convention errors 46/48 -> 0/48 (100% reduction,
-exact McNemar p=1.8e-09). Capability-gap errors (e.g. letter counting)
-are honestly NOT improved. See the benchmark in the repository.
+63 -> 20 failures across all 96 paired tasks (exact McNemar p=1.8e-09);
+repeated workspace-convention errors 46/48 -> 0/48 (100% reduction).
+Capability-gap errors (e.g. letter counting) are honestly NOT improved
+(17/48 -> 20/48). See the benchmark in the repository.
 
 ## Links
 - [Website](https://errlore.com)

{errlore-0.1.2 → errlore-0.1.3}/src/errlore/__init__.py

@@ -1,6 +1,6 @@
 """errlore -- memory for AI agents that learns from failures."""
 
-__version__ = "0.1.2"
+__version__ = "0.1.3"
 
 from errlore.facade import AgentMemory, Injection
 from errlore.lessons.store import LessonStore

{errlore-0.1.2 → errlore-0.1.3}/src/errlore/io/jsonl_writer.py

@@ -285,11 +285,6 @@ class JSONLWriter:
             self.atomic_rewrite(path, new_entries)
             return new_entries
 
-    def _invalidate_cache(self, path: Path) -> None:
-        """Drop the read cache entry for *path*."""
-        with self._cache_lock:
-            self._read_cache.pop(str(path), None)
-
     def read_all(self, path: Path) -> list[dict[str, object]]:
         """Read all valid records from a JSONL file.
 
@@ -330,13 +325,17 @@ class JSONLWriter:
                 except json.JSONDecodeError:
                     logger.warning("Skipping corrupted line in %s", path.name)
 
-        # Re-stat after parse to store a consistent snapshot.
+        # Cache only if the file is unchanged since the pre-parse stat.
+        # An append landing mid-parse would otherwise store the pre-append
+        # records under the post-append mtime/size — a poisoned cache that
+        # makes atomic_update silently drop the concurrent record.
         try:
             st2 = path.stat()
         except OSError:
             return records
 
-        with self._cache_lock:
-            self._read_cache[key] = (st2.st_mtime_ns, st2.st_size, records)
+        if st2.st_mtime_ns == st.st_mtime_ns and st2.st_size == st.st_size:
+            with self._cache_lock:
+                self._read_cache[key] = (st.st_mtime_ns, st.st_size, records)
 
         return [dict(r) for r in records]

{errlore-0.1.2 → errlore-0.1.3}/tests/test_regressions.py

@@ -5,6 +5,9 @@ Bug 1: TrustEngine state silently reset on process restart (facade used the
 Bug 2: Lost updates under concurrency -- resolve_error/reinforce/decay_unused
        did read -> atomic_rewrite outside a shared lock, so records appended
        by other threads between the read and the rewrite were wiped.
+Bug 3: read_all cached a pre-append snapshot under the post-append stat when
+       an append landed mid-parse, poisoning the cache; atomic_update then
+       silently deleted the concurrent record.
 """
 
 from __future__ import annotations
@@ -112,3 +115,56 @@ class TestNoLostUpdatesUnderConcurrency:
         ]
         seed_row = next(r for r in rows if r["id"] == seed_id)
         assert seed_row["applied_count"] == n_new
+
+
+class TestReadCacheNotPoisonedByMidParseAppend:
+    """Bug 3: read_all cached pre-append records under the post-append stat.
+
+    An append landing between the parse loop and the re-stat made read_all
+    store stale records keyed by the fresh mtime/size; every later read
+    returned stale data and atomic_update silently deleted the concurrent
+    record.  Fixed by caching only when pre-parse stat == post-parse stat.
+    """
+
+    def test_append_during_parse_does_not_poison_cache(
+        self, data_dir: Path, monkeypatch: object
+    ) -> None:
+        import io as _io
+
+        import errlore.io.jsonl_writer as jw
+
+        writer = jw.JSONLWriter()
+        path = data_dir / "records.jsonl"
+        writer.append(path, {"id": 1})
+
+        real_open = open
+        raced = {"done": False}
+
+        def racing_open(file: object, mode: str = "r", *args: object, **kwargs: object):  # type: ignore[no-untyped-def]
+            # Intercept only read_all's read of our file, exactly once:
+            # snapshot the current content, append a record behind the
+            # parser's back, then hand the parser the stale snapshot.
+            if str(file) == str(path) and mode == "r" and not raced["done"]:
+                raced["done"] = True
+                content = path.read_text(encoding="utf-8")
+                with real_open(path, "a", encoding="utf-8") as af:
+                    af.write(json.dumps({"id": 2}) + "\n")
+                return _io.StringIO(content)
+            return real_open(file, mode, *args, **kwargs)  # type: ignore[arg-type]
+
+        monkeypatch.setattr(jw, "open", racing_open, raising=False)  # type: ignore[attr-defined]
+
+        # The racy read itself legitimately sees the pre-append snapshot...
+        first = writer.read_all(path)
+        assert [r["id"] for r in first] == [1]
+
+        # ...but it must NOT have been cached: the next read sees both.
+        second = writer.read_all(path)
+        assert [r["id"] for r in second] == [1, 2], (
+            "read cache was poisoned by a mid-parse append"
+        )
+
+        # And atomic_update must not delete the concurrently appended record.
+        result = writer.atomic_update(path, lambda entries: entries)
+        assert result is not None
+        assert [r["id"] for r in result] == [1, 2]