contextgc 0.4.2__tar.gz → 0.4.4__tar.gz

{contextgc-0.4.2 → contextgc-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: contextgc
-Version: 0.4.2
+Version: 0.4.4
 Summary: Deterministic context compiler for AI agent transcripts. Retires superseded facts, compacts tool output, pins declared invariants. No dependencies, no model calls.
 Author: Jay Gopal Tripathy
 License: MIT
@@ -52,38 +52,65 @@ no network, no API key, no dependencies. Same input, same output, microseconds.
 pip install contextgc
 ```
 
-> **Not on PyPI yet.** This line was here before the project had ever been
-> published, which made it a lie in the most load-bearing spot in the README.
-> Publishing is wired up (`.github/workflows/release.yml`, tag `v0.4.0`) and
-> verifies the artifacts before uploading, but until a tag is pushed the
-> install has to come from the repository:
+> **On PyPI** as `contextgc`, published via Trusted Publishing with no stored
+> credentials. `pip install contextgc` gets you the library and nothing else —
+> it has **zero runtime dependencies** by design, and it is a library, not a
+> CLI: there is no `contextgc` command. The entry points are
+> `compile_messages`, `load_schema`, `patch_openai` and `StateDAG`.
 >
-> ```bash
-> pip install git+https://github.com/j4yop/context-hackdevengers
-> ```
+> If you found an earlier version of this README saying "Not on PyPI yet", it
+> was true when written and then stopped being true, which is how a README
+> starts lying in the most load-bearing spot it has. Every number in this file
+> is reproducible from the repository; see below for how to get it.
 
 ```python
-from contextgc import compile_messages
+from contextgc import compile_messages, load_schema
 
 messages = [
-    {"role": "user", "content": "Deliver order ORD-1 to Tower B, Flat 402. Severe peanut allergy."},
-    {"role": "assistant", "content": "Confirmed, routing to Tower B."},
-    {"role": "user", "content": "The elevator is broken. Deliver to the Clubhouse desk."},
-    {"role": "assistant", "content": "Updated, routing to Clubhouse."},
-    {"role": "user", "content": "Actually reroute to Gate 2. Code 4921."},
-    {"role": "assistant", "content": "Rerouted to Gate 2."},
+    {"role": "user", "content": "Deliver order ORD-1 to 402 Oak Street, Apt 5, Springfield, IL 62704. Pay with my credit card."},
+    {"role": "assistant", "content": "Confirmed: 402 Oak Street, Apt 5, on the credit card."},
+    {"role": "user", "content": "Actually reroute to 900 Pine Avenue, Suite 12, Chicago, IL 60601."},
+    {"role": "assistant", "content": "Rerouted to 900 Pine Avenue, Suite 12."},
     {"role": "user", "content": "Thanks."},
 ]
 
-compiled, telemetry = compile_messages(messages)
+compiled, telemetry = compile_messages(messages, schema=load_schema("logistics"))
 
 telemetry["active_state_slots"]
-# {'destination_address': 'Gate 2', 'dietary_allergy': 'peanut'}
+# {'delivery_address': '900 Pine Avenue, Suite 12', 'payment_method': 'credit card'}
 
 telemetry["retired_turn_indices"]
-# [1, 3]  -- the two assistant turns that asserted a now-stale address
+# [1, 2]  -- the two turns that asserted a now-stale address
 ```
 
+```
+  system: [ACTIVE_AGENT_STATE]
+  - delivery_address = "900 Pine Avenue, Suite 12" (turn 3 [inferred])
+  - payment_method = "credit card" (turn 0 [inferred])
+     user: Deliver order ORD-1 to 402 Oak Street, Apt 5, Springfield, IL 62704. Pay with my credit card.
+  assistant: Rerouted to 900 Pine Avenue, Suite 12.
+     user: Thanks.
+```
+
+Five turns in, three out. The address that was current for one turn is gone, and
+the one that superseded it is in a header the model can read without re-reading
+the transcript.
+
+> **Two things this example has to get right, and the version that was here
+> first got both wrong.** It did not pass a `schema`, and it cannot: with no
+> schema the read path has no vocabulary and extracts **nothing** — that is
+> deliberate, not a bug, since a built-in default domain would match prose in
+> every other domain. And the content has to be something the shipped schema
+> actually matches. The earlier version of this example used *"Tower B, Flat
+> 402"*, *"Gate 2"* and *"the Clubhouse desk"*, and claimed an output of
+> `{'destination_address': 'Gate 2', 'dietary_allergy': 'peanut'}`. Those two
+> slot names exist nowhere in this project, and none of that prose matches the
+> `logistics` patterns, which want a US street address (`Oak Street`, `Pine
+> Avenue`) and a phrase like *"my credit card"*. Run it and you got `{}` and the
+> transcript straight back. **The first example in a README is the whole
+> product for someone evaluating it in thirty seconds, and it was returning
+> nothing while looking like it worked.**
+
 Try it on your own transcript at **[context-hackdevengers.vercel.app](https://context-hackdevengers.vercel.app)** —
 paste a conversation on the left, get the compiled context back on the right.
 
@@ -223,11 +250,23 @@ last-write-wins state register only earns its keep where a conversation keeps
 changing its mind about one thing.
 
 ```bash
-pip install 'contextgc[bench]'
+git clone https://github.com/j4yop/context-hackdevengers
+cd context-hackdevengers
+pip install -e '.[bench]'
 python -m benchmarks fetch          # 85 MB parquet shard
 python -m benchmarks run --limit 40 --schema contextgc/schemas/coding.json
 ```
 
+> **A clone is required, and `pip install 'contextgc[bench]'` is not enough.**
+> The `bench` extra installs `pandas` and `pyarrow` and nothing else. The
+> harness lives in `benchmarks/`, which is not in the wheel — the wheel ships
+> `contextgc` only, deliberately, so the library stays dependency-free. So the
+> command above reads
+> `ModuleNotFoundError: No module named 'benchmarks'` from a pip install, and
+> that is the correct behaviour, not a bug to report. Every figure in this
+> README is reproducible from a clone; none of them is reproducible from the
+> wheel, because the wheel does not contain the thing that measured them.
+
 ```
 CORPUS  swe-agent-trajectories
   transcripts      40
@@ -407,8 +446,149 @@ because fixing the repeat bug in defect 6 changed what the compiler extracts.
 The 100% is not a claim that the tracker is perfect. It is 100% of what was
 labelled, and labelling is the bottleneck. Concretely, the limit is that **all 39
 labels are one slot**: precision is currently a statement about the
-`current_file` regex and nothing else. (`coding.failing_test` cannot be measured
-at all — it has 0 speech observations, so the read path never produces it.)
+`current_file` regex and nothing else. A worksheet for the second slot is built
+below.
+
+#### The second coding slot, unreachable and then not
+
+`coding.failing_test` matched **740 times** in the coding corpus and **not once in
+speech**. Every hit sat inside a pytest result — `FAILED tests/test_x.py::test_y` —
+which is machine output the read path is *forbidden* to infer from. So the slot
+could never fire, could never be labelled, and had no evidence behind its
+contract. It was carried in a shipping schema by a number that was large and
+true and meant nothing.
+
+The number hid this because a corpus-wide count cannot tell you where the
+matches were. `test_the_one_slot_that_never_fires_from_speech_says_so` existed
+precisely to catch it, and the fix was to the **pattern**, not to the
+measurement. Requiring the noun phrase that actually introduces a file — *the
+test file `test_dispatcher.py`*, *the tests in `test_run.py`*, *the failing test
+`test_x.py`*:
+
+| `coding.failing_test` | before | after |
+|---|---|---|
+| speech observations | **0** | **78** |
+| distinct values | 0 | **49** |
+| observations in tool output | 740 | 14 |
+
+Across 174,815 agent turns there are 81 noun-phrase hits, 71 of them not beside a
+shell command. Those are assertions, not commands — which is the distinction the
+old pattern had collapsed. Nothing the read path actually used is lost: it was
+already discarding the other 740.
+
+The read path now extracts `test_dispatcher.py`, `test_cognitive_complexity.py`
+and `tests/test_cognitive_complexity.py` from the first 40 transcripts, which
+before produced **zero**.
+
+Two honest costs. The derived contract is now `[...]test[...].py`, which accepts
+any path containing `test` — so it also accepts `latest_file.py` and
+`my_test_helper.py`, and those are recorded as residual false accepts rather than
+fitted away with a lookup table. And the slot is named `failing_test` while the
+prose it now matches says only *a test file*; that was true of the old pattern
+too, and renaming a shipped key is a breaking change not made to improve a
+sentence. **49 distinct values is thin** for precision, and the worksheet below
+exists to thicken it. It is measurable now, which it was not.
+
+#### A cheaper check that turns out not to be good enough
+
+With the slot reachable, the obvious next move is a check that needs no human.
+Every `failing_test` value the read path produces from speech can be tested for
+existence: does that path appear anywhere in what the agent's own tools listed?
+
+| slot | read from speech | names a file the tools listed | rate |
+|---|---|---|---|
+| `failing_test` | 78 | 78 | **100.0%** |
+| `current_file` | 38,375 | 38,253 | 99.7% |
+
+`failing_test` scores 100%. **So does `current_file`, and that is the whole
+finding: the check does not discriminate.** Both slots are near-perfect at it
+because both name files in a repository the agent is actively working in, so
+the file existing is nearly guaranteed and near-worthless as evidence that the
+read path chose *correctly*.
+
+It does rule out one real failure — the read path is not inventing paths that
+were never there — and it rules out nothing else. A `failing_test` naming the
+wrong-but-real test file passes it. So it is recorded as a floor, not a
+precision figure, and the human labels remain the only route to the number.
+
+#### The worksheet was a trap, and it is disarmed
+
+`failing_test` is now reachable, so the next step is a worksheet. The default
+window returns **2 rows**:
+
+```
+--limit  60 ->  2 rows       --limit 2000 -> 21 rows
+--limit 500 ->  7 rows       --limit 6000 -> 44 rows
+```
+
+Two rows read as *this slot cannot be labelled* when the shard holds 49 distinct
+values. A person who believed that would drop the slot and report coding
+precision as single-slot permanently — which is how it was single-slot in the
+first place, and the reason the 100% said so little. So the worksheet now names
+any slot short of its budget, and **refuses to guess the cause**: the window may
+be too small, or the corpus may have no more, and only raising `--limit` tells
+them apart. At `--limit 6000` the shortfall is real exhaustion — 44 of the 49
+available values, the other 5 already labelled — which is the answer the tool
+would not assert on its own.
+
+`ws.json` is built at `--per-entity 60 --limit 6000`: 44 `failing_test` rows and
+60 `current_file`, all `unlabelled`, no suggestions attached. It is waiting on
+a person. **Coding precision stays at 100% (n=36, all `current_file`) until
+someone fills it in**, and that sentence is the honest state of the number, not
+a rounding of it.
+
+#### The write path, checked at last
+
+Everything above scores what the **read path** inferred. The **write path** is the
+model declaring facts, and until now nothing had ever checked whether a declared
+fact was *true* — which is the gap that mattered, because a declared value wins
+over the read path by design. An unchecked declaration silently overrides
+everything the read path got right.
+
+The same records settle it: a capture is keyed by transcript and turn, the corpus
+has the booking record for that transcript, and the turn indices line up.
+
+```
+python -m benchmarks.ground_truth travel --declarations captures/airline-14b.json
+```
+
+| 14B airline capture | count |
+|---|---|
+| declared values | 109 |
+| corroborated by the record | **70** |
+| **contradicted by the record** | **12** |
+| unverifiable (no record saw the outcome) | 2 |
+| unchecked (slot has no record field) | 25 |
+
+**So the write path is wrong 12 times in 82 decidable declarations — 85.4%
+corroborated.** And the value gate rejects **11 of those 12**. Those it catches
+are the ones the earlier capture already showed: `passenger_id = "None"`,
+`flight_number = "H0MVIE"` (a reservation id declared as a flight),
+`flight_date = "2024-05-07T13:04:42"` (a timestamp, not a date), `passenger_id =
+"Yusuf Jackson"` (a name where an id belongs).
+
+**The twelfth is the one worth reading twice.** `cabin_class = "basic economy"`
+where the record says `economy`. It satisfies its contract, passes the gate, and
+is still wrong — the exact failure this project has said all along a contract
+cannot catch, now with a row in a table rather than an assurance. A contract
+checks shape, never truth.
+
+Two of the numbers above are only trustworthy because of bugs found while
+producing them:
+
+- The record was looked up by **corpus** name (`airline`) while the marker is
+  keyed by **schema** (`travel`), so every airline record resolved to nothing and
+  109 declarations scored 0 corroborated. That reads like a finding about the
+  write path and was a lookup that found nothing.
+- `2024-05-20` and `May 20, 2024` are the same flight date. Scoring them
+  differently called a *true* declaration a contradiction three times in one
+  conversation.
+
+**What this does not establish.** 25 declarations sit in slots with no record
+field at all — `total_baggages`, `cancellation_possible` — and nothing here can
+settle them either way. And n=82 across 6 transcripts on one model is a small
+base. The write path is *measured* now, which it was not, and it is still not
+*proven*: one well-formed false value in twelve is the number to remember.
 
 #### The same check on the third shipped schema, and what it cannot settle
 
@@ -423,34 +603,46 @@ whole line.
 
 | | count | share |
 |---|---|---|
-| corroborated by the record | 509 | 86.0% |
-| contradicted by the record | 70 | 11.8% |
-| unverifiable | 13 | 2.2% |
+| corroborated by the record | 560 | **97.4%** |
+| **contradicted by the record** | **15** | **2.6%** |
+| unverifiable | 17 | 3.0% |
 
 Three things that number needs beside it.
 
+**A record has to be scored at the right point in the conversation.** This is
+where most of the error was, and it was in the *measurement*, not the read path.
+`payment_history` says what was already charged and `address` says where the order
+is now, so a `delivery_address` slot — whose whole purpose is a *requested*
+change — looked like it was being judged against a record that was behind by
+definition. But the record is not behind. The agent applies the change, and the
+**next** tool result shows the new state:
+
+    apigen-1751 turn 13  "760 Elm Avenue"   record before: 592 Elm     after: 760 Elm
+    apigen-1817 turn 11  "123 Oak Street"   record before: 463 Main     after: 123 Oak
+    apigen-1865 turn 12  "828 River Road"  record before: 388 Spruce   after: 828 River
+
+So a value is checked against the state as of **just before** the turn, and against
+every state recorded **at or after** it. That took retail contradictions from **70
+to 19** — and all three cases above were in the 70, and all three had been read by
+hand before the data was blamed on the read path.
+
+**And a record that never saw the outcome is silence, not disagreement.** If nothing
+is recorded at or after the turn, the order had no chance to reflect the change, so
+a value matching nothing is *unverifiable*. A customer who says "charge the
+difference to my credit card" and then ends the conversation leaves no trace, and
+calling the read path wrong for agreeing with them measures nothing. That split
+took the 19 down to 15 — and it applies to the airline corpus too, where it moved
+contradictions from 5 to **2**.
+
 **A partial address is not a contradiction.** "464 Oak Street, Suite 664" is
-correct, and the record also holds "San Diego, CA 92135". 11 of the 35
-corroborated addresses stopped at the street line; they are counted as
-corroborated and the incompleteness is reported beside the rate. Demanding the
-full line first turned one of two address contradictions at 200 transcripts into
-exactly that, and reading the turn showed the address was right.
-
-**The retail record cannot settle a slot about a pending change.** `payment_history`
-says what was already charged; `address` says where the order is now. A
-`delivery_address` slot exists to capture a *requested* change, so the moment a
-conversation is about changing something, the record is behind by definition.
-Reading the contradicted turns, they are that shape — "I would like to change the
-shipping address to 123 Oak Street", "you can use my PayPal account for any price
-differences along the way" — and the read path is tracking what the customer just
-asked for. They stay counted as contradicted, because excluding them would flatter
-the number, but they are the record answering a different question rather than the
-tracker misreading one.
-
-**No split of those 70 is published.** An attempt to classify them by regex was
-wrong on turns that had already been read by hand, and a precise-looking number
-from an unreliable classifier is worse than none. The tool prints a hand-read
-sample and says so.
+correct, and the record also holds "San Diego, CA 92135". 25 of the 51
+corroborated addresses stopped at the street line; they count as corroborated and
+the incompleteness is reported beside the rate.
+
+The two travel contradictions that remain are the irreducible ones: a cabin class
+taken off a baggage-allowance table, and one turn where the agent says Economy
+while the record says `basic_economy`. The second is not a pattern bug at all — the
+read path faithfully recorded what the agent said.
 
 #### A ground truth that is not a person: the booking database
 
@@ -472,9 +664,9 @@ of three buckets, and reports them apart:
 
 | | count | share |
 |---|---|---|
-| corroborated by the record | 557 | 90.6% |
-| **contradicted by the record** | **5** | **0.8%** |
-| unverifiable (no record for that slot) | 53 | 8.6% |
+| corroborated by the record | 557 | **99.6%** |
+| **contradicted by the record** | **2** | **0.4%** |
+| unverifiable (no record for that slot) | 56 | 9.1% |
 
 The rate is over corroborated + contradicted. The unverifiable column is **not** a
 pass — it is extractions this check says nothing about, and folding it in would
@@ -646,6 +838,51 @@ to infer from tool output is a large part of why — it holds no value on the
 consecutive agent actions, which is a definition chosen here and not given by
 the data.
 
+#### The one staleness test that needs no edit signal
+
+The reason above is that a rate needs a ground truth for *when the file changed*.
+But there is a weaker question with a free answer, and it is the one that
+actually matters: **does the read path's value match the agent's most recent file
+statement?** If it does, the tracker is not stale, whatever the agent was doing
+in between. The vendored coding transcript answers it outright — 18 distinct
+file statements, in order:
+
+```
+memset.py -> reproduce.py -> memset.py -> lexicon/lexicon/cli.py
+```
+
+The read path reproduces that sequence exactly, including the round trip: the
+agent creates `reproduce.py`, works in it, then goes back to `cli.py`, and the
+read path follows it out of `reproduce.py` and back. Its final value equals the
+agent's final statement, so **the read path is not stale, and it is a faithful
+last-statement tracker.** 81% of its matches come from the agent's speech and
+19% from tool output; it takes the final one either way.
+
+Two things this does *not* license. It does not produce a staleness **rate**,
+because a rate needs the edit signal the corpus lacks. And it does not validate
+the confounded 44.9%: one trajectory agreeing with itself is not a measurement.
+
+The turn-by-turn version **is** available in the 14B capture, which records
+`declared` on every turn — and it is still not measurable, for a reason worth
+writing down. The capture has 6 transcripts; the vendored agent transcript is
+**one** file. Aligning each transcript's model declarations to a matching agent
+prefix requires slicing that one file six ways, which invents six trajectories
+out of one. I did it, got 8 apparent disagreements, and discarded all 8: they
+were artifacts of the slicing, and reporting them would have been the exact
+failure this section is about. **A measurement that needs data the corpus does
+not have is not a weak measurement, it is a fabricated one.**
+
+One oddity to leave alone on purpose. Half the values are doubled
+(`lexicon/lexicon/cli.py`, where the first `lexicon` is the conda prefix and the
+second is the repo). That is the **agent's own text**, faithfully captured, and
+normalising it would be fitting the tool to a transcript's confusion. Recorded
+here so it is not later "fixed".
+
+**Where this leaves the dimension.** Not unmeasured any more — measured, once,
+and negative: the tracker does not go stale on the trajectory available. Still
+no rate, and still n=1 trajectory. That is a better answer than the one it
+replaces, and it is still not a headline.
+
 **The two "confirmed errors" this project carried were mislabelled.** Both were
 recorded as `current_file` extractions judged incorrect because the turn's earlier
 prose discussed a different file. Re-reading the turns against the doctrine the
@@ -1082,10 +1319,18 @@ measurement for a guess:
 ```json
 "values": {
   "current_file": "(?:[\\w][\\w./-]*\\.(?:py|js|ts|tsx|jsx|go|rs|java|rb|cpp|hpp))",
-  "failing_test": "(?:(?<![A-Za-z0-9])test_[\\w./\\[\\]-]+)"
+  "failing_test": "(?:[\\w./-]*test[\\w./-]*\\.py)"
 }
 ```
 
+The second line is worth pausing on, because it is a contract that accepts
+`latest_file.py` — `la`+`test` — and `my_test_helper.py`. The `test_` anchoring
+this slot used to have is gone, because the pattern that reaches it from speech
+no longer requires a `test_` prefix. Both junk cases are recorded as residual
+false accepts. The trade is deliberate: a contract that rejects 2 of 49 real
+values and is unreachable buys less than one that accepts everything the read
+path can reach and admits its 2 false accepts in the open.
+
 Deriving them mattered more than the hand-written versions looked. The old
 `logistics.delivery_address` was `[\w\s.,'/-]+` and `payment_method` was
 `[\w ]+`, so on a 48-value set of wrong-shaped declarations — `None`, error