agent-formal-verify 0.4.2 → 0.4.4

package/CHANGES.md

@@ -1,5 +1,13 @@
 # Changes
 
+## 0.4.4 - bring existing models back in line with the code
+
+A TLA+ spec or Lean model keeps passing after the code it describes changes, because it never reads the code. When a repository already has models, the formal-verify skill now starts with them. It runs every one, finds where the functions it covers live now, and reads the commits that touched those files since the commit its header records as `Source read at`, so it follows code that moved to another file. It asks whether the model still describes the code and whether any of those commits says the guarantee changed; without that evidence the property stays, and a failure after the update is a bug candidate. Each brief has a section for an existing model: the agent edits the model's files, adds a mutation for each step the update adds, and records the commit it read the source at. When that commit is missing or no longer reachable from `HEAD`, as after a squash merge, the skill uses the last commit that touched the model. It proposes retiring a model whose functions are found nowhere and deletes it, with its CI steps and testing documentation row, when the user agrees. The reply keeps each model's verdict and adds whether it was unchanged or updated, or is proposed for retirement. A request to recheck only the existing models skips the search for new targets.
+
+## 0.4.3 - clear the directory's credential and download-and-run holds
+
+The Claude directory held every version since 0.3.3 for a credential read and a download-and-run command, and 0.3.2 stayed the version it served. Validating the tree one deletion at a time against a throwaway repository named each trigger. The credential rule matched a parameter named for a credential in `opencode.mjs` and in the metadata test, an environment lookup in the Kani helper's `search_path` whose variable name was an argument, and the test driver writing its whole environment to a file. `search_path` now takes the inherited value, which its two callers read by name, so the assembled `PATH` and `PYTHONPATH` are unchanged; the parameters are renamed; the fake kani-driver lives in `tests/fixtures/kani-driver.py` and checks `KANI_HOME` and the loader path against `driver.json` instead of writing its environment. The download-and-run rule matched the name of Lean's evaluation command in the Lean brief, which now says to evaluate `badPairs` at top level as the template does; the command stays in the template and the checker. The CI workflow's curl of the TLC release stays and shows as a warning, which does not hold a version. The 0.3.5 notes blamed the workflow token; it was never the trigger.
+
 ## 0.4.2 - OpenCode skips stray folders and rejects incomplete skills
 
 The OpenCode plugin read every folder under `skills/` as a skill, so one folder without a `SKILL.md`, such as a scratch folder in a local checkout, failed the plugin and registered neither skill. It now skips such folders, as Pi does. A `SKILL.md` without a `name` or `description` used to register with that field undefined; the plugin now fails with the file's path and the missing field. A single-quoted `description` lost nothing but kept its quotes; it now reads as YAML does. The two bundled skills register exactly as before.
@@ -14,7 +22,7 @@ The repository is now an OpenCode plugin for OpenCode 2.2 and later. `opencode p
 
 ## 0.3.5 - fetch TLC in CI without a token, and name no MCP package
 
-The CI workflow fetched the pinned TLC JAR with the GitHub CLI under the workflow's token, the only credential variable in the repository. The Claude directory scans every tracked file and read that as a credential taken from the environment and sent to a server. The workflow now downloads the release asset with curl and checks it against the pinned SHA256 through the shared helper before anything runs it. The Lean brief no longer links to a Lean language-server MCP package; the directory counts package launchers as download-and-run commands, and that link was the plugin's only reference to installable software. The 0.3.4 strip of the image manifests cleared the image hold and one of two credential findings; the token read was the other.
+The CI workflow downloads the pinned TLC JAR with curl and checks it against the pinned SHA256 through the shared helper before anything runs it, instead of fetching it with the GitHub CLI under the workflow's token. The Lean brief no longer links to a Lean language-server MCP package. Neither change cleared a directory hold; 0.4.3 names the triggers.
 
 ## 0.3.4 - keep image references out of release-note code
 

package/README.md

@@ -79,6 +79,7 @@ A passing check establishes its stated properties under its assumptions and boun
 - [Plugin updates](docs/reference.md#plugin-updates)
 - [Verification workflow](skills/formal-verify/SKILL.md)
 - [Release history](CHANGES.md)
+- [Agent instructions](AGENTS.md)
 
 ## Data handling
 

package/opencode.mjs

@@ -8,9 +8,9 @@ const skills = new URL("./skills/", import.meta.url);
 function parse(text, path) {
   const match = text.replaceAll("\r\n", "\n").match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/);
   if (!match) throw new Error(`${path} has no frontmatter`);
-  const field = (key) => {
-    const value = match[1].match(new RegExp(`^${key}: (.+)$`, "m"))?.[1];
-    if (value === undefined) throw new Error(`${path} has no ${key}`);
+  const field = (fieldName) => {
+    const value = match[1].match(new RegExp(`^${fieldName}: (.+)$`, "m"))?.[1];
+    if (value === undefined) throw new Error(`${path} has no ${fieldName}`);
     if (value.startsWith('"')) return JSON.parse(value);
     if (value.startsWith("'")) return value.slice(1, -1).replaceAll("''", "'");
     return value;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-formal-verify",
-  "version": "0.4.2",
+  "version": "0.4.4",
   "description": "Model-check protocols with TLA+, prove invariants with Lean 4, and check bounded Rust functions with Kani.",
   "author": {
     "name": "Michael Denyer",

package/skills/formal-verify/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: formal-verify
-description: "Model-check thread protocols with TLA+ and TLC, prove sequential invariants with Lean 4, and check bounded Rust functions with Kani. Map counter-examples to source locations, reproductions and fixes, and keep the models in CI. Use for /agent-formal-verify:formal-verify, formal verification, model checking, Lean proofs, Kani, bounded Rust verification, or concurrency bugs that runtime tests cannot reproduce."
+description: "Model-check thread protocols with TLA+ and TLC, prove sequential invariants with Lean 4, and check bounded Rust functions with Kani. Map counter-examples to source locations, reproductions and fixes, and keep the models in CI. Bring existing models back in line after the code changes. Use for /agent-formal-verify:formal-verify, formal verification, model checking, Lean proofs, Kani, bounded Rust verification, rechecking models after code changes, or concurrency bugs that runtime tests cannot reproduce."
 ---
 
 # Formal verify
@@ -23,6 +23,17 @@ Use the tool the user names. When the table gives another tool for part of the t
 
 ## 1. Pick the targets
 
+When the repository has models from an earlier run, start with them. Find every model, wherever it lives: `.tla` specs with their `.matrix` and `.mutations` files, Lean models with their `.mutations` and `.statements` files, and Kani harness files. Each is a row in the table this step writes.
+
+Take each model's sync commit from the `Source read at` line in its header when `git merge-base --is-ancestor <commit> HEAD` succeeds, and otherwise from `git log -1 --format=%H -- <model files>`. Find where the functions and shared variables it covers live now with `git grep -l -w -e <name> -- . ':(exclude)<model file>'`, one exclude per model file. When none is found, mark the row for retirement as step 5 says. Otherwise read `git log -p <sync>..HEAD -- <source files> <files found>`, which follows code that moved to another file. A model with no sync commit has no log to read, so compare it with the code in full.
+
+Record the sync commit, the commits read and the answers to two questions in the row:
+
+- Does the model still describe the code as it is now? If not, the model changes.
+- Does a doc comment, test, changelog entry or commit message in those commits say the guarantee changed? If so, the property changes with it, citing that evidence. If not, the property stays, and a failure after the update is a bug candidate.
+
+Then search for new targets in code no model covers, unless the user asked only to recheck the existing models.
+
 Search with `rg` for `pthread_cond`, `pthread_mutex`, `atomic_`, `std::condition_variable`, `sync.Cond`, `chan`, `select`, `asyncio.Condition`, `Semaphore`, `atexit` and `setjmp`. Also search for protocol state names such as `ready`, `stop`, `done`, `finished`, `pending`, `inflight`, `head`, `tail` and `exiting`. Classify each hit:
 
 - Model hand-written protocols, including hand-overs, bounded pipelines, queues with ordered output, shutdown, cleanup, retry and lease loops.
@@ -54,6 +65,8 @@ The helpers:
 - The Rust checker, [`scripts/kani-check.sh`](scripts/kani-check.sh), runs the pinned Kani on the crate's proof harnesses. It reports each harness's unwind bound and check count, rejects failed checks and unsatisfied cover conditions, and prints concrete playback tests when available. Copy [`examples/kani-template/`](examples/kani-template/src/lib.rs) for a bounded-queue example.
 - The mutation runner, [`scripts/mutate.py`](scripts/mutate.py), takes a model's `<Name>.mutations` file and applies each listed bug to a temporary copy of the model. It prints DETECTED or MISSED per mutation with what failed, UNCOVERED for a TLA+ property or Kani harness that no mutation targets, and a SUMMARY line. All three templates include a mutations file.
 
+Run every existing model's checker and mutations once, before any agent changes a model, and record the results. For a TLA+ or Lean model a pass shows only that the old description still holds; a Kani harness runs the code as it is now. A model that still describes the code needs no agent. When it passes, refresh its header as step 5 says; when it fails, take the failure through step 4. A Kani compile error means the harness no longer fits the code, so give it to an agent. A check that passes here and fails after the update was passing on a stale description; say so in the reply.
+
 ## 3. Model one target per agent
 
 Spawn one agent per target, in parallel when supported. Use the host runtime's agent tools and follow the user's model configuration when selecting agents. An agent does not see this skill. Its brief contains:
@@ -61,7 +74,8 @@ Spawn one agent per target, in parallel when supported. Use the host runtime's a
 - the target's row from the table, with its source files and line ranges;
 - the model's name, `<Name>`, and for a Lean target the absolute path of the Lake project, or for Kani the absolute path of the crate;
 - the absolute path of this skill's directory as `<skill-dir>`, and any `JAVA` setting;
-- the instruction to read `<skill-dir>/references/tla-brief.md` for a protocol, `<skill-dir>/references/lean-brief.md` for arithmetic, or `<skill-dir>/references/rust-brief.md` for a Rust function, in full and to follow it.
+- the instruction to read `<skill-dir>/references/tla-brief.md` for a protocol, `<skill-dir>/references/lean-brief.md` for arithmetic, or `<skill-dir>/references/rust-brief.md` for a Rust function, in full and to follow it;
+- for an existing model, its files.
 
 Name the brief file and leave its text out of the prompt. The [TLA+ brief](references/tla-brief.md), the [Lean brief](references/lean-brief.md) and the [Rust brief](references/rust-brief.md) hold the modelling requirements, the files to write, the commands to run and what to report. If delegation is unavailable, read the brief for each target and model the targets yourself, one at a time.
 
@@ -86,11 +100,13 @@ Add CI only when the user asked for it or the repository already runs models in
 
 Each fact about a model has one home:
 
-- The model's header comment names the source files and line ranges it follows and its assumptions. Refresh them when the code moves.
+- The model's header comment names the source files and line ranges it follows, the functions and shared variables it covers, the commit they were read at as `Source read at <commit>`, and its assumptions. Refresh them when the code moves.
 - The mutations file lists the bugs the model detects.
-- The project's testing documentation, when it has any, has one row per model: what it checks, in the words the Reply section asks for, and its boundary configuration.
+- The project's testing documentation, when it has any, has one row per model: what it checks, in the words the Reply section asks for, and its boundary configuration. Update the row when the model changes.
 - A pull request description tells its bug's story once and links to the model.
 
+When a model's covered code is gone, propose retiring it in the reply. With the user's agreement, delete its files, including a Lean model's `.statements` record, its CI steps and its row in the testing documentation.
+
 Do not add a report file per bug, and do not paste run tables or traces into several documents. Write `<skill-dir>` for the installed skill in any committed command; a path from this machine does not work for the next reader.
 
 ## What this does not cover
@@ -104,7 +120,7 @@ A passing TLA+ or Lean model establishes its stated properties under its assumpt
 Write the reply, and any pull request or document, for a reader who knows the code and has used none of these tools.
 
 1. Start with what was found, in the code's terms. For each bug, say what goes wrong for a caller or user, the code events in order with `file:line`, whether the shipped settings can reach it, how to reproduce it, and the fix or proposed fix, with a PR link when one was opened.
-2. Give a table with one row per target: what was checked, as a sentence about the code, and the verdict. Label each counter-example `reproduced`, `reachable at shipped settings`, or `model-only`. State each property as what it guarantees, such as "the bar never shows 100% before the work finishes". Add the model's name for it in backticks only so the reader can find it.
+2. Give a table with one row per target: what was checked, as a sentence about the code, and the verdict. Label each counter-example `reproduced`, `reachable at shipped settings`, or `model-only`. State each property as what it guarantees, such as "the bar never shows 100% before the work finishes". Add the model's name for it in backticks only so the reader can find it. For a model from an earlier run, add whether it was unchanged or updated, or is proposed for retirement, with its sync commit and the commits read for it.
 3. Say what each kind of evidence means where it first appears:
 
    | Evidence | Meaning to give |

package/skills/formal-verify/examples/tla-template/BoundedQueue.tla

@@ -9,8 +9,10 @@
    every size. BoundedQueue.mutations lists the bugs its properties must detect.
 
    To adapt it, replace the header with the source files and line ranges the
-   model follows, give each thread a program counter, and keep the top-level
-   formula named Spec, which tlc-matrix.sh checks. *)
+   model follows, the functions and shared variables it covers, and the commit
+   they were read at as `Source read at <commit>`. Give each thread a program
+   counter, and keep the top-level formula named Spec, which tlc-matrix.sh
+   checks. *)
 EXTENDS Naturals
 
 CONSTANTS Window, Items

package/skills/formal-verify/references/lean-brief.md

@@ -4,19 +4,23 @@ Follow this brief to prove one sequential or arithmetic property with Lean 4. Yo
 
 ## Files
 
-Write `Model/<Name>.lean`, `Model/<Name>.mutations` and `Model/<Name>.statements` in the Lake project. Start from the `BoundedQueue` files in `<skill-dir>/examples/lean-template/Model/`. Write no other file, so parallel agents share none. Put every declaration in `namespace <Name>`: the checker audits the project's modules together, and equal names clash. The header comment names the source files and line ranges the model follows and each assumption it makes.
+Write `Model/<Name>.lean`, `Model/<Name>.mutations` and `Model/<Name>.statements` in the Lake project. Start a new model from the `BoundedQueue` files in `<skill-dir>/examples/lean-template/Model/`, and edit an existing model's files in place. Write no other file, so parallel agents share none. Put every declaration in `namespace <Name>`: the checker audits the project's modules together, and equal names clash. The header comment names the source files and line ranges the model follows, the functions it covers, the commit they were read at as `Source read at <commit>`, and each assumption it makes.
 
 Every `.lean` file in the project must belong to a library in the lakefile. With the template's lakefile, that is every file under `Model/`. Add Mathlib only when the arithmetic needs it, since its dependencies can increase setup and build time.
 
+## Existing model
+
+The row in your task gives the commits since the model was last synced and says whether any of them changed the guarantee on purpose. Change the transcription where it no longer describes the code. Change a theorem statement only where the row cites that evidence. Re-freeze after either change, as the Check section allows, and rerun the differential vectors. Add a mutation for each branch or step the update adds. Update the header to the code you read.
+
 ## Model
 
 1. Transcribe the state type and functions with the code's integer widths, rounding, division and order of operations. Use `UInt32` or `Int` with explicit bounds where appropriate, and `Nat` only when values cannot go negative. Keep the source function names and cite `file:line` in doc comments. Preserve any bug during transcription so the model checks the code as written.
 2. State the property as a `Prop` with a `Decidable` instance. It may describe an invariant, an encoder round trip, an index bound or states the machine must never reach.
 3. Write every theorem statement with `sorry` for its proof, then freeze the model with `bash <skill-dir>/scripts/lean-check.sh <absolute-project-path> --freeze Model/<Name>.lean`. This records types and definition bodies even while proofs are unfinished. The FROZEN line confirms the record, not a proof. Added axioms still fail.
-4. Write and evaluate a bounded exhaustive check before proving the theorem. The template's `badPairs` finds states where one step breaks the invariant; `#eval badPairs 2 6` prints concrete counter-examples and `#guard` fails the build while any exist. Set the bound above the code's slot count, batch size, window and other constants so it includes boundary cases. Add a `#guard` that a bounded search from the initial state reaches the states the property depends on. The template's `reachable` search requires a full window.
+4. Write and evaluate a bounded exhaustive check before proving the theorem. The template's `badPairs` finds states where one step breaks the invariant; evaluating `badPairs 2 6` at top level, as the template does, prints concrete counter-examples and `#guard` fails the build while any exist. Set the bound above the code's slot count, batch size, window and other constants so it includes boundary cases. Add a `#guard` that a bounded search from the initial state reaches the states the property depends on. The template's `reachable` search requires a full window.
 5. Prove one theorem per step or function for every size. Split on the guards, use `simp only [...]` to expose arithmetic, then use `omega` or `decide` for a finite type. If a proof needs a fact about the real code that the model lacks, state it as a theorem hypothesis and name it in a comment. Do not declare it as an `axiom`. An unfinished proof may use `sorry`, but the Lean checker reports it as a failure. The checker also fails every declaration that rests on an axiom beyond `propext`, `Classical.choice` and `Quot.sound`, which also rules out `native_decide`.
 
-After a proof closes, print differential test vectors with `#eval`. Include boundary inputs and a spread of in-range inputs, with each input and the model's output on one line. Write a test in the target repository's own framework that calls the production function on those inputs and compares its output. A mismatch is a bug in the production code or the transcription; establish which before reporting. State how many vectors agree and the integer widths in both implementations. If the production comparison was not run, say so beside the proof verdict.
+After a proof closes, print differential test vectors by evaluating them at top level, as the template does. Include boundary inputs and a spread of in-range inputs, with each input and the model's output on one line. Write a test in the target repository's own framework that calls the production function on those inputs and compares its output. A mismatch is a bug in the production code or the transcription; establish which before reporting. State how many vectors agree and the integer widths in both implementations. If the production comparison was not run, say so beside the proof verdict.
 
 When a theorem needs a proof beyond bounded search and the host offers Lean language-server tools for goals and diagnostics, use them to shorten the loop. This plugin neither requires nor configures such a server.
 

package/skills/formal-verify/references/rust-brief.md

@@ -2,7 +2,8 @@
 
 Use Kani for a Rust function whose inputs and loops can be bounded. Call the real
 function from one `#[kani::proof]` harness per property. Do not transcribe its
-implementation. Start from [the bounded queue template](../examples/kani-template/src/lib.rs).
+implementation. Edit an existing harness in place, and start a new one from
+[the bounded queue template](../examples/kani-template/src/lib.rs).
 
 ## Files
 
@@ -13,9 +14,21 @@ Coordinate any shared crate or module wiring with its owner. Keep production
 files untouched unless the user has authorized a production fix.
 
 Cite the production source paths and line numbers that each property models in
-the harness header. List caller constraints, bounds and stub assumptions there,
-with source citations or an explicit statement that each is an assumption.
-Call the production function through its real crate or module interface.
+the harness header, the functions it calls, and the commit they were read at as
+`Source read at <commit>`. List caller constraints, bounds and stub assumptions
+there, with source citations or an explicit statement that each is an
+assumption. Call the production function through its real crate or module
+interface.
+
+## Existing model
+
+The row in your task gives the commits since the harness was last synced and
+says whether any of them changed the guarantee on purpose. The harness calls the
+production code, so drift shows as a compile error or a failed check. Update its
+bounds, stubs, contracts and covers where they no longer fit the code. Change a
+property only where the row cites that evidence. Add a mutation, with `detects`
+naming the harness, for each branch or step the update adds. Update the header
+to the code you read.
 
 ## Model
 

package/skills/formal-verify/references/tla-brief.md

@@ -4,7 +4,11 @@ Follow this brief to model one thread protocol with TLA+ and TLC. Your task give
 
 ## Files
 
-Write `tla/<Name>.tla`, `tla/<Name>.matrix` and `tla/<Name>.mutations` in the target repository, or the same names in its existing model directory. Start from the three `BoundedQueue` files in `<skill-dir>/examples/tla-template/`. Write no other file, so parallel agents share none. Name the top-level formula `Spec`. The header comment names the source files and line ranges the spec follows and each assumption it makes.
+Write `tla/<Name>.tla`, `tla/<Name>.matrix` and `tla/<Name>.mutations` in the target repository, or the same names in its existing model directory. Start a new model from the three `BoundedQueue` files in `<skill-dir>/examples/tla-template/`, and edit an existing model's files in place. Write no other file, so parallel agents share none. Name the top-level formula `Spec`. The header comment names the source files and line ranges the spec follows, the functions and shared variables it covers, the commit they were read at as `Source read at <commit>`, and each assumption it makes.
+
+## Existing model
+
+The row in your task gives the commits since the model was last synced and says whether any of them changed the guarantee on purpose. Change the spec where it no longer describes the code. Change a property only where the row cites that evidence. Add a mutation for each step, branch or wait the update adds, naming the property that must fail on it. Update the header to the code you read.
 
 ## Model
 

package/skills/formal-verify/scripts/kani.py

@@ -44,9 +44,11 @@ class Outcome:
     output: str
 
 
-def search_path(variable, *directories):
-    inherited = [os.environ[variable]] if variable in os.environ else []
-    return os.pathsep.join([*(str(directory) for directory in directories), *inherited])
+def search_path(inherited, *directories):
+    entries = [str(directory) for directory in directories]
+    if inherited is not None:
+        entries.append(inherited)
+    return os.pathsep.join(entries)
 
 
 class Kani:
@@ -67,8 +69,8 @@ class Kani:
         self.env = os.environ | {
             "KANI_HOME": str(home),
             "RUSTUP_TOOLCHAIN": str((bundle / "toolchain").resolve()),
-            "PATH": search_path("PATH", bundle / "bin", bundle / "pyroot/bin", bundle / "toolchain/bin"),
-            "PYTHONPATH": search_path("PYTHONPATH", bundle / "pyroot"),
+            "PATH": search_path(os.environ.get("PATH"), bundle / "bin", bundle / "pyroot/bin", bundle / "toolchain/bin"),
+            "PYTHONPATH": search_path(os.environ.get("PYTHONPATH"), bundle / "pyroot"),
         }
         # A parent cargo (cargo run, cargo test, a build script) exports its rustup toolchain's
         # lib directory here; the bundled compiler must load only the bundled toolchain.