external-review 1.0.0

package/docs/PLAYBOOK.md

@@ -0,0 +1,234 @@
+# Playbook
+
+Everything below came out of running this against a production app and being
+wrong in interesting ways. It is ordered by how much time each item saves.
+
+---
+
+## 1. The signature
+
+Every real defect a second model found shared one shape:
+
+> **The codebase already argued the correct rule somewhere else, and had not
+> applied it here.**
+
+Concretely, from one codebase in one day:
+
+| The rule, stated | Where it was not applied |
+|---|---|
+| "a non-bool value reads as false instead of throwing and **dropping the whole bike**" | Two hard casts, two lines above |
+| The sync path gated a one-time calibration on an adoption stamp | The link path gated it on "is the id null", so every reconnect recalibrated |
+| A helper whose own doc says "never silently substitute another value" | Seven call sites that substituted zero |
+| "back up the live document, but **only if it is good**" — argued in full | Two sibling stores that backed it up unconditionally, promoting corruption over the last good copy |
+| "a bad tier entry costs that tier's line, never the record" | The object that *owns* the tiers hard-cast and lost the record |
+
+Put this sentence in every review prompt. It converts an unbounded search into a
+bounded one: enumerate the codebase's stated intentions, then check each one's
+siblings.
+
+**How to hunt it yourself:** grep for comments that explain *why* — "must",
+"never", "otherwise", "this is why". Each one is a rule. For each rule, find the
+other places of that shape and check them.
+
+---
+
+## 2. Four ways a vacuity check lies
+
+You fixed something, wrote a regression test, and it passes. Before you believe
+it, revert the fix and confirm the test fails. That check itself fails in four
+ways, all observed:
+
+### The revert did not apply
+A scripted `replace` whose pattern did not match silently no-ops. The run
+reports "all passed", which reads as *the test is vacuous* when in fact the fix
+was never removed.
+
+```python
+old = "..."
+assert old in source, "REVERT DID NOT APPLY"   # ← this line
+source = source.replace(old, new, 1)
+```
+
+### It applied in the wrong place
+A blind replace hitting the first of four identical lines in a large file. You
+reverted something, just not the thing under test. Assert on a unique enough
+anchor, or assert the resulting line count changed as expected.
+
+### The run failed for the wrong reason
+A compile error or a missing fixture is not your assertion failing. Check the
+failure output names your test and your expectation — not the compiler.
+
+### The test never reached the defect
+The most common and the most convincing. The fixture looked right, the code path
+was never entered, and the assertion was true for an unrelated reason.
+
+Two real examples:
+
+- A test for a "cancelled animation leaves the highlight stale" bug asserted on
+  the **page position**, which moves whether or not the bug is present. It had to
+  assert on the **highlight**.
+- A test for "this value must not be shifted" set up a case where *no* shift
+  happened at all, so the code path under test was skipped entirely and the value
+  was untouched either way.
+
+**Guard:** make the precondition the test's first assertion.
+
+```dart
+expect(pinBefore, closeTo(30, 0.01),
+    reason: 'precondition: the pin is live, or this test proves nothing');
+```
+
+Know its limit: this catches a fixture that never reaches the defect. It does
+**not** catch a wrong *second* step — a re-run at the same timestamp that takes a
+different branch, say. Reverting the fix is the half that always works.
+
+---
+
+## 2b. A secret scanner's test fixtures look exactly like secrets
+
+Worth knowing before you write them. The first commit of this repo's own scanner
+tests was rejected by GitHub push protection: the fixture contained a
+Stripe-key-shaped string, which is precisely what the test needs to contain.
+
+The wrong fix is the allowlist link in the rejection message. It teaches the
+repository to ignore that class of finding, and the next one might be real.
+
+The right fix is to assemble the literal at runtime so it never appears in the
+source:
+
+```js
+const stripe = ['sk', 'live', '51H8xQ2eZvKYlo2Cabcdefghijklmnop'].join('_');
+```
+
+Same reasoning as §3 below: a fixture that trains a safety mechanism to stay
+quiet is worse than no fixture.
+
+---
+
+## 3. Your existing tests may pin the bug as correct
+
+Two separate codebases had a test whose "corrupt record" fixture was really a
+*recoverable* record with one mistyped field:
+
+```dart
+{'id': 'bad', 'year': 'not-a-number'}   // ← throws, so the record is skipped
+```
+
+The test asserted that this record gets **dropped**. It was green for months. It
+was asserting that recoverable rider data gets deleted.
+
+When you make decoding more tolerant, tests like this fail — and the reflex is to
+"fix the test". Stop and ask which behaviour is right. Then repoint the fixture
+at something genuinely undecodable (no id at all), so the skip path stays covered,
+and add a test for the opposite direction.
+
+---
+
+## 4. Assert intact, not present
+
+`expect(bike.components, hasLength(1))` passes for a component that survived
+decode as a husk with every field defaulted. That is the same data loss in a
+different shape.
+
+Assert the *irreplaceable* field — the baseline, the service date, the price.
+And where an autosave can make a loss permanent, drive the full round trip:
+
+```dart
+final once = Model.fromJson(damaged);
+final twice = Model.fromJson(once.toJson());
+expect(twice.components.single.serviceKmBaseline, 300);
+```
+
+---
+
+## 5. Prompt shapes that worked
+
+Scope to **one subsystem per pass**, and always include:
+
+- Point at the invariant docs by name; say a violation is usually a real bug.
+- The signature from §1, verbatim.
+- What to EXCLUDE, so passes do not overlap.
+- Required output per finding: severity, `file:line`, a concrete failure sequence
+  **with specific values**, why existing tests miss it, the minimal fix.
+- "Verify each claim against the code and quote the lines."
+- "No style, naming or refactor opinions."
+- "If a subsystem is sound, say so in one line rather than padding."
+- "A confident wrong finding is worse than no finding."
+- A cap: "stop at your 10 strongest." Quality tails off badly past that.
+
+### Always ask for "HELD UP"
+
+A section listing what it checked and found **correct**, one line each.
+
+This is your calibration instrument. If it independently re-derives things you
+know to be true, the findings are worth acting on. If it asserts something you
+know is false, discount the whole pass. On one run the held-up section
+independently traced a compliance firewall and confirmed no restricted data
+reached an AI path — which was worth more than any single finding.
+
+### Ask for "POLISH" on user-facing areas
+
+Up to five concrete UX gaps with `file:line`: a state with no affordance, an
+action with no feedback, a number with no unit, an empty state that says nothing.
+Different question, different answers.
+
+### Scope by journey, not by directory
+
+"The first-run path: install → connect → first item → its photo" finds what
+"review `lib/features/`" does not. Directory-shaped passes miss the handoffs
+between modules, which is where bugs live. Feature-shaped passes cross them by
+construction.
+
+---
+
+## 6. Reading findings
+
+Real outcomes from real passes:
+
+- **Refuted by reading the code.** The finding described a fall-through that did
+  not exist. Cost: ten minutes. Worth it.
+- **Wrong premise, real bug.** It named the wrong function as the trigger. The
+  bug was real via a different path. **Read past the premise.**
+- **First half wrong, second half a P0.** It claimed a fix had never been applied
+  (it had — the model read a stale copy), and the *rest* of the same finding
+  described a genuine data-loss path nobody had noticed.
+- **Stale line numbers are a tell** that a finding was written against an old
+  tree and never re-checked.
+
+---
+
+## 7. Practicalities
+
+- **Sync to a dated directory**, never to a long-lived checkout — it may carry
+  another session's uncommitted work, and you will review the wrong tree.
+- **A pass will wander** into sibling checkouts if they exist on the machine.
+  Put "work only inside the current working directory" in the prompt, and check
+  the first lines of the output to see which tree it actually read.
+- **Check the working directory of every pass you launch, not just the first.**
+  This bit hard. Launching two in one shell line:
+
+  ```bash
+  cd ~/review-dir && nohup runner … & nohup runner … &     # WRONG
+  ```
+
+  The `&&` binds to the FIRST command only. The second runs in your home
+  directory, finds a stale long-lived checkout next door, and reviews *that* —
+  producing findings against code you fixed hours ago, with no error anywhere.
+  The tell is in stderr: a correct pass reads `lib/foo.dart`, a wandering one
+  reads `myproject/lib/foo.dart`. Wrap each launch in its own subshell:
+
+  ```bash
+  (cd ~/review-dir && nohup runner … &)
+  (cd ~/review-dir && nohup runner … &)
+  ```
+
+  Related: never sync to a long-lived checkout in the first place. If the only
+  copy on the machine is the dated one you just made, a wandering pass has
+  nowhere stale to wander to.
+- **Run two or three concurrently** on different scopes; a pass is mostly waiting
+  on an API.
+- **`stderr` is the liveness signal** — it shows which files are being read.
+- **Reviewing your own session's recent work is the highest-value use.** That is
+  exactly where your blind spots are. Several of the worst defects found this way
+  were in code written hours earlier, including a comment that promised the exact
+  behaviour its own code failed to deliver.

package/docs/PRIVACY.md

@@ -0,0 +1,182 @@
+# What leaves your machine
+
+A review means sending source code to a third-party API. That is the deal, and
+it should be an informed one rather than a hopeful one.
+
+## What is sent
+
+**Your source files, in the clear, over TLS, to whichever provider serves the
+model you chose.** The runner reads files in the directory you point it at and
+puts their contents in the prompt. There is no filtering, no obfuscation, and no
+way to review code without sending it.
+
+Also sent: your prompt, and — for the CLI's own commands — nothing but the model
+id you asked about.
+
+## What is not sent
+
+- Anything matched by the sync exclusions (see below).
+- Your API key never leaves your machine except as an `Authorization` header to
+  OpenRouter, which is what it is for.
+- This tool has no telemetry. It makes no network calls except to
+  `openrouter.ai` and, for `sync`, to the host you name.
+
+## Deciding whether that is acceptable
+
+Run:
+
+```bash
+external-review providers <model-id>
+```
+
+For every machine that may serve your request, this prints:
+
+- the operator's **headquarters**,
+- its published **datacenter regions**,
+- links to its **actual privacy policy and terms**.
+
+These are facts published by OpenRouter, not a rating. This tool does not tell
+you which providers are trustworthy, because that is not a technical question
+and the answer differs per user. What it does is make sure you are not guessing.
+
+Things worth knowing while you decide:
+
+**A model is routed per request.** If it lists four endpoints, any of the four
+may serve a given call. To pin one, use OpenRouter's `provider.order` routing
+or pick a model with a single endpoint.
+
+**"Free" is a business model, and on OpenRouter the price is your data.** This
+is stronger than a caveat and it is the single most important thing on this
+page.
+
+To use free models at all you must enable, in Settings → Privacy:
+
+- *"Enable free endpoints that may train on inputs"*
+- *"Enable free endpoints that may publish prompts"*
+
+Without them, free models return
+`404: No endpoints available matching your guardrail restrictions and data
+policy`. So if free models work for you, those toggles are on, and **the code
+you send is permitted to be trained on and published.**
+
+**Stealth models are the same deal, more so.** A "stealth" or "cloaked" model is
+an unreleased model shipped under an anonymous name to gather real-world usage.
+That is the entire point: prompts and completions are logged and used to improve
+it. They are attractive for review work — often frontier-class, large context,
+free — and they are the least private option on the menu.
+
+**Zero Data Retention is the actual control.** ZDR means a provider will not
+store your data for any period, and cannot train on it. Enable it three ways:
+
+- account-wide, in privacy settings, globally or per model group;
+- per API key, as a guardrail, which is how you give a teammate a key that
+  cannot leak;
+- per request: `"zdr": true` in the provider preferences.
+
+The cost is real: ZDR **removes endpoints**, including most or all free ones,
+and it does not cover plugins such as web search, which carry their own
+policies.
+
+So the decision is simple to state, if not to make:
+
+| If the code is… | Use |
+|---|---|
+| open source, or you do not mind it training a model | free endpoints — that is the trade |
+| private but not contractually restricted | a paid endpoint whose terms you have read |
+| under NDA, customer contract, or a residency rule | ZDR, or a model you host yourself |
+
+Do not let "it is only a code review" carry the decision. A review prompt
+contains more of your source, in one place, than almost anything else you send
+anywhere.
+
+**Self-hosting removes the question.** A local model via Ollama or vLLM, or a
+model on infrastructure you control, sends nothing anywhere. It costs you
+capability, and for reviewing a subsystem that tradeoff is often fine.
+
+**Your obligations are yours.** If the code is under an NDA, a customer contract,
+export control, or a regulation with data-residency requirements, none of the
+above substitutes for checking. Ask, do not assume.
+
+## Secrets
+
+The most likely accident is not the model reading your code — you meant that.
+It is a credential riding along inside it.
+
+`external-review sync` excludes the usual shapes:
+
+```
+.env  .env.*  *.pem  *.der  *.key  *.jks  *.keystore  *.p12
+*.mobileprovision  id_rsa*  *.crt
+secrets.*  *.secrets.*  credentials.*  service-account*.json
+.git/  node_modules/  build/  dist/  target/  .venv/  __pycache__/
+```
+
+Then it **verifies over SSH that those paths are absent from the copy**, because
+an exclusion pattern that silently failed to match is the entire risk, and
+`rsync` exits 0 either way.
+
+**The default list cannot know about your repo.** Real example: a project's list
+covered `*.jks` and `strava.config.json` but not `garmin/developer_key.pem` — a
+code-signing key that would let someone publish releases as that developer. It
+was only caught because a second person reviewed the list.
+
+So: before the first sync, list your own credential paths and add them with
+`--exclude`. Then read the verification output rather than assuming it passed.
+
+If a secret does reach a review copy, treat it as disclosed: delete the copy,
+rotate the credential, and do not rely on the provider's retention policy to
+make it un-disclosed.
+
+## Running the review on a VM or VPS
+
+Worth doing. Also worth being precise about, because the reason people usually
+give for it is the one thing it does not help with.
+
+**It does NOT reduce what the model sees.** The prompt is byte-for-byte the
+same whether the runner executes on your laptop or on a box in a datacenter.
+If the endpoint may train on your code, it may train on it either way. Anyone
+telling you a VM makes the *disclosure* safer is confusing two different risks.
+
+**What it genuinely does:**
+
+**1. It contains the runner, which is the under-discussed risk.** A review
+runner is a third-party binary executing an agentic loop with filesystem access
+and, usually, shell access. It reads whatever it decides it needs. On your
+laptop that neighbourhood includes `~/.ssh`, your cloud credentials, your
+browser profile, your password-manager exports, your other clients' repositories
+and every uncommitted branch you have. On a throwaway VM it includes a dated
+copy of one project.
+
+That is not a hypothetical about malice — it is about scope. Agentic tools
+wander. On a real run, a pass launched from the wrong working directory found a
+sibling checkout and reviewed *that* instead, unprompted and without error. It
+had no reason to and no instruction to. It just could.
+
+**2. It keeps your API key off your daily machine.** The key lives on the box
+that uses it. A key on a laptop is a key in a laptop backup.
+
+**3. It forces a clean, dated, minimal copy.** You review exactly what you
+synced — not your working tree with its uncommitted experiments, its `.env.local`
+and its half-finished spike branch. This is also the only way `scan` and the
+exclusion verification can mean anything, because they run against a snapshot
+rather than a moving target.
+
+**4. It makes the blast radius disposable.** If something goes wrong — a secret
+synced by mistake, a runner that misbehaves — you destroy the VM. You cannot
+destroy your laptop.
+
+**How to set it up sensibly**
+
+- A cheap VPS or a local VM is fine. It needs a shell, `rsync` and the runner;
+  it does not need to be fast, because the work happens at the API.
+- Give it **no standing credentials** beyond the model API key. No cloud
+  provider keys, no deploy keys, no production access.
+- Sync to a **dated directory per review**, never to a long-lived checkout.
+  Delete it afterwards. A stale checkout on the review box is how a pass ends up
+  reviewing code you fixed hours ago and reporting it as broken.
+- Do not develop on it. The moment it becomes a second workstation it has the
+  same neighbourhood problem as the first one.
+
+**When it is not worth it:** reviewing open-source code you would publish
+anyway. The runner-containment argument still applies, but the stakes are low
+enough that a local run in a clean checkout is a reasonable trade.

package/examples/prompts/data-integrity.txt

@@ -0,0 +1,49 @@
+HARD CONSTRAINT: work ONLY inside the current working directory. Never read or
+reference anything outside it. If a file you want is not here, reason from what
+is and say what you could not check.
+
+You are a senior engineer reviewing this codebase for defects that LOSE, CORRUPT
+or SILENTLY MISREPORT the user's data. Read-only: report, do not edit files.
+
+Read the project's instruction files first (CLAUDE.md, AGENTS.md, README,
+CONTRIBUTING - whichever exist). They state this codebase's invariants, and a
+violation of one is usually a real bug.
+
+THE HIGHEST-YIELD THING TO HUNT: a rule stated and argued in ONE place, and
+silently not applied in its sibling case. Find a comment that explains WHY
+something must be done a certain way - "must", "never", "otherwise", "this is
+why" - then check every OTHER site of that same shape. In this codebase's
+history that pattern alone accounts for most of the real data-loss bugs.
+
+SCOPE: <fill in: the specific directories or subsystem>
+
+EXCLUDE (already audited, do not spend budget there): <fill in, or "nothing">
+
+WHAT TO HUNT, in priority order:
+1. PERSISTENCE. Any path where a record is dropped, truncated, or overwritten by
+   a worse copy. Decoding that throws and takes a whole record with it when one
+   field was recoverable. A backup slot that can be filled with a corrupt value.
+   A write that reports success it did not achieve.
+2. MIGRATIONS AND SCHEMA. Anything that must be idempotent across
+   decode -> encode -> decode and is not. A key that inflates or drops on a
+   re-read. Tolerance in one direction and a hard cast in the other.
+3. CONCURRENCY. A read-modify-write across an await. Two writers to one record
+   with no ordering. A cached snapshot written back over a newer value.
+4. USER INPUT. A typed value that is silently substituted, dropped, or
+   reinterpreted - especially numbers, dates, and anything locale-dependent.
+5. DESTRUCTIVE ACTIONS. Delete, replace, reset, disconnect: is each confirmable
+   or reversible, does any remove more than asked, does undo restore what was
+   actually removed?
+
+For EVERY finding: SEVERITY (P0 data loss / P1 wrong data shown / P2 other),
+file:line, a CONCRETE failure sequence with specific starting state and specific
+values, why existing tests miss it, and the minimal fix.
+
+Then a section "HELD UP": what you checked and found genuinely correct, one line
+each. Be specific - this is how your findings get calibrated, and a held-up list
+that independently re-derives known-good facts is what makes the rest credible.
+
+Rules: verify each claim against the code and quote the lines. No style, naming
+or refactor opinions. If an area is sound, say so in one line rather than
+padding. A confident wrong finding is worse than no finding. Rank by severity,
+stop at your 10 strongest.

package/examples/prompts/second-opinion.txt

@@ -0,0 +1,41 @@
+HARD CONSTRAINT: work ONLY inside the current working directory.
+
+You are reviewing code that another AI session wrote or changed VERY RECENTLY,
+and you are the second opinion on its own work. This is the highest-value use of
+an independent model: it is precisely where the first one's blind spots are.
+
+Read the project's instruction files first.
+
+THE CHANGES: <fill in - paste the diff, name the commits, or list the files>
+
+WHAT THE AUTHOR BELIEVED: <fill in - paste the commit messages or the summary it
+gave. You are checking whether the code does what these claim.>
+
+BE ADVERSARIAL ABOUT THE NEW CODE SPECIFICALLY. In particular:
+
+1. DOES IT DO WHAT ITS OWN COMMENTS SAY? A comment promising behaviour the code
+   does not deliver is this codebase's most common defect shape, and freshly
+   written code is where it appears. Check each claim against the lines under it.
+2. SECOND-ORDER EFFECTS. A fix that changes when something runs, what survives a
+   failure, or what a later step observes. Ask specifically: did this change
+   remove an accident that was hiding a different bug? Did it make a previously
+   unreachable path reachable?
+3. THE SIBLING SITES. If the fix applied a rule in one place, find every other
+   place of that shape and check whether it was applied there too. A partial
+   sweep is the normal outcome and it looks identical to a complete one.
+4. THE TESTS THAT SHIPPED WITH IT. For each new test, ask: would this fail if the
+   fix were reverted? Does the fixture actually reach the code path? Does it
+   assert the record is INTACT or merely present? Does an existing fixture now
+   pin the old broken behaviour as correct?
+5. WHAT THE CHANGE DID NOT COVER. The case the author did not think of - the
+   other axis, the other branch, the null, the concurrent writer.
+
+For EVERY finding: SEVERITY, file:line, a CONCRETE failure sequence, why the new
+tests miss it, and the minimal fix.
+
+Then a section "HELD UP": which of the author's claims you verified as true, one
+line each. Be specific about what you checked - this is what makes the findings
+worth acting on.
+
+Rules: verify each claim against the code and quote the lines. No style opinions.
+A confident wrong finding is worse than no finding. Stop at your 10 strongest.

package/examples/prompts/user-journey.txt

@@ -0,0 +1,47 @@
+HARD CONSTRAINT: work ONLY inside the current working directory. Never read or
+reference anything outside it.
+
+You are a senior engineer reviewing ONE USER JOURNEY end to end. Scope by what a
+person actually does, not by directory - the handoffs between modules are where
+the bugs are, and a directory-shaped review never crosses them.
+
+Read the project's instruction files first. They state the invariants.
+
+THE JOURNEY: <fill in, e.g. "fresh install -> sign in -> create the first item
+-> attach a photo to it -> see it in the list">
+
+Follow it through every file it touches, in order, including the seams between
+them.
+
+Read the project's instruction files first, and hunt above all for: a rule
+stated and argued in ONE place and silently not applied in its sibling case.
+
+WHAT TO HUNT:
+1. ABANDONMENT. What is left behind when the user backs out halfway, loses
+   connectivity, or kills the app mid-step? A half-created record, an orphaned
+   file, a one-shot flag consumed before the thing it guards actually happened,
+   a credential stored with nothing to use it on.
+2. LOST INPUT. Any value typed or picked that can be silently discarded - a
+   sheet dismissed mid-edit, a field never committed on close, an await after
+   which the writer is gone.
+3. STATE THAT LIES. A value read once and never refreshed; a summary of a
+   heterogeneous list that shows one item's setting while writing to all of
+   them; an optimistic update never rolled back on failure.
+4. FABRICATION. Any place a number or status is shown that the system does not
+   actually know - a zero rendered where "unknown" is the truth, an estimate
+   presented as a measurement, a default substituted for a failed parse.
+5. THE FIRST-RUN SPECIFICS. Anything that behaves differently on a fresh
+   install, after a reinstall, or on a device where a platform capability is
+   missing entirely.
+
+For EVERY finding: SEVERITY, file:line, a CONCRETE sequence of user actions
+leading to a specific wrong outcome, why existing tests miss it, the minimal fix.
+
+Then TWO more sections:
+- "HELD UP": what you checked and found correct, one line each.
+- "POLISH": up to 5 concrete UX gaps this journey has - a state with no
+  affordance, an action with no feedback, a number with no unit, an empty state
+  that says nothing useful. Each with file:line and one sentence of rationale.
+
+Rules: verify each claim against the code and quote the lines. No style opinions.
+A confident wrong finding is worse than no finding. Stop at your 10 strongest.

package/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "external-review",
+  "version": "1.0.0",
+  "description": "Review your code with a second, independent model - and know what it costs, where your source goes, and which findings to believe. Ships a Claude Code skill.",
+  "type": "module",
+  "bin": {
+    "external-review": "bin/external-review.mjs"
+  },
+  "files": [
+    "bin",
+    "skills",
+    "docs",
+    "examples",
+    "README.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "code-review",
+    "ai",
+    "claude",
+    "claude-code",
+    "skill",
+    "openrouter",
+    "llm",
+    "static-analysis",
+    "second-opinion",
+    "privacy"
+  ],
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/yevgavrikov/claude-external-review.git"
+  },
+  "bugs": {
+    "url": "https://github.com/yevgavrikov/claude-external-review/issues"
+  },
+  "homepage": "https://github.com/yevgavrikov/claude-external-review#readme",
+  "scripts": {
+    "test": "node --test \"test/**/*.test.mjs\""
+  }
+}