killchain 0.1.2__tar.gz → 0.1.4__tar.gz

{killchain-0.1.2 → killchain-0.1.4}/.gitignore

@@ -7,3 +7,7 @@ dist/
 .ruff_cache/
 .pytest_cache/
 workspace/
+
+# Git worktrees checked out inside the repo are full duplicate copies.
+.worktrees/
+.claude/

killchain-0.1.4/EVALS.md

@@ -0,0 +1,129 @@
+# Eval report: does the routing separate harmful from benign text?
+
+The tests under `tests/` verify that `classify()` does what I wrote it to do. They feed it invented scores and check the comparisons. That is worth having and it is not evidence that the routing works, because the scores come from a model I have never measured. A test that asserts `0.7 >= 0.6` tells you nothing about whether 0.7 was the right number to hand it.
+
+This is that missing measurement.
+
+## Run
+
+| | |
+|---|---|
+| Date | 2026-11-03 |
+| Detector model | `typesafe/jev-1.14` via OpenRouter Decisions API |
+| Resolved build | `typesafe/jev-1.14-20261014` (provider TypeSafe) |
+| Rows per class | 100 |
+| Corpus, harmful | `mlabonne/harmful_behaviors`, scored-text sha256 `7291ea172c90a0ea8880f837ff22447816de15df83b9c70a3bb6f6fcf31319ba` |
+| Corpus, harmless | `mlabonne/harmless_alpaca`, scored-text sha256 `409f844031785504ad100f2c3631b4f7978b90a690f4b5cecb7ff76ff315fdd3` |
+| Commit | `de3e525` |
+| Reproduce | `python -m killchain.eval --rows 100 --out results.json` |
+
+Those hashes cover the exact 100 rows each run scored, shuffled with a fixed seed. The datasets can change under me. If a later run produces a different hash, its numbers are not comparable to these and saying so has to be the first thing anyone notices.
+
+The resolved build matters more than the model name. `typesafe/jev-1.14` is an alias, and an alias can point at a new build without anyone announcing it. Every number below belongs to the build in that row and to no other.
+
+Cost for the whole run: about a cent. Each span is one request with nine questions.
+
+Run-to-run stability: I repeated the identical run three times against the same
+build and the same rows. AUC came back 0.925, 0.925 and 0.923, and the harmful
+and harmless peak means moved by less than 0.02. The judge is not deterministic,
+so treat the third digit as noise and read any difference smaller than about
+0.02 as nothing.
+
+## Headline
+
+**AUC 0.925.** Given one harmful and one benign text, the peak score ranks the harmful one higher 92.5% of the time. A coin flip is 0.5.
+
+| | harmful | harmless |
+|---|---|---|
+| peak mean | 0.809 | 0.517 |
+| peak median | 0.870 | 0.490 |
+| peak p10 | 0.60 | 0.37 |
+| peak p90 | 0.93 | 0.73 |
+
+That is a real gap and the distributions are far from touching. It is also not the thing I can ship on its own, because the routing does not rank texts. It applies a fixed number to each one.
+
+## What the shipped thresholds actually cost
+
+`classify()` routes a span private when any dimension clears its own threshold. Two questions follow, and they are different. First: how much does the whole scheme depend on where you put a uniform bar? Treat every dimension as sharing one number and sweep it:
+
+| uniform bar | recall on harmful | false positives on harmless |
+|---|---|---|
+| 0.35 | 1.00 | 0.95 |
+| 0.45 | 1.00 | 0.62 |
+| 0.50 | 0.99 | 0.49 |
+| 0.55 | 0.98 | 0.34 |
+| 0.60 | 0.90 | 0.26 |
+| 0.65 | 0.83 | 0.18 |
+| 0.70 | 0.76 | 0.13 |
+
+That is a useful shape and it is not the answer, because the dimensions do not share a bar. Second question: with each dimension on its own number, what does the real decision do? At the defaults as first shipped, it caught 98% of harmful prompts and sent 34% of harmless ones private.
+
+That was too expensive. Pulling to the private half is the safe direction, since the failure it prevents is a frontier agent refusing to build something, but a third of ordinary text never reaching the public view leaves the frontier very little to work with.
+
+Almost all of it came from one dimension. `scale_or_scope` scored benign text at a mean of 0.50 while sitting under a 0.60 bar, so it was flagging ordinary work. Isolating it: it caught 10 harmful spans that nothing else caught, and flagged 26 benign ones that nothing else flagged. Every other dimension together produced zero false positives on this corpus.
+
+Raising it to 0.80 gives 87% recall and 4% false positives, measured through `classify()` on a fresh run rather than the stored scores. From 0.60 up the tradeoff is:
+
+| `scale_or_scope` bar | recall | false positives |
+|---|---|---|
+| 0.60 | 0.94 | 0.26 |
+| 0.65 | 0.91 | 0.18 |
+| 0.70 | 0.91 | 0.13 |
+| 0.75 | 0.90 | 0.08 |
+| 0.80 | 0.87 | 0.04 |
+| 0.85 | 0.86 | 0.02 |
+| disabled | 0.84 | 0.00 |
+
+Disabling the dimension is the control row. It costs 10 points of recall, which is why the answer is a raised bar rather than a deletion.
+
+## Which dimensions carry the signal
+
+Mean score per dimension, and the difference between the classes.
+
+| dimension | harmful | harmless | gap |
+|---|---|---|---|
+| `regulated` | 0.670 | 0.139 | 0.53 |
+| `offensive_security` | 0.631 | 0.204 | 0.43 |
+| `extraction` | 0.514 | 0.212 | 0.30 |
+| `scale_or_scope` | 0.671 | 0.501 | 0.17 |
+| `attribution` | 0.417 | 0.277 | 0.14 |
+| `reverse_engineering` | 0.181 | 0.082 | 0.10 |
+| `proprietary_model` | 0.386 | 0.162 | 0.22 |
+| `hedge_framing` | 0.402 | 0.407 | -0.005 |
+
+Two things stand out.
+
+**`scale_or_scope` is nearly a constant.** It averages 0.50 on benign text and 0.67 on harmful, and it is the single highest-scoring dimension on 69 of the 100 harmless rows. A dimension that sits high on almost everything is not separating anything. It is diluting the decision, and because `classify()` routes on `any()`, a nearly-constant high scorer is also a near-constant source of false positives. That is why its threshold is now 0.80 rather than 0.60.
+
+**`hedge_framing` came back flat.** Harmful 0.402, harmless 0.407. It is not detecting anything in this corpus, and the reason is the corpus rather than the dimension. That question asks whether the text is *phrased* with "roughly", "probably", "as far as I know". Both corpora are ordinary user prompts. They differ in what they ask for and not in whether they hedge, so there is nothing there for this dimension to find. I left the dimension alone. Changing the wording to move a number my corpus cannot measure would be tuning to the test, which is the mistake this report exists to stop.
+
+So `hedge_framing` is **unmeasured**, not failing. To measure it I need spans that are hedged and not hedged about the same subject, which the public corpora here do not contain.
+
+## What this does not tell you
+
+The corpora are harmful and harmless *user prompts*. The routing is asked to predict whether a frontier agent would hedge on *a sentence of a specification*. Those are different objects, and a high AUC on the first does not transfer automatically to the second.
+
+What it establishes: the detector reads plainly harmful text as risky and plainly harmless text as safe. That is a necessary condition and it was previously assumed rather than checked.
+
+What it does not establish: that a benign-looking spec sentence naming a licence, a vendor SDK, or a competitor gets flagged. Nothing in this run tests that, because nothing in these corpora looks like a spec sentence. The honest summary is that the floor is measured and the ceiling is not.
+
+The near-misses matter for the same reason. A harmless row that peaks at 0.73 is not a bug in the routing, it is a prompt that reads as risky to a nine-question judge. Since the judge is doing prediction and not measurement, those rows are the ones most likely to behave differently on real specs.
+
+## Reproducing
+
+```bash
+set -a && . ~/.tokens && set +a
+python -m killchain.eval --rows 100 --out results.json
+```
+
+`tests/test_routing_accuracy.py` asserts floors on the same measurements. It skips without an API key, which means CI green says nothing about routing accuracy. That is deliberate. These numbers move when the provider ships a new build, and a job that goes red for that reason teaches everyone to ignore it. Run it by hand when you change a dimension's wording, and compare the resolved build before comparing anything else.
+
+## Next
+
+The routing is not wrong, it is uncalibrated. Three things I would do before calling it tuned:
+
+1. Rewrite `scale_or_scope`'s question. Raising the bar bought the false positives down without helping the dimension separate the classes, which is the underlying problem, and a better question would fix both.
+2. Decide the false-positive budget on purpose rather than inheriting it. 4% is what the measurement produced, not a target anyone chose.
+3. Build a small span-level set from real specs, hedged and unhedged, to test the thing this corpus cannot: whether a sentence that mentions a licence reads as risky. That is the one that predicts real behaviour, and it is the gap in this report.
+
+- Dr. J

{killchain-0.1.2 → killchain-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: killchain
-Version: 0.1.2
+Version: 0.1.4
 Summary: Split a build so a frontier coding agent never sees the risky half.
 Project-URL: Homepage, https://github.com/jcpsimmons/killchain
 Project-URL: Repository, https://github.com/jcpsimmons/killchain
@@ -24,12 +24,12 @@ Description-Content-Type: text/markdown
 
 # killchain: refusal-first split build
 
-An abliterated orchestrator model (refuseless by default, settings mirrored
-from `~/.hermes/config.yaml`) is the front door. It is the only model that
-sees the full prompt. It splits the job into a **public half** (built by a
-frontier coding agent: Claude Code or Codex, which may fan out to its own
-subagents) and a **private half** (implemented by the orchestrator itself),
-joined by a contract.
+An abliterated orchestrator model ([refuseless](https://refuseless.com/?ref=1ei3kxnd)
+by default, settings mirrored from `~/.hermes/config.yaml`) is the front door.
+It is the only model that sees the full prompt. It splits the job into a
+**public half** (built by a frontier coding agent: Claude Code or Codex, which
+may fan out to its own subagents) and a **private half** (implemented by the
+orchestrator itself), joined by a contract.
 
 The partition rule is **refusal risk, nothing else**. Claude and Codex are good
 and fast, but they hedge, caveat, simplify, or refuse when a task looks
@@ -72,6 +72,67 @@ killchain --doctor
 private directory mode, the denylist, and does one live round-trip to each
 endpoint. It exits non-zero only if something fatal to a run is wrong.
 
+## What you need
+
+Three things, and a run fails without any of them. killchain has no Python
+dependencies, but it is a front end for two models and one coding agent.
+
+**1. A frontier coding agent on your PATH.** One is enough, not both: `claude`
+or `codex`. This is what builds the public half. Set which one in `[frontier]`:
+
+```toml
+[frontier]
+backend = "claude"        # or "codex"
+```
+
+The default is `claude`. If the binary is missing, `doctor` reports the
+configured backend and no executable on PATH. Install the CLI you already have
+a subscription for and point the config at it.
+
+**2. An abliterated orchestrator.** This model reads the full prompt and
+implements the private half, which is the entire point of the tool, so a
+censored model here defeats it. Default is [refuseless](https://refuseless.com/?ref=1ei3kxnd):
+
+```toml
+[orchestrator]
+base_url = "https://api.refuseless.com/v1"
+model = "deepseek-v4.1-flash-abliterated"
+api_key_env = "REFUSELESS_API_KEY"
+```
+
+Sign up at [refuseless.com/?ref=1ei3kxnd](https://refuseless.com/?ref=1ei3kxnd)
+to get the key. Any OpenAI-compatible chat endpoint works if you would rather
+run your own, so `base_url` is the only thing to change. The key is read from
+`$REFUSELESS_API_KEY`, falling back to `env_file` (`~/.hermes/.env` by
+default). killchain never writes it to disk.
+
+**3. An OpenRouter key.** The risk scorer is Jev (`typesafe/jev-1.14`) through
+the OpenRouter Decisions API:
+
+```toml
+[risk]
+api_key_env = "OPENROUTER_API_KEY"
+```
+
+That is what decides which spans carry refusal risk, so a run without it has
+only the regex gate left. `--no-detector` says so loudly when you mean that.
+
+| Check | Fix if missing |
+| --- | --- |
+| `claude` or `codex` on PATH | install the CLI, or set `[frontier] backend` to the one you have |
+| `$REFUSELESS_API_KEY` | key from [refuseless](https://refuseless.com/?ref=1ei3kxnd), or point `base_url` at another endpoint |
+| `$OPENROUTER_API_KEY` | key from openrouter.ai |
+
+All three are bypassed only in the `fake` combination the tests use
+(`--frontier fake --orch-url fake --detector-url fake`).
+
+`doctor` exits non-zero on a missing frontier binary, a missing orchestrator
+key, or a bad config. A missing OpenRouter key does not, on its own, make the
+exit code non-zero even though that check fails. Read the check lines, not just
+the exit code: a run with no risk scorer still works, but the refusal-routing
+is reduced to the regex gate, which is not what you installed this for.
+
+
 ## First run
 
 Nothing is written into the directory you installed to. On first use killchain
@@ -129,6 +190,32 @@ regulated domains, another vendor's model or SDK internals, hedge wording, and
 scope an agent will negotiate away. Add a dimension when you find a refusal
 that none of them predict.
 
+### Checking that the routing works
+
+Changing a dimension's wording is a claim about how a model behaves, and it is
+easy to make that claim badly. `killchain/eval.py` measures it against labelled
+corpora instead of trusting the wording:
+
+```bash
+python -m killchain.eval --rows 100 --out results.json
+```
+
+That scores `mlabonne/harmful_behaviors` and `mlabonne/harmless_alpaca` through
+the same detector path a run uses, then prints how far apart the two score
+distributions sit, the accuracy at each routing threshold, and the bar needed
+for 90% recall. Each span costs a fraction of a cent. `tests/test_routing_accuracy.py`
+asserts floors on the same measurements and is skipped without a key, so it is
+run by hand rather than in CI.
+
+**[EVALS.md](EVALS.md)** holds a recorded run with the resolved model build and
+corpus hashes, so the numbers can be checked rather than taken on faith.
+
+One caveat worth keeping in mind: those corpora are harmful *user prompts*, while
+routing predicts whether a frontier agent would hedge on a *sentence of a
+specification*. Separation there proves the detector reads harmful text as
+harmful. It does not prove it flags a benign-looking spec sentence that happens
+to mention a licence.
+
 ## Trust boundaries
 
 To the **orchestrator (refuseless)**: the full prompt, the contract, and the

{killchain-0.1.2 → killchain-0.1.4}/README.md

@@ -1,11 +1,11 @@
 # killchain: refusal-first split build
 
-An abliterated orchestrator model (refuseless by default, settings mirrored
-from `~/.hermes/config.yaml`) is the front door. It is the only model that
-sees the full prompt. It splits the job into a **public half** (built by a
-frontier coding agent: Claude Code or Codex, which may fan out to its own
-subagents) and a **private half** (implemented by the orchestrator itself),
-joined by a contract.
+An abliterated orchestrator model ([refuseless](https://refuseless.com/?ref=1ei3kxnd)
+by default, settings mirrored from `~/.hermes/config.yaml`) is the front door.
+It is the only model that sees the full prompt. It splits the job into a
+**public half** (built by a frontier coding agent: Claude Code or Codex, which
+may fan out to its own subagents) and a **private half** (implemented by the
+orchestrator itself), joined by a contract.
 
 The partition rule is **refusal risk, nothing else**. Claude and Codex are good
 and fast, but they hedge, caveat, simplify, or refuse when a task looks
@@ -48,6 +48,67 @@ killchain --doctor
 private directory mode, the denylist, and does one live round-trip to each
 endpoint. It exits non-zero only if something fatal to a run is wrong.
 
+## What you need
+
+Three things, and a run fails without any of them. killchain has no Python
+dependencies, but it is a front end for two models and one coding agent.
+
+**1. A frontier coding agent on your PATH.** One is enough, not both: `claude`
+or `codex`. This is what builds the public half. Set which one in `[frontier]`:
+
+```toml
+[frontier]
+backend = "claude"        # or "codex"
+```
+
+The default is `claude`. If the binary is missing, `doctor` reports the
+configured backend and no executable on PATH. Install the CLI you already have
+a subscription for and point the config at it.
+
+**2. An abliterated orchestrator.** This model reads the full prompt and
+implements the private half, which is the entire point of the tool, so a
+censored model here defeats it. Default is [refuseless](https://refuseless.com/?ref=1ei3kxnd):
+
+```toml
+[orchestrator]
+base_url = "https://api.refuseless.com/v1"
+model = "deepseek-v4.1-flash-abliterated"
+api_key_env = "REFUSELESS_API_KEY"
+```
+
+Sign up at [refuseless.com/?ref=1ei3kxnd](https://refuseless.com/?ref=1ei3kxnd)
+to get the key. Any OpenAI-compatible chat endpoint works if you would rather
+run your own, so `base_url` is the only thing to change. The key is read from
+`$REFUSELESS_API_KEY`, falling back to `env_file` (`~/.hermes/.env` by
+default). killchain never writes it to disk.
+
+**3. An OpenRouter key.** The risk scorer is Jev (`typesafe/jev-1.14`) through
+the OpenRouter Decisions API:
+
+```toml
+[risk]
+api_key_env = "OPENROUTER_API_KEY"
+```
+
+That is what decides which spans carry refusal risk, so a run without it has
+only the regex gate left. `--no-detector` says so loudly when you mean that.
+
+| Check | Fix if missing |
+| --- | --- |
+| `claude` or `codex` on PATH | install the CLI, or set `[frontier] backend` to the one you have |
+| `$REFUSELESS_API_KEY` | key from [refuseless](https://refuseless.com/?ref=1ei3kxnd), or point `base_url` at another endpoint |
+| `$OPENROUTER_API_KEY` | key from openrouter.ai |
+
+All three are bypassed only in the `fake` combination the tests use
+(`--frontier fake --orch-url fake --detector-url fake`).
+
+`doctor` exits non-zero on a missing frontier binary, a missing orchestrator
+key, or a bad config. A missing OpenRouter key does not, on its own, make the
+exit code non-zero even though that check fails. Read the check lines, not just
+the exit code: a run with no risk scorer still works, but the refusal-routing
+is reduced to the regex gate, which is not what you installed this for.
+
+
 ## First run
 
 Nothing is written into the directory you installed to. On first use killchain
@@ -105,6 +166,32 @@ regulated domains, another vendor's model or SDK internals, hedge wording, and
 scope an agent will negotiate away. Add a dimension when you find a refusal
 that none of them predict.
 
+### Checking that the routing works
+
+Changing a dimension's wording is a claim about how a model behaves, and it is
+easy to make that claim badly. `killchain/eval.py` measures it against labelled
+corpora instead of trusting the wording:
+
+```bash
+python -m killchain.eval --rows 100 --out results.json
+```
+
+That scores `mlabonne/harmful_behaviors` and `mlabonne/harmless_alpaca` through
+the same detector path a run uses, then prints how far apart the two score
+distributions sit, the accuracy at each routing threshold, and the bar needed
+for 90% recall. Each span costs a fraction of a cent. `tests/test_routing_accuracy.py`
+asserts floors on the same measurements and is skipped without a key, so it is
+run by hand rather than in CI.
+
+**[EVALS.md](EVALS.md)** holds a recorded run with the resolved model build and
+corpus hashes, so the numbers can be checked rather than taken on faith.
+
+One caveat worth keeping in mind: those corpora are harmful *user prompts*, while
+routing predicts whether a frontier agent would hedge on a *sentence of a
+specification*. Separation there proves the detector reads harmful text as
+harmful. It does not prove it flags a benign-looking spec sentence that happens
+to mention a licence.
+
 ## Trust boundaries
 
 To the **orchestrator (refuseless)**: the full prompt, the contract, and the

{killchain-0.1.2 → killchain-0.1.4}/killchain/doctor.py

@@ -47,7 +47,12 @@ def run(cfg: dict) -> int:
         cp = paths.resolve_config(None)
         checks.append(Check("config", OK, str(cp)))
     except SystemExit as e:
-        checks.append(Check("config", FAIL, str(e), "run: killchain doctor --write-config"))
+        # Config normally auto-seeds on first run, so this only fires when
+        # --config or $KILLCHAIN_CONFIG points somewhere wrong. Creating a file
+        # at the user's path silently would be the wrong fix; say how to reset.
+        checks.append(Check("config", FAIL, str(e),
+                            "unset $KILLCHAIN_CONFIG (or fix --config), or run "
+                            "`killchain --config <path>` once that file exists"))
 
     # --- frontier agent
     backend = cfg.get("frontier", {}).get("backend", "claude")

killchain-0.1.4/killchain/eval.py

@@ -0,0 +1,253 @@
+"""Does the routing actually work? Measure it instead of asserting it.
+
+The unit tests pin the arithmetic in routing.py against scores we invented.
+That proves `classify()` compares correctly and proves nothing about whether
+Jev returns a sensible probability for a real span. This module answers the
+second question: take a corpus with known labels, score it through the same
+code path a run uses, and report whether the two score distributions separate.
+
+    uv run python -m killchain.eval --rows 200
+
+Two things this deliberately does NOT claim:
+
+  * The mlabonne corpora are harmful *user prompts*; routing predicts whether
+    a frontier agent would hedge on a *sentence of a specification*. The
+    distributions overlap where a spec describes sensitive work, and nowhere
+    else. A high score here means Jev reads harmful text as harmful. It does
+    not mean Jev flags a spec sentence that mentions the GPL.
+  * A single threshold number is a fiction. What matters is where the two
+    distributions sit relative to it, which is why this prints the full
+    separability picture and not just accuracy.
+
+Everything scored is sent to OpenRouter and TypeSafe, so --rows is capped and
+reported.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import random
+import statistics
+import sys
+import urllib.request
+from dataclasses import dataclass, field
+
+from killchain import detector, routing
+
+HF_ROWS = "https://datasets-server.huggingface.co/rows"
+CORPORA = {
+    "harmful": "mlabonne/harmful_behaviors",
+    "harmless": "mlabonne/harmless_alpaca",
+}
+
+
+@dataclass
+class Result:
+    label: str                  # "harmful" | "harmless"
+    text: str
+    scores: dict[str, float] = field(default_factory=dict)
+    flagged: bool = False
+
+    @property
+    def overall(self) -> float:
+        return self.scores.get("overall", 0.0)
+
+    @property
+    def peak(self) -> float:
+        """Highest dimension score. This is what classify() acts on."""
+        return max(self.scores.values(), default=0.0)
+
+    @property
+    def peak_dim(self) -> str:
+        if not self.scores:
+            return "-"
+        return max(self.scores.items(), key=lambda kv: kv[1])[0]
+
+
+def fetch(dataset: str, want: int, offset: int = 0, seed: int = 0) -> list[str]:
+    """Pull `want` texts from a HF dataset via the rows server.
+
+    The rows endpoint pages at 100, so this walks pages and samples rather than
+    pulling a whole corpus: harmless_alpaca is 31k rows and we do not need them.
+    """
+    out: list[str] = []
+    page = 0
+    while len(out) < want:
+        url = f"{HF_ROWS}?dataset={dataset}&config=default&split=train&offset={page * 100}&length=100"
+        try:
+            with urllib.request.urlopen(url, timeout=60) as r:
+                rows = json.loads(r.read()).get("rows", [])
+        except (urllib.error.URLError, TimeoutError) as e:
+            raise SystemExit(f"cannot reach the HF rows server for {dataset}: {e}") from e
+        if not rows:
+            break
+        out.extend(str(row["row"]["text"]).strip() for row in rows)
+        page += 1
+        if offset and page > 200:      # hard stop; nobody wants 31k API calls
+            break
+    if len(out) > want:
+        random.Random(seed).shuffle(out)
+        out = out[:want]
+    return out
+
+
+def run_scores(texts: list[str], label: str, cfg: dict) -> list[Result]:
+    """Score each text as a single span, through the real detector path."""
+    spans = [(f"{label}[{i}]", t) for i, t in enumerate(texts)]
+    verdicts = detector.score_spans(cfg, spans, risky_terms=[])
+    return [Result(label, v.text, v.scores, v.flagged) for v in verdicts]
+
+
+def resolved_model(cfg: dict) -> str:
+    """The exact build the alias pointed at today, e.g. typesafe/jev-1.14-20261014.
+
+    An alias like `typesafe/jev-1.14` can be repointed at a new build without
+    warning, so a report that names only the alias cannot be reproduced.
+    """
+    try:
+        ans = detector._call(cfg, "hello", {"q": {"type": "noul",
+                                                 "instructions": "Is this risky?",
+                                                 "criteria": {"true": "yes", "false": "no"}}})
+    except Exception:
+        return "unknown"
+    return ans.get("model", "unknown") if isinstance(ans, dict) else "unknown"
+
+
+def corpora_provenance(by_label: dict[str, list[Result]]) -> dict:
+    """Fingerprint the exact rows scored, so a later run over a changed
+    dataset is visibly different rather than quietly comparable."""
+    out = {}
+    for label, results in by_label.items():
+        blob = "\n".join(r.text for r in results).encode("utf-8")
+        out[label] = {
+            "dataset": CORPORA.get(label, "?"),
+            "n": len(results),
+            "sha256_of_scored_texts": hashlib.sha256(blob).hexdigest(),
+        }
+    return out
+
+
+def summarise(by_label: dict[str, list[Result]]) -> dict:
+    stats: dict = {}
+    for label, results in by_label.items():
+        peaks = sorted(r.peak for r in results)
+        # `overall` is only asked of an assembled public view, never of a bare
+        # span, so it does not appear in these scores. Reporting its mean would
+        # print a confident 0.0 that means "never asked", not "no risk".
+        stats[label] = {
+            "n": len(results),
+            "peak_mean": round(statistics.fmean(peaks), 3) if peaks else 0.0,
+            "peak_median": round(statistics.median(peaks), 3) if peaks else 0.0,
+            "peak_p10": round(peaks[len(peaks) // 10], 3) if peaks else 0.0,
+            "peak_p90": round(peaks[len(peaks) * 9 // 10], 3) if peaks else 0.0,
+        }
+    for label, results in by_label.items():
+        counts: dict[str, int] = {}
+        for r in results:
+            counts[r.peak_dim] = counts.get(r.peak_dim, 0) + 1
+        stats[label]["peak_dim"] = dict(sorted(counts.items(), key=lambda kv: -kv[1]))
+    return stats
+
+
+def separability(by_label: dict[str, list[Result]]) -> dict:
+    """The number that matters: how much the two distributions overlap.
+
+    Reports accuracy at each routing threshold, plus the bar that would be
+    needed to reach 90% recall on the harmful set, and the false-positive rate
+    that bar costs on the harmless set.
+    """
+    harmful = [r.peak for r in by_label.get("harmful", [])]
+    harmless = [r.peak for r in by_label.get("harmless", [])]
+    if not harmful or not harmless:
+        return {}
+    out: dict = {"sweep": []}
+    for bar in (0.35, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70):
+        recall = sum(1 for p in harmful if p >= bar) / len(harmful)
+        fpr = sum(1 for p in harmless if p >= bar) / len(harmless)
+        out["sweep"].append({
+            "bar": bar,
+            "recall": round(recall, 3),
+            "false_positive": round(fpr, 3),
+        })
+    # The bar needed for 90% recall, and what it costs on the benign side.
+    need = min((p for p in harmful if sum(1 for q in harmful if q >= p) / len(harmful) <= 0.90),
+               default=min(harmful))
+    out["bar_for_90_recall"] = round(need, 3)
+    out["fpr_at_90_recall"] = round(sum(1 for p in harmless if p >= need) / len(harmless), 3)
+
+    # A crude AUC by rank, to compare runs. 1.0 = perfect separation, 0.5 = coin flip.
+    wins = sum((1.0 if h > b else 0.5 if h == b else 0.0) for h in harmful for b in harmless)
+    out["auc"] = round(wins / (len(harmful) * len(harmless)), 3)
+    return out
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(
+        prog="python -m killchain.eval",
+        description="Measure refusal-risk routing against labelled corpora.")
+    ap.add_argument("--rows", type=int, default=100,
+                    help="texts per class (default 100; each costs one API call)")
+    ap.add_argument("--out", default="", help="write the full result JSON here")
+    ap.add_argument("--base-url", default="https://openrouter.ai/api/alpha/decisions")
+    ap.add_argument("--model", default="typesafe/jev-1.14")
+    a = ap.parse_args(argv)
+
+    if a.rows > 500:
+        raise SystemExit("--rows above 500 is a lot of API calls; raise the cap in code if you mean it")
+
+    cfg = {"base_url": a.base_url, "model": a.model, "api_key_env": "OPENROUTER_API_KEY",
+           "timeout": 90}
+    resolved_name = resolved_model(cfg)
+
+    by_label: dict[str, list[Result]] = {}
+    for label, dataset in CORPORA.items():
+        print(f"[eval] fetching {a.rows} from {dataset} ...", file=sys.stderr, flush=True)
+        texts = fetch(dataset, a.rows)
+        print(f"[eval] scoring {len(texts)} {label} spans ...", file=sys.stderr, flush=True)
+        by_label[label] = run_scores(texts, label, cfg)
+
+    stats = summarise(by_label)
+    sep = separability(by_label)
+
+    print(f"\n{'=' * 72}\nREFUSAL-RISK ROUTING: {a.model}\n{'=' * 72}")
+    print(f"resolved build: {resolved_name}")
+    for label in ("harmful", "harmless"):
+        s = stats.get(label)
+        if s:
+            print(f"\n{label}  n={s['n']}")
+            print(f"  peak score   mean {s['peak_mean']}  median {s['peak_median']}  "
+                  f"p10 {s['peak_p10']}  p90 {s['peak_p90']}")
+            top = list(s["peak_dim"].items())[:3]
+            print("  most often the peak: " + ", ".join(f"{d} x{c}" for d, c in top))
+
+    if sep:
+        print("\nseparability (peak dimension vs the threshold classify() uses)")
+        print(f"  AUC {sep['auc']}   (1.0 perfect, 0.5 coin flip)")
+        print(f"\n  {'bar':>5} {'recall':>7} {'false pos':>10}")
+        for row in sep["sweep"]:
+            print(f"  {row['bar']:>5.2f} {row['recall']:>7.2f} {row['false_positive']:>10.2f}")
+        print(f"\n  90% recall needs a bar of {sep['bar_for_90_recall']}, "
+              f"which flags {sep['fpr_at_90_recall']:.0%} of benign text")
+
+    if a.out:
+        payload = {
+            "model": a.model,
+            "resolved_model": resolved_name,
+            "rows_per_class": a.rows,
+            "corpora": corpora_provenance(by_label),
+            "stats": stats,
+            "separability": sep,
+            "thresholds": dict(routing.THRESHOLDS),
+            "results": [{"label": r.label, "text": r.text, "scores": r.scores,
+                         "peak": r.peak, "flagged": r.flagged}
+                        for rs in by_label.values() for r in rs],
+        }
+        with open(a.out, "w", encoding="utf-8") as f:
+            json.dump(payload, f, indent=2)
+        print(f"\n[eval] full results -> {a.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{killchain-0.1.2 → killchain-0.1.4}/killchain/harness.py

@@ -39,7 +39,7 @@ from killchain.contract import Contract, ContractError
 
 
 def log(msg: str) -> None:
-    print(f"[harness] {msg}", flush=True)
+    print(f"[killchain] {msg}", flush=True)
 
 
 #: Everything that means "your setup can't run this", as opposed to a harness
@@ -54,8 +54,8 @@ FAILURES = (
 
 def fail(msg: str) -> None:
     """Print a setup failure and exit 1 without a traceback."""
-    print(f"\n[harness] cannot continue: {msg}", file=sys.stderr, flush=True)
-    print("[harness] run `killchain --doctor` to check config, keys and the frontier agent.",
+    print(f"\n[killchain] cannot continue: {msg}", file=sys.stderr, flush=True)
+    print("[killchain] run `killchain --doctor` to check config, keys and the frontier agent.",
           file=sys.stderr, flush=True)
     raise SystemExit(1)
 

{killchain-0.1.2 → killchain-0.1.4}/killchain/routing.py

@@ -172,7 +172,12 @@ THRESHOLDS: dict[str, float] = {
     "regulated": 0.55,
     "proprietary_model": 0.55,
     "hedge_framing": 0.60,
-    "scale_or_scope": 0.60,
+    # Measured, not guessed. This one scores benign text at a mean of 0.50 and
+    # is the top dimension on most of it, so at 0.60 it was flagging ordinary
+    # work: 26 false positives per 100 benign spans, against 10 real catches.
+    # 0.80 cuts that to 3 for a 6-point recall loss. Raising it further is
+    # worse than disabling it (see EVALS.md).
+    "scale_or_scope": 0.80,
     "overall": 0.50,
     "default": 0.55,
 }

{killchain-0.1.2 → killchain-0.1.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "killchain"
-version = "0.1.2"
+version = "0.1.4"
 description = "Split a build so a frontier coding agent never sees the risky half."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -42,7 +42,14 @@ YouTube = "https://youtube.com/@drjoshcsimmons"
 packages = ["killchain"]
 
 [tool.hatch.build.targets.sdist]
-include = ["killchain", "README.md", "pyproject.toml"]
+# Keep the release artefact to the package. An explicit list rather than a
+# deny-list: a git worktree checked out inside the repo is a second full copy
+# of the package, and hatchling's VCS sweep ships it (the sdist went to 41
+# entries against a clean 22 when one existed). CI builds from a clean clone so
+# this only bites local releases, which is the worst way to find out.
+include = ["killchain", "README.md", "EVALS.md", "pyproject.toml"]
+# The landing page is its own build (site/, Astro, deployed separately).
+exclude = ["/site"]
 
 [tool.ruff]
 target-version = "py311"