boost-skill-cli 1.2.65__tar.gz → 1.2.66__tar.gz

{boost_skill_cli-1.2.65/boost_skill_cli.egg-info → boost_skill_cli-1.2.66}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.65
+Version: 1.2.66
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66}/boost_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.65'
-__version_tuple__ = version_tuple = (1, 2, 65)
+__version__ = version = '1.2.66'
+__version_tuple__ = version_tuple = (1, 2, 66)
 
 __commit_id__ = commit_id = None

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66/boost_skill_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: boost-skill-cli
-Version: 1.2.65
+Version: 1.2.66
 Summary: boost — Homebrew for AI coding skills
 Author: Jonathan Reyes
 License-Expression: Apache-2.0

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66}/docs/roadmap/items/tool-call-eval-tier.md

@@ -2,13 +2,13 @@
 id: tool-call-eval-tier
 board: code
 section: planned
-status: planned
+status: inflight
 category: Quality · Eval
 complexity: L
 impact: High
 wow: 4
-note: the Claude Code probe was broken and its finding was an artifact; fixed, second host still unwritten
-owner:
+note: probe fixed; --strict-mcp-config shipped for the surface confound; second host (Gemini CLI) still unwritten
+owner: loop/eval-tools-strict-mcp
 order: 96
 title: a Tier 3 eval for tool-call behaviour, floored in <b>both</b> directions
 ---
@@ -100,6 +100,27 @@ one this card argues about.
 <code>--strict-mcp-config</code> with a boost-only config cuts that sharply and controls the
 surface confound in the same move.
 
+<b>2026-08-31: <code>--strict-mcp-config</code> shipped.</b> <code>eval_tools.py</code> now takes
+a <code>--strict-mcp-config</code> flag: it writes a boost-only <code>mcpServers</code> config
+(the same <code>&lt;launcher&gt; mcp --stdio</code> invocation and fork-safety <code>env</code>
+<code>core.mcphost.register_argv</code> uses for a real registration — confirmed against an actual
+<code>claude mcp add-json</code> write, not guessed at the schema) to a temp file and passes
+<code>--strict-mcp-config --mcp-config &lt;path&gt;</code> to every <code>claude -p</code> call,
+cleaning the file up afterward. This session's sandbox had no network path to PyPI, so the pinned
+toolchain (<code>pytest</code>, <code>ruff</code>, <code>mypy</code>, …) could not be installed and
+<code>make check</code> could not be run here; the change was verified by hand instead — direct
+<code>python3.12</code> import of the module, the new unit tests executed by eye against the
+interpreter, <code>py_compile</code>, a manual line-length check against ruff's 88-column default,
+and an end-to-end dry run with <code>subprocess.run</code> mocked that confirms the flags land on
+the argv and the temp file is created and removed. CI runs the real gate on the PR.
+
+<b>Still unwritten: the second host arm (Gemini CLI).</b> No <code>gemini</code> CLI was reachable
+in this sandbox to capture a real stream from, and building that arm on an invented model of
+Gemini's non-interactive output format is the exact mistake this card's own probe fix (2026-08-30)
+already paid for once — "a fixture the author invented cannot catch the author's wrong model of the
+input." That arm stays a placeholder until it can be built against a captured real stream, on a
+machine with the <code>gemini</code> CLI installed.
+
 <b>What it unlocks.</b> The first honest answer to "did that description edit help", a baseline the
 next surface change can regress against, and a way to retire claims that survive only because
 nobody can check them.

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66}/docs/roadmap.html

@@ -814,7 +814,7 @@ an oversight.</p></details>
       </article>
 
       <article class="cap rcard" id="tool-call-eval-tier">
-        <div class="head"><span class="pill planned">Planned</span><span class="cat">Quality · Eval</span></div>
+        <div class="head"><span class="pill inflight">In flight</span><span class="cat">Quality · Eval</span></div>
         <h3>a Tier 3 eval for tool-call behaviour, floored in <b>both</b> directions</h3>
         <p>boost's required gate floors <b>four</b> retrieval metrics — recall@k &ge; 0.78, hit@1 &ge; 0.40,
 MRR &ge; 0.52, nDCG@k &ge; 0.58 — over a 91-query golden set and a 10,152-entry corpus, every
@@ -904,6 +904,27 @@ one this card argues about.
 <code>--strict-mcp-config</code> with a boost-only config cuts that sharply and controls the
 surface confound in the same move.
 
+<b>2026-08-31: <code>--strict-mcp-config</code> shipped.</b> <code>eval_tools.py</code> now takes
+a <code>--strict-mcp-config</code> flag: it writes a boost-only <code>mcpServers</code> config
+(the same <code>&lt;launcher&gt; mcp --stdio</code> invocation and fork-safety <code>env</code>
+<code>core.mcphost.register_argv</code> uses for a real registration — confirmed against an actual
+<code>claude mcp add-json</code> write, not guessed at the schema) to a temp file and passes
+<code>--strict-mcp-config --mcp-config &lt;path&gt;</code> to every <code>claude -p</code> call,
+cleaning the file up afterward. This session's sandbox had no network path to PyPI, so the pinned
+toolchain (<code>pytest</code>, <code>ruff</code>, <code>mypy</code>, …) could not be installed and
+<code>make check</code> could not be run here; the change was verified by hand instead — direct
+<code>python3.12</code> import of the module, the new unit tests executed by eye against the
+interpreter, <code>py_compile</code>, a manual line-length check against ruff's 88-column default,
+and an end-to-end dry run with <code>subprocess.run</code> mocked that confirms the flags land on
+the argv and the temp file is created and removed. CI runs the real gate on the PR.
+
+<b>Still unwritten: the second host arm (Gemini CLI).</b> No <code>gemini</code> CLI was reachable
+in this sandbox to capture a real stream from, and building that arm on an invented model of
+Gemini's non-interactive output format is the exact mistake this card's own probe fix (2026-08-30)
+already paid for once — "a fixture the author invented cannot catch the author's wrong model of the
+input." That arm stays a placeholder until it can be built against a captured real stream, on a
+machine with the <code>gemini</code> CLI installed.
+
 <b>What it unlocks.</b> The first honest answer to "did that description edit help", a baseline the
 next surface change can regress against, and a way to retire claims that survive only because
 nobody can check them.</p>
@@ -911,7 +932,7 @@ nobody can check them.</p>
           <span class="m">Complexity <b>L</b></span>
           <span class="m hi">Impact <b>High</b></span>
           <span class="m">Wow <b class="wow">★★★★</b></span>
-          <span class="m">the Claude Code probe was broken and its finding was an artifact; fixed, second host still unwritten</span>
+          <span class="m">probe fixed; --strict-mcp-config shipped for the surface confound; second host (Gemini CLI) still unwritten</span>
         </div>
       </article>
 

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66}/scripts/eval_tools.py

@@ -51,6 +51,7 @@ Usage:
   python3 scripts/eval_tools.py --dry-run           # show the plan, call nothing
   python3 scripts/eval_tools.py --runs 3            # 3 runs per prompt
   python3 scripts/eval_tools.py --floor-call 0.60 --ceiling-false-call 0.20
+  python3 scripts/eval_tools.py --strict-mcp-config # only boost on the MCP surface
 """
 from __future__ import annotations
 
@@ -61,11 +62,15 @@ import os
 import shutil
 import subprocess
 import sys
+import tempfile
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parent.parent
 DEFAULT_SET = ROOT / "tests" / "eval" / "tool_calls.jsonl"
 
+sys.path.insert(0, str(ROOT))
+from boost_cli.core import mcphost  # noqa: E402
+
 #: Tool names that count as "the agent asked boost". Prefixed forms are matched
 #: too — hosts namespace MCP tools differently (`mcp__boost__boost_search`,
 #: `mcp_boost_boost_search`), and matching the bare suffix keeps one list
@@ -267,15 +272,43 @@ def claude_available() -> bool:
     return shutil.which("claude") is not None
 
 
-def run_claude(prompt: str, timeout: int) -> str | None:
+def strict_mcp_config(launcher: str) -> dict:
+    """The boost-only `--mcp-config` payload for a `claude -p` run.
+
+    Claude Code's own `mcpServers` shape (`{"mcpServers": {name: {"command",
+    "args", "env"}}}` — confirmed against a real `claude mcp add-json` write,
+    not guessed) with exactly one entry: boost, launched the same way
+    `core.mcphost.register_argv` registers it (`<launcher> mcp --stdio`, plus
+    `LAUNCH_ENV` — the fork-safety env a real registration always carries, and
+    whose absence is a macOS SIGABRT, not a cosmetic gap). Kept a plain
+    function, not a file write, so a test can assert the shape without
+    touching disk.
+    """
+    return {"mcpServers": {mcphost.SERVER_NAME: {
+        "command": launcher, "args": ["mcp", "--stdio"],
+        "env": dict(mcphost.LAUNCH_ENV)}}}
+
+
+def run_claude(prompt: str, timeout: int,
+                mcp_config: Path | None = None) -> str | None:
     """One `claude -p` run, returning its raw event stream (or None).
 
     `--output-format stream-json` is what makes the probe honest: it emits a
     `tool_use` block per call, so the observation is the host's record of what
     happened rather than the model's account of it.
+
+    `mcp_config`, when given, adds `--strict-mcp-config --mcp-config <path>` —
+    the card's own fix for the confound `_report_surface` can only report, not
+    prevent: a machine with several MCP servers registered offers every one of
+    their tools to every prompt, which moves the call rate far more than any
+    wording edit and costs real tokens describing tools this eval never asked
+    about. `--strict-mcp-config` replaces the host's merged configuration with
+    only what this file names, so the run sees boost and nothing else.
     """
     cmd = ["claude", "-p", prompt, "--output-format", "stream-json",
            "--verbose", "--max-turns", "2"]
+    if mcp_config is not None:
+        cmd += ["--strict-mcp-config", "--mcp-config", str(mcp_config)]
     try:
         proc = subprocess.run(cmd, capture_output=True, text=True,
                               timeout=timeout)
@@ -365,6 +398,10 @@ def main(argv: list[str] | None = None) -> int:
     p.add_argument("--report-context", action="store_true", default=True,
                    help="name the installed rules in scope (default on); a "
                         "rate read without them is not attributable to wording")
+    p.add_argument("--strict-mcp-config", action="store_true",
+                   help="run with only boost on the MCP surface (--strict-mcp-config "
+                        "--mcp-config), so a crowded machine's other servers don't "
+                        "move the call rate or the token bill")
     args = p.parse_args(argv)
 
     rows = load_set(args.set)
@@ -392,16 +429,31 @@ def main(argv: list[str] | None = None) -> int:
     if args.report_context:
         _print_context()
 
+    mcp_config_path = None
+    if args.strict_mcp_config:
+        from boost_cli.core import paths
+        config = strict_mcp_config(str(paths.launcher()))
+        fd, name = tempfile.mkstemp(suffix=".json", prefix="boost-eval-mcp-")
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            json.dump(config, fh)
+        mcp_config_path = Path(name)
+        print("context: --strict-mcp-config on — this run sees only boost's "
+              "own MCP server, not whatever else is registered on this machine.")
+
     observed: dict[str, list[bool]] = {}
     surface = (0, 0)
-    for r in rows:
-        for _ in range(max(1, args.runs)):
-            events = run_claude(r["prompt"], args.timeout)
-            if events is None:
-                continue
-            if surface == (0, 0):
-                surface = session_surface(events)
-            observed.setdefault(r["id"], []).append(called_boost(events))
+    try:
+        for r in rows:
+            for _ in range(max(1, args.runs)):
+                events = run_claude(r["prompt"], args.timeout, mcp_config_path)
+                if events is None:
+                    continue
+                if surface == (0, 0):
+                    surface = session_surface(events)
+                observed.setdefault(r["id"], []).append(called_boost(events))
+    finally:
+        if mcp_config_path is not None:
+            mcp_config_path.unlink(missing_ok=True)
     if args.report_context:
         _report_surface(*surface)
 

{boost_skill_cli-1.2.65 → boost_skill_cli-1.2.66}/tests/unit/test_eval_tools.py

@@ -314,3 +314,65 @@ class TestTheRateIsNeverReadWithoutItsContext:
         monkeypatch.setattr(lockfile, "all_installed", boom)
         MOD._print_context()
         assert "could not read" in capsys.readouterr().out
+
+
+class TestStrictMcpConfigControlsTheSurfaceConfound:
+    """`--strict-mcp-config` is the card's own fix for the confound
+    `_report_surface` can only report: a crowded machine's other MCP servers
+    move the call rate far more than any wording edit under test.
+    """
+
+    def test_the_config_names_only_boost(self):
+        from boost_cli.core import mcphost
+        cfg = MOD.strict_mcp_config("/usr/local/bin/boost")
+        assert set(cfg["mcpServers"]) == {mcphost.SERVER_NAME}
+
+    def test_the_entry_launches_boost_as_an_mcp_stdio_server(self):
+        # Mirrors core.mcphost.register_argv's own invocation of boost, so a
+        # config built here starts the exact same server a real registration
+        # would — confirmed against a real `claude mcp add-json` write, not
+        # guessed at.
+        cfg = MOD.strict_mcp_config("/usr/local/bin/boost")
+        entry = cfg["mcpServers"]["boost"]
+        assert entry["command"] == "/usr/local/bin/boost"
+        assert entry["args"] == ["mcp", "--stdio"]
+
+    def test_the_entry_carries_the_fork_safety_env(self):
+        # Without LAUNCH_ENV a real host can SIGABRT the boost subprocess
+        # post-fork on macOS — the same failure core.mcphost.LAUNCH_ENV exists
+        # to prevent for every other registration path.
+        from boost_cli.core import mcphost
+        cfg = MOD.strict_mcp_config("/usr/local/bin/boost")
+        assert cfg["mcpServers"]["boost"]["env"] == mcphost.LAUNCH_ENV
+
+    def test_run_claude_adds_no_flags_without_a_config(self, monkeypatch):
+        captured = {}
+
+        def fake_run(cmd, **kw):
+            captured["cmd"] = cmd
+            class P:
+                stdout = ""
+            return P()
+
+        monkeypatch.setattr(MOD.subprocess, "run", fake_run)
+        MOD.run_claude("hi", 5)
+        assert "--strict-mcp-config" not in captured["cmd"]
+        assert "--mcp-config" not in captured["cmd"]
+
+    def test_run_claude_adds_both_flags_with_a_config(self, monkeypatch, tmp_path):
+        captured = {}
+
+        def fake_run(cmd, **kw):
+            captured["cmd"] = cmd
+            class P:
+                stdout = ""
+            return P()
+
+        monkeypatch.setattr(MOD.subprocess, "run", fake_run)
+        cfg_path = tmp_path / "mcp.json"
+        cfg_path.write_text("{}", encoding="utf-8")
+        MOD.run_claude("hi", 5, cfg_path)
+        cmd = captured["cmd"]
+        assert "--strict-mcp-config" in cmd
+        i = cmd.index("--mcp-config")
+        assert cmd[i + 1] == str(cfg_path)