claude-dev-env 1.95.0 → 2.0.0

package/skills/grok-spawn/reference/worker-briefs.md

@@ -0,0 +1,152 @@
+# Worker brief templates
+
+Prompt-part bodies for headless grok workers. Copy a template into a part file,
+fill every bracket, and list the part paths on the worker's `prompt_parts`.
+
+The batch launcher prepends a tool-profile header before the joined parts:
+
+- `readonly` — no write, edit, or shell
+- `build` — full tools; never commit, push, or call `gh`
+
+Put the role brief first, the task body next, the report contract last.
+
+---
+
+## Read-only investigation brief
+
+Use with `tool_profile: "readonly"`. For repo-only scans set `is_repo_only: true`
+so the launcher also passes `--disable-web-search`.
+
+```markdown
+# Role: read-only investigation
+
+You investigate only. You do not write files, edit files, or run shell commands.
+
+## Scope
+
+- Working directory: [absolute path]
+- Paths or symbols in scope: [list]
+- Question to answer: [one clear question]
+- Out of scope: [list]
+
+## Method
+
+1. Read the named paths and their direct callers or callees as needed.
+2. Cite every claim with `file:line`.
+3. Prefer measured facts (names, signatures, call sites) over guesses.
+4. When a fact is unproven, label it `unverified`.
+
+## Hard stops
+
+- No Write, Edit, or Bash.
+- No commits, pushes, or `gh`.
+- No expanding scope past the listed paths without noting it as an open question.
+
+## Done when
+
+You can answer the question with file:line evidence, or you can name the exact
+gap that blocks an answer.
+```
+
+---
+
+## Build brief
+
+Use with `tool_profile: "build"`. The worker may edit and run tests. The lead
+session owns every git and GitHub step.
+
+```markdown
+# Role: build worker
+
+You edit code and run tests for one closed task. You never commit, push, or call
+`gh`. The lead session stages, verifies, commits, pushes, and posts.
+
+## Scope
+
+- Working directory: [absolute path]
+- Task: [one sentence]
+- Files you may touch: [list]
+- Files you must not touch: [list]
+- Acceptance lines (each must map to evidence in the report):
+  1. [line]
+  2. [line]
+
+## Method
+
+1. Read the in-scope files before editing.
+2. Write or update tests that pin the acceptance lines (red first when you add
+   behavior).
+3. Make the smallest edit that satisfies the acceptance lines.
+4. Run the named test commands and capture pass/fail output.
+5. Stop with a stage-ready tree and a full report. Do not commit.
+
+## Hard stops
+
+- Never `git commit`, `git push`, or `gh`.
+- Never force-push, rewrite shared history, or change git config.
+- Never expand into files outside the allow list without an open question.
+- Never mark acceptance done without command output or a file:line proof.
+
+## Done when
+
+Every acceptance line has evidence, tests you own are green (or failures are
+listed with command output), and the report lists every changed file.
+```
+
+---
+
+## Report contract
+
+Append this part (or paste its sections into the task body) for every worker —
+readonly and build alike. The lead session uses it to verify and to relay open
+questions to an advisor when needed.
+
+```markdown
+# Report contract
+
+End your turn with exactly these sections, in this order. Use plain markdown.
+No tool calls after the report.
+
+## Changed files
+
+- List every path you wrote or edited, one per line.
+- Write `none` when the role is read-only or no edit landed.
+
+## Red-green evidence
+
+- For each test you added or changed: the failing command/output before the
+  fix (red), then the passing command/output after (green).
+- Write `n/a — investigation only` for read-only workers.
+
+## Acceptance mapping
+
+For each acceptance line from the brief:
+
+- **Acceptance:** [quote the line]
+- **Status:** met | not met | blocked
+- **Evidence:** [command output summary or `file:line` proof]
+
+## Test results
+
+- Commands run (full command lines)
+- Exit codes
+- Short pass/fail summary (paste key lines; do not dump huge logs)
+
+## Open questions
+
+- Questions for the lead session or advisor (empty list if none)
+- Unverified claims that need a second look
+- Scope gaps that blocked a full answer
+```
+
+---
+
+## Part assembly tips
+
+1. Keep task-specific detail in its own part file so the brief templates stay
+   reusable.
+2. Absolute paths only in `prompt_parts` — the launcher reads them as given.
+3. One worker, one closed scope. Split large work into more workers rather than
+   one long brief.
+4. The lead session fills bracketed fields before launch; workers never see the
+   skill folder unless you copy text into their part files.

package/skills/grokify/SKILL.md

@@ -12,7 +12,9 @@ One paste-ready handoff turns this session's work into a plan a Grok Build sessi
 
 ## Gotchas
 
-- Grok Build cannot spawn Claude subagents. The advisor is an out-of-process `claude -p` session — never write Agent-tool, `session-advisor`, or SendMessage instructions into the handoff.
+- Grok Build can use `spawn_subagent`, `--agent` / agent definitions, and can read skills under the user's Claude config paths. Skill evals measure these when `GROK_CAPABILITY_EVALS=1` (see `evals/README.md`).
+- The **grokify handoff's Claude-tier advisor** is an out-of-process `claude -p` bind/resume. Never write Claude Agent-tool, `session-advisor`, or SendMessage protocol into the handoff for that advisor path — product design for a Claude-model advisor, not a claim that Grok lacks agents.
+- Grok does not expose a Claude/GSD Workflow tool. When that tool is required, workflow orchestration stays with Claude.
 - A `--resume` after a usage-limit failover to another binary fails, because a session store belongs to the binary that minted it. The handoff must tell Grok to treat that failure as starting over: re-send the charter plus a compact recap, capture the new `session_id`.
 - Conversation-relative phrases ("as discussed", "the plan above", "the earlier choice") are dead text to Grok — every statement stands on its own.
 - Copy findings' measured numbers and `file:line` citations into the handoff exactly, and label each figure measured, bounded, or unverified.
@@ -51,8 +53,14 @@ The user types `/grokify`, alone or with guidance.
 |---|---|
 | `SKILL.md` | Trigger, process, fixed advisor structure. |
 | `templates/handoff-template.md` | Section-by-section skeleton of the handoff prompt. |
+| `capability-claims.test.mjs` | Offline static guards on capability wording. |
+| `evals/README.md` | How to run opt-in live capability evals. |
+| `evals/run-capability-evals.mjs` | Live E1–E5 runner (manual / opt-in only). |
+| `evals/parse-payload.test.mjs` | Offline unit tests for the eval output parser, run under `npm test`. |
 
 ## Folder map
 
 - `SKILL.md` — the whole workflow.
 - `templates/` — the handoff skeleton.
+- `evals/` — the opt-in live runner plus its offline parser unit tests.
+- `capability-claims.test.mjs` — offline claim guards for `npm test`.

package/skills/grokify/capability-claims.test.mjs

@@ -0,0 +1,28 @@
+import { test } from 'node:test';
+import { strict as assert } from 'node:assert';
+import { readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { dirname, join } from 'node:path';
+
+const skillDirectory = dirname(fileURLToPath(import.meta.url));
+const skillSource = readFileSync(join(skillDirectory, 'SKILL.md'), 'utf8');
+const handoffSource = readFileSync(
+  join(skillDirectory, 'templates', 'handoff-template.md'),
+  'utf8',
+);
+
+test('SKILL.md does not claim Grok cannot spawn Claude subagents', () => {
+  assert.equal(
+    skillSource.includes('cannot spawn Claude subagents'),
+    false,
+    'SKILL.md must not contain the false substring "cannot spawn Claude subagents"',
+  );
+});
+
+test('SKILL.md documents the out-of-process claude -p advisor path', () => {
+  assert.match(skillSource, /claude -p/);
+});
+
+test('handoff template keeps the claude -p advisor bind path', () => {
+  assert.match(handoffSource, /claude -p/);
+});

package/skills/grokify/evals/README.md

@@ -0,0 +1,72 @@
+# Grok capability evals (opt-in only)
+
+Live measurements of Grok Build capabilities used by the `grokify` skill wording.
+
+**Manual / opt-in only.** These evals never gate default `npm test`, pr-check, or CI.
+
+## Requirements
+
+- `grok` on `PATH` and authenticated (`grok models` exits 0). The runner calls `grok` by default; set `GROK_BIN` to point it at another binary.
+- Network access for the Grok API
+- A writable platform temp directory (`$env:TEMP` on Windows, `$TMPDIR` / `/tmp` elsewhere)
+- For E3, an agent `grok` can load under `--agent`. The runner passes the agent named by `GROK_CAPABILITY_EVAL_AGENT` (default `Explore`); set that variable to an agent your `grok` resolves when the default does not exist for you.
+
+## Run
+
+From the package root (`packages/claude-dev-env`):
+
+```powershell
+# Windows PowerShell
+$env:GROK_CAPABILITY_EVALS = "1"
+node skills/grokify/evals/run-capability-evals.mjs
+```
+
+```bash
+# POSIX
+GROK_CAPABILITY_EVALS=1 node skills/grokify/evals/run-capability-evals.mjs
+```
+
+From this skill folder (`packages/claude-dev-env/skills/grokify`):
+
+```powershell
+# Windows PowerShell
+$env:GROK_CAPABILITY_EVALS = "1"
+node evals/run-capability-evals.mjs
+```
+
+```bash
+# POSIX
+GROK_CAPABILITY_EVALS=1 node evals/run-capability-evals.mjs
+```
+
+Or pass `--run` without the env var (paths match the cwd above):
+
+```powershell
+# from package root
+node skills/grokify/evals/run-capability-evals.mjs --run
+# from this skill folder
+node evals/run-capability-evals.mjs --run
+```
+
+Without `GROK_CAPABILITY_EVALS=1` or `--run`, the script prints how to opt in and exits 0 (no live calls).
+
+## What it measures
+
+| Eval | Assertion |
+|------|-----------|
+| E1 | `can_spawn_subagent_tool === true` and tool list includes `spawn_subagent` |
+| E2 | `spawn_succeeded === true` (child reports `SPAWN_OK`) |
+| E3 | `skill_read_ok === true` and `agent_definition_loaded === true` (skill file read under `--agent`) |
+| E4 | Probe write under the eval cwd succeeds (soft: hooks log may show `global/settings` + `pre_tool_use`) |
+| E5 | `has_workflow_tool === false` and `result === "no_tool"` (both fields agree on absence) |
+
+Each live `grok` call uses a unique `--leader-socket` under a fresh temp directory so concurrent runs do not share a socket.
+
+## Offline static guard
+
+`../capability-claims.test.mjs` runs under package `npm test` (no live `grok`):
+
+- `SKILL.md` must not contain `cannot spawn Claude subagents`
+- `SKILL.md` and the handoff template must still mention `claude -p` for the advisor path
+
+`parse-payload.test.mjs` also runs under package `npm test` (no live `grok`): it unit-tests the runner's output parser (`tryParseJsonObject`, `extractResultText`, `parsePayload`, `isWorkflowToolAbsent`).

package/skills/grokify/evals/parse-payload.test.mjs

@@ -0,0 +1,171 @@
+import { test } from 'node:test';
+import { strict as assert } from 'node:assert';
+import {
+  tryParseJsonObject,
+  extractResultText,
+  parsePayload,
+  isWorkflowToolAbsent,
+} from './run-capability-evals.mjs';
+
+const E1_CLAIM_PAYLOAD = {
+  can_spawn_subagent_tool: true,
+  tool_names: ['spawn_subagent'],
+  agents_dir_exists: true,
+  skills_dir_exists: true,
+};
+
+test('tryParseJsonObject parses a pure JSON object string', () => {
+  const parsed = tryParseJsonObject(JSON.stringify(E1_CLAIM_PAYLOAD));
+  assert.deepEqual(parsed, E1_CLAIM_PAYLOAD);
+});
+
+test('tryParseJsonObject strips leading prose before the first brace', () => {
+  const withProse = `Here is the inventory:\n${JSON.stringify(E1_CLAIM_PAYLOAD)}`;
+  const parsed = tryParseJsonObject(withProse);
+  assert.deepEqual(parsed, E1_CLAIM_PAYLOAD);
+});
+
+test('tryParseJsonObject extracts the object when trailing prose carries braces', () => {
+  const withTrailingBraces = `${JSON.stringify(E1_CLAIM_PAYLOAD)}\nNote: see {run-directory} for logs.`;
+  const parsed = tryParseJsonObject(withTrailingBraces);
+  assert.deepEqual(parsed, E1_CLAIM_PAYLOAD);
+});
+
+test('tryParseJsonObject ignores a stray brace pair in leading prose', () => {
+  const withLeadingBraces = `Here is the result (see {} for details):\n${JSON.stringify(E1_CLAIM_PAYLOAD)}`;
+  const parsed = tryParseJsonObject(withLeadingBraces);
+  assert.deepEqual(parsed, E1_CLAIM_PAYLOAD);
+});
+
+test('tryParseJsonObject returns null for empty input', () => {
+  assert.equal(tryParseJsonObject(''), null);
+  assert.equal(tryParseJsonObject('   '), null);
+});
+
+test('extractResultText reads Grok top-level text field', () => {
+  const nestedJson = JSON.stringify(E1_CLAIM_PAYLOAD);
+  const envelope = JSON.stringify({ text: nestedJson, model: 'grok' });
+  assert.equal(extractResultText(envelope), nestedJson);
+});
+
+test('extractResultText prefers Claude result over Grok text', () => {
+  const envelope = JSON.stringify({
+    result: '{"from":"result"}',
+    text: '{"from":"text"}',
+  });
+  assert.equal(extractResultText(envelope), '{"from":"result"}');
+});
+
+test('extractResultText prefers Grok text over a conversational message', () => {
+  const envelope = JSON.stringify({
+    message: 'working',
+    text: '{"from":"text"}',
+  });
+  assert.equal(extractResultText(envelope), '{"from":"text"}');
+});
+
+test('extractResultText reads Claude message field', () => {
+  const envelope = JSON.stringify({ message: '{"from":"message"}' });
+  assert.equal(extractResultText(envelope), '{"from":"message"}');
+});
+
+test('extractResultText reads Claude stream result events', () => {
+  const envelope = JSON.stringify([
+    { type: 'assistant', content: 'thinking' },
+    {
+      type: 'result',
+      result: JSON.stringify(E1_CLAIM_PAYLOAD),
+    },
+  ]);
+  assert.equal(extractResultText(envelope), JSON.stringify(E1_CLAIM_PAYLOAD));
+});
+
+test('extractResultText prefers the last result event in a stream array', () => {
+  const intermediateClaim = { can_spawn_subagent_tool: false, tool_names: [] };
+  const envelope = JSON.stringify([
+    { type: 'assistant', content: 'thinking' },
+    { type: 'result', result: JSON.stringify(intermediateClaim) },
+    { type: 'assistant', content: 'retry' },
+    { type: 'result', result: JSON.stringify(E1_CLAIM_PAYLOAD) },
+  ]);
+  assert.equal(extractResultText(envelope), JSON.stringify(E1_CLAIM_PAYLOAD));
+});
+
+test('parsePayload extracts nested claim JSON from Grok text envelope', () => {
+  const envelope = JSON.stringify({
+    text: JSON.stringify(E1_CLAIM_PAYLOAD),
+  });
+  const payload = parsePayload(envelope);
+  assert.equal(payload.can_spawn_subagent_tool, true);
+  assert.deepEqual(payload.tool_names, ['spawn_subagent']);
+});
+
+test('parsePayload extracts nested claim JSON when Grok text has leading prose', () => {
+  const envelope = JSON.stringify({
+    text: `Capability inventory complete.\n${JSON.stringify(E1_CLAIM_PAYLOAD)}`,
+  });
+  const payload = parsePayload(envelope);
+  assert.equal(payload.can_spawn_subagent_tool, true);
+  assert.equal(payload.agents_dir_exists, true);
+});
+
+test('parsePayload still reads Claude result envelopes', () => {
+  const envelope = JSON.stringify({
+    result: JSON.stringify(E1_CLAIM_PAYLOAD),
+  });
+  const payload = parsePayload(envelope);
+  assert.equal(payload.can_spawn_subagent_tool, true);
+});
+
+test('parsePayload still reads Claude message envelopes', () => {
+  const envelope = JSON.stringify({
+    message: JSON.stringify(E1_CLAIM_PAYLOAD),
+  });
+  const payload = parsePayload(envelope);
+  assert.equal(payload.can_spawn_subagent_tool, true);
+});
+
+test('parsePayload still reads Claude stream array envelopes', () => {
+  const envelope = JSON.stringify([
+    { type: 'message', content: 'working' },
+    { type: 'result', result: JSON.stringify(E1_CLAIM_PAYLOAD) },
+  ]);
+  const payload = parsePayload(envelope);
+  assert.equal(payload.can_spawn_subagent_tool, true);
+  assert.deepEqual(payload.tool_names, ['spawn_subagent']);
+});
+
+test('isWorkflowToolAbsent accepts consistent absence', () => {
+  assert.equal(
+    isWorkflowToolAbsent({ has_workflow_tool: false, result: 'no_tool' }),
+    true,
+  );
+});
+
+test('isWorkflowToolAbsent rejects contradictory true and no_tool', () => {
+  assert.equal(
+    isWorkflowToolAbsent({ has_workflow_tool: true, result: 'no_tool' }),
+    false,
+  );
+});
+
+test('isWorkflowToolAbsent rejects contradictory false and has_tool', () => {
+  assert.equal(
+    isWorkflowToolAbsent({ has_workflow_tool: false, result: 'has_tool' }),
+    false,
+  );
+});
+
+test('isWorkflowToolAbsent rejects consistent presence', () => {
+  assert.equal(
+    isWorkflowToolAbsent({ has_workflow_tool: true, result: 'has_tool' }),
+    false,
+  );
+});
+
+test('isWorkflowToolAbsent rejects partial or missing fields', () => {
+  assert.equal(isWorkflowToolAbsent({ has_workflow_tool: false }), false);
+  assert.equal(isWorkflowToolAbsent({ result: 'no_tool' }), false);
+  assert.equal(isWorkflowToolAbsent(null), false);
+  assert.equal(isWorkflowToolAbsent({}), false);
+});