firegoat 0.7.0 → 0.8.0

package/README.md

@@ -2,7 +2,7 @@
 
 Local-first AX evals: how well coding agents use your SDK, CLI, API and docs.
 
-Status: early (v0.7). See `plan.md`.
+Status: early (v0.8). See `plan.md`.
 
 ## Use
 
@@ -61,11 +61,11 @@ A probe checks the sandbox before every run; if it does not hold, the run is not
 
 Off macOS there is no sandbox yet: run inside a throwaway container and pass `--host`.
 
-`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl`, runs the test's `checks` in the agent's folder, and writes `run.json` per run plus `summary.json` (prompt, goal, rubric, check results, paths). Nothing is uploaded.
+`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl`, runs the test's `checks` in the agent's folder, and writes `run.json` per run plus `summary.json` (prompt, end_goal, check results, paths). Nothing is uploaded.
 
-### Grading: your agent judges the goal
+### Grading: your agent judges the end goal
 
-firegoat runs only the script checks; they set the exit code. The `goal` and an optional `rubric:` list are handed over, not graded: whoever drives firegoat (usually your coding agent) judges them with its own process from `npx firegoat eval --json` / `run.json`, the transcript and the workdir. firegoat records no verdict beyond the checks.
+firegoat runs only the script checks; they set the exit code. Each test's `end_goal` is handed over, not graded: whoever drives firegoat (usually your coding agent) judges it with its own process from `npx firegoat eval --json` / `run.json`, the transcript and the workdir. firegoat records no verdict beyond the checks.
 
 Every command has `--help` written for an agent; start with `npx firegoat --help`. `firegoat.yaml` is looked up in the current directory, then parents up to the git root.
 

package/dist/cli.js

@@ -6947,7 +6947,7 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.7.0",
+  version: "0.8.0",
   repository: {
     type: "git",
     url: "git+https://github.com/team2027/firegoat-cli.git"
@@ -7221,10 +7221,10 @@ sandboxed folder; firegoat records the transcript and runs your script checks.
 
 Typical flow (for an agent driving firegoat on behalf of a human):
   1. npx firegoat init --yes          write firegoat.yaml (read: npx firegoat init --help)
-  2. edit firegoat.yaml               real user problems as prompts, a goal, checks
+  2. edit firegoat.yaml               real user problems as prompts, an end_goal, checks
   3. ask the human to run once:       npx firegoat claude login   (opens a browser)
   4. npx firegoat eval --json         run the tests; JSON summary on stdout
-  5. judge each run's goal + rubric yourself, from its run.json, transcript and
+  5. judge each run's end_goal yourself, from its run.json, transcript and
      workdir: firegoat only runs the script checks
 
 Commands:
@@ -7267,15 +7267,15 @@ firegoat.yaml reference:
   tests:
     - name: quickstart        letters, digits, '.', '_', '-'; unique
       prompt: ...             what a user would ask the agent; one concrete outcome
-      goal: ...               one sentence that is true when the problem is solved
-      rubric: [...]           optional criteria for whoever grades the run (you)
+      end_goal: ...           what is true when the problem is solved; not graded by
+                              firegoat, handed to you (goal: is a deprecated alias)
       checks:                 optional shell commands, run in the workdir after the
         - run: ...            agent; exit 0 = pass; they set eval's exit code
       timeout: 600            optional, seconds
 
   The agent's folder has no repo files. Checks get $FIREGOAT_WORKDIR,
   $FIREGOAT_SESSION (stream-json transcript), $FIREGOAT_OUT and the vars.
-  goal and rubric are not run by firegoat: they are handed to the caller to judge.
+  end_goal is not graded by firegoat: it is handed to the caller to judge.
 
 Example:
   setup: npm pack --pack-destination "$FIREGOAT_OUT"
@@ -7285,12 +7285,13 @@ Example:
       prompt: >-
         Using the acme SDK (already installed from the build under test), send a test
         event and print the event id. Credentials are in ACME_API_KEY. Keep it minimal.
-      goal: A script sends one event and prints its id.
-      rubric:
-        - uses the SDK's client, not raw HTTP
-        - does not install acme from the registry
+      end_goal: >-
+        A script sends one event with the SDK's client (not raw HTTP) and prints its
+        id, without installing acme from the registry.
       checks:
         - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_SESSION"
+        - run: |
+            ! grep -Eq 'npm (install|i) [^"]*acme' "$FIREGOAT_SESSION"
 
 Next: npx firegoat claude login (a human, once), then npx firegoat eval.
 
@@ -7316,13 +7317,13 @@ command to run.
 Output: .firegoat/runs/<runId>/
   summary.json            every run, plus totals and a grading hint
   <test>/<n>/run.json     one run: test, n, status and reason (from the checks),
-                          prompt, goal, rubric, checks [{run, code}], and paths:
+                          prompt, end_goal, checks [{run, code}], and paths:
                           dir, workdir, session, stderr, checksLog
   <test>/<n>/session.jsonl  the agent's transcript, stream-json (one event per line)
   setup.txt, <test>/<n>/workdir_setup.txt, checks.txt   logs
 
-Grading: firegoat runs only the script checks; they set the exit code. The goal and
-rubric are yours to judge, with your own process. session.jsonl is stream-json: the
+Grading: firegoat runs only the script checks; they set the exit code. end_goal is
+yours to judge, with your own process. session.jsonl is stream-json: the
 last line has type "result" with the agent's final message; tool calls are
 "tool_use" items in "assistant" messages; tool output is in "user" messages.
 
@@ -7464,7 +7465,7 @@ function posInt(v, path, fallback) {
     fail(path, "must be a positive integer");
   return v;
 }
-function parseConfig(text) {
+function parseConfig(text, warn = console.error) {
   let raw;
   try {
     raw = $parse(text);
@@ -7504,21 +7505,25 @@ function parseConfig(text) {
     if (!Array.isArray(checksRaw))
       fail(`${p}.checks`, "must be a list");
     const checks = [];
-    const rubric = rubricOf(t.rubric, `${p}.rubric`);
     checksRaw.forEach((c, j) => {
       const cp = `${p}.checks[${j}]`;
       if (!isRecord(c))
         fail(cp, "must be a mapping");
       if ("judge" in c)
-        rubric.push(str(c.judge, `${cp}.judge`));
-      else
-        checks.push({ run: str(c.run, `${cp}.run`) });
+        fail(`${cp}.judge`, "plain-language checks were removed; put the criteria in end_goal (your agent judges it) or write a run: check");
+      checks.push({ run: str(c.run, `${cp}.run`) });
     });
+    if (t.rubric !== undefined)
+      fail(`${p}.rubric`, "was removed in firegoat 0.8.0; put the criteria in end_goal");
+    if (t.goal !== undefined && t.end_goal !== undefined)
+      fail(`${p}.goal`, "use end_goal only (goal is its deprecated name)");
+    if (t.goal !== undefined)
+      warn(`${CONFIG_FILE}: ${p}.goal is deprecated; rename it to end_goal`);
+    const endGoalKey = t.goal !== undefined ? "goal" : "end_goal";
     return {
       name,
       prompt: str(t.prompt, `${p}.prompt`),
-      goal: str(t.goal, `${p}.goal`),
-      rubric,
+      end_goal: str(t[endGoalKey], `${p}.${endGoalKey}`),
       checks,
       timeout: posInt(t.timeout, `${p}.timeout`, DEFAULT_TIMEOUT)
     };
@@ -7533,15 +7538,6 @@ function parseConfig(text) {
   };
 }
 var NAME = /^[A-Za-z0-9._-]+$/;
-function rubricOf(raw, path) {
-  if (raw === undefined)
-    return [];
-  if (typeof raw === "string")
-    return [str(raw, path)];
-  if (!Array.isArray(raw))
-    fail(path, "must be a string or a list of strings");
-  return raw.map((r, i) => str(r, `${path}[${i}]`));
-}
 function commands(raw, path) {
   if (raw === undefined)
     return [];
@@ -7579,11 +7575,11 @@ function render(text, vars) {
     return v;
   });
 }
-function loadConfig(dir) {
+function loadConfig(dir, warn) {
   const file = `${dir}/${CONFIG_FILE}`;
   if (!existsSync3(file))
     throw new ConfigError(`${CONFIG_FILE} not found in ${dir}. Run: npx firegoat init`);
-  return parseConfig(readFileSync2(file, "utf8"));
+  return parseConfig(readFileSync2(file, "utf8"), warn);
 }
 function findConfigDir(cwd) {
   let dir = cwd;
@@ -7945,7 +7941,7 @@ function resolveBin(bin) {
 
 // src/eval.ts
 var CHECK_TIMEOUT_MS = 120000;
-var GRADING_HINT = "firegoat ran only the script checks. The goal and rubric are for the caller to judge, from each run's session (stream-json transcript) and workdir.";
+var GRADING_HINT = "firegoat ran only the script checks. Each run's end_goal is for the caller to judge, from its session (stream-json transcript) and workdir.";
 function blankRun(test, n, dir, prompt) {
   return {
     test: test.name,
@@ -7953,8 +7949,7 @@ function blankRun(test, n, dir, prompt) {
     status: "error",
     reason: "",
     prompt,
-    goal: test.goal,
-    rubric: test.rubric,
+    end_goal: test.end_goal,
     checks: [],
     dir,
     workdir: null,
@@ -8106,7 +8101,7 @@ async function runOne(opts) {
   if (!outcome.ok)
     return done("error", `not graded: ${outcome.reason}`);
   if (test.checks.length === 0)
-    return done("pass", "no script checks; the goal is for the grader");
+    return done("pass", "no script checks; end_goal is for the caller to judge");
   const log = createWriteStream3(summary.checksLog);
   const env = {
     ...process.env,
@@ -8156,7 +8151,7 @@ async function evalCommand(args, cwd) {
   const say = opts.json ? console.error : console.log;
   const warn = console.error;
   const root = findConfigDir(cwd);
-  const config = loadConfig(root);
+  const config = loadConfig(root, warn);
   let tests = config.tests;
   if (opts.names.length > 0) {
     const known = new Set(tests.map((t) => t.name));
@@ -8292,7 +8287,7 @@ async function evalCommand(args, cwd) {
   say(`
 ${pass} passed, ${fail} failed, ${error} not graded (${results.length} runs)`);
   say(`runs: ${runDir}`);
-  say(`goal and rubric are not graded by firegoat; per run: ${join6(runDir, "<test>", "<n>", "run.json")}`);
+  say(`end_goal is not graded by firegoat; per run: ${join6(runDir, "<test>", "<n>", "run.json")}`);
   if (opts.json)
     console.log(JSON.stringify(doc, null, 2));
   return error > 0 ? EXIT.notGraded : fail > 0 ? EXIT.fail : EXIT.pass;
@@ -8422,7 +8417,7 @@ function templates(p) {
     out.push({
       name: "cli-first-run",
       prompt: `I just heard about the \`${p.bin}\` command-line tool from ${p.name} and want to see it actually do something useful. Use it for one basic real task and save what it prints to cli-output.txt. Keep it minimal. Use only ${docs} as a guide.`,
-      goal: `cli-output.txt exists and holds real output from running \`${p.bin}\`, not an install log or an error.`,
+      end_goal: `cli-output.txt exists and holds real output from running \`${p.bin}\`, not an install log or an error.`,
       checks: [
         "test -s cli-output.txt",
         "! grep -qiE 'command not found|not recognized|ENOENT' cli-output.txt"
@@ -8433,7 +8428,7 @@ function templates(p) {
     out.push({
       name: "quickstart",
       prompt: `I want to use ${p.name} in a new ${LANG[p.ecosystem]} project. Get the smallest working example going: a program that uses ${p.name} to do one real thing, then run it and save what it prints to output.txt. Keep it minimal. Use only ${docs} as a guide.`,
-      goal: `A program in the workdir uses ${p.name}, runs without errors, and its output is saved in output.txt.`,
+      end_goal: `A program in the workdir uses ${p.name}, runs without errors, and its output is saved in output.txt.`,
       checks: [
         "test -s output.txt",
         `grep -rqsF ${GREP_EXCLUDES} -- ${sq(importName(p))} .`
@@ -8444,7 +8439,7 @@ function templates(p) {
     out.push({
       name: "quickstart",
       prompt: `I want to try ${p.name}. Follow its quickstart in a new project, get the smallest example working, then run it and save what it prints to output.txt. Keep it minimal. Use only ${docs} as a guide.`,
-      goal: `The ${p.name} quickstart example runs without errors and its output is saved in output.txt.`,
+      end_goal: `The ${p.name} quickstart example runs without errors and its output is saved in output.txt.`,
       checks: ["test -s output.txt"]
     });
   }
@@ -8493,7 +8488,7 @@ function renderYaml(p, tests) {
     "tests:"
   ];
   for (const t of tests) {
-    lines.push(`  - name: ${t.name}  # letters, digits, '.', '_' and '-'`, "    # What a user would ask the agent. Edit this.", "    prompt: >-", block(t.prompt, "      "), "    # One sentence that is true when the user's problem is solved.", "    goal: >-", block(t.goal, "      "), "    # Shell commands run in the agent's workdir afterwards; exit 0 = pass.", "    checks:", ...t.checks.map((c) => `      - run: ${JSON.stringify(c)}`), "    # Seconds before the agent run is stopped.", "    timeout: 600");
+    lines.push(`  - name: ${t.name}  # letters, digits, '.', '_' and '-'`, "    # What a user would ask the agent. Edit this.", "    prompt: >-", block(t.prompt, "      "), "    # One sentence that is true when the user's problem is solved. Not graded by", "    # firegoat: it goes to your agent (eval --json, run.json) to judge.", "    end_goal: >-", block(t.end_goal, "      "), "    # Shell commands run in the agent's workdir afterwards; exit 0 = pass.", "    checks:", ...t.checks.map((c) => `      - run: ${JSON.stringify(c)}`), "    # Seconds before the agent run is stopped.", "    timeout: 600");
   }
   return `${lines.join(`
 `)}

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "firegoat",
-	"version": "0.7.0",
+	"version": "0.8.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/team2027/firegoat-cli.git"