@arjunkhera/atlas 0.3.40 → 0.3.42

package/skills/fresh-agent-test/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: fresh-agent-test
+description: >
+  The fresh-agent test that ends onboarding a repo (target design flow 1, step 9): a fresh
+  agent does three tasks from the repo's files alone, and the judge crew picks them before
+  the run and grades them after. Trigger when onboarding reaches its last step, when the
+  owner asks to "run the fresh-agent test" or "judge the onboarding", or when a stage done
+  line names the test.
+---
+
+# The fresh-agent test
+
+A fresh agent does three tasks from a repo's files alone. If it passes all
+three, the files are enough for an agent with no memory. You, the lead,
+wrote or changed those files, so you never pick a task and never grade one.
+The `atlas:judge` crew does both, each time fresh. `atlas judge` does the
+exact work: the seal, the brief, the run and the check.
+
+Design: `docs/design-docs/fresh-agent-judge/` in the Atlas repo.
+
+## Before you start
+
+1. Name the item that the test proves, such as the stage or the onboarding
+   item. Call it the test item.
+2. Make a run folder outside the repo, such as a folder in the session's
+   scratch space.
+3. Read the commit to test: `git rev-parse HEAD` on the branch or the main
+   branch that holds the onboarding. Every run uses this commit.
+4. A run needs a sign-in that an empty home folder can use. A cloud
+   session has one. On a Mac, an empty home folder has no login. Set
+   `ANTHROPIC_API_KEY` for the run, or run the test from a cloud session.
+
+## Steps
+
+1. **Pick.** Send `atlas:judge` with the job "pick". Give it the repo
+   folder, the commit and the run folder. Add the onboarding pull request
+   when there is one. Add the task files of earlier tests of this repo.
+   Give it nothing else: no hint, no list of what you changed.
+2. **Seal.** Run `atlas judge seal <run folder>/tasks.md`. Fix nothing
+   yourself. On a shape error, send a new pick run with the error lines.
+3. **Record the seal.** Before any run, call `item_comment` on the test
+   item. Put in the seal line, the three runs it picked, and the whole of
+   `tasks.md`. When the file is too long for one comment, add a second
+   comment with the rest.
+4. **Run.** For each of the three picked runs, run
+   `atlas judge run <run folder>/tasks.md <id> --out <run folder> --repo <repo folder>`.
+   Run them one at a time. The command sends the brief itself, so never
+   send a task to an agent by another way.
+5. **Grade.** Send a new `atlas:judge` with the job "grade" and the run
+   folder. Never send the pick run's crew again.
+6. **Check.** Run
+   `atlas judge check --tasks <run folder>/tasks.md --verdict <run folder>/verdict.md --seal <hash from step 3> --runs <run folder>`.
+   Copy the hash from the tracker comment, not from the file.
+7. **Record the result.** Call `item_comment` on the test item. Put in the
+   output of the check and the whole of `verdict.md`. Link both comments
+   from the onboarding page or the pull request body.
+
+`check` exits 0 when all three pass, 1 when one fails, and 2 when the
+verdict does not count. On exit 2, send a new grade run with the error
+lines. Never edit `verdict.md` yourself.
+
+## When a task fails
+
+1. A failed task is a gap in the files. List it on the onboarding page.
+   Partial readiness still goes ahead (target design, flow 1).
+2. Fix the files. Then run a new test from step 1. The new pick gets the
+   earlier task files, so it picks new tasks. Never rerun the same tasks
+   after a fix: that teaches to the test.
+
+## When a run breaks
+
+1. A run that did not finish exits 1 and keeps its folder. You may run that
+   one task once more, into a new `--out` folder. Say so in the result
+   comment.
+2. When the test stops for good, call `item_comment` on the test item with
+   "abandoned" and the reason. Delete the run folder.
+
+## Never
+
+1. Never pick, grade or edit a task, a report or a verdict.
+2. Never give a crew or a run a hint about the files.
+3. Never count a test whose check exits 2.

package/skills/lead/SKILL.md

@@ -56,6 +56,7 @@ say that when you hand over a page.
 | Review a design or a diff | The three reviewer crews, as sdlc-task says |
 | Prove a pull request against its definition of done | The `atlas:verifier` crew |
 | Render a page for the owner | The `atlas:artifact-renderer` crew |
+| Run the fresh-agent test at the end of onboarding: pick, run and grade three tasks | The `atlas:fresh-agent-test` skill, which sends the `atlas:judge` crew |
 | Onboard an area to tests, review scenario words, write or convert a test, sort a red test, plant faults | The `atlas:tests` skill |
 
 A crew reads and reports. It never holds the Atlas tools.
@@ -104,6 +105,7 @@ set. A test keeps the two lists equal.
 | `atlas design` | `folder` to learn where designs live; `check` before you show a design; `build` to make its page |
 | `atlas view` | `build --product <id>` makes the project view page of one product, when the owner asks for it or after an Atlas release changes it. Publish it with only the grant the command prints |
 | `atlas upkeep` | `check` before you ask for a merge: the docs index, dead doc links, and the tracker item a pull request names. The daily upkeep run calls it on the main branch |
+| `atlas judge` | In the fresh-agent test only, as the `atlas:fresh-agent-test` skill says. `seal` after the pick, `brief` to read a sealed ask, `run` for each sealed task, `check` after the grade |
 
 A person merges every file that `atlas tooling` writes.
 A person merges every file that `atlas tests write` writes.

package/skills/tests/SKILL.md

@@ -358,6 +358,31 @@ not counted, and the test passes.
 After the product is ready, a lost connection, a server exit or a time-out is
 `fail`. `blocked` is for a guard refusal, a start-up fault and the first-call check.
 
+### A test in Python
+
+Use the Python library when the runner of the area is Python. Its header,
+`test/atlas/contract.py`, holds a worked example.
+
+1. Name the file `test/scenarios/<name>_test.py`. List a Python runner in
+   `runner` of `tests.yaml`. `runner` may be a list, one runner for each
+   language.
+2. Before the import, set `sys.dont_write_bytecode = True`. Put
+   `test/atlas` on `sys.path`, and import `scenario` from `contract`.
+   Run Python with `-B`: the check refuses bytecode in the test folder.
+   Never run it with `-O`, which removes every assert.
+3. Decorate the body: `@scenario("../../scenarios/<name>.md", adapter=adapter)`.
+4. Write each check as a `with` block that holds an assert statement:
+   `with run.check("<scenario>/e1#<fingerprint>"):`. Use `run.gate` for a gate.
+   To keep a short proof, set `proof.value` in `with ... as proof:`.
+5. Use `run.not_exercised` and `run.wait_for`, in snake case.
+6. Give the driver of each way in through the adapter (`define_adapter`).
+   Build its child environment with `child_env`.
+
+The library reads the scenario through node, so the runner needs node and
+python3. It does not yet start an environment, give run secrets, repeat a
+run, ask a judge or ship drivers. Each of these ends `blocked`. Write such a
+scenario with `contract.mjs`.
+
 ### A model in the test
 
 A model makes a test vary and costs money. Three parts of the kit handle it.

package/tests/bridge.mjs

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+// The bridge for a contract library in another language (design section 8.1).
+// A library in Python, say, must read a scenario exactly as the link check
+// reads it: the same parser, the same fingerprints, the same tests.yaml check
+// and the same guard comparison. So it does not parse the words itself. It
+// runs this file and reads one JSON object:
+//
+//   node bridge.mjs load <scenario.md>
+//   -> { root, file, environment, evidenceDir, guardRefusal, tests, envSpec, shippedDrivers, wayGuardParts,
+//        scenario: { id, title, file, sha256, method, fields, through, ways, repeat,
+//                    assertions: [{ id, fullId, kind, gate, ways, fp, names, line }],
+//                    tables: { T1: { name, kind, header, rows, fingerprint, cases } } } }
+//
+// `cases` is every case of a combination table (every combination, or a set
+// that holds every pair), so the other language need not make them again.
+// A fault in the scenario or in tests.yaml prints { error } and exits 1.
+// The environment and the folders come from the same variables as
+// contract.mjs: ATLAS_ENV, CI, ATLAS_EVIDENCE_DIR and ATLAS_GUARDS_FROM.
+import { existsSync, readFileSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { readTestsYaml, checkTestsYaml, compareGuardParts, resolveEnvironments, GUARD_PARTS } from './tests-yaml.mjs';
+import { SHIPPED_DRIVERS } from './drivers/index.mjs';
+import { readScenario, addFingerprints, combinations, pairs } from './scenario.mjs';
+
+function findRoot(file) {
+  let dir = dirname(file);
+  for (;;) {
+    if (existsSync(join(dir, 'atlas', 'tests.yaml'))) return dir;
+    const up = dirname(dir);
+    if (up === dir) throw new Error(`no atlas/tests.yaml above ${file}`);
+    dir = up;
+  }
+}
+
+export function loadForLibrary(path, env = process.env) {
+  const file = resolve(String(path));
+  const root = findRoot(file);
+  const yaml = readTestsYaml(join(root, 'atlas', 'tests.yaml'));
+  const findings = checkTestsYaml(yaml);
+  if (findings.length) throw new Error(`atlas/tests.yaml: ${findings.map((f) => `line ${f.line}: ${f.message}`).join('; ')}`);
+  const sc = readScenario(file);
+  if (sc.problems.length) throw new Error(`${path}: ${sc.problems.map((p) => `line ${p.line}: ${p.message}`).join('; ')}`);
+  const tests = yaml.value;
+  addFingerprints(sc, tests);
+  // The same rule as contract.mjs: a branch whose guard parts differ from the main branch copy is refused.
+  let guardRefusal = null;
+  if (env.ATLAS_GUARDS_FROM) {
+    try {
+      const changes = compareGuardParts(readFileSync(env.ATLAS_GUARDS_FROM, 'utf8'), yaml.value);
+      if (changes.length) guardRefusal = `the guard parts of atlas/tests.yaml differ from the main branch copy (${changes.map((c) => c.part).join(', ')}); the run is refused. A change to a guard part comes to the owner`;
+    } catch (error) {
+      guardRefusal = `ATLAS_GUARDS_FROM names a file that cannot be read as tests.yaml (${error.code ?? error.message}); the run is refused`;
+    }
+  }
+  const environment = env.ATLAS_ENV ?? (env.CI ? 'ci' : 'local');
+  const evidenceDir = env.ATLAS_EVIDENCE_DIR ?? join(root, 'test', 'evidence');
+  const tables = {};
+  for (const [name, t] of sc.tables) {
+    const cases = t.kind === 'combination' ? (t.cover === 'pair' ? pairs(t.dimensions) : combinations(t.dimensions)) : null;
+    tables[name] = { name, kind: t.kind, header: t.header ?? null, rows: t.rows ?? [], fingerprint: t.fingerprint, cases };
+  }
+  return {
+    root, file, environment, evidenceDir, guardRefusal, tests,
+    // What a library in another language must refuse until it has its own form of the guard.
+    shippedDrivers: SHIPPED_DRIVERS,
+    wayGuardParts: GUARD_PARTS.filter((p) => p.startsWith('ways-in.*.')).map((p) => p.slice('ways-in.*.'.length)),
+    envSpec: resolveEnvironments(tests.environments ?? {})[environment] ?? null,
+    scenario: {
+      id: sc.id, title: sc.title, file: sc.file, sha256: sc.sha256, method: sc.method,
+      names: [...new Set([...sc.start, ...sc.steps, ...sc.assertions].flatMap((x) => x.names ?? []))],
+      fields: sc.fields, through: sc.through, ways: sc.ways ?? sc.through, repeat: sc.repeat ?? null,
+      assertions: sc.assertions.map((a) => ({ id: a.id, fullId: a.fullId, kind: a.kind, gate: Boolean(a.gate), ways: a.ways ?? sc.through, fp: a.fp, names: a.names ?? [], line: a.line })),
+      tables,
+    },
+  };
+}
+
+export function main(argv, write = (text) => process.stdout.write(text)) {
+  const [verb, path] = argv;
+  if (verb !== 'load' || !path) { process.stderr.write('bridge: use "bridge.mjs load <scenario.md>"\n'); return 2; }
+  try {
+    write(`${JSON.stringify(loadForLibrary(path))}\n`);
+    return 0;
+  } catch (error) {
+    write(`${JSON.stringify({ error: error.message })}\n`);
+    return 1;
+  }
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) process.exitCode = main(process.argv.slice(2));