@henols/c64-re-tools 2.0.0-rc.3 → 2.0.0-rc.4

package/skills/c64-testing/references/scenario-format.md

@@ -40,19 +40,25 @@ Before the first step of each side, the script does a hard reset. The CPU stops
 | `{ "reset": "hard" }` | Reset the C64 (`hard` or `soft`). The CPU stops at the reset vector. |
 | `{ "frames": 150 }` | Run for an exact number of frames, from 1 to 10000. Use it after a reset to start the C64. |
 | `{ "load": true }` | Load the program of the side into memory. The CPU stays stopped. |
-| `{ "autostart": true }` | Load and start the program of the side as VICE autostart does. |
+| `{ "autostart": true }` | Load the program of the side, type `RUN` and wait until the program runs. The program must have a BASIC start line. The keys of a later `type` step go to the program. |
 | `{ "write": { "at": "seed", "bytes": [1, 2] } }` | Write bytes into memory: 1 to 4096 numbers from 0 to 255. |
 | `{ "registers": { "pc": "start", "a": 0 } }` | Set registers: `pc`, `a`, `x`, `y` or `sp`. A value is a number or a name. |
 | `{ "joystick": { "port": 2, "direction": "left", "fire": true } }` | Set a joystick. The joystick stays in this state until the next joystick step. |
-| `{ "type": "RUN\n" }` | Type text on the keyboard, at most 1024 keys. |
+| `{ "type": "RUN\n" }` | Type text on the keyboard, at most 1024 keys. The keys go into the keyboard queue of the KERNAL. A program that reads the keyboard hardware does not see them. A step cannot press one specific key of the keyboard matrix. |
 | `{ "runUntil": { "at": "main_loop" }, "timeoutFrames": 600 }` | Run until the CPU gets to an address. |
 | `{ "runUntil": { "memory": { "at": "game_state", "equals": 2 } }, "timeoutFrames": 600 }` | Run until a byte in memory has a value. |
+| `{ "runUntil": { "raster": 250 }, "timeoutFrames": 5 }` | Run until the raster beam gets to a line (0 to 311 for PAL, 0 to 262 for NTSC). |
+| `{ "runUntil": { "at": "irq_tick", "raster": 250 }, "timeoutFrames": 600 }` | Run until the CPU gets to an address while the beam is on that line. |
+| `{ "snapshot": "snapshots/start.vsf" }` | Restore a snapshot file. Usually with `side`: the snapshot of one program is not a state of the other. |
+| `{ "normalize": "unpacked/entry_state.json" }` | Write pages 0 to 3 and `$00`/`$01` from an entry state file of the c64-unpacker skill. Then both sides have the same zero page, stack and KERNAL work area. |
 | `{ "observe": "moved", ... }` | Record a checkpoint. |
 
 The `port` of a joystick is 1 or 2. The `direction` is `center`, `up`, `down`, `left`, `right`, `up-left`, `up-right`, `down-left` or `down-right`.
 
 `timeoutFrames` is from 1 to 30000. A `runUntil` that does not get to its target in `timeoutFrames` frames makes the result INCONCLUSIVE.
 
+Each step except `observe` can have `"side": "original"` or `"side": "rebuild"`. Then only that side does the step. Use it when only one side needs a step, for example a key for the intro of the original. An `observe` step has no `side`, because it compares both sides.
+
 An address is `$` and four hex digits, or a symbol name. Each side finds a name in its own symbols.
 
 ## Observations
@@ -61,7 +67,7 @@ An `observe` step has a checkpoint name. A scenario can have each name only one
 
 | Key | What the script compares |
 |---|---|
-| `memory` | A list of names, `$` addresses, or objects with `at`, `size` (1 to 4096, default 1) and `tolerance` (0 to 255, default 0). |
+| `memory` | A list of names, `$` addresses, or objects with `at`, `size` (1 to 4096, default 1), `tolerance` (0 to 255, default 0) and `exclude`. |
 | `registers` | A list of register names: `pc`, `a`, `x`, `y`, `sp`, `flags`. |
 | `screen` | The screen of the rebuild against the screen of the original. `maxMismatchRatio` (default 0) is the largest permitted part of different pixels. `mask` is a list of rectangles (`x`, `y`, `width`, `height`) that the comparison does not include. |
 | `vicii` | The VIC-II state: mode, screen address, scroll and colors. |
@@ -69,8 +75,42 @@ An `observe` step has a checkpoint name. A scenario can have each name only one
 
 A `tolerance` is the largest permitted difference for each byte. A `mask` has at most 64 rectangles. A scenario can observe the screen at most at 64 checkpoints.
 
+`exclude` is a list of bytes in the observed range that the comparison does not include. Each has `at` and a `reason`, for example `{ "at": "$d01e", "reason": "latched sprite collision, not writable" }`. The result lists each excluded byte with its reason.
+
 A `pc` value is an address. Compare `pc` only when both programs have the code at the same address.
 
+## Runs
+
+A run file (for `run.ts`) has:
+
+- `name`: a short name.
+- `start`: `{ "program": "game.prg" }` to autostart a program with a BASIC start line, `{ "program": "game.prg", "load": true }` to load it with the CPU stopped (set `pc` in a step), or `{ "snapshot": "snapshots/room0.vsf" }`.
+- `symbols`: optional, `knowledge`, `none` (default), or a JSON file as for a subject.
+- `videoStandard`: optional, `pal` (default) or `ntsc`.
+- `steps`: the steps `write`, `registers`, `joystick`, `type`, `frames` and `runUntil` of a scenario, and the run steps `snapshot`, `save`, `screen`, `coverage` and `observe`.
+
+A name in `save`, `screen`, `coverage` and `observe` has 1 to 64 letters, digits, dots, underscores or hyphens. The files go into the `--out` directory, with `results.json`. It has the observations, the files and the steps done. If a `runUntil` step did not reach its target, it also tells why the run stopped.
+
+Example: visit two rooms from the same start.
+
+```json
+{
+  "name": "rooms",
+  "start": { "snapshot": "snapshots/room0.vsf" },
+  "steps": [
+    { "write": { "at": "$0350", "bytes": [1] } },
+    { "frames": 50 },
+    { "screen": "room1" },
+    { "coverage": "room1" },
+    { "snapshot": "snapshots/room0.vsf" },
+    { "write": { "at": "$0350", "bytes": [2] } },
+    { "frames": 50 },
+    { "screen": "room2" },
+    { "coverage": "room2" }
+  ]
+}
+```
+
 ## Checks
 
 The script checks the scenario before it starts the emulator. A step with an unknown key or a wrong value stops the script with `invalid-input`. The message names the step.

package/skills/c64-testing/scripts/run.ts

@@ -0,0 +1,189 @@
+// The c64-testing run script (D45): runs steps in its own emulator, from a
+// program or a snapshot file, and writes what the steps ask for into a
+// directory: screens, coverage maps, snapshot files, and results.json with
+// the observations. One side, no comparison. Exit status: 0 when every step
+// ran, 1 when a step did not, 2 for a bad call.
+
+import { parseArgs } from "node:util";
+
+import { formatC64Address, parseC64Address } from "#src/c64.ts";
+import { runProgram } from "#src/host-client/autostart.ts";
+import { readCoverage } from "#src/host-client/coverage.ts";
+import { ACTION_STEP_KEYS, ACTION_STEP_KINDS, checkActionStep, isInteger, isObject, runActionStep, type ActionStep } from "#src/host-client/steps.ts";
+import { MAX_MEMORY_READ, WireFailure } from "#src/host-client/tools.ts";
+import { readProjectFile, writeProjectFile } from "#src/host-client/transfer.ts";
+import { ViceSessionClient } from "#src/host-client/vice-session.ts";
+import { resolveProjectPath } from "#src/project.ts";
+import { runScript, UsageError } from "#src/script.ts";
+import { loadSymbols } from "./scenario.ts";
+
+const USAGE = `run.ts <run.json> --out <dir>
+
+<run.json> is a run file relative to the project directory: where to start,
+optional symbols, and the steps. See references/scenario-format.md, section
+Runs. The files that the steps write, and results.json, go into <dir>. The
+result is one JSON object.`;
+
+type RunStep =
+  | ActionStep
+  | { snapshot: string }
+  | { save: string }
+  | { screen: string }
+  | { coverage: string }
+  | { observe: string; memory?: Array<string | { at: string; size?: number }>; registers?: boolean; vicii?: boolean; sprites?: number[] };
+
+interface Run {
+  name: string;
+  videoStandard?: "pal" | "ntsc";
+  start: { program: string; load?: boolean } | { snapshot: string };
+  symbols?: string;
+  steps: RunStep[];
+}
+
+const OWN_KINDS = ["snapshot", "save", "screen", "coverage", "observe"] as const;
+const OBSERVE_KEYS = ["observe", "memory", "registers", "vicii", "sprites"];
+/** A name of a file that a step writes: letters, digits, dots, underscores or hyphens. */
+const NAME = /^[A-Za-z0-9._-]{1,64}$/;
+
+function readRun(path: string): Run {
+  let value: unknown;
+  try {
+    value = JSON.parse(readProjectFile(path).bytes.toString("utf8"));
+  } catch (error) {
+    if (error instanceof WireFailure) throw error;
+    throw new UsageError(`the run ${path} is not JSON: ${(error as Error).message}`);
+  }
+  const fail = (message: string): never => {
+    throw new UsageError(`run ${path}: ${message}`);
+  };
+  if (!isObject(value)) return fail("is not a JSON object");
+  if (typeof value.name !== "string" || value.name === "") fail("needs a name");
+  if (value.videoStandard !== undefined && value.videoStandard !== "pal" && value.videoStandard !== "ntsc") fail("videoStandard must be pal or ntsc");
+  const start = value.start;
+  if (!isObject(start) || (typeof start.program !== "string") === (typeof start.snapshot !== "string")) fail("start needs program or snapshot");
+  if (isObject(start) && start.load !== undefined && typeof start.load !== "boolean") fail("start.load must be true or false");
+  if (value.symbols !== undefined && typeof value.symbols !== "string") fail("symbols must be knowledge, none or a JSON file");
+  if (!Array.isArray(value.steps) || value.steps.length === 0) fail("needs steps");
+  const labels = new Set<string>();
+  (value.steps as unknown[]).forEach((step, index) => {
+    const at = `step ${index + 1}`;
+    if (!isObject(step)) return fail(`${at} must be an object`);
+    const kinds = "observe" in step ? ["observe"] : Object.keys(step).filter((key) => (ACTION_STEP_KINDS as readonly string[]).includes(key) || (OWN_KINDS as readonly string[]).includes(key));
+    if (kinds.length !== 1) return fail(`${at} must have exactly one of ${[...ACTION_STEP_KINDS, ...OWN_KINDS].join(", ")}`);
+    const kind = kinds[0]!;
+    const allowed = kind === "observe" ? OBSERVE_KEYS : (ACTION_STEP_KEYS[kind] ?? [kind]);
+    for (const key of Object.keys(step)) if (!allowed.includes(key)) fail(`${at} has the key ${key}, which a ${kind} step does not have`);
+    if ((ACTION_STEP_KINDS as readonly string[]).includes(kind)) return checkActionStep(kind, step, at, fail);
+    const name = step[kind];
+    if (kind === "snapshot") {
+      if (typeof name !== "string" || name === "") fail(`${at}: snapshot needs a project file`);
+      return;
+    }
+    if (typeof name !== "string" || !NAME.test(name)) fail(`${at}: ${kind} needs a name of 1 to 64 letters, digits, dots, underscores or hyphens`);
+    if (kind !== "observe") return;
+    if (labels.has(name as string)) fail(`${at} needs a new observation name`);
+    labels.add(name as string);
+    if (step.memory !== undefined && (!Array.isArray(step.memory) || step.memory.some((item) => typeof item !== "string" && !(isObject(item) && typeof item.at === "string" && (item.size === undefined || isInteger(item.size, 1, MAX_MEMORY_READ)))))) {
+      fail(`${at}: memory is a list of names, $ addresses, or objects with at and size`);
+    }
+    if (step.registers !== undefined && typeof step.registers !== "boolean") fail(`${at}: registers must be true or false`);
+    if (step.vicii !== undefined && typeof step.vicii !== "boolean") fail(`${at}: vicii must be true or false`);
+    if (step.sprites !== undefined && (!Array.isArray(step.sprites) || !step.sprites.every((sprite) => isInteger(sprite, 0, 7)))) fail(`${at}: sprites is a list of numbers from 0 to 7`);
+  });
+  return value as unknown as Run;
+}
+
+async function runFile(path: string, out: string): Promise<{ output: Record<string, unknown>; failed: boolean }> {
+  const run = readRun(path);
+  const symbols = loadSymbols({ program: "", symbols: run.symbols ?? "none" });
+  const at = (location: string): number => {
+    if (location.startsWith("$")) return parseC64Address(location);
+    const address = symbols.get(location);
+    if (address === undefined) throw new UsageError(`the symbols have no ${location}`);
+    return address;
+  };
+  const files: string[] = [];
+  const write = (name: string, bytes: Uint8Array) => {
+    writeProjectFile(`${out}/${name}`, bytes, { what: "--out" });
+    files.push(name);
+  };
+  const observations: Record<string, unknown> = {};
+  const client = await ViceSessionClient.open({ videoStandard: run.videoStandard ?? "pal" });
+  let done = 0;
+  let reason: string | undefined;
+  try {
+    await client.reset({ mode: "hard", run: false });
+    if ("snapshot" in run.start) {
+      await client.snapshotRestore(readProjectFile(run.start.snapshot).bytes);
+    } else if (run.start.load === true) {
+      await client.execution({ action: "advance-frames", count: 150, space: "c64" });
+      await client.programLoad({ path: run.start.program });
+    } else {
+      const failed = await runProgram(client, run.start.program, readProjectFile(run.start.program).bytes);
+      if (failed !== undefined) reason = failed === "not-loaded" ? "The program was not in memory after autostart." : "The program did not start after RUN; it needs a BASIC start line.";
+    }
+    await client.memmap({ action: "clear" });
+    for (const step of reason === undefined ? run.steps : []) {
+      if ("snapshot" in step) {
+        await client.snapshotRestore(readProjectFile(step.snapshot).bytes);
+      } else if ("save" in step) {
+        await client.snapshot({ action: "save", name: "run-save" });
+        write(`${step.save}.vsf`, await client.snapshotExport("run-save"));
+        await client.snapshot({ action: "discard", name: "run-save" });
+      } else if ("screen" in step) {
+        write(`${step.screen}.png`, Buffer.from((await client.screenCapture()).png, "base64"));
+      } else if ("coverage" in step) {
+        write(`${step.coverage}.bin`, await readCoverage(client));
+        await client.memmap({ action: "clear" });
+      } else if ("observe" in step) {
+        const seen: Record<string, unknown> = {};
+        if (step.memory !== undefined) {
+          const memory: Record<string, string> = {};
+          for (const item of step.memory) {
+            const spec = typeof item === "string" ? { at: item } : item;
+            memory[spec.at] = (await client.memoryRead({ address: at(spec.at), size: spec.size ?? 1, space: "c64", view: "cpu" })).data;
+          }
+          seen.memory = memory;
+        }
+        if (step.registers === true) {
+          const registers = await client.registersGet("c64");
+          seen.registers = { ...registers, pc: formatC64Address(registers.pc) };
+        }
+        if (step.vicii === true) seen.vicii = await client.vicii();
+        if (step.sprites !== undefined) seen.sprites = (await client.sprites(step.sprites)).sprites;
+        observations[step.observe] = seen;
+      } else {
+        const notReached = await runActionStep(client, step, at);
+        if (notReached !== undefined && "runUntil" in step) {
+          reason = `Step ${done + 1} did not reach its target in ${step.timeoutFrames} frames. It stopped by ${notReached.stopReason} at ${formatC64Address(notReached.pc)}.`;
+          break;
+        }
+      }
+      done++;
+    }
+    const pc = (await client.registersGet("c64")).pc;
+    const results = { name: run.name, completed: reason === undefined, steps: run.steps.length, done, ...(reason === undefined ? {} : { reason }), pc: formatC64Address(pc), files: [...files], observations };
+    write("results.json", Buffer.from(`${JSON.stringify(results, null, 2)}\n`));
+    return { output: { ...results, files, observations: Object.keys(observations), out }, failed: reason !== undefined };
+  } finally {
+    await client.close();
+  }
+}
+
+await runScript(
+  async () => {
+    const { values, positionals } = parseArgs({ args: process.argv.slice(2), strict: true, allowPositionals: true, options: { out: { type: "string" }, help: { type: "boolean" } } });
+    if (values.help) throw new UsageError("");
+    if (positionals.length !== 1) throw new UsageError("give one run file");
+    if (values.out === undefined) throw new UsageError("run needs --out <dir>");
+    try {
+      resolveProjectPath(values.out);
+    } catch {
+      throw new UsageError(`--out must be a path relative to the project directory that stays inside it, not ${values.out}`);
+    }
+    const { output, failed } = await runFile(positionals[0]!, values.out);
+    if (failed) process.exitCode = 1;
+    return output;
+  },
+  { usage: USAGE },
+);