@henols/c64-re-tools 2.0.0-rc.8 → 2.0.0

package/skills/c64-static-analysis/SKILL.md

@@ -11,30 +11,30 @@ Find the static structure of a C64 program. Record the structure in the project
 
 ## Inputs
 
-- The program: a PRG file, relative to the project directory. Or 64 KiB of memory from `$0000`, for example a memory dump.
+- The program in the knowledge. Put it in with `image` of the c64-knowledge skill: a PRG, or a capture of the c64-unpacker skill.
+- Optional: the coverage in the knowledge. Add it with `coverage` of the c64-knowledge skill.
 - Entry points: addresses where code starts, for example the SYS address from the c64-basic skill. Routine symbols in the knowledge are entry points too. Ghidra needs at least one. DXA finds the SYS of a BASIC start by itself.
 - Optional: routines to decompile (Ghidra), or a file for the listing (DXA).
 
 ## Tools / execution path
 
-The script reads the current knowledge and runs DXA or Ghidra on this machine. The host runtime is not necessary. The script records the result in `.c64-re-tools/knowledge.db`.
+The script reads the program, the coverage and the current knowledge, and runs DXA or Ghidra on this machine. The host runtime is not necessary. The script records the result in the project knowledge, `.c64-re-tools/knowledge/`.
 
 ```
-node <skill>/scripts/analyze.ts extracted/game.prg --analyzer dxa --listing analysis/game.lst
-node <skill>/scripts/analyze.ts extracted/game.prg --entry '$080d' --decompile '$080d'
+node <skill>/scripts/analyze.ts --analyzer dxa --listing .c64-re-tools/local/game.lst
+node <skill>/scripts/analyze.ts --entry '$080d' --decompile '$080d'
 ```
 
 | Option | Meaning |
 |---|---|
 | `--analyzer dxa` | Use DXA: a fast first pass. Without this option, the script uses Ghidra. |
 | `--entry '$080d'` | An address where code starts. Use it again for more addresses. |
-| `--listing analysis/game.lst` | DXA only: write the disassembly listing to this project file. |
+| `--listing .c64-re-tools/local/game.lst` | DXA only: write the disassembly listing to this project file. |
 | `--decompile '$0818'` | Ghidra only: decompile the routine at this address. Use it again for more routines, at most 32. |
-| `--flat64k` | The file is 64 KiB of memory from `$0000`, not a PRG. |
 | `--max-listed 20` | List at most this number of labels, functions or candidates. The default is 50. Use 0 to list none. |
-| `--instructions analysis/instructions.json` | Ghidra only: write every instruction to this project file as JSON. Each has `address`, `bytes`, `mnemonic`, `mode` and `operand` in 6502 syntax. A branch, jump or call also has its `target`. With `--coverage`, `executed` tells if it ran. |
-| `--space drive8` | The knowledge space of the image: `drive8` for code from the 1541 drive, or `bank0` to `bank255` for a cartridge bank. The default is `c64`. The seeds come from that space, and the findings go into it. |
-| `--coverage coverage/attract.bin` | A coverage file of 64 KiB from `c64_memmap` (action `read` with `out`). Use it again for more runs. The executed code becomes entry points. |
+| `--instructions .c64-re-tools/local/instructions.json` | Ghidra only: write every instruction to this project file as JSON. Each has `address`, `bytes`, `mnemonic`, `mode` and `operand` in 6502 syntax. A branch, jump or call also has its `target`. When the knowledge has coverage, `executed` tells if it ran. |
+| `--space drive8` | The knowledge space of the program: `drive8` for code from the 1541 drive, or `bank0` to `bank255` for a cartridge bank. The default is `c64`. The seeds come from that space, and the findings go into it. |
+| `--macro-batch .c64-re-tools/local/macros.json` | Write the macro candidates to this project file as a batch for `apply` of the c64-knowledge skill. |
 
 The script prints one JSON object. Addresses are `$` and four hex digits.
 
@@ -49,9 +49,21 @@ Facts that an analyzer found are not seeds. A new analysis can then correct or r
 More seeds come from evidence:
 
 - The start of each `code` region and each entry of a `jumps` table from the user or the LLM is an entry point.
-- With `--coverage`, each executed instruction that starts a flow is an entry point. A flow starts after a gap, or after RTS, RTI, JMP or BRK. Then the analyzers find code that only a vector, an RTS table or an interrupt reaches.
+- When the knowledge has coverage, each executed instruction of the program that starts a flow is an entry point. A flow starts after a gap, or after RTS, RTI, JMP, BRK or JSR. Then the analyzers find code that only a vector, an RTS table or an interrupt reaches. They also find the code after a JSR to a routine that an analyzer thinks does not return.
+- For Ghidra, a flow start is a routine only when no executed branch or JMP from another flow gets to it. Also, no JSR can be immediately before it. Another flow start is code in a routine, and Ghidra does not make a function of it.
+- When the knowledge has coverage, Ghidra gives `coverageGaps`: the executed instruction starts in the program that are not in its instructions. Normally the count is 0. Each start in the list is missing from `--instructions` and from the source of c64-source. Examine the code at the start. Overlapped instructions and self-modified code can cause a gap. A start in a `dual-use` region is a known overlap, for example a BIT skip, and is not in the list.
+- When the knowledge has coverage, some RAM under a ROM is data for the analysis. At these addresses, the runs used the ROM and did not read or execute the RAM. Then a call into the KERNAL does not decode the RAM bytes under it as code. A knowledge routine or an `--entry` in such a range keeps its code. Ghidra does not follow BRK, because the vector at `$fffe` is in ROM unless the program removes the KERNAL.
 
-The analyzers take at most 1024 entry points. Knowledge routines and `--entry` come first, then code regions and tables, then coverage. `coverageSeeds` gives how many entry points the coverage gave and how many the analysis used.
+The analyzers take at most 1024 entry points. Knowledge routines and `--entry` come first, then code regions and tables, then coverage. Ghidra also takes at most 8192 flow starts that are code in a routine. `coverageSeeds` gives how many entry points the coverage gave and how many the analysis used. `romOnlyRanges` and `romOnlyUsed` give the same counts for the data ranges under ROM.
+
+### Project folder
+
+- Paths are relative to the project folder. The project folder is the nearest folder at or above the working directory that holds `.c64-re-tools/`.
+- If the working directory is not in a project and has projects below it, the script refuses and lists them. Then give `--project <folder>` to each script.
+- Write each file that you make for your work in `.c64-re-tools/local/` of the project. Git ignores that folder. Examples are results, batches, step files, runs, snapshots and memory dumps.
+- Move a file out of `.c64-re-tools/local/` only when it is the evidence for a finding. Move it before you give it to `--evidence` of c64-knowledge. `--evidence` refuses a file in `.c64-re-tools/local/`.
+- Put a run that is evidence in `runs/<name>/`, with its run file, its results and the snapshot files that it uses. Change the paths in the run file to the new paths. Then git keeps them, and another person can do the run again.
+- A shell redirect (`>`) is relative to the working directory, not to the project folder. From the folder above the project, write `<folder>/.c64-re-tools/local/result.json`.
 
 ## Workflow
 
@@ -60,13 +72,16 @@ The analyzers take at most 1024 entry points. Knowledge routines and `--entry` c
 3. Find the important routines. Give them names with the c64-knowledge skill.
 4. Run a deeper pass with Ghidra. Give the entry points with `--entry`. Ask for decompiled code of the routines that you must understand.
 5. Read the result:
-   - `changes` counts the symbols, regions and references that the analysis `added`, `changed`, `retired` or kept `unchanged`. `suppressed` counts the findings that the analysis did not write, because a removal blocks that place.
+   - `changes` counts the analyzer names (`names`), regions and references that the analysis `added`, `changed`, `retired` or kept `unchanged`. `symbols.retired` counts the old symbols of the analyzer that moved to its analyzer names. `suppressed` counts the findings that the analysis did not write, because a removal blocks that place.
    - DXA: `regions` lists the code and data regions. `labels` gives the number of names that DXA found (`count`) and lists the first names (`listed`).
    - Ghidra: `functions` gives the number of routines that Ghidra found (`count`) and lists the first routines (`listed`). `decompilations` gives C-like code for the requested routines.
    - If a list is not complete, `rest` gives the c64-knowledge command that lists all of them. For `candidates`, `rest` tells you to give a larger `--max-listed`.
    - `conflicts` lists the places where the analyzer and the current knowledge disagree.
-   - `idioms` tells what the script found by pattern: RTS dispatch tables (`rtsTables`), JMP tables (`jumpTables`) and BIT skips (`bitSkips`). The script writes them as knowledge with the origin `heuristic`, and seeds their targets into the same analysis. `candidates` gives DOP padding (`dop`) and opcodes that the program wrote and executed (`selfModified`). For `selfModified`, the write and the execution must be in one coverage file. A coverage file does not record the order. Thus a candidate can also be code that ran first, and that the program then wrote over. To prevent this, save one coverage file for each part of the program, for example the intro and the game. Clear the memory map between the parts. Each list has a `count`. The script does not write the candidates. Look at them and decide.
-   - With `--coverage`, a code finding replaces the DXA or Ghidra data where a run executed an instruction start. `coverageConflicts` lists the analyzer data regions where the coverage still shows executed code. Classify them as code with the c64-knowledge skill, or as `dual-use` when the bytes are also data.
+   - `idioms` tells what the script found by pattern: RTS dispatch tables (`rtsTables`), JMP tables (`jumpTables`) and BIT skips (`bitSkips`). It also finds the `CBM80` header of a cartridge bank at `$8000` (`cartridgeHeaders`). The two start vectors of the header become entry points. The script writes them as knowledge with the origin `heuristic`, and seeds their targets into the same analysis. `candidates` gives DOP padding (`dop`) and opcodes that the program wrote and executed (`selfModified`). For `selfModified`, the write and the execution must be in one coverage file. A coverage file does not record the order. Thus a candidate can also be code that ran first, and that the program then wrote over. To prevent this, save one coverage file for each part of the program, for example the intro and the game. A loader that copies relocated code is also such a part. Clear the memory map between the parts. Each list has a `count`. The script does not write the candidates. Look at them and decide.
+   - `candidates.pointers` gives code that makes an address from immediates. A `vector` or a `zero-page pointer` candidate stores a low byte (`low`) and a high byte (`high`) to a vector or to a zero-page pair that `(zp),y` uses. Its `target` is in the program or has a platform name. A `sprite pointer` candidate stores an immediate (`at`) to a sprite pointer of the screen at $0400 in VIC bank 0. Its `target` (the value × 64) is in a `sprite` region. For each candidate that you confirm, write operand expressions with `operand` of the c64-knowledge skill, for example `operand '<low>' --lo <target name>` and `operand '<high>' --hi <target name>`. Then a rebuild can move the target.
+   - `candidates.pitfalls` gives code that a C64 runs in a different way than it reads. `raster-bit-8`: the instruction at `at` reads $D011, and the instruction at `store` writes the value back. A read of $D011 gives bit 8 of the raster line in bit 7. Thus the write can move the raster interrupt to line 256 or later, and on PAL past line 311 the interrupt never comes. Examine each candidate. Where the code must not keep bit 7, add a comment and test the program.
+   - `candidates.macros` gives runs of instructions that a macro writes as one line, for example `+inc16 ptr`. The script knows these 6502 idioms: `inc16`, `dec16`, `add16_8`, `add16`, `sub16`, `set16`, `mov16` and `wait_line`. It reads the code regions after the analysis. An arg has the name of a `user` or `llm` symbol or a platform name. Each candidate has `macro`, `start`, `end`, `args` and `text`. `known` counts the uses that the knowledge has. `leftOut` counts the candidates with a label inside, or with a patch or a use over them. The c64-source skill writes such a candidate as instructions. With `--macro-batch`, `batch` gives the file, the count of writes and the `apply` command. The batch holds the macros that the knowledge does not have, and the uses. When the knowledge has a macro with the same name and another body, `otherMacros` tells that the batch does not have its uses. The script does not write the candidates. Apply the batch, then generate the source again.
+   - When the knowledge has coverage, a code finding replaces the DXA or Ghidra data where a run executed an instruction start. `coverageConflicts` lists the analyzer data regions where the coverage still shows executed code. Classify them as code with the c64-knowledge skill, or as `dual-use` when the bytes are also data. For many conflicts, write all classifications in one `apply` batch of the c64-knowledge skill.
 6. Give each important routine a name with the c64-knowledge skill. Then run the analysis again. The analyzers use the new names.
 
 The script records only structure: generated names, code and data regions, and references from Ghidra. The listing and the decompiled code are only for the current task. The script does not record them in the knowledge.
@@ -75,14 +90,16 @@ The script records only structure: generated names, code and data regions, and r
 
 Automatic write:
 - code and data regions
-- names that DXA generated (labels) and routine entry points with names that Ghidra generated
+- analyzer names: the labels that DXA generated and the functions that Ghidra found, for example `l17e2` and `FUN_17e2`
 - references from Ghidra: call, jump, read and write.
 
-The script never replaces a name or a region from the user or the LLM. A name that an analyzer only shows back stays with its owner. DXA and Ghidra do not replace the facts of each other, with one exception. With `--coverage`, a code finding replaces a DXA or Ghidra data region or data name where a run executed an instruction start. The revision keeps the coverage files as evidence. Thus run DXA first, then Ghidra with all coverage files.
+An analyzer name is a second name of the address. It is not a symbol. Each analyzer keeps its own name at an address. The generated source does not use an analyzer name, unless you give `--analyzer-names` to the c64-source skill. To use a name in the source, give it to the address with `rename` of the c64-knowledge skill. A Ghidra function that only branches and jumps get to is a `label`, not a `routine`.
+
+The script never replaces a name or a region from the user or the LLM. DXA and Ghidra do not replace the facts of each other, with one exception. When the knowledge has coverage, a code finding replaces a DXA or Ghidra data region where a run executed an instruction start. Thus add all coverage maps to the knowledge first. Then run DXA, then Ghidra.
 
 A `heuristic` region replaces the part of a DXA or Ghidra region where it lies, because an idiom is specific evidence about those bytes. DXA and Ghidra never replace a `heuristic` region. If they find code there, the result shows a conflict. A heuristic table can be too long or too short, because the script cannot know its length. Correct it with the c64-knowledge skill. Your region replaces it, and a removal holds.
 
-A new analysis replaces the earlier facts of the same analyzer in the analyzed range. The history keeps the earlier facts. DXA labels stay until a person or the LLM removes them, because DXA names only the addresses that it sees used.
+A new analysis replaces the earlier facts of the same analyzer in the analyzed range, also its analyzer names. The history keeps the earlier facts. Knowledge from an older version can have analyzer names as symbols. A new analysis moves the names of that analyzer to its analyzer names.
 
 A fact that the user or the LLM removes does not come back. The next analysis does not write a fact at that place, and counts it as `suppressed`. To remove a wrong fact, use the c64-knowledge skill.
 
@@ -100,7 +117,7 @@ The exit status is 0 for a result, also with conflicts. The exit status is 1 for
 
 - A conflict: the analyzer finds code where the knowledge has data, or data where the knowledge has code. The knowledge stays unchanged. Find the correct answer with the c64-emulator skill, then correct the knowledge with the c64-knowledge skill.
 - A conflict with the problem "another address has this name": the knowledge has the generated name at a different address. The knowledge stays unchanged.
-- A false reference through a vector: the code jumps through a vector, for example `JMP ($0408)`, and the program writes the vector at run time. The analyzer follows the bytes that the image has there. Do these steps:
+- A false reference through a vector: the code jumps through a vector, for example `JMP ($0408)`, and the program writes the vector at run time. The analyzer follows the bytes that the program has there. Do these steps:
   1. Find the real targets with the c64-emulator skill, or in a coverage file.
   2. If the vector bytes are also code at another time, classify them as `dual-use`.
   3. Remove the false reference with `unreference`. It does not come back.

package/skills/c64-static-analysis/scripts/analyze.ts

@@ -1,49 +1,54 @@
-// The c64-static-analysis script: analyzes a project program with DXA (fast
-// first pass) or Ghidra (deeper), which it runs itself, seeded from
-// current knowledge, and imports the structural findings into
-// .c64-re-tools/knowledge.db.
+// The c64-static-analysis script: analyzes the program in the knowledge
+// with DXA (fast first pass) or Ghidra (deeper), which it runs itself, seeded
+// from current knowledge and the coverage of the knowledge, and imports the
+// structural findings into .c64-re-tools/knowledge/.
 
-import { createHash } from "node:crypto";
 import { mkdirSync, writeFileSync } from "node:fs";
 import { dirname } from "node:path";
 import { parseArgs } from "node:util";
 
 import { COVERAGE, formatC64Address, parseC64Address, prgRange } from "#src/c64.ts";
-import { joinCoverage } from "#src/host-client/coverage.ts";
 import { WireFailure } from "#src/host-client/tools.ts";
-import { readProjectFile, writeProjectFile } from "#src/host-client/transfer.ts";
+import { writeProjectFile } from "#src/host-client/transfer.ts";
 import { analyze as analyzeWithDxa } from "#src/native/dxa.ts";
 import { analyze as analyzeWithGhidra } from "#src/native/ghidra/analyze.ts";
 import { localToolContext } from "#src/native/local.ts";
 import { GHIDRA_LIMITS } from "#src/native/types.ts";
 import { KnowledgeError, openForRead, withWrite } from "#src/knowledge/database.ts";
+import { analyzedBytes, programCoverage, requireProgram } from "#src/knowledge/program.ts";
 import { importFindings, runProvenance, type Analyzer, type ImportConflict, type ImportContext, type ImportResult, type NormalizedFindings } from "#src/knowledge/import.ts";
-import { currentRevision } from "#src/knowledge/read.ts";
+import { currentRevision, listMacros, listMacroUses, listPatches, listReferences, listRegions, listSymbols } from "#src/knowledge/read.ts";
 import { DEFAULT_SPACE, SPACE } from "#src/knowledge/schema.ts";
+import { platformNames } from "#src/platform.ts";
 import { resolveProjectPath } from "#src/project.ts";
 import { runScript, UsageError } from "#src/script.ts";
-import { codeEntries, coverageConflicts, coverageEntries, imageReader, readCoverageMaps } from "./coverage.ts";
-import { dxaFindings, ghidraFindings, seedsFromKnowledge } from "./findings.ts";
-import { findIdioms } from "./idioms.ts";
-import { instructionList } from "./instructions.ts";
+import { codeEntries, coverageConflicts, coverageEntries, coverageSeedKinds, executedStarts, imageReader, romOnlyRanges, type ImageByte } from "./lib/coverage.ts";
+import { dxaFindings, ghidraFindings, seedsFromKnowledge } from "./lib/findings.ts";
+import { findIdioms } from "./lib/idioms.ts";
+import { instructionList } from "./lib/instructions.ts";
+import { CATALOGUE, findMacroUses, type MacroCandidate } from "./lib/macros.ts";
+import { findPitfalls, type Pitfall } from "./lib/pitfalls.ts";
+import { findPointers, type PointerCandidate } from "./lib/pointers.ts";
 
-const USAGE = `analyze.ts <image> [options]
+const USAGE = `analyze.ts [options]
+
+The program and the coverage come from the knowledge: put them in with
+knowledge.ts image and coverage of the c64-knowledge skill. The executed code of
+the coverage is seeded, and analyzer data over it is reported.
 
-  <image>                 a PRG file, relative to the project directory
   --analyzer <name>       dxa (fast first pass) or ghidra (deeper, the default)
-  --flat64k               the image is 64 KiB of memory from $0000, not a PRG
   --entry <address>       an entry point, for example '$080d' (repeatable)
   --decompile <address>   ghidra: a routine to decompile, at most 32 (repeatable)
   --listing <file>        dxa: write the disassembly listing to this project file
   --instructions <file>   ghidra: write every instruction (address, bytes, mnemonic,
                           mode, operand, target, executed) to this project file as JSON
-  --coverage <file>       a coverage map of 64 KiB from c64_memmap (repeatable): its
-                          executed code is seeded, and analyzer data over it is reported
   --max-listed <n>        list at most n labels, functions or candidates (default 50,
                           0 for none); the counts are always given, and knowledge has
                           all labels and functions
-  --space <space>         the knowledge space of the image: c64 (the default), drive8
+  --space <space>         the knowledge space of the program: c64 (the default), drive8
                           for 1541 drive code, or bank0 to bank255 for a cartridge bank
+  --macro-batch <file>    write the macro candidates as an apply batch of c64-knowledge
+                          (the macros and their uses) to this project file
 
 Routine symbols in knowledge are entry points too. Ghidra needs at least one
 entry point; dxa finds the SYS of a BASIC start by itself. The result is one
@@ -56,6 +61,92 @@ function listed<T, U>(items: T[], max: number, show: (item: T) => U, all: string
   return { count: items.length, listed: items.slice(0, max).map(show), ...(items.length > max ? { rest: all } : {}) };
 }
 
+/** A pitfall with $ addresses and what goes wrong. */
+function showPitfall(pitfall: Pitfall) {
+  return {
+    kind: pitfall.kind,
+    at: formatC64Address(pitfall.at),
+    store: formatC64Address(pitfall.store),
+    problem: "A read of $D011 gives raster bit 8 in bit 7; the write sets bit 8 of the raster compare. Clear bit 7 (AND #$7F) unless the code wants a compare at line 256 or later.",
+  };
+}
+
+/** A pointer candidate with $ addresses, and the operand writes that would name it. */
+function showPointer(candidate: PointerCandidate) {
+  const hex = formatC64Address;
+  if (candidate.kind === "sprite pointer") {
+    return { kind: candidate.kind, at: hex(candidate.at), to: hex(candidate.to), value: `$${candidate.value.toString(16).padStart(2, "0")}`, target: hex(candidate.target), assumes: "VIC bank 0 and the screen at $0400" };
+  }
+  return { kind: candidate.kind, to: hex(candidate.to), target: hex(candidate.target), low: hex(candidate.low), high: hex(candidate.high) };
+}
+
+/** A macro candidate with $ addresses and its source line. */
+function showMacro(candidate: MacroCandidate) {
+  return { macro: candidate.macro, start: formatC64Address(candidate.start), end: formatC64Address(candidate.end), args: candidate.args, text: `+${candidate.macro} ${candidate.args.join(", ")}` };
+}
+
+/**
+ * The macro candidates in the current code regions of every origin,
+ * with the names of the knowledge in their args. A candidate is left out when
+ * a name or a reference target is inside it, or a patch or a use overlaps it:
+ * c64-source would write its instructions. The same use in the knowledge is
+ * known. With `batchPath`, the candidates go to an apply batch with the
+ * macros they need; a macro of the catalogue that the knowledge has with
+ * another body leaves its candidates out of the batch.
+ */
+function macroCandidates(space: string, byteAt: ImageByte, batchPath: string | undefined) {
+  const db = openForRead();
+  try {
+    const platform = space === DEFAULT_SPACE ? new Map(platformNames().map((entry) => [entry.address, entry.name])) : new Map<number, string>();
+    const semantic = new Map(listSymbols(db, undefined, undefined, ["user", "llm"], space).map((symbol) => [symbol.address, symbol.name]));
+    const code = listRegions(db, undefined, "code", undefined, space);
+    const found = findMacroUses(byteAt, code, (address) => semantic.get(address) ?? platform.get(address));
+    const named = new Set([...listSymbols(db, undefined, undefined, undefined, space).map((symbol) => symbol.address), ...listReferences(db, undefined, undefined, undefined, space).map((reference) => reference.to)]);
+    const patches = listPatches(db, undefined, undefined, undefined, space);
+    const uses = listMacroUses(db, undefined, undefined, undefined, space);
+    const overlaps = (a: { start: number; end: number }, b: { start: number; end: number }) => a.start <= b.end && b.start <= a.end;
+    const candidates: MacroCandidate[] = [];
+    let known = 0;
+    let leftOut = 0;
+    for (const candidate of found) {
+      const same = uses.find((use) => use.start === candidate.start && use.end === candidate.end && use.macro === candidate.macro && use.args.join(",") === candidate.args.join(","));
+      if (same !== undefined) known++;
+      else if ([...named].some((address) => address > candidate.start && address <= candidate.end) || patches.some((patch) => overlaps(patch, candidate)) || uses.some((use) => overlaps(use, candidate))) leftOut++;
+      else candidates.push(candidate);
+    }
+    if (batchPath === undefined) return { candidates, known, leftOut };
+    // A macro of the catalogue goes into the batch unless the knowledge has it; with another body, its uses stay out.
+    const macros = new Map(listMacros(db, undefined, space).map((macro) => [macro.name, macro]));
+    const batch: Array<Record<string, unknown>> = [];
+    const batched: MacroCandidate[] = [];
+    const other = new Set<string>();
+    for (const idiom of CATALOGUE.filter((item) => candidates.some((candidate) => candidate.macro === item.name))) {
+      const existing = macros.get(idiom.name);
+      if (existing === undefined) batch.push({ op: "macro", name: idiom.name, params: idiom.params, body: idiom.body, comment: idiom.comment, ...(space === DEFAULT_SPACE ? {} : { space }) });
+      else if (existing.params.join(",") !== idiom.params.join(",") || existing.body !== idiom.body) other.add(idiom.name);
+    }
+    for (const candidate of candidates) {
+      if (other.has(candidate.macro)) continue;
+      batched.push(candidate);
+      batch.push({ op: "use", start: formatC64Address(candidate.start), end: formatC64Address(candidate.end), macro: candidate.macro, args: candidate.args, ...(space === DEFAULT_SPACE ? {} : { space }) });
+    }
+    if (batch.length > 0) writeProjectFile(batchPath, Buffer.from(`${JSON.stringify(batch, null, 1)}\n`), { what: "--macro-batch" });
+    return {
+      candidates,
+      known,
+      leftOut,
+      batch: {
+        out: batchPath,
+        writes: batch.length,
+        ...(other.size === 0 ? {} : { otherMacros: [...other].map((name) => `The knowledge has another macro ${name}: its candidates are not in the batch.`) }),
+        ...(batch.length === 0 ? {} : { apply: `knowledge.ts apply ${batchPath} --reason "macro uses of 6502 idioms"` }),
+      },
+    };
+  } finally {
+    db?.close();
+  }
+}
+
 /** A decompiler that did not start gets the remedy: Ghidra ships no native decompiler for macOS. */
 function withRemedy(reason: string): string {
   return reason.startsWith("the Ghidra decompiler did not start")
@@ -93,10 +184,10 @@ function showConflict(conflict: ImportConflict, analyzer: Analyzer): Record<stri
 
 /**
  * Imports one analyzer's findings as one revision, refused when knowledge
- * changed since the seeds were read. A refused import creates no knowledge.db.
+ * changed since the seeds were read. A refused import creates no knowledge.
  * The revision records the image hash and the analyzer version.
  */
-function record(findings: NormalizedFindings, revision: number, description: string, provenance: Pick<ImportContext, "inputHash" | "toolVersion" | "evidence">) {
+function record(findings: NormalizedFindings, revision: string, description: string, provenance: Pick<ImportContext, "inputHash" | "toolVersion" | "evidence">) {
   let imported: ImportResult;
   try {
     imported = withWrite((db) => importFindings(db, findings, { expectedRevision: revision, description, ...provenance }));
@@ -108,7 +199,7 @@ function record(findings: NormalizedFindings, revision: number, description: str
   }
   return {
     revision: imported.revision,
-    changes: { symbols: imported.symbols, regions: imported.regions, references: imported.references },
+    changes: { names: imported.names, ...(imported.symbols.retired === 0 ? {} : { symbols: { retired: imported.symbols.retired } }), regions: imported.regions, references: imported.references },
     conflicts: imported.conflicts.map((conflict) => showConflict(conflict, findings.analyzer)),
   };
 }
@@ -120,20 +211,18 @@ async function run(argv: string[]): Promise<unknown> {
     allowPositionals: true,
     options: {
       analyzer: { type: "string" },
-      flat64k: { type: "boolean" },
       entry: { type: "string", multiple: true },
       decompile: { type: "string", multiple: true },
       listing: { type: "string" },
       "max-listed": { type: "string" },
-      coverage: { type: "string", multiple: true },
       instructions: { type: "string" },
       space: { type: "string" },
+      "macro-batch": { type: "string" },
       help: { type: "boolean" },
     },
   });
   if (values.help) throw new UsageError("");
-  if (positionals.length !== 1) throw new UsageError("give one image");
-  const image = positionals[0]!;
+  if (positionals.length !== 0) throw new UsageError("analyze takes no image: it reads the program from the knowledge (knowledge.ts image of the c64-knowledge skill puts it in)");
   const entries = addresses(values.entry, "--entry");
   const decompile = addresses(values.decompile, "--decompile");
   if (decompile.length > 32) throw new UsageError("--decompile takes at most 32 routines");
@@ -150,45 +239,68 @@ async function run(argv: string[]): Promise<unknown> {
     try {
       listingPath = resolveProjectPath(values.listing);
     } catch {
-      throw new UsageError(`--listing must be a path relative to the project directory that stays inside it, not ${values.listing}`);
+      throw new UsageError(`--listing must be a path relative to the project folder that stays inside it, not ${values.listing}`);
+    }
+  }
+
+  if (values["macro-batch"] !== undefined) {
+    try {
+      resolveProjectPath(values["macro-batch"]);
+    } catch {
+      throw new UsageError(`--macro-batch must be a path relative to the project folder that stays inside it, not ${values["macro-batch"]}`);
     }
   }
 
   const space = values.space ?? DEFAULT_SPACE;
   if (!SPACE.test(space)) throw new UsageError("--space must be c64, drive8, or bank0 to bank255");
-  // A coverage map is of the C64 CPU: VICE keeps none for the drive (D53).
-  if (space !== DEFAULT_SPACE && values.coverage !== undefined) throw new UsageError("--coverage works only in space c64");
-  const imageKind = values.flat64k ? "flat64k" : "prg";
-  const bytes = readProjectFile(image).bytes;
-  const runs = readCoverageMaps(values.coverage ?? []);
-  const coverage = runs.length === 0 ? undefined : joinCoverage(runs);
+  const reader = openForRead();
+  const program = requireProgram(reader, space);
+  // A coverage map is of the C64 CPU: VICE keeps none for the drive.
+  const covered = space === DEFAULT_SPACE ? programCoverage(reader) : undefined;
+  const coverage = covered?.map;
+  const { bytes, kind: imageKind } = analyzedBytes(program);
+  // The analyzer sees all bytes it gets; the coverage seeds only the program, as c64-source does.
   const byteAt = imageReader(bytes, imageKind);
-  // With coverage, a code finding over an executed instruction start replaces the data of the other analyzer (D58), and the files are the evidence.
+  const inProgram: ImageByte = (address) => (program.segments.some((segment) => address >= segment.start && address <= segment.end) ? program.byteAt(address) : undefined);
+  // With coverage, a code finding over an executed instruction start replaces the data of the other analyzer.
   const executed = coverage === undefined ? {} : { executed: (address: number) => (coverage[address]! & COVERAGE.ramExecute) !== 0 };
-  const coverageEvidence = coverage === undefined ? {} : { evidence: (values.coverage ?? []).map((path) => ({ kind: "coverage" as const, ref: path, sha256: createHash("sha256").update(readProjectFile(path).bytes).digest("hex") })) };
-
-  let loaded: { start: number; end: number };
-  try {
-    loaded = imageKind === "flat64k" ? { start: 0x0000, end: bytes.length - 1 } : prgRange(bytes);
-  } catch (error) {
-    throw new UsageError(`${image}: ${(error as Error).message}`);
-  }
-  const idioms = findIdioms(byteAt, { start: loaded.start, end: loaded.end }, runs);
+  const loaded = imageKind === "flat64k" ? { start: 0x0000, end: 0xffff } : prgRange(bytes);
+  const idioms = findIdioms(byteAt, { start: loaded.start, end: loaded.end }, covered);
 
-  const reader = openForRead();
   const revision = currentRevision(reader);
   const seeds = seedsFromKnowledge(reader, entries, space);
-  // Knowledge routines and --entry first, then code regions and JMP tables, then idioms, then coverage, up to the seed bound (D44).
-  const fromCoverage = coverage === undefined ? [] : coverageEntries(coverage, byteAt);
-  const ordered = [...new Set([...seeds.entryPoints, ...codeEntries(reader, byteAt, space), ...idioms.entryPoints, ...fromCoverage])];
+  // Knowledge routines and --entry first, then code regions and JMP tables, then idioms, then coverage, up to the seed bound.
+  const fromCoverage = coverage === undefined ? [] : coverageEntries(coverage, inProgram);
+  const known = [...seeds.entryPoints, ...codeEntries(reader, byteAt, space)];
+  // Ghidra takes the coverage as code starts without a function; DXA has one list.
+  const kinds = analyzer === "dxa" || coverage === undefined ? { routines: fromCoverage, codeStarts: [] } : coverageSeedKinds(coverage, inProgram);
+  const ordered = [...new Set([...known, ...idioms.entryPoints, ...kinds.routines])];
   seeds.entryPoints = ordered.slice(0, GHIDRA_LIMITS.entryPoints).sort((a, b) => a - b);
-  // Idiom tables are data for this run too; semantic data ranges come first.
-  seeds.dataRanges = [...seeds.dataRanges, ...idioms.dataRanges.filter((range) => !seeds.dataRanges.some((seed) => range.start <= seed.end && range.end >= seed.start))].slice(0, GHIDRA_LIMITS.dataRanges);
-  const coverageUsed = fromCoverage.filter((entry) => seeds.entryPoints.includes(entry)).length;
+  const codeStarts = kinds.codeStarts.filter((entry) => !seeds.entryPoints.includes(entry)).slice(0, GHIDRA_LIMITS.codeStarts);
+  // Idiom tables and the RAM under ROM that the runs never used are data for this run too; semantic data ranges come first.
+  // A range with an entry point of the knowledge or --entry in it is left out: that code stays code.
+  const fromRom = coverage === undefined ? [] : romOnlyRanges(coverage, byteAt).filter((range) => !known.some((entry) => entry >= range.start && entry <= range.end));
+  const dataRanges = [...seeds.dataRanges];
+  for (const range of [...idioms.dataRanges, ...fromRom]) if (!dataRanges.some((seed) => range.start <= seed.end && range.end >= seed.start)) dataRanges.push(range);
+  seeds.dataRanges = dataRanges.slice(0, GHIDRA_LIMITS.dataRanges);
+  const romUsed = fromRom.filter((range) => seeds.dataRanges.includes(range)).length;
+  const coverageUsed = fromCoverage.filter((entry) => seeds.entryPoints.includes(entry) || codeStarts.includes(entry)).length;
+  // Platform names are C64 names: drive code and banks get none.
+  const platform = space === DEFAULT_SPACE ? platformNames() : [];
+  const pointers = findPointers(byteAt, { start: loaded.start, end: loaded.end }, {
+    platform: (address) => platform.some((entry) => address >= entry.address && address <= entry.end),
+    sprites: listRegions(reader, undefined, "sprite", undefined, space),
+  });
   reader?.close();
   /** Imports the idioms as their own revision, after the analyzer's, and gives their part of the result. */
   const idiomsResult = () => {
-    const imported = withWrite((db) => importFindings(db, { ...idioms.findings, space }, { description: `Idioms in ${image}` }));
+    const imported = withWrite((db) => importFindings(db, { ...idioms.findings, space }, { description: "Idioms in the program" }));
+    // The macro candidates read the code regions after both imports.
+    const macros = macroCandidates(space, byteAt, values["macro-batch"]);
+    // Platform pitfalls are C64 facts: drive code and banks get none.
+    const pitfallDb = openForRead();
+    const pitfalls = space === DEFAULT_SPACE ? findPitfalls(byteAt, listRegions(pitfallDb, undefined, "code", undefined, space)) : [];
+    pitfallDb?.close();
     return {
       idioms: {
         ...idioms.counts,
@@ -199,6 +311,14 @@ async function run(argv: string[]): Promise<unknown> {
         candidates: {
           dop: listed(idioms.candidates.dop, max, formatC64Address, "a larger --max-listed"),
           selfModified: listed(idioms.candidates.selfModified, max, formatC64Address, "a larger --max-listed"),
+          pointers: listed(pointers, max, showPointer, "a larger --max-listed"),
+          pitfalls: listed(pitfalls, max, showPitfall, "a larger --max-listed"),
+          macros: {
+            ...listed(macros.candidates, max, showMacro, "a larger --max-listed, or --macro-batch <file>"),
+            ...(macros.known === 0 ? {} : { known: macros.known }),
+            ...(macros.leftOut === 0 ? {} : { leftOut: macros.leftOut }),
+            ...(macros.batch === undefined ? {} : { batch: macros.batch }),
+          },
         },
       },
     };
@@ -210,7 +330,7 @@ async function run(argv: string[]): Promise<unknown> {
     const conflicts = coverageConflicts(db, coverage);
     db?.close();
     return {
-      coverageSeeds: { entryPoints: fromCoverage.length, used: coverageUsed },
+      coverageSeeds: { entryPoints: fromCoverage.length, used: coverageUsed, romOnlyRanges: fromRom.length, romOnlyUsed: romUsed },
       coverageConflicts: conflicts.map(({ region, executed }) => ({
         range: { start: formatC64Address(region.start), end: formatC64Address(region.end) },
         knowledge: { type: region.type, origin: region.origin },
@@ -223,7 +343,7 @@ async function run(argv: string[]): Promise<unknown> {
     const { result, attachments } = await analyzeWithDxa({ imageKind, ...seeds }, bytes, localToolContext());
     const listing = attachments[0]!.toString("utf8");
     // The listing is written only after the import: a refused import leaves no new file.
-    const imported = record({ ...dxaFindings(result), space, ...executed }, revision, `DXA analysis of ${image}`, { ...runProvenance(bytes, result.toolVersion), ...coverageEvidence });
+    const imported = record({ ...dxaFindings(result), space, ...executed }, revision, "DXA analysis of the program", runProvenance(bytes, result.toolVersion));
     if (listingPath !== undefined) {
       try {
         mkdirSync(dirname(listingPath), { recursive: true });
@@ -247,9 +367,17 @@ async function run(argv: string[]): Promise<unknown> {
     };
   }
 
-  const { result } = await analyzeWithGhidra({ imageKind, ...seeds, decompile }, bytes, localToolContext());
-  const imported = record({ ...ghidraFindings(result), space, ...executed }, revision, `Ghidra analysis of ${image}`, { ...runProvenance(bytes, result.toolVersion), ...coverageEvidence });
+  const { result } = await analyzeWithGhidra({ imageKind, ...seeds, codeStarts, decompile }, bytes, localToolContext());
+  // Executed instruction starts that Ghidra did not decode: the instruction list and the source miss them.
+  const decoded = new Set(result.instructions.map((instruction) => instruction.address));
+  const imported = record({ ...ghidraFindings(result), space, ...executed }, revision, "Ghidra analysis of the program", runProvenance(bytes, result.toolVersion));
   const found = idiomsResult();
+  // A start inside a dual-use region is a known overlap, for example the second instruction of a BIT skip: one list cannot hold both.
+  // Read after the idioms, which write those regions.
+  const overlapDb = coverage === undefined ? undefined : openForRead();
+  const overlaps = listRegions(overlapDb, undefined, "dual-use", undefined, space);
+  overlapDb?.close();
+  const gaps = coverage === undefined ? [] : executedStarts(coverage, inProgram).filter((address) => address >= loaded.start && address <= loaded.end && !decoded.has(address) && !overlaps.some((region) => address >= region.start && address <= region.end));
   let instructions: { out: string; count: number } | undefined;
   if (values.instructions !== undefined) {
     const list = instructionList(result.instructions, coverage);
@@ -262,6 +390,7 @@ async function run(argv: string[]): Promise<unknown> {
     ...(instructions === undefined ? {} : { instructions }),
     ...found,
     ...coverageResult(),
+    ...(coverage === undefined ? {} : { coverageGaps: listed(gaps, max, formatC64Address, "a larger --max-listed") }),
     coverage: result.coverage.map((range) => ({ start: formatC64Address(range.start), end: formatC64Address(range.end) })),
     ...imported,
     functions: listed(result.functions, max, (fn) => ({ entry: formatC64Address(fn.entry), name: fn.name }), "c64-knowledge: knowledge.ts symbols --kind routine"),
@@ -272,4 +401,4 @@ async function run(argv: string[]): Promise<unknown> {
   };
 }
 
-await runScript(() => run(process.argv.slice(2)), { usage: USAGE, refusals: [KnowledgeError] });
+await runScript((argv) => run(argv), { usage: USAGE, refusals: [KnowledgeError] });

package/skills/c64-static-analysis/scripts/{coverage.ts → lib/coverage.ts}

@@ -1,27 +1,19 @@
-// Runtime evidence for static analysis (D44): coverage files as entry points,
+// Runtime evidence for static analysis: the coverage as entry points,
 // user/LLM code regions and JMP tables as entry points, and the check that
 // shows analyzer data over executed code.
 
 import type { DatabaseSync } from "node:sqlite";
 
-import { COVERAGE, instructionLength } from "#src/c64.ts";
-import { FULL_COVERAGE, readCoverageMaps as readMaps } from "#src/host-client/coverage.ts";
+import { COVERAGE, coverageEntries, coverageSeedKinds, executedStarts } from "#src/c64.ts";
+
+export { coverageEntries, coverageSeedKinds, executedStarts };
+import { FULL_COVERAGE } from "#src/host-client/coverage.ts";
 import { listRegions, type RegionRow } from "#src/knowledge/read.ts";
 import { CODE_REGION_TYPES, DEFAULT_SPACE, type Space } from "#src/knowledge/schema.ts";
-import { UsageError } from "#src/script.ts";
 
 const FULL_MAP = FULL_COVERAGE;
-/** Instructions after which the next instruction is not reached by falling through. */
-const FLOW_ENDS = new Set([0x00, 0x40, 0x4c, 0x60, 0x6c]);
 const ANALYZER_ORIGINS = new Set(["dxa", "ghidra", "heuristic"]);
 
-/** Reads coverage files, one map per file (a run). */
-export function readCoverageMaps(paths: string[]): Uint8Array[] {
-  return readMaps(paths, (message) => {
-    throw new UsageError(message);
-  });
-}
-
 /** The byte at an address of the analyzed image, or undefined outside it. */
 export type ImageByte = (address: number) => number | undefined;
 
@@ -31,22 +23,24 @@ export function imageReader(bytes: Uint8Array, kind: "prg" | "flat64k"): ImageBy
   return (address) => (address >= load && address - load + 2 < bytes.length ? bytes[address - load + 2] : undefined);
 }
 
+const ROM_ACCESS = COVERAGE.romRead | COVERAGE.romExecute;
+const RAM_USE = COVERAGE.ramRead | COVERAGE.ramExecute;
+
 /**
- * The executed instruction starts in RAM that begin a flow: the first after
- * a gap, or the first after an instruction that does not fall through. The
- * analyzers follow the rest of a flow themselves.
+ * The image bytes under ROM that the runs never used: the CPU read or ran the
+ * ROM there, and never read or ran the RAM. A JSR $FFE4 or an IRQ
+ * vector at $FFFE then leads into ROM, not into the leftover RAM bytes of a
+ * flat image, so the ranges are data for the analysis run.
  */
-export function coverageEntries(map: Uint8Array, image: ImageByte): number[] {
-  const entries: number[] = [];
-  let falls: number | undefined;
+export function romOnlyRanges(map: Uint8Array, image: ImageByte): Array<{ start: number; end: number }> {
+  const ranges: Array<{ start: number; end: number }> = [];
   for (let address = 0; address < FULL_MAP; address++) {
-    if ((map[address]! & COVERAGE.ramExecute) === 0) continue;
-    const opcode = image(address);
-    if (opcode === undefined) continue;
-    if (falls !== address) entries.push(address);
-    falls = FLOW_ENDS.has(opcode) ? undefined : address + instructionLength(opcode);
+    if ((map[address]! & ROM_ACCESS) === 0 || (map[address]! & RAM_USE) !== 0 || image(address) === undefined) continue;
+    const last = ranges.at(-1);
+    if (last !== undefined && last.end === address - 1) last.end = address;
+    else ranges.push({ start: address, end: address });
   }
-  return entries;
+  return ranges;
 }
 
 /** Entry points from semantic knowledge beyond routines: the start of each code region, and each entry of a JMP table with its target. */

package/skills/c64-static-analysis/scripts/{findings.ts → lib/findings.ts}

@@ -25,7 +25,7 @@ export function seedsFromKnowledge(db: DatabaseSync | undefined, extraEntryPoint
   return {
     entryPoints,
     dataRanges: listRegions(db, undefined, undefined, undefined, space)
-      // A JMP table is code and dual-use bytes are both: neither is a data range (D42).
+      // A JMP table is code and dual-use bytes are both: neither is a data range.
       .filter((region) => semantic(region) && !CODE_REGION_TYPES.includes(region.type) && region.type !== "dual-use")
       .map((region) => ({ start: region.start, end: region.end })),
     labels: symbols.map((symbol) => ({ address: symbol.address, name: symbol.name })),
@@ -46,11 +46,15 @@ export function ghidraFindings(result: GhidraResult): NormalizedFindings {
   for (const fn of result.functions) uses.set(fn.name, (uses.get(fn.name) ?? 0) + 1);
   const nameOf = (fn: GhidraResult["functions"][number]) =>
     fn.nameSource !== "seed" && uses.get(fn.name)! > 1 ? `${fn.name}_${fn.entry.toString(16).padStart(4, "0")}` : fn.name;
+  // A function that branches and jumps reach, and no call, is a place in a routine: a label.
+  const called = new Set(result.references.filter((reference) => reference.type === "call").map((reference) => reference.to));
+  const jumped = new Set(result.references.filter((reference) => reference.type === "jump").map((reference) => reference.to));
+  const kindOf = (fn: GhidraResult["functions"][number]) => (jumped.has(fn.entry) && !called.has(fn.entry) ? ("label" as const) : ("routine" as const));
   return {
     analyzer: "ghidra",
     coverage: result.coverage,
     authoritative: { symbols: result.completeness.functions, regions: result.completeness.regions, references: result.completeness.references },
-    symbols: result.functions.map((fn) => ({ address: fn.entry, name: nameOf(fn), kind: "routine" as const })),
+    symbols: result.functions.map((fn) => ({ address: fn.entry, name: nameOf(fn), kind: kindOf(fn) })),
     regions: result.regions.map((region) => ({ start: region.start, end: region.end, type: region.classification === "code" ? ("code" as const) : ("bytes" as const) })),
     references: result.references.map((reference) => ({ from: reference.from, to: reference.to, kind: reference.type })),
   };