@portll/cobolwork 0.5.0 → 0.7.0

package/README.md

@@ -54,7 +54,7 @@ request and writes SARIF for code scanning: [docs/github-action.md](docs/github-
 
 | Area | What is reported |
 |---|---|
-| Data flow | untrusted input - the command line, a job's `PARM` or in-stream data, a CICS terminal or web request - reaching an OS command, dynamic SQL, a dynamic `CALL`, `LINK` or `XCTL`, a file name, the internal reader, a subscript or length, decimal arithmetic, a record key or a log, across programs |
+| Data flow | untrusted input - the command line, a job's `PARM` or in-stream data, a CICS terminal or web request, a CICS queue item - reaching an OS command, dynamic SQL, a dynamic `CALL`, `LINK` or `XCTL`, a file name, the internal reader, a subscript or length, decimal arithmetic, a record key or a log, across programs |
 | Checks | a check counts only where it runs first, on every route; one that leaves the value safe for the sink clears the route |
 | Screen fields | a field the BMS map protects, read back and used to choose a record: the 3270 hidden form field |
 | Exfiltration | database rows and file records leaving through web calls, sockets, MQ, extrapartition queues or service calls |
@@ -63,13 +63,21 @@ request and writes SARIF for code scanning: [docs/github-action.md](docs/github-
 | JCL | credentials and security commands in in-stream data, destructive statements, `DLM=` tricks, FTP in cleartext or sending production data, production data touched by a test job |
 | The source | names nothing declares (code that cannot compile), shadowed copybooks, payloads hidden in columns 73-80 or aimed at AI readers |
 | The estate | production names outside production jobs, routable addresses, compiler and runtime versions with published advisories |
-| Assembler | a switch to key zero or supervisor state, an instruction run through `EX`, cross-memory calls, the security product called directly, and the CSECT or ENTRY a COBOL `CALL` or a job step reaches |
+| Assembler | a switch to key zero or supervisor state, an instruction run through `EX`, cross-memory calls, a module named at run time, the security product called directly, and the CSECT or ENTRY a COBOL `CALL` or a job step reaches |
+| IMS and Db2 | a PSB granting a program an option its own DL/I calls never use, and a segment a DL/I get retrieves reaching a sink; a PSB letting a program change every segment it reads, a SENSEG naming no segment of its DBD, a `KEYLEN` or field that DBDGEN, PSBGEN or ACBGEN refuses; a GRANT to `PUBLIC`, `WITH GRANT OPTION` or of a system authority, and the exit routines and load modules that run inside Db2 |
+| PL/I | input reaching a sink through a PL/I program, as in COBOL; a `FETCH` of a module named at run time, an `ON` unit that ignores its condition, and where pointers, entry variables or the preprocessor take the flow out of sight |
 | Cryptography | a single-length DES key, an MD5 or SHA-1 hash, or a fixed initialization vector asked of ICSF, read from IBM's parameter lists; an outbound CICS connection asking for HTTP |
-
-HLASM is read, not assembled: its statements are read from the cards and its operations looked up
-in a table that cites the IBM manual for each (`rules/hlasm-operations.json`). Macros are not
-expanded and conditional assembly is not evaluated, so an operation a site macro issues is seen
-where the macro is defined, and not where it is used.
+| Secrets | a credential written into a program or copybook: a literal `VALUE` on an item named for one, or a literal password in `EXEC SQL CONNECT` or `EXEC CICS SIGNON` |
+
+HLASM is read, not assembled. Each statement is parsed by what its operation is: a machine
+instruction against IBM's operand syntax for its mnemonic (`rules/hlasm-instructions.json`, every
+z/Architecture mnemonic with its page in the Principles of Operation), an assembler instruction,
+or a z/OS, Language Environment or HLASM Toolkit macro against the keywords IBM lists. The location
+and length of each symbol is worked out as the assembler would, and nothing after a statement of
+unknown length is placed. Macros are not expanded and conditional assembly is not evaluated, so an
+operation a site macro issues is seen where the macro is defined, and not where it is used. A scan
+names each macro it did not expand, each COPY member the tree does not hold, and each statement it
+could not read.
 
 [docs/rule-sets.md](docs/rule-sets.md) describes each in full, with what it deliberately leaves out.
 Vendor packs for CA ACF2 and Top Secret, Control-M and Connect:Direct load only for estates that
@@ -127,7 +135,10 @@ language, are not in it.
 and the paragraph or section it sits in (the job, step and DD for JCL), and the flagged statement's
 own text. No line number goes into it, so code added above a finding does not change it. `diff`
 compares findings by it, and SARIF carries it as `partialFingerprints["cobolwork/v1"]`. Two findings
-that only their position tells apart share one, and `summary.identity.shared` counts them.
+that only their position tells apart share one, and `summary.identity.shared` counts them. What goes
+into a fingerprint changes only in a major release, under a new version. [STABILITY.md](STABILITY.md)
+lists that and the other contracts a release keeps. `schema/cobolwork-finding.schema.json` describes
+a finding, and `schema/` holds a schema for every document cobolwork writes or reads.
 
 ### Findings and Claim Severity
 
@@ -215,8 +226,13 @@ entered ([docs/spec/evidence.md](docs/spec/evidence.md) §13.5).
 
 `COBOLWORK_ABENDS` names fuzz runs ironwork's harness wrote. Each abend an input caused becomes a
 finding at the line it happened, with the input and the run's journal, once that journal verifies:
-`input-causes-abend-s0c7`, `-s0c4`, `-subscript-range`, or `input-causes-abend` for any other code
-([docs/spec/evidence.md](docs/spec/evidence.md) §13.6).
+`input-causes-abend-s0c7`, `-s0c4` (in a CICS task, the ASRA whose message names that check),
+`-subscript-range`, `input-causes-hang` for an input that keeps a loop running past the statement limit (S322), `input-selects-program`
+for a CALL of a missing program (U4038 with CEE3501S, in a CICS task transaction abend 4038, or S806 from earlier ironwork releases) whose journal shows the input reaching the CALL, or `input-causes-abend` for any other
+code ([docs/spec/evidence.md](docs/spec/evidence.md) §13.6). It reads manifests in format
+`ironwork-fuzz/v1`, and `ironwork-fuzz-interface/v1` from a subprogram fuzzed with generated
+arguments, whose findings say no caller run shows a caller passes them and warn by default; it
+reports one in any other format as a problem instead of guessing at it.
 
 ### What each finding lets someone do, and the fix
 
@@ -285,7 +301,10 @@ clean result.
 Some rules need facts no repository holds: which dataset qualifiers are production, which DDs reach
 the internal reader, the compiler options and runtime versions in use, which libraries are
 authorised. They go in `cobolwork.site.json`, and `node diag/propose-site.mjs <path>` drafts one from
-the estate's own JCL for a person to correct. Without a fact, the rule that needs it says it did not
+the estate's own JCL for a person to correct. `schema/cobolwork.site.schema.json` describes the
+file. Its `version` is 1, and a file without one is read as version 0, which holds the same keys. A
+key cobolwork does not read is named under `summary.siteWarnings` rather than dropped, and a key
+opening with `_` is a note for people. Without a fact, the rule that needs it says it did not
 run, under `setsIncomplete`, rather than reporting a clean result. The benchmark tree has no site
 file, which is why three sets say so above. `advisoryCoverage` names the products the advisory rules
 searched, so a scan with no advisory finding says what that silence covers.
@@ -329,26 +348,65 @@ The parser is graded against GnuCOBOL's own listing (`cobc -t -Xref -ftsymbols`)
 data item with the size the compiler computed, every label, called programs, and which references
 write to a field. `diag/grade-against-gnucobol.mjs` runs that comparison over a corpus, on
 repositories never used while building the parser. The 100 and 300 sets were measured 2026-09-18,
-the 500 set 2026-09-26:
+the 500 set 2026-10-04:
 
 | Corpus | Files the compiler accepted | Data items | Sizes | Labels and calls |
 |---|---|---|---|---|
 | 100 repositories, held out | 489 | 100% recall, 100% precision | 0 disagree of 15,888 | 100% |
 | 300 repositories, held out | 2,210 | 99.9% / 100% | 8 disagree of 94,786 | 100% / 99.7% |
-| 500 repositories, held out | 21,624 | 99.98% / 99.998% | 5,881 disagree of 733,835 | 100% / 100% |
+| 500 repositories, held out | 21,707 | 100% / 100% | 0 disagree of 719,184 | 100% / 100% |
 
-On the 500 set, 5,643 of the 5,881 size disagreements come from one repository that vendors a COBOL
-research dataset; the other repositories disagree on 238 of 343,444. The grade covers only programs
-GnuCOBOL accepts, so a program with EXEC SQL or EXEC CICS is graded only through the precompiler
-stand-in in `diag/precompiler.mjs`, which rewrites what the parser would otherwise have to read. The
-tests compare the parser with the compiler's answers kept in `test/fixtures/parser/*.golden.json`,
+On the 500 set every data item, size and call agrees with the compiler: 845,704 items, 27,703
+calls. Of 146,004 labels one differs, an `EJECT` written in Area A, which GnuCOBOL reads as a
+paragraph and IBM as the listing directive it is. The grade covers only programs GnuCOBOL accepts, so a program with EXEC SQL or EXEC CICS is graded only
+through the precompiler stand-in in `diag/precompiler.mjs`, which rewrites what the parser would
+otherwise have to read. The tests compare the parser with the compiler's answers kept in `test/fixtures/parser/*.golden.json`,
 so they run without GnuCOBOL.
 
-`bench/cases/` holds 102 CWE-labelled cases, each paired with a near-miss negative: the same shape
+HLASM is graded against z390, an HLASM-compatible assembler, by `diag/hlasm-oracle.mjs` and
+`diag/hlasm-grade-z390.mjs`. Each file is assembled twice, the second time with sixteen bytes after
+every macro call, and only values the two runs agree on are graded. None of them depends on how
+z390's macros, or IBM's, expand. Measured 2026-10-03 over the files z390 assembles:
+
+| Corpus | Files | Symbol locations | Statement locations | Instruction lengths | External names |
+|---|---|---|---|---|---|
+| 12 repositories, dev | 48 | 935 agree of 936, 0 differ | 2,511 of 2,546, 0 differ | 2,273 of 2,273 | 55 of 55 |
+| 17 repositories, held out | 76 | 546 agree of 557, 0 differ | 1,631 of 1,675, 0 differ | 1,710 of 1,710 | 90 of 92 |
+
+A value not agreeing was not placed, never placed differently: it follows a statement of unknown
+length. Two of z390's values are not graded, because it differs from the Language Reference, which
+the reader follows: an EQU's length, where z390 gives 1 and the reference the length of the
+leftmost term; and anything after a literal pool where z390 pads a literal that the reference packs.
+A file with an error that can move a location is not graded either, such as an undefined symbol in
+a statement holding a literal, which z390 then leaves out of the pool. Of the statements outside conditional
+assembly, 99.5% parse on the dev corpus and 99.8% on the held-out one; the rest are counted by
+kind. `bench/hlasm-locate/` holds small programs, one assembler feature each, with z390's answers
+recorded beside them, so the tests check the locator without z390.
+
+PL/I has no compiler that may grade it, so its reader is measured by how much of a corpus it parses
+(`diag/pli-measure.mjs`, over `.pli`, `.pl1` and `.plx` files and `.inc` and `.pcx` include
+members), counted by statement kind, with every statement it does not parse named.
+The 500 set was used while building the reader. One of its repositories, `jcf608_PacificNationalBank`,
+is generated code and is counted apart. Measured 2026-10-07:
+
+| Corpus | Repositories | Files | Statements | Parsed |
+|---|---|---|---|---|
+| 500 repositories, dev, written by hand | 9 | 164 | 9,547 | 98.8% |
+| 500 repositories, dev, generated | 1 | 901 | 263,531 | 99.9% |
+| 200 repositories, held out | 3 | 31 | 1,768 | 99.8% |
+| 300 repositories, held out | 6 | 66 | 3,686 | 99.8% |
+
+Two repositories parse nothing, because the members measured there are `.inc` files that are not
+PL/I (three statements each). `jarora8_GitPlay` parses 89%, and most of the rest of
+the dev set's misses are its statements. Structure offsets are checked against hand-built fixtures
+and the Language Reference's own example, not against a compiler.
+
+`bench/cases/` holds 153 CWE-labelled cases, each paired with a near-miss negative: the same shape
 with the flaw removed. `node bench/run.mjs` scores any scanner's findings against them, by rule and
 file, never by line, and `npm test` fails if any case scores differently from its declaration.
-`--validate` compiles every COBOL case with GnuCOBOL and checks every JCL case against the
-statement grammar, a weaker witness, and says so.
+`--validate` compiles every COBOL case with GnuCOBOL, assembles every HLASM case with z390 when
+`Z390` names a release (stubbing the macros z390 does not ship), and checks every JCL case against
+the statement grammar, a weaker witness, and says so.
 
 ## Coverage on a busy machine
 
@@ -375,10 +433,10 @@ and to a COBIT 2019 practice by identifier alone:
 
 | File | Instrument | Rules mapped |
 |---|---|---|
-| `rules/compliance-dora.json` | Regulation (EU) 2022/2554 (DORA) | 207 |
-| `rules/compliance-ffiec.json` | FFIEC IT Examination Handbook | 205, and 2 recorded as unmapped |
-| `rules/compliance-nist80053.json` | NIST SP 800-53 Rev. 5.2.0 | 205, and 2 recorded as unmapped |
-| `rules/compliance-cobit2019.json` | COBIT 2019 (ISACA), identifiers only | 205, and 2 recorded as unmapped |
+| `rules/compliance-dora.json` | Regulation (EU) 2022/2554 (DORA) | 249 |
+| `rules/compliance-ffiec.json` | FFIEC IT Examination Handbook | 247, and 2 recorded as unmapped |
+| `rules/compliance-nist80053.json` | NIST SP 800-53 Rev. 5.2.0 | 247, and 2 recorded as unmapped |
+| `rules/compliance-cobit2019.json` | COBIT 2019 (ISACA), identifiers only | 247, and 2 recorded as unmapped |
 
 A COBIT 2019 row names the objective or practice and gives this project's own rationale; no ISACA
 text is reproduced, so reading what a practice says needs a copy of the framework. The practice is
@@ -386,12 +444,17 @@ chosen at the NIST control, through a crosswalk in `diag/map-compliance.mjs`, so
 from the rule's NIST clause.
 
 `node diag/map-compliance.mjs` refuses to write a quote the cached instrument does not contain, and
-every scan carries the mapping as `ruleCompliance`. The clause choice is a judgement that no
+every scan carries the mapping as `ruleCompliance`, and SARIF on each rule descriptor as `properties.compliance`. The clause choice is a judgement that no
 qualified assessor has reviewed. Where no control genuinely covers a rule, as for committing an LPAR
 name to a repository, the rule is recorded as unmapped with a reason rather than mapped to the
 nearest control that reads plausibly. What else the mapping does not claim is in
 [docs/rule-sets.md](docs/rule-sets.md#compliance).
 
+The same files map the evidence cobolwork and ironwork write: a run journal's records, its chain,
+the provenance statement and ironwork's options in force, each to its NIST control and COBIT
+practice, and `evidence verify` names the clauses of what it checked
+([docs/spec/evidence.md](docs/spec/evidence.md) §9).
+
 ## Tests
 
     npm test

package/STABILITY.md

@@ -0,0 +1,76 @@
+# Stability
+
+What a cobolwork release keeps from the release before it.
+
+From 0.9.0, the contracts below change only in a major release. Until then a minor release may
+change any of them, and its release notes say so.
+
+## The contracts
+
+- **Rule ids.** Each rule a rule set declares, as `cobolwork capabilities` and the SARIF tool
+  components list them.
+- **The command line.** Each command with its arguments and options, as `cobolwork capabilities`
+  lists them under `commands` and `globalOptions`.
+- **Exit statuses**, below.
+- **Documents and their schemas**, below.
+- **The fingerprint, `cobolwork/v1`.** What goes into a finding's fingerprint and pairing keys.
+  `test/identity-golden.test.mjs` pins the hashes, so a change to the inputs fails it.
+
+The JavaScript exports of `lib/` are not covered.
+
+## Exit statuses
+
+| Command | Status |
+|---|---|
+| Every command | 0 done; 2 a usage error, or the command could not run |
+| `build` | 0 pass, 1 fail, 3 undecided, 4 the compiler failed after a pass, 2 could not run |
+| `gate --exit-code` | 0 pass, 1 fail, 3 undecided, 2 could not run |
+| `evidence verify` | 0 verified and sealed, 1 broken or not sealed, 3 undetermined, 2 a usage error |
+
+## Documents
+
+Each JSON document cobolwork writes names itself in `tool` and gives its version in
+`schemaVersion`. `cobolwork capabilities` lists both under `documents`. Each schema describes every
+key its document carries at the top level, the scan, flow and diff schemas every key in `summary` and
+in each finding too, and a test holds the documents cobolwork writes to them.
+
+| Document | Version | Schema |
+|---|---|---|
+| `cobolwork` (`scan`) | 3 | `schema/cobolwork-report.schema.json`, each finding `schema/cobolwork-finding.schema.json` |
+| `cobolwork-flow` | 3 | `schema/cobolwork-flow.schema.json` |
+| `cobolwork-inventory` | 3 | `schema/cobolwork-inventory.schema.json` |
+| `cobolwork-diff` | 3 | `schema/cobolwork-diff.schema.json` |
+| `cobolwork-gate` | 3 | `schema/cobolwork-gate.schema.json` |
+| `cobolwork-build` | 1 | `schema/cobolwork-build.schema.json` |
+| `cobolwork-build-provenance` | 1 | `schema/cobolwork-build-provenance.schema.json` |
+| `cobolwork-capabilities` | 1 | `schema/cobolwork-capabilities.schema.json` |
+| `cobolwork-explain` | 1 | `schema/cobolwork-explain.schema.json` |
+| `cobolwork-parse` | 1 | `schema/cobolwork-parse.schema.json` |
+| `cobolwork-baseline` | 1 | `schema/cobolwork-baseline.schema.json` |
+| `cobolwork-evidence` | 1 | `schema/cobolwork-evidence.schema.json` |
+| SARIF | 2.1.0 | The coverage property bag: `schema/cobolwork-coverage.schema.json` |
+| CycloneDX SBOM | 1.6 | |
+| Evidence records | `cobolwork-evidence/v1` | `docs/spec/evidence.md`; record kinds in `test/fixtures/evidence/kinds.tsv` |
+
+The files cobolwork reads:
+
+| File | Version key | Schema |
+|---|---|---|
+| `cobolwork.site.json` | `version`: 1 | `schema/cobolwork.site.schema.json` |
+| Build policy | `policyVersion`: 1 | `schema/cobolwork.policy.schema.json` |
+| `cobolwork.baseline.json` | `version`: 1 | `schema/cobolwork.baseline.schema.json` |
+| Witness feed (`COBOLWORK_WITNESS`) | `version`: 1 | `schema/cobolwork-witness.schema.json` |
+| Reachability extract (`COBOLWORK_REACH`) | `version`: 1 | `schema/cobolwork-reach.schema.json` |
+| Execution coverage (`COBOLWORK_EXECUTION`) | ironwork's | `schema/cobolwork-execution.schema.json`, the keys read |
+
+A file of cobolwork's own without its version key is read as version 0, which holds the same keys
+as version 1. A newer version than the release reads is refused, and the message names both
+versions. A key the site file does not hold is named under `summary.siteWarnings`, and one a witness
+feed or reachability extract does not hold under `summary.feedWarnings`.
+
+## How a contract changes
+
+- **Additions** come in a minor release: a rule, a command, an option, or a key in a document
+  cobolwork writes. A program that reads cobolwork's documents should ignore keys it does not know.
+- **A rename** keeps the old name working as an alias for one major release, and using it warns.
+- **A deprecation** warns for at least one minor release before the major release that removes it.

package/bin/cobolwork.mjs

@@ -14,12 +14,13 @@ import { diffRefs } from '../lib/diff.mjs';
 import { gateRefs, VERDICT_EXIT } from '../lib/gate.mjs';
 import { build, buildSarif, buildSummaryLine } from '../lib/build.mjs';
 import { capabilities } from '../lib/capabilities.mjs';
+import { PARSE_SCHEMA_VERSION, BASELINE_RESULT_SCHEMA_VERSION } from '../lib/version.mjs';
 import { commitAt, revisionOf, toolRevision } from '../lib/revision.mjs';
 import { stampFingerprints } from '../lib/kernel/identity.mjs';
 import { pdsExportTree } from '../lib/kernel/source-tree.mjs';
 import { SITE_FILE } from '../lib/site.mjs';
 import { tally } from '../lib/kernel/findings.mjs';
-import { loadBaseline, applyBaseline, baselineEntries, BASELINE_FILE, SUPPRESSING } from '../lib/baseline.mjs';
+import { loadBaseline, applyBaseline, baselineEntries, BASELINE_FILE, BASELINE_VERSION, SUPPRESSING } from '../lib/baseline.mjs';
 import { readReport } from '../lib/tui/model.mjs';
 import { nodeTerminal } from '../lib/tui/terminal.mjs';
 import { runTui } from '../lib/tui/run.mjs';
@@ -28,7 +29,7 @@ import { printable } from '../lib/kernel/printable.mjs';
 import { startEvidence, recordInputs, recordHashed, recordFindings, recordOutput, recordVerdict, recordBaselineWrite, finishEvidence } from '../lib/evidence/run.mjs';
 import { evidenceCommand } from '../lib/evidence/cli.mjs';
 import { slsaStatement } from '../lib/evidence/slsa.mjs';
-import { parseAllowedSigners } from '../lib/evidence/sshsig.mjs';
+import { optableLevel } from '../lib/hlasm/optable.mjs';
 
 const STARTED = new Date().toISOString();
 
@@ -60,7 +61,7 @@ const USAGE = `cobolwork ${VERSION} — COBOL, JCL and CICS security analysis, n
                                what this cobolwork can do: commands and options, the version of every
                                document it writes, its fingerprint version, the file kinds it reads, and
                                the commit it runs from
-  cobolwork build <repo> [--base <ref>] [--policy <file>] [--provenance <file>] [--ironwork <path> | -- <compiler> <arg>...]
+  cobolwork build <repo> [--base <ref>] [--policy <file>] [--provenance <file>] [--ironwork <path> | [--precompile] -- <compiler> <arg>...]
                                the build gate: every finding ranked LOW to KNOWN-EXPLOITABLE, the
                                policy's blocking findings and compiler options checked, and the
                                compiler run only on a pass. Exits 0 pass, 1 fail, 3 undecided, 4 the
@@ -92,6 +93,13 @@ Options
   --pds-export          scan: the directory holds partitioned data sets' members as files, as
                         zowe zos-files download all-members writes them (hlq/llq/member.txt) or in a
                         directory named for each data set; findings name DATA.SET/MEMBER
+  --mvs38-forms, --no-mvs38-forms  scan: read (the default) or refuse the operands MVS 3.8's system
+                        macros take and z/OS 3.1's documentation does not list, such as MODESET
+                        EXTKEY=SUPR (key zero), GETMAIN P and ATTACH HIARCHY=
+  --hlasm-optable <table>  scan: read assembler source with this HLASM operation code table (UNI,
+                        DOS, 370, XA, ESA, ZOP, YOP, Z9 ... Z17, or a MACHINE name such as S390): a
+                        mnemonic outside it is a macro call. Without it the estate's assembly JCL
+                        PARM or a file's *PROCESS decides, and UNI otherwise
   --copylib <dir>[,<dir>]  copy libraries the estate keeps outside the repository, searched after the
                         tree's own copybooks, as COBCPY is; a copybook found there is read, not reported missing
   --report <file>       tui, explain: read a stored scan report instead of scanning
@@ -108,18 +116,32 @@ Options
                         the SLSA Provenance v1 predicate (unsigned; the pipeline signs it)
   --artifact <path>[,<path>]  build: what the compiler produced, named as subjects of the SLSA statement
   --equivalence <file>[,<file>]  build --base: ironwork equivalence statements for the programs the
-                        change edits; with --allowed-signers each must be signed by one. The policy's
-                        requireEquivalence (never, machineAuthored, always) says when one is required
+                        change edits, each signed by one of --allowed-signers. The policy's
+                        requireEquivalence (never, machineAuthored, always) says when one is required;
+                        where one may be, a build with no --allowed-signers fails the check
   --ironwork <path>     build: after a pass, run ironwork check on every program, for an estate that
                         compiles with IBM Enterprise COBOL; a program ironwork rejects exits 4, one it
                         does not model yet leaves the build undecided
+  --precompile          build -- <compiler>: after a pass, translate each program the compiler command
+                        names that holds EXEC SQL or EXEC CICS and run the compiler on it with
+                        -fsyntax-only; one it refuses exits 4 and the command after -- does not run
   --evidence <dir>      scan, flow, diff, gate, build, baseline, inventory: record this run in a
                         hash-chained journal and ledger there (or COBOLWORK_EVIDENCE); never inside
                         the tree being read
-  --allowed-signers <file>  evidence verify: OpenSSH allowed_signers for namespace cobolwork-evidence
+  --allowed-signers <file>  evidence verify and build: OpenSSH allowed_signers for namespace
+                        cobolwork-evidence; build refuses one inside the repository
   --anchor-git <repo>, --ref <ref>, --push, --max-unsealed <n>
                         evidence: the git witness, the ref that counts (default @{upstream}), whether
                         anchor pushes, and how many ledger records may follow the newest seal
+  --anchor-pin <commit>  evidence verify: the witness commit an earlier verify reported as
+                        witnessCommit; the witness must still hold every seal it held
+  --tsq <file>, --tsr <file>, --tsa-ca <file>
+                        evidence anchor writes an RFC 3161 request (and keeps it beside the seal);
+                        verify reads the response, checking imprint and nonce, and has OpenSSL check
+                        the authority's signature against the CA certificate
+  --cosign-bundle <file> with --cosign-key <file> or --certificate-identity <id> and
+  --certificate-oidc-issuer <url>, [--trusted-root <file>] [--insecure-ignore-tlog]
+                        evidence verify: a transparency-log bundle over a seal, checked by cosign
 
 Exit codes: 0 the command ran, 2 it could not run. A run that examined nothing says so in
 summary.filesScanned and summary.nosrc rather than reporting a clean zero.
@@ -158,6 +180,10 @@ function parseArgs(argv) {
     else if (a === '--rule') opts.rule = list();
     else if (a === '--advisories') opts.advisoryFeeds = [...new Set(list().map(x => resolve(x)))];
     else if (a === '--pds-export') opts.pdsExport = true;
+    else if (a === '--mvs38-forms') opts.mvs38Forms = true;
+    else if (a === '--no-mvs38-forms') opts.mvs38Forms = false;
+    else if (a === '--hlasm-optable') opts.hlasmOptable = value();
+    else if (a === '--precompile') opts.precompile = true;
     else if (a === '--copylib') opts.copylib = [...new Set(list().map(x => resolve(x)))];
     else if (a === '--report') opts.report = value();
     else if (a === '--keys') opts.keys = value();
@@ -182,6 +208,15 @@ function parseArgs(argv) {
     else if (a === '--push') opts.push = true;
     else if (a === '--max-unsealed') opts.maxUnsealed = value();
     else if (a === '--tsq') opts.tsq = value();
+    else if (a === '--anchor-pin') opts.anchorPin = value();
+    else if (a === '--tsr') opts.tsr = value();
+    else if (a === '--tsa-ca') opts.tsaCa = value();
+    else if (a === '--cosign-bundle') opts.cosignBundle = value();
+    else if (a === '--certificate-identity') opts.certificateIdentity = value();
+    else if (a === '--certificate-oidc-issuer') opts.certificateOidcIssuer = value();
+    else if (a === '--cosign-key') opts.cosignKey = value();
+    else if (a === '--trusted-root') opts.trustedRoot = value();
+    else if (a === '--insecure-ignore-tlog') opts.insecureIgnoreTlog = true;
     else if (a === '--expect-key') opts.expectKey = value();
     else if (a === '--help' || a === '-h') opts.help = true;
     else if (a === '--version' || a === '-v') opts.version = true;
@@ -275,7 +310,14 @@ if (opts.copylib && opts._.length && !COPYLIB_COMMANDS.includes(opts._[0])) {
 const notDirs = (opts.copylib || []).filter((d) => { try { return !statSync(d).isDirectory(); } catch { return true; } });
 if (notDirs.length) { process.stderr.write(`cobolwork: --copylib ${notDirs.join(', ')} is not a directory\n`); process.exit(2); }
 const systemDirs = opts.copylib || [];
-if (opts.pdsExport && opts._.length && opts._[0] !== 'scan') { process.stderr.write('cobolwork: --pds-export is for scan only\n'); process.exit(2); }
+if (opts.pdsExport && opts._.length && !['scan', 'diff', 'build'].includes(opts._[0])) { process.stderr.write('cobolwork: --pds-export is for scan, diff and build\n'); process.exit(2); }
+if (opts.hlasmOptable !== undefined) {
+  const level = optableLevel(opts.hlasmOptable) ?? optableLevel(opts.hlasmOptable, { machine: true });
+  if (!level) { process.stderr.write(`cobolwork: --hlasm-optable ${opts.hlasmOptable} names no HLASM operation code table\n`); process.exit(2); }
+  if (opts._.length && opts._[0] !== 'scan') { process.stderr.write('cobolwork: --hlasm-optable is for scan\n'); process.exit(2); }
+  opts.hlasmOptable = level;
+}
+if (opts.mvs38Forms !== undefined && opts._.length && opts._[0] !== 'scan') { process.stderr.write('cobolwork: --mvs38-forms and --no-mvs38-forms are for scan\n'); process.exit(2); }
 if (opts.pdsExport && opts.repos) { process.stderr.write('cobolwork: --pds-export reads one export; --repos does not apply\n'); process.exit(2); }
 if (opts.json && opts._.length && opts._[0] !== 'capabilities') {
   process.stderr.write(`cobolwork: --json is for capabilities; every other command writes JSON unless --format says otherwise\n`);
@@ -297,7 +339,7 @@ if (opts.evidence !== undefined && opts._.length && ![...JOURNALED, 'evidence'].
   process.stderr.write(`cobolwork: ${opts._[0]} records no evidence; ${JOURNALED.join(', ')} and evidence do\n`);
   process.exit(2);
 }
-const evidenceFlag = ['sshKey', 'signer', 'allowedSigners', 'anchorGit', 'ref', 'push', 'maxUnsealed', 'tsq', 'expectKey'].find((k) => opts[k] !== undefined);
+const evidenceFlag = ['sshKey', 'signer', 'allowedSigners', 'anchorGit', 'ref', 'push', 'maxUnsealed', 'tsq', 'expectKey', 'anchorPin', 'tsr', 'tsaCa', 'cosignBundle', 'certificateIdentity', 'certificateOidcIssuer', 'cosignKey', 'trustedRoot', 'insecureIgnoreTlog'].find((k) => opts[k] !== undefined);
 if (opts.equivalence && !opts.base) { process.stderr.write('cobolwork: --equivalence judges a change; it needs --base\n'); process.exit(2); }
 if (evidenceFlag && opts._.length && opts._[0] !== 'evidence' && !(evidenceFlag === 'allowedSigners' && opts._[0] === 'build')) {
   process.stderr.write(`cobolwork: --${evidenceFlag.replace(/[A-Z]/g, (c) => `-${c.toLowerCase()}`)} is for evidence only\n`);
@@ -346,7 +388,7 @@ try {
   } else if (command === 'evidence') {
     process.exitCode = evidenceCommand(target, opts, { toolVersion: VERSION, write: (s) => process.stdout.write(s) });
   } else if (command === 'scan' || command === 'flow') {
-    const flowOpts = { repos, fullTrace: opts.fullTrace === true, allRoutes: opts.allRoutes === true, systemDirs };
+    const flowOpts = { repos, fullTrace: opts.fullTrace === true, allRoutes: opts.allRoutes === true, systemDirs, mvs38Forms: opts.mvs38Forms !== false, hlasmOptable: opts.hlasmOptable ?? null };
     // The site file sits beside the members and is not one of them, so it is named rather than found.
     const site = resolve(root, SITE_FILE);
     const pds = opts.pdsExport ? pdsExportTree(root, { systemDirs }) : null;
@@ -371,7 +413,7 @@ try {
     warnCoverage(report);
   } else if (command === 'diff') {
     if (!opts.base) { process.stderr.write(`cobolwork: diff needs --base <ref>\n`); process.exit(2); }
-    const report = diffRefs(root, opts.base, opts.head || null, { only: opts.only, fullTrace: opts.fullTrace === true, systemDirs });
+    const report = diffRefs(root, opts.base, opts.head || null, { only: opts.only, fullTrace: opts.fullTrace === true, systemDirs, pdsExport: opts.pdsExport === true });
     stampRevisions(report.summary, opts.head || null);
     recordFindings(journal, { findings: [...report.findings, ...(report.introduced || [])] });
     if (opts.format === 'sarif') emit(toSarif({ ...report, findings: [...report.findings, ...report.introduced] }, { toolVersion: VERSION }), opts);
@@ -393,7 +435,7 @@ try {
     if (opts.only || opts.repos) { process.stderr.write('cobolwork: build judges one repository with every rule set; --only and --repos do not apply\n'); process.exit(2); }
     if (opts.baseline) { process.stderr.write('cobolwork: build reads the baseline the change was written against; --baseline does not apply, --no-baseline does\n'); process.exit(2); }
     if (opts.head && !opts.base) { process.stderr.write('cobolwork: build --head needs --base\n'); process.exit(2); }
-    const result = build(root, { base: opts.base || null, head: opts.head || null, policy: opts.policy || null, noBaseline: opts.noBaseline === true, compiler: compilerArgv, ironwork: opts.ironwork || null, advisoryFeeds: opts.advisoryFeeds || null, copylibs: systemDirs, equivalence: opts.equivalence || [], allowed: opts.allowedSigners ? parseAllowedSigners(readFileSync(resolve(opts.allowedSigners), 'utf8')) : null });
+    const result = build(root, { base: opts.base || null, head: opts.head || null, policy: opts.policy || null, noBaseline: opts.noBaseline === true, compiler: compilerArgv, ironwork: opts.ironwork || null, advisoryFeeds: opts.advisoryFeeds || null, copylibs: systemDirs, equivalence: opts.equivalence || [], allowedSigners: opts.allowedSigners || null, pdsExport: opts.pdsExport === true, precompile: opts.precompile === true });
     stampRevisions(result.doc.summary, opts.head || null);
     Object.assign(result.report.summary, { toolRevision: result.doc.summary.toolRevision, revision: result.doc.summary.revision });
     Object.assign(result.provenance, { toolRevision: result.doc.summary.toolRevision, revision: result.doc.summary.revision });
@@ -436,12 +478,12 @@ try {
     if (held.problems.length) { process.stderr.write(`cobolwork: ${path} holds entries that do not validate, so it is left as it is:\n  ${held.problems.join('\n  ')}\n`); process.exit(2); }
     const report = scanAll(root, { repos, only: opts.only, advisoryFeeds: opts.advisoryFeeds, systemDirs });
     const { entries, added } = baselineEntries(report.findings, held.entries, { action, reason: opts.reason, who: opts.who, expires: expires.toISOString(), at, rules: opts.rule || null });
-    const written = JSON.stringify({ _comment: 'Judgements over cobolwork findings, matched by fingerprint. A suppression lapses at its expires date and the finding comes back.', entries }, null, 1) + '\n';
+    const written = JSON.stringify({ _comment: 'Judgements over cobolwork findings, matched by fingerprint. A suppression lapses at its expires date and the finding comes back.', version: BASELINE_VERSION, entries }, null, 1) + '\n';
     writeFileSync(path, written);
     recordInputs(journal, 0, root);
     recordFindings(journal, report);
     recordBaselineWrite(journal, { before: held.entries, after: entries, who: opts.who, expires: expires.toISOString(), reason: opts.reason, path, text: written });
-    const line = `${JSON.stringify({ tool: 'cobolwork-baseline', path, added, entries: entries.length })}\n`;
+    const line = `${JSON.stringify({ tool: 'cobolwork-baseline', schemaVersion: BASELINE_RESULT_SCHEMA_VERSION, path, added, entries: entries.length })}\n`;
     recordOutput(journal, 'baseline', line);
     process.stdout.write(line);
   } else if (command === 'tui') {
@@ -475,7 +517,7 @@ try {
   } else if (command === 'parse') {
     const r = parseFile(root, { format: 'auto' });
     emit({
-      tool: 'cobolwork-parse', file: r.file, format: r.format, finalFormat: r.finalFormat,
+      tool: 'cobolwork-parse', schemaVersion: PARSE_SCHEMA_VERSION, file: r.file, format: r.format, finalFormat: r.finalFormat,
       programs: r.programs.map(p => ({ id: p.id, items: p.items.length, labels: p.labels.length, calls: p.calls.length, execs: p.execs.length, diagnostics: p.diags.length })),
       copies: r.copies.map(c => ({ name: c.name, status: c.status })),
       diagnostics: r.diags.length,

package/lib/baseline.mjs

@@ -22,6 +22,7 @@ import { resolvesInside } from './kernel/source-tree.mjs';
 import { printable } from './kernel/printable.mjs';
 
 export const BASELINE_FILE = 'cobolwork.baseline.json';
+export const BASELINE_VERSION = 1;
 export const SUPPRESSING = ['accept', 'false-positive', 'wont-fix', 'incorrect-scan-result'];
 export const ACTIONS = [...SUPPRESSING, 'note', 'resolved'];
 
@@ -75,6 +76,14 @@ export function loadBaseline(root, { explicit = null, use = true, mayBeAbsent =
     out.problems.push(`${path} is not readable as JSON: ${printable(err.message, 120)}`);
     return out;
   }
+  // A file with no version predates versioning and is read as version 0, which holds the same entries.
+  const version = raw && typeof raw === 'object' && raw.version !== undefined ? raw.version : 0;
+  if (!Number.isInteger(version) || version < 0) { out.problems.push(`${path} has version ${printable(JSON.stringify(version), 40)}, which is not a whole number from 0`); return out; }
+  if (version > BASELINE_VERSION) {
+    out.problems.push(`${path} is baseline version ${version}, and this cobolwork reads up to version ${BASELINE_VERSION}; upgrade cobolwork, so nothing was suppressed`);
+    return out;
+  }
+  out.version = BASELINE_VERSION;
   const entries = Array.isArray(raw?.entries) ? raw.entries : null;
   if (!entries) { out.problems.push(`${path} has no entries array`); return out; }
   entries.forEach((e, i) => {

package/lib/bms.mjs

@@ -56,11 +56,11 @@ const sublist = (v) => {
 };
 
 // L'NAME is NAME's length, and T', S', I', K', N', D' and O' its other attributes: the quote after the
-// letter opens no string when the letter starts a term and a symbol follows. CL8'TEXT' and D'1.5'
-// are constants, and their quotes do open one.
+// letter opens no string when the letter starts a term and a symbol, *, or a literal follows (L'*,
+// L'=F'1'). CL8'TEXT' and D'1.5' are constants, and their quotes do open one.
 // https://www.ibm.com/docs/en/hla-and-tf/1.6.0?topic=instructions-data-attributes
 const attributeReference = (text, i) => /[LTSIKNDO]/i.test(text[i - 1] || '')
-  && !/[A-Z0-9$#@_]/i.test(text[i - 2] || '') && /[A-Z$#@_&]/i.test(text[i + 1] || '');
+  && !/[A-Z0-9$#@_]/i.test(text[i - 2] || '') && /[A-Z$#@_&*=]/i.test(text[i + 1] || '');
 
 // Folds physical lines into statements. A map continues a statement in two ways, often both in one
 // file. An operand that fills the line to column 71 carries on in column 16 of the next line, which
@@ -70,12 +70,15 @@ const attributeReference = (text, i) => /[LTSIKNDO]/i.test(text[i - 1] || '')
 // assembler, so it is reported rather than read.
 // https://www.ibm.com/docs/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.asma400/cl.htm
 // https://www.ibm.com/docs/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.asma400/altwmac.htm
-export function foldStatements(src) {
+// batch: read on past END, as HLASM's BATCH option assembles the next program, and keep JCL, SMP/E and
+// IEBUPDTE control cards and an end-of-file mark, which no assembler statement starts with, as
+// not-assembler lines.
+export function foldStatements(src, { batch = false } = {}) {
   const phys = String(src).replace(/\r\n?/g, '\n').split('\n');
   const statements = [];
   const diags = [];
   let open = null;       // { st, reading, quoted }: the statement the next line continues
-  let ended = false;     // the assembler reads nothing after END
+  let ended = false;     // nothing after END is read, unless batch
 
   const remark = (text, line, why) => {
     if (/^[A-Z][A-Z0-9]*=/i.test(text)) diags.push({ sev: 'warn', line, text: `'${text}' ${why}, so the assembler reads it as a remark, not an operand` });
@@ -85,7 +88,8 @@ export function foldStatements(src) {
     for (let i = from; i < text.length; i++) {
       const c = text[i];
       if (o.quoted) { o.st.field += c; if (c === "'") o.quoted = false; continue; }
-      if (c === "'" && attributeReference(text, i)) { o.st.field += c; continue; }
+      // The letter of an attribute reference can end the line before: L in column 71, its quote in column 16.
+      if (c === "'" && attributeReference(o.st.field + text.slice(i), o.st.field.length)) { o.st.field += c; continue; }
       if (c === "'") { o.quoted = true; o.st.field += c; continue; }
       if (c === ' ' || c === '\t') {
         o.reading = o.st.field.endsWith(',');
@@ -105,6 +109,7 @@ export function foldStatements(src) {
   for (let i = 0; i < phys.length && !ended; i++) {
     const line = i + 1;
     const raw = phys[i];
+    let broke = null;
     if (raw.length > 80) diags.push({ sev: 'warn', line, text: 'line is longer than 80 columns' });
     const text = raw.slice(0, END_COLUMN);
     const continues = raw.length > END_COLUMN && raw[END_COLUMN] !== ' ';
@@ -113,6 +118,7 @@ export function foldStatements(src) {
       if (/\S/.test(text.slice(0, CONTINUE_COLUMN - 1))) {
         // A comment banner that runs into column 72 is common and harmless; a statement is not.
         if (open.st.kind === 'statement') diags.push({ sev: 'error', line, text: `a continuation must leave columns 1 to ${CONTINUE_COLUMN - 1} blank, so this line was read as a new statement` });
+        broke = open.st.kind;
         finish(open);
         open = null;
       } else {
@@ -130,6 +136,10 @@ export function foldStatements(src) {
     }
 
     if (!text.trim()) continue;
+    if (batch && /^(?:\/\/|\/\*|\+\+|\.\/|\x1a)/.test(text)) {
+      statements.push({ kind: 'not-assembler', line, endLine: line, lines: [line], text });
+      continue;
+    }
     if (/^\*|^\.\*/.test(text)) {
       const st = { kind: 'comment', line, endLine: line, lines: [line], text: text.slice(1) };
       if (continues) open = { st, reading: false, quoted: false };
@@ -143,6 +153,10 @@ export function foldStatements(src) {
       continue;
     }
     const st = { kind: 'statement', name: m[1] || null, operation: m[2].toUpperCase(), field: '', line, endLine: line, lines: [line] };
+    // The line before continued, so the assembler reads this one as part of it: as comment text
+    // after a comment, as an invalid continuation after a statement. It is read as a statement
+    // all the same, and marked.
+    if (broke) st.continues = broke;
     const o = { st, reading: true, quoted: false };
     let at = m[0].length;
     while (text[at] === ' ' || text[at] === '\t') at++;
@@ -150,7 +164,7 @@ export function foldStatements(src) {
     if (at < text.length) read(o, text, at, line);
     if (continues) open = o;
     else finish(o);
-    if (st.operation === 'END') ended = true;
+    if (st.operation === 'END') ended = !batch;
   }
 
   if (open) {