glamsterdam-compat-lab 0.2.2 → 0.3.1

package/CONTRIBUTING.md

@@ -22,7 +22,7 @@ pnpm test:update
 
 Releases are published from semver tags. After CI is green on `main`, create the GitHub release tag, then run the manual `Publish npm` workflow from `main` with the release tag as `release_tag`.
 
-Start with `dry_run=true`. For a real publish, prefer npm Trusted Publishing: configure `glamsterdam-compat-lab` on npm with the GitHub repository `CruzMolina/glamsterdam-compat-lab` and workflow file `npm-publish.yml`, then rerun the workflow with `dry_run=false`. If Trusted Publishing is not available yet, configure the repository `NPM_TOKEN` secret with an npm token that can publish `glamsterdam-compat-lab`.
+Start with `dry_run=true` for a new, unpublished release version. Real publishes use npm Trusted Publishing for `glamsterdam-compat-lab` with repository `CruzMolina/glamsterdam-compat-lab`, workflow file `npm-publish.yml`, and environment `npm-publish`. Do not configure `NPM_TOKEN` for the normal release path; use a short-lived token only as an emergency fallback, remove the GitHub secret immediately afterward, and revoke the npm token.
 
 The workflow checks out the requested semver tag, verifies that `package.json` matches the tag, installs dependencies, runs tests, builds, and then runs `npm publish --provenance`.
 

package/README.md

@@ -33,13 +33,20 @@ pnpm glamsterdam eips
 pnpm glamsterdam scan-bytecode fixtures/bytecode/storage-heavy.hex
 ```
 
-The package is configured as `glamsterdam-compat-lab` in `package.json`, but npm registry publication is still pending package-owner authentication. Until npm shows `glamsterdam-compat-lab@0.2.2`, the current installable release artifact is the v0.2.2 GitHub release tarball:
+Install the published CLI from npm:
 
 ```sh
-npm install -g https://github.com/CruzMolina/glamsterdam-compat-lab/releases/download/v0.2.2/glamsterdam-compat-lab-0.2.2.tgz
+npm install -g glamsterdam-compat-lab@0.3.1
+glamsterdam eips
 ```
 
-See [docs/release.md](docs/release.md) for npm publication status and maintainer release checks.
+The v0.3.1 GitHub release tarball remains available as a reproducible release artifact:
+
+```sh
+npm install -g https://github.com/CruzMolina/glamsterdam-compat-lab/releases/download/v0.3.1/glamsterdam-compat-lab-0.3.1.tgz
+```
+
+See [docs/release.md](docs/release.md) for maintainer release checks and npm publishing notes.
 
 The default output format is Markdown. Use `--format json` for machine-readable reports.
 
@@ -53,10 +60,26 @@ ETH_RPC_URL=https://your-execution-rpc.example pnpm glamsterdam scan-tx --tx 0x0
 pnpm glamsterdam scan-indexer fixtures/indexers/subgraph.yaml --format markdown
 pnpm glamsterdam scan-validator --config fixtures/validator/operator-config.yaml --format markdown
 pnpm glamsterdam report report-a.json report-b.json --format markdown
+pnpm glamsterdam compare-reports baseline-report.json candidate-report.json --format markdown
 ```
 
 Each scanner accepts `--registry <path>` and `--thresholds <path>` so EIP metadata and detector thresholds can be updated without editing detector code.
 
+## Examples
+
+- [Storage-heavy bytecode report](examples/storage-heavy-bytecode.md)
+- [Baseline comparison reports](examples/baseline-comparison.md)
+
+## Public dataset seed
+
+Fixture provenance lives in [fixtures/provenance.json](fixtures/provenance.json). The first deterministic dataset seed lives in [datasets/public-seed](datasets/public-seed) and includes generated JSON reports plus default-vs-research threshold comparisons for bytecode and trace fixtures.
+
+Regenerate it with:
+
+```sh
+pnpm dataset:generate
+```
+
 ## What the scanners can detect
 
 `scan-bytecode` normalizes EVM bytecode, disassembles opcodes while skipping PUSH data, counts relevant opcodes, and reports conservative risks around contract size, storage/account access, CREATE/CREATE2 usage, calldata copying, logs, and manual-review limits.
@@ -100,13 +123,15 @@ Use `--trace-out <path>` to save the fetched JSON-RPC trace response while also
 
 `scan-validator` parses JSON and YAML operator configs. It checks for execution, consensus, validator, builder/API, monitoring, and testnet/devnet metadata. It compares client names and versions against `data/client-compat/clients.example.json` or a user-provided matrix, but it does not guess compatibility.
 
+`compare-reports` accepts two saved JSON compatibility reports and emits deterministic JSON or Markdown deltas. It compares findings by stable finding ID, reports findings added, removed, changed, and unchanged, and highlights severity and confidence changes. It does not invent exact gas deltas; those must come from explicit input data or future client outputs.
+
 ## Report model
 
 Each scanner returns a `CompatibilityReport`:
 
 ```json
 {
-  "toolVersion": "0.2.2",
+  "toolVersion": "0.3.1",
   "fork": "glamsterdam",
   "target": {
     "kind": "bytecode",
@@ -139,6 +164,20 @@ Confidence means:
 - `medium`: strong heuristic
 - `low`: weak heuristic or incomplete input
 
+Comparison reports include baseline and candidate report references, risk and finding-count deltas, added/removed/changed/unchanged finding lists, and comparison assumptions and limitations. This supports workflows such as comparing default, research, and CI threshold-profile outputs:
+
+```sh
+pnpm glamsterdam scan-traces fixtures/traces/storage-heavy-trace.json \
+  --thresholds data/detectors/thresholds.json \
+  --format json > default-report.json
+
+pnpm glamsterdam scan-traces fixtures/traces/storage-heavy-trace.json \
+  --thresholds data/detectors/thresholds.research.json \
+  --format json > research-report.json
+
+pnpm glamsterdam compare-reports default-report.json research-report.json --format markdown
+```
+
 ## Updating the EIP registry
 
 Edit `data/eips/glamsterdam.json`.
@@ -180,7 +219,7 @@ Release publishing notes live in [docs/release.md](docs/release.md).
 
 ## Roadmap
 
-See [ROADMAP.md](ROADMAP.md) for planned phases. Phase 0 is released as `v0.1.0`; `v0.2.0` starts Phase 1 with RPC transaction trace ingestion and broader trace fixture coverage.
+See [ROADMAP.md](ROADMAP.md) for planned phases. Phase 0 is released as `v0.1.0`; `v0.2.0` starts Phase 1 with RPC transaction trace ingestion and broader trace fixture coverage; `v0.3.0` adds baseline comparison reports; `v0.3.1` expands public-safe fixture and dataset coverage.
 
 ## Disclaimer
 

package/ROADMAP.md

@@ -44,15 +44,21 @@ Target release: `v0.3.0`.
 
 Goal: compare compatibility reports across profiles and, later, across current-client and Glamsterdam-aware traces.
 
+- Add `compare-reports` for deterministic JSON and Markdown report comparisons
 - Compare one trace against multiple threshold profiles
-- Emit report deltas for findings added, removed, or changed in severity
-- Keep comparisons deterministic and JSON-friendly
+- Emit report deltas for findings added, removed, or changed in severity or confidence
+- Keep comparisons deterministic and JSON-friendly with golden fixtures and snapshots
 - Defer fork-specific gas deltas until they are present in explicit data files or client configs
 
 ## Phase 2: Public Dataset
 
+Status: seeded after `v0.3.0`; expanded with public-safe trace, indexer, and validator fixture coverage in `v0.3.1`.
+
 Goal: publish reproducible compatibility research.
 
+- Track fixture provenance, source type, redaction posture, expected scanner signals, and known metadata gaps
+- Generate a deterministic public seed dataset from safe-to-publish fixtures
+- Compare default and research threshold profiles for bytecode and trace fixtures
 - Scan popular contracts and protocol surfaces
 - Publish deterministic report artifacts
 - Generate aggregate risk statistics

package/data/client-compat/clients.example.json

@@ -13,6 +13,11 @@
           "version": "0.0.0-example",
           "status": "unknown",
           "notes": "Placeholder only. Replace with a sourced compatibility statement before relying on this matrix."
+        },
+        {
+          "version": "1.0.0-compatible",
+          "status": "compatible",
+          "notes": "Synthetic compatible status used by safe fixture variants; do not treat as a real client release."
         }
       ]
     },
@@ -24,6 +29,11 @@
           "version": "0.0.0-example",
           "status": "unknown",
           "notes": "Placeholder only. Replace with a sourced compatibility statement before relying on this matrix."
+        },
+        {
+          "version": "1.0.0-compatible",
+          "status": "compatible",
+          "notes": "Synthetic compatible status used by safe fixture variants; do not treat as a real client release."
         }
       ]
     },
@@ -35,6 +45,11 @@
           "version": "0.0.0-example",
           "status": "unknown",
           "notes": "Placeholder only. Replace with a sourced compatibility statement before relying on this matrix."
+        },
+        {
+          "version": "1.0.0-compatible",
+          "status": "compatible",
+          "notes": "Synthetic compatible status used by safe fixture variants; do not treat as a real client release."
         }
       ]
     }

package/datasets/public-seed/README.md

@@ -0,0 +1,26 @@
+# Public Seed Dataset
+
+This directory contains the first deterministic dataset seed for Glamsterdam Compatibility Lab. It is generated from safe-to-publish fixtures documented in `fixtures/provenance.json`.
+
+The seed is intentionally small. It is meant to prove the dataset workflow, not to measure aggregate public-chain readiness.
+
+## Contents
+
+- `manifest.json`: index of generated reports, comparisons, source fixtures, threshold profiles, and limitations.
+- `reports/`: JSON compatibility reports generated from source fixtures.
+- `comparisons/`: JSON comparison reports for default-vs-research threshold profiles on bytecode and trace fixtures.
+
+## Regenerate
+
+```sh
+pnpm dataset:generate
+```
+
+Then run:
+
+```sh
+pnpm test
+pnpm build
+```
+
+Review generated changes before publishing. Dataset comparisons are structural report differences only; they do not infer final Glamsterdam gas deltas or client behavior.

package/datasets/public-seed/comparisons/bytecode-storage-heavy--default-vs-research.json

@@ -0,0 +1,168 @@
+{
+  "toolVersion": "0.3.1",
+  "fork": "glamsterdam",
+  "comparison": {
+    "baseline": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "bytecode",
+        "name": "fixtures/bytecode/storage-heavy.hex"
+      },
+      "summary": {
+        "risk": "medium",
+        "findingCount": 6,
+        "highCount": 0,
+        "mediumCount": 3,
+        "lowCount": 2,
+        "unknownCount": 1
+      }
+    },
+    "candidate": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "bytecode",
+        "name": "fixtures/bytecode/storage-heavy.hex"
+      },
+      "summary": {
+        "risk": "medium",
+        "findingCount": 6,
+        "highCount": 0,
+        "mediumCount": 3,
+        "lowCount": 2,
+        "unknownCount": 1
+      }
+    }
+  },
+  "summary": {
+    "riskChange": {
+      "from": "medium",
+      "to": "medium",
+      "direction": "unchanged"
+    },
+    "findingCount": {
+      "baseline": 6,
+      "candidate": 6,
+      "delta": 0
+    },
+    "addedCount": 0,
+    "removedCount": 0,
+    "changedCount": 0,
+    "unchangedCount": 6,
+    "severityIncreasedCount": 0,
+    "severityDecreasedCount": 0,
+    "severityChangedCount": 0,
+    "confidenceIncreasedCount": 0,
+    "confidenceDecreasedCount": 0,
+    "confidenceChangedCount": 0
+  },
+  "changes": {
+    "added": [],
+    "removed": [],
+    "changed": [],
+    "unchanged": [
+      {
+        "key": "bytecode.calldata-copy-exposure",
+        "id": "bytecode.calldata-copy-exposure",
+        "title": "Calldata copy opcode is present",
+        "severity": "low",
+        "confidence": "medium",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-7976",
+          "EIP-7904",
+          "EIP-8038"
+        ]
+      },
+      {
+        "key": "bytecode.contract-creation-opcodes",
+        "id": "bytecode.contract-creation-opcodes",
+        "title": "Contract creation opcodes are present",
+        "severity": "medium",
+        "confidence": "high",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-8037"
+        ]
+      },
+      {
+        "key": "bytecode.log-opcodes-present",
+        "id": "bytecode.log-opcodes-present",
+        "title": "Log opcodes are present",
+        "severity": "low",
+        "confidence": "medium",
+        "domain": [
+          "contracts",
+          "indexer",
+          "monitoring"
+        ],
+        "relatedEips": [
+          "EIP-7708"
+        ]
+      },
+      {
+        "key": "bytecode.manual-review-required",
+        "id": "bytecode.manual-review-required",
+        "title": "Manual review is still required for runtime behavior",
+        "severity": "unknown",
+        "confidence": "low",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING"
+        ]
+      },
+      {
+        "key": "bytecode.state-account-opcode-exposure",
+        "id": "bytecode.state-account-opcode-exposure",
+        "title": "State and account access opcodes are prominent in bytecode",
+        "severity": "medium",
+        "confidence": "medium",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-7904",
+          "EIP-8038",
+          "EIP-7976"
+        ]
+      },
+      {
+        "key": "bytecode.storage-heavy-pattern",
+        "id": "bytecode.storage-heavy-pattern",
+        "title": "Storage-related opcodes appear frequently",
+        "severity": "medium",
+        "confidence": "medium",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-8038"
+        ]
+      }
+    ]
+  },
+  "assumptions": [
+    "Reports were compared by finding id. Repeated finding ids are disambiguated with deterministic occurrence suffixes.",
+    "Severity and confidence changes are structural report changes, not protocol gas estimates."
+  ],
+  "limitations": [
+    "The comparison does not infer exact gas deltas, final Glamsterdam parameters, or current-vs-Glamsterdam client behavior unless those values are already present in the input reports.",
+    "Added and removed findings can reflect threshold profile differences, fixture coverage changes, registry updates, or detector changes; review the source reports before treating a diff as a protocol risk change."
+  ]
+}

package/datasets/public-seed/comparisons/traces-besu-debug-structlogs--default-vs-research.json

@@ -0,0 +1,91 @@
+{
+  "toolVersion": "0.3.1",
+  "fork": "glamsterdam",
+  "comparison": {
+    "baseline": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "trace",
+        "name": "fixtures/traces/besu-debug-structlogs.json"
+      },
+      "summary": {
+        "risk": "low",
+        "findingCount": 1,
+        "highCount": 0,
+        "mediumCount": 0,
+        "lowCount": 1,
+        "unknownCount": 0
+      }
+    },
+    "candidate": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "trace",
+        "name": "fixtures/traces/besu-debug-structlogs.json"
+      },
+      "summary": {
+        "risk": "low",
+        "findingCount": 1,
+        "highCount": 0,
+        "mediumCount": 0,
+        "lowCount": 1,
+        "unknownCount": 0
+      }
+    }
+  },
+  "summary": {
+    "riskChange": {
+      "from": "low",
+      "to": "low",
+      "direction": "unchanged"
+    },
+    "findingCount": {
+      "baseline": 1,
+      "candidate": 1,
+      "delta": 0
+    },
+    "addedCount": 0,
+    "removedCount": 0,
+    "changedCount": 0,
+    "unchangedCount": 1,
+    "severityIncreasedCount": 0,
+    "severityDecreasedCount": 0,
+    "severityChangedCount": 0,
+    "confidenceIncreasedCount": 0,
+    "confidenceDecreasedCount": 0,
+    "confidenceChangedCount": 0
+  },
+  "changes": {
+    "added": [],
+    "removed": [],
+    "changed": [],
+    "unchanged": [
+      {
+        "key": "trace.logs-calls-visible",
+        "id": "trace.logs-calls-visible",
+        "title": "Trace includes logs or external calls",
+        "severity": "low",
+        "confidence": "high",
+        "domain": [
+          "contracts",
+          "indexer",
+          "monitoring"
+        ],
+        "relatedEips": [
+          "EIP-7708",
+          "EIP-7928"
+        ]
+      }
+    ]
+  },
+  "assumptions": [
+    "Reports were compared by finding id. Repeated finding ids are disambiguated with deterministic occurrence suffixes.",
+    "Severity and confidence changes are structural report changes, not protocol gas estimates."
+  ],
+  "limitations": [
+    "The comparison does not infer exact gas deltas, final Glamsterdam parameters, or current-vs-Glamsterdam client behavior unless those values are already present in the input reports.",
+    "Added and removed findings can reflect threshold profile differences, fixture coverage changes, registry updates, or detector changes; review the source reports before treating a diff as a protocol risk change."
+  ]
+}

package/datasets/public-seed/comparisons/traces-besu-mainnet-tracoor-debug-structlogs--default-vs-research.json

@@ -0,0 +1,140 @@
+{
+  "toolVersion": "0.3.1",
+  "fork": "glamsterdam",
+  "comparison": {
+    "baseline": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "trace",
+        "name": "fixtures/traces/besu-mainnet-tracoor-debug-structlogs.json"
+      },
+      "summary": {
+        "risk": "medium",
+        "findingCount": 4,
+        "highCount": 0,
+        "mediumCount": 1,
+        "lowCount": 2,
+        "unknownCount": 1
+      }
+    },
+    "candidate": {
+      "toolVersion": "0.3.1",
+      "fork": "glamsterdam",
+      "target": {
+        "kind": "trace",
+        "name": "fixtures/traces/besu-mainnet-tracoor-debug-structlogs.json"
+      },
+      "summary": {
+        "risk": "medium",
+        "findingCount": 4,
+        "highCount": 0,
+        "mediumCount": 1,
+        "lowCount": 2,
+        "unknownCount": 1
+      }
+    }
+  },
+  "summary": {
+    "riskChange": {
+      "from": "medium",
+      "to": "medium",
+      "direction": "unchanged"
+    },
+    "findingCount": {
+      "baseline": 4,
+      "candidate": 4,
+      "delta": 0
+    },
+    "addedCount": 0,
+    "removedCount": 0,
+    "changedCount": 0,
+    "unchangedCount": 4,
+    "severityIncreasedCount": 0,
+    "severityDecreasedCount": 0,
+    "severityChangedCount": 0,
+    "confidenceIncreasedCount": 0,
+    "confidenceDecreasedCount": 0,
+    "confidenceChangedCount": 0
+  },
+  "changes": {
+    "added": [],
+    "removed": [],
+    "changed": [],
+    "unchanged": [
+      {
+        "key": "trace.calldata-heavy-execution",
+        "id": "trace.calldata-heavy-execution",
+        "title": "Trace includes visible calldata-heavy execution",
+        "severity": "low",
+        "confidence": "medium",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-7976",
+          "EIP-7904",
+          "EIP-8038"
+        ]
+      },
+      {
+        "key": "trace.logs-calls-visible",
+        "id": "trace.logs-calls-visible",
+        "title": "Trace includes logs or external calls",
+        "severity": "low",
+        "confidence": "high",
+        "domain": [
+          "contracts",
+          "indexer",
+          "monitoring"
+        ],
+        "relatedEips": [
+          "EIP-7708",
+          "EIP-7928"
+        ]
+      },
+      {
+        "key": "trace.partial-evidence",
+        "id": "trace.partial-evidence",
+        "title": "Trace omits some useful compatibility evidence",
+        "severity": "unknown",
+        "confidence": "low",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-7976"
+        ]
+      },
+      {
+        "key": "trace.state-heavy-execution-medium",
+        "id": "trace.state-heavy-execution-medium",
+        "title": "Trace shows state-heavy execution",
+        "severity": "medium",
+        "confidence": "high",
+        "domain": [
+          "contracts",
+          "execution"
+        ],
+        "relatedEips": [
+          "GAS-REPRICING",
+          "EIP-7904",
+          "EIP-8038",
+          "EIP-7976"
+        ]
+      }
+    ]
+  },
+  "assumptions": [
+    "Reports were compared by finding id. Repeated finding ids are disambiguated with deterministic occurrence suffixes.",
+    "Severity and confidence changes are structural report changes, not protocol gas estimates."
+  ],
+  "limitations": [
+    "The comparison does not infer exact gas deltas, final Glamsterdam parameters, or current-vs-Glamsterdam client behavior unless those values are already present in the input reports.",
+    "Added and removed findings can reflect threshold profile differences, fixture coverage changes, registry updates, or detector changes; review the source reports before treating a diff as a protocol risk change."
+  ]
+}