@parity/product-deploy 0.7.28-rc.0

package/docs/e2e-bootstrap.md

@@ -0,0 +1,154 @@
+# E2E test setup
+
+The E2E suite (`test/e2e.test.js`, driven by `.github/workflows/e2e.yml`) deploys real content to Paseo Bulletin testnet via `bulletin-deploy` and verifies the on-chain round-trip. It consumes the **shared default pool** (derived from `DEV_PHRASE` — the same pool real users hit in production) for Bulletin chunk upload, so no pool bootstrapping is required.
+
+Three one-time setup items are needed before the workflow can pass. Do them once per testnet lifetime (redo if testnet is wiped).
+
+---
+
+## Paseo (stable testnet)
+
+### Prerequisites
+
+- `bulletin-deploy` built locally (`npm run build`).
+- Network access to Paseo Bulletin RPC (`wss://paseo-bulletin-rpc.polkadot.io`) and Asset Hub Paseo (DotNS).
+- Alice's dev mnemonic: `bottom drive obey lake curtain smoke basket hold race lonely fit walk`.
+
+### 1. Verify Alice Personhood status
+
+Both happy-path scenarios (S1, S2) deploy as Alice via DotNS. Registering a new un-reserved base name requires Personhood status from the chain's Personhood precompile:
+
+```bash
+node tools/check-pop-status.mjs
+```
+
+Re-run if the check cannot read Alice's status. Self-attestation is no longer available; ask the DotNS team to whitelist the signer if Alice needs a higher status for a label class.
+
+### 2. Fund and map Bob on Asset Hub Paseo
+
+Bob (`//Bob` from the dev phrase) is the owner of `e2eowned.dot` (see item 3). He needs:
+
+- **Balance** on Asset Hub Paseo for his on-chain fees. Request ~1 PAS from the Paseo faucet at [https://faucet.polkadot.io/](https://faucet.polkadot.io/) sending to Bob's SS58 address `5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty`.
+- **Revive mapping** so he can sign EVM transactions. `check-pop-status` reports Bob's deterministic H160 and Personhood status without submitting transactions. A later Bob-signed registration or transfer triggers mapping when the chain supports automatic mapping.
+
+```bash
+node tools/check-pop-status.mjs "bottom drive obey lake curtain smoke basket hold race lonely fit walk//Bob"
+```
+
+Expected output: Bob's SS58, his H160 (`0x41dccbd49b26c50d34355ed86ff0fa9e489d1e01`), and PoP status (`NoStatus (0)` initially). Idempotent.
+
+### 3. Register `e2eowned.dot` directly as Bob
+
+The S3 negative scenario asserts that `bulletin-deploy` refuses to deploy to a domain owned by a different account (exit 78 with transfer guidance). Have Bob register the label with a DotNS registration tool outside `bulletin-deploy` — no Alice intermediary, no Bulletin content needed (S3 never reads content).
+
+```bash
+node tools/register-test-fixture-paseo-next-v2.mjs e2eowned
+```
+
+Expected: `e2eowned.dot` is owned by Bob's H160 `0x41dccbd49b26c50d34355ed86ff0fa9e489d1e01`. The `e2eowned.dot` label requires PoP Full status and a mature commitment (the tool waits 30s after the minimum commitment age). The tool is idempotent: it exits 0 immediately if Bob already owns it, and transfers the label from Alice back to Bob if fixture drift is detected.
+
+If S3 ever fails because `e2eowned.dot` was transferred back to Alice by mistake, re-run this step to restore Bob's ownership.
+
+---
+
+## Paseo Next v2 (`--env paseo-next-v2`)
+
+Paseo Next v2 uses a separate Asset Hub (`wss://paseo-asset-hub-next-rpc.polkadot.io`) and Bulletin chain (`wss://paseo-bulletin-next-rpc.polkadot.io`) with different contract addresses. The `map_account` extrinsic does not exist on this chain — account mapping is triggered automatically when an account submits its first on-chain transaction.
+
+> **Note on PoP grants:** The paseo-next-v2 `POP_RULES` contract (`0x2002C1c15b88632Ad01c7770f6EbE1Ca05c8472E`) is **not permissionless** — `setUserPopStatus` can only be called by the contract owner (`0x4a519c30da0ec16aa9a73c26ea6ca6f701cce099`). `bulletin-deploy` itself cannot upgrade a signer; the chain team flips the Personhood precompile out of band. CI scenarios pick labels via `pickStableLabel`/`pickDirectLabel`/`pickIncLabel`/`pickRotLabel`, which auto-select between a PoP-Full base name (e.g. `e2epool.dot`) and a NoStatus fallback (e.g. `e2epoolns01.dot`) based on what `Personhood.personhoodStatus(<signer>)` returns at test start. The setup below covers both modes.
+
+> **⚠ Testnet wipes reset everything.** When paseo-next-v2 is reset (which happens periodically), Alice's Personhood precompile status drops to NoStatus *and* every `e2e*.dot` registration is gone. Re-run the relevant steps below after each wipe — there is no on-chain self-recovery. Issue #381 traces back to exactly this: Alice's status came back as Full but `e2epool.dot` was unregistered, so `setContenthash` reverted with `ERC721NonexistentToken`. As long as ownership stays in lockstep with Alice's PoP grade (Full ↔ PoP-Full labels registered; NoStatus ↔ NoStatus labels auto-register on first deploy), the nightly stays green.
+
+### Prerequisites
+
+- `bulletin-deploy` built locally (`npm run build`).
+- Alice (`5DfhGyQdFobKM8NsWvEeAKk5EQQgYe9AydgJ7rMB6E1EqRzV`) funded on paseo-next-v2 Asset Hub. The chain team must send funds manually — there is no public faucet for this chain.
+
+### 1. Authorize pool accounts on Bulletin Next
+
+```bash
+bulletin-bootstrap --env paseo-next-v2
+```
+
+This grants each pool account `TransactionStorage` quota on Bulletin Next. Alice (`//Alice`) must be funded and mapped on Asset Hub Next for this to succeed (Alice's mapping is triggered automatically by her first on-chain tx, so funding alone is sufficient).
+
+### 2. Verify Alice PoP status
+
+```bash
+node tools/check-pop-status.mjs --env paseo-next-v2
+```
+
+What you see decides which labels need pre-registration:
+
+- **`NoStatus (0)`** → no extra work. The tests pick the NoStatus fallback labels (`e2epoolns01`, `e2edirect01`, `e2eincpool01`, `e2erotpool01`) which auto-register on first deploy because their shape (base length ≥ 9 with two trailing digits) bypasses the `Requires Full personhood verification` gate.
+- **`ProofOfPersonhoodFull (2)`** → the chain team has flipped Alice on the Personhood precompile. The tests will now pick the PoP-Full stable labels (`e2epool`, `e2edirect`, `e2einc`, `e2erot`), and those **must already be registered to Alice** before the matrix runs. Register them once:
+
+  ```bash
+  # Pre-built fixture is fine — it's the contenthash, not the content, that the tests overwrite.
+  for label in e2epool e2edirect e2einc e2erot; do
+    node bin/bulletin-deploy test/fixtures/e2e-spa "${label}.dot" --env paseo-next-v2 --js-merkle
+  done
+  ```
+
+  Skipping this leaves `setContenthash` reverting with `ERC721NonexistentToken` on whichever PoP-Full label the scenario picks (issue #381).
+
+### 3. Fund Bob and trigger his account mapping
+
+Bob (`5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty`) needs PAS on paseo-next-v2 Asset Hub so he can pay fees for the `e2eowned.dot` registration. Have the chain team send ~1 PAS to his SS58 address. His H160 mapping (`0x41dccbd49b26c50d34355ed86ff0fa9e489d1e01`) is triggered automatically when he submits his first on-chain tx.
+
+Verify his mapping:
+
+```bash
+node tools/check-pop-status.mjs --env paseo-next-v2 \
+  "bottom drive obey lake curtain smoke basket hold race lonely fit walk//Bob"
+```
+
+Expected output: Bob's SS58, his H160 (`0x41dccbd49b26c50d34355ed86ff0fa9e489d1e01`), and PoP status.
+
+### 4. Verify Bob PoP status
+
+Bob owns both `e2eownedns02.dot` (NoStatus, used by S3 when Alice is NoStatus) and `e2eowned.dot` (PoP-Full, used by S3 when the Personhood precompile has flipped Alice to Full). The NoStatus branch needs no admin help; the PoP-Full branch needed Bob to register it back when the chain team granted him Full status once. After that one-shot registration the label persists until expiry, so Bob can stay NoStatus going forward — ownership and PoP-grade are decoupled after registration.
+
+### 5. Register `e2eownedns02.dot` directly as Bob
+
+Register the domain using the dedicated tool:
+
+```bash
+node tools/register-test-fixture-paseo-next-v2.mjs e2eownedns02
+```
+
+Expected: `e2eownedns02.dot` is owned by Bob (`0x41dccbd49b26c50d34355ed86ff0fa9e489d1e01`).
+The tool is idempotent: it leaves Bob-owned state alone and transfers the label back to Bob if a failed S3 fixture run accidentally left it owned by Alice.
+
+### 6. Ensure Bob owns `e2eowned.dot` on paseo-next-v2
+
+S3 picks `e2eowned.dot` when Alice is PoP-Full at test start. If the label is unregistered, Alice's deploy gets routed through the full register flow and her own H160 lands as owner — at which point every subsequent S3 run on the Full path "succeeds" cleanly (`exit 0`) instead of being rejected (`exit 78`), and the test fails. Bob must own this label.
+
+If Bob already owns it (`ownerOf` on `DOTNS_REGISTRAR` returns `0x41dccbd…`), skip. Otherwise:
+
+- **If nobody owns it yet:** registration needs Bob to hold Full PoP. Coordinate with the chain team to flip Bob's Personhood precompile to Full, then run `node tools/register-e2eowned-paseo-next-v2.mjs` (the script targets Bob via `//Bob`). After registration Bob can drop back to NoStatus.
+- **If Alice (or anyone other than Bob) is squatting:** the owner can call `transferFrom(<current>, Bob, tokenId)` on `DOTNS_REGISTRAR`. ERC721 transfer is unconditional on the recipient (no PoP check) and doesn't need Bob's signer. `tools/transfer-e2eowned-to-bob.mjs` does this for Alice → Bob in one shot (was added to recover from a botched PR-CI run on 2026-05-17, see issue #381 thread).
+
+---
+
+## No pre-registration needed for `e2epoolns01.dot` / direct NoStatus labels
+
+The stable happy-path labels auto-register to the selected test signer on first deploy. Subsequent runs exercise the update path (new contenthash under existing ownership). On paseo-next-v2 these labels must remain NoStatus-compatible because DotNS self-attestation is no longer available.
+
+## Verifying locally
+
+```bash
+# Paseo
+E2E=1 E2E_SIGNER=pool E2E_MERKLE=js E2E_SCENARIO=s1 \
+  BULLETIN_RPC=wss://paseo-bulletin-rpc.polkadot.io \
+  npm run test:e2e
+
+# Paseo Next v2
+E2E=1 E2E_SIGNER=pool E2E_MERKLE=js E2E_SCENARIO=s1 \
+  BULLETIN_RPC=wss://paseo-bulletin-next-rpc.polkadot.io \
+  BULLETIN_DEPLOY_ENV=paseo-next-v2 \
+  npm run test:e2e
+# Previously named DOTNS_ENV; kept as deprecated alias for one release.
+```
+
+Vary `E2E_SCENARIO` (`s1`, `s2`, `s3`), `E2E_SIGNER` (`pool`, `direct`), and `E2E_MERKLE` (`js`, `kubo`) to cover the full CI matrix.

package/docs/telemetry.md

@@ -0,0 +1,62 @@
+# Telemetry
+
+Telemetry is **off by default for external users**. It is enabled automatically for known internal Parity contexts and can also be controlled explicitly.
+
+## Opt in / opt out
+
+- `BULLETIN_DEPLOY_TELEMETRY=1`: explicit opt-in
+- `BULLETIN_DEPLOY_TELEMETRY=0`: force off
+
+Internal detection signals are OR'd together:
+
+1. `GITHUB_REPOSITORY` matches a known internal org
+2. `RUNNER_NAME` starts with `parity-`
+3. `git remote get-url origin` points at a known internal org
+
+## What is tracked
+
+- deploy duration and success/failure
+- storage phase timing
+- DotNS phase timing
+- pool account selection
+- source metadata such as repo, branch, and CI vs local
+- tool version
+
+## Ambient Sentry mode
+
+If another app embeds `bulletin-deploy` and already owns Sentry initialization, set these before importing or invoking the library:
+
+```sh
+BULLETIN_DEPLOY_USE_AMBIENT_SENTRY=1
+BULLETIN_DEPLOY_HOST_APP=<your-app-name>
+BULLETIN_DEPLOY_HOST_APP_VERSION=<your-app-version>
+```
+
+`BULLETIN_DEPLOY_HOST_APP_VERSION` is optional but recommended — it populates `deploy.host_app_version` on every span, enabling version-correlated triage in the dashboard.
+
+That makes `bulletin-deploy` reuse the existing Sentry client instead of calling its own `Sentry.init()`.
+
+Requirements:
+
+- the host app must initialize Sentry first
+- Sentry SDK compatibility still matters
+- quotas and issue grouping remain owned by the host project
+
+## Tagging test traffic
+
+Use `--tag` or `DEPLOY_TAG` to separate test and benchmark traffic from real deploys.
+
+Examples:
+
+```bash
+bulletin-deploy --tag e2e-ci-pr ./build my-app.dot
+DEPLOY_TAG=load-test bulletin-deploy ./build my-app.dot
+```
+
+Common tags in this repo:
+
+- `e2e-ci-pr`
+- `e2e-ci-nightly`
+- `e2e-local-smoke`
+- `e2e-local-pr`
+- `e2e-local-nightly`

package/docs/testing.md

@@ -0,0 +1,44 @@
+# Testing
+
+The repo has three practical test layers: offline unit tests, live-testnet E2E coverage, and GitHub Actions matrices that exercise the shipped reusable workflow.
+
+## Offline tests
+
+```bash
+npm test
+```
+
+This runs the local Node test suite without network access.
+
+## Live-testnet E2E
+
+The E2E suite deploys real content to Paseo Bulletin and verifies the on-chain round-trip.
+
+Local launchers:
+
+```bash
+npm run test:e2e:smoke
+npm run test:e2e:pr
+npm run test:e2e:nightly
+```
+
+Quiet mode:
+
+```bash
+E2E_QUIET=1 npm run test:e2e:smoke
+E2E_QUIET=1 npm run test:e2e:pr
+E2E_QUIET=1 npm run test:e2e:nightly
+```
+
+Each scenario writes a JUnit XML report under `e2e-reports/`.
+
+For one-time chain setup, see [E2E test setup](./e2e-bootstrap.md).
+
+## CI matrices
+
+`.github/workflows/e2e.yml` calls the shipped reusable `.github/workflows/deploy.yml` so the E2E jobs exercise the same path consumers use.
+
+- per-PR: stable happy-path coverage plus negative ownership coverage
+- nightly: broader signer, merkleization, and mirror-path coverage
+
+E2E deploys are tagged so telemetry can distinguish them from real-user traffic. See [Telemetry](./telemetry.md).

package/package.json

@@ -0,0 +1,82 @@
+{
+  "name": "@parity/product-deploy",
+  "version": "0.7.28-rc.0",
+  "private": false,
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/paritytech/bulletin-deploy.git"
+  },
+  "publishConfig": {
+    "registry": "https://registry.npmjs.org",
+    "access": "public"
+  },
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "bulletin-deploy": "./bin/bulletin-deploy",
+    "bulletin-bootstrap": "./bin/bulletin-bootstrap"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./deploy": {
+      "types": "./dist/deploy.d.ts",
+      "import": "./dist/deploy.js"
+    },
+    "./manifest-roundtrip": {
+      "types": "./dist/manifest-roundtrip.d.ts",
+      "import": "./dist/manifest-roundtrip.js"
+    }
+  },
+  "files": [
+    "dist",
+    "bin",
+    "docs",
+    "assets",
+    "tools/release-retry-wrapper.mjs"
+  ],
+  "scripts": {
+    "build": "tsup src/index.ts src/deploy.ts src/dotns.ts src/pool.ts src/telemetry.ts src/memory-report.ts src/merkle.ts src/gh-pages-mirror.ts src/version-check.ts src/bug-report.ts src/run-state.ts src/environments.ts src/errors.ts src/manifest.ts src/chunk-probe.ts src/manifest-embed.ts src/manifest-fetch.ts src/manifest-roundtrip.ts src/incremental-stats.ts src/chunker.ts src/personhood/encoding.ts src/personhood/hashing.ts src/personhood/constants.ts src/personhood/member-key.ts src/personhood/people-client.ts src/personhood/reprove.ts src/personhood/bind-personal-id.ts src/personhood/claim-pgas.ts src/personhood/bind-paid-alias.ts src/personhood/bootstrap.ts src/manifest/types.ts src/manifest/schema.ts src/manifest/byte-budget.ts src/manifest/config-load.ts src/manifest/publish.ts --format esm --dts --clean --target node22",
+    "refresh-environments": "node scripts/refresh-environments.mjs",
+    "check:watched-dependencies": "node tools/check-watched-dependencies.mjs",
+    "prepare": "npm run build",
+    "test": "npm run build && node --test test/test.js test/cli-help.test.js test/helpers/e2e-helpers.test.js test/environments.test.js test/refresh-environments.test.js test/watched-dependencies.test.js test/chunk-sharing-report.test.js test/product-manifest.test.js",
+    "test:e2e": "npm run build && node --test test/e2e.test.js",
+    "test:e2e:smoke": "bash scripts/e2e-pass.sh smoke",
+    "test:e2e:pr": "bash scripts/e2e-pass.sh pr",
+    "test:e2e:nightly": "bash scripts/e2e-pass.sh nightly",
+    "benchmark": "npm run build && node benchmark.js"
+  },
+  "dependencies": {
+    "@ipld/car": "^5.4.3",
+    "@ipld/dag-pb": "^4.1.3",
+    "@noble/hashes": "^1.7.2",
+    "@polkadot-api/metadata-builders": "^0.14.2",
+    "@polkadot-api/substrate-bindings": "^0.20.2",
+    "@polkadot-labs/hdkd": "^0.0.28",
+    "@polkadot-labs/hdkd-helpers": "^0.0.30",
+    "@polkadot/keyring": "^14.0.3",
+    "@polkadot/util-crypto": "^14.0.3",
+    "@sentry/node": "^9.14.0",
+    "ipfs-unixfs": "^11.2.0",
+    "ipfs-unixfs-importer": "^16.1.4",
+    "jiti": "^2.4.2",
+    "multiformats": "^13.4.1",
+    "polkadot-api": "^2.1.3",
+    "verifiablejs": "^1.2.0",
+    "viem": "^2.30.5"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "tsup": "^8.5.0",
+    "typescript": "^5.9.3",
+    "ws": "^8.20.1"
+  },
+  "minimumVersion": "0.5.6",
+  "engines": {
+    "node": ">=22"
+  }
+}

package/tools/release-retry-wrapper.mjs

@@ -0,0 +1,74 @@
+#!/usr/bin/env node
+// Selective retry wrapper for release E2E.
+//
+// Spawns a child process (the deploy CLI / test runner invocation), captures
+// stdout AND stderr, classifies the failure mode, and exits with 75 on
+// flake-class matches (retry-eligible) or the child's own exit code otherwise.
+//
+// Both streams are passed through to the parent's stdout/stderr so the GH
+// Actions job log still shows everything live.
+//
+// node --test captures each test-file subprocess's output and re-emits it as
+// TAP YAML on its own stdout — so deploy CLI errors (e.g. "ChainHead
+// disjointed") appear on stdout, not stderr. Capturing both is required.
+//
+// Configure nick-fields/retry@v3 with retry_on_exit_code: 75 so retries only
+// fire for the named transient classes. See
+// docs-internal/superpowers/specs/2026-05-22-ci-restructure-design.md.
+
+import { spawn } from "node:child_process";
+
+// Exact substrings that map to retry-eligible flake classes.
+// Patterns derived from Sentry telemetry (top transient errors over 30d on
+// the e2e-ci-pr and e2e-ci-release tags).
+const FLAKE_PATTERNS = [
+  "Invalid: Stale",                          // tx Invalid/Stale (nonce race) — papi 1.x format
+  '"type": "Stale"',                         // tx Invalid/Stale — papi 2.x JSON format
+  "ChainHead disjointed",                    // RPC reorg / WS flake
+  "Connection lost",                         // WS hard drop
+  "Account mapping did not take effect",     // Revive mapping race
+  "requires Node.js >=22",                   // parity-default runner downgrade (Node v18) — infra flake
+  "received a shutdown signal",              // runner process killed mid-job — CI infra flake
+];
+
+// output: combined stdout+stderr text from the child. Any flake pattern
+// appearing anywhere in the child's output makes the run retry-eligible.
+export function classifyForRetry(output, childExitCode = 1) {
+  if (childExitCode === 0) return 0;
+  for (const pat of FLAKE_PATTERNS) {
+    if (output.includes(pat)) return 75;
+  }
+  return childExitCode || 1;
+}
+
+// CLI entry: when run directly, spawn argv tail as a child and classify.
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const [cmd, ...args] = process.argv.slice(2);
+  if (!cmd) {
+    console.error("usage: release-retry-wrapper.mjs <command> [args...]");
+    process.exit(2);
+  }
+  const child = spawn(cmd, args, { stdio: ["inherit", "pipe", "pipe"] });
+  let outputBuf = "";
+  child.stdout.on("data", (chunk) => {
+    process.stdout.write(chunk); // pass through stdout to job log
+    outputBuf += chunk.toString();
+  });
+  child.stderr.on("data", (chunk) => {
+    process.stderr.write(chunk); // pass through stderr to job log
+    outputBuf += chunk.toString();
+  });
+  child.on("error", (err) => {
+    process.stderr.write(`[release-retry-wrapper] failed to spawn: ${err.message}\n`);
+    process.exit(1);
+  });
+  // Use `close` (not `exit`) so both pipes are fully drained before we
+  // classify — `exit` can fire before the last `data` chunk lands.
+  child.on("close", (code) => {
+    const cls = classifyForRetry(outputBuf, code ?? 1);
+    if (cls === 75) {
+      console.error("[release-retry-wrapper] flake-class match — exiting 75 to signal retry");
+    }
+    process.exit(cls);
+  });
+}