bulletin-deploy 0.4.1

package/README.md

@@ -0,0 +1,170 @@
+# bulletin-deploy
+
+Deploy static sites and apps to the Polkadot Triangle network with decentralized storage and human-readable `.dot` domains.
+
+## Quick Start
+
+```bash
+npm install -g github:paritytech/bulletin-deploy
+# Build your app (e.g. npm run build)
+bulletin-deploy ./dist my-app00.dot
+```
+
+Your site is live at `https://my-app00.dot.li`
+
+## Prerequisites
+
+- **Node.js 22+**
+- **IPFS Kubo** (for merkleizing directories)
+
+```bash
+# macOS
+brew install ipfs
+ipfs init
+
+# Linux
+wget https://dist.ipfs.tech/kubo/v0.33.0/kubo_v0.33.0_linux-amd64.tar.gz
+tar -xvzf kubo_v0.33.0_linux-amd64.tar.gz
+sudo bash kubo/install.sh
+ipfs init
+```
+
+## CLI Usage
+
+### Deploy an app
+
+```bash
+bulletin-deploy <build-dir> <domain.dot>
+```
+
+Examples:
+
+```bash
+# Basic deploy
+bulletin-deploy ./dist my-app00.dot
+
+# With DotNS owner mnemonic
+MNEMONIC="word1 word2 ..." bulletin-deploy ./dist my-app00.dot
+
+# Custom RPC endpoint
+bulletin-deploy --rpc wss://custom-bulletin.example.com ./dist my-app00.dot
+```
+
+### All options
+
+```
+Options:
+  --mnemonic "..."    DotNS owner mnemonic (or set MNEMONIC env var)
+  --rpc wss://...     Bulletin RPC (or set BULLETIN_RPC env var)
+  --help              Show help
+```
+
+## GitHub Actions
+
+1. Copy `workflows/deploy-on-pr.yml` to your repo's `.github/workflows/` directory
+2. Set the `MNEMONIC` secret in your repo settings (the mnemonic that owns your `.dot` domain)
+3. Customize the **Build** step for your framework (Vite, Next.js, etc.)
+4. Push and watch the deploy
+
+The template workflow:
+- Deploys on push to main and on PRs
+- Uses `nick-fields/retry@v3` for automatic retries on transient failures
+- Posts a comment on PRs with the live URL
+- Generates domain names as `<repo>-<branch>00.dot`
+
+## Programmatic API
+
+```javascript
+import { deploy, DotNS } from "@paritytech/bulletin-deploy";
+
+// Deploy a directory
+const result = await deploy("./dist", "my-app00.dot");
+console.log(result.cid, result.domainName);
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `BULLETIN_RPC` | `wss://paseo-bulletin-rpc.polkadot.io` | Bulletin chain WebSocket RPC |
+| `BULLETIN_POOL_SIZE` | `10` | Number of pool accounts to derive |
+| `BULLETIN_POOL_MNEMONIC` | Dev phrase (Alice) | Mnemonic for pool account derivation |
+| `MNEMONIC` | Dev phrase | DotNS domain owner mnemonic |
+| `BULLETIN_DEPLOY_TELEMETRY` | `1` (enabled) | Set to `0` to disable Sentry telemetry |
+| `IPFS_CID` | _(none)_ | Skip storage, use pre-existing CID |
+| `GITHUB_OUTPUT` | _(none)_ | GitHub Actions output file (set automatically in CI) |
+
+## How It Works
+
+```
+Build output ──> IPFS merkleize ──> CAR file ──> Chunk upload ──> DotNS
+    ./dist          ipfs add        .car         Bulletin        Asset Hub
+                                                 Storage         Registry
+```
+
+1. **Merkleize** your build directory with IPFS to produce a content-addressed CAR file
+2. **Chunk and upload** the CAR file to Bulletin's TransactionStorage (1MB chunks, 2 per batch)
+3. **Store the DAG root** that links all chunks together under a single CID
+4. **Register or update** your `.dot` domain on Asset Hub with the new contenthash
+
+Your site is immediately accessible at `https://your-domain.dot.li`
+
+## Resilience Features
+
+### Chunk-level retry
+
+Each batch of chunks is submitted with `Promise.allSettled`. Failed chunks are retried up to 3 times with a fresh nonce, serialized to avoid nonce conflicts. If a chunk fails all retries, the deploy aborts with a clear error.
+
+### Account pool
+
+Instead of using a single account for all storage transactions, bulletin-deploy derives a pool of accounts from a mnemonic. Each deploy selects the account with the most remaining authorization capacity. This prevents nonce conflicts between concurrent deploys and distributes the storage authorization budget.
+
+### Auto-authorization
+
+When a pool account's authorization drops below thresholds (50 transactions or 50MB), bulletin-deploy automatically tops it up by submitting an `authorize_account` transaction from Alice.
+
+## Telemetry
+
+Sentry telemetry is enabled by default for deploy observability.
+
+What's tracked:
+- Deploy duration and success/failure
+- Storage phase timing (merkleize, chunk upload, root node)
+- DotNS phase timing (registration, contenthash update)
+- Pool account selection
+- Chunk retry counts
+- Source metadata (repo, branch, PR number, CI vs local)
+
+Telemetry is enabled by default. Set `BULLETIN_DEPLOY_TELEMETRY=0` to disable.
+
+Dashboard: https://polkadot-community-foundation.sentry.io/dashboards/92523/
+
+## Troubleshooting
+
+| Error | Solution |
+|---|---|
+| `Payment` or authorization error | Pool account needs storage authorization — auto-authorization should handle this |
+| `Stale` or dropped from best chain | Bulletin chain reorg. Automatic retry handles this. |
+| `IPFS CLI not installed` | Install Kubo: `brew install ipfs && ipfs init` |
+| `CommitmentNotFound` | DotNS timing issue during registration. Retry the deploy. |
+| `All pool accounts exhausted` | Auto-authorization will top up the best available account |
+| `File exceeds 8MB limit` | File is automatically chunked. This shouldn't appear for directories. |
+| `fetchNonce timed out` | Bulletin RPC may be down. Check endpoint or try a different one. |
+
+## Configuration for Different Chains
+
+By default, bulletin-deploy targets the **Paseo testnet**:
+- Bulletin: `wss://paseo-bulletin-rpc.polkadot.io`
+- Asset Hub: `wss://asset-hub-paseo.dotters.network`
+
+To point at a different chain, set the `BULLETIN_RPC` environment variable:
+
+```bash
+# Local development
+BULLETIN_RPC=ws://127.0.0.1:9944 bulletin-deploy ./dist my-app00.dot
+
+# Custom endpoint
+BULLETIN_RPC=wss://your-bulletin-rpc.example.com bulletin-deploy ./dist my-app00.dot
+```
+
+The Asset Hub RPC endpoints are configured in `src/dotns.js` and support automatic failover across multiple providers.

package/benchmark.js

@@ -0,0 +1,163 @@
+import * as fs from "fs";
+import * as path from "path";
+import { createClient as createPolkadotClient } from "polkadot-api";
+import { Binary } from "@polkadot-api/substrate-bindings";
+import { getPolkadotSigner } from "polkadot-api/signer";
+import { getWsProvider } from "polkadot-api/ws-provider";
+import { withPolkadotSdkCompat } from "polkadot-api/polkadot-sdk-compat";
+import { sr25519CreateDerive } from "@polkadot-labs/hdkd";
+import { DEV_PHRASE, entropyToMiniSecret, mnemonicToEntropy } from "@polkadot-labs/hdkd-helpers";
+import * as dagPB from "@ipld/dag-pb";
+import { UnixFS } from "ipfs-unixfs";
+import { fetchNonce, TX_TIMEOUT_MS } from "./src/dotns.js";
+import { createCID, merkleize, chunk } from "./src/deploy.js";
+
+const BULLETIN_RPC = process.env.BULLETIN_RPC || "wss://paseo-bulletin-rpc.polkadot.io";
+const ALICE_SS58 = "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY";
+
+function toHashingEnum(mhCode) {
+  switch (mhCode) {
+    case 0x12: return { type: "Sha2_256", value: undefined };
+    case 0xb220: return { type: "Blake2b256", value: undefined };
+    default: throw new Error(`Unhandled: ${mhCode}`);
+  }
+}
+
+function createSigner() {
+  const entropy = mnemonicToEntropy(DEV_PHRASE);
+  const miniSecret = entropyToMiniSecret(entropy);
+  const derive = sr25519CreateDerive(miniSecret);
+  const keyPair = derive("//Alice");
+  return getPolkadotSigner(keyPair.publicKey, "Sr25519", keyPair.sign);
+}
+
+function watchTx(tx, signer, txOpts, label) {
+  return new Promise((resolve, reject) => {
+    let settled = false;
+    const settle = (fn) => (...args) => { if (!settled) { settled = true; clearTimeout(timer); try { sub.unsubscribe(); } catch {} fn(...args); } };
+    const timer = setTimeout(() => settle(reject)(new Error(`${label} timed out`)), TX_TIMEOUT_MS);
+    const sub = tx.signSubmitAndWatch(signer, txOpts).subscribe({
+      next: (event) => {
+        if (event.type === "txBestBlocksState" && event.found) {
+          if (event.ok) settle(resolve)();
+          else settle(reject)(new Error(`${label} dispatch error`));
+        }
+      },
+      error: (e) => settle(reject)(new Error(`${label}: ${e?.message?.slice(0, 100) || e}`)),
+    });
+  });
+}
+
+async function benchmarkConfig(carBuffer, chunkSize, batchSize, label) {
+  const chunks = chunk(carBuffer, chunkSize);
+  const rounds = Math.ceil(chunks.length / batchSize);
+  const concurrentMB = (chunkSize * batchSize / 1024 / 1024).toFixed(1);
+
+  console.log(`\n--- ${label} | ${chunks.length} chunks | ${rounds} rounds | ${concurrentMB}MB/round ---`);
+
+  const client = createPolkadotClient(withPolkadotSdkCompat(getWsProvider(BULLETIN_RPC)));
+  const unsafeApi = client.getUnsafeApi();
+  const signer = createSigner();
+
+  const start = Date.now();
+  try {
+    const startNonce = await fetchNonce(BULLETIN_RPC, ALICE_SS58);
+    const stored = [];
+
+    for (let b = 0; b < chunks.length; b += batchSize) {
+      const batch = chunks.slice(b, b + batchSize);
+      const batchStart = Date.now();
+      const batchPromises = batch.map((chunkData, j) => {
+        const i = b + j;
+        const nonce = startNonce + i;
+        const hashCode = 0x12;
+        const cid = createCID(chunkData, 0x55, hashCode);
+        const tx = unsafeApi.tx.TransactionStorage.store_with_cid_config({
+          cid: { codec: BigInt(0x55), hashing: toHashingEnum(hashCode) },
+          data: Binary.fromBytes(chunkData),
+        });
+        return watchTx(tx, signer, { mortality: { mortal: true, period: 256 }, nonce }, `chunk-${i}`)
+          .then(() => ({ cid, len: chunkData.length }));
+      });
+      const batchResults = await Promise.all(batchPromises);
+      stored.push(...batchResults);
+      const batchElapsed = ((Date.now() - batchStart) / 1000).toFixed(1);
+      console.log(`   Batch ${Math.floor(b / batchSize) + 1}/${rounds}: ${batch.length} chunks in ${batchElapsed}s`);
+    }
+
+    // Store root DAG node
+    const fileData = new UnixFS({ type: "file", blockSizes: stored.map((c) => BigInt(c.len)) });
+    const dagNode = dagPB.prepare({ Data: fileData.marshal(), Links: stored.map((c) => ({ Name: "", Tsize: c.len, Hash: c.cid })) });
+    const dagBytes = dagPB.encode(dagNode);
+    const rootCid = createCID(dagBytes, 0x70, 0x12);
+    const rootNonce = startNonce + chunks.length;
+    const rootTx = unsafeApi.tx.TransactionStorage.store_with_cid_config({
+      cid: { codec: BigInt(0x70), hashing: toHashingEnum(0x12) },
+      data: Binary.fromBytes(dagBytes),
+    });
+    await watchTx(rootTx, signer, { mortality: { mortal: true, period: 256 }, nonce: rootNonce }, "root");
+
+    const elapsed = ((Date.now() - start) / 1000).toFixed(1);
+    console.log(`   ROOT OK | Total: ${elapsed}s`);
+    client.destroy();
+    return { label, chunks: chunks.length, rounds, concurrentMB, elapsed, success: true };
+  } catch (e) {
+    const elapsed = ((Date.now() - start) / 1000).toFixed(1);
+    const error = e.message?.slice(0, 80);
+    console.log(`   FAILED (${elapsed}s): ${error}`);
+    client.destroy();
+    return { label, chunks: chunks.length, rounds, concurrentMB, elapsed, success: false, error };
+  }
+}
+
+// --- Main ---
+const buildDir = process.argv[2];
+if (!buildDir || !fs.existsSync(buildDir)) {
+  console.error("Usage: node benchmark.js <build-dir>");
+  process.exit(1);
+}
+
+console.log("Preparing CAR file...");
+const carPath = path.join(path.dirname(buildDir), `${path.basename(buildDir)}.car`);
+const { cid: ipfsCid } = await merkleize(buildDir, carPath);
+const carBuffer = fs.readFileSync(carPath);
+console.log(`CAR: ${(carBuffer.length / 1024 / 1024).toFixed(2)} MB | CID: ${ipfsCid}\n`);
+
+const configs = [
+  [2 * 1024 * 1024, 1, "2MB x1"],
+  [1 * 1024 * 1024, 2, "1MB x2"],
+  [1 * 1024 * 1024, 1, "1MB x1"],
+  [512 * 1024,      3, "512K x3"],
+  [512 * 1024,      4, "512K x4"],
+  [256 * 1024,      4, "256K x4"],
+  [256 * 1024,      6, "256K x6"],
+];
+
+const results = [];
+for (const [chunkSize, batchSize, label] of configs) {
+  const r = await benchmarkConfig(carBuffer, chunkSize, batchSize, label);
+  results.push(r);
+  // Wait between runs
+  if (label !== configs[configs.length - 1][2]) {
+    console.log("   Cooling 10s...");
+    await new Promise(r => setTimeout(r, 10000));
+  }
+}
+
+console.log(`\n\n${"=".repeat(70)}`);
+console.log("RESULTS");
+console.log(`${"=".repeat(70)}`);
+console.log(`| Config     | Chunks | Rounds | MB/round | Time    | Result |`);
+console.log(`|------------|--------|--------|----------|---------|--------|`);
+for (const r of results) {
+  const status = r.success ? "OK" : `FAIL`;
+  console.log(`| ${r.label.padEnd(10)} | ${String(r.chunks).padEnd(6)} | ${String(r.rounds).padEnd(6)} | ${r.concurrentMB.padEnd(8)} | ${(r.elapsed + "s").padEnd(7)} | ${status.padEnd(6)} |`);
+}
+if (results.some(r => !r.success)) {
+  console.log("\nFailed configs:");
+  for (const r of results.filter(r => !r.success)) {
+    console.log(`  ${r.label}: ${r.error}`);
+  }
+}
+
+process.exit(0);

package/bin/bulletin-deploy

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+
+import { deploy } from "../src/deploy.js";
+import { bootstrapPool } from "../src/pool.js";
+import * as fs from "fs";
+
+const args = process.argv.slice(2);
+
+const flags = {};
+const positional = [];
+for (let i = 0; i < args.length; i++) {
+  if (args[i] === "--bootstrap") { flags.bootstrap = true; }
+  else if (args[i] === "--pool-size") { flags.poolSize = parseInt(args[++i], 10); }
+  else if (args[i] === "--mnemonic") { flags.mnemonic = args[++i]; }
+  else if (args[i] === "--rpc") { flags.rpc = args[++i]; }
+  else if (args[i] === "--help" || args[i] === "-h") { flags.help = true; }
+  else { positional.push(args[i]); }
+}
+
+if (flags.help || (positional.length === 0 && !flags.bootstrap)) {
+  console.log(`Usage:
+  bulletin-deploy <build-dir> <domain.dot>  Deploy an app
+  bulletin-deploy --bootstrap               Initialize pool accounts
+
+Options:
+  --mnemonic "..."    DotNS owner mnemonic (or set MNEMONIC env var)
+  --rpc wss://...     Bulletin RPC (or set BULLETIN_RPC env var)
+  --pool-size N       Number of pool accounts (default: 10)
+  --help              Show this help`);
+  process.exit(0);
+}
+
+if (flags.rpc) process.env.BULLETIN_RPC = flags.rpc;
+if (flags.poolSize) process.env.BULLETIN_POOL_SIZE = String(flags.poolSize);
+
+try {
+  if (flags.bootstrap) {
+    const rpc = process.env.BULLETIN_RPC || "wss://paseo-bulletin-rpc.polkadot.io";
+    const poolSize = parseInt(process.env.BULLETIN_POOL_SIZE || "10", 10);
+    await bootstrapPool(rpc, poolSize);
+  } else {
+    const [buildDir, domain] = positional;
+    if (!buildDir) { console.error("Error: build directory required"); process.exit(1); }
+    if (!domain) { console.error("Error: domain required (e.g. my-app.dot)"); process.exit(1); }
+    if (!fs.existsSync(buildDir)) { console.error(`Error: ${buildDir} does not exist`); process.exit(1); }
+
+    const result = await deploy(buildDir, domain);
+
+    const output = process.env.GITHUB_OUTPUT;
+    if (output) {
+      fs.appendFileSync(output, `cid=${result.cid}\n`);
+      fs.appendFileSync(output, `domain=${result.domainName}\n`);
+    }
+
+    console.log(`CID: ${result.cid}`);
+    console.log(`Domain: ${result.domainName}`);
+  }
+  process.exit(0);
+} catch (error) {
+  console.error("Deployment failed:", error.message);
+  process.exit(1);
+}

package/package.json

@@ -0,0 +1,41 @@
+{
+  "name": "bulletin-deploy",
+  "version": "0.4.1",
+  "private": false,
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/paritytech/bulletin-deploy.git"
+  },
+  "publishConfig": {
+    "registry": "https://registry.npmjs.org",
+    "access": "public"
+  },
+  "type": "module",
+  "bin": {
+    "bulletin-deploy": "./bin/bulletin-deploy"
+  },
+  "exports": {
+    ".": "./src/index.js"
+  },
+  "scripts": {
+    "test": "node --test test/test.js",
+    "benchmark": "node benchmark.js"
+  },
+  "dependencies": {
+    "@ipld/dag-pb": "^4.1.3",
+    "@sentry/node": "^9.14.0",
+    "@noble/hashes": "^1.7.2",
+    "@polkadot-api/substrate-bindings": "^0.16.5",
+    "@polkadot-labs/hdkd": "^0.0.25",
+    "@polkadot-labs/hdkd-helpers": "^0.0.26",
+    "@polkadot/keyring": "^13.0.0",
+    "@polkadot/util-crypto": "^13.0.0",
+    "ipfs-unixfs": "^11.2.0",
+    "multiformats": "^13.4.1",
+    "polkadot-api": "^1.23.1",
+    "viem": "^2.30.5"
+  },
+  "engines": {
+    "node": ">=22"
+  }
+}