template-git-repo 0.1.0

package/README.md

@@ -0,0 +1,91 @@
+# template-git-repo
+
+One command to give a repository the CI, publishing, badges and Turborepo wiring
+worked out in [qwksearch-research-agent](https://github.com/OpenSourceAGI/qwksearch-research-agent):
+
+```bash
+bunx template-git-repo
+```
+
+It detects the repo slug, default branch, package manager, workspace layout and
+the package to advertise on npm, writes the workflows and their helper scripts,
+wires up Turborepo, and injects a badge block into your README. Nothing existing
+is overwritten without `--force`, and `--dry-run` prints the exact plan.
+
+## What it writes
+
+| Path | What it is |
+| --- | --- |
+| `.github/workflows/tests.yml` | Per-package tests with a **discovered** matrix, coverage + test analytics to Codecov |
+| `.github/workflows/npm-publish.yml` | Publishes every workspace package whose *content* changed — no version bookkeeping |
+| `.github/workflows/auto-merge-claude.yml` | Auto-merges agent PRs once their checks pass |
+| `.github/workflows/auto-merge-and-create-prs.yml` | Twice-daily sweep: merges green PRs, opens PRs for orphan branches |
+| `.github/workflows/deploy-test-reports.yml` | Publishes the HTML test report to Cloudflare Workers |
+| `scripts/*.mjs` | The five helpers those workflows call |
+| `turbo.json` | Pipeline whose task names the workflows use |
+| `codecov.yml` | Per-package flags with `carryforward` |
+| `README.md` | The badge block, between markers |
+
+Full setup notes: **[docs/ACTIONS.md](./docs/ACTIONS.md)** (workflows and secrets)
+and **[docs/BADGES.md](./docs/BADGES.md)** (every badge, and what you have to do
+outside the repo before it shows anything real).
+
+## Options
+
+```bash
+bunx template-git-repo --dry-run          # print the plan, change nothing
+bunx template-git-repo --force            # replace files that already exist
+bunx template-git-repo --actions-only     # just the workflows and their scripts
+bunx template-git-repo --badges-only      # just the README badge block
+bunx template-git-repo --no-turbo         # leave turbo.json and package.json alone
+```
+
+Badge inputs — each one enables the badge that needs it, and badges without their
+input are left out with a note rather than rendered broken:
+
+```bash
+bunx template-git-repo \
+  --doi 10.5281/zenodo.20951725 \
+  --docs https://example.com/docs \
+  --discord-id 1110227955554209923 --discord-invite https://discord.gg/xxxx \
+  --stack Claude,Cloudflare,Next.js
+```
+
+Run `bunx template-git-repo --help` for the rest.
+
+## The parts worth knowing
+
+**The test matrix is discovered, not written.** `scripts/list-test-packages.mjs`
+reads the `workspaces` globs and emits one matrix entry per package with a test
+script. A hand-maintained matrix fails silently — a package added to the repo but
+not to the matrix is never tested, and nothing goes red to say so.
+
+**Publishing compares content, not version numbers.** npm tarballs are
+reproducible, so `npm pack` integrity against the registry answers "did anything
+actually change". Version bumps happen as a consequence, not as a prerequisite,
+and they are taken from what the registry says is free — including the versions a
+half-finished publish reserved, which `latest` cannot show you.
+
+**Badges carry their own setup instructions.** `docs/BADGES.md` is generated from
+the catalog in `src/badges.js`, and a test fails if it drifts. A badge whose
+prerequisites nobody wrote down is a badge that reads `unknown` forever.
+
+**The badge block is re-runnable.** It lives between
+`<!-- template-git-repo:badges:start -->` and `...:end -->`, so running the CLI
+again after you publish a package or finish setting up Codecov replaces the block
+and nothing else.
+
+## Programmatic use
+
+```js
+import { renderBadgeBlock, buildContext } from 'template-git-repo';
+
+const context = buildContext({ overrides: { stack: 'Bun,Cloudflare' } });
+const { markdown, skipped } = renderBadgeBlock(context);
+```
+
+## Adding a badge
+
+Add an entry to `BADGES` in `src/badges.js` — `needs`, `setup` and `group` are
+what make it self-documenting — then `bun run docs:badges`. Nothing else needs to
+know about it.

package/bin/template-git-repo.js

@@ -0,0 +1,257 @@
+#!/usr/bin/env node
+/**
+ * One command that sets a repository up with the CI, publishing, badges and
+ * Turborepo wiring worked out in OpenSourceAGI/qwksearch-research-agent.
+ *
+ *   bunx template-git-repo
+ *
+ * Everything it can detect, it detects — repo slug, default branch, package
+ * manager, workspace layout, the package to advertise on npm — so the common
+ * case takes no flags. What it cannot know (a DOI, a Discord server, a docs
+ * URL) is passed in, and the badges that need those are skipped with a note
+ * rather than rendered broken.
+ *
+ * Nothing existing is overwritten without `--force`, and `--dry-run` prints the
+ * exact plan a real run would carry out.
+ */
+import fs from 'node:fs';
+import path from 'node:path';
+import process from 'node:process';
+import { buildContext } from '../src/context.js';
+import { renderBadgeBlock, BADGES } from '../src/badges.js';
+import { injectBadges, hasUnmarkedBadges } from '../src/readme.js';
+import { planFiles, applyPlan, wireTurbo } from '../src/apply.js';
+
+const HELP = `
+template-git-repo — set up a repo with the GitHub Actions, badges and Turborepo
+config from the qwksearch reference setup.
+
+Usage
+  bunx template-git-repo [options]
+
+What it writes
+  .github/workflows/   tests, npm publish, auto-merge, hosted test reports
+  scripts/*.mjs        the helpers those workflows call
+  turbo.json           pipeline whose task names the workflows use
+  codecov.yml          per-package flags with carryforward
+  README.md            the badge block, between markers
+
+Options
+  --dry-run              Print the plan and change nothing
+  --force                Overwrite files that already exist
+  --actions-only         Only the workflows and their scripts
+  --badges-only          Only the README badge block
+  --no-turbo             Do not write turbo.json or touch the root package.json
+  --yes, -y              Skip the confirmation prompt
+
+Context (all optional — detected where possible)
+  --repo <owner/repo>    Default: the origin remote
+  --branch <name>        Default: the remote's HEAD branch
+  --pm <bun|pnpm|yarn|npm>
+  --npm-package <name>   Package for the npm version/downloads badges
+  --workflow <file>      Workflow file for the CI badge (default tests.yml)
+
+Badge inputs (each enables the badge that needs it)
+  --doi <10.5281/...>        --docs <url>          --api <url>
+  --youtube <url>            --uptime <url>        --test-report <url>
+  --discord-id <id>          --discord-invite <url>
+  --stack <A,B,C>            --cloudflare-deploy
+  --exclude <id,id>          --only <id,id>
+
+  Badge ids: ${BADGES.map((b) => b.id).join(', ')}
+
+Docs
+  docs/ACTIONS.md   what each workflow does and which secrets it needs
+  docs/BADGES.md    how to set up each badge
+`;
+
+/**
+ * A deliberately small parser: this CLI takes `--flag` and `--flag value` and
+ * nothing else, so a dependency for it would cost more than it saves.
+ *
+ * @param {string[]} argv
+ * @returns {Record<string, string | boolean>}
+ */
+export function parseArgs(argv) {
+  const flags = {};
+
+  for (let i = 0; i < argv.length; i++) {
+    const arg = argv[i];
+    if (!arg.startsWith('-')) continue;
+
+    const key = arg.replace(/^--?/, '');
+    const next = argv[i + 1];
+
+    if (next && !next.startsWith('-')) {
+      flags[key] = next;
+      i++;
+    } else {
+      flags[key] = true;
+    }
+  }
+
+  return flags;
+}
+
+/**
+ * @param {Record<string, string | boolean>} flags
+ * @returns {Record<string, string | undefined>}
+ */
+function overridesFrom(flags) {
+  const text = (key) => (typeof flags[key] === 'string' ? flags[key] : undefined);
+
+  return {
+    repoSlug: text('repo'),
+    defaultBranch: text('branch'),
+    packageManager: text('pm'),
+    npmPackage: text('npm-package'),
+    workflowFile: text('workflow'),
+    doi: text('doi'),
+    docsUrl: text('docs'),
+    apiUrl: text('api'),
+    youtubeUrl: text('youtube'),
+    uptimeUrl: text('uptime'),
+    testReportUrl: text('test-report'),
+    discordId: text('discord-id'),
+    discordInvite: text('discord-invite'),
+    stack: text('stack'),
+    cloudflareDeploy: flags['cloudflare-deploy'] ? 'true' : undefined,
+  };
+}
+
+/**
+ * @param {string[]} [argv]
+ * @returns {Promise<number>} process exit code
+ */
+export async function main(argv = process.argv.slice(2)) {
+  const flags = parseArgs(argv);
+
+  if (flags.help || flags.h) {
+    console.log(HELP);
+    return 0;
+  }
+
+  const dryRun = Boolean(flags['dry-run']);
+  const force = Boolean(flags.force);
+  const badgesOnly = Boolean(flags['badges-only']);
+  const actionsOnly = Boolean(flags['actions-only']);
+  const skipTurbo = Boolean(flags['no-turbo']) || badgesOnly;
+
+  const context = buildContext({ overrides: overridesFrom(flags) });
+
+  if (!context.repoSlug) {
+    console.error(
+      'Could not work out which GitHub repo this is: no `origin` remote, or one that is not on github.com.\n' +
+        'Pass it explicitly:  bunx template-git-repo --repo owner/repo',
+    );
+    return 1;
+  }
+
+  console.log(`\n📦 ${context.repoSlug}  (${context.defaultBranch}, ${context.packageManager})`);
+  console.log(`   ${context.root}\n`);
+
+  let changed = 0;
+
+  // ── Workflows, scripts, turbo.json, codecov.yml ───────────────────────────
+  if (!badgesOnly) {
+    const include = actionsOnly ? ['.github', 'scripts'] : undefined;
+    const filePlan = planFiles({ context, force, include }).filter(
+      (entry) => !(skipTurbo && entry.path === 'turbo.json'),
+    );
+
+    for (const entry of filePlan) {
+      const mark = entry.action === 'skip' ? '·' : entry.action === 'overwrite' ? '~' : '+';
+      console.log(`  ${mark} ${entry.path}${entry.action === 'skip' ? '  (exists — use --force to replace)' : ''}`);
+    }
+
+    const { written } = applyPlan(filePlan, { dryRun });
+    changed += written.length;
+    console.log('');
+  }
+
+  // ── Root package.json: turbo scripts and devDependency ────────────────────
+  if (!skipTurbo && !actionsOnly) {
+    const manifestPath = path.join(context.root, 'package.json');
+
+    if (!fs.existsSync(manifestPath)) {
+      console.log('  · package.json not found at the repo root — skipping the Turborepo wiring\n');
+    } else {
+      const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
+      const { manifest: next, changes } = wireTurbo(manifest, { packagesDir: context.packagesDir });
+
+      if (changes.length === 0) {
+        console.log('  · package.json already has the turbo wiring\n');
+      } else {
+        console.log(`  ~ package.json  (${changes.join(', ')})\n`);
+        if (!dryRun) fs.writeFileSync(manifestPath, `${JSON.stringify(next, null, 2)}\n`);
+        changed++;
+      }
+    }
+  }
+
+  // ── README badges ─────────────────────────────────────────────────────────
+  if (!actionsOnly) {
+    const readmePath = ['README.md', 'readme.md', 'Readme.md']
+      .map((name) => path.join(context.root, name))
+      .find((candidate) => fs.existsSync(candidate)) ?? path.join(context.root, 'README.md');
+
+    const existing = fs.existsSync(readmePath) ? fs.readFileSync(readmePath, 'utf8') : '';
+
+    const { markdown, skipped } = renderBadgeBlock(context, {
+      only: typeof flags.only === 'string' ? flags.only.split(',') : undefined,
+      exclude: typeof flags.exclude === 'string' ? flags.exclude.split(',') : [],
+    });
+
+    const { content, action } = injectBadges(existing, markdown);
+
+    if (content === existing) {
+      console.log('  · README.md badges already up to date\n');
+    } else {
+      console.log(`  ~ ${path.basename(readmePath)}  (badge block ${action})`);
+      if (hasUnmarkedBadges(existing)) {
+        console.log('    ⚠ the README already had badges of its own — they were left in place, remove the duplicates by hand');
+      }
+      if (!dryRun) fs.writeFileSync(readmePath, content);
+      changed++;
+      console.log('');
+    }
+
+    if (skipped.length > 0) {
+      console.log('  Badges not included (nothing to point them at):');
+      for (const { id, missing } of skipped) {
+        console.log(`    ${id.padEnd(20)} needs ${missing.join(', ')}`);
+      }
+      console.log('    → see docs/BADGES.md, then re-run with the matching flag\n');
+    }
+  }
+
+  // ── What the human still has to do ────────────────────────────────────────
+  if (!badgesOnly) {
+    console.log('  Repository secrets the workflows expect:');
+    console.log('    CODECOV_TOKEN            coverage + test analytics uploads');
+    console.log('    NPM_TOKEN                only if you are not using npm trusted publishing');
+    console.log('    GIT_TOKEN                a PAT for auto-merge (GITHUB_TOKEN merges do not trigger workflows)');
+    console.log('    CLOUDFLARE_API_TOKEN     hosted test reports');
+    console.log('    CLOUDFLARE_ACCOUNT_ID    hosted test reports');
+    console.log('    → docs/ACTIONS.md explains each one\n');
+  }
+
+  if (dryRun) {
+    console.log(`Dry run — nothing was written (${changed} file(s) would change).\n`);
+  } else {
+    console.log(changed === 0 ? 'Already set up — nothing to do.\n' : `Done — ${changed} file(s) changed.\n`);
+  }
+
+  return 0;
+}
+
+// Only run when invoked as a program, so the module can be imported by tests.
+if (import.meta.url === `file://${process.argv[1]}`) {
+  main().then(
+    (code) => process.exit(code),
+    (error) => {
+      console.error(error);
+      process.exit(1);
+    },
+  );
+}

package/docs/ACTIONS.md

@@ -0,0 +1,195 @@
+# GitHub Actions setup
+
+What each workflow in `template/.github/workflows/` does, what it needs before it
+can work, and the failure it exists to prevent. These are the workflows from
+[OpenSourceAGI/qwksearch-research-agent](https://github.com/OpenSourceAGI/qwksearch-research-agent),
+generalized — most of the comments in them describe an incident.
+
+## The secrets, in one table
+
+Add these under **Settings → Secrets and variables → Actions → New repository secret**.
+
+| Secret | Needed by | How to get it | If it is missing |
+| --- | --- | --- | --- |
+| `CODECOV_TOKEN` | `tests.yml` | codecov.io → add the repo → Settings → Repository Upload Token | Uploads are rejected; the coverage badge stays `unknown`. Tests still run and still pass. |
+| `NPM_TOKEN` | `npm-publish.yml` | npmjs.com → Access Tokens → Granular, read-write on the packages | Only needed if you are *not* using trusted publishing. See below. |
+| `GIT_TOKEN` | `auto-merge-*.yml` | A fine-grained PAT with Contents: read-write and Pull requests: read-write | Falls back to `GITHUB_TOKEN`, which merges but does **not** trigger further workflows — so a merge would never fire the publish run. |
+| `CLOUDFLARE_API_TOKEN` | `deploy-test-reports.yml` | dash.cloudflare.com → My Profile → API Tokens → "Edit Cloudflare Workers" | The deploy step fails; the rest of the run is unaffected. |
+| `CLOUDFLARE_ACCOUNT_ID` | `deploy-test-reports.yml` | The hex id in any Cloudflare dashboard URL | Same. |
+
+Repository settings that matter as much as the secrets:
+
+- **Settings → General → Allow auto-merge** — without it, `--auto` is rejected and
+  the auto-merge workflows fall back to merging immediately, which skips waiting
+  for checks.
+- **Settings → Actions → General → Workflow permissions → Read and write** —
+  without it, the version-bump commit in `npm-publish.yml` cannot be pushed.
+- **Branch protection with at least one required check** — this is what `--auto`
+  waits on. With no required check there is nothing to wait for.
+
+---
+
+## `tests.yml` — test, coverage, and test analytics
+
+Runs every workspace package that has a test suite, uploads `coverage/lcov.info`
+to Codecov under a per-package flag, and uploads `junit.xml` to Codecov Test
+Analytics, which tracks per-test run time and flakiness and comments failures on
+the PR.
+
+**The matrix is discovered, not written.** A `discover` job runs
+`scripts/list-test-packages.mjs`, which reads the `workspaces` globs from the root
+`package.json` and emits one entry per package with a `test:coverage`, `test:ci`
+or `test` script. A hand-maintained matrix fails silently — a package added to
+the repo but not to the matrix is simply never tested, and nothing goes red to
+say so.
+
+What each package has to provide:
+
+```jsonc
+{
+  "scripts": {
+    // Whatever writes coverage/lcov.info and junit.xml. For Vitest:
+    "test:coverage": "vitest run --coverage"
+  },
+  // Optional: keep a currently-red suite from failing the workflow while it
+  // still uploads its results. Remove once the suite is green.
+  "ci": { "allowFailure": true }
+}
+```
+
+For Vitest, `junit.xml` comes from the reporter config, not a flag:
+
+```js
+reporters: process.env.CI ? ['default', 'junit'] : ['default'],
+outputFile: { junit: './junit.xml' },
+```
+
+**Why `if: ${{ !cancelled() }}` and not `always()`** on the upload steps: a
+cancelled run has nothing worth uploading, but a *failed* one is exactly when the
+report matters most. `always()` also runs on cancellation and clutters the
+dashboard with partial data.
+
+**Why `fail_ci_if_error: false`**: Codecov having an outage must not turn a green
+suite red.
+
+---
+
+## `npm-publish.yml` — publish what changed, without version bookkeeping
+
+Publishes every non-private workspace package whose *content* changed, on every
+push to the default branch. Nobody has to remember to bump a version.
+
+Five rules are encoded in it, each learned from a specific failure:
+
+1. **Compare pack integrity, not version numbers.** npm tarballs are reproducible
+   (normalized mtimes), so `npm pack --dry-run --json` integrity against
+   `npm view <pkg>@<version> dist.integrity` is a reliable "did anything change".
+   A version bump is not evidence of a change, and an unchanged version is not
+   evidence of no change.
+2. **Check the credential before building anything.** npm answers an unauthorized
+   PUT with `E404 ... could not be found or you do not have permission to access
+   it`, which reads like a missing package — and it arrives *after* every package
+   has been built, with a provenance statement already signed into the public
+   sigstore transparency log for each failed attempt.
+3. **Walk the workspace in dependency order** (`scripts/workspace-build-order.mjs`).
+   Package managers link workspace siblings as symlinks; a sibling that has not
+   been built has no `dist/`, and its `exports` → `types` entries point at files
+   that do not exist. Alphabetical order produces
+   `Cannot find module '<sibling>' or its corresponding type declarations`.
+4. **Never publish the literal `workspace:*` protocol** — no consumer outside the
+   monorepo can resolve it. `scripts/pin-workspace-deps.mjs` substitutes real
+   ranges at pack time; `scripts/restore-pinned-deps.mjs` takes them back out
+   afterwards, so only the version bump gets committed.
+5. **The registry decides which versions are spent** (`scripts/next-free-version.mjs`).
+   npm refuses a PUT for any version it has *ever* seen, including ones staged by
+   an interrupted publish that the `latest` dist-tag cannot show you:
+   `E409 ... Cannot publish over previously staged version`. The union of
+   `versions` and the `time` timeline is the real taken set.
+
+### Trusted publishing (recommended) vs `NPM_TOKEN`
+
+**Trusted publishing (OIDC)** — no secret at all. Leave `NPM_TOKEN` unset. The
+job's `id-token: write` permission is traded by npm ≥ 11.5.1 for a short-lived
+publish token. For each package, go to
+`npmjs.com/package/<name>/access` → **Trusted publisher** and name this
+repository plus `npm-publish.yml`. Nothing to rotate, nothing to expire.
+
+**`NPM_TOKEN`** — a granular access token with read-write on the packages. It
+expires after 90 days at most, and classic automation tokens no longer work for
+direct publishing. Setting the secret selects this path; leaving it unset selects
+OIDC.
+
+### Failure decoder
+
+| Symptom | Cause → fix |
+| --- | --- |
+| `E404` on publish for a package that exists | The credential cannot write. Rotate `NPM_TOKEN`, or finish the trusted-publisher setup for that package. |
+| `E409 Cannot publish over previously staged version` | A number the registry reserved. The workflow already retries at the next free version five times; more than that means the registry is refusing everything. |
+| `Cannot implicitly apply the latest tag` | The local version fell behind the registry, usually a bump commit that never landed back. The workflow syncs to `latest` and re-evaluates. |
+| `EUNSUPPORTEDPROTOCOL workspace:*` | `npm` was run where the workspace's own package manager should have been, or without `--no-workspaces` over its symlinks. |
+| `husky: not found` (exit 127) | A dependency ships a broken `prepare: "husky install"`. The no-op husky shim step handles it; do not remove that step. |
+| `vite: not found` during a package build | Dependencies were installed with plain `npm` inside a bun/pnpm workspace, which leaves devDependencies uninstalled. |
+
+---
+
+## `auto-merge-claude.yml` — merge agent PRs when they go green
+
+Fires on every PR event and, **only for the actors in its `if:` allowlist**,
+enables auto-merge. That allowlist is the entire safety model — edit it to the
+accounts you actually want merged unattended, and never widen it to all
+contributors.
+
+It tries `gh pr merge --auto` first (GitHub merges once required checks pass) and
+falls back to an immediate merge, because `--auto` is rejected outright on a repo
+with no branch protection — there is nothing for it to wait on.
+
+Long-lived branches (`production`, `prod`, `staging`, `develop`) are merged
+without `--delete-branch`.
+
+---
+
+## `auto-merge-and-create-prs.yml` — the twice-daily sweep
+
+The backstop for two things that fall through the cracks: an agent session that
+pushes a branch and never opens a PR, and a PR whose last check finished after
+the per-PR workflow had already run.
+
+It merges every open PR that is non-draft, `CLEAN`, not blocked by a requested
+change, and whose checks all concluded `SUCCESS`/`SKIPPED`/`NEUTRAL` — a *pending*
+check means "not yet", not "merge it". Then it opens a PR for every remote branch
+that has neither an open nor a previously merged PR and is not already an ancestor
+of the default branch.
+
+It deliberately does not use `actions/checkout`: it needs every remote ref, and
+`git init` + a full `git fetch` is cheaper than `fetch-depth: 0`.
+
+---
+
+## `deploy-test-reports.yml` — a test report with a URL
+
+Publishes the Vitest HTML reporter output to Cloudflare Workers on every push to
+the default branch, so the Test Report badge links to something current instead of
+to a workflow log.
+
+Needs an `apps/test-reports/` directory with a `wrangler.toml` whose
+`assets.directory` is `dist`, and a root `test:report` script that writes there.
+
+The test step is `continue-on-error: true` — a red suite is exactly when the
+report is worth reading. What it must not do is leave `dist/` missing: wrangler
+treats an absent `assets.directory` as a hard error, so the run would end on a
+config error instead of on the test signal. The placeholder-page step exists for
+that.
+
+---
+
+## Adding your own
+
+Keep the shape the rest of these use:
+
+- `concurrency.group` on anything that pushes or deploys, so two runs cannot race.
+- `fail-fast: false` on any matrix where one entry's failure should not hide the
+  others' results.
+- `if: ${{ !cancelled() }}` rather than `always()` on upload steps.
+- A comment at the top saying what the workflow is *for*, and a comment at any
+  step whose reason is not obvious from the code. The comments in these files are
+  the part that survives; the YAML is the easy half.