@buildinternet/releases-skills 0.50.0 → 0.52.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@buildinternet/releases-skills",
-  "version": "0.50.0",
+  "version": "0.52.0",
   "description": "Agent skills bundled with the Releases CLI. Markdown playbooks for changelog ingest, discovery, and analysis.",
   "type": "module",
   "license": "MIT",

package/skills/finding-changelogs/SKILL.md

@@ -193,6 +193,8 @@ For static providers, the adapter automatically uses Cloudflare's crawl API with
 
 **If a fast fetch returns incomplete content**, the adapter falls back to full rendering automatically. If you notice this happening repeatedly for a source, set `--render` to force headless rendering and note the reason in the playbook.
 
+**Rendering can't clear a bot challenge.** If a page sits behind a Cloudflare *Managed Challenge*, browser rendering fails too (symptom: `no_change` / 0 releases on a page that's clearly updating) — `--render` won't fix it. The external **Firecrawl monitoring** backend can fetch these instead; it's enabled per source via the admin API (`POST /v1/sources/:slug/firecrawl/sync { enabled: true }`), backend-only, no CLI verb. See the monorepo's `docs/architecture/firecrawl-monitoring.md`.
+
 The agent's role is to evaluate content completeness after the first fetch — check that releases have titles, dates, and content. If they do, the fast path is working. If releases are empty or missing, the page likely needs JS rendering.
 
 ## Source Selection and Scope

package/skills/managing-sources/SKILL.md

@@ -15,6 +15,7 @@ Operations can be performed via CLI commands or typed MCP/agent tools. Use which
 |-----------|-----|------------|
 | List sources | `releases list [slug] --json [--org <org>] [--query <text>] [--has-feed] [--category <c>] [--compact] [--limit <n>] [--page <n>]` | `list_catalog` (filter `kind: "source"` to exclude products); `list_sources` is a deprecated alias |
 | Add source | `releases admin source create <name> --url <url> [--type <type>] [--org <org>] [--feed-url <url>] [--primary]` | `manage_source` action "add" with name, url, type, organization, feed_url, **is_primary** (type auto-detected if omitted; only pass is_primary=true when the source is the org's primary changelog — see "Primary Sources") |
+| Add App Store source | `releases admin source create-appstore <url-or-id> [--platform ios\|macos] [--org <slug>] [--product <slug>] [--storefront <code>]` | _(no typed tool yet — CLI only)_ |
 | Edit source | `releases admin source update <identifier> [--primary] [--priority <p>]` | `manage_source` action "edit" with identifier, is_primary, fetch_priority, name, url, type (use only when changing an already-added source; prefer setting flags on "add") |
 | Remove source | `releases admin source delete <slug> [--ignore --reason <reason>]` | `manage_source` action "remove" with identifier |
 | Fetch releases | `releases admin source fetch <slug> [--dry-run] [--max <n>]` | `manage_source` action "fetch" with identifier |
@@ -47,10 +48,29 @@ Use `--json` (CLI) for structured output. Typed tools always return JSON.
 
 ## Adding Sources
 
-Required: **name** and **url**. Optional: **type** (github, scrape, feed, agent — auto-detected from URL if omitted), **organization** (org ID or slug to associate with), **feed_url** (direct feed URL if known).
+Required: **name** and **url**. Optional: **type** (github, scrape, feed, agent — auto-detected from URL if omitted), **organization** (org ID or slug to associate with), **feed_url** (direct feed URL if known). App Store apps (`appstore` type) are **not** created this way — use `create-appstore` (below); `source create` rejects `--type appstore` and pasted `apps.apple.com` URLs with a pointer to it.
 
 On slug collision the API auto-suffixes (`changelog` → `changelog-2`, `-3`, …) and the created row in the response tells you the resolved slug — no rename-and-retry needed.
 
+### App Store sources
+
+App Store apps need a dedicated command because the create flow resolves the iTunes listing, mints the current version as the first release, and backfills the product's avatar with the app icon:
+
+```
+releases admin source create-appstore <url-or-id> [--platform ios|macos] [--org <slug>] [--product <slug>] [--storefront <code>]
+```
+
+- `<url-or-id>` accepts an `apps.apple.com/.../id<trackId>` URL, a bare numeric track ID, or an `appstore:<trackId>` coordinate. `--platform` defaults to `ios` (`macos` = Mac App Store); `--storefront` defaults to `us`.
+- **Pre-create the product for a clean name.** With no `--product`, the endpoint names a *new* product after the (often verbose) App Store title — e.g. "Shopify: Sell online/in person". To control the name, create the product first and reference it:
+
+  ```
+  releases admin product create "Shopify" --org shopify
+  releases admin source create-appstore https://apps.apple.com/us/app/shopify/id719892358 --org shopify --product shopify
+  ```
+
+- **Keep writes serial.** The endpoint resolves the listing on the fly; concurrent creates for a brand-new org/product race on the org/product slug uniqueness constraint. Add one app at a time.
+- The command is idempotent on the app's track ID — re-running reports the existing source instead of creating a duplicate.
+
 ### Naming sources and products
 
 **Don't prefix names with the org name.** The org is already shown as context on every page — repeating it in each child source produces noise like "Datadog › Datadog dd-trace-py". Pick the bare, recognizable name instead.
@@ -201,6 +221,8 @@ Use `--render` when you know a source needs JavaScript execution. Use `--no-rend
 
 After adding a new scrape source with an unknown provider, check the first fetch results. If content is complete, consider setting `--no-render` and noting the provider behavior in the playbook.
 
+**Blocked by a Cloudflare Managed Challenge?** `--render`/`--no-render` only choose *how* we fetch — they don't help when the page returns a bot challenge that fails browser rendering itself (symptom: persistent `no_change` / 0 releases on a page that clearly updates, e.g. some vendor help pages). For those, the external **Firecrawl monitoring** backend can fetch the page instead. It's enabled per source via the admin API (`POST /v1/sources/:slug/firecrawl/sync { enabled: true }`), not a CLI verb, and not via `--metadata-set` (which skips monitor creation). See the monorepo's `docs/architecture/firecrawl-monitoring.md`.
+
 ## Duplicate Detection
 
 Before adding sources, search for overlapping URLs.

package/skills/parsing-changelogs/SKILL.md

@@ -15,6 +15,7 @@ The fetch pipeline follows this priority order:
 2. **Markdown fetch** — if `metadata.markdownUrl` is set, fetch raw markdown instead of rendered HTML.
 3. **Fast fetch (static providers)** — for providers known to serve pre-rendered HTML (Docusaurus, VitePress, WordPress, Ghost, Mintlify), fetch without headless browser rendering. Uses Cloudflare crawl API with `render: false`. ~10-30x faster than full rendering. Controlled by provider `staticContent` hint or per-source `renderRequired` metadata.
 4. **Cloudflare rendering** — for JS-heavy pages (React SPAs, Notion, etc.), use Cloudflare's browser rendering API to get the fully-rendered HTML. Fallback when fast fetch returns no content.
+5. **Firecrawl monitoring** — for sources behind a Cloudflare *Managed Challenge* that blocks even browser rendering (some vendor help/docs pages, e.g. OpenAI's), an external Firecrawl monitor scrapes the page on a schedule and POSTs changes to the backend, which extracts them through the same parse pipeline. This is a backend-only fetch backend, not a CLI fetch path: it's enabled per source via `metadata.firecrawl` through the admin API (`POST /v1/sources/:slug/firecrawl/sync { enabled: true }`), **not** via `--metadata-set` (that only patches the DB column and skips monitor creation). There's no `releases` verb for it yet — manage it via the API directly. See the monorepo's `docs/architecture/firecrawl-monitoring.md`.
 
 After fetching content, the pipeline parses it:
 - **Incremental parsing** — if the source already has releases in the database, extract only new ones by comparing against known releases. This is the default for subsequent fetches.

package/skills/releases-cli/references/admin.md

@@ -40,7 +40,7 @@ releases admin source create "Linear" --url https://linear.app/changelog --dry-r
 
 `--dry-run` still runs the URL dedup and exclusion checks (so you'll see "already exists" or "blocked URL" outcomes), but skips the write — including the auto-create-org side effect when `--org <name>` doesn't resolve.
 
-By default, `create` runs automated pre-checks (provider detection, feed discovery, markdown probing). Override with `--type github|scrape|feed|agent`. Batch mode (`--batch`) skips evaluation by default for speed.
+By default, `create` runs automated pre-checks (provider detection, feed discovery, markdown probing). Override with `--type github|scrape|feed|agent`. Batch mode (`--batch`) skips evaluation by default for speed. App Store apps use the dedicated `source create-appstore` verb (below) — `create` rejects `--type appstore` and pasted `apps.apple.com` URLs with a pointer to it.
 
 Provide a feed URL explicitly when it isn't easily discoverable:
 
@@ -49,12 +49,44 @@ releases admin source create "Claude Code" --url https://docs.anthropic.com/en/c
   --feed-url https://docs.anthropic.com/en/changelog/rss.xml
 ```
 
+Set source metadata **at create time** with `--keyword-allow` (feed keyword filter → `metadata.feedKeywordAllow`) or the general `--metadata-set key=value` (repeatable; same coercion as `source update --metadata-set`):
+
+```bash
+releases admin source create "Discord" --url https://discord.com/blog --type feed \
+  --feed-url https://discord.com/blog/rss.xml --keyword-allow changelog,patch-notes
+releases admin source create "Acme" --url https://acme.dev/changelog \
+  --metadata-set marketingFilter=true --metadata-set feedContentDepth=summary-only
+```
+
+Do this rather than a follow-up `source update --metadata-set`: `create` triggers the onboard workflow's auto-fetch, which reads the source's metadata **before** any post-create edit lands. Setting a feed filter on create keeps that first ingest filtered; setting it afterward races the auto-fetch and ingests the whole unfiltered feed.
+
 Evaluate without adding:
 
 ```bash
 releases admin discovery evaluate https://linear.app/changelog
 ```
 
+### Create App Store
+
+App Store apps have a dedicated verb because the create flow resolves the iTunes listing, mints the current version as the first release, and backfills the product's avatar with the app icon:
+
+```bash
+releases admin source create-appstore https://apps.apple.com/us/app/slack/id618783545 --org slack --product slack
+releases admin source create-appstore appstore:618783545 --platform ios --org slack
+releases admin source create-appstore 1496833156 --platform macos --dry-run
+```
+
+The identifier is an `apps.apple.com/.../id<trackId>` URL, a bare numeric track ID, or an `appstore:<trackId>` coordinate. `--platform` defaults to `ios` (`macos` = Mac App Store); `--storefront` defaults to `us`. The verb is idempotent on the track ID — re-running reports the existing source.
+
+With no `--product`, the endpoint names a _new_ product after the (often verbose) App Store title — e.g. "Shopify: Sell online/in person". To control the name, create the product first and reference it with `--product`:
+
+```bash
+releases admin product create "Shopify" --org shopify
+releases admin source create-appstore https://apps.apple.com/us/app/shopify/id719892358 --org shopify --product shopify
+```
+
+Add one app at a time — the listing is resolved on the fly, and concurrent creates for a brand-new org/product race on slug uniqueness.
+
 ### Update
 
 ```bash
@@ -96,6 +128,47 @@ Notes:
 - Remote concurrency defaults to 3, capped at 5. Duplicate source fetches are detected and blocked.
 - Smart fetch backoff: sources returning no changes back off exponentially (1h → 48h); error backoff caps at 72h.
 
+### Backfill (full history)
+
+Walk every scrape window of a windowed `scrape` source and upsert the whole history at once — the turnkey replacement for bespoke per-source backfill scripts. Idempotent (dedups by synthesized URL), and **dry-run by default**:
+
+```bash
+releases admin source backfill my-source                    # preview: counts + date range, nothing written
+releases admin source backfill my-source --no-dry-run        # write (or --commit)
+releases admin source backfill my-source --max-windows 100   # walk further back (endpoint clamps 1–200, default 50)
+releases admin source backfill my-source --wait              # deep Firecrawl backfill: block until the async workflow finishes
+releases admin source backfill my-source --markdown-file page.md --commit
+cat page.md | releases admin source backfill my-source --markdown-file - --commit
+```
+
+Notes:
+
+- Accepts a slug or `src_…` ID; the CLI resolves to the typed ID before calling (the endpoint rejects bare slugs as ambiguous across orgs).
+- `--markdown-file` supplies the full-page markdown for JS-heavy / bot-blocked sources the worker can't fetch itself. Without it the endpoint falls back to Firecrawl (if enabled on the source) then a plain fetch.
+- Scrape sources only. Non-scrape sources, an unfetchable body, or a missing `ANTHROPIC_API_KEY`/`FIRECRAWL_API_KEY` come back as a clear error.
+- A dry run reports `windows`, `extracted → unique`, and the date range; it warns if it hit the window cap (raise `--max-windows`).
+- Deep Firecrawl backfills run as a durable workflow (minutes). Like `admin overview batch`, the CLI **dispatches and returns the workflow instance ID by default** (non-blocking — the agent-friendly default), then either:
+  - poll it yourself: `releases admin source backfill-status <instanceId> [--json]` (single-shot; loop on the `--json` `status` field), or
+  - pass `--wait` to block and render the report inline.
+- Non-Firecrawl / `--markdown-file` sources stay **synchronous** — the report comes back in one call regardless of `--wait`.
+
+### Re-extract (from a stored snapshot)
+
+Re-run extraction over a source's captured raw body in R2 (`released-raw`) — **no live scrape, no Firecrawl credits, deterministic input**. Use it after extraction/parse logic improves to reprocess a source's history. Sibling of `backfill`, **dry-run by default**:
+
+```bash
+releases admin source reextract my-source                          # preview from the latest snapshot
+releases admin source reextract my-source --commit                 # write (or --no-dry-run)
+releases admin source reextract my-source --snapshot-id raw_abc123 --commit   # pin a specific capture
+releases admin source reextract my-source --max-windows 100 --json
+```
+
+Notes:
+
+- Slug or `src_…` ID, resolved to the typed ID before calling (bare slugs rejected). Scrape sources only.
+- Omitting `--snapshot-id` uses the most recent capture; the report's `snapshot` block names which one was used.
+- Actionable errors surface as-is: `no_snapshot`/`snapshot_not_found` (404, none stored / wrong id), `snapshot_expired` (410, body past the 90-day R2 lifecycle — re-scrape to capture fresh), missing `RAW_SNAPSHOTS`/`ANTHROPIC_API_KEY` (503).
+
 ### Poll (cheap change detection)
 
 ```bash
@@ -133,10 +206,14 @@ releases admin org link vercel --platform github --handle vercel
 releases admin org tag add vercel react serverless
 releases admin org alias add anthropic claude.ai claude.com
 releases admin org refresh vercel                         # fetch all sources + regenerate overview
+releases admin org delete vercel                          # reversible tombstone soft-delete
+releases admin org delete vercel --hard --yes             # permanent purge + FK cascade
 ```
 
 `org refresh` flags: `--max <n>` (per-source cap, default 20), `--concurrency <n>`, `--window <days>`, `--dry-run`, `--skip-overview`, `--json`.
 
+`org delete` soft-deletes by default (a reversible tombstone). `--hard` purges the row and cascade-deletes every dependent source, release, fetch-log, changelog file/chunk, summary, media asset, and webhook subscription; it prompts for a slug typeback unless `--yes` is passed (required in non-TTY/scripted contexts). You can pass a slug or an `org_…` ID either way — the CLI resolves to the typed ID the destructive path requires.
+
 ## Products
 
 Products group sources under multi-product orgs (e.g. Vercel → Next.js, Turborepo, v0):