crawlmark-mcp 0.1.0 → 0.1.1

package/.env.example

@@ -47,14 +47,14 @@ AHREFS_API_KEY=
 AHREFS_MONTHLY_UNIT_CAP=50000
 DEFAULT_CLIENT_MONTHLY_AHREFS_UNITS=10000
 
-# Semrush (optional): keyword volumes, rankings, competitors and backlinks from Semrush. Needs an SEO Toolkit Business
-# plan plus a separately bought API unit package (the balance is zero right after the upgrade).
-# SEMRUSH_API_KEY is the v3 key, under Subscription info > API units; it is used for every v3 report and the free
-# balance check. `crawlmark-cli setup --step semrush` checks it for free and saves it.
+# Semrush (optional): keyword volume (v4 Get Keyword Metrics, 20 units per keyword) and backlinks (v4 overview) from Semrush.
+# Semrush's v4 API has nothing else, so rankings, competitors and visibility come from DataForSEO or Ahrefs. The API needs
+# a plan that includes API units (50,000 come with SEO Classic Pro and Guru, and Semrush One Starter and Pro+) or a bought
+# unit package. SEMRUSH_API_KEY is a v4 key, from your profile > API keys; Semrush's older v3 keys aren't supported.
+# Semrush has no free key check, so the key is checked by the first paid call. `crawlmark-cli setup --step semrush` saves it.
+# (SEMRUSH_V4_API_KEY, the name 0.1.0 used for the v4 key, is still read and wins if both are set: in 0.1.0 SEMRUSH_API_KEY
+# held the old v3 key. `setup --step semrush` saves SEMRUSH_API_KEY and removes SEMRUSH_V4_API_KEY.)
 SEMRUSH_API_KEY=
-# Optional v4 key (your Semrush profile > API keys). When set, backlinks use the v4 overview; when unset, the older v3
-# report. Semrush has no free check for v4 keys, so it is first checked when backlinks_summary runs.
-SEMRUSH_V4_API_KEY=
 # Semrush API units crawlmark may use per calendar month (UTC), across all clients, and per client by default
 # (a client's monthlyUnits.semrush in clients.yaml overrides the per-client value).
 SEMRUSH_MONTHLY_UNIT_CAP=20000
@@ -65,7 +65,6 @@ DEFAULT_CLIENT_MONTHLY_SEMRUSH_UNITS=5000
 # 127.0.0.1 / localhost / [::1] only; the server and `setup` print a stderr line while one is in use.
 # AHREFS_BASE_URL=
 # SEMRUSH_BASE_URL=
-# SEMRUSH_BALANCE_URL=
 # CRUX_BASE_URL=
 # BING_BASE_URL=
 # CRAWLER_IPRANGES_BASE_URL=

package/README.md

@@ -10,7 +10,8 @@ It combines:
 | GA4 Data API | Organic sessions, engagement, key events, landing pages | Free |
 | Google Tag Manager | Live container contents for tag audits | Free |
 | DataForSEO | Keyword volume, live SERPs, ranked keywords, competitors, backlinks, site crawls | Pay per call |
-| Ahrefs or Semrush (beta) | Keyword volume, ranked keywords, competitors and backlinks, with your own API key | Your plan's API units |
+| Ahrefs (beta) | Keyword volume, ranked keywords, competitors and backlinks, with your own API key | Your plan's API units |
+| Semrush (beta) | Keyword volume and backlinks, with your own v4 API key (its v4 API has nothing else) | Your plan's API units |
 | Lighthouse | Performance, SEO and best-practice scores, lab Core Web Vitals | Free (runs locally) |
 | PageSpeed Insights | Real-user field data (CrUX) | Free with an API key (optional) |
 | Chrome UX Report History | Real-user Core Web Vitals week by week: 25 weeks by default, up to 40 | Free with an API key (optional) |
@@ -26,7 +27,7 @@ Install crawlmark
 npx crawlmark-mcp@latest init
 ```
 
-That one command sets up crawlmark and adds it to your AI apps: **Claude Code, Claude Desktop, Cursor and Codex**. It needs **Node 22.13 or later** (crawlmark uses the built-in `node:sqlite`). Other ways to install are below; whichever you pick, [set up your credentials](#set-up-credentials) next.
+That one command sets up crawlmark and adds it to your AI apps: **Claude Code, Claude Desktop, Cursor and Codex**. `npm install crawlmark-mcp` on its own only downloads the package and doesn't start setup; run `init` with `npx` as above (or install globally with `npm i -g crawlmark-mcp` and run `crawlmark-mcp init`). It needs **Node 22.13 or later** (crawlmark uses the built-in `node:sqlite`). Other ways to install are below; whichever you pick, [set up your credentials](#set-up-credentials) next.
 
 | System or use | Installation method |
 |---|---|
@@ -53,7 +54,7 @@ Every change is shown before it's made, other servers are listed by name only, a
 
 In a terminal you get arrow-key choices, masked secrets, a spinner with the time taken for each live check, and a short tip under each result. Links are clickable in terminals that support it (iTerm2, WezTerm, VS Code, Ghostty, Windows Terminal, kitty and recent GNOME terminals; `FORCE_HYPERLINK=1` or `0` overrides), and the steps that need a web page (DataForSEO API access, Ahrefs and Semrush keys, Google Cloud, PageSpeed) offer to open it with Enter. The summary box ends with three things to try asking Claude, using up to three of your own clients from `clients.yaml` (read locally and only shown in your terminal; `example.com` when you have none). With piped input, `CI=true`, `TERM=dumb`, `NO_COLOR` or `--yes`, the same steps run as plain numbered prompts without any of that. `init` and `setup` are the same command.
 
-Run `init` again once crawlmark is set up (a `.env` in its config home and at least one working data source) and it shows a menu instead of every step: add or change a data provider, change the Google sign-in, re-register your AI apps, refresh clients, set up report branding, check status, run the full setup again, or exit. `setup --step <name>` still goes straight to one step.
+Run `init` again once crawlmark is set up (a `.env` in its config home and at least one working data source) and it shows a menu instead of every step: add or change a data provider, change the Google sign-in, re-register your AI apps, add Semrush's or Ahrefs' own MCP server, refresh clients, set up report branding, check status, run the full setup again, or exit. `setup --step <name>` still goes straight to one step.
 
 `init`, `setup`, `--help` and `--version` check npm for a newer crawlmark at most once a day (a plain request for the latest version number, with a 1.5 second timeout and nothing about you or your clients) and print `Update available` when there is one. It never runs in CI, when output isn't a terminal, or with `--json`. Set `CRAWLMARK_NO_UPDATE_CHECK=1` (or the common `NO_UPDATE_NOTIFIER=1`) to turn it off; the MCP server itself never checks.
 
@@ -65,7 +66,7 @@ crawlmark prefers puppeteer's `chrome-headless-shell` (run `npx puppeteer browse
 
 ### Claude Desktop bundle (MCPB)
 
-Download `crawlmark.mcpb` from the release page and double-click it, or drag it onto Claude Desktop (**Settings > Extensions**). Claude Desktop shows a form. Every field is optional, so fill in only what you use:
+Download `crawlmark.mcpb` from the [latest release](https://github.com/idrewlong/crawlmark-mcp/releases/latest) and double-click it, or drag it onto Claude Desktop (**Settings > Extensions**). Claude Desktop shows a form. Every field is optional, so fill in only what you use:
 
 | Field | What it is |
 |---|---|
@@ -73,7 +74,7 @@ Download `crawlmark.mcpb` from the release page and double-click it, or drag it
 | Use the live DataForSEO API | Off (default) uses the free sandbox with sample data. On uses real data, billed to your balance |
 | Default SEO data provider | `dataforseo`, `ahrefs` or `semrush`. Blank uses the value in your crawlmark `.env` (from `setup --step provider`), else `dataforseo`. Any other value is reported by `setup_status` |
 | Ahrefs API key | API v3 key ([Ahrefs](#ahrefs)) |
-| Semrush API key (v3), Semrush API key (v4, optional) | [Semrush](#semrush) keys; the v4 key is only for the newer backlinks report |
+| Semrush API key (v4) | The [Semrush](#semrush) key, from your profile > API keys |
 | Ahrefs / Semrush monthly unit cap | API units crawlmark may use per month across all clients. Defaults 50,000 and 20,000 |
 | Google service-account key file | The JSON key from [option C](#option-c-service-account) |
 | Google OAuth client ID / secret | Only for signing in with your own Google account ([options A and B](#options-a-and-b-oauth)). After installing, sign in once from a terminal: `CRAWLMARK_HOME=<data folder> npx crawlmark-mcp setup --step google` |
@@ -105,7 +106,7 @@ The setup wizard covers everything here. Each step shows its current status firs
 | Command | Does |
 |---|---|
 | `crawlmark-cli init` (or `setup`) | The whole wizard |
-| `crawlmark-cli setup --step <node\|provider\|dataforseo\|ahrefs\|semrush\|google\|psi\|discovery\|register>` | One step. `provider` picks your SEO data providers and the default one |
+| `crawlmark-cli setup --step <node\|provider\|dataforseo\|ahrefs\|semrush\|google\|psi\|discovery\|register\|vendor-mcp>` | One step. `provider` picks your SEO data providers and the default one; `vendor-mcp` adds Semrush's or Ahrefs' own MCP server to your AI apps (see [below](#semrushs-and-ahrefs-own-mcp-servers-optional)) |
 | `crawlmark-cli setup --step crux` | A key for Core Web Vitals history (CrUX). Only run this way; it isn't part of the default wizard |
 | `crawlmark-cli setup --step bing` | A Bing Webmaster Tools API key. Only run this way; it isn't part of the default wizard |
 | `crawlmark-cli setup --step branding [--client <domain>]` | Report branding: agency name, logo, colours, footer, white label, paper size and default formats, with a one-page preview before saving. Only run this way (or from the setup menu) |
@@ -164,38 +165,47 @@ By hand, set `AHREFS_API_KEY` (and `SEO_DATA_PROVIDER=ahrefs` to make it the def
 
 > **Beta.** The Semrush integration is built to the official API documentation and tested against the documented example responses, but it hasn't been verified against a live account yet. Cost caps and key handling are fully tested. If anything looks wrong (an error you don't expect, a missing field, units that don't match your dashboard), please [open an issue](https://github.com/idrewlong/crawlmark-mcp/issues).
 
-Semrush is optional. Its API needs an **SEO Toolkit Business** plan **plus** a separately bought **API unit package**; the unit balance is zero until you buy one.
+Semrush is optional. crawlmark uses Semrush's **v4 API** with a v4 key; Semrush's older v3 keys aren't supported. Semrush's v4 SEO API has only two things crawlmark uses:
 
-1. Find your **v3 API key** under **Subscription info > API units**. crawlmark uses it for keyword volume, ranked keywords and competitors.
-2. Optionally create a **v4 key** in your profile under **API keys**. With it, backlinks use Semrush's newer v4 overview (45 units); without it, the older v3 report (40 units).
-3. Run `crawlmark-cli setup --step semrush`. It checks the v3 key with Semrush's free balance call and shows the units left ("key accepted": the check proves the key exists, not that your plan includes the API). A v4 key has no free check, so it's saved untested and checked the first time `backlinks_summary` runs.
+| What crawlmark does | From Semrush |
+|---|---|
+| `keyword_volume` | v4 Get Keyword Metrics, 20 units per keyword (Early Access per Semrush, so it may change) |
+| `backlinks_summary` and the `backlinks` report section | v4 backlinks overview, 45 units; no history (stored figures only) |
+| Everything else: `ranked_keywords`, `competitor_domains`, `organic_visibility`, `competitor_overview`, `keyword_ideas`, `keyword_gap`, Site Audit in `technical_issues` | Not in Semrush's v4 API. DataForSEO serves them when it is set up (the result says `fallbackFrom: semrush`); otherwise the call says to use DataForSEO or Ahrefs |
+
+API units come with the plan: Semrush says 50,000 units are included with SEO Classic Pro and Guru and with Semrush One Starter and Pro+, and SEO Classic Business or Semrush One Advanced can add unit packages.
+
+1. Create a **v4 key** in your Semrush profile under **API keys** (up to 100, revocable, with an expiry).
+2. Run `crawlmark-cli setup --step semrush` and paste it. Semrush has no free key check, so the key is saved as it is and checked by the first paid call (keyword volume or backlinks). A Semrush rejection on that call says to create a v4 key and re-run the step.
 
-By hand, set `SEMRUSH_API_KEY` (and `SEMRUSH_V4_API_KEY`, and `SEO_DATA_PROVIDER=semrush` to make it the default). The `semrush_status` tool shows the balance.
+By hand, set `SEMRUSH_API_KEY` (and `SEO_DATA_PROVIDER=semrush` to make it the default). If you used 0.1.0, a key saved as `SEMRUSH_V4_API_KEY` is still read, and if both are set the `SEMRUSH_V4_API_KEY` one wins, because in 0.1.0 `SEMRUSH_API_KEY` held the old v3 key. Re-running the step saves the key as `SEMRUSH_API_KEY` and removes `SEMRUSH_V4_API_KEY` from `.env`. The `semrush_status` tool and `setup status` show whether the key is set and what it serves.
+
+In v4 keyword metrics the CPC unit and the competition scale aren't documented, so CPC is passed through as Semrush returns it and the result carries a note saying so.
 
 Semrush's API terms don't allow keeping its data longer than a month, so crawlmark deletes stored Semrush figures 31 days after fetching them.
 
 ### Data providers
 
-Keyword volume, rankings, competitors, backlinks, keyword ideas and the competitor keyword gap can come from any of the three providers. DataForSEO is the default and is verified against its live API; Ahrefs and Semrush are in beta (see above). Live SERPs, on-page checks and crawls come from DataForSEO only.
+Keyword volume and backlinks can come from any of the three providers; rankings, competitors, visibility, keyword ideas and the competitor keyword gap from DataForSEO or Ahrefs (Semrush's v4 API doesn't have them, so a Semrush client is served by DataForSEO for those when it is set up). DataForSEO is the default and is verified against its live API; Ahrefs and Semrush are in beta (see above). Live SERPs, on-page checks and crawls come from DataForSEO only.
 
 | Capability | DataForSEO | Ahrefs | Semrush |
 |---|---|---|---|
-| `keyword_volume` | Google Ads volume | Keywords Explorer (12-month average) | Batch keyword overview (12-month average) |
-| `ranked_keywords` | Labs ranked keywords | Site Explorer organic keywords | Domain organic search keywords |
-| `competitor_domains` | Labs competitors | Site Explorer organic competitors | Competitors in organic search |
-| `backlinks_summary` | Backlinks summary (rank, spam score) | Backlinks stats plus Domain Rating | Backlinks overview (v4 with a v4 key, else v3) plus authority score |
-| `organic_visibility`, `organic_visibility` report section | Labs domain rank overview (position bands, estimated traffic); history: historical rank overview | Metrics plus keywords history (position bands, organic traffic); history: keywords and metrics history | Domain overview (keywords, traffic; no position bands); history: domain overview history |
-| `backlinks` report section | Backlinks summary; history: backlinks history (monthly, with new and lost referring domains) | Backlinks stats plus Domain Rating; history: referring domains and Domain Rating history | Backlinks overview; no history (stored figures only) |
-| `competitor_overview`, `competitor_overview` report section | Labs competitors with each competitor's whole footprint | Organic competitors | Competitors in organic search |
-| `technical_issues` sources (paid, opt-in) | A stored full crawl's summary (free) | Site Audit issues of your existing project | Site Audit of your existing project |
-| `keyword_ideas` | Labs keyword ideas (`related`) or keyword suggestions (`matching`) | Keywords Explorer related terms or matching terms | Related keywords or broad match |
-| `keyword_gap` | Labs domain intersection | Each competitor's organic keywords, minus what the client already gets Search Console impressions for | Domain vs Domain |
+| `keyword_volume` | Google Ads volume | Keywords Explorer (12-month average) | Get Keyword Metrics (v4, 20 units per keyword) |
+| `ranked_keywords` | Labs ranked keywords | Site Explorer organic keywords | Not available (DataForSEO or Ahrefs) |
+| `competitor_domains` | Labs competitors | Site Explorer organic competitors | Not available (DataForSEO or Ahrefs) |
+| `backlinks_summary` | Backlinks summary (rank, spam score) | Backlinks stats plus Domain Rating | Backlinks overview (v4) plus authority score |
+| `organic_visibility`, `organic_visibility` report section | Labs domain rank overview (position bands, estimated traffic); history: historical rank overview | Metrics plus keywords history (position bands, organic traffic); history: keywords and metrics history | Not available (DataForSEO or Ahrefs) |
+| `backlinks` report section | Backlinks summary; history: backlinks history (monthly, with new and lost referring domains) | Backlinks stats plus Domain Rating; history: referring domains and Domain Rating history | Backlinks overview (v4); no history (stored figures only) |
+| `competitor_overview`, `competitor_overview` report section | Labs competitors with each competitor's whole footprint | Organic competitors | Not available (DataForSEO or Ahrefs) |
+| `technical_issues` sources (paid, opt-in) | A stored full crawl's summary (free) | Site Audit issues of your existing project | Not available |
+| `keyword_ideas` | Labs keyword ideas (`related`) or keyword suggestions (`matching`) | Keywords Explorer related terms or matching terms | Not available (DataForSEO or Ahrefs) |
+| `keyword_gap` | Labs domain intersection | Each competitor's organic keywords, minus what the client already gets Search Console impressions for | Not available (DataForSEO or Ahrefs) |
 | `serp_check`, `check_ranks` with `live`, on-page checks, crawls | Yes | No (DataForSEO is used) | No (DataForSEO is used) |
 | Account check | `dataforseo_status` | `ahrefs_status` | `semrush_status` |
 
 - **The default.** `SEO_DATA_PROVIDER` (`dataforseo`, `ahrefs` or `semrush`; blank means `dataforseo`) picks the default. `crawlmark-cli setup --step provider` sets it, and so does the bundle's "Default SEO data provider" field. An unknown value doesn't stop the server: `setup_status` and every paid call say what's wrong, and free tools keep working.
 - **Per client.** A client's `dataProvider` in clients.yaml overrides the default: `crawlmark-cli clients set example.com --provider ahrefs` (or `none` to go back to the default), or `set_client_profile`. Ahrefs and Semrush need the client's market as a country: set `country` (`--country GB`), or it comes from a country-level `locationCode`.
-- **No silent fallback.** If a client's provider has no key, its paid calls fail with the command to fix it; crawlmark never spends on another account instead. If the provider lacks a capability (live SERPs, on-page checks, crawls), DataForSEO serves it when it's set up, and the result says `fallbackFrom`.
+- **No silent fallback.** If a client's provider has no key, its paid calls fail with the command to fix it; crawlmark never spends on another account instead. If the provider lacks a capability (live SERPs, on-page checks, crawls, or for Semrush anything but keyword volume and backlinks), DataForSEO serves it when it's set up, and the result says `fallbackFrom`.
 - **Units, not dollars.** DataForSEO is capped in USD. Ahrefs and Semrush are capped in their own API units, per month (UTC): `AHREFS_MONTHLY_UNIT_CAP` (default 50,000) and `SEMRUSH_MONTHLY_UNIT_CAP` (20,000) across all clients, and `DEFAULT_CLIENT_MONTHLY_AHREFS_UNITS` (10,000) and `DEFAULT_CLIENT_MONTHLY_SEMRUSH_UNITS` (5,000) per client, or a client's `monthlyUnits`. Each call checks its estimate first; units are never converted to dollars.
 - **Metrics differ.** Every result carries `provider`, and `backlinks_summary` labels the domain-strength number in `authority.metric` (`dataforseo_rank`, `domain_rating` or `semrush_score`). They aren't comparable across providers.
 
@@ -331,6 +341,62 @@ args = ["-y", "crawlmark-mcp"]
 
 For a source checkout, use `"command": "node"` and `"args": ["--disable-warning=ExperimentalWarning", "/absolute/path/to/crawlmark-mcp/dist/index.js"]`. Claude Desktop, Cursor and Codex may not load your shell's `PATH`, so if one can't find `node` or `npx`, use the absolute path (`which node`). Credentials come from `.env` in the config home, so no `env` block is needed unless your settings live somewhere other than the default; then add `"env": { "CRAWLMARK_HOME": "/path/to/home" }`.
 
+### Semrush's and Ahrefs' own MCP servers (optional)
+
+Semrush and Ahrefs each host their own MCP server. You can add either to your AI app next to crawlmark, to ask the vendor's tools ad-hoc questions. This is separate from crawlmark's own Semrush and Ahrefs support:
+
+| | crawlmark's Semrush / Ahrefs | The vendor's own MCP server |
+|---|---|---|
+| Sign-in | An API key in your crawlmark `.env` | OAuth, in your AI app |
+| Good for | Reports, cost caps, caching and history | Ad-hoc questions in chat |
+| Units | Counted against crawlmark's monthly caps | Spent directly from your account; crawlmark's caps, cache and reports don't cover them |
+
+crawlmark itself never calls these servers; it only adds them to your AI apps. For Ahrefs, that is what its terms ask for: per [Ahrefs' docs](https://docs.ahrefs.com/docs/mcp/introduction), its MCP endpoint is for interactive use in MCP-compatible clients, and scripts or other programmatic access must use the public API, which is what crawlmark's own Ahrefs features use (REST API v3). Ahrefs' MCP is available on paid plans from Lite and draws from the same monthly API unit pool as the REST API; connecting through OAuth creates an MCP-tagged key with a default limit of 200,000 units a month, which workspace admins can change in the API keys report. Semrush says its MCP calls use the same API units as its API, with no extra fee ([Semrush MCP docs](https://developer.semrush.com/api/basics/semrush-mcp/)).
+
+```bash
+crawlmark-cli setup --step vendor-mcp
+```
+
+The step asks which servers to add (Semrush and Ahrefs, pre-ticked if you picked them as data providers), then which of your detected apps. Each change is previewed and asked about first, the config file is kept as a `.bak`, other servers are never touched, and an entry that is already there is left alone. If an entry named `semrush` or `ahrefs` already exists with a different setting, it is replaced only if you say yes (the default is no). No API key is ever written to an app's config. In a normal `init`, if you picked Semrush or Ahrefs as a provider, it asks once after the register step whether to also add them (default no; never asked under `--yes` or in plain mode).
+
+To add them by hand (replace the name and URL for Ahrefs: `ahrefs`, `https://api.ahrefs.com/mcp/mcp`):
+
+**Claude Code:**
+
+```bash
+claude mcp add --transport http -s user semrush https://mcp.semrush.com/v2/mcp
+```
+
+Then run `/mcp` in Claude Code and sign in.
+
+**Cursor** (`~/.cursor/mcp.json`):
+
+```json
+{
+  "mcpServers": {
+    "semrush": { "url": "https://mcp.semrush.com/v2/mcp" }
+  }
+}
+```
+
+**Codex:**
+
+```bash
+codex mcp add semrush --url https://mcp.semrush.com/v2/mcp
+codex mcp login semrush
+```
+
+or in `~/.codex/config.toml`:
+
+```toml
+[mcp_servers.semrush]
+url = "https://mcp.semrush.com/v2/mcp"
+```
+
+**Claude Desktop** takes remote servers in its settings, not in `claude_desktop_config.json`: open **Settings > Connectors > Add custom connector**, enter a name (Semrush) and the URL, save, and sign in when asked.
+
+If a client can't do OAuth, Semrush also accepts the header `Authorization: Apikey <key>`; the wizard never writes that for you, so add it yourself if you need it.
+
 Once connected, run the **`getting-started`** prompt. It checks what's set up, explains what crawlmark can do with it, and offers a first report for your busiest client.
 
 Tools
@@ -384,7 +450,7 @@ Branded queries are excluded by default. See [docs/audits.md](docs/audits.md#opp
 | `site_hygiene` | robots.txt, sitemaps, structured data, hreflang (page tags, sitemap entries and `Link` headers), orphan pages (from the newest complete local or DataForSEO crawl), AI crawler access in robots.txt (AI search vs training-only crawlers) and llms.txt (informational: Google says it doesn't use it). Free |
 | `local_crawl` | Crawls the client's own site from your machine (free): status codes, redirects and chains, titles, meta descriptions, canonicals, H1s, internal links, depth, broken links, noindex pages and orphans against the sitemaps. Obeys robots.txt and Crawl-delay; same host only by default; up to 2,000 pages. Saved for `local_crawl_pages`, `site_hygiene` and audits |
 | `local_crawl_pages` | Pages from a saved local crawl, filtered by issue or status code. Free, local |
-| `technical_issues` | One deduplicated, severity-grouped list of technical problems from the saved audit, its site checks, the newest local crawl, a stored DataForSEO crawl and stored URL Inspection results, with the sources per issue and what's new, resolved or not re-checked since the last snapshot. Never starts a crawl. Free; your existing Ahrefs or Semrush Site Audit project can be added as a paid source with `allowPaid` (read-only) ([details](docs/audits.md#technical-issues)) |
+| `technical_issues` | One deduplicated, severity-grouped list of technical problems from the saved audit, its site checks, the newest local crawl, a stored DataForSEO crawl and stored URL Inspection results, with the sources per issue and what's new, resolved or not re-checked since the last snapshot. Never starts a crawl. Free; your existing Ahrefs Site Audit project can be added as a paid source with `allowPaid` (read-only) ([details](docs/audits.md#technical-issues)) |
 | `index_coverage` | Google's indexing of a sample of the client's URLs from stored URL Inspection results: indexed share, coverage states, the worst URLs, canonical mismatches, sitemap errors, the monthly trend and how many sitemap URLs were checked recently (or that all were). Free, local ([details](docs/audits.md#indexing-coverage)) |
 | `import_index_report` | Imports an export of Search Console's Pages report (Indexing > Pages > Export: the zip of CSVs, an Excel workbook or one CSV) so `index_coverage` and the indexing report section show Google's complete counts per reason, with example URLs from a reason's own export. Reads only `IMPORT_DIRS` or `<home>/imports`; parsed on your machine. Free ([details](docs/audits.md#importing-search-consoles-pages-report)) |
 | `inspect_index_coverage` | Inspects key pages, top Search Console pages and a weekly-rotating sample of sitemap URLs with URL Inspection (free, read-only) within a daily per-property budget, and stores the results. `all: true` checks every page when today's budget allows, else says how many daily runs a full pass needs |
@@ -452,7 +518,7 @@ All three take `estimateOnly: true` (the estimate and what's left of the caps, w
 | `backlinks_summary` | **Paid** (the client's data provider). Backlink totals for a domain |
 | `dataforseo_status` | Credentials check, sandbox or live mode, and balance. Free |
 | `ahrefs_status` | Ahrefs key check, plan, units used and reset date. Free |
-| `semrush_status` | Semrush key check, units left, and whether a v4 key is set. Free |
+| `semrush_status` | Whether the Semrush v4 key is set, what Semrush serves in crawlmark, and units used this month (no free key check exists). Free |
 
 Paid results carry `provider` (whose data it is) and `units` (Ahrefs or Semrush API units used; `null` for DataForSEO, which reports `costUsd`).
 
@@ -579,7 +645,7 @@ The config home holds `.env`, `config/clients.yaml`, `config/report-templates.ya
 | `DATAFORSEO_LIVE` | `false` | `true` selects the live API when no base URL is set |
 | `SEO_DATA_PROVIDER` | `dataforseo` | Default provider for keyword, ranking, competitor and backlink data: `dataforseo`, `ahrefs` or `semrush` |
 | `AHREFS_API_KEY` | | Ahrefs API v3 key |
-| `SEMRUSH_API_KEY`, `SEMRUSH_V4_API_KEY` | | Semrush v3 key, and the optional v4 key for backlinks |
+| `SEMRUSH_API_KEY` | | Semrush v4 key (profile > API keys). `SEMRUSH_V4_API_KEY`, the 0.1.0 name for the v4 key, is still read and wins if both are set (in 0.1.0 this variable held the old v3 key) |
 | `AHREFS_MONTHLY_UNIT_CAP`, `SEMRUSH_MONTHLY_UNIT_CAP` | `50000`, `20000` | Monthly API units across all clients |
 | `DEFAULT_CLIENT_MONTHLY_AHREFS_UNITS`, `DEFAULT_CLIENT_MONTHLY_SEMRUSH_UNITS` | `10000`, `5000` | Monthly API units per client (clients.yaml `monthlyUnits` overrides) |
 | `PSI_API_KEY` | | PageSpeed Insights key |

package/dist/analysis/tech-issues.js

@@ -3,16 +3,16 @@ const SEVERITIES = ["critical", "high", "medium", "low"];
 /**
  * The starter mapping (spec 13.5.3). crawlmark sources: `audit:<finding source>:<rule>` for the saved audit's
  * findings, `local_crawl:<rule>`, `site_hygiene:<rule>`, `gsc_inspection:<rule>`; DataForSEO crawl checks:
- * `dataforseo_crawl:<check>`. Ahrefs names and Semrush IDs are filled in by Stage B (research §1-2).
+ * `dataforseo_crawl:<check>`. Ahrefs names are filled in by Stage B (research §1).
  */
 export const CATALOGUE = [
-    { key: "broken-internal-link", title: "Internal link to a broken page", category: "links", match: ["local_crawl:broken-internal-link", "dataforseo_crawl:is_broken"], ahrefs: ["Page has links to broken page"], semrush: [8] },
-    { key: "server-error", title: "Page returns a server error (5xx)", category: "crawlability", match: ["audit:onpage:is_5xx_code", "dataforseo_crawl:is_5xx_code", "site_hygiene:sitemap-non200:5xx"], ahrefs: ["5XX page"], semrush: [1] },
-    { key: "client-error-page", title: "Page returns a client error (4xx)", category: "crawlability", match: ["audit:onpage:is_4xx_code", "dataforseo_crawl:is_4xx_code", "site_hygiene:sitemap-non200:4xx"], ahrefs: ["4XX page"], semrush: [2] },
+    { key: "broken-internal-link", title: "Internal link to a broken page", category: "links", match: ["local_crawl:broken-internal-link", "dataforseo_crawl:is_broken"], ahrefs: ["Page has links to broken page"] },
+    { key: "server-error", title: "Page returns a server error (5xx)", category: "crawlability", match: ["audit:onpage:is_5xx_code", "dataforseo_crawl:is_5xx_code", "site_hygiene:sitemap-non200:5xx"], ahrefs: ["5XX page"] },
+    { key: "client-error-page", title: "Page returns a client error (4xx)", category: "crawlability", match: ["audit:onpage:is_4xx_code", "dataforseo_crawl:is_4xx_code", "site_hygiene:sitemap-non200:4xx"], ahrefs: ["4XX page"] },
     { key: "missing-title", title: "Page has no title", category: "content", match: ["local_crawl:missing-title", "audit:onpage:no_title", "dataforseo_crawl:no_title"], ahrefs: ["Title tag missing or empty"] },
     { key: "duplicate-title", title: "Title is used by another page", category: "duplicates", match: ["local_crawl:duplicate-title", "audit:onpage:duplicate_title_tag", "dataforseo_crawl:duplicate_title_tag"], ahrefs: ["Multiple pages have same title"] },
     { key: "missing-description", title: "Page has no meta description", category: "content", match: ["local_crawl:missing-description", "audit:onpage:no_description", "dataforseo_crawl:no_description"], ahrefs: ["Meta description tag missing or empty"] },
-    { key: "missing-h1", title: "Page has no H1 heading", category: "content", match: ["audit:onpage:no_h1_tag", "dataforseo_crawl:no_h1_tag"], ahrefs: ["H1 tag missing or empty"], semrush: [103] },
+    { key: "missing-h1", title: "Page has no H1 heading", category: "content", match: ["audit:onpage:no_h1_tag", "dataforseo_crawl:no_h1_tag"], ahrefs: ["H1 tag missing or empty"] },
     { key: "redirect-chain", title: "Redirect chain (two or more hops)", category: "redirects", match: ["local_crawl:redirect-chain", "dataforseo_crawl:redirect_chain"], ahrefs: ["Redirect chain"] },
     { key: "noindex-in-sitemap", title: "Sitemap lists a noindex page", category: "indexability", match: ["site_hygiene:sitemap-noindex"], ahrefs: ["Noindex page in sitemap"] },
     { key: "canonical-mismatch", title: "Google chose a different canonical than the page declares", category: "indexability", match: ["audit:indexing:canonical-mismatch", "gsc_inspection:canonical-mismatch"] },
@@ -37,17 +37,13 @@ function matchKey(r) {
 }
 /**
  * Stage B: an Ahrefs Site Audit issue matches a row by its name (case-insensitive prefix: Ahrefs doesn't document its
- * issue IDs, research §1); a Semrush Site Audit issue by its documented numeric ID (S2). Unmatched ones keep their own key.
+ * issue IDs, research §1). Unmatched ones keep their own key.
  */
 function providerRow(r) {
     if (r.source === "ahrefs_site_audit") {
         const name = r.rule.toLowerCase();
         return CATALOGUE.find((row) => row.ahrefs?.some((a) => name.startsWith(a.toLowerCase())));
     }
-    if (r.source === "semrush_site_audit") {
-        const id = Number(r.rule);
-        return Number.isInteger(id) ? CATALOGUE.find((row) => row.semrush?.includes(id)) : undefined;
-    }
     return undefined;
 }
 /** Canonical key, title and category for a raw issue: its catalogue row, axe rules per rule, else `<source>:<rule>`. */

package/dist/cli-help.js

@@ -26,7 +26,7 @@ export function topHelp(o) {
         heading("Get started", o),
         "  init                  Set up credentials and clients, register your AI apps",
         "  setup --step <name>   Re-run one step: node, provider, dataforseo, ahrefs,",
-        "                        semrush, google, psi, discovery or register",
+        "                        semrush, google, psi, discovery, register or vendor-mcp",
         "  setup status          What's configured and what to do next",
         "  setup migrate --to    Copy your settings to another folder",
         heading("Google", o),
@@ -66,8 +66,10 @@ const GROUP_TEXT = {
         "  Claude Desktop, Cursor and Codex. Re-runnable; each step shows its status",
         "  first and can be skipped. init and setup are the same command.",
         "  --step <name>   Run one step: node, provider, dataforseo, ahrefs, semrush,",
-        "                  google, psi, discovery or register. provider picks your SEO",
-        "                  data providers and the default one. crux (a key for Core",
+        "                  google, psi, discovery, register or vendor-mcp. provider",
+        "                  picks your SEO data providers and the default one.",
+        "                  vendor-mcp adds Semrush's or Ahrefs' own MCP server to",
+        "                  your AI apps (OAuth, optional). crux (a key for Core",
         "                  Web Vitals history), bing (a Bing Webmaster Tools key) and",
         "                  branding (report logo, colours, footer, white label and",
         "                  default formats, with a one-page preview; --client <domain>",
@@ -111,7 +113,6 @@ const GROUP_TEXT = {
         "                [--country <XX|none>] [--bing-site <url|none>]",
         "                [--place-id <id|none>]",
         "                [--ahrefs-site-audit-project <id|none>]",
-        "                [--semrush-project <id|none>]",
         "",
         "clients list",
         "  Show every client. When Google is set up, newly discovered Search Console",
@@ -150,9 +151,6 @@ const GROUP_TEXT = {
         "                       The client's existing Ahrefs Site Audit project",
         "                       (app.ahrefs.com/site-audit/<id>), read by",
         "                       technical_issues; none matches by domain",
-        "  --semrush-project <id>",
-        "                       The client's existing Semrush project with Site",
-        "                       Audit; none matches by domain",
     ],
     competitors: (p) => [
         `Usage: ${p} competitors list <client>`,

package/dist/cli.js

@@ -399,8 +399,10 @@ const noneToEmpty = (v) => (v !== undefined && v.trim().toLowerCase() === "none"
 async function clientsSet(args, values) {
     if (args.length !== 1)
         return usageError("clients set takes exactly one client.");
-    if (Object.values({ a: values.area, n: values.niche, m: values.name, b: values["brand-terms"], p: values.provider, c: values.country, s: values["bing-site"], g: values["place-id"], ah: values["ahrefs-site-audit-project"], sp: values["semrush-project"] }).every((v) => v === undefined)) {
-        return usageError("clients set needs --area, --niche, --name, --brand-terms, --provider, --country, --bing-site, --place-id, --ahrefs-site-audit-project or --semrush-project.");
+    if (values["semrush-project"] !== undefined)
+        return usageError("--semrush-project is no longer supported: Semrush Site Audit needs Semrush's v3 API, which crawlmark no longer supports.");
+    if (Object.values({ a: values.area, n: values.niche, m: values.name, b: values["brand-terms"], p: values.provider, c: values.country, s: values["bing-site"], g: values["place-id"], ah: values["ahrefs-site-audit-project"] }).every((v) => v === undefined)) {
+        return usageError("clients set needs --area, --niche, --name, --brand-terms, --provider, --country, --bing-site, --place-id, or --ahrefs-site-audit-project.");
     }
     const provider = noneToEmpty(values.provider)?.trim().toLowerCase();
     if (provider !== undefined && provider !== "" && !PROVIDER_IDS.includes(provider)) {
@@ -411,7 +413,7 @@ async function clientsSet(args, values) {
         return usageError(`--country must be a two-letter ISO code such as US or GB, or none (got "${values.country}").`);
     }
     const brandTerms = values["brand-terms"] === undefined ? undefined : values["brand-terms"].split(",");
-    const r = await setClientProfile(args[0], { area: values.area, niche: values.niche, name: values.name, brandTerms, dataProvider: provider, country, bingSite: noneToEmpty(values["bing-site"]), gbpPlaceId: noneToEmpty(values["place-id"]), ahrefsSiteAuditProject: noneToEmpty(values["ahrefs-site-audit-project"]), semrushProject: noneToEmpty(values["semrush-project"]) }, { trustedPlaceId: true });
+    const r = await setClientProfile(args[0], { area: values.area, niche: values.niche, name: values.name, brandTerms, dataProvider: provider, country, bingSite: noneToEmpty(values["bing-site"]), gbpPlaceId: noneToEmpty(values["place-id"]), ahrefsSiteAuditProject: noneToEmpty(values["ahrefs-site-audit-project"]) }, { trustedPlaceId: true });
     for (const [k, v] of Object.entries(r.set))
         out(`Set ${k}: ${v}`);
     for (const k of r.cleared)
@@ -421,7 +423,6 @@ async function clientsSet(args, values) {
         (r.profile.bingSite !== undefined ? `Bing site ${r.profile.bingSite ?? "(matched by domain)"}. ` : "") +
         (r.profile.gbpPlaceId !== undefined ? `Google Business Profile place ID ${r.profile.gbpPlaceId ?? "(not set)"}. ` : "") +
         (r.profile.ahrefsSiteAuditProject !== undefined ? `Ahrefs Site Audit project ${r.profile.ahrefsSiteAuditProject ?? "(matched by domain)"}. ` : "") +
-        (r.profile.semrushProject !== undefined ? `Semrush project ${r.profile.semrushProject ?? "(matched by domain)"}. ` : "") +
         `File: ${r.clientsFile}`);
     return 0;
 }

package/dist/clients.js

@@ -479,13 +479,11 @@ opts = {}) {
             throw new Error(`"${stripControl(update.gbpPlaceId).slice(0, 100)}" is not a Google place ID (letters, digits, - and _, as local_profile returns it in matched.placeId).`);
         fields.gbpPlaceId = v;
     }
-    for (const k of ["ahrefsSiteAuditProject", "semrushProject"]) {
-        if (update[k] === undefined)
-            continue;
-        const v = update[k].trim();
+    if (update.ahrefsSiteAuditProject !== undefined) {
+        const v = update.ahrefsSiteAuditProject.trim();
         if (v !== "" && !PROJECT_ID_RE.test(v))
-            throw new Error(`"${stripControl(update[k]).slice(0, 60)}" isn't ${k === "semrushProject" ? "a Semrush project" : "an Ahrefs Site Audit project"} ID (digits, from the project's URL).`);
-        fields[k] = v;
+            throw new Error(`"${stripControl(update.ahrefsSiteAuditProject).slice(0, 60)}" isn't an Ahrefs Site Audit project ID (digits, from the project's URL).`);
+        fields.ahrefsSiteAuditProject = v;
     }
     let brandTerms;
     if (update.brandTerms !== undefined) {
@@ -495,7 +493,7 @@ opts = {}) {
             throw new Error(`Not a usable brand term: ${bad.map((t) => `"${t}"`).join(", ")}. Terms need a letter or digit and at most ${MAX_BRAND_TERM_LENGTH} characters.`);
     }
     if (Object.keys(fields).length === 0 && brandTerms === undefined)
-        throw new Error("Nothing to set: pass area, niche, name, brandTerms, dataProvider, country, bingSite, gbpPlaceId, ahrefsSiteAuditProject or semrushProject.");
+        throw new Error("Nothing to set: pass area, niche, name, brandTerms, dataProvider, country, bingSite, gbpPlaceId or ahrefsSiteAuditProject.");
     const client = await resolveClient(idOrDomain);
     if (fields.bingSite && !bingSiteOnDomain(fields.bingSite, client.domain)) {
         throw new Error(`bingSite ${fields.bingSite.slice(0, 200)} isn't on ${client.domain}: it would show another site's Bing data in this client's reports. Use a Bing site on ${client.domain} or a subdomain of it.`);
@@ -528,7 +526,6 @@ opts = {}) {
             ...(update.bingSite !== undefined ? { bingSite: updated.bingSite ?? null } : {}),
             ...(update.gbpPlaceId !== undefined ? { gbpPlaceId: updated.gbpPlaceId ?? null } : {}),
             ...(update.ahrefsSiteAuditProject !== undefined ? { ahrefsSiteAuditProject: updated.ahrefsSiteAuditProject ?? null } : {}),
-            ...(update.semrushProject !== undefined ? { semrushProject: updated.semrushProject ?? null } : {}),
         },
         clientsFile: env.CLIENTS_FILE,
     };

package/dist/config.js

@@ -71,6 +71,7 @@ const EnvSchema = z.object({
     // Phase 9: Ahrefs (API v3) and Semrush keys, and monthly unit caps. SEO_DATA_PROVIDER is read
     // leniently below (not here), so an unknown value never stops the server.
     AHREFS_API_KEY: z.string().optional(),
+    // Semrush's v4 API key (profile > API keys). SEMRUSH_V4_API_KEY is the name 0.1.0 used for the v4 key (there SEMRUSH_API_KEY was the old v3 key), so it wins when both are set.
     SEMRUSH_API_KEY: z.string().optional(),
     SEMRUSH_V4_API_KEY: z.string().optional(),
     AHREFS_MONTHLY_UNIT_CAP: z.coerce.number().int().positive().default(50_000),
@@ -95,7 +96,6 @@ const EnvSchema = z.object({
 const parsedEnv = EnvSchema.parse(stripEmptyValues(process.env));
 export const AHREFS_DEFAULT_BASE_URL = "https://api.ahrefs.com/v3";
 export const SEMRUSH_DEFAULT_BASE_URL = "https://api.semrush.com";
-export const SEMRUSH_DEFAULT_BALANCE_URL = "https://www.semrush.com/users/countapiunits.html";
 export const CRUX_DEFAULT_BASE_URL = "https://chromeuxreport.googleapis.com/v1";
 /** Bing Webmaster Tools JSON endpoint; methods are appended as /<Method>. */
 export const BING_DEFAULT_BASE_URL = "https://ssl.bing.com/webmaster/api.svc/json";
@@ -129,9 +129,9 @@ export const env = {
     GOOGLE_APPLICATION_CREDENTIALS: parsedEnv.GOOGLE_APPLICATION_CREDENTIALS
         ? resolveFromRoot(HOME.dir, parsedEnv.GOOGLE_APPLICATION_CREDENTIALS)
         : undefined,
+    SEMRUSH_API_KEY: parsedEnv.SEMRUSH_V4_API_KEY || parsedEnv.SEMRUSH_API_KEY || undefined,
     AHREFS_BASE_URL: urlOverride("AHREFS_BASE_URL", AHREFS_DEFAULT_BASE_URL),
     SEMRUSH_BASE_URL: urlOverride("SEMRUSH_BASE_URL", SEMRUSH_DEFAULT_BASE_URL),
-    SEMRUSH_BALANCE_URL: urlOverride("SEMRUSH_BALANCE_URL", SEMRUSH_DEFAULT_BALANCE_URL),
     CRUX_BASE_URL: urlOverride("CRUX_BASE_URL", CRUX_DEFAULT_BASE_URL),
     BING_BASE_URL: urlOverride("BING_BASE_URL", BING_DEFAULT_BASE_URL),
     CRAWLER_IPRANGES_BASE_URL: urlOverride("CRAWLER_IPRANGES_BASE_URL", CRAWLER_IPRANGES_DEFAULT_BASE_URL),
@@ -149,7 +149,6 @@ export const clientDefaults = {
 export const providerUrlNotice = urlOverrideNotice([
     { name: "AHREFS_BASE_URL", value: env.AHREFS_BASE_URL, fallback: AHREFS_DEFAULT_BASE_URL },
     { name: "SEMRUSH_BASE_URL", value: env.SEMRUSH_BASE_URL, fallback: SEMRUSH_DEFAULT_BASE_URL },
-    { name: "SEMRUSH_BALANCE_URL", value: env.SEMRUSH_BALANCE_URL, fallback: SEMRUSH_DEFAULT_BALANCE_URL },
     { name: "CRUX_BASE_URL", value: env.CRUX_BASE_URL, fallback: CRUX_DEFAULT_BASE_URL },
     { name: "BING_BASE_URL", value: env.BING_BASE_URL, fallback: BING_DEFAULT_BASE_URL },
     { name: "CRAWLER_IPRANGES_BASE_URL", value: env.CRAWLER_IPRANGES_BASE_URL, fallback: CRAWLER_IPRANGES_DEFAULT_BASE_URL },

package/dist/lib/clients.js

@@ -288,7 +288,7 @@ export function normalizeArea(input) {
 // ---------------------------------------------------------------- clients.yaml overrides
 /** A Google place ID (e.g. ChIJexamplePlaceId000001): letters, digits, - and _ only. Phase 12 C4. */
 export const PLACE_ID_RE = /^[A-Za-z0-9_-]{10,300}$/;
-/** Phase 13 B: Ahrefs and Semrush project IDs are digits. */
+/** Phase 13 B: Ahrefs project IDs are digits. */
 export const PROJECT_ID_RE = /^\d{1,20}$/;
 const OverrideSchema = z.object({
     domain: z.string().min(1),
@@ -334,8 +334,8 @@ const OverrideSchema = z.object({
     gbpPlaceId: z.string().regex(PLACE_ID_RE, { error: "must be a Google place ID (letters, digits, - and _), e.g. ChIJexamplePlaceId000001" }).optional(),
     /** Phase 13 B: the client's existing Ahrefs Site Audit project (its ID from app.ahrefs.com/site-audit/<id>); wins over matching by domain. */
     ahrefsSiteAuditProject: z.coerce.string().regex(PROJECT_ID_RE, { error: "must be an Ahrefs Site Audit project ID (digits)" }).optional(),
-    /** Phase 13 B: the client's existing Semrush project with Site Audit (its ID); wins over matching by domain. */
-    semrushProject: z.coerce.string().regex(PROJECT_ID_RE, { error: "must be a Semrush project ID (digits)" }).optional(),
+    /** Deprecated and ignored: Semrush Site Audit needed Semrush's v3 API, which crawlmark no longer supports. Accepted so an existing file still loads. */
+    semrushProject: z.coerce.string().optional(),
     /** Hide this domain: it isn't a client, and discovery never re-adds it. */
     ignore: z.boolean().optional(),
 });
@@ -346,6 +346,8 @@ const OverridesFileSchema = z
  * Validate parsed clients.yaml content. An empty file or empty `clients:` is
  * fine. Throws one readable error naming the bad entries.
  */
+let semrushProjectNoted = false;
+export const SEMRUSH_PROJECT_NOTE = "crawlmark: clients.yaml `semrushProject` is no longer used (Semrush Site Audit needs Semrush's v3 API, which crawlmark no longer supports); you can remove it.";
 export function parseOverrides(raw, source = "clients.yaml") {
     const parsed = OverridesFileSchema.safeParse(raw);
     if (!parsed.success) {
@@ -380,7 +382,13 @@ export function parseOverrides(raw, source = "clients.yaml") {
             if (!normalizeBrandTerm(brandTermText(t)))
                 throw new Error(`Invalid ${source}: clients.${slug}.brandTerms[${i}] must contain a letter or digit and be at most 100 characters.`);
         });
-        return { ...entry, domain: host, slug };
+        // The deprecated key is accepted and dropped; one stderr note per process says so.
+        const { semrushProject, ...rest } = entry;
+        if (semrushProject !== undefined && !semrushProjectNoted) {
+            semrushProjectNoted = true;
+            process.stderr.write(`${SEMRUSH_PROJECT_NOTE}\n`);
+        }
+        return { ...rest, domain: host, slug };
     });
 }
 /** DEFAULT_CLIENT_MONTHLY_AHREFS_UNITS / _SEMRUSH_UNITS defaults (D1). */
@@ -460,8 +468,6 @@ export function mergeClients(discovered, overrides, defaults, gtm = new Map()) {
             client.gbpPlaceId = y.gbpPlaceId;
         if (y?.ahrefsSiteAuditProject)
             client.ahrefsSiteAuditProject = y.ahrefsSiteAuditProject;
-        if (y?.semrushProject)
-            client.semrushProject = y.semrushProject;
         if (d?.ga4DisplayName && d.ga4DisplayName !== client.name)
             client.ga4Name = d.ga4DisplayName;
         return client;

package/dist/lib/country.js

@@ -49,17 +49,6 @@ export function resolveCountry(country, locationCode) {
 export function ahrefsCountry(cc) {
     return cc.trim().toLowerCase();
 }
-/**
- * The documented Semrush desktop databases, copied from docs/research/semrush-api.md
- * "Databases" (lowercase; the United Kingdom is `uk`). Mobile and `-ext` databases are never used.
- */
-export const SEMRUSH_DATABASES = new Set(("us uk ca ru de fr es it br au ar be ch dk fi hk ie il mx nl no pl se sg tr jp in hu af al dz ao am at az bh bd by bz bo ba bw bn bg cv kh cm cl co cr hr cy cz cd do ec eg sv ee et ge gh gr gt gy ht hn is id jm jo kz kw lv lb lt lu mg my mt mu md mn me ma mz na np nz ni ng om py pe ph pt ro sa sn rs sk si za kr lk th bs tt tn ua ae uy ve vn zm zw ly pa pk tw qa").split(" "));
-/** Semrush `database` for a country: lowercase, GB -> `uk`; null when it isn't a documented desktop database. */
-export function semrushDatabase(cc) {
-    const lower = cc.trim().toLowerCase();
-    const db = lower === "gb" ? "uk" : lower;
-    return SEMRUSH_DATABASES.has(db) ? db : null;
-}
 /** The message when a client has no country Ahrefs/Semrush can use (spec 9.1). */
 export function noCountryMessage(client) {
     return (`Set country: XX for ${client.domain} in clients.yaml (or clients set ${client.domain} --country XX); ` +

package/dist/lib/env-file.js

@@ -54,7 +54,7 @@ export function formatEnvValue(value) {
  * line it appears on (keeping `export` and spacing), so it never ends up with two
  * different values; a new key is appended. Returns the new text and what changed.
  */
-export function setEnvValues(text, updates) {
+export function setEnvValues(text, updates, remove = []) {
     for (const k of Object.keys(updates))
         if (!KEY_RE.test(k))
             throw new Error(`Invalid env key: ${k}`);
@@ -64,6 +64,15 @@ export function setEnvValues(text, updates) {
     const trailing = lines.length > 0 && lines[lines.length - 1] === "";
     if (trailing)
         lines.pop();
+    // Keys to delete (every line they appear on), e.g. a name an older version wrote.
+    const removed = new Set();
+    for (let i = lines.length - 1; i >= 0 && remove.length; i--) {
+        const m = LINE_RE.exec(lines[i]);
+        if (m && remove.includes(m[2])) {
+            removed.add(m[2]);
+            lines.splice(i, 1);
+        }
+    }
     const seen = new Set();
     const updated = new Set();
     for (let i = 0; i < lines.length; i++) {
@@ -87,7 +96,7 @@ export function setEnvValues(text, updates) {
         lines.push(`${key}=${formatEnvValue(value)}`);
         added.push(key);
     }
-    return { text: lines.length ? lines.join(eol) + eol : "", updated: [...updated], added };
+    return { text: lines.length ? lines.join(eol) + eol : "", updated: [...updated], added, removed: [...removed] };
 }
 export function readEnvFile(path) {
     return existsSync(path) ? parseEnvText(readFileSync(path, "utf8")) : {};
@@ -96,13 +105,13 @@ export function readEnvFile(path) {
  * Update keys in a `.env` file: atomic (temp file, then rename), mode 0600,
  * parent directory created with 0700, symlinks followed. Returns which keys were updated or added.
  */
-export function writeEnvFile(file, updates) {
+export function writeEnvFile(file, updates, remove = []) {
     // Follow a symlinked .env so the link survives the rename.
     const path = existsSync(file) ? realpathSync(file) : file;
     const before = existsSync(path) ? readFileSync(path, "utf8") : "";
-    const r = setEnvValues(before, updates);
+    const r = setEnvValues(before, updates, remove);
     mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
     writeFileAtomic(path, r.text, 0o600);
-    return { updated: r.updated, added: r.added };
+    return { updated: r.updated, added: r.added, removed: r.removed };
 }
 //# sourceMappingURL=env-file.js.map

package/dist/lib/issue-snapshots.js

@@ -1,7 +1,5 @@
 import { db } from "./store.js";
 export const ISSUE_SNAPSHOTS_KEPT = 24;
-/** Semrush's API terms: nothing from Semrush is kept longer than this (the Semrush Site Audit items in a snapshot). */
-export const SEMRUSH_ISSUE_RETENTION_DAYS = 31;
 const DAY_MS = 86_400_000;
 /**
  * Remove one source's items and source runs from stored issue snapshots (all of them, one client's, or only those
@@ -31,8 +29,6 @@ export function stripSourceFromIssueSnapshots(source, o = {}) {
     }
     return changed;
 }
-/** Semrush Site Audit items older than 31 days are removed (on every save and every read). */
-export const pruneSemrushIssueSnapshots = (now) => stripSourceFromIssueSnapshots("semrush_site_audit", { beforeMs: now - SEMRUSH_ISSUE_RETENTION_DAYS * DAY_MS });
 /** The newest snapshot of each calendar month is also kept for this many months. */
 export const MONTHLY_KEPT = 25;
 /** The first millisecond of the month `back` months before the month of `ms` (UTC). */
@@ -51,7 +47,6 @@ const fromRow = (r) => ({
 export function saveIssueSnapshot(s, now = Math.max(Date.now(), s.capturedAt)) {
     db.exec("BEGIN");
     try {
-        pruneSemrushIssueSnapshots(now);
         db.prepare("INSERT INTO issue_snapshots (id, client, captured_at, items, checked, settings) VALUES (?, ?, ?, ?, ?, ?)").run(s.id, s.client, s.capturedAt, JSON.stringify(s.items), JSON.stringify(s.checked), JSON.stringify(s.settings));
         // Kept: the newest ISSUE_SNAPSHOTS_KEPT, plus the newest of each calendar month (UTC) for MONTHLY_KEPT months, so a
         // report's base (the snapshot from the previous report) can't be pruned by frequent tool calls.
@@ -74,7 +69,6 @@ export function saveIssueSnapshot(s, now = Math.max(Date.now(), s.capturedAt)) {
 }
 /** The client's newest snapshot captured strictly before `beforeMs`, leaving out `excludeIds` (the current run's own). */
 export function latestIssueSnapshotBefore(client, beforeMs, excludeIds = [], now = Date.now()) {
-    pruneSemrushIssueSnapshots(now);
     const ex = excludeIds.length ? ` AND id NOT IN (${excludeIds.map(() => "?").join(", ")})` : "";
     const row = db.prepare(`SELECT * FROM issue_snapshots WHERE client = ? AND captured_at < ?${ex} ORDER BY captured_at DESC, id DESC LIMIT 1`).get(client, beforeMs, ...excludeIds);
     return row ? fromRow(row) : undefined;