n-seo 0.1.0

package/docs/INSTANCE.md

@@ -0,0 +1,365 @@
+# Running n-seo as an engine + instance
+
+n-seo can run two ways. In the simplest, you clone the repo, put your config
+in the checkout and run it there. In the second, the checkout is an **engine**
+you never edit, and everything that is yours — config, queue, content, data,
+logs — lives in a separate **instance** directory that the engine is pointed
+at. This document is about the second way, and about why you would want it.
+
+## Why split them
+
+- **Upgrades without merges.** Upstream changes touch engine files; your
+  files live somewhere else. Upgrading is `git pull` (or `npm update`) and a
+  test run. There is nothing to reconcile.
+- **Your work in your own repo.** The queue you curate, the drafts and
+  campaigns you write, the daily log that accumulates — those are yours, with
+  their own history, in a repo you control. Private if you want.
+- **Several instances, one engine.** A consultancy can run one engine and an
+  instance per client. A team can run a shared engine on one machine and
+  keep each portfolio separate.
+- **The upgrade path gets exercised.** Because upgrading is cheap, you do it
+  routinely, and the engine's `check` run is your regression test against
+  your own data.
+
+## Who owns what
+
+| Engine (the n-seo checkout) | Instance (your directory) |
+|---|---|
+| `src/`, `ingest/`, `ops/`, `probes/`, `public/`, `bin/`, `tests/` | `n-seo.config.json` (or whatever `$N_SEO_CONFIG` names) |
+| `n-seo.config.example.json` | `.env` |
+| `docs/PLAYBOOK.md`, `docs/OPERATING-RULES.md`, `docs/ARCHITECTURE.md` | `config/backlog.json`, `config/insights.json` |
+| | `content/drafts/*.md`, `content/campaigns/*.json` |
+| | `data/` (regenerable), `site/` (static export) |
+| | `docs/daily-log.md`, `docs/reports/` |
+| | the IndexNow key file |
+| | `.mcp.json`, `CLAUDE.md`, `.claude/` (your agent's rules) |
+
+The engine resolves the instance from, in order: `--instance <path>` on the
+CLI, then `$N_SEO_INSTANCE`, then the current directory. With none of those
+set and a config file next to the engine's `src/`, you are in in-place mode
+and the engine checkout *is* the instance. Nothing in the engine ever writes
+outside the instance directory.
+
+## Three ways to run
+
+### A. In place
+
+Clone, configure, run. One directory.
+
+```sh
+git clone https://github.com/en-dash-consulting/n-seo && cd n-seo
+npm install
+cp n-seo.config.example.json n-seo.config.json   # edit sites, auth
+python3 ops/doctor.py
+npm start
+```
+
+Upgrade:
+
+```sh
+git pull --ff-only && npm ci && npm run check
+```
+
+Merge conflicts are only possible in files you edited. After 0.1, upstream
+does not change `config/backlog.json`, `config/insights.json` or anything
+under `content/`, so a config-only setup pulls clean. If you edit engine
+files — a rule in `src/actions.ts`, a page in `src/views.tsx` — you are
+forking; add `upstream` as a remote and rebase or merge on your schedule:
+
+```sh
+git remote add upstream https://github.com/en-dash-consulting/n-seo
+git fetch upstream && git rebase upstream/main   # or merge
+```
+
+### B. Engine + instance directory
+
+Two directories. The engine is a plain checkout; the instance is scaffolded
+by the CLI and holds everything that is yours.
+
+```sh
+git clone https://github.com/en-dash-consulting/n-seo ~/tools/n-seo
+cd ~/tools/n-seo && npm install
+
+node ~/tools/n-seo/bin/n-seo.mjs init ~/sites/search-ops
+cd ~/sites/search-ops
+#   edit n-seo.config.json, .env
+n-seo doctor          # via the .mcp.json/env the scaffold wrote, or:
+node ~/tools/n-seo/bin/n-seo.mjs doctor --instance ~/sites/search-ops
+```
+
+`n-seo init` writes: `n-seo.config.json` (from the example), empty
+`config/backlog.json` and `config/insights.json`, `content/drafts/README.md`
+and `content/campaigns/README.md`, `.env` (from `.env.example`), a
+`.gitignore` that excludes `data/`, `site/` and `.env`, a `.mcp.json` that
+spawns the engine's MCP server with `N_SEO_INSTANCE` set, and a short README.
+
+To make `n-seo` a command, link the engine once:
+
+```sh
+cd ~/tools/n-seo && npm link      # puts `n-seo` on your PATH
+```
+
+Daily use, from inside the instance directory (or with `--instance`):
+
+```sh
+n-seo start            # dashboard on the config's port
+n-seo daily            # the morning run
+n-seo daily --only probe,gsc
+n-seo doctor
+n-seo export           # static export → <instance>/site/
+n-seo mcp              # stdio MCP server for this instance
+```
+
+Scheduling: the templates in `ops/templates/` run `ops/daily.py` and
+`src/server.tsx` from the engine checkout. For an instance, change the
+program to the CLI and name the instance, either as an argument:
+
+```xml
+<key>ProgramArguments</key>
+<array>
+  <string>/usr/bin/env</string>
+  <string>node</string>
+  <string>/Users/you/tools/n-seo/bin/n-seo.mjs</string>
+  <string>daily</string>
+  <string>--instance</string>
+  <string>/Users/you/sites/search-ops</string>
+</array>
+```
+
+or as an environment variable in the plist / unit / crontab:
+
+```xml
+<key>EnvironmentVariables</key>
+<dict>
+  <key>N_SEO_INSTANCE</key>
+  <string>/Users/you/sites/search-ops</string>
+</dict>
+```
+
+```cron
+0 7 * * * N_SEO_INSTANCE=/home/you/sites/search-ops /usr/bin/env node /home/you/tools/n-seo/bin/n-seo.mjs daily >> /home/you/sites/search-ops/data/daily-cron.log 2>&1
+```
+
+Log paths in the templates point at `data/` inside the checkout; in instance
+mode point them at `<instance>/data/` instead. Do the same for the dashboard
+service (`start` instead of `daily`, `KeepAlive` / `Restart=always` as in the
+templates).
+
+Upgrade:
+
+```sh
+n-seo upgrade
+```
+
+This runs `git pull --ff-only` in the engine, `npm ci` if the lockfile
+changed, then `npm run check` (typecheck plus both test suites). If the check
+fails it prints the rollback command — `git -C <engine> checkout <previous sha>`
+followed by `npm ci` — and exits non-zero. Nothing in the instance is touched
+by an upgrade.
+
+### C. npm dependency
+
+The instance is a small npm project that depends on the engine. Nothing to
+clone; upgrades are `npm update`.
+
+```sh
+mkdir ~/sites/search-ops && cd ~/sites/search-ops
+npm init -y
+npm install github:en-dash-consulting/n-seo     # or `n-seo@^0.1` once published
+npx n-seo init .
+```
+
+`package.json` scripts make the commands local:
+
+```json
+{
+  "scripts": {
+    "start": "n-seo start",
+    "daily": "n-seo daily",
+    "doctor": "n-seo doctor",
+    "export": "n-seo export",
+    "upgrade": "npm update n-seo && n-seo check"
+  },
+  "dependencies": { "n-seo": "github:en-dash-consulting/n-seo" }
+}
+```
+
+The engine lives in `node_modules/n-seo`; the instance is the current
+directory. `n-seo upgrade` recognizes an npm install and tells you to run
+`npm update n-seo` instead of pulling. Pin with a version or a commit
+(`github:en-dash-consulting/n-seo#<sha>`) when you need to.
+
+Python is still required on the machine — the engine's ingest scripts run
+with `python3` from the package directory.
+
+## The instance repo
+
+A typical instance, committed to its own git repo:
+
+```
+search-ops/
+  n-seo.config.json      sites, auth, modules, hooks         ← commit
+  .env                   Reddit creds, MCP token             ← never commit
+  config/backlog.json    your curated queue                  ← commit
+  config/insights.json   your narrative briefing             ← commit
+  content/drafts/        distribution drafts (markdown)      ← commit
+  content/campaigns/     outreach campaigns (json)           ← commit
+  docs/daily-log.md      appended by every run               ← commit
+  docs/reports/          analyze_gsc / analyze_ga4 output    ← commit
+  data/                  snapshots, regenerable              ← ignore
+  site/                  static export                       ← ignore (or commit if you deploy it from git)
+  indexnow.key           public by design                    ← commit
+  ops/                   your own scripts, run by hooks      ← commit
+  .mcp.json, CLAUDE.md, .claude/                             ← commit
+```
+
+`.gitignore` written by `n-seo init`:
+
+```
+data/
+site/
+.env
+__pycache__/
+node_modules/
+```
+
+If `modules.gitAutoCommit` is on, the daily run commits `docs/daily-log.md`,
+`docs/reports/` and `site/` (when tracked) in the **instance** repo and pushes
+if it has a remote. The engine checkout is never committed to by the run.
+
+## Hooks: your scripts in the daily run
+
+`hooks` in the config attach shell commands to the run. Each command runs
+with the instance directory as its working directory and these variables
+set: `N_SEO_ROOT` (the engine), `N_SEO_INSTANCE`, and for `afterStep`,
+`N_SEO_STEP`. Hook output goes to `data/daily-ops.log` like a step; a
+failing hook is recorded in `data/last-run.json` and never aborts the run.
+
+```json
+"hooks": {
+  "beforeRun": [],
+  "afterStep": {
+    "gsc": ["python3 ops/flag_new_queries.py"],
+    "daily-diff": ["git add docs/daily-log.md && git commit -qm \"daily log $(date +%F)\" || true", "git push -q origin main || true"]
+  },
+  "afterRun": [
+    "rsync -a --delete site/ deploy@mirror.example.com:/srv/search-ops/",
+    "CLOUDSDK_CONFIG=$HOME/.config/my-gcloud gcloud storage rsync site gs://my-search-ops-mirror --recursive --delete-unmatched-destination-objects"
+  ]
+}
+```
+
+Three patterns that cover most needs:
+
+1. **Publish a mirror after the run.** With `modules.staticExport` on, the
+   run leaves `site/` in the instance; an `afterRun` `rsync` or
+   `gcloud storage rsync` puts it behind whatever auth you already have.
+   Choose one of the two lines above, not both.
+2. **Push the instance repo as soon as the log is written.** The
+   `afterStep.daily-diff` pair above commits and pushes the day's entry
+   before the slower digest steps run. Leave `gitAutoCommit` off if you do
+   this, or you will get two commits a day.
+3. **Run your own analysis on fresh data.** Anything under the instance's
+   own `ops/` can read `data/` (the layout is in `docs/ARCHITECTURE.md`)
+   and write wherever you like. `afterStep.gsc` runs right after the Search
+   Console pull lands, before the audit and the scan.
+
+Step names, for `afterStep`: `probe`, `gsc`, `ga4`, `timeseries`,
+`metadata-audit`, `index-status`, `opportunity-scan`, `daily-diff`,
+`hn-digest`, `reddit-digest`, `static-export`. A hook on a step whose module
+is off never runs.
+
+## Extra Search Console properties
+
+A host can be verified in Search Console as both a domain property and a
+URL-prefix property, and the two report differently. The sites list takes one
+`gscProperty` per site; anything else you want pulled for history goes in
+`gscExtraProperties`:
+
+```json
+"gscExtraProperties": ["https://example.com/"]
+```
+
+They land in `data/gsc/<slug>/` with the same files as any property. A
+url-prefix property gets a `-urlprefix` suffix (`https://example.com/` →
+`data/gsc/example.com-urlprefix/`), so it never collides with the domain
+property's directory. Nothing in the UI reads extras — they are for your own
+scripts and for comparison.
+
+## Your agent, in the instance
+
+`n-seo init` writes a `.mcp.json` that spawns the engine's stdio MCP server
+with `N_SEO_INSTANCE` pointing at the instance, so opening the instance
+directory in Claude Code gives the agent the instance's queue and data:
+
+```json
+{
+  "mcpServers": {
+    "n-seo": {
+      "command": "node",
+      "args": ["/Users/you/tools/n-seo/bin/n-seo.mjs", "mcp"],
+      "env": { "N_SEO_INSTANCE": "/Users/you/sites/search-ops" }
+    }
+  }
+}
+```
+
+`CLAUDE.md` and `.claude/skills/` in the instance are yours: put the
+operating rules that are specific to your sites there (approval gates,
+freeze dates, who may post where). The engine's generic rules stay in
+`docs/OPERATING-RULES.md` and are also served as the MCP resource
+`seo://docs/operating-rules`. The `engine_info` tool reports the engine
+version, commit, mode and both paths, so an agent can tell which engine it
+is talking to.
+
+## Testing the upgrade path as routine
+
+The point of the split is that upgrading is boring. Make it a habit:
+
+- **Weekly, automatically.** A `beforeRun` hook on Mondays:
+  ```json
+  "hooks": { "beforeRun": ["[ \"$(date +%u)\" = 1 ] && n-seo upgrade || true"] }
+  ```
+  or a separate scheduled job (a second launchd plist or cron line) that
+  runs `n-seo upgrade` an hour before the daily run.
+- **`npm run check` is the gate.** `n-seo upgrade` refuses to leave the
+  engine on a commit whose typecheck or tests fail. Your `data/` is not
+  used by the tests (they run on a sandboxed copy with demo data), so a
+  failing check means the engine is broken, not your instance.
+- **Rollback is one command**, printed on failure:
+  ```sh
+  git -C ~/tools/n-seo checkout <previous-sha> && (cd ~/tools/n-seo && npm ci)
+  ```
+- **Pin when you need to.** `git -C ~/tools/n-seo checkout v0.1.0` (or a
+  sha) holds the engine; `n-seo upgrade` then fast-forwards only when you
+  move back to `main`. In npm mode, pin the version in `package.json`.
+- **Watch the Settings page.** It shows the engine version and commit the
+  dashboard is running, which is how you notice a scheduled upgrade
+  silently failed. `n-seo doctor` prints the same block.
+
+## FAQ
+
+**Can I change a rule (a threshold in the action engine, a probe check)?**
+Not from an instance — the rules are engine code. Either run in-place mode
+as a fork (mode A with an `upstream` remote) or send the change upstream as a
+pull request, which is the better outcome if the rule is generally right.
+Instance-level customization is what `config/backlog.json`, `watchPages`,
+`conversions`, the module fields and hooks are for.
+
+**Where do custom dashboard pages go?**
+Instance mode does not support them yet. Fork. If a page would be useful to
+others, propose it.
+
+**What happens when upstream changes a data file format?**
+The engine's readers, writers and `demo_data.py` move together in one
+commit, and `data/` is regenerable: the next daily run rewrites every file.
+If a format change ever needs a manual step, the changelog says so and
+`n-seo upgrade` prints it.
+
+**Can two instances share one `data/`?**
+No. `data/` belongs to an instance. Two instances with the same sites will
+pull the same data twice; that is fine within Google's quotas.
+
+**Does the engine ever write to the instance's `config/backlog.json`?**
+Only through the dashboard's accept / mark-watching / retire buttons, which
+you click. The daily run never writes the queue.

package/docs/MCP.md

@@ -0,0 +1,104 @@
+# MCP server — the same data, for agents
+
+`src/mcp.ts` exposes the control plane over the Model Context Protocol so an
+AI agent can read the queue and the metrics directly instead of scraping the
+dashboard. It is **read-only by design**: the queue is edited through the
+dashboard's accept/watch/retire flow, so nothing here writes anything. Every
+tool is annotated `readOnlyHint`.
+
+## Two transports
+
+### stdio (default — no secret)
+
+The client spawns the server as a child process and talks over stdin/stdout.
+There is no network listener and no token: the OS process boundary is the
+authentication, and the server can only read what you can.
+
+`.mcp.json` in the repo root registers it for **Claude Code**, so opening the
+repo in Claude Code is all the setup there is.
+
+```sh
+npm run mcp          # run it by hand (protocol on stdout, diagnostics on stderr)
+npm run mcp:smoke    # list tools and call a few against your data
+```
+
+**Claude Desktop** — add to its MCP settings, with `cwd` set to your checkout:
+
+```json
+{
+  "mcpServers": {
+    "n-seo": {
+      "command": "npx",
+      "args": ["tsx", "src/mcp-stdio.ts"],
+      "cwd": "/path/to/n-seo"
+    }
+  }
+}
+```
+
+### HTTP (for clients that cannot spawn a process)
+
+`POST /mcp` on the dashboard, streamable HTTP, stateless — a fresh server per
+request, so no session state crosses callers. It requires a bearer token,
+read from `SEO_MCP_TOKEN` (env or `.env`), else from
+`~/.config/n-seo/mcp-token` (chmod 600).
+
+```sh
+curl -X POST http://localhost:4600/mcp \
+  -H "Authorization: Bearer $(cat ~/.config/n-seo/mcp-token)" \
+  -H 'Content-Type: application/json' \
+  -H 'Accept: application/json, text/event-stream' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
+```
+
+**With no token set the endpoint returns 503 rather than serving.** It fails
+closed so that your search data is never readable by accident: the dashboard
+binds `127.0.0.1` by default, but `SEO_HOST=0.0.0.0` (for a tunnel or a
+container) exposes it to whatever can reach that interface. Rotate by writing a new
+value to the token file and restarting the dashboard.
+
+## Why local-only
+
+All the real data lives in `data/` on the machine that runs the daily job. An
+MCP server therefore only makes sense next to those files. To reach it from
+another device, tunnel to the dashboard port (Tailscale, `cloudflared`,
+an SSH tunnel) rather than exposing it — the bearer token is the only thing
+in front of it.
+
+## Tools
+
+| Tool | Returns |
+|---|---|
+| `list_actions` | The queue, ranked by impact per effort. Filter by `status` (active / watching / all), `host`, `tag`; `limit` |
+| `get_action` | One action in full (spec and success criteria), matched by title substring |
+| `list_sites` | Every configured site with 16-month GSC totals, 90-day session trend, probe status |
+| `site_report` | One site: totals, traffic mix (search / AI assistants / other), trend, landing pages, probe findings |
+| `top_queries` | GSC queries for a site, `window` 90d or 16mo |
+| `striking_distance` | Queries at position 5–15 with meaningful impressions (90d) |
+| `ctr_gaps` | Queries whose CTR trails what their position should earn (90d) |
+| `metadata_audit` | Per-page title/description findings from the daily crawl |
+| `trends_timeseries` | Daily clicks and sessions for a site over the last N days |
+| `ops_status` | Did the last run succeed, how stale is each dataset, current probe results — check before trusting numbers |
+| `daily_log` | The last N entries of `docs/daily-log.md` |
+| `opportunity_proposals` | Rising-query candidates, machine proposals awaiting accept, verdicts on watching items |
+| `conversions_status` | Whether conversion events are instrumented and their 28-day counts by source |
+| `campaigns` | Outreach campaigns from `content/campaigns/` — targets, plan, template ids |
+| `settings` | The effective config: sites, module switches, auth mode (never the key) |
+
+Resources: `seo://docs/playbook`, `seo://docs/daily-log`,
+`seo://docs/operating-rules`.
+
+## Example prompts
+
+- "What should I do first this week? Use `list_actions` and explain the top
+  three in terms of the evidence."
+- "Which pages on example.com have CTR gaps, and what would you retitle them
+  to? Respect the 28-day freeze on anything in the watching list."
+- "Is the data fresh? Check `ops_status` before answering anything else."
+- "Summarize the last five daily-log entries — what regressed, what moved."
+- "Read the opportunity proposals and tell me which ones are worth accepting
+  and why. Don't add anything to the queue."
+
+The rules in `CLAUDE.md` apply to an agent using these tools: it proposes,
+you accept; it never drafts community comments; it reads outputs rather than
+re-running pulls.

package/docs/OPERATING-RULES.md

@@ -0,0 +1,106 @@
+# Operating rules
+
+These are the rules the tooling encodes and the docs keep repeating. Each one
+exists because the opposite was tried and cost something. An agent working in
+this repo is bound by them too (see `CLAUDE.md`).
+
+## The queue is the single source of "what's next"
+
+`config/backlog.json` merged with the data-derived rules, shown on `/actions`
+and via the MCP `list_actions` tool. New findings become entries with
+evidence and a spec. Shipped work gets a `watching` note. Corrections from the
+owner are applied immediately.
+
+**Why:** the moment "what's next" lives in three places — a doc, a chat, a
+head — the data stops deciding and recency does.
+
+## Shipped work becomes *watching*; it is never deleted
+
+When an action ships, set `watching` to a dated note ("title trimmed 3/14,
+frozen until 4/11, watching CTR") and add the page to `shippedWatch` so the
+data-derived cards for that page show as watching too.
+
+**Why:** the point of the change was to move a number. If the item disappears
+you never learn whether it did, and the same page gets "fixed" again in six
+weeks by someone reading the same CTR gap.
+
+## Decisions ride the 90-day window
+
+Rules, audits and rankings read the `*_90d.json` files and the 90-day
+metadata audit. The 16-month pull is for totals and history only.
+
+**Why:** a page fixed last week must stop being accused within a season.
+Sixteen-month aggregates keep a dead query cluster looking alive and a fixed
+page looking broken.
+
+## Impact numbers order the queue. They are not forecasts
+
+Every action carries an impact estimate (clicks per month). It exists so
+that impact ÷ effort can sort the list. It is not a prediction, and it is
+never reported as an expected result. When observed data exists for a
+shipped item, recalibrate against it.
+
+**Why:** the estimates are rough by construction (industry CTR curves ×
+impressions). Treating them as promises leads to disappointment on the
+misses and over-investment on the flukes.
+
+## Title-change freeze: 28 days
+
+After rewriting a page's title or description, do not touch that page's
+metadata again for 28 days. Measure, then move.
+
+**Why:** search engines re-evaluate a page after a title change and need
+weeks of impression data to settle. Iterating inside that window destroys
+the measurement and, at scale, reads as manipulation.
+
+## Stagger metadata batches: ≈8 per week across all sites
+
+Even when the audit lists 25 pages with title problems, ship about eight a
+week.
+
+**Why:** a burst of simultaneous title changes across a site is a churn
+pattern that invites re-evaluation of the whole site rather than the pages
+you changed. Batching also keeps the 28-day freezes from all expiring on the
+same day, which is what makes weekly measurement readable.
+
+## Proposals never self-promote into the queue
+
+The opportunity scan writes `data/opportunity-proposals.json`. Its proposals
+show on `/actions` as *proposed* until you click accept (or add them to the
+backlog by hand). Its verdicts on watching items (succeeded / failed /
+keep-watching) are review triggers, not actions.
+
+**Why:** a machine adding items to its own to-do list has no brake. Keeping
+the accept step human keeps the queue something you believe in.
+
+## Participation is human
+
+The Hacker News and Reddit modules find threads where your first-hand
+experience applies and write a briefing: what the piece says, what the
+thread is debating, where you genuinely connect. They never draft the
+comment, and neither should any agent using this repo. Sustainable pace is
+two to four genuine comments a day.
+
+**Why:** generated participation is detectable, against the rules of the
+communities involved, and a ban destroys the account history that makes
+your own launches land later. The briefing gets you to the thread fast; the
+words are what make it worth being there.
+
+## Site changes ship as branches and PRs
+
+Changes to your sites go into an `seo-*` branch off freshly pulled main in
+the site's own repo, pushed, and reviewed as a PR. Never direct to main. If a
+checkout has uncommitted work, use a git worktree rather than touching it.
+
+**Why:** the change log is how movement gets attributed. A PR with the queue
+item's evidence in its description is the record that lets the next daily
+run say "this moved because of that."
+
+## Read the outputs; don't re-derive them
+
+The daily run pulls, probes, audits and logs. To answer "how are we doing,"
+read `docs/daily-log.md`, `data/last-run.json` and the dashboard — do not
+re-run the pipeline, re-crawl the sites, or recompute the trends.
+
+**Why:** each re-run spends API quota and time to reproduce a file that
+already exists, and it invites two versions of the truth.