n-seo 0.1.0

package/docs/SETUP-GOOGLE.md

@@ -0,0 +1,215 @@
+# Connecting Search Console and GA4
+
+The pipeline reads two Google APIs: Search Console (queries, pages, index
+coverage) and Google Analytics 4 (sessions, sources, landing pages). Both are
+free within generous quotas. The recommended way to authorize is a **service
+account with a JSON key** — a machine identity you add as a read-only user in
+each console. No gcloud install, no browser login, no token that expires when
+your session does.
+
+## 1. Pick a Google Cloud project
+
+Any project works; a dedicated one keeps it tidy. In the
+[Cloud Console](https://console.cloud.google.com/), create a project (or open
+an existing one). Billing is not required for these APIs.
+
+## 2. Enable the APIs
+
+APIs & Services → Library. Enable all three:
+
+- **Google Search Console API**
+- **Google Analytics Data API**
+- **Google Analytics Admin API** (used only to list properties; harmless to
+  skip if you already know your property id, but the doctor check uses it)
+
+## 3. Create a service account and key
+
+IAM & Admin → Service Accounts → Create service account. Name it something like
+`n-seo-reader`. **Grant it no project roles** — it needs none. Access to
+your Search Console and GA4 data comes from the consoles themselves in step 4.
+
+Open the account → Keys → Add key → Create new key → JSON. Store the download
+outside the repo, readable only by you:
+
+```sh
+mkdir -p ~/.config/n-seo
+mv ~/Downloads/<project>-<hash>.json ~/.config/n-seo/service-account.json
+chmod 600 ~/.config/n-seo/service-account.json
+```
+
+Note the account's email (`n-seo-reader@<project>.iam.gserviceaccount.com`).
+You will paste it into two places next.
+
+## 4. Grant access in the consoles
+
+### Search Console
+
+Open the property → Settings → Users and permissions → Add user. Paste the
+service account email, permission **Full**. (Restricted is enough for reading
+performance data; Full also allows the sitemaps endpoint, which the indexing
+sweep uses to report sitemap state.)
+
+If the "Add user" dialog rejects the service-account address — this happens
+on some domain properties — make the service account a verified owner instead:
+enable the **Site Verification API** in the same Cloud project, call it with
+the service account to request a DNS TXT token for the domain, add that TXT
+record at your DNS provider, then call the API's verify method. The service
+account then appears in Search Console as an owner and no UI step is needed.
+Google documents the flow under "Site Verification API — Getting started."
+
+### GA4
+
+Admin → Property → Property access management → Add users. Paste the same
+email, role **Viewer**. Adding it at the account level instead gives it every
+property under the account, which is handy if you run several sites.
+
+While you are in Admin, open **Property details** and note the numeric
+**Property ID** (nine or ten digits). That goes in the config.
+
+## 5. Fill in the config
+
+```sh
+cp n-seo.config.example.json n-seo.config.json
+```
+
+```json
+{
+  "google": {
+    "auth": "service-account-key",
+    "serviceAccountKey": "~/.config/n-seo/service-account.json"
+  },
+  "sites": [
+    {
+      "host": "example.com",
+      "label": "example",
+      "gscProperty": "sc-domain:example.com",
+      "gscHost": "example.com",
+      "ga4Property": "123456789",
+      "brand": "example"
+    }
+  ]
+}
+```
+
+`gscProperty` must match the property exactly as Search Console has it:
+`sc-domain:example.com` for a domain property, or the full URL prefix
+(`https://www.example.com/`, trailing slash included) for a URL-prefix
+property. `$GOOGLE_APPLICATION_CREDENTIALS` is honored if you prefer that to
+`serviceAccountKey`. See [ADDING-A-SITE.md](ADDING-A-SITE.md) for every field.
+
+## 6. Verify
+
+```sh
+python3 ingest/google_auth.py     # mints a token, lists the properties the account can see
+python3 ops/doctor.py             # full setup check: config, key, APIs, each site's access
+```
+
+The first command should print `token OK` followed by your properties and the
+permission level on each. If a site you configured is missing from that list,
+step 4 has not taken effect yet (it can take a few minutes).
+
+Then run the pipeline once:
+
+```sh
+python3 ops/daily.py
+```
+
+## Alternative auth modes
+
+Set `google.auth` to one of:
+
+| Mode | How it gets a token | When to use |
+|---|---|---|
+| `service-account-key` | Signs an OAuth JWT with the key file using `openssl` | Default. No extra tools |
+| `gcloud-impersonate` | `gcloud auth print-access-token --impersonate-service-account=<google.impersonate>` | You already use gcloud and would rather grant your user *Service Account Token Creator* on the SA than keep a key file. Still add the SA to the consoles as above |
+| `gcloud-user` | `gcloud auth print-access-token` for your own login | Rarely works: gcloud's default client does not carry the Search Console or Analytics scopes for user credentials, and Google blocks `application-default login` with those scopes. Kept for completeness |
+| `metadata` | The runtime service account from the GCE / Cloud Run / GKE metadata server, exchanged for a scoped token | Running on Google Cloud. No key file exists, so none can leak |
+
+## Running it somewhere other than your laptop
+
+A key file is a secret you have to mount, rotate and keep out of the image.
+On Google Cloud you can skip it: give the workload a service account and set
+
+```json
+"google": { "auth": "metadata" }
+```
+
+There is one wrinkle worth knowing, because the failure is otherwise
+baffling. The metadata server hands out a token scoped to `cloud-platform`,
+and the Search Console API checks for its own scope, so it rejects that
+token. n-seo therefore does what the `gcloud-impersonate` mode does, without
+gcloud: it takes the metadata token and asks IAM Credentials for a properly
+scoped one **for the same account**. That self-impersonation needs the
+account to hold Token Creator *on itself*:
+
+```sh
+SA=n-seo-runtime@PROJECT.iam.gserviceaccount.com
+gcloud iam service-accounts add-iam-policy-binding "$SA" \
+  --member="serviceAccount:$SA" \
+  --role=roles/iam.serviceAccountTokenCreator
+```
+
+Without it the first pull fails with a 403, and the error prints that exact
+command. Everything else is unchanged: the same service account still has to
+be added as a **Full** user in Search Console and a **Viewer** in GA4.
+
+Set `google.impersonate` as well if the workload should borrow a *different*
+account than the one it runs as; then that account needs Token Creator for
+the runtime account.
+
+`python3 ops/doctor.py` reports the detected account and whether the
+exchange works, so run it once on the box before trusting a schedule.
+
+### The LLM module on a server
+
+`modules.llm.command` shells out to a CLI, and a container has none signed
+in — so the opportunity scan produces no proposals and the digests no
+briefings. Point it at an HTTP endpoint instead:
+
+```json
+"llm": {
+  "enabled": true,
+  "http": {
+    "provider": "anthropic",
+    "model": "claude-sonnet-5",
+    "fastModel": "claude-haiku-4-5-20251001",
+    "apiKeyEnv": "ANTHROPIC_API_KEY"
+  }
+}
+```
+
+The key is read from the environment or the instance's `.env` under the name
+you give in `apiKeyEnv`; it never goes in the config file. `provider` may
+also be `openai`, which speaks the chat-completions shape and therefore also
+covers gateways and local servers that emulate it — add `baseUrl` to point
+somewhere other than the vendor. When `http` is set and its key resolves it
+wins; otherwise the `command` path still runs, so a laptop and a server can
+share one config file.
+
+## Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `403 … insufficient permission` / `User does not have sufficient permissions for this site` | The service account is not (yet) a user on that property. Re-check step 4; wait a few minutes |
+| `404` on a Search Console property | `gscProperty` does not match the console: `sc-domain:` vs URL prefix, missing trailing slash, `www` vs apex |
+| `invalid_grant` / `Invalid JWT` when minting a token | System clock off by more than a few minutes, or a corrupted key file. Check `date`; re-download the key |
+| `no key file was found` | The path in `serviceAccountKey` does not exist. `~` is expanded; relative paths resolve from the repo root |
+| GA4 returns `PERMISSION_DENIED` | Viewer not granted on that property, or the property id is the *measurement id* (`G-…`) instead of the numeric property id |
+| Every step fails at once | Offline. The daily run waits up to five minutes for the network before starting |
+
+Why not `gcloud auth application-default login --scopes=…`? Google rejects
+those scopes for gcloud's built-in OAuth client, so it fails regardless of what
+you do. The service-account key avoids the problem entirely.
+
+## Data windows and quotas
+
+- **Search Console** keeps 16 months of performance data and finalizes each
+  day about three days late. The pipeline pulls the full 16 months (for totals
+  and history) and the trailing 90 days (the decision window) and uses
+  `dataState: final`, so the most recent three days are always missing by
+  design.
+- **URL Inspection** allows 2,000 inspections per day and 600 per minute per
+  property. The indexing sweep caps itself at 400 URLs per host per run.
+- **GA4 Data API** has a daily token budget per property that ordinary use
+  never approaches; the four reports the pipeline runs are small.
+- Nothing here costs money.

package/docs/examples/campaign.json

@@ -0,0 +1,59 @@
+{
+  "slug": "docs-listing-outreach",
+  "name": "Docs listing and backlink outreach",
+  "site": "docs.example.com",
+  "summary": "Get docs.example.com listed on the curated resource pages and awesome-lists our audience already reads. Ten ranked targets, three templates, one send plan.",
+  "voice": "Short, specific, no marketing adjectives. Lead with what the reader's audience gets. One ask per message. Disclose that it is our project.",
+  "targets": [
+    {
+      "rank": 1,
+      "name": "awesome-example-tools",
+      "category": "awesome-list",
+      "url": "github.com/someone/awesome-example-tools",
+      "contact": "Pull request following CONTRIBUTING.md — one line, alphabetical, matching the existing entry format",
+      "angle": "The list has no interactive tutorial entry; ours is free and needs no account",
+      "value": "durable backlink from a 4k-star repo; steady referral traffic",
+      "likelihood": "high — the list accepts PRs weekly and has no competing entry",
+      "evidence": "17 open PRs merged in the last 60 days; two comparable tools listed",
+      "status": ""
+    },
+    {
+      "rank": 2,
+      "name": "Example University — course resource page",
+      "category": "education",
+      "url": "cs.example.edu/courses/101/resources",
+      "contact": "Instructor email on the syllabus page",
+      "angle": "The course's week 4 assignment covers exactly what our getting-started guide teaches",
+      "value": ".edu link; seasonal traffic at semester start",
+      "likelihood": "medium — instructors reply in batches at term boundaries",
+      "evidence": "Resource page updated each August; links to two similar tools",
+      "status": "sent 2026-08-14"
+    }
+  ],
+  "templates": [
+    {
+      "id": "A",
+      "audience": "awesome-list maintainers",
+      "subject": "Add docs.example.com (interactive getting-started, no account needed)",
+      "body": "Hi —\n\nPR attached adding docs.example.com under Tutorials. It's our project (disclosure), free, runs in the browser without an account, and covers the same ground as the two guides already listed but interactively.\n\nHappy to trim to one line or move sections if you prefer.\n\nThanks for maintaining the list."
+    },
+    {
+      "id": "B",
+      "audience": "instructors",
+      "subject": "A free interactive companion to your week-4 material",
+      "body": "Hi Professor —\n\nI build docs.example.com, a free interactive guide to <topic>. Your 101 course's week-4 assignment covers the same concepts, and a few students have told us they used the guide alongside it.\n\nIf it would be useful to list on the course resources page, here is the link: https://docs.example.com/getting-started. No account, no tracking beyond standard analytics.\n\nEither way, thank you for teaching this.\n\n— <your name>"
+    }
+  ],
+  "plan": [
+    { "day": "Mon", "action": "Open the awesome-list PR", "template": "A", "notes": "Read CONTRIBUTING.md first; match the entry format exactly" },
+    { "day": "Wed", "action": "Email the two instructors whose fall syllabi are already posted", "template": "B", "notes": "Personalize the week reference for each" },
+    { "day": "Fri", "action": "Check PR feedback; edit the existing PR rather than opening a new one", "template": "", "notes": "" }
+  ],
+  "week2": "Follow up on unanswered instructor emails once, then stop. Move to the next three targets.",
+  "later": "Once the awesome-list entry is merged, cite it in the next two pitches as social proof.",
+  "cautions": [
+    "One follow-up maximum per target.",
+    "Never send the same template to two maintainers of the same list.",
+    "Disclose that it is your project every time."
+  ]
+}

package/docs/examples/draft.md

@@ -0,0 +1,43 @@
+---
+title: "Dev.to: how we cut our docs site's JS shell to zero and doubled AI referrals"
+order: 1
+action: "Voice pass, then post on dev.to with the canonical URL set to the original on docs.example.com"
+channel: dev.to
+status: ready to post
+tags: distribution, geo
+notes: >
+  Cross-post, not a new article: set the canonical to the original so the
+  ranking stays with your domain. Post mid-week morning in your audience's
+  timezone. Reply to the first few comments the same day.
+---
+
+# How we cut our docs site's JS shell to zero and doubled AI referrals
+
+Six months ago every page on docs.example.com served about 400 bytes of
+visible text and a 1.2 MB bundle. Google coped. Nothing else did: Bing indexed
+eleven pages, and referrals from ChatGPT and Perplexity were a rounding error.
+
+Here is what we changed, in the order it mattered, with the numbers.
+
+## 1. Server-render the content, keep the app
+
+…
+
+## 2. Let the crawlers in
+
+…
+
+## 3. Add llms.txt and llms-full.txt
+
+…
+
+## What moved
+
+| | Before | After 90 days |
+|---|---|---|
+| Pages indexed by Bing | 11 | 212 |
+| AI-assistant referral sessions / month | 9 | 21 |
+| Google clicks / day | 140 | 165 |
+
+The whole change was a week of work. The measuring took longer than the
+doing, and that was the point.

package/ingest/analyze_ga4.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""GA4 analysis: 90-day traffic mix and the AI-referral (GEO) scoreboard.
+
+Reads data/ga4/<host>/*.json, writes docs/reports/ga4-findings-<YYYY-MM>.md
+and prints it.
+"""
+
+import json
+import re
+import sys
+from datetime import date
+
+import seo_config
+
+GA_DIR = seo_config.DATA / "ga4"
+
+AI_SOURCES = re.compile(
+    r"chatgpt|chat\.openai|openai\.com|perplexity|claude\.ai|copilot|gemini\.google"
+    r"|edgeservices|you\.com|poe\.com|phind|kagi|mistral|deepseek", re.I)
+SEARCH_SOURCES = re.compile(r"google|bing|duckduckgo|yahoo|ecosia|brave|yandex|baidu", re.I)
+
+
+def rows(site, name):
+    p = GA_DIR / site / f"{name}.json"
+    if not p.exists():
+        return []
+    d = json.loads(p.read_text())
+    out = []
+    for r in d.get("rows", []):
+        dims = [v["value"] for v in r.get("dimensionValues", [])]
+        mets = [float(v["value"]) for v in r.get("metricValues", [])]
+        out.append((dims, mets))
+    return out
+
+
+def main():
+    if not GA_DIR.exists():
+        print("no GA4 data yet — run ingest/pull_ga4.py")
+        return 0
+    out = [f"# GA4 Findings — {date.today().isoformat()}",
+           "\nWindow: last 90 days.\n",
+           "| Site | Sessions | AI-referral | Search | Direct/other | Top AI sources |",
+           "|---|---|---|---|---|---|"]
+    detail = []
+    for site_dir in sorted(p for p in GA_DIR.iterdir() if p.is_dir()):
+        site = site_dir.name
+        src = rows(site, "sources")
+        total = sum(m[0] for _, m in src)
+        ai = [("/".join(d), m[0]) for d, m in src if AI_SOURCES.search(d[0])]
+        search = sum(m[0] for d, m in src if SEARCH_SOURCES.search(d[0]) and not AI_SOURCES.search(d[0]))
+        ai_total = sum(v for _, v in ai)
+        ai.sort(key=lambda t: -t[1])
+        top_ai = ", ".join(f"{s} ({v:.0f})" for s, v in ai[:3]) or "—"
+        out.append(f"| {site} | {total:,.0f} | {ai_total:,.0f}"
+                   f" ({100*ai_total/total:.1f}%) | {search:,.0f} ({100*search/total:.1f}%)"
+                   f" | {total-ai_total-search:,.0f} | {top_ai} |" if total else
+                   f"| {site} | 0 | — | — | — | — |")
+
+        landing = rows(site, "landing")[:8]
+        detail.append(f"\n## {site}\n\n**Top landing pages (sessions / engagement):**\n")
+        for d, m in landing:
+            detail.append(f"- {d[0]} — {m[0]:.0f} sessions, {100*m[1]:.0f}% engaged")
+        if ai:
+            detail.append("\n**All AI sources:**\n")
+            for s, v in ai:
+                detail.append(f"- {s}: {v:.0f} sessions")
+
+    text = "\n".join(out) + "\n" + "\n".join(detail) + "\n"
+    dest_dir = seo_config.INSTANCE / "docs" / "reports"
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    dest = dest_dir / f"ga4-findings-{date.today():%Y-%m}.md"
+    dest.write_text(text)
+    print(text)
+    print(f"[saved to {dest}]")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/ingest/analyze_gsc.py

@@ -0,0 +1,136 @@
+#!/usr/bin/env python3
+"""Analyze pulled Search Console data into a markdown report: totals, 28-day
+trend, striking-distance queries, CTR gaps, top pages, dead pages.
+
+Reads data/gsc/<slug>/*.json (from pull_gsc.py), writes
+docs/reports/gsc-findings-<YYYY-MM>.md and prints it. Run it whenever you
+want a shareable snapshot; the dashboard shows the same data live.
+"""
+
+import json
+import sys
+from datetime import date, timedelta
+
+import seo_config
+
+# Rough expected CTR by average position (industry midpoints).
+EXPECTED_CTR = {1: 0.28, 2: 0.15, 3: 0.10, 4: 0.07, 5: 0.05, 6: 0.04}
+
+
+def load(slug, dataset):
+    p = seo_config.DATA / "gsc" / slug / f"{dataset}.json"
+    return json.loads(p.read_text())["rows"] if p.exists() else []
+
+
+def fmt_pct(x):
+    return f"{100 * x:.1f}%"
+
+
+def section(prop, slug, out):
+    queries = load(slug, "queries")
+    pages = load(slug, "pages")
+    dates = load(slug, "dates")
+
+    clicks = sum(r["clicks"] for r in dates)
+    imps = sum(r["impressions"] for r in dates)
+    out.append(f"\n## {prop}\n")
+    out.append(f"**16-month totals:** {clicks:,.0f} clicks · {imps:,.0f} impressions"
+               f" · overall CTR {fmt_pct(clicks / imps) if imps else 'n/a'}\n")
+
+    if dates:
+        by_day = {r["keys"][0]: r for r in dates}
+        today = date.today()
+
+        def window(offset_start, offset_end):
+            c = i = 0
+            for d in range(offset_start, offset_end):
+                r = by_day.get((today - timedelta(days=d)).isoformat())
+                if r:
+                    c += r["clicks"]
+                    i += r["impressions"]
+            return c, i
+        c1, i1 = window(3, 31)
+        c2, i2 = window(31, 59)
+        out.append(f"**Last 28d vs prior 28d:** {c1:,.0f} vs {c2:,.0f} clicks · "
+                   f"{i1:,.0f} vs {i2:,.0f} impressions\n")
+
+    hosts = {}
+    for r in pages:
+        host = r["keys"][0].split("/")[2]
+        h = hosts.setdefault(host, {"clicks": 0, "impressions": 0})
+        h["clicks"] += r["clicks"]
+        h["impressions"] += r["impressions"]
+    if len(hosts) > 1:
+        out.append("**By host:**\n")
+        for host, v in sorted(hosts.items(), key=lambda kv: -kv[1]["clicks"]):
+            out.append(f"- {host}: {v['clicks']:,.0f} clicks / {v['impressions']:,.0f} impressions")
+        out.append("")
+
+    top = sorted(queries, key=lambda r: -r["clicks"])[:15]
+    out.append("**Top queries by clicks:**\n")
+    out.append("| Query | Clicks | Impressions | CTR | Pos |")
+    out.append("|---|---|---|---|---|")
+    for r in top:
+        out.append(f"| {r['keys'][0]} | {r['clicks']:.0f} | {r['impressions']:.0f}"
+                   f" | {fmt_pct(r['ctr'])} | {r['position']:.1f} |")
+
+    striking = [r for r in queries if 5 <= r["position"] <= 15 and r["impressions"] >= 50]
+    striking.sort(key=lambda r: -r["impressions"])
+    out.append("\n**Striking distance (pos 5-15, >=50 impressions) — biggest prizes:**\n")
+    if striking:
+        out.append("| Query | Impressions | Clicks | Pos |")
+        out.append("|---|---|---|---|")
+        for r in striking[:15]:
+            out.append(f"| {r['keys'][0]} | {r['impressions']:.0f} | {r['clicks']:.0f} | {r['position']:.1f} |")
+    else:
+        out.append("_none at threshold_")
+
+    ctr_gap = []
+    for r in queries:
+        pos = round(r["position"])
+        if pos in EXPECTED_CTR and r["impressions"] >= 100:
+            expected = EXPECTED_CTR[pos]
+            if r["ctr"] < expected * 0.5:
+                ctr_gap.append((r, expected))
+    ctr_gap.sort(key=lambda t: -(t[0]["impressions"] * (t[1] - t[0]["ctr"])))
+    out.append("\n**CTR gaps (ranking well, clicked rarely — title/snippet problems):**\n")
+    if ctr_gap:
+        out.append("| Query | Impressions | CTR | Expected | Pos |")
+        out.append("|---|---|---|---|---|")
+        for r, exp in ctr_gap[:12]:
+            out.append(f"| {r['keys'][0]} | {r['impressions']:.0f} | {fmt_pct(r['ctr'])}"
+                       f" | ~{fmt_pct(exp)} | {r['position']:.1f} |")
+    else:
+        out.append("_none at threshold_")
+
+    top_pages = sorted(pages, key=lambda r: -r["clicks"])[:10]
+    out.append("\n**Top pages by clicks:**\n")
+    for r in top_pages:
+        out.append(f"- {r['keys'][0]} — {r['clicks']:.0f} clicks, {r['impressions']:.0f} imps, pos {r['position']:.1f}")
+
+    dead = [r for r in pages if r["impressions"] >= 200 and r["clicks"] <= 2]
+    dead.sort(key=lambda r: -r["impressions"])
+    if dead:
+        out.append("\n**High-impression, near-zero-click pages:**\n")
+        for r in dead[:10]:
+            out.append(f"- {r['keys'][0]} — {r['impressions']:.0f} imps, {r['clicks']:.0f} clicks, pos {r['position']:.1f}")
+    return out
+
+
+def main():
+    out = [f"# Search Console Findings — {date.today().isoformat()}",
+           "\nWindow: trailing 16 months (final data through ~3 days ago)."]
+    for prop, slug in seo_config.gsc_properties(include_extra=False).items():
+        section(prop, slug, out)
+    text = "\n".join(out) + "\n"
+    dest_dir = seo_config.INSTANCE / "docs" / "reports"
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    dest = dest_dir / f"gsc-findings-{date.today():%Y-%m}.md"
+    dest.write_text(text)
+    print(text)
+    print(f"\n[saved to {dest}]")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())