@ainyc/canonry 4.18.1 → 4.19.0

package/assets/agent-workspace/skills/canonry-setup/SKILL.md

@@ -88,6 +88,7 @@ GA4 is a first-class signal alongside citation tracking. Connect once with `cano
 | `references/aeo-analysis.md` | Interpreting sweep output, diagnosing regressions, planning content fixes |
 | `references/indexing.md` | Submitting URLs, checking GSC/Bing coverage, fixing indexing gaps |
 | `references/wordpress-integration.md` | Connecting to WordPress, editing pages, pushing staging → live |
+| `references/server-side-traffic.md` | Wiring server-log evidence (Cloud Run today; WordPress / others later) for AI Visibility — Server-Side. Connect, sync, manage sources, troubleshoot. |
 
 ---
 

package/assets/agent-workspace/skills/canonry-setup/references/server-side-traffic.md

@@ -0,0 +1,167 @@
+# Server-side traffic (AI Visibility — Server-Side)
+
+Server-side traffic ingestion captures **what AI engines actually do in
+your server logs** — bots crawling pages, AI products sending
+click-through arrivals — in addition to the citation data that measures
+**what models say** about you. The two surfaces are independent.
+
+## When to use it
+
+Reach for server-side traffic when an analyst or operator asks:
+
+- *"Is GPTBot / ClaudeBot / PerplexityBot actually fetching my pages?"*
+- *"Which paths are AI engines paying attention to?"*
+- *"Are users clicking through from chatgpt.com / claude.ai / etc.?"*
+- *"My citation rate is fine but there's no traffic — why?"*
+
+GA4 referrals (chatgpt.com → your site) catch click-throughs after they
+land. Server logs catch the upstream bot activity AND referrals at the
+edge — including arrivals GA4 missed because of cookie consent, ad
+blockers, or analytics gaps.
+
+## Architecture
+
+Two tables, populated from server-log adapters:
+
+| Table | What's in it |
+|---|---|
+| `crawler_events_hourly` | One row per `(project, source, hour, bot, verification, path, status)` — bot crawls rolled up by hour |
+| `ai_referral_events_hourly` | One row per `(project, source, hour, product, source_domain, evidence_type, landing_path, status)` — click-through arrivals rolled up by hour |
+| `raw_event_samples` | Bounded forensic samples (≤100 per sync) for spot-checking |
+
+Each `traffic_sources` row is one server-log integration for a project.
+Today's only adapter is `cloud-run`; future adapters slot in by
+implementing the same contract.
+
+## Connecting a Cloud Run source
+
+```bash
+# 1. Create a service account in the Cloud project that hosts the Cloud Run
+#    service. Grant it `roles/logging.viewer`. Download the JSON key.
+
+# 2. Connect from canonry CLI:
+canonry traffic connect cloud-run <project> \
+  --gcp-project <gcp-project-id> \
+  --service-account-key <path/to/key.json>
+
+# 3. (Optional) narrow to a specific service or location:
+canonry traffic connect cloud-run <project> \
+  --gcp-project <id> \
+  --service-account-key <path> \
+  --service my-service-name \
+  --location us-east1
+```
+
+Credentials are stored in `~/.canonry/config.yaml` (not the DB). The
+canonical key lives only on the host that runs `canonry serve`. The
+sync flow does NOT echo the private key back in any response.
+
+## Syncing data
+
+```bash
+# Manual sync — defaults to a 30-day lookback on the first run; subsequent
+# runs are clamped forward to lastSyncedAt to avoid re-pulling.
+canonry traffic sync <project> --source <id>
+
+# Override the lookback window (minutes):
+canonry traffic sync <project> --source <id> --since-minutes 4320  # 3 days
+```
+
+Cross-sync dedupe via the `last_event_ids` ring buffer means re-running a
+sync over an overlapping window cannot double-count rolled-up hourly
+hits. Safe to schedule (see "Scheduling" below) or trigger from CI.
+
+## Inspecting source state
+
+```bash
+# All sources with last-24h totals + latest sync run (single-call):
+canonry traffic status <project> --format json
+
+# Just the source list:
+canonry traffic sources <project> --format json
+
+# Windowed events (defaults to last 24h):
+canonry traffic events <project> --kind crawler --limit 200 --format json
+canonry traffic events <project> --kind ai-referral --since 2026-04-01 --until 2026-04-30
+```
+
+The `traffic status` composite returns the same per-source detail
+(24h crawler hits, AI-referral arrivals, raw-event-sample count, latest
+sync-run summary) whether you reach it via the CLI, the API, or the
+MCP `canonry_traffic_status` tool.
+
+## Where the data shows up
+
+| Surface | What's rendered |
+|---|---|
+| Project dashboard `/projects/:name/activity` | Live source table + 24h totals + GA4 referrals (combined view) |
+| Top-level `/traffic` route | Cross-project source admin (connect, sync, archive) |
+| `canonry report <project>` (HTML + SPA) | "AI Visibility — Server-Side" section, ranked above Indexing Health |
+| `canonry doctor --project <name>` | `traffic.source.connected`, `recent-data`, `credentials`, `scopes` checks |
+| MCP toolkit `traffic` | Tools: `canonry_traffic_status`, `_sources_list`, `_source_get`, `_events`, `_connect_cloud_run`, `_sync` |
+
+## Doctor signals
+
+The doctor checks are adapter-agnostic. When they fail or warn:
+
+| Check | Code | What to do |
+|---|---|---|
+| `traffic.source.connected` | `traffic.source.none` | No source — `canonry traffic connect cloud-run …` |
+| `traffic.source.connected` | `traffic.source.all-errored` | Re-connect the source. The check's `details.lastError` shows the underlying reason. |
+| `traffic.source.recent-data` | `traffic.recent-data.stale` | Last sync was >7d ago. Run `canonry traffic sync …` or schedule a recurring sync. |
+| `traffic.source.recent-data` | `traffic.recent-data.empty` | Source connected but no data in 30d. Verify config and credentials with `canonry traffic sources <project>`. |
+| `traffic.source.credentials` | `traffic.credentials.resolve-failed` | Service-account key in `~/.canonry/config.yaml` is invalid or expired. Re-connect. |
+
+## Scheduling
+
+`canonry schedule` supports `--kind traffic-sync`. Recurring syncs are
+safe because of the `last_event_ids` cross-sync dedupe ring buffer
+described above. Recommended cadence:
+
+| Cadence | Use case |
+|---|---|
+| `0 */6 * * *` (every 6h) | Production agencies tracking active client sites |
+| `0 0 * * *` (daily) | Lower-traffic sites or local dev |
+| Manual only | First few weeks while validating data |
+
+## Telemetry
+
+Every successful or failed sync emits a `traffic.synced` event to the
+canonry telemetry pipeline:
+
+```jsonc
+{
+  "event": "traffic.synced",
+  "errorCode": "PROVIDER_AUTH",       // present only when status='failed'
+  "properties": {
+    "status": "completed" | "failed",
+    "sourceType": "cloud-run",        // adapter type
+    "sourceId": "<uuid>",             // opaque
+    "pulledEvents": 234,
+    "crawlerHits": 200,
+    "aiReferralHits": 12,
+    "durationMs": 4150
+  }
+}
+```
+
+Counts are aggregate. The sourceId is an opaque UUID. No raw paths,
+domains, or PII are surfaced.
+
+## Limits & caveats
+
+- **Path-level citation cross-reference is not implemented yet.** The
+  citation store is domain-grain (`query_snapshots.cited_domains`). A
+  future iteration that lands URL-grain citation evidence will extend
+  the `topCrawledPaths` entry with a `citationState` flag. Until then,
+  treat the report's crawled-paths table as "engine attention" — the
+  signal is the bot fetched it, not whether it was cited.
+- **Verified vs unverified.** The headline numbers count only
+  rDNS-verified hits. Unverified bots claim a known UA but couldn't be
+  cross-confirmed via reverse-DNS — they may be the real bot or an
+  imitator. Don't promote unverified counts in client-facing copy.
+- **Cloud Run only in v1.** WordPress plugin and other adapters are
+  planned. The doctor checks and the report renderer are already
+  adapter-agnostic — adding a new adapter is just a new entry in
+  `traffic_sources.source_type` and a `TrafficSourceValidator`
+  registration.

package/dist/{chunk-7VDM3JBI.js → chunk-OHPZXTFC.js}

@@ -17118,6 +17118,7 @@ async function trafficRoutes(app, opts) {
       Math.min(windowEnd.getTime(), Math.max(requestedStartMs, lastSyncedMs))
     );
     const startedAt = windowEnd.toISOString();
+    const syncStartedAtMs = windowEnd.getTime();
     const runId = crypto20.randomUUID();
     app.db.insert(runs).values({
       id: runId,
@@ -17129,19 +17130,32 @@ async function trafficRoutes(app, opts) {
       startedAt,
       createdAt: startedAt
     }).run();
-    const markFailed = (msg) => {
+    const markFailed = (msg, errorCode) => {
       const failedAt = (/* @__PURE__ */ new Date()).toISOString();
       app.db.transaction((tx) => {
         tx.update(runs).set({ status: RunStatuses.failed, error: msg, finishedAt: failedAt }).where(eq23(runs.id, runId)).run();
         tx.update(trafficSources).set({ status: TrafficSourceStatuses.error, lastError: msg, updatedAt: failedAt }).where(eq23(trafficSources.id, sourceRow.id)).run();
       });
+      try {
+        opts.onTrafficSynced?.({
+          status: "failed",
+          sourceType: sourceRow.sourceType,
+          sourceId: sourceRow.id,
+          pulledEvents: 0,
+          crawlerHits: 0,
+          aiReferralHits: 0,
+          durationMs: Date.now() - syncStartedAtMs,
+          errorCode
+        });
+      } catch {
+      }
     };
     let accessToken;
     try {
       accessToken = await resolveAccessToken2(credential);
     } catch (e) {
       const msg = e instanceof Error ? e.message : String(e);
-      markFailed(msg);
+      markFailed(msg, "PROVIDER_AUTH");
       throw providerError(`Failed to resolve Cloud Run access token: ${msg}`);
     }
     let allEvents = [];
@@ -17158,7 +17172,7 @@ async function trafficRoutes(app, opts) {
       allEvents = page.events;
     } catch (e) {
       const msg = e instanceof Error ? e.message : String(e);
-      markFailed(msg);
+      markFailed(msg, "PROVIDER_PULL");
       throw providerError(`Cloud Run pull failed: ${msg}`);
     }
     const seenEventIds = new Set(parseJsonColumn(sourceRow.lastEventIds, []));
@@ -17292,6 +17306,18 @@ async function trafficRoutes(app, opts) {
       entityType: "traffic_source",
       entityId: sourceRow.id
     });
+    try {
+      opts.onTrafficSynced?.({
+        status: "completed",
+        sourceType: sourceRow.sourceType,
+        sourceId: sourceRow.id,
+        pulledEvents: report.totals.normalizedEvents,
+        crawlerHits: report.totals.crawlerHits,
+        aiReferralHits: report.totals.aiReferralHits,
+        durationMs: Date.now() - syncStartedAtMs
+      });
+    } catch {
+    }
     const response = {
       sourceId: sourceRow.id,
       runId,
@@ -18633,7 +18659,8 @@ async function apiRoutes(app, opts) {
     await api.register(trafficRoutes, {
       cloudRunCredentialStore: opts.cloudRunCredentialStore,
       pullCloudRunEvents: opts.pullCloudRunEvents,
-      resolveCloudRunAccessToken: opts.resolveCloudRunAccessToken
+      resolveCloudRunAccessToken: opts.resolveCloudRunAccessToken,
+      onTrafficSynced: opts.onTrafficSynced
     });
     await api.register(backlinksRoutes, {
       getBacklinksStatus: opts.getBacklinksStatus,
@@ -25830,6 +25857,17 @@ async function createServer(opts) {
     wordpressConnectionStore,
     ga4CredentialStore,
     cloudRunCredentialStore,
+    onTrafficSynced: (event) => {
+      trackEvent("traffic.synced", {
+        status: event.status,
+        sourceType: event.sourceType,
+        sourceId: event.sourceId,
+        pulledEvents: event.pulledEvents,
+        crawlerHits: event.crawlerHits,
+        aiReferralHits: event.aiReferralHits,
+        durationMs: event.durationMs
+      }, event.errorCode ? { errorCode: event.errorCode } : void 0);
+    },
     onRunCreated: (runId, projectId, providers2, location) => {
       jobRunner.executeRun(runId, projectId, providers2, location).catch((err) => {
         app.log.error({ runId, err }, "Job runner failed");

package/dist/cli.js

@@ -20,7 +20,7 @@ import {
   setTelemetrySource,
   showFirstRunNotice,
   trackEvent
-} from "./chunk-7VDM3JBI.js";
+} from "./chunk-OHPZXTFC.js";
 import {
   CliError,
   EXIT_SYSTEM_ERROR,

package/dist/index.js

@@ -1,6 +1,6 @@
 import {
   createServer
-} from "./chunk-7VDM3JBI.js";
+} from "./chunk-OHPZXTFC.js";
 import {
   loadConfig
 } from "./chunk-P3SFTXHG.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ainyc/canonry",
-  "version": "4.18.1",
+  "version": "4.19.0",
   "type": "module",
   "description": "Agent-first open-source AEO operating platform - track how answer engines cite your domain",
   "license": "FSL-1.1-ALv2",
@@ -63,19 +63,19 @@
     "@ainyc/canonry-db": "0.0.0",
     "@ainyc/canonry-intelligence": "0.0.0",
     "@ainyc/canonry-integration-bing": "0.0.0",
-    "@ainyc/canonry-api-routes": "0.0.0",
-    "@ainyc/canonry-integration-commoncrawl": "0.0.0",
     "@ainyc/canonry-integration-cloud-run": "0.0.0",
     "@ainyc/canonry-contracts": "0.0.0",
     "@ainyc/canonry-integration-google": "0.0.0",
     "@ainyc/canonry-integration-traffic": "0.0.0",
-    "@ainyc/canonry-integration-wordpress": "0.0.0",
+    "@ainyc/canonry-api-routes": "0.0.0",
     "@ainyc/canonry-provider-cdp": "0.0.0",
+    "@ainyc/canonry-integration-wordpress": "0.0.0",
     "@ainyc/canonry-provider-claude": "0.0.0",
-    "@ainyc/canonry-provider-gemini": "0.0.0",
+    "@ainyc/canonry-integration-commoncrawl": "0.0.0",
     "@ainyc/canonry-provider-local": "0.0.0",
+    "@ainyc/canonry-provider-openai": "0.0.0",
     "@ainyc/canonry-provider-perplexity": "0.0.0",
-    "@ainyc/canonry-provider-openai": "0.0.0"
+    "@ainyc/canonry-provider-gemini": "0.0.0"
   },
   "scripts": {
     "build": "tsx scripts/copy-agent-assets.ts && tsup && tsx build-web.ts",