@aiguru/google-web-operations 0.6.4

package/.env.example

@@ -0,0 +1,42 @@
+# Copy to .env. Never commit .env.
+
+# --- Google ------------------------------------------------------------------
+# Durable path (recommended): put the "Desktop app" OAuth client JSON from
+# Google Cloud Console at secrets/oauth-client.json and run once:
+#   npm run cli -- auth login
+# The refresh token is stored encrypted under secrets/ (DPAPI on Windows,
+# AES-256-GCM elsewhere). Nothing else is needed in this file.
+
+# Development bridge only: a short-lived bearer token, expires in ~1 hour.
+# Used when no stored authorisation exists, or when GWO_GOOGLE_AUTH=env.
+GOOGLE_ACCESS_TOKEN=
+GWO_GOOGLE_AUTH=
+
+# Optional. PageSpeed works without a key at a lower quota.
+GOOGLE_PAGESPEED_API_KEY=
+
+# --- Paths (optional overrides) ---------------------------------------------
+# Defaults resolve relative to the repository root, not the working directory.
+GWO_SITES_DIR=
+GWO_DATA_DIR=
+GWO_POLICY_FILE=
+GWO_SECRETS_DIR=
+# Force the portable AES store on Windows (tests, CI): GWO_TOKEN_STORE=aes
+GWO_TOKEN_STORE=
+
+# --- Client reports ---------------------------------------------------------
+# A .dotx/.docx whose header, footer and styles the monthly client report inherits.
+# Without it the report is a plain, correctly styled document.
+GWO_DOCX_TEMPLATE=
+
+# --- Delivery (worker --deliver) -------------------------------------------
+RESEND_API_KEY=
+GWO_REPORT_FROM=ops@aiguru.co.il
+GWO_REPORT_TO=ai@aiguru.co.il
+
+# --- GitHub (pull requests for repository-backed sites) --------------------
+# Fine-grained token: Contents write + Pull requests write on the site repos.
+GITHUB_TOKEN=
+
+# --- NPM JS ACCESS TOKEN - gwo-release --------------------
+NPM_TOKEN=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

package/README.md

@@ -0,0 +1,110 @@
+# Google Web Operations
+
+Agent-run SEO and GEO operations for a portfolio of client websites. One repository, one core, one MCP server, two hosts: Claude Code and Codex. A scheduled worker runs the sensing half without any agent.
+
+Repository: https://github.com/ai-gur/GWeb-Ops
+
+## What it does
+
+For each registered site: observe the live site and (with access) Search Console, Analytics, Tag Manager and PageSpeed; turn observations into ranked findings; decide against a playbook; act within a strict permission model; verify every write; record everything.
+
+Daily health pass, weekly improvement pass, monthly deep scan and client report, incident triage, onboarding.
+
+## Install (one package, both hosts)
+
+Needs Node 22.18 or newer. The compiled package is published to the public npm registry on every release (the source repository stays private):
+
+```
+npm i -g @aiguru/google-web-operations      # latest release
+gwo install                                 # MCP server + skills + slash commands for Claude Code and Codex; creates ~/.gwo
+gwo update                                  # later: fetch the latest release and re-register the hosts
+```
+
+The release page carries the same tarball (`google-web-operations-latest.tgz` and a versioned copy) for machines without registry access: `npm i -g ./google-web-operations-latest.tgz`. Until the `NPM_TOKEN` secret is set in the repository, releases go to GitHub Packages as `@ai-gur/google-web-operations` instead, which needs a GitHub token with `read:packages` in `~/.npmrc`.
+
+Restart the hosts, run `gwo doctor` to see where things stand, then start with `/gwo-setup` (Claude Code) or "set up Google Web Operations" (Codex): it connects the Google Cloud project, authorises the operations account and registers the first site, all from the chat. Operator state (sites, data, secrets, `.env`) lives in `~/.gwo` (`GWO_HOME`) and survives upgrades; `gwo install` again after `npm i -g` of a newer version.
+
+`gwo install claude`, `gwo install codex`, `--print` to preview, `gwo uninstall` to remove the host entries. A locally built tarball installs with `npm i -g ./dist/aiguru-google-web-operations-<version>.tgz` (the `./` matters, or npm reads it as a package name).
+
+## Development (repository checkout)
+
+```
+npm install
+npm run check                    # typecheck, tests, regenerate hosts, build the package, run its bin
+npm run pack:release             # dist/aiguru-google-web-operations-<version>.tgz (compiled JavaScript)
+npm run cli -- sites
+npm run cli -- scan aiguru       # crawls, runs 40+ rules, saves scan + snapshot
+npm run smoke:mcp                # boots the MCP server and lists tools
+node tools/install.ts --print    # what gwo install would do from this checkout
+```
+
+In a checkout, state stays next to the code (`sites/`, `.gwo-data/`, `secrets/`, `.env`). Site scans need no credentials. Google tools need one consent for the operations account, from the chat (`/gwo-setup`) or the CLI:
+
+```
+npm run cli -- auth login        # browser consent once; token stored encrypted under secrets/
+npm run cli -- discover --apply  # fills Search Console and GA4 ids into sites/*.yaml
+```
+
+### Commands
+
+`/gwo-setup`, `/gwo-add-site <url>`, `/gwo-daily`, `/gwo-weekly`, `/gwo-monthly`, `/gwo-triage <site>`, `/gwo-onboard <site>`, `/gwo-scan <site>`, `/gwo-measure <site>`. Codex reads the same skills; `hosts/codex/AGENTS.md` is there for projects that take instructions rather than skills.
+
+### Rendered checks (optional)
+
+`web_inspect_rendered` and `gwo inspect <url> --rendered` need Playwright once per machine: `npm i -g playwright && npx playwright install chromium`. Without it every other tool works; the rendered check explains what to install.
+
+### Worker (scheduled)
+
+```
+gwo worker daily                 # scan, diff, escalate, report
+gwo worker weekly --deliver      # also email the report through Resend
+gwo schedule install             # Windows Task Scheduler: 07:00 daily, Monday 07:30
+```
+
+Exit code 0 healthy, 2 escalations to triage, 1 scan failures. Reports are kept under `<home>/.gwo-data/_runs/`. For an always-on host, `gwo worker loop` or the container in `deploy/` (`docs/HOSTED-WORKER.md`).
+
+## Layout
+
+```
+apps/
+  mcp-server/      one server for both hosts
+  worker/          scheduled sensing with escalation rules, no agent needed
+  cli/             local admin
+packages/
+  core/            registry, operations engine, rule engine, write loop, types
+  connectors/
+    web-crawler/   fetch, robots, sitemap, parse
+    google/        OAuth, Search Console, GA4, Tag Manager, PageSpeed, discovery
+    odoo/          JSON-RPC site adapter
+    repo/          gwo.seo.json manifest adapter
+    github/        pull requests for repository-backed sites
+    resend/        report delivery
+  policy/          risk classes and enforcement
+  storage/         scans, snapshots, GTM snapshots, audit log
+  playbooks/       THE source of operating knowledge
+hosts/             GENERATED from playbooks. Never edit.
+sites/             one YAML per site (in ~/.gwo/sites when installed). secrets in *.secrets.yaml (gitignored)
+policies/          default.yaml, the permission model
+docs/              spec, security, adapters, ADRs, next-session bootstrap
+tests/
+tools/             build-hosts, install (gwo install), smoke-mcp, schedule.ps1
+bin/               gwo, the package executable
+```
+
+## Permission model
+
+| Class | Examples | Gate |
+|---|---|---|
+| `auto` | read, scan, report, submit sitemap, fill an empty meta description | none |
+| `approval_required` | any content, title, schema, canonical, redirect, tag or config change; pull requests | token `<site_id>:<action>` in-session; a Tag Manager publish also needs the review token of the diff shown (ADR-005) |
+| `protected` | delete property or container, account, retention, access, billing | never |
+
+Defined in `policies/default.yaml`, enforced in `packages/policy`. Every write runs through `performWrite()`: enforce, read before, execute, read after, verify, record.
+
+## Runtime
+
+TypeScript, executed natively by Node 22.18+ (type stripping). No build step, no transpiler at runtime. `tsc --noEmit` checks types. See ADR-004.
+
+## Status
+
+v0.6. Installable as one npm package with `gwo install` for both hosts and `gwo update` for the latest release (ADR-006); `gwo doctor` says what is missing. The scanner has an IS 5568 accessibility area, `web_inspect_rendered` compares static HTML with the rendered DOM (Playwright, optional), the worker runs hosted (`deploy/`, `docs/HOSTED-WORKER.md`), and the Odoo adapter addresses product pages. Hebrew quick start: `docs/GETTING-STARTED.he.md`. Site sensing, rule engine, policy, storage, playbooks, hosts, worker and CLI are live on two pilot sites. Google access is durable; Search Console, GA4, PageSpeed and Tag Manager reads are verified live. Site adapters for Odoo and repository-backed sites exist. Writes through the loop: sitemap submission, meta descriptions, titles, structured data (Odoo view arch or repository manifest), GA4 key events, pull requests, and Tag Manager changes staged in a `gwo-` workspace, diffed, and published under two tokens. The monthly client report renders to a Word document in the company template. The worker runs on a schedule with escalation rules and optional email delivery. Setup and site registration run from the chat. 44 MCP tools, 10 skills. See `docs/PRODUCT-TECHNICAL-SPEC-v0.6.md` and `docs/BOOTSTRAP-NEXT-PROMPT.md`.

package/apps/cli/src/index.js

@@ -0,0 +1,286 @@
+#!/usr/bin/env node
+/**
+ * Local admin CLI. Exercises the core without any agent host.
+ *
+ *   node apps/cli/src/index.ts sites
+ *   node apps/cli/src/index.ts scan <site> [--pages N] [--json]
+ *   node apps/cli/src/index.ts inspect <url> [--site <site>]
+ *   node apps/cli/src/index.ts diff <site>
+ *   node apps/cli/src/index.ts classify <action> [--site <site>]
+ *   node apps/cli/src/index.ts audit <site> [--limit N]
+ *   node apps/cli/src/index.ts auth login [--scope <extra>]... | status | logout
+ *   node apps/cli/src/index.ts discover [--apply]
+ *   node apps/cli/src/index.ts gtm workspaces|stage|wdiff|publish <site> ...
+ *   node apps/cli/src/index.ts content meta|title|schema <site> <path> ...
+ */
+import { readFileSync } from 'node:fs';
+import { operations } from "../../../packages/core/src/index.js";
+import * as admin from "../../../packages/core/src/admin.js";
+import { openPullRequest } from "../../../packages/connectors/github/src/index.js";
+import { login, logout, makeOauthProvider, tokenProvider, discover, SCOPES_FOR } from "../../../packages/connectors/google/src/index.js";
+const [, , cmd, ...rest] = process.argv;
+const positional = rest.filter((r, i) => !r.startsWith('--') && !(rest[i - 1] ?? '').startsWith('--'));
+const flag = (name) => { const i = rest.indexOf(name); return i >= 0 ? rest[i + 1] : undefined; };
+const has = (name) => rest.includes(name);
+const out = (v) => console.log(typeof v === 'string' ? v : JSON.stringify(v, null, 2));
+try {
+    switch (cmd) {
+        case 'sites': {
+            for (const s of operations.listSites({ includeDisabled: true })) {
+                console.log(`${s.enabled ? '*' : ' '} ${s.site_id.padEnd(16)} ${s.production_url.padEnd(34)} ${s.platform.padEnd(9)} ${s.tier}`);
+            }
+            break;
+        }
+        case 'scan': {
+            const site = positional[0];
+            if (!site)
+                throw new Error('usage: scan <site> [--pages N] [--json]');
+            const { scan, scanPath } = await operations.scanSite(site, { maxPages: Number(flag('--pages') ?? 50) });
+            if (has('--json'))
+                out(scan);
+            else {
+                out(operations.renderScan(scan));
+                console.log(`\nsaved: ${scanPath}`);
+            }
+            break;
+        }
+        case 'inspect': {
+            const url = positional[0];
+            if (!url)
+                throw new Error('usage: inspect <url> [--site <site>]');
+            if (has('--rendered')) {
+                const r = await operations.inspectRendered(url, flag('--site'));
+                out(r.rendered);
+            }
+            else
+                out(await operations.inspectUrl(url, flag('--site')));
+            break;
+        }
+        case 'diff': {
+            const site = positional[0];
+            if (!site)
+                throw new Error('usage: diff <site>');
+            out(operations.diffLatest(site));
+            break;
+        }
+        case 'classify': {
+            const action = positional[0];
+            if (!action)
+                throw new Error('usage: classify <action> [--site <site>]');
+            out(operations.classifyAction(action, flag('--site')));
+            break;
+        }
+        case 'audit': {
+            const site = positional[0];
+            if (!site)
+                throw new Error('usage: audit <site> [--limit N]');
+            out(operations.auditLog(site, Number(flag('--limit') ?? 50)));
+            break;
+        }
+        case 'auth': {
+            const sub = positional[0];
+            if (sub === 'login') {
+                // --scope may be repeated; accept the short names from SCOPES_FOR or full URLs.
+                const extra = rest.flatMap((r, i) => (r === '--scope' ? [rest[i + 1] ?? ''] : []))
+                    .map((s) => SCOPES_FOR[s] ?? s).filter(Boolean);
+                const r = await login({ scopes: extra, open: !has('--no-open') });
+                out({ authorised: true, ...r });
+            }
+            else if (sub === 'status') {
+                out({ active: tokenProvider().describe(), oauth: makeOauthProvider().status() });
+            }
+            else if (sub === 'logout') {
+                logout();
+                out('Stored Google authorisation removed.');
+            }
+            else
+                throw new Error('usage: auth login [--no-open] [--scope ga4_write|gtm_edit|gtm_publish|<url>]... | auth status | auth logout');
+            break;
+        }
+        case 'sitemap': {
+            const [sub, site] = positional;
+            if (sub !== 'submit' || !site)
+                throw new Error('usage: sitemap submit <site> [--approval <token>]');
+            const r = await operations.submitSitemap(site, flag('--approval') ? { approval: flag('--approval') } : {});
+            out({ status: r.status, verified: r.verified, reason: r.reason, before: r.audit.before, after: r.audit.after });
+            break;
+        }
+        case 'site': {
+            const [sub, site, p] = positional;
+            if (sub === 'ping' && site)
+                out(await operations.adapterPing(site));
+            else if (sub === 'pages' && site)
+                out(await operations.adapterPages(site));
+            else if (sub === 'seo' && site && p)
+                out(await operations.adapterPageSeo(site, p));
+            else if (sub === 'schema' && site && p)
+                out(await operations.adapterPageSchema(site, p));
+            else
+                throw new Error('usage: site ping <site> | site pages <site> | site seo <site> <path> | site schema <site> <path>');
+            break;
+        }
+        case 'content': {
+            const [sub, site, p] = positional;
+            const text = flag('--text');
+            const approval = flag('--approval') ? { approval: flag('--approval') } : {};
+            const show = (r) => out(r.status === 'invalid' ? r : { status: r.status, verified: r.verified, reason: r.reason, before: r.audit.before, after: r.audit.after });
+            if (sub === 'validate' && text)
+                out(has('--title') ? operations.validateTitle(text) : operations.validateMetaDescription(text));
+            else if (sub === 'meta' && site && p && text)
+                show(await operations.setMetaDescription(site, p, text, approval));
+            else if (sub === 'title' && site && p && text)
+                show(await operations.setTitle(site, p, text, approval));
+            else if (sub === 'schema' && site && p && flag('--file')) {
+                const parsed = JSON.parse(readFileSync(flag('--file'), 'utf8'));
+                show(await operations.setSchema(site, p, Array.isArray(parsed) ? parsed : [parsed], approval));
+            }
+            else
+                throw new Error('usage: content validate --text "..." [--title] | content meta|title <site> <path> --text "..." [--approval <token>] | content schema <site> <path> --file blocks.json [--approval <token>]');
+            break;
+        }
+        case 'ga4': {
+            const [sub, site, ev] = positional;
+            if (sub === 'key-event' && site && ev) {
+                const r = await operations.createKeyEvent(site, ev, flag('--approval') ? { approval: flag('--approval') } : {});
+                out({ status: r.status, verified: r.verified, reason: r.reason, before: r.audit.before, after: r.audit.after });
+            }
+            else
+                throw new Error('usage: ga4 key-event <site> <eventName> [--approval <token>]');
+            break;
+        }
+        case 'repo': {
+            const [sub, site] = positional;
+            if (sub !== 'pr' || !site)
+                throw new Error('usage: repo pr <site> [--approval <token>] [--change "..."]...');
+            const changes = rest.flatMap((r, i) => (r === '--change' ? [rest[i + 1] ?? ''] : [])).filter(Boolean);
+            const r = await openPullRequest(site, { changes, ...(flag('--approval') ? { approval: flag('--approval') } : {}) });
+            out({ status: r.status, verified: r.verified, reason: r.reason, pr: r.after });
+            break;
+        }
+        case 'gtm': {
+            const [sub, site] = positional;
+            if (sub === 'containers')
+                out(await operations.gtmContainers());
+            else if (sub === 'versions' && site)
+                out(await operations.gtmVersions(site));
+            else if (sub === 'snapshot' && site) {
+                const r = await operations.gtmSnapshot(site);
+                out(r.rendered);
+                console.log(`\nsaved: ${r.file}`);
+            }
+            else if (sub === 'diff' && site)
+                out(operations.gtmDiff(site));
+            else if (sub === 'workspaces' && site)
+                out(await operations.gtmWorkspaces(site));
+            else if (sub === 'stage' && site && flag('--file')) {
+                const entities = JSON.parse(readFileSync(flag('--file'), 'utf8'));
+                const r = await operations.gtmStage(site, entities, { ...(flag('--approval') ? { approval: flag('--approval') } : {}), ...(flag('--reason') ? { reason: flag('--reason') } : {}), ...(flag('--workspace') ? { workspaceId: flag('--workspace') } : {}) });
+                out(r.rendered);
+            }
+            else if (sub === 'wdiff' && site) {
+                const r = await operations.gtmWorkspaceDiff(site, flag('--workspace'));
+                out('error' in r ? r : r.rendered);
+            }
+            else if (sub === 'publish' && site && flag('--review')) {
+                out(await operations.gtmPublish(site, { review: flag('--review'), ...(flag('--approval') ? { approval: flag('--approval') } : {}), ...(flag('--workspace') ? { workspaceId: flag('--workspace') } : {}), ...(flag('--name') ? { versionName: flag('--name') } : {}) }));
+            }
+            else
+                throw new Error('usage: gtm containers | versions <site> | snapshot <site> | diff <site> | workspaces <site> | stage <site> --file entities.json [--reason "..."] [--approval <token>] | wdiff <site> [--workspace id] | publish <site> --review <token> [--approval <token>] [--name "..."]');
+            break;
+        }
+        case 'setup': {
+            out(admin.setupStatus());
+            break;
+        }
+        case 'doctor': {
+            out((await import("../../../packages/core/src/doctor.js")).renderDoctor());
+            break;
+        }
+        case 'add-site': {
+            const url = positional[0];
+            if (!url || !flag('--name'))
+                throw new Error('usage: add-site <url> --name "..." [--platform odoo|nextjs|wordpress|custom] [--odoo-url ...] [--odoo-db ...] [--tier full|advisory]');
+            const r = admin.addSite({ url, name: flag('--name'), ...(flag('--platform') ? { platform: flag('--platform') } : {}), ...(flag('--tier') ? { tier: flag('--tier') } : {}),
+                ...(flag('--odoo-url') && flag('--odoo-db') ? { odoo: { url: flag('--odoo-url'), db: flag('--odoo-db') } } : {}) });
+            out({ file: r.file, site_id: r.site.site_id });
+            break;
+        }
+        case 'set': {
+            const [site, p, value] = positional;
+            if (!site || !p || value === undefined)
+                throw new Error('usage: set <site> <dotted.path> <value>');
+            out(admin.updateSite(site, p, value));
+            break;
+        }
+        case 'suppress': {
+            const [site, rule] = positional;
+            if (!site || !rule || !flag('--reason'))
+                throw new Error('usage: suppress <site> <rule> [--path /x/] --reason "..."');
+            out(admin.addSuppression(site, { rule, ...(flag('--path') ? { path: flag('--path') } : {}), reason: flag('--reason') }));
+            break;
+        }
+        case 'report': {
+            const [sub, site] = positional;
+            if (sub !== 'client' || !site)
+                throw new Error('usage: report client <site> [--month YYYY-MM] [--lang he|en] [--next "..."] [--headline "..."]');
+            const r = await operations.renderClientReport(site, {
+                ...(flag('--month') ? { month: flag('--month') } : {}), ...(flag('--lang') ? { lang: flag('--lang') } : {}),
+                narrative: { ...(flag('--headline') ? { headline: flag('--headline') } : {}), ...(flag('--next') ? { next: flag('--next') } : {}) },
+            });
+            out(r.markdown);
+            console.log(`\ndocx: ${r.docx}\ntemplate: ${r.template ?? 'none'}${r.notes.length ? `\nnotes: ${r.notes.join('; ')}` : ''}`);
+            break;
+        }
+        case 'discover': {
+            const r = await discover({ apply: has('--apply') });
+            out({
+                inventory: { searchConsole: r.inventory.gsc.length, ga4: r.inventory.ga4.length, tagManager: r.inventory.gtm.length },
+                matches: r.matches,
+                applied: r.applied,
+                hint: has('--apply') ? undefined : 'Re-run with --apply to write matched ids into sites/*.yaml.',
+            });
+            break;
+        }
+        default:
+            if (cmd && cmd !== 'help' && cmd !== '--help' && cmd !== '-h')
+                console.error(`unknown command "${cmd}"\n`);
+            console.log([
+                'gwo: Google Web Operations',
+                '',
+                'Getting started',
+                '  gwo install [claude|codex|all] [--print]   register the MCP server, skills and commands with the hosts',
+                '  gwo doctor                                 what is set up, what is missing, the next step',
+                '  gwo add-site <url> --name "..." [--platform odoo|nextjs|wordpress|custom] [--odoo-url ... --odoo-db ...]',
+                '  gwo auth login [--scope ga4_write|gtm_edit|gtm_publish] | auth status | auth logout',
+                '  gwo discover [--apply]                     match Search Console, GA4 and Tag Manager to sites',
+                '  gwo update                                 latest release, then re-register the hosts',
+                '',
+                'Look',
+                '  gwo sites | scan <site> [--pages N] [--json] | inspect <url> [--site <site>] [--rendered] | diff <site>',
+                '  gwo audit <site> [--limit N] | classify <action> [--site <site>] | setup (JSON) | doctor (text)',
+                '  gwo gtm containers | versions <site> | snapshot <site> | diff <site> | workspaces <site>',
+                '  gwo site ping <site> | pages <site> | seo <site> <path> | schema <site> <path>',
+                '',
+                'Change (policy-gated; approval tokens look like <site>:<action>)',
+                '  gwo sitemap submit <site>',
+                '  gwo content validate --text "..." [--title]',
+                '  gwo content meta|title <site> <path> --text "..." [--approval <token>]',
+                '  gwo content schema <site> <path> --file blocks.json [--approval <token>]',
+                '  gwo ga4 key-event <site> <eventName> [--approval <token>]',
+                '  gwo gtm stage <site> --file entities.json [--reason "..."] [--approval <token>]',
+                '  gwo gtm wdiff <site> | gtm publish <site> --review <token> [--approval <token>]',
+                '  gwo repo pr <site> [--approval <token>] [--change "..."]...',
+                '  gwo set <site> <dotted.path> <value> | suppress <site> <rule> [--path /x/] --reason "..."',
+                '',
+                'Report and schedule',
+                '  gwo report client <site> [--month YYYY-MM] [--lang he|en] [--headline "..."] [--next "..."]',
+                '  gwo worker daily|weekly|monthly [--deliver] | worker loop | schedule install|status|remove',
+                '',
+                'Docs: docs/INSTALL.he.md, docs/GETTING-STARTED.he.md, README.md. In the chat: /gwo-setup.',
+            ].join('\n'));
+    }
+}
+catch (e) {
+    console.error(e.message);
+    process.exitCode = 1;
+}