@cloudgrid-io/mcp 0.13.0 → 0.15.0

package/src/corpus/templates/saas-marketing/cloudgrid.yaml

@@ -1,10 +1,228 @@
-# static — deploys as an inspiration (instant, works on hosted) via gridctl_drop;
-# this yaml is only for plugging it as an OWNED static runtime (async, local
-# edition only). To publish the page, prefer gridctl_drop, not grid plug.
+# ============================================================================
+# cloudgrid.yaml — Complete Example
+# ============================================================================
 #
-# needs: {}  # none — static, no infrastructure
+# This file shows EVERY field the platform supports, with comments explaining
+# each one. It is a reference — no real project would use all of these at once.
+#
+# STANDARD: every CloudGrid template ships its cloudgrid.yaml in THIS full form.
+# Uncomment ONLY the fields the archetype needs (always: name + services; plus
+# needs: for runtime apps). Leave everything else commented. Comments are
+# ignored by the parser, so the file deploys to exactly the active fields.
+#
+# Only two fields are required: name and services. Everything else is optional.
+# Full docs: docs/cloudgrid-yaml-reference.md
+# ============================================================================
+
+
+# ─── IDENTITY ───────────────────────────────────────────────────────────────
+
+# Entity slug (REQUIRED). 2-42 chars, lowercase alphanumeric + hyphens.
+# Must start and end with a letter or digit.
+# Becomes part of your public URL: <name>-<4hex>.<org>.cloudgrid.io
 name: saas-marketing
+
+# One-line summary. Max 280 chars. If omitted, Grid Brain auto-generates one.
+# description: Order processing API for the e-commerce platform
+
+# Single emoji or short text shown as the entity icon. Max 64 chars.
+# icon: "📦"
+
+# Entity type: 'app' (default) or 'agent'. An agent is an app with an agent: block.
+# kind: app
+
+
+# ─── NETWORKING ─────────────────────────────────────────────────────────────
+
+# Whether this entity serves public HTTP traffic.
+# true = public URL + Ingress (default). false = internal only.
+# expose: true
+
+# Additional domain names this entity serves on (the platform URL is always active).
+# custom_domains:
+#   - orders.example.com
+#   - shop.example.com
+
+
+# ─── HEALTH PROBE (entity-level override) ──────────────────────────────────
+
+# Override the default health check for the entire entity.
+# probe:
+#   path: /healthz               # HTTP path to probe (default: /).
+#   expected: "<500"             # '<500' (default), '200', '2xx'.
+
+
+# ─── DEPLOY NOTIFICATIONS ──────────────────────────────────────────────────
+
+# Slack notification to the org's default channel on deploy failure.
+# true = notify (default), false = silent.
+# notify: true
+
+
+# ─── SCALING ────────────────────────────────────────────────────────────────
+
+# Replica scaling intent.
+#   'auto'  — platform decides from the grid tier (default).
+#   'alert' — single-replica, emit a SPOF warning at plug time.
+#   1-10    — explicit replica count, clamped to the grid max.
+# scale: auto
+
+
+# ─── INFRASTRUCTURE NEEDS ──────────────────────────────────────────────────
+#
+# Declare infrastructure; the platform provisions it and injects connection
+# details as env vars. Nine needs: database, cache, kv, queue, pubsub, vector,
+# object_storage, disk, ai. Simplest form is `true`.
+#
+# needs:
+#   # database — MongoDB. Injects DATABASE_MONGODB_URL (+ legacy MONGODB_URL).
+#   #   true | pool | dedicated | { tier: external, secret: MY_DB }
+#   database: true
+#   # cache — Redis, LRU eviction (derived data). Injects CACHE_REDIS_URL (+ REDIS_URL).
+#   #   true | redis | { engine: external, secret: MY_REDIS_URL }
+#   cache: true
+#   # kv — Redis, no eviction (durable: flags, idempotency). Injects KV_REDIS_URL.
+#   kv: true
+#   # queue — Redis job-queue substrate (BullMQ). Injects QUEUE_REDIS_URL.
+#   queue: true
+#   # pubsub — Redis broadcast pub/sub. Injects PUBSUB_REDIS_URL.
+#   pubsub: true
+#   # vector — pgvector on Postgres. Injects VECTOR_PGVECTOR_URL (+ PGVECTOR_URL).
+#   #   true | pgvector | { engine: pgvector, dim: 1536 }
+#   vector: pgvector
+#   # object_storage — GCS bucket. Injects OBJECT_STORAGE_GCS_BUCKET, _GCS_REGION.
+#   #   true | { size: 10Gi }
+#   object_storage: true
+#   # disk — persistent FS mount (PVC). Injects DISK_PATH. Pins to 1 replica.
+#   #   true | { size: 1Gi, mount: /var/data }  (forbidden: /proc /sys /dev)
+#   disk: true
+#   # ai — AI Gateway via @cloudgrid-io/ai. Injects AI_GATEWAY_URL. true | false.
+#   ai: true
+
+
+# ─── INTER-ENTITY COMMUNICATION ────────────────────────────────────────────
+
+# Entities this app calls (outbound). Service-account token injected per pair.
+# calls:
+#   - org:atomic/billing-api
+
+# Inbound caller policy — who can call YOUR entity.
+# callers:
+#   policy: auto                 # 'auto' | 'manual' (default).
+#   scope: org                   # 'org' (default) | 'space' | 'explicit'.
+#   fallback: deny               # 'deny' (default) | 'warn'.
+#   # allowed_apps:              # required only when scope: explicit
+#   #   - org:atomic/frontend
+
+
+# ─── VAULT ──────────────────────────────────────────────────────────────────
+
+# Map org vault items to env vars. Keys UPPER_SNAKE_CASE; values = vault item keys.
+# vault:
+#   STRIPE_KEY: stripe-live-key
+#   SENDGRID_API_KEY: sendgrid-key
+
+
+# ─── CONNECTORS AND HOOKS ──────────────────────────────────────────────────
+
+# connectors:
+#   - id: team-slack
+#     target: org                # 'system' | 'org'.
+#     channel: "#deploys"
+# hooks:
+#   - on: deploy.failed          # release.published | deploy.failed | feedback.received | entity.created
+#     via: team-slack
+#     template: "Deploy failed for {entity_name}: {error}"
+#     channel: "#alerts"
+# triggers:
+#   - from: team-slack
+#     event: message
+#     channel: "#support"
+#     route: /webhooks/slack
+
+
+# ─── LOCAL DEVELOPMENT ──────────────────────────────────────────────────────
+
+# Config for `grid dev`. Redact fields in dev data snapshots.
+# dev:
+#   snapshot:
+#     mask:
+#       - users.email
+#       - users.phone
+
+
+# ─── AGENT METADATA ────────────────────────────────────────────────────────
+
+# An agent is an app with kind: agent (above) and an agent: block.
+# agent:
+#   purpose: Sync customer data from external CRM every night
+#   schedule: "0 2 * * *"
+#   trigger: webhook
+
+
+# ─── PRIVATE RESOURCE AUTO-DEPENDENCIES ────────────────────────────────────
+
+# When true (default), private resources are auto-injected as depends_on.
+# auto_depends_on_private_requires: true
+
+
+# ─── SERVICES (REQUIRED) ───────────────────────────────────────────────────
+#
+# At least one service is required. Key = service name (slug rules).
+# Types: node, nextjs, python, static, cron.
+#
 services:
   web:
+    # type (REQUIRED): node | nextjs | python | static | cron
     type: static
+    # HTTP route: '/' | '/api' | false (internal) | omitted (=false + warning).
+    # Only one service per entity can claim '/'.
     path: /
+    # Container port (default 8080; read process.env.PORT). Not for cron.
+    # port: 3000
+    # nextjs only: base_image: distroless (default) | alpine
+    # base_image: distroless
+    # node only: lang: javascript (default) | typescript
+    # lang: typescript
+    # Custom source location (default services/<name>/). '.' = project root.
+    # source:
+    #   path: src/worker
+    # Health probe: false/omitted (TCP) | true (/health) | "/path" | {path,expected}
+    # health:
+    #   path: /api/health
+    #   expected: "200"
+    # Static runtime env vars (strings). Reserved: PORT, APP_NAME, SERVICE_NAME,
+    # NODE_ENV, MONGODB_URL, REDIS_URL, AI_GATEWAY_URL, N8N_WEBHOOK_URL, CLOUDGRID_*.
+    # env:
+    #   LOG_LEVEL: info
+    # Sibling services/private resources that must be Ready first (no cron, no cycles).
+    # depends_on:
+    #   - worker
+    # Per-service needs override (merges with app-level needs).
+    # needs:
+    #   queue: true
+    # Static-only build step:
+    # build:
+    #   command: npm run build
+    #   output: dist
+    #   env:
+    #     VITE_API_URL: https://api.example.com
+    # node_version: "22"          # static build step: '18'|'20'|'22'|'24' (quoted)
+    # Cron-only: schedule (REQUIRED), timezone (UTC|EST|PST), run (job | https URL)
+    # schedule: "0 9 * * 1"
+    # timezone: UTC
+    # run: job
+    # Verbose per-service persistence (prefer needs.disk):
+    # persist:
+    #   - name: scratch
+    #     path: /tmp/scratch
+    #     size: 500Mi
+
+
+# ============================================================================
+# NOTES
+# - NOT here: org:/entity_id: (→ .cloudgrid/link.json), secrets (→ grid secrets set),
+#   Dockerfile (platform generates from services.*.type).
+# - Deploy: grid plug   |   Local with grid resources: grid dev
+# - Full reference: docs/cloudgrid-yaml-reference.md  |  CLI: docs/cli-reference.md
+# ============================================================================

package/src/corpus/templates/saas-marketing/index.md

@@ -17,11 +17,9 @@ The fillable HTML is the real template — fetch it with
 ## cloudgrid.yaml (reference only)
 
 ```yaml
-# static — deploys as an inspiration (instant, works on hosted) via gridctl_drop;
-# this yaml is only for plugging it as an OWNED static runtime (async, local
-# edition only). To publish the page, prefer gridctl_drop, not grid plug.
-#
-# needs: {}  # none — static, no infrastructure
+# On disk this file is the full-annotated reference (templates/_cloudgrid.yaml.reference) with EVERY
+# field present as a comment; only the fields below are uncommented, so it
+# deploys to exactly these active fields.
 name: saas-marketing
 services:
   web:

package/src/corpus/templates/semantic-search/AGENTS.md

@@ -0,0 +1,192 @@
+# AGENTS.md — semantic-search template (CloudGrid wiring guide)
+
+This template is a **runnable app**, not a blueprint — it ships real FastAPI +
+React code. Read this guide, adapt the naming/metadata to the user's domain, wire
+the secrets, and deploy. The four "baked-in learnings" (health-without-secrets,
+startup index, `$text` guard, no vector/no cron) are already in the code — do not
+regress them.
+
+---
+
+## 1. File tree
+
+Every service's code lives under `services/<name>/` — `path:` in `cloudgrid.yaml`
+is the URL mount, NOT the filesystem path. Files at the repo root fail with
+`Error: Service directory not found: …/services/<name>`.
+
+```
+cloudgrid.yaml                          # name + services(web,backend) + needs:{database:true}
+services/web/                           # React + Vite static frontend, served at /
+  package.json  vite.config.js  index.html
+  src/main.jsx  App.jsx  api.js  styles.css
+  src/components/SearchView.jsx ManagerLogin.jsx Settings.jsx Dashboard.jsx
+services/backend/                       # FastAPI, mounted at /backend
+  requirements.txt
+  src/main.py                           # uvicorn app.main:app (reads PORT)
+  src/app/
+    main.py       config.py   db.py      search.py   indexing.py
+    source.py     embeddings.py  answer.py  auth.py
+```
+
+---
+
+## 2. Mongo collections + fields
+
+The grid provisions one MongoDB database (`needs: { database: true }`). Two
+primary collections (`_id` is the Mongo ObjectId on every doc):
+
+**`documents`** — one row per source document
+- `title` (string), `collection` (string, free-form grouping tag, nullable),
+  `date` (ISO date string, nullable) — the metadata filters.
+- `source_filename` (string, unique), `source_rev` (string — content hash / rev
+  for incremental sync), `file_path` (string, source-internal, never exposed).
+- `has_media` (bool), `word_count` (int), `indexed_at` (Date).
+
+**`chunks`** — one embeddable passage per document (the retrieval unit)
+- `document_id` (ObjectId → documents), `chunk_index` (int),
+- `content` (string — indexed by the `$text` index, returned as the passage),
+- `embedding` (float[] — the semantic vector; `null` if embeddings unconfigured).
+
+Support collections: `settings` (feature flags, synonyms, `catalog.last_refresh`),
+`usage_events`, `index_reports`, `sessions`.
+
+**Adapt the metadata to the domain:** rename/extend `collection` and `date` in
+`indexing.py::_parse_metadata`, and mirror the field names in `search.py`,
+`main.py`, and the web `SearchView`.
+
+---
+
+## 3. How CloudGrid injects the datastore
+
+You never provision infra or set a connection string by hand. `needs:{database:
+true}` provisions Mongo and injects it as an env var at `grid dev` (local) and at
+runtime (after `grid plug`):
+
+| What | Declared in cloudgrid.yaml | Injected env var(s) | Read in code |
+|------|----------------------------|---------------------|--------------|
+| MongoDB | `needs: { database: true }` | `DATABASE_MONGODB_URL` (+ legacy `MONGODB_URL`) | `app/db.py::get_client` (lazy) |
+
+Read it LAZILY — `os.environ.get("DATABASE_MONGODB_URL") or
+os.environ.get("MONGODB_URL")` inside `get_client()`, never at module top level.
+`requires:` is the deprecated v1 alias; never author it and never set `needs:` and
+`requires:` together (the validator rejects the combination).
+
+**No `needs: vector`.** The ideal embedding store is pgvector
+(`needs: { vector: pgvector }` → `VECTOR_PGVECTOR_URL`), but the runtime
+role-grant is **blocked on platform issue #1545**. Until it lands, embeddings live
+in the Mongo `chunks.embedding` float array and are cosine-ranked in-app
+(`search.py::_cosine`). When #1545 ships, move the vectors to pgvector and replace
+the in-app cosine block with an `ORDER BY embedding <=> $q LIMIT k` query — the
+rest of the pipeline is unchanged.
+
+---
+
+## 4. Pluggable document source (SOURCE_TYPE)
+
+The source is where raw documents live, not the query DB. `SOURCE_TYPE` selects
+the adapter in `app/source.py`; all creds are gated in `app/config.py` and read
+lazily:
+
+- **`dropbox`** (default) — read-only, offline refresh-token flow. Secrets:
+  `DROPBOX_APP_KEY`, `DROPBOX_APP_SECRET`, `DROPBOX_REFRESH_TOKEN`,
+  `DROPBOX_FOLDER_PATH` (optional).
+- **`local`** — a directory on the container filesystem. Set `LOCAL_SOURCE_PATH`
+  (or add `needs: { disk: true }` and it defaults to the injected `DISK_PATH`).
+- **`url`** — a manifest of document URLs. Set `URL_SOURCE_MANIFEST` to a
+  newline- or comma-separated list.
+
+**Add your own adapter:** subclass `Source` (implement `list_files()` returning
+`SourceFile[]` and `download(path) -> bytes`), gate its config in
+`config.py::source_configured`, and return it from `get_source()`. Text
+extraction for txt/md/pdf/docx is shared in `source.extract_text`.
+
+---
+
+## 5. Embeddings (one config point)
+
+`app/embeddings.py` is the single provider seam — OpenAI-compatible by default.
+Secrets/config (all gated, read lazily):
+
+- `EMBEDDINGS_API_KEY` — the key (semantic search + generated answers degrade
+  gracefully to lexical/extractive when unset).
+- `EMBEDDINGS_MODEL` (default `text-embedding-3-small`) — changing it changes
+  vector dimensions and requires a full reindex.
+- `EMBEDDINGS_BASE_URL` (default `https://api.openai.com/v1`) — point at any
+  OpenAI-compatible endpoint.
+- `ANSWER_MODEL` (default `gpt-4o-mini`) — the grounded-answer chat model.
+
+---
+
+## 6. Secrets via `grid secrets set`
+
+Store secrets with the CLI; never commit them, never read them at import time:
+
+```bash
+grid secrets set EMBEDDINGS_API_KEY=sk-...
+grid secrets set SOURCE_TYPE=dropbox
+grid secrets set DROPBOX_APP_KEY=... DROPBOX_APP_SECRET=... DROPBOX_REFRESH_TOKEN=...
+grid secrets set MANAGER_PASSWORD_HASH=$(printf '%s' 'your-password' | shasum -a 256 | cut -d' ' -f1)
+```
+
+`MANAGER_PASSWORD_HASH` is the sha256 hex of the manager password, compared
+server-side in `auth.py`. The in-app "Change password" screen returns a new hash
+to store the same way. Non-secret public config can also go via `grid env`.
+
+---
+
+## 7. The baked-in learnings (do NOT regress)
+
+1. **Startup index.** `main.py` has `@app.on_event("startup")` →
+   `db.ensure_indexes()`, which creates the chunks `$text` index (plus the
+   documents metadata indexes) at boot. Without it, `$text` raises
+   `IndexNotFound` (a 500) on an empty catalog. `ensure_indexes()` is safe to call
+   repeatedly and never blocks startup if Mongo is briefly unavailable.
+2. **`$text` guard.** The lexical query in `search.py` is wrapped in
+   `except OperationFailure` so a missing index degrades to "lexical contributes
+   nothing", never a 500 — belt-and-suspenders on top of the startup index.
+3. **Health without secrets.** No secret is read at import time; every secret is
+   gated inside a function (`config.py`). `GET /backend/health` needs only Mongo.
+   Search/indexing degrade gracefully (empty results + a clear note) when the
+   source or embeddings are unconfigured. `GET /backend/status` reports capability
+   **booleans** (`source_configured`, `embeddings_configured`, …), never secret
+   values.
+4. **No `needs: vector` / no active cron.** `needs:` is `database` only (#1545).
+   The cron refresh service is commented out (#1585) — see §8.
+
+---
+
+## 8. Refresh: manager endpoint now, cron later
+
+The supported refresh path today is the manager-only endpoint
+`POST /backend/admin/refresh` (wired to the "Refresh now" button in Settings). It
+runs the same incremental sync pipeline (`indexing.run_sync`) a cron would.
+
+A scheduled `type: cron` refresh service is a **follow-up**: a Python cron entity
+is currently blocked on **platform issue #1585**. The `cloudgrid.yaml` keeps a
+`refresh:` cron service commented out with a `services/refresh` source pointing at
+the same `indexing.run_sync`. When #1585 lands, uncomment it (`schedule`,
+`timezone`, `run: job`) and add a thin `services/refresh/src/main.py` that calls
+`app.indexing.run_sync(triggered_by="cron")`.
+
+---
+
+## 9. Deploy + edition note
+
+This is a built + deployed multi-service app → the **local edition** (Claude
+Desktop / Claude Code) or the CLI; the hosted edition cannot build a runtime
+container.
+
+1. `grid init semantic-search` — creates the entity + `.cloudgrid/link.json`.
+2. Put the code under `services/web/` and `services/backend/`; set `cloudgrid.yaml`
+   to the active fields (web static+build, backend python, `needs:{database:true}`).
+3. `grid secrets set …` for embeddings + source + `MANAGER_PASSWORD_HASH` (§6).
+4. (Optional) `grid dev` to run locally against injected dev Mongo.
+5. **Build the frontend first:** `(cd services/web && npm install && npm run build)`.
+   CloudGrid validates that a `type: static` service carrying a `build:` block
+   already has `services/web/dist/index.html` on disk at plug time — plug fails
+   validation otherwise.
+6. `grid plug` — a runtime deploy is **ASYNC**: the first response is
+   `status: building`, not a live URL. Poll `grid status` (surface a liveness
+   signal, never a bare wait) until Ready.
+7. Sign in as manager, hit "Refresh now" to index the source, then search.
+   Re-plug the same entity to update the same URL.

package/src/corpus/templates/semantic-search/README.md

@@ -0,0 +1,74 @@
+# semantic-search template — CloudGrid-native document search
+
+A deployable, domain-agnostic **document search** app: index a folder of
+documents (PDF / DOCX / TXT / MD), then search them by keyword, by meaning, and
+by metadata — with a grounded answer mode and a manager admin. Data lives in the
+grid-shared MongoDB, so the catalog survives refresh and is shared across users.
+
+**Hybrid search** = Mongo `$text` (lexical) + in-app NumPy cosine over chunk
+embeddings (semantic) + metadata filters (date / collection), blended and
+aggregated to the document level with a best-passage highlight.
+
+## Services
+
+| Service | Type | Path | Entry |
+|---|---|---|---|
+| `web` | `static` (React, Vite) | `/` | build → `dist` |
+| `backend` | `python` (FastAPI) | `/backend` | `services/backend/src/main.py` |
+
+`needs: { database: true }` → CloudGrid injects `DATABASE_MONGODB_URL` (fallback
+`MONGODB_URL`). **No `needs: vector`** — pgvector runtime role-grant is blocked
+(platform #1545), so embeddings live in the Mongo `chunks` collection and are
+cosine-ranked in-app. A scheduled (cron) refresh is a follow-up, blocked on
+platform #1585 — ship the manager "Refresh now" endpoint for now (see AGENTS.md).
+
+## Pluggable document source
+
+`SOURCE_TYPE` selects the adapter, all credentials from env secrets:
+
+- `dropbox` (default) — read-only Dropbox via the offline refresh-token flow.
+- `local` — a directory on the container filesystem (`LOCAL_SOURCE_PATH`, or a
+  `needs: disk` mount at `DISK_PATH`).
+- `url` — a manifest of document URLs (`URL_SOURCE_MANIFEST`).
+
+Add your own by subclassing `Source` in
+`services/backend/src/app/source.py` and returning it from `get_source()`.
+
+## Embeddings
+
+Behind one config point (`services/backend/src/app/embeddings.py`), OpenAI by
+default. `EMBEDDINGS_API_KEY` from env; `EMBEDDINGS_MODEL` / `EMBEDDINGS_BASE_URL`
+to point at any OpenAI-compatible provider.
+
+## Secrets (set with `grid secrets set KEY=VALUE`)
+
+`EMBEDDINGS_API_KEY`, `SOURCE_TYPE`, and the source's own secrets
+(`DROPBOX_APP_KEY` / `DROPBOX_APP_SECRET` / `DROPBOX_REFRESH_TOKEN` /
+`DROPBOX_FOLDER_PATH`, or `LOCAL_SOURCE_PATH`, or `URL_SOURCE_MANIFEST`), and
+`MANAGER_PASSWORD_HASH` (sha256 hex of the manager password).
+
+The app starts and passes `GET /backend/health` with **none** of these set —
+search / indexing degrade to a clear "not indexed / missing secret" state.
+
+## Health without secrets (baked-in)
+
+- `@app.on_event("startup")` calls `db.ensure_indexes()` so the chunks `$text`
+  index exists at boot — `$text` returns `[]` on an empty catalog, never a 500.
+- The `$text` query is additionally guarded in `except OperationFailure`.
+- No secret is read at import time; every secret is gated inside a function.
+- `/backend/status` reports capability booleans, never secret values.
+
+## Deploy (manager does this)
+
+```bash
+grid init semantic-search   # creates the entity + .cloudgrid/link.json
+grid secrets set EMBEDDINGS_API_KEY=sk-...   # + source secrets + MANAGER_PASSWORD_HASH
+
+# Build the static frontend BEFORE plug — CloudGrid validates that a
+# type:static service with a build: block already has services/web/dist/index.html.
+(cd services/web && npm install && npm run build)
+
+grid plug                    # async build + deploy; poll to a live URL
+```
+
+See `AGENTS.md` for the full wiring guide.