@cloudgrid-io/mcp 0.15.0 → 0.16.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cloudgrid-io/mcp",
-  "version": "0.15.0",
+  "version": "0.16.1",
   "description": "MCP server for CloudGrid. Three editions: a local stdio server (full toolset), a hosted web server (light, CLI-free toolset), and a public docs server (search_cloudgrid_documentation) over MCP Streamable HTTP.",
   "type": "module",
   "bin": {
@@ -39,7 +39,8 @@
     "smoke:docs": "node test/smoke-docs.mjs",
     "start:docs": "node src/docs.js",
     "snapshot:corpus": "node scripts/snapshot-corpus.mjs",
-    "build:mcpb": "node scripts/build-mcpb.mjs"
+    "build:mcpb": "node scripts/build-mcpb.mjs",
+    "test:widgets": "node test/apps-widgets.test.mjs"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.0.0",

package/src/corpus/capability-map.md

@@ -57,7 +57,7 @@ requires-vs-needs caveat, validation rules), fetch the companion reference:
 | revenue dashboard, sales dashboard, MRR/revenue view | `revenue-dashboard` | `database: true` | runtime (async, poll) | local |
 | property listings, real estate site, rentals/homes listings | `property-listings` | `database: true` | runtime (async, poll) | local |
 | project management, projects and tasks tracker, team project board | `project-management` | `database: true` | runtime (async, poll) | local |
-| search over my documents / PDFs / notes / knowledge base, semantic search, document search, find across my files, searchable archive | `semantic-search` | `database: true` (NO `vector` #1545; NO cron #1585) | runtime (async, poll) | local |
+| search over my documents / PDFs / notes / knowledge base, semantic search, document search, find across my files, searchable archive | `semantic-search` | `database: true` (NO `vector` #1545; active daily refresh cron) | runtime (async, poll) | local |
 
 **Rule of thumb:** if the app must SAVE/remember data, share state across
 users/sessions, log in, or store submissions → it is persistent → runtime, local
@@ -86,14 +86,11 @@ Python `type: cron` is blocked on **#1585**). Read its `AGENTS.md` for the sourc
 adapter (dropbox/local/url) + embeddings wiring and the health-without-secrets /
 startup-index patterns.
 
-## Held / pending platform (not yet buildable)
-
-Two more archetypes are designed but blocked on platform work — the LLM should
-know they are coming but must NOT author them yet:
+## Held / pending platform
 
 | Intent | Would need | Status |
 |---|---|---|
-| scheduled task, cron job, "run every day/hour", periodic worker | a `type: cron` service (`schedule`, `timezone`) | **HELD** — platform issue #1543 (cron entities crash the online-check). Do not build. |
+| scheduled task, cron job, "run every day/hour", periodic worker | a `type: cron` service (`schedule`, `timezone`) | **SUPPORTED** on CLI 0.14.0 — Python and Node `type: cron` services validate, deploy, and fire (platform #1585 fixed). See the semantic-search `refresh` cron for a working Python-job example. |
 | RAG chatbot, "answer over my own docs", semantic search, retrieval-augmented Q&A | `ai-app` + `needs: { vector: pgvector }` | **HELD** — platform issue #1545 (pgvector crashes the deploy). Ship the plain `ai-app` (no vector) for now. |
 
 ## Blueprints (structure + cloudgrid.yaml; adapt the app — some pending platform support)
@@ -121,17 +118,17 @@ runtime, async build, **local edition only**. Auth/payment secrets map through a
 | billing dashboard, invoicing and payments dashboard, SaaS billing | `billing-dashboard` | `database: true` (+ Stripe via `vault:`) | runtime (async, poll) | local |
 | internal tools portal, admin tools hub, back-office portal | `internal-tools-portal` | `database: true` (+ auth via `vault:`) | runtime (async, poll) | local |
 | approval workflow, request approvals, multi-step approval system | `approval-workflow` | `database: true` (+ auth via `vault:`) | runtime (async, poll) | local |
-| booking system, reservations, appointment booking, scheduling | `booking-system` | `database: true` (+ Stripe/auth via `vault:`); reminder **cron pending #1543** | runtime (async, poll) | local |
-| calendar scheduler, team calendar, scheduling app, meeting scheduler | `calendar-scheduler` | `database: true` (+ optional auth/Stripe via `vault:`); reminder **cron pending #1543** | runtime (async, poll) | local |
-| appointment booking, clinic/salon booking, appointment scheduler | `appointment-booking` | `database: true` (+ auth/Stripe via `vault:`); reminder **cron pending #1543** | runtime (async, poll) | local |
-| restaurant website with reservations, table booking, restaurant site | `restaurant-reservations` | `database: true` (+ Stripe/email via `vault:`); reminder **cron pending #1543** | runtime (async, poll) | local |
-| travel booking portal, trip booking, flights/hotels booking, travel reservations | `travel-booking` | `database: true` (+ Stripe/auth via `vault:`); reminder **cron pending #1543** | runtime (async, poll) | local |
+| booking system, reservations, appointment booking, scheduling | `booking-system` | `database: true` (+ Stripe/auth via `vault:`); reminder cron supported (Python/Node `type: cron`, 0.14.0) | runtime (async, poll) | local |
+| calendar scheduler, team calendar, scheduling app, meeting scheduler | `calendar-scheduler` | `database: true` (+ optional auth/Stripe via `vault:`); reminder cron supported (Python/Node `type: cron`, 0.14.0) | runtime (async, poll) | local |
+| appointment booking, clinic/salon booking, appointment scheduler | `appointment-booking` | `database: true` (+ auth/Stripe via `vault:`); reminder cron supported (Python/Node `type: cron`, 0.14.0) | runtime (async, poll) | local |
+| restaurant website with reservations, table booking, restaurant site | `restaurant-reservations` | `database: true` (+ Stripe/email via `vault:`); reminder cron supported (Python/Node `type: cron`, 0.14.0) | runtime (async, poll) | local |
+| travel booking portal, trip booking, flights/hotels booking, travel reservations | `travel-booking` | `database: true` (+ Stripe/auth via `vault:`); reminder cron supported (Python/Node `type: cron`, 0.14.0) | runtime (async, poll) | local |
 | RAG, ask my docs, knowledge base chatbot, retrieval-augmented search over documents | `ai-knowledge-base` | `ai: true, database: true`; ideal `vector: pgvector` **pending #1545** (store embeddings in Mongo + cosine-rank until it lands) | runtime (async, poll) | local |
 
 The booking family (`booking-system`, `calendar-scheduler`, `appointment-booking`,
-`restaurant-reservations`, `travel-booking`) is fully buildable today **minus**
-the reminder cron — a `type: cron` service is HELD on platform issue **#1543**, so
-ship without it. `ai-knowledge-base` builds today on the AI Gateway + Mongo; its
+`restaurant-reservations`, `travel-booking`) is fully buildable today, **including**
+the reminder cron — Python and Node `type: cron` services work on CLI 0.14.0
+(platform #1585 fixed). `ai-knowledge-base` builds today on the AI Gateway + Mongo; its
 ideal `needs: { vector: pgvector }` embedding store is HELD on **#1545**, so store
 chunks in Mongo and cosine-rank in-app until it lands.
 
@@ -173,8 +170,27 @@ engine hint. Cron is NOT a need — it is a **service type** (`type: cron` with
 
 1. Read the request; match it against the workflow `when:` triggers above.
 2. Adopt that template's `needs:`. Persistence → `database` (`app-with-data` or
-   `api-service`); talks to an LLM → `ai: true` + `database` (`ai-app`). (Held for
-   now: scheduled → a `cron` service, #1543; RAG → `vector: pgvector`, #1545.)
+   `api-service`); talks to an LLM → `ai: true` + `database` (`ai-app`); scheduled
+   work → a `type: cron` service (Python/Node, supported on 0.14.0). (Held for now:
+   RAG's ideal `vector: pgvector`, #1545 — store embeddings in Mongo until it lands.)
 3. Static (`needs: none`) → publish as an inspiration with `gridctl_drop`
    (instant, any edition). Anything with a `needs:` → runtime, local edition,
    `gridctl_plug` a linked folder, then poll to a live URL.
+
+## FAQ
+
+**"What database does CloudGrid support?"**
+> All of them — use CloudGrid's managed database out of the box, or bring your own
+> by giving me the connection keys (Postgres, MySQL, MongoDB, Supabase, Neon, and
+> more). Which would you like?
+
+Under the hood: **managed** is `needs: { database: true }` (grid provisions Mongo,
+injects `DATABASE_MONGODB_URL`); **bring-your-own** is
+`needs: { database: { tier: external, secret: MY_DB } }` plus
+`grid secrets set MY_DB=<connection-string>` (the connection string lives in env
+secrets, never committed).
+
+**"Can I run something on a schedule / every day?"**
+> Yes — add a `type: cron` service (`schedule` + `timezone`). Python and Node cron
+> jobs are supported on CLI 0.14.0. See the semantic-search `refresh` cron for a
+> working Python-job example.

package/src/corpus/install.md

@@ -80,6 +80,33 @@ Or the MCP server, in `~/.cursor/mcp.json`:
 
 Remote variant, nothing installed: `{ "mcpServers": { "cloudgrid": { "url": "https://mcp.cloudgrid.io/mcp" } } }`
 
+### Gemini CLI
+
+Add the MCP server with the built-in command:
+
+```
+gemini mcp add cloudgrid npx -y @cloudgrid-io/mcp
+```
+
+Or edit `~/.gemini/settings.json` (user) or `.gemini/settings.json` (project) directly:
+
+```json
+{ "mcpServers": { "cloudgrid": { "command": "npx", "args": ["-y", "@cloudgrid-io/mcp"] } } }
+```
+
+### Antigravity
+
+Google's Antigravity IDE/CLI shares one MCP config at `~/.gemini/config/mcp_config.json`
+(open it from the IDE: Settings, Customizations, Open MCP Config / Manage MCP Servers,
+View raw config). Add the local stdio server:
+
+```json
+{ "mcpServers": { "cloudgrid": { "command": "npx", "args": ["-y", "@cloudgrid-io/mcp"] } } }
+```
+
+Antigravity uses `serverUrl` (not `url`) for remote HTTP servers, so the remote variant is
+`{ "mcpServers": { "cloudgrid": { "serverUrl": "https://mcp.cloudgrid.io/mcp" } } }`.
+
 ### Manual
 
 ```

package/src/corpus/templates/semantic-search/AGENTS.md

@@ -3,7 +3,7 @@
 This template is a **runnable app**, not a blueprint — it ships real FastAPI +
 React code. Read this guide, adapt the naming/metadata to the user's domain, wire
 the secrets, and deploy. The four "baked-in learnings" (health-without-secrets,
-startup index, `$text` guard, no vector/no cron) are already in the code — do not
+startup index, `$text` guard, no `needs: vector`) are already in the code — do not
 regress them.
 
 ---
@@ -15,7 +15,7 @@ is the URL mount, NOT the filesystem path. Files at the repo root fail with
 `Error: Service directory not found: …/services/<name>`.
 
 ```
-cloudgrid.yaml                          # name + services(web,backend) + needs:{database:true}
+cloudgrid.yaml                          # name + services(web,backend,refresh) + needs:{database:true}
 services/web/                           # React + Vite static frontend, served at /
   package.json  vite.config.js  index.html
   src/main.jsx  App.jsx  api.js  styles.css
@@ -26,6 +26,11 @@ services/backend/                       # FastAPI, mounted at /backend
   src/app/
     main.py       config.py   db.py      search.py   indexing.py
     source.py     embeddings.py  answer.py  auth.py
+services/refresh/                       # type: cron, run: job — scheduled reindex (§8)
+  requirements.txt  README.md
+  src/main.py                           # calls app.indexing.run_sync(triggered_by="cron")
+  src/app/                              # VENDORED copies of the backend modules run_sync needs:
+    config.py  db.py  source.py  embeddings.py  indexing.py  __init__.py
 ```
 
 ---
@@ -150,23 +155,42 @@ to store the same way. Non-secret public config can also go via `grid env`.
    source or embeddings are unconfigured. `GET /backend/status` reports capability
    **booleans** (`source_configured`, `embeddings_configured`, …), never secret
    values.
-4. **No `needs: vector` / no active cron.** `needs:` is `database` only (#1545).
-   The cron refresh service is commented out (#1585) — see §8.
+4. **No `needs: vector`.** `needs:` is `database` only — pgvector's runtime
+   role-grant is still blocked (#1545). (The scheduled cron refresh IS active — see
+   §8.)
 
 ---
 
-## 8. Refresh: manager endpoint now, cron later
-
-The supported refresh path today is the manager-only endpoint
-`POST /backend/admin/refresh` (wired to the "Refresh now" button in Settings). It
-runs the same incremental sync pipeline (`indexing.run_sync`) a cron would.
-
-A scheduled `type: cron` refresh service is a **follow-up**: a Python cron entity
-is currently blocked on **platform issue #1585**. The `cloudgrid.yaml` keeps a
-`refresh:` cron service commented out with a `services/refresh` source pointing at
-the same `indexing.run_sync`. When #1585 lands, uncomment it (`schedule`,
-`timezone`, `run: job`) and add a thin `services/refresh/src/main.py` that calls
-`app.indexing.run_sync(triggered_by="cron")`.
+## 8. Refresh: on-demand endpoint + scheduled cron
+
+Two refresh paths, both running the same incremental pipeline
+(`indexing.run_sync`):
+
+1. **On-demand** — the manager-only endpoint `POST /backend/admin/refresh` (wired
+   to the "Refresh now" button in Settings). `triggered_by="manager"`.
+2. **Scheduled** — a `type: cron`, `run: job` Python service (`services/refresh/`)
+   that runs daily at `0 3 * * *` UTC and calls
+   `app.indexing.run_sync(triggered_by="cron")`. This is ACTIVE — Python `type:
+   cron` now validates, deploys, and fires on CLI 0.14.0 (platform #1585 fixed).
+
+**How the cron reuses the pipeline (and the drift note).** Each CloudGrid service
+builds in its own isolated container from its own folder, so a cross-folder import
+of `services/backend/src/app` will not resolve. The reuse mechanism is a
+**vendored copy**: the backend modules `run_sync` transitively needs —
+`config.py db.py source.py embeddings.py indexing.py __init__.py` — are copied
+verbatim into `services/refresh/src/app/`. The cron connects directly to the
+grid-injected `DATABASE_MONGODB_URL` (no HTTP hop to the backend, no self-URL
+dependency). If you change the pipeline, chunking, source adapters, embeddings, or
+the schema in the backend, **re-copy the changed module(s)** into
+`services/refresh/src/app/` so the scheduled run matches the endpoint. See
+`services/refresh/README.md`.
+
+**Secrets the cron needs:** the same ones as the backend, read lazily — `SOURCE_TYPE`
++ its source creds (`DROPBOX_*` / `LOCAL_SOURCE_PATH` / `URL_SOURCE_MANIFEST`) and
+`EMBEDDINGS_API_KEY` (+ `EMBEDDINGS_MODEL`, `EMBEDDINGS_BASE_URL`). With none set,
+the scheduled run is a clean no-op (`status: skipped`, 0 documents, exit 0), never a
+crash. Its outcome lands in the same `index_reports` collection, visible via the
+manager index-report and in `grid logs` as a `cron refresh: status=… files=…` line.
 
 ---
 

package/src/corpus/templates/semantic-search/README.md

@@ -19,8 +19,9 @@ aggregated to the document level with a best-passage highlight.
 `needs: { database: true }` → CloudGrid injects `DATABASE_MONGODB_URL` (fallback
 `MONGODB_URL`). **No `needs: vector`** — pgvector runtime role-grant is blocked
 (platform #1545), so embeddings live in the Mongo `chunks` collection and are
-cosine-ranked in-app. A scheduled (cron) refresh is a follow-up, blocked on
-platform #1585 — ship the manager "Refresh now" endpoint for now (see AGENTS.md).
+cosine-ranked in-app. Refresh runs two ways: the manager "Refresh now" endpoint
+(on demand) and an active Python `type: cron` job (`services/refresh/`, daily 03:00
+UTC) — both call the same `indexing.run_sync` (see AGENTS.md §8).
 
 ## Pluggable document source
 

package/src/corpus/templates/semantic-search/cloudgrid.yaml

@@ -191,16 +191,18 @@ services:
   backend:
     type: python
     path: /backend
-  # A scheduled `type: cron` refresh service is a follow-up — a Python cron
-  # entity is currently blocked on platform issue #1585. Until it lands, use the
-  # manager "Refresh now" endpoint (POST /backend/admin/refresh). See AGENTS.md.
-  # refresh:
-  #   type: cron
-  #   schedule: "0 3 * * *"
-  #   timezone: UTC
-  #   run: job
-  #   source:
-  #     path: services/refresh
+  # Scheduled reindex — Python `type: cron` job. Runs the SAME incremental
+  # pipeline as the manager "Refresh now" endpoint (app.indexing.run_sync),
+  # connecting directly to the injected DATABASE_MONGODB_URL. `run: job` = batch
+  # (no HTTP server, no path/port). Cron has no injected self-URL, so the backend
+  # `app/` package is vendored into services/refresh/src/app/ and called locally —
+  # see services/refresh/README.md (incl. the drift note). Degrades gracefully:
+  # with no source/embeddings secrets it records a report and exits 0 (no-op).
+  refresh:
+    type: cron
+    schedule: "0 3 * * *"       # daily at 03:00
+    timezone: UTC
+    run: job                    # batch job — runs services/refresh/src/main.py once
 # CloudGrid managed Mongo → injects DATABASE_MONGODB_URL (+ legacy MONGODB_URL).
 # NO `needs: vector` — pgvector runtime role-grant is still blocked (#1545);
 # embeddings live in the Mongo `chunks` collection and are cosine-ranked in-app.

package/src/corpus/templates/semantic-search/index.md

@@ -26,13 +26,15 @@ level with a best-passage highlight.
    `except OperationFailure` as belt-and-suspenders.
 5. **No `needs: vector`** — pgvector runtime role-grant is blocked (#1545), so
    embeddings live in the Mongo `chunks` collection (a float array) and are
-   cosine-ranked in-app. **No active cron** — a Python `type: cron` refresh is
-   blocked (#1585); ship the manager "Refresh now" endpoint and add cron later.
+   cosine-ranked in-app. **The refresh cron IS active** — a Python `type: cron`,
+   `run: job` service (`services/refresh/`) reindexes daily at 03:00 UTC, and the
+   manager "Refresh now" endpoint reindexes on demand; both call the same
+   `indexing.run_sync` (the cron vendors the backend `app/` modules — see AGENTS.md §8).
 
 ## File tree
 
 ```
-cloudgrid.yaml                                 # name + services(web static+build, backend python) + needs:{database:true}
+cloudgrid.yaml                                 # name + services(web static+build, backend python, refresh cron) + needs:{database:true}
 services/web/                                  # React + Vite static frontend (LTR, neutral)
   package.json  vite.config.js  index.html
   src/main.jsx  App.jsx  api.js  styles.css
@@ -50,6 +52,10 @@ services/backend/                              # FastAPI backend, mounted at /ba
     embeddings.py                               # one config point (OpenAI-compatible)
     answer.py                                   # grounded answer + citations (extractive fallback)
     auth.py                                     # manager session auth (MANAGER_PASSWORD_HASH)
+services/refresh/                              # type: cron, run: job — scheduled reindex (daily 03:00 UTC)
+  requirements.txt  README.md
+  src/main.py                                  # calls app.indexing.run_sync(triggered_by="cron")
+  src/app/                                     # VENDORED copies: config db source embeddings indexing __init__
 ```
 
 ## cloudgrid.yaml (active fields)
@@ -69,14 +75,19 @@ services:
   backend:
     type: python
     path: /backend
+  refresh:
+    type: cron
+    schedule: "0 3 * * *"
+    timezone: UTC
+    run: job
 needs:
   database: true
 ```
 
 > **Capability:** the need is `database: true` — the deployer provisions Mongo and
 > injects `DATABASE_MONGODB_URL` (+ legacy `MONGODB_URL`). NO `needs: vector`
-> (#1545) and NO active cron (#1585). `requires:` is the deprecated v1 alias —
-> don't mix it with `needs:`.
+> (#1545). Three services: `web` (static), `backend` (python), `refresh` (cron).
+> `requires:` is the deprecated v1 alias — don't mix it with `needs:`.
 
 ## Configure + deploy
 

package/src/corpus/templates/semantic-search/services/backend/src/app/indexing.py

@@ -1,7 +1,8 @@
 """Incremental indexing pipeline: discover -> extract -> chunk -> embed -> store.
 
-Shared by the manager 'Refresh now' endpoint (and, once platform issue #1585
-lands, a `type: cron` refresh service). It is resilient: if the source or
+Shared by the manager 'Refresh now' endpoint AND the scheduled `type: cron`
+refresh service (services/refresh, which vendors this module). It is resilient:
+if the source or
 embeddings are not configured it records a clear, non-crashing report and
 returns. Only new/changed files (by rev / content-hash) are reprocessed.
 """

package/src/corpus/templates/semantic-search/services/backend/src/app/main.py

@@ -205,8 +205,8 @@ def index_report():
 def refresh_now():
     """Manager-only on-demand incremental sync.
 
-    This is the supported refresh path today. A scheduled `type: cron` refresh
-    service is a follow-up (blocked on platform issue #1585); see AGENTS.md.
+    Runs the same pipeline as the scheduled `type: cron` refresh service
+    (services/refresh); this is the on-demand path. See AGENTS.md §8.
     """
     report = indexing.run_sync(triggered_by="manager")
     report.pop("_id", None)

package/src/corpus/templates/semantic-search/services/refresh/README.md

@@ -0,0 +1,42 @@
+# services/refresh — scheduled cron reindex
+
+A `type: cron`, `run: job` Python service that reruns the incremental indexing
+pipeline on a schedule (default daily `0 3 * * *`, UTC). It is the scheduled
+sibling of the manager-only `POST /backend/admin/refresh` endpoint — both call the
+exact same `app.indexing.run_sync`.
+
+## How it works
+
+- `src/main.py` is the batch entry point. CloudGrid runs it to completion once per
+  schedule and exits; there is no HTTP server. It calls
+  `app.indexing.run_sync(triggered_by="cron")`.
+- It connects DIRECTLY to the grid-injected `DATABASE_MONGODB_URL` (from the
+  app-level `needs: { database: true }`, inherited by every service including
+  cron). It does NOT call the backend over HTTP — there is no injected self-URL or
+  documented service-to-service DNS, so a network hop would be fragile.
+- Graceful no-op: with no source / embeddings secrets set, `run_sync` records a
+  report (`status: skipped`, `source_configured: false`, 0 documents) and returns
+  WITHOUT raising. The run exits 0. Only a genuine pipeline error exits non-zero.
+
+## Why the code is vendored (and the drift note)
+
+Each CloudGrid service builds in its OWN isolated container from its own folder, so
+a cross-folder import of `services/backend/src/app` does not resolve at build time.
+The reuse mechanism is therefore a **vendored copy**: the backend modules that
+`run_sync` transitively needs are copied verbatim into `services/refresh/src/app/`:
+
+    config.py  db.py  source.py  embeddings.py  indexing.py  __init__.py
+
+These are exact copies of `services/backend/src/app/*`. **Drift consideration:** if
+you change the indexing pipeline, chunking, source adapters, embeddings, or the DB
+schema in the backend, re-copy the changed module(s) here so the scheduled run
+matches the on-demand endpoint. Keep the two in lockstep. (`answer.py`, `search.py`,
+`auth.py`, `main.py` are NOT needed — the cron only indexes, it never serves or
+searches — so they are intentionally not vendored.)
+
+## Secrets it uses
+
+Same secrets as the backend, read lazily at run time (never at import):
+`SOURCE_TYPE` + its source creds (e.g. `DROPBOX_*` / `LOCAL_SOURCE_PATH` /
+`URL_SOURCE_MANIFEST`), and `EMBEDDINGS_API_KEY` (+ `EMBEDDINGS_MODEL`,
+`EMBEDDINGS_BASE_URL`). With none set, the run is a clean no-op.

package/src/corpus/templates/semantic-search/services/refresh/requirements.txt

@@ -0,0 +1,9 @@
+# Cron-job deps — the subset of the backend's requirements that the vendored
+# indexing pipeline (app/config,db,source,embeddings,indexing) actually needs.
+# No fastapi/uvicorn (this is a batch job, not a server) and no numpy (in-app
+# cosine ranking lives in the backend's search.py, which the cron does not run).
+# pypdf / python-docx are imported lazily by source.extract_text for pdf/docx.
+pymongo==4.8.0
+httpx==0.27.0
+pypdf==4.2.0
+python-docx==1.1.2

package/src/corpus/templates/semantic-search/services/refresh/src/app/config.py

@@ -0,0 +1,110 @@
+"""Central configuration and secret gating.
+
+CRITICAL CloudGrid rule: NEVER read a secret at module import time. Every value
+here is resolved lazily inside a function, so the process starts and passes
+`GET /backend/health` even when NO secret is set. Search and indexing degrade
+gracefully (empty results + a clear "not indexed / missing secret" note) rather
+than crashing.
+
+The Mongo connection is the ONLY hard dependency for health, and it is injected
+automatically by CloudGrid (`needs: database`) as DATABASE_MONGODB_URL.
+
+Two pluggable config points:
+  - SOURCE_TYPE  selects the document-source adapter (dropbox | local | url).
+  - EMBEDDINGS_* selects the embeddings provider (OpenAI-compatible by default).
+"""
+from __future__ import annotations
+
+import os
+
+
+# ── Mongo (injected by the grid — the only health-critical config) ──────────
+
+def mongo_url() -> str | None:
+    """Injected by CloudGrid `needs: database`. Never set by hand."""
+    return os.environ.get("DATABASE_MONGODB_URL") or os.environ.get("MONGODB_URL")
+
+
+# ── Embeddings (one config point, gated) ─────────────────────────────────────
+
+def embeddings_api_key() -> str | None:
+    return os.environ.get("EMBEDDINGS_API_KEY")
+
+
+def embeddings_model() -> str:
+    # Changing the model changes vector dimensions and requires a full reindex.
+    return os.environ.get("EMBEDDINGS_MODEL", "text-embedding-3-small")
+
+
+def embeddings_base_url() -> str:
+    # Any OpenAI-compatible endpoint. Default is OpenAI.
+    return os.environ.get("EMBEDDINGS_BASE_URL", "https://api.openai.com/v1")
+
+
+def embeddings_configured() -> bool:
+    return bool(embeddings_api_key())
+
+
+def answer_model() -> str:
+    return os.environ.get("ANSWER_MODEL", "gpt-4o-mini")
+
+
+# ── Source adapter (one config point, gated) ─────────────────────────────────
+
+def source_type() -> str:
+    """dropbox | local | url. Selects the document-source adapter."""
+    return os.environ.get("SOURCE_TYPE", "dropbox")
+
+
+def dropbox_config() -> dict[str, str | None]:
+    return {
+        "app_key": os.environ.get("DROPBOX_APP_KEY"),
+        "app_secret": os.environ.get("DROPBOX_APP_SECRET"),
+        "refresh_token": os.environ.get("DROPBOX_REFRESH_TOKEN"),
+        "folder_path": os.environ.get("DROPBOX_FOLDER_PATH", ""),
+    }
+
+
+def local_source_path() -> str:
+    """Directory the local adapter reads. Point at a `needs: disk` mount
+    (DISK_PATH) or a bundled folder."""
+    return os.environ.get("LOCAL_SOURCE_PATH", os.environ.get("DISK_PATH", ""))
+
+
+def url_source_manifest() -> str | None:
+    """A newline- or comma-separated list of document URLs the url adapter
+    fetches. Set as a secret or plain env var."""
+    return os.environ.get("URL_SOURCE_MANIFEST")
+
+
+def source_configured() -> bool:
+    st = source_type()
+    if st == "dropbox":
+        c = dropbox_config()
+        return bool(c["app_key"] and c["app_secret"] and c["refresh_token"])
+    if st == "local":
+        return bool(local_source_path())
+    if st == "url":
+        return bool(url_source_manifest())
+    return False
+
+
+# ── Manager auth (gated) ──────────────────────────────────────────────────────
+
+def manager_password_hash() -> str | None:
+    """A sha256 hex digest of the manager password. Compared server-side."""
+    return os.environ.get("MANAGER_PASSWORD_HASH")
+
+
+def manager_auth_configured() -> bool:
+    return bool(manager_password_hash())
+
+
+# ── Search blend weights ─────────────────────────────────────────────────────
+
+def semantic_weight() -> float:
+    return float(os.environ.get("SEMANTIC_WEIGHT", "0.55"))
+
+
+def keyword_weight() -> float:
+    return float(os.environ.get("KEYWORD_WEIGHT", "0.45"))

package/src/corpus/templates/semantic-search/services/refresh/src/app/db.py

@@ -0,0 +1,89 @@
+"""Lazy MongoDB access.
+
+The client and connection string are resolved LAZILY (inside get_client), never
+at module top level — so importing this module never touches the network and
+never requires the env var to exist. CloudGrid injects DATABASE_MONGODB_URL at
+runtime from `needs: database`.
+
+Collections:
+  documents  — one row per source document (title, collection tag, date, ...).
+  chunks     — one embeddable passage per document (content + embedding array).
+  settings   — feature flags, synonyms, catalog metadata (last_refresh).
+  usage_events, index_reports, sessions — telemetry / admin.
+"""
+from __future__ import annotations
+
+from typing import Optional
+
+from pymongo import ASCENDING, MongoClient, TEXT
+from pymongo.database import Database
+
+from . import config
+
+_client: Optional[MongoClient] = None
+
+
+def get_client() -> MongoClient:
+    global _client
+    if _client is None:
+        uri = config.mongo_url()
+        if not uri:
+            raise RuntimeError(
+                "DATABASE_MONGODB_URL is not set. CloudGrid injects it "
+                "automatically from `needs: database` — run with `grid dev` "
+                "locally or deploy with `grid plug`. Do not set it by hand."
+            )
+        # Short server-selection timeout so health checks fail fast, not hang.
+        _client = MongoClient(uri, serverSelectionTimeoutMS=3000)
+    return _client
+
+
+def get_db() -> Database:
+    # Default DB name comes from the connection-string path segment the grid injects.
+    return get_client().get_default_database()
+
+
+def ping() -> bool:
+    """True if Mongo is reachable. Used by /backend/health."""
+    get_client().admin.command("ping")
+    return True
+
+
+def ensure_indexes() -> None:
+    """Create the indexes the search relies on. Safe to call repeatedly.
+
+    Called at FastAPI startup so the chunks $text index exists before any query
+    — without it, `$text` raises IndexNotFound (a 500) on an empty catalog.
+    """
+    db = get_db()
+    # Text index on chunk content for lexical / $text search.
+    db.chunks.create_index([("content", TEXT)], name="chunks_content_text")
+    db.chunks.create_index([("document_id", ASCENDING)], name="chunks_document_id")
+    # Metadata filter indexes on documents.
+    db.documents.create_index([("date", ASCENDING)], name="documents_date")
+    db.documents.create_index([("collection", ASCENDING)], name="documents_collection")
+    db.documents.create_index(
+        [("source_filename", ASCENDING)], name="documents_source_filename", unique=True
+    )
+
+
+# ── Collection accessors ─────────────────────────────────────────────────────
+
+def documents():
+    return get_db().documents
+
+
+def chunks():
+    return get_db().chunks
+
+
+def settings():
+    return get_db().settings
+
+
+def events():
+    return get_db().usage_events
+
+
+def index_reports():
+    return get_db().index_reports