@leadbay/mcp 0.4.0 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,103 @@
 # Changelog — @leadbay/mcp
 
+## 0.6.0 — UNRELEASED
+
+Massive spec-coverage upgrade. Each tool declares **annotations** (`readOnlyHint` / `destructiveHint` / `idempotentHint` / `openWorldHint`); 17 composites + 12 highest-leverage granulars declare typed `outputSchema` + emit `structuredContent`. New surfaces: `prompts/*` (5 canned slash-commands), `resources/*` (`lead://`, `lens://`, `org://taste-profile`), `notifications/progress` (per-lead streaming on every long-running composite), `notifications/cancelled` → `ToolContext.signal` (bulk-store entries marked `cancelled` so subsequent status polls return `BULK_CANCELLED`), `elicitation/create` (refine_prompt clarification + report_outreach user_confirmed anti-poisoning).
+
+**Hardening**: every `inputSchema` declares `additionalProperties: false`. Runtime conformance test pins `structuredContent` against `outputSchema` for every declarer (drift-catcher). Static-scan audit of every error-hint string asserts each names a recovery action. `report_outreach.verification` rejects extra keys at runtime (closes the SDK's nested-additionalProperties limitation). `score_0_to_10` deprecated alias of `boost_score` removed in 0.7.0.
+
+**Token economy**: pagination payloads carry `has_more` + `next_page`. `research_lead` truncates large payloads with a `truncation_hint`. Per-tool opt-in `response_format: "json" | "markdown"` lets chat-rendering agents pick the cheaper render (research_lead first; pattern reusable). `LEADBAY_DEBUG=1` enables a per-tools/call observability line on stderr.
+
+**DXT → MCPB**: bundle now publishes both `leadbay-X.Y.Z.dxt` (legacy) and `leadbay-X.Y.Z.mcpb` (new Claude Desktop format) for one cycle. Manifest `dxt_version` field stays for backwards-compat with current installers; field rename will follow Anthropic's spec when finalised. The two filenames have identical content; downstream installers can match either glob.
+
+**Versioning + docs**: 0.4.0 / 0.6.0 bumps; README §3a "Spec primitives in action" adds wire-level JSON-RPC transcripts for every primitive; MIGRATION.md 0.5 → 0.6 walkthrough.
+
+## 0.5.0 — 2026-05-04
+
+### `leadbay_import_and_qualify` — new composite
+
+End-to-end import + AI qualification in one call. Wraps `leadbay_import_leads` (chunking, mapping preflight, custom-field validation), then fans out `web_fetch` on every imported leadId, polls until each lead's qualification answers populate, and returns the results. When the wall-clock budget overflows, returns a `qualify_id` UUID handle for resumable retrieval via the new `leadbay_qualify_status` tool.
+
+**Inputs** (mostly mirrors `leadbay_import_leads`):
+
+- `domains` OR `records` — same shape as import_leads.
+- `mappings.fields` and `mappings.custom_fields` — same as 0.3.0 import_leads. Custom fields surface as first-class via `leadbay_list_mappable_fields` (also new).
+- `dry_run: "preview"` — special mode: uploads the CSV in dry-run and returns the wizard's per-column AI mapping hints + sample rows + custom-field candidates from the org catalog matched against unmapped column names by exact / case-insensitive / fuzzy-substring. NO ai_rescore quota consumed.
+- `total_budget_ms` (default 900_000 = 15 min), `per_lead_budget_ms` (default 90_000), `per_phase_budget_ms` (default 300_000).
+- `skip_already_qualified` (default `true`) — skips `web_fetch` launch on leads with a non-null `ai_agent_lead_score`. Saves quota.
+
+**Outputs** (full mode):
+
+```
+{
+  kind: "result",
+  qualify_id: "<UUIDv4>" | null,
+  import_ids: [...],
+  imported: [{ leadId, domain?, name, rowId? }],
+  not_imported: [...],
+  qualified: [{ lead_id, qualifications: [{ question, score, response, computed_at }], qualification_summary, signals_count, ... }],
+  still_running: [{ lead_id }],
+  failed: [...],
+  quota_exceeded: bool,
+  skipped_already_qualified: [...],
+  reused?: bool, seconds_since_original?: number,
+  cancelled?: bool, budget_exhausted?: bool,
+  region, _meta
+}
+```
+
+`qualify_id` is persisted to `~/.leadbay/bulks.json` (30-day TTL, 5-min idempotency window) — same store as `leadbay_enrich_titles` but with a `kind: "qualify"` discriminator. Re-calling with the same records+mapping within 5 min returns the same handle (`reused: true`).
+
+### `leadbay_qualify_status` — new resumable retrieval
+
+```
+leadbay_qualify_status({ qualify_id })
+  → { qualify_id, status, lead_ids, qualified, still_running, ... }
+```
+
+Refreshes the per-lead state (`/web_fetch` + `/ai_agent_responses`) at call time. No backend mutation. Survives MCP restart.
+
+### `leadbay_list_mappable_fields` — new discovery
+
+```
+leadbay_list_mappable_fields()
+  → { standard_fields: [{name, description, mapping_value}], custom_fields: [{id, name, type, description, mapping_value}], region, _meta }
+```
+
+Lists every CRM field the agent can target in `mappings.fields`. Standard fields come from a static catalog with human descriptions; custom fields come from `GET /crm/custom_fields`. The `mapping_value` field is what the agent passes verbatim (e.g., `"CUSTOM.8"`).
+
+### `leadbay_import_leads` 0.3.0 — `mappings.custom_fields` shorthand
+
+In 0.2.0 the only way to map a custom field was `mappings.fields[col] = "CUSTOM.<id>"` (raw wire format). 0.3.0 adds `mappings.custom_fields[col] = <id>` (numeric) or `<name>` (string), resolved against `/crm/custom_fields` before the wizard sees it. New error codes: `IMPORT_CUSTOM_FIELD_UNKNOWN`, `IMPORT_INVALID_CUSTOM_MAPPING`, `IMPORT_CUSTOM_FIELD_NAME_AMBIGUOUS`, `IMPORT_CUSTOM_FIELD_CATALOG_REQUIRED`, `IMPORT_MAPPING_DUPLICATE_CUSTOM`. Catalog GET is suppressed when the mapping references no custom fields (saves a round trip).
+
+### Bulk store schema widening
+
+`BulkRecord` now has a `kind: "enrich" | "qualify"` discriminator. Old enrich rows (no `kind` field) default to `"enrich"` on read. Existing `leadbay_bulk_enrich_status` callers see no change. Cross-kind id queries are surfaced with the new `BULK_WRONG_KIND` error code that points the caller at the right tool.
+
+### `not_in_lens` partition — terminate the silent infinite-poll
+
+Both `leadbay_import_and_qualify` and `leadbay_qualify_status` now surface a `not_in_lens: string[]` array — lead ids that exist in the org (the wizard imported them) but are NOT admitted to the active lens. The backend's `queueAiRescoreForLead` is a no-op for these leads — they will never appear in `qualified[]`. Surfacing them in a distinct partition means the agent's poll loop terminates.
+
+Discovered by iter-17 live e2e: imported 4 leads (Apple, Stripe, Datadog, GitHub) into a lens whose scoring rules only admitted Datadog. Datadog got `ai_agent_lead_score: -13` + 3/3 qualifications cleanly; Apple and Stripe sat in `still_running[]` indefinitely with no signal to stop polling. Now they land in `not_in_lens` with an actionable agent prompt: "either change the active lens or accept the lead won't be qualified."
+
+`qualify_status` re-checks lens membership at each call — a lead added to the lens after the original `import_and_qualify` automatically migrates from `not_in_lens` back into the regular qualify pipeline on the next status call.
+
+### Aligned with backend `/imports/{id}/leads` (PR #1801)
+
+`leadbay_import_and_qualify` now sources the qualify-phase lead set from `GET /1.5/imports/{importId}/leads` (added backend-side 2026-05-06). This is the spec-prescribed source of truth — distinct lead ids the import touched (matched-existing AND newly-created). Replaces the per-record reconciliation pagination for the qualify input. Falls back gracefully to the per-record set when the endpoint is unavailable (older backend / 400 in_progress race).
+
+Verified live: import 970bd47a-… (apple.com matched, salesforce.com uncrawled) → /leads returned `{lead_ids: [0a788a89-...]}` cleanly; qualify_id 08ad4555-… retrieved end-to-end.
+
+### Monitor-membership disclosure
+
+Both `leadbay_import_leads` and `leadbay_import_and_qualify` now flag in their tool descriptions that imported leads are NOT auto-promoted to the user's Monitor tab. Lens-scoring rules decide — only above-threshold leads get `in_monitor: true`. This was a real surprise discovered in production (journal entry `leadbay-monitor-lens-filter`, captured 2026-05-05). Surfacing it in the description prevents the agent from telling users "I imported your leads, check Monitor" — answer is the CRM-imports list, not Monitor.
+
+### Stable qualification ordering + human_summary
+
+Both `leadbay_import_and_qualify` and `leadbay_qualify_status` now sort `qualifications[]` by the org's `ai_agent_questions` catalog order — the same question appears at the same index across calls so LLM agents can position-index reliably. Catalog comes from `client.resolveTasteProfile()` (cached 10min). Falls back to alphabetical when the catalog is empty.
+
+Each `qualified[]` entry also carries an optional `human_summary` string of the form `answered X/Y — <signal> on '<question>'[, <signal> on '<question>']` where `<signal>` is `strong positive` (score=20), `positive` (10), `neutral` (0), or `negative` (-10). Top-2 by absolute score. Saves the agent from reading every per-question response when it just needs the gist.
+
 ## 0.4.0 — 2026-05-04
 
 ### `leadbay_import_leads` 0.2.0 — custom field mapping

package/MIGRATION.md

@@ -1,3 +1,99 @@
+# Migration: leadbay-mcp 0.5.x → 0.6.0 (UNRELEASED)
+
+The "MCP best-practice" upgrade. Five behaviour additions and ONE
+softer-than-it-looks behavior callout. All changes are MCP-spec
+aligned (2025-11-25).
+
+## 1. Tool annotations on every tool
+
+Every tool now declares `annotations: { readOnlyHint, destructiveHint,
+idempotentHint, openWorldHint, title }` in the `tools/list` payload.
+Capable clients (Claude Desktop, Cursor) use these hints to decide
+whether to auto-approve a call or prompt for confirmation. No agent-
+side change required.
+
+## 2. `additionalProperties: false` on every tool's inputSchema
+
+**Behavior callout**: any client that was passing extra unrecognized
+fields will now get a schema rejection. This was a deliberate
+hardening — extra fields were never documented, and the rejection
+closes a prompt-injection surface (the eval doc explicitly endorsed
+this). To migrate: drop the unknown field. The error message names
+the rejected field.
+
+If you discover an unknown field in your call shape, double-check the
+relevant tool's documented `inputSchema`; if you believe a field
+should exist, file an issue.
+
+## 3. `outputSchema` + `structuredContent` on top-5 composites
+
+`leadbay_pull_leads`, `leadbay_research_lead`, `leadbay_account_status`,
+`leadbay_bulk_qualify_leads`, `leadbay_report_outreach` now declare
+`outputSchema` AND emit `structuredContent` on success alongside the
+existing text content. Clients that read structuredContent get the
+typed payload without re-parsing through the LLM. Backwards-compat:
+text content is unchanged.
+
+## 4. `boost_score` on `research_lead.qualification[]`
+
+The qualification entry was previously labelled `score_0_to_10`. The
+actual scale is the discrete `-10|0|10|20` boost (per
+`AiAgentResponse.score`), NOT a 0-10 average. 0.6.0 introduces:
+
+```json
+{
+  "question": "...",
+  "boost_score": 10,
+  "score_scale": "-10|0|10|20",
+  "score_0_to_10": 10,    // DEPRECATED — same value as boost_score
+  "response": "...",
+  "computed_at": "..."
+}
+```
+
+`score_0_to_10` is kept as a deprecated alias for one minor version
+and removed in 0.7.0. Switch to `boost_score` now.
+
+## 5. New spec primitives
+
+The server now advertises and serves:
+
+- **`prompts/*`** — 5 canned slash-commands (`leadbay_daily_check_in`,
+  `leadbay_research_a_domain`, `leadbay_refine_audience`,
+  `leadbay_log_outreach`, `leadbay_qualify_top_n`). Use these from
+  the client UI; agents that read prompts compose workflows
+  automatically.
+- **`resources/*`** — three URI schemes addressable by client:
+  `lead://{uuid}/profile`, `lens://{id}/definition`,
+  `org://taste-profile`. Cache-friendly for capable clients.
+- **`notifications/progress`** — long-running composites (today:
+  `leadbay_bulk_qualify_leads`) stream per-lead progress updates
+  when the client opts in via `_meta.progressToken`.
+- **`notifications/cancelled`** → `ToolContext.signal` — clients
+  that send cancellation now actually stop in-flight work in
+  composites that observe `ctx.signal` (`leadbay_import_and_qualify`,
+  `leadbay_import_leads`, etc.).
+
+Clients without these capabilities negotiate away — backwards-compat
+is preserved.
+
+## 6. Pagination metadata: `has_more` + `next_page`
+
+`leadbay_pull_leads` and `leadbay_discover_leads` payloads now
+include `has_more: boolean` and `next_page: number | null` next to
+the existing `pagination` object. Spec-aligned; agents no longer
+need to compute `page < pages - 1`.
+
+## 7. Truncation steering on `research_lead`
+
+When the response would exceed ~25k characters, the composite now
+emits `truncated: true` plus `truncation_hint: "Pass concise:true to
+filter to hot signals only."`. Load-bearing fields (qualification,
+firmographics, contacts) are never trimmed; only signals.entries get
+trimmed first.
+
+---
+
 # Migration: leadbay-mcp 0.2.x → 0.3.0
 
 This release fixes [product#3504](https://github.com/leadbay/product/issues/3504): the default-installed MCP server's system prompt told the agent to call tools that the server didn't actually expose. Three behavior changes you need to know about.
@@ -210,3 +306,143 @@ Default is read-only (exposeWrite=false, exposeGranular=false).
   to detect when `enrichment.done` flips.
 - A `DELETE /lenses/{draftId}` endpoint — not testable in our tenant; treated
   as best-effort with `orphan_draft_id` surfaced on cleanup failure.
+
+---
+
+# Migration: leadbay-mcp 0.4.x → 0.5.0
+
+The 0.5.0 release adds three new tools and extends `leadbay_import_leads` to support custom fields. This is purely additive — existing 0.4.x callers see no behavior change.
+
+## What's new
+
+| Tool | Purpose |
+|---|---|
+| `leadbay_list_mappable_fields` | Discovery: lists every CRM field (standard + this org's custom fields) the agent can target in `mappings.fields`. |
+| `leadbay_import_and_qualify` | The mission verb: imports leads + triggers AI qualification + returns per-question answers in one call. Resumable via `qualify_id`. |
+| `leadbay_qualify_status` | Retrieves a previously-launched qualification by `qualify_id` (handle persists 30d). |
+| `leadbay_import_leads` 0.3.0 | Adds `mappings.custom_fields` ergonomic shorthand alongside the raw `mappings.fields[col] = "CUSTOM.<id>"` wire format. |
+
+## Worked example: discover → import → qualify
+
+```jsonc
+// 1. Discover what fields are mappable on this org.
+leadbay_list_mappable_fields()
+// → {
+//   "standard_fields": [{name:"LEAD_NAME", description:"...", mapping_value:"LEAD_NAME"}, ...],
+//   "custom_fields":   [{id:"8", name:"priority_test", type:"TEXT", mapping_value:"CUSTOM.8"}, ...],
+//   "_meta": {region: "us", endpoint: "GET /crm/custom_fields", latency_ms: 78}
+// }
+
+// 2. (Optional) preview the wizard's mapping suggestions for a sample.
+leadbay_import_and_qualify({
+  records: [{Brand: "Apple", Site: "apple.com", Priority: "high"}],
+  dry_run: "preview"
+})
+// → {
+//   kind: "preview",
+//   mapping_hints: [{column: "Site", suggested_field: "LEAD_WEBSITE", ai_confidence: 95}],
+//   custom_field_candidates: [{column: "Priority", candidates: [{id:"8", mapping_value:"CUSTOM.8", reason:"fuzzy_substring_match"}]}],
+//   sample_rows: [{Brand: "Apple", Site: "apple.com", Priority: "high"}],
+//   import_id: "<uuid>"
+// }
+
+// 3. Run the full flow with the chosen mapping.
+leadbay_import_and_qualify({
+  records: [{Brand: "Apple", Site: "apple.com", Priority: "high"}],
+  mappings: {
+    fields: {Brand: "LEAD_NAME", Site: "LEAD_WEBSITE"},
+    custom_fields: {Priority: "priority_test"}   // resolved against /crm/custom_fields
+  }
+})
+// → {
+//   kind: "result",
+//   qualify_id: "<uuidv4>",                   // resumable handle
+//   import_ids: ["<uuid>"],
+//   imported: [{leadId: "...", domain: "apple.com", name: "APPLE", rowId: "..."}],
+//   not_imported: [],
+//   qualified: [{
+//     lead_id: "...",
+//     qualifications: [{question: "Is the company...", score: 20, response: "yes...", computed_at: "..."}, ...],
+//     qualification_summary: {answered: 3, total: 3, avg_qualification_boost: 13.3},
+//     human_summary: "answered 3/3 — strong positive on 'Is the company...', positive on '...'",
+//     signals_count: 12
+//   }],
+//   still_running: [],
+//   chosen_budgets: {strategy: "small", total_budget_ms: 180000, wall_clock_estimate_ms: 60000, ...}
+// }
+
+// 4. If still_running was non-empty, retrieve later via qualify_id.
+leadbay_qualify_status({qualify_id: "<uuid>"})
+// → same shape, refreshed against backend at call time. failed[] populated
+//   for any leads that 404'd between launch and status-check.
+```
+
+## Custom fields wire format
+
+The backend serializer accepts `"CUSTOM.<id>"` as a value in the `mappings.fields` map. 0.3.0 of `leadbay_import_leads` accepts both:
+
+```jsonc
+// raw wire form (passes through unchanged):
+mappings.fields = {col: "CUSTOM.8"}
+
+// ergonomic shorthand (resolved against /crm/custom_fields):
+mappings.custom_fields = {col: 8}             // numeric id
+mappings.custom_fields = {col: "8"}           // string-shaped id
+mappings.custom_fields = {col: "priority_test"}  // exact name (case-insensitive fallback)
+```
+
+New error codes (all surfaced with hint pointing at next action):
+`IMPORT_CUSTOM_FIELD_UNKNOWN`, `IMPORT_INVALID_CUSTOM_MAPPING`, `IMPORT_CUSTOM_FIELD_NAME_AMBIGUOUS`, `IMPORT_CUSTOM_FIELD_CATALOG_REQUIRED`, `IMPORT_MAPPING_DUPLICATE_CUSTOM`, `IMPORT_MAPPING_CONFLICT_TARGET`.
+
+## Bulk store schema widening
+
+`~/.leadbay/bulks.json` now has a `kind: "enrich" | "qualify"` discriminator. Old enrich rows (no `kind` field) are read with `kind: "enrich"` defaulted in. Existing `leadbay_bulk_enrich_status` callers see no change. Querying an enrich `bulk_id` via `leadbay_qualify_status` (or vice versa) returns the new `BULK_WRONG_KIND` error pointing at the right tool.
+
+## Adaptive budgets
+
+When `leadbay_import_and_qualify` is called with no `total_budget_ms` or `per_lead_budget_ms`, the composite picks a strategy from input size:
+
+| Input size | Strategy | total_budget_ms | per_lead_budget_ms |
+|---|---|---|---|
+| ≤ 5 leads | small | 3 min | 60s |
+| 6 – 20 leads | default | 10 min | 90s |
+| > 20 leads | large | 25 min | 120s |
+
+The chosen values appear on the response as `chosen_budgets: {strategy, total_budget_ms, per_lead_budget_ms, wall_clock_estimate_ms}` so the agent can communicate the wall-clock to the human user.
+
+## See also
+
+- [CHANGELOG.md 0.5.0 entry](./CHANGELOG.md) — full release notes.
+
+## Error-code reference (0.5.0)
+
+Quick-reference for new error codes shipped in 0.5.0:
+
+| Code | When | Next action |
+|---|---|---|
+| `IMPORT_CUSTOM_FIELD_UNKNOWN` | Caller passed `CUSTOM.<id>` or `mappings.custom_fields[col]` that doesn't exist on this org. | Call `leadbay_list_mappable_fields()`; pick a real id/name from `custom_fields[]`. |
+| `IMPORT_INVALID_CUSTOM_MAPPING` | Caller passed a mapping value that's neither a StandardCrmFieldType nor a well-formed `CUSTOM.<digits>`. | Use a value from `leadbay_list_mappable_fields()` `mapping_value` field. |
+| `IMPORT_CUSTOM_FIELD_NAME_AMBIGUOUS` | Two custom fields share the name (case-insensitive) the caller used. | Pass the numeric id instead. |
+| `IMPORT_MAPPING_DUPLICATE_CUSTOM` | Same column appears in both `mappings.fields` and `mappings.custom_fields`. | Pick one of the two maps; remove the duplicate. |
+| `IMPORT_MAPPING_CONFLICT_TARGET` | Two columns map to the same StandardCrmFieldType (e.g. both → `LEAD_NAME`). | Pick the column that contains the real value; drop the others from the mapping. |
+| `IMPORT_CUSTOM_FIELD_CATALOG_REQUIRED` | Internal: catalog couldn't be fetched. | Retry; or use raw `CUSTOM.<id>` in `mappings.fields` instead of shorthand. |
+| `BULK_TRACKER_UNAVAILABLE` | MCP server has no BulkTracker (qualify_id persistence). | Restart with `LEADBAY_BULK_STORE_ALLOW_MEMORY=1` or upgrade. |
+| `BULK_INVALID_ID` | `qualify_id` is not a valid UUIDv4. | Pass the id returned by the prior `leadbay_import_and_qualify` call verbatim. |
+| `BULK_NOT_FOUND` | Handle expired (>30d TTL) or never existed. | Re-launch via `leadbay_import_and_qualify`. |
+| `BULK_PENDING` | Launch in flight or crashed mid-launch. | Retry in a few seconds; if persists >60s, relaunch. |
+| `BULK_LAUNCH_FAILED` | The original launch failed permanently. | Re-launch. |
+| `BULK_WRONG_KIND` | Caller passed an enrich `bulk_id` to `leadbay_qualify_status` (or vice versa). | Switch tools — enrich → `leadbay_bulk_enrich_status`, qualify → `leadbay_qualify_status`. |
+| `IMPORT_PREVIEW_NO_UPLOAD` | Preview mode hit all-malformed input; nothing was uploaded. | Check that at least one record/domain is well-formed. |
+
+## Per-tool prereqs (0.5.0)
+
+Quick scan of what each new tool requires before invocation:
+
+| Tool | Admin role | Active billing | LEADBAY_MCP_WRITE |
+|---|---|---|---|
+| `leadbay_list_mappable_fields` | no | no | no (read-only) |
+| `leadbay_qualify_status` | no | no | no (read-only) |
+| `leadbay_import_and_qualify` | yes | yes | yes (`=1`, default ON in 0.3.0+) |
+| `leadbay_import_leads` (0.3.0) | yes | yes | yes |
+
+Pre-flight: call `leadbay_account_status` to read `user.admin` and `organization.plan`. Both writes will return typed errors (`IMPORT_ADMIN_REQUIRED` / `IMPORT_BILLING_REQUIRED` / `FORBIDDEN`) at the call site too — pre-flighting just gives the agent a chance to ask the user politely instead of attempting a write that 403s.