@doist/todoist-mcp 12.5.2 → 12.5.3

package/dist/mcp-server.d.ts

@@ -1,6 +1,6 @@
 import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
 import { FEATURE_NAMES, Feature, FeatureName, Features } from './mcp-helpers.js';
-export declare const instructions = "\n## Todoist Task and Project Management Tools\n\nYou have access to comprehensive Todoist management tools for personal productivity and team collaboration. Use these tools to help users manage tasks, projects, sections, comments, and assignments effectively.\n\n### Core Capabilities:\n- Create, update, complete, and search tasks with rich metadata (priorities, due dates, durations, assignments)\n- Manage projects and sections with flexible organization\n- Handle comments and collaboration features\n- Bulk assignment operations for team workflows\n- Get overviews and insights about workload and progress\n\n### Tool Usage Guidelines:\n\n**Task Management:**\n- **add-tasks**: Create tasks (max 25 per call) with content, description, priority (`p1`, `p2`, `p3`, `p4` strings only; `p1` highest and `p4` lowest/default; integers are not accepted), dueString (natural language like \"tomorrow\", \"next Friday\", \"2024-12-25\"; also use natural language for recurrences and do not prefix them with `recurring`), deadlineDate (ISO 8601 format like \"2025-12-31\" for immovable constraints), duration (formats like \"2h\", \"90m\", \"2h30m\"), and assignments to project collaborators\n- **update-tasks**: Modify existing tasks - get task IDs from search results first, only include fields that need changes. Supports priority updates using `p1`/`p2`/`p3`/`p4` string values (`p1` highest, `p4` lowest/default; integers are not accepted), due date updates via dueString and due date removal via \"dueString: remove\", plus deadlineDate (ISO 8601 format like \"2025-12-31\") updates and removals via \"deadlineDate: remove\". **IMPORTANT: Do NOT use update-tasks to reschedule/move task dates \u2014 use reschedule-tasks instead.** update-tasks replaces the entire due string which destroys recurrence patterns on recurring tasks. Never echo back a task's existing projectId/sectionId/parentId \u2014 those fields are treated as a move.\n- **reschedule-tasks**: **Always use this tool when moving/rescheduling task due dates to a different date.** This tool preserves recurring schedules and existing time-of-day. Accepts YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS. Works for both recurring and non-recurring tasks. Do NOT use update-tasks for rescheduling.\n- **complete-tasks**: Mark tasks as done using task IDs\n- **uncomplete-tasks**: Reopen completed tasks using task IDs\n- **find-tasks**: Search by text, project/section/parent container, responsible user, labels, a raw Todoist `filter` string (e.g. \"today\", \"p1\", \"##Work\", \"(today | overdue) & p1\"), or a saved filter by ID or name (`filterIdOrName`). Requires at least one search parameter. `filter`/`filterIdOrName` cannot be combined with projectId/sectionId/parentId, and `filter` and `filterIdOrName` are mutually exclusive.\n- **find-tasks-by-date**: Get tasks by date range (startDate: YYYY-MM-DD or 'today' which includes overdue tasks) or specific day counts\n- **find-completed-tasks**: View completed tasks by completion date or original due date; if since/until are omitted, defaults to the last 7 days (returns all collaborators unless filtered). For a history of actual task-completion events, including recurring task occurrences, use find-activity instead.\n\n**Project & Organization:**\n- **add-projects/update-projects/find-projects**: Manage project lifecycle with names, descriptions (Markdown), favorites, view styles (list/board/calendar), and workspace assignment for new projects (by name or ID). find-projects returns active projects by default; pass archivedStatus ('archived' or 'all') to include archived projects. Every returned project includes an isArchived field\n- **project-management**: Archive or unarchive projects by ID\n- To delete a project (active or archived), use **delete-object** with type \"project\". Note: workspace projects must be archived first; personal projects can be deleted regardless\n- **project-move**: Move projects between personal and workspace contexts\n- **add-sections/update-sections/find-sections**: Organize tasks within projects using sections\n- **get-overview**: Get comprehensive Markdown overview of entire account or specific project with task hierarchies. Project data includes parentId (sub-projects), folderId (workspace folder membership), and childOrder (sibling ordering)\n- **list-workspaces**: Get all workspaces for the user with details like plan type, role, and settings\n\n**Reminders:**\n- **add-reminders**: Create reminders for tasks. Three types: \"relative\" (minutes before due), \"absolute\" (specific date/time), or \"location\" (geofence-triggered). Each reminder must specify a taskId.\n- **find-reminders**: Find reminders by task ID (returns both time-based and location reminders), or get a specific reminder by ID (use reminderId for time-based, locationReminderId for location-based).\n- **update-reminders**: Update existing reminders. Must specify the reminder type (\"relative\", \"absolute\", or \"location\") and ID.\n- Relative and absolute reminders support an **isUrgent** flag to mark a reminder as urgent.\n- Reminders can be deleted using **delete-object** with type \"reminder\" (time-based) or \"location_reminder\" (location-based).\n\n**Collaboration & Comments:**\n- **add-comments/update-comments/find-comments**: Manage task and project discussions\n- **view-attachment**: View file attachments from comments. Pass the fileUrl from a comment's fileAttachment. Returns images inline, text files as text, and binary files as embedded resources.\n- **find-project-collaborators**: Look up Todoist users (collaborators, teammates) by name or email to get their user ID \u2014 use for \"find/who is X\" questions or any time you need to resolve a person's name to an ID. By default searches collaborators of every shared project the authenticated user can access (plus the authenticated user themselves). An empty result means the person is not a collaborator on any shared project, not that they do not exist. Pass projectId to scope to a single project\n- **manage-assignments**: Bulk assign/unassign/reassign up to 50 tasks with atomic operations and dry-run validation\n\n**Filters:**\n- **find-filters**: List all personal filters or search by name; filters are saved task views using query syntax\n- **add-filters**: Create personal filters with name, query (e.g. \"today & p1\"), color, and favorite flag\n- **update-filters**: Modify existing filters' name, query, color, or favorite status\n\n**Templates:**\n- **export-project-template**: Export a project as a Todoist template \u2014 CSV content (format \"file\") or a shareable link (format \"url\"). Prefer \"url\" for large projects. To read a project's contents rather than export it, use find-tasks.\n- **import-project-template**: Add a template's tasks, sections and comments to an existing project. Source it with templateId (a gallery slug like \"product-launch\", a personal template ID like \"UT_28Ex...\", or a full Todoist template URL) or with csvFileContent (CSV from export-project-template). Templates cannot be listed through this server, so only use the ID or URL the user gave you. Gallery templates work for any account; personal templates only for the account that saved them. For \"start a new project from this template\", call add-projects first and import into the new project. Imports write immediately and cannot be undone.\n\n**Activity & Audit:**\n- **find-activity**: Retrieve activity logs to monitor and audit changes. Shows events from all users by default; use initiatorId to filter by specific user. Filter by object type (task/project/comment), event type (added/updated/deleted/completed/uncompleted/archived/unarchived/shared/left), objects (objectId, projectId, taskId), and an inclusive dateFrom/exclusive dateTo range. For \u201Cwhat did I complete?\u201D or \u201Cwhat got done?\u201D questions, including recurring task occurrences, use objectType=\"task\", eventType=\"completed\", and the requested date range. Activity history retention depends on the user plan.\n- **get-productivity-stats**: Get comprehensive productivity statistics including daily/weekly completion breakdowns, goal streaks (current, last, max), karma score and trends, and historical karma data. No parameters required.\n\n**Project Health & Insights:**\n- **get-project-health**: Get comprehensive health assessment for a project including completion progress (completed/active counts, percentage), health status (EXCELLENT/ON_TRACK/AT_RISK/CRITICAL), description, and task-level recommendations. Use includeContext=true for detailed metrics (overdue tasks, weekly activity, avg completion time) and full task data. Health data may be stale \u2014 check isStale flag.\n- **get-project-activity-stats**: Get daily and optional weekly activity statistics for a project over a configurable time window (1-12 weeks). Useful for identifying activity trends.\n- **analyze-project-health**: Trigger a new health analysis for a project. Use when health data is stale. The analysis may take time \u2014 use get-project-health afterward to see updated results.\n- **get-workspace-insights**: Get aggregated health and progress insights across all projects in a workspace. Accepts workspace name or ID, with optional project ID filtering.\n\n**General Operations:**\n- **delete-object**: Remove projects, sections, tasks, comments, labels, filters, reminders, or location reminders by type and ID. Deletes both active and archived projects (workspace projects must be archived first; use find-projects with archivedStatus to locate archived projects)\n- **fetch-object**: Fetch a single task, project, comment, or section by its ID. Pass includeChildren=true to also get its direct children (subtasks for a task, sub-projects for a project) with a childCount - use this to check whether a task hides subtasks rather than a speculative find-tasks call\n- **reorder-objects**: Reorder sibling projects or sections, and optionally move projects to a new parent. For projects: set order to reorder siblings, and/or set parentId to move under a new parent (use \"root\" for top level). For sections: set order to reorder within a project\n- **user-info**: Get user details including timezone, goals, and plan information\n\n### Best Practices:\n\n1. **Task Creation**: Write clear, actionable task titles. Use natural language for due dates (\"tomorrow\", \"next Monday\"). Set appropriate priorities and include detailed descriptions when needed.\n\n2. **Search Strategy**: Use specific search queries combining multiple filters for precise results. When searching for tasks, start with broader queries and narrow down as needed.\n\n3. **Assignments & user lookup**: Always validate project collaborators exist before assigning tasks. Use find-project-collaborators to verify user access. Also use find-project-collaborators (with just a searchTerm and no projectId) to resolve a user's ID whenever the user references a person by name or email \u2014 it searches collaborators of all shared projects you can access, plus yourself.\n\n4. **Bulk Operations**: When working with multiple items, prefer bulk tools (complete-tasks, manage-assignments) over individual operations for better performance.\n\n5. **Date Handling**: All dates respect user timezone settings. Use 'today' keyword for dynamic date filtering (includes overdue tasks). **When rescheduling/moving tasks to a different date, always use reschedule-tasks** \u2014 never update-tasks with dueString, as that destroys recurrence on recurring tasks.\n\n6. **Labels**: Use label filtering with AND/OR operators for advanced task organization. Most search tools support labels parameter. Use **find-labels** to discover personal and shared labels \u2014 use label **names** (not IDs) when filtering tasks, and use label **IDs** only with **delete-object** and **update-labels** (for personal label updates). Use **add-labels** to create new personal labels. Use **update-labels** to rename or recolor personal labels (by ID), or to rename shared labels (by name) \u2014 note that shared labels support renaming only, not color/order/favorite changes.\n\n7. **Pagination**: Large result sets use cursor-based pagination. Use limit parameter to control result size (default varies by tool).\n\n8. **Error Handling**: All tools provide detailed error messages and next-step suggestions. Pay attention to validation feedback for corrective actions. Batch tools (e.g. add-tasks, update-tasks) report per-item `failures` alongside successes \u2014 a single failed item does not undo the rest of the batch. When an item fails, **do not retry the whole batch**; inspect its failure reason and only re-send the items that are actually fixable.\n\n### Common Workflows:\n\n- **Daily Planning**: Use find-tasks-by-date with 'today' and get-overview for project status\n- **Team Assignment**: find-project-collaborators \u2192 add-tasks with responsibleUser \u2192 manage-assignments for bulk changes\n- **User Lookup**: find-project-collaborators with just a searchTerm (no projectId) to resolve a name or email to a Todoist user ID across all shared-project collaborators you can access\n- **Task Search**: find-tasks with multiple filters \u2192 update-tasks or complete-tasks based on results\n- **Project Organization**: add-projects \u2192 add-sections \u2192 add-tasks with projectId and sectionId\n- **Completion History**: find-activity with objectType=\"task\", eventType=\"completed\", dateFrom, and dateTo to report what was actually completed in a period, including recurring task occurrences; use initiatorId for one collaborator\n- **Progress Reviews**: find-completed-tasks (defaults to last 7 days; optionally use explicit date ranges) \u2192 get-overview for project summaries\n- **Activity Auditing**: find-activity with event/object filters to track changes, monitor team activity, or investigate specific actions\n- **Productivity Analysis**: Use the productivity-analysis prompt for comprehensive analysis combining user-info, get-productivity-stats, and find-completed-tasks data into actionable insights\n- **Project Health Reviews**: get-project-health \u2192 analyze-project-health if stale \u2192 get-project-health with includeContext=true for detailed metrics \u2192 get-workspace-insights for cross-project overview\n\nAlways provide clear, actionable task titles and descriptions. Use the overview tools to give users context about their workload and project status.\n";
+export declare const instructions = "\n## Todoist Task and Project Management Tools\n\nYou have access to comprehensive Todoist management tools for personal productivity and team collaboration. Use these tools to help users manage tasks, projects, sections, comments, and assignments effectively.\n\n### Core Capabilities:\n- Create, update, complete, and search tasks with rich metadata (priorities, due dates, durations, assignments)\n- Manage projects and sections with flexible organization\n- Handle comments and collaboration features\n- Bulk assignment operations for team workflows\n- Get overviews and insights about workload and progress\n\n### Choosing between tools\n\nWhat each tool does and how to fill its parameters is in the tool's own description and input schema. This section covers only what those cannot: which tool to reach for, and how tools relate to each other.\n\n**Dates**\n\n- To move a task to a different date use **reschedule-tasks**, never **update-tasks**. update-tasks replaces the whole due string, which destroys recurrence on recurring tasks.\n- Never send a task's existing projectId, sectionId or parentId back to **update-tasks** \u2014 those fields are treated as a move.\n- All dates respect the user's timezone.\n\n**Finding things**\n\n- For \"what did I complete?\", use **find-activity** with objectType=\"task\" and eventType=\"completed\". It reports completion events, including each occurrence of a recurring task. **find-completed-tasks** lists completed tasks, which is not the same question.\n- To resolve a person's name or email to a user ID, or to answer \"who is X?\", use **find-project-collaborators** with just a searchTerm. It covers every shared project you can access plus yourself, so an empty result means they collaborate on none of them \u2014 not that they do not exist.\n- Before assigning work to someone, call **find-project-collaborators** again with the target projectId. Resolving an ID only proves they collaborate on *some* project you can see; assignment fails unless they collaborate on that one.\n- To find out whether a task hides subtasks, use **fetch-object** with includeChildren rather than a speculative **find-tasks** call.\n- Filter tasks by label **name**. Label IDs are only for **delete-object** and **update-labels**. Shared labels can be renamed but not recoloured, reordered or favourited.\n\n**Deleting and archiving**\n\n- **delete-object** removes every object type; there is no per-type delete tool. Reminders use type \"reminder\", location reminders \"location_reminder\".\n- A workspace project must be archived with **project-management** before it can be deleted. A personal project can be deleted directly.\n\n**Templates**\n\n- Templates cannot be listed through this server, so only use an ID or URL the user gave you. To start a new project from one, call **add-projects** first and import into it. Imports write immediately and cannot be undone.\n\n**Project health**\n\n- Health data may be stale \u2014 check the isStale flag, and call **analyze-project-health** to refresh it before reading again.\n\n**Batches**\n\n- Prefer a batch tool over one call per item.\n- Where a batch tool returns per-item `failures` alongside successes, one failure does not undo the rest: never retry the whole batch, re-send only the items whose failure reason is fixable. **reschedule-tasks** is the exception \u2014 it has no per-item failures and throws if any task fails, so retry it as a whole.\n\nUse the **productivity-analysis** prompt for a combined productivity review \u2014 it pulls together user-info, get-productivity-stats and find-completed-tasks.\n\nAlways provide clear, actionable task titles and descriptions. Use the overview tools to give users context about their workload and project status.\n";
 /**
  * Create the MCP server.
  * @param todoistApiKey - The API key for the todoist account.

package/dist/mcp-server.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"mcp-server.d.ts","sourceRoot":"","sources":["../src/mcp-server.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,yCAAyC,CAAA;AAGnE,OAAO,EACH,aAAa,EACb,KAAK,OAAO,EACZ,KAAK,WAAW,EAChB,KAAK,QAAQ,EAEhB,MAAM,kBAAkB,CAAA;AAKzB,eAAO,MAAM,YAAY,kqcAuGxB,CAAA;AAED;;;;;;GAMG;AACH,iBAAS,YAAY,CAAC,EAClB,aAAa,EACb,OAAO,EACP,QAAa,GAChB,EAAE;IACC,aAAa,EAAE,MAAM,CAAA;IACrB,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB,QAAQ,CAAC,EAAE,QAAQ,CAAA;CACtB,aA4CA;AAED,OAAO,EAAE,aAAa,EAAE,KAAK,OAAO,EAAE,KAAK,WAAW,EAAE,KAAK,QAAQ,EAAE,YAAY,EAAE,CAAA"}
+{"version":3,"file":"mcp-server.d.ts","sourceRoot":"","sources":["../src/mcp-server.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,yCAAyC,CAAA;AAGnE,OAAO,EACH,aAAa,EACb,KAAK,OAAO,EACZ,KAAK,WAAW,EAChB,KAAK,QAAQ,EAEhB,MAAM,kBAAkB,CAAA;AAKzB,eAAO,MAAM,YAAY,ksHAmDxB,CAAA;AAED;;;;;;GAMG;AACH,iBAAS,YAAY,CAAC,EAClB,aAAa,EACb,OAAO,EACP,QAAa,GAChB,EAAE;IACC,aAAa,EAAE,MAAM,CAAA;IACrB,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB,QAAQ,CAAC,EAAE,QAAQ,CAAA;CACtB,aA4CA;AAED,OAAO,EAAE,aAAa,EAAE,KAAK,OAAO,EAAE,KAAK,WAAW,EAAE,KAAK,QAAQ,EAAE,YAAY,EAAE,CAAA"}

package/dist/{require-valid-todoist-token-Cqt_drLE.js → require-valid-todoist-token-26-h3alF.js}

@@ -1,4 +1,4 @@
-import { Z as A, _, $ as T } from "./mcp-server-Cgo8Mi4X.js";
+import { Z as A, _, $ as T } from "./mcp-server-9sgQGVw_.js";
 const v = /* @__PURE__ */ new Set([401, 403]);
 function y(e) {
   const t = T(e);

package/dist/tools/find-tasks-by-date.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"find-tasks-by-date.d.ts","sourceRoot":"","sources":["../../src/tools/find-tasks-by-date.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AAsBvB,eAAO,MAAM,UAAU;;;;;;;;;;;;;;;;;;;;;CA+CtB,CAAA;AAUD,QAAA,MAAM,eAAe;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAqF0C,CAAA;AAyF/D,OAAO,EAAE,eAAe,EAAE,CAAA"}
+{"version":3,"file":"find-tasks-by-date.d.ts","sourceRoot":"","sources":["../../src/tools/find-tasks-by-date.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AAsBvB,eAAO,MAAM,UAAU;;;;;;;;;;;;;;;;;;;;;CAiDtB,CAAA;AAUD,QAAA,MAAM,eAAe;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAqF0C,CAAA;AAyF/D,OAAO,EAAE,eAAe,EAAE,CAAA"}

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@doist/todoist-mcp",
-    "version": "12.5.2",
+    "version": "12.5.3",
     "type": "module",
     "main": "./dist/index.js",
     "types": "./dist/index.d.ts",
@@ -76,6 +76,7 @@
         "@modelcontextprotocol/sdk": "^1.25.0"
     },
     "devDependencies": {
+        "@anthropic-ai/sdk": "0.115.0",
         "@modelcontextprotocol/inspector": "0.22.0",
         "@semantic-release/changelog": "6.0.3",
         "@semantic-release/exec": "7.1.0",

package/scripts/eval-instructions.ts

@@ -0,0 +1,415 @@
+#!/usr/bin/env npx tsx
+/**
+ * Measure whether the server's tool surface leads a model to the right tool.
+ *
+ * The token-footprint test says what the surface costs; it cannot say whether
+ * the surface still works. This does: it sends the same tool definitions and
+ * instructions a real client sends, gives the model a prompt, and checks which
+ * tool it reaches for and how it fills the arguments.
+ *
+ * It reads the surface from the working tree, so the way to compare two
+ * versions is to run it on each and diff the pass rates:
+ *
+ *   npx tsx scripts/eval-instructions.ts --label before
+ *   git stash pop
+ *   npx tsx scripts/eval-instructions.ts --label after
+ *
+ * Tool calls are never executed — only the first call of each turn is
+ * inspected — so this touches no Todoist data and needs no Todoist token. It
+ * does spend money on model calls; see --repeats and --models.
+ *
+ * Auth comes from the standard Anthropic credential chain, so `ant auth login`
+ * is enough (no ANTHROPIC_API_KEY needed).
+ *
+ * Usage:
+ *   npx tsx scripts/eval-instructions.ts [--label NAME] [--repeats N]
+ *                                        [--models a,b] [--scenario ID]
+ */
+import { mkdirSync, writeFileSync } from 'node:fs'
+import Anthropic from '@anthropic-ai/sdk'
+import { z } from 'zod'
+import { instructions } from '../src/mcp-server.js'
+import { registeredTools } from '../src/tool-registry.js'
+import { createLimiter } from '../src/utils/concurrency.js'
+import { ToolNames } from '../src/utils/tool-names.js'
+
+type Check = (input: Record<string, unknown>) => string | null
+
+/** First item of a batch tool's array argument, e.g. add-tasks' `tasks`. */
+function firstItem(input: Record<string, unknown>, key: string): Record<string, unknown> | null {
+    const list = input[key]
+    if (!Array.isArray(list) || list.length === 0) return null
+    const item = list[0]
+    return item && typeof item === 'object' ? (item as Record<string, unknown>) : null
+}
+
+type Scenario = {
+    id: string
+    prompt: string
+    /**
+     * Any of these counts as the right tool. Use for a rule that names the
+     * tool to reach for. Omit when the rule is "don't do X" — an allowlist
+     * then fails legitimate lookup steps it did not anticipate, which is
+     * noise, not signal.
+     */
+    expect?: string[]
+    /** Calling any of these fails the scenario. Use for "don't do X" rules. */
+    forbid?: string[]
+    /** Extra assertion on the arguments; return a reason on failure. */
+    check?: Check
+    /** What this scenario is protecting. */
+    guards: string
+}
+
+const SCENARIOS: Scenario[] = [
+    {
+        // The ID is in the prompt on purpose. Without one the model has to
+        // search first, which is correct behaviour and tells us nothing about
+        // the rule under test.
+        id: 'reschedule-not-update',
+        prompt: 'Move task 6XG4Vw2c9J ("Weekly review", repeats every Monday) to next Tuesday.',
+        // The rule is "not update-tasks", so anything else -- including a
+        // preliminary lookup -- is acceptable.
+        forbid: [ToolNames.UPDATE_TASKS],
+        guards: 'instructions: reschedule-tasks vs update-tasks (recurrence loss)',
+    },
+    {
+        id: 'completed-via-activity',
+        prompt: 'What did I actually get done last week?',
+        expect: [ToolNames.FIND_ACTIVITY],
+        check: (input) => {
+            // Exact match: "uncompleted" contains "completed".
+            if (input.eventType !== 'completed') {
+                return `eventType is ${JSON.stringify(input.eventType)}, not "completed"`
+            }
+            if (!input.dateFrom || !input.dateTo) {
+                return 'no dateFrom/dateTo, so this searches all history rather than last week'
+            }
+            return null
+        },
+        guards: 'instructions: find-activity vs find-completed-tasks, over a bounded range',
+    },
+    {
+        id: 'resolve-person',
+        prompt: 'Who is Sarah?',
+        expect: [ToolNames.FIND_PROJECT_COLLABORATORS],
+        guards: 'instructions: resolving a name to a user ID',
+    },
+    {
+        id: 'subtask-check',
+        prompt: 'Does task 6XG4Vw2c9J have any subtasks?',
+        expect: [ToolNames.FETCH_OBJECT],
+        check: (input) =>
+            input.includeChildren === true ? null : 'includeChildren not set to true',
+        guards: 'instructions: fetch-object over a speculative find-tasks',
+    },
+    {
+        id: 'label-by-name',
+        prompt: 'Show me my tasks labelled urgent.',
+        expect: [ToolNames.FIND_TASKS, ToolNames.FIND_LABELS],
+        check: (input) => {
+            const labels = input.labels
+            const used = Array.isArray(labels) ? labels.map(String) : []
+            if (used.some((l) => l.toLowerCase() === 'urgent')) return null
+            // find-labels legitimately takes a search term rather than a labels array.
+            const search = String(input.searchTerm ?? input.name ?? '').toLowerCase()
+            return search.includes('urgent')
+                ? null
+                : `label name not used: ${JSON.stringify(input)}`
+        },
+        guards: 'instructions: filter by label name, not ID',
+    },
+    {
+        id: 'archive-before-delete',
+        prompt: 'Delete workspace project 6XQ3Plan99 ("Q3 Planning").',
+        // The rule is "archive first", so the only wrong first move is the
+        // delete itself. Archiving and any lookup (find-projects, get-overview,
+        // fetch-object) are all legitimate openers.
+        forbid: [ToolNames.DELETE_OBJECT],
+        guards: 'instructions: workspace projects archive before delete',
+    },
+    {
+        id: 'priority-string',
+        prompt: 'Set task 6XG4Vw2c9J to high priority.',
+        expect: [ToolNames.UPDATE_TASKS],
+        check: (input) => {
+            const json = JSON.stringify(input)
+            if (/"priority"\s*:\s*"p[1-4]"/.test(json)) return null
+            if (/"priority"\s*:\s*\d/.test(json)) return `priority sent as an integer: ${json}`
+            return `no p1-p4 priority found: ${json}`
+        },
+        guards: 'input field description: priority is "p1".."p4", never an integer',
+    },
+    {
+        id: 'recurring-due-string',
+        prompt: 'Add a task "Water the plants" due every Monday.',
+        expect: [ToolNames.ADD_TASKS],
+        check: (input) => {
+            const task = firstItem(input, 'tasks')
+            if (!task) return `no tasks array: ${JSON.stringify(input)}`
+            const dueString = String(task.dueString ?? '')
+            // The recurrence has to be in dueString, not smuggled into content.
+            if (!dueString.toLowerCase().includes('monday')) {
+                return `recurrence not in dueString: ${JSON.stringify(task)}`
+            }
+            return /^\s*recurring/i.test(dueString)
+                ? `dueString prefixed with "recurring": ${dueString}`
+                : null
+        },
+        guards: 'input field description: no "recurring" prefix on dueString',
+    },
+    {
+        id: 'today-includes-overdue',
+        prompt: 'Show me the tasks due today.',
+        expect: [ToolNames.FIND_TASKS_BY_DATE, ToolNames.GET_OVERVIEW],
+        check: (input) => {
+            // get-overview takes no date argument; only assert on the dated tool.
+            if (!('startDate' in input)) return null
+            // A concrete YYYY-MM-DD satisfies the schema but loses the overdue
+            // behaviour the keyword carries.
+            return input.startDate === 'today'
+                ? null
+                : `startDate is ${JSON.stringify(input.startDate)}, not the 'today' keyword`
+        },
+        guards: "input field description: startDate 'today' and its overdue behaviour",
+    },
+    {
+        id: 'no-container-echo',
+        prompt: 'Rename task 6XG4Vw2c9J to "Draft the Q4 report".',
+        expect: [ToolNames.UPDATE_TASKS],
+        check: (input) => {
+            const json = JSON.stringify(input)
+            return /"(projectId|sectionId|parentId)"/.test(json)
+                ? `echoed a container field, which is treated as a move: ${json}`
+                : null
+        },
+        guards: 'instructions: never echo projectId/sectionId/parentId back',
+    },
+]
+
+const DEFAULT_MODELS = ['claude-haiku-4-5', 'claude-sonnet-5']
+const DEFAULT_REPEATS = 5
+const MAX_CONCURRENCY = 4
+
+function parseArgs() {
+    const args = process.argv.slice(2)
+    const get = (flag: string) => {
+        const i = args.indexOf(flag)
+        return i === -1 ? undefined : args[i + 1]
+    }
+    const label = get('--label') ?? 'run'
+    // The label becomes a filename, so keep it to characters that cannot escape
+    // the output directory.
+    if (!/^[A-Za-z0-9_-]+$/.test(label)) {
+        console.error(`--label must match [A-Za-z0-9_-]+, got "${label}"`)
+        process.exit(1)
+    }
+    return {
+        label,
+        repeats: Number(get('--repeats') ?? DEFAULT_REPEATS),
+        models: (get('--models') ?? DEFAULT_MODELS.join(','))
+            .split(',')
+            .map((m) => m.trim())
+            .filter(Boolean),
+        scenario: get('--scenario'),
+    }
+}
+
+/**
+ * The tool definitions as an MCP client forwards them to the Messages API.
+ *
+ * Note the Messages API tool shape has no output-schema field, so a tool's
+ * `outputSchema` never reaches the model here however the server advertises it.
+ */
+function buildTools(): Anthropic.Tool[] {
+    return registeredTools.map((tool) => ({
+        name: tool.name,
+        description: tool.description,
+        // `InputSchema` carries an index signature, so the extra JSON Schema
+        // keys Zod emits pass through rather than being rejected by the cast.
+        input_schema: z.toJSONSchema(z.object(tool.parameters), {
+            unrepresentable: 'any',
+            io: 'input',
+            target: 'draft-7',
+        }) as Anthropic.Tool.InputSchema,
+    }))
+}
+
+type Usage = { input: number; output: number; cacheWrite: number; cacheRead: number }
+
+type Attempt = {
+    scenario: string
+    model: string
+    calledTool: string | null
+    pass: boolean
+    reason: string | null
+    /**
+     * The request itself failed (auth, rate limit, transport). Distinct from a
+     * model that answered but chose wrong — counting these as routing failures
+     * would let a run with bad credentials report 0% and be saved as a result.
+     */
+    errored: boolean
+    usage: Usage
+}
+
+async function runAttempt(
+    client: Anthropic,
+    model: string,
+    scenario: Scenario,
+    tools: Anthropic.Tool[],
+): Promise<Attempt> {
+    const base = {
+        scenario: scenario.id,
+        model,
+        errored: false,
+        usage: { input: 0, output: 0, cacheWrite: 0, cacheRead: 0 },
+    }
+    try {
+        const response = await client.messages.create({
+            model,
+            max_tokens: 4096,
+            // Tools render before system, so one breakpoint here caches both.
+            system: [{ type: 'text', text: instructions, cache_control: { type: 'ephemeral' } }],
+            tools,
+            messages: [{ role: 'user', content: scenario.prompt }],
+        })
+
+        base.usage = {
+            input: response.usage.input_tokens,
+            output: response.usage.output_tokens,
+            cacheWrite: response.usage.cache_creation_input_tokens ?? 0,
+            cacheRead: response.usage.cache_read_input_tokens ?? 0,
+        }
+
+        const call = response.content.find((b) => b.type === 'tool_use')
+        if (!call) {
+            return { ...base, calledTool: null, pass: false, reason: 'no tool call' }
+        }
+        if (scenario.forbid?.includes(call.name)) {
+            return {
+                ...base,
+                calledTool: call.name,
+                pass: false,
+                reason: `called ${call.name}, which this rule forbids`,
+            }
+        }
+        if (scenario.expect && !scenario.expect.includes(call.name)) {
+            return {
+                ...base,
+                calledTool: call.name,
+                pass: false,
+                reason: `expected ${scenario.expect.join(' or ')}`,
+            }
+        }
+        // An argument check written for a specific tool must not run against a
+        // different one a forbid-only scenario legitimately allows.
+        if (scenario.forbid && !scenario.expect) {
+            return { ...base, calledTool: call.name, pass: true, reason: null }
+        }
+        const reason = scenario.check?.(call.input as Record<string, unknown>) ?? null
+        return { ...base, calledTool: call.name, pass: reason === null, reason }
+    } catch (error) {
+        const message = error instanceof Error ? error.message : String(error)
+        return {
+            ...base,
+            calledTool: null,
+            pass: false,
+            errored: true,
+            reason: `request failed: ${message}`,
+        }
+    }
+}
+
+async function main() {
+    const { label, repeats, models, scenario: only } = parseArgs()
+    const scenarios = only ? SCENARIOS.filter((s) => s.id === only) : SCENARIOS
+    if (scenarios.length === 0) {
+        console.error(`No scenario matching "${only}"`)
+        process.exit(1)
+    }
+
+    const client = new Anthropic()
+    const tools = buildTools()
+    // queueTimeoutMs: 0 disables the default queue deadline, which exists for
+    // request-scoped work and does not apply to a batch script.
+    const limit = createLimiter(MAX_CONCURRENCY, { queueTimeoutMs: 0 })
+
+    console.log(`label:     ${label}`)
+    console.log(`tools:     ${tools.length}`)
+    console.log(`scenarios: ${scenarios.length} x ${repeats} repeats x ${models.length} models`)
+    console.log(`models:    ${models.join(', ')}\n`)
+
+    const attempts: Attempt[] = []
+    for (const model of models) {
+        const jobs = scenarios.flatMap((s) => Array.from({ length: repeats }, () => s))
+        // Run one first so it writes the shared prefix to cache; the rest read it.
+        const first = jobs[0]
+        if (!first) continue
+        attempts.push(await runAttempt(client, model, first, tools))
+        attempts.push(
+            ...(await Promise.all(
+                jobs.slice(1).map((s) => limit(() => runAttempt(client, model, s, tools))),
+            )),
+        )
+        console.log(`${model}: done`)
+    }
+
+    console.log(`\n=== ${label} ===`)
+    for (const model of models) {
+        console.log(`\n${model}`)
+        for (const s of scenarios) {
+            const all = attempts.filter((a) => a.model === model && a.scenario === s.id)
+            const rows = all.filter((a) => !a.errored)
+            const passed = rows.filter((a) => a.pass).length
+            const rate = rows.length ? Math.round((passed / rows.length) * 100) : 0
+            const mark = rate === 100 ? '✓' : rate >= 60 ? '~' : '✗'
+            console.log(`  ${mark} ${String(rate).padStart(3)}%  ${s.id}`)
+            const failure = rows.find((a) => !a.pass)
+            if (failure) {
+                console.log(
+                    `           ${failure.calledTool ?? 'no call'} — ${failure.reason ?? ''}`,
+                )
+            }
+        }
+        const rows = attempts.filter((a) => a.model === model && !a.errored)
+        const passed = rows.filter((a) => a.pass).length
+        console.log(`  overall: ${passed}/${rows.length}`)
+    }
+
+    const errored = attempts.filter((a) => a.errored)
+    if (errored.length > 0) {
+        console.error(
+            `\n${errored.length}/${attempts.length} requests failed outright ` +
+                `(not counted as routing failures). First: ${errored[0]?.reason}`,
+        )
+    }
+
+    const total = attempts.reduce(
+        (acc, a) => ({
+            input: acc.input + a.usage.input,
+            output: acc.output + a.usage.output,
+            cacheWrite: acc.cacheWrite + a.usage.cacheWrite,
+            cacheRead: acc.cacheRead + a.usage.cacheRead,
+        }),
+        { input: 0, output: 0, cacheWrite: 0, cacheRead: 0 },
+    )
+    console.log(
+        `\ntokens: ${total.input} uncached in, ${total.cacheWrite} cache write, ` +
+            `${total.cacheRead} cache read, ${total.output} out`,
+    )
+
+    mkdirSync('tmp/eval', { recursive: true })
+    const out = `tmp/eval/${label}.json`
+    writeFileSync(out, JSON.stringify({ label, models, repeats, attempts }, null, 2))
+    console.log(`\nwrote ${out}`)
+
+    // A run that could not obtain its samples is not a result to compare against.
+    if (errored.length > 0) {
+        process.exitCode = 1
+    }
+}
+
+main().catch((error) => {
+    console.error(error)
+    process.exit(1)
+})