hive-skills 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (238) hide show
  1. package/LICENSE +28 -0
  2. package/README.md +235 -0
  3. package/assets/LICENSE +32 -0
  4. package/assets/THIRD_PARTY_NOTICES.md +72 -0
  5. package/assets/manifest.json +1290 -0
  6. package/assets/skills/authored/code-review/composable/BUNDLE.md +192 -0
  7. package/assets/skills/authored/code-review/composable/INDEX.md +13 -0
  8. package/assets/skills/authored/code-review/composable/VERSION +1 -0
  9. package/assets/skills/authored/code-review/composable/mini/01-review-method.md +34 -0
  10. package/assets/skills/authored/code-review/composable/mini/02-security-review.md +20 -0
  11. package/assets/skills/authored/code-review/composable/mini/03-correctness-bugs.md +19 -0
  12. package/assets/skills/authored/code-review/composable/mini/04-concurrency.md +16 -0
  13. package/assets/skills/authored/code-review/composable/mini/05-error-handling.md +12 -0
  14. package/assets/skills/authored/code-review/composable/mini/06-performance.md +11 -0
  15. package/assets/skills/authored/code-review/composable/mini/07-api-design.md +20 -0
  16. package/assets/skills/authored/code-review/composable/mini/08-missing-tests.md +25 -0
  17. package/assets/skills/authored/code-review/composable/mini/09-refactoring-and-communication.md +32 -0
  18. package/assets/skills/authored/data-analysis/composable/BUNDLE.md +186 -0
  19. package/assets/skills/authored/data-analysis/composable/INDEX.md +13 -0
  20. package/assets/skills/authored/data-analysis/composable/VERSION +1 -0
  21. package/assets/skills/authored/data-analysis/composable/mini/01-data-quality-profiling.md +26 -0
  22. package/assets/skills/authored/data-analysis/composable/mini/02-aggregation-pitfalls.md +21 -0
  23. package/assets/skills/authored/data-analysis/composable/mini/03-time-series.md +19 -0
  24. package/assets/skills/authored/data-analysis/composable/mini/04-segmentation.md +23 -0
  25. package/assets/skills/authored/data-analysis/composable/mini/05-discount-pricing.md +17 -0
  26. package/assets/skills/authored/data-analysis/composable/mini/06-insight-generation.md +25 -0
  27. package/assets/skills/authored/data-analysis/composable/mini/07-executive-writing.md +26 -0
  28. package/assets/skills/authored/data-analysis/composable/mini/08-recommendations.md +20 -0
  29. package/assets/skills/authored/data-analysis/composable/mini/09-reproducibility-assumptions.md +20 -0
  30. package/assets/skills/authored/financial-analysis/composable/BUNDLE.md +1293 -0
  31. package/assets/skills/authored/financial-analysis/composable/INDEX.md +17 -0
  32. package/assets/skills/authored/financial-analysis/composable/VERSION +1 -0
  33. package/assets/skills/authored/financial-analysis/composable/mini/00-core.md +69 -0
  34. package/assets/skills/authored/financial-analysis/composable/mini/01-profitability-efficiency-ratios.md +173 -0
  35. package/assets/skills/authored/financial-analysis/composable/mini/02-liquidity-leverage-ratios.md +120 -0
  36. package/assets/skills/authored/financial-analysis/composable/mini/03-valuation-ratios-interpretation.md +107 -0
  37. package/assets/skills/authored/financial-analysis/composable/mini/04-dcf-projection-wacc.md +122 -0
  38. package/assets/skills/authored/financial-analysis/composable/mini/05-dcf-terminal-value-sensitivity.md +75 -0
  39. package/assets/skills/authored/financial-analysis/composable/mini/06-comparables-precedents.md +78 -0
  40. package/assets/skills/authored/financial-analysis/composable/mini/07-budget-variance-analysis.md +47 -0
  41. package/assets/skills/authored/financial-analysis/composable/mini/08-driver-based-forecasting-scenarios.md +111 -0
  42. package/assets/skills/authored/financial-analysis/composable/mini/09-rolling-forecasts-accuracy.md +114 -0
  43. package/assets/skills/authored/financial-analysis/composable/mini/10-reporting-exec-summary.md +58 -0
  44. package/assets/skills/authored/financial-analysis/composable/mini/11-industry-adaptations.md +130 -0
  45. package/assets/skills/authored/financial-analysis/composable/mini/12-toolkit-scripts-io.md +102 -0
  46. package/assets/skills/authored/python-api/composable/BUNDLE.md +389 -0
  47. package/assets/skills/authored/python-api/composable/INDEX.md +13 -0
  48. package/assets/skills/authored/python-api/composable/VERSION +1 -0
  49. package/assets/skills/authored/python-api/composable/mini/01-project-structure.md +44 -0
  50. package/assets/skills/authored/python-api/composable/mini/02-routing-and-app.md +38 -0
  51. package/assets/skills/authored/python-api/composable/mini/03-pydantic-models.md +45 -0
  52. package/assets/skills/authored/python-api/composable/mini/04-validation-and-errors.md +59 -0
  53. package/assets/skills/authored/python-api/composable/mini/05-dependencies-and-auth.md +60 -0
  54. package/assets/skills/authored/python-api/composable/mini/06-pagination-filtering.md +44 -0
  55. package/assets/skills/authored/python-api/composable/mini/07-async-performance.md +30 -0
  56. package/assets/skills/authored/python-api/composable/mini/08-testing.md +45 -0
  57. package/assets/skills/authored/python-api/composable/mini/09-middleware-observability.md +29 -0
  58. package/assets/skills/authored/tech-writing/composable/BUNDLE.md +220 -0
  59. package/assets/skills/authored/tech-writing/composable/INDEX.md +12 -0
  60. package/assets/skills/authored/tech-writing/composable/VERSION +1 -0
  61. package/assets/skills/authored/tech-writing/composable/mini/01-audience-analysis.md +26 -0
  62. package/assets/skills/authored/tech-writing/composable/mini/02-document-types.md +27 -0
  63. package/assets/skills/authored/tech-writing/composable/mini/03-readme-quickstart-changelog.md +22 -0
  64. package/assets/skills/authored/tech-writing/composable/mini/04-information-architecture.md +21 -0
  65. package/assets/skills/authored/tech-writing/composable/mini/05-style-clarity.md +29 -0
  66. package/assets/skills/authored/tech-writing/composable/mini/06-breaking-changes-migrations.md +33 -0
  67. package/assets/skills/authored/tech-writing/composable/mini/07-troubleshooting-security.md +30 -0
  68. package/assets/skills/authored/tech-writing/composable/mini/08-faq-voice-tone.md +32 -0
  69. package/assets/skills/converted/claude-api/composable/BUNDLE.md +14996 -0
  70. package/assets/skills/converted/claude-api/composable/INDEX.md +98 -0
  71. package/assets/skills/converted/claude-api/composable/VERSION +1 -0
  72. package/assets/skills/converted/claude-api/composable/mini/00-core.md +235 -0
  73. package/assets/skills/converted/claude-api/composable/mini/01-model-api-quick-reference.md +258 -0
  74. package/assets/skills/converted/claude-api/composable/mini/02-common-pitfalls.md +42 -0
  75. package/assets/skills/converted/claude-api/composable/mini/10-models.md +135 -0
  76. package/assets/skills/converted/claude-api/composable/mini/11-platform-availability.md +96 -0
  77. package/assets/skills/converted/claude-api/composable/mini/12-claude-platform-on-aws.md +59 -0
  78. package/assets/skills/converted/claude-api/composable/mini/13-live-sources.md +143 -0
  79. package/assets/skills/converted/claude-api/composable/mini/14-error-codes.md +256 -0
  80. package/assets/skills/converted/claude-api/composable/mini/15-prompt-caching.md +223 -0
  81. package/assets/skills/converted/claude-api/composable/mini/16-token-counting.md +56 -0
  82. package/assets/skills/converted/claude-api/composable/mini/17-tool-use-concepts.md +444 -0
  83. package/assets/skills/converted/claude-api/composable/mini/18-agent-design.md +101 -0
  84. package/assets/skills/converted/claude-api/composable/mini/19-anthropic-cli.md +246 -0
  85. package/assets/skills/converted/claude-api/composable/mini/20-model-migration.md +1301 -0
  86. package/assets/skills/converted/claude-api/composable/mini/30-managed-agents-overview.md +71 -0
  87. package/assets/skills/converted/claude-api/composable/mini/31-managed-agents-core.md +278 -0
  88. package/assets/skills/converted/claude-api/composable/mini/32-managed-agents-environments.md +219 -0
  89. package/assets/skills/converted/claude-api/composable/mini/33-managed-agents-tools.md +366 -0
  90. package/assets/skills/converted/claude-api/composable/mini/34-managed-agents-events.md +252 -0
  91. package/assets/skills/converted/claude-api/composable/mini/35-managed-agents-outcomes.md +106 -0
  92. package/assets/skills/converted/claude-api/composable/mini/36-managed-agents-multiagent.md +101 -0
  93. package/assets/skills/converted/claude-api/composable/mini/37-managed-agents-webhooks.md +123 -0
  94. package/assets/skills/converted/claude-api/composable/mini/38-managed-agents-memory.md +197 -0
  95. package/assets/skills/converted/claude-api/composable/mini/39-managed-agents-scheduled-deployments.md +146 -0
  96. package/assets/skills/converted/claude-api/composable/mini/40-managed-agents-client-patterns.md +211 -0
  97. package/assets/skills/converted/claude-api/composable/mini/41-managed-agents-onboarding.md +82 -0
  98. package/assets/skills/converted/claude-api/composable/mini/42-managed-agents-api-reference.md +441 -0
  99. package/assets/skills/converted/claude-api/composable/mini/43-managed-agents-self-hosted-sandboxes.md +174 -0
  100. package/assets/skills/converted/claude-api/composable/mini/50-python-readme.md +564 -0
  101. package/assets/skills/converted/claude-api/composable/mini/51-python-tool-use.md +590 -0
  102. package/assets/skills/converted/claude-api/composable/mini/52-python-streaming-batches-files.md +549 -0
  103. package/assets/skills/converted/claude-api/composable/mini/53-python-managed-agents.md +335 -0
  104. package/assets/skills/converted/claude-api/composable/mini/55-typescript-readme.md +404 -0
  105. package/assets/skills/converted/claude-api/composable/mini/56-typescript-tool-use.md +548 -0
  106. package/assets/skills/converted/claude-api/composable/mini/57-typescript-streaming-batches-files.md +384 -0
  107. package/assets/skills/converted/claude-api/composable/mini/58-typescript-managed-agents.md +358 -0
  108. package/assets/skills/converted/claude-api/composable/mini/60-csharp-readme.md +361 -0
  109. package/assets/skills/converted/claude-api/composable/mini/61-csharp-tool-use.md +163 -0
  110. package/assets/skills/converted/claude-api/composable/mini/62-csharp-streaming-batches-files.md +64 -0
  111. package/assets/skills/converted/claude-api/composable/mini/65-go-readme.md +185 -0
  112. package/assets/skills/converted/claude-api/composable/mini/66-go-tool-use.md +219 -0
  113. package/assets/skills/converted/claude-api/composable/mini/67-go-streaming-files.md +63 -0
  114. package/assets/skills/converted/claude-api/composable/mini/68-go-managed-agents.md +564 -0
  115. package/assets/skills/converted/claude-api/composable/mini/70-java-readme.md +237 -0
  116. package/assets/skills/converted/claude-api/composable/mini/71-java-tool-use.md +227 -0
  117. package/assets/skills/converted/claude-api/composable/mini/72-java-streaming-files.md +49 -0
  118. package/assets/skills/converted/claude-api/composable/mini/73-java-managed-agents.md +443 -0
  119. package/assets/skills/converted/claude-api/composable/mini/75-php-readme.md +173 -0
  120. package/assets/skills/converted/claude-api/composable/mini/76-php-tool-use.md +252 -0
  121. package/assets/skills/converted/claude-api/composable/mini/77-php-streaming-batches-files.md +53 -0
  122. package/assets/skills/converted/claude-api/composable/mini/78-php-managed-agents.md +443 -0
  123. package/assets/skills/converted/claude-api/composable/mini/80-ruby-readme.md +133 -0
  124. package/assets/skills/converted/claude-api/composable/mini/81-ruby-tool-use-streaming.md +56 -0
  125. package/assets/skills/converted/claude-api/composable/mini/82-ruby-managed-agents.md +394 -0
  126. package/assets/skills/converted/claude-api/composable/mini/85-curl-examples.md +255 -0
  127. package/assets/skills/converted/claude-api/composable/mini/86-curl-managed-agents.md +340 -0
  128. package/assets/skills/converted/claude-api/composable/presets/csharp.md +1151 -0
  129. package/assets/skills/converted/claude-api/composable/presets/curl.md +1154 -0
  130. package/assets/skills/converted/claude-api/composable/presets/go.md +1598 -0
  131. package/assets/skills/converted/claude-api/composable/presets/java.md +1523 -0
  132. package/assets/skills/converted/claude-api/composable/presets/php.md +1488 -0
  133. package/assets/skills/converted/claude-api/composable/presets/python.md +2605 -0
  134. package/assets/skills/converted/claude-api/composable/presets/ruby.md +1146 -0
  135. package/assets/skills/converted/claude-api/composable/presets/shared.md +6478 -0
  136. package/assets/skills/converted/claude-api/composable/presets/typescript.md +2261 -0
  137. package/assets/skills/converted/docx/composable/BUNDLE.md +617 -0
  138. package/assets/skills/converted/docx/composable/INDEX.md +12 -0
  139. package/assets/skills/converted/docx/composable/VERSION +1 -0
  140. package/assets/skills/converted/docx/composable/mini/00-core.md +26 -0
  141. package/assets/skills/converted/docx/composable/mini/01-reading-converting.md +36 -0
  142. package/assets/skills/converted/docx/composable/mini/02-creating-setup-layout.md +85 -0
  143. package/assets/skills/converted/docx/composable/mini/03-creating-elements.md +257 -0
  144. package/assets/skills/converted/docx/composable/mini/04-editing-existing.md +57 -0
  145. package/assets/skills/converted/docx/composable/mini/05-xml-reference.md +128 -0
  146. package/assets/skills/converted/internal-comms/composable/BUNDLE.md +220 -0
  147. package/assets/skills/converted/internal-comms/composable/INDEX.md +11 -0
  148. package/assets/skills/converted/internal-comms/composable/VERSION +1 -0
  149. package/assets/skills/converted/internal-comms/composable/mini/00-core.md +30 -0
  150. package/assets/skills/converted/internal-comms/composable/mini/01-3p-updates.md +49 -0
  151. package/assets/skills/converted/internal-comms/composable/mini/02-company-newsletter.md +67 -0
  152. package/assets/skills/converted/internal-comms/composable/mini/03-faq-answers.md +32 -0
  153. package/assets/skills/converted/internal-comms/composable/mini/04-general-comms.md +18 -0
  154. package/assets/skills/converted/mcp-builder/composable/BUNDLE.md +2460 -0
  155. package/assets/skills/converted/mcp-builder/composable/INDEX.md +23 -0
  156. package/assets/skills/converted/mcp-builder/composable/VERSION +1 -0
  157. package/assets/skills/converted/mcp-builder/composable/mini/00-core.md +43 -0
  158. package/assets/skills/converted/mcp-builder/composable/mini/01-workflow-research-planning.md +54 -0
  159. package/assets/skills/converted/mcp-builder/composable/mini/02-implementation-process.md +78 -0
  160. package/assets/skills/converted/mcp-builder/composable/mini/03-naming-conventions.md +33 -0
  161. package/assets/skills/converted/mcp-builder/composable/mini/04-response-formats.md +21 -0
  162. package/assets/skills/converted/mcp-builder/composable/mini/05-pagination.md +24 -0
  163. package/assets/skills/converted/mcp-builder/composable/mini/06-transport.md +47 -0
  164. package/assets/skills/converted/mcp-builder/composable/mini/07-annotations.md +20 -0
  165. package/assets/skills/converted/mcp-builder/composable/mini/08-security.md +35 -0
  166. package/assets/skills/converted/mcp-builder/composable/mini/09-error-handling.md +32 -0
  167. package/assets/skills/converted/mcp-builder/composable/mini/10-testing-and-documentation.md +21 -0
  168. package/assets/skills/converted/mcp-builder/composable/mini/11-python-implementation.md +432 -0
  169. package/assets/skills/converted/mcp-builder/composable/mini/12-python-advanced-and-checklist.md +233 -0
  170. package/assets/skills/converted/mcp-builder/composable/mini/13-node-implementation.md +717 -0
  171. package/assets/skills/converted/mcp-builder/composable/mini/14-node-advanced-and-checklist.md +200 -0
  172. package/assets/skills/converted/mcp-builder/composable/mini/15-evaluation-design.md +200 -0
  173. package/assets/skills/converted/mcp-builder/composable/mini/16-evaluation-running.md +198 -0
  174. package/assets/skills/converted/mcp-builder/composable/presets/node-server.md +1381 -0
  175. package/assets/skills/converted/mcp-builder/composable/presets/python-server.md +1129 -0
  176. package/assets/skills/converted/pdf/composable/BUNDLE.md +1284 -0
  177. package/assets/skills/converted/pdf/composable/INDEX.md +14 -0
  178. package/assets/skills/converted/pdf/composable/VERSION +1 -0
  179. package/assets/skills/converted/pdf/composable/mini/00-core.md +46 -0
  180. package/assets/skills/converted/pdf/composable/mini/01-creation.md +176 -0
  181. package/assets/skills/converted/pdf/composable/mini/02-extraction.md +178 -0
  182. package/assets/skills/converted/pdf/composable/mini/03-merge-split-manipulate.md +240 -0
  183. package/assets/skills/converted/pdf/composable/mini/04-images-ocr.md +168 -0
  184. package/assets/skills/converted/pdf/composable/mini/05-encryption.md +51 -0
  185. package/assets/skills/converted/pdf/composable/mini/06-forms.md +300 -0
  186. package/assets/skills/converted/pdf/composable/mini/07-libraries-reference.md +89 -0
  187. package/assets/skills/converted/pptx/composable/BUNDLE.md +911 -0
  188. package/assets/skills/converted/pptx/composable/INDEX.md +12 -0
  189. package/assets/skills/converted/pptx/composable/VERSION +1 -0
  190. package/assets/skills/converted/pptx/composable/mini/00-core.md +37 -0
  191. package/assets/skills/converted/pptx/composable/mini/01-reading.md +37 -0
  192. package/assets/skills/converted/pptx/composable/mini/02-creating-from-scratch.md +422 -0
  193. package/assets/skills/converted/pptx/composable/mini/03-editing-templates.md +207 -0
  194. package/assets/skills/converted/pptx/composable/mini/04-design.md +89 -0
  195. package/assets/skills/converted/pptx/composable/mini/05-qa-and-rendering.md +91 -0
  196. package/assets/skills/converted/pptx/composable/presets/creating.md +659 -0
  197. package/assets/skills/converted/pptx/composable/presets/editing.md +485 -0
  198. package/assets/skills/converted/skill-creator/composable/BUNDLE.md +1681 -0
  199. package/assets/skills/converted/skill-creator/composable/INDEX.md +21 -0
  200. package/assets/skills/converted/skill-creator/composable/VERSION +1 -0
  201. package/assets/skills/converted/skill-creator/composable/mini/00-core.md +59 -0
  202. package/assets/skills/converted/skill-creator/composable/mini/01-creating-a-skill.md +27 -0
  203. package/assets/skills/converted/skill-creator/composable/mini/02-skill-writing-guide.md +71 -0
  204. package/assets/skills/converted/skill-creator/composable/mini/03-test-cases.md +21 -0
  205. package/assets/skills/converted/skill-creator/composable/mini/04-running-and-evaluating.md +128 -0
  206. package/assets/skills/converted/skill-creator/composable/mini/05-improving-the-skill.md +30 -0
  207. package/assets/skills/converted/skill-creator/composable/mini/06-blind-comparison.md +5 -0
  208. package/assets/skills/converted/skill-creator/composable/mini/07-description-optimization.md +74 -0
  209. package/assets/skills/converted/skill-creator/composable/mini/08-packaging.md +11 -0
  210. package/assets/skills/converted/skill-creator/composable/mini/09-environment-specific.md +38 -0
  211. package/assets/skills/converted/skill-creator/composable/mini/10-scripts-and-references.md +34 -0
  212. package/assets/skills/converted/skill-creator/composable/mini/11-json-schemas.md +414 -0
  213. package/assets/skills/converted/skill-creator/composable/mini/12-agent-grader.md +225 -0
  214. package/assets/skills/converted/skill-creator/composable/mini/13-agent-comparator.md +204 -0
  215. package/assets/skills/converted/skill-creator/composable/mini/14-agent-analyzer.md +276 -0
  216. package/assets/skills/converted/skill-creator/composable/presets/authoring.md +198 -0
  217. package/assets/skills/converted/skill-creator/composable/presets/evaluating.md +1334 -0
  218. package/assets/skills/converted/skill-creator/composable/presets/optimizing.md +160 -0
  219. package/assets/skills/converted/skill-creator/composable/presets.json +5 -0
  220. package/assets/skills/meta/ccs-skill-creator/composable/BUNDLE.md +465 -0
  221. package/assets/skills/meta/ccs-skill-creator/composable/INDEX.md +8 -0
  222. package/assets/skills/meta/ccs-skill-creator/composable/VERSION +1 -0
  223. package/assets/skills/meta/ccs-skill-creator/composable/mini/00-core.md +46 -0
  224. package/assets/skills/meta/ccs-skill-creator/composable/mini/01-create-new-skill.md +167 -0
  225. package/assets/skills/meta/ccs-skill-creator/composable/mini/02-convert-existing-skill.md +120 -0
  226. package/assets/skills/meta/ccs-skill-creator/composable/mini/03-review-and-maintain.md +112 -0
  227. package/assets/skills/sources/anthropic/PROVENANCE.md +3 -0
  228. package/assets/skills/sources/anthropic/claude-api/LICENSE.txt +202 -0
  229. package/assets/skills/sources/anthropic/docx/LICENSE.txt +30 -0
  230. package/assets/skills/sources/anthropic/internal-comms/LICENSE.txt +202 -0
  231. package/assets/skills/sources/anthropic/mcp-builder/LICENSE.txt +202 -0
  232. package/assets/skills/sources/anthropic/pdf/LICENSE.txt +30 -0
  233. package/assets/skills/sources/anthropic/pptx/LICENSE.txt +30 -0
  234. package/assets/skills/sources/anthropic/skill-creator/LICENSE.txt +202 -0
  235. package/assets/skills/sources/financial-analyst/PROVENANCE.md +1 -0
  236. package/assets/tools/hive.py +728 -0
  237. package/dist/cli.js +7494 -0
  238. package/package.json +40 -0
@@ -0,0 +1,200 @@
1
+ # Node/TypeScript — Advanced Features, Build & Quality Checklist
2
+
3
+ Continues `13-node-implementation.md`. Advanced MCP features, building/running, and the finalization checklist.
4
+
5
+ ## Advanced MCP Features
6
+
7
+ ### Resource Registration
8
+
9
+ Expose data as resources for efficient, URI-based access:
10
+
11
+ ```typescript
12
+ import { ResourceTemplate } from "@modelcontextprotocol/sdk/types.js";
13
+
14
+ // Register a resource with URI template
15
+ server.registerResource(
16
+ {
17
+ uri: "file://documents/{name}",
18
+ name: "Document Resource",
19
+ description: "Access documents by name",
20
+ mimeType: "text/plain"
21
+ },
22
+ async (uri: string) => {
23
+ // Extract parameter from URI
24
+ const match = uri.match(/^file:\/\/documents\/(.+)$/);
25
+ if (!match) {
26
+ throw new Error("Invalid URI format");
27
+ }
28
+
29
+ const documentName = match[1];
30
+ const content = await loadDocument(documentName);
31
+
32
+ return {
33
+ contents: [{
34
+ uri,
35
+ mimeType: "text/plain",
36
+ text: content
37
+ }]
38
+ };
39
+ }
40
+ );
41
+
42
+ // List available resources dynamically
43
+ server.registerResourceList(async () => {
44
+ const documents = await getAvailableDocuments();
45
+ return {
46
+ resources: documents.map(doc => ({
47
+ uri: `file://documents/${doc.name}`,
48
+ name: doc.name,
49
+ mimeType: "text/plain",
50
+ description: doc.description
51
+ }))
52
+ };
53
+ });
54
+ ```
55
+
56
+ **When to use Resources vs Tools:**
57
+ - **Resources:** for data access with simple URI-based parameters
58
+ - **Tools:** for complex operations requiring validation and business logic
59
+ - **Resources:** when data is relatively static or template-based
60
+ - **Tools:** when operations have side effects or complex workflows
61
+
62
+ ### Transport Options
63
+
64
+ The TypeScript SDK supports two main transport mechanisms:
65
+
66
+ #### Streamable HTTP (Recommended for Remote Servers)
67
+
68
+ ```typescript
69
+ import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
70
+ import express from "express";
71
+
72
+ const app = express();
73
+ app.use(express.json());
74
+
75
+ app.post('/mcp', async (req, res) => {
76
+ // Create new transport for each request (stateless, prevents request ID collisions)
77
+ const transport = new StreamableHTTPServerTransport({
78
+ sessionIdGenerator: undefined,
79
+ enableJsonResponse: true
80
+ });
81
+
82
+ res.on('close', () => transport.close());
83
+
84
+ await server.connect(transport);
85
+ await transport.handleRequest(req, res, req.body);
86
+ });
87
+
88
+ app.listen(3000);
89
+ ```
90
+
91
+ #### stdio (For Local Integrations)
92
+
93
+ ```typescript
94
+ import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
95
+
96
+ const transport = new StdioServerTransport();
97
+ await server.connect(transport);
98
+ ```
99
+
100
+ **Transport selection:**
101
+ - **Streamable HTTP:** web services, remote access, multiple clients
102
+ - **stdio:** command-line tools, local development, subprocess integration
103
+
104
+ ### Notification Support
105
+
106
+ Notify clients when server state changes:
107
+
108
+ ```typescript
109
+ // Notify when tools list changes
110
+ server.notification({
111
+ method: "notifications/tools/list_changed"
112
+ });
113
+
114
+ // Notify when resources change
115
+ server.notification({
116
+ method: "notifications/resources/list_changed"
117
+ });
118
+ ```
119
+
120
+ Use notifications sparingly — only when server capabilities genuinely change.
121
+
122
+ ## Code Best Practices
123
+
124
+ ### Code Composability and Reusability
125
+
126
+ Prioritize composability and code reuse (the general rule is in `00-core.md`): extract common functionality into reusable helper functions, shared API clients, centralized error handling, composable business logic, and shared markdown/JSON field-selection & formatting. NEVER copy-paste similar code between tools; if you write similar logic twice, extract it. Share pagination, filtering, field selection, and formatting; centralize authentication/authorization.
127
+
128
+ ## Building and Running
129
+
130
+ Always build your TypeScript code before running:
131
+
132
+ ```bash
133
+ # Build the project
134
+ npm run build
135
+
136
+ # Run the server
137
+ npm start
138
+
139
+ # Development with auto-reload
140
+ npm run dev
141
+ ```
142
+
143
+ Always ensure `npm run build` completes successfully before considering the implementation complete.
144
+
145
+ ## Quality Checklist
146
+
147
+ Before finalizing your Node/TypeScript MCP server implementation, ensure:
148
+
149
+ ### Strategic Design
150
+ - [ ] Tools enable complete workflows, not just API endpoint wrappers
151
+ - [ ] Tool names reflect natural task subdivisions
152
+ - [ ] Response formats optimize for agent context efficiency
153
+ - [ ] Human-readable identifiers used where appropriate
154
+ - [ ] Error messages guide agents toward correct usage
155
+
156
+ ### Implementation Quality
157
+ - [ ] FOCUSED IMPLEMENTATION: Most important and valuable tools implemented
158
+ - [ ] All tools registered using `registerTool` with complete configuration
159
+ - [ ] All tools include `title`, `description`, `inputSchema`, and `annotations`
160
+ - [ ] Annotations correctly set (readOnlyHint, destructiveHint, idempotentHint, openWorldHint)
161
+ - [ ] All tools use Zod schemas for runtime input validation with `.strict()` enforcement
162
+ - [ ] All Zod schemas have proper constraints and descriptive error messages
163
+ - [ ] All tools have comprehensive descriptions with explicit input/output types
164
+ - [ ] Descriptions include return value examples and complete schema documentation
165
+ - [ ] Error messages are clear, actionable, and educational
166
+
167
+ ### TypeScript Quality
168
+ - [ ] TypeScript interfaces are defined for all data structures
169
+ - [ ] Strict TypeScript is enabled in tsconfig.json
170
+ - [ ] No use of `any` type - use `unknown` or proper types instead
171
+ - [ ] All async functions have explicit Promise<T> return types
172
+ - [ ] Error handling uses proper type guards (e.g., `axios.isAxiosError`, `z.ZodError`)
173
+
174
+ ### Advanced Features (where applicable)
175
+ - [ ] Resources registered for appropriate data endpoints
176
+ - [ ] Appropriate transport configured (stdio or streamable HTTP)
177
+ - [ ] Notifications implemented for dynamic server capabilities
178
+ - [ ] Type-safe with SDK interfaces
179
+
180
+ ### Project Configuration
181
+ - [ ] Package.json includes all necessary dependencies
182
+ - [ ] Build script produces working JavaScript in dist/ directory
183
+ - [ ] Main entry point is properly configured as dist/index.js
184
+ - [ ] Server name follows format: `{service}-mcp-server`
185
+ - [ ] tsconfig.json properly configured with strict mode
186
+
187
+ ### Code Quality
188
+ - [ ] Pagination is properly implemented where applicable
189
+ - [ ] Large responses check CHARACTER_LIMIT constant and truncate with clear messages
190
+ - [ ] Filtering options are provided for potentially large result sets
191
+ - [ ] All network operations handle timeouts and connection errors gracefully
192
+ - [ ] Common functionality is extracted into reusable functions
193
+ - [ ] Return types are consistent across similar operations
194
+
195
+ ### Testing and Build
196
+ - [ ] `npm run build` completes successfully without errors
197
+ - [ ] dist/index.js created and executable
198
+ - [ ] Server runs: `node dist/index.js --help`
199
+ - [ ] All imports resolve correctly
200
+ - [ ] Sample tool calls work as expected
@@ -0,0 +1,200 @@
1
+ # Evaluations — Designing Questions (Phase 4)
2
+
3
+ Create comprehensive evaluations that test whether LLMs can effectively use your MCP server to answer realistic, complex questions using ONLY the tools provided. Running the evaluation harness is covered in `16-evaluation-running.md`.
4
+
5
+ ## Purpose
6
+
7
+ The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs — with no other context and access ONLY to the MCP server — to answer realistic and difficult questions.
8
+
9
+ ## Overview & Requirements
10
+
11
+ Create **10 human-readable questions** requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be:
12
+ - Realistic
13
+ - Clear and concise
14
+ - Unambiguous
15
+ - Complex, requiring potentially dozens of tool calls or steps
16
+ - Answerable with a single, verifiable value that you identify in advance
17
+
18
+ The four-step creation process (referenced from the workflow): (1) **Tool Inspection** — list available tools and understand their capabilities; (2) **Content Exploration** — use READ-ONLY operations to explore available data; (3) **Question Generation** — create 10 complex, realistic questions; (4) **Answer Verification** — solve each question yourself to verify answers.
19
+
20
+ ## Question Guidelines
21
+
22
+ ### Core Requirements
23
+
24
+ 1. **Questions MUST be independent** — each should NOT depend on the answer to any other question, and should not assume prior write operations from processing another question.
25
+ 2. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use** — should not instruct or require modifying state to arrive at the correct answer.
26
+ 3. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX** — must require another LLM to use multiple (potentially dozens of) tools or steps to answer.
27
+
28
+ ### Complexity and Depth
29
+
30
+ 4. **Questions must require deep exploration** — consider multi-hop questions requiring multiple sub-questions and sequential tool calls; each step should benefit from information found in previous steps.
31
+ 5. **Questions may require extensive paging** — may need paging through multiple pages of results; may require querying old data (1–2 years out-of-date) to find niche information; the questions must be DIFFICULT.
32
+ 6. **Questions must require deep understanding** — rather than surface-level knowledge; may pose complex ideas as True/False questions requiring evidence; may use multiple-choice format where the LLM must search different hypotheses.
33
+ 7. **Questions must not be solvable with straightforward keyword search** — do not include specific keywords from the target content; use synonyms, related concepts, or paraphrases; require multiple searches, analyzing multiple related items, extracting context, then deriving the answer.
34
+
35
+ ### Tool Testing
36
+
37
+ 8. **Questions should stress-test tool return values** — may elicit tools returning large JSON objects or lists that overwhelm the LLM; should require understanding multiple modalities of data (IDs and names; timestamps and datetimes — months, days, years, seconds; file IDs, names, extensions, mimetypes; URLs, GIDs, etc.); should probe the tool's ability to return all useful forms of data.
38
+ 9. **Questions should MOSTLY reflect real human use cases** — the kinds of information-retrieval tasks that humans assisted by an LLM would care about.
39
+ 10. **Questions may require dozens of tool calls** — this challenges LLMs with limited context and encourages MCP server tools to reduce information returned.
40
+ 11. **Include ambiguous questions** — may be ambiguous OR require difficult decisions on which tools to call; force the LLM to potentially make mistakes or misinterpret; ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER.
41
+
42
+ ### Stability
43
+
44
+ 12. **Questions must be designed so the answer DOES NOT CHANGE** — do not ask questions that rely on dynamic "current state." For example, do not count: number of reactions to a post, number of replies to a thread, number of members in a channel.
45
+ 13. **DO NOT let the MCP server RESTRICT the kinds of questions you create** — create challenging and complex questions; some may not be solvable with the available tools; questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN); questions may require dozens of tool calls.
46
+
47
+ ## Answer Guidelines
48
+
49
+ ### Verification
50
+
51
+ 1. **Answers must be VERIFIABLE via direct string comparison.** If the answer can be written in many formats, clearly specify the output format in the QUESTION (e.g., "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else."). Answer should be a single verifiable value such as: user ID, user name, display name, first name, last name; channel ID, channel name; message ID, string; URL, title; numerical quantity; timestamp, datetime; boolean (True/False); email address, phone number; file ID, file name, file extension; multiple choice answer. Answers must not require special formatting or complex, structured output. Answer will be verified using DIRECT STRING COMPARISON.
52
+
53
+ ### Readability
54
+
55
+ 2. **Answers should generally prefer HUMAN-READABLE formats** — names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d — rather than opaque IDs (though IDs are acceptable). The VAST MAJORITY of answers should be human-readable.
56
+
57
+ ### Stability
58
+
59
+ 3. **Answers must be STABLE/STATIONARY** — look at old content (conversations that have ended, projects that have launched, questions answered); create questions based on "closed" concepts that will always return the same answer; questions may ask to consider a fixed time window to insulate from non-stationary answers; rely on context UNLIKELY to change. Example: if finding a paper name, be SPECIFIC enough so the answer is not confused with papers published later.
60
+ 4. **Answers must be CLEAR and UNAMBIGUOUS** — designed so there is a single, clear answer derivable from the MCP server tools.
61
+
62
+ ### Diversity
63
+
64
+ 5. **Answers must be DIVERSE** — a single verifiable value in diverse modalities and formats. User concept: user ID, user name, display name, first name, last name, email address, phone number. Channel concept: channel ID, channel name, channel topic. Message concept: message ID, message string, timestamp, month, day, year.
65
+ 6. **Answers must NOT be complex structures** — not a list of values, not a complex object, not a list of IDs or strings, not natural language text — UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON and can be realistically reproduced (unlikely that an LLM would return the same list in any other order or format).
66
+
67
+ ## Evaluation Process
68
+
69
+ 1. **Documentation Inspection** — read the target API docs to understand available endpoints and functionality; if ambiguity exists, fetch additional info from the web; parallelize as much as possible; ensure each subagent is ONLY examining documentation from the file system or web.
70
+ 2. **Tool Inspection** — list the tools available in the MCP server; inspect input/output schemas, docstrings, and descriptions WITHOUT calling the tools at this stage.
71
+ 3. **Developing Understanding** — repeat steps 1 & 2, iterating multiple times; think about the tasks you want to create and refine understanding; at NO stage READ the code of the MCP server implementation itself; use intuition to create reasonable, realistic, but VERY challenging tasks.
72
+ 4. **Read-Only Content Inspection** — USE the MCP server tools with READ-ONLY, NON-DESTRUCTIVE operations ONLY to identify specific content (users, channels, messages, projects, tasks) for realistic questions. Do NOT call tools that modify state. Do NOT read the server's code. Parallelize with individual sub-agents pursuing independent explorations (each performing only read-only, non-destructive, idempotent operations). BE CAREFUL: some tools may return LOTS OF DATA and exhaust context — make INCREMENTAL, SMALL, TARGETED calls; in all tool call requests use the `limit` parameter (<10); use pagination.
73
+ 5. **Task Generation** — create 10 human-readable questions an LLM can answer with the MCP server, following all question and answer guidelines above.
74
+
75
+ ## Output Format
76
+
77
+ Each QA pair consists of a question and an answer, in an XML file:
78
+
79
+ ```xml
80
+ <evaluation>
81
+ <qa_pair>
82
+ <question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
83
+ <answer>Website Redesign</answer>
84
+ </qa_pair>
85
+ <qa_pair>
86
+ <question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
87
+ <answer>sarah_dev</answer>
88
+ </qa_pair>
89
+ <qa_pair>
90
+ <question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question>
91
+ <answer>7</answer>
92
+ </qa_pair>
93
+ <qa_pair>
94
+ <question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
95
+ <answer>data-pipeline</answer>
96
+ </qa_pair>
97
+ </evaluation>
98
+ ```
99
+
100
+ A single-pair illustration of the same structure:
101
+
102
+ ```xml
103
+ <evaluation>
104
+ <qa_pair>
105
+ <question>Find discussions about AI model launches with animal codenames. One model needed a specific safety designation that uses the format ASL-X. What number X was being determined for the model named after a spotted wild cat?</question>
106
+ <answer>3</answer>
107
+ </qa_pair>
108
+ <!-- More qa_pairs... -->
109
+ </evaluation>
110
+ ```
111
+
112
+ ## Examples
113
+
114
+ ### Good Questions
115
+
116
+ **Example 1 — Multi-hop question requiring deep exploration (GitHub MCP):**
117
+ ```xml
118
+ <qa_pair>
119
+ <question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question>
120
+ <answer>Python</answer>
121
+ </qa_pair>
122
+ ```
123
+ Good because: requires multiple searches to find archived repositories; needs to identify which had the most forks before archival; requires examining repository details for the language; answer is a simple verifiable value; based on historical (closed) data that won't change.
124
+
125
+ **Example 2 — Requires understanding context without keyword matching (Project Management MCP):**
126
+ ```xml
127
+ <qa_pair>
128
+ <question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question>
129
+ <answer>Product Manager</answer>
130
+ </qa_pair>
131
+ ```
132
+ Good because: doesn't use the specific project name; requires finding completed projects from a specific timeframe; needs to identify the project lead and their role; requires understanding context from retrospective documents; answer is human-readable and stable; based on completed work.
133
+
134
+ **Example 3 — Complex aggregation requiring multiple steps (Issue Tracker MCP):**
135
+ ```xml
136
+ <qa_pair>
137
+ <question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question>
138
+ <answer>alex_eng</answer>
139
+ </qa_pair>
140
+ ```
141
+ Good because: requires filtering bugs by date, priority, and status; needs to group by assignee and calculate resolution rates; requires understanding timestamps to determine 48-hour windows; tests pagination; answer is a single username; based on historical data.
142
+
143
+ **Example 4 — Requires synthesis across multiple data types (CRM MCP):**
144
+ ```xml
145
+ <qa_pair>
146
+ <question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question>
147
+ <answer>Healthcare</answer>
148
+ </qa_pair>
149
+ ```
150
+ Good because: requires understanding subscription tier changes; needs to identify upgrade events in a specific timeframe; requires comparing contract values; must access account industry information; answer is simple and verifiable; based on completed historical transactions.
151
+
152
+ ### Poor Questions
153
+
154
+ **Example 1 — Answer changes over time:**
155
+ ```xml
156
+ <qa_pair>
157
+ <question>How many open issues are currently assigned to the engineering team?</question>
158
+ <answer>47</answer>
159
+ </qa_pair>
160
+ ```
161
+ Poor because: the answer changes as issues are created, closed, or reassigned; not based on stable data; relies on dynamic "current state."
162
+
163
+ **Example 2 — Too easy with keyword search:**
164
+ ```xml
165
+ <qa_pair>
166
+ <question>Find the pull request with title "Add authentication feature" and tell me who created it.</question>
167
+ <answer>developer123</answer>
168
+ </qa_pair>
169
+ ```
170
+ Poor because: solvable with a straightforward keyword search for the exact title; doesn't require deep exploration or understanding; no synthesis or analysis needed.
171
+
172
+ **Example 3 — Ambiguous answer format:**
173
+ ```xml
174
+ <qa_pair>
175
+ <question>List all the repositories that have Python as their primary language.</question>
176
+ <answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer>
177
+ </qa_pair>
178
+ ```
179
+ Poor because: the answer is a list that could be returned in any order; difficult to verify with direct string comparison; an LLM might format differently (JSON array, comma-separated, newline-separated); better to ask for a specific aggregate (count) or superlative (most stars).
180
+
181
+ ## Verification Process
182
+
183
+ After creating evaluations:
184
+ 1. **Examine the XML file** to understand the schema.
185
+ 2. **Load each task instruction** and, in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF.
186
+ 3. **Flag any operations** that require WRITE or DESTRUCTIVE operations.
187
+ 4. **Accumulate all CORRECT answers** and replace any incorrect answers in the document.
188
+ 5. **Remove any `<qa_pair>`** that requires WRITE or DESTRUCTIVE operations.
189
+
190
+ Parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end.
191
+
192
+ ## Tips for Quality Evaluations
193
+
194
+ 1. **Think Hard and Plan Ahead** before generating tasks.
195
+ 2. **Parallelize Where Opportunity Arises** to speed up the process and manage context.
196
+ 3. **Focus on Realistic Use Cases** that humans would actually want to accomplish.
197
+ 4. **Create Challenging Questions** that test the limits of the MCP server's capabilities.
198
+ 5. **Ensure Stability** by using historical data and closed concepts.
199
+ 6. **Verify Answers** by solving the questions yourself using the MCP server tools.
200
+ 7. **Iterate and Refine** based on what you learn during the process.
@@ -0,0 +1,198 @@
1
+ # Evaluations — Running the Harness
2
+
3
+ After creating your evaluation file (see `15-evaluation-design.md`), use the provided evaluation harness (`scripts/evaluation.py`) to test your MCP server.
4
+
5
+ ## Setup
6
+
7
+ 1. **Install Dependencies**
8
+ ```bash
9
+ pip install -r scripts/requirements.txt
10
+ ```
11
+ Or install manually:
12
+ ```bash
13
+ pip install anthropic mcp
14
+ ```
15
+
16
+ 2. **Set API Key**
17
+ ```bash
18
+ export ANTHROPIC_API_KEY=your_api_key_here
19
+ ```
20
+
21
+ ## Evaluation File Format
22
+
23
+ Evaluation files use XML format with `<qa_pair>` elements:
24
+
25
+ ```xml
26
+ <evaluation>
27
+ <qa_pair>
28
+ <question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question>
29
+ <answer>Website Redesign</answer>
30
+ </qa_pair>
31
+ <qa_pair>
32
+ <question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
33
+ <answer>sarah_dev</answer>
34
+ </qa_pair>
35
+ </evaluation>
36
+ ```
37
+
38
+ ## Running Evaluations
39
+
40
+ The evaluation script (`scripts/evaluation.py`) supports three transport types.
41
+
42
+ **Important:**
43
+ - **stdio transport:** the evaluation script automatically launches and manages the MCP server process for you. Do not run the server manually.
44
+ - **sse/http transports:** you must start the MCP server separately before running the evaluation. The script connects to the already-running server at the specified URL.
45
+
46
+ ### 1. Local STDIO Server
47
+
48
+ For locally-run MCP servers (script launches the server automatically):
49
+
50
+ ```bash
51
+ python scripts/evaluation.py \
52
+ -t stdio \
53
+ -c python \
54
+ -a my_mcp_server.py \
55
+ evaluation.xml
56
+ ```
57
+
58
+ With environment variables:
59
+ ```bash
60
+ python scripts/evaluation.py \
61
+ -t stdio \
62
+ -c python \
63
+ -a my_mcp_server.py \
64
+ -e API_KEY=abc123 \
65
+ -e DEBUG=true \
66
+ evaluation.xml
67
+ ```
68
+
69
+ ### 2. Server-Sent Events (SSE)
70
+
71
+ For SSE-based MCP servers (you must start the server first):
72
+
73
+ ```bash
74
+ python scripts/evaluation.py \
75
+ -t sse \
76
+ -u https://example.com/mcp \
77
+ -H "Authorization: Bearer token123" \
78
+ -H "X-Custom-Header: value" \
79
+ evaluation.xml
80
+ ```
81
+
82
+ ### 3. HTTP (Streamable HTTP)
83
+
84
+ For HTTP-based MCP servers (you must start the server first):
85
+
86
+ ```bash
87
+ python scripts/evaluation.py \
88
+ -t http \
89
+ -u https://example.com/mcp \
90
+ -H "Authorization: Bearer token123" \
91
+ evaluation.xml
92
+ ```
93
+
94
+ ## Command-Line Options
95
+
96
+ ```
97
+ usage: evaluation.py [-h] [-t {stdio,sse,http}] [-m MODEL] [-c COMMAND]
98
+ [-a ARGS [ARGS ...]] [-e ENV [ENV ...]] [-u URL]
99
+ [-H HEADERS [HEADERS ...]] [-o OUTPUT]
100
+ eval_file
101
+
102
+ positional arguments:
103
+ eval_file Path to evaluation XML file
104
+
105
+ optional arguments:
106
+ -h, --help Show help message
107
+ -t, --transport Transport type: stdio, sse, or http (default: stdio)
108
+ -m, --model Claude model to use (default: claude-3-7-sonnet-20250219)
109
+ -o, --output Output file for report (default: print to stdout)
110
+
111
+ stdio options:
112
+ -c, --command Command to run MCP server (e.g., python, node)
113
+ -a, --args Arguments for the command (e.g., server.py)
114
+ -e, --env Environment variables in KEY=VALUE format
115
+
116
+ sse/http options:
117
+ -u, --url MCP server URL
118
+ -H, --header HTTP headers in 'Key: Value' format
119
+ ```
120
+
121
+ ## Output
122
+
123
+ The evaluation script generates a detailed report including:
124
+
125
+ - **Summary Statistics:** Accuracy (correct/total); Average task duration; Average tool calls per task; Total tool calls.
126
+ - **Per-Task Results:** Prompt and expected response; Actual response from the agent; Whether the answer was correct (✅/❌); Duration and tool call details; Agent's summary of its approach; Agent's feedback on the tools.
127
+
128
+ ### Save Report to File
129
+
130
+ ```bash
131
+ python scripts/evaluation.py \
132
+ -t stdio \
133
+ -c python \
134
+ -a my_server.py \
135
+ -o evaluation_report.md \
136
+ evaluation.xml
137
+ ```
138
+
139
+ ## Complete Example Workflow
140
+
141
+ 1. **Create your evaluation file** (`my_evaluation.xml`):
142
+ ```xml
143
+ <evaluation>
144
+ <qa_pair>
145
+ <question>Find the user who created the most issues in January 2024. What is their username?</question>
146
+ <answer>alice_developer</answer>
147
+ </qa_pair>
148
+ <qa_pair>
149
+ <question>Among all pull requests merged in Q1 2024, which repository had the highest number? Provide the repository name.</question>
150
+ <answer>backend-api</answer>
151
+ </qa_pair>
152
+ <qa_pair>
153
+ <question>Find the project that was completed in December 2023 and had the longest duration from start to finish. How many days did it take?</question>
154
+ <answer>127</answer>
155
+ </qa_pair>
156
+ </evaluation>
157
+ ```
158
+
159
+ 2. **Install dependencies**:
160
+ ```bash
161
+ pip install -r scripts/requirements.txt
162
+ export ANTHROPIC_API_KEY=your_api_key
163
+ ```
164
+
165
+ 3. **Run evaluation**:
166
+ ```bash
167
+ python scripts/evaluation.py \
168
+ -t stdio \
169
+ -c python \
170
+ -a github_mcp_server.py \
171
+ -e GITHUB_TOKEN=ghp_xxx \
172
+ -o github_eval_report.md \
173
+ my_evaluation.xml
174
+ ```
175
+
176
+ 4. **Review the report** in `github_eval_report.md` to: see which questions passed/failed; read the agent's feedback on your tools; identify areas for improvement; iterate on your MCP server design.
177
+
178
+ ## Troubleshooting
179
+
180
+ ### Connection Errors
181
+ - **STDIO:** verify the command and arguments are correct.
182
+ - **SSE/HTTP:** check the URL is accessible and headers are correct.
183
+ - Ensure any required API keys are set in environment variables or headers.
184
+
185
+ ### Low Accuracy
186
+ If many evaluations fail:
187
+ - Review the agent's feedback for each task.
188
+ - Check if tool descriptions are clear and comprehensive.
189
+ - Verify input parameters are well-documented.
190
+ - Consider whether tools return too much or too little data.
191
+ - Ensure error messages are actionable.
192
+
193
+ ### Timeout Issues
194
+ If tasks are timing out:
195
+ - Use a more capable model (e.g., `claude-3-7-sonnet-20250219`).
196
+ - Check if tools are returning too much data.
197
+ - Verify pagination is working correctly.
198
+ - Consider simplifying complex questions.