nexakit 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +28 -0
- package/LICENSE +26 -0
- package/README.md +79 -2
- package/agents/claude/docs-researcher.md +9 -0
- package/agents/claude/implementation-tutor.md +9 -0
- package/agents/claude/project-explorer.md +9 -0
- package/agents/claude/quality-reviewer.md +9 -0
- package/agents/claude/requirements-analyst.md +9 -0
- package/agents/claude/solution-architect.md +9 -0
- package/agents/codex/docs-researcher.toml +10 -0
- package/agents/codex/implementation-tutor.toml +10 -0
- package/agents/codex/project-explorer.toml +10 -0
- package/agents/codex/quality-reviewer.toml +10 -0
- package/agents/codex/requirements-analyst.toml +11 -0
- package/agents/codex/solution-architect.toml +10 -0
- package/assets/brand/README.md +38 -0
- package/assets/brand/agent-toolkit-mark.png +0 -0
- package/assets/brand/agent-toolkit-mark.svg +13 -0
- package/assets/brand/agent-toolkit-tui-hero.png +0 -0
- package/assets/brand/nexakit-readme-hero.png +0 -0
- package/assets/brand/nexakit-readme-hero.svg +21 -0
- package/assets/runtime/package-lock.json +1425 -0
- package/bin/agent-task.js +14 -0
- package/bin/agent-toolkit-gemini.js +13 -0
- package/bin/agent-toolkit-inbox.js +14 -0
- package/bin/agent-toolkit-mcp.js +17 -0
- package/bin/agent-toolkit.js +13 -0
- package/bin/nexakit.js +13 -0
- package/bin/project-memory-mcp.js +22 -0
- package/bin/project-memory.js +7 -0
- package/catalogs/skills/api-design-principles/LICENSE.txt +21 -0
- package/catalogs/skills/api-design-principles/SKILL.md +110 -0
- package/catalogs/skills/api-design-principles/assets/api-design-checklist.md +155 -0
- package/catalogs/skills/api-design-principles/assets/rest-api-template.py +185 -0
- package/catalogs/skills/api-design-principles/references/details.md +413 -0
- package/catalogs/skills/api-design-principles/references/graphql-schema-design.md +589 -0
- package/catalogs/skills/api-design-principles/references/rest-best-practices.md +408 -0
- package/catalogs/skills/nodejs-backend-patterns/LICENSE.txt +21 -0
- package/catalogs/skills/nodejs-backend-patterns/SKILL.md +45 -0
- package/catalogs/skills/nodejs-backend-patterns/references/advanced-patterns.md +430 -0
- package/catalogs/skills/nodejs-backend-patterns/references/details.md +599 -0
- package/catalogs/skills/supabase-postgres-best-practices/CHANGELOG.md +73 -0
- package/catalogs/skills/supabase-postgres-best-practices/LICENSE.txt +21 -0
- package/catalogs/skills/supabase-postgres-best-practices/SKILL.md +64 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/_contributing.md +170 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/_sections.md +39 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/_template.md +34 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/advanced-full-text-search.md +55 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/advanced-jsonb-indexing.md +49 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/conn-idle-timeout.md +46 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/conn-limits.md +44 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/conn-pooling.md +41 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/conn-prepared-statements.md +46 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/data-batch-inserts.md +54 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/data-n-plus-one.md +53 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/data-pagination.md +50 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/data-upsert.md +50 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/lock-advisory.md +56 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/lock-deadlock-prevention.md +68 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/lock-short-transactions.md +50 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/lock-skip-locked.md +54 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/monitor-explain-analyze.md +45 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/monitor-pg-stat-statements.md +55 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/monitor-vacuum-analyze.md +55 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/query-composite-indexes.md +44 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/query-covering-indexes.md +40 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/query-index-types.md +48 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/query-missing-indexes.md +43 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/query-partial-indexes.md +45 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-constraints.md +80 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-data-types.md +46 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-foreign-key-indexes.md +59 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-lowercase-identifiers.md +55 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-partitioning.md +55 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/schema-primary-keys.md +61 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/security-privileges.md +54 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/security-rls-basics.md +50 -0
- package/catalogs/skills/supabase-postgres-best-practices/references/security-rls-performance.md +63 -0
- package/catalogs/skills/vercel-composition-patterns/AGENTS.md +946 -0
- package/catalogs/skills/vercel-composition-patterns/LICENSE.txt +27 -0
- package/catalogs/skills/vercel-composition-patterns/README.md +60 -0
- package/catalogs/skills/vercel-composition-patterns/SKILL.md +89 -0
- package/catalogs/skills/vercel-composition-patterns/metadata.json +11 -0
- package/catalogs/skills/vercel-composition-patterns/rules/_sections.md +29 -0
- package/catalogs/skills/vercel-composition-patterns/rules/_template.md +24 -0
- package/catalogs/skills/vercel-composition-patterns/rules/architecture-avoid-boolean-props.md +100 -0
- package/catalogs/skills/vercel-composition-patterns/rules/architecture-compound-components.md +112 -0
- package/catalogs/skills/vercel-composition-patterns/rules/patterns-children-over-render-props.md +87 -0
- package/catalogs/skills/vercel-composition-patterns/rules/patterns-explicit-variants.md +100 -0
- package/catalogs/skills/vercel-composition-patterns/rules/react19-no-forwardref.md +42 -0
- package/catalogs/skills/vercel-composition-patterns/rules/state-context-interface.md +191 -0
- package/catalogs/skills/vercel-composition-patterns/rules/state-decouple-implementation.md +113 -0
- package/catalogs/skills/vercel-composition-patterns/rules/state-lift-state.md +125 -0
- package/catalogs/skills/vercel-react-best-practices/AGENTS.md +3810 -0
- package/catalogs/skills/vercel-react-best-practices/LICENSE.txt +27 -0
- package/catalogs/skills/vercel-react-best-practices/README.md +123 -0
- package/catalogs/skills/vercel-react-best-practices/SKILL.md +149 -0
- package/catalogs/skills/vercel-react-best-practices/metadata.json +15 -0
- package/catalogs/skills/vercel-react-best-practices/rules/_sections.md +46 -0
- package/catalogs/skills/vercel-react-best-practices/rules/_template.md +28 -0
- package/catalogs/skills/vercel-react-best-practices/rules/advanced-effect-event-deps.md +56 -0
- package/catalogs/skills/vercel-react-best-practices/rules/advanced-event-handler-refs.md +55 -0
- package/catalogs/skills/vercel-react-best-practices/rules/advanced-init-once.md +42 -0
- package/catalogs/skills/vercel-react-best-practices/rules/advanced-use-latest.md +39 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-api-routes.md +38 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-cheap-condition-before-await.md +37 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-defer-await.md +82 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-dependencies.md +51 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-parallel.md +28 -0
- package/catalogs/skills/vercel-react-best-practices/rules/async-suspense-boundaries.md +99 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-analyzable-paths.md +63 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-barrel-imports.md +60 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-conditional.md +31 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-defer-third-party.md +49 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-dynamic-imports.md +35 -0
- package/catalogs/skills/vercel-react-best-practices/rules/bundle-preload.md +50 -0
- package/catalogs/skills/vercel-react-best-practices/rules/client-event-listeners.md +74 -0
- package/catalogs/skills/vercel-react-best-practices/rules/client-localstorage-schema.md +71 -0
- package/catalogs/skills/vercel-react-best-practices/rules/client-passive-event-listeners.md +48 -0
- package/catalogs/skills/vercel-react-best-practices/rules/client-swr-dedup.md +56 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-batch-dom-css.md +107 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-cache-function-results.md +80 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-cache-property-access.md +28 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-cache-storage.md +70 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-combine-iterations.md +32 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-early-exit.md +50 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-flatmap-filter.md +60 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-hoist-regexp.md +45 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-index-maps.md +37 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-length-check-first.md +49 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-min-max-loop.md +82 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-request-idle-callback.md +105 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-set-map-lookups.md +24 -0
- package/catalogs/skills/vercel-react-best-practices/rules/js-tosorted-immutable.md +57 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-activity.md +26 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-animate-svg-wrapper.md +47 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-conditional-render.md +40 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-content-visibility.md +38 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-hoist-jsx.md +46 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-hydration-no-flicker.md +82 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-hydration-suppress-warning.md +30 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-resource-hints.md +85 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-script-defer-async.md +68 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-svg-precision.md +28 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rendering-usetransition-loading.md +75 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-defer-reads.md +39 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-dependencies.md +45 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-derived-state-no-effect.md +40 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-derived-state.md +29 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-functional-setstate.md +74 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-lazy-state-init.md +58 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-memo-with-default-value.md +38 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-memo.md +44 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-move-effect-to-event.md +45 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-no-inline-components.md +82 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-simple-expression-in-memo.md +35 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-split-combined-hooks.md +64 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-transitions.md +40 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-use-deferred-value.md +59 -0
- package/catalogs/skills/vercel-react-best-practices/rules/rerender-use-ref-transient-values.md +73 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-after-nonblocking.md +73 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-auth-actions.md +96 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-cache-lru.md +41 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-cache-react.md +76 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-dedup-props.md +65 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-hoist-static-io.md +149 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-no-shared-module-state.md +50 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-parallel-fetching.md +83 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-parallel-nested-fetching.md +34 -0
- package/catalogs/skills/vercel-react-best-practices/rules/server-serialization.md +38 -0
- package/catalogs/skills/webapp-testing/LICENSE.txt +202 -0
- package/catalogs/skills/webapp-testing/SKILL.md +96 -0
- package/catalogs/skills/webapp-testing/examples/console_logging.py +35 -0
- package/catalogs/skills/webapp-testing/examples/element_discovery.py +40 -0
- package/catalogs/skills/webapp-testing/examples/static_html_automation.py +33 -0
- package/catalogs/skills/webapp-testing/scripts/with_server.py +106 -0
- package/catalogs/skills.json +1115 -0
- package/docs/skills/evidencias/2026-10-28-evaluacion-inicial-mcp-seguridad.md +118 -0
- package/docs/skills/evidencias/2026-10-28-incorporacion-mcp-seguridad.json +152 -0
- package/docs/skills/evidencias/2026-10-28-recomendador-preparacion.md +50 -0
- package/package.json +73 -4
- package/plugins/nexakit-modelo/.claude-plugin/plugin.json +1 -0
- package/plugins/nexakit-modelo/hooks/hooks.json +1 -0
- package/plugins/nexakit-modelo/hooks/register.js +48 -0
- package/plugins/nexakit-modelo/hooks/status.js +8 -0
- package/plugins/nexakit-modelo/tests/register.test.ts +21 -0
- package/plugins/nexakit-panel/.claude-plugin/plugin.json +1 -0
- package/plugins/nexakit-panel/README.md +20 -0
- package/plugins/nexakit-panel/hooks/hooks.json +1 -0
- package/plugins/nexakit-panel/hooks/lienzo.js +382 -0
- package/plugins/nexakit-panel/hooks/panel.js +279 -0
- package/plugins/nexakit-panel/hooks/register.tsx +248 -0
- package/plugins/nexakit-panel/tests/lienzo.test.ts +142 -0
- package/plugins/nexakit-panel/tests/panel.test.ts +240 -0
- package/plugins/nexakit-panel/tests/register.test.ts +167 -0
- package/plugins/nexakit-panel/types/index.d.ts +41 -0
- package/plugins/nexakit-sesion/.claude-plugin/plugin.json +1 -0
- package/plugins/nexakit-sesion/README.md +12 -0
- package/plugins/nexakit-sesion/hooks/aviso.js +38 -0
- package/plugins/nexakit-sesion/hooks/hooks.json +1 -0
- package/plugins/nexakit-sesion/hooks/register.js +28 -0
- package/plugins/nexakit-sesion/tests/register.test.ts +96 -0
- package/plugins/niveles/.claude-plugin/plugin.json +5 -0
- package/plugins/niveles/agents/nivel-alto.md +13 -0
- package/plugins/niveles/agents/nivel-exigente.md +13 -0
- package/plugins/niveles/agents/nivel-ligero.md +13 -0
- package/plugins/niveles/agents/nivel-normal.md +13 -0
- package/schemas/agent-client-adapter.schema.json +786 -0
- package/schemas/agent-client-auth-diagnostic.schema.json +801 -0
- package/schemas/agent-client-isolated-variant.schema.json +105 -0
- package/schemas/agent-client-isolated-verification.schema.json +99 -0
- package/schemas/agent-client-managed-preferences-invocation.schema.json +67 -0
- package/schemas/agent-client-managed-preferences.schema.json +63 -0
- package/schemas/agent-client-native-diagnostics.schema.json +85 -0
- package/schemas/agent-client-native-evaluation.schema.json +171 -0
- package/schemas/agent-client-native-preflight.schema.json +90 -0
- package/schemas/agent-client-output.schema.json +551 -0
- package/schemas/agent-client-runtime-fixture.schema.json +90 -0
- package/schemas/agent-client-runtime-local.schema.json +95 -0
- package/schemas/agent-client-runtime.schema.json +1288 -0
- package/schemas/agent-client-selective-config.schema.json +77 -0
- package/schemas/agent-client-supervised-local.schema.json +135 -0
- package/schemas/agent-client-supervised.schema.json +186 -0
- package/schemas/agent-model-preference.schema.json +573 -0
- package/schemas/agent-model-selection.schema.json +311 -0
- package/schemas/agent-task-approvals.schema.json +362 -0
- package/schemas/agent-task-prepare.schema.json +1324 -0
- package/schemas/agent-task-verify.schema.json +198 -0
- package/schemas/project-memory-update.schema.json +33 -0
- package/schemas/project-memory.schema.json +298 -0
- package/schemas/skill-catalog.schema.json +246 -0
- package/schemas/task-workflow-plan-output.schema.json +35 -0
- package/scripts/prepare-package.js +6 -0
- package/scripts/validate-skill-catalog.js +41 -0
- package/skills/cerrar-avisens/SKILL.md +12 -0
- package/skills/cerrar-memoria/SKILL.md +96 -0
- package/skills/cerrar-memoria/references/cierre.md +76 -0
- package/skills/decision-records/SKILL.md +27 -0
- package/skills/decision-records/agents/openai.yaml +7 -0
- package/skills/decision-records/references/adr-template.md +35 -0
- package/skills/differential-review/LICENSE.txt +427 -0
- package/skills/differential-review/SKILL.md +225 -0
- package/skills/differential-review/adversarial.md +203 -0
- package/skills/differential-review/agents/openai.yaml +6 -0
- package/skills/differential-review/assets/trail-of-bits-mark.svg +1 -0
- package/skills/differential-review/methodology.md +234 -0
- package/skills/differential-review/patterns.md +300 -0
- package/skills/differential-review/reporting.md +369 -0
- package/skills/elegir-modelo/SKILL.md +68 -0
- package/skills/elegir-modelo/modelos.mjs +108 -0
- package/skills/frontend-design/LICENSE.txt +177 -0
- package/skills/frontend-design/SKILL.md +71 -0
- package/skills/gemini/SKILL.md +52 -0
- package/skills/imagen/SKILL.md +12 -0
- package/skills/investigar/SKILL.md +11 -0
- package/skills/mcp-builder/LICENSE.txt +202 -0
- package/skills/mcp-builder/SKILL.md +236 -0
- package/skills/mcp-builder/reference/evaluation.md +602 -0
- package/skills/mcp-builder/reference/mcp_best_practices.md +249 -0
- package/skills/mcp-builder/reference/node_mcp_server.md +970 -0
- package/skills/mcp-builder/reference/python_mcp_server.md +719 -0
- package/skills/mcp-builder/scripts/connections.py +151 -0
- package/skills/mcp-builder/scripts/evaluation.py +373 -0
- package/skills/mcp-builder/scripts/example_evaluation.xml +22 -0
- package/skills/mcp-builder/scripts/requirements.txt +2 -0
- package/skills/ordenar-proyecto/SKILL.md +56 -0
- package/skills/project-handoff/SKILL.md +53 -0
- package/skills/project-handoff/agents/openai.yaml +7 -0
- package/skills/project-handoff/references/handoff-template.md +35 -0
- package/skills/quality-gate/LICENSE.txt +26 -0
- package/skills/quality-gate/SKILL.md +44 -0
- package/skills/quality-gate/agents/openai.yaml +7 -0
- package/skills/quality-gate/references/review-checklist.md +42 -0
- package/skills/revisar/SKILL.md +11 -0
- package/skills/revisar-skill/LICENSE-find-skills.txt +26 -0
- package/skills/revisar-skill/SKILL.md +94 -0
- package/skills/revision-cruzada/SKILL.md +59 -0
- package/skills/web-design-guidelines/AGENTS.md +156 -0
- package/skills/web-design-guidelines/LICENSE.txt +21 -0
- package/skills/web-design-guidelines/SKILL.md +29 -0
- package/src/agent-models.js +131 -0
- package/src/brand.js +117 -0
- package/src/claude-code-runner.js +642 -0
- package/src/cli-executable.js +88 -0
- package/src/cli.js +778 -0
- package/src/client-adapter-plan.js +271 -0
- package/src/client-auth-diagnostic-parser.js +211 -0
- package/src/client-auth-diagnostic.js +257 -0
- package/src/client-isolated-variant.js +106 -0
- package/src/client-isolated-verification.js +130 -0
- package/src/client-local-platform.js +14 -0
- package/src/client-managed-preferences-invocation.js +209 -0
- package/src/client-managed-preferences.js +78 -0
- package/src/client-native-access.js +134 -0
- package/src/client-native-diagnostics.js +253 -0
- package/src/client-native-evaluation.js +106 -0
- package/src/client-native-preflight-files.js +68 -0
- package/src/client-native-preflight.js +95 -0
- package/src/client-output-json.js +118 -0
- package/src/client-output-parser.js +278 -0
- package/src/client-runtime-admission.js +80 -0
- package/src/client-runtime-coordinator.js +70 -0
- package/src/client-runtime-core.js +92 -0
- package/src/client-runtime-fixture-permit.js +20 -0
- package/src/client-runtime-fixture-protocol.js +39 -0
- package/src/client-runtime-state.js +67 -0
- package/src/client-runtime-store-files.js +91 -0
- package/src/client-runtime-store.js +239 -0
- package/src/client-runtime-supervisor.js +146 -0
- package/src/client-selective-config.js +195 -0
- package/src/client-selective-files.js +126 -0
- package/src/client-supervised-codes.js +32 -0
- package/src/client-supervised-contract.js +52 -0
- package/src/client-supervised-coordinator.js +36 -0
- package/src/client-supervised-permit.js +20 -0
- package/src/client-supervised-protocol.js +37 -0
- package/src/client-supervised-store.js +167 -0
- package/src/client-supervised-supervisor.js +167 -0
- package/src/client-supervised.js +160 -0
- package/src/codex-cli-runner.js +358 -0
- package/src/codex-delivery.js +101 -0
- package/src/connect.js +703 -0
- package/src/doctor.js +795 -0
- package/src/gemini-cli.js +76 -0
- package/src/gemini-files.js +45 -0
- package/src/gemini-tools.js +249 -0
- package/src/init.js +473 -0
- package/src/install-lock.js +127 -0
- package/src/install-registry.js +61 -0
- package/src/legacy-repair.js +71 -0
- package/src/level-agents.js +111 -0
- package/src/link-states.js +172 -0
- package/src/memory-connection-artifacts.js +192 -0
- package/src/memory-connection-descriptor.js +158 -0
- package/src/memory-connection-diagnostic.js +177 -0
- package/src/memory-connection-files.js +191 -0
- package/src/memory-connection-manifest.js +118 -0
- package/src/memory-connection-package.js +176 -0
- package/src/memory-connection-plan.js +211 -0
- package/src/memory-connection-skill.js +211 -0
- package/src/memory-connection-toml.js +153 -0
- package/src/memory-locator.js +118 -0
- package/src/memory-mcp-policy.js +181 -0
- package/src/memory-mcp-server.js +166 -0
- package/src/memory-mcp-transport.js +59 -0
- package/src/missing-dependencies.js +21 -0
- package/src/model-command.js +147 -0
- package/src/model-preference.js +235 -0
- package/src/model-selection.js +368 -0
- package/src/native/managed-preferences.c +100 -0
- package/src/native/managed-preferences.h +17 -0
- package/src/package-lock-path.js +13 -0
- package/src/panel-cli.js +182 -0
- package/src/peer-handoff.js +27 -0
- package/src/peer-inbox-cli.js +211 -0
- package/src/peer-review-mcp.js +210 -0
- package/src/peer-review-store.js +532 -0
- package/src/platform.js +24 -0
- package/src/project-root.js +41 -0
- package/src/project-signals.js +339 -0
- package/src/recommend-report.js +77 -0
- package/src/review-instructions.js +211 -0
- package/src/review-policy.js +35 -0
- package/src/revision-adapters.js +157 -0
- package/src/revision-context.js +86 -0
- package/src/revision-isolation.js +222 -0
- package/src/revision-process.js +103 -0
- package/src/revision-provenance.js +89 -0
- package/src/revision-snapshot.js +120 -0
- package/src/revision.js +242 -0
- package/src/runtime-fixtures/runtime-supervisor/child.cjs +3 -0
- package/src/runtime-fixtures/runtime-supervisor/fixture.cjs +49 -0
- package/src/runtime-fixtures/supervised/child.cjs +3 -0
- package/src/runtime-fixtures/supervised/fixture.cjs +47 -0
- package/src/sensitive-data.js +21 -0
- package/src/setup-apply.js +231 -0
- package/src/setup-plan.js +231 -0
- package/src/setup-state.js +277 -0
- package/src/setup-tui.js +555 -0
- package/src/skill-catalog-add.js +91 -0
- package/src/skill-catalog-audit.js +68 -0
- package/src/skill-catalog.js +170 -0
- package/src/skill-files.js +39 -0
- package/src/skill-recommendations.js +1673 -0
- package/src/skill-scan-report.js +14 -0
- package/src/skill-scan.js +178 -0
- package/src/stable-command.js +53 -0
- package/src/stable-install.js +222 -0
- package/src/status.js +200 -0
- package/src/superpowers.js +133 -0
- package/src/sync-apply.js +227 -0
- package/src/sync-files.js +109 -0
- package/src/sync-inspect.js +394 -0
- package/src/sync-ledger.js +94 -0
- package/src/sync-markdown.js +151 -0
- package/src/sync-proposal.js +84 -0
- package/src/sync-record.js +459 -0
- package/src/sync.js +360 -0
- package/src/task-approvals.js +367 -0
- package/src/task-cli.js +79 -0
- package/src/task-external-verifier.js +584 -0
- package/src/task-prepare-files.js +115 -0
- package/src/task-prepare.js +180 -0
- package/src/task-verify-cli.js +67 -0
- package/src/task-verify-files.js +188 -0
- package/src/task-verify.js +118 -0
- package/src/task-workflow-mcp.js +544 -0
- package/src/task-workflow-store.js +254 -0
- package/src/terminal-text.js +68 -0
- package/src/toolkit-cli.js +543 -0
- package/src/toolkit-commands-guide.js +107 -0
- package/src/toolkit-setup.js +614 -0
- package/src/usage-metrics.js +43 -0
- package/templates/agent-team/01-encargo.md +158 -0
- package/templates/agent-team/02-propuesta.md +132 -0
- package/templates/agent-team/03-revision.md +150 -0
- package/templates/agent-team/README.md +213 -0
- package/templates/model-inventory/01-inventario.md +154 -0
- package/templates/model-inventory/README.md +115 -0
- package/templates/project-memory/README.md +67 -0
- package/templates/project-memory/project-memory.example.json +53 -0
- package/templates/project-memory/skill/project-id/SKILL.md +37 -0
- package/templates/project-memory/skill/project-id/references/project-profile.md +44 -0
- package/templates/project-memory/vault/00-INICIO/Estado actual.md +44 -0
- package/templates/project-memory/vault/00-INICIO/Fuentes de verdad.md +30 -0
- package/templates/project-memory/vault/00-INICIO/Inicio del proyecto.md +44 -0
- package/templates/project-memory/vault/00-INICIO/Mantenimiento de la memoria.md +26 -0
- package/templates/project-memory/vault/00-INICIO/Mapa de la memoria.md +33 -0
- package/templates/project-memory/vault/00-INICIO/Registro de sincronizaci/303/263n.md" +28 -0
- package/templates/project-memory/vault/01-PRODUCTO/Producto.md +16 -0
- package/templates/project-memory/vault/02-ARQUITECTURA/Arquitectura.md +16 -0
- package/templates/project-memory/vault/03-IMPLEMENTACION/Implementaci/303/263n.md +15 -0
- package/templates/project-memory/vault/04-DECISIONES/Decisiones.md +19 -0
- package/templates/project-memory/vault/05-ENTREGAS/Entregas.md +12 -0
- package/templates/project-memory/vault/06-OPERACION/Operaci/303/263n.md +15 -0
- package/templates/project-memory/vault/07-AGENTES/Protocolo de agentes.md +33 -0
- package/templates/project-memory/vault/08-RECURSOS/Recursos.md +12 -0
- package/templates/project-memory/vault/09-ARCHIVO/README.md +5 -0
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
"""Lightweight connection handling for MCP servers."""
|
|
2
|
+
|
|
3
|
+
from abc import ABC, abstractmethod
|
|
4
|
+
from contextlib import AsyncExitStack
|
|
5
|
+
from typing import Any
|
|
6
|
+
|
|
7
|
+
from mcp import ClientSession, StdioServerParameters
|
|
8
|
+
from mcp.client.sse import sse_client
|
|
9
|
+
from mcp.client.stdio import stdio_client
|
|
10
|
+
from mcp.client.streamable_http import streamablehttp_client
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
class MCPConnection(ABC):
|
|
14
|
+
"""Base class for MCP server connections."""
|
|
15
|
+
|
|
16
|
+
def __init__(self):
|
|
17
|
+
self.session = None
|
|
18
|
+
self._stack = None
|
|
19
|
+
|
|
20
|
+
@abstractmethod
|
|
21
|
+
def _create_context(self):
|
|
22
|
+
"""Create the connection context based on connection type."""
|
|
23
|
+
|
|
24
|
+
async def __aenter__(self):
|
|
25
|
+
"""Initialize MCP server connection."""
|
|
26
|
+
self._stack = AsyncExitStack()
|
|
27
|
+
await self._stack.__aenter__()
|
|
28
|
+
|
|
29
|
+
try:
|
|
30
|
+
ctx = self._create_context()
|
|
31
|
+
result = await self._stack.enter_async_context(ctx)
|
|
32
|
+
|
|
33
|
+
if len(result) == 2:
|
|
34
|
+
read, write = result
|
|
35
|
+
elif len(result) == 3:
|
|
36
|
+
read, write, _ = result
|
|
37
|
+
else:
|
|
38
|
+
raise ValueError(f"Unexpected context result: {result}")
|
|
39
|
+
|
|
40
|
+
session_ctx = ClientSession(read, write)
|
|
41
|
+
self.session = await self._stack.enter_async_context(session_ctx)
|
|
42
|
+
await self.session.initialize()
|
|
43
|
+
return self
|
|
44
|
+
except BaseException:
|
|
45
|
+
await self._stack.__aexit__(None, None, None)
|
|
46
|
+
raise
|
|
47
|
+
|
|
48
|
+
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
|
49
|
+
"""Clean up MCP server connection resources."""
|
|
50
|
+
if self._stack:
|
|
51
|
+
await self._stack.__aexit__(exc_type, exc_val, exc_tb)
|
|
52
|
+
self.session = None
|
|
53
|
+
self._stack = None
|
|
54
|
+
|
|
55
|
+
async def list_tools(self) -> list[dict[str, Any]]:
|
|
56
|
+
"""Retrieve available tools from the MCP server."""
|
|
57
|
+
response = await self.session.list_tools()
|
|
58
|
+
return [
|
|
59
|
+
{
|
|
60
|
+
"name": tool.name,
|
|
61
|
+
"description": tool.description,
|
|
62
|
+
"input_schema": tool.inputSchema,
|
|
63
|
+
}
|
|
64
|
+
for tool in response.tools
|
|
65
|
+
]
|
|
66
|
+
|
|
67
|
+
async def call_tool(self, tool_name: str, arguments: dict[str, Any]) -> Any:
|
|
68
|
+
"""Call a tool on the MCP server with provided arguments."""
|
|
69
|
+
result = await self.session.call_tool(tool_name, arguments=arguments)
|
|
70
|
+
return result.content
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
class MCPConnectionStdio(MCPConnection):
|
|
74
|
+
"""MCP connection using standard input/output."""
|
|
75
|
+
|
|
76
|
+
def __init__(self, command: str, args: list[str] = None, env: dict[str, str] = None):
|
|
77
|
+
super().__init__()
|
|
78
|
+
self.command = command
|
|
79
|
+
self.args = args or []
|
|
80
|
+
self.env = env
|
|
81
|
+
|
|
82
|
+
def _create_context(self):
|
|
83
|
+
return stdio_client(
|
|
84
|
+
StdioServerParameters(command=self.command, args=self.args, env=self.env)
|
|
85
|
+
)
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
class MCPConnectionSSE(MCPConnection):
|
|
89
|
+
"""MCP connection using Server-Sent Events."""
|
|
90
|
+
|
|
91
|
+
def __init__(self, url: str, headers: dict[str, str] = None):
|
|
92
|
+
super().__init__()
|
|
93
|
+
self.url = url
|
|
94
|
+
self.headers = headers or {}
|
|
95
|
+
|
|
96
|
+
def _create_context(self):
|
|
97
|
+
return sse_client(url=self.url, headers=self.headers)
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
class MCPConnectionHTTP(MCPConnection):
|
|
101
|
+
"""MCP connection using Streamable HTTP."""
|
|
102
|
+
|
|
103
|
+
def __init__(self, url: str, headers: dict[str, str] = None):
|
|
104
|
+
super().__init__()
|
|
105
|
+
self.url = url
|
|
106
|
+
self.headers = headers or {}
|
|
107
|
+
|
|
108
|
+
def _create_context(self):
|
|
109
|
+
return streamablehttp_client(url=self.url, headers=self.headers)
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
def create_connection(
|
|
113
|
+
transport: str,
|
|
114
|
+
command: str = None,
|
|
115
|
+
args: list[str] = None,
|
|
116
|
+
env: dict[str, str] = None,
|
|
117
|
+
url: str = None,
|
|
118
|
+
headers: dict[str, str] = None,
|
|
119
|
+
) -> MCPConnection:
|
|
120
|
+
"""Factory function to create the appropriate MCP connection.
|
|
121
|
+
|
|
122
|
+
Args:
|
|
123
|
+
transport: Connection type ("stdio", "sse", or "http")
|
|
124
|
+
command: Command to run (stdio only)
|
|
125
|
+
args: Command arguments (stdio only)
|
|
126
|
+
env: Environment variables (stdio only)
|
|
127
|
+
url: Server URL (sse and http only)
|
|
128
|
+
headers: HTTP headers (sse and http only)
|
|
129
|
+
|
|
130
|
+
Returns:
|
|
131
|
+
MCPConnection instance
|
|
132
|
+
"""
|
|
133
|
+
transport = transport.lower()
|
|
134
|
+
|
|
135
|
+
if transport == "stdio":
|
|
136
|
+
if not command:
|
|
137
|
+
raise ValueError("Command is required for stdio transport")
|
|
138
|
+
return MCPConnectionStdio(command=command, args=args, env=env)
|
|
139
|
+
|
|
140
|
+
elif transport == "sse":
|
|
141
|
+
if not url:
|
|
142
|
+
raise ValueError("URL is required for sse transport")
|
|
143
|
+
return MCPConnectionSSE(url=url, headers=headers)
|
|
144
|
+
|
|
145
|
+
elif transport in ["http", "streamable_http", "streamable-http"]:
|
|
146
|
+
if not url:
|
|
147
|
+
raise ValueError("URL is required for http transport")
|
|
148
|
+
return MCPConnectionHTTP(url=url, headers=headers)
|
|
149
|
+
|
|
150
|
+
else:
|
|
151
|
+
raise ValueError(f"Unsupported transport type: {transport}. Use 'stdio', 'sse', or 'http'")
|
|
@@ -0,0 +1,373 @@
|
|
|
1
|
+
"""MCP Server Evaluation Harness
|
|
2
|
+
|
|
3
|
+
This script evaluates MCP servers by running test questions against them using Claude.
|
|
4
|
+
"""
|
|
5
|
+
|
|
6
|
+
import argparse
|
|
7
|
+
import asyncio
|
|
8
|
+
import json
|
|
9
|
+
import re
|
|
10
|
+
import sys
|
|
11
|
+
import time
|
|
12
|
+
import traceback
|
|
13
|
+
import xml.etree.ElementTree as ET
|
|
14
|
+
from pathlib import Path
|
|
15
|
+
from typing import Any
|
|
16
|
+
|
|
17
|
+
from anthropic import Anthropic
|
|
18
|
+
|
|
19
|
+
from connections import create_connection
|
|
20
|
+
|
|
21
|
+
EVALUATION_PROMPT = """You are an AI assistant with access to tools.
|
|
22
|
+
|
|
23
|
+
When given a task, you MUST:
|
|
24
|
+
1. Use the available tools to complete the task
|
|
25
|
+
2. Provide summary of each step in your approach, wrapped in <summary> tags
|
|
26
|
+
3. Provide feedback on the tools provided, wrapped in <feedback> tags
|
|
27
|
+
4. Provide your final response, wrapped in <response> tags
|
|
28
|
+
|
|
29
|
+
Summary Requirements:
|
|
30
|
+
- In your <summary> tags, you must explain:
|
|
31
|
+
- The steps you took to complete the task
|
|
32
|
+
- Which tools you used, in what order, and why
|
|
33
|
+
- The inputs you provided to each tool
|
|
34
|
+
- The outputs you received from each tool
|
|
35
|
+
- A summary for how you arrived at the response
|
|
36
|
+
|
|
37
|
+
Feedback Requirements:
|
|
38
|
+
- In your <feedback> tags, provide constructive feedback on the tools:
|
|
39
|
+
- Comment on tool names: Are they clear and descriptive?
|
|
40
|
+
- Comment on input parameters: Are they well-documented? Are required vs optional parameters clear?
|
|
41
|
+
- Comment on descriptions: Do they accurately describe what the tool does?
|
|
42
|
+
- Comment on any errors encountered during tool usage: Did the tool fail to execute? Did the tool return too many tokens?
|
|
43
|
+
- Identify specific areas for improvement and explain WHY they would help
|
|
44
|
+
- Be specific and actionable in your suggestions
|
|
45
|
+
|
|
46
|
+
Response Requirements:
|
|
47
|
+
- Your response should be concise and directly address what was asked
|
|
48
|
+
- Always wrap your final response in <response> tags
|
|
49
|
+
- If you cannot solve the task return <response>NOT_FOUND</response>
|
|
50
|
+
- For numeric responses, provide just the number
|
|
51
|
+
- For IDs, provide just the ID
|
|
52
|
+
- For names or text, provide the exact text requested
|
|
53
|
+
- Your response should go last"""
|
|
54
|
+
|
|
55
|
+
|
|
56
|
+
def parse_evaluation_file(file_path: Path) -> list[dict[str, Any]]:
|
|
57
|
+
"""Parse XML evaluation file with qa_pair elements."""
|
|
58
|
+
try:
|
|
59
|
+
tree = ET.parse(file_path)
|
|
60
|
+
root = tree.getroot()
|
|
61
|
+
evaluations = []
|
|
62
|
+
|
|
63
|
+
for qa_pair in root.findall(".//qa_pair"):
|
|
64
|
+
question_elem = qa_pair.find("question")
|
|
65
|
+
answer_elem = qa_pair.find("answer")
|
|
66
|
+
|
|
67
|
+
if question_elem is not None and answer_elem is not None:
|
|
68
|
+
evaluations.append({
|
|
69
|
+
"question": (question_elem.text or "").strip(),
|
|
70
|
+
"answer": (answer_elem.text or "").strip(),
|
|
71
|
+
})
|
|
72
|
+
|
|
73
|
+
return evaluations
|
|
74
|
+
except Exception as e:
|
|
75
|
+
print(f"Error parsing evaluation file {file_path}: {e}")
|
|
76
|
+
return []
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
def extract_xml_content(text: str, tag: str) -> str | None:
|
|
80
|
+
"""Extract content from XML tags."""
|
|
81
|
+
pattern = rf"<{tag}>(.*?)</{tag}>"
|
|
82
|
+
matches = re.findall(pattern, text, re.DOTALL)
|
|
83
|
+
return matches[-1].strip() if matches else None
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
async def agent_loop(
|
|
87
|
+
client: Anthropic,
|
|
88
|
+
model: str,
|
|
89
|
+
question: str,
|
|
90
|
+
tools: list[dict[str, Any]],
|
|
91
|
+
connection: Any,
|
|
92
|
+
) -> tuple[str, dict[str, Any]]:
|
|
93
|
+
"""Run the agent loop with MCP tools."""
|
|
94
|
+
messages = [{"role": "user", "content": question}]
|
|
95
|
+
|
|
96
|
+
response = await asyncio.to_thread(
|
|
97
|
+
client.messages.create,
|
|
98
|
+
model=model,
|
|
99
|
+
max_tokens=4096,
|
|
100
|
+
system=EVALUATION_PROMPT,
|
|
101
|
+
messages=messages,
|
|
102
|
+
tools=tools,
|
|
103
|
+
)
|
|
104
|
+
|
|
105
|
+
messages.append({"role": "assistant", "content": response.content})
|
|
106
|
+
|
|
107
|
+
tool_metrics = {}
|
|
108
|
+
|
|
109
|
+
while response.stop_reason == "tool_use":
|
|
110
|
+
tool_use = next(block for block in response.content if block.type == "tool_use")
|
|
111
|
+
tool_name = tool_use.name
|
|
112
|
+
tool_input = tool_use.input
|
|
113
|
+
|
|
114
|
+
tool_start_ts = time.time()
|
|
115
|
+
try:
|
|
116
|
+
tool_result = await connection.call_tool(tool_name, tool_input)
|
|
117
|
+
tool_response = json.dumps(tool_result) if isinstance(tool_result, (dict, list)) else str(tool_result)
|
|
118
|
+
except Exception as e:
|
|
119
|
+
tool_response = f"Error executing tool {tool_name}: {str(e)}\n"
|
|
120
|
+
tool_response += traceback.format_exc()
|
|
121
|
+
tool_duration = time.time() - tool_start_ts
|
|
122
|
+
|
|
123
|
+
if tool_name not in tool_metrics:
|
|
124
|
+
tool_metrics[tool_name] = {"count": 0, "durations": []}
|
|
125
|
+
tool_metrics[tool_name]["count"] += 1
|
|
126
|
+
tool_metrics[tool_name]["durations"].append(tool_duration)
|
|
127
|
+
|
|
128
|
+
messages.append({
|
|
129
|
+
"role": "user",
|
|
130
|
+
"content": [{
|
|
131
|
+
"type": "tool_result",
|
|
132
|
+
"tool_use_id": tool_use.id,
|
|
133
|
+
"content": tool_response,
|
|
134
|
+
}]
|
|
135
|
+
})
|
|
136
|
+
|
|
137
|
+
response = await asyncio.to_thread(
|
|
138
|
+
client.messages.create,
|
|
139
|
+
model=model,
|
|
140
|
+
max_tokens=4096,
|
|
141
|
+
system=EVALUATION_PROMPT,
|
|
142
|
+
messages=messages,
|
|
143
|
+
tools=tools,
|
|
144
|
+
)
|
|
145
|
+
messages.append({"role": "assistant", "content": response.content})
|
|
146
|
+
|
|
147
|
+
response_text = next(
|
|
148
|
+
(block.text for block in response.content if hasattr(block, "text")),
|
|
149
|
+
None,
|
|
150
|
+
)
|
|
151
|
+
return response_text, tool_metrics
|
|
152
|
+
|
|
153
|
+
|
|
154
|
+
async def evaluate_single_task(
|
|
155
|
+
client: Anthropic,
|
|
156
|
+
model: str,
|
|
157
|
+
qa_pair: dict[str, Any],
|
|
158
|
+
tools: list[dict[str, Any]],
|
|
159
|
+
connection: Any,
|
|
160
|
+
task_index: int,
|
|
161
|
+
) -> dict[str, Any]:
|
|
162
|
+
"""Evaluate a single QA pair with the given tools."""
|
|
163
|
+
start_time = time.time()
|
|
164
|
+
|
|
165
|
+
print(f"Task {task_index + 1}: Running task with question: {qa_pair['question']}")
|
|
166
|
+
response, tool_metrics = await agent_loop(client, model, qa_pair["question"], tools, connection)
|
|
167
|
+
|
|
168
|
+
response_value = extract_xml_content(response, "response")
|
|
169
|
+
summary = extract_xml_content(response, "summary")
|
|
170
|
+
feedback = extract_xml_content(response, "feedback")
|
|
171
|
+
|
|
172
|
+
duration_seconds = time.time() - start_time
|
|
173
|
+
|
|
174
|
+
return {
|
|
175
|
+
"question": qa_pair["question"],
|
|
176
|
+
"expected": qa_pair["answer"],
|
|
177
|
+
"actual": response_value,
|
|
178
|
+
"score": int(response_value == qa_pair["answer"]) if response_value else 0,
|
|
179
|
+
"total_duration": duration_seconds,
|
|
180
|
+
"tool_calls": tool_metrics,
|
|
181
|
+
"num_tool_calls": sum(len(metrics["durations"]) for metrics in tool_metrics.values()),
|
|
182
|
+
"summary": summary,
|
|
183
|
+
"feedback": feedback,
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
|
|
187
|
+
REPORT_HEADER = """
|
|
188
|
+
# Evaluation Report
|
|
189
|
+
|
|
190
|
+
## Summary
|
|
191
|
+
|
|
192
|
+
- **Accuracy**: {correct}/{total} ({accuracy:.1f}%)
|
|
193
|
+
- **Average Task Duration**: {average_duration_s:.2f}s
|
|
194
|
+
- **Average Tool Calls per Task**: {average_tool_calls:.2f}
|
|
195
|
+
- **Total Tool Calls**: {total_tool_calls}
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
"""
|
|
199
|
+
|
|
200
|
+
TASK_TEMPLATE = """
|
|
201
|
+
### Task {task_num}
|
|
202
|
+
|
|
203
|
+
**Question**: {question}
|
|
204
|
+
**Ground Truth Answer**: `{expected_answer}`
|
|
205
|
+
**Actual Answer**: `{actual_answer}`
|
|
206
|
+
**Correct**: {correct_indicator}
|
|
207
|
+
**Duration**: {total_duration:.2f}s
|
|
208
|
+
**Tool Calls**: {tool_calls}
|
|
209
|
+
|
|
210
|
+
**Summary**
|
|
211
|
+
{summary}
|
|
212
|
+
|
|
213
|
+
**Feedback**
|
|
214
|
+
{feedback}
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
"""
|
|
218
|
+
|
|
219
|
+
|
|
220
|
+
async def run_evaluation(
|
|
221
|
+
eval_path: Path,
|
|
222
|
+
connection: Any,
|
|
223
|
+
model: str = "claude-3-7-sonnet-20250219",
|
|
224
|
+
) -> str:
|
|
225
|
+
"""Run evaluation with MCP server tools."""
|
|
226
|
+
print("🚀 Starting Evaluation")
|
|
227
|
+
|
|
228
|
+
client = Anthropic()
|
|
229
|
+
|
|
230
|
+
tools = await connection.list_tools()
|
|
231
|
+
print(f"📋 Loaded {len(tools)} tools from MCP server")
|
|
232
|
+
|
|
233
|
+
qa_pairs = parse_evaluation_file(eval_path)
|
|
234
|
+
print(f"📋 Loaded {len(qa_pairs)} evaluation tasks")
|
|
235
|
+
|
|
236
|
+
results = []
|
|
237
|
+
for i, qa_pair in enumerate(qa_pairs):
|
|
238
|
+
print(f"Processing task {i + 1}/{len(qa_pairs)}")
|
|
239
|
+
result = await evaluate_single_task(client, model, qa_pair, tools, connection, i)
|
|
240
|
+
results.append(result)
|
|
241
|
+
|
|
242
|
+
correct = sum(r["score"] for r in results)
|
|
243
|
+
accuracy = (correct / len(results)) * 100 if results else 0
|
|
244
|
+
average_duration_s = sum(r["total_duration"] for r in results) / len(results) if results else 0
|
|
245
|
+
average_tool_calls = sum(r["num_tool_calls"] for r in results) / len(results) if results else 0
|
|
246
|
+
total_tool_calls = sum(r["num_tool_calls"] for r in results)
|
|
247
|
+
|
|
248
|
+
report = REPORT_HEADER.format(
|
|
249
|
+
correct=correct,
|
|
250
|
+
total=len(results),
|
|
251
|
+
accuracy=accuracy,
|
|
252
|
+
average_duration_s=average_duration_s,
|
|
253
|
+
average_tool_calls=average_tool_calls,
|
|
254
|
+
total_tool_calls=total_tool_calls,
|
|
255
|
+
)
|
|
256
|
+
|
|
257
|
+
report += "".join([
|
|
258
|
+
TASK_TEMPLATE.format(
|
|
259
|
+
task_num=i + 1,
|
|
260
|
+
question=qa_pair["question"],
|
|
261
|
+
expected_answer=qa_pair["answer"],
|
|
262
|
+
actual_answer=result["actual"] or "N/A",
|
|
263
|
+
correct_indicator="✅" if result["score"] else "❌",
|
|
264
|
+
total_duration=result["total_duration"],
|
|
265
|
+
tool_calls=json.dumps(result["tool_calls"], indent=2),
|
|
266
|
+
summary=result["summary"] or "N/A",
|
|
267
|
+
feedback=result["feedback"] or "N/A",
|
|
268
|
+
)
|
|
269
|
+
for i, (qa_pair, result) in enumerate(zip(qa_pairs, results))
|
|
270
|
+
])
|
|
271
|
+
|
|
272
|
+
return report
|
|
273
|
+
|
|
274
|
+
|
|
275
|
+
def parse_headers(header_list: list[str]) -> dict[str, str]:
|
|
276
|
+
"""Parse header strings in format 'Key: Value' into a dictionary."""
|
|
277
|
+
headers = {}
|
|
278
|
+
if not header_list:
|
|
279
|
+
return headers
|
|
280
|
+
|
|
281
|
+
for header in header_list:
|
|
282
|
+
if ":" in header:
|
|
283
|
+
key, value = header.split(":", 1)
|
|
284
|
+
headers[key.strip()] = value.strip()
|
|
285
|
+
else:
|
|
286
|
+
print(f"Warning: Ignoring malformed header: {header}")
|
|
287
|
+
return headers
|
|
288
|
+
|
|
289
|
+
|
|
290
|
+
def parse_env_vars(env_list: list[str]) -> dict[str, str]:
|
|
291
|
+
"""Parse environment variable strings in format 'KEY=VALUE' into a dictionary."""
|
|
292
|
+
env = {}
|
|
293
|
+
if not env_list:
|
|
294
|
+
return env
|
|
295
|
+
|
|
296
|
+
for env_var in env_list:
|
|
297
|
+
if "=" in env_var:
|
|
298
|
+
key, value = env_var.split("=", 1)
|
|
299
|
+
env[key.strip()] = value.strip()
|
|
300
|
+
else:
|
|
301
|
+
print(f"Warning: Ignoring malformed environment variable: {env_var}")
|
|
302
|
+
return env
|
|
303
|
+
|
|
304
|
+
|
|
305
|
+
async def main():
|
|
306
|
+
parser = argparse.ArgumentParser(
|
|
307
|
+
description="Evaluate MCP servers using test questions",
|
|
308
|
+
formatter_class=argparse.RawDescriptionHelpFormatter,
|
|
309
|
+
epilog="""
|
|
310
|
+
Examples:
|
|
311
|
+
# Evaluate a local stdio MCP server
|
|
312
|
+
python evaluation.py -t stdio -c python -a my_server.py eval.xml
|
|
313
|
+
|
|
314
|
+
# Evaluate an SSE MCP server
|
|
315
|
+
python evaluation.py -t sse -u https://example.com/mcp -H "Authorization: Bearer token" eval.xml
|
|
316
|
+
|
|
317
|
+
# Evaluate an HTTP MCP server with custom model
|
|
318
|
+
python evaluation.py -t http -u https://example.com/mcp -m claude-3-5-sonnet-20241022 eval.xml
|
|
319
|
+
""",
|
|
320
|
+
)
|
|
321
|
+
|
|
322
|
+
parser.add_argument("eval_file", type=Path, help="Path to evaluation XML file")
|
|
323
|
+
parser.add_argument("-t", "--transport", choices=["stdio", "sse", "http"], default="stdio", help="Transport type (default: stdio)")
|
|
324
|
+
parser.add_argument("-m", "--model", default="claude-3-7-sonnet-20250219", help="Claude model to use (default: claude-3-7-sonnet-20250219)")
|
|
325
|
+
|
|
326
|
+
stdio_group = parser.add_argument_group("stdio options")
|
|
327
|
+
stdio_group.add_argument("-c", "--command", help="Command to run MCP server (stdio only)")
|
|
328
|
+
stdio_group.add_argument("-a", "--args", nargs="+", help="Arguments for the command (stdio only)")
|
|
329
|
+
stdio_group.add_argument("-e", "--env", nargs="+", help="Environment variables in KEY=VALUE format (stdio only)")
|
|
330
|
+
|
|
331
|
+
remote_group = parser.add_argument_group("sse/http options")
|
|
332
|
+
remote_group.add_argument("-u", "--url", help="MCP server URL (sse/http only)")
|
|
333
|
+
remote_group.add_argument("-H", "--header", nargs="+", dest="headers", help="HTTP headers in 'Key: Value' format (sse/http only)")
|
|
334
|
+
|
|
335
|
+
parser.add_argument("-o", "--output", type=Path, help="Output file for evaluation report (default: stdout)")
|
|
336
|
+
|
|
337
|
+
args = parser.parse_args()
|
|
338
|
+
|
|
339
|
+
if not args.eval_file.exists():
|
|
340
|
+
print(f"Error: Evaluation file not found: {args.eval_file}")
|
|
341
|
+
sys.exit(1)
|
|
342
|
+
|
|
343
|
+
headers = parse_headers(args.headers) if args.headers else None
|
|
344
|
+
env_vars = parse_env_vars(args.env) if args.env else None
|
|
345
|
+
|
|
346
|
+
try:
|
|
347
|
+
connection = create_connection(
|
|
348
|
+
transport=args.transport,
|
|
349
|
+
command=args.command,
|
|
350
|
+
args=args.args,
|
|
351
|
+
env=env_vars,
|
|
352
|
+
url=args.url,
|
|
353
|
+
headers=headers,
|
|
354
|
+
)
|
|
355
|
+
except ValueError as e:
|
|
356
|
+
print(f"Error: {e}")
|
|
357
|
+
sys.exit(1)
|
|
358
|
+
|
|
359
|
+
print(f"🔗 Connecting to MCP server via {args.transport}...")
|
|
360
|
+
|
|
361
|
+
async with connection:
|
|
362
|
+
print("✅ Connected successfully")
|
|
363
|
+
report = await run_evaluation(args.eval_file, connection, args.model)
|
|
364
|
+
|
|
365
|
+
if args.output:
|
|
366
|
+
args.output.write_text(report)
|
|
367
|
+
print(f"\n✅ Report saved to {args.output}")
|
|
368
|
+
else:
|
|
369
|
+
print("\n" + report)
|
|
370
|
+
|
|
371
|
+
|
|
372
|
+
if __name__ == "__main__":
|
|
373
|
+
asyncio.run(main())
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
<evaluation>
|
|
2
|
+
<qa_pair>
|
|
3
|
+
<question>Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years. What is the final amount in dollars (rounded to 2 decimal places)?</question>
|
|
4
|
+
<answer>11614.72</answer>
|
|
5
|
+
</qa_pair>
|
|
6
|
+
<qa_pair>
|
|
7
|
+
<question>A projectile is launched at a 45-degree angle with an initial velocity of 50 m/s. Calculate the total distance (in meters) it has traveled from the launch point after 2 seconds, assuming g=9.8 m/s². Round to 2 decimal places.</question>
|
|
8
|
+
<answer>87.25</answer>
|
|
9
|
+
</qa_pair>
|
|
10
|
+
<qa_pair>
|
|
11
|
+
<question>A sphere has a volume of 500 cubic meters. Calculate its surface area in square meters. Round to 2 decimal places.</question>
|
|
12
|
+
<answer>304.65</answer>
|
|
13
|
+
</qa_pair>
|
|
14
|
+
<qa_pair>
|
|
15
|
+
<question>Calculate the population standard deviation of this dataset: [12, 15, 18, 22, 25, 30, 35]. Round to 2 decimal places.</question>
|
|
16
|
+
<answer>7.61</answer>
|
|
17
|
+
</qa_pair>
|
|
18
|
+
<qa_pair>
|
|
19
|
+
<question>Calculate the pH of a solution with a hydrogen ion concentration of 3.5 × 10^-5 M. Round to 2 decimal places.</question>
|
|
20
|
+
<answer>4.46</answer>
|
|
21
|
+
</qa_pair>
|
|
22
|
+
</evaluation>
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ordenar-proyecto
|
|
3
|
+
description: Úsala cuando se pida organizar archivos, carpetas o documentación, o cuando un trabajo autorizado necesite ubicar archivos nuevos dentro de un proyecto existente.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Ordenar un proyecto
|
|
7
|
+
|
|
8
|
+
Ubica cada archivo según su responsabilidad y las convenciones del proyecto.
|
|
9
|
+
El resultado debe ser fácil de recorrer y conservar sus referencias funcionales.
|
|
10
|
+
|
|
11
|
+
## Elegir el alcance
|
|
12
|
+
|
|
13
|
+
| Encargo | Actuación |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| Añadir un archivo | Elegir la carpeta existente que corresponda y actualizar su índice si lo necesita. |
|
|
16
|
+
| Reorganizar un área | Inventariar esa área y sus consumidores; preparar movimientos y verificar referencias. |
|
|
17
|
+
| Reorganizar código | Revisar además imports, exports, configuración, recursos y puntos de entrada afectados. |
|
|
18
|
+
|
|
19
|
+
Un trabajo pequeño conserva el alcance solicitado. La preferencia por el orden
|
|
20
|
+
no inicia una reorganización completa del repositorio en cada tarea.
|
|
21
|
+
|
|
22
|
+
## Reorganizar
|
|
23
|
+
|
|
24
|
+
1. Lee las instrucciones del proyecto y revisa el estado de los archivos.
|
|
25
|
+
Identifica cambios previos, archivos generados y ubicaciones con significado
|
|
26
|
+
para herramientas o contratos.
|
|
27
|
+
2. Agrupa por responsabilidad, tema o función siguiendo el criterio existente.
|
|
28
|
+
Usa nombres claros y una profundidad que facilite encontrar los archivos.
|
|
29
|
+
Crea una carpeta cuando haya contenido concreto para ella.
|
|
30
|
+
3. Prepara un mapa de origen a destino. Comprueba destinos únicos, colisiones y
|
|
31
|
+
referencias mediante búsquedas como `rg`. Delimita qué consumidores necesitan
|
|
32
|
+
actualizarse aunque vivan fuera del área que se mueve.
|
|
33
|
+
4. Mueve conservando contenido y cambios ajenos. Actualiza enlaces relativos,
|
|
34
|
+
índices e imports según su nueva ubicación. Una URL anclada a un commit
|
|
35
|
+
histórico conserva la ruta de esa revisión; una ruta ficticia de un ejemplo
|
|
36
|
+
se evalúa por su función, sin convertirla automáticamente en una ruta real.
|
|
37
|
+
5. Comprueba que cada archivo original sigue presente y compara las referencias
|
|
38
|
+
antes y después. En documentación revisa destinos locales; en código ejecuta
|
|
39
|
+
las comprobaciones relevantes del área. Informa fallos previos por separado.
|
|
40
|
+
6. Resume la estructura, referencias actualizadas y evidencia de comprobación.
|
|
41
|
+
Mantén el criterio en el índice o instrucciones existentes del área cuando
|
|
42
|
+
eso ayude a ubicar los próximos archivos.
|
|
43
|
+
|
|
44
|
+
## Ejemplo
|
|
45
|
+
|
|
46
|
+
Si `docs/auth.md` pasa a `docs/guias/auth.md`, su enlace `../src/auth.js`
|
|
47
|
+
debe pasar a `../../src/auth.js`; también hay que actualizar los enlaces hacia
|
|
48
|
+
esa guía desde el README y otros documentos. Una copia o referencia externa
|
|
49
|
+
no queda actualizada por mover el archivo local.
|
|
50
|
+
|
|
51
|
+
## Errores frecuentes
|
|
52
|
+
|
|
53
|
+
- Mover solo las guías sin revisar sus consumidores externos al directorio.
|
|
54
|
+
- Aplicar una sustitución global que reescribe URLs históricas o fixtures.
|
|
55
|
+
- Crear carpetas vacías o un cajón de varios sin criterio de clasificación.
|
|
56
|
+
- Afirmar que un archivo conservado demuestra que sus enlaces o imports funcionan.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: project-handoff
|
|
3
|
+
description: Prepara un traspaso compacto entre sesiones, personas o agentes con estado comprobado, cambios, riesgos y siguiente paso. Úsala al pausar, cambiar de responsable o cerrar una etapa; no la uses para copiar toda la conversación o inventar progreso.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Traspaso de proyecto y nota de seguimiento
|
|
7
|
+
|
|
8
|
+
Produce un resumen compacto y verificable que permita continuar en otra sesión o con otra IA sin reconstruir todo el contexto.
|
|
9
|
+
|
|
10
|
+
## 1. Cuándo se dispara
|
|
11
|
+
|
|
12
|
+
Una tarea se considera **larga** cuando tiene un plan de implementación de varios pasos (en `03-IMPLEMENTACION/Planes/` o `docs/superpowers/plans/`) o cuando ya acumula commits o cambios verificables en una rama y no va a completarse en la sesión actual.
|
|
13
|
+
|
|
14
|
+
Si una tarea larga queda a medias —por pausa a petición del usuario, cambio de IA, bloqueo que detiene la sesión o cierre antes de terminar la fase—, escribe o actualiza la nota de seguimiento siguiendo [la plantilla de traspaso](references/handoff-template.md).
|
|
15
|
+
|
|
16
|
+
## 2. Permiso único por sesión y ubicación en la bóveda
|
|
17
|
+
|
|
18
|
+
- **Permiso de escritura:** la bóveda está fuera del repositorio del proyecto y el MCP de memoria es estrictamente de solo lectura. Pide **una sola autorización por sesión** para crear o actualizar la nota de seguimiento usando tus **herramientas de archivos**. Si el usuario no concede permiso (o el proyecto no tiene bóveda conectada), entrega el traspaso directamente en el chat.
|
|
19
|
+
- **Ubicación y nombre:** `<bóveda>/03-IMPLEMENTACION/Planes/<AAAA-MM-DD>-<slug>-seguimiento.md`, reutilizando la fecha y el `<slug>` base del plan activo (o la fecha actual y el nombre de la rama sin prefijo si no hay plan). Mientras la tarea siga con `estado: en curso`, actualiza siempre esa misma nota sin crear duplicados.
|
|
20
|
+
- **Cierre al fusionar:** cuando la rama se integra en `main` y se ejecuta `cerrar-memoria`, las decisiones y el estado final pasan a las notas permanentes y la nota de seguimiento cambia su frontmatter a `estado: completado`.
|
|
21
|
+
|
|
22
|
+
## 3. Contenido obligatorio y prohibiciones
|
|
23
|
+
|
|
24
|
+
La nota de seguimiento incluye únicamente estas cinco secciones comprobables:
|
|
25
|
+
|
|
26
|
+
1. **Estado:** rama, base, último commit, archivos tocados, resultado real de las pruebas y qué parte del plan está hecha.
|
|
27
|
+
2. **Cambio de diseño:** cualquier ajuste acordado con el usuario respecto al plan original.
|
|
28
|
+
3. **Revisiones:** rondas de revisión ejecutadas, con qué IA y modelo, hallazgos corregidos y descartados.
|
|
29
|
+
4. **Pendiente:** tareas o pasos que faltan y su condición de finalización, o causa del bloqueo y qué lo desbloquea.
|
|
30
|
+
5. **Cómo retomar:** los comandos exactos (3–4 líneas) para verificar el árbol y continuar desde el punto exacto.
|
|
31
|
+
|
|
32
|
+
**Prohibido:** nunca escribas secretos, tokens ni credenciales; nunca vuelques conversaciones enteras ni razonamientos internos; nunca modifiques el archivo del plan en curso para meter seguimiento; y nunca ejecutes `git commit` ni `git push` automáticos en la bóveda por el mero hecho de guardar el seguimiento.
|
|
33
|
+
|
|
34
|
+
## 4. Cuándo recomendar una sesión nueva
|
|
35
|
+
|
|
36
|
+
Recomienda al usuario cerrar la sesión actual y abrir otra —**nunca la cierres por tu cuenta**— en estos cuatro casos:
|
|
37
|
+
|
|
38
|
+
1. **Empieza una fase o una tarea grande nueva** tras terminar o dejar lista la anterior.
|
|
39
|
+
2. **Se instalaron o actualizaron skills o instrucciones globales** (por ejemplo, tras ejecutar `npx nexakit`), porque cada IA solo las lee al abrir la sesión.
|
|
40
|
+
3. **Se cambia de proyecto** o de repositorio principal.
|
|
41
|
+
4. **La sesión ya resumió su contexto o avisa de que va a hacerlo** por límite de ventana.
|
|
42
|
+
|
|
43
|
+
Antes de recomendar abrir una sesión nueva, escribe o actualiza primero el traspaso en la nota de seguimiento y entrega al usuario una frase corta lista para copiar al abrir la sesión nueva:
|
|
44
|
+
|
|
45
|
+
`«Retoma <tarea>; el contexto está en <nota de seguimiento>»`
|
|
46
|
+
|
|
47
|
+
No recomiendes abrir una sesión nueva entre pasos de una misma tarea mientras la sesión siga corta.
|
|
48
|
+
|
|
49
|
+
## Calidad mínima
|
|
50
|
+
|
|
51
|
+
- Cada afirmación importante debe poder rastrearse a evidencia comprobada.
|
|
52
|
+
- Un bloqueo debe indicar causa y qué lo desbloquea.
|
|
53
|
+
- El siguiente agente no debe asumir que una prueba pasó si no se ejecutó.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
interface:
|
|
2
|
+
display_name: "Project Handoff"
|
|
3
|
+
short_description: "Prepara traspasos verificables entre sesiones y agentes"
|
|
4
|
+
default_prompt: "Usa $project-handoff para resumir el estado comprobado y dejar un siguiente paso claro."
|
|
5
|
+
|
|
6
|
+
policy:
|
|
7
|
+
allow_implicit_invocation: true
|