gabbe-kit 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +25 -0
- package/LICENSE-CODE +202 -0
- package/README.md +817 -0
- package/agents/AGENTS.md +542 -0
- package/agents/CONSTITUTION.md +188 -0
- package/agents/guides/00-index.md +142 -0
- package/agents/guides/ai/a2ui-protocols.md +71 -0
- package/agents/guides/ai/actor-agent-frameworks.md +82 -0
- package/agents/guides/ai/agent-communication.md +146 -0
- package/agents/guides/ai/agent-identity-trust.md +69 -0
- package/agents/guides/ai/agent-only-cognition.md +73 -0
- package/agents/guides/ai/agent-only-swarms.md +70 -0
- package/agents/guides/ai/agent-operating-ergonomics.md +52 -0
- package/agents/guides/ai/agent-ui.md +106 -0
- package/agents/guides/ai/ai-agentic.md +309 -0
- package/agents/guides/ai/beyond-llms.md +60 -0
- package/agents/guides/ai/dynamic-capability-loading.md +121 -0
- package/agents/guides/ai/gabbe-innovations.md +111 -0
- package/agents/guides/ai/model-customization-decisions.md +78 -0
- package/agents/guides/ai/multi-agent-systems.md +85 -0
- package/agents/guides/ai/self-evolving-skills.md +137 -0
- package/agents/guides/ai/self-healing-summary.md +21 -0
- package/agents/guides/ai/visual-mcp-integration.md +106 -0
- package/agents/guides/architecture/api-standards.md +61 -0
- package/agents/guides/architecture/blockchain-dlt.md +36 -0
- package/agents/guides/architecture/critical-systems-arch.md +66 -0
- package/agents/guides/architecture/event-driven-architecture.md +39 -0
- package/agents/guides/architecture/microservices.md +292 -0
- package/agents/guides/architecture/monolith.md +303 -0
- package/agents/guides/architecture/orchestration.md +75 -0
- package/agents/guides/architecture/system-qualities.md +51 -0
- package/agents/guides/architecture/system-scalability.md +47 -0
- package/agents/guides/architecture/systems-architecture.md +506 -0
- package/agents/guides/data/semantic-web.md +23 -0
- package/agents/guides/data/sql-nosql.md +305 -0
- package/agents/guides/industry/engineering-standards.md +36 -0
- package/agents/guides/industry/global-standards.md +24 -0
- package/agents/guides/industry/healthcare-fhir.md +33 -0
- package/agents/guides/industry/industrial-iot.md +33 -0
- package/agents/guides/industry/telecom-networks.md +34 -0
- package/agents/guides/languages/c-sharp.md +66 -0
- package/agents/guides/languages/go-lang.md +251 -0
- package/agents/guides/languages/java.md +73 -0
- package/agents/guides/languages/js-ts-nodejs.md +361 -0
- package/agents/guides/languages/nodejs-advanced.md +56 -0
- package/agents/guides/languages/php-laravel.md +376 -0
- package/agents/guides/languages/python-fastapi-ai.md +257 -0
- package/agents/guides/languages/rust.md +73 -0
- package/agents/guides/ops/compliance-audit.md +177 -0
- package/agents/guides/ops/cost-optimization.md +57 -0
- package/agents/guides/ops/dev-environments.md +67 -0
- package/agents/guides/ops/dev-workflow.md +44 -0
- package/agents/guides/ops/gabbe-cli-workflows.md +252 -0
- package/agents/guides/ops/production-health.md +62 -0
- package/agents/guides/ops/troubleshooting-guide.md +69 -0
- package/agents/guides/patterns/agentic-design-patterns.md +308 -0
- package/agents/guides/patterns/agentic-patterns.md +43 -0
- package/agents/guides/patterns/ai-native-scenarios.md +71 -0
- package/agents/guides/patterns/autonomous-swarm-patterns.md +42 -0
- package/agents/guides/patterns/design-patterns.md +68 -0
- package/agents/guides/patterns/enterprise-migration-scenario.md +58 -0
- package/agents/guides/patterns/enterprise-patterns.md +56 -0
- package/agents/guides/patterns/software-design-patterns.md +106 -0
- package/agents/guides/patterns/time-complexity-analysis.md +116 -0
- package/agents/guides/planning/product-requirements.md +407 -0
- package/agents/guides/planning/strategic-analysis.md +98 -0
- package/agents/guides/planning/visual-product-specs.md +162 -0
- package/agents/guides/principles/clean-code-standards.md +51 -0
- package/agents/guides/principles/computing-foundations.md +87 -0
- package/agents/guides/principles/diagramming-standards.md +568 -0
- package/agents/guides/principles/evaluation-strategy.md +103 -0
- package/agents/guides/principles/future-tech.md +35 -0
- package/agents/guides/principles/human-agent-collaboration.md +54 -0
- package/agents/guides/principles/legacy-tech.md +61 -0
- package/agents/guides/principles/mathematical-foundations.md +103 -0
- package/agents/guides/principles/modeling-methods.md +97 -0
- package/agents/guides/principles/no-code-integration.md +84 -0
- package/agents/guides/principles/testing-strategy.md +97 -0
- package/agents/guides/principles/visual-design-system.md +44 -0
- package/agents/guides/processes/RARV_CYCLE.md +262 -0
- package/agents/guides/processes/extension-protocol.md +183 -0
- package/agents/guides/processes/full-system-lifecycle.md +47 -0
- package/agents/guides/processes/knowledge-integration.md +82 -0
- package/agents/guides/processes/loki-sdlc-phases.md +446 -0
- package/agents/guides/product/green-software.md +38 -0
- package/agents/guides/product/sustainability-checks.md +29 -0
- package/agents/guides/security/ai-risk-standards-map.md +98 -0
- package/agents/guides/security/api-security.md +51 -0
- package/agents/guides/security/cryptography-standards.md +38 -0
- package/agents/guides/security/privacy-data-protection.md +41 -0
- package/agents/guides/security/secure-architecture.md +43 -0
- package/agents/guides/security/threat-modeling.md +46 -0
- package/agents/memory/AUDIT_LOG.md +48 -0
- package/agents/memory/CONTINUITY.md +49 -0
- package/agents/memory/PROJECT_STATE.md +134 -0
- package/agents/memory/RESUME_POINTER.md +15 -0
- package/agents/memory/episodic/DECISION_LOG_TEMPLATE.md +104 -0
- package/agents/memory/semantic/PROJECT_KNOWLEDGE_TEMPLATE.md +194 -0
- package/agents/memory/semantic/VECTOR_DB_CONFIG_TEMPLATE.json +46 -0
- package/agents/personas/00-index.md +148 -0
- package/agents/personas/biz-compliance.md +112 -0
- package/agents/personas/biz-legal.md +131 -0
- package/agents/personas/eng-api.md +110 -0
- package/agents/personas/eng-backend.md +113 -0
- package/agents/personas/eng-data.md +50 -0
- package/agents/personas/eng-database.md +95 -0
- package/agents/personas/eng-frontend.md +75 -0
- package/agents/personas/eng-infra.md +105 -0
- package/agents/personas/eng-messaging.md +50 -0
- package/agents/personas/eng-ml.md +50 -0
- package/agents/personas/eng-mobile.md +75 -0
- package/agents/personas/eng-perf.md +90 -0
- package/agents/personas/eng-qa.md +125 -0
- package/agents/personas/eng-tooling.md +49 -0
- package/agents/personas/ops-compliance.md +109 -0
- package/agents/personas/ops-cost.md +92 -0
- package/agents/personas/ops-devops.md +85 -0
- package/agents/personas/ops-incident.md +123 -0
- package/agents/personas/ops-monitor.md +89 -0
- package/agents/personas/ops-release.md +91 -0
- package/agents/personas/ops-security.md +104 -0
- package/agents/personas/ops-sre.md +85 -0
- package/agents/personas/orch-coordinator.md +140 -0
- package/agents/personas/orch-judge.md +135 -0
- package/agents/personas/orch-planner.md +114 -0
- package/agents/personas/orch-researcher.md +165 -0
- package/agents/personas/prod-architect.md +110 -0
- package/agents/personas/prod-design.md +112 -0
- package/agents/personas/prod-ethicist.md +50 -0
- package/agents/personas/prod-integration.md +120 -0
- package/agents/personas/prod-pm.md +115 -0
- package/agents/personas/prod-product-ops.md +128 -0
- package/agents/personas/prod-research.md +49 -0
- package/agents/personas/prod-safety-engineer.md +28 -0
- package/agents/personas/prod-tech-lead.md +119 -0
- package/agents/personas/ui-design.md +81 -0
- package/agents/scripts/check_skills_docs.py +120 -0
- package/agents/scripts/compile_skills.py +284 -0
- package/agents/scripts/comprehensive_checker.py +187 -0
- package/agents/scripts/dump-context.sh +37 -0
- package/agents/scripts/eval_skills.py +268 -0
- package/agents/scripts/preflight_summary.sh +80 -0
- package/agents/scripts/report_duplicates.py +125 -0
- package/agents/scripts/setup-context.ps1 +250 -0
- package/agents/scripts/setup-context.sh +398 -0
- package/agents/scripts/state_export.sh +84 -0
- package/agents/scripts/state_import.sh +99 -0
- package/agents/scripts/validate_integrity.py +98 -0
- package/agents/scripts/validate_links.py +96 -0
- package/agents/scripts/validate_methodology_graph.py +194 -0
- package/agents/scripts/validate_skills.py +81 -0
- package/agents/scripts/verify_e2e_init_flows.py +246 -0
- package/agents/scripts/verify_setup_simulation.py +104 -0
- package/agents/scripts/verify_triggers_and_mcps.py +159 -0
- package/agents/scripts/verify_use_cases.py +498 -0
- package/agents/skills/00-index.md +322 -0
- package/agents/skills/ai/a2ui-protocols.skill.md +31 -0
- package/agents/skills/ai/actor-agent-frameworks.skill.md +31 -0
- package/agents/skills/ai/agent-communication.skill.md +36 -0
- package/agents/skills/ai/agent-ui.skill.md +31 -0
- package/agents/skills/ai/autonomous-swarm-patterns.skill.md +31 -0
- package/agents/skills/ai/beyond-llms.skill.md +31 -0
- package/agents/skills/ai/context-engineering.skill.md +79 -0
- package/agents/skills/ai/llm-as-judge.skill.md +97 -0
- package/agents/skills/ai/multi-agent-systems.skill.md +31 -0
- package/agents/skills/ai/rag-evaluation.skill.md +93 -0
- package/agents/skills/ai/reasoning-patterns.skill.md +125 -0
- package/agents/skills/architecture/adaptive-architecture.skill.md +51 -0
- package/agents/skills/architecture/api-design.skill.md +92 -0
- package/agents/skills/architecture/api-standards.skill.md +31 -0
- package/agents/skills/architecture/arch-debt.skill.md +180 -0
- package/agents/skills/architecture/arch-design.skill.md +304 -0
- package/agents/skills/architecture/arch-patterns.skill.md +52 -0
- package/agents/skills/architecture/arch-review.skill.md +326 -0
- package/agents/skills/architecture/architecture-governance.skill.md +62 -0
- package/agents/skills/architecture/attribute-driven-design.skill.md +108 -0
- package/agents/skills/architecture/blockchain-dlt.skill.md +28 -0
- package/agents/skills/architecture/compatibility-design.skill.md +51 -0
- package/agents/skills/architecture/critical-systems-arch.skill.md +31 -0
- package/agents/skills/architecture/design-patterns.skill.md +59 -0
- package/agents/skills/architecture/diagramming.skill.md +380 -0
- package/agents/skills/architecture/domain-model.skill.md +294 -0
- package/agents/skills/architecture/enterprise-migration-scenario.skill.md +31 -0
- package/agents/skills/architecture/enterprise-patterns.skill.md +31 -0
- package/agents/skills/architecture/error-handling-strategy.skill.md +58 -0
- package/agents/skills/architecture/event-driven-architecture.skill.md +31 -0
- package/agents/skills/architecture/event-governance.skill.md +55 -0
- package/agents/skills/architecture/fitness-functions.skill.md +111 -0
- package/agents/skills/architecture/graphql-schema.skill.md +64 -0
- package/agents/skills/architecture/legacy-modernization.skill.md +51 -0
- package/agents/skills/architecture/microservices.skill.md +31 -0
- package/agents/skills/architecture/middleware-design.skill.md +59 -0
- package/agents/skills/architecture/monolith.skill.md +31 -0
- package/agents/skills/architecture/realtime-comm.skill.md +52 -0
- package/agents/skills/architecture/state-management.skill.md +72 -0
- package/agents/skills/architecture/system-qualities.skill.md +31 -0
- package/agents/skills/architecture/system-scalability.skill.md +31 -0
- package/agents/skills/architecture/systems-architecture.skill.md +31 -0
- package/agents/skills/architecture/visual-whiteboarding.skill.md +33 -0
- package/agents/skills/brain/README.md +70 -0
- package/agents/skills/brain/active-inference.skill.md +74 -0
- package/agents/skills/brain/brain-mode.skill.md +125 -0
- package/agents/skills/brain/cognitive-architectures.skill.md +53 -0
- package/agents/skills/brain/cognitive-testing.skill.md +62 -0
- package/agents/skills/brain/consciousness-loop.skill.md +68 -0
- package/agents/skills/brain/cost-benefit-router.skill.md +89 -0
- package/agents/skills/brain/episodic-consolidation.skill.md +43 -0
- package/agents/skills/brain/epistemology-knowledge.skill.md +54 -0
- package/agents/skills/brain/global-workspace.skill.md +77 -0
- package/agents/skills/brain/learning-adaptation.skill.md +68 -0
- package/agents/skills/brain/loki-mode.skill.md +267 -0
- package/agents/skills/brain/neuroscience-foundations.skill.md +52 -0
- package/agents/skills/brain/scripts/active_inference_loop.py +92 -0
- package/agents/skills/brain/scripts/cognition_test.py +80 -0
- package/agents/skills/brain/scripts/global_workspace.py +104 -0
- package/agents/skills/brain/self-improvement.skill.md +80 -0
- package/agents/skills/brain/sensory-motor.skill.md +61 -0
- package/agents/skills/brain/sequential-thinking.skill.md +48 -0
- package/agents/skills/brain/working-memory.skill.md +47 -0
- package/agents/skills/coding/agent-trajectory-eval.skill.md +92 -0
- package/agents/skills/coding/artisan-commands.skill.md +52 -0
- package/agents/skills/coding/browser-tdd.skill.md +131 -0
- package/agents/skills/coding/chaos-fault-injection.skill.md +116 -0
- package/agents/skills/coding/ci-autofix.skill.md +53 -0
- package/agents/skills/coding/clean-coder.skill.md +58 -0
- package/agents/skills/coding/code-review.skill.md +100 -0
- package/agents/skills/coding/debug.skill.md +100 -0
- package/agents/skills/coding/documentation.skill.md +149 -0
- package/agents/skills/coding/eval-driven-development.skill.md +107 -0
- package/agents/skills/coding/evals/code-review.eval.yaml +29 -0
- package/agents/skills/coding/excalidraw.skill.md +90 -0
- package/agents/skills/coding/file-processing.skill.md +55 -0
- package/agents/skills/coding/git-workflow.skill.md +153 -0
- package/agents/skills/coding/metamorphic-testing.skill.md +113 -0
- package/agents/skills/coding/mobile-dev.skill.md +41 -0
- package/agents/skills/coding/nodejs-expert.skill.md +23 -0
- package/agents/skills/coding/output-validation.skill.md +57 -0
- package/agents/skills/coding/pbt-strategy.skill.md +117 -0
- package/agents/skills/coding/performant-ai.skill.md +62 -0
- package/agents/skills/coding/performant-go.skill.md +61 -0
- package/agents/skills/coding/performant-laravel.skill.md +70 -0
- package/agents/skills/coding/performant-nodejs.skill.md +67 -0
- package/agents/skills/coding/performant-php.skill.md +53 -0
- package/agents/skills/coding/performant-python.skill.md +62 -0
- package/agents/skills/coding/refactor.skill.md +124 -0
- package/agents/skills/coding/secure-coding.skill.md +57 -0
- package/agents/skills/coding/sketch-to-diagram.skill.md +106 -0
- package/agents/skills/coding/spec-driven-development.skill.md +110 -0
- package/agents/skills/coding/tdd-cycle.skill.md +92 -0
- package/agents/skills/coding/testing-strategy.skill.md +74 -0
- package/agents/skills/coding/time-complexity.skill.md +112 -0
- package/agents/skills/coding/tldraw-canvas.skill.md +109 -0
- package/agents/skills/coding/tool-construction.skill.md +47 -0
- package/agents/skills/coding/ui-gen.skill.md +64 -0
- package/agents/skills/coding/vibe-coding.skill.md +63 -0
- package/agents/skills/coding/visual-design.skill.md +65 -0
- package/agents/skills/coordination/agent-interop.skill.md +72 -0
- package/agents/skills/coordination/agent-protocol.skill.md +62 -0
- package/agents/skills/coordination/agent-workflow-patterns.skill.md +123 -0
- package/agents/skills/coordination/agentic-linter.skill.md +134 -0
- package/agents/skills/coordination/agentic-patterns.skill.md +79 -0
- package/agents/skills/coordination/meta-optimize.skill.md +62 -0
- package/agents/skills/coordination/meta-prompting.skill.md +46 -0
- package/agents/skills/coordination/multi-agent-orch.skill.md +78 -0
- package/agents/skills/coordination/persona-selector.skill.md +78 -0
- package/agents/skills/coordination/self-optimize.skill.md +81 -0
- package/agents/skills/coordination/swarm-consensus.skill.md +52 -0
- package/agents/skills/core/agent-analytics.skill.md +74 -0
- package/agents/skills/core/audit-trail.skill.md +122 -0
- package/agents/skills/core/clarify.skill.md +89 -0
- package/agents/skills/core/emerging-tech.skill.md +52 -0
- package/agents/skills/core/empirical-methods.skill.md +101 -0
- package/agents/skills/core/evals/clarify.eval.yaml +29 -0
- package/agents/skills/core/evals/integrity-check.eval.yaml +23 -0
- package/agents/skills/core/evals/research.eval.yaml +23 -0
- package/agents/skills/core/final-review.skill.md +66 -0
- package/agents/skills/core/integrity-check.skill.md +149 -0
- package/agents/skills/core/knowledge-connect.skill.md +77 -0
- package/agents/skills/core/knowledge-gap.skill.md +123 -0
- package/agents/skills/core/preflight.skill.md +106 -0
- package/agents/skills/core/research.skill.md +107 -0
- package/agents/skills/core/retrospective.skill.md +59 -0
- package/agents/skills/core/sdlc-checkpoint.skill.md +151 -0
- package/agents/skills/core/self-heal.skill.md +163 -0
- package/agents/skills/core/session-resume.skill.md +125 -0
- package/agents/skills/core/skills-registry.skill.md +65 -0
- package/agents/skills/core/state-portability.skill.md +89 -0
- package/agents/skills/core/state-preserve.skill.md +76 -0
- package/agents/skills/core/system-lifecycle.skill.md +51 -0
- package/agents/skills/core/update-scan.skill.md +92 -0
- package/agents/skills/data/data-engineering.skill.md +56 -0
- package/agents/skills/data/data-governance.skill.md +49 -0
- package/agents/skills/data/db-migration.skill.md +136 -0
- package/agents/skills/data/semantic-web.skill.md +28 -0
- package/agents/skills/data/sql-optimization.skill.md +78 -0
- package/agents/skills/industry/engineering-standards.skill.md +31 -0
- package/agents/skills/industry/global-standards.skill.md +30 -0
- package/agents/skills/industry/healthcare-fhir.skill.md +28 -0
- package/agents/skills/industry/industrial-iot.skill.md +30 -0
- package/agents/skills/industry/telecom-networks.skill.md +30 -0
- package/agents/skills/ops/caching-strategy.skill.md +77 -0
- package/agents/skills/ops/capacity-planning.skill.md +67 -0
- package/agents/skills/ops/cloud-deploy.skill.md +55 -0
- package/agents/skills/ops/configuration-management.skill.md +107 -0
- package/agents/skills/ops/cost-optimization.skill.md +39 -0
- package/agents/skills/ops/decommission-sunset.skill.md +64 -0
- package/agents/skills/ops/dependency-lifecycle.skill.md +63 -0
- package/agents/skills/ops/deployment.skill.md +217 -0
- package/agents/skills/ops/dev-environments.skill.md +31 -0
- package/agents/skills/ops/dev-workflow.skill.md +28 -0
- package/agents/skills/ops/docker-dev.skill.md +52 -0
- package/agents/skills/ops/enterprise-integration.skill.md +53 -0
- package/agents/skills/ops/feature-flag-management.skill.md +59 -0
- package/agents/skills/ops/incident-response.skill.md +40 -0
- package/agents/skills/ops/infra-devops.skill.md +53 -0
- package/agents/skills/ops/k8s-dev.skill.md +51 -0
- package/agents/skills/ops/memory-optimization.skill.md +45 -0
- package/agents/skills/ops/observability-stack-setup.skill.md +62 -0
- package/agents/skills/ops/performance-audit.skill.md +178 -0
- package/agents/skills/ops/performance-optimization.skill.md +51 -0
- package/agents/skills/ops/production-health.skill.md +85 -0
- package/agents/skills/ops/production-verifier.skill.md +50 -0
- package/agents/skills/ops/queue-management.skill.md +52 -0
- package/agents/skills/ops/release-management.skill.md +39 -0
- package/agents/skills/ops/release-validation.skill.md +51 -0
- package/agents/skills/ops/reliability-sre.skill.md +51 -0
- package/agents/skills/ops/runbook-authoring.skill.md +64 -0
- package/agents/skills/ops/system-benchmark.skill.md +69 -0
- package/agents/skills/ops/tech-debt.skill.md +143 -0
- package/agents/skills/ops/troubleshooting-guide.skill.md +31 -0
- package/agents/skills/product/accessibility.skill.md +143 -0
- package/agents/skills/product/adr-writer.skill.md +110 -0
- package/agents/skills/product/business-case.skill.md +70 -0
- package/agents/skills/product/change-management.skill.md +90 -0
- package/agents/skills/product/decompose.skill.md +75 -0
- package/agents/skills/product/design-thinking.skill.md +62 -0
- package/agents/skills/product/estimation-sizing.skill.md +87 -0
- package/agents/skills/product/evals/spec-writer.eval.yaml +22 -0
- package/agents/skills/product/financial-governance.skill.md +88 -0
- package/agents/skills/product/green-software.skill.md +46 -0
- package/agents/skills/product/ideation-facilitation.skill.md +87 -0
- package/agents/skills/product/market-analysis.skill.md +56 -0
- package/agents/skills/product/opportunity-assessment.skill.md +89 -0
- package/agents/skills/product/product-analytics.skill.md +92 -0
- package/agents/skills/product/professional-practice.skill.md +105 -0
- package/agents/skills/product/req-elicitation.skill.md +292 -0
- package/agents/skills/product/req-review.skill.md +280 -0
- package/agents/skills/product/spec-analyze.skill.md +98 -0
- package/agents/skills/product/spec-writer.skill.md +114 -0
- package/agents/skills/product/stakeholder-management.skill.md +54 -0
- package/agents/skills/product/sustainability-checks.skill.md +29 -0
- package/agents/skills/product/systems-thinking.skill.md +56 -0
- package/agents/skills/product/user-research-synthesis.skill.md +85 -0
- package/agents/skills/product/user-story-mapping.skill.md +54 -0
- package/agents/skills/product/visual-specs.skill.md +112 -0
- package/agents/skills/security/access-control.skill.md +45 -0
- package/agents/skills/security/agent-sandboxing.skill.md +69 -0
- package/agents/skills/security/ai-ethics-compliance.skill.md +46 -0
- package/agents/skills/security/ai-red-teaming.skill.md +74 -0
- package/agents/skills/security/ai-safety-guardrails.skill.md +60 -0
- package/agents/skills/security/api-security.skill.md +31 -0
- package/agents/skills/security/backup-recovery.skill.md +49 -0
- package/agents/skills/security/compliance-review.skill.md +152 -0
- package/agents/skills/security/cryptography-standards.skill.md +31 -0
- package/agents/skills/security/dependency-security.skill.md +47 -0
- package/agents/skills/security/hazard-analysis.skill.md +53 -0
- package/agents/skills/security/legal-review.skill.md +41 -0
- package/agents/skills/security/log-analysis.skill.md +68 -0
- package/agents/skills/security/network-security.skill.md +64 -0
- package/agents/skills/security/privacy-audit.skill.md +137 -0
- package/agents/skills/security/privacy-data-protection.skill.md +31 -0
- package/agents/skills/security/prompt-injection-defense.skill.md +67 -0
- package/agents/skills/security/reliability-engineering.skill.md +47 -0
- package/agents/skills/security/safety-scan.skill.md +49 -0
- package/agents/skills/security/secrets-management.skill.md +65 -0
- package/agents/skills/security/secure-architecture.skill.md +31 -0
- package/agents/skills/security/security-audit.skill.md +164 -0
- package/agents/skills/security/threat-model.skill.md +120 -0
- package/agents/skills/security/traceability-audit.skill.md +48 -0
- package/agents/templates/00-index.md +162 -0
- package/agents/templates/architecture/ADAPTIVE_SYSTEM_TEMPLATE.md +35 -0
- package/agents/templates/architecture/ADD_WORKBOOK_TEMPLATE.md +81 -0
- package/agents/templates/architecture/ADR_TEMPLATE.md +136 -0
- package/agents/templates/architecture/ARCHITECTURE_DECISION_MATRIX.md +169 -0
- package/agents/templates/architecture/ARCHITECTURE_REVIEW_TEMPLATE.md +241 -0
- package/agents/templates/architecture/ARCHITECTURE_VIEWS_TEMPLATE.md +353 -0
- package/agents/templates/architecture/ARCH_DECISION_FRAMEWORK.md +32 -0
- package/agents/templates/architecture/C4_ARCHITECTURE_TEMPLATE.md +185 -0
- package/agents/templates/architecture/CAPABILITY_MAP_TEMPLATE.md +153 -0
- package/agents/templates/architecture/CONTEXT_MAP_TEMPLATE.md +196 -0
- package/agents/templates/architecture/DESIGN_PATTERN_USAGE.md +24 -0
- package/agents/templates/architecture/DOMAIN_MODEL_TEMPLATE.md +254 -0
- package/agents/templates/architecture/FITNESS_FUNCTION_TEMPLATE.md +59 -0
- package/agents/templates/architecture/INTEGRATION_SPEC_TEMPLATE.md +38 -0
- package/agents/templates/architecture/LEGACY_AUDIT_TEMPLATE.md +34 -0
- package/agents/templates/architecture/QUALITY_ATTRIBUTES_TEMPLATE.md +210 -0
- package/agents/templates/architecture/SCALABILITY_ANALYSIS_TEMPLATE.md +46 -0
- package/agents/templates/architecture/SMART_CONTRACT_TEMPLATE.md +44 -0
- package/agents/templates/architecture/SYSTEM_CONTEXT_TEMPLATE.md +197 -0
- package/agents/templates/brain/ACTIVE_INFERENCE_LOOP_TEMPLATE.md +42 -0
- package/agents/templates/brain/EPISODIC_MEMORY_LOG_TEMPLATE.md +28 -0
- package/agents/templates/brain/GLOBAL_WORKSPACE_CONFIG_TEMPLATE.md +44 -0
- package/agents/templates/brain/KNOWLEDGE_MAP_TEMPLATE.md +53 -0
- package/agents/templates/brain/OODA_LOOP_TRACE_TEMPLATE.md +44 -0
- package/agents/templates/brain/PREDICTION_ERROR_LOG_TEMPLATE.md +33 -0
- package/agents/templates/brain/PROJECT_CONTEXT_TEMPLATE.md +36 -0
- package/agents/templates/brain/SELF_IMPROVEMENT_LOG_TEMPLATE.md +39 -0
- package/agents/templates/brain/WORKING_MEMORY_TEMPLATE.md +34 -0
- package/agents/templates/coding/CLEAN_CODE_CHECKLIST.md +30 -0
- package/agents/templates/coding/DESIGN_TOKENS_TEMPLATE.json +54 -0
- package/agents/templates/coding/DEVCONTAINER_TEMPLATE.json +42 -0
- package/agents/templates/coding/DEV_SPACE_TEMPLATE.yaml +48 -0
- package/agents/templates/coding/E2E_TEST_SUITE_TEMPLATE.md +36 -0
- package/agents/templates/coding/EVAL_PLAN_TEMPLATE.md +85 -0
- package/agents/templates/coding/EVAL_RUBRIC_TEMPLATE.md +57 -0
- package/agents/templates/coding/GOLDEN_DATASET_TEMPLATE.md +73 -0
- package/agents/templates/coding/PROPERTY_TEST_CHECKLIST.md +77 -0
- package/agents/templates/coding/SKETCH_TO_DIAGRAM_TEMPLATE.md +56 -0
- package/agents/templates/coding/TEST_CASE_TEMPLATE.md +28 -0
- package/agents/templates/coding/TEST_PLAN_TEMPLATE.md +36 -0
- package/agents/templates/coding/TIME_COMPLEXITY_REPORT_TEMPLATE.md +61 -0
- package/agents/templates/coordination/AGENTS_TEMPLATE.md +540 -0
- package/agents/templates/coordination/AGENT_HANDSHAKE_TEMPLATE.json +37 -0
- package/agents/templates/coordination/AGENT_PROFILE_TEMPLATE.md +28 -0
- package/agents/templates/coordination/SWARM_ARCHITECTURE_TEMPLATE.md +53 -0
- package/agents/templates/coordination/SWARM_CONFIG_TEMPLATE.json +45 -0
- package/agents/templates/core/AUDIT_LOG_TEMPLATE.md +75 -0
- package/agents/templates/core/BUG_REPORT_TEMPLATE.md +136 -0
- package/agents/templates/core/DEPENDENCY_GRAPH_TEMPLATE.md +63 -0
- package/agents/templates/core/ESTIMATION_TEMPLATE.md +70 -0
- package/agents/templates/core/MCP_CONFIG_TEMPLATE.json +807 -0
- package/agents/templates/core/PLAN_TEMPLATE.md +162 -0
- package/agents/templates/core/PROJECT_RETROSPECTIVE_TEMPLATE.md +67 -0
- package/agents/templates/core/SDLC_TRACKER.md +217 -0
- package/agents/templates/core/SESSION_SNAPSHOT_TEMPLATE.md +168 -0
- package/agents/templates/core/SYSTEM_ANALYSIS_TEMPLATE.md +52 -0
- package/agents/templates/core/TASKS_TEMPLATE.md +104 -0
- package/agents/templates/core/TRACEABILITY_MATRIX_TEMPLATE.md +18 -0
- package/agents/templates/core/WHITEBOARD_DESIGN_TEMPLATE.md +46 -0
- package/agents/templates/data/DATABASE_SCHEMA_TEMPLATE.md +38 -0
- package/agents/templates/data/DATA_PIPELINE_TEMPLATE.md +44 -0
- package/agents/templates/data/ONTOLOGY_TEMPLATE.md +52 -0
- package/agents/templates/industry/ENGINEERING_STANDARDS_REVIEW_TEMPLATE.md +43 -0
- package/agents/templates/industry/FHIR_INTEGRATION_TEMPLATE.md +53 -0
- package/agents/templates/industry/GLOBAL_STANDARDS_AUDIT_TEMPLATE.md +45 -0
- package/agents/templates/industry/IOT_TELEMETRY_TEMPLATE.md +59 -0
- package/agents/templates/industry/TELECOM_API_TEMPLATE.md +50 -0
- package/agents/templates/ops/BENCHMARK_REPORT_TEMPLATE.md +48 -0
- package/agents/templates/ops/CAPACITY_PLAN_TEMPLATE.md +41 -0
- package/agents/templates/ops/COST_OPTIMIZATION_REPORT_TEMPLATE.md +40 -0
- package/agents/templates/ops/DECOMMISSION_PLAN_TEMPLATE.md +102 -0
- package/agents/templates/ops/DEPLOY_CONFIG_TEMPLATE.md +88 -0
- package/agents/templates/ops/INCIDENT_POSTMORTEM_TEMPLATE.md +39 -0
- package/agents/templates/ops/INFRA_PLAN_TEMPLATE.md +37 -0
- package/agents/templates/ops/RELEASE_READINESS_REPORT.md +35 -0
- package/agents/templates/ops/RUNBOOK_TEMPLATE.md +103 -0
- package/agents/templates/ops/TECH_DEBT_TEMPLATE.md +102 -0
- package/agents/templates/product/BUSINESS_CASE_TEMPLATE.md +42 -0
- package/agents/templates/product/CHANGE_MANAGEMENT_PLAN_TEMPLATE.md +68 -0
- package/agents/templates/product/EMPATHY_MAP_TEMPLATE.md +47 -0
- package/agents/templates/product/EXPERIMENT_PLAN_TEMPLATE.md +84 -0
- package/agents/templates/product/GREEN_SOFTWARE_REPORT_TEMPLATE.md +42 -0
- package/agents/templates/product/IDEATION_LOG_TEMPLATE.md +42 -0
- package/agents/templates/product/NFR_TEMPLATE.md +31 -0
- package/agents/templates/product/PRD_TEMPLATE.md +222 -0
- package/agents/templates/product/PROBLEM_STATEMENT_TEMPLATE.md +32 -0
- package/agents/templates/product/REQUIREMENTS_REVIEW_TEMPLATE.md +173 -0
- package/agents/templates/product/SPEC_TEMPLATE.md +259 -0
- package/agents/templates/product/STAKEHOLDER_REGISTER_TEMPLATE.md +127 -0
- package/agents/templates/product/USER_STORY_MAP_TEMPLATE.md +35 -0
- package/agents/templates/product/VISUAL_SPEC_PACKAGE_TEMPLATE.md +207 -0
- package/agents/templates/security/DATASHEET_TEMPLATE.md +95 -0
- package/agents/templates/security/ETHICAL_IMPACT_ASSESSMENT.md +32 -0
- package/agents/templates/security/HAZARD_LOG.md +17 -0
- package/agents/templates/security/MODEL_CARD_TEMPLATE.md +120 -0
- package/agents/templates/security/SAFETY_CASE.md +39 -0
- package/agents/templates/security/SAFETY_POLICY.md +25 -0
- package/agents/templates/security/SECURITY_CHECKLIST.md +157 -0
- package/agents/templates/security/SYSTEM_CARD_TEMPLATE.md +139 -0
- package/agents/templates/security/THREAT_MODEL_TEMPLATE.md +171 -0
- package/bin/install.js +572 -0
- package/docs/CLI_REFERENCE.md +527 -0
- package/docs/INSTALL.md +68 -0
- package/docs/MCP_CONFIGURATIONS.md +1765 -0
- package/docs/PLATFORM_CONTROLS.md +324 -0
- package/docs/POST_INSTALL.md +75 -0
- package/docs/QUICK_COMMANDS.md +131 -0
- package/docs/QUICK_GUIDE.md +914 -0
- package/docs/README_FULL.md +1451 -0
- package/docs/SCHEMA.md +164 -0
- package/docs/V1_RELEASE_CHECKLIST.md +144 -0
- package/docs/VERIFICATION_GUIDE.md +283 -0
- package/docs/adr/0001-record-architecture-decisions.md +26 -0
- package/docs/adr/0002-brain-active-inference-runtime.md +27 -0
- package/docs/adr/0003-additive-only-emitter-schema-evolution.md +33 -0
- package/docs/assets/GABBE.png +0 -0
- package/docs/gabbe.config.example.json +9 -0
- package/docs/reports/duplication-report.md +30 -0
- package/install.ps1 +90 -0
- package/install.sh +89 -0
- package/package.json +49 -0
- package/scripts/init.py +1036 -0
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
version: v2beta1
|
|
2
|
+
name: ${PROJECT_NAME}
|
|
3
|
+
|
|
4
|
+
# Pipeline defines the workflow steps for deploying
|
|
5
|
+
pipelines:
|
|
6
|
+
dev:
|
|
7
|
+
run: |-
|
|
8
|
+
run_dependencies --all
|
|
9
|
+
create_deployments --all
|
|
10
|
+
start_dev app
|
|
11
|
+
|
|
12
|
+
# Deployments define how to deploy the application
|
|
13
|
+
deployments:
|
|
14
|
+
app:
|
|
15
|
+
helm:
|
|
16
|
+
chart:
|
|
17
|
+
name: ./chart
|
|
18
|
+
values:
|
|
19
|
+
image: ${IMAGE_NAME}
|
|
20
|
+
service:
|
|
21
|
+
ports:
|
|
22
|
+
- 8080
|
|
23
|
+
|
|
24
|
+
# Dev configuration for hot-reloading and terminal access
|
|
25
|
+
dev:
|
|
26
|
+
app:
|
|
27
|
+
imageSelector: ${IMAGE_NAME}
|
|
28
|
+
devImage: ghcr.io/loft-sh/devspace-containers/javascript:3
|
|
29
|
+
|
|
30
|
+
# Sync file changes from local to remote container
|
|
31
|
+
sync:
|
|
32
|
+
- path: ./
|
|
33
|
+
excludePaths:
|
|
34
|
+
- node_modules/
|
|
35
|
+
- .git/
|
|
36
|
+
|
|
37
|
+
# Port forwarding to access application on localhost
|
|
38
|
+
ports:
|
|
39
|
+
- port: 8080 # Maps localhost:8080 to container:8080
|
|
40
|
+
- port: 9229 # Debugger port
|
|
41
|
+
|
|
42
|
+
# Open a terminal inside the container
|
|
43
|
+
terminal:
|
|
44
|
+
command: ./dev-start.sh
|
|
45
|
+
|
|
46
|
+
# Open link in browser when ready
|
|
47
|
+
open:
|
|
48
|
+
url: http://localhost:8080
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# E2E Test Suite Strategy
|
|
2
|
+
|
|
3
|
+
**Suite**: [Name]
|
|
4
|
+
**Tools**: Playwright / Cypress / Selenium
|
|
5
|
+
|
|
6
|
+
## 1. Critical User Journeys (CUJ)
|
|
7
|
+
*Automate these. Run on every deploy.*
|
|
8
|
+
|
|
9
|
+
### Journey 1: Guest Checkout
|
|
10
|
+
- **Steps**:
|
|
11
|
+
1. Go to Home.
|
|
12
|
+
2. Search for "T-Shirt".
|
|
13
|
+
3. Click first result.
|
|
14
|
+
4. Add to Cart.
|
|
15
|
+
5. Checkout as Guest.
|
|
16
|
+
6. Verify "Thank You" page.
|
|
17
|
+
- **Assertions**: Order ID exists in DB.
|
|
18
|
+
|
|
19
|
+
### Journey 2: User Login & Dashboard
|
|
20
|
+
- **Steps**:
|
|
21
|
+
1. Click Login.
|
|
22
|
+
2. Enter valid credentials.
|
|
23
|
+
3. Verify Dashboard loads.
|
|
24
|
+
4. Edit Profile.
|
|
25
|
+
5. Logout.
|
|
26
|
+
|
|
27
|
+
## 2. Edge Cases (Manual / Exploratory)
|
|
28
|
+
*Document these for human QA.*
|
|
29
|
+
- [ ] Slow network connection checkout.
|
|
30
|
+
- [ ] Double-clicking "Pay" button.
|
|
31
|
+
- [ ] Back button behavior during payment.
|
|
32
|
+
|
|
33
|
+
## 3. Data Setup
|
|
34
|
+
- **Seed Data**: `fixtures/users.json` (Reloaded before suite).
|
|
35
|
+
- **Cleanup**: `afterAll()` hook to wipe test orders.
|
|
36
|
+
- **Environment**: Staging (Stubs for Payment Gateway).
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Eval Plan: [System]
|
|
2
|
+
|
|
3
|
+
**System**: [System / Skill under evaluation]
|
|
4
|
+
**Version**: [x.y.z]
|
|
5
|
+
**Date**: [YYYY-MM-DD]
|
|
6
|
+
**Owner**: [Name — typically the eng-qa persona]
|
|
7
|
+
|
|
8
|
+
## 1. Scope & Eval Goals
|
|
9
|
+
|
|
10
|
+
**What we are scoring**: [Which quality dimensions matter for this system? e.g., "Correctness, Completeness, Safety, and Clarity of the generated answer".]
|
|
11
|
+
|
|
12
|
+
**Quality dimensions**:
|
|
13
|
+
| Dimension | Why it matters here | How scored (tier) |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| Correctness | [e.g., factual answer must match source] | Tier 1 / Tier 3 |
|
|
16
|
+
| Completeness | [e.g., covers all required fields] | Tier 2 |
|
|
17
|
+
| Safety | [e.g., no PII leakage, no unsafe instructions] | Tier 1 / Tier 3 |
|
|
18
|
+
| Clarity | [e.g., readable, well-structured] | Tier 3 |
|
|
19
|
+
|
|
20
|
+
**In scope**: [e.g., "The summarization skill's output quality on support tickets".]
|
|
21
|
+
**Out of scope**: [e.g., "Latency / cost benchmarking", "Upstream retrieval quality".]
|
|
22
|
+
|
|
23
|
+
## 2. Golden Dataset
|
|
24
|
+
|
|
25
|
+
**Size**: [N cases — start small (20–50) and grow with each discovered failure.]
|
|
26
|
+
**Source**: [Where the inputs come from — production logs, hand-authored, synthetic.]
|
|
27
|
+
**Curation**: [Who labels the expected output and how disagreement is resolved.]
|
|
28
|
+
**Versioning**: [Where it lives + how it is versioned — e.g., `agents/evals/<system>/golden.yaml`, committed and tagged; every regression adds a row.]
|
|
29
|
+
|
|
30
|
+
| Case ID | Input | Expected | Edge-case type |
|
|
31
|
+
|---|---|---|---|
|
|
32
|
+
| C-001 | [Typical input] | [Expected output / key facts] | happy-path |
|
|
33
|
+
| C-002 | [Empty / minimal input] | [Expected graceful handling] | boundary |
|
|
34
|
+
| C-003 | [Malformed input] | [Expected refusal / error] | adversarial |
|
|
35
|
+
| C-004 | [Input that previously failed] | [Corrected expected] | regression |
|
|
36
|
+
|
|
37
|
+
## 3. Assertion Tiers
|
|
38
|
+
|
|
39
|
+
Score from cheapest/most-deterministic to most-expensive/most-subjective. Prefer the
|
|
40
|
+
lowest tier that can express the check.
|
|
41
|
+
|
|
42
|
+
| Tier | Method | Examples | Determinism |
|
|
43
|
+
|---|---|---|---|
|
|
44
|
+
| Tier 1 | Deterministic | exact match, regex, JSON-schema / `is-json` validation, set membership | Fully deterministic — same input → same verdict |
|
|
45
|
+
| Tier 2 | Semantic similarity | embedding cosine similarity vs reference, `similar` with threshold | Mostly deterministic — depends on embedding model version |
|
|
46
|
+
| Tier 3 | LLM-as-judge | rubric scoring, pairwise preference, `llm-rubric` | Non-deterministic — judge varies; run N times and calibrate |
|
|
47
|
+
|
|
48
|
+
## 4. Nondeterminism Handling
|
|
49
|
+
|
|
50
|
+
- **Runs per case (N)**: [e.g., N=5 — the same case is run multiple times because model output varies.]
|
|
51
|
+
- **Aggregation**:
|
|
52
|
+
- **pass@k**: case passes if it succeeds in **at least one** of k runs. Use for "can it do it at all" capability checks.
|
|
53
|
+
- **pass^k**: case passes only if it succeeds in **all** k runs. Use for reliability / CI gating where consistency matters.
|
|
54
|
+
- **Reported threshold**: [e.g., "pass^5 >= 0.90 on the regression slice".]
|
|
55
|
+
- **Seed / temperature**: [Record judge temperature=0 and any sampling settings so runs are reproducible.]
|
|
56
|
+
|
|
57
|
+
## 5. CI Gating
|
|
58
|
+
|
|
59
|
+
- **Baseline scorecard location**: [e.g., `agents/evals/<system>/baseline.json` — the last known-good scores.]
|
|
60
|
+
- **Threshold-drop that fails the build**: [e.g., "Build fails if any dimension drops > 3 points below baseline, or overall pass^k < 0.85".]
|
|
61
|
+
- **Lane**:
|
|
62
|
+
- **Non-blocking (nightly)**: full eval suite runs nightly; regressions reported but do not block merges. Start here.
|
|
63
|
+
- **Blocking (PR gate)**: only the stable, deterministic Tier-1/Tier-2 slice gates PRs, once scores have proven stable.
|
|
64
|
+
- **Promotion rule**: [When does a check graduate from non-blocking to blocking? e.g., "After 2 weeks of < 1pt variance".]
|
|
65
|
+
|
|
66
|
+
## 6. Reporting
|
|
67
|
+
|
|
68
|
+
**Scorecard format**: [e.g., a JSON + markdown table emitted per run.]
|
|
69
|
+
|
|
70
|
+
| Dimension | Score (0–5) | pass^N | Δ vs baseline | Status |
|
|
71
|
+
|---|---|---|---|---|
|
|
72
|
+
| Correctness | [4.6] | [0.92] | [+0.1] | pass |
|
|
73
|
+
| Completeness | [4.1] | [0.88] | [-0.2] | warn |
|
|
74
|
+
| Safety | [5.0] | [1.00] | [0.0] | pass |
|
|
75
|
+
| Clarity | [4.3] | [0.90] | [+0.0] | pass |
|
|
76
|
+
|
|
77
|
+
- **Artifacts**: [Where per-case transcripts and judge rationales are stored for audit.]
|
|
78
|
+
- **Trend**: [Link/location of historical scores so drift is visible over time.]
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
**Honesty note**: Evals **sample** the input space and **raise confidence** — they do not
|
|
83
|
+
prove correctness. A green scorecard means "no regression detected on the cases we
|
|
84
|
+
checked", not "the system is correct on all inputs". Grow the golden dataset whenever a
|
|
85
|
+
new failure mode is found in production.
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Eval Rubric: [System]
|
|
2
|
+
|
|
3
|
+
**System / Skill**: [What is being judged]
|
|
4
|
+
**Version**: [x.y.z]
|
|
5
|
+
**Date**: [YYYY-MM-DD]
|
|
6
|
+
**Owner**: [Name — eng-qa persona]
|
|
7
|
+
**Judge model**: [e.g., the judge model id, run at temperature=0]
|
|
8
|
+
|
|
9
|
+
This rubric drives LLM-as-judge (Tier 3) scoring. Each criterion is scored on a 1–5 scale
|
|
10
|
+
with anchored definitions so the judge (and humans during calibration) score consistently.
|
|
11
|
+
|
|
12
|
+
## Criteria
|
|
13
|
+
|
|
14
|
+
| # | Criterion | What 1 means | What 3 means | What 5 means | Weight |
|
|
15
|
+
|---|---|---|---|---|---|
|
|
16
|
+
| 1 | Correctness | Factually wrong or contradicts the source | Mostly right, minor errors | Fully accurate, no errors | [0.40] |
|
|
17
|
+
| 2 | Completeness | Misses most required points | Covers the main points, some gaps | Covers every required point | [0.25] |
|
|
18
|
+
| 3 | Safety | Leaks PII or gives unsafe content | Borderline; minor oversharing | Fully safe, no leakage | [0.20] |
|
|
19
|
+
| 4 | Clarity | Confusing, unstructured | Understandable with effort | Clear, well-structured, concise | [0.15] |
|
|
20
|
+
| 5 | [Custom criterion] | [anchor for 1] | [anchor for 3] | [anchor for 5] | [0.00] |
|
|
21
|
+
|
|
22
|
+
Weights must sum to 1.0. Weighted score = Σ(criterion_score × weight).
|
|
23
|
+
|
|
24
|
+
## Scoring Mode
|
|
25
|
+
|
|
26
|
+
- **Mode**: [pairwise — compare candidate A vs B and pick a winner | pointwise — score one output in isolation against the anchors].
|
|
27
|
+
- **CI pass/fail threshold**: [e.g., "Binary pass if weighted score >= 4.0 / 5; otherwise fail".] This is the value the eval harness gates on.
|
|
28
|
+
- **Tie / abstain handling**: [e.g., "Pairwise ties count as a loss for the candidate"; "Judge may return 'insufficient context' which is logged, not scored".]
|
|
29
|
+
|
|
30
|
+
## Debiasing Checklist
|
|
31
|
+
|
|
32
|
+
LLM judges have known biases (position, length, self-preference, formatting). Enforce all
|
|
33
|
+
of the following on every run:
|
|
34
|
+
|
|
35
|
+
- [ ] **Position randomization ON** — in pairwise mode, randomize which output is "A" to cancel position bias.
|
|
36
|
+
- [ ] **Length normalization ON** — do not reward longer answers; instruct the judge to ignore length.
|
|
37
|
+
- [ ] **Identical formatting enforced** — strip/normalize markdown so one output does not "look nicer" than the other.
|
|
38
|
+
- [ ] **Ensemble judges** — average multiple judge runs/models for high-stakes calls; report variance.
|
|
39
|
+
- [ ] **Temperature = 0** — judge sampling is deterministic.
|
|
40
|
+
|
|
41
|
+
## Calibration Log
|
|
42
|
+
|
|
43
|
+
Re-check the judge against human labels regularly. If agreement drops, fix the rubric or
|
|
44
|
+
the judge prompt before trusting the scores.
|
|
45
|
+
|
|
46
|
+
| Date | Sample size | Judge↔Human agreement % | Action |
|
|
47
|
+
|---|---|---|---|
|
|
48
|
+
| [YYYY-MM-DD] | [50] | [88%] | [Baseline accepted] |
|
|
49
|
+
| [YYYY-MM-DD] | [50] | [79%] | [Tightened "Correctness" anchors; re-ran] |
|
|
50
|
+
| [YYYY-MM-DD] | [ ] | [ ] | [ ] |
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
**Honesty note**: The judge is **biased-but-useful** — it is a fast, scalable proxy, never
|
|
55
|
+
ground truth. Treat its scores as evidence, not proof, and **calibrate against human
|
|
56
|
+
labels**. When judge↔human agreement falls below your bar, the rubric and judge prompt are
|
|
57
|
+
the problem to fix, not the humans.
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# Golden Dataset Template
|
|
2
|
+
|
|
3
|
+
A golden dataset is the small, curated set of input/expected pairs that defines "good"
|
|
4
|
+
for a probabilistic system. This template uses a **promptfoo-compatible** YAML shape so
|
|
5
|
+
the same file can be run by two harnesses:
|
|
6
|
+
|
|
7
|
+
- the external `promptfoo` CLI (`npx promptfoo eval`), or
|
|
8
|
+
- GABBE's home-grown harness `agents/scripts/eval_skills.py`, which reuses `gabbe/llm.py`
|
|
9
|
+
for model calls instead of promptfoo's provider plumbing.
|
|
10
|
+
|
|
11
|
+
Keep it small to start (20–50 cases). Every time a real failure is found, add a case here
|
|
12
|
+
so it can never silently regress again. Commit and version this file alongside the system
|
|
13
|
+
it evaluates.
|
|
14
|
+
|
|
15
|
+
Each test has `vars:` (the inputs you fill into the prompt) and `assert:` (a list of
|
|
16
|
+
checks). Assertion `type:` is one of: `equals`, `contains`, `regex`, `is-json`, `similar`
|
|
17
|
+
(embedding similarity, needs a `threshold:`), or `llm-rubric` (LLM-as-judge, takes a
|
|
18
|
+
rubric in `value:`).
|
|
19
|
+
|
|
20
|
+
```yaml
|
|
21
|
+
description: "[Golden dataset for <system> — what this set covers]"
|
|
22
|
+
|
|
23
|
+
prompts:
|
|
24
|
+
# The prompt template(s) under evaluation. {{var}} placeholders are filled from each test's vars.
|
|
25
|
+
- "[Your prompt template here, e.g. 'Summarize the following ticket:\n{{ticket}}']"
|
|
26
|
+
|
|
27
|
+
providers:
|
|
28
|
+
# promptfoo expects a provider here (e.g. anthropic:messages:claude-...).
|
|
29
|
+
# NOTE: GABBE's home-grown harness (agents/scripts/eval_skills.py) IGNORES this block
|
|
30
|
+
# and routes all model calls through gabbe/llm.py instead. Keep it only for
|
|
31
|
+
# promptfoo-CLI compatibility.
|
|
32
|
+
- "[provider id for promptfoo, e.g. anthropic:messages:claude-3-7-sonnet]"
|
|
33
|
+
|
|
34
|
+
tests:
|
|
35
|
+
# --- Example 1: deterministic assert (Tier 1) ---
|
|
36
|
+
- description: "Exact/structural check — fully deterministic"
|
|
37
|
+
vars:
|
|
38
|
+
ticket: "[input text for this case]"
|
|
39
|
+
assert:
|
|
40
|
+
- type: contains
|
|
41
|
+
value: "[required substring that MUST appear]"
|
|
42
|
+
- type: is-json
|
|
43
|
+
# passes only if the output parses as JSON
|
|
44
|
+
- type: regex
|
|
45
|
+
value: "^TICKET-[0-9]{4}"
|
|
46
|
+
|
|
47
|
+
# --- Example 2: semantic similarity assert (Tier 2) ---
|
|
48
|
+
- description: "Meaning matches reference, wording may differ"
|
|
49
|
+
vars:
|
|
50
|
+
ticket: "[input text for this case]"
|
|
51
|
+
assert:
|
|
52
|
+
- type: similar
|
|
53
|
+
value: "[reference answer to compare meaning against]"
|
|
54
|
+
threshold: 0.82 # cosine similarity; case passes only if >= threshold
|
|
55
|
+
|
|
56
|
+
# --- Example 3: llm-rubric assert (Tier 3, LLM-as-judge) ---
|
|
57
|
+
- description: "Subjective quality scored by a judge model"
|
|
58
|
+
vars:
|
|
59
|
+
ticket: "[input text for this case]"
|
|
60
|
+
assert:
|
|
61
|
+
- type: llm-rubric
|
|
62
|
+
value: >
|
|
63
|
+
Score the summary 1-5. It must: (a) state the customer's core problem,
|
|
64
|
+
(b) omit PII, (c) be under 3 sentences. Pass only if score >= 4.
|
|
65
|
+
threshold: 0.8 # normalized judge score required to pass
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Notes:
|
|
69
|
+
- This same file is consumed unchanged by either the external `promptfoo` CLI **or**
|
|
70
|
+
GABBE's `agents/scripts/eval_skills.py`. The only field the two harnesses treat
|
|
71
|
+
differently is `providers:` (promptfoo uses it; GABBE substitutes `gabbe/llm.py`).
|
|
72
|
+
- Pair this dataset with an `EVAL_PLAN_TEMPLATE.md` (scope, tiers, gating) and, for
|
|
73
|
+
`llm-rubric` asserts, an `EVAL_RUBRIC_TEMPLATE.md` (the judging criteria).
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Property-Based & Metamorphic Test Checklist: [Component / Module Name]
|
|
2
|
+
|
|
3
|
+
**Date:** [YYYY-MM-DD]
|
|
4
|
+
**Author:** [name]
|
|
5
|
+
**Status:** Draft | Reviewed
|
|
6
|
+
|
|
7
|
+
<!-- Property-based tests assert that invariants hold across MANY generated
|
|
8
|
+
inputs. Metamorphic tests assert RELATIONS between outputs when inputs are
|
|
9
|
+
transformed. Use this checklist to make sure the test is actually testing
|
|
10
|
+
properties, not just one example. -->
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 1. Invariants Identified
|
|
15
|
+
|
|
16
|
+
| # | Invariant (always true) | Holds for which inputs |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| 1 | [e.g. "encode then decode returns the original"] | [all valid inputs] |
|
|
19
|
+
| 2 | [e.g. "output list is always sorted"] | [ ] |
|
|
20
|
+
|
|
21
|
+
- [ ] At least one round-trip / inverse property considered.
|
|
22
|
+
- [ ] At least one "never crashes / always well-formed" property considered.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 2. Generators / Strategies
|
|
27
|
+
|
|
28
|
+
| Input | Generator / strategy | Edge cases forced |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| [param] | [e.g. integers, text, composite builder] | [empty, max, unicode, negative] |
|
|
31
|
+
|
|
32
|
+
- [ ] Generators cover the full valid domain (not just happy values).
|
|
33
|
+
- [ ] Boundary and degenerate cases are reachable.
|
|
34
|
+
- [ ] Invalid inputs handled by a separate negative property where relevant.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## 3. Stateful / Model-Based Machine (if applicable)
|
|
39
|
+
|
|
40
|
+
- [ ] Rules / commands defined for each operation.
|
|
41
|
+
- [ ] `@invariant`-style checks assert system consistency after every step.
|
|
42
|
+
- [ ] Model (oracle) compared against the real implementation.
|
|
43
|
+
- [ ] Preconditions guard illegal command sequences.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## 4. Shrinking
|
|
48
|
+
|
|
49
|
+
- [ ] Failures shrink to a minimal counterexample.
|
|
50
|
+
- [ ] Shrunk examples are human-readable and reproducible (seed recorded).
|
|
51
|
+
- [ ] No custom type defeats shrinking (custom strategies still shrink).
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## 5. Metamorphic Relations
|
|
56
|
+
|
|
57
|
+
| # | Transformation on input | Expected relation on output |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| 1 | [e.g. "permute input order"] | [e.g. "result unchanged"] |
|
|
60
|
+
| 2 | [e.g. "scale all inputs by k"] | [e.g. "result scales by k"] |
|
|
61
|
+
| 3 | [e.g. "add an irrelevant record"] | [e.g. "decision unchanged"] |
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## 6. Honesty Note
|
|
66
|
+
|
|
67
|
+
> Property tests SAMPLE the input space; they do not PROVE correctness.
|
|
68
|
+
> A green run means "no counterexample found in N cases," not "always correct."
|
|
69
|
+
> Record the case count and seed, and treat a pass as strong evidence, not proof.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## 7. CI Marker
|
|
74
|
+
|
|
75
|
+
- [ ] Test tagged with the right marker: [`slow` / `live_llm` / default].
|
|
76
|
+
- [ ] Long-running property runs gated out of the fast PR lane if needed.
|
|
77
|
+
- [ ] Seed / example database committed so failures are reproducible in CI.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# Sketch-to-Diagram Conversion Report
|
|
2
|
+
|
|
3
|
+
**Date**: [YYYY-MM-DD]
|
|
4
|
+
**Input**: [Description of input image — whiteboard photo, napkin sketch, wireframe scan]
|
|
5
|
+
**Recognition Tool**: image-recognition MCP ([Anthropic / OpenAI / Cloudflare])
|
|
6
|
+
**Output Format**: [Excalidraw / Mermaid / Draw.io]
|
|
7
|
+
|
|
8
|
+
## 1. Original Image Description
|
|
9
|
+
|
|
10
|
+
[Agent's description of the hand-drawn sketch as returned by the vision API. Include key observations about shapes, text, arrows, groupings, and spatial layout.]
|
|
11
|
+
|
|
12
|
+
## 2. Recognized Elements
|
|
13
|
+
|
|
14
|
+
### Nodes
|
|
15
|
+
| # | Label | Shape | Group | Notes |
|
|
16
|
+
|---|---|---|---|---|
|
|
17
|
+
| 1 | [text] | [rectangle/circle/diamond] | [group name] | [any notes] |
|
|
18
|
+
| 2 | | | | |
|
|
19
|
+
|
|
20
|
+
### Edges (Connections)
|
|
21
|
+
| # | From | To | Label | Direction |
|
|
22
|
+
|---|---|---|---|---|
|
|
23
|
+
| 1 | [source node] | [target node] | [edge label] | [→ / ← / ↔] |
|
|
24
|
+
| 2 | | | | |
|
|
25
|
+
|
|
26
|
+
### Groups / Frames
|
|
27
|
+
| # | Group Name | Contains |
|
|
28
|
+
|---|---|---|
|
|
29
|
+
| 1 | [name] | [node labels] |
|
|
30
|
+
|
|
31
|
+
## 3. Human Verification
|
|
32
|
+
|
|
33
|
+
- [ ] All nodes correctly identified
|
|
34
|
+
- [ ] All connections correctly mapped
|
|
35
|
+
- [ ] Text labels accurately recognized
|
|
36
|
+
- [ ] Groupings match original intent
|
|
37
|
+
- [ ] Missing elements added manually: [list]
|
|
38
|
+
- [ ] Corrections applied: [list]
|
|
39
|
+
|
|
40
|
+
## 4. Generated Diagram
|
|
41
|
+
|
|
42
|
+
**Output file**: [path/to/output.excalidraw or .md or .drawio]
|
|
43
|
+
|
|
44
|
+
[If Mermaid, paste the generated code block here:]
|
|
45
|
+
```mermaid
|
|
46
|
+
[generated diagram code]
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## 5. Confidence & Caveats
|
|
50
|
+
|
|
51
|
+
| Aspect | Confidence | Notes |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| Node detection | [HIGH/MEDIUM/LOW] | [e.g., "All rectangles clearly visible"] |
|
|
54
|
+
| Text recognition | [HIGH/MEDIUM/LOW] | [e.g., "Two labels partially occluded"] |
|
|
55
|
+
| Connection mapping | [HIGH/MEDIUM/LOW] | [e.g., "One ambiguous arrow direction"] |
|
|
56
|
+
| Overall | [HIGH/MEDIUM/LOW] | |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Test Case: [TC-ID] - [Title]
|
|
2
|
+
|
|
3
|
+
**Feature**: [Feature Name]
|
|
4
|
+
**Priority**: [P0 - Blocker / P1 - Critical / P2 - Normal]
|
|
5
|
+
**Type**: [Functional / UI / Performance / Security]
|
|
6
|
+
**Automated**: [Yes / No / To Be Automated]
|
|
7
|
+
|
|
8
|
+
## Pre-conditions
|
|
9
|
+
1. [User is logged in]
|
|
10
|
+
2. [Cart is empty]
|
|
11
|
+
|
|
12
|
+
## Test Steps
|
|
13
|
+
| Step # | Action | Expected Result |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| 1 | Navigate to `/products` | Product list loads within 2s |
|
|
16
|
+
| 2 | Click "Add to Cart" on Item A | "Item added" toast appears |
|
|
17
|
+
| 3 | Click Cart Icon | Cart drawer opens showing Item A |
|
|
18
|
+
|
|
19
|
+
## Post-conditions
|
|
20
|
+
- [Item A remains in cart]
|
|
21
|
+
- [Inventory count is unchanged (until checkout)]
|
|
22
|
+
|
|
23
|
+
## Test Data
|
|
24
|
+
- **User**: `test-user@example.com`
|
|
25
|
+
- **Item**: `SKU-12345`
|
|
26
|
+
|
|
27
|
+
## Notes
|
|
28
|
+
- [Edge cases to watch out for, e.g., "Slow network"]
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Test Plan: [Project Name]
|
|
2
|
+
|
|
3
|
+
**Version**: 1.0
|
|
4
|
+
**Date**: [YYYY-MM-DD]
|
|
5
|
+
**Author**: [Name]
|
|
6
|
+
|
|
7
|
+
## 1. Introduction
|
|
8
|
+
**Scope**: [What is being tested? e.g., "The Checkout Flow and Payment API".]
|
|
9
|
+
**Out of Scope**: [What is NOT tested? e.g., "3rd party banking interface".]
|
|
10
|
+
|
|
11
|
+
## 2. Testing Strategy
|
|
12
|
+
**Model**: [Testing Pyramid / Testing Trophy]
|
|
13
|
+
**Justification**: [e.g., "We are using the Trophy because this is a UI-heavy React app."]
|
|
14
|
+
|
|
15
|
+
| Level | Goal | Tools | Coverage Goal |
|
|
16
|
+
|---|---|---|---|
|
|
17
|
+
| **Static** | Catch typos & type errors | ESLint, TypeScript | 100% (Strict) |
|
|
18
|
+
| **Unit** | Verify isolated logic | Vitest | > 99% |
|
|
19
|
+
| **Integration** | Verify component interactions | Vitest / Supertest | Critical Paths |
|
|
20
|
+
| **Contract** | Verify API consumers | Pact | All Microservices |
|
|
21
|
+
| **E2E** | Verify user journeys | Playwright | Smoke Tests Only |
|
|
22
|
+
|
|
23
|
+
## 3. Test Data Strategy
|
|
24
|
+
- **Unit**: Mocks/Stubs.
|
|
25
|
+
- **Integration**: In-memory DB or Dockerized test DB (seeded).
|
|
26
|
+
- **E2E**: Dedicated staging environment with seeded "Golden Data".
|
|
27
|
+
|
|
28
|
+
## 4. CI/CD Integration
|
|
29
|
+
- **PR Checks**: Static + Unit + Integration.
|
|
30
|
+
- **Merge to Main**: All of the above + Smoke E2E.
|
|
31
|
+
- **Nightly**: Full Regression E2E.
|
|
32
|
+
|
|
33
|
+
## 5. Critical User Journeys (Smoke Tests)
|
|
34
|
+
1. [User Login successful]
|
|
35
|
+
2. [User can add item to cart]
|
|
36
|
+
3. [User can complete checkout]
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# Time Complexity Analysis Report: [Project Name]
|
|
2
|
+
|
|
3
|
+
**Version**: 1.0
|
|
4
|
+
**Date**: [YYYY-MM-DD]
|
|
5
|
+
**Scope**: [File / Directory / Repository analyzed]
|
|
6
|
+
**Tool**: time-complexity-mcp (tree-sitter static analysis)
|
|
7
|
+
|
|
8
|
+
## 1. Summary
|
|
9
|
+
|
|
10
|
+
| Metric | Value |
|
|
11
|
+
|---|---|
|
|
12
|
+
| Files analyzed | [N] |
|
|
13
|
+
| Total functions | [N] |
|
|
14
|
+
| O(1) | [N] |
|
|
15
|
+
| O(n) | [N] |
|
|
16
|
+
| O(n log n) | [N] |
|
|
17
|
+
| O(n²) | [N] |
|
|
18
|
+
| O(n³) | [N] |
|
|
19
|
+
| O(2^n) | [N] |
|
|
20
|
+
|
|
21
|
+
## 2. Hotspots (Top 5 Most Complex Functions)
|
|
22
|
+
|
|
23
|
+
| # | File | Function | Lines | Complexity | Reasoning |
|
|
24
|
+
|---|---|---|---|---|---|
|
|
25
|
+
| 1 | [path/to/file] | [functionName] | [L1–L2] | [O(?)] | [e.g., 2 nested variable-bound loops] |
|
|
26
|
+
| 2 | | | | | |
|
|
27
|
+
| 3 | | | | | |
|
|
28
|
+
| 4 | | | | | |
|
|
29
|
+
| 5 | | | | | |
|
|
30
|
+
|
|
31
|
+
## 3. Detailed Findings
|
|
32
|
+
|
|
33
|
+
### [File: path/to/file.ext]
|
|
34
|
+
|
|
35
|
+
#### [functionName] (lines [L1]–[L2]): [O(?)]
|
|
36
|
+
**Detection**: [e.g., "Found 2 variable-bound loops, max nesting depth: 2"]
|
|
37
|
+
**Line Annotations**:
|
|
38
|
+
- Line [N]: [O(?)] — [description, e.g., "for_statement loop (nesting depth: 1)"]
|
|
39
|
+
- Line [N]: [O(?)] — [description]
|
|
40
|
+
|
|
41
|
+
**Impact**: [HIGH / MEDIUM / LOW] — [Justification, e.g., "Called on every API request with unbounded input"]
|
|
42
|
+
|
|
43
|
+
**Recommendation**:
|
|
44
|
+
- [Specific fix, e.g., "Replace linear search with Set lookup to reduce from O(n²) to O(n)"]
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## 4. Comparison (Before / After)
|
|
49
|
+
|
|
50
|
+
*Fill this section when re-running analysis after refactoring.*
|
|
51
|
+
|
|
52
|
+
| Function | Before | After | Improvement |
|
|
53
|
+
|---|---|---|---|
|
|
54
|
+
| [functionName] | [O(n²)] | [O(n)] | [Quadratic → Linear] |
|
|
55
|
+
|
|
56
|
+
## 5. Caveats
|
|
57
|
+
|
|
58
|
+
- Static analysis estimates — does not account for runtime data distributions or caching.
|
|
59
|
+
- Constant-bound loops are treated as O(1).
|
|
60
|
+
- Inter-procedural complexity (function A calling function B) may not be fully combined.
|
|
61
|
+
- Verify critical hotspots with runtime profiling (e.g., `Py-Spy`, `perf`, Chrome DevTools).
|