exonous 0.0.0-stage → 2.10.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +100 -2
- package/assets/THIRD-PARTY-NOTICES.md +29 -0
- package/assets/adapters/opencode/exonous.js +54 -0
- package/assets/agents/exo-backend.md +22 -0
- package/assets/agents/exo-cto.md +22 -0
- package/assets/agents/exo-data-ai.md +22 -0
- package/assets/agents/exo-designer.md +22 -0
- package/assets/agents/exo-devops.md +22 -0
- package/assets/agents/exo-frontend.md +22 -0
- package/assets/agents/exo-infra.md +22 -0
- package/assets/agents/exo-mobile.md +22 -0
- package/assets/agents/exo-pm.md +22 -0
- package/assets/agents/exo-qa.md +22 -0
- package/assets/agents/exo-researcher.md +22 -0
- package/assets/agents/exo-security.md +22 -0
- package/assets/agents/roster.json +209 -0
- package/assets/agents/taxis.js +189 -0
- package/assets/editors/cursor/frontend-skills.mdc +27 -0
- package/assets/editors/hooks/cursor/hooks.json +11 -0
- package/assets/machine/PR-TEMPLATE.md +21 -0
- package/assets/machine/hooks/anamnesis.cjs +61 -0
- package/assets/machine/hooks/ethos.cjs +48 -0
- package/assets/machine/hooks/kairos.cjs +46 -0
- package/assets/machine/hooks/lib/cli-options.cjs +133 -0
- package/assets/machine/hooks/lib/memory.cjs +488 -0
- package/assets/machine/hooks/lib/metron.cjs +52 -0
- package/assets/machine/hooks/lib/paths.cjs +99 -0
- package/assets/machine/hooks/lib/payload.cjs +81 -0
- package/assets/machine/hooks/lib/pr-template-validate.cjs +103 -0
- package/assets/machine/hooks/lib/release.cjs +58 -0
- package/assets/machine/hooks/lib/repo.cjs +138 -0
- package/assets/machine/hooks/lib/sanitize.cjs +37 -0
- package/assets/machine/hooks/lib/selfupdate.cjs +354 -0
- package/assets/machine/hooks/lib/shell.cjs +722 -0
- package/assets/machine/hooks/lib/template.cjs +70 -0
- package/assets/machine/hooks/lib/token.cjs +110 -0
- package/assets/machine/hooks/lib/validate-pr-body.cjs +21 -0
- package/assets/machine/hooks/pyle.cjs +302 -0
- package/assets/rules/nomos.md +59 -0
- package/assets/rules/praxis.md +38 -0
- package/assets/skills/code-review/SKILL.md +124 -0
- package/assets/skills/commit-changes/SKILL.md +49 -0
- package/assets/skills/deliver/SKILL.md +114 -0
- package/assets/skills/deliver/references/parallel.md +37 -0
- package/assets/skills/doc/SKILL.md +140 -0
- package/assets/skills/doc/doc.mjs +277 -0
- package/assets/skills/issue-board/SKILL.md +84 -0
- package/assets/skills/issue-board/board.mjs +389 -0
- package/assets/skills/issue-board/config.example.json +9 -0
- package/assets/skills/issue-board/render.py +189 -0
- package/assets/skills/make-pr/SKILL.md +24 -0
- package/assets/skills/native-ui/SKILL.md +67 -0
- package/assets/skills/native-ui/reference/apple.md +180 -0
- package/assets/skills/native-ui/reference/mobile-feel.md +221 -0
- package/assets/skills/native-ui/reference/swift.md +289 -0
- package/assets/skills/project-memory/SKILL.md +59 -0
- package/assets/skills/project-memory/config.example.json +4 -0
- package/assets/skills/project-memory/project-memory.mjs +478 -0
- package/assets/skills/security-review/SKILL.md +102 -0
- package/assets/skills/session-handoff/SKILL.md +23 -0
- package/assets/skills/session-handoff/config.example.json +3 -0
- package/assets/skills/ui-design/SKILL.md +75 -0
- package/assets/skills/ui-design/reference/adapt.md +151 -0
- package/assets/skills/ui-design/reference/adapt.native.md +35 -0
- package/assets/skills/ui-design/reference/android.md +50 -0
- package/assets/skills/ui-design/reference/animate.md +87 -0
- package/assets/skills/ui-design/reference/audit.md +76 -0
- package/assets/skills/ui-design/reference/audit.native.md +126 -0
- package/assets/skills/ui-design/reference/bolder.md +33 -0
- package/assets/skills/ui-design/reference/clarify.md +84 -0
- package/assets/skills/ui-design/reference/colorize.md +86 -0
- package/assets/skills/ui-design/reference/craft-floor.md +25 -0
- package/assets/skills/ui-design/reference/critique.md +345 -0
- package/assets/skills/ui-design/reference/degraded/asset-producer.md +35 -0
- package/assets/skills/ui-design/reference/degraded/documenter.md +23 -0
- package/assets/skills/ui-design/reference/degraded/finish-reviewer.md +43 -0
- package/assets/skills/ui-design/reference/degraded/manual-edit-applier.md +56 -0
- package/assets/skills/ui-design/reference/delight.md +63 -0
- package/assets/skills/ui-design/reference/distill.md +98 -0
- package/assets/skills/ui-design/reference/doctor.md +49 -0
- package/assets/skills/ui-design/reference/document.md +273 -0
- package/assets/skills/ui-design/reference/extract.md +52 -0
- package/assets/skills/ui-design/reference/harden.md +122 -0
- package/assets/skills/ui-design/reference/hooks.md +114 -0
- package/assets/skills/ui-design/reference/init.md +47 -0
- package/assets/skills/ui-design/reference/ios.md +55 -0
- package/assets/skills/ui-design/reference/layout.md +81 -0
- package/assets/skills/ui-design/reference/live-setup.md +75 -0
- package/assets/skills/ui-design/reference/live.md +326 -0
- package/assets/skills/ui-design/reference/new-work.md +350 -0
- package/assets/skills/ui-design/reference/onboard.md +85 -0
- package/assets/skills/ui-design/reference/operate.md +29 -0
- package/assets/skills/ui-design/reference/optimize.md +101 -0
- package/assets/skills/ui-design/reference/overdrive.md +83 -0
- package/assets/skills/ui-design/reference/polish.md +77 -0
- package/assets/skills/ui-design/reference/quieter.md +95 -0
- package/assets/skills/ui-design/reference/routing.md +37 -0
- package/assets/skills/ui-design/reference/shape.md +43 -0
- package/assets/skills/ui-design/reference/styles/brutalist.md +45 -0
- package/assets/skills/ui-design/reference/styles/design-md.md +70 -0
- package/assets/skills/ui-design/reference/styles/direction.md +95 -0
- package/assets/skills/ui-design/reference/styles/full-output.md +42 -0
- package/assets/skills/ui-design/reference/styles/minimalist.md +64 -0
- package/assets/skills/ui-design/reference/styles/premium.md +65 -0
- package/assets/skills/ui-design/reference/styles/redesign.md +59 -0
- package/assets/skills/ui-design/reference/typeset.md +73 -0
- package/assets/skills/ui-design/reference/visualize.md +156 -0
- package/assets/skills/ui-design/scripts/ENGINE-LICENSE +191 -0
- package/assets/skills/ui-design/scripts/VERSION +1 -0
- package/assets/skills/ui-design/scripts/bin/darwin-arm64/impeccable +0 -0
- package/assets/skills/ui-design/scripts/command-metadata.json +94 -0
- package/assets/skills/ui-design/scripts/data/font-index-failures.json +121 -0
- package/assets/skills/ui-design/scripts/data/font-index.json +1 -0
- package/assets/skills/ui-design/scripts/impeccable +206 -0
- package/assets/skills/ui-design/scripts/impeccable.cmd +214 -0
- package/assets/skills/ui-design/scripts/live-browser-dom.js +167 -0
- package/assets/skills/ui-design/scripts/live-browser-ignores.js +242 -0
- package/assets/skills/ui-design/scripts/live-browser-session.js +144 -0
- package/assets/skills/ui-design/scripts/live-browser.js +12990 -0
- package/assets/skills/ui-design/scripts/modern-screenshot.umd.js +14 -0
- package/assets/skills/ui-design/scripts/ui-design +6 -0
- package/assets/skills/ui-design/scripts/ui-design.cmd +3 -0
- package/assets/skills/ui-design/scripts/ui-design.mjs +124 -0
- package/assets/skills/ui-mockup/SKILL.md +68 -0
- package/assets/skills/ui-mockup/reference/brandkit.md +101 -0
- package/assets/skills/ui-mockup/reference/image-to-code.md +83 -0
- package/assets/skills/ui-mockup/reference/mobile-comps.md +94 -0
- package/assets/skills/ui-mockup/reference/web-comps.md +83 -0
- package/assets/skills/ui-motion/SKILL.md +112 -0
- package/assets/skills/ui-motion/reference/expo.md +224 -0
- package/assets/skills/ui-motion/reference/opportunities.md +81 -0
- package/assets/skills/ui-motion/reference/prototype.md +238 -0
- package/assets/skills/ui-motion/reference/review.md +163 -0
- package/assets/skills/ui-motion/reference/toasts.md +144 -0
- package/assets/skills/ui-motion/reference/ui-libraries.md +71 -0
- package/assets/skills/ui-motion/reference/vocabulary.md +130 -0
- package/assets/skills/ui-motion/reference/web.md +294 -0
- package/assets/skills/ui-patterns/SKILL.md +105 -0
- package/assets/skills/ui-patterns/references/stacks/html-tailwind.md +40 -0
- package/assets/skills/ui-patterns/references/stacks/nextjs.md +52 -0
- package/assets/skills/ui-patterns/references/stacks/nuxt.md +100 -0
- package/assets/skills/ui-patterns/references/stacks/react-native.md +48 -0
- package/assets/skills/ui-patterns/references/stacks/react.md +49 -0
- package/assets/skills/ui-patterns/references/stacks/shadcn.md +57 -0
- package/assets/skills/ui-patterns/references/stacks/vue.md +37 -0
- package/assets/techne/.claude-plugin/marketplace.json +31 -0
- package/assets/techne/.claude-plugin/plugin.json +24 -0
- package/assets/techne/.mcp.json +8 -0
- package/assets/techne/commands/build-fix.md +48 -0
- package/assets/techne/commands/refactor-clean.md +46 -0
- package/assets/techne/commands/test-coverage.md +42 -0
- package/assets/techne/hooks/hooks.json +49 -0
- package/assets/techne/rules/angular/coding-style.md +49 -0
- package/assets/techne/rules/angular/hooks.md +23 -0
- package/assets/techne/rules/angular/patterns.md +48 -0
- package/assets/techne/rules/angular/security.md +26 -0
- package/assets/techne/rules/angular/testing.md +46 -0
- package/assets/techne/rules/arkts/coding-style.md +63 -0
- package/assets/techne/rules/arkts/hooks.md +38 -0
- package/assets/techne/rules/arkts/patterns.md +42 -0
- package/assets/techne/rules/arkts/security.md +35 -0
- package/assets/techne/rules/arkts/testing.md +39 -0
- package/assets/techne/rules/common/agents.md +6 -0
- package/assets/techne/rules/common/code-review.md +15 -0
- package/assets/techne/rules/common/coding-style.md +13 -0
- package/assets/techne/rules/common/development-workflow.md +5 -0
- package/assets/techne/rules/common/git-workflow.md +30 -0
- package/assets/techne/rules/common/hooks.md +7 -0
- package/assets/techne/rules/common/patterns.md +3 -0
- package/assets/techne/rules/common/performance.md +3 -0
- package/assets/techne/rules/common/security.md +5 -0
- package/assets/techne/rules/common/testing.md +3 -0
- package/assets/techne/rules/cpp/coding-style.md +31 -0
- package/assets/techne/rules/cpp/hooks.md +26 -0
- package/assets/techne/rules/cpp/patterns.md +31 -0
- package/assets/techne/rules/cpp/security.md +37 -0
- package/assets/techne/rules/cpp/testing.md +31 -0
- package/assets/techne/rules/csharp/coding-style.md +30 -0
- package/assets/techne/rules/csharp/hooks.md +24 -0
- package/assets/techne/rules/csharp/patterns.md +28 -0
- package/assets/techne/rules/csharp/security.md +32 -0
- package/assets/techne/rules/csharp/testing.md +27 -0
- package/assets/techne/rules/dart/coding-style.md +47 -0
- package/assets/techne/rules/dart/hooks.md +26 -0
- package/assets/techne/rules/dart/patterns.md +38 -0
- package/assets/techne/rules/dart/security.md +44 -0
- package/assets/techne/rules/dart/testing.md +39 -0
- package/assets/techne/rules/fsharp/coding-style.md +34 -0
- package/assets/techne/rules/fsharp/hooks.md +25 -0
- package/assets/techne/rules/fsharp/patterns.md +43 -0
- package/assets/techne/rules/fsharp/security.md +32 -0
- package/assets/techne/rules/fsharp/testing.md +27 -0
- package/assets/techne/rules/golang/coding-style.md +30 -0
- package/assets/techne/rules/golang/hooks.md +17 -0
- package/assets/techne/rules/golang/patterns.md +43 -0
- package/assets/techne/rules/golang/security.md +33 -0
- package/assets/techne/rules/golang/testing.md +20 -0
- package/assets/techne/rules/java/coding-style.md +51 -0
- package/assets/techne/rules/java/hooks.md +18 -0
- package/assets/techne/rules/java/patterns.md +55 -0
- package/assets/techne/rules/java/security.md +33 -0
- package/assets/techne/rules/java/testing.md +33 -0
- package/assets/techne/rules/kotlin/coding-style.md +50 -0
- package/assets/techne/rules/kotlin/hooks.md +17 -0
- package/assets/techne/rules/kotlin/patterns.md +44 -0
- package/assets/techne/rules/kotlin/security.md +34 -0
- package/assets/techne/rules/kotlin/testing.md +38 -0
- package/assets/techne/rules/nuxt/coding-style.md +20 -0
- package/assets/techne/rules/nuxt/hooks.md +16 -0
- package/assets/techne/rules/nuxt/patterns.md +46 -0
- package/assets/techne/rules/nuxt/security.md +19 -0
- package/assets/techne/rules/nuxt/testing.md +15 -0
- package/assets/techne/rules/perl/coding-style.md +33 -0
- package/assets/techne/rules/perl/hooks.md +22 -0
- package/assets/techne/rules/perl/patterns.md +43 -0
- package/assets/techne/rules/perl/security.md +39 -0
- package/assets/techne/rules/perl/testing.md +42 -0
- package/assets/techne/rules/php/coding-style.md +30 -0
- package/assets/techne/rules/php/hooks.md +24 -0
- package/assets/techne/rules/php/patterns.md +26 -0
- package/assets/techne/rules/php/security.md +27 -0
- package/assets/techne/rules/php/testing.md +28 -0
- package/assets/techne/rules/python/coding-style.md +38 -0
- package/assets/techne/rules/python/fastapi.md +57 -0
- package/assets/techne/rules/python/hooks.md +19 -0
- package/assets/techne/rules/python/patterns.md +37 -0
- package/assets/techne/rules/python/security.md +26 -0
- package/assets/techne/rules/python/testing.md +32 -0
- package/assets/techne/rules/react/coding-style.md +24 -0
- package/assets/techne/rules/react/hooks.md +62 -0
- package/assets/techne/rules/react/patterns.md +48 -0
- package/assets/techne/rules/react/security.md +45 -0
- package/assets/techne/rules/react/testing.md +42 -0
- package/assets/techne/rules/react-native/accessibility.md +14 -0
- package/assets/techne/rules/react-native/coding-style.md +16 -0
- package/assets/techne/rules/react-native/hooks.md +16 -0
- package/assets/techne/rules/react-native/patterns.md +18 -0
- package/assets/techne/rules/react-native/performance.md +14 -0
- package/assets/techne/rules/react-native/production-readiness.md +16 -0
- package/assets/techne/rules/react-native/security.md +16 -0
- package/assets/techne/rules/react-native/testing.md +14 -0
- package/assets/techne/rules/ruby/coding-style.md +40 -0
- package/assets/techne/rules/ruby/hooks.md +33 -0
- package/assets/techne/rules/ruby/patterns.md +33 -0
- package/assets/techne/rules/ruby/security.md +39 -0
- package/assets/techne/rules/ruby/testing.md +35 -0
- package/assets/techne/rules/rust/coding-style.md +41 -0
- package/assets/techne/rules/rust/hooks.md +16 -0
- package/assets/techne/rules/rust/patterns.md +42 -0
- package/assets/techne/rules/rust/security.md +33 -0
- package/assets/techne/rules/rust/testing.md +30 -0
- package/assets/techne/rules/swift/coding-style.md +30 -0
- package/assets/techne/rules/swift/hooks.md +20 -0
- package/assets/techne/rules/swift/patterns.md +30 -0
- package/assets/techne/rules/swift/security.md +22 -0
- package/assets/techne/rules/swift/testing.md +28 -0
- package/assets/techne/rules/typescript/coding-style.md +38 -0
- package/assets/techne/rules/typescript/hooks.md +14 -0
- package/assets/techne/rules/typescript/patterns.md +12 -0
- package/assets/techne/rules/typescript/security.md +14 -0
- package/assets/techne/rules/typescript/testing.md +12 -0
- package/assets/techne/rules/vue/coding-style.md +19 -0
- package/assets/techne/rules/vue/hooks.md +13 -0
- package/assets/techne/rules/vue/patterns.md +29 -0
- package/assets/techne/rules/vue/security.md +11 -0
- package/assets/techne/rules/vue/testing.md +11 -0
- package/assets/techne/rules/web/coding-style.md +25 -0
- package/assets/techne/rules/web/design-quality.md +21 -0
- package/assets/techne/rules/web/hooks.md +39 -0
- package/assets/techne/rules/web/patterns.md +23 -0
- package/assets/techne/rules/web/performance.md +25 -0
- package/assets/techne/rules/web/security.md +23 -0
- package/assets/techne/rules/web/testing.md +21 -0
- package/assets/techne/scripts/hooks/config-guard.js +50 -0
- package/assets/techne/scripts/hooks/edit-tracker.js +26 -0
- package/assets/techne/scripts/hooks/lib/shell.js +204 -0
- package/assets/techne/scripts/hooks/no-verify-guard.js +205 -0
- package/assets/techne/scripts/hooks/run.js +92 -0
- package/assets/techne/scripts/hooks/stop-checks.js +136 -0
- package/assets/techne/skills/angular/SKILL.md +34 -0
- package/assets/techne/skills/angular/reference/components-and-templates.md +63 -0
- package/assets/techne/skills/angular/reference/dependency-injection.md +44 -0
- package/assets/techne/skills/angular/reference/forms.md +97 -0
- package/assets/techne/skills/angular/reference/routing.md +50 -0
- package/assets/techne/skills/angular/reference/signals-and-state.md +59 -0
- package/assets/techne/skills/angular/reference/styling-motion-a11y.md +54 -0
- package/assets/techne/skills/angular/reference/testing.md +52 -0
- package/assets/techne/skills/angular/reference/tooling.md +31 -0
- package/assets/techne/skills/cpp/SKILL.md +27 -0
- package/assets/techne/skills/cpp/reference/coding-standards.md +160 -0
- package/assets/techne/skills/databases/SKILL.md +35 -0
- package/assets/techne/skills/databases/reference/clickhouse.md +102 -0
- package/assets/techne/skills/databases/reference/migrations.md +94 -0
- package/assets/techne/skills/databases/reference/mysql.md +101 -0
- package/assets/techne/skills/databases/reference/postgres.md +72 -0
- package/assets/techne/skills/databases/reference/prisma.md +105 -0
- package/assets/techne/skills/databases/reference/redis.md +108 -0
- package/assets/techne/skills/delivery/SKILL.md +35 -0
- package/assets/techne/skills/delivery/reference/deployment-patterns.md +58 -0
- package/assets/techne/skills/delivery/reference/docker-patterns.md +77 -0
- package/assets/techne/skills/delivery/reference/git-workflow.md +64 -0
- package/assets/techne/skills/delivery/reference/github-ops.md +67 -0
- package/assets/techne/skills/delivery/reference/jira-integration.md +53 -0
- package/assets/techne/skills/delivery/reference/kubernetes-patterns.md +67 -0
- package/assets/techne/skills/dotnet/SKILL.md +27 -0
- package/assets/techne/skills/dotnet/reference/dotnet-core.md +131 -0
- package/assets/techne/skills/flutter/SKILL.md +29 -0
- package/assets/techne/skills/flutter/reference/code-review.md +76 -0
- package/assets/techne/skills/flutter/reference/patterns.md +201 -0
- package/assets/techne/skills/go/SKILL.md +28 -0
- package/assets/techne/skills/go/reference/patterns.md +87 -0
- package/assets/techne/skills/go/reference/testing.md +142 -0
- package/assets/techne/skills/java/SKILL.md +29 -0
- package/assets/techne/skills/java/reference/java-standards.md +117 -0
- package/assets/techne/skills/java/reference/springboot.md +108 -0
- package/assets/techne/skills/kotlin/SKILL.md +35 -0
- package/assets/techne/skills/kotlin/reference/android-clean-architecture.md +115 -0
- package/assets/techne/skills/kotlin/reference/compose-multiplatform-patterns.md +113 -0
- package/assets/techne/skills/kotlin/reference/kotlin-coroutines-flows.md +93 -0
- package/assets/techne/skills/kotlin/reference/kotlin-patterns.md +74 -0
- package/assets/techne/skills/kotlin/reference/kotlin-testing.md +115 -0
- package/assets/techne/skills/ml/SKILL.md +34 -0
- package/assets/techne/skills/ml/reference/ai-regression-testing.md +99 -0
- package/assets/techne/skills/ml/reference/cost-aware-llm-pipeline.md +133 -0
- package/assets/techne/skills/ml/reference/eval-harness.md +94 -0
- package/assets/techne/skills/ml/reference/mle-workflow.md +177 -0
- package/assets/techne/skills/ml/reference/prompt-optimizer.md +107 -0
- package/assets/techne/skills/ml/reference/pytorch-patterns.md +164 -0
- package/assets/techne/skills/php/SKILL.md +27 -0
- package/assets/techne/skills/php/reference/laravel.md +142 -0
- package/assets/techne/skills/python/SKILL.md +33 -0
- package/assets/techne/skills/python/reference/django.md +104 -0
- package/assets/techne/skills/python/reference/fastapi.md +118 -0
- package/assets/techne/skills/python/reference/python-core.md +88 -0
- package/assets/techne/skills/python/reference/python-testing.md +105 -0
- package/assets/techne/skills/quality/SKILL.md +34 -0
- package/assets/techne/skills/quality/reference/benchmark.md +55 -0
- package/assets/techne/skills/quality/reference/browser-qa.md +60 -0
- package/assets/techne/skills/quality/reference/coding-standards.md +88 -0
- package/assets/techne/skills/quality/reference/e2e-testing.md +99 -0
- package/assets/techne/skills/quality/reference/tdd-workflow.md +89 -0
- package/assets/techne/skills/quality/reference/verification-loop.md +80 -0
- package/assets/techne/skills/react/SKILL.md +30 -0
- package/assets/techne/skills/react/reference/nextjs-turbopack.md +42 -0
- package/assets/techne/skills/react/reference/react-patterns.md +139 -0
- package/assets/techne/skills/react/reference/react-performance.md +101 -0
- package/assets/techne/skills/react/reference/react-testing.md +139 -0
- package/assets/techne/skills/react-native/SKILL.md +28 -0
- package/assets/techne/skills/react-native/reference/expo-patterns.md +196 -0
- package/assets/techne/skills/research/SKILL.md +35 -0
- package/assets/techne/skills/research/reference/architecture-decision-records.md +65 -0
- package/assets/techne/skills/research/reference/codebase-onboarding.md +44 -0
- package/assets/techne/skills/research/reference/deep-research.md +50 -0
- package/assets/techne/skills/research/reference/iterative-retrieval.md +62 -0
- package/assets/techne/skills/research/reference/product-lens.md +38 -0
- package/assets/techne/skills/research/reference/search-first.md +50 -0
- package/assets/techne/skills/rust/SKILL.md +28 -0
- package/assets/techne/skills/rust/reference/patterns.md +108 -0
- package/assets/techne/skills/rust/reference/testing.md +142 -0
- package/assets/techne/skills/services/SKILL.md +32 -0
- package/assets/techne/skills/services/reference/api-contracts.md +139 -0
- package/assets/techne/skills/services/reference/backend-layers.md +95 -0
- package/assets/techne/skills/services/reference/errors-and-resilience.md +146 -0
- package/assets/techne/skills/services/reference/hexagonal-boundaries.md +93 -0
- package/assets/techne/skills/services/reference/mcp-servers.md +50 -0
- package/assets/techne/skills/services/reference/nestjs.md +96 -0
- package/assets/techne/skills/swift/SKILL.md +29 -0
- package/assets/techne/skills/swift/reference/concurrency.md +83 -0
- package/assets/techne/skills/swift/reference/swiftui.md +139 -0
- package/assets/techne/skills/vue/SKILL.md +29 -0
- package/assets/techne/skills/vue/reference/nuxt4-patterns.md +90 -0
- package/assets/techne/skills/vue/reference/ui-to-vue.md +90 -0
- package/assets/techne/skills/vue/reference/vue-patterns.md +151 -0
- package/assets/techne/skills/web/SKILL.md +30 -0
- package/assets/techne/skills/web/reference/bun-runtime.md +55 -0
- package/assets/techne/skills/web/reference/frontend-patterns.md +54 -0
- package/assets/techne/skills/web/reference/seo.md +77 -0
- package/assets/techne/skills/web/reference/vite-patterns.md +96 -0
- package/bin/native.cjs +30 -0
- package/native/aarch64-apple-darwin/exonous +0 -0
- package/native/aarch64-unknown-linux-gnu/exonous +0 -0
- package/native/x86_64-apple-darwin/exonous +0 -0
- package/native/x86_64-pc-windows-msvc/exonous.exe +0 -0
- package/native/x86_64-unknown-linux-gnu/exonous +0 -0
- package/package.json +1 -6
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# Kotlin testing
|
|
2
|
+
|
|
3
|
+
Standard stack: **Kotest** (specs + matchers), **MockK** (mocks/spies), **kotlinx-coroutines-test** (`runTest`, virtual time), **Kover** (coverage). Align red-green-refactor with **techne:quality**; do not mix JUnit-style assertion libraries on the same suite.
|
|
4
|
+
|
|
5
|
+
## Spec styles (pick one per module)
|
|
6
|
+
|
|
7
|
+
- **StringSpec** — flat `"description" { }` cases; good for pure functions.
|
|
8
|
+
- **FunSpec** — `test("…") { }`; familiar JUnit shape; pairs well with MockK `beforeTest`.
|
|
9
|
+
- **BehaviorSpec** — Given/When/Then for multi-step workflows.
|
|
10
|
+
- **DescribeSpec** — nested `describe`/`context`/`it`.
|
|
11
|
+
|
|
12
|
+
## MockK essentials
|
|
13
|
+
|
|
14
|
+
```kotlin
|
|
15
|
+
class BillingServiceTest : FunSpec({
|
|
16
|
+
val gateway = mockk<PaymentGateway>()
|
|
17
|
+
val sut = BillingService(gateway)
|
|
18
|
+
|
|
19
|
+
beforeTest { clearMocks(gateway) }
|
|
20
|
+
|
|
21
|
+
test("charges on success") {
|
|
22
|
+
coEvery { gateway.charge(any()) } returns ChargeResult.Approved
|
|
23
|
+
sut.pay(OrderFixtures.valid).isSuccess shouldBe true
|
|
24
|
+
coVerify(exactly = 1) { gateway.charge(match { it.amount > 0 }) }
|
|
25
|
+
}
|
|
26
|
+
})
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- Blocking APIs: `every` / `verify`.
|
|
30
|
+
- Suspend APIs: `coEvery` / `coVerify`.
|
|
31
|
+
- Capture args: `slot<T>()` + `capture(slot)`.
|
|
32
|
+
- Prefer real data classes over mocking them; use `relaxed = true` only for loggers/noise.
|
|
33
|
+
|
|
34
|
+
## Coroutines in tests
|
|
35
|
+
|
|
36
|
+
```kotlin
|
|
37
|
+
test("debounced search fires once") {
|
|
38
|
+
runTest {
|
|
39
|
+
val repo = FakeSearchRepo()
|
|
40
|
+
val engine = SearchEngine(repo, debounceMs = 300)
|
|
41
|
+
|
|
42
|
+
engine.submitQuery("a")
|
|
43
|
+
engine.submitQuery("ab")
|
|
44
|
+
engine.submitQuery("abc")
|
|
45
|
+
advanceTimeBy(350)
|
|
46
|
+
|
|
47
|
+
repo.searchCallCount shouldBe 1
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- Inject `StandardTestDispatcher` when testing custom scopes; call `advanceUntilIdle()` before assertions on `StateFlow`.
|
|
53
|
+
|
|
54
|
+
## Flow assertions
|
|
55
|
+
|
|
56
|
+
- Collect with Turbine: `flow.test { awaitItem(); cancelAndIgnoreRemainingEvents() }`.
|
|
57
|
+
- Or `toList()` on finite flows inside `runTest`.
|
|
58
|
+
|
|
59
|
+
## Property tests
|
|
60
|
+
|
|
61
|
+
```kotlin
|
|
62
|
+
test("sort is idempotent") {
|
|
63
|
+
checkAll(Arb.list(Arb.int())) { nums ->
|
|
64
|
+
val once = nums.sorted()
|
|
65
|
+
once shouldBe once.sorted()
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Define `Arb` builders for domain types (email, money) near test fixtures.
|
|
71
|
+
|
|
72
|
+
## Data-driven cases
|
|
73
|
+
|
|
74
|
+
Use Kotest `withData` for table-driven inputs (valid/invalid parsers, date formats).
|
|
75
|
+
|
|
76
|
+
## Lifecycle fixtures
|
|
77
|
+
|
|
78
|
+
- `beforeSpec` / `afterSpec` for expensive setup (in-memory DB).
|
|
79
|
+
- Extract reusable listeners into Kotest extensions registered with `register(ext)`.
|
|
80
|
+
|
|
81
|
+
## Kover
|
|
82
|
+
|
|
83
|
+
```kotlin
|
|
84
|
+
kover {
|
|
85
|
+
reports {
|
|
86
|
+
filters.excludes.classes("*.generated.*", "*.BuildConfig")
|
|
87
|
+
verify.rule { minBound(80) }
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Commands: `./gradlew test`, `./gradlew koverHtmlReport`, `./gradlew koverVerify`, `./gradlew test --tests "com.acme.pkg.ClassName"`.
|
|
93
|
+
|
|
94
|
+
| Layer | Coverage expectation |
|
|
95
|
+
|-------|---------------------|
|
|
96
|
+
| Critical domain rules | 100% target |
|
|
97
|
+
| Public service API | 90%+ |
|
|
98
|
+
| General code | 80%+ floor |
|
|
99
|
+
|
|
100
|
+
## Ktor HTTP tests
|
|
101
|
+
|
|
102
|
+
```kotlin
|
|
103
|
+
test("GET /health") {
|
|
104
|
+
testApplication {
|
|
105
|
+
application { installHealthRoutes() }
|
|
106
|
+
client.get("/health").status shouldBe HttpStatusCode.OK
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Practices
|
|
112
|
+
|
|
113
|
+
- Assert behavior and contracts, not private methods.
|
|
114
|
+
- Fix flaky tests; do not `@Ignore` without a tracked defect.
|
|
115
|
+
- CI: run `test` + `koverXmlReport` + `koverVerify` (see **techne:delivery** for pipeline wiring).
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ml
|
|
3
|
+
description: Production ML and LLM systems—data contracts, PyTorch training, eval gates, prompt design, cost-aware API routing, and agent regression tests. Use for model promotion, train/serve parity, drift, embeddings, classifiers, forecasting, or LLM spend control.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
End-to-end machine learning engineering: turn notebook experiments into reproducible pipelines with measurable promotion gates, monitored serving, and LLM cost guardrails. Pair **techne:python** for implementation, **techne:databases** for labels and feature stores, **techne:services** for inference APIs, **techne:quality** for automated tests, **exo-data-ai** for deep implementation, **atlas** for code structure, and **biblos** for library APIs.
|
|
7
|
+
|
|
8
|
+
## Non-negotiable rules
|
|
9
|
+
|
|
10
|
+
1. **Write the prediction contract first** — target, decision owner, output schema, latency budget, fallback when features or the model are unavailable.
|
|
11
|
+
2. **Lock the data contract** — entity grain, label timestamp, feature freshness, split policy, snapshot ID; reject features not knowable at prediction time.
|
|
12
|
+
3. **Compare against baseline and current production** before treating offline gains as real; slice metrics for high-risk cohorts.
|
|
13
|
+
4. **Automate promotion gates and fail closed** — declare thresholds before training finishes; missing metrics block release.
|
|
14
|
+
5. **Share or equivalence-test train vs serve transforms** — never copy preprocessing manually into serving without a parity test.
|
|
15
|
+
6. **Package artifacts with version, config hash, dataset reference, and preprocessing** — safe deserialization; no secrets or PII in artifacts or prediction logs.
|
|
16
|
+
7. **PyTorch: device-agnostic code, explicit seeds, `train()`/`eval()` before loops, `weights_only=True` on load** — profile before optimizing memory.
|
|
17
|
+
8. **LLM calls: route by task complexity to the cheapest tier that meets quality** — set batch budgets up front; retry only transient errors; cache stable system prompts when the provider supports it. **Check current provider pricing** when estimating cost; do not rely on stale per-token tables in docs.
|
|
18
|
+
9. **Model tier for agents: economical models for planning, research, review, and docs; robust models for production training, inference, and pipeline code** — same split applies to LLM judge vs implementation work.
|
|
19
|
+
10. **Define evals before building** — capability vs regression; prefer deterministic code graders; target pass@3 ≥ 0.90 for new capability, pass^3 = 1.00 for release-critical regression paths.
|
|
20
|
+
11. **Do not trust agent self-review on agent-written changes** — run automated tests and build first; add a regression case for every production bug (especially dual-path sandbox vs live logic).
|
|
21
|
+
12. **Monitor system health and model quality** — latency, feature drift, prediction drift, delayed labels; every deploy names rollback artifact and trigger.
|
|
22
|
+
13. **Scope to the system** — skip feature stores, GPUs, or online serving if a contract, a baseline, an eval script and a rollback plan already cover the need.
|
|
23
|
+
14. **Security and privacy** — run **security-review** / **exo-security** on datasets, prompts, and deserialization paths; session continuity via **/session-handoff**, not legacy save/resume commands.
|
|
24
|
+
|
|
25
|
+
## References
|
|
26
|
+
|
|
27
|
+
| File | Load when |
|
|
28
|
+
|------|-----------|
|
|
29
|
+
| [reference/mle-workflow.md](reference/mle-workflow.md) | Contracts, reproducible pipelines, promotion, serving, monitoring, error analysis |
|
|
30
|
+
| [reference/pytorch-patterns.md](reference/pytorch-patterns.md) | Modules, training/eval loops, DataLoader, AMP, checkpoints, perf |
|
|
31
|
+
| [reference/cost-aware-llm-pipeline.md](reference/cost-aware-llm-pipeline.md) | Model routing, budgets, retries, prompt caching, spend estimation |
|
|
32
|
+
| [reference/eval-harness.md](reference/eval-harness.md) | Eval-driven development, graders, pass@k / pass^k, eval artifacts |
|
|
33
|
+
| [reference/prompt-optimizer.md](reference/prompt-optimizer.md) | Rewriting user prompts for agents (advisory only—does not execute the task) |
|
|
34
|
+
| [reference/ai-regression-testing.md](reference/ai-regression-testing.md) | Agent blind spots, sandbox API tests, bug-driven regression suite |
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# AI regression testing
|
|
2
|
+
|
|
3
|
+
When the same agent writes and reviews code, shared assumptions hide bugs. Automated tests—especially contract tests on API shapes and dual code paths—catch failures reviews miss.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
- Agent touched routes, inference handlers, or feature-flagged paths
|
|
8
|
+
- A bug was fixed and must not return
|
|
9
|
+
- Sandbox/mock mode exists for fast tests without production DB
|
|
10
|
+
|
|
11
|
+
## Core failure mode
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
agent implements fix → agent reviews fix → “looks fine” → bug remains
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Highest-frequency pattern: **sandbox vs production branches return different shapes** (field added on one path only).
|
|
18
|
+
|
|
19
|
+
## Strategy: test where bugs happened
|
|
20
|
+
|
|
21
|
+
Do not chase coverage percentages. Add tests adjacent to incident sites; similar mistake classes cluster in auth, multi-path handlers, and optimistic UI.
|
|
22
|
+
|
|
23
|
+
| Priority | Pattern | Test focus |
|
|
24
|
+
|----------|---------|------------|
|
|
25
|
+
| High | Dual-path mismatch | Same JSON keys in mock/sandbox mode |
|
|
26
|
+
| High | DB SELECT omitting new column | Response includes required fields |
|
|
27
|
+
| Medium | Error leaves stale UI state | State cleared on failure |
|
|
28
|
+
| Medium | Optimistic UI without rollback | Prior state restored on API error |
|
|
29
|
+
|
|
30
|
+
## Sandbox-first API tests
|
|
31
|
+
|
|
32
|
+
Force mock/sandbox env in test setup so suites stay fast and DB-free.
|
|
33
|
+
|
|
34
|
+
```typescript
|
|
35
|
+
// vitest setup excerpt
|
|
36
|
+
process.env.SANDBOX_MODE = "true";
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Build requests against route handlers directly; assert **response contract**, not SQL internals.
|
|
40
|
+
|
|
41
|
+
```typescript
|
|
42
|
+
const REQUIRED = ["id", "email", "full_name", "notification_settings"];
|
|
43
|
+
|
|
44
|
+
it("GET /api/user/profile exposes contract", async () => {
|
|
45
|
+
const res = await GET(createRequest("/api/user/profile"));
|
|
46
|
+
const { status, json } = await parse(res);
|
|
47
|
+
expect(status).toBe(200);
|
|
48
|
+
for (const key of REQUIRED) {
|
|
49
|
+
expect(json.data).toHaveProperty(key);
|
|
50
|
+
}
|
|
51
|
+
});
|
|
52
|
+
|
|
53
|
+
it("regression: notification_settings present (BUG-R1)", async () => {
|
|
54
|
+
const { json } = await parse(await GET(createRequest("/api/user/profile")));
|
|
55
|
+
expect("notification_settings" in json.data).toBe(true);
|
|
56
|
+
});
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Name tests after the incident id when useful (`BUG-R1`).
|
|
60
|
+
|
|
61
|
+
## Bug-check workflow
|
|
62
|
+
|
|
63
|
+
1. **Automated first (mandatory)** — unit/API tests, then typecheck/build. Failures outrank subjective review.
|
|
64
|
+
2. **Review second** — dual-path parity, SELECT completeness, error rollback, race-prone optimistic updates. Use **exo-qa** / **code-review**; do not skip step 1.
|
|
65
|
+
3. **For each fix** — add or extend a regression test before closing.
|
|
66
|
+
|
|
67
|
+
## Dual-path parity example
|
|
68
|
+
|
|
69
|
+
```typescript
|
|
70
|
+
// bad: new field only on live branch
|
|
71
|
+
if (sandbox()) return { data: { id, email } };
|
|
72
|
+
return { data: { id, email, notification_settings } };
|
|
73
|
+
|
|
74
|
+
// good: parallel shapes (null placeholders ok in sandbox)
|
|
75
|
+
const settings = sandbox() ? null : row.notification_settings;
|
|
76
|
+
return { data: { id, email, notification_settings: settings } };
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## ML-specific regressions
|
|
80
|
+
|
|
81
|
+
- Train/serve feature parity: golden-vector test through both paths
|
|
82
|
+
- Promotion gate script: missing metric file fails CI
|
|
83
|
+
- LLM routing: log snapshot tests for tier selection on fixed inputs (no live API in CI when mocked)
|
|
84
|
+
|
|
85
|
+
Pair with **techne:ml** eval harness for agent/LLM behavior; pair with **techne:quality** for broader TDD.
|
|
86
|
+
|
|
87
|
+
## DO / DON'T
|
|
88
|
+
|
|
89
|
+
**DO**
|
|
90
|
+
|
|
91
|
+
- Write regression test when bug is reproduced (ideally before fix)
|
|
92
|
+
- Assert outward API/schema contracts
|
|
93
|
+
- Keep sandbox tests sub-second total when possible
|
|
94
|
+
|
|
95
|
+
**DON'T**
|
|
96
|
+
|
|
97
|
+
- Replace tests with agent self-review
|
|
98
|
+
- Skip mock path because “it's fake data”
|
|
99
|
+
- Add broad integration suites before first incident in that area
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
# Cost-aware LLM pipeline
|
|
2
|
+
|
|
3
|
+
Control API spend while preserving quality on hard tasks: route models by complexity, enforce budgets, retry narrowly, and cache stable prompt prefixes.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
- Apps calling Anthropic, OpenAI, or compatible APIs at scale
|
|
8
|
+
- Batch jobs where token volume dominates cost
|
|
9
|
+
- Multi-tier architectures (small model first, escalate on failure)
|
|
10
|
+
|
|
11
|
+
## Model tiering (no stale price tables)
|
|
12
|
+
|
|
13
|
+
1. **Classify each request** — input size, structured output need, reasoning depth, safety sensitivity, latency SLO.
|
|
14
|
+
2. **Map classes to provider tiers** — fast/cheap for extraction, formatting, routing, and simple Q&A; mid for most coding assistance; largest tier only for multi-step reasoning, long context synthesis, or high-stakes generation.
|
|
15
|
+
3. **Estimate spend from usage metadata** — use `input_tokens`, `output_tokens`, and cache read/write fields returned by the API; multiply by **current** list rates from the provider dashboard or pricing page at deploy time. Store rates in config/env refreshed on schedule, not copied into skills.
|
|
16
|
+
4. **Log routing decisions** — model id, tier reason, token counts, estimated USD — so thresholds can be tuned from production data.
|
|
17
|
+
|
|
18
|
+
Heuristic example (thresholds are project-specific, not universal):
|
|
19
|
+
|
|
20
|
+
```python
|
|
21
|
+
from dataclasses import dataclass
|
|
22
|
+
|
|
23
|
+
@dataclass(frozen=True)
|
|
24
|
+
class RoutePolicy:
|
|
25
|
+
small_model: str
|
|
26
|
+
large_model: str
|
|
27
|
+
char_escalate: int = 10_000
|
|
28
|
+
item_escalate: int = 30
|
|
29
|
+
|
|
30
|
+
def pick_model(policy: RoutePolicy, text_len: int, n_items: int, force: str | None = None) -> str:
|
|
31
|
+
if force:
|
|
32
|
+
return force
|
|
33
|
+
if text_len >= policy.char_escalate or n_items >= policy.item_escalate:
|
|
34
|
+
return policy.large_model
|
|
35
|
+
return policy.small_model
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Immutable budget tracking
|
|
39
|
+
|
|
40
|
+
Never mutate running totals in place — append-only records simplify audit and replay.
|
|
41
|
+
|
|
42
|
+
```python
|
|
43
|
+
from dataclasses import dataclass
|
|
44
|
+
|
|
45
|
+
@dataclass(frozen=True, slots=True)
|
|
46
|
+
class UsageLine:
|
|
47
|
+
model: str
|
|
48
|
+
input_tokens: int
|
|
49
|
+
output_tokens: int
|
|
50
|
+
est_usd: float
|
|
51
|
+
|
|
52
|
+
@dataclass(frozen=True, slots=True)
|
|
53
|
+
class Ledger:
|
|
54
|
+
cap_usd: float
|
|
55
|
+
lines: tuple[UsageLine, ...] = ()
|
|
56
|
+
|
|
57
|
+
def append(self, line: UsageLine) -> "Ledger":
|
|
58
|
+
return Ledger(self.cap_usd, (*self.lines, line))
|
|
59
|
+
|
|
60
|
+
@property
|
|
61
|
+
def spent(self) -> float:
|
|
62
|
+
return sum(x.est_usd for x in self.lines)
|
|
63
|
+
|
|
64
|
+
@property
|
|
65
|
+
def over_cap(self) -> bool:
|
|
66
|
+
return self.spent > self.cap_usd
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Compute `est_usd` with a small pricing function fed from config (updated when providers change rates).
|
|
70
|
+
|
|
71
|
+
## Retries
|
|
72
|
+
|
|
73
|
+
Retry **only** transient failures (connection, rate limit, 5xx). Fail immediately on auth, validation, and policy errors to avoid burning budget.
|
|
74
|
+
|
|
75
|
+
```python
|
|
76
|
+
RETRYABLE = (APIConnectionError, RateLimitError, InternalServerError)
|
|
77
|
+
|
|
78
|
+
def with_backoff(fn, attempts: int = 3):
|
|
79
|
+
for i in range(attempts):
|
|
80
|
+
try:
|
|
81
|
+
return fn()
|
|
82
|
+
except RETRYABLE:
|
|
83
|
+
if i == attempts - 1:
|
|
84
|
+
raise
|
|
85
|
+
time.sleep(2 ** i)
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Prompt caching
|
|
89
|
+
|
|
90
|
+
When the provider supports ephemeral cache blocks, mark long static system instructions; keep variable user content outside the cached segment. Worth it when the stable prefix exceeds ~1k tokens and repeats across calls.
|
|
91
|
+
|
|
92
|
+
## Pipeline composition
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
def invoke(user_text: str, policy: RoutePolicy, ledger: Ledger, price_fn) -> tuple[Result, Ledger]:
|
|
96
|
+
model = pick_model(policy, len(user_text), estimate_items(user_text))
|
|
97
|
+
if ledger.over_cap:
|
|
98
|
+
raise BudgetError(ledger.spent, ledger.cap_usd)
|
|
99
|
+
resp = with_backoff(lambda: client.messages.create(
|
|
100
|
+
model=model,
|
|
101
|
+
messages=build_messages(static_system, user_text, use_cache=True),
|
|
102
|
+
))
|
|
103
|
+
line = UsageLine(
|
|
104
|
+
model=model,
|
|
105
|
+
input_tokens=resp.usage.input_tokens,
|
|
106
|
+
output_tokens=resp.usage.output_tokens,
|
|
107
|
+
est_usd=price_fn(model, resp.usage),
|
|
108
|
+
)
|
|
109
|
+
return parse(resp), ledger.append(line)
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
## Agent model governance (house rule)
|
|
113
|
+
|
|
114
|
+
| Phase | Model tier |
|
|
115
|
+
|-------|----------------|
|
|
116
|
+
| Planning, research, eval design, docs, prompt critique | Economical |
|
|
117
|
+
| Production pipeline code, training scripts, inference paths | Robust |
|
|
118
|
+
|
|
119
|
+
Apply the same split when choosing a **judge model** vs the model under test — judges can often sit one tier lower if the rubric is structured.
|
|
120
|
+
|
|
121
|
+
## Practices
|
|
122
|
+
|
|
123
|
+
- Default to cheapest tier; escalate on measurable quality failure, not preemptively
|
|
124
|
+
- Set batch caps before loops; stop early on `over_cap`
|
|
125
|
+
- Centralize model ids in config
|
|
126
|
+
- Never retry 401/400-class errors
|
|
127
|
+
|
|
128
|
+
## Anti-patterns
|
|
129
|
+
|
|
130
|
+
- Largest model for every call
|
|
131
|
+
- Retry-all-errors policies
|
|
132
|
+
- Mutable global spend counters
|
|
133
|
+
- Hardcoded per-token constants in repo without refresh path
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Eval harness
|
|
2
|
+
|
|
3
|
+
Eval-driven development (EDD): define pass/fail behavior before implementation, grade with deterministic tools when possible, and track reliability with pass@k metrics. Applies to agent workflows, LLM features, and classical ML promotion checks alike.
|
|
4
|
+
|
|
5
|
+
## Philosophy
|
|
6
|
+
|
|
7
|
+
- Evals are the contract for “done” — like unit tests for nondeterministic components
|
|
8
|
+
- Run continuously during changes; regressions block merge
|
|
9
|
+
- Version eval definitions with application code
|
|
10
|
+
|
|
11
|
+
## Eval types
|
|
12
|
+
|
|
13
|
+
**Capability** — new behavior must succeed:
|
|
14
|
+
|
|
15
|
+
```markdown
|
|
16
|
+
[CAPABILITY: feature-id]
|
|
17
|
+
Task: …
|
|
18
|
+
Success:
|
|
19
|
+
- [ ] …
|
|
20
|
+
Expected artifact: …
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
**Regression** — existing behavior must not break:
|
|
24
|
+
|
|
25
|
+
```markdown
|
|
26
|
+
[REGRESSION: feature-id]
|
|
27
|
+
Baseline: git revision or tag
|
|
28
|
+
Checks:
|
|
29
|
+
- case-a: PASS/FAIL
|
|
30
|
+
Result: X/Y (was Y/Y)
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Graders (prefer top to bottom)
|
|
34
|
+
|
|
35
|
+
| Grader | Use |
|
|
36
|
+
|--------|-----|
|
|
37
|
+
| Code | Shell/pytest/grep/build — deterministic |
|
|
38
|
+
| Rule | JSON schema, regex, type constraints |
|
|
39
|
+
| Model | Rubric-scored LLM judge for open-ended text |
|
|
40
|
+
| Human | Ambiguous UX, safety, compliance |
|
|
41
|
+
|
|
42
|
+
Model graders consume tokens — use economical tier for rubric scoring unless the task requires deep reasoning; keep rubrics structured (numbered criteria, JSON scores).
|
|
43
|
+
|
|
44
|
+
**Security-sensitive paths:** combine code/rule graders with **security-review**; do not fully automate sign-off.
|
|
45
|
+
|
|
46
|
+
## Metrics
|
|
47
|
+
|
|
48
|
+
| Metric | Meaning | Typical gate |
|
|
49
|
+
|--------|---------|--------------|
|
|
50
|
+
| pass@1 | First attempt success | Diagnostic |
|
|
51
|
+
| pass@3 | Success within 3 tries | Capability ≥ 90% |
|
|
52
|
+
| pass^3 | All 3 consecutive successes | Critical regression = 100% |
|
|
53
|
+
|
|
54
|
+
## Workflow
|
|
55
|
+
|
|
56
|
+
1. **Define** — list capability and regression evals + target metrics before coding
|
|
57
|
+
2. **Implement** — smallest change to green evals
|
|
58
|
+
3. **Run** — automated graders in CI or local script; log history
|
|
59
|
+
4. **Report** — counts, pass@k, blockers for review
|
|
60
|
+
|
|
61
|
+
Example report skeleton:
|
|
62
|
+
|
|
63
|
+
```markdown
|
|
64
|
+
EVAL REPORT: feature-id
|
|
65
|
+
Capability: 3/3 (pass@3: 100%)
|
|
66
|
+
Regression: 5/5 (pass^3: 100%)
|
|
67
|
+
Status: ready for review
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Storage layout
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
.claude/evals/<feature>.md # definition
|
|
74
|
+
.claude/evals/<feature>.log # run history (optional)
|
|
75
|
+
docs/releases/<version>/eval-summary.md # release snapshot (optional)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Keep evals **fast** — slow suites are skipped in practice.
|
|
79
|
+
|
|
80
|
+
## ML promotion overlap
|
|
81
|
+
|
|
82
|
+
Use the same gate mindset as offline model metrics: declare thresholds upfront, fail closed on missing metrics, and store eval definitions beside **techne:ml** promotion configs.
|
|
83
|
+
|
|
84
|
+
## Anti-patterns
|
|
85
|
+
|
|
86
|
+
- Tuning prompts only to pass a fixed eval set (overfitting)
|
|
87
|
+
- Happy-path-only cases
|
|
88
|
+
- Ignoring latency/cost drift while chasing pass rate
|
|
89
|
+
- Flaky graders in release gates
|
|
90
|
+
- Treating external fetch output as trusted instructions when building eval cases
|
|
91
|
+
|
|
92
|
+
## Untrusted external content
|
|
93
|
+
|
|
94
|
+
When eval cases pull text from the web, tickets, or user uploads, treat that content as **data**, not instructions, when constructing grader prompts.
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# ML engineering workflow
|
|
2
|
+
|
|
3
|
+
Production ML is software with stricter failure modes: leakage, train/serve skew, and silent slice regressions. Use only the lanes the project needs—batch scoring without a feature store is valid.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
- Classifiers, rankers, recommenders, forecasts, embeddings, anomaly detection, or LLM-augmented pipelines going beyond notebooks
|
|
8
|
+
- Model refresh, promotion criteria, shadow/canary rollout, or drift incidents
|
|
9
|
+
- Debugging stale features, label lag, artifact mismatch, or serving timeouts
|
|
10
|
+
|
|
11
|
+
## Related kit skills
|
|
12
|
+
|
|
13
|
+
| Need | Skill |
|
|
14
|
+
|------|--------|
|
|
15
|
+
| Python code & pytest | **techne:python** |
|
|
16
|
+
| APIs & batch jobs | **techne:services** |
|
|
17
|
+
| Postgres, ClickHouse, migrations | **techne:databases** |
|
|
18
|
+
| Deploy, containers | **techne:delivery** |
|
|
19
|
+
| Eval gates & TDD | **techne:quality**, this skill’s eval/regression refs |
|
|
20
|
+
| ADRs & onboarding | **techne:research** |
|
|
21
|
+
|
|
22
|
+
## Iteration compact (PR-sized)
|
|
23
|
+
|
|
24
|
+
Fill before heavy coding:
|
|
25
|
+
|
|
26
|
+
```text
|
|
27
|
+
Goal / who cares / decision owner:
|
|
28
|
+
Behavior changed by the model:
|
|
29
|
+
Primary metric & guardrails (latency, cost, calibration):
|
|
30
|
+
Mistake budget & unacceptable errors:
|
|
31
|
+
Assumptions & constraints:
|
|
32
|
+
Labels & data snapshot ID:
|
|
33
|
+
Baseline & candidate signals:
|
|
34
|
+
Eval slices & rollback:
|
|
35
|
+
Next experiment:
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Decision loop
|
|
39
|
+
|
|
40
|
+
1. Name the downstream **action**, not the model architecture.
|
|
41
|
+
2. State who pays for false positives vs false negatives vs latency vs compute.
|
|
42
|
+
3. Hypothesize separating signals; pick a simple baseline that must be beaten.
|
|
43
|
+
4. Score options with evidence; prefer the smallest change that cuts the top mistake class.
|
|
44
|
+
5. Record decision, counterargument, and next reversible step (**techne:research** ADR when irreversible).
|
|
45
|
+
|
|
46
|
+
## Metric economics
|
|
47
|
+
|
|
48
|
+
| Situation | Lean toward |
|
|
49
|
+
|-----------|-------------|
|
|
50
|
+
| Bad positive action is costly | Precision |
|
|
51
|
+
| Missed positives are costly | Recall |
|
|
52
|
+
| Ordering matters more than one threshold | AUC / NDCG |
|
|
53
|
+
| Balanced tradeoff with explainable F1 | F1 (explicitly justify) |
|
|
54
|
+
|
|
55
|
+
Always report confusion-matrix clusters, not headline accuracy alone. Treat delayed feedback as biased, lagged labels—not instant ground truth.
|
|
56
|
+
|
|
57
|
+
## Data & features
|
|
58
|
+
|
|
59
|
+
- Every feature family needs a **why it separates** story and a **leakage check** (event time, label availability).
|
|
60
|
+
- Handle imbalance via threshold, weights, or resampling—document the choice.
|
|
61
|
+
- Missing values: informative absence vs impute vs abstain.
|
|
62
|
+
- Add capacity only after error analysis shows baseline failure modes that new signal could fix.
|
|
63
|
+
|
|
64
|
+
## Error analysis loop
|
|
65
|
+
|
|
66
|
+
After each run or threshold change:
|
|
67
|
+
|
|
68
|
+
1. Bucket false positives, false negatives, abstentions, low-confidence, infra failures.
|
|
69
|
+
2. Cluster by language, cohort, source, freshness, sparsity, label source.
|
|
70
|
+
3. Route clusters to: better labels, features, threshold, or product fallback.
|
|
71
|
+
4. Capture each important cluster as an eval slice, regression test, or dashboard panel.
|
|
72
|
+
|
|
73
|
+
## Core pipeline stages
|
|
74
|
+
|
|
75
|
+
### 1. Prediction contract
|
|
76
|
+
|
|
77
|
+
- Entity, inputs, outputs (incl. confidence), batch vs online vs stream
|
|
78
|
+
- Fallback when model, features, or dependencies fail
|
|
79
|
+
- Human override path for high-impact decisions
|
|
80
|
+
- Retention and audit rules for inputs, predictions, labels
|
|
81
|
+
|
|
82
|
+
### 2. Data contract
|
|
83
|
+
|
|
84
|
+
- Primary key grain; label definition and **label time**
|
|
85
|
+
- Feature time and freshness SLA; point-in-time join rules
|
|
86
|
+
- Train/val/test/backtest splits (no random split on time-series without justification)
|
|
87
|
+
- Allowed nulls, ranges, enums; PII excluded from training artifacts
|
|
88
|
+
|
|
89
|
+
### 3. Reproducible training
|
|
90
|
+
|
|
91
|
+
- Frozen config (dataclass or typed YAML); pinned deps; seeds documented
|
|
92
|
+
- Log dataset version, code revision, config hash, metrics, artifact URI
|
|
93
|
+
- Preprocessing saved **with** the model; idempotent steps for retries
|
|
94
|
+
- Prefer pure transforms; avoid mutating shared frames in place
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
import hashlib
|
|
98
|
+
from dataclasses import dataclass
|
|
99
|
+
from pathlib import Path
|
|
100
|
+
|
|
101
|
+
@dataclass(frozen=True)
|
|
102
|
+
class RunConfig:
|
|
103
|
+
dataset_uri: str
|
|
104
|
+
artifact_dir: Path
|
|
105
|
+
seed: int
|
|
106
|
+
lr: float
|
|
107
|
+
batch_size: int
|
|
108
|
+
|
|
109
|
+
def run_tag(cfg: RunConfig, revision: str) -> str:
|
|
110
|
+
payload = f"{cfg.dataset_uri}:{cfg.seed}:{cfg.lr}:{cfg.batch_size}"
|
|
111
|
+
digest = hashlib.sha256(payload.encode()).hexdigest()[:12]
|
|
112
|
+
return f"{revision[:12]}-{digest}"
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### 4. Promotion
|
|
116
|
+
|
|
117
|
+
Declare gates **before** training completes. Example pattern:
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
GATES = {
|
|
121
|
+
"auc": ("min", 0.82),
|
|
122
|
+
"calibration_error": ("max", 0.04),
|
|
123
|
+
"p95_latency_ms": ("max", 80),
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
def require_gates(metrics: dict[str, float]) -> None:
|
|
127
|
+
missing = [k for k in GATES if k not in metrics]
|
|
128
|
+
if missing:
|
|
129
|
+
raise ValueError(f"missing gate metrics: {missing}")
|
|
130
|
+
bad = {
|
|
131
|
+
k: metrics[k]
|
|
132
|
+
for k, (op, bound) in GATES.items()
|
|
133
|
+
if (op == "min" and metrics[k] < bound) or (op == "max" and metrics[k] > bound)
|
|
134
|
+
}
|
|
135
|
+
if bad:
|
|
136
|
+
raise ValueError(f"promotion blocked: {bad}")
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Offline gates are necessary, not sufficient—plan shadow, canary, or controlled A/B when behavior changes.
|
|
140
|
+
|
|
141
|
+
### 5. Serving
|
|
142
|
+
|
|
143
|
+
- Validate inputs (types, ranges, staleness); enforce timeouts and resource limits
|
|
144
|
+
- Log model version and correlation IDs; avoid PII in logs
|
|
145
|
+
- Integration tests: missing features, stale features, empty batch, fallback path
|
|
146
|
+
|
|
147
|
+
### 6. Operations
|
|
148
|
+
|
|
149
|
+
Track availability, errors, p50/p95 latency, feature null rates, distribution drift, confidence drift, label arrival health, business guardrails. Rollback = previous artifact + config + traffic switch—no retrain required.
|
|
150
|
+
|
|
151
|
+
## Observation ledger (per iteration)
|
|
152
|
+
|
|
153
|
+
```text
|
|
154
|
+
Change / why:
|
|
155
|
+
Metric & slice movement:
|
|
156
|
+
FP / FN themes:
|
|
157
|
+
Decision & tradeoff accepted:
|
|
158
|
+
Regression added:
|
|
159
|
+
Next step:
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
## Review checklist
|
|
163
|
+
|
|
164
|
+
- [ ] Prediction and data contracts documented
|
|
165
|
+
- [ ] Leakage reviewed for prediction-time availability
|
|
166
|
+
- [ ] Reproducible from config + data version + seed
|
|
167
|
+
- [ ] Baseline and production comparison with slices
|
|
168
|
+
- [ ] Gates automated; train/serve parity tested
|
|
169
|
+
- [ ] Monitoring and rollback defined
|
|
170
|
+
|
|
171
|
+
## Anti-patterns
|
|
172
|
+
|
|
173
|
+
- Notebook-only state required to reproduce
|
|
174
|
+
- Validation/test tuned repeatedly on the same holdout
|
|
175
|
+
- Training preprocessing diverges from serving
|
|
176
|
+
- Uptime-only monitoring without data or prediction quality
|
|
177
|
+
- Rollback requires full retrain
|