@geraldmaron/construct 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +69 -0
- package/LICENSE +21 -0
- package/README.md +344 -0
- package/agents/prompts/cx-accessibility.md +29 -0
- package/agents/prompts/cx-ai-engineer.md +31 -0
- package/agents/prompts/cx-architect.md +40 -0
- package/agents/prompts/cx-business-strategist.md +25 -0
- package/agents/prompts/cx-data-analyst.md +30 -0
- package/agents/prompts/cx-data-engineer.md +32 -0
- package/agents/prompts/cx-debugger.md +27 -0
- package/agents/prompts/cx-designer.md +33 -0
- package/agents/prompts/cx-devil-advocate.md +36 -0
- package/agents/prompts/cx-docs-keeper.md +50 -0
- package/agents/prompts/cx-engineer.md +36 -0
- package/agents/prompts/cx-evaluator.md +26 -0
- package/agents/prompts/cx-explorer.md +38 -0
- package/agents/prompts/cx-legal-compliance.md +25 -0
- package/agents/prompts/cx-operations.md +28 -0
- package/agents/prompts/cx-orchestrator.md +30 -0
- package/agents/prompts/cx-platform-engineer.md +25 -0
- package/agents/prompts/cx-product-manager.md +30 -0
- package/agents/prompts/cx-qa.md +42 -0
- package/agents/prompts/cx-rd-lead.md +25 -0
- package/agents/prompts/cx-release-manager.md +31 -0
- package/agents/prompts/cx-researcher.md +31 -0
- package/agents/prompts/cx-reviewer.md +36 -0
- package/agents/prompts/cx-security.md +37 -0
- package/agents/prompts/cx-sre.md +26 -0
- package/agents/prompts/cx-test-automation.md +35 -0
- package/agents/prompts/cx-trace-reviewer.md +76 -0
- package/agents/prompts/cx-ux-researcher.md +30 -0
- package/agents/registry.json +879 -0
- package/agents/teams.json +94 -0
- package/bin/construct +613 -0
- package/commands/build/feature.md +21 -0
- package/commands/build/fix.md +20 -0
- package/commands/design/access.md +19 -0
- package/commands/design/flow.md +16 -0
- package/commands/design/ui.md +18 -0
- package/commands/measure/experiment.md +18 -0
- package/commands/measure/metrics.md +18 -0
- package/commands/measure/results.md +18 -0
- package/commands/plan/api.md +19 -0
- package/commands/plan/challenge.md +18 -0
- package/commands/plan/decide.md +20 -0
- package/commands/plan/feature.md +49 -0
- package/commands/plan/requirements.md +24 -0
- package/commands/remember/context.md +26 -0
- package/commands/remember/handoff.md +19 -0
- package/commands/remember/runbook.md +20 -0
- package/commands/review/code.md +21 -0
- package/commands/review/quality.md +19 -0
- package/commands/review/security.md +16 -0
- package/commands/ship/ready.md +19 -0
- package/commands/ship/release.md +19 -0
- package/commands/ship/status.md +18 -0
- package/commands/understand/docs.md +22 -0
- package/commands/understand/research.md +22 -0
- package/commands/understand/this.md +23 -0
- package/commands/understand/why.md +18 -0
- package/commands/work/clean.md +32 -0
- package/commands/work/drive.md +69 -0
- package/commands/work/optimize-prompts.md +46 -0
- package/commands/work/parallel-review.md +40 -0
- package/db/migrations/001_init.sql +38 -0
- package/langfuse/docker-compose.yml +82 -0
- package/lib/audit-skills.mjs +118 -0
- package/lib/auto-docs.mjs +293 -0
- package/lib/cli-commands.mjs +409 -0
- package/lib/codex-config.mjs +114 -0
- package/lib/comment-lint.mjs +191 -0
- package/lib/completions.mjs +170 -0
- package/lib/context-state.mjs +99 -0
- package/lib/cost.mjs +213 -0
- package/lib/distill.mjs +403 -0
- package/lib/efficiency.mjs +139 -0
- package/lib/env-config.mjs +53 -0
- package/lib/eval-harness.mjs +59 -0
- package/lib/features.mjs +209 -0
- package/lib/headhunt.mjs +597 -0
- package/lib/hooks/adaptive-lint.mjs +112 -0
- package/lib/hooks/agent-tracker.mjs +50 -0
- package/lib/hooks/bootstrap-guard.mjs +90 -0
- package/lib/hooks/comment-lint.mjs +26 -0
- package/lib/hooks/config-protection.mjs +52 -0
- package/lib/hooks/console-warn.mjs +43 -0
- package/lib/hooks/context-window-recovery.mjs +90 -0
- package/lib/hooks/continuation-enforcer.mjs +72 -0
- package/lib/hooks/dep-audit.mjs +155 -0
- package/lib/hooks/drive-guard.mjs +89 -0
- package/lib/hooks/edit-accumulator.mjs +36 -0
- package/lib/hooks/edit-error-recovery.mjs +46 -0
- package/lib/hooks/edit-guard.mjs +109 -0
- package/lib/hooks/env-check.mjs +80 -0
- package/lib/hooks/guard-bash.mjs +83 -0
- package/lib/hooks/mcp-audit.mjs +57 -0
- package/lib/hooks/mcp-health-check.mjs +51 -0
- package/lib/hooks/mcp-task-scope.mjs +47 -0
- package/lib/hooks/model-fallback.mjs +105 -0
- package/lib/hooks/pre-compact.mjs +212 -0
- package/lib/hooks/pre-push-gate.mjs +129 -0
- package/lib/hooks/read-tracker.mjs +129 -0
- package/lib/hooks/registry-sync.mjs +23 -0
- package/lib/hooks/scan-secrets.mjs +76 -0
- package/lib/hooks/session-start.mjs +95 -0
- package/lib/hooks/stop-notify.mjs +191 -0
- package/lib/hooks/stop-typecheck.mjs +83 -0
- package/lib/hooks/task-completed-guard.mjs +43 -0
- package/lib/hooks/teammate-idle-guard.mjs +54 -0
- package/lib/hooks/workflow-guard.mjs +62 -0
- package/lib/host-capabilities.mjs +123 -0
- package/lib/init-docs.mjs +382 -0
- package/lib/mcp/server.mjs +1123 -0
- package/lib/mcp-catalog.json +243 -0
- package/lib/mcp-manager.mjs +433 -0
- package/lib/mcp-platform-config.mjs +104 -0
- package/lib/model-router.mjs +810 -0
- package/lib/opencode-config.mjs +44 -0
- package/lib/opencode-runtime-plugin.mjs +909 -0
- package/lib/opencode-telemetry.mjs +433 -0
- package/lib/orchestration-policy.mjs +258 -0
- package/lib/prompt-composer.mjs +196 -0
- package/lib/prompt-metadata.mjs +68 -0
- package/lib/review.mjs +429 -0
- package/lib/role-preload.mjs +52 -0
- package/lib/schemas/decision.json +50 -0
- package/lib/schemas/implementation.json +51 -0
- package/lib/schemas/review-report.json +32 -0
- package/lib/schemas/test-report.json +43 -0
- package/lib/server/index.mjs +334 -0
- package/lib/server/static/app.js +841 -0
- package/lib/server/static/index.html +820 -0
- package/lib/service-manager.mjs +225 -0
- package/lib/setup.mjs +319 -0
- package/lib/status.mjs +707 -0
- package/lib/storage/backend.mjs +44 -0
- package/lib/storage/embeddings.mjs +70 -0
- package/lib/storage/hybrid-query.mjs +184 -0
- package/lib/storage/sql-store.mjs +55 -0
- package/lib/storage/state-source.mjs +101 -0
- package/lib/storage/sync.mjs +164 -0
- package/lib/storage/vector-store.mjs +59 -0
- package/lib/telemetry/backends/langfuse.mjs +58 -0
- package/lib/telemetry/backends/noop.mjs +16 -0
- package/lib/telemetry/langfuse-ingest.mjs +108 -0
- package/lib/telemetry/langfuse-model-sync.mjs +270 -0
- package/lib/telemetry/team-rollup.mjs +143 -0
- package/lib/toolkit-env.mjs +30 -0
- package/lib/validator.mjs +181 -0
- package/lib/workflow-state.mjs +794 -0
- package/package.json +53 -0
- package/personas/construct.md +94 -0
- package/platforms/claude/CLAUDE.md +30 -0
- package/platforms/claude/settings.template.json +472 -0
- package/platforms/opencode/config.template.json +65 -0
- package/platforms/opencode/plugins/construct-fallback.js +5 -0
- package/platforms/opencode/sync-config.mjs +69 -0
- package/rules/common/agents.md +55 -0
- package/rules/common/code-review.md +129 -0
- package/rules/common/coding-style.md +95 -0
- package/rules/common/comments.md +139 -0
- package/rules/common/cx-agent-routing.md +61 -0
- package/rules/common/cx-skill-routing.md +11 -0
- package/rules/common/development-workflow.md +49 -0
- package/rules/common/git-workflow.md +29 -0
- package/rules/common/hooks.md +35 -0
- package/rules/common/patterns.md +36 -0
- package/rules/common/performance.md +71 -0
- package/rules/common/security.md +34 -0
- package/rules/common/testing.md +62 -0
- package/rules/golang/coding-style.md +37 -0
- package/rules/golang/hooks.md +22 -0
- package/rules/golang/patterns.md +50 -0
- package/rules/golang/security.md +39 -0
- package/rules/golang/testing.md +36 -0
- package/rules/python/coding-style.md +47 -0
- package/rules/python/hooks.md +24 -0
- package/rules/python/patterns.md +44 -0
- package/rules/python/security.md +35 -0
- package/rules/python/testing.md +43 -0
- package/rules/swift/coding-style.md +52 -0
- package/rules/swift/hooks.md +25 -0
- package/rules/swift/patterns.md +71 -0
- package/rules/swift/security.md +38 -0
- package/rules/swift/testing.md +50 -0
- package/rules/typescript/coding-style.md +204 -0
- package/rules/typescript/hooks.md +27 -0
- package/rules/typescript/patterns.md +57 -0
- package/rules/typescript/security.md +33 -0
- package/rules/typescript/testing.md +23 -0
- package/rules/web/coding-style.md +101 -0
- package/rules/web/design-quality.md +68 -0
- package/rules/web/hooks.md +125 -0
- package/rules/web/patterns.md +84 -0
- package/rules/web/performance.md +69 -0
- package/rules/web/security.md +62 -0
- package/rules/web/testing.md +60 -0
- package/rules/zh/README.md +113 -0
- package/rules/zh/agents.md +55 -0
- package/rules/zh/code-review.md +129 -0
- package/rules/zh/coding-style.md +53 -0
- package/rules/zh/development-workflow.md +49 -0
- package/rules/zh/git-workflow.md +29 -0
- package/rules/zh/hooks.md +35 -0
- package/rules/zh/patterns.md +36 -0
- package/rules/zh/performance.md +60 -0
- package/rules/zh/security.md +34 -0
- package/rules/zh/testing.md +34 -0
- package/skills/ai/agent-dev.md +102 -0
- package/skills/ai/llm-security.md +105 -0
- package/skills/ai/ml-ops.md +169 -0
- package/skills/ai/orchestration-workflow.md +87 -0
- package/skills/ai/prompt-and-eval.md +114 -0
- package/skills/ai/prompt-optimizer.md +114 -0
- package/skills/ai/rag-system.md +104 -0
- package/skills/architecture/api-design.md +100 -0
- package/skills/architecture/caching.md +101 -0
- package/skills/architecture/cloud-native.md +97 -0
- package/skills/architecture/message-queue.md +100 -0
- package/skills/architecture/security-arch.md +105 -0
- package/skills/development/cpp.md +127 -0
- package/skills/development/go.md +129 -0
- package/skills/development/java.md +134 -0
- package/skills/development/kotlin.md +140 -0
- package/skills/development/mobile-crossplatform.md +144 -0
- package/skills/development/python.md +109 -0
- package/skills/development/rust.md +127 -0
- package/skills/development/shell.md +127 -0
- package/skills/development/swift.md +129 -0
- package/skills/development/typescript.md +129 -0
- package/skills/devops/ci-cd.md +108 -0
- package/skills/devops/containerization.md +106 -0
- package/skills/devops/cost-optimization.md +105 -0
- package/skills/devops/data-engineering.md +181 -0
- package/skills/devops/database.md +95 -0
- package/skills/devops/dependency-management.md +91 -0
- package/skills/devops/devsecops.md +96 -0
- package/skills/devops/git-workflow.md +100 -0
- package/skills/devops/incident-response.md +167 -0
- package/skills/devops/monorepo.md +87 -0
- package/skills/devops/observability.md +103 -0
- package/skills/devops/performance.md +104 -0
- package/skills/devops/testing.md +104 -0
- package/skills/docs/adr-workflow.md +32 -0
- package/skills/docs/backlog-proposal-workflow.md +19 -0
- package/skills/docs/customer-profile-workflow.md +22 -0
- package/skills/docs/evidence-ingest-workflow.md +23 -0
- package/skills/docs/init-docs.md +160 -0
- package/skills/docs/init-project.md +50 -0
- package/skills/docs/prd-workflow.md +57 -0
- package/skills/docs/prfaq-workflow.md +25 -0
- package/skills/docs/product-intelligence-review.md +22 -0
- package/skills/docs/product-intelligence-workflow.md +62 -0
- package/skills/docs/product-signal-workflow.md +27 -0
- package/skills/docs/research-workflow.md +28 -0
- package/skills/docs/runbook-workflow.md +24 -0
- package/skills/exploration/repo-map.md +297 -0
- package/skills/frameworks/django.md +159 -0
- package/skills/frameworks/nextjs.md +148 -0
- package/skills/frameworks/react.md +132 -0
- package/skills/frameworks/spring-boot.md +165 -0
- package/skills/frontend-design/accessibility.md +153 -0
- package/skills/frontend-design/component-patterns.md +111 -0
- package/skills/frontend-design/engineering.md +122 -0
- package/skills/frontend-design/state-management.md +118 -0
- package/skills/frontend-design/ui-aesthetics.md +102 -0
- package/skills/frontend-design/ux-principles.md +126 -0
- package/skills/quality-gates/review-work.md +90 -0
- package/skills/quality-gates/verify-change.md +94 -0
- package/skills/quality-gates/verify-module.md +96 -0
- package/skills/quality-gates/verify-quality.md +93 -0
- package/skills/quality-gates/verify-security.md +95 -0
- package/skills/roles/architect.ai-systems.md +37 -0
- package/skills/roles/architect.data.md +37 -0
- package/skills/roles/architect.enterprise.md +37 -0
- package/skills/roles/architect.integration.md +37 -0
- package/skills/roles/architect.md +68 -0
- package/skills/roles/architect.platform.md +37 -0
- package/skills/roles/data-analyst.experiment.md +37 -0
- package/skills/roles/data-analyst.md +68 -0
- package/skills/roles/data-analyst.product-intelligence.md +37 -0
- package/skills/roles/data-analyst.product.md +37 -0
- package/skills/roles/data-analyst.telemetry.md +37 -0
- package/skills/roles/data-engineer.pipeline.md +37 -0
- package/skills/roles/data-engineer.vector-retrieval.md +37 -0
- package/skills/roles/data-engineer.warehouse.md +37 -0
- package/skills/roles/debugger.md +68 -0
- package/skills/roles/designer.accessibility.md +43 -0
- package/skills/roles/designer.md +68 -0
- package/skills/roles/engineer.ai.md +49 -0
- package/skills/roles/engineer.data.md +49 -0
- package/skills/roles/engineer.md +85 -0
- package/skills/roles/engineer.platform.md +49 -0
- package/skills/roles/operator.docs.md +43 -0
- package/skills/roles/operator.md +68 -0
- package/skills/roles/operator.release.md +43 -0
- package/skills/roles/operator.sre.md +43 -0
- package/skills/roles/orchestrator.md +66 -0
- package/skills/roles/product-manager.ai-product.md +37 -0
- package/skills/roles/product-manager.business-strategy.md +43 -0
- package/skills/roles/product-manager.enterprise.md +37 -0
- package/skills/roles/product-manager.growth.md +37 -0
- package/skills/roles/product-manager.md +67 -0
- package/skills/roles/product-manager.platform.md +37 -0
- package/skills/roles/product-manager.product.md +37 -0
- package/skills/roles/qa.ai-eval.md +37 -0
- package/skills/roles/qa.api-contract.md +37 -0
- package/skills/roles/qa.data-pipeline.md +37 -0
- package/skills/roles/qa.md +68 -0
- package/skills/roles/qa.test-automation.md +49 -0
- package/skills/roles/qa.web-ui.md +37 -0
- package/skills/roles/researcher.explorer.md +43 -0
- package/skills/roles/researcher.md +68 -0
- package/skills/roles/researcher.ux.md +43 -0
- package/skills/roles/reviewer.devil-advocate.md +43 -0
- package/skills/roles/reviewer.evaluator.md +43 -0
- package/skills/roles/reviewer.md +68 -0
- package/skills/roles/reviewer.trace.md +43 -0
- package/skills/roles/security.ai.md +37 -0
- package/skills/roles/security.appsec.md +37 -0
- package/skills/roles/security.cloud.md +37 -0
- package/skills/roles/security.legal-compliance.md +49 -0
- package/skills/roles/security.md +68 -0
- package/skills/roles/security.privacy.md +37 -0
- package/skills/roles/security.supply-chain.md +37 -0
- package/skills/routing.md +139 -0
- package/skills/security/blue-team.md +83 -0
- package/skills/security/code-audit.md +94 -0
- package/skills/security/pentest.md +84 -0
- package/skills/security/red-team.md +81 -0
- package/skills/security/threat-intel.md +89 -0
- package/skills/security/vuln-research.md +87 -0
- package/skills/utility/clean-code.md +165 -0
- package/sync-agents.mjs +899 -0
- package/templates/docs/adr.md +27 -0
- package/templates/docs/backlog-proposal.md +26 -0
- package/templates/docs/customer-profile.md +34 -0
- package/templates/docs/evidence-brief.md +37 -0
- package/templates/docs/incident-report.md +46 -0
- package/templates/docs/memo.md +27 -0
- package/templates/docs/meta-prd.md +56 -0
- package/templates/docs/one-pager.md +24 -0
- package/templates/docs/prd-business.md +61 -0
- package/templates/docs/prd-platform.md +62 -0
- package/templates/docs/prd.md +47 -0
- package/templates/docs/prfaq.md +31 -0
- package/templates/docs/product-intelligence-report.md +31 -0
- package/templates/docs/research-brief.md +29 -0
- package/templates/docs/rfc-platform.md +60 -0
- package/templates/docs/rfc.md +46 -0
- package/templates/docs/runbook.md +33 -0
- package/templates/docs/signal-brief.md +25 -0
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/ai/ml-ops.md — ML Operations (MLOps) — ```
|
|
3
|
+
|
|
4
|
+
## The MLOps Lifecycle ```
|
|
5
|
+
-->
|
|
6
|
+
# ML Operations (MLOps)
|
|
7
|
+
|
|
8
|
+
## The MLOps Lifecycle
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
Data → Feature Engineering → Training → Evaluation → Registry → Deployment → Monitoring → (loop)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Each stage should be reproducible, versioned, and automated.
|
|
15
|
+
|
|
16
|
+
## Data Management
|
|
17
|
+
|
|
18
|
+
### Data versioning
|
|
19
|
+
|
|
20
|
+
Use **DVC** (Data Version Control) alongside git to version datasets and models:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
dvc add data/training.csv # track large file outside git
|
|
24
|
+
dvc push # push to remote storage (S3, GCS, Azure)
|
|
25
|
+
git add data/training.csv.dvc # commit the pointer
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
### Feature stores
|
|
29
|
+
|
|
30
|
+
For team-shared features or online serving:
|
|
31
|
+
- **Feast** (open source): offline (data warehouse) + online (Redis/DynamoDB) store
|
|
32
|
+
- **Tecton**, **Hopsworks** for managed options
|
|
33
|
+
|
|
34
|
+
Feature pipelines should be:
|
|
35
|
+
1. Deterministic (same input → same output)
|
|
36
|
+
2. Backfillable (can recompute historical features)
|
|
37
|
+
3. Point-in-time correct (no future data leakage)
|
|
38
|
+
|
|
39
|
+
## Experiment Tracking
|
|
40
|
+
|
|
41
|
+
Use **MLflow** or **Weights & Biases** to log:
|
|
42
|
+
|
|
43
|
+
```python
|
|
44
|
+
import mlflow
|
|
45
|
+
|
|
46
|
+
with mlflow.start_run(run_name="baseline-xgb"):
|
|
47
|
+
mlflow.log_params({"max_depth": 6, "n_estimators": 100})
|
|
48
|
+
model.fit(X_train, y_train)
|
|
49
|
+
metrics = evaluate(model, X_val, y_val)
|
|
50
|
+
mlflow.log_metrics(metrics)
|
|
51
|
+
mlflow.sklearn.log_model(model, "model")
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Minimum to log per experiment: hyperparameters, dataset version/hash, evaluation metrics, model artifact.
|
|
55
|
+
|
|
56
|
+
## Training Pipelines
|
|
57
|
+
|
|
58
|
+
### Pipeline orchestration
|
|
59
|
+
|
|
60
|
+
Use **Kubeflow Pipelines**, **Prefect**, **Airflow**, or **ZenML**:
|
|
61
|
+
|
|
62
|
+
```python
|
|
63
|
+
@pipeline
|
|
64
|
+
def training_pipeline():
|
|
65
|
+
data = ingest_data()
|
|
66
|
+
features = engineer_features(data)
|
|
67
|
+
model = train_model(features)
|
|
68
|
+
evaluate_model(model, features)
|
|
69
|
+
push_to_registry(model)
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
- Each step is a containerized function with explicit inputs/outputs
|
|
73
|
+
- Cache step outputs to avoid re-running expensive transformations
|
|
74
|
+
- Use hardware-specific steps (GPU) only where necessary
|
|
75
|
+
|
|
76
|
+
### Reproducibility requirements
|
|
77
|
+
|
|
78
|
+
- Pin library versions (`requirements.txt` or `conda.yml` locked)
|
|
79
|
+
- Log random seeds
|
|
80
|
+
- Log git commit hash with the run
|
|
81
|
+
- Use deterministic data splits (fixed random seed, or pre-split stored files)
|
|
82
|
+
|
|
83
|
+
## Model Registry
|
|
84
|
+
|
|
85
|
+
Central store for model versions and deployment state:
|
|
86
|
+
|
|
87
|
+
| Stage | Meaning |
|
|
88
|
+
|---|---|
|
|
89
|
+
| Staging | Candidate — passes evaluation, not yet in production |
|
|
90
|
+
| Production | Currently serving |
|
|
91
|
+
| Archived | Superseded |
|
|
92
|
+
|
|
93
|
+
MLflow Model Registry, SageMaker Model Registry, or Vertex AI Model Registry all follow this pattern.
|
|
94
|
+
|
|
95
|
+
## Deployment Patterns
|
|
96
|
+
|
|
97
|
+
### Online serving (real-time inference)
|
|
98
|
+
|
|
99
|
+
- **Containerize** the model + serving code: `docker build -t model-server .`
|
|
100
|
+
- **Serve** via FastAPI / Triton Inference Server / TorchServe
|
|
101
|
+
- **Deploy** to Kubernetes with horizontal pod autoscaling on GPU/CPU utilization
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
# FastAPI serving example
|
|
105
|
+
@app.post("/predict")
|
|
106
|
+
async def predict(req: PredictRequest) -> PredictResponse:
|
|
107
|
+
features = feature_pipeline(req.input)
|
|
108
|
+
score = model.predict(features)
|
|
109
|
+
return PredictResponse(score=score)
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### Batch inference
|
|
113
|
+
|
|
114
|
+
- Schedule periodic jobs (Airflow, Prefect) to score datasets
|
|
115
|
+
- Write results to a data warehouse or feature store
|
|
116
|
+
- Use Spark for large-scale batch scoring
|
|
117
|
+
|
|
118
|
+
### A/B testing / Shadow mode
|
|
119
|
+
|
|
120
|
+
- **Shadow mode**: new model receives traffic and logs predictions but does not serve users — validate before switching
|
|
121
|
+
- **A/B test**: split traffic (e.g., 10% new model) and measure business metrics, not just accuracy
|
|
122
|
+
- Use a feature flag system to control routing
|
|
123
|
+
|
|
124
|
+
## Monitoring
|
|
125
|
+
|
|
126
|
+
### What to monitor
|
|
127
|
+
|
|
128
|
+
| Signal | Why |
|
|
129
|
+
|---|---|
|
|
130
|
+
| Prediction distribution | Detect output drift |
|
|
131
|
+
| Feature distribution | Detect input drift (data pipeline change) |
|
|
132
|
+
| Model latency (p50, p95, p99) | SLO compliance |
|
|
133
|
+
| Error rate / null predictions | Serving failures |
|
|
134
|
+
| Business metric (CTR, revenue) | Downstream impact |
|
|
135
|
+
|
|
136
|
+
### Drift detection
|
|
137
|
+
|
|
138
|
+
```python
|
|
139
|
+
from evidently.report import Report
|
|
140
|
+
from evidently.metric_preset import DataDriftPreset
|
|
141
|
+
|
|
142
|
+
report = Report(metrics=[DataDriftPreset()])
|
|
143
|
+
report.run(reference_data=reference_df, current_data=production_df)
|
|
144
|
+
report.save_html("drift_report.html")
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Tools: **Evidently**, **WhyLogs**, **Seldon Alibi Detect**.
|
|
148
|
+
|
|
149
|
+
Trigger retraining when:
|
|
150
|
+
- Statistical drift test fails (KS test, PSI > 0.2)
|
|
151
|
+
- Business metric degrades below threshold for N consecutive periods
|
|
152
|
+
- Scheduled periodic retraining cadence (baseline)
|
|
153
|
+
|
|
154
|
+
## Infrastructure
|
|
155
|
+
|
|
156
|
+
- **Container registry**: ECR, GCR, GHCR
|
|
157
|
+
- **Object storage**: S3/GCS for datasets, model artifacts, logs
|
|
158
|
+
- **Feature store**: Redis (online), BigQuery/Snowflake (offline)
|
|
159
|
+
- **Orchestration**: Kubeflow, Airflow, Prefect, Vertex Pipelines
|
|
160
|
+
- **Serving**: Kubernetes + Knative / SageMaker Endpoints / Vertex AI Endpoints
|
|
161
|
+
- **Monitoring**: Prometheus + Grafana for infrastructure; Evidently/WhyLogs for model health
|
|
162
|
+
|
|
163
|
+
## Common Pitfalls
|
|
164
|
+
|
|
165
|
+
- **Training-serving skew**: feature engineering in training differs from serving — solve with a shared feature pipeline
|
|
166
|
+
- **Data leakage**: future information in training features inflates metrics — use strict temporal splits
|
|
167
|
+
- **Model staleness without detection**: set up drift monitoring day 1, not after the model degrades
|
|
168
|
+
- **No rollback plan**: always keep N-1 model version in registry and test rollback procedure
|
|
169
|
+
- **GPU waste**: jobs that could run on CPU scheduled on GPU — profile resource usage
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/ai/orchestration-workflow.md — Construct Orchestration Workflow — Use this skill when the request involves agent orchestration, phase transitions,
|
|
3
|
+
|
|
4
|
+
Use this skill when the request involves agent orchestration, phase transitions, task keys, handoff quality, workflow state, or project alignment. ## Core Model
|
|
5
|
+
-->
|
|
6
|
+
# Construct Orchestration Workflow
|
|
7
|
+
|
|
8
|
+
Use this skill when the request involves agent orchestration, phase transitions, task keys, handoff quality, workflow state, or project alignment.
|
|
9
|
+
|
|
10
|
+
## Core Model
|
|
11
|
+
|
|
12
|
+
Construct is an executive-aligned, control-plane workflow:
|
|
13
|
+
|
|
14
|
+
- **Universal Orchestrator**: **Construct** is the single point of entry. It owns the outcome from strategy to production, dispatching specialists autonomously.
|
|
15
|
+
- **Executive Checkpoints**: High-gate phases (Planning and Verification) require explicit Customer/Executive sign-off via the dashboard or CLI.
|
|
16
|
+
- **Specialists**: `cx-*` agents own bounded work packages.
|
|
17
|
+
- **Skills**: Reusable execution playbooks (searched via `search_skills`).
|
|
18
|
+
- **Hooks**: Enforce continuity and system integrity (e.g. `pre-push-gate`, `dep-audit`).
|
|
19
|
+
- **Resumption Protocol**: Every new session MUST begin with `workflow_status` and `project_context` to prevent "state amnesia."
|
|
20
|
+
- **.cx/workflow.json**: The authoritative local task graph (Beads pattern).
|
|
21
|
+
- **Cass (Memory MCP)**: Preserves durable project context across sessions and platforms.
|
|
22
|
+
|
|
23
|
+
## Required State
|
|
24
|
+
|
|
25
|
+
For non-trivial work, create or update `.cx/workflow.json` with:
|
|
26
|
+
|
|
27
|
+
- `phase`: current phase (research, plan, implement, validate, operate)
|
|
28
|
+
- `currentTaskKey`: active task
|
|
29
|
+
- `tasks[].key`: stable `todo:N` key
|
|
30
|
+
- `tasks[].phase`: corresponding phase
|
|
31
|
+
- `tasks[].owner`: persona or cx-specialist
|
|
32
|
+
- `tasks[].status`: todo, in-progress, blocked, blocked_needs_user, blocked_needs_executive, done, or skipped
|
|
33
|
+
- `tasks[].readFirst`: files, docs, or memory queries to inspect first
|
|
34
|
+
- `tasks[].doNotChange`: protected files or surfaces
|
|
35
|
+
- `tasks[].acceptanceCriteria`: binary pass/fail checks
|
|
36
|
+
- `tasks[].verification`: evidence proving completion
|
|
37
|
+
|
|
38
|
+
## Commands
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
construct do "Goal" # Unified natural language entry point
|
|
42
|
+
construct workflow approve # Executive sign-off on current phase
|
|
43
|
+
construct workflow status # Check alignment and progress
|
|
44
|
+
construct workflow align # Sync state and identify drift
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Worker Packets
|
|
48
|
+
|
|
49
|
+
Specialists do not run open-ended conversations with the user. They return one terminal state:
|
|
50
|
+
|
|
51
|
+
- `DONE`: changed files, decisions, verification evidence
|
|
52
|
+
- `BLOCKED`: blocker, attempted steps, safest next action
|
|
53
|
+
- `NEEDS_MAIN_INPUT`: `{ taskKey, worker, blocker, question, safeDefault, context }`
|
|
54
|
+
|
|
55
|
+
When a worker needs user input, mark the task `blocked_needs_user`, return the `NEEDS_MAIN_INPUT` packet, and let the active persona ask the user in the main session.
|
|
56
|
+
|
|
57
|
+
## Phase Gates
|
|
58
|
+
|
|
59
|
+
Plan -> Implement:
|
|
60
|
+
- Requirements are explicit.
|
|
61
|
+
- Risks have been challenged.
|
|
62
|
+
- Tasks have TASK_KEY, owner, readFirst, doNotChange, and acceptance criteria.
|
|
63
|
+
|
|
64
|
+
Implement -> Validate:
|
|
65
|
+
- Implementation is complete.
|
|
66
|
+
- Verification evidence is recorded.
|
|
67
|
+
- Changed files are listed.
|
|
68
|
+
|
|
69
|
+
Validate -> Operate:
|
|
70
|
+
- CRITICAL and HIGH findings are resolved.
|
|
71
|
+
- Remaining risks are explicitly accepted.
|
|
72
|
+
|
|
73
|
+
Operate -> Done:
|
|
74
|
+
- Health checks pass.
|
|
75
|
+
- Rollback path is documented.
|
|
76
|
+
- Release or deployment status is recorded.
|
|
77
|
+
|
|
78
|
+
## Output Discipline
|
|
79
|
+
|
|
80
|
+
Keep orchestration mechanics mostly internal. The user should see:
|
|
81
|
+
|
|
82
|
+
- current phase
|
|
83
|
+
- current task or blocker
|
|
84
|
+
- decisions that changed the plan
|
|
85
|
+
- verification results
|
|
86
|
+
|
|
87
|
+
Do not dump full handoff templates unless the host cannot dispatch the next phase or the user asks to see the handoff.
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/ai/prompt-and-eval.md — Prompt Engineering and Evaluation — Use this skill when designing prompts, evaluating model performance, or optimizi
|
|
3
|
+
|
|
4
|
+
Use this skill when designing prompts, evaluating model performance, or optimizing LLM behavior. ## Prompt Structure
|
|
5
|
+
-->
|
|
6
|
+
# Prompt Engineering and Evaluation
|
|
7
|
+
|
|
8
|
+
Use this skill when designing prompts, evaluating model performance, or optimizing LLM behavior.
|
|
9
|
+
|
|
10
|
+
## Prompt Structure
|
|
11
|
+
|
|
12
|
+
### System Prompt
|
|
13
|
+
- Define the role and persona
|
|
14
|
+
- State capabilities and constraints
|
|
15
|
+
- Specify output format and style
|
|
16
|
+
- Include examples of correct behavior
|
|
17
|
+
- Keep stable across conversations; parameterize the variable parts
|
|
18
|
+
|
|
19
|
+
### User Prompt
|
|
20
|
+
- Be specific: state the task, context, and expected output
|
|
21
|
+
- Provide relevant context; omit irrelevant information
|
|
22
|
+
- Break complex tasks into steps
|
|
23
|
+
- Use delimiters to separate instructions from data
|
|
24
|
+
|
|
25
|
+
### Few-Shot Examples
|
|
26
|
+
- a small set of examples is typically sufficient (often 3-7)
|
|
27
|
+
- Cover the range of expected inputs and edge cases
|
|
28
|
+
- Order examples from simple to complex
|
|
29
|
+
- Use consistent formatting across examples
|
|
30
|
+
- Include examples of correct refusal or uncertainty
|
|
31
|
+
|
|
32
|
+
## Techniques
|
|
33
|
+
|
|
34
|
+
### Chain of Thought
|
|
35
|
+
- "Think step by step" or explicit reasoning scaffolding
|
|
36
|
+
- Improves accuracy on math, logic, and multi-step reasoning
|
|
37
|
+
- Increases token usage; use when accuracy justifies cost
|
|
38
|
+
- Can be made invisible to users with structured output parsing
|
|
39
|
+
|
|
40
|
+
### Self-Consistency
|
|
41
|
+
- Generate multiple responses with temperature >0
|
|
42
|
+
- Select the most common answer (majority vote)
|
|
43
|
+
- Improves reliability on ambiguous or difficult tasks
|
|
44
|
+
- Cost multiplied by number of samples
|
|
45
|
+
|
|
46
|
+
### Structured Output
|
|
47
|
+
- Request JSON, YAML, or XML with a defined schema
|
|
48
|
+
- Provide the schema in the prompt
|
|
49
|
+
- Validate outputs programmatically
|
|
50
|
+
- Use constrained decoding when available
|
|
51
|
+
|
|
52
|
+
### Retrieval-Augmented Generation
|
|
53
|
+
- Include retrieved context with clear source attribution
|
|
54
|
+
- Instruct the model to answer only from provided context
|
|
55
|
+
- See `skills/ai/rag-system.md` for full RAG guidance
|
|
56
|
+
|
|
57
|
+
## Anti-Patterns
|
|
58
|
+
|
|
59
|
+
- Vague instructions ("do your best", "be creative")
|
|
60
|
+
- Conflicting constraints in the same prompt
|
|
61
|
+
- Excessive context that buries the actual task
|
|
62
|
+
- Over-relying on temperature to fix prompt quality issues
|
|
63
|
+
- Not testing prompts against edge cases before deployment
|
|
64
|
+
|
|
65
|
+
## Evaluation Framework
|
|
66
|
+
|
|
67
|
+
### Offline Evaluation
|
|
68
|
+
1. Build a labeled dataset: input + expected output
|
|
69
|
+
2. Define metrics per task type (see below)
|
|
70
|
+
3. Run the full pipeline on the eval set
|
|
71
|
+
4. Track metrics over time; catch regressions
|
|
72
|
+
5. Automate eval on every prompt or model change
|
|
73
|
+
|
|
74
|
+
### Online Evaluation
|
|
75
|
+
- A/B test prompt variants on live traffic
|
|
76
|
+
- Collect user feedback: thumbs up/down, corrections
|
|
77
|
+
- Monitor: latency, cost, user satisfaction, task completion
|
|
78
|
+
|
|
79
|
+
### Metrics by Task Type
|
|
80
|
+
|
|
81
|
+
| Task | Primary Metrics |
|
|
82
|
+
|---|---|
|
|
83
|
+
| Classification | Precision, recall, F1, confusion matrix |
|
|
84
|
+
| Generation | Faithfulness, relevance, fluency, human preference |
|
|
85
|
+
| Extraction | Exact match, field-level accuracy, hallucination rate |
|
|
86
|
+
| Summarization | ROUGE, faithfulness, compression ratio, human eval |
|
|
87
|
+
| Code generation | Pass rate on test cases, correctness, security |
|
|
88
|
+
| Conversation | Task completion, turns to resolution, user satisfaction |
|
|
89
|
+
|
|
90
|
+
### Human Evaluation
|
|
91
|
+
- Use when automated metrics are insufficient
|
|
92
|
+
- Blind evaluation: evaluators do not know which variant they rate
|
|
93
|
+
- Inter-annotator agreement to measure consistency
|
|
94
|
+
- Rubric-based scoring with clear criteria
|
|
95
|
+
- Sample size sufficient for statistical significance
|
|
96
|
+
|
|
97
|
+
## Model Selection
|
|
98
|
+
|
|
99
|
+
| Consideration | Guidance |
|
|
100
|
+
|---|---|
|
|
101
|
+
| Latency | Smaller models for real-time; larger for batch |
|
|
102
|
+
| Cost | Price per token varies 10-100x across model tiers |
|
|
103
|
+
| Accuracy | Benchmark on your specific task, not general leaderboards |
|
|
104
|
+
| Context window | Match to your typical input + output size |
|
|
105
|
+
| Fine-tuning | When prompting alone cannot capture domain patterns |
|
|
106
|
+
|
|
107
|
+
## Optimization Workflow
|
|
108
|
+
|
|
109
|
+
1. Start with a clear prompt on the strongest available model
|
|
110
|
+
2. Evaluate on a representative dataset
|
|
111
|
+
3. Iterate on the prompt to fix failure modes
|
|
112
|
+
4. Once prompt is stable, test on smaller/cheaper models
|
|
113
|
+
5. Fine-tune only when prompt engineering plateaus
|
|
114
|
+
6. Continuously monitor and re-evaluate after deployment
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/ai/prompt-optimizer.md — Closed-loop prompt auto-optimization guide.
|
|
3
|
+
|
|
4
|
+
Uses Langfuse traces and quality scores as the feedback signal, Claude as the optimizer,
|
|
5
|
+
and the agent registry + construct sync as the deployment layer.
|
|
6
|
+
-->
|
|
7
|
+
# Prompt Auto-Optimization Loop
|
|
8
|
+
|
|
9
|
+
Construct's prompt improvement system uses Langfuse traces and quality scores as the feedback signal, Claude as the optimizer, and the agent registry (`agents/registry.json`) as the deployment layer. This is a closed loop: production data → failure analysis → improved prompt → staging → promotion.
|
|
10
|
+
|
|
11
|
+
## Running the optimizer
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
# Analyze and push staging version
|
|
15
|
+
construct optimize cx-engineer
|
|
16
|
+
|
|
17
|
+
# Dry run — see failure patterns without pushing
|
|
18
|
+
construct optimize cx-engineer --dry-run
|
|
19
|
+
|
|
20
|
+
# List all agents with current quality scores
|
|
21
|
+
construct optimize --list
|
|
22
|
+
|
|
23
|
+
# Tune parameters
|
|
24
|
+
construct optimize cx-debugger --threshold=0.65 --days=14 --min-traces=15
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
The optimizer requires Python 3.12+ (`pip3` must be available). It will auto-install `dspy-ai` and `requests` on first run. Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` in `.env`. DSPy uses the same LLM key Construct uses — no separate setup.
|
|
28
|
+
|
|
29
|
+
## When to run
|
|
30
|
+
|
|
31
|
+
- Triggered by `/work:optimize-prompts` (manual) or the scheduled task `prompt-optimization-weekly`
|
|
32
|
+
- Automatically suggested when `cx-trace-reviewer` finds an agent with median quality score below 0.65 over the past 7 days
|
|
33
|
+
- Never run optimization on prompts with fewer than 20 scored traces — insufficient signal
|
|
34
|
+
|
|
35
|
+
## Step 1 — Gather signal
|
|
36
|
+
|
|
37
|
+
Retrieve the current production prompt for the target agent from `agents/registry.json` (or the corresponding `promptFile` if using extracted prompts).
|
|
38
|
+
|
|
39
|
+
Then retrieve recent traces via the Langfuse REST API:
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
GET {LANGFUSE_BASEURL}/api/public/traces?tags={agentName}&limit=50
|
|
43
|
+
# Auth: Basic base64(LANGFUSE_PUBLIC_KEY:LANGFUSE_SECRET_KEY)
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
To fetch quality scores for traces:
|
|
47
|
+
```
|
|
48
|
+
GET {LANGFUSE_BASEURL}/api/public/scores?traceId={id}&name=quality
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Filter to scores where `value < 0.7`. For each low-scoring trace, extract: the prompt used, the user input, the model output, the quality score, and any human comments.
|
|
52
|
+
|
|
53
|
+
## Step 2 — Diagnose failure patterns
|
|
54
|
+
|
|
55
|
+
Analyze the low-scoring traces as a batch. Identify recurring failure modes. Common patterns:
|
|
56
|
+
|
|
57
|
+
| Pattern | Diagnostic signal |
|
|
58
|
+
|---|---|
|
|
59
|
+
| Output too verbose | Long outputs consistently score low; user messages are short questions |
|
|
60
|
+
| Missing context | Outputs lack specific file/line references; traces show no Read tool calls |
|
|
61
|
+
| Wrong routing | Agent performs work outside its stated role |
|
|
62
|
+
| Hallucination risk | Outputs assert facts not present in tool results |
|
|
63
|
+
| Format drift | Output format varies; scoring is inconsistent on structure |
|
|
64
|
+
| Insufficient depth | Outputs are correct but shallow; scored down for missing detail |
|
|
65
|
+
|
|
66
|
+
Write a failure summary: top 3 patterns with supporting trace count and representative examples.
|
|
67
|
+
|
|
68
|
+
## Step 3 — Generate improved prompt
|
|
69
|
+
|
|
70
|
+
Write an improved prompt that directly addresses the diagnosed failures. Rules:
|
|
71
|
+
|
|
72
|
+
1. **Keep what works** — compare high-scoring traces (>0.8) to low-scoring ones. Only change what's associated with failures.
|
|
73
|
+
2. **Surgical edits, not rewrites** — changing everything risks breaking current strengths. Identify the specific clauses that correlate with failures.
|
|
74
|
+
3. **Be explicit, not vague** — if the failure is "too verbose", add a concrete rule ("respond in under 150 words for questions that fit on one line") not a general note ("be concise").
|
|
75
|
+
4. **Add a self-check instruction** — append a brief checklist the agent runs before responding, derived from the top failure patterns.
|
|
76
|
+
|
|
77
|
+
## Step 4 — Push to staging
|
|
78
|
+
|
|
79
|
+
Update the agent's prompt in `agents/registry.json` (or the corresponding `promptFile`) with a staging marker comment. Tag the version by writing to `.cx/decisions/prompt-staging-{agent}-{date}.md`.
|
|
80
|
+
|
|
81
|
+
Log the candidate prompt as a span attribute on a test run batch using `cx_trace`. Tag spans with `promptVersion: staging-{timestamp}` and score them with `cx_score` as traces complete.
|
|
82
|
+
|
|
83
|
+
Do not overwrite the production prompt in the registry until promotion is confirmed.
|
|
84
|
+
|
|
85
|
+
## Step 5 — Monitor staging
|
|
86
|
+
|
|
87
|
+
After at least 20 scored traces on the staging version:
|
|
88
|
+
- Compare median quality score: staging vs production
|
|
89
|
+
- If staging median > production median + 0.05: promote
|
|
90
|
+
- If staging median ≤ production median: rollback (restore previous prompt, document why)
|
|
91
|
+
|
|
92
|
+
To promote: update the agent registry with the accepted prompt and run `construct sync`.
|
|
93
|
+
|
|
94
|
+
## Step 6 — Document the optimization
|
|
95
|
+
|
|
96
|
+
Write to `.cx/decisions/` with:
|
|
97
|
+
- Which agent was optimized
|
|
98
|
+
- Previous vs new median quality score
|
|
99
|
+
- Top 3 failure patterns addressed
|
|
100
|
+
- What changed in the prompt (diff summary)
|
|
101
|
+
- Date and version numbers
|
|
102
|
+
|
|
103
|
+
This becomes the audit trail for future optimizations and regression analysis.
|
|
104
|
+
|
|
105
|
+
## Regression detection
|
|
106
|
+
|
|
107
|
+
Run `cx-trace-reviewer` after every promotion. If the newly promoted prompt shows worse performance after 48 hours of production traffic:
|
|
108
|
+
1. Revert to the previous prompt in the registry and run `construct sync`
|
|
109
|
+
2. Start a new optimization cycle with the regression as the primary failure signal
|
|
110
|
+
|
|
111
|
+
## What this does not replace
|
|
112
|
+
|
|
113
|
+
- **DSPy**: If you need algorithmic optimization over large datasets with measurable metrics (classification, structured output), DSPy is the right tool and integrates with Langfuse for tracing. This skill is for natural-language agent prompts where the metric is quality score.
|
|
114
|
+
- **Human review**: Always read the generated prompt before pushing to staging. The LLM optimizer can introduce subtle regressions. Automated promotion should only happen after confirming the diff makes sense.
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/ai/rag-system.md — RAG System Design — Use this skill when building retrieval-augmented generation pipelines, vector se
|
|
3
|
+
|
|
4
|
+
Use this skill when building retrieval-augmented generation pipelines, vector search, or knowledge-grounded AI systems. ## Document Processing
|
|
5
|
+
-->
|
|
6
|
+
# RAG System Design
|
|
7
|
+
|
|
8
|
+
Use this skill when building retrieval-augmented generation pipelines, vector search, or knowledge-grounded AI systems.
|
|
9
|
+
|
|
10
|
+
## Document Processing
|
|
11
|
+
|
|
12
|
+
### Chunking Strategies
|
|
13
|
+
- Fixed size: 512-1024 tokens with 10-20% overlap; simple, predictable
|
|
14
|
+
- Semantic: split on paragraph, section, or topic boundaries; preserves coherence
|
|
15
|
+
- Recursive: try large chunks, split smaller only when needed
|
|
16
|
+
- Document-aware: respect headers, tables, code blocks as atomic units
|
|
17
|
+
- Always preserve metadata: source, page, section, timestamp
|
|
18
|
+
|
|
19
|
+
### Preprocessing
|
|
20
|
+
- Extract text from PDFs, DOCX, HTML; preserve structure
|
|
21
|
+
- Normalize whitespace, encoding, and character sets
|
|
22
|
+
- Strip boilerplate (headers, footers, navigation) from web pages
|
|
23
|
+
- Deduplicate identical or near-identical chunks
|
|
24
|
+
- Handle tables and images: extract text, generate descriptions
|
|
25
|
+
|
|
26
|
+
## Embedding
|
|
27
|
+
|
|
28
|
+
- Choose embedding model based on retrieval quality, not just speed
|
|
29
|
+
- Match embedding model to chunk size (most optimize for 256-512 tokens)
|
|
30
|
+
- Normalize vectors if using cosine similarity
|
|
31
|
+
- Benchmark retrieval accuracy on a representative eval set before deploying
|
|
32
|
+
- Consider late-interaction models (ColBERT) for higher accuracy at more compute
|
|
33
|
+
|
|
34
|
+
## Vector Database
|
|
35
|
+
|
|
36
|
+
| System | Strength |
|
|
37
|
+
|---|---|
|
|
38
|
+
| Pinecone | Managed, serverless scaling |
|
|
39
|
+
| Weaviate | Hybrid search, multi-modal |
|
|
40
|
+
| Qdrant | Filtering, payload storage |
|
|
41
|
+
| pgvector | Postgres-native, simple deployment |
|
|
42
|
+
| Chroma | Local development, lightweight |
|
|
43
|
+
|
|
44
|
+
### Indexing
|
|
45
|
+
- HNSW for high recall with reasonable memory
|
|
46
|
+
- IVF for large-scale with memory constraints
|
|
47
|
+
- Tune `ef_construction` and `M` parameters for recall vs build time
|
|
48
|
+
- Partition by tenant or document collection for isolation
|
|
49
|
+
|
|
50
|
+
## Retrieval
|
|
51
|
+
|
|
52
|
+
### Hybrid Search
|
|
53
|
+
- Combine dense vector search with sparse keyword search (BM25)
|
|
54
|
+
- Use reciprocal rank fusion to merge result lists
|
|
55
|
+
- Keyword search catches exact matches that embeddings miss
|
|
56
|
+
- Dense search captures semantic similarity that keywords miss
|
|
57
|
+
|
|
58
|
+
### Reranking
|
|
59
|
+
- Cross-encoder reranker on top-k candidates (k=20-50)
|
|
60
|
+
- Improves precision significantly at moderate latency cost
|
|
61
|
+
- Use a smaller, faster model than the generation LLM
|
|
62
|
+
|
|
63
|
+
### Query Processing
|
|
64
|
+
- Query expansion: rephrase user query into multiple search queries
|
|
65
|
+
- HyDE: generate a hypothetical answer, embed it, search with that vector
|
|
66
|
+
- Decomposition: break complex queries into sub-queries, retrieve for each
|
|
67
|
+
- Filter by metadata when the query implies scope (date, source, category)
|
|
68
|
+
|
|
69
|
+
## Context Assembly
|
|
70
|
+
|
|
71
|
+
- Order retrieved chunks by relevance
|
|
72
|
+
- Deduplicate overlapping content
|
|
73
|
+
- Include source attribution for each chunk
|
|
74
|
+
- Respect context window limits; prioritize quality over quantity
|
|
75
|
+
- Typical prompt structure: system instructions, retrieved context, user query
|
|
76
|
+
|
|
77
|
+
## Evaluation
|
|
78
|
+
|
|
79
|
+
### Retrieval Metrics
|
|
80
|
+
- Recall@k: did the relevant chunks appear in top-k results?
|
|
81
|
+
- MRR: how high is the first relevant result?
|
|
82
|
+
- NDCG: overall ranking quality of returned results
|
|
83
|
+
|
|
84
|
+
### Generation Metrics
|
|
85
|
+
- Faithfulness: does the answer stay grounded in retrieved context?
|
|
86
|
+
- Relevance: does the answer address the user's question?
|
|
87
|
+
- Completeness: does the answer cover all aspects of the query?
|
|
88
|
+
- Hallucination rate: percentage of claims not supported by context
|
|
89
|
+
|
|
90
|
+
### Eval Process
|
|
91
|
+
1. Build a labeled dataset: query + relevant passages + expected answer
|
|
92
|
+
2. Measure retrieval quality independent of generation
|
|
93
|
+
3. Measure generation quality with gold-standard context
|
|
94
|
+
4. Measure end-to-end with the full pipeline
|
|
95
|
+
5. Automate eval runs on every pipeline change
|
|
96
|
+
|
|
97
|
+
## Common Failures
|
|
98
|
+
|
|
99
|
+
- Chunk too large: dilutes relevance signal
|
|
100
|
+
- Chunk too small: loses context needed for coherent answers
|
|
101
|
+
- Missing metadata: cannot filter or attribute sources
|
|
102
|
+
- No reranking: first-stage retrieval alone has low precision
|
|
103
|
+
- No eval: impossible to know if changes improve or degrade quality
|
|
104
|
+
- Stale index: documents updated but embeddings not refreshed
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
<!--
|
|
2
|
+
skills/architecture/api-design.md — API Design — Use this skill when designing REST, GraphQL, or gRPC APIs.
|
|
3
|
+
|
|
4
|
+
Use this skill when designing REST, GraphQL, or gRPC APIs. ## REST Design Principles
|
|
5
|
+
-->
|
|
6
|
+
# API Design
|
|
7
|
+
|
|
8
|
+
Use this skill when designing REST, GraphQL, or gRPC APIs.
|
|
9
|
+
|
|
10
|
+
## REST Design Principles
|
|
11
|
+
|
|
12
|
+
### Resource Naming
|
|
13
|
+
- Use plural nouns: `/users`, `/orders`, `/products`
|
|
14
|
+
- Nest for relationships: `/users/{id}/orders`
|
|
15
|
+
- Limit nesting to two levels; use query params for deeper filtering
|
|
16
|
+
- Use kebab-case for multi-word paths: `/order-items`
|
|
17
|
+
- Avoid verbs in URLs; let HTTP methods express the action
|
|
18
|
+
|
|
19
|
+
### HTTP Methods
|
|
20
|
+
- GET: read, idempotent, cacheable
|
|
21
|
+
- POST: create, not idempotent
|
|
22
|
+
- PUT: full replace, idempotent
|
|
23
|
+
- PATCH: partial update, idempotent
|
|
24
|
+
- DELETE: remove, idempotent
|
|
25
|
+
|
|
26
|
+
### Status Codes
|
|
27
|
+
- 200 OK: successful read or update
|
|
28
|
+
- 201 Created: successful creation, include Location header
|
|
29
|
+
- 204 No Content: successful delete
|
|
30
|
+
- 400 Bad Request: validation failure
|
|
31
|
+
- 401 Unauthorized: missing or invalid authentication
|
|
32
|
+
- 403 Forbidden: authenticated but not authorized
|
|
33
|
+
- 404 Not Found: resource does not exist
|
|
34
|
+
- 409 Conflict: state conflict (duplicate, version mismatch)
|
|
35
|
+
- 429 Too Many Requests: rate limited
|
|
36
|
+
- 500 Internal Server Error: unhandled server failure
|
|
37
|
+
|
|
38
|
+
### Versioning
|
|
39
|
+
- URL path versioning (`/v1/users`) for major breaking changes
|
|
40
|
+
- Header versioning for minor variations
|
|
41
|
+
- Support at most two major versions concurrently
|
|
42
|
+
- Deprecation period: minimum 6 months with Sunset header
|
|
43
|
+
|
|
44
|
+
## Response Envelope
|
|
45
|
+
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"data": {},
|
|
49
|
+
"error": null,
|
|
50
|
+
"meta": { "page": 1, "limit": 20, "total": 142 }
|
|
51
|
+
}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
- Consistent structure across all endpoints
|
|
55
|
+
- Paginate all list endpoints; default limit 20, max 100
|
|
56
|
+
- Use cursor-based pagination for large or real-time datasets
|
|
57
|
+
- Include total count only when it is cheap to compute
|
|
58
|
+
|
|
59
|
+
## GraphQL Design
|
|
60
|
+
|
|
61
|
+
- Separate queries and mutations clearly
|
|
62
|
+
- Use input types for mutation arguments
|
|
63
|
+
- Implement connection-based pagination (Relay spec)
|
|
64
|
+
- Set query depth and complexity limits
|
|
65
|
+
- Disable introspection in production
|
|
66
|
+
- Use DataLoader to batch and deduplicate database queries
|
|
67
|
+
- Version via schema evolution, not URL versioning
|
|
68
|
+
|
|
69
|
+
## gRPC Design
|
|
70
|
+
|
|
71
|
+
- Define services and messages in `.proto` files
|
|
72
|
+
- Use streaming for large datasets or real-time updates
|
|
73
|
+
- Implement health checking service
|
|
74
|
+
- Set deadlines on all RPCs
|
|
75
|
+
- Use interceptors for auth, logging, and metrics
|
|
76
|
+
- Version via package namespacing
|
|
77
|
+
|
|
78
|
+
## Error Design
|
|
79
|
+
|
|
80
|
+
- Machine-readable error code (string enum, not just HTTP status)
|
|
81
|
+
- Human-readable message
|
|
82
|
+
- Field-level validation errors as an array
|
|
83
|
+
- Request ID for correlation
|
|
84
|
+
- Do not expose internal details (stack traces, SQL errors)
|
|
85
|
+
|
|
86
|
+
## Rate Limiting
|
|
87
|
+
|
|
88
|
+
- Per-user and per-IP limits
|
|
89
|
+
- Return `Retry-After` header with 429 responses
|
|
90
|
+
- Use sliding window or token bucket algorithm
|
|
91
|
+
- Higher limits for authenticated requests
|
|
92
|
+
- Separate limits for read and write operations
|
|
93
|
+
|
|
94
|
+
## Documentation
|
|
95
|
+
|
|
96
|
+
- OpenAPI 3.x spec for REST; keep it as the source of truth
|
|
97
|
+
- Auto-generate client SDKs from the spec
|
|
98
|
+
- Include request/response examples for every endpoint
|
|
99
|
+
- Document authentication requirements per endpoint
|
|
100
|
+
- Migration or upgrade note for every breaking change, reflected in the project's canonical docs surface
|