@oriro/orirocli 0.1.11 → 0.3.1

package/skills/craft/master-architect/references/ai-ml.md

@@ -1,317 +1,317 @@
-# AI/ML Engineering Reference
-
-## LLM INTEGRATION — DECISION TREE
-
-```
-Need to add AI to a product?
-  ↓
-What's the primary use case?
-  ├── Generate text/content        → Direct LLM call (OpenAI/Anthropic/Gemini)
-  ├── Answer questions about docs  → RAG pipeline
-  ├── Take actions autonomously    → Agent system
-  ├── Classify/structure data      → Fine-tuned model or structured output
-  ├── Search by meaning            → Vector similarity search
-  └── Train on proprietary data    → Fine-tuning or LoRA
-
-Speed vs Cost tradeoff:
-  Fastest + expensive:  GPT-4o, Claude Opus, Gemini Ultra
-  Balanced:             Claude Sonnet, GPT-4o-mini, Gemini Flash
-  Fastest + cheapest:   Claude Haiku, GPT-3.5-turbo, Gemini Flash 8B
-  Fully local (no cost): Ollama + Llama3/Mistral (needs GPU)
-```
-
----
-
-## RAG (RETRIEVAL-AUGMENTED GENERATION)
-
-The standard pattern for "AI that knows your data."
-
-### Architecture
-
-```
-[Documents] → [Chunker] → [Embedder] → [Vector DB]
-                                              ↓
-[User Query] → [Embedder] → [Vector Search] → [Top-K Chunks]
-                                              ↓
-                              [LLM] ← [Prompt + Chunks + Query]
-                                ↓
-                          [Answer + Citations]
-```
-
-### Chunking Strategy (Critical — Most Teams Get This Wrong)
-
-```
-Fixed size (bad):     500-token chunks with no semantic awareness
-Sentence (better):   Split on sentence boundaries, overlap 1-2 sentences
-Semantic (best):     Use embedding similarity to find natural break points
-Document-aware:       Respect headers, sections, tables — don't split mid-table
-
-Overlap: Always include 10-20% overlap between chunks to preserve context at boundaries.
-Chunk size: 256-512 tokens for factual Q&A, 512-1024 for summarization tasks.
-```
-
-### Embedding Models
-
-```
-OpenAI text-embedding-3-large:  Best quality (3072 dims), $0.00013/1K tokens
-OpenAI text-embedding-3-small:  Good quality (1536 dims), $0.00002/1K tokens — default choice
-Cohere embed-english-v3:        Competitive, supports int8 quantization (smaller storage)
-sentence-transformers (local):  all-MiniLM-L6-v2 — fast, free, 384 dims — good for prototyping
-BGE-M3:                         Best open-source multilingual model
-```
-
-### Vector Databases
-
-```
-Pinecone:       Managed, easiest ops, good at scale — $70+/mo
-Weaviate:       Open source + cloud, GraphQL API, good hybrid search
-Qdrant:         Open source, Rust-based (fast), great for self-hosted
-pgvector:       PostgreSQL extension — USE THIS FIRST if already on Postgres
-Chroma:         Simple, Python-native, great for prototyping
-Redis (Vector): If already on Redis, add vector search — no new infra
-Milvus:         Enterprise scale, self-hosted, complex ops
-
-Default choice: pgvector (zero new infra if on PostgreSQL)
-Scale choice:   Qdrant (self-hosted) or Pinecone (managed)
-```
-
-### RAG Quality Improvements
-
-```
-Hybrid search:     Combine vector similarity + BM25 keyword search. Almost always better.
-Reranking:         Cross-encoder reranker on top-K results (Cohere Rerank, BGE-reranker)
-Query expansion:   LLM rewrites query into 3 variants → union of results
-HyDE:              Generate hypothetical answer → embed it → use as search vector
-Metadata filters:  Always filter by tenant_id, date range, document type before vector search
-Self-querying:     LLM extracts metadata filters from natural language query
-```
-
----
-
-## AGENT SYSTEMS
-
-### When to Use Agents
-
-Agents are LLMs that can take actions (tool calls) in a loop until a goal is met. Use when:
-
-- Task requires multiple steps with conditional branching
-- System needs to call external APIs, read files, run queries
-- Requires human-in-the-loop approval at decision points
-- Single LLM call is insufficient to complete the task
-
-### Agent Patterns
-
-```
-ReAct (Reason + Act):     LLM reasons → selects tool → observes result → repeats
-                          Best for: general-purpose agents, debugging, research
-
-Plan-and-Execute:         Planner LLM creates steps → Executor LLM runs each step
-                          Best for: complex multi-step workflows, parallelizable tasks
-
-Reflection:               Agent checks its own output, critiques, revises
-                          Best for: code generation, writing, analysis
-
-Multi-Agent:              Specialized agents (Researcher, Coder, Critic, Orchestrator)
-                          Best for: complex pipelines, separation of concerns
-                          Risk: Harder to debug, latency compounds
-```
-
-### Tool Design (Critical)
-
-```python
-# Good tool design: atomic, single-responsibility, always returns structured data
-def search_transactions(
-    user_id: str,
-    date_from: str,
-    date_to: str,
-    amount_min: float | None = None
-) -> dict:
-    """
-    Search transactions for a user within a date range.
-    Returns: {"transactions": [...], "total": int, "error": str | None}
-    """
-    # Always return structured dict, never raw strings
-    # Always handle errors — agent needs to know what failed
-    # Always validate inputs before external calls
-```
-
-### Frameworks
-
-```
-LangChain:      Most popular, large ecosystem, can be overengineered — use carefully
-LlamaIndex:     Best for RAG and document Q&A pipelines
-LangGraph:      Graph-based agent workflows, better for complex state machines
-CrewAI:         Multi-agent coordination, role-based agents
-Anthropic Claude API: Function calling native — use directly for simpler agents
-OpenAI Assistants: Managed threads + tools, good for product integrations
-AutoGen:        Microsoft's multi-agent framework — good for code generation agents
-```
-
----
-
-## ML PIPELINES
-
-### Standard ML Pipeline Architecture
-
-```
-Data → Ingest → Validate → Feature Engineering → Train → Evaluate → Deploy → Monitor
-         ↓           ↓              ↓               ↓         ↓          ↓        ↓
-       Airbyte    Great Exp      dbt/Spark        MLflow    MLflow    BentoML  Evidently
-       Fivetran   Pandera        Feast (store)    W&B       W&B       Seldon   WhyLabs
-```
-
-### MLOps Stack (Production Grade)
-
-```
-Experiment tracking:  MLflow (open source) or W&B (better UX, $)
-Feature store:        Feast (open source) or Tecton (managed, $$$)
-Model registry:       MLflow Model Registry or W&B Artifacts
-Model serving:        BentoML (flexible) or Seldon Core (K8s) or Vertex AI
-Data validation:      Great Expectations or Pandera
-Drift monitoring:     Evidently AI or WhyLabs
-Pipeline orchestration: Airflow, Prefect, or Vertex AI Pipelines (if on GCP)
-```
-
-### Model Selection Framework
-
-```
-Classic ML (tabular data, interpretability needed):
-  └── Gradient boosting: XGBoost, LightGBM, CatBoost
-      Linear models: scikit-learn (fast, interpretable)
-
-Deep Learning (images, text, complex patterns):
-  └── PyTorch (research + production)
-      TensorFlow/Keras (GCP integration, Vertex AI)
-      JAX (cutting edge, Google research)
-
-LLM-powered (natural language tasks):
-  └── Use API (OpenAI, Anthropic, Cohere) — don't train from scratch
-      Fine-tune only if: domain-specific, latency-critical, cost-critical at scale
-      LoRA/QLoRA for efficient fine-tuning on consumer hardware
-
-Computer Vision:
-  └── Pretrained on ImageNet (ResNet, EfficientNet, ViT) → fine-tune
-      YOLO family for real-time object detection
-      SAM (Segment Anything) for segmentation
-```
-
----
-
-## PROMPT ENGINEERING (Production Quality)
-
-### System Prompt Structure
-
-```
-[ROLE + EXPERTISE]
-[TASK DEFINITION]
-[OUTPUT FORMAT — be explicit]
-[CONSTRAINTS — what NOT to do]
-[EXAMPLES — 2-3 few-shot examples for complex tasks]
-```
-
-### Techniques That Actually Work
-
-```
-Chain of Thought (CoT):    "Think step by step" → better reasoning on math/logic
-Few-shot:                   2-3 examples in prompt → significant quality boost
-XML tags:                   <output>, <reasoning>, <answer> → cleaner parsing
-Structured output:          Force JSON via function calling or response_format
-Temperature:               0-0.3 for factual/code, 0.7-1.0 for creative
-Constrained decoding:      Grammar-based sampling (outlines, guidance) for strict formats
-Self-consistency:          Sample N times, majority vote → more reliable answers
-```
-
-### Cost Optimization
-
-```
-Cache prompts:    Anthropic prompt caching (90% discount on cached tokens)
-Batch API:        OpenAI/Anthropic batch endpoints — 50% cheaper, async
-Model routing:    Haiku/3.5-mini for simple tasks, Opus/4 for complex
-Prompt compression: Remove redundant whitespace, compress examples
-Context pruning:  Don't send full conversation history — summarize older turns
-```
-
----
-
-## AI EVALUATION FRAMEWORK
-
-Never ship AI features without evals. This is the new unit testing.
-
-### Eval Types
-
-```
-Functional:     Does it do the task? (factual accuracy, task completion rate)
-Safety:         Does it avoid harmful outputs? (refusal rate on red-team prompts)
-Quality:        How good is it? (human preference, rubric-based scoring)
-Regression:     Did the latest change make things worse? (track over time)
-Latency:        p50/p95 response time under load
-Cost:           Average tokens/request × price/token
-```
-
-### Tooling
-
-```
-Promptfoo:      Open source eval framework — run evals in CI
-RAGAS:          RAG-specific eval (faithfulness, relevance, recall)
-LangSmith:      LangChain's eval + tracing platform
-Braintrust:     Modern eval platform, good UX
-Weights & Biases: Eval + experiment tracking combo
-Custom evals:   For domain-specific tasks — always build at least 50 golden examples
-```
-
-### Golden Dataset Rule
-
-Before shipping any AI feature: build 50-100 "golden" input/output pairs that represent the expected behavior. Every model or prompt change is tested against this set. If quality drops on >10% of cases, don't ship.
-
----
-
-## VECTOR SEARCH PATTERNS (pgvector — Production)
-
-```sql
--- Setup (PostgreSQL)
-CREATE EXTENSION IF NOT EXISTS vector;
-ALTER TABLE documents ADD COLUMN embedding vector(1536);
-CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
-
--- Hybrid search (vector + keyword) — almost always better than pure vector
-WITH vector_search AS (
-  SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
-  FROM documents
-  WHERE tenant_id = $2  -- ALWAYS filter by tenant first
-  ORDER BY embedding <=> $1::vector
-  LIMIT 20
-),
-keyword_search AS (
-  SELECT id, content, ts_rank(to_tsvector('english', content), query) AS rank
-  FROM documents, to_tsquery('english', $3) query
-  WHERE tenant_id = $2 AND to_tsvector('english', content) @@ query
-  LIMIT 20
-)
-SELECT DISTINCT ON (id) id, content,
-  COALESCE(v.similarity, 0) * 0.7 + COALESCE(k.rank, 0) * 0.3 AS score
-FROM vector_search v
-FULL OUTER JOIN keyword_search k USING (id)
-ORDER BY id, score DESC
-LIMIT 10;
-```
-
----
-
-## AI SECURITY
-
-```
-Prompt injection:     Never concatenate user input directly into system prompts
-                      Use: separate user_message from system_message at API level
-
-Data leakage:         Validate LLM output doesn't contain other users' data
-                      Especially in RAG: filter metadata BEFORE vector search
-
-Model abuse:          Rate limit AI endpoints aggressively (10x stricter than normal)
-                      Log all AI requests with user_id for audit trail
-
-Jailbreaks:           Use Anthropic Constitutional AI or OpenAI moderation API
-                      Layer your own content classifier for sensitive domains
-
-API key exposure:     AI API keys are high-value targets — rotate monthly
-                      Separate keys per environment, never share between tenants
-```
+# AI/ML Engineering Reference
+
+## LLM INTEGRATION — DECISION TREE
+
+```
+Need to add AI to a product?
+  ↓
+What's the primary use case?
+  ├── Generate text/content        → Direct LLM call (OpenAI/Anthropic/Gemini)
+  ├── Answer questions about docs  → RAG pipeline
+  ├── Take actions autonomously    → Agent system
+  ├── Classify/structure data      → Fine-tuned model or structured output
+  ├── Search by meaning            → Vector similarity search
+  └── Train on proprietary data    → Fine-tuning or LoRA
+
+Speed vs Cost tradeoff:
+  Fastest + expensive:  GPT-4o, Claude Opus, Gemini Ultra
+  Balanced:             Claude Sonnet, GPT-4o-mini, Gemini Flash
+  Fastest + cheapest:   Claude Haiku, GPT-3.5-turbo, Gemini Flash 8B
+  Fully local (no cost): Ollama + Llama3/Mistral (needs GPU)
+```
+
+---
+
+## RAG (RETRIEVAL-AUGMENTED GENERATION)
+
+The standard pattern for "AI that knows your data."
+
+### Architecture
+
+```
+[Documents] → [Chunker] → [Embedder] → [Vector DB]
+                                              ↓
+[User Query] → [Embedder] → [Vector Search] → [Top-K Chunks]
+                                              ↓
+                              [LLM] ← [Prompt + Chunks + Query]
+                                ↓
+                          [Answer + Citations]
+```
+
+### Chunking Strategy (Critical — Most Teams Get This Wrong)
+
+```
+Fixed size (bad):     500-token chunks with no semantic awareness
+Sentence (better):   Split on sentence boundaries, overlap 1-2 sentences
+Semantic (best):     Use embedding similarity to find natural break points
+Document-aware:       Respect headers, sections, tables — don't split mid-table
+
+Overlap: Always include 10-20% overlap between chunks to preserve context at boundaries.
+Chunk size: 256-512 tokens for factual Q&A, 512-1024 for summarization tasks.
+```
+
+### Embedding Models
+
+```
+OpenAI text-embedding-3-large:  Best quality (3072 dims), $0.00013/1K tokens
+OpenAI text-embedding-3-small:  Good quality (1536 dims), $0.00002/1K tokens — default choice
+Cohere embed-english-v3:        Competitive, supports int8 quantization (smaller storage)
+sentence-transformers (local):  all-MiniLM-L6-v2 — fast, free, 384 dims — good for prototyping
+BGE-M3:                         Best open-source multilingual model
+```
+
+### Vector Databases
+
+```
+Pinecone:       Managed, easiest ops, good at scale — $70+/mo
+Weaviate:       Open source + cloud, GraphQL API, good hybrid search
+Qdrant:         Open source, Rust-based (fast), great for self-hosted
+pgvector:       PostgreSQL extension — USE THIS FIRST if already on Postgres
+Chroma:         Simple, Python-native, great for prototyping
+Redis (Vector): If already on Redis, add vector search — no new infra
+Milvus:         Enterprise scale, self-hosted, complex ops
+
+Default choice: pgvector (zero new infra if on PostgreSQL)
+Scale choice:   Qdrant (self-hosted) or Pinecone (managed)
+```
+
+### RAG Quality Improvements
+
+```
+Hybrid search:     Combine vector similarity + BM25 keyword search. Almost always better.
+Reranking:         Cross-encoder reranker on top-K results (Cohere Rerank, BGE-reranker)
+Query expansion:   LLM rewrites query into 3 variants → union of results
+HyDE:              Generate hypothetical answer → embed it → use as search vector
+Metadata filters:  Always filter by tenant_id, date range, document type before vector search
+Self-querying:     LLM extracts metadata filters from natural language query
+```
+
+---
+
+## AGENT SYSTEMS
+
+### When to Use Agents
+
+Agents are LLMs that can take actions (tool calls) in a loop until a goal is met. Use when:
+
+- Task requires multiple steps with conditional branching
+- System needs to call external APIs, read files, run queries
+- Requires human-in-the-loop approval at decision points
+- Single LLM call is insufficient to complete the task
+
+### Agent Patterns
+
+```
+ReAct (Reason + Act):     LLM reasons → selects tool → observes result → repeats
+                          Best for: general-purpose agents, debugging, research
+
+Plan-and-Execute:         Planner LLM creates steps → Executor LLM runs each step
+                          Best for: complex multi-step workflows, parallelizable tasks
+
+Reflection:               Agent checks its own output, critiques, revises
+                          Best for: code generation, writing, analysis
+
+Multi-Agent:              Specialized agents (Researcher, Coder, Critic, Orchestrator)
+                          Best for: complex pipelines, separation of concerns
+                          Risk: Harder to debug, latency compounds
+```
+
+### Tool Design (Critical)
+
+```python
+# Good tool design: atomic, single-responsibility, always returns structured data
+def search_transactions(
+    user_id: str,
+    date_from: str,
+    date_to: str,
+    amount_min: float | None = None
+) -> dict:
+    """
+    Search transactions for a user within a date range.
+    Returns: {"transactions": [...], "total": int, "error": str | None}
+    """
+    # Always return structured dict, never raw strings
+    # Always handle errors — agent needs to know what failed
+    # Always validate inputs before external calls
+```
+
+### Frameworks
+
+```
+LangChain:      Most popular, large ecosystem, can be overengineered — use carefully
+LlamaIndex:     Best for RAG and document Q&A pipelines
+LangGraph:      Graph-based agent workflows, better for complex state machines
+CrewAI:         Multi-agent coordination, role-based agents
+Anthropic Claude API: Function calling native — use directly for simpler agents
+OpenAI Assistants: Managed threads + tools, good for product integrations
+AutoGen:        Microsoft's multi-agent framework — good for code generation agents
+```
+
+---
+
+## ML PIPELINES
+
+### Standard ML Pipeline Architecture
+
+```
+Data → Ingest → Validate → Feature Engineering → Train → Evaluate → Deploy → Monitor
+         ↓           ↓              ↓               ↓         ↓          ↓        ↓
+       Airbyte    Great Exp      dbt/Spark        MLflow    MLflow    BentoML  Evidently
+       Fivetran   Pandera        Feast (store)    W&B       W&B       Seldon   WhyLabs
+```
+
+### MLOps Stack (Production Grade)
+
+```
+Experiment tracking:  MLflow (open source) or W&B (better UX, $)
+Feature store:        Feast (open source) or Tecton (managed, $$$)
+Model registry:       MLflow Model Registry or W&B Artifacts
+Model serving:        BentoML (flexible) or Seldon Core (K8s) or Vertex AI
+Data validation:      Great Expectations or Pandera
+Drift monitoring:     Evidently AI or WhyLabs
+Pipeline orchestration: Airflow, Prefect, or Vertex AI Pipelines (if on GCP)
+```
+
+### Model Selection Framework
+
+```
+Classic ML (tabular data, interpretability needed):
+  └── Gradient boosting: XGBoost, LightGBM, CatBoost
+      Linear models: scikit-learn (fast, interpretable)
+
+Deep Learning (images, text, complex patterns):
+  └── PyTorch (research + production)
+      TensorFlow/Keras (GCP integration, Vertex AI)
+      JAX (cutting edge, Google research)
+
+LLM-powered (natural language tasks):
+  └── Use API (OpenAI, Anthropic, Cohere) — don't train from scratch
+      Fine-tune only if: domain-specific, latency-critical, cost-critical at scale
+      LoRA/QLoRA for efficient fine-tuning on consumer hardware
+
+Computer Vision:
+  └── Pretrained on ImageNet (ResNet, EfficientNet, ViT) → fine-tune
+      YOLO family for real-time object detection
+      SAM (Segment Anything) for segmentation
+```
+
+---
+
+## PROMPT ENGINEERING (Production Quality)
+
+### System Prompt Structure
+
+```
+[ROLE + EXPERTISE]
+[TASK DEFINITION]
+[OUTPUT FORMAT — be explicit]
+[CONSTRAINTS — what NOT to do]
+[EXAMPLES — 2-3 few-shot examples for complex tasks]
+```
+
+### Techniques That Actually Work
+
+```
+Chain of Thought (CoT):    "Think step by step" → better reasoning on math/logic
+Few-shot:                   2-3 examples in prompt → significant quality boost
+XML tags:                   <output>, <reasoning>, <answer> → cleaner parsing
+Structured output:          Force JSON via function calling or response_format
+Temperature:               0-0.3 for factual/code, 0.7-1.0 for creative
+Constrained decoding:      Grammar-based sampling (outlines, guidance) for strict formats
+Self-consistency:          Sample N times, majority vote → more reliable answers
+```
+
+### Cost Optimization
+
+```
+Cache prompts:    Anthropic prompt caching (90% discount on cached tokens)
+Batch API:        OpenAI/Anthropic batch endpoints — 50% cheaper, async
+Model routing:    Haiku/3.5-mini for simple tasks, Opus/4 for complex
+Prompt compression: Remove redundant whitespace, compress examples
+Context pruning:  Don't send full conversation history — summarize older turns
+```
+
+---
+
+## AI EVALUATION FRAMEWORK
+
+Never ship AI features without evals. This is the new unit testing.
+
+### Eval Types
+
+```
+Functional:     Does it do the task? (factual accuracy, task completion rate)
+Safety:         Does it avoid harmful outputs? (refusal rate on red-team prompts)
+Quality:        How good is it? (human preference, rubric-based scoring)
+Regression:     Did the latest change make things worse? (track over time)
+Latency:        p50/p95 response time under load
+Cost:           Average tokens/request × price/token
+```
+
+### Tooling
+
+```
+Promptfoo:      Open source eval framework — run evals in CI
+RAGAS:          RAG-specific eval (faithfulness, relevance, recall)
+LangSmith:      LangChain's eval + tracing platform
+Braintrust:     Modern eval platform, good UX
+Weights & Biases: Eval + experiment tracking combo
+Custom evals:   For domain-specific tasks — always build at least 50 golden examples
+```
+
+### Golden Dataset Rule
+
+Before shipping any AI feature: build 50-100 "golden" input/output pairs that represent the expected behavior. Every model or prompt change is tested against this set. If quality drops on >10% of cases, don't ship.
+
+---
+
+## VECTOR SEARCH PATTERNS (pgvector — Production)
+
+```sql
+-- Setup (PostgreSQL)
+CREATE EXTENSION IF NOT EXISTS vector;
+ALTER TABLE documents ADD COLUMN embedding vector(1536);
+CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
+
+-- Hybrid search (vector + keyword) — almost always better than pure vector
+WITH vector_search AS (
+  SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
+  FROM documents
+  WHERE tenant_id = $2  -- ALWAYS filter by tenant first
+  ORDER BY embedding <=> $1::vector
+  LIMIT 20
+),
+keyword_search AS (
+  SELECT id, content, ts_rank(to_tsvector('english', content), query) AS rank
+  FROM documents, to_tsquery('english', $3) query
+  WHERE tenant_id = $2 AND to_tsvector('english', content) @@ query
+  LIMIT 20
+)
+SELECT DISTINCT ON (id) id, content,
+  COALESCE(v.similarity, 0) * 0.7 + COALESCE(k.rank, 0) * 0.3 AS score
+FROM vector_search v
+FULL OUTER JOIN keyword_search k USING (id)
+ORDER BY id, score DESC
+LIMIT 10;
+```
+
+---
+
+## AI SECURITY
+
+```
+Prompt injection:     Never concatenate user input directly into system prompts
+                      Use: separate user_message from system_message at API level
+
+Data leakage:         Validate LLM output doesn't contain other users' data
+                      Especially in RAG: filter metadata BEFORE vector search
+
+Model abuse:          Rate limit AI endpoints aggressively (10x stricter than normal)
+                      Log all AI requests with user_id for audit trail
+
+Jailbreaks:           Use Anthropic Constitutional AI or OpenAI moderation API
+                      Layer your own content classifier for sensitive domains
+
+API key exposure:     AI API keys are high-value targets — rotate monthly
+                      Separate keys per environment, never share between tenants
+```