onto-mcp 0.3.0 → 0.3.2

package/.onto/domains/llm-native-development/extension_cases.md

@@ -1,474 +0,0 @@
----
-version: 2
-last_updated: "2026-03-30"
-source: manual
-status: established
----
-
-# LLM-Native Development Domain — Extension Scenarios
-
-The evolution agent simulates each scenario to verify whether the existing structure for LLM-powered systems breaks under real-world changes.
-
-**Area key**: 1=Model Integration, 2=Prompt & Context Design, 3=Retrieval & Knowledge Systems, 4=Agentic Systems, 5=Evaluation & Testing, 6=Safety & Alignment, 7=Production Operations, 8=Data & Model Adaptation
-
----
-
-## Case 1: New Model Generation
-
-### Situation
-
-A new foundation model (GPT-5, Claude 4, Gemini 2) introduces expanded capabilities — larger context, new modalities, improved reasoning, new API features. The system must evaluate, integrate, and potentially restructure without disrupting production.
-
-### Case Study: Claude 3 → Sonnet 4 Migration
-
-Each Anthropic model release changed instruction-following behavior, tool calling format, and system prompt handling. Teams pinned to `claude-3-sonnet-20240229` faced migration requiring: re-evaluation of all prompt templates, restructured tool schemas, and full evaluation suite re-runs. Teams using `-latest` endpoints experienced silent production behavior changes — output format shifts, changed refusal patterns, altered verbosity.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 1 (Model) | **Primary** | Selection criteria, API integration, version pinning, routing logic |
-| 2 (Prompt) | **High** | System prompt restructuring, token budget shift, structured output |
-| 5 (Evaluation) | **High** | Full suite re-run, benchmark recalibration |
-| 7 (Operations) | **High** | Canary deployment, cost/latency changes, monitoring thresholds |
-| 3, 4, 6, 8 | Low–Medium | Chunk size (3), tool formats (4), refusal patterns (6), fine-tune transfer (8) |
-
-### Verification Checklist
-
-- [ ] Model version pinned → dependency_rules.md §Model API Dependencies
-- [ ] Migration plan: affected components → evaluate replacement → run eval → canary deploy
-- [ ] All prompt templates tested against new model → domain_scope.md Area 2
-- [ ] Evaluation suite run; results compared to baseline → domain_scope.md Area 5
-- [ ] Cost/latency impact analyzed → domain_scope.md Area 7
-- [ ] Rollback plan defined → dependency_rules.md §Model API Dependencies
-- [ ] Safety guardrails verified against new behavior → domain_scope.md Area 6
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| dependency_rules.md | Modify | §Model API Dependencies |
-| domain_scope.md | Verify | §Technology Assumptions |
-| structure_spec.md | Verify | Token budget rules |
-
----
-
-## Case 2: New Retrieval Paradigm
-
-### Situation
-
-A new retrieval architecture (graph RAG, multi-modal RAG, hybrid search) changes how external knowledge is stored, indexed, and retrieved for LLM consumption.
-
-### Case Study: LlamaIndex Knowledge Graph Integration
-
-LlamaIndex added knowledge graph-based retrieval alongside vector search. A healthcare company migrating found: chunking strategy became irrelevant for structured data (entities replaced chunks), embedding model selection became secondary to entity extraction quality, and retrieval evaluation changed from "relevance@k" to "path accuracy." The hybrid approach required a routing layer deciding which backend to query based on query type.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 3 (Retrieval) | **Primary** | Chunking, indexing, retrieval algorithms, storage backend all restructured |
-| 5 (Evaluation) | **High** | Retrieval metrics change; new golden sets needed |
-| 2 (Prompt) | **Medium** | Retrieved content format changes (triples vs. chunks) |
-| 7 (Operations) | **Medium** | New infrastructure (graph DB); different latency/cost profile |
-| 1, 4, 6, 8 | Low | Model interface unchanged (1), tool interface changes (4) |
-
-### Verification Checklist
-
-- [ ] Handoff point with Area 2 preserved: retrieval results = Area 3 boundary → domain_scope.md §Handoff point
-- [ ] Embedding migration plan if changing backend → dependency_rules.md §Embedding Model Dependencies
-- [ ] Retrieval evaluation metrics updated → domain_scope.md Area 5
-- [ ] Hybrid search strategy defined → domain_scope.md Area 3
-- [ ] LLM-favored structure patterns adapted → domain_scope.md Area 3
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Verify | Area 3, §Handoff point with Area 2 |
-| dependency_rules.md | Modify | §Embedding Model Dependencies, §Runtime Data Flow |
-| concepts.md | Modify | New terms: knowledge graph retrieval, graph RAG |
-
----
-
-## Case 3: New Agent Protocol
-
-### Situation
-
-A new standardized protocol for agent-tool interaction (MCP successor, A2A maturation) replaces or supplements existing integration patterns.
-
-### Case Study: MCP — 0 to 97M npm Downloads in 1 Year
-
-MCP standardized how agents discover and invoke tools via JSON-RPC. Before MCP, each framework had proprietary tool patterns. Migration required: tool schema restructuring to JSON Schema-based definitions, stateful server connections replacing stateless calls, connection lifecycle management, and a bridge layer for legacy APIs during transition. A2A emerged as complementary standard for inter-agent communication.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 4 (Agentic) | **Primary** | Tool integration restructured; server/client design; discovery mechanisms |
-| 6 (Safety) | **High** | New injection surface; tool permission models change |
-| 7 (Operations) | **Medium** | Server availability monitoring; connection lifecycle; new failure modes |
-| 1, 2, 3, 5, 8 | Low | Tool-calling format (1), tool descriptions (2), MCP resources (3) |
-
-### Verification Checklist
-
-- [ ] MCP server dependencies documented with fallback behavior → dependency_rules.md §MCP Server Dependencies
-- [ ] Schema validation at connection time → dependency_rules.md §MCP Server Dependencies
-- [ ] Tool permission model defined → domain_scope.md Area 6
-- [ ] Backward compatibility plan for existing integrations documented
-- [ ] Protocol-specific monitoring added → domain_scope.md Area 7
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| dependency_rules.md | Modify | §MCP Server Dependencies, §Framework Dependencies |
-| domain_scope.md | Verify | Area 4, §Reference Standards/Frameworks |
-| concepts.md | Modify | Protocol terms: MCP, A2A, ACI |
-
----
-
-## Case 4: AI Regulation Compliance
-
-### Situation
-
-New AI regulations (EU AI Act, etc.) impose compliance requirements: audit logging, risk classification, transparency, human oversight.
-
-### Case Study: EU AI Act Risk Classification
-
-An LLM recruitment screening tool was classified "high-risk" (Annex III), requiring: risk management with continuous monitoring, technical documentation including training data provenance, human oversight (no autonomous hiring decisions), automatic decision logging, and documented evaluation methodology. Systems designed with safety and observability from the start required minimal changes; those treating these as afterthoughts required extensive refactoring.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 6 (Safety) | **Primary** | Risk classification, content policy, PII handling, regulatory compliance |
-| 7 (Operations) | **High** | Mandatory audit logging, compliance monitoring, incident reporting |
-| 5 (Evaluation) | **High** | Auditable evaluation methodology required |
-| 4 (Agentic) | **High** | Human oversight constrains agent autonomy |
-| 1, 2, 3, 8 | Low–Medium | Provider certifications (1), disclosure instructions (2), provenance (3, 8) |
-
-### Verification Checklist
-
-- [ ] System risk classification determined → domain_scope.md Area 6
-- [ ] Audit logging covers all LLM interactions → domain_scope.md Area 7
-- [ ] Human oversight mechanism for high-risk decisions → domain_scope.md Area 6
-- [ ] Evaluation methodology documented and reproducible → domain_scope.md Area 5
-- [ ] PII detection and redaction operational → domain_scope.md Area 6
-- [ ] All 8 areas audited against regulatory requirements
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Verify | Area 6, Area 7 |
-| dependency_rules.md | Modify | §Truth Source Hierarchy (regulatory priority) |
-| concepts.md | Modify | Regulatory terms: risk classification, audit trail |
-
----
-
-## Case 5: Context Window Expansion
-
-### Situation
-
-Context windows increase by 10x (200K → 2M → 10M tokens), changing fundamental trade-offs. Techniques designed for limited context (chunking, summarization, multi-step retrieval) may become unnecessary.
-
-### Case Study: Gemini 1M Context — Is RAG Still Necessary?
-
-Gemini 1.5 Pro launched with 1M tokens (February 2024). Results showed: "lost in the middle" effects made retrieval still valuable for precision; 1M tokens at $3.50/M made naive "stuff everything" 100x more expensive than targeted retrieval; latency scaled with context length. RAG shifted from "mandatory" to "optimization for cost, latency, and precision" — conditional rather than required.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 2 (Prompt) | **Primary** | Token budget restructured; context rot amplified; utilization strategy changed |
-| 3 (Retrieval) | **High** | Chunking relevance decreases; RAG becomes conditional |
-| 7 (Operations) | **High** | Cost per request increases dramatically; caching for larger payloads |
-| 1 (Model) | **Medium** | Context window as selection differentiator |
-| 4, 5, 6, 8 | Low–Medium | Longer agent history (4), long-context eval (5), larger injection surface (6) |
-
-### Verification Checklist
-
-- [ ] Technology assumption updated → domain_scope.md §Technology Assumptions
-- [ ] Token budget strategy revised → structure_spec.md
-- [ ] RAG necessity re-evaluated per use case → domain_scope.md Area 3
-- [ ] Cost analysis: long context vs. retrieval → domain_scope.md Area 7
-- [ ] "Lost in the middle" effects tested → domain_scope.md Area 5
-- [ ] Quantitative criteria re-evaluated → structure_spec.md
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Modify | §Technology Assumptions, Area 2, Area 3 |
-| structure_spec.md | Modify | Token budget rules, quantitative criteria |
-| dependency_rules.md | Verify | §Design-Time Constraint Flow |
-
----
-
-## Case 6: Model-as-a-Service Disruption
-
-### Situation
-
-Deployment shifts from cloud API to on-premise/edge with open-weight models, changing infrastructure, cost structures, and operational responsibilities.
-
-### Case Study: Llama Open-Weight Local Deployment
-
-A financial services company migrated from OpenAI API to self-hosted Llama 3 70B for data residency. Infrastructure responsibility shifted entirely (GPU procurement, model serving via vLLM/TGI, auto-scaling). Cost structure inverted: no per-token costs but $30K+/month GPU cluster. Safety guardrails (content filtering, PII redaction) had to be built from scratch — API providers include these by default. The domain_scope.md assumption "Models accessed via API" was invalidated.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 1 (Model) | **Primary** | Model serving replaces API; quantization; batching; version self-management |
-| 7 (Operations) | **Primary** | GPU infrastructure, serving, scaling, hardware monitoring — new domain |
-| 6 (Safety) | **High** | All guardrails built in-house; no provider-side filtering |
-| 8 (Adaptation) | **High** | Direct model access enables fine-tuning |
-| 2, 3, 4, 5 | Low–Medium | Chat template differences (2), self-hosted embeddings (3), tool support gaps (4) |
-
-### Verification Checklist
-
-- [ ] Technology assumption invalidation documented → domain_scope.md §Technology Assumptions
-- [ ] Area 7 expanded for infrastructure management → domain_scope.md Area 7
-- [ ] Model serving infrastructure documented → domain_scope.md Area 1
-- [ ] Safety guardrails built for self-hosted model → domain_scope.md Area 6
-- [ ] Cost model updated: fixed vs. per-token → domain_scope.md Area 7
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Modify | §Technology Assumptions, Area 1, Area 7 |
-| dependency_rules.md | Modify | §Model API Dependencies (extend to self-hosted) |
-| concepts.md | Modify | New terms: model serving, quantization, vLLM |
-
----
-
-## Case 7: Multi-Modal Native
-
-### Situation
-
-Vision, audio, and other modalities become first-class inputs alongside text, rather than supplementary capabilities.
-
-### Case Study: GPT-4V and Claude 3 Vision
-
-A document processing company migrated from OCR-then-LLM to direct image-to-text: the OCR pipeline was eliminated, prompts referenced visual elements requiring spatial reasoning instructions, knowledge bases needed multi-modal embeddings (CLIP) incompatible with text-only embeddings, evaluation required annotated image golden sets, and token costs increased (1,000-2,000 tokens per page image vs. 200-400 for text). The domain_scope.md assumption "Text is primary modality" was challenged.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 2 (Prompt) | **Primary** | Multi-modal input design; spatial/temporal references; token budget for images |
-| 3 (Retrieval) | **High** | Multi-modal embeddings; cross-modal search; non-text storage |
-| 1 (Model) | **High** | Modality support as selection criterion; multi-modal routing |
-| 5 (Evaluation) | **High** | Visual understanding metrics; multi-modal golden sets |
-| 4, 6, 7, 8 | Low–Medium | Vision for agents (4), image injection (6), larger payloads (7) |
-
-### Verification Checklist
-
-- [ ] Technology assumption updated → domain_scope.md §Technology Assumptions
-- [ ] Multi-modal input design patterns documented → domain_scope.md Area 2
-- [ ] Multi-modal embedding model selected → dependency_rules.md §Embedding Model Dependencies
-- [ ] Cross-modal search assessed → domain_scope.md Area 3
-- [ ] Multi-modal evaluation metrics defined → domain_scope.md Area 5
-- [ ] Token budget includes image/audio costs → structure_spec.md
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Modify | §Technology Assumptions, Area 2, Area 3 |
-| dependency_rules.md | Modify | §Embedding Model Dependencies |
-| concepts.md | Modify | New terms: multi-modal embedding, cross-modal search |
-
----
-
-## Case 8: Fine-Tuning Democratization
-
-### Situation
-
-Fine-tuning costs drop 100x through parameter-efficient techniques, making model adaptation a default practice. Area 8 shifts from optional to mandatory.
-
-### Case Study: LoRA/QLoRA Cost Reduction
-
-A startup fine-tuned Llama 2 7B with QLoRA on 10K customer support conversations in 4 hours on a single A100 ($12). The fine-tuned model outperformed GPT-4 with elaborate prompts (92% vs. 87% accuracy). Prompt engineering effort dropped 70% — domain knowledge was in weights, not context. New capabilities needed: dataset curation, experiment tracking, model evaluation before/after, adapter serving. The domain_scope.md assumption "Fine-tuning is optional" was invalidated.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 8 (Adaptation) | **Primary** | Becomes mandatory: dataset engineering, training, experiment tracking |
-| 5 (Evaluation) | **High** | Pre/post comparison; contamination checks; regression testing |
-| 6 (Safety) | **High** | Fine-tuning can remove guardrails; alignment verification post-training |
-| 1 (Model) | **High** | Adapter serving; model routing includes fine-tuned variants |
-| 2, 3, 4, 7 | Low–Medium | Shorter prompts (2), reduced retrieval needs (3), A/B testing (7) |
-
-### Verification Checklist
-
-- [ ] Technology assumption invalidation documented → domain_scope.md §Technology Assumptions
-- [ ] Area 8 elevated to required → domain_scope.md Area 8
-- [ ] Dataset engineering pipeline documented → domain_scope.md Area 8
-- [ ] Pre/post fine-tuning evaluation → domain_scope.md Area 5
-- [ ] Safety alignment verified after training → domain_scope.md Area 6
-- [ ] Adapter serving infrastructure → domain_scope.md Area 1, Area 7
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Modify | §Technology Assumptions, Area 8 |
-| dependency_rules.md | Modify | §Feedback Loops (Loop 1 becomes critical path) |
-| concepts.md | Modify | New terms: LoRA, QLoRA, adapter, PEFT |
-
----
-
-## Case 9: Agent Autonomy Expansion
-
-### Situation
-
-Agents gain direct computer interface capabilities — browser control, desktop applications, file systems — extending autonomy beyond structured tool calls to open-ended environment interaction.
-
-### Case Study: Claude Computer Use and Browser Use (78K Stars)
-
-A QA company adopted browser use agents: architecture shifted from predefined tool calls to observe-act loops (screenshot → reason → mouse/keyboard action). Safety boundaries became critical — agents could navigate anywhere, fill any form. Novel failure modes: loops, unintended navigation, interaction with pop-ups. Evaluation shifted from step-based ("correct tool call?") to outcome-based ("goal achieved through any path?").
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 4 (Agentic) | **Primary** | Observe-act loops; browser/computer control; environment interaction |
-| 6 (Safety) | **Primary** | Sandboxing; action permissions; escalation for irreversible actions |
-| 5 (Evaluation) | **High** | Outcome-based evaluation; trajectory analysis; reliability metrics |
-| 7 (Operations) | **High** | Sandbox infrastructure; session recording; resource limits |
-| 1, 2, 3, 8 | Low–Medium | Vision required (1), boundary instructions (2) |
-
-### Verification Checklist
-
-- [ ] Observe-act pattern documented → domain_scope.md Area 4
-- [ ] Sandbox environment defined and enforced → domain_scope.md Area 6
-- [ ] Action permission model defined → domain_scope.md Area 6
-- [ ] Escalation policy for irreversible actions → domain_scope.md Area 6
-- [ ] Outcome-based evaluation metrics → domain_scope.md Area 5
-- [ ] Stuck/loop detection and termination → domain_scope.md Area 4
-- [ ] Resource limits (session duration, allowed domains) → domain_scope.md Area 7
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Verify | Area 4, Area 6 |
-| dependency_rules.md | Verify | §Design-Time Constraint Flow (safety → agent) |
-| concepts.md | Modify | New terms: observe-act loop, action permission model |
-
----
-
-## Case 10: Evaluation Paradigm Shift
-
-### Situation
-
-AI-as-judge (LLM-based evaluation) becomes the primary evaluation method, replacing human annotation and rule-based metrics at scale.
-
-### Case Study: LLM-as-Judge Replacing Human Annotation
-
-A customer support company migrated from human evaluation (500/week at $15 each = $7,500) to LLM-as-judge (50,000/week at $0.03 each = $1,500). Consistency improved, but systematic biases emerged (preference for longer responses, over-rating confident incorrect answers). Meta-evaluation became necessary — "evaluating the evaluator." Self-evaluation bias required separate judge and generation models. The methodology shifted from "collecting human judgments" to "calibrating automated judges."
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 5 (Evaluation) | **Primary** | AI-judge pipelines, meta-evaluation, judge calibration, bias detection |
-| 1 (Model) | **Medium** | Separate judge model needed; judge selection criteria differ |
-| 2 (Prompt) | **Medium** | Judge prompt design: rubrics, criteria, output format |
-| 7 (Operations) | **Medium** | Evaluation pipeline costs; judge model drift monitoring |
-| 3, 4, 6, 8 | Low–Medium | Factual verification (3), safety eval (6), training labels (8) |
-
-### Verification Checklist
-
-- [ ] AI-as-judge methodology documented → domain_scope.md Area 5
-- [ ] Judge model ≠ generation model enforced → domain_scope.md Area 5
-- [ ] Meta-evaluation: judge validated against human golden set → domain_scope.md Area 5
-- [ ] Judge bias detection and calibration procedure defined
-- [ ] Human evaluation preserved for calibration maintenance → domain_scope.md Area 5
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| domain_scope.md | Verify | Area 5 |
-| dependency_rules.md | Verify | §Feedback Loops (Loop 1), §Metric Ownership Rules |
-| concepts.md | Modify | New terms: AI-as-judge, meta-evaluation, judge calibration |
-
----
-
-## Case 11: LLM-Favored Structure Adoption
-
-### Situation
-
-Organization-wide adoption of LLM-optimized knowledge patterns — File=Concept, YAML frontmatter, system maps, llms.txt — as the documentation standard.
-
-### Case Study: Onto's Structure and llms.txt Adoption
-
-A 200-person engineering org migrated from Confluence wiki (3,000+ nested, duplicated pages) to File=Concept structure (1,200 files with clear ownership). YAML frontmatter enabled automated knowledge graph generation. LLMs consumed documentation 40% more accurately (RAG retrieval precision). Migration cost: 6 engineer-months. The llms.txt standard, CLAUDE.md, .cursorrules, and AGENTS.md demonstrate vendor-specific implementations of the same principle: structured, machine-readable knowledge surfaces.
-
-### Impact Analysis
-
-| Area | Impact Level | Key Concern |
-|---|---|---|
-| 3 (Retrieval) | **Primary** | File=Concept; metadata-driven retrieval; navigation paths; system maps |
-| 2 (Prompt) | **Medium** | Frontmatter provides context; reduces prompt engineering |
-| 4 (Agentic) | **Medium** | CLAUDE.md/AGENTS.md define per-directory agent behavior |
-| 5 (Evaluation) | **Medium** | Structure quality measurable: orphan rate, duplication, completeness |
-| 1, 6, 7, 8 | Low | Indirect benefits across all areas |
-
-### Verification Checklist
-
-- [ ] File=Concept: one concept per file → domain_scope.md Area 3
-- [ ] YAML frontmatter on all knowledge files → structure_spec.md
-- [ ] System map provides navigation → domain_scope.md Area 3
-- [ ] Duplication eliminated → dependency_rules.md §Duplication Prevention Rules
-- [ ] Navigation paths acyclic → dependency_rules.md §Acyclicity
-- [ ] Retrieval precision measured before/after → domain_scope.md Area 5
-- [ ] Structure validation automated (lint, orphan detection) → domain_scope.md Area 7
-
-### Affected Files
-
-| File | Impact | Section |
-|---|---|---|
-| structure_spec.md | Modify | Frontmatter spec, File=Concept rules |
-| dependency_rules.md | Verify | §Duplication Prevention, §Acyclicity, §Referential Integrity |
-| domain_scope.md | Verify | Area 3, §Reference Standards/Frameworks |
-
----
-
-## Scenario Interconnections
-
-| Scenario | Triggers / Interacts With | Reason |
-|---|---|---|
-| Case 1 (New Model) | → Case 5 (Context), Case 7 (Multi-Modal), Case 10 (Eval) | New models expand context, add modalities, change eval baselines |
-| Case 2 (Retrieval) | → Case 5 (Context), Case 11 (Structure) | Larger context changes RAG calculus; structure affects retrieval |
-| Case 3 (Protocol) | → Case 9 (Autonomy), Case 4 (Regulation) | Protocols enable new capabilities; standardization aids compliance |
-| Case 4 (Regulation) | → Case 9 (Autonomy), Case 10 (Eval) | Regulation constrains agents; mandates auditable evaluation |
-| Case 5 (Context) | → Case 2 (Retrieval), Case 11 (Structure) | Reduced RAG necessity; less aggressive structuring needed |
-| Case 6 (MaaS Disruption) | → Case 8 (Fine-Tuning), Case 4 (Regulation) | Local deployment enables fine-tuning; data residency drives on-prem |
-| Case 7 (Multi-Modal) | → Case 2 (Retrieval), Case 9 (Autonomy) | Multi-modal retrieval is distinct paradigm; vision enables computer use |
-| Case 8 (Fine-Tuning) | → Case 6 (MaaS), Case 10 (Eval) | Fine-tuning incentivizes local; more variants need scalable eval |
-| Case 9 (Autonomy) | → Case 4 (Regulation), Case 3 (Protocol) | Autonomous agents face stricter scrutiny; need safety protocols |
-| Case 10 (Eval Shift) | → Case 8 (Fine-Tuning) | AI judges generate training labels at scale |
-| Case 11 (Structure) | → Case 2 (Retrieval) | Better structure improves retrieval across all paradigms |
-
-### Cascade Patterns
-
-**Cascade 1 — Model → Context → Retrieval → Structure**: New model with 10M context (Case 1) → context expansion (Case 5) → reduced RAG necessity (Case 2) → knowledge structure adaptation (Case 11). All 4 cases require coordinated response.
-
-**Cascade 2 — Regulation → Safety → Agent → Protocol**: AI regulation (Case 4) → tightened safety requirements → constrained agent autonomy (Case 9) → standardized safety protocols (Case 3). Compliance becomes root cause for multiple architectural changes.
-
-**Cascade 3 — Local Deployment → Fine-Tuning → Eval Scaling**: Self-hosted models (Case 6) → accessible fine-tuning (Case 8) → many model variants needing evaluation → AI-as-judge for throughput (Case 10). Evaluation pipeline becomes bottleneck.
-
----
-
-## Related Documents
-- domain_scope.md — Sub-area definitions (Areas 1-8), membership criteria, technology assumptions, handoff points
-- dependency_rules.md — Inter-area dependencies, model/MCP/embedding dependencies, feedback loops
-- structure_spec.md — Token budget rules, quantitative criteria, frontmatter specification
-- competency_qs.md — Questions each area must answer; updated when scenarios reveal gaps
-- concepts.md — Term definitions; updated when new terms emerge from scenarios

package/.onto/domains/llm-native-development/logic_rules.md

@@ -1,123 +0,0 @@
----
-version: 2
-last_updated: "2026-03-30"
-source: manual
-status: established
----
-
-# LLM-Native Development Domain — Logic Rules
-
-Classification axis: **system construction concern** — rules classified by the design concern of the LLM-powered system they govern.
-
-## File–Concept Correspondence Logic (→Area 3)
-
-- One concept must be defined in exactly one concept file. If the same concept is defined in two files, it is a source of truth conflict
-- A filename must represent the concept the file defines. A mismatch between filename and content is a self-describing violation
-- If a concept file exists, it must define at least one concept. Empty files are not allowed
-- Meta files (INDEX.md, ARCHITECTURE.md, etc.) do not define concepts and serve the role of specifying structural relationships among other files. They are excluded from the "File = Concept" correspondence
-
-## Frontmatter Conformance Logic (→Area 3)
-
-- The specification for frontmatter (required fields, format) has structure_spec.md as its source of truth. When other files reference frontmatter, they follow structure_spec.md's definitions
-- All references declared in frontmatter (depends_on, related_to, parent) must point to actually existing files
-- The type field value in frontmatter must belong to the type list defined in the system. An undefined type is unclassifiable
-- If a relationship declared in frontmatter is bidirectional, the reverse relationship must also exist in the target file's frontmatter. In this case, follow the cycle exception clause in dependency_rules.md
-
-## Hierarchy Logic (→Area 3)
-
-- Directory hierarchy represents containment relationships of concepts. A parent directory must be a category encompassing the concepts in its child directories
-- Files within the same directory must be concepts at the same abstraction level. If abstraction levels differ, separate into directories
-- As directory depth increases, concepts become more specific. Depth reversal (child is more abstract) is a structural contradiction
-
-## Navigation Path Logic (→Area 3)
-
-- A reference chain must exist from the entry point to any concept file. A file that cannot be reached is an isolated document
-- If a reference chain has cycles, the LLM may fall into infinite traversal. Circular references must be explicitly marked in frontmatter (see cycle exception clause in dependency_rules.md)
-- When A references B and B references C, if understanding C's content from A requires traversing through B, a direct A→C reference should be added (navigation shortcut)
-
-## Change Propagation Logic (→Area 3)
-
-- When concept X's definition changes, all files referencing X are verification targets
-- When adding a new concept file, if existing concept files require modification beyond the meta file (INDEX.md), it is a signal of high coupling. Homonym registration (concepts.md) is exceptionally allowed
-- When deleting a file, all references to that file must be removed everywhere (referential integrity)
-
-## Model Integration Logic (→Area 1)
-
-- If model routing is used (directing requests to different models based on task type, cost, or complexity), fallback paths must be defined for every route. A route without a fallback is a single point of failure — if the target model is unavailable, the request fails silently or errors out
-- Model version pinning: if the model version is not pinned to a specific release (e.g., `gpt-4-0613` rather than `gpt-4`), behavior changes are non-deterministic. Provider-side model updates can alter output quality, format, or latency without notice
-- If multiple models are used for different tasks within the same system, capability requirements must be documented per model. Documentation must include: task type, required capabilities (e.g., tool use, structured output, long context), minimum acceptable quality level, and cost constraints
-- Model selection must be justified against the task's requirements. Using a more capable (and expensive) model where a simpler model suffices is a cost inefficiency. Using a less capable model where quality is critical is a quality risk
-- If inference optimization is applied (quantization, batching, KV cache), the optimization's impact on output quality must be measured, not assumed. Quantization in particular can degrade performance on tasks requiring precise reasoning
-
-## Prompt Design Logic (→Area 2)
-
-- Instruction hierarchy: system prompt > tool definitions > user prompt. When instructions at different levels conflict, higher-priority instructions take precedence. If the system prompt says "always respond in JSON" and the user prompt says "respond in plain text," the system prompt wins
-- If structured output is required (JSON, XML, function call schemas), the output schema must be validated before consumption. Trusting model output to conform to a schema without validation is unsafe — models can produce malformed output even when instructed to follow a schema
-- Token budget constraint: system prompt tokens + retrieved context tokens + user input tokens ≤ context window size - expected output tokens. Violating this constraint causes truncation (silent data loss) or API errors. The budget must be calculated before the API call, not after
-- Context rot: as conversation length grows, the earliest context loses influence on model behavior. Critical information (safety rules, output format requirements, identity constraints) must be re-injected at regular intervals or placed in positions of high attention (beginning and end of the context window)
-- Prompt templates must be versioned. A prompt change that alters system behavior is functionally equivalent to a code change. Unversioned prompts make it impossible to reproduce past behavior or diagnose regressions
-- Few-shot examples must be representative of the target distribution. Biased examples cause biased outputs. If the task distribution changes, the examples must be updated
-
-## Retrieval Logic (→Area 3)
-
-- RAG pipeline correctness: retrieval relevance must be verified independently of generation quality. If the final output is incorrect, the cause may be (a) irrelevant retrieval, (b) correct retrieval but incorrect generation, or (c) both. Diagnosing the root cause requires evaluating retrieval and generation separately
-- Chunking strategy must preserve semantic boundaries. Splitting mid-sentence or mid-paragraph destroys context that the embedding model needs to produce meaningful vectors. Chunk boundaries should align with document structure (paragraphs, sections, logical units)
-- If hybrid search is used (combining keyword-based search and semantic/vector search), the combination strategy must be explicit. Options include: score fusion (weighted sum of scores), rank fusion (reciprocal rank fusion), or cascade (keyword filter → semantic rerank). An unspecified combination strategy produces non-reproducible results
-- Embedding model selection must match the content domain. General-purpose embeddings may underperform on domain-specific content (medical, legal, code). If retrieval quality is insufficient, evaluate domain-specific or fine-tuned embedding models before increasing chunk count
-- Retrieved context must carry provenance metadata (source document, chunk ID, relevance score). Without provenance, it is impossible to debug retrieval failures or trace hallucinations back to their source
-
-## Agentic Systems Logic (→Area 4)
-
-- Tool definitions must be non-overlapping in their described capabilities. If two tools can accomplish the same task, the agent's choice between them is non-deterministic. Either remove the overlap or add explicit routing instructions that disambiguate when to use each tool
-- Multi-agent communication must have termination conditions. Without explicit termination (maximum iterations, convergence criteria, timeout), agent loops can run indefinitely, consuming resources and producing no useful output. Every loop must have a defined exit condition
-- Agent state must be explicitly managed. Relying on conversation history as implicit state is fragile — context windows are finite, and conversation history can be truncated, summarized, or lost. Critical state (task progress, accumulated results, decision points) must be stored in structured form (scratchpads, databases, state objects)
-- MCP tool schemas must be self-describing. The agent must be able to determine a tool's applicability from the schema alone (name, description, parameter descriptions) without external documentation. A tool schema that requires reading separate documentation to understand is poorly designed
-- Agent instructions must explicitly list which tools are available and when each should be used. If tools are available but not mentioned in instructions, the agent may discover them through schema inspection but cannot reliably determine appropriate usage context
-- For long-running agent tasks (multi-session persistence), structured progress tracking must be maintained. The agent must be able to resume from the last checkpoint without re-executing completed steps. Progress format: completed steps, current step, remaining steps, accumulated artifacts
-
-## Evaluation Logic (→Area 5)
-
-- Evaluation criteria must be defined before system development begins, not retrofitted after (spec-first principle). Defining evaluation criteria after seeing system output introduces confirmation bias — criteria are unconsciously shaped to match existing behavior rather than desired behavior
-- AI-as-judge evaluation must disclose: (a) the judge model identity and version, (b) the judge prompt used, and (c) known biases of the judge model (e.g., positional bias, verbosity bias, self-preference). Without disclosure, evaluation results are not reproducible or interpretable
-- A/B testing for system output quality requires statistical significance thresholds defined in advance. Without predefined thresholds, there is a temptation to stop testing when results look favorable (peeking), which inflates false positive rates
-- Golden set (reference test data) must be maintained separately from training/fine-tuning data. Contamination of the golden set with training data produces artificially inflated evaluation scores
-- Evaluation pipelines must be automated and repeatable. Manual evaluation is acceptable for initial exploration but does not scale. If the system is in production, evaluation must run on a regular cadence without human intervention
-
-## Safety Logic (→Area 6)
-
-- Defense-in-depth: no single safety mechanism is sufficient. Minimum layers: input filtering (block malicious prompts before they reach the model) + output filtering (block harmful content before it reaches the user) + monitoring (detect patterns that individual filters miss). Removing any layer increases risk
-- Prompt injection defense must not degrade normal user experience. A defense mechanism that blocks 5% of legitimate user requests to prevent 0.1% of injection attempts has an unacceptable false positive rate. Defense mechanisms must be evaluated for both efficacy (true positive rate) and cost (false positive rate)
-- Content policy rules must be testable. Each rule must have at least one positive test case (content that should be blocked) and one negative test case (content that should pass). Untestable rules cannot be verified and may silently fail or over-block
-- Safety constraints must be applied at the system level, not delegated to the model's built-in safety. Model-level safety is a useful baseline but is not configurable, not auditable, and can change without notice when the provider updates the model
-- Red teaming must be performed before deployment and on a regular cadence after deployment. The threat landscape evolves — attack techniques that did not exist during initial red teaming may emerge later
-
-## Operations Logic (→Area 7)
-
-- Cost tracking granularity must match billing granularity. If the provider bills per token, the system must track per-token usage. If the system tracks only per-request, cost attribution to specific features or users is impossible, making cost optimization guesswork
-- Quality drift detection requires a baseline. The baseline must be established during the evaluation phase (→Area 5) using the golden set. Without a baseline, there is no reference point to determine whether quality has degraded. Drift detection compares current output quality against the baseline at regular intervals
-- Feedback loop: user feedback must be actionable. Feedback without a defined path to system improvement is waste. For each feedback type (thumbs up/down, free-text correction, escalation), there must be a defined process: collection → aggregation → analysis → system change → verification
-- Logging must capture sufficient information to reproduce any LLM interaction: input (full prompt), output (full response), model version, latency, token count, and any tool calls. Insufficient logging makes debugging production issues impossible
-- Incident response for LLM-specific failures must account for failure modes unique to LLM systems: model provider outage, sudden quality degradation (model update), cost spike (unexpected token usage), safety filter false positives (legitimate requests blocked), and prompt injection in production
-
-## Data & Model Adaptation Logic (→Area 8)
-
-- Fine-tuning decision: fine-tuning is justified only when prompting alone cannot achieve the required quality, latency, or cost targets. Fine-tuning introduces a maintenance burden (dataset management, retraining on model updates) that prompting avoids
-- Training data quality has an upper bound on model quality. No amount of training can overcome systematically flawed data. Data quality assessment (accuracy, consistency, completeness, representativeness) must precede training
-- If parameter-efficient fine-tuning is used (LoRA, QLoRA, adapters), the base model version must be pinned. A base model update invalidates all existing adapters — adapters trained on version N are not guaranteed to work correctly on version N+1
-- Evaluation of fine-tuned models must compare against the base model on the same evaluation set. Without this comparison, there is no evidence that fine-tuning improved performance. Improvement must exceed the cost of fine-tuning to be justified
-
-## Constraint Conflict Checking (Cross-Cutting)
-
-When design constraints from different areas impose contradictory requirements, the conflict must be identified, documented, and resolved. Unresolved conflicts produce systems with unpredictable behavior.
-
-- **Safety vs. Functionality**: When safety constraints conflict with functionality (e.g., output filtering blocks valid responses, input filtering rejects legitimate queries), safety takes precedence. However, the conflict must be documented with: (a) the specific safety rule, (b) the functionality it degrades, (c) the false positive rate, and (d) a plan to reduce the false positive rate without weakening safety
-- **Cost vs. Quality**: When cost constraints conflict with quality (e.g., a cheaper model degrades output quality, reduced retrieval scope misses relevant context), the trade-off must be explicit in the system specification. The specification must state: which quality metrics are affected, by how much, and what the cost saving is. Implicit quality degradation (choosing the cheap option without measuring the impact) is prohibited
-- **Latency vs. Completeness**: When latency requirements conflict with completeness (e.g., retrieval timeout cuts off relevant results, generation is truncated to meet response time targets), the system must degrade gracefully. Partial results must be clearly marked as partial, and the user must be informed that completeness was sacrificed for speed
-- **Privacy vs. Personalization**: When privacy constraints conflict with personalization (e.g., PII redaction removes information needed for personalized responses), privacy takes precedence. The system must provide useful responses without requiring PII, or obtain explicit user consent for PII usage with clear data handling policies
-
-## Related Documents
-- concepts.md — Term definitions within this domain
-- dependency_rules.md — Cycle exception clause, reference direction rules
-- structure_spec.md — Source of truth for frontmatter specifications and structural rules
-- domain_scope.md — Scope definition, sub-area membership criteria, and boundary tests
-- competency_qs.md — Questions this domain's logic must be able to answer

package/.onto/domains/llm-native-development/prompt_interface.md

@@ -1,49 +0,0 @@
----
-version: 1
-last_updated: "2026-03-29"
-source: bundled-domain-baseline
-status: established
----
-
-# LLM-Native Development Domain — Prompt/Interface Design
-
-This document defines the design criteria for structures that issue instructions to LLMs — system prompts, tool definitions, role definitions, and response formats.
-While the existing 7 files cover "the structure of a system that describes knowledge," this file covers the separate concern dimension of "designing the knowledge consumption method."
-
-## System Prompt Structure
-
-- Prompts are structured in the order of role definition (who) → context (what) → task instructions (how) → constraints
-- Role definitions are managed in individual files within the roles/ directory. They are not inlined into the prompt but read from files and included
-- Context loading is either self-loading (agent reads files directly) or injection (team lead includes content). The choice depends on the agent's file reading capability
-
-## Role Definition File Structure
-
-Role definition files (roles/{agent-id}.md) include the following elements:
-- **Specialization**: The concern this role verifies/performs
-- **Role Description**: The core question this role answers
-- **Core Questions**: 3~6 items. Define the judgment axes of the role
-- **Domain Document Mapping**: Domain documents this role references (at least 1)
-
-## Tool Definition (Tool Schema)
-
-- Tool definitions follow JSON Schema format
-- Tool names directly express the action (e.g., `search_files`, `read_document`)
-- Tool descriptions must be specific enough for the LLM to judge tool selection
-- Required parameters and optional parameters are clearly distinguished
-
-## Response Format Constraints (Structured Output)
-
-- When structured output is needed, specify the output format in the prompt
-- Format specification methods: choose between examples or schemas (JSON Schema)
-- When the format is complex, separate it into a file (templates/ directory) and reference from the prompt
-
-## Context Window Utilization Strategy
-
-- Prompt size is recommended to be 30% or less of the total context window. The remainder is allocated to work targets and responses
-- For repeatedly executed prompts, separate static parts (role, rules) from dynamic parts (work target, previous results)
-- When static parts do not change, consider caching or compression
-
-## Related Documents
-- concepts.md — Definitions of terms used in prompts
-- structure_spec.md — Physical location rules for role files
-- domain_scope.md — Higher-level definition of the "Consumption Interface" concern area