mdcontext 0.0.1 → 0.2.0

package/research/dogfood/strategy-c/c5-diver-execution.md

@@ -0,0 +1,221 @@
+# Report: C5 - Diver: Execution Model
+
+## Mission
+
+Deep-dive into: Execution Model (Jobs, Contexts, Workflows)
+
+## Command Log
+
+| #   | Command                                                                                       | Purpose                            | Result                                  | Useful?     |
+| --- | --------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------------------- | ----------- |
+| 1   | `mdcontext --help`                                                                            | Understand available commands      | Full command listing with examples      | Yes         |
+| 2   | `mdcontext tree`                                                                              | List all markdown files            | 23 files found, identified key docs     | Yes         |
+| 3   | `mdcontext tree docs/04-EXECUTION_MODEL.md`                                                   | Show document outline              | 38 sections, 7418 tokens total          | Very useful |
+| 4   | `mdcontext context docs/04-EXECUTION_MODEL.md -t 4000`                                        | Get overview of execution model    | Truncated to 36%, got main sections     | Useful      |
+| 5   | `mdcontext search "job" --json`                                                               | Find job-related content           | 10 results across multiple files        | Useful      |
+| 6   | `mdcontext search "execution context"`                                                        | Find execution context content     | 10 results with context lines           | Useful      |
+| 7   | `mdcontext context docs/04-EXECUTION_MODEL.md --section "2. Jobs"`                            | Deep dive on Jobs                  | Full Jobs section extracted             | Very useful |
+| 8   | `mdcontext context docs/04-EXECUTION_MODEL.md --section "3. Execution Contexts"`              | Deep dive on Contexts              | Full Execution Contexts section         | Very useful |
+| 9   | `mdcontext context docs/04-EXECUTION_MODEL.md --section "4. Workflows in Execution"`          | Workflows in execution             | Full workflows section                  | Very useful |
+| 10  | `mdcontext tree docs/06-WORKFLOWS.md`                                                         | Outline of dedicated Workflows doc | 37 sections, 8541 tokens                | Useful      |
+| 11  | `mdcontext context docs/06-WORKFLOWS.md --section "1. Workflow Philosophy"`                   | Workflow philosophy                | Core principles extracted               | Very useful |
+| 12  | `mdcontext context docs/06-WORKFLOWS.md --section "2. What Problem Workflows Solve"`          | Problem statement                  | Clear value proposition                 | Useful      |
+| 13  | `mdcontext search "parallel execution"`                                                       | Find parallel patterns             | 10 results, multiple examples           | Useful      |
+| 14  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "5. Execution Coordination Patterns"` | Coordination patterns              | Pipeline, Fan-out, Background patterns  | Very useful |
+| 15  | `mdcontext search "context resumption"`                                                       | Find resumption info               | 6 results across docs                   | Useful      |
+| 16  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "6. Context Creation and Resumption"` | Resumption mechanics               | Critical resumption model explained     | Very useful |
+| 17  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "7. Scope and Mutual Exclusion"`      | Concurrency rules                  | Hard invariant explained                | Very useful |
+| 18  | `mdcontext search "scope"`                                                                    | Find scope usage                   | 10 results showing scope patterns       | Useful      |
+| 19  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "16. Guarantees"`                     | System guarantees                  | 7 key guarantees listed                 | Very useful |
+| 20  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "18. Summary"`                        | Executive summary                  | Clean separation of concerns            | Very useful |
+| 21  | `mdcontext tree docs/07-EXAMPLE_WORKFLOWS.md`                                                 | Example workflows outline          | 5 detailed examples, 14423 tokens       | Useful      |
+| 22  | `mdcontext context docs/07-EXAMPLE_WORKFLOWS.md --section "Summary Across All Examples"`      | Examples summary                   | Common patterns across use cases        | Very useful |
+| 23  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "8. Steps and Agents"`                | Steps and Agents                   | Atomic execution units                  | Very useful |
+| 24  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "17. Why This Model Scales"`          | Scalability                        | Scale dimensions explained              | Very useful |
+| 25  | `mdcontext search "disposable"`                                                               | Find disposable concept            | 10 results showing ephemeral nature     | Useful      |
+| 26  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "10. External Workflow Integration"`  | External framework integration     | BMAD, RD-Agent integration pattern      | Very useful |
+| 27  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "11. Key Execution Patterns"`         | Execution patterns                 | Ad-hoc, guided, background, parallel    | Very useful |
+| 28  | `mdcontext context docs/04-EXECUTION_MODEL.md --section "12. Control Plane Integration"`      | Control plane                      | Intervention capabilities               | Very useful |
+| 29  | `mdcontext search "checkpoint"`                                                               | Find checkpoint content            | 10 results showing checkpoint usage     | Useful      |
+| 30  | `mdcontext context docs/06-WORKFLOWS.md --section "12. Checkpoints in Depth"`                 | Checkpoint mechanics               | 6 checkpoint types, enforcement rules   | Very useful |
+| 31  | `mdcontext search "immutable"`                                                                | Find immutability patterns         | 10 results confirming append-only model | Useful      |
+| 32  | `mdcontext stats`                                                                             | Index statistics                   | 23 docs, 924 sections, no embeddings    | Useful      |
+
+## Findings
+
+### Key Discoveries
+
+- **Execution is scoped, disposable attempts guided by durable intent** - This is the philosophical core
+- **Jobs coordinate, Contexts execute, Workflows guide** - Clean separation of concerns
+- **Immutability transforms conflicts into choices** - No race conditions, no merge logic
+- **Resumption creates new contexts** - No hidden state, perfect auditability
+- **External frameworks (BMAD, RD-Agent) execute WITHIN the model** - Not above it
+
+### Model Components
+
+#### Jobs: Human-Meaningful Coordination Envelopes
+
+> "A Job is a human-meaningful coordination envelope that groups one or more Execution Contexts to achieve an objective."
+> Source: docs/04-EXECUTION_MODEL.md, Section 2
+
+Jobs provide:
+
+- **Continuity**: Survive execution attempts
+- **Grouping**: Relate multiple execution contexts
+- **Navigation**: Stable handle for UI/API
+- **Signals**: Aggregate status and attention needs
+- **Artifact reference**: Point to key outputs
+
+Jobs have NO stored state - status is derived from signals. They can coordinate serial, parallel, or mixed execution patterns.
+
+#### Execution Contexts: Disposable Actors
+
+> "An Execution Context is a temporary, scoped binding that performs one unit of concrete work."
+> Source: docs/04-EXECUTION_MODEL.md, Section 3
+
+Key duality:
+
+- **Contexts are disposable**: May be deleted anytime, not resumed
+- **But emit permanent records**: Events logged immutably, artifacts preserved, provenance captured
+
+**The Hard Concurrency Rule**:
+
+> "No two Execution Contexts may concurrently mutate the same scope of work."
+> Source: docs/04-EXECUTION_MODEL.md, Section 3
+
+Because the system is fully immutable:
+
+- Contexts do not mutate - they produce new artifacts
+- "Conflict" is just parallel alternatives
+- Resolution is a separate, explicit act
+
+#### Workflows: Guidance Not Execution
+
+> "Reusable coordination patterns that shape how work unfolds, without prescribing execution logic."
+> Source: docs/06-WORKFLOWS.md, Section 1
+
+Workflows consist of:
+
+- **Entry Signals**: When relevant
+- **Roles**: Which perspectives needed
+- **Phases**: Broad stages of work
+- **Activities**: What kind of work happens
+- **Checkpoints**: Deliberate pauses for human re-engagement
+
+**Critical Rule**: An Execution Context may reference AT MOST ONE workflow.
+
+### Relevant Quotes/Sections Found
+
+> "HumanWork treats execution as a series of scoped, disposable attempts guided by durable intent."
+> Source: docs/04-EXECUTION_MODEL.md, Section 1
+
+> "Execution Contexts do not resume. Instead: A new Execution Context is created. It reads Workspace Memory to understand prior work."
+> Source: docs/04-EXECUTION_MODEL.md, Section 6
+
+> "This transforms conflicts from correctness problems into choice problems."
+> Source: docs/04-EXECUTION_MODEL.md, Section 3
+
+> "Workflows do not execute. They provide coordination patterns."
+> Source: docs/04-EXECUTION_MODEL.md, Section 4
+
+### Execution Coordination Patterns
+
+**Pattern A: Serial Execution (Pipeline)**
+
+- Context 1: Design -> spec
+- Context 2: Implement -> code
+- Context 3: Review -> approval
+- Job survives and maintains continuity
+
+**Pattern B: Parallel Execution (Fan-Out/Fan-In)**
+
+- Context A, B, C run concurrently on distinct scopes
+- Context D compares and merges
+- Resolution is explicit, not automatic
+
+**Pattern C: Background + Foreground**
+
+- Long-running analysis + interactive brainstorming
+- Both execute because scopes are distinct
+
+### Steps and Agents
+
+- **Steps**: Atomic units within Execution Context, owned by role, emit progress
+- **Agents**: Stateless workers, receive assembled context, must report status before token exhaustion
+
+**Agent Replaceability**: Work continues seamlessly if agent crashes/exhausts tokens because context is external.
+
+### System Guarantees
+
+1. Single-scope coherence: no concurrent mutation of same scope
+2. Human control: execution always interruptible
+3. Deterministic replay: all execution reproducible from records
+4. Cost attribution: every execution traceable
+5. No hidden state: all context external and inspectable
+6. Agent replaceability: work continues regardless of agent instance
+7. Workflow interoperability: external systems integrate cleanly
+
+### Summary of Theme
+
+The HumanWork execution model separates:
+
+- **Human intent** (Jobs)
+- **Work attempts** (Execution Contexts)
+- **Guidance patterns** (Workflows)
+- **Concrete execution** (Steps/Agents)
+
+This enables:
+
+- Parallel exploration without conflicts
+- Long-running work without loss of control
+- Framework integration without lock-in
+- Enterprise safety without bureaucracy
+- Scale without brittleness
+
+The model scales from $3 individual explorations to $3000 enterprise compliance reviews without changing architecture.
+
+## Proposed Spec Changes
+
+- [ ] Add glossary entry clarifying "scope" types (explicit, implicit, tag-based, artifact-based)
+- [ ] Create diagram showing Job -> Context -> Steps -> Agents hierarchy
+- [ ] Document checkpoint timeout escalation paths in more detail
+- [ ] Add explicit guidance on when to use parallel vs serial execution patterns
+- [ ] Clarify what "Semantic Memory" contributes to context resumption (mentioned but not detailed)
+- [ ] Add example of scope conflict resolution workflow
+
+## Tool Evaluation
+
+### What Worked Well
+
+- **`--section` flag is excellent**: Precise extraction of specific sections without reading full documents
+- **`tree` on individual files**: Instant document outline with token counts per section - perfect for planning
+- **Search with context lines**: Default context before/after matches aids understanding
+- **Section numbering**: Consistent 1.2, 1.3 scheme makes navigation predictable
+- **Token budgeting**: `-t 4000` respects limits while showing included/excluded sections
+- **Stats command**: Quick index health check
+
+### What Was Frustrating
+
+- **Initial `--section` syntax error**: Passing multiple `--section` flags tried to read flag as filename (had to call once per section)
+- **No embeddings**: Semantic search would have helped find conceptual relationships
+- **Search results capped at 10**: Sometimes wanted to see more matches
+- **Context command truncation**: 36% of 7418 tokens shown with default budget; would prefer explicit "this section won't fit" warning
+
+### Confidence Level
+
+[X] High - The execution model is thoroughly documented with clear definitions, examples, and guarantees. The section-based extraction worked extremely well for deep-diving into complex technical documentation.
+
+### Would Use Again? (1-5)
+
+**5** - The `--section` flag is a game-changer for deep dives. Being able to see the full document outline with token counts, then surgically extract specific sections, made this investigation highly efficient. The search found relevant content across files quickly. This is exactly how technical documentation exploration should work.
+
+**Key workflow discovered**:
+
+1. `tree <file>` - see structure and token counts
+2. `context <file> --section "X"` - extract needed sections
+3. `search "term"` - find cross-references
+4. Repeat as needed
+
+Total commands: 32
+Useful commands: 32 (100%)
+Highly useful commands: ~20 (63%)

package/research/dogfood/strategy-c/c6-diver-org.md

@@ -0,0 +1,221 @@
+# Report: C6 - Diver: Org-Workspace Model
+
+## Mission
+
+Deep-dive into: Org-Workspace-Cost Model (Theme #5 from C1 Explorer)
+
+## Command Log
+
+| #   | Command                                                            | Purpose                         | Result                                  | Useful? |
+| --- | ------------------------------------------------------------------ | ------------------------------- | --------------------------------------- | ------- |
+| 1   | `mdcontext search "workspace"`                                     | Find workspace content          | 10 results spanning 4 docs              | Yes     |
+| 2   | `mdcontext search "org"`                                           | Find org content                | 10 results, good coverage               | Yes     |
+| 3   | `mdcontext search "cost"`                                          | Find cost-related content       | 10 results, found key docs              | Yes     |
+| 4   | `mdcontext context docs/03-ORG_WORKSPACE_MODEL.md`                 | Get main doc overview           | Truncated to 24%, but overview helpful  | Partial |
+| 5   | `mdcontext context --section "5. Authority Model"`                 | Get authority details           | Complete section retrieved              | Yes     |
+| 6   | `mdcontext context --sections`                                     | List all sections               | Full section index shown                | Yes     |
+| 7   | `mdcontext context --section "6. Cost and Resource Model"`         | Get cost model details          | Complete section with all subsections   | Yes     |
+| 8   | `mdcontext context --section "9. Org-Workspace Information Flow"`  | Cross-workspace learning        | Found key principle on authority flow   | Yes     |
+| 9   | `mdcontext search "promotion"`                                     | Find artifact promotion content | 10 results, good workflow examples      | Yes     |
+| 10  | `mdcontext context --section "Phase 11: Promotion to Org Level"`   | Example workflow                | Concrete promotion example              | Yes     |
+| 11  | `mdcontext search "governance"`                                    | Find governance content         | 10 results including enterprise docs    | Yes     |
+| 12  | `mdcontext context --section "12. Why This Model Works"`           | Model rationale                 | Scaling from individuals to enterprises | Yes     |
+| 13  | `mdcontext search "job"`                                           | Find job layer content          | 10 results, found architecture section  | Yes     |
+| 14  | `mdcontext context --section "6. Job Layer"`                       | Get job definition              | Complete job layer explanation          | Yes     |
+| 15  | `mdcontext search "execution context"`                             | Find context layer content      | 10 results                              | Yes     |
+| 16  | `mdcontext context --section "7. Execution Context Layer"`         | Get context definition          | Full lifecycle and semantics            | Yes     |
+| 17  | `mdcontext context --section "2. High-Level Stack"`                | Get hierarchy diagram           | ASCII stack visualization               | Yes     |
+| 18  | `mdcontext context --section "15. Cost and Resource Model"` (arch) | Architecture cost summary       | Concise cost flow explanation           | Yes     |
+| 19  | `mdcontext search "enterprise"`                                    | Find enterprise examples        | Example 5 compliance review             | Yes     |
+| 20  | `mdcontext context --section "Example 5: Enterprise"`              | Get enterprise workflow         | Detailed compliance review walkthrough  | Yes     |
+| 21  | `mdcontext search "knowledge graph"`                               | Find org knowledge graph        | Found definitions and usage             | Yes     |
+| 22  | `mdcontext context --section "Org Knowledge Graph"`                | Get knowledge graph details     | Derived graph properties                | Yes     |
+| 23  | `mdcontext context --section "13. Summary"`                        | Get model summary               | Core principles list                    | Yes     |
+| 24  | `mdcontext search "policy"`                                        | Find policy overlay content     | 10 results on optional controls         | Yes     |
+| 25  | `mdcontext context --section "Cost Attribution"` (exec)            | Get cost attribution levels     | Three-level cost hierarchy              | Yes     |
+| 26  | `mdcontext search "authority"`                                     | Find authority invariants       | Key authority principles                | Yes     |
+| 27  | `mdcontext search "boundary"`                                      | Find containment boundaries     | Workspace isolation semantics           | Yes     |
+| 28  | `mdcontext context --section "Workspace Boundaries"`               | Get boundary details            | Strict isolation guarantees             | Yes     |
+| 29  | `mdcontext context --section "17. Org Knowledge Graph"` (prim)     | Formal knowledge graph def      | Guarantees and properties               | Yes     |
+
+## Findings
+
+### Key Discoveries
+
+1. **Two-Tier Model is Intentional**: The Org/Workspace split enables "shared learning without central control, reuse without coupling, governance without bureaucracy, cost visibility without micromanagement"
+
+2. **Authority Never Flows Downward**: Critical invariant - Org-level artifacts can be suggested/referenced but never auto-attach, auto-execute, or auto-override Workspace decisions
+
+3. **Agents Explicitly Excluded from Org-Level Authority**: Agents cannot create Workspaces, promote artifacts, or invite users - "keeps humans firmly in control"
+
+4. **Governance is an Overlay, Not Foundation**: Default is permissive (trust-by-default), enterprise controls are optional policy additions
+
+5. **Cost Visibility Over Control**: Philosophy emphasizes transparency and self-regulation over enforcement
+
+### Model Hierarchy
+
+```
+Human(s)
+   |
+Control Plane (ALWAYS LISTENING, PREEMPTIVE)
+   |
+Org (identity, ownership, learning)
+   |
+Workspace (shared reality, work boundary)
+   |
+Jobs (intent over time, coordination)
+   |
+Execution Contexts (disposable actors, scoped attempts)
+   |
+Workflows/Steps/Agents/Tools
+   |
+Artifacts/Records (immutable facts)
+```
+
+- **Org**: Top-level identity and ownership boundary. Owns Workspaces, participants, Org-level Artifacts, Knowledge Graph, policies, and cost allocation. Represents an individual, family, team, company, or any collective.
+
+- **Workspace**: The ONLY place where work occurs. Provides strict isolation - execution state, artifacts, history, and cost all exist within a Workspace. Can reference other Workspaces but never implicitly share state.
+
+- **Job**: Human-meaningful coordination envelope that groups Execution Contexts. Provides continuity across execution attempts, UI handles, signal aggregation. Does NOT execute or store state - survives deletion of its contexts.
+
+- **Context (Execution Context)**: The unit of actual work. Ephemeral, scoped, disposable. Binds a scope of work, roles, and optionally a workflow. Creates permanent records (events) but is itself temporary.
+
+### Cost Model
+
+**Three-Level Attribution:**
+
+1. **Execution Context Level (Most Granular)**
+   - Token consumption
+   - Tool invocations
+   - Compute time
+   - Storage of context artifacts
+
+2. **Job Level (Aggregated)**
+   - Sum of all associated Execution Contexts
+   - Visible to humans
+   - Used for budget decisions
+
+3. **Workspace Level (Boundary)**
+   - Sum of all Jobs
+   - Org-level reporting
+   - Policy enforcement point
+
+**Key Principles:**
+
+- Cost is owned at Org level, attributed at Workspace level, accrued at Execution Context level
+- Dormant Workspaces incur minimal cost
+- Navigation and discovery are low-cost by design
+- Default Workspace is intentionally lightweight (zero-friction onboarding)
+
+**Future Controls (Optional Overlays):**
+
+- Soft budgets per Workspace
+- Org-level usage alerts
+- Creation limits
+- Execution throttles
+- Approval workflows
+
+### Cross-Workspace Learning (Promotion)
+
+**Upward Flow (Workspace to Org):**
+
+- Artifacts promoted via explicit human or policy action
+- Patterns extracted for Org Knowledge Graph
+- Learning aggregated
+- Flow is one-way, lossy, asynchronous, revocable
+
+**Downward Flow (Org to Workspace):**
+
+- Org artifacts can be suggested, referenced, or explicitly imported
+- May NEVER auto-attach, auto-execute, or auto-override
+
+> **"Truth is local. Meaning is global. Authority never flows downward."**
+
+### Relevant Quotes/Sections Found
+
+> "This document defines the identity, containment, cost, and authority model for HumanWork."
+> Source: docs/03-ORG_WORKSPACE_MODEL.md, intro
+
+> "Workspace boundaries are strict and enforced... Workspaces may reference one another, but they never implicitly share state."
+> Source: docs/03-ORG_WORKSPACE_MODEL.md, Workspace Boundaries
+
+> "Cost is: owned at the Org level, attributed at the Workspace level, accrued at the Execution Context level, visible in real-time"
+> Source: docs/01-ARCHITECTURE.md, Section 15
+
+> "No authority flows from Org to Workspace automatically"
+> Source: docs/02-PRIMITIVES.md, Section 19 Invariants
+
+> "Agents: Never have Org-level roles, Only operate within Workspaces, Inherit authority from Workspace roles, Cannot create Workspaces, Cannot promote artifacts, Cannot invite users"
+> Source: docs/03-ORG_WORKSPACE_MODEL.md, Agents Never Have Org-Level Authority
+
+### Enterprise Governance Example
+
+The compliance review example (Example 5) demonstrates:
+
+- 10 participants across 5 functional areas
+- Heavy checkpoint enforcement
+- Parallel review streams (security, legal, technical)
+- Remediation loops with clear ownership
+- Executive approval flow with dual sign-off
+- Automatic audit trail generation
+- Org-level promotion of compliance artifacts
+- Cost: $3,240 tracked and attributed
+- Time: 23 days with full visibility
+
+### Summary of Theme
+
+The Org-Workspace model enables governance through:
+
+1. **Clear Containment**: Two-tier hierarchy separates identity/ownership (Org) from execution/work (Workspace)
+
+2. **Cost Attribution by Design**: Every cost element flows up from Context to Job to Workspace to Org, making budget decisions transparent
+
+3. **Promotion Mechanism**: Cross-workspace learning happens through explicit artifact promotion, preserving autonomy while enabling reuse
+
+4. **Authority Boundaries**: Agents operate only within Workspaces; humans control cross-workspace flows
+
+5. **Overlay Architecture**: Governance is additive - enterprises add policy controls, individuals get trust-by-default
+
+6. **Audit Trail**: Immutable events enable regulatory-grade compliance without bureaucratic overhead
+
+## Proposed Spec Changes
+
+- [ ] Add explicit cost budget alerts/limits API to the Cost Model section
+- [ ] Define formal promotion workflow stages (propose, review, approve, publish)
+- [ ] Clarify how Org Knowledge Graph recommendations surface in Workspace UX
+- [ ] Add section on Workspace archival policies for cost management
+- [ ] Define multi-Org scenarios (user belonging to multiple Orgs) and cross-Org patterns
+- [ ] Add monitoring/dashboard primitives for cost visibility
+
+## Tool Evaluation
+
+### What Worked Well
+
+- **Search was fast and relevant**: Every search returned useful results with context
+- **Section targeting**: `--section` flag allowed precise extraction of specific topics
+- **Hierarchical navigation**: `--sections` gave clear document structure for navigation
+- **Line-level context**: Search showed surrounding lines which helped understand context
+- **Token efficiency**: Got focused content without loading entire documents
+
+### What Was Frustrating
+
+- **Initial context truncation**: Default context command only showed 24% of large document
+- **Two-word search failure**: `mdcontext search "job context"` returned 0 results (had to search separately)
+- **No semantic search**: Tip kept reminding to run `--embed` but this was out of scope
+- **Section name matching**: Had to get exact section names (e.g., "6" didn't work, needed "6. Cost and Resource Model")
+
+### Confidence Level
+
+[X] High / Medium / Low
+
+The Org-Workspace-Cost Model is well-documented with:
+
+- Dedicated document (03-ORG_WORKSPACE_MODEL.md)
+- Architecture section coverage
+- Primitives definitions
+- Detailed enterprise example workflow
+- Clear invariants and guarantees
+
+### Would Use Again? (1-5)
+
+**4** - The tool effectively supported deep-dive research. Section targeting was invaluable for large documents. The combination of search (for discovery) and context (for retrieval) worked well. Lost one point for multi-word search limitations and initial truncation confusion.