@smartergpt/lexrunner 1.0.0

package/docs/TOKEN_TRACKING.md

@@ -0,0 +1,162 @@
+# Token Usage Tracking
+
+This module provides measurement-only token usage tracking to help analyze context consumption before making optimization decisions.
+
+## Overview
+
+Token tracking uses a simple character-based heuristic (chars/4) to estimate token counts. This is intentionally rough - we need directional data, not precision.
+
+## Components
+
+### Token Estimator (`src/util/tokenEstimator.ts`)
+
+Provides functions to estimate token counts:
+
+```typescript
+import { estimateTokens, estimateTokensFromFile } from "./util/tokenEstimator.js";
+
+// Estimate from text
+const tokens = estimateTokens("hello world"); // ~3 tokens
+
+// Estimate from file
+const fileTokens = estimateTokensFromFile("/path/to/file.txt", fs);
+```
+
+### Token Logger (`src/monitoring/tokenLogger.ts`)
+
+Writes structured JSONL logs to `.smartergpt.local/runner/logs/token-usage.jsonl`:
+
+```typescript
+import { createTokenLogger } from "./monitoring/tokenLogger.js";
+
+const logger = createTokenLogger({ profileDir: "/path/to/profile" });
+
+// Log text content
+logger.logText("load-instruction", "AGENTS.md", fileContent);
+
+// Log from file path
+logger.logFile("load-persona", "/path/to/persona.md");
+
+// Log with metadata
+logger.log("cli-operation", "merge-weave", 250, {
+  operation: "merge",
+  prCount: 5,
+});
+
+await logger.close();
+```
+
+### Token Report Command
+
+Analyze token usage:
+
+```bash
+# Human-readable summary
+lex-pr token-report
+
+# JSON output
+lex-pr token-report --json
+
+# Specify profile directory
+lex-pr token-report --profile-dir /custom/path
+```
+
+## Usage Example
+
+```typescript
+import { createTokenLogger } from "./monitoring/tokenLogger.js";
+import * as fs from "fs";
+
+async function trackInstructionLoading(profileDir: string) {
+  const logger = createTokenLogger({ profileDir });
+
+  // Track instruction file loading
+  const instructionFiles = ["AGENTS.md", ".github/copilot-instructions.md", "CLAUDE.md"];
+
+  for (const file of instructionFiles) {
+    if (fs.existsSync(file)) {
+      logger.logFile("load-instruction", file, { type: "instruction" });
+    }
+  }
+
+  // Track persona loading
+  const personaFiles = [".smartergpt/personas/senior-dev.md", ".smartergpt/personas/eager-pm.md"];
+
+  for (const file of personaFiles) {
+    if (fs.existsSync(file)) {
+      logger.logFile("load-persona", file, { type: "persona" });
+    }
+  }
+
+  await logger.close();
+}
+```
+
+## Log Format
+
+JSONL format (one JSON object per line):
+
+```jsonl
+{"timestamp":"2025-12-15T00:00:00.000Z","operation":"load-instruction","source":"AGENTS.md","estimatedTokens":500,"metadata":{"type":"instruction"}}
+{"timestamp":"2025-12-15T00:00:01.000Z","operation":"load-persona","source":"senior-dev.md","estimatedTokens":200,"metadata":{"type":"persona"}}
+```
+
+## Report Output
+
+Human-readable:
+
+```
+Token Usage Report
+============================================================
+
+Total Tokens: 1,250
+Total Entries: 5
+Log File: /path/to/token-usage.jsonl
+
+By Source:
+------------------------------------------------------------
+  AGENTS.md
+    Tokens: 500 (40.0%)
+    Entries: 1
+  copilot-instructions.md
+    Tokens: 300 (24.0%)
+    Entries: 1
+
+By Operation:
+------------------------------------------------------------
+  load-instruction: 800 (64.0%)
+  load-persona: 200 (16.0%)
+```
+
+JSON format is also available via `--json` flag for programmatic analysis.
+
+## Design Principles
+
+1. **Measurement-only**: No behavior changes to existing logic
+2. **Opt-in**: Token logging is enabled by default but can be disabled
+3. **Structured output**: JSONL format for easy parsing and analysis
+4. **Deterministic**: Stable, sortable output for diffing
+5. **Lightweight**: No heavy dependencies, simple heuristic
+
+## Future Instrumentation Points
+
+Potential areas to add token tracking:
+
+- Instruction file loading (AGENTS.md, copilot-instructions.md)
+- Persona file loading (.smartergpt/personas/\*.md)
+- Major CLI operations (merge-weave, fanout, etc.)
+- Tool call responses (if metadata available)
+- Context window checkpoints
+
+## Testing
+
+```bash
+npm test -- tests/token-usage.spec.ts
+```
+
+## Notes
+
+- Token estimation is approximate (chars/4 heuristic)
+- Logs accumulate over time - consider log rotation policies
+- Log file location: `<profile-dir>/runner/logs/token-usage.jsonl`
+- Designed for analysis, not production metrics

package/docs/WORKFLOW_GUIDANCE.md

@@ -0,0 +1,277 @@
+# Workflow Guidance (LPR-037)
+
+## Overview
+
+The `workflow.guide` MCP tool provides context-aware guidance for lexrunner workflows, helping agents navigate complex operations by providing:
+
+- **Next steps**: Available actions in the current workflow phase
+- **Common issues**: Known problems and their solutions
+- **Recommendations**: Suggested next action
+- **Documentation links**: Relevant documentation for the phase
+
+## Architecture
+
+Following Guff's Option 1 approach from the backlog analysis, this feature:
+
+- **Separates concerns**: Deterministic data vs. contextual guidance
+- **Doesn't modify existing responses**: Guidance is opt-in via helper command
+- **Maintains simplicity**: MCP server stays focused on I/O
+
+## Workflow Phases
+
+### 1. Initial
+
+Starting point before any operations.
+
+**Next steps:**
+
+- `plan.create` (required) - Create execution plan
+
+**Recommended action:** `plan.create`
+
+### 2. Post-Plan-Creation
+
+Plan has been created, ready for validation and gate execution.
+
+**Next steps:**
+
+- `gates.run` (required) - Execute gates to validate PRs
+- `plan.validate` (optional) - Check for schema errors
+- `merge.order` (optional) - Calculate dependency-based merge order
+
+**Common issues:**
+
+- Plan schema validation errors → Use `plan.validate`
+- Unknown dependencies → Check PR dependency declarations
+
+**Recommended action:** `gates.run`
+
+### 3. Post-Gates-Run
+
+Gates have been executed, check results before merge.
+
+**Next steps:**
+
+- `status` (required) - Check gate execution results
+- `merge.apply` (optional) - Apply merge operations (use dryRun first)
+
+**Common issues:**
+
+- Gate failures → Check `artifacts/PR-XXX/gateName/` for logs
+- PRs blocked from merge → Use `status` tool to see why
+
+**Recommended action:** `status`
+
+### 4. Pre-Merge
+
+Ready to merge, perform final checks.
+
+**Next steps:**
+
+- `merge.apply` (required) - Apply merge operations
+
+**Common issues:**
+
+- Merge conflicts detected → Resolve manually or use AI conflict resolution
+- Branch protection prevents merge → Ensure status checks pass and approvals obtained
+
+**Recommended action:** `merge.apply`
+
+### 5. Post-Merge
+
+Merge completed successfully.
+
+**Next steps:** None (workflow complete)
+
+### 6. Error-Recovery
+
+Error occurred, attempting recovery.
+
+**Next steps:**
+
+- `doctor` (optional) - Run diagnostics
+- `status` (optional) - Check current state
+
+**Common issues:**
+
+- Command execution failed → Check error messages and use `doctor`
+
+## Usage
+
+### MCP Tool
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/call",
+  "params": {
+    "name": "workflow.guide",
+    "arguments": {
+      "phase": "post-plan-creation"
+    }
+  }
+}
+```
+
+**Response:**
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": {
+    "content": [{
+      "type": "text",
+      "text": "{
+        \"phase\": \"post-plan-creation\",
+        \"nextSteps\": [
+          {
+            \"command\": \"gates.run\",
+            \"description\": \"Execute gates to validate PRs before merging\",
+            \"required\": true
+          },
+          ...
+        ],
+        \"commonIssues\": [...],
+        \"documentation\": \"docs/merge-weave-workflow.md\",
+        \"canProceed\": true,
+        \"recommendedAction\": \"gates.run\"
+      }"
+    }]
+  }
+}
+```
+
+### Programmatic Usage (TypeScript)
+
+```typescript
+import { WorkflowStateMachine } from "lexrunner/src/mcp/workflow/state-machine.js";
+
+// Create state machine
+const sm = new WorkflowStateMachine("initial");
+
+// Get current guide
+const guide = sm.getGuide();
+console.log(`Recommended: ${guide.recommendedAction}`);
+
+// Complete a step
+sm.completeStep("plan.create");
+
+// Transition to next phase
+sm.transition("post-plan-creation");
+```
+
+### Stateless Helper
+
+```typescript
+import { createWorkflowGuide } from "lexrunner/src/mcp/workflow/state-machine.js";
+
+// Get guidance without maintaining state
+const guide = createWorkflowGuide("post-gates-run");
+console.log(guide.nextSteps);
+console.log(guide.commonIssues);
+```
+
+## API Reference
+
+### WorkflowGuide
+
+```typescript
+interface WorkflowGuide {
+  phase: string;
+  nextSteps: WorkflowAction[];
+  commonIssues: CommonIssue[];
+  documentation?: string;
+  canProceed: boolean;
+  recommendedAction?: string;
+}
+```
+
+### WorkflowAction
+
+```typescript
+interface WorkflowAction {
+  command: string; // MCP tool command name
+  description: string; // What this action does
+  required: boolean; // Must be performed before proceeding
+}
+```
+
+### CommonIssue
+
+```typescript
+interface CommonIssue {
+  symptom: string; // Error message or symptom pattern
+  solution: string; // Recommended solution
+}
+```
+
+## Implementation Details
+
+### Files Created
+
+- `src/mcp/types/guided-response.ts` - Type definitions
+- `src/mcp/workflow/state-machine.ts` - State machine and guidance logic
+- `tests/workflow-state-machine.spec.ts` - State machine tests (29 tests)
+- `tests/mcp-workflow-guide.spec.ts` - MCP integration tests (5 tests)
+
+### Integration Points
+
+1. **MCP Server** (`src/mcp/server.ts`):
+   - Added `workflow.guide` tool definition
+   - Added `handleWorkflowGuide` handler
+   - Added `WorkflowGuideArgs` validation schema
+
+2. **MCP Server Wrapper** (`mcp-server.mjs`):
+   - Added `workflow.guide` tool entry
+   - Imports `createWorkflowGuide` from built dist
+
+3. **CLI Exports** (`src/cli.ts`):
+   - Exported `createWorkflowGuide` for MCP server usage
+
+## Testing
+
+```bash
+# Run workflow state machine tests
+npm test -- tests/workflow-state-machine.spec.ts
+
+# Run MCP integration tests
+npm test -- tests/mcp-workflow-guide.spec.ts
+
+# Manual MCP server test
+echo '{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "workflow.guide", "arguments": {"phase": "post-plan-creation"}}}' | \
+  node ./mcp-server.mjs
+```
+
+## Future Enhancements
+
+While this implementation provides **stateless guidance** (Option 1 from Guff's analysis), the `GuidedMCPResponse<T>` interface in `guided-response.ts` shows how responses could be enhanced with workflow tracking in the future if needed.
+
+This would enable:
+
+- **Contextual prompts**: AI-generated suggestions based on current state
+- **Decision points**: Branching workflows with multiple options
+- **Progress tracking**: History of completed steps
+
+However, these enhancements are **not implemented** in this phase to maintain architectural simplicity.
+
+## Design Rationale
+
+Per Guff's backlog analysis:
+
+> **The Core Value Proposition:** "Agents waste tokens and time figuring out what to do next. I want the MCP to guide them through correct workflows."
+
+This implementation:
+
+✅ **Separates concerns** - Deterministic data ≠ guidance  
+✅ **Opt-in guidance** - Call `workflow.guide` when needed  
+✅ **Minimal architectural impact** - No changes to existing tools  
+✅ **Deterministic** - Same phase always returns same guidance  
+✅ **Testable** - Comprehensive test coverage
+
+## Related Documentation
+
+- [Merge Weave Workflow](./MERGE_WEAVE_USAGE_GUIDE.md) - Referenced in guidance
+- [MCP CLI Parity](./MCP-CLI-PARITY.md) - MCP tool alignment
+- [Terms](./TERMS.md) - Canonical terminology

package/docs/adr/ADR-000-product-naming-and-branding.md

@@ -0,0 +1,115 @@
+# ADR-000: Product Naming and Branding (LexRunner + Lex)
+
+**Date:** November 6, 2025
+**Status:** Accepted
+**Author:** Go-to-Market Initiative
+
+---
+
+## Context
+
+The lexrunner project serves two audiences with distinct personas:
+
+1. **Paid Product (Proprietary):** Enterprise teams using the merge-weave CLI and merge pyramid orchestration.
+2. **Open Source Core (MIT):** Community developers building on architectural policy, frames, and episodic memory.
+
+The current naming conflates both, making it unclear which product is which and which license applies to each component.
+
+---
+
+## Decision
+
+### Product Names
+
+| Product       | Brand       | Repo                    | License     | Notes                                       |
+| ------------- | ----------- | ----------------------- | ----------- | ------------------------------------------- |
+| **LexRunner** | `LexRunner` | `Guffawaffle/LexRunner` | Proprietary | Paid product; merge-weave orchestration CLI |
+| **Lex**       | `Lex`       | `Guffawaffle/lex`       | MIT         | OSS core; frames, memory, policy, atlas     |
+
+### CLI Name
+
+- **CLI binary/command:** `lex-pr` remains unchanged (backward compatible).
+- **Tagline:** `lex-pr` (powered by LexRunner).
+- **Package name (npm, future):** `@guffawaffle/lexrunner` (for the paid product).
+
+### Repository Names
+
+- `Guffawaffle/LexRunner` → Renamed from `lexrunner` to align with branding.
+- `Guffawaffle/lex` → Stays as-is (MIT badge + README).
+
+### Release Tag Prefix
+
+- **Canonical tag prefix (starting now):** `lexrunner-v*` (e.g., `lexrunner-v0.1.0`, `lexrunner-v1.0.0`).
+- **CI/CD:** `.github/workflows/release.yml` triggers on `lexrunner-v*` tags.
+- **Note:** Old `v*` tags may exist; new releases use `lexrunner-v*` exclusively.
+
+---
+
+## Rationale
+
+1. **Clarity:** Distinguishes the paid orchestration layer (LexRunner) from the OSS foundations (Lex).
+2. **Portability:** Lex can be adopted independently; LexRunner is built on top of Lex.
+3. **Legal/Licensing:** Clear separation eases compliance and customer communication.
+4. **Brand Consistency:** "LexRunner" projects a premium, purpose-built identity; "Lex" projects open-source accessibility.
+5. **Backward Compatibility:** CLI name (`lex-pr`) does not change; existing scripts continue to work.
+
+---
+
+## Implications
+
+### Immediate (Phase 1)
+
+1. Update `lexrunner/README.md` with LexRunner branding.
+2. Update `lex/README.md` with Lex (MIT) branding and cross-reference.
+3. Add badges: "Proprietary • Paid" (LexRunner), "MIT • OSS" (Lex).
+4. Create `.github/workflows/release.yml` trigger on `lexrunner-v*` tags.
+
+### Near-Term (Phase 2)
+
+1. Update issue templates to use LexRunner/Lex where appropriate.
+2. Update docs and references across both repos.
+3. Plan first release tag: `lexrunner-v0.1.0`.
+
+### Future (Phase 3)
+
+1. npm package `@guffawaffle/lexrunner` (currently undefined; may remain as-is).
+2. Branding assets: logos, website, marketing materials.
+3. User documentation split (paid vs. OSS guides).
+
+---
+
+## Consequences
+
+- **Pros:**
+  - Clear, memorable branding for both products.
+  - Easy to communicate ("LexRunner for enterprise orchestration; Lex for policy foundations").
+  - Supports future monetization and licensing strategies.
+
+- **Cons:**
+  - Documentation needs updating across both repos.
+  - Users unfamiliar with the split may initially be confused.
+  - Tag migration from `v*` to `lexrunner-v*` is a one-time maintenance task.
+
+---
+
+## Alternatives Considered
+
+1. **Rename the CLI to `lexrunner-pr`:** Rejected; breaks backward compatibility and adds verbosity.
+2. **Rename repos to `lexrunner` and `lex-core`:** Rejected; GitHub repos stable; branding works via docs.
+3. **Merge Lex into lexrunner:** Rejected; compromises MIT license and OSS adoption story.
+
+---
+
+## References
+
+- `Guffawaffle/LexRunner` (https://github.com/Guffawaffle/LexRunner)
+- `Guffawaffle/lex` (https://github.com/Guffawaffle/lex)
+- `docs/TERMS.md` (canonical terminology)
+- `.smartergpt/intent.md` (workspace profile example)
+
+---
+
+## Sign-Off
+
+Accepted by: Go-to-Market Team
+Date: November 6, 2025

package/docs/adr/ADR-001-plan-json-frozen-input.md

@@ -0,0 +1,68 @@
+# ADR-001: Plan.json as Frozen Runtime Input
+
+**Status:** Accepted
+**Date:** 2025-11-25
+**Authors:** lexrunner team
+
+---
+
+## Context
+
+The runner needs to compute dependency order, execute gates, and merge PRs. The question: what should its input be?
+
+Options considered:
+
+1. **Live GitHub API** — Query PRs in real-time
+2. **Issue descriptions** — Parse from GitHub Issues
+3. **Frozen file** — Single JSON file that captures all needed state
+
+---
+
+## Decision
+
+We adopt `plan.json` as the **single frozen runtime input** at integration time.
+
+### Scope Clarification
+
+- **Integration-time (Runner):** `plan.json` is the **only** input the runner reads to compute order and merges.
+- **Development-time (Issues & Coding):** Implementation agents (humans/Copilot) use Issue descriptions, specs, and project principles to do the work. The runner does not constrain this phase.
+
+### Runner Contract
+
+```typescript
+// Runner reads only plan.json
+const plan = loadPlan("./plan.json");
+
+// Runner never reads:
+// - GitHub Issues directly
+// - .smartergpt/ at runtime
+// - Environment variables for logic (only for credentials)
+```
+
+### Plan Generation (Out of Scope)
+
+A separate **plan generator** tool/workflow transforms Issues/PR graph into `plan.json`. This is explicitly out of scope for the runner.
+
+---
+
+## Consequences
+
+### Positive
+
+- **Deterministic** — Same `plan.json` → same decisions (modulo external state)
+- **Replayable** — Can re-run with same input for debugging
+- **Auditable** — Plan is a durable artifact
+- **Decoupled** — Plan generation can evolve independently
+
+### Negative
+
+- **Two-step process** — Generate plan, then run
+- **Staleness risk** — Plan can diverge from live PR state
+- **Not real-time** — Changes to PRs require re-generation
+
+---
+
+## References
+
+- `/AGENTS.md` — Section 2: Runner Contract
+- `/docs/TERMS.md` — Plan definition

package/docs/adr/ADR-002-two-track-separation.md

@@ -0,0 +1,84 @@
+# ADR-002: Two-Track Separation
+
+**Status:** Accepted
+**Date:** 2025-11-25
+**Authors:** lexrunner team
+
+---
+
+## Context
+
+The runner needs to be packageable and distributable. At the same time, users need a place to store local state, profiles, and deliverables. The question: how do we separate these concerns?
+
+---
+
+## Decision
+
+We adopt **two-track separation**:
+
+### Track A: Core Runner
+
+Location: Repository root (`src/**`, CLI, MCP adapter, packaging, CI)
+
+Characteristics:
+
+- **Stateless** — Never stores user/work artifacts
+- **Packageable** — Can be distributed via npm
+- **Runtime-independent** — Never reads `.smartergpt/` at runtime
+
+Contents:
+
+- `src/` — TypeScript source
+- `dist/` — Compiled output
+- `package.json` — Dependencies
+- CI recipes and packaging
+
+### Track B: Portable Workspace
+
+Location: `.smartergpt/**`
+
+Characteristics:
+
+- **Per-workspace** — Gitignored, not distributed
+- **Replaceable** — Can be swapped between projects
+- **Example profile** — The repo contains a canonical example
+
+Tracked files:
+
+- `intent.md`
+- `scope.yml`
+- `deps.yml`
+- `gates.yml`
+- `stack.yml`
+- `pull-request-template.md`
+
+Ignored directories:
+
+- `.smartergpt/runner/`
+- `.smartergpt/cache/`
+- `.smartergpt/deliverables/`
+
+**Deliverables are posted as PR comments, not committed.**
+
+---
+
+## Consequences
+
+### Positive
+
+- **Repos can adopt lexrunner** without inheriting personal workspace
+- **Workspace travels across repos** without changing the runner
+- **Clean distribution** — npm package contains only code
+- **Portable profiles** — Users can share profiles as templates
+
+### Negative
+
+- **Two directories to understand** — Learning curve
+- **No default writable workspace** — Users must initialize/provide a profile (typically `.smartergpt.local/`; `.smartergpt/` is a tracked example)
+
+---
+
+## References
+
+- `/AGENTS.md` — Section 6: Two-Track Separation
+- `/docs/TERMS.md` — Workspace profile definition