opkg 0.11.2 → 0.11.3

package/specs/platforms/map-pipeline.md

@@ -1,801 +0,0 @@
-# Map Pipeline Documentation
-
-## Overview
-
-The **Map Pipeline** is a MongoDB aggregation-inspired transformation system that enables powerful document-level field transformations during package installation. It provides a declarative, composable approach to transforming package content as files flow from source to target.
-
-**Design Philosophy:**
-- **MongoDB-aligned:** Uses `$operation` syntax matching MongoDB aggregation operators
-- **Sequential execution:** Operations applied in order, each receiving the result of the previous
-- **Document-level:** Transforms entire documents (frontmatter, JSON, YAML, TOML)
-- **Context-aware:** Access file metadata through `$$` context variables
-- **Type-safe:** Full TypeScript support with comprehensive validation
-
-## Implementation
-
-**Initial Commit:** `e464927d2f5d2480c439a83c5cfc2e04053ac22d`  
-**MongoDB Alignment:** January 9, 2026 (Breaking Change)
-
-**Core files:**
-- `src/core/flows/map-pipeline/index.ts` - Main pipeline executor
-- `src/core/flows/map-pipeline/types.ts` - TypeScript type definitions
-- `src/core/flows/map-pipeline/operations/` - Individual operation implementations
-- `schemas/map-pipeline-v1.json` - JSON Schema validation
-- `tests/flows/map-pipeline.test.ts` - Comprehensive test suite
-
-## Architecture
-
-### Pipeline Structure
-
-```jsonc
-{
-  "map": [
-    { "$operation1": config1 },
-    { "$operation2": config2 },
-    { "$operation3": config3 }
-  ]
-}
-```
-
-### Execution Flow
-
-```
-Input Document
-    ↓
-$operation1 (receives input, produces result1)
-    ↓
-$operation2 (receives result1, produces result2)
-    ↓
-$operation3 (receives result2, produces result3)
-    ↓
-Output Document
-```
-
-**Key characteristics:**
-- Operations execute sequentially
-- Each operation receives full document
-- Operations are isolated (no side effects)
-- Document is deep-cloned at start to prevent mutations
-- All operations use `$` prefix (MongoDB convention)
-
-### Context Variables
-
-Every operation has access to file context:
-
-```typescript
-interface MapContext {
-  filename: string;  // Filename without extension: "code-reviewer"
-  dirname: string;   // Parent directory: "agents"
-  path: string;      // Relative path: "agents/code-reviewer.md"
-  ext: string;       // Extension with dot: ".md"
-}
-```
-
-**Usage in operations:**
-- Prefix with `$$` to inject: `$$filename`, `$$dirname`, `$$path`, `$$ext`
-- Escape literal `$$`: use `\$$` to get literal string `$$filename`
-
-## Seven Core Operations
-
-### 1. `$set` - Set Field Values
-
-Sets one or more fields with literal values or context variables.
-
-**Examples:**
-
-```jsonc
-// Set from context
-{ "$set": { "name": "$$filename" } }
-
-// Set multiple fields
-{ "$set": { 
-  "name": "$$filename",
-  "version": "1.0.0",
-  "status": "active"
-}}
-
-// Nested fields (dot notation)
-{ "$set": { "config.model": "sonnet" } }
-
-// Escaped literal
-{ "$set": { "template": "\\$$placeholder" } }
-// Results in: { "template": "$$placeholder" }
-```
-
-**MongoDB equivalent:** `$set` (exact match)
-
-**See:** `src/core/flows/map-pipeline/operations/set.ts`
-
----
-
-### 2. `$rename` - Rename Fields
-
-Renames fields with support for wildcards.
-
-**Examples:**
-
-```jsonc
-// Simple rename
-{ "$rename": { "mcp": "mcpServers" } }
-
-// Multiple renames
-{ "$rename": { 
-  "old1": "new1",
-  "old2": "new2"
-}}
-
-// Wildcard rename (preserves structure)
-{ "$rename": { "mcp.*": "mcpServers.*" } }
-```
-
-**Wildcard example:**
-```json
-// Input
-{ "mcp": { "server1": {}, "server2": {} } }
-
-// After { "$rename": { "mcp.*": "mcpServers.*" } }
-{ "mcpServers": { "server1": {}, "server2": {} } }
-```
-
-**MongoDB equivalent:** `$rename` (exact match)
-
-**See:** `src/core/flows/map-pipeline/operations/rename.ts`
-
----
-
-### 3. `$unset` - Remove Fields
-
-Removes one or more fields from the document.
-
-**Examples:**
-
-```jsonc
-// Single field
-{ "$unset": "deprecated" }
-
-// Multiple fields
-{ "$unset": ["legacy", "temp", "internal"] }
-
-// Nested field
-{ "$unset": "config.debug.verbose" }
-```
-
-**MongoDB equivalent:** `$unset` (exact match)
-
-**See:** `src/core/flows/map-pipeline/operations/unset.ts`
-
----
-
-### 4. `$switch` - Pattern-Based Value Replacement
-
-Pattern matching with value replacement (like switch statement).
-
-**Examples:**
-
-```jsonc
-// String pattern matching (glob syntax)
-{
-  "$switch": {
-    "field": "model",
-    "cases": [
-      { "pattern": "anthropic/claude-sonnet-*", "value": "sonnet" },
-      { "pattern": "anthropic/claude-opus-*", "value": "opus" }
-    ],
-    "default": "inherit"
-  }
-}
-
-// Object pattern matching
-{
-  "$switch": {
-    "field": "permission",
-    "cases": [
-      { 
-        "pattern": { "edit": "deny", "bash": "deny" }, 
-        "value": "restricted" 
-      },
-      { 
-        "pattern": { "*": "deny" }, 
-        "value": "ignore" 
-      }
-    ]
-  }
-}
-```
-
-**Pattern matching rules:**
-- **String patterns:** Use glob syntax (`*`, `?`)
-- **Object patterns:** Match object shape (all keys must match)
-- **First match wins:** Stops checking after first successful match
-- **Default optional:** If provided, used when no patterns match
-
-**MongoDB equivalent:** Similar to `$switch` (aggregation expression)
-
-**See:** `src/core/flows/map-pipeline/operations/switch.ts`
-
----
-
-### 5. `$pipeline` - Multi-Step Field Transformation
-
-Transforms a field through a pipeline of operations (MongoDB-aligned syntax).
-
-**Available operations:**
-
-| Operation | Purpose | Input → Output |
-|-----------|---------|----------------|
-| `{ "$filter": { "match": { "value": X } } }` | Filter by value | Object → Object (filtered) |
-| `{ "$filter": { "match": { "key": X } } }` | Filter by key | Object → Object (filtered) |
-| `{ "$objectToArray": { "extract": "keys" } }` | Extract keys | Object → Array |
-| `{ "$objectToArray": { "extract": "values" } }` | Extract values | Object → Array |
-| `{ "$objectToArray": { "extract": "entries" } }` | To entries | Object → Array of [key, value] |
-| `{ "$map": { "each": "capitalize" } }` | Transform elements | Array → Array (transformed) |
-| `{ "$reduce": { "type": "join", "separator": "," } }` | Join to string | Array → String |
-| `{ "$reduce": { "type": "split", "separator": "," } }` | Split to array | String → Array |
-| `{ "$arrayToObject": { "value": X } }` | Array to object | Array → Object |
-| `{ "$replace": { "pattern": "...", "with": "..." } }` | String replacement | String → String |
-
-**Example:**
-
-```jsonc
-// Convert tools object to CSV string
-{
-  "$pipeline": {
-    "field": "tools",
-    "operations": [
-      { "$filter": { "match": { "value": true } } },   // Keep only true values
-      { "$objectToArray": { "extract": "keys" } },     // Extract keys
-      { "$map": { "each": "capitalize" } },            // Capitalize each
-      { "$reduce": { "type": "join", "separator": ", " } } // Join to string
-    ]
-  }
-}
-```
-
-**Input:**
-```yaml
-tools:
-  write: false
-  edit: false
-  bash: true
-  read: true
-```
-
-**Output:**
-```yaml
-tools: Bash, Read
-```
-
-**Empty result handling:**  
-If transformation results in empty string or empty array, the field is automatically removed.
-
-**MongoDB alignment:**
-- `$filter` - matches MongoDB's `$filter`
-- `$objectToArray` - matches MongoDB's `$objectToArray`
-- `$map` - matches MongoDB's `$map`
-- `$reduce` - inspired by MongoDB's `$reduce`
-- `$arrayToObject` - matches MongoDB's `$arrayToObject`
-- `$replace` - similar to MongoDB's `$replaceOne`/`$replaceAll`
-
-**See:** `src/core/flows/map-pipeline/operations/transform.ts`
-
----
-
-### 6. `$copy` - Copy Field with Optional Transformation
-
-Copies a field to new location with optional pattern-based transformation.
-
-**Examples:**
-
-```jsonc
-// Simple copy
-{
-  "$copy": {
-    "from": "config",
-    "to": "settings"
-  }
-}
-
-// Copy with transformation
-{
-  "$copy": {
-    "from": "permission",
-    "to": "permissionMode",
-    "transform": {
-      "cases": [
-        { "pattern": { "edit": "deny", "bash": "deny" }, "value": "plan" },
-        { "pattern": { "*": "deny" }, "value": "ignore" }
-      ],
-      "default": "default"
-    }
-  }
-}
-```
-
-**MongoDB equivalent:** Custom operation (not in MongoDB)
-
-**See:** `src/core/flows/map-pipeline/operations/copy.ts`
-
----
-
-### 7. `$pipe` - Apply Transform Registry Operations
-
-Applies external transforms from the transform registry within the map pipeline.
-Enables format conversions and validations to be interleaved with schema transformations.
-
-**Examples:**
-
-```jsonc
-// Convert to TOML at end of pipeline
-{
-  "$pipe": ["json-to-toml"]
-}
-
-// Parse TOML at start of pipeline
-{
-  "$pipe": ["toml-to-json"]
-}
-
-// Chain multiple transforms
-{
-  "$pipe": ["filter-comments", "jsonc", "validate-schema"]
-}
-```
-
-**Complete flow example:**
-
-```jsonc
-{
-  "from": ["mcp.jsonc", "mcp.json"],
-  "to": ".codex/mcp-servers.toml",
-  "map": [
-    // Schema transformations
-    { "$rename": { "mcp": "mcp_servers" } },
-    {
-      "$pipeline": {
-        "field": "mcp_servers.*.headers.Authorization",
-        "operations": [
-          { "$extract": { 
-              "pattern": "^Bearer \\$\\{env:([A-Z_]+)\\}$",
-              "group": 1
-          }}
-        ]
-      }
-    },
-    // More transformations...
-    
-    // Convert to TOML at the end
-    { "$pipe": ["json-to-toml"] }
-  ]
-}
-```
-
-**Use cases:**
-- Format conversion: `json-to-toml`, `toml-to-json`, `yaml-to-json`
-- Pre-processing: `filter-comments`, `jsonc` (parse JSONC)
-- Post-processing: `validate-schema`, `format-toml`
-- Mid-pipeline validation: Apply validation between transformation steps
-
-**MongoDB equivalent:** Custom operation (bridges map pipeline with external systems)
-
-**See:** `src/core/flows/map-pipeline/operations/pipe.ts`
-
-## Complete Examples
-
-### Example 1: Agent Name Injection
-
-```jsonc
-{
-  "from": "agents/**/*.md",
-  "to": ".claude/agents/**/*.md",
-  "map": [
-    { "$set": { "name": "$$filename" } }
-  ]
-}
-```
-
-**File:** `agents/code-reviewer.md`  
-**Result:** Frontmatter gets `name: code-reviewer`
-
----
-
-### Example 2: Model Simplification
-
-```jsonc
-{
-  "from": "agents/**/*.md",
-  "to": ".claude/agents/**/*.md",
-  "map": [
-    { "$set": { "name": "$$filename" } },
-    {
-      "$switch": {
-        "field": "model",
-        "cases": [
-          { "pattern": "anthropic/claude-sonnet-*", "value": "sonnet" },
-          { "pattern": "anthropic/claude-opus-*", "value": "opus" }
-        ],
-        "default": "inherit"
-      }
-    }
-  ]
-}
-```
-
-**Transforms:**  
-`anthropic/claude-sonnet-4-20250514` → `sonnet`
-
----
-
-### Example 3: Complete Claude Agent Transformation
-
-```jsonc
-{
-  "from": "agents/**/*.md",
-  "to": ".claude/agents/**/*.md",
-  "map": [
-    // 1. Inject filename as name
-    { "$set": { "name": "$$filename" } },
-    
-    // 2. Simplify model ID
-    {
-      "$pipeline": {
-        "field": "model",
-        "operations": [
-          { "$replace": { "pattern": "^anthropic/", "with": "" } },
-          { "$replace": { "pattern": "(-[0-9]+)\\.([0-9]+)", "with": "$1-$2", "flags": "g" } }
-        ]
-      }
-    },
-    
-    // 3. Transform tools object to CSV
-    {
-      "$pipeline": {
-        "field": "tools",
-        "operations": [
-          { "$filter": { "match": { "value": true } } },
-          { "$objectToArray": { "extract": "keys" } },
-          { "$map": { "each": "capitalize" } },
-          { "$reduce": { "type": "join", "separator": ", " } }
-        ]
-      }
-    },
-    
-    // 4. Transform permission to mode
-    {
-      "$copy": {
-        "from": "permission",
-        "to": "permissionMode",
-        "transform": {
-          "cases": [
-            { "pattern": { "edit": "deny", "bash": "deny" }, "value": "plan" }
-          ],
-          "default": "default"
-        }
-      }
-    },
-    
-    // 5. Remove original permission field
-    { "$unset": "permission" }
-  ]
-}
-```
-
-**Input:**
-```yaml
----
-model: anthropic/claude-sonnet-4-20250514
-tools:
-  write: false
-  bash: true
-  read: true
-permission:
-  edit: deny
-  bash: deny
----
-# Code Reviewer
-```
-
-**Output:**
-```yaml
----
-name: code-reviewer
-model: claude-sonnet-4-20250514
-tools: Bash, Read
-permissionMode: plan
----
-# Code Reviewer
-```
-
-## `$pipeline` Operation Details
-
-### `$filter` - Filter Object Entries
-
-```jsonc
-// Filter by value
-{ "$filter": { "match": { "value": true } } }
-
-// Filter by key
-{ "$filter": { "match": { "key": "enabled" } } }
-```
-
-**MongoDB equivalent:** `$filter` operator
-
----
-
-### `$objectToArray` - Convert Object to Array
-
-```jsonc
-// Extract keys only
-{ "$objectToArray": { "extract": "keys" } }
-// { a: 1, b: 2 } → ["a", "b"]
-
-// Extract values only
-{ "$objectToArray": { "extract": "values" } }
-// { a: 1, b: 2 } → [1, 2]
-
-// Extract entries (default)
-{ "$objectToArray": { "extract": "entries" } }
-// { a: 1, b: 2 } → [["a", 1], ["b", 2]]
-
-// Shorthand for entries
-{ "$objectToArray": true }
-```
-
-**MongoDB equivalent:** `$objectToArray` (exact match for entries mode)
-
----
-
-### `$arrayToObject` - Convert Array to Object
-
-```jsonc
-// Array of strings to object with fixed value
-{ "$arrayToObject": { "value": true } }
-// ["bash", "read"] → { bash: true, read: true }
-
-// With context variable
-{ "$arrayToObject": { "value": "$$filename" } }
-// ["tool1", "tool2"] → { tool1: "code-reviewer", tool2: "code-reviewer" }
-```
-
-**MongoDB equivalent:** `$arrayToObject` (similar, but MongoDB uses entries format)
-
----
-
-### `$map` - Transform Array Elements
-
-```jsonc
-{ "$map": { "each": "capitalize" } }
-// ["hello", "world"] → ["Hello", "World"]
-
-{ "$map": { "each": "uppercase" } }
-// ["hello", "world"] → ["HELLO", "WORLD"]
-
-{ "$map": { "each": "lowercase" } }
-// ["HELLO", "WORLD"] → ["hello", "world"]
-```
-
-**MongoDB equivalent:** `$map` (similar structure)
-
----
-
-### `$reduce` - Reduce Array to Single Value
-
-```jsonc
-// Join array to string
-{ "$reduce": { "type": "join", "separator": ", " } }
-// ["bash", "read"] → "bash, read"
-
-// Split string to array
-{ "$reduce": { "type": "split", "separator": ", " } }
-// "bash, read" → ["bash", "read"]
-
-// Sum numbers
-{ "$reduce": { "type": "sum" } }
-// [1, 2, 3] → 6
-
-// Count elements
-{ "$reduce": { "type": "count" } }
-// ["a", "b", "c"] → 3
-```
-
-**MongoDB equivalent:** `$reduce` (inspired by MongoDB's `$reduce`, but simplified)
-
----
-
-### `$replace` - String Replacement with Regex
-
-```jsonc
-// Simple replacement
-{ "$replace": { "pattern": "^anthropic/", "with": "" } }
-// "anthropic/claude-sonnet" → "claude-sonnet"
-
-// With capture groups
-{ "$replace": { "pattern": "(-[0-9]+)\\.([0-9]+)", "with": "$1-$2", "flags": "g" } }
-// "claude-4.5" → "claude-4-5"
-```
-
-**MongoDB equivalent:** `$replaceOne` / `$replaceAll`
-
-## Validation
-
-Pipelines are validated against `schemas/map-pipeline-v1.json` before execution.
-
-**Valid:**
-```jsonc
-[
-  { "$set": { "name": "$$filename" } },
-  { "$rename": { "old": "new" } }
-]
-```
-
-**Invalid - multiple operations per object:**
-```jsonc
-[
-  { 
-    "$set": { "name": "$$filename" },
-    "$rename": { "old": "new" }  // Error!
-  }
-]
-```
-
-**Invalid - unknown operation:**
-```jsonc
-[
-  { "$unknown": { "field": "value" } }  // Error!
-]
-```
-
-## Testing
-
-Comprehensive test suite: `tests/flows/map-pipeline.test.ts`
-
-**Run tests:**
-```bash
-npm test -- map-pipeline
-```
-
-**Test coverage:**
-- All 6 operations
-- Context variable resolution
-- Nested field access
-- Wildcard patterns
-- Pattern matching
-- Empty result handling
-- Error cases
-- Integration with flow execution
-
-## Best Practices
-
-### 1. Keep Pipelines Simple
-
-```jsonc
-// Good
-{ "$rename": { "mcp": "mcpServers" } }
-
-// Avoid unnecessary complexity
-[
-  { "$copy": { "from": "mcp", "to": "temp" } },
-  { "$unset": "mcp" },
-  { "$rename": { "temp": "mcpServers" } }
-]
-```
-
-### 2. Order Operations Logically
-
-```jsonc
-[
-  { "$set": { ... } },       // 1. Add fields
-  { "$switch": { ... } },    // 2. Transform values
-  { "$pipeline": { ... } },  // 3. Complex transforms
-  { "$copy": { ... } },      // 4. Derive fields
-  { "$unset": [...] }        // 5. Cleanup
-]
-```
-
-### 3. Leverage Context Variables
-
-```jsonc
-{ "$set": { 
-  "name": "$$filename",
-  "source": "$$path",
-  "category": "$$dirname"
-}}
-```
-
-### 4. Document Complex Pipelines
-
-```jsonc
-{
-  "map": [
-    // Extract filename as agent name
-    { "$set": { "name": "$$filename" } },
-    
-    // Convert full model ID to short name for Claude compatibility
-    {
-      "$pipeline": {
-        "field": "model",
-        "operations": [
-          { "$replace": { "pattern": "^anthropic/", "with": "" } }
-        ]
-      }
-    }
-  ]
-}
-```
-
-### 5. Test Incrementally
-
-```bash
-# Preview transformations
-opkg install @user/package --dry-run
-
-# Verify specific file
-cat .claude/agents/code-reviewer.md
-```
-
-## Performance
-
-**Optimized execution:**
-- Document cloned once at start (prevents mutations)
-- All operations execute in-memory (no I/O)
-- Operations are O(1) to O(n) where n = number of fields
-- Pattern matching uses optimized glob library
-- Pipeline operations are O(operations × elements)
-
-**Recommendations:**
-- Keep pipelines under 10 operations
-- Combine multiple `$set` operations into one
-- Use `$copy` with transform instead of separate operations
-
-## Related Documentation
-
-- **Breaking Changes:** `BREAKING_CHANGES.md` - MongoDB alignment migration
-- **MongoDB Alignment:** `MONGODB_ALIGNMENT.md` - Complete migration guide
-- **Examples:** `MONGODB_ALIGNMENT_EXAMPLE.md` - Step-by-step examples
-- **Type Definitions:** `src/core/flows/map-pipeline/types.ts` - TypeScript types
-- **JSON Schema:** `schemas/map-pipeline-v1.json` - Validation schema
-- **Tests:** `tests/flows/map-pipeline.test.ts` - Test suite
-- **Flow Documentation:** `specs/platforms/flows.md` - Flow system overview
-
-## MongoDB Alignment (Breaking Change)
-
-**Date:** January 9, 2026
-
-The pipeline syntax has been updated to align with MongoDB aggregation conventions:
-
-### What Changed
-
-1. **`$transform` renamed to `$pipeline`**
-2. **`steps` renamed to `operations`**
-3. **All sub-operations now use `$` prefix:**
-   - `filter` → `$filter`
-   - `keys`/`values`/`entries` → `$objectToArray`
-   - `map` → `$map`
-   - `join`/`split` → `$reduce`
-   - `arrayToObject` → `$arrayToObject`
-   - `replace` → `$replace`
-
-### Migration Required
-
-See `BREAKING_CHANGES.md` for complete migration guide.
-
-## Summary
-
-The Map Pipeline provides a powerful, MongoDB-aligned transformation system for document-level operations:
-
-✓ **7 core operations:** set, rename, unset, switch, pipeline, copy, pipe  
-✓ **Context-aware:** Access file metadata with `$$` variables  
-✓ **Sequential execution:** Predictable, composable transformations  
-✓ **Type-safe:** Full TypeScript support  
-✓ **Validated:** JSON Schema enforcement  
-✓ **Tested:** Comprehensive test coverage  
-✓ **Performant:** In-memory operations with minimal overhead  
-✓ **MongoDB-aligned:** Familiar operators for MongoDB developers  
-✓ **Transform integration:** `$pipe` bridges map pipeline with transform registry
-
-For complete examples and usage patterns, see the related documentation above.
-
----
-
-**Map Pipeline v1.0** - Implemented in commit `e464927d2f5d2480c439a83c5cfc2e04053ac22d`  
-**MongoDB Alignment** - Updated January 9, 2026 (Breaking Change)  
-**`$pipe` Operation** - Added January 2026