@mmerterden/multi-agent-pipeline 20.2.0 → 20.3.0

package/pipeline/skills/shared/external/context-compression/SKILL.md

@@ -1,266 +1,134 @@
 ---
 name: context-compression
-description: "When agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. Use when an agent session has grown too long to keep in full and history has to be compressed."
-risk: unknown
-source: community
+description: "Keeping long LLM agent sessions on track: what a summary must preserve, incremental structured summaries, state in files, compact versus restart versus subagent. Use when an agent session outgrows its context."
+metadata:
+  risk: safe
+  source: multi-agent-pipeline
+  date_added: "2026-09-25"
 ---
 
-# Context Compression Strategies
+# Context Compression for Agent Sessions
 
-When agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. The correct optimization target is tokens per task: total tokens consumed to complete a task, including re-fetching costs when compression loses critical information.
+A long agent session fills its context window with tool output, file contents
+and dead ends. At some point the history has to shrink. The goal is not the
+smallest summary; it is finishing the task with the fewest total tokens. A
+summary that drops a file path or an error message costs more later, when the
+agent re-reads, re-runs or repeats the mistake.
 
-## When to Use
-Activate this skill when:
-- Agent sessions exceed context window limits
-- Codebases exceed context windows (5M+ token systems)
-- Designing conversation summarization strategies
-- Debugging cases where agents "forget" what files they modified
-- Building evaluation frameworks for compression quality
+## What must survive a compression
 
-## Core Concepts
+Treat these as mandatory fields. If a summary cannot fill one, it says so
+explicitly rather than leaving it out.
 
-Context compression trades token savings against information loss. Three production-ready approaches exist:
+| Field | Why it matters | Example |
+|-------|----------------|---------|
+| Goal and acceptance criteria | Without it the agent optimises the last message instead of the task | "Add CSV export to the orders page; export must respect active filters; tests green" |
+| Decisions and their reasons | Prevents re-litigating settled questions | "Stream rows instead of building the file in memory: exports reach 400k rows" |
+| File trail | The most commonly lost item; the agent must know what it touched | "Created `export/csv_writer.py`; modified `orders/views.py` (new `export` action); read but unchanged `orders/filters.py`" |
+| Exact errors still open | Paraphrased errors cannot be searched or matched | The literal failing assertion and test id |
+| Ruled-out approaches | Stops loops | "Tried pandas `to_csv`: 3x memory, rejected" |
+| Open tasks, in order | Tells the next turn what to do first | Checklist with the current item marked |
+| Environment facts | Hard-won and cheap to store | Branch name, how to run the tests, a flaky test to ignore |
 
-1. **Anchored Iterative Summarization**: Maintain structured, persistent summaries with explicit sections for session intent, file modifications, decisions, and next steps. When compression triggers, summarize only the newly-truncated span and merge with the existing summary. Structure forces preservation by dedicating sections to specific information types.
+What can go: full file contents that can be re-read, verbose tool output once
+its conclusion is recorded, superseded plans, conversational back-and-forth.
 
-2. **Opaque Compression**: Produce compressed representations optimized for reconstruction fidelity. Achieves highest compression ratios (99%+) but sacrifices interpretability. Cannot verify what was preserved.
+## Structured, incremental summaries
 
-3. **Regenerative Full Summary**: Generate detailed structured summaries on each compression. Produces readable output but may lose details across repeated compression cycles due to full regeneration rather than incremental merging.
+Free-form "summarise the conversation so far" prompts drift: each pass rewrites
+from scratch and quietly loses details that seemed minor. Two habits prevent
+that.
 
-The critical insight: structure forces preservation. Dedicated sections act as checklists that the summarizer must populate, preventing silent information drift.
-
-## Detailed Topics
-
-### Why Tokens-Per-Task Matters
-
-Traditional compression metrics target tokens-per-request. This is the wrong optimization. When compression loses critical details like file paths or error messages, the agent must re-fetch information, re-explore approaches, and waste tokens recovering context.
-
-The right metric is tokens-per-task: total tokens consumed from task start to completion. A compression strategy saving 0.5% more tokens but causing 20% more re-fetching costs more overall.
-
-### The Artifact Trail Problem
-
-Artifact trail integrity is the weakest dimension across all compression methods, scoring 2.2-2.5 out of 5.0 in evaluations. Even structured summarization with explicit file sections struggles to maintain complete file tracking across long sessions.
-
-Coding agents need to know:
-- Which files were created
-- Which files were modified and what changed
-- Which files were read but not changed
-- Function names, variable names, error messages
-
-This problem likely requires specialized handling beyond general summarization: a separate artifact index or explicit file-state tracking in agent scaffolding.
-
-### Structured Summary Sections
-
-Effective structured summaries include explicit sections:
+**A fixed template.** Give the summariser named sections matching the table
+above. Empty sections are visible gaps, which is the point.
 
 ```markdown
-## Session Intent
-[What the user is trying to accomplish]
-
-## Files Modified
-- auth.controller.ts: Fixed JWT token generation
-- config/redis.ts: Updated connection pooling
-- tests/auth.test.ts: Added mock setup for new config
-
-## Decisions Made
-- Using Redis connection pool instead of per-request connections
-- Retry logic with exponential backoff for transient failures
-
-## Current State
-- 14 tests passing, 2 failing
-- Remaining: mock setup for session service tests
-
-## Next Steps
-1. Fix remaining test failures
-2. Run full test suite
-3. Update documentation
+## Goal
+## Acceptance criteria
+## Decisions (with reason)
+## Files
+- created:
+- modified (what changed):
+- read only:
+## Open errors (verbatim)
+## Ruled out
+## Next steps
+## Environment
 ```
 
-This structure prevents silent loss of file paths or decisions because each section must be explicitly addressed.
-
-### Compression Trigger Strategies
-
-When to trigger compression matters as much as how to compress:
-
-| Strategy | Trigger Point | Trade-off |
-|----------|---------------|-----------|
-| Fixed threshold | 70-80% context utilization | Simple but may compress too early |
-| Sliding window | Keep last N turns + summary | Predictable context size |
-| Importance-based | Compress low-relevance sections first | Complex but preserves signal |
-| Task-boundary | Compress at logical task completions | Clean summaries but unpredictable timing |
-
-The sliding window approach with structured summaries provides the best balance of predictability and quality for most coding agent use cases.
-
-### Probe-Based Evaluation
-
-Traditional metrics like ROUGE or embedding similarity fail to capture functional compression quality. A summary may score high on lexical overlap while missing the one file path the agent needs.
-
-Probe-based evaluation directly measures functional quality by asking questions after compression:
-
-| Probe Type | What It Tests | Example Question |
-|------------|---------------|------------------|
-| Recall | Factual retention | "What was the original error message?" |
-| Artifact | File tracking | "Which files have we modified?" |
-| Continuation | Task planning | "What should we do next?" |
-| Decision | Reasoning chain | "What did we decide about the Redis issue?" |
-
-If compression preserved the right information, the agent answers correctly. If not, it guesses or hallucinates.
-
-### Evaluation Dimensions
-
-Six dimensions capture compression quality for coding agents:
-
-1. **Accuracy**: Are technical details correct? File paths, function names, error codes.
-2. **Context Awareness**: Does the response reflect current conversation state?
-3. **Artifact Trail**: Does the agent know which files were read or modified?
-4. **Completeness**: Does the response address all parts of the question?
-5. **Continuity**: Can work continue without re-fetching information?
-6. **Instruction Following**: Does the response respect stated constraints?
-
-Accuracy shows the largest variation between compression methods (0.6 point gap). Artifact trail is universally weak (2.2-2.5 range).
-
-## Practical Guidance
-
-### Three-Phase Compression Workflow
-
-For large codebases or agent systems exceeding context windows, apply compression through three phases:
-
-1. **Research Phase**: Produce a research document from architecture diagrams, documentation, and key interfaces. Compress exploration into a structured analysis of components and dependencies. Output: single research document.
-
-2. **Planning Phase**: Convert research into implementation specification with function signatures, type definitions, and data flow. A 5M token codebase compresses to approximately 2,000 words of specification.
-
-3. **Implementation Phase**: Execute against the specification. Context remains focused on the spec rather than raw codebase exploration.
-
-### Using Example Artifacts as Seeds
-
-When provided with a manual migration example or reference PR, use it as a template to understand the target pattern. The example reveals constraints that static analysis cannot surface: which invariants must hold, which services break on changes, and what a clean migration looks like.
-
-This is particularly important when the agent cannot distinguish essential complexity (business requirements) from accidental complexity (legacy workarounds). The example artifact encodes that distinction.
-
-### Implementing Anchored Iterative Summarization
-
-1. Define explicit summary sections matching your agent's needs
-2. On first compression trigger, summarize truncated history into sections
-3. On subsequent compressions, summarize only new truncated content
-4. Merge new summary into existing sections rather than regenerating
-5. Track which information came from which compression cycle for debugging
-
-### When to Use Each Approach
-
-**Use anchored iterative summarization when:**
-- Sessions are long-running (100+ messages)
-- File tracking matters (coding, debugging)
-- You need to verify what was preserved
-
-**Use opaque compression when:**
-- Maximum token savings required
-- Sessions are relatively short
-- Re-fetching costs are low
-
-**Use regenerative summaries when:**
-- Summary interpretability is critical
-- Sessions have clear phase boundaries
-- Full context review is acceptable on each compression
-
-### Compression Ratio Considerations
-
-| Method | Compression Ratio | Quality Score | Trade-off |
-|--------|-------------------|---------------|-----------|
-| Anchored Iterative | 98.6% | 3.70 | Best quality, slightly less compression |
-| Regenerative | 98.7% | 3.44 | Good quality, moderate compression |
-| Opaque | 99.3% | 3.35 | Best compression, quality loss |
-
-The 0.7% additional tokens retained by structured summarization buys 0.35 quality points. For any task where re-fetching costs matter, this trade-off favors structured approaches.
-
-## Examples
-
-**Example 1: Debugging Session Compression**
-
-Original context (89,000 tokens, 178 messages):
-- 401 error on /api/auth/login endpoint
-- Traced through auth controller, middleware, session store
-- Found stale Redis connection
-- Fixed connection pooling, added retry logic
-- 14 tests passing, 2 failing
-
-Structured summary after compression:
-
-```markdown
-## Session Intent
-Debug 401 Unauthorized error on /api/auth/login despite valid credentials.
-
-## Root Cause
-Stale Redis connection in session store. JWT generated correctly but session could not be persisted.
-
-## Files Modified
-- auth.controller.ts: No changes (read only)
-- middleware/cors.ts: No changes (examined)
-- config/redis.ts: Fixed connection pooling configuration
-- services/session.service.ts: Added retry logic for transient failures
-- tests/auth.test.ts: Updated mock setup
-
-## Test Status
-14 passing, 2 failing (mock setup issues)
-
-## Next Steps
-1. Fix remaining test failures (mock session service)
-2. Run full test suite
-3. Deploy to staging
-```
-
-**Example 2: Probe Response Quality**
-
-After compression, asking "What was the original error?":
-
-Good response (structured summarization):
-> "The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users received this error with valid credentials. Root cause was stale Redis connection in session store."
-
-Poor response (aggressive compression):
-> "We were debugging an authentication issue. The login was failing. We fixed some configuration problems."
-
-The structured response preserves endpoint, error code, and root cause. The aggressive response loses all technical detail.
-
-## Guidelines
-
-1. Optimize for tokens-per-task, not tokens-per-request
-2. Use structured summaries with explicit sections for file tracking
-3. Trigger compression at 70-80% context utilization
-4. Implement incremental merging rather than full regeneration
-5. Test compression quality with probe-based evaluation
-6. Track artifact trail separately if file tracking is critical
-7. Accept slightly lower compression ratios for better quality retention
-8. Monitor re-fetching frequency as a compression quality signal
-
-## Integration
-
-This skill connects to several others in the collection:
-
-- context-degradation - Compression is a mitigation strategy for degradation
-- context-optimization - Compression is one optimization technique among many
-- evaluation - Probe-based evaluation applies to compression testing
-- memory-systems - Compression relates to scratchpad and summary memory patterns
-
-## References
-
-Internal reference:
-- Evaluation Framework Reference - Detailed probe types and scoring rubrics
-
-Related skills in this collection:
-- context-degradation - Understanding what compression prevents
-- context-optimization - Broader optimization strategies
-- evaluation - Building evaluation frameworks
-
-External resources:
-- Factory Research: Evaluating Context Compression for AI Agents (December 2025)
-- Research on LLM-as-judge evaluation methodology (Zheng et al., 2023)
-- Netflix Engineering: "The Infinite Software Crisis" - Three-phase workflow and context compression at scale (AI Summit 2025)
-
----
-
-## Skill Metadata
-
-**Created**: 2025-12-22
-**Last Updated**: 2025-12-26
-**Author**: Agent Skills for Context Engineering Contributors
-**Version**: 1.1.0
-
+**Merge, do not regenerate.** When compression triggers again, summarise only
+the span being dropped and merge it into the existing summary section by
+section. Earlier facts are carried forward unchanged unless the new span
+explicitly supersedes them. Regenerating the whole summary each time compounds
+loss across cycles.
+
+Keep verbatim anything that will be matched against later: identifiers, paths,
+command lines, error strings, version numbers. Summarise reasoning; never
+summarise literals.
+
+## Externalise instead of remembering
+
+The cheapest context is the one that lives on disk and is re-read on demand.
+
+- **Progress file.** A `NOTES.md` or task file the agent updates as it works:
+  plan, checklist, decisions. After compression or restart, reading one file
+  restores the working state.
+- **Git as a log.** Small commits with clear messages record what changed and
+  why. `git diff` and `git log` recover the file trail exactly, which no summary
+  can match.
+- **Test output files.** Write long test or build output to a file and keep only
+  the failing lines in context.
+- **Scratch artefacts.** Search results, API responses and generated data go to
+  files that are referenced by path.
+
+An agent that externalises well can survive aggressive compression, because the
+summary only needs pointers.
+
+## Compact, restart or delegate
+
+| Situation | Choose | Reason |
+|-----------|--------|--------|
+| Same task, history cluttered with resolved detail | Compact | The thread of reasoning is still valuable |
+| Task finished, next task unrelated | Restart | Old context only adds noise and cost |
+| Session derailed: repeated failures, contradictory instructions piling up | Restart with a hand-written brief | A clean start with the lessons outperforms a summary of confusion |
+| A sub-question needs lots of reading but only a short answer (search a codebase, read long docs, review a diff) | Delegate to a subagent | The reading stays in the subagent's context; only the conclusion returns |
+| Independent pieces of work | Parallel subagents | Each gets a focused context; the parent keeps the plan |
+
+When delegating, give the subagent a self-contained brief: goal, constraints,
+where to look, and what shape the answer should take. Ask for conclusions with
+file and line references, not dumps.
+
+## Triggering compression
+
+- Compress before the window is full, not at the limit. Leave room for the next
+  tool results and for the model to reason; compressing at the last moment
+  forces a rushed summary.
+- Prefer natural boundaries: after a subtask completes, after tests pass, before
+  starting a new phase. Mid-edit compression is where file trails get lost.
+- Clear stale tool results first (old file reads, superseded searches). That
+  often buys enough room without summarising reasoning at all.
+
+## Measuring whether compression loses too much
+
+Compression quality is measured by what the agent can still do, not by how
+readable the summary is.
+
+- **Probe questions.** After compressing, ask the agent questions whose answers
+  were in the dropped span: which files did you modify, what was the exact
+  error, why did we reject approach X. Score the answers.
+- **Task completion.** Run the same set of long tasks with and without the
+  compression strategy; compare success rate and total tokens.
+- **Re-work signals.** Count re-reads of files already read, repeated commands
+  and re-attempts of ruled-out approaches after each compression. A rising count
+  means the summary is dropping something the agent needs.
+
+## Common failure shapes
+
+- **The forgetting editor.** After compression the agent re-creates a file it
+  already created, or edits the wrong copy. Cause: missing file trail. Fix: the
+  Files section plus `git status` after every restore.
+- **The loop.** The agent retries a rejected approach. Cause: no ruled-out list.
+- **Goal drift.** The agent finishes a side quest and stops. Cause: goal and
+  acceptance criteria summarised away or reworded until vague.
+- **Paraphrased errors.** "A type error in the export module" cannot be matched
+  against the next test run. Keep the literal text.

package/pipeline/skills/shared/external/core-bluetooth/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: core-bluetooth
 description: "Build direct Bluetooth Low Energy workflows with Core Bluetooth. Use when implementing BLE central or peripheral GATT communication, scanning or connecting with CBCentralManager, discovering services and characteristics, reading/writing/subscribing with CBPeripheral, publishing local services with CBPeripheralManager, handling Bluetooth authorization, background BLE modes, state restoration, write flow control, or CBUUID-based workflows. For privacy-preserving accessory setup/picker flows, use accessorysetupkit first and return here for post-setup GATT communication."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # Core Bluetooth

package/pipeline/skills/shared/external/core-data/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: core-data
 description: "Build, review, or improve Core Data persistence in apps that have not adopted SwiftData. Use when working with NSManagedObject subclasses, NSFetchedResultsController for list-driven UI, NSBatchInsertRequest / NSBatchDeleteRequest / NSBatchUpdateRequest for bulk operations, NSPersistentHistoryChangeRequest for persistent history tracking and multi-target sync, NSStagedMigrationManager for staged schema migrations (iOS 17+), NSCompositeAttributeDescription for composite attributes (iOS 17+), or when integrating Core Data threading with Swift Concurrency. For Core Data + SwiftData coexistence or migration, see the swiftdata skill instead."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # Core Data

package/pipeline/skills/shared/external/core-motion/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: core-motion
 description: "Access Core Motion accelerometer, gyroscope, magnetometer, device-motion, pedometer, activity-recognition, altitude, headphone motion, batched high-frequency workout motion, and water-submersion/depth data. Use when reading device sensors, counting steps, detecting walking/running/driving/cycling, tracking altitude, building motion interactions, handling AirPods head tracking, or implementing watchOS dive/depth features."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # CoreMotion

package/pipeline/skills/shared/external/core-nfc/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: core-nfc
 description: "Read and write NFC tags using CoreNFC. Use when scanning NDEF tags, reading ISO7816/ISO15693/FeliCa/MIFARE tags, writing NDEF messages, handling NFC session lifecycle, configuring NFC entitlements, or implementing background tag reading in iOS apps."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # CoreNFC

package/pipeline/skills/shared/external/coreml/SKILL.md

@@ -1,8 +1,9 @@
 ---
 name: coreml
 description: "Integrate Core ML models in iOS apps for on-device machine learning inference. Covers model loading (.mlmodel, .mlpackage, .mlmodelc), predictions with auto-generated classes and MLFeatureProvider, compute unit configuration (CPU, GPU, Neural Engine), MLTensor, VNCoreMLRequest, MLComputePlan, multi-model pipelines, and deployment strategies. Use when loading Core ML models, making predictions, configuring compute units, or profiling model performance."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
-
 # Core ML Swift Integration
 
 Load, configure, and run Core ML models in iOS apps. This skill covers the

package/pipeline/skills/shared/external/council/SKILL.md

@@ -1,12 +1,13 @@
 ---
 name: council
-version: 1.0.0
 description: |
   Convene a multi-voice council for ambiguous decisions, tradeoffs, and
   go/no-go calls. Use when several valid paths exist and you need structured
   disagreement before choosing, not code review or implementation planning.
   Triggers on: "should we X or Y", "second opinion", "is this worth it",
   "ship now or wait", requests for dissent or multiple perspectives.
+metadata:
+  version: 1.0.0
 ---
 
 # Council

package/pipeline/skills/shared/external/cryptokit/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: cryptokit
 description: "Use Apple CryptoKit for Swift cryptographic primitives. Use when hashing with SHA-2 or SHA-3, generating HMACs, encrypting with AES-GCM or ChaChaPoly, signing with P256/P384/P521/Curve25519 or ML-DSA keys, performing ECDH, HPKE, ML-KEM, or X-Wing key exchange, using Secure Enclave CryptoKit keys, or migrating CommonCrypto code to CryptoKit."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # CryptoKit

package/pipeline/skills/shared/external/css-modern/SKILL.md

@@ -1,8 +1,9 @@
 ---
 name: css-modern
 description: "Modern CSS: container queries, nesting, @layer, Grid/Flexbox patterns, custom properties, animations, and transitions. Use when writing or reviewing modern CSS: layout, container queries, custom properties, animation."
-tags: [css, grid, flexbox, container-queries, nesting, animations, frontend]
-version: "2025.1"
+metadata:
+  tags: [css, grid, flexbox, container-queries, nesting, animations, frontend]
+  version: "2025.1"
 ---
 
 # Modern CSS (2024-2025)

package/pipeline/skills/shared/external/database-patterns/SKILL.md

@@ -1,8 +1,9 @@
 ---
 name: database-patterns
 description: "Database patterns: SQL best practices (PostgreSQL), indexing, query optimization, migrations, schema design, ORMs (SQLAlchemy, Prisma, Drizzle). Use when designing a schema, writing migrations, or a query needs optimising."
-tags: [database, sql, postgresql, prisma, drizzle, sqlalchemy, backend]
-version: "2025.1"
+metadata:
+  tags: [database, sql, postgresql, prisma, drizzle, sqlalchemy, backend]
+  version: "2025.1"
 ---
 
 # Database Patterns

package/pipeline/skills/shared/external/debugging-instruments/SKILL.md

@@ -1,6 +1,8 @@
 ---
 name: debugging-instruments
 description: "Debug iOS apps and profile performance using LLDB, Memory Graph Debugger, and Instruments. Use when diagnosing crashes, memory leaks, retain cycles, main thread hangs, slow rendering, build failures, or when profiling CPU, memory, energy, and network usage."
+metadata:
+  source: "dpearson2699/swift-ios-skills (PolyForm Perimeter 1.0.0)"
 ---
 
 # Debugging and Instruments