c64bridge 0.5.1 → 0.6.0

package/doc/meta-tools/analysis.md

@@ -0,0 +1,255 @@
+# Meta Tools Impact Analysis
+
+Purpose: Analyze unimplemented meta tools to identify high-impact, low-effort candidates that will most improve agent capabilities for BASIC/ASM coding, graphics generation, SID music composition, and C64 filesystem navigation.
+
+## Current Implementation Status
+
+### Implemented (Phase 0-1): 11 tools
+- ✅ `firmware_info_and_healthcheck` — Verify firmware readiness
+- ✅ `wait_for_screen_text` — Poll screen for text/regex matches
+- ✅ `verify_and_write_memory` — Safe memory write with verification
+- ✅ `start_background_task`, `stop_background_task`, `list_background_tasks`, `stop_all_background_tasks` — Background task orchestration
+- ✅ `find_paths_by_name` — Container-aware file discovery
+- ✅ `memory_dump_to_file` — Export memory ranges to files
+- ✅ `config_snapshot_and_restore` — Configuration backup/restore
+- ✅ `program_shuffle` — Run multiple programs sequentially
+- ✅ `batch_run_with_assertions` — Run programs with validation
+- ✅ `bundle_run_artifacts` — Collect run artifacts
+- ✅ `compile_run_verify_cycle` — Complete build-test cycle
+
+### Not Implemented: ~60 tools across multiple domains
+
+## Impact Analysis for Agent Capabilities
+
+### 1. C64 Filesystem Navigation & Program Discovery (CRITICAL)
+
+**Problem**: Agents struggle to find and run programs in large C64 file collections (D64/D71/D81/T64 images)
+
+**High-Impact Tools**:
+
+- **`find_and_run_program_by_name`** ⭐⭐⭐⭐⭐ (Effort: 45min, Benefit: HIGH)
+  - Impact: Enables agents to search entire filesystem and run programs by name
+  - Use cases: "Run Archon from my games collection", "Load the demo called Boulder"
+  - Dependencies: `find_paths_by_name` (✅ implemented)
+  - **PRIORITY: #1** — Essential for effective C64 interaction
+
+- **`filesystem_stats_by_extension`** ⭐⭐⭐ (Effort: 35min, Benefit: MEDIUM)
+  - Impact: Helps agents understand file collections and provide insights
+  - Use cases: "How many games do I have?", "List all PRG files"
+  - Dependencies: None
+  - **PRIORITY: #3** — Good context for navigation
+
+- **`classify_prg_basic_or_mc`** ⭐⭐⭐ (Effort: 45min, Benefit: MEDIUM)
+  - Impact: Distinguishes BASIC from machine code programs
+  - Use cases: Better error messages, appropriate testing strategies
+  - Dependencies: `container_aware_walk_and_classify`
+  - **PRIORITY: #8** — Useful but not urgent
+
+### 2. Graphics Generation & Extraction (HIGH)
+
+**Problem**: Agents need better tools to create, manipulate, and extract C64 graphics
+
+**High-Impact Tools**:
+
+- **`extract_sprites_from_ram`** ⭐⭐⭐⭐ (Effort: 80min, Benefit: MEDIUM)
+  - Impact: Extract sprites from running programs for analysis/reuse
+  - Use cases: "Show me the sprites from this game", sprite library building
+  - Dependencies: `memory_dump_to_file` (✅ implemented)
+  - **PRIORITY: #5** — Moderate effort, good payoff for graphics work
+
+- **`rip_charset_from_ram`** ⭐⭐⭐⭐ (Effort: 75min, Benefit: MEDIUM)
+  - Impact: Extract custom character sets for reuse/study
+  - Use cases: Font library creation, PETSCII art analysis
+  - Dependencies: `memory_dump_to_file` (✅ implemented)
+  - **PRIORITY: #6** — Similar to sprites, good for graphics
+
+- **`screen_capture_timeline`** ⭐⭐⭐ (Effort: 50min, Benefit: MEDIUM)
+  - Impact: Record program execution over time
+  - Use cases: Demo capture, animation analysis, debugging
+  - Dependencies: `wait_for_screen_text` (✅ implemented)
+  - **PRIORITY: #9** — Nice to have but lower urgency
+
+### 3. SID Music & Audio (HIGH)
+
+**Problem**: Agents need better workflows for creating and testing SID music
+
+**High-Impact Tools**:
+
+- **`silence_and_verify`** ⭐⭐⭐⭐⭐ (Effort: 35min, Benefit: HIGH)
+  - Impact: Essential for reliable SID testing and music composition
+  - Use cases: Verify silence before playing, clean state for tests
+  - Dependencies: None
+  - **PRIORITY: #2** — Quick win, high value for music work
+
+- **`music_compile_play_analyze`** ⭐⭐⭐⭐ (Effort: 90min, Benefit: HIGH)
+  - Impact: Complete music development workflow in one tool
+  - Use cases: "Create a C major scale and verify it plays correctly"
+  - Dependencies: `silence_and_verify`
+  - **PRIORITY: #4** — After silence_and_verify, excellent workflow tool
+
+- **`sid_param_sweep`** ⭐⭐⭐ (Effort: 90min, Benefit: MEDIUM)
+  - Impact: Experiment with SID parameters systematically
+  - Use cases: Find best ADSR settings, explore waveforms
+  - Dependencies: `silence_and_verify`
+  - **PRIORITY: #10** — Advanced feature, implement later
+
+### 4. Debug Streaming & Assembly Verification (COMPLEX)
+
+**Problem**: Verifying assembly execution requires debug stream integration
+
+**High-Impact Tools** (all require infrastructure):
+
+- **`debug_stream_watch`** ⭐⭐⭐⭐⭐ (Effort: 55min, Benefit: MEDIUM)
+  - Impact: Foundation for all debug streaming features
+  - Use cases: Monitor CPU/VIC/1541 activity
+  - Dependencies: UDP ingest pipeline (NOT YET IMPLEMENTED)
+  - **PRIORITY: Post-GA** — Requires infrastructure work first
+
+- **`debug_loop_run_and_capture`** ⭐⭐⭐⭐⭐ (Effort: 80min, Benefit: HIGH)
+  - Impact: Verify assembly programs by comparing expected vs actual execution
+  - Use cases: "Did my IRQ handler fire correctly?", "Verify timing"
+  - Dependencies: `debug_stream_watch` + infrastructure
+  - **PRIORITY: Post-GA** — Very valuable but infrastructure-gated
+
+- **`verify_raster_irq_line`** ⭐⭐⭐⭐ (Effort: 70min, Benefit: HIGH)
+  - Impact: Critical for graphics programming verification
+  - Use cases: Validate IRQ setup in demos and games
+  - Dependencies: `debug_loop_run_and_capture`
+  - **PRIORITY: Post-GA** — Excellent feature but requires debug foundation
+
+**Note**: All debug streaming tools are marked Post-GA because they require:
+- UDP packet ingest pipeline with backpressure handling
+- Sampling/windowing for high-rate streams
+- Performance validation via load tests
+- Observability infrastructure
+
+### 5. Drive & Filesystem Management (MEDIUM)
+
+**High-Impact Tools**:
+
+- **`drive_mount_and_verify`** ⭐⭐⭐ (Effort: 45min, Benefit: MEDIUM)
+  - Impact: Reliable drive mounting with retry logic
+  - Use cases: Mount disk images for program loading
+  - Dependencies: `firmware_info_and_healthcheck` (✅ implemented)
+  - **PRIORITY: #7** — Good utility, implement after higher priorities
+
+- **`create_and_mount_blank_d64`** ⭐⭐ (Effort: 35min, Benefit: MEDIUM)
+  - Impact: Create working disks for file operations
+  - Use cases: Save programs, create disk collections
+  - Dependencies: `drive_mount_and_verify`
+  - **PRIORITY: #11** — Lower priority, niche use case
+
+### 6. Memory & Disassembly (MEDIUM)
+
+**High-Impact Tools**:
+
+- **`memory_snapshot_and_diff`** ⭐⭐⭐⭐ (Effort: 50min, Benefit: HIGH)
+  - Impact: Debug memory changes during program execution
+  - Use cases: "What changed after running this?", reverse engineering
+  - Dependencies: `memory_dump_to_file` (✅ implemented)
+  - **PRIORITY: #12** — Useful for advanced debugging
+
+- **`disassemble_ram_region`** ⭐⭐⭐ (Effort: 60min, Benefit: MEDIUM)
+  - Impact: Understand machine code in memory
+  - Use cases: Reverse engineering, learning assembly
+  - Dependencies: None (agent-side disassembler)
+  - **PRIORITY: #13** — Nice to have, not urgent
+
+## Reprioritized Rollout Order
+
+Based on impact analysis focusing on:
+1. **Immediate agent effectiveness** (find and run programs, compose music)
+2. **Low-hanging fruit** (small effort, high impact)
+3. **Foundation for future work** (enables other tools)
+4. **Deferring complex infrastructure** (debug streaming to Post-GA)
+
+### New High-Priority Sequence
+
+#### Quick Wins (Phase 2a — High Impact, Low Effort)
+1. **`find_and_run_program_by_name`** (45min) — Enables program discovery
+2. **`silence_and_verify`** (35min) — Essential for SID work
+3. **`filesystem_stats_by_extension`** (35min) — Context for file operations
+4. **`music_compile_play_analyze`** (90min) — Complete music workflow
+
+#### Graphics & Extraction (Phase 2b — Medium Effort, High Value)
+5. **`extract_sprites_from_ram`** (80min) — Graphics extraction
+6. **`rip_charset_from_ram`** (75min) — Character set extraction
+7. **`drive_mount_and_verify`** (45min) — Reliable disk handling
+
+#### Advanced Features (Phase 2c — Lower Priority)
+8. **`classify_prg_basic_or_mc`** (45min) — Program classification
+9. **`screen_capture_timeline`** (50min) — Timeline recording
+10. **`sid_param_sweep`** (90min) — SID experimentation
+11. **`create_and_mount_blank_d64`** (35min) — Disk creation
+12. **`memory_snapshot_and_diff`** (50min) — Memory comparison
+13. **`disassemble_ram_region`** (60min) — Disassembly
+
+#### Container & Classification (Phase 3 — When Filesystem Work Needed)
+- **`container_aware_walk_and_classify`** (60min)
+- **`dedup_scan`** (60min)
+- **`dedup_plan_and_apply`** (75min)
+
+#### Debug Streaming (Phase 5 — Post Infrastructure)
+- All debug stream tools deferred until UDP infrastructure is ready
+- These are high-value but require significant foundation work
+
+## Key Insights
+
+### Why This Order?
+
+1. **`find_and_run_program_by_name` is #1** because it's the most requested feature — agents need to find programs in disk collections. Without this, users must manually locate files.
+
+2. **`silence_and_verify` is #2** because it's a quick win (35min) that unblocks all SID testing and music composition work. Currently, SID tests can be flaky without proper silence verification.
+
+3. **Graphics tools (#5-6)** are grouped together because they share similar patterns (RAM scanning, pattern detection, export). Implementing them consecutively leverages shared code.
+
+4. **Debug streaming deferred** because it needs infrastructure (UDP pipeline, backpressure, performance testing) that would take weeks to implement properly. Better to ship other high-value features first.
+
+### Impact on Agent Capabilities
+
+After implementing the Phase 2a quick wins, agents will be able to:
+- ✅ Find and run any program in a C64 file collection by name
+- ✅ Reliably test SID music with proper silence verification
+- ✅ Understand file collection contents and statistics
+- ✅ Create and verify complete music compositions in one workflow
+
+After Phase 2b graphics tools:
+- ✅ Extract sprites and character sets from running programs
+- ✅ Build reusable graphics libraries
+- ✅ Analyze and learn from existing C64 graphics
+
+### ROI Summary
+
+**Top 4 Quick Wins (Phase 2a)**: ~205 minutes (3.4 hours)
+- Unlocks: Program discovery, music composition, file navigation
+- Impact: Transforms agent from "follow instructions" to "find and create"
+
+**Graphics Suite (Phase 2b)**: ~200 minutes (3.3 hours)
+- Unlocks: Sprite/charset extraction, graphics analysis
+- Impact: Enables graphics-focused workflows
+
+**Total High-Priority Work**: ~6.7 hours for 11 tools
+- Massive improvement in agent capabilities
+- All low-risk (no infrastructure dependencies)
+- Clear, testable outcomes
+
+### Deferred Work: Debug Streaming
+
+Debug streaming tools (Phase 5) are incredibly valuable but require:
+- UDP packet ingest infrastructure (~40 hours)
+- Performance validation (~8 hours)
+- Observability & monitoring (~8 hours)
+- Integration testing (~16 hours)
+
+**Total infrastructure: ~72 hours** before first debug tool can ship (see `rollout.md` Phase 5 for detailed breakdown)
+
+**Decision**: Ship high-value, low-complexity features first (Phases 2a-3), then revisit debug streaming as a separate project with dedicated infrastructure sprint.
+
+## Recommendations
+
+1. **Immediate**: Implement Phase 2a (quick wins) in next sprint
+2. **Near-term**: Follow with Phase 2b (graphics) when graphics work is prioritized
+3. **Medium-term**: Phase 2c and Phase 3 tools as needed based on user feedback
+4. **Long-term**: Plan debug streaming infrastructure project, then implement Phase 5
+
+This prioritization maximizes near-term agent effectiveness while deferring complex infrastructure work that would block faster progress.

package/doc/{meta-tools.md → meta-tools/catalog.md}

@@ -168,8 +168,8 @@ Container-aware traversal: All tools in this group recurse into disk/tape contai
 
 File-Type Determination (CBM directory and PRG content):
 - Step 1: Fetch metadata/listing for host files; detect container types (`.d64/.d71/.d81/.t64`).
-- Step 2: For containers, parse directory entries (CBM type byte at offset $02) to classify `SEQ/PRG/USR/REL/UNKNOWN`.
-- Step 3: If type is `PRG`, inspect first bytes of file data: load address, plausible next-line pointer, and presence of tokens ≥ $80 before `00 00` termination → classify as `BASIC`; else `PRG-MC`.
+- Step 2: For containers, parse directory entries: CBM file type byte at offset $02 (0x02 = SEQ, 0x82 = PRG, etc.) to classify `SEQ/PRG/USR/REL/UNKNOWN`.
+- Step 3: If type is `PRG`, inspect first bytes of file data: load address, plausible next-line pointer, and presence of BASIC tokens (≥ $80) before `00 00` (end-of-program marker) → classify as `BASIC`; else `PRG-MC` (machine code).
 - Practical note: When byte reads are not supported by firmware for host files, use `load_prg` (non-running) + `readmem` to examine the loaded header without executing.
 
 - "dedup_scan"

package/doc/meta-tools/notes.md

@@ -0,0 +1,174 @@
+# Meta Tools Implementation Notes
+
+This document tracks observations, decisions, and learnings from the meta tools rollout.
+
+## Overview
+
+Meta tools are high-level orchestration tools that compose multiple MCP operations into single, stateful workflows. They reduce round-trips between client and server and enable complex agent behaviors.
+
+## Current State (as of 2025-10-30)
+
+### Implemented Tools (11)
+- Phase 0 foundations: firmware_info_and_healthcheck, wait_for_screen_text, verify_and_write_memory, background task suite, find_paths_by_name, memory_dump_to_file, config_snapshot_and_restore
+- Phase 1 orchestration: program_shuffle, batch_run_with_assertions, bundle_run_artifacts, compile_run_verify_cycle
+
+### File Organization
+- Source: `src/tools/meta/` with submodules by domain (diagnostics, screen, memory, background, filesystem, config, program, artifacts, compilation)
+- Tests: Integrated into existing test suites, no dedicated meta/ test folder yet
+- Docs: **Reorganized into `doc/meta-tools/`** (matching improve-knowledge pattern)
+
+## Documentation Structure
+
+Following the improve-knowledge precedent:
+```
+doc/meta-tools/
+  ├── catalog.md        (formerly meta-tools.md — comprehensive tool definitions)
+  ├── rollout.md        (formerly meta-tools-rollout.md — implementation checklist)
+  ├── analysis.md       (NEW — impact analysis and prioritization)
+  └── notes.md          (this file — observations and decisions)
+```
+
+Benefits:
+- Cleaner doc/ root folder
+- Related documents grouped together
+- Easier to find meta tools documentation
+- Consistent with improve-knowledge structure
+
+## Prioritization Rationale
+
+### Why Reprioritize?
+
+Original rollout followed dependency order and implementation phases. However, based on actual agent usage patterns, some unimplemented tools would provide much higher value:
+
+1. **Filesystem discovery** — Agents frequently struggle to find programs in disk collections
+2. **SID testing** — Current music workflows lack proper silence verification
+3. **Graphics extraction** — Users want to analyze and reuse sprites/charsets from programs
+4. **Debug streaming** — Highly valuable but requires significant infrastructure
+
+### Decision: High-Impact, Low-Effort First
+
+New approach prioritizes:
+- Quick wins (35-45 minute tools with high impact)
+- Foundation tools that enable agent creativity (find programs, compose music)
+- Tools with clear, testable outcomes
+- Deferring infrastructure-heavy work (debug streaming)
+
+See `analysis.md` for detailed impact breakdown.
+
+## Implementation Patterns
+
+### Meta Tool Anatomy
+
+Each meta tool typically:
+1. **Composes** existing MCP tools (no direct REST calls when possible)
+2. **Maintains state** agent-side (snapshots, buffers, registries)
+3. **Provides retry logic** and error recovery
+4. **Returns structured results** with metadata and URIs
+5. **Respects hardware safeguards** (pause/resume, timeouts, mutual exclusion)
+
+### Testing Strategy
+
+- Unit tests for tool logic (parameter validation, error handling)
+- Integration tests with mock C64Client
+- No hardware required for CI/CD
+- Real hardware testing manual (for now)
+
+### Documentation Requirements
+
+For each implemented tool:
+- [ ] Entry in `catalog.md` with full specification
+- [ ] Checkbox in `rollout.md` marked complete
+- [ ] Tests covering happy path and one failure case
+- [ ] Usage examples in tool description
+- [ ] Dependencies clearly documented
+
+## Technical Decisions
+
+### Background Tasks Registry
+
+**Decision**: In-memory registry with optional persistence to disk
+**Rationale**: Simplicity for MVP; disk persistence can be added later if needed
+**Trade-offs**: Tasks lost on restart, but acceptable for experimental features
+
+### Container-Aware Filesystem Tools
+
+**Decision**: Implement discovery that recurses into D64/D71/D81/T64 containers
+**Rationale**: C64 programs are often stored in disk images, not loose files
+**Implementation**: Use wildcards when firmware supports them; fall back to mount+BASIC scraping when necessary
+
+### Debug Streaming Infrastructure
+
+**Decision**: Defer all debug streaming tools to Post-GA
+**Rationale**: Requires UDP ingest, backpressure handling, performance validation (~72 hours of infrastructure work)
+**Alternative considered**: Implement basic version with TCP — rejected because packet loss handling is essential
+**Next steps**: Plan dedicated infrastructure sprint before tackling Phase 5 debug tools
+
+### File Organization
+
+**Decision**: Keep meta tools in `src/tools/meta/` submodule structure
+**Rationale**: 
+- Clear separation from core tools
+- Easy to mark as experimental/beta
+- Modular structure by domain (diagnostics, screen, memory, etc.)
+- No impact on MCP tool registration (all tools exposed equally)
+
+**Trade-offs**: Slightly deeper import paths, but better organization
+
+## Lessons Learned
+
+### From Phase 0-1 Implementation
+
+1. **Background tasks are versatile** — Can schedule any tool, not just specific operations
+2. **Verification patterns reusable** — verify_and_write_memory pattern applies to many tools
+3. **Screen polling essential** — wait_for_screen_text foundation for many workflows
+4. **Artifacts bundling popular** — bundle_run_artifacts frequently requested by users
+
+### From Impact Analysis
+
+1. **Filesystem navigation is critical** — Most common agent failure: "I can't find that program"
+2. **SID testing needs improvement** — Silence verification missing causes flaky tests
+3. **Debug streaming is desirable but complex** — High demand, but infrastructure not ready
+4. **Graphics tools have broad appeal** — Sprite/charset extraction enables creative reuse
+
+## Future Considerations
+
+### Potential Improvements
+
+1. **Meta tool composition** — Allow meta tools to call other meta tools
+2. **Persistent background tasks** — Survive server restarts
+3. **Progress callbacks** — Report progress during long-running operations
+4. **Cancellation** — Abort in-progress meta tools gracefully
+5. **Dry-run mode** — Preview actions without executing (especially for destructive ops)
+
+### Infrastructure Needs
+
+For debug streaming (Phase 5):
+- UDP packet receiver with reassembly
+- Backpressure and flow control
+- Circular buffer for trace windows
+- Packet loss detection and reporting
+- Performance benchmarks and load tests
+- Observability (metrics, logging, tracing)
+
+### Documentation Gaps
+
+- No architecture diagram for meta tool system
+- Limited examples of meta tool composition patterns
+- No performance guidelines (when to avoid meta tools)
+- Testing best practices not documented
+
+## Open Questions
+
+1. **Should meta tools be separately versioned?** Currently experimental, but what's the graduation criteria?
+2. **How to handle breaking changes?** Meta tools compose other tools — cascade impacts?
+3. **Rate limiting?** Some meta tools could issue many requests — need throttling?
+4. **Observability?** How to monitor meta tool usage and performance in production?
+
+## References
+
+- `catalog.md` — Full meta tool definitions with effort/benefit matrix
+- `rollout.md` — Implementation checklist with dependencies
+- `analysis.md` — Impact analysis and prioritization rationale
+- `doc/improve-knowledge/` — Similar documentation structure pattern
+- `doc/agent-state-spec.md` — Background task registry specification
+- `src/tools/meta/` — Implementation source code