compact-agent 1.10.1 → 1.11.0

package/resources/ecc/skills/redis-patterns/SKILL.md

@@ -0,0 +1,403 @@
+---
+name: redis-patterns
+description: Redis data structure patterns, caching strategies, distributed locks, rate limiting, pub/sub, and connection management for production applications.
+origin: ECC
+---
+
+# Redis Patterns
+
+Quick reference for Redis best practices across common backend use cases.
+
+## How It Works
+
+Redis is an in-memory data structure store that supports strings, hashes, lists, sets, sorted sets, streams, and more. Individual Redis commands are atomic on a single instance; multi-step workflows require Lua scripts, MULTI/EXEC transactions, or explicit synchronization to stay atomic. Data is optionally persisted via RDB snapshots or AOF logs. Clients communicate over TCP using the RESP protocol; connection pools are essential to avoid per-request handshake overhead.
+
+## When to Activate
+
+- Adding caching to an application
+- Implementing rate limiting or throttling
+- Building distributed locks or coordination
+- Setting up session or token storage
+- Using Pub/Sub or Redis Streams for messaging
+- Configuring Redis in production (pooling, eviction, clustering)
+
+## Data Structure Cheat Sheet
+
+| Use Case | Structure | Example Key |
+|----------|-----------|-------------|
+| Simple cache | String | `product:123` |
+| User session | Hash | `session:abc` |
+| Leaderboard | Sorted Set | `scores:weekly` |
+| Unique visitors | Set | `visitors:2024-01-01` |
+| Activity feed | List | `feed:user:456` |
+| Event stream | Stream | `events:orders` |
+| Counters / rate limits | String (INCR) | `ratelimit:user:123` |
+| Bloom filter / HLL | HyperLogLog | `hll:pageviews` |
+
+## Core Patterns
+
+### Cache-Aside (Lazy Loading)
+
+```python
+import redis
+import json
+
+r = redis.Redis(host='localhost', port=6379, decode_responses=True)
+
+def get_product(product_id: int):
+    cache_key = f"product:{product_id}"
+    cached = r.get(cache_key)
+
+    if cached:
+        return json.loads(cached)
+
+    product = db.query("SELECT * FROM products WHERE id = %s", product_id)
+    r.setex(cache_key, 3600, json.dumps(product))  # TTL: 1 hour
+    return product
+```
+
+### Write-Through Cache
+
+```python
+def update_product(product_id: int, data: dict):
+    # Write to DB first
+    db.execute("UPDATE products SET ... WHERE id = %s", product_id)
+
+    # Immediately update cache
+    cache_key = f"product:{product_id}"
+    r.setex(cache_key, 3600, json.dumps(data))
+```
+
+### Cache Invalidation
+
+```python
+# Tag-based invalidation — group related keys under a set
+def cache_product(product_id: int, category_id: int, data: dict):
+    key = f"product:{product_id}"
+    tag = f"tag:category:{category_id}"
+    pipe = r.pipeline(transaction=True)
+    pipe.setex(key, 3600, json.dumps(data))
+    pipe.sadd(tag, key)
+    pipe.expire(tag, 3600)
+    pipe.execute()
+
+def invalidate_category(category_id: int):
+    tag = f"tag:category:{category_id}"
+    keys = r.smembers(tag)
+    if keys:
+        r.delete(*keys)
+    r.delete(tag)
+```
+
+### Session Storage
+
+```python
+import time
+import uuid
+
+def create_session(user_id: int, ttl: int = 86400) -> str:
+    session_id = str(uuid.uuid4())
+    key = f"session:{session_id}"
+    pipe = r.pipeline(transaction=True)
+    pipe.hset(key, mapping={
+        "user_id": user_id,
+        "created_at": int(time.time()),
+    })
+    pipe.expire(key, ttl)
+    pipe.execute()
+    return session_id
+
+def get_session(session_id: str) -> dict | None:
+    data = r.hgetall(f"session:{session_id}")
+    return data if data else None
+
+def delete_session(session_id: str):
+    r.delete(f"session:{session_id}")
+```
+
+## Rate Limiting
+
+### Fixed Window (Simple)
+
+```python
+def is_rate_limited(user_id: int, limit: int = 100, window: int = 60) -> bool:
+    key = f"ratelimit:{user_id}:{int(time.time()) // window}"
+    pipe = r.pipeline(transaction=True)
+    pipe.incr(key)
+    pipe.expire(key, window)
+    count, _ = pipe.execute()
+    return count > limit
+```
+
+### Sliding Window (Lua — Atomic)
+
+```lua
+-- sliding_window.lua
+local key = KEYS[1]
+local now = tonumber(ARGV[1])
+local window = tonumber(ARGV[2])
+local limit = tonumber(ARGV[3])
+
+redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
+local count = redis.call('ZCARD', key)
+
+if count < limit then
+    -- Use unique member (now + sequence) to avoid collisions within the same millisecond
+    local seq_key = key .. ':seq'
+    local seq = redis.call('INCR', seq_key)
+    redis.call('EXPIRE', seq_key, math.ceil(window / 1000))
+    redis.call('ZADD', key, now, now .. '-' .. seq)
+    redis.call('EXPIRE', key, math.ceil(window / 1000))
+    return 1
+end
+return 0
+```
+
+```python
+sliding_window = r.register_script(open('sliding_window.lua').read())
+
+def allow_request(user_id: int) -> bool:
+    key = f"ratelimit:sliding:{user_id}"
+    now = int(time.time() * 1000)
+    return bool(sliding_window(keys=[key], args=[now, 60000, 100]))
+```
+
+## Distributed Locks
+
+### Distributed Lock (Single Node — SET NX PX)
+
+```python
+import uuid
+
+def acquire_lock(resource: str, ttl_ms: int = 5000) -> str | None:
+    lock_key = f"lock:{resource}"
+    token = str(uuid.uuid4())
+    acquired = r.set(lock_key, token, px=ttl_ms, nx=True)
+    return token if acquired else None
+
+def release_lock(resource: str, token: str) -> bool:
+    release_script = """
+    if redis.call('get', KEYS[1]) == ARGV[1] then
+        return redis.call('del', KEYS[1])
+    else
+        return 0
+    end
+    """
+    result = r.eval(release_script, 1, f"lock:{resource}", token)
+    return bool(result)
+
+# Usage
+token = acquire_lock("order:payment:123")
+if token:
+    try:
+        process_payment()
+    finally:
+        release_lock("order:payment:123", token)
+```
+
+> For multi-node setups use the `redlock-py` library which implements the full Redlock algorithm.
+
+## Pub/Sub & Streams
+
+### Pub/Sub (Fire-and-Forget)
+
+```python
+# Publisher
+def publish_event(channel: str, payload: dict):
+    r.publish(channel, json.dumps(payload))
+
+# Subscriber (blocking — run in separate thread/process)
+def subscribe_events(channel: str):
+    pubsub = r.pubsub()
+    pubsub.subscribe(channel)
+    for message in pubsub.listen():
+        if message['type'] == 'message':
+            handle(json.loads(message['data']))
+```
+
+### Redis Streams (Durable Queue)
+
+```python
+# Producer
+def emit(stream: str, event: dict):
+    r.xadd(stream, event, maxlen=10000)  # Cap stream length
+
+# Consumer group — guarantees at-least-once delivery
+try:
+    r.xgroup_create('events:orders', 'processor', id='0', mkstream=True)
+except Exception:
+    pass  # Group already exists
+
+def consume(stream: str, group: str, consumer: str):
+    while True:
+        messages = r.xreadgroup(group, consumer, {stream: '>'}, count=10, block=2000)
+        for _, entries in (messages or []):
+            for msg_id, data in entries:
+                process(data)
+                r.xack(stream, group, msg_id)
+```
+
+> Prefer **Streams** over Pub/Sub when you need delivery guarantees, consumer groups, or replay.
+
+## Key Design
+
+### Naming Conventions
+
+```
+# Pattern: resource:id:field
+user:123:profile
+order:456:status
+cache:product:789
+
+# Pattern: namespace:resource:id
+myapp:session:abc123
+myapp:ratelimit:user:123
+
+# Pattern: resource:date (time-bound keys)
+stats:pageviews:2024-01-01
+```
+
+### TTL Strategy
+
+| Data Type | Suggested TTL |
+|-----------|--------------|
+| User session | 24h (`86400`) |
+| API response cache | 5–15 min |
+| Rate limit window | Match window size |
+| Short-lived tokens | 5–10 min |
+| Leaderboard | 1h–24h |
+| Static/reference data | 1h–1 week |
+
+Always set a TTL. Keys without TTL accumulate indefinitely and cause memory pressure.
+
+## Connection Management
+
+### Connection Pooling
+
+```python
+from redis import ConnectionPool, Redis
+
+pool = ConnectionPool(
+    host='localhost',
+    port=6379,
+    db=0,
+    max_connections=20,
+    decode_responses=True,
+    socket_connect_timeout=2,
+    socket_timeout=2,
+)
+
+r = Redis(connection_pool=pool)
+```
+
+### Cluster Mode
+
+```python
+from redis.cluster import RedisCluster
+
+r = RedisCluster(
+    startup_nodes=[{"host": "redis-1", "port": 6379}],
+    decode_responses=True,
+    skip_full_coverage_check=True,
+)
+```
+
+### Sentinel (High Availability)
+
+```python
+from redis.sentinel import Sentinel
+
+sentinel = Sentinel(
+    [('sentinel-1', 26379), ('sentinel-2', 26379)],
+    socket_timeout=0.5,
+)
+master = sentinel.master_for('mymaster', decode_responses=True)
+replica = sentinel.slave_for('mymaster', decode_responses=True)
+```
+
+## Eviction Policies
+
+| Policy | Behavior | Best For |
+|--------|----------|----------|
+| `noeviction` | Error on write when full | Queues / critical data |
+| `allkeys-lru` | Evict least recently used | General cache |
+| `volatile-lru` | LRU only among keys with TTL | Mixed data store |
+| `allkeys-lfu` | Evict least frequently used | Skewed access patterns |
+| `volatile-ttl` | Evict soonest-to-expire | Prioritize long-lived data |
+
+Set via `redis.conf`: `maxmemory-policy allkeys-lru`
+
+## Anti-Patterns
+
+| Anti-Pattern | Problem | Fix |
+|---|---|---|
+| Keys with no TTL | Memory grows unbounded | Always set TTL |
+| `KEYS *` in production | Blocks the server (O(N)) | Use `SCAN` cursor |
+| Storing large blobs (>100KB) | Slow serialization, memory pressure | Store reference + fetch from object store |
+| Single Redis for everything | No isolation between cache & queue | Use separate DBs or instances |
+| Ignoring connection pool limits | Connection exhaustion under load | Size pool to workload |
+| Not handling cache miss stampede | Thundering herd on cold start | Use locks or probabilistic early expiry |
+| `FLUSHALL` without thought | Wipes entire instance | Scope deletes by key pattern |
+
+### Cache Miss Stampede Prevention
+
+```python
+import threading
+
+_locks: dict[str, threading.Lock] = {}
+_locks_mutex = threading.Lock()
+
+def get_with_lock(key: str, fetch_fn, ttl: int = 300):
+    cached = r.get(key)
+    if cached:
+        return json.loads(cached)
+
+    with _locks_mutex:
+        if key not in _locks:
+            _locks[key] = threading.Lock()
+        lock = _locks[key]
+    with lock:
+        cached = r.get(key)  # Re-check after acquiring lock
+        if cached:
+            return json.loads(cached)
+        value = fetch_fn()
+        r.setex(key, ttl, json.dumps(value))
+        return value
+```
+
+> Note: for multi-process deployments, replace the in-process lock with `acquire_lock`/`release_lock` from the Distributed Locks section above.
+
+## Examples
+
+**Add caching to a Django/Flask API endpoint:**
+Use cache-aside with `setex` and a 5-minute TTL on the response. Key on the request parameters.
+
+**Rate-limit an API by user:**
+Use fixed-window with `pipeline(transaction=True)` for low-traffic endpoints; use sliding-window Lua for accurate per-user throttling.
+
+**Coordinate a background job across workers:**
+Use `acquire_lock` with a TTL that exceeds the expected job duration. Always release in a `finally` block.
+
+**Fan-out notifications to multiple subscribers:**
+Use Pub/Sub for fire-and-forget. Switch to Streams if you need guaranteed delivery or replay for late consumers.
+
+## Quick Reference
+
+| Pattern | When to Use |
+|---------|-------------|
+| Cache-aside | Read-heavy, tolerate slight staleness |
+| Write-through | Strong consistency required |
+| Distributed lock | Prevent concurrent access to a resource |
+| Sliding window rate limit | Accurate per-user throttling |
+| Redis Streams | Durable event queue with consumer groups |
+| Pub/Sub | Broadcast with no delivery guarantees needed |
+| Sorted Set leaderboard | Ranked scoring, pagination |
+| HyperLogLog | Approximate unique count at low memory |
+
+## Related
+
+- Skill: `postgres-patterns` — relational data patterns
+- Skill: `backend-patterns` — API and service layer patterns
+- Skill: `database-migrations` — schema versioning
+- Skill: `django-patterns` — Django cache framework integration
+- Agent: `database-reviewer` — full database review workflow

package/resources/ecc/skills/regex-vs-llm-structured-text/SKILL.md

@@ -0,0 +1,220 @@
+---
+name: regex-vs-llm-structured-text
+description: Decision framework for choosing between regex and LLM when parsing structured text — start with regex, add LLM only for low-confidence edge cases.
+origin: ECC
+---
+
+# Regex vs LLM for Structured Text Parsing
+
+A practical decision framework for parsing structured text (quizzes, forms, invoices, documents). The key insight: regex handles 95-98% of cases cheaply and deterministically. Reserve expensive LLM calls for the remaining edge cases.
+
+## When to Activate
+
+- Parsing structured text with repeating patterns (questions, forms, tables)
+- Deciding between regex and LLM for text extraction
+- Building hybrid pipelines that combine both approaches
+- Optimizing cost/accuracy tradeoffs in text processing
+
+## Decision Framework
+
+```
+Is the text format consistent and repeating?
+├── Yes (>90% follows a pattern) → Start with Regex
+│   ├── Regex handles 95%+ → Done, no LLM needed
+│   └── Regex handles <95% → Add LLM for edge cases only
+└── No (free-form, highly variable) → Use LLM directly
+```
+
+## Architecture Pattern
+
+```
+Source Text
+    │
+    ▼
+[Regex Parser] ─── Extracts structure (95-98% accuracy)
+    │
+    ▼
+[Text Cleaner] ─── Removes noise (markers, page numbers, artifacts)
+    │
+    ▼
+[Confidence Scorer] ─── Flags low-confidence extractions
+    │
+    ├── High confidence (≥0.95) → Direct output
+    │
+    └── Low confidence (<0.95) → [LLM Validator] → Output
+```
+
+## Implementation
+
+### 1. Regex Parser (Handles the Majority)
+
+```python
+import re
+from dataclasses import dataclass
+
+@dataclass(frozen=True)
+class ParsedItem:
+    id: str
+    text: str
+    choices: tuple[str, ...]
+    answer: str
+    confidence: float = 1.0
+
+def parse_structured_text(content: str) -> list[ParsedItem]:
+    """Parse structured text using regex patterns."""
+    pattern = re.compile(
+        r"(?P<id>\d+)\.\s*(?P<text>.+?)\n"
+        r"(?P<choices>(?:[A-D]\..+?\n)+)"
+        r"Answer:\s*(?P<answer>[A-D])",
+        re.MULTILINE | re.DOTALL,
+    )
+    items = []
+    for match in pattern.finditer(content):
+        choices = tuple(
+            c.strip() for c in re.findall(r"[A-D]\.\s*(.+)", match.group("choices"))
+        )
+        items.append(ParsedItem(
+            id=match.group("id"),
+            text=match.group("text").strip(),
+            choices=choices,
+            answer=match.group("answer"),
+        ))
+    return items
+```
+
+### 2. Confidence Scoring
+
+Flag items that may need LLM review:
+
+```python
+@dataclass(frozen=True)
+class ConfidenceFlag:
+    item_id: str
+    score: float
+    reasons: tuple[str, ...]
+
+def score_confidence(item: ParsedItem) -> ConfidenceFlag:
+    """Score extraction confidence and flag issues."""
+    reasons = []
+    score = 1.0
+
+    if len(item.choices) < 3:
+        reasons.append("few_choices")
+        score -= 0.3
+
+    if not item.answer:
+        reasons.append("missing_answer")
+        score -= 0.5
+
+    if len(item.text) < 10:
+        reasons.append("short_text")
+        score -= 0.2
+
+    return ConfidenceFlag(
+        item_id=item.id,
+        score=max(0.0, score),
+        reasons=tuple(reasons),
+    )
+
+def identify_low_confidence(
+    items: list[ParsedItem],
+    threshold: float = 0.95,
+) -> list[ConfidenceFlag]:
+    """Return items below confidence threshold."""
+    flags = [score_confidence(item) for item in items]
+    return [f for f in flags if f.score < threshold]
+```
+
+### 3. LLM Validator (Edge Cases Only)
+
+```python
+def validate_with_llm(
+    item: ParsedItem,
+    original_text: str,
+    client,
+) -> ParsedItem:
+    """Use LLM to fix low-confidence extractions."""
+    response = client.messages.create(
+        model="claude-haiku-4-5-20251001",  # Cheapest model for validation
+        max_tokens=500,
+        messages=[{
+            "role": "user",
+            "content": (
+                f"Extract the question, choices, and answer from this text.\n\n"
+                f"Text: {original_text}\n\n"
+                f"Current extraction: {item}\n\n"
+                f"Return corrected JSON if needed, or 'CORRECT' if accurate."
+            ),
+        }],
+    )
+    # Parse LLM response and return corrected item...
+    return corrected_item
+```
+
+### 4. Hybrid Pipeline
+
+```python
+def process_document(
+    content: str,
+    *,
+    llm_client=None,
+    confidence_threshold: float = 0.95,
+) -> list[ParsedItem]:
+    """Full pipeline: regex -> confidence check -> LLM for edge cases."""
+    # Step 1: Regex extraction (handles 95-98%)
+    items = parse_structured_text(content)
+
+    # Step 2: Confidence scoring
+    low_confidence = identify_low_confidence(items, confidence_threshold)
+
+    if not low_confidence or llm_client is None:
+        return items
+
+    # Step 3: LLM validation (only for flagged items)
+    low_conf_ids = {f.item_id for f in low_confidence}
+    result = []
+    for item in items:
+        if item.id in low_conf_ids:
+            result.append(validate_with_llm(item, content, llm_client))
+        else:
+            result.append(item)
+
+    return result
+```
+
+## Real-World Metrics
+
+From a production quiz parsing pipeline (410 items):
+
+| Metric | Value |
+|--------|-------|
+| Regex success rate | 98.0% |
+| Low confidence items | 8 (2.0%) |
+| LLM calls needed | ~5 |
+| Cost savings vs all-LLM | ~95% |
+| Test coverage | 93% |
+
+## Best Practices
+
+- **Start with regex** — even imperfect regex gives you a baseline to improve
+- **Use confidence scoring** to programmatically identify what needs LLM help
+- **Use the cheapest LLM** for validation (Haiku-class models are sufficient)
+- **Never mutate** parsed items — return new instances from cleaning/validation steps
+- **TDD works well** for parsers — write tests for known patterns first, then edge cases
+- **Log metrics** (regex success rate, LLM call count) to track pipeline health
+
+## Anti-Patterns to Avoid
+
+- Sending all text to an LLM when regex handles 95%+ of cases (expensive and slow)
+- Using regex for free-form, highly variable text (LLM is better here)
+- Skipping confidence scoring and hoping regex "just works"
+- Mutating parsed objects during cleaning/validation steps
+- Not testing edge cases (malformed input, missing fields, encoding issues)
+
+## When to Use
+
+- Quiz/exam question parsing
+- Form data extraction
+- Invoice/receipt processing
+- Document structure parsing (headers, sections, tables)
+- Any structured text with repeating patterns where cost matters

package/resources/ecc/skills/remotion-video-creation/SKILL.md

@@ -0,0 +1,43 @@
+---
+name: remotion-video-creation
+description: Best practices for Remotion - Video creation in React. 29 domain-specific rules covering 3D, animations, audio, captions, charts, transitions, and more.
+metadata:
+  tags: remotion, video, react, animation, composition, three.js, lottie
+---
+
+## When to use
+
+Use this skills whenever you are dealing with Remotion code to obtain the domain-specific knowledge.
+
+## How to use
+
+Read individual rule files for detailed explanations and code examples:
+
+- [rules/3d.md](rules/3d.md) - 3D content in Remotion using Three.js and React Three Fiber
+- [rules/animations.md](rules/animations.md) - Fundamental animation skills for Remotion
+- [rules/assets.md](rules/assets.md) - Importing images, videos, audio, and fonts into Remotion
+- [rules/audio.md](rules/audio.md) - Using audio and sound in Remotion - importing, trimming, volume, speed, pitch
+- [rules/calculate-metadata.md](rules/calculate-metadata.md) - Dynamically set composition duration, dimensions, and props
+- [rules/can-decode.md](rules/can-decode.md) - Check if a video can be decoded by the browser using Mediabunny
+- [rules/charts.md](rules/charts.md) - Chart and data visualization patterns for Remotion
+- [rules/compositions.md](rules/compositions.md) - Defining compositions, stills, folders, default props and dynamic metadata
+- [rules/display-captions.md](rules/display-captions.md) - Displaying captions in Remotion with TikTok-style pages and word highlighting
+- [rules/extract-frames.md](rules/extract-frames.md) - Extract frames from videos at specific timestamps using Mediabunny
+- [rules/fonts.md](rules/fonts.md) - Loading Google Fonts and local fonts in Remotion
+- [rules/get-audio-duration.md](rules/get-audio-duration.md) - Getting the duration of an audio file in seconds with Mediabunny
+- [rules/get-video-dimensions.md](rules/get-video-dimensions.md) - Getting the width and height of a video file with Mediabunny
+- [rules/get-video-duration.md](rules/get-video-duration.md) - Getting the duration of a video file in seconds with Mediabunny
+- [rules/gifs.md](rules/gifs.md) - Displaying GIFs synchronized with Remotion's timeline
+- [rules/images.md](rules/images.md) - Embedding images in Remotion using the Img component
+- [rules/import-srt-captions.md](rules/import-srt-captions.md) - Importing .srt subtitle files into Remotion using @remotion/captions
+- [rules/lottie.md](rules/lottie.md) - Embedding Lottie animations in Remotion
+- [rules/measuring-dom-nodes.md](rules/measuring-dom-nodes.md) - Measuring DOM element dimensions in Remotion
+- [rules/measuring-text.md](rules/measuring-text.md) - Measuring text dimensions, fitting text to containers, and checking overflow
+- [rules/sequencing.md](rules/sequencing.md) - Sequencing patterns for Remotion - delay, trim, limit duration of items
+- [rules/tailwind.md](rules/tailwind.md) - Using TailwindCSS in Remotion
+- [rules/text-animations.md](rules/text-animations.md) - Typography and text animation patterns for Remotion
+- [rules/timing.md](rules/timing.md) - Interpolation curves in Remotion - linear, easing, spring animations
+- [rules/transcribe-captions.md](rules/transcribe-captions.md) - Transcribing audio to generate captions in Remotion
+- [rules/transitions.md](rules/transitions.md) - Scene transition patterns for Remotion
+- [rules/trimming.md](rules/trimming.md) - Trimming patterns for Remotion - cut the beginning or end of animations
+- [rules/videos.md](rules/videos.md) - Embedding videos in Remotion - trimming, volume, speed, looping, pitch