@softspark/ai-toolkit 1.0.0

package/app/skills/database-patterns/SKILL.md

@@ -0,0 +1,297 @@
+---
+name: database-patterns
+description: "Loaded when user asks about database schema or query optimization"
+effort: medium
+user-invocable: false
+---
+
+# Database Patterns Skill
+
+## ORM Selection
+
+| Scenario | ORM |
+|----------|-----|
+| Node.js, type-safe | Prisma |
+| Node.js, SQL-first | Drizzle |
+| Python, async | SQLAlchemy 2.0 |
+| Python, simple | SQLModel |
+| PHP | Doctrine, Eloquent |
+
+---
+
+## Schema Design
+
+### Naming Conventions
+
+```sql
+-- Tables: plural, snake_case
+CREATE TABLE user_profiles (...);
+
+-- Columns: snake_case
+user_id, created_at, is_active
+
+-- Indexes: idx_{table}_{columns}
+CREATE INDEX idx_users_email ON users(email);
+
+-- Foreign keys: fk_{table}_{ref_table}
+CONSTRAINT fk_orders_users FOREIGN KEY (user_id) REFERENCES users(id)
+```
+
+### Common Patterns
+
+#### Soft Delete
+```sql
+ALTER TABLE users ADD COLUMN deleted_at TIMESTAMP NULL;
+
+-- Query active records
+SELECT * FROM users WHERE deleted_at IS NULL;
+```
+
+#### Audit Columns
+```sql
+created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+created_by UUID REFERENCES users(id),
+updated_by UUID REFERENCES users(id)
+```
+
+#### UUID vs Serial
+| Use Case | Type |
+|----------|------|
+| Internal only | SERIAL/BIGSERIAL |
+| External/distributed | UUID |
+| Human readable | SERIAL with prefix |
+
+---
+
+## Index Strategies
+
+### When to Index
+- Foreign keys (always)
+- Columns in WHERE clauses
+- Columns in ORDER BY
+- Columns in JOIN conditions
+
+### Index Types
+| Type | Use Case |
+|------|----------|
+| B-tree | Equality, range (default) |
+| Hash | Equality only |
+| GIN | Arrays, JSONB, full-text |
+| GiST | Geometric, full-text |
+| BRIN | Large sequential data |
+
+### Composite Index Order
+```sql
+-- Good: matches query pattern
+CREATE INDEX idx_orders_user_date ON orders(user_id, created_at);
+SELECT * FROM orders WHERE user_id = 1 AND created_at > '2024-01-01';
+
+-- Index used for:
+-- WHERE user_id = 1
+-- WHERE user_id = 1 AND created_at > ...
+
+-- Index NOT used for:
+-- WHERE created_at > '2024-01-01' (missing leading column)
+```
+
+---
+
+## Query Optimization
+
+### Explain Analyze
+```sql
+EXPLAIN ANALYZE
+SELECT * FROM users WHERE email = 'test@example.com';
+```
+
+### Common Issues
+
+| Issue | Solution |
+|-------|----------|
+| Seq Scan on large table | Add index |
+| High row estimate | Update statistics |
+| Nested Loop on large sets | Consider hash join |
+| Sort in memory | Increase work_mem |
+
+### N+1 Prevention
+```python
+# Bad: N+1
+for user in users:
+    print(user.orders)  # Query per user
+
+# Good: Eager loading
+users = User.query.options(joinedload(User.orders)).all()
+```
+
+---
+
+## Migration Best Practices
+
+### Safe Migrations
+```sql
+-- Add column (safe)
+ALTER TABLE users ADD COLUMN phone VARCHAR(20);
+
+-- Add NOT NULL column (safe pattern)
+ALTER TABLE users ADD COLUMN phone VARCHAR(20);
+UPDATE users SET phone = '' WHERE phone IS NULL;
+ALTER TABLE users ALTER COLUMN phone SET NOT NULL;
+
+-- Rename column (use application-level)
+-- 1. Add new column
+-- 2. Copy data
+-- 3. Update application
+-- 4. Remove old column
+```
+
+### Migration Checklist
+- [ ] Tested on production-like data
+- [ ] Rollback script ready
+- [ ] No long locks on large tables
+- [ ] Indexes created concurrently
+- [ ] Application handles both states
+
+---
+
+## Connection Pooling
+
+### PgBouncer Settings
+```ini
+[pgbouncer]
+pool_mode = transaction
+max_client_conn = 1000
+default_pool_size = 20
+```
+
+### Application Settings
+| Framework | Pool Size Formula |
+|-----------|------------------|
+| General | (cores * 2) + disk spindles |
+| Read-heavy | cores * 4 |
+| Write-heavy | cores * 2 |
+
+---
+
+## Vector Database (Qdrant) Patterns
+
+### Client Setup
+```python
+from qdrant_client import QdrantClient
+from qdrant_client.models import Distance, VectorParams, PointStruct
+
+# Sync client
+client = QdrantClient(host="localhost", port=6333)
+
+# Async client
+from qdrant_client import AsyncQdrantClient
+async_client = AsyncQdrantClient(host="localhost", port=6333)
+```
+
+### Collection Management
+```python
+# Create collection (single vector)
+client.create_collection(
+    collection_name="documents",
+    vectors_config=VectorParams(size=384, distance=Distance.COSINE)
+)
+
+# Create collection (multi-vector)
+from qdrant_client.models import VectorParams
+
+client.create_collection(
+    collection_name="multimodal",
+    vectors_config={
+        "text": VectorParams(size=384, distance=Distance.COSINE),
+        "image": VectorParams(size=512, distance=Distance.EUCLID),
+    }
+)
+```
+
+### Upserting Vectors
+```python
+# Single upsert
+client.upsert(
+    collection_name="documents",
+    points=[
+        PointStruct(
+            id=1,
+            vector=[0.1, 0.2, 0.3, ...],  # 384-dim vector
+            payload={"title": "Doc 1", "category": "tech"}
+        )
+    ]
+)
+
+# Batch upsert
+points = [
+    PointStruct(id=i, vector=vectors[i], payload=payloads[i])
+    for i in range(len(vectors))
+]
+client.upsert(collection_name="documents", points=points, batch_size=100)
+```
+
+### Searching Vectors
+```python
+from qdrant_client.models import Filter, FieldCondition, MatchValue
+
+# Basic search
+results = client.search(
+    collection_name="documents",
+    query_vector=[0.1, 0.2, ...],
+    limit=10
+)
+
+# Search with filter
+results = client.search(
+    collection_name="documents",
+    query_vector=[0.1, 0.2, ...],
+    query_filter=Filter(
+        must=[
+            FieldCondition(key="category", match=MatchValue(value="tech"))
+        ]
+    ),
+    limit=10,
+    with_payload=True,
+    score_threshold=0.7
+)
+
+# Search with range filter
+from qdrant_client.models import Range
+
+results = client.search(
+    collection_name="documents",
+    query_vector=query_vector,
+    query_filter=Filter(
+        must=[
+            FieldCondition(key="price", range=Range(gte=10, lte=100))
+        ]
+    ),
+    limit=10
+)
+```
+
+### Payload Indexing
+```python
+# Create payload index for faster filtering
+client.create_payload_index(
+    collection_name="documents",
+    field_name="category",
+    field_schema="keyword"  # or "integer", "float", "bool"
+)
+```
+
+### Best Practices
+| Aspect | Recommendation |
+|--------|----------------|
+| Batch Size | 100-1000 points per upsert |
+| Vector Dim | Match your embedding model (384, 768, 1536) |
+| Filters | Index frequently filtered fields |
+| Distance | COSINE for normalized, EUCLID for raw |
+| Sharding | Use for >1M vectors |
+
+### Distance Metrics
+| Metric | Best For | Normalized |
+|--------|----------|------------|
+| COSINE | Text embeddings | Yes |
+| EUCLID | Image embeddings | No |
+| DOT | When vectors pre-normalized | Yes |

package/app/skills/debug/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: debug
+description: "Debug errors and trace root causes systematically"
+user-invocable: true
+effort: medium
+argument-hint: "[symptom]"
+agent: debugger
+context: fork
+allowed-tools: Bash, Read, Grep
+---
+
+# Debug Helper
+
+$ARGUMENTS
+
+Systematic debugging for application issues.
+
+## Project context
+
+- Recent logs: !`docker compose logs --tail 20 2>/dev/null || tail -20 logs/*.log 2>/dev/null || echo "no-logs-found"`
+
+## Automated Error Parsing
+
+Pipe error output through the error parser for structured diagnosis:
+
+```bash
+# Pipe from failing command
+your_command 2>&1 | python3 "$(dirname "$0")/scripts/error-parser.py"
+
+# Or from a log file
+cat /var/log/app/error.log | python3 scripts/error-parser.py
+```
+
+The script outputs JSON with:
+- **language**: detected language (python/node/go/php)
+- **error_type**: extracted error class (e.g., ModuleNotFoundError)
+- **message**: the error message text
+- **category**: classification (import, reference, type, connection, timeout, memory, permission, syntax)
+- **stack_frames**: parsed file/line/function from the stack trace
+- **files_to_check**: unique files from the trace, ordered by relevance
+- **common_causes**: likely root causes for this error category
+
+Use the parsed output to focus investigation on the right files and hypotheses.
+
+---
+
+## Debugging Workflow
+
+### 1. Check Logs
+
+```bash
+# Application logs (auto-detect environment)
+# Docker:
+docker compose logs --tail 100 {service} 2>&1 | grep -i error
+
+# Bare metal / systemd:
+journalctl -u {service} --since "1 hour ago" | grep -i error
+
+# Log files:
+tail -100 logs/app.log | grep -i error
+```
+
+### 2. Check Service Health
+
+```bash
+# Docker environment
+docker compose ps
+
+# Process check
+ps aux | grep -E "(node|python|java|php)" | grep -v grep
+
+# HTTP health endpoints
+curl -sf http://localhost:{port}/health
+```
+
+### 3. Interactive Debug
+
+```bash
+# Python
+python3 -c "import module; print(module.function('test'))"
+
+# Node.js
+node -e "const m = require('./module'); console.log(m.fn('test'))"
+
+# PHP
+php -r "require 'vendor/autoload.php'; echo MyClass::method('test');"
+```
+
+### 4. Database Checks
+
+```bash
+# PostgreSQL
+psql -U postgres -c "SELECT version();"
+
+# MySQL
+mysql -e "SELECT VERSION();"
+
+# Redis
+redis-cli ping && redis-cli info memory
+
+# MongoDB
+mongosh --eval "db.runCommand({ping:1})"
+```
+
+## Common Debug Scenarios
+
+### API Returns 500
+```bash
+# Check server logs for stack traces
+grep -A5 "Traceback\|Error\|Exception" logs/app.log
+```
+
+### Slow Performance
+```bash
+# Resource usage
+top -bn1 | head -20     # CPU/memory
+iostat -x 1 3            # Disk I/O
+ss -tlnp                 # Open connections
+```
+
+### Connection Issues
+```bash
+# Test connectivity
+curl -I http://localhost:{port}
+nc -zv {host} {port}
+```
+
+## Parallel Hypothesis Debugging (Agent Teams)
+
+For complex bugs (open >1h, unclear root cause), spawn teammates to investigate competing hypotheses:
+
+```
+Create an agent team to debug this issue:
+- Teammate 1 (debugger): "Investigate if [bug] is caused by [hypothesis A: database issue].
+  Check logs, connection pools, timeouts, query performance."
+  Use Opus.
+- Teammate 2 (debugger): "Investigate if [bug] is caused by [hypothesis B: race condition].
+  Look for async issues, locking, concurrency, shared state."
+  Use Opus.
+- Teammate 3 (debugger): "Investigate if [bug] is caused by [hypothesis C: configuration drift].
+  Compare env vars, config files, recent changes, dependency versions."
+  Use Opus.
+Have them talk to each other to challenge each other's theories.
+Report consensus when done.
+```
+
+## Debug Checklist
+
+- [ ] Identified error/symptom
+- [ ] Checked relevant logs
+- [ ] Verified service health
+- [ ] Reproduced issue
+- [ ] Formed hypothesis
+- [ ] Tested fix

package/app/skills/debug/scripts/error-parser.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+"""Parse error messages and stack traces from stdin for structured diagnosis.
+
+Reads raw error output piped from a failing command or log file,
+detects the programming language (Python, Node.js, Go, PHP), extracts
+the error type, message, and stack frames, classifies the error into a
+category, and outputs a JSON object with common causes and files to
+investigate.
+
+Usage::
+
+    your_command 2>&1 | python3 error-parser.py
+    cat /var/log/app/error.log | python3 error-parser.py
+"""
+from __future__ import annotations
+
+import json
+import re
+import sys
+from typing import Any
+
+
+LANGUAGE_SIGNATURES: dict[str, list[str]] = {
+    "python": [r"Traceback \(most recent call last\)", r"File \".*\", line \d+"],
+    "node": [r"at Object\.<anonymous>", r"at Module\.", r"node:internal", r"at .*\(.*\.js:\d+:\d+\)"],
+    "go": [r"goroutine \d+", r"panic:", r"\.go:\d+"],
+    "php": [r"Fatal error:", r"PHP (?:Warning|Notice|Fatal)", r"Stack trace:", r"\.php:\d+"],
+}
+
+ERROR_CATEGORIES: dict[str, list[str]] = {
+    "import": [r"ModuleNotFoundError", r"ImportError", r"Cannot find module", r"no required module", r"undefined: "],
+    "reference": [r"NameError", r"ReferenceError", r"undefined variable", r"Undefined variable"],
+    "type": [r"TypeError", r"type mismatch", r"cannot use .* as type"],
+    "connection": [r"ConnectionRefusedError", r"ECONNREFUSED", r"connection refused", r"dial tcp"],
+    "timeout": [r"TimeoutError", r"ETIMEDOUT", r"context deadline exceeded", r"Maximum execution time"],
+    "memory": [r"MemoryError", r"heap out of memory", r"out of memory", r"Allowed memory size"],
+    "permission": [r"PermissionError", r"EACCES", r"permission denied", r"Access denied"],
+    "syntax": [r"SyntaxError", r"Unexpected token", r"syntax error", r"Parse error"],
+}
+
+COMMON_CAUSES: dict[str, list[str]] = {
+    "import": ["Missing dependency (run pip install / npm install)", "Typo in module name", "Virtual environment not activated"],
+    "reference": ["Variable used before assignment", "Typo in variable/function name", "Missing import"],
+    "type": ["Wrong argument type passed", "None/null not handled", "API response shape changed"],
+    "connection": ["Service not running", "Wrong host/port", "Firewall blocking connection"],
+    "timeout": ["Service overloaded", "Network latency", "Query too slow - add index"],
+    "memory": ["Large dataset loaded into memory", "Memory leak in loop", "Increase memory limit"],
+    "permission": ["File ownership issue", "Missing sudo/elevation", "SELinux/AppArmor policy"],
+    "syntax": ["Missing bracket/parenthesis", "Invalid JSON/YAML", "Wrong language version"],
+}
+
+
+def detect_language(text: str) -> str:
+    """Detect the programming language from characteristic error patterns.
+
+    Scores each supported language by counting how many of its
+    signature patterns match the input text and returns the language
+    with the highest score.
+
+    Args:
+        text: Raw error output text.
+
+    Returns:
+        Detected language name or ``"unknown"``.
+    """
+    scores: dict[str, int] = {}
+    for lang, patterns in LANGUAGE_SIGNATURES.items():
+        score = sum(1 for p in patterns if re.search(p, text))
+        if score > 0:
+            scores[lang] = score
+    if not scores:
+        return "unknown"
+    return max(scores, key=scores.get)
+
+
+def categorize_error(text: str) -> str:
+    """Categorize error text into a high-level domain.
+
+    Args:
+        text: Raw error output text.
+
+    Returns:
+        Category string (e.g. ``"import"``, ``"connection"``) or
+        ``"unknown"``.
+    """
+    for category, patterns in ERROR_CATEGORIES.items():
+        for p in patterns:
+            if re.search(p, text, re.IGNORECASE):
+                return category
+    return "unknown"
+
+
+def extract_frames(text: str, language: str) -> list[dict[str, Any]]:
+    """Extract stack trace frames based on the detected language.
+
+    Args:
+        text: Raw error output text.
+        language: Detected language (``"python"``, ``"node"``, ``"go"``,
+            ``"php"``).
+
+    Returns:
+        List of frame dictionaries with ``file``, ``line``, and
+        ``function`` keys.
+    """
+    frames: list[dict[str, Any]] = []
+    if language == "python":
+        for m in re.finditer(r'File "([^"]+)", line (\d+)(?:, in (.+))?', text):
+            frames.append({"file": m.group(1), "line": int(m.group(2)), "function": m.group(3) or ""})
+    elif language == "node":
+        for m in re.finditer(r"at (?:(.+?) \()?(.+?):(\d+):(\d+)\)?", text):
+            frames.append({"file": m.group(2), "line": int(m.group(3)), "function": m.group(1) or ""})
+    elif language == "go":
+        for m in re.finditer(r"(\S+\.go):(\d+)", text):
+            frames.append({"file": m.group(1), "line": int(m.group(2)), "function": ""})
+    elif language == "php":
+        for m in re.finditer(r"(?:#\d+\s+)?(\S+\.php)(?:\((\d+)\))?(?::\s*(.+))?", text):
+            frames.append({"file": m.group(1), "line": int(m.group(2)) if m.group(2) else 0, "function": m.group(3) or ""})
+    return frames
+
+
+def extract_error_message(text: str, language: str) -> tuple[str, str]:
+    """Extract the primary error type and message from error output.
+
+    Uses language-specific regex patterns to find the error class and
+    description. Falls back to searching for lines containing common
+    error keywords.
+
+    Args:
+        text: Raw error output text.
+        language: Detected language identifier.
+
+    Returns:
+        A ``(error_type, message)`` tuple.
+    """
+    patterns: dict[str, str] = {
+        "python": r"(\w+Error|\w+Exception|\w+Warning):\s*(.+)",
+        "node": r"(\w+Error|\w+Exception):\s*(.+)",
+        "go": r"(?:panic:\s*(.+)|fatal error:\s*(.+))",
+        "php": r"(?:Fatal error|PHP \w+):\s*(.+?)(?:\s+in\s+|$)",
+    }
+    pattern = patterns.get(language, r"(\w+Error|\w+Exception):\s*(.+)")
+    m = re.search(pattern, text)
+    if m:
+        groups = [g for g in m.groups() if g]
+        if len(groups) >= 2:
+            return groups[0], groups[1].strip()
+        elif groups:
+            return "Error", groups[0].strip()
+    # Fallback: first non-empty line that looks like an error
+    for line in text.split("\n"):
+        line = line.strip()
+        if line and any(kw in line.lower() for kw in ["error", "exception", "fatal", "panic"]):
+            return "Error", line[:200]
+    return "Unknown", "Could not parse error message"
+
+
+def main() -> None:
+    """Entry point: read stdin, parse errors, and print JSON to stdout."""
+    text = sys.stdin.read()
+    if not text.strip():
+        print(json.dumps({"error": "No input provided. Pipe error output to stdin."}))
+        return
+
+    language = detect_language(text)
+    error_type, message = extract_error_message(text, language)
+    category = categorize_error(text)
+    frames = extract_frames(text, language)
+    causes = COMMON_CAUSES.get(category, ["Unknown error category - manual investigation required"])
+
+    # Files to check - unique files from stack frames
+    files_to_check = list(dict.fromkeys(f["file"] for f in frames if f["file"]))[:10]
+
+    result: dict[str, Any] = {
+        "language": language,
+        "error_type": error_type,
+        "message": message,
+        "category": category,
+        "stack_frames": frames[:20],
+        "files_to_check": files_to_check,
+        "common_causes": causes,
+        "raw_lines": len(text.split("\n")),
+    }
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    main()