xtrm-tools 2.4.0 → 2.4.2

package/skills/creating-service-skills/references/script_quality_standards.md

@@ -0,0 +1,425 @@
+# Script Quality Standards for Service Skills
+
+> Distilled from the mercury-market-data implementation (Feb 2026).
+> Updated with lessons from the processing-papers implementation (Feb 2026).
+> Apply these standards to every script generated in Phase 2 of the service-skill-builder workflow.
+
+---
+
+## Table of Contents
+
+- [Mandatory DB Connection Pattern](#mandatory-db-connection-pattern)
+- [Schema Verification Before Writing Any SQL](#schema-verification-before-writing-any-sql)
+- [Makefile Standard](#makefile-standard)
+- [Design Principles](#design-principles)
+- [health_probe.py Standards](#health_probepy-standards)
+- [log_hunter.py Standards](#log_hunterpy-standards)
+- [Specialist Script Standards](#specialist-script-standards)
+- [Common Pitfalls](#common-pitfalls)
+
+---
+
+## Mandatory DB Connection Pattern
+
+**Every script that touches the database MUST use this exact pattern.** No exceptions.
+
+```python
+#!/usr/bin/env python3
+import sys
+from pathlib import Path
+from dotenv import load_dotenv
+
+# Resolve project root (depth depends on script location within .claude/skills/)
+project_root = Path(__file__).resolve().parent.parent.parent.parent.parent
+env_file = project_root / ".env"
+if env_file.exists():
+    load_dotenv(str(env_file))
+
+sys.path.insert(0, str(project_root))
+from shared.db_pool_manager import execute_db_query
+```
+
+**Why:** System `python3` may lack `dotenv` and project deps. Always test with `venv/bin/python3`.
+**Never:** Raw `psycopg2`, hardcoded DSN strings, or skipping the `load_dotenv` call.
+
+---
+
+## Schema Verification Before Writing Any SQL
+
+Run these queries against the live DB **before** writing any script SQL. Paste the results
+into the delegation prompt so the agent never guesses column or table names.
+
+```sql
+-- Step 1: Confirm which tables actually exist
+SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename;
+
+-- Step 2: For each output table — get exact column names and types
+SELECT column_name, data_type, is_nullable
+FROM information_schema.columns
+WHERE table_name = '<your_table>'
+ORDER BY ordinal_position;
+```
+
+**Critical rule:** If a table has no timestamp column, use `COUNT(*)` for freshness checks —
+never guess a column name. Verify with `information_schema.columns` first.
+
+---
+
+## Makefile Standard
+
+Every `scripts/` directory MUST contain a `Makefile` with these standard targets.
+This is auto-generated by the scaffolder in Phase 1 and should be updated in Phase 2
+to add any service-specific targets.
+
+```makefile
+# Skill diagnostic scripts for <service-id>
+# Usage: make <target>   (from this directory)
+# Override python: make health PYTHON=../../venv/bin/python3
+
+PYTHON := python3
+
+.PHONY: health health-json data data-json logs errors db help
+
+help:
+	@echo "Available targets:"
+	@echo "  health      - Run health probe (human readable)"
+	@echo "  health-json - Run health probe (JSON output)"
+	@echo "  data        - Show latest DB records"
+	@echo "  data-json   - Show latest DB records (JSON, limit 5)"
+	@echo "  logs        - Tail and analyze recent logs"
+	@echo "  errors      - Show errors/criticals only"
+	@echo "  db          - Run DB helper example queries"
+
+health:
+	$(PYTHON) health_probe.py
+
+health-json:
+	$(PYTHON) health_probe.py --json
+
+data:
+	$(PYTHON) data_explorer.py
+
+data-json:
+	$(PYTHON) data_explorer.py --json --limit 5
+
+logs:
+	$(PYTHON) log_hunter.py --tail 50
+
+errors:
+	$(PYTHON) log_hunter.py --errors-only --tail 50
+
+db:
+	$(PYTHON) db_helper.py
+```
+
+---
+
+## Design Principles
+
+### 1. Service-Specific, Not Generic
+
+The single most important rule. Generic scripts provide zero value.
+
+**Wrong (generic stub output):**
+```python
+error_patterns = [
+    r"(ERROR|CRITICAL|FATAL|EXCEPTION)",
+    r"ConnectionError",
+    r"SyntaxError",
+    r"ImportError",
+]
+```
+
+**Right (service-specific, sourced from actual codebase):**
+```python
+PATTERNS = [
+    # From yfinance source: actual exception class names
+    ("Rate limit",    r"YFRateLimitError|429.*yahoo|Too Many Requests",  "error"),
+    ("Missing data",  r"YFPricesMissingError|no timezone found|Period.*invalid", "warning"),
+    # From DB layer: actual psycopg2 messages
+    ("DB connect",    r"could not connect|password authentication failed",  "critical"),
+    ("DB write",      r"relation.*does not exist|column.*does not exist",   "error"),
+]
+```
+
+**How to find real patterns:** Read the service's entry point script, exception handlers, and log statements. Search for `logger.error`, `raise`, `except` blocks, and error message strings.
+
+---
+
+### 2. Port Awareness: Host vs. Container
+
+Scripts in `skills/` run on the host machine, not inside Docker. Port mappings matter.
+
+| Context | Use This Port |
+|---------|--------------|
+| Host scripts (`skills/*.py`) | External mapped port (e.g., `5433` for TimescaleDB `5433:5432`) |
+| Docker service env vars | Internal port (`5432`) |
+| `docker exec` commands | N/A — resolves via container DNS |
+
+**Always use env vars with correct defaults:**
+```python
+DB_HOST = os.getenv("DB_HOST", "localhost")
+DB_PORT = int(os.getenv("DB_PORT", "5433"))   # External mapped port
+```
+
+---
+
+### 3. Read-Before-Write Discipline
+
+When a stub file already exists and you are replacing it, **always read it first**. Write tools fail with "File has not been read yet" otherwise. New files (no existing content) can be created directly.
+
+---
+
+### 4. Dual Output Mode
+
+Every script must support both human-readable (default) and machine-readable (`--json`) output.
+
+```python
+parser.add_argument("--json", action="store_true", help="Output as JSON")
+
+if args.json:
+    print(json.dumps(result, indent=2, default=str))
+    return
+# ... human-readable output below
+```
+
+---
+
+### 5. Actionable Remediation in Output
+
+When a health probe or log hunter detects a critical problem, it must print the exact fix command — not a generic "check the logs."
+
+```python
+if by_sev["critical"]:
+    print(f"\n  ⚠  Critical issues detected.")
+    if any("OAuth expired" in h["labels"] for h in by_sev["critical"]):
+        print(f"     Fix: docker exec -it {CONTAINER} python scripts/auth.py --refresh")
+    if any("DB connect" in h["labels"] for h in by_sev["critical"]):
+        print(f"     Fix: docker compose restart timescaledb && docker compose restart {CONTAINER}")
+```
+
+---
+
+## health_probe.py Standards
+
+### Structure
+
+```python
+CONTAINER = "service-name"   # Exact Docker container name
+
+def check_container() -> dict:
+    """docker inspect for status. Always present."""
+    ...
+
+def check_<domain>() -> dict:
+    """Service-specific check (DB freshness, file presence, HTTP endpoint, etc.)"""
+    ...
+
+def main():
+    # 1. Collect all checks
+    # 2. --json: dump report dict
+    # 3. Human: print formatted table
+    # 4. Print fix commands on failure
+```
+
+### For DB-writing services: Freshness Table
+
+```python
+# Define per-table stale thresholds based on service update frequency
+FRESHNESS_CHECKS = [
+    # (table_name,          timestamp_col,  stale_threshold_minutes)
+    ("candles_5m",          "timestamp",    30),    # 5m feed → stale if >30m old
+    ("outright_snapshots",  "snapshot_ts",  10),    # continuous → stale if >10m old
+    ("volatility_snapshots","snapshot_ts",  1500),  # daily job → stale if >25h old
+]
+```
+
+Stale threshold logic: `update_interval × 3` is a reasonable default, but adjust for business criticality.
+
+### For HTTP API services: Endpoint Probing
+
+Do not ping generic ports. Probe the actual API routes the service exposes:
+
+```python
+HEALTH_ENDPOINTS = [
+    ("FastAPI health",    "http://localhost:8000/api/system/health", 3),
+    ("Background server", "http://localhost:5002/health",            2),
+]
+# Optional smoke tests against real data endpoints
+SMOKE_ENDPOINTS = [
+    ("Market snapshot",  "http://localhost:8000/api/market/snapshot"),
+    ("Volatility data",  "http://localhost:8000/api/analytics/volatility"),
+]
+```
+
+### For one-shot services (migrations, backfills): Exit Code
+
+```python
+# docker inspect returns status="exited" and exit_code="0" on success
+result = subprocess.run(
+    ["docker", "inspect", "--format",
+     "{{.State.Status}} {{.State.ExitCode}} {{.State.FinishedAt}}",
+     CONTAINER],
+    capture_output=True, text=True
+)
+```
+
+A one-shot service is healthy if `exit_code == "0"` and expected tables/schemas exist in the DB.
+
+### For file watcher services: Mount + State File
+
+```python
+def check_scid_mount() -> dict:
+    result = subprocess.run(
+        ["docker", "exec", CONTAINER, "ls", "/data/scid"],
+        capture_output=True, text=True, timeout=10
+    )
+    files = [f for f in result.stdout.splitlines() if f.endswith(".scid")]
+    return {"accessible": result.returncode == 0, "file_count": len(files)}
+
+def check_state_file() -> dict:
+    result = subprocess.run(
+        ["docker", "exec", CONTAINER, "cat", "/app/state/watcher_state.json"],
+        capture_output=True, text=True
+    )
+    if result.returncode != 0:
+        return {"present": False}
+    return {"present": True, "state": json.loads(result.stdout)}
+```
+
+---
+
+## log_hunter.py Standards
+
+### Pattern Structure
+
+```python
+PATTERNS = [
+    # (label,           regex_pattern,              severity)
+    ("OAuth expired",   r"invalid_grant|token.*expired",  "critical"),
+    ("PDF parse",       r"PdfReadError|pdf.*format.*changed", "error"),
+    ("No data",         r"No new.*report|0 reports.*found",   "warning"),
+    ("Report saved",    r"report.*ingested|saved.*DB",         "info"),
+]
+```
+
+**Severity levels:**
+- `critical`: Service needs restart or manual intervention to recover
+- `error`: Functionality is impaired, data may be incomplete
+- `warning`: Degraded state, worth monitoring
+- `info`: Normal operation confirmation
+
+### Severity Ordering
+
+Always use `sev_order` so that the highest severity "wins" when a line matches multiple patterns:
+
+```python
+sev_order = {"critical": 0, "error": 1, "warning": 2, "info": 3}
+if matched_severity is None or sev_order[severity] < sev_order[matched_severity]:
+    matched_severity = severity
+```
+
+### Required CLI Flags
+
+```python
+parser.add_argument("--tail",        type=int,  default=200)
+parser.add_argument("--since",       type=str,  default=None)   # Docker --since (e.g. "1h", "2026-01-01")
+parser.add_argument("--errors-only", action="store_true")       # Skip info entries
+parser.add_argument("--json",        action="store_true")
+```
+
+### Pattern Design Rules
+
+1. Test patterns against the **actual log format** of the service, not hypothetical messages.
+2. Use `re.IGNORECASE` — log levels and messages vary in capitalization.
+3. Prefer specific class names (`YFPricesMissingError`) over generic keywords (`Error`).
+4. For Rust services, add: `r"thread '.*' panicked|panicked at '"` as a critical pattern.
+5. Always include at least 2 `info` patterns for normal operation confirmation — so the absence of info lines itself becomes a signal.
+
+### Anti-patterns to avoid
+
+| Anti-pattern | Why It Fails |
+|---|---|
+| `r"ERROR"` | Matches comment text, variable names, and dozens of false positives |
+| `r"Exception"` | Too broad — every Python `try/except` emits this |
+| `r"ConnectionError"` | Only catches one subclass; misses `OperationalError`, `InterfaceError`, etc. |
+| Single `error_patterns` list without severity | Provides no triage — everything looks equally bad |
+
+---
+
+## Specialist Script Standards
+
+### data_explorer.py (for DB-writing services)
+
+Purpose: Let an agent query the service's output tables interactively without writing SQL.
+
+```python
+# Always support:
+parser.add_argument("--symbol",  help="Filter to a specific symbol")
+parser.add_argument("--history", action="store_true", help="Show time series, not just latest")
+parser.add_argument("--limit",   type=int, default=20)
+parser.add_argument("--json",    action="store_true")
+```
+
+Use `DISTINCT ON (symbol) ... ORDER BY symbol, timestamp DESC` for "latest per symbol" queries. Use parameterized queries: `WHERE symbol = %s`.
+
+### endpoint_tester.py (for HTTP API services)
+
+Test every real route in the API, not just `/health`. Measure response time and size:
+
+```python
+ENDPOINTS = [
+    # (label, method, path, expected_status, timeout_s)
+    ("Health check",    "GET", "/api/system/health",          200, 3),
+    ("Market overview", "GET", "/api/market/overview",        200, 5),
+    ("Symbol detail",   "GET", "/api/market/ES=F",            200, 5),
+    # ... all actual routes
+]
+```
+
+Report slow endpoints (>2s) separately from failed ones.
+
+### state_inspector.py (for stateful file watchers)
+
+Read the state file via `docker exec` and compute lag between current file size and processed byte offset:
+
+```python
+scid_size  = get_file_size_in_container(container, filepath)
+lag_bytes  = scid_size - state["byte_offset"]
+lag_flag   = " ⚠" if lag_bytes > 1_000_000 else ""
+```
+
+### coverage_checker.py (for one-shot backfill services)
+
+Report per-entity (spread, symbol, etc.) row counts, date ranges, and gaps:
+
+```sql
+SELECT entity_id, COUNT(*) AS rows,
+       MIN(ts) AS earliest, MAX(ts) AS latest
+FROM output_table
+GROUP BY entity_id ORDER BY entity_id;
+```
+
+Also detect missing entities against a known expected list, and find time-series gaps using `LAG()`.
+
+---
+
+## Common Pitfalls
+
+| Pitfall | Prevention |
+|---------|-----------|
+| Script uses port 5432 from host | Default to 5433 (external mapped port); document the discrepancy |
+| Script uses HTTP port scanning instead of real routes | Read docker-compose to find actual port mappings; check the service's API routes |
+| OAuth token path is wrong | `docker exec container ls /expected/path` to verify before hardcoding |
+| **DB table name is guessed** | Run `SELECT tablename FROM pg_tables WHERE schemaname='public'` first; include output in delegation prompt |
+| **DB column name is guessed** | Run `SELECT column_name, data_type FROM information_schema.columns WHERE table_name='X'` per table; include in delegation prompt |
+| **Assumed timestamp column on every table** | Check `information_schema.columns` — if no timestamp exists, use `COUNT(*)` for freshness; never guess |
+| `try` block with no matching `except` | Every DB call needs a complete `try/except`; bare `try` blocks crash silently |
+| Function renamed but call sites not updated | After any rename, grep the scripts dir for the old name before finishing |
+| Delegation with no `-y` flag (Qwen) | Qwen requires `-y` for non-interactive file writes; without it, research happens but no files are written |
+| Using `ccs gemini` instead of `gemini -p` | Gemini: `gemini -p "..."` · Qwen: `qwen -y "..."` · GLM: `env -u CLAUDECODE ccs glm -p "..."` |
+| Scripts tested with system python3 | Always test with `venv/bin/python3`; system python may lack dotenv and other deps |
+| Log patterns too broad | Read the actual `logger.error()` calls in the source code |
+| Missing `--since` flag | Log hunters without `--since` can't be used for incremental monitoring |
+| `health_probe.py` doesn't print fix commands | Always add actionable remediation text after detecting critical states |
+| No `scripts/Makefile` | Every skill must have a Makefile with standard targets; scaffolder generates it in Phase 1 |

package/skills/creating-service-skills/references/service_skill_system_guide.md

@@ -0,0 +1,278 @@
+# Service Skill System: Architecture & Operations Guide
+
+> Distilled from real-world Docker microservices projects.
+> This guide is project-agnostic — adapt all examples to your stack.
+
+---
+
+## Table of Contents
+
+- [1. System Overview](#1-system-overview)
+- [2. System Architecture](#2-system-architecture)
+- [3. Mandatory Two-Phase Workflow](#3-mandatory-two-phase-workflow)
+- [4. Service Type Classification](#4-service-type-classification)
+- [5. Directory Structure](#5-directory-structure)
+- [6. Skill Lifecycle](#6-skill-lifecycle)
+- [7. Quality Gates](#7-quality-gates)
+- [8. Best Practices](#8-best-practices)
+- [9. Anti-Patterns](#9-anti-patterns)
+
+---
+
+## 1. System Overview
+
+The **Service Skill System** transforms an AI agent from a generic assistant into a service-aware operator. Each Docker service in your project gets a dedicated **skill package**: a structured combination of operational documentation and executable diagnostic scripts.
+
+### What a Skill Provides
+
+| Layer | Contents | Purpose |
+|-------|----------|---------|
+| `SKILL.md` | Operational manual | How the service works, how to debug it |
+| `scripts/health_probe.py` | Container + data freshness checks | Is the service healthy right now? |
+| `scripts/log_hunter.py` | Pattern-based log analysis | What is the service logging and why? |
+| `scripts/<specialist>.py` | Service-specific inspector | What state does this service hold? |
+
+Without scripts, a skill is documentation only. Without documentation, scripts have no context. Both are required.
+
+---
+
+## 2. System Architecture
+
+### Three Components
+
+**A. The Builder (`service-skill-builder`)**
+The meta-skill that generates other skills.
+- **Input**: `docker-compose*.yml`, Dockerfiles, entry-point source code
+- **Engine**: `scripts/main.py` (Phase 1 skeleton generator)
+- **Output**: `SKILL.md`, `REFINEMENT_BRIEF.md`, stub scripts → then replaced in Phase 2
+
+**B. The Health Checker (`scripts/skill_health_check.py`)**
+Detects drift between skills and the live codebase.
+- Compares service modification timestamps vs. skill generation timestamps
+- Identifies services with no skill (coverage gaps)
+- Reports stale skills needing a re-dive
+
+**C. The Generated Skills**
+Individual packages per service (e.g., `.claude/skills/my-service/`).
+
+---
+
+## 3. Mandatory Two-Phase Workflow
+
+**Phase 1 and Phase 2 are both required. The skeleton alone is never sufficient.**
+
+### Phase 1: Automated Skeleton
+
+Run the generator against your project root:
+
+```bash
+# Discover all Docker services
+python3 .claude/skills/service-skill-builder/scripts/main.py --scan
+
+# Generate skeleton for one service
+python3 .claude/skills/service-skill-builder/scripts/main.py <service-name>
+```
+
+The skeleton provides:
+- Structural facts: port mappings, env var names, image names, volumes
+- `REFINEMENT_BRIEF.md` listing every open question
+- Generic stub scripts (placeholder only — **must be replaced**)
+
+**The skeleton cannot tell you:**
+- What the service actually writes to the database (column names, stale thresholds)
+- What real error messages look like in the logs
+- What "healthy" vs. "degraded" vs. "failed" looks like
+- What exact commands fix common failures
+
+### Phase 2: Agentic Deep Dive
+
+Read the source code. Answer every question in `REFINEMENT_BRIEF.md` using `Grep`, `Glob`, `Read`, and Serena LSP tools. Do not guess. Do not leave placeholders.
+
+**Mandatory investigation areas:**
+
+#### Container & Runtime
+- What is the exact entry point? (Dockerfile CMD + docker-compose `command:`)
+- Is this a long-running daemon, a cron job, or a one-shot? → determines health strategy
+- Which env vars cause a crash if missing? Which are optional?
+- What volumes does it read from? Write to?
+- What is the restart policy and why?
+
+#### Data Layer
+- Which tables does it write? Which does it only read?
+- What is the timestamp column for each output table (`created_at`, `snapshot_ts`, `asof_ts`, etc.)?
+- What is a realistic "stale" threshold in minutes per table? (Rule of thumb: update_interval × 3)
+- Does it use Redis, S3, local files, or other external state?
+- Are queries parameterized? (Check `%s`, `%(name)s`, `?` patterns — never f-strings in SQL)
+
+#### Failure Modes
+Build this table with ≥5 rows from code comments, exception handlers, and READMEs:
+
+| Symptom | Likely Cause | Resolution |
+|---------|-------------|------------|
+| (what you see in logs or alerts) | (root cause) | (exact docker/shell command to fix) |
+
+#### Log Patterns
+Search for `logger.error`, `logger.warning`, `raise`, `except`, and `panic!` in the source:
+- What appears in logs during normal healthy operation? (→ `info` patterns)
+- What appears during recoverable errors? (→ `warning` / `error` patterns)
+- What appears during critical failures requiring restart? (→ `critical` patterns)
+- For Rust services: what does a panic look like? (`thread '.*' panicked`)
+
+---
+
+## 4. Service Type Classification
+
+Classify before writing scripts. The service type determines which scripts to write beyond the baseline `health_probe.py` and `log_hunter.py`.
+
+| Service Type | Health Probe Strategy | Specialist Script |
+|---|---|---|
+| **Continuous DB writer** | Table freshness (age of most recent row per table) | `data_explorer.py` |
+| **HTTP API server** | HTTP probe against real routes (not just port scan) | `endpoint_tester.py` |
+| **One-shot / migration** | Container exit code + expected tables/schemas present | `coverage_checker.py` |
+| **File watcher** | Mount path accessible + state file present + DB recency | `state_inspector.py` |
+| **Email / API poller** | Container running + auth token file present | service-specific |
+| **Scheduled backup** | Recent backup files in staging dir + daemon running | service-specific |
+| **MCP stdio server** | Data source freshness in DB (no HTTP to probe) | service-specific |
+
+---
+
+## 5. Directory Structure
+
+```
+.claude/skills/
+├── service-skill-builder/          # Meta-skill (system core)
+│   ├── SKILL.md
+│   ├── references/
+│   │   ├── service_skill_system_guide.md   # This file
+│   │   └── script_quality_standards.md     # Script design rules
+│   └── scripts/
+│       ├── main.py                 # Phase 1 skeleton generator
+│       ├── skill_health_check.py   # Drift detection
+│       ├── discovery.py            # Docker Compose parser
+│       ├── analysis.py             # AST/regex code analyzer
+│       ├── devops_audit.py         # CI/CD/observability audit
+│       └── generator.py            # Skill file generation logic
+│
+├── my-service-a/                   # Generated skill (long-running daemon)
+│   ├── SKILL.md
+│   └── scripts/
+│       ├── health_probe.py         # Container + DB freshness checks
+│       ├── log_hunter.py           # Pattern-matched log analysis
+│       └── data_explorer.py        # Query output tables interactively
+│
+├── my-service-b/                   # Generated skill (HTTP API)
+│   ├── SKILL.md
+│   └── scripts/
+│       ├── health_probe.py
+│       ├── log_hunter.py
+│       └── endpoint_tester.py      # Probe all real API routes
+│
+└── my-service-c/                   # Generated skill (file watcher)
+    ├── SKILL.md
+    └── scripts/
+        ├── health_probe.py
+        ├── log_hunter.py
+        └── state_inspector.py      # Read state file, compute lag
+```
+
+Agent mirrors — always sync after creating or updating skills:
+
+```bash
+for d in .claude/skills/my-*/; do
+  svc=$(basename "$d")
+  cp -r "$d" ".agent/skills/$svc/"
+  cp -r "$d" ".gemini/skills/$svc/"
+done
+```
+
+---
+
+## 6. Skill Lifecycle
+
+### When to Generate a Skill
+- A new Docker service is added to the project
+- An existing service is significantly refactored
+
+### When to Update a Skill
+- The service's database schema changes
+- New error conditions are added to the code
+- The entry point or restart policy changes
+- The health check script's stale thresholds no longer reflect reality
+
+### Detecting Drift
+
+```bash
+# Check all skills for staleness
+python3 .claude/skills/service-skill-builder/scripts/skill_health_check.py --all
+```
+
+Output example:
+```
+my-service-a: HEALTHY
+my-service-b: STALE (service code modified 2026-01-15, skill generated 2025-11-01)
+my-service-c: MISSING (no skill exists)
+```
+
+A skill is **STALE** when the service's source code or docker-compose definition has been modified more recently than the skill was generated. This is a signal to re-run Phase 2 for the affected service.
+
+---
+
+## 7. Quality Gates
+
+A skill is **complete** (not draft) when all of the following are true:
+
+- [ ] No `[PENDING RESEARCH]` markers remain in SKILL.md
+- [ ] All stub scripts have been replaced with service-specific implementations
+- [ ] `health_probe.py` queries actual output tables with correct stale thresholds
+- [ ] `log_hunter.py` patterns are sourced from the real codebase (not invented)
+- [ ] At least one specialist script exists if the service has unique inspectable state
+- [ ] The Troubleshooting table has ≥5 rows based on real failure modes
+- [ ] All CLI commands in SKILL.md are verified against the actual docker-compose config
+- [ ] Scripts have been synced to `.agent/skills/` and `.gemini/skills/` mirrors
+
+---
+
+## 8. Best Practices
+
+### One Service, One Skill
+Keep skills granular. A skill for `my-api` should not also document `my-worker`. Tightly coupled services (e.g., Redis master/replica) may share a skill if they are always operated together.
+
+### Read Source, Not Docs
+Internal README files go stale. The entry point script, exception handlers, and log statements are the ground truth. Always grep the source code for actual error messages before writing log patterns.
+
+### Port Awareness
+Scripts in `skills/` run on the **host machine**, not inside Docker. Always use the external mapped port:
+
+```python
+# ✅ Host script (external mapped port)
+DB_PORT = int(os.getenv("DB_PORT", "5433"))
+
+# ❌ Wrong for a host script (container-internal port)
+DB_PORT = int(os.getenv("DB_PORT", "5432"))
+```
+
+### Executable Knowledge
+Prefer putting logic into `scripts/` (executed without reading into context) over text-only descriptions in SKILL.md. An agent that can run `health_probe.py` learns the truth about service health in one step. An agent reading stale prose may act on incorrect assumptions.
+
+### Actionable Remediation
+Every critical failure detected by a script must print the exact command to fix it — not "check the logs." For example:
+
+```python
+if not token_present:
+    print(f"  Fix: docker exec -it {CONTAINER} python scripts/auth.py --refresh")
+```
+
+---
+
+## 9. Anti-Patterns
+
+| Anti-pattern | Why It Fails |
+|---|---|
+| Skip Phase 2 because Phase 1 looks complete | Skeleton has correct port numbers but wrong table names, wrong log patterns, wrong stale thresholds |
+| Copy log patterns from another service's skill | Different services emit different errors; shared patterns produce false positives and miss real failures |
+| Use port 5432 in host scripts | Container-internal port is unreachable from host; scripts silently hang |
+| Write `health_probe.py` without fix commands | Agent sees a failure but has no recovery path |
+| Leave `[PENDING RESEARCH]` markers | The skill is unusable — an agent acting on incomplete info may apply wrong fixes |
+| Forget to sync to `.agent/` and `.gemini/` | Other agent runtimes use stale or missing skills |
+| Use `r"ERROR"` as a log pattern | Matches variable names, comments, thousands of false positives |
+| Hardcode table names without verifying | `SELECT tablename FROM pg_tables WHERE schemaname='public'` first |