xtrm-tools 2.0.0

package/skills/documenting/README.md

@@ -0,0 +1,148 @@
+# Documenting Skill
+
+Comprehensive documentation management for projects using:
+- **Serena SSOT** (.serena/memories/)
+- **CHANGELOG.md** (Keep a Changelog format)
+- **README.md**
+- **CLAUDE.md / AGENT.md**
+
+## Quick Start
+
+### Initialize Project Documentation
+
+```bash
+# Create CHANGELOG.md
+python3 scripts/changelog/init_changelog.py ./CHANGELOG.md
+
+# Create .serena/memories directory
+mkdir -p .serena/memories
+```
+
+### Document a Change
+
+**Using Orchestrator (Recommended)**:
+
+```bash
+python3 scripts/orchestrator.py . feature "Add new search feature" \
+  --scope=search \
+  --category=backend
+```
+
+**Manual Workflow**:
+
+1. Update CHANGELOG:
+   ```bash
+   python3 scripts/changelog/add_entry.py CHANGELOG.md Added "New feature"
+   ```
+
+2. Create SSOT (if needed):
+   ```bash
+   python3 scripts/generate_template.py ssot .serena/memories/ssot_search_engine_2026-02-01.md \
+     title="Search Engine SSOT" \
+     scope="search-engine" \
+     domain="backend"
+   ```
+
+3. Update README.md and CLAUDE.md manually
+
+4. Validate:
+   ```bash
+   python3 scripts/changelog/validate_changelog.py CHANGELOG.md
+   python3 scripts/validate_metadata.py .serena/memories/ssot_*.md
+   ```
+
+### Release a Version
+
+```bash
+# Bump CHANGELOG
+python3 scripts/changelog/bump_release.py CHANGELOG.md 1.2.0
+
+# Validate
+python3 scripts/changelog/validate_changelog.py CHANGELOG.md
+
+# Commit and tag
+git add CHANGELOG.md
+git commit -m "chore: release v1.2.0"
+git tag -a v1.2.0 -m "Release 1.2.0"
+```
+
+## Scripts Reference
+
+### CHANGELOG Management
+
+| Script | Purpose | Example |
+|--------|---------|---------|
+| `init_changelog.py` | Create new CHANGELOG.md | `init_changelog.py ./CHANGELOG.md` |
+| `add_entry.py` | Add entry to [Unreleased] | `add_entry.py CHANGELOG.md Added "Feature X"` |
+| `bump_release.py` | Release new version | `bump_release.py CHANGELOG.md 1.2.0` |
+| `validate_changelog.py` | Validate format | `validate_changelog.py CHANGELOG.md` |
+
+### SSOT Management
+
+| Script | Purpose | Example |
+|--------|---------|---------|
+| `generate_template.py` | Create new SSOT | `generate_template.py ssot file.md title="X"` |
+| `validate_metadata.py` | Validate SSOT metadata | `validate_metadata.py file.md` |
+| `bump_version.sh` | Calculate next version | `bump_version.sh 1.0.0 patch` |
+
+### Orchestration
+
+| Script | Purpose | Example |
+|--------|---------|---------|
+| `orchestrator.py` | Coordinate all docs | `orchestrator.py . feature "X"` |
+
+## File Structure
+
+```
+~/.claude/skills/documenting/
+├── SKILL.md                      # Skill definition
+├── README.md                     # This file
+├── scripts/
+│   ├── changelog/
+│   │   ├── init_changelog.py    # Create CHANGELOG
+│   │   ├── add_entry.py         # Add entries
+│   │   ├── bump_release.py      # Release versions
+│   │   └── validate_changelog.py # Validate format
+│   ├── orchestrator.py          # Coordination
+│   ├── generate_template.py     # SSOT templates
+│   ├── validate_metadata.py     # SSOT validation
+│   └── bump_version.sh          # Version bumping
+├── templates/
+│   └── CHANGELOG.md.template    # CHANGELOG template
+├── references/
+│   ├── metadata-schema.md       # SSOT schema
+│   ├── changelog-format.md      # Keep a Changelog guide
+│   └── taxonomy.md              # SSOT taxonomy
+└── tests/
+    ├── test_changelog.py        # CHANGELOG tests
+    └── test_orchestrator.py     # Orchestrator tests
+```
+
+## Change Type Guide
+
+- **feature**: New functionality → CHANGELOG: Added
+- **bugfix**: Bug fixes → CHANGELOG: Fixed
+- **refactor**: Code restructuring → CHANGELOG: Changed
+- **breaking**: Breaking changes → CHANGELOG: Changed (with **BREAKING** prefix)
+- **docs**: Documentation → CHANGELOG: Changed
+- **chore**: Maintenance → CHANGELOG: Changed
+
+## Validation
+
+All scripts validate before modifying files:
+- CHANGELOG: Keep a Changelog 1.0.0 format
+- SSOT: Serena metadata schema
+- Versions: Semantic Versioning 2.0.0
+
+## Testing
+
+```bash
+# Run all tests
+python -m pytest tests/
+
+# Test specific module
+python -m pytest tests/test_changelog.py -v
+
+# Integration test
+bash tests/integration_test.sh
+```

package/skills/documenting/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: documenting
+description: >-
+  Maintain SSOT documentation with drift detection. Runs drift_detector.py scan
+  on invoke to identify stale memories. Creates/updates Serena memories with
+  auto-generated INDEX blocks. MUST be suggested after any feature, refactor,
+  or architecture change is verified complete.
+gemini-command: document
+gemini-prompt: |
+  1. Identify the task: Create new memory, Update existing, or Validate compliance.
+  2. For finalizing changes, use the orchestrator script to update CHANGELOG.md and README.md.
+  3. Ensure SSOT memories are stored in .serena/memories/ with correct naming conventions.
+  4. Validate YAML frontmatter metadata before completing the task.
+version: 2.0.0
+---
+
+# Documenting Skill
+
+This skill provides workflows and tools for maintaining the Serena Single Source of Truth (SSOT) documentation system/memories.
+
+## 🚨 AGENT GUIDANCE: When to Suggest This Skill
+
+**MANDATORY: Suggest or Use this skill in the following scenarios:**
+
+### ✅ Explicit Triggers:
+- User asks to "document this", "create memory", "update ssot".
+- User asks to "validate documentation", "check metadata".
+- User asks to "list memories", "show patterns".
+- User asks to "bump version" of a document.
+
+### 🤖 Autonomous Triggers (End of Task):
+- **Condition**: A task (feature, refactor, bugfix) is **completed** and **verified** (tests passed).
+- **Action**: The agent MUST check if SSOT documentation needs creation or update.
+- **Guideline**: "Is this a new component? Update `_ssot`. Is this a new pattern? Create `_pattern`. Did I change architecture? Update `_ssot`."
+
+### 📋 Detection Pattern:
+```javascript
+const shouldDocument = 
+  /(document|ssot|memory|metadata|changelog|bump version)/i.test(userMessage) ||
+  (taskCompleted && (newFeature || refactor || architectureChange));
+```
+
+---
+
+## Core Capabilities
+
+1. **Create Memories**: Generate new SSOT documents with correct metadata and structure.
+2. **Update Memories**: Bump versions and maintain changelogs.
+3. **Validate Compliance**: Ensure files follow naming conventions and metadata schemas.
+4. **Navigate**: List and find memories by category.
+
+## Workflows
+
+**🚨 MANDATORY FIRST STEP FOR ALL WORKFLOWS:**
+
+Before using ANY Serena tools, activate the project:
+
+```javascript
+mcp__plugin_serena_serena__activate_project({ project: "/path/to/current/working/directory" })
+```
+
+---
+
+### Step 1: Detect drift
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/drift_detector.py" scan
+```
+
+Review the output. If nothing is stale and no explicit documentation request was made → confirm to user and stop.
+
+### Step 2: Decide action
+
+| Situation | Action |
+|---|---|
+| New feature shipped | Create new SSOT memory OR update existing |
+| Refactor / architecture change | Update relevant SSOT, bump minor version |
+| Bug fix only | CHANGELOG entry only (skip memory update unless behaviour changed) |
+| SKILL.md drift flagged | Update skill + run `validate_metadata.py` on it |
+
+### Step 3: Create or update memory
+
+**Creating a new memory:**
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/generate_template.py" \
+  ssot <name>_ssot.md title="..." domain="..." subcategory="..."
+```
+Fill `[PENDING]` placeholders. Add `tracks:` globs pointing to the files this memory documents.
+
+**Updating an existing memory:**
+1. Read the `<!-- INDEX -->` block only — identify which sections need updating
+2. Use `search_for_pattern` to jump directly to stale sections (avoids reading the full file)
+3. Bump `version:` (patch = content fix, minor = new section, major = full rewrite)
+4. Update `updated:` timestamp to today
+
+### Step 4: Regenerate INDEX
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/validate_metadata.py" <memory-file>
+```
+
+This regenerates the `<!-- INDEX -->` block automatically from the current `##` headings.
+
+### Step 5: Update CHANGELOG
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/changelog/add_entry.py" \
+  <changelog_file> <type> "<summary>"
+```
+
+> `<changelog_file>`: path to target CHANGELOG.md (e.g. `CHANGELOG.md` or `.serena/memories/CHANGELOG.md`)
+
+Types: `Added`, `Changed`, `Fixed`, `Removed`.

package/skills/documenting/examples/example_pattern.md

@@ -0,0 +1,70 @@
+---
+title: Circuit Breaker Pattern
+version: 1.1.0
+updated: 2026-01-20T10:00:00Z
+scope: pattern-circuit-breaker
+category: pattern
+subcategory: reliability
+domain: [infra, reliability, resiliency]
+changelog:
+  - 1.1.0 (2026-01-20): Added exponential backoff configuration.
+  - 1.0.0 (2025-12-01): Initial pattern definition.
+---
+
+## Purpose
+Standardize the implementation of the Circuit Breaker pattern to prevent cascading failures when external services are unavailable.
+
+## Pattern Description
+Wrap calls to external services (CME API, Database, Redis) in a circuit breaker that tracks failure rates. When failures exceed a threshold, the breaker "trips" (opens) and immediately fails subsequent calls for a cooldown period, allowing the downstream system to recover.
+
+## When to Apply
+- Any HTTP call to external APIs
+- Database connections
+- RPC calls between internal microservices
+
+## Implementation
+
+### Standard Configuration
+We use `pybreaker` or the internal `ResiliencyClient` wrapper.
+
+**Default Thresholds:**
+- `fail_max`: 5 failures
+- `reset_timeout`: 60 seconds
+
+### Example
+
+```python
+import pybreaker
+from mercury.core.resiliency import circuit_breaker
+
+# Define breaker
+api_breaker = pybreaker.CircuitBreaker(
+    fail_max=5,
+    reset_timeout=60
+)
+
+@api_breaker
+def call_external_api():
+    # ... implementation ...
+    pass
+
+# Or using our wrapper
+@circuit_breaker(service="market_data")
+def fetch_quotes():
+    pass
+```
+
+## Trade-offs
+
+### Benefits
+- Prevents resource exhaustion (thread pools, connections)
+- Fails fast, improving user experience during outages
+- Allows self-healing of downstream systems
+
+### Costs
+- Adds complexity to testing (need to mock failures)
+- Can mask intermittent network blips if calibrated too sensitively
+
+## Related Patterns
+- `pattern_retry_logic_2025-11.md` - Use Retries with Circuit Breaker, not instead of.
+- `pattern_bulkhead_2025-12.md` - Isolating failure domains.

package/skills/documenting/examples/example_reference.md

@@ -0,0 +1,70 @@
+---
+title: Database Query Patterns Reference
+scope: reference-db-queries
+category: reference
+subcategory: database
+domain: [data, sql, performance]
+---
+
+## Purpose
+Reference guide for standard SQL query patterns used in the Mercury project to ensure performance and maintainability.
+
+## Naming Conventions
+
+| Object Type | Convention | Example |
+|-------------|------------|---------|
+| Table | `snake_case`, plural | `market_orders` |
+| Column | `snake_case` | `created_at` |
+| Primary Key | `id` | `id` |
+| Foreign Key | `[table_singular]_id` | `user_id` |
+| Index | `idx_[table]_[columns]` | `idx_orders_user_id` |
+
+## Standard Queries
+
+### Time-Series Selection
+Efficiently selecting time-series data using the timescaleDB hypertables.
+
+```sql
+-- GOOD: Uses partition key (time)
+SELECT * FROM ticks
+WHERE symbol = 'ES'
+  AND time >= NOW() - INTERVAL '1 hour'
+ORDER BY time DESC;
+
+-- BAD: Missing time constraint (scans all partitions)
+SELECT * FROM ticks
+WHERE symbol = 'ES'
+ORDER BY time DESC;
+```
+
+### Pagination
+Keyset pagination is preferred over OFFSET/LIMIT for large datasets.
+
+```sql
+-- GOOD: Keyset pagination
+SELECT * FROM orders
+WHERE id < :last_seen_id
+ORDER BY id DESC
+LIMIT 50;
+
+-- BAD: Offset pagination
+SELECT * FROM orders
+ORDER BY id DESC
+LIMIT 50 OFFSET 10000;
+```
+
+### Upsert Pattern
+Standard pattern for inserting or updating records.
+
+```sql
+INSERT INTO volatility_surfaces (expiry, strike, vol, updated_at)
+VALUES (:expiry, :strike, :vol, NOW())
+ON CONFLICT (expiry, strike)
+DO UPDATE SET
+    vol = EXCLUDED.vol,
+    updated_at = NOW();
+```
+
+## Additional Resources
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
+- [TimescaleDB Best Practices](https://docs.timescale.com/)

package/skills/documenting/examples/example_ssot_analytics.md

@@ -0,0 +1,64 @@
+---
+title: Analytics Volatility SSOT
+version: 1.0.0
+updated: 2026-01-20T12:00:00Z
+scope: analytics-volatility
+category: ssot
+subcategory: volatility
+domain: [analytics, options, math]
+applicability: core-pricing-engine
+changelog:
+  - 1.0.0 (2026-01-20): Initial baseline creation.
+---
+
+## Purpose
+Define the Single Source of Truth for the Volatility Analytics component, including surface construction, interpolation methods, and smile dynamics.
+
+## Overview
+The Volatility component is responsible for constructing implied volatility surfaces from market data (CME options) and providing interpolation services for the pricing engine.
+
+## Architecture
+
+### Data Flow
+1. **Ingestion**: Raw option ticks -> `ingestion_tick`
+2. **Processing**: Strike alignment and filtering -> `VolatilityProcessor`
+3. **Construction**: Spline fitting -> `SurfaceBuilder`
+4. **Storage**: Coefficients stored in Postgres -> `volatility_surfaces` table
+
+### Key Classes
+- `VolatilitySurface`: Main interface for querying vol at (expiry, strike)
+- `SABRModel`: Implementation of the SABR stochastic volatility model
+- `CubicSplineInterpolator`: Fallback interpolation method
+
+## Current State
+
+### What Works
+- Real-time surface updates for major expirations (Quarterly)
+- SABR calibration for liquid strikes
+- Persistence to database
+
+### Known Limitations
+- Weekly options often have insufficient liquidity for stable calibration
+- Wings (deep OTM) extrapolation can be unstable in high vol environments
+- No support for negative strikes (required for some rates products)
+
+## Configuration
+
+### Environment Variables
+```bash
+VOL_SURFACE_UPDATE_FREQ=60  # seconds
+VOL_MODEL_TYPE=sabr         # or 'spline'
+```
+
+## Related SSOTs
+- `ssot_data_ingestion_tick_2026-01-14.md` - Upstream data source
+- `ssot_analytics_pricing_2025-12-10.md` - Downstream consumer
+
+## Next Steps
+- [ ] Implement wing dampening for deep OTM strikes
+- [ ] Add support for negative strikes (Shifted Lognormal)
+- [ ] Optimize recalibration trigger to ignore noise
+
+## References
+- [Hagan et al. (2002) Managing Smile Risk](https://link-to-paper)
+- `reference_volatility_math.md` (Internal Math Spec)

package/skills/documenting/examples/example_workflow.md

@@ -0,0 +1,141 @@
+# Example Workflow: Documenting a New Feature
+
+This example shows the complete workflow for documenting a new feature in a project.
+
+## Scenario
+
+You've just completed a new "semantic search" feature for your project:
+- Added new API endpoint `/api/search`
+- Implemented vector search with ChromaDB
+- Added new configuration options
+- Created comprehensive tests
+
+## Step 1: Document with Orchestrator
+
+```bash
+cd ~/projects/my-project
+
+python3 ~/.claude/skills/documenting/scripts/orchestrator.py . feature 
+  "Semantic search with vector embeddings" 
+  --scope=search 
+  --category=backend
+```
+
+**Output:**
+```
+📝 Documentation Update Results
+============================================================
+✅ CHANGELOG.md updated
+
+💡 README.md suggestions:
+  - Consider adding to ## Features section: Semantic search with vector embeddings
+  - Update ## Usage section with new example
+
+💡 CLAUDE.md suggestions:
+  - Review ## Architecture section for: Semantic search with vector embeddings
+
+✅ All documentation validated
+```
+
+## Step 2: Review CHANGELOG
+
+```bash
+grep -A 10 "\[Unreleased\]" CHANGELOG.md
+```
+
+**Output:**
+```markdown
+## [Unreleased]
+
+### Added
+- Semantic search with vector embeddings
+```
+
+## Step 3: Update README.md
+
+Based on orchestrator suggestions, update README:
+
+```markdown
+## Features
+
+- **Semantic Search**: Natural language search using vector embeddings
+  - Powered by OpenAI embeddings and ChromaDB
+  - Fast approximate nearest neighbor search
+  - Configurable similarity threshold
+```
+
+```markdown
+## Usage
+
+### Semantic Search
+
+```python
+from search import SearchEngine
+
+engine = SearchEngine()
+results = engine.search("find documents about machine learning")
+
+for result in results:
+    print(f"{result.title} (similarity: {result.score})")
+```
+```
+
+## Step 4: Update CLAUDE.md
+
+Based on suggestions, update architecture:
+
+```markdown
+## Architecture
+
+- **Search Layer**: `services/search_service.py` - Vector search implementation
+  - Uses ChromaDB for vector storage
+  - OpenAI embeddings via `embedding_service`
+  - Configurable reranking with FlashRank
+```
+
+## Step 5: Create SSOT Memory
+
+For significant architectural additions:
+
+```bash
+python3 ~/.claude/skills/documenting/scripts/generate_template.py ssot 
+  .serena/memories/ssot_search_semantic_2026-02-01.md 
+  title="Semantic Search SSOT" 
+  scope="search-semantic" 
+  subcategory="search" 
+  domain="backend,ml"
+```
+
+Edit the generated file to document:
+- Search algorithm design
+- Vector storage architecture
+- Embedding strategy
+- Performance characteristics
+
+## Step 6: Validate Everything
+
+```bash
+# Validate CHANGELOG
+python3 ~/.claude/skills/documenting/scripts/changelog/validate_changelog.py CHANGELOG.md
+
+# Validate SSOT
+python3 ~/.claude/skills/documenting/scripts/validate_metadata.py 
+  .serena/memories/ssot_search_semantic_2026-02-01.md
+
+# Or use orchestrator validation
+python3 ~/.claude/skills/documenting/scripts/orchestrator.py . validate
+```
+
+## Step 7: Commit
+
+```bash
+git add 
+  CHANGELOG.md 
+  README.md 
+  CLAUDE.md 
+  .serena/memories/ssot_search_semantic_2026-02-01.md
+
+git commit -m "feat(search): add semantic search with vector embeddings
+
+- Implement vector search using ChromaDB
+- Add OpenAI embedding generation

package/skills/documenting/references/changelog-format.md

@@ -0,0 +1,97 @@
+# CHANGELOG Format Reference
+
+This project uses [Keep a Changelog 1.0.0](https://keepachangelog.com/en/1.0.0/) format.
+
+## Structure
+
+```markdown
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- New features
+
+### Changed
+- Changes to existing functionality
+
+### Deprecated
+- Soon-to-be removed features
+
+### Removed
+- Removed features
+
+### Fixed
+- Bug fixes
+
+### Security
+- Security fixes
+
+## [1.0.0] - 2026-02-01
+
+### Added
+- Initial release
+```
+
+## Categories (in order)
+
+1. **Added** - New features
+2. **Changed** - Changes to existing functionality
+3. **Deprecated** - Soon-to-be removed features
+4. **Removed** - Removed features
+5. **Fixed** - Bug fixes
+6. **Security** - Security vulnerabilities fixed
+
+## Version Format
+
+- Use [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
+- Format: `[X.Y.Z] - YYYY-MM-DD`
+- Example: `[1.2.3] - 2026-02-01`
+
+## Entry Guidelines
+
+- Start with action verb
+- Be specific and concise
+- Include context if needed
+- Use bullet points (-)
+- Sub-bullets for details (indented)
+
+### Examples
+
+**Good:**
+```markdown
+### Added
+- **Universal Context**: Support for indexing multiple sources simultaneously
+  - Zero-Config Auto-Discovery for current project
+  - Context-Aware Search with source filtering
+- Reranking with FlashRank (ms-marco-TinyBERT-L-2-v2 model)
+```
+
+**Bad:**
+```markdown
+### Added
+- Added stuff
+- Fixed things
+- Improvements
+```
+
+## Breaking Changes
+
+Mark breaking changes with **BREAKING**: prefix:
+
+```markdown
+### Changed
+- **BREAKING**: Renamed `old_function()` to `new_function()`
+```
+
+## Scripts
+
+- `init_changelog.py` - Create new CHANGELOG.md
+- `add_entry.py` - Add entry to [Unreleased]
+- `bump_release.py` - Move [Unreleased] to version
+- `validate_changelog.py` - Validate format