xtrm-tools 0.7.3 → 0.7.4

package/.xtrm/skills/default/documenting/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: documenting
+description: >-
+  Maintain SSOT documentation with drift detection. Runs drift_detector.py scan
+  on invoke to identify stale memories. Creates/updates Serena memories with
+  auto-generated INDEX blocks. MUST be suggested after any feature, refactor,
+  or architecture change is verified complete.
+gemini-command: document
+gemini-prompt: |
+  1. Identify the task: Create new memory, Update existing, or Validate compliance.
+  2. For finalizing changes, use the orchestrator script to update CHANGELOG.md and README.md.
+  3. Ensure SSOT memories are stored in .serena/memories/ with correct naming conventions.
+  4. Validate YAML frontmatter metadata before completing the task.
+version: 2.0.0
+---
+
+# Documenting Skill
+
+This skill provides workflows and tools for maintaining the Serena Single Source of Truth (SSOT) documentation system/memories.
+
+## 🚨 AGENT GUIDANCE: When to Suggest This Skill
+
+**MANDATORY: Suggest or Use this skill in the following scenarios:**
+
+### ✅ Explicit Triggers:
+- User asks to "document this", "create memory", "update ssot".
+- User asks to "validate documentation", "check metadata".
+- User asks to "list memories", "show patterns".
+- User asks to "bump version" of a document.
+
+### 🤖 Autonomous Triggers (End of Task):
+- **Condition**: A task (feature, refactor, bugfix) is **completed** and **verified** (tests passed).
+- **Action**: The agent MUST check if SSOT documentation needs creation or update.
+- **Guideline**: "Is this a new component? Update `_ssot`. Is this a new pattern? Create `_pattern`. Did I change architecture? Update `_ssot`."
+
+### 📋 Detection Pattern:
+```javascript
+const shouldDocument = 
+  /(document|ssot|memory|metadata|changelog|bump version)/i.test(userMessage) ||
+  (taskCompleted && (newFeature || refactor || architectureChange));
+```
+
+---
+
+## Core Capabilities
+
+1. **Create Memories**: Generate new SSOT documents with correct metadata and structure.
+2. **Update Memories**: Bump versions and maintain changelogs.
+3. **Validate Compliance**: Ensure files follow naming conventions and metadata schemas.
+4. **Navigate**: List and find memories by category.
+
+## Workflows
+
+**🚨 MANDATORY FIRST STEP FOR ALL WORKFLOWS:**
+
+Before using ANY Serena tools, activate the project:
+
+```javascript
+mcp__plugin_serena_serena__activate_project({ project: "/path/to/current/working/directory" })
+```
+
+---
+
+### Step 1: Detect drift
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/drift_detector.py" scan
+```
+
+Review the output. If nothing is stale and no explicit documentation request was made → confirm to user and stop.
+
+### Step 2: Decide action
+
+| Situation | Action |
+|---|---|
+| New feature shipped | Create new SSOT memory OR update existing |
+| Refactor / architecture change | Update relevant SSOT, bump minor version |
+| Bug fix only | CHANGELOG entry only (skip memory update unless behaviour changed) |
+| SKILL.md drift flagged | Update skill + run `validate_metadata.py` on it |
+
+### Step 3: Create or update memory
+
+**Creating a new memory:**
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/generate_template.py" \
+  ssot <name>_ssot.md title="..." domain="..." subcategory="..."
+```
+Fill `[PENDING]` placeholders. Add `tracks:` globs pointing to the files this memory documents.
+
+**Updating an existing memory:**
+1. Read the `<!-- INDEX -->` block only — identify which sections need updating
+2. Use `search_for_pattern` to jump directly to stale sections (avoids reading the full file)
+3. Bump `version:` (patch = content fix, minor = new section, major = full rewrite)
+4. Update `updated:` timestamp to today
+
+### Step 4: Regenerate INDEX
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/validate_metadata.py" <memory-file>
+```
+
+This regenerates the `<!-- INDEX -->` block automatically from the current `##` headings.
+
+### Step 5: Update CHANGELOG
+
+```bash
+python3 "$HOME/.claude/skills/documenting/scripts/changelog/add_entry.py" \
+  <changelog_file> <type> "<summary>"
+```
+
+> `<changelog_file>`: path to target CHANGELOG.md (e.g. `CHANGELOG.md` or `.serena/memories/CHANGELOG.md`)
+
+Types: `Added`, `Changed`, `Fixed`, `Removed`.

package/.xtrm/skills/default/documenting/examples/example_pattern.md

@@ -0,0 +1,70 @@
+---
+title: Circuit Breaker Pattern
+version: 1.1.0
+updated: 2026-01-20T10:00:00Z
+scope: pattern-circuit-breaker
+category: pattern
+subcategory: reliability
+domain: [infra, reliability, resiliency]
+changelog:
+  - 1.1.0 (2026-01-20): Added exponential backoff configuration.
+  - 1.0.0 (2025-12-01): Initial pattern definition.
+---
+
+## Purpose
+Standardize the implementation of the Circuit Breaker pattern to prevent cascading failures when external services are unavailable.
+
+## Pattern Description
+Wrap calls to external services (CME API, Database, Redis) in a circuit breaker that tracks failure rates. When failures exceed a threshold, the breaker "trips" (opens) and immediately fails subsequent calls for a cooldown period, allowing the downstream system to recover.
+
+## When to Apply
+- Any HTTP call to external APIs
+- Database connections
+- RPC calls between internal microservices
+
+## Implementation
+
+### Standard Configuration
+We use `pybreaker` or the internal `ResiliencyClient` wrapper.
+
+**Default Thresholds:**
+- `fail_max`: 5 failures
+- `reset_timeout`: 60 seconds
+
+### Example
+
+```python
+import pybreaker
+from mercury.core.resiliency import circuit_breaker
+
+# Define breaker
+api_breaker = pybreaker.CircuitBreaker(
+    fail_max=5,
+    reset_timeout=60
+)
+
+@api_breaker
+def call_external_api():
+    # ... implementation ...
+    pass
+
+# Or using our wrapper
+@circuit_breaker(service="market_data")
+def fetch_quotes():
+    pass
+```
+
+## Trade-offs
+
+### Benefits
+- Prevents resource exhaustion (thread pools, connections)
+- Fails fast, improving user experience during outages
+- Allows self-healing of downstream systems
+
+### Costs
+- Adds complexity to testing (need to mock failures)
+- Can mask intermittent network blips if calibrated too sensitively
+
+## Related Patterns
+- `pattern_retry_logic_2025-11.md` - Use Retries with Circuit Breaker, not instead of.
+- `pattern_bulkhead_2025-12.md` - Isolating failure domains.

package/.xtrm/skills/default/documenting/examples/example_reference.md

@@ -0,0 +1,70 @@
+---
+title: Database Query Patterns Reference
+scope: reference-db-queries
+category: reference
+subcategory: database
+domain: [data, sql, performance]
+---
+
+## Purpose
+Reference guide for standard SQL query patterns used in the Mercury project to ensure performance and maintainability.
+
+## Naming Conventions
+
+| Object Type | Convention | Example |
+|-------------|------------|---------|
+| Table | `snake_case`, plural | `market_orders` |
+| Column | `snake_case` | `created_at` |
+| Primary Key | `id` | `id` |
+| Foreign Key | `[table_singular]_id` | `user_id` |
+| Index | `idx_[table]_[columns]` | `idx_orders_user_id` |
+
+## Standard Queries
+
+### Time-Series Selection
+Efficiently selecting time-series data using the timescaleDB hypertables.
+
+```sql
+-- GOOD: Uses partition key (time)
+SELECT * FROM ticks
+WHERE symbol = 'ES'
+  AND time >= NOW() - INTERVAL '1 hour'
+ORDER BY time DESC;
+
+-- BAD: Missing time constraint (scans all partitions)
+SELECT * FROM ticks
+WHERE symbol = 'ES'
+ORDER BY time DESC;
+```
+
+### Pagination
+Keyset pagination is preferred over OFFSET/LIMIT for large datasets.
+
+```sql
+-- GOOD: Keyset pagination
+SELECT * FROM orders
+WHERE id < :last_seen_id
+ORDER BY id DESC
+LIMIT 50;
+
+-- BAD: Offset pagination
+SELECT * FROM orders
+ORDER BY id DESC
+LIMIT 50 OFFSET 10000;
+```
+
+### Upsert Pattern
+Standard pattern for inserting or updating records.
+
+```sql
+INSERT INTO volatility_surfaces (expiry, strike, vol, updated_at)
+VALUES (:expiry, :strike, :vol, NOW())
+ON CONFLICT (expiry, strike)
+DO UPDATE SET
+    vol = EXCLUDED.vol,
+    updated_at = NOW();
+```
+
+## Additional Resources
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
+- [TimescaleDB Best Practices](https://docs.timescale.com/)

package/.xtrm/skills/default/documenting/examples/example_ssot_analytics.md

@@ -0,0 +1,64 @@
+---
+title: Analytics Volatility SSOT
+version: 1.0.0
+updated: 2026-01-20T12:00:00Z
+scope: analytics-volatility
+category: ssot
+subcategory: volatility
+domain: [analytics, options, math]
+applicability: core-pricing-engine
+changelog:
+  - 1.0.0 (2026-01-20): Initial baseline creation.
+---
+
+## Purpose
+Define the Single Source of Truth for the Volatility Analytics component, including surface construction, interpolation methods, and smile dynamics.
+
+## Overview
+The Volatility component is responsible for constructing implied volatility surfaces from market data (CME options) and providing interpolation services for the pricing engine.
+
+## Architecture
+
+### Data Flow
+1. **Ingestion**: Raw option ticks -> `ingestion_tick`
+2. **Processing**: Strike alignment and filtering -> `VolatilityProcessor`
+3. **Construction**: Spline fitting -> `SurfaceBuilder`
+4. **Storage**: Coefficients stored in Postgres -> `volatility_surfaces` table
+
+### Key Classes
+- `VolatilitySurface`: Main interface for querying vol at (expiry, strike)
+- `SABRModel`: Implementation of the SABR stochastic volatility model
+- `CubicSplineInterpolator`: Fallback interpolation method
+
+## Current State
+
+### What Works
+- Real-time surface updates for major expirations (Quarterly)
+- SABR calibration for liquid strikes
+- Persistence to database
+
+### Known Limitations
+- Weekly options often have insufficient liquidity for stable calibration
+- Wings (deep OTM) extrapolation can be unstable in high vol environments
+- No support for negative strikes (required for some rates products)
+
+## Configuration
+
+### Environment Variables
+```bash
+VOL_SURFACE_UPDATE_FREQ=60  # seconds
+VOL_MODEL_TYPE=sabr         # or 'spline'
+```
+
+## Related SSOTs
+- `ssot_data_ingestion_tick_2026-01-14.md` - Upstream data source
+- `ssot_analytics_pricing_2025-12-10.md` - Downstream consumer
+
+## Next Steps
+- [ ] Implement wing dampening for deep OTM strikes
+- [ ] Add support for negative strikes (Shifted Lognormal)
+- [ ] Optimize recalibration trigger to ignore noise
+
+## References
+- [Hagan et al. (2002) Managing Smile Risk](https://link-to-paper)
+- `reference_volatility_math.md` (Internal Math Spec)

package/.xtrm/skills/default/documenting/examples/example_workflow.md

@@ -0,0 +1,141 @@
+# Example Workflow: Documenting a New Feature
+
+This example shows the complete workflow for documenting a new feature in a project.
+
+## Scenario
+
+You've just completed a new "semantic search" feature for your project:
+- Added new API endpoint `/api/search`
+- Implemented vector search with ChromaDB
+- Added new configuration options
+- Created comprehensive tests
+
+## Step 1: Document with Orchestrator
+
+```bash
+cd ~/projects/my-project
+
+python3 ~/.claude/skills/documenting/scripts/orchestrator.py . feature 
+  "Semantic search with vector embeddings" 
+  --scope=search 
+  --category=backend
+```
+
+**Output:**
+```
+📝 Documentation Update Results
+============================================================
+✅ CHANGELOG.md updated
+
+💡 README.md suggestions:
+  - Consider adding to ## Features section: Semantic search with vector embeddings
+  - Update ## Usage section with new example
+
+💡 CLAUDE.md suggestions:
+  - Review ## Architecture section for: Semantic search with vector embeddings
+
+✅ All documentation validated
+```
+
+## Step 2: Review CHANGELOG
+
+```bash
+grep -A 10 "\[Unreleased\]" CHANGELOG.md
+```
+
+**Output:**
+```markdown
+## [Unreleased]
+
+### Added
+- Semantic search with vector embeddings
+```
+
+## Step 3: Update README.md
+
+Based on orchestrator suggestions, update README:
+
+```markdown
+## Features
+
+- **Semantic Search**: Natural language search using vector embeddings
+  - Powered by OpenAI embeddings and ChromaDB
+  - Fast approximate nearest neighbor search
+  - Configurable similarity threshold
+```
+
+```markdown
+## Usage
+
+### Semantic Search
+
+```python
+from search import SearchEngine
+
+engine = SearchEngine()
+results = engine.search("find documents about machine learning")
+
+for result in results:
+    print(f"{result.title} (similarity: {result.score})")
+```
+```
+
+## Step 4: Update CLAUDE.md
+
+Based on suggestions, update architecture:
+
+```markdown
+## Architecture
+
+- **Search Layer**: `services/search_service.py` - Vector search implementation
+  - Uses ChromaDB for vector storage
+  - OpenAI embeddings via `embedding_service`
+  - Configurable reranking with FlashRank
+```
+
+## Step 5: Create SSOT Memory
+
+For significant architectural additions:
+
+```bash
+python3 ~/.claude/skills/documenting/scripts/generate_template.py ssot 
+  .serena/memories/ssot_search_semantic_2026-02-01.md 
+  title="Semantic Search SSOT" 
+  scope="search-semantic" 
+  subcategory="search" 
+  domain="backend,ml"
+```
+
+Edit the generated file to document:
+- Search algorithm design
+- Vector storage architecture
+- Embedding strategy
+- Performance characteristics
+
+## Step 6: Validate Everything
+
+```bash
+# Validate CHANGELOG
+python3 ~/.claude/skills/documenting/scripts/changelog/validate_changelog.py CHANGELOG.md
+
+# Validate SSOT
+python3 ~/.claude/skills/documenting/scripts/validate_metadata.py 
+  .serena/memories/ssot_search_semantic_2026-02-01.md
+
+# Or use orchestrator validation
+python3 ~/.claude/skills/documenting/scripts/orchestrator.py . validate
+```
+
+## Step 7: Commit
+
+```bash
+git add 
+  CHANGELOG.md 
+  README.md 
+  CLAUDE.md 
+  .serena/memories/ssot_search_semantic_2026-02-01.md
+
+git commit -m "feat(search): add semantic search with vector embeddings
+
+- Implement vector search using ChromaDB
+- Add OpenAI embedding generation

package/.xtrm/skills/default/documenting/references/changelog-format.md

@@ -0,0 +1,97 @@
+# CHANGELOG Format Reference
+
+This project uses [Keep a Changelog 1.0.0](https://keepachangelog.com/en/1.0.0/) format.
+
+## Structure
+
+```markdown
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- New features
+
+### Changed
+- Changes to existing functionality
+
+### Deprecated
+- Soon-to-be removed features
+
+### Removed
+- Removed features
+
+### Fixed
+- Bug fixes
+
+### Security
+- Security fixes
+
+## [1.0.0] - 2026-02-01
+
+### Added
+- Initial release
+```
+
+## Categories (in order)
+
+1. **Added** - New features
+2. **Changed** - Changes to existing functionality
+3. **Deprecated** - Soon-to-be removed features
+4. **Removed** - Removed features
+5. **Fixed** - Bug fixes
+6. **Security** - Security vulnerabilities fixed
+
+## Version Format
+
+- Use [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
+- Format: `[X.Y.Z] - YYYY-MM-DD`
+- Example: `[1.2.3] - 2026-02-01`
+
+## Entry Guidelines
+
+- Start with action verb
+- Be specific and concise
+- Include context if needed
+- Use bullet points (-)
+- Sub-bullets for details (indented)
+
+### Examples
+
+**Good:**
+```markdown
+### Added
+- **Universal Context**: Support for indexing multiple sources simultaneously
+  - Zero-Config Auto-Discovery for current project
+  - Context-Aware Search with source filtering
+- Reranking with FlashRank (ms-marco-TinyBERT-L-2-v2 model)
+```
+
+**Bad:**
+```markdown
+### Added
+- Added stuff
+- Fixed things
+- Improvements
+```
+
+## Breaking Changes
+
+Mark breaking changes with **BREAKING**: prefix:
+
+```markdown
+### Changed
+- **BREAKING**: Renamed `old_function()` to `new_function()`
+```
+
+## Scripts
+
+- `init_changelog.py` - Create new CHANGELOG.md
+- `add_entry.py` - Add entry to [Unreleased]
+- `bump_release.py` - Move [Unreleased] to version
+- `validate_changelog.py` - Validate format

package/.xtrm/skills/default/documenting/references/metadata-schema.md

@@ -0,0 +1,136 @@
+# SSOT Metadata Schema
+
+This reference defines the required and optional YAML frontmatter fields for Serena memories.
+
+## Schema Overview
+
+Every markdown file in the memories directory **MUST** start with a YAML frontmatter block enclosed in `---`.
+
+```yaml
+---
+title: Human Readable Title
+version: 1.0.0
+updated: 2026-01-20T10:00:00Z
+scope: identifier-slug
+category: ssot
+subcategory: component
+domain: [tag1, tag2]
+---
+```
+
+## Field Definitions
+
+### Required Fields (All Categories)
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `title` | string | Clear, descriptive title of the document | "Analytics Volatility SSOT" |
+| `scope` | string | Unique slug identifying the document's scope | "analytics-volatility" |
+| `category` | enum | One of the standard categories | "ssot" |
+
+### Category-Specific Requirements
+
+#### SSOT (`ssot_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `version` | semver | **Required** | Semantic version (x.y.z) |
+| `updated` | timestamp | **Required** | ISO8601 timestamp of last update |
+| `subcategory` | string | **Required** | Specific component name |
+| `domain` | array | **Required** | List of relevant domain tags |
+| `changelog` | array | **Required** | List of version history entries |
+| `tracks` | array | Optional | Glob patterns (fnmatch) for files this memory documents. Used by drift_detector.py to detect documentation drift. |
+
+#### Pattern (`pattern_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `version` | semver | **Required** | Pattern version |
+| `updated` | timestamp | **Required** | ISO8601 timestamp |
+| `domain` | array | **Required** | Tags for applicability |
+| `tracks` | array | Optional | Glob patterns for files this memory documents. Used by drift_detector.py. |
+
+#### Plan (`plan_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `status` | enum | **Required** | draft, in-progress, completed, abandoned |
+| `plan_ref` | string | Optional | ID of related ticket/issue |
+| `tracks` | array | Optional | Glob patterns for files this memory documents. Used by drift_detector.py. |
+
+#### Reference (`reference_`)
+
+No required category-specific fields beyond the common required fields. The `tracks` field is also available as an optional field (glob patterns for files this memory documents, used by drift_detector.py).
+
+#### Archive (`archive_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `archived_date` | date | **Required** | When it was archived (YYYY-MM-DD) |
+| `replacement` | string | Optional | Link to replacement SSOT |
+
+## Valid Values
+
+### Categories
+- `ssot`
+- `pattern`
+- `plan`
+- `reference`
+- `archive`
+- `troubleshoot`
+
+### Status (for Plans)
+- `draft`
+- `review`
+- `approved`
+- `in-progress`
+- `completed`
+- `on-hold`
+- `cancelled`
+
+## Example Frontmatter
+
+### Complete SSOT Example
+```yaml
+---
+title: Volatility Surface Analytics SSOT
+version: 2.1.0
+updated: 2026-01-14T15:30:00+00:00
+scope: analytics-volatility
+category: ssot
+subcategory: volatility
+domain: [analytics, math, options]
+applicability: core-pricing-engine
+tracks:
+  - "src/analytics/**/*.py"
+  - "src/analytics/config.yaml"
+changelog:
+  - 2.1.0 (2026-01-14): Added SABR model details.
+  - 2.0.0 (2025-12-20): Major refactor of surface construction.
+  - 1.0.0 (2025-11-01): Initial baseline.
+---
+```
+
+### Plan Example
+```yaml
+---
+title: Migration to FastAPI
+version: 0.5.0
+updated: 2025-12-28T10:00:00Z
+scope: infra-api-migration
+category: plan
+subcategory: api
+status: in-progress
+domain: [infra, backend]
+---
+```
+
+### Archive Example
+```yaml
+---
+title: Legacy Flask API Documentation
+version: 1.2.0
+updated: 2025-10-15T00:00:00Z
+scope: legacy-api
+category: archive
+subcategory: api
+archived_date: 2025-12-28
+replacement: ssot_infra_api_architecture_2025-12-28.md
+---
+```