xtrm-tools 2.0.0

package/skills/documenting/references/metadata-schema.md

@@ -0,0 +1,136 @@
+# SSOT Metadata Schema
+
+This reference defines the required and optional YAML frontmatter fields for Serena memories.
+
+## Schema Overview
+
+Every markdown file in the memories directory **MUST** start with a YAML frontmatter block enclosed in `---`.
+
+```yaml
+---
+title: Human Readable Title
+version: 1.0.0
+updated: 2026-01-20T10:00:00Z
+scope: identifier-slug
+category: ssot
+subcategory: component
+domain: [tag1, tag2]
+---
+```
+
+## Field Definitions
+
+### Required Fields (All Categories)
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `title` | string | Clear, descriptive title of the document | "Analytics Volatility SSOT" |
+| `scope` | string | Unique slug identifying the document's scope | "analytics-volatility" |
+| `category` | enum | One of the standard categories | "ssot" |
+
+### Category-Specific Requirements
+
+#### SSOT (`ssot_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `version` | semver | **Required** | Semantic version (x.y.z) |
+| `updated` | timestamp | **Required** | ISO8601 timestamp of last update |
+| `subcategory` | string | **Required** | Specific component name |
+| `domain` | array | **Required** | List of relevant domain tags |
+| `changelog` | array | **Required** | List of version history entries |
+| `tracks` | array | Optional | Glob patterns (fnmatch) for files this memory documents. Used by drift_detector.py to detect documentation drift. |
+
+#### Pattern (`pattern_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `version` | semver | **Required** | Pattern version |
+| `updated` | timestamp | **Required** | ISO8601 timestamp |
+| `domain` | array | **Required** | Tags for applicability |
+| `tracks` | array | Optional | Glob patterns for files this memory documents. Used by drift_detector.py. |
+
+#### Plan (`plan_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `status` | enum | **Required** | draft, in-progress, completed, abandoned |
+| `plan_ref` | string | Optional | ID of related ticket/issue |
+| `tracks` | array | Optional | Glob patterns for files this memory documents. Used by drift_detector.py. |
+
+#### Reference (`reference_`)
+
+No required category-specific fields beyond the common required fields. The `tracks` field is also available as an optional field (glob patterns for files this memory documents, used by drift_detector.py).
+
+#### Archive (`archive_`)
+| Field | Type | Requirement | Description |
+|-------|------|-------------|-------------|
+| `archived_date` | date | **Required** | When it was archived (YYYY-MM-DD) |
+| `replacement` | string | Optional | Link to replacement SSOT |
+
+## Valid Values
+
+### Categories
+- `ssot`
+- `pattern`
+- `plan`
+- `reference`
+- `archive`
+- `troubleshoot`
+
+### Status (for Plans)
+- `draft`
+- `review`
+- `approved`
+- `in-progress`
+- `completed`
+- `on-hold`
+- `cancelled`
+
+## Example Frontmatter
+
+### Complete SSOT Example
+```yaml
+---
+title: Volatility Surface Analytics SSOT
+version: 2.1.0
+updated: 2026-01-14T15:30:00+00:00
+scope: analytics-volatility
+category: ssot
+subcategory: volatility
+domain: [analytics, math, options]
+applicability: core-pricing-engine
+tracks:
+  - "src/analytics/**/*.py"
+  - "src/analytics/config.yaml"
+changelog:
+  - 2.1.0 (2026-01-14): Added SABR model details.
+  - 2.0.0 (2025-12-20): Major refactor of surface construction.
+  - 1.0.0 (2025-11-01): Initial baseline.
+---
+```
+
+### Plan Example
+```yaml
+---
+title: Migration to FastAPI
+version: 0.5.0
+updated: 2025-12-28T10:00:00Z
+scope: infra-api-migration
+category: plan
+subcategory: api
+status: in-progress
+domain: [infra, backend]
+---
+```
+
+### Archive Example
+```yaml
+---
+title: Legacy Flask API Documentation
+version: 1.2.0
+updated: 2025-10-15T00:00:00Z
+scope: legacy-api
+category: archive
+subcategory: api
+archived_date: 2025-12-28
+replacement: ssot_infra_api_architecture_2025-12-28.md
+---
+```

package/skills/documenting/references/taxonomy.md

@@ -0,0 +1,81 @@
+# SSOT Taxonomy & Naming Conventions
+
+This reference defines the standard naming conventions and categorization hierarchy for Serena memories and SSOT documentation.
+
+## Naming Convention
+All memory files must follow the pattern:
+`[subject]_[category].md`
+
+### Components
+
+- --subject--: The primary system, component, or area (e.g., `analytics_volatility`, `infra_docker_ops`
+sion).
+- --category--: The type of document (see Category Suffixes below)
+
+### Category Suffixes
+
+| Category | Suffix | Purpose | Example |
+|-----------|----------|------------------------------------|-------------------------------|
+| **SSOT** | `_ssot` | Single Source of Truth for a component/system | `analytics_stir_ssot.md` |
+|  **Pattern** | `_pattern` | Reusable design pattern or standard | `refactoring_security_pattern.md` |
+|  **Plan** | `_plan` | Implementation plan or roadmap | `implementation_curve_feed_plan.md` |
+|  **Reference** | `_reference` | Look-up tables, cheat sheets, API docs | `database_query_patterns_reference.md` |
+|  **Troubleshoot** | `_troubleshoot` | Guide for resolving specific issues | `docker_port_config_troubleshoot.md` |
+|  **Archive** | `_archive` | Deprecated documentation (kept for history) | `meta_project_overview_archive.md` |
+
+
+## Subject Hierarchy (Domains)
+
+Use these standard domains as prefixes for the **subject** part to group documentation:
+
+### Analytics (`analytics_*`)
+- [volatility]: Volatility surface and modeling
+- [curve]: Yield curve construction
+- [stir]: Short Term Interest Rates
+- [reporting]: End-of-day and ad-hoc reporting
+- [correlation]: Asset correlation matrices
+- [amt]: Automated Market Trading components
+- [path]: Path-dependent option pricing
+- [snapshot_feed]: Real-time data snapshots
+
+### Data (`data_*`)
+- [config]: Instrument and system configuration
+- [ingestion_reliability]: Data pipeline health and monitoring
+- [ingestion_sr3]: SR3 specjfic ingestion
+- [ingestion_tick]: Tick data capture
+
+
+### Infrastructure (`infra_*`)
+- [docker_ops]: Container orchestration and operations
+- [security_migrations]: Security updates and user migration 
+- [mcp_server]: Model Context Protocol server configuration
+- [api_architecture]: FastAPI/Backend architecture
+
+
+### Meta (`meta_*`)
+- [update_guidelines]: Documentation standards (this document)
+- [project_overview]: High-level project goals and status
+- [project_structure]: Codebase organization 
+
+
+### Testing (`testing_*`)
+- [qa]: Quality Assurance processes
+- [integration]: Integration test patterns
+
+## Tagging Strategy
+Use the `domain` frontmatter field (array) for cross-cutting concerns.
+
+**Common Tags:**
+- `fastapi`
+- `docker`
+- `security`
+- `performance`
+- `database`J
+- `refactoring`
+- `deprecation`
+
+## Directory Structure
+
+While all memories currently reside in a flat structure in `.serena/memories/`, the naming convention allows for virtual folder organization.
+
+Future agents may implement physical subdirectories if the flat list exceeds manageable limits (e.g., >200 active SSOTs).

package/skills/documenting/references/versioning-rules.md

@@ -0,0 +1,78 @@
+# Versioning Rules for SSOT
+
+This reference defines the semantic versioning rules for maintaining Serena memories.
+
+## Core Principle
+
+Treat documentation versioning with the same rigor as software versioning. This allows agents to determine if their context is outdated and understand the magnitude of changes.
+
+## Version Format
+
+Use Semantic Versioning 2.0.0: `MAJOR.MINOR.PATCH`
+
+### Format: `x.y.z`
+
+- **x (Major)**: Breaking changes / Rewrite
+- **y (Minor)**: New features / Significant additions
+- **z (Patch)**: Corrections / Clarifications / Minor updates
+
+## Bumping Rules
+
+### Patch (x.y.Z) -> (x.y.Z+1)
+Increment the patch version for:
+- Typos and grammar fixes
+- Clarifications of existing content
+- Adding examples
+- Updating links or references
+- Minor formatting changes
+- Metadata updates (e.g., adding a tag)
+
+### Minor (x.Y.0) -> (x.Y+1.0)
+Increment the minor version for:
+- Adding a new section or component description
+- Extending the scope of the document
+- Documenting new features added to the system
+- Significant rewrites of specific sections that don't change the overall architecture
+- Adding a new known limitation or workaround
+
+### Major (X.0.0) -> (X+1.0.0)
+Increment the major version for:
+- Complete rewrite of the document
+- Major architectural changes to the system described
+- Fundamental change in the document's purpose or scope
+- Marking large sections as deprecated/removed
+- Changing the "Single Source of Truth" authority (e.g., merging two SSOTs)
+
+## Changelog Maintenance
+
+Every SSOT document must have a `changelog` list in the frontmatter.
+
+### Format
+```yaml
+changelog:
+  - VERSION (DATE): SUMMARY
+```
+
+### Guidelines
+- **Reverse Chronological Order**: Newest entries first.
+- **Concise**: Keep summaries under 140 characters if possible.
+- **Date Format**: YYYY-MM-DD.
+
+### Example
+```yaml
+changelog:
+  - 1.2.0 (2026-01-20): Added section on Redis caching strategy.
+  - 1.1.1 (2026-01-15): Fixed typos in configuration examples.
+  - 1.1.0 (2026-01-10): Documented new async processing pipeline.
+  - 1.0.0 (2026-01-01): Initial release.
+```
+
+## Workflow for Updating
+
+1. **Assess Change**: Determine if it's Patch, Minor, or Major.
+2. **Update Content**: Make the changes in the markdown body.
+3. **Update Frontmatter**:
+   - Update `version` field.
+   - Update `updated` timestamp (ISO8601).
+   - Add entry to top of `changelog`.
+4. **Commit**: Save the file.

package/skills/documenting/scripts/bump_version.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+# Semantic version bumping utility for SSOT memories
+#
+# Usage: bump_version.sh <current_version> <bump_type>
+# bump_type: major | minor | patch
+#
+# Examples:
+#   bump_version.sh 1.2.3 patch  -> 1.2.4
+#   bump_version.sh 1.2.3 minor  -> 1.3.0
+#   bump_version.sh 1.2.3 major  -> 2.0.0
+
+set -e
+
+if [ $# -ne 2 ]; then
+    echo "Usage: $0 <current_version> <bump_type>"
+    echo "bump_type: major | minor | patch"
+    echo ""
+    echo "Examples:"
+    echo "  $0 1.2.3 patch  -> 1.2.4"
+    echo "  $0 1.2.3 minor  -> 1.3.0"
+    echo "  $0 1.2.3 major  -> 2.0.0"
+    exit 1
+fi
+
+CURRENT_VERSION=$1
+BUMP_TYPE=$2
+
+# Validate version format
+if ! echo "$CURRENT_VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
+    echo "ERROR: Invalid version format: $CURRENT_VERSION"
+    echo "Expected format: x.y.z (e.g., 1.2.3)"
+    exit 1
+fi
+
+# Parse version components
+IFS='.' read -r MAJOR MINOR PATCH <<< "$CURRENT_VERSION"
+
+# Bump version based on type
+case "$BUMP_TYPE" in
+    major)
+        MAJOR=$((MAJOR + 1))
+        MINOR=0
+        PATCH=0
+        ;;
+    minor)
+        MINOR=$((MINOR + 1))
+        PATCH=0
+        ;;
+    patch)
+        PATCH=$((PATCH + 1))
+        ;;
+    *)
+        echo "ERROR: Invalid bump type: $BUMP_TYPE"
+        echo "Must be one of: major, minor, patch"
+        exit 1
+        ;;
+esac
+
+NEW_VERSION="${MAJOR}.${MINOR}.${PATCH}"
+echo "$NEW_VERSION"

package/skills/documenting/scripts/changelog/add_entry.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""
+Add entry to CHANGELOG.md [Unreleased] section.
+
+Automatically:
+- Places entry in correct category
+- Creates category if missing
+- Maintains category ordering per Keep a Changelog
+- Preserves existing entries
+"""
+
+import re
+import sys
+from enum import Enum
+from pathlib import Path
+from typing import Optional
+
+
+class ChangeCategory(Enum):
+    """Keep a Changelog categories in proper order."""
+    ADDED = "Added"
+    CHANGED = "Changed"
+    DEPRECATED = "Deprecated"
+    REMOVED = "Removed"
+    FIXED = "Fixed"
+    SECURITY = "Security"
+
+
+CATEGORY_ORDER = [cat.value for cat in ChangeCategory]
+
+
+def add_entry(changelog_content: str, category: ChangeCategory, description: str) -> str:
+    """
+    Add entry to [Unreleased] section under specified category.
+
+    Args:
+        changelog_content: Full CHANGELOG.md content
+        category: ChangeCategory enum value
+        description: Entry text (without leading "- ")
+
+    Returns:
+        Updated changelog content
+    """
+    lines = changelog_content.splitlines()
+
+    # Find [Unreleased] section
+    unreleased_idx = None
+    for i, line in enumerate(lines):
+        if re.match(r"^## \[Unreleased\]", line):
+            unreleased_idx = i
+            break
+
+    if unreleased_idx is None:
+        raise ValueError("CHANGELOG missing [Unreleased] section")
+
+    # Find next version section (end of Unreleased)
+    next_version_idx = len(lines)
+    for i in range(unreleased_idx + 1, len(lines)):
+        if re.match(r"^## \[.+\]", lines[i]):
+            next_version_idx = i
+            break
+
+    # Find or create category section
+    category_name = category.value
+    category_idx = None
+
+    for i in range(unreleased_idx + 1, next_version_idx):
+        if lines[i].strip() == f"### {category_name}":
+            category_idx = i
+            break
+
+    if category_idx is None:
+        # Category doesn't exist, create it in proper order
+        category_idx = _insert_category_in_order(
+            lines,
+            unreleased_idx,
+            next_version_idx,
+            category_name
+        )
+        # Update next_version_idx as lines shifted
+        # Wait, if I insert, I need to know how many lines. 
+        # _insert_category_in_order returns the index of the new category header.
+        # But if it inserted blank lines, the indices after it shifted. 
+        # But next_version_idx is an index, so if I insert BEFORE it, it shifts.
+        # Let's check _insert_category_in_order implementation logic.
+        # It mutates 'lines' list.
+        # If it inserts 2 lines, next_version_idx should increase by 2.
+        # But wait, next_version_idx was calculated based on initial list.
+        # I should probably re-calculate or just use the return value.
+        # But wait, the function mutates the list.
+        # Let's look at the implementation below.
+        
+        # Recalculate next_version_idx to be safe? 
+        # Or rely on finding the category index.
+        pass
+
+    # Re-find next version idx just to be safe if lines shifted
+    for i in range(unreleased_idx + 1, len(lines)):
+        if re.match(r"^## \[.+\]", lines[i]):
+            next_version_idx = i
+            break
+    else:
+        next_version_idx = len(lines)
+        
+    # Re-find category index
+    for i in range(unreleased_idx + 1, next_version_idx):
+        if lines[i].strip() == f"### {category_name}":
+            category_idx = i
+            break
+
+    # Find where to insert the entry (after category header)
+    insert_idx = category_idx + 1
+
+    # Skip existing entries to add at end of category
+    while insert_idx < len(lines) and insert_idx < next_version_idx and lines[insert_idx].strip().startswith("- "):
+        insert_idx += 1
+
+    # Insert the new entry
+    entry_line = f"- {description}"
+    lines.insert(insert_idx, entry_line)
+
+    return '\n'.join(lines)
+
+
+def _insert_category_in_order(
+    lines: list,
+    unreleased_idx: int,
+    next_version_idx: int,
+    category_name: str
+) -> int:
+    """
+    Insert category header in proper Keep a Changelog order.
+
+    Returns:
+        Index where category header was inserted
+    """
+    category_order_idx = CATEGORY_ORDER.index(category_name)
+
+    # Find existing categories in Unreleased section
+    existing_categories = []
+    # Note: next_version_idx is based on current lines state
+    for i in range(unreleased_idx + 1, next_version_idx):
+        match = re.match(r"^### (.+)$", lines[i])
+        if match:
+            cat = match.group(1)
+            if cat in CATEGORY_ORDER:
+                existing_categories.append((i, cat))
+
+    # Find insertion point
+    insert_idx = next_version_idx
+    for idx, cat in existing_categories:
+        cat_order_idx = CATEGORY_ORDER.index(cat)
+        if category_order_idx < cat_order_idx:
+            # Insert before this category
+            insert_idx = idx
+            break
+
+    # Insert category header with blank line before if not first
+    if existing_categories:
+        lines.insert(insert_idx, "")
+        insert_idx += 1
+    else:
+        # First category, add blank line after [Unreleased]
+        lines.insert(unreleased_idx + 1, "")
+        insert_idx = unreleased_idx + 2
+
+    lines.insert(insert_idx, f"### {category_name}")
+
+    return insert_idx
+
+
+def add_entry_to_file(
+    filepath: Path,
+    category: ChangeCategory,
+    description: str
+) -> None:
+    """Add entry to CHANGELOG file."""
+    if not filepath.exists():
+        raise FileNotFoundError(f"CHANGELOG not found: {filepath}")
+
+    content = filepath.read_text(encoding='utf-8')
+    updated = add_entry(content, category, description)
+    filepath.write_text(updated, encoding='utf-8')
+
+    print(f"✅ Added to {filepath.name}:")
+    print(f"   [{category.value}] {description}")
+
+
+def main():
+    """CLI entry point."""
+    if len(sys.argv) != 4:
+        print("Usage: add_entry.py <changelog_file> <category> <description>")
+        print("")
+        print("Categories: Added, Changed, Deprecated, Removed, Fixed, Security")
+        print("")
+        print("Example:")
+        print('  add_entry.py CHANGELOG.md Added "New semantic search feature"')
+        sys.exit(1)
+
+    filepath = Path(sys.argv[1])
+    category_str = sys.argv[2]
+    description = sys.argv[3]
+
+    # Validate category
+    try:
+        category = ChangeCategory[category_str.upper()]
+    except KeyError:
+        print(f"ERROR: Invalid category '{category_str}'")
+        print(f"Valid: {', '.join(CATEGORY_ORDER)}")
+        sys.exit(1)
+
+    add_entry_to_file(filepath, category, description)
+
+
+if __name__ == "__main__":
+    main()