@smilintux/skmemory 0.5.0 → 0.9.2

package/skill.yaml

@@ -0,0 +1,46 @@
+name: skmemory
+version: "0.6.0"
+description: >
+  Persistent sovereign memory layer. Short-term, mid-term, and long-term
+  stores with HMAC integrity seals and automatic promotion. Backends for
+  file, SQLite, vector, and graph storage.
+
+author:
+  name: smilinTux.org
+  email: hello@smilintux.org
+
+tags: [memory, persistence, sovereign, integrity, sqlite, vector]
+category: core
+
+tools:
+  - name: memory_store
+    description: Store a new memory with tags and importance score
+    entrypoint: skmemory.skill:memory_store
+
+  - name: memory_search
+    description: Full-text search across all memory layers
+    entrypoint: skmemory.skill:memory_search
+
+  - name: memory_recall
+    description: Recall a specific memory by ID
+    entrypoint: skmemory.skill:memory_recall
+
+  - name: memory_list
+    description: List memories filtered by layer, tags, or importance
+    entrypoint: skmemory.skill:memory_list
+
+  - name: memory_forget
+    description: Delete a memory by ID
+    entrypoint: skmemory.skill:memory_forget
+
+  - name: memory_promote
+    description: Promote a memory to a higher persistence layer
+    entrypoint: skmemory.skill:memory_promote
+
+  - name: memory_health
+    description: Check memory store health and statistics
+    entrypoint: skmemory.skill:memory_health
+
+  - name: memory_verify
+    description: Verify HMAC integrity seals on stored memories
+    entrypoint: skmemory.skill:memory_verify

package/skmemory/HA.md

@@ -0,0 +1,296 @@
+# SKMemory High Availability & Routing for SKVector and SKGraph Backends
+
+> Self-contained endpoint routing for SKVector and SKGraph backends.
+> No external load balancer. No new dependencies. Backward compatible.
+
+## Overview
+
+SKMemory's SKVector and SKGraph backends can run on multiple nodes across a
+Tailscale mesh (or any network). The **EndpointSelector** sits between config
+resolution and backend construction: it discovers endpoints, probes their
+latency, selects the fastest healthy one, and fails over automatically.
+
+Key properties:
+- **On-demand probing** with TTL cache (no background threads)
+- **Config endpoints take precedence** over heartbeat discovery
+- **Graceful degradation** — missing heartbeats, Tailscale, or config all fail silently
+- **Backward compatible** — single `skvector_url` configs work unchanged
+
+## Architecture
+
+### Routing Layer Diagram
+
+```mermaid
+graph TB
+    CLI[skmemory CLI / Agent] --> ES[EndpointSelector]
+    ES --> |probe| EP1[SKVector @ home:6333]
+    ES --> |probe| EP2[SKVector @ vps:6333]
+    ES --> |probe| EP3[SKGraph @ home:6379]
+    ES --> |probe| EP4[SKGraph @ cloud:6379]
+    ES --> |select best| MS[MemoryStore]
+    MS --> SQLite[SQLite Primary]
+    MS --> SKVectorBackend[SKVectorBackend]
+    MS --> SKGraphBackend[SKGraphBackend]
+    HB[Heartbeat Mesh] -.->|discover| ES
+
+    style ES fill:#f9f,stroke:#333
+    style MS fill:#bbf,stroke:#333
+```
+
+The selector picks a URL, then backends are created with that URL. Backend
+internals are **never modified** — the selector is a pure URL resolver.
+
+### Endpoint Selection Flowchart
+
+```mermaid
+flowchart TD
+    A[Load config.yaml] --> B{Multi-endpoint<br/>or heartbeat_discovery?}
+    B -->|No| C[Use single URL as-is]
+    B -->|Yes| D[Build EndpointSelector]
+    D --> E{Heartbeat<br/>discovery enabled?}
+    E -->|Yes| F[Read ~/.skcapstone/heartbeats/*.json]
+    F --> G[Merge discovered endpoints]
+    E -->|No| G
+    G --> H{Probe results<br/>stale? >30s}
+    H -->|Yes| I[TCP probe all endpoints]
+    H -->|No| J[Use cached results]
+    I --> K[Apply routing strategy]
+    J --> K
+    K --> L[Return best URL]
+    L --> M[Create backend with URL]
+```
+
+### Failover Sequence
+
+```mermaid
+sequenceDiagram
+    participant CLI as skmemory CLI
+    participant ES as EndpointSelector
+    participant A as SKVector @ home:6333
+    participant B as SKVector @ vps:6333
+
+    CLI->>ES: select_skvector()
+    ES->>A: TCP probe (port 6333)
+    A--xES: Connection refused
+    Note over ES: fail_count++ (1/3)
+    ES->>B: TCP probe (port 6333)
+    B-->>ES: Connected (12ms)
+    ES-->>CLI: vps:6333 (healthy)
+
+    Note over CLI: Later, home comes back...
+    CLI->>ES: select_skvector()
+    ES->>A: TCP probe (port 6333)
+    A-->>ES: Connected (2ms)
+    Note over ES: fail_count reset to 0
+    ES-->>CLI: home:6333 (lowest latency)
+```
+
+### Heartbeat Discovery Flow
+
+```mermaid
+sequenceDiagram
+    participant Peer as Agent @ VPS
+    participant FS as Syncthing
+    participant Local as Local Agent
+    participant ES as EndpointSelector
+
+    Peer->>FS: heartbeat.json<br/>{services: [{name: "skvector", port: 6333}],<br/>tailscale_ip: "100.64.0.5"}
+    FS->>Local: Sync heartbeat file
+    Local->>ES: discover_from_heartbeats()
+    ES->>ES: Parse services from heartbeat
+    ES->>ES: Build URL: http://100.64.0.5:6333
+    ES->>ES: Add as replica endpoint
+    ES->>ES: Probe new endpoint
+    Note over ES: Endpoint available for routing
+```
+
+## Routing Strategies
+
+### `failover` (default)
+
+The simplest strategy. Uses the first healthy endpoint in the list.
+If it goes down, moves to the next one.
+
+| Reads | Writes | Use Case |
+|-------|--------|----------|
+| First healthy | First healthy | Simple HA, single primary |
+
+### `latency`
+
+Always picks the endpoint with the lowest measured TCP latency.
+Best for globally distributed agents where network proximity matters.
+
+| Reads | Writes | Use Case |
+|-------|--------|----------|
+| Lowest latency healthy | Lowest latency healthy | Globally distributed agents |
+
+### `local-first`
+
+Prefers `localhost` / `127.0.0.1` if available, then falls back to
+lowest latency. The most common strategy — agents that have a local
+Docker stack should use it.
+
+| Reads | Writes | Use Case |
+|-------|--------|----------|
+| localhost if available | localhost if available | Prefer local Docker stack |
+
+### `read-local-write-primary`
+
+Reads go to the closest healthy endpoint (local preference).
+Writes go **only** to endpoints with `role: primary`.
+Good for setups with one write primary and multiple read replicas.
+
+| Reads | Writes | Use Case |
+|-------|--------|----------|
+| Lowest latency | Only `role=primary` | Eventual consistency OK |
+
+## Configuration
+
+### Single Node (backward compatible)
+
+No changes needed. Old configs work as before:
+
+```yaml
+# ~/.skmemory/config.yaml
+skvector_url: http://localhost:6333
+skgraph_url: redis://localhost:6379
+```
+
+Or via environment variables:
+```bash
+export SKMEMORY_SKVECTOR_URL=http://localhost:6333
+export SKMEMORY_SKGRAPH_URL=redis://localhost:6379
+```
+
+### Multi-Node (Tailscale mesh)
+
+```yaml
+# ~/.skmemory/config.yaml
+skvector_endpoints:
+  - url: http://localhost:6333
+    role: primary
+  - url: http://100.64.0.5:6333
+    role: replica
+    tailscale_ip: "100.64.0.5"
+
+skgraph_endpoints:
+  - url: redis://localhost:6379
+    role: primary
+  - url: redis://100.64.0.5:6379
+    role: replica
+
+routing_strategy: local-first
+```
+
+### Global Distribution
+
+```yaml
+# ~/.skmemory/config.yaml
+skvector_endpoints:
+  - url: https://us-east.qdrant.example.com:6333
+    role: primary
+  - url: https://eu-west.qdrant.example.com:6333
+    role: replica
+  - url: https://ap-south.qdrant.example.com:6333
+    role: replica
+
+routing_strategy: latency
+```
+
+### Heartbeat Auto-Discovery
+
+Let agents find each other's backends via the heartbeat mesh:
+
+```yaml
+# ~/.skmemory/config.yaml
+skvector_url: http://localhost:6333
+routing_strategy: latency
+heartbeat_discovery: true
+```
+
+Agents that run `skcapstone heartbeat pulse` will advertise any locally
+detected services (SKVector on 6333, SKGraph on 6379). Other agents read
+these heartbeats and add the endpoints automatically.
+
+## CLI Commands
+
+```bash
+# Show endpoint rankings, latency, and health
+skmemory routing status
+
+# Force re-probe all endpoints
+skmemory routing probe
+```
+
+## What We Built This For (Ideal Use Case)
+
+A sovereign agent running on a home server with SKVector and SKGraph in Docker.
+A second instance runs on a VPS. Both are connected via Tailscale. When the
+home server goes down for maintenance, the agent on the VPS automatically
+routes to its local SKVector instance (or another VPS peer). When the home
+server comes back, agents detect the lower latency and route back.
+
+No ops team. No service mesh. No external load balancer. Just config and
+heartbeats.
+
+## Pros & Challenges
+
+### Strengths
+
+- **Zero-downtime maintenance** — drain a node, agents auto-route elsewhere
+- **Self-contained** — no external load balancer, service mesh, or ops team
+- **Self-healing** — unhealthy endpoints auto-recover when they come back
+- **Latency-optimized** — agents always use the closest backend
+- **Backward compatible** — single URL configs work unchanged
+- **No new dependencies** — stdlib `socket` for probing
+- **Cross-platform** — works on Linux, macOS, Windows
+
+### Challenges & Solutions
+
+| Challenge | Solution | When It Matters |
+|-----------|----------|----------------|
+| Write consistency with multiple primaries | Use `read-local-write-primary` — one write endpoint | Multiple writers to same data |
+| Stale reads from replicas | Acceptable for memory search (not bank transactions) | Real-time requirements |
+| Probe overhead on CLI startup | Lazy probing with 30s cache — first call probes, subsequent use cache | High-frequency CLI usage |
+| Heartbeat directory not available | Graceful fallback to config-only endpoints | skcapstone not installed |
+| Tailscale IP detection fails | Falls back to hostname | Non-Tailscale deployments |
+| Large number of endpoints (50+) | Probe only top-N by last known latency | Enterprise scale |
+
+### What This Doesn't Solve
+
+- **Cross-region write replication** — Use Qdrant distributed mode or external replication
+- **Strong consistency** — This is eventual consistency; fine for AI memory, not for transactions
+- **Automatic replica provisioning** — You still deploy Qdrant/FalkorDB manually; routing just finds them
+
+## Future Scaling
+
+### Qdrant Distributed Mode
+
+For 10M+ memories, Qdrant's built-in sharding distributes data across nodes.
+The endpoint selector would point to the Qdrant cluster entry point; Qdrant
+handles internal routing.
+
+### Redis Sentinel for FalkorDB
+
+Redis Sentinel provides automatic primary election for FalkorDB. The selector
+could query Sentinel for the current primary instead of probing directly.
+
+### Write Consistency
+
+For strong write consistency across regions:
+1. Single-writer topology (one primary, N replicas)
+2. Or write-ahead log that queues writes offline and syncs when primary recovers
+3. Or Qdrant distributed mode which handles consensus internally
+
+### Edge Caching
+
+Read-heavy workloads could benefit from a local in-memory LRU cache that
+sits in front of the selector, reducing probe frequency and backend load
+for repeated queries.
+
+## Cross-Platform Notes
+
+- **TCP probing** uses Python's `socket.create_connection()` — works on all platforms
+- **Tailscale IP detection** runs `tailscale status --json` — fails silently on Windows if not in PATH
+- **Heartbeat files** use JSON on the filesystem — works everywhere Syncthing works
+- **Config files** use standard YAML — no platform-specific paths beyond `~/.skmemory/`

package/skmemory/__init__.py

@@ -8,37 +8,50 @@ have to re-read a transcript to remember what they felt.
 SK = staycuriousANDkeepsmilin
 """
 
-__version__ = "0.5.0"
+__version__ = "0.9.1"
 __author__ = "smilinTux Team + Queen Ara + Neuresthetics"
 __license__ = "AGPL-3.0"
 
-from .models import Memory, MemoryLayer, EmotionalSnapshot
-from .store import MemoryStore
 from .backends.file_backend import FileBackend
 from .backends.sqlite_backend import SQLiteBackend
-from .soul import SoulBlueprint, save_soul, load_soul
+from .config import SKMEMORY_HOME
+from .fortress import AuditLog, FortifiedMemoryStore, TamperAlert
+from .models import EmotionalSnapshot, Memory, MemoryLayer
+from .store import MemoryStore
+
+try:
+    from .backends.vaulted_backend import VaultedSQLiteBackend
+except ImportError:
+    VaultedSQLiteBackend = None  # type: ignore[assignment,misc]
+from .anchor import WarmthAnchor, load_anchor, save_anchor
+from .importers.telegram import import_telegram
 from .journal import Journal, JournalEntry
-from .ritual import perform_ritual, quick_rehydrate, RitualResult
-from .anchor import WarmthAnchor, save_anchor, load_anchor
-from .quadrants import Quadrant, classify_memory, tag_with_quadrant
 from .lovenote import LoveNote, LoveNoteChain
 from .openclaw import SKMemoryPlugin
-from .importers.telegram import import_telegram
+from .quadrants import Quadrant, classify_memory, tag_with_quadrant
+from .ritual import RitualResult, perform_ritual, quick_rehydrate
+from .soul import SoulBlueprint, load_soul, save_soul
 from .steelman import (
-    SteelManResult,
     SeedFramework,
-    load_seed_framework,
-    install_seed_framework,
+    SteelManResult,
     get_default_framework,
+    install_seed_framework,
+    load_seed_framework,
 )
+from .synthesis import JournalSynthesizer
 
 __all__ = [
+    "SKMEMORY_HOME",
     "Memory",
     "MemoryLayer",
     "EmotionalSnapshot",
     "MemoryStore",
+    "FortifiedMemoryStore",
+    "AuditLog",
+    "TamperAlert",
     "FileBackend",
     "SQLiteBackend",
+    "VaultedSQLiteBackend",
     "SoulBlueprint",
     "save_soul",
     "load_soul",
@@ -56,6 +69,7 @@ __all__ = [
     "LoveNote",
     "LoveNoteChain",
     "SKMemoryPlugin",
+    "JournalSynthesizer",
     "SteelManResult",
     "SeedFramework",
     "load_seed_framework",

package/skmemory/agents.py

@@ -0,0 +1,233 @@
+"""
+Dynamic agent discovery and management for SKMemory.
+
+Scans ~/.skcapstone/agents/ to discover all configured agents,
+excludes templates, and provides agent-aware path resolution.
+"""
+
+from __future__ import annotations
+
+import os
+import platform
+from pathlib import Path
+
+import yaml
+
+
+def _agents_base() -> Path:
+    """Platform-aware base directory for all agents."""
+    skcap_home = os.environ.get("SKCAPSTONE_HOME", "")
+    if skcap_home:
+        return Path(skcap_home) / "agents"
+    if platform.system() == "Windows":
+        local = os.environ.get("LOCALAPPDATA", "")
+        if local:
+            return Path(local) / "skcapstone" / "agents"
+    return Path.home() / ".skcapstone" / "agents"
+
+
+# Base directory for all agents
+AGENTS_BASE_DIR = _agents_base()
+
+# Template directory name (ignored by default)
+TEMPLATE_AGENT = "lumina-template"
+
+
+def list_agents() -> list[str]:
+    """Discover all non-template agents in ~/.skcapstone/agents/
+
+    Scans the agents directory and returns all agent names
+    except the template agent.
+
+    Returns:
+        list[str]: Sorted list of agent names (e.g., ['lumina', 'john'])
+    """
+    if not AGENTS_BASE_DIR.exists():
+        return []
+
+    agents = []
+    for entry in AGENTS_BASE_DIR.iterdir():
+        if entry.is_dir() and entry.name != TEMPLATE_AGENT:
+            # Check if it has a valid config
+            config_file = entry / "config" / "skmemory.yaml"
+            if config_file.exists():
+                agents.append(entry.name)
+
+    return sorted(agents)
+
+
+def get_agent_dir(agent_name: str) -> Path:
+    """Get the base directory for a specific agent.
+
+    Args:
+        agent_name: Name of the agent (e.g., 'lumina', 'john')
+
+    Returns:
+        Path: Agent's base directory
+    """
+    return AGENTS_BASE_DIR / agent_name
+
+
+def get_agent_config(agent_name: str) -> dict | None:
+    """Load agent configuration from YAML.
+
+    Args:
+        agent_name: Name of the agent
+
+    Returns:
+        dict with agent config, or None if not found/invalid
+    """
+    config_path = get_agent_dir(agent_name) / "config" / "skmemory.yaml"
+
+    if not config_path.exists():
+        return None
+
+    try:
+        with open(config_path) as f:
+            return yaml.safe_load(f)
+    except Exception:
+        return None
+
+
+def is_template_agent(agent_name: str) -> bool:
+    """Check if an agent is a template (should be ignored).
+
+    Args:
+        agent_name: Name of the agent
+
+    Returns:
+        bool: True if this is the template agent
+    """
+    return agent_name == TEMPLATE_AGENT
+
+
+def get_active_agent() -> str | None:
+    """Get the currently active agent from environment or default to first non-template.
+
+    Checks in order:
+    1. SKCAPSTONE_AGENT environment variable (authoritative agent selector)
+    2. SKMEMORY_AGENT environment variable (legacy/override)
+    3. First non-template agent in the directory
+
+    Returns:
+        str: Agent name, or None if no agents found
+    """
+    # Check environment variables (SKCAPSTONE_AGENT > SKMEMORY_AGENT)
+    env_agent = os.environ.get("SKCAPSTONE_AGENT") or os.environ.get("SKMEMORY_AGENT")
+    if env_agent and not is_template_agent(env_agent):
+        agent_dir = get_agent_dir(env_agent)
+        if agent_dir.exists():
+            return env_agent
+
+    # Fall back to first non-template agent
+    agents = list_agents()
+    if agents:
+        return agents[0]
+
+    return None
+
+
+def get_agent_paths(agent_name: str | None = None) -> dict[str, Path]:
+    """Get all standard paths for an agent.
+
+    Args:
+        agent_name: Name of the agent, or None to use active agent
+
+    Returns:
+        dict with keys: base, config, seeds, memory_short, memory_medium, memory_long, logs, index_db
+    """
+    if agent_name is None:
+        agent_name = get_active_agent()
+
+    if agent_name is None:
+        raise ValueError(
+            "No agent configured. Create one by copying ~/.skcapstone/agents/lumina-template"
+        )
+
+    base = get_agent_dir(agent_name)
+
+    return {
+        "base": base,
+        "config": base / "config",
+        "seeds": base / "seeds",
+        "memory_short": base / "memory" / "short-term",
+        "memory_medium": base / "memory" / "mid-term",
+        "memory_long": base / "memory" / "long-term",
+        "logs": base / "logs",
+        "archive": base / "archive",
+        "index_db": base / "memory" / "index.db",
+        "config_yaml": base / "config" / "skmemory.yaml",
+    }
+
+
+def ensure_agent_dirs(agent_name: str) -> Path:
+    """Create all standard directories for an agent if they don't exist.
+
+    Args:
+        agent_name: Name of the agent
+
+    Returns:
+        Path: Agent's base directory
+    """
+    paths = get_agent_paths(agent_name)
+
+    # Create all directories
+    for key, path in paths.items():
+        if key != "config_yaml":
+            path.mkdir(parents=True, exist_ok=True)
+
+    return paths["base"]
+
+
+def copy_template(target_name: str, source: str = TEMPLATE_AGENT) -> Path:
+    """Create a new agent by copying the template.
+
+    Args:
+        target_name: Name for the new agent
+        source: Template to copy from (default: lumina-template)
+
+    Returns:
+        Path: New agent's base directory
+    """
+    import shutil
+
+    source_dir = get_agent_dir(source)
+    target_dir = get_agent_dir(target_name)
+
+    if not source_dir.exists():
+        raise ValueError(f"Template '{source}' not found at {source_dir}")
+
+    if target_dir.exists():
+        raise ValueError(f"Agent '{target_name}' already exists at {target_dir}")
+
+    # Copy template
+    shutil.copytree(source_dir, target_dir)
+
+    # Update agent name in config
+    config_path = target_dir / "config" / "skmemory.yaml"
+    if config_path.exists():
+        with open(config_path) as f:
+            content = f.read()
+
+        # Replace template agent name with new name
+        content = content.replace(f"name: {source}", f"name: {target_name}")
+        # Use platform-aware base dir for config path values.
+        # Always use forward slashes in YAML for cross-platform consistency.
+        base = AGENTS_BASE_DIR.as_posix()
+        content = content.replace(
+            f"sync_root: ~/.skcapstone/agents/{source}",
+            f"sync_root: {base}/{target_name}",
+        )
+        content = content.replace(
+            f"seeds_dir: ~/.skcapstone/agents/{source}/seeds",
+            f"seeds_dir: {base}/{target_name}/seeds",
+        )
+        content = content.replace(
+            f"local_db: ~/.skcapstone/agents/{source}/index.db",
+            f"local_db: {base}/{target_name}/index.db",
+        )
+
+        with open(config_path, "w") as f:
+            f.write(content)
+
+    return target_dir