caducus 0.1.0__tar.gz

caducus-0.1.0/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: caducus
+Version: 0.1.0
+Summary: CLI for collecting ops events and running reinforcement-memory analysis.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: virtuus>=0.4.0
+Provides-Extra: dev
+Requires-Dist: behave>=1.2.6; extra == "dev"
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+Requires-Dist: build>=1.2.2; extra == "dev"
+Requires-Dist: python-semantic-release>=10.5.3; extra == "dev"
+Provides-Extra: reinforcement-memory
+Requires-Dist: biblicus[reinforcement-memory]>=1.8.0; extra == "reinforcement-memory"
+
+# Caducus
+
+Caducus helps operations teams understand what is going wrong right now across logs, alerts, dead-letter queues, and other operational event streams.
+
+It is a CLI-first system for collecting timestamped operational events, normalizing them into a canonical schema, storing them as plain JSON, and using semantic reinforcement memory to surface recurring patterns, fresh anomalies, and just-in-time context during incidents.
+
+## Why Caducus Exists
+
+Operational signals are scattered across many systems:
+
+- CloudWatch logs
+- alerting systems
+- dead-letter queues
+- notifications and incident messages
+
+Each source captures part of the truth, but not the whole picture. Caducus is intended to bring those signals together into one stream of timestamped event records that can be analyzed as a living memory of operational behavior.
+
+The goal is not just to search historical data. The goal is to create a radar for what looks unusual, active, or important now.
+
+## How It Works
+
+Caducus is designed around a simple flow:
+
+1. Collect operational events from source systems.
+2. Normalize them into canonical event records with text, timestamps, source identity, and generalized metadata.
+3. Persist them as JSON files in a Virtuus-backed folder structure.
+4. Analyze event groups using Biblicus reinforcement memory.
+5. Surface patterns, anomalies, and context for operators.
+
+This keeps the system inspectable and composable. The underlying data lives in plain folders, not inside a black-box database.
+
+## CLI-First MVP
+
+The initial product is a CLI utility.
+
+The MVP is focused on a coherent end-to-end flow:
+
+- collect events from operational sources
+- store them in a canonical schema
+- run analysis over selected event groups
+- inspect recent events and analysis outputs from the command line
+
+Initial source areas for the MVP are:
+
+- CloudWatch Logs
+- SQS dead-letter queues
+- one alert source
+
+Configuration is intended to be layered through YAML, environment variables, and CLI overrides. Caducus will own collection and orchestration while allowing Biblicus-related analysis settings to flow through the Caducus configuration tree without duplicating Biblicus's schema.
+
+## Architecture At A Glance
+
+Caducus is intentionally thin:
+
+- **Caducus** handles collection, normalization, orchestration, and CLI workflows.
+- **Virtuus** provides file-backed JSON storage and retrieval.
+- **Biblicus** provides semantic reinforcement-memory analysis.
+
+```mermaid
+flowchart LR
+    sources[OpsSources] --> caducus[Caducus]
+    caducus --> events[CanonicalEvents]
+    events --> virtuus[VirtuusStorage]
+    caducus --> biblicus[BiblicusAnalysis]
+    biblicus --> radar[OpsRadar]
+```
+
+## Running the demo
+
+Real HDFS data uses **component-derived group IDs**: each log row’s `component` becomes `hdfs-demo:<component>` (e.g. `hdfs-demo:dfs.DataNode$DataXceiver`). You must use a group ID that exists in your ingested data.
+
+### Quick demo (small fixture, no download)
+
+```bash
+pip install -e ".[reinforcement-memory]"
+caducus demo run --input tests/fixtures/demo_hdfs_sample.csv --group-id "hdfs-demo:DataNode" --data-dir /tmp/caducus-demo
+```
+
+The fixture has components `DataNode` and `NameNode`, so valid group IDs are `hdfs-demo:DataNode` and `hdfs-demo:NameNode`.
+
+### Full demo on real HDFS data
+
+1. Install optional deps (Biblicus reinforcement-memory and the datasets library for the download script):
+
+   ```bash
+   pip install -e ".[reinforcement-memory]"
+   pip install datasets
+   ```
+
+2. Download a subset of the [HDFS_v1](https://huggingface.co/datasets/logfit-project/HDFS_v1) dataset:
+
+   ```bash
+   python scripts/download_hdfs_demo.py --output demo_data/hdfs_sample.csv --max-rows 10000
+   ```
+
+3. Ingest and list available groups (group IDs come from the CSV `component` column):
+
+   ```bash
+   caducus demo ingest --input demo_data/hdfs_sample.csv --data-dir ./caducus-data
+   caducus groups --data-dir ./caducus-data
+   ```
+
+4. Run analysis for one of the listed group IDs:
+
+   ```bash
+   caducus analyze --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+   Or do ingest and analyze in one step (use a group ID that exists in the CSV):
+
+   ```bash
+   caducus demo run --input demo_data/hdfs_sample.csv --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+## Releases
+
+Caducus uses `python-semantic-release` with Conventional Commits.
+
+Use commit messages like:
+
+- `feat: add CloudWatch collector checkpointing`
+- `fix: quote group IDs containing dollar signs in docs`
+- `feat!: change canonical event schema`
+
+Release behavior:
+
+- `feat:` triggers a minor release
+- `fix:` triggers a patch release
+- `feat!:` or a `BREAKING CHANGE:` footer triggers a major release
+
+The release workflow lives in `.github/workflows/release.yml` and runs on pushes to `main`. It will:
+
+1. Determine the next version from commit messages.
+2. Update `project.version` in `pyproject.toml` and `src/caducus/__init__.py`.
+3. Generate `CHANGELOG.md`, create a tag, and create a GitHub Release.
+4. Publish the built distributions to PyPI.
+
+PyPI publishing is configured for GitHub Actions trusted publishing. Before the first live release, configure the `caducus` project on PyPI to trust this repository's `release.yml` workflow.
+
+## Roadmap
+
+Caducus is intended to grow beyond the initial CLI foundation over time.
+
+Planned directions include:
+
+- broader source integrations across operational systems
+- deeper analysis of concepts and entities derived from operational activity
+- richer incident context and root-cause workflows
+- a future web UI and embeddable components for other applications
+
+## Repository Direction
+
+This repository is being built outside-in. Product definition and behavior specifications come first, followed by the minimum implementation needed to satisfy them.

caducus-0.1.0/README.md

@@ -0,0 +1,153 @@
+# Caducus
+
+Caducus helps operations teams understand what is going wrong right now across logs, alerts, dead-letter queues, and other operational event streams.
+
+It is a CLI-first system for collecting timestamped operational events, normalizing them into a canonical schema, storing them as plain JSON, and using semantic reinforcement memory to surface recurring patterns, fresh anomalies, and just-in-time context during incidents.
+
+## Why Caducus Exists
+
+Operational signals are scattered across many systems:
+
+- CloudWatch logs
+- alerting systems
+- dead-letter queues
+- notifications and incident messages
+
+Each source captures part of the truth, but not the whole picture. Caducus is intended to bring those signals together into one stream of timestamped event records that can be analyzed as a living memory of operational behavior.
+
+The goal is not just to search historical data. The goal is to create a radar for what looks unusual, active, or important now.
+
+## How It Works
+
+Caducus is designed around a simple flow:
+
+1. Collect operational events from source systems.
+2. Normalize them into canonical event records with text, timestamps, source identity, and generalized metadata.
+3. Persist them as JSON files in a Virtuus-backed folder structure.
+4. Analyze event groups using Biblicus reinforcement memory.
+5. Surface patterns, anomalies, and context for operators.
+
+This keeps the system inspectable and composable. The underlying data lives in plain folders, not inside a black-box database.
+
+## CLI-First MVP
+
+The initial product is a CLI utility.
+
+The MVP is focused on a coherent end-to-end flow:
+
+- collect events from operational sources
+- store them in a canonical schema
+- run analysis over selected event groups
+- inspect recent events and analysis outputs from the command line
+
+Initial source areas for the MVP are:
+
+- CloudWatch Logs
+- SQS dead-letter queues
+- one alert source
+
+Configuration is intended to be layered through YAML, environment variables, and CLI overrides. Caducus will own collection and orchestration while allowing Biblicus-related analysis settings to flow through the Caducus configuration tree without duplicating Biblicus's schema.
+
+## Architecture At A Glance
+
+Caducus is intentionally thin:
+
+- **Caducus** handles collection, normalization, orchestration, and CLI workflows.
+- **Virtuus** provides file-backed JSON storage and retrieval.
+- **Biblicus** provides semantic reinforcement-memory analysis.
+
+```mermaid
+flowchart LR
+    sources[OpsSources] --> caducus[Caducus]
+    caducus --> events[CanonicalEvents]
+    events --> virtuus[VirtuusStorage]
+    caducus --> biblicus[BiblicusAnalysis]
+    biblicus --> radar[OpsRadar]
+```
+
+## Running the demo
+
+Real HDFS data uses **component-derived group IDs**: each log row’s `component` becomes `hdfs-demo:<component>` (e.g. `hdfs-demo:dfs.DataNode$DataXceiver`). You must use a group ID that exists in your ingested data.
+
+### Quick demo (small fixture, no download)
+
+```bash
+pip install -e ".[reinforcement-memory]"
+caducus demo run --input tests/fixtures/demo_hdfs_sample.csv --group-id "hdfs-demo:DataNode" --data-dir /tmp/caducus-demo
+```
+
+The fixture has components `DataNode` and `NameNode`, so valid group IDs are `hdfs-demo:DataNode` and `hdfs-demo:NameNode`.
+
+### Full demo on real HDFS data
+
+1. Install optional deps (Biblicus reinforcement-memory and the datasets library for the download script):
+
+   ```bash
+   pip install -e ".[reinforcement-memory]"
+   pip install datasets
+   ```
+
+2. Download a subset of the [HDFS_v1](https://huggingface.co/datasets/logfit-project/HDFS_v1) dataset:
+
+   ```bash
+   python scripts/download_hdfs_demo.py --output demo_data/hdfs_sample.csv --max-rows 10000
+   ```
+
+3. Ingest and list available groups (group IDs come from the CSV `component` column):
+
+   ```bash
+   caducus demo ingest --input demo_data/hdfs_sample.csv --data-dir ./caducus-data
+   caducus groups --data-dir ./caducus-data
+   ```
+
+4. Run analysis for one of the listed group IDs:
+
+   ```bash
+   caducus analyze --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+   Or do ingest and analyze in one step (use a group ID that exists in the CSV):
+
+   ```bash
+   caducus demo run --input demo_data/hdfs_sample.csv --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+## Releases
+
+Caducus uses `python-semantic-release` with Conventional Commits.
+
+Use commit messages like:
+
+- `feat: add CloudWatch collector checkpointing`
+- `fix: quote group IDs containing dollar signs in docs`
+- `feat!: change canonical event schema`
+
+Release behavior:
+
+- `feat:` triggers a minor release
+- `fix:` triggers a patch release
+- `feat!:` or a `BREAKING CHANGE:` footer triggers a major release
+
+The release workflow lives in `.github/workflows/release.yml` and runs on pushes to `main`. It will:
+
+1. Determine the next version from commit messages.
+2. Update `project.version` in `pyproject.toml` and `src/caducus/__init__.py`.
+3. Generate `CHANGELOG.md`, create a tag, and create a GitHub Release.
+4. Publish the built distributions to PyPI.
+
+PyPI publishing is configured for GitHub Actions trusted publishing. Before the first live release, configure the `caducus` project on PyPI to trust this repository's `release.yml` workflow.
+
+## Roadmap
+
+Caducus is intended to grow beyond the initial CLI foundation over time.
+
+Planned directions include:
+
+- broader source integrations across operational systems
+- deeper analysis of concepts and entities derived from operational activity
+- richer incident context and root-cause workflows
+- a future web UI and embeddable components for other applications
+
+## Repository Direction
+
+This repository is being built outside-in. Product definition and behavior specifications come first, followed by the minimum implementation needed to satisfy them.

caducus-0.1.0/pyproject.toml

@@ -0,0 +1,56 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "caducus"
+version = "0.1.0"
+description = "CLI for collecting ops events and running reinforcement-memory analysis."
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = [
+  "PyYAML>=6.0",
+  "virtuus>=0.4.0",
+]
+
+[project.optional-dependencies]
+dev = [
+  "behave>=1.2.6",
+  "pytest>=8.0",
+  "ruff>=0.4.0",
+  "black>=24.0",
+  "build>=1.2.2",
+  "python-semantic-release>=10.5.3",
+]
+reinforcement-memory = [
+  "biblicus[reinforcement-memory]>=1.8.0",
+]
+
+[project.scripts]
+caducus = "caducus.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 88
+
+[tool.black]
+line-length = 88
+
+[tool.semantic_release]
+allow_zero_version = true
+build_command = "python -m build"
+commit_parser = "conventional"
+version_toml = [
+  "pyproject.toml:project.version",
+]
+version_variables = [
+  "src/caducus/__init__.py:__version__",
+]
+
+[tool.semantic_release.branches.main]
+match = "main"
+
+[tool.semantic_release.changelog.default_templates]
+changelog_file = "CHANGELOG.md"

caducus-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

caducus-0.1.0/src/caducus/__init__.py

@@ -0,0 +1,3 @@
+"""Caducus — collect ops events and run reinforcement-memory analysis."""
+
+__version__ = "0.1.0"

caducus-0.1.0/src/caducus/biblicus_adapter.py

@@ -0,0 +1,104 @@
+"""Thin adapter over Biblicus reinforcement memory for Caducus analysis."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    from virtuus._python.table import Table
+
+try:
+    from biblicus.analysis.reinforcement_memory import (
+        LocalVectorStore,
+        ReinforcementMemory,
+        TimestampedText,
+        hash_embedder,
+    )
+except ImportError as e:
+    raise ImportError(
+        "Caducus analysis requires biblicus with reinforcement-memory. "
+        "Install with: pip install 'biblicus[reinforcement-memory]' or pip install -e ../Biblicus"
+    ) from e
+
+from caducus.storage import get_events_for_group
+
+
+def _events_to_timestamped_text(rows: list[dict[str, Any]], group_id: str) -> list[TimestampedText]:
+    """Map Caducus canonical event rows to Biblicus TimestampedText."""
+    out = []
+    for r in rows:
+        meta = dict(r.get("metadata") or {})
+        meta["source"] = r.get("source", "")
+        meta["group_id"] = r.get("group_id", group_id)
+        out.append(
+            TimestampedText(
+                id=r["id"],
+                group_id=group_id,
+                timestamp=r.get("timestamp", ""),
+                text=r.get("text", ""),
+                metadata=meta,
+            )
+        )
+    return out
+
+
+def _analysis_dirs_from_config(
+    data_dir: str, config: dict[str, Any] | None
+) -> tuple[str, str]:
+    """
+    Resolve analysis_dir and vector_dir from Caducus data_dir and optional biblicus config.
+    Returns (analysis_dir, vector_dir).
+    """
+    analysis_dir = os.path.join(data_dir, "analysis")
+    vector_dir = os.path.join(analysis_dir, "vectors")
+    if config:
+        rm = config.get("biblicus", {}).get("reinforcement_memory") or {}
+        if rm.get("data_dir"):
+            analysis_dir = rm["data_dir"]
+            vector_dir = os.path.join(analysis_dir, "vectors")
+        vs = rm.get("vector_store")
+        if isinstance(vs, dict) and vs.get("kind") == "local" and vs.get("path"):
+            vector_dir = vs["path"]
+    return analysis_dir, vector_dir
+
+
+def run_analysis_for_group(
+    data_dir: str,
+    group_id: str,
+    table: "Table",
+    config: dict[str, Any] | None = None,
+) -> None:
+    """
+    Load canonical events for the group, run Biblicus reinforcement-memory analysis,
+    and print structured topic output to stdout.
+
+    Config may contain a biblicus.reinforcement_memory subtree with data_dir and
+    vector_store (e.g. { kind: local, path: ... }). If absent, defaults are used.
+    """
+    analysis_dir, vector_dir = _analysis_dirs_from_config(data_dir, config)
+    Path(analysis_dir).mkdir(parents=True, exist_ok=True)
+    Path(vector_dir).mkdir(parents=True, exist_ok=True)
+
+    memory = ReinforcementMemory(
+        data_dir=analysis_dir,
+        vector_store=LocalVectorStore(vector_dir),
+        embed=hash_embedder(),
+    )
+
+    rows = get_events_for_group(table, group_id)
+    if not rows:
+        print(f"No events found for group_id={group_id}")
+        return
+
+    texts = _events_to_timestamped_text(rows, group_id)
+    memory.ingest(texts)
+    result = memory.analyze(group_id=group_id)
+
+    print(f"Group: {result.group_id}  Texts: {result.texts_analyzed}  Run: {result.run_id}")
+    for t in result.topics:
+        line = f"  {t.label}  [{t.memory_tier}/{t.lifecycle_tier}]  n={t.member_count}"
+        if t.root_cause:
+            line += f"  cause: {t.root_cause[:60]}..."
+        print(line)

caducus-0.1.0/src/caducus/cli.py

@@ -0,0 +1,109 @@
+"""Caducus CLI entrypoint."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+
+def main() -> int:
+    """Entry point for the caducus command."""
+    parser = argparse.ArgumentParser(prog="caducus", description="Collect ops events and run analysis.")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    demo = sub.add_parser("demo", help="Demo dataset commands")
+    demo_sub = demo.add_subparsers(dest="demo_command", required=True)
+
+    demo_ingest = demo_sub.add_parser("ingest", help="Ingest a local demo dataset file into canonical events")
+    demo_ingest.add_argument("--input", required=True, help="Path to demo dataset file (e.g. CSV or JSONL)")
+    demo_ingest.add_argument("--data-dir", default="./caducus-data", help="Caducus data directory")
+    demo_ingest.set_defaults(func=cmd_demo_ingest)
+
+    demo_run = demo_sub.add_parser("run", help="Ingest demo file and run analysis for one group")
+    demo_run.add_argument("--input", required=True, help="Path to demo dataset file")
+    demo_run.add_argument("--group-id", required=True, help="Group ID to analyze")
+    demo_run.add_argument("--data-dir", default=None, help="Caducus data directory")
+    demo_run.add_argument("--config", action="append", default=[], metavar="KEY=VALUE", help="Config override (repeatable)")
+    demo_run.add_argument("--configuration", action="append", default=[], metavar="FILE", dest="config_files", help="Config YAML file (repeatable)")
+    demo_run.set_defaults(func=cmd_demo_run)
+
+    analyze_p = sub.add_parser("analyze", help="Run reinforcement-memory analysis for a group")
+    analyze_p.add_argument("--group-id", required=True, help="Group ID to analyze")
+    analyze_p.add_argument("--data-dir", default=None, help="Caducus data directory")
+    analyze_p.add_argument("--config", action="append", default=[], metavar="KEY=VALUE", help="Config override (repeatable)")
+    analyze_p.add_argument("--configuration", action="append", default=[], metavar="FILE", dest="config_files", help="Config YAML file (repeatable)")
+    analyze_p.set_defaults(func=cmd_analyze)
+
+    groups_p = sub.add_parser("groups", help="List group IDs that have events (use after ingest)")
+    groups_p.add_argument("--data-dir", default="./caducus-data", help="Caducus data directory")
+    groups_p.set_defaults(func=cmd_groups)
+
+    args = parser.parse_args()
+    if hasattr(args, "func"):
+        return args.func(args)
+    return 0
+
+
+def cmd_demo_ingest(args: argparse.Namespace) -> int:
+    """Ingest demo dataset file into canonical events."""
+    from caducus.collectors.demo_dataset import ingest_demo_file
+    from caducus.storage import get_events_table
+
+    table = get_events_table(args.data_dir)
+    count = ingest_demo_file(args.input, table)
+    print(f"Ingested {count} events into {args.data_dir}")
+    return 0
+
+
+def _load_config(args: argparse.Namespace) -> tuple[dict, str]:
+    """Load merged config and resolve data_dir. Returns (config, data_dir)."""
+    from caducus.config import get_data_dir, load_config
+
+    config = load_config(
+        config_file_paths=getattr(args, "config_files", None) or [],
+        overrides=getattr(args, "config", None) or [],
+    )
+    data_dir = get_data_dir(config, getattr(args, "data_dir", None))
+    return config, data_dir
+
+
+def cmd_demo_run(args: argparse.Namespace) -> int:
+    """Ingest demo file and run analysis for one group."""
+    from caducus.biblicus_adapter import run_analysis_for_group
+    from caducus.collectors.demo_dataset import ingest_demo_file
+    from caducus.storage import get_events_table
+
+    config, data_dir = _load_config(args)
+    table = get_events_table(data_dir)
+    ingest_demo_file(args.input, table)
+    run_analysis_for_group(data_dir, args.group_id, table, config=config)
+    return 0
+
+
+def cmd_analyze(args: argparse.Namespace) -> int:
+    """Run analysis for a group from stored canonical events."""
+    from caducus.biblicus_adapter import run_analysis_for_group
+    from caducus.storage import get_events_table
+
+    config, data_dir = _load_config(args)
+    table = get_events_table(data_dir)
+    run_analysis_for_group(data_dir, args.group_id, table, config=config)
+    return 0
+
+
+def cmd_groups(args: argparse.Namespace) -> int:
+    """List group IDs that have events in storage."""
+    from caducus.storage import get_events_table, list_group_ids
+
+    table = get_events_table(args.data_dir)
+    group_ids = list_group_ids(table)
+    if not group_ids:
+        print("No groups found. Run 'caducus demo ingest' first.")
+        return 0
+    for gid in group_ids:
+        print(gid)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

caducus-0.1.0/src/caducus/collectors/__init__.py

@@ -0,0 +1 @@
+"""Caducus event collectors."""

caducus-0.1.0/src/caducus/collectors/demo_dataset.py

@@ -0,0 +1,83 @@
+"""Demo dataset collector: HDFS-style log rows -> canonical events."""
+
+from __future__ import annotations
+
+import csv
+import uuid
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Iterator
+
+from caducus.events import CanonicalEvent
+
+if TYPE_CHECKING:
+    from virtuus._python.table import Table
+
+SOURCE_ID = "hdfs-demo"
+
+
+def _parse_timestamp(date_str: str, time_str: str) -> str:
+    """Turn YYMMDD and HHMMSS into ISO 8601. Assumes 20xx for year."""
+    if not date_str or not time_str:
+        return ""
+    try:
+        y, m, d = int(date_str[:2]), int(date_str[2:4]), int(date_str[4:6])
+        year = 2000 + y if y < 50 else 1900 + y
+        h = int(time_str[:2])
+        mi = int(time_str[2:4])
+        s = int(time_str[4:6]) if len(time_str) >= 6 else 0
+        return f"{year:04d}-{m:02d}-{d:02d}T{h:02d}:{mi:02d}:{s:02d}Z"
+    except (ValueError, IndexError):
+        return ""
+
+
+def _row_to_event(row: dict[str, Any], index: int) -> CanonicalEvent:
+    """Map one HDFS-style row to a canonical event."""
+    date_str = str(row.get("date", ""))
+    time_str = str(row.get("time", ""))
+    content = str(row.get("content", ""))
+    component = str(row.get("component", "unknown")).strip() or "unknown"
+    event_id = row.get("id") or str(uuid.uuid4())
+    timestamp = _parse_timestamp(date_str, time_str)
+    group_id = f"{SOURCE_ID}:{component}"
+    metadata: dict[str, Any] = {
+        "level": row.get("level"),
+        "pid": row.get("pid"),
+        "block_id": row.get("block_id"),
+        "anomaly": row.get("anomaly"),
+    }
+    return CanonicalEvent(
+        id=event_id,
+        timestamp=timestamp,
+        source=SOURCE_ID,
+        group_id=group_id,
+        text=content,
+        metadata={k: v for k, v in metadata.items() if v is not None},
+    )
+
+
+def _read_csv(path: Path) -> Iterator[dict[str, Any]]:
+    """Read CSV with headers. Normalize keys to lowercase."""
+    with open(path, newline="", encoding="utf-8", errors="replace") as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            yield {k.strip().lower(): v for k, v in row.items() if k}
+
+
+def ingest_demo_file(input_path: str, table: "Table") -> int:
+    """
+    Read a demo dataset file (CSV with HDFS-style columns) and write canonical events.
+
+    Expected columns: date, time, level, component, pid, content, block_id, anomaly.
+    """
+    path = Path(input_path)
+    if not path.exists():
+        raise FileNotFoundError(f"Demo file not found: {input_path}")
+
+    events: list[CanonicalEvent] = []
+    for i, row in enumerate(_read_csv(path)):
+        ev = _row_to_event(row, i)
+        events.append(ev)
+
+    from caducus.storage import put_events
+
+    return put_events(table, events)

caducus-0.1.0/src/caducus/config.py

@@ -0,0 +1,105 @@
+"""Caducus configuration: YAML merge, env, and CLI overrides."""
+
+from __future__ import annotations
+
+import os
+import re
+from pathlib import Path
+from typing import Any
+
+
+def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
+    """Merge override into base recursively. Override wins for scalars."""
+    out = dict(base)
+    for k, v in override.items():
+        if k in out and isinstance(out[k], dict) and isinstance(v, dict):
+            out[k] = _deep_merge(out[k], v)
+        else:
+            out[k] = v
+    return out
+
+
+def _set_dotted(root: dict[str, Any], key: str, value: Any) -> None:
+    """Set a dotted key (e.g. biblicus.reinforcement_memory.data_dir) into root."""
+    parts = key.split(".")
+    cur = root
+    for i, part in enumerate(parts[:-1]):
+        if part not in cur:
+            cur[part] = {}
+        cur = cur[part]
+    cur[parts[-1]] = value
+
+
+def _env_substitute_string(s: str) -> str:
+    """Replace {{ VAR }} and {{ VAR|default }} in s with env or default."""
+    if not isinstance(s, str):
+        return s
+
+    def repl(match: re.Match[str]) -> str:
+        var = match.group(1).strip()
+        pipe = match.group(2)
+        default = match.group(3).strip() if pipe else ""
+        return os.environ.get(var, default)
+
+    return re.sub(r"\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*(\|\s*([^}]*))?\s*\}\}", repl, s)
+
+
+def _env_substitute(obj: Any) -> Any:
+    """Recursively substitute {{ VAR }} in string values."""
+    if isinstance(obj, dict):
+        return {k: _env_substitute(v) for k, v in obj.items()}
+    if isinstance(obj, list):
+        return [_env_substitute(v) for v in obj]
+    if isinstance(obj, str):
+        return _env_substitute_string(obj)
+    return obj
+
+
+def load_yaml(path: Path) -> dict[str, Any]:
+    """Load a single YAML file. Returns {} if file missing or invalid."""
+    import yaml
+
+    if not path.exists():
+        return {}
+    try:
+        with open(path, encoding="utf-8") as f:
+            data = yaml.safe_load(f)
+        return dict(data) if data else {}
+    except Exception:
+        return {}
+
+
+def load_config(
+    config_file_paths: list[str] | None = None,
+    overrides: list[str] | None = None,
+) -> dict[str, Any]:
+    """
+    Build merged config from YAML files, env, and dotted overrides.
+
+    - config_file_paths: paths to YAML files (later override earlier).
+    - overrides: list of "key=value" (dotted keys, e.g. biblicus.reinforcement_memory.data_dir=/path).
+
+    CADUCUS_DATA_DIR and BIBLICUS_* env vars are not auto-injected here;
+    CLI can apply them as overrides before calling this, or callers can merge.
+    """
+    merged: dict[str, Any] = {}
+    if config_file_paths:
+        for p in config_file_paths:
+            data = load_yaml(Path(p))
+            merged = _deep_merge(merged, data)
+    merged = _env_substitute(merged)
+    if overrides:
+        for o in overrides:
+            if "=" not in o:
+                continue
+            key, _, value = o.partition("=")
+            key = key.strip().strip(".")
+            _set_dotted(merged, key, value)
+    return merged
+
+
+def get_data_dir(config: dict[str, Any], cli_data_dir: str | None) -> str:
+    """Resolve data_dir: CLI wins, then config, then default."""
+    if cli_data_dir:
+        return cli_data_dir
+    return config.get("data_dir") or os.environ.get("CADUCUS_DATA_DIR") or "./caducus-data"

caducus-0.1.0/src/caducus/events.py

@@ -0,0 +1,45 @@
+"""Canonical event model for Caducus."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+
+@dataclass
+class CanonicalEvent:
+    """
+    A single canonical ops event.
+
+    Schema is defined in docs/caducus-biblicus-contract.md.
+    """
+
+    id: str
+    timestamp: str  # ISO 8601
+    source: str
+    group_id: str
+    text: str
+    metadata: dict[str, Any]
+
+    def to_dict(self) -> dict[str, Any]:
+        """Serialize for storage."""
+        return {
+            "id": self.id,
+            "timestamp": self.timestamp,
+            "source": self.source,
+            "group_id": self.group_id,
+            "text": self.text,
+            "metadata": self.metadata,
+        }
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "CanonicalEvent":
+        """Deserialize from storage."""
+        return cls(
+            id=data["id"],
+            timestamp=data["timestamp"],
+            source=data["source"],
+            group_id=data["group_id"],
+            text=data["text"],
+            metadata=data.get("metadata", {}),
+        )

caducus-0.1.0/src/caducus/storage.py

@@ -0,0 +1,57 @@
+"""Virtuus-backed storage for Caducus canonical events."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Iterator
+
+from virtuus import Database
+
+from caducus.events import CanonicalEvent
+
+if TYPE_CHECKING:
+    from virtuus._python.table import Table
+
+EVENTS_SCHEMA = {
+    "tables": {
+        "events": {
+            "primary_key": "id",
+            "directory": "events",
+            "storage": "index_only",
+            "gsis": {
+                "by_group": {
+                    "partition_key": "group_id",
+                    "sort_key": "timestamp",
+                },
+            },
+        },
+    },
+}
+
+
+def get_events_table(data_dir: str) -> "Table":
+    """Return the Virtuus events table for the given data root."""
+    db = Database.from_schema_dict(EVENTS_SCHEMA, data_root=data_dir)
+    return db.tables["events"]
+
+
+def put_events(table: "Table", events: list[CanonicalEvent]) -> int:
+    """Write canonical events to the table. Returns count written."""
+    for ev in events:
+        table.put(ev.to_dict())
+    return len(events)
+
+
+def get_events_for_group(table: "Table", group_id: str) -> list[dict[str, Any]]:
+    """Return all events for a group_id, ordered by timestamp ascending."""
+    return table.query_gsi("by_group", group_id, None, descending=False)
+
+
+def list_group_ids(table: "Table") -> list[str]:
+    """Return sorted distinct group_id values present in the events table."""
+    seen: set[str] = set()
+    for row in table.scan():
+        gid = row.get("group_id")
+        if gid:
+            seen.add(gid)
+    return sorted(seen)

caducus-0.1.0/src/caducus.egg-info/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: caducus
+Version: 0.1.0
+Summary: CLI for collecting ops events and running reinforcement-memory analysis.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: virtuus>=0.4.0
+Provides-Extra: dev
+Requires-Dist: behave>=1.2.6; extra == "dev"
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+Requires-Dist: build>=1.2.2; extra == "dev"
+Requires-Dist: python-semantic-release>=10.5.3; extra == "dev"
+Provides-Extra: reinforcement-memory
+Requires-Dist: biblicus[reinforcement-memory]>=1.8.0; extra == "reinforcement-memory"
+
+# Caducus
+
+Caducus helps operations teams understand what is going wrong right now across logs, alerts, dead-letter queues, and other operational event streams.
+
+It is a CLI-first system for collecting timestamped operational events, normalizing them into a canonical schema, storing them as plain JSON, and using semantic reinforcement memory to surface recurring patterns, fresh anomalies, and just-in-time context during incidents.
+
+## Why Caducus Exists
+
+Operational signals are scattered across many systems:
+
+- CloudWatch logs
+- alerting systems
+- dead-letter queues
+- notifications and incident messages
+
+Each source captures part of the truth, but not the whole picture. Caducus is intended to bring those signals together into one stream of timestamped event records that can be analyzed as a living memory of operational behavior.
+
+The goal is not just to search historical data. The goal is to create a radar for what looks unusual, active, or important now.
+
+## How It Works
+
+Caducus is designed around a simple flow:
+
+1. Collect operational events from source systems.
+2. Normalize them into canonical event records with text, timestamps, source identity, and generalized metadata.
+3. Persist them as JSON files in a Virtuus-backed folder structure.
+4. Analyze event groups using Biblicus reinforcement memory.
+5. Surface patterns, anomalies, and context for operators.
+
+This keeps the system inspectable and composable. The underlying data lives in plain folders, not inside a black-box database.
+
+## CLI-First MVP
+
+The initial product is a CLI utility.
+
+The MVP is focused on a coherent end-to-end flow:
+
+- collect events from operational sources
+- store them in a canonical schema
+- run analysis over selected event groups
+- inspect recent events and analysis outputs from the command line
+
+Initial source areas for the MVP are:
+
+- CloudWatch Logs
+- SQS dead-letter queues
+- one alert source
+
+Configuration is intended to be layered through YAML, environment variables, and CLI overrides. Caducus will own collection and orchestration while allowing Biblicus-related analysis settings to flow through the Caducus configuration tree without duplicating Biblicus's schema.
+
+## Architecture At A Glance
+
+Caducus is intentionally thin:
+
+- **Caducus** handles collection, normalization, orchestration, and CLI workflows.
+- **Virtuus** provides file-backed JSON storage and retrieval.
+- **Biblicus** provides semantic reinforcement-memory analysis.
+
+```mermaid
+flowchart LR
+    sources[OpsSources] --> caducus[Caducus]
+    caducus --> events[CanonicalEvents]
+    events --> virtuus[VirtuusStorage]
+    caducus --> biblicus[BiblicusAnalysis]
+    biblicus --> radar[OpsRadar]
+```
+
+## Running the demo
+
+Real HDFS data uses **component-derived group IDs**: each log row’s `component` becomes `hdfs-demo:<component>` (e.g. `hdfs-demo:dfs.DataNode$DataXceiver`). You must use a group ID that exists in your ingested data.
+
+### Quick demo (small fixture, no download)
+
+```bash
+pip install -e ".[reinforcement-memory]"
+caducus demo run --input tests/fixtures/demo_hdfs_sample.csv --group-id "hdfs-demo:DataNode" --data-dir /tmp/caducus-demo
+```
+
+The fixture has components `DataNode` and `NameNode`, so valid group IDs are `hdfs-demo:DataNode` and `hdfs-demo:NameNode`.
+
+### Full demo on real HDFS data
+
+1. Install optional deps (Biblicus reinforcement-memory and the datasets library for the download script):
+
+   ```bash
+   pip install -e ".[reinforcement-memory]"
+   pip install datasets
+   ```
+
+2. Download a subset of the [HDFS_v1](https://huggingface.co/datasets/logfit-project/HDFS_v1) dataset:
+
+   ```bash
+   python scripts/download_hdfs_demo.py --output demo_data/hdfs_sample.csv --max-rows 10000
+   ```
+
+3. Ingest and list available groups (group IDs come from the CSV `component` column):
+
+   ```bash
+   caducus demo ingest --input demo_data/hdfs_sample.csv --data-dir ./caducus-data
+   caducus groups --data-dir ./caducus-data
+   ```
+
+4. Run analysis for one of the listed group IDs:
+
+   ```bash
+   caducus analyze --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+   Or do ingest and analyze in one step (use a group ID that exists in the CSV):
+
+   ```bash
+   caducus demo run --input demo_data/hdfs_sample.csv --group-id "hdfs-demo:dfs.DataNode$DataXceiver" --data-dir ./caducus-data
+   ```
+
+## Releases
+
+Caducus uses `python-semantic-release` with Conventional Commits.
+
+Use commit messages like:
+
+- `feat: add CloudWatch collector checkpointing`
+- `fix: quote group IDs containing dollar signs in docs`
+- `feat!: change canonical event schema`
+
+Release behavior:
+
+- `feat:` triggers a minor release
+- `fix:` triggers a patch release
+- `feat!:` or a `BREAKING CHANGE:` footer triggers a major release
+
+The release workflow lives in `.github/workflows/release.yml` and runs on pushes to `main`. It will:
+
+1. Determine the next version from commit messages.
+2. Update `project.version` in `pyproject.toml` and `src/caducus/__init__.py`.
+3. Generate `CHANGELOG.md`, create a tag, and create a GitHub Release.
+4. Publish the built distributions to PyPI.
+
+PyPI publishing is configured for GitHub Actions trusted publishing. Before the first live release, configure the `caducus` project on PyPI to trust this repository's `release.yml` workflow.
+
+## Roadmap
+
+Caducus is intended to grow beyond the initial CLI foundation over time.
+
+Planned directions include:
+
+- broader source integrations across operational systems
+- deeper analysis of concepts and entities derived from operational activity
+- richer incident context and root-cause workflows
+- a future web UI and embeddable components for other applications
+
+## Repository Direction
+
+This repository is being built outside-in. Product definition and behavior specifications come first, followed by the minimum implementation needed to satisfy them.

caducus-0.1.0/src/caducus.egg-info/SOURCES.txt

@@ -0,0 +1,17 @@
+README.md
+pyproject.toml
+src/caducus/__init__.py
+src/caducus/biblicus_adapter.py
+src/caducus/cli.py
+src/caducus/config.py
+src/caducus/events.py
+src/caducus/storage.py
+src/caducus.egg-info/PKG-INFO
+src/caducus.egg-info/SOURCES.txt
+src/caducus.egg-info/dependency_links.txt
+src/caducus.egg-info/entry_points.txt
+src/caducus.egg-info/requires.txt
+src/caducus.egg-info/top_level.txt
+src/caducus/collectors/__init__.py
+src/caducus/collectors/demo_dataset.py
+tests/test_demo_slice.py

caducus-0.1.0/src/caducus.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

caducus-0.1.0/src/caducus.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+caducus = caducus.cli:main

caducus-0.1.0/src/caducus.egg-info/requires.txt

@@ -0,0 +1,13 @@
+PyYAML>=6.0
+virtuus>=0.4.0
+
+[dev]
+behave>=1.2.6
+pytest>=8.0
+ruff>=0.4.0
+black>=24.0
+build>=1.2.2
+python-semantic-release>=10.5.3
+
+[reinforcement-memory]
+biblicus[reinforcement-memory]>=1.8.0

caducus-0.1.0/src/caducus.egg-info/top_level.txt

@@ -0,0 +1 @@
+caducus

caducus-0.1.0/tests/test_demo_slice.py

@@ -0,0 +1,121 @@
+"""Tests for the demo vertical slice: ingest and analysis."""
+
+from __future__ import annotations
+
+import os
+import tempfile
+from pathlib import Path
+
+import pytest
+
+from caducus.collectors.demo_dataset import ingest_demo_file
+from caducus.events import CanonicalEvent
+from caducus.storage import get_events_table, get_events_for_group, put_events
+
+
+FIXTURES_DIR = Path(__file__).resolve().parent / "fixtures"
+
+
+def test_canonical_event_roundtrip() -> None:
+    """CanonicalEvent serializes and deserializes."""
+    ev = CanonicalEvent(
+        id="e1",
+        timestamp="2024-01-01T12:00:00Z",
+        source="hdfs-demo",
+        group_id="hdfs-demo:DataNode",
+        text="Receiving block blk_123",
+        metadata={"level": "INFO"},
+    )
+    d = ev.to_dict()
+    ev2 = CanonicalEvent.from_dict(d)
+    assert ev2.id == ev.id
+    assert ev2.text == ev.text
+    assert ev2.metadata == ev.metadata
+
+
+def test_demo_ingest_writes_canonical_events() -> None:
+    """Demo ingest reads CSV and writes one canonical event per row."""
+    csv_path = FIXTURES_DIR / "demo_hdfs_sample.csv"
+    if not csv_path.exists():
+        pytest.skip("fixture demo_hdfs_sample.csv not found")
+    with tempfile.TemporaryDirectory() as tmp:
+        table = get_events_table(tmp)
+        count = ingest_demo_file(str(csv_path), table)
+        assert count == 5
+        rows = get_events_for_group(table, "hdfs-demo:DataNode")
+        assert len(rows) == 3
+        rows_nn = get_events_for_group(table, "hdfs-demo:NameNode")
+        assert len(rows_nn) == 2
+        for r in rows + rows_nn:
+            assert "id" in r and "text" in r and "timestamp" in r
+            assert r["source"] == "hdfs-demo"
+            assert "group_id" in r
+
+
+def test_analyze_requires_biblicus() -> None:
+    """Analyze path imports Biblicus; we only test that adapter imports."""
+    try:
+        from caducus.biblicus_adapter import _events_to_timestamped_text
+    except ImportError:
+        pytest.skip("biblicus reinforcement-memory not installed")
+    from biblicus.analysis.reinforcement_memory import TimestampedText
+
+    rows = [
+        {
+            "id": "1",
+            "timestamp": "2024-01-01T12:00:00Z",
+            "source": "hdfs-demo",
+            "group_id": "hdfs-demo:DataNode",
+            "text": "Receiving block",
+            "metadata": {},
+        }
+    ]
+    texts = _events_to_timestamped_text(rows, "hdfs-demo:DataNode")
+    assert len(texts) == 1
+    assert isinstance(texts[0], TimestampedText)
+    assert texts[0].text == "Receiving block"
+
+
+def test_groups_lists_group_ids_after_ingest() -> None:
+    """After ingest, groups command returns sorted distinct group_ids."""
+    csv_path = FIXTURES_DIR / "demo_hdfs_sample.csv"
+    if not csv_path.exists():
+        pytest.skip("fixture demo_hdfs_sample.csv not found")
+    with tempfile.TemporaryDirectory() as tmp:
+        table = get_events_table(tmp)
+        ingest_demo_file(str(csv_path), table)
+        from caducus.storage import list_group_ids
+
+        group_ids = list_group_ids(table)
+        assert "hdfs-demo:DataNode" in group_ids
+        assert "hdfs-demo:NameNode" in group_ids
+        assert group_ids == sorted(group_ids)
+
+
+def test_demo_run_produces_stable_topic_output() -> None:
+    """End-to-end: ingest fixture, run analyze for a real group, assert topic output shape."""
+    try:
+        from caducus.biblicus_adapter import run_analysis_for_group
+    except ImportError:
+        pytest.skip("biblicus reinforcement-memory not installed")
+    csv_path = FIXTURES_DIR / "demo_hdfs_sample.csv"
+    if not csv_path.exists():
+        pytest.skip("fixture demo_hdfs_sample.csv not found")
+    with tempfile.TemporaryDirectory() as tmp:
+        table = get_events_table(tmp)
+        ingest_demo_file(str(csv_path), table)
+        import io
+        import sys
+
+        out = io.StringIO()
+        old_stdout = sys.stdout
+        sys.stdout = out
+        try:
+            run_analysis_for_group(tmp, "hdfs-demo:DataNode", table, config=None)
+        finally:
+            sys.stdout = old_stdout
+        text = out.getvalue()
+    assert "Group:" in text
+    assert "Texts:" in text or "texts_analyzed" in text or "n=" in text
+    assert "n=" in text
+    assert "[" in text and "/" in text