zettelforge 2.4.0__tar.gz → 2.4.2__tar.gz

{zettelforge-2.4.0 → zettelforge-2.4.2}/CHANGELOG.md

@@ -6,6 +6,146 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+## [2.4.2] - 2026-04-24
+
+Patch release bundling the RFC-010 enrichment-pipeline hotfix with the
+RFC-009 Phase 0.5 latency-attribution instrumentation. Response to the
+2026-04-24 Vigil telemetry audit.
+
+### Fixed
+
+- **RFC-010 hotfix — `OllamaProvider` timeout plumbing** (#88). The
+  constructor's `**_: Any` absorbed the configured `timeout` kwarg, so
+  `ollama.Client(host=...)` was built with no timeout and `remember()`
+  could hang up to 66.5s on a slow backend. `timeout` is now a
+  first-class parameter (default 60.0s) threaded through to the client.
+- **RFC-010 hotfix — consolidation shutdown race** (#88). A third
+  `iterate_notes()` site at `consolidation.py:224` was not covered by
+  PR #84's two-site guard. Added a two-layer defense: fast-path
+  `_accepting` pre-check plus a narrow `BackendClosedError` catch on
+  the iterator itself. Clean skip instead of `consolidation_failed`
+  log noise during `atexit`.
+
+### Added
+
+- **RFC-009 Phase 0.5 — per-phase timers in `remember()`** (#90).
+  `memory_manager.remember()` now wraps each direct-store phase
+  (`construct`, `write_note`, `lance_index`, `entity_index`,
+  `consolidation_observe`, `supersession`, `kg_update`,
+  `enrichment_dispatch`) in `time.perf_counter()` and emits the
+  breakdown inside the existing `ocsf_api_activity` event as
+  `phase_timings_ms`. Pure observability. Enables Vigil-side latency
+  attribution without host-side profilers, which do not apply to a
+  library-per-turn deployment. `enrichment_dispatch` is intentionally
+  skipped in `sync=True` runs so inline LLM work cannot corrupt the
+  dispatch bucket.
+- **Phase 0.5 preliminary attribution artifact** (#91) —
+  `docs/superpowers/research/2026-04-24-phase-0.5-attribution-prelim.md`.
+  Analyses 961 real `remember()` calls from Vigil's v2.4.1 OCSF log
+  and finds **98.4% of `remember()` wall-clock is one LanceDB `Update`
+  on the `notes_cti` shard**, which has 7,356 uncompacted fragments
+  versus 458 on the healthy `notes_general` shard. Reshapes RFC-009's
+  Phase 1–6 priority ordering: those phases target the LLM / queue /
+  consolidation paths, which are not what drives the 5.7s average.
+  To be refined or falsified with `phase_timings_ms` data from this
+  release.
+
+### Does NOT address
+
+- The ~2,329 enrichment-job drops/day are still present. Those are
+  caused by HTTP 200 + empty Ollama responses (Ollama returns
+  successfully but with no parseable body), not by hangs — RFC-010's
+  timeout fix does not touch them. The durable outbox + circuit
+  breaker in RFC-009 Phases 1–3 (v2.5.0) is the real fix.
+- LanceDB fragment accumulation on `notes_cti` is identified here but
+  not fixed here. RFC-009 is being revised to add periodic compaction
+  to Phase 1 scope.
+
+## [2.4.1] - 2026-04-24
+
+Operational telemetry (RFC-007), TypeDB authentication hardening, and a
+tranche of SQLite backend correctness fixes surfaced by the sqlite
+review in issue #83.
+
+### Added
+
+- **Operational telemetry** (RFC-007, #85) — per-query recall /
+  synthesis metrics captured to `~/.amem/telemetry/telemetry_YYYY-MM-DD.jsonl`
+  when `ZETTELFORGE_LOG_LEVEL=DEBUG`. Five shipped components:
+  - `TelemetryCollector` class (`start_query` / `log_recall` /
+    `log_synthesis` / `log_feedback` / `auto_feedback_from_synthesis`)
+    with INFO/DEBUG-gated field sets, 1-hour TTL on in-memory query
+    context, and thread-safe JSONL append.
+  - `MemoryManager` integration — `recall()` and `synthesize()` gain a
+    non-breaking `actor=` kwarg; OCSF events extended via the
+    sanctioned `unmapped` object with a `zf_` prefix (class_uid 6002
+    compliant). `recall()` wraps `retriever.retrieve()` and
+    `graph_retriever.retrieve_note_ids()` with narrow-scope
+    `perf_counter` deltas for `vector_latency_ms` / `graph_latency_ms`.
+  - Daily aggregator (`python -m zettelforge.scripts.telemetry_aggregator`)
+    emitting a `DailyMetrics` JSON report (latency averages, tier
+    distribution, unused-notes count, top-utility notes).
+  - Human-evaluation workflow — 6-question rubric (`docs/human-evaluation-rubric.md`),
+    sampler script (`python -m zettelforge.scripts.human_eval_sampler`)
+    that selects 20 random briefings as a fill-in Markdown template,
+    and a `--write-events` path to append `event_type: "human_eval"`
+    entries back to telemetry.
+  - Optional Streamlit dashboard (`streamlit run
+    src/zettelforge/scripts/telemetry_dashboard.py`) — query volume,
+    latency p50/p95/max, tier distribution, utility trend,
+    unused-notes warning.
+  - Privacy contract: raw note content never persisted (IDs / tiers /
+    source_types / domains only); query text truncated at 200 chars
+    INFO / 500 chars DEBUG; local-only, no network calls.
+
+### Fixed
+
+- **SQLite shutdown NPE** (#84, issue #83 H3) — `close()` and
+  `initialize()` are now lock-protected and idempotent. Readers and
+  writers raise a clean `BackendClosedError` (new, in
+  `storage_backend`) instead of the opaque `AttributeError: 'NoneType'
+  object has no attribute 'execute'` seen 170× in production logs on
+  2026-04-23 during atexit. `memory_manager._enrichment_loop` and
+  `_drain_enrichment_queue` catch `BackendClosedError` and exit
+  cleanly.
+- **SQLite torn snapshot** (#84, issue #83 C1) — `export_snapshot()`
+  now uses `sqlite3.Connection.backup()` for a page-consistent copy.
+  The previous `shutil.copy2` path could produce a corrupt backup
+  missing `-wal` / `-shm` sidecars, unsafe for DR restore.
+- **SQLite reindex race** (#84, issue #83 C2) — `reindex_vector()` now
+  uses a single-lock targeted `UPDATE` on the `embedding_vector`
+  column. The previous `get_note_by_id → rewrite_note` path spanned
+  two lock acquisitions and could clobber concurrent
+  `mark_access_dirty` / `evolve` / supersede edits via
+  `INSERT OR REPLACE`.
+
+### Security
+
+- **TypeDB authentication hardening** (#82) — removed known-insecure
+  `admin` / `password` defaults from `TypeDBConfig` and
+  `config.default.yaml`. `TypeDBConfig.__repr__` now redacts
+  non-empty passwords as `***`. The config loader resolves
+  `${TYPEDB_USERNAME}` / `${TYPEDB_PASSWORD}` env-var references in
+  YAML (same pattern already used for `llm.api_key`), so secrets can
+  stay in env / container secret stores rather than on disk.
+  Migration: set `TYPEDB_USERNAME` / `TYPEDB_PASSWORD` in your
+  environment or use the `${VAR}` references in a local
+  `config.yaml`. Direct env overrides (`TYPEDB_USERNAME=…`) already
+  worked and are unaffected.
+
+### Docs
+
+- **Architecture Deep Dive + Module Inventory for v2.4.0** (#80) —
+  reference-level architecture documentation.
+- **RFC-007 Operational Telemetry** (#85) — full design doc including
+  the four subagent-resolved frictions (caller-opt-in query_id
+  correlation, narrow-scope latency instrumentation, OCSF unmapped
+  extension, hybrid `__new__`-bypass integration tests).
+- **Human Evaluation Rubric** (#85) — 6-question monthly review
+  rubric with scoring summary table.
+- **Troubleshoot guide** (#85) — "Operational telemetry" subsection
+  covering the three CLI entry points and the privacy contract.
+
 ## [2.4.0] - 2026-04-19
 
 Detection-rules-as-memory, MCP Registry publication, SQLite concurrency

{zettelforge-2.4.0 → zettelforge-2.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zettelforge
-Version: 2.4.0
+Version: 2.4.2
 Summary: ZettelForge: Agentic Memory System with vector search, knowledge graph, and synthesis
 Project-URL: Homepage, https://github.com/rolandpg/zettelforge
 Project-URL: Documentation, https://docs.threatrecall.ai
@@ -59,7 +59,9 @@ Description-Content-Type: text/markdown
 
 **The only agentic memory system built for cyber threat intelligence.**
 
-Persistent memory for AI agents and Claude Code — with CTI entity extraction, STIX knowledge graphs, threat-actor alias resolution, and offline-first RAG. MCP server included. No cloud, no API keys.
+When a senior analyst leaves, two or three years of context walks out with them — customer environments, prior investigations, actor TTPs, false-positive patterns, every hard-won "wait, we've seen this before." ZettelForge is an agentic memory system built so that context stays with the team.
+
+It extracts CVEs, threat actors, IOCs, and ATT&CK techniques from analyst notes and threat reports, resolves aliases (APT28 = Fancy Bear = STRONTIUM = Sofacy), builds a STIX 2.1 knowledge graph, and serves every past investigation back to your analysts — and to Claude Code via MCP — in natural language. Runs entirely in-process. No API keys. No cloud. No data leaves the host.
 
 [![PyPI](https://img.shields.io/pypi/v/zettelforge)](https://pypi.org/project/zettelforge/)
 [![Downloads/month](https://static.pepy.tech/personalized-badge/zettelforge?period=month&units=international_system&left_color=grey&right_color=blue&left_text=downloads%2Fmonth)](https://pepy.tech/projects/zettelforge)
@@ -67,7 +69,7 @@ Persistent memory for AI agents and Claude Code — with CTI entity extraction,
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/rolandpg/zettelforge/actions/workflows/ci.yml/badge.svg)](https://github.com/rolandpg/zettelforge/actions)
 
-**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted](https://threatrecall.ai)**
+**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai)**
 
 <p align="center">
 <a href="https://www.buymeacoffee.com/xypher22pr0" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-green.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;" ></a>
@@ -78,24 +80,25 @@ Persistent memory for AI agents and Claude Code — with CTI entity extraction,
 
 > If ZettelForge fits a CTI workflow you run, a star is the fastest signal that this category is worth continuing to invest in.
 
-## Why ZettelForge?
+## The problem
 
-General-purpose memory systems don't understand threat intelligence. They can't tell APT28 from Fancy Bear, don't know that CVE-2024-3094 is the XZ Utils backdoor, and can't track how intelligence evolves across reports. When your agent forgets context between investigations, you end up re-reading the same reports and re-building the same mental models.
+Every SOC loses analysts. When they leave, investigation context, actor attribution, and environment-specific false-positive patterns go with them. Their replacements re-open the same tickets, re-read the same reports, and re-build the same mental models from scratch.
 
-ZettelForge was built from the ground up for analysts who think in threat graphs, not chat histories. It extracts CVEs, threat actors, IOCs, and MITRE ATT&CK techniques automatically, resolves aliases across naming conventions, builds a knowledge graph with causal relationships, and retrieves memories using intent-aware blended search -- all offline, with no API keys or cloud dependencies.
+General-purpose AI memory systems don't fix this for security teams. They can't tell APT28 from Fancy Bear, don't know that CVE-2024-3094 is the XZ Utils backdoor, can't parse Sigma or YARA, and have no concept of MITRE ATT&CK technique IDs. When a CTI analyst gives them a year of intel reports, they get back fuzzy semantic search over chat history.
 
->"Memory augmentation closes 33% of the gap between small and large models on CTI tasks (CTI-REALM, Microsoft 2026)." [1]
+ZettelForge was built for analysts who think in threat graphs. It extracts CVEs, threat actors, IOCs, and ATT&CK techniques automatically, resolves aliases across naming conventions, builds a knowledge graph with causal relationships, and retrieves memories using intent-aware blended search — all in-process, with no external API dependency.
 
+>"Memory augmentation closes 33% of the gap between small and large models on CTI tasks (CTI-REALM, Microsoft 2026)." [1]
 
-| Feature | ZettelForge | Mem0 | Graphiti | Cognee |
-|---------|------------|------|----------|--------|
+| Capability | ZettelForge | Mem0 | Graphiti | Cognee |
+|---|---|---|---|---|
 | CTI entity extraction (CVEs, actors, IOCs) | Yes | No | No | No |
 | STIX 2.1 ontology | Yes | No | No | No |
 | Threat actor alias resolution | Yes (APT28 = Fancy Bear) | No | No | No |
 | Knowledge graph with causal triples | Yes | No | Yes | Yes |
 | Intent-classified retrieval (5 types) | Yes | No | No | No |
-| Offline / local-first (no API keys) | Yes | No | No | No |
-| OCSF audit logging | Yes | No | No | No |
+| In-process / no external API required | Yes | No | No | No |
+| Audit logs in OCSF schema | Yes | No | No | No |
 | MCP server (Claude Code) | Yes | No | No | No |
 
 ## Data Pipeline
@@ -110,21 +113,21 @@ ZettelForge was built from the ground up for analysts who think in threat graphs
 
 ## Features
 
-**Entity Extraction** -- Automatically identifies CVEs, threat actors, IOCs (IPs, domains, hashes, URLs, emails), MITRE ATT&CK techniques, campaigns, intrusion sets, tools, people, locations, and organizations. Regex + LLM NER with STIX 2.1 types throughout.
+**Entity Extraction** — Automatically identifies CVEs, threat actors, IOCs (IPs, domains, hashes, URLs, emails), MITRE ATT&CK techniques, campaigns, intrusion sets, tools, people, locations, and organizations. Regex + LLM NER with STIX 2.1 types throughout.
 
-**Knowledge Graph** -- Entities become nodes, co-occurrence becomes edges. LLM infers causal triples ("APT28 *uses* Cobalt Strike"). Temporal edges and supersession track how intelligence evolves.
+**Knowledge Graph** — Entities become nodes, co-occurrence becomes edges. LLM infers causal triples ("APT28 *uses* Cobalt Strike"). Temporal edges and supersession track how intelligence evolves.
 
-**Alias Resolution** -- APT28, Fancy Bear, Sofacy, STRONTIUM all resolve to the same actor node. Works automatically on store and recall.
+**Alias Resolution** — APT28, Fancy Bear, Sofacy, STRONTIUM all resolve to the same actor node. Works automatically on store and recall.
 
-**Blended Retrieval** -- Vector similarity (768-dim fastembed, ONNX) + graph traversal (BFS over knowledge graph edges), weighted by intent classification. Five intent types: factual, temporal, relational, exploratory, causal.
+**Blended Retrieval** — Vector similarity (768-dim fastembed, ONNX) + graph traversal (BFS over knowledge graph edges), weighted by intent classification. Five intent types: factual, temporal, relational, exploratory, causal.
 
-**Memory Evolution** -- With `evolve=True`, new intel is compared to existing memory. LLM decides ADD, UPDATE, DELETE, or NOOP. Stale intel gets superseded. Contradictions get resolved. Duplicates get skipped.
+**Memory Evolution** — With `evolve=True`, new intel is compared to existing memory. LLM decides ADD, UPDATE, DELETE, or NOOP. Stale intel gets superseded. Contradictions get resolved. Duplicates get skipped.
 
-**RAG Synthesis** -- Synthesize answers across all stored memories with direct_answer format.
+**RAG Synthesis** — Synthesize answers across all stored memories with `direct_answer` format.
 
-**Offline-First** -- fastembed (ONNX) for embeddings, llama-cpp-python for LLM features. No API keys, no cloud dependencies.
+**In-process by architecture** — fastembed (ONNX) for embeddings, llama-cpp-python for optional local LLM inference, SQLite + LanceDB for storage, and Ollama on localhost by default. No external API keys are required. Outbound network access may occur on first run when embedding/LLM models are downloaded; after models are preloaded, it can run fully offline (including on air-gapped hosts).
 
-**OCSF Audit Logging** -- Every operation is logged in OCSF format (FedRAMP AU controls).
+**Audit logging in OCSF schema** — Every operation emits a structured event in the Open Cybersecurity Schema Framework format. What you do with the log stream (SIEM, WORM store, nothing) is up to you.
 
 ## Quick Start
 
@@ -137,7 +140,7 @@ from zettelforge import MemoryManager
 
 mm = MemoryManager()
 
-# Store threat intel -- entities extracted automatically
+# Store threat intel — entities extracted automatically
 mm.remember("APT28 uses Cobalt Strike for lateral movement via T1021")
 
 # Recall with alias resolution
@@ -148,7 +151,7 @@ results = mm.recall("What tools does Fancy Bear use?")
 answer = mm.synthesize("Summarize known APT28 TTPs")
 ```
 
-No TypeDB, no Ollama, no Docker -- just `pip install`. Embeddings run in-process via fastembed. LLM features (extraction, synthesis) activate when Ollama is available.
+No TypeDB, no Ollama, no Docker — just `pip install`. Embeddings run in-process via fastembed. LLM features (extraction, synthesis) activate when Ollama is available.
 
 ### With Ollama (enables LLM features)
 
@@ -160,7 +163,7 @@ ollama pull qwen2.5:3b && ollama serve
 ### Memory Evolution
 
 ```python
-# New intel arrives -- evolve=True enables memory evolution:
+# New intel arrives — evolve=True enables memory evolution:
 # LLM extracts facts, compares to existing notes, decides ADD/UPDATE/DELETE/NOOP
 mm.remember(
     "APT28 has shifted tactics. They dropped DROPBEAR and now exploit edge devices.",
@@ -173,25 +176,25 @@ mm.remember(
 
 Every `remember()` call triggers a pipeline:
 
-1. **Entity Extraction** -- regex + LLM NER identifies CVEs, intrusion sets, threat actors, tools, campaigns, ATT&CK techniques, IOCs (IPv4, domain, URL, MD5/SHA1/SHA256, email), people, locations, organizations, events, activities, and temporal references (19 types)
-2. **Knowledge Graph Update** -- entities become nodes, co-occurrence becomes edges, LLM infers causal triples
-3. **Vector Embedding** -- 768-dim fastembed (ONNX, in-process, 7ms/embed) stored in LanceDB
-4. **Supersession Check** -- entity overlap detection marks stale notes as superseded
-5. **Dual-Stream Write** -- fast path returns in ~45ms; causal enrichment is deferred to a background worker
+1. **Entity Extraction** — regex + LLM NER identifies CVEs, intrusion sets, threat actors, tools, campaigns, ATT&CK techniques, IOCs (IPv4, domain, URL, MD5/SHA1/SHA256, email), people, locations, organizations, events, activities, and temporal references (19 types)
+2. **Knowledge Graph Update** — entities become nodes, co-occurrence becomes edges, LLM infers causal triples
+3. **Vector Embedding** — 768-dim fastembed (ONNX, in-process, 7ms/embed) stored in LanceDB
+4. **Supersession Check** — entity overlap detection marks stale notes as superseded
+5. **Dual-Stream Write** — fast path returns in ~45ms; causal enrichment is deferred to a background worker
 
 Every `recall()` call blends two retrieval strategies:
 
-1. **Vector similarity** -- semantic search over embeddings
-2. **Graph traversal** -- BFS over knowledge graph edges, scored by hop distance
-3. **Intent routing** -- query classified as factual/temporal/relational/causal/exploratory, weights adjusted per type
-4. **Cross-encoder reranking** -- ms-marco-MiniLM reorders final results by relevance
+1. **Vector similarity** — semantic search over embeddings
+2. **Graph traversal** — BFS over knowledge graph edges, scored by hop distance
+3. **Intent routing** — query classified as factual/temporal/relational/causal/exploratory, weights adjusted per type
+4. **Cross-encoder reranking** — ms-marco-MiniLM reorders final results by relevance
 
 ## Benchmarks
 
 Evaluated against published academic benchmarks:
 
 | Benchmark | What it measures | Score |
-|-----------|-----------------|-------|
+|---|---|---|
 | **CTI Retrieval** | Attribution, CVE linkage, multi-hop | **75.0%** |
 | **RAGAS** | Retrieval quality (keyword presence) | **78.1%** |
 | **LOCOMO** (ACL 2024) | Conversational memory recall | **22.0%** *(with Ollama cloud models)* |
@@ -221,7 +224,7 @@ Exposed tools: `remember`, `recall`, `synthesize`, `entity`, `graph`, `stats`.
 
 Sigma and YARA rules are first-class memory primitives. Parse, validate, and ingest a rule and its tags become graph edges: MITRE ATT&CK techniques, CVEs, threat-actor aliases, tools, and malware families resolve against the same ontology as every other note. A shared `DetectionRule` supertype carries `SigmaRule` and `YaraRule` subtypes, so a single rule UUID is addressable across both formats.
 
-Sigma rules are validated against the vendored [SigmaHQ JSON schema](https://github.com/SigmaHQ/sigma-specification). YARA rules are parsed with plyara and validated against the [CCCS YARA metadata standard](https://github.com/CybercentreCanada/CCCS-Yara) (tiers: `strict`, `warn`, `non_cccs`). Ingest is idempotent -- re-ingesting an unchanged rule returns the original note via a content-hashed `source_ref`.
+Sigma rules are validated against the vendored [SigmaHQ JSON schema](https://github.com/SigmaHQ/sigma-specification). YARA rules are parsed with plyara and checked against the [CCCS YARA metadata standard](https://github.com/CybercentreCanada/CCCS-Yara) (tiers: `strict`, `warn`, `non_cccs`). Ingest is idempotent — re-ingesting an unchanged rule returns the original note via a content-hashed `source_ref`.
 
 ```python
 from zettelforge import MemoryManager
@@ -238,11 +241,11 @@ ingest_yara("rules/webshell_china_chopper.yar", mm, tier="warn")
 python -m zettelforge.sigma.ingest /path/to/sigma/rules/
 python -m zettelforge.yara.ingest /path/to/yara/rules/ --tier warn
 
-# CI fixture check -- parse + validate, no writes
+# CI fixture check — parse + validate, no writes
 python -m zettelforge.sigma.ingest rules/ --dry-run
 ```
 
-An LLM rule explainer (`zettelforge.detection.explainer.explain`) produces a structured JSON summary -- intent, key fields, evasion notes, false-positive hypotheses -- for any `DetectionRule`. It runs synchronously on demand in v1; async enrichment-queue wiring is v1.1. Rate-limited via `ZETTELFORGE_EXPLAIN_RPM` (default 60 calls/minute).
+An LLM rule explainer (`zettelforge.detection.explainer.explain`) produces a structured JSON summary — intent, key fields, evasion notes, false-positive hypotheses — for any `DetectionRule`. It runs synchronously on demand in v1; async enrichment-queue wiring is v1.1. Rate-limited via `ZETTELFORGE_EXPLAIN_RPM` (default 60 calls/minute).
 
 References: [Sigma spec](https://github.com/SigmaHQ/sigma-specification), [SigmaHQ rules](https://github.com/SigmaHQ/sigma), [CCCS YARA](https://github.com/CybercentreCanada/CCCS-Yara), [YARA docs](https://yara.readthedocs.io).
 
@@ -263,35 +266,31 @@ See [examples/athf_bridge.py](examples/athf_bridge.py).
 
 ## Extensions
 
-ZettelForge is a complete, production-ready agentic memory system.
-Everything documented above works out of the box.
+ZettelForge ships a complete agentic memory core. Everything documented above works from a single `pip install`.
 
-For teams that need TypeDB-scale graph storage, OpenCTI integration,
-or multi-tenant deployment, optional extensions are available:
+For teams that want TypeDB-scale graph storage, OpenCTI integration, or multi-tenant deployment, optional extensions are available:
 
 | Extension | What it adds |
-|-----------|-------------|
+|---|---|
 | TypeDB STIX 2.1 backend | Schema-enforced ontology with inference rules |
 | OpenCTI sync | Bi-directional sync with OpenCTI instances |
 | Multi-tenant auth | OAuth/JWT with per-tenant isolation |
 | Sigma rule generation | Detection rules from extracted IOCs |
 
-Extensions are installed separately:
+Extensions install separately:
 
 ```bash
 pip install zettelforge-enterprise
 ```
 
-**Hosted option:** [ThreatRecall](https://threatrecall.ai) provides
-managed ZettelForge with all extensions, so you don't have to run
-infrastructure yourself.
+**Hosted (private beta):** [ThreatRecall](https://threatrecall.ai) is the managed SaaS version of ZettelForge with enterprise extensions enabled. Currently accepting waitlist signups and a limited number of design partners.
 
 ## Configuration
 
 | Variable | Default | Description |
-|----------|---------|-------------|
+|---|---|---|
 | `AMEM_DATA_DIR` | `~/.amem` | Data directory |
-| `ZETTELFORGE_BACKEND` | `sqlite` | SQLite community backend. TypeDB is available via extension. |
+| `ZETTELFORGE_BACKEND` | `sqlite` | SQLite community backend. TypeDB available via extension. |
 | `ZETTELFORGE_LLM_PROVIDER` | `local` | `local` (llama-cpp) or `ollama` |
 
 See [config.default.yaml](config.default.yaml) for all options.
@@ -302,9 +301,11 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup.
 
 ## License
 
-MIT -- See [LICENSE](LICENSE).
+MIT — See [LICENSE](LICENSE).
+
+## About the author
 
-**Made by Patrick Roland**.
+Built by **Patrick Roland** — Director of SOC Services at Summit 7 Systems, where he built the Vigilance MxDR practice from the ground up. Navy nuclear veteran, CISSP, CCP (CMMC 2.0 Professional). [LinkedIn](https://www.linkedin.com/in/patrickgroland/).
 
 ## Support the Project
 

{zettelforge-2.4.0 → zettelforge-2.4.2}/README.md

@@ -4,7 +4,9 @@
 
 **The only agentic memory system built for cyber threat intelligence.**
 
-Persistent memory for AI agents and Claude Code — with CTI entity extraction, STIX knowledge graphs, threat-actor alias resolution, and offline-first RAG. MCP server included. No cloud, no API keys.
+When a senior analyst leaves, two or three years of context walks out with them — customer environments, prior investigations, actor TTPs, false-positive patterns, every hard-won "wait, we've seen this before." ZettelForge is an agentic memory system built so that context stays with the team.
+
+It extracts CVEs, threat actors, IOCs, and ATT&CK techniques from analyst notes and threat reports, resolves aliases (APT28 = Fancy Bear = STRONTIUM = Sofacy), builds a STIX 2.1 knowledge graph, and serves every past investigation back to your analysts — and to Claude Code via MCP — in natural language. Runs entirely in-process. No API keys. No cloud. No data leaves the host.
 
 [![PyPI](https://img.shields.io/pypi/v/zettelforge)](https://pypi.org/project/zettelforge/)
 [![Downloads/month](https://static.pepy.tech/personalized-badge/zettelforge?period=month&units=international_system&left_color=grey&right_color=blue&left_text=downloads%2Fmonth)](https://pepy.tech/projects/zettelforge)
@@ -12,7 +14,7 @@ Persistent memory for AI agents and Claude Code — with CTI entity extraction,
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/rolandpg/zettelforge/actions/workflows/ci.yml/badge.svg)](https://github.com/rolandpg/zettelforge/actions)
 
-**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted](https://threatrecall.ai)**
+**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai)**
 
 <p align="center">
 <a href="https://www.buymeacoffee.com/xypher22pr0" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-green.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;" ></a>
@@ -23,24 +25,25 @@ Persistent memory for AI agents and Claude Code — with CTI entity extraction,
 
 > If ZettelForge fits a CTI workflow you run, a star is the fastest signal that this category is worth continuing to invest in.
 
-## Why ZettelForge?
+## The problem
 
-General-purpose memory systems don't understand threat intelligence. They can't tell APT28 from Fancy Bear, don't know that CVE-2024-3094 is the XZ Utils backdoor, and can't track how intelligence evolves across reports. When your agent forgets context between investigations, you end up re-reading the same reports and re-building the same mental models.
+Every SOC loses analysts. When they leave, investigation context, actor attribution, and environment-specific false-positive patterns go with them. Their replacements re-open the same tickets, re-read the same reports, and re-build the same mental models from scratch.
 
-ZettelForge was built from the ground up for analysts who think in threat graphs, not chat histories. It extracts CVEs, threat actors, IOCs, and MITRE ATT&CK techniques automatically, resolves aliases across naming conventions, builds a knowledge graph with causal relationships, and retrieves memories using intent-aware blended search -- all offline, with no API keys or cloud dependencies.
+General-purpose AI memory systems don't fix this for security teams. They can't tell APT28 from Fancy Bear, don't know that CVE-2024-3094 is the XZ Utils backdoor, can't parse Sigma or YARA, and have no concept of MITRE ATT&CK technique IDs. When a CTI analyst gives them a year of intel reports, they get back fuzzy semantic search over chat history.
 
->"Memory augmentation closes 33% of the gap between small and large models on CTI tasks (CTI-REALM, Microsoft 2026)." [1]
+ZettelForge was built for analysts who think in threat graphs. It extracts CVEs, threat actors, IOCs, and ATT&CK techniques automatically, resolves aliases across naming conventions, builds a knowledge graph with causal relationships, and retrieves memories using intent-aware blended search — all in-process, with no external API dependency.
 
+>"Memory augmentation closes 33% of the gap between small and large models on CTI tasks (CTI-REALM, Microsoft 2026)." [1]
 
-| Feature | ZettelForge | Mem0 | Graphiti | Cognee |
-|---------|------------|------|----------|--------|
+| Capability | ZettelForge | Mem0 | Graphiti | Cognee |
+|---|---|---|---|---|
 | CTI entity extraction (CVEs, actors, IOCs) | Yes | No | No | No |
 | STIX 2.1 ontology | Yes | No | No | No |
 | Threat actor alias resolution | Yes (APT28 = Fancy Bear) | No | No | No |
 | Knowledge graph with causal triples | Yes | No | Yes | Yes |
 | Intent-classified retrieval (5 types) | Yes | No | No | No |
-| Offline / local-first (no API keys) | Yes | No | No | No |
-| OCSF audit logging | Yes | No | No | No |
+| In-process / no external API required | Yes | No | No | No |
+| Audit logs in OCSF schema | Yes | No | No | No |
 | MCP server (Claude Code) | Yes | No | No | No |
 
 ## Data Pipeline
@@ -55,21 +58,21 @@ ZettelForge was built from the ground up for analysts who think in threat graphs
 
 ## Features
 
-**Entity Extraction** -- Automatically identifies CVEs, threat actors, IOCs (IPs, domains, hashes, URLs, emails), MITRE ATT&CK techniques, campaigns, intrusion sets, tools, people, locations, and organizations. Regex + LLM NER with STIX 2.1 types throughout.
+**Entity Extraction** — Automatically identifies CVEs, threat actors, IOCs (IPs, domains, hashes, URLs, emails), MITRE ATT&CK techniques, campaigns, intrusion sets, tools, people, locations, and organizations. Regex + LLM NER with STIX 2.1 types throughout.
 
-**Knowledge Graph** -- Entities become nodes, co-occurrence becomes edges. LLM infers causal triples ("APT28 *uses* Cobalt Strike"). Temporal edges and supersession track how intelligence evolves.
+**Knowledge Graph** — Entities become nodes, co-occurrence becomes edges. LLM infers causal triples ("APT28 *uses* Cobalt Strike"). Temporal edges and supersession track how intelligence evolves.
 
-**Alias Resolution** -- APT28, Fancy Bear, Sofacy, STRONTIUM all resolve to the same actor node. Works automatically on store and recall.
+**Alias Resolution** — APT28, Fancy Bear, Sofacy, STRONTIUM all resolve to the same actor node. Works automatically on store and recall.
 
-**Blended Retrieval** -- Vector similarity (768-dim fastembed, ONNX) + graph traversal (BFS over knowledge graph edges), weighted by intent classification. Five intent types: factual, temporal, relational, exploratory, causal.
+**Blended Retrieval** — Vector similarity (768-dim fastembed, ONNX) + graph traversal (BFS over knowledge graph edges), weighted by intent classification. Five intent types: factual, temporal, relational, exploratory, causal.
 
-**Memory Evolution** -- With `evolve=True`, new intel is compared to existing memory. LLM decides ADD, UPDATE, DELETE, or NOOP. Stale intel gets superseded. Contradictions get resolved. Duplicates get skipped.
+**Memory Evolution** — With `evolve=True`, new intel is compared to existing memory. LLM decides ADD, UPDATE, DELETE, or NOOP. Stale intel gets superseded. Contradictions get resolved. Duplicates get skipped.
 
-**RAG Synthesis** -- Synthesize answers across all stored memories with direct_answer format.
+**RAG Synthesis** — Synthesize answers across all stored memories with `direct_answer` format.
 
-**Offline-First** -- fastembed (ONNX) for embeddings, llama-cpp-python for LLM features. No API keys, no cloud dependencies.
+**In-process by architecture** — fastembed (ONNX) for embeddings, llama-cpp-python for optional local LLM inference, SQLite + LanceDB for storage, and Ollama on localhost by default. No external API keys are required. Outbound network access may occur on first run when embedding/LLM models are downloaded; after models are preloaded, it can run fully offline (including on air-gapped hosts).
 
-**OCSF Audit Logging** -- Every operation is logged in OCSF format (FedRAMP AU controls).
+**Audit logging in OCSF schema** — Every operation emits a structured event in the Open Cybersecurity Schema Framework format. What you do with the log stream (SIEM, WORM store, nothing) is up to you.
 
 ## Quick Start
 
@@ -82,7 +85,7 @@ from zettelforge import MemoryManager
 
 mm = MemoryManager()
 
-# Store threat intel -- entities extracted automatically
+# Store threat intel — entities extracted automatically
 mm.remember("APT28 uses Cobalt Strike for lateral movement via T1021")
 
 # Recall with alias resolution
@@ -93,7 +96,7 @@ results = mm.recall("What tools does Fancy Bear use?")
 answer = mm.synthesize("Summarize known APT28 TTPs")
 ```
 
-No TypeDB, no Ollama, no Docker -- just `pip install`. Embeddings run in-process via fastembed. LLM features (extraction, synthesis) activate when Ollama is available.
+No TypeDB, no Ollama, no Docker — just `pip install`. Embeddings run in-process via fastembed. LLM features (extraction, synthesis) activate when Ollama is available.
 
 ### With Ollama (enables LLM features)
 
@@ -105,7 +108,7 @@ ollama pull qwen2.5:3b && ollama serve
 ### Memory Evolution
 
 ```python
-# New intel arrives -- evolve=True enables memory evolution:
+# New intel arrives — evolve=True enables memory evolution:
 # LLM extracts facts, compares to existing notes, decides ADD/UPDATE/DELETE/NOOP
 mm.remember(
     "APT28 has shifted tactics. They dropped DROPBEAR and now exploit edge devices.",
@@ -118,25 +121,25 @@ mm.remember(
 
 Every `remember()` call triggers a pipeline:
 
-1. **Entity Extraction** -- regex + LLM NER identifies CVEs, intrusion sets, threat actors, tools, campaigns, ATT&CK techniques, IOCs (IPv4, domain, URL, MD5/SHA1/SHA256, email), people, locations, organizations, events, activities, and temporal references (19 types)
-2. **Knowledge Graph Update** -- entities become nodes, co-occurrence becomes edges, LLM infers causal triples
-3. **Vector Embedding** -- 768-dim fastembed (ONNX, in-process, 7ms/embed) stored in LanceDB
-4. **Supersession Check** -- entity overlap detection marks stale notes as superseded
-5. **Dual-Stream Write** -- fast path returns in ~45ms; causal enrichment is deferred to a background worker
+1. **Entity Extraction** — regex + LLM NER identifies CVEs, intrusion sets, threat actors, tools, campaigns, ATT&CK techniques, IOCs (IPv4, domain, URL, MD5/SHA1/SHA256, email), people, locations, organizations, events, activities, and temporal references (19 types)
+2. **Knowledge Graph Update** — entities become nodes, co-occurrence becomes edges, LLM infers causal triples
+3. **Vector Embedding** — 768-dim fastembed (ONNX, in-process, 7ms/embed) stored in LanceDB
+4. **Supersession Check** — entity overlap detection marks stale notes as superseded
+5. **Dual-Stream Write** — fast path returns in ~45ms; causal enrichment is deferred to a background worker
 
 Every `recall()` call blends two retrieval strategies:
 
-1. **Vector similarity** -- semantic search over embeddings
-2. **Graph traversal** -- BFS over knowledge graph edges, scored by hop distance
-3. **Intent routing** -- query classified as factual/temporal/relational/causal/exploratory, weights adjusted per type
-4. **Cross-encoder reranking** -- ms-marco-MiniLM reorders final results by relevance
+1. **Vector similarity** — semantic search over embeddings
+2. **Graph traversal** — BFS over knowledge graph edges, scored by hop distance
+3. **Intent routing** — query classified as factual/temporal/relational/causal/exploratory, weights adjusted per type
+4. **Cross-encoder reranking** — ms-marco-MiniLM reorders final results by relevance
 
 ## Benchmarks
 
 Evaluated against published academic benchmarks:
 
 | Benchmark | What it measures | Score |
-|-----------|-----------------|-------|
+|---|---|---|
 | **CTI Retrieval** | Attribution, CVE linkage, multi-hop | **75.0%** |
 | **RAGAS** | Retrieval quality (keyword presence) | **78.1%** |
 | **LOCOMO** (ACL 2024) | Conversational memory recall | **22.0%** *(with Ollama cloud models)* |
@@ -166,7 +169,7 @@ Exposed tools: `remember`, `recall`, `synthesize`, `entity`, `graph`, `stats`.
 
 Sigma and YARA rules are first-class memory primitives. Parse, validate, and ingest a rule and its tags become graph edges: MITRE ATT&CK techniques, CVEs, threat-actor aliases, tools, and malware families resolve against the same ontology as every other note. A shared `DetectionRule` supertype carries `SigmaRule` and `YaraRule` subtypes, so a single rule UUID is addressable across both formats.
 
-Sigma rules are validated against the vendored [SigmaHQ JSON schema](https://github.com/SigmaHQ/sigma-specification). YARA rules are parsed with plyara and validated against the [CCCS YARA metadata standard](https://github.com/CybercentreCanada/CCCS-Yara) (tiers: `strict`, `warn`, `non_cccs`). Ingest is idempotent -- re-ingesting an unchanged rule returns the original note via a content-hashed `source_ref`.
+Sigma rules are validated against the vendored [SigmaHQ JSON schema](https://github.com/SigmaHQ/sigma-specification). YARA rules are parsed with plyara and checked against the [CCCS YARA metadata standard](https://github.com/CybercentreCanada/CCCS-Yara) (tiers: `strict`, `warn`, `non_cccs`). Ingest is idempotent — re-ingesting an unchanged rule returns the original note via a content-hashed `source_ref`.
 
 ```python
 from zettelforge import MemoryManager
@@ -183,11 +186,11 @@ ingest_yara("rules/webshell_china_chopper.yar", mm, tier="warn")
 python -m zettelforge.sigma.ingest /path/to/sigma/rules/
 python -m zettelforge.yara.ingest /path/to/yara/rules/ --tier warn
 
-# CI fixture check -- parse + validate, no writes
+# CI fixture check — parse + validate, no writes
 python -m zettelforge.sigma.ingest rules/ --dry-run
 ```
 
-An LLM rule explainer (`zettelforge.detection.explainer.explain`) produces a structured JSON summary -- intent, key fields, evasion notes, false-positive hypotheses -- for any `DetectionRule`. It runs synchronously on demand in v1; async enrichment-queue wiring is v1.1. Rate-limited via `ZETTELFORGE_EXPLAIN_RPM` (default 60 calls/minute).
+An LLM rule explainer (`zettelforge.detection.explainer.explain`) produces a structured JSON summary — intent, key fields, evasion notes, false-positive hypotheses — for any `DetectionRule`. It runs synchronously on demand in v1; async enrichment-queue wiring is v1.1. Rate-limited via `ZETTELFORGE_EXPLAIN_RPM` (default 60 calls/minute).
 
 References: [Sigma spec](https://github.com/SigmaHQ/sigma-specification), [SigmaHQ rules](https://github.com/SigmaHQ/sigma), [CCCS YARA](https://github.com/CybercentreCanada/CCCS-Yara), [YARA docs](https://yara.readthedocs.io).
 
@@ -208,35 +211,31 @@ See [examples/athf_bridge.py](examples/athf_bridge.py).
 
 ## Extensions
 
-ZettelForge is a complete, production-ready agentic memory system.
-Everything documented above works out of the box.
+ZettelForge ships a complete agentic memory core. Everything documented above works from a single `pip install`.
 
-For teams that need TypeDB-scale graph storage, OpenCTI integration,
-or multi-tenant deployment, optional extensions are available:
+For teams that want TypeDB-scale graph storage, OpenCTI integration, or multi-tenant deployment, optional extensions are available:
 
 | Extension | What it adds |
-|-----------|-------------|
+|---|---|
 | TypeDB STIX 2.1 backend | Schema-enforced ontology with inference rules |
 | OpenCTI sync | Bi-directional sync with OpenCTI instances |
 | Multi-tenant auth | OAuth/JWT with per-tenant isolation |
 | Sigma rule generation | Detection rules from extracted IOCs |
 
-Extensions are installed separately:
+Extensions install separately:
 
 ```bash
 pip install zettelforge-enterprise
 ```
 
-**Hosted option:** [ThreatRecall](https://threatrecall.ai) provides
-managed ZettelForge with all extensions, so you don't have to run
-infrastructure yourself.
+**Hosted (private beta):** [ThreatRecall](https://threatrecall.ai) is the managed SaaS version of ZettelForge with enterprise extensions enabled. Currently accepting waitlist signups and a limited number of design partners.
 
 ## Configuration
 
 | Variable | Default | Description |
-|----------|---------|-------------|
+|---|---|---|
 | `AMEM_DATA_DIR` | `~/.amem` | Data directory |
-| `ZETTELFORGE_BACKEND` | `sqlite` | SQLite community backend. TypeDB is available via extension. |
+| `ZETTELFORGE_BACKEND` | `sqlite` | SQLite community backend. TypeDB available via extension. |
 | `ZETTELFORGE_LLM_PROVIDER` | `local` | `local` (llama-cpp) or `ollama` |
 
 See [config.default.yaml](config.default.yaml) for all options.
@@ -247,9 +246,11 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup.
 
 ## License
 
-MIT -- See [LICENSE](LICENSE).
+MIT — See [LICENSE](LICENSE).
+
+## About the author
 
-**Made by Patrick Roland**.
+Built by **Patrick Roland** — Director of SOC Services at Summit 7 Systems, where he built the Vigilance MxDR practice from the ground up. Navy nuclear veteran, CISSP, CCP (CMMC 2.0 Professional). [LinkedIn](https://www.linkedin.com/in/patrickgroland/).
 
 ## Support the Project