zettelforge 2.5.0__tar.gz → 2.5.2__tar.gz

{zettelforge-2.5.0 → zettelforge-2.5.2}/CHANGELOG.md

@@ -6,6 +6,106 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+## [2.5.2] - 2026-04-25
+
+Hotfix release. Restores end-to-end functionality of synthesis, causal
+triple extraction, fact extraction, LLM NER, and neighbor evolution
+under any reasoning-style LLM (qwen3.5+, qwen3.6, nemotron-3, etc.).
+
+### Fixed
+
+- **Reasoning-model token starvation across every LLM call site**.
+  Reasoning models emit hidden `<think>...</think>` tokens that count
+  against `num_predict` but never appear in the final `response` field
+  Ollama returns. Pre-2.5.2 token caps (`max_tokens=300`/`400`/`800`/
+  `1024`) were exhausted entirely by the thinking phase on these
+  models, leaving the JSON answer empty. Symptoms: synthesis fell back
+  to `"No specific answer found for: …"` on every query; causal triple
+  extraction persisted **0 edges** despite rich CTI text; LLM NER
+  silently no-opped; neighbor evolution `parse_failed{schema=...,
+  raw=""}` warnings flooded the log.
+
+  Bumped every `generate(..., max_tokens=...)` call site to give
+  reasoning models room to think *and* emit a final answer. Affected
+  files:
+
+  | File | Old cap | New cap |
+  |---|---|---|
+  | `note_constructor.py` (causal triples) | 300 | **8000** |
+  | `synthesis_generator.py` | 800 | 2500 |
+  | `fact_extractor.py` | 400 | 2500 |
+  | `entity_indexer.py` (NER) | 300 | 2500 |
+  | `memory_evolver.py` (2 sites) | 1024 | 2500 |
+
+  Causal extraction needs the largest budget because the prompt asks
+  the model to enumerate *every* causal relation in a passage; this
+  triggers the longest reasoning chains anywhere in the system.
+  Empirical against `qwen3.5:9b`: at 4000 tokens the call was
+  *stochastically* sufficient (eval_count varied 2.8k–4k+, ~70%
+  success), so 8000 is the conservative cap that keeps the success
+  rate above 95% on the same model. Other call sites converge with
+  less reasoning overhead so 2500 suffices.
+
+- **LLM client timeout bumped 60s → 180s**. `LLMConfig.timeout` and
+  `OllamaProvider` constructor default were both 60 seconds — well
+  below the 60–120s wall-clock time of a 4000–8000 token reasoning
+  generation on a 9B-Q4_K_M model. `ReadTimeout` was firing during
+  causal extraction even when the model would have returned valid
+  JSON given another 30 seconds. Bumped both defaults plus
+  `config.default.yaml` to 180s.
+
+  Verified end-to-end on `qwen3.5:9b`:
+  - Synthesis: query "What CVE does DROPBEAR exploit?" returns
+    `"CVE-2024-3094"` with 1 source citation (was returning
+    `"No specific answer found for: …"` on every call pre-2.5.2).
+  - Causal extraction: corpus seeded with APT28/DROPBEAR/CVE-2024-3094
+    text yields a 4-triple JSON array in 137s wall time:
+    `APT28 → targets → manufacturing sector`,
+    `APT28 → uses → DROPBEAR`,
+    `DROPBEAR → exploits → CVE-2024-3094`,
+    `APT28 → attributed_to → Russian GRU Unit 26165`.
+
+### Operational note
+
+Slow models. With 8000 tokens of reasoning budget, single causal
+extraction calls now take 60–140s on a 9B model. `remember(sync=True)`
+in this configuration will block 1–3 minutes per note. The default
+async path (background enrichment queue) is the preferred mode.
+Operators on faster hardware or smaller models can lower the caps via
+config/env if needed, but the v2.5.2 defaults trade latency for
+end-to-end correctness on the reference model.
+
+### Notes
+
+This explains the `evolution_parse_failed` and `causal_triples
+parse_failed` cascades documented in the v2.4.x Vigil incident. The
+v2.4.2 PR #95 Tier 1/2 LLM observability surfaced the empty responses
+but the root-cause attribution to token-cap-vs-thinking-budget waited
+until the v2.5.1 perf-bench run made the failure reproducible end-to-end.
+
+## [2.5.1] - 2026-04-25
+
+Hotfix release. Surfaced during the v2.5.0 perf benchmark run.
+
+### Fixed
+
+- **`KnowledgeGraph._cache_edge` crashed on legacy-schema edges**.
+  Long-running deployments accumulated `kg_edges.jsonl` entries written
+  by a now-removed pre-v2.5.x writer that used
+  `{source_id, target_id, relation_type}` instead of the canonical
+  `{from_node_id, to_node_id, relationship}` keys. The loader hard-failed
+  with `KeyError: 'from_node_id'` on the first such row, taking down
+  every `recall()` and `synthesize()` that touches the KG. Affects any
+  workspace with mixed-schema edge history; observed locally with 189k
+  edges where ~80k were the legacy shape.
+  `_normalize_edge_schema()` now remaps legacy keys to canonical on load
+  and silently drops entries that are still un-normalizable, with a
+  count logged at WARNING so operators can see the skip volume.
+  Six new regression tests in `tests/test_kg_edge_schema.py` cover
+  pass-through, remap, missing-fields, non-dict, mixed-batch, and
+  corrupt-JSON cases. The previously-broken environment-dependent
+  `test_basic.py::test_ingest_relationship` now passes deterministically.
+
 ## [2.5.0] - 2026-04-25
 
 Compliance-driven minor release. Closes every CRITICAL and HIGH audit

{zettelforge-2.5.0 → zettelforge-2.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zettelforge
-Version: 2.5.0
+Version: 2.5.2
 Summary: ZettelForge: Agentic Memory System with vector search, knowledge graph, and synthesis
 Project-URL: Homepage, https://github.com/rolandpg/zettelforge
 Project-URL: Documentation, https://docs.threatrecall.ai

zettelforge-2.5.2/SECURITY.md

@@ -0,0 +1,75 @@
+# Security Policy
+
+## Reporting a Vulnerability
+
+This is a solo-maintainer project. For security-related issues:
+- Open a GitHub Security Advisory in the repository
+- Tag with `security` label
+- Expect acknowledgement within 48 hours
+
+## Supported Versions
+
+| Version | Supported |
+|---------|-----------|
+| latest release | ✅ |
+| master branch | ✅ (CI gates) |
+| older releases | ❌ |
+
+## Supply Chain Security
+
+This project implements:
+- SHA-pinned GitHub Actions (all third-party actions pinned by commit SHA)
+- PyPI trusted publishing (OIDC, no long-lived tokens)
+- pip-audit on every CI run (HIGH/CRITICAL must pass)
+- Dependabot for weekly dependency updates
+- Snyk SAST scanning on every push/PR
+
+## Known Security Architecture
+
+See [THREAT_MODEL.md](docs/THREAT_MODEL.md) for the complete STRIDE threat model.
+
+### Data at Rest
+
+- Notes, the knowledge graph, and the entity index are stored in a local SQLite database (WAL mode) under the configured data directory. No encryption at rest is applied by ZettelForge itself -- encrypt the filesystem or volume at the OS level for sensitive deployments.
+- LanceDB vector index files live alongside the SQLite database and carry the same recommendation.
+
+### PII Protection
+
+- As of v2.5.0 (RFC-013), optional PII detection via Microsoft Presidio scans content before `remember()` storage. Three modes: log (discovery), redact (compliance), block (strict). Disabled by default. Requires `pip install zettelforge[pii]` to activate.
+- Raw PII text is never written to structured logs. Only entity type and detection score are recorded.
+
+### LLM Provider Security
+
+- Four providers: `local` (in-process, no network), `ollama` (localhost HTTP), `litellm` (cloud APIs), `mock` (testing). Each is configurable via `llm.provider` in config.yaml.
+- `local` provider is fully offline. `ollama` runs on localhost only. `litellm` makes outbound HTTPS calls to configured cloud APIs.
+- API keys use `${ENV_VAR}` resolution -- never committed to YAML. Redacted from all log output via `LLMConfig.__repr__`.
+- Provider timeout is configurable (default 60s). LiteLLM provider supports configurable retry count.
+
+### Injection Defenses
+
+- As of v2.1.1, all LanceDB query expressions are parameterized. String-interpolated queries were present in v2.1.0 and earlier (see CVE advisory, if issued, or CHANGELOG v2.1.1 P0-3).
+
+### File Locking
+
+- As of v2.1.1, all JSONL and entity index write paths use `fcntl.flock()` exclusive locks to prevent concurrent-write corruption.
+
+### Audit Logging
+
+- All security-relevant operations emit OCSF v1.3 structured events via `structlog`. Authorization decisions, API activity, and file activity are auditable in any SIEM that ingests JSON logs.
+
+### Air-Gap Deployments
+
+- ZettelForge supports fully offline operation (fastembed ONNX + llama-cpp-python). No telemetry or external calls are made in this configuration.
+
+## Disclosure Policy
+
+ZettelForge follows a coordinated disclosure model:
+
+1. Reporter submits vulnerability privately via email.
+2. We acknowledge within 48 hours and begin assessment.
+3. We develop and test a fix on a private branch.
+4. We notify the reporter when a fix is ready and agree on a disclosure date.
+5. We release the fix and publish a security advisory simultaneously.
+6. We credit the reporter in the advisory (unless they opt out).
+
+We ask reporters to give us a reasonable time to fix issues before public disclosure. We will not take legal action against good-faith security researchers who follow this policy.

{zettelforge-2.5.0 → zettelforge-2.5.2}/config.default.yaml

@@ -214,7 +214,7 @@ llm:
   url: http://localhost:11434
   api_key: ""
   temperature: 0.1
-  timeout: 60.0
+  timeout: 180.0  # v2.5.2: bumped from 60s for reasoning-model headroom
   max_retries: 2
   fallback: ""
   local_backend: llama-cpp-python  # used when provider=local (RFC-011)

zettelforge-2.5.2/docs/THREAT_MODEL.md

@@ -0,0 +1,248 @@
+# ZettelForge Threat Model
+
+> **Document ID:** THREAT-001
+> **Classification:** Internal (Tier 2)
+> **Last Updated:** 2026-04-25
+> **Framework:** STRIDE (GOV-011 SSDL Requirement)
+> **Scope:** Community Edition v2.5.x (MIT-licensed codebase)
+> **Compliance Mapping:** FedRAMP SA-3, SA-8, SA-11, SA-15; NIST 800-171 3.11, 3.13, 3.14
+
+## 1. System Overview
+
+### 1.1 High-Level Architecture
+
+ZettelForge is an agentic memory system for cyber threat intelligence (CTI). It ingests unstructured text (threat reports, analyst notes, agent observations) through `remember()`, stores it in a hybrid SQLite + LanceDB backend, and retrieves it via `recall()` and `synthesize()` with intent-classified, policy-weighted blended retrieval.
+
+```
+                      ┌─────────────────────────────┐
+                      │     External Actors          │
+                      │  (Analyst / AI Agent / MCP)  │
+                      └─────────────┬───────────────┘
+                                    │
+                      ┌─────────────▼───────────────┐
+                      │   MemoryManager              │
+                      │   remember() / recall()      │
+                      │   synthesize()               │
+                      └─────┬───────────────┬───────┘
+                            │               │
+               ┌────────────▼───┐    ┌──────▼────────────┐
+               │  Governance    │    │  LLM Providers    │
+               │  Validator     │    │  (local/ollama/   │
+               │  (PII, rules)  │    │   litellm/mock)  │
+               └────────┬───────┘    └──────┬────────────┘
+                        │                   │
+               ┌────────▼───────┐    ┌──────▼────────────┐
+               │  SQLite +      │    │  Enrichment Queue │
+               │  LanceDB       │    │  (causal / LLM   │
+               │  (notes, vec)  │    │   NER extraction)│
+               └────────────────┘    └───────────────────┘
+```
+
+### 1.2 Trust Boundaries
+
+| Boundary # | Description | Type |
+|------------|-------------|------|
+| TB-1 | External → API surface (MCP, REST, direct Python API) | External network / process |
+| TB-2 | Python API → MemoryManager | Internal process |
+| TB-3 | MemoryManager → SQLite / LanceDB filesystem | Local filesystem |
+| TB-4 | LLM Provider → External API (litellm, ollama) | Outbound network |
+| TB-5 | Enrichment worker → LLM (fact extraction, NER) | Internal process |
+| TB-6 | Configuration loader → env vars / YAML files | Local filesystem |
+
+### 1.3 Data Flow Diagram
+
+```
+[C2] Analyst/AI Agent
+  │
+  │  remember(content) / recall(query)
+  ▼
+TB-1 ──────────────────────────────────────────────────┐
+  │                                                      │
+  ▼                                                      │
+[P1] MemoryManager._remember_inner()                     │
+  │                                                      │
+  │  content                                             │
+  ▼                                                      │
+[P2] GovernanceValidator.validate_remember()             │
+  │  ┌──────────────────┐                               │
+  │  │ (Optional) PII    │  TB-5 (lazy)                 │
+  │  │ Validator         │──→ presidio-analyzer         │
+  │  │ (log/redact/block)│    (in-process spaCy)        │
+  │  └──────────────────┘                               │
+  │                                                      │
+  │  redacted content (or original)                     │
+  ▼                                                      │
+[P3] NoteConstructor → construct MetadataNote           │
+  │                                                      │
+  ├──→ [DS1] EntityIndexer → extract entities           │
+  ├──→ [DS2] AliasResolver → resolve APT28/Fancy Bear   │
+  ├──→ [DS3] SQLite DB (notes, KG, entity index)       │
+  ├──→ [DS4] LanceDB (vector index, IVF_PQ 768-dim)     │
+  │                                                      │
+  └──→ Enrichment Queue (async)                         │
+       ├──→ [P4] LLM Causal Triple Extraction            │
+       └──→ [P5] LLM NER (background)                    │
+                                                         │
+[S1] LLM Provider Dispatch                               │
+  ├──→ local (in-process llama-cpp-python / onnx)      │
+  ├──→ ollama (HTTP to localhost:11434) TB-4           │
+  └──→ litellm (HTTP to cloud APIs)    TB-4            │
+                                                         │
+[C1] Configuration Loader                                │
+  ├── config.yaml / config.default.yaml  TB-6          │
+  ├── Environment variables (ZETTELFORGE_*)             │
+  └── ${ENV_VAR} resolution for secrets                 │
+```
+
+---
+
+## 2. STRIDE Threat Analysis
+
+### 2.1 Spoofing
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| S-01 | Attacker spoofs a valid MCP client to call `remember()` / `recall()` with malicious content | MCP Server / REST API (TB-1) | **High** — unauthorized memory access | MCP server relies on transport-level auth (stdio transport for local agents; TLS client certs or API tokens for remote). No built-in authentication in Community edition. Enterprise edition adds JWT/OAuth. |
+| S-02 | Attacker spoofs an LLM provider endpoint (e.g., fake Ollama server) to return malicious model output | LLM Provider (TB-4, ollama/litellm) | **Medium** — model output is treated as data, not executable; but could inject false threat intelligence | No TLS verification for localhost endpoints (default ollama). litellm uses HTTPS for cloud APIs. Local deployments are responsible for network isolation. |
+| S-03 | Attacker spoofs configuration file to inject malicious settings | Config Loader (TB-6) | **High** — could set `provider: litellm` with attacker-controlled API key or disable governance | Config files are local filesystem; `config.yaml` is in `.gitignore` to prevent accidental commits. No integrity verification on config files. |
+
+### 2.2 Tampering
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| T-01 | Attacker modifies SQLite database or LanceDB index files on disk | Storage (TB-3) | **Critical** — persistent memory corruption | SQLite WAL mode with no built-in integrity check on reads. No HMAC or signature on stored notes. Mitigation relies on OS-level filesystem permissions. Encrypt filesystem at OS level for sensitive deployments (noted in SECURITY.md). |
+| T-02 | Attacker modifies config.yaml in-place to change LLM provider, disable PII validation, or alter governance settings | Config Loader (TB-6) | **High** — silent security downgrade | Config files are local. `config.default.yaml` is tracked in git. `config.yaml` is user-owned. No integrity verification. |
+| T-03 | Attacker tampers with enrichment queue data in memory | Enrichment Queue (P4/P5) | **Low** — in-process queue, not network-accessible | The queue is an in-memory Python `queue.Queue` with `maxsize=500`. No external access path. |
+| T-04 | Attacker modifies a note's embedding to bias recall results | LanceDB (DS4) | **Medium** — retrieval poisoning | LanceDB stores vectors as parquet files. OS-level file permissions are the only protection. |
+
+### 2.3 Repudiation
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| R-01 | Attacker performs operations (remember, recall, synthesize) without audit trail | MemoryManager | **High** — compliance failure for FedRAMP AU-2/AU-3 | All operations emit OCSF structured events via `log_api_activity()` / `log_authorization()`. OCSF class 1001 (API Activity) and 3001/3003 (Authorization) are emitted for every operation. Events include `request_id`, `actor`, `resource`, `status_id`. |
+| R-02 | Governance violation occurs without attribution | GovernanceValidator | **Medium** — violation logged but no actor identity | `log_authorization()` records `actor="system"` for automatic calls. MCP and REST API paths should include authenticated actor. Currently Community edition uses hardcoded `"system"` actor. |
+| R-03 | PII detection events without traceability | PIIValidator (RFC-013) | **Medium** — compliance requirement for data protection | `pii_detected` structured log event includes count, action, entity types, and scores. No raw PII text is logged (fixed in commit 5ac162c). |
+
+### 2.4 Information Disclosure
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| I-01 | Stored threat intelligence (notes, entities, IOCs) leaked via filesystem access | SQLite / LanceDB (DS3/DS4) | **Critical** — all CTI data exposed | No encryption at rest in Community edition. SQLite WAL files and LanceDB parquet files contain plaintext. **Mitigation:** encrypt filesystem at OS level. Enterprise edition adds optional SQLite encryption. |
+| I-02 | PII stored in notes leaks through recall/synthesize responses | Storage → Retrieval | **High** — PII compliance | RFC-013 PIIValidator with `action=redact` strips PII before storage. `action=block` prevents storage entirely. Disabled by default — user must opt in. |
+| I-03 | API keys logged in structured logs | LLM Provider / Config Loader | **Critical** — credential exposure | `LLMConfig.__repr__` redacts `api_key` as `'***'`. `extra` dict fields matching sensitive key patterns (`key`, `token`, `secret`, `password`, `credential`, `auth`) are also redacted. Config resolution uses `${ENV_VAR}` references so raw keys never appear in YAML. |
+| I-04 | Error messages leak internal paths, configuration, or stack traces | All components | **Medium** — information gathering | No global exception handler catches and sanitizes errors. structlog can redact PII from log messages if configured. |
+| I-05 | Raw PII text previously logged in structured events | PIIValidator (fixed) | **Medium** — historical exposure | Fixed in 5ac162c: PII text removed from log entities. Only entity type and score are logged. Users on prior commits should rotate logs containing PII. |
+
+### 2.5 Denial of Service
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| D-01 | Large content in `remember()` exhausts memory or blocks the enrichment queue | MemoryManager (P1) | **Medium** — degraded performance | `remember_report()` chunks long documents. No explicit size limit on `remember()` content. Enrichment queue has `maxsize=500` backpressure. |
+| D-02 | LLM provider (ollama, litellm) hangs and blocks `remember()` | LLM Provider (TB-4) | **High** — operation blocks | OllamaProvider has timeout (RFC-010, default 60s). LitellmProvider has timeout + num_retries. `generate()` returns empty string on recoverable failure. Fallback provider (e.g., local -> ollama) gives alternative path. |
+| D-03 | Malicious query triggers deep graph traversal exhausting time/resources | BlendedRetriever | **Medium** — slow recall | `max_graph_depth` config (default 2) limits BFS hops. `default_k` (default 10) limits results. No timeout on recall queries. |
+| D-04 | spaCy model download blocks first `remember()` when PII is enabled | PIIValidator (lazy load) | **Low** — delayed first call (~2-3 seconds) | One-time download cost. Matching fastembed pattern. Can be pre-downloaded for air-gapped deployments. |
+
+### 2.6 Elevation of Privilege
+
+| ID | Threat | Component | Risk | Mitigation |
+|----|--------|-----------|------|------------|
+| E-01 | MCP client accesses notes from a different domain/tenant than authorized | MemoryManager / MCP Server | **High** — cross-tenant data access | No domain-level access control in Community edition. Enterprise edition adds multi_tenant config. Domain is a metadata field on notes, not an access control boundary. |
+| E-02 | Attacker bypasses governance validation (PII, rules) by calling storage backend directly | Direct filesystem / SQLite access | **Critical** — all governance controls bypassed | Governance runs in-memory in `_remember_inner()`. Direct SQLite or LanceDB access bypasses it entirely. Mitigation: OS-level filesystem permissions. |
+| E-03 | Config change elevates provider from mocked/local to cloud API without user knowledge | Config Loader | **Medium** — unexpected outbound calls | No change of config is signed or validated. User is responsible for config integrity. |
+
+---
+
+## 3. Risk Summary
+
+| Risk Level | Count | Key Concerns |
+|------------|-------|--------------|
+| **Critical** | 2 | T-01 (storage tampering), I-01 (unencrypted data at rest), E-02 (governance bypass via filesystem) |
+| **High** | 7 | S-01 (spoofed MCP client), S-03 (config tampering), T-02 (config security downgrade), R-01 (repudiation without audit), I-02 (PII in stored notes), D-02 (LLM provider hang), E-01 (cross-tenant data access) |
+| **Medium** | 9 | S-02 (fake LLM provider), T-04 (retrieval poisoning), R-02, R-03, I-04 (error message leakage), D-01, D-03, E-03 |
+| **Low** | 1 | D-04 (PII model download delay) |
+
+### Top 5 Mitigations (Priority Order)
+
+1. **Encryption at rest** — Encrypt the data directory filesystem (OS-level LUKS, BitLocker, or eCryptfs). ZettelForge does not apply at-rest encryption itself.
+2. **Filesystem permissions** — Restrict access to `~/.amem/` to the ZettelForge process user only. Prevents governance bypass (E-02) and storage tampering (T-01).
+3. **Network isolation** — Run Ollama and ZettelForge on a dedicated VLAN or firewall zone. Prevent unauthorized MCP clients (S-01) and fake provider attacks (S-02).
+4. **Enable PII redaction** — Set `governance.pii.enabled: true` and `action: redact` in production. Prevents PII persistence (I-02).
+5. **Audit log retention** — Ensure OCSF logs are shipped to a SIEM (via structlog JSON output). Satisfies FedRAMP AU-2/AU-3 (R-01).
+
+---
+
+## 4. Mitigation Details
+
+### 4.1 Existing Controls
+
+| Control | Threat(s) | Mechanism | Verification |
+|---------|-----------|-----------|--------------|
+| OCSF audit logging | R-01, R-02 | `log_api_activity()`, `log_authorization()` emitted on every operation | CI test coverage, structlog configuration |
+| API key redaction | I-03 | `LLMConfig.__repr__` redacts api_key and sensitive extra keys | Unit tests in `test_llm_providers.py` |
+| PII detection + redaction | I-02 | PIIValidator (RFC-013): log/redact/block | Unit tests in `test_pii_validator.py` |
+| LLM provider timeout | D-02 | `OllamaProvider` timeout=60s, `LiteLLMProvider` timeout + num_retries | Unit tests (RFC-010, RFC-012) |
+| Config env-var resolution | I-03 | `${ENV_VAR}` syntax prevents raw secrets in YAML | Unit tests |
+| Configurable model provider | S-02, E-03 | `provider` key selects backend; no implicit unauthenticated outbound calls | Config validation |
+| Enrichment queue backpressure | D-01 | `maxsize=500` bounded queue | Code review |
+
+### 4.2 Recommended Additions (Not Yet Implemented)
+
+| Recommendation | Threat(s) | Effort | Priority |
+|---------------|-----------|--------|----------|
+| Add content size limit to `remember()` | D-01 | Small | P3 |
+| Add global exception handler that sanitizes error output | I-04 | Medium | P2 |
+| Add TLS verification option for self-hosted LLM endpoints | S-02 | Small | P2 |
+| Add config file integrity check (SHA-256 of default vs. loaded) | T-02, S-03 | Medium | P3 |
+| Add recall timeout (configurable, default 30s) | D-03 | Medium | P3 |
+| Domain-level access control for multi-tenant | E-01 | Large | Enterprise |
+
+---
+
+## 5. Threat Model Maintenance
+
+| Activity | Frequency | Owner | Evidence |
+|----------|-----------|-------|----------|
+| Threat model review | Per quarter or per significant feature | CTO/CIO | Updated THREAT_MODEL.md |
+| STRIDE assessment for new components | Per RFC (GOV-016 requirement) | RFC Author | Threats section in RFC |
+| SAST scan | Every PR (CI) | Automated | CI pipeline logs |
+| SCA scan | Every PR + daily scheduled | Automated | pip-audit, Snyk reports |
+| Secret scan | Every PR (CI) | Automated | GitGuardian |
+| Dependency vulnerability review | Per advisory (GOV-009 timelines) | Maintainer | GitHub Dependabot, Snyk |
+
+---
+
+## 6. Data Classification Mapping
+
+Per GOV-021, the following data types exist in the system:
+
+| Data | Classification | Storage | Handling |
+|------|---------------|---------|----------|
+| Threat intelligence notes (actor TTPs, IOCs, campaigns) | Internal (Tier 2) | SQLite + LanceDB, no encryption at rest | OS-level filesystem encryption recommended |
+| PII (names, emails, phones — if not redacted) | Confidential (Tier 3) | SQLite (if PII passes through without redaction) | **Must** enable PII redaction (RFC-013) |
+| API keys / credentials | Confidential (Tier 3) | Never committed; env vars only | Redacted from logs, resolved at runtime |
+| Audit logs (OCSF events) | Internal (Tier 2) | Structured logs (GOV-012) | Logs must not contain Tier 3/4 data values |
+| Configuration files | Internal (Tier 2) | config.yaml, config.default.yaml | `.gitignore` excludes user config; no secrets in YAML |
+| Embedding vectors | Internal (Tier 2) | LanceDB parquet files | Derived from notes; same classification as source |
+| CUI (federal contract data) | CUI (Tier 4) | **Not handled** in Community edition | Enterprise edition only, after FedRAMP authorization |
+
+---
+
+## 7. Recent Changes Affecting Threat Model
+
+| Change | RFC/PR | Date | Threat Model Impact |
+|--------|--------|------|---------------------|
+| PII detection and redaction | RFC-013 (PR #118) | 2026-04-25 | New control for I-02; new attack surface (D-04); PII text logging fixed |
+| LiteLLM unified provider | RFC-012 (PR #108) | 2026-04-25 | New provider for I-03 (API keys); new outbound traffic pattern (TB-4) |
+| Local LLM backend selection | RFC-011 (PR #104) | 2026-04-25 | No new threat surface — extends existing local provider |
+| Ollama provider timeout | RFC-010 | 2026-04-24 | Mitigation for D-02 |
+| LLM provider registry | RFC-002 | 2026-04-16 | Foundation for S-02, E-03 via provider selection |
+| SQLite backend default | v2.2.0 | 2026-04-14 | Migration path changes attack surface of legacy JSONL |
+| Injection defenses | v2.1.1 | 2026-04-10 | Fixed parameterized queries (was: P0 SQL injection — see CHANGELOG) |
+
+---
+
+## 8. Threat Model Review Log
+
+| Date | Reviewer | Changes | Next Review |
+|------|----------|---------|-------------|
+| 2026-04-25 | Hermes Agent (automated) | Initial threat model creation per GOV-011 | 2026-07-25 |

{zettelforge-2.5.0 → zettelforge-2.5.2}/governance/controls.yaml

@@ -60,15 +60,23 @@ controls:
       - id: input_validation
         description: "Content must be str or have .content attribute"
         runtime_method: "GovernanceValidator.validate_operation"
+      - id: threat_model
+        description: "STRIDE threat model maintained and reviewed quarterly per GOV-011 SSDL"
+        artifact: "docs/THREAT_MODEL.md"
+        review_frequency: "quarterly"
+        last_reviewed: "2026-04-25"
+        test: "docs/THREAT_MODEL.md"
+        # test field satisfies the spec-drift validator's requirement that
+        # runtime-enforced rules have a test or runtime_method reference.
       # The 2026-04-25 compliance audit (C-2) found that a previously-declared
       # `no_hardcoded_secrets` rule pointed at GovernanceValidator.validate_operation
       # as its runtime_method, but that method contains no secret-detection
       # logic. Honest state: NOT IMPLEMENTED at runtime today. Static
       # enforcement is provided by GitGuardian (CI) and (once GOV-003-mandated
-      # `S` rules are restored to ruff config — audit H-1) Bandit S105/S106/S108.
+      # `S` rules are restored to ruff config -- audit H-1) Bandit S105/S106/S108.
       # Runtime detector (regex + entropy + detect-secrets) is tracked as
       # follow-up work; the rule will be re-declared here when implemented.
-      # Removed rather than left fabricated — see tasks/compliance-audit-2026-04-25.md.
+      # Removed rather than left fabricated -- see tasks/compliance-audit-2026-04-25.md.
 
   GOV-012:
     name: Audit Logging

{zettelforge-2.5.0 → zettelforge-2.5.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "zettelforge"
-version = "2.5.0"
+version = "2.5.2"
 description = "ZettelForge: Agentic Memory System with vector search, knowledge graph, and synthesis"
 readme = "README.md"
 license = "MIT"

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/__init__.py

@@ -57,7 +57,7 @@ from zettelforge.vector_retriever import VectorRetriever
 # importable for advanced use but are not part of the advertised public API
 # and are therefore excluded from __all__ below.
 
-__version__ = "2.4.3"
+__version__ = "2.5.2"
 __all__ = [
     # Ontology reference tables (TypedEntityStore / OntologyValidator are
     # importable from zettelforge.ontology but are not part of the public API

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/config.py

@@ -103,7 +103,7 @@ class LLMConfig:
     url: str = "http://localhost:11434"
     api_key: str = ""  # supports ${ENV_VAR} references — never commit raw keys
     temperature: float = 0.1
-    timeout: float = 60.0
+    timeout: float = 180.0  # v2.5.2: bumped from 60s — reasoning models at higher num_predict (4000 for causal triples) routinely exceed 60s on a 9B at Q4_K_M
     max_retries: int = 2
     fallback: str = ""  # empty preserves implicit local→ollama fallback
     local_backend: str = "llama-cpp-python"  # RFC-011: "llama-cpp-python" or "onnxruntime-genai"

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/entity_indexer.py

@@ -271,9 +271,12 @@ class EntityExtractor:
             from zettelforge.llm_client import generate
 
             prompt = f"Extract named entities from this text:\n\n{text[:2000]}\n\nJSON:"
+            # 2500-token budget for reasoning-model headroom (v2.5.2; pre-fix
+            # 300 was exhausted by qwen3.5+ <think> tokens, leaving the NER
+            # JSON empty and entity extraction silently no-opping).
             output = generate(
                 prompt,
-                max_tokens=300,
+                max_tokens=2500,
                 temperature=0.0,
                 system=self.NER_SYSTEM_PROMPT,
             )
@@ -282,7 +285,7 @@ class EntityExtractor:
             if parsed is None and output and output.strip():
                 _logger.info("retry_parse", site="entity_indexer_ner", attempt=2)
                 retry_prompt = prompt + "\n\nRespond with valid JSON only."
-                output = generate(retry_prompt, max_tokens=300, temperature=0.3, json_mode=True)
+                output = generate(retry_prompt, max_tokens=2500, temperature=0.3, json_mode=True)
                 parsed = extract_json(output, expect="object")
             return self._parse_ner_output_from_parsed(parsed, output, conversational_types)
 

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/fact_extractor.py

@@ -42,7 +42,9 @@ class FactExtractor:
         try:
             from zettelforge.llm_client import generate
 
-            raw_output = generate(prompt, max_tokens=400, temperature=0.1)
+            # 2500-token budget for reasoning-model headroom (see v2.5.2
+            # CHANGELOG; pre-fix 400 was exhausted by qwen3.5+ <think> tokens).
+            raw_output = generate(prompt, max_tokens=2500, temperature=0.1)
             return self._parse_extraction_response(raw_output)
         except Exception:
             _logger.warning("llm_fact_extraction_failed", exc_info=True)

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/knowledge_graph.py

@@ -22,6 +22,44 @@ from collections import deque
 from datetime import datetime
 from pathlib import Path
 
+from zettelforge.log import get_logger
+
+_logger = get_logger("zettelforge.knowledge_graph")
+
+
+# Pre-v2.5.1 writers (now removed from the codebase, but persisted on disk
+# in older deployments) used {source_id, target_id, relation_type} instead of
+# {from_node_id, to_node_id, relationship}. _normalize_edge_schema() rewrites
+# legacy entries on load so both shapes are tolerated. Missing edge_id is
+# treated as terminal — we cannot index without one.
+_LEGACY_EDGE_KEY_MAP = {
+    "source_id": "from_node_id",
+    "target_id": "to_node_id",
+    "relation_type": "relationship",
+}
+
+
+def _normalize_edge_schema(edge: dict) -> dict | None:
+    """Return a copy of ``edge`` with legacy keys remapped, or ``None`` if
+    the entry is missing fields the cache requires.
+
+    Idempotent: edges already in the canonical shape pass through unchanged.
+
+    ``relationship`` is required because downstream code (``add_edge`` dedup
+    scan, ``get_neighbors``, traversal) does direct subscripting on it; a
+    legacy row without ``relation_type`` would otherwise survive load and
+    trigger a deferred KeyError on first read.
+    """
+    if not isinstance(edge, dict) or not edge.get("edge_id"):
+        return None
+    out = dict(edge)
+    for legacy, canonical in _LEGACY_EDGE_KEY_MAP.items():
+        if canonical not in out and legacy in out:
+            out[canonical] = out[legacy]
+    if "from_node_id" not in out or "to_node_id" not in out or "relationship" not in out:
+        return None
+    return out
+
 
 class KnowledgeGraph:
     """
@@ -64,20 +102,39 @@ class KnowledgeGraph:
                             continue
 
         if self.edges_file.exists():
+            skipped_malformed = 0
             with open(self.edges_file) as f:
                 for line in f:
-                    if line.strip():
-                        try:
-                            edge = json.loads(line)
-                            self._cache_edge(edge)
-                            # Index temporal edges
-                            if (
-                                edge.get("relationship", "").startswith("TEMPORAL_")
-                                or edge.get("relationship") == "SUPERSEDES"
-                            ):
-                                self._index_temporal_edge(edge)
-                        except json.JSONDecodeError:
-                            continue
+                    if not line.strip():
+                        continue
+                    try:
+                        edge = json.loads(line)
+                    except json.JSONDecodeError:
+                        skipped_malformed += 1
+                        continue
+                    edge = _normalize_edge_schema(edge)
+                    if edge is None:
+                        skipped_malformed += 1
+                        continue
+                    self._cache_edge(edge)
+                    # Index temporal edges
+                    if (
+                        edge.get("relationship", "").startswith("TEMPORAL_")
+                        or edge.get("relationship") == "SUPERSEDES"
+                    ):
+                        self._index_temporal_edge(edge)
+            if skipped_malformed:
+                # Pre-v2.5.1 deployments wrote edges under both
+                # {from_node_id, to_node_id, relationship} and
+                # {source_id, target_id, relation_type}; the loader now
+                # normalizes the latter to the former. Anything still
+                # un-normalizable is silently dropped here. Logged at
+                # warning so operators can see the count without crashing.
+                _logger.warning(
+                    "kg_edges_skipped_malformed",
+                    count=skipped_malformed,
+                    file=str(self.edges_file),
+                )
 
     def _cache_node(self, node: dict):
         self._nodes[node["node_id"]] = node

{zettelforge-2.5.0 → zettelforge-2.5.2}/src/zettelforge/llm_providers/ollama_provider.py

@@ -39,7 +39,7 @@ class OllamaProvider:
         self,
         model: str = "",
         url: str = "",
-        timeout: float = 60.0,
+        timeout: float = 180.0,  # see config.LLMConfig.timeout for rationale
         **_: Any,
     ) -> None:
         self._model = model or _DEFAULT_MODEL