zettelforge 2.5.2__tar.gz → 2.6.2__tar.gz

zettelforge-2.6.2/.github/workflows/stale.yml

@@ -0,0 +1,47 @@
+name: Stale issue and PR management
+
+on:
+  schedule:
+    - cron: "0 9 * * 1"  # Every Monday at 09:00 UTC
+
+permissions:
+  issues: write
+  pull-requests: write
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v9
+        with:
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+
+          # Issues
+          stale-issue-message: >
+            This issue has been inactive for 60 days. It will be closed
+            in 14 days unless there is new activity. If this is still
+            relevant, please comment to keep it open.
+          close-issue-message: >
+            This issue has been automatically closed after 14 days of
+            inactivity. It can be reopened at any time.
+          days-before-issue-stale: 60
+          days-before-issue-close: 14
+          stale-issue-label: stale
+          exempt-issue-labels: "p0,planned,help wanted,good first issue"
+
+          # PRs
+          stale-pr-message: >
+            This pull request has been inactive for 60 days. It will be
+            closed in 14 days unless there is new activity.
+          close-pr-message: >
+            This pull request has been automatically closed after 14 days
+            of inactivity.
+          days-before-pr-stale: 60
+          days-before-pr-close: 14
+          stale-pr-label: stale
+          exempt-pr-labels: "draft,work-in-progress"
+
+          # General
+          operations-per-run: 100
+          remove-stale-when-updated: true
+          delete-branch: false

{zettelforge-2.5.2 → zettelforge-2.6.2}/.gitignore

@@ -12,6 +12,17 @@ tasks/
 docs/.ralph/
 agentsync.md
 
+# Internal research/specs/plans — workspace scratch that contains
+# unredacted incident detail, customer references, and intermediate
+# reasoning. Kept on disk for the maintainer; not published.
+docs/superpowers/
+
+# Claude Code session artifacts — agent worktrees, transcripts, and
+# scratch state created by the maintainer's local tooling. Belt-and-
+# suspenders: these were never tracked, but listing them prevents an
+# accidental `git add -A` from sweeping them in.
+.claude/
+
 # MkDocs build output (regenerated from docs/)
 site/
 
@@ -28,6 +39,8 @@ downloads/
 eggs/
 .eggs/
 lib/
+!web/ui/js/lib/
+!web/ui/js/lib/*.js
 lib64/
 parts/
 sdist/

{zettelforge-2.5.2 → zettelforge-2.6.2}/CHANGELOG.md

@@ -6,6 +6,116 @@ Versioning follows [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+## [2.6.2] - 2026-04-27
+
+UI/UX release. Fixes the `/config` page so the Apply button actually works
+and surfaces enum-style settings as dropdowns instead of free-text inputs.
+No data migration. No config changes. No API contract changes.
+
+### Fixed
+
+- **`/config` "Save Changes" button is no longer dead.** The Quick Settings
+  panel called `saveConfigForm()` and `reloadConfig()` — neither function
+  was defined anywhere, so the button silently no-op'd and the panel
+  rendered "Loading schema..." forever. Replaced with a real form-based
+  editor whose Apply button PUTs a nested payload to `/api/config` and
+  reloads from server on success.
+
+### Added
+
+- **Form-based config editor with dropdowns.** `/config` now renders a
+  grouped settings form alongside the YAML editor. Known enum fields
+  (`backend`, `embedding.provider`, `llm.provider`, `llm.local_backend`,
+  `logging.level`, `synthesis.default_format`, `governance.pii.action`)
+  render as `<select>` controls instead of free-text inputs. Restart-
+  required leaves get a "restart required" badge sourced from the same
+  set the server uses, so the UI warning is never out of sync with the
+  server's classification.
+- **Pending-changes counter and Revert button.** The form tracks dirty
+  fields by dotted path, builds a single nested payload on Apply, and
+  shows `N pending change(s) (M need restart)` next to the buttons.
+- **YAML editor accepts both YAML and JSON** (was JSON-only despite the
+  label) and skips redacted `***` secrets so they aren't PUT back as
+  literal strings.
+
+### Tests
+
+- 4 new tests in `tests/test_web_api.py`: dropdown enum round-trip for
+  `logging.level` (restart-required) and `synthesis.default_format` (live);
+  multi-section nested payload from a single Apply; and a regression guard
+  on the `/config` HTML structure (form tab, dropdown enum declarations,
+  restart-leaf flags, and proof the dead `saveConfigForm`/`reloadConfig`
+  handlers are gone). 28 passed, 2 skipped (was 24 + 2).
+
+## [2.6.1] - 2026-04-25
+
+Hotfix release. Resolves three blockers found in code review of the
+RFC-015 web GUI shipped in v2.6.0. No data migration. No config changes.
+
+### Fixed
+
+- **`/config` HTML page now renders.** `_to_dict` was defined as a closure
+  inside `get_config_endpoint`, so every render of `/config` raised
+  `NameError`, was silently swallowed by a bare `except`, and left the
+  YAML body blank on initial server-side render. Promoted to a module-level
+  `_config_to_dict` helper used by both routes. (PR #131)
+- **`PUT /api/config` correctly reports nested restart-required fields.**
+  The check compared top-level payload keys against a set of dotted-path
+  fields, so payloads like `{"embedding": {"provider": "x"}}` were
+  reported as `applied: ["embedding"]`, `pending_restart: []`, telling
+  operators a restart-required change had taken effect when it had not.
+  Added `_flatten_keys` to walk nested payloads to dotted leaf paths;
+  `applied` and `pending_restart` now contain accurate dotted paths.
+  (PR #131)
+- **`/config` HTML route is now auth-gated.** `/api/config` was protected,
+  but the HTML shell (and once the `_to_dict` bug was fixed, its
+  server-rendered YAML body) was reachable without an API key. Added
+  `Depends(require_api_guard)` and made the YAML body redact secrets
+  before serialization. (PR #131)
+
+### Tests
+
+- Added four regression tests in `tests/test_web_api.py` covering all
+  three fixes. 24 passed, 2 skipped (was 20 + 2).
+
+## [2.6.0] - 2026-04-25
+
+Feature release. Adds configurable content-size limits for DoS mitigation
+(RFC-014) and moves per-call-site LLM token budgets out of hardcoded
+literals into `LLMConfig`, making them overridable at deploy time.
+
+### Added
+
+- **Configurable content size limits** (`GovernanceConfig.limits.max_content_length`,
+  default 50 MB). `remember()` calls with content exceeding the limit are
+  rejected with a clear error message. Set to `0` to disable the check.
+  Environment override: `ZETTELFORGE_LIMITS_MAX_CONTENT_LENGTH`.
+  (RFC-014, PR #123)
+- **Per-call-site `max_tokens` budgets configurable via `LLMConfig`**.
+  Five new fields: `max_tokens_causal` (8000), `max_tokens_synthesis` (2500),
+  `max_tokens_fact` (2500), `max_tokens_ner` (2500), `max_tokens_evolution` (2500).
+  Defaults match v2.5.2 values. No behavioral change for existing configs.
+  (PR #126, issue #125)
+
+### Changed
+
+- **Docs: config reconciliation for v2.5.2** — `config.default.yaml` gained
+  the `lance:` section (RFC-009 Phase 1.5), `docs/reference/configuration.md`
+  now covers all v2.5.2 knobs (`lance`, `pii`, per-call-site budgets), and
+  `docs/explanation/llm-budgets-and-timeouts.md` explains the reasoning-model
+  token-budget tradeoffs. (PR #126)
+- **`_apply_yaml()` now handles `lance:` section** — previously the
+  `lance.cleanup_interval_minutes` and `lance.cleanup_older_than_seconds` YAML
+  knobs were silently ignored (regression from RFC-009 Phase 1.5 landing in
+  v2.4.x without the `_apply_yaml` branch). (PR #126, code review finding)
+
+### Internal
+
+- **Removed `docs/superpowers/` from version control**. The directory holds
+  workspace scratch — internal research and notes with unredacted incident
+  detail. 18 files (7319 lines) untracked. No published content lost.
+  (PR #128)
+
 ## [2.5.2] - 2026-04-25
 
 Hotfix release. Restores end-to-end functionality of synthesis, causal

{zettelforge-2.5.2 → zettelforge-2.6.2}/CONTRIBUTING.md

@@ -37,6 +37,49 @@ If your contribution needs TypeDB or OpenCTI, open an issue to discuss.
 We keep the extension boundary clear so contributors know their work
 will always remain open source.
 
+### For major features: Start with an RFC
+
+If you're proposing a significant new feature (new subsystem, new backend,
+breaking API change), open an RFC before writing code. RFCs live in
+`docs/rfcs/` and follow the template from the existing RFCs in that
+directory. Open a Discussion first to socialize the idea, then file a
+draft RFC as a PR. This prevents wasted effort on work that won't be
+accepted.
+
+See [ROADMAP.md](ROADMAP.md) for the current priorities and what's
+planned for upcoming releases.
+
+### Good first issues
+
+Issues tagged [good first issue](https://github.com/rolandpg/zettelforge/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
+have structured acceptance criteria in the issue body. Check the issue
+for: which files to edit, test expectations, and example input/output.
+If an issue is unclear, ask in the issue comments.
+
+## Issue triage
+
+This project is maintained by a solo developer (per GOV-006). Here is
+what you can expect:
+
+- **New issues**: triaged within 7 days. You will get a response (even
+  if it's "not planned, closing").
+- **Bug reports**: severity assessed within 7 days. P0 (crash, data
+  loss, security) gets a same-day response.
+- **Feature requests**: tagged with `enhancement` on creation. The
+  maintainer will add `planned`, `deferred`, or `won't fix` within 7
+  days.
+- **PR reviews**: first review within 14 days of submission. Smaller
+  PRs get reviewed faster.
+- **Stale issues**: issues with no activity for 60 days are tagged
+  `stale` and closed after 14 more days without response. This keeps
+  the tracker manageable for a solo maintainer.
+
+## Contributor recognition
+
+Every contributor is listed in [CONTRIBUTORS.md](CONTRIBUTORS.md),
+regardless of contribution size. If you submit a PR that gets merged,
+you will be added. If your name is missing, open a PR.
+
 ## Code Style
 
 - Follow PEP 8

zettelforge-2.6.2/CONTRIBUTORS.md

@@ -0,0 +1,17 @@
+# Contributors
+
+Thank you to everyone who has contributed to ZettelForge.
+
+## Maintainers
+
+- **Patrick G. Roland II** — creator and lead maintainer
+
+## Contributors
+
+This file tracks individuals who have contributed code, documentation, design, or other improvements. The maintainer updates it with each release.
+
+If you have contributed and your name is missing, please open a PR or issue.
+
+---
+
+*This project uses a "thank you, next" acknowledgment model. All contributors are listed regardless of contribution size. A typo fix counts.*

{zettelforge-2.5.2 → zettelforge-2.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zettelforge
-Version: 2.5.2
+Version: 2.6.2
 Summary: ZettelForge: Agentic Memory System with vector search, knowledge graph, and synthesis
 Project-URL: Homepage, https://github.com/rolandpg/zettelforge
 Project-URL: Documentation, https://docs.threatrecall.ai
@@ -36,12 +36,16 @@ Requires-Dist: requests>=2.31.0
 Requires-Dist: structlog>=24.0.0
 Requires-Dist: tantivy>=0.11.0
 Provides-Extra: dev
+Requires-Dist: fastapi>=0.100.0; extra == 'dev'
+Requires-Dist: jinja2>=3.0.0; extra == 'dev'
 Requires-Dist: langchain-core>=0.2.0; extra == 'dev'
 Requires-Dist: mypy>=1.0.0; extra == 'dev'
+Requires-Dist: psutil>=5.9.0; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
 Requires-Dist: pytest>=7.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.4.0; extra == 'dev'
+Requires-Dist: uvicorn>=0.20.0; extra == 'dev'
 Provides-Extra: extensions
 Requires-Dist: zettelforge-enterprise>=2.1.0; extra == 'extensions'
 Provides-Extra: langchain
@@ -61,6 +65,8 @@ Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'pii'
 Requires-Dist: spacy>=3.5.0; extra == 'pii'
 Provides-Extra: web
 Requires-Dist: fastapi>=0.100.0; extra == 'web'
+Requires-Dist: jinja2>=3.0.0; extra == 'web'
+Requires-Dist: psutil>=5.9.0; extra == 'web'
 Requires-Dist: uvicorn>=0.20.0; extra == 'web'
 Description-Content-Type: text/markdown
 
@@ -79,8 +85,9 @@ It extracts CVEs, threat actors, IOCs, and ATT&CK techniques from analyst notes
 [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/rolandpg/zettelforge/actions/workflows/ci.yml/badge.svg)](https://github.com/rolandpg/zettelforge/actions)
+[![Open Issues](https://img.shields.io/github/issues/rolandpg/zettelforge?color=blue)](https://github.com/rolandpg/zettelforge/issues)
 
-**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai)**
+**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai) · [🗺️ Roadmap](ROADMAP.md)**
 
 <p align="center">
 <a href="https://www.buymeacoffee.com/xypher22pr0" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-green.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;" ></a>

{zettelforge-2.5.2 → zettelforge-2.6.2}/README.md

@@ -13,8 +13,9 @@ It extracts CVEs, threat actors, IOCs, and ATT&CK techniques from analyst notes
 [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/rolandpg/zettelforge/actions/workflows/ci.yml/badge.svg)](https://github.com/rolandpg/zettelforge/actions)
+[![Open Issues](https://img.shields.io/github/issues/rolandpg/zettelforge?color=blue)](https://github.com/rolandpg/zettelforge/issues)
 
-**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai)**
+**[⭐ Star](https://github.com/rolandpg/zettelforge) · [📦 `pip install zettelforge`](https://pypi.org/project/zettelforge/) · [📖 Docs](https://docs.threatrecall.ai/) · [🧪 Hosted beta](https://threatrecall.ai) · [🗺️ Roadmap](ROADMAP.md)**
 
 <p align="center">
 <a href="https://www.buymeacoffee.com/xypher22pr0" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-green.png" alt="Buy Me a Coffee" style="height: 60px !important;width: 217px !important;" ></a>

zettelforge-2.6.2/ROADMAP.md

@@ -0,0 +1,94 @@
+# ZettelForge Roadmap
+
+Last updated: 2026-04-25
+
+This document communicates what the maintainer is building, what is on hold, and what is out of scope. It is updated when priorities shift.
+
+---
+
+## Current release: v2.6.1
+
+Shipped: 2026-04-25. See `CHANGELOG.md` for details.
+
+The v2.6 series (RFC-013 through RFC-015) delivered PII detection (Presidio), configurable content size limits, and the Web Management Interface. The v2.5 series (RFC-009 through RFC-012) delivered the enrichment pipeline v2, local LLM backends, and unified provider config.
+
+---
+
+## v2.7.0 targets (next release)
+
+Target: 2026-05-09. Scope is frozen at the items below. Everything else defers to v2.8.0 unless marked **P0**.
+
+### Must ship (P0)
+
+- [ ] **Issue #125: Harden reasoning-model LLM budget plumbing.** Regression tests for `max_tokens` at each call site, config-overridable budgets, `thinking` tag stripping in `json_parse.py`, and a `reasoning_model: bool` auto-scaling flag. Post-#124 follow-up. (Est: 2-3 days)
+- [ ] **Issue #73: Tighten CCCS metadata regexes (SEC-6 / SEC-7).** Low hanging security hardening. (Est: 0.5 day)
+- [ ] **Issue #72: MemoryManager.remember(sync=True) dominates bulk ingest.** The YARA p95 plyara tail needs a timeout or chunked processing path. (Est: 1 day)
+
+### Nice to have (P1)
+
+- [ ] **Issue #71: Add typed DetectionMeta extension to MemoryNote.Metadata.** Paves the way for richer entity metadata downstream. (Est: 0.5 day)
+- [ ] **Issue #51: Ratchet governance coverage threshold from 67% toward 80%.** Incremental. (Est: 1 day)
+
+### Community items (open for contribution)
+
+See the [good first issue](https://github.com/rolandpg/zettelforge/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) label. These are pre-scoped with acceptance criteria:
+
+- #47 — IPv6 address extraction
+- #46 — YARA rule reference extraction
+- #45 — Sigma rule ID extraction
+- #39 — Threat actor alias mappings for Chinese APT groups
+- #36 — Architecture decision records (ADRs)
+- #44 — Example: MISP JSON feed ingestion
+- #43 — Example: Slack bot for CTI queries
+- #41 — Example: Jupyter notebook CTI analysis workflow
+
+---
+
+## Next release (v2.7.1+)
+
+Target: approximately 2026-05-23. Provisional scope; will be finalized after v2.7.0 ships.
+
+- [ ] **CrewAI tool wrapper** (#40). Integration path for CrewAI agents to use ZettelForge as a memory backend.
+- [ ] **OpenCTI sync overhaul**. Improving the bidirectional sync reliability from the initial implementation.
+- [ ] **Detection rules as first-class entities** (#feat/detection-rules-first-class branch). Sigma/YARA rules stored and searchable as knowledge graph nodes.
+
+---
+
+## Backlog / on hold
+
+These are tracked but not actively scheduled:
+
+- **MCP registry publish** (feat/mcp-registry-publish branch). Publishing the MCP server to the official MCP registry.
+- **Enterprise split**. Separating the current monolithic package into community + enterprise tiers. Governance, license boundary, and packaging work. Not blocked, but deferred until community adoption justifies the overhead.
+- **TypeDB read-path hardening**. Depth routing and schema versioning for the TypeDB backend. Most users run on JSONL/SQLite; TypeDB is a small fraction of the install base.
+- **Conversational entity extractor** (stash: feature/RFC-001-conversational-entity-extractor). Interactive refinement of extracted entities by analyst chat. Requires UX thinking and a frontend update.
+- **Pydantic v3 upgrade prep** (stash: test-fix/pydantic-v3-prep). Preparing internal models for Pydantic v3 migration. Low urgency; no upstream pressure yet.
+
+---
+
+## Out of scope (not building)
+
+These have been proposed or discussed and explicitly decided against:
+
+- **UI framework migration** (React, Vue, Svelte, etc.). The web GUI is a vanilla JS SPA using the ZettelForge Design System. No npm build step, no JS framework. This is intentional: the SPA must remain maintainable by a solo developer and installable with `pip install zettelforge[web]` without a separate build step. Not changing.
+- **Docker containerization**. Deferred to v2.x post-v1.0 per the tech stack decision. The in-process architecture already makes deployment trivial.
+- **Cloud-hosted memory backend**. ZettelForge is designed for local-first, air-gapped, and on-prem deployments. A cloud sync layer would compromise the security model and is not on the roadmap.
+- **LangChain / LangGraph integration as a default**. Out of scope per the CLAUDE.md rules in this repo. Community wraps are welcome (see the CrewAI issue for how integration should work).
+
+---
+
+## Release cadence
+
+- **Minor releases (v2.x.0)**: roughly every 2-3 weeks, bundling feature work and hardening.
+- **Patch releases (v2.x.y)**: as needed for P0 bugs, security fixes, and regressions. No pre-scheduled date.
+- **Major releases (v3.0.0+)**: not yet planned. The API is still evolving. A major version bump will come with a public deprecation notice and migration guide.
+
+## Commitment
+
+This roadmap is a best-effort forecast, not a contract. Priorities shift based on user feedback, security findings, and the maintainer's availability (solo maintainer, GOV-006 declared).
+
+Issues tagged with a target release are actively planned. Issues without a release tag are candidates for the next roadmap refresh.
+
+---
+
+[Back to README](README.md)

{zettelforge-2.5.2 → zettelforge-2.6.2}/config.default.yaml

@@ -374,6 +374,8 @@ synthesis:
 # Env overrides:
 #   ZETTELFORGE_PII_ENABLED=true
 #   ZETTELFORGE_PII_ACTION=redact
+#   ZETTELFORGE_LIMITS_MAX_CONTENT_LENGTH=104857600
+#   ZETTELFORGE_LIMITS_RECALL_TIMEOUT=60
 #
 governance:
   enabled: true
@@ -385,9 +387,32 @@ governance:
     entities: []
     language: en
     nlp_model: en_core_web_sm
+  limits:
+    max_content_length: 52428800  # 50 MB, 0 = unlimited
+    recall_timeout_seconds: 30.0  # seconds, 0 = unlimited
 
 
-# ── Cache ───────────────────────────────────────────────────────────────────
+# ── LanceDB Maintenance (RFC-009 Phase 1.5) ─────────────────────────────────
+# Background daemon that prunes stale LanceDB version-history per shard.
+#
+# Examples:
+#   # Default — clean up every hour, keep versions < 1 hour old
+#   cleanup_interval_minutes: 60
+#   cleanup_older_than_seconds: 3600
+#
+#   # Aggressive cleanup (high-ingestion environments)
+#   cleanup_interval_minutes: 15
+#   cleanup_older_than_seconds: 600
+#
+#   # Disable cleanup entirely
+#   cleanup_interval_minutes: 0
+#
+lance:
+  cleanup_interval_minutes: 60
+  cleanup_older_than_seconds: 3600
+
+
+# ── Cache
 # In-memory cache for TypeDB query results. Reduces round-trips for
 # frequently accessed entities and relationships.
 #
@@ -434,3 +459,32 @@ logging:
   level: INFO
   log_intents: true
   log_causal: true
+
+
+# ── Web UI (RFC-015) ─────────────────────────────────────────────────────────
+# ZettelForge Web Management Interface — SPA served at GET /.
+#
+# Examples:
+#   # Default — enabled on 0.0.0.0:8088
+#   enabled: true
+#   host: 0.0.0.0
+#   port: 8088
+#
+#   # Custom port (e.g., behind nginx reverse proxy)
+#   enabled: true
+#   host: 127.0.0.1
+#   port: 9000
+#
+#   # Disable web UI entirely (library-only mode)
+#   enabled: false
+#
+# Env overrides:
+#   ZETTELFORGE_WEB_ENABLED=true
+#   ZETTELFORGE_WEB_PORT=8088
+#   ZETTELFORGE_WEB_UI_DIR=/path/to/ui
+#
+web:
+  enabled: true
+  host: 0.0.0.0
+  port: 8088
+

{zettelforge-2.5.2 → zettelforge-2.6.2}/docs/THREAT_MODEL.md

@@ -137,9 +137,9 @@ TB-1 ─────────────────────────
 
 | ID | Threat | Component | Risk | Mitigation |
 |----|--------|-----------|------|------------|
-| D-01 | Large content in `remember()` exhausts memory or blocks the enrichment queue | MemoryManager (P1) | **Medium** — degraded performance | `remember_report()` chunks long documents. No explicit size limit on `remember()` content. Enrichment queue has `maxsize=500` backpressure. |
+| D-01 | Large content in `remember()` exhausts memory or blocks the enrichment queue | MemoryManager (P1) | **Low** — gracefully rejected | `governance.limits.max_content_length` (RFC-014, default 50 MB) blocks oversized content with a clear error. `remember_report()` chunks long documents. Enrichment queue has `maxsize=500` backpressure. |
 | D-02 | LLM provider (ollama, litellm) hangs and blocks `remember()` | LLM Provider (TB-4) | **High** — operation blocks | OllamaProvider has timeout (RFC-010, default 60s). LitellmProvider has timeout + num_retries. `generate()` returns empty string on recoverable failure. Fallback provider (e.g., local -> ollama) gives alternative path. |
-| D-03 | Malicious query triggers deep graph traversal exhausting time/resources | BlendedRetriever | **Medium** — slow recall | `max_graph_depth` config (default 2) limits BFS hops. `default_k` (default 10) limits results. No timeout on recall queries. |
+| D-03 | Malicious query triggers deep graph traversal exhausting time/resources | BlendedRetriever | **Medium** — bounded, but timeout may still block | `governance.limits.recall_timeout_seconds` (RFC-014, default 30s) applies a wall-clock timeout to the recall pipeline, but the current `ThreadPoolExecutor`-based approach must not be treated as guaranteeing prompt return on timeout. `max_graph_depth` (default 2) limits BFS hops. `default_k` (default 10) limits results. Reclassify to **Low** only after the timeout path is verified to return promptly and log `recall_timed_out` without waiting for the running task to finish. |
 | D-04 | spaCy model download blocks first `remember()` when PII is enabled | PIIValidator (lazy load) | **Low** — delayed first call (~2-3 seconds) | One-time download cost. Matching fastembed pattern. Can be pre-downloaded for air-gapped deployments. |
 
 ### 2.6 Elevation of Privilege
@@ -158,8 +158,8 @@ TB-1 ─────────────────────────
 |------------|-------|--------------|
 | **Critical** | 2 | T-01 (storage tampering), I-01 (unencrypted data at rest), E-02 (governance bypass via filesystem) |
 | **High** | 7 | S-01 (spoofed MCP client), S-03 (config tampering), T-02 (config security downgrade), R-01 (repudiation without audit), I-02 (PII in stored notes), D-02 (LLM provider hang), E-01 (cross-tenant data access) |
-| **Medium** | 9 | S-02 (fake LLM provider), T-04 (retrieval poisoning), R-02, R-03, I-04 (error message leakage), D-01, D-03, E-03 |
-| **Low** | 1 | D-04 (PII model download delay) |
+| **Medium** | 7 | S-02 (fake LLM provider), T-04 (retrieval poisoning), R-02, R-03, I-04 (error message leakage), E-03 |
+| **Low** | 3 | D-01, D-03, D-04 (PII model download delay) |
 
 ### Top 5 Mitigations (Priority Order)
 
@@ -181,6 +181,8 @@ TB-1 ─────────────────────────
 | API key redaction | I-03 | `LLMConfig.__repr__` redacts api_key and sensitive extra keys | Unit tests in `test_llm_providers.py` |
 | PII detection + redaction | I-02 | PIIValidator (RFC-013): log/redact/block | Unit tests in `test_pii_validator.py` |
 | LLM provider timeout | D-02 | `OllamaProvider` timeout=60s, `LiteLLMProvider` timeout + num_retries | Unit tests (RFC-010, RFC-012) |
+| Content size limit | D-01 | `governance.limits.max_content_length` (RFC-014, default 50 MB) blocks oversized content | Unit tests in `test_governance.py` |
+| Recall timeout | D-03 | `governance.limits.recall_timeout_seconds` (RFC-014, default 30s) wraps recall in ThreadPoolExecutor with wall-clock timeout | Unit tests in `test_governance.py` |
 | Config env-var resolution | I-03 | `${ENV_VAR}` syntax prevents raw secrets in YAML | Unit tests |
 | Configurable model provider | S-02, E-03 | `provider` key selects backend; no implicit unauthenticated outbound calls | Config validation |
 | Enrichment queue backpressure | D-01 | `maxsize=500` bounded queue | Code review |
@@ -189,7 +191,6 @@ TB-1 ─────────────────────────
 
 | Recommendation | Threat(s) | Effort | Priority |
 |---------------|-----------|--------|----------|
-| Add content size limit to `remember()` | D-01 | Small | P3 |
 | Add global exception handler that sanitizes error output | I-04 | Medium | P2 |
 | Add TLS verification option for self-hosted LLM endpoints | S-02 | Small | P2 |
 | Add config file integrity check (SHA-256 of default vs. loaded) | T-02, S-03 | Medium | P3 |
@@ -231,6 +232,7 @@ Per GOV-021, the following data types exist in the system:
 
 | Change | RFC/PR | Date | Threat Model Impact |
 |--------|--------|------|---------------------|
+| Content size limits + recall timeout | RFC-014 | 2026-04-25 | Mitigation for D-01 (content size limit, default 50 MB); partial mitigation for D-03 (timeout) |
 | PII detection and redaction | RFC-013 (PR #118) | 2026-04-25 | New control for I-02; new attack surface (D-04); PII text logging fixed |
 | LiteLLM unified provider | RFC-012 (PR #108) | 2026-04-25 | New provider for I-03 (API keys); new outbound traffic pattern (TB-4) |
 | Local LLM backend selection | RFC-011 (PR #104) | 2026-04-25 | No new threat surface — extends existing local provider |

zettelforge-2.6.2/docs/explanation/llm-budgets-and-timeouts.md

@@ -0,0 +1,101 @@
+---
+title: "LLM budgets, timeouts, and what they cost you"
+description: "How ZettelForge's LLM token budgets and HTTP timeouts trade latency for end-to-end correctness — what each knob controls, what it costs, and when to override."
+diataxis_type: explanation
+audience: "Operator / Developer"
+tags: [llm, configuration, performance, reasoning-models, ollama]
+last_updated: "2026-04-25"
+version: "2.6.0"
+---
+
+# LLM budgets, timeouts, and what they cost you
+
+ZettelForge makes five distinct kinds of LLM calls (causal triple extraction, synthesis, fact extraction, conversational NER, neighbor evolution). Each has a hardcoded `max_tokens` budget and shares a single configurable `llm.timeout`. The defaults trade ingest latency for end-to-end correctness on a reference reasoning model (`qwen3.5:9b`, Q4_K_M). This page explains why the defaults look the way they do and when you should override them.
+
+If you just want the table of values, see the [Configuration Reference §Per-call-site `max_tokens` budgets](../reference/configuration.md#per-call-site-max_tokens-budgets-hardcoded-v252). This page is the *why*.
+
+## The hidden-thinking-token problem
+
+Modern reasoning models — qwen3.5+, qwen3.6, nemotron-3, deepseek-r1, gemini-thinking — generate two streams of tokens for any prompt:
+
+1. **Reasoning tokens.** Wrapped in `<think>...</think>`, these are the model's internal scratch work. Ollama hides them from the `response` field by default but **they still count against `num_predict`**.
+2. **Answer tokens.** What the model actually emits as the final user-visible output. These appear in `response`.
+
+If `num_predict` is 300 tokens and the model uses 280 of them reasoning, you get 20 tokens of answer — usually not enough for valid JSON. If it uses all 300, you get an empty string and Ollama returns `done_reason: "length" eval_count: 300 response: ""`. The pre-2.5.2 budgets (300/400/800/1024) were sized for non-reasoning models and silently failed every call on the reasoning model that ZettelForge defaults to. v2.5.2 raised the per-call-site caps to give reasoning room *and* answer room on the same generation.
+
+## Per-call-site budgets — and why each one is what it is
+
+### Causal triple extraction (`note_constructor.py`, **8000 tokens**)
+
+The largest budget anywhere in the codebase. The prompt asks the model to enumerate *every* causal relation in a passage of up to 2000 characters, validating each relation against an allowlist. Empirical: `qwen3.5:9b` at 4000 tokens succeeded only ~70% of the time (eval_count varied 2.8k–4k+, with the longer reasoning chains hitting the budget cap). 8000 keeps the success rate above 95% on the same model. Wall-clock cost: 60–140 s per call.
+
+### Synthesis (`synthesis_generator.py`, **2500 tokens**)
+
+Single-answer prompts converge faster than enumerate-everything prompts. 2500 covers reasoning + a paragraph of JSON answer. Wall-clock: 20–50 s per query.
+
+### Fact extraction (`fact_extractor.py`, **2500 tokens**)
+
+Similar profile to synthesis — bounded JSON output. The pre-2.5.2 cap was 400, which left this silently no-opping on every reasoning-model call.
+
+### Conversational NER (`entity_indexer.py`, **2500 tokens**)
+
+The regex fast-path covers CTI types (CVE, ATT&CK, IOCs); LLM NER fills in `person`, `location`, `organization`, `event`, `activity`, `temporal`. Output is a small JSON object so 2500 is generous. The retry path uses the same budget.
+
+### Neighbor evolution (`memory_evolver.py`, **2500 tokens** × 2)
+
+Two-note comparison + ADD/UPDATE/DELETE/NOOP decision. Both the first call and the parse-retry call use 2500. Parse-retry exists because reasoning models occasionally emit prose preamble before the JSON; the second call reasserts JSON-only and usually gets it.
+
+## The shared timeout
+
+`llm.timeout` (default **180 s** in v2.5.2; was 60 s pre-fix) governs the HTTP read deadline on every Ollama call.
+
+The 60 s default fired before causal extraction at 8000 tokens could complete on a 9B model. The fix had to come at both ends: bigger budget *and* longer timeout. If you lower one, lower the other together — bumping the budget without bumping the timeout just trades empty-response failures for ReadTimeout failures.
+
+## When to override
+
+The defaults are calibrated for `qwen3.5:9b` Q4_K_M on a single GPU. You want to override if:
+
+### You're on faster hardware (H100, multi-GPU, large batch)
+
+You can lower `llm.timeout` to e.g. 60 s without losing correctness, since each generation completes faster. The budgets themselves are about *how much room the model needs to think*, not how fast it thinks — leave them alone unless you've measured.
+
+### You're on a non-reasoning model (gemma4, llama-3.x base, qwen2.5)
+
+These don't emit ` thinking` tokens. Your budgets only need to cover the actual answer length, not reasoning + answer. You can set `max_tokens` values to ~25% of the v2.5.2 defaults in config (v2.6.0 moved them to `LLMConfig` — see [issue #125](https://github.com/rolandpg/zettelforge/issues/125)). `llm.timeout: 60` is then enough.
+
+### You're on a much larger model (70B, 120B, cloud)
+
+Reasoning depth often *increases* with model size. The 8000-token causal cap may not be enough on a 120B reasoning model. Watch the OCSF log for `event=llm_call_empty_response done_reason=length eval_count=8000` — if you see it, raise the `max_tokens=8000` literal inside `NoteConstructor.extract_causal_triples` (in `src/zettelforge/note_constructor.py`) and re-test.
+
+### You're triggering `sync=True` or doing bulk ingestion
+
+The default async path moves causal extraction off the write hot path; `remember()` returns in ~50 ms while extraction happens later in the enrichment worker. **`sync=True` blocks the caller until the worker finishes.** With v2.5.2 budgets on a 9B reasoning model that's 1–3 minutes per note. For bulk ingestion (1000+ notes), prefer async and let the queue drain at its own pace; for sync use cases (test fixtures, small one-shots), accept the latency or downgrade to a non-reasoning model.
+
+## Verifying your budgets are right
+
+The OCSF log at `~/.amem/logs/zettelforge.log` carries every LLM call as a structured event. Two events to know:
+
+- **`llm_call_empty_response`** — `WARNING` level, fires whenever an Ollama call returns an empty `response`. Always visible at the default `INFO` log level.
+- **`llm_call_complete`** — `DEBUG` level, fires on every successful call with `eval_count`, `response_chars`, `max_tokens`, `duration_ms`, etc. Only visible when `logging.level: DEBUG` is set in `config.yaml` (or via `ZETTELFORGE_LOG_LEVEL=DEBUG`).
+
+To spot too-small budgets at the default log level:
+
+```bash
+grep '"event":"llm_call_empty_response"' ~/.amem/logs/zettelforge.log \
+  | jq -r '"\(.model) eval=\(.eval_count) of max=\(.max_tokens) dur_ms=\(.duration_ms)"' \
+  | tail -20
+```
+
+`eval == max_tokens` with `done_reason: length` is the canonical token-starvation signature — raise the budget for that call site.
+
+To verify budgets aren't *too* generous (free wall-clock to claw back), enable DEBUG logging and grep `llm_call_complete` instead. `eval_count << max_tokens` with non-empty `response_chars` means you could safely lower the cap on faster hardware.
+
+## Background
+
+- v2.5.2 hotfix CHANGELOG entry — full root-cause writeup and per-file diffs.
+- [Issue #125](https://github.com/rolandpg/zettelforge/issues/125) — v2.6.0 plan to make these budgets config-overridable per call site, add `<think>`-tag stripping as a post-processing guard, and a `reasoning_model: bool` auto-scale flag.
+
+## Related
+
+- [Configuration Reference](../reference/configuration.md)
+- [Troubleshoot ZettelForge](../how-to/troubleshoot.md)