unread 0.1.0__tar.gz

unread-0.1.0/.claude/skills/unread-dev/SKILL.md

@@ -0,0 +1,418 @@
+---
+name: unread-dev
+description: |
+  Contribute code to the `unread` Python package (github.com/maxbolgarin/unread):
+  the local Python CLI that pulls Telegram chats, transcribes voice, analyzes
+  with OpenAI, answers questions over the synced corpus (`unread ask`), and
+  manages the local DB (backup/restore, doctor, prune). Use this skill when
+  editing files under `unread/`, adding a new CLI command or flag, writing
+  / modifying a preset, changing the cache or pipeline, fixing a bug, or
+  writing tests in that repo — or when the user says "in unread",
+  "the Telegram analyzer repo", or references `unread/cli.py`,
+  `analyzer/pipeline.py`, `db/repo.py`, `ask/`, or `interactive.py`.
+---
+
+# unread — contributor guide
+
+Python 3.11+, uv, Typer, Telethon, OpenAI, aiosqlite. **Every OpenAI call is
+async; every DB call is async. No synchronous I/O in command paths.**
+
+## 1. First check before writing code
+
+```bash
+cd ~/path/to/unread          # config + storage are cwd-relative
+uv sync --extra dev              # if not already done
+uv run pytest -q                 # baseline: should be all green
+uv run ruff check .              # lint clean
+uv run ruff format --check .     # format clean
+unread doctor                       # validates Telegram session, OpenAI key, ffmpeg, DB integrity, pricing
+```
+
+Before claiming work is done, run the first three — that's what CI runs on Python 3.11, 3.12, and 3.13.
+
+## 2. Layout map
+
+```
+unread/
+  cli.py                  Typer app — every CLI command + help panels.
+                          Wraps `_run = asyncio.run`. Lazy imports inside handlers
+                          keep `unread --help` ~100 ms.
+  interactive.py          Wizard pickers (questionary + prompt_toolkit).
+                          Step graph: chat → thread → preset → period → enrich → output → mark_read → confirm.
+                          `_expand_printable_for_search()` extends `string.printable`
+                          at module import so non-Latin type-to-filter works.
+  config.py               Settings loader (.env + config.toml → Pydantic with
+                          extra="forbid"). `_StrictCfg` base for every block.
+                          BOM-safe `_load_dotenv`. New: `AskCfg`, `high_impact_reactions`.
+  models.py               Dataclasses: Message, Chat, Subscription, ResolvedRef…
+  core/
+    pipeline.py           prepare_chat_run() — shared prefix for analyze/dump/download-media
+                          (resolve → backfill → iter → per-topic unread filter → enrich).
+    run.py                PreparedRun dataclass (handoff between prefix + consumer).
+    paths.py              slugify, chat_slug, topic_slug, compute_window (UTC),
+                          parse_ymd (UTC), derive_internal_id, has_explicit_period.
+  db/
+    repo.py               Single Repo class. All DB access goes through it.
+    schema.sql            Source of truth; idempotently applied on every Repo.open.
+                          `Repo._apply_additive_migrations()` adds known missing columns
+                          for older local DBs; destructive changes still require
+                          "delete data.sqlite + re-sync".
+  tg/
+    client.py             Telethon client factory (session at storage/session.sqlite.session).
+    resolver.py           @name / link / fuzzy / numeric id → ResolvedRef.
+    sync.py               backfill() — incremental fetch + Rich progress bar
+                          (determinate when from_msg_id+forward, indeterminate otherwise).
+    dialogs.py            list_unread_dialogs(); correct_forum_unread() helper —
+                          fixes Telegram's stale dialog-level forum counts via
+                          GetForumTopicsRequest.
+    topics.py folders.py  Forum topics + dialog-filter (chat folder) helpers.
+                          `chat_folder_index(client) -> dict[chat_id, list[folder_titles]]`.
+    links.py              Telegram link parsing / template building.
+    commands.py           `unread chats add/list/...` handlers + `cmd_doctor`.
+                          `cmd_doctor` checks: env files, secrets, ffmpeg, disk,
+                          DB integrity, Telegram session, OpenAI reachability,
+                          presets, chat AND audio pricing coverage.
+  analyzer/
+    prompts.py            Preset loader + _base.md composer; BASE_VERSION cache-bust knob.
+                          name == filename stem enforced; placeholder dry-run validation.
+    filters.py            FilterOpts(min_msg_chars, include_transcripts, text_only,
+                          sender_substring, sender_id). filter_messages applies all.
+    chunker.py            Token-aware chunking. Raises on degenerate budgets
+                          (< 2000 tokens) instead of silent clamp-to-500.
+    formatter.py          Render messages for the LLM prompt. `_high_impact_marker`
+                          prefixes [high-impact] for messages whose reaction sum
+                          ≥ settings.analyze.high_impact_reactions.
+    hasher.py             batch_hash / reduce_hash / text_hash — cache keys.
+    openai_client.py      chat_complete() with retry-on-truncation. Uses
+                          max_completion_tokens (NOT max_tokens — deprecated).
+    pipeline.py           run_analysis() — map-reduce orchestration with progress bar.
+                          Public `estimate_cost(n_messages, preset, settings)` used by
+                          --max-cost / --dry-run / wizard cost preview.
+                          `AVG_TOKENS_PER_MSG = 60` (Cyrillic-heavy estimate).
+    commands.py           `unread analyze` + batch flow + output rendering.
+                          Helpers: `_expand_citations`, `_self_check`, `_post_to_chat`,
+                          `_post_to_saved_messages` shim, `_split_for_telegram`,
+                          `_CITATION_RE`.
+  ask/                    `unread ask` — Q&A across the synced corpus.
+    commands.py             cmd_ask + _run_single_turn (one Q→A iteration);
+                            handles the post-answer follow-up loop / --semantic /
+                            --rerank / --build-index / --refresh / --show-retrieved /
+                            --max-cost.
+    retrieval.py            Keyword LIKE retrieval over text||transcript;
+                            return_scores=True returns list[(Message, int)].
+    rerank.py               Cheap-model 1-5 ratings; tolerant `_parse_ratings`.
+                            On total batch failure, returns keyword-sorted pool.
+    embeddings.py           OpenAI text-embedding-3-small index (build_index +
+                            semantic_search). Vectors stored as
+                            array.array('f').tobytes() in message_embeddings.
+                            Cosine in pure Python (stdlib `math`, no numpy).
+  enrich/                 Per-kind enrichers run BEFORE filtering:
+    pipeline.py             enrich_messages() orchestrator + Rich Progress bar +
+                            per-doc_id asyncio.Lock to dedup parallel work.
+    audio.py                voice/videonote/video → Whisper transcript (media_enrichments).
+                            Logs include chat_id / msg_id / msg_date.
+    image.py                photo → vision-model description.
+    document.py             pdf/docx/txt/code → local extraction (no OpenAI).
+    link.py                 URL fetch + filter_model summary (link_enrichments).
+                            Per-message inline summary, NOT full-article analysis —
+                            see `unread/website/` for that.
+    video.py                Thin wrapper over audio.py.
+    base.py                 EnrichOpts / EnrichResult / EnrichStats.
+  youtube/                Non-Telegram source: `unread analyze <youtube-url>`.
+    urls.py                 is_youtube_url + extract_video_id (cheap stdlib check).
+    metadata.py             YoutubeMetadata dataclass + yt-dlp metadata fetch.
+    transcript.py           captions (VTT) → text + timed cues, OR audio download
+                            + Whisper segmentation. TranscriptResult shape.
+    paths.py                reports/youtube/<channel>/<video>-<preset>-<ts>.md.
+    commands.py             cmd_analyze_youtube — synth-message build, dispatch to
+                            run_analysis with source_kind="video", link_template
+                            "?t={msg_id}s" so citations jump to the moment.
+                            Cache row in `youtube_videos` (metadata + transcript +
+                            timed_cues_json).
+  website/                Non-Telegram source: `unread analyze <website-url>`.
+    urls.py                 is_website_url + is_telegram_url (so t.me/... still
+                            flows to the resolver) + normalize_url (drops tracking
+                            params) + page_id (sha256(normalized_url)[:16]) +
+                            domain_of.
+    metadata.py             WebsiteMetadata dataclass.
+    content.py              fetch_page (httpx) + trafilatura extractor with BS4
+                            fallback (semantic-tag pass, then whole-body get_text).
+                            _segment_paragraphs splits at blank lines, then
+                            sentences, then hard-cut. _SPA_MARKERS +
+                            _explain_empty_extraction produce a tailored error
+                            for JS-rendered SPAs vs short non-SPA vs large-but-empty.
+    paths.py                reports/website/<domain>/<page>-<preset>-<ts>.md.
+    commands.py             cmd_analyze_website — synth-message build, dispatch to
+                            run_analysis with source_kind="website", link_template
+                            = page URL (no fragment). Cache row in `website_pages`
+                            (metadata + paragraphs_json + content_hash).
+  media/                  Raw media download + `unread download-media` (deprecated;
+                          folded into `dump --save-media`).
+  export/                 `unread dump` (md/jsonl/csv).
+  util/                   flood, logging, pricing, tokens (tiktoken wrapper).
+presets/                  Per-language preset trees: presets/<lang>/*.md.
+  ru/                     RU bodies + _base.md + _reduce.md.
+  en/                     EN bodies + _base.md + _reduce.md.
+                          Bundled in each lang: summary, broad, digest, action_items,
+                          decisions, highlights, links, questions, quotes, reactions,
+                          single_msg, multichat. Active language picked by
+                          settings.locale.language (default "en").
+unread/i18n.py         User-visible strings keyed by language. `t(key, lang=None)`
+                          resolves from settings at call time. Add new strings here,
+                          NOT inline in module code.
+unread/settings/       `unread settings` command. DB-backed overrides for
+  commands.py             [locale] / [openai] config blocks. Persisted in the
+                          `app_settings` SQLite table; applied on every
+                          `open_repo()` via `db/repo.py:_apply_db_overrides`.
+                          Allow-list of overlay keys: `_OVERRIDE_KEYS`.
+tests/                    pytest-asyncio auto mode; all tests live here.
+                          `test_regressions_deep_review.py` is the "bugs locked in"
+                          file — every audit / bug-fix lands a regression test there.
+```
+
+## 3. Invariants — DO NOT VIOLATE
+
+These were debugged into existence. Breaking them causes silent data corruption or wasted spend.
+
+1. **Never cache truncated results.** `pipeline._call_cached` gates `cache_put` on `not res.truncated`. The cache row also has a `truncated` column, and `_call_cached` re-runs on a hit with `truncated=1`.
+
+2. **`build_messages` order is `system → static_context → dynamic`.** `openai_client.build_messages` enforces this. OpenAI's prompt cache only hits when the stable prefix leads.
+
+3. **Bump `prompt_version`** in a preset's frontmatter when its body changes. Without the bump, users get stale cached results.
+
+4. **Bump `BASE_VERSION`** (in `analyzer/prompts.py`, currently `"v4"`) when editing `presets/<lang>/_base.md` (any language) or the per-language `_FORUM_CONTEXT` constants or `compose_system_prompt`'s structural behavior. Busts `analysis_cache` for **every** preset at once.
+
+5. **`redact_old_messages` WHERE clause must include "IS NOT NULL".** Otherwise rowcount reports age-matching rows, not rows actually changed. Regression test in `tests/test_repo_cache.py`.
+
+6. **Use `max_completion_tokens`, not `max_tokens`.** Required on gpt-5+ and reasoning models. `openai_client._completion` already does this.
+
+7. **Unread-default anchor is `read_inbox_max_id`.** Batch flow (`_run_no_ref` in analyzer + dump) and `--folder` filter rely on `msg_id > read_inbox_max_id`.
+
+8. **Config + storage are cwd-relative.** `config.py` resolves `.env`, `config.toml`, and `storage/` from `Path.cwd()`. Don't hardcode install-dir paths.
+
+9. **New run-style commands go through `core/pipeline.prepare_chat_run`** *unless* the source isn't a Telegram chat (YouTube, websites, future RSS / eBook / etc.). For non-Telegram sources, dispatch BEFORE `prepare_chat_run` from `cmd_analyze`, build synthetic `Message` rows with `chat_id=0`, and call `run_analysis` directly with `source_kind` + `link_template_override`. Both `youtube/commands.py` and `website/commands.py` follow this pattern — copy that shape, don't re-implement the Telegram prefix.
+
+10. **`AnalysisOptions.options_payload` must include every option that affects analysis output.** Adding a new flag that changes the prompt or the message set without updating `options_payload` silently returns stale cached results. Recent additions: `sender_substring`, `sender_id`, `enrich_options` block, `youtube_video_id`, **`website_page_id` + `website_content_hash`**, `source_kind`. For mutable sources (websites — pages can be edited live), the **content hash** is what makes "fetch unchanged → cache hit, fetch changed → cache miss" work; pure URL keying would silently return stale analyses of edited articles.
+
+11. **`schema.sql` statements use `IF NOT EXISTS`.** Re-applying schema on every connect is the design — `tests/test_regressions_deep_review.py:test_schema_apply_is_idempotent` enforces it. Destructive changes (column type changes, drops) need explicit handling in `_apply_additive_migrations` plus a documented "delete `data.sqlite` + re-sync" path.
+
+12. **`unread ask` semantic scores map to `[0, 100]`.** Cosine `[-1, 1]` is mapped via `(s + 1) * 50` so rerank's "default 0 for unrated" sort works for negative similarities. Don't shortcut to `round(s * 100)`.
+
+13. **Citations regex `_CITATION_RE` matches `[^)]+` for the URL.** The LLM is instructed to emit Telegram URLs (no parens). Wikipedia-style URLs with parens would truncate; that's a documented limitation, not a regression.
+
+14. **Presets live under `presets/<language>/`.** The loader does not fall back across languages — each language is autonomous. Adding a new user-visible string → put English+native entries in `unread/i18n.py:_STRINGS`, never inline in module code. Both `language` and `content_language` must enter `AnalysisOptions.options_payload` and `Repo.put_last_run_args` payload.
+
+15. **`language` and `content_language` have non-overlapping responsibilities.** `language` = UI / saved-report headings only (wizard, `## Sources`, `## Verification`, truncation banner, report metadata). `content_language` = prompts / LLM input (which `presets/<lang>/` directory the loader reads, image+link enricher prompts, ask system prompt, formatter labels in the LLM input, verification audit prompt). When in doubt: anything the LLM sees → `content_language`; anything ONLY the user sees → `language`.
+
+16. **Persistent settings overlay applies via `open_repo`.** Adding a new override key requires extending three places together: `db/repo.py:_OVERRIDE_KEYS`, `db/repo.py:_apply_db_overrides` (with the actual settings mutation), and `unread/settings/commands.py:_KNOWN_KEYS` (so `unread settings set <key>` validates). Don't apply overrides anywhere except `_apply_db_overrides` — keeping the surface small avoids drift.
+
+## 4. Patterns to follow
+
+### Async tests with real SQLite
+
+```python
+from pathlib import Path
+import pytest
+from unread.db.repo import Repo
+
+@pytest.fixture
+async def repo(tmp_path: Path) -> Repo:
+    r = await Repo.open(tmp_path / "t.sqlite")
+    yield r
+    await r.close()
+
+async def test_x(repo: Repo) -> None:
+    ...   # pytest-asyncio auto mode; no @pytest.mark.asyncio needed
+```
+
+`pytest-asyncio` auto mode is enabled in `pyproject.toml`. Never introduce in-memory stubs for the DB layer — every repo test uses a real temp SQLite via `tmp_path`.
+
+### Stubbing OpenAI (no network in tests)
+
+```python
+from unread.analyzer import openai_client
+
+async def test_retry(monkeypatch):
+    async def fake_completion(oai, model, messages, max_tokens, temperature):
+        # return a minimal fake response with .choices[0].message.content,
+        # .choices[0].finish_reason, and .usage.prompt_tokens/completion_tokens
+        ...
+    monkeypatch.setattr(openai_client, "_completion", fake_completion)
+    # ...call chat_complete and assert
+```
+
+For `ask` rerank tests, stub `unread.ask.rerank.chat_complete` directly — see `test_rerank_total_failure_returns_keyword_sorted_pool`.
+
+### Typer command stub (keeps `unread --help` fast)
+
+```python
+@app.command(rich_help_panel=PANEL_MAIN)
+def newcmd(...) -> None:
+    """One-line help."""
+    from unread.analyzer.commands import cmd_newcmd    # lazy
+    _run(cmd_newcmd(...))
+```
+
+For sub-commands, register on the existing Typer instances: `chats_app`, `cache_app`, `reports_app`. Don't import Telethon/OpenAI/heavy modules at module top of `cli.py`.
+
+### Lint exceptions — don't "fix" them
+
+`pyproject.toml` intentionally suppresses these (read the comment block there):
+
+- `RUF001 / RUF002` — em-dashes and `×` in Russian UX strings
+- `PLC0415` — lazy imports in CLI command bodies
+- `B008` — `typer.Option(...)` in function defaults
+- `PLW0603` — settings singleton uses module-level global
+- `E501` — long lines in SQL strings and log messages
+- `PLR09xx` — intentionally flat functions
+
+`RUF003` (ambiguous chars in *comments*) is still enforced. Use `*`, not `×`, in comments.
+
+## 5. Preset format (`presets/<lang>/*.md`)
+
+Presets live under per-language directories (`presets/en/`, `presets/ru/`).
+The active language is `settings.locale.language` (default `"en"`); the
+loader does NOT fall back across languages — each language is autonomous.
+Use `prompts.get_presets(language)` to get the dict for a specific
+language; `prompts.PRESETS` is a backward-compat lazy proxy that resolves
+to the active locale's tree at access time.
+
+```
+---
+name: summary                   # MUST match filename stem (load-time check)
+prompt_version: v2              # BUMP when prompt body changes (per-language; bump only the language(s) you edited)
+description: One-liner shown by the wizard's preset picker  # NEW — optional
+needs_reduce: true              # false → skip map-reduce even on long inputs
+filter_model: gpt-5.4-nano      # map-phase model
+final_model: gpt-5.4-mini       # reduce / single-chunk model (note: default is mini, not flagship)
+output_budget_tokens: 4000      # reduce / single-chunk max_tokens
+map_output_tokens: 1500         # per-chunk map max_tokens
+enrich: [link, image]           # optional — declares this preset's enrichment needs
+---
+<system prompt goes here>
+---USER---
+<user template — must contain {period} {title} {msg_count} {messages}>
+```
+
+Files starting with `_` (`_reduce.md`, `_base.md`) are helpers, not presets. Missing `---USER---` → whole body becomes system, `DEFAULT_USER_TAIL[language]` is used. Missing standard placeholders are auto-appended; **typo placeholders fail at load time** with a readable error (see `_validate_user_template`). Name vs filename stem mismatch also fails at load time.
+
+**Adding a new preset:** drop the `.md` under `presets/<lang>/` with `description:` set in the frontmatter; the wizard picks it up automatically. Translate to other supported languages.
+
+**When to bump `output_budget_tokens`:** if a user reports the "⚠ Output truncated" banner. Also bump `prompt_version` (otherwise the cached truncated-first / fine-later mix gets reused).
+
+## 6. Adding a new CLI command — checklist
+
+1. **Handler in `unread/<subpkg>/commands.py`** — `async def cmd_newcmd(...)`, uses `open_repo` + `tg_client` async context managers.
+2. **If it's a run-style command** (resolve chat → fetch messages → do X): use `core/pipeline.prepare_chat_run`.
+3. **Typer stub in `cli.py`** — lazy import, match `@app.command(rich_help_panel=PANEL_*)`. Pick the right panel.
+4. **If the command costs money**: add `--max-cost` (use `analyzer.pipeline.estimate_cost` or `count_tokens + chat_cost`) and `--dry-run`.
+5. **If the command's flags affect analysis output**: extend `AnalysisOptions.options_payload` so the cache key reflects them.
+6. **Tests**: unit-test pure logic — don't test Typer wiring itself.
+7. **Update README.md** if user-facing.
+8. **Update `unread-usage` skill** so future sessions surface the new command.
+9. **Update CLAUDE.md** if you've added a new invariant.
+
+## 7. Adding a new top-level config block
+
+1. New `_StrictCfg` subclass in `config.py`.
+2. Add as a `Field(default_factory=…)` on `Settings`.
+3. Reference via `get_settings().<section>.<field>`.
+4. Document in `config.toml.example` (commented-out, showing default).
+5. `_StrictCfg` inheritance + `Settings.model_config = SettingsConfigDict(extra="forbid")` makes typos fail loudly.
+
+## 8. Rerunning the user's request — `--repeat-last` plumbing
+
+`Repo.put_last_run_args(chat_id, thread_id, args_dict)` persists a JSON-safe slice of cmd_analyze's flags after every successful run. `Repo.get_last_run_args(chat_id, thread_id=0)` reads them back. The `--repeat-last` block in `cmd_analyze` overrides default-shaped values with saved ones; explicit CLI flags still win.
+
+When adding a new flag that should participate in `--repeat-last`:
+1. Add to the dict written by `repo.put_last_run_args(...args=...)` in `_run_single`.
+2. Add a "default-shaped → use saved" branch in the `--repeat-last` overlay in `cmd_analyze`.
+3. Be explicit about JSON-safety — Path, datetime, EnrichOpts must flatten to scalars.
+
+## 9. Schema additions
+
+Add the table at the bottom of `db/schema.sql` with `IF NOT EXISTS`. For columns on existing tables, add to `_apply_additive_migrations` in `db/repo.py` so older DBs catch up. Recent additions:
+
+- `chat_last_run_args(chat_id, thread_id, args_json, updated_at)` — backs `--repeat-last`.
+- `message_embeddings(chat_id, msg_id, model, vector BLOB, created_at)` — backs `unread ask --semantic`.
+- `analysis_cache.truncated INTEGER NOT NULL DEFAULT 0` (additive on existing rows).
+- `messages.reactions TEXT` (additive on existing rows).
+- `youtube_videos(video_id, …, transcript, transcript_timed_json, …)` — backs `unread analyze <youtube-url>`.
+- `website_pages(page_id, url, paragraphs_json, content_hash, extractor, …)` — backs `unread analyze <website-url>`. `page_id = sha256(normalized_url)[:16]`; `content_hash = sha256(joined paragraphs)[:32]` flows into `AnalysisOptions.options_payload`.
+
+Test idempotency in `test_regressions_deep_review.py:test_schema_apply_is_idempotent` and additive migrations in `test_schema_apply_upgrades_legacy_tables`.
+
+## 10. Cost guard / budget pattern
+
+```python
+from unread.analyzer.pipeline import estimate_cost
+
+lo, hi = estimate_cost(n_messages=N, preset=preset, settings=get_settings())
+if hi is not None and hi > max_cost:
+    if yes:
+        raise typer.Exit(2)
+    if not typer.confirm(f"Estimate ${lo:.4f}–${hi:.4f} > ${max_cost}; run anyway?"):
+        raise typer.Exit(0)
+```
+
+For `ask`-shape commands (no map-reduce), use `count_tokens(prompt, model) + chat_cost(model, prompt_tokens, 0, max_tokens)` — see `ask/commands.py:_run_single_turn`.
+
+## 11. Citation expansion / audit pattern
+
+`_expand_citations(body, *, chat_id, repo, context_n, thread_id, cap=30)` — regex-parses `[#msg_id](url)` citations, looks up surrounding messages via `Repo.get_messages_around`, appends `<details>` blocks. Cap exists to prevent a runaway LLM from 10×-ing the report file. Backticks in body text are escaped to avoid breaking the inline `[ts #msg_id]` code span.
+
+`_self_check(result, messages, repo)` — cheap-model audit pass; appends `## Verification` to the report. Best-effort; failure returns empty string and the run continues.
+
+Both run between `run_analysis` and `_print_and_write` in `_run_single`. They mutate `result.final_result` so downstream rendering / posting / `--post-saved` see the augmented body.
+
+## 12. Posting analyses to Telegram
+
+`_post_to_chat(client, repo, result, *, title, target)` — resolves any chat ref via `tg/resolver.resolve` and sends. `me` / `saved` / empty target → `client.get_me()`. `_split_for_telegram` chunks on paragraph → line → hard-cut to stay under Telegram's 4000-char per-message ceiling. Failures warn but don't crash the run (the report is already on disk).
+
+`_post_to_saved_messages` is a back-compat shim that calls `_post_to_chat(target="me")` so legacy `--post-saved` plumbing keeps working without threading `repo` through.
+
+## 13. `unread ask` retrieval pipeline
+
+```
+cmd_ask
+  ├─ tokenize (bilingual stop-words)
+  ├─ optional --refresh: backfill scoped chats
+  ├─ optional --build-index: build embeddings, exit
+  ├─ retrieve_messages (keyword) OR semantic_search (cosine)
+  ├─ optional rerank (cheap-model 1-5 ratings)
+  ├─ format_messages with per-chat link templates
+  ├─ count_tokens cost preview / --max-cost guard
+  ├─ chat_complete (multi-turn message history during the post-answer follow-up loop)
+  └─ render or save
+```
+
+When adding a flag that affects retrieval (limit, scope, model): respect rerank's `keep` logic — `candidate_limit = max(limit, ask_cfg.rerank_top_k)` if rerank is on. Re-sort chronologically after rerank for the LLM context.
+
+For the post-answer follow-up loop: `_build_history_messages(system, history, new_user_text)` produces `[system, user, assistant, …, user]`. System prompt stays at index 0 so prompt-cache hits on the first turn are byte-identical to single-shot.
+
+Use `phase=ask` in `chat_complete` context for cost attribution.
+
+## 14. Wizard step graph — don't break it
+
+Order: `chat → thread (forum) → preset (analyze) → period → enrich → output → mark_read → confirm`. Period before enrich is deliberate — the enrich picker shows period-scoped media counts via `Repo.media_breakdown(...)` keyed by `_period_to_db_filters(period, custom_since, custom_until, custom_from_msg, chat)`.
+
+When adding a step:
+- BACK transitions in `output` and `mark_read` go to `enrich` (or to `preset`/`chat` for run-on-all).
+- ESC must be wired via `_bind_escape(question, BACK)` to return BACK; first step uses `None` (cancel).
+- Folder column in pickers comes from `tg/folders.chat_folder_index(client)` — re-fetch when relevant.
+
+`_expand_printable_for_search()` runs at module import and extends `string.printable` with Cyrillic / Greek / Arabic / Hebrew / Latin Extended ranges. CJK is excluded on purpose (would register tens of thousands of bindings).
+
+## 15. Before opening a PR — the CI-equivalent local check
+
+```bash
+uv run pytest -q                     # must be 100% green; no skips without a reason
+uv run ruff check .                  # must be "All checks passed!"
+uv run ruff format --check .         # must report "N files already formatted"
+```
+
+These three are exactly what `.github/workflows/ci.yml` runs on Python 3.11, 3.12, 3.13. CI caches `uv.lock` — **`uv.lock` must be committed** (history shows a CI failure from this exact footgun).
+
+Run `unread doctor` after any DB / config / dependency change.
+
+When in doubt, check `CLAUDE.md` at the repo root — it's the canonical pointer to invariants and anti-patterns. `ROADMAP.md` tracks deferred work; `tests/test_regressions_deep_review.py` is the running list of bugs locked in by tests.