scourgify 1.0.0__tar.gz

scourgify-1.0.0/.github/workflows/publish.yml

@@ -0,0 +1,61 @@
+name: Publish
+
+# Two triggers, one workflow:
+#   - push to main      -> auto-publish to TestPyPI (every release commit)
+#   - workflow_dispatch -> pick target manually (testpypi or pypi)
+#
+# Production PyPI stays gated behind manual dispatch — uploads are permanent, so
+# target=pypi should be a deliberate human action, never an accidental push.
+#
+# Trusted Publishing (OIDC) authenticates both indexes — no API tokens stored.
+# Requires a matching trusted publisher registered on PyPI *and* TestPyPI:
+#   owner=elfensky, repo=scourgify, workflow=publish.yml, environment=pypi
+# (if any of those differ in your PyPI registration, update them here to match).
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+    inputs:
+      target:
+        description: "Where to publish"
+        type: choice
+        options:
+          - testpypi
+          - pypi
+        default: testpypi
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: ${{ inputs.target == 'pypi' && 'https://pypi.org/project/scourgify/' || 'https://test.pypi.org/project/scourgify/' }}
+    permissions:
+      id-token: write # required for Trusted Publishing (OIDC)
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v8.1.0
+        with:
+          python-version: "3.13"
+
+      - name: Verify before publishing
+        run: |
+          uv sync
+          uv run tests/test_core.py
+
+      - name: Build sdist and wheel
+        run: uv build
+
+      # Push to main (inputs.target unset) and dispatch target=testpypi land here.
+      - name: Publish to TestPyPI
+        if: github.event_name == 'push' || inputs.target == 'testpypi'
+        run: uv publish --index testpypi --trusted-publishing always
+
+      # PyPI only on explicit dispatch target=pypi — a push to main can never
+      # reach this step.
+      - name: Publish to PyPI
+        if: github.event_name == 'workflow_dispatch' && inputs.target == 'pypi'
+        run: uv publish --trusted-publishing always

scourgify-1.0.0/.gitignore

@@ -0,0 +1,28 @@
+# Personal library data — review maps, proposals, intermediates (derived from YOUR Calibre library)
+data/
+
+# Ignore stray CSVs anywhere; the bundled generic defaults/ are the only tracked CSVs
+*.csv
+!src/scourgify/defaults/*.csv
+
+# Calibre DB snapshots / backups
+*.db
+
+# Generated per-user config is fine to ship as a default, but ignore local overrides
+overrides/
+
+# Python / OS cruft
+__pycache__/
+*.pyc
+.venv/
+.DS_Store
+
+# compiled Apple FM bridge binary (build from afm.swift; lives next to it in the package)
+/afm
+src/scourgify/afm
+*.o
+classify_state.json
+
+# build artifacts
+/dist/
+*.egg-info/

scourgify-1.0.0/CLAUDE.md

@@ -0,0 +1,153 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+scourgify normalizes a [FanFicFare](https://github.com/JimmXinu/FanFicFare)-imported
+[Calibre](https://calibre-ebook.com) library — consolidating tags, fandoms, characters, relationships,
+genres, and status. It is data-driven (bundled `defaults/` + per-user `overrides/` + `config.toml`),
+audit-first, and reversible. Python stdlib + Calibre's own CLI + `rich`; tests in `tests/` (plain asserts,
+no framework needed). **rich dependency rules by surface:** `wizard.py`/`ui.py` may hard-import rich (the
+wizard is rich-first; `ui.py` raises a friendly install hint if missing). The core tools
+(`wrangle`/`classify`/`staleness`) import rich under `try/except` and every rich use there needs a plain
+fallback (scripting/CI without rich must keep working). `_writer.py` runs under `calibre-debug` (Calibre's
+bundled Python has empty site-packages) — never import rich (or `ui`/`wizard`) there.
+
+## Running it
+
+Everything keys off `CALIBRE_LIBRARY` (the folder containing `metadata.db`):
+
+```bash
+export CALIBRE_LIBRARY="$HOME/Calibre/fanfiction"
+uv run scourgify                                     # no args = the interactive wizard (rich required; TTY only)
+uv run scourgify setup                               # interactive health check + setup (FanFicFare, columns, config)
+uv run scourgify audit                               # read-only dry-run of every pass
+uv run scourgify apply --apply                       # write changes (Calibre CLOSED for the write step)
+```
+
+(`uv run scourgify` from a checkout; an installed copy — `pipx install scourgify` — drops the `uv run`.)
+
+**`wizard.py`** (launched by bare `scourgify`) is a guiding menu wizard: on launch it
+detects an un-set-up library (missing columns / no config.toml) and routes to setup; the header shows
+books, column health, new/changed-since-last-run count, pending proposal, Calibre-open warning; the
+menu default adapts via `recommend()` (setup → review-pending → maintenance → audit). Item 0
+(`act_flow`) is a guided maintenance run sequencing wrangle → staleness → classify → review with
+per-step explanations and skips. It calls the same engine functions the subcommands do (previews →
+confirm → write), so guardrails and auto-backup apply identically; guardrail `SystemExit`s return to
+the menu. `ui.py` holds the shared rich Console + prompt helpers (lintle `term.py` pattern). classify
+runs render a live dashboard (`classify._Dashboard`: progress, tagged/failed/rate, throughput
+sparkline, rising candidates).
+
+**Packaging.** The code is a proper installable package under `src/scourgify/` (hatchling; on PyPI as
+`scourgify`). The single `scourgify` console command (`cli.py`) dispatches argv to the tools: bare → wizard,
+`setup`/`audit`/`apply` → wrangle, `classify`, `staleness`. Bundled `defaults/` (and `_writer.py`, `afm.swift`)
+ship **inside** the package (read-only at runtime); per-user `config.toml`, `overrides/`, and `data/` resolve
+against the **current working directory** — so `uv run` from the repo (CWD = repo root) behaves exactly as
+before, while an installed copy writes proposals under wherever it's invoked. `common.HERE` is the package
+dir (use it only for shipped read-only files); anything user-writable keys off `os.getcwd()`.
+
+**Everything runs under normal CPython** — the installed `scourgify` command, `uv run scourgify`, or plain
+`python3` with rich installed. The core operating rule is about *reads vs writes*, not which interpreter:
+- **Reads** (audit, classify proposal, setup health check) — read-only `sqlite3 ... mode=ro`; fine while Calibre is open.
+- **Writes** — the standalone tool computes the change-set, serializes it to JSON, and shells out **once** to
+  `calibre-debug -e _writer.py -- ops.json` (Calibre's API is the only fast batch-write path; `calibredb set_metadata`
+  is one book per process). `run_writer()` (in **common.py**; imported by wrangle/classify/staleness) does this,
+  **automatically snapshots metadata.db to `/tmp/ff_<ts>.db` first** (prints the path — that's the rollback), and
+  **refuses to run while Calibre is open** (it locks the DB). The user never types `calibre-debug`. Master rollback =
+  the full "Export all Calibre data" backup.
+- **`_writer.py`** is the only file that imports Calibre — a generic ops executor (`create_column` / `set_field` /
+  `stamp_now` / `set_pref`).
+- **`common.py`** is the shared core: lazy `CALIBRE_LIBRARY` resolution (importing any module never exits),
+  `ro_connect()`, link-table-aware `read_custom_column()`, `norm`/`ascii_fold`, the minimal TOML `load_config()`,
+  and `run_writer()`. Don't re-implement any of these in a tool script.
+
+Verification: `uv run tests/test_core.py` (plain asserts, pytest-compatible, no library/network needed) pins the
+pure core — `transform`, trope-chain resolution, `parse_resp`, the TOML reader. `scourgify audit` remains the
+against-your-library check: full new state, before/after counts, and SAFETY lines asserting **no book loses its
+last fandom or character** plus a **tag mass-deletion guardrail** (`apply` aborts if tags would shrink >25% and
+>200 assignments — the signature of an over-broad junk rule; `--force` overrides).
+
+## Maintenance loop (after new FanFicFare downloads)
+
+**Order matters: deterministic cleanup (wrangle) FIRST, content tagging (classify) second** — raw
+junk tags inflate a book's tag count and would hide it from the classifier's sparse-book targeting.
+
+```
+FFF fetch → uv run scourgify apply --apply           # 1. junk-drop/canonicalize the new raw tags (idempotent)
+          → uv run scourgify staleness --apply       # 2. free; re-derive #status from #updated age
+          → uv run scourgify classify --incremental  # 3. cheap; only books changed since last wrangle
+          → review data/classify_proposal.csv        # 4.
+          → uv run scourgify classify --apply        # 5. Calibre closed (writes shell to calibre-debug)
+```
+
+(Or the wizard: `uv run scourgify` → menu 3 → 4 → 5 → 6.)
+
+**⚠️ Cost:** a full Gemini `classify --fresh` pass over the library ≈ **€50** in tokens. Never run `--fresh`
+casually — use `--incremental` (only changed/new books), `--batch N`, or `--engine apple` (free, on-device).
+Confirm with the user before any full cloud run (classify itself gates cloud runs >200 books behind a
+confirmation / `--yes`). **Do NOT bulk re-fetch FFF metadata** — it re-pollutes columns not protected by
+`custom_cols_newonly`.
+
+## Architecture
+
+**`wrangle.py` — the unified engine.** Subcommands `audit` / `apply` / `setup`. Loads three layers:
+`defaults/` (generic, shipped) ← `config.toml` (column map + behavior toggles) ← `overrides/` (per-user,
+**gitignored**, same file formats, wins on conflict). `load_maps()` builds the in-memory maps; `transform()`
+is the per-book core: fandom alias→canonical, character folding (global + fandom-scoped), genre
+split→canon→route, tag junk-drop / trope-route / redundancy-strip. Strips a redundant tag only when the
+concept already lives in that book's structured column (**backfill-before-strip**).
+
+**The FFF→Calibre column model** (see README "FanFicFare → Calibre columns"): `category`→`#fandoms`,
+`characters`→`#characters`, `ships`→`#relationships`, `genre`→`#genres`, `status`→`#status`, real
+`series`→builtin Series. Two gotchas the tool exists to fix: `include_in_series:category` stuffing fandoms
+into the numbered Series field, and aggressive franchise unification (e.g. all Fate/Nasuverse → `Type-Moon`).
+
+**`classify.py` — content-based tagging** (separate from the deterministic engine; uses an LLM). Two outputs
+per book: `added_tags` (chosen from the controlled vocab `defaults/classify_vocab.txt` → applied) and
+`proposed_new` (novel candidates → aggregated to `classify_newtags_ranked.csv` for review→promotion, so the
+vocab grows without freeform noise). Engines `--engine apple|claude|openai|gemini` (keys via env:
+`ANTHROPIC_/OPENAI_/GEMINI_API_KEY`); `apple` = on-device, free, single-threaded. Concurrency via
+`ThreadPoolExecutor` (`--workers`), retry/backoff, incremental save + resume. `--text-fallback` samples the
+book's own prose (EPUB via zipfile, other formats via `ebook-convert`) when the `#comments` description is
+too thin. `--incremental` re-tags only books whose `#updated` is newer than their per-book **`#wrangled`**
+datetime marker (auto-created and stamped on `--apply`) — state lives in the library, no external file.
+Proposals/outputs live in `data/` (gitignored); `--apply` archives the proposal to
+`classify_proposal_applied_<ts>.csv` so stale rows never re-add hand-removed tags.
+
+**`staleness.py`** — re-derives `#status` for the activity family {In-Progress, Hiatus, Abandoned} from
+`#updated` age (`<2y`→In-Progress, `2–5y`→Hiatus, `≥5y`→Abandoned); idempotent + self-correcting on re-run.
+Completed/Dropped/Rewritten and date-less books are never touched.
+
+**`build_defaults.py`** — maintainer tool: regenerates `defaults/` from the source library's gitignored
+review-map CSVs (in `data/`). Curated cross-library knowledge (e.g. franchise unification) lives in its `CURATED_FAN`.
+
+## Gotchas worth knowing before editing
+
+- **Column creation needs the legacy DB object**, then a reopen: `DB(LIB).create_custom_column(...)` →
+  re-instantiate `DB(LIB).new_api` before the new column is usable in the same process. `Cache` has no
+  `all_field_keys` — use `api.field_metadata.all_field_keys()`.
+- **Single-value columns may use a link table.** Read a custom column by detecting
+  `books_custom_column_{id}_link`; fall back to the `book` column in `custom_column_{id}` if absent.
+- **`tropes.csv` is parsed leniently** (`read_tropes` + `resolve_trope_chains` in `wrangle.py`):
+  delimiter-sniffed (`,` or `;`), positional columns, unknown route → `tag` (so freeform notes don't crash),
+  and variant→canonical chains/cycles are resolved to a terminal at load. Hand-editing it is expected.
+- **Gemini hard-blocks ~1% of extreme content** as `PROHIBITED_CONTENT` (non-configurable; `safetySettings`
+  only relaxes the 4 HARM categories). It's deterministic — recover those books with `--engine openai` or
+  `--engine apple`. `classify.py` logs failures to `classify_failures.csv`.
+- **No `tomllib`** under `calibre-debug`'s Python — `common.py` ships a minimal TOML reader (quote-aware so
+  values can contain `#`; tolerates trailing comments on section headers).
+
+## Repo conventions
+
+- **Personal library data is gitignored and lives in `data/`** (review maps, proposals, cluster
+  intermediates); `.gitignore` also ignores stray `*.csv` **except** `!src/scourgify/defaults/*.csv`, plus
+  `*.db`, `overrides/`, the compiled `afm` binary (`/afm` and `src/scourgify/afm`), and build artifacts
+  (`/dist/`, `*.egg-info/`). Only the generic `defaults/` ship (bundled inside the package).
+- **`attic/`** holds the original single-purpose pipeline (`apply_*.py`, `generate_*.py`, `dryrun.py`,
+  `recover_xianxia.py`), kept as provenance — see `attic/README.md`. `scourgify` supersedes it; the attic
+  scripts read CSVs from their own directory and predate the auto-backup, so prefer the live tools.
+- `src/scourgify/afm.swift` is the Apple Foundation Models bridge for `scourgify classify --engine apple`;
+  it ships in the package (a `swift` toolchain runs it as-is), or build the faster binary with
+  `swiftc -O src/scourgify/afm.swift -o src/scourgify/afm` (requires macOS 26+ / Apple Intelligence).
+- **Publishing:** `uv build` → `uv publish --index testpypi` (dry-run) → `uv publish` (PyPI). The wheel must
+  contain `scourgify/defaults/*`, `_writer.py`, `afm.swift` and **not** `data/`, `overrides/`, or the `afm`
+  binary — verify with `unzip -l dist/*.whl` after a layout change.

scourgify-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 elfensky
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scourgify-1.0.0/PKG-INFO

@@ -0,0 +1,259 @@
+Metadata-Version: 2.4
+Name: scourgify
+Version: 1.0.0
+Summary: Normalize and consolidate tags/fandoms/characters/genres in a FanFicFare-imported Calibre library
+Project-URL: Homepage, https://github.com/elfensky/scourgify
+Project-URL: Repository, https://github.com/elfensky/scourgify
+Project-URL: Issues, https://github.com/elfensky/scourgify/issues
+Author-email: Andrei Lavrenov <andrei@lav.ren>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: calibre,ebook,epub,fanficfare,fanfiction,metadata,normalization,tags
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: rich>=13
+Description-Content-Type: text/markdown
+
+# scourgify
+
+The tag-wrangler / canonizer for your fanfiction library — normalize and consolidate **tags,
+fandoms, characters, relationships and genres** in a
+[FanFicFare](https://github.com/JimmXinu/FanFicFare)-imported [Calibre](https://calibre-ebook.com)
+library. Data-driven from **~1,700 bundled generic defaults**, fully customizable, audit-first and
+reversible.
+
+```bash
+pipx install scourgify                                # or: uv tool install scourgify  (one dependency: rich)
+
+export CALIBRE_LIBRARY="$HOME/Calibre/fanfiction"     # folder containing metadata.db
+scourgify                                             # ← the wizard: everything below, menu-driven
+```
+
+Requires **Calibre installed** — the tool reads via read-only sqlite and shells out to Calibre's own
+`calibre-debug` for writes. From a checkout, `uv run scourgify` (or `uvx --from . scourgify`) runs it
+without installing; [uv](https://docs.astral.sh/uv/) handles the environment (one dependency: rich).
+
+**The wizard** (no arguments) is the intended way in — and it steers you: on a fresh library it
+detects missing columns/config and offers to run **setup** immediately; afterwards the status header
+shows book count, column health, how many books are **new/changed since the last run**, and any
+pending proposal, and the menu **defaults to whatever the library needs next**. Menu item **0 —
+maintenance** walks the whole loop in the right order (wrangle → staleness → classify → review),
+each step explained, previewed, and skippable. Classify runs show a **live dashboard** (progress +
+tagged/failed/rate + throughput sparkline + rising tag candidates). Every write previews first,
+asks for confirmation, and auto-backs-up `metadata.db`.
+
+Each step is also a plain scriptable subcommand:
+
+```bash
+scourgify setup                                      # interactive health check + setup (FanFicFare, columns, config)
+scourgify audit                                      # read-only dry-run report of every pass
+scourgify apply --apply                              # write changes (Calibre CLOSED for this step)
+```
+
+Everything runs under plain `python3`. The tool reads via read-only sqlite and, for the actual writes,
+shells out **once** to `calibre-debug -e _writer.py` (Calibre's API is the only fast batch-write path) —
+so any command that writes (`apply --apply`, `setup` creating columns, `classify.py --apply`) needs
+Calibre **closed**, and refuses to run while it's open. You never invoke `calibre-debug` yourself.
+
+**`setup` is the first-run wizard + re-runnable health check.** It verifies, with `✓/⚠/✗` status and
+`Y/n` prompts (default-yes; `--yes` to auto-accept): the library; that the **FanFicFare plugin is
+installed and configured**, flagging + offering to fix the known gotchas (fandom-vs-series mapping,
+`include_in_series:category`, unprotected `#genres`); that every needed column exists (`#fandoms`,
+`#characters`, `#relationships`, `#genres`, `#status`, plus `#updated` and `#wrangled` for staleness /
+incremental classification), creating any that are missing; and writes `config.toml` (preserving your
+behavior toggles). Safe to re-run anytime.
+
+**rich** is required for the wizard and powers the live dashboards/tables everywhere else; the plain
+subcommands still degrade to text without it (rich is `try/except`-imported in the core tools, so
+scripting/CI without rich keeps working, and `_writer.py` under Calibre's bundled Python needs none).
+
+The engine reads:
+- **`defaults/`** — bundled, generic fanfic knowledge (FFN `Harry P.`→`Harry Potter`, JP→English
+  fandom titles, `no beta we die like…` = junk, …). Ships with the tool; not specific to anyone.
+- **`config.toml`** — your column mapping + opinionated behavior toggles (generated by `setup`).
+- **`overrides/`** *(optional)* — your own files (same formats as `defaults/`) that **extend and win
+  over** the defaults.
+
+---
+
+## FanFicFare → Calibre columns (how the linking works)
+
+FanFicFare scrapes metadata fields from each story and writes them into Calibre columns. The mapping
+lives in the FFF Calibre-plugin config (stored per-library in the `metadata.db` preference
+`namespaced:FanFicFarePlugin:settings` → key `custom_cols`). scourgify's `setup` reads that
+mapping, and creates any recommended columns you're missing.
+
+**Recommended mapping** (FFF metadata field → Calibre column):
+
+| FanFicFare field | Calibre column | Type | Holds |
+|---|---|---|---|
+| `category`   | `#fandoms`       | text, multiple | fandom(s) — **map this to `category`, not `series`** (see gotcha) |
+| `characters` | `#characters`    | text, multiple | characters |
+| `ships`      | `#relationships` | text, multiple | pairings |
+| `genre`      | `#genres`        | text, multiple | genres |
+| `status`     | `#status`        | text           | In-Progress / Completed / … |
+| `series`     | **Series** (built-in) | series    | the real site/AO3 series |
+| `numWords`   | `#words`         | int            | word count |
+| `numChapters`| `#chapters`      | int            | chapter count |
+| `dateUpdated`| `#updated`       | datetime       | last-updated date |
+| `storyUrl`   | `#storyurl`      | text           | source URL (also stored as the `url` identifier) |
+| *subject tags* | `tags` (built-in) | —            | freeform tags (what this tool normalizes most) |
+
+### ⚠️ The fandom-vs-series gotcha
+FanFicFare's `personal.ini` setting **`include_in_series:category`** stuffs the *fandom* into FFF's
+`series` field. If your `custom_cols` then maps **`#fandoms ← series`**, two things break: your
+**Series** column fills with fandom names (not real series), and **#fandoms** is fed from that
+fandom-stuffed series field. Fix:
+1. Remove `include_in_series:category` from `personal.ini` → `series` becomes the real (e.g. AO3) series.
+2. Map `#fandoms ← category` (the true fandom field).
+
+`wrangle.py setup` detects and offers to fix both, plus the protection below (the original
+`attic/apply_fff_config.py` does the same standalone). After that, real series fills in going
+forward and fandoms come from `category`.
+
+**Why fandom-as-series is especially bad:** Calibre's **Series** is a *numbered* field — every book
+gets a `series_index` (`A Fandom Name [1]`, `[2]`, …). So fandom-as-series doesn't just duplicate the
+fandom, it invents a bogus **ordered hierarchy**: dozens of unrelated stories become "book 1, book 2…
+of Harry Potter," a sequence that reflects nothing real. Clearing it (see `attic/apply_other.py`) and
+mapping `#fandoms ← category` removes the fake ordering; real series (where the index is meaningful,
+e.g. a genuine 3-part AO3 series) then populate correctly.
+
+### Franchise unification (fandom granularity)
+Related works in one universe (e.g. `Fate/stay night`, `Fate/Zero`, `Fate/Grand Order`) are distinct
+titles but one fandom. The bundled `defaults/fandoms.csv` unifies the obvious franchises to a single
+canonical — **the Fate/Nasuverse works all map to `Type-Moon`** (the studio/umbrella name the
+Nasuverse fandom uses). Prefer an **English title** as canonical wherever one exists (e.g.
+`The Saga of Tanya the Evil`, not `Youjo Senki`; `Puella Magi Madoka Magica`, not the romaji). This
+is a granularity *preference*: if you'd rather keep `Fate/Zero` separate from `Fate/stay night`,
+remove those rows from your `overrides/fandoms.csv` (or leave them unmapped). Note some franchises
+should **stay split** — Disney works are mostly standalone worlds (keep `DuckTales`, don't fold to a
+`Disney` mega-fandom), and `Overlord (Game)` vs `Overlord (Anime)` are unrelated. Curated
+unifications live in `build_defaults.py`'s `CURATED_FAN`.
+
+### Protecting your cleanup from re-pollution
+FFF's **`custom_cols_newonly`** (`{column: bool}`) controls overwrite-on-update: when `true`, FFF
+only writes that column **if it's empty**, so a metadata refresh won't clobber your normalized
+values. Recommended: `newonly:true` for `#genres`; leave `#status` **writable** so FanFicFare refreshes it on fetch (`staleness.py` re-derives the activity inference). The **built-in `tags` column is
+never protected** by this — so new downloads/updates re-add raw tag junk, and you re-run
+`wrangle.py` to clean it (see *Maintenance*).
+
+---
+
+## Customizing
+
+**`config.toml`** — column map + behavior toggles (all have sane, opinionated defaults):
+
+| Toggle | Default | Effect |
+|---|---|---|
+| `fold_characters` | `true` | apply abbreviation→full-name defaults |
+| `ascii_only_tags` | `true` | transliterate non-ASCII tags to plain ASCII |
+| `au_as` / `crossover_as` / `reincarnation_as` / `time_travel_as` | `genre` | put these tropes in `#genres` (`tag` to keep in tags) |
+| `fold_ratings` | `false` | fold `Erotica`→`Smut`, `Adult`→`Mature` |
+| `keep_categories` | `true` | keep `Multi`/`Gen`/`F/M` tags (`false` drops them) |
+
+**`overrides/`** — drop in `characters.csv`, `fandoms.csv`, `tropes.csv`, `junk.txt`,
+`genres_allow.txt`, … (same formats as `defaults/`). Anything here is merged on top of the bundled
+defaults and wins on conflicts. This is where *your* preferences live — the code stays generic.
+
+**`defaults/` formats:**
+- `characters.csv` — `variant,canonical,fandom` (blank fandom = global; set = homonym-scoped, e.g. `Luke C.` differs in Marvel vs PJO)
+- `fandoms.csv` — `alias,canonical`
+- `tropes.csv` — `variant,canonical,route` (route = `tag`|`genre`|`character`|`fandom`)
+- `genres_split.csv` (`combined,atoms`), `genres_canon.csv` (`variant,canonical`), `genres_allow.txt` (the genre vocabulary)
+- `junk.txt` — drop list (plain line = case-insensitive exact; `re:<regex>` = regex)
+- `ratings.txt` — content-rating/warning vocabulary
+
+---
+
+## Safety model
+`audit` and `apply` compute the full new state in memory and assert **no book loses its last fandom
+or character** (backfill-before-strip), aborting without writing if that fails. A second guardrail
+aborts if tags would **mass-shrink** (>25% of assignments and >200 lost — the signature of an
+over-broad `junk.txt` rule; `--force` overrides after you've checked). A redundant tag is only
+stripped when the concept already lives in that book's structured column. `audit` is read-only
+(plain `python3`, fine with Calibre open); `apply`/`setup` use the Calibre API (Calibre **closed**).
+**Every write automatically snapshots `metadata.db` to `/tmp/ff_<timestamp>.db` first** and prints
+the path — that's your instant rollback (master rollback = a full "Export all Calibre data" backup).
+
+## Maintenance — after new downloads / updates
+New stories arrive **raw** (junky subject tags, unfolded names). **Order matters — deterministic
+cleanup first, content tagging second**, because raw junk tags inflate a book's tag count and would
+hide it from the classifier's "sparsely tagged" targeting:
+
+```bash
+scourgify apply --apply                # 1. wrangle FIRST: junk-drop/canonicalize the new raw tags (idempotent)
+scourgify staleness --apply            # 2. free: re-derive #status from #updated age (independent, any time)
+scourgify classify --incremental       # 3. cheap: content-tag only new/changed books -> proposal
+                                       # 4. review data/classify_proposal.csv
+scourgify classify --apply             # 5. apply the reviewed tags + stamp #wrangled
+```
+
+Or just `scourgify` and walk the wizard menu in order: **3 wrangle → 4 staleness →
+5 classify → 6 review**. Re-running wrangle is always safe — it's idempotent and won't regress
+curated genres (it uses the full `genres_allow.txt`).
+
+---
+
+## Content-based tagging — `scourgify classify`
+Reads each book's description (`#comments`) and produces **two outputs**: (1) `added_tags` — tags chosen
+from the **controlled vocabulary** (`defaults/classify_vocab.txt`), which get applied; and (2) `proposed_new`
+— short novel tags *not* in the vocab, aggregated by frequency into `classify_newtags_ranked.csv` so you can
+review and **promote** the recurring ones into the vocab. Grows the tag set deliberately, without freeform noise.
+
+```bash
+scourgify classify --engine apple --limit 50        # propose -> data/classify_proposal.csv (dry-run, read-only)
+scourgify classify --apply                          # add the proposed tags (Calibre CLOSED)
+```
+- `--engine apple` — on-device **Apple Foundation Models** via `afm.swift` (free, private; macOS 26+,
+  Apple Intelligence). Ships as source; a `swift` toolchain runs it as-is, or from a checkout build the
+  faster binary once: `swiftc -O src/scourgify/afm.swift -o src/scourgify/afm`. Lower quality — prone to over-tagging,
+  so the prompt caps at `--max-tags 6` and dumps (>2× cap) are rejected.
+- `--engine claude|openai|gemini` — cloud APIs (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY`);
+  defaults `claude-haiku-4-5` / `gpt-4o-mini` / `gemini-2.5-flash`, override with `--model`. Sharper; cheap.
+- Only books with `< --min-tags` (default 2) tags **and** a description are touched. Always dry-run
+  until `--apply`. Edit `defaults/classify_vocab.txt` to shape the allowed tag set.
+- Long runs **save incrementally and resume** on re-run (skip books already in the proposal; `--fresh`
+  to restart). `--batch N` processes only N new books per run — handy for pacing API spend/rate limits.
+  A **spend gate** asks for confirmation (or `--yes`) before sending more than 200 books to a cloud engine.
+- Proposals/outputs live in `data/` (gitignored): `classify_proposal.csv`, `classify_newtags_ranked.csv`,
+  `classify_failures.csv`. On `--apply` the proposal is **archived** to `classify_proposal_applied_<ts>.csv`,
+  so a later apply can never re-add tags you've since hand-removed in Calibre.
+- **Incremental maintenance (`--incremental`):** after new FanFicFare downloads, `classify.py --incremental` (re)tags
+  only books whose `#updated` is newer than their own **`#wrangled`** marker (a datetime column auto-created and
+  stamped on `--apply`), plus any still untagged — cents instead of a full pass. State lives *in the library*
+  (travels with `metadata.db`, no external file). A full cloud `--fresh` run is expensive; reserve it for vocab changes.
+
+## Custom maps from your library (`overrides/`)
+The bundled `defaults/` are generic. Two helper workflows mined library-specific maps into `overrides/`
+(gitignored): AO3-style tag clustering (`overrides/tropes.csv`) and fandom **universe-unification**
+(`overrides/fandoms.csv`, e.g. `Avengers`/`Captain America (Movies)` → `Marvel`, `Game of Thrones (TV)`
+→ `A Song of Ice and Fire`). The engine loads these on top of the defaults automatically.
+
+## Repo layout
+The package lives in **`src/scourgify/`**; the single `scourgify` command (`cli.py`) dispatches
+bare → wizard, `setup`/`audit`/`apply` → wrangle, `classify`, and `staleness`.
+- **`cli.py`** — the `scourgify` entry point (argv dispatcher over the tools below)
+- **`wrangle.py`** — the engine: `setup` / `audit` / `apply`; with no command it launches the wizard
+- **`wizard.py` + `ui.py`** — the interactive wizard and its rich terminal helpers (the one
+  rich-required surface)
+- **`classify.py`** — content-based tagging (LLM engines) · **`staleness.py`** — `#status` re-derivation
+- **`common.py`** — shared core: library resolution, read-only sqlite + custom-column reading,
+  config, and `run_writer()` (the single write funnel, with automatic backup)
+- **`_writer.py`** — the only file that imports Calibre; a generic ops executor invoked under
+  `calibre-debug` by `run_writer()`, never by hand
+- **`defaults/`** — bundled generic maps, shipped inside the package (read-only at runtime)
+- Per-user files resolve against the **working directory**: `config.toml`, `overrides/` (your maps,
+  gitignored), and `data/` (proposals/intermediates, gitignored)
+- **`build_defaults.py`** (repo root) — maintainer tool: regenerates `defaults/` from the review maps in `data/`
+- **`tests/`** — `uv run tests/test_core.py`, no library needed
+- **`attic/`** — the original single-purpose pipeline, kept as provenance (see `attic/README.md`);
+  `scourgify` supersedes it.
+
+The per-library review-map CSVs in `data/` are gitignored (they contain your library's actual data);
+only the generic `defaults/` ship with the repo.