topicsleuth 0.1.0__tar.gz

topicsleuth-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dachary Carey
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

topicsleuth-0.1.0/PKG-INFO

@@ -0,0 +1,618 @@
+Metadata-Version: 2.4
+Name: topicsleuth
+Version: 0.1.0
+Summary: Topic detection and tracking over a stream of documents: link each item to a topic or open a new one, keep the evidence, and measure how topics move.
+Author-email: Dachary Carey <dc@dacharycarey.com>
+License-Expression: MIT
+Project-URL: Homepage, https://topicsleuth.dev
+Project-URL: Repository, https://github.com/agent-ecosystem/topicsleuth
+Project-URL: Issues, https://github.com/agent-ecosystem/topicsleuth/issues
+Keywords: topic detection,topic tracking,topic detection and tracking,first-story detection,llm,vector search,mongodb
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml
+Requires-Dist: numpy
+Requires-Dist: python-dotenv
+Provides-Extra: mongo
+Requires-Dist: pymongo>=4.6; extra == "mongo"
+Provides-Extra: voyage
+Requires-Dist: voyageai; extra == "voyage"
+Provides-Extra: anthropic
+Requires-Dist: anthropic; extra == "anthropic"
+Provides-Extra: all
+Requires-Dist: pymongo>=4.6; extra == "all"
+Requires-Dist: voyageai; extra == "all"
+Requires-Dist: anthropic; extra == "all"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Dynamic: license-file
+
+# topicsleuth
+
+You're interested in something specific. New content arrives every day:
+papers, release notes, posts, discussion threads. topicsleuth remembers
+them for you. It works out which ongoing question each new document
+belongs to, keeps the content related to every question in one place,
+and tells you which questions are gathering momentum, settling, or going
+quiet. It is the memory behind the topics you care about, so
+"what happened to that thing from two weeks ago?" has an answer.
+
+If that is the problem you have, the next section gets you running. The
+rest of this page explains what the library does and how, so you can
+decide whether it fits before you commit to it.
+
+## Try it in five minutes
+
+To use topicsleuth with the default backends, you need:
+
+- Python 3.12 or higher
+- A MongoDB Atlas cluster
+- A vector search index on the `topics` collection with:
+  - path `embedding`
+  - cosine similarity
+  - number of dimensions matching your embedding model
+  - index name `topic_embedding_index` (or set `TOPICSLEUTH_VECTOR_INDEX`)
+- API keys for:
+  - Voyage embeddings
+  - Anthropic
+
+Other databases and models work too; see "Bring your own models and
+database" below.
+
+### Install topicsleuth
+
+```bash
+pip install "topicsleuth[all]"
+```
+
+### Tell it what you're interested in
+
+The judge works from prompts that describe the field it is working in.
+The defaults describe a feed of papers, release notes and posts about
+a technical field. If that is not the content you're interested in,
+write a settings file before the first run, so the first topics it
+opens are the right shape. Two things go in it: the domain, as a
+noun phrase, and a `rubric` block with the prose the judge reads about
+your field.
+
+```yaml
+domain: municipal housing policy
+rubric:
+  source_kinds: council minutes, local news, planning filings, advocacy reports
+  reading_by_kind: a filing bears on a question by proposing a change, a minute by recording what was decided, a news story by reporting an outcome, an advocacy report by arguing a position
+  topic_size: 10-50 observations over a year
+  question_examples:
+    - question: Are cities loosening parking minimums near transit?
+      subject: parking
+    - question: Is inclusionary zoning producing units where it is adopted?
+      subject: zoning
+  subset_example: The candidate asks whether cities are loosening parking minimums near transit; the item is one city's ordinance doing so. The item is a subset, not a different question.
+  counter_examples:
+    - A council agenda that lists every item of the month, with candidates all rated `adjacent_question`: skip, not a "city council activity" meta-topic.
+```
+
+Every key is optional, and the full list with what each one does is on
+`RubricConfig` in `topicsleuth/rubric.py`. Only the wording about your
+field is yours to set. The labels the judge applies, the rules it follows
+and the questions it answers are fixed, because the code depends on them.
+`topicsleuth show-prompts` prints the prompts exactly as the judge will
+see them, which is the quickest way to check that the result reads well.
+
+### Use the library
+
+To use the default clients, import from `backends`. Otherwise,
+import `record`, `settings`, and `synthesize` to make and store
+observations related to your topics:
+
+```python
+from datetime import date
+
+from topicsleuth.backends.anthropic import AnthropicJudge
+from topicsleuth.backends.mongo import MongoStore
+from topicsleuth.backends.voyage import VoyageEmbedder
+from topicsleuth.record import record
+from topicsleuth.settings import load_settings
+from topicsleuth.synthesize import synthesize_items
+
+settings = load_settings("settings.yaml")
+store = MongoStore.from_uri(MONGODB_URI, database="briefing")
+embedder = VoyageEmbedder(model="voyage-4", dimensions=settings.embedding.dimensions)
+judge = AnthropicJudge(model="claude-opus-4-8")
+
+items = fetch_new_documents()        # your dicts: url, title, summary, source_type, tags ...
+
+class Eval:                          # what the library needs per document
+    def __init__(self, url, embedding, admitted):
+        self.url, self.embedding, self.above_threshold = url, embedding, admitted
+
+vectors = embedder.embed_documents([f"{d['title']}\n\n{d.get('summary', '')}" for d in items])
+evals = [Eval(d["url"], v, True) for d, v in zip(items, vectors)]
+
+decisions = synthesize_items(items, evals, store, settings, judge, embedder)
+result = record(items, evals, decisions, store, settings=settings, embedder=embedder,
+                run_id=f"my-feed-{date.today()}")   # so `clean --run-id` can remove this run
+
+for obs in result.observations_created:
+    print(f"{obs['item_title']} -> {obs['topic_id']} ({obs['connection_type']})")
+for obs in result.already_recorded:    # held from an earlier run; not judged or written again
+    print(f"{obs['item_title']} was already on {obs['topic_id']}")
+for failure in result.failures:        # retry these next run; do not mark them done
+    print(f"{failure.item_url} failed at {failure.stage}: {failure.error}")
+```
+
+Recording is safe to retry. A document is evidence for a topic once: the
+store holds one observation per document per topic, so running the same
+batch twice, or two programs admitting the same document, writes it once,
+and a document the store already holds is not judged again; the result
+lists it under `already_recorded`. A document that could not be
+finished is listed in `result.failures` with the stage it stopped at.
+Keep it in your queue and hand it over again next run: every stage is
+idempotent, so the retry completes what the failed run left undone (a
+snapshot, a status, an embedding) without repeating what it did. Pass a
+`run_id` and everything the run writes is stamped with it, which is what
+lets `clean --run-id` remove one run and nothing else.
+
+## Keep track of content across days, not just today
+
+Anyone who reads a feed and writes a summary ends up with the same thing:
+a good account of today, and no connection to yesterday. The paper from
+February, the product launch in June and the tutorial this morning are
+one story; an idea becomes a product which begets a practice. Your daily
+summary can't tell that story because it has no memory.
+
+Think of topicsleuth as the tool that helps you keep a notebook with one
+page per question. A subject like "security" is too broad. Instead, the
+topic is a question the field keeps working on, such as "do coding
+assistants make developers write less secure code?" A question is the
+right size for a page: specific enough that a document clearly belongs
+on it or not, but also broad enough to gather evidence for months.
+
+When you let topicsleuth keep that notebook, each piece of content that
+bears on a question becomes an **observation** on that topic's page. A
+document that bears on two questions, say a paper that is both about
+evaluation design and about how a coding harness runs commands, gets an
+entry on both pages, each judged on its own. An observation records which
+document it was, how strongly it bears on the question, and a label that
+says how it connects to what's already in the topic. The label is there so you can see at a glance when you open a topic
+which entries agree with each other and which ones pushed the question
+somewhere new. An observation can relate to its topic in one of four ways:
+
+- **supports**: adds evidence consistent with what the topic already says.
+  Most observations are this.
+- **extends**: broadens the question by bringing in a facet, method, or
+  kind of source the page had not covered.
+- **contradicts**: offers evidence against the topic's claim.
+- **shifts**: marks a change in the topic's trajectory, such as the first
+  vendor to turn a practice on by default.
+
+When a piece of content arrives that raises a question with no page yet,
+it starts a new one. And because the entries on each page are dated, it can
+measure what a summary never could: how fast a question is gathering
+evidence, from how many different sources, and where the question is in
+its life. Every topic carries a status that moves as evidence arrives or
+stops arriving:
+
+- **emerging**: a new page with fewer than five observations. It may be
+  the start of something, or a one-off.
+- **active**: five or more observations. The question has drawn repeated
+  evidence.
+- **mature**: twenty or more observations, a well-established thread. From
+  here on, a weak match (strength below 0.7) is still filed but marked
+  minor, so a mature topic does not drift on tangential links. Both numbers
+  are settings, under `record`.
+- **fading**: an active or mature topic that has gone a week with no new
+  evidence. Fresh evidence sends it back to active.
+- **dormant**: a fading topic whose last evidence is more than thirty days
+  old. Fresh evidence sends it back to emerging, to earn its place again.
+
+Topics you merge or split are marked **retired** and keep a pointer to
+what replaced them, so old links still resolve.
+
+## Know whether it fits before you build on it
+
+You might find topicsleuth useful when:
+
+- You have a steady stream of content about one area of interest.
+- You want continuity across days, not only a digest of each day.
+- You are willing to pay for at least one call to a strong language model
+  per document you choose to track. The judgment of where a document belongs
+  is made by a model, and that is where the cost goes.
+  
+You will also need a database with vector search; MongoDB Atlas works out
+of the box, and anything else needs an adapter (see "Bring your own
+models and database" for what one has to provide).
+
+It is the wrong tool if you want to discover clusters in a corpus you
+already have (that is topic modeling), if you want a dashboard or alerts
+(this produces the record a dashboard would read), or if you were
+searching for something to do with Kafka topics.
+
+## How a piece of content finds its topic
+
+Let's follow one document through the library: a changelog entry announcing
+that a coding tool now runs commands in a sandbox.
+
+**First, the library looks nearby.** Every topic and every document has an
+embedding, a vector that places its meaning so that similar things sit
+close together. The library pulls the topics nearest to the document,
+around thirty, and adds any that share a tag with it. This is quick and
+cheap, and it only finds the neighborhood; it does not decide anything.
+
+Tags are the one piece of your own vocabulary the library uses. A
+document arrives with whatever tags your program gave it, free-form
+strings such as `security` or `benchmarks`, and the judge sees them with
+the document. When the judge opens a topic it gives the page tags of its
+own, a merge or a split rewrites them, and topics you seed or add by hand
+carry the tags you wrote. From then on the tags are a second way to find
+the neighborhood: a topic that shares a tag with the document joins the
+candidates even when its embedding sits far away. Each entry also keeps a
+copy of its document's tags, so you can see later how a page's evidence
+breaks down. A tag every document carries, such as the name of the feed,
+would match every page; list it under `ignored_tags` in settings and the
+library drops it from matching and from the tags of new topics.
+
+**Then a language model decides.** Being near a topic is not the same as
+belonging to it. A document can sit right beside a topic and be about a
+different question. So the library shows a language model, which it calls
+the **judge**, the document and the similar topics, each with its
+one-sentence summary and a few of the entries already on it. The lib asks
+the judge if the document's question is the same as the existing topic,
+a narrower case of it, a neighbor of it, or does it only share vocabulary?
+The document goes on the page of every similar topic that is the same
+question or a narrower case, as an observation on each, scored for each.
+If there is none, the judge may propose a new topic, or decide the
+document is interesting but not the start of anything.
+
+The sandbox changelog lands on "are coding tools isolating what agents
+execute by default?", even though every other entry on that page is a
+paper and this is a release note. That is the point of the whole design:
+a topic gathers evidence of every kind related to the shared question.
+
+**New topics get a second opinion.** Putting a document on a slightly
+wrong page costs one noisy entry, easily moved later. Opening a page that
+should have been part of an existing one is worse: from then on, evidence
+splits between the two and nothing repairs it on its own. Because the two
+mistakes are not equal, every proposed topic goes to a second, independent
+model call that sees only the closest existing topics and the proposal,
+and can confirm it, fold the document into an existing topic instead, or
+drop it.
+
+**Finally, it is written down.** The entry is recorded with a strength
+that comes from three yes-or-no questions the judge answers about fit,
+never a number the model invents. The page's statistics are refreshed,
+and if the topic has crossed a threshold, its status changes.
+
+## Keep the judge honest
+
+The decision at the heart of the library is a judgment, and judgments
+drift. Edit the prompt, upgrade the model, change how many candidates the
+judge sees, and the topic assignment results can change without any
+noise letting you know something is different.
+
+In real-world use, changing the judge model from one Opus version to a
+newer one resulted in far more new topics being created versus observations
+linking to existing topics. Nothing in a daily briefing hints that 
+evidence is spreading across an increasing percentage of one-item topics
+until you notice the topic library has grown by hundreds.
+
+Treat the judge the way you would treat any classifier: take a baseline,
+change one thing, and measure. The library provides a tool for this; the
+ability to replay the decision on documents the store already holds,
+without writing anything:
+
+```bash
+# A baseline on 30 documents already in the store, drawn twice.
+topicsleuth tune synthesize --sample-query '{"source": "my-feed"}' --n 30 --seed 42 --label baseline
+topicsleuth tune synthesize --sample-query '{"source": "my-feed"}' --n 30 --seed 7 --label baseline
+
+# Each baseline writes tuning_runs/synthesize-<label>-n<n>-seed<seed>.json,
+# so the two runs above land in two files.
+
+# After a change, use `--replay` with the baseline report, then the diff.
+topicsleuth tune synthesize --replay tuning_runs/synthesize-baseline-n30-seed42.json --label after-42
+python scripts/diff_tuning_runs.py tuning_runs/synthesize-baseline-n30-seed42.json tuning_runs/synthesize-after-42-replay.json
+```
+
+Generate a baseline report over a sample of documents that you can
+use to to check how the judge assesses documents before and after
+a change. Run at least two sample sets before assessing whether a
+change has had the desired result. A fair share of documents often
+sit right on the boundary between "link" and "new topic" and can
+flip on their own from run to run. Two samples of thirty can tell
+you whether a shift is real or just those documents.
+
+The topicsleuth `tune synthesize` command can take a `--seed` flag
+that accepts a random integer to create a sample of `--n` documents
+from your data store. Any two integers can create random seeds, but
+keep using the *same* two integers so every experiment is judged on
+the same two samples. On every later run, `--replay` the report, so
+the comparison is always one to one even after the store has grown.
+
+Drawing a sample needs a store that can sample. The default store is
+MongoDB, which supports sampling. If you're using another store,
+write the first report by hand, as a list of document URLs in the shape
+of `data/examples/replay.example.json`, and `--replay` from it.
+
+The report tells you how many documents linked, proposed or skipped, how
+strong the links were, and for each call, where the chosen topic came from
+and how large the prompt was. The diff tells you which documents changed
+their minds. The `tune` command reads your settings file through
+`TOPICSLEUTH_SETTINGS`, so a change to the rubric block is replayed by
+editing the file.
+
+### What you can change to tune library performance
+
+Five things move the judge, listed here in order from the largest
+effect to the smallest. All of them live in the settings file. The
+first three can be overridden for a single replay from the command line,
+so you can try a value before you keep it.
+
+- **The model.** The judge is an object you construct, and `--model`
+  swaps it for one replay. Expect the boundary to move: a different model
+  draws the line between "link" and "new topic" somewhere else, which is
+  why a model change gets the full two-seed treatment.
+- **Which topics the judge sees.** `candidates.vector_limit` is how many
+  nearest topics go in the prompt, `candidates.tag_overlap` adds every
+  topic sharing a tag with the document, and `candidates.ignored_tags`
+  keeps a routing tag from matching everything. A judge cannot link to a
+  topic it was not shown. Flags: `--limit`, `--no-tag-overlap`,
+  `--ignore-tags`.
+- **The second opinion.** `synthesize.confirm_proposals` turns the
+  reviewer on, `confirm_candidates` is how many nearest topics it sees,
+  and `confirm_backend` picks the judge's second call or a decision model
+  with `confirm_link_threshold` as its bar. These change how many
+  proposals survive without touching the rubric. Flags: `--no-confirm`,
+  `--confirm-backend`, `--confirm-threshold`. In the same group,
+  `secondary_link_floor` drops a document's second or later link when its
+  strength is below the floor, and `--secondary-link-floor` tries one.
+- **The prose about your field.** `domain` and the `rubric` block from
+  "Tell it what you're interested in". Counter-examples are the working lever: a
+  replay shows you the decisions the judge gets wrong, and one
+  well-chosen counter-example states that case and its right answer.
+- **The rubric itself.** The labels, rules and questions in
+  `topicsleuth/rubric.py`. Change them in a fork, replay both seeds, and
+  expect every baseline you have to need re-taking.
+
+The problems below are the ones a store develops in practice. Each names
+the report line that shows it and the lever that moves it.
+
+### Too many new topics
+
+Evidence spreads across near-duplicate pages, each looks slow, and
+`suggest-merges` keeps finding pairs. In the report, the `propose_new`
+share is high and `confirmation outcomes` shows most proposals confirmed.
+Read the proposed topics and the diff first. If the right page existed
+and the judge was shown it, the fix is in the prose: write the case up
+as a counter-example in your `rubric` block. If the reviewer is waving
+proposals through, give it more to compare against with
+`confirm_candidates`, or switch `confirm_backend` to `decision` and lower
+`confirm_link_threshold` a step at a time. If the right page was not
+among the candidates at all, that is the next problem.
+
+### The right topic was never shown
+
+The report's `linked topic vector rank` says where each chosen topic sat
+in the nearest-neighbor list. When links cluster near the bottom of the
+list, or the diff shows documents finding a better page only after you
+raised `--limit`, the judge is being starved of candidates. Raise
+`vector_limit` in steps of ten and watch `mean input tokens`, which is
+the cost of each step. If `linked topic channel` shows the tag channel
+doing real work, check that `ignored_tags` is not dropping a tag that
+matters. The reverse also happens: a tag every document carries turns the
+channel into "every topic", and the fix is to list it.
+
+### One topic swallows everything
+
+A page with hundreds of entries whose summary could cover half the feed.
+In the report, `gate overrides` counts the times the judge wanted a new
+topic and the rules forced a link, and the strength buckets lean toward
+the low bands. The rules are doing what they are for, so the fix is in
+the store and the prose. Split the page with `split-topic`, and make
+`topic_size` and the `question_examples` say what narrow looks like in
+your field. A counter-example of the form "an item that only mentions X,
+judged against the X page, rated `surface_overlap`: do not link" closes
+the door.
+
+### The judge reasons about the wrong kind of document
+
+The reasoning traces talk about papers and releases when your feed is
+filings and minutes. The judge is working from the default prose. Set
+the `rubric` block, check it with `show-prompts`, and take a fresh
+baseline, because every comparison from before was made by a different
+judge.
+
+### Weak secondary links
+
+A document that bears on two questions is filed on both pages, and the
+second link is often the weaker one: the judge said the question is the
+same but answered the fit questions "no". On a well-evidenced topic that
+entry is marked minor, which is correct, but a page can fill with minor
+entries from documents whose main home is elsewhere. The report's
+`LINKS` line shows how many links were secondary, their strength
+buckets, and how many a floor dropped. Set `synthesize.secondary_link_floor`
+a step at a time (0.5 drops the "all three no" and "not citable" bands)
+and replay both seeds; a replayed report shows what each value would
+have dropped without a new model call. The first link is never dropped.
+
+### It costs too much
+
+`mean input tokens per call` in the report is the bill. The candidate
+list is most of it, so `vector_limit` is the first thing to lower and
+`confirm_candidates` the second. A cheaper model is worth a replay to see
+what it gives up, and `--model` tries it without committing to it.
+
+When the question is whether the judge's rubric is right at all, rather
+than whether a change moved it, build a small labeled set: pairs of a
+document and a candidate topic, labeled by a person or an independent
+model without seeing what the judge said. The `scripts/` folder can
+produce such labels from a model and compare raters against each other
+and against the judge. Keep your baselines and label sets with the code
+that owns your store, since they describe its content; `data/examples/`
+shows the shape of every file.
+
+## Keep the store tidy
+
+No judge declines to open a page for a document that is genuinely a
+one-off, and a notebook that has grown for months needs editing. The
+`topicsleuth` command does that against a store configured through the
+environment (`MONGODB_URI`, `TOPICSLEUTH_DATABASE`, `TOPICSLEUTH_SETTINGS`
+for your settings file, and optionally the model names; a `.env` file in
+the working directory is read). Every
+command that writes takes `--dry-run` and shows the plan first.
+
+```bash
+topicsleuth audit                               # how big the store is and how evidence is spread
+topicsleuth suggest-merges                      # pairs of topics that look like the same question
+topicsleuth merge-topics <source> <target>      # fold one into the other; the judge rewrites the survivor
+topicsleuth split-topic <id>                    # the judge proposes sub-topics and reassigns the entries
+topicsleuth add-topic topic.yaml --link <url>   # open a page by hand and move named entries onto it
+topicsleuth move-observation <url> <topic_id>   # move one document's entry to another page
+topicsleuth reprocess-tail --max-obs 1 --older-than-days 60   # clear pages that never got a second entry
+topicsleuth seed topics.yaml                    # start an empty store from a list of topics
+topicsleuth backfill-embeddings                 # embed topics that have none; refresh well-evidenced ones
+topicsleuth clean --hours 28                    # remove what the last run wrote
+```
+
+`reprocess-tail` clears the pages but does not re-judge their documents:
+it writes their URLs to a file (under `tuning_runs/` by default, or
+`--urls-out`), and your feeder hands those documents over again on its
+next run so they find a home among the topics that exist now.
+`backfill-embeddings` is for after an import or a change of embedding
+model: it embeds every topic that has no vector and recomputes the
+vector of every topic with at least five entries from the entries'
+own embeddings.
+
+Nothing here re-judges a document, except that a split asks the judge to
+re-assess each entry's strength and connection type against its narrower
+sub-topic. Merging and moving keep both. All three keep each entry's
+identity, date and metadata and note where it came from, and a retired
+topic keeps a pointer to what replaced it, so old links still resolve. A
+document that was evidence on both sides of a merge or move keeps the
+target's entry, with the other folded into it and noted. A
+split is checked in full before anything is written: every entry assigned
+exactly once, to a sub-topic that exists, with a known connection type.
+
+These commands may run while a feeder is writing. A document caught
+between a command and the feeder fails that run and is retried on the
+next, and a merge or split carries along anything that arrived on its
+source while it ran. If a command dies part-way through deleting a
+topic, the topic stays marked for up to fifteen minutes, during which
+recording and the other commands skip it and report it as busy; the next
+`clean` or `reprocess-tail` then finishes or undoes the interrupted
+deletion on its own.
+
+`clean` removes the observations, topics and snapshots written in a
+window, by when they were written rather than when their evidence
+happened, and with `--run-id` only what one run wrote (pass `run_id` to
+`record` and the run stamps everything it writes). It is not a full undo.
+A topic written in the window that still holds older evidence is kept,
+and a status, embedding or description a run changed is not restored.
+A topic another command is working on is left alone and reported as
+busy; run `clean` again once that command has finished.
+
+## Bring your own content
+
+topicsleuth never fetches documents and never decides which ones matter.
+That is the job of your own program, which this page calls the **feeder**:
+it gathers documents, chooses which to track, embeds them, and hands them
+over, then does what it likes with the outcome. The library owns the
+notebook, and it is the only code that writes to it, so two feeders
+sharing one store cannot disagree about where things belong.
+
+## Bring your own models and database
+
+Every service the library talks to is an object you construct and pass
+in: the embedding model, the language model that judges, the database,
+and optionally a decision-only model that reviews proposals in place of
+the judge's second call. Each is a `typing.Protocol` in
+`topicsleuth.interfaces`, with the methods listed there; anything that has
+them works, with no base class to inherit. The model protocols are a
+method or two. The store protocol is larger, and a few of its operations
+must be atomic: a conditional update for the lease a maintenance command
+takes on a topic, for publishing a topic's embedding and for registering
+a refresh it still owes, and a transaction wherever one operation writes
+two documents (deleting a leased topic's entries, moving or folding an
+entry onto a topic that must still be live). The docstrings on the
+protocol say which, and the store contract test in the source tree is the
+specification: an adapter that passes it will behave under retries and
+concurrent writers the way the defaults do. Defaults for Voyage, Anthropic,
+MongoDB Atlas and TypeSafe Jev are included as extras (`voyage`,
+`anthropic`, `mongo`), and the core package itself depends only on pyyaml,
+numpy and python-dotenv.
+
+A few decisions have to be shared by everyone writing to one store, and
+those live in settings rather than code: the domain and the prose the
+prompts are written with, the embedding dimensionality the index was
+built with, how candidates are assembled, and whether and how proposals
+are reviewed. The defaults ship with the package and
+`load_settings(path, overrides)` layers yours on top:
+
+```yaml
+domain: the field this store follows   # a noun phrase; every prompt is rendered with it
+rubric: {}                             # the prose about your field; see "Tell it what you're interested in"
+embedding:
+  dimensions: 1024                     # match your embedding model and your index
+candidates:
+  tag_overlap: true                    # topics sharing a tag with the document join the candidates
+  ignored_tags: []                     # a tag every document carries would match everything; list it here
+  vector_limit: 30                     # how many nearest topics the judge sees
+synthesize:
+  confirm_proposals: true              # the second opinion on every new topic
+  confirm_candidates: 5                # how many nearest topics the reviewer sees
+  confirm_backend: judge               # judge | decision
+  confirm_link_threshold: 0.7          # decision backend: how sure it must be to fold a proposal in
+  secondary_link_floor: 0.0            # drop a document's second or later link below this strength
+record:
+  minor_floor_after: 20                # from this many observations on a topic...
+  minor_floor_strength: 0.7            # ...a new one below this strength is marked minor
+```
+
+Loading history rather than today's documents? Pass `date_from_item=True`
+to `record` so entries are dated by publication rather than by the run,
+and load oldest first. A batch dated before a topic's latest snapshot is
+inserted into its history: the snapshot counts only evidence up to its
+date, and the topic's current status is left to the newer evidence. Pass
+`backfill=True` as well and every entry the batch writes is stamped
+`backfill`, so you can tell re-homed history from evidence that arrived
+fresh.
+
+## Test without spending anything
+
+The package ships stand-ins for every service: an embedder that hashes
+text to a stable vector, a judge that returns scripted answers, a decision
+model driven by a table, and an in-memory store. Together they run the
+whole loop with no network, so a feeder's tests are ordinary unit tests.
+
+```python
+from topicsleuth.backends.fake import HashEmbedder, ScriptedJudge
+from topicsleuth.backends.memory import MemoryStore
+```
+
+The `topicsleuth` command uses the same stand-ins when
+`TOPICSLEUTH_STORE=memory`, `TOPICSLEUTH_JUDGE=scripted`,
+`TOPICSLEUTH_EMBEDDER=hash` and `TOPICSLEUTH_DECISION=scripted` are set,
+which is how its own tests run. If
+you write a store adapter, the store contract test in the source tree's
+`tests/` folder (not in the wheel) runs against it.
+
+## Where things stand
+
+Early. The matching, recording, curation and tuning code has run daily
+against a live store for months, but the public interface is new and may
+move between minor versions, so pin a release. Releases are published to
+PyPI from version tags, and `DESIGN.md` explains the reasoning behind the
+interfaces and the decision for anyone who wants to go deeper.
+
+The name follows the topic detection and tracking literature, where the
+tracked thing is a topic, a document is a story, noticing a new topic is
+first-story detection, and attaching a document to one is tracking.
+
+MIT licensed.