linkedparticles 1.149.0__tar.gz → 1.172.0__tar.gz

linkedparticles-1.172.0/PKG-INFO

@@ -0,0 +1,320 @@
+Metadata-Version: 2.5
+Name: linkedparticles
+Version: 1.172.0
+Summary: Auditable, provenance-tracked memory for AI agents — the Particles engine and its API, CLI, and MCP surfaces
+Project-URL: Homepage, https://linkedparticles.org
+Project-URL: Documentation, https://docs.linkedparticles.org
+Project-URL: Repository, https://github.com/LinkedParticles/particles-engine-py
+Author: The Particles authors
+License: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: agent-memory,agents,ai-memory,audit,belief-store,beliefs,citations,claim-extraction,epistemic,epistemics,json-ld,knowledge-graph,knowledge-management,llm,mcp,memory,provenance,psum,rag,uncertainty
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Environment :: Web Environment
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: aiosqlite>=0.20
+Requires-Dist: alembic>=1.13
+Requires-Dist: apscheduler>=3.10
+Requires-Dist: fastapi>=0.111
+Requires-Dist: filelock>=3.13
+Requires-Dist: greenlet>=3.5.0
+Requires-Dist: jinja2>=3.1
+Requires-Dist: linkedparticles-core==1.172.0
+Requires-Dist: mcp<2,>=1.27.1
+Requires-Dist: opentelemetry-api>=1.27
+Requires-Dist: python-multipart>=0.0.9
+Requires-Dist: sqlalchemy[asyncio]<2.2,>=2.0
+Requires-Dist: typer>=0.12
+Requires-Dist: uvicorn[standard]>=0.30
+Requires-Dist: warcio>=1.7
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.3; extra == 'langchain'
+Provides-Extra: otel
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27; extra == 'otel'
+Requires-Dist: opentelemetry-instrumentation-fastapi>=0.48b0; extra == 'otel'
+Requires-Dist: opentelemetry-instrumentation-httpx>=0.48b0; extra == 'otel'
+Requires-Dist: opentelemetry-instrumentation-logging>=0.48b0; extra == 'otel'
+Requires-Dist: opentelemetry-instrumentation-sqlalchemy>=0.48b0; extra == 'otel'
+Requires-Dist: opentelemetry-sdk>=1.27; extra == 'otel'
+Description-Content-Type: text/markdown
+
+# linkedparticles
+
+**[linkedparticles.org](https://linkedparticles.org)** ·
+**[docs.linkedparticles.org](https://docs.linkedparticles.org)** ·
+[PyPI](https://pypi.org/project/linkedparticles/) ·
+[Apache-2.0](https://github.com/LinkedParticles/particles-engine-py/blob/main/LICENSE)
+
+<!-- Ownership proof for the official Model Context Protocol registry: the
+     registry matches this token against the server name in server.json before
+     it will accept a listing for the linkedparticles PyPI package. Keep it
+     byte-identical to server.json's `name`. -->
+<!-- mcp-name: org.linkedparticles/particles -->
+
+> **Particles is shared memory for humans and AI agents.** Each particle is one
+> claim, plus what you need to judge it: who said it, where, when, and how
+> confident they were. Facts, opinions, and memories are all claims, recorded
+> the same way as particles. Particles are not edited or deleted. Particles are
+> superseded, retracted, or disputed in the open. How much to trust it is a
+> perspective applied at query time, never baked into the record.
+
+`linkedparticles` is the **Engine** of the Particles reference implementation:
+the package you install to actually run a belief store. Give it documents,
+pages, chat logs, or an agent's own session notes; get back claim-granularity
+beliefs, each carrying calibrated confidence, an uncertainty kind, a resolved
+subject, and provenance back to the exact source bytes. Ask a question and the
+answer cites the beliefs it was built from. Nothing is ever overwritten, so you
+can ask the store what it believed a year ago and why it stopped.
+
+**When your agent is wrong, you can see exactly why, and fix it at the source.**
+
+The Engine is a library first. The HTTP API, the CLI, the read-only MCP server,
+and the resident daemon are all *surfaces* over it.
+
+## Install
+
+```bash
+pip install linkedparticles
+```
+
+Python 3.11+. `linkedparticles-core`, the store-free Client layer, is pulled
+in automatically.
+
+## Sixty seconds
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+particles db init
+
+# Deposit a source (file, URL, or literal text) into the append-only corpus
+particles deposit https://en.wikipedia.org/wiki/Pluto
+
+# Extract claim-granularity beliefs with confidence, subjects, and provenance
+particles extract --all-pending
+
+# Ask in natural language; the answer cites the particles behind it
+particles query "Why is Pluto not a planet?"
+
+# Health-check the store: contradictions, staleness, orphaned links
+particles lint
+```
+
+Because nothing is overwritten, the store can replay its own history. `--as-of`
+is the **assertion-time** lens. It answers *what did the store believe at T, and
+why did it stop*, not what was true of the world at T:
+
+```bash
+particles query "How many planets are in the Solar System?" --as-of 2000-01-01
+# → the belief as it stood then, and why it no longer stands: SUPERSEDED,
+#   retired 2006-08-24, superseded by "Pluto is a dwarf planet."
+```
+
+There is a clickable version of that belief history on the
+[front door](https://linkedparticles.org). Step by step:
+[getting started](https://docs.linkedparticles.org/user-guide/getting-started/),
+then [as-of time travel](https://docs.linkedparticles.org/user-guide/as-of/).
+
+### Wiring it into an agent
+
+```bash
+particles mcp serve                 # read-only MCP server (stdio), any MCP client
+particles init claude-code          # session-start digest, session-end harvest
+particles engine serve              # HTTP API, for a shared or remote engine
+particles export obsidian ./vault   # also: anki, wiki, logseq, notion, graph, jsonl
+```
+
+It can also stand in for the reference memory server, and it plugs into
+LangChain as a retriever and a tool set. See
+[Claude Code memory](https://docs.linkedparticles.org/user-guide/claude-code/),
+[swapping in for the reference memory server](https://docs.linkedparticles.org/user-guide/memory-server-swap/),
+and [using from LangChain](https://docs.linkedparticles.org/user-guide/integrations/).
+
+## Does it hold up?
+
+Measured on **LongMemEval**, the multi-session agent-memory benchmark, with a
+full-context oracle and a no-memory floor run as controls in the same harness:
+
+| | Recall@40 | End-to-end QA |
+|---|---:|---:|
+| **Particles memory** (shipped default, `top_k` 40) | **0.969** | **0.804** |
+| Full-context oracle (baseline, n=148) | — | 0.878 |
+| No memory (floor) | — | 0.047 |
+
+Method, per-question-type breakdowns, comparator memories, and the
+budget-matched arm (including the arms where Particles *loses*) are on the
+[benchmarks page](https://docs.linkedparticles.org/benchmarks/), with the
+report JSONs of record beside them.
+
+## Security posture
+
+A memory layer is an *input-shaping* attack surface: it does not merely hold
+data, it shapes what the agent believes and does next, so a poisoned claim is
+an instruction on a delay timer. Before this code was opened we ran an
+adversarial application-security audit over the whole package: the HTTP, CLI,
+and MCP surfaces, the filesystem writers, the egress layer, and the build
+pipeline. The verdict, verbatim: **GO-WITH-FIXES**, with **33 findings (2
+High, 7 Medium, 20 Low, 4 Info)**. The ranked must-fix set merged the following
+day; the last open finding closed in `v1.128.0`.
+
+What that bought, and what it did not:
+
+- **Prompt injection is contained, not eliminated.** Every LLM call site that
+  touches attacker-controllable text keeps trusted instructions in the system
+  turn and wraps the untrusted material in a per-call, 128-bit-nonce data
+  fence, with structural backstops behind it: a JSON contract enforced at the
+  parser, a citation-id membership gate on synthesized prose. This raises the
+  bar materially. It is hardening, not immunity.
+- **The model is never given tools, and no model output is executed.** No
+  function-calling; nothing the model emits becomes a shell command, a SQL
+  fragment, or a fetch. An injection can at worst distort *claims*, never
+  trigger *actions*.
+- **Outbound fetches are validated per hop.** The host is resolved, the address
+  is checked against a blocklist, and the connection is made to *that vetted
+  address*, re-resolved and re-validated on every redirect. That closes DNS
+  rebinding and redirect SSRF, not just the first lookup. The two fetches that
+  run as subprocesses reach the same guarantee by pinning to addresses this
+  process vetted.
+- **Fail-closed auth, and no raw SQL.** The API refuses to boot on a
+  non-loopback bind without a real bearer key, and the token comparison is
+  constant-time. Every query in the data layer is typed ORM with bound
+  parameters: no `text()`, no string-built SQL, no dynamic `ORDER BY`.
+- **Local-first by default.** Loopback bind, a local SQLite file, telemetry
+  off, and the MCP server as a locally-spawned stdio child rather than a
+  network service. The only outbound traffic in the default posture is what you
+  configured.
+
+The real answer to the residual is epistemic rather than technical: every
+particle carries its provenance, source trust is a read-time lens that
+discounts a distrusted source without rewriting anything, and `lint` / `review`
+turn your rulings on contradictions into a reusable trust policy. The store is
+a record of *what sources said, weighted by how much you trust them*, not an
+oracle, so a poisoned source is something you can see, discount, and retract
+with the audit trail intact.
+
+The limitations we ask you to read before relying on any of this are in
+[SECURITY.md](https://github.com/LinkedParticles/particles-engine-py/blob/main/SECURITY.md):
+the unauthenticated read surface, verbatim storage of whatever you deposit
+(secrets included), the single-operator trust model, and the MCP write
+boundary. SECURITY.md is also where to report a vulnerability; please report
+privately.
+
+## Why Particles?
+
+<!-- BEGIN PROJECTED: what-is (manifest: docs/projection/readme.yaml) -->
+Most Retrieval-Augmented Generation systems re-derive knowledge from scratch on every query, operating over raw text chunks without any persistent record of what was learned, how confidently it was held, or where it came from. Particles takes a different approach: rather than retrieving passages, it stores discrete, citable, revisable claims (natural-language sentences paired with structured metadata for confidence, provenance, and uncertainty) so that an agent's knowledge accumulates as an auditable ledger rather than evaporating between sessions. Corpus-entry-level provenance is required at Core, meaning every belief can be traced back to the source that produced it, and the particle store itself is a derived view that can be rebuilt from the corpus by re-running extractors, giving the system both durability and reproducibility.
+
+The Particles standard defines the particle schema, the source-corpus model, the extraction protocol, and a set of operations (deposit, extract, query, lint, review, and reindex) that together form the spine along which an agent's knowledge is built, queried, audited, and revised one belief at a time. A developer working with this loop deposits a source into the append-only corpus, extracts claim-granularity particles with resolved subjects, queries by effective-confidence ranking, lints for contradictions and staleness, and reviews inconsistencies into a reusable trust policy. Because particles carry explicit confidence and provenance, Particles can answer queries that require finding relevant claims about a subject and synthesising them with provenance intact, rather than returning a ranked list of chunks whose epistemic status is opaque.
+
+Particles is a minimal interoperable substrate for claim-granularity agent knowledge: narrower than a formal-ontology knowledge graph such as RDF/SPARQL and more structured than augmented markdown. Anchoring every particle to a subject is what makes this precision possible: the Subject store is the standard's catalogue of real-world entities against which particles are indexed, aligning beliefs to canonical entities that can, in principle, be mapped to external ontologies. A developer choosing Particles is therefore choosing a substrate where each belief is a first-class object with an identity, a confidence score, a source, and a subject, not a position in a vector space.
+
+<!-- sources: p-21f11766, p-3f71ca93, p-7931228a, p-80f9b726, p-a5a42fe0, p-ba8cfa2c, p-cb10c54c, p-d3fd6c56 -->
+<!-- END PROJECTED: what-is -->
+
+## Design rationale
+
+<!-- BEGIN PROJECTED: design-rationale (manifest: docs/projection/readme.yaml) -->
+The foundation of Particles is the guarantee that nothing already written is ever overwritten. The Source Corpus is constituted as an append-only archive of all materials from which particles have been or may be extracted, meaning that every ingested source and every derived claim accumulates in place rather than displacing what came before. Provenance is carried by each particle from the moment of its creation, and when a source must be handled in a mutable mode rather than a purely append-only one, the system does not silently absorb the change: particles whose provenance location falls within the changed regions are flagged with `PROVENANCE_STALE`, making the disturbance visible rather than burying it. Supersession and retraction therefore propagate deterministically through the provenance graph, touching every claim whose origin is implicated, so the ledger remains a faithful record of what was asserted and when, never a quietly revised version of the past.
+
+Trust in Particles is not a property baked into a stored record but a lens applied at the moment of reading. The raw confidence value is stored immutably, calibrated at the time a particle is created and never subsequently altered; what changes with each query is the effective confidence, which is computed at query time rather than stored. Trust weighting, recency decay, and the as-of instant are all applied per observer at query time, never written back into the record. This means that a belief does not need to be retracted or rewritten to sink in the rankings as its source ages or as the extractor that produced it loses credibility: the stored particle remains intact while the effective confidence computed over it falls, allowing stale beliefs to recede naturally without any mutation of the underlying ledger. When `as_of` is set to a past instant T, the query evaluates the store on the assertion-time axis (what the store believed at T) rather than on any world-time validity dimension, so the same immutable corpus can answer questions about present belief and historical belief without ambiguity.
+
+Contradictions in Particles are treated as first-class citizens of the knowledge graph rather than anomalies to be suppressed. When two sources or two principals disagree, the conflict-resolution ladder surfaces an `INCONSISTENCY` record for review instead of silently overwriting either side, preserving both competing claims in the ledger and making the disagreement legible to any downstream consumer. Contestedness, the property that marks a particle as being in tension with one or more others, must be computed at read time and is never stored; like effective confidence, it is a read-time lens rather than a durable annotation, which means that as new particles arrive or old ones are superseded, the contested status of any given claim is always freshly derived from the current state of the graph rather than from a stale flag written at some earlier moment.
+
+Memory in Particles maintains itself through a periodic, automated health discipline. The Lint operation performs health checks for contradictions, stale claims, orphan pages, and missing cross-references, scanning the corpus on a scheduled basis so that problems accumulate into visible findings rather than silently degrading the quality of the belief store. Because nothing is overwritten and contradictions are surfaced rather than resolved by fiat, Lint operates on a stable substrate: it traverses the same immutable record that queries read, and its findings feed back into the trust-policy review cycle, closing the loop between ingestion, querying, and the ongoing stewardship of the ledger's integrity.
+
+<!-- sources: p-0296afc0, p-8476c2ac, p-8767ad66, p-9cf41a46, p-c3b1cef6, p-cea0186c, p-d0d41702, p-f1ced6c4, p-f35e1dee -->
+<!-- END PROJECTED: design-rationale -->
+
+## Architecture
+
+<!-- BEGIN PROJECTED: architecture (manifest: docs/projection/readme.yaml) -->
+A Particles deployment is organized around two distinct storage subsystems: a Source Corpus store and a Particle store. The Source Corpus store is an append-only object store, compatible with a local filesystem, an S3-compatible backend, or a content-addressed store, while corpus entries and snapshot metadata are held in queryable form, either relational or document store. Together these two subsystems form the durable foundation on which all operations act.
+
+The Core SDK exposes both a clean Python API and a CLI, and the six core operations (Deposit, Extract, Query, Lint, Reindex, and Review) each map to a CLI subcommand and a `POST` API endpoint in the reference SDK. A developer therefore has two symmetric surfaces for every operation: a scriptable command line suitable for pipelines and a programmatic endpoint suitable for application integration.
+
+The journey from a raw source to a set of particles begins with Deposit, which writes the source into the append-only corpus, and continues with Extract, which drives the extraction pipeline. That pipeline selects the most specific registered extractor whose applicability specification matches the deposited source; the general extractor is used if and only if no domain-specific extractor applies. Once particles exist in the Particle store, Query retrieves them ranked by effective confidence, Lint checks for contradictions and staleness, Reindex rebuilds derived indexes, and Review consolidates inconsistencies into a reusable trust policy. Each step in this sequence is a discrete, named operation rather than an implicit side effect, which means a developer can invoke any stage independently and observe its inputs and outputs in isolation.
+
+<!-- sources: p-3bd3740c, p-5a941962, p-8f670063, p-a4bdde07, p-ac28fba0, p-cb1cd190, p-e5639e78 -->
+<!-- END PROJECTED: architecture -->
+
+> The *Why Particles?*, *Design rationale*, and *Architecture* sections are
+> **cited projections of a particle store**, not hand-authored prose. Each
+> marked block is rendered from
+> [`docs/projection/readme.yaml`](https://github.com/LinkedParticles/particles-engine-py/blob/main/docs/projection/readme.yaml)
+> and the corpus bundle committed beside it, and the trailer under each block
+> names the exact claims it was built from. Edit the manifest or the underlying
+> particles and re-render, never the prose between the sentinels. Check it
+> yourself: `python scripts/projection_drift.py` restores the bundle into a
+> throwaway store, re-derives the deterministic render, and fails if any block
+> has drifted from the claims it cites.
+>
+> This is the project eating its own dogfood: the README argues for
+> claim-granularity knowledge with provenance, and is itself assembled that way.
+
+## Documentation
+
+Everything below is served at
+**[docs.linkedparticles.org](https://docs.linkedparticles.org)**.
+
+| Guide | What it covers |
+|---|---|
+| [User guide](https://docs.linkedparticles.org/user-guide/) | Depositing, extracting, querying, as-of time travel, exporting, agent integrations |
+| [Operator guide](https://docs.linkedparticles.org/operator-guide/) | Configuration, tuning, lint and review, remote engines, containers, observability, troubleshooting |
+| [Plugin-author guide](https://docs.linkedparticles.org/plugin-author-guide/) | Writing extractors, exporters, and benchmark suites |
+| [CLI reference](https://docs.linkedparticles.org/cli-reference/) | Every verb and flag |
+| [HTTP API](https://docs.linkedparticles.org/api/http/) | The OpenAPI contract |
+| [Benchmarks](https://docs.linkedparticles.org/benchmarks/) | LongMemEval results, method, and comparators |
+
+The standard itself (whitepaper, technical specification, and the normative
+schema, context, and vocabulary artifacts) lives at
+**[linkedparticles.org](https://linkedparticles.org)**.
+
+## The three repositories
+
+| Repo | What it is |
+|---|---|
+| [`particles-standard`](https://github.com/LinkedParticles/particles-standard) | The standard: whitepaper, technical specification, normative schema + SHACL artifacts, conformance fixtures |
+| [`particles-core-py`](https://github.com/LinkedParticles/particles-core-py) | The Python Client layer ([`linkedparticles-core`](https://pypi.org/project/linkedparticles-core/)) |
+| [`particles-engine-py`](https://github.com/LinkedParticles/particles-engine-py) | **This repo**: the Python Engine layer + surfaces (`linkedparticles`) |
+
+## Deployment
+
+Container and chart artifacts live under
+[`deploy/`](https://github.com/LinkedParticles/particles-engine-py/tree/main/deploy).
+The engine ships as a single-writer service; see
+[running in a container](https://docs.linkedparticles.org/operator-guide/container-deployment/).
+
+## Contributing
+
+See
+[CONTRIBUTING.md](https://github.com/LinkedParticles/particles-engine-py/blob/main/CONTRIBUTING.md)
+and
+[ARCHITECTURE.md](https://github.com/LinkedParticles/particles-engine-py/blob/main/ARCHITECTURE.md).
+Contributions are accepted under a Developer Certificate of Origin sign-off;
+there is no CLA.
+
+## License
+
+Apache-2.0. See
+[LICENSE](https://github.com/LinkedParticles/particles-engine-py/blob/main/LICENSE)
+and
+[NOTICE](https://github.com/LinkedParticles/particles-engine-py/blob/main/NOTICE).

linkedparticles-1.172.0/alembic/versions/040_backfill_particle_tag_edges.py

@@ -0,0 +1,88 @@
+# SPDX-FileCopyrightText: 2026 The Particles authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Backfill ``particle_tag_edges`` from ``particles.tags_json``.
+
+A particle's tags live in two places: ``particles.tags_json`` (the canonical
+list, round-tripped to ``Particle.tags``) and ``particle_tag_edges`` (one row
+per tag, the index every tag-filtered read joins on). Until this release
+``insert_particle`` wrote only the JSON column; the edge rows came only from
+``set_particle_tags`` / ``add_particle_tags``. A particle born with tags (an
+agent assertion with ``tags``, a supersession or subject-assign successor
+inheriting its predecessor's tags, a conversation deposit) was therefore
+invisible to every tag-filtered query.
+
+**The backfill.** For every particle with a non-empty ``tags_json``, insert the
+edge rows it is missing. Rows already present are left alone, so the migration
+is idempotent and safe on a store where some particles were retagged by hand.
+Malformed JSON is skipped, not repaired: the column is read as-is by
+``ParticleRow.to_model``, and this migration does not change what a particle
+says about itself.
+
+``downgrade()`` is a deliberate no-op. A backfilled edge cannot be told apart
+from one ``set_particle_tags`` wrote, and each one agrees with the particle's
+own tag list, so removing them would only reintroduce the defect.
+
+Revision ID: 040
+Revises: 039
+"""
+
+from __future__ import annotations
+
+import json
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision = "040"
+down_revision = "039"
+branch_labels = None
+depends_on = None
+
+
+def backfill_tag_edges(bind: sa.engine.Connection) -> int:
+    """Insert the edge rows each tagged particle is missing.
+
+    Takes the connection rather than calling ``op.get_bind()`` so the backfill
+    is exercisable outside an Alembic context, as in migration 037. See
+    ``tests/test_migration_040_tag_edges.py``.
+
+    Returns the number of edge rows inserted.
+    """
+    existing = {
+        (pid, tag)
+        for pid, tag in bind.execute(
+            sa.text("SELECT particle_id, tag FROM particle_tag_edges")
+        ).fetchall()
+    }
+    rows = bind.execute(
+        sa.text("SELECT id, tags_json FROM particles WHERE tags_json IS NOT NULL")
+    ).fetchall()
+    missing: list[dict[str, str]] = []
+    for pid, tags_json in rows:
+        try:
+            tags = json.loads(tags_json)
+        except (TypeError, ValueError):
+            continue
+        if not isinstance(tags, list):
+            continue
+        for tag in dict.fromkeys(t for t in tags if isinstance(t, str) and t):
+            if (pid, tag) not in existing:
+                existing.add((pid, tag))
+                missing.append({"pid": pid, "tag": tag})
+    if missing:
+        bind.execute(
+            sa.text("INSERT INTO particle_tag_edges (particle_id, tag) VALUES (:pid, :tag)"),
+            missing,
+        )
+    return len(missing)
+
+
+def upgrade() -> None:
+    backfill_tag_edges(op.get_bind())
+
+
+def downgrade() -> None:
+    pass

linkedparticles-1.172.0/alembic/versions/041_snapshot_extraction_attempts.py

@@ -0,0 +1,57 @@
+# SPDX-FileCopyrightText: 2026 The Particles authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Count extraction claims per snapshot.
+
+One non-null integer column on ``snapshots``:
+
+* ``extraction_attempts`` — incremented each time extraction claims the
+  snapshot (the IN_PROGRESS write), never reset. The consolidation extract
+  catch-up orders its PENDING queue on it, least-tried first,
+  then by capture time.
+
+Why a column. The pipeline hands a snapshot back to PENDING when every LLM
+call for it failed, which is right for a transient outage and wrong for a
+failure that recurs: a reply cut at the output budget, or a pooled batch that
+outlives ``llm.batch.max_wait_seconds``, fails the same way the next night.
+Ordered on capture time alone, those snapshots were first in line every night.
+Measured on the owner's store 2026-09-25: the same six transcript snapshots,
+captured 2026-07-19 to 2026-08-08, took most of the per-run cap on every run
+since mid-September while 337 PENDING snapshots, every memory file written in
+September among them, waited behind them. Nothing on the row recorded that a
+snapshot had been tried, so no ordering could route around it.
+
+**No backfill.** Every existing row starts at 0. A snapshot that was already
+failing repeatedly is tried once more on the first run after this revision,
+and from then on sorts behind every snapshot not yet tried.
+
+Revision ID: 041
+Revises: 040
+"""
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision = "041"
+down_revision = "040"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("snapshots") as batch:
+        batch.add_column(
+            sa.Column(
+                "extraction_attempts",
+                sa.Integer(),
+                nullable=False,
+                server_default="0",
+            )
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("snapshots") as batch:
+        batch.drop_column("extraction_attempts")

linkedparticles-1.172.0/alembic/versions/042_born_retired.py

@@ -0,0 +1,140 @@
+# SPDX-FileCopyrightText: 2026 The Particles authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Mark rows born retired, so the as-of lens never reads them as beliefs.
+
+One non-null boolean column on ``particles``:
+
+* ``born_retired`` — true for a row inserted in a status other than
+  ``ACTIVE``: an INCONSISTENCY record, or the losing candidate of a §6.6
+  INCONSISTENT verdict persisted ``PROVENANCE_STALE`` / ``CONFLICT_PENDING``.
+  ``insert_particle`` stamps it from this revision on.
+
+Why a column. The as-of lens hid these rows by their
+current status and reason (``INCONSISTENCY``, ``CONFLICT_PENDING``), and
+resolving a conflict changes both. The loser moves to ``CONFLICT_RESOLVED``
+(PREFER_A, the trust cascade), ``SUPERSEDED`` (promotion) or ``RETRACTED``
+(DISCARD); the record moves to ``RETRACTED`` or ``PROVENANCE_STALE``. Each then
+looked like a retired belief, dated from the review, so an as-of query between
+its birth and the review returned a claim the store never held. Nothing else
+on the row survives the resolution: ``retired_at`` stays NULL (the row never
+left ACTIVE), but it is NULL on pre-029 retirements too.
+
+**The backfill** is exact on every store this SDK wrote. It marks:
+
+1. every row still ``INCONSISTENCY`` or ``CONFLICT_PENDING``;
+2. every resolved INCONSISTENCY record: content opening ``INCONSISTENCY: ``
+   (the one template prefix every version has written), not ACTIVE, no
+   ``retired_at``;
+3. every row an INCONSISTENCY record, open or resolved, names as its B claim
+   (the second PARTICLE provenance ref), when the row exists, is not ACTIVE,
+   and has no ``retired_at``. Since the only writer of an
+   INCONSISTENCY record is the quarantine plan, and its B is the quarantined
+   candidate; before the B ref was a dangling id, so it matches no
+   row.
+
+The ``retired_at`` guard skips any row stamped on leaving ACTIVE, which a
+born-retired row never is.
+
+``downgrade()`` drops the column.
+
+Revision ID: 042
+Revises: 041
+"""
+
+from __future__ import annotations
+
+import json
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision = "042"
+down_revision = "041"
+branch_labels = None
+depends_on = None
+
+#: The content prefix of every INCONSISTENCY record template this SDK has
+#: written (``core.conflict_resolution.build_inconsistency_particle``).
+_RECORD_PREFIX = "INCONSISTENCY: "
+
+
+def _b_claim_id(provenance_json: str | None) -> str | None:
+    """The B claim an INCONSISTENCY record names: its second PARTICLE ref."""
+    try:
+        refs = json.loads(provenance_json or "[]")
+    except (TypeError, ValueError):
+        return None
+    if not isinstance(refs, list):
+        return None
+    particle_refs = [
+        r.get("corpus_entry_id")
+        for r in refs
+        if isinstance(r, dict) and r.get("type") == "PARTICLE"
+    ]
+    b = particle_refs[1] if len(particle_refs) > 1 else None
+    return b if isinstance(b, str) else None
+
+
+def backfill_born_retired(bind: sa.engine.Connection) -> int:
+    """Set ``born_retired`` on every born-retired row already stored.
+
+    Takes the connection rather than calling ``op.get_bind()`` so the backfill
+    is exercisable outside an Alembic context, as in migration 040. See
+    ``tests/test_migration_042_born_retired.py``.
+
+    Returns the number of rows marked.
+    """
+    marked = {
+        pid
+        for (pid,) in bind.execute(
+            sa.text(
+                "SELECT id FROM particles"
+                " WHERE status = 'INCONSISTENCY' OR status_reason = 'CONFLICT_PENDING'"
+            )
+        ).fetchall()
+    }
+    # Rows that left no trace of ever being believed: not ACTIVE, never stamped.
+    unstamped = bind.execute(
+        sa.text(
+            "SELECT id, content, provenance_json FROM particles"
+            " WHERE status != 'ACTIVE' AND retired_at IS NULL"
+        )
+    ).fetchall()
+    unstamped_ids = {pid for pid, _content, _prov in unstamped}
+    records = [
+        (pid, prov) for pid, content, prov in unstamped if content.startswith(_RECORD_PREFIX)
+    ]
+    marked.update(pid for pid, _prov in records)
+    open_records = bind.execute(
+        sa.text("SELECT provenance_json FROM particles WHERE status = 'INCONSISTENCY'")
+    ).fetchall()
+    record_provenance = [prov for (prov,) in open_records] + [prov for _pid, prov in records]
+    b_ids = {b for prov in record_provenance if (b := _b_claim_id(prov)) is not None}
+    marked |= b_ids & unstamped_ids
+    if marked:
+        bind.execute(
+            sa.text("UPDATE particles SET born_retired = :t WHERE id = :pid"),
+            [{"t": True, "pid": pid} for pid in sorted(marked)],
+        )
+    return len(marked)
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("particles") as batch:
+        batch.add_column(
+            sa.Column(
+                "born_retired",
+                sa.Boolean(),
+                nullable=False,
+                server_default=sa.false(),
+            )
+        )
+    backfill_born_retired(op.get_bind())
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("particles") as batch:
+        batch.drop_column("born_retired")

linkedparticles-1.172.0/alembic/versions/043_snapshot_extracted_through.py

@@ -0,0 +1,45 @@
+# SPDX-FileCopyrightText: 2026 The Particles authors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Record how far extraction read each snapshot.
+
+One nullable integer column on ``snapshots``:
+
+* ``extracted_through`` — the offset, in bytes of the raw content, up to which
+  extraction read the snapshot. The content's length when the read reached the
+  end; the largest paragraph boundary whose decoded prefix was read in full
+  when the read was cut at ``extraction.max_llm_calls_per_source``. Written in
+  the transaction that marks the snapshot COMPLETE after an extraction.
+
+Why a column. An APPEND_ONLY entry's snapshot is extracted as a delta from the
+last snapshot the store extracted, and the delta starts where that read
+stopped. Recording the stop is what lets text a capped read skipped be read on
+the next pass instead of never.
+
+**No backfill.** Every existing row stays NULL. The pipeline derives the offset
+of such a base from the chunk hashes its entry's claims carry the first time it
+is used as one, and otherwise counts it as fully extracted.
+
+Revision ID: 043
+Revises: 042
+"""
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision = "043"
+down_revision = "042"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("snapshots") as batch:
+        batch.add_column(sa.Column("extracted_through", sa.Integer(), nullable=True))
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("snapshots") as batch:
+        batch.drop_column("extracted_through")