tether-vcs 0.1.0__tar.gz

tether_vcs-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

tether_vcs-0.1.0/PKG-INFO

@@ -0,0 +1,398 @@
+Metadata-Version: 2.4
+Name: tether-vcs
+Version: 0.1.0
+Summary: jj-style version control for heterogeneous datasets (files/object stores, Icechunk, Neon, Iceberg, Delta, Lance, lakeFS, DuckLake, Dolt, git)
+Keywords: version-control,data-versioning,icechunk,iceberg,neon,delta-lake,lance,lakefs,ducklake,dolt,jujutsu
+Author: tether contributors
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Version Control
+Classifier: Typing :: Typed
+Requires-Dist: tomlkit>=0.12
+Requires-Dist: typer>=0.12 ; extra == 'all'
+Requires-Dist: obstore>=0.11 ; extra == 'all'
+Requires-Dist: icechunk>=1.0 ; extra == 'all'
+Requires-Dist: httpx>=0.27 ; extra == 'all'
+Requires-Dist: psycopg[binary]>=3.1 ; extra == 'all'
+Requires-Dist: pyiceberg>=0.10 ; extra == 'all'
+Requires-Dist: pyarrow>=15 ; extra == 'all'
+Requires-Dist: deltalake>=1.0 ; extra == 'all'
+Requires-Dist: pylance>=1.0 ; extra == 'all'
+Requires-Dist: lakefs>=0.8 ; extra == 'all'
+Requires-Dist: duckdb>=1.3 ; extra == 'all'
+Requires-Dist: pymysql>=1.1 ; extra == 'all'
+Requires-Dist: obstore>=0.11 ; extra == 'azure'
+Requires-Dist: typer>=0.12 ; extra == 'cli'
+Requires-Dist: deltalake>=1.0 ; extra == 'delta'
+Requires-Dist: pymysql>=1.1 ; extra == 'dolt'
+Requires-Dist: duckdb>=1.3 ; extra == 'ducklake'
+Requires-Dist: obstore>=0.11 ; extra == 'gcs'
+Requires-Dist: pyiceberg>=0.10 ; extra == 'iceberg'
+Requires-Dist: pyarrow>=15 ; extra == 'iceberg'
+Requires-Dist: icechunk>=1.0 ; extra == 'icechunk'
+Requires-Dist: lakefs>=0.8 ; extra == 'lakefs'
+Requires-Dist: pylance>=1.0 ; extra == 'lance'
+Requires-Dist: httpx>=0.27 ; extra == 'neon'
+Requires-Dist: psycopg[binary]>=3.1 ; extra == 'neon'
+Requires-Dist: obstore>=0.11 ; extra == 'objectstore'
+Requires-Dist: obstore>=0.11 ; extra == 's3'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://evanlyall.com/tether/
+Project-URL: Documentation, https://evanlyall.com/tether/
+Project-URL: Repository, https://github.com/elyall/tether
+Project-URL: Changelog, https://github.com/elyall/tether/blob/main/CHANGELOG.md
+Provides-Extra: all
+Provides-Extra: azure
+Provides-Extra: cli
+Provides-Extra: delta
+Provides-Extra: dolt
+Provides-Extra: ducklake
+Provides-Extra: gcs
+Provides-Extra: iceberg
+Provides-Extra: icechunk
+Provides-Extra: lakefs
+Provides-Extra: lance
+Provides-Extra: neon
+Provides-Extra: objectstore
+Provides-Extra: s3
+Description-Content-Type: text/markdown
+
+# tether
+
+**jj-style version control for heterogeneous datasets.**
+
+> Status: alpha. See the [changelog](https://github.com/elyall/tether/blob/main/CHANGELOG.md).
+>
+> State: vibe coded with Claude Fable 5.1. **USE AT YOUR OWN RISK.**
+
+`tether` ties heterogeneous data objects -- files and object-store prefixes (S3,
+GCS, Azure), [Icechunk](https://icechunk.io) repositories, [Neon](https://neon.com)
+Postgres databases, [Apache Iceberg](https://iceberg.apache.org) tables,
+[Delta Lake](https://delta.io) tables, [Lance](https://lance.org) datasets,
+[lakeFS](https://lakefs.io) repositories, [DuckLake](https://ducklake.select)
+catalogs, [Dolt](https://www.dolthub.com) databases, and git/[jj](https://jj-vcs.dev)
+repositories -- to the commits of an existing git or jj repository.
+
+A `tether commit` records each object's state and, for systems that support it,
+creates a durable native reference (an Icechunk tag, a Neon child branch, a git
+tag, an Iceberg tag, a Lance tag, a lakeFS tag, a Dolt tag) so the exact state
+can be recovered and branched from later.
+`tether new` forks fresh writable branches off any committed state. The VCS
+supplies history, undo, bookmarks, workspaces, and sharing; `tether` only
+implements what is novel: cross-system fan-out, drift detection, and the
+pin/fork lifecycle.
+
+Think of it as **DVC for *branchable* systems**: like DVC it commits small
+manifests into your git/jj repo, but where DVC only fingerprints files, tether
+also *pins* and *forks* live systems.
+
+Documentation: <https://evanlyall.com/tether/> -- user guide (getting
+started, concepts, pinning, branching and writing, CLI, configuration,
+backends, writing a backend) plus the generated API and CLI reference. The
+guide sources live in [`user_guide/`](https://github.com/elyall/tether/tree/main/user_guide).
+
+## Why this exists (prior art)
+
+Every existing tool versions a single layer:
+
+| Tool | Scope | Relationship to tether |
+| --- | --- | --- |
+| lakeFS, Quilt, Oxen, DataChain | objects / files | analog of our `file` objects; lakeFS is also a backend |
+| **DVC** (lakeFS-owned) | files/objects in git | closest structural analog; no pin/fork of live systems |
+| Dolt, pgGit, Neon, Databricks Lakebase | one database | vendor-bound; no external-object pins; Dolt and Neon are backends |
+| Nessie, Bauplan | Iceberg catalog branching | Iceberg-only |
+| Icechunk, Lance, Delta, DuckLake | one dataset / table / catalog | we use them as backends |
+| [Yggdrasil](https://github.com/replikativ/yggdrasil) (replikativ) | cross-system: Clojure protocol stack (snapshot / branch / merge / watch) over Git, ZFS, Btrfs, IPFS, Iceberg, Datahike, lakeFS, Dolt, Podman, with an HLC-coordinated workspace | the closest conceptual sibling; not adoptable from Python (JVM library; its own README marks the Python binding as unmaintained since the initial release) |
+| Dagster observable assets | staleness detection | analog of our snapshot/drift step; no branching |
+
+Nothing provides unified version control *across* files + Icechunk + Postgres +
+Iceberg with pinning and forking. So tether borrows jj's working-copy model
+(fingerprint, snapshot on every command, stale-working-copy detection), DVC's
+manifests-in-VCS layout, and [Yggdrasil](https://github.com/replikativ/yggdrasil)'s
+observe-then-record shape (a workspace that watches independent systems and
+records their snapshots, rather than a store that holds the data).
+
+## Install
+
+```bash
+pip install tether-vcs[cli]                 # core + CLI
+pip install tether-vcs[cli,icechunk,neon]   # add backends you need
+pip install tether-vcs[all]                 # everything
+```
+
+The distribution is `tether-vcs` (PyPI reserves the bare name); the package you
+import and the command you run are both `tether`.
+
+Extras: `cli`, `objectstore` (S3/GCS/Azure for `file`; `s3`/`gcs`/`azure` are
+aliases), `icechunk`, `neon`, `iceberg`, `delta`, `lance`, `lakefs`, `ducklake`,
+`dolt`, `all`. `git`/`jj` must be on `PATH`.
+
+## Quickstart
+
+```bash
+tether init
+tether add zarr/imaging --kind icechunk s3://bucket/imaging.zarr.icechunk --write fork
+tether add db/rosebud   --kind neon --project-id prj-123 --database neondb --role runner
+tether add raw/plate1   --kind file s3://bucket/raw/plate1/     # Observed unless versioned
+tether add raw/manifest --kind file gs://bucket/manifest.csv --file versioned   # Addressable
+tether add features     --kind lance s3://bucket/features.lance
+tether add events       --kind delta s3://bucket/events        # Addressable (retention-bound)
+tether add lake         --kind lakefs --repository analytics --branch main --prefix raw/
+tether add warehouse    --kind ducklake ducklake:postgres:dbname=lake --table events
+tether add ledger       --kind dolt --host dolt.internal --database ledger --branch main
+tether add code         --kind git . --remote origin
+
+tether status                       # fan-out: modified / unpinned / drifted per object
+tether commit -m "Baseline imaging + metrics"   # pins, writes manifests, jj/git commit
+tether new main                     # fork fresh writable branches off main's pins
+tether open db/rosebud              # -> postgresql://...tether.ws.ab12cd34...
+tether open zarr/imaging -r main    # read-only handle at main's pinned tag
+tether diff main @ --content         # what changed inside each object, natively
+tether verify --all-history --deep
+tether gc --dry-run
+```
+
+Python:
+
+```python
+from tether import Repo
+
+repo = Repo.find(".")
+repo.status()
+repo.commit("baseline")
+repo.new("main")
+handle = repo.open("zarr/imaging")  # writable IcechunkHandle
+session = handle.session  # native icechunk session
+ro = repo.open("zarr/imaging", rev="main")  # read-only at the pinned tag
+```
+
+Set `TETHER_REV=<vcs rev>` and `repo.open(key)` defaults to a read-only handle at
+that commit's pinned state -- convenient for downstream, reproducible reads.
+
+## Concept map (jj / DVC -> tether)
+
+| jj / DVC | tether |
+| --- | --- |
+| working copy | the VCS working dir; the committed thing is the manifests |
+| `TreeState.snapshot()` | **snapshot**: concurrent fan-out fingerprint -> `workspace.toml` |
+| `commit` | **pin** fan-out + write state into manifests + `jj/git commit` |
+| `new <rev>` | checkout + **fork** fan-out (fresh writable branch per object) |
+| stale working copy | manifest hash recorded at fork; differs from HEAD => refuse writes |
+| bookmarks / op log / undo / workspaces / push | delegated to the VCS |
+| `.dvc` files | `.tether/objects/<key>.toml` (committed) |
+| `jj diff` | **diff**: object-level manifest diff; `--content` asks each backend for its native diff (files, tables, arrays, fragments, commits) |
+| -- | **verify** (pins still resolve) and **gc** (drop unreferenced pins) |
+
+## Repository layout
+
+```
+<dataset-root>/
+  tether.toml                 # committed: snapshot.auto (CLI snapshots by default), verify.on_status,
+                              #   new.auto_fork (commit re-forks working refs), default policies,
+                              #   backend options, credential *references*
+  .tether/
+    .gitignore                # ignores workspace.toml
+    objects/<key>.toml        # committed, one per object: kind, locator, policy, state, pin
+    listings/<hash>.jsonl     # committed, content-addressed per-file listings for directory/prefix
+                              #   states (so `diff --content` can compare them); pruned by gc
+    workspace.toml            # untracked: workspace_id, base manifest hash, working refs, last snapshot
+```
+
+`pin_id = blake2b(kind, locator identity, state)[:12]`; the native ref is
+`tether.<pin_id>`. Identical state yields the same pin, so re-committing an
+unchanged object is a no-op and identical states dedupe to one pin.
+
+## Capability tiers
+
+Backends declare capabilities; commands degrade explicitly by tier.
+
+- **Observed** (`FINGERPRINT`): drift detection only. `commit` records state and
+  marks it `recoverable = false`. E.g. a local file by mtime/size.
+- **Addressable** (`+ ADDRESSABLE`): an immutable version id can be read later;
+  `open -r <rev>` works, nothing to create or GC. E.g. an S3 object in a
+  versioned bucket, a Delta table version.
+- **Pinnable** (`+ PIN`): can create/delete a durable, GC-proof ref; `commit`
+  pins, `verify`/`gc` apply. E.g. an Icechunk tag.
+- **Forkable** (`+ FORK`): a writable branch can be created off a pin; `new`
+  forks, `open` returns a writable handle. E.g. Icechunk, Neon, Iceberg, Lance,
+  lakeFS, Dolt, git/jj.
+
+Orthogonal flags: `CHEAP_FINGERPRINT`, `RETENTION_BOUND`, `NEEDS_QUIESCENCE`,
+`ATOMIC_REF`, `DIFF` (the backend can describe what changed between two
+recorded states; see [Content diffs](#content-diffs)).
+
+### Compatibility matrix
+
+| Backend | Max tier | State | Pin | Fork | Notable flags / caveats |
+| --- | --- | --- | --- | --- | --- |
+| `file` (local) | Observed | size, mtime_ns (or dir digest) | -- | -- | `CHEAP`; immutable-by-default, drift is an error |
+| `file` (S3 / GCS / Azure object) | Addressable | size, etag, version_id | -- | -- | `CHEAP`; `--file versioned` on a versioning-enabled bucket; one `HEAD` |
+| `file` (S3 / GCS / Azure prefix) | Observed | count, size, etag digest | -- | -- | `CHEAP`; one paged `LIST`, no per-object calls |
+| `icechunk` | Forkable | snapshot_id | tag | branch | `ATOMIC_REF`; tags immutable, excluded from expiry |
+| `neon` | Forkable | lsn, next_xid | protected child branch @ parent_lsn | child of pin | `NEEDS_QUIESCENCE`, `RETENTION_BOUND`; no merge/promote, leaf-only gc, quotas |
+| `git` / `jj` | Forkable | sha, change_id, dirty | tag (pushed if `remote`) | branch | `CHEAP`, `ATOMIC_REF`; local path only for now |
+| `iceberg` | Forkable | snapshot_id, metadata_location | tag (`native`) or recorded id (`record`) | branch | `RETENTION_BOUND`; `record` for S3 Tables (no native ref) |
+| `delta` | Addressable | version, table_id | -- | -- | `CHEAP`, `RETENTION_BOUND`; no native tags; `VACUUM`/log retention bound readability |
+| `lance` | Forkable | branch, version | tag on (branch, version) | branch | `ATOMIC_REF`; tagged versions exempt from cleanup; version numbers are branch-scoped |
+| `lakefs` | Forkable | commit_id (+ dirty) | tag | branch | `CHEAP`, `ATOMIC_REF`; repo-wide pins, `prefix` scopes the handle |
+| `ducklake` | Addressable | snapshot_id, snapshot_time_us | -- | -- | `CHEAP`, `RETENTION_BOUND`; catalog-wide snapshots; `ducklake_expire_snapshots` bounds readability |
+| `dolt` | Forkable | commit (+ dirty) | tag | branch | `ATOMIC_REF`; over MySQL protocol to `dolt sql-server`; handles are `db/ref` revision URLs |
+| `memory` | Forkable | snapshot_id | tag | branch | reference impl for tests |
+
+Command requirements: `status`/`snapshot`/`verify` need `FINGERPRINT`; `commit`
+pins `PIN` objects (records `ADDRESSABLE`, records + warns for Observed, or fails
+with `--strict`); `new` forks `FORK` objects; `gc` applies to `PIN` objects;
+`diff --content` applies to `DIFF` objects.
+
+### Content diffs
+
+`tether diff A B` compares manifests: which objects were added, removed, or
+changed (with pin ids). `--content` additionally asks each changed object's
+backend for its *native* diff, using metadata only (no data is read), run
+concurrently across objects:
+
+| Backend | Unit | Source | Example detail |
+| --- | --- | --- | --- |
+| `file` (dir / prefix) | files | two stored listings (`.tether/listings/`) | `modified a.tif +4.0 KiB` |
+| `file` (object) | objects | recorded size / etag / version | `etag a -> b, version v1 -> v2` |
+| `git` / `jj` | files | `git diff --name-status` + `--numstat` | `modified main.py +1 -1` |
+| `icechunk` | nodes | `Repository.diff(from, to)` | `modified /x 2 chunks` |
+| `iceberg` | snapshots | snapshot summaries along `b`'s ancestry to `a` | `append: +5 rows, +1 files` |
+| `delta` | commits | transaction-log `history()` in `(a, b]` | `WRITE: +2 rows, +1 files` |
+| `lance` | fragments | fragment ids / deletions / schema at both versions | `added fragment 1 +1 rows` |
+| `lakefs` | objects | server-side `Reference.diff`, scoped to `prefix` | `added raw/c.bin 1 B` |
+| `ducklake` | tables | snapshot `changes` + `ducklake_table_changes` | `modified main.t +5000 rows, -1 rows` |
+| `dolt` | tables | `dolt_diff_summary` + `dolt_diff_stat` | `modified t +6 rows` |
+| `neon` | -- | no native diff; not supported (yet) | |
+
+Directory and prefix states are digests, so the `file` backend hands the
+engine a per-file listing at commit time; it is stored content-addressed under
+`.tether/listings/` and committed with the manifests (the same trick as DVC's
+`.dir` files). Diffs of commits made before listings existed fall back to a
+count/size summary. `gc` removes listings no manifest references. Entries are
+capped at 2000 per object; counts are exact.
+
+## Writing a backend
+
+Implement the `ObjectBackend` protocol (`tether.backends.base`): declare
+`kind` and `capabilities`, then `identity`, `fingerprint`, `pin`, `unpin`,
+`list_pins`, `verify`, `fork`, `delete_working_ref`, and `open`. A backend whose
+tier varies per object (like `file`) can override
+`effective_capabilities(locator, policy)`. To support `diff --content`, add
+`DIFF` and implement `diff(locator, a, b, *, listings)` returning an
+`ObjectDiff`; if your state is a digest, also implement `listing(locator,
+state)` and the engine will store and hand back the text. Register it with
+`register_backend`.
+
+Validate it against the importable conformance suite:
+
+```python
+from tether.testing import run_conformance
+
+
+class MyHarness:
+    capabilities = ...  # optional; defaults to backend.capabilities
+    backend = MyBackend()
+
+    def new_object(self): ...  # create a fresh system, return a locator
+    def mutate(self, locator, working_ref): ...  # change current state
+
+
+run_conformance(MyHarness())  # runs only the tier-appropriate checks
+```
+
+## Caveats
+
+- **No cross-system atomicity.** A commit records each object's state within a
+  capture window; point-consistency requires quiesced writers (the Neon check
+  helps; `tether commit` refuses while writers are active unless `--force`).
+- **Pin-then-commit ordering.** Pins are created before the VCS commit; if the
+  commit never lands, pins leak until `gc` (which scans VCS history + the current
+  workspace). Re-run `tether verify` after merging manifests across branches.
+- **Neon lineage.** Pins are child branches, so a working branch can't be deleted
+  while its pins exist, `gc` only drops leaf pins, and fork-per-`new` deepens the
+  tree. Neon cannot merge/promote a child into `main`; use `--write track` on
+  `main` for production. `protected` needs a paid plan; branch quotas vary.
+- **Neon fingerprint noise/cost.** LSN moves without user writes (`next_xid` is
+  the real change signal); snapshot may wake a suspended compute. Use
+  `--no-snapshot` to skip.
+- **Files on unversioned storage** are Observed: detectably but not recoverably
+  drifted. Bucket/container versioning promotes single objects to Addressable;
+  prefixes stay Observed (tether does not record per-object version ids).
+  Object-store credentials come from the environment; `[backends.file]
+  storage_options` in `tether.toml` passes non-secret options (region,
+  endpoint, account name) to obstore.
+- **Iceberg catalog variance.** S3 Tables disables maintenance when user refs
+  exist; use `--pin record` there.
+- **Delta has no refs.** Versions are recorded, not pinned; `VACUUM` (7-day
+  default) and log retention (30 days) bound how long `open -r` works. A
+  recreated table (new metadata id) verifies as drifted, not re-addressed.
+- **Lance versions are branch-scoped.** State is `(branch, version)`; an
+  untouched fork reports its parent's address so no-op commits stay no-ops.
+  Lance refuses to delete a branch a tag references, so a pinned working branch
+  outlives its workspace until `gc` drops the tag (a re-fork picks `name.2`).
+- **lakeFS staging.** Uncommitted lakeFS changes are not in any commit; a dirty
+  branch is reported in `status` and refused at `commit`.
+- **DuckLake attachments.** A DuckDB-file metadata catalog can be attached only
+  once per process, so tether attaches per operation and a `DuckLakeHandle` owns
+  its attachment: `close()` it (or use it as a context manager) before the next
+  `status`, and don't hold your own writer connection open in the same process.
+  Postgres/SQLite/MySQL catalogs have no such limit. Snapshots are catalog-wide;
+  `table` only scopes the handle. Put `CREATE SECRET` statements in
+  `[backends.ducklake] init_sql` rather than credentials in the locator.
+- **Dolt working sets.** Uncommitted changes on a branch are not in any commit;
+  a dirty branch is reported and refused at `commit`. The password comes from
+  `$DOLT_PASSWORD` (configurable via `[backends.dolt] password_env`), never the
+  manifest. Requires a Dolt with the `dirty` column on `dolt_branches` for dirty
+  detection (older servers report clean).
+- **Secrets** are never stored in manifests -- only credential *references* (env
+  var / secret names).
+
+## Performance notes
+
+tether is an orchestrator; its cost is the network round-trips it fans out, not
+Python.
+
+- **Fan-out uses threads, not asyncio.** Every backend SDK tether wraps (obstore,
+  psycopg, pyiceberg, deltalake, lance, lakefs, duckdb, pymysql, `git`) is synchronous, the
+  per-object Python work is microseconds, and a thread pool gives the same
+  wall-clock concurrency without forcing an async protocol on backend authors.
+  `snapshot`, `verify`, and the fork step of `new` all run concurrently.
+- **History walks are batched.** `gc` and `verify --all-history` stream every
+  manifest at every commit through one `git cat-file --batch` process (jj repos
+  are git-backed, so the same plumbing applies), caching trees and blobs by
+  object id and verifying each distinct `(system, state, pin)` once. On a repo
+  with 200 commits and 5 objects this is ~20-40 ms end to end versus 6-12 s
+  with one process per read.
+- **Fingerprints are metadata-only.** Local trees use `os.scandir`
+  (~4 us/file); a remote object is one `HEAD`; a prefix is one paged `LIST`.
+- **Content diffs are metadata-only too.** Each backend reads manifests, logs,
+  or catalog tables -- never data -- so `diff --content` costs about one
+  fingerprint per changed object (two manifest reads for Icechunk/Lance, one log
+  walk for Delta, one catalog query for DuckLake, one prolly-tree diff for
+  Dolt). Two stored file listings diff in pure Python at ~50 ms per 100k paths.
+
+## Non-goals
+
+- Being in the data path (tether hands back native handles and steps aside).
+- Reimplementing history/undo/branching/sharing (that is the VCS's job).
+- Cross-system transactions or a query layer.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff format --check . && uv run ruff check .
+uv run ty check
+uv run pytest
+```
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE).