zesus 0.1.0a1__tar.gz

zesus-0.1.0a1/.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+*.png binary
+*.sqlite binary

zesus-0.1.0a1/.github/workflows/ci.yml

@@ -0,0 +1,18 @@
+name: CI
+on: [push, pull_request]
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python: ["3.10", "3.11", "3.12", "3.13"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check src tests
+      - run: pytest -q

zesus-0.1.0a1/.gitignore

@@ -0,0 +1,19 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+build/
+dist/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+# scan outputs and extracted data never belong in the repo
+*.sqlite
+*.sqlite-wal
+*.sqlite-shm
+*.log.jsonl
+*.img
+/out/
+/work/
+/dev/testimage.toml
+.claude/

zesus-0.1.0a1/AI_POLICY.md

@@ -0,0 +1,58 @@
+# AI use policy
+
+## How this project was made
+
+Zesus was written with [Claude Code](https://claude.com/claude-code), using Anthropic's
+Claude Opus 5.5 model, under the direction of a human maintainer. The maintainer:
+
+* set the goals;
+* made the design decisions;
+* tested the tool against a real damaged pool;
+* reviewed the results.
+
+Commits co-written with AI carry a `Co-Authored-By` trailer that says so.
+
+## AI-assisted contributions are welcome
+
+You may use AI tools to write code, tests or documentation for Zesus. We care about the
+quality of a contribution, not about which tools produced it.
+
+## Your contribution is your responsibility
+
+* **You are the author.** Whatever you submit, you are accountable for it: its correctness,
+  its licensing, and its consequences. "The AI wrote it" is not an explanation for a bug, a
+  hallucinated API, or a broken test. Low-quality contributions will be treated as
+  low-quality contributions, whatever tool produced them.
+* **Test it and review it yourself before you submit it.** Run the test suite. Exercise the
+  change for real, against an image where that makes sense. Read every line. If you cannot
+  explain what a change does and why it is correct, it is not ready.
+* **Check the facts.** Anything that claims something about ZFS on-disk structures,
+  filesystem formats or tool behaviour must be verified against the OpenZFS source,
+  published documentation, or a real image. Do not trust a model's description alone.
+  Recovery tools that act on invented facts destroy people's data, or give them false
+  confidence.
+
+## Commit messages and pull requests
+
+* **Descriptive and to the point.** Say what changed and why. Skip the boilerplate,
+  filler and marketing tone.
+* **Free of hallucinations.** Describe only what the change actually does, how you actually
+  tested it, and what you actually observed. Do not claim tests pass that you did not run,
+  cite issues that do not exist, or describe behaviour you did not verify.
+* **Disclose substantial AI help.** Add a trailer such as
+  `Co-Authored-By: <model or tool>` or `Assisted-by: <tool>` to the commit message. It is not
+  a penalty. It helps reviewers know where to look harder.
+
+## This project's own rules still apply
+
+AI assistance changes nothing in [CONTRIBUTING.md](CONTRIBUTING.md). In particular:
+
+* nothing may write to the evidence;
+* nothing may silently guess;
+* parsers must survive damaged input;
+* **never paste real case data (disk images, maps, file names, file contents) into an AI
+  service** that you do not control. It is someone's private data.
+
+Maintainers may ask how a change was produced and tested. They may decline contributions
+that show signs of unreviewed generation, such as code that does not match its description,
+tests that do not test anything, or confidently wrong explanations.

zesus-0.1.0a1/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 (unreleased)
+
+* First version: pool/label/uberblock discovery; pool history decoding; DSL traversal
+  across the whole uberblock ring; resumable raw-device carving; multi-generation zvol
+  reconstruction; data verification with fallback to older generations; GPT/MBR; ext4
+  inventory with virtual journal replay; extraction of volumes, partitions and files
+  with gap reports; local web UI.
+* `zesus send` and the web UI's "send" action: rsync recovered files back to a server, then
+  restore original names (including ones Windows can't store), symlinks, modes, times and
+  owners from the map.

zesus-0.1.0a1/CLAUDE.md

@@ -0,0 +1,109 @@
+# CLAUDE.md
+
+Guidance for Claude Code (and humans) working on **Zesus**, *ZFS Emergency Storage Undelete
+Support*. Zesus is a forensic scanner and extractor that recovers destroyed ZFS volumes and
+datasets, even after they have left the uberblock ring.
+
+## Non-negotiable rules
+
+1. **Never write to the evidence.**
+   * All source access goes through `zesus.io.RawSource` / `SliceSource` (opened `O_RDONLY`).
+   * `zesus.io.guard` installs an audit hook that raises `SourceWriteAttempt` on any write-mode
+     open, truncate, rename or remove of a protected path.
+   * Do not add write paths to `src/zesus/io/`. `tests/unit/test_guard.py` checks this
+     statically.
+   * Outputs (maps, extractions) must never resolve to the source.
+2. **Never guess silently.**
+   * Every recovered byte is checksum-verified against a parent block pointer.
+   * Anything inferred or unverified is labelled: statuses `ok_stale`, carved provenance,
+     `--include-unverified`.
+   * Gaps are zero- or pattern-filled *and* reported (`*.gaps.json`, ddrescue mapfile,
+     manifest).
+3. **Privacy.** Test images may contain someone's personal files. Inspect structure
+   (superblocks, directories, counts) freely, but do not print or read file *contents*
+   unless the owner asks. Validate data integrity with self-checking formats (archive CRCs,
+   checksums) instead.
+4. **Parsers must be damage-tolerant.** Truncated, zeroed or random input must not crash a
+   phase. Record a problem and continue.
+
+## Layout
+
+```
+src/zesus/
+  io/          read-only sources + write guard
+  zfs/         on-disk format: label/uberblock, nvlist (XDR + native), blkptr, checksum,
+               compress, dnode, objset, zap, dsl, history, zpl (ZFS filesystems), pool, reader
+  carve/       scanner (resumable raw carving) · classify · reconstruct (multi-generation
+               tree merge + orphan placement + root clustering) · verify · fastsum (numba)
+  volume/      LogicalVolume (gap-aware device over a reconstructed zvol) · Coverage
+  partitions/  GPT, MBR plugins
+  fs/          plugin API (api.py), registry, ext4 plugin (+ virtual JBD2 replay), identifiers
+  extract/     extraction engine (+ LocalNamer: collision-safe local names), sparse output,
+               send.py (stage → rsync → generated POSIX restore script)
+  scan/        pipeline (phase runner, resumable, downstream invalidation) · phases
+  map/         schema.sql, db.py (WAL, in-place migrations), codes.py (block statuses)
+  cli/         main (scan/info/report/web), extract_cmd (extract/ls), report, fullreport
+  web/         FastAPI + single-file vanilla JS UI (localhost only)
+dev/           imgtool.py (inspect an image), make_ext4_fixtures.sh
+tests/         unit (fixtures in tests/fixtures), integration (needs ZESUS_TEST_IMAGE)
+docs/          how-to-use.md (user recipes), architecture.md, map-format.md, writing-plugins.md
+```
+
+Scan phases, in order: `history → datasets → carve → reconstruct → verify → contents`.
+Re-running a phase resets every later phase except `carve`, which is additive and resumes
+by chunk.
+
+## Commands
+
+```bash
+pip install -e ".[dev,all]"
+pytest -q                                      # unit tests; ~1 s
+ZESUS_TEST_IMAGE=/path/pool.img pytest -q      # plus integration tests
+ruff check src tests dev examples              # must be clean
+python dev/imgtool.py --image IMG uberblocks   # also: labels, history, datasets, objset, zap, dva, bp, carve-sample
+wsl -e sh dev/make_ext4_fixtures.sh            # rebuild ext4 test images (needs e2fsprogs)
+```
+
+For fast iteration on a big image, use a small slice:
+`zesus scan IMG -o t.sqlite --phases history,datasets,reconstruct,verify,contents --limit-blocks 102400`.
+
+## Conventions and gotchas
+
+* **Map schema changes** need a migration in `MapDB._migrate()`. Rebuild tables with
+  "create new → copy → drop → rename" and `legacy_alter_table=ON`. A plain `RENAME` rewrites
+  other tables' foreign keys. `tests/unit/test_map_migration.py` covers this.
+* **The map is an index, not a data store.** Volumes store L1 candidate pointers per span
+  (`volume_spans`), not every L0 pointer. Derive L0 pointers through `LogicalVolume` or
+  `carve.verify.chosen_blocks`.
+* **Big volumes:** keep per-block work in numpy. A zvol can have 50M+ blocks, so avoid
+  Python dicts or loops per block (see `Summary`, `ChildIndex`, and the coverage-driven gap
+  loop in the extractor).
+* **Spinning disks:** always read in physical order (`read_many`, `windows()`).
+* **Carved roots:** associate them with a dataset by *shared tree blocks* (`cluster()`),
+  never by size or geometry alone.
+* **Windows:** pass `encoding="utf-8"` to every `read_text`/`write_text`, since the default
+  is cp1252. Paths from the evidence may be un-decodable: use `surrogateescape`, and
+  `safe_name()` when creating files.
+* **Plugins** register via entry points (`zesus.filesystems`, `zesus.partitions`,
+  `zesus.identifiers`). Core must not import plugin-specific code, except the built-ins
+  in the registries.
+* Commit messages: imperative subject, a body explaining *why*. Follow [AI_POLICY.md](AI_POLICY.md): state only what
+  was actually done and tested, and keep the `Co-Authored-By` trailer on AI-assisted commits.
+
+## Status and next steps
+
+Validated end to end on a real 1 TB single-disk pool:
+* A destroyed 880 GiB zvol (GPT + ext4) was rebuilt from 1,202 carved tree generations.
+* Carved-only and ring+carved reconstructions matched slot for slot.
+* Extracted files passed independent integrity checks.
+
+Open items, roughly by value:
+1. Carved-only reconstruction for destroyed ZFS **filesystem** datasets (the volume path in
+   `phase_reconstruct` shows the approach: cluster carved `os_type=2` objsets and walk
+   them with `ZplFilesystem`).
+2. Test fixtures made by real OpenZFS (small file-backed pools with destroyed zvols and
+   datasets) for CI. These need a machine with ZFS. Add a `dev/make_zfs_fixtures.sh`.
+3. RAIDZ/dRAID reconstruction in `zfs/vdev.py` (they are currently reported as unsupported).
+4. More filesystem plugins: XFS and NTFS inventory (identification exists already).
+5. Verification of zvol **snapshots** as their own volumes.
+6. Encrypted datasets (with a user-supplied key).

zesus-0.1.0a1/CODE_OF_CONDUCT.md

@@ -0,0 +1,54 @@
+# Code of Conduct
+
+This project follows the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+Its standards and enforcement guidelines apply here in full. The sections below add
+project-specific expectations; they do not replace anything in the Covenant.
+
+## Be professional
+
+Be respectful and assume good faith. People often arrive here after losing important
+data, sometimes under real pressure. Help them, or leave the thread to someone who will.
+
+Disagreements are expected. Keep them technical: argue about code, data and evidence,
+never about the person. If a discussion turns into an argument, step away and let a
+maintainer settle it.
+
+## Stay on topic: no politics
+
+Zesus exists to recover data from ZFS pools. Every project space stays on that purpose.
+
+**Political, ideological, religious and social-cause content is not allowed in any project
+space.** This applies equally to every cause, side and viewpoint, without exception. It is
+not a statement about any of those causes, or about whether they matter. It is a statement
+that this project is not the place for them.
+
+Project spaces are:
+* issues, pull requests, discussions and review comments;
+* commit messages;
+* code, comments, documentation, release notes and the README;
+* project-owned chat channels and social accounts.
+
+Not allowed there, for example:
+* slogans, banners, badges or messages for or against any political figure, party,
+  government, movement, conflict or cause;
+* political symbols or messages in code, test data, sample files or artwork added to the
+  repository;
+* arguing about current events, or answering someone else's political remark.
+
+Contributors are free to hold and express any views they like outside project spaces.
+However, directing public attacks, harassment or bad-faith criticism at the project, its
+maintainers or its community from outside project spaces is incompatible with contributing
+to it. Honest, good-faith criticism is always welcome, wherever it is made: a public bug
+report, a critical review, or a write-up of where Zesus failed you. (Though we would prefer
+that you raise it with us first, so that we can make sure no one else has to deal with the
+same problem.)
+
+If someone brings politics in, **do not reply to it**. Report it to the maintainers, who
+will remove or hide the content and, if needed, lock the thread. Repeated or deliberate
+violations are handled under the Covenant's
+[enforcement guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/#enforcement-guidelines).
+
+## Reporting
+
+Report unacceptable behaviour privately to the maintainers, through the contact listed on the
+GitHub repository. Reports are handled confidentially.

zesus-0.1.0a1/CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+Thanks for helping. Data recovery tools are only as good as the formats and failure
+cases they have seen.
+
+## Setup
+
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev,all]"
+pytest
+ruff check src tests
+```
+
+Integration tests run against a real ZFS image when `ZESUS_TEST_IMAGE` is set.
+
+## Conduct
+
+Be professional, keep discussions technical, and keep politics out of the project entirely.
+See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
+
+## Using AI tools
+
+AI-assisted contributions are welcome. You remain responsible for everything you submit:
+test it, review it, and keep commit messages factual. See [AI_POLICY.md](AI_POLICY.md).
+
+## Ground rules
+
+* **The source is sacred.** Nothing may write to the evidence. All access goes through
+  `zesus.io.RawSource`, and `tests/unit/test_guard.py` enforces the rule. Pull
+  requests that add write paths to `zesus/io` will not be merged.
+* **Never guess silently.** Anything unverified or inferred must be labelled as such in
+  the map and in output reports.
+* **Be damage-tolerant.** Parsers must survive truncated, zeroed and random input. Log
+  what could not be read and continue.
+* New filesystem support belongs in a plugin (see `docs/writing-plugins.md`).
+* Add tests with small real fixtures where possible. Scripts that generate fixtures go in
+  `dev/`.
+
+## Reporting a recovery problem
+
+Please include:
+* the output of `zesus info MAP`;
+* the JSON log (`*.log.jsonl`);
+* your OpenZFS version and pool features (`zpool get all`, if the pool still imports).
+
+Never attach disk images or maps from real cases publicly. They contain your data.

zesus-0.1.0a1/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.