sandcrawler 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +5 -0
  3. data/LICENSE.txt +21 -0
  4. data/README.md +157 -0
  5. data/docs/findings.md +49 -0
  6. data/docs/releasing.md +28 -0
  7. data/docs/verification.md +80 -0
  8. data/exe/sandcrawler +5 -0
  9. data/lib/sandcrawler/acquisition/mounted_volume.rb +81 -0
  10. data/lib/sandcrawler/acquisition/picotool.rb +227 -0
  11. data/lib/sandcrawler/acquisition/serial_protocol.rb +136 -0
  12. data/lib/sandcrawler/acquisition/serial_shell.rb +162 -0
  13. data/lib/sandcrawler/acquisition/serial_templates.rb +63 -0
  14. data/lib/sandcrawler/cli/devices.rb +109 -0
  15. data/lib/sandcrawler/cli/offline.rb +199 -0
  16. data/lib/sandcrawler/cli.rb +149 -0
  17. data/lib/sandcrawler/errors.rb +11 -0
  18. data/lib/sandcrawler/extractor.rb +157 -0
  19. data/lib/sandcrawler/fat/allocation_table.rb +139 -0
  20. data/lib/sandcrawler/fat/boot_sector.rb +142 -0
  21. data/lib/sandcrawler/fat/dir_entry.rb +80 -0
  22. data/lib/sandcrawler/fat/lfn_assembler.rb +60 -0
  23. data/lib/sandcrawler/fat/volume.rb +198 -0
  24. data/lib/sandcrawler/fat/volume_locator.rb +108 -0
  25. data/lib/sandcrawler/image/raw_image.rb +38 -0
  26. data/lib/sandcrawler/image/slice.rb +25 -0
  27. data/lib/sandcrawler/image/source.rb +25 -0
  28. data/lib/sandcrawler/image/uf2_image.rb +83 -0
  29. data/lib/sandcrawler/log.rb +44 -0
  30. data/lib/sandcrawler/manifest.rb +37 -0
  31. data/lib/sandcrawler/path_sanitizer.rb +61 -0
  32. data/lib/sandcrawler/recovery/carver.rb +76 -0
  33. data/lib/sandcrawler/recovery/orphans.rb +52 -0
  34. data/lib/sandcrawler/recovery/undelete.rb +43 -0
  35. data/lib/sandcrawler/version.rb +5 -0
  36. data/lib/sandcrawler.rb +25 -0
  37. metadata +81 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: e23c0fbd03d38450155ff6452cda9ef0ea2df57a62510d14ff6b1dd0e7e0b7f5
4
+ data.tar.gz: 8ca4a9276f597ee1287680311337ba9cc8d7ce56ae003bee26835a8a1448de5c
5
+ SHA512:
6
+ metadata.gz: 49b59f162a6edfc9dcc2aee9a08053d3f1fa51c065f85dffe196ef71e892c4a6dec0c7bf39c6bfb4305c421ec027d32508949ee50935306da53f941cc63ceea5
7
+ data.tar.gz: b9fcf2b93c4da162d72f889492c00f3b5518b91973713c3cc61e0611c489d4954be437e6a37d7f65adf2ef288c20caee392af52d5a716261356da0df005583c7
data/CHANGELOG.md ADDED
@@ -0,0 +1,5 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 (unreleased)
4
+
5
+ - Initial release.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2026 Yudai Takada
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,157 @@
1
+ # Sandcrawler
2
+
3
+ **Take a flash dump before using `flash_nuke.uf2`, reinstalling R2P2, or formatting a drive.** These operations can overwrite the files you want to recover. Keep the original dump and a second copy on another disk.
4
+
5
+ Sandcrawler salvages files from Raspberry Pi Pico / Pico W (RP2040) and Pico 2 / Pico 2 W (RP2350) flash images. It reads FAT12, FAT16, FAT32, and long filenames (including Japanese), and recovers deleted files, orphaned cluster chains, and text fragments.
6
+
7
+ Sandcrawler does not repair or write to the device. Its picotool adapter permits only `version`, `info`, and `save`. Hardware and VM compatibility still require the checks in [docs/findings.md](docs/findings.md).
8
+
9
+ ## Install
10
+
11
+ Ruby 3.1 or newer is required. The core has no runtime gem dependencies. From this checkout:
12
+
13
+ ```sh
14
+ gem build sandcrawler.gemspec
15
+ gem install ./sandcrawler-0.1.0.gem
16
+ sandcrawler --version
17
+ ```
18
+
19
+ Install [picotool](https://github.com/raspberrypi/picotool) separately for flash acquisition; RP2350 requires picotool 2.x. Set `PICOTOOL=/path/to/picotool` or pass `--picotool /path/to/picotool` to choose an executable. The serial route additionally needs `gem install serialport`.
20
+
21
+ ## Recover a device
22
+
23
+ Prefer BOOTSEL acquisition, which does not need R2P2 to start. Hold BOOTSEL while connecting the board, then:
24
+
25
+ ```sh
26
+ sandcrawler doctor
27
+ sandcrawler dump -o ./rescue
28
+ sandcrawler scan ./rescue/dump.bin
29
+ sandcrawler ls ./rescue/dump.bin --long --deleted
30
+ sandcrawler extract ./rescue/dump.bin -o ./rescue/out --deleted --orphans
31
+ ```
32
+
33
+ `dump` reads twice by default and compares SHA-256 hashes. If they differ, it reads once more and chooses the strict majority for each 4 KiB block. Every pass is retained. Blocks without a majority are listed in `dump.json` and cause exit code 5; producing an image does not imply its contents are reliable. `dump.bin` is made read-only and has a companion `dump.bin.sha256` file.
34
+
35
+ Each picotool command has a 120-second timeout and a 1 MiB limit on each of stdout and stderr. If a failed save produced bytes, they are retained as `passN-all-failed.bin` or `passN-range-failed.bin` with an `acquisition_error` record and hash. Missing required hashes or acquisition metadata cause an integrity error.
36
+
37
+ `scan` reports all plausible volumes with offsets and scores. `ls` and `extract` choose the highest-ranked candidate unless `--offset` selects another. `auto -o ./rescue` offers interactive route selection and runs acquisition and extraction together.
38
+
39
+ If BOOTSEL is unavailable and the mounted R2P2 volume is readable:
40
+
41
+ ```sh
42
+ sandcrawler copy /mnt/r2p2 -o ./copied
43
+ ```
44
+
45
+ Use a read-only mount (`mount -o ro` on Linux). Although Sandcrawler only reads the source, the operating system may write hidden metadata to writable removable volumes. Symlinks are skipped. Keep the destination outside the source volume.
46
+
47
+ Detected file changes during copying produce `SOURCE_CHANGED` warnings and a partial-success result. Copying a running filesystem does not provide a consistent snapshot.
48
+
49
+ For a running R2P2 shell on CDC0:
50
+
51
+ ```sh
52
+ sandcrawler serial --port /dev/ttyACM0 -o ./serial-copy
53
+ ```
54
+
55
+ Without `--port`, the CLI lists candidates and asks for a selection. Choose the shell port, not debug or MIDI. Serial acquisition interrupts `app.rb`, enters IRB, probes APIs, and verifies hexadecimal chunks against a byte count and checksum. Session-specific marker tokens distinguish protocol responses from unrelated output. Failed transfers are retried up to three attempts. Device startup and unrelated running software can still alter flash; use BOOTSEL when possible.
56
+
57
+ Serial requests have a 10-second idle timeout and a one-hour total deadline. Each request accepts at most 65,537 response records and 40 MiB of encoded responses, supporting files up to 16 MiB with the fixed 256-byte chunks. Incomplete listings, duplicate names, invalid frames, and oversized responses are rejected. These limits are built in; there are no CLI options to increase them.
58
+
59
+ ## Commands and options
60
+
61
+ | Command | Purpose |
62
+ |---|---|
63
+ | `doctor` | Check Ruby, picotool, USB access, and optional serial support |
64
+ | `dump -o DIR` | Read flash; `--passes N` sets the initial read count (default 2) |
65
+ | `scan IMAGE` | Rank FAT candidates in raw or UF2 images |
66
+ | `ls IMAGE` | List files; `--long` adds metadata, `--deleted` includes deleted entries |
67
+ | `extract IMAGE -o DIR` | Extract files; `--deleted`, `--orphans`, and `--carve` enable recovery |
68
+ | `copy MOUNTPOINT -o DIR` | Copy a mounted volume with hashes |
69
+ | `serial -o DIR [--port PORT]` | Copy through the R2P2 shell |
70
+ | `auto -o DIR` | Choose a route interactively |
71
+
72
+ `--help` lists options. `--json` emits machine-readable results, `--verbose` adds diagnostics, and `--quiet` suppresses routine progress. `--force` permits a nonempty destination; use a new output directory to preserve earlier recovery results unchanged.
73
+
74
+ | Image option | Meaning |
75
+ |---|---|
76
+ | `--offset N` | Volume byte offset, decimal or `0x` hexadecimal |
77
+ | `--fat 1` / `--fat 2` | Force a FAT copy; otherwise erased sectors may fall back to the other copy |
78
+ | `--codepage NAME` | Short filename encoding (default `IBM437`; for example `CP932`) |
79
+ | `--reference-image FILE` | Borrow geometry from a healthy image with the same firmware and layout |
80
+ | `--geometry KEY=VALUE,...` | Supply geometry when the boot sector is damaged |
81
+
82
+ ## Damaged volumes
83
+
84
+ Try a healthy reference dump from a **separate test board** with the same firmware and flash layout:
85
+
86
+ ```sh
87
+ sandcrawler extract broken.bin -o ./with-reference --reference-image healthy.bin
88
+ ```
89
+
90
+ Manual geometry requires `bps` (bytes per sector), `spc` (sectors per cluster), `rsvd` (reserved sectors), `fats` (copy count), `fatsz` (sectors per FAT), and `root` (FAT12/16 root entry count, or zero for FAT32). Optional keys are `offset` (byte offset), `total` or `totsec` (total sectors, otherwise inferred from remaining image size), `root_cluster` (FAT32, default 2), and `media` (default 248). The following values are illustrative; substitute measured geometry:
91
+
92
+ ```sh
93
+ sandcrawler extract broken.bin -o ./manual \
94
+ --geometry bps=4096,spc=1,rsvd=1,fats=2,fatsz=1,root=128,total=256,offset=0x100000
95
+ sandcrawler extract broken.bin -o ./carved --carve
96
+ ```
97
+
98
+ Missing fields, duplicate keys, conflicting `total`/`totsec`, and invalid geometry produce usage error 2. When a FAT32 backup boot sector has matching FAT copies and a plausible root entry, detection can recover the volume's original offset and records a `BACKUP_BOOT_SECTOR` warning.
99
+
100
+ With no readable volume, `--carve` searches the whole image. Deleted-file recovery assumes consecutive free clusters and stops before allocated data. Original fragmentation and overwritten bytes cannot be reconstructed reliably. Orphan chains have no authoritative filename or length, so final clusters can include padding. Carving groups adjacent valid UTF-8 blocks with at least 95% printable characters; it cannot establish file boundaries or recover arbitrary binaries. Inspect every recovery result.
101
+
102
+ ## Results
103
+
104
+ ```text
105
+ out/
106
+ files/ live files; .partial suffix when incomplete
107
+ deleted/ best-effort undeleted files
108
+ lost+found/ unreferenced chains
109
+ carved/ text fragments
110
+ manifest.json provenance, paths, hashes, statuses, warnings
111
+ sandcrawler.log diagnostics
112
+ ```
113
+
114
+ The manifest records the source image's SHA-256 and size, geometry and FAT selection, original names, safe output paths, hashes, timestamps, attributes, and warnings. Its `directories` list also records directory metadata and creation failures. Invalid FAT timestamps are `null`. Unsafe path components, Windows reserved names, and case-insensitive collisions are renamed; original names remain in the manifest. Invalid UTF-8 path and name bytes are preserved in companion `_hex` fields. Console diagnostics escape control characters.
115
+
116
+ `ok` means no read/structure error was detected, not that historical contents are proven intact. `truncated` and `chain_error` identify incomplete or broken chains; `suspect_erased` identifies all-`0xFF` clusters. `read_error` records an unreadable file or carving block; readable prefixes are preserved and other files or blocks are still attempted. Recovery warnings explain assumptions. Keep the source image after extraction.
117
+
118
+ Extraction rechecks the image and any reference image against their original SHA-256 hashes before finishing. A changed or unreadable input produces `SOURCE_CHANGED` and exit code 5; recovered files and the failure manifest remain available for inspection.
119
+
120
+ | Exit code | Meaning |
121
+ |---|---|
122
+ | 0 | Success |
123
+ | 1 | General error |
124
+ | 2 | Invalid usage |
125
+ | 3 | Device unavailable |
126
+ | 4 | Partial success; inspect statuses and warnings |
127
+ | 5 | Integrity error, including unresolved dump disagreement or changed extraction input |
128
+
129
+ ## Troubleshooting
130
+
131
+ | Symptom | Next step |
132
+ |---|---|
133
+ | No device | Check BOOTSEL, a data-capable cable, and USB permissions; Linux may need picotool's udev rules |
134
+ | Different dump hashes | Connect directly without a hub, retain all passes, and inspect unstable blocks |
135
+ | No FAT candidate | Check the offset, use matching reference geometry, then try carving |
136
+ | Garbled short filename | Try the firmware's code page, such as `CP932` |
137
+ | Partial files | Inspect `lost+found/` and `carved/`; keep the original image |
138
+ | Serial timeout / unsupported API | Check CDC0 and firmware/VM details; use BOOTSEL if available |
139
+
140
+ ## Development
141
+
142
+ ```sh
143
+ bundle install
144
+ bundle exec rake test
145
+ bundle exec rubocop
146
+ bundle exec ruby exe/sandcrawler --version
147
+ ```
148
+
149
+ Install dosfstools and mtools for independent integration fixtures (`brew install dosfstools mtools` or `sudo apt install dosfstools mtools`), with their executables on `PATH`. `rake test` builds missing fixtures when tools are available; otherwise it warns, runs synthetic unit tests, and skips external-fixture integration. Require full coverage with `REQUIRE_FIXTURES=1 bundle exec rake test`.
150
+
151
+ `script/build_fixtures.sh` creates FAT12 with 512/4096-byte sectors, FAT16, FAT32, Japanese names, empty/multi-cluster files, deleted files, embedded volumes, and corruption cases. Expected trees come from `mcopy` and are checked against original content. Generated files are ignored by Git. CI generates on Ubuntu and shares fixtures with Ubuntu, macOS, and Windows jobs for Ruby 3.1 and maintained Ruby branches.
152
+
153
+ See [software verification](docs/verification.md) for requirement coverage, adversarial cases, and the platforms actually exercised. Hardware acceptance is separate from image tests. Follow [the hardware checklist](docs/findings.md) before claiming a board/firmware/VM combination is validated. See [release instructions](docs/releasing.md) for publication checks.
154
+
155
+ ## License
156
+
157
+ [MIT](LICENSE.txt).
data/docs/findings.md ADDED
@@ -0,0 +1,49 @@
1
+ # Hardware findings and acceptance checklist
2
+
3
+ Status: **not measured on physical hardware**. Synthetic images and mocked transports cannot answer the six questions below. No Pico has been flashed, erased, deliberately corrupted, or used for acquisition during implementation. Test offsets are not device measurements.
4
+
5
+ ## Open questions
6
+
7
+ | Question | Current result | Evidence needed |
8
+ |---|---|---|
9
+ | FAT offset and size | Unverified; detected or supplied per image | Firmware, board, matching dumps, `scan --json` output |
10
+ | Logical sector size | Unverified; parser accepts 512/1024/2048/4096 | Physical BPB and independently checked extraction hashes |
11
+ | MBR presence | Unverified; scanner considers partition starts | MBR bytes, partition entries, selected offset |
12
+ | Logical-to-physical mapping | Unverified; parser assumes linear mapping | Multi-cluster files match independent source hashes |
13
+ | Startup flash writes | Unverified; do not assume boot preserves evidence | Before/after dumps around a controlled boot |
14
+ | File/Dir APIs for each VM | Unverified; serial probes capabilities | Capability transcript and transfer hashes per firmware/VM |
15
+
16
+ Nonlinear storage would require an additional mapping layer before raw recovery can be trusted. No mapping has been inferred from an untested firmware version.
17
+
18
+ ## Record each test
19
+
20
+ Use a separate test board. Never experiment on the only recoverable copy of a device's data. Record:
21
+
22
+ - Board, chip, flash capacity, firmware repository and exact release/commit, build settings, VM.
23
+ - Host OS, Ruby/picotool versions, dump hashes, retained artifact location.
24
+ - FAT offset/size/type, sector/cluster sizes, MBR result.
25
+ - Original file hashes and hashes from each acquisition route.
26
+ - Raw serial capability transcript, retries, missing APIs.
27
+ - Flash blocks changed on startup, exact boot procedure, warnings, exit codes, observed losses.
28
+
29
+ ## Procedure
30
+
31
+ 1. On the test board only, install chosen firmware and create `home/app.rb`, nested directories, a Japanese filename, an empty file, and a 100 KB binary. Record their SHA-256 hashes on the host.
32
+ 2. Safely unmount, disconnect, reconnect in BOOTSEL, and run `sandcrawler dump -o before`. Require matching pass hashes.
33
+ 3. Run `scan before/dump.bin --json` and `extract before/dump.bin -o extracted`. Compare every output hash to the originals. This supplies evidence for geometry and linear mapping.
34
+ 4. On the test board only, boot normally and record what runs. Return to BOOTSEL, dump to `after`, and compare 4 KiB blocks. Record changed offsets and repeat only if needed for the experiment.
35
+ 5. Capture the serial capability output and copy via CDC0. Copy the read-only mounted volume separately. Compare both routes against originals and flash extraction.
36
+ 6. Delete a known file on the test board, dump again, and check `extract --deleted` against the original content.
37
+ 7. Only with expendable data and a disposable test board, interrupt a write and evaluate `--deleted --orphans --carve`. Record exact losses. Never run this experiment on a rescue target.
38
+
39
+ ## Acceptance matrix
40
+
41
+ | Board | VM | BOOTSEL dump | Mounted copy | Serial copy |
42
+ |---|---|---|---|---|
43
+ | Pico | femtoruby | Pending (required) | Pending (required) | Pending (required) |
44
+ | Pico 2 | femtoruby | Pending (required) | Pending (optional) | Pending (optional) |
45
+ | Pico 2 | picoruby | Pending (required) | Pending (optional) | Pending (required) |
46
+ | Pico W | firmware-specific | Pending (optional) | Pending (optional) | Pending (optional) |
47
+ | Pico 2 W | firmware-specific | Pending (optional) | Pending (optional) | Pending (optional) |
48
+
49
+ Completion requires measurements for all six questions and every required matrix cell. Software tests alone do not complete this checklist.
data/docs/releasing.md ADDED
@@ -0,0 +1,28 @@
1
+ # Release procedure
2
+
3
+ Publishing is a separate maintainer action. Implementation does not publish a gem or create release tags.
4
+
5
+ 1. Review `docs/findings.md`; state hardware limitations in release notes.
6
+ 2. Update `lib/sandcrawler/version.rb` and `CHANGELOG.md`.
7
+ 3. Run full tests with generated fixtures, lint, and build:
8
+
9
+ ```sh
10
+ REQUIRE_FIXTURES=1 bundle exec rake test
11
+ bundle exec rubocop
12
+ gem build sandcrawler.gemspec
13
+ ```
14
+
15
+ 4. Install the artifact in a temporary gem home and smoke-test the installed command (substitute the release version):
16
+
17
+ ```sh
18
+ gem install --install-dir ./tmp/release-gems ./sandcrawler-0.1.0.gem --no-document
19
+ GEM_HOME=./tmp/release-gems ./tmp/release-gems/bin/sandcrawler --version
20
+ GEM_HOME=./tmp/release-gems ./tmp/release-gems/bin/sandcrawler doctor
21
+ ```
22
+
23
+ A build machine may lack devices or optional tools. Record `doctor`'s result without claiming hardware validation.
24
+
25
+ 5. Verify CI and package contents: executable, every runtime Ruby file, README, license, changelog, and docs. Generated images and tests must be excluded.
26
+ 6. Verify ownership/availability of the `sandcrawler` name on RubyGems, enable MFA, and review the commit/version. After publication is explicitly approved, `bundle exec rake release` creates/pushes the tag and publishes the gem.
27
+
28
+ The gemspec requires MFA and allows pushes only to `https://rubygems.org`.
@@ -0,0 +1,80 @@
1
+ # Software verification
2
+
3
+ This records the software acceptance scope for the implementation and adversarial review on 2026-10-28. Physical Pico testing and execution on Windows are outside this verification run. They are not implied by the passing software tests; physical acceptance remains documented in [findings.md](findings.md).
4
+
5
+ ## Independent acceptance oracle
6
+
7
+ `script/build_fixtures.sh` uses dosfstools and mtools to construct FAT12 (512-byte and 4096-byte sectors), FAT16, and FAT32 images. It writes known original files and obtains the expected trees with `mcopy`. The generator verifies those trees against the originals before marking the corpus complete. Generated artifacts are excluded from the repository.
8
+
9
+ `test/acceptance/offline_acceptance_test.rb` exercises the CLI rather than only the FAT reader. It checks:
10
+
11
+ - All four filesystem types: exact directory and filename sets, every file's SHA-256, empty files, nested files, Japanese filenames, and multi-cluster binary content.
12
+ - A complete flash image and an equivalent UF2 with blocks in reverse order: identical extracted trees and volume geometry, the correct embedded-volume offset, and source hashes.
13
+ - An erased FAT copy and an erased boot sector with a healthy reference image: unchanged live file contents, explicit recovery warnings, and successful status.
14
+ - A deleted file: exact recovered bytes and a status that identifies the recovery assumption.
15
+ - An erased directory: unaffected siblings plus orphan contents matching the lost originals; unknown file lengths are not treated as authoritative.
16
+ - A cyclic chain: the exact readable prefix has a `.partial` suffix, intact siblings remain correct, and the command reports partial success.
17
+
18
+ Every acceptance extraction also checks that stdout JSON matches `manifest.json`, that recorded file hashes and sizes match the output bytes, that recorded timestamps match restored timestamps, and that the input image's hash is unchanged.
19
+
20
+ ## Requirement coverage
21
+
22
+ | Requirement | Software evidence | Boundary |
23
+ |---|---|---|
24
+ | FR-01 / FR-02: acquisition and repeated reads | Fake picotool process tests verify allowed commands, pass counts, hashes, strict block majority, range fallback, and failures | Real USB devices and picotool-to-board compatibility require physical testing |
25
+ | FR-03: raw/UF2 volume detection | Embedded-volume, MBR, UF2 validation, and full CLI raw/UF2 parity tests | A damaged boot sector can still require manual or reference geometry |
26
+ | FR-04 / FR-05: FAT parsing and faithful extraction | Independent expected trees for FAT12/16/32; LFN/codepage/timestamp tests; output acceptance checks | Invalid timestamps remain unknown; FAT timestamps have no timezone metadata |
27
+ | FR-06: mismatched and erased FAT copies | Copy selection, FAT32 active-copy, mixed-sector and erased-allocation tests | Disagreement without a uniquely healthy copy remains uncertain and is reported |
28
+ | FR-07 / FR-08 / FR-09: recovery | Deleted, orphan, cycle, cross-link, and carving tests with exact content expectations | Deleted fragmentation and original orphan/carved file boundaries cannot be proven |
29
+ | FR-10: serial transfer | Generated template execution plus pipe-based peers, checksums, listing completeness, retries, protocol validation and bounded failures | Actual firmware prompts and VM APIs require physical testing; individual file transfers are limited to 16 MiB |
30
+ | FR-11: mounted copy | Tree/content/timestamp tests, hidden files, literal mount names, symlink rejection, changed files and failed reads | The host OS may independently write to a writable source mount; this is not an atomic snapshot |
31
+ | FR-12 / NFR-04: provenance | Manifest/stdout agreement, source and output hashes, directory metadata, original byte names, failure reporting and post-extraction source verification | Source state changing outside Sandcrawler cannot establish historical provenance |
32
+ | NFR-01: read-only acquisition | Command/argument allowlists and fixed read-only serial templates | Serial startup interrupts the application; other device software is outside the tool's control |
33
+ | NFR-02: minimal dependencies | Gem metadata and installed executable checks; the core uses the Ruby standard library | Serial communication additionally needs the optional `serialport` gem |
34
+ | NFR-03: platforms and Ruby | Local macOS and Linux container runs; Ruby 3.1 and 4.0 checks; CI configuration includes Windows | No Windows execution or remote CI run is claimed |
35
+ | NFR-05: damaged-input robustness | Seeded metadata mutation, bounded chains/protocols, injected read/write failures, and preserved partial results | These are defined adversarial cases, not a proof over every possible byte sequence |
36
+ | NFR-06: safe output | Traversal/reserved-name/collision tests, symlink rejection, and atomic output checks | Output directories must not be concurrently manipulated by another process |
37
+
38
+ ## Adversarial fixes verified
39
+
40
+ The review used concrete malformed images, injected I/O failures, fake subprocesses, and scripted serial peers. Regression tests cover the corrected behavior:
41
+
42
+ - FAT32 backup boot sectors no longer become a false volume at the backup sector's own offset when the primary is erased and the image has trailing bytes. The recovered original offset and `BACKUP_BOOT_SECTOR` evidence are recorded.
43
+ - FAT structural `.` and `..` entries are identified before codepage decoding. Metadata corruption, cycles, cross-links, erased allocation entries, and directory terminators are checked without falsely treating damaged results as intact files.
44
+ - A failed streamed read retains the readable prefix with a `.partial` suffix. Early EOF is compared with the expected size. Directory creation failures are recorded while independent entries remain recoverable.
45
+ - Invalid UTF-8 source and entry names no longer abort JSON or diagnostic logging. Companion `_hex` fields retain original path and name bytes, distinct invalid-byte directory names remain distinct, and terminal control characters are escaped.
46
+ - Missing, duplicate, contradictory, and invalid manual geometry produce usage error 2. Image and reference hashes are rechecked after extraction; changes produce `SOURCE_CHANGED`, exit 5, and a saved failure manifest.
47
+ - Mounted paths containing glob characters no longer produce silently empty copies. Source symlinks and replaced source entries are rejected. Detected rewrites during copying are reported instead of marking the result `ok`.
48
+ - Serial listings require a verified terminator, count, checksum, and unique names. Unexpected, incomplete, or malformed transfer frames fail. Requests stop after the idle timeout, one-hour deadline, 65,537 records, or 40 MiB of encoded responses.
49
+ - picotool commands stop after 120 seconds by default or when either stdout or stderr exceeds 1 MiB. Failed saves preserve any acquired bytes and their hashes before fallback. Invalid UTF-8 device information remains serializable with original bytes in `info_hex`. Required acquisition sidecar failures and changed pass lengths are integrity errors.
50
+
51
+ Final independent probes reconfirmed the invalid-name error path, directory collision separation, invalid geometry exit codes, and erased-primary FAT32 backup recovery. The independent CLI acceptance file still passed after these fixes: 6 tests, 264 assertions, no failures, errors, or skips.
52
+
53
+ ## Reproduce
54
+
55
+ ```sh
56
+ bundle install
57
+ bundle exec rake fixtures
58
+ REQUIRE_FIXTURES=1 bundle exec rake test
59
+ bundle exec rubocop
60
+ actionlint
61
+ gem build sandcrawler.gemspec
62
+ ```
63
+
64
+ The strict `REQUIRE_FIXTURES=1` run must contain no fixture-related skips. Missing dosfstools/mtools or an absent generated corpus must not be interpreted as successful independent acceptance. The CI workflow generates fixtures once on Ubuntu and supplies the same corpus to each platform job.
65
+
66
+ The acceptance file can also be run on its own:
67
+
68
+ ```sh
69
+ REQUIRE_FIXTURES=1 bundle exec ruby -Itest test/acceptance/offline_acceptance_test.rb
70
+ ```
71
+
72
+ After all adversarial fixes, the complete suite passed **127 tests and 1,686 assertions**, with **zero failures, errors, or skips**, in each environment below:
73
+
74
+ | Environment | Ruby | Result |
75
+ |---|---|---|
76
+ | macOS | 4.0.9 | Full suite, including generated fixtures, passed |
77
+ | macOS | 3.1.0 | Full suite passed; unrelated globally installed minitest plugins disabled |
78
+ | Linux container | 4.0.9 | Full suite passed with networking disabled and the checkout mounted read-only |
79
+
80
+ Mutation checks include 200 FAT metadata cases and 100 UF2 cases with fixed seeds. These supplement the independent mtools comparisons and exact damaged-file expectations. No Windows execution, physical-device acceptance, remote CI result, or RubyGems publication is claimed.
data/exe/sandcrawler ADDED
@@ -0,0 +1,5 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ require_relative "../lib/sandcrawler"
5
+ exit Sandcrawler::CLI.new.run(ARGV)
@@ -0,0 +1,81 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Sandcrawler
4
+ module Acquisition
5
+ class MountedVolume
6
+ WARNING = "Mounted volumes may be changed by the OS. On Linux use mount -o ro; " \
7
+ "on other systems prefer a BOOTSEL dump."
8
+
9
+ def initialize(path)
10
+ @path = File.realpath(path)
11
+ raise UsageError, "Mount point is not a directory: #{path}" unless File.directory?(@path)
12
+ end
13
+
14
+ def copy(out, force: false, log: nil)
15
+ reject_overlap!(out)
16
+ output = Extractor.new(out, source: { type: "mounted_volume", path: @path }, force: force, log: log)
17
+ output.warning("MOUNT_WRITE_RISK", WARNING)
18
+ Dir.open(@path, &:read)
19
+ Dir.glob("**/*", File::FNM_DOTMATCH, base: @path).sort.each do |relative|
20
+ next if %w[. ..].include?(File.basename(relative))
21
+
22
+ copy_entry(output, File.join(@path, relative))
23
+ end
24
+ output.finish
25
+ rescue SystemCallError, IOError => e
26
+ raise unless output
27
+
28
+ output.warning("COPY_FAILED", "#{@path}: #{e.message}")
29
+ output.finish
30
+ end
31
+
32
+ private
33
+
34
+ def reject_overlap!(out)
35
+ destination = File.expand_path(out)
36
+ ancestor = destination
37
+ ancestor = File.dirname(ancestor) until File.exist?(ancestor) || File.symlink?(ancestor)
38
+ resolved = File.join(File.realpath(ancestor), destination.delete_prefix(ancestor).sub(%r{\A/}, ""))
39
+ if resolved == @path || resolved.start_with?(@path + File::SEPARATOR) || @path.start_with?(resolved + File::SEPARATOR)
40
+ raise SafetyError, "Source and output directories must not overlap"
41
+ end
42
+ end
43
+
44
+ def copy_entry(output, path)
45
+ relative = path.delete_prefix(@path + File::SEPARATOR)
46
+ stat = File.lstat(path)
47
+ if stat.symlink? || File.realpath(path) != path
48
+ output.warning("SYMLINK_SKIPPED", relative)
49
+ elsif stat.directory?
50
+ Dir.open(path, &:read)
51
+ output.directory(path: relative, mtime: stat.mtime)
52
+ elsif stat.file?
53
+ flags = File::RDONLY
54
+ flags |= File::NOFOLLOW if File.const_defined?(:NOFOLLOW)
55
+ File.open(path, flags) do |file|
56
+ opened = file.stat
57
+ current = File.lstat(path)
58
+ if File.realpath(path) != path || !opened.file? ||
59
+ [current.dev, current.ino] != [stat.dev, stat.ino] || !File.identical?(file, path)
60
+ raise IOError, "Source changed while opening #{relative}"
61
+ end
62
+
63
+ file.binmode
64
+ record = output.write(path: relative, data: file, mtime: stat.mtime,
65
+ original_name: File.basename(path), expected_size: stat.size)
66
+ after = file.stat
67
+ if [stat.size, stat.mtime, stat.ctime] != [after.size, after.mtime, after.ctime]
68
+ record[:status] = "read_error" if record[:status] == "ok"
69
+ output.warning("SOURCE_CHANGED", "#{relative} changed during copying; verify recovered bytes")
70
+ end
71
+ end
72
+ else
73
+ output.warning("SPECIAL_FILE_SKIPPED", relative)
74
+ end
75
+ rescue SystemCallError, IOError => e
76
+ output.warning("COPY_FAILED", "#{relative}: #{e.message}")
77
+ output.write(path: relative, data: "".b, status: "read_error", error: e.message)
78
+ end
79
+ end
80
+ end
81
+ end