diskstack 1.0.0__tar.gz

diskstack-1.0.0/CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+## 1.0.0 - 2026-09-12
+
+First release.
+
+- Merges two or more dumps of one floppy disk into a single sector image.
+- Reads flux (`.scp`, KryoFlux `.raw` streams, `.hfe`) and sector images
+  (`.img`, `.ima`, `.st`, `.adf`, `.imd`), in any mix.
+- Decodes every revolution of every flux capture, not just the first two.
+- Groups read attempts by the decoded `(cylinder, head, sector id)`, so
+  captures with different index phases and motor speeds still line up.
+- Three tiers per sector: a passing CRC wins, else the attempts are rebuilt
+  into candidate payloads that have to satisfy a check value off the disk,
+  else unresolved.
+- Three reconstruction routes, reported per sector as `method`: a byte-wise
+  majority, a majority of one payload per input file so a capture with more
+  revolutions cannot outvote the rest, and each read on its own, which
+  recovers a sector whose CRC bytes rather than payload were damaged.
+- Treats Greaseweazle's `-=[BAD SECTOR]=-` filler as a failed read.
+- Prints a `gw read --tracks=` command naming only the tracks still unresolved.
+- Writes `diskstack-report.json` with per-sector provenance, schema 1.
+- IBM FM/MFM and AmigaDOS MFM formats, auto-detected.
+- Decodes flux one track per core, `-j/--jobs` to change it, falling back to a
+  single process if the worker pool cannot start.
+- Flags a sector as unstable when one capture read it two different ways across
+  its own revolutions, and says so beside the re-read command, because weak
+  bits do not settle down however many more passes you make.
+- Refuses to write the merged image or the report over one of the inputs.
+- Warns when a sector image is not the size the chosen format implies, since
+  the extra sectors get ignored and a missing tail counts as unread.
+- Takes only the sectors a truncated `.img` or `.adf` actually holds. The
+  readers pad a short image to the format length and mark what they invented
+  CRC-clean, which would otherwise win the merge with zeroes.
+- Rejects an output path it cannot write before reading any input rather than
+  after decoding all of them.
+- Calls a sector verified only when some read of it satisfied a check value off
+  the disk, and ends a merge of sector images by saying nothing confirmed them
+  instead of claiming a CRC did.
+- Flags a sector contested when two dumps both read it cleanly and disagreed,
+  keeps the best-ranked dump's copy, and names those sectors on stdout, since
+  nothing in a sector image can arbitrate between them.
+- Applies a repeated `--pll` to every flux container. KryoFlux streams and HFE
+  images previously took the first setting and ignored the rest.
+- Refuses a report path equal to the merged image.
+- Counts a KryoFlux input as the whole stream set rather than the single file
+  named on the command line.
+- Compares each run against the report it replaces and says how many sectors
+  the new capture recovered, or that nothing moved, which is the loop's stop
+  condition. Also in the report as `since_last_report`.
+- Stops calling a sector unstable when a repeated `--pll` read one revolution
+  two ways. Weak bits are revolutions disagreeing with each other, not two
+  decodes of the same revolution.
+- Rejects an input it cannot open, including a KryoFlux name that does not fit
+  the per-track pattern, before decoding any of the others.
+- Measures a KryoFlux set by the track files the reader will open, so a second
+  capture sitting in the same directory is not counted in.
+- Keeps decoded flux and the detected format in `.diskstack-cache` beside the
+  output, so adding a capture to the stack only decodes the new one. Three
+  captures of a 360K disk go from 6.5 seconds to 0.4 on a repeat run.
+  `--cache DIR` moves it, `--no-cache` turns it off, and every input says in
+  the report whether it was decoded or reused.

diskstack-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Christian Bosch
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

diskstack-1.0.0/MANIFEST.in

@@ -0,0 +1,10 @@
+include CHANGELOG.md
+graft examples
+graft tests
+graft tools
+
+# The captures are 19 MB each and are rebuilt by make_fixtures.py.
+prune tests/fixtures
+include tests/fixtures/make_fixtures.py
+
+global-exclude *.py[cod]

diskstack-1.0.0/NOTICE

@@ -0,0 +1,55 @@
+diskstack bundles part of Greaseweazle.
+
+    https://github.com/keirf/greaseweazle
+    commit 26690f89967d519e0106ab9566019a026b920bb4 (master, post-v1.23)
+
+Greaseweazle is released into the public domain under the Unlicense. The full
+text is in diskstack/_vendor/greaseweazle/COPYING.
+
+Greaseweazle is not published on PyPI, so its flux and sector decoders are
+vendored rather than depended on. tools/vendor.py regenerates the whole tree
+from that commit; nothing under diskstack/_vendor/ is edited by hand.
+
+What is vendored: src/greaseweazle/ minus usb.py, cli.py and tools/. TrackSet,
+range_str and columnify are extracted from tools/util.py into a standalone
+module because the rest of that file drives the USB hardware and needs
+pyserial.
+
+Local changes, all applied automatically by tools/vendor.py:
+
+  * imports rewritten from `greaseweazle.*` to
+    `diskstack._vendor.greaseweazle.*`
+
+  * greaseweazle/__init__.py is generated. It carries the vendored commit as
+    __version__ and imports codec.codec, because codec/codec.py imports every
+    codec at module scope and those codecs import back into codec.ibm.ibm, so
+    importing e.g. image.imd first lands on a partially-initialised module.
+    Upstream never hits this because its CLI always reaches codec.codec first.
+
+  * codec/ibm/ibm.py: the two `print("Unknown mark %02x")` warnings are
+    dropped. Damaged tracks produce them constantly and they would corrupt the
+    table diskstack prints on stdout.
+
+  * optimised/__init__.py: the two warnings about the missing C extension are
+    dropped, for the same reason. The pure-Python fallback is used and is fast
+    enough (about 75 ms per three-revolution track).
+
+  * image/imd.py: the reader now records the per-sector data-error flag
+    (rec & 4) and the data-unavailable code (rec == 0) as a failed CRC instead
+    of discarding both. An IMD full of error sectors is exactly the input
+    diskstack exists to merge.
+
+  * tools/util.py: TrackSet.TrackIter gets an __iter__ returning itself. An
+    iterator has to be iterable, and Python 3.13 enforces it, so without this
+    every `for track in diskdef.tracks` raises TypeError there.
+
+  * codec/codec.py: importlib.resources.open_text is replaced with
+    importlib.resources.files(...).joinpath(...).open(), which is the form
+    that still works once the package is installed as a wheel.
+
+The test fixtures download Transylvania.img from the fluxfox test corpus:
+
+    https://github.com/dbalsom/fluxfox/tree/main/tests/images/transylvania
+
+See the LICENSE.txt in that directory for its terms. Nothing from it is
+redistributed here; tests/fixtures/make_fixtures.py fetches it on demand.

diskstack-1.0.0/PKG-INFO

@@ -0,0 +1,329 @@
+Metadata-Version: 2.4
+Name: diskstack
+Version: 1.0.0
+Summary: Merge several dumps of one floppy disk into the best possible image, and say what is still missing
+Author-email: Christian Bosch <cbosch101@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Booyaka101/diskstack
+Project-URL: Repository, https://github.com/Booyaka101/diskstack
+Project-URL: Issues, https://github.com/Booyaka101/diskstack/issues
+Project-URL: Changelog, https://github.com/Booyaka101/diskstack/blob/main/CHANGELOG.md
+Keywords: floppy,flux,greaseweazle,supercard-pro,scp,preservation,disk-image,data-recovery,retrocomputing,amiga,mfm
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Archiving
+Classifier: Topic :: System :: Recovery Tools
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+License-File: diskstack/_vendor/greaseweazle/COPYING
+Requires-Dist: click>=8.1
+Requires-Dist: bitarray>=2.9
+Requires-Dist: crcmod>=1.7
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.16; extra == "dev"
+Dynamic: license-file
+
+# diskstack
+
+Merge several dumps of one floppy disk into the best possible image, and get
+told exactly what is still missing.
+
+## The loop
+
+Read the disk twice, stack the dumps, and diskstack hands you back a
+Greaseweazle command that re-reads only the tracks that are still bad:
+
+```
+$ diskstack disk_a.scp disk_b.scp -o merged.img
+
+720 sectors: 714 clean (from 2 files), 6 unresolved
+
+Tracks needing attention
+  cyl  head  clean  voted  unresolved  missing  sector ids still bad
+   17     0      7      0           2        0  4,7
+   18     0      7      0           2        0  4,7
+   19     0      7      0           2        0  4,7
+
+Re-read just these tracks, then run diskstack again with the new capture added:
+  gw read retry.scp --tracks=c=17-19:h=0
+```
+
+Three tracks instead of eighty, so the drive spends seconds on the media
+rather than minutes. Feed the new capture back in:
+
+```
+$ gw read disk_c.scp --tracks=c=17-19:h=0
+$ diskstack disk_a.scp disk_b.scp disk_c.scp -o merged.img
+
+720 sectors: 714 clean (from 3 files), 6 recovered by vote
+Since the last report: 6 recovered.
+
+Every sector confirmed by CRC. Nothing left to re-read.
+```
+
+That run produced a file byte-identical to the original disk image. Repeat
+until the unresolved count reaches zero or stops moving. diskstack reads the
+report it is about to overwrite, so it tells you which of the two happened:
+either how many sectors the new capture recovered, or that nothing changed and
+another pass at the same settings is unlikely to. It also keeps the decoded
+flux, so the second run only decodes the capture you just added: re-running
+the three-capture merge above takes 0.4 seconds against 6.5 the first time.
+The full session is in
+[examples/console-session.txt](examples/console-session.txt) and the report it
+wrote is in [examples/diskstack-report.json](examples/diskstack-report.json).
+
+## Install
+
+```
+pip install diskstack
+```
+
+Python 3.11 or newer. No hardware, no network, no configuration.
+
+Until the PyPI upload lands, `pip install
+git+https://github.com/Booyaka101/diskstack` gets you the same thing.
+
+## Use
+
+```
+diskstack disk_a.scp disk_b.scp disk_c.img -o merged.img
+```
+
+Inputs are two or more dumps of the same disk, in any mix of:
+
+| what | extensions |
+| --- | --- |
+| flux | `.scp` (SuperCard Pro), `.raw` (KryoFlux streams), `.hfe` (HxC) |
+| sector images | `.img`, `.ima`, `.st`, `.adf`, `.imd` |
+
+For KryoFlux, name any one of the per-track stream files and the rest of the
+set is picked up from the directory.
+
+Output is a sector image: `.img`, `.ima`, `.st`, `.adf` or `.imd`. Flux goes in
+but never comes out. Along with it you get `diskstack-report.json` next to the
+image and the table above on stdout.
+
+Formats understood are IBM FM and MFM (PC 160K through 2.88M, Atari ST) and
+AmigaDOS MFM. `diskstack --list-formats` prints the list. Detection is
+automatic; `--format ibm.720` overrides it.
+
+The exit code is 0 when every sector is confirmed and 2 when something is still
+unresolved, so a retry loop can be scripted.
+
+## How it decides
+
+Every sector of every revolution of every input is a separate read attempt.
+Attempts are grouped by the `(cylinder, head, sector id)` in their ID address
+mark, never by where they sit in the flux, because two captures of the same
+disk do not line up: the index phase differs and so does the motor speed.
+
+Per sector, in order:
+
+1. **clean** - an attempt whose own CRC or checksum passes. Taken as is.
+2. **recovered by vote** - no attempt passed, so the attempts get recombined
+   into candidate payloads and the first one that satisfies a check value off
+   the disk wins. Three routes are tried, best evidence first:
+   `majority` votes byte by byte across every attempt, `per_source_majority`
+   collapses each input file to one payload first so a capture with more
+   revolutions cannot outvote the files that read the sector correctly, and
+   `cross_check` tries each individual read as it stands, which is what
+   recovers a sector whose payload was fine and whose own CRC bytes were the
+   damaged part. The CRC bytes came off the same damaged track, so they get
+   voted on too and the voted check value is tried alongside the ones read.
+   Ties go to the dump that read the rest of the disk best. The report says
+   which route it was.
+3. **unresolved** - the vote still failed. The largest cluster of attempts that
+   agree exactly is written out, the sector is listed in the table, and its
+   track goes into the re-read command.
+
+A sector that is good in any input is never worse in the output. There is a
+test for that.
+
+Only a handful of reconstructions are tried, deliberately. A CRC16 accepts the
+wrong payload once in 65536 tries, so every extra candidate thrown at the check
+value buys recovery at the price of a small chance of confidently writing out
+garbage. Three routes that each mean something beats a brute-force sweep over
+the contested bytes.
+
+The check value has to come off the disk, which means a stack of only sector
+images can never reach step 2. `.img` and `.adf` files carry data and nothing
+to verify it with, so identical bytes in two images prove only that both dumps
+read the same thing. Mixing in one flux capture is what gives the vote
+something to check against.
+
+That distinction runs through the whole report. A sector is `verified` when
+some read of it satisfied a check value off the disk, and a merge of sector
+images alone never gets that, so the closing line says the sectors came from
+inputs that carry no check value instead of claiming a CRC confirmed them. A
+sector is `contested` when two dumps both looked good and held different
+bytes. Nothing in a sector image can arbitrate that, so diskstack keeps the
+best-ranked dump's copy, prints which sectors it happened to, and leaves the
+judgement to you. It is what deliberate weak bits look like, and also what two
+dumps of two different disks look like.
+
+A sector the merge could not confirm is marked `unstable` if one capture read
+it two different ways across its own revolutions. That is what weak bits look
+like, and they are usually deliberate, so more passes over the disk will not
+settle them. The stdout note says so next to the re-read command. Two `--pll`
+settings reading one revolution two ways is a different thing, a decode that
+went marginal rather than a disk that moved, and does not count.
+
+Sectors Greaseweazle filled with `-=[BAD SECTOR]=-` are treated as failed
+reads, not as data, so a `.img` from an earlier bad session still contributes
+its good sectors. `--keep-filler` turns that off.
+
+A truncated `.img` or `.adf` contributes only the sectors the file actually
+holds bytes for. The readers pad a short image out to the format's length and
+mark every sector they invented as CRC-clean, so without that check a dump that
+stopped halfway would win the merge with 512 bytes of nothing per sector.
+
+## Options
+
+```
+  -o, --output PATH               Merged sector image to write (.img, .ima,
+                                  .st, .adf, .imd).  [required]
+  -r, --report PATH               JSON report path  [default: diskstack-
+                                  report.json beside the output]
+  --no-report                     Do not write the JSON report.
+  -f, --format TEXT               Disk format, e.g. ibm.720 or amiga.amigados.
+                                  Detected from the inputs when not given.
+  --list-formats                  Print the disk formats diskstack understands
+                                  and exit.
+  --pll SPEC                      Flux PLL settings, e.g.
+                                  period=5:phase=60:lowpass=1.5. Repeat to
+                                  decode each capture several ways and let the
+                                  results compete.
+  --revs N                        Use only the first N revolutions of each
+                                  flux capture [default: all of them]  [x>=1]
+  --keep-filler                   Trust sectors written as '-=[BAD SECTOR]=-'
+                                  filler instead of treating them as failed
+                                  reads.
+  --fill-unresolved [best|filler|zero]
+                                  What to write for a sector no input
+                                  confirmed: its best guess, bad-sector
+                                  filler, or zeroes.  [default: best]
+  --no-vote                       Skip the byte-wise majority vote; keep only
+                                  sectors whose own CRC passes.
+  -j, --jobs N                    Worker processes for decoding flux [default:
+                                  one per core, up to 8]  [x>=0]
+  --cache DIR                     Where to keep decoded flux between runs
+                                  [default: .diskstack-cache beside the
+                                  output]
+  --no-cache                      Decode every flux capture again instead of
+                                  reusing the decode a previous run stored.
+  --retry-name NAME               Filename used in the printed gw read
+                                  command.  [default: retry.scp]
+  -q, --quiet                     Print the summary and the re-read command,
+                                  no tables.
+  -V, --version                   Show the version and exit.
+  -h, --help                      Show this message and exit.
+```
+
+Decoding flux is the slow part, so it runs one track per core by default. A
+three-capture 40-cylinder merge takes about six seconds on eight cores against
+half a minute on one. `--jobs 1` forces the single-process path, which is also
+what happens automatically if the worker pool cannot start.
+
+`--pll` is worth knowing about. Repeating it decodes each flux capture several
+ways, and since two PLL settings disagree about different marginal bitcells,
+they become independent attempts that the vote can use:
+
+```
+diskstack a.scp b.scp -o merged.img --pll period=5:phase=60 --pll lowpass=1.5
+```
+
+It applies to every flux container, `.scp`, `.raw` and `.hfe` alike.
+
+Decoded flux is kept in `.diskstack-cache` beside the output, along with the
+detected disk format, so a re-run only pays for the captures it has not seen.
+An entry is named for a hash of the input's bytes and of every setting that
+changes the decode, which means a fresh dump written over the same filename is
+a new entry rather than a stale hit, and a capture that moved or was renamed
+still counts as one it has seen. `--cache DIR` puts it somewhere else,
+`--no-cache` turns it off, and deleting the directory costs nothing but the
+next decode. It drops its least recently used entries once it passes 512 MB;
+three captures of a 360K disk take 377 KB.
+
+## The report
+
+`diskstack-report.json` carries one entry per sector of the disk: its status,
+how many attempts it got, how many agreed, and which file and revolution each
+contributing read came from. Plus per-input totals and the re-read trackspecs.
+`schema` is versioned and starts at 1.
+
+```json
+{
+  "cyl": 17, "head": 0, "sec_id": 4, "size": 512,
+  "status": "unresolved", "attempts": 6, "good": 0, "agreement": 3,
+  "discarded": 0, "unstable": false, "contested": false, "verified": false,
+  "method": null,
+  "sources": [{"path": "capture_a.scp", "rev": 0}]
+}
+```
+
+`method` names the route that rebuilt a recovered sector and is null for any
+other status. `discarded` counts attempts thrown away because they decoded to
+the wrong length for the sector. `verified` and `contested` are the two above,
+and `totals` counts both across the disk.
+
+`since_last_report` holds the `recovered`, `lost` and `still_bad` counts
+against the report this run replaced, or null on the first run and whenever
+the old report covers a different set of sectors. Each input carries `cached`,
+saying whether its sectors were decoded on this run or reused from an earlier
+one.
+
+## What it does not do
+
+v1 is deliberately narrow. No GCR, so no Apple II and no Commodore. No writing
+flux back out. No hardware access, so nothing here talks to a Greaseweazle
+or a KryoFlux. No copy-protection preservation: this produces sector images,
+which is the wrong container for weak bits and long tracks. No network calls.
+
+## Validation, honestly
+
+The three-capture test builds genuine SuperCard Pro flux from a real PC floppy
+image, damages each capture a different way, gives each its own index phase and
+its own motor speed within 1.5 percent, and asserts the merged output is
+byte-identical to the original. The AmigaDOS codec gets the same treatment one
+track at a time, since the fixture disk is a PC one. That is a real decode of
+real flux through Greaseweazle's own decoders, but the damage is synthetic. It
+has not yet been run against dumps of a genuinely deteriorated disk.
+
+If you have a disk that needs several passes, I would like to see the report.
+Open an issue with the `diskstack-report.json` and what the disk is.
+
+## Tests
+
+```
+pip install -e ".[dev]"
+python tests/fixtures/make_fixtures.py     # downloads 360 KB, writes 55 MB
+python -m pytest tests -q
+ruff check .
+```
+
+The fixture builder needs the network once. Everything else runs offline.
+
+## Built on Greaseweazle
+
+The flux and sector decoders are [Greaseweazle's](
+https://github.com/keirf/greaseweazle), vendored under
+`diskstack/_vendor/greaseweazle` because Greaseweazle is not published on PyPI.
+It is public domain under the Unlicense. `tools/vendor.py` regenerates the tree
+from a pinned commit and applies every local change; `NOTICE` lists them.
+
+Runtime dependencies are `click`, `bitarray` and `crcmod`. The last two are
+what the vendored decoders need.
+
+## License
+
+MIT. See `LICENSE`, and `NOTICE` for the vendored code.