pieusb 0.3.2__tar.gz → 0.3.7__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
pieusb-0.3.7/CHANGELOG ADDED
@@ -0,0 +1,230 @@
1
+ # v0.3.7
2
+
3
+ - **The `exp_rel_*` ceiling is a 16-bit Timer 1 overflow, and is now derived rather
4
+ than guessed.** The relative exposure scales the *device's own* absolute exposure
5
+ time into a 16-bit count:
6
+
7
+ timer = exposure_time * exp_rel / 100 (mod 65536)
8
+
9
+ so the usable maximum is `65535 * 100 // exposure_time` -- 1598 for the 4100 a
10
+ ProScan 10T reports. Past it the register wraps and the exposure *drops*, and a
11
+ wrapped result below 100 meets the same floor that makes `exp_rel=50` behave as
12
+ 100. A binary search puts the step at exactly 1598/1599, for all three channels,
13
+ which is where the product crosses 65535. The model predicts every measurement
14
+ taken to within 4%:
15
+
16
+ exp_rel 1086 1500 | 1598 1599 | 2200 3300 5000 7500 10000
17
+ predicted x10.9 x15.0 | x15.98 x1.00 | x6.02 x1.03 x2.05 x11.1 x4.09
18
+ measured x10.5 x14.7 | clip x1.00 | x5.79 x1.06 x2.01 x10.7 x3.94
19
+
20
+ New `option.max_relative_exposure(exposure_time)`; `MAX_TESTED_RELATIVE_EXPOSURE`
21
+ (a hard-coded 1500, itself a raised guess from 1086) is replaced by
22
+ `MAX_RELATIVE_EXPOSURE`, the ceiling implied by `OBSERVED_EXPOSURE_TIME`, used by
23
+ `OptionsTable.validate()` which has no device reading available
24
+ - `auto_exp` clamps each channel to the ceiling the device's own reported exposure
25
+ time implies, per channel, and reports the shortfall. A scanner that has not
26
+ warmed up reports no exposure time at all, which says nothing about the register
27
+ it uses, so the observed ceiling is assumed and said out loud rather than
28
+ abandoning the correction
29
+ - This also explains why `exp_time_*` looked inert but is not irrelevant: the device
30
+ ignores what is written to it and keeps its own value, and that value is the
31
+ multiplicand deciding where `exp_rel_*` overflows
32
+
33
+ # v0.3.6
34
+
35
+ - **`auto_exp` now meters, and it works.** It runs a preview pass at the device's
36
+ preview resolution with `exp_rel_*` at 100, takes the 99th percentile of each
37
+ colour plane, and scales that channel's relative exposure to land it at 85% of
38
+ full scale. Per channel, no cap on the correction beyond the measured-linear
39
+ range, and no dependency on anything the device reports about itself. On a PIE
40
+ ProScan 10T against a colour negative it derives 247/563/1086, which puts all
41
+ three channels within 1% of each other at ~88% of full scale with nothing
42
+ clipped -- where untouched they sit at 4.39 : 1.93 : 1.00 with red at 34%
43
+ - **`exp_rel_*` is the exposure control on this hardware, and `gain_*`,
44
+ `exp_time_*` and `light` are inert.** Measured by sweeping each in isolation at
45
+ 300 dpi / 16-bit: gain 10/30/60, exp_time 500/1500/2500/2937/6000/10000 and
46
+ light 4/5/6/7 all produce the same image to within 0.2%, while `exp_rel_*` is
47
+ linear to within 4% and clamps at the bottom -- 50 behaves as 100. The top is set
48
+ by a Timer 1 overflow, explained in v0.3.7. `exp_rel_*` is the
49
+ per-line integration period, so the line rate halves as it doubles. SANE
50
+ hard-codes all three to 100 and never varies them (`pieusb.c:878-882`), which is
51
+ why neither backend could ever expose this film
52
+ - **`auto_exp` no longer adopts the device's own calibration.** `gain_*`,
53
+ `exp_time_*`, `offset_*` and `light` from GET GAIN OFFSET are inert, so the
54
+ v0.3.5 behaviour was a no-op that cost a command. `Scanner._adopt_device_calibration`
55
+ and `_set_from_device` are gone; `CALIBRATION_OPTIONS` is replaced by
56
+ `METERED_OPTIONS`, the three `exp_rel_*` names, which are restored after the scan
57
+ that derived them. The v0.3.5 note that the light byte was "worth roughly the
58
+ entire exposure range" was wrong -- it came from a comparison confounded by
59
+ resolution. Sending 0 was still incorrect and 4 is still the right default
60
+ - Shading correction counts and warns about clamped samples. Clipping is not
61
+ uniform: the per-column gain is >1 where the lamp falls off, so edge columns
62
+ reach the ceiling at a lower raw value than centre columns, and highlights near
63
+ full scale clip column by column and read as vertical banding rather than flat
64
+ white. Measured at `exp_rel` 400, red clipped 12.7% of samples, concentrated at
65
+ the left and right edges with 55 percentage points of column-to-column swing
66
+ - `OptionsTable.validate()` no longer warns on every `exp_rel_*` away from 100 --
67
+ it is meant to be moved. It now warns only above the Timer 1 ceiling (see v0.3.7)
68
+ or below 100, where the device clamps and the setting does nothing. Warned about
69
+ rather than refused, since the real ceiling depends on a value only Scanner reads
70
+ - The per-scan settings line now includes the relative exposures, since `auto_exp`
71
+ restores them afterwards and the log is the only record of what a scan ran with
72
+ - The high-resolution stall is a device characteristic with no host-side fix: the
73
+ host is unattended for 1.9% of a 5000 dpi scan, the device's buffer never fills
74
+ (it drains to 0 every fifth read), and the C backend stalls identically at
75
+ matched bit depth. It is also self-correcting -- raising `exp_rel_*` slows the
76
+ line rate to match the readout, and a 5000 dpi scan at x11.1 took 150 s against
77
+ 190 s at 100%, because the reversals stop
78
+
79
+ # v0.3.5
80
+
81
+ - **`auto_exp` is now "scan with the calibration the scanner reports for itself"**,
82
+ and no longer runs a preview pass. It reads GET GAIN OFFSET and sends the gain,
83
+ exposure, offset and light it finds there -- `SCAN_CALIBRATION_AUTO`
84
+ (`pieusb_specific.c:1988`), which is the C backend's *default* calibration mode
85
+ (`:733`) and what its working scans run with. The firmware optimises those values
86
+ per channel while warming up, on this lamp at its current temperature, so they
87
+ beat anything derivable from outside. Without `auto_exp` the options are sent
88
+ exactly as set, unchanged
89
+ - **Removed the "calibration from preview" port**, along with `pieusb.calibration`
90
+ and the extra pass it cost. That path divides by `settings.saturationLevel`, and
91
+ this hardware never populates it: GET GAIN OFFSET reports `0-0-0` for it cold and
92
+ warm alike -- confirmed on a PIE ProScan 10T, in the C backend's own captured
93
+ response (`pieusb_scancmd.c:1136-1146`), and in a SANE trace of the C reading it.
94
+ A zero there makes `dg` 0, and `updateGain2` turns a `dg` of 0 into gain 0 and
95
+ exposure 0. There was nothing for a preview to meter against, so there is no
96
+ longer a preview. `ScanPhase.METERING` is kept for compatibility but never
97
+ emitted; a scan now reports one sweep of the phases, not two
98
+ - **Fixed the light level being sent as 0 on every scan.** Byte 15 of SET GAIN
99
+ OFFSET is the lamp level; it was hard-coded to 0 with a `# Light, maybe in SANE
100
+ is 5?` comment. The C never sends 0 -- AUTO echoes back what GET GAIN OFFSET
101
+ reported at byte 75 (`pieusb_scancmd.c:1048`, `1111`), and the DEFAULT and OPTIONS
102
+ modes send `DEFAULT_LIGHT`/`OPT_LIGHT`, both 4. The documented band is 4..7: the
103
+ firmware starts at 7 or 6 and decrements as the lamp warms, settling at 4, where
104
+ "the scanner produces stable scans" (`pieusb_scancmd.h:208-213`). This
105
+ under-exposed *every* scan, manual ones included. Measured on hardware: a scan at
106
+ the *minimum* exposure (2937) with light 4 reaches the same level as one at
107
+ maximum exposure with light 0 -- the byte was worth roughly the entire exposure
108
+ range
109
+ - New `light` option, default 4, so the lamp level is settable and appears in the
110
+ option table. `auto_exp` adopts the scanner's own value over it
111
+ - `extraEntries` (byte 16) and `doubleTimes` (byte 17) are both sent as 0, which is
112
+ what a backend trace of a working C scan shows going out with the device's own
113
+ gain and exposure. Not the `DEFAULT_ADDITIONAL_ENTRIES` of 1 from
114
+ `pieusb_specific.h:108`: only the DEFAULT and OPTIONS calibration modes assign
115
+ that, while AUTO leaves the field alone because `get_gain_offset()` does not
116
+ decode it. The SET GAIN OFFSET payload now matches a captured C payload byte for
117
+ byte
118
+ - A cold scanner reports zeros for its whole calibration -- exposure
119
+ `(0, 0, 0, 4100)`, gain `(0, 0, 0, 15)`, light 0 -- and fills the real values in
120
+ during its first scan. `auto_exp` detects that, warns, and scans with the options
121
+ as set rather than adopting zeros. Practical consequence: the first `auto_exp`
122
+ scan of a session is not calibrated, so scan twice and discard the first
123
+ - Every scan logs the exposure, gain, offset and light it actually sent, at INFO.
124
+ `auto_exp` restores those options afterwards, so the log is the only place the
125
+ values a scan ran with can be seen after the fact
126
+ - `tests/exercise.py`: new `--gain-offset` prints the scanner's own saturation
127
+ levels, exposure times, gain, offset and light (pair with `--dry-run` for the
128
+ readout alone). Console logging now shows INFO, which is where the line above
129
+ lands; `-v` is still needed for the per-command DEBUG chatter
130
+ - `tests/exercise.py` also writes `<prefix>_rgb.tif` (and `_ir.tif`) next to the
131
+ .npy: uncompressed TIFF at the scan's own bit depth, pixels exactly as scanned.
132
+ Hand-rolled, because the package has no image-library dependency and Pillow
133
+ cannot write 16-bit RGB TIFF -- it drops the low byte of every sample, and
134
+ mis-reads such files too, which is worth knowing if you have been inspecting
135
+ scans through it
136
+
137
+ # v0.3.3
138
+
139
+ - Relaxed python and numpy minimum requirements
140
+
141
+ # v0.3.2
142
+
143
+ - Made it possible to set the resolution to the maximum reported by the inquiry
144
+
145
+ # v0.3.1
146
+
147
+ - **Breaking:** `UpdateData.scanned_lines`/`total_lines` are replaced by a single
148
+ `progress` float, 0.0 -> 1.0 within the current `phase`. Line counts only ever
149
+ meant anything during SCANNING, which left every other phase reporting `None`
150
+ and each consumer writing the same division; a fraction is what a progress
151
+ indicator wants and it now applies to every phase. CALIBRATING reports the
152
+ shading read's fraction and WARMING_UP its progress through the START SCAN
153
+ retry budget; phases whose length the device does not announce report 0.0
154
+ - **Breaking:** the `calibrate` option is replaced by `reuse_calibration`, with
155
+ the opposite polarity and the same default behaviour (`calibrate=True` is
156
+ `reuse_calibration=False`, still the default). The old name promised something
157
+ it could not deliver: skipping calibration is a request, not a decision, and
158
+ the option now says so
159
+ - Skipping calibration is honoured, where before it changed the SET MODE quality
160
+ bit and nothing else. A pass acquires a shading reference unless
161
+ `reuse_calibration` is set AND one is already cached; the scanner can still
162
+ refuse, which it signals by answering START SCAN with MUST_CALIBRATE, and then
163
+ the pass calibrates anyway. `ScanPhase.CALIBRATING` is reported only when a
164
+ pass actually calibrates
165
+ - Shading references are cached on the `Scanner` for the life of the device
166
+ session, so a pass that skips calibration corrects from the one an earlier pass
167
+ read. Following the C backend, which keeps `shading_ref`/`shading_mean` on the
168
+ open device handle (`pieusb_specific.h:292-294`). Practical consequence: keep
169
+ one `Scanner` open across a batch, since a fresh one starts with a cold cache
170
+ and calibrates on its first scan regardless of the option. The cache is dropped
171
+ if the device starts reporting a different shading width, and an unusable read
172
+ keeps the previous reference rather than falling back to raw pixels
173
+ - With `auto_exp`, the metering pass calibrates and the real pass reuses that
174
+ reference: one calibration per scan instead of two
175
+ - `set_options()` takes `skip_shading_analysis` as a keyword argument, since
176
+ whether a pass can skip depends on the Scanner's cache and not on an option
177
+ alone
178
+ - The scan sequence proper is now `Scanner._scan_pass(started, emit)`, shared by
179
+ the real scan and the auto-exposure metering pass, which differ only in the
180
+ options in force and in the `emit` callback that labels their progress.
181
+ Replaces the `_phase_override` attribute that relabelled updates inside
182
+ `_emit()`
183
+ - Implemented the `auto_exp` option, which until now warned and did nothing. It
184
+ runs a preview pass at the device's preview resolution, measures the 99th
185
+ percentile of each colour plane against the CCD saturation levels, and derives
186
+ new `gain_*`/`exp_*` settings before the real scan. A port of the SANE
187
+ backend's "calibration from preview" path (`sanei_pieusb_analyze_preview`,
188
+ `sanei_pieusb_set_gain_offset`, `updateGain2`), in the new
189
+ `pieusb.calibration` module. The preview pass reports its progress under a new
190
+ `ScanPhase.METERING` so it is distinguishable from the real scan
191
+ - `_get_gain_offset()` now also returns the `saturation_level` triple that
192
+ auto-exposure meters against (GET GAIN OFFSET byte 54)
193
+ - **Breaking:** split the exposure options, which conflated two independent
194
+ device controls behind one name. `exp_r`/`exp_g`/`exp_b`/`exp_i` are gone;
195
+ there are now two families:
196
+ - `exp_time_r`/`_g`/`_b`/`_i` -- ABSOLUTE integration time in Timer 1 counts,
197
+ carried by SET GAIN OFFSET. This is the real exposure control and the one
198
+ auto-exposure moves. Its default changes from 100 to 2937 (SANE's
199
+ DEFAULT_EXPOSURE, the value the firmware itself falls back to) and its upper
200
+ bound is the inquiry maximum times 4 -- without which the device's own
201
+ default is outside the range the device reports
202
+ - `exp_rel_r`/`_g`/`_b` -- RELATIVE exposure percentage, sent by the
203
+ SCSI_EXPOSURE write. No infrared entry, because the device has none. **Leave
204
+ this at 100.** SANE hard-codes it and exposes no option for it, so no other
205
+ value has been exercised against this hardware; it is redundant with
206
+ `exp_time_*`; and auto-exposure's saturation reference assumes it is at 100.
207
+ `OptionsTable.validate()` warns once per scan if it is moved
208
+ The previous single option fed *both* commands, so the value written to the
209
+ SET GAIN OFFSET exposure time was the relative percentage's 100 rather than an
210
+ exposure time. Existing callers setting `exp_*` must rename to `exp_time_*`
211
+ - Added `Unit.PERCENT` and `Unit.TIMER_COUNTS`; exposure times are not
212
+ expressible in microseconds without the Timer 1 clock rate, which the device
213
+ does not report
214
+
215
+ # 0.2.0
216
+
217
+ - Added wait() API to block until the scan is finished
218
+ - Added close(), which cancels a running scan and waits for the worker before
219
+ releasing the USB interface. `with Scanner(...)` now goes through it, so
220
+ leaving the block mid-scan no longer closes the device under the worker
221
+ - Fixed Scanner construction: the option attribute accessors recursed on
222
+ `self.params` before the option table existed, so `Scanner(info)` always
223
+ raised RecursionError
224
+ - Reading an option attribute returns its value rather than the internal
225
+ Parameter, and an unknown name raises AttributeError rather than KeyError
226
+ - Setting an option while a scan is running raises ScanInProgress
227
+
228
+ # 0.1.0
229
+
230
+ Initial release
@@ -0,0 +1,424 @@
1
+ # High-resolution scan stalls — concluded investigation
2
+
3
+ **Concluded:** 2026-08-11
4
+ **Hardware:** PIE ProScan 10T, rev 1.70, native 10000 dpi, bed 14370 × 9570 native units (36.5 × 24.3 mm)
5
+
6
+ ## Outcome
7
+
8
+ The stall is a device characteristic with no host-side fix — and it fixed itself as
9
+ a side effect of the exposure work.
10
+
11
+ - The host is unattended for **1.9%** of a 5000 dpi scan. It is not the cause.
12
+ - The device's buffer **never fills**; it drains to 0 every fifth read.
13
+ - The C backend stalls identically at matched bit depth.
14
+ - The scanner reads out at ~1.1 MB/s regardless of resolution, and the reversals
15
+ are what hold it there.
16
+ - **Raising `exp_rel_*` slows the line rate to match the readout and the reversals
17
+ stop.** A 5000 dpi 16-bit scan at ×11.1 took **150 s against 190 s** at 100%.
18
+
19
+ So the correct exposure for a colour negative is also the exposure at which the
20
+ mechanism runs smoothly. `auto_exp` now derives it (v0.3.6), which means this needs
21
+ no further action.
22
+
23
+ The instrumentation written to measure all this was temporary and has been removed;
24
+ §3 records what it was, in enough detail to rebuild if a similar question comes up.
25
+
26
+ ---
27
+
28
+ ## 1. Symptom
29
+
30
+ At 5000 dpi the scanner pauses mid-scan, backs the carriage up, and re-scans the
31
+ same few lines. It does not happen at low resolutions. Working hypothesis: the
32
+ device's internal line buffer fills because the host does not drain it fast
33
+ enough, and the firmware recovers by reversing to re-acquire the lines it could
34
+ not buffer.
35
+
36
+ Nothing is known to be *wrong* with the resulting images — the symptom is time
37
+ lost and mechanical wear, plus the risk of duplicated or shifted lines.
38
+
39
+ **This hypothesis is wrong.** Measured, the host is unattended for 1.9% of a
40
+ 5000 dpi scan. See §4.
41
+
42
+ ## 2. Why the hypothesis looked plausible
43
+
44
+ > The per-read overhead table below is accurate and still the best wire-level
45
+ > reference. The *reasoning* built on it — resolution scaling and the host-side
46
+ > contributors — was disproved in §4.
47
+
48
+ ### Fixed per-read protocol overhead
49
+
50
+ Every `GET SCANNED LINES` (max 255 lines) costs the following before one pixel
51
+ byte moves, all in `src/pieusb/transport.py`:
52
+
53
+ | Phase | Cost | Location |
54
+ |---|---|---|
55
+ | IEEE-1284 handshake | 11 control transfers **+ unconditional 3 ms `sleep`** | `ieee_command`, :139-146 |
56
+ | CDB | 6 control transfers, one byte each | `scsi_transaction`, :196-197 |
57
+ | Intermediate status | 1 control transfer | :199 |
58
+ | Bulk size | 1 control transfer | `_bulk_size`, :114 |
59
+ | **Bulk data** | `ceil(bytes / 16 KiB)` reads | `_bulk_in`, :124 |
60
+ | Final status | 1 control transfer | :209 |
61
+
62
+ ≈20 control round-trips plus 3 ms of sleep per read. **This is faithful to the C
63
+ backend** — the 3 ms is `pieusb_usb.c:477`, the 255-line cap is
64
+ `pieusb_specific.c:2248`, the 16 KiB bulk chunking is `pieusb_usb.c:667`. It is
65
+ not a porting bug.
66
+
67
+ ### The resolution scaling ~~argument~~ (WRONG)
68
+
69
+ The gap between reads is roughly constant in wall-clock time, but bytes per line
70
+ grow linearly with dpi. The device buffer is a fixed number of *bytes*, so it
71
+ holds proportionally fewer *lines* as resolution rises, while the carriage
72
+ delivers lines faster.
73
+
74
+ Measured: the gap is indeed constant (~66 ms at both resolutions) but it is far
75
+ too small to matter — 1.9% of a 5000 dpi scan. The premise held; the conclusion
76
+ did not.
77
+
78
+ Estimated 5000 dpi rgb 16-bit full-frame scan (derived from the INQUIRY bed; the
79
+ real geometry comes from `GET PARAMETERS` at scan time):
80
+
81
+ - width ≈ 7185 px, height ≈ 4785 lines, 3 planes → ≈14355 tagged lines
82
+ - 14372 bytes/line → 255-line read ≈ **3.66 MB**, ≈**57 reads**, ≈206 MB total
83
+
84
+ For comparison, at 300 dpi a line is ~860 B — roughly 16× fewer bytes per line.
85
+
86
+ ### Host-side additions to the gap (ALL NEGLIGIBLE)
87
+
88
+ Three suspects, all measured and all noise:
89
+
90
+ 1. **`self._emit(...)` per chunk** calls `on_update` on the worker thread — in
91
+ `tests/exercise.py`, terminal I/O. Measured **0.1 ms**; the 5000 dpi runs with
92
+ and without the progress bar are identical to three digits.
93
+ 2. **`bytes(out)` in `_bulk_in`** copies ~3.66 MB per read. ~1 ms out of 4330.
94
+ 3. **The per-line Python decode loop**, 255 iterations plus page faults on the
95
+ 206 MB `planes` array. Measured **1.0 ms** total per read.
96
+
97
+ ## 3. Instrumentation used (since removed)
98
+
99
+ Written for this investigation, kept out of the repository once it had answered the
100
+ question. Not committed, so it exists only as this description.
101
+
102
+ | Piece | What it did |
103
+ |---|---|
104
+ | `src/pieusb/profile.py` | `ReadRecord` / `ReadProfile`: one record per pixel read, a `summary()` for the log and `write_csv()` |
105
+ | `UASDevice.scsi_transaction` | filled `dev.last_timing` with per-phase timings — `ieee_s`, `cdb_s`, `status_s`, `bulk_size_s`, `bulk_s`, `bulk_bytes`, absolute `bulk_start`/`bulk_end`, `setup_s`, `total_s` — plus a `dev.transactions` counter so a retried command was distinguishable |
106
+ | `UASDevice.link_info()` | reported the bulk endpoint's `wMaxPacketSize` (64 = full speed, 512 = high) |
107
+ | `transport.BULK_CHUNK` | made overridable to test whether the data phase was chunk-limited |
108
+ | `Scanner._scan_pass` | built a `ReadProfile` in the pixel loop and logged the summary at INFO |
109
+ | `Scanner.probe_device` | polled GET PARAMETERS after every read for `available_lines` and the motor bit |
110
+ | `Scanner._read_state()` | ported READ STATE (0xDD) |
111
+ | `Scanner._get_scan_parameters()` | decoded the motor-direction bit at byte 16 |
112
+ | `tests/exercise.py` | `--profile FILE`, `--probe-device`, `--quiet-progress`, `--bulk-chunk KIB`, a `USB link:` line |
113
+
114
+ The field that mattered was **`gap_s`**: end of one bulk transfer to the start of
115
+ the next — the window in which the device is scanning and nothing is draining it.
116
+ Not a sum of the other fields; it spans the previous cycle's decode and callback
117
+ plus this cycle's setup.
118
+
119
+ Two findings about the device's own status commands are worth keeping:
120
+
121
+ - **`GET PARAMETERS` byte 14 = `available_lines`** is the buffer-fill gauge,
122
+ non-zero only mid-scan (`pieusb_scancmd.h:140`), and byte 16 bit 0 is the motor
123
+ direction (`:141`), which the C declares and never decodes. Both readable
124
+ mid-scan. `available_lines` survives in `_get_scan_parameters()`.
125
+ - **`READ STATE` (0xDD) is rejected mid-scan** by this device: CHECK CONDITION,
126
+ ILLEGAL REQUEST / INVALID COMMAND OPERATION CODE (key 0x05, code 0x20). The C
127
+ only issues it at the top of `sane_start`, before START SCAN.
128
+
129
+ ## 4. RESULTS — measured 2026-08-11
130
+
131
+ Raw CSVs and logs were kept out of the repository; the numbers below are the whole
132
+ of it. **The buffer-starvation hypothesis is wrong.** The host is
133
+ idle almost the whole scan.
134
+
135
+ | | 1000 dpi | 5000 dpi | 5000 dpi + progress bar |
136
+ |---|---|---|---|
137
+ | reads | 12 | 57 | 57 |
138
+ | MB/read | 0.73 | 3.66 | 3.66 |
139
+ | total | 8.3 MB in 6.9 s | 206 MB in 192.9 s | 206 MB in 192.9 s |
140
+ | `setup_s` mean | 59.5 ms | 59.7 ms | 59.7 ms |
141
+ | `bulk_s` mean | 513 ms | 3319 ms | 3319 ms |
142
+ | `decode_s` mean | 0.5 ms | 1.0 ms | 1.0 ms |
143
+ | `emit_s` mean | 0.0 ms | 0.0 ms | 0.1 ms |
144
+ | **`gap_s` mean** | **65.7 ms** | **66.5 ms** | **66.5 ms** |
145
+ | **unattended** | **10.5%** | **1.9%** | **1.9%** |
146
+ | aggregate rate | 1.20 MB/s | 1.07 MB/s | 1.07 MB/s |
147
+ | retries | 0 | 0 | 0 |
148
+
149
+ ### What this kills
150
+
151
+ - **The gap is 66 ms and 1.9% of a 5000 dpi scan.** The host cannot be
152
+ overflowing the device's buffer; it spends 98% of the scan blocked inside
153
+ `ep_in.read()`.
154
+ - **`decode_s` is 1 ms and `emit_s` is 0.1 ms.** The progress bar and the
155
+ per-line Python loop are noise. The 5000 dpi runs with and without the bar are
156
+ identical to three digits. Every host-side fix considered was pointless — the
157
+ ~1 ms `bytes(out)` copy included.
158
+ - **The gap is constant at ~66 ms across a 25× change in data volume**, so it is
159
+ pure protocol overhead and does not scale with resolution. It matters *more* at
160
+ low resolution (10.5% at 1000 dpi), the opposite of the prediction.
161
+
162
+ ### What it points at instead
163
+
164
+ **Scan time is set by data volume at a flat ~1.1 MB/s.** 25× the bytes took 28×
165
+ the time. A mechanical limit would scale with *lines* (5×), not bytes.
166
+
167
+ `setup_s` is 59.7 ms, not the ~11 ms estimated — 20 control transfers at ~3 ms
168
+ each. Three milliseconds for a one-byte control transfer is a slow tunnel.
169
+
170
+ The `bulk_s` series at 5000 dpi is a clean period-3 cycle:
171
+
172
+ ```
173
+ 4.33 1.34 4.34 4.33 1.34 4.34 4.33 1.34 4.34 ...
174
+ ```
175
+
176
+ - slow reads: 3.66 MB / 4.33 s = **0.85 MB/s**
177
+ - fast reads: 3.66 MB / 1.34 s = **2.73 MB/s**
178
+ - cycle mean: 11 MB / 10.01 s = **1.10 MB/s**
179
+
180
+ 1000 dpi shows the same shape: 0.35 s for the first three reads (2.09 MB/s, buffer
181
+ already primed), settling to 0.64 s (1.15 MB/s).
182
+
183
+ The 2.7 MB/s peaks matter: they are **above full-speed USB's 1.5 MB/s ceiling**,
184
+ so the link is not full speed and 1.1 MB/s is not a raw link limit. The device
185
+ bursts from a primed buffer, then the read blocks waiting for production.
186
+
187
+ Working explanation: this transport is SCSI tunnelled over **IEEE-1284** over USB
188
+ (`ieee_command`, `PORT_PAR_CTRL`, `PORT_PAR_DATA`). EPP/ECP parallel throughput is
189
+ ~1–2 MB/s in practice, which is exactly what is measured, and ~3 ms per one-byte
190
+ control transfer is what a byte-at-a-time tunnel over that bridge would cost. If
191
+ that is the ceiling, **the back-up is the scanner's own flow control** — it fills
192
+ its buffer faster than the bridge drains it, and a CCD carriage cannot pause in
193
+ place, so it decelerates, reverses and re-approaches. Nothing in the host software
194
+ would change that.
195
+
196
+ ### Instrumentation bug found
197
+
198
+ `--probe-device` crashed 1 read into the 5000 dpi scan:
199
+
200
+ ```
201
+ [cmd 0xdd] status=0x02
202
+ CHECK CONDITION key=0x05 code=0x20 qualifier=0x00
203
+ ```
204
+
205
+ `0xDD` is READ STATE; key 0x05 / code 0x20 is ILLEGAL REQUEST / INVALID COMMAND
206
+ OPERATION CODE. **This device rejects READ STATE while a scan is running.** The C
207
+ only issues it at the top of `sane_start`, before START SCAN. My mistake.
208
+
209
+ `GET PARAMETERS` (0x0f) *is* accepted mid-scan — it returned 18 bytes cleanly
210
+ right before the crash — and carries the motor bit anyway, so the probe now uses
211
+ GET PARAMETERS alone. `_read_state()` is kept for idle use with the restriction
212
+ documented. Fixed, uncommitted.
213
+
214
+ Also from that log: real geometry at 5000 dpi is **7184 × 4786**, 14368 bytes/line,
215
+ 3 planes → 14358 tagged lines, and `available_lines` was 0 before START SCAN as
216
+ expected.
217
+
218
+ ### Round 2 — chunk size ruled out, and a confound in the C comparison
219
+
220
+ **`--bulk-chunk 512` changed nothing** -- identical to the 16 KiB run to three
221
+ significant figures: 1.07 MB/s aggregate, `bulk_s` p50
222
+ 4327 ms, same period-3 cycle. 229 reads of 16 KiB and 8 reads of 512 KiB perform
223
+ the same, so the data phase is not chunk-limited — `ep_in.read()` is waiting on the
224
+ device. Cross it off.
225
+
226
+ **`scanimage` at 5000 dpi reportedly never backed up.** Before concluding the C
227
+ avoids something we do not, note that **the C's default bit depth is 8**:
228
+ `bpp_list[2] = 8` (`pieusb_specific.c:460`, element 0 is the SANE list count),
229
+ assigned to `OPT_BIT_DEPTH` at `:671`. Unless `--depth 16` was passed explicitly,
230
+ that run moved **103 MB where ours moved 206 MB**, at half the bytes per line —
231
+ which is exactly the quantity that decides whether the mechanism outruns the
232
+ bridge. The comparison is not apples-to-apples until the depth is pinned.
233
+
234
+ Two further observations that argue *against* the C being cleverer:
235
+
236
+ - Its read loop is `malloc` → `sanei_pieusb_cmd_get_scanned_lines` → copy into the
237
+ buffer → `free` (`pieusb_specific.c:2232-2300`), with no sleeps, no extra
238
+ commands and no waits. Its CDB carries the line count with the byte count passed
239
+ separately (`pieusb_scancmd.c:464`) — byte-for-byte what we do. It also copies
240
+ into a *file-backed* buffer, which is more work than our numpy write.
241
+ - On the AUTO path it applies `gain-adjust` 0.8, shortening exposure to ~3706
242
+ against our 4100 — a ~10% *higher* line rate, making it more likely to outrun
243
+ the bridge, not less.
244
+
245
+ **New reading of the period-3 cycle: it may be the back-up itself.**
246
+
247
+ | | tagged lines/s | rate |
248
+ |---|---|---|
249
+ | slow reads (4.33 s) | 59 | 0.85 MB/s |
250
+ | fast reads (1.34 s) | 190 | 2.73 MB/s |
251
+
252
+ If the 4.33 s reads are slow because they contain a reversal, and 190 lines/s is
253
+ the clean rate, then a scan with no back-up would move 206 MB at 2.73 MB/s in
254
+ **~75 s instead of 193 s**. That is a falsifiable prediction: a 16-bit `scanimage`
255
+ run at 5000 dpi should take ~75-95 s if it truly streams, and ~190 s if it backs
256
+ up like ours.
257
+
258
+ ### Round 3 — CONCLUDED: the buffer never fills, it empties
259
+
260
+ 57 probed reads at 5000 dpi / 16-bit. `available_lines`:
261
+
262
+ ```
263
+ 74 31 60 114 72 30 63 114 71 28 66 112 69 27 69 111 66 24 75 107 63 21 78 106
264
+ 62 20 81 105 60 0 21 81 104 60 0 20 81 104 60 0 20 81 104 60 0 20 81 104 60 0
265
+ 20 81 104 61 0 21 2
266
+ ```
267
+
268
+ min **0**, max **114** (1.64 MB at 14370 B/line), mean 58. From read 29 it settles
269
+ into a perfectly regular period-5 cycle — `0, 20, 81, 104, 60` — **draining to
270
+ empty every fifth read**. The motor bit was never observed set.
271
+
272
+ **The buffer never overflows. It repeatedly empties.** The host outpaces the
273
+ device, so the device is the slow party and the host cannot be the cause of the
274
+ reversal. This holds regardless of whether 114 lines is the buffer's capacity or
275
+ merely as much as it happened to accumulate: at the observed ~74 tagged lines/s,
276
+ the 66 ms gap (133 ms with probes) adds **5-10 lines** of fill. Overflowing even a
277
+ 114-line buffer would need the host to stop reading for ~1.5 s.
278
+
279
+ Both host-side theories are therefore dead:
280
+
281
+ 1. Not a large-gap overflow — the gap is 1.9% of the scan.
282
+ 2. Not a small-buffer overflow — the buffer drains to zero, and the gap is worth
283
+ ≤10 lines of the ≥114 available.
284
+
285
+ The oscillation is production against 255-line gulps, and the period tracks the
286
+ timing rather than anything in the device: it was 3 without probes, 4-5 with them.
287
+
288
+ **Conclusion: this is a device characteristic and there is no host-side fix.** The
289
+ scanner reads out at ~1.1 MB/s regardless of resolution (1.20 MB/s at 1000 dpi,
290
+ 1.07 at 5000), which sets a floor of ~190 s on a 206 MB scan. The reversal is
291
+ internal — most plausibly the carriage being unable to creep smoothly at the
292
+ ~0.127 mm/s that 5000 dpi at that readout rate demands — and the dead time during
293
+ each reversal is what drains the buffer to zero and holds the average at 1.1 MB/s.
294
+
295
+ Caveat on the evidence: `available_lines` is sampled once per read, immediately
296
+ after a read drains it, so these are post-drain levels rather than peaks. Peaks are
297
+ at most ~10 lines higher (production during the gap), which does not change the
298
+ conclusion.
299
+
300
+ ### Round 3 addendum — `exp_time` appears to be inert on this device
301
+
302
+ Raising `exp_time_*` from 4100 to 8200 produced **no change in image brightness**.
303
+ That is consistent with two earlier oddities that were noted and not chased:
304
+
305
+ - Sending exposure `(0, 0, 0)` produced R max 10218, not a black frame.
306
+ - The device always reports exposure as a flat 4100 for all four channels, which
307
+ `pieusb_scancmd.h:194-199` describes as a *reset default* pattern rather than a
308
+ per-channel optimum.
309
+
310
+ This was the thread that led to §6, where every control was swept in isolation:
311
+ `exp_rel_*` turned out to be the only one that does anything, and `gain_*`,
312
+ `exp_time_*` and `light` are all inert.
313
+
314
+ ## 5. Device quirks already established
315
+
316
+ Confirmed on this unit, from `scan.log` and a SANE backend trace:
317
+
318
+ - **Saturation levels are always `(0, 0, 0)`**, cold and warm. Auto-exposure by
319
+ metering is therefore impossible — `dg = 0` → gain 0, exposure 0. This is why
320
+ the "calibration from preview" port was removed.
321
+ - **A cold scanner reports zeros for its whole calibration**: exposure
322
+ `(0, 0, 0, 4100)`, gain `(0, 0, 0, 15)`, offset all 0, light 0. It fills them in
323
+ during its **first scan**. Irrelevant now that nothing reads those values, but it
324
+ is why `auto_exp` used to need two scans.
325
+ - **Warm values** on this unit: exposure `(4100, 4100, 4100, 4100)`, gain
326
+ `(44, 28, 37, 57/58)`, offset `(12..18, 16..25, 16..26, 16..25)`, light `4`.
327
+ Exposure is a flat constant, not a per-channel optimum.
328
+ - **The light byte does nothing.** Measured directly: light 4, 5, 6 and 7 produce
329
+ the same image to within 0.1%. An earlier claim here that it was "worth roughly
330
+ the entire exposure range" came from a comparison confounded by resolution and was
331
+ wrong. Sending 0 was still incorrect — outside the documented 4..7 band and not
332
+ what the C sends — so the default of 4 stands on correctness grounds alone.
333
+ - **The C's `gain-adjust` default is `* 0.8`**, not the 1.0 its comment at
334
+ `pieusb_specific.c:743` claims (`gain_adjust_list[2]`). We send the device's
335
+ values unscaled, equivalent to `* 1.0`.
336
+
337
+ ## 6. What the exposure work settled
338
+
339
+ Chasing the stall turned up that most of the exposure controls do nothing. Measured
340
+ by sweeping each in isolation at 300 dpi / 16-bit:
341
+
342
+ | control | values tried | effect |
343
+ |---|---|---|
344
+ | `gain_*` | 10, 30, 60 | none (±0.1%) |
345
+ | `exp_time_*` | 500, 1500, 2500, 2937, 6000, 10000 | none (±0.2%) |
346
+ | `light` | 4, 5, 6, 7 | none (±0.1%) |
347
+ | `exp_rel_*` | 50 | none — clamps at 100 |
348
+ | `exp_rel_*` | 200, 250, 400, 800, 1086, 1500 | **×1.97, ×2.46, ×3.93, ×8.03, ×10.5, ×14.7** |
349
+ | `exp_rel_*` | 1598 / 1599 | ×15.98 / **×1.00** — the Timer 1 overflow step |
350
+ | `exp_rel_*` | 2200, 3300, 5000, 7500, 10000 | wrapped: ×5.8, ×1.1, ×2.0, ×10.7, ×3.9 |
351
+
352
+ `exp_rel_*` is the exposure system: linear to within 4%, per channel, upward only,
353
+ and bounded by a **16-bit Timer 1 overflow**. It scales the device's own absolute
354
+ exposure time into that register:
355
+
356
+ timer = exposure_time * exp_rel / 100 (mod 65536)
357
+
358
+ so the ceiling is `65535 * 100 // exposure_time`, which for the 4100 this unit
359
+ reports is 1598. A binary search put the step at exactly 1598/1599 for all three
360
+ channels. Above it the exposure *drops* — and lands on ×1.00 rather than near zero
361
+ because a wrapped result below 100 meets the same floor that makes `exp_rel=50`
362
+ behave as 100. The model predicts every measurement to within 4%:
363
+
364
+ exp_rel 1086 1500 | 1598 1599 | 2200 3300 5000 7500 10000
365
+ predicted x10.9 x15.0 | x15.98 x1.00 | x6.02 x1.03 x2.05 x11.1 x4.09
366
+ measured x10.5 x14.7 | clip x1.00 | x5.79 x1.06 x2.01 x10.7 x3.94
367
+
368
+ Which also explains why `exp_time_*` looked inert without being irrelevant: the
369
+ device ignores what is written to it and keeps its own value, and that value is the
370
+ multiplicand. It is the per-line integration period, so the line rate halves
371
+ as it doubles — which is why it also cures the stall.
372
+
373
+ Per-channel 247/563/1086 puts all three channels within 1% of each other at ~88% of
374
+ full scale with nothing clipped, against 4.39 : 1.93 : 1.00 with red at 34%
375
+ untouched. `auto_exp` derives exactly those numbers from a metering pass (v0.3.6).
376
+
377
+ Driving any channel to saturation produces **vertical banding**, not flat white: the
378
+ shading correction's per-column gain is >1 where the lamp falls off, so edge columns
379
+ reach the ceiling at a lower raw value than centre columns. At `exp_rel` 400 red
380
+ clipped 12.7% of samples, concentrated at the left and right edges with 55
381
+ percentage points of column-to-column swing. `apply_shading_correction` now warns
382
+ when it clamps.
383
+
384
+ ## 7. Open question, unresolved
385
+
386
+ `~/scan_tests/scanner2.c` sets `calibration = "from preview"` on both passes
387
+ (`:369`, `:386`), so its real scan should take the `SCAN_CALIBRATION_PREVIEW`
388
+ branch, compute `dg = 0` from the zero saturation levels, and send exposure 0 /
389
+ gain 0 — which produced a dark scan when this library did the same thing. It
390
+ instead produces a good scan. Either `preview_done` is false there, or the device
391
+ tolerates exposure 0 differently than observed.
392
+
393
+ Not on the critical path — `auto_exp` now adopts the device's calibration and
394
+ works — but it means the mechanism behind that C program's good output is not
395
+ fully understood. One command settles it:
396
+
397
+ ```bash
398
+ SANE_DEBUG_PIEUSB=5 ./scanner2 ... 2>&1 | grep -E "mode = |calibration data from|additional gain"
399
+ ```
400
+
401
+ - `get calibration data from scanner` → it took the AUTO path; everything is
402
+ explained by warm-vs-cold.
403
+ - `from preview` with `additional gain 0.000000` and a good scan → the device
404
+ substitutes its own values for zeros, and something else explains the dark scan
405
+ this library produced.
406
+
407
+ ## 8. Key C source references
408
+
409
+ Paths relative to `~/backends/backend/`.
410
+
411
+ | What | Where |
412
+ |---|---|
413
+ | 3 ms per-command sleep | `pieusb_usb.c:477` |
414
+ | Bulk read, 16 KiB chunks | `pieusb_usb.c:651-670` |
415
+ | 255-line read cap | `pieusb_specific.c:2248` |
416
+ | `SET GAIN OFFSET` payload layout | `pieusb_scancmd.c:1102-1116` |
417
+ | `GET GAIN OFFSET` field offsets | `pieusb_scancmd.c:1042-1051` |
418
+ | Light level semantics (band 4..7) | `pieusb_scancmd.h:208-213` |
419
+ | `available_lines`, motor field | `pieusb_scancmd.h:140-141` |
420
+ | `READ STATE` decode, scanning/motor bits | `pieusb_scancmd.c:1167-1193`, `pieusb_scancmd.h:129` |
421
+ | Calibration modes; AUTO is the default | `pieusb_specific.c:733`, `:1885-2046` |
422
+ | `updateGain2` | `pieusb_specific.c:2528` |
423
+ | `gain-adjust` option, default `* 0.8` | `pieusb_specific.c:735-743` |
424
+ | `sane_start` sequence | `pieusb.c:865-1140` |
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.4
1
+ Metadata-Version: 2.5
2
2
  Name: pieusb
3
- Version: 0.3.2
3
+ Version: 0.3.7
4
4
  Summary: Interface and acquire images from the Reflecta (a.k.a PacificImage) line of film scanners.
5
5
  Project-URL: Homepage, https://github.com/davide-b/pieusb
6
6
  Project-URL: Issues, https://github.com/davide-b/pieusb/issues
@@ -8,9 +8,9 @@ License-Expression: GPL-3.0-or-later
8
8
  License-File: LICENSE
9
9
  Classifier: Operating System :: OS Independent
10
10
  Classifier: Programming Language :: Python :: 3
11
- Requires-Python: >=3.13
11
+ Requires-Python: >=3.11
12
12
  Requires-Dist: libusb-package>=1.0.30.0
13
- Requires-Dist: numpy>=2.4.4
13
+ Requires-Dist: numpy>=1.23.2
14
14
  Requires-Dist: pyusb>=1.3.1
15
15
  Description-Content-Type: text/markdown
16
16