trace-digitiser 1.2.0__tar.gz → 2.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- trace_digitiser-2.1.0/CHANGELOG.md +170 -0
- trace_digitiser-2.1.0/MANIFEST.in +7 -0
- trace_digitiser-2.1.0/PKG-INFO +343 -0
- trace_digitiser-2.1.0/README.md +303 -0
- trace_digitiser-2.1.0/RELEASING.md +69 -0
- trace_digitiser-2.1.0/examples/README.md +37 -0
- trace_digitiser-2.1.0/examples/digitise_published_figure.py +47 -0
- trace_digitiser-2.1.0/examples/lopatina2014_fig4.jpg +0 -0
- trace_digitiser-2.1.0/examples/validate_with_known_data.py +44 -0
- trace_digitiser-2.1.0/examples/walkthrough.ipynb +669 -0
- trace_digitiser-2.1.0/pyproject.toml +91 -0
- trace_digitiser-2.1.0/src/trace_digitiser/__init__.py +361 -0
- trace_digitiser-2.1.0/src/trace_digitiser/_figure.py +480 -0
- trace_digitiser-2.1.0/src/trace_digitiser/_results.py +240 -0
- trace_digitiser-2.1.0/src/trace_digitiser/axis_labels.py +1358 -0
- trace_digitiser-2.1.0/src/trace_digitiser/bars.py +324 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/calibration.py +207 -11
- trace_digitiser-2.1.0/src/trace_digitiser/cli.py +640 -0
- trace_digitiser-2.1.0/src/trace_digitiser/diagnostics.py +428 -0
- trace_digitiser-2.1.0/src/trace_digitiser/digitise.py +112 -0
- trace_digitiser-2.1.0/src/trace_digitiser/explore.py +193 -0
- trace_digitiser-2.1.0/src/trace_digitiser/frames.py +547 -0
- trace_digitiser-2.1.0/src/trace_digitiser/heatmap.py +368 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/io.py +36 -5
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/line_detection.py +0 -1
- trace_digitiser-2.1.0/src/trace_digitiser/markers.py +399 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/models.py +88 -15
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/ocr.py +28 -9
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/panel_detection.py +271 -115
- trace_digitiser-2.1.0/src/trace_digitiser/points.py +194 -0
- trace_digitiser-2.1.0/src/trace_digitiser/polar.py +320 -0
- trace_digitiser-2.1.0/src/trace_digitiser/provenance.py +162 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/resampling.py +0 -1
- trace_digitiser-2.1.0/src/trace_digitiser/rows.py +313 -0
- trace_digitiser-2.1.0/src/trace_digitiser/straighten.py +95 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/synthetic.py +0 -1
- trace_digitiser-2.1.0/src/trace_digitiser/trace_detection.py +653 -0
- trace_digitiser-2.1.0/src/trace_digitiser/validation.py +1244 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/x_calibration.py +100 -11
- trace_digitiser-2.1.0/src/trace_digitiser.egg-info/PKG-INFO +343 -0
- trace_digitiser-2.1.0/src/trace_digitiser.egg-info/SOURCES.txt +64 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser.egg-info/requires.txt +3 -0
- trace_digitiser-2.1.0/tests/__init__.py +0 -0
- trace_digitiser-2.1.0/tests/conftest.py +125 -0
- trace_digitiser-2.1.0/tests/data/lopatina2014_fig4_reference.csv +697 -0
- trace_digitiser-2.1.0/tests/test_axis_labels.py +440 -0
- trace_digitiser-2.1.0/tests/test_calibration.py +116 -0
- trace_digitiser-2.1.0/tests/test_cli.py +132 -0
- trace_digitiser-2.1.0/tests/test_digitise.py +304 -0
- trace_digitiser-2.1.0/tests/test_frames.py +208 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/tests/test_integration.py +49 -1
- trace_digitiser-2.1.0/tests/test_io.py +61 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/tests/test_panel_detection.py +0 -2
- trace_digitiser-2.1.0/tests/test_published_figure.py +101 -0
- trace_digitiser-2.1.0/tests/test_readers.py +417 -0
- trace_digitiser-2.1.0/tests/test_straighten.py +149 -0
- trace_digitiser-2.1.0/tests/test_trace_detection.py +172 -0
- trace_digitiser-2.1.0/tests/test_validation.py +253 -0
- trace_digitiser-1.2.0/PKG-INFO +0 -178
- trace_digitiser-1.2.0/README.md +0 -157
- trace_digitiser-1.2.0/pyproject.toml +0 -43
- trace_digitiser-1.2.0/src/trace_digitiser/__init__.py +0 -276
- trace_digitiser-1.2.0/src/trace_digitiser/cli.py +0 -179
- trace_digitiser-1.2.0/src/trace_digitiser/diagnostics.py +0 -283
- trace_digitiser-1.2.0/src/trace_digitiser/digitise.py +0 -85
- trace_digitiser-1.2.0/src/trace_digitiser/trace_detection.py +0 -431
- trace_digitiser-1.2.0/src/trace_digitiser.egg-info/PKG-INFO +0 -178
- trace_digitiser-1.2.0/src/trace_digitiser.egg-info/SOURCES.txt +0 -29
- trace_digitiser-1.2.0/tests/test_trace_detection.py +0 -35
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/LICENSE +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/setup.cfg +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/geometry.py +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser/summarise.py +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser.egg-info/dependency_links.txt +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser.egg-info/entry_points.txt +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/src/trace_digitiser.egg-info/top_level.txt +0 -0
- {trace_digitiser-1.2.0 → trace_digitiser-2.1.0}/tests/test_geometry.py +0 -0
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to trace-digitiser are recorded here.
|
|
4
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
|
|
5
|
+
and the project uses [semantic versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [2.1.0] - 2026-11-14
|
|
8
|
+
|
|
9
|
+
Everything in 2.0.0 works as before. What is new is readers for other kinds of
|
|
10
|
+
figure, and helpers for graphical tools such as the
|
|
11
|
+
[digitiser workbench](https://github.com/comparativechrono/digitiser-workbench).
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Points**: `digitise_points` (`trace-digitiser points`) reads the markers of
|
|
16
|
+
a scatter or dot plot, or of a line drawn with markers, whatever their
|
|
17
|
+
shape, separating markers that overlap; and the error bars on them,
|
|
18
|
+
vertical or horizontal, to the middle of their caps.
|
|
19
|
+
- **Bars**: `digitise_bars` (`trace-digitiser bars`) reads where each bar
|
|
20
|
+
ends, upright or lying down, rising or hanging from the baseline, and its
|
|
21
|
+
error bar; a legend's patch is left out.
|
|
22
|
+
- **Rows**: `digitise_rows` (`trace-digitiser rows`) reads figures drawn as a
|
|
23
|
+
stack of rows - actograms, double-plotted records, birthograms - bin by bin,
|
|
24
|
+
and the dots that mark a peak in them.
|
|
25
|
+
- **Heat maps**: `digitise_heatmap` (`trace-digitiser heatmap`) reads each
|
|
26
|
+
cell's value from its colour on the colour bar, whose labels are read like
|
|
27
|
+
an axis's (or from a matplotlib colour map).
|
|
28
|
+
- **Polar plots**: `digitise_polar` (`trace-digitiser polar`) finds the polar
|
|
29
|
+
axes and their rings, and reads each point's angle and distance.
|
|
30
|
+
- All of them return a `FigureResult` and write a CSV file, a provenance file
|
|
31
|
+
and, if asked, an overlay, as `digitise` does.
|
|
32
|
+
- `describe_figure` (`trace-digitiser describe`): what the automatic steps
|
|
33
|
+
find in a figure - plot areas, each axis's tick labels and readings, the
|
|
34
|
+
colours drawn, a colour bar, polar axes - as plain values.
|
|
35
|
+
- `pixels_to_values`: points picked by hand, in the figure's units.
|
|
36
|
+
- Tick labels are read on the right-hand side of a plot too (a colour bar, a
|
|
37
|
+
second y axis): `calibrate_axis(..., "y2")`.
|
|
38
|
+
- `y_axis="right"` (`--y-axis right`) reads a line's values against a second
|
|
39
|
+
y axis on the right of the plot (a twin axis); with `trace_colours`, the
|
|
40
|
+
lines drawn against each axis are digitised in a run of their own.
|
|
41
|
+
- `describe_figure` gives the tick marks found on each plot's axes, for a
|
|
42
|
+
program that lets you click on a tick.
|
|
43
|
+
- Colours can be given by name (`"black"`, `"steelblue"`, `"tab:blue"`).
|
|
44
|
+
- Plots stacked on a shared x axis, of which only the lowest has labels, take
|
|
45
|
+
its x scale (with a message saying so).
|
|
46
|
+
- A plot area hidden among others in a grid (a heat map whose dark colours
|
|
47
|
+
hide its axis lines) is found by its frame.
|
|
48
|
+
|
|
49
|
+
### Changed
|
|
50
|
+
|
|
51
|
+
- An axis scale is held to medium confidence when the tick labels support a
|
|
52
|
+
different scale just as well. Tesseract compiled to WebAssembly (as it runs
|
|
53
|
+
in a browser) read the superscript minus signs of a log axis's exponents
|
|
54
|
+
differently from the Tesseract program, and two scales then fitted equally;
|
|
55
|
+
between log scales that tie, the one that steps by a decade is chosen.
|
|
56
|
+
- The sides of filled areas (bars) no longer make a plot area of their own
|
|
57
|
+
inside the real one, which bar charts with a legend could cause.
|
|
58
|
+
- A long axis (ten or more labels) whose labels nearly all agree is not read
|
|
59
|
+
again in the slower ways; fitting the labels of a long axis (the 59 years of
|
|
60
|
+
a birthogram) takes a second or two rather than half a minute, with the same
|
|
61
|
+
result.
|
|
62
|
+
- A heat map's colours are looked up on the colour bar once each, rather than
|
|
63
|
+
once for each pixel: Figure 4 of Hearn and Whitmore (2026) takes 4 seconds
|
|
64
|
+
rather than 45, with the same values.
|
|
65
|
+
- pytesseract is imported only when it is first used.
|
|
66
|
+
|
|
67
|
+
## [2.0.0] - 2026-11-14
|
|
68
|
+
|
|
69
|
+
Every call and command-line option of 1.2.0 still works, with the same
|
|
70
|
+
meaning. The version number changes because the **results** change: figures
|
|
71
|
+
are analysed in a new way, and the values that come out differ from those of
|
|
72
|
+
1.2.0. By the package's own tests against known data they are much closer to
|
|
73
|
+
what was drawn (55 of 57 test traces recovered to within 2 % with nothing
|
|
74
|
+
supplied, against 20; see the README). Anything digitised with 1.x that
|
|
75
|
+
matters should be digitised again.
|
|
76
|
+
|
|
77
|
+
### Added
|
|
78
|
+
|
|
79
|
+
- **Plot areas are found without being told the layout.** Axis lines are
|
|
80
|
+
located to a fraction of a pixel and assembled into plot areas: any number
|
|
81
|
+
of panels, boxed or L-shaped axes, with or without gridlines and tick
|
|
82
|
+
marks. `layout_mode`, `expected_rows` and the rest remain as hints and
|
|
83
|
+
checks.
|
|
84
|
+
- **Tilted scans are straightened.** The tilt of the figure is measured and
|
|
85
|
+
undone first (`straighten="auto"`, `--straighten`), so a page that lay
|
|
86
|
+
askew on the scanner is digitised as accurately as one that did not.
|
|
87
|
+
- **Each automatic step can be replaced by the answer.** `plot_area` for the
|
|
88
|
+
position of the axes, `y_ref` / `x_ref` for reference points on an axis,
|
|
89
|
+
`y_scale` / `x_scale` for logarithmic axes, `trace_colours` for the
|
|
90
|
+
colours to follow, `exclude` for legends and annotations, `achromatic` for
|
|
91
|
+
black and grey traces, `ocr=False` to do without Tesseract.
|
|
92
|
+
- **The x axis can be calibrated** from `x_min` / `x_max`, from reference
|
|
93
|
+
points, or from numeric tick labels (`x_value` in the output).
|
|
94
|
+
- **Axis scales read from tick labels come with a confidence rating**, and a
|
|
95
|
+
scale that too few labels agree on is declined rather than guessed. The
|
|
96
|
+
rating, and the labels the scale rests on, are in the panel metadata.
|
|
97
|
+
- **Messages about anything that needs checking**: `result.warnings`,
|
|
98
|
+
`DigitiserWarning`, and `CHECK:` lines on the command line.
|
|
99
|
+
- **An overlay image** of the result drawn over the figure (`save_overlay`,
|
|
100
|
+
`--overlay`, `result.overlay()`): the quickest way to see whether a
|
|
101
|
+
digitisation is right.
|
|
102
|
+
- **A provenance file** for every run: version, SHA-256 of the image, all
|
|
103
|
+
settings, what was detected, and all messages.
|
|
104
|
+
- **Validation tools** (`trace_digitiser.validation`): `round_trip` tests the
|
|
105
|
+
digitiser on your own data drawn at the size of your figure;
|
|
106
|
+
`run_validation` (`--validate`) and `run_axis_validation` test it on 44
|
|
107
|
+
figures and 750 axes drawn from known data.
|
|
108
|
+
- New columns in the trace file: `n_pixels`, `interpolated`, `x_value`. New
|
|
109
|
+
columns at the end of the panel metadata: the sub-pixel plot area and how
|
|
110
|
+
each axis was calibrated. The 1.2.0 columns keep their names and order.
|
|
111
|
+
- Image arrays are accepted as well as file names; transparent images are
|
|
112
|
+
placed on white, and photographs are turned according to their EXIF data.
|
|
113
|
+
- `--version`; exit status 1 if any image could not be processed.
|
|
114
|
+
- Examples: two scripts, a notebook, and a published figure (CC BY) that is
|
|
115
|
+
also used in the tests.
|
|
116
|
+
- The repository itself: continuous testing on Python 3.10 to 3.14, and a
|
|
117
|
+
release workflow that publishes to PyPI through trusted publishing (see
|
|
118
|
+
`RELEASING.md`).
|
|
119
|
+
|
|
120
|
+
### Changed
|
|
121
|
+
|
|
122
|
+
- **Values differ from 1.2.0**, chiefly because:
|
|
123
|
+
- the plot area is now bounded by the centres of the axis lines; 1.2.0 put
|
|
124
|
+
its edges up to 13 pixels outside them when tick marks pointed outwards,
|
|
125
|
+
which shifted and rescaled every value read with `y_min` / `y_max`;
|
|
126
|
+
- traces are followed down onto the axis lines when their colour allows,
|
|
127
|
+
and thin, pale, grey and black traces are no longer missed;
|
|
128
|
+
- gridlines, text and compression fringes are no longer reported as
|
|
129
|
+
extra traces (1.2.0 did so in about half of single-trace test figures).
|
|
130
|
+
- Axis labels are read in a new way (located geometrically, read by
|
|
131
|
+
Tesseract in several ways, and fitted on the principle that tick values
|
|
132
|
+
step by exactly the same amount). On the test axes it never gave a wrong
|
|
133
|
+
scale with high confidence; 1.2.0 misread decimal and logarithmic axes by
|
|
134
|
+
large factors without warning. It takes about half a second a figure
|
|
135
|
+
rather than twenty.
|
|
136
|
+
- Tesseract is now optional: without it, give the axis limits.
|
|
137
|
+
- Labels under 8 pixels high are read but never trusted outright; the
|
|
138
|
+
result carries a message asking for the values to be checked.
|
|
139
|
+
- `mean_estimate` and `center_estimate` are documented for what they are
|
|
140
|
+
(statistics of the trace's pixels in a column), with the consequence for
|
|
141
|
+
dense records spelt out in the README.
|
|
142
|
+
|
|
143
|
+
### Fixed
|
|
144
|
+
|
|
145
|
+
- A one-pixel shift in the trace mask (from a 2×2 morphological opening)
|
|
146
|
+
that erased one-pixel lines and produced all-white "traces".
|
|
147
|
+
- `tests/conftest.py` was missing from the 1.2.0 source distribution, so its
|
|
148
|
+
tests could not be run from it.
|
|
149
|
+
- `use_lab=True` needed scikit-learn, and one helper needed SciPy, without
|
|
150
|
+
either being declared. scikit-learn is now the optional extra
|
|
151
|
+
`trace-digitiser[lab]`; SciPy is no longer used.
|
|
152
|
+
- `eval` of text read back from a data frame replaced by `ast.literal_eval`.
|
|
153
|
+
- The trace-accuracy test that failed in 1.2.0 (RMSE 102 against a limit of
|
|
154
|
+
50) passes, and a new one holds all eight example figures to 7.5.
|
|
155
|
+
|
|
156
|
+
### Removed
|
|
157
|
+
|
|
158
|
+
- The slow "exhaustive" reading of y labels is no longer tried by default
|
|
159
|
+
(`calibrate_y(..., exhaustive_fallback=True)` still does it). On axes that
|
|
160
|
+
the new method declined, it was right in none of twenty trials.
|
|
161
|
+
|
|
162
|
+
## [1.2.0] - 2026-06-07
|
|
163
|
+
|
|
164
|
+
First version tracked in this repository; identical to the 1.2.0 files on PyPI.
|
|
165
|
+
Releases 0.1.0 to 1.0.0 (6-7 June 2026) were uploaded to PyPI directly and are
|
|
166
|
+
not recorded here.
|
|
167
|
+
|
|
168
|
+
[2.1.0]: https://github.com/comparativechrono/trace-digitiser/compare/v2.0.0...v2.1.0
|
|
169
|
+
[2.0.0]: https://github.com/comparativechrono/trace-digitiser/compare/1d34f75...v2.0.0
|
|
170
|
+
[1.2.0]: https://pypi.org/project/trace-digitiser/1.2.0/
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Ship the complete test suite and the examples in the source distribution.
|
|
2
|
+
# (Without this, setuptools includes tests/test_*.py but not conftest.py or
|
|
3
|
+
# the reference data, so the tests in the sdist cannot run.)
|
|
4
|
+
recursive-include tests *.py *.csv
|
|
5
|
+
recursive-include examples *.py *.md *.jpg *.ipynb
|
|
6
|
+
prune examples/output
|
|
7
|
+
include CHANGELOG.md RELEASING.md
|
|
@@ -0,0 +1,343 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: trace-digitiser
|
|
3
|
+
Version: 2.1.0
|
|
4
|
+
Summary: Recover the data from figure images - lines, points, bars, actograms, heat maps - on your own computer: no uploads, no AI model.
|
|
5
|
+
Author: Trace Digitiser Contributors
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/comparativechrono/trace-digitiser
|
|
8
|
+
Project-URL: Repository, https://github.com/comparativechrono/trace-digitiser
|
|
9
|
+
Project-URL: Issues, https://github.com/comparativechrono/trace-digitiser/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/comparativechrono/trace-digitiser/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: plot digitizer,graph digitiser,data extraction,data rescue,computer vision,time series,circadian,actigraphy,actogram,heat map,scatter plot,bar chart
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
22
|
+
Classifier: Topic :: Scientific/Engineering
|
|
23
|
+
Classifier: Topic :: Scientific/Engineering :: Image Processing
|
|
24
|
+
Requires-Python: >=3.10
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
License-File: LICENSE
|
|
27
|
+
Requires-Dist: opencv-python-headless>=4.8
|
|
28
|
+
Requires-Dist: pytesseract>=0.3.10
|
|
29
|
+
Requires-Dist: Pillow>=10.0
|
|
30
|
+
Requires-Dist: pandas>=2.0
|
|
31
|
+
Requires-Dist: numpy>=1.24
|
|
32
|
+
Requires-Dist: matplotlib>=3.7
|
|
33
|
+
Provides-Extra: lab
|
|
34
|
+
Requires-Dist: scikit-learn>=1.2; extra == "lab"
|
|
35
|
+
Provides-Extra: dev
|
|
36
|
+
Requires-Dist: pytest>=7.4; extra == "dev"
|
|
37
|
+
Requires-Dist: pytest-cov>=4.1; extra == "dev"
|
|
38
|
+
Requires-Dist: ruff>=0.4; extra == "dev"
|
|
39
|
+
Dynamic: license-file
|
|
40
|
+
|
|
41
|
+
# trace-digitiser
|
|
42
|
+
|
|
43
|
+
Get the numbers back out of a figure when the figure is all that survives — on your own computer.
|
|
44
|
+
|
|
45
|
+
`trace-digitiser` takes an image of a figure (a scan of an old paper, a figure saved from a PDF),
|
|
46
|
+
finds the plot area, follows each trace, and writes the values to a CSV file. It reads line plots,
|
|
47
|
+
and also [points, bars, actograms, heat maps and polar plots](#other-kinds-of-figure). It is
|
|
48
|
+
ordinary image processing, written to be read and checked.
|
|
49
|
+
|
|
50
|
+
**Nothing is uploaded and no AI model is involved.** The only other program it runs is
|
|
51
|
+
[Tesseract](https://github.com/tesseract-ocr/tesseract), also on your machine and optional, to read
|
|
52
|
+
the numbers printed beside the axes. A figure that you are not free to pass to an online service
|
|
53
|
+
can therefore still be digitised. (Whether the values in a published figure may be *reused* is a
|
|
54
|
+
matter for the licence of the source and the law where you work; no tool can settle that for you.)
|
|
55
|
+
|
|
56
|
+
A worked example with pictures is in
|
|
57
|
+
[`examples/walkthrough.ipynb`](https://github.com/comparativechrono/trace-digitiser/blob/main/examples/walkthrough.ipynb).
|
|
58
|
+
|
|
59
|
+
## Install
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
pip install trace-digitiser
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Python 3.10 or later. To have axis values read from the figure, also install Tesseract
|
|
66
|
+
(`sudo apt-get install tesseract-ocr`, `brew install tesseract`, or the
|
|
67
|
+
[Windows installer](https://github.com/UB-Mannheim/tesseract/wiki)). Without it everything else
|
|
68
|
+
works, and you type in the axis limits yourself — which is the better practice anyway.
|
|
69
|
+
|
|
70
|
+
## Use
|
|
71
|
+
|
|
72
|
+
From the command line:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
trace-digitiser figure.png --y-min 0 --y-max 300 --x-min 0 --x-max 7 --overlay -o results/
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
or from Python:
|
|
79
|
+
|
|
80
|
+
```python
|
|
81
|
+
from trace_digitiser import digitise
|
|
82
|
+
|
|
83
|
+
result = digitise("figure.png", y_min=0, y_max=300, x_min=0, x_max=7,
|
|
84
|
+
output_dir="results", save_overlay=True)
|
|
85
|
+
|
|
86
|
+
result.trace_data # one row per pixel column of each trace
|
|
87
|
+
result.warnings # anything you should check
|
|
88
|
+
result.overlay_path # the digitised traces drawn over the figure
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
`y_min` and `y_max` are the values at the bottom and top axis lines, `x_min` and `x_max` those at
|
|
92
|
+
the left and right. Nothing else is needed for most figures: the plot areas (any number of panels)
|
|
93
|
+
and the traces (coloured, black or grey) are found automatically, and a scan that lies a few
|
|
94
|
+
degrees askew is straightened first.
|
|
95
|
+
|
|
96
|
+
Leave the limits out and the scale is read from the tick labels instead:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
trace-digitiser figure.png --overlay -o results/
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
Panels: 2, Traces: 2
|
|
104
|
+
panel 1: y 0.31 to 299, read from tick labels (medium confidence); x not calibrated
|
|
105
|
+
CHECK: Panel 1: the y-axis scale was read from 7 of 7 tick labels (300, 250, 200, 150, 100, 50, 0);
|
|
106
|
+
the print is only 6 pixels high. Confidence in it is medium: check those values against
|
|
107
|
+
the figure, or give y_min and y_max.
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
A scale read from labels always comes with a statement of how far to trust it and, unless that is
|
|
111
|
+
"high", of the labels it rests on. If too few labels can be read to agree on a scale, the axis is
|
|
112
|
+
left uncalibrated (values from 0 to 1) and you are told so: the aim is never to pass off a wrong
|
|
113
|
+
scale as a good one ([how well that is met](#how-accurate-is-it)).
|
|
114
|
+
|
|
115
|
+
## Other kinds of figure
|
|
116
|
+
|
|
117
|
+
Points, bars, stacked rows, heat maps and polar plots each have a reader of their own. They find
|
|
118
|
+
the plot areas and read the axes exactly as `digitise` does, take the same settings for them
|
|
119
|
+
(`y_min`, `y_ref`, `plot_area`, `exclude` …), and give a table with a row for each thing found:
|
|
120
|
+
|
|
121
|
+
| Figure | Python | Command line | One row for each |
|
|
122
|
+
|---|---|---|---|
|
|
123
|
+
| Scatter or dot plot, line with markers | `digitise_points` | `trace-digitiser points` | point, with its error bars |
|
|
124
|
+
| Bar chart | `digitise_bars` | `trace-digitiser bars` | bar: where it ends, and its error bar |
|
|
125
|
+
| Actogram, double-plotted record, any stack of rows | `digitise_rows` | `trace-digitiser rows` | bin (or column) of each row |
|
|
126
|
+
| Heat map with a colour bar | `digitise_heatmap` | `trace-digitiser heatmap` | cell, from its colour on the bar |
|
|
127
|
+
| Polar plot | `digitise_polar` | `trace-digitiser polar` | point: its angle and distance |
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
from trace_digitiser import digitise_points, digitise_bars, digitise_rows, digitise_heatmap
|
|
131
|
+
|
|
132
|
+
digitise_points("forest.png", save_overlay=True) # markers, joined by a line or not; error bars too
|
|
133
|
+
digitise_bars("bars.png", y_min=-25, y_max=0) # the legend's patch is left out
|
|
134
|
+
digitise_rows("birthogram.png", row_values=range(1955, 2014), row_spacing_value=1.1,
|
|
135
|
+
bins=24, dots="black") # each row's fill, month by month, and its dot
|
|
136
|
+
digitise_heatmap("ccf.png", rows=21) # the colour bar's labels are read too
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Each returns a `FigureResult`: `result.data` (the table), `result.warnings`, `result.overlay()`
|
|
140
|
+
and the files written (`<image>_points.csv`, `<image>_points_provenance.json` …).
|
|
141
|
+
|
|
142
|
+
For a program with a graphical interface — the
|
|
143
|
+
[digitiser workbench](https://github.com/comparativechrono/digitiser-workbench), which runs all of
|
|
144
|
+
this in a web browser, where you can mark plot areas, ticks and colours with the mouse —
|
|
145
|
+
`describe_figure` reports what the automatic steps find (plot areas, the tick marks on each axis,
|
|
146
|
+
each axis's tick labels and what was read from them, the colours drawn, a colour bar, polar axes)
|
|
147
|
+
so that they can be shown and corrected, and `pixels_to_values` turns points picked by hand into
|
|
148
|
+
the figure's units. `trace-digitiser describe figure.png` prints the same as JSON.
|
|
149
|
+
|
|
150
|
+
## Check the result
|
|
151
|
+
|
|
152
|
+
1. **Look at the overlay** (`--overlay`). It shows the plot area that was found, the centre and
|
|
153
|
+
envelope of each digitised trace drawn over the original, and the positions of the axis values.
|
|
154
|
+
If the lines sit on the traces, the digitisation is right; if they do not, you can see where.
|
|
155
|
+
2. **Read the messages.** Everything the program is unsure of is printed as a `CHECK:` line
|
|
156
|
+
(in Python: `result.warnings`, also raised as `DigitiserWarning`). A clean run prints none.
|
|
157
|
+
3. **Keep the provenance file.** `<name>_automated_provenance.json` records the version, the
|
|
158
|
+
SHA-256 of the image, every setting, what was detected and all messages, so that a digitisation
|
|
159
|
+
can be reported and repeated exactly.
|
|
160
|
+
|
|
161
|
+
## Output
|
|
162
|
+
|
|
163
|
+
| File | Contents |
|
|
164
|
+
|------|----------|
|
|
165
|
+
| `<name>_automated_digitised_trace_by_column.csv` | One row per pixel column per trace (below). |
|
|
166
|
+
| `<name>_automated_panel_metadata.csv` | Where each plot area is, its axis calibrations and how they were obtained. |
|
|
167
|
+
| `<name>_automated_digitised_summary_by_label.csv` | Means per labelled interval, when the x axis carries interval names (`1N`, `1D`, …). |
|
|
168
|
+
| `<name>_automated_provenance.json` | Everything needed to report and repeat the run. |
|
|
169
|
+
| `<name>_automated_overlay.png` | With `--overlay`: the result drawn over the figure. |
|
|
170
|
+
| `<name>_automated_straightened.png` | Only if the figure was tilted: the image turned upright, which is what pixel positions in the other files refer to. |
|
|
171
|
+
|
|
172
|
+
Columns of the trace file:
|
|
173
|
+
|
|
174
|
+
| Column | Meaning |
|
|
175
|
+
|--------|---------|
|
|
176
|
+
| `panel_id`, `trace_id` | Panels are numbered row by row from the top left; traces within a panel from 1. |
|
|
177
|
+
| `x_pixel`, `x_norm_0_to_1` | The pixel column, and its position between the left (0) and right (1) axis lines. |
|
|
178
|
+
| `x_value` | The x value, if the x axis is calibrated. |
|
|
179
|
+
| `center_estimate` | Middle of the trace's ink in this column (median of its pixels). |
|
|
180
|
+
| `mean_estimate` | Mean of the trace's pixels in this column. |
|
|
181
|
+
| `upper_envelope`, `lower_envelope` | Top and bottom of the trace's ink in this column. |
|
|
182
|
+
| `n_pixels`, `interpolated` | How many pixels the trace has in this column; `True` if it has none and the values were interpolated from its neighbours. |
|
|
183
|
+
| `scale_type` | Where the y values come from: `user_supplied`, `ocr_value`, `normalised_0_to_1`, … |
|
|
184
|
+
| `trace_median_rgb`, `trace_pixel_count` | The trace's colour and size, for identifying it. |
|
|
185
|
+
|
|
186
|
+
`result.resample(...)` and `--sampling-interval` put the trace on an even grid in x.
|
|
187
|
+
|
|
188
|
+
## What the values are — and are not
|
|
189
|
+
|
|
190
|
+
A digitiser recovers what was **drawn**, not what was measured.
|
|
191
|
+
|
|
192
|
+
* For an ordinary curve — a thin line with at most a few samples per pixel column —
|
|
193
|
+
`center_estimate` is the plotted value, to within a fraction of a pixel.
|
|
194
|
+
* For a **dense record**, where many samples share each pixel column and the trace is drawn as a
|
|
195
|
+
solid band (an activity record of several days in a small figure, for instance), the drawing
|
|
196
|
+
shows only the *range* of the samples in each column. `upper_envelope` and `lower_envelope`
|
|
197
|
+
recover that range; `center_estimate` is its middle. **None of them is the mean of the samples**,
|
|
198
|
+
and for spiky data the middle of the range can be far from the mean. The timing of the rhythm
|
|
199
|
+
survives; the level does not.
|
|
200
|
+
|
|
201
|
+
How much this matters for a particular kind of data and size of figure can be measured rather than
|
|
202
|
+
guessed: see [Test it on data you know](#test-it-on-data-you-know).
|
|
203
|
+
|
|
204
|
+
Positions cannot be better than the image. In a plot area 350 pixels wide showing seven days, one
|
|
205
|
+
pixel is half an hour.
|
|
206
|
+
|
|
207
|
+
## When something is not found
|
|
208
|
+
|
|
209
|
+
Each automatic step can be replaced by telling the program the answer. Pixel positions count from
|
|
210
|
+
the top-left corner of the image, starting at 0; any image viewer shows them.
|
|
211
|
+
|
|
212
|
+
| Problem | Option | Python |
|
|
213
|
+
|---------|--------|--------|
|
|
214
|
+
| The figure is tilted and was not straightened, or was turned when it should not have been | `--straighten DEGREES` (its anticlockwise tilt), `--straighten never` | `straighten=1.3`, `straighten=False` |
|
|
215
|
+
| The plot area is wrong or not found | `--plot-area LEFT,TOP,RIGHT,BOTTOM` (repeat per panel) | `plot_area=(l, t, r, b)` |
|
|
216
|
+
| The wrong number of panels | `--layout`, `--rows`, `--cols`, `--panels` | `layout_mode`, `expected_rows`, … |
|
|
217
|
+
| The axis continues beyond its last tick, or has no line at the top or right | `--y-ref ROW:VALUE` twice or more (also `--x-ref`) | `y_ref=[(row, value), …]` |
|
|
218
|
+
| A logarithmic axis, with limits given | `--y-scale log` | `y_scale="log"` |
|
|
219
|
+
| A line drawn against a second y axis, on the right | `--y-axis right` (with `--trace-colour`) | `y_axis="right"` |
|
|
220
|
+
| A trace is missed, or two are merged | `--trace-colour RRGGBB` (repeat per trace) | `trace_colours=["#578dd2", …]` |
|
|
221
|
+
| A legend or annotation is taken for a trace | `--exclude LEFT,TOP,RIGHT,BOTTOM` | `exclude=(l, t, r, b)` |
|
|
222
|
+
| A black trace is invented or ignored | `--achromatic never` / `always` | `achromatic="never"` |
|
|
223
|
+
| No Tesseract, or it should not be used | `--no-ocr` | `ocr=False` |
|
|
224
|
+
|
|
225
|
+
## How accurate is it?
|
|
226
|
+
|
|
227
|
+
The package carries its own tests against known answers, and you can rerun them.
|
|
228
|
+
|
|
229
|
+
**Traces.** `trace-digitiser --validate results/` draws 44 figures (57 traces) from known data —
|
|
230
|
+
different axis styles, twelve colours including black and grey, 60 to 300 dpi, JPEG compression,
|
|
231
|
+
noise and blur, line widths, multi-panel layouts, several traces per panel, a logarithmic axis,
|
|
232
|
+
jagged and dense records, tilted scans — digitises them, and compares. Errors are in percent of the
|
|
233
|
+
height of the plot. A trace counts as *recovered* if its panel and curve were found with no
|
|
234
|
+
spurious extra trace, x positions are within 1 % and the RMS error in y is within 2 %.
|
|
235
|
+
|
|
236
|
+
| | found | recovered | median error | time per figure¹ |
|
|
237
|
+
|---|---|---|---|---|
|
|
238
|
+
| **2.0, nothing supplied** | 57 of 57 | 55 | 0.13 % | 0.5 s |
|
|
239
|
+
| **2.0, axis limits supplied** | 57 of 57 | 55 | 0.09 % | 0.3 s |
|
|
240
|
+
| 1.2.0, panel layout supplied | 48 of 57 | 20 | 0.49 % | 20 s |
|
|
241
|
+
| 1.2.0, panel layout and axis limits supplied | 48 of 57 | 13 | 1.39 % | 0.3 s |
|
|
242
|
+
|
|
243
|
+
¹ On a two-core machine.
|
|
244
|
+
|
|
245
|
+
The two traces that 2.0 does not recover are dense records in low-resolution figures (errors of
|
|
246
|
+
2.3 % and 4.3 %), for the reason given [above](#what-the-values-are--and-are-not). Of the traces
|
|
247
|
+
that 1.2.0 found but did not recover, most came with a spurious second trace.
|
|
248
|
+
|
|
249
|
+
**Axis scales.** Reading a scale from tick labels is the one step that depends on OCR, and OCR of
|
|
250
|
+
small print is unreliable, so it is tested separately:
|
|
251
|
+
`python -m trace_digitiser.validation results/ --axes` reads 750 axes (whole numbers, decimals,
|
|
252
|
+
negative values, thousands, clock and angle steps, logarithmic; print from 5 to 24 pixels high;
|
|
253
|
+
three typefaces; JPEG compression and blur).
|
|
254
|
+
|
|
255
|
+
| Height of the labels | axes | read correctly | declined | read wrongly, with a warning | read wrongly, no warning |
|
|
256
|
+
|---|---|---|---|---|---|
|
|
257
|
+
| 8 pixels or more | 477 | 464 | 13 | 0 | 0 |
|
|
258
|
+
| under 8 pixels | 273 | 257 (all with a warning to check them) | 15 | 1 | 0 |
|
|
259
|
+
|
|
260
|
+
*Declined* means that no scale was offered. All of those at 8 pixels or more are logarithmic axes
|
|
261
|
+
whose raised exponents are too small to read, though the rest of the label is not. The outcome for small
|
|
262
|
+
print depends on how the text was rasterised: with the test figures drawn by matplotlib 3.7, which
|
|
263
|
+
renders small print more softly than current versions, 236 of 292 axes under 8 pixels were read
|
|
264
|
+
correctly, 46 declined and 10 read wrongly with a warning — and still none without.
|
|
265
|
+
|
|
266
|
+
Labels under 8 pixels high are never given high confidence, however well they agree, because print
|
|
267
|
+
that small is misread in consistent ways — the decimal point dropped from every label, say. If
|
|
268
|
+
your figure has such labels, enlarging the image does not help; typing in two numbers does.
|
|
269
|
+
|
|
270
|
+
### Test it on data you know
|
|
271
|
+
|
|
272
|
+
The best test of a digitisation is one on data like yours. `round_trip` draws a known series as a
|
|
273
|
+
figure of whatever size and style you choose, digitises the drawing, and reports the errors:
|
|
274
|
+
|
|
275
|
+
```python
|
|
276
|
+
from trace_digitiser.validation import round_trip
|
|
277
|
+
|
|
278
|
+
check = round_trip(t, y, "check/", xlim=(0, 7), ylim=(0, 300),
|
|
279
|
+
panel_px=(347, 137), dpi=72, linewidth=0.8, fmt="jpg")
|
|
280
|
+
check["metrics"] # errors of centre and envelopes, correlation with the mean per column, …
|
|
281
|
+
check["overlay"] # the picture to look at
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
[`examples/validate_with_known_data.py`](https://github.com/comparativechrono/trace-digitiser/blob/main/examples/validate_with_known_data.py)
|
|
285
|
+
does this for a week of minute-by-minute activity drawn at the size of a small journal figure.
|
|
286
|
+
|
|
287
|
+
## Limits
|
|
288
|
+
|
|
289
|
+
* The plot must have at least a left and a bottom axis line, drawn in black or grey. A figure
|
|
290
|
+
that is askew is straightened automatically if the tilt is under five degrees; beyond that,
|
|
291
|
+
or for a photograph taken at an angle (where the axes are no longer parallel), correct the
|
|
292
|
+
image first.
|
|
293
|
+
* One trace per colour per panel. Two traces of the same colour and dashed or dotted lines are not
|
|
294
|
+
separated or joined up. Markers are read by `digitise_points`.
|
|
295
|
+
* Where traces cross or overlap, the one drawn underneath is hidden and its values there are
|
|
296
|
+
interpolated (`interpolated` is `True`). Markers that overlap heavily (the inner turns of a
|
|
297
|
+
spiral on a polar plot, say) are separated by their size, and some may be missed or doubled:
|
|
298
|
+
check the overlay.
|
|
299
|
+
* Axis scales are read only from numeric labels on linear or logarithmic axes (on the left, below,
|
|
300
|
+
or for a colour bar or a second y axis, on the right); dates, month names and broken axes are
|
|
301
|
+
not understood: give `x_ref` / `y_ref` for those. Limits typed in are assumed to be at the axis
|
|
302
|
+
lines unless `y_ref` / `x_ref` say otherwise.
|
|
303
|
+
* Shaded bands are not digitised. A fill drawn with an outline of its own colour (matplotlib's
|
|
304
|
+
`fill_between`) is measured to the middle of the outline; bars are measured to their edges, to
|
|
305
|
+
the nearest pixel where the bars' edges are not antialiased.
|
|
306
|
+
|
|
307
|
+
## Development
|
|
308
|
+
|
|
309
|
+
```bash
|
|
310
|
+
git clone https://github.com/comparativechrono/trace-digitiser.git
|
|
311
|
+
cd trace-digitiser
|
|
312
|
+
pip install -e ".[dev]"
|
|
313
|
+
pytest # the tests that need Tesseract are skipped without it
|
|
314
|
+
ruff check src tests
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
`src/trace_digitiser/` in the order the work is done:
|
|
318
|
+
|
|
319
|
+
| Module | Does |
|
|
320
|
+
|--------|------|
|
|
321
|
+
| `straighten.py` | Measures the tilt of a scan and turns it upright. |
|
|
322
|
+
| `frames.py`, `panel_detection.py` | Finds axis lines and assembles them into plot areas. |
|
|
323
|
+
| `axis_labels.py`, `calibration.py`, `x_calibration.py` | Locates and reads tick labels, fits the scale, rates its confidence; or takes the limits given. |
|
|
324
|
+
| `trace_detection.py` | Separates the traces from gridlines, axes and text. |
|
|
325
|
+
| `digitise.py` | Converts each trace to values, column by column. |
|
|
326
|
+
| `ocr.py`, `summarise.py` | Interval labels along the x axis, and means per interval. |
|
|
327
|
+
| `diagnostics.py`, `provenance.py`, `io.py` | Overlay, record of the run, files. |
|
|
328
|
+
| `validation.py`, `synthetic.py` | Figures drawn from known data, and the comparisons. |
|
|
329
|
+
| `_figure.py` | What every reader does first: the image, straightening, plot areas and axes. |
|
|
330
|
+
| `markers.py`, `points.py` | Markers and their error bars. |
|
|
331
|
+
| `bars.py`, `rows.py`, `heatmap.py`, `polar.py` | Bar charts, stacks of rows, heat maps and polar plots. |
|
|
332
|
+
| `explore.py` | `describe_figure` and `pixels_to_values`, for graphical tools. |
|
|
333
|
+
| `_results.py` | What the readers of points, bars, rows, heat maps and polar plots give back. |
|
|
334
|
+
| `cli.py` | The command line. |
|
|
335
|
+
|
|
336
|
+
Changes are listed in [CHANGELOG.md](https://github.com/comparativechrono/trace-digitiser/blob/main/CHANGELOG.md);
|
|
337
|
+
making a release is described in
|
|
338
|
+
[RELEASING.md](https://github.com/comparativechrono/trace-digitiser/blob/main/RELEASING.md).
|
|
339
|
+
|
|
340
|
+
## Licence
|
|
341
|
+
|
|
342
|
+
MIT; see `LICENSE`. The example figure has its own licence and attribution, given in
|
|
343
|
+
[`examples/README.md`](https://github.com/comparativechrono/trace-digitiser/blob/main/examples/README.md).
|