roo-calamine 0.0.1-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +8 -0
- data/LICENSE +21 -0
- data/README.md +330 -0
- data/lib/roo/calamine/3.3/roo_calamine.so +0 -0
- data/lib/roo/calamine/3.4/roo_calamine.so +0 -0
- data/lib/roo/calamine/4.0/roo_calamine.so +0 -0
- data/lib/roo/calamine/context.rb +57 -0
- data/lib/roo/calamine/sheet_doc.rb +66 -0
- data/lib/roo/calamine/version.rb +7 -0
- data/lib/roo/calamine.rb +47 -0
- metadata +78 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 29a7cb8a6caa3952530441c1f49385791c1f25307e88c17095b522f3d0dfe987
|
|
4
|
+
data.tar.gz: f5ea4edd830abacd2b33ae1a1500dae56fc7ea2e1b392ccbf236b6093266c801
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 5e8f453a6d829f5fb7406143dd991ec17a0c9277a2a9d199d4ec2ae5efa2cfd81b851d04f096908b49cea76aa93e71396ce16738fbedfde0d107b50e6685f16b
|
|
7
|
+
data.tar.gz: 423e4424b591c39a5a81cd510fbaa9d55647a72a7c2a173c48f6da5e2d13d5489cdb38e31962246d43560cd27e77cca719b9584df09552e4e86dcf2d73ab7ad9
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.0.1 (2026-11-10)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* make roo read xlsx files 4-12x faster with identical results ([#1](https://github.com/salasebas/roo-calamine/issues/1)) ([dce0aa4](https://github.com/salasebas/roo-calamine/commit/dce0aa4737b6e376ef01eb4f2aa8f602331d4d0b))
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sebastian Sala
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,330 @@
|
|
|
1
|
+
# roo-calamine
|
|
2
|
+
|
|
3
|
+
Add one line to your Gemfile and [roo](https://github.com/roo-rb/roo) reads
|
|
4
|
+
`.xlsx` and `.xlsm` files 4–12× faster, with the same results, and with a
|
|
5
|
+
quarter of the memory for anything that loads a whole sheet.
|
|
6
|
+
Your roo code does not change.
|
|
7
|
+
|
|
8
|
+
```ruby
|
|
9
|
+
# Gemfile
|
|
10
|
+
gem "roo"
|
|
11
|
+
gem "roo-calamine"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
```ruby
|
|
15
|
+
xlsx = Roo::Spreadsheet.open("orders.xlsx") # or Roo::Excelx.new("orders.xlsx")
|
|
16
|
+
xlsx.each_row_streaming { |row| ... } # now read by the native reader
|
|
17
|
+
xlsx.parse(headers: true) # this too
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
> **Unofficial.** roo-calamine is not affiliated with or endorsed by roo-rb or
|
|
21
|
+
> the roo maintainers. It depends on roo `~> 4.0` and changes only how roo
|
|
22
|
+
> reads worksheet XML.
|
|
23
|
+
|
|
24
|
+
## Contents
|
|
25
|
+
|
|
26
|
+
- [Install](#install)
|
|
27
|
+
- [What it accelerates](#what-it-accelerates)
|
|
28
|
+
- [What stays roo's own code](#what-stays-roos-own-code)
|
|
29
|
+
- [Turning it off](#turning-it-off)
|
|
30
|
+
- [Platforms and JRuby](#platforms-and-jruby)
|
|
31
|
+
- [Correctness: roo is the oracle](#correctness-roo-is-the-oracle)
|
|
32
|
+
- [Benchmarks](#benchmarks)
|
|
33
|
+
- [Design](#design)
|
|
34
|
+
- [Development](#development)
|
|
35
|
+
- [Releasing](#releasing)
|
|
36
|
+
- [Credits](#credits)
|
|
37
|
+
|
|
38
|
+
## Install
|
|
39
|
+
|
|
40
|
+
```ruby
|
|
41
|
+
gem "roo-calamine"
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Bundler requires `roo/calamine` on `Bundler.require`, and that turns the
|
|
45
|
+
native reader on. Without Bundler, `require "roo/calamine"`.
|
|
46
|
+
|
|
47
|
+
Precompiled gems exist for x86_64 and aarch64 Linux (glibc and musl), x86_64
|
|
48
|
+
and arm64 macOS, and 64-bit Windows (`x64-mingw-ucrt`), for Ruby 3.3, 3.4,
|
|
49
|
+
and 4.0. On those platforms `gem install` needs no Rust toolchain. Anywhere
|
|
50
|
+
else the source gem compiles the extension, which needs Rust 1.86 or later.
|
|
51
|
+
|
|
52
|
+
## What it accelerates
|
|
53
|
+
|
|
54
|
+
roo reads every worksheet in one of two places, and roo-calamine replaces
|
|
55
|
+
both, through `Module#prepend` on `Roo::Excelx::SheetDoc`:
|
|
56
|
+
|
|
57
|
+
| roo method | What it now does |
|
|
58
|
+
|---|---|
|
|
59
|
+
| `SheetDoc#cells`, used by `#cell`, `#row`, `#column`, `#each`, `#parse`, `#first_row` … `#last_column`, `#formula`, `#formulas`, `#celltype`, `#excelx_type`, `#excelx_value`, `#empty?`, `#to_s`, `#to_csv` … | reads the sheet in Rust and builds the same `Roo::Excelx::Cell` objects with roo's own constructors |
|
|
60
|
+
| `SheetDoc#each_row_streaming` and `#each_cell`, used by `#each_row_streaming` with `pad_cells`, `offset`, and `max_rows` | streams rows from Rust; a row becomes cells only when roo asks for them |
|
|
61
|
+
|
|
62
|
+
Because the hook sits under `Roo::Excelx`, it covers `Roo::Spreadsheet.open`,
|
|
63
|
+
`Roo::Excelx.new` called directly, and streams (`Roo::Excelx.new(io)`), with
|
|
64
|
+
every roo option (`expand_merged_ranges`, `empty_cell`, `no_hyperlinks`,
|
|
65
|
+
`disable_html_wrapper`, `only_visible_sheets`, `cell_max`, …).
|
|
66
|
+
|
|
67
|
+
## What stays roo's own code
|
|
68
|
+
|
|
69
|
+
The native reader returns roo's own cell objects, so everything that works
|
|
70
|
+
on a cell keeps working through roo's Ruby code: `formatted_value`,
|
|
71
|
+
`excelx_format`, `font`, `hyperlink`, `comment`, `label`, `images`, styles,
|
|
72
|
+
shared strings (including the HTML that roo builds for rich text), base
|
|
73
|
+
dates, and the number-format rules that decide between Integer, Float, Date,
|
|
74
|
+
DateTime and Time.
|
|
75
|
+
|
|
76
|
+
The reader hands a sheet back to roo, which reads it with Nokogiri as usual,
|
|
77
|
+
when the XML uses something it does not reproduce exactly the way Nokogiri
|
|
78
|
+
does:
|
|
79
|
+
|
|
80
|
+
- an XML encoding other than UTF-8 (some Windows Phone exports are UTF-16);
|
|
81
|
+
- a document type declaration, whose entities and attribute defaults Nokogiri
|
|
82
|
+
applies;
|
|
83
|
+
- an entity reference other than the five XML ones and character references;
|
|
84
|
+
- XML that is not well-formed: characters XML does not allow, unbalanced or
|
|
85
|
+
missing tags, malformed attributes and names, `]]>` in text, `--` in
|
|
86
|
+
comments, anything but whitespace, comments, and processing instructions
|
|
87
|
+
around the root element, and a late XML declaration.
|
|
88
|
+
|
|
89
|
+
For `each_row_streaming`, the rows read before such a spot are kept and roo
|
|
90
|
+
continues from the next row, so nothing is read twice and the rows come out
|
|
91
|
+
in order.
|
|
92
|
+
|
|
93
|
+
## Turning it off
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
Roo::Calamine.enabled = false # this process, from the next sheet read on
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
```sh
|
|
100
|
+
ROO_CALAMINE=0 bundle exec rails server # also: false, off
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Both are read on every sheet read, so they work at runtime. With either one,
|
|
104
|
+
roo reads exactly as it does without the gem. `Roo::Calamine.enabled?` tells
|
|
105
|
+
whether the next read uses the native reader, and `Roo::Calamine.available?`
|
|
106
|
+
whether the extension is loaded at all.
|
|
107
|
+
|
|
108
|
+
## Platforms and JRuby
|
|
109
|
+
|
|
110
|
+
| | |
|
|
111
|
+
|---|---|
|
|
112
|
+
| Ruby | CRuby 3.3, 3.4, 4.0 (roo's own floor is 3.3) |
|
|
113
|
+
| roo | `~> 4.0` |
|
|
114
|
+
| OS | Linux (glibc, musl), macOS, Windows |
|
|
115
|
+
| JRuby 10 | installs and stays inert: roo reads every file itself |
|
|
116
|
+
|
|
117
|
+
A native extension cannot load on JRuby. There the gem's `extconf.rb` writes
|
|
118
|
+
an empty Makefile, so `gem install` and `bundle install` succeed without Rust,
|
|
119
|
+
`require "roo/calamine"` loads nothing native, `Roo::Calamine.available?` and
|
|
120
|
+
`enabled?` are `false`, and `Roo::Excelx` is not patched. CI proves this on
|
|
121
|
+
JRuby 10 (`test/jruby_smoke.rb`).
|
|
122
|
+
|
|
123
|
+
## Correctness: roo is the oracle
|
|
124
|
+
|
|
125
|
+
The goal is behavior identical to roo. Every check below takes roo's own
|
|
126
|
+
output as the expected result.
|
|
127
|
+
|
|
128
|
+
**roo 4.0.0's own test suite.** `rake parity` checks out the roo `v4.0.0` tag
|
|
129
|
+
(pinned by SHA), runs its Excelx tests and specs once with plain roo and once
|
|
130
|
+
with roo-calamine loaded, and requires the same results:
|
|
131
|
+
|
|
132
|
+
| Suite | Tests | roo: failed / skipped | roo-calamine: failed / skipped | Native reads | roo cell parses (plugin run) | Same |
|
|
133
|
+
|---|---:|---:|---:|---:|---:|---|
|
|
134
|
+
| `test/roo/test_excelx.rb` | 25 | 0 / 1 | 0 / 1 | 19 | 14 | yes |
|
|
135
|
+
| `test/test_roo.rb` | 41 | 0 / 1 | 0 / 1 | 66 | 0 | yes |
|
|
136
|
+
| `test/excelx` | 51 | 0 / 0 | 0 / 0 | 0 | 0 | yes |
|
|
137
|
+
| `spec/lib/roo/excelx_spec.rb` | 65 | 0 / 0 | 0 / 0 | 35 | 0 | yes |
|
|
138
|
+
| `spec/lib/roo/strict_spec.rb` | 8 | 0 / 0 | 0 / 0 | 12 | 0 | yes |
|
|
139
|
+
| `spec/lib/roo/excelx` | 62 | 0 / 0 | 0 / 0 | 1 | 0 | yes |
|
|
140
|
+
|
|
141
|
+
The skip in both runs is roo's own (`LONG_RUN`). The 14 cells roo parses in
|
|
142
|
+
the plugin run belong to `Pfand_from_windows_phone.xlsx`, whose worksheets
|
|
143
|
+
are UTF-16 and go to roo on purpose.
|
|
144
|
+
|
|
145
|
+
**Every roo fixture, every read path.** `test/roo_fixtures_test.rb` opens
|
|
146
|
+
each of the 84 `.xlsx`/`.xlsm` files in roo's `test/files` and compares plain
|
|
147
|
+
roo with roo-calamine on: the full `cells` hash (every cell's class, value,
|
|
148
|
+
type, formatted value, and every instance variable) under five sets of open
|
|
149
|
+
options; `each_row_streaming` under five sets of `pad_cells`/`offset`/
|
|
150
|
+
`max_rows`; and the `Roo::Base` API (bounds, rows, columns, formulas,
|
|
151
|
+
`to_csv`). It also asserts that the native reader, not roo's parser, served
|
|
152
|
+
each read. 252 tests, 0 differences.
|
|
153
|
+
|
|
154
|
+
**Edge cases.** `test/edge_cases_test.rb` builds workbooks for the cases a
|
|
155
|
+
reader most easily gets wrong, and pins roo's own answer for each first:
|
|
156
|
+
Integer vs Float from the written text (`1` vs `1.0`, `1E3`, a 20-digit
|
|
157
|
+
integer), Date vs DateTime vs Time, the 1904 date system, 1-based and
|
|
158
|
+
missing coordinates, padded and empty cells, merged ranges, hyperlinks,
|
|
159
|
+
shared strings with rich text and `_x000D_`, inline strings, formulas
|
|
160
|
+
(normal, shared, and after the value), cached values, error cells, booleans,
|
|
161
|
+
CDATA and entities, prefixed namespaces, errors roo raises, and a stream
|
|
162
|
+
that hands over to roo halfway.
|
|
163
|
+
|
|
164
|
+
### Known differences
|
|
165
|
+
|
|
166
|
+
No difference in what roo returns has been found. Differences that remain by
|
|
167
|
+
design:
|
|
168
|
+
|
|
169
|
+
- Code that overrides roo's private cell parsing, such as
|
|
170
|
+
`Roo::Excelx::SheetDoc#cell_from_xml` or `#extract_cells`, or
|
|
171
|
+
`Roo::Excelx::Cell.cell_class`, is bypassed while the native reader is on.
|
|
172
|
+
Public methods and the `Cell` classes themselves are used as they are.
|
|
173
|
+
- On a worksheet whose XML breaks partway, `each_row_streaming` raises the
|
|
174
|
+
same `Nokogiri::XML::SyntaxError` as roo, but yields every row before the
|
|
175
|
+
bad spot; roo stops a little earlier, where libxml2's read-ahead reaches
|
|
176
|
+
it (241 of 250 good rows in `test/edge_cases_test.rb`).
|
|
177
|
+
- Resource use changes (see [Benchmarks](#benchmarks)): peak memory is 4–5×
|
|
178
|
+
lower for `cells`-based reads and about 10 MB higher for streaming, and each
|
|
179
|
+
sheet read starts a short-lived native thread that parses ahead of Ruby.
|
|
180
|
+
|
|
181
|
+
## Benchmarks
|
|
182
|
+
|
|
183
|
+
Ruby 3.4.9 (arm64-darwin23), Darwin 25.6.0, Apple M5 Pro, 18 cores, median of 3 runs
|
|
184
|
+
|
|
185
|
+
| File | Scenario | roo | roo + roo-calamine | Speedup | calx (reference) |
|
|
186
|
+
|---|---|---:|---:|---:|---:|
|
|
187
|
+
| mixed-100k.xlsx | `Roo::Spreadsheet.open` + `each_row_streaming` | 14.26 s, 122 MB | 1.31 s, 135 MB | 10.9× | 0.54 s, 187 MB |
|
|
188
|
+
| mixed-100k.xlsx | `parse(headers: true)` | 20.96 s, 3431 MB | 3.48 s, 860 MB | 6.0× | 0.74 s, 298 MB |
|
|
189
|
+
| mixed-100k.xlsx | 10,000 random `cell(row, col)` | 20.74 s, 3267 MB | 1.94 s, 660 MB | 10.7× | 0.52 s, 258 MB |
|
|
190
|
+
| shared-strings-100k.xlsx | `Roo::Spreadsheet.open` + `each_row_streaming` | 5.25 s, 124 MB | 0.43 s, 133 MB | 12.3× | 0.24 s, 122 MB |
|
|
191
|
+
| shared-strings-100k.xlsx | `parse(headers: true)` | 6.27 s, 1663 MB | 1.47 s, 396 MB | 4.3× | 0.36 s, 201 MB |
|
|
192
|
+
| shared-strings-100k.xlsx | 10,000 random `cell(row, col)` | 5.40 s, 1598 MB | 0.70 s, 307 MB | 7.8× | 0.25 s, 176 MB |
|
|
193
|
+
| small.xlsx | `Roo::Spreadsheet.open` + `each_row_streaming` | 0.02 s, 48 MB | 0.00 s, 43 MB | 4.5× | 0.00 s, 35 MB |
|
|
194
|
+
| small.xlsx | `parse(headers: true)` | 0.01 s, 48 MB | 0.01 s, 43 MB | 2.1× | 0.00 s, 35 MB |
|
|
195
|
+
| small.xlsx | 10,000 random `cell(row, col)` | 0.02 s, 48 MB | 0.01 s, 45 MB | 1.7× | 0.00 s, 35 MB |
|
|
196
|
+
|
|
197
|
+
`bench/run.rb` generates the workbooks with
|
|
198
|
+
[caxlsx](https://github.com/caxlsx/caxlsx): 100,000 rows × 20 columns of
|
|
199
|
+
mixed types, 100,000 rows × 10 columns of repeated shared strings, and 100
|
|
200
|
+
rows × 20 columns. Each cell of the table is the median of three runs, each
|
|
201
|
+
in a fresh process; memory is the process's peak RSS. Before timing, one run
|
|
202
|
+
of roo and one of roo-calamine hash every value they read, and the two
|
|
203
|
+
hashes must match. [calx](https://github.com/7a6163/calx) is a reference for
|
|
204
|
+
a reader that skips roo's cell model: it returns plain values, not roo
|
|
205
|
+
cells, so it is not a drop-in replacement.
|
|
206
|
+
|
|
207
|
+
```sh
|
|
208
|
+
cd bench && bundle install && bundle exec ruby run.rb
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
## Design
|
|
212
|
+
|
|
213
|
+
### Why a plugin
|
|
214
|
+
|
|
215
|
+
roo is maintained and widely used, so a fork would split users. A pull request
|
|
216
|
+
to roo would make every roo user build a Rust extension. A plugin keeps roo
|
|
217
|
+
as it is and maintains only the fast path; anything the fast path does not
|
|
218
|
+
handle stays roo's own code.
|
|
219
|
+
|
|
220
|
+
### The seam
|
|
221
|
+
|
|
222
|
+
Three places were candidates: (a) a `Roo::Excelx` subclass registered in
|
|
223
|
+
`Roo::CLASS_FOR_EXTENSION`, (b) replacing how the cells of a sheet are
|
|
224
|
+
built, and (c) a `prepend` on roo's classes. roo-calamine uses (b) through
|
|
225
|
+
(c): `Roo::Excelx::SheetDoc` is prepended with `#cells`,
|
|
226
|
+
`#each_row_streaming`, and `#each_cell`.
|
|
227
|
+
|
|
228
|
+
All three run the same native code, so they cost the same per cell; what
|
|
229
|
+
differs is what they cover. On the 100k × 20 workbook, `Sheet#cells` takes
|
|
230
|
+
2.2 s through both `Roo::Spreadsheet.open` and `Roo::Excelx.new`. (a) would
|
|
231
|
+
miss `Roo::Excelx.new` and still have to replace sheet reading inside the
|
|
232
|
+
subclass. Prepending `SheetDoc` covers every entry point, keeps roo's own
|
|
233
|
+
`Sheet#each_row` (with its `pad_cells`, `offset`, and `max_rows` logic), and
|
|
234
|
+
lets the kill switch call `super`.
|
|
235
|
+
|
|
236
|
+
### The engine: calamine's parser stack, not calamine's cell model
|
|
237
|
+
|
|
238
|
+
The extension was planned over [calamine](https://github.com/tafia/calamine)
|
|
239
|
+
0.38. calamine's cell reader turns every number into an `f64` and returns
|
|
240
|
+
shared strings already resolved, but roo's results depend on the raw XML:
|
|
241
|
+
|
|
242
|
+
- roo returns `1` (Integer) for `<v>1</v>` and `1.0` (Float) for `<v>1.0</v>`,
|
|
243
|
+
and exposes the raw text as `cell_value`/`excelx_value`. caxlsx, Apache POI
|
|
244
|
+
and others write `1.0` for floats, so calamine would turn those Float
|
|
245
|
+
columns into Integer. calamine declined to keep integers apart
|
|
246
|
+
([tafia/calamine#669](https://github.com/tafia/calamine/pull/669): "all
|
|
247
|
+
numbers are f64").
|
|
248
|
+
- roo wraps rich-text shared strings in HTML by their index; calamine returns
|
|
249
|
+
plain text.
|
|
250
|
+
- roo's handling of `t="e"`, `t="inlineStr"` with a `<v>`, phonetic text, and
|
|
251
|
+
rows without `r` differs from calamine's.
|
|
252
|
+
|
|
253
|
+
Measured on roo, XML parsing was not the cost anyway: on 400,000 cells,
|
|
254
|
+
Nokogiri's parse took 0.23 s of roo's 2.8 s; the rest was per-cell Ruby work.
|
|
255
|
+
So the extension reads the worksheet with
|
|
256
|
+
[quick-xml](https://github.com/tafia/quick-xml), the XML parser calamine is
|
|
257
|
+
built on, into raw cell records that mirror what roo's `cell_from_xml` sees,
|
|
258
|
+
and builds cells with roo's own constructors. The name stays; calamine
|
|
259
|
+
itself is not a dependency.
|
|
260
|
+
|
|
261
|
+
### Optimizations
|
|
262
|
+
|
|
263
|
+
Each was measured, and adopted only with a win and unchanged results.
|
|
264
|
+
|
|
265
|
+
| | Measured | Decision |
|
|
266
|
+
|---|---|---|
|
|
267
|
+
| **1. Streaming, not whole-sheet loading** | Both read paths stream the XML; nothing loads a whole sheet into Rust. `each_row_streaming(max_rows: 100)` on the 100k file returns in 0.08 s. Peak RSS for streaming is in the table. | Adopted |
|
|
268
|
+
| **2. Parsing without the GVL** | A native thread parses the next batch of about 16,384 cells while Ruby turns the current one into cells; Ruby releases the GVL only when it has to wait for a batch. 100k × 20, alone / four threads reading at once / next to a busy Ruby thread, median of three: `cells` 1.58 / 6.96 / 4.07 s, streaming 1.18 / 4.70 / 3.93 s. Holding the GVL while parsing, as calx does: 2.49 / 11.65 / 6.75 s and 2.29 / 8.61 / 5.23 s. Releasing it around each batch instead: 2.52 / 10.15 / 5.24 s and 2.63 / 5.53 / 5.82 s, and next to a busy thread with 128-row batches 19.8 s, because each handoff waits for the other thread's time slice. (Alternatives: single runs.) | Adopted, as a read-ahead thread |
|
|
269
|
+
| **3. One String per shared string** | Cells reuse the String objects of roo's own shared-string table, as roo does. 1M shared-string cells: 0.73 s and 2.0M allocated objects, against 0.80 s and 3.0M with a new String per cell (roo: 5.46 s, 14.6M). Inline strings and numbers keep one String per cell, because roo returns distinct, mutable strings for them. | Adopted |
|
|
270
|
+
| **4. No conversions roo does not need** | No Nokogiri nodes, attribute strings, or `"A1"` strings per cell; a style's format and type are looked up once per style; a streamed row becomes cells only when roo yields it: `each_row_streaming(offset: 90_000)` takes 0.67 s against 1.82 s for every row. Lazy Date objects inside `cells` were rejected: roo's cells compute their value in the constructor, and the oracle tests compare every instance variable. | Adopted, except lazy values |
|
|
271
|
+
|
|
272
|
+
A fifth change came from profiling: keeping a row's text in one buffer and
|
|
273
|
+
its attributes as typed values cut the Rust parse of the 100k × 20 sheet from
|
|
274
|
+
0.79 s to 0.47 s. The well-formedness checks added later cost some of that
|
|
275
|
+
back, on the read-ahead thread, where it does not show in wall time
|
|
276
|
+
(streaming the same sheet: 1.41–1.45 s before and after).
|
|
277
|
+
|
|
278
|
+
## Development
|
|
279
|
+
|
|
280
|
+
```sh
|
|
281
|
+
bundle install
|
|
282
|
+
bundle exec rake compile # build the extension into lib/roo/calamine/
|
|
283
|
+
bundle exec rake test # unit, fixture, and edge-case tests (fetches roo 4.0.0's fixtures)
|
|
284
|
+
bundle exec rake parity # roo 4.0.0's own suites, with and without roo-calamine
|
|
285
|
+
bundle exec rubocop
|
|
286
|
+
cargo test && cargo clippy --all-targets -- -D warnings && cargo fmt --check
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
## Releasing
|
|
290
|
+
|
|
291
|
+
Releases are cut by [release-please](https://github.com/googleapis/release-please)
|
|
292
|
+
from Conventional Commit titles (pull requests are squash-merged), and
|
|
293
|
+
published by `.github/workflows/release.yml` through RubyGems.org
|
|
294
|
+
[trusted publishing](https://guides.rubygems.org/trusted-publishing/): no API
|
|
295
|
+
key is stored anywhere.
|
|
296
|
+
|
|
297
|
+
1. Merging a pull request into `main` updates release-please's release pull
|
|
298
|
+
request (version and CHANGELOG). `release.yml` runs the required checks
|
|
299
|
+
on it, because pull requests opened with `GITHUB_TOKEN` trigger no
|
|
300
|
+
workflows by themselves.
|
|
301
|
+
2. Merging the release pull request tags `vX.Y.Z`, creates the GitHub
|
|
302
|
+
release, builds the source gem and the seven platform gems, installs each
|
|
303
|
+
on its platform without Rust, and pushes all eight to RubyGems.org from the
|
|
304
|
+
`rubygems` environment.
|
|
305
|
+
|
|
306
|
+
The trusted publisher on RubyGems.org (registered as a pending publisher
|
|
307
|
+
before the first release, since the gem did not exist yet):
|
|
308
|
+
|
|
309
|
+
| Field | Value |
|
|
310
|
+
|---|---|
|
|
311
|
+
| Gem name | `roo-calamine` |
|
|
312
|
+
| Repository owner | `salasebas` |
|
|
313
|
+
| Repository name | `roo-calamine` |
|
|
314
|
+
| Workflow filename | `release.yml` |
|
|
315
|
+
| Environment | `rubygems` |
|
|
316
|
+
|
|
317
|
+
## Credits
|
|
318
|
+
|
|
319
|
+
- [roo](https://github.com/roo-rb/roo) (MIT) does all the spreadsheet work;
|
|
320
|
+
this gem only reads worksheet XML for it.
|
|
321
|
+
- [calamine](https://github.com/tafia/calamine) (MIT) and its XML parser
|
|
322
|
+
[quick-xml](https://github.com/tafia/quick-xml) (MIT).
|
|
323
|
+
- [calx](https://github.com/7a6163/calx) (MIT), the reference for the native
|
|
324
|
+
extension; `.github/workflows/build-gems.yml` adapts its cross-compilation
|
|
325
|
+
workflow.
|
|
326
|
+
- [magnus](https://github.com/matsadler/magnus) and
|
|
327
|
+
[rb-sys](https://github.com/oxidize-rb/rb-sys) for the Ruby bindings and
|
|
328
|
+
the platform gems.
|
|
329
|
+
|
|
330
|
+
MIT License. See [LICENSE](LICENSE).
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Roo
|
|
4
|
+
module Calamine
|
|
5
|
+
# The answers the native reader needs from roo while it builds cells. Each
|
|
6
|
+
# one comes from the roo object or method that SheetDoc#cell_from_xml uses,
|
|
7
|
+
# so formats, shared strings, and errors are roo's own.
|
|
8
|
+
class Context
|
|
9
|
+
attr_reader :hyperlinks, :empty_cell
|
|
10
|
+
|
|
11
|
+
def initialize(shared, hyperlinks, empty_cell)
|
|
12
|
+
@shared = shared
|
|
13
|
+
@hyperlinks = hyperlinks
|
|
14
|
+
@empty_cell = empty_cell
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def base_date
|
|
18
|
+
@shared.base_date
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def base_timestamp
|
|
22
|
+
@shared.base_timestamp
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def style(index)
|
|
26
|
+
format = @shared.styles.style_format(index)
|
|
27
|
+
[format.to_s, Roo::Excelx::Format.to_type(format)]
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def style_index(text)
|
|
31
|
+
text.to_i
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def coordinate(reference)
|
|
35
|
+
Roo::Utils.extract_coordinate(reference)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# roo's own memoized shared-string arrays: plain text, and the HTML that
|
|
39
|
+
# SharedStrings#use_html? tests (nil when the HTML wrapper is disabled).
|
|
40
|
+
def shared_strings
|
|
41
|
+
strings = @shared.shared_strings
|
|
42
|
+
html_disabled = strings.instance_variable_get(:@options)[:disable_html_wrapper]
|
|
43
|
+
[strings.to_a, (strings.to_html unless html_disabled)]
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def shared_string(text)
|
|
47
|
+
shared_string_at(@shared.shared_strings, text.to_i)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
private
|
|
51
|
+
|
|
52
|
+
def shared_string_at(strings, index)
|
|
53
|
+
strings.use_html?(index) ? strings.to_html[index] : strings[index]
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
end
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Roo
|
|
4
|
+
module Calamine
|
|
5
|
+
# Prepended to Roo::Excelx::SheetDoc, where every worksheet read in roo
|
|
6
|
+
# starts: Sheet#cells (and with it #cell, #row, #each, #parse) and
|
|
7
|
+
# Sheet#each_row (#each_row_streaming). Everything else, including roo's
|
|
8
|
+
# padding, offset, and row limits, stays roo's code.
|
|
9
|
+
module SheetDoc
|
|
10
|
+
# Cells the native reader parses ahead of Ruby, per batch.
|
|
11
|
+
BATCH_CELLS = 16_384
|
|
12
|
+
|
|
13
|
+
def cells(relationships)
|
|
14
|
+
return super if @cells || !native?
|
|
15
|
+
|
|
16
|
+
cells = {}
|
|
17
|
+
reader = Native::Reader.new(@path, false, BATCH_CELLS)
|
|
18
|
+
converter = Native::Converter.new(Context.new(@shared, hyperlinks(relationships), @options[:empty_cell]), false)
|
|
19
|
+
nil while reader.fill_cells(cells, converter)
|
|
20
|
+
expand_merged_ranges(cells) if @options[:expand_merged_ranges]
|
|
21
|
+
@cells = cells
|
|
22
|
+
rescue UnsupportedError
|
|
23
|
+
super
|
|
24
|
+
ensure
|
|
25
|
+
reader&.close
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Yields Native::Row objects; #each_cell turns one into cells when roo
|
|
29
|
+
# asks for it, so rows roo skips are never converted, as in roo.
|
|
30
|
+
def each_row_streaming(&)
|
|
31
|
+
return super unless native?
|
|
32
|
+
|
|
33
|
+
yielded = 0
|
|
34
|
+
reader = Native::Reader.new(@path, true, BATCH_CELLS)
|
|
35
|
+
while (rows = reader.next_rows)
|
|
36
|
+
rows.each do |row|
|
|
37
|
+
yield row
|
|
38
|
+
yielded += 1
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
nil
|
|
42
|
+
rescue UnsupportedError
|
|
43
|
+
# roo reads the sheet again; skip the rows already yielded.
|
|
44
|
+
skip = yielded
|
|
45
|
+
super { |row| skip.zero? ? yield(row) : skip -= 1 }
|
|
46
|
+
ensure
|
|
47
|
+
# Also when the caller stops early, as roo does for max_rows.
|
|
48
|
+
reader&.close
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def each_cell(row_xml, &)
|
|
52
|
+
return super unless row_xml.is_a?(Native::Row)
|
|
53
|
+
|
|
54
|
+
@calamine_stream_converter ||= Native::Converter.new(Context.new(@shared, hyperlinks(@relationships), true),
|
|
55
|
+
true)
|
|
56
|
+
@calamine_stream_converter.row_cells(row_xml).each(&)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
private
|
|
60
|
+
|
|
61
|
+
def native?
|
|
62
|
+
@path && Calamine.enabled?
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
end
|
data/lib/roo/calamine.rb
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "roo"
|
|
4
|
+
require "roo/excelx"
|
|
5
|
+
require_relative "calamine/version"
|
|
6
|
+
|
|
7
|
+
module Roo
|
|
8
|
+
# Reads `.xlsx` and `.xlsm` worksheets for Roo::Excelx with a native
|
|
9
|
+
# extension. Loading this file is all it takes; `Roo::Calamine.enabled =
|
|
10
|
+
# false` or `ROO_CALAMINE=0` switches back to roo's own reader at runtime.
|
|
11
|
+
module Calamine
|
|
12
|
+
# Raised by the native reader for XML it does not reproduce exactly the
|
|
13
|
+
# way roo reads it. Roo::Calamine rescues it and lets roo read the sheet.
|
|
14
|
+
class UnsupportedError < StandardError; end
|
|
15
|
+
|
|
16
|
+
DISABLING_ENV_VALUES = %w[0 false off].freeze
|
|
17
|
+
private_constant :DISABLING_ENV_VALUES
|
|
18
|
+
|
|
19
|
+
@enabled = true
|
|
20
|
+
|
|
21
|
+
class << self
|
|
22
|
+
attr_writer :enabled
|
|
23
|
+
|
|
24
|
+
# Whether the native extension is loaded. It is `false` on Ruby
|
|
25
|
+
# implementations other than CRuby, such as JRuby.
|
|
26
|
+
def available?
|
|
27
|
+
defined?(Native::Reader) ? true : false
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Whether the next worksheet read goes through the native reader.
|
|
31
|
+
def enabled?
|
|
32
|
+
@enabled && available? && !DISABLING_ENV_VALUES.include?(ENV.fetch("ROO_CALAMINE", "").downcase)
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
if RUBY_ENGINE == "ruby"
|
|
39
|
+
begin
|
|
40
|
+
require "roo/calamine/#{RUBY_VERSION[/\d+\.\d+/]}/roo_calamine"
|
|
41
|
+
rescue LoadError
|
|
42
|
+
require "roo/calamine/roo_calamine"
|
|
43
|
+
end
|
|
44
|
+
require_relative "calamine/context"
|
|
45
|
+
require_relative "calamine/sheet_doc"
|
|
46
|
+
Roo::Excelx::SheetDoc.prepend(Roo::Calamine::SheetDoc)
|
|
47
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: roo-calamine
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.0.1
|
|
5
|
+
platform: aarch64-linux
|
|
6
|
+
authors:
|
|
7
|
+
- Sebastian Sala
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2026-11-10 00:00:00.000000000 Z
|
|
12
|
+
dependencies:
|
|
13
|
+
- !ruby/object:Gem::Dependency
|
|
14
|
+
name: roo
|
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
|
16
|
+
requirements:
|
|
17
|
+
- - "~>"
|
|
18
|
+
- !ruby/object:Gem::Version
|
|
19
|
+
version: '4.0'
|
|
20
|
+
type: :runtime
|
|
21
|
+
prerelease: false
|
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
23
|
+
requirements:
|
|
24
|
+
- - "~>"
|
|
25
|
+
- !ruby/object:Gem::Version
|
|
26
|
+
version: '4.0'
|
|
27
|
+
description: Unofficial plugin for roo. Add it to the Gemfile and Roo::Excelx reads
|
|
28
|
+
worksheets through a Rust extension instead of Nokogiri, building the same Roo::Excelx::Cell
|
|
29
|
+
objects. Anything the native reader does not reproduce exactly goes to roo's own
|
|
30
|
+
code. Not affiliated with roo-rb.
|
|
31
|
+
email:
|
|
32
|
+
executables: []
|
|
33
|
+
extensions: []
|
|
34
|
+
extra_rdoc_files: []
|
|
35
|
+
files:
|
|
36
|
+
- CHANGELOG.md
|
|
37
|
+
- LICENSE
|
|
38
|
+
- README.md
|
|
39
|
+
- lib/roo/calamine.rb
|
|
40
|
+
- lib/roo/calamine/3.3/roo_calamine.so
|
|
41
|
+
- lib/roo/calamine/3.4/roo_calamine.so
|
|
42
|
+
- lib/roo/calamine/4.0/roo_calamine.so
|
|
43
|
+
- lib/roo/calamine/context.rb
|
|
44
|
+
- lib/roo/calamine/sheet_doc.rb
|
|
45
|
+
- lib/roo/calamine/version.rb
|
|
46
|
+
homepage: https://github.com/salasebas/roo-calamine
|
|
47
|
+
licenses:
|
|
48
|
+
- MIT
|
|
49
|
+
metadata:
|
|
50
|
+
homepage_uri: https://github.com/salasebas/roo-calamine
|
|
51
|
+
source_code_uri: https://github.com/salasebas/roo-calamine/tree/v0.0.1
|
|
52
|
+
changelog_uri: https://github.com/salasebas/roo-calamine/blob/v0.0.1/CHANGELOG.md
|
|
53
|
+
bug_tracker_uri: https://github.com/salasebas/roo-calamine/issues
|
|
54
|
+
rubygems_mfa_required: 'true'
|
|
55
|
+
post_install_message:
|
|
56
|
+
rdoc_options: []
|
|
57
|
+
require_paths:
|
|
58
|
+
- lib
|
|
59
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
60
|
+
requirements:
|
|
61
|
+
- - ">="
|
|
62
|
+
- !ruby/object:Gem::Version
|
|
63
|
+
version: '3.3'
|
|
64
|
+
- - "<"
|
|
65
|
+
- !ruby/object:Gem::Version
|
|
66
|
+
version: 4.1.dev
|
|
67
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
68
|
+
requirements:
|
|
69
|
+
- - ">="
|
|
70
|
+
- !ruby/object:Gem::Version
|
|
71
|
+
version: 3.3.11
|
|
72
|
+
requirements: []
|
|
73
|
+
rubygems_version: 3.5.23
|
|
74
|
+
signing_key:
|
|
75
|
+
specification_version: 4
|
|
76
|
+
summary: A native reader that makes roo read .xlsx and .xlsm files faster, with the
|
|
77
|
+
same results.
|
|
78
|
+
test_files: []
|