calcsheet-native 0.1.0-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +36 -0
- data/LICENSE.txt +21 -0
- data/README.md +65 -0
- data/lib/calcsheet/native/3.3/calcsheet_native.so +0 -0
- data/lib/calcsheet/native/3.4/calcsheet_native.so +0 -0
- data/lib/calcsheet/native/4.0/calcsheet_native.so +0 -0
- data/lib/calcsheet/native/csv_reader.rb +122 -0
- data/lib/calcsheet/native/ods_reader.rb +148 -0
- data/lib/calcsheet/native/version.rb +8 -0
- data/lib/calcsheet/native/xlsx_reader.rb +208 -0
- data/lib/calcsheet/native.rb +31 -0
- metadata +84 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: fde3e8255bd8a834f6ddb21d36271b9890d1ea5f4bb30a72b5dfeebecd7e7784
|
|
4
|
+
data.tar.gz: 7cffd309408492eb7653cce47fd7ef4a7cf328b6a7cfc38d4bbed459c8e1706d
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: fa982556f9f62090b9b2f8a397f9a7100fa408d3b879d25e7405ab950312b4730d6e3d5cdbd2f84065e85c53e2da27c131c6bac506266bdc72d9958ada839da5
|
|
7
|
+
data.tar.gz: b26e370a8312173df81ea4ed5a30dcbad2acb01432c143dcdfc4b40fcb8de7c928b73f1b892ed0f853ef272b511565b0105eb47060c03d3efaeeb8a86b41870b
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this gem are documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this gem adheres to
|
|
6
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-10-28
|
|
11
|
+
|
|
12
|
+
First release, for calcsheet 0.2 and 0.3 (protocol 1), with precompiled gems for `x86_64-linux`, `aarch64-linux`,
|
|
13
|
+
`x86_64-linux-musl`, `aarch64-linux-musl`, `arm64-darwin`, `x86_64-darwin` and `x64-mingw-ucrt` (Ruby 3.3, 3.4 and
|
|
14
|
+
4.0) beside the source gem.
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- The gem skeleton: a Rust extension built with magnus and rb_sys, `Calcsheet::Native::PROTOCOL` 1 and
|
|
19
|
+
`Calcsheet::Native.backends`.
|
|
20
|
+
- The native CSV backend: a Rust tokenizer that accepts and refuses what the stdlib CSV parser does, run without the
|
|
21
|
+
GVL, under calcsheet's own CSV reader (dialect, encodings, numbering, checks and conversion). About twice as fast
|
|
22
|
+
as the Ruby backend on large files, with the same memory.
|
|
23
|
+
- The native XLSX backend: the XML of each sheet is read in Rust, without the GVL, for a plain subset of XML and the
|
|
24
|
+
values calcsheet accepts; anything else is read again by calcsheet's own parser, so results and errors are always
|
|
25
|
+
the Ruby backend's. The archive, its limits, the workbook, styles and shared strings stay calcsheet's. About 1.7
|
|
26
|
+
times as fast as the Ruby backend, with the same memory.
|
|
27
|
+
- The native ODS backend, built the same way: the XML of each table is read in Rust, the Scanner that finds tables
|
|
28
|
+
and rows, the archive and the cell styles stay calcsheet's. About twice as fast as the Ruby backend, with the same
|
|
29
|
+
memory.
|
|
30
|
+
- XLSX shared strings read in Rust, in one table shared by the sheets, and a faster XML lexer: with YJIT, XLSX
|
|
31
|
+
reads in 8.5 s per million rows instead of 10.3 s, ODS in 12.3 s instead of 13.6 s.
|
|
32
|
+
- An ODS read finds its sheet without listing every sheet (calcsheet's ADR 013): with YJIT, 9.1 s per million rows
|
|
33
|
+
instead of 12.3 s, and about 40 MB of peak memory instead of 100.
|
|
34
|
+
|
|
35
|
+
[Unreleased]: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/compare/v0.1.0...develop
|
|
36
|
+
[0.1.0]: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/tags/v0.1.0
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Demetra Opinioni.net Srl
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# calcsheet-native
|
|
2
|
+
|
|
3
|
+
Native reader backends for [calcsheet](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet): a Rust
|
|
4
|
+
core reads CSV, XLSX and ODS (inflating, tokenizing, limits) without the GVL and hands raw cells to calcsheet, which
|
|
5
|
+
converts them exactly as its Ruby backends do. With the same file and the same options both backends must give the
|
|
6
|
+
same rows, cell errors and exceptions; the only allowed differences are listed in
|
|
7
|
+
[docs/parity-exceptions.md](docs/parity-exceptions.md).
|
|
8
|
+
|
|
9
|
+
## Status
|
|
10
|
+
|
|
11
|
+
Released on rubygems.org. CSV, XLSX and ODS are read natively. Precompiled gems exist for `x86_64-linux`,
|
|
12
|
+
`aarch64-linux`, `x86_64-linux-musl`, `aarch64-linux-musl`, `arm64-darwin`, `x86_64-darwin` and `x64-mingw-ucrt`, each
|
|
13
|
+
for Ruby 3.3, 3.4 and 4.0; any other platform installs the source gem, which needs the Rust toolchain and libclang
|
|
14
|
+
(see Development).
|
|
15
|
+
|
|
16
|
+
Seconds per million rows of 11 columns (Ruby 4.0, x86_64; XLSX and ODS measured on 100,000 rows), end to end
|
|
17
|
+
with the values converted; the peak memory is the same with both backends:
|
|
18
|
+
|
|
19
|
+
| File | Ruby | Ruby + YJIT | Native | Native + YJIT |
|
|
20
|
+
|---|---|---|---|---|
|
|
21
|
+
| CSV | 7.5 s | 3.9 s | 3.7 s | 2.0 s |
|
|
22
|
+
| XLSX | 27.8 s | 16.9 s | 12.0 s | 8.5 s |
|
|
23
|
+
| ODS | 35.7 s | 17.7 s | 14.8 s | 9.1 s |
|
|
24
|
+
|
|
25
|
+
With YJIT, the conversion of the cells in Ruby (calcsheet's ADR 011) is 30 to 45% of the native time. Two reads in
|
|
26
|
+
two threads take 1.7 to 2 times as long as one.
|
|
27
|
+
|
|
28
|
+
## Usage
|
|
29
|
+
|
|
30
|
+
Add it next to calcsheet; nothing else changes in the application:
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
gem "calcsheet", "~> 0.3"
|
|
34
|
+
gem "calcsheet-native"
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
It needs calcsheet 0.2 or 0.3 (protocol 1). calcsheet uses it under `backend: :auto` (the default) for the formats it reads, and falls back to the Ruby backends,
|
|
38
|
+
with one warning, when it is installed but unusable.
|
|
39
|
+
|
|
40
|
+
## Development
|
|
41
|
+
|
|
42
|
+
Requirements: Ruby 3.3 or later, the Rust toolchain of `rust-toolchain.toml` (rustup installs it) and libclang
|
|
43
|
+
(Debian: `libclang-dev`), which rb_sys uses to generate the Ruby bindings.
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
bundle install
|
|
47
|
+
bundle exec rake compile
|
|
48
|
+
bundle exec rake # compile, specs, calcsheet's suite on the native backends, RuboCop
|
|
49
|
+
cargo fmt --check && cargo clippy --all-targets -- -D warnings
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
The Gemfile takes calcsheet from the tag of its release, whose suite `rake parity` runs (the published gem leaves
|
|
53
|
+
it out).
|
|
54
|
+
|
|
55
|
+
A release builds the precompiled gems and the source gem into `pkg/` with Docker, from a checkout of its tag, then
|
|
56
|
+
pushes each of them (rubygems.org asks for the security key of every push):
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
bin/build-native-gems
|
|
60
|
+
for gem in pkg/*.gem; do gem push "$gem"; done
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## License
|
|
64
|
+
|
|
65
|
+
MIT, see [LICENSE.txt](LICENSE.txt).
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "csv"
|
|
4
|
+
|
|
5
|
+
module Calcsheet
|
|
6
|
+
module Native
|
|
7
|
+
# The native CSV backend: calcsheet's Ruby reader (dialect detection, encodings, numbering, checks and the
|
|
8
|
+
# conversion of the fields) with the records tokenized in Rust instead of by the stdlib CSV parser. A dialect the
|
|
9
|
+
# tokenizer does not handle (a separator or quote that is not one ASCII character, or a line ending the sample
|
|
10
|
+
# did not show) is parsed by the stdlib, as the Ruby backend does.
|
|
11
|
+
#
|
|
12
|
+
# It depends on the private methods of Calcsheet::Readers::CsvReader of the calcsheet minor version it supports.
|
|
13
|
+
class CsvReader < Calcsheet::Readers::CsvReader
|
|
14
|
+
# The line endings the tokenizer handles.
|
|
15
|
+
ROW_SEPARATORS = [ "\n", "\r", "\r\n" ].freeze
|
|
16
|
+
|
|
17
|
+
# The stdlib CSV parser's place for the reader: CsvReader#shift pulls records with +each+ and +next+, and
|
|
18
|
+
# reads +row_sep+ when a record is malformed.
|
|
19
|
+
class Records
|
|
20
|
+
# Records parsed per call into Rust.
|
|
21
|
+
BATCH = 1024
|
|
22
|
+
EMPTY = [].freeze
|
|
23
|
+
|
|
24
|
+
# @return [String] the line ending
|
|
25
|
+
attr_reader :row_sep
|
|
26
|
+
# @return [Integer] the physical lines the last record ends
|
|
27
|
+
attr_reader :breaks
|
|
28
|
+
# @return [Boolean] whether the last record has the stray line break outside quoted fields
|
|
29
|
+
attr_reader :stray
|
|
30
|
+
|
|
31
|
+
# @param input [Calcsheet::Readers::CsvInput]
|
|
32
|
+
# @param dialect [Calcsheet::Readers::CsvDialect::Dialect]
|
|
33
|
+
def initialize(input, dialect)
|
|
34
|
+
@input = input
|
|
35
|
+
@row_sep = dialect.row_sep
|
|
36
|
+
@tokenizer = CsvTokenizer.new(dialect.col_sep, dialect.quote_char, dialect.row_sep)
|
|
37
|
+
@fields = @all_breaks = @strays = EMPTY
|
|
38
|
+
@index = 0
|
|
39
|
+
@breaks = 0
|
|
40
|
+
@stray = false
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# @return [Records] self: #next gives the records
|
|
44
|
+
def each
|
|
45
|
+
self
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# @return [Array<String, nil>] the fields of the next record
|
|
49
|
+
# @raise [StopIteration] at the end
|
|
50
|
+
# @raise [CSV::MalformedCSVError] for a malformed record
|
|
51
|
+
# @raise [EncodingError] from the input, at the record that holds the invalid bytes
|
|
52
|
+
def next
|
|
53
|
+
if @index == @fields.size
|
|
54
|
+
@fields, @all_breaks, @strays = batch
|
|
55
|
+
@index = 0
|
|
56
|
+
raise StopIteration if @fields.empty?
|
|
57
|
+
end
|
|
58
|
+
index = @index
|
|
59
|
+
@index += 1
|
|
60
|
+
@breaks = @all_breaks[index]
|
|
61
|
+
@stray = @strays[index]
|
|
62
|
+
@fields[index]
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
67
|
+
# The next records, reading the input only when the text read ends inside a record.
|
|
68
|
+
def batch
|
|
69
|
+
loop do
|
|
70
|
+
records = @tokenizer.records(BATCH)
|
|
71
|
+
return records if records
|
|
72
|
+
raise CSV::MalformedCSVError.new(*@tokenizer.error) if records == false
|
|
73
|
+
|
|
74
|
+
chunk = @input.gets(nil, Calcsheet::Readers::CsvInput::CHUNK_SIZE)
|
|
75
|
+
chunk ? @tokenizer.feed(chunk) : @tokenizer.finish
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
private
|
|
81
|
+
|
|
82
|
+
# Parses with the tokenizer when it handles the dialect.
|
|
83
|
+
def restart(forced)
|
|
84
|
+
super
|
|
85
|
+
@csv = Records.new(@input, @dialect) if tokenizes?(@dialect)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# The text the parser reads, kept for the tokenizer.
|
|
89
|
+
def input
|
|
90
|
+
@input = super
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def tokenizes?(dialect)
|
|
94
|
+
ROW_SEPARATORS.include?(dialect.row_sep) && [ dialect.col_sep, dialect.quote_char ].all? { _1.bytesize == 1 }
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# CsvReader#each_row, with the line breaks and the stray line break of each record from the tokenizer instead
|
|
98
|
+
# of its raw text.
|
|
99
|
+
def each_row(sheet_index, cast, types, start, &block)
|
|
100
|
+
forced = forced_dialect(cast.options)
|
|
101
|
+
restart(forced) if @csv.nil? || start <= @records || forced != @forced
|
|
102
|
+
return super unless @csv.is_a?(Records)
|
|
103
|
+
|
|
104
|
+
strip = cast.options.strip_formula_prefix
|
|
105
|
+
builder = RowBuilder.new(cast, types)
|
|
106
|
+
while (fields = shift)
|
|
107
|
+
number = @records += 1
|
|
108
|
+
line = @line
|
|
109
|
+
@line += @csv.breaks
|
|
110
|
+
mixed_line_endings(number, line, @csv.row_sep) if @csv.stray
|
|
111
|
+
next if number < start
|
|
112
|
+
|
|
113
|
+
last = fields.size - 1
|
|
114
|
+
last -= 1 while last >= 0 && blank_field?(fields[last], number, last)
|
|
115
|
+
next if last.negative?
|
|
116
|
+
|
|
117
|
+
yield raw_row(builder, number, line, fields, last, strip)
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Calcsheet
|
|
4
|
+
module Native
|
|
5
|
+
# The native ODS backend: calcsheet's Ruby reader (the zip archive and its limits, the Scanner that finds the
|
|
6
|
+
# tables, the rows of +start_at:+ and the cell styles) with the XML of each table read in Rust instead of by Ox.
|
|
7
|
+
# The Rust side reads a plain subset of XML and the values calcsheet accepts; on anything else (a comment, CDATA,
|
|
8
|
+
# a reference in an attribute, a value calcsheet refuses, an error of the zip stream...) the table is read again
|
|
9
|
+
# by calcsheet's own parser, which gives its exact rows or error. The rows already yielded are not yielded again,
|
|
10
|
+
# and the inflated total of the container and what the Scanner learned are set back to their state before the
|
|
11
|
+
# first attempt, so that the second one charges and caches exactly as calcsheet does.
|
|
12
|
+
#
|
|
13
|
+
# It depends on the private methods of Calcsheet::Readers::OdsReader and the internals of ZipContainer and
|
|
14
|
+
# Ods::Scanner of the calcsheet minor version it supports.
|
|
15
|
+
class OdsReader < Calcsheet::Readers::OdsReader
|
|
16
|
+
# Thrown when calcsheet's parser must read the table.
|
|
17
|
+
FALLBACK = :calcsheet_native_fallback
|
|
18
|
+
# Rows per call into Rust, and bytes read from the stream at a time.
|
|
19
|
+
ROWS = 512
|
|
20
|
+
CHUNK = 64 * 1024
|
|
21
|
+
# The RowBuilder kinds of OdsTable's cells, and the reasons of a missing value.
|
|
22
|
+
KINDS = %i[text number iso_date duration boolean error missing].freeze
|
|
23
|
+
MISSING = %i[invalid_decimal invalid_date invalid_boolean].freeze
|
|
24
|
+
# The inflated total of ZipContainer.
|
|
25
|
+
TOTAL = :@inflated_total
|
|
26
|
+
|
|
27
|
+
@fallbacks = 0
|
|
28
|
+
|
|
29
|
+
class << self
|
|
30
|
+
# @return [Integer] the table reads handed to calcsheet's parser in this process, for specs and diagnostics
|
|
31
|
+
attr_accessor :fallbacks
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
private
|
|
35
|
+
|
|
36
|
+
def each_row(index, cast, types, start_row, &block)
|
|
37
|
+
charged = @zip.instance_variable_get(TOTAL)
|
|
38
|
+
learned = @scanner.instance_variables.to_h { [ _1, @scanner.instance_variable_get(_1) ] }
|
|
39
|
+
last = 0
|
|
40
|
+
finished = catch(FALLBACK) do
|
|
41
|
+
read_natively(index, cast, types, start_row) do |row|
|
|
42
|
+
last = row.number
|
|
43
|
+
yield row
|
|
44
|
+
end
|
|
45
|
+
true
|
|
46
|
+
end
|
|
47
|
+
return if finished
|
|
48
|
+
|
|
49
|
+
self.class.fallbacks += 1
|
|
50
|
+
@zip.instance_variable_set(TOTAL, charged)
|
|
51
|
+
(@scanner.instance_variables - learned.keys).each { @scanner.remove_instance_variable(_1) }
|
|
52
|
+
learned.each { |name, value| @scanner.instance_variable_set(name, value) }
|
|
53
|
+
super(index, cast, types, start_row) { |row| yield row if row.number > last }
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
# Yields the rows read by OdsTable, as ContentParser yields them; throws FALLBACK when calcsheet's parser must
|
|
57
|
+
# read the table. The errors of the caller's block are its own.
|
|
58
|
+
def read_natively(index, cast, types, start_row)
|
|
59
|
+
guarded { sheet_at(index) }
|
|
60
|
+
io = guarded { @scanner.io_at_sheet(index, row: start_row) } or return
|
|
61
|
+
table = guarded do
|
|
62
|
+
prefixes = @scanner.prefixes
|
|
63
|
+
@formats = @scanner.cell_formats
|
|
64
|
+
OdsTable.new(prefixes.values_at(:table, :office, :text, :calcext), @formats.keys, @max_cell_bytes,
|
|
65
|
+
io.first_row, start_row)
|
|
66
|
+
end
|
|
67
|
+
@format_list = @formats.values
|
|
68
|
+
@column_formats = []
|
|
69
|
+
builder = RowBuilder.new(cast, types, number_errors: :raw)
|
|
70
|
+
# Read into one String: a new one per chunk would leave the whole table as garbage before the GC runs.
|
|
71
|
+
buffer = String.new(capacity: CHUNK)
|
|
72
|
+
loop do
|
|
73
|
+
batch = guarded { table.rows(ROWS) }
|
|
74
|
+
throw FALLBACK if batch == false
|
|
75
|
+
if batch.nil?
|
|
76
|
+
chunk = guarded { io.read(CHUNK, buffer) }
|
|
77
|
+
chunk ? table.feed(chunk) : table.finish
|
|
78
|
+
next
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
firsts, lasts, ends, cells, @column_formats, done = batch
|
|
82
|
+
from = 0
|
|
83
|
+
firsts.each_with_index do |number, position|
|
|
84
|
+
to = ends[position]
|
|
85
|
+
guarded { add_cells(builder, cells, from, to) }
|
|
86
|
+
from = to
|
|
87
|
+
last = lasts[position]
|
|
88
|
+
row = builder.finish(last)
|
|
89
|
+
next if row.nil? || last < start_row
|
|
90
|
+
|
|
91
|
+
[ number, start_row ].max.upto(last - 1) { |each| yield RawRow.new(number: each, values: row.values.dup, errors: row.errors&.dup) }
|
|
92
|
+
yield row
|
|
93
|
+
end
|
|
94
|
+
break if done
|
|
95
|
+
end
|
|
96
|
+
ensure
|
|
97
|
+
io&.close
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# Runs the block, throwing FALLBACK for any error it raises: calcsheet's parser raises it again, where it
|
|
101
|
+
# would.
|
|
102
|
+
def guarded
|
|
103
|
+
yield
|
|
104
|
+
rescue StandardError
|
|
105
|
+
throw FALLBACK
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# Adds the cells +from...to+ of a batch to the builder, as ContentParser#finish_cell does.
|
|
109
|
+
def add_cells(builder, cells, from, to)
|
|
110
|
+
while from < to
|
|
111
|
+
column = cells[from]
|
|
112
|
+
repeat = cells[from + 1]
|
|
113
|
+
kind = KINDS[cells[from + 2]]
|
|
114
|
+
missing = MISSING[cells[from + 3]] if kind == :missing
|
|
115
|
+
raw = cells[from + 4]
|
|
116
|
+
style = cells[from + 5]
|
|
117
|
+
if repeat == 1
|
|
118
|
+
builder.add(column, kind, raw, format_of(kind, style, column), missing: missing)
|
|
119
|
+
elsif kind == :number || kind == :iso_date
|
|
120
|
+
builder.add_run(column, column + repeat - 1, kind, raw, missing: missing) { format_of(kind, style, _1) }
|
|
121
|
+
else
|
|
122
|
+
builder.add_run(column, column + repeat - 1, kind, raw, missing: missing)
|
|
123
|
+
end
|
|
124
|
+
from += 6
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
# ContentParser#format_of: a number takes only the padding of a zero-padding style, a date its style's format.
|
|
129
|
+
def format_of(kind, style, column)
|
|
130
|
+
return unless kind == :number || kind == :iso_date
|
|
131
|
+
|
|
132
|
+
format = format_at(style, column)
|
|
133
|
+
kind == :iso_date || (format && format[0] == :general) ? format : nil
|
|
134
|
+
end
|
|
135
|
+
|
|
136
|
+
# ContentParser#format_at: the format of the cell's or row's style (-1: a style without one), else of the
|
|
137
|
+
# default cell style of its column (-2).
|
|
138
|
+
def format_at(style, column)
|
|
139
|
+
return if @formats.empty?
|
|
140
|
+
return @format_list[style] if style >= 0
|
|
141
|
+
return if style == -1
|
|
142
|
+
|
|
143
|
+
position = (0...@column_formats.size / 3).bsearch { |entry| @column_formats[entry * 3 + 1] >= column }
|
|
144
|
+
@format_list[@column_formats[position * 3 + 2]] if position && @column_formats[position * 3] <= column
|
|
145
|
+
end
|
|
146
|
+
end
|
|
147
|
+
end
|
|
148
|
+
end
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Calcsheet
|
|
4
|
+
module Native
|
|
5
|
+
# The native XLSX backend: calcsheet's Ruby reader (the zip archive and its limits, the workbook, styles and
|
|
6
|
+
# shared strings) with the XML of each sheet read in Rust instead of by Ox. The Rust side reads a plain subset of
|
|
7
|
+
# XML and the values calcsheet accepts; on anything else (a comment, CDATA, a reference in an attribute, a value
|
|
8
|
+
# calcsheet refuses, an error of the zip stream...) the sheet is read again by calcsheet's own parser, which
|
|
9
|
+
# gives its exact rows or error. The rows already yielded are not yielded again, and the bytes the first attempt
|
|
10
|
+
# inflated are not charged twice against +max_uncompressed_size+.
|
|
11
|
+
#
|
|
12
|
+
# A read from a later row (the rows after the header, +start_at:+) parses the rows before it too, where calcsheet
|
|
13
|
+
# passes over them with RowSkipper: what the subset reads, RowSkipper passes over without an error, and anything
|
|
14
|
+
# else is again left to calcsheet.
|
|
15
|
+
#
|
|
16
|
+
# It depends on the private methods of Calcsheet::Readers::XlsxReader and the internals of ZipContainer of the
|
|
17
|
+
# calcsheet minor version it supports.
|
|
18
|
+
class XlsxReader < Calcsheet::Readers::XlsxReader
|
|
19
|
+
# calcsheet's SharedStrings over a StringTable read in Rust, which the sheet readers share: one copy of the
|
|
20
|
+
# strings.
|
|
21
|
+
class SharedStrings < Readers::Xlsx::SharedStrings
|
|
22
|
+
# @return [StringTable]
|
|
23
|
+
attr_reader :native
|
|
24
|
+
|
|
25
|
+
# @param table [StringTable]
|
|
26
|
+
def initialize(table) # rubocop:disable Lint/MissingSuper -- the buffer and offsets of the superclass are in Rust
|
|
27
|
+
@native = table
|
|
28
|
+
@size = table.size
|
|
29
|
+
@bytesize = table.bytesize
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# @see Readers::Xlsx::SharedStrings#unchecked
|
|
33
|
+
def unchecked(index)
|
|
34
|
+
@native.unchecked(index)
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
# Thrown when calcsheet's parser must read the sheet.
|
|
39
|
+
FALLBACK = :calcsheet_native_fallback
|
|
40
|
+
# Rows per call into Rust, and bytes read from the zip stream at a time.
|
|
41
|
+
ROWS = 512
|
|
42
|
+
CHUNK = 64 * 1024
|
|
43
|
+
# The chunks of shared strings between two minor GCs (see #read_shared_strings).
|
|
44
|
+
GC_CHUNKS = 32
|
|
45
|
+
# The kinds of XlsxSheet's cells; the text of a SHARED cell is its shared string.
|
|
46
|
+
NUMBER = 0
|
|
47
|
+
SHARED = 1
|
|
48
|
+
INLINE = 2
|
|
49
|
+
BOOLEAN = 3
|
|
50
|
+
ERROR = 4
|
|
51
|
+
DATE = 5
|
|
52
|
+
# The inflated total of ZipContainer, restored before calcsheet's parser reads the sheet again.
|
|
53
|
+
TOTAL = :@inflated_total
|
|
54
|
+
|
|
55
|
+
@fallbacks = 0
|
|
56
|
+
|
|
57
|
+
class << self
|
|
58
|
+
# @return [Integer] the sheet reads handed to calcsheet's parser in this process, for specs and diagnostics
|
|
59
|
+
attr_accessor :fallbacks
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
private
|
|
63
|
+
|
|
64
|
+
def each_row(index, cast, types, start_row, &block)
|
|
65
|
+
path = @meta.sheet_path(index)
|
|
66
|
+
return super unless @zip.entry?(path)
|
|
67
|
+
|
|
68
|
+
charged = @zip.instance_variable_get(TOTAL)
|
|
69
|
+
strings = @shared_strings
|
|
70
|
+
last = 0
|
|
71
|
+
finished = catch(FALLBACK) do
|
|
72
|
+
read_natively(path, cast, types, start_row) do |row|
|
|
73
|
+
last = row.number
|
|
74
|
+
yield row
|
|
75
|
+
end
|
|
76
|
+
true
|
|
77
|
+
end
|
|
78
|
+
return if finished
|
|
79
|
+
|
|
80
|
+
self.class.fallbacks += 1
|
|
81
|
+
@zip.instance_variable_set(TOTAL, charged)
|
|
82
|
+
@shared_strings = strings
|
|
83
|
+
super(index, cast, types, start_row) { |row| yield row if row.number > last }
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
# Yields the rows read by XlsxSheet; throws FALLBACK when calcsheet's parser must read the sheet. The errors of
|
|
87
|
+
# the caller's block are its own.
|
|
88
|
+
def read_natively(path, cast, types, start_row)
|
|
89
|
+
stream = guarded { @zip.stream(path) }
|
|
90
|
+
sheet = XlsxSheet.new(@max_cell_bytes)
|
|
91
|
+
builder = RowBuilder.new(cast, types, number_errors: :as_read)
|
|
92
|
+
@style_formats = {}
|
|
93
|
+
# Read into one String: a new one per chunk would leave the whole part as garbage before the GC runs.
|
|
94
|
+
buffer = String.new(capacity: CHUNK)
|
|
95
|
+
loop do
|
|
96
|
+
batch = guarded { sheet.rows(ROWS) }
|
|
97
|
+
throw FALLBACK if batch == false
|
|
98
|
+
if batch == :strings
|
|
99
|
+
guarded do
|
|
100
|
+
table = shared_strings
|
|
101
|
+
if table.is_a?(SharedStrings)
|
|
102
|
+
sheet.table(table.native)
|
|
103
|
+
else
|
|
104
|
+
sheet.strings(table.instance_variable_get(:@buffer), table.instance_variable_get(:@offsets))
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
next
|
|
108
|
+
end
|
|
109
|
+
if batch.nil?
|
|
110
|
+
chunk = guarded { stream.read(CHUNK, buffer) }
|
|
111
|
+
chunk ? sheet.feed(chunk) : sheet.finish
|
|
112
|
+
next
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
numbers, ends, cells, done = batch
|
|
116
|
+
from = 0
|
|
117
|
+
numbers.each_with_index do |number, position|
|
|
118
|
+
to = ends[position]
|
|
119
|
+
guarded { add_cells(builder, cells, from, to) }
|
|
120
|
+
from = to
|
|
121
|
+
row = builder.finish(number)
|
|
122
|
+
yield row if row && number >= start_row
|
|
123
|
+
end
|
|
124
|
+
break if done
|
|
125
|
+
end
|
|
126
|
+
ensure
|
|
127
|
+
stream&.close
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
# Runs the block, throwing FALLBACK for any error it raises: calcsheet's parser raises it again, where it
|
|
131
|
+
# would.
|
|
132
|
+
def guarded
|
|
133
|
+
yield
|
|
134
|
+
rescue StandardError
|
|
135
|
+
throw FALLBACK
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
# Adds the cells +from...to+ of a batch to the builder, as SheetParser#finish_cell does.
|
|
139
|
+
def add_cells(builder, cells, from, to)
|
|
140
|
+
while from < to
|
|
141
|
+
column = cells[from]
|
|
142
|
+
style = cells[from + 2]
|
|
143
|
+
text = cells[from + 3]
|
|
144
|
+
text = read_text(text) if cells[from + 4]
|
|
145
|
+
case cells[from + 1]
|
|
146
|
+
when NUMBER then builder.add(column, :number, text, format(style))
|
|
147
|
+
when SHARED then builder.add(column, :text, text)
|
|
148
|
+
when INLINE then builder.add(column, :text, Readers::Xlsx::Escapes.decode!(text))
|
|
149
|
+
when BOOLEAN then builder.add(column, :boolean, text)
|
|
150
|
+
when ERROR then builder.add(column, :error, text)
|
|
151
|
+
when DATE then builder.add(column, :iso_date, text, format(style))
|
|
152
|
+
else builder.add(column, :text, text)
|
|
153
|
+
end
|
|
154
|
+
from += 6
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
# The line ends and references of a text, as SheetParser#text reads them.
|
|
159
|
+
def read_text(text)
|
|
160
|
+
text = Readers::References.line_ends(text) if text.include?("\r")
|
|
161
|
+
text.include?("&") ? Readers::References.decode(text) : text
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
# The +[kind, pad]+ of a number or date cell's style (SheetParser#style_kind): 0 without one.
|
|
165
|
+
def format(style)
|
|
166
|
+
return @meta.style_kind(0) if style.nil?
|
|
167
|
+
|
|
168
|
+
@style_formats[style] ||= begin
|
|
169
|
+
throw FALLBACK unless style.zero? || style.between?(1, @meta.style_count - 1)
|
|
170
|
+
@meta.style_kind(style)
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
# The shared strings read in Rust, as calcsheet's SharedStrings; calcsheet loads them itself when the Rust side
|
|
175
|
+
# does not read them, with the inflated total set back first.
|
|
176
|
+
def load_shared_strings
|
|
177
|
+
return super unless @zip.entry?(SHARED_STRINGS)
|
|
178
|
+
|
|
179
|
+
charged = @zip.instance_variable_get(TOTAL)
|
|
180
|
+
table = catch(FALLBACK) { read_shared_strings }
|
|
181
|
+
return table if table
|
|
182
|
+
|
|
183
|
+
@zip.instance_variable_set(TOTAL, charged)
|
|
184
|
+
super
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
def read_shared_strings
|
|
188
|
+
stream = guarded { @zip.stream(SHARED_STRINGS, ratio_floor: false) }
|
|
189
|
+
reader = XlsxStrings.new(@max_cell_bytes)
|
|
190
|
+
buffer = String.new(capacity: CHUNK)
|
|
191
|
+
chunks = 0
|
|
192
|
+
loop do
|
|
193
|
+
result = guarded { reader.read }
|
|
194
|
+
throw FALLBACK if result == false
|
|
195
|
+
return SharedStrings.new(result) if result
|
|
196
|
+
|
|
197
|
+
chunk = guarded { stream.read(CHUNK, buffer) }
|
|
198
|
+
chunk ? reader.feed(chunk) : reader.finish
|
|
199
|
+
# The inflater leaves its pieces as garbage, and this loop allocates too little for the GC to run: a
|
|
200
|
+
# minor GC every GC_CHUNKS keeps the peak memory at the Ruby backend's.
|
|
201
|
+
GC.start(full_mark: false) if (chunks += 1) % GC_CHUNKS == 0
|
|
202
|
+
end
|
|
203
|
+
ensure
|
|
204
|
+
stream&.close
|
|
205
|
+
end
|
|
206
|
+
end
|
|
207
|
+
end
|
|
208
|
+
end
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "calcsheet"
|
|
4
|
+
require_relative "native/version"
|
|
5
|
+
|
|
6
|
+
begin
|
|
7
|
+
# A precompiled platform gem holds one extension per Ruby minor version; a gem built from source holds one.
|
|
8
|
+
require_relative "native/#{RUBY_VERSION[/\A\d+\.\d+/]}/calcsheet_native"
|
|
9
|
+
rescue LoadError
|
|
10
|
+
require_relative "native/calcsheet_native"
|
|
11
|
+
end
|
|
12
|
+
require_relative "native/csv_reader"
|
|
13
|
+
require_relative "native/ods_reader"
|
|
14
|
+
require_relative "native/xlsx_reader"
|
|
15
|
+
|
|
16
|
+
module Calcsheet
|
|
17
|
+
# The native reader backends of calcsheet, which Calcsheet.open finds through +require "calcsheet/native"+ (see
|
|
18
|
+
# Calcsheet::NativeBackends in calcsheet).
|
|
19
|
+
module Native
|
|
20
|
+
# The version of the interface with calcsheet: Calcsheet::NativeBackends::PROTOCOLS lists those it accepts.
|
|
21
|
+
PROTOCOL = 1
|
|
22
|
+
|
|
23
|
+
# The formats read natively.
|
|
24
|
+
BACKENDS = { csv: CsvReader, xlsx: XlsxReader, ods: OdsReader }.freeze
|
|
25
|
+
|
|
26
|
+
# @return [Hash{Symbol => Class}] the backend class of each format read natively
|
|
27
|
+
def self.backends
|
|
28
|
+
BACKENDS
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: calcsheet-native
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: aarch64-linux
|
|
6
|
+
authors:
|
|
7
|
+
- Demetra Opinioni.net Srl
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2026-10-22 00:00:00.000000000 Z
|
|
12
|
+
dependencies:
|
|
13
|
+
- !ruby/object:Gem::Dependency
|
|
14
|
+
name: calcsheet
|
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
|
16
|
+
requirements:
|
|
17
|
+
- - ">="
|
|
18
|
+
- !ruby/object:Gem::Version
|
|
19
|
+
version: '0.2'
|
|
20
|
+
- - "<"
|
|
21
|
+
- !ruby/object:Gem::Version
|
|
22
|
+
version: '0.4'
|
|
23
|
+
type: :runtime
|
|
24
|
+
prerelease: false
|
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
26
|
+
requirements:
|
|
27
|
+
- - ">="
|
|
28
|
+
- !ruby/object:Gem::Version
|
|
29
|
+
version: '0.2'
|
|
30
|
+
- - "<"
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '0.4'
|
|
33
|
+
description: |
|
|
34
|
+
calcsheet-native reads CSV, XLSX and ODS for calcsheet with a Rust core that runs without the GVL, and gives
|
|
35
|
+
the same rows as calcsheet's Ruby backends.
|
|
36
|
+
email:
|
|
37
|
+
- developers@opinioni.net
|
|
38
|
+
executables: []
|
|
39
|
+
extensions: []
|
|
40
|
+
extra_rdoc_files: []
|
|
41
|
+
files:
|
|
42
|
+
- CHANGELOG.md
|
|
43
|
+
- LICENSE.txt
|
|
44
|
+
- README.md
|
|
45
|
+
- lib/calcsheet/native.rb
|
|
46
|
+
- lib/calcsheet/native/3.3/calcsheet_native.so
|
|
47
|
+
- lib/calcsheet/native/3.4/calcsheet_native.so
|
|
48
|
+
- lib/calcsheet/native/4.0/calcsheet_native.so
|
|
49
|
+
- lib/calcsheet/native/csv_reader.rb
|
|
50
|
+
- lib/calcsheet/native/ods_reader.rb
|
|
51
|
+
- lib/calcsheet/native/version.rb
|
|
52
|
+
- lib/calcsheet/native/xlsx_reader.rb
|
|
53
|
+
homepage: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native
|
|
54
|
+
licenses:
|
|
55
|
+
- MIT
|
|
56
|
+
metadata:
|
|
57
|
+
allowed_push_host: https://rubygems.org
|
|
58
|
+
source_code_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/tree/main
|
|
59
|
+
bug_tracker_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/issues
|
|
60
|
+
changelog_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/blob/main/CHANGELOG.md
|
|
61
|
+
rubygems_mfa_required: 'true'
|
|
62
|
+
post_install_message:
|
|
63
|
+
rdoc_options: []
|
|
64
|
+
require_paths:
|
|
65
|
+
- lib
|
|
66
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
67
|
+
requirements:
|
|
68
|
+
- - ">="
|
|
69
|
+
- !ruby/object:Gem::Version
|
|
70
|
+
version: '3.3'
|
|
71
|
+
- - "<"
|
|
72
|
+
- !ruby/object:Gem::Version
|
|
73
|
+
version: 4.1.dev
|
|
74
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
75
|
+
requirements:
|
|
76
|
+
- - ">="
|
|
77
|
+
- !ruby/object:Gem::Version
|
|
78
|
+
version: '0'
|
|
79
|
+
requirements: []
|
|
80
|
+
rubygems_version: 3.5.23
|
|
81
|
+
signing_key:
|
|
82
|
+
specification_version: 4
|
|
83
|
+
summary: Native reader backends for calcsheet.
|
|
84
|
+
test_files: []
|