calcsheet-native 0.1.0-x86_64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 247329e9d2cc5fdbc61257c10d23f7547c9e7af7db9cb0b38235c42b69489dd5
4
+ data.tar.gz: 39f8cef0f4e6fa9f9708969fd70a600d424c0b4bb7a883bbb5a0074c4a48db59
5
+ SHA512:
6
+ metadata.gz: e56613339236d5dcd2af21bf166f553e9abf70880bd3751ab9aa9edef3557fe7a367987eef57ecba318a395326384b86d54ef41a7ef7ace2826dd1fef62b605b
7
+ data.tar.gz: 2b6ac6a8ad75db78b9c0bd1b832c262237b0b27f75c764c0679e79a80c9efecf675cc5ef23c40839c5764890481cc6e38ae9661c7db625111e082e0264657c9f
data/CHANGELOG.md ADDED
@@ -0,0 +1,36 @@
1
+ # Changelog
2
+
3
+ All notable changes to this gem are documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this gem adheres to
6
+ [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.1.0] - 2026-10-28
11
+
12
+ First release, for calcsheet 0.2 and 0.3 (protocol 1), with precompiled gems for `x86_64-linux`, `aarch64-linux`,
13
+ `x86_64-linux-musl`, `aarch64-linux-musl`, `arm64-darwin`, `x86_64-darwin` and `x64-mingw-ucrt` (Ruby 3.3, 3.4 and
14
+ 4.0) beside the source gem.
15
+
16
+ ### Added
17
+
18
+ - The gem skeleton: a Rust extension built with magnus and rb_sys, `Calcsheet::Native::PROTOCOL` 1 and
19
+ `Calcsheet::Native.backends`.
20
+ - The native CSV backend: a Rust tokenizer that accepts and refuses what the stdlib CSV parser does, run without the
21
+ GVL, under calcsheet's own CSV reader (dialect, encodings, numbering, checks and conversion). About twice as fast
22
+ as the Ruby backend on large files, with the same memory.
23
+ - The native XLSX backend: the XML of each sheet is read in Rust, without the GVL, for a plain subset of XML and the
24
+ values calcsheet accepts; anything else is read again by calcsheet's own parser, so results and errors are always
25
+ the Ruby backend's. The archive, its limits, the workbook, styles and shared strings stay calcsheet's. About 1.7
26
+ times as fast as the Ruby backend, with the same memory.
27
+ - The native ODS backend, built the same way: the XML of each table is read in Rust, the Scanner that finds tables
28
+ and rows, the archive and the cell styles stay calcsheet's. About twice as fast as the Ruby backend, with the same
29
+ memory.
30
+ - XLSX shared strings read in Rust, in one table shared by the sheets, and a faster XML lexer: with YJIT, XLSX
31
+ reads in 8.5 s per million rows instead of 10.3 s, ODS in 12.3 s instead of 13.6 s.
32
+ - An ODS read finds its sheet without listing every sheet (calcsheet's ADR 013): with YJIT, 9.1 s per million rows
33
+ instead of 12.3 s, and about 40 MB of peak memory instead of 100.
34
+
35
+ [Unreleased]: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/compare/v0.1.0...develop
36
+ [0.1.0]: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/tags/v0.1.0
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2026 Demetra Opinioni.net Srl
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,65 @@
1
+ # calcsheet-native
2
+
3
+ Native reader backends for [calcsheet](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet): a Rust
4
+ core reads CSV, XLSX and ODS (inflating, tokenizing, limits) without the GVL and hands raw cells to calcsheet, which
5
+ converts them exactly as its Ruby backends do. With the same file and the same options both backends must give the
6
+ same rows, cell errors and exceptions; the only allowed differences are listed in
7
+ [docs/parity-exceptions.md](docs/parity-exceptions.md).
8
+
9
+ ## Status
10
+
11
+ Released on rubygems.org. CSV, XLSX and ODS are read natively. Precompiled gems exist for `x86_64-linux`,
12
+ `aarch64-linux`, `x86_64-linux-musl`, `aarch64-linux-musl`, `arm64-darwin`, `x86_64-darwin` and `x64-mingw-ucrt`, each
13
+ for Ruby 3.3, 3.4 and 4.0; any other platform installs the source gem, which needs the Rust toolchain and libclang
14
+ (see Development).
15
+
16
+ Seconds per million rows of 11 columns (Ruby 4.0, x86_64; XLSX and ODS measured on 100,000 rows), end to end
17
+ with the values converted; the peak memory is the same with both backends:
18
+
19
+ | File | Ruby | Ruby + YJIT | Native | Native + YJIT |
20
+ |---|---|---|---|---|
21
+ | CSV | 7.5 s | 3.9 s | 3.7 s | 2.0 s |
22
+ | XLSX | 27.8 s | 16.9 s | 12.0 s | 8.5 s |
23
+ | ODS | 35.7 s | 17.7 s | 14.8 s | 9.1 s |
24
+
25
+ With YJIT, the conversion of the cells in Ruby (calcsheet's ADR 011) is 30 to 45% of the native time. Two reads in
26
+ two threads take 1.7 to 2 times as long as one.
27
+
28
+ ## Usage
29
+
30
+ Add it next to calcsheet; nothing else changes in the application:
31
+
32
+ ```ruby
33
+ gem "calcsheet", "~> 0.3"
34
+ gem "calcsheet-native"
35
+ ```
36
+
37
+ It needs calcsheet 0.2 or 0.3 (protocol 1). calcsheet uses it under `backend: :auto` (the default) for the formats it reads, and falls back to the Ruby backends,
38
+ with one warning, when it is installed but unusable.
39
+
40
+ ## Development
41
+
42
+ Requirements: Ruby 3.3 or later, the Rust toolchain of `rust-toolchain.toml` (rustup installs it) and libclang
43
+ (Debian: `libclang-dev`), which rb_sys uses to generate the Ruby bindings.
44
+
45
+ ```sh
46
+ bundle install
47
+ bundle exec rake compile
48
+ bundle exec rake # compile, specs, calcsheet's suite on the native backends, RuboCop
49
+ cargo fmt --check && cargo clippy --all-targets -- -D warnings
50
+ ```
51
+
52
+ The Gemfile takes calcsheet from the tag of its release, whose suite `rake parity` runs (the published gem leaves
53
+ it out).
54
+
55
+ A release builds the precompiled gems and the source gem into `pkg/` with Docker, from a checkout of its tag, then
56
+ pushes each of them (rubygems.org asks for the security key of every push):
57
+
58
+ ```sh
59
+ bin/build-native-gems
60
+ for gem in pkg/*.gem; do gem push "$gem"; done
61
+ ```
62
+
63
+ ## License
64
+
65
+ MIT, see [LICENSE.txt](LICENSE.txt).
@@ -0,0 +1,122 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "csv"
4
+
5
+ module Calcsheet
6
+ module Native
7
+ # The native CSV backend: calcsheet's Ruby reader (dialect detection, encodings, numbering, checks and the
8
+ # conversion of the fields) with the records tokenized in Rust instead of by the stdlib CSV parser. A dialect the
9
+ # tokenizer does not handle (a separator or quote that is not one ASCII character, or a line ending the sample
10
+ # did not show) is parsed by the stdlib, as the Ruby backend does.
11
+ #
12
+ # It depends on the private methods of Calcsheet::Readers::CsvReader of the calcsheet minor version it supports.
13
+ class CsvReader < Calcsheet::Readers::CsvReader
14
+ # The line endings the tokenizer handles.
15
+ ROW_SEPARATORS = [ "\n", "\r", "\r\n" ].freeze
16
+
17
+ # The stdlib CSV parser's place for the reader: CsvReader#shift pulls records with +each+ and +next+, and
18
+ # reads +row_sep+ when a record is malformed.
19
+ class Records
20
+ # Records parsed per call into Rust.
21
+ BATCH = 1024
22
+ EMPTY = [].freeze
23
+
24
+ # @return [String] the line ending
25
+ attr_reader :row_sep
26
+ # @return [Integer] the physical lines the last record ends
27
+ attr_reader :breaks
28
+ # @return [Boolean] whether the last record has the stray line break outside quoted fields
29
+ attr_reader :stray
30
+
31
+ # @param input [Calcsheet::Readers::CsvInput]
32
+ # @param dialect [Calcsheet::Readers::CsvDialect::Dialect]
33
+ def initialize(input, dialect)
34
+ @input = input
35
+ @row_sep = dialect.row_sep
36
+ @tokenizer = CsvTokenizer.new(dialect.col_sep, dialect.quote_char, dialect.row_sep)
37
+ @fields = @all_breaks = @strays = EMPTY
38
+ @index = 0
39
+ @breaks = 0
40
+ @stray = false
41
+ end
42
+
43
+ # @return [Records] self: #next gives the records
44
+ def each
45
+ self
46
+ end
47
+
48
+ # @return [Array<String, nil>] the fields of the next record
49
+ # @raise [StopIteration] at the end
50
+ # @raise [CSV::MalformedCSVError] for a malformed record
51
+ # @raise [EncodingError] from the input, at the record that holds the invalid bytes
52
+ def next
53
+ if @index == @fields.size
54
+ @fields, @all_breaks, @strays = batch
55
+ @index = 0
56
+ raise StopIteration if @fields.empty?
57
+ end
58
+ index = @index
59
+ @index += 1
60
+ @breaks = @all_breaks[index]
61
+ @stray = @strays[index]
62
+ @fields[index]
63
+ end
64
+
65
+ private
66
+
67
+ # The next records, reading the input only when the text read ends inside a record.
68
+ def batch
69
+ loop do
70
+ records = @tokenizer.records(BATCH)
71
+ return records if records
72
+ raise CSV::MalformedCSVError.new(*@tokenizer.error) if records == false
73
+
74
+ chunk = @input.gets(nil, Calcsheet::Readers::CsvInput::CHUNK_SIZE)
75
+ chunk ? @tokenizer.feed(chunk) : @tokenizer.finish
76
+ end
77
+ end
78
+ end
79
+
80
+ private
81
+
82
+ # Parses with the tokenizer when it handles the dialect.
83
+ def restart(forced)
84
+ super
85
+ @csv = Records.new(@input, @dialect) if tokenizes?(@dialect)
86
+ end
87
+
88
+ # The text the parser reads, kept for the tokenizer.
89
+ def input
90
+ @input = super
91
+ end
92
+
93
+ def tokenizes?(dialect)
94
+ ROW_SEPARATORS.include?(dialect.row_sep) && [ dialect.col_sep, dialect.quote_char ].all? { _1.bytesize == 1 }
95
+ end
96
+
97
+ # CsvReader#each_row, with the line breaks and the stray line break of each record from the tokenizer instead
98
+ # of its raw text.
99
+ def each_row(sheet_index, cast, types, start, &block)
100
+ forced = forced_dialect(cast.options)
101
+ restart(forced) if @csv.nil? || start <= @records || forced != @forced
102
+ return super unless @csv.is_a?(Records)
103
+
104
+ strip = cast.options.strip_formula_prefix
105
+ builder = RowBuilder.new(cast, types)
106
+ while (fields = shift)
107
+ number = @records += 1
108
+ line = @line
109
+ @line += @csv.breaks
110
+ mixed_line_endings(number, line, @csv.row_sep) if @csv.stray
111
+ next if number < start
112
+
113
+ last = fields.size - 1
114
+ last -= 1 while last >= 0 && blank_field?(fields[last], number, last)
115
+ next if last.negative?
116
+
117
+ yield raw_row(builder, number, line, fields, last, strip)
118
+ end
119
+ end
120
+ end
121
+ end
122
+ end
@@ -0,0 +1,148 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Calcsheet
4
+ module Native
5
+ # The native ODS backend: calcsheet's Ruby reader (the zip archive and its limits, the Scanner that finds the
6
+ # tables, the rows of +start_at:+ and the cell styles) with the XML of each table read in Rust instead of by Ox.
7
+ # The Rust side reads a plain subset of XML and the values calcsheet accepts; on anything else (a comment, CDATA,
8
+ # a reference in an attribute, a value calcsheet refuses, an error of the zip stream...) the table is read again
9
+ # by calcsheet's own parser, which gives its exact rows or error. The rows already yielded are not yielded again,
10
+ # and the inflated total of the container and what the Scanner learned are set back to their state before the
11
+ # first attempt, so that the second one charges and caches exactly as calcsheet does.
12
+ #
13
+ # It depends on the private methods of Calcsheet::Readers::OdsReader and the internals of ZipContainer and
14
+ # Ods::Scanner of the calcsheet minor version it supports.
15
+ class OdsReader < Calcsheet::Readers::OdsReader
16
+ # Thrown when calcsheet's parser must read the table.
17
+ FALLBACK = :calcsheet_native_fallback
18
+ # Rows per call into Rust, and bytes read from the stream at a time.
19
+ ROWS = 512
20
+ CHUNK = 64 * 1024
21
+ # The RowBuilder kinds of OdsTable's cells, and the reasons of a missing value.
22
+ KINDS = %i[text number iso_date duration boolean error missing].freeze
23
+ MISSING = %i[invalid_decimal invalid_date invalid_boolean].freeze
24
+ # The inflated total of ZipContainer.
25
+ TOTAL = :@inflated_total
26
+
27
+ @fallbacks = 0
28
+
29
+ class << self
30
+ # @return [Integer] the table reads handed to calcsheet's parser in this process, for specs and diagnostics
31
+ attr_accessor :fallbacks
32
+ end
33
+
34
+ private
35
+
36
+ def each_row(index, cast, types, start_row, &block)
37
+ charged = @zip.instance_variable_get(TOTAL)
38
+ learned = @scanner.instance_variables.to_h { [ _1, @scanner.instance_variable_get(_1) ] }
39
+ last = 0
40
+ finished = catch(FALLBACK) do
41
+ read_natively(index, cast, types, start_row) do |row|
42
+ last = row.number
43
+ yield row
44
+ end
45
+ true
46
+ end
47
+ return if finished
48
+
49
+ self.class.fallbacks += 1
50
+ @zip.instance_variable_set(TOTAL, charged)
51
+ (@scanner.instance_variables - learned.keys).each { @scanner.remove_instance_variable(_1) }
52
+ learned.each { |name, value| @scanner.instance_variable_set(name, value) }
53
+ super(index, cast, types, start_row) { |row| yield row if row.number > last }
54
+ end
55
+
56
+ # Yields the rows read by OdsTable, as ContentParser yields them; throws FALLBACK when calcsheet's parser must
57
+ # read the table. The errors of the caller's block are its own.
58
+ def read_natively(index, cast, types, start_row)
59
+ guarded { sheet_at(index) }
60
+ io = guarded { @scanner.io_at_sheet(index, row: start_row) } or return
61
+ table = guarded do
62
+ prefixes = @scanner.prefixes
63
+ @formats = @scanner.cell_formats
64
+ OdsTable.new(prefixes.values_at(:table, :office, :text, :calcext), @formats.keys, @max_cell_bytes,
65
+ io.first_row, start_row)
66
+ end
67
+ @format_list = @formats.values
68
+ @column_formats = []
69
+ builder = RowBuilder.new(cast, types, number_errors: :raw)
70
+ # Read into one String: a new one per chunk would leave the whole table as garbage before the GC runs.
71
+ buffer = String.new(capacity: CHUNK)
72
+ loop do
73
+ batch = guarded { table.rows(ROWS) }
74
+ throw FALLBACK if batch == false
75
+ if batch.nil?
76
+ chunk = guarded { io.read(CHUNK, buffer) }
77
+ chunk ? table.feed(chunk) : table.finish
78
+ next
79
+ end
80
+
81
+ firsts, lasts, ends, cells, @column_formats, done = batch
82
+ from = 0
83
+ firsts.each_with_index do |number, position|
84
+ to = ends[position]
85
+ guarded { add_cells(builder, cells, from, to) }
86
+ from = to
87
+ last = lasts[position]
88
+ row = builder.finish(last)
89
+ next if row.nil? || last < start_row
90
+
91
+ [ number, start_row ].max.upto(last - 1) { |each| yield RawRow.new(number: each, values: row.values.dup, errors: row.errors&.dup) }
92
+ yield row
93
+ end
94
+ break if done
95
+ end
96
+ ensure
97
+ io&.close
98
+ end
99
+
100
+ # Runs the block, throwing FALLBACK for any error it raises: calcsheet's parser raises it again, where it
101
+ # would.
102
+ def guarded
103
+ yield
104
+ rescue StandardError
105
+ throw FALLBACK
106
+ end
107
+
108
+ # Adds the cells +from...to+ of a batch to the builder, as ContentParser#finish_cell does.
109
+ def add_cells(builder, cells, from, to)
110
+ while from < to
111
+ column = cells[from]
112
+ repeat = cells[from + 1]
113
+ kind = KINDS[cells[from + 2]]
114
+ missing = MISSING[cells[from + 3]] if kind == :missing
115
+ raw = cells[from + 4]
116
+ style = cells[from + 5]
117
+ if repeat == 1
118
+ builder.add(column, kind, raw, format_of(kind, style, column), missing: missing)
119
+ elsif kind == :number || kind == :iso_date
120
+ builder.add_run(column, column + repeat - 1, kind, raw, missing: missing) { format_of(kind, style, _1) }
121
+ else
122
+ builder.add_run(column, column + repeat - 1, kind, raw, missing: missing)
123
+ end
124
+ from += 6
125
+ end
126
+ end
127
+
128
+ # ContentParser#format_of: a number takes only the padding of a zero-padding style, a date its style's format.
129
+ def format_of(kind, style, column)
130
+ return unless kind == :number || kind == :iso_date
131
+
132
+ format = format_at(style, column)
133
+ kind == :iso_date || (format && format[0] == :general) ? format : nil
134
+ end
135
+
136
+ # ContentParser#format_at: the format of the cell's or row's style (-1: a style without one), else of the
137
+ # default cell style of its column (-2).
138
+ def format_at(style, column)
139
+ return if @formats.empty?
140
+ return @format_list[style] if style >= 0
141
+ return if style == -1
142
+
143
+ position = (0...@column_formats.size / 3).bsearch { |entry| @column_formats[entry * 3 + 1] >= column }
144
+ @format_list[@column_formats[position * 3 + 2]] if position && @column_formats[position * 3] <= column
145
+ end
146
+ end
147
+ end
148
+ end
@@ -0,0 +1,8 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Calcsheet
4
+ module Native
5
+ # The version of the gem; the Rust core (Calcsheet::Native.core_version) has the same.
6
+ VERSION = "0.1.0"
7
+ end
8
+ end
@@ -0,0 +1,208 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Calcsheet
4
+ module Native
5
+ # The native XLSX backend: calcsheet's Ruby reader (the zip archive and its limits, the workbook, styles and
6
+ # shared strings) with the XML of each sheet read in Rust instead of by Ox. The Rust side reads a plain subset of
7
+ # XML and the values calcsheet accepts; on anything else (a comment, CDATA, a reference in an attribute, a value
8
+ # calcsheet refuses, an error of the zip stream...) the sheet is read again by calcsheet's own parser, which
9
+ # gives its exact rows or error. The rows already yielded are not yielded again, and the bytes the first attempt
10
+ # inflated are not charged twice against +max_uncompressed_size+.
11
+ #
12
+ # A read from a later row (the rows after the header, +start_at:+) parses the rows before it too, where calcsheet
13
+ # passes over them with RowSkipper: what the subset reads, RowSkipper passes over without an error, and anything
14
+ # else is again left to calcsheet.
15
+ #
16
+ # It depends on the private methods of Calcsheet::Readers::XlsxReader and the internals of ZipContainer of the
17
+ # calcsheet minor version it supports.
18
+ class XlsxReader < Calcsheet::Readers::XlsxReader
19
+ # calcsheet's SharedStrings over a StringTable read in Rust, which the sheet readers share: one copy of the
20
+ # strings.
21
+ class SharedStrings < Readers::Xlsx::SharedStrings
22
+ # @return [StringTable]
23
+ attr_reader :native
24
+
25
+ # @param table [StringTable]
26
+ def initialize(table) # rubocop:disable Lint/MissingSuper -- the buffer and offsets of the superclass are in Rust
27
+ @native = table
28
+ @size = table.size
29
+ @bytesize = table.bytesize
30
+ end
31
+
32
+ # @see Readers::Xlsx::SharedStrings#unchecked
33
+ def unchecked(index)
34
+ @native.unchecked(index)
35
+ end
36
+ end
37
+
38
+ # Thrown when calcsheet's parser must read the sheet.
39
+ FALLBACK = :calcsheet_native_fallback
40
+ # Rows per call into Rust, and bytes read from the zip stream at a time.
41
+ ROWS = 512
42
+ CHUNK = 64 * 1024
43
+ # The chunks of shared strings between two minor GCs (see #read_shared_strings).
44
+ GC_CHUNKS = 32
45
+ # The kinds of XlsxSheet's cells; the text of a SHARED cell is its shared string.
46
+ NUMBER = 0
47
+ SHARED = 1
48
+ INLINE = 2
49
+ BOOLEAN = 3
50
+ ERROR = 4
51
+ DATE = 5
52
+ # The inflated total of ZipContainer, restored before calcsheet's parser reads the sheet again.
53
+ TOTAL = :@inflated_total
54
+
55
+ @fallbacks = 0
56
+
57
+ class << self
58
+ # @return [Integer] the sheet reads handed to calcsheet's parser in this process, for specs and diagnostics
59
+ attr_accessor :fallbacks
60
+ end
61
+
62
+ private
63
+
64
+ def each_row(index, cast, types, start_row, &block)
65
+ path = @meta.sheet_path(index)
66
+ return super unless @zip.entry?(path)
67
+
68
+ charged = @zip.instance_variable_get(TOTAL)
69
+ strings = @shared_strings
70
+ last = 0
71
+ finished = catch(FALLBACK) do
72
+ read_natively(path, cast, types, start_row) do |row|
73
+ last = row.number
74
+ yield row
75
+ end
76
+ true
77
+ end
78
+ return if finished
79
+
80
+ self.class.fallbacks += 1
81
+ @zip.instance_variable_set(TOTAL, charged)
82
+ @shared_strings = strings
83
+ super(index, cast, types, start_row) { |row| yield row if row.number > last }
84
+ end
85
+
86
+ # Yields the rows read by XlsxSheet; throws FALLBACK when calcsheet's parser must read the sheet. The errors of
87
+ # the caller's block are its own.
88
+ def read_natively(path, cast, types, start_row)
89
+ stream = guarded { @zip.stream(path) }
90
+ sheet = XlsxSheet.new(@max_cell_bytes)
91
+ builder = RowBuilder.new(cast, types, number_errors: :as_read)
92
+ @style_formats = {}
93
+ # Read into one String: a new one per chunk would leave the whole part as garbage before the GC runs.
94
+ buffer = String.new(capacity: CHUNK)
95
+ loop do
96
+ batch = guarded { sheet.rows(ROWS) }
97
+ throw FALLBACK if batch == false
98
+ if batch == :strings
99
+ guarded do
100
+ table = shared_strings
101
+ if table.is_a?(SharedStrings)
102
+ sheet.table(table.native)
103
+ else
104
+ sheet.strings(table.instance_variable_get(:@buffer), table.instance_variable_get(:@offsets))
105
+ end
106
+ end
107
+ next
108
+ end
109
+ if batch.nil?
110
+ chunk = guarded { stream.read(CHUNK, buffer) }
111
+ chunk ? sheet.feed(chunk) : sheet.finish
112
+ next
113
+ end
114
+
115
+ numbers, ends, cells, done = batch
116
+ from = 0
117
+ numbers.each_with_index do |number, position|
118
+ to = ends[position]
119
+ guarded { add_cells(builder, cells, from, to) }
120
+ from = to
121
+ row = builder.finish(number)
122
+ yield row if row && number >= start_row
123
+ end
124
+ break if done
125
+ end
126
+ ensure
127
+ stream&.close
128
+ end
129
+
130
+ # Runs the block, throwing FALLBACK for any error it raises: calcsheet's parser raises it again, where it
131
+ # would.
132
+ def guarded
133
+ yield
134
+ rescue StandardError
135
+ throw FALLBACK
136
+ end
137
+
138
+ # Adds the cells +from...to+ of a batch to the builder, as SheetParser#finish_cell does.
139
+ def add_cells(builder, cells, from, to)
140
+ while from < to
141
+ column = cells[from]
142
+ style = cells[from + 2]
143
+ text = cells[from + 3]
144
+ text = read_text(text) if cells[from + 4]
145
+ case cells[from + 1]
146
+ when NUMBER then builder.add(column, :number, text, format(style))
147
+ when SHARED then builder.add(column, :text, text)
148
+ when INLINE then builder.add(column, :text, Readers::Xlsx::Escapes.decode!(text))
149
+ when BOOLEAN then builder.add(column, :boolean, text)
150
+ when ERROR then builder.add(column, :error, text)
151
+ when DATE then builder.add(column, :iso_date, text, format(style))
152
+ else builder.add(column, :text, text)
153
+ end
154
+ from += 6
155
+ end
156
+ end
157
+
158
+ # The line ends and references of a text, as SheetParser#text reads them.
159
+ def read_text(text)
160
+ text = Readers::References.line_ends(text) if text.include?("\r")
161
+ text.include?("&") ? Readers::References.decode(text) : text
162
+ end
163
+
164
+ # The +[kind, pad]+ of a number or date cell's style (SheetParser#style_kind): 0 without one.
165
+ def format(style)
166
+ return @meta.style_kind(0) if style.nil?
167
+
168
+ @style_formats[style] ||= begin
169
+ throw FALLBACK unless style.zero? || style.between?(1, @meta.style_count - 1)
170
+ @meta.style_kind(style)
171
+ end
172
+ end
173
+
174
+ # The shared strings read in Rust, as calcsheet's SharedStrings; calcsheet loads them itself when the Rust side
175
+ # does not read them, with the inflated total set back first.
176
+ def load_shared_strings
177
+ return super unless @zip.entry?(SHARED_STRINGS)
178
+
179
+ charged = @zip.instance_variable_get(TOTAL)
180
+ table = catch(FALLBACK) { read_shared_strings }
181
+ return table if table
182
+
183
+ @zip.instance_variable_set(TOTAL, charged)
184
+ super
185
+ end
186
+
187
+ def read_shared_strings
188
+ stream = guarded { @zip.stream(SHARED_STRINGS, ratio_floor: false) }
189
+ reader = XlsxStrings.new(@max_cell_bytes)
190
+ buffer = String.new(capacity: CHUNK)
191
+ chunks = 0
192
+ loop do
193
+ result = guarded { reader.read }
194
+ throw FALLBACK if result == false
195
+ return SharedStrings.new(result) if result
196
+
197
+ chunk = guarded { stream.read(CHUNK, buffer) }
198
+ chunk ? reader.feed(chunk) : reader.finish
199
+ # The inflater leaves its pieces as garbage, and this loop allocates too little for the GC to run: a
200
+ # minor GC every GC_CHUNKS keeps the peak memory at the Ruby backend's.
201
+ GC.start(full_mark: false) if (chunks += 1) % GC_CHUNKS == 0
202
+ end
203
+ ensure
204
+ stream&.close
205
+ end
206
+ end
207
+ end
208
+ end
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "calcsheet"
4
+ require_relative "native/version"
5
+
6
+ begin
7
+ # A precompiled platform gem holds one extension per Ruby minor version; a gem built from source holds one.
8
+ require_relative "native/#{RUBY_VERSION[/\A\d+\.\d+/]}/calcsheet_native"
9
+ rescue LoadError
10
+ require_relative "native/calcsheet_native"
11
+ end
12
+ require_relative "native/csv_reader"
13
+ require_relative "native/ods_reader"
14
+ require_relative "native/xlsx_reader"
15
+
16
+ module Calcsheet
17
+ # The native reader backends of calcsheet, which Calcsheet.open finds through +require "calcsheet/native"+ (see
18
+ # Calcsheet::NativeBackends in calcsheet).
19
+ module Native
20
+ # The version of the interface with calcsheet: Calcsheet::NativeBackends::PROTOCOLS lists those it accepts.
21
+ PROTOCOL = 1
22
+
23
+ # The formats read natively.
24
+ BACKENDS = { csv: CsvReader, xlsx: XlsxReader, ods: OdsReader }.freeze
25
+
26
+ # @return [Hash{Symbol => Class}] the backend class of each format read natively
27
+ def self.backends
28
+ BACKENDS
29
+ end
30
+ end
31
+ end
metadata ADDED
@@ -0,0 +1,84 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: calcsheet-native
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: x86_64-linux
6
+ authors:
7
+ - Demetra Opinioni.net Srl
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2026-10-22 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: calcsheet
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0.2'
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '0.4'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: '0.2'
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '0.4'
33
+ description: |
34
+ calcsheet-native reads CSV, XLSX and ODS for calcsheet with a Rust core that runs without the GVL, and gives
35
+ the same rows as calcsheet's Ruby backends.
36
+ email:
37
+ - developers@opinioni.net
38
+ executables: []
39
+ extensions: []
40
+ extra_rdoc_files: []
41
+ files:
42
+ - CHANGELOG.md
43
+ - LICENSE.txt
44
+ - README.md
45
+ - lib/calcsheet/native.rb
46
+ - lib/calcsheet/native/3.3/calcsheet_native.so
47
+ - lib/calcsheet/native/3.4/calcsheet_native.so
48
+ - lib/calcsheet/native/4.0/calcsheet_native.so
49
+ - lib/calcsheet/native/csv_reader.rb
50
+ - lib/calcsheet/native/ods_reader.rb
51
+ - lib/calcsheet/native/version.rb
52
+ - lib/calcsheet/native/xlsx_reader.rb
53
+ homepage: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native
54
+ licenses:
55
+ - MIT
56
+ metadata:
57
+ allowed_push_host: https://rubygems.org
58
+ source_code_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/tree/main
59
+ bug_tracker_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/issues
60
+ changelog_uri: https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet-native/-/blob/main/CHANGELOG.md
61
+ rubygems_mfa_required: 'true'
62
+ post_install_message:
63
+ rdoc_options: []
64
+ require_paths:
65
+ - lib
66
+ required_ruby_version: !ruby/object:Gem::Requirement
67
+ requirements:
68
+ - - ">="
69
+ - !ruby/object:Gem::Version
70
+ version: '3.3'
71
+ - - "<"
72
+ - !ruby/object:Gem::Version
73
+ version: 4.1.dev
74
+ required_rubygems_version: !ruby/object:Gem::Requirement
75
+ requirements:
76
+ - - ">="
77
+ - !ruby/object:Gem::Version
78
+ version: '0'
79
+ requirements: []
80
+ rubygems_version: 3.5.23
81
+ signing_key:
82
+ specification_version: 4
83
+ summary: Native reader backends for calcsheet.
84
+ test_files: []