calcsheet 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.yardopts +9 -0
- data/CHANGELOG.md +62 -0
- data/LICENSE.txt +21 -0
- data/README.md +921 -0
- data/lib/calcsheet/book.rb +164 -0
- data/lib/calcsheet/cast.rb +578 -0
- data/lib/calcsheet/column.rb +117 -0
- data/lib/calcsheet/column_letter.rb +34 -0
- data/lib/calcsheet/configuration.rb +186 -0
- data/lib/calcsheet/decimal_text.rb +19 -0
- data/lib/calcsheet/entry_name.rb +28 -0
- data/lib/calcsheet/errors.rb +186 -0
- data/lib/calcsheet/excel_serial.rb +65 -0
- data/lib/calcsheet/format_detector.rb +109 -0
- data/lib/calcsheet/formula_protection.rb +31 -0
- data/lib/calcsheet/grid.rb +20 -0
- data/lib/calcsheet/header_keys.rb +66 -0
- data/lib/calcsheet/limited_inflater.rb +326 -0
- data/lib/calcsheet/merge_check.rb +41 -0
- data/lib/calcsheet/odf.rb +45 -0
- data/lib/calcsheet/option_validation.rb +160 -0
- data/lib/calcsheet/options.rb +10 -0
- data/lib/calcsheet/raw_row.rb +18 -0
- data/lib/calcsheet/raw_types.rb +15 -0
- data/lib/calcsheet/readers/csv_dialect.rb +264 -0
- data/lib/calcsheet/readers/csv_input.rb +128 -0
- data/lib/calcsheet/readers/csv_reader.rb +239 -0
- data/lib/calcsheet/readers/ods/cell_formats.rb +115 -0
- data/lib/calcsheet/readers/ods/content_parser.rb +594 -0
- data/lib/calcsheet/readers/ods/count.rb +30 -0
- data/lib/calcsheet/readers/ods/cursor.rb +121 -0
- data/lib/calcsheet/readers/ods/patterns.rb +70 -0
- data/lib/calcsheet/readers/ods/row_seek.rb +157 -0
- data/lib/calcsheet/readers/ods/scanner.rb +302 -0
- data/lib/calcsheet/readers/ods/style_scan.rb +108 -0
- data/lib/calcsheet/readers/ods/tag_text.rb +76 -0
- data/lib/calcsheet/readers/ods_reader.rb +71 -0
- data/lib/calcsheet/readers/ox_errors.rb +40 -0
- data/lib/calcsheet/readers/references.rb +62 -0
- data/lib/calcsheet/readers/row_batches.rb +45 -0
- data/lib/calcsheet/readers/sax_sheet.rb +108 -0
- data/lib/calcsheet/readers/stream_remainder.rb +66 -0
- data/lib/calcsheet/readers/tag_scan.rb +119 -0
- data/lib/calcsheet/readers/xlsx/escapes.rb +36 -0
- data/lib/calcsheet/readers/xlsx/local_name.rb +54 -0
- data/lib/calcsheet/readers/xlsx/number_format.rb +89 -0
- data/lib/calcsheet/readers/xlsx/row_skipper.rb +162 -0
- data/lib/calcsheet/readers/xlsx/shared_strings.rb +209 -0
- data/lib/calcsheet/readers/xlsx/sheet_parser.rb +435 -0
- data/lib/calcsheet/readers/xlsx/styles.rb +156 -0
- data/lib/calcsheet/readers/xlsx/workbook_meta.rb +228 -0
- data/lib/calcsheet/readers/xlsx_reader.rb +122 -0
- data/lib/calcsheet/reading.rb +281 -0
- data/lib/calcsheet/row.rb +110 -0
- data/lib/calcsheet/sample.rb +10 -0
- data/lib/calcsheet/schema.rb +140 -0
- data/lib/calcsheet/sheet_info.rb +21 -0
- data/lib/calcsheet/sheet_names.rb +114 -0
- data/lib/calcsheet/sheet_selection.rb +58 -0
- data/lib/calcsheet/source.rb +263 -0
- data/lib/calcsheet/stream_body.rb +81 -0
- data/lib/calcsheet/time_zone.rb +96 -0
- data/lib/calcsheet/untrusted.rb +32 -0
- data/lib/calcsheet/version.rb +6 -0
- data/lib/calcsheet/write_book.rb +202 -0
- data/lib/calcsheet/write_sheet.rb +282 -0
- data/lib/calcsheet/writers/csv_writer.rb +138 -0
- data/lib/calcsheet/writers/ods_writer.rb +437 -0
- data/lib/calcsheet/writers/xlsx_writer.rb +350 -0
- data/lib/calcsheet/writers/xml_text.rb +35 -0
- data/lib/calcsheet/writers/zip_buffered.rb +55 -0
- data/lib/calcsheet/zip_container.rb +190 -0
- data/lib/calcsheet/zip_directory.rb +202 -0
- data/lib/calcsheet/zip_end_record.rb +117 -0
- data/lib/calcsheet.rb +295 -0
- metadata +200 -0
data/README.md
ADDED
|
@@ -0,0 +1,921 @@
|
|
|
1
|
+
# Calcsheet
|
|
2
|
+
|
|
3
|
+
Streaming CSV, XLSX and ODS reader and writer for Ruby, with typed cells and no dependency on Rails.
|
|
4
|
+
|
|
5
|
+
Calcsheet reads and writes spreadsheets one row at a time, so a file of a million rows costs tens of megabytes of
|
|
6
|
+
memory, not gigabytes. It is built for imports and exports of large lists:
|
|
7
|
+
|
|
8
|
+
- **Exact values.** Decimals are `BigDecimal` built from the cell text, integers are `Integer`, dates are `Date`,
|
|
9
|
+
datetimes are `Time` with the wall clock of the file. Text is never turned into numbers: `"+393331234567"` and
|
|
10
|
+
`"00013"` survive every round trip.
|
|
11
|
+
- **Row numbers the user sees.** Error reports say "sheet 'Contacts (2)', row 4812, column C", exactly where Excel
|
|
12
|
+
or LibreOffice shows the cell.
|
|
13
|
+
- **A two-phase import flow.** Sample the first rows, let the user map the columns, then process the file in
|
|
14
|
+
batches, collecting cell errors instead of stopping at the first one.
|
|
15
|
+
- **Streaming output.** Write to a file, to any IO (sockets and other non-seekable ones included), or straight into
|
|
16
|
+
an HTTP response as a Rack body. Sheets longer than the format allows continue in "Name (2)", "Name (3)"...
|
|
17
|
+
- **Defensive limits** for untrusted uploads: file size, rows, zip bombs, entries and sheets.
|
|
18
|
+
|
|
19
|
+
## Contents
|
|
20
|
+
|
|
21
|
+
- [Installation](#installation)
|
|
22
|
+
- [Reading](#reading)
|
|
23
|
+
- [Writing](#writing)
|
|
24
|
+
- [Types](#types)
|
|
25
|
+
- [Row numbers](#row-numbers)
|
|
26
|
+
- [Errors](#errors)
|
|
27
|
+
- [Limits](#limits)
|
|
28
|
+
- [CSV](#csv)
|
|
29
|
+
- [XLSX and ODS](#xlsx-and-ods)
|
|
30
|
+
- [Streams and non-seekable input](#streams-and-non-seekable-input)
|
|
31
|
+
- [Time zones](#time-zones)
|
|
32
|
+
- [Configuration](#configuration)
|
|
33
|
+
- [Options reference](#options-reference)
|
|
34
|
+
- [Concurrency](#concurrency)
|
|
35
|
+
- [Performance](#performance)
|
|
36
|
+
- [Compatibility](#compatibility)
|
|
37
|
+
- [Not supported](#not-supported)
|
|
38
|
+
- [Development](#development)
|
|
39
|
+
- [Design decisions](#design-decisions)
|
|
40
|
+
|
|
41
|
+
## Installation
|
|
42
|
+
|
|
43
|
+
Calcsheet will be published on [rubygems.org](https://rubygems.org/gems/calcsheet); its Git tag is created at the
|
|
44
|
+
release. Once it is released, add it to your Gemfile:
|
|
45
|
+
|
|
46
|
+
```rb
|
|
47
|
+
gem "calcsheet", "~> 0.1"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The runtime dependencies are `ox` (a SAX parser with a C extension), `rubyzip`, `zip_kit`, `bigdecimal` and `csv`.
|
|
51
|
+
Nothing needs Rails or ActiveSupport.
|
|
52
|
+
|
|
53
|
+
## Reading
|
|
54
|
+
|
|
55
|
+
### Quick start
|
|
56
|
+
|
|
57
|
+
The typical import has two phases: show the beginning of the file so that the user can map the columns, then read
|
|
58
|
+
everything with that mapping, in batches, keeping the rows that have errors aside.
|
|
59
|
+
|
|
60
|
+
```ruby
|
|
61
|
+
require "calcsheet"
|
|
62
|
+
|
|
63
|
+
Calcsheet.open("contacts.xlsx") do |book|
|
|
64
|
+
book.format # => :xlsx
|
|
65
|
+
book.sheets.map(&:name) # => ["Contacts", "Contacts (2)"]
|
|
66
|
+
|
|
67
|
+
# 1. The header and the first rows of each visible sheet, with native values.
|
|
68
|
+
sample = book.sample(rows: 5).first
|
|
69
|
+
sample.header.values # => ["Phone", "Name", "Email", "Postal code", "Price", "Created at"]
|
|
70
|
+
sample.rows.first.values # => ["+393331234567", "Ada", "ada@example.com", "00013", BigDecimal("12.5"), Time.utc(2026, 1, 1, 10, 0, 0)]
|
|
71
|
+
|
|
72
|
+
# 2. Can the sheets be read as one? Every non-empty sheet must have the same header.
|
|
73
|
+
book.merge_check.mergeable? # => true
|
|
74
|
+
end
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
The user's mapping becomes a column schema: each key names a column, by 0-based position (`at:`) or by header name
|
|
78
|
+
(`header:`, compared ignoring case and whitespace), with an optional declared type and `required:` flag.
|
|
79
|
+
|
|
80
|
+
```ruby
|
|
81
|
+
require "calcsheet"
|
|
82
|
+
|
|
83
|
+
columns = {
|
|
84
|
+
phone: { at: 0, type: :string },
|
|
85
|
+
email: { header: "email", required: true },
|
|
86
|
+
postal_code: { header: "postal code", type: :string },
|
|
87
|
+
price: { at: 4, type: :decimal },
|
|
88
|
+
created_at: { at: 5, type: :datetime }
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
imported = []
|
|
92
|
+
rejected = []
|
|
93
|
+
Calcsheet.open("contacts.xlsx") do |book|
|
|
94
|
+
book.each_batch(size: 1000, sheet: :all, columns: columns) do |rows|
|
|
95
|
+
valid, invalid = rows.partition(&:valid?)
|
|
96
|
+
imported.concat(valid.map(&:to_h)) # for example, insert_all into a table
|
|
97
|
+
rejected.concat(invalid.flat_map(&:errors)) # report them to the user
|
|
98
|
+
end
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
imported.size # => 2
|
|
102
|
+
imported.first # => { phone: "+393331234567", email: "ada@example.com", postal_code: "00013", price: BigDecimal("12.5"), created_at: Time.utc(2026, 1, 1, 10, 0, 0) }
|
|
103
|
+
rejected.map(&:message) # => ["sheet 'Contacts (2)', row 2, column C (email): required"]
|
|
104
|
+
rejected.map { |error| [error.sheet, error.row, error.column_letter, error.key, error.reason] } # => [["Contacts (2)", 2, "C", :email, :required]]
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
### Opening a file
|
|
108
|
+
|
|
109
|
+
```rb
|
|
110
|
+
Calcsheet.open(source, format: nil, **options) { |book| ... } # yields the book, closes it, returns the block's value
|
|
111
|
+
book = Calcsheet.open(source, **options) # without a block the caller must call book.close
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
- `source` is a path (`String` or `Pathname`) or an IO: a `File`, a `Tempfile`, a `StringIO`, an upload object, a
|
|
115
|
+
pipe. A seekable IO is read from its start whatever its position (an upload already read once for a preview is
|
|
116
|
+
fine). An IO given by the caller is never closed. An upload object is named after its `original_filename`,
|
|
117
|
+
whether it seeks itself (Rack's `Rack::Multipart::UploadedFile`, which delegates to its Tempfile) or not; one that
|
|
118
|
+
cannot seek but answers `to_io` (Rails' `ActionDispatch::Http::UploadedFile`) is read through the file `to_io`
|
|
119
|
+
returns, without a copy.
|
|
120
|
+
- The format is detected from the content: a zip with `xl/workbook.xml` is XLSX, a zip whose `mimetype` is
|
|
121
|
+
`application/vnd.oasis.opendocument.spreadsheet` is ODS, anything else is CSV. The extension only breaks a tie.
|
|
122
|
+
`format: :csv`, `:xlsx` or `:ods` skips detection. An XLS (Excel 97–2003) file, an encrypted (password-protected)
|
|
123
|
+
workbook, which is an OLE2 container like XLS, or another kind of zip raises `Calcsheet::UnsupportedFormatError`.
|
|
124
|
+
Entry names are compared with a leading `/` removed and `\` read as `/`, in detection as in reading; two entries
|
|
125
|
+
of the same name, and an archive comment that holds an end of central directory signature (the zip library would
|
|
126
|
+
read it as the record), raise `MalformedFileError`.
|
|
127
|
+
- `options` are any keys of the [options reference](#options-reference), for this book.
|
|
128
|
+
- `book.format` is `:csv`, `:xlsx` or `:ods`; `book.sheets` lists every sheet in workbook order as
|
|
129
|
+
`Calcsheet::SheetInfo` (`name`, `index`, `hidden?`), hidden ones included. A CSV file has one sheet, named after
|
|
130
|
+
the file without its directories and extension (an upload's `original_filename`, Windows separators included;
|
|
131
|
+
`"csv"` for an IO without a path).
|
|
132
|
+
|
|
133
|
+
### Rows
|
|
134
|
+
|
|
135
|
+
```rb
|
|
136
|
+
book.rows(**read) # => a lazy Enumerator of Calcsheet::Row
|
|
137
|
+
book.each_row(**read) { |row| ... }
|
|
138
|
+
book.each_batch(size: 1000, **read) { |rows| ... }
|
|
139
|
+
book.sample(rows: 20, sheets: :all, include_hidden: false) # => [Calcsheet::Sample(sheet:, header:, rows:)]
|
|
140
|
+
book.merge_check(sheets: :all, include_hidden: false) # => Calcsheet::MergeCheck(sheet_count:, mismatches:)
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
`rows` reads one batch at a time while it is iterated: `book.rows.first(10)` reads only the beginning of the
|
|
144
|
+
sheet. The batch size never changes what is yielded. `sample` returns, for each selected sheet, its header row
|
|
145
|
+
(`nil` for an empty sheet) and its first data rows, reading only the beginning of each sheet.
|
|
146
|
+
|
|
147
|
+
The read options (`**read`) are these, plus any key of the [options reference](#options-reference) as an override
|
|
148
|
+
for that read (for example `time_zone:` or `decimal_separator:`), except the options that apply when the file is
|
|
149
|
+
opened: `max_file_size`, `max_uncompressed_size`, `max_compression_ratio`, `compression_ratio_floor`,
|
|
150
|
+
`max_zip_entries`, `max_sheets` and `max_cell_bytes` raise `ArgumentError` in a read; pass them to `Calcsheet.open`.
|
|
151
|
+
`sample` and `merge_check` take `include_hidden:` too.
|
|
152
|
+
|
|
153
|
+
| Read option | Default | Meaning |
|
|
154
|
+
|---|---|---|
|
|
155
|
+
| `sheet:` | first visible sheet | An index, a name, an Array of them, or `:all` (every visible sheet; like the default, the first sheet when every sheet is hidden). Several sheets are read as one: see [merged sheets](#merged-sheets). |
|
|
156
|
+
| `header:` | `true` | The first non-blank row is the header and is not yielded. `false` yields every row. |
|
|
157
|
+
| `columns:` | `nil` | A [column schema](#column-schema). Without one every column is returned with its native value. |
|
|
158
|
+
| `types:` | `:native` | `:string` reads every column without a declared type as text. |
|
|
159
|
+
| `on_error:` | `:collect` | `:collect` attaches cell errors to the row; `:raise` raises the first one as `Calcsheet::CellError`. |
|
|
160
|
+
| `start_at:` | `nil` | Resume an interrupted job: a row number, or `{ sheet:, row: }` when reading several sheets. |
|
|
161
|
+
| `skip_blank:` | `true` | Rows without a non-blank cell are not yielded; row numbers keep the gaps. |
|
|
162
|
+
| `include_hidden:` | `false` | `sheet: :all` includes the hidden sheets. A hidden sheet is otherwise read only when named or indexed. |
|
|
163
|
+
| `required_headers:` | `nil` | Header names that must be present (ignoring case and whitespace), checked before the first row. |
|
|
164
|
+
|
|
165
|
+
Each `Calcsheet::Row` has:
|
|
166
|
+
|
|
167
|
+
- `sheet` (the sheet name), `sheet_index`, `number` (the row number the user sees, see [row numbers](#row-numbers))
|
|
168
|
+
and `line` (CSV only: the physical line where the record starts);
|
|
169
|
+
- `values`: in file order, or in schema order with a schema;
|
|
170
|
+
- `row[index]`, `row[key]` and `to_h`: keys are the schema keys, or without a schema the
|
|
171
|
+
header texts stripped of surrounding whitespace, non-breaking spaces included (a numeric header by its plain
|
|
172
|
+
text: `"2.5"`, `"10"`). A blank header cell becomes its
|
|
173
|
+
column letter (`"C"`), a repeated header gets a counter (`"Phone"`, `"Phone (2)"`), and cells beyond the header are
|
|
174
|
+
keyed by their column letter, with the same counter when a header has that text (`"D (2)"` after a header `"D"`,
|
|
175
|
+
which `row["D"]` keeps meaning);
|
|
176
|
+
- `errors` (an Array of `Calcsheet::CellError`) and `valid?`.
|
|
177
|
+
|
|
178
|
+
Without a schema, `values` are as long as the row with trailing blank cells trimmed; in header mode they are padded
|
|
179
|
+
with `nil` to the width of the header, and cells beyond the header are kept.
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
require "calcsheet"
|
|
183
|
+
|
|
184
|
+
Calcsheet.open("contacts.xlsx") do |book|
|
|
185
|
+
row = book.rows.first
|
|
186
|
+
row.number # => 2
|
|
187
|
+
row[0] # => "+393331234567"
|
|
188
|
+
row["Postal code"] # => "00013"
|
|
189
|
+
row.to_h.keys # => ["Phone", "Name", "Email", "Postal code", "Price", "Created at"]
|
|
190
|
+
|
|
191
|
+
# Resume at row 3 of "Contacts": the rest of that sheet, then the following sheets.
|
|
192
|
+
book.rows(sheet: :all, start_at: { sheet: "Contacts", row: 3 }).map { |r| [r.sheet, r.number] } # => [["Contacts", 3], ["Contacts (2)", 2]]
|
|
193
|
+
end
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
`start_at:` skips without parsing where the format allows it: XLSX byte-scans for the `r` attribute of the rows, ODS
|
|
197
|
+
counts row start tags, CSV parses the records it skips (quotes make byte scanning unsafe) but builds no rows. The
|
|
198
|
+
byte scans pass over comments, CDATA sections and processing instructions, and assume that the XML before their
|
|
199
|
+
target is well-formed. A `<` that starts no complete tag before a row, a sheet or a markup opening usually raises
|
|
200
|
+
`MalformedFileError` (a tag cut by a chunk boundary may be passed over unchecked); but on other malformed input, such as a complete tag inside an attribute value, the scans
|
|
201
|
+
(`start_at:`, and for ODS the sheet listing) may disagree with a full read, misnumbering rows or missing a sheet.
|
|
202
|
+
They never allocate beyond their bounds or raise errors other than Calcsheet's (ADR 008, in the [repository](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/docs/adr/008-limits-for-untrusted-files.md)).
|
|
203
|
+
|
|
204
|
+
### Column schema
|
|
205
|
+
|
|
206
|
+
```rb
|
|
207
|
+
columns: {
|
|
208
|
+
phone: { at: 1, type: :string }, # by 0-based position: the primary use
|
|
209
|
+
first_name: { at: 3 }, # native type
|
|
210
|
+
price: { header: "Price", type: :decimal }, # by header name
|
|
211
|
+
active: { at: 10, type: :boolean, required: true }
|
|
212
|
+
}
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
- Mapping by position is the primary mode, since header names may repeat. `at:` is an Integer from 0 to 16,383
|
|
216
|
+
(the last spreadsheet column, XFD), checked when the schema is built. Mapping by header name compares names
|
|
217
|
+
downcased with all whitespace removed; a name that matches no column raises `Calcsheet::ColumnNotFoundError`, one
|
|
218
|
+
that matches several raises `Calcsheet::AmbiguousColumnError`, both before the first row. `header:` mapping needs
|
|
219
|
+
`header: true`.
|
|
220
|
+
- Columns the schema does not name are dropped from `values` and `to_h`. Keys are Symbols (a String key is
|
|
221
|
+
converted), so `:a` and `"a"` together raise `ArgumentError`; several keys may map one position only with the same
|
|
222
|
+
type.
|
|
223
|
+
- `type:` is one of `:string`, `:integer`, `:decimal`, `:date`, `:datetime`, `:boolean` (see
|
|
224
|
+
[declared types](#declared-types)); without it the value is native.
|
|
225
|
+
- `required: true` makes a blank cell (`nil` or whitespace only) a cell error with reason `:required`. A
|
|
226
|
+
whitespace-only cell reads as `nil` for every declared type but `:string`, in every position, never as a cast
|
|
227
|
+
error; `:string` and native values keep the text as it is.
|
|
228
|
+
- The header row is always read with native values: declared types apply to data rows only.
|
|
229
|
+
|
|
230
|
+
### Merged sheets
|
|
231
|
+
|
|
232
|
+
Reading several sheets (`sheet: :all`, or an Array) yields them as one stream of rows:
|
|
233
|
+
|
|
234
|
+
- it requires `header: true`;
|
|
235
|
+
- before the first row is yielded, the headers are compared with the normalization of header mapping, trailing
|
|
236
|
+
blank header cells ignored; completely empty sheets are skipped. A difference raises
|
|
237
|
+
`Calcsheet::HeaderMismatchError`, whose `mismatches` are `{ sheet:, expected:, actual: }` with the full header
|
|
238
|
+
rows of the first non-empty sheet and of the differing one;
|
|
239
|
+
- the header rows of the later sheets are not yielded, and row numbers are per sheet.
|
|
240
|
+
|
|
241
|
+
`book.merge_check` runs the same comparison without reading the rows: `mergeable?`, `mismatches` and `sheet_count`
|
|
242
|
+
(the sheets checked, empty ones included). A workbook with one sheet, and every CSV file, is trivially mergeable.
|
|
243
|
+
|
|
244
|
+
Hidden sheets often hold lookup data, so `:all` means the visible sheets. The default sheet is the first visible
|
|
245
|
+
one; a hidden sheet is read when it is named or indexed, or with `include_hidden: true`.
|
|
246
|
+
|
|
247
|
+
### Cell errors
|
|
248
|
+
|
|
249
|
+
With `on_error: :collect` (the default) a cell that cannot be converted, or a blank `required:` cell, becomes `nil`
|
|
250
|
+
in `values` and a `Calcsheet::CellError` in `row.errors`, and reading goes on. With `on_error: :raise` the first one
|
|
251
|
+
is raised. Problems of the file itself (damaged, too large, missing sheet) always raise.
|
|
252
|
+
|
|
253
|
+
```ruby
|
|
254
|
+
require "calcsheet"
|
|
255
|
+
|
|
256
|
+
Calcsheet.open("contacts.csv") do |book|
|
|
257
|
+
columns = { phone: { at: 0 }, price: { header: "Price", type: :decimal }, created_at: { at: 3, type: :datetime } }
|
|
258
|
+
first, second = book.rows(columns: columns, decimal_separator: ",").to_a
|
|
259
|
+
|
|
260
|
+
first.to_h # => { phone: "+393331234567", price: BigDecimal("12.50"), created_at: Time.utc(2026, 1, 1, 10, 0, 0) }
|
|
261
|
+
second[:price] # => nil
|
|
262
|
+
second.errors.map(&:message) # => ["sheet 'contacts', row 3, column C (price): invalid decimal \"n/a\""]
|
|
263
|
+
|
|
264
|
+
begin
|
|
265
|
+
book.each_row(columns: columns, decimal_separator: ",", on_error: :raise) { |row| row }
|
|
266
|
+
rescue Calcsheet::CellError => error
|
|
267
|
+
error.row # => 3
|
|
268
|
+
error.value # => "n/a"
|
|
269
|
+
error.reason # => :invalid_decimal
|
|
270
|
+
end
|
|
271
|
+
end
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
## Writing
|
|
275
|
+
|
|
276
|
+
### To a file
|
|
277
|
+
|
|
278
|
+
```ruby
|
|
279
|
+
require "calcsheet"
|
|
280
|
+
|
|
281
|
+
Calcsheet.write("price-list.xlsx") do |book| # the format comes from the extension: .csv, .xlsx or .ods
|
|
282
|
+
book.sheet("Prices", columns: [
|
|
283
|
+
{ header: "Code", type: :string, width: 12 },
|
|
284
|
+
{ header: "Price", type: :decimal, scale: 2 },
|
|
285
|
+
{ header: "Valid from", type: :date },
|
|
286
|
+
{ header: "Updated at", type: :datetime },
|
|
287
|
+
{ header: "Active", type: :boolean },
|
|
288
|
+
{ header: "Note", key: :note }
|
|
289
|
+
]) do |sheet|
|
|
290
|
+
sheet << ["00013", BigDecimal("12.3456"), Date.new(2026, 1, 1), Time.utc(2026, 1, 1, 10, 0, 0), true, "=not a formula"]
|
|
291
|
+
sheet << { "Code" => "00014", "Price" => 7, note: "a Hash row" } # keyed by header, or by key:; missing keys are empty
|
|
292
|
+
end
|
|
293
|
+
end
|
|
294
|
+
|
|
295
|
+
Calcsheet.open("price-list.xlsx") do |book|
|
|
296
|
+
row = book.rows.first
|
|
297
|
+
row["Code"] # => "00013"
|
|
298
|
+
row["Price"] # => BigDecimal("12.3456")
|
|
299
|
+
row["Note"] # => "=not a formula"
|
|
300
|
+
end
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
`Calcsheet.write(target, format: nil, **options) { |book| ... }` writes a path (format from the extension, written in
|
|
304
|
+
binary mode and closed) or any object responding to `write` or `<<` (`format:` required). A path is written to a
|
|
305
|
+
temporary file in the same directory (that of the file a symlink points to, which a dangling symlink gets created),
|
|
306
|
+
renamed over the target once the block has completed, so an existing file stays untouched until then and keeps its
|
|
307
|
+
permissions, though not its owner and group; when the block raises, the temporary file is deleted and the target is
|
|
308
|
+
unchanged. A target that cannot be written raises `Errno::EACCES`, as opening it would. A target that is not a regular file (a FIFO, `/dev/null`), or
|
|
309
|
+
one in a directory that is not writable, is written directly instead: then a failed block leaves what was already
|
|
310
|
+
written. An IO keeps what was already written, and is never closed.
|
|
311
|
+
|
|
312
|
+
`book.sheet(name, columns:, header: true) { |sheet| ... }` writes one sheet (`header:` is `true` or `false`); sheets
|
|
313
|
+
are written one after another, and a finished sheet cannot be reopened. Each column is a Hash:
|
|
314
|
+
|
|
315
|
+
| Key | Default | Meaning |
|
|
316
|
+
|---|---|---|
|
|
317
|
+
| `header:` | required | The header text, and the key of Hash rows. |
|
|
318
|
+
| `type:` | `:string` | `:string`, `:integer`, `:decimal`, `:date`, `:datetime` or `:boolean`. |
|
|
319
|
+
| `width:` | `nil` | Column width in characters (XLSX and ODS): a number above 0 and at most 255. |
|
|
320
|
+
| `scale:` | `nil` | Decimals shown by XLSX and ODS for a `:decimal` column. Display only: see below. |
|
|
321
|
+
| `key:` | `nil` | The key of the column in Hash rows, instead of the header. |
|
|
322
|
+
| `formula_protection:` | `true` | CSV only: see [formula protection](#formula-protection). |
|
|
323
|
+
|
|
324
|
+
`sheet << row` takes an Array in column order or a Hash (an unknown key raises `ArgumentError`). Every value must
|
|
325
|
+
fit its column type, otherwise `Calcsheet::WriteError` names the sheet, row and column, nothing of that row is
|
|
326
|
+
written and the sheet stays usable:
|
|
327
|
+
|
|
328
|
+
| Column type | Accepted values |
|
|
329
|
+
|---|---|
|
|
330
|
+
| `:string` | a `String` (converted to UTF-8) |
|
|
331
|
+
| `:integer` | an `Integer` |
|
|
332
|
+
| `:decimal` | an `Integer` or a finite `BigDecimal`; a `Float` is refused, so that no binary rounding slips in |
|
|
333
|
+
| `:date` | a `Date` (not a `DateTime`) |
|
|
334
|
+
| `:datetime` | a `Time`, written in whole seconds (rounded) |
|
|
335
|
+
| `:boolean` | `true` or `false` |
|
|
336
|
+
|
|
337
|
+
`nil` writes an empty cell. A `:string` column writes text even for `"00013"` or `"+39..."`. In every format a number
|
|
338
|
+
is written with all its digits, so one whose text would exceed 1,024 characters (the most the readers read, see
|
|
339
|
+
[limits](#limits)) raises `WriteError`. So does a text longer than `max_cell_bytes` (1 MiB by default; `nil` for no
|
|
340
|
+
limit), which the readers would refuse, counting the `'` CSV puts before a formula. A written header longer than
|
|
341
|
+
`max_cell_bytes` (counted the same way) raises `WriteError` from `book.sheet`, before anything of that sheet is
|
|
342
|
+
written.
|
|
343
|
+
|
|
344
|
+
**`scale:` is display only.** XLSX and ODS store the full value and show it with `scale` decimals; CSV ignores
|
|
345
|
+
`scale` and writes the full value. Calcsheet never rounds a written decimal: round the values yourself before
|
|
346
|
+
writing them if the file must hold rounded values. Numbers with more than 15 significant digits are written as
|
|
347
|
+
given: Calcsheet reads integers and numbers of more than 17 significant digits back exactly, rounds fractions of 16
|
|
348
|
+
or 17 to 15, and Excel keeps only 15.
|
|
349
|
+
|
|
350
|
+
### To an IO
|
|
351
|
+
|
|
352
|
+
```ruby
|
|
353
|
+
require "calcsheet"
|
|
354
|
+
require "stringio"
|
|
355
|
+
|
|
356
|
+
io = StringIO.new
|
|
357
|
+
Calcsheet.write(io, format: :csv, excel_it: true) do |book|
|
|
358
|
+
book.sheet("Contacts", columns: [{ header: "Phone" }, { header: "Price", type: :decimal }]) do |sheet|
|
|
359
|
+
sheet << ["+393331234567", BigDecimal("12.5")]
|
|
360
|
+
end
|
|
361
|
+
end
|
|
362
|
+
io.string # => "\uFEFFPhone;Price\r\n'+393331234567;12,5\r\n"
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
The IO may be non-seekable (a socket, a pipe): no format needs to seek its output. A background job can write a
|
|
366
|
+
`Tempfile` and attach it elsewhere.
|
|
367
|
+
|
|
368
|
+
### As an HTTP response
|
|
369
|
+
|
|
370
|
+
`Calcsheet.stream(format:, **options) { |book| ... }` returns a Rack body: its `each` runs the block and yields the
|
|
371
|
+
file in binary chunks of at least 64 KiB (the last one may be shorter) while the rows are produced, and memory
|
|
372
|
+
stays flat. The first chunk leaves once 64 KiB of output exist: for CSV after about 64 KiB of rows, for XLSX and
|
|
373
|
+
ODS after the compressor has produced 64 KiB of deflated data, which takes several times more rows. Options and format are checked when the body is built, before any byte.
|
|
374
|
+
The size is not known in advance (chunked transfer). Each call of `each` writes the whole file again; an exception in the block propagates
|
|
375
|
+
from `each` after the chunks already sent, so the client receives a truncated file.
|
|
376
|
+
|
|
377
|
+
```ruby
|
|
378
|
+
require "calcsheet"
|
|
379
|
+
|
|
380
|
+
body = Calcsheet.stream(format: :xlsx) do |book|
|
|
381
|
+
book.sheet("Contacts", columns: [{ header: "Email" }]) do |sheet|
|
|
382
|
+
%w[ada@example.com grace@example.com].each { |email| sheet << [email] }
|
|
383
|
+
end
|
|
384
|
+
end
|
|
385
|
+
headers = {
|
|
386
|
+
"content-type" => "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
|
387
|
+
"content-disposition" => %(attachment; filename="contacts.xlsx")
|
|
388
|
+
}
|
|
389
|
+
app = ->(_env) { [200, headers, body] } # a Rack application
|
|
390
|
+
|
|
391
|
+
# What the server does with the body:
|
|
392
|
+
_status, _headers, chunks = app.call({})
|
|
393
|
+
File.open("export.xlsx", "wb") { |file| chunks.each { |chunk| file.write(chunk) } }
|
|
394
|
+
Calcsheet.open("export.xlsx") { |book| book.rows.map { |row| row["Email"] } } # => ["ada@example.com", "grace@example.com"]
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
In a Rails controller, assign the body to `response_body`:
|
|
398
|
+
|
|
399
|
+
```rb
|
|
400
|
+
class ContactsController < ApplicationController
|
|
401
|
+
def export
|
|
402
|
+
headers["Content-Type"] = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
|
403
|
+
headers["Content-Disposition"] = %(attachment; filename="contacts.xlsx")
|
|
404
|
+
self.response_body = Calcsheet.stream(format: :xlsx) do |book|
|
|
405
|
+
book.sheet("Contacts", columns: [{ header: "Email" }, { header: "Created at", type: :datetime }]) do |sheet|
|
|
406
|
+
Contact.find_each { |contact| sheet << [contact.email, contact.created_at] }
|
|
407
|
+
end
|
|
408
|
+
end
|
|
409
|
+
end
|
|
410
|
+
end
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
The block runs while the server sends the response, after the action returns. A middleware that buffers bodies (to
|
|
414
|
+
compute an ETag or a Content-Length, for example) defeats the streaming: check the middleware stack.
|
|
415
|
+
|
|
416
|
+
### Sheet names and overflow
|
|
417
|
+
|
|
418
|
+
Sheet names are validated for every format: 1 to 31 characters, counted as Excel counts them (a character beyond
|
|
419
|
+
the Basic Multilingual Plane, such as an emoji, counts twice), none of `[ ] : * ? / \` and no control characters,
|
|
420
|
+
not starting or ending with an apostrophe, and unique ignoring case. An invalid name raises `ArgumentError`, unless
|
|
421
|
+
`sanitize_sheet_names: true` replaces the forbidden characters with `_`, shortens the name (never inside a character
|
|
422
|
+
or a combining sequence) and numbers duplicates (`"Name (2)"`).
|
|
423
|
+
|
|
424
|
+
XLSX and ODS hold at most 1,048,576 rows per sheet. When a sheet reaches the limit (header included), the writer
|
|
425
|
+
continues in a new sheet named after the first with a counter (`"Contacts"`, `"Contacts (2)"`, `"Contacts (3)"`,
|
|
426
|
+
shortening the base name to keep 31 characters) and repeats the header (a row that would be refused opens no
|
|
427
|
+
continuation); the block does not notice, and `sheet.name` tells the current one. `max_rows_per_sheet:` lowers the
|
|
428
|
+
limit; a value above the format's limit raises `ArgumentError` before anything is written. Raising instead of continuing is not offered: in streaming the overflow
|
|
429
|
+
is discovered after a million rows were sent. A CSV file has no row limit and holds a single sheet: a second
|
|
430
|
+
`book.sheet` raises `Calcsheet::WriteError`.
|
|
431
|
+
|
|
432
|
+
```ruby
|
|
433
|
+
require "calcsheet"
|
|
434
|
+
|
|
435
|
+
Calcsheet.write("long.ods", max_rows_per_sheet: 3) do |book|
|
|
436
|
+
book.sheet("Contacts", columns: [{ header: "Id", type: :integer }]) do |sheet|
|
|
437
|
+
(1..5).each { |id| sheet << [id] }
|
|
438
|
+
end
|
|
439
|
+
end
|
|
440
|
+
|
|
441
|
+
Calcsheet.open("long.ods") do |book|
|
|
442
|
+
book.sheets.map(&:name) # => ["Contacts", "Contacts (2)", "Contacts (3)"]
|
|
443
|
+
book.rows(sheet: :all).map { |row| row["Id"] } # => [1, 2, 3, 4, 5]
|
|
444
|
+
end
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
## Types
|
|
448
|
+
|
|
449
|
+
### Native values
|
|
450
|
+
|
|
451
|
+
Without a declared type a cell reads as:
|
|
452
|
+
|
|
453
|
+
| Cell | Value |
|
|
454
|
+
|---|---|
|
|
455
|
+
| CSV field | `String`, never guessed; an empty field is `nil` |
|
|
456
|
+
| XLSX/ODS text | `String` |
|
|
457
|
+
| XLSX/ODS number, integral | `Integer` (`BigDecimal` beyond 30 digits) |
|
|
458
|
+
| XLSX/ODS number, fractional | `BigDecimal`; `decimals: :float` returns `Float` |
|
|
459
|
+
| XLSX number with a date format, ODS date without time | `Date` |
|
|
460
|
+
| XLSX number with a date and time format, ODS date with time | `Time` (see [time zones](#time-zones)) |
|
|
461
|
+
| XLSX number with a time-only or duration format (`hh:mm`, `[h]:mm:ss`), ODS time | `Integer` seconds (`BigDecimal` when fractional; `Float` with `decimals: :float`) |
|
|
462
|
+
| boolean | `true` or `false` |
|
|
463
|
+
| error (`#DIV/0!`...) | `nil`, with a cell error of reason `:cell_error` |
|
|
464
|
+
| formula | its cached value; `nil` when the file caches none |
|
|
465
|
+
| merged range | the value in the top-left cell, `nil` in the others |
|
|
466
|
+
|
|
467
|
+
- XLSX stores numbers as text with up to 17 significant digits; Excel itself keeps 15. Calcsheet builds the
|
|
468
|
+
`BigDecimal` from the text, never through `Float`, and rounds it half up to 15 significant digits only when the
|
|
469
|
+
text is the full print of a double: a fraction or an exponent with 16 or 17 significant digits
|
|
470
|
+
(`0.30000000000000004` reads as `0.3`). Integer text is always exact (`9007199254740993`, `2**64`), and so is text
|
|
471
|
+
with more than 17 significant digits. A `BigDecimal` costs about 160 bytes more than a `Float`; that matters only
|
|
472
|
+
when millions of values are kept in memory.
|
|
473
|
+
- The XML character references of XLSX text and sheet names and of ODS text, cell attributes and sheet names
|
|
474
|
+
(`&`, `A`) are decoded by Calcsheet, CDATA sections are kept as written: a reference to NUL, a surrogate or
|
|
475
|
+
a code point beyond U+10FFFF, an unknown entity and an `&` that starts no reference raise `MalformedFileError`
|
|
476
|
+
(the XML parser would cut or change the text).
|
|
477
|
+
- Date or datetime is decided by the number format of the cell, never by the value: a datetime at midnight stays a
|
|
478
|
+
`Time`. XLSX built-in formats 14–17 and 27–36 and 50–58 are dates, 22 a datetime, 18–21 and 45–47 times and
|
|
479
|
+
durations; custom codes are classified by their date and time tokens outside quotes and brackets. ODS cells
|
|
480
|
+
follow their data style the same way. The 1904 date system is honored.
|
|
481
|
+
- A number under a plain zero-padding format (`00000`) gets its leading zeros back when the column is declared
|
|
482
|
+
`:string` (`13` becomes `"00013"`).
|
|
483
|
+
|
|
484
|
+
### Declared types
|
|
485
|
+
|
|
486
|
+
A schema `type:` converts from any representation; a value that does not convert is a cell error with the reason
|
|
487
|
+
shown.
|
|
488
|
+
|
|
489
|
+
| Type | From a number, date or boolean cell | From text (CSV, text cells) | Reasons |
|
|
490
|
+
|---|---|---|---|
|
|
491
|
+
| `:string` | plain digits without exponent (`393331234567`, `0.0101`); dates and datetimes in ISO 8601, times of day as `HH:MM:SS`; `"true"`/`"false"` | the text | |
|
|
492
|
+
| `:integer` | integral numbers only; booleans as `1`/`0` | `"12"`, `"12.0"`, with `decimal_separator:` and `thousands_separator:` | `:invalid_integer`, `:not_integral` |
|
|
493
|
+
| `:decimal` | the exact `BigDecimal`, also for integral numbers (`7` gives `BigDecimal("7")`) | `decimal_separator:` (`"."` or `","`), optional `thousands_separator:`, surrounding whitespace ignored | `:invalid_decimal` |
|
|
494
|
+
| `:date` | date and datetime cells (a datetime is truncated) | `date_formats:`, then `datetime_formats:` | `:invalid_date` |
|
|
495
|
+
| `:datetime` | datetime and date cells (a date gives midnight) | `datetime_formats:`, then `date_formats:` | `:invalid_datetime` |
|
|
496
|
+
| `:boolean` | boolean cells; the numbers `1` and `0` | `true_words:` / `false_words:`, case-insensitive | `:invalid_boolean` |
|
|
497
|
+
|
|
498
|
+
A number with a general (non-date) format declared `:date` or `:datetime` is a cell error: reading it as a date
|
|
499
|
+
serial would silently turn years or ids into dates of 1905. Text dates match a format only as a whole and only with
|
|
500
|
+
a real calendar date. A blank `required:` cell has reason `:required`, an error cell `:cell_error`.
|
|
501
|
+
|
|
502
|
+
## Row numbers
|
|
503
|
+
|
|
504
|
+
`row.number` is the row the user sees in the application that opens the file:
|
|
505
|
+
|
|
506
|
+
- **XLSX**: the `r` attribute of the row; when a generator leaves it out, rows are counted.
|
|
507
|
+
- **ODS**: table rows counted, repeated rows (`number-rows-repeated`) included.
|
|
508
|
+
- **CSV**: the 1-based **record** number, counting the header record and blank lines, because Excel and LibreOffice
|
|
509
|
+
show a quoted multi-line field as one row. `row.line` gives the physical line where the record starts, for text
|
|
510
|
+
editors.
|
|
511
|
+
|
|
512
|
+
Header, blank and skipped rows keep their numbers, so the numbers have gaps instead of shifting. In merged reading
|
|
513
|
+
the number is per sheet. `CellError#row`, `MalformedFileError#row` and `WriteError#row` use the same numbering.
|
|
514
|
+
|
|
515
|
+
```ruby
|
|
516
|
+
require "calcsheet"
|
|
517
|
+
require "stringio"
|
|
518
|
+
|
|
519
|
+
csv = StringIO.new(%(Name,Note\r\nAda,"two\r\nlines"\r\n\r\nGrace,one line\r\n))
|
|
520
|
+
Calcsheet.open(csv) do |book|
|
|
521
|
+
book.rows.map { |row| [row.number, row.line] } # => [[2, 2], [4, 5]]
|
|
522
|
+
end
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
## Errors
|
|
526
|
+
|
|
527
|
+
```text
|
|
528
|
+
Calcsheet::Error < StandardError
|
|
529
|
+
├── UnsupportedFormatError not CSV, XLSX or ODS (XLS gets its own message)
|
|
530
|
+
├── MalformedFileError damaged file; sheet, row, column_index when known; the parser error as cause
|
|
531
|
+
├── LimitExceededError limit, max, actual
|
|
532
|
+
│ ├── FileTooLargeError max_file_size
|
|
533
|
+
│ ├── TooManyRowsError max_rows
|
|
534
|
+
│ ├── TooManySheetsError max_sheets
|
|
535
|
+
│ └── ZipBombError max_uncompressed_size, max_compression_ratio, max_zip_entries
|
|
536
|
+
├── SheetNotFoundError
|
|
537
|
+
├── HeaderMismatchError mismatches: [{ sheet:, expected:, actual: }]
|
|
538
|
+
├── ColumnNotFoundError
|
|
539
|
+
├── AmbiguousColumnError
|
|
540
|
+
├── CellError sheet, row, column_index, column_letter, key, value, reason
|
|
541
|
+
└── WriteError sheet, row, column_index
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
- Messages name the location in user terms ("sheet 'Contacts (2)', row 4812, column C (price): invalid decimal
|
|
545
|
+
\"abc\""); the attributes let applications build their own. `reason` is a Symbol (`:invalid_decimal`,
|
|
546
|
+
`:required`...) that applications can translate.
|
|
547
|
+
- The parsers' own exceptions never leak: they are wrapped in `MalformedFileError`, the original kept as `cause`.
|
|
548
|
+
- Invalid arguments and options raise `ArgumentError` (options are checked before any row is read or any byte is
|
|
549
|
+
written); so do invalid sheet names when writing, and a second read of a non-seekable CSV stream. A closed book
|
|
550
|
+
raises `IOError` ("closed book"), also at the next batch of a read that was going on when it was closed.
|
|
551
|
+
|
|
552
|
+
## Limits
|
|
553
|
+
|
|
554
|
+
Every limit is an option, set globally or per call. Exceeding one raises a `Calcsheet::LimitExceededError`
|
|
555
|
+
subclass whose `limit` names the option, with the observed value (`actual`) and the maximum (`max`). Every limit
|
|
556
|
+
but `max_rows` applies when the file is opened: give it to `Calcsheet.open` or the configuration, not to a read.
|
|
557
|
+
|
|
558
|
+
| Option | Default | Bounds |
|
|
559
|
+
|---|---|---|
|
|
560
|
+
| `max_file_size` | 1 GiB | the bytes of the source; a stream is counted as it is read or copied |
|
|
561
|
+
| `max_rows` | none | the rows one read yields (`rows`, `each_row`, `each_batch`; not `sample`) |
|
|
562
|
+
| `max_uncompressed_size` | 8 GiB | the bytes inflated from the zip archive, cumulative per book: every pass counts |
|
|
563
|
+
| `max_compression_ratio` | 100 | inflated / compressed bytes per entry, beyond `compression_ratio_floor` |
|
|
564
|
+
| `compression_ratio_floor` | 512 MiB | the bytes a streamed entry may inflate to whatever its ratio (the XLSX shared strings, kept in memory, get a fixed 16 MiB instead); `nil` or `0` for none |
|
|
565
|
+
| `max_zip_entries` | 10,000 | the entries of the archive, checked before its central directory is loaded |
|
|
566
|
+
| `max_sheets` | 1,000 | the sheets of an XLSX or ODS workbook |
|
|
567
|
+
| `max_cell_bytes` | 1 MiB | the text of one cell: an XLSX or ODS string, a CSV field once parsed, a text written; `nil` for none |
|
|
568
|
+
|
|
569
|
+
- `max_rows` has no default: set it when reading untrusted files. An ODS describes a million rows of content in a few
|
|
570
|
+
hundred bytes (a repeated row whose cells hold a value), and the reader yields every one of them; the zip limits
|
|
571
|
+
do not see it, `max_rows` does. Wide sparse rows cost too: an XLSX row whose one cell is in column XFD becomes an
|
|
572
|
+
Array of 16,384 values, so 50,000 such rows from a 250 KB file take seconds and hundreds of MB while they are
|
|
573
|
+
read, whatever the schema; `max_rows` keeps that bounded.
|
|
574
|
+
- The zip limits count the bytes actually inflated, never trusting the sizes the archive declares, and the CRC of
|
|
575
|
+
every entry read to its end is verified.
|
|
576
|
+
- The central directory is checked before rubyzip loads it: it must lie before the end records, and the local
|
|
577
|
+
headers and declared data of the entries must not overlap or reach into it, so that loading it reads every local
|
|
578
|
+
header once and every declared compressed size is real. Otherwise the archive raises `MalformedFileError`.
|
|
579
|
+
- While loading the central directory, rubyzip prints `WARNING: invalid date/time in zip entry.` to standard error
|
|
580
|
+
for an entry whose date is invalid (at most `max_zip_entries` times). Calcsheet leaves rubyzip's global settings
|
|
581
|
+
alone; an application can silence the warning with `Zip.warn_invalid_date = false` (after `require "zip"`).
|
|
582
|
+
- `max_uncompressed_size` is cumulative because an ODS keeps every sheet in one `content.xml`, inflated again from
|
|
583
|
+
its start for each sheet read, sampled or merge-checked: however many passes a crafted file causes, one book
|
|
584
|
+
never inflates more than the budget. `max_sheets` bounds those passes.
|
|
585
|
+
- Verbose, repetitive XML (ODS exports especially) deflates beyond 100:1, so an entry may always inflate up to
|
|
586
|
+
`compression_ratio_floor`; only beyond it does the ratio apply. Memory stays flat either way: entries are
|
|
587
|
+
streamed. The XLSX shared strings are the exception: they are kept in memory, 8 bytes per string beyond their
|
|
588
|
+
text, so 512 MiB of empty `<si/>` would take gigabytes. `compression_ratio_floor` does not apply to
|
|
589
|
+
`xl/sharedStrings.xml`: it may inflate to a fixed 16 MiB whatever its ratio (enough for a small file of long,
|
|
590
|
+
repetitive strings; at most about 54 MB of empty `<si/>` while the table is built), and beyond that only as far
|
|
591
|
+
as `max_compression_ratio` allows.
|
|
592
|
+
- The XML of untrusted files is bounded too. The tags the byte scanners read are at most 64 KiB each: XLSX row start
|
|
593
|
+
tags when `start_at:` skips rows, and the table, row, column and row group tags the ODS scanner reads to find
|
|
594
|
+
sheets and rows; other tags are passed over unbuffered, whatever their length. The part
|
|
595
|
+
of an ODS `content.xml` before its first sheet is at most 16 MiB, and so are the column elements and the row group
|
|
596
|
+
start tags kept when `start_at:` skips rows, inside at most 1,000 nested row groups. The XLSX parts read whole, `xl/workbook.xml` and
|
|
597
|
+
its relationships, are at most 1 MiB each, checked while they are inflated, and parsed one at a time (a part of
|
|
598
|
+
tiny elements still costs Ox tens of times its size); `xl/styles.xml` is streamed, keeping only its number formats
|
|
599
|
+
(at most 100,000, codes of at most 1 KiB) and cell formats (at most 1,000,000). Rows beyond 1,048,576, cells
|
|
600
|
+
beyond column XFD and XLSX rows out of order raise `MalformedFileError`.
|
|
601
|
+
- A cell whose text is longer than `max_cell_bytes` raises `MalformedFileError` naming the sheet, the row, the
|
|
602
|
+
column and the limit (for XLSX shared strings, the index of the string), before the text is joined or converted
|
|
603
|
+
(a CSV field only once it is parsed, see below).
|
|
604
|
+
The text of a number, a date, a time or a boolean is at most 1,024 bytes whatever the option; text read as
|
|
605
|
+
`:integer` or `:decimal` longer than that is an invalid value of the cell. Ox, the XML parser, buffers a text or a
|
|
606
|
+
tag whole before Calcsheet sees it, so in an XLSX or ODS part the bytes between two `<` are at most six times
|
|
607
|
+
`max_cell_bytes` plus 64 KiB (6 MiB by default): longer ones raise `MalformedFileError` while the entry is
|
|
608
|
+
inflated. A comment, a CDATA section or an attribute value that holds a `<` is not measured that way; Ox buffers
|
|
609
|
+
it whole, within the zip limits, at two to three times its size.
|
|
610
|
+
- Error messages show at most 200 characters of an attribute value, a cell value, a sheet name or a zip entry name
|
|
611
|
+
read from the file, and of a column header, a column key, a sheet name or a refused value given to a writer
|
|
612
|
+
(listing at most 10 unknown keys of a row), followed by `...` when cut; `error.sheet` and `error.value` keep the
|
|
613
|
+
whole text.
|
|
614
|
+
- A CSV field is checked against `max_cell_bytes` only after the standard library's `CSV::Parser` has built it, so
|
|
615
|
+
the option bounds what Calcsheet keeps and converts, not what parsing costs: there is no per-cell bound for CSV.
|
|
616
|
+
One huge field costs several times its size inside the parser (a single field of 100 MiB peaked at about 390 MB
|
|
617
|
+
unquoted and 490 MB quoted), bounded only by `max_file_size`. For untrusted CSV uploads, set a `max_file_size`
|
|
618
|
+
well below the 1 GiB default, sized to the memory one read may take.
|
|
619
|
+
|
|
620
|
+
```ruby
|
|
621
|
+
require "calcsheet"
|
|
622
|
+
|
|
623
|
+
begin
|
|
624
|
+
Calcsheet.open("contacts.xlsx", max_file_size: 1_000) { |book| book.sheets }
|
|
625
|
+
rescue Calcsheet::FileTooLargeError => error
|
|
626
|
+
error.limit # => :max_file_size
|
|
627
|
+
error.max # => 1_000
|
|
628
|
+
end
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
## CSV
|
|
632
|
+
|
|
633
|
+
### Reading
|
|
634
|
+
|
|
635
|
+
- **Encoding.** A UTF-8 BOM means UTF-8, a UTF-16 LE or BE BOM means UTF-16 (Excel's "Unicode Text"), transcoded
|
|
636
|
+
to UTF-8. Without a BOM, a seekable source is validated as UTF-8 over the whole file with a fast byte-level pass,
|
|
637
|
+
because files often start with ASCII rows and have accented names further down: valid means UTF-8, otherwise
|
|
638
|
+
Windows-1252, common in Excel exports. A non-seekable stream decides on the first 50 lines of its first 64 KiB
|
|
639
|
+
(ASCII counts as UTF-8); invalid bytes later raise `MalformedFileError`. `encoding:` forces any encoding Ruby can
|
|
640
|
+
transcode to UTF-8 and skips detection.
|
|
641
|
+
- **Separator and quote.** The separator is detected among `;`, `,`, tab and `|` by parsing the first 50 records of
|
|
642
|
+
a 64 KiB sample with each candidate and choosing the most consistent column count greater than one, so
|
|
643
|
+
separators inside quoted fields do not mislead it (ties go in that order). The quote character is `"`; `'` is
|
|
644
|
+
chosen only when no separator parses the sample with `"` (a quoted field left open by the end of the 64 KiB sample
|
|
645
|
+
does not count) and `'` gives consistent records of more than one field, so a quoted `"Smith, John"` stays one
|
|
646
|
+
value. A single-column file reads with `,` and `"`. `col_sep:` and `quote_char:` force them.
|
|
647
|
+
- **Lines** end with `\n`, `\r\n` or `\r`, one of them per file: the first line ending outside quoted fields decides,
|
|
648
|
+
and another one later outside quoted fields raises `MalformedFileError` ("mixed line endings") with its record
|
|
649
|
+
number and line, never leaving a stray `\r` in a value or merging two records. Quoted fields keep any line break
|
|
650
|
+
as written. Parsing is strict: a malformed record raises `MalformedFileError` with its record number and line.
|
|
651
|
+
- Values are always `String` (empty fields `nil`); declared types convert them with `decimal_separator:`,
|
|
652
|
+
`thousands_separator:`, `date_formats:`, `datetime_formats:`, `true_words:` and `false_words:`.
|
|
653
|
+
- `strip_formula_prefix: true` removes the `'` that [formula protection](#formula-protection) puts before `=`, `+`,
|
|
654
|
+
`-`, `@`, tab or carriage return, so that files written by Calcsheet import back cleanly. The default is `false`.
|
|
655
|
+
|
|
656
|
+
### Writing
|
|
657
|
+
|
|
658
|
+
| Option | Default | With `excel_it: true` |
|
|
659
|
+
|---|---|---|
|
|
660
|
+
| `col_sep` | `,` | `;` |
|
|
661
|
+
| `bom` | no BOM | UTF-8 BOM |
|
|
662
|
+
| `decimal_separator` | `.` | `,` |
|
|
663
|
+
| `date_format` | `%Y-%m-%d` | `%d/%m/%Y` |
|
|
664
|
+
| `datetime_format` | `%Y-%m-%d %H:%M:%S` | `%d/%m/%Y %H:%M:%S` |
|
|
665
|
+
| `row_sep` | `\r\n` | `\r\n` |
|
|
666
|
+
|
|
667
|
+
`excel_it: true` is the preset that Excel with Italian (and similar European) regional settings opens directly. An
|
|
668
|
+
explicit `col_sep`, `bom`, `date_format` or `datetime_format`, configured or given per call, wins over the preset;
|
|
669
|
+
`decimal_separator` wins only when given in the call, since its default (`"."`) is also the reading default.
|
|
670
|
+
Output is UTF-8. Decimals are written in full without exponent (`scale` is ignored), booleans as `true`/`false`.
|
|
671
|
+
Times are written as their wall clock, without an offset: `%z` in `datetime_format` prints `+0000`.
|
|
672
|
+
|
|
673
|
+
### Formula protection
|
|
674
|
+
|
|
675
|
+
A spreadsheet that opens a CSV file evaluates a field starting with `=`, `+`, `-`, `@`, tab or carriage return as a
|
|
676
|
+
formula, even when the field is quoted: quoting does not protect, because the quotes are removed before the content
|
|
677
|
+
is interpreted. Calcsheet therefore writes a leading `'` before such text in `:string` columns and in headers
|
|
678
|
+
([OWASP](https://owasp.org/www-community/attacks/CSV_Injection)). This is on by default; disable it per column with
|
|
679
|
+
`formula_protection: false`.
|
|
680
|
+
|
|
681
|
+
The consequence: in CSV, `+393331234567` is written as `'+393331234567`. Without the prefix the spreadsheet would
|
|
682
|
+
read a number and drop the `+`. Disable protection for a phone column only when the file is not meant to be opened
|
|
683
|
+
by a spreadsheet application, and read Calcsheet's own exports with `strip_formula_prefix: true`.
|
|
684
|
+
|
|
685
|
+
XLSX and ODS need no prefix: `:string` cells are typed text, never evaluated, so `+39...` and `=...` are stored
|
|
686
|
+
verbatim.
|
|
687
|
+
|
|
688
|
+
## XLSX and ODS
|
|
689
|
+
|
|
690
|
+
Reading:
|
|
691
|
+
|
|
692
|
+
- XLSX sheets are resolved through `xl/workbook.xml` and its relationships; the shared strings and the styles are
|
|
693
|
+
read from `xl/sharedStrings.xml` and `xl/styles.xml`, where Excel and LibreOffice write them. The shared strings
|
|
694
|
+
table is the only part kept in memory, packed in one binary string with offsets: its text plus 8 bytes per string,
|
|
695
|
+
twice that while the table is built (about 1.4 times the text for strings of 20 bytes; a table of empty strings
|
|
696
|
+
costs 8 bytes per `<si/>`, 16 while it is built). Because its cost follows what it inflates to,
|
|
697
|
+
`xl/sharedStrings.xml` gets not `compression_ratio_floor` but a fixed floor of 16 MiB, beyond which
|
|
698
|
+
`max_compression_ratio` bounds it. Rich text runs
|
|
699
|
+
are joined, phonetic runs skipped. Inline strings, `t="d"` ISO dates (read by the kind of their
|
|
700
|
+
number format, like numbers: a date-only format gives a `Date`), booleans, errors and cells or rows without `r`
|
|
701
|
+
attributes are supported. ISO dates of the year 0 are cell errors `:invalid_date`, in ODS too. `hidden` and `veryHidden` sheets are hidden.
|
|
702
|
+
- ODS `content.xml` is streamed; repeated rows and cells are counted, never expanded, so the huge trailing runs
|
|
703
|
+
LibreOffice writes cost nothing. Text joins paragraphs with `\n` and honors `text:s`, `text:tab` and
|
|
704
|
+
`text:line-break`; numbers come from `office:value`, so decimals are exact. The namespace prefixes are read from
|
|
705
|
+
the document, not assumed. A sheet whose table style says `table:display="false"` is hidden. Sampling finds the
|
|
706
|
+
start of each sheet by scanning bytes and parses only its first rows.
|
|
707
|
+
|
|
708
|
+
Writing:
|
|
709
|
+
|
|
710
|
+
- The header row is bold and frozen (XLSX `pane`, ODS `settings.xml`) when `header: true`. Columns get their
|
|
711
|
+
`width:`; ODS widths are rounded to whole characters, from 1 to 255.
|
|
712
|
+
- Number formats: `scale:` decimals (`0.00`), dates `yyyy-mm-dd`, datetimes `yyyy-mm-dd hh:mm:ss`. `scale:` is
|
|
713
|
+
display only: the full value is stored.
|
|
714
|
+
- XLSX text is written as inline strings (no shared strings table) and every sheet is streamed as its rows arrive,
|
|
715
|
+
so memory stays flat. XLSX dates range from 1900-01-01 to 9999-12-31, ODS years from 1 to 9999; others raise
|
|
716
|
+
`WriteError`. An XLSX cell holds at most 32,767 characters of text, counted as Excel does (a character beyond the
|
|
717
|
+
Basic Multilingual Plane, such as an emoji, counts twice); longer text raises `WriteError` without writing the row,
|
|
718
|
+
and a longer header raises it before the sheet starts.
|
|
719
|
+
ODS and CSV have no such limit beyond `max_cell_bytes`.
|
|
720
|
+
- The ODS `mimetype` entry comes first, stored, without data descriptor or extra field, as LibreOffice requires.
|
|
721
|
+
- Characters XML 1.0 does not allow are removed from text. A sheet has at most 16,384 columns.
|
|
722
|
+
- Sheets overflow into `"Name (2)"`, `"Name (3)"`... (see [overflow](#sheet-names-and-overflow)).
|
|
723
|
+
|
|
724
|
+
## Streams and non-seekable input
|
|
725
|
+
|
|
726
|
+
- A **seekable** source (a path, a `File`, a `Tempfile`, a `StringIO`) is read in place.
|
|
727
|
+
- XLSX and ODS need random access to the zip central directory: a **non-seekable** stream is copied to a
|
|
728
|
+
`Tempfile` first (counted against `max_file_size` while copying), and the copy is deleted when the book closes.
|
|
729
|
+
- A non-seekable CSV stream opened with `format: :csv` is read **directly**, without a copy: the detection sample
|
|
730
|
+
(at most 64 KiB) is kept and replayed ahead of the rest of the stream. Such a stream can be read **once**: a
|
|
731
|
+
second read, for example `rows` after `sample`, raises `ArgumentError`. Read a file or a seekable IO, or open the
|
|
732
|
+
stream without `format:`, to read it more than once.
|
|
733
|
+
- A non-seekable stream opened **without** `format:` is spooled to a `Tempfile` to detect its format, CSV included,
|
|
734
|
+
and can then be read any number of times.
|
|
735
|
+
- A seekable CSV source is parsed again from its start whenever a read needs records already passed (`rows` after
|
|
736
|
+
`sample`, a second `rows`).
|
|
737
|
+
|
|
738
|
+
## Time zones
|
|
739
|
+
|
|
740
|
+
Spreadsheet datetimes carry no zone: they are wall-clock fields. Calcsheet keeps that meaning.
|
|
741
|
+
|
|
742
|
+
- **Reading** returns a `Time` in UTC with the wall-clock fields of the file. `time_zone:` builds it in a zone
|
|
743
|
+
instead: `"UTC"`, a UTC offset such as `"+01:00"`, or any object responding to
|
|
744
|
+
`local(year, month, day, hour, min, sec)`, for example `ActiveSupport::TimeZone["Europe/Rome"]` (Calcsheet does
|
|
745
|
+
not depend on ActiveSupport). CSV text read as `:datetime` follows the same rule.
|
|
746
|
+
- XLSX `t="d"` values and ODS dates that carry `Z` or an offset are instants: they are returned in UTC, or converted
|
|
747
|
+
into `time_zone:` when one is set. A duck-typed zone converts them when it responds to `utc_to_local` or
|
|
748
|
+
`utc_offset` (as `ActiveSupport::TimeZone` does); one that responds to `local` alone leaves them in UTC.
|
|
749
|
+
- **Writing** a `Time` writes its wall clock in its own offset, in whole seconds (rounded, since spreadsheets store
|
|
750
|
+
fractions of a day). `time_zone:` converts it first; a duck-typed zone must then respond to `utc_to_local` (as
|
|
751
|
+
`ActiveSupport::TimeZone` does) or `utc_offset`.
|
|
752
|
+
|
|
753
|
+
```ruby
|
|
754
|
+
require "calcsheet"
|
|
755
|
+
|
|
756
|
+
Calcsheet.open("contacts.xlsx") do |book|
|
|
757
|
+
book.rows.first["Created at"] # => Time.utc(2026, 1, 1, 10, 0, 0)
|
|
758
|
+
book.rows(time_zone: "+01:00").first["Created at"] # => Time.new(2026, 1, 1, 10, 0, 0, "+01:00")
|
|
759
|
+
end
|
|
760
|
+
|
|
761
|
+
Calcsheet.write("times.ods", time_zone: "UTC") do |book|
|
|
762
|
+
book.sheet("Times", columns: [{ header: "At", type: :datetime }]) do |sheet|
|
|
763
|
+
sheet << [Time.new(2026, 1, 1, 10, 0, 0, "+01:00")]
|
|
764
|
+
end
|
|
765
|
+
end
|
|
766
|
+
Calcsheet.open("times.ods") { |book| book.rows.first["At"] } # => Time.utc(2026, 1, 1, 9, 0, 0)
|
|
767
|
+
```
|
|
768
|
+
|
|
769
|
+
## Configuration
|
|
770
|
+
|
|
771
|
+
Global defaults are set with `Calcsheet.configure`; every option can be overridden per call
|
|
772
|
+
(`Calcsheet.open(path, max_rows: 100_000)`, `book.rows(time_zone: "UTC")`,
|
|
773
|
+
`Calcsheet.write("contacts.csv", excel_it: true)`), and per call wins; the options marked "open" in the
|
|
774
|
+
[options reference](#options-reference) only when the file is opened. An unknown option key raises `ArgumentError`. `Calcsheet.reset_config!` restores the defaults.
|
|
775
|
+
|
|
776
|
+
```ruby
|
|
777
|
+
require "calcsheet"
|
|
778
|
+
|
|
779
|
+
Calcsheet.configure do |config|
|
|
780
|
+
config.max_file_size = 200 * 1024 * 1024
|
|
781
|
+
config.time_zone = "+01:00"
|
|
782
|
+
# Arrays are frozen copies: assign a new Array instead of changing the current one.
|
|
783
|
+
config.date_formats = [*config.date_formats, "%d.%m.%Y"]
|
|
784
|
+
end
|
|
785
|
+
Calcsheet.config.date_formats # => ["%Y-%m-%d", "%d/%m/%Y", "%d.%m.%Y"]
|
|
786
|
+
```
|
|
787
|
+
|
|
788
|
+
## Options reference
|
|
789
|
+
|
|
790
|
+
Every key of `Calcsheet::Configuration::DEFAULTS`. Read options apply to `Calcsheet.open` and every read method,
|
|
791
|
+
open options to `Calcsheet.open` only (a read method raises `ArgumentError` for them), write options to
|
|
792
|
+
`Calcsheet.write` and `Calcsheet.stream`.
|
|
793
|
+
|
|
794
|
+
Every value is checked when it is used: by the call that passes it, or for a configured value by the next
|
|
795
|
+
`Calcsheet.open`, `Calcsheet.write` or `Calcsheet.stream`, before anything is read or written; a book already open
|
|
796
|
+
keeps the options it was opened with. A wrong type or value raises `ArgumentError` naming the option. The zip-bomb
|
|
797
|
+
limits `max_uncompressed_size`, `max_zip_entries` (positive Integers) and `max_compression_ratio` (a positive number)
|
|
798
|
+
cannot be switched off; the other limits are positive Integers, or `nil` for none (`compression_ratio_floor` may also
|
|
799
|
+
be `0`); `batch_size` is a positive Integer and `sample_rows` a non-negative one; the booleans are `true` or `false`;
|
|
800
|
+
word and format lists are Arrays of non-empty Strings.
|
|
801
|
+
|
|
802
|
+
| Option | Default | Applies to | Meaning |
|
|
803
|
+
|---|---|---|---|
|
|
804
|
+
| `max_file_size` | `1_073_741_824` (1 GiB) | open | Bytes of the source; `nil` for no limit, which leaves an untrusted upload unbounded. |
|
|
805
|
+
| `max_rows` | `nil` | read | Rows one read may yield; `nil` for no limit. |
|
|
806
|
+
| `max_uncompressed_size` | `8_589_934_592` (8 GiB) | open | Bytes inflated from a zip archive, cumulative per book; always a positive Integer. |
|
|
807
|
+
| `max_compression_ratio` | `100` | open | Inflated / compressed bytes per zip entry, beyond the floor; always a positive number (a Float too). |
|
|
808
|
+
| `compression_ratio_floor` | `536_870_912` (512 MiB) | open | Bytes any zip entry may inflate to whatever its ratio, except the XLSX shared strings (a fixed 16 MiB); `nil` or `0` for none. |
|
|
809
|
+
| `max_zip_entries` | `10_000` | open | Entries of a zip archive; always a positive Integer. |
|
|
810
|
+
| `max_sheets` | `1_000` | open | Sheets of an XLSX or ODS workbook; `nil` for no limit. |
|
|
811
|
+
| `max_cell_bytes` | `1_048_576` (1 MiB) | open | Bytes of the text of one cell; `nil` for no limit, which also lifts the bound on what the XML parser buffers. See [limits](#limits). |
|
|
812
|
+
| `decimals` | `:bigdecimal` | read | `:bigdecimal` or `:float`: `:float` returns native fractional numbers, and fractional durations, as `Float`. |
|
|
813
|
+
| `time_zone` | `nil` | read, write | `nil` (UTC), `"UTC"` or `"Z"`, a UTC offset such as `"+01:00"` or `"+0100"`, or an object responding to `local` (and, for writing, to `utc_to_local` or `utc_offset`): see [time zones](#time-zones). |
|
|
814
|
+
| `decimal_separator` | `"."` | read, write CSV | `"."` or `","`: text read as `:decimal` or `:integer`; CSV output. |
|
|
815
|
+
| `thousands_separator` | `nil` | read | `nil`, `"."`, `","`, `" "`, `"'"`, `"\u00A0"` or `"\u202F"`, never the decimal separator. Groups the integer digits of text read as `:decimal` or `:integer` by three (`"1.234.567,89"`), or is absent (`"1234,5"`); text it groups otherwise (`"12.5"`) is invalid. |
|
|
816
|
+
| `date_formats` | `["%Y-%m-%d", "%d/%m/%Y"]` | read | `strptime` formats of text read as `:date` (and `:datetime`), with the directives `%Y` (a year of four digits) `%y %m %d %e %b %B %h %a %A %H %k %I %l %M %S %p %P %F %T %R %%`; others (`%z`, `%N`, `%s`, `%j`, `%G`, `%V`, flags such as `%-d`) raise `ArgumentError`. |
|
|
817
|
+
| `datetime_formats` | `["%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%d/%m/%Y %H:%M:%S", "%d/%m/%Y %H:%M"]` | read | `strptime` formats of text read as `:datetime` (and `:date`), with the directives of `date_formats`. |
|
|
818
|
+
| `true_words` | `["true", "1"]` | read | Text read as `true` by `:boolean`, case-insensitive. |
|
|
819
|
+
| `false_words` | `["false", "0"]` | read | Text read as `false` by `:boolean`, case-insensitive. |
|
|
820
|
+
| `batch_size` | `1000` | read | Rows read at a time, and the default size of `each_batch`. |
|
|
821
|
+
| `sample_rows` | `20` | read | Default `rows:` of `sample`. |
|
|
822
|
+
| `max_rows_per_sheet` | `1_048_576` | write | Rows per XLSX or ODS sheet, header included, before the overflow; can only lower the format's limit. |
|
|
823
|
+
| `strip_formula_prefix` | `false` | read CSV | Remove the `'` of formula protection. |
|
|
824
|
+
| `encoding` | `nil` | read CSV | Forces the encoding; `nil` detects it. |
|
|
825
|
+
| `col_sep` | `nil` | read, write CSV | One character other than `"`, CR or LF. Forces the separator when reading (`nil` detects it); `nil` writes `,` (`;` with `excel_it`). |
|
|
826
|
+
| `quote_char` | `nil` | read CSV | One character other than CR, LF or `col_sep`: forces the quote character; `nil` detects it. |
|
|
827
|
+
| `row_sep` | `"\r\n"` | write CSV | Line ending: `"\r\n"`, `"\n"` or `"\r"`. |
|
|
828
|
+
| `bom` | `nil` | write CSV | `true` writes a UTF-8 BOM; `nil` follows `excel_it`. |
|
|
829
|
+
| `date_format` | `nil` | write CSV | `strftime` format of dates; `nil`: `%Y-%m-%d` (`%d/%m/%Y` with `excel_it`). |
|
|
830
|
+
| `datetime_format` | `nil` | write CSV | `strftime` format of times; `nil`: `%Y-%m-%d %H:%M:%S` (`%d/%m/%Y %H:%M:%S` with `excel_it`). |
|
|
831
|
+
| `excel_it` | `false` | write CSV | The preset of `;`, BOM, decimal comma and `%d/%m/%Y` dates. |
|
|
832
|
+
| `sanitize_sheet_names` | `false` | write | Repair invalid sheet names and number duplicates instead of raising. |
|
|
833
|
+
|
|
834
|
+
## Concurrency
|
|
835
|
+
|
|
836
|
+
Call `Calcsheet.configure` (or `Calcsheet.config`) once at boot, before threads read or write: the configuration is
|
|
837
|
+
created lazily, and that first creation is not thread-safe. Afterwards every read and write takes its own frozen
|
|
838
|
+
snapshot of the options, and books are independent; a single book must not be shared between threads.
|
|
839
|
+
|
|
840
|
+
Ox parses while holding Ruby's global VM lock, so two large reads in two threads of one process take turns instead
|
|
841
|
+
of running in parallel (measured: 66 s for two reads that take 27 s each alone). Process large files in separate
|
|
842
|
+
worker processes, or with one worker thread per process, until a native backend that releases the lock exists.
|
|
843
|
+
|
|
844
|
+
## Performance
|
|
845
|
+
|
|
846
|
+
1,000,000 rows of 11 typed columns (id, phone, zero-padded postal code, names, email, city, decimal price, date,
|
|
847
|
+
datetime, boolean), Ruby 3.4, each case in its own process; peak RSS includes about 27 MB for the interpreter and
|
|
848
|
+
the gem. `bundle exec rspec --tag performance` runs these cases with their memory bounds.
|
|
849
|
+
|
|
850
|
+
| Case | Rows/s | Peak RSS |
|
|
851
|
+
|---|---:|---:|
|
|
852
|
+
| Read XLSX, inline strings | 43,000 | 32 MB |
|
|
853
|
+
| Read XLSX, shared strings (saved by LibreOffice, 70 MB) | 35,000 | 146 MB |
|
|
854
|
+
| Read ODS | 28,000 | 96 MB |
|
|
855
|
+
| Read CSV | 74,000 | 53 MB |
|
|
856
|
+
| Write XLSX | 63,000 | 31 MB |
|
|
857
|
+
| Write ODS | 61,000 | 31 MB |
|
|
858
|
+
| Write CSV | 87,000 | 29 MB |
|
|
859
|
+
|
|
860
|
+
Memory does not grow with the number of rows, apart from the XLSX shared strings table, whose size the file itself
|
|
861
|
+
decides.
|
|
862
|
+
|
|
863
|
+
## Compatibility
|
|
864
|
+
|
|
865
|
+
- Ruby 3.3 or later; tested on Ruby 3.3, 3.4 and 4.0.
|
|
866
|
+
- The suite reads XLSX and ODS files saved by LibreOffice and XLSX files saved by Excel for the web, and checks
|
|
867
|
+
every value; a workbook saved with the 1904 date system by Excel is still pending. Written files are checked by
|
|
868
|
+
reading them back with Calcsheet and with LibreOffice.
|
|
869
|
+
|
|
870
|
+
## Not supported
|
|
871
|
+
|
|
872
|
+
- XLS (Excel 97–2003): the only Ruby reader is GPL-licensed. Save the file as XLSX, ODS or CSV.
|
|
873
|
+
- Random access to cells (`sheet["B7"]`) and editing existing files while keeping their formatting.
|
|
874
|
+
- Writing formulas, rich formatting, charts, conditional formats, data validation, images.
|
|
875
|
+
- Merged reading of sheets without a header row.
|
|
876
|
+
|
|
877
|
+
Possible future work: a faster native reader as a separate, optional gem (see [ADR 007](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/docs/adr/007-optional-native-backend.md)); random access to rows
|
|
878
|
+
(`book.row(900_000)`) by inflating a sheet once into a temporary file and indexing the byte offset of each row
|
|
879
|
+
(about 8 MB per million rows), then seeking and parsing one row; editing an uploaded file while keeping everything
|
|
880
|
+
else (copying the untouched zip entries, rewriting only the changed cells with new strings inline, marking formulas
|
|
881
|
+
for recalculation, splitting ODS repeated cells) for "fix the errors in your file" flows; XLS, if a non-GPL reader
|
|
882
|
+
appears.
|
|
883
|
+
|
|
884
|
+
## Development
|
|
885
|
+
|
|
886
|
+
```sh
|
|
887
|
+
bundle exec rspec # the suite; slow and external-tool cases are excluded
|
|
888
|
+
bundle exec rspec --tag performance # 1M-row reads and writes with their time and memory bounds
|
|
889
|
+
bundle exec rspec --tag libreoffice # written files opened by LibreOffice headless (needs soffice)
|
|
890
|
+
bundle exec rubocop
|
|
891
|
+
bundle exec rake # the default task: the suite, then RuboCop
|
|
892
|
+
bundle exec yard stats --list-undoc # the public API left undocumented (options in .yardopts)
|
|
893
|
+
bundle exec yard # the API documentation, in tmp/yard
|
|
894
|
+
```
|
|
895
|
+
|
|
896
|
+
- Every fixture is synthetic (neutral names, `example.com` addresses); `spec/fixtures/README.md` says how each one
|
|
897
|
+
was produced. Large files are generated by the specs that need them and never committed.
|
|
898
|
+
- The code comments are YARD in RDoc markup (`+code+`). The public API is what `.yardopts` shows; implementation
|
|
899
|
+
classes and modules are tagged `@api private` (hidden by `--hide-api private`) and documented for maintainers.
|
|
900
|
+
- Runtime dependencies are pinned loosely in the gemspec (`~>`, or a bounded range such as `bigdecimal >= 3.1, < 5`); `Gemfile.lock` adopts only releases at least two
|
|
901
|
+
weeks old.
|
|
902
|
+
- CI (one pipeline per push: the merge request pipeline when one is open) runs:
|
|
903
|
+
- `rspec` on Ruby 3.3, 3.4 and 4.0, with a JUnit test report and a Cobertura coverage report that merge requests
|
|
904
|
+
show; it fails below 99% line or 98.5% branch coverage (checked only when `CI` is set, so focused local runs
|
|
905
|
+
are not);
|
|
906
|
+
- `rubocop`;
|
|
907
|
+
- `gem_build`, which checks the files of the built gem, and `gem_smoke`, which installs it on Ruby 3.3 outside the
|
|
908
|
+
bundle, with the locked runtime dependencies, and writes and reads a file in each format;
|
|
909
|
+
- `audit`: `bundle-audit` on the locked gems, allowed to fail;
|
|
910
|
+
- `deps_latest`, on schedules, merge requests and the default branch: the suite on the newest runtime dependencies
|
|
911
|
+
the gemspec allows, in a job-local copy of the Gemfile (`Gemfile.canary`, git-ignored), allowed to fail;
|
|
912
|
+
- `performance`, the `--tag performance` suite, by hand.
|
|
913
|
+
|
|
914
|
+
## Design decisions
|
|
915
|
+
|
|
916
|
+
The reasons behind these choices, with the benchmarks that shaped them, are recorded as architecture decision
|
|
917
|
+
records in [`docs/adr/`](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/tree/main/docs/adr) of the repository.
|
|
918
|
+
|
|
919
|
+
## License
|
|
920
|
+
|
|
921
|
+
Released under the [MIT License](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/LICENSE.txt).
|