calcsheet 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. checksums.yaml +7 -0
  2. data/.yardopts +9 -0
  3. data/CHANGELOG.md +62 -0
  4. data/LICENSE.txt +21 -0
  5. data/README.md +921 -0
  6. data/lib/calcsheet/book.rb +164 -0
  7. data/lib/calcsheet/cast.rb +578 -0
  8. data/lib/calcsheet/column.rb +117 -0
  9. data/lib/calcsheet/column_letter.rb +34 -0
  10. data/lib/calcsheet/configuration.rb +186 -0
  11. data/lib/calcsheet/decimal_text.rb +19 -0
  12. data/lib/calcsheet/entry_name.rb +28 -0
  13. data/lib/calcsheet/errors.rb +186 -0
  14. data/lib/calcsheet/excel_serial.rb +65 -0
  15. data/lib/calcsheet/format_detector.rb +109 -0
  16. data/lib/calcsheet/formula_protection.rb +31 -0
  17. data/lib/calcsheet/grid.rb +20 -0
  18. data/lib/calcsheet/header_keys.rb +66 -0
  19. data/lib/calcsheet/limited_inflater.rb +326 -0
  20. data/lib/calcsheet/merge_check.rb +41 -0
  21. data/lib/calcsheet/odf.rb +45 -0
  22. data/lib/calcsheet/option_validation.rb +160 -0
  23. data/lib/calcsheet/options.rb +10 -0
  24. data/lib/calcsheet/raw_row.rb +18 -0
  25. data/lib/calcsheet/raw_types.rb +15 -0
  26. data/lib/calcsheet/readers/csv_dialect.rb +264 -0
  27. data/lib/calcsheet/readers/csv_input.rb +128 -0
  28. data/lib/calcsheet/readers/csv_reader.rb +239 -0
  29. data/lib/calcsheet/readers/ods/cell_formats.rb +115 -0
  30. data/lib/calcsheet/readers/ods/content_parser.rb +594 -0
  31. data/lib/calcsheet/readers/ods/count.rb +30 -0
  32. data/lib/calcsheet/readers/ods/cursor.rb +121 -0
  33. data/lib/calcsheet/readers/ods/patterns.rb +70 -0
  34. data/lib/calcsheet/readers/ods/row_seek.rb +157 -0
  35. data/lib/calcsheet/readers/ods/scanner.rb +302 -0
  36. data/lib/calcsheet/readers/ods/style_scan.rb +108 -0
  37. data/lib/calcsheet/readers/ods/tag_text.rb +76 -0
  38. data/lib/calcsheet/readers/ods_reader.rb +71 -0
  39. data/lib/calcsheet/readers/ox_errors.rb +40 -0
  40. data/lib/calcsheet/readers/references.rb +62 -0
  41. data/lib/calcsheet/readers/row_batches.rb +45 -0
  42. data/lib/calcsheet/readers/sax_sheet.rb +108 -0
  43. data/lib/calcsheet/readers/stream_remainder.rb +66 -0
  44. data/lib/calcsheet/readers/tag_scan.rb +119 -0
  45. data/lib/calcsheet/readers/xlsx/escapes.rb +36 -0
  46. data/lib/calcsheet/readers/xlsx/local_name.rb +54 -0
  47. data/lib/calcsheet/readers/xlsx/number_format.rb +89 -0
  48. data/lib/calcsheet/readers/xlsx/row_skipper.rb +162 -0
  49. data/lib/calcsheet/readers/xlsx/shared_strings.rb +209 -0
  50. data/lib/calcsheet/readers/xlsx/sheet_parser.rb +435 -0
  51. data/lib/calcsheet/readers/xlsx/styles.rb +156 -0
  52. data/lib/calcsheet/readers/xlsx/workbook_meta.rb +228 -0
  53. data/lib/calcsheet/readers/xlsx_reader.rb +122 -0
  54. data/lib/calcsheet/reading.rb +281 -0
  55. data/lib/calcsheet/row.rb +110 -0
  56. data/lib/calcsheet/sample.rb +10 -0
  57. data/lib/calcsheet/schema.rb +140 -0
  58. data/lib/calcsheet/sheet_info.rb +21 -0
  59. data/lib/calcsheet/sheet_names.rb +114 -0
  60. data/lib/calcsheet/sheet_selection.rb +58 -0
  61. data/lib/calcsheet/source.rb +263 -0
  62. data/lib/calcsheet/stream_body.rb +81 -0
  63. data/lib/calcsheet/time_zone.rb +96 -0
  64. data/lib/calcsheet/untrusted.rb +32 -0
  65. data/lib/calcsheet/version.rb +6 -0
  66. data/lib/calcsheet/write_book.rb +202 -0
  67. data/lib/calcsheet/write_sheet.rb +282 -0
  68. data/lib/calcsheet/writers/csv_writer.rb +138 -0
  69. data/lib/calcsheet/writers/ods_writer.rb +437 -0
  70. data/lib/calcsheet/writers/xlsx_writer.rb +350 -0
  71. data/lib/calcsheet/writers/xml_text.rb +35 -0
  72. data/lib/calcsheet/writers/zip_buffered.rb +55 -0
  73. data/lib/calcsheet/zip_container.rb +190 -0
  74. data/lib/calcsheet/zip_directory.rb +202 -0
  75. data/lib/calcsheet/zip_end_record.rb +117 -0
  76. data/lib/calcsheet.rb +295 -0
  77. metadata +200 -0
data/README.md ADDED
@@ -0,0 +1,921 @@
1
+ # Calcsheet
2
+
3
+ Streaming CSV, XLSX and ODS reader and writer for Ruby, with typed cells and no dependency on Rails.
4
+
5
+ Calcsheet reads and writes spreadsheets one row at a time, so a file of a million rows costs tens of megabytes of
6
+ memory, not gigabytes. It is built for imports and exports of large lists:
7
+
8
+ - **Exact values.** Decimals are `BigDecimal` built from the cell text, integers are `Integer`, dates are `Date`,
9
+ datetimes are `Time` with the wall clock of the file. Text is never turned into numbers: `"+393331234567"` and
10
+ `"00013"` survive every round trip.
11
+ - **Row numbers the user sees.** Error reports say "sheet 'Contacts (2)', row 4812, column C", exactly where Excel
12
+ or LibreOffice shows the cell.
13
+ - **A two-phase import flow.** Sample the first rows, let the user map the columns, then process the file in
14
+ batches, collecting cell errors instead of stopping at the first one.
15
+ - **Streaming output.** Write to a file, to any IO (sockets and other non-seekable ones included), or straight into
16
+ an HTTP response as a Rack body. Sheets longer than the format allows continue in "Name (2)", "Name (3)"...
17
+ - **Defensive limits** for untrusted uploads: file size, rows, zip bombs, entries and sheets.
18
+
19
+ ## Contents
20
+
21
+ - [Installation](#installation)
22
+ - [Reading](#reading)
23
+ - [Writing](#writing)
24
+ - [Types](#types)
25
+ - [Row numbers](#row-numbers)
26
+ - [Errors](#errors)
27
+ - [Limits](#limits)
28
+ - [CSV](#csv)
29
+ - [XLSX and ODS](#xlsx-and-ods)
30
+ - [Streams and non-seekable input](#streams-and-non-seekable-input)
31
+ - [Time zones](#time-zones)
32
+ - [Configuration](#configuration)
33
+ - [Options reference](#options-reference)
34
+ - [Concurrency](#concurrency)
35
+ - [Performance](#performance)
36
+ - [Compatibility](#compatibility)
37
+ - [Not supported](#not-supported)
38
+ - [Development](#development)
39
+ - [Design decisions](#design-decisions)
40
+
41
+ ## Installation
42
+
43
+ Calcsheet will be published on [rubygems.org](https://rubygems.org/gems/calcsheet); its Git tag is created at the
44
+ release. Once it is released, add it to your Gemfile:
45
+
46
+ ```rb
47
+ gem "calcsheet", "~> 0.1"
48
+ ```
49
+
50
+ The runtime dependencies are `ox` (a SAX parser with a C extension), `rubyzip`, `zip_kit`, `bigdecimal` and `csv`.
51
+ Nothing needs Rails or ActiveSupport.
52
+
53
+ ## Reading
54
+
55
+ ### Quick start
56
+
57
+ The typical import has two phases: show the beginning of the file so that the user can map the columns, then read
58
+ everything with that mapping, in batches, keeping the rows that have errors aside.
59
+
60
+ ```ruby
61
+ require "calcsheet"
62
+
63
+ Calcsheet.open("contacts.xlsx") do |book|
64
+ book.format # => :xlsx
65
+ book.sheets.map(&:name) # => ["Contacts", "Contacts (2)"]
66
+
67
+ # 1. The header and the first rows of each visible sheet, with native values.
68
+ sample = book.sample(rows: 5).first
69
+ sample.header.values # => ["Phone", "Name", "Email", "Postal code", "Price", "Created at"]
70
+ sample.rows.first.values # => ["+393331234567", "Ada", "ada@example.com", "00013", BigDecimal("12.5"), Time.utc(2026, 1, 1, 10, 0, 0)]
71
+
72
+ # 2. Can the sheets be read as one? Every non-empty sheet must have the same header.
73
+ book.merge_check.mergeable? # => true
74
+ end
75
+ ```
76
+
77
+ The user's mapping becomes a column schema: each key names a column, by 0-based position (`at:`) or by header name
78
+ (`header:`, compared ignoring case and whitespace), with an optional declared type and `required:` flag.
79
+
80
+ ```ruby
81
+ require "calcsheet"
82
+
83
+ columns = {
84
+ phone: { at: 0, type: :string },
85
+ email: { header: "email", required: true },
86
+ postal_code: { header: "postal code", type: :string },
87
+ price: { at: 4, type: :decimal },
88
+ created_at: { at: 5, type: :datetime }
89
+ }
90
+
91
+ imported = []
92
+ rejected = []
93
+ Calcsheet.open("contacts.xlsx") do |book|
94
+ book.each_batch(size: 1000, sheet: :all, columns: columns) do |rows|
95
+ valid, invalid = rows.partition(&:valid?)
96
+ imported.concat(valid.map(&:to_h)) # for example, insert_all into a table
97
+ rejected.concat(invalid.flat_map(&:errors)) # report them to the user
98
+ end
99
+ end
100
+
101
+ imported.size # => 2
102
+ imported.first # => { phone: "+393331234567", email: "ada@example.com", postal_code: "00013", price: BigDecimal("12.5"), created_at: Time.utc(2026, 1, 1, 10, 0, 0) }
103
+ rejected.map(&:message) # => ["sheet 'Contacts (2)', row 2, column C (email): required"]
104
+ rejected.map { |error| [error.sheet, error.row, error.column_letter, error.key, error.reason] } # => [["Contacts (2)", 2, "C", :email, :required]]
105
+ ```
106
+
107
+ ### Opening a file
108
+
109
+ ```rb
110
+ Calcsheet.open(source, format: nil, **options) { |book| ... } # yields the book, closes it, returns the block's value
111
+ book = Calcsheet.open(source, **options) # without a block the caller must call book.close
112
+ ```
113
+
114
+ - `source` is a path (`String` or `Pathname`) or an IO: a `File`, a `Tempfile`, a `StringIO`, an upload object, a
115
+ pipe. A seekable IO is read from its start whatever its position (an upload already read once for a preview is
116
+ fine). An IO given by the caller is never closed. An upload object is named after its `original_filename`,
117
+ whether it seeks itself (Rack's `Rack::Multipart::UploadedFile`, which delegates to its Tempfile) or not; one that
118
+ cannot seek but answers `to_io` (Rails' `ActionDispatch::Http::UploadedFile`) is read through the file `to_io`
119
+ returns, without a copy.
120
+ - The format is detected from the content: a zip with `xl/workbook.xml` is XLSX, a zip whose `mimetype` is
121
+ `application/vnd.oasis.opendocument.spreadsheet` is ODS, anything else is CSV. The extension only breaks a tie.
122
+ `format: :csv`, `:xlsx` or `:ods` skips detection. An XLS (Excel 97–2003) file, an encrypted (password-protected)
123
+ workbook, which is an OLE2 container like XLS, or another kind of zip raises `Calcsheet::UnsupportedFormatError`.
124
+ Entry names are compared with a leading `/` removed and `\` read as `/`, in detection as in reading; two entries
125
+ of the same name, and an archive comment that holds an end of central directory signature (the zip library would
126
+ read it as the record), raise `MalformedFileError`.
127
+ - `options` are any keys of the [options reference](#options-reference), for this book.
128
+ - `book.format` is `:csv`, `:xlsx` or `:ods`; `book.sheets` lists every sheet in workbook order as
129
+ `Calcsheet::SheetInfo` (`name`, `index`, `hidden?`), hidden ones included. A CSV file has one sheet, named after
130
+ the file without its directories and extension (an upload's `original_filename`, Windows separators included;
131
+ `"csv"` for an IO without a path).
132
+
133
+ ### Rows
134
+
135
+ ```rb
136
+ book.rows(**read) # => a lazy Enumerator of Calcsheet::Row
137
+ book.each_row(**read) { |row| ... }
138
+ book.each_batch(size: 1000, **read) { |rows| ... }
139
+ book.sample(rows: 20, sheets: :all, include_hidden: false) # => [Calcsheet::Sample(sheet:, header:, rows:)]
140
+ book.merge_check(sheets: :all, include_hidden: false) # => Calcsheet::MergeCheck(sheet_count:, mismatches:)
141
+ ```
142
+
143
+ `rows` reads one batch at a time while it is iterated: `book.rows.first(10)` reads only the beginning of the
144
+ sheet. The batch size never changes what is yielded. `sample` returns, for each selected sheet, its header row
145
+ (`nil` for an empty sheet) and its first data rows, reading only the beginning of each sheet.
146
+
147
+ The read options (`**read`) are these, plus any key of the [options reference](#options-reference) as an override
148
+ for that read (for example `time_zone:` or `decimal_separator:`), except the options that apply when the file is
149
+ opened: `max_file_size`, `max_uncompressed_size`, `max_compression_ratio`, `compression_ratio_floor`,
150
+ `max_zip_entries`, `max_sheets` and `max_cell_bytes` raise `ArgumentError` in a read; pass them to `Calcsheet.open`.
151
+ `sample` and `merge_check` take `include_hidden:` too.
152
+
153
+ | Read option | Default | Meaning |
154
+ |---|---|---|
155
+ | `sheet:` | first visible sheet | An index, a name, an Array of them, or `:all` (every visible sheet; like the default, the first sheet when every sheet is hidden). Several sheets are read as one: see [merged sheets](#merged-sheets). |
156
+ | `header:` | `true` | The first non-blank row is the header and is not yielded. `false` yields every row. |
157
+ | `columns:` | `nil` | A [column schema](#column-schema). Without one every column is returned with its native value. |
158
+ | `types:` | `:native` | `:string` reads every column without a declared type as text. |
159
+ | `on_error:` | `:collect` | `:collect` attaches cell errors to the row; `:raise` raises the first one as `Calcsheet::CellError`. |
160
+ | `start_at:` | `nil` | Resume an interrupted job: a row number, or `{ sheet:, row: }` when reading several sheets. |
161
+ | `skip_blank:` | `true` | Rows without a non-blank cell are not yielded; row numbers keep the gaps. |
162
+ | `include_hidden:` | `false` | `sheet: :all` includes the hidden sheets. A hidden sheet is otherwise read only when named or indexed. |
163
+ | `required_headers:` | `nil` | Header names that must be present (ignoring case and whitespace), checked before the first row. |
164
+
165
+ Each `Calcsheet::Row` has:
166
+
167
+ - `sheet` (the sheet name), `sheet_index`, `number` (the row number the user sees, see [row numbers](#row-numbers))
168
+ and `line` (CSV only: the physical line where the record starts);
169
+ - `values`: in file order, or in schema order with a schema;
170
+ - `row[index]`, `row[key]` and `to_h`: keys are the schema keys, or without a schema the
171
+ header texts stripped of surrounding whitespace, non-breaking spaces included (a numeric header by its plain
172
+ text: `"2.5"`, `"10"`). A blank header cell becomes its
173
+ column letter (`"C"`), a repeated header gets a counter (`"Phone"`, `"Phone (2)"`), and cells beyond the header are
174
+ keyed by their column letter, with the same counter when a header has that text (`"D (2)"` after a header `"D"`,
175
+ which `row["D"]` keeps meaning);
176
+ - `errors` (an Array of `Calcsheet::CellError`) and `valid?`.
177
+
178
+ Without a schema, `values` are as long as the row with trailing blank cells trimmed; in header mode they are padded
179
+ with `nil` to the width of the header, and cells beyond the header are kept.
180
+
181
+ ```ruby
182
+ require "calcsheet"
183
+
184
+ Calcsheet.open("contacts.xlsx") do |book|
185
+ row = book.rows.first
186
+ row.number # => 2
187
+ row[0] # => "+393331234567"
188
+ row["Postal code"] # => "00013"
189
+ row.to_h.keys # => ["Phone", "Name", "Email", "Postal code", "Price", "Created at"]
190
+
191
+ # Resume at row 3 of "Contacts": the rest of that sheet, then the following sheets.
192
+ book.rows(sheet: :all, start_at: { sheet: "Contacts", row: 3 }).map { |r| [r.sheet, r.number] } # => [["Contacts", 3], ["Contacts (2)", 2]]
193
+ end
194
+ ```
195
+
196
+ `start_at:` skips without parsing where the format allows it: XLSX byte-scans for the `r` attribute of the rows, ODS
197
+ counts row start tags, CSV parses the records it skips (quotes make byte scanning unsafe) but builds no rows. The
198
+ byte scans pass over comments, CDATA sections and processing instructions, and assume that the XML before their
199
+ target is well-formed. A `<` that starts no complete tag before a row, a sheet or a markup opening usually raises
200
+ `MalformedFileError` (a tag cut by a chunk boundary may be passed over unchecked); but on other malformed input, such as a complete tag inside an attribute value, the scans
201
+ (`start_at:`, and for ODS the sheet listing) may disagree with a full read, misnumbering rows or missing a sheet.
202
+ They never allocate beyond their bounds or raise errors other than Calcsheet's (ADR 008, in the [repository](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/docs/adr/008-limits-for-untrusted-files.md)).
203
+
204
+ ### Column schema
205
+
206
+ ```rb
207
+ columns: {
208
+ phone: { at: 1, type: :string }, # by 0-based position: the primary use
209
+ first_name: { at: 3 }, # native type
210
+ price: { header: "Price", type: :decimal }, # by header name
211
+ active: { at: 10, type: :boolean, required: true }
212
+ }
213
+ ```
214
+
215
+ - Mapping by position is the primary mode, since header names may repeat. `at:` is an Integer from 0 to 16,383
216
+ (the last spreadsheet column, XFD), checked when the schema is built. Mapping by header name compares names
217
+ downcased with all whitespace removed; a name that matches no column raises `Calcsheet::ColumnNotFoundError`, one
218
+ that matches several raises `Calcsheet::AmbiguousColumnError`, both before the first row. `header:` mapping needs
219
+ `header: true`.
220
+ - Columns the schema does not name are dropped from `values` and `to_h`. Keys are Symbols (a String key is
221
+ converted), so `:a` and `"a"` together raise `ArgumentError`; several keys may map one position only with the same
222
+ type.
223
+ - `type:` is one of `:string`, `:integer`, `:decimal`, `:date`, `:datetime`, `:boolean` (see
224
+ [declared types](#declared-types)); without it the value is native.
225
+ - `required: true` makes a blank cell (`nil` or whitespace only) a cell error with reason `:required`. A
226
+ whitespace-only cell reads as `nil` for every declared type but `:string`, in every position, never as a cast
227
+ error; `:string` and native values keep the text as it is.
228
+ - The header row is always read with native values: declared types apply to data rows only.
229
+
230
+ ### Merged sheets
231
+
232
+ Reading several sheets (`sheet: :all`, or an Array) yields them as one stream of rows:
233
+
234
+ - it requires `header: true`;
235
+ - before the first row is yielded, the headers are compared with the normalization of header mapping, trailing
236
+ blank header cells ignored; completely empty sheets are skipped. A difference raises
237
+ `Calcsheet::HeaderMismatchError`, whose `mismatches` are `{ sheet:, expected:, actual: }` with the full header
238
+ rows of the first non-empty sheet and of the differing one;
239
+ - the header rows of the later sheets are not yielded, and row numbers are per sheet.
240
+
241
+ `book.merge_check` runs the same comparison without reading the rows: `mergeable?`, `mismatches` and `sheet_count`
242
+ (the sheets checked, empty ones included). A workbook with one sheet, and every CSV file, is trivially mergeable.
243
+
244
+ Hidden sheets often hold lookup data, so `:all` means the visible sheets. The default sheet is the first visible
245
+ one; a hidden sheet is read when it is named or indexed, or with `include_hidden: true`.
246
+
247
+ ### Cell errors
248
+
249
+ With `on_error: :collect` (the default) a cell that cannot be converted, or a blank `required:` cell, becomes `nil`
250
+ in `values` and a `Calcsheet::CellError` in `row.errors`, and reading goes on. With `on_error: :raise` the first one
251
+ is raised. Problems of the file itself (damaged, too large, missing sheet) always raise.
252
+
253
+ ```ruby
254
+ require "calcsheet"
255
+
256
+ Calcsheet.open("contacts.csv") do |book|
257
+ columns = { phone: { at: 0 }, price: { header: "Price", type: :decimal }, created_at: { at: 3, type: :datetime } }
258
+ first, second = book.rows(columns: columns, decimal_separator: ",").to_a
259
+
260
+ first.to_h # => { phone: "+393331234567", price: BigDecimal("12.50"), created_at: Time.utc(2026, 1, 1, 10, 0, 0) }
261
+ second[:price] # => nil
262
+ second.errors.map(&:message) # => ["sheet 'contacts', row 3, column C (price): invalid decimal \"n/a\""]
263
+
264
+ begin
265
+ book.each_row(columns: columns, decimal_separator: ",", on_error: :raise) { |row| row }
266
+ rescue Calcsheet::CellError => error
267
+ error.row # => 3
268
+ error.value # => "n/a"
269
+ error.reason # => :invalid_decimal
270
+ end
271
+ end
272
+ ```
273
+
274
+ ## Writing
275
+
276
+ ### To a file
277
+
278
+ ```ruby
279
+ require "calcsheet"
280
+
281
+ Calcsheet.write("price-list.xlsx") do |book| # the format comes from the extension: .csv, .xlsx or .ods
282
+ book.sheet("Prices", columns: [
283
+ { header: "Code", type: :string, width: 12 },
284
+ { header: "Price", type: :decimal, scale: 2 },
285
+ { header: "Valid from", type: :date },
286
+ { header: "Updated at", type: :datetime },
287
+ { header: "Active", type: :boolean },
288
+ { header: "Note", key: :note }
289
+ ]) do |sheet|
290
+ sheet << ["00013", BigDecimal("12.3456"), Date.new(2026, 1, 1), Time.utc(2026, 1, 1, 10, 0, 0), true, "=not a formula"]
291
+ sheet << { "Code" => "00014", "Price" => 7, note: "a Hash row" } # keyed by header, or by key:; missing keys are empty
292
+ end
293
+ end
294
+
295
+ Calcsheet.open("price-list.xlsx") do |book|
296
+ row = book.rows.first
297
+ row["Code"] # => "00013"
298
+ row["Price"] # => BigDecimal("12.3456")
299
+ row["Note"] # => "=not a formula"
300
+ end
301
+ ```
302
+
303
+ `Calcsheet.write(target, format: nil, **options) { |book| ... }` writes a path (format from the extension, written in
304
+ binary mode and closed) or any object responding to `write` or `<<` (`format:` required). A path is written to a
305
+ temporary file in the same directory (that of the file a symlink points to, which a dangling symlink gets created),
306
+ renamed over the target once the block has completed, so an existing file stays untouched until then and keeps its
307
+ permissions, though not its owner and group; when the block raises, the temporary file is deleted and the target is
308
+ unchanged. A target that cannot be written raises `Errno::EACCES`, as opening it would. A target that is not a regular file (a FIFO, `/dev/null`), or
309
+ one in a directory that is not writable, is written directly instead: then a failed block leaves what was already
310
+ written. An IO keeps what was already written, and is never closed.
311
+
312
+ `book.sheet(name, columns:, header: true) { |sheet| ... }` writes one sheet (`header:` is `true` or `false`); sheets
313
+ are written one after another, and a finished sheet cannot be reopened. Each column is a Hash:
314
+
315
+ | Key | Default | Meaning |
316
+ |---|---|---|
317
+ | `header:` | required | The header text, and the key of Hash rows. |
318
+ | `type:` | `:string` | `:string`, `:integer`, `:decimal`, `:date`, `:datetime` or `:boolean`. |
319
+ | `width:` | `nil` | Column width in characters (XLSX and ODS): a number above 0 and at most 255. |
320
+ | `scale:` | `nil` | Decimals shown by XLSX and ODS for a `:decimal` column. Display only: see below. |
321
+ | `key:` | `nil` | The key of the column in Hash rows, instead of the header. |
322
+ | `formula_protection:` | `true` | CSV only: see [formula protection](#formula-protection). |
323
+
324
+ `sheet << row` takes an Array in column order or a Hash (an unknown key raises `ArgumentError`). Every value must
325
+ fit its column type, otherwise `Calcsheet::WriteError` names the sheet, row and column, nothing of that row is
326
+ written and the sheet stays usable:
327
+
328
+ | Column type | Accepted values |
329
+ |---|---|
330
+ | `:string` | a `String` (converted to UTF-8) |
331
+ | `:integer` | an `Integer` |
332
+ | `:decimal` | an `Integer` or a finite `BigDecimal`; a `Float` is refused, so that no binary rounding slips in |
333
+ | `:date` | a `Date` (not a `DateTime`) |
334
+ | `:datetime` | a `Time`, written in whole seconds (rounded) |
335
+ | `:boolean` | `true` or `false` |
336
+
337
+ `nil` writes an empty cell. A `:string` column writes text even for `"00013"` or `"+39..."`. In every format a number
338
+ is written with all its digits, so one whose text would exceed 1,024 characters (the most the readers read, see
339
+ [limits](#limits)) raises `WriteError`. So does a text longer than `max_cell_bytes` (1 MiB by default; `nil` for no
340
+ limit), which the readers would refuse, counting the `'` CSV puts before a formula. A written header longer than
341
+ `max_cell_bytes` (counted the same way) raises `WriteError` from `book.sheet`, before anything of that sheet is
342
+ written.
343
+
344
+ **`scale:` is display only.** XLSX and ODS store the full value and show it with `scale` decimals; CSV ignores
345
+ `scale` and writes the full value. Calcsheet never rounds a written decimal: round the values yourself before
346
+ writing them if the file must hold rounded values. Numbers with more than 15 significant digits are written as
347
+ given: Calcsheet reads integers and numbers of more than 17 significant digits back exactly, rounds fractions of 16
348
+ or 17 to 15, and Excel keeps only 15.
349
+
350
+ ### To an IO
351
+
352
+ ```ruby
353
+ require "calcsheet"
354
+ require "stringio"
355
+
356
+ io = StringIO.new
357
+ Calcsheet.write(io, format: :csv, excel_it: true) do |book|
358
+ book.sheet("Contacts", columns: [{ header: "Phone" }, { header: "Price", type: :decimal }]) do |sheet|
359
+ sheet << ["+393331234567", BigDecimal("12.5")]
360
+ end
361
+ end
362
+ io.string # => "\uFEFFPhone;Price\r\n'+393331234567;12,5\r\n"
363
+ ```
364
+
365
+ The IO may be non-seekable (a socket, a pipe): no format needs to seek its output. A background job can write a
366
+ `Tempfile` and attach it elsewhere.
367
+
368
+ ### As an HTTP response
369
+
370
+ `Calcsheet.stream(format:, **options) { |book| ... }` returns a Rack body: its `each` runs the block and yields the
371
+ file in binary chunks of at least 64 KiB (the last one may be shorter) while the rows are produced, and memory
372
+ stays flat. The first chunk leaves once 64 KiB of output exist: for CSV after about 64 KiB of rows, for XLSX and
373
+ ODS after the compressor has produced 64 KiB of deflated data, which takes several times more rows. Options and format are checked when the body is built, before any byte.
374
+ The size is not known in advance (chunked transfer). Each call of `each` writes the whole file again; an exception in the block propagates
375
+ from `each` after the chunks already sent, so the client receives a truncated file.
376
+
377
+ ```ruby
378
+ require "calcsheet"
379
+
380
+ body = Calcsheet.stream(format: :xlsx) do |book|
381
+ book.sheet("Contacts", columns: [{ header: "Email" }]) do |sheet|
382
+ %w[ada@example.com grace@example.com].each { |email| sheet << [email] }
383
+ end
384
+ end
385
+ headers = {
386
+ "content-type" => "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
387
+ "content-disposition" => %(attachment; filename="contacts.xlsx")
388
+ }
389
+ app = ->(_env) { [200, headers, body] } # a Rack application
390
+
391
+ # What the server does with the body:
392
+ _status, _headers, chunks = app.call({})
393
+ File.open("export.xlsx", "wb") { |file| chunks.each { |chunk| file.write(chunk) } }
394
+ Calcsheet.open("export.xlsx") { |book| book.rows.map { |row| row["Email"] } } # => ["ada@example.com", "grace@example.com"]
395
+ ```
396
+
397
+ In a Rails controller, assign the body to `response_body`:
398
+
399
+ ```rb
400
+ class ContactsController < ApplicationController
401
+ def export
402
+ headers["Content-Type"] = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
403
+ headers["Content-Disposition"] = %(attachment; filename="contacts.xlsx")
404
+ self.response_body = Calcsheet.stream(format: :xlsx) do |book|
405
+ book.sheet("Contacts", columns: [{ header: "Email" }, { header: "Created at", type: :datetime }]) do |sheet|
406
+ Contact.find_each { |contact| sheet << [contact.email, contact.created_at] }
407
+ end
408
+ end
409
+ end
410
+ end
411
+ ```
412
+
413
+ The block runs while the server sends the response, after the action returns. A middleware that buffers bodies (to
414
+ compute an ETag or a Content-Length, for example) defeats the streaming: check the middleware stack.
415
+
416
+ ### Sheet names and overflow
417
+
418
+ Sheet names are validated for every format: 1 to 31 characters, counted as Excel counts them (a character beyond
419
+ the Basic Multilingual Plane, such as an emoji, counts twice), none of `[ ] : * ? / \` and no control characters,
420
+ not starting or ending with an apostrophe, and unique ignoring case. An invalid name raises `ArgumentError`, unless
421
+ `sanitize_sheet_names: true` replaces the forbidden characters with `_`, shortens the name (never inside a character
422
+ or a combining sequence) and numbers duplicates (`"Name (2)"`).
423
+
424
+ XLSX and ODS hold at most 1,048,576 rows per sheet. When a sheet reaches the limit (header included), the writer
425
+ continues in a new sheet named after the first with a counter (`"Contacts"`, `"Contacts (2)"`, `"Contacts (3)"`,
426
+ shortening the base name to keep 31 characters) and repeats the header (a row that would be refused opens no
427
+ continuation); the block does not notice, and `sheet.name` tells the current one. `max_rows_per_sheet:` lowers the
428
+ limit; a value above the format's limit raises `ArgumentError` before anything is written. Raising instead of continuing is not offered: in streaming the overflow
429
+ is discovered after a million rows were sent. A CSV file has no row limit and holds a single sheet: a second
430
+ `book.sheet` raises `Calcsheet::WriteError`.
431
+
432
+ ```ruby
433
+ require "calcsheet"
434
+
435
+ Calcsheet.write("long.ods", max_rows_per_sheet: 3) do |book|
436
+ book.sheet("Contacts", columns: [{ header: "Id", type: :integer }]) do |sheet|
437
+ (1..5).each { |id| sheet << [id] }
438
+ end
439
+ end
440
+
441
+ Calcsheet.open("long.ods") do |book|
442
+ book.sheets.map(&:name) # => ["Contacts", "Contacts (2)", "Contacts (3)"]
443
+ book.rows(sheet: :all).map { |row| row["Id"] } # => [1, 2, 3, 4, 5]
444
+ end
445
+ ```
446
+
447
+ ## Types
448
+
449
+ ### Native values
450
+
451
+ Without a declared type a cell reads as:
452
+
453
+ | Cell | Value |
454
+ |---|---|
455
+ | CSV field | `String`, never guessed; an empty field is `nil` |
456
+ | XLSX/ODS text | `String` |
457
+ | XLSX/ODS number, integral | `Integer` (`BigDecimal` beyond 30 digits) |
458
+ | XLSX/ODS number, fractional | `BigDecimal`; `decimals: :float` returns `Float` |
459
+ | XLSX number with a date format, ODS date without time | `Date` |
460
+ | XLSX number with a date and time format, ODS date with time | `Time` (see [time zones](#time-zones)) |
461
+ | XLSX number with a time-only or duration format (`hh:mm`, `[h]:mm:ss`), ODS time | `Integer` seconds (`BigDecimal` when fractional; `Float` with `decimals: :float`) |
462
+ | boolean | `true` or `false` |
463
+ | error (`#DIV/0!`...) | `nil`, with a cell error of reason `:cell_error` |
464
+ | formula | its cached value; `nil` when the file caches none |
465
+ | merged range | the value in the top-left cell, `nil` in the others |
466
+
467
+ - XLSX stores numbers as text with up to 17 significant digits; Excel itself keeps 15. Calcsheet builds the
468
+ `BigDecimal` from the text, never through `Float`, and rounds it half up to 15 significant digits only when the
469
+ text is the full print of a double: a fraction or an exponent with 16 or 17 significant digits
470
+ (`0.30000000000000004` reads as `0.3`). Integer text is always exact (`9007199254740993`, `2**64`), and so is text
471
+ with more than 17 significant digits. A `BigDecimal` costs about 160 bytes more than a `Float`; that matters only
472
+ when millions of values are kept in memory.
473
+ - The XML character references of XLSX text and sheet names and of ODS text, cell attributes and sheet names
474
+ (`&amp;`, `&#65;`) are decoded by Calcsheet, CDATA sections are kept as written: a reference to NUL, a surrogate or
475
+ a code point beyond U+10FFFF, an unknown entity and an `&` that starts no reference raise `MalformedFileError`
476
+ (the XML parser would cut or change the text).
477
+ - Date or datetime is decided by the number format of the cell, never by the value: a datetime at midnight stays a
478
+ `Time`. XLSX built-in formats 14–17 and 27–36 and 50–58 are dates, 22 a datetime, 18–21 and 45–47 times and
479
+ durations; custom codes are classified by their date and time tokens outside quotes and brackets. ODS cells
480
+ follow their data style the same way. The 1904 date system is honored.
481
+ - A number under a plain zero-padding format (`00000`) gets its leading zeros back when the column is declared
482
+ `:string` (`13` becomes `"00013"`).
483
+
484
+ ### Declared types
485
+
486
+ A schema `type:` converts from any representation; a value that does not convert is a cell error with the reason
487
+ shown.
488
+
489
+ | Type | From a number, date or boolean cell | From text (CSV, text cells) | Reasons |
490
+ |---|---|---|---|
491
+ | `:string` | plain digits without exponent (`393331234567`, `0.0101`); dates and datetimes in ISO 8601, times of day as `HH:MM:SS`; `"true"`/`"false"` | the text | |
492
+ | `:integer` | integral numbers only; booleans as `1`/`0` | `"12"`, `"12.0"`, with `decimal_separator:` and `thousands_separator:` | `:invalid_integer`, `:not_integral` |
493
+ | `:decimal` | the exact `BigDecimal`, also for integral numbers (`7` gives `BigDecimal("7")`) | `decimal_separator:` (`"."` or `","`), optional `thousands_separator:`, surrounding whitespace ignored | `:invalid_decimal` |
494
+ | `:date` | date and datetime cells (a datetime is truncated) | `date_formats:`, then `datetime_formats:` | `:invalid_date` |
495
+ | `:datetime` | datetime and date cells (a date gives midnight) | `datetime_formats:`, then `date_formats:` | `:invalid_datetime` |
496
+ | `:boolean` | boolean cells; the numbers `1` and `0` | `true_words:` / `false_words:`, case-insensitive | `:invalid_boolean` |
497
+
498
+ A number with a general (non-date) format declared `:date` or `:datetime` is a cell error: reading it as a date
499
+ serial would silently turn years or ids into dates of 1905. Text dates match a format only as a whole and only with
500
+ a real calendar date. A blank `required:` cell has reason `:required`, an error cell `:cell_error`.
501
+
502
+ ## Row numbers
503
+
504
+ `row.number` is the row the user sees in the application that opens the file:
505
+
506
+ - **XLSX**: the `r` attribute of the row; when a generator leaves it out, rows are counted.
507
+ - **ODS**: table rows counted, repeated rows (`number-rows-repeated`) included.
508
+ - **CSV**: the 1-based **record** number, counting the header record and blank lines, because Excel and LibreOffice
509
+ show a quoted multi-line field as one row. `row.line` gives the physical line where the record starts, for text
510
+ editors.
511
+
512
+ Header, blank and skipped rows keep their numbers, so the numbers have gaps instead of shifting. In merged reading
513
+ the number is per sheet. `CellError#row`, `MalformedFileError#row` and `WriteError#row` use the same numbering.
514
+
515
+ ```ruby
516
+ require "calcsheet"
517
+ require "stringio"
518
+
519
+ csv = StringIO.new(%(Name,Note\r\nAda,"two\r\nlines"\r\n\r\nGrace,one line\r\n))
520
+ Calcsheet.open(csv) do |book|
521
+ book.rows.map { |row| [row.number, row.line] } # => [[2, 2], [4, 5]]
522
+ end
523
+ ```
524
+
525
+ ## Errors
526
+
527
+ ```text
528
+ Calcsheet::Error < StandardError
529
+ ├── UnsupportedFormatError not CSV, XLSX or ODS (XLS gets its own message)
530
+ ├── MalformedFileError damaged file; sheet, row, column_index when known; the parser error as cause
531
+ ├── LimitExceededError limit, max, actual
532
+ │ ├── FileTooLargeError max_file_size
533
+ │ ├── TooManyRowsError max_rows
534
+ │ ├── TooManySheetsError max_sheets
535
+ │ └── ZipBombError max_uncompressed_size, max_compression_ratio, max_zip_entries
536
+ ├── SheetNotFoundError
537
+ ├── HeaderMismatchError mismatches: [{ sheet:, expected:, actual: }]
538
+ ├── ColumnNotFoundError
539
+ ├── AmbiguousColumnError
540
+ ├── CellError sheet, row, column_index, column_letter, key, value, reason
541
+ └── WriteError sheet, row, column_index
542
+ ```
543
+
544
+ - Messages name the location in user terms ("sheet 'Contacts (2)', row 4812, column C (price): invalid decimal
545
+ \"abc\""); the attributes let applications build their own. `reason` is a Symbol (`:invalid_decimal`,
546
+ `:required`...) that applications can translate.
547
+ - The parsers' own exceptions never leak: they are wrapped in `MalformedFileError`, the original kept as `cause`.
548
+ - Invalid arguments and options raise `ArgumentError` (options are checked before any row is read or any byte is
549
+ written); so do invalid sheet names when writing, and a second read of a non-seekable CSV stream. A closed book
550
+ raises `IOError` ("closed book"), also at the next batch of a read that was going on when it was closed.
551
+
552
+ ## Limits
553
+
554
+ Every limit is an option, set globally or per call. Exceeding one raises a `Calcsheet::LimitExceededError`
555
+ subclass whose `limit` names the option, with the observed value (`actual`) and the maximum (`max`). Every limit
556
+ but `max_rows` applies when the file is opened: give it to `Calcsheet.open` or the configuration, not to a read.
557
+
558
+ | Option | Default | Bounds |
559
+ |---|---|---|
560
+ | `max_file_size` | 1 GiB | the bytes of the source; a stream is counted as it is read or copied |
561
+ | `max_rows` | none | the rows one read yields (`rows`, `each_row`, `each_batch`; not `sample`) |
562
+ | `max_uncompressed_size` | 8 GiB | the bytes inflated from the zip archive, cumulative per book: every pass counts |
563
+ | `max_compression_ratio` | 100 | inflated / compressed bytes per entry, beyond `compression_ratio_floor` |
564
+ | `compression_ratio_floor` | 512 MiB | the bytes a streamed entry may inflate to whatever its ratio (the XLSX shared strings, kept in memory, get a fixed 16 MiB instead); `nil` or `0` for none |
565
+ | `max_zip_entries` | 10,000 | the entries of the archive, checked before its central directory is loaded |
566
+ | `max_sheets` | 1,000 | the sheets of an XLSX or ODS workbook |
567
+ | `max_cell_bytes` | 1 MiB | the text of one cell: an XLSX or ODS string, a CSV field once parsed, a text written; `nil` for none |
568
+
569
+ - `max_rows` has no default: set it when reading untrusted files. An ODS describes a million rows of content in a few
570
+ hundred bytes (a repeated row whose cells hold a value), and the reader yields every one of them; the zip limits
571
+ do not see it, `max_rows` does. Wide sparse rows cost too: an XLSX row whose one cell is in column XFD becomes an
572
+ Array of 16,384 values, so 50,000 such rows from a 250 KB file take seconds and hundreds of MB while they are
573
+ read, whatever the schema; `max_rows` keeps that bounded.
574
+ - The zip limits count the bytes actually inflated, never trusting the sizes the archive declares, and the CRC of
575
+ every entry read to its end is verified.
576
+ - The central directory is checked before rubyzip loads it: it must lie before the end records, and the local
577
+ headers and declared data of the entries must not overlap or reach into it, so that loading it reads every local
578
+ header once and every declared compressed size is real. Otherwise the archive raises `MalformedFileError`.
579
+ - While loading the central directory, rubyzip prints `WARNING: invalid date/time in zip entry.` to standard error
580
+ for an entry whose date is invalid (at most `max_zip_entries` times). Calcsheet leaves rubyzip's global settings
581
+ alone; an application can silence the warning with `Zip.warn_invalid_date = false` (after `require "zip"`).
582
+ - `max_uncompressed_size` is cumulative because an ODS keeps every sheet in one `content.xml`, inflated again from
583
+ its start for each sheet read, sampled or merge-checked: however many passes a crafted file causes, one book
584
+ never inflates more than the budget. `max_sheets` bounds those passes.
585
+ - Verbose, repetitive XML (ODS exports especially) deflates beyond 100:1, so an entry may always inflate up to
586
+ `compression_ratio_floor`; only beyond it does the ratio apply. Memory stays flat either way: entries are
587
+ streamed. The XLSX shared strings are the exception: they are kept in memory, 8 bytes per string beyond their
588
+ text, so 512 MiB of empty `<si/>` would take gigabytes. `compression_ratio_floor` does not apply to
589
+ `xl/sharedStrings.xml`: it may inflate to a fixed 16 MiB whatever its ratio (enough for a small file of long,
590
+ repetitive strings; at most about 54 MB of empty `<si/>` while the table is built), and beyond that only as far
591
+ as `max_compression_ratio` allows.
592
+ - The XML of untrusted files is bounded too. The tags the byte scanners read are at most 64 KiB each: XLSX row start
593
+ tags when `start_at:` skips rows, and the table, row, column and row group tags the ODS scanner reads to find
594
+ sheets and rows; other tags are passed over unbuffered, whatever their length. The part
595
+ of an ODS `content.xml` before its first sheet is at most 16 MiB, and so are the column elements and the row group
596
+ start tags kept when `start_at:` skips rows, inside at most 1,000 nested row groups. The XLSX parts read whole, `xl/workbook.xml` and
597
+ its relationships, are at most 1 MiB each, checked while they are inflated, and parsed one at a time (a part of
598
+ tiny elements still costs Ox tens of times its size); `xl/styles.xml` is streamed, keeping only its number formats
599
+ (at most 100,000, codes of at most 1 KiB) and cell formats (at most 1,000,000). Rows beyond 1,048,576, cells
600
+ beyond column XFD and XLSX rows out of order raise `MalformedFileError`.
601
+ - A cell whose text is longer than `max_cell_bytes` raises `MalformedFileError` naming the sheet, the row, the
602
+ column and the limit (for XLSX shared strings, the index of the string), before the text is joined or converted
603
+ (a CSV field only once it is parsed, see below).
604
+ The text of a number, a date, a time or a boolean is at most 1,024 bytes whatever the option; text read as
605
+ `:integer` or `:decimal` longer than that is an invalid value of the cell. Ox, the XML parser, buffers a text or a
606
+ tag whole before Calcsheet sees it, so in an XLSX or ODS part the bytes between two `<` are at most six times
607
+ `max_cell_bytes` plus 64 KiB (6 MiB by default): longer ones raise `MalformedFileError` while the entry is
608
+ inflated. A comment, a CDATA section or an attribute value that holds a `<` is not measured that way; Ox buffers
609
+ it whole, within the zip limits, at two to three times its size.
610
+ - Error messages show at most 200 characters of an attribute value, a cell value, a sheet name or a zip entry name
611
+ read from the file, and of a column header, a column key, a sheet name or a refused value given to a writer
612
+ (listing at most 10 unknown keys of a row), followed by `...` when cut; `error.sheet` and `error.value` keep the
613
+ whole text.
614
+ - A CSV field is checked against `max_cell_bytes` only after the standard library's `CSV::Parser` has built it, so
615
+ the option bounds what Calcsheet keeps and converts, not what parsing costs: there is no per-cell bound for CSV.
616
+ One huge field costs several times its size inside the parser (a single field of 100 MiB peaked at about 390 MB
617
+ unquoted and 490 MB quoted), bounded only by `max_file_size`. For untrusted CSV uploads, set a `max_file_size`
618
+ well below the 1 GiB default, sized to the memory one read may take.
619
+
620
+ ```ruby
621
+ require "calcsheet"
622
+
623
+ begin
624
+ Calcsheet.open("contacts.xlsx", max_file_size: 1_000) { |book| book.sheets }
625
+ rescue Calcsheet::FileTooLargeError => error
626
+ error.limit # => :max_file_size
627
+ error.max # => 1_000
628
+ end
629
+ ```
630
+
631
+ ## CSV
632
+
633
+ ### Reading
634
+
635
+ - **Encoding.** A UTF-8 BOM means UTF-8, a UTF-16 LE or BE BOM means UTF-16 (Excel's "Unicode Text"), transcoded
636
+ to UTF-8. Without a BOM, a seekable source is validated as UTF-8 over the whole file with a fast byte-level pass,
637
+ because files often start with ASCII rows and have accented names further down: valid means UTF-8, otherwise
638
+ Windows-1252, common in Excel exports. A non-seekable stream decides on the first 50 lines of its first 64 KiB
639
+ (ASCII counts as UTF-8); invalid bytes later raise `MalformedFileError`. `encoding:` forces any encoding Ruby can
640
+ transcode to UTF-8 and skips detection.
641
+ - **Separator and quote.** The separator is detected among `;`, `,`, tab and `|` by parsing the first 50 records of
642
+ a 64 KiB sample with each candidate and choosing the most consistent column count greater than one, so
643
+ separators inside quoted fields do not mislead it (ties go in that order). The quote character is `"`; `'` is
644
+ chosen only when no separator parses the sample with `"` (a quoted field left open by the end of the 64 KiB sample
645
+ does not count) and `'` gives consistent records of more than one field, so a quoted `"Smith, John"` stays one
646
+ value. A single-column file reads with `,` and `"`. `col_sep:` and `quote_char:` force them.
647
+ - **Lines** end with `\n`, `\r\n` or `\r`, one of them per file: the first line ending outside quoted fields decides,
648
+ and another one later outside quoted fields raises `MalformedFileError` ("mixed line endings") with its record
649
+ number and line, never leaving a stray `\r` in a value or merging two records. Quoted fields keep any line break
650
+ as written. Parsing is strict: a malformed record raises `MalformedFileError` with its record number and line.
651
+ - Values are always `String` (empty fields `nil`); declared types convert them with `decimal_separator:`,
652
+ `thousands_separator:`, `date_formats:`, `datetime_formats:`, `true_words:` and `false_words:`.
653
+ - `strip_formula_prefix: true` removes the `'` that [formula protection](#formula-protection) puts before `=`, `+`,
654
+ `-`, `@`, tab or carriage return, so that files written by Calcsheet import back cleanly. The default is `false`.
655
+
656
+ ### Writing
657
+
658
+ | Option | Default | With `excel_it: true` |
659
+ |---|---|---|
660
+ | `col_sep` | `,` | `;` |
661
+ | `bom` | no BOM | UTF-8 BOM |
662
+ | `decimal_separator` | `.` | `,` |
663
+ | `date_format` | `%Y-%m-%d` | `%d/%m/%Y` |
664
+ | `datetime_format` | `%Y-%m-%d %H:%M:%S` | `%d/%m/%Y %H:%M:%S` |
665
+ | `row_sep` | `\r\n` | `\r\n` |
666
+
667
+ `excel_it: true` is the preset that Excel with Italian (and similar European) regional settings opens directly. An
668
+ explicit `col_sep`, `bom`, `date_format` or `datetime_format`, configured or given per call, wins over the preset;
669
+ `decimal_separator` wins only when given in the call, since its default (`"."`) is also the reading default.
670
+ Output is UTF-8. Decimals are written in full without exponent (`scale` is ignored), booleans as `true`/`false`.
671
+ Times are written as their wall clock, without an offset: `%z` in `datetime_format` prints `+0000`.
672
+
673
+ ### Formula protection
674
+
675
+ A spreadsheet that opens a CSV file evaluates a field starting with `=`, `+`, `-`, `@`, tab or carriage return as a
676
+ formula, even when the field is quoted: quoting does not protect, because the quotes are removed before the content
677
+ is interpreted. Calcsheet therefore writes a leading `'` before such text in `:string` columns and in headers
678
+ ([OWASP](https://owasp.org/www-community/attacks/CSV_Injection)). This is on by default; disable it per column with
679
+ `formula_protection: false`.
680
+
681
+ The consequence: in CSV, `+393331234567` is written as `'+393331234567`. Without the prefix the spreadsheet would
682
+ read a number and drop the `+`. Disable protection for a phone column only when the file is not meant to be opened
683
+ by a spreadsheet application, and read Calcsheet's own exports with `strip_formula_prefix: true`.
684
+
685
+ XLSX and ODS need no prefix: `:string` cells are typed text, never evaluated, so `+39...` and `=...` are stored
686
+ verbatim.
687
+
688
+ ## XLSX and ODS
689
+
690
+ Reading:
691
+
692
+ - XLSX sheets are resolved through `xl/workbook.xml` and its relationships; the shared strings and the styles are
693
+ read from `xl/sharedStrings.xml` and `xl/styles.xml`, where Excel and LibreOffice write them. The shared strings
694
+ table is the only part kept in memory, packed in one binary string with offsets: its text plus 8 bytes per string,
695
+ twice that while the table is built (about 1.4 times the text for strings of 20 bytes; a table of empty strings
696
+ costs 8 bytes per `<si/>`, 16 while it is built). Because its cost follows what it inflates to,
697
+ `xl/sharedStrings.xml` gets not `compression_ratio_floor` but a fixed floor of 16 MiB, beyond which
698
+ `max_compression_ratio` bounds it. Rich text runs
699
+ are joined, phonetic runs skipped. Inline strings, `t="d"` ISO dates (read by the kind of their
700
+ number format, like numbers: a date-only format gives a `Date`), booleans, errors and cells or rows without `r`
701
+ attributes are supported. ISO dates of the year 0 are cell errors `:invalid_date`, in ODS too. `hidden` and `veryHidden` sheets are hidden.
702
+ - ODS `content.xml` is streamed; repeated rows and cells are counted, never expanded, so the huge trailing runs
703
+ LibreOffice writes cost nothing. Text joins paragraphs with `\n` and honors `text:s`, `text:tab` and
704
+ `text:line-break`; numbers come from `office:value`, so decimals are exact. The namespace prefixes are read from
705
+ the document, not assumed. A sheet whose table style says `table:display="false"` is hidden. Sampling finds the
706
+ start of each sheet by scanning bytes and parses only its first rows.
707
+
708
+ Writing:
709
+
710
+ - The header row is bold and frozen (XLSX `pane`, ODS `settings.xml`) when `header: true`. Columns get their
711
+ `width:`; ODS widths are rounded to whole characters, from 1 to 255.
712
+ - Number formats: `scale:` decimals (`0.00`), dates `yyyy-mm-dd`, datetimes `yyyy-mm-dd hh:mm:ss`. `scale:` is
713
+ display only: the full value is stored.
714
+ - XLSX text is written as inline strings (no shared strings table) and every sheet is streamed as its rows arrive,
715
+ so memory stays flat. XLSX dates range from 1900-01-01 to 9999-12-31, ODS years from 1 to 9999; others raise
716
+ `WriteError`. An XLSX cell holds at most 32,767 characters of text, counted as Excel does (a character beyond the
717
+ Basic Multilingual Plane, such as an emoji, counts twice); longer text raises `WriteError` without writing the row,
718
+ and a longer header raises it before the sheet starts.
719
+ ODS and CSV have no such limit beyond `max_cell_bytes`.
720
+ - The ODS `mimetype` entry comes first, stored, without data descriptor or extra field, as LibreOffice requires.
721
+ - Characters XML 1.0 does not allow are removed from text. A sheet has at most 16,384 columns.
722
+ - Sheets overflow into `"Name (2)"`, `"Name (3)"`... (see [overflow](#sheet-names-and-overflow)).
723
+
724
+ ## Streams and non-seekable input
725
+
726
+ - A **seekable** source (a path, a `File`, a `Tempfile`, a `StringIO`) is read in place.
727
+ - XLSX and ODS need random access to the zip central directory: a **non-seekable** stream is copied to a
728
+ `Tempfile` first (counted against `max_file_size` while copying), and the copy is deleted when the book closes.
729
+ - A non-seekable CSV stream opened with `format: :csv` is read **directly**, without a copy: the detection sample
730
+ (at most 64 KiB) is kept and replayed ahead of the rest of the stream. Such a stream can be read **once**: a
731
+ second read, for example `rows` after `sample`, raises `ArgumentError`. Read a file or a seekable IO, or open the
732
+ stream without `format:`, to read it more than once.
733
+ - A non-seekable stream opened **without** `format:` is spooled to a `Tempfile` to detect its format, CSV included,
734
+ and can then be read any number of times.
735
+ - A seekable CSV source is parsed again from its start whenever a read needs records already passed (`rows` after
736
+ `sample`, a second `rows`).
737
+
738
+ ## Time zones
739
+
740
+ Spreadsheet datetimes carry no zone: they are wall-clock fields. Calcsheet keeps that meaning.
741
+
742
+ - **Reading** returns a `Time` in UTC with the wall-clock fields of the file. `time_zone:` builds it in a zone
743
+ instead: `"UTC"`, a UTC offset such as `"+01:00"`, or any object responding to
744
+ `local(year, month, day, hour, min, sec)`, for example `ActiveSupport::TimeZone["Europe/Rome"]` (Calcsheet does
745
+ not depend on ActiveSupport). CSV text read as `:datetime` follows the same rule.
746
+ - XLSX `t="d"` values and ODS dates that carry `Z` or an offset are instants: they are returned in UTC, or converted
747
+ into `time_zone:` when one is set. A duck-typed zone converts them when it responds to `utc_to_local` or
748
+ `utc_offset` (as `ActiveSupport::TimeZone` does); one that responds to `local` alone leaves them in UTC.
749
+ - **Writing** a `Time` writes its wall clock in its own offset, in whole seconds (rounded, since spreadsheets store
750
+ fractions of a day). `time_zone:` converts it first; a duck-typed zone must then respond to `utc_to_local` (as
751
+ `ActiveSupport::TimeZone` does) or `utc_offset`.
752
+
753
+ ```ruby
754
+ require "calcsheet"
755
+
756
+ Calcsheet.open("contacts.xlsx") do |book|
757
+ book.rows.first["Created at"] # => Time.utc(2026, 1, 1, 10, 0, 0)
758
+ book.rows(time_zone: "+01:00").first["Created at"] # => Time.new(2026, 1, 1, 10, 0, 0, "+01:00")
759
+ end
760
+
761
+ Calcsheet.write("times.ods", time_zone: "UTC") do |book|
762
+ book.sheet("Times", columns: [{ header: "At", type: :datetime }]) do |sheet|
763
+ sheet << [Time.new(2026, 1, 1, 10, 0, 0, "+01:00")]
764
+ end
765
+ end
766
+ Calcsheet.open("times.ods") { |book| book.rows.first["At"] } # => Time.utc(2026, 1, 1, 9, 0, 0)
767
+ ```
768
+
769
+ ## Configuration
770
+
771
+ Global defaults are set with `Calcsheet.configure`; every option can be overridden per call
772
+ (`Calcsheet.open(path, max_rows: 100_000)`, `book.rows(time_zone: "UTC")`,
773
+ `Calcsheet.write("contacts.csv", excel_it: true)`), and per call wins; the options marked "open" in the
774
+ [options reference](#options-reference) only when the file is opened. An unknown option key raises `ArgumentError`. `Calcsheet.reset_config!` restores the defaults.
775
+
776
+ ```ruby
777
+ require "calcsheet"
778
+
779
+ Calcsheet.configure do |config|
780
+ config.max_file_size = 200 * 1024 * 1024
781
+ config.time_zone = "+01:00"
782
+ # Arrays are frozen copies: assign a new Array instead of changing the current one.
783
+ config.date_formats = [*config.date_formats, "%d.%m.%Y"]
784
+ end
785
+ Calcsheet.config.date_formats # => ["%Y-%m-%d", "%d/%m/%Y", "%d.%m.%Y"]
786
+ ```
787
+
788
+ ## Options reference
789
+
790
+ Every key of `Calcsheet::Configuration::DEFAULTS`. Read options apply to `Calcsheet.open` and every read method,
791
+ open options to `Calcsheet.open` only (a read method raises `ArgumentError` for them), write options to
792
+ `Calcsheet.write` and `Calcsheet.stream`.
793
+
794
+ Every value is checked when it is used: by the call that passes it, or for a configured value by the next
795
+ `Calcsheet.open`, `Calcsheet.write` or `Calcsheet.stream`, before anything is read or written; a book already open
796
+ keeps the options it was opened with. A wrong type or value raises `ArgumentError` naming the option. The zip-bomb
797
+ limits `max_uncompressed_size`, `max_zip_entries` (positive Integers) and `max_compression_ratio` (a positive number)
798
+ cannot be switched off; the other limits are positive Integers, or `nil` for none (`compression_ratio_floor` may also
799
+ be `0`); `batch_size` is a positive Integer and `sample_rows` a non-negative one; the booleans are `true` or `false`;
800
+ word and format lists are Arrays of non-empty Strings.
801
+
802
+ | Option | Default | Applies to | Meaning |
803
+ |---|---|---|---|
804
+ | `max_file_size` | `1_073_741_824` (1 GiB) | open | Bytes of the source; `nil` for no limit, which leaves an untrusted upload unbounded. |
805
+ | `max_rows` | `nil` | read | Rows one read may yield; `nil` for no limit. |
806
+ | `max_uncompressed_size` | `8_589_934_592` (8 GiB) | open | Bytes inflated from a zip archive, cumulative per book; always a positive Integer. |
807
+ | `max_compression_ratio` | `100` | open | Inflated / compressed bytes per zip entry, beyond the floor; always a positive number (a Float too). |
808
+ | `compression_ratio_floor` | `536_870_912` (512 MiB) | open | Bytes any zip entry may inflate to whatever its ratio, except the XLSX shared strings (a fixed 16 MiB); `nil` or `0` for none. |
809
+ | `max_zip_entries` | `10_000` | open | Entries of a zip archive; always a positive Integer. |
810
+ | `max_sheets` | `1_000` | open | Sheets of an XLSX or ODS workbook; `nil` for no limit. |
811
+ | `max_cell_bytes` | `1_048_576` (1 MiB) | open | Bytes of the text of one cell; `nil` for no limit, which also lifts the bound on what the XML parser buffers. See [limits](#limits). |
812
+ | `decimals` | `:bigdecimal` | read | `:bigdecimal` or `:float`: `:float` returns native fractional numbers, and fractional durations, as `Float`. |
813
+ | `time_zone` | `nil` | read, write | `nil` (UTC), `"UTC"` or `"Z"`, a UTC offset such as `"+01:00"` or `"+0100"`, or an object responding to `local` (and, for writing, to `utc_to_local` or `utc_offset`): see [time zones](#time-zones). |
814
+ | `decimal_separator` | `"."` | read, write CSV | `"."` or `","`: text read as `:decimal` or `:integer`; CSV output. |
815
+ | `thousands_separator` | `nil` | read | `nil`, `"."`, `","`, `" "`, `"'"`, `"\u00A0"` or `"\u202F"`, never the decimal separator. Groups the integer digits of text read as `:decimal` or `:integer` by three (`"1.234.567,89"`), or is absent (`"1234,5"`); text it groups otherwise (`"12.5"`) is invalid. |
816
+ | `date_formats` | `["%Y-%m-%d", "%d/%m/%Y"]` | read | `strptime` formats of text read as `:date` (and `:datetime`), with the directives `%Y` (a year of four digits) `%y %m %d %e %b %B %h %a %A %H %k %I %l %M %S %p %P %F %T %R %%`; others (`%z`, `%N`, `%s`, `%j`, `%G`, `%V`, flags such as `%-d`) raise `ArgumentError`. |
817
+ | `datetime_formats` | `["%Y-%m-%dT%H:%M:%S", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%d/%m/%Y %H:%M:%S", "%d/%m/%Y %H:%M"]` | read | `strptime` formats of text read as `:datetime` (and `:date`), with the directives of `date_formats`. |
818
+ | `true_words` | `["true", "1"]` | read | Text read as `true` by `:boolean`, case-insensitive. |
819
+ | `false_words` | `["false", "0"]` | read | Text read as `false` by `:boolean`, case-insensitive. |
820
+ | `batch_size` | `1000` | read | Rows read at a time, and the default size of `each_batch`. |
821
+ | `sample_rows` | `20` | read | Default `rows:` of `sample`. |
822
+ | `max_rows_per_sheet` | `1_048_576` | write | Rows per XLSX or ODS sheet, header included, before the overflow; can only lower the format's limit. |
823
+ | `strip_formula_prefix` | `false` | read CSV | Remove the `'` of formula protection. |
824
+ | `encoding` | `nil` | read CSV | Forces the encoding; `nil` detects it. |
825
+ | `col_sep` | `nil` | read, write CSV | One character other than `"`, CR or LF. Forces the separator when reading (`nil` detects it); `nil` writes `,` (`;` with `excel_it`). |
826
+ | `quote_char` | `nil` | read CSV | One character other than CR, LF or `col_sep`: forces the quote character; `nil` detects it. |
827
+ | `row_sep` | `"\r\n"` | write CSV | Line ending: `"\r\n"`, `"\n"` or `"\r"`. |
828
+ | `bom` | `nil` | write CSV | `true` writes a UTF-8 BOM; `nil` follows `excel_it`. |
829
+ | `date_format` | `nil` | write CSV | `strftime` format of dates; `nil`: `%Y-%m-%d` (`%d/%m/%Y` with `excel_it`). |
830
+ | `datetime_format` | `nil` | write CSV | `strftime` format of times; `nil`: `%Y-%m-%d %H:%M:%S` (`%d/%m/%Y %H:%M:%S` with `excel_it`). |
831
+ | `excel_it` | `false` | write CSV | The preset of `;`, BOM, decimal comma and `%d/%m/%Y` dates. |
832
+ | `sanitize_sheet_names` | `false` | write | Repair invalid sheet names and number duplicates instead of raising. |
833
+
834
+ ## Concurrency
835
+
836
+ Call `Calcsheet.configure` (or `Calcsheet.config`) once at boot, before threads read or write: the configuration is
837
+ created lazily, and that first creation is not thread-safe. Afterwards every read and write takes its own frozen
838
+ snapshot of the options, and books are independent; a single book must not be shared between threads.
839
+
840
+ Ox parses while holding Ruby's global VM lock, so two large reads in two threads of one process take turns instead
841
+ of running in parallel (measured: 66 s for two reads that take 27 s each alone). Process large files in separate
842
+ worker processes, or with one worker thread per process, until a native backend that releases the lock exists.
843
+
844
+ ## Performance
845
+
846
+ 1,000,000 rows of 11 typed columns (id, phone, zero-padded postal code, names, email, city, decimal price, date,
847
+ datetime, boolean), Ruby 3.4, each case in its own process; peak RSS includes about 27 MB for the interpreter and
848
+ the gem. `bundle exec rspec --tag performance` runs these cases with their memory bounds.
849
+
850
+ | Case | Rows/s | Peak RSS |
851
+ |---|---:|---:|
852
+ | Read XLSX, inline strings | 43,000 | 32 MB |
853
+ | Read XLSX, shared strings (saved by LibreOffice, 70 MB) | 35,000 | 146 MB |
854
+ | Read ODS | 28,000 | 96 MB |
855
+ | Read CSV | 74,000 | 53 MB |
856
+ | Write XLSX | 63,000 | 31 MB |
857
+ | Write ODS | 61,000 | 31 MB |
858
+ | Write CSV | 87,000 | 29 MB |
859
+
860
+ Memory does not grow with the number of rows, apart from the XLSX shared strings table, whose size the file itself
861
+ decides.
862
+
863
+ ## Compatibility
864
+
865
+ - Ruby 3.3 or later; tested on Ruby 3.3, 3.4 and 4.0.
866
+ - The suite reads XLSX and ODS files saved by LibreOffice and XLSX files saved by Excel for the web, and checks
867
+ every value; a workbook saved with the 1904 date system by Excel is still pending. Written files are checked by
868
+ reading them back with Calcsheet and with LibreOffice.
869
+
870
+ ## Not supported
871
+
872
+ - XLS (Excel 97–2003): the only Ruby reader is GPL-licensed. Save the file as XLSX, ODS or CSV.
873
+ - Random access to cells (`sheet["B7"]`) and editing existing files while keeping their formatting.
874
+ - Writing formulas, rich formatting, charts, conditional formats, data validation, images.
875
+ - Merged reading of sheets without a header row.
876
+
877
+ Possible future work: a faster native reader as a separate, optional gem (see [ADR 007](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/docs/adr/007-optional-native-backend.md)); random access to rows
878
+ (`book.row(900_000)`) by inflating a sheet once into a temporary file and indexing the byte offset of each row
879
+ (about 8 MB per million rows), then seeking and parsing one row; editing an uploaded file while keeping everything
880
+ else (copying the untouched zip entries, rewriting only the changed cells with new strings inline, marking formulas
881
+ for recalculation, splitting ODS repeated cells) for "fix the errors in your file" flows; XLS, if a non-GPL reader
882
+ appears.
883
+
884
+ ## Development
885
+
886
+ ```sh
887
+ bundle exec rspec # the suite; slow and external-tool cases are excluded
888
+ bundle exec rspec --tag performance # 1M-row reads and writes with their time and memory bounds
889
+ bundle exec rspec --tag libreoffice # written files opened by LibreOffice headless (needs soffice)
890
+ bundle exec rubocop
891
+ bundle exec rake # the default task: the suite, then RuboCop
892
+ bundle exec yard stats --list-undoc # the public API left undocumented (options in .yardopts)
893
+ bundle exec yard # the API documentation, in tmp/yard
894
+ ```
895
+
896
+ - Every fixture is synthetic (neutral names, `example.com` addresses); `spec/fixtures/README.md` says how each one
897
+ was produced. Large files are generated by the specs that need them and never committed.
898
+ - The code comments are YARD in RDoc markup (`+code+`). The public API is what `.yardopts` shows; implementation
899
+ classes and modules are tagged `@api private` (hidden by `--hide-api private`) and documented for maintainers.
900
+ - Runtime dependencies are pinned loosely in the gemspec (`~>`, or a bounded range such as `bigdecimal >= 3.1, < 5`); `Gemfile.lock` adopts only releases at least two
901
+ weeks old.
902
+ - CI (one pipeline per push: the merge request pipeline when one is open) runs:
903
+ - `rspec` on Ruby 3.3, 3.4 and 4.0, with a JUnit test report and a Cobertura coverage report that merge requests
904
+ show; it fails below 99% line or 98.5% branch coverage (checked only when `CI` is set, so focused local runs
905
+ are not);
906
+ - `rubocop`;
907
+ - `gem_build`, which checks the files of the built gem, and `gem_smoke`, which installs it on Ruby 3.3 outside the
908
+ bundle, with the locked runtime dependencies, and writes and reads a file in each format;
909
+ - `audit`: `bundle-audit` on the locked gems, allowed to fail;
910
+ - `deps_latest`, on schedules, merge requests and the default branch: the suite on the newest runtime dependencies
911
+ the gemspec allows, in a job-local copy of the Gemfile (`Gemfile.canary`, git-ignored), allowed to fail;
912
+ - `performance`, the `--tag performance` suite, by hand.
913
+
914
+ ## Design decisions
915
+
916
+ The reasons behind these choices, with the benchmarks that shaped them, are recorded as architecture decision
917
+ records in [`docs/adr/`](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/tree/main/docs/adr) of the repository.
918
+
919
+ ## License
920
+
921
+ Released under the [MIT License](https://gitlab.opinioni.net/demetra-developers/rubygems/calcsheet/-/blob/main/LICENSE.txt).