hypertabular-wasm 0.9.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5640b48d2db902a13bddf2b875eca7ca06f3d2fa0a318414c1ce166db2c5d365
-  data.tar.gz: f5bced5d2e07a2da961f749a94cfb96cf6023fcd58325095dc8efd06dabe1b7b
+  metadata.gz: a5f5417b9520fde201cba9a16cdf1f873689538554633d8f3a60c2ad8304d4f9
+  data.tar.gz: 81cd168e899fa6fb4347a6d89b575c3bc16364360fa97afd96075b943465b10e
 SHA512:
-  metadata.gz: ab47d6aaecbb62db8d315e0ea26b5bf87e52758235427df25923dc9d20a5cbab902f80a5e7fbff59bad525b8ff593698cc1e8972b2a6d44668693852ea007cee
-  data.tar.gz: 4a67aa99cc3957a732df52578da3e14390424744389ad91c8baa065020bc9ebc074860f8f2529e495bc4e067d95c01c8b35567049bd93203b169aea9255df861
+  metadata.gz: 3de75793df8238ae3af084000369f50deee80048c2e6b1caf3919c2fa8646f737cbe2df47dd98f940f0436c3e6d99628aa4a9dc2f6bb0f98ade720d510a000ba
+  data.tar.gz: f85379e972e936f0e3346124f0e8cb2d3303b89bcf09911c0de3eb41745c8dddd850fac845067ca7786634043082b4547e94e0b246725a53f6ae80231351a5c5

data/README.md

@@ -1,6 +1,6 @@
 # hypertabular
 
-Delimited text — CSV, TSV, any single-byte ASCII separator — and workbooks — XLSX and ODS —
+Delimited text — CSV, TSV, any single-byte ASCII separator — and workbooks — XLSX, ODS and Numbers —
 read a batch at a time into typed columns, with a
 [HyperCast](https://github.com/SkunkWerkx/HyperCast) verdict for every cell.
 
@@ -51,21 +51,25 @@ A plan known up front can be handed over at once —
 `DelimitedReader.open(path, dialect, plan)`, `book.sheet(which, options, plan)` — which binds it
 before the header is read.
 
-- **The header** (`reader.header`, `sheet.header`) is a `HyperTabular::Header`: the frozen
-  Array of frozen names it always was, which also says where a name is. `header.ordinal(name)`
+- **The header** (`reader.header`, `sheet.header`) is a `HyperTabular::Header`: a frozen
+  Array of frozen names, which also says where a name is. `header.ordinal(name)`
   is the first column with exactly that name — case and spaces included — and a `KeyError`
   naming it when there is none (or, with a block, the block's value, as `Hash#fetch` does);
   `header.find_ordinal(name)` answers `nil` instead.
 - **Binding** happens once, before the first read: `bind` again raises `RuntimeError`, and so
   does a `read` before `bind` — which `bind` then cures. A plan that is not Columns is an
-  `ArgumentError` and leaves the reader unbound. `bound?` says which it is. A `nil` plan
+  `ArgumentError` and leaves the reader unbound; so is, once a delimited header has fixed
+  the width every record has, a column at or past it, named in the message (a plan handed
+  to the constructor is held to the header as soon as it is read). Without a header such a
+  column reads as empty, and a plan reaching past 256 source columns reads shorter batches,
+  as many rows as about 8 MiB of cell table holds. `bound?` says which it is. A `nil` plan
   means "bind later"; `[]` is an empty plan, bound at once.
   `reader.column_count` is the header's count, or the first record's once it has been read.
 - **Rows.** `batch.each`, `reader.each` and `sheet.each` yield a `HyperTabular::Row` —
   `index`, `line`, `get(column)` (also `[]` and `verdict`), `value(column)`, `raw(column)`,
   `to_a` — each answering what the batch's own accessor does for that row. A batch is a copy,
   so a row stays good after the read has moved on. A Row deconstructs as its verdicts, for
-  `case row in [HyperCast::Success(value: id), *]`. `reader.each_row` still yields each row as
+  `case row in [HyperCast::Success(value: id), *]`. `reader.each_row` yields each row as
   a frozen Array of verdicts, and `sheet.each_batch` walks a sheet's batches.
 - **A workbook from an IO.** `Workbook.new(io)` reads anything with `#read` to its end into a
   String the workbook owns, as soon as it is opened; `close_source: true` closes the IO once it
@@ -140,7 +144,7 @@ Selection happens once, at `require`, in that order. Both backends read a batch
 call and hand back the same bytes; everything above that crossing — the readers, the
 workbook, `Batch` and every value and verdict it builds — is the same Ruby on either, and
 `spec/native_backend_spec.rb` holds the two to the same answer over every cell of both corpora.
-So the extension is not where a read gets faster (the crossing was already once per batch);
+So the extension is not where a read gets faster (either backend crosses once per batch);
 what it buys is a platform gem with no shared library for Fiddle to find, and Ruby in the
 browser, where there is no Fiddle at all.
 
@@ -188,6 +192,12 @@ delimited text from a String or any IO.
   build naming the ones the gem carries. Built and tested against ruby_wasm 2.10.
 - **No Rust toolchain.** The gem's `extconf.rb` only hands rbwasm the prebuilt archive. The
   first `rbwasm build` compiles Ruby itself and takes 15–20 minutes; later builds reuse it.
+- **macOS with Homebrew: limit pkg-config to `openssl@3`.** Run the build as
+  `PKG_CONFIG_LIBDIR="$(brew --prefix openssl@3)/lib/pkgconfig" bundle exec rbwasm build …`.
+  Without it Homebrew's pkg-config files reach into the build twice: its `openssl` is OpenSSL
+  4, which the native Ruby rbwasm builds first does not compile against, and its `libyaml`
+  is a macOS library that psych then links into the wasm interpreter, which fails for Ruby
+  4.0.
 - **Static linking only,** into ruby.wasm's default `wasm32-unknown-wasip1` interpreter.
 - **Beside HyperCast.** Every Rust extension that carries std defines a few of the same
   symbols (`rust_eh_personality`, rb-sys's `ruby_abi_version`, one of std's); this gem and
@@ -209,3 +219,37 @@ bundle exec rake native:dev             # builds the Magnus extension for this R
 bundle exec rspec                       # the extension, and its agreement with Fiddle
 bundle exec rake docs:check
 ```
+
+## Verifying provenance
+
+Every gem RubyGems.org serves — the universal fallback, each of the seven precompiled platform
+gems and `hypertabular-wasm` — verifies from the release tag alone: the release attestation,
+signed by this repository's `release.yml` at the tag, names every one of them, with the
+release inventory as its predicate.
+
+```sh
+gem fetch hypertabular -v <version> --platform <platform>   # or omit --platform for the universal gem
+gh attestation verify hypertabular-<version>-<platform>.gem --repo SkunkWerkx/HyperTabular \
+  --source-ref refs/tags/v<version> --predicate-type https://github.com/SkunkWerkx/.github/predicates/release/v1
+```
+
+Each gem's build provenance is this repository's too: the `ruby-pack` job attests
+`ruby/pkg/*.gem` as soon as they are built and `rubygems-publish` pushes those preserved
+bytes, so plain `--repo` verifies it:
+
+```sh
+gh attestation verify hypertabular-<version>-<platform>.gem --repo SkunkWerkx/HyperTabular
+```
+
+That is the release's second layer of checking, not the only one: before any gem is built, the
+same job verifies every native binary it packs — the Fiddle libraries, the Magnus extensions
+(one per Ruby ABI per platform), the `wasm32-wasip1` archives — against *their own* build
+provenance, signed from `SkunkWerkx/.github` by `hyper-build-native.yml` and
+`hyper-build-wasm.yml` (so that check needs `--signer-repo SkunkWerkx/.github` added), and
+refuses to proceed on an unverified one. RubyGems.org has no unpublish and no
+duplicate-version overwrite, so all of this happens while a bad artifact is still reversible,
+and the push itself is a separate dispatch from the preserved bytes. The release run then
+re-fetches every gem from rubygems.org and fails unless each matches the attested bytes
+exactly, turning "rubygems.org stores an upload verbatim" into a per-release check rather than
+an assumption. The root README's [Provenance](../README.md#provenance) section has the whole
+picture.

data/lib/hypertabular/column.rb

@@ -32,7 +32,8 @@ module HyperTabular
     # HyperCast::NumFormat (ArgumentError), an unknown door or an undeclared option
     # (KeyError, as HyperCast raises for the same).
     def initialize(ordinal:, door:, declared: nil, format: nil)
-      raise ArgumentError, "ordinal must be an Integer in 0..#{Column::MAX_ORDINAL}; got #{ordinal.inspect}" unless
+      raise ArgumentError, "ordinal must be an Integer at least 0, and an ordinal must be below 2^20 " \
+                           "(1,048,576); got #{ordinal.inspect}" unless
         ordinal.is_a?(Integer) && ordinal.between?(0, Column::MAX_ORDINAL)
 
       DOORS.fetch(door)
@@ -159,8 +160,10 @@ module HyperTabular
     end
   end
 
-  # The widest source ordinal a column can name.
-  Column::MAX_ORDINAL = (1 << 31) - 1
+  # The widest source ordinal a column can name: an ordinal must be below 2^20. No workbook
+  # format has that many columns (XLSX and ODS stop at 16,384, Numbers at 1,000), and at the
+  # cap one row of cell table is at most 8 MiB.
+  Column::MAX_ORDINAL = (1 << 20) - 1
 
   # The doors that read a HyperCast::NumFormat: the integers, the reals and the decimal.
   Column::NUMERIC = %i[i8 i16 i32 i64 u8 u16 u32 u64 f32 f64 decimal].freeze

data/lib/hypertabular/delimited_reader.rb

@@ -114,6 +114,8 @@ module HyperTabular
       @failure = nil
       take(source)
       @header = dialect.has_header ? read_header : nil
+      # A plan bound before the header was read is held to it as soon as it is.
+      within_header(planned) if planned
     end
 
     # The header's names, when the dialect declares a header: a Header — the frozen Array of
@@ -130,13 +132,16 @@ module HyperTabular
     # Columns, in output order, typically built from the #header's ordinals — once, before
     # the first read. Returns the reader.
     #
-    # ArgumentError for a plan that is not Columns, which leaves the reader unbound;
-    # RuntimeError if the reader already has a plan; IOError on a closed reader.
+    # ArgumentError for a plan that is not Columns, or that reads a column at or past the
+    # width the header gives every record, which leaves the reader unbound; RuntimeError if
+    # the reader already has a plan; IOError on a closed reader.
     def bind(plan)
       raise IOError, "closed reader" if @closed
       raise "the reader already has a plan: a plan is bound once" if bound?
 
-      bind_kernel(Column.plan(plan))
+      plan = Column.plan(plan)
+      within_header(plan)
+      bind_kernel(plan)
       self
     end
 
@@ -243,6 +248,19 @@ module HyperTabular
     UNBOUND_MESSAGE = "the reader has no plan yet: bind one before reading".freeze
     private_constant :UNBOUND, :UNBOUND_MESSAGE
 
+    # ArgumentError for the first column of +plan+ at or past the width the header has given
+    # every record: such a column could only ever be empty.
+    def within_header(plan)
+      width = @header.nil? ? nil : column_count
+      return if width.nil?
+
+      index = plan.index { |column| column.ordinal >= width }
+      return if index.nil?
+
+      raise ArgumentError, "plan column #{index} reads source column #{plan[index].ordinal}, " \
+                           "past the #{width} columns the header gives every record"
+    end
+
     # Sizes the kernel's arrays for +plan+ (already checked to be Columns) and takes it.
     def bind_kernel(plan)
       # Cell-table entries one row takes: the widest ordinal the plan reads, plus two.

data/lib/hypertabular/runtime/delimited.rb

@@ -43,6 +43,17 @@ module HyperTabular
 
       ARENA_BYTES = 4096
       NAMES = 64
+      # The cell-table entries a reader holds at most, unless one row takes more: what a
+      # batch of 4096 rows takes for a plan reaching 256 source columns, so that an ordinary
+      # plan's batches are full, and a plan reaching far past any column a record holds
+      # costs one row's entries rather than a batch of them.
+      CELL_BUDGET = 4096 * 257
+
+      # The rows of +per_row+ entries the cell table holds: as many as the budget allows,
+      # at least one and at most a batch. The core fills only the rows the table holds.
+      def self.cell_rows(per_row, batch_rows)
+        (CELL_BUDGET / [per_row, 1].max).clamp(1, batch_rows)
+      end
 
       CONTRACT = "hypertabular: libhypertabular reported a contract violation — a binding bug, " \
                  "please report it".freeze
@@ -57,6 +68,9 @@ module HyperTabular
       # #bind.
       attr_reader :columns, :per_row
 
+      # The cell-table entries the reader holds.
+      def cell_entries = @cells_cap
+
       # The loaded core's version word, major << 16 | minor << 8 | patch.
       def self.version
         Runtime.function(:hypertabular_version).call
@@ -87,7 +101,7 @@ module HyperTabular
         @batch_rows = batch_rows
         @columns = Columns.new(specs, sizes, batch_rows)
         @per_row = per_row
-        @cells_cap = per_row * batch_rows
+        @cells_cap = per_row * Delimited.cell_rows(per_row, batch_rows)
         @cells = Runtime.buffer(SPAN_BYTES * @cells_cap)
       end
 
@@ -142,12 +156,14 @@ module HyperTabular
           needed = finished
           case code
           when OK
-            @cramped = @rows.positive? && @rows < @batch_rows && @arena_used * 2 >= @arena_cap
+            # A batch the cell table ends early (CELL_BUDGET) is no reason to grow the arena.
+            held = [@batch_rows, @cells_cap / @per_row].min
+            @cramped = @rows.positive? && @rows < held && @arena_used * 2 >= @arena_cap
             return code
           when ERR_STRUCTURE then return code
           when ERR_CELLS
             @per_row = [@per_row, needed].max
-            @cells_cap = @per_row * @batch_rows
+            @cells_cap = @per_row * Delimited.cell_rows(@per_row, @batch_rows)
             @cells = Runtime.buffer(SPAN_BYTES * @cells_cap)
           when ERR_ARENA then grow_arena(needed)
           else raise CONTRACT

data/lib/hypertabular/runtime/workbook.rb

@@ -139,7 +139,7 @@ module HyperTabular
           styles: :hypertabular_workbook_styles
         }.freeze
 
-        # :xlsx or :ods; the date system's code (1 for 1900, 2 for 1904); the sheets as
+        # :xlsx, :ods or :numbers; the date system's code (1 for 1900, 2 for 1904); the sheets as
         # [name, hidden, part, index]; the shared strings' bytes; the state template every
         # sheet copies; and the tables a sheet's calls are handed.
         attr_reader :format, :epoch, :sheets, :strings, :state, :tables, :container, :length
@@ -175,7 +175,7 @@ module HyperTabular
             format, epoch, strings_bytes, _count, needed, *failure = @out[0, OPENED_BYTES].unpack(OPENED)
             case code
             when OK
-              @format = format == 2 ? :ods : :xlsx
+              @format = { 2 => :ods, 3 => :numbers }.fetch(format, :xlsx)
               @epoch = epoch
               return strings_bytes
             when ERR_WINDOW then scratch.grow_window(needed)

data/lib/hypertabular/tabular_error.rb

@@ -23,7 +23,7 @@ module HyperTabular
     KINDS = {
       1 => :unclosed_quote, 2 => :column_count, 3 => :row_too_long, 16 => :not_a_zip, 17 => :container,
       18 => :encrypted, 19 => :method, 20 => :missing_part, 21 => :xml, 22 => :deflate,
-      23 => :not_a_workbook, 24 => :shared_string, 25 => :too_large
+      23 => :not_a_workbook, 24 => :shared_string, 25 => :too_large, 26 => :iwa
     }.freeze
 
     # What is wrong: :unclosed_quote (the input ended inside a quoted cell), :column_count
@@ -32,7 +32,11 @@ module HyperTabular
     # :container (the zip's own structure is broken), :encrypted, :method (a part is
     # compressed by neither stored nor deflate), :missing_part, :xml (a part ends inside an
     # XML construct), :deflate (a part is not a whole deflate stream), :not_a_workbook,
-    # :shared_string (a cell names a shared string the table does not have) or :too_large.
+    # :shared_string (a cell names a shared string the table does not have), :too_large or
+    # :iwa (a Numbers part is not what the IWA packaging says: found is 1 for a broken chunk
+    # or Snappy block, 2 for a malformed message, 3 for an object a reference names that is
+    # not there, expected its identifier, 4 for cell storage of a version the core does not
+    # read, expected the version).
     attr_reader :kind
 
     # Zero-based index of the offending record — the header and skipped blank lines
@@ -89,9 +93,11 @@ module HyperTabular
       when :missing_part then "Part #{record}, which the workbook cannot be read without, is missing."
       when :xml then "Part #{record} of the workbook ends inside an XML construct (byte #{byte})."
       when :deflate then "Part #{record} of the workbook is not a whole deflate stream (byte #{byte})."
-      when :not_a_workbook then "The zip is neither an XLSX nor an ODS workbook."
+      when :not_a_workbook then "The zip is not an XLSX, ODS or Numbers workbook."
       when :shared_string then "Row #{line} names shared string #{found}; the table has #{expected}."
       when :too_large then "The workbook holds more text than a batch can address."
+      when :iwa
+        "Part #{record} of the Numbers document is not what the IWA packaging says (byte #{byte}, detail #{found})."
       else "The workbook's zip structure is broken."
       end
     end

data/lib/hypertabular/workbook.rb

@@ -19,7 +19,7 @@ module HyperTabular
   # A header, empty rows skipped, DelimitedReader::DEFAULT_BATCH_ROWS rows a batch.
   SheetOptions::DEFAULT = SheetOptions.new
 
-  # An XLSX or ODS workbook held in memory, its sheets listed and its shared strings and
+  # An XLSX, ODS or Numbers workbook held in memory, its sheets listed and its shared strings and
   # styles loaded: what a Sheet reads from.
   #
   #   book = HyperTabular::Workbook.open("orders.xlsx")
@@ -64,7 +64,7 @@ module HyperTabular
     # (chart sheets, macro sheets) are not among them.
     attr_reader :sheets
 
-    # Which kind of workbook this is: :xlsx or :ods.
+    # Which kind of workbook this is: :xlsx, :ods or :numbers.
     def format
       @book.format
     end

data/lib/hypertabular.rb

@@ -6,7 +6,7 @@ require_relative "hypertabular/runtime/delimited"
 require_relative "hypertabular/runtime/workbook"
 
 # Tabular parsing with a HyperCast verdict for every cell: delimited text — CSV, TSV, any
-# single-byte ASCII separator — and workbooks — XLSX and ODS — read a batch at a time into
+# single-byte ASCII separator — and workbooks — XLSX, ODS and Numbers — read a batch at a time into
 # typed columns by one Rust core (libhypertabular), called through Fiddle.
 #
 #   plan = [HyperTabular::Column.i32(0), HyperTabular::Column.text(1)]
@@ -38,7 +38,7 @@ require_relative "hypertabular/runtime/workbook"
 module HyperTabular
   # This gem's own version — kept in lockstep with hypertabular.gemspec and the core's
   # rust/Cargo.toml.
-  VERSION = "0.9.0"
+  VERSION = "1.0.0"
 
   class << self
     # Whether the native core loaded and exports the ABI this binding was built against.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: hypertabular-wasm
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Brian Buvinghausen
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.0
+        version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.0
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement