ruby-hdf5 0.0.2 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4591bafb2585ec00df81fc3831cc7037cc02dee81f15919ec7cf1b5d2f08a4c2
-  data.tar.gz: beb661f3c825adaab6b132cee69a402c1319d05143a0bec1516f2e0dec1180b2
+  metadata.gz: e7932dfc6c7cc1ffcfb6dc30c8963785e1e1e2ba206add86e634878d999e166d
+  data.tar.gz: cd6e121152d557821c64deb08a08286528911db013a48a0d7d7a1bff8c6da7a0
 SHA512:
-  metadata.gz: bf13704291dfbdb2d0d61cf03504574a553e3d16e6db980355df5408ee425e340cfa1bc7b3850c4f877aa066958f20ab4cb5cbb07935f6302eb403771c6b4458
-  data.tar.gz: ac8511ab873fea84933c38c65c15c899cfb3dbc7a63d2a062f41d78722bcc87288d84997e5f296c38691deb2660e065046e7fdf13e2c611127ef8fc84b1ce2ea
+  metadata.gz: f7270f19f0bd3732147faad5f94f0c429f02c93e81ad66985adefddff43ac41fbacb2834073d6b195c14f4ff905df4a60735ea143fee79fab1116a96f76c7e54
+  data.tar.gz: 060cb8b49f0373ca6ac788274c1bb5bc187fe92bab799b240f731034b59abebd789c956e23da6ccdbef4dd577e37be38e20121879ce98caea20c326829cb2e2e

data/README.md

@@ -2,119 +2,122 @@
 
 [![test](https://github.com/red-data-tools/ruby-hdf5/actions/workflows/test.yml/badge.svg)](https://github.com/red-data-tools/ruby-hdf5/actions/workflows/test.yml)
 
-Ruby bindings for the HDF5 library.
+Ruby bindings for HDF5 with Numo::NArray support.
 
-## Scope
+## Requirements
 
-This gem currently provides practical high-level wrappers for:
-
-- opening and creating files
-- creating groups
-- creating, writing, and reading one-dimensional numeric datasets
-- reading and writing numeric attributes
-
-Unsupported at this stage:
-
-- string dataset read/write
-- multidimensional array write
-
-Integer datasets and integer attributes currently use native C `int` under the hood.
-Values outside that range are rejected with `HDF5::Error` to avoid silent overflow.
-
-`Group#list_datasets` filters datasets from group entries by checking object type per entry.
-For very large groups, this may be slower than `Group#list_entries`.
-
-## Supported HDF5 Versions
-
-- HDF5 1.10
-- HDF5 1.14
-- HDF5 2.x
-
-HDF5 versions older than 1.10 are not supported.
+- Ruby 3.4 or later
+- HDF5 1.10 or later (`libhdf5` shared library)
 
 ## Install
 
-Add to your Gemfile:
+Add the gem to your Gemfile:
 
 ```ruby
 gem 'ruby-hdf5'
 ```
 
-Install:
+Then install dependencies:
 
 ```sh
 bundle install
 ```
 
-System library (`libhdf5`) is required.
-
-## Runtime Notes
-
-- The gem loads `libhdf5` through FFI.
-- If the shared library cannot be found automatically, set `HDF5_LIB_PATH`.
-
-Examples:
+Set `HDF5_LIB_PATH` only when `libhdf5` cannot be found automatically. It may be a library directory or a shared-library path.
 
 ```sh
-# Point to a directory containing libhdf5.so
-export HDF5_LIB_PATH=/usr/lib
-
-# Or point directly to the shared object
 export HDF5_LIB_PATH=/usr/lib/libhdf5.so
 ```
 
-## Quick Start
+## Usage
 
-### Read an existing file
+Create a file and write a Numo array:
 
 ```ruby
 require 'hdf5'
 
-HDF5::File.open('example.h5') do |file|
-	group = file['foo']
-	dataset = group['bar_int']
+matrix = Numo::SFloat.new(100, 64).seq
+
+HDF5::File.open('numbers.h5', 'w') do |file|
+	dataset = file.require_group('measurements').create_dataset('signal', matrix)
+	dataset.attrs['unit'] = 'a.u.'
+end
+```
+
+Read data and inspect its type:
+
+```ruby
+HDF5::File.open('numbers.h5') do |file|
+	dataset = file['measurements/signal']
 	p dataset.shape
-	p dataset.dtype
+	p dataset.dtype.to_sym
 	p dataset.read
 end
 ```
 
-### Create and write a file
+Use a block with `HDF5::File.open` to close the file automatically. Supported modes are `r`, `r+`, `w`, `x`, and `a`.
 
-```ruby
-require 'hdf5'
+## Common Tasks
 
-HDF5::File.create('numbers.h5') do |file|
-	file.create_group('values') do |group|
-		group.create_dataset('ints', [1, 2, 3, 4])
-	end
-end
+Read a row, a column, or a strided selection without reading the complete dataset:
 
-reopened = HDF5::File.open('numbers.h5')
-p reopened['values']['ints'].read
-reopened.close
+```ruby
+HDF5::File.open('numbers.h5') do |file|
+	dataset = file['measurements/signal']
+	row = dataset[10, true]
+	column = dataset[true, 0]
+	every_tenth_row = dataset[HDF5.slice(0...100, step: 10), true]
+end
 ```
 
-## Error Handling
+Append rows to an extendible dataset. Extendible datasets must use chunked storage:
+
+```ruby
+HDF5::File.open('samples.h5', 'w') do |file|
+	samples = file.create_dataset(
+		'samples',
+		shape: [0, 2],
+		dtype: :float32,
+		maxshape: [nil, 2],
+		chunks: [256, 2]
+	)
+	samples.append(Numo::SFloat[[1.0, 2.0], [3.0, 4.0]])
+end
+```
 
-High-level API failures raise `HDF5::Error`.
+Process a dataset in bounded-memory blocks:
 
 ```ruby
-begin
-	HDF5::File.open('missing.h5')
-rescue HDF5::Error => e
-	warn e.message
+sum = 0.0
+
+HDF5::File.open('numbers.h5') do |file|
+	file['measurements/signal'].each_block(max_bytes: 4 * 1024 * 1024) do |_selection, block|
+		sum += block.sum
+	end
 end
 ```
 
-## Development
+## Main Operations
+
+- Hierarchy: `[]`, `create_group`, `require_group`, `keys`, `delete`, `move`
+- Datasets: `create_dataset`, `read`, `write`, `[]`, `[]=`, `read_into`
+- Dataset metadata: `shape`, `ndim`, `size`, `dtype`, `chunks`, `maxshape`, `fillvalue`
+- Storage: chunking, gzip, shuffle, Fletcher32, resize, and append
+- Iteration: `each_block(max_bytes:)` and `each_chunk`
+- Attributes: `attrs[]`, `attrs[]=`, `attrs.create`, `attrs.modify`, `attrs.delete`
+
+Datasets support Numo numeric arrays, scalar values, variable-length UTF-8 strings, h5py-compatible bool values, and h5py-compatible complex values.
+
+## Limitations
 
-After more than a decade, it is clear that the Ruby community does not have enough resources to sustainably maintain an HDF5 library. For that reason, development of this library is intentionally AI-assisted. Something is better than nothing.
+- Fixed-length strings, general compound / enum / reference types, and variable-length numeric types are unsupported.
+- Fancy indexing, boolean masks, negative slice steps, and general broadcasting are unsupported.
+- SWMR, MPI, and VDS creation are unsupported.
 
-## Acknowledgement
+## Examples
 
-[https://github.com/edmundhighcock/hdf5](https://github.com/edmundhighcock/hdf5)
+See [examples/README.md](examples/README.md) for standalone examples, ordered from basic file I/O through chunking, resizing, and links.
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+MIT. See [LICENSE.txt](LICENSE.txt).

data/lib/hdf5/attribute.rb

@@ -1,43 +1,34 @@
 module HDF5
   class Attribute
-    def initialize(dataset_id, attr_name)
+    def initialize(dataset_id, attr_name, context = nil)
       @dataset_id = dataset_id
       @attr_name = attr_name
+      context&.ensure_open!(dataset_id)
       @attr_id = HDF5::FFI.H5Aopen(@dataset_id, @attr_name, HDF5::DEFAULT_PROPERTY_LIST)
       raise HDF5::Error, 'Failed to open attribute' if @attr_id < 0
     end
 
     def read
       type_id = HDF5::FFI.H5Aget_type(@attr_id)
-      space_id = HDF5::FFI.H5Aget_space(@attr_id)
-
-      size = HDF5::FFI.H5Sget_simple_extent_npoints(space_id)
-
-      buffer = \
-        case HDF5::FFI.H5Tget_class(type_id)
-        when :H5T_INTEGER
-          ::FFI::MemoryPointer.new(:int, size)
-        when :H5T_FLOAT
-          ::FFI::MemoryPointer.new(:double, size)
-        when :H5T_STRING
-          ::FFI::MemoryPointer.new(:pointer, size)
-        else
-          raise HDF5::Error, 'Unsupported data type'
-        end
+      raise HDF5::Error, 'Failed to get attribute datatype' if type_id < 0
 
-      status = HDF5::FFI.H5Aread(@attr_id, type_id, buffer)
+      space_id = HDF5::FFI.H5Aget_space(@attr_id)
+      raise HDF5::Error, 'Failed to get attribute dataspace' if space_id < 0
+      return read_string(type_id, space_id) if HDF5::FFI.H5Tget_class(type_id) == :H5T_STRING
+
+      dtype_object = DType.for_hdf5(type_id)
+      attribute_shape = shape(space_id)
+      size = attribute_shape.empty? ? 1 : attribute_shape.inject(:*)
+      buffer = ::FFI::MemoryPointer.new(:char, size * dtype_object.itemsize)
+      status = HDF5::FFI.H5Aread(@attr_id, dtype_object.memory_type_id, buffer)
       raise HDF5::Error, 'Failed to read attribute' if status < 0
 
-      case HDF5::FFI.H5Tget_class(type_id)
-      when :H5T_INTEGER
-        buffer.read_array_of_int(size)
-      when :H5T_FLOAT
-        buffer.read_array_of_double(size)
-      when :H5T_STRING
-        buffer.read_pointer.read_string
-      else
-        raise HDF5::Error, 'Unsupported data type'
-      end
+      result = HDF5::DataHelpers.from_binary(dtype_object, buffer.read_bytes(size * dtype_object.itemsize),
+                                             attribute_shape)
+      return result unless attribute_shape.empty?
+
+      scalar = result.extract
+      dtype_object.kind == :bool ? !scalar.zero? : scalar
     ensure
       HDF5::FFI.H5Tclose(type_id) if type_id && type_id >= 0
       HDF5::FFI.H5Sclose(space_id) if space_id && space_id >= 0
@@ -49,15 +40,54 @@ module HDF5
       HDF5::FFI.H5Aclose(@attr_id)
       @attr_id = nil
     end
+
+    private
+
+    def shape(space_id)
+      rank = HDF5::FFI.H5Sget_simple_extent_ndims(space_id)
+      raise HDF5::Error, 'Failed to get attribute rank' if rank < 0
+      return [] if rank.zero?
+
+      dimensions = ::FFI::MemoryPointer.new(:ulong_long, rank)
+      status = HDF5::FFI.H5Sget_simple_extent_dims(space_id, dimensions, nil)
+      raise HDF5::Error, 'Failed to get attribute shape' if status < 0
+
+      dimensions.read_array_of_uint64(rank)
+    end
+
+    def read_string(type_id, space_id)
+      unless HDF5::StringCodec.variable?(type_id)
+        raise UnsupportedTypeError, 'Fixed-length string attributes are not yet supported'
+      end
+
+      attribute_shape = shape(space_id)
+      count = attribute_shape.empty? ? 1 : attribute_shape.inject(:*)
+      buffer = ::FFI::MemoryPointer.new(:pointer, count)
+      status = HDF5::FFI.H5Aread(@attr_id, type_id, buffer)
+      raise HDF5::Error, 'Failed to read string attribute' if status < 0
+
+      HDF5::StringCodec.read_values(buffer, count, attribute_shape)
+    ensure
+      if buffer
+        active_error = $ERROR_INFO
+        reclaim_status = HDF5::FFI.H5Dvlen_reclaim(type_id, space_id, HDF5::DEFAULT_PROPERTY_LIST, buffer)
+        if reclaim_status.negative? && active_error.nil?
+          raise HDF5::Error,
+                'Failed to reclaim variable-length string attribute'
+        end
+      end
+    end
   end
 
   class AttributeManager
-    def initialize(dataset_id)
+    def initialize(dataset_id, context = nil)
       @dataset_id = dataset_id
+      @context = context
     end
 
     def [](attr_name)
-      attr = Attribute.new(@dataset_id, attr_name)
+      @context&.ensure_open!(@dataset_id)
+      attr = Attribute.new(@dataset_id, attr_name, @context)
       attr.read
     ensure
       attr.close if attr
@@ -67,9 +97,92 @@ module HDF5
       write(attr_name, value)
     end
 
+    def keys
+      @context&.ensure_open!(@dataset_id)
+      names = []
+      index = ::FFI::MemoryPointer.new(:ulong_long)
+      index.write_ulong_long(0)
+      callback = ::FFI::Function.new(:int, %i[int64_t string pointer pointer]) do |_, name, _, _|
+        names << name
+        0
+      end
+      status = HDF5::FFI.H5Aiterate2(@dataset_id, :H5_INDEX_NAME, :H5_ITER_NATIVE, index, callback, nil)
+      raise HDF5::Error, 'Failed to iterate over attributes' if status < 0
+
+      names
+    end
+
+    def key?(attr_name)
+      @context&.ensure_open!(@dataset_id)
+      exists = HDF5::FFI.H5Aexists(@dataset_id, attr_name)
+      raise HDF5::Error, "Failed to check attribute existence: #{attr_name}" if exists.negative?
+
+      exists.positive?
+    end
+
+    def delete(attr_name)
+      @context&.ensure_open!(@dataset_id)
+      status = HDF5::FFI.H5Adelete(@dataset_id, attr_name)
+      raise HDF5::Error, "Failed to delete attribute: #{attr_name}" if status < 0
+
+      self
+    end
+
+    def create(attr_name, value)
+      raise HDF5::Error, "Attribute already exists: #{attr_name}" if key?(attr_name)
+
+      write(attr_name, value)
+    end
+
+    def modify(attr_name, value)
+      @context&.ensure_open!(@dataset_id)
+      raise HDF5::Error, "Attribute not found: #{attr_name}" unless key?(attr_name)
+
+      attr_id = HDF5::FFI.H5Aopen(@dataset_id, attr_name, HDF5::DEFAULT_PROPERTY_LIST)
+      raise HDF5::Error, "Failed to open attribute: #{attr_name}" if attr_id < 0
+
+      type_id = HDF5::FFI.H5Aget_type(attr_id)
+      space_id = HDF5::FFI.H5Aget_space(attr_id)
+      raise HDF5::Error, "Failed to inspect attribute: #{attr_name}" if type_id < 0 || space_id < 0
+
+      if HDF5::FFI.H5Tget_class(type_id) == :H5T_STRING
+        unless HDF5::StringCodec.variable?(type_id)
+          raise UnsupportedTypeError, 'Fixed-length string attributes are not yet supported'
+        end
+
+        string_values, string_shape = HDF5::StringCodec.normalize_data(value)
+        unless string_shape == attribute_shape(space_id)
+          raise HDF5::ShapeError,
+                'Attribute shape must not change when modifying'
+        end
+
+        buffer, _string_pointers = HDF5::StringCodec.buffer_for_values(string_values)
+        status = HDF5::FFI.H5Awrite(attr_id, type_id, buffer)
+      else
+        dtype_object = DType.for_hdf5(type_id)
+        values = HDF5::DataHelpers.normalize_data(value, label: 'Attribute data')
+        expected_shape = attribute_shape(space_id)
+        raise HDF5::Error, 'Attribute shape must not change when modifying' unless values.shape == expected_shape
+
+        converted = dtype_object.numo_class.cast(values)
+        status = HDF5::FFI.H5Awrite(attr_id, dtype_object.memory_type_id, HDF5::DataHelpers.buffer_for(converted))
+      end
+      raise HDF5::Error, "Failed to modify attribute: #{attr_name}" if status < 0
+
+      value
+    ensure
+      HDF5::FFI.H5Sclose(space_id) if space_id && space_id >= 0
+      HDF5::FFI.H5Tclose(type_id) if type_id && type_id >= 0
+      HDF5::FFI.H5Aclose(attr_id) if attr_id && attr_id >= 0
+    end
+
     def write(attr_name, value)
-      values = normalize_data(value)
-      datatype_id = datatype_id_for(values)
+      @context&.ensure_open!(@dataset_id)
+      string_data = HDF5::StringCodec.string_data?(value)
+      string_values, string_shape = HDF5::StringCodec.normalize_data(value) if string_data
+      values = HDF5::DataHelpers.normalize_data(value, label: 'Attribute data') unless string_data
+      dtype_object = DType.for_numo(values) unless string_data
+      type_id = string_data ? HDF5::StringCodec.datatype_id : dtype_object.storage_type_id
 
       exists = HDF5::FFI.H5Aexists(@dataset_id, attr_name)
       raise HDF5::Error, "Failed to check attribute existence: #{attr_name}" if exists.negative?
@@ -79,78 +192,57 @@ module HDF5
         raise HDF5::Error, "Failed to replace attribute: #{attr_name}" if status < 0
       end
 
-      dims = ::FFI::MemoryPointer.new(:ulong_long, 1)
-      dims.write_array_of_ulong_long([values.length])
-      dataspace_id = HDF5::FFI.H5Screate_simple(1, dims, nil)
+      dataspace_id = create_dataspace(string_data ? string_shape : values.shape)
       raise HDF5::Error, 'Failed to create attribute dataspace' if dataspace_id < 0
 
       attr_id = HDF5::FFI.H5Acreate2(
         @dataset_id,
         attr_name,
-        datatype_id,
+        type_id,
         dataspace_id,
         HDF5::DEFAULT_PROPERTY_LIST,
         HDF5::DEFAULT_PROPERTY_LIST
       )
       raise HDF5::Error, "Failed to create attribute: #{attr_name}" if attr_id < 0
 
-      buffer = buffer_for(values)
-      status = HDF5::FFI.H5Awrite(attr_id, datatype_id, buffer)
+      buffer, _string_pointers = if string_data
+                                   HDF5::StringCodec.buffer_for_values(string_values)
+                                 else
+                                   [HDF5::DataHelpers.buffer_for(values), nil]
+                                 end
+      memory_type_id = string_data ? type_id : dtype_object.memory_type_id
+      status = HDF5::FFI.H5Awrite(attr_id, memory_type_id, buffer)
       raise HDF5::Error, "Failed to write attribute: #{attr_name}" if status < 0
 
       value
     ensure
       HDF5::FFI.H5Aclose(attr_id) if attr_id && attr_id >= 0
       HDF5::FFI.H5Sclose(dataspace_id) if dataspace_id && dataspace_id >= 0
+      HDF5::FFI.H5Tclose(type_id) if string_data && type_id && type_id >= 0
     end
 
     private
 
-    def normalize_data(value)
-      values = value.is_a?(Array) ? value : [value]
-      raise HDF5::Error, 'Attribute data must not be empty' if values.empty?
-      raise HDF5::Error, 'Nested arrays are not supported for attributes' if values.any? { |item| item.is_a?(Array) }
+    def attribute_shape(space_id)
+      rank = HDF5::FFI.H5Sget_simple_extent_ndims(space_id)
+      raise HDF5::Error, 'Failed to get attribute rank' if rank < 0
+      return [] if rank.zero?
 
-      values
-    end
+      dimensions = ::FFI::MemoryPointer.new(:ulong_long, rank)
+      status = HDF5::FFI.H5Sget_simple_extent_dims(space_id, dimensions, nil)
+      raise HDF5::Error, 'Failed to get attribute shape' if status < 0
 
-    def datatype_id_for(values)
-      if values.all? { |item| item.is_a?(Integer) }
-        validate_native_int_range!(values)
-        HDF5::FFI.H5T_NATIVE_INT
-      elsif values.all? { |item| item.is_a?(Numeric) }
-        HDF5::FFI.H5T_NATIVE_DOUBLE
-      else
-        raise HDF5::Error, 'Only numeric attribute data is supported'
-      end
+      dimensions.read_array_of_uint64(rank)
     end
 
-    def buffer_for(values)
-      if values.all? { |item| item.is_a?(Integer) }
-        buffer = ::FFI::MemoryPointer.new(:int, values.length)
-        buffer.write_array_of_int(values)
-      else
-        buffer = ::FFI::MemoryPointer.new(:double, values.length)
-        buffer.write_array_of_double(values.map(&:to_f))
-      end
-
-      buffer
-    end
+    def create_dataspace(shape)
+      return HDF5::FFI.H5Screate(:H5S_SCALAR) if shape.empty?
 
-    def native_int_bounds
-      bits = ::FFI.type_size(:int) * 8
-      max = (1 << (bits - 1)) - 1
-      min = -(1 << (bits - 1))
-      [min, max]
+      dimensions = ::FFI::MemoryPointer.new(:ulong_long, shape.length)
+      dimensions.write_array_of_ulong_long(shape)
+      HDF5::FFI.H5Screate_simple(shape.length, dimensions, nil)
     end
 
-    def validate_native_int_range!(values)
-      min, max = native_int_bounds
-      out_of_range = values.find { |value| value < min || value > max }
-      return unless out_of_range
-
-      raise HDF5::Error,
-            "Integer value #{out_of_range} is outside native int range (#{min}..#{max}). Use a smaller value."
-    end
+    prepend FileContext.guard(:[], :[]=, :keys, :key?, :delete, :create, :modify, :write)
   end
 end

data/lib/hdf5/data_helpers.rb

@@ -0,0 +1,54 @@
+module HDF5
+  module DataHelpers
+    module_function
+
+    def normalize_data(data, label: 'Data')
+      return data if data.is_a?(Numo::NArray) && DType.for_numo(data)
+
+      values = data.is_a?(Array) ? data.flatten : [data]
+      raise HDF5::Error, "#{label} must not be empty" if values.empty?
+
+      dtype = if values.all? { |value| value.is_a?(Integer) }
+                DType.for_symbol(:int64)
+              elsif values.all? { |value| [true, false].include?(value) }
+                DType.for_symbol(:bool)
+              elsif values.all? { |value| value.is_a?(Numeric) } && values.any? { |value| value.is_a?(Complex) }
+                DType.for_symbol(:complex128)
+              elsif values.all? { |value| value.is_a?(Numeric) }
+                DType.for_symbol(:float64)
+              else
+                raise HDF5::Error, "Only numeric #{label.downcase} is supported"
+              end
+      normalized = dtype.kind == :bool ? normalize_booleans(data) : data
+      dtype.numo_class.cast(normalized)
+    end
+
+    def buffer_for(narray)
+      dtype = DType.for_numo(narray)
+      binary = if dtype.kind == :bool
+                 narray.to_a.flatten.map { |value| value.zero? ? 0 : 1 }.pack('C*')
+               else
+                 narray.to_binary
+               end
+      expected_bytes = narray.size * dtype.itemsize
+      raise HDF5::Error, 'Numo binary representation has an unexpected size' unless binary.bytesize == expected_bytes
+
+      ::FFI::MemoryPointer.new(:char, expected_bytes).tap { |buffer| buffer.put_bytes(0, binary) }
+    end
+
+    def from_binary(dtype, binary, shape)
+      return dtype.numo_class.from_binary(binary, shape) unless dtype.kind == :bool
+
+      bytes = binary.unpack('C*')
+      return Numo::Bit.new.store(bytes.first) if shape.empty?
+
+      Numo::UInt8.cast(bytes).reshape(*shape).ne(0)
+    end
+
+    def normalize_booleans(value)
+      return value.map { |item| normalize_booleans(item) } if value.is_a?(Array)
+
+      value ? 1 : 0
+    end
+  end
+end