ndav 0.0.7 → 0.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ddc0e563bff86b51222b18b00f243b94b0f2e5a69dad3e7694c80a349775bf30
4
- data.tar.gz: 0afbc73643e49372d2b5104358acdc7460b26d7e6cc8892d45a67e4530cbe3d9
3
+ metadata.gz: 95b06d29aad658c09480cbe88ad3030c77aca3bad0e98ea20d02f29168de76cc
4
+ data.tar.gz: 6fce2648d0ff6a4ed82741f9357194c1d5de8b2c81183d1d16de9804cb19adcb
5
5
  SHA512:
6
- metadata.gz: f3a264a0204009e7f6b7505cae42667abe159e7ce7cbc5a27ffa88578c5691138b1b938795e14083b6f4cc9605dbc53b70a3990df495443ec9f5a8ebd664d5b5
7
- data.tar.gz: bafa151218a7e1acb22acd915fe08acb5147228980a2f441cd928ba66ae4425fb4fcd7bf9509e8ac28f5113e555ee9927b9a71b3d13ca2064134f144a524958c
6
+ metadata.gz: edd30f7e86ef25155769684c2432383ae9f8fd4b788eaddd25067ebd99fd9f210f5d88a48018dd76cc8b5baab378a404d61f0610e98401aff0e10c2f6b891101
7
+ data.tar.gz: 917020fcfde03cfab43b4d47b631fba0e94d4c94a0b72f0b72080f933d50123874c50dcf682e391fa4dd86a61a8b770df94ec2af96c26cbbcf80baeb07b7e1cd
data/.yardopts CHANGED
@@ -1,4 +1,3 @@
1
1
  --title="NDAV - N-Dimensional Array View"
2
2
  -
3
3
  LICENSE.txt
4
- lib/ndav/ffi.rb
data/README.md CHANGED
@@ -112,7 +112,7 @@ On the other hand, it also exports MemoryView. You can pass NDAV arrays *directl
112
112
  samples = waveform.to_ndav
113
113
  whisper.full(params, samples)
114
114
 
115
- Additinally, classes that `include` `NDAV::MemoryViewable`, including `Numo::NArray`, `OnnxRuntime::OrtValue` and `Torch::Tensor`, export MemoryViews:
115
+ Additionally, classes that `include` `NDAV::MemoryViewable`, including `Numo::NArray`, `OnnxRuntime::OrtValue` and `Torch::Tensor`, export MemoryViews:
116
116
 
117
117
  memory_view = Fiddle::MemoryView.new(numo)
118
118
  memory_view = Fiddle::MemoryView.new(ort_value)
@@ -124,6 +124,10 @@ Notice that NDAV is just a memory view and libraries share a memory address. If
124
124
 
125
125
  Additionally, you potentially encounter odd data corruption or segmentation fault. These might be bugs in bridge libraries such as [ndav-numo-narray][]. As a user, you don't need to worry about this kind of memory management, but it's worth knowing such situations may occur.
126
126
 
127
+ ### Notice On Export Flags ###
128
+
129
+ When the first argument to `NDAV.new` or `.new_override` is a MemoryView producer, NDAV assumes it exports the same metadata - such as `byte_size`, `format`, and `shape` - regardless of any flags. If you find a MemoryView producer which exports different metadata dependent on the flags, please file an issue.
130
+
127
131
  INSTALLATION
128
132
  ------------
129
133
 
@@ -140,11 +144,11 @@ BRIDGES
140
144
 
141
145
  There are some bridges using NDAV:
142
146
 
143
- * {file:ndav/ffi ndav/ffi}: [`FFI::MemoryPointer`][FFI Pointer], [`FFI::Pointer`][FFI Pointer] <-> `NDAV`
144
147
  * [ndav-numo-narray][]: [`Numo::NArray`][Numo::NArray] <-> `NDAV`
145
148
  * [ndav-ort_value][]: [`OnnxRuntime::OrtValue`][ONNX Runtime Ruby] <-> `NDAV`
146
149
  * [ndav-torch-tensor][]: [`Torch::Tensor`][Torch.rb] <-> `NDAV`
147
150
  * [onnxruntime-torch-tensor][]: [`Torch::Tensor`][Torch.rb] <-> [`OnnxRuntime::OrtValue`][ONNX Runtime Ruby] via `NDAV`
151
+ * [ndav-magick-image][]: [`Magick::Image`][RMagick] <-> `NDAV`
148
152
 
149
153
  APPLICATIONS AND LIBRARIES WHICH USE NDAV
150
154
  -----------------------------------------
@@ -159,7 +163,7 @@ Refer to existing bridge implementations listed above to create your bridge.
159
163
 
160
164
  The points are:
161
165
 
162
- * Implement `FromNDAV#from_ndav`, `MemoryViewable#ndav_descriptor`, optionally `ToNDAV#to_ndav`, {NDAV.register register} them, and `NDAV.from_your_data` and `NDAV#to_your_data` are automatically derived and the array become to able to export a MemoryView
166
+ * Implement `FromNDAV#from_ndav`, `MemoryViewable#ndav_descriptor`, optionally `ToNDAV#to_ndav`, {NDAV.register register} them, and `NDAV.from_your_data` and `NDAV#to_your_data` are automatically derived and the array becomes able to export a MemoryView
163
167
  * When initializing NDAV object from your object, use `lifetime` keyword argument for {NDAV#initialize} effectively to prevent Ruby from GCing your object, which would lead to a dangling pointer
164
168
  * When initializing your object from NDAV object, keep NDAV object alive to prevent Ruby from GCing NDAV object, which would lead to a dangling pointer, [ndav-numo-narray][], for instance, embeds the NDAV object in an instance variable
165
169
 
@@ -183,13 +187,13 @@ BSD-2-Clause license. See {file:LICENSE.txt} file.
183
187
  [ONNX Runtime Ruby]: https://github.com/ankane/onnxruntime-ruby
184
188
  [Red Arrow]: https://github.com/apache/arrow/tree/main/ruby
185
189
  [Red Arrow Numo::NArray]: https://github.com/red-data-tools/red-arrow-numo-narray
186
- [ndav/ffi]: https://gitlab.com/KitaitiMakoto/ndav/-/blob/main/lib/ndav/ffi.rb
187
- [FFI Pointer]: https://github.com/ffi/ffi/wiki/Pointers
188
190
  [whispercpp]: https://github.com/ggml-org/whisper.cpp/tree/master/bindings/ruby
189
191
  [ndav-numo-narray]: https://gitlab.com/KitaitiMakoto/ndav-numo-narray
190
192
  [ndav-ort_value]: https://gitlab.com/KitaitiMakoto/ndav-ort_value
191
193
  [ndav-torch-tensor]: https://gitlab.com/KitaitiMakoto/ndav-torch-tensor
192
194
  [onnxruntime-torch-tensor]: https://gitlab.com/KitaitiMakoto/onnxruntime-torch-tensor
195
+ [ndav-magick-image]: https://gitlab.com/KitaitiMakoto/ndav-magick-image
196
+ [RMagick]: https://rmagick.github.io/
193
197
  [GTCRN]: https://gitlab.com/KitaitiMakoto/gtcrn
194
198
  [TorchAudio]: https://github.com/ankane/torchaudio-ruby
195
199
  [Itak]: https://gitlab.com/KitaitiMakoto/itak
data/Rakefile CHANGED
@@ -8,7 +8,7 @@ DL_BUILD_PATH = File.join("ext", DL_NAME)
8
8
  DL_PATH = File.join("lib", DL_NAME)
9
9
 
10
10
  SRC = FileList["ext/**.{h,c,rb}"]
11
- CLEAN.include SRC.select {|src| src.end_with?(".o", ".so", ".bundle", ".dll") }
11
+ CLEAN.include FileList["ext/**.{o,so,bundle,dll}"]
12
12
 
13
13
  task default: :test
14
14
 
data/ext/ndav.c CHANGED
@@ -1,7 +1,16 @@
1
1
  #include <ruby.h>
2
2
  #include <ruby/memory_view.h>
3
3
 
4
+ #ifndef NUM2PTR
5
+ #if SIZEOF_LONG == SIZEOF_VOIDP
6
+ #define NUM2PTR(x) ((void *)(NUM2LONG(x)))
7
+ #else
8
+ #define NUM2PTR(x) ((void *)(NUM2ULL(x)))
9
+ #endif // SIZEOF_LONG == SIZEOF_VOIDP
10
+ #endif // NUM2PTR
11
+
4
12
  static ID id_ndav_validated_descriptor;
13
+ static VALUE sym_data;
5
14
  static VALUE sym_addr;
6
15
  static VALUE sym_shape;
7
16
  static VALUE sym_strides;
@@ -14,9 +23,10 @@ static VALUE sym_item_size;
14
23
  static ID id_new;
15
24
  static VALUE cItemComponent;
16
25
 
17
- typedef struct {
18
- VALUE descriptor;
19
- } private_data_t;
26
+ struct private_data {
27
+ VALUE data;
28
+ ssize_t buf[];
29
+ };
20
30
 
21
31
  static bool
22
32
  fill_size_array(VALUE src, ssize_t *dest, ssize_t size)
@@ -31,23 +41,6 @@ fill_size_array(VALUE src, ssize_t *dest, ssize_t size)
31
41
  return true;
32
42
  }
33
43
 
34
- static void
35
- release_memview_common(rb_memory_view_t *view)
36
- {
37
- if (view->shape) {
38
- free((void *)view->shape);
39
- view->shape = NULL;
40
- }
41
- if (view->strides) {
42
- free((void *)view->strides);
43
- view->strides = NULL;
44
- }
45
- if (view->sub_offsets) {
46
- free((void *)view->sub_offsets);
47
- view->sub_offsets = NULL;
48
- }
49
- }
50
-
51
44
  static bool
52
45
  ndav_memory_viewable_get_memory_view(const VALUE ndav, rb_memory_view_t *view, int flags)
53
46
  {
@@ -69,67 +62,49 @@ ndav_memory_viewable_get_memory_view(const VALUE ndav, rb_memory_view_t *view, i
69
62
  }
70
63
  tmp.readonly = readonly;
71
64
  tmp.obj = ndav;
72
- // TODO: Commonalize
65
+ VALUE data = rb_hash_aref(desc, sym_data);
73
66
  VALUE item_size = rb_hash_aref(desc, sym_item_size);
74
67
  tmp.item_size = NUM2SSIZET(item_size);
75
68
  tmp.byte_size = byte_size;
76
69
  VALUE ndim = rb_hash_aref(desc, sym_ndim);
77
70
  tmp.ndim = NUM2SSIZET(ndim);
78
71
  VALUE format = rb_hash_aref(desc, sym_format);
79
- tmp.format = StringValueCStr(format);
80
-
81
- // TODO: Commonalize
72
+ char *view_format = StringValueCStr(format);
82
73
  VALUE shape = rb_hash_aref(desc, sym_shape);
83
- ssize_t *view_shape = (ssize_t *)malloc(sizeof(ssize_t) * tmp.ndim);
84
- if (!view_shape) {
85
- release_memview_common(&tmp);
86
- return false;
87
- }
88
- if (!fill_size_array(shape, view_shape, tmp.ndim)) {
89
- free(view_shape);
90
- release_memview_common(&tmp);
91
- return false;
92
- }
93
- tmp.shape = view_shape;
94
-
95
74
  VALUE strides = rb_hash_aref(desc, sym_strides);
96
- ssize_t *view_strides = (ssize_t *)malloc(sizeof(ssize_t) * tmp.ndim);
97
- if (!view_strides) {
98
- release_memview_common(&tmp);
99
- return false;
100
- }
101
- if (!fill_size_array(strides, view_strides, tmp.ndim)) {
102
- free(view_strides);
103
- release_memview_common(&tmp);
104
- return false;
105
- }
106
- tmp.strides = view_strides;
107
-
108
75
  VALUE sub_offsets = rb_hash_aref(desc, sym_sub_offsets);
109
- if (!NIL_P(sub_offsets)) {
110
- ssize_t *view_sub_offsets = (ssize_t *)malloc(sizeof(ssize_t) * tmp.ndim);
111
- if (!view_sub_offsets) {
112
- release_memview_common(&tmp);
76
+
77
+ size_t num_ary = NIL_P(sub_offsets) ? 2 : 3;
78
+ long format_len = RSTRING_LEN(format);
79
+ struct private_data *private_data = malloc(sizeof(struct private_data) + sizeof(ssize_t) * tmp.ndim * num_ary + sizeof(char) * (format_len + 1));
80
+ if (!private_data) {
113
81
  return false;
114
- }
115
- if (!fill_size_array(sub_offsets, view_sub_offsets, tmp.ndim)) {
116
- free(view_sub_offsets);
117
- release_memview_common(&tmp);
82
+ }
83
+ ssize_t pdt_idx;
84
+ if (!fill_size_array(shape, &private_data->buf[0], tmp.ndim)) {
85
+ free(private_data);
118
86
  return false;
119
- }
120
- tmp.sub_offsets = view_sub_offsets;
121
87
  }
122
-
123
- private_data_t *private_data = malloc(sizeof(private_data_t));
124
- if (!private_data) {
125
- release_memview_common(&tmp);
126
- rb_warn("failed to alloc private_data");
127
- return false;
88
+ tmp.shape = &private_data->buf[0];
89
+ if (!fill_size_array(strides, &private_data->buf[tmp.ndim], tmp.ndim)) {
90
+ free(private_data);
91
+ return false;
128
92
  }
129
- private_data->descriptor = Qnil;
130
- rb_gc_register_address(&private_data->descriptor);
131
- private_data->descriptor = desc;
132
- tmp.private_data = private_data;
93
+ tmp.strides = &private_data->buf[tmp.ndim];
94
+ if (!NIL_P(sub_offsets)) {
95
+ pdt_idx = tmp.ndim * 2;
96
+ if (!fill_size_array(sub_offsets, &private_data->buf[pdt_idx], tmp.ndim)) {
97
+ free(private_data);
98
+ return false;
99
+ }
100
+ tmp.sub_offsets = &private_data->buf[pdt_idx];
101
+ }
102
+ pdt_idx = tmp.ndim * num_ary;
103
+ memcpy((void *)&private_data->buf[pdt_idx], view_format, format_len + 1);
104
+ tmp.format = (char *)&private_data->buf[pdt_idx];
105
+ private_data->data = data;
106
+ rb_gc_register_address(&private_data->data);
107
+ tmp.private_data = (void *)private_data;
133
108
 
134
109
  *view = tmp;
135
110
 
@@ -139,13 +114,12 @@ ndav_memory_viewable_get_memory_view(const VALUE ndav, rb_memory_view_t *view, i
139
114
  static bool
140
115
  ndav_memory_viewable_release_memory_view(const VALUE ndav, rb_memory_view_t *view)
141
116
  {
142
- release_memview_common(view);
143
117
  if (view->private_data) {
144
- private_data_t *private_data = (private_data_t *)view->private_data;
145
- rb_gc_unregister_address(&private_data->descriptor);
146
- free(private_data);
147
- view->private_data = NULL;
118
+ struct private_data *private_data = view->private_data;
119
+ rb_gc_unregister_address(&private_data->data);
148
120
  }
121
+ free(view->private_data);
122
+ view->private_data = NULL;
149
123
 
150
124
  return true;
151
125
  }
@@ -246,6 +220,7 @@ void
246
220
  Init_ndav(void)
247
221
  {
248
222
  id_ndav_validated_descriptor = rb_intern("ndav_validated_descriptor");
223
+ sym_data = ID2SYM(rb_intern("data"));
249
224
  sym_addr = ID2SYM(rb_intern("addr"));
250
225
  sym_shape = ID2SYM(rb_intern("shape"));
251
226
  sym_strides = ID2SYM(rb_intern("strides"));
@@ -24,7 +24,7 @@ class NDAV
24
24
  unless mdl.const_defined?(:ToNDAV)
25
25
  mdl.const_set(:ToNDAV, Module.new {
26
26
  def to_ndav(lifetime: self, **)
27
- ::NDAV.new(self, lifetime:, **)
27
+ ::NDAV.new_override(self, lifetime:, **)
28
28
  end
29
29
  })
30
30
  end
@@ -54,7 +54,7 @@ class NDAV
54
54
  elsif array.respond_to?(:to_ndav)
55
55
  array.to_ndav(*, **)
56
56
  else
57
- ::NDAV.new(array, *, **)
57
+ ::NDAV.new_override(array, *, **)
58
58
  end
59
59
  end
60
60
  end
data/lib/ndav/flags.rb CHANGED
@@ -13,11 +13,13 @@ class NDAV
13
13
 
14
14
  module_function
15
15
 
16
- def decode(flags)
16
+ def decode_for_request(flags)
17
17
  FLAGS.each_with_object({}) {|(name, value), state|
18
18
  state[name] = case name
19
19
  when :simple
20
20
  flags == value
21
+ when :row_major, :column_major
22
+ flags & FLAGS[:any_contiguous] == FLAGS[name]
21
23
  when :any_contiguous
22
24
  (flags & FLAGS[:row_major]) == FLAGS[:row_major] ||
23
25
  (flags & FLAGS[:column_major]) == FLAGS[:column_major]
@@ -26,5 +28,9 @@ class NDAV
26
28
  end
27
29
  }
28
30
  end
31
+
32
+ def combine(*flags)
33
+ flags.reduce(FLAGS[:simple]) {|encoded, flag| encoded | FLAGS[flag]}
34
+ end
29
35
  end
30
36
  end
@@ -5,6 +5,7 @@ class NDAV
5
5
  # +#ndav_descriptor+ 's keyword arguments are flags of MemoryView including
6
6
  # +simple+, +writable+, +format+, +multi_dimensional+, +strides+, +row_major+, +column_major+, +any_contiguous+, +indirect+.
7
7
  # For the meaning of them, see MemoryView document.
8
+ # +any_contiguous+ ' meaning is ambiguous but assumed as +row_major+ **OR** +column_major+ is requested here.
8
9
  #
9
10
  # It must return a Hash whose keys includes:
10
11
  #
@@ -52,66 +53,85 @@ class NDAV
52
53
 
53
54
  private
54
55
 
56
+ # TODO: Distinguish validation error and other errors
55
57
  def ndav_validated_descriptor(flags = ::NDAV::Flags::FLAGS[:simple])
56
- flags = ::NDAV::Flags.decode(flags)
58
+ flags = ::NDAV::Flags.decode_for_request(flags)
57
59
  desc = ndav_descriptor(**flags).to_h
58
60
  desc => {data:, shape:, strides:, format:, byte_size:, readonly?: readonly}
59
61
  sub_offsets = desc[:sub_offsets]
60
62
 
61
63
  [:data, :shape, :strides, :format, :byte_size, :readonly?].each do |key|
62
64
  if desc[key].nil?
63
- warn ":#{key} is nil"
64
- return false
65
+ raise ":#{key} is nil"
65
66
  end
66
67
  end
67
68
  [:shape, :strides].each do |key|
68
69
  unless desc[key].instance_of? Array
69
- warn "#{key} not Array"
70
- return false
70
+ raise "#{key} not Array"
71
71
  end
72
72
  end
73
- if sub_offsets && !sub_offsets.instance_of?(Array)
74
- warn "sub_offsets not Array"
75
- return false
73
+ ndim = shape.length
74
+ raise "wrong length of strides" unless strides.length == ndim
75
+ if sub_offsets
76
+ raise "sub_offsets not Array" unless sub_offsets.instance_of?(Array)
77
+ raise "wrong length of sub_offsets" unless sub_offsets.length == ndim
76
78
  end
79
+ format = format.to_s
77
80
  item_size = ::NDAV::ITEM_SIZES[format]
78
- unless shape.reduce(item_size, :*) == byte_size
79
- warn "item size from format and shape not match byte_size"
80
- return false
81
- end
82
81
 
83
82
  if flags[:writable] && readonly
84
- warn "not writable"
85
- return false
83
+ raise "not writable"
86
84
  end
87
85
 
88
86
  # TODO: Research the meaning of MULTI_DIMENSIONAL and validate it
89
87
 
90
88
  if flags[:indirect] && !sub_offsets
91
- warn "indirect requested but sub_offsets is NULL"
92
- return false
89
+ raise "indirect requested but sub_offsets is NULL"
93
90
  end
94
91
 
95
92
  row_major_contiguous = ::NDAV.row_major_contiguous?(format:, shape:, strides:)
96
93
  column_major_contiguous = ::NDAV.column_major_contiguous?(format:, shape:, strides:)
97
94
  if flags[:row_major] && !row_major_contiguous
98
- return false
95
+ raise "row-major contiguous requested but it isn't"
99
96
  end
100
97
  if flags[:column_major] && !column_major_contiguous
101
- return false
98
+ raise "column-major contiguous requested but it isn't"
102
99
  end
103
100
  if flags[:any_contiguous] && !row_major_contiguous && !column_major_contiguous
104
- return false
101
+ raise "contiguous requested but it isn't"
102
+ end
103
+ if row_major_contiguous || column_major_contiguous
104
+ unless shape.reduce(item_size, :*) == byte_size
105
+ raise "item size from format and shape not match byte_size"
106
+ end
107
+ else
108
+ max_offset = 0
109
+ min_offset = 0
110
+ shape.zip(strides) do |s, stride|
111
+ next if s == 0
112
+
113
+ extent = (s - 1) * stride
114
+ max_offset += [extent, 0].max
115
+ min_offset += [extent, 0].min
116
+ end
117
+ if min_offset < 0
118
+ raise "min offset must be positive"
119
+ end
120
+ if max_offset + item_size > byte_size
121
+ raise "max offset too large"
122
+ end
105
123
  end
106
124
 
107
125
  desc.merge(
108
126
  addr: ::Fiddle::Pointer.to_ptr(data).to_i,
109
- ndim: shape.length,
127
+ ndim:,
128
+ format:,
110
129
  item_size:,
111
130
  sub_offsets:
112
131
  )
113
132
  rescue => err
114
133
  warn err
134
+ warn err.backtrace.join($/) if $VERBOSE
115
135
  return false
116
136
  end
117
137
  end
data/lib/ndav.rb CHANGED
@@ -17,7 +17,7 @@ class NDAV
17
17
  8 => "q"
18
18
  }
19
19
 
20
- attr_reader :ndim, :shape, :format, :item_size, :byte_size, :strides, :sub_offsets, :to_s, :lifetime
20
+ attr_reader :ndim, :shape, :format, :item_size, :byte_size, :strides, :sub_offsets, :lifetime
21
21
 
22
22
  class << self
23
23
  # * Doesn't modify byte_size
@@ -104,58 +104,58 @@ class NDAV
104
104
  end
105
105
  end
106
106
 
107
- def initialize(obj, readonly: nil, format: nil, item_size: nil, shape: nil, strides: nil, sub_offsets: nil, lifetime: nil)
108
- # We cannot determine flags until ndav_descriptor is called but some MemoryView producer like Fiddle::Pointer and IO::Buffer determines some members by flags.
109
- # For instance, Fiddle::Pointer's MemoryView is readonly when writable flag is not passed and is not readonly when writable flag is passed.
110
- # How should we solve this mismatching? Delaying to determine the flags?
111
- # Considers when it becomes actual problem.
112
- @fmv = obj.kind_of?(::Fiddle::MemoryView) ? obj : ::Fiddle::MemoryView.new(obj)
113
- # Fiddle::MemoryView#to_s is zero-copy
114
- # Fiddle::MemoryView#to_s embeds obj into the String, so we can prevent GC from collecting it
115
- @to_s = @fmv.to_s
107
+ def initialize(obj, format: nil, item_size: nil, shape: nil, strides: nil, sub_offsets: nil, lifetime: nil)
108
+ @obj = obj
116
109
  @lifetime = lifetime
117
- @byte_size = @fmv.byte_size
118
-
119
- case [format, item_size]
120
- in [nil, nil]
121
- @format = @fmv.format || "C"
122
- @item_size = @fmv.item_size
123
- in [_, nil]
124
- @format = format
125
- @item_size = ITEM_SIZES[@format]
126
- in [nil, _]
127
- @item_size = item_size
128
- @format = DEFAULT_FORMATS[@item_size] || @fmv.format
129
- else
130
- @format = format
131
- @item_size = item_size
132
- end
110
+ is_fmv = @obj.kind_of? Fiddle::MemoryView
111
+ fmv = is_fmv ? @obj : Fiddle::MemoryView.new(@obj)
112
+ begin
113
+ @byte_size = fmv.byte_size
114
+
115
+ case [format, item_size]
116
+ in [nil, nil]
117
+ @format = fmv.format || "C"
118
+ @item_size = fmv.item_size
119
+ in [_, nil]
120
+ @format = format
121
+ @item_size = ITEM_SIZES[@format]
122
+ in [nil, _]
123
+ @item_size = item_size
124
+ @format = DEFAULT_FORMATS[@item_size] || fmv.format
125
+ else
126
+ @format = format
127
+ @item_size = item_size
128
+ end
133
129
 
134
- case [shape, @byte_size]
135
- in [nil, nil]
136
- @shape = @fmv.shape
137
- @byte_size = @shape.reduce(@item_size, :*)
138
- in [_, nil]
139
- @shape = shape
140
- @byte_size = @shape.reduce(@item_size, :*)
141
- in [nil, _]
142
- @shape = @fmv.shape || [@byte_size / @item_size]
143
- else
144
- @shape = shape
145
- end
130
+ case [shape, @byte_size]
131
+ in [nil, nil]
132
+ @shape = fmv.shape
133
+ @byte_size = @shape.reduce(@item_size, :*)
134
+ in [_, nil]
135
+ @shape = shape
136
+ @byte_size = @shape.reduce(@item_size, :*)
137
+ in [nil, _]
138
+ @shape = fmv.shape || [@byte_size / @item_size]
139
+ else
140
+ @shape = shape
141
+ end
146
142
 
147
- @ndim = @shape.size
148
- @readonly = readonly.nil? ? @fmv.readonly? : readonly
149
- @strides = strides || @fmv.strides || self.class.default_strides(shape: @shape, item_size: @item_size)
150
- @sub_offsets = sub_offsets || @fmv.sub_offsets
151
- @row_major_contiguous = self.class.row_major_contiguous?(format: @format, shape: @shape, strides: @strides)
152
- @column_major_contiguous = self.class.column_major_contiguous?(format: @format, shape: @shape, strides: @strides)
143
+ @ndim = @shape.size
144
+ @strides = strides || fmv.strides || self.class.default_strides(shape: @shape, item_size: @item_size)
145
+ @sub_offsets = sub_offsets || fmv.sub_offsets
146
+ @row_major_contiguous = self.class.row_major_contiguous?(format: @format, shape: @shape, strides: @strides)
147
+ @column_major_contiguous = self.class.column_major_contiguous?(format: @format, shape: @shape, strides: @strides)
148
+ ensure
149
+ fmv.release unless is_fmv
150
+ end
153
151
 
154
152
  validate
155
153
  end
156
154
 
157
- def readonly?
158
- @readonly
155
+ def to_s
156
+ # Fiddle::MemoryView#to_s is zero-copy
157
+ # Fiddle::MemoryView#to_s embeds obj into the String, so we can prevent GC from collecting it
158
+ (@obj.kind_of?(Fiddle::MemoryView) ? @obj : Fiddle::MemoryView.new(@obj)).to_s
159
159
  end
160
160
 
161
161
  def item_desc
@@ -171,27 +171,33 @@ class NDAV
171
171
  end
172
172
 
173
173
  def to_ptr
174
- ::Fiddle::Pointer[@to_s]
174
+ ::Fiddle::Pointer[to_s]
175
175
  end
176
176
  alias to_fiddle_pointer to_ptr
177
177
 
178
- def release
179
- @fmv.release
180
- @fmv = @ndim = @readonly = @shape = @format = @item_size = @byte_size = @strides = @sub_offsets = @lifetime = @to_s = @row_major_contiguous = @column_major_contiguous = nil
181
- end
182
-
183
178
  private
184
179
 
185
- def ndav_descriptor(**)
186
- return false unless @fmv
187
-
180
+ def ndav_descriptor(writable:, **)
181
+ fmv = if @obj.kind_of? Fiddle::MemoryView
182
+ raise "not writable" if writable && @obj.readonly?
183
+ @obj
184
+ else
185
+ if Fiddle::VERSION <= "1.1.8"
186
+ fmv = Fiddle::MemoryView.new(@obj)
187
+ raise "not writable" if writable && fmv.readonly?
188
+ fmv
189
+ else
190
+ flgs = Flags.combine(writable ? :writable : :simple)
191
+ Fiddle::MemoryView.new(@obj, flgs)
192
+ end
193
+ end
188
194
  {
189
- data: @to_s,
195
+ data: fmv.to_s,
190
196
  shape: @shape,
191
197
  strides: @strides,
192
198
  format: @format,
193
199
  byte_size: @byte_size,
194
- readonly?: @readonly,
200
+ readonly?: fmv.readonly?,
195
201
  sub_offsets: @sub_offsets
196
202
  }
197
203
  end
@@ -206,14 +212,14 @@ class NDAV
206
212
  r = @byte_size % @item_size
207
213
  raise ArgumentError, "byte_size must be n-times of item_size" unless r.zero?
208
214
 
209
- raise ArgumentError, "shape length doen't match ndim" unless @shape.length == @ndim
215
+ raise ArgumentError, "shape length doesn't match ndim" unless @shape.length == @ndim
210
216
  if @row_major_contiguous || @column_major_contiguous
211
217
  unless @byte_size == @shape.reduce(@item_size, :*)
212
- raise ArgumentError, "shape not match byte_size"
218
+ raise ArgumentError, "shape not match byte_size. byte_size: #{@byte_size}, item_size: #{@item_size}, shape: #{@shape}"
213
219
  end
214
220
  end
215
221
 
216
- raise ArgumentError, "strides length doen't match ndim" unless @strides.length == @ndim
222
+ raise ArgumentError, "strides length doesn't match ndim" unless @strides.length == @ndim
217
223
  if @row_major_contiguous
218
224
  stride = @item_size
219
225
  @strides.reverse_each.with_index do |s, i|
@@ -234,7 +240,7 @@ class NDAV
234
240
  # NDAV uses Fiddle::MemoryView#to_s and it doesn't allow negative min offset
235
241
  # Some MemoryView might allow negative min offset
236
242
  # Remove this check if such the MemoryView producer is found
237
- raise ArgumentError, "min offset must be positive"
243
+ raise ArgumentError, "min offset must be non-negative"
238
244
  end
239
245
  raise ArgumentError, "max offset too large" if max_offset + @item_size > @byte_size
240
246
  end
data/ndav.gemspec CHANGED
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = "ndav"
3
- s.version = "0.0.7"
3
+ s.version = "0.0.9"
4
4
  s.summary = "N-Dimensional Array View"
5
5
  s.authors = ["Kitaiti Makoto"]
6
6
  s.licenses = ["BSD-2-Clause"]
@@ -20,6 +20,5 @@ Gem::Specification.new do |s|
20
20
  s.add_development_dependency "rubygems-tasks"
21
21
  s.add_development_dependency "rubygems-requirements-system"
22
22
  s.add_development_dependency "red-arrow"
23
- s.add_development_dependency "ffi"
24
23
  s.add_development_dependency "yard"
25
24
  end
data/test/helper.rb CHANGED
@@ -4,7 +4,6 @@ require "test/unit/notify"
4
4
  require "fiddle"
5
5
  require "arrow"
6
6
  require "ndav"
7
- require "ndav/ffi"
8
7
 
9
8
  class TestBase < Test::Unit::TestCase
10
9
  private
@@ -16,7 +15,6 @@ class TestBase < Test::Unit::TestCase
16
15
  assert_equal [2], array.strides
17
16
  assert_equal 2, array.item_size
18
17
  assert_equal 6, array.byte_size
19
- assert_true array.readonly?
20
18
  assert_equal [1, 2, 3].pack("s*"), array.to_s
21
19
  end
22
20
  end
data/test/test_flags.rb CHANGED
@@ -4,25 +4,19 @@ class TestFlags < TestBase
4
4
  include NDAV::Flags
5
5
 
6
6
  def test_any_contiguous
7
- decoded = decode(combine_flags(:any_contiguous))
8
- assert_true decoded[:row_major]
9
- assert_true decoded[:column_major]
7
+ decoded = decode_for_request(combine(:any_contiguous))
8
+ assert_false decoded[:row_major]
9
+ assert_false decoded[:column_major]
10
10
  assert_true decoded[:any_contiguous]
11
11
 
12
- decoded = decode(combine_flags(:row_major))
12
+ decoded = decode_for_request(combine(:row_major))
13
13
  assert_true decoded[:row_major]
14
14
  assert_false decoded[:column_major]
15
15
  assert_true decoded[:any_contiguous]
16
16
 
17
- decoded = decode(combine_flags(:column_major))
17
+ decoded = decode_for_request(combine(:column_major))
18
18
  assert_false decoded[:row_major]
19
19
  assert_true decoded[:column_major]
20
20
  assert_true decoded[:any_contiguous]
21
21
  end
22
-
23
- private
24
-
25
- def combine_flags(*flags)
26
- flags.reduce(0) {|composed, name| composed | FLAGS[name]}
27
- end
28
22
  end
data/test/test_ndav.rb CHANGED
@@ -29,6 +29,27 @@ class TestNDAV < TestBase
29
29
  assert_int_array ndav
30
30
  end
31
31
 
32
+ def test_flags
33
+ omit if Fiddle::VERSION <= "1.1.8"
34
+
35
+ flags = NDAV::Flags.combine(:writable)
36
+ ptr = Fiddle::Pointer[[1, 2, 3].pack("s*")]
37
+ ndav = NDAV.new(ptr)
38
+
39
+ Fiddle::MemoryView.export ndav do |memview|
40
+ assert_true memview.readonly?
41
+ end
42
+
43
+ Fiddle::MemoryView.export ndav, flags do |memview|
44
+ assert_false memview.readonly?
45
+
46
+ p = Fiddle::Pointer[memview.to_s]
47
+ p[0] = 4
48
+ assert_equal [4, 2, 3], memview.to_s.unpack("s*")
49
+ end
50
+ assert_equal [4, 2, 3], ndav.to_s.unpack("s*")
51
+ end
52
+
32
53
  def test_string
33
54
  str = [1, 2, 3].pack("s*")
34
55
  ndav = NDAV.from_string(str, format: "s")
@@ -57,18 +78,6 @@ class TestNDAV < TestBase
57
78
  assert_true ndav.column_major_contiguous?
58
79
  end
59
80
 
60
- def test_release
61
- ndav = NDAV.new(@array)
62
-
63
- ndav.release
64
- [:ndim, :shape, :format, :item_size, :byte_size, :strides, :sub_offsets, :lifetime, :to_s].each do |attr|
65
- assert_nil ndav.send(attr)
66
- end
67
- assert_nil ndav.readonly?
68
- assert_nil ndav.row_major_contiguous?
69
- assert_nil ndav.column_major_contiguous?
70
- end
71
-
72
81
  def test_parse_item_format
73
82
  assert_equal [
74
83
  NDAV::ItemComponent.new("f", false, IO::Buffer::HOST_ENDIAN == IO::Buffer::LITTLE_ENDIAN, 0, 4, 1)
@@ -190,17 +199,5 @@ class TestNDAV < TestBase
190
199
  assert_equal 6, memview.byte_size
191
200
  end
192
201
  end
193
-
194
- def test_overwrite_readonly?
195
- assert_raise ArgumentError do
196
- NDAV.new(@array, readonly: false)
197
- end
198
-
199
- ndav = NDAV.new_override(@array, readonly: false)
200
- Fiddle::MemoryView.export ndav do |memview|
201
- assert_false memview.readonly?
202
- assert_equal 6, memview.byte_size
203
- end
204
- end
205
202
  end
206
203
  end
data/test/test_ractor.rb CHANGED
@@ -3,6 +3,11 @@ require_relative "helper"
3
3
  class TestRactor < TestBase
4
4
  ractor
5
5
 
6
+ def setup
7
+ omit if RUBY_VERSION < "4.0.0"
8
+ pend
9
+ end
10
+
6
11
  def test_ractor
7
12
  ractor = Ractor.new {
8
13
  Ractor.receive
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ndav
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.7
4
+ version: 0.0.9
5
5
  platform: ruby
6
6
  authors:
7
7
  - Kitaiti Makoto
@@ -135,20 +135,6 @@ dependencies:
135
135
  - - ">="
136
136
  - !ruby/object:Gem::Version
137
137
  version: '0'
138
- - !ruby/object:Gem::Dependency
139
- name: ffi
140
- requirement: !ruby/object:Gem::Requirement
141
- requirements:
142
- - - ">="
143
- - !ruby/object:Gem::Version
144
- version: '0'
145
- type: :development
146
- prerelease: false
147
- version_requirements: !ruby/object:Gem::Requirement
148
- requirements:
149
- - - ">="
150
- - !ruby/object:Gem::Version
151
- version: '0'
152
138
  - !ruby/object:Gem::Dependency
153
139
  name: yard
154
140
  requirement: !ruby/object:Gem::Requirement
@@ -179,7 +165,6 @@ files:
179
165
  - ext/ndav.c
180
166
  - lib/ndav.rb
181
167
  - lib/ndav/converter.rb
182
- - lib/ndav/ffi.rb
183
168
  - lib/ndav/flags.rb
184
169
  - lib/ndav/import.rb
185
170
  - lib/ndav/item_component.rb
@@ -188,7 +173,6 @@ files:
188
173
  - test/helper.rb
189
174
  - test/test_converter.rb
190
175
  - test/test_export.rb
191
- - test/test_ffi.rb
192
176
  - test/test_flags.rb
193
177
  - test/test_ndav.rb
194
178
  - test/test_package.rb
@@ -212,7 +196,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
212
196
  - !ruby/object:Gem::Version
213
197
  version: '0'
214
198
  requirements: []
215
- rubygems_version: 4.0.16
199
+ rubygems_version: 4.0.20
216
200
  specification_version: 4
217
201
  summary: N-Dimensional Array View
218
202
  test_files: []
data/lib/ndav/ffi.rb DELETED
@@ -1,97 +0,0 @@
1
- require "ffi"
2
-
3
- class NDAV
4
- module FFI
5
- TYPE_SIZE_TO_FORMAT = {
6
- 1 => "c",
7
- 2 => "s",
8
- 4 => "l",
9
- 8 => "q"
10
- }
11
-
12
- module MemoryPointer
13
- module FromNDAV
14
- # NOTE: Copies data
15
- def from_ndav(array)
16
- from_string(array.to_s)
17
- end
18
- end
19
-
20
- module ToNDAV
21
- def to_ndav(format: TYPE_SIZE_TO_FORMAT[type_size], lifetime: self, **)
22
- shape = [size / type_size]
23
- byte_size = shape.reduce(::NDAV::ITEM_SIZES[format], :*)
24
- ptr = ::Fiddle::Pointer.new(address, byte_size)
25
- ::NDAV.new(ptr, shape:, format:, lifetime:, **)
26
- end
27
- end
28
-
29
- module MemoryViewable
30
- def ndav_descriptor(**)
31
- shape = [size / type_size]
32
- format = TYPE_SIZE_TO_FORMAT[type_size]
33
- item_size = ::NDAV::ITEM_SIZES[format]
34
- byte_size = shape.reduce(item_size, :*)
35
- ptr = ::Fiddle::Pointer.new(address, byte_size)
36
-
37
- {
38
- data: Fiddle::MemoryView.new(ptr).to_s,
39
- shape:,
40
- strides: ::NDAV.default_strides(shape:, item_size:),
41
- format:,
42
- byte_size:,
43
- readonly?: true
44
- }
45
- end
46
- end
47
-
48
- module Converter
49
- def FFIMemoryPointer(array, *, **)
50
- if ::FFI::MemoryPointer === array
51
- array
52
- elsif array.respond_to? :to_ffi_memory_pointer
53
- array.to_ffi_memory_pointer(*, **)
54
- elsif array.respond_to? :to_ndav
55
- array.to_ndav(*, **).to_ffi_memory_pointer
56
- else
57
- ::NDAV.new(array, *, **).to_ffi_memory_pointer
58
- end
59
- end
60
- end
61
- end
62
-
63
- module Pointer
64
- module FromNDAV
65
- def from_ndav(array)
66
- new(array.item_size, array.to_ptr.to_i)
67
- end
68
- end
69
-
70
- module ToNDAV
71
- def to_ndav(shape:, format: TYPE_SIZE_TO_FORMAT[type_size], lifetime: self, **)
72
- byte_size = shape.reduce(::NDAV::ITEM_SIZES[format], :*)
73
- ptr = ::Fiddle::Pointer.new(address, byte_size)
74
- ::NDAV.new(ptr, shape:, format:, lifetime:, **)
75
- end
76
- end
77
-
78
- module Converter
79
- def FFIPointer(array, *, **)
80
- # use instance_of? instead of kind_of? beacuase MemoryPointer is subclass of Pointer
81
- if array.instance_of? ::FFI::Pointer
82
- array
83
- elsif array.respond_to? :to_ffi_pointer
84
- array.to_ffi_pointer(*, **)
85
- elsif array.respond_to? :to_ndav
86
- array.to_ndav(*, **).to_ffi_pointer
87
- else
88
- ::NDAV.new(array, *, **).to_ffi_pointer
89
- end
90
- end
91
- end
92
- end
93
-
94
- ::NDAV.register ::FFI::MemoryPointer, MemoryPointer, name: :ffi_memory_pointer
95
- ::NDAV.register ::FFI::Pointer, Pointer, name: :ffi_pointer
96
- end
97
- end
data/test/test_ffi.rb DELETED
@@ -1,117 +0,0 @@
1
- require_relative "helper"
2
-
3
- class TestFFI < TestBase
4
- def test_ndav_from_ffi_memory_pointer
5
- ptr = ::FFI::MemoryPointer.new(:int16, 3)
6
- ptr.write_array_of_int16([1, 2, 3])
7
- ndav = ::NDAV.from_ffi_memory_pointer(ptr, shape: [3], format: "s")
8
-
9
- assert_int_array ndav
10
- end
11
-
12
- def test_ndav_from_ffi_pointer
13
- str = [1, 2, 3].pack("s*")
14
- address = ::Fiddle::Pointer[str].to_i
15
- ptr = ::FFI::Pointer.new(:int16, address)
16
- ndav = ::NDAV.from_ffi_pointer(ptr, shape: [3], format: "s", lifetime: str)
17
-
18
- assert_int_array ndav
19
- end
20
-
21
- def test_ndav_to_ffi_memory_pointer
22
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
23
- ptr = ndav.to_ffi_memory_pointer
24
-
25
- assert_equal [1, 2, 3].pack("s*"), ptr.read_string(ndav.byte_size)
26
- end
27
-
28
- def test_ndav_to_ffi_pointer
29
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
30
- ptr = ndav.to_ffi_pointer
31
-
32
- assert_equal [1, 2, 3].pack("s*"), ptr.read_string(ndav.byte_size)
33
- assert_equal ndav.to_ptr.to_i, ptr.address
34
- end
35
-
36
- def test_ffi_memory_pointer_from_ndav
37
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
38
- ptr = ::FFI::MemoryPointer.from_ndav(ndav)
39
-
40
- assert_equal [1, 2, 3].pack("s*"), ptr.read_string(ndav.byte_size)
41
- end
42
-
43
- def test_ffi_pointer_from_ndav
44
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
45
- ptr = ::FFI::Pointer.from_ndav(ndav)
46
-
47
- assert_equal [1, 2, 3].pack("s*"), ptr.read_string(ndav.byte_size)
48
- assert_equal ndav.to_ptr.to_i, ptr.address
49
- end
50
-
51
- def test_memory_view_from_ffi_memory_pointer
52
- ptr = ::FFI::MemoryPointer.new(:int16, 3)
53
- ptr.write_array_of_int16([1, 2, 3])
54
- fmv = Fiddle::MemoryView.new(ptr)
55
-
56
- assert_int_array fmv
57
- end
58
-
59
- class TestConverter < self
60
- def test_ndav_from_ffi_memory_pointer
61
- ptr = ::FFI::MemoryPointer.new(:int16, 3)
62
- ptr.write_array_of_int16([1, 2, 3])
63
- ndav = ::NDAV::Converter::NDAV(ptr)
64
-
65
- assert_int_array ndav
66
- end
67
-
68
- def test_ffi_memory_pointer_from_ndav
69
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
70
-
71
- assert_instance_of ::FFI::MemoryPointer, ::NDAV::Converter::FFIMemoryPointer(ndav)
72
- end
73
-
74
- def test_FFIMemoryPointer
75
- c = Class.new {
76
- include ::NDAV::Converter
77
-
78
- def to_ffi_memory_pointer(ndav)
79
- FFIMemoryPointer(ndav)
80
- end
81
- }
82
-
83
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
84
-
85
- assert_instance_of ::FFI::MemoryPointer, c.new.to_ffi_memory_pointer(ndav)
86
- end
87
-
88
- def test_ndav_from_ffi_pointer
89
- str = [1, 2, 3].pack("s*")
90
- address = ::Fiddle::Pointer[str].to_i
91
- ptr = ::FFI::Pointer.new(:int16, address)
92
- ndav = ::NDAV::Converter::NDAV(ptr, shape: [3])
93
-
94
- assert_int_array ndav
95
- end
96
-
97
- def test_ffi_pointer_from_ndav
98
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
99
-
100
- assert_instance_of ::FFI::Pointer, ::NDAV::Converter::FFIPointer(ndav)
101
- end
102
-
103
- def test_FFIPointer
104
- c = Class.new {
105
- include ::NDAV::Converter
106
-
107
- def to_ffi_pointer(ndav)
108
- FFIPointer(ndav)
109
- end
110
- }
111
-
112
- ndav = ::NDAV.new(Arrow::Int16Array.new([1, 2, 3]))
113
-
114
- assert_instance_of ::FFI::Pointer, c.new.to_ffi_pointer(ndav)
115
- end
116
- end
117
- end