fast-protowire 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a0c2962e443067325c5f4a089b72e499b9e273f8f563464e9830ed9cdf6eccfa
4
- data.tar.gz: cdc454d06627bf2d1f28ddc35127024e119e06e748120cfceac91fa7e4e40860
3
+ metadata.gz: e4332160b8041d794b7f19703e3c9ccfa0887bc1eb0d376b6e9d6e1947cf79c9
4
+ data.tar.gz: 990b87f8ffcf4b7684dd45bfef79f2ec93d66d6ad60b491109c5ab65cdfa7ec2
5
5
  SHA512:
6
- metadata.gz: 5b6b5ea5c39ebc3fb9b25490fbea312dccdaaf450ab08b7a9be794056dbc3565e814f98890629664e405333e340e61d493760a57adc99b072b2cf59ff8399bc6
7
- data.tar.gz: b788d96fe73458a33391b5339d35f7be57a37ec3bb203a2d645c590a5cb98d6ef4f9a5641788d8a12f20c7a51911c27287cba01cbac49e0d4b3c9e22f944be1d
6
+ metadata.gz: 69a56b231c6adb17d312ad7a90c2c223323d4283f564f5bc6b0853ee37136c5a3ffdc4ff81f86d70047f3fc10d5d1f510cc3152c5b3cd628f1bf965d5a50c8b9
7
+ data.tar.gz: 8f9478ae25d88e9ca0d8ce55d64ffa248c7e480d2323436eceb3b6757b8a0668faf18d4b32b89fef7df4379461d1b50e7d652f5428ab67026c8df1c82abc3ee2
data/AGENTS.md CHANGED
@@ -8,7 +8,9 @@ Standing context for agents working in this repository.
8
8
  bundle install
9
9
  bundle exec sus
10
10
  bundle exec rubocop
11
- protoc --proto_path=fixtures/proto --ruby_out=fixtures/pb fixtures/proto/*.proto # regenerate the reference schemas
11
+ bundle exec ruby benchmark/messages.rb # encode, build and decode against google-protobuf; BENCH_QUICK=1 for a short run
12
+ protoc --proto_path=fixtures/proto --proto_path=/opt/homebrew/include --ruby_out=fixtures/pb fixtures/proto/*.proto
13
+ # regenerates the reference schemas; the second path is where protoc keeps google/protobuf/timestamp.proto
12
14
  ```
13
15
 
14
16
  ## What this is
data/CHANGELOG.md CHANGED
@@ -1,5 +1,32 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.2.0
4
+
5
+ - Encoding allocates nothing per nested message, packed field or map
6
+ entry: each is written into the parent buffer behind a length prefix
7
+ filled in afterwards. A family of 36,000 metrics x 12 labels encodes in
8
+ one object, its output, where it took 647,000, and 22% faster.
9
+ `Wire.append_length_delimited_from` now appends in place and returns the
10
+ width of the prefix it wrote, a hint to pass back for the next value;
11
+ `Wire.reserve_length` and `Wire.close_length` are its pieces, and
12
+ `Wire.varint_size` and `Wire.append_bytes` are public.
13
+ - Fixed-width values pack with literal formats, so a double costs no Array
14
+ on Ruby 3.4+; text is appended with `String#append_as_bytes` there, tag
15
+ and one-byte size in the same call and no binary copy for UTF-8.
16
+ Zigzag no longer allocates for negative values.
17
+ - Decoding reads nested messages, packed fields and map entries in place
18
+ (`Reader#read_nested`), unpacks fixed-width values at an offset
19
+ (`Reader#read_fixed`) and splits tags without an Array, so it allocates
20
+ only the messages, containers and Strings it returns: the same family
21
+ decodes in 1.4M objects where it took 7.1M, 1.7x faster.
22
+ - `Message.new` no longer allocates an empty keyword Hash per message.
23
+ - Removed: `Reader#read_packed` (use `read_nested`) and the interpretive
24
+ `Field#encode`, which the compiled encoder had superseded.
25
+ - `test/fast/protowire/allocations.rb` holds the budgets; the Prometheus
26
+ client model (`fixtures/proto/metrics.proto`, upstream `client_model`)
27
+ joins the parity suite; `benchmark/messages.rb` measures encode, build
28
+ and decode against google-protobuf, published on the benchmarks page.
29
+
3
30
  ## 0.1.0
4
31
 
5
32
  - Initial release: `Wire`, `Reader`, `Enum` and the `Message` DSL covering
data/README.md CHANGED
@@ -50,6 +50,23 @@ and the output is byte-identical to what `protoc`-generated code and `google-pro
50
50
  produce for the same values. The [tutorial](docs/tutorials/encode-a-message.md) walks
51
51
  through a full schema and proves that.
52
52
 
53
+ ## Performance
54
+
55
+ Measured against `google-protobuf` on the Prometheus client model, a family of 36,000
56
+ metrics with twelve labels each (Ruby 4.0.7; conditions and every table on the
57
+ [benchmarks page](docs/explanation/benchmarks.md)):
58
+
59
+ - Encoding allocates **one object, the output**, whatever the message's size or depth;
60
+ 0.1.0 allocated 647,000 for this family. Decoding allocates only the messages,
61
+ containers and Strings it returns, 5x fewer than 0.1.0 and 1.7x faster.
62
+ - Built a message at a time, the way an exposition builds series, `google-protobuf`
63
+ leaves **504,001 native arenas and 225 MiB** behind for an 11.76 MB body and spends
64
+ 1.66 s of every ten builds in GC; fast-protowire leaves 16.5 MiB, no arenas, and 0.26 s.
65
+ - `google-protobuf` is native, and 10 to 20x faster per operation on an existing tree
66
+ or one nested Hash. This gem trades that speed for memory that is roughly the size of
67
+ the output; fast-prometheus's scrape path goes further and writes series with `Wire`
68
+ directly, with no message per series at all.
69
+
53
70
  ## Documentation
54
71
 
55
72
  ### Tutorials
@@ -73,10 +90,12 @@ through a full schema and proves that.
73
90
 
74
91
  - [Design: the wire format and nothing else](docs/explanation/design.md) — what google-protobuf costs per message, what this gem does instead, and what it leaves out.
75
92
  - [How encoding works](docs/explanation/encoding.md) — compiled encoders, buffers, field order, presence, and decoding.
93
+ - [Benchmarks](docs/explanation/benchmarks.md) — encode, build and decode against google-protobuf, what changed since 0.1.0, and how to reproduce them.
76
94
 
77
95
  ## Development
78
96
 
79
97
  ```bash
80
98
  bundle exec sus
81
99
  bundle exec rubocop
100
+ BENCH_QUICK=1 bundle exec ruby benchmark/messages.rb # encode, build and decode against google-protobuf
82
101
  ```
@@ -141,14 +141,6 @@ module Fast
141
141
  end
142
142
  end
143
143
 
144
- def encode(buffer, value)
145
- case rule
146
- when :repeated then encode_repeated(buffer, value)
147
- when :map then encode_map(buffer, value)
148
- else encode_one(buffer, value)
149
- end
150
- end
151
-
152
144
  # Reads one occurrence of this field into +current+ (the value already
153
145
  # held) and returns the value to store.
154
146
  def decode(reader, wire_type, current)
@@ -157,7 +149,7 @@ module Fast
157
149
  when :map then decode_map_entry(reader, wire_type, current)
158
150
  else
159
151
  if type == :message && current
160
- current.merge_from(Reader.new(expect(reader, wire_type).read_length_delimited))
152
+ expect(reader, wire_type).read_nested { |nested| current.merge_from(nested) }
161
153
  else
162
154
  read_one(reader, wire_type)
163
155
  end
@@ -178,26 +170,38 @@ module Fast
178
170
  end
179
171
  end
180
172
 
181
- def encode_one(buffer, value)
182
- case type
183
- when :message then Wire.append_length_delimited(buffer, @tag, value.encode)
184
- when :string, :bytes then Wire.append_length_delimited(buffer, @tag, value)
185
- else
186
- buffer << @tag
187
- append_scalar(buffer, value)
188
- end
189
- end
190
-
191
173
  def read_one(reader, wire_type)
192
174
  expect(reader, wire_type)
193
175
  case type
194
- when :message then message_class.decode(reader.read_length_delimited)
176
+ when :message then reader.read_nested { |nested| message_class.new.merge_from(nested) }
195
177
  when :string then reader.read_length_delimited.force_encoding(Encoding::UTF_8)
196
178
  when :bytes then reader.read_length_delimited
197
179
  else read_scalar(reader)
198
180
  end
199
181
  end
200
182
 
183
+ # Appends the tag and one value. A nested message is encoded straight
184
+ # into the buffer behind a length prefix filled in after it, with no
185
+ # buffer of its own; +width+ carries the prefix width from one value
186
+ # to the next, a hint shared by every encode of the field. (A map's
187
+ # key and value fields build the entry writer together, which is why
188
+ # this is protected rather than private.)
189
+ def writer
190
+ tag = @tag
191
+ case type
192
+ when :message
193
+ width = 1
194
+ ->(buffer, value) { width = Wire.append_length_delimited_from(buffer, tag, width) { |b| value.encode(b) } }
195
+ when :string, :bytes then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value) }
196
+ else
197
+ scalar = scalar_writer
198
+ lambda do |buffer, value|
199
+ buffer << tag
200
+ scalar.call(buffer, value)
201
+ end
202
+ end
203
+ end
204
+
201
205
  private
202
206
 
203
207
  def resolve_type(type)
@@ -265,17 +269,19 @@ module Fast
265
269
  end
266
270
  end
267
271
 
272
+ # Packed: every value bare, behind one tag and length.
268
273
  def repeated_step(ivar)
269
274
  if packed?
270
275
  tag = @tag
271
276
  scalar = scalar_writer
277
+ width = 1
272
278
  lambda do |message, buffer|
273
279
  values = message.instance_variable_get(ivar)
274
280
  next if values.empty?
275
281
 
276
- payload = String.new
277
- values.each { |value| scalar.call(payload, value) }
278
- Wire.append_length_delimited(buffer, tag, payload)
282
+ width = Wire.append_length_delimited_from(buffer, tag, width) do |b|
283
+ values.each { |value| scalar.call(b, value) }
284
+ end
279
285
  end
280
286
  else
281
287
  write = writer
@@ -283,36 +289,27 @@ module Fast
283
289
  end
284
290
  end
285
291
 
292
+ # Each entry is a message { key = 1; value = 2 }.
286
293
  def map_step(ivar)
287
294
  tag = @tag
288
295
  key = @key_field.writer
289
296
  value = @value_field.writer
297
+ width = 1
290
298
  lambda do |message, buffer|
291
299
  message.instance_variable_get(ivar).each do |k, v|
292
- entry = String.new
293
- key.call(entry, k)
294
- value.call(entry, v)
295
- Wire.append_length_delimited(buffer, tag, entry)
300
+ width = Wire.append_length_delimited_from(buffer, tag, width) do |b|
301
+ key.call(b, k)
302
+ value.call(b, v)
303
+ end
296
304
  end
297
305
  end
298
306
  end
299
307
 
300
- # Appends the tag and one value.
301
- def writer
302
- tag = @tag
303
- case type
304
- when :message then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value.encode) }
305
- when :string, :bytes then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value) }
306
- else
307
- scalar = scalar_writer
308
- lambda do |buffer, value|
309
- buffer << tag
310
- scalar.call(buffer, value)
311
- end
312
- end
313
- end
308
+ # -- writers -----------------------------------------------------------
314
309
 
315
310
  # Appends one scalar value without its tag, as packed fields need.
311
+ # Fixed-width types get a lambda each so the pack format is a literal:
312
+ # Ruby elides the Array in [value].pack(literal, buffer:), and only then.
316
313
  def scalar_writer
317
314
  case type
318
315
  when :int32, :int64, :uint32, :uint64 then ->(buffer, value) { Wire.append_varint(buffer, value) }
@@ -322,9 +319,12 @@ module Fast
322
319
  when :enum
323
320
  enum = @enum
324
321
  ->(buffer, value) { Wire.append_varint(buffer, value.is_a?(Symbol) ? enum.resolve(value) : value) }
325
- else
326
- format = FIXED_FORMATS.fetch(type)
327
- ->(buffer, value) { [value].pack(format, buffer: buffer) }
322
+ when :double then ->(buffer, value) { [value].pack("E", buffer: buffer) }
323
+ when :float then ->(buffer, value) { [value].pack("e", buffer: buffer) }
324
+ when :fixed32 then ->(buffer, value) { [value].pack("L<", buffer: buffer) }
325
+ when :sfixed32 then ->(buffer, value) { [value].pack("l<", buffer: buffer) }
326
+ when :fixed64 then ->(buffer, value) { [value].pack("Q<", buffer: buffer) }
327
+ when :sfixed64 then ->(buffer, value) { [value].pack("q<", buffer: buffer) }
328
328
  end
329
329
  end
330
330
 
@@ -409,44 +409,11 @@ module Fast
409
409
  integer
410
410
  end
411
411
 
412
- # -- encoding ----------------------------------------------------------
413
-
414
- def encode_repeated(buffer, values)
415
- return if values.empty?
416
-
417
- if packed?
418
- Wire.append_length_delimited_from(buffer, @tag) { |b| values.each { |v| append_scalar(b, v) } }
419
- else
420
- values.each { |v| encode_one(buffer, v) }
421
- end
422
- end
423
-
424
- def encode_map(buffer, hash)
425
- hash.each do |key, value|
426
- Wire.append_length_delimited_from(buffer, @tag) do |entry|
427
- @key_field.encode_one(entry, key)
428
- @value_field.encode_one(entry, value)
429
- end
430
- end
431
- end
432
-
433
- def append_scalar(buffer, value)
434
- case type
435
- when :int32, :int64, :uint32, :uint64 then Wire.append_varint(buffer, value)
436
- when :sint32 then Wire.append_varint(buffer, Wire.zigzag32(value))
437
- when :sint64 then Wire.append_varint(buffer, Wire.zigzag64(value))
438
- when :bool then buffer << (value ? 1 : 0)
439
- when :enum then Wire.append_varint(buffer, enum_number(value))
440
- else buffer << [value].pack(FIXED_FORMATS.fetch(type))
441
- end
442
- end
443
-
444
412
  # -- decoding ----------------------------------------------------------
445
413
 
446
414
  def decode_repeated(reader, wire_type, values)
447
415
  if packable? && wire_type == Wire::LENGTH_DELIMITED
448
- packed = reader.read_packed
449
- values << read_scalar(packed) until packed.eof?
416
+ reader.read_nested { |packed| values << read_scalar(packed) until packed.eof? }
450
417
  else
451
418
  values << read_one(reader, wire_type)
452
419
  end
@@ -454,18 +421,19 @@ module Fast
454
421
  end
455
422
 
456
423
  def decode_map_entry(reader, wire_type, hash)
457
- entry = Reader.new(expect(reader, wire_type).read_length_delimited)
458
- key = @key_field.default_value
459
- value = @value_field.default_value
460
- until entry.eof?
461
- number, entry_wire_type = entry.read_tag
462
- case number
463
- when 1 then key = @key_field.read_one(entry, entry_wire_type)
464
- when 2 then value = @value_field.decode(entry, entry_wire_type, value)
465
- else entry.skip(entry_wire_type)
424
+ expect(reader, wire_type).read_nested do |entry|
425
+ key = @key_field.default_value
426
+ value = @value_field.default_value
427
+ until entry.eof?
428
+ tag = entry.read_varint
429
+ case tag >> 3
430
+ when 1 then key = @key_field.read_one(entry, tag & 0x7)
431
+ when 2 then value = @value_field.decode(entry, tag & 0x7, value)
432
+ else entry.skip(tag & 0x7)
433
+ end
466
434
  end
435
+ hash[key] = value
467
436
  end
468
- hash[key] = value
469
437
  hash
470
438
  end
471
439
 
@@ -484,8 +452,8 @@ module Fast
484
452
  when :uint64 then reader.read_varint
485
453
  when :sint32, :sint64 then Wire.unzigzag(reader.read_varint)
486
454
  when :bool then reader.read_varint != 0
487
- when :double, :fixed64, :sfixed64 then reader.read_bytes(8).unpack1(FIXED_FORMATS.fetch(type))
488
- else reader.read_bytes(4).unpack1(FIXED_FORMATS.fetch(type))
455
+ when :double, :fixed64, :sfixed64 then reader.read_fixed(FIXED_FORMATS.fetch(type), 8)
456
+ else reader.read_fixed(FIXED_FORMATS.fetch(type), 4)
489
457
  end
490
458
  end
491
459
 
@@ -493,9 +461,6 @@ module Fast
493
461
  value &= (1 << bits) - 1
494
462
  value >= (1 << (bits - 1)) ? value - (1 << bits) : value
495
463
  end
496
-
497
- # A map's key and value fields build the entry writer together.
498
- protected :writer
499
464
  end
500
465
  end
501
466
  end
@@ -143,11 +143,12 @@ module Fast
143
143
 
144
144
  # Scalar and message ivars stay unset until written (an unset ivar reads
145
145
  # as nil); repeated and map fields get their container up front so it
146
- # can be mutated in place.
147
- def initialize(attributes = nil, **keywords)
146
+ # can be mutated in place. +attributes+ is a Hash, or the keywords a
147
+ # caller writes instead of one.
148
+ def initialize(attributes = nil)
148
149
  self.class.container_fields.each { |field| instance_variable_set(field.ivar, field.default_value) }
149
150
  @unknown_fields = nil
150
- (attributes || keywords).each do |name, value|
151
+ attributes&.each do |name, value|
151
152
  field = self.class.fields[name.to_sym]
152
153
  raise ArgumentError, "unknown field #{name.inspect} for #{self.class}" unless field
153
154
 
@@ -174,12 +175,14 @@ module Fast
174
175
  # merge) and returns self.
175
176
  def merge_from(reader)
176
177
  until reader.eof?
177
- number, wire_type = reader.read_tag
178
- field = self.class.fields_by_number[number]
178
+ key = reader.read_varint
179
+ wire_type = key & 0x7
180
+ field = self.class.fields_by_number[key >> 3]
179
181
  if field
180
182
  write_field(field, field.decode(reader, wire_type, instance_variable_get(field.ivar)))
181
183
  else
182
- (@unknown_fields ||= String.new) << Wire.varint((number << 3) | wire_type) << reader.skip(wire_type)
184
+ Wire.append_varint(@unknown_fields ||= String.new, key)
185
+ @unknown_fields << reader.skip(wire_type)
183
186
  end
184
187
  end
185
188
  self
@@ -26,28 +26,40 @@ module Fast
26
26
  [key >> 3, key & 0x7]
27
27
  end
28
28
 
29
+ # Most varints (tags, small lengths) are one byte; the loop is only
30
+ # entered past it, and is a bare while because Kernel#loop costs an
31
+ # object per call.
29
32
  def read_varint
30
- result = 0
31
- shift = 0
32
- loop do
33
- raise DecodeError, "truncated varint" if @position >= @limit
33
+ byte = read_byte
34
+ return byte if byte < 0x80
34
35
 
35
- byte = @buffer.getbyte(@position)
36
- @position += 1
37
- result |= (byte & 0x7f) << shift
38
- return result if byte < 0x80
36
+ result = byte & 0x7f
37
+ shift = 7
38
+ while byte >= 0x80
39
+ raise DecodeError, "varint too long" if shift > 63
39
40
 
41
+ byte = read_byte
42
+ result |= (byte & 0x7f) << shift
40
43
  shift += 7
41
- raise DecodeError, "varint too long" if shift > 63
42
44
  end
45
+ result
43
46
  end
44
47
 
45
48
  def read_fixed32
46
- read_bytes(4).unpack1("L<")
49
+ read_fixed("L<", 4)
47
50
  end
48
51
 
49
52
  def read_fixed64
50
- read_bytes(8).unpack1("Q<")
53
+ read_fixed("Q<", 8)
54
+ end
55
+
56
+ # Reads +width+ bytes as one value of the pack +format+, in place.
57
+ def read_fixed(format, width)
58
+ raise DecodeError, "truncated field" if @position + width > @limit
59
+
60
+ value = @buffer.unpack1(format, offset: @position)
61
+ @position += width
62
+ value
51
63
  end
52
64
 
53
65
  def read_bytes(length)
@@ -62,14 +74,19 @@ module Fast
62
74
  read_bytes(read_varint)
63
75
  end
64
76
 
65
- # A reader bounded to the next length-delimited value, for packed fields.
66
- def read_packed
77
+ # Bounds the reader to the next length-delimited value for the block
78
+ # and returns the block's result: nested messages, packed fields and
79
+ # map entries are read in place, with no copy of their bytes.
80
+ def read_nested
67
81
  length = read_varint
68
- raise DecodeError, "truncated packed field" if @position + length > @limit
82
+ limit = @limit
83
+ raise DecodeError, "truncated field" if @position + length > limit
69
84
 
70
- reader = Reader.new(@buffer, @position, @position + length)
71
- @position += length
72
- reader
85
+ @limit = @position + length
86
+ result = yield self
87
+ @position = @limit
88
+ @limit = limit
89
+ result
73
90
  end
74
91
 
75
92
  # Skips one value of +wire_type+ and returns its raw bytes, so unknown
@@ -89,6 +106,14 @@ module Fast
89
106
 
90
107
  private
91
108
 
109
+ def read_byte
110
+ raise DecodeError, "truncated varint" if @position >= @limit
111
+
112
+ byte = @buffer.getbyte(@position)
113
+ @position += 1
114
+ byte
115
+ end
116
+
92
117
  def skip_group
93
118
  loop do
94
119
  number, wire_type = read_tag
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Fast
4
4
  module Protowire
5
- VERSION = "0.1.0"
5
+ VERSION = "0.2.0"
6
6
  end
7
7
  end
@@ -3,8 +3,9 @@
3
3
  module Fast
4
4
  module Protowire
5
5
  # The wire format itself: tags, varints, zigzag, fixed-width values and
6
- # length prefixes, appended to a binary String buffer. Everything above
7
- # this module is bookkeeping about which field gets which of these.
6
+ # length prefixes, appended to a binary String buffer, and the size each
7
+ # takes so a length prefix can be written before its payload. Everything
8
+ # above this module is bookkeeping about which field gets which of these.
8
9
  module Wire
9
10
  VARINT = 0
10
11
  FIXED64 = 1
@@ -14,7 +15,9 @@ module Fast
14
15
  FIXED32 = 5
15
16
 
16
17
  UINT64_MASK = (1 << 64) - 1
17
- private_constant :UINT64_MASK
18
+ # Zero bytes a length prefix is widened to, by varint width.
19
+ PLACEHOLDERS = Array.new(11) { |width| ("\0" * width).b.freeze }.freeze
20
+ private_constant :UINT64_MASK, :PLACEHOLDERS
18
21
 
19
22
  module_function
20
23
 
@@ -38,34 +41,100 @@ module Fast
38
41
  buffer << value
39
42
  end
40
43
 
44
+ # Bytes append_varint writes for +value+.
45
+ def varint_size(value)
46
+ return 10 if value.negative?
47
+
48
+ size = 1
49
+ while value > 0x7f
50
+ value >>= 7
51
+ size += 1
52
+ end
53
+ size
54
+ end
55
+
56
+ # (* 2 rather than << 1: a left shift of a negative Fixnum allocates.)
41
57
  def zigzag32(value)
42
- (value << 1) ^ (value >> 31)
58
+ (value * 2) ^ (value >> 31)
43
59
  end
44
60
 
45
61
  def zigzag64(value)
46
- (value << 1) ^ (value >> 63)
62
+ (value * 2) ^ (value >> 63)
47
63
  end
48
64
 
49
65
  def unzigzag(value)
50
66
  (value >> 1) ^ -(value & 1)
51
67
  end
52
68
 
53
- # A length-delimited field: +tag+ then the varint size of +bytes+ then
54
- # +bytes+. Non-ASCII text is appended as its bytes so the binary buffer
55
- # never trips Ruby's encoding compatibility check.
56
- def append_length_delimited(buffer, tag, bytes)
69
+ # A length-delimited field whose payload the block appends to +buffer+
70
+ # itself, behind a length prefix filled in once the payload is there,
71
+ # so a nested message, packed field or map entry costs no buffer of
72
+ # its own. Returns the width of the prefix it wrote: a caller writing
73
+ # many alike values passes it back as +width+ and the prefix is
74
+ # rarely resized.
75
+ def append_length_delimited_from(buffer, tag, width = 1)
57
76
  buffer << tag
58
- append_varint(buffer, bytes.bytesize)
59
- buffer << (bytes.ascii_only? ? bytes : bytes.b)
77
+ start = reserve_length(buffer, width)
78
+ yield buffer
79
+ close_length(buffer, start, width)
80
+ end
81
+
82
+ # Appends +width+ placeholder bytes for a length prefix and returns the
83
+ # position after them, for close_length.
84
+ def reserve_length(buffer, width = 1)
85
+ buffer << PLACEHOLDERS[width]
86
+ buffer.bytesize
87
+ end
88
+
89
+ # Writes the varint size of everything appended since +start+ into the
90
+ # +width+ placeholder bytes before it, first resizing the placeholder
91
+ # when the size needs a different width, which moves only the payload.
92
+ # Returns the width written. (Growing a String in place reallocates it
93
+ # to exactly the new size, so on a large buffer a resize costs far more
94
+ # than the bytes moved; hence the width hint.)
95
+ def close_length(buffer, start, width = 1)
96
+ length = buffer.bytesize - start
97
+ needed = length < 0x80 ? 1 : varint_size(length)
98
+ buffer.bytesplice(start - width, width, PLACEHOLDERS[needed]) unless needed == width
99
+ position = start - width
100
+ while length > 0x7f
101
+ buffer.setbyte(position, (length & 0x7f) | 0x80)
102
+ position += 1
103
+ length >>= 7
104
+ end
105
+ buffer.setbyte(position, length)
106
+ needed
60
107
  end
61
108
 
62
- # A length-delimited field whose payload the block writes into a fresh
63
- # buffer. (Writing into the parent and inserting the size afterwards
64
- # looks cheaper but String#insert costs O(size) on the parent.)
65
- def append_length_delimited_from(buffer, tag)
66
- payload = String.new
67
- yield payload
68
- append_length_delimited(buffer, tag, payload)
109
+ # Text of any encoding goes into the binary buffer as bytes. Ruby 3.4's
110
+ # String#append_as_bytes does that with no copy, and takes the tag and
111
+ # a one-byte size in the same call; before it, non-ASCII text is
112
+ # appended as a binary copy so the buffer never trips Ruby's encoding
113
+ # compatibility check.
114
+ if String.method_defined?(:append_as_bytes)
115
+ # A length-delimited field: +tag+, the varint size of +bytes+, +bytes+.
116
+ def append_length_delimited(buffer, tag, bytes)
117
+ size = bytes.bytesize
118
+ return buffer.append_as_bytes(tag, size, bytes) if size < 0x80
119
+
120
+ buffer << tag
121
+ append_varint(buffer, size)
122
+ buffer.append_as_bytes(bytes)
123
+ end
124
+
125
+ def append_bytes(buffer, bytes)
126
+ buffer.append_as_bytes(bytes)
127
+ end
128
+ else
129
+ def append_length_delimited(buffer, tag, bytes)
130
+ buffer << tag
131
+ append_varint(buffer, bytes.bytesize)
132
+ append_bytes(buffer, bytes)
133
+ end
134
+
135
+ def append_bytes(buffer, bytes)
136
+ buffer << (bytes.encoding == Encoding::BINARY || bytes.ascii_only? ? bytes : bytes.b)
137
+ end
69
138
  end
70
139
  end
71
140
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fast-protowire
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Eric Jacobs
@@ -46,7 +46,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
46
46
  - !ruby/object:Gem::Version
47
47
  version: '0'
48
48
  requirements: []
49
- rubygems_version: 4.0.10
49
+ rubygems_version: 4.0.20
50
50
  specification_version: 4
51
51
  summary: 'Protocol Buffers wire format for Ruby: declare messages, encode and decode
52
52
  bytes, no runtime.'