fast-protowire 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/AGENTS.md +3 -1
- data/CHANGELOG.md +27 -0
- data/README.md +19 -0
- data/lib/fast/protowire/field.rb +58 -93
- data/lib/fast/protowire/message.rb +9 -6
- data/lib/fast/protowire/reader.rb +42 -17
- data/lib/fast/protowire/version.rb +1 -1
- data/lib/fast/protowire/wire.rb +87 -18
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e4332160b8041d794b7f19703e3c9ccfa0887bc1eb0d376b6e9d6e1947cf79c9
|
|
4
|
+
data.tar.gz: 990b87f8ffcf4b7684dd45bfef79f2ec93d66d6ad60b491109c5ab65cdfa7ec2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 69a56b231c6adb17d312ad7a90c2c223323d4283f564f5bc6b0853ee37136c5a3ffdc4ff81f86d70047f3fc10d5d1f510cc3152c5b3cd628f1bf965d5a50c8b9
|
|
7
|
+
data.tar.gz: 8f9478ae25d88e9ca0d8ce55d64ffa248c7e480d2323436eceb3b6757b8a0668faf18d4b32b89fef7df4379461d1b50e7d652f5428ab67026c8df1c82abc3ee2
|
data/AGENTS.md
CHANGED
|
@@ -8,7 +8,9 @@ Standing context for agents working in this repository.
|
|
|
8
8
|
bundle install
|
|
9
9
|
bundle exec sus
|
|
10
10
|
bundle exec rubocop
|
|
11
|
-
|
|
11
|
+
bundle exec ruby benchmark/messages.rb # encode, build and decode against google-protobuf; BENCH_QUICK=1 for a short run
|
|
12
|
+
protoc --proto_path=fixtures/proto --proto_path=/opt/homebrew/include --ruby_out=fixtures/pb fixtures/proto/*.proto
|
|
13
|
+
# regenerates the reference schemas; the second path is where protoc keeps google/protobuf/timestamp.proto
|
|
12
14
|
```
|
|
13
15
|
|
|
14
16
|
## What this is
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,32 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.2.0
|
|
4
|
+
|
|
5
|
+
- Encoding allocates nothing per nested message, packed field or map
|
|
6
|
+
entry: each is written into the parent buffer behind a length prefix
|
|
7
|
+
filled in afterwards. A family of 36,000 metrics x 12 labels encodes in
|
|
8
|
+
one object, its output, where it took 647,000, and 22% faster.
|
|
9
|
+
`Wire.append_length_delimited_from` now appends in place and returns the
|
|
10
|
+
width of the prefix it wrote, a hint to pass back for the next value;
|
|
11
|
+
`Wire.reserve_length` and `Wire.close_length` are its pieces, and
|
|
12
|
+
`Wire.varint_size` and `Wire.append_bytes` are public.
|
|
13
|
+
- Fixed-width values pack with literal formats, so a double costs no Array
|
|
14
|
+
on Ruby 3.4+; text is appended with `String#append_as_bytes` there, tag
|
|
15
|
+
and one-byte size in the same call and no binary copy for UTF-8.
|
|
16
|
+
Zigzag no longer allocates for negative values.
|
|
17
|
+
- Decoding reads nested messages, packed fields and map entries in place
|
|
18
|
+
(`Reader#read_nested`), unpacks fixed-width values at an offset
|
|
19
|
+
(`Reader#read_fixed`) and splits tags without an Array, so it allocates
|
|
20
|
+
only the messages, containers and Strings it returns: the same family
|
|
21
|
+
decodes in 1.4M objects where it took 7.1M, 1.7x faster.
|
|
22
|
+
- `Message.new` no longer allocates an empty keyword Hash per message.
|
|
23
|
+
- Removed: `Reader#read_packed` (use `read_nested`) and the interpretive
|
|
24
|
+
`Field#encode`, which the compiled encoder had superseded.
|
|
25
|
+
- `test/fast/protowire/allocations.rb` holds the budgets; the Prometheus
|
|
26
|
+
client model (`fixtures/proto/metrics.proto`, upstream `client_model`)
|
|
27
|
+
joins the parity suite; `benchmark/messages.rb` measures encode, build
|
|
28
|
+
and decode against google-protobuf, published on the benchmarks page.
|
|
29
|
+
|
|
3
30
|
## 0.1.0
|
|
4
31
|
|
|
5
32
|
- Initial release: `Wire`, `Reader`, `Enum` and the `Message` DSL covering
|
data/README.md
CHANGED
|
@@ -50,6 +50,23 @@ and the output is byte-identical to what `protoc`-generated code and `google-pro
|
|
|
50
50
|
produce for the same values. The [tutorial](docs/tutorials/encode-a-message.md) walks
|
|
51
51
|
through a full schema and proves that.
|
|
52
52
|
|
|
53
|
+
## Performance
|
|
54
|
+
|
|
55
|
+
Measured against `google-protobuf` on the Prometheus client model, a family of 36,000
|
|
56
|
+
metrics with twelve labels each (Ruby 4.0.7; conditions and every table on the
|
|
57
|
+
[benchmarks page](docs/explanation/benchmarks.md)):
|
|
58
|
+
|
|
59
|
+
- Encoding allocates **one object, the output**, whatever the message's size or depth;
|
|
60
|
+
0.1.0 allocated 647,000 for this family. Decoding allocates only the messages,
|
|
61
|
+
containers and Strings it returns, 5x fewer than 0.1.0 and 1.7x faster.
|
|
62
|
+
- Built a message at a time, the way an exposition builds series, `google-protobuf`
|
|
63
|
+
leaves **504,001 native arenas and 225 MiB** behind for an 11.76 MB body and spends
|
|
64
|
+
1.66 s of every ten builds in GC; fast-protowire leaves 16.5 MiB, no arenas, and 0.26 s.
|
|
65
|
+
- `google-protobuf` is native, and 10 to 20x faster per operation on an existing tree
|
|
66
|
+
or one nested Hash. This gem trades that speed for memory that is roughly the size of
|
|
67
|
+
the output; fast-prometheus's scrape path goes further and writes series with `Wire`
|
|
68
|
+
directly, with no message per series at all.
|
|
69
|
+
|
|
53
70
|
## Documentation
|
|
54
71
|
|
|
55
72
|
### Tutorials
|
|
@@ -73,10 +90,12 @@ through a full schema and proves that.
|
|
|
73
90
|
|
|
74
91
|
- [Design: the wire format and nothing else](docs/explanation/design.md) — what google-protobuf costs per message, what this gem does instead, and what it leaves out.
|
|
75
92
|
- [How encoding works](docs/explanation/encoding.md) — compiled encoders, buffers, field order, presence, and decoding.
|
|
93
|
+
- [Benchmarks](docs/explanation/benchmarks.md) — encode, build and decode against google-protobuf, what changed since 0.1.0, and how to reproduce them.
|
|
76
94
|
|
|
77
95
|
## Development
|
|
78
96
|
|
|
79
97
|
```bash
|
|
80
98
|
bundle exec sus
|
|
81
99
|
bundle exec rubocop
|
|
100
|
+
BENCH_QUICK=1 bundle exec ruby benchmark/messages.rb # encode, build and decode against google-protobuf
|
|
82
101
|
```
|
data/lib/fast/protowire/field.rb
CHANGED
|
@@ -141,14 +141,6 @@ module Fast
|
|
|
141
141
|
end
|
|
142
142
|
end
|
|
143
143
|
|
|
144
|
-
def encode(buffer, value)
|
|
145
|
-
case rule
|
|
146
|
-
when :repeated then encode_repeated(buffer, value)
|
|
147
|
-
when :map then encode_map(buffer, value)
|
|
148
|
-
else encode_one(buffer, value)
|
|
149
|
-
end
|
|
150
|
-
end
|
|
151
|
-
|
|
152
144
|
# Reads one occurrence of this field into +current+ (the value already
|
|
153
145
|
# held) and returns the value to store.
|
|
154
146
|
def decode(reader, wire_type, current)
|
|
@@ -157,7 +149,7 @@ module Fast
|
|
|
157
149
|
when :map then decode_map_entry(reader, wire_type, current)
|
|
158
150
|
else
|
|
159
151
|
if type == :message && current
|
|
160
|
-
|
|
152
|
+
expect(reader, wire_type).read_nested { |nested| current.merge_from(nested) }
|
|
161
153
|
else
|
|
162
154
|
read_one(reader, wire_type)
|
|
163
155
|
end
|
|
@@ -178,26 +170,38 @@ module Fast
|
|
|
178
170
|
end
|
|
179
171
|
end
|
|
180
172
|
|
|
181
|
-
def encode_one(buffer, value)
|
|
182
|
-
case type
|
|
183
|
-
when :message then Wire.append_length_delimited(buffer, @tag, value.encode)
|
|
184
|
-
when :string, :bytes then Wire.append_length_delimited(buffer, @tag, value)
|
|
185
|
-
else
|
|
186
|
-
buffer << @tag
|
|
187
|
-
append_scalar(buffer, value)
|
|
188
|
-
end
|
|
189
|
-
end
|
|
190
|
-
|
|
191
173
|
def read_one(reader, wire_type)
|
|
192
174
|
expect(reader, wire_type)
|
|
193
175
|
case type
|
|
194
|
-
when :message then message_class.
|
|
176
|
+
when :message then reader.read_nested { |nested| message_class.new.merge_from(nested) }
|
|
195
177
|
when :string then reader.read_length_delimited.force_encoding(Encoding::UTF_8)
|
|
196
178
|
when :bytes then reader.read_length_delimited
|
|
197
179
|
else read_scalar(reader)
|
|
198
180
|
end
|
|
199
181
|
end
|
|
200
182
|
|
|
183
|
+
# Appends the tag and one value. A nested message is encoded straight
|
|
184
|
+
# into the buffer behind a length prefix filled in after it, with no
|
|
185
|
+
# buffer of its own; +width+ carries the prefix width from one value
|
|
186
|
+
# to the next, a hint shared by every encode of the field. (A map's
|
|
187
|
+
# key and value fields build the entry writer together, which is why
|
|
188
|
+
# this is protected rather than private.)
|
|
189
|
+
def writer
|
|
190
|
+
tag = @tag
|
|
191
|
+
case type
|
|
192
|
+
when :message
|
|
193
|
+
width = 1
|
|
194
|
+
->(buffer, value) { width = Wire.append_length_delimited_from(buffer, tag, width) { |b| value.encode(b) } }
|
|
195
|
+
when :string, :bytes then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value) }
|
|
196
|
+
else
|
|
197
|
+
scalar = scalar_writer
|
|
198
|
+
lambda do |buffer, value|
|
|
199
|
+
buffer << tag
|
|
200
|
+
scalar.call(buffer, value)
|
|
201
|
+
end
|
|
202
|
+
end
|
|
203
|
+
end
|
|
204
|
+
|
|
201
205
|
private
|
|
202
206
|
|
|
203
207
|
def resolve_type(type)
|
|
@@ -265,17 +269,19 @@ module Fast
|
|
|
265
269
|
end
|
|
266
270
|
end
|
|
267
271
|
|
|
272
|
+
# Packed: every value bare, behind one tag and length.
|
|
268
273
|
def repeated_step(ivar)
|
|
269
274
|
if packed?
|
|
270
275
|
tag = @tag
|
|
271
276
|
scalar = scalar_writer
|
|
277
|
+
width = 1
|
|
272
278
|
lambda do |message, buffer|
|
|
273
279
|
values = message.instance_variable_get(ivar)
|
|
274
280
|
next if values.empty?
|
|
275
281
|
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
282
|
+
width = Wire.append_length_delimited_from(buffer, tag, width) do |b|
|
|
283
|
+
values.each { |value| scalar.call(b, value) }
|
|
284
|
+
end
|
|
279
285
|
end
|
|
280
286
|
else
|
|
281
287
|
write = writer
|
|
@@ -283,36 +289,27 @@ module Fast
|
|
|
283
289
|
end
|
|
284
290
|
end
|
|
285
291
|
|
|
292
|
+
# Each entry is a message { key = 1; value = 2 }.
|
|
286
293
|
def map_step(ivar)
|
|
287
294
|
tag = @tag
|
|
288
295
|
key = @key_field.writer
|
|
289
296
|
value = @value_field.writer
|
|
297
|
+
width = 1
|
|
290
298
|
lambda do |message, buffer|
|
|
291
299
|
message.instance_variable_get(ivar).each do |k, v|
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
300
|
+
width = Wire.append_length_delimited_from(buffer, tag, width) do |b|
|
|
301
|
+
key.call(b, k)
|
|
302
|
+
value.call(b, v)
|
|
303
|
+
end
|
|
296
304
|
end
|
|
297
305
|
end
|
|
298
306
|
end
|
|
299
307
|
|
|
300
|
-
#
|
|
301
|
-
def writer
|
|
302
|
-
tag = @tag
|
|
303
|
-
case type
|
|
304
|
-
when :message then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value.encode) }
|
|
305
|
-
when :string, :bytes then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value) }
|
|
306
|
-
else
|
|
307
|
-
scalar = scalar_writer
|
|
308
|
-
lambda do |buffer, value|
|
|
309
|
-
buffer << tag
|
|
310
|
-
scalar.call(buffer, value)
|
|
311
|
-
end
|
|
312
|
-
end
|
|
313
|
-
end
|
|
308
|
+
# -- writers -----------------------------------------------------------
|
|
314
309
|
|
|
315
310
|
# Appends one scalar value without its tag, as packed fields need.
|
|
311
|
+
# Fixed-width types get a lambda each so the pack format is a literal:
|
|
312
|
+
# Ruby elides the Array in [value].pack(literal, buffer:), and only then.
|
|
316
313
|
def scalar_writer
|
|
317
314
|
case type
|
|
318
315
|
when :int32, :int64, :uint32, :uint64 then ->(buffer, value) { Wire.append_varint(buffer, value) }
|
|
@@ -322,9 +319,12 @@ module Fast
|
|
|
322
319
|
when :enum
|
|
323
320
|
enum = @enum
|
|
324
321
|
->(buffer, value) { Wire.append_varint(buffer, value.is_a?(Symbol) ? enum.resolve(value) : value) }
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
322
|
+
when :double then ->(buffer, value) { [value].pack("E", buffer: buffer) }
|
|
323
|
+
when :float then ->(buffer, value) { [value].pack("e", buffer: buffer) }
|
|
324
|
+
when :fixed32 then ->(buffer, value) { [value].pack("L<", buffer: buffer) }
|
|
325
|
+
when :sfixed32 then ->(buffer, value) { [value].pack("l<", buffer: buffer) }
|
|
326
|
+
when :fixed64 then ->(buffer, value) { [value].pack("Q<", buffer: buffer) }
|
|
327
|
+
when :sfixed64 then ->(buffer, value) { [value].pack("q<", buffer: buffer) }
|
|
328
328
|
end
|
|
329
329
|
end
|
|
330
330
|
|
|
@@ -409,44 +409,11 @@ module Fast
|
|
|
409
409
|
integer
|
|
410
410
|
end
|
|
411
411
|
|
|
412
|
-
# -- encoding ----------------------------------------------------------
|
|
413
|
-
|
|
414
|
-
def encode_repeated(buffer, values)
|
|
415
|
-
return if values.empty?
|
|
416
|
-
|
|
417
|
-
if packed?
|
|
418
|
-
Wire.append_length_delimited_from(buffer, @tag) { |b| values.each { |v| append_scalar(b, v) } }
|
|
419
|
-
else
|
|
420
|
-
values.each { |v| encode_one(buffer, v) }
|
|
421
|
-
end
|
|
422
|
-
end
|
|
423
|
-
|
|
424
|
-
def encode_map(buffer, hash)
|
|
425
|
-
hash.each do |key, value|
|
|
426
|
-
Wire.append_length_delimited_from(buffer, @tag) do |entry|
|
|
427
|
-
@key_field.encode_one(entry, key)
|
|
428
|
-
@value_field.encode_one(entry, value)
|
|
429
|
-
end
|
|
430
|
-
end
|
|
431
|
-
end
|
|
432
|
-
|
|
433
|
-
def append_scalar(buffer, value)
|
|
434
|
-
case type
|
|
435
|
-
when :int32, :int64, :uint32, :uint64 then Wire.append_varint(buffer, value)
|
|
436
|
-
when :sint32 then Wire.append_varint(buffer, Wire.zigzag32(value))
|
|
437
|
-
when :sint64 then Wire.append_varint(buffer, Wire.zigzag64(value))
|
|
438
|
-
when :bool then buffer << (value ? 1 : 0)
|
|
439
|
-
when :enum then Wire.append_varint(buffer, enum_number(value))
|
|
440
|
-
else buffer << [value].pack(FIXED_FORMATS.fetch(type))
|
|
441
|
-
end
|
|
442
|
-
end
|
|
443
|
-
|
|
444
412
|
# -- decoding ----------------------------------------------------------
|
|
445
413
|
|
|
446
414
|
def decode_repeated(reader, wire_type, values)
|
|
447
415
|
if packable? && wire_type == Wire::LENGTH_DELIMITED
|
|
448
|
-
packed
|
|
449
|
-
values << read_scalar(packed) until packed.eof?
|
|
416
|
+
reader.read_nested { |packed| values << read_scalar(packed) until packed.eof? }
|
|
450
417
|
else
|
|
451
418
|
values << read_one(reader, wire_type)
|
|
452
419
|
end
|
|
@@ -454,18 +421,19 @@ module Fast
|
|
|
454
421
|
end
|
|
455
422
|
|
|
456
423
|
def decode_map_entry(reader, wire_type, hash)
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
424
|
+
expect(reader, wire_type).read_nested do |entry|
|
|
425
|
+
key = @key_field.default_value
|
|
426
|
+
value = @value_field.default_value
|
|
427
|
+
until entry.eof?
|
|
428
|
+
tag = entry.read_varint
|
|
429
|
+
case tag >> 3
|
|
430
|
+
when 1 then key = @key_field.read_one(entry, tag & 0x7)
|
|
431
|
+
when 2 then value = @value_field.decode(entry, tag & 0x7, value)
|
|
432
|
+
else entry.skip(tag & 0x7)
|
|
433
|
+
end
|
|
466
434
|
end
|
|
435
|
+
hash[key] = value
|
|
467
436
|
end
|
|
468
|
-
hash[key] = value
|
|
469
437
|
hash
|
|
470
438
|
end
|
|
471
439
|
|
|
@@ -484,8 +452,8 @@ module Fast
|
|
|
484
452
|
when :uint64 then reader.read_varint
|
|
485
453
|
when :sint32, :sint64 then Wire.unzigzag(reader.read_varint)
|
|
486
454
|
when :bool then reader.read_varint != 0
|
|
487
|
-
when :double, :fixed64, :sfixed64 then reader.
|
|
488
|
-
else reader.
|
|
455
|
+
when :double, :fixed64, :sfixed64 then reader.read_fixed(FIXED_FORMATS.fetch(type), 8)
|
|
456
|
+
else reader.read_fixed(FIXED_FORMATS.fetch(type), 4)
|
|
489
457
|
end
|
|
490
458
|
end
|
|
491
459
|
|
|
@@ -493,9 +461,6 @@ module Fast
|
|
|
493
461
|
value &= (1 << bits) - 1
|
|
494
462
|
value >= (1 << (bits - 1)) ? value - (1 << bits) : value
|
|
495
463
|
end
|
|
496
|
-
|
|
497
|
-
# A map's key and value fields build the entry writer together.
|
|
498
|
-
protected :writer
|
|
499
464
|
end
|
|
500
465
|
end
|
|
501
466
|
end
|
|
@@ -143,11 +143,12 @@ module Fast
|
|
|
143
143
|
|
|
144
144
|
# Scalar and message ivars stay unset until written (an unset ivar reads
|
|
145
145
|
# as nil); repeated and map fields get their container up front so it
|
|
146
|
-
# can be mutated in place.
|
|
147
|
-
|
|
146
|
+
# can be mutated in place. +attributes+ is a Hash, or the keywords a
|
|
147
|
+
# caller writes instead of one.
|
|
148
|
+
def initialize(attributes = nil)
|
|
148
149
|
self.class.container_fields.each { |field| instance_variable_set(field.ivar, field.default_value) }
|
|
149
150
|
@unknown_fields = nil
|
|
150
|
-
|
|
151
|
+
attributes&.each do |name, value|
|
|
151
152
|
field = self.class.fields[name.to_sym]
|
|
152
153
|
raise ArgumentError, "unknown field #{name.inspect} for #{self.class}" unless field
|
|
153
154
|
|
|
@@ -174,12 +175,14 @@ module Fast
|
|
|
174
175
|
# merge) and returns self.
|
|
175
176
|
def merge_from(reader)
|
|
176
177
|
until reader.eof?
|
|
177
|
-
|
|
178
|
-
|
|
178
|
+
key = reader.read_varint
|
|
179
|
+
wire_type = key & 0x7
|
|
180
|
+
field = self.class.fields_by_number[key >> 3]
|
|
179
181
|
if field
|
|
180
182
|
write_field(field, field.decode(reader, wire_type, instance_variable_get(field.ivar)))
|
|
181
183
|
else
|
|
182
|
-
(@unknown_fields ||= String.new
|
|
184
|
+
Wire.append_varint(@unknown_fields ||= String.new, key)
|
|
185
|
+
@unknown_fields << reader.skip(wire_type)
|
|
183
186
|
end
|
|
184
187
|
end
|
|
185
188
|
self
|
|
@@ -26,28 +26,40 @@ module Fast
|
|
|
26
26
|
[key >> 3, key & 0x7]
|
|
27
27
|
end
|
|
28
28
|
|
|
29
|
+
# Most varints (tags, small lengths) are one byte; the loop is only
|
|
30
|
+
# entered past it, and is a bare while because Kernel#loop costs an
|
|
31
|
+
# object per call.
|
|
29
32
|
def read_varint
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
loop do
|
|
33
|
-
raise DecodeError, "truncated varint" if @position >= @limit
|
|
33
|
+
byte = read_byte
|
|
34
|
+
return byte if byte < 0x80
|
|
34
35
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
36
|
+
result = byte & 0x7f
|
|
37
|
+
shift = 7
|
|
38
|
+
while byte >= 0x80
|
|
39
|
+
raise DecodeError, "varint too long" if shift > 63
|
|
39
40
|
|
|
41
|
+
byte = read_byte
|
|
42
|
+
result |= (byte & 0x7f) << shift
|
|
40
43
|
shift += 7
|
|
41
|
-
raise DecodeError, "varint too long" if shift > 63
|
|
42
44
|
end
|
|
45
|
+
result
|
|
43
46
|
end
|
|
44
47
|
|
|
45
48
|
def read_fixed32
|
|
46
|
-
|
|
49
|
+
read_fixed("L<", 4)
|
|
47
50
|
end
|
|
48
51
|
|
|
49
52
|
def read_fixed64
|
|
50
|
-
|
|
53
|
+
read_fixed("Q<", 8)
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
# Reads +width+ bytes as one value of the pack +format+, in place.
|
|
57
|
+
def read_fixed(format, width)
|
|
58
|
+
raise DecodeError, "truncated field" if @position + width > @limit
|
|
59
|
+
|
|
60
|
+
value = @buffer.unpack1(format, offset: @position)
|
|
61
|
+
@position += width
|
|
62
|
+
value
|
|
51
63
|
end
|
|
52
64
|
|
|
53
65
|
def read_bytes(length)
|
|
@@ -62,14 +74,19 @@ module Fast
|
|
|
62
74
|
read_bytes(read_varint)
|
|
63
75
|
end
|
|
64
76
|
|
|
65
|
-
#
|
|
66
|
-
|
|
77
|
+
# Bounds the reader to the next length-delimited value for the block
|
|
78
|
+
# and returns the block's result: nested messages, packed fields and
|
|
79
|
+
# map entries are read in place, with no copy of their bytes.
|
|
80
|
+
def read_nested
|
|
67
81
|
length = read_varint
|
|
68
|
-
|
|
82
|
+
limit = @limit
|
|
83
|
+
raise DecodeError, "truncated field" if @position + length > limit
|
|
69
84
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
85
|
+
@limit = @position + length
|
|
86
|
+
result = yield self
|
|
87
|
+
@position = @limit
|
|
88
|
+
@limit = limit
|
|
89
|
+
result
|
|
73
90
|
end
|
|
74
91
|
|
|
75
92
|
# Skips one value of +wire_type+ and returns its raw bytes, so unknown
|
|
@@ -89,6 +106,14 @@ module Fast
|
|
|
89
106
|
|
|
90
107
|
private
|
|
91
108
|
|
|
109
|
+
def read_byte
|
|
110
|
+
raise DecodeError, "truncated varint" if @position >= @limit
|
|
111
|
+
|
|
112
|
+
byte = @buffer.getbyte(@position)
|
|
113
|
+
@position += 1
|
|
114
|
+
byte
|
|
115
|
+
end
|
|
116
|
+
|
|
92
117
|
def skip_group
|
|
93
118
|
loop do
|
|
94
119
|
number, wire_type = read_tag
|
data/lib/fast/protowire/wire.rb
CHANGED
|
@@ -3,8 +3,9 @@
|
|
|
3
3
|
module Fast
|
|
4
4
|
module Protowire
|
|
5
5
|
# The wire format itself: tags, varints, zigzag, fixed-width values and
|
|
6
|
-
# length prefixes, appended to a binary String buffer
|
|
7
|
-
#
|
|
6
|
+
# length prefixes, appended to a binary String buffer, and the size each
|
|
7
|
+
# takes so a length prefix can be written before its payload. Everything
|
|
8
|
+
# above this module is bookkeeping about which field gets which of these.
|
|
8
9
|
module Wire
|
|
9
10
|
VARINT = 0
|
|
10
11
|
FIXED64 = 1
|
|
@@ -14,7 +15,9 @@ module Fast
|
|
|
14
15
|
FIXED32 = 5
|
|
15
16
|
|
|
16
17
|
UINT64_MASK = (1 << 64) - 1
|
|
17
|
-
|
|
18
|
+
# Zero bytes a length prefix is widened to, by varint width.
|
|
19
|
+
PLACEHOLDERS = Array.new(11) { |width| ("\0" * width).b.freeze }.freeze
|
|
20
|
+
private_constant :UINT64_MASK, :PLACEHOLDERS
|
|
18
21
|
|
|
19
22
|
module_function
|
|
20
23
|
|
|
@@ -38,34 +41,100 @@ module Fast
|
|
|
38
41
|
buffer << value
|
|
39
42
|
end
|
|
40
43
|
|
|
44
|
+
# Bytes append_varint writes for +value+.
|
|
45
|
+
def varint_size(value)
|
|
46
|
+
return 10 if value.negative?
|
|
47
|
+
|
|
48
|
+
size = 1
|
|
49
|
+
while value > 0x7f
|
|
50
|
+
value >>= 7
|
|
51
|
+
size += 1
|
|
52
|
+
end
|
|
53
|
+
size
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
# (* 2 rather than << 1: a left shift of a negative Fixnum allocates.)
|
|
41
57
|
def zigzag32(value)
|
|
42
|
-
(value
|
|
58
|
+
(value * 2) ^ (value >> 31)
|
|
43
59
|
end
|
|
44
60
|
|
|
45
61
|
def zigzag64(value)
|
|
46
|
-
(value
|
|
62
|
+
(value * 2) ^ (value >> 63)
|
|
47
63
|
end
|
|
48
64
|
|
|
49
65
|
def unzigzag(value)
|
|
50
66
|
(value >> 1) ^ -(value & 1)
|
|
51
67
|
end
|
|
52
68
|
|
|
53
|
-
# A length-delimited field
|
|
54
|
-
#
|
|
55
|
-
#
|
|
56
|
-
|
|
69
|
+
# A length-delimited field whose payload the block appends to +buffer+
|
|
70
|
+
# itself, behind a length prefix filled in once the payload is there,
|
|
71
|
+
# so a nested message, packed field or map entry costs no buffer of
|
|
72
|
+
# its own. Returns the width of the prefix it wrote: a caller writing
|
|
73
|
+
# many alike values passes it back as +width+ and the prefix is
|
|
74
|
+
# rarely resized.
|
|
75
|
+
def append_length_delimited_from(buffer, tag, width = 1)
|
|
57
76
|
buffer << tag
|
|
58
|
-
|
|
59
|
-
buffer
|
|
77
|
+
start = reserve_length(buffer, width)
|
|
78
|
+
yield buffer
|
|
79
|
+
close_length(buffer, start, width)
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Appends +width+ placeholder bytes for a length prefix and returns the
|
|
83
|
+
# position after them, for close_length.
|
|
84
|
+
def reserve_length(buffer, width = 1)
|
|
85
|
+
buffer << PLACEHOLDERS[width]
|
|
86
|
+
buffer.bytesize
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
# Writes the varint size of everything appended since +start+ into the
|
|
90
|
+
# +width+ placeholder bytes before it, first resizing the placeholder
|
|
91
|
+
# when the size needs a different width, which moves only the payload.
|
|
92
|
+
# Returns the width written. (Growing a String in place reallocates it
|
|
93
|
+
# to exactly the new size, so on a large buffer a resize costs far more
|
|
94
|
+
# than the bytes moved; hence the width hint.)
|
|
95
|
+
def close_length(buffer, start, width = 1)
|
|
96
|
+
length = buffer.bytesize - start
|
|
97
|
+
needed = length < 0x80 ? 1 : varint_size(length)
|
|
98
|
+
buffer.bytesplice(start - width, width, PLACEHOLDERS[needed]) unless needed == width
|
|
99
|
+
position = start - width
|
|
100
|
+
while length > 0x7f
|
|
101
|
+
buffer.setbyte(position, (length & 0x7f) | 0x80)
|
|
102
|
+
position += 1
|
|
103
|
+
length >>= 7
|
|
104
|
+
end
|
|
105
|
+
buffer.setbyte(position, length)
|
|
106
|
+
needed
|
|
60
107
|
end
|
|
61
108
|
|
|
62
|
-
#
|
|
63
|
-
#
|
|
64
|
-
#
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
109
|
+
# Text of any encoding goes into the binary buffer as bytes. Ruby 3.4's
|
|
110
|
+
# String#append_as_bytes does that with no copy, and takes the tag and
|
|
111
|
+
# a one-byte size in the same call; before it, non-ASCII text is
|
|
112
|
+
# appended as a binary copy so the buffer never trips Ruby's encoding
|
|
113
|
+
# compatibility check.
|
|
114
|
+
if String.method_defined?(:append_as_bytes)
|
|
115
|
+
# A length-delimited field: +tag+, the varint size of +bytes+, +bytes+.
|
|
116
|
+
def append_length_delimited(buffer, tag, bytes)
|
|
117
|
+
size = bytes.bytesize
|
|
118
|
+
return buffer.append_as_bytes(tag, size, bytes) if size < 0x80
|
|
119
|
+
|
|
120
|
+
buffer << tag
|
|
121
|
+
append_varint(buffer, size)
|
|
122
|
+
buffer.append_as_bytes(bytes)
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def append_bytes(buffer, bytes)
|
|
126
|
+
buffer.append_as_bytes(bytes)
|
|
127
|
+
end
|
|
128
|
+
else
|
|
129
|
+
def append_length_delimited(buffer, tag, bytes)
|
|
130
|
+
buffer << tag
|
|
131
|
+
append_varint(buffer, bytes.bytesize)
|
|
132
|
+
append_bytes(buffer, bytes)
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
def append_bytes(buffer, bytes)
|
|
136
|
+
buffer << (bytes.encoding == Encoding::BINARY || bytes.ascii_only? ? bytes : bytes.b)
|
|
137
|
+
end
|
|
69
138
|
end
|
|
70
139
|
end
|
|
71
140
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: fast-protowire
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Eric Jacobs
|
|
@@ -46,7 +46,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
46
46
|
- !ruby/object:Gem::Version
|
|
47
47
|
version: '0'
|
|
48
48
|
requirements: []
|
|
49
|
-
rubygems_version: 4.0.
|
|
49
|
+
rubygems_version: 4.0.20
|
|
50
50
|
specification_version: 4
|
|
51
51
|
summary: 'Protocol Buffers wire format for Ruby: declare messages, encode and decode
|
|
52
52
|
bytes, no runtime.'
|