fast-protowire 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e4332160b8041d794b7f19703e3c9ccfa0887bc1eb0d376b6e9d6e1947cf79c9
-  data.tar.gz: 990b87f8ffcf4b7684dd45bfef79f2ec93d66d6ad60b491109c5ab65cdfa7ec2
+  metadata.gz: 9d6f10862b8163b7b4d5210479a6b27b62298e0951b24425db97cdec94356a10
+  data.tar.gz: 5a882709766e557ae6a65b3107ec3cc8de6fba07f90de3293607518cf880b4f8
 SHA512:
-  metadata.gz: 69a56b231c6adb17d312ad7a90c2c223323d4283f564f5bc6b0853ee37136c5a3ffdc4ff81f86d70047f3fc10d5d1f510cc3152c5b3cd628f1bf965d5a50c8b9
-  data.tar.gz: 8f9478ae25d88e9ca0d8ce55d64ffa248c7e480d2323436eceb3b6757b8a0668faf18d4b32b89fef7df4379461d1b50e7d652f5428ab67026c8df1c82abc3ee2
+  metadata.gz: 8a54a74df181260136e00d9227f050c1ef785096515b4a91197f939af58df1d865bea3eb5e660db97335111a1728407edccfce11afd8b60a4335076e95b1a258
+  data.tar.gz: 5048f2b557f2e3c4f44b4a2cb95110b11142c6b7c94975dbca68f7a3fc1550ea2e9b120460b4e36bb2ed7f316bdb476b60b2b334f449728c2a8ba869242a8036

data/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Changelog
 
+## 0.3.0
+
+- A declared field arriving with a wire type it does not accept is kept as
+  an unknown field and written back verbatim, where it used to raise
+  `DecodeError`: schema drift is not corruption, and the reference keeps it.
+  So is a map entry carrying more than a key and a value — an undeclared
+  subfield, or a key or value with a wire type the entry does not accept —
+  kept whole among the parent message's unknown fields and left out of the
+  map. An entry that omits its key or its value decodes to that type's
+  default, an empty message for a message-typed value.
+- Hostile input is bounded rather than fatal. Nesting deeper than
+  `Reader::MAX_DEPTH` (100, the reference's limit; nested messages, map
+  entries and groups count, a packed field does not) raises `DecodeError`
+  instead of overflowing the VM stack with `SystemStackError`. Field number
+  0 anywhere in the input, a group closed by an `END_GROUP` carrying another
+  number, and a proto3 `string` whose bytes are not valid UTF-8 raise
+  `DecodeError` too. A varint carrying bits above 64 is truncated rather
+  than read wide, as every implementation does, and a `sint32` is truncated
+  to 32 bits before its zigzag is undone.
+- Decoding is about bytes, so a `Reader` reads its input whatever the String
+  is tagged: a non-binary one is read through a binary view made once on
+  construction, and the input is never modified. `encode` holds its buffer
+  to the same rule: an empty String of any encoding is retagged binary, and
+  one already holding text raises `ArgumentError` rather than widening the
+  bytes it appends into that encoding's characters.
+- `Message#==`, `eql?` and `hash` compare the declared fields only. Unknown
+  fields are no longer part of the comparison, as the reference does not
+  compare them; they are still kept, still readable through
+  `unknown_fields`, and still written back by `encode`. Byte identity is
+  `a.encode == b.encode`.
+- A `float` field narrows to single precision on assignment rather than on
+  encode, so what it reads back is what the wire carries and
+  `decode(encode(m)) == m` holds: `0.1` reads back `0.10000000149011612`,
+  `3.5e38` reads `Infinity`, `1e-50` reads `0.0`. A 32-bit NaN is still
+  written canonical (`7fc00000`) because Ruby's `pack("e")` drops the sign
+  and payload bits the reference preserves; that one is documented, not
+  fixed. `double` NaNs round-trip whole.
+- A declaration the wire format has no form for raises `ArgumentError` where
+  it is written rather than at the first encode: a field number outside
+  1 ... 2²⁹−1 or inside 19000 ... 19999, which the specification reserves for
+  the implementation, and `packed: true` on a `string`, `bytes` or message
+  field.
+- `Reader#read_packed` is back for packed fields, which are read in place
+  and cost no depth level (0.2.0's entry recorded it removed).
+  `Reader#read_key` returns a tag as the raw key holding number and wire
+  type together; `Reader#skip` takes the field number the value arrived
+  under, `skip(wire_type, number = nil)`, so a group is closed only by its
+  own number; `Field#accepts?(wire_type)`, `Wire.binary_buffer` and
+  `Wire::UINT64_MASK` are public.
+- Decoding costs what it did on 0.2.0, guards included: the 36,000-metric
+  family decodes in 0.801 s where 0.2.0 takes 0.784 s on the same machine
+  and Ruby (4.0.7, five timed calls, same session). A field's wire type is
+  computed once, when the field is declared, rather than on every read,
+  where the per-field check had been paying for it twice. What it allocates
+  is unchanged, 1,404,005 objects for that family and one object to encode
+  it, and so are the budgets in `test/fast/protowire/allocations.rb`; the
+  benchmarks page carries the re-measured tables.
+
 ## 0.2.0
 
 - Encoding allocates nothing per nested message, packed field or map

data/README.md

@@ -58,11 +58,11 @@ metrics with twelve labels each (Ruby 4.0.7; conditions and every table on the
 
 - Encoding allocates **one object, the output**, whatever the message's size or depth;
   0.1.0 allocated 647,000 for this family. Decoding allocates only the messages,
-  containers and Strings it returns, 5x fewer than 0.1.0 and 1.7x faster.
+  containers and Strings it returns, 5x fewer than 0.1.0 and 1.3x faster.
 - Built a message at a time, the way an exposition builds series, `google-protobuf`
   leaves **504,001 native arenas and 225 MiB** behind for an 11.76 MB body and spends
-  1.66 s of every ten builds in GC; fast-protowire leaves 16.5 MiB, no arenas, and 0.26 s.
-- `google-protobuf` is native, and 10 to 20x faster per operation on an existing tree
+  1.77 s of every ten builds in GC; fast-protowire leaves 16.5 MiB, no arenas, and 0.27 s.
+- `google-protobuf` is native, and 9 to 25x faster per operation on an existing tree
   or one nested Hash. This gem trades that speed for memory that is roughly the size of
   the output; fast-prometheus's scrape path goes further and writes series with `Wire`
   directly, with no message per series at all.

data/lib/fast/protowire/field.rb

@@ -30,9 +30,15 @@ module Fast
 
       MAP_KEY_TYPES = %i[int32 int64 uint32 uint64 sint32 sint64 fixed32 fixed64 sfixed32 sfixed64 bool string].freeze
 
-      private_constant :SCALAR_WIRE_TYPES, :INTEGER_RANGES, :FIXED_FORMATS, :PACKABLE, :MAP_KEY_TYPES
+      # The field numbers a tag can carry, less the range the specification
+      # reserves for the implementation.
+      NUMBERS = (1..(1 << 29) - 1)
+      RESERVED_NUMBERS = (19_000..19_999)
 
-      attr_reader :name, :number, :type, :rule, :oneof, :ivar, :enum
+      private_constant :SCALAR_WIRE_TYPES, :INTEGER_RANGES, :FIXED_FORMATS, :PACKABLE, :MAP_KEY_TYPES,
+                       :NUMBERS, :RESERVED_NUMBERS
+
+      attr_reader :name, :number, :type, :rule, :oneof, :ivar, :enum, :wire_type
 
       # +type+ is a scalar Symbol, an Enum module, a Message class, or a
       # String / Proc naming a Message class resolved on first use (for
@@ -40,13 +46,17 @@ module Fast
       # :repeated or :map.
       def initialize(name, type, number, rule:, owner:, packed: nil, default: nil, oneof: nil, key_type: nil)
         @name = name
-        @number = number
+        @number = validate_number(number)
         @rule = rule
         @owner = owner
         @oneof = oneof
         @ivar = :"@#{name}"
         resolve_type(type)
+        @wire_type = rule == :map ? Wire::LENGTH_DELIMITED : SCALAR_WIRE_TYPES.fetch(@type)
+        raise ArgumentError, "#{name}: a #{@type} field cannot be packed" if packed && !packable?
+
         @packed = rule == :repeated && (packed.nil? ? owner.syntax == :proto3 && packable? : packed)
+        @strict_utf8 = @type == :string && owner.syntax == :proto3
         @default = default
         @tag = Wire.tag(number, packed? ? Wire::LENGTH_DELIMITED : wire_type)
         return unless map?
@@ -55,6 +65,9 @@ module Fast
 
         @key_field = Field.new(:key, key_type, 1, rule: :optional, owner: owner)
         @value_field = Field.new(:value, type, 2, rule: :optional, owner: owner)
+        # The only two tags an entry accepts: number and wire type together.
+        @key_tag = (1 << 3) | @key_field.wire_type
+        @value_tag = (2 << 3) | @value_field.wire_type
       end
 
       def message_class
@@ -65,10 +78,6 @@ module Fast
                            end
       end
 
-      def wire_type
-        map? ? Wire::LENGTH_DELIMITED : SCALAR_WIRE_TYPES.fetch(type)
-      end
-
       def repeated?
         rule == :repeated
       end
@@ -85,6 +94,13 @@ module Fast
         PACKABLE.include?(type)
       end
 
+      # Whether a value of +wire_type+ can be read into this field: its own,
+      # plus the packed form of a packable repeated field however it was
+      # declared. Anything else is schema drift, kept as an unknown field.
+      def accepts?(wire_type)
+        wire_type == self.wire_type || (repeated? && packable? && wire_type == Wire::LENGTH_DELIMITED)
+      end
+
       # Whether an unset field is distinguishable from one set to its default.
       def explicit_presence?
         rule != :implicit || !oneof.nil? || type == :message
@@ -142,11 +158,13 @@ module Fast
       end
 
       # Reads one occurrence of this field into +current+ (the value already
-      # held) and returns the value to store.
-      def decode(reader, wire_type, current)
+      # held) and returns the value to store. +message+ is the message being
+      # read into, for a map entry it cannot accept: that belongs among the
+      # message's unknown fields rather than in the map.
+      def decode(reader, wire_type, current, message)
         case rule
         when :repeated then decode_repeated(reader, wire_type, current)
-        when :map then decode_map_entry(reader, wire_type, current)
+        when :map then decode_map_entry(reader, wire_type, current, message)
         else
           if type == :message && current
             expect(reader, wire_type).read_nested { |nested| current.merge_from(nested) }
@@ -158,6 +176,15 @@ module Fast
 
       protected
 
+      # A second occurrence of a message field merges into the first; of
+      # anything else, replaces it. (A map's value field is read this way by
+      # the map field, which is why this is protected rather than private.)
+      def decode_singular(reader, wire_type, current)
+        return read_one(reader, wire_type) unless type == :message && current
+
+        expect(reader, wire_type).read_nested { |nested| current.merge_from(nested) }
+      end
+
       def coerce_one(value)
         case type
         when :string then coerce_string(value)
@@ -174,12 +201,20 @@ module Fast
         expect(reader, wire_type)
         case type
         when :message then reader.read_nested { |nested| message_class.new.merge_from(nested) }
-        when :string then reader.read_length_delimited.force_encoding(Encoding::UTF_8)
+        when :string
+          @strict_utf8 ? read_string(reader) : reader.read_length_delimited.force_encoding(Encoding::UTF_8)
         when :bytes then reader.read_length_delimited
         else read_scalar(reader)
         end
       end
 
+      # What a map entry holds for this field when the entry omits it: the
+      # type's default, except that a message value is an empty instance, as
+      # the reference materialises one (and nil would not encode).
+      def entry_default
+        type == :message ? message_class.new : default_value
+      end
+
       # Appends the tag and one value. A nested message is encoded straight
       # into the buffer behind a length prefix filled in after it, with no
       # buffer of its own; +width+ carries the prefix width from one value
@@ -204,6 +239,13 @@ module Fast
 
       private
 
+      def validate_number(number)
+        raise ArgumentError, "field number #{number} is reserved" if RESERVED_NUMBERS.cover?(number)
+        raise ArgumentError, "field number #{number} is outside #{NUMBERS}" unless NUMBERS.cover?(number)
+
+        number
+      end
+
       def resolve_type(type)
         case type
         when Symbol
@@ -292,8 +334,7 @@ module Fast
       # Each entry is a message { key = 1; value = 2 }.
       def map_step(ivar)
         tag = @tag
-        key = @key_field.writer
-        value = @value_field.writer
+        key, value = entry_writers
         width = 1
         lambda do |message, buffer|
           message.instance_variable_get(ivar).each do |k, v|
@@ -365,10 +406,13 @@ module Fast
         value.encoding == Encoding::BINARY ? value : value.b
       end
 
+      # A +float+ holds what the wire holds: the value narrowed to single
+      # precision on assignment, as the reference narrows it, so what is
+      # read back is what a decode of the encoding reads.
       def coerce_float(value)
         raise ::TypeError, "#{name} expects a number, got #{value.class}" unless value.is_a?(Numeric)
 
-        value.to_f
+        type == :float ? [value.to_f].pack("e").unpack1("e") : value.to_f
       end
 
       def coerce_bool(value)
@@ -413,30 +457,64 @@ module Fast
 
       def decode_repeated(reader, wire_type, values)
         if packable? && wire_type == Wire::LENGTH_DELIMITED
-          reader.read_nested { |packed| values << read_scalar(packed) until packed.eof? }
+          reader.read_packed { |packed| values << read_scalar(packed) until packed.eof? }
         else
           values << read_one(reader, wire_type)
         end
         values
       end
 
-      def decode_map_entry(reader, wire_type, hash)
+      # An entry the map accepts holds its key, its value, or both; one that
+      # carries anything else — an undeclared subfield, or the key or value
+      # with a wire type the entry does not accept — is not a map entry at
+      # all. The reference keeps such an entry among the parent message's
+      # unknown fields and leaves the map alone, so this does too.
+      def decode_map_entry(reader, wire_type, hash, message)
         expect(reader, wire_type).read_nested do |entry|
-          key = @key_field.default_value
-          value = @value_field.default_value
+          key = nil
+          value = nil
+          unknown = nil
           until entry.eof?
-            tag = entry.read_varint
-            case tag >> 3
-            when 1 then key = @key_field.read_one(entry, tag & 0x7)
-            when 2 then value = @value_field.decode(entry, tag & 0x7, value)
-            else entry.skip(tag & 0x7)
+            tag = entry.read_key
+            case tag
+            when @key_tag then key = @key_field.read_one(entry, tag & 0x7)
+            when @value_tag then value = @value_field.decode_singular(entry, tag & 0x7, value)
+            else
+              Wire.append_varint(unknown ||= String.new, tag)
+              unknown << entry.skip(tag & 0x7, tag >> 3)
             end
           end
-          hash[key] = value
+          if unknown
+            keep_entry(message, key, value, unknown)
+          else
+            # An absent key or value reads as its type's default, an empty
+            # message for a message-typed value, as the reference
+            # materialises one. (Only nil is absent: false is a bool key.)
+            hash[key.nil? ? @key_field.default_value : key] = value.nil? ? @value_field.entry_default : value
+          end
         end
         hash
       end
 
+      # A rejected entry, into the message's unknown fields (its own buffer,
+      # made here when the message has none yet) and written the way the
+      # reference writes one: the subfields it did carry, in number order and
+      # omitted when they are at their default — a message value is written
+      # whenever it was there — then the bytes it carried besides.
+      def keep_entry(message, key, value, unknown)
+        buffer = message.unknown_fields || message.instance_variable_set(:@unknown_fields, String.new)
+        write_key, write_value = entry_writers
+        Wire.append_length_delimited_from(buffer, @tag) do |b|
+          write_key.call(b, key) unless key.nil? || @key_field.omit?(key)
+          write_value.call(b, value) unless value.nil? || @value_field.omit?(value)
+          b << unknown
+        end
+      end
+
+      def entry_writers
+        @entry_writers ||= [@key_field.writer, @value_field.writer]
+      end
+
       def expect(reader, wire_type)
         return reader if wire_type == self.wire_type
 
@@ -450,13 +528,23 @@ module Fast
         when :int64 then signed(reader.read_varint, 64)
         when :uint32 then reader.read_varint & 0xFFFF_FFFF
         when :uint64 then reader.read_varint
-        when :sint32, :sint64 then Wire.unzigzag(reader.read_varint)
+        when :sint32 then Wire.unzigzag(reader.read_varint & 0xFFFF_FFFF)
+        when :sint64 then Wire.unzigzag(reader.read_varint)
         when :bool then reader.read_varint != 0
         when :double, :fixed64, :sfixed64 then reader.read_fixed(FIXED_FORMATS.fetch(type), 8)
         else reader.read_fixed(FIXED_FORMATS.fetch(type), 4)
         end
       end
 
+      # proto3 requires a parser to reject a string field that is not valid
+      # UTF-8; proto2 does not, and the reference keeps it.
+      def read_string(reader)
+        string = reader.read_length_delimited.force_encoding(Encoding::UTF_8)
+        raise DecodeError, "#{name}: string is not valid UTF-8" if @strict_utf8 && !string.valid_encoding?
+
+        string
+      end
+
       def signed(value, bits)
         value &= (1 << bits) - 1
         value >= (1 << (bits - 1)) ? value - (1 << bits) : value

data/lib/fast/protowire/message.rb

@@ -97,6 +97,7 @@ module Fast
         def compile_encoder
           steps = sorted_fields.map(&:encoder_step)
           define_method(:encode) do |buffer = String.new|
+            Wire.binary_buffer(buffer)
             steps.each { |step| step.call(self, buffer) }
             buffer << @unknown_fields if @unknown_fields
             buffer
@@ -172,17 +173,23 @@ module Fast
 
       # Reads fields from +reader+ into this message (protobuf merge
       # semantics: later scalars win, repeated fields append, nested messages
-      # merge) and returns self.
+      # merge) and returns self. A declared field arriving with a wire type
+      # it does not accept is schema drift rather than corruption, so it is
+      # kept as an unknown field, as the reference does; so is a map entry
+      # carrying a subfield the entry does not accept, kept whole.
       def merge_from(reader)
         until reader.eof?
           key = reader.read_varint
           wire_type = key & 0x7
-          field = self.class.fields_by_number[key >> 3]
-          if field
-            write_field(field, field.decode(reader, wire_type, instance_variable_get(field.ivar)))
+          number = key >> 3
+          raise DecodeError, "field number 0" if number.zero? # as Reader#read_key, inlined for the loop
+
+          field = self.class.fields_by_number[number]
+          if field&.accepts?(wire_type)
+            write_field(field, field.decode(reader, wire_type, instance_variable_get(field.ivar), self))
           else
             Wire.append_varint(@unknown_fields ||= String.new, key)
-            @unknown_fields << reader.skip(wire_type)
+            @unknown_fields << reader.skip(wire_type, number)
           end
         end
         self
@@ -197,8 +204,12 @@ module Fast
         end
       end
 
-      # Two messages are equal when every field reads the same; an implicit
-      # field set to its default is the same as one never set.
+      # Two messages are equal when every declared field reads the same; an
+      # implicit field set to its default is the same as one never set.
+      # Unknown fields are not compared, as the reference does not compare
+      # them: two messages that differ only in what they carry for fields
+      # this class never declared are the same message. Byte identity is
+      # +a.encode == b.encode+.
       def ==(other)
         other.class == self.class && comparable_values == other.comparable_values
       end
@@ -228,11 +239,10 @@ module Fast
       protected
 
       def comparable_values
-        values = self.class.fields.each_value.map do |field|
+        self.class.fields.each_value.map do |field|
           value = instance_variable_get(field.ivar)
           value.nil? && !field.explicit_presence? ? field.default_value : value
         end
-        values << @unknown_fields
       end
 
       private

data/lib/fast/protowire/reader.rb

@@ -8,10 +8,18 @@ module Fast
     # A cursor over an encoded message. Reads the primitive wire values and
     # skips what it is not asked to interpret.
     class Reader
+      # How deep nested messages and groups may go before the input is taken
+      # to be hostile rather than deep; the reference's limit.
+      MAX_DEPTH = 100
+
+      # Decoding is about bytes, so the input is read as bytes: a String
+      # tagged anything else is copied into a binary view once, here, rather
+      # than leaking its encoding into every slice handed out below.
       def initialize(buffer, position = 0, limit = buffer.bytesize)
-        @buffer = buffer
+        @buffer = buffer.encoding == Encoding::BINARY ? buffer : buffer.b
         @position = position
         @limit = limit
+        @depth = 0
       end
 
       attr_reader :position
@@ -22,10 +30,20 @@ module Fast
 
       # Returns [field number, wire type].
       def read_tag
-        key = read_varint
+        key = read_key
         [key >> 3, key & 0x7]
       end
 
+      # The next field's key, the varint holding its number and wire type
+      # together. Field number 0 exists on no wire: the reference rejects it
+      # wherever it appears, and so does this.
+      def read_key
+        key = read_varint
+        raise DecodeError, "field number 0" if key < 8
+
+        key
+      end
+
       # Most varints (tags, small lengths) are one byte; the loop is only
       # entered past it, and is a bare while because Kernel#loop costs an
       # object per call.
@@ -42,7 +60,9 @@ module Fast
           result |= (byte & 0x7f) << shift
           shift += 7
         end
-        result
+        # Only a tenth byte can carry bits above 64; every implementation
+        # truncates them rather than reading a wider value.
+        shift > 63 ? result & Wire::UINT64_MASK : result
       end
 
       def read_fixed32
@@ -75,29 +95,52 @@ module Fast
       end
 
       # Bounds the reader to the next length-delimited value for the block
-      # and returns the block's result: nested messages, packed fields and
-      # map entries are read in place, with no copy of their bytes.
+      # and returns the block's result, so a packed repeated field is read
+      # in place, with no copy of its bytes.
+      def read_packed
+        length = read_varint
+        limit = @limit
+        raise DecodeError, "truncated field" if @position + length > limit
+
+        @limit = @position + length
+        result = yield self
+        @position = @limit
+        @limit = limit
+        result
+      end
+
+      # The same, for a value read by recursing into it — a nested message or
+      # a map entry — counting the nesting so deeply nested input raises
+      # rather than overflowing the VM stack. A packed field holds scalars
+      # and does not recurse, so it costs no level and reads through
+      # read_packed; the bounding is spelled out twice rather than delegated
+      # because every nested message pays for the call.
       def read_nested
         length = read_varint
         limit = @limit
         raise DecodeError, "truncated field" if @position + length > limit
+        raise DecodeError, "nested deeper than #{MAX_DEPTH}" if @depth >= MAX_DEPTH
 
         @limit = @position + length
+        @depth += 1
         result = yield self
+        @depth -= 1
         @position = @limit
         @limit = limit
         result
       end
 
       # Skips one value of +wire_type+ and returns its raw bytes, so unknown
-      # fields survive a decode/encode round trip.
-      def skip(wire_type)
+      # fields survive a decode/encode round trip. +number+ is the field
+      # number the value arrived under: a group is closed by its own number
+      # and nothing else. Without one, any END_GROUP closes it.
+      def skip(wire_type, number = nil)
         start = @position
         case wire_type
         when Wire::VARINT then read_varint
         when Wire::FIXED64 then read_bytes(8)
         when Wire::LENGTH_DELIMITED then read_length_delimited
-        when Wire::START_GROUP then skip_group
+        when Wire::START_GROUP then skip_group(number)
         when Wire::FIXED32 then read_bytes(4)
         else raise DecodeError, "unknown wire type #{wire_type}"
         end
@@ -114,14 +157,20 @@ module Fast
         byte
       end
 
-      def skip_group
+      def skip_group(number)
+        raise DecodeError, "nested deeper than #{MAX_DEPTH}" if @depth >= MAX_DEPTH
+
+        @depth += 1
         loop do
-          number, wire_type = read_tag
-          return if wire_type == Wire::END_GROUP
-          raise DecodeError, "invalid group" if number.zero?
+          inner, wire_type = read_tag
+          if wire_type == Wire::END_GROUP
+            raise DecodeError, "group #{number} closed by #{inner}" unless number.nil? || inner == number
 
-          skip(wire_type)
+            break
+          end
+          skip(wire_type, inner)
         end
+        @depth -= 1
       end
     end
   end

data/lib/fast/protowire/version.rb

@@ -2,6 +2,6 @@
 
 module Fast
   module Protowire
-    VERSION = "0.2.0"
+    VERSION = "0.3.0"
   end
 end

data/lib/fast/protowire/wire.rb

@@ -17,10 +17,21 @@ module Fast
       UINT64_MASK = (1 << 64) - 1
       # Zero bytes a length prefix is widened to, by varint width.
       PLACEHOLDERS = Array.new(11) { |width| ("\0" * width).b.freeze }.freeze
-      private_constant :UINT64_MASK, :PLACEHOLDERS
+      private_constant :PLACEHOLDERS
 
       module_function
 
+      # A buffer written to here must be binary: appending a byte to a text
+      # String appends that encoding's character for it instead, silently
+      # writing something else. An empty buffer is simply retagged, so the
+      # usual ways of making one (+"", String.new("")) still work.
+      def binary_buffer(buffer)
+        return buffer if buffer.encoding == Encoding::BINARY
+        raise ArgumentError, "buffer must be a binary String, got #{buffer.encoding}" unless buffer.empty?
+
+        buffer.force_encoding(Encoding::BINARY)
+      end
+
       # The key for +number+ / +wire_type+ as frozen bytes.
       def tag(number, wire_type)
         varint((number << 3) | wire_type).freeze

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fast-protowire
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Eric Jacobs