fast-protowire 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a0c2962e443067325c5f4a089b72e499b9e273f8f563464e9830ed9cdf6eccfa
+  data.tar.gz: cdc454d06627bf2d1f28ddc35127024e119e06e748120cfceac91fa7e4e40860
+SHA512:
+  metadata.gz: 5b6b5ea5c39ebc3fb9b25490fbea312dccdaaf450ab08b7a9be794056dbc3565e814f98890629664e405333e340e61d493760a57adc99b072b2cf59ff8399bc6
+  data.tar.gz: b788d96fe73458a33391b5339d35f7be57a37ec3bb203a2d645c590a5cb98d6ef4f9a5641788d8a12f20c7a51911c27287cba01cbac49e0d4b3c9e22f944be1d

data/AGENTS.md

@@ -0,0 +1,35 @@
+# AGENTS.md
+
+Standing context for agents working in this repository.
+
+## Commands
+
+```bash
+bundle install
+bundle exec sus
+bundle exec rubocop
+protoc --proto_path=fixtures/proto --ruby_out=fixtures/pb fixtures/proto/*.proto   # regenerate the reference schemas
+```
+
+## What this is
+
+The Protocol Buffers wire format, and nothing above it: a `Wire` writer, a
+`Reader`, and a `Message` DSL that declares fields and derives encode/decode
+from the declaration. No descriptors, reflection, JSON mapping or generated
+code. `google-protobuf` is a development dependency only — the parity suite
+(`test/fast/protowire/parity.rb`) encodes the same attributes with both and
+requires identical bytes.
+
+## Style Rules
+
+- 2-space indentation, `# frozen_string_literal: true` on every source file, double-quoted strings
+- No runtime dependencies. If a feature needs one, it does not belong here.
+- Encoded bytes must match the reference implementation for every case the
+  DSL can express; add a parity case before changing encoding behavior.
+
+## Test Rules
+
+- `sus`; tests are terse and exercise public interfaces only
+- No mock/stub of the class under test
+- Schemas used by tests live in `fixtures/proto` (source of truth), `fixtures/pb`
+  (protoc output, checked in) and `fixtures/schema.rb` (the DSL mirror)

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+- Initial release: `Wire`, `Reader`, `Enum` and the `Message` DSL covering
+  every scalar type, proto2 and proto3 presence, packed and unpacked
+  repeated fields, maps, oneofs, nested and recursive messages, unknown
+  field preservation and merge-on-decode, verified byte-for-byte against
+  `google-protobuf`. `encode` is compiled per message class.

data/README.md

@@ -0,0 +1,82 @@
+# fast-protowire
+
+The Protocol Buffers wire format for Ruby, with nothing on top of it: declare
+a message's fields, get `encode` and `decode` for exactly those bytes.
+
+It is not a replacement for `google-protobuf`. There are no descriptors,
+no reflection, no JSON mapping and no generated code. It exists for
+libraries that emit or read a fixed, known schema and want the memory cost
+of doing so to be roughly the size of the encoded output, rather than a
+native message object and arena for every field, as `google-protobuf`
+allocates.
+
+## Installation
+
+```ruby
+gem "fast-protowire"
+```
+
+`require "fast/protowire"` has no dependencies.
+
+## Quickstart
+
+Declarations mirror the `.proto` text. Run this with `bundle exec ruby` from a project
+that has the gem:
+
+```ruby
+require "fast/protowire"
+
+class LabelPair < Fast::Protowire::Message
+  field :name, :string, 1
+  field :value, :string, 2
+end
+
+class Counter < Fast::Protowire::Message
+  field :value, :double, 1
+end
+
+class Metric < Fast::Protowire::Message
+  repeated :label, LabelPair, 1
+  field :counter, Counter, 3
+end
+
+metric = Metric.new(label: [{ name: "method", value: "GET" }], counter: { value: 12.0 })
+bytes = metric.encode
+Metric.decode(bytes) == metric # => true
+```
+
+Fields encode in field-number order, unknown fields survive a decode/encode round trip,
+and the output is byte-identical to what `protoc`-generated code and `google-protobuf`
+produce for the same values. The [tutorial](docs/tutorials/encode-a-message.md) walks
+through a full schema and proves that.
+
+## Documentation
+
+### Tutorials
+
+- [Tutorial: declare, encode and decode a message](docs/tutorials/encode-a-message.md) — declare a slice of the Prometheus client model, encode, decode, and check the bytes against google-protobuf.
+
+### How-to guides
+
+- [How to declare a schema from a .proto file](docs/how-to/declare-a-schema-from-proto.md) — the keyword-by-keyword translation, including proto2, oneofs, maps, packing and forward references.
+- [How to stream a large repeated field](docs/how-to/stream-a-large-repeated-field.md) — append thousands of entries to a message without holding them all at once.
+- [How to use declared messages with protocol-grpc](docs/how-to/use-with-protocol-grpc.md) — request and response classes for `protocol-grpc` and `async-grpc` stubs.
+- [How to verify parity against google-protobuf](docs/how-to/verify-parity-against-google-protobuf.md) — add a case to the parity suite, or run the same check in your project.
+
+### Reference
+
+- [Reference: Message](docs/reference/message.md) — the declaration DSL, the instance API, and the errors.
+- [Reference: field types](docs/reference/field-types.md) — every scalar type with its wire type, Ruby value, range, presence and packing rules; enums.
+- [Reference: Wire and Reader](docs/reference/wire.md) — the primitives for writing and reading the wire format directly.
+
+### Explanation
+
+- [Design: the wire format and nothing else](docs/explanation/design.md) — what google-protobuf costs per message, what this gem does instead, and what it leaves out.
+- [How encoding works](docs/explanation/encoding.md) — compiled encoders, buffers, field order, presence, and decoding.
+
+## Development
+
+```bash
+bundle exec sus
+bundle exec rubocop
+```

data/lib/fast/protowire/enum.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+module Fast
+  module Protowire
+    # Builds an enum module: one constant per value, plus lookup in both
+    # directions. Fields declared with the module store Symbols for known
+    # values and Integers for values the module does not name.
+    #
+    #   MetricType = Fast::Protowire::Enum.define(COUNTER: 0, GAUGE: 1)
+    #   MetricType::GAUGE           # => 1
+    #   MetricType.lookup(1)        # => :GAUGE
+    #   MetricType.resolve(:GAUGE)  # => 1
+    module Enum
+      def self.define(**values)
+        Module.new do
+          extend Enum
+          values.each { |name, number| const_set(name, number) }
+          @by_name = values.freeze
+          @by_number = values.invert.freeze
+        end
+      end
+
+      def lookup(number)
+        @by_number[number]
+      end
+
+      def resolve(name)
+        @by_name[name]
+      end
+
+      def values
+        @by_name
+      end
+
+      # The value an unset field reads as: proto3 requires a zero value,
+      # proto2 falls back to the first declared one.
+      def default
+        @by_number[0] || @by_name.each_key.first
+      end
+    end
+  end
+end

data/lib/fast/protowire/errors.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Fast
+  module Protowire
+    class Error < StandardError; end
+
+    # Bytes that do not parse as a message of the declared shape.
+    class DecodeError < Error; end
+  end
+end

data/lib/fast/protowire/field.rb

@@ -0,0 +1,501 @@
+# frozen_string_literal: true
+
+require_relative "errors"
+require_relative "wire"
+require_relative "enum"
+
+module Fast
+  module Protowire
+    # One declared field: its number, type and rule, and how to validate,
+    # encode and decode values of it. Built by the Message DSL.
+    class Field
+      SCALAR_WIRE_TYPES = {
+        double: Wire::FIXED64, float: Wire::FIXED32,
+        int32: Wire::VARINT, int64: Wire::VARINT, uint32: Wire::VARINT, uint64: Wire::VARINT,
+        sint32: Wire::VARINT, sint64: Wire::VARINT, bool: Wire::VARINT, enum: Wire::VARINT,
+        fixed32: Wire::FIXED32, sfixed32: Wire::FIXED32, fixed64: Wire::FIXED64, sfixed64: Wire::FIXED64,
+        string: Wire::LENGTH_DELIMITED, bytes: Wire::LENGTH_DELIMITED, message: Wire::LENGTH_DELIMITED
+      }.freeze
+
+      INTEGER_RANGES = {
+        int32: (-(1 << 31))...(1 << 31), sint32: (-(1 << 31))...(1 << 31), sfixed32: (-(1 << 31))...(1 << 31),
+        uint32: 0...(1 << 32), fixed32: 0...(1 << 32),
+        int64: (-(1 << 63))...(1 << 63), sint64: (-(1 << 63))...(1 << 63), sfixed64: (-(1 << 63))...(1 << 63),
+        uint64: 0...(1 << 64), fixed64: 0...(1 << 64)
+      }.freeze
+
+      FIXED_FORMATS = { double: "E", float: "e", fixed32: "L<", sfixed32: "l<", fixed64: "Q<", sfixed64: "q<" }.freeze
+
+      PACKABLE = (SCALAR_WIRE_TYPES.keys - %i[string bytes message]).freeze
+
+      MAP_KEY_TYPES = %i[int32 int64 uint32 uint64 sint32 sint64 fixed32 fixed64 sfixed32 sfixed64 bool string].freeze
+
+      private_constant :SCALAR_WIRE_TYPES, :INTEGER_RANGES, :FIXED_FORMATS, :PACKABLE, :MAP_KEY_TYPES
+
+      attr_reader :name, :number, :type, :rule, :oneof, :ivar, :enum
+
+      # +type+ is a scalar Symbol, an Enum module, a Message class, or a
+      # String / Proc naming a Message class resolved on first use (for
+      # recursive schemas). +rule+ is :implicit, :optional, :required,
+      # :repeated or :map.
+      def initialize(name, type, number, rule:, owner:, packed: nil, default: nil, oneof: nil, key_type: nil)
+        @name = name
+        @number = number
+        @rule = rule
+        @owner = owner
+        @oneof = oneof
+        @ivar = :"@#{name}"
+        resolve_type(type)
+        @packed = rule == :repeated && (packed.nil? ? owner.syntax == :proto3 && packable? : packed)
+        @default = default
+        @tag = Wire.tag(number, packed? ? Wire::LENGTH_DELIMITED : wire_type)
+        return unless map?
+
+        raise ArgumentError, "map key type #{key_type.inspect} is not allowed" unless MAP_KEY_TYPES.include?(key_type)
+
+        @key_field = Field.new(:key, key_type, 1, rule: :optional, owner: owner)
+        @value_field = Field.new(:value, type, 2, rule: :optional, owner: owner)
+      end
+
+      def message_class
+        @message_class ||= case @message_ref
+                           when Proc then @message_ref.call
+                           when String then namespace_of(@owner).const_get(@message_ref)
+                           else @message_ref
+                           end
+      end
+
+      def wire_type
+        map? ? Wire::LENGTH_DELIMITED : SCALAR_WIRE_TYPES.fetch(type)
+      end
+
+      def repeated?
+        rule == :repeated
+      end
+
+      def map?
+        rule == :map
+      end
+
+      def packed?
+        @packed
+      end
+
+      def packable?
+        PACKABLE.include?(type)
+      end
+
+      # Whether an unset field is distinguishable from one set to its default.
+      def explicit_presence?
+        rule != :implicit || !oneof.nil? || type == :message
+      end
+
+      def default_value
+        case rule
+        when :repeated then []
+        when :map then {}
+        else scalar_default
+        end
+      end
+
+      # True when a field without presence carries nothing worth emitting.
+      # Floats compare bitwise, as the reference encoder does: -0.0 is sent.
+      def omit?(value)
+        return value.empty? if repeated? || map?
+
+        case type
+        when :enum then enum_number(value).zero?
+        when :message then false
+        when :double, :float then value.zero? && (1.0 / value).positive?
+        else value == scalar_default
+        end
+      end
+
+      # A step of the compiled Message#encode: a lambda taking the message
+      # and the buffer that appends this field, or nothing when the field is
+      # unset or at a default it need not send. Everything the step needs
+      # (ivar name, tag bytes, enum, nested writers) is captured when it is
+      # built, so encoding does no per-field dispatch.
+      def encoder_step
+        ivar = @ivar
+        case rule
+        when :repeated then repeated_step(ivar)
+        when :map then map_step(ivar)
+        else singular_step(ivar)
+        end
+      end
+
+      # Validates and normalizes a value the way an assignment would.
+      def coerce(value)
+        case rule
+        when :repeated
+          raise ::TypeError, "#{name} expects an Array" unless value.respond_to?(:to_ary)
+
+          value.to_ary.map { |v| coerce_one(v) }
+        when :map
+          raise ::TypeError, "#{name} expects a Hash" unless value.respond_to?(:to_hash)
+
+          value.to_hash.to_h { |k, v| [@key_field.coerce_one(k), @value_field.coerce_one(v)] }
+        else
+          value.nil? ? nil : coerce_one(value)
+        end
+      end
+
+      def encode(buffer, value)
+        case rule
+        when :repeated then encode_repeated(buffer, value)
+        when :map then encode_map(buffer, value)
+        else encode_one(buffer, value)
+        end
+      end
+
+      # Reads one occurrence of this field into +current+ (the value already
+      # held) and returns the value to store.
+      def decode(reader, wire_type, current)
+        case rule
+        when :repeated then decode_repeated(reader, wire_type, current)
+        when :map then decode_map_entry(reader, wire_type, current)
+        else
+          if type == :message && current
+            current.merge_from(Reader.new(expect(reader, wire_type).read_length_delimited))
+          else
+            read_one(reader, wire_type)
+          end
+        end
+      end
+
+      protected
+
+      def coerce_one(value)
+        case type
+        when :string then coerce_string(value)
+        when :bytes then coerce_bytes(value)
+        when :double, :float then coerce_float(value)
+        when :bool then coerce_bool(value)
+        when :enum then coerce_enum(value)
+        when :message then coerce_message(value)
+        else coerce_integer(value)
+        end
+      end
+
+      def encode_one(buffer, value)
+        case type
+        when :message then Wire.append_length_delimited(buffer, @tag, value.encode)
+        when :string, :bytes then Wire.append_length_delimited(buffer, @tag, value)
+        else
+          buffer << @tag
+          append_scalar(buffer, value)
+        end
+      end
+
+      def read_one(reader, wire_type)
+        expect(reader, wire_type)
+        case type
+        when :message then message_class.decode(reader.read_length_delimited)
+        when :string then reader.read_length_delimited.force_encoding(Encoding::UTF_8)
+        when :bytes then reader.read_length_delimited
+        else read_scalar(reader)
+        end
+      end
+
+      private
+
+      def resolve_type(type)
+        case type
+        when Symbol
+          unless SCALAR_WIRE_TYPES.key?(type) && type != :message
+            raise ArgumentError,
+                  "unknown field type #{type.inspect}"
+          end
+
+          @type = type
+        when Module
+          if type.is_a?(Enum)
+            @type = :enum
+            @enum = type
+          else
+            @type = :message
+            @message_ref = type
+          end
+        when String, Proc
+          @type = :message
+          @message_ref = type
+        else
+          raise ArgumentError, "unknown field type #{type.inspect}"
+        end
+      end
+
+      # A String type names a class relative to where the owner is declared,
+      # so a message can refer to itself or a sibling declared later.
+      def namespace_of(owner)
+        namespace = owner.name.to_s.rpartition("::").first
+        namespace.empty? ? Object : Object.const_get(namespace)
+      end
+
+      def scalar_default
+        return @default unless @default.nil?
+
+        case type
+        when :string then ""
+        when :bytes then "".b
+        when :double, :float then 0.0
+        when :bool then false
+        when :enum then @enum.default
+        when :message then nil
+        else 0
+        end
+      end
+
+      def enum_number(value)
+        value.is_a?(Symbol) ? @enum.resolve(value) : value
+      end
+
+      # -- compiled encoder steps --------------------------------------------
+
+      def singular_step(ivar)
+        write = writer
+        if explicit_presence?
+          ->(message, buffer) { (value = message.instance_variable_get(ivar)).nil? || write.call(buffer, value) }
+        else
+          omit = omitter
+          lambda do |message, buffer|
+            value = message.instance_variable_get(ivar)
+            write.call(buffer, value) unless value.nil? || omit.call(value)
+          end
+        end
+      end
+
+      def repeated_step(ivar)
+        if packed?
+          tag = @tag
+          scalar = scalar_writer
+          lambda do |message, buffer|
+            values = message.instance_variable_get(ivar)
+            next if values.empty?
+
+            payload = String.new
+            values.each { |value| scalar.call(payload, value) }
+            Wire.append_length_delimited(buffer, tag, payload)
+          end
+        else
+          write = writer
+          ->(message, buffer) { message.instance_variable_get(ivar).each { |value| write.call(buffer, value) } }
+        end
+      end
+
+      def map_step(ivar)
+        tag = @tag
+        key = @key_field.writer
+        value = @value_field.writer
+        lambda do |message, buffer|
+          message.instance_variable_get(ivar).each do |k, v|
+            entry = String.new
+            key.call(entry, k)
+            value.call(entry, v)
+            Wire.append_length_delimited(buffer, tag, entry)
+          end
+        end
+      end
+
+      # Appends the tag and one value.
+      def writer
+        tag = @tag
+        case type
+        when :message then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value.encode) }
+        when :string, :bytes then ->(buffer, value) { Wire.append_length_delimited(buffer, tag, value) }
+        else
+          scalar = scalar_writer
+          lambda do |buffer, value|
+            buffer << tag
+            scalar.call(buffer, value)
+          end
+        end
+      end
+
+      # Appends one scalar value without its tag, as packed fields need.
+      def scalar_writer
+        case type
+        when :int32, :int64, :uint32, :uint64 then ->(buffer, value) { Wire.append_varint(buffer, value) }
+        when :sint32 then ->(buffer, value) { Wire.append_varint(buffer, Wire.zigzag32(value)) }
+        when :sint64 then ->(buffer, value) { Wire.append_varint(buffer, Wire.zigzag64(value)) }
+        when :bool then ->(buffer, value) { buffer << (value ? 1 : 0) }
+        when :enum
+          enum = @enum
+          ->(buffer, value) { Wire.append_varint(buffer, value.is_a?(Symbol) ? enum.resolve(value) : value) }
+        else
+          format = FIXED_FORMATS.fetch(type)
+          ->(buffer, value) { [value].pack(format, buffer: buffer) }
+        end
+      end
+
+      # Decides whether a field without presence is at a value it need not
+      # send. Floats compare bitwise, as the reference encoder does.
+      def omitter
+        case type
+        when :string, :bytes then ->(value) { value.empty? } # rubocop:disable Style/SymbolProc
+        when :bool then ->(value) { !value } # rubocop:disable Style/SymbolProc
+        when :double, :float then ->(value) { value.zero? && (1.0 / value).positive? }
+        when :enum
+          enum = @enum
+          ->(value) { (value.is_a?(Symbol) ? enum.resolve(value) : value).zero? }
+        else
+          default = scalar_default
+          ->(value) { value == default }
+        end
+      end
+
+      # -- validation --------------------------------------------------------
+
+      def coerce_string(value)
+        raise ::TypeError, "#{name} expects a String, got #{value.class}" unless value.is_a?(String)
+
+        string = case value.encoding
+                 when Encoding::UTF_8 then value
+                 when Encoding::BINARY then value.dup.force_encoding(Encoding::UTF_8)
+                 else value.encode(Encoding::UTF_8)
+                 end
+        raise ::ArgumentError, "#{name}: string is not valid UTF-8" unless string.valid_encoding?
+
+        string
+      end
+
+      def coerce_bytes(value)
+        raise ::TypeError, "#{name} expects a String, got #{value.class}" unless value.is_a?(String)
+
+        value.encoding == Encoding::BINARY ? value : value.b
+      end
+
+      def coerce_float(value)
+        raise ::TypeError, "#{name} expects a number, got #{value.class}" unless value.is_a?(Numeric)
+
+        value.to_f
+      end
+
+      def coerce_bool(value)
+        return value if [true, false].include?(value)
+
+        raise ::TypeError, "#{name} expects true or false, got #{value.class}"
+      end
+
+      def coerce_enum(value)
+        case value
+        when Symbol
+          raise ::RangeError, "#{name}: unknown enum value #{value.inspect}" unless @enum.resolve(value)
+
+          value
+        when Integer
+          raise ::RangeError, "#{name}: #{value} is out of range" unless INTEGER_RANGES[:int32].cover?(value)
+
+          @enum.lookup(value) || value
+        else
+          raise ::TypeError, "#{name} expects a Symbol or Integer, got #{value.class}"
+        end
+      end
+
+      def coerce_message(value)
+        return message_class.new(value) if value.is_a?(Hash)
+        return value if value.is_a?(message_class)
+
+        raise ::TypeError, "#{name} expects a #{message_class}, got #{value.class}"
+      end
+
+      def coerce_integer(value)
+        raise ::TypeError, "#{name} expects an Integer, got #{value.class}" unless value.is_a?(Numeric)
+
+        integer = value.to_i
+        raise ::RangeError, "#{name}: #{value} is not an integer" unless integer == value
+        raise ::RangeError, "#{name}: #{value} is out of range" unless INTEGER_RANGES.fetch(type).cover?(integer)
+
+        integer
+      end
+
+      # -- encoding ----------------------------------------------------------
+
+      def encode_repeated(buffer, values)
+        return if values.empty?
+
+        if packed?
+          Wire.append_length_delimited_from(buffer, @tag) { |b| values.each { |v| append_scalar(b, v) } }
+        else
+          values.each { |v| encode_one(buffer, v) }
+        end
+      end
+
+      def encode_map(buffer, hash)
+        hash.each do |key, value|
+          Wire.append_length_delimited_from(buffer, @tag) do |entry|
+            @key_field.encode_one(entry, key)
+            @value_field.encode_one(entry, value)
+          end
+        end
+      end
+
+      def append_scalar(buffer, value)
+        case type
+        when :int32, :int64, :uint32, :uint64 then Wire.append_varint(buffer, value)
+        when :sint32 then Wire.append_varint(buffer, Wire.zigzag32(value))
+        when :sint64 then Wire.append_varint(buffer, Wire.zigzag64(value))
+        when :bool then buffer << (value ? 1 : 0)
+        when :enum then Wire.append_varint(buffer, enum_number(value))
+        else buffer << [value].pack(FIXED_FORMATS.fetch(type))
+        end
+      end
+
+      # -- decoding ----------------------------------------------------------
+
+      def decode_repeated(reader, wire_type, values)
+        if packable? && wire_type == Wire::LENGTH_DELIMITED
+          packed = reader.read_packed
+          values << read_scalar(packed) until packed.eof?
+        else
+          values << read_one(reader, wire_type)
+        end
+        values
+      end
+
+      def decode_map_entry(reader, wire_type, hash)
+        entry = Reader.new(expect(reader, wire_type).read_length_delimited)
+        key = @key_field.default_value
+        value = @value_field.default_value
+        until entry.eof?
+          number, entry_wire_type = entry.read_tag
+          case number
+          when 1 then key = @key_field.read_one(entry, entry_wire_type)
+          when 2 then value = @value_field.decode(entry, entry_wire_type, value)
+          else entry.skip(entry_wire_type)
+          end
+        end
+        hash[key] = value
+        hash
+      end
+
+      def expect(reader, wire_type)
+        return reader if wire_type == self.wire_type
+
+        raise DecodeError, "field #{name} (#{number}) has wire type #{wire_type}, expected #{self.wire_type}"
+      end
+
+      def read_scalar(reader)
+        case type
+        when :int32 then signed(reader.read_varint, 32)
+        when :enum then coerce_enum(signed(reader.read_varint, 32))
+        when :int64 then signed(reader.read_varint, 64)
+        when :uint32 then reader.read_varint & 0xFFFF_FFFF
+        when :uint64 then reader.read_varint
+        when :sint32, :sint64 then Wire.unzigzag(reader.read_varint)
+        when :bool then reader.read_varint != 0
+        when :double, :fixed64, :sfixed64 then reader.read_bytes(8).unpack1(FIXED_FORMATS.fetch(type))
+        else reader.read_bytes(4).unpack1(FIXED_FORMATS.fetch(type))
+        end
+      end
+
+      def signed(value, bits)
+        value &= (1 << bits) - 1
+        value >= (1 << (bits - 1)) ? value - (1 << bits) : value
+      end
+
+      # A map's key and value fields build the entry writer together.
+      protected :writer
+    end
+  end
+end

data/lib/fast/protowire/message.rb

@@ -0,0 +1,263 @@
+# frozen_string_literal: true
+
+require_relative "errors"
+require_relative "wire"
+require_relative "reader"
+require_relative "field"
+
+module Fast
+  module Protowire
+    # Base class for declared messages. Subclasses describe their fields with
+    # the class-level DSL, mirroring the .proto text:
+    #
+    #   class LabelPair < Fast::Protowire::Message
+    #     syntax :proto2
+    #     optional :name, :string, 1
+    #     optional :value, :string, 2
+    #   end
+    #
+    #   class Metric < Fast::Protowire::Message
+    #     repeated :label, LabelPair, 1
+    #     optional :counter, Counter, 3
+    #   end
+    #
+    # +field+ declares proto3 implicit presence (omitted when at the default);
+    # +optional+ / +required+ declare explicit presence (has_x? tells set from
+    # unset); +repeated+ and +map+ hold Arrays and Hashes; +oneof+ groups
+    # fields so setting one clears the others. Fields encode in field-number
+    # order, unknown fields survive decode and re-encode, so bytes match what
+    # the reference implementation produces for the same values.
+    class Message
+      class << self
+        attr_reader :fields, :fields_by_number, :oneofs
+
+        def inherited(subclass)
+          super
+          subclass.instance_variable_set(:@fields, {})
+          subclass.instance_variable_set(:@fields_by_number, {})
+          subclass.instance_variable_set(:@oneofs, {})
+          subclass.instance_variable_set(:@syntax, :proto3)
+        end
+
+        def syntax(value = nil)
+          return @syntax if value.nil?
+
+          raise ArgumentError, "syntax must be :proto2 or :proto3" unless %i[proto2 proto3].include?(value)
+
+          @syntax = value
+        end
+
+        # proto3 `type name = N;` — implicit presence. Under proto2 syntax
+        # this is the same as +optional+, since proto2 has no implicit fields.
+        def field(name, type, number, **options)
+          add(name, type, number, rule: @syntax == :proto2 ? :optional : :implicit, **options)
+        end
+
+        def optional(name, type, number, **options)
+          add(name, type, number, rule: :optional, **options)
+        end
+
+        def required(name, type, number, **options)
+          add(name, type, number, rule: :required, **options)
+        end
+
+        def repeated(name, type, number, packed: nil)
+          add(name, type, number, rule: :repeated, packed: packed)
+        end
+
+        def map(name, key_type, value_type, number)
+          add(name, value_type, number, rule: :map, key_type: key_type)
+        end
+
+        # Fields declared inside the block are members of the oneof +name+;
+        # the message gains a +name+ reader returning the set member's name.
+        def oneof(name, &block)
+          raise ArgumentError, "oneof declarations do not nest" if @current_oneof
+
+          @current_oneof = name
+          @oneofs[name] = []
+          instance_eval(&block)
+          members = @oneofs[name]
+          define_method(name) { members.find { |member| !instance_variable_get(:"@#{member}").nil? } }
+        ensure
+          @current_oneof = nil
+        end
+
+        def decode(bytes)
+          new.merge_from(Reader.new(bytes))
+        end
+
+        def encode(message)
+          message.encode
+        end
+
+        # Defines this class's own #encode from one step per field in number
+        # order (see Field#encoder_step). Runs once, on the first encode after
+        # the last declaration.
+        def compile_encoder
+          steps = sorted_fields.map(&:encoder_step)
+          define_method(:encode) do |buffer = String.new|
+            steps.each { |step| step.call(self, buffer) }
+            buffer << @unknown_fields if @unknown_fields
+            buffer
+          end
+        end
+
+        def sorted_fields
+          @sorted_fields ||= @fields.values.sort_by(&:number).freeze
+        end
+
+        def container_fields
+          @container_fields ||= @fields.values.select { |field| field.repeated? || field.map? }.freeze
+        end
+
+        private
+
+        def add(name, type, number, rule:, **options)
+          raise ArgumentError, "field #{name} already declared" if @fields.key?(name)
+          raise ArgumentError, "field number #{number} already used" if @fields_by_number.key?(number)
+
+          field = Field.new(name, type, number, rule: rule, owner: self, oneof: @current_oneof, **options)
+          @fields[name] = field
+          @fields_by_number[number] = field
+          @oneofs[@current_oneof] << name if @current_oneof
+          @sorted_fields = nil
+          @container_fields = nil
+          remove_method(:encode) if instance_methods(false).include?(:encode)
+          define_accessors(field)
+        end
+
+        def define_accessors(field)
+          ivar = field.ivar
+          define_method(field.name) do
+            value = instance_variable_get(ivar)
+            value.nil? ? field.default_value : value
+          end
+          define_method(:"#{field.name}=") { |value| write_field(field, field.coerce(value)) }
+          return unless field.explicit_presence?
+
+          define_method(:"has_#{field.name}?") { !instance_variable_get(ivar).nil? }
+          define_method(:"clear_#{field.name}") { instance_variable_set(ivar, nil) }
+        end
+      end
+
+      # Scalar and message ivars stay unset until written (an unset ivar reads
+      # as nil); repeated and map fields get their container up front so it
+      # can be mutated in place.
+      def initialize(attributes = nil, **keywords)
+        self.class.container_fields.each { |field| instance_variable_set(field.ivar, field.default_value) }
+        @unknown_fields = nil
+        (attributes || keywords).each do |name, value|
+          field = self.class.fields[name.to_sym]
+          raise ArgumentError, "unknown field #{name.inspect} for #{self.class}" unless field
+
+          write_field(field, field.coerce(value))
+        end
+      end
+
+      attr_reader :unknown_fields
+
+      # Appends this message's bytes to +buffer+ and returns it. The first
+      # call compiles the class's own encode (see compile_encoder); this
+      # generic one is only ever reached before that.
+      def encode(buffer = String.new)
+        self.class.compile_encoder
+        encode(buffer)
+      end
+
+      def to_proto(buffer = String.new)
+        encode(buffer)
+      end
+
+      # Reads fields from +reader+ into this message (protobuf merge
+      # semantics: later scalars win, repeated fields append, nested messages
+      # merge) and returns self.
+      def merge_from(reader)
+        until reader.eof?
+          number, wire_type = reader.read_tag
+          field = self.class.fields_by_number[number]
+          if field
+            write_field(field, field.decode(reader, wire_type, instance_variable_get(field.ivar)))
+          else
+            (@unknown_fields ||= String.new) << Wire.varint((number << 3) | wire_type) << reader.skip(wire_type)
+          end
+        end
+        self
+      end
+
+      def to_h
+        self.class.fields.each_value.with_object({}) do |field, hash|
+          value = instance_variable_get(field.ivar)
+          next if value.nil? && field.explicit_presence?
+
+          hash[field.name] = hashify(value.nil? ? field.default_value : value)
+        end
+      end
+
+      # Two messages are equal when every field reads the same; an implicit
+      # field set to its default is the same as one never set.
+      def ==(other)
+        other.class == self.class && comparable_values == other.comparable_values
+      end
+      alias eql? ==
+
+      def hash
+        comparable_values.hash
+      end
+
+      def inspect
+        set = self.class.fields.each_value.filter_map do |field|
+          value = instance_variable_get(field.ivar)
+          "#{field.name}: #{value.inspect}" unless value.nil? || field.omit?(value)
+        end
+        "#<#{self.class.name} #{set.join(', ')}>"
+      end
+
+      def initialize_copy(source)
+        super
+        self.class.fields.each_value do |field|
+          value = source.instance_variable_get(field.ivar)
+          instance_variable_set(field.ivar, deep_copy(value))
+        end
+        @unknown_fields = source.unknown_fields&.dup
+      end
+
+      protected
+
+      def comparable_values
+        values = self.class.fields.each_value.map do |field|
+          value = instance_variable_get(field.ivar)
+          value.nil? && !field.explicit_presence? ? field.default_value : value
+        end
+        values << @unknown_fields
+      end
+
+      private
+
+      def write_field(field, value)
+        if field.oneof && !value.nil?
+          self.class.oneofs[field.oneof].each { |member| instance_variable_set(:"@#{member}", nil) }
+        end
+        instance_variable_set(field.ivar, value)
+      end
+
+      def hashify(value)
+        case value
+        when Message then value.to_h
+        when Array then value.map { |v| v.is_a?(Message) ? v.to_h : v }
+        when Hash then value.transform_values { |v| v.is_a?(Message) ? v.to_h : v }
+        else value
+        end
+      end
+
+      def deep_copy(value)
+        case value
+        when Message, String then value.dup
+        when Array then value.map { |v| deep_copy(v) }
+        when Hash then value.to_h { |k, v| [k, deep_copy(v)] }
+        else value
+        end
+      end
+    end
+  end
+end

data/lib/fast/protowire/reader.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+require_relative "errors"
+require_relative "wire"
+
+module Fast
+  module Protowire
+    # A cursor over an encoded message. Reads the primitive wire values and
+    # skips what it is not asked to interpret.
+    class Reader
+      def initialize(buffer, position = 0, limit = buffer.bytesize)
+        @buffer = buffer
+        @position = position
+        @limit = limit
+      end
+
+      attr_reader :position
+
+      def eof?
+        @position >= @limit
+      end
+
+      # Returns [field number, wire type].
+      def read_tag
+        key = read_varint
+        [key >> 3, key & 0x7]
+      end
+
+      def read_varint
+        result = 0
+        shift = 0
+        loop do
+          raise DecodeError, "truncated varint" if @position >= @limit
+
+          byte = @buffer.getbyte(@position)
+          @position += 1
+          result |= (byte & 0x7f) << shift
+          return result if byte < 0x80
+
+          shift += 7
+          raise DecodeError, "varint too long" if shift > 63
+        end
+      end
+
+      def read_fixed32
+        read_bytes(4).unpack1("L<")
+      end
+
+      def read_fixed64
+        read_bytes(8).unpack1("Q<")
+      end
+
+      def read_bytes(length)
+        raise DecodeError, "truncated field" if @position + length > @limit
+
+        bytes = @buffer.byteslice(@position, length)
+        @position += length
+        bytes
+      end
+
+      def read_length_delimited
+        read_bytes(read_varint)
+      end
+
+      # A reader bounded to the next length-delimited value, for packed fields.
+      def read_packed
+        length = read_varint
+        raise DecodeError, "truncated packed field" if @position + length > @limit
+
+        reader = Reader.new(@buffer, @position, @position + length)
+        @position += length
+        reader
+      end
+
+      # Skips one value of +wire_type+ and returns its raw bytes, so unknown
+      # fields survive a decode/encode round trip.
+      def skip(wire_type)
+        start = @position
+        case wire_type
+        when Wire::VARINT then read_varint
+        when Wire::FIXED64 then read_bytes(8)
+        when Wire::LENGTH_DELIMITED then read_length_delimited
+        when Wire::START_GROUP then skip_group
+        when Wire::FIXED32 then read_bytes(4)
+        else raise DecodeError, "unknown wire type #{wire_type}"
+        end
+        @buffer.byteslice(start, @position - start)
+      end
+
+      private
+
+      def skip_group
+        loop do
+          number, wire_type = read_tag
+          return if wire_type == Wire::END_GROUP
+          raise DecodeError, "invalid group" if number.zero?
+
+          skip(wire_type)
+        end
+      end
+    end
+  end
+end

data/lib/fast/protowire/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Fast
+  module Protowire
+    VERSION = "0.1.0"
+  end
+end

data/lib/fast/protowire/wire.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+module Fast
+  module Protowire
+    # The wire format itself: tags, varints, zigzag, fixed-width values and
+    # length prefixes, appended to a binary String buffer. Everything above
+    # this module is bookkeeping about which field gets which of these.
+    module Wire
+      VARINT = 0
+      FIXED64 = 1
+      LENGTH_DELIMITED = 2
+      START_GROUP = 3
+      END_GROUP = 4
+      FIXED32 = 5
+
+      UINT64_MASK = (1 << 64) - 1
+      private_constant :UINT64_MASK
+
+      module_function
+
+      # The key for +number+ / +wire_type+ as frozen bytes.
+      def tag(number, wire_type)
+        varint((number << 3) | wire_type).freeze
+      end
+
+      def varint(value)
+        append_varint(String.new(capacity: 10), value)
+      end
+
+      # Base-128 little-endian; negative values are sign-extended to 64 bits
+      # exactly as int32/int64 fields require (ten bytes).
+      def append_varint(buffer, value)
+        value &= UINT64_MASK if value.negative?
+        while value > 0x7f
+          buffer << ((value & 0x7f) | 0x80)
+          value >>= 7
+        end
+        buffer << value
+      end
+
+      def zigzag32(value)
+        (value << 1) ^ (value >> 31)
+      end
+
+      def zigzag64(value)
+        (value << 1) ^ (value >> 63)
+      end
+
+      def unzigzag(value)
+        (value >> 1) ^ -(value & 1)
+      end
+
+      # A length-delimited field: +tag+ then the varint size of +bytes+ then
+      # +bytes+. Non-ASCII text is appended as its bytes so the binary buffer
+      # never trips Ruby's encoding compatibility check.
+      def append_length_delimited(buffer, tag, bytes)
+        buffer << tag
+        append_varint(buffer, bytes.bytesize)
+        buffer << (bytes.ascii_only? ? bytes : bytes.b)
+      end
+
+      # A length-delimited field whose payload the block writes into a fresh
+      # buffer. (Writing into the parent and inserting the size afterwards
+      # looks cheaper but String#insert costs O(size) on the parent.)
+      def append_length_delimited_from(buffer, tag)
+        payload = String.new
+        yield payload
+        append_length_delimited(buffer, tag, payload)
+      end
+    end
+  end
+end

data/lib/fast/protowire.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require_relative "protowire/version"
+require_relative "protowire/errors"
+require_relative "protowire/wire"
+require_relative "protowire/reader"
+require_relative "protowire/enum"
+require_relative "protowire/field"
+require_relative "protowire/message"
+
+module Fast
+  # Protocol Buffers wire format: declare messages, encode and decode bytes.
+  # No descriptors, reflection or JSON — the wire layer only.
+  module Protowire
+  end
+end

metadata

@@ -0,0 +1,53 @@
+--- !ruby/object:Gem::Specification
+name: fast-protowire
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Eric Jacobs
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- AGENTS.md
+- CHANGELOG.md
+- README.md
+- lib/fast/protowire.rb
+- lib/fast/protowire/enum.rb
+- lib/fast/protowire/errors.rb
+- lib/fast/protowire/field.rb
+- lib/fast/protowire/message.rb
+- lib/fast/protowire/reader.rb
+- lib/fast/protowire/version.rb
+- lib/fast/protowire/wire.rb
+homepage: https://github.com/jetpks/fast-protowire
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/jetpks/fast-protowire
+  source_code_uri: https://github.com/jetpks/fast-protowire
+  changelog_uri: https://github.com/jetpks/fast-protowire/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.10
+specification_version: 4
+summary: 'Protocol Buffers wire format for Ruby: declare messages, encode and decode
+  bytes, no runtime.'
+test_files: []