field_struct-avro-native 1.0.0.pre1-aarch64-linux

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 6a5c8c1562a029724d9c3d43b48eeff6159e8dbefdb6c991c73ac7f4a70f4b95
4
+ data.tar.gz: c3571349f1b9eb02c2f9e513e40ae37a5c378df72b8491c1adb7616fc8b6c6e0
5
+ SHA512:
6
+ metadata.gz: 16c4fcf65081a8919f84256b8fef6574de6a8bd51b666b34c09878990d4a944c02d4f4d3376ac5b5887842f422ac578fa863b29a3a0361b2320f0ddb5c981027
7
+ data.tar.gz: 2968de8060ab63e52ddb04b54c83d0b2ca4ea1642f43cd1df87a9208df23361e76ca44a1d08b5e6d99f1ba86e6017a1822cedc550743ad93cf29d1b1e1e69fd6
data/CHANGELOG.md ADDED
@@ -0,0 +1,5 @@
1
+ # Changelog
2
+
3
+ ## 1.0.0.pre1
4
+
5
+ - First release: an optional Rust encoder for `FieldStruct::Avro.encode`, loaded with `require 'field_struct/avro/native'`.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Adrian Madrid
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,122 @@
1
+ # field_struct-avro-native
2
+
3
+ An optional Rust encoder for [field_struct-avro](../field_struct-avro). It writes the same Avro bytes as the
4
+ pure-Ruby path, about 2.6x faster end to end on an Order-like struct.
5
+
6
+ ```ruby run
7
+ require 'field_struct/avro'
8
+
9
+ class Beacon < FieldStruct::Base
10
+ required :id, :integer
11
+ required :tower, :string
12
+ end
13
+
14
+ pure = FieldStruct::Avro.encode(Beacon.new(id: 7, tower: 'north'))
15
+
16
+ require 'field_struct/avro/native'
17
+
18
+ FieldStruct::Avro::Native.enabled? # => true
19
+ FieldStruct::Avro.encode(Beacon.new(id: 7, tower: 'north')) == pure # => true
20
+ ```
21
+
22
+ Without the `require`, nothing changes. field_struct-avro doesn't depend on this gem.
23
+
24
+ ## What it does
25
+
26
+ `FieldStruct::Avro.encode` builds a datum with `to_datum` and hands it to the avro gem's binary writer. That
27
+ writer is most of the cost: it validates the whole datum, then writes varints a byte at a time. This gem replaces
28
+ only that last step. It compiles each parsed writer schema once into an encoding plan, keeps it on the schema
29
+ object (so it follows the adapter's cache: a rebuilt or cleared entry gets a new schema and a new plan), and
30
+ walks the datum in Rust into one buffer.
31
+
32
+ The contract is byte-for-byte identical output. When the plan is not sure about a datum, it doesn't raise.
33
+ It hands the datum to the pure path, which then writes it or raises exactly the error field_struct-avro
34
+ would raise. It falls back for:
35
+
36
+ - a value of a class the plan doesn't expect, like an Integer in a `double` field or a DateTime in a `date` field
37
+ - a value out of range: an integer past `int`/`long`, a decimal past its precision or scale, a time past `long`
38
+ - a String in an encoding other than UTF-8 (7-bit US-ASCII and binary Strings are fine)
39
+ - a record datum that isn't exactly what `to_datum` builds (String keys, one per field, in field order)
40
+ - a union member it can't rule in or out the way the avro gem would
41
+ - schema shapes it doesn't cover: `fixed`, logical types other than `decimal` (on bytes), `date`,
42
+ `timestamp-millis/micros/nanos` and the identity ones like `uuid`
43
+
44
+ Two avro gem 1.12 quirks the plan copies by falling back: a decimal zero at scale 0 raises "Rounding
45
+ necessary", and a map with a key that is not valid UTF-8 raises inside a union.
46
+
47
+ ## Install
48
+
49
+ ```ruby
50
+ gem 'field_struct-avro'
51
+ gem 'field_struct-avro-native'
52
+ ```
53
+
54
+ Ruby 3.3 or newer. Precompiled gems aren't published yet, so installing builds the extension from source, and the
55
+ machine needs Rust (stable) and a C toolchain.
56
+
57
+ If the extension can't load, `require 'field_struct/avro/native'` raises `LoadError`. The per-datum fallback above
58
+ doesn't cover that case. To keep the native encoder optional, rescue it around the require:
59
+
60
+ ```ruby
61
+ begin
62
+ require 'field_struct/avro/native'
63
+ rescue LoadError
64
+ # field_struct-avro keeps encoding in pure Ruby
65
+ end
66
+ ```
67
+
68
+ ## Building
69
+
70
+ You need Rust (stable) and a C toolchain. From this directory:
71
+
72
+ ```sh
73
+ bundle install
74
+ bundle exec rake compile # builds lib/field_struct/avro/field_struct_avro_native.bundle (.so on Linux)
75
+ bundle exec rake # compile, specs, field_struct-avro's suite with native, rubocop, clippy, cargo test
76
+ ```
77
+
78
+ The Ruby-free kernels (zigzag varints, decimal bytes) live in their own crate, `ext/field_struct_avro_kernel`,
79
+ so `cargo test` runs without linking libruby.
80
+
81
+ ## Release builds
82
+
83
+ `bundle exec rake native gem` builds a platform gem for the machine you are on. The other platforms are built in
84
+ containers: `bundle exec rb-sys-dock --platform aarch64-linux --ruby-versions 3.3,3.4,4.0 --build` (needs Docker; the
85
+ gem lands in `pkg/`). The tag workflow, `field_struct-avro-native-release.yml`, does this for every platform plus the
86
+ source gem, smoke-tests each, and uploads them as artifacts. It never publishes.
87
+ `support/smoke_install.sh <gem>` installs a built gem into a throwaway `GEM_HOME` and runs an encode.
88
+
89
+ ## Proof
90
+
91
+ - `spec/field_struct/avro/native/datum_fuzz_spec.rb` - 9,000 random schema/datum pairs per run, edges and wrong
92
+ shapes included: whenever the plan writes, the pure path writes the same bytes, and it must take most datums
93
+ and leave both good and bad ones to the pure path
94
+ - `struct_differential_spec.rb` - 3,000 random structs over every field type the adapter maps, at each timestamp
95
+ precision: same bytes, and they decode and re-encode to the same bytes
96
+ - `fallback_spec.rb` - each error the adapter raises comes out the same with native on: a time too precise for
97
+ its timestamp, an invalid struct, a decimal past its precision or scale, an integer past 64 bits
98
+ - `rake upstream` - field_struct-avro's whole suite with `FIELD_STRUCT_AVRO_NATIVE=1`
99
+
100
+ ## Numbers
101
+
102
+ Ruby 4.0.7 with YJIT on an M-series Mac, `adapters/field_struct-avro/benchmarks/perf.rb` with and without
103
+ `-rfield_struct/avro/native`, two runs each:
104
+
105
+ | workload | pure | native | speedup |
106
+ |---|---:|---:|---:|
107
+ | encode, 1 order | 23,476 i/s (42.6 us) | 61,468 i/s (16.3 us) | 2.6x |
108
+ | encode, 1000 orders | 23.2 i/s (43.2 us/order) | 61.6 i/s (16.2 us/order) | 2.7x |
109
+ | allocations per order | 417 | 155 | |
110
+
111
+ The write step alone goes from 26.4 us to 1.9 us. What is left is `to_datum` (12.6 us), so the next step, if
112
+ we want one, is a plan that reads the struct directly instead of the datum Hash.
113
+
114
+ ## Not yet
115
+
116
+ - Published precompiled platform gems. The release workflow builds them (x86_64-linux, aarch64-linux, arm64-darwin)
117
+ but nothing is published yet, so installing compiles from source.
118
+ - `decode` stays pure Ruby. The avro gem's decimal reader is the hot spot there, ceiling about 1.6x.
119
+
120
+ ## Docs
121
+
122
+ The guide on the docs site, with every example run in CI: https://gliders-rb.github.io/adapters/avro-native/
@@ -0,0 +1,124 @@
1
+ # frozen_string_literal: true
2
+
3
+ module FieldStruct
4
+ module Avro
5
+ module Native
6
+ # Turns a parsed +Avro::Schema+ into the flat node list +Plan.new+ takes, or +nil+ for a schema with a shape the
7
+ # native encoder does not cover (then every encode with it runs the pure path).
8
+ #
9
+ # Each node is an Array starting with a tag; child nodes are referenced by index, so a recursive record is a
10
+ # node that points back at itself.
11
+ class Compiler
12
+ # @return [Hash{Symbol => Integer}] the tag number of each node kind
13
+ TAGS = {
14
+ null: 0, boolean: 1, int: 2, long: 3, float: 4, double: 5, bytes: 6, string: 7,
15
+ decimal: 8, date: 9, timestamp: 10, enum: 11, array: 12, map: 13, record: 14, union: 15
16
+ }.freeze
17
+
18
+ # Ticks per second of each timestamp logical type.
19
+ #
20
+ # @return [Hash{String => Integer}]
21
+ TICKS = { 'timestamp-millis' => 1_000, 'timestamp-micros' => 1_000_000, 'timestamp-nanos' => 1_000_000_000 }.freeze
22
+
23
+ # The logical types written with a conversion; any other logical type must use the identity adapter.
24
+ #
25
+ # @return [Hash{String => Array(Symbol, String)}] base type and adapter name
26
+ CONVERTED = {
27
+ 'decimal' => [:bytes, 'BytesDecimal'], 'date' => [:int, 'IntDate'],
28
+ 'timestamp-millis' => [:long, 'TimestampMillis'], 'timestamp-micros' => [:long, 'TimestampMicros'],
29
+ 'timestamp-nanos' => [:long, 'TimestampNanos']
30
+ }.freeze
31
+
32
+ # @return [Array<Symbol>]
33
+ PRIMITIVES = %i[null boolean int long float double bytes string].freeze
34
+ # @return [Array<Symbol>]
35
+ COMPOSITES = %i[record enum array map union].freeze
36
+
37
+ # Raised inside the walk for a shape with no native encoding.
38
+ class Unsupported < StandardError; end
39
+
40
+ # @param schema [::Object] a parsed +Avro::Schema+
41
+ # @return [Array(::Array<::Array<::Object>>, ::Integer), nil] the nodes and the root index, or nil when unsupported
42
+ def self.call(schema)
43
+ compiler = new
44
+ root = compiler.index(schema)
45
+ [compiler.nodes, root]
46
+ rescue Unsupported
47
+ nil
48
+ end
49
+
50
+ # @return [::Array<::Array<::Object>>]
51
+ attr_reader :nodes
52
+
53
+ def initialize
54
+ @nodes = []
55
+ @seen = {}.compare_by_identity
56
+ end
57
+
58
+ # @param schema [::Object] a parsed +Avro::Schema+
59
+ # @return [::Integer] the node index of the schema, compiling it on first sight
60
+ def index(schema)
61
+ found = @seen[schema]
62
+ return found if found
63
+
64
+ at = @nodes.size
65
+ @seen[schema] = at
66
+ @nodes << nil
67
+ @nodes[at] = node(schema)
68
+ at
69
+ end
70
+
71
+ private
72
+
73
+ def node(schema)
74
+ type = schema.type_sym
75
+ return primitive(schema, type) if PRIMITIVES.include?(type)
76
+ raise Unsupported unless COMPOSITES.include?(type)
77
+
78
+ [TAGS.fetch(type), send(type, schema)]
79
+ end
80
+
81
+ def record(schema) = schema.fields.map { |field| [field.name.to_s, index(field.type)] }
82
+ def enum(schema) = schema.symbols.map(&:to_s)
83
+ def array(schema) = index(schema.items)
84
+ def map(schema) = index(schema.values)
85
+ def union(schema) = schema.schemas.map { |member| index(member) }
86
+
87
+ def primitive(schema, type)
88
+ logical = schema.logical_type
89
+ return [TAGS.fetch(type)] if logical.nil? || identity?(schema)
90
+
91
+ base, adapter = CONVERTED[logical]
92
+ raise Unsupported unless base == type && adapter?(schema.type_adapter, ::Avro::LogicalTypes.const_get(adapter))
93
+
94
+ converted(schema, logical)
95
+ end
96
+
97
+ def converted(schema, logical)
98
+ case logical
99
+ when 'decimal' then decimal(schema)
100
+ when 'date' then [TAGS[:date]]
101
+ else [TAGS[:timestamp], TICKS.fetch(logical)]
102
+ end
103
+ end
104
+
105
+ def decimal(schema)
106
+ precision = schema.precision
107
+ scale = schema.scale || 0
108
+ raise Unsupported unless precision.is_a?(::Integer) && precision.positive? && scale.is_a?(::Integer) && scale >= 0
109
+
110
+ [TAGS[:decimal], precision, scale]
111
+ end
112
+
113
+ # The avro gem keeps some adapters as modules and builds others per schema (decimal).
114
+ def adapter?(adapter, expected)
115
+ adapter.equal?(expected) || adapter.instance_of?(expected)
116
+ end
117
+
118
+ def identity?(schema)
119
+ schema.type_adapter.equal?(::Avro::LogicalTypes::Identity)
120
+ end
121
+ end
122
+ end
123
+ end
124
+ end
@@ -0,0 +1,12 @@
1
+ # frozen_string_literal: true
2
+
3
+ module FieldStruct
4
+ module Avro
5
+ module Native
6
+ # The gem version.
7
+ #
8
+ # @return [::String]
9
+ VERSION = '1.0.0.pre1'
10
+ end
11
+ end
12
+ end
@@ -0,0 +1,76 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'bigdecimal'
4
+ require 'date'
5
+ require 'field_struct/avro'
6
+ require_relative 'native/version'
7
+ require_relative 'native/compiler'
8
+
9
+ # A precompiled platform gem keeps one binary per Ruby minor version in a folder named for it; a source build
10
+ # (and `rake compile`) puts the single binary next to this file.
11
+ begin
12
+ require_relative "#{RUBY_VERSION[/\A\d+\.\d+/]}/field_struct_avro_native"
13
+ rescue LoadError
14
+ require_relative 'field_struct_avro_native'
15
+ end
16
+
17
+ # The field_struct namespace.
18
+ module FieldStruct
19
+ # Apache Avro for field_struct classes.
20
+ module Avro
21
+ # An optional Rust encoder behind +FieldStruct::Avro.encode+. Requiring this file swaps the binary write: the
22
+ # datum +to_datum+ built is written by an encoding plan compiled once per parsed writer schema. A datum the plan is
23
+ # not sure about (a value of an unexpected class, out of range, needing rounding) is written by the pure-Ruby path
24
+ # instead, so the bytes and every error stay what the pure path gives.
25
+ module Native
26
+ # The instance variable a parsed schema keeps its compiled plan in, so the plan lives and dies with the
27
+ # schema the adapter's cache holds (a rebuilt or cleared entry gets a new schema and a new plan).
28
+ PLAN_IVAR = :@field_struct_avro_native_plan
29
+ private_constant :PLAN_IVAR
30
+
31
+ class << self
32
+ # @return [Boolean] whether +FieldStruct::Avro.encode+ goes through the native encoder
33
+ def enabled?
34
+ FieldStruct::Avro.singleton_class.ancestors.include?(Write)
35
+ end
36
+
37
+ # The Avro binary of a datum, or nil when the native encoder leaves it to the pure path.
38
+ #
39
+ # @param schema [::Object] a parsed +Avro::Schema+
40
+ # @param datum [::Object] what +FieldStruct::Avro.to_datum+ returns for that schema
41
+ # @return [::String, nil] binary bytes, or nil
42
+ def encode(schema, datum)
43
+ plan = plan_for(schema)
44
+ plan&.encode(datum)
45
+ end
46
+
47
+ private
48
+
49
+ def plan_for(schema)
50
+ plan = schema.instance_variable_get(PLAN_IVAR)
51
+ return plan || nil unless plan.nil?
52
+
53
+ plan = compile(schema)
54
+ schema.instance_variable_set(PLAN_IVAR, plan || false) unless schema.frozen?
55
+ plan
56
+ end
57
+
58
+ def compile(schema)
59
+ nodes, root = Compiler.call(schema)
60
+ nodes && Plan.new(nodes, root)
61
+ end
62
+ end
63
+
64
+ # Prepended to +FieldStruct::Avro+'s singleton class: the binary write tries the plan first.
65
+ module Write
66
+ private
67
+
68
+ def write(parsed, datum, struct)
69
+ Native.encode(parsed, datum) || super
70
+ end
71
+ end
72
+
73
+ FieldStruct::Avro.singleton_class.prepend(Write)
74
+ end
75
+ end
76
+ end
@@ -0,0 +1,61 @@
1
+ # The field_struct namespace.
2
+ module FieldStruct
3
+ # Apache Avro for field_struct classes.
4
+ module Avro
5
+ # An optional Rust encoder behind +FieldStruct::Avro.encode+. Requiring this file swaps the binary write: the
6
+ # datum +to_datum+ built is written by an encoding plan compiled once per parsed writer schema. A datum the plan is
7
+ # not sure about (a value of an unexpected class, out of range, needing rounding) is written by the pure-Ruby path
8
+ # instead, so the bytes and every error stay what the pure path gives.
9
+ module Native
10
+ include FieldStruct::Avro::Native::Write
11
+ VERSION: ::String
12
+
13
+ # _@return_ — whether +FieldStruct::Avro.encode+ goes through the native encoder
14
+ def self.enabled?: () -> bool
15
+
16
+ # The Avro binary of a datum, or nil when the native encoder leaves it to the pure path.
17
+ #
18
+ # _@param_ `schema` — a parsed +Avro::Schema+
19
+ #
20
+ # _@param_ `datum` — what +FieldStruct::Avro.to_datum+ returns for that schema
21
+ #
22
+ # _@return_ — binary bytes, or nil
23
+ def self.encode: (::Object schema, ::Object datum) -> ::String?
24
+
25
+ # Prepended to +FieldStruct::Avro+'s singleton class: the binary write tries the plan first.
26
+ module Write
27
+ end
28
+
29
+ # Turns a parsed +Avro::Schema+ into the flat node list +Plan.new+ takes, or +nil+ for a schema with a shape the
30
+ # native encoder does not cover (then every encode with it runs the pure path).
31
+ #
32
+ # Each node is an Array starting with a tag; child nodes are referenced by index, so a recursive record is a
33
+ # node that points back at itself.
34
+ class Compiler
35
+ TAGS: ::Hash[Symbol, Integer]
36
+ TICKS: ::Hash[String, Integer]
37
+ CONVERTED: ::Hash[String, [Symbol, String]]
38
+ PRIMITIVES: ::Array[Symbol]
39
+ COMPOSITES: ::Array[Symbol]
40
+
41
+ # _@param_ `schema` — a parsed +Avro::Schema+
42
+ #
43
+ # _@return_ — the nodes and the root index, or nil when unsupported
44
+ def self.call: (::Object schema) -> [::Array[::Array[::Object]], ::Integer]?
45
+
46
+ def initialize: () -> void
47
+
48
+ # _@param_ `schema` — a parsed +Avro::Schema+
49
+ #
50
+ # _@return_ — the node index of the schema, compiling it on first sight
51
+ def index: (::Object schema) -> ::Integer
52
+
53
+ attr_reader nodes: ::Array[::Array[::Object]]
54
+
55
+ # Raised inside the walk for a shape with no native encoding.
56
+ class Unsupported < StandardError
57
+ end
58
+ end
59
+ end
60
+ end
61
+ end
metadata ADDED
@@ -0,0 +1,82 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: field_struct-avro-native
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0.pre1
5
+ platform: aarch64-linux
6
+ authors:
7
+ - Adrian Madrid
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2026-11-17 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: field_struct-avro
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: 1.0.0.pre1
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '2'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: 1.0.0.pre1
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '2'
33
+ description: Compiles each writer schema into an encoding plan and writes field_struct-avro's
34
+ datums to Avro binary in Rust. Anything it is not sure about goes to the pure-Ruby
35
+ path, so output and errors stay the same.
36
+ email:
37
+ - adrian.madrid@timeworxs.com
38
+ executables: []
39
+ extensions: []
40
+ extra_rdoc_files: []
41
+ files:
42
+ - CHANGELOG.md
43
+ - LICENSE.txt
44
+ - README.md
45
+ - lib/field_struct/avro/3.3/field_struct_avro_native.so
46
+ - lib/field_struct/avro/3.4/field_struct_avro_native.so
47
+ - lib/field_struct/avro/4.0/field_struct_avro_native.so
48
+ - lib/field_struct/avro/native.rb
49
+ - lib/field_struct/avro/native/compiler.rb
50
+ - lib/field_struct/avro/native/version.rb
51
+ - sig/field_struct-avro-native.rbs
52
+ homepage: https://github.com/gliders-rb/gems
53
+ licenses:
54
+ - MIT
55
+ metadata:
56
+ source_code_uri: https://github.com/gliders-rb/gems/tree/main/adapters/field_struct-avro-native
57
+ changelog_uri: https://github.com/gliders-rb/gems/blob/main/adapters/field_struct-avro-native/CHANGELOG.md
58
+ rubygems_mfa_required: 'true'
59
+ post_install_message:
60
+ rdoc_options: []
61
+ require_paths:
62
+ - lib
63
+ required_ruby_version: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - ">="
66
+ - !ruby/object:Gem::Version
67
+ version: '3.3'
68
+ - - "<"
69
+ - !ruby/object:Gem::Version
70
+ version: 4.1.dev
71
+ required_rubygems_version: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ requirements: []
77
+ rubygems_version: 3.5.23
78
+ signing_key:
79
+ specification_version: 4
80
+ summary: 'An optional Rust encoder for field_struct-avro: the same Avro bytes, written
81
+ faster.'
82
+ test_files: []