pgoutput-parser 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 58a2b938bf7d260f461f769757d66f2a0ef1a8127b85ba415091fabfde9aee19
-  data.tar.gz: efacd6a3e298aa165066170a4a4c826a35a48d75f10d2a8a281ba90e709ac1e6
+  metadata.gz: 5822df660843e811e03208f6004f7700821953d9c602d70dd3d059f7fb9b82f7
+  data.tar.gz: 0231a9f0d3fcc295976b534bfa7350e08f0156c96a1daebb5528e095df2a63db
 SHA512:
-  metadata.gz: 83ecef417fc9b40a21c4c4440dfc5be4d0de4a03cb7c6b4439efb17d176888bf0080d803d61d1ad334cc440ae661287073f242b976aac3fa85991115fe558fcf
-  data.tar.gz: 1e31f54d1c0ea212aeb1e78a01f6c6a8dba6a982717d8bffb75ec87d61e32f89001f4e828c7ad2be643bcabcfb347146e66170750eedce66560b5a612c227fc7
+  metadata.gz: 6221a4d4a884f2877fdb9fe0086ce37b955137f204de2c207892de697be28a4d67f4290e12b027531e12ee9ae4277233124989d6d0073a4d9a7584fc587eac78
+  data.tar.gz: c868e1446cb07ee253f0ece9a5f8fc104a9ca5bf9e3181e169a12d4affcd03e58a1ca240bf86107a35b388b007b0911985ebba214a6d667b2a886af5f8c90995

data/CHANGELOG.md

@@ -7,9 +7,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+---
+
+## [0.2.0] - 2026-06-06
+
 ### Added
 
-* Placeholder for future development.
+* Added `Pgoutput::TupleArityError` for relation/tuple column-count mismatches.
+* Added a dependency-free parser throughput benchmark for `BinaryParser` and
+  cached `RelationTracker` DML processing.
+* Added parser support for non-streaming Message (`M`), Origin (`O`), Type (`Y`),
+  and Truncate (`T`) messages.
+* Added benchmark tuning controls for iterations, warmup, Ractor count, and
+  scenario selection.
+* Added benchmark relation-cache tuning for comparing the default Hash cache
+  with optional `Ratomic::Map`.
+* Added generated documentation pages for the glossary and detailed
+  `RelationTracker` usage.
+* Added optional `Ratomic::Map` relation-cache coverage and benchmark smoke
+  paths.
+* Added grouped unit, integration, and behavior test layout aligned with the CDC
+  component test shape.
+* Added explicit BinaryParser coverage for unterminated C strings and negative
+  byte lengths.
+
+### Changed
+
+* `Pgoutput::RelationTracker` now rejects DML tuples whose value count differs from
+  the cached `Relation` column count instead of silently assigning incomplete
+  column metadata.
+* Raised the minimum supported Ruby version to Ruby 4.0 to align with the CDC
+  ecosystem's Ractor-first parser/runtime direction.
+* `Pgoutput::RelationTracker` now accepts an injectable `relation_cache:` object
+  compatible with Hash-style `#[]=` and `#fetch`.
+* GitHub Actions now validates RBS signatures and runs a small benchmark smoke
+  test against Hash and Ratomic relation-cache paths.
 
 ---
 
@@ -112,6 +144,6 @@ A future companion project (`pgoutput-decoder`) may provide PostgreSQL type deco
 
 ---
 
-[Unreleased]: https://github.com/your-github-username/pgoutput-parser/compare/v0.1.0...HEAD
-[0.1.0]: https://github.com/your-github-username/pgoutput-parser/releases/tag/v0.1.0
-
+[Unreleased]: https://github.com/kanutocd/pgoutput-parser/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/kanutocd/pgoutput-parser/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/kanutocd/pgoutput-parser/releases/tag/v0.1.0

data/README.md

@@ -10,7 +10,7 @@ It intentionally does **not** convert PostgreSQL values into application-specifi
 
 ## Requirements
 
-- Ruby 3.4+
+- Ruby 4+
 - PostgreSQL 10+
 
 ---
@@ -18,7 +18,7 @@ It intentionally does **not** convert PostgreSQL values into application-specifi
 ## Features
 
 - Pure Ruby implementation
-- Ruby 3.4+
+- Ruby 4+
 - Ractor-safe parsed messages
 - Immutable protocol message objects
 - PostgreSQL logical replication protocol support
@@ -28,6 +28,9 @@ It intentionally does **not** convert PostgreSQL values into application-specifi
 - YARD documentation included
 - No runtime dependencies
 
+The generated documentation also includes a project glossary:
+[docs/glossary.md](docs/glossary.md).
+
 ---
 
 ## Why Another pgoutput Library?
@@ -46,13 +49,17 @@ This keeps the parser small, predictable, dependency-free, and faithful to Postg
 
 ## Supported MVP Scope
 
-Supports the core pgoutput row-change replication messages:
+Supports the core non-streaming pgoutput logical replication messages:
 
 - Begin (`B`)
+- Message (`M`)
+- Origin (`O`)
 - Relation (`R`)
+- Type (`Y`)
 - Insert (`I`)
 - Update (`U`)
 - Delete (`D`)
+- Truncate (`T`)
 - Commit (`C`)
 
 The currently supported message formats are stable across PostgreSQL 10 through PostgreSQL 18.
@@ -70,10 +77,6 @@ TupleData supports all base column markers:
 
 Future releases may add support for:
 
-- Message (`M`)
-- Truncate (`T`)
-- Origin (`O`)
-- Type (`Y`)
 - Stream Start (`S`)
 - Stream Stop (`E`)
 - Stream Commit (`c`)
@@ -275,6 +278,13 @@ delete.old_tuple
 
 `RelationTracker` keeps a local relation cache so tuple values can be associated with PostgreSQL column OIDs defined by preceding Relation (`R`) messages.
 
+The tracker accepts an optional `relation_cache:` argument. The default is a
+plain Hash, but callers can inject `Ratomic::Map` for a Ractor-safe cache in
+experimental or parallel setups.
+
+For a deeper guide, including stream-order behavior, tuple arity validation, and
+`Ratomic::Map` usage, see [docs/relation_tracker.md](docs/relation_tracker.md).
+
 No type conversion is performed.
 
 Only protocol metadata is attached.
@@ -290,6 +300,10 @@ message.new_tuple.map(&:oid)
 
 The relation tracker itself is stateful and maintains relation metadata encountered in the replication stream.
 
+If a DML tuple's value count does not match the cached relation column count, `RelationTracker` raises
+`Pgoutput::TupleArityError`. This keeps malformed payloads or mismatched stream state from being silently
+annotated with incomplete column metadata.
+
 ---
 
 ## Ractor Safety
@@ -392,12 +406,16 @@ The tracker maintains relation metadata discovered during the stream and therefo
 
 ```ruby
 Pgoutput::Messages::Begin
+Pgoutput::Messages::Message
+Pgoutput::Messages::Origin
 Pgoutput::Messages::Relation
+Pgoutput::Messages::Type
 Pgoutput::Messages::Column
 Pgoutput::Messages::TupleValue
 Pgoutput::Messages::Insert
 Pgoutput::Messages::Update
 Pgoutput::Messages::Delete
+Pgoutput::Messages::Truncate
 Pgoutput::Messages::Commit
 ```
 
@@ -435,6 +453,50 @@ COVERAGE=true bundle exec rake test
 
 ---
 
+## Benchmarking
+
+Run the parser throughput benchmark:
+
+```bash
+ruby benchmark/parser_throughput.rb
+```
+
+The benchmark reports single-process parser throughput, relation-tracker throughput, and Ractor-parallel throughput. It is intended to show both the single-thread baseline and the Ruby 4 Ractor path this parser is designed to support. Relation-tracker scenarios can also compare the default Hash relation cache with an optional `Ratomic::Map` cache.
+
+Tune the run with environment variables:
+
+| Variable | Default | Description |
+| -------- | ------- | ----------- |
+| `PGOUTPUT_BENCH_ITERATIONS` | `100000` | Iterations per selected scenario. |
+| `PGOUTPUT_BENCH_WARMUP` | `1000` | Warmup iterations before timing. |
+| `PGOUTPUT_BENCH_RACTORS` | `2` or CPU count, whichever is lower | Number of Ractor workers for Ractor scenarios. |
+| `PGOUTPUT_BENCH_SCENARIOS` | `all` | Comma-separated scenarios: `binary`, `tracker_dml`, `tracker_mixed`, `ractor_binary`, `ractor_tracker`, or `all`. |
+| `PGOUTPUT_BENCH_RELATION_CACHE` | `hash` | Comma-separated relation-cache backends for tracker scenarios: `hash`, `ratomic`, or `all`. |
+
+Examples:
+
+```bash
+PGOUTPUT_BENCH_ITERATIONS=10000 ruby benchmark/parser_throughput.rb
+PGOUTPUT_BENCH_SCENARIOS=binary,tracker_mixed ruby benchmark/parser_throughput.rb
+PGOUTPUT_BENCH_RACTORS=4 PGOUTPUT_BENCH_SCENARIOS=ractor_binary,ractor_tracker ruby benchmark/parser_throughput.rb
+PGOUTPUT_BENCH_RELATION_CACHE=all PGOUTPUT_BENCH_SCENARIOS=tracker_mixed,ractor_tracker ruby benchmark/parser_throughput.rb
+```
+
+Sample Ruby 4 output:
+
+```text
+pgoutput-parser throughput
+iterations=1000 warmup=10 ractors=2 scenarios=tracker_mixed,ractor_tracker relation_cache=hash,ratomic ruby=4.0.5
+RelationTracker hash               7000 messages in   0.163s        42891 msg/s
+RelationTracker ratomic            7000 messages in   0.131s        53579 msg/s
+Ractor RelationTracker hash       14000 messages in   0.197s        71097 msg/s
+Ractor RelationTracker ratomic      14000 messages in   0.146s        96190 msg/s
+```
+
+Interpret the Ractor rows as aggregate throughput across workers. They are not a replacement for the single-process rows; they demonstrate the parser's shareable-message design under parallel execution.
+
+---
+
 ## Development
 
 Generate YARD documentation:

data/docs/glossary.md

@@ -0,0 +1,145 @@
+# Glossary
+
+This glossary defines terms as they are used by `pgoutput-parser`. It focuses on
+the protocol parsing layer and avoids application-level decoding concepts that
+belong to higher CDC components.
+
+## Binary Value
+
+A tuple value sent by PostgreSQL with the `b` TupleData marker. The parser
+preserves the bytes exactly as received and does not interpret the PostgreSQL
+binary format.
+
+## BinaryParser
+
+The stateless entry point for parsing one pgoutput message payload. It decodes
+the wire-format tag and fields into an immutable `Pgoutput::Messages` object.
+
+## Column Flag
+
+The per-column flag byte in a Relation (`R`) message. PostgreSQL uses flag `1`
+to identify replica identity key columns.
+
+## Commit LSN
+
+The PostgreSQL log sequence number associated with a transaction commit. The
+parser exposes it as protocol metadata and does not use it for ordering logic.
+
+## CopyData Payload
+
+The PostgreSQL replication protocol frame body that contains one pgoutput
+message. This gem expects callers to provide that payload; it does not manage
+the PostgreSQL connection or replication stream.
+
+## DML Message
+
+A data manipulation message emitted by pgoutput for row or table changes. In
+this parser, DML messages are Insert (`I`), Update (`U`), Delete (`D`), and
+Truncate (`T`).
+
+## Immutable Message
+
+A parsed protocol object that has been made shareable with `Ractor`. Parsed
+messages can be passed across Ractors without sharing mutable parser state.
+
+## LSN
+
+Log sequence number. PostgreSQL uses LSN values to identify positions in the
+write-ahead log. This parser keeps LSN values as integers and does not convert
+them into PostgreSQL's textual `X/Y` notation.
+
+## Message Tag
+
+The first byte of a pgoutput payload. It identifies the message type, such as
+`R` for Relation, `I` for Insert, or `C` for Commit.
+
+## OID
+
+Object identifier. In this gem, OIDs are most commonly relation IDs and
+PostgreSQL type IDs exposed by Relation (`R`) and Type (`Y`) messages.
+
+## pgoutput
+
+PostgreSQL's built-in logical replication output plugin. It serializes
+transaction metadata, relation metadata, and row changes into a binary protocol.
+
+## Relation
+
+PostgreSQL table metadata sent by a Relation (`R`) message. It includes the
+relation ID, schema name, table name, replica identity setting, and columns.
+
+## RelationTracker
+
+The stream-order parser wrapper that remembers Relation (`R`) messages and uses
+them to annotate later DML tuple values with PostgreSQL type OIDs.
+
+## Replica Identity
+
+PostgreSQL metadata that determines what old-row data is available for Update
+and Delete messages. The parser exposes the replica identity byte from Relation
+messages and preserves old-key or old-full tuples when PostgreSQL sends them.
+
+## Ractor-Safe
+
+Safe to pass between Ruby Ractors. Parsed messages are Ractor-safe; parser and
+tracker instances remain mutable and should be scoped to one owner unless the
+caller supplies an explicitly Ractor-safe relation cache.
+
+## Ratomic
+
+An optional Ruby library that provides Ractor-oriented concurrent data
+structures. `pgoutput-parser` does not require Ratomic at runtime, but benchmark
+and development code can use it to evaluate Ractor-safe relation cache behavior.
+
+## Ratomic::Map
+
+A Ractor-safe map implementation from Ratomic. `RelationTracker` can use
+`Ratomic::Map` as its `relation_cache:` when callers need relation metadata in a
+cache that can be shared across Ractor-oriented execution designs.
+
+## Raw Tuple Value
+
+The uninterpreted bytes for a tuple column value. Text values and binary values
+are both kept as raw strings; NULL and unchanged TOAST markers have no raw
+payload.
+
+## Text Value
+
+A tuple value sent by PostgreSQL with the `t` TupleData marker. The parser
+preserves the text bytes and leaves type conversion to a decoder layer.
+
+## TOAST
+
+PostgreSQL's storage mechanism for large column values. In pgoutput TupleData,
+the `u` marker means an unchanged TOAST value was not resent.
+
+## Truncate
+
+A pgoutput DML message that reports table truncation. It contains relation IDs
+and option bits such as CASCADE and RESTART IDENTITY.
+
+## Tuple Arity
+
+The number of values in tuple data. `RelationTracker` validates DML tuple arity
+against cached Relation column metadata before annotating type OIDs.
+
+## TupleData
+
+The pgoutput structure that carries row values for Insert, Update, and Delete
+messages. Each value is marked as NULL, unchanged TOAST, text, or binary.
+
+## Type Decoding
+
+Conversion from PostgreSQL raw tuple bytes into application-level Ruby values.
+This gem intentionally does not perform type decoding; that responsibility
+belongs to a higher-level decoder component.
+
+## Type Modifier
+
+PostgreSQL column type metadata carried in Relation messages. For example,
+typmods can encode precision or length constraints for some PostgreSQL types.
+
+## WAL
+
+Write-ahead log. PostgreSQL logical replication streams changes derived from WAL
+through output plugins such as pgoutput.

data/docs/index.md

@@ -0,0 +1,239 @@
+# pgoutput-parser
+
+A high-performance, Ractor-safe PostgreSQL `pgoutput` logical replication protocol parser written in pure Ruby.
+
+`pgoutput-parser` parses PostgreSQL logical replication `CopyData` payloads into immutable protocol message objects.
+
+It focuses exclusively on PostgreSQL's `pgoutput` wire format:
+
+* Transaction boundaries
+* Relation metadata
+* DML message structure
+* Tuple payload markers
+* Raw tuple values
+
+It intentionally does **not** decode PostgreSQL values into application-level Ruby objects.
+
+That responsibility belongs to higher layers such as:
+
+```text
+pgoutput-decoder
+```
+
+---
+
+# Architecture
+
+```text
+PostgreSQL WAL
+       |
+       v
+CopyData payload
+       |
+       v
+pgoutput-parser
+       |
+       v
+Immutable protocol messages
+```
+
+The parser is the protocol layer of the CDC ecosystem.
+
+```text
+PostgreSQL
+      |
+      v
+pgoutput-parser
+      |
+      v
+pgoutput-decoder
+      |
+      v
+cdc-core
+```
+
+---
+
+# Quick Start
+
+```ruby
+require "pgoutput"
+
+stream = Pgoutput::RelationTracker.new
+
+stream.process(relation_payload)
+
+insert = stream.process(insert_payload)
+
+insert.relation_id
+# => 42
+
+insert.tuple.first.raw
+# => "7"
+
+insert.tuple.first.oid
+# => 23
+```
+
+---
+
+# Core Concepts
+
+## BinaryParser
+
+Parses a single `pgoutput` payload.
+
+```ruby
+message = Pgoutput::BinaryParser
+  .new(payload)
+  .parse
+```
+
+Use this when stream state is not required.
+
+---
+
+## RelationTracker
+
+Tracks PostgreSQL relation metadata across a replication stream.
+
+```ruby
+stream = Pgoutput::RelationTracker.new
+
+stream.process(relation_payload)
+
+message = stream.process(insert_payload)
+```
+
+This allows tuple values to be associated with PostgreSQL column OIDs.
+
+If tuple data does not match the cached relation column count, `RelationTracker`
+raises `Pgoutput::TupleArityError`.
+
+---
+
+# Supported Messages
+
+Current MVP support:
+
+* Begin (`B`)
+* Message (`M`)
+* Origin (`O`)
+* Relation (`R`)
+* Type (`Y`)
+* Insert (`I`)
+* Update (`U`)
+* Delete (`D`)
+* Truncate (`T`)
+* Commit (`C`)
+
+Supported across PostgreSQL 10–18.
+
+---
+
+# Tuple Values
+
+Tuple values are preserved exactly as PostgreSQL sends them.
+
+```ruby
+value.raw
+# => "2026-05-31 12:34:56+00"
+```
+
+No application-level decoding occurs.
+
+---
+
+# Binary Values
+
+Binary tuple values are preserved exactly as received.
+
+```ruby
+value.raw
+# => "\x00\x00\x00\x07".b
+```
+
+The parser does not interpret binary payloads.
+
+---
+
+# Ractor Safety
+
+All parsed protocol messages are immutable and shareable.
+
+```ruby
+message = stream.process(update_payload)
+
+Ractor.shareable?(message)
+# => true
+```
+
+Passing messages across Ractors:
+
+```ruby
+message = stream.process(update_payload)
+
+result = Ractor.new(message) do |update|
+  update.new_tuple.map(&:raw)
+end.take
+```
+
+---
+
+# Non-Goals
+
+`pgoutput-parser` intentionally does not:
+
+* Open replication connections
+* Manage replication slots
+* Track WAL positions
+* Reconnect to PostgreSQL
+* Decode PostgreSQL values
+* Build CDC pipelines
+* Integrate with ActiveRecord
+
+Its sole responsibility is protocol parsing.
+
+---
+
+# Public API
+
+See the generated API documentation for:
+
+* `Pgoutput::BinaryParser`
+* `Pgoutput::RelationTracker`
+* `Pgoutput::Messages::*`
+
+---
+
+# Development
+
+Generate documentation:
+
+```bash
+bundle exec yard doc
+```
+
+Run tests:
+
+```bash
+bundle exec rake test
+```
+
+Run coverage:
+
+```bash
+COVERAGE=true bundle exec rake test
+```
+
+Run Steep:
+
+```bash
+bundle exec steep check
+```
+
+---
+
+# License
+
+MIT