oxygene 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 16f8b923311bdcd734666a03e4beff53a786b13a131a29e2ae834996d6c8545e
-  data.tar.gz: c3c3ebc73bc19bdc937425757a1563c181e2790d6e3a87a76d086675540ed68c
+  metadata.gz: d85155d1ce53a32be48e557bd81995deea20b096f0d5ac7797aad798f6527680
+  data.tar.gz: 70d1b127ab875cc3a7d2f04b922aa48c950cb2db33b46240e35a29080edc5fc8
 SHA512:
-  metadata.gz: 30e917d087c398a27af26984842f5a71901093599310244f12794614ec1f5326614a2465021d6d2936c633ca9429246be493fa21e13ef4464054e5b02f7096af
-  data.tar.gz: 58c550605cdaf83a38aeb0a49c67fa4badafeed1ac5f2991b969179e609db485e05f17aa21cc4c42e092a06ec44f791c7fa02e3bcd6225602d7b7fbf69086941
+  metadata.gz: ec92728d10052f07d4fced7e9aa973bb14f655f67fb1f4c1684535b7d4e2dcaac2fda995fa64243253c1ef56e0ee66eb33d1ba66b31fb76c37437beaacf0f85e
+  data.tar.gz: 133a216c0b0016c18d269355eb5336fcaa754cb30b8fa648786644648f06820ccec4e9608d00f8de57229ea3d493bd10de4bc06368ac901a71c674605d841159

data/AGENTS.md

@@ -0,0 +1,9 @@
+* Ruby versions:
+  - for any testing, use the Ruby 3.4 configured in `asdf` instead of the ancient system Ruby 2.6
+  - since Ruby 3.2 you don't need to require 'set', it's a built-in class
+
+* unit tests:
+  - don't add automated tests to the repo while adding/changing library code unless specifically asked, or when changing the code breaks some existing tests and they need to be fixed
+  - use the classic RSpec `foo.should be_...` matcher style instead of `expect()`, except for calling matchers on blocks as in `expect { ... }.to(not) raise_error...`
+  - use the classic `it "should do this"` naming for test cases instead of `it "does this"`
+  - use raw `describe` instead of `RSpec.describe`

data/CHANGELOG.md

@@ -1,3 +1,31 @@
+## [0.1.0] - 2026-08-17
+
+* added custom, much faster Base32 encoder/decoder, dropping `base32` gem dependency
+* added `CARRepo` (subclass of `CARArchive`) for parsing and processing ATProto repo CAR archives:
+  - use `#walk_all_nodes { |path, section_cid| ... }` to iterate over all records in the repo
+  - use `#commit` or `#commit_section` to access repo commit data
+* reorganized `CID` class:
+  - it now stores only the initial JSON/binary form and avoids generating the other until requested
+  - `#json_form` returns JSON string starting with 'b' (`to_s` is an alias)
+  - `#cbor_form` returns the binary string starting with `\x00`
+  - `#raw_data` returns the binary string *without* the CBOR `\x00` prefix
+  - `#data` is an alias for `#raw_data` for backwards compatibility
+  - both the input and output strings for JSON/binary form are frozen so they can't be modified from outside
+  - added more data validations
+* optimized section parsing & lookup in `CARArchive`:
+  - it keeps an internal Hash of sections as { raw CID data => section }, but doesn't build it by default unless explicitly enabled
+  - use `section_with_cid(cid, use_map: true)` to populate and use the lookup hashmap
+  - either a `CID` object or its `#cbor_form` can be used for lookup
+  - use `section_with_cid(cid, return_body: false)` to return a `CARSection` instead of its decoded body object directly; this will be the default behavior in a future version
+  - various internal optimizations in the decoding code
+  - added more data validations
+  - added `#parsed_sections` helper
+* changed the API of `CARSection`:
+  - use `#json_body` to return the body with values recursively converted to use `$bytes` and `$link` where needed, as before
+  - use `#decoded_body` to return the body only decoded from CBOR, but without the recursive conversion
+  - `#body` is an alias for `#json_body` for now for backwards compatibility
+* added unit tests & YARD docs
+
 ## [0.0.1] - 2026-08-05
 
 - initial release - extracted `CID` and `CARArchive` from Skyfall

data/README.md

@@ -1,13 +1,153 @@
 # Oxygène
 
+[![Gem Version](https://badge.fury.io/rb/oxygene.svg?icon=si%3Arubygems&icon_color=%23ff6251)](https://rubygems.org/gems/oxygene) [![YARD Docs](http://img.shields.io/badge/yard-docs-blue.svg)](https://rubydoc.info/gems/oxygene)
+
 *(n) Gaz incolore, inodore et sans saveur qui compose 1/5ème de l'air atmosphérique.*
 
-Various data decoding primitives for working with AT Protocol and the Atmosphere (including CAR, CID, CBOR). (Extracted from [Skyfall](https://tangled.org/mackuba.eu/skyfall), work in progress.)
+Various data decoding primitives for working with AT Protocol and the Atmosphere (including CAR, CID, CBOR).
 
 > [!NOTE]
 > Part of ATProto Ruby SDK: [ruby.sdk.blue](https://ruby.sdk.blue)
 
 
+## Purpose
+
+Oxygene includes various data decoding related primitives for working with AT Protocol repositories and firehose events. Provides at least partial implementations of standards such as [CAR](https://dasl.ing/car.html), [CID](https://dasl.ing/cid.html) and [Base 32](https://datatracker.ietf.org/doc/html/rfc4648#section-6).
+
+This gem was extracted from [Skyfall](https://ruby.sdk.blue/skyfall/) and is mostly used there for the CBOR firehose decoding, but can also be used standalone for other ATProto data handling purposes, like decoding downloaded .car account repos.
+
+The functionality currently includes:
+
+- Base32 encoder/decoder
+- CAR archive/repo parser
+- CID wrapper
+- and CBOR decoding (through the [cbor gem](https://github.com/cabo/cbor-ruby) for now)
+
+The gem intentionally only includes support for the parts of these standards that are in use in ATProto, i.e. it doesn't handle all possible types of CIDs defined in the CID standard, only generates lowercase Base32, and so on.
+
+
+## Installation
+
+To use Oxygene, you need a reasonably new version of Ruby – it should run on Ruby 2.6 and above, although it's recommended to use a version that's still getting maintainance updates, i.e. currently 3.3+. A compatible version should be preinstalled on macOS Big Sur and above and on many Linux systems. Otherwise, you can install one using tools such as [RVM](https://rvm.io), [asdf](https://asdf-vm.com), [ruby-install](https://github.com/postmodern/ruby-install) or [ruby-build](https://github.com/rbenv/ruby-build), or `rpm` or `apt-get` on Linux (see more installation options on [ruby-lang.org](https://www.ruby-lang.org/en/downloads/)).
+
+To install the gem, run the command:
+
+    [sudo] gem install oxygene
+
+Or add this to your app's `Gemfile`:
+
+    gem 'oxygene', '~> 0.1'
+
+
+## Usage
+
+### CID
+
+The `Oxygene::CID` class is an immutable object wrapper for CIDs (Content Identifiers), both in the binary/CBOR form and in the Base32-encoded JSON string form.
+
+To create one from a CBOR tag object (`CBOR::Tagged`):
+
+```rb
+data = CBOR.decode(file)
+cid = CID.from_cbor_tag(data['cid'])
+```
+
+To create one from a JSON representation:
+
+```rb
+cid = CID.from_json(event['cid'])
+```
+
+Both can be converted to the other representation:
+
+```rb
+cid.cbor_form
+# => "\x00\x01q\x12 \x1A\x88\xF3Z\xFC..."
+
+cid.raw_data
+# same but without the 0 prefix
+# => "\x01q\x12 \x1A\x88\xF3Z\xFC..."
+
+cid.json_form
+# => "bafyreia2rdzvv7fjlp..."
+```
+
+
+### CAR archives & repos
+
+`Oxygene::CARArchive` is a wrapper for any CAR archive file, including e.g. in particular the kind that's contained in a `blocks` field in a `:commit` firehose message.
+
+Create it passing it the binary data to decode:
+
+```rb
+car = Oxygene::CARArchive.new(message.blocks)
+```
+
+The decoder automatically parses the CAR file header containing the "root" CIDs, and makes those available as `#roots` array. The remaining sections are parsed lazily on demand.
+
+You can parse and access the sections in the following ways:
+
+- `#sections` returns an array of all sections in the archive, parsing them if needed
+- `#parsed_sections` returns the sections parsed so far (initially `[]`)
+- `section_with_cid(cid)` looks up a section by CID; if it was already parsed, it's returned immediately, otherwise remaining sections are parsed until a match is found (or the end of the file is reached)
+    - **Note:** parsed sections are searched sequentially with `detect`. If you need to look up multiple sections repeatedly, use the second variant below.
+- `section_with_cid(cid, use_map: true)` also looks up a section, but builds up and uses a `Hash` mapping CIDs to sections for quick lookup. For performance, creating this section index is opt-in, since it's not needed if only one section will be looked up.
+
+Sections are represented as `Oxygene::CARSection` objects. A section has a `#cid`, and can return the body decoded from the binary CBOR data in two ways:
+
+- `#decoded_body` decodes the CBOR into a Ruby `Hash`, but besides that leaves it as is; this means that e.g. CIDs are included as `CBOR::Tagged` objects
+- `#json_body` additionally makes a recursive conversion of the object to an "ATProto JSON" as described in [ATProto Data Model](https://atproto.com/specs/data-model):
+  - CIDs are converted to `{ "$link": "(base32 representation)" }`
+  - binary strings are converted to `{ "$bytes": "(base64 encoded data)" }`
+
+For backwards compatibility reasons, `#section_with_cid` currently returns the `#json_body` of a section directly, unless you pass `return_body: false` to return a `CARSection`. This will be changed in a future version.
+
+For .car account repos, there is additionally a subclass of `CARArchive` called `CARRepo`, which lets you extract the repo's records from the archive's Merkle Search Tree.
+
+Create a repo reader like this:
+
+```rb
+car = Oxygene::CARRepo.new(File.read(repo_path))
+```
+
+Then, iterate over the records using the `#walk_all_nodes` method. The records are indexed by the record "path", which is the NSID collection + the rkey joined with a `/`, and are returned in alphabetical order sorted by that path key (so by collection first and then by rkey within a collection):
+
+```rb
+car.walk_all_nodes do |key, cid|
+  collection, rkey = key.split('/')
+  if collection == 'app.bsky.feed.post'
+    record = car.section_with_cid(cid, use_map: true)
+    # ... save or print record w/ rkey
+  end
+end
+```
+
+
+### Base32
+
+To encode a binary string into Base32:
+
+```rb
+string = Oxygene::Base32.encode(data)
+```
+
+To decode Base32 into binary data:
+
+```rb
+data = Oxygene::Base32.decode(string)
+```
+
+As an optimization, both methods also allow you to specify an offset in the input string from which to start reading, and a prefix to put at the beginning of the output buffer, so you can avoid some string allocations in a hot code path:
+
+```rb
+# skip the \x00 and add a starting 'b'
+json_cid = Oxygene::Base32.encode(cbor_tag_cid, 1, 'b')
+
+# skip the 'b' and add a starting \x00
+binary_cid = Oxygene::Base32.decode(json_cid, 1, "\x00")
+```
+
+
 ## Credits
 
 Copyright © 2026 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/did:plc:oio4hkxaop4ao4wz2pp3f4cr)).

data/lib/oxygene/base32.rb

@@ -0,0 +1,284 @@
+# frozen_string_literal: true
+
+require_relative 'errors'
+
+module Oxygene
+
+  #
+  # Fast, optimized implementation of [RFC 4648 Base32](https://datatracker.ietf.org/doc/html/rfc4648#section-6)
+  # for encoding and decoding binary data.
+  #
+  # Decoding accepts either lowercase or uppercase input and accepts `=` padding if present. Encoder only
+  # creates Base32 output using lowercase alphabet and without padding, since that's what it used in
+  # DASL/ATProto CIDs.
+  #
+  # Based on the code of the [base32 gem](https://github.com/stesla/base32) by Samantha Tesla (MIT).
+  #
+
+  module Base32
+    BASE32_ALPHABET = "abcdefghijklmnopqrstuvwxyz234567".freeze
+
+    BASE32_ENCODE_TABLE = Array.new(1024) { |i|
+      (BASE32_ALPHABET.getbyte(i >> 5).chr + BASE32_ALPHABET.getbyte(i & 31).chr).freeze
+    }.freeze
+
+    BASE32_DECODE_TABLE = begin
+      table = Array.new(256, 255)
+
+      BASE32_ALPHABET.each_byte.with_index do |byte, value|
+        table[byte] = value
+        table[byte - 32] = value if byte >= 97 && byte <= 122
+      end
+
+      table.freeze
+    end
+
+    private_constant :BASE32_ALPHABET, :BASE32_ENCODE_TABLE, :BASE32_DECODE_TABLE
+
+
+    # Encodes a binary string into Base32 (lowercase and without padding).
+    #
+    # @param data [String] the input string to encode
+    # @param start_offset [Integer] byte offset in `data` from which bytes should be read
+    # @param prefix [String] string to add at the beginning of the encoded result
+    #
+    # @return [String] a new string containing the prefix and the input encoded into Base32
+    # @raise [ArgumentError] if `start_offset` is negative or beyond the end of `data`
+
+    def self.encode(data, start_offset = 0, prefix = "")
+      total_size = data.bytesize
+      raise ArgumentError, "Start offset can't be negative" if start_offset < 0
+      raise ArgumentError, "Start offset is larger than the length of data" if start_offset > total_size
+
+      encoded_size = total_size - start_offset
+      output = prefix.dup
+      offset = start_offset
+      full_block_end = total_size - (encoded_size % 5)
+
+      # A single base32 character is one of the 32 values in the BASE32_ALPHABET string
+      # above, i.e. 5 bits. The BASE32_ENCODE_TABLE array stores a (flattened) 32x32 table
+      # of all possible combinations of two-character pairs (10 bits).
+      #
+      # Instead of taking the 40 bits of a 5-byte slice of the original string, slicing
+      # it into 8 5-bit pieces and looking up 8 separate base32 characters, we process
+      # 10 bits at a time here, looking up 4 two-character pairs. This allows us to do
+      # only half the amount of array lookups, shifts and bitwise ands.
+
+      while offset < full_block_end
+        value = (data.getbyte(offset) << 32) |
+                (data.getbyte(offset + 1) << 24) |
+                (data.getbyte(offset + 2) << 16) |
+                (data.getbyte(offset + 3) << 8) |
+                (data.getbyte(offset + 4))
+
+        output << BASE32_ENCODE_TABLE[(value >> 30) & 1023]
+        output << BASE32_ENCODE_TABLE[(value >> 20) & 1023]
+        output << BASE32_ENCODE_TABLE[(value >> 10) & 1023]
+        output << BASE32_ENCODE_TABLE[(value) & 1023]
+
+        offset += 5
+      end
+
+      # Finish the output for the final incomplete block of less than 40 bits:
+
+      case total_size - offset
+      when 1
+        output << BASE32_ENCODE_TABLE[data.getbyte(offset) << 2]
+      when 2
+        value = (data.getbyte(offset) << 8) | data.getbyte(offset + 1)
+
+        output << BASE32_ENCODE_TABLE[value >> 6]
+        output << BASE32_ENCODE_TABLE[(value & 63) << 4]
+      when 3
+        value = (data.getbyte(offset) << 16) |
+                (data.getbyte(offset + 1) << 8) |
+                (data.getbyte(offset + 2))
+
+        output << BASE32_ENCODE_TABLE[value >> 14]
+        output << BASE32_ENCODE_TABLE[(value >> 4) & 1023]
+        output << BASE32_ALPHABET.getbyte((value & 15) << 1)
+      when 4
+        value = (data.getbyte(offset) << 24) |
+                (data.getbyte(offset + 1) << 16) |
+                (data.getbyte(offset + 2) << 8) |
+                (data.getbyte(offset + 3))
+
+        output << BASE32_ENCODE_TABLE[value >> 22]
+        output << BASE32_ENCODE_TABLE[(value >> 12) & 1023]
+        output << BASE32_ENCODE_TABLE[(value >> 2) & 1023]
+        output << BASE32_ALPHABET.getbyte((value & 3) << 3)
+      end
+
+      output
+    end
+
+    # Decodes Base32-encoded data into the original byte string.
+    #
+    # For compatibility, the decoder accepts both lowercase or uppercase inputs and trailing
+    # `=` padding if present. If you want to reject uppercase characters or padding, you need
+    # to perform such checks manually.
+    #
+    # @param data [String] the Base32-encoded input string
+    # @param start_offset [Integer] byte offset in `data` from which Base32 characters should be read
+    # @param prefix [String] string to add at the beginning of the decoded result
+    #
+    # @return [String] a new binary string containing the prefix and the decoded bytes
+    # @raise [ArgumentError] if `start_offset` is negative or beyond the end of `data`
+    # @raise [DecodeError] if the data length, padding, characters used or trailing bits are invalid
+
+    def self.decode(data, start_offset = 0, prefix = "")
+      total_size = data.bytesize
+      raise ArgumentError, "Start offset can't be negative" if start_offset < 0
+      raise ArgumentError, "Start offset is larger than the length of data" if start_offset > total_size
+
+      encoded_end = total_size
+
+      if encoded_end > start_offset && data.getbyte(encoded_end - 1) == 61 # '='
+        # Only decode until the beginning of padding
+        encoded_end -= 1 while encoded_end > start_offset && data.getbyte(encoded_end - 1) == 61
+
+        padding_size = total_size - encoded_end
+
+        # Full blocks are 8 5-bit Base32 characters decoded into 5 8-bit bytes.
+        # Here, we check if in the last incomplete block the padding (if included)
+        # covers the whole rest of the block.
+        #
+        # Because how 5-bit characters map into 8-bit bytes, only a final incomplete
+        # block of 2, 4, 5 or 7 characters makes sense - 1, 3 and 6 don't include enough
+        # bits to create another byte. Which is why padding_size can't be 7 here.
+        #
+        # "n & 7" is the same as "n % 8", just slightly faster.
+
+        unpadded_remainder = (encoded_end - start_offset) & 7
+        expected_padding = (8 - unpadded_remainder) & 7
+
+        if ((total_size - start_offset) & 7) != 0 || padding_size != expected_padding || padding_size > 6
+          raise DecodeError, "Invalid Base32 padding"
+        end
+      end
+
+      encoded_size = encoded_end - start_offset
+      remainder = encoded_size & 7
+
+      # See above for why only these are allowed
+
+      unless remainder == 0 || remainder == 2 || remainder == 4 || remainder == 5 || remainder == 7
+        raise DecodeError, "Invalid Base32 length"
+      end
+
+      output = prefix.dup.force_encoding(Encoding::BINARY)
+      offset = start_offset
+      full_block_end = encoded_end - remainder
+      table = BASE32_DECODE_TABLE
+
+      # Process full blocks of 8 5-bit Base32 characters and decode them into 5 8-bit
+      # bytes. The characters are first mapped into indexes of 0...32, and then combined
+      # into a single 40-bit number, which is sliced into 5 bytes.
+      #
+      # The BASE32_DECODE_TABLE table is a table mapping character indexes 0...256 to
+      # the 0...32 Base32 value. Using this table, a character's index can be looked up in
+      # O(1) time using its .ord code. The special value 255 means that there is no Base32
+      # character with a given ASCII code and that an exception should be raised.
+
+      while offset < full_block_end
+        v0 = table[data.getbyte(offset)]
+        v1 = table[data.getbyte(offset + 1)]
+        v2 = table[data.getbyte(offset + 2)]
+        v3 = table[data.getbyte(offset + 3)]
+        v4 = table[data.getbyte(offset + 4)]
+        v5 = table[data.getbyte(offset + 5)]
+        v6 = table[data.getbyte(offset + 6)]
+        v7 = table[data.getbyte(offset + 7)]
+
+        # Combine the bits of the 8 index values using OR. If any of the 8 characters
+        # has any bits above the 6th (in practice it could only be 255), the combined
+        # value will also have the bits set so it will be higher than 31.
+
+        invalid_value = v0 | v1 | v2 | v3 | v4 | v5 | v6 | v7
+        invalid_character!(data, offset, offset + 8, table) if invalid_value > 31
+
+        value = (v0 << 35) | (v1 << 30) | (v2 << 25) | (v3 << 20) | (v4 << 15) | (v5 << 10) | (v6 << 5) | v7
+
+        output << ((value >> 32) & 255)
+        output << ((value >> 24) & 255)
+        output << ((value >> 16) & 255)
+        output << ((value >> 8) & 255)
+        output << (value & 255)
+
+        offset += 8
+      end
+
+      # Finish the output for the final incomplete block of less than 40 bits:
+
+      case remainder
+      when 2
+        v0 = table[data.getbyte(offset)]
+        v1 = table[data.getbyte(offset + 1)]
+        invalid_character!(data, offset, encoded_end, table) if (v0 | v1) > 31
+
+        value = (v0 << 5) | v1
+        raise DecodeError, "Invalid Base32 trailing bits" unless (value & 3) == 0
+
+        output << (value >> 2)
+
+      when 4
+        v0 = table[data.getbyte(offset)]
+        v1 = table[data.getbyte(offset + 1)]
+        v2 = table[data.getbyte(offset + 2)]
+        v3 = table[data.getbyte(offset + 3)]
+        invalid_character!(data, offset, encoded_end, table) if (v0 | v1 | v2 | v3) > 31
+
+        value = (v0 << 15) | (v1 << 10) | (v2 << 5) | v3
+        raise DecodeError, "Invalid Base32 trailing bits" unless (value & 15) == 0
+
+        output << ((value >> 12) & 255)
+        output << ((value >> 4) & 255)
+
+      when 5
+        v0 = table[data.getbyte(offset)]
+        v1 = table[data.getbyte(offset + 1)]
+        v2 = table[data.getbyte(offset + 2)]
+        v3 = table[data.getbyte(offset + 3)]
+        v4 = table[data.getbyte(offset + 4)]
+        invalid_character!(data, offset, encoded_end, table) if (v0 | v1 | v2 | v3 | v4) > 31
+
+        value = (v0 << 20) | (v1 << 15) | (v2 << 10) | (v3 << 5) | v4
+        raise DecodeError, "Invalid Base32 trailing bits" unless (value & 1) == 0
+
+        output << ((value >> 17) & 255)
+        output << ((value >> 9) & 255)
+        output << ((value >> 1) & 255)
+
+      when 7
+        v0 = table[data.getbyte(offset)]
+        v1 = table[data.getbyte(offset + 1)]
+        v2 = table[data.getbyte(offset + 2)]
+        v3 = table[data.getbyte(offset + 3)]
+        v4 = table[data.getbyte(offset + 4)]
+        v5 = table[data.getbyte(offset + 5)]
+        v6 = table[data.getbyte(offset + 6)]
+        invalid_character!(data, offset, encoded_end, table) if (v0 | v1 | v2 | v3 | v4 | v5 | v6) > 31
+
+        value = (v0 << 30) | (v1 << 25) | (v2 << 20) | (v3 << 15) |
+                (v4 << 10) | (v5 << 5) | v6
+        raise DecodeError, "Invalid Base32 trailing bits" unless (value & 7) == 0
+
+        output << ((value >> 27) & 255)
+        output << ((value >> 19) & 255)
+        output << ((value >> 11) & 255)
+        output << ((value >> 3) & 255)
+      end
+
+      output
+    end
+
+
+    def self.invalid_character!(data, offset, end_offset, table)
+      offset += 1 while offset < end_offset && table[data.getbyte(offset)] <= 31
+      character = data.byteslice(offset, 1)
+      raise DecodeError, "Invalid Base32 character: #{character.inspect}"
+    end
+
+    private_class_method :invalid_character!
+  end
+end