xitdb 0.1.0

data/docs/ruby-profile-v1.md

@@ -0,0 +1,226 @@
+# Ruby profile version 1 wire contract
+
+This contract defines the `rb01` profile without changing upstream format version 1.
+
+## File identity
+
+The upstream header has hash size 32 and hash identifier `rb01`. Hash indexes use SHA-256 over the canonical key bytes defined below.
+
+The physical root is an upstream array list of history entries. Each entry references an encoded logical root map.
+
+The portable profile uses its upstream header and layout. It does not inspect or decode Ruby envelopes.
+
+## Scalar notation
+
+- W1: `U(n)` is minimal unsigned LEB128 for a nonnegative integer.
+- W2: `B(bytes)` is `U(bytes.bytesize)` followed by those bytes.
+- W3: `T(text)` is `B(text.encode("UTF-8"))`, with valid UTF-8 required.
+- W4: `Z(n)` is one sign byte, then `B(magnitude)`. The sign is 0 for nonnegative and 1 for negative.
+- W5: `Q(r)` is `Z(r.numerator)` followed by `U(r.denominator)`, with a positive, reduced denominator.
+
+Integer magnitudes use minimal big-endian bytes. Zero has an empty magnitude and sign 0. A zero integer therefore has `Z(0) = 00 00`.
+
+`F64(f)` is the eight bytes from Ruby's `pack("G")`. Multi-byte fixed integers use big-endian order.
+
+Every field length must fit inside its enclosing byte payload. Nonminimal integers, truncated fields, invalid flags, and trailing bytes raise `InvalidValueError`.
+
+## Encoded scalar values
+
+Upstream byte values carry format tag `rb`, hexadecimal `72 62`. Their payload starts with version byte `01` and a type byte.
+
+| Type byte | Ruby value | Remaining payload |
+| --- | --- | --- |
+| `01` | String | `T(encoding.name)`, `B(bytes)` |
+| `02` | Symbol | `T(name.encoding.name)`, `B(name.bytes)` |
+| `03` | Boolean | One byte, `00` for false or `01` for true |
+| `04` | Large Integer | `Z(value)` |
+| `05` | Time | `Q(value.to_r)`, `Z(value.utc_offset)`, one byte for `utc?` |
+| `06` | Date | `Z(value.jd)`, `F64(value.start)` |
+| `07` | DateTime | `Q(value.ajd)`, `Q(value.offset)`, `F64(value.start)` |
+| `08` | BigDecimal | Decimal encoding defined below |
+| `09` | Rational | `Q(value)` |
+| `0a` | Registered custom value | `T(codec_id)`, `B(codec_bytes)` |
+
+Integers fitting a native upstream integer slot use that slot. Signed 64-bit values use `int`; remaining unsigned 64-bit values use `uint`.
+
+Values outside those ranges use type `04`. Floats use upstream float slots. Stored null uses an occupied upstream `none` slot.
+
+Time decoding uses the exact rational instant and recorded UTC offset. The UTC flag distinguishes UTC presentation from a fixed zero offset.
+
+The instant retains subnanosecond precision. The offset field accepts integer seconds only; fractional-second UTC offsets raise `UnsupportedValueError`.
+
+Date and DateTime decoding restores the recorded calendar start. Their date fields use the public `date` library constructors.
+
+Untagged byte values represent explicit binary data. Ruby reads them as frozen ASCII-8BIT strings. Public byte streams address these untagged values.
+
+Other format tags remain opaque tagged-byte values. An unknown custom codec retains its complete encoded payload and codec identifier.
+
+An unsupported Ruby payload version or builtin type raises `UnsupportedEncodingError`. A registered custom loader never runs during file opening or unrelated traversal.
+
+### Decimal encoding
+
+The first decimal byte is 0 for finite, 1 for positive infinity, 2 for negative infinity, or 3 for NaN.
+
+Infinite and NaN encodings end after that byte. A finite value adds one sign byte, `T(digits)`, and `Z(exponent)`.
+
+Sign 0 is positive; sign 1 is negative. Digits and exponent come from `BigDecimal#split`, whose base must be 10.
+
+Digits contain normalized ASCII decimal digits. Zero uses digits `0` and exponent 0. Its sign preserves positive or negative zero.
+
+## Canonical hash keys
+
+These bytes determine key equality and hashing. The stored entry separately retains the original encoded key for enumeration.
+
+| Prefix | Accepted key | Remaining canonical bytes |
+| --- | --- | --- |
+| `00` | nil | None |
+| `01` | false | None |
+| `02` | true | None |
+| `03` | Integer | `Z(value)` |
+| `04` | Float | `F64(value)`, with signed zero normalized to positive zero |
+| `05` | String | Canonical string body defined below |
+| `06` | Symbol | Canonical string body for `value.to_s` |
+
+NaN keys raise `UnsupportedValueError`. Infinite float keys are accepted in hash indexes.
+
+A canonical string body is `T(encoding_label)` followed by `B(bytes)`. The encoding label is empty for every empty string.
+
+For nonempty ASCII-only strings with an ASCII-compatible encoding, the label is also empty. Other strings use their canonical `Encoding#name`.
+
+This preserves Ruby's empty-string equality across encodings. It also preserves the distinction between non-ASCII binary and UTF-8 strings.
+
+String and Symbol prefixes remain distinct. Integer and Float prefixes remain distinct.
+
+Golden vectors include:
+
+| Input | Hexadecimal canonical bytes |
+| --- | --- |
+| nil | `00` |
+| false | `01` |
+| true | `02` |
+| 0 | `03 00 00` |
+| 1 | `03 00 01 01` |
+| -1 | `03 01 01 01` |
+| Empty String in any encoding | `05 00 00` |
+| ASCII String `name` | `05 00 04 6e 61 6d 65` |
+| Symbol `name` | `06 00 04 6e 61 6d 65` |
+
+The decoder does not reconstruct keys from canonical bytes. Entry records retain their original Ruby representation.
+
+## Container envelopes
+
+Each envelope is an upstream array list with the exact fields shown below. Marker fields are upstream unsigned integer slots.
+
+Encoded child values can be native scalar slots, encoded scalar bytes, or container envelopes. Internal indexes are ordinary upstream containers.
+
+| Marker | Logical type | Envelope fields |
+| --- | --- | --- |
+| `0x52420101` | Array | `[marker, array_list]` |
+| `0x52420102` | Linked array | `[marker, linked_array_list]` |
+| `0x52420103` | Ordered map | `[marker, next_id, lookup, order]` |
+| `0x52420104` | Ordered set | `[marker, next_id, lookup, order]` |
+| `0x52420105` | Sorted map | `[marker, key_kind, sorted_map]` |
+| `0x52420106` | Sorted set | `[marker, key_kind, sorted_map]` |
+
+Arrays and linked arrays contain encoded children. Their envelopes do not require materializing those children.
+
+Unexpected markers, field counts, or physical collection kinds raise `InvalidValueError`. The core remains able to inspect the physical values.
+
+### Ordered maps and sets
+
+`next_id` is an upstream unsigned integer starting at zero. `lookup` is a counted upstream hash map. `order` is an upstream sorted map.
+
+A lookup entry uses `SHA256(canonical_key_bytes)` as its hash. Its physical key is the canonical byte string.
+
+Its physical value is a three-element upstream array list: `[encoded_key, encoded_value, insertion_id]`. Sets store occupied null as `encoded_value`.
+
+An order entry maps the eight-byte big-endian insertion identifier to the 32-byte lookup hash. Both indexes reference the same logical entry.
+
+Insertion creates one entry and increments `next_id`. Updating an equal key retains its original encoded key and insertion identifier.
+
+Deletion removes both index entries. Reinsertion receives a fresh identifier. Clearing resets both indexes and `next_id`.
+
+When `next_id` equals `2**64 - 1`, another insertion raises `RangeError`. Updates and deletions remain valid.
+
+The logical size comes from the counted lookup map. Iteration follows the order index and resolves its referenced entries.
+
+### Sorted maps and sets
+
+Key kind is 1 for String, 2 for Integer, 3 for Float, or 4 for Time.
+
+A sorted map stores order-preserving key bytes and a two-element record: `[encoded_original_key, encoded_value]`.
+
+A sorted set also uses an upstream sorted map. Its value is the encoded original member. This retains representation beyond the comparison key.
+
+Updating an equal comparison key retains its original encoded key or member. Every operation uses the declared key kind.
+
+## Ordered key encoding
+
+Ordered encodings use unsigned byte comparison. They contain no locale-dependent data and do not use Ruby's randomized hash.
+
+### Integers
+
+For nonnegative `n`, define `O(n)` as an eight-byte magnitude length followed by minimal big-endian magnitude bytes.
+
+`O(0)` has length zero and no magnitude bytes. The length field is an unsigned 64-bit integer.
+
+Ordered signed integers use prefix `00` for negative, `01` for zero, and `02` for positive.
+
+Negative values append the bytewise complement of `O(abs(n))`. Positive values append `O(n)`. Zero ends after its prefix.
+
+These encodings are prefix-free and compare in numeric order. Magnitude lengths exceeding the physical file range raise `RangeError`.
+
+### Floats
+
+Reject NaN. Normalize either signed zero to positive zero. Read the IEEE-754 bits as an unsigned 64-bit integer.
+
+For negative values, complement all 64 bits. For nonnegative values, toggle the sign bit. Write the resulting eight bytes big-endian.
+
+This places negative infinity first and positive infinity last. It gives either zero representation one lookup key.
+
+### Strings
+
+String keys must convert to valid UTF-8. Their ordered bytes are the resulting UTF-8 bytes without a length prefix.
+
+The entry record retains the original encoding and bytes. Invalid or unconvertible keys raise `UnsupportedEncodingError`.
+
+This comparison deliberately uses UTF-8 text order. Ordinary Ruby hash-key encoding retains its separate equality contract.
+
+### Exact time instants
+
+Time keys compare `value.to_r`, including fractions smaller than a nanosecond. Their comparison ignores presentation offsets.
+
+For nonnegative rational `x`, define a prefix-free encoding `R(x)`:
+
+1. Compute `q, remainder = x.numerator.divmod(x.denominator)`.
+2. Write `O(q)`.
+3. For zero remainder, append byte `00` and stop.
+4. Otherwise, append byte `01` and the complement of `R(x.denominator / remainder)`.
+
+The division in step 4 denotes an exact Rational. An iterative continued-fraction implementation avoids Ruby stack growth.
+
+For signed rational instants, use the signed prefixes from integer encoding. Negative instants append the complement of `R(abs(x))`.
+
+Positive instants append `R(x)`. Zero ends after its prefix. Prefix-free child encodings make byte complementation reverse their order.
+
+This requires work proportional to the continued-fraction representation. It does not emit one byte per denominator unit.
+
+## Ownership and loader behavior
+
+View reads return frozen scalar values. Materialization creates independent values and mutable containers and strings where Ruby permits mutation.
+
+Custom loaders receive a frozen copy of their payload. They must return a fresh owned value, without shared mutable state between calls.
+
+The profile freezes the returned object's outer state for view reads. A custom codec owns the immutability of references inside that value.
+
+Materialization invokes the loader independently. No implicit constant lookup or arbitrary object loader interprets stored class names.
+
+## Required fixtures
+
+Golden fixtures cover every scalar type, every container envelope, key equality, signed zero, and unknown custom codecs.
+
+Ordered-key fixtures compare encoded byte order with numeric or temporal order. They include negative values, huge integers, infinities, and exact subnanosecond instants.
+
+Malformed fixtures cover truncated lengths, nonminimal integers, invalid encodings, invalid fields, and conflicting index records.
+
+Changes to canonical keys, payload interpretation, or container layouts require a different profile identifier. Version-one readers never infer a changed layout from data shape.

data/docs/ruby.md

@@ -0,0 +1,61 @@
+# Ruby profile
+
+`require "xitdb/ruby"` provides `XitDB::Ruby.open(path, readonly: false, lock_timeout: 5.0, codecs: [])` and `XitDB::Ruby.memory(codecs: [])`.
+Configuration belongs to each handle. The file identifier `rb01` selects SHA-256 over canonical typed keys and version-one container envelopes.
+The [portable guide](portable.md) defines the shared transaction, snapshot, history, ownership, stream, and view contracts.
+
+## Native values and keys
+
+Supported exact builtin value classes include nil, booleans, Integer, Float, String, Symbol, Hash, Array, Set, Time, Date, DateTime, BigDecimal, and Rational.
+Integers have arbitrary size. Strings preserve bytes and encoding names. Unknown encodings raise `UnsupportedEncodingError` on decoding.
+Time preserves its exact rational instant, including subnanosecond fractions, UTC flag, and integer UTC offset.
+The offset wire field is an integer: fractional-second UTC offsets raise `UnsupportedValueError`. Instant precision is unchanged.
+Named timezone rules are not stored. Date and DateTime preserve their calendar start and date or instant.
+BigDecimal preserves numeric value and signed zero. Rational preserves numerator and denominator.
+Float values preserve IEEE representations, including infinities and NaN outside keys.
+
+Ordinary keys and set members accept exact NilClass, TrueClass, FalseClass, Integer, Float, String, and Symbol instances.
+Symbols and Strings remain distinct, as do Integer and Float keys. Float keys normalize signed zero and reject NaN.
+ASCII-compatible String keys with equal ASCII bytes compare equally. Non-ASCII keys retain encoding-sensitive equality.
+Ordinary keys never use custom codecs. Containers, temporal values, and custom objects cannot be ordinary keys.
+
+Maps and sets preserve insertion order. Updating retains position; deleting and reinserting moves an entry to the end.
+Native containers select the ordinary Ruby-profile representation. Counts are available without scanning members.
+Hash defaults, default procs, identity comparison, singleton behavior, executable objects, and cyclic inputs are unsupported.
+Scalar and container subclasses need an explicit value codec. Unregistered subclasses reject before overridden methods execute.
+
+## Sorted collections
+
+Declare `XitDB.sorted_map(key: :integer)` or `XitDB.sorted_set(key: :string)` explicitly.
+Key kinds are `:string`, `:integer`, `:float`, and `:time`.
+Strings use valid UTF-8 byte order. Integers include arbitrary-size values; Float rejects NaN.
+Time keys compare exact rational instants, independent of presentation offset.
+`rank`, inclusive `from`, `from_index`, and `page` use the storage tree. Snapshot pagination remains stable through later commits.
+
+## Custom values
+
+```ruby
+Money = Struct.new(:cents)
+codec = XitDB::Ruby::Codec.new(
+  id: "example/money/v1", type: Money,
+  dump: ->(money) { money.cents.to_s },
+  load: ->(bytes) { Money.new(Integer(bytes)) }
+)
+XitDB::Ruby.memory(codecs: [codec]) do |db|
+  db.transaction { |root| root[:price] = Money.new(125) }
+  p db.snapshot[:price].cents # 125
+end
+```
+
+Dumps return byte Strings. Use stable, schema-versioned identifiers. Duplicate IDs, overlapping registrations, and builtin overrides raise `CodecConfigurationError`.
+Register subclass values explicitly; registering codecs never extends key support.
+Loader input is a frozen copy. Return a fresh owned value for each call.
+The profile freezes the returned object's outer state. The codec must control mutable references inside that object.
+`to_ruby` invokes loaders independently. Unknown codecs return immutable `EncodedValue` objects that retain complete payload bytes.
+Opening and unrelated traversal never load classes or run arbitrary object deserialization.
+
+Callbacks can read transaction views. Coercion, dump, load, and conversion callbacks cannot start a second logical mutation during one Core transaction's active mutation.
+The guard applies across Context instances; nested writes raise `TransactionStateError` before changing state.
+Callback failure aborts the affected assignment. Byte-stream callbacks retain their separate non-overlapping sibling-write behavior.
+
+See [native workflows](../examples/ruby.rb) and the [wire contract](ruby-profile-v1.md).

data/examples/indexed_users.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+require "xitdb"
+require "json"
+
+XitDB.memory do |db|
+  db.transaction do |root|
+    root["users"] = {}
+    root["by_name"] = XitDB.sorted_map
+    [%w[1 Ada], %w[2 Grace]].each do |id, name|
+      root["users"][id] = { "name" => name }
+      root["by_name"][name] = id
+    end
+  end
+  snapshot = db.snapshot
+  names = snapshot["by_name"].page(offset: 0, limit: 2).map do |_name, id|
+    snapshot["users"][id]["name"]
+  end
+  raise "rank differs" unless snapshot["by_name"].rank("Grace") == 1
+  raise "range differs" unless snapshot["by_name"].from("Grace").first == %w[Grace 2]
+
+  puts JSON.generate(names: names)
+end

data/examples/portable.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+require "xitdb"
+require "json"
+require "tmpdir"
+require "stringio"
+
+Dir.mktmpdir("xitdb-portable-") do |directory|
+  XitDB.open(File.join(directory, "source.xit"), hash: :sha256) do |db|
+    db.transaction do |root|
+      root["profile"] = { "name" => "Ada", "visits" => 0 }
+      root["tags"] = Set["ruby", "storage"]
+      root["events"] = XitDB.linked_array
+      root["events"].push("created", "visited")
+      root["counted"] = XitDB.map(counted: true)
+      root["counted"]["n"] = XitDB.uint((2**64) - 1)
+      root["opaque"] = XitDB.bytes("payload", format_tag: "zz")
+      root.write_bytes("attachment") do |io|
+        io.write("abc")
+        io.write("def")
+      end
+    end
+    before = db.snapshot
+    db.transaction { |root| root["profile"]["visits"] += 1 }
+    raise "snapshot changed" unless before.dig("profile", "visits").zero?
+    raise "revision changed" unless db.snapshot(revision: before.revision).dig("profile", "visits").zero?
+    raise "stream differs" unless db.snapshot.open_bytes("attachment", &:read) == "abcdef"
+    raise "linked slice differs" unless db.snapshot["events"].split(1).last.to_ruby == ["visited"]
+    raise "history missing" unless db.history.fetch(0).dig("profile", "visits").zero?
+
+    db.revert(before)
+    raise "revert differs" unless db.snapshot.dig("profile", "visits").zero?
+
+    db.compact(to: File.join(directory, "compact.xit")).close
+    XitDB.open(File.join(directory, "compact.xit"), readonly: true) do |compact|
+      raise "compaction differs" unless compact.snapshot.dig("profile", "name") == "Ada"
+    end
+
+    # Bootstrap a separate replica, then send one incremental update.
+    full = StringIO.new("".b)
+    finish = db.core.replica_state.committed_size
+    db.core.read_committed(0, finish, full)
+    full.rewind
+    replica = XitDB::Core::Database.copy_replica(to: File.join(directory, "replica.xit"), source_io: full,
+                                                 length: finish)
+    begin
+      db.transaction { |root| root["profile"]["visits"] = 2 }
+      start = db.core.replication_start(replica.replica_state)
+      raise "full staging required" if start.zero?
+
+      finish = db.core.replica_state.committed_size
+      delta = StringIO.new("".b)
+      db.core.read_committed(start, finish, delta)
+      delta.rewind
+      replica.apply_replicated(start, delta, finish - start)
+      raise "replica differs" unless replica.replica_state == db.core.replica_state
+    ensure
+      replica.close
+    end
+    puts JSON.generate(name: db.snapshot.dig("profile", "name"), visits: db.snapshot.dig("profile", "visits"))
+  end
+end

data/examples/ruby.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "xitdb/ruby"
+require "json"
+require "tmpdir"
+require "bigdecimal"
+require "date"
+
+Money = Struct.new(:cents)
+codec = XitDB::Ruby::Codec.new(id: "example/money/v1", type: Money,
+                               dump: ->(money) { money.cents.to_s },
+                               load: ->(bytes) { Money.new(Integer(bytes)) })
+
+Dir.mktmpdir("xitdb-ruby-") do |directory|
+  XitDB::Ruby.memory(codecs: [codec]) do |db|
+    instant = Time.at(Rational(1, 10**12)).utc
+    db.transaction do |root|
+      root[:enabled] = true
+      root[:values] = [2**200, Rational(2, 3), BigDecimal("1.25"), Date.new(2026, 11, 4),
+                       DateTime.new(2026, 11, 4), instant, "é".encode("UTF-16LE"), Money.new(125)]
+      root[:keys] = { name: "symbol", "name" => "string", 1 => "integer", 1.0 => "float" }
+      root[:members] = Set[:second, :first]
+      root[:times] = XitDB.sorted_map(key: :time)
+      root[:times][instant] = "created"
+    end
+    snapshot = db.snapshot
+    raise "exact instant lost" unless snapshot[:values][5].to_r == instant.to_r
+    raise "codec differs" unless snapshot[:values][7].cents == 125
+    raise "key identity lost" unless snapshot[:keys].size == 4
+    raise "set order lost" unless snapshot[:members].to_a == %i[second first]
+    raise "time lookup differs" unless snapshot[:times].from(instant).first.last == "created"
+
+    XitDB::Ruby.memory(codecs: [codec]) do |copy|
+      copy.transaction { |root| root.context.copy_from(snapshot[:values], at: [:copied]) }
+      raise "cross-file copy differs" unless copy.snapshot[:copied][0] == 2**200
+    end
+    # Convert a representable subset with an explicit key/value transform.
+    XitDB::Ruby.memory do |source|
+      source.transaction { |root| root[:enabled] = true }
+      portable = source.convert(to: File.join(directory, "portable.xit"), profile: :portable, history: :latest,
+                                transform: lambda { |value:, role:, path:| # rubocop:disable Lint/UnusedBlockArgument -- Keep the callback's public keyword contract.
+                                  if role == :key
+                                    value.to_s
+                                  else
+                                    (value == true ? 1 : value)
+                                  end
+                                })
+      begin
+        raise "conversion differs" unless portable.snapshot["enabled"] == 1
+      ensure
+        portable.close
+      end
+    end
+    puts JSON.generate(enabled: snapshot[:enabled])
+  end
+end