xitdb 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7ff2dced063b5fa4980d7e5f30c71d016a338b6986980b6a8003156b3b23c2d0
+  data.tar.gz: aa3fac0c1fbed621cbbe30bf059b2ea038a8fcd4660052d02c5a0d17648b7c78
+SHA512:
+  metadata.gz: cdd84a81d4ebe0faa251f986f58eaacabdda3a1fa24968c8c8147ae1d75d3402e796854d6441f963d5a5310fe692153b4e898713810cac41a13e0e0686651b4c
+  data.tar.gz: 693ded1aa3dd47139bf04f99f139e5f5396810ab80b0038d911f190368b3c086d3102f26fdcaba7a95d5d85176fe69e192e9e95755a02378453eeab846d98368

data/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+## Unreleased
+
+## 0.1.0 - 2026-11-05
+
+Initial release of the `xitdb` gem. Requires MRI 3.3 or later.
+
+### Added
+
+- Pure Ruby storage for upstream xitdb maps, sets, counted collections, array lists, linked array lists, and sorted collections.
+- Portable `sha1` and `sha2` files with the full signed and unsigned upstream integer ranges.
+- Transactions with coordinated writers, lock timeouts, rollback, commit outcome reporting, and synchronized file publication.
+- Stable snapshots, incremental collection views, history, revision tokens, and revert operations.
+- Sorted range queries, rank lookup, and pagination through storage indexes.
+- Byte streams with bounded transfers, compaction, and byte-preserving full and incremental replication.
+- The `rb01` Ruby profile with typed keys, insertion-ordered maps and sets, and typed sorted indexes.
+- Exact encodings for arbitrary-size integers, booleans, Symbols, String encodings, temporal values, BigDecimal, and Rational.
+- Per-handle custom value codecs and immutable preservation of unknown encoded values.
+- Explicit profile conversion and context-level copying between databases.
+- RBS signatures, packaged examples, and guides for portable storage, Ruby values, the core API, and the Ruby wire format.
+- Package installation tests, reciprocal checks against pinned Go, TypeScript, Java, and Zig references, and Linux memory workloads.
+
+### Limitations
+
+- Portable keys and set members must be Strings. Booleans, Symbols, and native temporal values require the Ruby profile.
+- Conversion can duplicate data shared across histories, increasing runtime and output size. See [the measurements](docs/benchmarks.md).
+- The public API can change before version 1.0.

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (Expat)
+
+Copyright (c) contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,131 @@
+# xitdb for Ruby
+
+[`xitdb`](https://rubygems.org/gems/xitdb) provides transactional collections in one file, implemented in Ruby. It requires MRI 3.3 or later.
+It requires no compiler or database service. Runtime dependencies are `date` and `bigdecimal`.
+
+This is a Ruby implementation of [xitdb](https://github.com/xit-vcs/xitdb), a binary format for immutable database structures.
+The repository is named `xitdb-ruby`; the gem and require path are `xitdb`.
+
+The portable profile reads and writes upstream xitdb structures. The Ruby profile adds typed keys, ordered maps, and Ruby values.
+Both profiles use the same storage core. Requiring the Ruby extension does not change portable behavior.
+
+## Install
+
+Add the gem to your application's bundle:
+
+```sh
+bundle add xitdb
+```
+
+Or install it directly:
+
+```sh
+gem install xitdb
+```
+
+## Quick start
+
+Use String keys and portable values to share files with other xitdb implementations:
+
+```ruby
+require "xitdb"
+
+XitDB.open("shared.xit") do |db|
+  db.transaction { |root| root["profile"] = { "name" => "Ada", "visits" => 0 } }
+  before = db.snapshot
+  db.transaction { |root| root["profile"]["visits"] += 1 }
+  p before.dig("profile", "visits")       # 0
+  p db.snapshot.dig("profile", "visits")  # 1
+end
+```
+
+Use the Ruby profile for Symbol keys, booleans, arbitrary-size integers, and other Ruby values:
+
+```ruby
+require "xitdb/ruby"
+
+XitDB::Ruby.memory do |db|
+  db.transaction do |root|
+    root[:enabled] = true
+    root[:large] = 2**200
+  end
+  p db.snapshot[:enabled] # true
+end
+```
+
+## Choose a profile
+
+| Entry point | Values and keys | File identifier |
+| --- | --- | --- |
+| `XitDB.open`, `XitDB.memory` | Portable integers, floats, bytes, collections, and String keys | `sha1` by default; `hash: :sha256` selects `sha2` |
+| `XitDB::Ruby.open`, `XitDB::Ruby.memory` | Ruby scalar values, typed keys, and ordered maps and sets | `rb01` |
+| `XitDB::Core::Database` | Raw slots, every upstream collection, and arbitrary roots | Explicit storage configuration |
+
+Opening never converts a profile. Public entry points reject a mismatched profile and pre-existing zero-byte files.
+Missing paths are created. Block forms close the handle; callers close handles returned without a block.
+
+## Transactions and limits
+
+Write through `db.transaction`. Normal block completion commits; an exception aborts the transaction.
+Snapshots retain their committed values through later writes and remain usable until the database handle closes.
+Transaction views expire when their block ends. Use `to_ruby` to obtain independent Ruby containers.
+
+Writers serialize across cooperating threads and processes. Lock timeout raises `XitDB::BusyError` before the transaction block runs.
+File commits synchronize data and root publication. A `XitDB::CommitError` reports whether publication failed, succeeded, or has an unknown outcome.
+See [locking and durability](docs/core.md#locking-and-durability) before implementing retries.
+
+Portable maps and sets require String keys or members; portable values exclude booleans and Symbols.
+The Ruby profile supports these types and preserves insertion order in ordinary maps and sets.
+Profile conversion is explicit. Conversion can duplicate shared data across histories; see [measurements and costs](docs/benchmarks.md).
+
+Version 0.1.0 is the initial release. The public API can change before 1.0.
+
+## Documentation
+
+Read the [portable guide](docs/portable.md), [Ruby guide](docs/ruby.md), and [core guide](docs/core.md).
+The [Ruby wire contract](docs/ruby-profile-v1.md) defines cross-language interpretation of `rb01`.
+Other languages can preserve Ruby-profile bytes without decoding their logical values.
+
+## Run the workflows
+
+1. Run `ruby examples/portable.rb` for nested updates, snapshots, history, streams, compaction, and replication.
+2. Run `ruby examples/ruby.rb` for native values, codecs, cross-file copying, and explicit conversion.
+3. Run `ruby examples/indexed_users.rb` for atomic record/index updates and sorted pagination.
+
+Examples require the installed package and write temporary files only. Each emits one JSON object.
+From a checkout, run them with `bundle exec ruby -Ilib examples/portable.rb`, substituting the example filename.
+
+## Verify a checkout
+
+Run `bundle install` to install the locked development dependencies.
+Run `bundle exec ruby script/ci.rb` to check the checkout.
+It checks Ruby style, library types, and Change Risk Anti-Patterns (CRAP), then inspects the locally built package.
+The tests execute packaged examples from temporary storage and inspect package contents.
+The package check uses an isolated application bundle and local gem resolution. It does not install dependencies from the network.
+
+`bundle exec rake` runs the same checks using the locked development dependencies.
+Plain `rake` also works when those versions are installed on Ruby's gem paths.
+RuboCop loads the Minitest, Performance, and Rake plugins. Individual checks are available:
+
+| Command | Check |
+| --- | --- |
+| `bundle exec rake rubocop` | Project style rules, with size metrics exempted |
+| `bundle exec rake types` | Steep diagnostics and 100% typed calls in every library file |
+| `bundle exec rake crap` | Fast tests with fresh SimpleCov coverage, then a CRAP limit of 10 |
+| `bundle exec rake test` | Fast tests without the analysis tools |
+
+RBS signatures ship in `sig/`. Custom codec results have an unknown type until callers validate their class.
+Type coverage measures checked method calls. It does not prove that arbitrary codec output meets an application's domain constraints.
+
+The workflow defines separate `reference` and `linux-memory` jobs alongside Linux MRI 3.3, 3.4, and 4.0 and macOS jobs.
+Configure every job as a required branch check. Workflow definitions alone do not enable repository branch protection.
+Reference tools are mandatory for `ruby script/interop.rb --all`; unavailable tools fail that gate.
+The [measurement guide](docs/benchmarks.md) records bounded-memory evidence and conversion costs.
+
+## Releases
+
+See [CHANGELOG.md](CHANGELOG.md) for release changes and [the release guide](docs/releasing.md) for the publishing procedure.
+
+## License
+
+The package preserves the [MIT license notice](LICENSE). Upstream xitdb references remain pinned in `test/interop/manifest.json`.

data/docs/benchmarks.md

@@ -0,0 +1,135 @@
+# Memory and performance measurements
+
+The stress command measures file databases inside an isolated Linux container with a hard memory budget.
+The benchmark command measures fixed workloads without timing pass thresholds.
+Both commands require Docker and the cached `ruby:4.0.9-slim` image.
+They use `--pull=never` and `--network=none` and fail when the required capability is unavailable.
+They install nothing.
+
+## Run the workloads
+
+1. Run `ruby script/stress.rb --limit-mib 256 --data-mib 768`.
+2. Run `ruby script/benchmark.rb --output pkg/benchmark.json`.
+3. Run `bundle exec ruby -Ilib:test test/stress/memory_test.rb` for the covering tests.
+
+The covering stress test executes the same 768 MiB command and saves its report in `pkg/stress.json`.
+It therefore supplies the command evidence without a second standalone stress run.
+The stress suite is separate from `rake test`.
+Set `XITDB_STRESS_REPORT=PATH` to validate a previously captured report and republish it without starting another full workload.
+This explicit mode checks recorded measurements and the report publication path; it does not provide a fresh Linux execution result.
+
+Docker sets the container's memory and combined memory-plus-swap limits to the requested budget.
+The child verifies Linux cgroup v2 `memory.max` and `memory.swap.max` before creating data.
+Swap must be zero.
+The report records these actual limits, `memory.peak`, and `memory.events`.
+The cgroup peak includes kernel charges and file cache, not only Ruby's resident memory.
+The budget applies during fixture creation, operations, and verification.
+
+With Docker's systemd cgroup driver, the harness sets `memory.high` to 75% of the hard limit through runc's systemd annotation.
+For the 256 MiB workload, this starts reclamation at 192 MiB while retaining the 256 MiB maximum and zero swap.
+This gives the kernel time to reclaim filesystem cache before the hard limit can trigger an OOM kill.
+The child verifies the configured threshold and records `memory_high_bytes` in its report.
+Docker's cgroupfs driver retains its hard-limit behavior and records `memory_high_bytes` as `"max"`.
+See [the kernel's memory controller guide](https://docs.kernel.org/admin-guide/cgroup-v2.html#usage-guidelines)
+and [runc's systemd properties](https://github.com/opencontainers/runc/blob/main/docs/systemd.md#auxiliary-properties).
+
+Each command checks available scratch space before starting.
+Datasets use one task-owned temporary directory.
+Profiles run sequentially, and each destination closes and unlinks before the next maintenance operation.
+The parent removes the directory on normal completion or failure.
+If the child fails, the parent removes its own container and reports the failure.
+It does not use a weaker memory limit or skip a workload.
+
+## Stress workload
+
+Each profile stores 8,192 deterministic sorted rows and a 768 MiB byte value.
+The generator writes one reusable 64 KiB chunk and retains one digest.
+Two histories share the byte value and sorted rows, with different revision scalars.
+`dataset_bytes` comes from the committed source file's actual size.
+`payload_bytes` records the complete generated byte count.
+
+Lookup checks present and absent keys.
+Pagination checks every sorted row through pages of at most 37 entries.
+Streaming checks the complete byte count and SHA-256 digest through public `open_bytes`.
+Conversion copies both histories to the other profile.
+Compaction retains the latest history in the original profile.
+Both outputs receive lookup, pagination, history, revision, byte-count, and digest checks.
+Verification times are separate from conversion and compaction times.
+Destination file sizes record the output cost of each operation.
+
+Converted Ruby Strings can use tagged envelopes that public `open_bytes` does not accept.
+Their verification uses the existing profile byte descriptor and bounded range reads.
+Original explicit byte streams use public `open_bytes` in both profiles.
+Source reads exceeding 64 KiB fail throughout measured operations and destination verification.
+The workload never decodes the large scalar or materializes a database root.
+
+The offset-cache observer reads `OffsetMap#peak_cache_size` when each map closes.
+It retains one maximum integer, without retaining maps, entries, or per-access records.
+The covering test requires a positive observed peak no greater than 16,384 entries.
+
+## Benchmark fields and interpretation
+
+Each benchmark workload uses a 1 MiB payload, 256 sorted rows, and two histories.
+Portable and Ruby profiles run against file and memory sources.
+Every operation has three elapsed samples and three allocation counts.
+The output records minimum, median, and maximum elapsed seconds, Ruby version, source sizes, destination sizes, and fixture times.
+Allocation counts use `GC.stat(:total_allocated_objects)` differences.
+They count Ruby objects, not bytes or retained memory.
+Maintenance verification runs outside the operation's elapsed and allocation sample.
+
+File workloads separate freshly reopened handles from warm handles.
+Memory workloads separate newly constructed handles from warm handles.
+A warm handle receives one complete unmeasured workload before sampling.
+Each memory first-handle sample receives a separately generated fixture.
+File reopen samples reuse the source file and reopen its database handle.
+All states use identical logical parameters and checks.
+
+Operating-system cache control is unavailable and recorded as `os_cache_control: unavailable`.
+Reopening a file does not establish a cold filesystem cache.
+The output makes no cold-cache timing claim.
+The samples describe one run, without comparisons to other implementations or portable performance guarantees.
+
+## Shared-history characterization
+
+The benchmark also measures 25 shared histories in each profile.
+One fixture shares a 128-element array.
+Another retains a depth-600 array chain and a 64-element array in every history.
+Fixture creation, conversion, and successful verification have separate timings.
+Each record includes history length, source size, destination size, and their ratio.
+
+Conversion has a ten-second diagnostic timeout for these fixtures.
+A timeout records elapsed time through abort cleanup and the actual remaining destination size.
+It has status `timeout`, no verification sample, and no successful conversion sample.
+The cutoff bounds this diagnostic; it is not a performance pass threshold.
+Conversion copies shared logical nodes separately, so retained histories can increase runtime and output size substantially.
+
+## Observed run
+
+An aarch64 Linux run used MRI 4.0.9 and `ruby:4.0.9-slim` on November 5, 2026.
+The 768 MiB workload completed in both profiles under a 268,435,456-byte cgroup limit with zero swap.
+The observed cgroup peak equaled that limit; OOM and OOM-kill event counts were zero.
+Both offset caches peaked at 16,384 entries. Independent inspection directly observed disk spill for the Ruby profile.
+Other validation jobs ran concurrently, and operating-system cache state was uncontrolled.
+These measurements establish this workload's completion and output cost.
+
+| Source profile | Source bytes | Fixture seconds | Conversion seconds | Conversion output bytes | Compaction seconds | Compaction output bytes |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Portable | 818,396,326 | 39.962 | 53.047 | 1,644,815,197 | 8.370 | 805,827,485 |
+| Ruby | 833,361,937 | 130.068 | 32.251 | 1,633,809,495 | 8.866 | 807,393,651 |
+
+Conversion preserved both histories and independently copied their shared payloads.
+Compaction preserved one latest history.
+Conversion verification took 6.030 and 4.601 seconds; compaction verification took 4.636 and 5.970 seconds, respectively.
+
+The separate 1 MiB benchmark peaked at 197,292,032 cgroup bytes and recorded zero OOM events.
+Its shared-history results were:
+
+| Source profile | Depth | Fixture seconds | Source bytes | Conversion result | Destination bytes |
+| --- | ---: | ---: | ---: | --- | ---: |
+| Portable | 0 | 0.069 | 48,219 | Completed in 1.303 s | 1,560,437 |
+| Ruby | 0 | 0.148 | 100,661 | Completed in 1.030 s | 941,187 |
+| Portable | 600 | 0.328 | 220,945 | Timed out after 10.019 s including cleanup | 0 |
+| Ruby | 600 | 1.391 | 648,584 | Timed out after 10.018 s including cleanup | 0 |
+
+The shallow conversions expanded output to approximately 32.36 and 9.35 times source size.
+The deep cases provide interrupted diagnostic results, not completed conversion timings.

data/docs/core.md

@@ -0,0 +1,123 @@
+# Storage core and maintenance
+
+`require "xitdb/core"` exposes raw upstream storage. Public profile handles also expose `db.core`.
+Core accepts arbitrary valid root layouts; public profiles require a history array containing logical root maps.
+Unknown format tags remain uninterpreted bytes. Raw graphs can contain sharing and cycles that public materialization rejects.
+
+## Raw collections
+
+```ruby
+require "xitdb/core"
+
+db = XitDB::Core::Database.memory
+begin
+  db.transaction do |root|
+    list = XitDB::Core::ArrayList.new(root)
+    list.append(XitDB::Core::Slot.uint(42))
+  end
+  list = XitDB::Core::ArrayList.new(db.root_cursor)
+  p list.cursor_at(0).slot.value # 42
+ensure
+  db.close
+end
+```
+
+Core constructors include `ArrayList`, `LinkedArrayList`, `HashMap`, and `SortedMap`.
+`HashMap` supports counted and set variants; `SortedMap` supports set mode.
+Callers provide raw digest bytes to hash collections and raw key bytes to sorted collections.
+Slots contain a tag, full flag, and unsigned 64-bit payload. Use `db.bytes(data, format_tag:)` during a transaction for byte slots.
+Core cursor ownership and transaction-local references are lower-level contracts; public views provide value capture and operation rollback.
+Core empty stores may initialize. Public opening rejects pre-existing zero-byte files.
+
+## Conversion and copying
+
+`db.convert(to:, profile:, history: :all, transform: nil)` returns an open writable destination handle owned by the caller.
+Always close that handle. Destinations must be missing or zero bytes, including for empty sources.
+`history:` accepts only `:all` and `:latest`. Empty sources retain zero history entries.
+Same-profile symbols retain source configuration; cross-profile symbols select defaults.
+Exact configured `XitDB::Profiles::Portable` and `XitDB::Profiles::Ruby` instances create fresh target profiles and registries.
+
+```ruby
+require "xitdb/ruby"
+
+XitDB::Ruby.memory do |db|
+  db.transaction { |root| root[:enabled] = true }
+  portable = db.convert(to: "converted.xit", profile: :portable, history: :latest,
+    transform: ->(value:, path:, role:) { role == :key ? value.to_s : (value == true ? 1 : value) })
+  begin
+    p portable.snapshot["enabled"] # 1
+  ensure
+    portable.close
+  end
+end
+```
+
+Transforms run once for each source scalar or logical key. Keywords are `value:`, `path:`, and `role:` (`:key` or `:value`).
+Paths are immutable and relative to the copied source root. Returned products are encoded without recursively applying the transform.
+Ruby-only keys, booleans, native numeric sorted keys, and unknown custom codecs need explicit portable representations.
+Collapsing distinct keys raises `KeyCollisionError`. Opaque portable `rb` payloads also require an explicit transform for Ruby targets.
+Sorted kinds remain sorted. Portable-to-Ruby sorted keys use the String kind.
+
+Inside a destination transaction, use `root.context.copy_from(source_view, at: ["parent", "copy"], transform: nil)`.
+It returns the installed destination view. `at:` requires a full Array path with existing parents.
+`at: []` explicitly replaces the root with a Map or sorted Map. Previous child views expire; the transaction root remains usable.
+No view-level copy wrapper exists. Direct foreign-view assignment remains an error.
+
+Conversion traverses roots and histories incrementally. Transform-free Strings and opaque bytes stream in 64 KiB chunks.
+A transform may allocate its complete scalar input. Shared data is copied independently across histories.
+The [measurements](benchmarks.md) record 25-history shallow conversions expanding to 32.36 and 9.35 times source size.
+Depth-600 repeated-history diagnostics exceeded ten seconds and were interrupted; those are not completed conversion timings.
+Compaction retains only the latest root and can preserve physical sharing. It is separate from logical conversion.
+
+## Replication
+
+Run [the portable example](../examples/portable.rb) for a complete bootstrap and incremental update.
+Use binary caller-owned IO. Transport objects stay open and input consumes only the declared length.
+
+1. Capture `finish = source.replica_state.committed_size` from a Core handle.
+2. Export `source.read_committed(0, finish, io)`, then rewind the transport.
+3. Bootstrap with `Core::Database.copy_replica(to: missing_path, source_io: io, length: finish)`.
+4. Negotiate updates with `start = source.replication_start(replica.replica_state)`.
+5. For positive start, export through a captured finish and call `replica.apply_replicated(start, io, finish - start)`.
+
+`ReplicaState` is immutable with `committed_size` and `checksum`. `replication_start` returns the matched size or zero.
+`read_committed` and `apply_replicated` return nil. `copy_replica` returns a separate validated open writable Core handle; its caller closes it.
+A zero negotiation requires a full copy to another missing or zero-byte path. `apply_replicated(0, ...)` raises `ArgumentError`.
+Application code controls replacement and reader reopening. No API renames a staged file over a live replica.
+Replication preserves bytes and profile metadata. It does not convert profiles or merge independent writers.
+
+Known incremental aborts restore the previous size. Known staging aborts attempt to restore zero, subject to storage cleanup success.
+Staging synchronization failure raises `CommitError` with outcome `:unknown` and retains on-disk output, which may be partial.
+Do not assume that an uncertain staged image is complete or valid. Transport failures preserve their original exception identity.
+Short input raises `EOFError`; invalid trailers raise `CorruptionError`.
+
+## Locking and durability
+
+Writers serialize across cooperating processes and threads. The writer refreshes committed state after acquiring its lock.
+`lock_timeout:` defaults to five seconds. `BusyError` occurs before the transaction callback runs.
+Snapshot reads use positional access and do not acquire the exclusive writer lock.
+Closing waits for active operations; closing inside that handle's transaction raises `TransactionStateError`.
+
+File publication synchronizes data before publishing and synchronizing the root header. A successful transaction returns after confirmed synchronization.
+Each writable pathname handle synchronizes its containing directory after its first file synchronization, including handles opened on existing files.
+This lets cooperating initializers and writers establish pathname durability independently. Later commits on that handle synchronize only the file.
+Opening an existing file validates its contents; it does not confirm durability of bytes left by an earlier failed publication.
+Creation rejects dangling final symlinks with `Errno::ENOENT`; open the intended target path directly. Symlinks to existing files remain supported.
+Memory handles provide atomic visibility without disk durability.
+`CommitError#outcome` is `:not_committed` before publication, `:unknown` during uncertain publication, or `:committed` after confirmed synchronization.
+Uncertain publication, failed rollback, or lock-release failure requires reopening before further writes. Callbacks are never automatically retried.
+Library errors inherit `XitDB::Error`; standard argument, missing-key, index, and range failures retain Ruby error classes.
+
+## Physical contract
+
+The header occupies 20 bytes. The root header adds a nine-byte slot and eight-byte committed size.
+A trailer contains a root slot, commit identifier, and CRC-64/XZ checksum, occupying 25 bytes.
+An initialized empty database occupies 62 bytes. Parsing checks object ranges against committed boundaries before allocating.
+Graph traversal uses an explicit stack. Write buffers are bounded at 2 MiB, byte transfers at 64 KiB, and offset caches spill after 16,384 entries.
+
+`XitDB::Core::Bytes` owns physical byte storage, including inline selection, lengths, format tags, and bounded transfers.
+Scalar materialization, streams, conversion, and compaction use that implementation. Profiles retain payload interpretation and encoding rules.
+Compaction preserves external byte records, including short payloads and unknown format tags.
+
+The port preserves the package's [MIT license notice](../LICENSE).
+Reference revisions and source hashes are recorded in `test/interop/manifest.json` in the source checkout.

data/docs/portable.md

@@ -0,0 +1,99 @@
+# Portable profile
+
+`require "xitdb"` provides `XitDB.open(path, readonly: false, hash: nil, lock_timeout: 5.0)` and `XitDB.memory(hash: :sha1)`.
+New files use SHA-1 with identifier `sha1`. Use `hash: :sha256` to create SHA-256 files with identifier `sha2`.
+Hash collections inherit upstream collision semantics. Bulk input and merge reject distinct keys that collapse to one byte key with `KeyCollisionError`.
+Recognized file hashing is selected on open. Unknown or absent hash identifiers require explicit compatible configuration.
+
+## Transactions and lifetime
+
+A transaction block receives a writable root map. Normal completion publishes its mutations and returns a committed `Snapshot`.
+The block runs once after the writer lock succeeds. Exceptions, `break`, `throw`, and non-local `return` abort publication.
+`next` completes normally. Its return value does not replace the root.
+A transaction without mutations reuses the previous snapshot. Reassigning an equal value can append history.
+
+Nested writes to the same file raise `TransactionStateError`. Each assignment is atomic even if its error is rescued inside the transaction.
+Coercion and codec callbacks may read transaction views. They cannot start another logical mutation while one is applying, including through another Context.
+Byte-stream callbacks run outside finalization and permit non-overlapping sibling writes.
+
+`db.snapshot` captures committed data, including inside a transaction. Transaction views expose the uncommitted data.
+Snapshot views remain valid until handle close. Transaction views expire when their block ends and belong to their thread.
+Replacing or deleting an ancestor expires earlier child views. Errors are `ClosedError`, `ExpiredViewError`, or `ReadOnlyError` as applicable.
+
+Assignment captures input immediately. Assigning a view from the same transaction captures its value without later aliasing.
+Foreign views require explicit copying or `to_ruby`; direct assignment raises `ForeignValueError`.
+Read Strings are frozen. `to_ruby` creates independent mutable containers and Strings.
+Cyclic inputs or materialization of cyclic graphs raise `CyclicValueError` with the offending path.
+
+## Values
+
+| Input | Representation |
+| --- | --- |
+| `nil` | Occupied null slot, distinguishable from absence with `key?` |
+| Integer from `-2**63` through `2**64 - 1` | Exact signed or unsigned upstream integer |
+| `XitDB.uint(n)` | Explicit unsigned slot |
+| Float | IEEE 64-bit float |
+| String | Untagged bytes |
+| Hash, Array, Set | Ordinary upstream map, array list, or set |
+| `XitDB.bytes(data, format_tag: "zz")` | Opaque tagged bytes |
+
+Map keys and set members must be Strings. Boolean, Symbol, temporal, and arbitrary object values require the Ruby profile or an explicit application encoding.
+Strings preserve bytes. Reads use UTF-8 for valid UTF-8 bytes and ASCII-8BIT otherwise; original encoding metadata is not retained.
+Hash defaults, default procs, and unsupported input behavior raise `UnsupportedValueError`.
+
+## Collections and costs
+
+Declarations include `XitDB.map(counted:)`, `XitDB.set(counted:)`, `XitDB.array`, `XitDB.linked_array`, `XitDB.sorted_map`, and `XitDB.sorted_set`.
+Assign a declaration, then mutate its returned view. Native Hash, Array, and Set select ordinary representations.
+Views include `Enumerable`; they do not subclass native collections.
+
+| View | Reads | Writes inside a transaction |
+| --- | --- | --- |
+| Map | `[]`, `fetch`, `dig`, `key?`, `each`, `each_key`, `each_value`, `keys`, `values` | `[]=`, `delete`, `clear`, `merge!` |
+| Array | `[]`, `fetch`, `first`, `last`, `length`, `each` | `[]=`, `<<`, `push`, `pop`, `insert`, `delete_at`, `concat`, `clear` |
+| Set | `include?`, `each` | `add`, `delete`, `clear` |
+| Sorted map/set | Corresponding reads, `rank`, `from`, `from_index`, `page` | Corresponding writes |
+| Linked array | Array reads, slices, `split(index)` | Array writes and concatenation |
+
+All views support `size`, `empty?`, `to_ruby`, and bounded `inspect`. Missing map keys return nil; `fetch` raises `KeyError`.
+Array positions support negative indexes. Assignment beyond the end fills the gap with stored nil.
+Reads never create missing parents. Unimplemented operations raise ordinary `NoMethodError`.
+Plain hash map/set size requires traversal. Counted maps/sets, arrays, and sorted collections have constant-time size.
+Equality can traverse the subtree. Scalar reads allocate that scalar; `keys`, `values`, and eager enumeration allocate their results.
+
+Enumeration is incremental; `each` without a block returns an enumerator. Snapshot enumeration stays read-only and stable.
+Transaction enumeration permits replacing values and editing descendants. Changing membership or order invalidates its iterator with `ConcurrentModificationError`.
+
+Sorted collections compare raw bytes lexicographically. `from(key)` is inclusive; `from_index(index)` seeks by rank.
+`page(offset:, limit:)` returns an incremental enumerator and seeks without scanning skipped entries.
+Keep one snapshot for stable pagination. Applications maintain secondary indexes in the same transaction as their records.
+See [indexed users](../examples/indexed_users.rb).
+
+## History and streams
+
+`db.history[index]` returns a snapshot or nil. Negative indexes count backward; `fetch` raises `IndexError` for a missing index.
+Enumeration is incremental. A file without history has a virtual empty map and no revision.
+`snapshot.revision` is opaque and immutable. `db.snapshot(revision: token)` validates its ancestor proof and reopens that history entry.
+Tokens survive ordinary appends but identify a file generation, not a globally canonical value hash.
+`db.revert(snapshot_or_revision)` appends the selected root and retains subsequent history. Foreign tokens raise `ForeignRevisionError`.
+
+`view.write_bytes(key) { |io| io.write(chunk) }` streams an untagged value. A callback failure discards that assignment.
+`view.open_bytes(key) { |io| ... }` yields a bounded reader with `read`, `readpartial`, `seek`, `rewind`, `pos`, `size`, and `eof?`.
+Use these block-scoped IO objects inside their blocks. Reads do not exceed the selected value.
+Byte transfers use 64 KiB chunks; writes buffer at most 2 MiB. Ordinary String lookup still allocates the complete String.
+Pass a length to `read` for bounded application buffers; `read` without a length allocates the entire remaining byte value.
+
+## Maintenance and compatibility
+
+`db.compact(to: path)` returns an open writable handle to a separate file with one latest root, preserving profile and hashing.
+The caller owns and closes the returned handle.
+The destination must be missing or zero bytes; an initialized empty database is nonempty and rejected.
+Source snapshots remain valid. Application code controls file replacement and reader reopening.
+See [conversion and replication](core.md) for separate-destination ownership and failure contracts.
+
+The pinned Go, TypeScript, Java, and Zig references have reciprocal storage and maintenance checks.
+TypeScript's Number convenience reader loses precision outside the exact Number integer range.
+Its signed `Slot.fromBytes` carrier also makes `readUint` reject values above `2**63 - 1` with `ExpectedUnsignedLongException`.
+Java's convenience `Database.Uint` writer rejects negative long carriers, and `readUint` rejects those stored carriers.
+Java's public raw Slot writer preserves these unsigned bits. TypeScript can also write and retain the full unsigned bits.
+Raw-bit, compaction, and replication checks pass separately from these convenience API limits. Ruby retains the full upstream integer range.

data/docs/releasing.md

@@ -0,0 +1,37 @@
+# Releasing
+
+The gem is named `xitdb`. Releases use the `vVERSION` tag and RubyGems trusted publishing from GitHub Actions.
+The release workflow requires every CI job to pass before publishing: Ruby quality checks, four reference implementations, and Linux memory tests.
+
+## Publisher setup
+
+Register a [pending trusted publisher](https://rubygems.org/profile/oidc/pending_trusted_publishers/new) for the first release:
+
+| Field | Value |
+| --- | --- |
+| Gem name | `xitdb` |
+| Repository owner | `svyatov` |
+| Repository name | `xitdb-ruby` |
+| Workflow filename | `release.yml` |
+| Environment | `release` |
+
+Leave the optional workflow repository fields empty. Create the `release` environment in the GitHub repository settings.
+After the first publication, RubyGems converts the pending publisher into a publisher for the gem.
+See [RubyGems trusted publishing](https://guides.rubygems.org/trusted-publishing/) for authentication details.
+
+## Release procedure
+
+1. Set `XitDB::VERSION` in `lib/xitdb/version.rb` to the release version.
+2. Update README examples and move changelog entries into a dated version section.
+3. Run `bundle exec ruby script/ci.rb` to verify quality and the installed package.
+4. Commit the release changes and push them to GitHub.
+5. Create the matching tag, for example `git tag -a v0.1.0 -m "Release 0.1.0"`.
+6. Push the tag with `git push origin v0.1.0`.
+7. Check the Release workflow and confirm the version and checksum on RubyGems before announcing publication.
+
+The workflow rejects tags that differ from the packaged version.
+It builds the gem before requesting credentials, then authenticates through GitHub OIDC with permission to publish `xitdb`.
+No persistent RubyGems API key is required.
+
+If publisher registration fails, correct the RubyGems settings and rerun the failed job on the same tag.
+For a published version, make corrections in a subsequent release; RubyGems does not permit replacement of a version.