lutaml-store 0.2.4 → 0.3.0

data/docs/src/pages/cloud-contract.md

@@ -0,0 +1,48 @@
+---
+layout: ../layouts/Docs.astro
+---
+
+# Cloud Store Contract
+
+The lutaml cloud store API contract — distilled from api.relaton.org and generalized. Any service implementing it is a lutaml cloud store.
+
+## The contract
+
+```
+GET {base}/collections                       → { collections: [...] }
+GET {base}/collections/{c}/manifest          → Manifest (JSON)
+GET {base}/collections/{c}/entries/{key}     → record bytes (ETag; 404 = not-found)
+GET {base}/collections/{c}/shards/{n}        → { keys: [...] }   (optional)
+```
+
+## Rules
+
+1. **Keys are URL-safe storage keys** — a single path segment with no raw `/`. Domain identifiers that contain slashes (docids like `ISO/IEC DIR 1`) travel in `entries[].metadata.docid`, and clients resolve reference → storage key through the manifest.
+
+2. **404 is an answer, not an error** — clients raise `NotFoundError` (Ruby) or return `{ ok: false, reason: "not_found" }` (TS). Only 5xx / transport failures are retryable.
+
+3. **Manifests are immutable per generation** — `version`, `generated`, `count` move together; shards are derived from the same generation.
+
+4. **Sharding semantics are domain-owned** — the store/source only fetches named parts; the domain computes which shard (relaton: crc32 of pubid root number).
+
+## The Manifest schema
+
+```json
+{
+  "version": 1,
+  "generated": "2026-09-27T00:00:00Z",
+  "count": 178681,
+  "shards": 256,
+  "entries": [
+    { "key": "rfc 7231", "digest": "sha256:...", "shard": 12, "metadata": { "docid": "RFC 7231" } }
+  ]
+}
+```
+
+## Reference implementation
+
+api.relaton.org serves 30 collections (flavors) with 178k+ records:
+
+- **Browsers** get server-rendered HTML: collections index, searchable record table, framed record pages
+- **API clients** get JSON manifests and raw records from the same URLs
+- **Content negotiation** on the `Accept` header — no separate site

data/docs/src/pages/formats.md

@@ -0,0 +1,54 @@
+---
+layout: ../layouts/Docs.astro
+title: Formats
+---
+
+# Formats
+
+A **Format** handler defines how model entries serialize to and from files. All handlers inherit from `Format::Base` and are registered in `Format::FORMATS`, resolved via `Format.resolve(:symbol)`. Registering a new format never touches existing code.
+
+| Symbol | Format | Multi-doc | Extension | Binary |
+|---|---|---|---|---|
+| `:yaml` | Single YAML document | no | `.yaml` | no |
+| `:yamls` | YAML stream (multi-document) | yes | `.yaml` | no |
+| `:json` | Single JSON document | no | `.json` | no |
+| `:jsonl` | JSON Lines (one object per line) | yes | `.jsonl` | no |
+| `:marshal` | Ruby Marshal | no | `.marshal` | yes |
+| `:xml` | XML | yes | `.xml` | no |
+
+## Handler interface
+
+```ruby
+class MyFormat < Lutaml::Store::Format::Base
+  def serialize(models)  = ...
+  def deserialize(data)  = ...
+  def serialize_many(models) = ...   # multi-doc formats only
+  def deserialize_many(data)  = ...  # multi-doc formats only
+  def extension  = ".myfmt"
+  def glob_pattern = "*.myfmt"
+  def binary? = false
+end
+
+Lutaml::Store::Format.register(:myfmt, MyFormat)
+```
+
+## Choosing a format
+
+- **`:yamls`** — human-editable multi-record files (the Glossarist pattern): one file per concept collection, streamed documents.
+- **`:jsonl`** — append-heavy, line-oriented processing; one model per line.
+- **`:marshal`** — fastest round-trip, Ruby-only, opaque to humans.
+- **`:xml`** — interchange with XML-based pipelines; round-trips LutaML Model XML mappings.
+- **`:json` / `:yaml`** — single-model files, configuration-style data.
+
+## Usage with DatabaseStore
+
+Pass `format:` per model registration:
+
+```ruby
+store = Lutaml::Store.new(
+  adapter: { type: :filesystem, location: "./glossary" },
+  models: [{ model: GlossaryTerm, key: :id, format: :yamls }]
+)
+```
+
+`FormatSerializer` bridges the handler into the store's serializer interface, so CRUD operations transparently read and write the chosen format on disk.

data/docs/src/pages/http-cache.md

@@ -0,0 +1,43 @@
+---
+layout: ../layouts/Docs.astro
+title: HTTP Caching
+---
+
+# HTTP Caching
+
+`Lutaml::Store::HttpCache` provides HTTP-aware caching for network-backed sources, used by the `:https` and `:rest` sources. It is model-driven: cache state serializes through `to_json`/`from_json` on LutaML Model classes, not hand-rolled hashes.
+
+## What it supports
+
+- **ETags** — stored and re-sent as `If-None-Match`
+- **Conditional requests** — a `304 Not Modified` response resolves to the cached body with zero transfer
+- **Cache-Control** — `max-age`, `no-cache`, `no-store` directives are honored
+- **Vary** — responses varied by header are cached separately
+
+## Enabling on the HTTPS source
+
+```ruby
+src = Lutaml::Store::Source.for(:https,
+  base_url: "https://data.example.com/index/",
+  cache: { path: "./cache/http" }   # HttpCacheConfig hash
+)
+
+src.read("some-key")   # first call: 200, cached
+src.read("some-key")   # revalidates with ETag; 304 → cached bytes
+```
+
+## Revalidation semantics
+
+1. **Fresh window** (`max-age` not elapsed): served from cache without a request.
+2. **Stale**: the source sends `If-None-Match`; on `304` the cached body is reused and freshness is extended.
+3. **`no-store` / missing validators**: the response is returned but not cached.
+
+## Transport injection
+
+The `:https` source accepts an injected `transport:` callable for specs and alternate HTTP stacks, so cache behavior is fully testable without the network:
+
+```ruby
+transport = ->(uri, headers) { fake_response }
+src = Lutaml::Store::Source.for(:https,
+  base_url: "https://example.com/", transport: transport, cache: { path: tmpdir })
+```

data/docs/src/pages/index.md

@@ -0,0 +1,82 @@
+---
+layout: ../layouts/Docs.astro
+---
+
+# Lutaml::Store
+
+Store-centric database-style API for [Lutaml::Model](https://github.com/lutaml/lutaml-model) objects, with model registry, polymorphic support, composite relationships, and multiple storage backends.
+
+It also provides **read-only Sources** for working with data repositories that live outside your process — GitHub Pages, any static host, a REST API, a local package directory, or a `.zip` distribution — through one facade that never guesses which layer answered.
+
+## What it does
+
+| Layer | Role |
+|---|---|
+| **Repository** | The uniform facade: one read API over any source (cloud, local, zip), explicit cache, online/offline modes |
+| **Source** | Read-only views: `Directory`, `Zip`, `Https` (ETag/304), `Rest` (the lutaml cloud store API) |
+| **Manifest** | The enumeration SSOT: keys, sha256 digests, opaque shard metadata |
+| **Mirror** | Pull any source into a GCR-style package; pack to a distributable `.zip` |
+| **DatabaseStore** | High-level CRUD with model registry, polymorphism, composites |
+| **CacheStore** | TTL-aware cache with LRU eviction |
+| **HttpCache** | HTTP-aware caching with ETags, conditional requests, Cache-Control |
+| **PackageStore** | Structured multi-model packages with ZIP and directory transports |
+
+## Installation
+
+```ruby
+gem "lutaml-store"
+```
+
+For SQLite backend support, also add:
+
+```ruby
+gem "sqlite3"
+```
+
+## Quick start: read a LutaML data repository
+
+The same read API whether the data lives in the cloud, a local package, or a downloaded `.zip`:
+
+```ruby
+require "lutaml/store"
+
+# From a cloud API (api.relaton.org is the reference implementation)
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:rest,
+    base_url: "https://api.relaton.org", collection: "ietf"),
+  cache: Lutaml::Store::Source.for(:directory, path: "~/.cache/relaton/ietf"),
+)
+
+# From a local GCR-style package — identical read API
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:directory, path: "~/gcr/relaton/ietf"),
+)
+
+# From a downloaded .zip distribution — same API
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:zip, path: "ietf.zip"),
+)
+```
+
+## Read, search, pull
+
+```ruby
+repo.read("RFC 7231")           # bytes
+repo.exist?("RFC 7231")         # true
+repo.keys                       # all keys
+repo.manifest                   # Lutaml::Store::Manifest
+
+repo.get("RFC 7231", MyModel)   # typed model — declared, never inferred
+repo.search(docid: "RFC 7231")  # metadata filter
+
+repo.pull!(into: "~/gcr/relaton", collection: "ietf")  # GCR-style package
+```
+
+## Offline mode
+
+```ruby
+offline = Lutaml::Store::Repository.new(
+  source: repo.source, cache: repo.cache, mode: :offline
+)
+offline.read("RFC 7231")  # served from the local package, no network
+```

data/docs/src/pages/quick-start.md

@@ -0,0 +1,109 @@
+---
+layout: ../layouts/Docs.astro
+---
+
+# Quick Start
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'lutaml-store'
+```
+
+And then execute:
+
+```sh
+$ bundle install
+```
+
+Or install it yourself as:
+
+```sh
+$ gem install lutaml-store
+```
+
+For SQLite backend support, also add:
+
+```ruby
+gem 'sqlite3'
+```
+
+## Define your models
+
+```ruby
+require 'lutaml/model'
+require 'lutaml/store'
+
+class Studio < Lutaml::Model::Serializable
+  attribute :studio_key, :string
+  attribute :name, :string
+  attribute :location, :string
+end
+
+class PotteryClass < Lutaml::Model::Serializable
+  attribute :studio, Studio
+  attribute :class_id, :string
+  attribute :description, :string
+end
+
+class Enrollment < Lutaml::Model::Serializable
+  attribute :pottery_class, PotteryClass
+  attribute :student_name, :string
+end
+```
+
+## DatabaseStore: CRUD with model registry
+
+```ruby
+store = Lutaml::Store.new(
+  adapter: { type: :filesystem, location: "./data" },
+  models: [
+    { model: Studio, key: :studio_key },
+    { model: PotteryClass, key: :class_id },
+    { model: Enrollment, key: :student_name }
+  ]
+)
+
+# Save a model
+store.save(Studio.new(studio_key: "st-001", name: "Riverside Pottery", location: "123 River St"))
+
+# Fetch by key
+studio = store.fetch(model: Studio, studio_key: "st-001")
+
+# Update with dot-notation
+store.update(model: Studio, studio_key: "st-001") do |s|
+  s.name = "Riverside Pottery Studio"
+end
+
+# Delete
+store.destroy(model: Studio, studio_key: "st-001")
+```
+
+## CacheStore: TTL-aware caching
+
+```ruby
+cache = Lutaml::Store::CacheStore.new(
+  adapter: { type: :memory },
+  default_ttl: 3600,     # 1 hour
+  max_size: 1000         # LRU eviction
+)
+
+cache.set("key", "value", ttl: 1800)
+cache.get("key")          # "value" (within TTL)
+cache.exists?("key")      # true
+cache.fetch("key") { expensive_computation }  # block on miss
+```
+
+## PackageStore: multi-model packages
+
+```ruby
+package = Lutaml::Store::PackageStore.new(definition)
+package.add_model(Studio.new(studio_key: "st-001", name: "Riverside"))
+package.save("./my-package", transport: :directory)
+
+# Load it back
+loaded = Lutaml::Store::PackageStore.load(definition, "./my-package")
+loaded.fetch_model(Studio, "st-001")
+```

data/docs/src/pages/sources.md

@@ -0,0 +1,83 @@
+---
+layout: ../layouts/Docs.astro
+---
+
+# Sources
+
+A **Source** is a read-only view over a LutaML data repository. It answers four questions:
+
+- Does a key exist? (`exist?`)
+- What are its bytes? (`read`)
+- Which keys exist? (`keys`, `each_key`)
+- What does the repository's manifest declare? (`manifest`)
+
+Sources **never write**. Selection is explicit — `Source.for(type, options)` raises `ConfigurationError` for a missing option; there is no discovery, no fallback, and no magic.
+
+## Available sources
+
+| Type | What | Key options |
+|---|---|---|
+| `:directory` | A local package directory (`manifest.json` + `entries/`) | `path:` |
+| `:zip` | A `.zip` package read in place | `path:` |
+| `:https` | Any static HTTP host (Pages, raw, buckets) | `base_url:` |
+| `:rest` | The lutaml cloud store API | `base_url:`, `collection:` |
+
+## Error contract
+
+A definitive miss raises `NotFoundError`; transport trouble raises `BackendError`. Consumers can distinguish "absent" from "network broke":
+
+```ruby
+begin
+  source.read("RFC 9999")
+rescue Lutaml::Store::NotFoundError
+  # definitively not in the repository
+rescue Lutaml::Store::BackendError
+  # network/service failure — retryable
+end
+```
+
+## Usage
+
+```ruby
+# Directory source
+src = Lutaml::Store::Source.for(:directory, path: "~/gcr/relaton/ietf")
+
+# Zip source
+src = Lutaml::Store::Source.for(:zip, path: "ietf-distribution.zip")
+
+# HTTPS source (any static host)
+src = Lutaml::Store::Source.for(:https,
+  base_url: "https://raw.githubusercontent.com/relaton/relaton-data-ietf/main/data/")
+
+# REST source (the lutaml cloud store contract)
+src = Lutaml::Store::Source.for(:rest,
+  base_url: "https://api.relaton.org", collection: "ietf")
+
+# Common operations
+src.keys                      # => ["RFC 7231", "RFC 3986", ...]
+src.read("RFC 7231")          # => bytes
+src.exist?("RFC 7231")        # => true
+src.get("RFC 7231", MyModel)  # => typed model (model class declared)
+src.search(docid: "RFC 7231") # => matching manifest entries
+```
+
+## The Https transport
+
+The `:https` source supports:
+- **ETag/304 revalidation** via an optional `cache:` config (`HttpCacheConfig` hash)
+- **Injected transport** (`transport:` callable) for tests and alternate HTTP stacks
+- **Redirect budget** (`max_redirects:` default 3)
+- **Auth headers** (`headers:`) for private repositories
+
+## The Rest contract
+
+The `:rest` source implements the lutaml cloud store API contract:
+
+```
+GET {base}/collections                      → collection list
+GET {base}/collections/{c}/manifest         → Manifest
+GET {base}/collections/{c}/entries/{key}    → record
+GET {base}/collections/{c}/shards/{n}       → shard keys (optional)
+```
+
+404 is a definitive "no such key/collection"; every other non-2xx is a `BackendError`.

data/docs/src/pages/stores.md

@@ -0,0 +1,110 @@
+---
+layout: ../layouts/Docs.astro
+title: Stores
+---
+
+# Stores
+
+## DatabaseStore — typed CRUD
+
+```ruby
+store = Lutaml::Store.new(
+  adapter: { type: :filesystem, location: "./data" },
+  models: [
+    { model: Studio, key: :studio_key },
+    { model: PotteryClass, key: :class_id },
+    { model: Enrollment, key: :student_name }
+  ]
+)
+
+store.save(Studio.new(studio_key: "st-001", name: "Riverside Pottery"))
+studio = store.fetch(model: Studio, studio_key: "st-001")
+
+store.update(model: Studio, studio_key: "st-001") do |s|
+  s.name = "Riverside Pottery Studio"
+end
+
+store.destroy(model: Studio, studio_key: "st-001")
+```
+
+### Polymorphic models
+
+Registering multiple models against the same key attribute enables polymorphic storage: fetching returns whichever registered model was stored under the key. Configure per-registration:
+
+```ruby
+models: [
+  { model: Person, key: :email },
+  { model: Organization, key: :email, polymorphic: true }
+]
+```
+
+### Composite models
+
+Nested registered models are stored independently; `CompositeModelHandler` restores object references on fetch. A `PotteryClass` that embeds a `Studio` keeps the studio in its own key space — updating the studio once updates every referencing object's view of it.
+
+## BasicStore — key-value layer
+
+```ruby
+basic = Lutaml::Store::BasicStore.new(
+  adapter: { type: :memory },
+  cache: { type: :memory, ttl: 60 },
+  monitor: my_monitor,          # optional
+  events: [my_subscriber]       # optional
+)
+
+basic.set("key", "value")
+basic.get("key")        # => "value"
+basic.exists?("key")    # => true
+basic.keys              # => ["key"]
+basic.all               # => { "key" => "value" }
+basic.delete("key")
+```
+
+## CacheStore — TTL and LRU
+
+```ruby
+cache = Lutaml::Store::CacheStore.new(
+  adapter: { type: :memory },
+  default_ttl: 3600,
+  max_size: 1000
+)
+
+cache.set("key", "value", ttl: 1800)
+cache.get("key")                          # "value" within TTL, nil after
+cache.fetch("key") { expensive_call }     # block runs only on miss
+```
+
+Entries past their TTL are evicted lazily on access; when `max_size` is exceeded the least-recently-used entries are evicted first.
+
+## PackageStore — multi-model packages
+
+```ruby
+definition = Lutaml::Store::PackageDefinition.new do |d|
+  d.model Studio
+  d.model PotteryClass
+  d.metadata version: "1.0"
+end
+
+package = Lutaml::Store::PackageStore.new(definition)
+package.add_model(Studio.new(studio_key: "st-001", name: "Riverside"))
+package.save("./my-package", transport: :directory)
+package.save("./my-package.zip", transport: :zip)
+
+loaded = Lutaml::Store::PackageStore.load(definition, "./my-package")
+loaded.fetch_model(Studio, "st-001")
+```
+
+`PackageDefinition` declares which models, assets, and metadata a package contains; `DirectoryTransport` and `ZipTransport` handle reading and writing. Formats are chosen per model entry (see [Formats](/lutaml-store/formats/)).
+
+## FormatSerializer — store in any format
+
+`FormatSerializer` wraps a Format handler to implement the serialize/deserialize interface, letting `DatabaseStore` persist entries as YAMLS, XML, Marshal, etc. instead of the default hash serialization:
+
+```ruby
+store = Lutaml::Store.new(
+  adapter: { type: :filesystem, location: "./data" },
+  models: [{ model: GlossaryTerm, key: :id, format: :yamls }]
+)
+```
+
+This is the pattern used for Glossarist-style multi-document YAML files.

data/docs/src/styles/global.css

@@ -0,0 +1,10 @@
+@import "tailwindcss";
+@plugin "@tailwindcss/typography";
+
+.light-only { display: block; }
+.dark-only { display: none; }
+
+@media (prefers-color-scheme: dark) {
+  .light-only { display: none; }
+  .dark-only { display: block; }
+}

data/lib/lutaml/store/format.rb

@@ -31,6 +31,19 @@ module Lutaml
         const_get(entry).new
       end
 
+      # Self-describing content: XML documents start with "<?xml" or "<",
+      # JSON with "{" or "["; everything else is YAML, the ecosystem's
+      # default. Extensionless records (object-storage keys carry no
+      # extension) declare their format this way instead of by guesswork
+      # out of band.
+      def self.guess(data)
+        stripped = data.to_s.lstrip
+        return :xml if stripped.start_with?("<?xml", "<")
+        return :json if stripped.start_with?("{", "[")
+
+        :yaml
+      end
+
       def self.for_extension(ext)
         extension_map[ext] || extension_map[".#{ext.to_s.sub(/\A\./, "")}"]
       end

data/lib/lutaml/store/manifest.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+require "lutaml/model"
+require "digest"
+
+module Lutaml
+  module Store
+    # The enumeration document of a LutaML data repository: which keys
+    # exist, where each one lives relative to the source root, and the
+    # sha256 digest to verify it against.
+    #
+    # Sharding metadata (`shards`, `Entry#shard`) is carried but never
+    # interpreted: computing "which shard holds this key" is domain logic
+    # (e.g. relaton's crc32 over the pubid root number). The store only
+    # enumerates.
+    class Manifest
+      include Lutaml::Model::Serialize
+
+      VERSION = 1
+
+      class Entry
+        include Lutaml::Model::Serialize
+
+        attribute :key, :string
+        attribute :location, :string
+        attribute :digest, :string
+        attribute :shard, :integer
+        attribute :metadata, :hash, default: {}
+
+        def digest_for(body)
+          "sha256:#{Digest::SHA256.hexdigest(body)}"
+        end
+
+        def matches?(body)
+          return true unless digest
+
+          digest == digest_for(body)
+        end
+      end
+
+      attribute :version, :integer, default: VERSION
+      attribute :generated, :string
+      attribute :count, :integer, default: 0
+      attribute :shards, :integer, default: 0
+      attribute :entries, Manifest::Entry, collection: true, initialize_empty: true
+
+      # @param text [String]
+      # @param format [Symbol] :json or :yaml
+      # @return [Manifest]
+      def self.parse(text, format: :json)
+        case format.to_sym
+        when :json then from_json(text)
+        when :yaml then from_yaml(text)
+        else raise ConfigurationError, "unsupported manifest format: #{format}"
+        end
+      end
+
+      def self.build(entries, generated: nil, shards: 0, version: VERSION)
+        new(
+          version: version,
+          generated: (generated || Time.now.utc).iso8601,
+          count: entries.size,
+          shards: shards,
+          entries: entries
+        )
+      end
+
+      def keys
+        entries.map(&:key)
+      end
+
+      def entry_for(key)
+        entries.find { |e| e.key == key }
+      end
+
+      def key?(key)
+        !entry_for(key).nil?
+      end
+
+      def shard_of_declared?
+        shards.positive?
+      end
+    end
+  end
+end