lutaml-store 0.2.4 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b6093f17ea0bca9240b608835a29b8f060a0c8c41be950e93fc0713e8c1b9444
-  data.tar.gz: f0f9ee8ace22fa22ec62408425ba7fa479648c57158c92967bc8e235fd037285
+  metadata.gz: 4481af10b7c80db97bffbe2e1ad5119d615370596c46c88c84e536411d1ffdda
+  data.tar.gz: 618ad367561533d0aa576ef713dfb384173e34ec1ec0047e3cbab400af9c4dce
 SHA512:
-  metadata.gz: 97a8aa5dc0d52617b1f4d97bd8378664eead01e299410d8dd506b775be23d69616008cde2e053b6a649819726048afdd2b227a455979630b2f07d04e76deed6d
-  data.tar.gz: b80834cd232f87d378c0a3a5f7ea17ef2c7ea2d68103d890756eabb2f2c85e20e1b1f193ec64cd0198fdb4235c0ff68ca6de87e788b7aabaa26b49a081e2023e
+  metadata.gz: 2e22cb07d0781f6aba2c2ebc043322e5a756519284740b74661906b6f00cbce8536291967868b86fd2a1908de46e6c1d4450f35c517d4919f73020d65890dc63
+  data.tar.gz: 26f2ef6b87a85e2c020571319640ceed71955ace41c97c2052857b97bd8d68e599cf466cd349f31c802a0fda7172cd857ce1bfbed952076b7425aea07c7d2de0

data/CLAUDE.md

@@ -49,6 +49,12 @@ All inherit from `Format::Base`. Six formats: `Yaml`, `Yamls`, `Json`, `Jsonl`,
 
 All inherit from `Adapter::Base`. Three backends: `Memory`, `FileSystem`, `SQLite`. Registered in `Adapter` module and resolved via `Adapter.resolve(:type, options)`. New adapters can be added with `Adapter.register(:custom, CustomClass)` without modifying existing code (OCP).
 
+`Base#each_key` defaults to `keys.each`; `Base#update(key) { |old| new }` defaults to get + set inside `transaction`. Memory, FileSystem and Sqlite override `update` to be atomic. `FileSystem` percent-encodes file names (digest + key in `.meta` for long names), writes non-String values as JSON (`format: "json"` in `.meta`), and locks `<root>/.lock` (`with_lock(:ex/:sh)`, re-entrant per thread; a shared lock cannot be upgraded).
+
+Inside `Lutaml::Store`, a bare `Monitor` resolves to `Lutaml::Store::Monitor` (the stats collector) — write `::Monitor` for Ruby's re-entrant lock.
+
+Specs that load `sqlite3` must sort after `autoload_spec.rb` (see `sqlite_scale_spec.rb`). Run specs with a UTF-8 locale (`LANG=C.UTF-8`); the anti-pattern guard fails under US-ASCII.
+
 ### PackageStore and transports
 
 `PackageStore` provides structured multi-model persistence. `PackageDefinition` declares which models, assets, and metadata the package contains. Transports (`DirectoryTransport`, `ZipTransport`) handle reading/writing to disk. Format handlers determine serialization per model entry.

data/README.adoc

@@ -17,8 +17,64 @@ It offers a unified interface for storing and retrieving complex model
 hierarchies, batch file I/O with multiple serialization formats, HTTP-aware
 caching, and package-based persistence with ZIP and directory transports.
 
+It also provides **read-only Sources** for working with data repositories that
+live outside your process — GitHub Pages, any static host, a REST API, a local
+package directory, or a `.zip` distribution — through one facade that never
+guesses which layer answered.
+
+== Quick start: read a LutaML data repository
+
+The same read API whether the data lives in the cloud, a local package, or a
+downloaded `.zip` — backends are configured, never guessed:
+
+[source,ruby]
+----
+require "lutaml/store"
+
+# From a cloud API (api.relaton.org is the reference implementation)
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:rest,
+    base_url: "https://api.relaton.org", collection: "ietf"),
+  cache: Lutaml::Store::Source.for(:directory, path: "~/.cache/relaton/ietf"),
+)
+
+# From a local GCR-style package — identical read API
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:directory, path: "~/gcr/relaton/ietf"),
+)
+
+# From a downloaded .zip distribution — same API
+repo = Lutaml::Store::Repository.new(
+  source: Lutaml::Store::Source.for(:zip, path: "ietf.zip"),
+)
+
+# Read
+repo.read("RFC 7231")        # => bytes
+repo.exist?("RFC 7231")      # => true
+repo.keys                     # => ["RFC 7231", ...]
+repo.manifest                 # => Lutaml::Store::Manifest
+
+# Parse into your model — the model class is declared, never inferred
+repo.get("RFC 7231", MyModel)
+
+# Search by metadata fields
+repo.search(docid: "RFC 7231")
+
+# Pull the whole collection into a local GCR-style package
+repo.pull!(into: "~/gcr/relaton", collection: "ietf")
+
+# Offline mode: never touch the source, serve from the local package only
+offline = Lutaml::Store::Repository.new(source: repo.source, cache: repo.cache, mode: :offline)
+offline.read("RFC 7231")     # served from the cache, no network
+
+
+
 == Features
 
+* **Repository** — the uniform facade: one read API over any source (cloud, local, zip), explicit cache, online/offline modes
+* **Source** (read-only) — Directory, Zip, Https (ETag/304), Rest (the lutaml cloud store API); 404 = NotFoundError, transport = BackendError
+* **Manifest** — the enumeration SSOT: keys, sha256 digests, opaque shard metadata
+* **Mirror** — pull any source into a GCR-style package (incremental, digest-verified); pack to a distributable .zip
 * **DatabaseStore** — high-level CRUD with model registry, polymorphism, composites
 * **PackageStore** — structured multi-model packages with ZIP and directory transport
 * **BasicStore** — low-level key-value store with optional cache, events, monitoring
@@ -172,7 +228,21 @@ with optional caching, monitoring, and event emission.
 === Storage adapters
 
 All adapters inherit from `Adapter::Base` and provide `get`, `set`, `delete`,
-`exists?`, `keys`, `all`, `clear`, `size`, `each_key`, and bulk operations.
+`exists?`, `keys`, `all`, `clear`, `size`, `each_key`, `update`, and bulk
+operations. `each_key` defaults to `keys.each`, so a custom adapter only needs
+`keys`.
+
+`update(key) { |old| new }` is a read-modify-write: the block gets the current
+value (`nil` when the key is missing) and its result is stored. It is atomic
+across threads for Memory, and across threads and processes for FileSystem and
+SQLite. `BasicStore#update` calls it and refreshes the read cache.
+
+[source,ruby]
+----
+store = Lutaml::Store::BasicStore.new(adapter_type: :filesystem,
+                                      adapter_options: { path: "./data" })
+store.update("counter") { |n| (n.to_i + 1).to_s }
+----
 
 [cols="1,1,3"]
 |===
@@ -585,6 +655,21 @@ store = Lutaml::Store.new(
 Persistent file-based storage with SHA-256 integrity checks. Files organized
 by key in subdirectories.
 
+* *File names.* Every byte outside `[A-Za-z0-9._-]` is percent-encoded, so
+  `ISO/19115` and `ISO:19115` get different files and `keys` returns the
+  original keys. A leading `.` is encoded too, so no file is hidden or outside
+  the root. A name longer than 200 bytes becomes `~` plus the SHA-256 of the
+  key; the `.meta` file beside it keeps the key. Files written by 0.3.0 (with
+  `_` in place of unsafe characters) are still read, and move to the new name
+  on the next write of that key.
+* *Values.* A `String` is written as is. Any other value (for example the
+  Hash that `DatabaseStore` saves) is written as JSON and read back as JSON.
+* *Locks.* Writes hold an exclusive `flock` on `<path>/.lock` and reads hold a
+  shared one. Data and `.meta` files are written to a unique temp file and
+  renamed into place, so readers never see a partial write.
+* *Case-insensitive file systems* (the macOS default) can still merge keys that
+  differ only in letter case.
+
 === SQLite
 
 [source,ruby]
@@ -625,8 +710,20 @@ Error hierarchy:
 
 == Thread safety
 
-All adapters use mutex-based synchronization. Safe for concurrent use across
-threads.
+* `Adapter::Memory`: writes are synchronized with a Monitor; reads use a
+  frozen snapshot.
+* `Adapter::FileSystem`: safe across threads and processes that share one
+  directory (see the FileSystem section above). `transaction { ... }` holds the exclusive
+  lock for the whole block.
+* `Adapter::Sqlite`: SQLite locking with a busy timeout; `update` uses
+  `BEGIN IMMEDIATE`.
+* `CacheStore`: all operations run under one lock per instance. The TTL and
+  LRU bookkeeping is per process.
+* `BasicStore`: the optional read cache is per process. Disable it
+  (`cache: { enabled: false }`) when other processes write the same store.
+
+A `get` followed by a `set` is never atomic. Use `update` for a
+read-modify-write.
 
 == Development
 

data/docs/astro.config.mjs

@@ -0,0 +1,12 @@
+import { defineConfig } from "astro/config";
+import vue from "@astrojs/vue";
+import tailwindcss from "@tailwindcss/vite";
+
+export default defineConfig({
+  site: "https://lutaml.github.io",
+  base: "/lutaml-store",
+  integrations: [vue()],
+  vite: {
+    plugins: [tailwindcss()],
+  },
+});