seamless-database 2.0.2__tar.gz → 2.3.0__tar.gz

seamless_database-2.3.0/LICENSE.txt

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Author: Sjoerd de Vries, MBI platform.
+Copyright (c) 2023-2026 CNRS. 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

{seamless_database-2.0.2 → seamless_database-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: seamless-database
-Version: 2.0.2
+Version: 2.3.0
 Summary: SQLite-backed metadata database service for Seamless
 Author: Sjoerd de Vries
 License-Expression: MIT
@@ -18,8 +18,10 @@ Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Software Development :: Libraries
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
+License-File: LICENSE.txt
 Requires-Dist: aiohttp
 Requires-Dist: peewee
+Dynamic: license-file
 
 # seamless-database
 
@@ -35,13 +37,60 @@ The database stores the following kinds of records:
 |-------|---------|
 | **Transformation** | Maps a transformation checksum to its result checksum |
 | **RevTransformation** | Reverse lookup: finds which transformations produced a given result |
-| **BufferInfo** | Stores buffer metadata (length, dtype, encoding, etc.) for a checksum |
+| **HashType** | Stores the packed checksum classification word used by new HashType validation and conversion checks |
 | **SyntacticToSemantic** | Maps between syntactic and semantic checksums per celltype |
-| **Expression** | Caches expression evaluation results (input checksum + path + celltype → result checksum) |
-| **MetaData** | Stores execution metadata for transformations (executor, environment, timing) |
-| **IrreproducibleTransformation** | Records transformations whose results are not reproducible |
+| **Expression** | Caches expression evaluation results keyed by `(input_checksum, path, input_celltype, celltype)` |
+| **MetaData** | Stores a canonical execution record for each successful, non-probe transformation |
+| **IrreproducibleTransformation** | Records transformations whose results are not reproducible; metadata is preserved on migration |
 
-All data is persisted in a single SQLite file (typically `seamless.db`).
+All data is persisted in a single SQLite file (typically `seamless.db`). The current protocol version is **2.3**.
+
+## Expression cache schema
+
+Expression `input_celltype` describes the input checksum's interpretation;
+`celltype` describes the produced result. Both are part of the composite cache
+key. Database, jobserver, remote client, and Dask payloads use those same names.
+The unreleased old `celltype`/`target_celltype` keys are not accepted aliases.
+Existing development caches need these statements, in order, or a recreated
+expression cache table:
+
+```sql
+ALTER TABLE expression RENAME COLUMN celltype TO input_celltype;
+ALTER TABLE expression RENAME COLUMN target_celltype TO celltype;
+```
+
+## Execution records
+
+`MetaData` stores one canonical execution record per successful, non-probe transformation, keyed by `tf_checksum`. Records are **write-once**: subsequent calls to a recorded transformation hit the normal `Transformation` cache and do not re-enter the record path.
+
+Two record sizes coexist under the same schema:
+
+- **Minimal record (default)** — `schema_version`, `tf_checksum`, `result_checksum`, `seamless_version`, `execution_mode`, `remote_target`, `wall_time_seconds`, `cpu_time_user_seconds`, `cpu_time_system_seconds`, `memory_peak_bytes`, `gpu_memory_peak_bytes`. The hot path pays only timing/memory capture and one write.
+- **Full record (`record: true` in `seamless.profile.yaml`)** — adds environment fingerprints (content-addressed `node`, `environment`, `queue` sub-buffer checksums), compilation context, validation snapshots, contract-violation lists, execution envelope (requested cluster/queue/node, scratch/fingertip flags, resolved `__env__`), and per-job freshness/retry/worker fields.
+
+The validator on `PUT metadata` checks identity only — record syntax (integer `schema_version`, body `tf_checksum`/`result_checksum` matching the request, sane `checksum_fields` if present) — not the full payload schema. Identical duplicates are idempotent successes; differing bodies are rejected. Once `IrreproducibleTransformation` rows exist for a `tf_checksum`, `PUT metadata` for that checksum is rejected to avoid silently migrating it back into the normal cache.
+
+When a normal entry moves to `IrreproducibleTransformation`, its metadata body travels with it unchanged.
+
+### Schema upgrade from legacy
+
+The legacy `meta_data` table had two columns (`checksum PRIMARY KEY`, `metadata JSON`). On startup, `seamless-database`:
+
+- creates the upgraded table fresh if absent,
+- drops and recreates the legacy table if it is empty,
+- preserves an already-upgraded table on subsequent starts,
+- and **fails loudly** if a non-empty legacy table is present (it must be migrated explicitly).
+
+### Endpoints (request types)
+
+- `PUT metadata` — atomically creates missing `Transformation` and `RevTransformation` rows alongside the `MetaData` row. Body: `{type: "metadata", checksum: <tf>, result: <result>, value: <record>}`.
+- `GET metadata` — return the canonical record body for a `tf_checksum`.
+- `GET irreproducible` — return all rows for a `tf_checksum`, optionally filtered by `result`. Each row includes `checksum`, `result`, and `metadata`.
+- `PUT irreproducible` — move a normal entry into `IrreproducibleTransformation`, preserving metadata.
+
+The remote-client equivalents are `set_execution_record`, `get_execution_record`, and `get_irreproducible_records` in `seamless-remote`.
+
+See the [agent contract](https://github.com/sjdv1982/seamless/blob/main/docs/agent/contracts/execution-records.md) for the full behavioral spec.
 
 ## Role in the Seamless ecosystem
 

{seamless_database-2.0.2 → seamless_database-2.3.0}/README.md

@@ -12,13 +12,60 @@ The database stores the following kinds of records:
 |-------|---------|
 | **Transformation** | Maps a transformation checksum to its result checksum |
 | **RevTransformation** | Reverse lookup: finds which transformations produced a given result |
-| **BufferInfo** | Stores buffer metadata (length, dtype, encoding, etc.) for a checksum |
+| **HashType** | Stores the packed checksum classification word used by new HashType validation and conversion checks |
 | **SyntacticToSemantic** | Maps between syntactic and semantic checksums per celltype |
-| **Expression** | Caches expression evaluation results (input checksum + path + celltype → result checksum) |
-| **MetaData** | Stores execution metadata for transformations (executor, environment, timing) |
-| **IrreproducibleTransformation** | Records transformations whose results are not reproducible |
+| **Expression** | Caches expression evaluation results keyed by `(input_checksum, path, input_celltype, celltype)` |
+| **MetaData** | Stores a canonical execution record for each successful, non-probe transformation |
+| **IrreproducibleTransformation** | Records transformations whose results are not reproducible; metadata is preserved on migration |
 
-All data is persisted in a single SQLite file (typically `seamless.db`).
+All data is persisted in a single SQLite file (typically `seamless.db`). The current protocol version is **2.3**.
+
+## Expression cache schema
+
+Expression `input_celltype` describes the input checksum's interpretation;
+`celltype` describes the produced result. Both are part of the composite cache
+key. Database, jobserver, remote client, and Dask payloads use those same names.
+The unreleased old `celltype`/`target_celltype` keys are not accepted aliases.
+Existing development caches need these statements, in order, or a recreated
+expression cache table:
+
+```sql
+ALTER TABLE expression RENAME COLUMN celltype TO input_celltype;
+ALTER TABLE expression RENAME COLUMN target_celltype TO celltype;
+```
+
+## Execution records
+
+`MetaData` stores one canonical execution record per successful, non-probe transformation, keyed by `tf_checksum`. Records are **write-once**: subsequent calls to a recorded transformation hit the normal `Transformation` cache and do not re-enter the record path.
+
+Two record sizes coexist under the same schema:
+
+- **Minimal record (default)** — `schema_version`, `tf_checksum`, `result_checksum`, `seamless_version`, `execution_mode`, `remote_target`, `wall_time_seconds`, `cpu_time_user_seconds`, `cpu_time_system_seconds`, `memory_peak_bytes`, `gpu_memory_peak_bytes`. The hot path pays only timing/memory capture and one write.
+- **Full record (`record: true` in `seamless.profile.yaml`)** — adds environment fingerprints (content-addressed `node`, `environment`, `queue` sub-buffer checksums), compilation context, validation snapshots, contract-violation lists, execution envelope (requested cluster/queue/node, scratch/fingertip flags, resolved `__env__`), and per-job freshness/retry/worker fields.
+
+The validator on `PUT metadata` checks identity only — record syntax (integer `schema_version`, body `tf_checksum`/`result_checksum` matching the request, sane `checksum_fields` if present) — not the full payload schema. Identical duplicates are idempotent successes; differing bodies are rejected. Once `IrreproducibleTransformation` rows exist for a `tf_checksum`, `PUT metadata` for that checksum is rejected to avoid silently migrating it back into the normal cache.
+
+When a normal entry moves to `IrreproducibleTransformation`, its metadata body travels with it unchanged.
+
+### Schema upgrade from legacy
+
+The legacy `meta_data` table had two columns (`checksum PRIMARY KEY`, `metadata JSON`). On startup, `seamless-database`:
+
+- creates the upgraded table fresh if absent,
+- drops and recreates the legacy table if it is empty,
+- preserves an already-upgraded table on subsequent starts,
+- and **fails loudly** if a non-empty legacy table is present (it must be migrated explicitly).
+
+### Endpoints (request types)
+
+- `PUT metadata` — atomically creates missing `Transformation` and `RevTransformation` rows alongside the `MetaData` row. Body: `{type: "metadata", checksum: <tf>, result: <result>, value: <record>}`.
+- `GET metadata` — return the canonical record body for a `tf_checksum`.
+- `GET irreproducible` — return all rows for a `tf_checksum`, optionally filtered by `result`. Each row includes `checksum`, `result`, and `metadata`.
+- `PUT irreproducible` — move a normal entry into `IrreproducibleTransformation`, preserving metadata.
+
+The remote-client equivalents are `set_execution_record`, `get_execution_record`, and `get_irreproducible_records` in `seamless-remote`.
+
+See the [agent contract](https://github.com/sjdv1982/seamless/blob/main/docs/agent/contracts/execution-records.md) for the full behavioral spec.
 
 ## Role in the Seamless ecosystem