loom-kernel 8.0.0__tar.gz → 9.0.0__tar.gz

{loom_kernel-8.0.0 → loom_kernel-9.0.0}/CHANGELOG.md

@@ -412,7 +412,7 @@
   selection can be checked without running it. A bare string as `include`
   raises `TypeError`. The runner now filters through it. `InvalidStageError`
   is exported from `loom.etl.compiler` as well; it is the same class as
-  `loom.etl.runner.InvalidStageError` and `loom.etl.InvalidStageError`.
+  `loom.etl.InvalidStageError`.
 - **etl:** `iter_processes`, `iter_steps_in_process` and `iter_all_steps` are
   exported from `loom.etl.compiler`. They flatten a compiled plan in
   execution order, parallel groups in declaration order, so code that walks
@@ -459,6 +459,78 @@
 
 ### core
 
+- **core:** `RuleViolation` and `RuleViolations` are no longer exported by
+  `loom.core.use_case` or `loom.core.use_case.rule`. Import them from
+  `loom.core.errors`, their only public home.
+- **core:** wrapping a repository in `CachedRepository` raises `RuntimeError`
+  when one of its `@cache_query` reads has a missing return annotation, one
+  the defining module cannot resolve, or one outside the supported grammar,
+  naming the repository and the method; a `@cached` repository therefore fails
+  when it is built. Before, wrapping emitted a `DeprecationWarning` and logged
+  `CacheQueryReturnTypeUnusable`, and the cached call returned the decoded
+  payload instead of the declared type. Annotate the read's return type with a
+  struct, a scalar, or a `list`, `tuple` or optional of those. A
+  `@cache_query` coroutine that is not defined on the repository class, such
+  as one set on the instance, raises `RuntimeError` when it is looked up on
+  the wrapper, since no codec was built for it; define it as a method of the
+  class.
+- **core:** a field annotated `X | UnsetType` infers `X`'s column type.
+  `UNSET` means "not provided", not another type, so `str | UnsetType` now
+  compiles to `String` and `int | UnsetType` to `Integer` instead of `JSON`,
+  and `Annotated[str, String(20)] | UnsetType` reads its declared type. Such
+  a column is scalar: a REST route may declare it filterable, `like` and
+  `ilike` included for text, and sortable by offset; a cursor sort on it is
+  still refused, since a cursor key must be a bare type. Nullability is
+  unchanged: `str | None | UnsetType` admits `None` and is nullable,
+  `UnsetType` alone keeps the column `NOT NULL`. An undeclared
+  `Decimal | UnsetType`, plain `Enum | UnsetType` or `Literal[...] | UnsetType`
+  now needs a declared column type, as it does without `UnsetType`. On Mongo,
+  a `str | UnsetType` or `UUID | UnsetType` primary key is now accepted, and
+  a range filter or sort on a `Decimal | UnsetType` field is refused, as on a
+  `Decimal` field. Alembic autogenerate will show the type change from `JSON`
+  for existing tables with such columns.
+- **core:** a column's nullability follows its field's annotation. A field
+  whose annotation admits `None` (`x: int | None = None`, also inside
+  `Annotated`) compiles to a nullable column; any other field compiles to
+  `NOT NULL`, and a primary key column is `NOT NULL` whatever its annotation.
+  Before, every column was `NOT NULL` unless declared otherwise, so the model
+  accepted `None` and the database refused it on insert. `nullable=` on
+  `ColumnField`, `ScopedField` or `Field` now defaults to `None` (follow the
+  annotation) and a declared `True` or `False` still wins. Alembic
+  autogenerate will propose `nullable` changes for existing tables whose
+  Optional fields were created `NOT NULL`; to keep such a column `NOT NULL`,
+  declare `nullable=False`. ETL schema contracts keep their own rule (a
+  column is nullable unless declared otherwise), which this change does not
+  touch.
+- **core:** a `uuid.UUID` field without a declared column type is now a
+  `Uuid` column (`loom.core.model.Uuid`, SQLAlchemy `Uuid`) instead of
+  `JSON`: `UUID` on Postgres, `CHAR(32)` holding the hex form on dialects
+  without a native UUID type, such as SQLite. Such a field is now a scalar
+  column, so REST `filterable`, `sortable` and `ALL` accept it on every
+  backend, `mongo` included; on SQLite a cursor walk sorted by it pages in
+  UUID order and an equality filter on it matches. Streaming `IntoTable` with the
+  SQLAlchemy backend binds a `UUID` field the same way instead of failing to
+  serialise it as JSON. Migration: an existing table with such a column needs
+  a revision changing its type; Alembic autogenerate reports it as a type
+  change from `JSON` to `Uuid`. Postgres does not cast `json` to `uuid`, so
+  the revision needs `postgresql_using="(<column> #>> '{}')::uuid"`. On a
+  dialect without a native UUID type the old rows hold JSON text (the quoted
+  UUID with dashes), which the `Uuid` column cannot read, so after the type
+  change the revision rewrites each value to its 32-character hex. On
+  SQLite: `op.batch_alter_table` for the type change, then
+  `UPDATE <table> SET <column> = replace(json_extract(<column>, '$'), '-', '')`.
+  Rewriting before the type change loses values on SQLite: a hex form made
+  only of digits is stored as a number in the `JSON` column.
+- **core:** a `Decimal` field, plain or optional, declares its column type: a
+  SQLAlchemy model with an undeclared one is refused when it is compiled,
+  naming the field. loom inferred `NUMERIC(10, 2)`, and Postgres silently
+  rounds a value to that scale (`1.2345` is stored as `1.23`) and refuses one
+  past that precision. Migration: declare it, e.g.
+  `Annotated[Decimal, Numeric(precision=20, scale=6)]` or
+  `ColumnField(Numeric(precision=20, scale=6))`; `Numeric()` with no
+  precision is now an unbounded `NUMERIC` instead of `NUMERIC(10, 2)`, and
+  `Numeric(scale=...)` without a precision raises `ValueError`, as SQLAlchemy
+  would drop the scale.
 - **core:** `sql_endpoint.auth` admits only `identity` and `external`;
   `auth: jwt` is refused when the config is loaded, and the alias and its
   `DeprecationWarning` are gone. Migration: replace `auth: jwt` with
@@ -550,6 +622,81 @@
   or `RuleViolation` it returns no longer fails the rule. Migrate
   `Rule.check(F(Order).total, via=lambda v: v <= 0, message="total must be > 0")`
   to `Rule.check(F(Order).total, via=lambda v: "total must be > 0" if v <= 0 else None)`.
+- **core:** cursor signing keys are required (ADR 0004). An application whose
+  persistence backend serves repositories (`sqlalchemy`, `mongo`), and a
+  Celery worker that registers repositories, read `persistence.cursors` at
+  startup and fail with a `ConfigError` without a key:
+
+  ```yaml
+  persistence:
+    cursors:
+      current_key_id: "2026-10"
+      keys:
+        "2026-10": {env: CURSOR_KEY_2026_10}   # or {path: /run/secrets/...}
+  ```
+
+  Keys are read from a file or an environment variable, never written in the
+  configuration, as standard base64 decoding to at least 32 random bytes
+  (`openssl rand -base64 32`); a passphrase is refused. Older key ids stay in
+  `keys` to verify the tokens they signed during a rotation.
+  The default is `signed` in every environment; `signing: ephemeral` (a
+  random per-process key, for `development.yaml`) and `signing: unsigned`
+  (for `test.yaml`) are explicit opt-outs, and both log a WARNING at
+  startup. In production and staging the deploy check `loom.cursors.E001`
+  fails startup, and `loom check --deploy` fails, when a `database` or
+  `persistence.mongo` section is set and `signing` is not `signed` or
+  `current_key_id` names no entry of `keys`. No `config.overridable` entry
+  may open a key under `persistence.cursors`. A key's `env` source must not
+  name a `LOOM_*` variable. Tokens issued before this release are refused:
+  clients restart from the first page.
+- **core:** a cursor page orders only by non-null columns of a type a token
+  carries; a dotted path, a JSON, `bytes` or optional column answers
+  `400 unsupported_query` before the query runs (`require_cursor_sort`), and
+  a page whose last row holds a non-finite number or a value the token
+  cannot carry answers 400 instead of 500. An empty cursor is an invalid
+  token on SQLAlchemy as on Mongo. Tokens bind the model as
+  `module.qualname`; `UnsupportedQuery` and the client's 400 name it by its
+  qualified name alone. `CursorCodec.encode`/`decode`, `require_cursor_sort`
+  and `QuerySpecCompiler(model=...)` take the model class (or a name). Mongo
+  issues the next token in the model's types, so a page sorted by a date or
+  UUID column, or keyed by a UUID, reaches the next page.
+- **core:** one `CursorCodec` per process: the registration modules register
+  it with `register_cursor_codec`, which raises `ConfigError` when a
+  different codec is already registered, and, in a deployed environment,
+  for an `ephemeral` or `unsigned` codec unless `CursorCodec.from_context`
+  built it from a configuration silencing `loom.cursors.E001`. Without a
+  registered `Environment` the process environment decides (`LOOM_ENV`, else
+  production).
+- **core:** `loom.core.checks.silenced_checks(config)` returns the ids
+  `checks.silenced` lists, as the deploy-check runner reads them.
+- **core:** the unsigned module functions `encode_cursor` and
+  `decode_cursor` are gone. `CursorCodec` (with `CursorConfig` and
+  `CursorKeySource`, exported from `loom.core.repository.abc`) issues and
+  verifies tokens: `encode(cursor, *, model, query, scope)` and
+  `decode(token, *, backend, model, query, scope, columns, primary_key)`,
+  where `scope` is required and `""` binds a token to no tenant, `columns`
+  maps each column name to its annotation and `primary_key` names the key
+  columns in order. A sort key or primary-key value whose type is not its
+  column's is refused (`400 unsupported_query`), as is a sort field missing
+  from `columns`. The backends
+  register the codec they build in DI, so a service paginating by hand
+  resolves it with `container.resolve(CursorCodec)`. A decoded datetime key
+  is always aware. A `time` sort key is carried too, with the offset it was
+  issued with; before, a page sorted by a column holding `time` values
+  raised `TypeError` when it issued the next cursor.
+- **core:** `RepositorySQLAlchemy`, `RepositoryMongo`, `QuerySpecCompiler`
+  (with a required `scope` and `columns` too), `SQLAlchemyDefaultRepositoryBuilder`,
+  `MongoDefaultRepositoryBuilder`, `build_sqlalchemy_repository_registration_module`,
+  `build_mongo_repository_registration_module` and
+  `loom.testing.build_repository_harness` take a required keyword-only
+  `cursor_codec`. A custom repository `builder` passes
+  `context.container.resolve(CursorCodec)`. `extract_next_cursor` returns the
+  next `Cursor` position instead of a token, and `SessionManager` gains
+  `current_settings()`. SQLAlchemy repositories bind cursors to the caller's
+  `loom.scope.*` session values. `loom.core.environment` gains
+  `configured_variable(name)`, which reads a variable a configuration names
+  and refuses a `LOOM_*` name.
+
 - **core:** the `LoomType` protocol gains `from_query(params)`, which
   validates query-string values strictly. A number takes the JSON number
   grammar and must be finite, an integer its integer subset within
@@ -567,8 +714,7 @@
   takes the tuple as its first argument and raises `TypeError` for anything
   else. A cursor token issued for such a model before this release is refused with
   `UnsupportedQuery`, `400 unsupported_query`; clients restart from the first
-  page. `decode_cursor` takes `id_count`, the number of key columns, and
-  `get_id_attributes` returns every key column name.
+  page. `get_id_attributes` returns every key column name.
 
 - **core:** `compile_all` raises `ValueError` for a model with a composite
   primary key that declares a relation or a projection whose loader runs
@@ -663,6 +809,8 @@
   present convert those rows deliberately (for example to UTC with
   `(c::timetz AT TIME ZONE 'UTC')::time`) before changing the type. MongoDB is
   unaffected: it stores `date` and `time` values as ISO 8601 strings.
+- **core:** `to_column_value` is imported from `loom.core.model.convert`; it is
+  no longer exported by `loom.core.repository.sqlalchemy.values`.
 
 ### ai
 
@@ -671,9 +819,15 @@
   example `spec_malformed`, `engine_not_found` and `mcp_server_unreachable`.
   Build an issue with `AgentErrorCode.X.issue(...)` and keyword arguments,
   such as `AgentErrorCode.ENGINE_NOT_FOUND.issue(component="openai-agents",
-  available=["pydantic-ai"])`. `loom.ai.errors.from_message` is removed too:
-  `AgentCompilationError` still turns a bare string into an `UNSPECIFIED`
-  issue. Messages keep their text; a list that is empty renders `none`.
+  available=["pydantic-ai"])`. `loom.ai.errors.from_message` is removed too.
+  Messages keep their text; a list that is empty renders `none`.
+- **ai:** `AgentCompilationError` takes `AgentCompilationIssue` values only;
+  bare strings and the `AgentErrorCode.UNSPECIFIED` code they became are
+  removed. Build an issue from an `AgentErrorCode` member with `.issue()`.
+  The five pydantic-ai engine failures that raised bare strings now carry
+  their own codes, with the same messages: `TOOL_NAME_COLLISION`,
+  `TOOL_NAME_TOO_LONG`, `USECASE_NOT_COMPILED`, `CAPABILITY_KIND_NO_BUILDER`
+  and `MCP_CONNECTION_NOT_OPENED`.
 - **ai:** `AgentCompilationIssue` is a type alias of
   `CompileIssue[AgentErrorCode]` from `loom.core.errors.codes`, not a class.
   It has the same fields, but `isinstance` against it raises `TypeError`
@@ -752,6 +906,9 @@
   declares both a pydantic `type_ref` output and `output_check` serialises
   an accepted answer twice per attempt — once for the check's own builtins
   projection, once for the wire — accepted rather than optimised away.
+- **ai:** a provider's `credentials_ref` is read through
+  `loom.core.environment.configured_variable` and may not name a `LOOM_*`
+  variable.
 
 ### rest
 
@@ -814,6 +971,33 @@
   `msgspec.ValidationError`. The root `title` is dropped on pydantic
   OpenAPI schemas for request bodies and responses alike, matching the
   untitled document a Struct always produced.
+- **rest:** a `sortable` column must be a field of the items the route
+  answers with (`PageResult[T]`, `CursorResult[T]` or `list[T]`); a column
+  the response does not show, or a result type that does not say, fails
+  compilation naming the route and the field, since ordering by it reveals
+  its values. `sortable=ALL` opens every scalar column, hidden ones included,
+  and its startup warning says so. `filterable` follows the same rule, with
+  `filterable=ALL` as its escape hatch. "Shown" is decided by field name; a
+  pydantic field declared `exclude=True` is not shown.
+- **rest:** breaking: a `QuerySpec` route that can serve cursor pages (cursor
+  mode, or the default `allow_pagination_override=True`) must answer with
+  items that show its primary key, which every cursor token carries. A route
+  whose response hides the key, or whose result type is untyped (`str`,
+  `dict[str, Any]`, ...), now fails compilation naming the route and the key.
+  `sortable=ALL` opens it with a warning; a route that must hide its keys
+  serves offset pages only (`pagination_mode` offset and
+  `allow_pagination_override=False`). On such a route the key must also be of
+  a type a token carries, in a scalar column: a key declared in a JSON or
+  array column, which a SQL keyset cannot order, fails compilation instead
+  of on the first cursor request. A bare `UUID` key, a scalar `Uuid` column,
+  compiles and pages on SQLAlchemy and Mongo alike.
+- **rest:** `create_fastapi_app(interfaces=...)` is removed and `routes` is
+  required; use `create_fastapi_app(result, RouteSources(python=[...]))`.
+- **rest:** the `QUERY_PARAM_PAGE`, `QUERY_PARAM_LIMIT`,
+  `QUERY_PARAM_PAGINATION`, `QUERY_PARAM_AFTER`, `QUERY_PARAM_CURSOR`,
+  `QUERY_PARAM_SORT`, `QUERY_PARAM_DIRECTION` and `QUERY_PARAM_PROFILE`
+  re-exports of `loom.rest.fastapi.openapi` are removed; use the matching
+  `loom.rest.constants.QueryParam` member (`QueryParam.PAGE`, ...).
 
 ### prefect
 
@@ -904,11 +1088,8 @@
 - **etl:** a `WithAgent` compiled by a runner whose config has no `ai:`
   section fails with the new code `AGENT_RUNNER_NOT_CONFIGURED` instead of
   `AGENT_NOT_FOUND`, which now means only that no artifact declares the name.
-- **etl:** the names of `loom.etl.runner.filtering` (`_filter_plan`,
-  `_filter_process`, `_collect_all_names`, `_map_pipeline_nodes`,
-  `_map_process_nodes`) are deprecated and removed in loom 3.0; each emits a
-  `DeprecationWarning` naming its replacement, `loom.etl.compiler.filter_plan`
-  or, for `_collect_all_names`, `iter_processes` and `iter_all_steps`.
+- **etl:** `loom.etl.runner.filtering` is removed; use
+  `loom.etl.compiler.filter_plan`.
 - **etl:** the Polars writer refuses, under `SchemaMode.STRICT` and
   `EVOLVE`, a frame column that is a `Struct` or `List` where the Delta table
   has `String`, or `String` where the table has a `Struct` or `List`, at any
@@ -927,8 +1108,7 @@
   `Unknown step or process names: [...]` instead of `No steps or processes
   match include=[...]`, the constructor's parameter is `unknown`, and the
   names are kept on the new `unknown` attribute. Positional construction is
-  unchanged; `InvalidStageError(include=...)` still works, emits a
-  `DeprecationWarning` and is removed in loom 3.0. The error survives pickle
+  unchanged. The error survives pickle
   with its `unknown` names, so it crosses process boundaries intact.
 - **etl:** `ThreadDispatcher` runs each task in a copy of the caller's
   context, as `asyncio.to_thread` does, so a step of a parallel group sees
@@ -939,16 +1119,91 @@
   caller's context among the requirements on an implementation.
 - **etl:** `WithAgent(max_usd=...)` keeps the budget as a `Decimal`. An
   `int`, or another `numbers.Integral` such as `numpy.int64`, is converted
-  exactly. A `float` emits a `DeprecationWarning`, is converted through its
-  `str()` (`0.1` becomes `Decimal("0.1")`) and is removed in loom 3.0. A
-  `bool`, accepted before, and any other type, such as `str`, raise
-  `TypeError`. `NaN`, `sNaN` and `Infinity`, as a `Decimal` or a `float`,
-  raise `ValueError`, like zero and a negative amount; a `Decimal` `NaN` or
-  `sNaN` used to raise `decimal.InvalidOperation`, and a `Decimal`
-  `Infinity`, a `float` `inf` and a `float` `nan` were accepted.
+  exactly. A `bool`, accepted before, and any other type, such as `str`,
+  raise `TypeError`. `NaN`, `sNaN` and `Infinity` raise `ValueError`, like
+  zero and a negative amount; a `Decimal` `NaN` or `sNaN` used to raise
+  `decimal.InvalidOperation`, and a `Decimal` `Infinity` was accepted.
+- **etl:** `InvalidStageError(include=...)` is removed; use
+  `InvalidStageError(unknown=...)` or pass the names positionally.
+- **etl:** `WithAgent(max_usd=<float>)` is removed and raises `TypeError`;
+  pass a `Decimal`, such as `Decimal("2.50")`, or an `int`.
+- **etl:** `loom.etl.CheckpointCleaner` and
+  `loom.etl.checkpoint.CheckpointCleaner` are removed; use
+  `FsspecTempCleaner`, now exported from `loom.etl` as well.
+- **etl:** the module `loom.etl.declarative.target._schema_mode` is removed;
+  import `SchemaMode` from `loom.core.schema` (or `loom.etl`).
+- **etl:** `FromClickHouse.select(columns)` is removed; use
+  `FromClickHouse.columns(*columns)`.
+- **etl:** `ClickHouseSourceSpec.table` is removed; use
+  `ClickHouseSourceSpec.table_ref.ref`.
+- **etl:** `loom.etl.backends.polars._reader._apply_source_schema` is
+  removed. It was a test helper and has no public replacement.
+- **etl:** the module `loom.etl.runner.errors` is removed, and neither
+  `loom.etl.runner` nor `loom.etl.runner.core` exports `InvalidStageError`;
+  import it from `loom.etl.compiler` (or `loom.etl`).
+- **etl:** `loom.etl.io.sources._clickhouse` no longer re-exports
+  `FromClickHouse` and `ClickHouseSourceSpec`; import them from
+  `loom.etl.declarative.source` (`FromClickHouse` also from `loom.etl`).
 
 ### streaming
 
+- **streaming:** `ConsumerSettings.delivery` is required. A Kafka consumer
+  config without `delivery` fails to decode, so compiling a flow that reads
+  from Kafka raises `CompilationError` with `KAFKA_CONFIG_INVALID` naming the
+  missing field. Before, a missing `delivery` resolved to `at_most_once`.
+  Declare `delivery: at_most_once` to keep that behaviour, or
+  `delivery: at_least_once`. `ConsumerSettings.effective_delivery()` is
+  removed; read `ConsumerSettings.delivery`.
+- **streaming:** `ConsumerSettings.enable_auto_commit` is removed, together
+  with the `DELIVERY_CONFLICT` compile code. The consumer settings forbid
+  unknown fields, so a config that sets `enable_auto_commit` fails to decode
+  (`KAFKA_CONFIG_INVALID`). Declare `delivery` instead: `enable_auto_commit:
+  false` was `delivery: at_least_once`, and `true` was `delivery:
+  at_most_once`.
+- **streaming:** `ConsumerSettings.poll_timeout_ms` is removed. The
+  partitioned source ignored it and logged a deprecation warning; a config
+  that sets it now fails to decode (`KAFKA_CONFIG_INVALID`). Remove the key;
+  the partitioned source is tuned with `batch_size` and `poll_backoff_ms`.
+- **streaming:** `CompilationError` takes `CompilationIssue` values only;
+  bare strings and the `StreamingErrorCode.UNSPECIFIED` code they became are
+  removed. Build an issue from a `StreamingErrorCode` member with `.issue()`.
+- **streaming:** `CompilationError.errors` is removed. Read the messages from
+  `CompilationError.issues` (`[issue.message for issue in error.issues]`).
+- **streaming:** an `IntoTable` sink section (`streaming.sinks.<name>`) with a
+  key its backend does not read fails at compile with `SINK_CONFIG_INVALID`,
+  naming every unknown key; on Delta this covers the keys under `staging`
+  too, and a `staging` or `writer_properties` that is not a mapping is
+  refused. Before, such a key was ignored. A SQLAlchemy sink that references a
+  shared `database` reads no pool keys (`pool_size`, `echo`, ...) from its own
+  section, so they are refused there; set them under `database.<name>`.
+  A SQLAlchemy or ClickHouse sink section that sets both `url` and `database`
+  is refused too, instead of one silently winning: a SQLAlchemy sink sets one
+  or the other, and a ClickHouse sink takes its url from `database.<name>`, so
+  it sets `database` only.
+  Delta's `part_max_records` is removed and therefore refused: it was an alias
+  of `writer_properties.max_row_group_size`, which is the key to set. Without
+  `writer_properties` the row groups still hold at most 10 000 rows.
+  `DeltaSinkConfig` has no `part_max_records` attribute.
+- **streaming:** `CompiledSingleSource` and `CompiledMultiSource` take a
+  required `settings_key`, the config key their consumer settings were
+  resolved from (`kafka.consumer` or `kafka.consumers.<ref>`);
+  `KafkaSettings.consumer_key_for(ref)` names it.
+- **streaming:** `FromTopic`, `FromMultiTypeTopic`, `FromMongoCDC` and
+  `IntoTopic` take no `shape`, `CompiledSingleSource`, `CompiledMultiSource`
+  and `CompiledMongoCDCSource` have no `shape` field, and `StreamShape.MANY`
+  is removed. The runtime never read a declared shape: every source emits one
+  record per message, so a source declared `BATCH` was shape-checked as
+  batches while its nodes received records, and an `IntoTopic` writes one
+  record per message whatever it declared. Every flow now starts as a record
+  stream, and an `IntoTopic` ends its path (shape `none`). Migration: drop
+  `shape=`; to work on batches, use `CollectBatch` and, back to records,
+  `ForEach`.
+- **streaming:** a batch reaching a `Drain` or a `Broadcast` fails compilation
+  with `SHAPE_MISMATCH`, whatever the delivery. Under at-most-once delivery
+  `CollectBatch → Drain` compiled and dropped the batch; under at-least-once
+  the batch's offsets never completed. Under at-least-once a `Broadcast` with
+  more than one route raised `TypeError` (`Expected Message, got list`) on a
+  batch. Put a `ForEach` before the node.
 - **streaming:** `BatchExpandStep.execute` returns one list of payloads per
   message of its batch, in batch order, instead of one flat list. List `i`
   holds what `messages[i]` expands into; each output takes that message's
@@ -975,9 +1230,11 @@
   `CompileIssue` instead), `loom_message_type()` is gone, and
   `CompilationIssue.prefixed()` is removed. Messages keep their text.
 - **streaming:** a `Broadcast` followed by another node reports
-  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level, in a
-  branch and in a Router branch. It used to report `TERMINAL_NOT_LAST` with
-  component `Broadcast`.
+  `BROADCAST_NOT_LAST` with component `broadcast`, at the top level and in a
+  branch. It used to report `TERMINAL_NOT_LAST` with component `Broadcast`. In
+  a Router branch a `Broadcast` reports `ROUTER_BRANCH_UNSAFE_NODE`, and one
+  followed by another node also `TERMINAL_NOT_LAST` with component
+  `router branch <label> > Broadcast`.
 - **streaming:** a Router branch that ends in `IntoTopic`, `IntoTable` (any
   `IntoSink`) or `Drain` hands the message to that terminal. The branch's
   terminal was skipped: with a flow `output` the message went to the output
@@ -1060,11 +1317,71 @@
 - **streaming:** a sync `With(process=...)` hands on one message at a time, as
   `WithAsync` does. After a `CollectBatch` it handed on whole batches, so as
   the last node of a flow without `output` its messages never completed.
+- **streaming:** `IntoTable` with `Backend.SQLALCHEMY` binds each column with
+  the type core compiles for the field instead of its own map:
+  - a `StrEnum` or `IntEnum` field is written as its scalar (`String`,
+    `Integer`) instead of as JSON;
+  - a column type declared with `ColumnField` (on a `BaseModel`) or
+    `Annotated[T, <ColumnType>]` is used;
+  - a `Decimal`, plain `Enum` or `Literal` field without a declared column
+    type is refused when the partition is built, with core's message, instead
+    of being written as `Numeric()` (a `Decimal`) or as JSON. Declare it, e.g.
+    `Annotated[Decimal, Numeric(precision=12, scale=2)]`, or
+    `Annotated[Decimal, Numeric()]` for an unbounded `NUMERIC`.
+- **streaming:** `IntoTable` with `Backend.CLICKHOUSE` hands a `date` field to
+  clickhouse-connect as a `date`, as core keeps it for a column, instead of as
+  its ISO-8601 text. clickhouse-connect writes a `Date` column only from a
+  `date` and refused the text ("unsupported operand type(s) for -: 'str' and
+  'datetime.date'"); a `String` column now refuses the `date` ("object of type
+  'datetime.date' has no len()"). A `dict` value is written as compact JSON
+  text from `msgspec.json`, keys still sorted (`{"a":1}`, was `{"a": 1}`).
 
 ## 🐛 Fixes
 
 ### core
 
+- **core:** `RepositoryDynamoDB` reads, updates and deletes by a `UUID`
+  primary key. `get_by_id`, `get_by`, `exists_by`, `update` and `delete`
+  handed the raw key to boto3's `TypeSerializer`, which raised
+  `TypeError: Unsupported type "<class 'uuid.UUID'>"`; `create` already stored
+  the key as a string through `msgspec.to_builtins`. The key is now encoded
+  with that same conversion, so it matches its stored form and round-trips.
+  `update` keeps the key as stored instead of rewriting it from the argument.
+  Attribute values are encoded as before.
+- **cache:** security: the repository cache keeps each caller's entries apart.
+  The entity key (`get_by_id`, and the rows behind a cached page), the
+  `list_paginated` index key, the `list_with_query` key and the key of a
+  `@cache_query` read (`scope="entity"` and `scope="list"`) now carry the
+  caller's cache scope, so a row, a page of ids, a first page with its next
+  cursor or a custom read's result cached for one caller is no longer served
+  to another that row-level security hides it from. The scope comes from the
+  new, provisional `SupportsCacheScope.cache_scope()` (`""` for a repository
+  without it): `RepositorySQLAlchemy` builds it from every setting the
+  session-settings provider returns, so a policy reading a product setting
+  such as `app.tenant_id` is kept apart too, while cursor tokens stay bound to
+  the `loom.scope.*` settings only. Inside a caller-owned transaction (the
+  unit of work a use case runs in, or a `@transactional` scope) every cached
+  read goes straight to the repository, with no lookup and no store, and so
+  does a call to a `@cache_query` method whose `session` parameter, passed by
+  position or by name, is not `None`. This holds where the backend reports
+  such a transaction, SQLAlchemy and Mongo with transactions; DynamoDB and
+  Mongo with `transactions: false` keep caching, and a read-only use case run
+  outside another unit of work opens none and keeps using the cache. Writes
+  still evict every scope's entries: the scope is not part of the dependency
+  tags, and a write bumps the same generation counters whatever its scope. The
+  scope is read once per cache operation and read again before its writes,
+  which are skipped if it changed. `CachedRepository` implements
+  `cache_scope()` and `has_caller_scoped_session()` by forwarding the wrapped
+  repository's, so a wrapper around it keys by the same scope and stays out of
+  the same transactions, and refuses at construction, with a `RuntimeError`, a
+  repository whose `session_manager` has session settings but which does not
+  implement `cache_scope()`. The list filter, query and `@cache_query`
+  arguments fingerprints in cache keys are now the full SHA-256 instead of its
+  first 64 bits. `entity_key` and `list_index_key` take a required
+  keyword-only `scope`, and the new `query_index_key` and `custom_query_key`
+  build the `list_with_query` and `@cache_query` keys. Existing entity, list,
+  query and `@cache_query` entries are no longer read after the upgrade and
+  expire by their TTL.
 - **core:** one environment selector and deploy checks (ADR 0005). loom had
   two unconnected notions of environment (the logger's `dev`/`prod`, where an
   unknown value meant `dev`, and `FlowCtx.environment`) and nothing checked a
@@ -1115,10 +1432,25 @@
   startup. `parse_environment` validates a value naming an environment.
 - **core:** the SQLAlchemy query compiler converts a filter value, and each
   `in` item, as the repository converts a value it writes
-  (`loom.core.repository.sqlalchemy.values.to_column_value`, now shared by
+  (`loom.core.model.convert.to_column_value`, now shared by
   both): an `Enum` member is bound as its value, a JSON column's value is
   reduced to builtins. A plain `Enum` member as the value of a filter on a
   string column failed to bind.
+- **core:** pagination cursors are signed and decoded strictly (ADR 0004).
+  A cursor token was base64url JSON with no integrity, and a datetime key
+  decoded without a time zone, so a client could forge a position and a
+  consumer comparing it with aware datetimes raised `TypeError` (a 500). A
+  token is now `base64url(payload).base64url(tag)`, the tag an HMAC-SHA256
+  over a domain-separated input bound to the model, the normalised filters,
+  the sort and the caller's scope, verified with `hmac.compare_digest` before
+  the payload is parsed. A malformed, unsigned, forged or rebound token, a
+  key signed by an unknown key id, a datetime key without a time zone, a key
+  of an unknown type, the wrong number of keys or the wrong backend is
+  `UnsupportedQuery` (`400 unsupported_query`), and the message never echoes
+  the token. A naive datetime key, such as a `DateTime(tz=False)` column, is
+  issued as UTC with an explicit offset and compared as naive UTC against
+  that column, on SQLite and Postgres alike.
+
 - **core:** a `BoundaryValidationError` no longer echoes the rejected value
   (#347). msgspec's `Invalid enum value 'x'` (an `Enum`, `IntEnum` or `Literal`
   field or dict key) and `Invalid value 'x'` (a tagged-union tag) become
@@ -1147,8 +1479,9 @@
   column only. The cursor now orders and compares on
   every key column after the sort keys and carries every key value in the
   token; offset pages and `list_paginated` order by every key column too.
-  Tokens of single-column keys are unchanged. A cached repository over such a
-  model keyed its entries by the first key column too, so a warm
+  That fix leaves tokens of single-column keys as they were. A cached
+  repository over such a model keyed its entries by the first key column
+  too, so a warm
   `list_paginated` or first cursor page served the rows sharing a first key
   value as copies of one of them; it now caches and reloads every row by its
   full key, one `get_by_id` per missing row.
@@ -1184,6 +1517,12 @@
 
 ### etl
 
+- **etl:** the error for a missing destination table names only advice that
+  works under the policy in force. Under `missing_table_policy='error'` it
+  used to suggest `SchemaMode.OVERWRITE`, which that policy ignores; it now
+  suggests `missing_table_policy='create'`, `'schema_mode'` with
+  `SchemaMode.OVERWRITE`, or creating the table first.
+
 - **etl:** `replace(allow_empty=False)` on Spark writes the rows it checked.
   The writer ran `DataFrame.isEmpty()` and then let the write compute the
   frame again, so a source that changed in between, or a nondeterministic
@@ -1364,6 +1703,32 @@
 
 ### streaming
 
+- **streaming:** `SINK_CANNOT_TRACK_COMMITS` and
+  `INLINE_SINK_CANNOT_SKIP_COMMITS` point their `field` at the `delivery` of
+  the consumer the source was bound through, `kafka.consumer.delivery` or
+  `kafka.consumers.<ref>.delivery`, instead of `streaming.delivery`, which no
+  config reads.
+- **streaming:** `ForEach` accepts the batch a `BatchStep` produces, so
+  `CollectBatch → BatchStep → ForEach → IntoTopic` compiles; it failed with
+  `shape mismatch: expected many but got batch before ForEach`. Under
+  at-least-once delivery each offset completes once, after its write is
+  flushed.
+- **streaming:** a batch reaching a node that takes records fails compilation
+  with `SHAPE_MISMATCH`: `IntoTopic`, `IntoTable` (any `IntoSink`), `Fork`,
+  `ExpandRoutes`, `CollectBatch` (it groups records), the flow `output`
+  (component `flow.output`, field `output`)
+  and a Broadcast route `output` (component `broadcast branch <n> > output`,
+  field `output`). Such flows used to compile. `CollectBatch → BatchStep →
+  IntoTopic` raised `'list' object has no attribute 'meta'` at runtime. Put a
+  `ForEach` before the node. `Drain` and `Broadcast` are under Behaviour
+  changes.
+- **streaming:** a `BatchExpandStep` subclass outputs records to the shape
+  check. The compiler matched step output shapes by exact type, so every step
+  subclass kept the incoming shape and a record step after a
+  `BatchExpandStep` failed with `expected record but got batch`. The shape
+  check reads a step's `input_shape` and `output_shape` from its class,
+  except in a Router branch: there a `BatchStep` runs one record at a time,
+  on a one-message batch, so it is record to record.
 - **streaming:** a `Broadcast` is validated as a `Broadcast`. The compiler
   treated it as a plain terminal, so the nodes of its branches were never
   shape-checked. A branch with a shape mismatch, such as a record step after
@@ -1433,6 +1798,49 @@
   offset at the step, also when the batch holds several messages of one
   offset that an earlier expanding node produced. The new return type is under
   Behaviour changes.
+- **streaming:** `IntoTable` with `Backend.SQLALCHEMY` binds a `bytes` field as
+  `LargeBinary` (`bytea` on Postgres). It bound it as text, which Postgres
+  refused for a `bytea` column ("column is of type bytea but expression is of
+  type character varying"). IntoTable now takes each column's type from core's
+  model introspection (`loom.core.backend.sqlalchemy.sa_table_for`), the type
+  a compiled model declares, and builds each row with core's new
+  `loom.core.repository.sqlalchemy.values.to_column_row`, which converts and
+  checks values as the repository does when it writes them: a plain `Enum` as
+  its value, a struct in a JSON column as a dict, an `UNSET` value left out so
+  the column's database default applies, also when other rows of the batch
+  set it, and a `time` whose UTC offset its column cannot take refused with
+  `BoundaryValidationError` (`boundary_validation`) instead of being written
+  without the offset. Relation and projection fields are no longer columns
+  for any backend: the SQLAlchemy and Delta rows included them and failed on
+  their `UNSET` value, and the ClickHouse insert named them. A `date` or
+  `time` field is bound as `Date` or `Time`, as core infers it, so it is
+  written into a Postgres `date` or `time` column; it was bound as `String`
+  and Postgres refused it ("column is of type date but expression is of type
+  character varying").
+- **streaming:** `IntoTable` with `Backend.DELTA` or `Backend.CLICKHOUSE`
+  refuses an `UNSET` column value when the item is written, with
+  `BoundaryValidationError` (`boundary_validation`) naming the field and the
+  table. Delta failed later, at a mini-batch or flush, with pyarrow's
+  `ArrowInvalid`, and ClickHouse passed the `UNSET` object to
+  clickhouse-connect's `insert`. Neither can leave a column out for one row as
+  `Backend.SQLALCHEMY` does.
+- **streaming:** `IntoTable` with `Backend.DELTA` or `Backend.CLICKHOUSE`
+  converts each column value as `Backend.SQLALCHEMY` and core's repository
+  do, with `loom.core.model.convert.to_column_value`: a plain `Enum` becomes
+  its value and a nested `msgspec.Struct` or `LoomStruct` a dict, also inside a
+  list or a dict, while `bytes`, `datetime`, `date`, `time`, `Decimal` and
+  `UUID` are kept. Delta refused an `Enum` or a struct with pyarrow's
+  `ArrowInvalid` ("did not recognize Python value type when inferring an
+  Arrow data type"); a nested struct is now a Delta struct column, as a `dict`
+  field already was, and keeps its `datetime` and `Decimal` types.
+  clickhouse-connect refused an `Enum` or a struct ("object of type ... has no
+  len()") and `json.dumps` a `dict` holding a `Decimal` or `datetime` ("Object
+  of type Decimal is not JSON serializable"); an `Enum` is now inserted as its
+  value and a struct or `dict` as JSON text from `msgspec.json`, a `datetime`
+  inside it as ISO-8601 text and a `Decimal` as decimal text.
+  `to_column_value` moved to `loom.core.model.convert` (under Behaviour
+  changes), so the Delta and ClickHouse writers use it without SQLAlchemy
+  installed.
 
 ### prefect
 

{loom_kernel-8.0.0 → loom_kernel-9.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 8.0.0
+Version: 9.0.0
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3
@@ -457,6 +457,10 @@ app:
     title: My Store API
     version: 0.1.0
 
+persistence:
+  cursors:
+    signing: ephemeral   # development.yaml only; production.yaml configures keys
+
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
 

{loom_kernel-8.0.0 → loom_kernel-9.0.0}/README.md

@@ -333,6 +333,10 @@ app:
     title: My Store API
     version: 0.1.0
 
+persistence:
+  cursors:
+    signing: ephemeral   # development.yaml only; production.yaml configures keys
+
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
 

{loom_kernel-8.0.0 → loom_kernel-9.0.0}/src/loom/ai/engines/pydantic_ai/_capabilities.py

@@ -115,9 +115,7 @@ class _UsecaseGrant:
 def _grant(key: str, use_case: type[Compilable]) -> _UsecaseGrant:
     execution = use_case.__execution_plan__
     if execution is None:
-        raise AgentCompilationError(
-            [f"use case '{key}' carries no compiled execution plan; it was never compiled"]
-        )
+        raise AgentCompilationError([AgentErrorCode.USECASE_NOT_COMPILED.issue(key=key)])
     return _UsecaseGrant(
         tool_name=tool_name("usecase", key), key=key, use_case=use_case, execution=execution
     )
@@ -637,7 +635,11 @@ def _build(plan: AgentPlan, destination: _Destination, context: BuildContext) ->
         binding = _KINDS.get(type(capability))
         if binding is None:
             raise AgentCompilationError(
-                [f"{context.agent}: capability kind '{capability.kind}' has no builder"]
+                [
+                    AgentErrorCode.CAPABILITY_KIND_NO_BUILDER.issue(
+                        component=context.agent, kind=capability.kind
+                    )
+                ]
             )
         if binding.destination is destination:
             yield binding.builder(capability, context)