loom-kernel 2.18.0__tar.gz → 2.19.0__tar.gz

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/CHANGELOG.md

@@ -90,6 +90,9 @@
 
 ### ai
 
+- **ai:** `AiConfig` logs a warning for every key of `ai.prices` that no
+  binding of `ai.models` names, a price no run is priced with (a misspelt
+  model id, for instance).
 - **ai:** `provider: typesafe` binds TypeSafe's Jev, a decision model that
   fills a typed output from the state it is given and never writes text, so a
   role whose answer is a choice or a verdict can run on it. Extra
@@ -153,12 +156,14 @@
   assembles an unentered `AgentRuntime` from an `ai:` section: engine,
   artifacts under `root`, compilation. `create_app` now builds its runtime
   through it. `AgentRuntime.plans` exposes the compiled plans.
-- **ai:** `AgentPlan.fingerprint`, a stable sha256 of what decides an
-  agent's answers: compiled instructions, output schema, output check (by
-  import path), policies, `spec_version`, the kind and name of every
-  capability and the bound provider, model, `output_mode` and `options`; not
-  the name, metadata, price, where a capability is served from, region,
-  endpoint, credentials or `streaming`.
+- **ai:** `AgentPlan.fingerprint`, a stable sha256 of the declared facts
+  that decide an agent's answers: `spec_version`, compiled instructions,
+  output schema, output check (by import path), the kind and declared facts
+  of every capability (including the `include`/`exclude` filters of `mcp`
+  and `a2a`) and the bound provider, model, `output_mode` and `options`; not
+  the policies, the content of the skills, tools or use cases a capability
+  names, the name, metadata, price, where a capability is served from,
+  region, endpoint, credentials or `streaming`.
 - **ai:** `loom.ai.etl.PolarsAgentRunner` runs an agent over a Polars frame
   for ETL steps (see `etl`): a runtime of its own per call that also sees the
   caller's context variables, concurrency that waits instead of refusing, one
@@ -250,6 +255,14 @@
   attribute name (the flow name with `-` replaced by `_`), as the worker
   imports it, and raises `ValueError` otherwise; a module not yet imported is
   imported from that file. Any other object raises `TypeError`.
+- **prefect:** a flow can notify an AWS SNS topic. The `notifications` list
+  takes `kind: sns` with `topic_arn` and the same `on_failure` (default
+  `true`) and `on_completion` (default `false`) filters as `kind: slack`; an
+  empty `topic_arn` skips the entry. Matching terminal states are published
+  through boto3 (`prefect` extra) in the topic's region with subject
+  `[<state>] <flow>` and a plain-text message with the run details, the end of
+  the error message and the Prefect link. A failed publish is logged and does
+  not fail the run. `SnsNotifier` is exported from `loom.prefect.notify`.
 
 ### etl
 
@@ -333,11 +346,14 @@
   exported from `loom.etl`, carrying `table_ref`) before it creates or writes
   anything, so a dimension rebuilt from a read that came back empty keeps its
   rows and the step fails. Polars checks the row count of the frame the
-  replace already collects, at no extra cost; Spark runs `isEmpty()`, one
-  job that stops at the first row, before the write recomputes the frame. On
-  Polars a `Null`-typed column still fails first, with `SchemaError`. On
-  Spark a streaming DataFrame, whose rows cannot be counted before the
-  write, raises `ValueError` and the table is left as it was.
+  replace already collects, at no extra cost. Spark computes the frame once
+  with `localCheckpoint(eager=True)`, checks that result with `isEmpty()`,
+  writes it and then frees its blocks, as described under Fixes; a Spark
+  Connect DataFrame, which has no `localCheckpoint()` in pyspark 3.5, is
+  persisted, counted, written and unpersisted instead. On Polars a
+  `Null`-typed column still fails first, with `SchemaError`. On Spark a
+  streaming DataFrame, whose rows cannot be counted before the write, raises
+  `ValueError` and the table is left as it was.
   The default, `allow_empty=True`, keeps today's behaviour, and `ReplaceSpec`
   gains the `allow_empty` field.
 - **etl:** `MaintenanceSpec`, `VacuumSpec`, `CompactSpec` and `ZOrderSpec` are
@@ -370,6 +386,51 @@
 
 ### ai
 
+- **ai:** `AgentPlan.fingerprint`, and so the ETL `agent_version` column,
+  changes once for every existing agent. It no longer covers any policy, so
+  changing `retries`, a timeout, `max_iterations`, `max_tool_calls`,
+  `max_requests`, a token cap, `max_usd`, `on_unpriced_spend` or
+  `max_history_bytes` keeps it; and it now covers the `include`/`exclude`
+  filters of `mcp` and `a2a` capabilities, so changing a filter changes it.
+  A step that keeps `agent_version` in its upsert keys and its anti-join
+  re-runs, and pays for, every row of the anti-join window on its next
+  execution after the upgrade.
+- **ai:** in `PolarsAgentRunner.map`, a field of a pydantic output whose type
+  is another pydantic model is now a `Struct` column of that model's fields,
+  named by alias, instead of a `String` holding its JSON. Writing it into a
+  Delta table created before fails with `SchemaError` naming the column (see
+  the `etl` entry on casts between text and nested types). To keep writing to
+  that table, rewrite it once with the column decoded through the model,
+  `pl.Series(name, [None if c is None else Model.model_validate_json(c).model_dump(by_alias=True) for c in frame[name]], dtype=<the Struct type>)`,
+  and `schema=SchemaMode.OVERWRITE` (delta-rs `schema_mode="overwrite"`).
+  `pl.col(name).str.json_decode(<the Struct type>)` fails with
+  `ComputeError` when the model has a `date` field (Polars 1.39.3).
+- **ai:** in `PolarsAgentRunner.map`, a `String` column holding a value's JSON
+  now holds the JSON of a string too: `"5"` is written `"\"5\""` and `5` is
+  written `5`, where both were `5`. This covers a field typed as a union of
+  several types, `Any` or a `Literal` mixing value types whose answer is a
+  string. A `UUID`, `time`, `timedelta` or `bytes` field keeps its plain text
+  (`P1D` is written `P1D`). Rows written before hold the raw string and rows
+  written after hold its JSON encoding. A cell alone does not always tell
+  them apart (an old `5` may have been the string `"5"`), but `agent_version`
+  does: every agent's fingerprint changes with this release, so the old rows
+  carry the old version. Re-run the old rows of such a field, or rewrite each
+  old cell whose answer was a string as the JSON encoding of its text.
+- **ai:** the `agent_cost_usd` column `PolarsAgentRunner.map` writes is
+  `Decimal(38, 12)` instead of `Float64`, carrying `AgentUsage.cost` without a
+  float conversion, rounded half to even to 12 decimal places. Writing into a
+  Delta table created before does not fail: an append, upsert or replace with
+  `SchemaMode.STRICT` or `EVOLVE` casts the column back to the table's
+  `Float64`. To keep the decimal, rewrite the table once with the column cast
+  to `pl.Decimal(38, 12)` and `schema=SchemaMode.OVERWRITE` (delta-rs
+  `schema_mode="overwrite"`); the costs already stored keep the precision of
+  the float they were written as.
+- **ai:** an `ai:` section with `max_concurrent_runs` below 1 fails to load
+  with `MAX_CONCURRENT_RUNS_INVALID`, naming the key and the minimum. Before,
+  it loaded; with `0` the runtime refused every run with `TOO_MANY_RUNS` and
+  `PolarsAgentRunner.map` on a frame with a prompt waited on a semaphore of
+  zero slots and did not return, and a negative value failed when the runtime
+  was built.
 - **ai:** `load_specs`, and so `ai.specs`, refuses a glob that leaves the
   root it resolves against — a `..` segment, an absolute pattern or a match
   that resolves outside the root through a symlink — with
@@ -446,14 +507,41 @@
 - **prefect:** the private `_deploy_single(flow, meta, work_pool, env)` of
   `loom.prefect.deploy._discovery` now takes `(flow, meta, plan)`; callers
   migrate to the public `deploy_flow(flow, env=..., work_pool=...)`.
+- **prefect:** `discover_and_deploy_etls(flows_package=...)` checks every
+  flow of the package, its work pool and that its module exposes it under its
+  attribute name as `deploy_flow` does, before it registers the first
+  deployment. A flow without a work pool for the environment, one its module
+  exposes under another attribute name, or one whose source file cannot be
+  imported raises `ValueError` and no deployment of the package is
+  registered. Flows used to be registered one by one: a flow without a work
+  pool raised after the flows before it were registered, and a flow exposed
+  under another name or not importable from its source file was registered
+  with an entrypoint the worker cannot import.
 
 ### etl
 
+- **etl:** a `WithAgent` compiled by a runner whose config has no `ai:`
+  section fails with the new code `AGENT_RUNNER_NOT_CONFIGURED` instead of
+  `AGENT_NOT_FOUND`, which now means only that no artifact declares the name.
 - **etl:** the names of `loom.etl.runner.filtering` (`_filter_plan`,
   `_filter_process`, `_collect_all_names`, `_map_pipeline_nodes`,
   `_map_process_nodes`) are deprecated and removed in loom 3.0; each emits a
   `DeprecationWarning` naming its replacement, `loom.etl.compiler.filter_plan`
   or, for `_collect_all_names`, `iter_processes` and `iter_all_steps`.
+- **etl:** the Polars writer refuses, under `SchemaMode.STRICT` and
+  `EVOLVE`, a frame column that is a `Struct` or `List` where the Delta table
+  has `String`, or `String` where the table has a `Struct` or `List`, at any
+  depth of a `Struct` or `List`, with `SchemaError` naming the column and
+  both types; nothing is written. The check is by dtype, so an all-null
+  `String` column written into a `List` column is refused too. Before, a
+  `Struct` was cast to its Polars display text (`{1.0}` for a `Float64`
+  field) and a `String` to a `List` wrapped or nulled each value, both
+  written without an error; a `List` to `String` and a `String` to `Struct`
+  failed inside Polars. The Spark writer does not refuse these casts: it
+  writes a `Struct` or `List` into a `String` column as text (`{1.0}`,
+  `[1]`). To change the column's type,
+  rewrite the table once with the column converted and
+  `schema=SchemaMode.OVERWRITE` (delta-rs `schema_mode="overwrite"`).
 - **etl:** `InvalidStageError` names what it reports: its message reads
   `Unknown step or process names: [...]` instead of `No steps or processes
   match include=[...]`, the constructor's parameter is `unknown`, and the
@@ -461,6 +549,22 @@
   unchanged; `InvalidStageError(include=...)` still works, emits a
   `DeprecationWarning` and is removed in loom 3.0. The error survives pickle
   with its `unknown` names, so it crosses process boundaries intact.
+- **etl:** `ThreadDispatcher` runs each task in a copy of the caller's
+  context, as `asyncio.to_thread` does, so a step of a parallel group sees
+  the `ContextVar` values of the run that dispatched it (the Prefect flow run
+  among them) and its OpenTelemetry context. A parallel group's step spans
+  therefore nest under their process span instead of starting a trace of
+  their own. `ParallelDispatcher` lists running each task in a copy of the
+  caller's context among the requirements on an implementation.
+- **etl:** `WithAgent(max_usd=...)` keeps the budget as a `Decimal`. An
+  `int`, or another `numbers.Integral` such as `numpy.int64`, is converted
+  exactly. A `float` emits a `DeprecationWarning`, is converted through its
+  `str()` (`0.1` becomes `Decimal("0.1")`) and is removed in loom 3.0. A
+  `bool`, accepted before, and any other type, such as `str`, raise
+  `TypeError`. `NaN`, `sNaN` and `Infinity`, as a `Decimal` or a `float`,
+  raise `ValueError`, like zero and a negative amount; a `Decimal` `NaN` or
+  `sNaN` used to raise `decimal.InvalidOperation`, and a `Decimal`
+  `Infinity`, a `float` `inf` and a `float` `nan` were accepted.
 
 ## 🐛 Fixes
 
@@ -491,6 +595,42 @@
 
 ### etl
 
+- **etl:** `replace(allow_empty=False)` on Spark writes the rows it checked.
+  The writer ran `DataFrame.isEmpty()` and then let the write compute the
+  frame again, so a source that changed in between, or a nondeterministic
+  plan, could still replace the table with no rows. The frame is now
+  computed once with `localCheckpoint(eager=True)`, and the check and the
+  write both read that result. Those rows are kept in executor storage, not
+  in reliable storage: losing an executor before the write ends fails the
+  write instead of recomputing the frame. Once the write ends, whether it
+  succeeded or failed, and when the check refuses the frame, the writer
+  unpersists the checkpointed RDD, so its blocks do not stay on the executors
+  for the rest of the session. A Spark Connect DataFrame has no
+  `localCheckpoint()` in pyspark 3.5: it is persisted with `MEMORY_AND_DISK`,
+  counted, written and unpersisted. That is weaker: Spark recomputes a cached
+  block it loses, so the write may read rows other than the ones counted.
+  Persisting and unpersisting go through Spark's shared cache manager, which
+  matches cached data by plan, so a frame with a plan equivalent to the
+  written frame's, cached before the write, is unpersisted too. Freeing the
+  rows is best effort: a failure is logged as a warning and never replaces
+  the outcome of the write, its error or `EmptyReplaceError`. The write
+  policy's `_is_empty` hook becomes `_pin_rows`, which returns the frame to
+  write along with whether it is empty, and gains `_release_pinned`, which
+  frees what `_pin_rows` kept and must not raise.
+- **etl:** the Polars checkpoint store memory-maps only a local checkpoint.
+  It recognised cloud URIs by a lower-case prefix list without `r2://`, so a
+  checkpoint under `r2://` or an upper-case scheme such as `S3://` was read
+  with `memory_map=True`; it now uses the scheme set of
+  `loom.etl._uri`, compared case-insensitively.
+- **etl:** documentation states what the writers do. `replace(streaming=...)`
+  and `ReplaceSpec.streaming` claimed a streaming Arrow write on Polars; no
+  backend reads the flag for a full replace, and the collect engine follows
+  the step's `streaming` attribute. `vacuum(full=True)` warns, in the guide
+  and in `VacuumSpec`, `MaintainTable.vacuum` and `MaintenanceVacuumConfig`,
+  that it deletes every unreferenced file under the table directory older
+  than the retention, whoever wrote it. The pipelines page states that
+  `replace_where` with an empty frame deletes the rows matching its
+  predicate.
 - **etl:** `upsert` and `update` on the Polars backend write with the route's
   `writer` and `commit` settings. The delta-rs MERGE received neither, so the
   Parquet files it rewrote came out with delta-rs defaults (SNAPPY, row groups
@@ -534,6 +674,20 @@
 
 ### ai
 
+- **ai:** `PolarsAgentRunner.map` runs at most `ai.max_concurrent_runs`
+  worker tasks, which take the rows in order, instead of creating one task
+  per row up front; the order of the output rows is unchanged. A run
+  cancelled while the batch was not raises a `RuntimeError` naming how many
+  rows were left without an outcome and the first of them, where the run's
+  `asyncio.CancelledError` reached the caller.
+- **ai:** `PolarsAgentRunner.validate` reports an output with a field no
+  column can hold (a union of two models, for instance) as
+  `OUTPUT_MISMATCH` instead of raising its `TypeError`, and checks the output
+  columns before asking whether the model is priced.
+- **ai:** an agent output type that contains itself, directly
+  (`children: list[Node]`) or through another type, no longer makes
+  `PolarsAgentRunner.validate` and `map` fail with `RecursionError`: the
+  repeated struct or pydantic model is a `String` column holding its JSON.
 - **ai:** a configured `ai.prices` rate prices runs again with pydantic-ai
   2.54. That release records a response's usage before the
   `after_model_request` hook loom priced it in, so every run reported
@@ -573,6 +727,22 @@
   `SERVICE_UNAVAILABLE`, which `HttpErrorMapper` maps to `503`. Any other
   exception from the authenticator, and the `401` refusal, are unchanged.
 
+### prefect
+
+- **prefect:** every log line the bridge sends carries the run's
+  `flow_run_id`, and a step's line its `task_run_id` as well. A step's lines
+  went out with `task_run_id` only, so a reader scoping a run's logs by flow
+  run, such as Prefect's flow run view, missed every step line; a step of a
+  parallel group now keeps the flow run too, through `ThreadDispatcher`'s
+  context copy. A line emitted inside a step with no flow run bound omits
+  `flow_run_id` instead of sending `null`.
+- **prefect:** `deploy_flow` refuses a flow built from a YAML declaration
+  with a `TypeError` naming `discover_and_deploy_etls(config=...)`, which
+  deploys it, instead of the entrypoint module's `ConfigError`. When the flow
+  module's `__getattr__` raises anything but `AttributeError` while the flow
+  is looked up, `deploy_flow` raises its documented `ValueError`, chained to
+  that exception.
+
 ## ♻️ Refactor
 
 ### etl
@@ -591,6 +761,12 @@
   spec type than the protocol it is dispatched through; it now accepts the
   protocol's type and refuses a spec of another kind with a `TypeError`,
   which the registry's dispatch by kind already made unreachable.
+- **etl:** `PolarsSourceReader` decodes JSON columns through
+  `decode_json_columns`; the private `_apply_json_decode` staticmethod and
+  module function of `loom.etl.backends.polars._reader` are removed.
+- **etl:** the private `_is_cloud_path` of `loom.etl.checkpoint._cleaners`
+  moves, with the same scheme set, to `loom.etl._uri.is_cloud_path`,
+  and the Polars backend defines its frame type variable once.
 
 # 🚀 Release 2.1.2 ([#238](https://github.com/the-reacher-data/loom-py/pull/238))
 

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: loom-kernel
-Version: 2.18.0
+Version: 2.19.0
 Summary: Loom Python project
 License-File: LICENSE
 Classifier: Programming Language :: Python :: 3

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/compiler/_fingerprint.py

@@ -18,32 +18,22 @@ _CAPABILITY_NAMES: Final[Mapping[str, tuple[str, ...]]] = MappingProxyType(
     {
         "usecase": ("keys",),
         "sql": ("connection",),
-        "mcp": ("server",),
+        "mcp": ("server", "include", "exclude"),
         "skills": ("library", "names"),
         "python": ("factory_ref",),
-        "a2a": ("agent",),
+        "a2a": ("agent", "include", "exclude"),
         "native": ("tool",),
     }
 )
 
 
 def plan_fingerprint(plan: AgentPlan) -> str:
-    """Return the sha256 of what decides *plan*'s answers.
-
-    Args:
-        plan: Compiled plan.
-
-    Returns:
-        Hex digest over the instructions, output schema and check, policies,
-        format version, the kind and name of every capability and the model
-        binding, without the deployment details.
-    """
+    """Return the sha256 behind :attr:`AgentPlan.fingerprint`, which states what it covers."""
     facts = {
         "spec_version": plan.spec_version,
         "instructions": [[block.text, block.template] for block in plan.instructions],
         "output_schema": plan.output.schema,
         "output_check": _reference(plan.output_check),
-        "policies": plan.policies,
         "capabilities": [_capability(capability) for capability in plan.capabilities],
         "provider": plan.inference.provider,
         "model": plan.inference.model,

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/compiler/_plan.py

@@ -408,16 +408,24 @@ class AgentPlan(LoomFrozenStruct, frozen=True, kw_only=True):
 
     @property
     def fingerprint(self) -> str:
-        """Stable sha256 of everything that decides this agent's answers.
-
-        Covers the instructions as compiled, the output schema, the output
-        check, the policies, the format version, the kind and name of every
-        capability and the bound provider, model, output mode and options.
-        Leaves out the name, description, metadata, provenance, configured
-        price, where a capability is served from (URL, directory,
-        credentials, timeouts) and the binding's region, endpoint,
-        credentials and streaming flag. The output check enters by its import
-        path, so editing the check's body keeps the fingerprint.
+        """Stable sha256 of the declared facts that decide this agent's answers.
+
+        Covers the format version, the instructions as compiled, the output
+        schema, the output check, the kind and declared facts of every
+        capability (use case keys, SQL connection, MCP server, skill library
+        and names, Python factory reference, A2A agent, native tool, and the
+        ``include``/``exclude`` filters of MCP and A2A) and the bound
+        provider, model, output mode and options. It covers what the plan
+        declares, not the content of the skills, tools or use cases it names:
+        editing a skill's text or a tool's code keeps the fingerprint. Leaves
+        out the name, description, metadata, provenance, configured price,
+        every policy (each bounds a run and fails it past that bound;
+        ``max_history_bytes`` also withholds from the hook a history above
+        it; none decides the content of an accepted answer), where a
+        capability is served from (URL, directory, credentials, timeouts) and
+        the binding's region, endpoint, credentials and streaming flag. The
+        output check enters by its import path, so editing the check's body
+        keeps the fingerprint.
         """
         return plan_fingerprint(self)
 

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/config.py

@@ -28,6 +28,7 @@ from loom.ai.errors import (
     endpoint_auth_missing,
     inference_target_incomplete,
     max_agent_depth_invalid,
+    max_concurrent_runs_invalid,
     mcp_auth_conflict,
     mcp_auth_strategy_unknown,
     mcp_credentials_inline,
@@ -343,7 +344,8 @@ class AiConfig(LoomFrozenStruct, frozen=True, kw_only=True):
         prices: Prices keyed by vendor model id (``InferenceTarget.model``).
             A plan bound to a listed model is priced with these rates rather
             than the engine's own, which is what keeps ``policies.max_usd``
-            enforceable for a model the engine cannot price.
+            enforceable for a model the engine cannot price. A key no
+            binding of ``models`` names is logged as a warning.
         skills_root: Filesystem root bare skill library names resolve against.
         mcp_servers: Named remote MCP servers artifacts refer to by name.
         a2a_agents: Named remote A2A agents artifacts refer to by name.
@@ -361,7 +363,8 @@ class AiConfig(LoomFrozenStruct, frozen=True, kw_only=True):
             that did open and whose tool listing times out still fails
             start-up, and no client becomes lazy — one that never opened is
             not reconnected later.
-        max_concurrent_runs: Per-worker run limit (FR-033a).
+        max_concurrent_runs: Per-worker run limit (FR-033a); below 1 the
+            config fails with ``MAX_CONCURRENT_RUNS_INVALID``.
         max_prompt_bytes: Enforced while reading the request body, against
             the ``prompt`` field alone (FR-015).
         max_state_bytes: Enforced while reading the request body, against the
@@ -422,8 +425,21 @@ class AiConfig(LoomFrozenStruct, frozen=True, kw_only=True):
             issues.append(remote_clients_unknown(self.remote_clients, _REMOTE_CLIENTS_MODES))
         if self.max_agent_depth < 1:
             issues.append(max_agent_depth_invalid(self.max_agent_depth))
+        if self.max_concurrent_runs < 1:
+            issues.append(max_concurrent_runs_invalid(self.max_concurrent_runs))
         if issues:
             raise AgentCompilationError(issues)
+        _warn_unused_prices(self.prices, self.models)
+
+
+def _warn_unused_prices(
+    prices: Mapping[str, ModelPrice], models: Mapping[str, InferenceTarget]
+) -> None:
+    bound = {target.model for target in models.values()}
+    for model in sorted(prices.keys() - bound):
+        _logger.warning(
+            "ai.prices[%r] names no model of ai.models; no run is priced with it", model
+        )
 
 
 _UrlIssue = Callable[[str, str, str], AgentCompilationIssue]

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/engines/pydantic_ai/_limits.py

@@ -1,4 +1,4 @@
-"""Spend caps projected onto the engine's own ``UsageLimits``.
+"""Spend caps projected onto the engine's own ``UsageLimits``, and the pricing of its responses.
 
 ``PolicySpec`` carries five caps (FR-040), four of them optional and one,
 ``max_requests``, always set to a default; :func:`usage_limits` projects all

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/errors.py

@@ -155,6 +155,7 @@ class AgentErrorCode(StrEnum):
     AGENT_SPECS_ESCAPE_ROOT = "AGENT_SPECS_ESCAPE_ROOT"
     REMOTE_CLIENTS_UNKNOWN = "REMOTE_CLIENTS_UNKNOWN"
     MAX_AGENT_DEPTH_INVALID = "MAX_AGENT_DEPTH_INVALID"
+    MAX_CONCURRENT_RUNS_INVALID = "MAX_CONCURRENT_RUNS_INVALID"
 
     # Use-case agent markers (model-as-actor)
     AGENT_MARKER_UNKNOWN = "AGENT_MARKER_UNKNOWN"
@@ -1317,6 +1318,23 @@ def max_agent_depth_invalid(value: int) -> AgentCompilationIssue:
     )
 
 
+def max_concurrent_runs_invalid(value: int) -> AgentCompilationIssue:
+    """The run limit admits no run.
+
+    Args:
+        value: The rejected value of ``ai.max_concurrent_runs``.
+    """
+    return AgentCompilationIssue(
+        code=AgentErrorCode.MAX_CONCURRENT_RUNS_INVALID,
+        message=(
+            f"ai.max_concurrent_runs: {value} is below the minimum of 1; a worker "
+            "with no run slot refuses every run and an ETL batch answers no row"
+        ),
+        component="ai",
+        field="ai.max_concurrent_runs",
+    )
+
+
 # ---------------------------------------------------------------------------
 # Compatibility factories
 # ---------------------------------------------------------------------------

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/etl/_batch.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 import asyncio
 import logging
-from collections.abc import Callable, Mapping, Sequence
+from collections.abc import Callable, Iterator, Mapping, Sequence
 from dataclasses import dataclass
 from decimal import Decimal
 from typing import Any, Final
@@ -58,21 +58,43 @@ class Batch:
     to_builtins: Callable[[Any], Mapping[str, Any]]
 
     async def answer_all(self, prompts: Sequence[str | None]) -> list[RowOutcome]:
-        """Enter the runtime, answer every prompt in order, and close it in the same task."""
-        slots = asyncio.Semaphore(self.limit)
-        identity = Identity(subject=f"etl:{self.name}")
-        async with self.runtime:
-            return list(await asyncio.gather(*(self._answer(p, slots, identity) for p in prompts)))
+        """Enter the runtime, answer every prompt in order, and close it in the same task.
+
+        *limit* workers take the prompts in order from one shared iterator, so
+        the batch creates at most *limit* tasks whatever the number of rows.
 
-    async def _answer(
-        self, prompt: str | None, slots: asyncio.Semaphore, identity: Identity
-    ) -> RowOutcome:
+        Raises:
+            RuntimeError: A run was cancelled while the batch was not, so its
+                worker stopped and left rows without an outcome.
+        """
+        identity = Identity(subject=f"etl:{self.name}")
+        outcomes: list[RowOutcome | None] = [None] * len(prompts)
+        pending = iter(enumerate(prompts))
+        async with self.runtime, asyncio.TaskGroup() as workers:
+            for _ in range(min(self.limit, len(prompts))):
+                workers.create_task(self._work(pending, outcomes, identity))
+        if None in outcomes:
+            raise RuntimeError(
+                f"agent {self.name!r} left {outcomes.count(None)} row(s) without an outcome, the "
+                f"first row {outcomes.index(None)}: a run was cancelled while the batch was not"
+            )
+        return [outcome for outcome in outcomes if outcome is not None]
+
+    async def _work(
+        self,
+        pending: Iterator[tuple[int, str | None]],
+        outcomes: list[RowOutcome | None],
+        identity: Identity,
+    ) -> None:
+        for row, prompt in pending:
+            outcomes[row] = await self._answer(prompt, identity)
+
+    async def _answer(self, prompt: str | None, identity: Identity) -> RowOutcome:
         if prompt is None:
             return RowOutcome(output=None, error=PROMPT_MISSING, usage=None)
-        async with slots:
-            if not self.ledger.reserve(self.reservation):
-                return RowOutcome(output=None, error=BUDGET_EXHAUSTED, usage=None)
-            return await self._run(prompt, identity)
+        if not self.ledger.reserve(self.reservation):
+            return RowOutcome(output=None, error=BUDGET_EXHAUSTED, usage=None)
+        return await self._run(prompt, identity)
 
     async def _run(self, prompt: str, identity: Identity) -> RowOutcome:
         usage: AgentUsage | None = None

{loom_kernel-2.18.0 → loom_kernel-2.19.0}/src/loom/ai/etl/_columns.py

@@ -13,7 +13,7 @@ import msgspec
 import msgspec.inspect as mi
 import polars as pl
 
-from loom.core.model import loom_type
+from loom.core.model import is_pydantic_model, loom_type
 
 DECIMAL_DTYPE: Final = pl.Decimal(38, 9)
 
@@ -42,9 +42,14 @@ def output_columns(output_type: type[Any]) -> dict[str, Column]:
     and integer choices ``Int64``; ``float`` ``Float64``; ``bool``
     ``Boolean``; ``date`` ``Date``; ``datetime`` a UTC ``Datetime`` (naive when
     the field requires naive values, a naive value otherwise read as UTC);
-    ``Decimal`` ``Decimal(38, 9)``; lists and sets ``List``; nested structs
-    ``Struct``; an optional field the type of its value. Any other shape is
-    ``String`` holding the JSON of the value.
+    ``Decimal`` ``Decimal(38, 9)``; ``UUID``, ``time``, ``timedelta`` and
+    ``bytes`` ``String`` holding the text the output type serialises them to;
+    lists and sets ``List``; nested structs and pydantic models ``Struct``;
+    an optional field the type of its value. A struct or model nested in
+    itself, directly or through other types, is ``String`` holding the JSON
+    of the value from that repetition down. Any other shape (a union of
+    several types, ``Any``, ``dict``, a fixed-length tuple, a ``Literal``
+    mixing value types) is ``String`` holding the JSON of the value.
 
     Args:
         output_type: A strict ``msgspec.Struct`` or ``pydantic.BaseModel``.
@@ -52,39 +57,58 @@ def output_columns(output_type: type[Any]) -> dict[str, Column]:
     Returns:
         The columns, in field order.
     """
-    return {name: _column(mi.type_info(annotation)) for name, annotation in _fields(output_type)}
+    path = frozenset({output_type})
+    return {
+        name: _column(mi.type_info(annotation), path) for name, annotation in _fields(output_type)
+    }
 
 
 def _fields(output_type: type[Any]) -> Iterable[tuple[str, Any]]:
     if loom_type(output_type).library == "msgspec":
         return [(f.encode_name, f.type) for f in msgspec.structs.fields(output_type)]
+    return _model_fields(output_type)
+
+
+def _model_fields(model: type[Any]) -> list[tuple[str, Any]]:
     return [
         (info.serialization_alias or info.alias or name, info.annotation)
-        for name, info in output_type.model_fields.items()
+        for name, info in model.model_fields.items()
     ]
 
 
-def _column(info: mi.Type) -> Column:
+def _column(info: mi.Type, path: frozenset[type[Any]]) -> Column:
     if isinstance(info, mi.Metadata):
-        return _column(info.type)
+        return _column(info.type, path)
     if isinstance(info, mi.UnionType):
-        return _optional(info)
+        return _optional(info, path)
     if isinstance(info, mi.LiteralType):
         return _choice(info.values)
     if isinstance(info, mi.EnumType):
         return _choice(tuple(member.value for member in info.cls))
     if isinstance(info, mi.CollectionType):
-        return _list(_column(info.item_type))
+        return _list(_column(info.item_type, path))
     if isinstance(info, mi.StructType | mi.DataclassType | mi.TypedDictType):
-        return _struct({f.encode_name: _column(f.type) for f in info.fields})
+        return _nested(info.cls, [(f.encode_name, f.type) for f in info.fields], path)
+    if isinstance(info, mi.CustomType) and is_pydantic_model(info.cls):
+        fields = [(name, mi.type_info(a)) for name, a in _model_fields(info.cls)]
+        return _nested(info.cls, fields, path)
     if isinstance(info, mi.DateTimeType):
         return Column(pl.Datetime("us", None if info.tz is False else "UTC"), _datetime)
     return _SCALARS.get(type(info), _JSON)
 
 
-def _optional(info: mi.UnionType) -> Column:
+def _nested(
+    cls: type[Any], fields: Iterable[tuple[str, mi.Type]], path: frozenset[type[Any]]
+) -> Column:
+    if cls in path:
+        return _JSON
+    inner = path | {cls}
+    return _struct({name: _column(info, inner) for name, info in fields})
+
+
+def _optional(info: mi.UnionType, path: frozenset[type[Any]]) -> Column:
     values = [member for member in info.types if not isinstance(member, mi.NoneType)]
-    return _column(values[0]) if len(values) == 1 else _JSON
+    return _column(values[0], path) if len(values) == 1 else _JSON
 
 
 def _choice(values: tuple[Any, ...]) -> Column:
@@ -122,18 +146,23 @@ def _decimal(value: str | float | int) -> Decimal:
 
 
 def _json(value: Any) -> str:
-    return value if isinstance(value, str) else msgspec.json.encode(value).decode()
+    return msgspec.json.encode(value).decode()
 
 
 _JSON: Final = Column(pl.String(), _json)
+_TEXT: Final = Column(pl.String(), _same)
 
 _SCALARS: Final[Mapping[type[mi.Type], Column]] = MappingProxyType(
     {
-        mi.StrType: Column(pl.String(), _same),
+        mi.StrType: _TEXT,
         mi.IntType: Column(pl.Int64(), _same),
         mi.FloatType: Column(pl.Float64(), float),
         mi.BoolType: Column(pl.Boolean(), _same),
         mi.DateType: Column(pl.Date(), date.fromisoformat),
         mi.DecimalType: Column(DECIMAL_DTYPE, _decimal),
+        mi.UUIDType: _TEXT,
+        mi.TimeType: _TEXT,
+        mi.TimeDeltaType: _TEXT,
+        mi.BytesType: _TEXT,
     }
 )