loom-kernel 0.5.0__tar.gz → 0.7.0__tar.gz

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/CHANGELOG.md

@@ -1,3 +1,219 @@
+# 🚀 Release 0.7.0 ([#25](https://github.com/the-reacher-data/loom-py/pull/25)) ([`0d8941a`](https://github.com/the-reacher-data/loom-py/commit/0d8941a724ec768b162a07e756bac3d53157ec85))
+
+
+## ✨ Features
+### config
+- **config:** add StructBinder and migrate streaming binding resolution<br>
+  > Introduces StructBinder in core/config/binder.py — a Strategy that
+  > injects constructor arguments from a config mapping via msgspec.convert,
+  > covering primitives, Literal constraints, and LoomFrozenStruct subclasses.
+  > Migrates streaming _instantiate_binding to use it, removing ~30 lines of
+  > private helpers and adding typed ConfigError on resolution failures.
+
+- **config:** add ConfigContext for typed section extraction and binding resolution<br>
+  > Single entry-point for runner and bootstrap code to read config:
+  > section() extracts typed sub-trees, bind() injects constructor args
+  > from a config path + overrides, resolve() materializes ConfigBinding
+  > declarations. Accepts an optional StructBinder for strict-mode control.
+
+
+
+## 🐛 Fixes
+### core
+- **core:** clean async bridge timeout and celery typing
+
+### etl
+- **etl:** align missing storage error with config contract
+
+
+
+
+## ♻️ Refactor
+### model
+- **model:** migrate public config structs to LoomFrozenStruct
+
+### streaming
+- **streaming:** clean up _instantiate_binding before extraction<br>
+  > Replace mutable dict() + .update() with a single dict unpacking
+  > Remove the dead param.annotation fallback (get_type_hints already
+  > resolves all forward refs; the fallback silently passed strings to
+  > _is_struct_annotation which always returned False)
+  > Extract _SKIP_KINDS constant to name the variadic-parameter guard
+  > Raise ConfigError instead of TypeError for missing required struct
+  > params (semantically a config error, not a type error; both are
+  > caught by the enclosing _resolve_binding handler so no behaviour
+  > change in streaming)
+  > All 7 binding tests pass.
+
+
+### core
+- **core:** homogenize config and runner boundaries
+- **core:** add runner and compiler protocols
+- **core:** apply runner lifecycle protocols
+- **core:** harden runner lifecycle abstraction<br>
+  > shutdown_runner/flush_runner now catch and log exceptions instead of
+  > propagating, preserving the original exception when called from finally
+  > blocks; contract updated in docstrings and covered by new tests
+  > CompilerProtocol drops @runtime_checkable — no production isinstance
+  > usage existed; Protocol remains valid for static typing
+  > StreamingRunner.prepare_run() calls shutdown_runner(self) before
+  > overwriting self._shutdown, preventing resource leaks on double calls
+  > _build_backend_options extracted from loom.celery.config into
+  > loom.core.async_bridge.build_backend_options, eliminating duplication
+  > between the Celery and streaming domains
+  > _CeleryAsyncRuntime._signals_connected ClassVar removed; the
+  > module-level _SIGNALS_CONNECTED flag is the single source of truth
+  > for the process-level signal guard
+  > ETLRunner.flush() delegates to ETLExecutor.flush(), which checks
+  > SupportsFlush via isinstance instead of getattr duck-typing
+  > TypeVars in compiler.py renamed to _RequestT/_PlanT to signal they
+  > are implementation details of the Protocol, not public API
+
+
+### etl
+- **etl:** make flush an optional runner capability
+
+### celery
+- **celery:** adopt runner shutdown helper
+- **celery:** replace _SIGNALS_CONNECTED sentinel with dispatch_uid<br>
+  > Eliminates the module-level mutable boolean that violated the no-global-
+  > mutable-state architecture rule. Idempotency is now handled by Celery's
+  > dispatch_uid mechanism: disconnect+connect on each call ensures the most-
+  > recently registered closure is always active, so a second bootstrap_worker
+  > call wires up fresh dependencies rather than leaving stale closures from
+  > the first call.
+  > This is strictly better than the sentinel: no global state, no race
+  > condition between threads calling bootstrap_worker concurrently, and
+  > correct closure replacement when called multiple times in tests.
+
+
+### rest
+- **rest:** use observability config directly
+
+
+
+
+
+
+
+# 🚀 Release 0.6.0 ([#22](https://github.com/the-reacher-data/loom-py/pull/22)) ([`5146569`](https://github.com/the-reacher-data/loom-py/commit/51465697115036ed05f620a1099272e4fd216501))
+
+
+## ✨ Features
+### core
+- **core:** add unified ObservabilityRuntime<br>
+  > Introduces the core observability package: LifecycleEvent, LifecycleObserver
+  > protocol, ObservabilityConfig, and ObservabilityRuntime as a single fan-out
+  > engine replacing the fragmented per-module observer wiring. Adds
+  > StructlogLifecycleObserver, OtelLifecycleObserver, NoopObserver, and
+  > PrometheusLifecycleAdapter (with Pushgateway support for ETL batch jobs).
+  > Backward-compat re-exports (safe_observe, notify_observers) kept for the
+  > streaming composite observer until the legacy cleanup commit.
+
+
+
+## 🐛 Fixes
+### observability
+- **observability:** restore default bootstraps
+
+### rest
+- **rest:** correct camel-to-snake conversion for acronyms in filter fields
+
+### streaming
+- **streaming:** propagate trace ids through bytewax paths
+- **streaming:** keep input trace ids through the micro
+- **streaming:** bridge otel trace ids from messages
+
+
+## 📖 Documentation
+- align yaml config and dummy repo links
+- restore dummy repo urls
+- fix markdown links for streaming dummies
+
+### prometheus
+- **prometheus:** update KafkaPrometheusMetrics docstring metric names
+
+
+
+## ♻️ Refactor
+### observability
+- **observability:** simplify otel log correlation
+- **observability:** simplify otel exporter imports
+- **observability:** expose ObservabilityRuntime in public API and clean architecture debt<br>
+  > Export ObservabilityRuntime and LifecycleStatus from loom.core.observability package
+  > Compute dict(meta) once in span() instead of three separate allocations
+  > Replace _SIGNALS_CONNECTED module-level global with ClassVar on _CeleryAsyncRuntime
+  > Add RuntimeError to bootstrap_worker Raises docstring
+  > Fix TraceIdMiddleware docstring example (removed spurious ObservabilityRuntime reference)
+  > Add missing observability_runtime param to _make_handler docstring
+
+- **observability:** unify runtime across services
+- **observability:** clean up dead code and docs
+
+### streaming
+- **streaming:** migrate to unified ObservabilityRuntime<br>
+  > Replace StreamingObservabilityConfig and the manual observer construction
+  > in _load_observability_runtime() with ObservabilityRuntime.from_config().
+  > Observability config now lives under streaming.runtime.observability in YAML,
+  > eliminating the separate streaming.observability section.
+  > Delete src/loom/streaming/_observability.py (StreamingObservabilityConfig)
+  > Delete src/loom/streaming/observability/ directory (old observer stack)
+  > Add observability: ObservabilityConfig field to BytewaxRuntimeConfig
+  > StreamingRunner.from_config() calls ObservabilityRuntime.from_config() directly
+  > StreamingTestRunner defaults to ObservabilityRuntime.noop(); callers pass
+  > observability_runtime= explicitly when needed
+  > Update tests to use new config shape and ObservabilityRuntime([observer])
+
+- **streaming:** replace KafkaStreamingObserver with LifecycleEvent/TRANSPORT<br>
+  > Eliminates the parallel KafkaStreamingObserver protocol hierarchy (NoopKafkaObserver,
+  > StructlogKafkaObserver) and models all Kafka transport events as LifecycleEvent with
+  > Scope.TRANSPORT. KafkaPrometheusMetrics now implements the LifecycleObserver protocol
+  > via a single on_event() dispatcher instead of four typed callback methods.
+
+- **streaming:** preserve trace lineage across boundaries
+
+### prometheus
+- **prometheus:** add KafkaMetricName enum and drop loom_ prefix<br>
+  > Extracts Prometheus metric names into a public KafkaMetricName StrEnum so
+  > callers can reference metric names without magic strings. Removes the loom_
+  > namespace prefix from all four Kafka instruments (produced_total,
+  > consumed_total, encode_duration_seconds, decode_duration_seconds).
+
+- **prometheus:** drop loom prefix from lifecycle metrics
+
+### rest
+- **rest:** use core observability runtime
+- **rest:** read prometheus from observability config
+
+### celery
+- **celery:** adopt async bridge and runtime config
+
+### etl
+- **etl:** make spark pytest plugin opt-in
+
+
+
+## ✅ Tests
+### kafka
+- **kafka:** update metric name assertions after loom_ prefix removal
+
+### observability
+- **observability:** cover lineage and runtime branches
+
+### integration
+- **integration:** add in-memory REST, observability, and bootstrap integration tests
+
+### etl
+- **etl:** cover prometheus flush on runner shutdown
+
+
+## 🛠 Chores
+### deps
+- **deps:** bump click to 8.3.3
+
+
+
+
 # 🚀 Release 0.5.0 ([#20](https://github.com/the-reacher-data/loom-py/pull/20)) ([`70f7cf8`](https://github.com/the-reacher-data/loom-py/commit/70f7cf85275fcc1f590e06f980dfac91cad50893))
 
 

loom_kernel-0.7.0/CHANGELOG_RELEASE.md

@@ -0,0 +1,96 @@
+# 🚀 Release 0.7.0 ([#25](https://github.com/the-reacher-data/loom-py/pull/25)) ([`0d8941a`](https://github.com/the-reacher-data/loom-py/commit/0d8941a724ec768b162a07e756bac3d53157ec85))
+
+
+## ✨ Features
+### config
+- **config:** add StructBinder and migrate streaming binding resolution<br>
+  > Introduces StructBinder in core/config/binder.py — a Strategy that
+  > injects constructor arguments from a config mapping via msgspec.convert,
+  > covering primitives, Literal constraints, and LoomFrozenStruct subclasses.
+  > Migrates streaming _instantiate_binding to use it, removing ~30 lines of
+  > private helpers and adding typed ConfigError on resolution failures.
+
+- **config:** add ConfigContext for typed section extraction and binding resolution<br>
+  > Single entry-point for runner and bootstrap code to read config:
+  > section() extracts typed sub-trees, bind() injects constructor args
+  > from a config path + overrides, resolve() materializes ConfigBinding
+  > declarations. Accepts an optional StructBinder for strict-mode control.
+
+
+
+## 🐛 Fixes
+### core
+- **core:** clean async bridge timeout and celery typing
+
+### etl
+- **etl:** align missing storage error with config contract
+
+
+
+
+## ♻️ Refactor
+### model
+- **model:** migrate public config structs to LoomFrozenStruct
+
+### streaming
+- **streaming:** clean up _instantiate_binding before extraction<br>
+  > Replace mutable dict() + .update() with a single dict unpacking
+  > Remove the dead param.annotation fallback (get_type_hints already
+  > resolves all forward refs; the fallback silently passed strings to
+  > _is_struct_annotation which always returned False)
+  > Extract _SKIP_KINDS constant to name the variadic-parameter guard
+  > Raise ConfigError instead of TypeError for missing required struct
+  > params (semantically a config error, not a type error; both are
+  > caught by the enclosing _resolve_binding handler so no behaviour
+  > change in streaming)
+  > All 7 binding tests pass.
+
+
+### core
+- **core:** homogenize config and runner boundaries
+- **core:** add runner and compiler protocols
+- **core:** apply runner lifecycle protocols
+- **core:** harden runner lifecycle abstraction<br>
+  > shutdown_runner/flush_runner now catch and log exceptions instead of
+  > propagating, preserving the original exception when called from finally
+  > blocks; contract updated in docstrings and covered by new tests
+  > CompilerProtocol drops @runtime_checkable — no production isinstance
+  > usage existed; Protocol remains valid for static typing
+  > StreamingRunner.prepare_run() calls shutdown_runner(self) before
+  > overwriting self._shutdown, preventing resource leaks on double calls
+  > _build_backend_options extracted from loom.celery.config into
+  > loom.core.async_bridge.build_backend_options, eliminating duplication
+  > between the Celery and streaming domains
+  > _CeleryAsyncRuntime._signals_connected ClassVar removed; the
+  > module-level _SIGNALS_CONNECTED flag is the single source of truth
+  > for the process-level signal guard
+  > ETLRunner.flush() delegates to ETLExecutor.flush(), which checks
+  > SupportsFlush via isinstance instead of getattr duck-typing
+  > TypeVars in compiler.py renamed to _RequestT/_PlanT to signal they
+  > are implementation details of the Protocol, not public API
+
+
+### etl
+- **etl:** make flush an optional runner capability
+
+### celery
+- **celery:** adopt runner shutdown helper
+- **celery:** replace _SIGNALS_CONNECTED sentinel with dispatch_uid<br>
+  > Eliminates the module-level mutable boolean that violated the no-global-
+  > mutable-state architecture rule. Idempotency is now handled by Celery's
+  > dispatch_uid mechanism: disconnect+connect on each call ensures the most-
+  > recently registered closure is always active, so a second bootstrap_worker
+  > call wires up fresh dependencies rather than leaving stale closures from
+  > the first call.
+  > This is strictly better than the sentinel: no global state, no race
+  > condition between threads calling bootstrap_worker concurrently, and
+  > correct closure replacement when called multiple times in tests.
+
+
+### rest
+- **rest:** use observability config directly
+
+
+
+
+

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loom-kernel
-Version: 0.5.0
+Version: 0.7.0
 Summary: Loom Python project
 License-File: LICENSE
 Requires-Python: >=3.11
@@ -396,8 +396,13 @@ app:
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
 
-metrics:
-  enabled: false
+observability:
+  log:
+    enabled: false
+  otel:
+    enabled: false
+  prometheus:
+    enabled: false
 ```
 
 ```python

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/README.md

@@ -329,8 +329,13 @@ app:
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
 
-metrics:
-  enabled: false
+observability:
+  log:
+    enabled: false
+  otel:
+    enabled: false
+  prometheus:
+    enabled: false
 ```
 
 ```python

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/conf.py

@@ -69,7 +69,7 @@ def _skip_duplicate_reexports(
         ("loom.core.errors", "RuleViolations"),
         ("loom.core.use_case", "RuleViolation"),
         ("loom.core.use_case", "RuleViolations"),
-        ("loom.etl.observability", "OtelConfig"),
+        ("loom.etl.lineage", "ETLObservabilityConfig"),
         ("loom.testing", "CompilationError"),
         ("loom.rest.model", "PaginationMode"),
     }

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/etl/pipelines.md

@@ -165,9 +165,15 @@ storage:
   tmp_root: /var/lib/loom/lake/_tmp
 
 observability:
-  log: true
-  slow_step_threshold_ms: 30000
-  run_sink:
+  log:
+    enabled: true
+  otel:
+    enabled: false
+  prometheus:
+    enabled: true
+    pushgateway_url: ${oc.env:PUSHGATEWAY_URL,http://127.0.0.1:9091}
+  lineage:
+    enabled: true
     # Choose exactly one destination:
     root: /var/lib/loom/lake/_runs
     # database: ops

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/etl/testing.md

@@ -42,7 +42,8 @@ def test_double_amount(loom_polars_runner):
     result.assert_count(2)
 ```
 
-Pytest fixtures auto-registered: `loom_polars_runner`, `loom_spark_runner`.
+Pytest fixtures: `loom_polars_runner`. Import `loom.etl.testing.spark` or
+register it explicitly in Spark-enabled suites to use `loom_spark_runner`.
 
 ## Spark integration tests
 

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/examples-repo/index.md

@@ -578,12 +578,15 @@ app:
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///./store.db}
 
-trace:
-  enabled: ${oc.decode:${oc.env:TRACE_ENABLED,true}}
-
-metrics:
-  enabled: ${oc.decode:${oc.env:METRICS_ENABLED,true}}
-  path: /metrics
+observability:
+  log:
+    enabled: true
+  otel:
+    enabled: false
+  prometheus:
+    enabled: ${oc.decode:${oc.env:METRICS_ENABLED,true}}
+    config:
+      path: /metrics
 ```
 
 ### Manifest mode (explicit registry)

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/getting-started/etl.md

@@ -90,9 +90,15 @@ storage:
   tmp_root: /var/lib/loom/lake/_tmp
 
 observability:
-  log: true
-  slow_step_threshold_ms: 30000
-  run_sink:
+  log:
+    enabled: true
+  otel:
+    enabled: false
+  prometheus:
+    enabled: true
+    pushgateway_url: ${oc.env:PUSHGATEWAY_URL,http://127.0.0.1:9091}
+  lineage:
+    enabled: true
     root: /var/lib/loom/lake/_runs
 ```
 

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/getting-started/rest.md

@@ -58,6 +58,12 @@ app:
 
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
+
+observability:
+  log:
+    enabled: false
+  otel:
+    enabled: false
 ```
 
 ```python
@@ -330,12 +336,15 @@ app:
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///store.db}
 
-metrics:
-  enabled: true
-  path: /metrics
-
-trace:
-  enabled: true
+observability:
+  log:
+    enabled: true
+  otel:
+    enabled: false
+  prometheus:
+    enabled: true
+    config:
+      path: /metrics
 ```
 
 ```python

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/getting-started/streaming.md

@@ -4,7 +4,7 @@
 steps, batching, branching, and transport wiring.
 
 The companion demo repository for this subsystem is
-`dummy-loom-streaming <https://github.com/the-reacher-data/dummy-loom-streaming>`_.
+[dummy-loom-streaming](https://github.com/the-reacher-data/dummy-loom-streaming).
 
 ## Install
 
@@ -94,4 +94,4 @@ Use `loom.streaming.nodes`, `loom.streaming.kafka`, and
 surface.
 
 For a runnable end-to-end example, see the companion repository:
-`dummy-loom-streaming <https://github.com/the-reacher-data/dummy-loom-streaming>`_.
+[dummy-loom-streaming](https://github.com/the-reacher-data/dummy-loom-streaming).

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/guides/celery.md

@@ -282,6 +282,12 @@ celery:
 
   # For integration tests: run tasks synchronously in the calling process.
   task_always_eager: false
+
+  # Async bridge settings used by async jobs/callbacks.
+  runtime:
+    backend: asyncio          # asyncio or trio
+    use_uvloop: true          # only applies to asyncio on supported platforms
+    shutdown_timeout_ms: 10000
 ```
 
 (per-job-overrides-from-yaml)=
@@ -497,6 +503,9 @@ celery:
   broker_url: "redis://localhost:6379/15"
   result_backend: "redis://localhost:6379/15"
   task_always_eager: true
+  runtime:
+    backend: asyncio
+    use_uvloop: false
 ```
 
 Or patch it in a pytest fixture:

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/guides/etl.md

@@ -165,9 +165,15 @@ storage:
   tmp_root: /var/lib/loom/lake/_tmp
 
 observability:
-  log: true
-  slow_step_threshold_ms: 30000
-  run_sink:
+  log:
+    enabled: true
+  otel:
+    enabled: false
+  prometheus:
+    enabled: true
+    pushgateway_url: ${oc.env:PUSHGATEWAY_URL,http://127.0.0.1:9091}
+  lineage:
+    enabled: true
     # Choose exactly one destination:
     root: /var/lib/loom/lake/_runs
     # database: ops

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/overview.md

@@ -19,7 +19,7 @@ design rules:
 - prefer explicit contracts over reflection-heavy magic
 
 The companion demo repository for the streaming subsystem is
-`dummy-loom-streaming <https://github.com/the-reacher-data/dummy-loom-streaming>`_.
+[dummy-loom-streaming](https://github.com/the-reacher-data/dummy-loom-streaming).
 
 ## At a glance
 

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/reference/api/etl.rst

@@ -28,7 +28,7 @@ Execution Infrastructure
 
    loom.etl.storage
    loom.etl.checkpoint
-   loom.etl.observability
+   loom.etl.lineage
 
 Backends
 --------

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/rest/celery.md

@@ -282,6 +282,12 @@ celery:
 
   # For integration tests: run tasks synchronously in the calling process.
   task_always_eager: false
+
+  # Async bridge settings used by async jobs/callbacks.
+  runtime:
+    backend: asyncio          # asyncio or trio
+    use_uvloop: true          # only applies to asyncio on supported platforms
+    shutdown_timeout_ms: 10000
 ```
 
 (rest-per-job-overrides-from-yaml)=
@@ -497,6 +503,9 @@ celery:
   broker_url: "redis://localhost:6379/15"
   result_backend: "redis://localhost:6379/15"
   task_always_eager: true
+  runtime:
+    backend: asyncio
+    use_uvloop: false
 ```
 
 Or patch it in a pytest fixture:

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/rest/examples.md

@@ -578,12 +578,15 @@ app:
 database:
   url: ${oc.env:DATABASE_URL,sqlite+aiosqlite:///./store.db}
 
-trace:
-  enabled: ${oc.decode:${oc.env:TRACE_ENABLED,true}}
-
-metrics:
-  enabled: ${oc.decode:${oc.env:METRICS_ENABLED,true}}
-  path: /metrics
+observability:
+  log:
+    enabled: false
+  otel:
+    enabled: false
+  prometheus:
+    enabled: ${oc.decode:${oc.env:METRICS_ENABLED,true}}
+    config:
+      path: /metrics
 ```
 
 ### Manifest mode (explicit registry)

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/docs/streaming/bytewax.md

@@ -4,7 +4,7 @@
 `StreamFlow` into a real Bytewax dataflow.
 
 For a runnable end-to-end reference implementation, see:
-`dummy-loom-streaming <https://github.com/the-reacher-data/dummy-loom-streaming>`_.
+[dummy-loom-streaming](https://github.com/the-reacher-data/dummy-loom-streaming).
 
 ## What it does
 

{loom_kernel-0.5.0 → loom_kernel-0.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "loom-kernel"
-version = "0.5.0"
+version = "0.7.0"
 description = "Loom Python project"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -79,9 +79,6 @@ celery = [
     "redis>=5.0,<6.0",
 ]
 
-[project.entry-points.pytest11]
-loom-spark = "loom.etl.testing.spark"
-
 [project.entry-points."loom.etl.backends"]
 polars = "loom.etl.backends.polars.provider:PolarsProvider"
 spark = "loom.etl.backends.spark.provider:SparkProvider"
@@ -223,7 +220,7 @@ markers = [
 
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.5.0"
+version = "0.7.0"
 tag_format = "v$version"
 version_files = [
     "pyproject.toml:project.version",