query-cascade 0.2.2__tar.gz → 0.2.3__tar.gz

{query_cascade-0.2.2 → query_cascade-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: query-cascade
-Version: 0.2.2
+Version: 0.2.3
 Summary: Minimal demand-driven query framework for incremental computation.
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -212,7 +212,7 @@ Properties and methods:
 - **`accumulator(name: str) → Accumulator`** — Return a named **`Accumulator`** (see below). Names are keys in the optional **`effects`** map passed to synchronous **`QueryHandle`** calls and **`submit`**.
 - **`snapshot() → Snapshot`** — Capture **`Snapshot(revision=engine.revision)`** for use as **`snapshot=`** on reads. Reads through that snapshot see input and memo state as of that revision; live **`set`** calls do not disturb snapshot reads.
 - **`submit(query, *args, *, snapshot=None, effects=None, executor=None) → concurrent.futures.Future`** — Run **`query(*args)`** on **`executor`**, or on a **lazily created** per-engine **`ThreadPoolExecutor`** when **`executor` is `None`**. The engine captures **`cancel_epoch`** at schedule time; if inputs move on before the task runs, the future may complete with **`QueryCancelled`**. **`snapshot`** defaults to a fresh **`snapshot()`** at submit time when omitted.
-- **`compute_many(calls, *, workers=None, snapshot=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. There is no **`effects`** parameter on this path—use **`QueryHandle.__call__`** or **`submit`** if you need to collect accumulator output in a dict.
+- **`compute_many(calls, *, workers=None, snapshot=None, effects=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. When **`effects`** is a dict, accumulator output is collected and appended into **`effects[name]`** deterministically in **call order** (not completion order).
 - **`shutdown(*, wait: bool = True, cancel_futures: bool = False)`** — Shut down the **default** thread pool created by **`submit`**, if any. Call when discarding the engine if you need threads torn down promptly (for example in tests).
 - **`traces() → list[TraceEvent]`** — Copy of recent trace events (subject to **`trace_limit`**).
 - **`clear_traces()`** — Drop all buffered trace events.
@@ -452,6 +452,7 @@ Answer Yes/No:
 | `dynamic_macro_expansion.py` | Query that **changes downstream dependencies** at runtime |
 | `snapshot_isolation.py` | Snapshot reads while live inputs change |
 | `concurrent_background_work.py` | Dedup under concurrency + cancellation after input changes |
+| `compute_many_with_accumulators.py` | `compute_many(..., effects=...)` accumulator collection |
 | `persistence_and_inspection.py` | Save/load and graph summaries |
 | `gil_parallel_speedup.py` | Threaded CPU benchmark: GIL vs free-threaded |
 

{query_cascade-0.2.2 → query_cascade-0.2.3}/README.md

@@ -199,7 +199,7 @@ Properties and methods:
 - **`accumulator(name: str) → Accumulator`** — Return a named **`Accumulator`** (see below). Names are keys in the optional **`effects`** map passed to synchronous **`QueryHandle`** calls and **`submit`**.
 - **`snapshot() → Snapshot`** — Capture **`Snapshot(revision=engine.revision)`** for use as **`snapshot=`** on reads. Reads through that snapshot see input and memo state as of that revision; live **`set`** calls do not disturb snapshot reads.
 - **`submit(query, *args, *, snapshot=None, effects=None, executor=None) → concurrent.futures.Future`** — Run **`query(*args)`** on **`executor`**, or on a **lazily created** per-engine **`ThreadPoolExecutor`** when **`executor` is `None`**. The engine captures **`cancel_epoch`** at schedule time; if inputs move on before the task runs, the future may complete with **`QueryCancelled`**. **`snapshot`** defaults to a fresh **`snapshot()`** at submit time when omitted.
-- **`compute_many(calls, *, workers=None, snapshot=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. There is no **`effects`** parameter on this path—use **`QueryHandle.__call__`** or **`submit`** if you need to collect accumulator output in a dict.
+- **`compute_many(calls, *, workers=None, snapshot=None, effects=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. When **`effects`** is a dict, accumulator output is collected and appended into **`effects[name]`** deterministically in **call order** (not completion order).
 - **`shutdown(*, wait: bool = True, cancel_futures: bool = False)`** — Shut down the **default** thread pool created by **`submit`**, if any. Call when discarding the engine if you need threads torn down promptly (for example in tests).
 - **`traces() → list[TraceEvent]`** — Copy of recent trace events (subject to **`trace_limit`**).
 - **`clear_traces()`** — Drop all buffered trace events.
@@ -439,6 +439,7 @@ Answer Yes/No:
 | `dynamic_macro_expansion.py` | Query that **changes downstream dependencies** at runtime |
 | `snapshot_isolation.py` | Snapshot reads while live inputs change |
 | `concurrent_background_work.py` | Dedup under concurrency + cancellation after input changes |
+| `compute_many_with_accumulators.py` | `compute_many(..., effects=...)` accumulator collection |
 | `persistence_and_inspection.py` | Save/load and graph summaries |
 | `gil_parallel_speedup.py` | Threaded CPU benchmark: GIL vs free-threaded |
 

{query_cascade-0.2.2 → query_cascade-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "query-cascade"
-version = "0.2.2"
+version = "0.2.3"
 description = "Minimal demand-driven query framework for incremental computation."
 requires-python = ">=3.12"
 readme = "README.md"

{query_cascade-0.2.2 → query_cascade-0.2.3}/src/cascade/engine.py

@@ -3,7 +3,7 @@ from __future__ import annotations
 import concurrent.futures
 import os
 import threading
-from typing import Any, Callable, Iterable, Literal, Mapping, Sequence
+from typing import Any, Callable, Iterable, Iterator, Literal, Mapping, Sequence
 
 from ._errors import CancellationError, CycleError, QueryCancelled
 from ._evaluator import Evaluator
@@ -262,18 +262,95 @@ class Engine:
         *,
         workers: int | None = None,
         snapshot: Snapshot | None = None,
+        effects: dict[str, list[Any]] | None = None,
     ) -> list[Any]:
         if not calls:
             return []
         run_snapshot = snapshot or self.snapshot()
         worker_count = workers or min(32, max(1, len(calls)))
         scheduler = WorkStealingExecutor(worker_count)
+        per_call_effects: list[dict[str, list[Any]] | None]
+        if effects is None:
+            per_call_effects = [None] * len(calls)
+        else:
+            per_call_effects = [{} for _ in range(len(calls))]
         for idx, (query, args) in enumerate(calls):
             scheduler.submit_indexed(
                 idx,
-                lambda q=query, a=args: self._query_call(q.id, q.raw, a, snapshot=run_snapshot),
+                lambda q=query, a=args, e=per_call_effects[idx]: self._query_call(
+                    q.id,
+                    q.raw,
+                    a,
+                    snapshot=run_snapshot,
+                    effects=e,
+                ),
             )
-        return scheduler.run(len(calls))
+        results = scheduler.run(len(calls))
+        if effects is not None:
+            for call_effects in per_call_effects:
+                if not call_effects:
+                    continue
+                for name, items in call_effects.items():
+                    if not items:
+                        continue
+                    effects.setdefault(name, []).extend(items)
+        return results
+
+    def compute_many_stream(
+        self,
+        calls: Sequence[tuple[_QueryHandle, tuple[Any, ...]]],
+        *,
+        workers: int | None = None,
+        snapshot: Snapshot | None = None,
+        effects: dict[str, list[Any]] | None = None,
+    ) -> Iterator[tuple[int, Any, dict[str, list[Any]]]]:
+        """Yield completed results as each call finishes.
+
+        Yields ``(index, value, call_effects)`` where ``index`` matches the position in
+        ``calls`` and ``call_effects`` contains accumulator output for that call when
+        ``effects`` was provided (otherwise an empty dict).
+
+        Unlike :meth:`compute_many`, this API yields items in *completion order*.
+        If ``effects`` is provided, it is populated after all calls complete, merged
+        deterministically in call order (matching :meth:`compute_many`).
+        """
+        if not calls:
+            return
+
+        run_snapshot = snapshot or self.snapshot()
+        worker_count = workers or min(32, max(1, len(calls)))
+
+        per_call_effects: list[dict[str, list[Any]]]
+        per_call_effects = [{} for _ in range(len(calls))]
+
+        with concurrent.futures.ThreadPoolExecutor(max_workers=worker_count) as pool:
+            future_to_index: dict[concurrent.futures.Future[Any], int] = {}
+            for idx, (query, args) in enumerate(calls):
+                fut = pool.submit(
+                    self._query_call,
+                    query.id,
+                    query.raw,
+                    args,
+                    snapshot=run_snapshot,
+                    effects=per_call_effects[idx] if effects is not None else None,
+                )
+                future_to_index[fut] = idx
+
+            for fut in concurrent.futures.as_completed(future_to_index):
+                idx = future_to_index[fut]
+                value = fut.result()
+                call_effects = per_call_effects[idx] if effects is not None else {}
+                item = (idx, value, call_effects)
+                yield item
+
+        if effects is not None:
+            for call_effects in per_call_effects:
+                if not call_effects:
+                    continue
+                for name, items in call_effects.items():
+                    if not items:
+                        continue
+                    effects.setdefault(name, []).extend(items)
 
     def traces(self) -> list[TraceEvent]:
         return self._store.traces()

{query_cascade-0.2.2 → query_cascade-0.2.3}/src/query_cascade.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: query-cascade
-Version: 0.2.2
+Version: 0.2.3
 Summary: Minimal demand-driven query framework for incremental computation.
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -212,7 +212,7 @@ Properties and methods:
 - **`accumulator(name: str) → Accumulator`** — Return a named **`Accumulator`** (see below). Names are keys in the optional **`effects`** map passed to synchronous **`QueryHandle`** calls and **`submit`**.
 - **`snapshot() → Snapshot`** — Capture **`Snapshot(revision=engine.revision)`** for use as **`snapshot=`** on reads. Reads through that snapshot see input and memo state as of that revision; live **`set`** calls do not disturb snapshot reads.
 - **`submit(query, *args, *, snapshot=None, effects=None, executor=None) → concurrent.futures.Future`** — Run **`query(*args)`** on **`executor`**, or on a **lazily created** per-engine **`ThreadPoolExecutor`** when **`executor` is `None`**. The engine captures **`cancel_epoch`** at schedule time; if inputs move on before the task runs, the future may complete with **`QueryCancelled`**. **`snapshot`** defaults to a fresh **`snapshot()`** at submit time when omitted.
-- **`compute_many(calls, *, workers=None, snapshot=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. There is no **`effects`** parameter on this path—use **`QueryHandle.__call__`** or **`submit`** if you need to collect accumulator output in a dict.
+- **`compute_many(calls, *, workers=None, snapshot=None, effects=None) → list[Any]`** — Run many queries in parallel with a work-stealing scheduler. **`calls`** is a sequence of **`(query_handle, args_tuple)`**. Result order matches **`calls`**. **`workers`** defaults to a sensible value from **`len(calls)`** (capped); **`workers=0`** falls back to that same default. **`snapshot`** defaults to **`snapshot()`** when omitted. When **`effects`** is a dict, accumulator output is collected and appended into **`effects[name]`** deterministically in **call order** (not completion order).
 - **`shutdown(*, wait: bool = True, cancel_futures: bool = False)`** — Shut down the **default** thread pool created by **`submit`**, if any. Call when discarding the engine if you need threads torn down promptly (for example in tests).
 - **`traces() → list[TraceEvent]`** — Copy of recent trace events (subject to **`trace_limit`**).
 - **`clear_traces()`** — Drop all buffered trace events.
@@ -452,6 +452,7 @@ Answer Yes/No:
 | `dynamic_macro_expansion.py` | Query that **changes downstream dependencies** at runtime |
 | `snapshot_isolation.py` | Snapshot reads while live inputs change |
 | `concurrent_background_work.py` | Dedup under concurrency + cancellation after input changes |
+| `compute_many_with_accumulators.py` | `compute_many(..., effects=...)` accumulator collection |
 | `persistence_and_inspection.py` | Save/load and graph summaries |
 | `gil_parallel_speedup.py` | Threaded CPU benchmark: GIL vs free-threaded |
 

{query_cascade-0.2.2 → query_cascade-0.2.3}/tests/test_engine_compute_many.py

@@ -83,6 +83,79 @@ def test_compute_many_with_zero_workers_falls_back_to_default_worker_selection()
     assert result == [i + 1 for i in range(6)]
 
 
+def test_compute_many_can_collect_accumulator_effects() -> None:
+    engine = Engine()
+    progress = engine.accumulator("progress")
+
+    @engine.query
+    def job(name: str) -> str:
+        progress.push(("start", name))
+        progress.push(("done", name))
+        return name.upper()
+
+    effects: dict[str, list[object]] = {}
+    result = engine.compute_many([(job, ("a",)), (job, ("b",)), (job, ("c",))], workers=3, effects=effects)
+    assert result == ["A", "B", "C"]
+    assert effects == {
+        "progress": [
+            ("start", "a"),
+            ("done", "a"),
+            ("start", "b"),
+            ("done", "b"),
+            ("start", "c"),
+            ("done", "c"),
+        ]
+    }
+
+
+def test_compute_many_stream_yields_in_completion_order() -> None:
+    engine = Engine()
+
+    @engine.query
+    def delayed(i: int) -> int:
+        time.sleep(0.004 * (4 - i))
+        return i
+
+    calls = [(delayed, (i,)) for i in range(4)]
+    items = list(engine.compute_many_stream(calls, workers=4))
+    # i=3 sleeps least, so it should complete first.
+    assert [idx for idx, _value, _effects in items] == [3, 2, 1, 0]
+    assert sorted(value for _idx, value, _effects in items) == [0, 1, 2, 3]
+
+
+def test_compute_many_stream_can_yield_and_collect_accumulator_effects() -> None:
+    engine = Engine()
+    progress = engine.accumulator("progress")
+
+    @engine.query
+    def job(name: str, seconds: float) -> str:
+        progress.push(("start", name))
+        time.sleep(seconds)
+        progress.push(("done", name))
+        return name.upper()
+
+    calls = [(job, ("a", 0.03)), (job, ("b", 0.01)), (job, ("c", 0.02))]
+    effects: dict[str, list[object]] = {}
+    items = list(engine.compute_many_stream(calls, workers=3, effects=effects))
+
+    # Stream yields in completion order (b, c, a).
+    assert [value for _idx, value, _call_effects in items] == ["B", "C", "A"]
+    # Each yielded item includes just that call's effects.
+    by_value = {value: call_effects["progress"] for _idx, value, call_effects in items}
+    assert by_value["A"] == [("start", "a"), ("done", "a")]
+    assert by_value["B"] == [("start", "b"), ("done", "b")]
+    assert by_value["C"] == [("start", "c"), ("done", "c")]
+    # Aggregate effects dict is merged in call order, matching compute_many.
+    assert effects["progress"] == [
+        ("start", "a"),
+        ("done", "a"),
+        ("start", "b"),
+        ("done", "b"),
+        ("start", "c"),
+        ("done", "c"),
+    ]
+
+
 def test_work_stealing_run_raises_first_recorded_task_error() -> None:
     scheduler = WorkStealingExecutor(workers=2)