forkreduce 0.1.0__tar.gz → 0.2.0__tar.gz

{forkreduce-0.1.0 → forkreduce-0.2.0}/.github/workflows/ci.yml

@@ -11,6 +11,7 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.11", "3.12"]
+        extras: ["temporal", "openai,temporal,full"]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
@@ -18,7 +19,12 @@ jobs:
           python-version: ${{ matrix.python-version }}
           cache: pip
       - name: Install
-        run: python -m pip install -e ".[dev]"
+        run: python -m pip install -e ".[dev,${{ matrix.extras }}]" "pydantic-ai-slim[temporal]>=2.57,<3"
+      - name: Verify provider-free slim installation
+        if: matrix.extras == 'temporal'
+        run: |
+          python -c "from importlib.util import find_spec; assert find_spec('openai') is None"
+          python -c "from importlib.metadata import distributions; assert 'pydantic-ai' not in {d.metadata['Name'].lower().replace('_', '-') for d in distributions()}"
       - name: Ruff
         run: |
           ruff check src tests examples

forkreduce-0.2.0/CHANGELOG.md

@@ -0,0 +1,119 @@
+# Changelog
+
+## 0.2.0 — 2026-10-25
+
+### Added
+
+- Referenced dynamic schemas via `OutputSpecWriter(store).store(spec, ref_namespace=...) -> OutputSpecRef`: content-addressed preparation, per-writer deduplication, namespace/context-aware reads, digest/name verification, missing-blob errors and per-activity resolution caching. Inline specs and existing default payload bytes remain unchanged. Complete `run_spec_refs_offline.py` uses activity-generated bounded batches/public orchestration for 300 large distinct schemas, both converters and replay; no `queries_ref` field or new default activity registration.
+- Opt-in `FollowUpQuery(query, history="fresh")` restarts repairs from the shared prefix; default `continue` retains history inheritance. Tail-only repairs retain their schema; changing schemas requires a full follow-up query. No prefix-hash/default behavior change.
+- `build(query_separator=QUERY_SEPARATOR, dynamic_output_instructions=True)` supports exact caller-owned dynamic bytes. Only nondefault separators used by `message_layout="single"` enter prefix hashes/fingerprints; split separators are ignored. Suppressing dynamic guidance affects only OutputSpec tails/fingerprints, not prefix hashes or static output tails. Nondefault frozen layouts use `ConfiguredFrozenPrefix` / `QueryLayout`, preserving plain snapshot fields/bytes; factories restore both options.
+- Validators may accept `(output, OutputValidationContext(query_id, round, meta))`; copied context is query-local, supports sync/async/native feedback, and has no prefix-hash impact. Existing one-argument callbacks remain supported.
+
+- Opt-in `document_format="raw"` preserves exact text/control characters without framework wrappers/end markers; `message_layout="single"` combines the initial prefix and tail with exported `QUERY_SEPARATOR`, retaining the head across continuations/feedback. `DocumentFormat` / `MessageLayout` are public aliases, snapshots restore both settings, and raw mode explicitly rejects built-in block/citation operations. Only nondefaults enter the hash/fingerprint encoding; literal v0.1.1 defaults remain byte/hash identical.
+- Complete `examples/run_temporal_offline.py` and bilingual integration recipes combine default factories, bounded metadata, namespace-routed immutable SQLite references, return-mode hook repairs, dynamic schemas, warmup/queues, prefixed sandbox definitions, tracing and raw/single layouts. It requires a cached local server and disables provider requests; both converters support replay.
+
+- Worker-local `build(instrument=False|True|InstrumentationSettings)` applies to ordinary, feedback and dynamic Agents without changing hashes/fingerprints; explicit default-off remains independent of global Agent defaults. `forkreduce_temporal_names(prefix=...)` / serializable `TemporalNames` provide sandboxed, cached prefixed activity/workflow definitions while preserving original default registrations, child/query IDs, application hooks and custom batch choices. Real in-memory tracing and same-worker official TemporalDurability/PydanticAIPlugin coexistence are covered.
+
+- Public serializable `OutputSpec` / `DynamicOutputs` support per-question JSON schemas and dictionary outputs, with JSON Schema validation, local refs/composition, and native output-feedback retries. Prompted schemas stay in tails; opt-in `dynamic_outputs="generic_tool"` adds one fixed permissive output tool rather than changing tools per query. Dynamic-only auto sessions without that tool use prompted mode. Both Temporal converters handle specs/follow-ups and frozen dynamic settings. Only enabling/changing the frozen tool/mode contract changes hashes; per-question schemas do not.
+
+- Public workflow primitives `plan_batches`, `batch_child_id`, `require_query_references`, `validate_query_references`, `query_failure`, and `execute_query` support application-owned orchestration. Built-in pipelines add `AfterQueryInput` / `AfterQueryDecision`, per-query `after_query_activity` with independent options, bounded repair rounds, safe `after_query_failed` records, and small copied `result_meta`. `batch_workflow` selects an application batch with the same input/output contract. Defaults and hashes are unchanged; repair IDs append `#r{round}`.
+
+- `MapQueriesInput.activity_concurrency`, `task_queue`, and `warmup="none"|"first"|"per_batch"` control direct/per-child query concurrency, child/activity queue routing, and deterministic warmup gates. A first returned failure releases warmup; raise/cancellation propagation remains unchanged. Default settings preserve fan-out/queue behavior and do not change prefix hashes or session execution fingerprints.
+
+- Bounded same-branch output feedback via `Policy(output_feedback_retries=0)` and worker-local `build(output_validator=...)` sync/async callbacks raising `ModelRetry`. Schema/cross-field errors share native Pydantic AI retry semantics; every actual request is budgeted. Results expose `feedback_retries_used` and matching metadata; frozen snapshots preserve the numeric setting, not callbacks. Default zero retains old request behavior and hashes; opt-in feedback/validators change execution fingerprints, not prefix hashes.
+
+- SDK-independent, JSON-serializable `ActivityOptions`, available on `build`, individual Temporal query payloads, and `MapQueriesInput`: start-to-close, heartbeat, schedule-to-close, total attempts, and exponential-backoff settings. Defaults remain five minutes and `Policy.retries + 1` attempts. Configured heartbeats cover the full asynchronous query activity, with cancellation cleanup.
+- Optional bounded child batching via `MapQueriesInput(batch_size=..., child_concurrency=...)` and `QueryBatchWorkflow`. Ordered results and deterministic child IDs include parent workflow ID, prefix hash, and batch index; IDs survive workflow retries that change run ID. Single-workflow/list-input mode stays the default.
+- `PrefixStore`, `FrozenPrefix`, `freeze_prefix`, `set_prefix_store`, and worker-local `register_session_factory` support document reconstruction from references. Batched mode requires external transcript/result storage and returns `messages_ref` / `output_ref`, keeping transcript and structured-output bodies out of parent history. The in-memory store remains test-only, not a durable backend.
+- Real time-skipping workflow tests for the default converter and `PydanticAIPlugin` / `PydanticAIPayloadConverter`, including offline `TestModel`/`FunctionModel`, shared-worker `TemporalDurability`, replay, bounded children, heartbeat/retry behavior, and large-document payload bounds.
+- Opt-in `Citation` schema and `CITATION_INSTRUCTIONS`: a contiguous quote must fit inside its single referenced block; split cross-block evidence into separate citations. Citation validators remain strict and unchanged.
+- Capability-aware `output_mode="auto"` (the new build default), requested `Session.output_mode`, actual `Session.resolved_output_mode`, and `QueryResult.resolved_output_mode` (including failures/reduce combinations; legacy default `None`). Auto resolves from public model profile fields and shallow-merged model/build settings, accounting for thinking restrictions; it never selects native output or retries in a different mode.
+- `on_unforced_tools="warn"|"error"|"ignore"` for explicit tools without effective forcing support (`UnforcedToolsWarning` / `UnforcedToolsError`), including capability-changing query settings. Auto query overrides that would flip the resolved mode raise `OutputModeConflictError` before budget/provider use; build a new session instead.
+- When thinking can affect forcing capability, auto rejects unknown/malformed non-null thinking controls with `ForkReduceError` code `unknown_thinking_control`; explicit tools uses the unforced-tools policy, while explicit prompted remains honored. Unconditionally supported/unsupported profiles resolve directly without rejecting irrelevant thinking controls. Recognized OpenAI, Anthropic, Google, and common `extra_body` controls take precedence over unified thinking, with conflicting controls conservatively active.
+- `FrozenPrefix` retains requested mode, optional resolved mode for legacy snapshots, and warning policy. Factories restore the requested mode/policy and matching thinking-related settings; workers reject resolved-mode/hash mismatches.
+- Opt-in `MapQueriesInput.on_query_failure="return"` returns ordered mixed `QueryActivityOutput` successes/failures after Temporal retries, with safe bounded messages and observed attempt counts. Workflow cancellation still propagates; default `"raise"` and list inputs remain unchanged.
+- `set_default_session_factory` reconstructs arbitrary frozen prefixes after worker restart, with exact factories taking precedence. Missing factories now raise retryable `session_factory_missing`; snapshot/hash mismatches remain permanent.
+- `freeze_prefix(..., app_meta=...)` persists up to 16 KiB compact UTF-8 JSON routing metadata for factories without changing prefix identity or execution fingerprints; snapshots without metadata remain readable.
+- Frozen `StoreContext`, context-aware store signatures alongside legacy put/get, namespace-prefixed generated keys, and explicit application-owned `delete_ref`. English/Chinese recipes combine durable storage, default factories, metadata and partial-failure returns.
+
+### Changed
+
+- Store contracts require immutable returned references. Both SQLite README stores now use content-addressed, insert-only references rather than overwriting a reused logical key; tests retain earlier transcript bytes across rewrites and independent deletion. This fixes the outstanding immutable-reference review finding without changing framework key/hash identity.
+- Internal `child_workflow_id` is replaced by public `batch_child_id(parent_id, prefix_hash, index)`; the child-ID algorithm and generated IDs are byte-identical.
+
+- Base dependency is now `pydantic-ai-slim>=2.57,<3`; `[openai]` installs its OpenAI support and `[full]` restores the complete `pydantic-ai` provider bundle. HTTPX is an explicit dependency for the existing transport API. CI covers provider-free slim/Temporal and full/OpenAI installations. Core imports do not require a provider SDK; calling the OpenAI helper without its extra raises an actionable `ImportError`. No hash or default execution change.
+
+- Reduce tails explicitly enumerate allowed input/fork IDs and per-question source mappings when `question_id` fields are present, including nested outputs and tree aliases. Document/block IDs are not reduce provenance IDs. Citation-repair tails repeat the single-block constraint.
+- English and Chinese documentation include the 2-attempt / 3-minute / 30-second-heartbeat recipe, plugin coexistence, batching/reference storage, and reasoning-token headroom. Primary-source links explain model/vendor forced-tool restrictions (HTTP 400 or silent `auto`, especially with thinking), exact Pydantic AI 2.61 profile semantics/defaults, and MiMo thinking's forced `temperature=1.0` / `top_p=0.95` despite requested zero temperature.
+
+### Fixed
+
+- The large-schema example uses the SDK's default test-server cache/download behavior when `TEMPORAL_TEST_SERVER` is unset; an explicit cached binary retains offline execution. Direct checkout execution no longer needs manual `PYTHONPATH`. Optional OpenAI imports have a scoped mypy missing-import override so provider-free slim remains strictly type-checked.
+- Worker session registration rejects distinct validator objects whose canonical fingerprints collide, preserving the first registration until explicit unregister/replacement. The guard is process-local; no object identities enter serialized fingerprints or default payloads.
+- Check/repair and reduce aggregation retain accumulated `feedback_retries_used` and matching metadata, counting actual calls rather than continuation history.
+- Both plain/default and configured frozen snapshots validate document/message layout and dynamic-instruction settings against reconstructed or cached sessions, with non-retryable mismatch errors. Default snapshot/payload bytes stay unchanged.
+
+- Keep a single periodic heartbeat task alive across prefix hydration, continuation-history reads, async session-factory construction, model calls, and transcript/structured-output uploads. Previously heartbeats stopped outside the model await, so a healthy slow store could time out and repeat a successful model call on retry. Real-worker regressions cover each reference I/O boundary and the async factory with a 200 ms timeout, a 650 ms wait, and exactly one attempt under both converters. Blocking the event loop remains outside heartbeat protection.
+
+### Compatibility and cache identity
+
+- No prefix-hash algorithm change: hashes use actual tools/prompted mode, never an auto marker. Auto resolving to tools preserves the previous default tools hash; unsupported forcing now defaults to prompted and changes that old default hash. Explicit tools/prompted hashes remain stable. Thinking settings/profile changes can change auto resolution; per-query changes cannot flip it. Reduce/repair guidance changes only tails. Adopting `Citation` or `CITATION_INSTRUCTIONS` changes schema/instruction bytes and therefore `prefix_hash` (including prompted schema fingerprints); updated examples consequently miss old provider prefix caches. `ActivityOptions`, batching, and reference storage do not change prefix identity.
+- Activity/workflow names and legacy list inputs remain supported. Batch results intentionally use references rather than inline bodies. Child IDs identify orchestration, not exactly-once provider calls; retries can repeat external effects. Retain referenced data through retries/continuations. Drain or version incompatible in-flight workflows and replay representative histories before changing deployment code/converters.
+- Verified dependency targets: Pydantic AI 2.61.0, OpenAI 3.37.0, Temporal SDK 1.33.0 and 1.34.0. Core requirement is `pydantic-ai-slim>=2.57,<3`; `[openai]` adds OpenAI support and `[full]` installs `pydantic-ai>=2.57,<3`. The adapter does not require raising its existing `temporalio>=1.9` floor. Official Pydantic AI plugin users must also satisfy that integration's own `[temporal]` requirements.
+- Version metadata is prepared as 0.2.0. This preparation does not publish the package, create a tag/release, merge the PR, or change deployments.
+
+### Stable query failure kinds
+
+Public `QueryFailureKind` is exported from `forkreduce.contrib.temporal` as a `Literal`; `QueryActivityOutput.failure_kind=None` denotes success. The complete non-`None` set is:
+
+| `failure_kind` | Meaning |
+| --- | --- |
+| `retryable_exhausted` | Retryable failure with Temporal maximum attempts reached. |
+| `permanent` | Non-retryable application failure other than budget exhaustion. |
+| `budget` | Application budget exhaustion. |
+| `timeout` | Temporal activity timeout. |
+| `cancelled` | Activity cancellation, not parent workflow cancellation. |
+| `unknown` | Ambiguous terminal application/SDK failure or failed child batch. |
+
+### Upgrade notes
+
+- **References and refinements:** configure the message store for spec reads; retain immutable schema blobs through retries/repairs and set query routing metadata. Reuse preparation writers for deduplication; cross-process deduplication requires content-addressed storage. Keep generated batches bounded rather than returning a giant list. Restore `frozen.query_separator` and `frozen.dynamic_output_instructions`; nondefault separators change identity only in single layout. Guidance suppression affects only OutputSpec tails, where callers supply their own instructions. Use full follow-up queries for schema changes and `FollowUpQuery(..., history="fresh")` for resets. With feedback, size `start_to_close >= single_call_time * (output_feedback_retries + 1)` plus I/O/validation overhead; outer attempts are independent.
+
+- **Document/message layout:** defaults remain blocks/split and preserve the legacy identity encoding. Raw/single opt-ins change prefix hashes and execution fingerprints; restore both frozen flags in factories and version/drain incompatible in-flight work. Raw metadata rendering and framework block/citation operations are unsupported; applications own deterministic bytes and provider acceptance. The new complete offline example needs the checkout dev/Temporal extras for in-memory OpenTelemetry support and never downloads its test server.
+
+- **Observability and registration:** rebuild models and instrumentation in worker session factories, never serialize providers/exporters/functions. Configure tracing/export explicitly and consider `include_content=False`. Prefixed definitions still use the documented sandbox passthrough runner; register/replay the same definitions. Official plugin clients supply the worker plugin once. forkreduce query activities remain their own Agent path, not a TemporalDurability forwarding shim.
+
+- **Dynamic outputs:** restore `FrozenPrefix.dynamic_outputs` in factories. A static frozen tools session needs the generic tool declared up front before accepting dynamic schemas; otherwise choose a separate prompted/dynamic session. Remote schema references are rejected; both strict modes validate schema/types without coercion, with extra format/JSON checks in strict mode. Dictionary outputs need application validators/hooks for BaseModel-specific citation operations. Run real-model regressions before switching output modes; offline tests establish contracts, not model quality.
+
+- **Per-query hooks:** register the named hook on the selected queue and retain referenced histories through all repairs. `max_rounds` is an inclusive round index (initial zero); reaching the cap while still requesting repair yields a failed record. Hooked direct queries also require reference storage. Hook retries are independent of model retries; custom batches own their implementation of hook/concurrency/cancellation semantics. The additional synthetic failure type is `after_query_failed`; raw application types remain whitelisted separately.
+
+- **Scheduling:** workers on an explicit `task_queue` must register child workflows and activities there. `activity_concurrency` is per batch, not a global distributed semaphore. First warmup uses a singleton first child when batching; changing orchestration for existing executions requires draining/versioning and replay as described below.
+
+- **Output feedback:** restore `FrozenPrefix.output_feedback_retries` and worker-local validators in session factories. Feedback is bounded within one local/activity attempt; the existing outer/Temporal retry budget remains separate. Validation messages are intentionally sent to the model and must not contain secrets. Temporal failure counts describe the final observed attempt rather than a guessed cross-attempt total.
+
+- **Provider dependencies:** applications relying on previously transitive vendor SDKs must install `forkreduce[full]` or their provider's slim extra explicitly. `forkreduce[temporal]` continues to install the Temporal SDK; official Pydantic AI plugin users must satisfy that plugin's own optional dependencies.
+
+- **Cache identity:** the hash algorithm is unchanged. For unchanged schemas/instructions, the new default `output_mode="auto"` preserves the 0.1.1 tools hash when the effective profile supports forced output tools. Users whose profile forbids forcing, or whose active thinking prevents forcing, now default to prompted and get a different prefix/hash and a cold provider cache. Explicit tools/prompted retain their previous hashes. Adopting `Citation`, `CITATION_INSTRUCTIONS`, changed output descriptions, or the updated example schemas/instructions also changes hashes. Auto mode is frozen per session; a query thinking override that would flip it raises `OutputModeConflictError` before provider use—build a separate session.
+- **In-flight Temporal executions:** before deploying 0.2.0, stop admitting new work to the old task queue and let its running workflows/children/activities finish on the old worker code and converter. Keep referenced blobs available through retries and continuations. Then switch clients/workers together. If executions cannot be drained, isolate the new code on a new task queue or compatible worker version and replay representative old histories before routing existing work to it; converter roundtrips alone do not prove arbitrary old workflow-code replay compatibility.
+- **Worker sandbox:** register both `MapQueriesWorkflow` and `QueryBatchWorkflow` for batching. A default-converter worker must pass `SandboxedWorkflowRunner(restrictions=SandboxRestrictions.default.with_passthrough_modules("forkreduce", "pydantic_ai"))` as its `workflow_runner`; both classes are in `temporalio.worker.workflow_sandbox`. Apply the same runner configuration when replaying histories. With `PydanticAIPlugin`, install Pydantic AI's `[temporal]` dependencies and inherit the client plugin on the worker rather than installing it twice.
+- **Heartbeat limits:** set a finite start-to-close timeout in addition to heartbeat_timeout. The full-activity heartbeat protects cooperative async waits, including shared-store I/O, not CPU work blocking the event loop; offload that work to a thread or a separate activity. Heartbeats signal liveness, not business progress or exactly-once external effects.
+- **Query failures:** `on_query_failure="raise"` remains the default, including list inputs. Opt into `"return"` and inspect the stable `QueryFailureKind` table above before loading successful references. Ambiguous legacy `application`/`activity` kinds are now `unknown`. `failure_type` retains raw `ApplicationError.type` only for `validation`, `transport`, `permanent`, `budget`, `session_factory_missing`, and `ref_namespace_mismatch`; other raw types become `None`. Exhausted validation uses `retryable_exhausted` / `validation`, not a fabricated code; missing factory uses `retryable_exhausted` / `session_factory_missing`. A failed child batch (`ChildWorkflowError`) returns `unknown` / synthetic whitelisted `batch_failed` for every query in that batch, with `attempts=0`. Messages never echo arbitrary exception/payload text. `attempts=0` means no observed count, not the configured retry limit. Parent workflow cancellation always propagates, including return mode.
+- **Worker recovery:** initialize a default factory on every worker; an exact factory overrides it. The intentional configuration change is retryable `session_factory_missing` instead of a permanent error. Recommend `ActivityOptions(maximum_attempts=5, retry_backoff=2, max_backoff=30, backoff_coefficient=2.0)` and choose attempts/schedule-to-close to provide a sufficient worker-repair/restart recovery window. Retry budgets still apply; expired windows cannot recover. Reconstruction/hash validation remains permanent.
+- **Metadata and stores:** `app_meta` defaults to `{}` and never changes hashes/fingerprints. Keep secrets and bodies out; the limit is 16 KiB compact UTF-8 JSON. Legacy stores continue to work. Set `ref_namespace` to isolate generated keys; absent namespace preserves old keys. `query_activity_input` does not hydrate snapshots or infer namespaces from opaque references: supply matching query metadata if initial prefix reads require routing before hydration. Explicit query/snapshot disagreement is rejected as non-retryable `ref_namespace_mismatch`. Stores may consume keyword-only `context`; deletion is optional, explicit, and never automatic. Supported deletion must treat missing references, repeated deletes and never-created references as successful no-ops while preserving other references; unsupported deletion still raises `NotImplementedError`. Both SQLite README samples implement this with SQL DELETE. Retain blobs until all retries/continuations are finished.
+
+## 0.1.1
+
+Backward compatible. Calls that do not opt in to rendering `Block.meta` keep the same prefix hash as 0.1.0.
+
+### Added
+
+- `openai_compatible_model()` accepts `timeout`, `profile` (a pydantic-ai `ModelProfile`), `extra_body`, and the common sampling settings (`temperature`, `top_p`, `max_tokens`, `seed`, penalties, `stop_sequences`, `parallel_tool_calls`, `extra_headers`). A `settings` dict passes any other `ModelSettings` key. Explicit arguments override `settings`.
+- `build(..., model_settings=...)` sends those pydantic-ai `ModelSettings` on every call. `Query.model_settings` overrides individual keys for one call. The merge is shallow, so a query `extra_body` replaces the session `extra_body` as a whole.
+- The Temporal execution fingerprint includes request-body settings and the resolved model profile. `timeout` and `extra_headers` are transport options and are not part of the fingerprint. A per-query override is not part of session identity; `QueryActivityInput.model_settings` carries it to the worker.
+- Optional `Block.meta` rendering via `build(..., meta_keys=[...])` or `meta_renderer=...`. Rendered meta is part of the prefix and `prefix_hash`. The default still omits meta.
+
+Settings that change prompt or tool bytes (`meta_keys`, `meta_renderer`, and a `profile` that changes tool or schema rendering) miss the provider prefix cache. Sampling parameters, `timeout`, `extra_headers`, and `extra_body` do not change the frozen prefix. `extra_body` and sampling parameters still change the request body, so a different value is a different execution fingerprint.
+
+## 0.1.0
+
+Initial public release.