plexus-python-pulse-demeter 1.1.132__tar.gz → 1.1.134__tar.gz

{plexus_python_pulse_demeter-1.1.132 → plexus_python_pulse_demeter-1.1.134}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: plexus-python-pulse-demeter
-Version: 1.1.132
+Version: 1.1.134
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
@@ -27,8 +27,8 @@ Requires-Dist: uvicorn; extra == "api"
 Provides-Extra: cli
 Requires-Dist: argcomplete; extra == "cli"
 Provides-Extra: workflow
-Requires-Dist: requests<3,>=2.32; extra == "workflow"
-Requires-Dist: pydantic<3,>=2.8; extra == "workflow"
+Requires-Dist: requests>=2.32; extra == "workflow"
+Requires-Dist: pydantic>=2.11; extra == "workflow"
 Provides-Extra: test
 Requires-Dist: ddt>=1.7; extra == "test"
 Requires-Dist: moto[s3,server]>=5.1; extra == "test"
@@ -36,4 +36,4 @@ Requires-Dist: pytest-cov>=5.0; extra == "test"
 Requires-Dist: pytest-httpx2>=1.0; extra == "test"
 Requires-Dist: pytest-order>=1.3; extra == "test"
 Requires-Dist: pytest-postgresql>=6.1; extra == "test"
-Requires-Dist: pytest>=8.3; extra == "test"
+Requires-Dist: pytest>=9.2; extra == "test"

{plexus_python_pulse_demeter-1.1.132 → plexus_python_pulse_demeter-1.1.134}/k8s/local/README.md

@@ -72,7 +72,7 @@ kubectl create -f k8s/local/example-business-seeder.yaml
 
 Wait for the seeder to finish, then allow the `transform`, `concat`, and `repeat` schedules to process the message. Each
 schedule runs once per minute, so completion can take several minutes. A successful run can have an empty batch; check
-the persisted work items as well as the workflow status. List workflows and inspect a completed workflow's logs:
+the persisted stage inputs as well as the workflow status. List workflows and inspect a completed workflow's logs:
 
 ```sh
 kubectl get workflows -n argo
@@ -85,25 +85,34 @@ Forward the API service in a separate terminal:
 kubectl port-forward -n plexus-pulse service/plexus-pulse-demeter-app 8000:80
 ```
 
-Query the persisted work items from another terminal:
+Query the persisted stage inputs from another terminal:
 
 ```sh
-curl -fsS 'http://localhost:8000/pulse_demeter/pipeline/work_items/?pipeline_name=example_business%401'
+curl -fsS 'http://localhost:8000/pulse_demeter/pipeline/stage_inputs/?pipeline_name=example_business%401'
 ```
 
-For a fresh database with the default `hello world` input, the response's `work_items` list should contain four work
-items, all with `pipeline_name: "example_business@1"` and `status: "SUCCEEDED"`:
+For a fresh database with the default `hello world` input, the response's `stage_inputs` list should contain four stage
+inputs, all with `pipelineName: "example_business@1"` and `status: "SUCCEEDED"`:
 
-| Producer → Consumer    | `stage_output` | Expected `work_item_result`                                   |
-|------------------------|----------------|---------------------------------------------------------------|
-| `source` → `transform` | `message`      | `{"lowered": "hello world", "reversed": "dlrow olleh"}`       |
-| `transform` → `concat` | `lowered`      | `{"message": "hello worlddlrow olleh"}`                       |
-| `transform` → `concat` | `reversed`     | `{"message": "hello worlddlrow olleh"}`                       |
-| `concat` → `repeat`    | `result`       | `{"message": "hello worlddlrow ollehhello worlddlrow olleh"}` |
+| Producer → Consumer    | `topic`    | Expected `result`                                             |
+|------------------------|------------|---------------------------------------------------------------|
+| `source` → `transform` | `message`  | `{"lowered": "hello world", "reversed": "dlrow olleh"}`       |
+| `transform` → `concat` | `lowered`  | `{"message": "hello worlddlrow olleh"}`                       |
+| `transform` → `concat` | `reversed` | `{"message": "hello worlddlrow olleh"}`                       |
+| `concat` → `repeat`    | `result`   | `{"message": "hello worlddlrow ollehhello worlddlrow olleh"}` |
 
-Each work item's `outcome` should name its consumer stage, and all four work items should share the source message's
-SHA-256 `dedup_key`. The final value is stored in the `concat` → `repeat` work item's `result`; no further work item
-is created for the terminal stage's output.
+Each stage input's `outcome` should name its consumer stage. Query the persisted stage outputs as well:
+
+```sh
+curl -fsS 'http://localhost:8000/pulse_demeter/pipeline/stage_outputs/?pipeline_name=example_business%401'
+```
+
+Join each input's `stageOutputUuid` to an output's `uuid` to find its `producerStage`, `topic`, `dedupKey`, and `data`;
+these fields live on stage outputs, while execution status, `outcome`, and `result` live on stage inputs. All outputs
+share the source message's SHA-256 `dedupKey`. There should be five outputs: the source message, the two transform
+branches, the concatenated message, and the repeated message. The final value is stored in the`concat` → `repeat` stage
+input's `result` and in the terminal `repeat` stage output's `data`. The terminal output is persisted even though it has
+no downstream stage input.
 
 The business workspace is under `$WORKSPACE_DIR/$WORKFLOW_UID` on the workflow's shared volume. The heartbeat process
 may exit with code 143 when its workflow finishes.
@@ -113,38 +122,40 @@ may exit with code 143 when its workflow finishes.
 ### Stages and data flow
 
 The [seeder](example-business-seeder.dockerfile) registers `example_business@1` if it does not exist, verifies the
-topology if it does, and submits a source work item for each input message. The message's SHA-256 digest is used as the
-deduplication key. The [business code](example-business.dockerfile) processes three stages:
+topology if it does, and creates a source stage output plus a linked `transform` stage input for each input message. The
+message's SHA-256 digest is used as the deduplication key. Internal output creation does not fan out inputs
+automatically, so the seeder explicitly creates the linked input with the topology's retry limit and properties.
+The [business code](example-business.dockerfile) processes three stages:
 
 1. `transform` consumes a source message and submits two outputs: the lowercase message (`lowered`) and the original
    message reversed (`reversed`).
 2. `concat` joins those outputs by deduplication key within the claimed batch, concatenates the lowercase and reversed
-   messages, and submits a `result` output. If only one branch is in the batch, it skips that work item so it can be
+   messages, and submits a `result` output. If only one branch is in the batch, it skips that stage input so it can be
    claimed again later. The join does not fetch missing inputs or retain them outside the batch.
-3. `repeat` consumes the concatenated message, doubles it, and records the result on its input work item. It also
-   submits a `result` output, but the API creates downstream work items only for declared consumers. No stage consumes
-   this output.
+3. `repeat` consumes the concatenated message, doubles it, and records the result on its stage input. It also
+   submits a `result` output, but the API creates downstream stage inputs only for declared consumers. No stage consumes
+   this output, but it is still persisted.
 
 The [example resources](example-business.yaml) define one CronWorkflow per business stage, each with a claim limit of
-1,000 work items per incoming `(producer_stage, stage_output)` pair. A `concat` batch can therefore contain up to 1,000
-`lowered` work items and 1,000 `reversed` work items, for a total of 2,000. The limit does not guarantee matching
-deduplication keys across inputs; unmatched work items still need to be skipped. `concurrencyPolicy: Forbid` prevents
+1,000 stage inputs per incoming `(producer_stage, topic)` pair. A `concat` batch can therefore contain up to 1,000
+`lowered` stage inputs and 1,000 `reversed` stage inputs, for a total of 2,000. The limit does not guarantee matching
+deduplication keys across inputs; unmatched stage inputs still need to be skipped. `concurrencyPolicy: Forbid` prevents
 overlapping runs of the same CronWorkflow; different stages can run concurrently. Their data dependencies are enforced
-by the availability of work items in PostgreSQL.
+by the availability of stage inputs in PostgreSQL.
 
 ### Batch execution and the flow capsule
 
 The [dispatcher](../argo/plexus-pulse-demeter-pipeline-templates.yaml) runs each scheduled batch in three steps:
 
 1. **Initialize.** Unless a matching local capsule already exists, the worker recovers expired stage run attempts for
-   the pipeline and stage, then checks out a batch. Checkout creates a stage run, stage run attempt, and work item
-   attempts and claims up to the configured limit of eligible queued work items per incoming `(producer_stage,
-   stage_output)` pair in one PostgreSQL transaction. Each input is ordered by priority, availability time, and sequence
+   the pipeline and stage, then checks out a batch. Checkout creates a stage run, stage run attempt, and stage input
+   attempts and claims up to the configured limit of eligible queued stage inputs per incoming `(producer_stage,
+   topic)` pair in one PostgreSQL transaction. Each input is ordered by priority, availability time, and sequence
    number. The Argo workflow UID determines the stage run's deduplication key, so repeated checkout requests reuse the
-   same claim rather than claim more work items. The worker saves the returned SQLite file as
+   same claim rather than claim more stage inputs. The worker saves the returned SQLite file as
    `pulse-demeter-flow-capsule.db` in the shared workspace.
 2. **Process.** A daemon sends a heartbeat every 60 seconds while the business container processes the capsule's fixed
-   batch. Business code records work item results with `FlowCapsule.commit`, `fail`, or `skip`, and records downstream
+   batch. Business code records stage input results with `FlowCapsule.commit`, `fail`, or `skip`, and records downstream
    outputs with `submit`. These calls write to the local capsule; they do not update PostgreSQL directly.
 3. **Finalize.** The exit handler uses the Argo workflow status to decide whether to upload the capsule or report a
    stage run failure, as described below.
@@ -152,24 +163,24 @@ The [dispatcher](../argo/plexus-pulse-demeter-pipeline-templates.yaml) runs each
 ### Completion, retries, and recovery
 
 When Argo reports `Succeeded`, the worker freezes a SQLite snapshot and uploads it as the multipart `file` field to the
-stage-run-attempt `commit/` endpoint. The API applies work item results and creates downstream work items in one
-transaction, then marks the stage run and stage run attempt as succeeded. Stage run success does not imply every work
-item succeeded: explicitly failed work items and claimed work items without a result follow the work item failure rules.
-Business code should record a result for every work item.
+stage-run-attempt `commit/` endpoint. The API applies stage input results, persists stage outputs, and creates
+downstream stage inputs in one transaction, then marks the stage run and stage run attempt as succeeded. Stage run
+success does not imply every stage input succeeded: explicitly failed stage inputs and claimed stage inputs without a
+result follow the stage input failure rules. Business code should record a result for every stage input.
 
-A skipped attempt remains in history and its work item is queued without consuming retry allowance. Failed work items
-are queued while their retry limit permits it, otherwise they become failed. Work items requeued by a capsule commit
-become eligible again after 60 seconds. Work item attempt numbers continue increasing even when skips do not consume
-retry allowance.
+A skipped attempt remains in history and its stage input is queued without consuming retry allowance. Failed stage
+inputs are queued while their retry limit permits it, otherwise they become failed. Stage inputs requeued by a capsule
+commit become eligible again after 60 seconds. Stage input attempt numbers continue increasing even when skips do not
+consume retry allowance.
 
-For any Argo status other than `Succeeded`, the worker calls `fail/` without uploading the capsule. Running work item
-attempts become failed, and their work items are immediately eligible for another run if their retry limit permits it.
-The separate `cancel/` API operation instead marks running attempts as cancelled and immediately requeues their work
-items without consuming retry allowance. The exit handler does not call `cancel/` automatically.
+For any Argo status other than `Succeeded`, the worker calls `fail/` without uploading the capsule. Running stage input
+attempts become failed, and their stage inputs are immediately eligible for another run if their retry limit permits it.
+The separate `cancel/` API operation instead marks running attempts as cancelled and immediately requeues their stage
+inputs without consuming retry allowance. The exit handler does not call `cancel/` automatically.
 
 Recovery expires running stage run attempts whose last heartbeat is at least 30 minutes old, marks their stage runs
-failed, and immediately requeues eligible work items under the same retry-limit rules as failure. Recovery runs during a
-new initialization, rather than on an independent timer. Heartbeats and commits for expired or recovered attempts
+failed, and immediately requeues eligible stage inputs under the same retry-limit rules as failure. Recovery runs during
+a new initialization, rather than on an independent timer. Heartbeats and commits for expired or recovered attempts
 receive HTTP 409.
 
 The lifecycle endpoints are under `/pulse_demeter/pipeline`: `POST /stage_runs/recover/`, `POST /stage_runs/checkout/`,

{plexus_python_pulse_demeter-1.1.132 → plexus_python_pulse_demeter-1.1.134}/k8s/local/example-business-seeder.dockerfile

@@ -10,7 +10,7 @@ import json
 import urllib.error
 import urllib.request
 
-from plexus.pulse.demeter.app.models.pipeline import WorkItem, Pipeline
+from plexus.pulse.demeter.app.models.pipeline import Pipeline, StageInput, StageOutput
 from plexus.pulse.demeter.app.core.pipeline import validate_topology
 
 
@@ -21,33 +21,33 @@ pipeline = Pipeline(
             {
                 "name": "source",
                 "executor": "manual",
-                "outputs": ["message"],
+                "topics": ["message"],
             },
             {
                 "name": "transform",
                 "executor": "argo",
                 "props": {"normalization": "lowercase"},
                 "consume": [
-                    {"producer": "source", "output": "message", "max_retrials": 2},
+                    {"producer": "source", "topic": "message", "max_retrials": 2},
                 ],
-                "outputs": ["lowered", "reversed"],
+                "topics": ["lowered", "reversed"],
             },
             {
                 "name": "concat",
                 "executor": "argo",
                 "consume": [
-                    {"producer": "transform", "output": "lowered", "max_retrials": 2},
-                    {"producer": "transform", "output": "reversed", "max_retrials": 2, "props": {"branch": "reversed"}},
+                    {"producer": "transform", "topic": "lowered", "max_retrials": 2},
+                    {"producer": "transform", "topic": "reversed", "max_retrials": 2, "props": {"branch": "reversed"}},
                 ],
-                "outputs": ["result"],
+                "topics": ["result"],
             },
             {
                 "name": "repeat",
                 "executor": "argo",
                 "consume": [
-                    {"producer": "concat", "output": "result", "max_retrials": 2},
+                    {"producer": "concat", "topic": "result", "max_retrials": 2},
                 ],
-                "outputs": ["result"],
+                "topics": ["result"],
             },
         ]
     },
@@ -58,7 +58,8 @@ def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--api-url", default="http://localhost:8000")
     parser.add_argument("--count", type=int, default=1, help="Number of distinct inputs per message")
-    parser.add_argument("--batch-size", type=int, default=1000, help="Work items per API request")
+    parser.add_argument("--batch-size", type=int, default=1000,
+                        help="Messages per batch (one output request and one input request)")
     parser.add_argument("messages", nargs="+", help="Messages to process")
 
     args = parser.parse_args()
@@ -76,7 +77,7 @@ def main() -> None:
             with urllib.request.urlopen(request, timeout=30) as response:
                 return json.load(response)
         except urllib.error.HTTPError as exc:
-            raise RuntimeError(f"{path}: HTTP {exc.code}: {exc.read().decode()}") from exc
+            raise RuntimeError(f"'{path}': HTTP '{exc.code}': {exc.read().decode()}") from exc
 
     with urllib.request.urlopen(base + "/pipelines/", timeout=30) as response:
         existing_pipelines = [Pipeline.model_validate(item) for item in json.load(response)]
@@ -84,7 +85,7 @@ def main() -> None:
     if existing_pipeline is None:
         post("/pipeline/", pipeline.model_dump(mode="json"))
     elif validate_topology(existing_pipeline.topology) != validate_topology(pipeline.topology):
-        raise RuntimeError(f"existing pipeline {pipeline.name} has a different topology")
+        raise RuntimeError(f"existing pipeline '{pipeline.name}' has a different topology")
 
     messages = (
         message if args.count == 1 else f"{message} {index:05d}"
@@ -92,26 +93,37 @@ def main() -> None:
         for index in range(args.count)
     )
 
-    work_items = []
-    for message in messages:
-        digest = hashlib.sha256(message.encode()).hexdigest()
-        db_work_item = WorkItem(
-            pipeline_name=pipeline.name,
-            producer_stage="source",
-            consumer_stage="transform",
-            stage_output="message",
-            work_item_name="input-" + digest[:16],
-            dedup_key=digest,
-            data={"message": message},
-            available_at=datetime.datetime.now(datetime.UTC),
-        )
-        work_items.append(db_work_item.model_dump(mode="json"))
-
+    source_input = next(
+        source for source in validate_topology(pipeline.topology).stage("transform").consume
+        if source.producer == "source" and source.topic == "message"
+    )
     submitted = 0
-    for batch in itertools.batched(work_items, args.batch_size):
-        created_work_items = post("/work_items/", list(batch))
-        submitted += len(created_work_items)
-        print(f"Submitted {submitted}/{len(work_items)} work items", flush=True)
+    for batch in itertools.batched(messages, args.batch_size):
+        stage_outputs = [
+            StageOutput(
+                pipeline_name=pipeline.name,
+                producer_stage="source",
+                topic="message",
+                dedup_key=hashlib.sha256(message.encode()).hexdigest(),
+                data={"message": message},
+            ).model_dump(mode="json")
+            for message in batch
+        ]
+        created_stage_outputs = post("/stage_outputs/", stage_outputs)
+        stage_inputs = [
+            StageInput(
+                pipeline_name=pipeline.name,
+                stage_output_uuid=stage_output["uuid"],
+                consumer_stage="transform",
+                props=source_input.props,
+                max_retrials=source_input.max_retrials,
+                available_at=datetime.datetime.now(datetime.UTC),
+            ).model_dump(mode="json")
+            for stage_output in created_stage_outputs
+        ]
+        created_stage_inputs = post("/stage_inputs/", stage_inputs)
+        submitted += len(created_stage_inputs)
+        print(f"Submitted {submitted} messages as stage outputs and stage inputs", flush=True)
 
 
 if __name__ == "__main__":

plexus_python_pulse_demeter-1.1.134/k8s/local/example-business.dockerfile

@@ -0,0 +1,89 @@
+FROM ruyangshou/plexus-pulse-demeter-tools:latest
+
+# language=python
+COPY <<'EOF' /app/example_business.py
+from plexus.pulse.demeter.utils import flowutils
+
+
+def main() -> None:
+    capsule = flowutils.capsule()
+
+    stage_run = capsule.stage_run()
+    stage = stage_run.stage
+    if stage not in {"transform", "concat", "repeat"}:
+        raise ValueError(f"unsupported example stage: '{stage}'")
+
+    stage_inputs = capsule.stage_inputs()
+    if stage == "transform":
+        for stage_input in stage_inputs:
+            message = str(stage_input.data.get("message", ""))
+            lowered_message, reversed_message = message.lower(), message[::-1]
+
+            capsule.submit(
+                topic="lowered",
+                dedup_key=stage_input.dedup_key,
+                data={"message": lowered_message},
+            )
+            capsule.submit(
+                topic="reversed",
+                dedup_key=stage_input.dedup_key,
+                data={"message": reversed_message},
+            )
+            capsule.commit(
+                stage_input.stage_input_uuid,
+                outcome=stage,
+                result={"lowered": lowered_message, "reversed": reversed_message},
+            )
+
+            print(f"{stage_input.stage_input_uuid}: {message!r} -> {lowered_message!r}, {reversed_message!r}", flush=True)
+
+    elif stage == "concat":
+        # Inner join the two branch outputs by the source message's digest.
+        # A digest is shared by all outputs for one source message.
+        lowered_by_key = {stage_input.dedup_key: stage_input
+                          for stage_input in stage_inputs if stage_input.topic == "lowered"}
+        reversed_by_key = {stage_input.dedup_key: stage_input
+                           for stage_input in stage_inputs if stage_input.topic == "reversed"}
+        joined_keys = lowered_by_key.keys() & reversed_by_key.keys()
+
+        for stage_input in stage_inputs:
+            if stage_input.dedup_key not in joined_keys:
+                capsule.skip(stage_input.stage_input_uuid)
+
+        for dedup_key in joined_keys:
+            lowered_stage_input = lowered_by_key[dedup_key]
+            reversed_stage_input = reversed_by_key[dedup_key]
+            lowered_message = lowered_stage_input.data.get("message", "")
+            reversed_message = reversed_stage_input.data.get("message", "")
+            concatenated_message = lowered_message + reversed_message
+
+            capsule.submit(
+                topic="result",
+                dedup_key=dedup_key,
+                data={"message": concatenated_message},
+            )
+            capsule.commit(lowered_stage_input.stage_input_uuid, outcome=stage, result={"message": concatenated_message})
+            capsule.commit(reversed_stage_input.stage_input_uuid, outcome=stage, result={"message": concatenated_message})
+
+            print(f"{dedup_key}: {concatenated_message!r}", flush=True)
+
+    else:
+        for stage_input in stage_inputs:
+            message = str(stage_input.data.get("message", ""))
+            repeated_message = message * 2
+
+            capsule.submit(
+                topic="result",
+                dedup_key=stage_input.dedup_key,
+                data={"message": repeated_message},
+            )
+            capsule.commit(stage_input.stage_input_uuid, outcome=stage, result={"message": repeated_message})
+
+            print(f"{stage_input.stage_input_uuid}: {message!r} -> {repeated_message!r}", flush=True)
+
+
+if __name__ == "__main__":
+    main()
+EOF
+
+ENTRYPOINT ["/app/venv/bin/python", "/app/example_business.py"]

{plexus_python_pulse_demeter-1.1.132 → plexus_python_pulse_demeter-1.1.134}/pyproject.toml

@@ -31,8 +31,8 @@ cli = [
     "argcomplete",
 ]
 workflow = [
-    "requests>=2.32,<3",
-    "pydantic>=2.8,<3",
+    "requests>=2.32",
+    "pydantic>=2.11",
 ]
 test = [
     "ddt>=1.7",
@@ -41,7 +41,7 @@ test = [
     "pytest-httpx2>=1.0",
     "pytest-order>=1.3",
     "pytest-postgresql>=6.1",
-    "pytest>=8.3",
+    "pytest>=9.2",
 ]
 
 [project]
@@ -82,8 +82,8 @@ cli = [
     "argcomplete",
 ]
 workflow = [
-    "requests>=2.32,<3",
-    "pydantic>=2.8,<3",
+    "requests>=2.32",
+    "pydantic>=2.11",
 ]
 test = [
     "ddt>=1.7",
@@ -92,7 +92,7 @@ test = [
     "pytest-httpx2>=1.0",
     "pytest-order>=1.3",
     "pytest-postgresql>=6.1",
-    "pytest>=8.3",
+    "pytest>=9.2",
 ]
 
 [project.scripts]
@@ -109,6 +109,7 @@ namespaces = true
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]
+asyncio_default_fixture_loop_scope = "function"
 
 [tool.coverage.run]
 branch = true