taskmaestro 0.2.0__tar.gz → 0.3.0__tar.gz

{taskmaestro-0.2.0 → taskmaestro-0.3.0}/.github/workflows/ci.yml

@@ -5,9 +5,19 @@ on:
     branches: [main]
   pull_request:
 
+# Least privilege: the CI job only reads the repository.
+permissions:
+  contents: read
+
+# Cancel superseded runs for the same ref (e.g. force-pushes to a PR).
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   ci:
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     strategy:
       fail-fast: false
       matrix:
@@ -20,18 +30,20 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: pyproject.toml
 
       - name: Install dependencies
         run: pip install -e ".[dev]"
 
       - name: Ruff check
-        run: ruff check taskmaestro/ tests/
+        run: ruff check taskmaestro/ tests/ examples/
 
       - name: Ruff format check
-        run: ruff format --check taskmaestro/ tests/
+        run: ruff format --check taskmaestro/ tests/ examples/
 
       - name: Mypy type check
         run: mypy taskmaestro
 
       - name: Run tests with coverage
-        run: pytest --cov=taskmaestro --cov-report=term-missing
+        run: pytest --cov=taskmaestro --cov-report=term-missing --cov-fail-under=100

{taskmaestro-0.2.0 → taskmaestro-0.3.0}/.github/workflows/publish.yml

@@ -4,10 +4,15 @@ on:
   push:
     tags: ["v*"]
 
+# Default to read-only; the publish job opts into id-token below.
+permissions:
+  contents: read
+
 jobs:
   build:
     name: Build distributions
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     steps:
       - uses: actions/checkout@v4
 
@@ -15,6 +20,8 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: "3.12"
+          cache: pip
+          cache-dependency-path: pyproject.toml
 
       - name: Install build
         run: pip install build
@@ -32,6 +39,7 @@ jobs:
     name: Publish to PyPI
     needs: build
     runs-on: ubuntu-latest
+    timeout-minutes: 10
     environment: pypi
     permissions:
       id-token: write
@@ -43,4 +51,6 @@ jobs:
           path: dist/
 
       - name: Publish via trusted publishing
-        uses: pypa/gh-action-pypi-publish@release/v1
+        # Third-party action pinned to a full commit SHA (tag v1.14.2); the
+        # `release/v1` branch is mutable and could be repointed.
+        uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2

{taskmaestro-0.2.0 → taskmaestro-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: taskmaestro
-Version: 0.2.0
+Version: 0.3.0
 Summary: A Python 3.12+ library for defining and executing typed DAG task workflows
 Project-URL: Homepage, https://github.com/OPM/taskmaestro
 Project-URL: Repository, https://github.com/OPM/taskmaestro
@@ -15,6 +15,7 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
 Requires-Python: >=3.12
@@ -36,6 +37,8 @@ Description-Content-Type: text/markdown
 
 A Python 3.12+ library for defining and executing typed DAG task workflows with Pydantic models, lifecycle hooks, and fail-fast semantics.
 
+For an agent-friendly, runnable YAML example with validation and troubleshooting steps, see the [agent quickstart](docs/agent-quickstart.md).
+
 ## Installation
 
 ```bash
@@ -140,16 +143,55 @@ You define **Tasks** (typed units of work), compose them into a **Workflow** (li
 | Concept | Description |
 |---|---|
 | **Task** | Subclass `Task[I, O]` with Pydantic models for input and output, then implement `run(input, ctx)`. Each task can declare an optional `timeout_seconds`. For tasks with multiple named outputs, use inline `Inputs`/`Outputs` classes inside the task body. |
-| **Workflow** | Build a linear pipeline with `Workflow(tasks=[...])` or a DAG with `Workflow.builder()`. The builder accepts `depends_on` for single dependencies, fan-in dicts (`{"field": UpstreamTask}`), and `(Task, "field")` tuples for output field routing. Use `config_fields` to declare which input fields come from `JobConfiguration`. Workflows are validated at build time for cycles, type compatibility, and input completeness. |
+| **Workflow** | Build a linear pipeline with `Workflow(tasks=[...])` or a DAG with `Workflow.builder()`. Prefer `builder.task()` and task handles for unambiguous dependencies; the fluent `add_task()` API remains supported. The builder accepts `collect()` for gathering outputs into collection fields and `mapped_over=TaskMap(...)` for sequential expansion over configured mappings. Use `config_fields` to declare which input fields come from `JobConfiguration`. Workflows are validated at build time for cycles, type compatibility, and input completeness. |
 | **Job** | Binds a Workflow to a typed config (the root task's input). Tracks `status` (`pending` → `running` → `completed`/`failed`), the final `result`, any `error`, and per-task `task_results`. Optionally accepts a `JobConfiguration` for per-task static config values. A job can only be run once. |
 | **Runner** | Executes tasks in topological order, stopping on the first failure (fail-fast). Supports per-task and per-job timeouts via `signal.alarm` (Unix only). Dispatches lifecycle events to registered hooks. |
 | **ExecutionContext** | Passed to every `run()` call. Provides a `logger`, an auto-generated `correlation_id` (UUID), a `scratch_dir` (temporary directory), and a service registry (`register()`/`resolve()`) for injecting shared resources like DB connections. |
 | **Hooks** | Subclass `BaseHook` and override methods like `on_job_start`, `on_task_complete`, etc. Hook errors are swallowed and reported via `warnings.warn()`, so they never crash the job. Built-ins: `LoggingHook`, `TimingHook`, `ResultPersistenceHook`. |
 | **ObjectModel** | Generic `ObjectModel[T]` base model for wrapping arbitrary (non-Pydantic) objects. Enables `arbitrary_types_allowed` so fields can hold native library objects like database connections or API clients. |
 
+For common cases, run a workflow directly without constructing `Job` and `Runner`:
+
+```python
+result = workflow.run(
+    Input(value=5),
+    task_config={"configured_task": {"option": "value"}},
+    hooks=[LoggingHook()],
+)
+```
+
+The explicit `Job` and `Runner` API remains available for advanced lifecycle control.
+
+## Task Handles
+
+`builder.task()` adds a task and returns a handle to that specific instance. Handles avoid ambiguous class and string references, especially when the same task class is registered more than once:
+
+```python
+builder = Workflow.builder(name="parallel_wells")
+model = builder.task(LoadModel)
+well_1 = builder.task(LoadWellPath, name="well_1", depends_on=model)
+well_2 = builder.task(LoadWellPath, name="well_2", depends_on=model)
+builder.task(Process, name="proc_1", depends_on=well_1)
+builder.task(Process, name="proc_2", depends_on=well_2)
+workflow = builder.build()
+```
+
+Use `handle.field("field_name")` to route one output field. Named input dependencies can be passed directly to `task()`, while keyword arguments to `collect()` provide a concise keyed collection:
+
+```python
+merged = builder.task(
+    MergeResults,
+    primary=producer.field("result"),
+    checks=collect(tests=tests, lint=lint, types=types),
+)
+builder.set_result_task(merged)
+```
+
+Handles are accepted anywhere dependency references are accepted. A handle from a different builder is rejected. Use the `depends_on` dictionary form when an input field conflicts with a reserved builder argument such as `name` or `config_fields`. The existing fluent `add_task()` API remains fully supported for backward compatibility.
+
 ## Named Task Instances
 
-The same Task class can appear multiple times in a workflow with different names. Use the `name=` parameter in `add_task()`:
+The same Task class can appear multiple times in a workflow with different names. With the fluent API, use the `name=` parameter in `add_task()`:
 
 ```python
 workflow = (
@@ -190,6 +232,8 @@ job = Job(workflow=workflow, config=EmptyConfig(), job_configuration=job_config)
 result = Runner().run(job)
 ```
 
+Every declared config field, on root, dependent and mapped tasks alike, must have a value in the `JobConfiguration`; otherwise `Job(...)` raises `WorkflowDefinitionError`.
+
 ## Nested Workflows
 
 A workflow can be wrapped as a typed task and used inside a larger workflow. Its input
@@ -261,6 +305,155 @@ workflow = (
 )
 ```
 
+## Collecting Multiple Outputs
+
+Use `collect()` when several task outputs should populate one `list[T]` or
+`dict[str, T]` field. Positional members preserve declaration order:
+
+```python
+from taskmaestro import collect
+
+class GridInput(BaseModel):
+    surfaces: list[Surface]
+
+workflow = (
+    Workflow.builder("create_grid")
+    .add_task(LoadSurface, name="top")
+    .add_task(GenerateSurface, name="middle")
+    .add_task(LoadSurface, name="base")
+    .add_task(
+        CreateGrid,
+        depends_on={"surfaces": collect("top", "middle", "base")},
+    )
+    .build()
+)
+```
+
+Use a mapping to preserve aliases in a `dict[str, T]`, and use `(task, "field")`
+to collect a specific output field:
+
+```python
+depends_on={
+    "surfaces": collect({
+        "top": ("top_loader", "surface"),
+        "base": ("base_loader", "surface"),
+    })
+}
+```
+
+The equivalent YAML forms are:
+
+```yaml
+depends_on:
+  surfaces:
+    collect:
+      - top
+      - [middle, generated_surface]
+      - base
+```
+
+```yaml
+depends_on:
+  surfaces:
+    collect:
+      top: [top_loader, surface]
+      base: [base_loader, surface]
+```
+
+Every member is checked against the field's element type when the workflow is
+built. Subtypes are accepted. `collect()` and `collect({})` explicitly create
+empty list and dictionary inputs, respectively.
+
+## Mapped Tasks
+
+A mapped task invokes one task declaration for every entry in a configured
+mapping. Mapped items execute sequentially in mapping declaration order.
+Each item gets a fresh task instance and child `ExecutionContext`.
+
+```python
+builder = Workflow.builder("create_grid")
+connection = builder.task(ConnectToResInsight)
+surfaces = builder.map_task(
+    LoadRegularSurface,
+    name="load_surfaces",
+    depends_on={"resinsight": connection},
+    config_fields=["unit"],
+    over="surfaces",
+    key_as="surface_name",
+    value_as="path",
+    error_mode="fail_fast",
+)
+builder.task(CreateGrid, depends_on={"surfaces": surfaces})
+workflow = builder.build()
+```
+
+The mapped task's input model contains the injected key and value fields, not
+the source mapping:
+
+```python
+class LoadSurfaceInput(BaseModel):
+    resinsight: RipsInstance
+    unit: str
+    surface_name: str  # key_as
+    path: str          # value_as
+```
+
+Configure the source through `JobConfiguration`:
+
+```python
+job_configuration = JobConfiguration({
+    "load_surfaces": {
+        "unit": "meters",
+        "surfaces": {
+            "top": "/data/top.irap",
+            "base": "/data/base.irap",
+        },
+    },
+})
+```
+
+The logical output is a `MappedOutput[O]` Pydantic root model containing an
+insertion-ordered `dict[str, O]`, where `O` is the task's declared output type.
+When a mapped task is connected to a named `dict[str, O]` input field, its
+`root` value is unwrapped automatically:
+
+```python
+class CreateGridInput(BaseModel):
+    surfaces: dict[str, RegularSurface]
+```
+
+The equivalent YAML task declaration is:
+
+```yaml
+- task: resinsight.load_regular_surface
+  name: load_surfaces
+  map:
+    over: surfaces
+    key_as: surface_name
+    value_as: path
+    error_mode: fail_fast
+  depends_on:
+    resinsight: resinsight.connect
+  config_fields: [unit]
+```
+
+Input YAML:
+
+```yaml
+load_surfaces:
+  unit: meters
+  surfaces:
+    top: /data/top.irap
+    base: /data/base.irap
+```
+
+`fail_fast` stops at the first failed item. `collect_all` attempts every item
+and reports an aggregate `MappedTaskExecutionError`. An empty mapping succeeds
+with `MappedOutput(root={})`. Per-item records are available in
+`job.mapped_item_results`, and
+built-in logging, timing, and persistence hooks observe individual items.
+Concurrent mapped execution is intentionally deferred.
+
 ## ObjectModel
 
 `ObjectModel[T]` wraps arbitrary (non-Pydantic) objects so they can flow through workflows. Use it as a type alias for simple wrappers, or subclass it to add extra fields:
@@ -312,6 +505,25 @@ without importing plugin modules, or `get_registered_task(name)` and
 `get_registered_workflow(name)` to load one plugin. Duplicate names and invalid plugin
 types raise `PluginLoadError`.
 
+Agents can inspect installed task plugins from the CLI without loading every plugin:
+
+```bash
+taskmaestro tasks list --json
+taskmaestro tasks describe acme.prepare --json
+```
+
+`list --json` emits `{"tasks": ["acme.prepare", ...]}` in sorted order; an empty list
+means no task plugins are installed. `describe --json` loads only the named plugin
+and emits its registered `identifier`, task `name`, `timeout_seconds`, and Pydantic
+`input_schema` / `output_schema` (JSON Schema objects). These commands inspect
+**installed entry points**, not task classes local to a workflow YAML file. Without
+`--json`, `list` prints one identifier per line and `describe` prints indented JSON.
+For runtime-only Python objects (such as `ObjectModel[rips.EclipseCase]`), schema
+fields include `"not": {}`, `"x-taskmaestro-opaque": true`, and
+`"x-taskmaestro-python-type"`. They cannot be supplied as JSON; wire them from
+upstream tasks or a Python context instead. Other unsupported schema constructs,
+unknown identifiers, and invalid plugins report an error on stderr and exit with status 2.
+
 ## YAML Configuration
 
 Workflows can be defined entirely in YAML instead of Python. A `task:` value may be
@@ -354,8 +566,9 @@ context:
 
 ```yaml
 # input.yaml
-text: "Python is a high-level programming language..."
-title: "Python Overview"
+prepare_text:
+  text: "Python is a high-level programming language..."
+  title: "Python Overview"
 ```
 
 Load and run:
@@ -371,7 +584,18 @@ result = loaded.run()
 result = run_workflow_from_yaml("workflow.yaml", "input.yaml")
 ```
 
-YAML supports named task instances (`name:`), per-task input config (keyed by task name in the input file), fan-in dicts, and output field routing via `[task, field]` lists.
+YAML input always uses per-task configuration: every top-level key in `input.yaml` must be a registered task instance name, and its value must be a mapping or `null`. Unknown task names and scalar task values are rejected. Fields are validated against the task's input model and can configure root tasks, downstream tasks, and mapped tasks.
+
+YAML also supports named task instances (`name:`), fan-in dictionaries, and output field routing via `[task, field]` lists. Named instances use their instance name as the input key:
+
+```yaml
+load_well_path_1:
+  path: first.dev
+load_well_path_2:
+  path: second.dev
+```
+
+When the same task class (or the same inner YAML file) appears more than once under different `name:`s, `depends_on` and `result_task` must use the instance name — referencing the class path is rejected as ambiguous.
 
 Use `workflow:` instead of `task:` to compose another YAML workflow. Paths are resolved
 relative to the containing workflow file, and `workflow_input:` optionally supplies the
@@ -435,19 +659,99 @@ WorkflowRunnerError (base)
 ├── JobStateError                 # e.g., re-running a completed job
 ├── ConfigLoadError               # YAML config loading failure
 └── TaskExecutionError            # Runtime task failure
+    ├── MappedTaskExecutionError  # One or more mapped items failed
     ├── TaskOutputTypeError       # Output type mismatch
     └── TaskTimeoutError          # Task exceeded timeout
 ```
 
+## Command-Line Interface
+
+Installed packages provide a `taskmaestro` command for YAML workflows:
+
+```bash
+taskmaestro validate workflow.yaml --input input.yaml
+taskmaestro graph workflow.yaml --input input.yaml
+taskmaestro run workflow.yaml --input input.yaml --log-level INFO
+taskmaestro workflow describe workflow.yaml --json
+```
+
+By default, `run` prints the final output as JSON and reports errors on stderr;
+`validate` prints a human-readable confirmation. `graph` prints Mermaid markup.
+
+For automation, use `--json` with `validate` or `run`:
+
+```bash
+taskmaestro validate workflow.yaml --input input.yaml --json
+taskmaestro run workflow.yaml --input input.yaml --json
+```
+
+Successful validation emits `{"status":"valid","workflow":"..."}`. A successful
+run emits `{"status":"completed","workflow":"...","result":{...}}`. A failure
+emits one JSON object on stdout, for example:
+
+```json
+{"status":"failed","workflow":"example","failed_task":"prepare","error":{"code":"task_failed","type":"ValueError","message":"Task failed","task":"prepare","field":null,"issues":[]}}
+```
+
+Errors contain `code`, exception `type`, a safe `message`, nullable `task` and
+`field`, and `issues` (field paths and error codes, without input values).
+Loading failures have status `invalid` and `code: "configuration_error"`; task
+failures have status `failed` and `code: "task_failed"`. If a completed result
+cannot be encoded as JSON (e.g. a Python-only object), `run --json` returns
+`code: "serialization_error"`. Missing configuration fields produce `issues`
+with code `missing`; when no structured field metadata is available, `field`
+is `null`. Use text mode when you need the original exception message.
+
+Exit codes: `0` success, `1` task or result-serialization failure, `2` workflow
+configuration failure. In JSON mode logs and ordinary Python `print()` output
+from imports/tasks go to stderr, reserving stdout for the result document.
+Application code can still write directly to file descriptor 1; JSON mode is
+not a sandbox. Error objects intentionally omit raw exception messages, but
+application-generated stderr may contain sensitive data.
+
+Inspect a workflow **before its input file is complete** with
+`taskmaestro workflow describe workflow.yaml --json`. The JSON contains the
+workflow name and result task plus topologically ordered task instances. Each
+instance includes its `name`, `python_type`, `depends_on` references (`task` /
+`field`), `config_fields`, `required_input_fields`, optional `map` configuration,
+and Pydantic input/output schemas. Collection dependencies include their kind
+(`positional` or `keyed`) and member references. `python_type` identifies the
+loaded class; it may differ from the plugin entry-point identifier used in YAML.
+
+Pass `--input input.yaml` to check the input against the workflow. The input
+may be incomplete (for example a template whose runtime-picker fields a UI
+fills in): `provided_config_fields` lists the field names set for each task,
+`missing_config_fields` lists declared `config_fields` the input does not set,
+and `config_values` maps exactly the provided names to their values; without
+`--input`, all three are `null`. Other problems, such as unknown task keys or a
+missing or invalid map source, still fail. `validate` and `run` reject missing
+configuration fields. Values are what `input.yaml` contains for
+the task, **before** Pydantic validation, so opaque markers (for example
+`{"__resinsight_ref__": "EclipseCase", "case_id": 0}`) come through unchanged.
+Dates and datetimes become ISO 8601 strings, paths become strings, sets become
+sorted lists and non-string mapping keys become JSON key strings; binary values,
+NaN/infinity and keys that collide after conversion are rejected as invalid.
+Fields without a value fall back to `input_schema.properties.<field>.default`.
+**The output contains the input values, including any secrets.** This check
+does not run tasks or hooks and does not validate every configured value's
+runtime type. Loading YAML still imports
+Python modules (and nested workflows); **do not inspect untrusted YAML or plugins**
+under a privileged account. In JSON mode inspection failures have status
+`invalid`, an `error` object, and exit code 2. Without `--json`, inspection
+prints indented JSON.
+
+`python -m taskmaestro ...` is equivalent, which is useful when the scripts directory is not on `PATH`.
+
 ## Examples
 
-Three full example pipelines are included in the `examples/` directory:
+Four full example pipelines are included in the `examples/` directory:
 
 | Example | Features |
 |---|---|
 | `examples/text_analysis/` | DAG with fan-out/fan-in, output field routing, inline `Inputs`/`Outputs` classes, YAML config, Mermaid visualization |
 | `examples/resinsight/` | `ObjectModel[T]` for gRPC objects, `JobConfiguration` with per-task config, named task instances, `config_fields`, YAML config |
 | `examples/image_processing/` | Nested workflows through `Workflow.as_task()` and YAML `workflow:`, typed boundaries, expanded Mermaid subgraph |
+| `examples/release_pipeline/` | Keyed `collect()` dependencies, mapped tasks, mapped output routing, per-task YAML config |
 
 Run an example: