dirigent-examples 0.15.0__py3-none-any.whl

dirigent_examples/shelves/graph/fan-in.yaml

@@ -0,0 +1,80 @@
+# Fan-in: reading a whole fan-out back as one list, in one downstream step.
+#
+# fan-out.yaml maps a step over a list. This is the other half: what the step *after* it
+# sees. Three rules, and they are the whole of it:
+#
+#   1. A fan-out step's output is the list of its items' outputs, in item order.
+#      ${steps.fetch.output} is a JSON array here, not an object.
+#   2. The downstream step runs ONCE, not once per item. It has no for_each of its own, so
+#      it is one attempt that reads the whole batch.
+#   3. Only items that SUCCEEDED are in that list. Under items: continue a failed item is
+#      absent rather than null, so the list can be shorter than the input, and index 0 is
+#      the first item that worked rather than the first one asked for.
+#
+# The list can be indexed: ${steps.fetch.output.0.status} is the integer 200. Bear rule 3
+# in mind before using it -- with one failure upstream, index 0 is a different dataset than
+# it was yesterday. Reading the whole list is the honest form.
+#
+# Note the deliberate asymmetry with the CLI, where `-p regions.0=east` is refused. Reading
+# an existing structure by index is unambiguous: the value is there, and it is either a list
+# or it is not. Writing into one that may not exist is not: a dotted path cannot tell the
+# index 0 from an object key named "0", and cannot say how long the array should be.
+#
+# Two edges worth knowing. A fan-out that tolerated a failure still counts as succeeded, so
+# the default rule: all_success runs this step; the run ends completed_with_errors. But if
+# *every* item fails the step itself failed, and a downstream step forced to run anyway
+# with rule: all_done is refused -- there is no stored output to read, and the attempt
+# settles as rejected rather than silently receiving an empty list.
+#
+# One block, used twice, and it runs no code on the worker, so this one needs no allowlist:
+#
+#   dg run --local examples/graph/fan-in.yaml
+#   dg run --local examples/graph/fan-in.yaml -p datasets='["temperature","humidity"]'
+#
+# The latency point: four slow fetches cost four parked rows and nothing else. The engine
+# is not holding four workers for the duration; it is holding four rows in the database.
+
+format: dirigent/v1
+kind: pipeline
+code: fan-in
+name: Branches that join
+description: Fetch several datasets in parallel, then send the whole batch on in one call.
+
+tags: [graph, http]
+
+requires:
+  blocks:
+    - http.request
+
+params:
+  type: object
+  properties:
+    datasets:
+      type: array
+      default: [temperature, humidity, pressure, wind]
+      items:
+        type: string
+
+steps:
+  fetch:
+    block: http.request
+    for_each: "${params.datasets}"
+    items: continue
+    config:
+      url: https://postman-echo.com/get
+      method: GET
+      query:
+        dataset: "${item}"
+
+  report:
+    block: http.request
+    depends_on: [fetch]
+    config:
+      url: https://postman-echo.com/post
+      method: POST
+      body:
+        # A fan-out step's output is the list of its elements' outputs, in the order the
+        # elements were created, so one reference collects the whole fan and an index
+        # reaches one element of it.
+        collected: "${steps.fetch.output}"
+        first_status: "${steps.fetch.output.0.status}"

dirigent_examples/shelves/graph/fan-out.yaml

@@ -0,0 +1,66 @@
+# Fan-out: one step mapped over a list, with one run item per element.
+#
+# for_each is expanded when the run is created, so the item grid exists from the moment the
+# run is visible. It may read params, run, and item -- but not another step's output,
+# because the cardinality has to be known before anything executes.
+#
+# items: continue is the item policy: a region that fails does not stop the others, its
+# failure is recorded against its own run item, and the run reports completed_with_errors
+# rather than failed. Under the default, fail_fast, any failed item fails the whole step.
+#
+# What a failed item costs downstream: the step still counts as succeeded, so an ordinary
+# all_success edge out of it runs. But the step's output is the list of its items' outputs
+# and a failed item is absent from it -- not null, absent -- so a step reading the batch
+# sees only what worked. If every item fails the step failed, and then the edge is skipped.
+# examples/graph/fan-in.yaml is that downstream step.
+#
+# This one uses shell.run, which executes code on the worker and is refused unless the
+# instance allowlists it: export DIRIGENT_ENABLED_UNSAFE_BLOCKS='["shell.run"]', or pass
+# --enable-unsafe shell.run to a local run.
+#
+#   dg run --local examples/graph/fan-out.yaml -p day=2026-01-01
+#   dg run --local examples/graph/fan-out.yaml -p regions='["east","west","north","south"]'
+
+format: dirigent/v1
+kind: pipeline
+code: fan-out
+name: One step fans out
+description: Push one payload per region, tolerating a region that refuses it.
+
+tags: [graph, execute, http]
+
+params:
+  type: object
+  required: [day]
+  properties:
+    day:
+      type: string
+      format: date
+    regions:
+      type: array
+      default: [east, west, north]
+      items:
+        type: string
+
+steps:
+  push:
+    block: http.request
+    for_each: "${params.regions}"
+    # One element per region, each its own attempt. items: continue lets the rest finish
+    # when one region refuses; items: halt would stop the fan-out at the first failure.
+    items: continue
+    config:
+      url: https://postman-echo.com/post
+      method: POST
+      body:
+        day: "${params.day}"
+        region: "${item}"
+
+  audit:
+    block: shell.run
+    depends_on: [push]
+    # all_done runs the audit once every element has settled, failures included. The
+    # default all_success would skip it precisely when there is something to audit.
+    rule: all_done
+    config:
+      argv: [echo, "the push step has settled for every region"]

dirigent_examples/shelves/graph/linear.yaml

@@ -0,0 +1,66 @@
+# A four-step chain, and how one step reads another's output.
+#
+# depends_on draws the edge; ${steps.<name>.output.<field>} reads across it. A step may
+# only read from a step it depends on, transitively, and apply checks that: the reference
+# language has no way to reach sideways into an unrelated branch.
+#
+# The answer travels as a value in the fetch step's output, and storage.write is what puts
+# it in the run's scratch space, so the copy step has a real object to move.
+#
+# report uses shell.run, which executes code on the worker and is refused unless the
+# instance allowlists it:
+#   export DIRIGENT_ENABLED_UNSAFE_BLOCKS='["shell.run"]'
+#   dg run --local examples/graph/linear.yaml -p day=2026-01-01
+# or, without touching the environment:
+#   dg run --local examples/graph/linear.yaml -p day=2026-01-01 --enable-unsafe shell.run
+
+format: dirigent/v1
+kind: pipeline
+code: linear
+name: A straight line of steps
+description: Fetch a payload, store it, archive the object, then report what was archived.
+
+tags: [graph, execute, http, storage]
+
+params:
+  type: object
+  required: [day]
+  properties:
+    day:
+      type: string
+      format: date
+
+steps:
+  fetch:
+    block: http.request
+    config:
+      url: https://postman-echo.com/get
+      method: GET
+      query:
+        day: "${params.day}"
+
+  store:
+    block: storage.write
+    depends_on: [fetch]
+    config:
+      # storage.write is the only way a value leaves the run, so the answer the fetch
+      # carries becomes an object here and nowhere else.
+      target: "${run.scratch}/incoming/${params.day}.json"
+      value: "${steps.fetch.output.body}"
+
+  archive:
+    block: storage.copy
+    # depends_on is what orders the run; nothing infers an edge from one step reading
+    # another's output, so the dependency is stated even though the reference implies it.
+    depends_on: [store]
+    config:
+      source: "${steps.store.output.uri}"
+      target: "${run.scratch}/archive/${params.day}.json"
+
+  report:
+    block: shell.run
+    depends_on: [archive]
+    config:
+      argv:
+        - echo
+        - "service ${steps.fetch.output.status}, archived ${steps.archive.output.bytes_copied} bytes"

dirigent_examples/shelves/graph/parallel-branches.yaml

@@ -0,0 +1,57 @@
+# A diamond: one root, two independent branches, one join.
+#
+# Parallelism is implicit. Nothing declares that read_side and write_side run at the same
+# time; they simply have no edge between them, so both become claimable the moment their
+# shared prerequisite succeeds and whichever workers are free pick them up.
+#
+# The join uses the default rule, all_success, so summarise runs only if both branches
+# succeeded. Change it to all_done and it would run regardless.
+#
+# This one uses shell.run, which executes code on the worker and is refused unless the
+# instance allowlists it: export DIRIGENT_ENABLED_UNSAFE_BLOCKS='["shell.run"]', or pass
+# --enable-unsafe shell.run to a local run.
+#
+#   dg run --local examples/graph/parallel-branches.yaml
+
+format: dirigent/v1
+kind: pipeline
+code: parallel-branches
+name: Branches in parallel
+description: Call two endpoints in parallel and summarise both results.
+
+tags: [graph, execute, http]
+
+steps:
+  prepare:
+    block: shell.run
+    config:
+      argv: [echo, "starting both branches"]
+
+  read_side:
+    block: http.request
+    depends_on: [prepare]
+    config:
+      url: https://postman-echo.com/get
+      method: GET
+      query:
+        branch: read
+
+  write_side:
+    block: http.request
+    # Both branches depend on prepare and on nothing else, which is the whole of what makes
+    # them parallel: the engine runs whatever is claimable, and nothing here orders these two.
+    depends_on: [prepare]
+    config:
+      url: https://postman-echo.com/post
+      method: POST
+      body:
+        branch: write
+
+  summarise:
+    block: shell.run
+    depends_on: [read_side, write_side]
+    rule: all_success
+    config:
+      argv:
+        - echo
+        - "read ${steps.read_side.output.status}, write ${steps.write_side.output.status}"

dirigent_examples/shelves/graph/parallel-sleep.yaml

@@ -0,0 +1,55 @@
+# Fan-out and fan-in with no fan-out syntax at all: four plain steps, drawn by their edges.
+#
+# The three sleeps declare no depends_on, so they are all roots, and every root is claimable
+# the moment the run is created. Nothing serialises them but worker concurrency: what bounds
+# this run is how many block calls a worker carries at a time, not the shape of the DAG. Run
+# serially the waits would take nine seconds, and this run takes about five. Time it -- the
+# example proves its own claim.
+#
+# report depends on all three, and the default trigger rule is all_success, so it waits for
+# every one of them and reads each one's output. That is the join: no barrier step, no
+# gather syntax, just an edge from each sleep.
+#
+# time.sleep is a sensor, so none of those seconds occupy a worker: each poke reads the clock
+# and returns, and the wait is a parked row the engine wakes later. The same shape with
+# shell.run and sleep would hold a worker slot for the whole wait.
+#
+# The asymmetry is worth seeing: the sleeps need no allowlist, and report does, because
+# shell.run executes code on the worker and is refused unless the instance allows it.
+#
+#   time dg run --local examples/graph/parallel-sleep.yaml --enable-unsafe shell.run
+
+format: dirigent/v1
+kind: pipeline
+code: parallel-sleep
+name: Sleeps in parallel
+description: Three independent waits that run at once, and one step that waits for all of them.
+
+tags: [graph, execute, sensor]
+
+steps:
+  # None of the three sleeps depends on another, so they are claimable at once and the run
+  # takes as long as the longest rather than as long as the sum.
+  slow:
+    block: time.sleep
+    config:
+      for: 5s
+
+  medium:
+    block: time.sleep
+    config:
+      for: 3s
+
+  quick:
+    block: time.sleep
+    config:
+      for: 1s
+
+  report:
+    block: shell.run
+    depends_on: [slow, medium, quick]
+    config:
+      argv:
+        - echo
+        - "3 steps completed slow=${steps.slow.output.waited_ms}ms
+          medium=${steps.medium.output.waited_ms}ms quick=${steps.quick.output.waited_ms}ms"

dirigent_examples/shelves/graph/skip-diamond.yaml

@@ -0,0 +1,105 @@
+# The happy path, where the error handler is skipped and the run is green anyway.
+#
+# error-handler.yaml runs the failure path: the load fails, the one_failed branch fires. This
+# is the same diamond taking the other road, and it exists because a skipped step is the
+# outcome people misread most often.
+#
+# A skip is not a failure. It is not a step that did nothing, either, and it is not an error
+# suppressed. It is the engine reporting that an edge condition was evaluated and came out
+# false, which is an ordinary, correct, green thing for a run to contain. A run whose alert
+# branch was skipped is a run where nothing needed alerting about.
+#
+# Four steps, four outcomes worth reading off the graph as written:
+#
+#   verify          succeeds   the load is healthy
+#   raise_alarm     SKIPPED    rule: one_failed, and nothing failed, so the edge is false
+#   record_success  succeeds   rule: all_success, and the edge is true
+#   close_out       succeeds   rule: all_done, which is satisfied by a skip as much as by a
+#                              success -- this is why cleanup hangs off all_done and never
+#                              off all_success, which would wait on a step that never ran
+#
+# Flip it with the parameter and the same four steps swap two of those outcomes: raise_alarm
+# runs, record_success is skipped, close_out still runs. Same document, same edges, no
+# branching syntax anywhere -- only the outcome of one step changed.
+#
+# One caution the graph does not show. one_failed reads the outcome a dependent sees, and a
+# step marked continue_on_failure: true shows its dependents a success even when it failed.
+# So an error handler hung under a tolerated step never fires. Tolerating a failure and
+# handling one are opposite instructions; optional-step.yaml is the one that tolerates.
+#
+# This one uses shell.run, which executes code on the worker and is refused unless the
+# instance allowlists it: export DIRIGENT_ENABLED_UNSAFE_BLOCKS='["shell.run"]', or pass
+# --enable-unsafe shell.run to a local run.
+#
+#   dg run --local examples/graph/skip-diamond.yaml --enable-unsafe shell.run              # green, one skip
+#   dg run --local examples/graph/skip-diamond.yaml --enable-unsafe shell.run -p fail=true # the other road
+
+format: dirigent/v1
+kind: pipeline
+code: skip-diamond
+name: Skip diamond
+description: |
+  A load that works, an alarm that is therefore skipped, and a close-out that runs either way.
+
+  **A skip is not a failure.** It is an edge condition that evaluated false, and a run
+  full of them can be perfectly green.
+
+  | Step | Rule | On the happy path |
+  | --- | --- | --- |
+  | `verify` | -- | succeeds |
+  | `raise_alarm` | `one_failed` | skipped |
+  | `record_success` | `all_success` | succeeds |
+  | `close_out` | `all_done` | succeeds |
+
+  Pass `-p fail=true` and the middle two swap places. `close_out` does not move, which is
+  the whole reason cleanup hangs off `all_done`.
+
+tags: [graph, execute]
+
+requires:
+  blocks:
+    - shell.run
+
+params:
+  type: object
+  properties:
+    fail:
+      type: boolean
+      description: Make the verification fail, to take the branch this example does not.
+      default: false
+
+steps:
+  verify:
+    block: shell.run
+    config:
+      # Exits 0 unless asked otherwise, so the default run takes the road where the handler
+      # is skipped rather than the one error-handler.yaml already shows.
+      command: "if [ '${params.fail}' = 'true' ]; then echo 'checksum mismatch' >&2; exit 1; fi; echo 'verified 412 rows'"
+
+  raise_alarm:
+    block: shell.run
+    depends_on: [verify]
+    # Skipped on the happy path. Nothing failed, so there is nothing for a handler to handle,
+    # and the engine says so in the outcome rather than running it and having it decide.
+    rule: one_failed
+    config:
+      argv: [echo, "alarm raised: verification failed"]
+
+  record_success:
+    block: shell.run
+    depends_on: [verify]
+    # The default rule, written out here only because its opposite is directly above it.
+    rule: all_success
+    config:
+      argv: [echo, "recorded a clean verification"]
+
+  close_out:
+    block: shell.run
+    depends_on: [raise_alarm, record_success]
+    # Exactly one of the two above ran and the other was skipped, and all_done is satisfied by
+    # both -- a skip is settled. all_success here would itself be skipped: a skipped
+    # prerequisite can never satisfy it, so the skip would propagate down and this step would
+    # never run -- the mistake this step exists to not make.
+    rule: all_done
+    config:
+      argv: [echo, "closed out, whichever road was taken"]

dirigent_examples/shelves/graph/wide-fan.yaml

@@ -0,0 +1,147 @@
+# Graph width: one start, eight branches that are all claimable at once, and one join.
+#
+# This is deep-chain.yaml turned ninety degrees. There the ten steps drew a column and the
+# run could not go faster than one worker; here the eight branches draw a row, every one of
+# them depends only on the split, and the run goes exactly as fast as the workers allow.
+#
+# Width is not a keyword. Nothing here declares parallelism: the eight branches are parallel
+# because none of them names another in depends_on, and that is the whole mechanism. The
+# engine claims every step whose prerequisites have settled, so all eight become claimable in
+# the same instant the split settles. What actually bounds them is worker concurrency -- the
+# number of block calls a worker carries at a time -- which is an instance setting and not
+# something a document gets a say in.
+#
+# collect is the join. It names all eight, and the default rule is all_success, so it waits
+# for every one of them before it runs once. That is the same shape parallel-sleep.yaml draws
+# with three sensors; the difference is that this one has a common start above the fan, which
+# makes it a proper diamond rather than eight roots.
+#
+# Nothing here reaches the network or runs code on the worker, so it needs no allowlist.
+#
+#   dg run --local examples/graph/wide-fan.yaml
+#   dg run --local examples/graph/wide-fan.yaml -p factor=10
+
+format: dirigent/v1
+kind: pipeline
+code: wide-fan
+name: Wide fan
+description: |
+  One step splits, eight run at once, one joins them back.
+
+  A DAG's **width** is how many steps are claimable at the same moment, and it is what a
+  run demands of the workers. Its opposite is `deep-chain`, where the same ten steps stand
+  in a single line and no worker count helps.
+
+  Nothing declares the parallelism. The eight branches are parallel because none of them
+  names another.
+
+tags: [graph, transform]
+
+requires:
+  blocks:
+    - transform.jq
+
+params:
+  type: object
+  properties:
+    factor:
+      type: integer
+      description: What each branch multiplies its own region weight by.
+      default: 2
+
+steps:
+  # The single root. Everything below reads it, so nothing is claimable until it settles --
+  # which is what makes the eight start together rather than trickling.
+  split:
+    block: transform.jq
+    config:
+      input:
+        factor: "${params.factor}"
+      program: "{factor: .factor, regions: 8}"
+
+  # Eight separate steps rather than one for_each, because this example is about the shape of
+  # the graph. fan-out.yaml is the same parallelism expressed as one step over a list, and
+  # that is what a real pipeline over eight regions would use.
+  bagmati:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"bagmati\", weight: (.factor * 3)}"
+
+  gandaki:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"gandaki\", weight: (.factor * 5)}"
+
+  karnali:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"karnali\", weight: (.factor * 7)}"
+
+  koshi:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"koshi\", weight: (.factor * 11)}"
+
+  lumbini:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"lumbini\", weight: (.factor * 13)}"
+
+  madhesh:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"madhesh\", weight: (.factor * 17)}"
+
+  sudurpashchim:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"sudurpashchim\", weight: (.factor * 19)}"
+
+  national:
+    block: transform.jq
+    depends_on: [split]
+    config:
+      input: "${steps.split.output.value}"
+      program: "{region: \"national\", weight: (.factor * 23)}"
+
+  # The join. Eight edges in, one attempt, and every branch's output read by name. There is
+  # no gather syntax and no barrier step: the edges are the join.
+  collect:
+    block: transform.jq
+    depends_on:
+      - bagmati
+      - gandaki
+      - karnali
+      - koshi
+      - lumbini
+      - madhesh
+      - sudurpashchim
+      - national
+    config:
+      input:
+        - "${steps.bagmati.output.value}"
+        - "${steps.gandaki.output.value}"
+        - "${steps.karnali.output.value}"
+        - "${steps.koshi.output.value}"
+        - "${steps.lumbini.output.value}"
+        - "${steps.madhesh.output.value}"
+        - "${steps.sudurpashchim.output.value}"
+        - "${steps.national.output.value}"
+      program: |
+        {branches: length,
+         total_weight: (map(.weight) | add),
+         heaviest: (max_by(.weight) | .region)}

dirigent_examples/shelves/hello-world.yaml

@@ -0,0 +1,30 @@
+# The smallest thing dirigent can run: one step, one block, no parameters.
+#
+# value.const emits its configured value and touches nothing, so this runs on a fresh
+# instance with nothing on the unsafe allowlist:
+#
+#   dg apply examples/hello-world.yaml
+#   dg run hello-world --watch
+
+format: dirigent/v1
+kind: pipeline
+code: hello-world
+# The code above is the identity: it is what a URL, a reference and `dg run` carry.
+# The name below is display only -- free-form, and nothing ever references by it.
+name: Hello, world
+description: |
+  Emit a greeting, and nothing else.
+
+  This is the **smallest** document the format admits: one step, one block, no
+  parameters. A description is markdown, so the UI renders it as prose:
+
+  - `value.const` emits its config as its output, with nothing granted
+  - everything else here is a default
+
+tags: [demo]
+
+steps:
+  greet:
+    block: value.const
+    config:
+      value: "hello from dirigent"

dirigent_examples/shelves/open-data/README.md

@@ -0,0 +1,67 @@
+# Open data
+
+Real pipelines against real public systems. Every source on this shelf is a live, public API,
+and every one of them but two is keyless: no account, no token, no registration, nothing to
+stand up. `dg run --local examples/open-data/<file>.yaml` reaches the internet and comes back
+with today's data.
+
+The two exceptions say so in their headers and in the table below. `kobo-submissions-to-csv`
+needs an account token, because the submissions are somebody's household survey and there is no
+anonymous read; `odk-central-submissions` needs a server and an account for the same reason.
+They are here because a shelf of only-open sources would never show where a credential belongs
+-- in a connection the instance holds, not in the document.
+
+That distinction is what "it runs" means on each row. Every keyless document here has been run
+end to end against its live source and ended `succeeded` with real data;
+`gdacs-disaster-updates` was run twice under one `dg run --local --root`, a day apart, to see
+both the empty first day and the second day's difference. The two credentialed ones stop where the credential does: their
+documents validate and their graphs are the same shape, but nobody's Kobo token or ODK Central
+server is in this repository, so the request itself is what a reader has to supply.
+
+Nothing here uses a block outside the core catalog, and nothing runs code on a worker: no
+`shell.run`, no allowlist entry, no `--enable-unsafe`. Reshaping is jq, encoding is a codec,
+waiting is a sensor.
+
+| File | What it teaches |
+| --- | --- |
+| [open-meteo-weekly-report.yaml](open-meteo-weekly-report.yaml) | A windowed weekly schedule, a columnar API transposed to rows, and csv written under the window's start |
+| [who-gho-indicators-to-parquet.yaml](who-gho-indicators-to-parquet.yaml) | A fan-out read back as one list, flattened into one table, written as one parquet file, and a manifest that counts what landed |
+| [world-bank-population-trend.yaml](world-bank-population-trend.yaml) | Paging made visible: an envelope checked with `error()`, year-on-year arithmetic in jq, and a carried schema gating the rows |
+| [usgs-earthquakes-alert.yaml](usgs-earthquakes-alert.yaml) | Haversine in jq, thresholds carried as data rather than spliced into a program, and posting **only** when something matched |
+| [overpass-health-facilities.yaml](overpass-health-facilities.yaml) | A query language in a query parameter, nodes and ways reconciled to one shape, and the same rows written as csv and as parquet |
+| [wikidata-country-reference.yaml](wikidata-country-reference.yaml) | SPARQL with content negotiation, the W3C results envelope unwrapped, and a reference table saved for other pipelines to read |
+| [gdacs-disaster-updates.yaml](gdacs-disaster-updates.yaml) | A daily window, deduplication against yesterday's saved list, and what the first day looks like when there is no yesterday |
+| [hdx-dataset-watch.yaml](hdx-dataset-watch.yaml) | The marker pattern: storage standing in for state, and a marker overwritten under `rule: all_done`, whichever way the comparison went |
+| [github-releases-relay.yaml](github-releases-relay.yaml) | A rate limit as a design constraint, string comparison of ISO stamps, and the remaining budget recorded beside the result |
+| [nominatim-geocode-facilities.yaml](nominatim-geocode-facilities.yaml) | One request per second expressed as the shape of the graph: a chain with `time.sleep` between the lookups, not a fan-out |
+| [kobo-submissions-to-csv.yaml](kobo-submissions-to-csv.yaml) | Where a credential belongs, and why a csv needs its columns named up front |
+| [odk-central-submissions.yaml](odk-central-submissions.yaml) | The same survey story with the secret in a connection, and an OData feed with grouped questions |
+| [feeds-composition.yaml](feeds-composition.yaml) | `pipeline.run` over two of the above: parameters down, statuses up, and data across only through storage |
+
+## The two patterns worth stealing
+
+**Do nothing, successfully.** There is no `if` in the format and no conditional edge. What there
+is: a decision written to storage as ndjson, and `storage.exists` asked for it with
+`min_size: 1b` and `on_timeout: skip`. An empty decision encodes as an empty file, the floor is
+never met, the sensor skips, and every step behind it skips with it -- while the run still ends
+`succeeded`. `usgs-earthquakes-alert.yaml`, `hdx-dataset-watch.yaml` and
+`github-releases-relay.yaml` each use it, and each says why at the step.
+
+**Remember, without a state block.** Nothing carries over between runs except what a run wrote.
+A marker is a small object at a stable path: a sensor asks whether it is there, a transform
+reads it, the comparison decides, and the last step overwrites it under `rule: all_done`.
+`hdx-dataset-watch.yaml` is the whole pattern in one file. Under `--local` the artifact root is
+a throwaway directory, so every local run is a first sighting; on an instance, a stable prefix
+or an `s3://` bucket is what makes the second run the interesting one.
+
+## Two things the reference language will not do
+
+A `${...}` reference reads `params`, `steps.<name>.output.<path>`, `item`, and `run` --
+`run.scratch`, `run.id`, `run.window.start`, `run.window.end`. Two consequences show up
+repeatedly on this shelf:
+
+- **A parameter default is literal text.** `default: ${run.scratch}/marker.json` is those
+  characters, not a path. Anything that has to resolve belongs in a step's config.
+- **A jq program never has a value spliced into it.** Data reaches a program through `input`,
+  which is why the earthquake filter carries its thresholds on every row it tests and the
+  briefing puts the country in its input rather than in its program.