dirigent-examples 0.15.0__py3-none-any.whl

dirigent_examples/shelves/recipes/README.md

@@ -0,0 +1,130 @@
+# Recipes
+
+One question per file. Every recipe here is a small, complete, runnable answer to something
+a person doing data work actually asks -- how do I group and sum, how do I join two lists,
+what survives a parquet round trip, where do credentials live -- and its header comment is
+the lesson: what it does, what goes in and what comes out, why each step is configured the
+way it is, and what to change to make it yours.
+
+Nothing here needs infrastructure. The data is written inline, generated by a jq program, or
+fetched from [Postman Echo](https://postman-echo.com), a public service that answers with
+the request it was given. Nothing needs the unsafe-block allowlist: no recipe on this shelf
+runs code on a worker.
+
+```bash
+dg run --local examples/recipes/jq-group-by-and-sum.yaml
+```
+
+One recipe is handed something: [schema-referenced.yaml](schema-referenced.yaml) names a
+schema the instance holds, so a local run is given it, and without it the run is refused
+before anything executes -- which is the point that recipe makes.
+
+```bash
+dg run --local --schema examples/schemas/ou-record.json examples/recipes/schema-referenced.yaml
+```
+
+Four of them fail on purpose, and their headers say so. Failing is the lesson in two of them
+and a parameter away in the others:
+
+| Recipe | Ends as | Because |
+| --- | --- | --- |
+| [schema-refuses-then-rule.yaml](schema-refuses-then-rule.yaml) | `failed` | The gate refuses a level of 0, the load is skipped, and the `one_failed` handler runs |
+| [reconcile-two-sources.yaml](reconcile-two-sources.yaml) | `failed` | Three rows differ beyond the tolerance and the limit is two; `-p max_differing=5` passes |
+| [http-success-status-list.yaml](http-success-status-list.yaml) | `succeeded` | 418 is listed as success; `-p status=500` is not, and fails |
+| [csv-header-rules.yaml](csv-header-rules.yaml) | `succeeded` | `-p header=station,station,celsius` provokes the codec's refusal |
+
+## jq: reshaping a value
+
+| File | The question it answers |
+| --- | --- |
+| [jq-group-by-and-sum.yaml](jq-group-by-and-sum.yaml) | How do I total a column per group, with a grand total that still adds up? |
+| [jq-join-two-lists.yaml](jq-join-two-lists.yaml) | How do I join two lists on a key, without losing the rows that match nothing? |
+| [jq-pivot-wide-to-long.yaml](jq-pivot-wide-to-long.yaml) | How do I melt a column-per-measure table into one record per measurement? |
+| [jq-long-to-wide.yaml](jq-long-to-wide.yaml) | How do I widen it back, with an explicit fill in every unreported cell? |
+| [jq-dedupe-by-key.yaml](jq-dedupe-by-key.yaml) | Which row wins when a key repeats -- first, last, or newest by timestamp? |
+| [jq-window-dates.yaml](jq-window-dates.yaml) | How do I build a window of days and ISO weeks, and why is `strptime` alone not a date? |
+| [jq-nested-to-flat.yaml](jq-nested-to-flat.yaml) | How do I flatten nested records into the flat ones a csv or parquet writer accepts? |
+| [jq-defaults-and-nulls.yaml](jq-defaults-and-nulls.yaml) | How do I default a field without `//` turning every `false` back into `true`? |
+| [jq-string-cleaning.yaml](jq-string-cleaning.yaml) | How do I trim, case-fold and extract the strings a spreadsheet export brings in? |
+| [jq-top-n.yaml](jq-top-n.yaml) | How do I take the top N without hiding the ties or the tail? |
+| [jq-running-totals.yaml](jq-running-totals.yaml) | How do I add cumulative sums, deltas and a moving average to an ordered series? |
+| [jq-validate-in-jq-vs-schema.yaml](jq-validate-in-jq-vs-schema.yaml) | When do I sort a batch in jq, and when do I gate it with a schema? |
+
+## map and filter: the element-wise verbs
+
+| File | The question it answers |
+| --- | --- |
+| [map-enrich-with-lookup.yaml](map-enrich-with-lookup.yaml) | How do I enrich every row from a lookup table a `map.jq` program cannot see? |
+| [filter-by-predicate.yaml](filter-by-predicate.yaml) | How do I write a predicate that answers true or false, since truthiness is not applied? |
+| [filter-then-map-then-reduce.yaml](filter-then-map-then-reduce.yaml) | Which verb is which, in the order a pipeline usually needs them? |
+
+## Converting between formats
+
+| File | The question it answers |
+| --- | --- |
+| [csv-to-ndjson.yaml](csv-to-ndjson.yaml) | How do I read a csv, and who gives the string cells their types? |
+| [ndjson-to-parquet.yaml](ndjson-to-parquet.yaml) | How do I write records as parquet, and read the file back to see what landed? |
+| [parquet-round-trip-types.yaml](parquet-round-trip-types.yaml) | Which types survive a parquet round trip, and which does a csv flatten? |
+| [csv-header-rules.yaml](csv-header-rules.yaml) | What are the header rules, and which two headers are refused outright? |
+| [json-to-csv-flattening.yaml](json-to-csv-flattening.yaml) | How do I turn a nested API payload into a csv somebody opens in a spreadsheet? |
+
+## Validating a shape
+
+| File | The question it answers |
+| --- | --- |
+| [schema-carried.yaml](schema-carried.yaml) | How do I carry the shape in the document, so the pipeline runs with nothing handed to it? |
+| [schema-referenced.yaml](schema-referenced.yaml) | How do I name a schema the instance holds, and declare that dependency? |
+| [schema-formats.yaml](schema-formats.yaml) | Which `format` keywords actually assert here, and what does a pack contribute? |
+| [schema-refuses-then-rule.yaml](schema-refuses-then-rule.yaml) | What runs after a gate refuses -- and what does not? |
+
+## Storage
+
+| File | The question it answers |
+| --- | --- |
+| [storage-write-then-read.yaml](storage-write-then-read.yaml) | How do I park a payload in storage and pick it up again a step later? |
+| [storage-copy-dated-archive.yaml](storage-copy-dated-archive.yaml) | How do I keep a dated copy, and what should the key layout be? |
+| [storage-exists-gate.yaml](storage-exists-gate.yaml) | How do I wait for an object, and what stops me reading a half-written one? |
+| [large-output-to-storage.yaml](large-output-to-storage.yaml) | What happens to a payload past the inline threshold, and what should I do about it? |
+| [storage-manifest-of-a-fan-out.yaml](storage-manifest-of-a-fan-out.yaml) | How do I write one file per item and then say, in the run, what landed? |
+| [report-to-file.yaml](report-to-file.yaml) | How do I render a page, write it to a file, and read it back to see that it landed? |
+
+## HTTP and webhooks
+
+| File | The question it answers |
+| --- | --- |
+| [http-get-with-query.yaml](http-get-with-query.yaml) | How do I build a query without hand-writing a URL, and which field holds the answer? |
+| [http-post-json-echo.yaml](http-post-json-echo.yaml) | How do I POST a body assembled upstream, as structure rather than as text? |
+| [http-fetch-validate-post.yaml](http-fetch-validate-post.yaml) | How do I fetch, check what came back, reshape it, check what I made, and post it on? |
+| [http-headers-and-auth-connection.yaml](http-headers-and-auth-connection.yaml) | Where do credentials live, and what belongs in a header map instead? |
+| [http-save-body-to-storage.yaml](http-save-body-to-storage.yaml) | How do I keep a response body as an object somebody else can fetch? |
+| [http-post-file-from-storage.yaml](http-post-file-from-storage.yaml) | How do I POST a file that lives in storage? |
+| [http-success-status-list.yaml](http-success-status-list.yaml) | How do I accept a non-2xx that is a real answer, without silencing failures? |
+| [http-follow-redirects.yaml](http-follow-redirects.yaml) | Why is a redirect not followed by default, and what do I lose when it is? |
+| [http-timeout-override.yaml](http-timeout-override.yaml) | Which of the three deadlines around a call am I actually setting? |
+| [webhook-post-hmac.yaml](webhook-post-hmac.yaml) | What exactly gets signed, with what, and what does a receiver check? |
+| [webhook-post-summary.yaml](webhook-post-summary.yaml) | What belongs in a notification body somebody has to act on? |
+| [report-to-webhook.yaml](report-to-webhook.yaml) | How do I POST a rendered page to a receiver, and what travels beside it? |
+
+## Whole small pipelines
+
+| File | The question it answers |
+| --- | --- |
+| [etl-csv-clean-validate-parquet.yaml](etl-csv-clean-validate-parquet.yaml) | What does a whole small ETL look like, from a messy csv to a checked parquet file? |
+| [report-daily-digest.yaml](report-daily-digest.yaml) | How do I compute a day's numbers once and let the csv, the digest and the notification agree? |
+| [reconcile-two-sources.yaml](reconcile-two-sources.yaml) | How do I say exactly how two systems disagree, in four buckets and with a tolerance? |
+| [pagination-by-fan-out.yaml](pagination-by-fan-out.yaml) | How do I fetch several pages at once, and what does `for_each` not let me do? |
+
+## The run's own report
+
+| File | The question it answers |
+| --- | --- |
+| [report-built-in.yaml](report-built-in.yaml) | How do I get a page about the run itself, without writing a line of template? |
+| [http-post-report.yaml](http-post-report.yaml) | How do I write that page myself, and which of the run's facts may it read? |
+
+## The pages behind them
+
+[docs/jq.md](../../docs/jq.md) teaches the language, [docs/transforms.md](../../docs/transforms.md)
+the four verbs and their frames, [docs/json-schema.md](../../docs/json-schema.md) the shapes,
+[docs/reports.md](../../docs/reports.md) every fact a report template may read,
+and [docs/blocks.md](../../docs/blocks.md) is generated from the live catalog, so it is the
+honest answer to what a block's config actually takes.

dirigent_examples/shelves/recipes/csv-header-rules.yaml

@@ -0,0 +1,147 @@
+# The four rules the csv codec keeps about headers, and the two headers it refuses.
+#
+# In:  a csv whose rows are ragged, and a list of records whose keys are not all the same.
+# Out: {read, written, header} -- the ragged csv read into records, and records written back
+#      out as a csv whose header covers every key any of them has.
+#
+# A convert step reads one URI and writes another, so each half below is three hops: the
+# document is put in storage, converted, and read back out. The two conversions are the
+# middle hop of each half, and the csv rules are what they keep.
+#
+# Reading, two rules:
+#
+#   The header names the columns, and every cell is a string. A row with fewer cells than
+#   the header names gets an empty string in the missing ones rather than a missing key, so
+#   every record has the same shape.
+#   A row with MORE cells than the header names is refused, naming the row: cells with no
+#   column are data nobody can address, and dropping them silently is worse than stopping.
+#
+# Writing, two rules:
+#
+#   The header is the union of the keys the rows have, in first-seen order across the
+#   records. A key only the third record carries is still a column, and the records without
+#   it get an empty cell -- a header taken from the first record alone would silently drop
+#   data. Within one record the order is whatever the value carries by the time it reaches
+#   the codec, which is not always the order it was typed in; a report whose header is part
+#   of its contract builds its objects field by field in a jq step, which keeps that order.
+#   A nested value has no csv spelling, so it is refused naming the row and the key. Flatten
+#   first: jq-nested-to-flat.yaml is that step.
+#
+# Two headers are refused outright when reading, and both refusals are worth provoking once:
+#
+#   station,station,celsius   ->  the csv header repeats a column name: 'station' at columns
+#                                 1, 2; a record key names one column, so rename them before
+#                                 converting
+#   station,,celsius          ->  the csv header has no name at column 2, and a record key
+#                                 names its column; name it before converting
+#
+# Both are refusals rather than repairs because a repair has to invent a name -- station_2,
+# column_2 -- and every reader downstream then depends on an invention this codec made up.
+#
+# To see either one, run with -p header='station,station,celsius'; that run fails on purpose
+# with the message above.
+#
+#   dg run --local examples/recipes/csv-header-rules.yaml
+#   dg run --local examples/recipes/csv-header-rules.yaml -p header=station,station,celsius
+#   dg run --local examples/recipes/csv-header-rules.yaml -p header=station,,celsius
+
+format: dirigent/v1
+kind: pipeline
+code: csv-header-rules
+name: CSV header rules
+description: How the csv codec reads a ragged file and writes a union header, and the two headers it refuses to read at all.
+
+tags: [recipes, storage, transform]
+
+requires:
+  blocks:
+    - value.const
+    - transform.jq
+    - storage.write
+    - convert.std
+    - storage.read
+
+params:
+  type: object
+  properties:
+    header:
+      type: string
+      default: station,date,celsius
+      description: The header row the reading half is given; a repeated or empty name is refused.
+
+steps:
+  ragged:
+    block: transform.jq
+    config:
+      input: ${params.header}
+      # The rows are short on purpose: st-2 has no celsius cell at all.
+      program: |
+        . + "\nst-1,2026-01-01,4.5\nst-2,2026-01-01\nst-3,2026-01-01,7\n"
+
+  ragged_file:
+    block: storage.write
+    depends_on: [ragged]
+    config:
+      target: ${run.scratch}/ragged.csv
+      text: ${steps.ragged.output.value}
+      content_type: text/csv
+
+  read:
+    block: convert.std
+    depends_on: [ragged_file]
+    config:
+      source: ${steps.ragged_file.output.uri}
+      target: ${run.scratch}/ragged.json
+      from: csv
+      to: json
+
+  records:
+    block: storage.read
+    depends_on: [read]
+    config:
+      # The converted object is application/json, so it comes back as a value.
+      source: ${steps.read.output.target}
+
+  uneven:
+    block: value.const
+    config:
+      value:
+        - { station: st-1, celsius: 4.5 }
+        # A key the first record does not have: it is still a column.
+        - { station: st-2, celsius: -1, battery: 96 }
+        - { station: st-3, note: replaced sensor }
+
+  uneven_file:
+    block: storage.write
+    depends_on: [uneven]
+    config:
+      target: ${run.scratch}/uneven.json
+      # Canonical JSON: the keys reach the codec sorted, whatever order they were typed in.
+      value: ${steps.uneven.output.value}
+
+  write:
+    block: convert.std
+    depends_on: [uneven_file]
+    config:
+      source: ${steps.uneven_file.output.uri}
+      target: ${run.scratch}/uneven.csv
+      from: json
+      to: csv
+
+  written:
+    block: storage.read
+    depends_on: [write]
+    config:
+      source: ${steps.write.output.target}
+
+  report:
+    block: transform.jq
+    depends_on: [records, written]
+    config:
+      input:
+        read: ${steps.records.output.value}
+        written: ${steps.written.output.text}
+      program: |
+        {read: .read,
+         written: .written,
+         header: (.written | split("\n")[0] | split(","))}

dirigent_examples/shelves/recipes/csv-to-ndjson.yaml

@@ -0,0 +1,107 @@
+# A csv turned into ndjson, and the one thing the conversion will not do for you: types.
+#
+# In:  an inline csv with a header row -- station, date, celsius, active.
+# Out: ndjson in the run's scratch space, one JSON object per line, and beside it the same
+#      records with celsius as a number and active as a boolean.
+#
+# All three text formats convert.std knows say the same thing -- a sequence of records --
+# so a conversion is reading that sequence out of one spelling and writing it in another.
+# csv to ndjson is the one a line-oriented sink wants, because a consumer can read one line
+# at a time instead of holding the whole array.
+#
+# A convert step is a storage operation, the way storage.copy is: it reads one URI and
+# writes another, and no record passes through the step itself. So the csv is put at a URI
+# first and the ndjson is read back from one afterwards, which is the three hops below.
+#
+#   source    storage.write, the inline csv at a URI. A real pipeline writes the body an
+#             http.request answered with here instead, and nothing else in the document
+#             moves.
+#   as_ndjson convert.std csv -> ndjson. Output: source, target, bytes_written.
+#   lines     storage.read of the target. Nothing recognises the .ndjson extension, so the
+#             step says what the object is; application/x-ndjson is text rather than the
+#             JSON family, so it comes back as `text`.
+#
+# Every cell read out of a csv is a string. "4.5" comes back as the three characters, not as
+# a number, because a csv carries no types and a codec that guessed would be wrong on the
+# column where 007 is a code and 1-2 is a range. So the typing step is a jq program, in the
+# document that knows what the columns mean, and it is the second half of every csv read.
+#
+# ndjson is one JSON value per line, which is why the typing program splits on newlines,
+# drops the empty last one, and fromjson each.
+#
+# To change it: -p empty_is_null=true reads an empty cell as null rather than as the empty
+# string, which is the right rule for a numeric column and the wrong one for a text column.
+#
+#   dg run --local examples/recipes/csv-to-ndjson.yaml
+#   dg run --local examples/recipes/csv-to-ndjson.yaml -p empty_is_null=true
+
+format: dirigent/v1
+kind: pipeline
+code: csv-to-ndjson
+name: CSV to ndjson, with the typing that follows
+description: Re-spell a csv as one JSON object per line, then give the string cells the types the columns actually have.
+
+tags: [recipes, storage, transform]
+
+requires:
+  blocks:
+    - storage.write
+    - convert.std
+    - storage.read
+    - transform.jq
+
+params:
+  type: object
+  properties:
+    empty_is_null:
+      type: boolean
+      default: false
+      description: Read an empty cell as null rather than as an empty string.
+
+steps:
+  source:
+    block: storage.write
+    config:
+      target: ${run.scratch}/readings.csv
+      text: |
+        station,date,celsius,active
+        st-1,2026-01-01,4.5,true
+        st-2,2026-01-01,-1,false
+        st-3,2026-01-01,,true
+      content_type: text/csv
+
+  as_ndjson:
+    block: convert.std
+    depends_on: [source]
+    config:
+      source: ${steps.source.output.uri}
+      target: ${run.scratch}/readings.ndjson
+      from: csv
+      to: ndjson
+
+  lines:
+    block: storage.read
+    depends_on: [as_ndjson]
+    config:
+      source: ${steps.as_ndjson.output.target}
+      content_type: application/x-ndjson
+
+  typed:
+    block: transform.jq
+    depends_on: [lines]
+    config:
+      input:
+        lines: ${steps.lines.output.text}
+        empty_is_null: ${params.empty_is_null}
+      program: |
+        . as {$lines, $empty_is_null}
+        | [$lines
+           | split("\n")[]
+           | select(length > 0)
+           | fromjson
+           | {station,
+              date,
+              celsius: (if .celsius == ""
+                        then (if $empty_is_null then null else 0 end)
+                        else (.celsius | tonumber) end),
+              active: (.active == "true")}]

dirigent_examples/shelves/recipes/etl-csv-clean-validate-parquet.yaml

@@ -0,0 +1,207 @@
+# A whole small ETL: a messy csv in, a checked parquet file out.
+#
+# In:  an inline csv with padded strings, mixed case, a typed-as-text numeric column, an
+#      empty cell, and a duplicate row.
+# Out: a parquet file in the run's scratch space, written only from rows that passed a
+#      schema gate, plus a report of what was dropped and why.
+#
+# The order of the nine steps is the recipe, and every one of them is a different kind of
+# work that wants its own step:
+#
+#   source     storage.write, the inline csv at a URI. A conversion reads one URI and writes
+#              another, the way storage.copy does, so the csv it reads has to be an object
+#              first. A real pipeline writes a downloaded body here instead.
+#   parse      convert.std csv -> json. Nothing passes through the step: bytes at one URI,
+#              bytes at another.
+#   rows       storage.read of the parsed object, which is where the records enter the run.
+#              Every cell arrives as a string, because a csv has no types and a codec that
+#              guessed would be wrong on the column where 007 is a code.
+#   clean      the string work: trim, case-fold, and turn the text columns that are really
+#              numbers into numbers. This is where a csv's missing type information is
+#              supplied, by the document that knows what the columns mean.
+#   sort       the rows that cannot be loaded are separated from the ones that can, with a
+#              reason kept per row. Dropping them silently is what makes a load report "412
+#              rows" when 460 arrived.
+#   gate       validate.schema over the accepted rows. The clean step and the schema are two
+#              statements of the same expectation, and the gate is what makes them agree:
+#              anything the cleaner let through that the schema refuses fails the run here,
+#              at the boundary, rather than inside a parquet file somebody reads next month.
+#   checked    storage.write of what the gate passed, because the writer below takes a URI
+#              and not a value. Nothing but the gate's own output is written here.
+#   write      convert.arrow json -> parquet. Types survive here, which is the reason for
+#              doing all of the above before writing rather than after.
+#   report     the run's own account of itself: counts, the rejected keys, and the file.
+#
+# Everything downstream of the gate reads ${steps.gate.output.value}, which is the input
+# unchanged. That reference is the document's proof that no unchecked row reached the file.
+#
+# To change it: -p min_celsius=-50 loosens the range the schema allows; -p strict=true makes
+# the sorter refuse a duplicate key instead of keeping the first, which fails the run rather
+# than quietly preferring one row over another.
+#
+# The schema is carried in the document, so no server stores this one: it runs with
+# `dg run --local`, and on an instance the same shape is created once with `dg schema create`.
+#
+#   dg run --local examples/recipes/etl-csv-clean-validate-parquet.yaml
+#   dg run --local examples/recipes/etl-csv-clean-validate-parquet.yaml -p strict=true   # fails, on purpose
+
+format: dirigent/v1
+kind: pipeline
+code: etl-csv-clean-validate-parquet
+name: CSV to checked parquet
+description: Parse a messy csv, clean and type it, sort out the unloadable rows, hold the rest to a schema, and write parquet.
+
+tags: [recipes, storage, transform, validate, parquet]
+
+requires:
+  blocks:
+    - storage.write
+    - storage.read
+    - convert.std
+    - convert.arrow
+    - transform.jq
+    - validate.schema
+
+schemas:
+  recipe-reading-rows:
+    type: array
+    minItems: 1
+    items:
+      type: object
+      required: [station, day, celsius, active]
+      additionalProperties: false
+      properties:
+        station: { type: string, pattern: "^st-[0-9]+$" }
+        day: { type: string, format: date }
+        celsius: { type: number, minimum: -90, maximum: 60 }
+        active: { type: boolean }
+
+params:
+  type: object
+  properties:
+    day:
+      type: string
+      format: date
+      default: "2026-01-01"
+      description: The day the file is named for.
+    strict:
+      type: boolean
+      default: false
+      description: Fail on a duplicate station instead of keeping the first row for it.
+
+steps:
+  source:
+    block: storage.write
+    config:
+      target: ${run.scratch}/readings-${params.day}.csv
+      # Inline so the recipe needs no network. In a real pipeline the body an http.request
+      # answered with is written here, and nothing below moves.
+      text: |
+        station,day,celsius,active
+          ST-1 ,2026-01-01,4.5,TRUE
+        st-2,2026-01-01,-1,false
+        st-3,2026-01-01,,true
+        st-1,2026-01-01,4.5,true
+        st-4,2026-01-01,not-a-number,true
+      content_type: text/csv
+
+  parse:
+    block: convert.std
+    depends_on: [source]
+    config:
+      source: ${steps.source.output.uri}
+      target: ${run.scratch}/readings-${params.day}.json
+      from: csv
+      to: json
+
+  rows:
+    block: storage.read
+    depends_on: [parse]
+    config:
+      source: ${steps.parse.output.target}
+      max_size: 8mb
+
+  clean:
+    block: transform.jq
+    depends_on: [rows]
+    config:
+      input:
+        rows: ${steps.rows.output.value}
+        day: ${params.day}
+      program: |
+        def trim: sub("^\\s+"; "") | sub("\\s+$"; "");
+        . as {$rows, $day}
+        | [$rows[]
+           | {station: (.station | trim | ascii_downcase),
+              day: (if .day == "" then $day else .day end),
+              # An empty cell is not a measurement of zero, so it stays null and is sorted
+              # out below rather than invented here.
+              celsius: (if .celsius == "" then null
+                        else (try (.celsius | tonumber) catch null) end),
+              # A csv boolean is a word, and which word depends on who exported it.
+              active: ((.active | ascii_downcase) == "true")}]
+
+  sort:
+    block: transform.jq
+    depends_on: [clean]
+    config:
+      input:
+        rows: ${steps.clean.output.value}
+        strict: ${params.strict}
+      program: |
+        . as {$rows, $strict}
+        | [$rows | group_by(.station)[] | {station: .[0].station, count: length}]
+          as $by_station
+        | ([$by_station[] | select(.count > 1) | .station]) as $duplicated
+        | if $strict and ($duplicated | length) > 0
+          then error("duplicate stations: \($duplicated | join(", "))")
+          else . end
+        | {accepted: [$rows
+             | group_by(.station)[]
+             # First row wins for a duplicated key, which is a decision and not an accident.
+             | .[0]
+             | select(.celsius != null)],
+           rejected: [$rows[]
+             | select(.celsius == null)
+             | {station, reason: "celsius is absent or not a number"}],
+           duplicated: $duplicated}
+
+  gate:
+    block: validate.schema
+    depends_on: [sort]
+    config:
+      input: ${steps.sort.output.value.accepted}
+      schema: recipe-reading-rows
+
+  checked:
+    block: storage.write
+    depends_on: [gate]
+    config:
+      target: ${run.scratch}/checked-${params.day}.json
+      # From the gate, never from the sorter.
+      value: ${steps.gate.output.value}
+
+  write:
+    block: convert.arrow
+    depends_on: [checked]
+    config:
+      source: ${steps.checked.output.uri}
+      target: ${run.scratch}/readings-${params.day}.parquet
+      from: json
+      to: parquet
+
+  report:
+    block: transform.jq
+    depends_on: [sort, gate, write]
+    config:
+      input:
+        sorted: ${steps.sort.output.value}
+        checked: ${steps.gate.output.value}
+        uri: ${steps.write.output.target}
+        bytes: ${steps.write.output.bytes_written}
+      program: |
+        {file: .uri, file_bytes: .bytes,
+         written: (.checked | length),
+         rejected: (.sorted.rejected | length),
+         rejected_rows: .sorted.rejected,
+         duplicated_stations: .sorted.duplicated}