mq-bridge-basic 0.4.18__tar.gz → 0.4.19__tar.gz

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/CHANGELOG.md

@@ -2,6 +2,143 @@
 
 All notable changes to `mq-bridge`. Newest first.
 
+## 0.4.19
+
+CSV in more shapes: the reader takes other separators and quoting, and a CSV sink no longer
+mixes JSON into its cells. The second is a behaviour change, listed below.
+
+### Added
+
+- **CSV dialects.** `format: csv` took only comma-separated, `"`-quoted files with a header. A
+  `csv` block on the `file` endpoint and on an `object_store` source now sets `separator` (one
+  character, `tab`, `space`, hex, or `auto`), `quote` (a character or `none`), `header` and
+  `columns`. `separator: auto` is for sources and takes whichever of `,` `;` tab `|` occurs
+  most often outside quotes in the first record. That covers Excel's semicolon exports, TSV
+  from `mongoexport` and `psql`, and headerless dumps; the book's file page lists the setting
+  per tool. On the command line it is one parameter, `csv={"separator":"auto"}`.
+  `FileConfig` and `ObjectStoreConfig` gain the field `csv`, and `models` the types
+  `CsvConfig` and `CsvNested`; code that builds either struct without `..Default::default()`
+  needs the new field.
+- **CSV to object storage.** An `object_store` sink rejected `format: csv`. It now writes each
+  batch as a CSV file with its own header, taken from that batch's first row, in the dialect
+  the `csv` block sets. Set `csv.columns` when every object must have the same columns.
+- **A warning for short CSV rows.** A row with fewer fields than the header was read with the
+  missing ones empty and nothing said, which is what a wrong `separator` looks like. It is now
+  warned about once per process, as a row with too many fields already was.
+
+- **`aggregate`: `on_error`, time-based averages, late-message detection, `read_only`.**
+  - `on_error: drop | fail | skip` says what happens to a message that cannot be folded.
+    `drop` is the default and what happened before: an input logs, acks and drops it, so an
+    HTTP caller gets `202`. `fail` nacks it instead (HTTP `500`). `skip` leaves out only the
+    entries that cannot be computed, runs the others, passes the message on and names the
+    skipped entries in the metadata `mqb.aggregate.skipped`. Before, one missing field in one
+    entry discarded the message for all entries.
+  - `ema`, `ema_stddev` and `ema_variance` take a half-life in place of `alpha`
+    (`ema(reading, 5m)`): a value counts half after that time, however many messages came.
+    The new `time` names the payload field with the event time (epoch seconds or
+    milliseconds, or RFC 3339); without it the clock is used.
+  - With `time`, a message older than the newest one its state has seen is folded without
+    decay and marked with the metadata `mqb.aggregate.late`. It is detected, not reordered.
+  - `read_only: true` reads the states from the `store` and never changes them, for a dry run
+    or a second route that only reads.
+  - The error for a missing field now says `is missing or not a number`.
+  - `AggregateMiddleware` gains the fields `time`, `on_error` and `read_only`, and `models`
+    the enum `AggregateOnError`; code that builds the struct without `..Default::default()`
+    needs the new fields.
+- **Descriptive names for the file formats.** `format: raw` writes only the payload, which
+  for JSON payloads is plain JSON lines, while `format: json` writes the whole message
+  envelope; the names suggested the opposite. `payload`, `envelope_json`, `envelope_text`
+  and `envelope` are now accepted for `raw`, `json`, `text` and `normal`, on `file`,
+  `object_store` and `mongodb`. The old names keep working and are what is written back.
+- **`mqb copy`: pipes, `--limit`, progress, compression by file name.**
+  - `-` is stdin as the source and stdout as the target (`-?format=csv` takes parameters). A
+    `file` source on a pipe, FIFO or terminal is read once without seeking, and its end ends
+    the copy, with or without `--drain`. Before, `file:///dev/stdin` failed with `Illegal seek`.
+    With stdout as the target the `copied …` summary goes to stderr.
+  - `--limit N` stops after `N` rows reached the destination; with `--filter`, `N` matching rows.
+  - On a terminal, a running row count is shown on stderr every two seconds.
+  - A `file://` path ending in `.gz`, `.zst` or `.lz4` sets `compression`; `compression=none`
+    overrides. A sink named `out.jsonl.gz` used to be written uncompressed.
+  - A copy without `--drain` now exits when its route ends by itself, on a permanent error or
+    the end of a stream. It used to wait for Ctrl-C.
+- **SQL sink: `columns: auto` and `key`.** A SQL sink could write a record's fields into columns
+  only through a hand-written `insert_query`, percent-encoded on the command line, and an upsert
+  needed its own `ON CONFLICT` clause. With `columns: auto` each top-level JSON field goes into
+  the table column of the same name, and `key: id` turns the insert into an upsert, so
+  `mqb copy --from 'postgres://…?table=orders&cursor_column=id' --to 'postgres://…?table=orders&columns=auto&key=id'`
+  copies a table and can be rerun.
+  - The table must exist. A column a record does not name keeps its default (insert) or its
+    value (update). A field without a column is not written and is logged once; with
+    `extra_column: <name>` those fields go into that column as one JSON object.
+  - PostgreSQL casts each value to the column type, so `numeric`, `timestamptz`, `uuid`,
+    `jsonb`, enum and array columns take the text a SQL or CSV source delivers.
+  - `bulk_copy` works with `columns: auto`, not with `key`. `columns` cannot be combined with
+    `insert_query` or `auto_create_table`.
+  - `SqlxConfig` gains the fields `columns`, `key` and `extra_column`, and `models` the type `SqlColumns`.
+- **SQL source: `timestamps: rfc3339`.** A Postgres `timestamptz` column is read in Postgres's
+  text form, `2026-10-04 09:15:23.923277+00`, which few other systems parse. With
+  `timestamps: rfc3339` a `cursor_column` source renders it in UTC as
+  `2026-10-04T09:15:23.923277Z`; a `timestamp` without zone gets the `T` and no offset. The
+  default is unchanged. `SqlxConfig` gains the field `timestamps` and `models` the type
+  `SqlTimestamps`.
+
+### Behaviour changes
+
+- **`aggregate`: states in memory only are opt-in.** An `aggregate` without
+  a `store` kept its states in the process, lost on restart and wrong with two instances, and
+  that was the default. It now needs `consistency: single_writer` next to it; with neither a
+  `store` nor that, the route fails to start with an error naming both. Add
+  `consistency: single_writer` to a config from 0.4.17 or 0.4.18 that has no `store` to keep
+  what it did.
+- **A CSV sink flattens nested objects.** A nested object used to be written as JSON text in
+  one cell, so the output of `aggregate` (`{"stats":{"n":3,"avg":2.5}}`) came out as a mix of
+  CSV and JSON. It now becomes one column per leaf, named by its path: `stats.n`, `stats.avg`.
+  **A new file gets different columns than before; a file that already has a header keeps
+  it.** Set `csv.nested: json` for the previous output. Arrays are still written as JSON text.
+- **A drain does not wait for an endpoint.** With `exit_on_empty` (`mqb copy --drain`) a route
+  that failed reconnected up to ten times, 5 s apart, before it gave up: about 50 s against a
+  closed port, a WebSocket path that answers 404, a NATS subject outside every stream or a gRPC
+  topic nobody serves. It now ends on the first failure when nothing has been delivered yet,
+  with the endpoint's error. A drain that has delivered something keeps the ten reconnects, so
+  a short outage in the middle of a long copy does not end it. Routes without `exit_on_empty`
+  reconnect without limit, as before. Add a `retry` middleware to a drain that should wait for
+  its sink.
+- **A CSV file that ends inside a quoted field fails a drain.** The reader took everything
+  after the open quote as one field and reported success. A drain now writes the records before
+  it and fails, naming the byte offset of the open record; a tailing route waits for the writer
+  to finish the record.
+- **Blank lines in a line-based file are skipped.** `raw`, `json`, `text` and `normal` sources
+  emitted an empty line as a message with an empty payload, which a SQL or Parquet sink then
+  rejected. The CSV reader already skipped them. Row counts change for files with blank lines.
+- **A `file` route whose source and sink are the same path is refused.** It doubled the file
+  under a drain and would grow it without end otherwise.
+- **SQLite `has no column named` is not retried.** A sink writing to a table without the
+  expected columns retried for ever; it is now a non-retryable error like the same case on
+  Postgres and MySQL.
+- **An `object_store` source on a local path that does not exist fails a drain.** It reported
+  `copied 0 rows`. An empty directory or cloud prefix still does. A drain of a local path also
+  ends after one listing instead of waiting out the idle interval.
+- **NATS: a missing `subject` or `stream` stops the route.** It was retried as a connection
+  error until the start timeout.
+- **Clearer errors and quieter logs.** A queue read on a plain SQL table names
+  `cursor_column`; Postgres errors no longer end in `at line N` (a line of the server's source);
+  a closed database port reports `Connection refused` at once; a `filter` that does not compile
+  lists the operators, and one with a single `=` says so; a gRPC server names the topic it has
+  no consumer for; the drop summary counts messages per cause; an unknown `aggregate` field
+  lists the built-ins; `csv` settings without `format: csv` and an `insert_query` token that
+  binds NULL each warn once. The WebSocket "falling back to routed mode" line is now DEBUG and
+  the MQTT note on restart loss INFO.
+
+### Fixed
+
+- **`mqb copy --wait` reports a failed route at once.** A route that failed before it delivered
+  anything was retried until the wait ran out, and a later empty attempt could then report
+  success. `--wait` now retries only a drain that ended cleanly with nothing to copy.
+- **`aggregate` with a shared `store` no longer keeps the state of a message it rejects.** A
+  message whose result had no place in the payload (`into: a.b` where `a` is not an object) was
+  counted in the store and then failed, so under `on_error: fail` every redelivery counted it
+  again. It now fails before any state changes, as it already did without a shared store.
+
 ## 0.4.18
 
 A new generic output for search engines, plus documentation and packaging work. This release

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/Cargo.lock

@@ -3689,7 +3689,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.4.18"
+version = "0.4.19"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3792,7 +3792,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.4.18"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3806,7 +3806,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.4.18"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3861,7 +3861,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.4.18"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "async-trait",

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/Cargo.toml

@@ -40,7 +40,7 @@ dbg_macro = "warn"
 exit = "warn"
 
 [workspace.package]
-version = "0.4.18"
+version = "0.4.19"
 
 [workspace.dependencies]
 anyhow = "1.0"

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mq-bridge-basic
-Version: 0.4.18
+Version: 0.4.19
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Database

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/README.md

@@ -93,7 +93,7 @@ Every row works from the Rust crate, the Python and Node.js packages, and the ze
 | **MongoDB** as source or sink | `mongodb` endpoint · `mongodb` | [MongoDB](https://marcomq.github.io/mq-bridge/connectors/mongodb.html) |
 | **ClickHouse** bulk insert and resumable reads | `clickhouse` endpoint · `clickhouse` | [ClickHouse](https://marcomq.github.io/mq-bridge/connectors/clickhouse.html) |
 | **Search engines** and other bulk document APIs (Meilisearch, Typesense, Elasticsearch, Qdrant, PostgREST / Supabase) | `http_bulk` endpoint · `http-bulk` | [HTTP bulk](https://marcomq.github.io/mq-bridge/connectors/http-bulk.html) |
-| **Object storage**: S3 (and S3-compatible, e.g. R2), GCS, Azure Blob, or a local directory, as **Parquet** or **JSONL**; **CSV** as input | `object_store` endpoint with `format: parquet \| raw \| json \| csv` · `object-store`, `parquet` | [Object storage](https://marcomq.github.io/mq-bridge/connectors/object-store.html) |
+| **Object storage**: S3 (and S3-compatible, e.g. R2), GCS, Azure Blob, or a local directory, as **Parquet**, **JSONL** or **CSV** | `object_store` endpoint with `format: parquet \| raw \| json \| csv` · `object-store`, `parquet` | [Object storage](https://marcomq.github.io/mq-bridge/connectors/object-store.html) |
 | **Files**: CSV and JSONL, read and write | `file` endpoint · built in | [File](https://marcomq.github.io/mq-bridge/connectors/file.html) |
 | **Analytics / warehouses**: Snowflake, BigQuery, Databricks / Spark, Athena / Trino, DuckDB **via Parquet on S3/GCS/Azure**. These are not native connectors: mq-bridge writes the Parquet files, the warehouse loads or queries them | `object_store` with `format: parquet` · `parquet` | [Snowflake](https://marcomq.github.io/mq-bridge/cookbook/snowflake.html) · [BigQuery](https://marcomq.github.io/mq-bridge/cookbook/bigquery.html) · [Databricks / Spark](https://marcomq.github.io/mq-bridge/cookbook/databricks.html) · [Athena / Trino](https://marcomq.github.io/mq-bridge/cookbook/athena.html) · [DuckDB](https://marcomq.github.io/mq-bridge/cookbook/duckdb.html) |
 | **HTTP, gRPC, WebSocket** as server or client, including request/reply | `http`, `grpc`, `websocket` endpoints · `http`, `grpc`, `websocket` | [HTTP](https://marcomq.github.io/mq-bridge/connectors/http.html) · [gRPC](https://marcomq.github.io/mq-bridge/connectors/grpc.html) · [Request / reply](https://marcomq.github.io/mq-bridge/tutorials/request-reply.html) |
@@ -116,6 +116,8 @@ Throughput is tracked continuously on the public [benchmark dashboard](https://m
 | CSV → JSONL, 1M mixed-type rows (~116 MiB) | **2,824,858 rows/s**, ~28 MiB RAM | Meltano (`tap-csv` → `target-jsonl`): ~19,500 rows/s, ~444 MiB RAM — **~145x slower**<br>DuckDB, all cores: 2,036,659 rows/s — mq-bridge **~1.4x faster**, ~17x less memory |
 | Kafka → file, 1M rows, no transform | **~65% faster** than Sea Streamer | Sea Streamer, both on mimalloc (~80% faster vs. its default-allocator build) |
 
+The CSV row reuses an existing, independently published 1M-row CSV → JSONL workload, on a byte-identical fixture: 7 mixed-type columns, seed 42, 121,981,421 bytes, SHA-256 `a84894e0…0c45b221` ([full definition](apps/mq-bridge-app/benches/etl/README.md#6--csv--jsonl-vs-meltano)). Whole-process wall-clock, no transformation, Meltano run on the same machine and file.
+
 CSV figures: mq-bridge 0.4.12. DuckDB is a throughput ceiling for the conversion itself, not an ETL tool. Methodology and reporting rules are in [`benches/ETL_BENCHMARKS.md`](benches/ETL_BENCHMARKS.md); the raw numbers, baselines and reproducible helpers are in the [ETL benchmark harness](apps/mq-bridge-app/benches/etl/README.md).
 
 
@@ -170,17 +172,12 @@ tuning use the same settings in both.
 
 ## Status
 
-`mq-bridge` is young (created in 2025), but its reliability behavior is exercised by an automated integration and performance suite across **every** supported endpoint, in each of the queue and subscriber modes that endpoint supports:
+`mq-bridge` was created in 2025. Its reliability behavior is exercised by an automated integration and performance suite across supported endpoints, in each of the queue and subscriber modes that endpoint supports:
 
 *   All endpoints showed **no data loss during in-flight broker restarts**; MQTT publish confirmation was hardened until a chaos test drove in-flight loss to **zero**.
 *   Postgres CDC has a **restart-safety test**: an un-acked, in-flight batch is redelivered after a database restart with no loss and no gap.
 
-Known rough edges:
-
-- old or very new broker-server versions, or unusual broker settings
-- subscribe/event and response patterns where the backend has no native equivalent (emulated); MongoDB request/reply is only covered by automated tests so far
-- NATS without JetStream (integration tests run with JetStream only)
-- **TLS**: one `TlsConfig` shape is shared by all transports and covered by automated tests, but per-backend certificate matrices are still being expanded, so verify non-trivial TLS setups yourself
+It is used as rust library in [armature](https://github.com/quinnjr/armature) and as python lib in [omniload](https://github.com/panodata/omniload). 
 
 ## Running Tests
 The project includes integration and performance tests. Most backend tests require Docker.

{mq_bridge_basic-0.4.18 → mq_bridge_basic-0.4.19}/docs/REFERENCE.md

@@ -700,7 +700,43 @@ stream. File compression/encryption supports only the default `consume` mode. `c
 too: the header row is written into the first member, so the decoded stream is a normal CSV
 file.
 
-A `csv` sink takes its columns from the first row it writes, in that row's field order.
+A `csv` sink takes its columns from the first row it writes, in that row's field order. An
+`object_store` sink writes each batch as a CSV file of its own, with its own header taken from
+that batch's first row; set `columns` when every object must have the same ones.
+
+`csv` on its own means comma-separated, `"`-quoted, with a header record. The `csv` block
+describes any other dialect, for the `file` and `object_store` endpoints:
+
+```yaml endpoint
+input:
+  file:
+    path: "export.csv"
+    format: csv
+    csv:
+      separator: ";"          # one character, `tab`, `space`, hex (`0x1f`), or `auto`
+      quote: "'"              # one character, or `none`; default `"`
+      header: false           # first record is data; a source then needs `columns`
+      columns: [id, name]     # replaces the header's names; a sink writes exactly these
+```
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| `separator` | `,` | Field separator. `auto` (sources only) picks whichever of `,` `;` tab `\|` occurs most often outside quotes in the first record, and `,` on a tie. |
+| `quote` | `"` | Quote character; a doubled one inside a quoted field is a literal. `none` reads every byte as data and makes a sink fail a value it would have to quote. |
+| `header` | `true` | Whether the first record names the columns. A sink with `false` writes rows only. |
+| `columns` | — | Source: the keys to use, instead of the header's. Sink: the columns to write, in this order; other keys are dropped. |
+| `nested` | `flatten` | Sink only. `flatten` writes a nested object as one `parent.child` column per leaf; `json` writes its JSON text into one cell. Arrays are always JSON text. |
+
+`delimiter` stays the *record* separator and is independent of all this; it only must not
+contain the separator or the quote character. Common exports: Excel "CSV UTF-8" in a
+semicolon locale is `separator: ";"` (the byte-order mark and CRLF are handled already),
+`mongoexport --type=tsv` and `psql -A -F $'\t'` are `separator: tab`, and `psql --csv` and
+`COPY … WITH (FORMAT csv)` are the default. PostgreSQL's `COPY` *text* format (backslash
+escapes, `\N`) and UTF-16 files are not CSV dialects and are not read.
+
+A row with more fields than the header drops the extras, and one with fewer reads the missing
+ones as empty. Each is warned about once per process; a short row is the usual sign of a wrong
+`separator`.
 
 A file **source** must declare the same `compression`/`encryption` the data was written with.
 A mismatch (wrong key, wrong codec, or a missing field) is a permanent decode failure: the
@@ -1151,12 +1187,16 @@ to keep a counter: the arithmetic runs here, so no request is made per message.
 | `output` | expression | the state itself; shapes what is written to the message |
 | `emit` | `updated` \| `previous` | `updated` |
 | `entries` | list of `{key, expression, fields, into, output, emit}` | `[]`; further aggregates |
-| `store` | URL | – (states live in memory); `postgres://…/db[/table]`, `sqlite://…`, `mongodb://host/db[/collection]` |
-| `consistency` | `shared` \| `single_writer` | `shared`; only used with `store` |
+| `store` | URL | –; `postgres://…/db[/table]`, `sqlite://…`, `mongodb://host/db[/collection]`. Without it `consistency: single_writer` is required |
+| `consistency` | `shared` \| `single_writer` | `shared`, which needs a `store`; `single_writer` without a `store` keeps the states in memory only |
 | `max_keys` | integer | `1000000`; states kept in memory per entry, `0` is unlimited |
+| `time` | dotted payload path | – (the clock); the message's event time, for `fields` entries |
+| `on_error` | `drop` \| `fail` \| `skip` | `drop`; what happens to a message that cannot be folded |
+| `read_only` | bool | `false`; reads the states of the `store` and never changes them |
 
 ```yaml middleware
 - aggregate:
+    consistency: single_writer
     key: "${payload:sensor_id}"
     into: sensor
     expression: "{ total: (state.total ?? 0) + reading, count: (state.count ?? 0) + 1 }"
@@ -1166,6 +1206,11 @@ to keep a counter: the arithmetic runs here, so no request is made per message.
 `{"sensor_id": 7, "reading": 20, "sensor": {"total": 20, "count": 1}}`; the next message for sensor 7
 continues from that state.
 
+**States in memory only are opt-in.** Without a `store` the states are lost on restart and
+not shared between instances, so that has to be asked for with `consistency: single_writer`,
+as above. An `aggregate` with neither a `store` nor `consistency: single_writer` is a startup
+error.
+
 **Expressions** are the ones [`filter`](#filter) and [`transform`](#transform) use. They read
 payload fields by name and metadata as `meta.<key>`, plus `state`: the value `expression`
 returned for the previous message with the same key. For a key seen for the first time
@@ -1184,6 +1229,7 @@ to forget it. Keep the weighted sum and the weight instead and divide when writi
 
 ```yaml middleware
 - aggregate:
+    consistency: single_writer
     key: "${payload:sensor_id}"
     into: sensor.avg_reading
     expression: "{ s: (state.s ?? 0) * 0.99 + reading, w: (state.w ?? 0) * 0.99 + 1 }"
@@ -1197,6 +1243,7 @@ the usual form still reads 216.
 
 ```yaml middleware
 - aggregate:
+    consistency: single_writer
     entries:
       - key: "${payload:sensor_id}"
         into: features.sensor
@@ -1212,6 +1259,7 @@ times faster, because no expression runs and the payload is not parsed into a tr
 
 ```yaml middleware
 - aggregate:
+    consistency: single_writer
     key: "${payload:sensor_id}"
     into: sensor
     fields:
@@ -1228,14 +1276,84 @@ times faster, because no expression runs and the payload is not parsed into a tr
 | `mean(path)` | arithmetic mean |
 | `stddev(path)`, `variance(path)` | sample standard deviation and variance (divided by n − 1) of all values; `null` for the first message |
 | `ema(path, alpha)` | moving average weighting each new value with `alpha` (0 < alpha ≤ 1), without a start bias: the first message reads its own value |
-| `ema_stddev(path, alpha)`, `ema_variance(path, alpha)` | the same over values weighted like `ema`, so old values fade; `null` for the first message |
+| `ema(path, half_life)` | the same, weighted by time: a value counts half after `half_life`, e.g. `ema(reading, 5m)`. Units: `ms`, `s`, `m`, `h`, `d` |
+| `ema_stddev(path, alpha)`, `ema_variance(path, alpha)` | the same over values weighted like `ema`, so old values fade; `null` for the first message. Both take a half-life in place of `alpha` too |
 
 `path` is a dotted payload path such as `reading` or `device.reading`. Its value must be a
-number or a numeric string (what a CSV source delivers); a message without one fails like a
-message with a failing expression. The result is an object with the field names in
+number or a numeric string (what a CSV source delivers); a message without one cannot be
+folded, see [`on_error`](#aggregate-on-error). The result is an object with the field names in
 alphabetical order, e.g. `{"avg": 20.0, "high": 20.0, "n": 1, "total": 20.0}`. An entry takes
 either `expression` or `fields`; entries of both kinds can be mixed. `fields` takes no `output`.
 
+**Time.** `ema(path, alpha)` counts messages: ten messages in one second fade a value as
+much as ten messages in one day. With a half-life the weight follows the time between two
+messages of a key instead. `time` names the payload field holding the message's time: epoch
+seconds, epoch milliseconds (any number above 1e11), or an RFC 3339 text such as
+`2026-10-03T12:00:00Z`; a text without a zone is read as UTC. Without `time` the clock of
+this process is used, which makes a replay compute different values.
+
+```yaml middleware
+- aggregate:
+    consistency: single_writer
+    key: "${payload:sensor_id}"
+    into: sensor
+    time: measured_at
+    fields:
+      n: count
+      avg: ema(reading, 5m)
+      spread: ema_stddev(reading, 1h)
+```
+
+**Late messages are detected, not corrected.** With `time` set, every `fields` state
+remembers the newest time it has seen. A message with an older time is folded as if it had
+arrived at that newest time (nothing decays for it), and it carries the metadata
+`mqb.aggregate.late` naming the entries concerned by their `into`, comma-separated. A
+[`filter`](#filter) or a `switch` can act on it as `meta["mqb.aggregate.late"]`. The first
+occurrence is logged as a warning. `time` adds one slot to each `fields` state; states stored
+before it was set are read and extended. Expression entries do not use `time`.
+
+<a id="aggregate-on-error"></a>
+**A message that cannot be folded.** Its payload is not a JSON object, a `key` has no value,
+a field read by `fields` or `time` is missing or not a number, or an expression fails. One
+entry that cannot be computed is enough. `on_error` decides what follows; no state changes
+for an entry that was not computed.
+
+| `on_error` | On an input | On an output |
+|---|---|---|
+| `drop` (default) | logged, acked and dropped. **The source sees a success**: an HTTP caller gets `202` | only that message fails, as non-retryable |
+| `fail` | logged and nacked, so the source sees the failure: an HTTP caller gets `500`. A source that redelivers nacked messages (a broker) delivers it again | as `drop` |
+| `skip` | the entries that cannot be computed are left out, the others run, and the message goes on. A message no entry can fold goes on unchanged | the same |
+
+A message that left entries out under `skip` carries the metadata `mqb.aggregate.skipped`
+with their `into` paths, comma-separated, and the first occurrence is logged as a warning;
+the reason for each is logged at debug level.
+
+```yaml middleware
+- aggregate:
+    consistency: single_writer
+    on_error: skip
+    entries:
+      - { key: "${payload:sensor_id}", into: sensor, fields: { avg: "ema(reading, 0.01)" } }
+      - { key: "${payload:site_id}", into: site, fields: { high: max(reading) } }
+```
+
+**Reading without writing.** `read_only: true` computes and writes into the message as
+usual but stores nothing, so a dry run or a second route can use the states another route
+maintains. `emit: updated` shows what the state would become with this message,
+`emit: previous` the stored state as it is. It needs a `store`; states are loaded per batch
+whatever `consistency` says, and each message of a batch sees the stored state, not the
+messages before it.
+
+```yaml middleware
+- aggregate:
+    store: "postgres://localhost/telemetry/sensor_states"
+    read_only: true
+    emit: previous
+    key: "${payload:sensor_id}"
+    into: sensor
+    fields: { n: count, avg: "ema(reading, 0.01)" }
+```
+
 **Keeping the states in a store.** With `store` the states survive a restart. The table or
 collection defaults to `mqb_aggregate_<route>` and is created on start; PostgreSQL, SQLite
 and MongoDB are supported.
@@ -1283,7 +1401,7 @@ and MongoDB are supported.
 **What to expect.**
 
 - **Without `store`, state lives in memory only.** It is lost on restart and it is not shared
-  between instances.
+  between instances, which is why it needs `consistency: single_writer`.
 - **Memory is bounded by `max_keys`.** An entry keeps at most that many states in memory,
   one million unless set. When it is full, the half that was used least recently is dropped.
   Without a `store` those states are forgotten: their key starts again from an empty state,
@@ -1295,13 +1413,13 @@ and MongoDB are supported.
   first: on an input, list it *after* `aggregate` (see [Ordering](#ordering--read-this-before-combining-middleware)).
 - **Order matters for order-dependent states** such as a moving average. Within a batch
   messages are folded in order; with a route `concurrency` above 1 the order across batches
-  is not guaranteed.
-- **A message that cannot be folded fails alone** and changes no state: its payload is not a
-  JSON object, its `key` has no value, or an expression fails. On an input it is logged,
-  acked and dropped; on an output only that message fails, as non-retryable.
+  is not guaranteed. With `time`, a message older than its state is marked, not reordered.
+- **A message that cannot be folded fails alone** and changes no state. By default an input
+  drops it and still acks it, so an HTTP caller gets `202`; see
+  [`on_error`](#aggregate-on-error) for `fail` and `skip`.
 - **No windows and no expiry by time.** There is no "last hour" and no time to live: a state
   covers every message of its key, and a row in the store is never removed. `ema` and
-  `ema_stddev` stand in for a sliding window. For fixed windows put the window into the key,
+  `ema_stddev` stand in for a sliding window, by message count or with a half-life by time. For fixed windows put the window into the key,
   e.g. `key: "${payload:sensor_id}:${payload:hour}"`, and use a `store` with `shared`, which
   keeps no state in memory; the rows of past windows stay until you delete them.
 - **No exactly-once and no key ownership.** A stream processor commits state and input