mq-bridge-basic 0.4.17__tar.gz → 0.4.19__tar.gz

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.19}/.github/workflows/docs.yml

@@ -43,6 +43,10 @@ jobs:
       - name: Build book
         run: mdbook build apps/mq-bridge-app/dev/docs
 
+      # mdbook copies non-Markdown files from the book source; fail if that ever stops.
+      - name: Check llms.txt is published
+        run: test -s apps/mq-bridge-app/dev/docs/book/llms.txt
+
       # The benchmark dashboard lives on the gh-pages branch. Copy it into the
       # book so this workflow is the only thing that publishes to Pages.
       - name: Checkout benchmark data

mq_bridge_basic-0.4.17/CLAUDE.md → mq_bridge_basic-0.4.19/AGENTS.md

@@ -3,16 +3,16 @@
 > **Looking for what middleware or structural endpoints exist, and how to configure them?**
 > [REFERENCE.md](docs/REFERENCE.md) is the complete, authoritative list — every middleware
 > (`retry`, `dlq`, `transform`, `id`, `filter`, `deduplication`, `weak_join`, `buffer`, `limiter`, `delay`,
-> `cookie_jar`, `encryption`, `compression`, `pack`, `unpack`, `metrics`, `otel`, `lookup`, `aggregate`, `random_panic`, `custom`) and every
+> `cookie_jar`, `encryption`, `compression`, `pack`, `unpack`, `timeout`, `metrics`, `otel`, `lookup`, `aggregate`, `random_panic`, `custom`) and every
 > structural endpoint (`ref`,
-> `fanout`, `switch`, `request`, `response`, `reader`, `static`, `stream_buffer`, `null`,
+> `fanout`, `switch`, `sequence`, `request`, `response`, `reader`, `static`, `stream_buffer`, `null`,
 > `custom`), each with its fields, defaults, and a working YAML example. Do not infer these
 > from the enum definitions in `models.rs`; the reference records the behaviour and the
 > spelling traps too. Every snippet in it is parsed by `tests/reference_docs_test.rs`.
 
 ## Project Overview
 
-`mq-bridge` is an asynchronous message bridging library for Rust that connects different messaging systems, data stores, and protocols. It acts as a **programmable integration layer**, allowing for transformation, filtering, handling, events, and complex routing.
+`mq-bridge` is an asynchronous Rust library that moves messages and data between brokers, databases, object storage, files and HTTP. What it can do, and where each capability is documented, is in [llms.txt](llms.txt) and the README's "Capabilities at a glance" table; this file covers how to work on the engine. The app in `apps/mq-bridge-app` has its own [AGENTS.md](apps/mq-bridge-app/AGENTS.md).
 
 ## Core Architecture
 
@@ -41,7 +41,10 @@ src/
 ├── canonical_message.rs   # Unified message format
 ├── traits.rs             # Core traits (MessageConsumer, MessagePublisher, Handler)
 ├── models.rs             # Configuration models (Route, Endpoint, Middleware)
+├── models/               # models.rs submodules: builders, defaults, secrets, serde support
+├── config_file.rs        # Loading routes from YAML/JSON files
 ├── route.rs              # Route execution logic (sequential/concurrent)
+├── shutdown.rs           # Graceful shutdown signalling
 ├── endpoints/            # Endpoint implementations
 │   ├── mod.rs           # Factory functions for creating consumers/publishers
 │   ├── amqp.rs          # AMQP (RabbitMQ) consumer/publisher
@@ -49,12 +52,13 @@ src/
 │   ├── clickhouse.rs    # ClickHouse sink + cursor source
 │   ├── dir_spool/       # Crash-safe directory FIFO queue (payload file + JSON sidecar)
 │   ├── file/            # File-based endpoints
-│   ├── grpc.rs          # gRPC consumer/publisher
+│   ├── grpc/            # gRPC consumer/publisher
 │   ├── http/            # HTTP consumer/publisher (+ streaming)
+│   ├── http_bulk/       # Bulk JSON documents over HTTP (search engines): publisher, paged consumer, presets
 │   ├── ibm_mq.rs        # IBM MQ (client loaded at runtime via dlopen)
 │   ├── kafka.rs         # Kafka consumer/publisher
 │   ├── memory/          # In-memory channels + IPC transports
-│   ├── mongodb.rs       # MongoDB consumer/publisher/change streams
+│   ├── mongodb/         # MongoDB consumer/publisher/change streams
 │   ├── mqtt.rs          # MQTT consumer/publisher
 │   ├── nats.rs          # NATS consumer/publisher
 │   ├── object_store.rs  # Cloud object storage (S3 / GCS / Azure)
@@ -68,6 +72,7 @@ src/
 │   └── structural/      # Structural endpoints (no external system)
 │       ├── fanout.rs          # Broadcast to every listed endpoint
 │       ├── switch.rs          # Content-based routing
+│       ├── sequence.rs        # Drain several inputs one after another (backfill, then stream)
 │       ├── request.rs         # Request/reply call, forward the response
 │       ├── response.rs        # Reply to the origin of the current request
 │       ├── reader.rs          # Trigger a pull from a consumer
@@ -77,8 +82,10 @@ src/
 ├── middleware/           # Middleware implementations
 │   ├── aggregate/       # Per-key running state (sums, counters, moving averages)
 │   ├── buffer.rs        # Batch accumulation
+│   ├── compression.rs   # Per-message payload compression
 │   ├── cookie_jar.rs    # Cookie / metadata persistence across requests
-│   ├── deduplication.rs # Message deduplication (sled)
+│   ├── deduplication.rs # Message deduplication
+│   ├── deduplication/   # Dedup stores (sled, memory, SQL, MongoDB)
 │   ├── delay.rs         # Artificial delay
 │   ├── deferred_commit.rs # Hold commits for batches a middleware emptied
 │   ├── dlq.rs           # Dead-letter queue
@@ -89,15 +96,22 @@ src/
 │   ├── lookup.rs        # Enrich from another endpoint's response
 │   ├── metrics.rs       # Metrics collection
 │   ├── otel.rs          # OpenTelemetry spans + traceparent propagation
+│   ├── pack.rs          # pack / unpack: one transport message per batch
 │   ├── random_panic.rs  # Testing middleware
+│   ├── raw_json.rs      # Shared raw-JSON payload helpers
 │   ├── retry.rs         # Exponential backoff retry
+│   ├── timeout.rs       # Bound each send; fail as retryable
 │   ├── transform/       # Declarative JSON mapping + schema coercion
 │   └── weak_join.rs     # Correlation-keyed join
 ├── support/              # Cross-cutting helpers
 │   ├── compression.rs   # gzip / lz4 / zstd
 │   ├── connection_registry.rs # Shared connection reuse
 │   ├── crypto.rs        # AEAD core (used by encryption middleware + at-rest)
-│   └── interpolation.rs # `${namespace:selector}` templating
+│   ├── interpolation.rs # `${namespace:selector}` templating
+│   ├── parquet.rs       # Parquet encode/decode for object_store
+│   ├── source_ranges.rs # Covered-range recovery for replay-safe file/object sinks
+│   ├── pack.rs          # Batch envelope used by pack / unpack
+│   └── plugin_abi.rs    # C ABI shared by host and native plugins
 ├── command_handler.rs    # Command handler wrapper
 ├── event_handler.rs      # Event handler wrapper
 ├── event_store.rs        # In-memory event store
@@ -105,6 +119,7 @@ src/
 ├── publisher.rs          # Standalone publisher API
 ├── checkpoint.rs         # Durable cursor stores (file/s3/postgres/mongodb)
 ├── extensions.rs         # Custom endpoint/middleware factory registration
+├── plugin/               # Native plugin host, SDK and conformance suite
 ├── response.rs           # Ergonomic response helpers
 ├── test_utils.rs         # Shared test helpers
 ├── errors.rs             # Error types
@@ -142,20 +157,7 @@ Routes are defined via:
 - JSON files
 - Environment variables (prefix: `MQB__`, separator: `__`)
 
-Example:
-```yaml
-kafka_to_nats:
-  concurrency: 4
-  batch_size: 128
-  input:
-    kafka:
-      topic: "orders"
-      url: "localhost:9092"
-  output:
-    nats:
-      subject: "orders.processed"
-      url: "nats://localhost:4222"
-```
+The grammar and examples are in [CONFIGURATION.md](docs/CONFIGURATION.md).
 
 ### Common Patterns
 
@@ -188,20 +190,4 @@ kafka_to_nats:
 
 ### Dependencies
 
-- **tokio**: Async runtime
-- **serde**: Serialization/deserialization
-- **anyhow**: Error handling
-- **tracing**: Structured logging
-- **config**: Configuration management
-- **async-trait**: Async trait support
-- **uuid**: Unique identifiers (v4 and v7)
-
-Optional (feature-gated):
-- **rdkafka**: Kafka support
-- **async-nats**: NATS support
-- **lapin**: AMQP support
-- **rumqttc**: MQTT support
-- **mongodb**: MongoDB support
-- **actix/reqwest**: HTTP support
-- **sled**: Deduplication storage
-- **metrics**: Metrics collection
+Dependencies and the feature that gates each optional one are in `Cargo.toml`, with comments explaining the non-obvious choices. The crate has no default features.

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.19}/CHANGELOG.md

@@ -2,6 +2,315 @@
 
 All notable changes to `mq-bridge`. Newest first.
 
+## 0.4.19
+
+CSV in more shapes: the reader takes other separators and quoting, and a CSV sink no longer
+mixes JSON into its cells. The second is a behaviour change, listed below.
+
+### Added
+
+- **CSV dialects.** `format: csv` took only comma-separated, `"`-quoted files with a header. A
+  `csv` block on the `file` endpoint and on an `object_store` source now sets `separator` (one
+  character, `tab`, `space`, hex, or `auto`), `quote` (a character or `none`), `header` and
+  `columns`. `separator: auto` is for sources and takes whichever of `,` `;` tab `|` occurs
+  most often outside quotes in the first record. That covers Excel's semicolon exports, TSV
+  from `mongoexport` and `psql`, and headerless dumps; the book's file page lists the setting
+  per tool. On the command line it is one parameter, `csv={"separator":"auto"}`.
+  `FileConfig` and `ObjectStoreConfig` gain the field `csv`, and `models` the types
+  `CsvConfig` and `CsvNested`; code that builds either struct without `..Default::default()`
+  needs the new field.
+- **CSV to object storage.** An `object_store` sink rejected `format: csv`. It now writes each
+  batch as a CSV file with its own header, taken from that batch's first row, in the dialect
+  the `csv` block sets. Set `csv.columns` when every object must have the same columns.
+- **A warning for short CSV rows.** A row with fewer fields than the header was read with the
+  missing ones empty and nothing said, which is what a wrong `separator` looks like. It is now
+  warned about once per process, as a row with too many fields already was.
+
+- **`aggregate`: `on_error`, time-based averages, late-message detection, `read_only`.**
+  - `on_error: drop | fail | skip` says what happens to a message that cannot be folded.
+    `drop` is the default and what happened before: an input logs, acks and drops it, so an
+    HTTP caller gets `202`. `fail` nacks it instead (HTTP `500`). `skip` leaves out only the
+    entries that cannot be computed, runs the others, passes the message on and names the
+    skipped entries in the metadata `mqb.aggregate.skipped`. Before, one missing field in one
+    entry discarded the message for all entries.
+  - `ema`, `ema_stddev` and `ema_variance` take a half-life in place of `alpha`
+    (`ema(reading, 5m)`): a value counts half after that time, however many messages came.
+    The new `time` names the payload field with the event time (epoch seconds or
+    milliseconds, or RFC 3339); without it the clock is used.
+  - With `time`, a message older than the newest one its state has seen is folded without
+    decay and marked with the metadata `mqb.aggregate.late`. It is detected, not reordered.
+  - `read_only: true` reads the states from the `store` and never changes them, for a dry run
+    or a second route that only reads.
+  - The error for a missing field now says `is missing or not a number`.
+  - `AggregateMiddleware` gains the fields `time`, `on_error` and `read_only`, and `models`
+    the enum `AggregateOnError`; code that builds the struct without `..Default::default()`
+    needs the new fields.
+- **Descriptive names for the file formats.** `format: raw` writes only the payload, which
+  for JSON payloads is plain JSON lines, while `format: json` writes the whole message
+  envelope; the names suggested the opposite. `payload`, `envelope_json`, `envelope_text`
+  and `envelope` are now accepted for `raw`, `json`, `text` and `normal`, on `file`,
+  `object_store` and `mongodb`. The old names keep working and are what is written back.
+- **`mqb copy`: pipes, `--limit`, progress, compression by file name.**
+  - `-` is stdin as the source and stdout as the target (`-?format=csv` takes parameters). A
+    `file` source on a pipe, FIFO or terminal is read once without seeking, and its end ends
+    the copy, with or without `--drain`. Before, `file:///dev/stdin` failed with `Illegal seek`.
+    With stdout as the target the `copied …` summary goes to stderr.
+  - `--limit N` stops after `N` rows reached the destination; with `--filter`, `N` matching rows.
+  - On a terminal, a running row count is shown on stderr every two seconds.
+  - A `file://` path ending in `.gz`, `.zst` or `.lz4` sets `compression`; `compression=none`
+    overrides. A sink named `out.jsonl.gz` used to be written uncompressed.
+  - A copy without `--drain` now exits when its route ends by itself, on a permanent error or
+    the end of a stream. It used to wait for Ctrl-C.
+- **SQL sink: `columns: auto` and `key`.** A SQL sink could write a record's fields into columns
+  only through a hand-written `insert_query`, percent-encoded on the command line, and an upsert
+  needed its own `ON CONFLICT` clause. With `columns: auto` each top-level JSON field goes into
+  the table column of the same name, and `key: id` turns the insert into an upsert, so
+  `mqb copy --from 'postgres://…?table=orders&cursor_column=id' --to 'postgres://…?table=orders&columns=auto&key=id'`
+  copies a table and can be rerun.
+  - The table must exist. A column a record does not name keeps its default (insert) or its
+    value (update). A field without a column is not written and is logged once; with
+    `extra_column: <name>` those fields go into that column as one JSON object.
+  - PostgreSQL casts each value to the column type, so `numeric`, `timestamptz`, `uuid`,
+    `jsonb`, enum and array columns take the text a SQL or CSV source delivers.
+  - `bulk_copy` works with `columns: auto`, not with `key`. `columns` cannot be combined with
+    `insert_query` or `auto_create_table`.
+  - `SqlxConfig` gains the fields `columns`, `key` and `extra_column`, and `models` the type `SqlColumns`.
+- **SQL source: `timestamps: rfc3339`.** A Postgres `timestamptz` column is read in Postgres's
+  text form, `2026-10-04 09:15:23.923277+00`, which few other systems parse. With
+  `timestamps: rfc3339` a `cursor_column` source renders it in UTC as
+  `2026-10-04T09:15:23.923277Z`; a `timestamp` without zone gets the `T` and no offset. The
+  default is unchanged. `SqlxConfig` gains the field `timestamps` and `models` the type
+  `SqlTimestamps`.
+
+### Behaviour changes
+
+- **`aggregate`: states in memory only are opt-in.** An `aggregate` without
+  a `store` kept its states in the process, lost on restart and wrong with two instances, and
+  that was the default. It now needs `consistency: single_writer` next to it; with neither a
+  `store` nor that, the route fails to start with an error naming both. Add
+  `consistency: single_writer` to a config from 0.4.17 or 0.4.18 that has no `store` to keep
+  what it did.
+- **A CSV sink flattens nested objects.** A nested object used to be written as JSON text in
+  one cell, so the output of `aggregate` (`{"stats":{"n":3,"avg":2.5}}`) came out as a mix of
+  CSV and JSON. It now becomes one column per leaf, named by its path: `stats.n`, `stats.avg`.
+  **A new file gets different columns than before; a file that already has a header keeps
+  it.** Set `csv.nested: json` for the previous output. Arrays are still written as JSON text.
+- **A drain does not wait for an endpoint.** With `exit_on_empty` (`mqb copy --drain`) a route
+  that failed reconnected up to ten times, 5 s apart, before it gave up: about 50 s against a
+  closed port, a WebSocket path that answers 404, a NATS subject outside every stream or a gRPC
+  topic nobody serves. It now ends on the first failure when nothing has been delivered yet,
+  with the endpoint's error. A drain that has delivered something keeps the ten reconnects, so
+  a short outage in the middle of a long copy does not end it. Routes without `exit_on_empty`
+  reconnect without limit, as before. Add a `retry` middleware to a drain that should wait for
+  its sink.
+- **A CSV file that ends inside a quoted field fails a drain.** The reader took everything
+  after the open quote as one field and reported success. A drain now writes the records before
+  it and fails, naming the byte offset of the open record; a tailing route waits for the writer
+  to finish the record.
+- **Blank lines in a line-based file are skipped.** `raw`, `json`, `text` and `normal` sources
+  emitted an empty line as a message with an empty payload, which a SQL or Parquet sink then
+  rejected. The CSV reader already skipped them. Row counts change for files with blank lines.
+- **A `file` route whose source and sink are the same path is refused.** It doubled the file
+  under a drain and would grow it without end otherwise.
+- **SQLite `has no column named` is not retried.** A sink writing to a table without the
+  expected columns retried for ever; it is now a non-retryable error like the same case on
+  Postgres and MySQL.
+- **An `object_store` source on a local path that does not exist fails a drain.** It reported
+  `copied 0 rows`. An empty directory or cloud prefix still does. A drain of a local path also
+  ends after one listing instead of waiting out the idle interval.
+- **NATS: a missing `subject` or `stream` stops the route.** It was retried as a connection
+  error until the start timeout.
+- **Clearer errors and quieter logs.** A queue read on a plain SQL table names
+  `cursor_column`; Postgres errors no longer end in `at line N` (a line of the server's source);
+  a closed database port reports `Connection refused` at once; a `filter` that does not compile
+  lists the operators, and one with a single `=` says so; a gRPC server names the topic it has
+  no consumer for; the drop summary counts messages per cause; an unknown `aggregate` field
+  lists the built-ins; `csv` settings without `format: csv` and an `insert_query` token that
+  binds NULL each warn once. The WebSocket "falling back to routed mode" line is now DEBUG and
+  the MQTT note on restart loss INFO.
+
+### Fixed
+
+- **`mqb copy --wait` reports a failed route at once.** A route that failed before it delivered
+  anything was retried until the wait ran out, and a later empty attempt could then report
+  success. `--wait` now retries only a drain that ended cleanly with nothing to copy.
+- **`aggregate` with a shared `store` no longer keeps the state of a message it rejects.** A
+  message whose result had no place in the payload (`into: a.b` where `a` is not an object) was
+  counted in the store and then failed, so under `on_error: fail` every redelivery counted it
+  again. It now fails before any state changes, as it already did without a shared store.
+
+## 0.4.18
+
+A new generic output for search engines, plus documentation and packaging work. This release
+changes behaviour in several places, listed under "Behaviour changes": read that section before
+upgrading.
+
+### Added
+
+- **`http_bulk`: a generic bulk document endpoint for search engines.** It sends a batch as one
+  NDJSON or JSON-array request and deletes by id, and reads the outcome from the HTTP status,
+  from one result line or array entry per document, or from a polled job. An action line per
+  document, an envelope around the array and `gzip`/`zstd`/`lz4` request bodies are
+  configuration too. The target is described in configuration, so no product is named in
+  code; the book has tested recipes for Meilisearch, Typesense, Elasticsearch, Qdrant,
+  PostgREST and CouchDB. On the command line it is `http-bulk:?config_file=<yaml>` or
+  `http-bulk:?config=<YAML or JSON>`. As an input, `read` pages through a JSON listing by a
+  cursor taken from the response, from the last document, or counted as an offset; the position
+  is saved with `read.cursor_id` and `read.checkpoint_store` and works with `mqb copy --resume`. Reading
+  was run against CouchDB's change feed, Qdrant, Meilisearch and PostgREST. Behind the new
+  `http-bulk` feature, which `full` and `portable` include. `EndpointType` gains the variant
+  `HttpBulk`. A polled job has a deadline, `result.job.timeout_ms` (default 5 minutes): when it
+  passes, or the poll keeps failing that long, the documents of that request fail as retryable
+  with the job id and the last state or poll error seen.
+- **`http_bulk` answers lookups in bulk.** A `query` section makes the endpoint the `from` of a
+  `lookup`: every message renders `request`, one HTTP request carries the whole batch
+  (`ndjson`, or `json_array` inside an `envelope` with a `{requests}` marker), and the answers
+  are matched by position in the array at `responses`, with optional pointers `value`, `found`
+  and `error`. It covers Elasticsearch `_msearch` and `_mget` and Qdrant's batch search. A
+  missing answer is `null`; an entry that reports an error makes the batch fall back to one
+  request per message, with a warning, so only that message fails. `query` excludes `upsert`,
+  `delete` and `operation`, and an output now needs `upsert` or `query`.
+- **`http_bulk` authentication: OAuth2 client credentials and AWS SigV4.** `auth.oauth2`
+  (`token_url`, `client_id`, `client_secret`, `scope`) fetches a token, caches it, replaces it
+  before it expires and once more after a 401; concurrent batches share one token request.
+  `auth.aws_sigv4` (`region`, `service`, optional static keys, else the default credential
+  chain) signs each request and needs the `aws` feature. Both apply to upserts, deletes,
+  queries, reads and job polls; the secrets are extracted like `headers`. The `aws` feature
+  now depends on `aws-sigv4` directly, a crate the AWS SDK already brought in.
+- **`http_bulk` partial updates.** `upsert.document` is a template around each document with a
+  `{document}` marker, for `ndjson` and `json_array`. The `elasticsearch` endpoint gains
+  `mode: update` (`{"doc":…,"doc_as_upsert":true}` behind an `update` action) and `auth`.
+- **`typesense` and `elasticsearch` endpoints.** Each is an `http_bulk` output with the
+  requests filled in, so it takes a few fields (`url`, `collection` or `index`, `api_key`,
+  `operation`) and a URI of its own: `mqb copy … 'typesense://host:8108/books?api_key=…'`,
+  with `typesense+https://` for HTTPS. `mqb` has both built in; a library registers one with
+  `endpoints::http_bulk::register_preset("typesense")` and addresses it as a `custom`
+  endpoint. An installed plugin of the same name replaces one through `MQB_PLUGIN_OVERRIDE`.
+- **The Meilisearch book page covers bulk-load speed and embeddings.** Index settings before
+  loading, batch sizes with measurements, pgvector columns as `_vectors`, Supabase, keeping
+  joined fields current and a rebuild without downtime.
+- **Book pages for the plugin connectors.** Apache Pulsar, Meilisearch and the Connect plugin
+  (Redpanda Connect components) each have a connector page. The Connect inputs, outputs and
+  processors are listed with their fields, generated by `dev/scripts/gen-connect-docs.py`.
+- **Warehouse recipes in the cookbook.** Loading the Parquet that `object_store` writes into
+  Snowflake, BigQuery and Databricks / Spark, querying it from Athena / Trino, and when to use
+  the ClickHouse endpoint versus `s3()`.
+- **`llms.txt`.** A capability index for agents, at the repository root and published with the
+  book. The docs workflow fails if the book stops shipping it.
+- **A Debezium recipe in the cookbook.** MySQL, MariaDB, SQL Server and Oracle change capture by
+  running Debezium Server beside mq-bridge, handed over by HTTP or through a broker.
+- **A feature-flag table on docs.rs.** The crate docs list each Cargo feature with what it
+  enables, and link to the README's capabilities table.
+- **Helpers for plugin authors.** A host now accepts the schema `#[derive(JsonSchema)]` gives
+  an enum field (`support::config_schema::flatten`), so a plugin no longer hand-writes one.
+  `SentBatch::from_outcomes` builds a partial result from per-message outcomes, and
+  `support::ndjson::chunk` frames payloads as size-limited NDJSON request bodies.
+  `support::http_status` classifies an HTTP status as retryable or not,
+  `support::poll_job::poll_until` waits for an asynchronous job to end or a deadline to pass,
+  `support::change_op::ChangeOp` sorts a change record into upsert, delete or truncate, and
+  `plugin::test_support::StubHttpServer` replaces the remote system in a test. See
+  [docs/PLUGINS.md](docs/PLUGINS.md#shared-helpers).
+- **Plugin ABI 1.3: a loaded plugin learns how its route ended.** The host now calls
+  `publisher_disconnect` with the outcome before it closes a publisher, so
+  `disconnect_outcome()` returns `Completed`, `Stopped` or `Failed` inside a plugin's
+  `on_disconnect_hook`. It returned `None` there before. Plugins built for 1.0 to 1.2 load
+  unchanged. C plugins that use `MQB_NO_PUBLISHER` or `MQB_BLOCKING_PUBLISHER` get a stub.
+- **An endpoint can ask for its own default batch size.** `mqb copy` and the MCP server's
+  `start_route` use the top-level schema annotation `x-mqb-default-batch-size` of a custom
+  output when no batch size is given. The general default stays 1,024; of the built-in
+  endpoints only the `meilisearch` preset declares one, 50,000.
+- **Plugin repository template.** [examples/plugin-template](examples/plugin-template) holds a
+  working endpoint with its npm and Python packages, conda recipe, Homebrew formula script and
+  CI and release workflows, plus a script that renames it.
+
+### Changed
+
+- **`meilisearch` in `mqb` is now built on `http_bulk`; the plugin is no longer compiled in.**
+  The built-in endpoint keeps the URI and the fields `url`, `api_key`, `index`, `primary_key`,
+  `operation`, `delete_values`, `max_request_bytes`, `compression`, the timeouts and the input
+  fields. It refuses `settings`, `method`, a template in `index`, `update_where`,
+  `update_into`, `swap_into`, `create_index`, `wait_for_task` and `task_timeout_ms` at startup
+  and names the `mq-bridge-meilisearch` plugin, which still has them: install it and set
+  `MQB_PLUGIN_OVERRIDE=meilisearch`, or build with the cargo feature `meilisearch`, which
+  `full` no longer includes. `delete_values` now defaults to `delete` and `d`.
+- **Secret extraction covers custom endpoints and middlewares.** `extract_config_secrets` and
+  the app's `balanced` config security mode now move credentials out of a `custom` config, such
+  as a Connect, Pulsar or Meilisearch endpoint: fields named like a credential and URLs
+  carrying `user:pass@`. They used to stay in the config file. The cookbook's Secrets page
+  lists what is recognised and what is left in place.
+- **Package metadata describes the ETL scope.** The crates.io, PyPI and npm descriptions,
+  keywords and categories now cover CDC, Parquet and object storage as well as the brokers. The
+  homepage points at the book instead of the repository.
+- **`AGENTS.md` holds the contributor context** that was in `CLAUDE.md`, which now imports it.
+
+### Behaviour changes
+
+Each of these changes what an existing setup sees. Check them before upgrading.
+
+- **Replies carry the request's message id.** On NATS, memory and MongoDB request/reply, and on
+  the `static` and `reader` endpoints, a reply's `message_id` is now the request's id and no
+  longer the responder's. **Anything that keyed on the responder's id sees new values: a
+  `deduplication` on a `request` endpoint's `forward_to` starts from an empty history after the
+  upgrade.** `reader` keeps the pulled message's own id in the metadata `mqb.reader.message_id`.
+- **Unknown fields in a `file` or `object_store` endpoint are rejected.** `file: { path: x,
+  fromat: raw }` used to load and run with the default format. It now fails with the field's
+  name, as the other endpoints did.
+- **The CSV sink keeps the first row's field order.** Columns were written alphabetically, so
+  the column order of an existing CSV export changes. The order is read once per file.
+- **The SQL cursor source emits Postgres `json`, `jsonb` and array columns as nested JSON.**
+  They arrived as strings: `"j":"{\"k\": 1}"` is now `"j":{"k":1}`, and `"{1,2}"` is `[1,2]`.
+  `numeric` stays a string, which keeps its precision.
+- **`mqb copy` logs to stderr.** Only the `copied …` summary is on stdout, so a script can read
+  it. `MQB_LOG_STDOUT=1` puts the log lines back on stdout.
+- **`mqb copy` exits 1 when rows it read could not be decrypted**, for example an `encryption`
+  with the wrong key. It printed `copied 0 of 1_000 rows` and exited 0. Rows a `transform`
+  rejects are still reported in the summary with exit 0.
+- **`mqb copy` reports dead-lettered rows separately**: `copied 0 rows, dead-lettered 1_000`
+  where it said `copied 1_000 rows`. The exit code is still 0.
+- **`mqb --config <file>` fails when the file does not exist.** It started with default
+  settings and waited. The implicit `config.yml` is still optional.
+- **`static` as a `--drain` source emits one batch and ends.** It never ended.
+- **A SQLite sink creates its database file** when the URL names no `mode`. It failed with
+  `unable to open database file` unless the URL carried `mode=rwc`.
+- **`mqb copy --drain` stops at once on an authentication failure** (a wrong Postgres, MySQL,
+  RabbitMQ or Redis password, MongoDB `capture_all` without a replica set) instead of retrying
+  for the start window. A bridge keeps retrying, and so does an unreachable host.
+- **An unreachable MQTT broker (sink) or Kafka broker (source) fails the start** after the
+  start window, as other endpoints do. `copy --drain` used to retry until stopped.
+- **A SQL cursor source without a checkpoint continues after a reconnect** instead of reading
+  the table from the first row again. The position is held in memory for the life of the
+  route, so a restart of the process still starts over unless `--resume` or a checkpoint is set.
+- **A `null` output keeps its middlewares in the app's config.** `output: { "null": null,
+  middlewares: [...] }` was loaded as no output at all.
+- **`mqb copy` warns** when the target is `response:` and the source cannot take a reply, and
+  when a SQL source has neither `cursor_column` nor `delete_after_read`, which leases the rows
+  for a minute and leaves them in place.
+
+### Fixed
+
+- **Request/reply replies are matched in batches.** A reply from a `static` responder, and from
+  NATS, memory and MongoDB request/reply, kept its own message id. A batch matches replies to
+  requests by id, so in a batch larger than one the reply was dropped and an HTTP caller got
+  `202 Message processed`. Replies now carry the request's id, as HTTP and SQL replies did.
+- **MQTT: the first batch of a QoS 1/2 publisher is no longer sent twice.** The first CONNACK
+  was treated as a session reset, so the batch was reported unconfirmed, retried after 5 s and
+  delivered again.
+- **Kafka: draining a group that is already at the end returns at once** instead of waiting out
+  the 30 s join timeout, and the end-of-partition event is no longer logged as an `ERROR`.
+- **MongoDB: `consume: snapshot` no longer returns the bridge's own `<collection>:sequencer`
+  document** as a row.
+- **A configuration that can never work fails at once with its reason**, instead of being
+  retried for the 5 s start window: an invalid filter expression, a bad encryption key, a
+  middleware on the wrong side, a `limiter` of 0, a directory as a file source.
+- **Clearer errors.** A file sink that cannot be opened shows the OS reason; a start timeout
+  with no recorded cause says an endpoint was still connecting; `${payload:…}` in a `copy` URI
+  explains the `$%7B…%7D` spelling; the NATS publisher no longer warns that `stream` is ignored
+  when it is used.
+- **`mqb copy` reports the rate of delivered rows** after an `unpack`, not of packs read.
+- **The `compression` middleware uses every core.** Batches of 256 messages and more are
+  compressed and decompressed on the blocking pool, in order: zstd 316k → 1.29M rows/s, gzip
+  160k → 400k, lz4 1.9M → 3.9M. The output is byte-identical.
+
 ## 0.4.17
 
 ### Added

mq_bridge_basic-0.4.19/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.19}/Cargo.lock

@@ -3689,7 +3689,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.4.17"
+version = "0.4.19"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3704,6 +3704,7 @@ dependencies = [
  "aws-config",
  "aws-sdk-sns",
  "aws-sdk-sqs",
+ "aws-sigv4",
  "base64 0.23.1",
  "byteorder",
  "bytes",
@@ -3791,7 +3792,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.4.17"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3805,7 +3806,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.4.17"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3830,6 +3831,7 @@ name = "mq-bridge-plugin-fixture"
 version = "0.1.0"
 dependencies = [
  "anyhow",
+ "async-channel",
  "async-trait",
  "cbindgen",
  "libloading",
@@ -3859,7 +3861,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.4.17"
+version = "0.4.19"
 dependencies = [
  "anyhow",
  "async-trait",

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.19}/Cargo.toml

@@ -4,12 +4,14 @@ version.workspace = true
 edition = "2021"
 authors = ["Marco Mengelkoch"]
 rust-version = "1.85"  # Base crate MSRV (non-optional uuid 1.24 needs 1.85); optional amqp/mongodb/nats need Rust 1.88.
-description = "An asynchronous message bridging library connecting Kafka, MQTT, AMQP, NATS, MongoDB, HTTP, and more."
+description = "Embeddable async ETL and messaging library: brokers (Kafka, NATS, AMQP, MQTT), databases incl. Postgres/MongoDB CDC, Parquet on S3/GCS/Azure object storage, HTTP/gRPC."
 repository = "https://github.com/marcomq/mq-bridge"
+homepage = "https://marcomq.github.io/mq-bridge/"
+documentation = "https://docs.rs/mq-bridge"
 license = "MIT OR Apache-2.0"
 readme = "README.md"
-keywords = ["messaging", "kafka", "nats", "amqp", "cqrs"]
-categories = ["asynchronous", "network-programming"]
+keywords = ["kafka", "messaging", "etl", "cdc", "cqrs"]
+categories = ["asynchronous", "network-programming", "database"]
 # Test-only proto; build.rs skips it when absent so downstream builds skip the codegen.
 # `apps/` is a sibling product, not part of the library: cargo only auto-excludes
 # the app's member crates, so the UI, book and benches would ship in the tarball.
@@ -21,7 +23,7 @@ workspace = true
 [workspace]
 members = ["bindings-common", "python/mq-bridge-py"]
 # apps/mq-bridge-app is its own workspace with its own Cargo.lock; see its manifest.
-exclude = ["apps"]
+exclude = ["apps", "examples/plugin-template"]
 resolver = "2"
 
 # Lints that are off by default in rustc/clippy and mark code that was never meant to
@@ -38,7 +40,7 @@ dbg_macro = "warn"
 exit = "warn"
 
 [workspace.package]
-version = "0.4.17"
+version = "0.4.19"
 
 [workspace.dependencies]
 anyhow = "1.0"
@@ -214,6 +216,9 @@ tokio-websockets = { version = "0.13", optional = true, default-features = false
 aws-config = { version = "1.10", optional = true, default-features = false, features = ["behavior-version-latest", "rt-tokio", "default-https-client"] }
 aws-sdk-sqs = { version = "1.105", optional = true, default-features = false, features = ["rt-tokio", "default-https-client"] }
 aws-sdk-sns = { version = "1.107", optional = true, default-features = false, features = ["rt-tokio", "default-https-client"] }
+# Signs `http_bulk` requests (managed OpenSearch). Already in the tree through aws-config.
+# A low floor: 1.6 needs aws-smithy-types 1.8, which a dependent may pin below.
+aws-sigv4 = { version = "1.4", optional = true }
 
 # SQLx
 sqlx = { version = "0.9", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "mysql", "any", "sqlite-bundled"], optional = true }
@@ -279,7 +284,7 @@ middleware = ["metrics", "otel", "dedup", "compression", "encryption", "filter",
 # on ~19% of 17-significant-digit doubles (e.g. a Postgres `double precision` value
 # moved through a JSON payload). Enable if bit-exact float round-tripping matters.
 float-roundtrip = ["serde_json/float_roundtrip"]
-full-common = ["middleware", "yaml", "plugin", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "zeromq-omq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "websocket", "postgres-cdc", "object-store", "parquet", "avro", "rustls-aws-lc"]
+full-common = ["middleware", "yaml", "plugin", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "zeromq-omq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "http-bulk", "websocket", "postgres-cdc", "object-store", "parquet", "avro", "rustls-aws-lc"]
 
 # ibm-mq (dlopen) is safe to include: it needs no IBM SDK at build, and at
 # runtime the client is loaded lazily only when an ibm-mq endpoint is opened
@@ -315,7 +320,7 @@ vendored-protoc = ["dep:protoc-bin-vendored"]
 # kafka (librdkafka), sqlx (libsqlite3-sys), grpc (protoc), ibm-mq/ibm-mq-static
 # (IBM C client). postgres-cdc is excluded too: its wire protocol is pure Rust
 # but it pulls in sqlx (C).
-portable = ["metrics", "otel", "dedup", "yaml", "plugin", "mongodb", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "clickhouse", "websocket", "object-store", "rustls-aws-lc"]
+portable = ["metrics", "otel", "dedup", "yaml", "plugin", "mongodb", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "clickhouse", "http-bulk", "websocket", "object-store", "rustls-aws-lc"]
 
 # Functionality features
 # YAML config files for Route/Publisher::from_file and deploy_file; JSON works without it.
@@ -375,7 +380,7 @@ mongodb = ["dep:mongodb"]
 mqtt = ["rumqttc", "rustls", "tokio-rustls", "rustls-pemfile"]
 http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 websocket = ["dep:tokio-websockets"]
-aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns"]
+aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns", "aws-sigv4"]
 # Runtime-loaded IBM MQ client via dlopen: builds with no SDK present; the IBM
 # redistributable client is only needed at runtime, and only if an ibm-mq
 # endpoint is actually used.
@@ -398,6 +403,8 @@ redis-streams = ["dep:redis"]
 grpc = ["tonic", "tonic-reflection", "prost", "prost-types", "prost-reflect", "tokio-stream", "tonic-prost", "tonic-prost-build", "rustls", "vendored-protoc"]
 sqlx = ["dep:sqlx"]
 clickhouse = ["dep:reqwest", "dep:flate2", "dep:lz4_flex", "dep:zstd", "dep:percent-encoding"]
+# Generic bulk document sink over HTTP (search engines and similar APIs).
+http-bulk = ["dep:reqwest", "dep:percent-encoding", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 # Cloud object-store checkpoint backend (S3/GCS/Azure/R2). Talks cloud REST via reqwest.
 object-store = ["dep:object_store"]
 # `format: parquet` on object_store. Column codecs reuse the `compression` crates.

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.19}/PKG-INFO

@@ -1,7 +1,12 @@
 Metadata-Version: 2.4
 Name: mq-bridge-basic
-Version: 0.4.17
+Version: 0.4.19
 Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Database
+Classifier: Topic :: System :: Networking
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Topic :: Software Development :: Libraries
 Classifier: License :: OSI Approved :: MIT License
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
@@ -24,7 +29,9 @@ Keywords: messaging,tokio,pyo3,maturin,mq-bridge
 License: MIT OR Apache-2.0
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
-Project-URL: Homepage, https://github.com/marcomq/mq-bridge
+Project-URL: Book, https://marcomq.github.io/mq-bridge/
+Project-URL: Documentation, https://marcomq.github.io/mq-bridge/getting-started/library-quick-start.html
+Project-URL: Homepage, https://marcomq.github.io/mq-bridge/
 Project-URL: Issues, https://github.com/marcomq/mq-bridge/issues
 Project-URL: Repository, https://github.com/marcomq/mq-bridge