mq-bridge-basic 0.4.17__tar.gz → 0.4.18__tar.gz

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/.github/workflows/docs.yml

@@ -43,6 +43,10 @@ jobs:
       - name: Build book
         run: mdbook build apps/mq-bridge-app/dev/docs
 
+      # mdbook copies non-Markdown files from the book source; fail if that ever stops.
+      - name: Check llms.txt is published
+        run: test -s apps/mq-bridge-app/dev/docs/book/llms.txt
+
       # The benchmark dashboard lives on the gh-pages branch. Copy it into the
       # book so this workflow is the only thing that publishes to Pages.
       - name: Checkout benchmark data

mq_bridge_basic-0.4.17/CLAUDE.md → mq_bridge_basic-0.4.18/AGENTS.md

@@ -3,16 +3,16 @@
 > **Looking for what middleware or structural endpoints exist, and how to configure them?**
 > [REFERENCE.md](docs/REFERENCE.md) is the complete, authoritative list — every middleware
 > (`retry`, `dlq`, `transform`, `id`, `filter`, `deduplication`, `weak_join`, `buffer`, `limiter`, `delay`,
-> `cookie_jar`, `encryption`, `compression`, `pack`, `unpack`, `metrics`, `otel`, `lookup`, `aggregate`, `random_panic`, `custom`) and every
+> `cookie_jar`, `encryption`, `compression`, `pack`, `unpack`, `timeout`, `metrics`, `otel`, `lookup`, `aggregate`, `random_panic`, `custom`) and every
 > structural endpoint (`ref`,
-> `fanout`, `switch`, `request`, `response`, `reader`, `static`, `stream_buffer`, `null`,
+> `fanout`, `switch`, `sequence`, `request`, `response`, `reader`, `static`, `stream_buffer`, `null`,
 > `custom`), each with its fields, defaults, and a working YAML example. Do not infer these
 > from the enum definitions in `models.rs`; the reference records the behaviour and the
 > spelling traps too. Every snippet in it is parsed by `tests/reference_docs_test.rs`.
 
 ## Project Overview
 
-`mq-bridge` is an asynchronous message bridging library for Rust that connects different messaging systems, data stores, and protocols. It acts as a **programmable integration layer**, allowing for transformation, filtering, handling, events, and complex routing.
+`mq-bridge` is an asynchronous Rust library that moves messages and data between brokers, databases, object storage, files and HTTP. What it can do, and where each capability is documented, is in [llms.txt](llms.txt) and the README's "Capabilities at a glance" table; this file covers how to work on the engine. The app in `apps/mq-bridge-app` has its own [AGENTS.md](apps/mq-bridge-app/AGENTS.md).
 
 ## Core Architecture
 
@@ -41,7 +41,10 @@ src/
 ├── canonical_message.rs   # Unified message format
 ├── traits.rs             # Core traits (MessageConsumer, MessagePublisher, Handler)
 ├── models.rs             # Configuration models (Route, Endpoint, Middleware)
+├── models/               # models.rs submodules: builders, defaults, secrets, serde support
+├── config_file.rs        # Loading routes from YAML/JSON files
 ├── route.rs              # Route execution logic (sequential/concurrent)
+├── shutdown.rs           # Graceful shutdown signalling
 ├── endpoints/            # Endpoint implementations
 │   ├── mod.rs           # Factory functions for creating consumers/publishers
 │   ├── amqp.rs          # AMQP (RabbitMQ) consumer/publisher
@@ -49,12 +52,13 @@ src/
 │   ├── clickhouse.rs    # ClickHouse sink + cursor source
 │   ├── dir_spool/       # Crash-safe directory FIFO queue (payload file + JSON sidecar)
 │   ├── file/            # File-based endpoints
-│   ├── grpc.rs          # gRPC consumer/publisher
+│   ├── grpc/            # gRPC consumer/publisher
 │   ├── http/            # HTTP consumer/publisher (+ streaming)
+│   ├── http_bulk/       # Bulk JSON documents over HTTP (search engines): publisher, paged consumer, presets
 │   ├── ibm_mq.rs        # IBM MQ (client loaded at runtime via dlopen)
 │   ├── kafka.rs         # Kafka consumer/publisher
 │   ├── memory/          # In-memory channels + IPC transports
-│   ├── mongodb.rs       # MongoDB consumer/publisher/change streams
+│   ├── mongodb/         # MongoDB consumer/publisher/change streams
 │   ├── mqtt.rs          # MQTT consumer/publisher
 │   ├── nats.rs          # NATS consumer/publisher
 │   ├── object_store.rs  # Cloud object storage (S3 / GCS / Azure)
@@ -68,6 +72,7 @@ src/
 │   └── structural/      # Structural endpoints (no external system)
 │       ├── fanout.rs          # Broadcast to every listed endpoint
 │       ├── switch.rs          # Content-based routing
+│       ├── sequence.rs        # Drain several inputs one after another (backfill, then stream)
 │       ├── request.rs         # Request/reply call, forward the response
 │       ├── response.rs        # Reply to the origin of the current request
 │       ├── reader.rs          # Trigger a pull from a consumer
@@ -77,8 +82,10 @@ src/
 ├── middleware/           # Middleware implementations
 │   ├── aggregate/       # Per-key running state (sums, counters, moving averages)
 │   ├── buffer.rs        # Batch accumulation
+│   ├── compression.rs   # Per-message payload compression
 │   ├── cookie_jar.rs    # Cookie / metadata persistence across requests
-│   ├── deduplication.rs # Message deduplication (sled)
+│   ├── deduplication.rs # Message deduplication
+│   ├── deduplication/   # Dedup stores (sled, memory, SQL, MongoDB)
 │   ├── delay.rs         # Artificial delay
 │   ├── deferred_commit.rs # Hold commits for batches a middleware emptied
 │   ├── dlq.rs           # Dead-letter queue
@@ -89,15 +96,22 @@ src/
 │   ├── lookup.rs        # Enrich from another endpoint's response
 │   ├── metrics.rs       # Metrics collection
 │   ├── otel.rs          # OpenTelemetry spans + traceparent propagation
+│   ├── pack.rs          # pack / unpack: one transport message per batch
 │   ├── random_panic.rs  # Testing middleware
+│   ├── raw_json.rs      # Shared raw-JSON payload helpers
 │   ├── retry.rs         # Exponential backoff retry
+│   ├── timeout.rs       # Bound each send; fail as retryable
 │   ├── transform/       # Declarative JSON mapping + schema coercion
 │   └── weak_join.rs     # Correlation-keyed join
 ├── support/              # Cross-cutting helpers
 │   ├── compression.rs   # gzip / lz4 / zstd
 │   ├── connection_registry.rs # Shared connection reuse
 │   ├── crypto.rs        # AEAD core (used by encryption middleware + at-rest)
-│   └── interpolation.rs # `${namespace:selector}` templating
+│   ├── interpolation.rs # `${namespace:selector}` templating
+│   ├── parquet.rs       # Parquet encode/decode for object_store
+│   ├── source_ranges.rs # Covered-range recovery for replay-safe file/object sinks
+│   ├── pack.rs          # Batch envelope used by pack / unpack
+│   └── plugin_abi.rs    # C ABI shared by host and native plugins
 ├── command_handler.rs    # Command handler wrapper
 ├── event_handler.rs      # Event handler wrapper
 ├── event_store.rs        # In-memory event store
@@ -105,6 +119,7 @@ src/
 ├── publisher.rs          # Standalone publisher API
 ├── checkpoint.rs         # Durable cursor stores (file/s3/postgres/mongodb)
 ├── extensions.rs         # Custom endpoint/middleware factory registration
+├── plugin/               # Native plugin host, SDK and conformance suite
 ├── response.rs           # Ergonomic response helpers
 ├── test_utils.rs         # Shared test helpers
 ├── errors.rs             # Error types
@@ -142,20 +157,7 @@ Routes are defined via:
 - JSON files
 - Environment variables (prefix: `MQB__`, separator: `__`)
 
-Example:
-```yaml
-kafka_to_nats:
-  concurrency: 4
-  batch_size: 128
-  input:
-    kafka:
-      topic: "orders"
-      url: "localhost:9092"
-  output:
-    nats:
-      subject: "orders.processed"
-      url: "nats://localhost:4222"
-```
+The grammar and examples are in [CONFIGURATION.md](docs/CONFIGURATION.md).
 
 ### Common Patterns
 
@@ -188,20 +190,4 @@ kafka_to_nats:
 
 ### Dependencies
 
-- **tokio**: Async runtime
-- **serde**: Serialization/deserialization
-- **anyhow**: Error handling
-- **tracing**: Structured logging
-- **config**: Configuration management
-- **async-trait**: Async trait support
-- **uuid**: Unique identifiers (v4 and v7)
-
-Optional (feature-gated):
-- **rdkafka**: Kafka support
-- **async-nats**: NATS support
-- **lapin**: AMQP support
-- **rumqttc**: MQTT support
-- **mongodb**: MongoDB support
-- **actix/reqwest**: HTTP support
-- **sled**: Deduplication storage
-- **metrics**: Metrics collection
+Dependencies and the feature that gates each optional one are in `Cargo.toml`, with comments explaining the non-obvious choices. The crate has no default features.

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/CHANGELOG.md

@@ -2,6 +2,178 @@
 
 All notable changes to `mq-bridge`. Newest first.
 
+## 0.4.18
+
+A new generic output for search engines, plus documentation and packaging work. This release
+changes behaviour in several places, listed under "Behaviour changes": read that section before
+upgrading.
+
+### Added
+
+- **`http_bulk`: a generic bulk document endpoint for search engines.** It sends a batch as one
+  NDJSON or JSON-array request and deletes by id, and reads the outcome from the HTTP status,
+  from one result line or array entry per document, or from a polled job. An action line per
+  document, an envelope around the array and `gzip`/`zstd`/`lz4` request bodies are
+  configuration too. The target is described in configuration, so no product is named in
+  code; the book has tested recipes for Meilisearch, Typesense, Elasticsearch, Qdrant,
+  PostgREST and CouchDB. On the command line it is `http-bulk:?config_file=<yaml>` or
+  `http-bulk:?config=<YAML or JSON>`. As an input, `read` pages through a JSON listing by a
+  cursor taken from the response, from the last document, or counted as an offset; the position
+  is saved with `read.cursor_id` and `read.checkpoint_store` and works with `mqb copy --resume`. Reading
+  was run against CouchDB's change feed, Qdrant, Meilisearch and PostgREST. Behind the new
+  `http-bulk` feature, which `full` and `portable` include. `EndpointType` gains the variant
+  `HttpBulk`. A polled job has a deadline, `result.job.timeout_ms` (default 5 minutes): when it
+  passes, or the poll keeps failing that long, the documents of that request fail as retryable
+  with the job id and the last state or poll error seen.
+- **`http_bulk` answers lookups in bulk.** A `query` section makes the endpoint the `from` of a
+  `lookup`: every message renders `request`, one HTTP request carries the whole batch
+  (`ndjson`, or `json_array` inside an `envelope` with a `{requests}` marker), and the answers
+  are matched by position in the array at `responses`, with optional pointers `value`, `found`
+  and `error`. It covers Elasticsearch `_msearch` and `_mget` and Qdrant's batch search. A
+  missing answer is `null`; an entry that reports an error makes the batch fall back to one
+  request per message, with a warning, so only that message fails. `query` excludes `upsert`,
+  `delete` and `operation`, and an output now needs `upsert` or `query`.
+- **`http_bulk` authentication: OAuth2 client credentials and AWS SigV4.** `auth.oauth2`
+  (`token_url`, `client_id`, `client_secret`, `scope`) fetches a token, caches it, replaces it
+  before it expires and once more after a 401; concurrent batches share one token request.
+  `auth.aws_sigv4` (`region`, `service`, optional static keys, else the default credential
+  chain) signs each request and needs the `aws` feature. Both apply to upserts, deletes,
+  queries, reads and job polls; the secrets are extracted like `headers`. The `aws` feature
+  now depends on `aws-sigv4` directly, a crate the AWS SDK already brought in.
+- **`http_bulk` partial updates.** `upsert.document` is a template around each document with a
+  `{document}` marker, for `ndjson` and `json_array`. The `elasticsearch` endpoint gains
+  `mode: update` (`{"doc":…,"doc_as_upsert":true}` behind an `update` action) and `auth`.
+- **`typesense` and `elasticsearch` endpoints.** Each is an `http_bulk` output with the
+  requests filled in, so it takes a few fields (`url`, `collection` or `index`, `api_key`,
+  `operation`) and a URI of its own: `mqb copy … 'typesense://host:8108/books?api_key=…'`,
+  with `typesense+https://` for HTTPS. `mqb` has both built in; a library registers one with
+  `endpoints::http_bulk::register_preset("typesense")` and addresses it as a `custom`
+  endpoint. An installed plugin of the same name replaces one through `MQB_PLUGIN_OVERRIDE`.
+- **The Meilisearch book page covers bulk-load speed and embeddings.** Index settings before
+  loading, batch sizes with measurements, pgvector columns as `_vectors`, Supabase, keeping
+  joined fields current and a rebuild without downtime.
+- **Book pages for the plugin connectors.** Apache Pulsar, Meilisearch and the Connect plugin
+  (Redpanda Connect components) each have a connector page. The Connect inputs, outputs and
+  processors are listed with their fields, generated by `dev/scripts/gen-connect-docs.py`.
+- **Warehouse recipes in the cookbook.** Loading the Parquet that `object_store` writes into
+  Snowflake, BigQuery and Databricks / Spark, querying it from Athena / Trino, and when to use
+  the ClickHouse endpoint versus `s3()`.
+- **`llms.txt`.** A capability index for agents, at the repository root and published with the
+  book. The docs workflow fails if the book stops shipping it.
+- **A Debezium recipe in the cookbook.** MySQL, MariaDB, SQL Server and Oracle change capture by
+  running Debezium Server beside mq-bridge, handed over by HTTP or through a broker.
+- **A feature-flag table on docs.rs.** The crate docs list each Cargo feature with what it
+  enables, and link to the README's capabilities table.
+- **Helpers for plugin authors.** A host now accepts the schema `#[derive(JsonSchema)]` gives
+  an enum field (`support::config_schema::flatten`), so a plugin no longer hand-writes one.
+  `SentBatch::from_outcomes` builds a partial result from per-message outcomes, and
+  `support::ndjson::chunk` frames payloads as size-limited NDJSON request bodies.
+  `support::http_status` classifies an HTTP status as retryable or not,
+  `support::poll_job::poll_until` waits for an asynchronous job to end or a deadline to pass,
+  `support::change_op::ChangeOp` sorts a change record into upsert, delete or truncate, and
+  `plugin::test_support::StubHttpServer` replaces the remote system in a test. See
+  [docs/PLUGINS.md](docs/PLUGINS.md#shared-helpers).
+- **Plugin ABI 1.3: a loaded plugin learns how its route ended.** The host now calls
+  `publisher_disconnect` with the outcome before it closes a publisher, so
+  `disconnect_outcome()` returns `Completed`, `Stopped` or `Failed` inside a plugin's
+  `on_disconnect_hook`. It returned `None` there before. Plugins built for 1.0 to 1.2 load
+  unchanged. C plugins that use `MQB_NO_PUBLISHER` or `MQB_BLOCKING_PUBLISHER` get a stub.
+- **An endpoint can ask for its own default batch size.** `mqb copy` and the MCP server's
+  `start_route` use the top-level schema annotation `x-mqb-default-batch-size` of a custom
+  output when no batch size is given. The general default stays 1,024; of the built-in
+  endpoints only the `meilisearch` preset declares one, 50,000.
+- **Plugin repository template.** [examples/plugin-template](examples/plugin-template) holds a
+  working endpoint with its npm and Python packages, conda recipe, Homebrew formula script and
+  CI and release workflows, plus a script that renames it.
+
+### Changed
+
+- **`meilisearch` in `mqb` is now built on `http_bulk`; the plugin is no longer compiled in.**
+  The built-in endpoint keeps the URI and the fields `url`, `api_key`, `index`, `primary_key`,
+  `operation`, `delete_values`, `max_request_bytes`, `compression`, the timeouts and the input
+  fields. It refuses `settings`, `method`, a template in `index`, `update_where`,
+  `update_into`, `swap_into`, `create_index`, `wait_for_task` and `task_timeout_ms` at startup
+  and names the `mq-bridge-meilisearch` plugin, which still has them: install it and set
+  `MQB_PLUGIN_OVERRIDE=meilisearch`, or build with the cargo feature `meilisearch`, which
+  `full` no longer includes. `delete_values` now defaults to `delete` and `d`.
+- **Secret extraction covers custom endpoints and middlewares.** `extract_config_secrets` and
+  the app's `balanced` config security mode now move credentials out of a `custom` config, such
+  as a Connect, Pulsar or Meilisearch endpoint: fields named like a credential and URLs
+  carrying `user:pass@`. They used to stay in the config file. The cookbook's Secrets page
+  lists what is recognised and what is left in place.
+- **Package metadata describes the ETL scope.** The crates.io, PyPI and npm descriptions,
+  keywords and categories now cover CDC, Parquet and object storage as well as the brokers. The
+  homepage points at the book instead of the repository.
+- **`AGENTS.md` holds the contributor context** that was in `CLAUDE.md`, which now imports it.
+
+### Behaviour changes
+
+Each of these changes what an existing setup sees. Check them before upgrading.
+
+- **Replies carry the request's message id.** On NATS, memory and MongoDB request/reply, and on
+  the `static` and `reader` endpoints, a reply's `message_id` is now the request's id and no
+  longer the responder's. **Anything that keyed on the responder's id sees new values: a
+  `deduplication` on a `request` endpoint's `forward_to` starts from an empty history after the
+  upgrade.** `reader` keeps the pulled message's own id in the metadata `mqb.reader.message_id`.
+- **Unknown fields in a `file` or `object_store` endpoint are rejected.** `file: { path: x,
+  fromat: raw }` used to load and run with the default format. It now fails with the field's
+  name, as the other endpoints did.
+- **The CSV sink keeps the first row's field order.** Columns were written alphabetically, so
+  the column order of an existing CSV export changes. The order is read once per file.
+- **The SQL cursor source emits Postgres `json`, `jsonb` and array columns as nested JSON.**
+  They arrived as strings: `"j":"{\"k\": 1}"` is now `"j":{"k":1}`, and `"{1,2}"` is `[1,2]`.
+  `numeric` stays a string, which keeps its precision.
+- **`mqb copy` logs to stderr.** Only the `copied …` summary is on stdout, so a script can read
+  it. `MQB_LOG_STDOUT=1` puts the log lines back on stdout.
+- **`mqb copy` exits 1 when rows it read could not be decrypted**, for example an `encryption`
+  with the wrong key. It printed `copied 0 of 1_000 rows` and exited 0. Rows a `transform`
+  rejects are still reported in the summary with exit 0.
+- **`mqb copy` reports dead-lettered rows separately**: `copied 0 rows, dead-lettered 1_000`
+  where it said `copied 1_000 rows`. The exit code is still 0.
+- **`mqb --config <file>` fails when the file does not exist.** It started with default
+  settings and waited. The implicit `config.yml` is still optional.
+- **`static` as a `--drain` source emits one batch and ends.** It never ended.
+- **A SQLite sink creates its database file** when the URL names no `mode`. It failed with
+  `unable to open database file` unless the URL carried `mode=rwc`.
+- **`mqb copy --drain` stops at once on an authentication failure** (a wrong Postgres, MySQL,
+  RabbitMQ or Redis password, MongoDB `capture_all` without a replica set) instead of retrying
+  for the start window. A bridge keeps retrying, and so does an unreachable host.
+- **An unreachable MQTT broker (sink) or Kafka broker (source) fails the start** after the
+  start window, as other endpoints do. `copy --drain` used to retry until stopped.
+- **A SQL cursor source without a checkpoint continues after a reconnect** instead of reading
+  the table from the first row again. The position is held in memory for the life of the
+  route, so a restart of the process still starts over unless `--resume` or a checkpoint is set.
+- **A `null` output keeps its middlewares in the app's config.** `output: { "null": null,
+  middlewares: [...] }` was loaded as no output at all.
+- **`mqb copy` warns** when the target is `response:` and the source cannot take a reply, and
+  when a SQL source has neither `cursor_column` nor `delete_after_read`, which leases the rows
+  for a minute and leaves them in place.
+
+### Fixed
+
+- **Request/reply replies are matched in batches.** A reply from a `static` responder, and from
+  NATS, memory and MongoDB request/reply, kept its own message id. A batch matches replies to
+  requests by id, so in a batch larger than one the reply was dropped and an HTTP caller got
+  `202 Message processed`. Replies now carry the request's id, as HTTP and SQL replies did.
+- **MQTT: the first batch of a QoS 1/2 publisher is no longer sent twice.** The first CONNACK
+  was treated as a session reset, so the batch was reported unconfirmed, retried after 5 s and
+  delivered again.
+- **Kafka: draining a group that is already at the end returns at once** instead of waiting out
+  the 30 s join timeout, and the end-of-partition event is no longer logged as an `ERROR`.
+- **MongoDB: `consume: snapshot` no longer returns the bridge's own `<collection>:sequencer`
+  document** as a row.
+- **A configuration that can never work fails at once with its reason**, instead of being
+  retried for the 5 s start window: an invalid filter expression, a bad encryption key, a
+  middleware on the wrong side, a `limiter` of 0, a directory as a file source.
+- **Clearer errors.** A file sink that cannot be opened shows the OS reason; a start timeout
+  with no recorded cause says an endpoint was still connecting; `${payload:…}` in a `copy` URI
+  explains the `$%7B…%7D` spelling; the NATS publisher no longer warns that `stream` is ignored
+  when it is used.
+- **`mqb copy` reports the rate of delivered rows** after an `unpack`, not of packs read.
+- **The `compression` middleware uses every core.** Batches of 256 messages and more are
+  compressed and decompressed on the blocking pool, in order: zstd 316k → 1.29M rows/s, gzip
+  160k → 400k, lz4 1.9M → 3.9M. The output is byte-identical.
+
 ## 0.4.17
 
 ### Added

mq_bridge_basic-0.4.18/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/Cargo.lock

@@ -3689,7 +3689,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge"
-version = "0.4.17"
+version = "0.4.18"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -3704,6 +3704,7 @@ dependencies = [
  "aws-config",
  "aws-sdk-sns",
  "aws-sdk-sqs",
+ "aws-sigv4",
  "base64 0.23.1",
  "byteorder",
  "bytes",
@@ -3791,7 +3792,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-bindings-common"
-version = "0.4.17"
+version = "0.4.18"
 dependencies = [
  "anyhow",
  "fast-uuid-v7",
@@ -3805,7 +3806,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-node"
-version = "0.4.17"
+version = "0.4.18"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -3830,6 +3831,7 @@ name = "mq-bridge-plugin-fixture"
 version = "0.1.0"
 dependencies = [
  "anyhow",
+ "async-channel",
  "async-trait",
  "cbindgen",
  "libloading",
@@ -3859,7 +3861,7 @@ dependencies = [
 
 [[package]]
 name = "mq-bridge-py"
-version = "0.4.17"
+version = "0.4.18"
 dependencies = [
  "anyhow",
  "async-trait",

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/Cargo.toml

@@ -4,12 +4,14 @@ version.workspace = true
 edition = "2021"
 authors = ["Marco Mengelkoch"]
 rust-version = "1.85"  # Base crate MSRV (non-optional uuid 1.24 needs 1.85); optional amqp/mongodb/nats need Rust 1.88.
-description = "An asynchronous message bridging library connecting Kafka, MQTT, AMQP, NATS, MongoDB, HTTP, and more."
+description = "Embeddable async ETL and messaging library: brokers (Kafka, NATS, AMQP, MQTT), databases incl. Postgres/MongoDB CDC, Parquet on S3/GCS/Azure object storage, HTTP/gRPC."
 repository = "https://github.com/marcomq/mq-bridge"
+homepage = "https://marcomq.github.io/mq-bridge/"
+documentation = "https://docs.rs/mq-bridge"
 license = "MIT OR Apache-2.0"
 readme = "README.md"
-keywords = ["messaging", "kafka", "nats", "amqp", "cqrs"]
-categories = ["asynchronous", "network-programming"]
+keywords = ["kafka", "messaging", "etl", "cdc", "cqrs"]
+categories = ["asynchronous", "network-programming", "database"]
 # Test-only proto; build.rs skips it when absent so downstream builds skip the codegen.
 # `apps/` is a sibling product, not part of the library: cargo only auto-excludes
 # the app's member crates, so the UI, book and benches would ship in the tarball.
@@ -21,7 +23,7 @@ workspace = true
 [workspace]
 members = ["bindings-common", "python/mq-bridge-py"]
 # apps/mq-bridge-app is its own workspace with its own Cargo.lock; see its manifest.
-exclude = ["apps"]
+exclude = ["apps", "examples/plugin-template"]
 resolver = "2"
 
 # Lints that are off by default in rustc/clippy and mark code that was never meant to
@@ -38,7 +40,7 @@ dbg_macro = "warn"
 exit = "warn"
 
 [workspace.package]
-version = "0.4.17"
+version = "0.4.18"
 
 [workspace.dependencies]
 anyhow = "1.0"
@@ -214,6 +216,9 @@ tokio-websockets = { version = "0.13", optional = true, default-features = false
 aws-config = { version = "1.10", optional = true, default-features = false, features = ["behavior-version-latest", "rt-tokio", "default-https-client"] }
 aws-sdk-sqs = { version = "1.105", optional = true, default-features = false, features = ["rt-tokio", "default-https-client"] }
 aws-sdk-sns = { version = "1.107", optional = true, default-features = false, features = ["rt-tokio", "default-https-client"] }
+# Signs `http_bulk` requests (managed OpenSearch). Already in the tree through aws-config.
+# A low floor: 1.6 needs aws-smithy-types 1.8, which a dependent may pin below.
+aws-sigv4 = { version = "1.4", optional = true }
 
 # SQLx
 sqlx = { version = "0.9", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "mysql", "any", "sqlite-bundled"], optional = true }
@@ -279,7 +284,7 @@ middleware = ["metrics", "otel", "dedup", "compression", "encryption", "filter",
 # on ~19% of 17-significant-digit doubles (e.g. a Postgres `double precision` value
 # moved through a JSON payload). Enable if bit-exact float round-tripping matters.
 float-roundtrip = ["serde_json/float_roundtrip"]
-full-common = ["middleware", "yaml", "plugin", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "zeromq-omq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "websocket", "postgres-cdc", "object-store", "parquet", "avro", "rustls-aws-lc"]
+full-common = ["middleware", "yaml", "plugin", "mongodb", "kafka", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "zeromq-omq", "redis-streams", "sled", "grpc", "sqlx", "clickhouse", "http-bulk", "websocket", "postgres-cdc", "object-store", "parquet", "avro", "rustls-aws-lc"]
 
 # ibm-mq (dlopen) is safe to include: it needs no IBM SDK at build, and at
 # runtime the client is loaded lazily only when an ibm-mq endpoint is opened
@@ -315,7 +320,7 @@ vendored-protoc = ["dep:protoc-bin-vendored"]
 # kafka (librdkafka), sqlx (libsqlite3-sys), grpc (protoc), ibm-mq/ibm-mq-static
 # (IBM C client). postgres-cdc is excluded too: its wire protocol is pure Rust
 # but it pulls in sqlx (C).
-portable = ["metrics", "otel", "dedup", "yaml", "plugin", "mongodb", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "clickhouse", "websocket", "object-store", "rustls-aws-lc"]
+portable = ["metrics", "otel", "dedup", "yaml", "plugin", "mongodb", "amqp", "nats", "mqtt", "http", "aws", "zeromq", "redis-streams", "clickhouse", "http-bulk", "websocket", "object-store", "rustls-aws-lc"]
 
 # Functionality features
 # YAML config files for Route/Publisher::from_file and deploy_file; JSON works without it.
@@ -375,7 +380,7 @@ mongodb = ["dep:mongodb"]
 mqtt = ["rumqttc", "rustls", "tokio-rustls", "rustls-pemfile"]
 http = ["dep:arc-swap", "hyper", "hyper-util", "hyper-rustls", "http-body-util", "dep:http-body", "rustls", "tokio-rustls", "rustls-pemfile", "webpki-roots", "h2", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 websocket = ["dep:tokio-websockets"]
-aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns"]
+aws = ["aws-config", "aws-sdk-sqs", "aws-sdk-sns", "aws-sigv4"]
 # Runtime-loaded IBM MQ client via dlopen: builds with no SDK present; the IBM
 # redistributable client is only needed at runtime, and only if an ibm-mq
 # endpoint is actually used.
@@ -398,6 +403,8 @@ redis-streams = ["dep:redis"]
 grpc = ["tonic", "tonic-reflection", "prost", "prost-types", "prost-reflect", "tokio-stream", "tonic-prost", "tonic-prost-build", "rustls", "vendored-protoc"]
 sqlx = ["dep:sqlx"]
 clickhouse = ["dep:reqwest", "dep:flate2", "dep:lz4_flex", "dep:zstd", "dep:percent-encoding"]
+# Generic bulk document sink over HTTP (search engines and similar APIs).
+http-bulk = ["dep:reqwest", "dep:percent-encoding", "dep:flate2", "dep:lz4_flex", "dep:zstd"]
 # Cloud object-store checkpoint backend (S3/GCS/Azure/R2). Talks cloud REST via reqwest.
 object-store = ["dep:object_store"]
 # `format: parquet` on object_store. Column codecs reuse the `compression` crates.

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/PKG-INFO

@@ -1,7 +1,12 @@
 Metadata-Version: 2.4
 Name: mq-bridge-basic
-Version: 0.4.17
+Version: 0.4.18
 Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Database
+Classifier: Topic :: System :: Networking
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Topic :: Software Development :: Libraries
 Classifier: License :: OSI Approved :: MIT License
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
@@ -24,7 +29,9 @@ Keywords: messaging,tokio,pyo3,maturin,mq-bridge
 License: MIT OR Apache-2.0
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
-Project-URL: Homepage, https://github.com/marcomq/mq-bridge
+Project-URL: Book, https://marcomq.github.io/mq-bridge/
+Project-URL: Documentation, https://marcomq.github.io/mq-bridge/getting-started/library-quick-start.html
+Project-URL: Homepage, https://marcomq.github.io/mq-bridge/
 Project-URL: Issues, https://github.com/marcomq/mq-bridge/issues
 Project-URL: Repository, https://github.com/marcomq/mq-bridge
 

{mq_bridge_basic-0.4.17 → mq_bridge_basic-0.4.18}/README.md

@@ -81,6 +81,32 @@ mqb copy 'kafka://localhost:9092?topic=orders' 'nats://localhost:4222?subject=or
 ```
 
 
+## Capabilities at a glance
+
+Every row works from the Rust crate, the Python and Node.js packages, and the zero-code app; the Cargo feature is only needed when you build the Rust crate yourself. The linked [book](https://marcomq.github.io/mq-bridge/) documents the library as well as the app.
+
+| Capability | How (endpoint / middleware · Cargo feature) | Docs |
+| :--- | :--- | :--- |
+| **Message brokers**: Kafka, NATS (JetStream), RabbitMQ (AMQP), MQTT, Redis Streams, AWS SQS/SNS, IBM MQ, ZeroMQ | `kafka`, `nats`, `amqp`, `mqtt`, `redis_streams`, `aws`, `ibmmq`, `zeromq` endpoints · features `kafka`, `nats`, `amqp`, `mqtt`, `redis-streams`, `aws`, `ibm-mq`, `zeromq` | [Connectors](https://marcomq.github.io/mq-bridge/connectors/index.html) |
+| **Change Data Capture (CDC)**: stream Postgres or MongoDB changes to Kafka, files, a warehouse, … | `postgres_cdc` (logical replication, `pgoutput`) · `postgres-cdc`; `mongodb` with `consume: capture_all` / `capture_new` (change streams) · `mongodb` | [Postgres CDC tutorial](https://marcomq.github.io/mq-bridge/tutorials/postgres-cdc.html) · [MongoDB](https://marcomq.github.io/mq-bridge/connectors/mongodb.html) |
+| **SQL databases**: PostgreSQL, MySQL, MariaDB, SQLite as source (cursor polling) or sink (insert / upsert) | `sqlx` endpoint · `sqlx` | [PostgreSQL / MySQL / SQLite](https://marcomq.github.io/mq-bridge/connectors/postgres.html) · [Upserts](https://marcomq.github.io/mq-bridge/cookbook/upserts.html) |
+| **MongoDB** as source or sink | `mongodb` endpoint · `mongodb` | [MongoDB](https://marcomq.github.io/mq-bridge/connectors/mongodb.html) |
+| **ClickHouse** bulk insert and resumable reads | `clickhouse` endpoint · `clickhouse` | [ClickHouse](https://marcomq.github.io/mq-bridge/connectors/clickhouse.html) |
+| **Search engines** and other bulk document APIs (Meilisearch, Typesense, Elasticsearch, Qdrant, PostgREST / Supabase) | `http_bulk` endpoint · `http-bulk` | [HTTP bulk](https://marcomq.github.io/mq-bridge/connectors/http-bulk.html) |
+| **Object storage**: S3 (and S3-compatible, e.g. R2), GCS, Azure Blob, or a local directory, as **Parquet** or **JSONL**; **CSV** as input | `object_store` endpoint with `format: parquet \| raw \| json \| csv` · `object-store`, `parquet` | [Object storage](https://marcomq.github.io/mq-bridge/connectors/object-store.html) |
+| **Files**: CSV and JSONL, read and write | `file` endpoint · built in | [File](https://marcomq.github.io/mq-bridge/connectors/file.html) |
+| **Analytics / warehouses**: Snowflake, BigQuery, Databricks / Spark, Athena / Trino, DuckDB **via Parquet on S3/GCS/Azure**. These are not native connectors: mq-bridge writes the Parquet files, the warehouse loads or queries them | `object_store` with `format: parquet` · `parquet` | [Snowflake](https://marcomq.github.io/mq-bridge/cookbook/snowflake.html) · [BigQuery](https://marcomq.github.io/mq-bridge/cookbook/bigquery.html) · [Databricks / Spark](https://marcomq.github.io/mq-bridge/cookbook/databricks.html) · [Athena / Trino](https://marcomq.github.io/mq-bridge/cookbook/athena.html) · [DuckDB](https://marcomq.github.io/mq-bridge/cookbook/duckdb.html) |
+| **HTTP, gRPC, WebSocket** as server or client, including request/reply | `http`, `grpc`, `websocket` endpoints · `http`, `grpc`, `websocket` | [HTTP](https://marcomq.github.io/mq-bridge/connectors/http.html) · [gRPC](https://marcomq.github.io/mq-bridge/connectors/grpc.html) · [Request / reply](https://marcomq.github.io/mq-bridge/tutorials/request-reply.html) |
+| **Schema validation and type coercion** (JSON Schema subset, field mapping) | `transform` middleware · built in | [Transform & schema mapping](https://marcomq.github.io/mq-bridge/cookbook/transform.html) |
+| **Reliability**: retries, dead-letter queue, deduplication, rate limiting | `retry`, `dlq`, `deduplication`, `limiter` middleware · `dedup` for deduplication | [Middleware reference](docs/REFERENCE.md) · [Delivery guarantees](docs/DELIVERY.md) |
+| **Broker-free tests**: run a route or handler against in-memory channels, no Docker | `memory` endpoint · built in | [Embed the library](https://marcomq.github.io/mq-bridge/tutorials/embedding.html) |
+| **ETL without code** | [`mq-bridge-app`](apps/mq-bridge-app): `mqb copy <from> <to>`, YAML routes, desktop UI, MCP server | [Quick start: `mqb copy`](https://marcomq.github.io/mq-bridge/quick-start.html) |
+| **More systems through plugins**: Apache Pulsar, Meilisearch, and Redpanda Connect's inputs, outputs and processors | native plugins · `plugin` | [Plugin endpoints](https://marcomq.github.io/mq-bridge/reference/endpoints.html#plugin-endpoints) · [Connect plugin](https://marcomq.github.io/mq-bridge/connectors/connect.html): [inputs](https://marcomq.github.io/mq-bridge/connectors/connect-inputs.html), [outputs](https://marcomq.github.io/mq-bridge/connectors/connect-outputs.html), [processors](https://marcomq.github.io/mq-bridge/connectors/connect-processors.html) |
+
+The Connect plugin links only Redpanda Connect components free of Redpanda Community License code (bundled code is Apache-2.0 and MIT), so some upstream components, such as its Kafka, AWS and Snowflake ones, are not included; see [what is not included](https://marcomq.github.io/mq-bridge/connectors/connect.html#what-is-not-included).
+
+For AI agents: [`llms.txt`](llms.txt) is a curated index of the documentation.
+
 ## Benchmarks
 
 Throughput is tracked continuously on the public [benchmark dashboard](https://marcomq.github.io/mq-bridge/dev/bench/). Like-for-like ETL comparisons, measured through the zero-code [`mq-bridge-app`](apps/mq-bridge-app):