mq-bridge-py-basic 0.4.6__tar.gz → 0.4.8__tar.gz

{mq_bridge_py_basic-0.4.6 → mq_bridge_py_basic-0.4.8}/.github/workflows/ci.yml

@@ -42,6 +42,11 @@ jobs:
       - name: Run clippy
         run: cargo clippy --all-targets --all-features -- -D warnings
 
+      # Not a workspace default member, so nothing else builds it. It pins the
+      # exhaustive `GrpcConfig` literal a downstream crate would write.
+      - name: Check downstream config-compat fixture
+        run: cargo check -p grpc-config-compat
+
   # Every feature subset still gets its own `cargo check`, but sequentially in one
   # job: the matrix cost 8 runner slots at the moment integration-build's dependents
   # were starving, and none of this is on the critical path.
@@ -87,6 +92,15 @@ jobs:
       
       - name: Cache cargo build
         uses: Swatinem/rust-cache@v2
+
+      # setup-python provides a non-PEP-668 interpreter; the system python3 on
+      # ubuntu-latest is externally managed and rejects a plain pip install.
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install Python gRPC compatibility tools
+        run: python3 -m pip install grpcio-tools
       
       - name: Run unit tests
         run: cargo test --lib --features=full

{mq_bridge_py_basic-0.4.6 → mq_bridge_py_basic-0.4.8}/CHANGELOG.md

@@ -2,6 +2,193 @@
 
 All notable changes to `mq-bridge`. Newest first.
 
+## 0.4.8
+
+### Added
+
+- **Dynamic gRPC sinks.** The descriptor keys now work on a route's `output` too: a unary method
+  makes one call per message and returns the reply as its response, a client-streaming method
+  streams a batch into one call. Server-streaming and bidirectional methods, and `request`, are
+  rejected there. Status codes that mean "never going to succeed" are non-retryable, so such a
+  message is dead-lettered instead of replayed.
+
+- **gRPC server reflection, on both sides.** `reflection: true` discovers descriptors from the
+  remote server instead of a descriptor file, and server mode hosts reflection v1 and v1alpha,
+  so `grpcurl` can introspect an embedded server.
+
+- **Metadata and credentials for dynamic gRPC calls.** `metadata`, `binary_metadata`,
+  `bearer_token`, and `api_key`/`api_key_name`, sent on the reflection call as well. They never
+  appear in errors or logs, and are startup errors on Bridge and server mode rather than a silent
+  unauthenticated connection — authenticate that with TLS client certificates.
+
+- **Separate gRPC deadlines.** `connect_timeout_ms`, `request_timeout_ms`,
+  `idle_stream_timeout_ms` (retryable: the route reconnects), and `overall_timeout_ms` (terminal:
+  reconnecting would reset the cap). All default to disabled.
+
+- **`descriptor_set_bytes`** passes a compiled gRPC `FileDescriptorSet` straight from embedded
+  Rust callers, with no temporary file.
+
+- **Provenance on dynamic gRPC responses.** A deterministic id per response, plus `grpc.service`,
+  `grpc.method`, `grpc.response_index`, and `grpc.ack_guarantee=none` metadata.
+
+- **[docs/GRPC.md](docs/GRPC.md).** The full gRPC guide, including why a generic
+  descriptor-driven server is intentionally not implemented. The Bridge contract moved to
+  `src/endpoints/grpc/proto/mqbridge/bridge.proto`, unchanged on the wire, and CI now checks a
+  generated Python client against server mode.
+
+### Changed
+
+- **Adding the dedicated gRPC deadline fields is a breaking Rust API change for exhaustive
+  `GrpcConfig` struct literals.** Downstream Rust callers must specify the new fields or use
+  `..Default::default()` / `GrpcConfig::new`. A downstream compile fixture now locks the complete
+  literal shape so future additions cannot make this break silently.
+
+- **gRPC `bearer_token` and `api_key` now require an `https://` endpoint.** A credential sent over
+  plaintext h2c is rejected rather than put on the wire in the clear. Deployments that terminated
+  TLS at a sidecar and spoke plaintext gRPC to it must point the endpoint at `https://`.
+
+- **The gRPC endpoint's `timeout_ms` and `server_streaming` are deprecated but still accepted.**
+  `timeout_ms` stays the fallback for connection and request setup; it no longer bounds a dynamic
+  stream, which needs the dedicated keys. RPC shape comes from the descriptor, so a disagreeing
+  `server_streaming` warns instead of failing startup.
+
+- **gRPC descriptor keys on an `output` select the dynamic publisher.** They were previously
+  ignored there and the Bridge publisher was used.
+
+- **gRPC failures preserve code, message, and trailing metadata** in a `GrpcStatusError`. Its
+  `Display`/`Debug` omit trailer values so a peer's credentials cannot leak into logs.
+
+- **Filtering and structural forwarding retain bulk writes where possible.** An input `filter`
+  now reads additional full source batches after dropping rows until it refills the requested
+  batch size; a naturally short source batch remains a flush boundary so live routes do not
+  wait indefinitely. A `request` still performs its request/reply calls individually and
+  concurrently, unless its `to` endpoint requires ordered publishing; ordered requests run one
+  at a time. Results are restored to input order and responses plus error fallbacks are sent to
+  `forward_to` in one `send_batch` call. Batch-capable destinations such as MongoDB can
+  therefore continue using `insert_many` through both paths.
+
+- **Buffered concurrent routes warn that destination order is not guaranteed.** Buffering
+  preserves order within each batch, but with route `concurrency > 1` separate destination
+  writes may complete out of source order. Route validation now recommends `concurrency: 1`
+  when destination order matters.
+
+## 0.4.7
+
+### Changed
+
+- **The startup failure message distinguishes its two causes.** A route that never became ready
+  reports `failed to start: did not become ready within {n}ms`; a route whose task ended before
+  signalling ready reports `failed to start: ended before it signalled ready (outcome: ...)`.
+  The second case returns immediately and never involved the timeout, so reporting it as one
+  sent the reader looking for a stall that never happened. Both now carry the cause the
+  reconnect loop recorded, which the old message dropped. Code matching on the old
+  `failed to start within {n}ms or encountered an error` text needs updating; the substring
+  `failed to start` is unchanged.
+
+### Fixed
+
+- **A route that ran to completion is no longer reported as one that failed to start.** The run
+  task signals ready before its consume loop, so a drain over a small source can reach its
+  terminal while that signal is still buffered — leaving the reconnect loop's `select!` with two
+  ready branches and free to take the terminal one, dropping the startup channel unread.
+  `Route::run()` then returned an error for a drain that had already written its output, and
+  `mqb copy --drain` exited non-zero on a successful copy. Both terminal arms now forward a
+  ready signal that was already emitted.
+
+- **A completed route reports the messages it dropped even after an earlier reconnect failure.**
+  The drop report was suppressed whenever `EndpointStatus::error` was already set, including by a
+  transient failure the route recovered from. Only a `Failed` route's cause now outranks it.
+
+- **A plugin endpoint that rejects its config no longer reconnects forever.** `create_consumer`
+  and `create_publisher` flattened the plugin's status into a bare message, and the reconnect
+  loop decides by downcasting to `ConsumerError`/`PublisherError` — so every failure to open
+  looked like a connection fault worth retrying. A plugin answering `MQB_ERR_INVALID_CONFIG`,
+  `MQB_ERR_UNSUPPORTED`, `MQB_ERR_PERMANENT` or `MQB_ERR_PANIC` now stops the route with that
+  cause; `MQB_ERR_CONNECTION` and `MQB_ERR_RETRYABLE` still get the reconnect loop.
+
+## 0.4.6
+
+### Added
+
+- **`filter` middleware.** Keeps only the messages matching an expression and drops the rest,
+  on the input or the output: `- filter: "amount > 100"`. Payload fields are read by bare name
+  (`amount`, `order.status`), metadata under `meta.` and always as text, so a numeric comparison
+  needs a cast (`number(meta.http_status_code) >= 400`); `&&`/`||` and `and`/`or` both work. A
+  metadata-only expression never parses the payload. **Prefer the input** — a filter there drops
+  the message before the rest of the route touches it and acks it at the source. Requires the
+  new `filter` feature (it pulls the `zen-expression` engine), which is part of `middleware` and
+  `full` but **not** of `portable`.
+
+- **Predicate mode for `switch`.** `when: [{ if, to }, ...]` routes on the same expression
+  language, first match wins, so overlapping thresholds are safe to write and a payload-derived
+  key no longer has to be promoted into metadata first. A `switch` uses exactly one of `when` or
+  `metadata_key`/`cases`; naming both or neither is a startup error, as is a `when` list in a
+  build without the `filter` feature. Value lookup stays the cheaper mode when the key is
+  already in metadata.
+
+- **`expression` on `transform`.** A Zen expression producing the output document, applied after
+  `mapping` and before `schema` — for values that have to be calculated rather than copied.
+  Needs the `zen` feature.
+
+- **A `fanout` can reply.** If one branch produces a response — a `response` or `static` leg, or
+  a `request` whose `forward_to` replies — that response goes back to the caller, so a
+  request/reply input can fan a message out and still answer. Only one branch may answer: if
+  several do, the first in list order wins, and the route warns once rather than on every
+  message. A branch that *fails* nacks the whole fan-out, so the caller gets a `500` rather than
+  an answer that hides a lost message; opt out per branch with a `request` whose `forward_to`
+  does not reply, or a `dlq` on a plain branch.
+
+- **`pass_through_status` on the `http` endpoint.** On a **publisher**, a non-2xx response
+  becomes response data instead of a publisher error, so a route can branch on
+  `http_status_code` with a `switch` rather than treating every 404 as a failed send. Transport
+  failures and unreadable responses are still errors. On a **consumer**, once every output sink
+  opts in, a transient sink failure answers the request with `502` instead of stopping and
+  reconnecting a non-streaming request/reply route. Composite outputs such as `fanout` need the
+  flag on every leaf sink — a mixed output keeps the normal stop-and-reconnect policy — and
+  streamable HTTP inputs keep their protocol-specific error frames.
+
+### Changed
+
+- **A middleware that empties a batch no longer acknowledges ahead of the route.** `filter` and
+  `deduplication` can filter a batch down to nothing, and acking that straight away jumps ahead
+  of batches the route is still writing on a source with cumulative acks — the ordered sequencer
+  only sees the commits `receive_batch` returns, so a crash in that window lost them. On those
+  sources the emptied batch's commit is now held and runs from inside the next retained batch's
+  commit, which the sequencer does order.
+
+- **An `object_store` sink on `name_by: auto` switches to `write_time` names when a row-dropping
+  middleware is on the route.** A source-range name covers one *contiguous* run of positions, so
+  a batch with holes punched in it would be written as one object per surviving run — a filter
+  keeping 80% of rows turns one upload into roughly a hundred. The route logs one line at
+  startup saying it made the switch. This covers `filter`, `deduplication`, `weak_join`,
+  `transform` with `on_error: reject`, and a `switch` in `when` mode with no `default`. Set
+  `name_by: source_position` explicitly to keep replay-safe names and accept the fragmentation.
+
+- **`filter` expressions and a `switch`'s `when` cases are compiled when the config is loaded.**
+  An invalid expression is now a startup error naming the expression, instead of a per-message
+  failure that only surfaces once traffic arrives.
+
+### Fixed
+
+- **`switch` forwards connect and disconnect hooks to its destinations**, `cases` and `default`
+  alike, and one failing teardown no longer skips the rest.
+
+- **A failed `request` whose `forward_to` answers the caller surfaces the error** instead of
+  echoing the original message back as a success. The route then nacks (HTTP `500`) and a
+  `retry` or `dlq` on the endpoint sees the failure as usual. Forwarding to a plain sink still
+  acks, so routing a status with a `switch` is unchanged.
+
+- **Concurrent MongoDB consumers no longer claim documents out from under each other.** Two
+  consumers polling within the same second write the same `locked_until`, which is all the claim
+  query had to work with, so one poll could read back documents another had just taken. Each
+  claim now carries a unique `claim_token`, and the read-back, unlock, ack and delete are all
+  scoped to it — an expired lease that was re-claimed elsewhere is no longer deleted from under
+  its new owner.
+
+## 0.4.5
+
+No library changes: CI workflow timeouts and a version bump.
+
 ## 0.4.4
 
 ### Breaking