comq 0.18.0 → 0.20.0

package/discussions/poison-message-handling.md

@@ -6,9 +6,8 @@
 > became shared and delay-named, the delay became configurable, and message provenance moved to
 > where it survives a retry.
 >
-> **Parts 1 and 2 are implemented.** Part 3 — the `Retry` / `Park` verdict API — is decided but
-> not yet scheduled; see the end. Text below is written as a proposal because that is what it
-> was; it describes what the code now does, except where Part 3 says otherwise.
+> **All three parts are implemented.** Text below is written as a proposal because that is what
+> it was; it describes what the code now does.
 
 When a consumer callback rejects, comq is supposed to retry the message a few times and then
 give up on it. What it actually does is kill the process, and take every other consumer in that
@@ -1111,10 +1110,15 @@ Nothing blocking. The three questions this document opened are resolved:
 - **The noun** — `Park` / `comq.parked.<queue>`. See Part 1.
 - **`Park` on an `io.reply` producer** — rejected as a category error, treated at runtime as a
   bare rejection plus a diagnostic. See Part 3.
-- **Whether Part 3 is built** — **decided, not yet scheduled.** Parts 1 and 2 are the emergency
-  and do not depend on it; a consumer of comq gets retry-then-park by upgrading and writing no
-  code. But the parking queue takes its noun from a verdict class that would not otherwise
-  exist, and without verdicts a message a contract has already refused costs every attempt —
-  two minutes at the event default — and then lands in the parking queue beside the
-  messages that genuinely failed. "Decided, not yet scheduled" rather than "open", because
-  people read this deciding whether to build on it.
+- **Whether Part 3 is built** — built. Parts 1 and 2 shipped first as the emergency, and a
+  consumer of comq got retry-then-park by upgrading and writing no code; the verdicts followed
+  as an additive minor.
+
+One thing the design did not settle, found while building it: it placed the `Park`-from-a-Producer
+rejection in `#failed`, which cannot implement it — `Channel` takes a topology but not its type
+([channel.js:64](../source/channel.js#L64)), so it cannot tell the request channel from the event
+one. It lives in `#getRequestConsumer` ([io.js:297](../source/io.js#L297)) instead, which knows by
+construction that it is wrapping a `Producer`. And rather than emit a diagnostic nobody may be
+subscribed to, it rethrows with an explanation, which reaches the parked message as its
+`x-comq-reason` and both existing diagnostics as their exception — louder, and no new event name
+to document.

package/docs/headers.md

@@ -28,6 +28,7 @@ the message properties.
 | `x-comq-key` | a retried or parked message | The routing key it was originally published with. |
 | `x-comq-queue` | a parked message | The queue it was consumed from. |
 | `x-comq-reason` | a parked message | The `message` of the exception that ended its attempts. |
+| `x-comq-cause` | a parked message | The `message` of that exception's `cause`, when it has one — a verdict is usually thrown with the failure that prompted it. |
 | `x-comq-at` | a parked message | When it was parked, as `Date.now()`. |
 
 `x-comq-exchange` and `x-comq-key` are recorded on the first failure rather than read at parking

package/features/events.poison.feature

@@ -40,3 +40,14 @@ Feature: Poison events
     When an event is emitted to the `poison_open` exchange
     And after 1000ms
     Then `bystander` has received 1 event
+
+  Scenario: A consumer refusing a message parks it at once
+
+    A `Park` says the consumer will never process this message, so climbing the
+    ladder would only delay the inevitable.
+
+    Given that events from the `poison_refused` exchange are refused as poison
+    When an event is emitted to the `poison_refused` exchange
+    Then the event is attempted 1 time
+    And the message is parked
+    And the parked message is kept, and says it came from the `poison_refused` exchange

package/features/rpc.call.feature

@@ -0,0 +1,71 @@
+Feature: Calls under a key
+
+  A call reaches the one connection holding its key. A call under a key nobody holds is refused
+  at once, and a holder that stops withdraws its key before it finishes what it holds.
+
+  Background:
+    Given an active connection to the broker
+
+  Scenario: Answered by the holder of the key
+    Given a holder answering `sums` under the `a` key:
+      """
+      ({ a, b }) => a + b
+      """
+    When the consumer calls `sums` under the `a` key with:
+      """yaml
+      a: 1
+      b: 2
+      """
+    Then the consumer receives the reply:
+      """yaml
+      3
+      """
+
+  Scenario: Answered by the holder of that key only
+    Given a holder answering `names` under the `a` key:
+      """
+      () => 'a'
+      """
+    And a holder answering `names` under the `b` key:
+      """
+      () => 'b'
+      """
+    When the consumer calls `names` under the `b` key
+    Then the consumer receives the reply:
+      """yaml
+      b
+      """
+
+  Scenario: Refused when nobody holds the key
+    Given a holder answering `names` under the `a` key:
+      """
+      () => 'a'
+      """
+    When the consumer calls `names` under the `c` key
+    Then the call is refused as unroutable
+
+  Scenario: A sealed holder answers what it holds and refuses what follows
+    Given a holder on another connection answering `slow` under the `a` key in 500ms
+    When the consumer calls `slow` under the `a` key
+    And after 100ms
+    And the holder is sealed
+    Then the consumer receives the reply
+    When the consumer calls `slow` under the `a` key
+    Then the call is refused as unroutable
+
+  Scenario: A second holder waits for the key
+    Given a holder on another connection answering `echo` under the `a` key
+    When a second holder on another connection holds `echo` under the `a` key
+    Then the second holder finds the key taken
+    When the first holder disconnects
+    Then the second holder holds the key
+    When the consumer calls `echo` under the `a` key
+    Then the consumer receives the reply
+
+  Scenario: A caller stops waiting for a holder that crashed
+    Given a holder never answering `void` under the `a` key
+    When the consumer calls `void` under the `a` key with a 1000ms timeout
+    And after 100ms
+    And the silent holder crashes
+    Then the consumer stops waiting
+    And a call to `void` under the `a` key is refused within 10 seconds

package/features/rpc.call.recovery.feature

@@ -0,0 +1,75 @@
+@heavy
+Feature: Calls under a key across failures
+
+  A holder that loses its connection holds its key again once the connection is restored, and a
+  caller's timeout ends its wait whatever the broker is doing.
+
+  Scenario: A holder holds its key again after the broker restarts
+    Given an active connection to the broker
+    And a holder on another connection answering `echo` under the `a` key
+    When the broker has crashed
+    And the broker is up
+    Then a call to `echo` under the `a` key is answered within 60 seconds
+
+  Scenario: A holder holds its key again once the broker lets its silent connection go
+
+    The holder reconnects while the broker still holds the connection that went silent, and with
+    it the key. The holder finds the key taken until the broker lets that connection go, and
+    claims it then.
+
+    Given watchdog interval is set to 2000ms with 60s AMQP heartbeat
+    And a network that can go silent
+    And an active connection to the broker
+    And a holder answering `echo` under the `a` key:
+      """
+      (payload) => payload
+      """
+    When the network goes silent
+    Then the connection is lost within 10 seconds
+    And the holder finds its key taken while the broker holds the silent connection
+    When the silent connection is let go
+    Then a call to `echo` under the `a` key from another connection is answered within 60 seconds
+
+  Scenario: A call in flight when the broker crashes ends
+    Given an active connection to the broker
+    And a holder on another connection answering `slow` under the `a` key in 2000ms
+    When the consumer calls `slow` under the `a` key
+    And after 200ms
+    And the broker has crashed
+    And the broker is up
+    Then the call ends
+
+  Scenario: A caller stops waiting for a request at its timeout while the broker is down
+    Given an active connection to the broker
+    And a producer replying `echo` queue
+    When the broker has crashed
+    And the consumer sends a request to the `echo` queue with a 1000ms timeout
+    Then the consumer stops waiting within 5 seconds
+
+  Scenario: A caller stops waiting for a call at its timeout while the broker is down
+    Given an active connection to the broker
+    And a holder on another connection answering `echo` under the `a` key
+    When the broker has crashed
+    And the consumer calls `echo` under the `a` key with a 1000ms timeout
+    Then the consumer stops waiting within 5 seconds
+
+  Scenario: A request re-sent after the broker restarts expires at its caller's deadline
+
+    The request is lost with the broker and sent again once the connection is restored, carrying
+    the time its caller has left.
+
+    Given an active connection to the broker
+    When the consumer sends a request to the `resent` queue with a 30000ms timeout
+    And after 200ms
+    And the broker has crashed
+    And the broker is up
+    Then the consumer stops waiting within 40 seconds
+    When a producer counting requests to the `resent` queue
+    Then the producer receives nothing
+
+  Scenario: A call reaches its holder while one of the brokers is down
+    Given an active sharded connection
+    And a holder on another connection answering `echo` under the `a` key
+    When one of the brokers has crashed
+    And the consumer calls `echo` under the `a` key 20 times
+    Then every call is answered

package/features/rpc.call.shards.feature

@@ -0,0 +1,24 @@
+Feature: Calls under a key over a sharded connection
+
+  A call is published to one shard, and a shard where nobody holds the key returns it: it is
+  published on the next one, and refused once every shard has returned it.
+
+  Background:
+    Given an active sharded connection
+
+  Scenario: Answered by a holder on one broker only
+    Given a holder connected to broker 0 answering `echo` under the `a` key
+    When the consumer calls `echo` under the `a` key 20 times
+    Then every call is answered
+
+  Scenario: Refused when nobody holds the key on any broker
+    Given a holder connected to broker 0 answering `echo` under the `a` key
+    When the consumer calls `echo` under the `b` key
+    Then the call is refused as unroutable
+
+  Scenario: A key taken on one broker is held on the other, and on both once it is let go
+    Given a holder connected to broker 0 answering `whom` under the `a` key as `first`
+    When a second holder on another connection holds `whom` under the `a` key as `second`
+    Then the second holder finds the key taken
+    When the first holder disconnects
+    Then every call to `whom` under the `a` key is answered by `second` within 30 seconds

package/features/rpc.parked.feature

@@ -0,0 +1,20 @@
+Feature: Parked requests
+
+  Background:
+    Given an active connection to the broker
+
+  Scenario: A parked request can still be answered
+
+    A caller that sets no timeout is certain of an eventual answer for as long as it is
+    alive, and parking is what keeps that true when no consumer can produce one: the
+    request is kept, along with who was asking. Somebody else can answer it, and the
+    caller — still waiting, still holding an exclusive reply queue — receives it.
+
+    A caller that sets a timeout trades that for an end to the wait: once it stops
+    waiting, a parked request is addressed to a caller that has gone.
+
+    Given a producer failing every request to the `unanswerable` queue
+    When a request is sent to the `unanswerable` queue
+    Then the request is parked
+    When the parked request from the `unanswerable` queue is answered by hand
+    Then the caller receives the reply

package/features/rpc.timeout.feature

@@ -0,0 +1,19 @@
+Feature: Request timeout
+
+  A caller may say how long it waits for a reply. A request nobody took by then is dropped by the
+  broker; one already taken is still processed, and its reply is discarded.
+
+  Background:
+    Given an active connection to the broker
+
+  Scenario: A caller stops waiting at its timeout
+    Given a producer replying `slow` queue in 1000ms
+    When the consumer sends a request to the `slow` queue with a 200ms timeout
+    Then the consumer stops waiting
+
+  Scenario: A request nobody took in time is never processed
+    When the consumer sends a request to the `later` queue with a 200ms timeout
+    Then the consumer stops waiting
+    When after 300ms
+    And a producer counting requests to the `later` queue
+    Then the producer receives nothing