@mark1russell7/machines 0.3.14 → 0.3.15

package/README.md

@@ -50,8 +50,10 @@ The algebra reads these documents. It gives the closed form of each timer, the f
 | `SUPAVISOR_SERVER` | `supavisor-server` | Supavisor 2.9.13 in transaction mode: one server connection of the pool of a tenant | proxy | 10 implemented, 1 observed |
 | `AMQPLIB` | `amqplib` | amqplib 2.2.0: one connection to an AMQP 0-9-1 broker, its heartbeat and an RPC of its channel | driver | 18 implemented, 1 documented |
 | `RABBITMQ_CONNECTION` | `rabbitmq-connection` | RabbitMQ 4.3.6: one connection of AMQP 0-9-1 on the broker | server | 14 implemented, 1 documented |
+| `ATEN_DETECTOR` | `aten` | aten 0.6.0: one node that the failure detector of another node of RabbitMQ watches | server | 7 implemented |
+| `RA_MEMBER` | `ra-member` | ra 3.1.10: one member of the Raft cluster of a quorum queue of RabbitMQ 4.3.6 | server | 25 implemented |
 
-Twenty-one documents operate on Linux TCP (`over`). Linux TCP, conntrack, the Sentinel and the replication of mongod operate on no other document. The `conditions` of each document give the range where it holds, for example a range of kernels.
+Twenty-one documents operate on Linux TCP (`over`). Linux TCP, conntrack, the Sentinel, the replication of mongod, aten and the member of Ra operate on no other document. The `conditions` of each document give the range where it holds, for example a range of kernels.
 
 The package also gives these parts:
 
@@ -97,6 +99,12 @@ A change to the version of a system changes its sources too. Each fact of the do
 
 In this repository, a snapshot of the predictions of the cells (`packages/cells/src/__snapshots__/predictions.json`) shows each prediction that a change to a document moves.
 
+### 0.3.15
+
+- **The cluster of a quorum queue of RabbitMQ 4.3.6.** Two new documents, each from the source in the release of RabbitMQ:
+  - `aten`: the failure detector between the nodes, aten 0.6.0. Each node sends a heartbeat to each other node each 100 ms. The detector polls each 5 s in RabbitMQ (1 s at the defaults of aten). It reports a node down when the time since its last heartbeat is 1.5 times its gaps. Thus a silent node is down 150 ms to approximately 5.2 s after its last heartbeat.
+  - `ra-member`: one member of the Raft cluster of a quorum queue, ra 3.1.10. When aten reports the node of the leader down, a follower waits 5 s to 6 s, and then it asks for pre-votes. A majority makes a candidate, and the votes of a majority make the leader. A leader ignores aten, thus it stays the leader in a minority and commits nothing.
+
 ### 0.3.14
 
 - **amqplib 2.2.0 and RabbitMQ 4.3.6.** Two new documents, each from the source at its tag:

package/build/index.d.ts

@@ -355,6 +355,47 @@ export declare const PROCEDURES_WS_SERVER: Machine;
  */
 export declare const AMQPLIB: Machine;
 //#endregion
+//#region src/rabbit/aten.d.ts
+/**
+ * This document is aten 0.6.0, the failure detector between the nodes of RabbitMQ 4.3.6. It gives one
+ * node that another node watches: up or down, as the detector of the watching node sees it.
+ *
+ * Each node sends a heartbeat to each connected node each 100 ms. The sink of the watching node keeps
+ * the gaps between the heartbeats of each node. Its detector polls each 5000 ms: aten polls each 1000 ms
+ * by its own default, and RabbitMQ sets 5000 ms, for fewer false elections.
+ *
+ * At a poll, the probability of a failure is a share of the gaps. A gap counts when 1.5 times the gap
+ * is not longer than the time since the last heartbeat. The node is down when that share reaches 0.99.
+ * With gaps of approximately 100 ms, that is approximately 150 ms after the last heartbeat, at the first
+ * poll after it. Thus a silent node is down 150 ms to approximately 5.2 s after its last heartbeat,
+ * long before the tick of the distribution.
+ *
+ * A closed connection of the distribution (a stopped node) removes the data of the node from the
+ * sink. The next poll then reports the node down.
+ */
+export declare const ATEN_DETECTOR: Machine;
+//#endregion
+//#region src/rabbit/ra-member.d.ts
+/**
+ * This document is ra 3.1.10, for one member of the Raft cluster of a quorum queue of RabbitMQ 4.3.6.
+ * It gives the roles of the member, the election timers, and the change of leader.
+ *
+ * A follower watches the node of its leader with aten. When aten reports that node down, the follower
+ * arms a long election timeout. The timeout is the poll interval of aten, 5 s in RabbitMQ, and a random
+ * part up to 1 s. A valid message of the leader cancels it. When the timeout fires, the follower asks for pre-votes in
+ * its term.
+ *
+ * A member grants a pre-vote when the log of the candidate is not behind its own. A member that is
+ * still a follower of a live leader grants it too. A majority of pre-votes makes a candidate. The
+ * candidate starts a new term and asks for votes, and a majority makes it the leader. The new leader
+ * of the queue tells each publisher of the queue. Then the channel of each publisher sends its pending
+ * publishes again to the new leader.
+ *
+ * A leader ignores the events of aten. Thus a leader that loses the majority stays the leader in its
+ * minority, and it commits nothing, until it sees a higher term.
+ */
+export declare const RA_MEMBER: Machine;
+//#endregion
 //#region src/rabbit/rabbitmq-connection.d.ts
 /**
  * This document is RabbitMQ 4.3.6, for one connection of AMQP 0-9-1 on the broker. It gives the reader of

package/build/index.js

@@ -11489,6 +11489,595 @@ const AMQPLIB = machineSchema.parse({
 	}
 });
 //#endregion
+//#region src/rabbit/aten.ts
+const ATEN = "aten 0.6.0";
+/**
+* This document is aten 0.6.0, the failure detector between the nodes of RabbitMQ 4.3.6. It gives one
+* node that another node watches: up or down, as the detector of the watching node sees it.
+*
+* Each node sends a heartbeat to each connected node each 100 ms. The sink of the watching node keeps
+* the gaps between the heartbeats of each node. Its detector polls each 5000 ms: aten polls each 1000 ms
+* by its own default, and RabbitMQ sets 5000 ms, for fewer false elections.
+*
+* At a poll, the probability of a failure is a share of the gaps. A gap counts when 1.5 times the gap
+* is not longer than the time since the last heartbeat. The node is down when that share reaches 0.99.
+* With gaps of approximately 100 ms, that is approximately 150 ms after the last heartbeat, at the first
+* poll after it. Thus a silent node is down 150 ms to approximately 5.2 s after its last heartbeat,
+* long before the tick of the distribution.
+*
+* A closed connection of the distribution (a stopped node) removes the data of the node from the
+* sink. The next poll then reports the node down.
+*/
+const ATEN_DETECTOR = machineSchema.parse({
+	format: "mark1russell7.machine/1",
+	id: "aten",
+	title: "aten 0.6.0: one node that the failure detector of another node of RabbitMQ watches, up or down",
+	layer: "server",
+	version: "0.6.0",
+	conditions: {
+		aten: "0.6.0",
+		ra: "3.1.10",
+		rabbitmq: "4.3.6",
+		nodes: "two nodes of one cluster, connected by the Erlang distribution"
+	},
+	sources: [
+		{
+			ref: `${ATEN} src/aten.app.src: the defaults poll_interval 1000, scaling_factor 1.5, heartbeat_interval 100, detection_threshold 0.99`,
+			rung: "implemented"
+		},
+		{
+			ref: "rabbitmq v4.3.6 deps/rabbitmq_prelaunch/src/rabbit_prelaunch_conf.erl:112-120 RabbitMQ sets the poll interval of aten to 5000 ms, for fewer false elections of quorum queues",
+			rung: "implemented"
+		},
+		{
+			ref: "run 39218896659 of github.com/mark1russell7/systems: after a freeze and a black hole of the node of the leader, the new leader came 6.1 s to 10.4 s later, and not 1 s to 4 s",
+			rung: "observed"
+		},
+		{
+			ref: `${ATEN} src/aten_emitter.erl:49-54 and :95-98 the emitter arms its next send heartbeat_interval after it sends to each node, :63-82 it sends to each node in nodes() with noconnect and nosuspend`,
+			rung: "implemented"
+		},
+		{
+			ref: `${ATEN} src/aten_sink.erl:69-79 a heartbeat adds a sample to the state of its node, :81-87 nodedown removes the state of the node`,
+			rung: "implemented"
+		},
+		{
+			ref: `${ATEN} src/aten_detect.erl:17 a window of 1000 samples, :44-52 a sample is the gap since the heartbeat before it, :54-68 the probability is the share of the gaps S with S × factor ≤ the time since the last heartbeat`,
+			rung: "implemented"
+		},
+		{
+			ref: `${ATEN} src/aten_detector.erl:53-59 the threshold and the poll interval, :105-113 each poll arms the next poll first, then reads the probabilities, :148-186 a node goes down when its probability crosses the threshold between two polls, or when it leaves the sink`,
+			rung: "implemented"
+		},
+		{
+			ref: `${ATEN} src/aten_detector.erl:134-146 the detector sends {node_event, Node, down} or up to each process that watches the node`,
+			rung: "implemented"
+		},
+		{
+			ref: "rabbitmq v4.3.6 deps/rabbit/priv/schema/rabbit.schema:2540-2589 raft.adaptive_failure_detector.poll_interval, scaling_factor, heartbeat_interval and detection_threshold map to aten",
+			rung: "implemented"
+		}
+	],
+	states: {
+		up: {
+			initial: true,
+			note: "the probability of the node is below the threshold at the last poll"
+		},
+		down: { note: "{node_event, Node, down}: the probability crossed the threshold, or the node left the sink" }
+	},
+	events: {
+		"rx.heartbeat": {
+			kind: "rx",
+			note: "a heartbeat of the node reaches the sink of the watching node"
+		},
+		"rx.nodedown": {
+			kind: "rx",
+			note: "the connection of the distribution to the node closed: the sink removes the node"
+		},
+		"timer.detect": {
+			kind: "timer",
+			note: "the first poll at which the share of the gaps reaches the threshold"
+		},
+		"timer.left": {
+			kind: "timer",
+			note: "the first poll after the node left the sink"
+		}
+	},
+	knobs: {
+		heartbeat_interval: {
+			key: "raft.adaptive_failure_detector.heartbeat_interval (aten heartbeat_interval)",
+			scope: "node",
+			unit: "ms",
+			default: 100,
+			note: "aten.app.src"
+		},
+		poll_interval: {
+			key: "raft.adaptive_failure_detector.poll_interval (aten poll_interval)",
+			scope: "node",
+			unit: "ms",
+			default: 5e3,
+			note: "RabbitMQ sets 5000 (rabbit_prelaunch_conf.erl:112-120). The default of aten itself is 1000 (aten.app.src)"
+		},
+		scaling_factor: {
+			key: "raft.adaptive_failure_detector.scaling_factor (aten scaling_factor)",
+			scope: "node",
+			unit: "count",
+			default: 1.5,
+			note: "aten.app.src"
+		},
+		detection_threshold: {
+			key: "raft.adaptive_failure_detector.detection_threshold (aten detection_threshold)",
+			scope: "node",
+			unit: "count",
+			default: .99,
+			note: "aten.app.src: the share of the gaps that the time since the last heartbeat must reach"
+		},
+		gap_slack: {
+			key: "how much longer than heartbeat_interval the 99th percentile gap is",
+			scope: "node",
+			unit: "ms",
+			default: 10,
+			note: "an assumption of this document, not a value of aten: the emitter arms its next send after its sends (aten_emitter.erl:95-98), and the BEAM schedules it a little late"
+		},
+		slack: {
+			key: "the lateness of a timer of the BEAM for one step",
+			scope: "node",
+			unit: "ms",
+			default: 10,
+			note: "an assumption of this document, not a value of aten: a send_after that fires late"
+		}
+	},
+	timers: {
+		detect: {
+			duration: {
+				expr: "scaling_factor * heartbeat_interval",
+				unit: "ms"
+			},
+			lateness: {
+				expr: "scaling_factor * gap_slack + poll_interval + slack",
+				unit: "ms"
+			},
+			arms: ["rx.heartbeat"],
+			fires: "timer.detect",
+			note: "150 ms after the last heartbeat at the defaults, and up to one poll of 5 s later"
+		},
+		nodedown_poll: {
+			duration: {
+				const: 0,
+				unit: "ms"
+			},
+			lateness: {
+				expr: "poll_interval + slack",
+				unit: "ms"
+			},
+			arms: ["rx.nodedown"],
+			fires: "timer.left"
+		}
+	},
+	realizes: {
+		"systems.mechanism.heartbeat-detection-time": { timer: "detect" },
+		"systems.mechanism.heartbeat-interval": {
+			duration: {
+				knob: "heartbeat_interval",
+				unit: "ms"
+			},
+			lateness: {
+				knob: "gap_slack",
+				unit: "ms"
+			},
+			note: "each node sends a heartbeat to each connected node at this interval (aten_emitter.erl:95-98)"
+		}
+	},
+	transitions: [
+		{
+			id: "aten.heartbeat",
+			from: "up",
+			to: "up",
+			on: "rx.heartbeat",
+			note: "a new sample of the gap",
+			source: "aten_sink.erl:69-79"
+		},
+		{
+			id: "aten.down",
+			from: "up",
+			to: "down",
+			on: "timer.detect",
+			timer: "detect",
+			emits: ["node_event down"],
+			source: "aten_detector.erl:148-186"
+		},
+		{
+			id: "aten.left",
+			from: "up",
+			to: "down",
+			on: "timer.left",
+			timer: "nodedown_poll",
+			emits: ["node_event down"],
+			source: "aten_detector.erl:159-175"
+		},
+		{
+			id: "aten.up",
+			from: "down",
+			to: "up",
+			on: "rx.heartbeat",
+			emits: ["node_event up"],
+			note: "at the next poll the share is below the threshold again",
+			source: "aten_detector.erl:153-155"
+		}
+	]
+});
+//#endregion
+//#region src/rabbit/ra-member.ts
+const RA = "ra 3.1.10 src";
+const RMQ$1 = "rabbitmq v4.3.6 deps/rabbit/src";
+/**
+* This document is ra 3.1.10, for one member of the Raft cluster of a quorum queue of RabbitMQ 4.3.6.
+* It gives the roles of the member, the election timers, and the change of leader.
+*
+* A follower watches the node of its leader with aten. When aten reports that node down, the follower
+* arms a long election timeout. The timeout is the poll interval of aten, 5 s in RabbitMQ, and a random
+* part up to 1 s. A valid message of the leader cancels it. When the timeout fires, the follower asks for pre-votes in
+* its term.
+*
+* A member grants a pre-vote when the log of the candidate is not behind its own. A member that is
+* still a follower of a live leader grants it too. A majority of pre-votes makes a candidate. The
+* candidate starts a new term and asks for votes, and a majority makes it the leader. The new leader
+* of the queue tells each publisher of the queue. Then the channel of each publisher sends its pending
+* publishes again to the new leader.
+*
+* A leader ignores the events of aten. Thus a leader that loses the majority stays the leader in its
+* minority, and it commits nothing, until it sees a higher term.
+*/
+const RA_MEMBER = machineSchema.parse({
+	format: "mark1russell7.machine/1",
+	id: "ra-member",
+	title: "ra 3.1.10: one member of the Raft cluster of a quorum queue of RabbitMQ 4.3.6, its elections and its leader",
+	layer: "server",
+	version: "3.1.10",
+	conditions: {
+		ra: "3.1.10",
+		aten: "0.6.0",
+		rabbitmq: "4.3.6",
+		queue: "a quorum queue of three members, one on each node of the cluster, each one a voter"
+	},
+	sources: [
+		{
+			ref: "rabbitmq v4.3.6 rabbitmq-components.mk:58 dep_ra = hex 3.1.10",
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:66-68 DEFAULT_BROADCAST_TIME 100, DEFAULT_ELECTION_MULT 5, TICK_INTERVAL_MS 1000`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:384 the member reads the poll interval of aten: 1000 ms by default, and 5000 ms in RabbitMQ (rabbit_prelaunch_conf.erl:112-120)`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:1931-1950 the election timeouts: really_short rand(broadcast), short rand(5 × broadcast) + broadcast, medium rand(poll) + broadcast, long rand(10 × broadcast) + poll`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:2350-2357 an election timeout that is set is not set again, and :2359-2367 each change of role cancels it`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:802-825 a follower that is not new arms a long timeout when it enters the role`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:1844-1851 a valid message of the leader cancels the election timeout of a follower`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:899-919 aten reports the node of the leader down: a long timeout; up again: the timeout stops`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:869-896 the monitor of the leader goes down: short for noconnection, really_short for each other reason`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:1958-1981 a follower watches the node of a new leader with aten, and logs the new leader`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:1625-1626 the election timeout of a follower asks for pre-votes, and :2900-2923 a pre-vote round is in the current term`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:739-743 a member in pre-vote arms a long timeout, and ra_server.erl:1255-1256 its timeout asks again`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:2925-2985 a member grants a pre-vote when the term is not lower and the log of the candidate is not behind; a follower that declines arms its own long timeout`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:1229-1245 a majority of pre-votes makes a candidate, and :2880-2899 the candidate starts a new term and asks for votes`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:674-677 a candidate arms a short timeout, and ra_server.erl:1161-1162 its timeout starts a new election`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:1045-1061 a majority of votes makes the leader`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:1742-1755 each vote request is a call with the poll interval of aten as its timeout`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:597-598 a leader ignores the events of aten`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:928-942 and :946-960 a leader that sees a vote request or a pre-vote request of a higher term becomes a follower`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server.erl:1008-1032 a transfer of leadership: the leader waits, and it sends an election timeout to the member that gets the leadership`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:683-687 and :749-753 a candidate and a member in pre-vote reject a command`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RA}/ra_server_proc.erl:1336-1345 each change from or to leader is a NOTICE in the log, and each other change is DEBUG`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RMQ$1}/rabbit_fifo.erl:1111-1123 a new leader of the queue sends leader_change to each publisher`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RMQ$1}/rabbit_fifo_client.erl:738-760 the channel sends each pending publish again to the new leader, on leader_change or on a rejection`,
+			rung: "implemented"
+		},
+		{
+			ref: `${RMQ$1}/rabbit_quorum_queue.erl:175 and :2427-2441 the tick of a member is quorum_tick_interval, 5000 ms; RabbitMQ sets no broadcast time`,
+			rung: "implemented"
+		}
+	],
+	states: {
+		follower: {
+			initial: true,
+			note: "it follows the leader, and its election timeout runs when no valid message of the leader cancels it"
+		},
+		pre_vote: { note: "it asks for pre-votes in its term, and it rejects each command" },
+		candidate: { note: "a new term, its vote for itself, and a vote request to each member. A round with no majority before the timeout starts a new election after a medium timeout (101 ms to 1.1 s)" },
+		leader: { note: "it commits with a majority. It ignores aten: without a majority it stays the leader and commits nothing" }
+	},
+	events: {
+		"local.leader-node-down": {
+			kind: "local",
+			note: "{node_event, Node, down}: aten reports the node of the leader down"
+		},
+		"local.leader-node-up": {
+			kind: "local",
+			note: "{node_event, Node, up}: aten reports the node of the leader up again"
+		},
+		"rx.leader-noconnection": {
+			kind: "rx",
+			note: "the monitor of the leader goes down with noconnection: the distribution to its node closed"
+		},
+		"rx.leader-exit": {
+			kind: "rx",
+			note: "the monitor of the leader goes down with another reason: the process of the leader is gone"
+		},
+		"rx.leader-message": {
+			kind: "rx",
+			note: "a valid append_entries or heartbeat of the leader (record_leader_msg)"
+		},
+		"timer.election": {
+			kind: "timer",
+			note: "the state timeout election_timeout"
+		},
+		"rx.pre-votes": {
+			kind: "rx",
+			note: "a majority of granted pre-votes, its own one too"
+		},
+		"rx.pre-vote-declined": {
+			kind: "rx",
+			note: "a pre-vote request whose candidate has a log behind this member: it declines and arms its own timeout"
+		},
+		"rx.votes": {
+			kind: "rx",
+			note: "a majority of votes, its own one too"
+		},
+		"rx.higher-term": {
+			kind: "rx",
+			note: "a message of a term that is not lower: a leader of that term, or a vote request of a higher term"
+		},
+		"app.transfer": {
+			kind: "app",
+			note: "the leader sends election_timeout to the member that gets the leadership (rabbitmq-queues transfer_leadership)"
+		}
+	},
+	knobs: {
+		broadcast_time: {
+			key: "broadcast_time of the ra server",
+			scope: "queue",
+			unit: "ms",
+			default: 100,
+			note: "ra_server_proc.erl:66: RabbitMQ sets none"
+		},
+		election_mult: {
+			key: "DEFAULT_ELECTION_MULT",
+			scope: "build",
+			unit: "count",
+			default: 5,
+			note: "ra_server_proc.erl:67"
+		},
+		aten_poll_interval: {
+			key: "raft.adaptive_failure_detector.poll_interval (aten poll_interval)",
+			scope: "node",
+			unit: "ms",
+			default: 5e3,
+			note: "ra_server_proc.erl:384 reads it. RabbitMQ sets 5000 (rabbit_prelaunch_conf.erl:112-120)"
+		},
+		tick_timeout: {
+			key: "quorum_tick_interval",
+			scope: "node",
+			unit: "ms",
+			default: 5e3,
+			note: "rabbit_quorum_queue.erl:175, :2435-2441"
+		},
+		slack: {
+			key: "the lateness of a timer of the BEAM for one step",
+			scope: "node",
+			unit: "ms",
+			default: 10,
+			note: "an assumption of this document, not a value of ra: a state timeout that fires late"
+		},
+		commit_work: {
+			key: "the local work of one command: the writes and the fsyncs of the logs of a leader and a follower, and the apply",
+			scope: "node",
+			unit: "ms",
+			default: 50,
+			note: "an assumption of this document, not a value of ra: the WAL writes each batch and syncs it with fdatasync before the commit (ra_log_wal.erl:89, :772-798)"
+		},
+		rtt: {
+			key: "the round trip between members",
+			scope: "path",
+			unit: "ms",
+			default: null
+		}
+	},
+	timers: {
+		election_long: {
+			duration: {
+				expr: "aten_poll_interval + 1",
+				unit: "ms"
+			},
+			lateness: {
+				expr: "broadcast_time * election_mult * 2 - 1 + slack",
+				unit: "ms"
+			},
+			arms: [
+				"enter:follower",
+				"enter:pre_vote",
+				"local.leader-node-down",
+				"rx.pre-vote-declined"
+			],
+			cancels: ["rx.leader-message", "local.leader-node-up"],
+			fires: "timer.election",
+			note: "longer than one poll of aten, thus a false report of aten can end before an election (ra_server_proc.erl:1944-1949)"
+		},
+		election_short: {
+			duration: {
+				expr: "broadcast_time + 1",
+				unit: "ms"
+			},
+			lateness: {
+				expr: "broadcast_time * election_mult - 1 + slack",
+				unit: "ms"
+			},
+			arms: ["enter:candidate", "rx.leader-noconnection"],
+			cancels: ["rx.leader-message"],
+			fires: "timer.election"
+		},
+		election_really_short: {
+			duration: {
+				const: 1,
+				unit: "ms"
+			},
+			lateness: {
+				expr: "broadcast_time - 1 + slack",
+				unit: "ms"
+			},
+			arms: ["rx.leader-exit"],
+			cancels: ["rx.leader-message"],
+			fires: "timer.election"
+		}
+	},
+	realizes: { "systems.mechanism.election-timeout": { timer: "election_long" } },
+	transitions: [
+		{
+			id: "ra.leader-message",
+			from: "follower",
+			to: "follower",
+			on: "rx.leader-message",
+			note: "the election timeout stops",
+			source: "ra_server_proc.erl:1844-1851"
+		},
+		{
+			id: "ra.leader-node-down",
+			from: "follower",
+			to: "follower",
+			on: "local.leader-node-down",
+			note: "the long election timeout starts",
+			source: "ra_server_proc.erl:899-908"
+		},
+		{
+			id: "ra.pre-vote",
+			from: "follower",
+			to: "pre_vote",
+			on: ["timer.election", "app.transfer"],
+			source: "ra_server.erl:1625-1626, :2900-2923"
+		},
+		{
+			id: "ra.pre-vote-again",
+			from: "pre_vote",
+			to: "pre_vote",
+			on: "timer.election",
+			timer: "election_long",
+			source: "ra_server.erl:1255-1256"
+		},
+		{
+			id: "ra.candidate",
+			from: "pre_vote",
+			to: "candidate",
+			on: "rx.pre-votes",
+			source: "ra_server.erl:1229-1245, :2880-2899"
+		},
+		{
+			id: "ra.election-again",
+			from: "candidate",
+			to: "candidate",
+			on: "timer.election",
+			timer: "election_short",
+			source: "ra_server.erl:1161-1162"
+		},
+		{
+			id: "ra.leader",
+			from: "candidate",
+			to: "leader",
+			on: "rx.votes",
+			emits: ["candidate -> leader", "leader_change"],
+			note: "the NOTICE of the change, and leader_change to each publisher of the queue",
+			source: "ra_server.erl:1045-1061, ra_server_proc.erl:1336-1345, rabbit_fifo.erl:1111-1123"
+		},
+		{
+			id: "ra.follow",
+			from: ["pre_vote", "candidate"],
+			to: "follower",
+			on: "rx.higher-term",
+			source: "ra_server.erl:1062-1084"
+		},
+		{
+			id: "ra.abdicate",
+			from: "leader",
+			to: "follower",
+			on: "rx.higher-term",
+			emits: ["leader -> follower"],
+			source: "ra_server.erl:928-942, :946-960"
+		}
+	],
+	observe: { "rabbitmq-log": {
+		fidelity: "lossy",
+		states: {
+			leader: "leader",
+			follower: "follower"
+		},
+		hides: ["pre_vote", "candidate"],
+		note: "at the default level the log has each change from or to leader (NOTICE), and the new leader that a follower detects (INFO). The elections are DEBUG"
+	} }
+});
+//#endregion
 //#region src/rabbit/rabbitmq-connection.ts
 const RMQ = "v4.3.6";
 /**
@@ -12512,9 +13101,11 @@ const MACHINES = byId([
 	SUPAVISOR_CLIENT,
 	SUPAVISOR_SERVER,
 	AMQPLIB,
-	RABBITMQ_CONNECTION
+	RABBITMQ_CONNECTION,
+	ATEN_DETECTOR,
+	RA_MEMBER
 ]);
 //#endregion
-export { AMQPLIB, GO_TRANSPORT, HIKARICP_PGJDBC, IOREDIS_SENTINEL, IOREDIS_STATUS, LINUX_TCP, MACHINES, MANIFESTATIONS, MONGODB_NODE, MONGOD_REPL, MONGOD_SESSION, NF_CONNTRACK_TCP, NODE_HTTP_AGENT, NODE_HTTP_SERVER, NODE_POSTGRES, NODE_TIMER_EARLY, PGBOUNCER_CLIENT, PGBOUNCER_SERVER, PGPOOL_SESSION, POSTGRES_SESSION, PROCEDURES_WS_CLIENT, PROCEDURES_WS_SERVER, RABBITMQ_CONNECTION, REDIS_CLIENT, SENTINEL, SUPAVISOR_CLIENT, SUPAVISOR_SERVER, UNDICI_CLIENT, UNDICI_TICK_MS, goLateSeconds, goTimerLateness, nodeEarlySeconds, nodeLateSeconds, nodeTimerLateness, undiciFastEarly, undiciFastLateness };
+export { AMQPLIB, ATEN_DETECTOR, GO_TRANSPORT, HIKARICP_PGJDBC, IOREDIS_SENTINEL, IOREDIS_STATUS, LINUX_TCP, MACHINES, MANIFESTATIONS, MONGODB_NODE, MONGOD_REPL, MONGOD_SESSION, NF_CONNTRACK_TCP, NODE_HTTP_AGENT, NODE_HTTP_SERVER, NODE_POSTGRES, NODE_TIMER_EARLY, PGBOUNCER_CLIENT, PGBOUNCER_SERVER, PGPOOL_SESSION, POSTGRES_SESSION, PROCEDURES_WS_CLIENT, PROCEDURES_WS_SERVER, RABBITMQ_CONNECTION, RA_MEMBER, REDIS_CLIENT, SENTINEL, SUPAVISOR_CLIENT, SUPAVISOR_SERVER, UNDICI_CLIENT, UNDICI_TICK_MS, goLateSeconds, goTimerLateness, nodeEarlySeconds, nodeLateSeconds, nodeTimerLateness, undiciFastEarly, undiciFastLateness };
 
 //# sourceMappingURL=index.js.map