librats 2.1.4 → 2.3.0

package/native-src/src/librats/subsystems/relay.h

@@ -0,0 +1,211 @@
+#pragma once
+
+/**
+ * @file relay.h
+ * @brief Reaching a peer through a node both ends can already reach.
+ *
+ * The last rung of the connectivity ladder. A direct dial handles the easy cases;
+ * PortMappingService and HolePunch handle most of the rest. What is left is the
+ * pair of nodes for which no endpoint either one can advertise will ever work —
+ * a symmetric NAT on one side, a network that drops UDP and blocks inbound TCP,
+ * a punch that simply never lands. For them the only way through is to borrow a
+ * path: a node connected to both carries the bytes.
+ *
+ * ── What is relayed, and what is not ────────────────────────────────────────
+ * The relayed thing is a *byte stream*, not a message. It becomes a `Circuit` and
+ * a `RelayLink` (transport/relay_link.h), the reactor adopts it as an ordinary
+ * Connection, and everything above the Link runs unchanged — which is why:
+ *
+ *   - the Noise_XX handshake is END TO END. The relay moves ciphertext it cannot
+ *     read, cannot forge and cannot replay. It is a pipe, not a party, and it
+ *     learns nothing beyond who is talking to whom and how much;
+ *   - the peers authenticate each other's real keys, so a relay cannot substitute
+ *     itself for either end;
+ *   - every subsystem — pub/sub, file transfer, PEX — works over a relayed peer
+ *     with no code of its own. To the application it is simply a peer, and
+ *     PeerInfo::transport is the only thing that says otherwise.
+ *
+ * ── Finding a relay ─────────────────────────────────────────────────────────
+ * This stage covers the case where the two ends share a peer — the same topology
+ * HolePunch already relies on for its rendezvous, and the one a bootstrap or DHT
+ * mesh produces on its own. The initiator asks a few of its peers "do you hold
+ * this id?" (Probe) and opens a circuit with the first that says yes. Probing
+ * first, rather than opening with everybody at once, is what keeps a successful
+ * search from producing three redundant connections to the same peer — each with
+ * its own handshake — for the peer table to then tear two of down.
+ *
+ * Not covered here, deliberately: reaching a peer with whom we share NO peer. That
+ * needs reservations (a node asking a relay to hold a slot for it) and a way to
+ * advertise "reachable via R" as an address, which touches identify and PEX. The
+ * version byte and the free op-codes in the wire format leave room for it.
+ *
+ * ── Getting off the relay again ─────────────────────────────────────────────
+ * A circuit is a fallback, not a destination: it costs a third node bandwidth and
+ * a round trip. So when one comes up, this module asks HolePunch to try the target
+ * again — now that the two ends are peers, they can exchange what a punch needs
+ * over the very circuit that is carrying them. If the punch lands, PeerTable::add
+ * prefers the direct link at BOTH ends (any direct transport outranks Relay) and
+ * swaps the route with no disconnect event: the application never sees the seam.
+ *
+ * ── Carrying other peers' traffic ───────────────────────────────────────────
+ * Serving as a relay is off by default. Forwarding a rendezvous, as HolePunch
+ * does, is a few dozen bytes; forwarding a connection is somebody else's file
+ * transfer on your uplink, and that is a decision to be made rather than assumed.
+ * A node that turns it on is protected by, in order of how much they matter:
+ *
+ *   - the end-to-end credit window (transport/relay_link.h), which bounds what one
+ *     circuit can make the relay hold no matter what either end does. Without it a
+ *     slow receiver would grow the relay's send queue until the relay dropped that
+ *     peer entirely — losing all of its traffic, not just the circuit's;
+ *   - forwarding only between peers it already holds. A relay never dials, never
+ *     resolves an address, and can only ever deliver to somebody that chose to
+ *     connect to it, so it cannot be turned into an open reflector;
+ *   - no chaining. A circuit is refused if either end is itself only reachable
+ *     through a relay, which is what stops loops and multi-hop amplification;
+ *   - per-circuit byte and duration caps, per-peer and total circuit counts, and a
+ *     rate limit on the requests themselves.
+ *
+ * ── Threading ───────────────────────────────────────────────────────────────
+ * Message handlers run on reactor threads; one worker thread drives attempt
+ * timeouts and the relay's own expiry. The tables are behind one mutex, which is
+ * never held across a call into a circuit or the reactor — a circuit is touched
+ * only by the reactor thread that owns its connection, which is by construction
+ * the thread that owns its carrier's (see node/circuit_service.h).
+ */
+
+#include "librats/util/rats_export.h"
+#include "librats/core/service_registry.h"
+#include "librats/node/peer_network.h"
+#include "librats/peer/peer_id.h"
+#include "librats/subsystems/relay_service.h"
+#include "librats/transport/relay_link.h"   // Circuit::kDefaultWindow
+
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
+#include <cstdint>
+#include <memory>
+#include <mutex>
+#include <thread>
+
+namespace librats {
+
+class RATS_API Relay final : public Subsystem, public RelayService {
+public:
+    struct Config {
+        // ── Using relays to reach peers we cannot dial ──────────────────────
+
+        /// Open circuits through other peers. Off makes this a relay-only node.
+        bool enable_client = true;
+
+        /// Peers asked whether they hold a target, per attempt. Small: each probe
+        /// is a question to somebody who probably cannot help, and one useful
+        /// answer is all an attempt needs.
+        size_t max_probes = 3;
+
+        /// Circuits this node may hold open as a client at once. Bounds both memory
+        /// and how many relayed peers a discovery module can talk us into.
+        size_t max_outbound_circuits = 8;
+
+        /// How long an attempt may take from the first probe to a live connection.
+        /// Covers the probe round trip, the circuit setup, and the handshake across
+        /// two hops.
+        std::chrono::milliseconds open_timeout{8000};
+
+        /// How long after a failed attempt before the same target may be tried
+        /// again. Only an attempt WE started ever ends in one.
+        std::chrono::milliseconds cooldown{60000};
+
+        // ── Accepting circuits opened to us ─────────────────────────────────
+
+        /// Accept circuits other peers open toward us. Off makes this node
+        /// unreachable by relay, which is a choice a well-connected node can make.
+        bool accept_inbound = true;
+
+        /// Circuits opened TO us that may be live at once.
+        size_t max_inbound_circuits = 16;
+
+        // ── Carrying other peers' traffic ───────────────────────────────────
+
+        /// Serve as a relay. OFF by default: unlike a hole-punch rendezvous, this
+        /// spends real bandwidth on somebody else's connection, so it is opted into
+        /// rather than assumed. A mesh in which nobody serves cannot relay at all.
+        bool serve = false;
+
+        /// Circuits this node will carry for others at once.
+        size_t max_circuits = 32;
+
+        /// Circuits one peer may have us carrying, so a single peer cannot take the
+        /// whole budget.
+        size_t max_circuits_per_peer = 2;
+
+        /// Bytes one circuit may move before it is closed. 0 removes the cap. The
+        /// default is generous for a fallback path and still bounds what one pair
+        /// of peers can spend of ours.
+        uint64_t max_bytes_per_circuit = 64ull * 1024 * 1024;
+
+        /// How long one circuit may live. 0 removes the cap.
+        std::chrono::seconds max_circuit_duration{600};
+
+        /// Requests (probes and circuit openings) one peer may make per window.
+        size_t                    request_budget = 8;
+        std::chrono::milliseconds request_window{1000};
+
+        // ── Both ends ───────────────────────────────────────────────────────
+
+        /// Bytes this node is willing to have in flight per circuit — the credit
+        /// window it advertises (see transport/relay_link.h). Raising it makes a
+        /// relayed peer faster on a long path and asks the relay to hold more.
+        uint32_t window = Circuit::kDefaultWindow;
+
+        /// When a relayed peer connects, ask HolePunch to try it directly, so the
+        /// circuit is replaced by a direct link as soon as one is possible. Costs
+        /// nothing when no HolePunch is attached.
+        bool upgrade_with_hole_punch = true;
+
+        /// Bookkeeping cadence — attempt timeouts, cooldowns, circuit expiry.
+        std::chrono::milliseconds tick{250};
+    };
+
+    Relay();
+    explicit Relay(Config config);
+    ~Relay() override;
+
+    void attach(NodeContext& ctx) override;
+    void start() override;
+    void stop() override;
+
+    /// @copydoc RelayService::connect_via_relay
+    bool connect_via_relay(const PeerId& target) override;
+
+    // — diagnostics and tests —
+    /// Circuits this node terminates (relayed peers, live or still being set up).
+    size_t circuits() const;
+    /// Circuits this node is carrying on behalf of other peers.
+    size_t carried_circuits() const;
+    /// Bytes forwarded on behalf of other peers.
+    uint64_t carried_bytes() const;
+    /// Attempts to reach a peer through a relay that are currently running.
+    size_t attempts() const;
+
+private:
+    /// Everything mutable, held by shared_ptr because a Connection — and therefore
+    /// the Link that reports its circuit released — can outlive this subsystem: the
+    /// node stops subsystems before it stops the reactors that own the connections.
+    struct State;
+
+    Config                 config_;
+    std::shared_ptr<State> state_;
+    /// Kept from attach() so start() can resolve HolePunchService — which may be
+    /// attached after us, and is only guaranteed to have registered by then.
+    ServiceRegistry*       services_ = nullptr;
+
+    std::thread             worker_;
+    std::atomic<bool>       running_{false};
+    std::mutex              worker_mutex_;
+    std::condition_variable worker_cv_;
+
+    void loop();
+};
+
+} // namespace librats

package/native-src/src/librats/subsystems/relay_service.h

@@ -0,0 +1,46 @@
+#pragma once
+
+/**
+ * @file relay_service.h
+ * @brief Capability that lets a sibling module ask for a peer to be reached
+ *        through a third node, by PeerId.
+ *
+ * Published by Relay via ServiceRegistry (see service_registry.h). It is the last
+ * rung of the connectivity ladder — a direct dial, then a hole punch, then this —
+ * and the module that gives up on the rung above should not have to know anything
+ * about how the rung below works. HolePunch, for instance, learns that a target is
+ * unreachable (a symmetric NAT it declined to punch, or a rendezvous that ran out
+ * of attempts) and this is the whole contract it needs to hand the id on: no relay
+ * selection, no circuits, no policy, all of which are Relay's own business.
+ *
+ * Resolving it returns nullptr when relaying is not enabled, which is exactly what
+ * makes the fallback optional at the consumer's side:
+ *
+ *     if (auto* relay = ctx.services.get<RelayService>()) relay->connect_via_relay(id);
+ *
+ * The provider registers during attach(), so resolve it in start() (every attach()
+ * runs before any start()) rather than in your own attach(), where the provider may
+ * not have been attached yet. The pointer is NON-owning and valid while the node is.
+ */
+
+#include "librats/util/rats_export.h"
+#include "librats/peer/peer_id.h"
+
+namespace librats {
+
+struct RATS_API RelayService {
+    virtual ~RelayService() = default;
+
+    /// Try to reach `target` through a node both ends are connected to.
+    /// Non-blocking; success surfaces as an ordinary peer-connected event, and the
+    /// resulting peer is ordinary in every way except that its bytes take a detour
+    /// (PeerInfo::transport is TransportKind::Relay).
+    ///
+    /// @return whether an attempt actually started — false is routine (already
+    ///         connected, an attempt is already running, the target is in cooldown,
+    ///         or this node has no peer that could carry the circuit) and needs no
+    ///         handling by the caller.
+    virtual bool connect_via_relay(const PeerId& target) = 0;
+};
+
+} // namespace librats

package/native-src/src/librats/transport/connection.cpp

@@ -51,6 +51,46 @@ uint8_t Connection::reactor_index() const noexcept { return reactor_.index(); }
 bool Connection::send(FrameHeader header, ByteView payload) {
     if (state_ != ConnState::Established) return false;  // frames only flow post-handshake
 
+    // The one thing that is refused outright rather than queued: a message too
+    // large to be framed at all. The block prefix tops out at kMaxBlockSize and the
+    // peer's decoder rejects anything past it, so queueing this would spend the
+    // link only to have the far end hang up on a protocol error it was handed on
+    // purpose. Waiting cannot help either — no amount of draining makes it fit — so
+    // the honest answer is a refusal, with the frame unqueued and the connection
+    // untouched. Checked before encrypt() because the nonce counters run in
+    // lockstep: a message encrypted and then dropped would break every one after.
+    if (framer::kHeaderSize + payload.size() + session_->overhead() > framer::kMaxBlockSize) {
+        LOG_WARN("connection", "Peer " << remote_id_.short_hex() << " refused a "
+                 << payload.size() << " B message: past the " << framer::kMaxBlockSize
+                 << " B block ceiling; split it");
+        return false;
+    }
+
+    // Weighed on the backlog as it stands BEFORE this message, never after it has
+    // been added. A message is indivisible — there is no queueing half of one — so
+    // judging a caller by the mark its own message has just crossed answers "that
+    // message was bigger than a limit nobody published" with a disconnection, on a
+    // connection that is idle, healthy and draining at full speed. What the mark is
+    // for is a peer that cannot keep up, and the evidence for that is a caller
+    // piling MORE on top of a backlog that is already over it.
+    //
+    // So one message may always be queued, whatever its size — the queue may exceed
+    // the mark by exactly that message and no more — and offering another before it
+    // has drained is what makes a caller a slow consumer. Nothing is ever dropped
+    // or refused here, which is what the layers above rely on: a relayed circuit
+    // (transport/relay_link.cpp) reports bytes as written the moment it hands them
+    // over, and a byte stream cannot survive one of them going missing.
+    const size_t backlog_before = backlog();
+    if (backlog_before > send_high_water_) {
+        LOG_WARN("connection", "Peer " << remote_id_.short_hex() << " offered more with "
+                 << backlog_before << " B still queued past the high-water mark ("
+                 << send_high_water_ << " B); closing as slow consumer");
+        close_reason_ = CloseReason::SlowConsumer;
+        state_ = ConnState::Closing;
+        reactor_.close(id_, CloseReason::SlowConsumer);
+        return false;
+    }
+
     Bytes inner;
     framer::encode_message(inner, header, payload);
 
@@ -64,14 +104,6 @@ bool Connection::send(FrameHeader header, ByteView payload) {
     queue_block(std::move(cipher));
 
     const size_t backlog_now = backlog();
-    if (backlog_now > send_high_water_) {
-        LOG_WARN("connection", "Peer " << remote_id_.short_hex() << " over send high-water ("
-                 << backlog_now << " B); closing as slow consumer");
-        close_reason_ = CloseReason::SlowConsumer;
-        state_ = ConnState::Closing;
-        reactor_.close(id_, CloseReason::SlowConsumer);
-        return false;
-    }
 
     // The queue has grown past what a caller should keep adding to. Everything
     // still goes out — nothing is dropped here — but the answer to "may I send

package/native-src/src/librats/transport/connection.h

@@ -85,8 +85,11 @@ public:
 
 class Connection {
 public:
-    /// High-water mark for the memory held by the send queue; exceeding it closes the
-    /// connection with CloseReason::SlowConsumer.
+    /// High-water mark for the memory held by the send queue. A caller that offers
+    /// another message while the queue is *still* over it has the connection closed
+    /// with CloseReason::SlowConsumer. The mark is never applied to the message that
+    /// crosses it: one message may always be queued, whatever its size (see send()),
+    /// so the queue's ceiling is this plus one message rather than this exactly.
     static constexpr size_t kDefaultSendHighWater = 8 * 1024 * 1024;
 
     /// Where send() starts answering "no room". A quarter of the hard limit, so a
@@ -125,6 +128,13 @@ public:
 
     /// Queue an application frame for the peer. No-op unless Established.
     ///
+    /// Never refuses and never drops: whatever it is handed is queued, however
+    /// large. The size of a single message is not what the high-water mark is
+    /// about — a message cannot be queued by halves, so a connection that is
+    /// draining perfectly well must not be torn down over one big frame. What
+    /// closes a connection is offering *another* message while the queue is still
+    /// over the mark, which is the actual evidence that the peer is not keeping up.
+    ///
     /// @return whether there is still room for more. False means the queue is
     ///         past its low-water mark — this frame is queued like any other,
     ///         nothing is dropped, but the caller should stop and wait for

package/native-src/src/librats/transport/reactor.cpp

@@ -150,6 +150,27 @@ void Reactor::start_dial(ConnId id, const std::string& host, int port, Transport
     conn->set_dial_address(host, static_cast<uint16_t>(port));
 }
 
+void Reactor::adopt_link(ConnId id, std::unique_ptr<Link> link, ConnRole role, bool connected) {
+    // A unique_ptr cannot be captured by a std::function, which is what the task
+    // queue holds — so the link travels in a shared holder it is moved out of on
+    // arrival. If the task is never run (a reactor stopped under us), the holder
+    // dies with it and takes the link with it, which is the right outcome.
+    auto holder = std::make_shared<std::unique_ptr<Link>>(std::move(link));
+    execute([this, id, holder, role, connected] {
+        Connection* conn = adopt(std::move(*holder), role, id);
+        if (!conn) return;
+        // Already-connected links skip the Connecting state entirely, exactly as an
+        // accepted socket does; the rest wait for the PollOut that wake() will bring
+        // once the far end answers.
+        if (connected) conn->start_handshake();
+    });
+}
+
+void Reactor::wake(ConnId id, uint32_t events) {
+    if (events == 0) return;
+    execute([this, id, events] { dispatch_events(id, events); });
+}
+
 void Reactor::abort_dial(ConnId id, const std::string& host, int port) {
     resolve_dial(id);  // the id is spent either way; release any cancellation slot
     delegate_.on_dial_aborted(index_, id, host, static_cast<uint16_t>(port));
@@ -243,6 +264,21 @@ void Reactor::run() {
         if (mux_) timeout = mux_->next_timeout_ms(timeout);
         const int n = poller_->wait(events, kMaxEvents, timeout);
 
+        // Empty the wakeup pipe BEFORE taking the task snapshot, never after.
+        //
+        // post() pushes a task and then writes a byte, and those two arrive here as
+        // two separate facts. Draining last would let a byte written *while this
+        // turn was running its tasks* be thrown away along with the ones that
+        // brought us here — and the task it belonged to was pushed after the
+        // snapshot, so it does not run either. It would then sit in the queue with
+        // nothing left to announce it, until the next poll timeout woke the loop
+        // for some other reason: up to kMaxPollMs of latency, at random.
+        //
+        // Draining first inverts that. A byte written any time after this line is
+        // still in the pipe when the loop comes back round, so wait() returns at
+        // once — whether or not its task made this turn's snapshot. The cost is one
+        // recv per turn, which is nothing against the work a turn does.
+        drain_wakeup();
         drain_tasks(task_batch);                         // connect/close/send-arm
         for (int i = 0; i < n; ++i) handle_event(events[i]);
         timers_.run_due();
@@ -325,7 +361,9 @@ void Reactor::drain_wakeup() {
 void Reactor::handle_event(const PollResult& ev) {
     const socket_t fd = ev.fd;
 
-    if (fd == wakeup_.fd()) { drain_wakeup(); return; }
+    // Already emptied at the top of the turn — see run(). Nothing to do but
+    // recognise it, so it is not mistaken for a connection's socket.
+    if (fd == wakeup_.fd()) return;
     if (fd == server_socket_) { if (ev.events & PollIn) do_accept(); return; }
     if (mux_ && fd == mux_->socket()) {
         // Any event, not just PollIn. An error on a datagram socket belongs to one
@@ -479,16 +517,26 @@ void Reactor::mark_for_close(ConnId id, CloseReason reason) {
 }
 
 void Reactor::flush_dirty() {
-    if (dirty_.empty()) return;
     // Swapped out first: a flush can close a connection, and a close can queue
     // further work — none of which should extend the batch being walked.
+    //
+    // Looped, because a flush can also *book another one*: writing a relayed
+    // circuit hands its bytes to the connection carrying it (see
+    // transport/relay_link.h), which books its own flush from inside this one. A
+    // single pass would leave those bytes queued until the next turn of the loop —
+    // i.e. behind a poll wait — adding tens of milliseconds to every hop of a
+    // relayed path. The bound is a backstop against a pathological chain, not an
+    // expected limit: honest work needs two passes at most.
     std::vector<ConnId> batch;
-    batch.swap(dirty_);
-    for (const ConnId id : batch) {
-        Connection* conn = find(id);
-        if (!conn) continue;                                   // gone since it booked
-        if (conn->state() != ConnState::Established) continue;  // closing; nothing to owe
-        if (!conn->flush_pending()) mark_for_close(id, conn->close_reason());
+    for (int pass = 0; pass < kMaxFlushPasses && !dirty_.empty(); ++pass) {
+        batch.clear();
+        batch.swap(dirty_);
+        for (const ConnId id : batch) {
+            Connection* conn = find(id);
+            if (!conn) continue;                                   // gone since it booked
+            if (conn->state() != ConnState::Established) continue;  // closing; nothing to owe
+            if (!conn->flush_pending()) mark_for_close(id, conn->close_reason());
+        }
     }
 }
 

package/native-src/src/librats/transport/reactor.h

@@ -104,6 +104,41 @@ public:
     /// Valid for an id whose dial has not started yet — it is cancelled instead.
     void close(ConnId id, CloseReason reason);
 
+    // — links this reactor did not open itself ──────────────────────────────
+    //
+    // A relayed circuit is a byte stream that comes out of another peer's
+    // connection rather than out of a socket (see transport/relay_link.h), so the
+    // module that speaks the relay protocol builds the Link and hands it over.
+    // These three are that hand-over, and they are deliberately the whole of it:
+    // the reactor learns nothing about relaying, and the module learns nothing
+    // about reactors beyond "connections need to be woken".
+    //
+    // The circuit MUST be given to the reactor that owns its carrier's connection.
+    // That is what keeps every byte of it on one thread, exactly like the streams
+    // the mux hands over above, and it is the caller's job to arrange (see
+    // node/circuit_service.h).
+
+    /// Reserve a ConnId without starting anything. Thread-safe, and synchronous for
+    /// the same reason connect() is: the caller has to be able to name the
+    /// connection — to wake it, to close it — from the moment it asks for one,
+    /// rather than after a task it posted has run.
+    ConnId reserve_conn_id() noexcept {
+        return next_conn_id_.fetch_add(1, std::memory_order_relaxed);
+    }
+
+    /// Adopt an already-built Link under the id `reserve_conn_id()` handed out.
+    /// Thread-safe; the adoption itself runs on the reactor thread (inline when the
+    /// caller is already on it).
+    /// @param connected whether the far end is already there. Such a link starts its
+    ///        handshake at once, like an accepted socket; one that is not waits for
+    ///        a PollOut from wake(), exactly as an outbound connect does.
+    void adopt_link(ConnId id, std::unique_ptr<Link> link, ConnRole role, bool connected);
+
+    /// Deliver poll-equivalent events (PollIn/PollOut/PollErr) to a connection whose
+    /// Link has no socket for the poller to watch. Thread-safe. A no-op for an id
+    /// that is gone, so a late wake for a torn-down circuit costs nothing.
+    void wake(ConnId id, uint32_t events);
+
     /// Drop a dial that lost its race — but only while it is still a dial. Once an
     /// attempt has established it is not an attempt any more, it is one of possibly
     /// several links to a peer, and which of those survives is the peer table's
@@ -175,6 +210,9 @@ private:
     // Negligible idle cost; on epoll/kqueue the loop still wakes on real events,
     // so this only caps idle latency.
     static constexpr int kMaxPollMs = 50;
+    /// Passes flush_dirty() makes over the connections booked for a write. More than
+    /// one because a write can book another connection's write — see flush_dirty().
+    static constexpr int kMaxFlushPasses = 4;
     /// Deadline from adopt() to reaching Established (covers connect + handshake).
     static constexpr std::chrono::milliseconds kEstablishTimeout{15000};
     /// Cadence of the housekeeping sweep over this reactor's connections (currently