iodine 0.7.38 → 0.7.43

data/SPEC-WebSocket-Draft.md

@@ -5,62 +5,72 @@ This draft is also implemented by [the Agoo server](https://github.com/ohler55/a
 ---
 ## Purpose
 
-This document details WebSocket / EventSource connection support for Rack servers.
+This document details a Rack specification extension for WebSocket / EventSource servers.
 
-The purpose of these specifications is:
+The purpose of this specification is:
 
-1. To allow separation of concerns between the transport layer and the application, thereby allowing the application to be server agnostic.
+1. To improve application safety by phasing out the use of `hijack` and replacing it with the use of application object callbacks.
 
-    Simply put, when choosing between conforming servers, the application doesn’t need to have any knowledge about the chosen server.
+    This should make it easer for applications to accept WebSocket and EventSource (SSE) connections without exposing themselves to risks and errors related to IO / network logic (such as slow client attacks, buffer flooding, etc').
 
-2. To support “native" (server-side) WebSocket and EventSource (SSE) connections and using application side callbacks.
+2. To improve separation of concerns between servers and applications, moving the IO / network logic related to WebSocket and EventSource (SSE) back to the server.
 
-    Simply put, to make it easy for applications to accept WebSocket and EventSource (SSE) connections from WebSocket and EventSource clients (commonly browsers) while abstracting away any transport layer details.
-
-3. Allow applications to use WebSocket and EventSource (SSE) on HTTP/2 servers. Note: current `hijack` practices will break network connections when attempting to implement EventSource (SSE).
+    Simply put, when using a server that support this extension, the application / framework doesn’t need to have any knowledge about networking, transport protocols, IO streams, polling, etc'.
 
 ## Rack WebSockets / EventSource
 
-Servers that publish WebSocket and/or EventSource (SSE) support using the `env['rack.upgrade?']` value MUST follow the requirements set in this document.
+Servers that publish WebSocket and/or EventSource (SSE) support using the `env['rack.upgrade?']` value **MUST** follow the requirements set in this document.
 
 This document reserves the Rack `env` Hash keys of `rack.upgrade?` and `rack.upgrade`.
 
-A conforming server MUST set `env['rack.upgrade?']` to `:websocket` for incoming WebSocket connections and `:sse` for incoming EventSource (SSE) connections.
+A conforming server **MUST** set `env['rack.upgrade?']` to `:websocket` for incoming WebSocket connections and `:sse` for incoming EventSource (SSE) connections. 
+
+If a connection is not "upgradeable", a conforming server **SHOULD** set `env['rack.upgrade?']` to either `nil` or `false`. Setting the `env['rack.upgrade?']` to either `false` or `nil` should make it easier for applications to test for server support during a normal HTTP request.
+
+If the connection is upgradeable and a client application set a value for `env['rack.upgrade']`:
+
+* the server **MUST** use that value as a WebSocket / EventSource Callback Object unless the response status code `>= 300` (redirection / error status code).
+
+* The response `body` **MUST NOT** be sent when switching to a Callback Object.
+
+If a connection is **NOT** upgradeable and a client application set a value for `env['rack.upgrade']`:
 
-If a connection is not "upgradable", a conforming server SHOULD set `env['rack.upgrade?']` to either `nil` or `false`. 
+* The server **SHOULD** ignore the Callback Object and process the response as if it did not exist.
 
-The historical use of `upgrade.websocket?` and `upgrade.websocket` (iodine 0.4.x) will be gradually deprecated.
+* A server **MAY** use the Callback Object to allow a client to "hijack" the data stream as raw data stream. Such behavior **MUST** be documented.
 
 ### The WebSocket / EventSource Callback Object
 
 WebSocket and EventSource connection upgrade and handling is performed using a Callback Object.
 
-The Callback Object could be a any object which implements any of the following callbacks:
+The Callback Object could be a any object which implements any (of none) of the following callbacks:
 
-* `on_open(client)` WILL be called once the connection had been established.
+* `on_open(client)` **MUST** be called once the connection had been established and/or the Callback Object had been linked to the `client` object.
 
-* `on_message(client, data)` WILL be called when incoming WebSocket data is received.
+* `on_message(client, data)` **MUST** be called when incoming WebSocket data is received.
 
     This callback is ignored for EventSource connections.
 
-    `data` will be a String with an encoding of UTF-8 for text messages and `binary` encoding for non-text messages (as specified by the WebSocket Protocol).
+    `data` **MUST** be a String with an encoding of UTF-8 for text messages and `binary` encoding for non-text messages (as specified by the WebSocket Protocol).
 
     The *callback object* **MUST** assume that the `data` String will be a **recyclable buffer** and that it's content will be corrupted the moment the `on_message` callback returns.
 
-    Servers **MAY**, optionally, implement a **recyclable buffer** for the `on_message` callback. However, this is optional and it is *not* required.
+    Servers **MAY**, optionally, implement a **recyclable buffer** for the `on_message` callback. However, this is optional, is *not* required and might result in issues in cases where the client code is less than pristine.
 
-* `on_drained(client)` **MAY** be called when the the `client.write` buffer becomes empty. **If** `client.pending` returns a non-zero value, the `on_drained` callback **MUST** be called once the write buffer becomes empty.
+* `on_drained(client)` **MAY** be called when the `client.write` buffer becomes empty. **If** `client.pending` ever returns a non-zero value (see later on), the `on_drained` callback **MUST** be called once the write buffer becomes empty.
 
 * `on_shutdown(client)` **MAY** be called during the server's graceful shutdown process, _before_ the connection is closed and in addition to the `on_close` function (which is called _after_ the connection is closed.
 
-* `on_close(client)` **MUST** be called _after_ the connection was closed for whatever reason (socket errors, parsing errors, timeouts, client disconnection, `client.close` being called, etc').
+* `on_close(client)` **MUST** be called _after_ the connection was closed for whatever reason (socket errors, parsing errors, timeouts, client disconnection, `client.close` being called, etc') or the Callback Object was replaced by another Callback Object.
 
 
 The server **MUST** provide the Callback Object with a `client` object, that supports the following methods (this approach promises applications could be server agnostic):
 
-* `write(data)` will schedule the data to be sent. `data` **MUST** be a String.
+* `env` **MUST** return the Rack `env` hash related to the originating HTTP request. Some changes to the `env` hash (such as removal of the IO hijacking support) **MAY** be implemented by the server.
 
-    A call to `write` only promises that the data is scheduled to be sent. Servers are encouraged to avoid blocking and return immediately, deferring the actual `write` operation for later.
+* `write(data)` **MUST** schedule **all** the data to be sent. `data` **MUST** be a String. Servers **MAY** silently convert non-String objects to JSON if an application attempts to `write` a non-String value, otherwise servers **SHOULD** throw an exception.
+
+    A call to `write` only promises that the data is scheduled to be sent. Implementation details may differ across servers.
 
     `write` shall return `true` on success and `false` if the connection is closed.
 
@@ -70,27 +80,52 @@ The server **MUST** provide the Callback Object with a `client` object, that sup
 
     * If `data` is binary encoded it will be sent as non-text (as specified by the WebSocket Protocol).
 
-    A server **SHOULD** document whether `write` will block. It is **RECOMMENDED** that servers implement buffered IO, allowing `write` to return immediately when resources allow and block (or, possibly, disconnect) when the IO buffer is full.
+    A server **SHOULD** document its concurrency model, allowing developers to know whether `write` will block or not, whether buffered IO is implemented, etc'.
+
+    For example, evented servers are encouraged to avoid blocking and return immediately, deferring the actual `write` operation for later. However, (process/thread/fiber) per-connection based servers **MAY** choose to return only after all the data was sent. Documenting these differences will allows applications to choose the model that best fits their needs and environments.
 
-* `close` closes the connection once all the data in the outgoing queue was sent. If `close` is called while there is still data to be sent, `close` will only take effect once the data was sent.
+* `close` closes the connection once all the data scheduled using `write` was sent. If `close` is called while there is still data to be sent, `close` **SHOULD** return immediately and only take effect once the data was sent.
 
     `close` shall always return `nil`.
 
-* `open?` returns `true` if the connection isn't known to have been closed and `false` if the connection is known to be closed or marked to be closed.
+* `open?` **MUST** return `false` **if** the connection was never open, is known to be closed or marked to be closed. Otherwise `true` **MUST** be returned.
 
 * `pending` **MUST** return -1 if the connection is closed. Otherwise, `pending` **SHOULD** return the number of pending writes (messages in the `write` queue\*) that need to be processed before the next time the `on_drained` callback is called.
 
-    Servers **MAY** choose to always return the value `0` if they never call the `on_drained` callback and the connection is open.
+    Servers **MAY** choose to always return the value `0` **ONLY IF** they never call the `on_drained` callback and the connection is open.
+
+    Servers that return a positive number **MUST** call the `on_drained` callback when a call to `pending` would return the value `0`.
+
+    \*Servers that divide large messages into a number of smaller messages (implement message fragmentation) **MAY** count each fragment separately, as if the fragmentation was performed by the user and `write` was called more than once per message.
+
+* `pubsub?` **MUST** return `false` **unless** the pub/sub extension is supported.
+
+   Pub/Sub patterns are idiomatic for WebSockets and EventSource connections but their API is out of scope for this extension.
 
-    Servers that return a positive number MUST call the `on_drained` callback when a call to `pending` would return the value `0`.
+* `class` **MUST** return the client's Class, allowing it be extended with additional features (such as Pub/Sub, etc').
 
-    \*Servers that divide large messages into a number of smaller messages (implement message fragmentation) MAY count each fragment separately, as if the fragmentation was performed by the user and `write` was called more than once per message.
+    **Note**: Ruby adds this method automatically to every class, no need to do a thing.
 
-* `env` returns the Rack `env` hash related to the originating HTTP request. Some changes to the `env` hash (such as removal of the IO hijacking support) MAY be implemented by the server.
+The server **MAY** support the following (optional) methods for the `client` object:
 
-WebSocket `ping` / `pong`, timeouts and network considerations should be implemented by the server. It is **RECOMMENDED** (but not required) that the server send `ping`s to prevent connection timeouts and detect network failure.
+* `handler` if implemented, **MUST** return the callback object linked to the `client` object.
 
-Server settings **MAY** (not required) be provided to allow for customization and adaptation for different network environments or WebSocket extensions. It is **RECOMMENDED** that any settings be available as command line arguments and **not** incorporated into the application's logic.
+* `handler=` if implemented, **MUST** set a new Callback Object for `client`.
+
+    This allows applications to switch from one callback object to another (i.e., in case of credential upgrades).
+
+    Once a new Callback Object was set, the server **MUST** call the old handler's `on_close` callback and **afterwards** call the new handler's `on_open` callback.
+
+    It is **RECOMMENDED** (but not required) that this also updates the value for `env['rack.upgrade']`.
+
+* `timeout` / `timeout=` allows applications to get / set connection timeouts dynamically and separately for each connection. Servers **SHOULD** provide a global setting for the default connection timeout. It is **RECOMMENDED** (but not required) that a global / default timeout setting be available from the command line (CLI).
+
+* `protocol` if implemented, **MUST** return the same value that was originally set by `env['rack.upgrade?']`.
+
+
+WebSocket `ping` / `pong`, timeouts and network considerations **SHOULD** be implemented by the server. It is **RECOMMENDED** (but not required) that the server send `ping`s to prevent connection timeouts and to detect network failure. Clients **SHOULD** also consider sending `ping`s to detect network errors (dropped connections).
+
+Server settings **MAY** be provided to allow for customization and adaptation for different network environments or WebSocket extensions. It is **RECOMMENDED** that any settings be available as command line arguments and **not** incorporated into the application's logic.
 
 ---
 
@@ -100,62 +135,65 @@ Server settings **MAY** (not required) be provided to allow for customization an
 
 * **Server**:
 
+    When a regular HTTP request arrives (non-upgradeable), the server will set the `env['rack.upgrade?']` flag to `false`, indicating that: 1. this specific request is NOT upgradable; and 2. the server supports this specification for either WebSocket and/or EventSource connections.
+
     When a WebSocket upgrade request arrives, the server will set the `env['rack.upgrade?']` flag to `:websocket`, indicating that: 1. this specific request is upgradable; and 2. the server supports this specification for WebSocket connections.
 
-    When an EventSource request arrives, the server will set the `env['rack.upgrade?']` flag to `:sse`, indicating that: 1. this specific request is an EventSource request; and 2. the server supports this specification.
+    When an EventSource request arrives, the server will set the `env['rack.upgrade?']` flag to `:sse`, indicating that: 1. this specific request is an EventSource request; and 2. the server supports this specification for EventSource connections.
 
 * **Client**:
 
-    When a client decides to upgrade a request, they will place an appropriate Callback Object in the `env['rack.upgrade']` Hash key.
+    If a client decides to upgrade a request, they will place an appropriate Callback Object in the `env['rack.upgrade']` Hash key.
 
 * **Server**:
 
-    1. The server will review the `env` Hash *before* sending the response. If the `env['rack.upgrade']` was set, the server will perform the upgrade.
+    1. If the application's response status indicates an error or a redirection (status code `>= 300`), the server shall ignore the Callback Object and/or remove it from the `env` Hash, ignoring the rest of the steps that follow.
 
-    2. The server will send the correct response status and headers, as well as any headers present in the response object. The server will also perform any required housekeeping, such as closing the response body, if it exists.
+    2. The server will review the `env` Hash *before* sending the response. If the `env['rack.upgrade']` was set, the server will perform the upgrade.
 
-         The response status provided by the response object shall be ignored and the correct response status shall be set by the server.
+    3. The server will send the correct response status and headers, as well as any headers present in the response object. The server will also perform any required housekeeping, such as closing the response body, if it exists.
 
-         If the application's response status indicates an error or a redirection (status code >= 300), the server shall ignore the Callback Object.
+         The response status provided by the response object shall be ignored and the correct response status shall be set by the server.
 
-    3. Once the upgrade had completed, the server will call the `on_open` callback.
+    4. Once the upgrade had completed, the server will call the `on_open` callback.
 
         No other callbacks shall be called until the `on_open` callback had returned.
 
-        WebSocket messages shall be handled by the `on_message` callback in the same order in which they arrive and the `on_message` will **not** be executed concurrently for the same connection.
+        WebSocket messages shall be handled by the `on_message` callback in the same order in which they arrive and the `on_message` **SHOULD NOT** be executed concurrently for the same connection.
 
-        The `on_close` callback will **not** be called while any other callback is running (`on_open`, `on_message`, `on_drained`, etc').
+        The `on_close` callback **MUST NOT** be called while any other callback is running (`on_open`, `on_message`, `on_drained`, etc').
 
-        The `on_drained` callback might be called concurrently with the `on_message` callback, allowing data to be sent even while other data is being processed. Multi-threading considerations may apply.
+        The `on_drained` callback **MAY** be called concurrently with the `on_message` callback, allowing data to be sent even while incoming data is being processed. Multi-threading considerations apply.
 
 ## Example Usage
 
 The following is an example WebSocket echo server implemented using this specification:
 
 ```ruby
-class WSConnection
+module WSConnection
     def on_open(client)
-        puts "WebSocket connection established."
+        puts "WebSocket connection established (#{client.object_id})."
     end
     def on_message(client, data)
-        client.write data
+        client.write data # echo the data back
         puts "on_drained MUST be implemented if #{ pending } != 0."
     end
     def on_drained(client)
-        puts "Yap,on_drained is implemented."
+        puts "If this line prints out, on_drained is supported by the server."
     end
     def on_shutdown(client)
         client.write "The server is going away. Goodbye."
     end
     def on_close(client)
-        puts "WebSocket connection closed."
+        puts "WebSocket connection closed (#{client.object_id})."
     end
+    extend self
 end
 
 module App
    def self.call(env)
        if(env['rack.upgrade?'.freeze] == :websocket)
-           env['rack.upgrade'.freeze] = WSConnection.new
+           env['rack.upgrade'.freeze] = WSConnection
            return [0, {}, []]
        end
        return [200, {"Content-Length" => "12", "Content-Type" => "text/plain"}, ["Hello World!"]]
@@ -165,23 +203,22 @@ end
 run App
 ```
 
-The following example uses Push notifications for both WebSocket and SSE connections. The Pub/Sub API isn't part of this specification but it is supported by iodine:
+The following example uses Push notifications for both WebSocket and SSE connections. The Pub/Sub API is subject to a separate Pub/Sub API extension and isn't part of this specification (it is, however, supported by iodine):
 
 ```ruby
-class Chat
-    def initialize(nickname)
-        @nickname = nickname
-    end
+module Chat
     def on_open(client)
+        client.class.prepend MyPubSubModule unless client.pubsub?
         client.subscribe "chat"
-        client.publish "chat", "#{@nickname} joined the chat."
+        client.publish "chat", "#{env[:nickname]} joined the chat."
     end
     def on_message(client, data)
-        client.publish "chat", "#{@nickname}: #{data}"
+        client.publish "chat", "#{env[:nickname]}: #{data}"
     end
     def on_close(client)
-        client.publish "chat", "#{@nickname}: left the chat."
+        client.publish "chat", "#{env[:nickname]}: left the chat."
     end
+    extend self
 end
 
 module App
@@ -189,7 +226,7 @@ module App
        if(env['rack.upgrade?'.freeze])
            nickname = env['PATH_INFO'][1..-1]
            nickname = "Someone" if nickname == "".freeze
-           env['rack.upgrade'.freeze] = Chat.new(nickname)
+           env[:nickname] = nickname
            return [0, {}, []]
        end
        return [200, {"Content-Length" => "12", "Content-Type" => "text/plain"}, ["Hello World!"]]

data/examples/async_task.ru

@@ -0,0 +1,92 @@
+# This is a task scheduling WebSocket push example application.
+#
+# Benchmark HTTPP with `ab` or `wrk` (a 5 seconds benchmark with 2000 concurrent clients):
+#
+#      ab -c 2000 -t 5 -n 1000000 -k http://127.0.0.1:3000/
+#      wrk -c2000 -d5 -t12 http://localhost:3000/
+#
+# Test websocket tasks using the browser. For example:
+#      ws = new WebSocket("ws://localhost:3000/userID"); ws.onmessage = function(e) {console.log(e.data);}; ws.onclose = function(e) {console.log("closed")};
+#      ws.onopen = function(e) {ws.send(JSON.stringify({'task': 'echo', 'data': 'Hello!'}));};
+require 'iodine'
+require 'json'
+
+TASK_PUBLISHING_ENGINE = Iodine::PubSub::PROCESS
+
+# This module handles tasks and send them back to the frontend
+module TaskHandler
+  def echo msg
+    msg = Iodine::JSON.parse(msg, symbolize_names: true)
+    publish_to = msg.delete(:from)
+    Iodine.publish(publish_to, msg.to_json, TASK_PUBLISHING_ENGINE) if publish_to
+    puts "performed 'echo' task"
+  rescue => e
+    puts "JSON task message error? #{e.message} - under attack?"
+  end
+
+  def add msg
+    msg = Iodine::JSON.parse(msg, symbolize_names: true)
+    raise "addition task requires an array of numbers" unless msg[:data].is_a?(Array)
+    msg[:data] = msg[:data].inject(0){|sum,x| sum + x }
+    publish_to = msg.delete(:from)
+    Iodine.publish(publish_to, msg.to_json, TASK_PUBLISHING_ENGINE) if publish_to
+    puts "performed 'add' task"
+  rescue => e
+    puts
+     "JSON task message error? #{e.message} - under attack?"
+  end
+
+  def listen2tasks
+    Iodine.subscribe(:echo) {|ch,msg| TaskHandler.echo(msg) }
+    Iodine.subscribe(:add) {|ch,msg| TaskHandler.add(msg) }
+  end
+
+  extend self
+end
+
+module WebsocketClient
+  def on_open client
+    # Pub/Sub directly to the client (or use a block to process the messages)
+    client.subscribe client.env['PATH_INFO'.freeze]
+  end
+  def on_message client, data
+    # Strings and symbol channel names are equivalent.
+    msg = Iodine::JSON.parse(data, symbolize_names: true)
+    raise "no valid task" unless ["echo".freeze, "add".freeze].include? msg[:task]
+    msg[:from] = client.env['PATH_INFO'.freeze]
+    client.publish msg[:task], msg.to_json, TASK_PUBLISHING_ENGINE
+  rescue => e
+      puts "JSON message error? #{e.message}\n\t#{data}\n\t#{msg}"
+  end
+  extend self
+end
+
+APP = Proc.new do |env|
+  if env['rack.upgrade?'.freeze] == :websocket 
+    env['rack.upgrade'.freeze] = WebsocketClient 
+    [0,{}, []] # It's possible to set cookies for the response.
+  elsif env['rack.upgrade?'.freeze] == :sse
+    puts "SSE connections can only receive data from the server, the can't write." 
+    env['rack.upgrade'.freeze] = WebsocketClient
+    [0,{}, []] # It's possible to set cookies for the response.
+  else
+    [200, {"Content-Type" => "text/plain"}, ["Send messages with WebSockets using JSON.\ni.e.: {\"task\":\"add\", \"data\":[1,2]}"]]
+  end
+end
+
+# test automatically for Redis extensions.
+if(Iodine::PubSub.default.is_a? Iodine::PubSub::Redis)
+  TASK_PUBLISHING_ENGINE = Iodine::PubSub.default
+  if(ARGV.include? "worker")
+    TaskHandler.listen2tasks
+    Iodine.workers = 1
+    Iodine.threads = 16 if Iodine.threads == 0
+    Iodine.start
+    exit(0)
+  end
+else
+  TaskHandler.listen2tasks
+end
+
+# # or in config.ru
+run APP

data/exe/iodine

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
+IODINE_PARSE_CLI = true
 require 'iodine'
 
 # Load Rack if available (assume it will be used).

data/ext/iodine/extconf.rb

@@ -1,34 +1,5 @@
 require 'mkmf'
 
-if ENV['CC']
-  ENV['CPP'] ||= ENV['CC']
-  puts "detected user prefered compiler (#{ENV['CC']}):", `#{ENV['CC']} -v`
-elsif find_executable('clang') && puts('testing clang for stdatomic support...').nil? && system("printf \"\#include <stdatomic.h>\nint main(void) {}\" | clang -include stdatomic.h -xc -o /dev/null -", out: '/dev/null')
-  $CC = ENV['CC'] = 'clang'
-  $CPP = ENV['CPP'] = 'clang'
-  puts "using clang compiler v. #{`clang -dumpversion`}."
-elsif find_executable('gcc') && (`gcc -dumpversion 2>&1`.to_i >= 5)
-  $CC = ENV['CC'] = 'gcc'
-  $CPP = ENV['CPP'] = find_executable('g++') ? 'g++' : 'gcc'
-  puts "using gcc #{ `gcc -dumpversion 2>&1`.to_i }"
-elsif find_executable('gcc-6')
-  $CC = ENV['CC'] = 'gcc-6'
-  $CPP = ENV['CPP'] = find_executable('g++-6') ? 'g++-6' : 'gcc-6'
-  puts 'using gcc-6 compiler.'
-elsif find_executable('gcc-5')
-  $CC = ENV['CC'] = 'gcc-5'
-  $CPP = ENV['CPP'] = find_executable('g++-5') ? 'g++-5' : 'gcc-5'
-  puts 'using gcc-5 compiler.'
-elsif find_executable('gcc-4.9')
-  $CC = ENV['CC'] = 'gcc-4.9'
-  $CPP = ENV['CPP'] = find_executable('g++-4.9') ? 'g++-4.9' : 'gcc-4.9'
-  puts 'using gcc-4.9 compiler.'
-else
-  puts 'using an unknown (old?) compiler... who knows if this will work out... we hope.'
-end
-
-
-
 # Test polling
 def iodine_test_polling_support
   iodine_poll_test_kqueue = <<EOS
@@ -129,10 +100,5 @@ EOS
     end
   end
 end
-# $defs << "-DFIO_USE_RISKY_HASH"
-
-RbConfig::MAKEFILE_CONFIG['CFLAGS'] = $CFLAGS = "-std=c11 -DFIO_PRINT_STATE=0 #{$CFLAGS} #{$CFLAGS == ENV['CFLAGS'] ? "" : ENV['CFLAGS']}"
-RbConfig::MAKEFILE_CONFIG['CC'] = $CC = ENV['CC'] if ENV['CC']
-RbConfig::MAKEFILE_CONFIG['CPP'] = $CPP = ENV['CPP'] if ENV['CPP']
 
 create_makefile 'iodine/iodine'

data/ext/iodine/fio.c

@@ -36,6 +36,12 @@ Feel free to copy, use and enjoy according to the license provided.
 
 #include <arpa/inet.h>
 
+#if HAVE_OPENSSL
+#include <openssl/bio.h>
+#include <openssl/err.h>
+#include <openssl/ssl.h>
+#endif
+
 /* force poll for testing? */
 #ifndef FIO_ENGINE_POLL
 #define FIO_ENGINE_POLL 0
@@ -277,26 +283,6 @@ static inline fio_packet_s *fio_packet_alloc(void) {
 Core Connection Data Clearing
 ***************************************************************************** */
 
-/* set the minimal max_protocol_fd */
-static void fio_max_fd_min(uint32_t fd) {
-  if (fio_data->max_protocol_fd > fd)
-    return;
-  fio_lock(&fio_data->lock);
-  if (fio_data->max_protocol_fd < fd)
-    fio_data->max_protocol_fd = fd;
-  fio_unlock(&fio_data->lock);
-}
-
-/* set the minimal max_protocol_fd */
-static void fio_max_fd_shrink(void) {
-  fio_lock(&fio_data->lock);
-  uint32_t fd = fio_data->max_protocol_fd;
-  while (fd && fd_data(fd).protocol == NULL)
-    --fd;
-  fio_data->max_protocol_fd = fd;
-  fio_unlock(&fio_data->lock);
-}
-
 /* resets connection data, marking it as either open or closed. */
 static inline int fio_clear_fd(intptr_t fd, uint8_t is_open) {
   fio_packet_s *packet;
@@ -318,6 +304,13 @@ static inline int fio_clear_fd(intptr_t fd, uint8_t is_open) {
       .counter = fd_data(fd).counter + 1,
       .packet_last = &fd_data(fd).packet,
   };
+  if (fio_data->max_protocol_fd < fd) {
+    fio_data->max_protocol_fd = fd;
+  } else {
+    while (fio_data->max_protocol_fd &&
+           !fd_data(fio_data->max_protocol_fd).open)
+      --fio_data->max_protocol_fd;
+  }
   fio_unlock(&(fd_data(fd).sock_lock));
   if (rw_hooks && rw_hooks->cleanup)
     rw_hooks->cleanup(rw_udata);
@@ -336,8 +329,8 @@ static inline int fio_clear_fd(intptr_t fd, uint8_t is_open) {
   if (protocol && protocol->on_close) {
     fio_defer(deferred_on_close, (void *)fd2uuid(fd), protocol);
   }
-  if (is_open)
-    fio_max_fd_min(fd);
+  FIO_LOG_DEBUG("FD %d re-initialized (state: %p-%s).", (int)fd,
+                (void *)fd2uuid(fd), (is_open ? "open" : "closed"));
   return 0;
 }
 
@@ -2887,7 +2880,7 @@ void fio_close(intptr_t uuid) {
   }
   if (uuid_data(uuid).packet || uuid_data(uuid).sock_lock) {
     uuid_data(uuid).close = 1;
-    fio_poll_add_write(fio_uuid2fd(uuid));
+    fio_force_event(uuid, FIO_EVENT_ON_READY);
     return;
   }
   fio_force_close(uuid);
@@ -3019,6 +3012,7 @@ test_errno:
   case ENOSPC:        /* fallthrough */
   case EADDRNOTAVAIL: /* fallthrough */
   case EINTR:
+  case 0:
     return 1;
   case EFAULT:
     FIO_LOG_ERROR("fio_flush EFAULT - possible memory address error sent to "
@@ -3032,8 +3026,8 @@ test_errno:
     fio_force_close(uuid);
     return -1;
   }
-  fprintf(stderr, "UUID error: %p (%d)\n", (void *)uuid, errno);
-  perror("No errno handler");
+  FIO_LOG_DEBUG("UUID error: %p (%d): %s\n", (void *)uuid, errno,
+                strerror(errno));
   return 0;
 
 invalid:
@@ -3099,6 +3093,19 @@ const fio_rw_hook_s FIO_DEFAULT_RW_HOOKS = {
     .cleanup = fio_hooks_default_cleanup,
 };
 
+static inline void fio_rw_hook_validate(fio_rw_hook_s *rw_hooks) {
+  if (!rw_hooks->read)
+    rw_hooks->read = fio_hooks_default_read;
+  if (!rw_hooks->write)
+    rw_hooks->write = fio_hooks_default_write;
+  if (!rw_hooks->flush)
+    rw_hooks->flush = fio_hooks_default_flush;
+  if (!rw_hooks->before_close)
+    rw_hooks->before_close = fio_hooks_default_before_close;
+  if (!rw_hooks->cleanup)
+    rw_hooks->cleanup = fio_hooks_default_cleanup;
+}
+
 /**
  * Replaces an existing read/write hook with another from within a read/write
  * hook callback.
@@ -3112,19 +3119,10 @@ int fio_rw_hook_replace_unsafe(intptr_t uuid, fio_rw_hook_s *rw_hooks,
   int replaced = -1;
   uint8_t was_locked;
   intptr_t fd = fio_uuid2fd(uuid);
-  if (!rw_hooks->read)
-    rw_hooks->read = fio_hooks_default_read;
-  if (!rw_hooks->write)
-    rw_hooks->write = fio_hooks_default_write;
-  if (!rw_hooks->flush)
-    rw_hooks->flush = fio_hooks_default_flush;
-  if (!rw_hooks->before_close)
-    rw_hooks->before_close = fio_hooks_default_before_close;
-  if (!rw_hooks->cleanup)
-    rw_hooks->cleanup = fio_hooks_default_cleanup;
+  fio_rw_hook_validate(rw_hooks);
   /* protect against some fulishness... but not all of it. */
   was_locked = fio_trylock(&fd_data(fd).sock_lock);
-  if (fd2uuid(fd) == uuid) {
+  if (uuid_is_valid(uuid)) {
     fd_data(fd).rw_hooks = rw_hooks;
     fd_data(fd).rw_udata = udata;
     replaced = 0;
@@ -3138,16 +3136,7 @@ int fio_rw_hook_replace_unsafe(intptr_t uuid, fio_rw_hook_s *rw_hooks,
 int fio_rw_hook_set(intptr_t uuid, fio_rw_hook_s *rw_hooks, void *udata) {
   if (fio_is_closed(uuid))
     goto invalid_uuid;
-  if (!rw_hooks->read)
-    rw_hooks->read = fio_hooks_default_read;
-  if (!rw_hooks->write)
-    rw_hooks->write = fio_hooks_default_write;
-  if (!rw_hooks->flush)
-    rw_hooks->flush = fio_hooks_default_flush;
-  if (!rw_hooks->before_close)
-    rw_hooks->before_close = fio_hooks_default_before_close;
-  if (!rw_hooks->cleanup)
-    rw_hooks->cleanup = fio_hooks_default_cleanup;
+  fio_rw_hook_validate(rw_hooks);
   intptr_t fd = fio_uuid2fd(uuid);
   fio_rw_hook_s *old_rw_hooks;
   void *old_udata;
@@ -3249,7 +3238,6 @@ static int fio_attach__internal(void *uuid_, void *protocol_) {
     /* adding a new uuid to the reactor */
     fio_poll_add(fio_uuid2fd(uuid));
   }
-  fio_max_fd_min(fio_uuid2fd(uuid));
   return 0;
 
 invalid_uuid:
@@ -3510,18 +3498,19 @@ static void fio_on_fork(void) {
   fio_poll_init();
   fio_state_callback_on_fork();
 
+  /* don't pass open connections belonging to the parent onto the child. */
   const size_t limit = fio_data->capa;
   for (size_t i = 0; i < limit; ++i) {
     fd_data(i).sock_lock = FIO_LOCK_INIT;
     fd_data(i).protocol_lock = FIO_LOCK_INIT;
-    if (fd_data(i).protocol) {
+    if (fd_data(i).protocol && fd_data(i).open) {
+      /* open without protocol might be waiting for the child (listening) */
       fd_data(i).protocol->rsv = 0;
       fio_force_close(fd2uuid(i));
     }
   }
 
   fio_pubsub_on_fork();
-  fio_max_fd_shrink();
   uint16_t old_active = fio_data->active;
   fio_data->active = 0;
   fio_defer_perform();
@@ -3681,24 +3670,30 @@ static void fio_review_timeout(void *arg, void *ignr) {
   uint16_t timeout = fd_data(fd).timeout;
   if (!timeout)
     timeout = 300; /* enforced timout settings */
-  if (!fd_data(fd).protocol || (fd_data(fd).active + timeout >= review))
+  if (!fd_data(fd).open || fd_data(fd).active + timeout >= review)
     goto finish;
-  tmp = protocol_try_lock(fd, FIO_PR_LOCK_STATE);
-  if (!tmp) {
-    if (errno == EBADF)
-      goto finish;
-    goto reschedule;
+  if (fd_data(fd).protocol) {
+    tmp = protocol_try_lock(fd, FIO_PR_LOCK_STATE);
+    if (!tmp) {
+      if (errno == EBADF)
+        goto finish;
+      goto reschedule;
+    }
+    if (prt_meta(tmp).locks[FIO_PR_LOCK_TASK] ||
+        prt_meta(tmp).locks[FIO_PR_LOCK_WRITE])
+      goto unlock;
+    fio_defer_push_task(deferred_ping, (void *)fio_fd2uuid((int)fd), NULL);
+  unlock:
+    protocol_unlock(tmp, FIO_PR_LOCK_STATE);
+  } else {
+    /* open FD but no protocol? RW hook thing or listening sockets? */
+    if (fd_data(fd).rw_hooks != &FIO_DEFAULT_RW_HOOKS)
+      fio_close(fd2uuid(fd));
   }
-  if (prt_meta(tmp).locks[FIO_PR_LOCK_TASK] ||
-      prt_meta(tmp).locks[FIO_PR_LOCK_WRITE])
-    goto unlock;
-  fio_defer_push_task(deferred_ping, (void *)fio_fd2uuid((int)fd), NULL);
-unlock:
-  protocol_unlock(tmp, FIO_PR_LOCK_STATE);
 finish:
   do {
     fd++;
-  } while (!fd_data(fd).protocol && (fd <= fio_data->max_protocol_fd));
+  } while (!fd_data(fd).open && (fd <= fio_data->max_protocol_fd));
 
   if (fio_data->max_protocol_fd < fd) {
     fio_data->need_review = 1;
@@ -3714,7 +3709,6 @@ static void fio_cycle_schedule_events(void) {
   static time_t last_to_review = 0;
   fio_mark_time();
   fio_timer_schedule();
-  fio_max_fd_shrink();
   if (fio_signal_children_flag) {
     /* hot restart support */
     fio_signal_children_flag = 0;
@@ -3923,16 +3917,22 @@ void fio_start FIO_IGNORE_MACRO(struct fio_start_args args) {
   fio_data->is_worker = 0;
 
   fio_state_callback_force(FIO_CALL_PRE_START);
-
   FIO_LOG_INFO(
       "Server is running %u %s X %u %s with facil.io " FIO_VERSION_STRING
       " (%s)\n"
+#if HAVE_OPENSSL
+      "* Linked to %s\n"
+#endif
       "* Detected capacity: %d open file limit\n"
       "* Root pid: %d\n"
       "* Press ^C to stop\n",
       fio_data->workers, fio_data->workers > 1 ? "workers" : "worker",
       fio_data->threads, fio_data->threads > 1 ? "threads" : "thread",
-      fio_engine(), fio_data->capa, (int)fio_data->parent);
+      fio_engine(),
+#if HAVE_OPENSSL
+      OpenSSL_version(0),
+#endif
+      fio_data->capa, (int)fio_data->parent);
 
   if (args.workers > 1) {
     for (int i = 0; i < args.workers && fio_data->active; ++i) {