hyperion-rb 1.4.0 → 1.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94ad9628c5683d6b9c9d2d8359066e4bda37d29c7ec2e4fb4e48ae726f8ff939
-  data.tar.gz: 7ebf3c87d10a384e388cf7f2c28d276c0975044a49fe09e1174ffa331cab76b4
+  metadata.gz: fd27fc15210a2436de9b88664413eeb7cd7cf129e0921f88d7d15edd87b76d11
+  data.tar.gz: 8dd98630e170ee1467197542d69d034b15aa1d1a1bc2441e780d3290c2c59f1d
 SHA512:
-  metadata.gz: 9fb844690d5a06aa55335211bdaf3a5276b48f7cfd8a248bb714258ae02549a7e595bc1088987ae9c77d3f28aa1efec91fcd8cc401c26264620f48d78d85faf3
-  data.tar.gz: d050bffe3c8fed2ae1ce434fb7ccf8904a418e3fa18c6afd3e4314b841215ca0dec621edb7bc5f80350e4807e04503b1e785786fb57503b716e52afaf2e30a20
+  metadata.gz: a2ae7959ab5fe99207c2c53db22060385a6eeb3aad355d05b4b6f14b16c16d1057592eda2f63b2a72cbfeb62ec7ebae5ec5945f317e84a27c5bac9c6fdfbe614
+  data.tar.gz: 6fb3d77d7c631b98e366ef921859dcc9af1f5f2e3f6eb6e97bebfd9453f6403c6846d46c66628204785bd9c0253e01984696b59eab690c53a79f3238c6f1f15a

data/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## [1.4.2] - 2026-04-27
+
+Audit-driven cleanup. No behaviour changes; fiber-correctness + docs polish.
+
+### Fixed
+- **`Hyperion::Logger` access buffer was fiber-local, not thread-local** — pre-1.4.2 the access-log write buffer was stored via `Thread.current[@buffer_key]`. Under an `Async::Scheduler` (TLS / h2 / `--async-io` plain HTTP/1.1) every handler fiber got its own private buffer, so the 4 KiB `ACCESS_FLUSH_BYTES` batching never fired — each fiber's buffer accumulated 1-3 lines before its connection closed and `flush_access_buffer` wrote them. At 24k r/s this meant ~12-24k `write(2)` syscalls/sec instead of the designed ~750/sec. Switched to `Thread#thread_variable_*` so all fibers on the same OS thread share one buffer and the batching actually fires. Same root cause as the 1.4.1 Metrics fix; surfaced by a code-audit grep for residual `Thread.current[:key]` patterns.
+- **`Logger#cached_timestamp` and `ResponseWriter#cached_date`** — same fix. Pre-1.4.2 the per-second / per-millisecond Time-formatting caches were per-fiber, so under Async every fiber rebuilt the iso8601 / httpdate String on its first call after a tick. Now per-OS-thread, shared across fibers; one allocation per second per thread total.
+
+### Added
+- **Prometheus exporter example output** in the README's Metrics section — shows what `curl -H 'X-Hyperion-Admin-Token: ...' /-/metrics` actually returns (HELP/TYPE lines, status-code labels, auto-export of unknown counters), plus the Prometheus scraper config sketch.
+- **Regression spec** for the access-buffer cross-fiber bug — two fibers on the same OS thread write through one logger; verifies a single buffer is registered (not one per fiber) and both lines land via `flush_all`.
+- **4 new Metrics specs** (already shipped in 1.4.1; called out here for coverage tracking) — cross-fiber on same thread, cross-thread, cross-fiber-on-different-thread, many-fibers-on-same-thread.
+
+### Changed
+- **README benchmark section** version-stamped: clarifies that the headline numbers were measured against the noted Hyperion version (most are 1.2.0 hello-world / 1.3.0 PG-bound) and that 1.3.0+ `--async-io` + 1.4.0+ TLS-inline + 1.4.1+ Metrics fix preserve or improve these numbers. We re-run the headline configs each release.
+
+## [1.4.1] - 2026-04-27
+
+### Fixed
+- **`Hyperion::Metrics` fiber-key bug** — pre-1.4.1 the metrics module stored counters via `Thread.current[:key]`, which is FIBER-local in Ruby 1.9+. Under an `Async::Scheduler` (TLS / h2 / `--async-io` plain HTTP/1.1) every handler fiber got its own private counters Hash that `Hyperion.stats` could never see — increments were stranded, the dispatch counters and `:bytes_written` etc. read as zero from any non-handler-fiber observer (including the Prometheus `/-/metrics` exporter when scraped from a different fiber). Switched to `Thread#thread_variable_*` (truly thread-local across fibers) plus direct counter-Hash list storage so snapshots also survive thread death. Verified via 4 new specs: cross-fiber on same thread, cross-thread, cross-fiber-on-different-thread, many-fibers-on-same-thread (210 increments aggregated correctly). Surfaced by hyperion-async-pg 0.4.0's bench round, which couldn't read `:requests_async_dispatched` from spec assertions even though the increments were firing.
+
 ## [1.4.0] - 2026-04-27
 
 Default-behaviour change for TLS users: HTTP/1.1-over-TLS now dispatches inline on the calling fiber instead of hopping through the worker thread pool. Fiber-cooperative libraries (`hyperion-async-pg`, `async-redis`) work on the TLS h1 path without `--async-io`. No code-path changes for plain HTTP/1.1 default behaviour.

data/README.md

@@ -25,7 +25,7 @@ bundle exec hyperion config.ru
 
 ## Benchmarks
 
-All numbers are real wrk runs against published Hyperion configs. Hyperion ships **with default-ON structured access logs**; Puma comparisons use Puma defaults (no per-request log emission).
+All numbers are real wrk runs against published Hyperion configs. Hyperion ships **with default-ON structured access logs**; Puma comparisons use Puma defaults (no per-request log emission). Each section is stamped with the Hyperion version it was measured against — newer versions (1.3.0+ `--async-io`, 1.4.0+ TLS h1 inline, 1.4.1+ Metrics fiber-key fix) preserve or improve these numbers; we re-run the headline configs each release and have not seen regressions on these workloads.
 
 ### Hello-world Rack app
 
@@ -95,7 +95,7 @@ Ubuntu 24.04 / 16 vCPU / Ruby 3.3.3, Postgres 17 over WAN, `wrk -t4 -c200 -d20s`
 1. **Linear scaling with pool size** under `--async-io` — `r/s ≈ pool × 12` on this WAN bench. Single-worker pool=200 hits 2381 r/s, **42× Puma `-t 5`** and **5.9× Puma's best** (`-t 30`).
 2. **Mixed workload doesn't kill the win** — Hyperion `--async-io` pool=128 actually goes *up* on mixed (1740 vs 1344 r/s) because CPU work overlaps other fibers' PG-wait windows. This is the honest "what happens to a real Rails handler" answer.
 3. **Hyperion ≈ Falcon within 3-7%** across pool sizes; both fiber-native architectures extract similar value from `hyperion-async-pg`.
-4. **RSS at single-worker scale isn't the architectural moat** — Linux thread stacks are demand-paged; PG connection buffers dominate RSS at pool sizes ≤ 200. The MB-vs-GB story shows up at **idle keep-alive connection scale** (10k+ conns), not in this PG-bound throughput bench. See [Concurrency at scale](#concurrency-at-scale-architectural-advantages) for the connection-count win.
+4. **RSS at single-worker scale isn't the architectural moat** — Linux thread stacks are demand-paged; PG connection buffers dominate RSS at pool sizes ≤ 200. The architectural win is **handler concurrency under load**, not idle memory: Hyperion's fiber path runs thousands of in-flight handler invocations per OS thread, so wait-bound handlers don't queue at `max_threads`. See [Concurrency at scale](#concurrency-at-scale-architectural-advantages) for both the throughput-under-load row and a measured 10k-idle-keepalive RSS sweep against Puma and Falcon.
 5. **`-w 4` cold-start caveat** — multi-worker p99 inflates because the bench rackup uses lazy per-process pool init (each worker pays full pool fill on its first request). Production apps avoid this with `on_worker_boot { Hyperion::AsyncPg::FiberPool.new(...).fill }`.
 
 Three things must all be true to get this win:
@@ -176,7 +176,21 @@ These workloads demonstrate structural differences between Hyperion's fiber-per-
 | Hyperion `-w 1 -t 10` | 93,090 | 6,910 | 3,446 | 27.01 s |
 | Puma `-w 1 -t 10:10`  | 77,340 | 22,660 | 706 | 109.59 s |
 
-Hyperion holds each connection in a ~1 KB fiber stack; Puma needs an OS thread (~1–8 MB each, capped at `max_threads`). At 10k concurrent connections Hyperion serves **~5× the throughput** of Puma with **~20% fewer dropped requests**, while the per-connection bookkeeping cost is bounded by fiber size, not by `max_threads`.
+At 10k concurrent connections under load Hyperion serves **~5× the throughput** of Puma with **~20% fewer dropped requests**. The per-connection bookkeeping cost is bounded by fiber size, not by `max_threads` — workers don't get pinned to long-lived sockets, so a slow handler doesn't starve other connections.
+
+**Memory at idle keep-alive scale — 10,000 idle HTTP/1.1 keep-alive connections:**
+
+Each client opens a TCP connection, sends one keep-alive GET, drains the response, then holds the socket open without sending a follow-up request. RSS is sampled once a second across a 30s idle hold. Same hello-world rackup, single worker, no TLS. Hyperion runs with `async_io true` (fiber-per-connection on the plain HTTP/1.1 path).
+
+| | held | dropped | peak RSS | RSS after drain |
+|---|---:|---:|---:|---:|
+| Hyperion `-w 1 -t 5 --async-io` | 10,000 / 10,000 | 0 | 173 MB | 155 MB |
+| Puma `-w 0 -t 100`               | 10,000 / 10,000 | 0 | 101 MB | 104 MB |
+| Falcon `--count 1`               | 10,000 / 10,000 | 0 | 429 MB | 440 MB |
+
+All three hold 10k idle conns without OOMing or dropping — the "MB-per-thread" intuition that thread-based servers can't reach this scale doesn't survive contact with Linux's demand-paged thread stacks plus Puma's reactor-based keep-alive handling. Per-conn RSS lands at ~14 KB (Hyperion fiber + parser state), ~7 KB (Puma reactor entry + tiny thread share), ~36 KB (Falcon Async::Task + protocol-http stack). Bounded, not unbounded — for all three.
+
+The architectural difference shows up under **load**, not at idle: Puma can only run `max_threads` handler invocations concurrently, so wait-bound handlers (DB, HTTP, Redis) starve at higher request concurrency than `max_threads`. Hyperion's fiber-per-connection model + `--async-io` gives one OS thread thousands of in-flight handler executions, paired with [hyperion-async-pg](https://github.com/exodusgaming-io/hyperion-async-pg) for non-blocking DB. The 10k-conn throughput row above (5× Puma) is the consequence — same idle RSS shape, very different behaviour once the handlers actually do work.
 
 **HTTP/2 multiplexing — 1 connection × 100 concurrent streams (handler sleeps 50 ms):**
 
@@ -194,6 +208,9 @@ Hyperion fans 100 in-flight streams across separate fibers within a single TCP c
 bundle exec ruby bench/compare.rb
 HYPERION_WORKERS=4 PUMA_WORKERS=4 FALCON_COUNT=4 bundle exec ruby bench/compare.rb
 
+# Idle keep-alive RSS sweep (1k / 5k / 10k conns, 30s hold per server)
+./bench/keepalive_memory.sh
+
 # Real Rails / Grape: see bench/db.ru for the schema
 ```
 
@@ -346,6 +363,31 @@ Hyperion.stats
 # => {connections_accepted: 1234, connections_active: 7, requests_total: 8910, …}
 ```
 
+### Prometheus exporter
+
+When `admin_token` is set in your config, Hyperion mounts a `/-/metrics` endpoint that emits Prometheus text-format v0.0.4. Same token guards both `/-/metrics` (GET) and `/-/quit` (POST); auth is via the `X-Hyperion-Admin-Token` header.
+
+```sh
+$ curl -s -H 'X-Hyperion-Admin-Token: secret' http://127.0.0.1:9292/-/metrics
+# HELP hyperion_requests_total Total HTTP requests handled
+# TYPE hyperion_requests_total counter
+hyperion_requests_total 8910
+# HELP hyperion_bytes_written_total Total bytes written to response sockets
+# TYPE hyperion_bytes_written_total counter
+hyperion_bytes_written_total 2351023
+# HELP hyperion_responses_status_total Responses by HTTP status code
+# TYPE hyperion_responses_status_total counter
+hyperion_responses_status_total{status="200"} 8521
+hyperion_responses_status_total{status="404"} 12
+hyperion_responses_status_total{status="500"} 3
+# … and so on for sendfile_responses_total, rejected_connections_total,
+# slow_request_aborts_total, requests_async_dispatched_total, etc.
+```
+
+Any counter not in the known set (added by app middleware via `Hyperion.metrics.increment(:custom_thing)`) is auto-exported as `hyperion_custom_thing` with a generic HELP line — no Hyperion config change required.
+
+Point your scraper at it: in Prometheus' `scrape_configs`, set `metrics_path: /-/metrics` and `bearer_token` (or use a custom header relabel — Prometheus 2.42+ supports `authorization.credentials_file` paired with a custom `header` block). Network-isolate the admin endpoints if the listener is internet-facing — see [docs/REVERSE_PROXY.md](docs/REVERSE_PROXY.md) for the nginx `location /-/ { return 404; }` recipe.
+
 ## TLS + HTTP/2
 
 Provide a PEM cert + key:

data/lib/hyperion/logger.rb

@@ -150,7 +150,7 @@ module Hyperion
                build_access_text(ts, method, path, query, status, duration_ms, remote_addr, http_version)
              end
 
-      buf = Thread.current[@buffer_key] || allocate_access_buffer
+      buf = Thread.current.thread_variable_get(@buffer_key) || allocate_access_buffer
       buf << line
       return if buf.bytesize < ACCESS_FLUSH_BYTES
 
@@ -164,7 +164,7 @@ module Hyperion
     # loop when a connection closes (so log lines from a closing keep-alive
     # session don't get stuck behind the buffer until the next connection).
     def flush_access_buffer
-      buf = Thread.current[@buffer_key]
+      buf = Thread.current.thread_variable_get(@buffer_key)
       return if buf.nil? || buf.empty?
 
       @out.write(buf)
@@ -215,7 +215,7 @@ module Hyperion
     # Mutex is taken once per thread (not per request).
     def allocate_access_buffer
       buf = +''
-      Thread.current[@buffer_key] = buf
+      Thread.current.thread_variable_set(@buffer_key, buf)
       @access_buffers_mutex.synchronize { @access_buffers << buf }
       buf
     end
@@ -229,11 +229,21 @@ module Hyperion
     end
 
     # Cached UTC iso8601(3) timestamp, refreshed at most once per millisecond
-    # per thread. At 24k r/s with 16 threads we render ~1500 r/s/thread; only
-    # ~1000 of those allocate a new String. The other 500 reuse the cached one.
+    # per OS thread. At 24k r/s with 16 threads we render ~1500 r/s/thread;
+    # only ~1000 of those allocate a new String. The other 500 reuse the
+    # cached one. Stored as a thread variable (truly thread-local across
+    # fibers) so under Async every fiber on this thread shares the same
+    # cache and the per-second amortisation actually fires; with the prior
+    # `Thread.current[:k]` storage each fiber would re-build the iso8601
+    # String on its first call after a millisecond tick.
     def cached_timestamp
       now_ms = Process.clock_gettime(Process::CLOCK_REALTIME, :millisecond)
-      cache = (Thread.current[:__hyperion_ts_cache__] ||= [-1, ''])
+      thread = Thread.current
+      cache = thread.thread_variable_get(:__hyperion_ts_cache__)
+      if cache.nil?
+        cache = [-1, '']
+        thread.thread_variable_set(:__hyperion_ts_cache__, cache)
+      end
       return cache[1] if cache[0] == now_ms
 
       cache[0] = now_ms

data/lib/hyperion/metrics.rb

@@ -7,6 +7,22 @@ module Hyperion
   # all threads that have ever incremented (one short mutex section, only
   # taken when the operator asks for stats).
   #
+  # Storage: counters live behind `Thread#thread_variable_*`, which is the
+  # only TRUE thread-local in Ruby 1.9+ — `Thread.current[:key]` is in fact
+  # FIBER-local, so under an `Async::Scheduler` (TLS path, h2 streams, the
+  # 1.3.0+ `--async-io` plain HTTP/1.1 path) every handler fiber would get
+  # its own private counters Hash that `snapshot` could never find.
+  # Verified with hyperion-async-pg 0.4.0's bench round; before the fix
+  # the dispatch counters dropped requests entirely under `--async-io` and
+  # an external scrape (Prometheus exporter on a different fiber than the
+  # handler) saw the dispatch buckets at zero.
+  #
+  # Cross-fiber races on the same OS thread: the `+=` is technically read-
+  # modify-write, but Ruby's fiber scheduler only preempts at IO boundaries
+  # (Fiber.scheduler-aware system calls), and `Hash#[]=` is purely Ruby —
+  # no preemption mid-increment, no torn writes. Two fibers cannot
+  # interleave a single `+=` on the same OS thread.
+  #
   # Reset semantics: counters monotonically increase. Operators that want
   # rate-of-change should snapshot, sleep, snapshot, diff.
   #
@@ -14,16 +30,40 @@ module Hyperion
   #   Hyperion.stats -> Hash with all current values across all threads.
   class Metrics
     def initialize
-      @threads = Set.new
-      @threads_mutex = Mutex.new
-      # Each Metrics instance has its own thread-local key so spec runs that
-      # build fresh Metrics objects don't share state across examples.
+      # Direct list of every per-thread counters Hash ever allocated through
+      # this Metrics instance. We hold the Hash refs ourselves (instead of
+      # holding Thread refs and looking the Hash up via thread-local
+      # storage) so snapshot survives thread death — counters from a
+      # short-lived worker that already exited still aggregate. Tiny per-
+      # thread footprint (one Hash + one slot in this Array).
+      @thread_counters = []
+      @counters_mutex = Mutex.new
+      # Per-instance thread-local key so spec runs that build fresh Metrics
+      # objects don't share state across examples.
       @thread_key = :"__hyperion_metrics_#{object_id}__"
     end
 
-    # Hot path: one TLS lookup + one hash op. No mutex.
+    # Hot path: one thread-variable lookup + one hash op. No mutex on the
+    # increment fast path; the mutex is taken only on first allocation per
+    # OS thread (very rare) and on snapshot.
+    #
+    # Storage uses Thread#thread_variable_*, which is the only TRUE thread-
+    # local in Ruby 1.9+ — Thread.current[:key] is in fact FIBER-local, so
+    # under an Async::Scheduler (TLS path, h2 streams, the 1.3.0+ --async-io
+    # plain HTTP/1.1 path) every handler fiber would get its own private
+    # counters Hash that snapshot could never aggregate. Verified with
+    # hyperion-async-pg 0.4.0's bench round; before the fix the dispatch
+    # counters dropped requests under --async-io.
+    #
+    # Cross-fiber races on the same OS thread: the `+=` is read-modify-write,
+    # but Ruby's fiber scheduler only preempts at IO boundaries (Fiber-
+    # scheduler-aware system calls). Hash#[]= is purely Ruby — no
+    # preemption mid-increment, no torn writes. Two fibers cannot
+    # interleave a single `+=` on the same OS thread.
     def increment(key, by = 1)
-      counters = Thread.current[@thread_key] ||= register_thread_counters
+      thread = Thread.current
+      counters = thread.thread_variable_get(@thread_key)
+      counters = register_thread_counters(thread) if counters.nil?
       counters[key] += by
     end
 
@@ -37,14 +77,9 @@ module Hyperion
 
     def snapshot
       result = Hash.new(0)
-      @threads_mutex.synchronize do
-        @threads.delete_if { |t| !t.alive? }
-        @threads.each do |t|
-          counters = t[@thread_key]
-          next unless counters
-
-          counters.each { |k, v| result[k] += v }
-        end
+      counters_snapshot = @counters_mutex.synchronize { @thread_counters.dup }
+      counters_snapshot.each do |counters|
+        counters.each { |k, v| result[k] += v }
       end
       result.default = nil
       result
@@ -52,16 +87,17 @@ module Hyperion
 
     # Tests can call .reset! between examples to avoid cross-spec leakage.
     def reset!
-      @threads_mutex.synchronize do
-        @threads.each { |t| t[@thread_key]&.clear }
+      @counters_mutex.synchronize do
+        @thread_counters.each(&:clear)
       end
     end
 
     private
 
-    def register_thread_counters
+    def register_thread_counters(thread)
       counters = Hash.new(0)
-      @threads_mutex.synchronize { @threads << Thread.current }
+      thread.thread_variable_set(@thread_key, counters)
+      @counters_mutex.synchronize { @thread_counters << counters }
       counters
     end
   end

data/lib/hyperion/response_writer.rb

@@ -142,10 +142,19 @@ module Hyperion
 
     # Cached HTTP `Date:` header at second resolution. `Time.now.httpdate`
     # allocates several strings; at high r/s the cache reuses one String per
-    # second per thread instead of allocating per response.
+    # second per OS thread instead of allocating per response. Stored as a
+    # thread variable (truly thread-local across fibers) so under Async
+    # every fiber on this thread shares the same cache — otherwise each
+    # fiber would rebuild the httpdate String on its first response after
+    # a second tick.
     def cached_date
       now_s = Process.clock_gettime(Process::CLOCK_REALTIME, :second)
-      cache = (Thread.current[:__hyperion_date_cache__] ||= [-1, ''])
+      thread = Thread.current
+      cache = thread.thread_variable_get(:__hyperion_date_cache__)
+      if cache.nil?
+        cache = [-1, '']
+        thread.thread_variable_set(:__hyperion_date_cache__, cache)
+      end
       return cache[1] if cache[0] == now_s
 
       cache[0] = now_s

data/lib/hyperion/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Hyperion
-  VERSION = '1.4.0'
+  VERSION = '1.4.2'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: hyperion-rb
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.4.2
 platform: ruby
 authors:
 - Andrey Lobanov