hyperion-rb 1.4.1 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64221d424c994e0757262dca6984cd2b65bf9ec3da6c85094daa717c716da906
-  data.tar.gz: 236188ff1777b49178bb4b2bfe75fbea9309ec6f5d56ec01329943dfa1afbb36
+  metadata.gz: ab7691ac6671b0e0c9606c281c55659c76675b71f3d461d1fb5bf6a03680861b
+  data.tar.gz: b7ad35585d56e59d4a7b5c9fcb6d4e016e72b4c3f99496ba675ca7e871865718
 SHA512:
-  metadata.gz: 9b196f8d046c828546f8f5fbac91e0300e8704a70f11b47096a9b0fb05caf2ec34f63e2f33768ab53b41413aaa4e957ac499aa4779ecd54b6a987deef7fd464e
-  data.tar.gz: cb3d64f757736bcbc2632492e24b0794a67c98bfdcae7a1e53d89f583cf2eecc404e060fd3ec10e441ddfdcd11273c41e0d46fa9b80be904b29a7718fe0258ba
+  metadata.gz: 8911a91c7932b332a9d5f069099c7f6ded94d9b5978dffd259881ab482066d5328c508ca5983101c6d9d04b18c1353664766bf759ec66cb034d3bcdf84f01a89
+  data.tar.gz: 7c948c98eb9aea2cb31595e08deca0c4e98c2281105a18fc0419678da25a04ac9b04f9defe564ea660104cf935399582506eb87769f6f9fbbca74f568c8f904b

data/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## [1.5.0] - 2026-04-27
+
+Audit-driven CLI + adapter polish. No breaking changes; pure additions to the operator surface and a hardening of the host-header parser.
+
+### Added
+- **CLI flag coverage for 8 Config DSL settings.** Pre-1.5.0 these settings could only be reached by writing a `config/hyperion.rb` file; operators who don't keep one in their repo had no way to flip them without authoring one. They now flow through the same CLI > config-file > default precedence as the rest of the flags:
+  - `--max-body-bytes BYTES` (Integer, default 16 MiB)
+  - `--max-header-bytes BYTES` (Integer, default 64 KiB)
+  - `--max-pending COUNT` (Integer, default unbounded)
+  - `--max-request-read-seconds SECONDS` (Float, default 60)
+  - `--admin-token TOKEN` (String, default unset) — gates `POST /-/quit` and `GET /-/metrics`
+  - `--admin-token-file PATH` — sibling that reads the token from disk; refuses to load if the file is missing, unreadable, world-readable (perms must mask `0o007`), or empty. Production deployments should prefer this over `--admin-token` because argv is visible via `ps`.
+  - `--worker-max-rss-mb MB` (Integer, default unset) — RSS-based worker recycling
+  - `--idle-keepalive SECONDS` (Float, default 5)
+  - `--graceful-timeout SECONDS` (Integer, default 30)
+- **`Hyperion::CLI.parse_argv!` extracted as a public class method** so the flag-to-`cli_opts` mapping is unit-testable without booting a server. `CLI.run` is now a thin wrapper around it.
+- **README CLI flags table** extended with the 8 new flags plus `--[no-]yjit` / `--[no-]async-io` (already wired but previously undocumented in the table).
+- **17 new specs**:
+  - 14 in `spec/hyperion/cli_flags_spec.rb` cover per-flag parsing, the `merge_cli!` handoff for all 8 new flags, the CLI-wins precedence rule, and the four `--admin-token-file` abort paths (missing / unreadable / world-readable / empty).
+  - 3 in `spec/hyperion/adapter/rack_spec.rb` cover plain IPv4-with-port, bare hostname (no port), and the malformed-bracket regression below.
+
+### Fixed
+- **`Hyperion::Adapter::Rack#split_host` accepted malformed bracketed IPv6.** Pre-1.5.0 a `Host: [::1` header (no closing bracket) was returned as-is in `SERVER_NAME`, leaking attacker-controlled bytes into Rack env where downstream URL generators / SSRF allow-lists / audit logs would trust them. The adapter now fails closed to `localhost:80` and bumps a `:malformed_host_header` counter so operators can alert on attack-pattern volume. No raise — Rack apps don't expect a server adapter to throw on header-parse failures, so we degrade gracefully instead.
+
+### Security
+- `--admin-token` help text warns that argv is visible via `ps` and points operators at `--admin-token-file` for production. The token value is never echoed back in any log line.
+
+## [1.4.2] - 2026-04-27
+
+Audit-driven cleanup. No behaviour changes; fiber-correctness + docs polish.
+
+### Fixed
+- **`Hyperion::Logger` access buffer was fiber-local, not thread-local** — pre-1.4.2 the access-log write buffer was stored via `Thread.current[@buffer_key]`. Under an `Async::Scheduler` (TLS / h2 / `--async-io` plain HTTP/1.1) every handler fiber got its own private buffer, so the 4 KiB `ACCESS_FLUSH_BYTES` batching never fired — each fiber's buffer accumulated 1-3 lines before its connection closed and `flush_access_buffer` wrote them. At 24k r/s this meant ~12-24k `write(2)` syscalls/sec instead of the designed ~750/sec. Switched to `Thread#thread_variable_*` so all fibers on the same OS thread share one buffer and the batching actually fires. Same root cause as the 1.4.1 Metrics fix; surfaced by a code-audit grep for residual `Thread.current[:key]` patterns.
+- **`Logger#cached_timestamp` and `ResponseWriter#cached_date`** — same fix. Pre-1.4.2 the per-second / per-millisecond Time-formatting caches were per-fiber, so under Async every fiber rebuilt the iso8601 / httpdate String on its first call after a tick. Now per-OS-thread, shared across fibers; one allocation per second per thread total.
+
+### Added
+- **Prometheus exporter example output** in the README's Metrics section — shows what `curl -H 'X-Hyperion-Admin-Token: ...' /-/metrics` actually returns (HELP/TYPE lines, status-code labels, auto-export of unknown counters), plus the Prometheus scraper config sketch.
+- **Regression spec** for the access-buffer cross-fiber bug — two fibers on the same OS thread write through one logger; verifies a single buffer is registered (not one per fiber) and both lines land via `flush_all`.
+- **4 new Metrics specs** (already shipped in 1.4.1; called out here for coverage tracking) — cross-fiber on same thread, cross-thread, cross-fiber-on-different-thread, many-fibers-on-same-thread.
+
+### Changed
+- **README benchmark section** version-stamped: clarifies that the headline numbers were measured against the noted Hyperion version (most are 1.2.0 hello-world / 1.3.0 PG-bound) and that 1.3.0+ `--async-io` + 1.4.0+ TLS-inline + 1.4.1+ Metrics fix preserve or improve these numbers. We re-run the headline configs each release.
+
 ## [1.4.1] - 2026-04-27
 
 ### Fixed

data/README.md

@@ -25,7 +25,7 @@ bundle exec hyperion config.ru
 
 ## Benchmarks
 
-All numbers are real wrk runs against published Hyperion configs. Hyperion ships **with default-ON structured access logs**; Puma comparisons use Puma defaults (no per-request log emission).
+All numbers are real wrk runs against published Hyperion configs. Hyperion ships **with default-ON structured access logs**; Puma comparisons use Puma defaults (no per-request log emission). Each section is stamped with the Hyperion version it was measured against — newer versions (1.3.0+ `--async-io`, 1.4.0+ TLS h1 inline, 1.4.1+ Metrics fiber-key fix) preserve or improve these numbers; we re-run the headline configs each release and have not seen regressions on these workloads.
 
 ### Hello-world Rack app
 
@@ -255,6 +255,17 @@ Three layers, in precedence order: explicit CLI flag > environment variable > `c
 | `--log-format FORMAT` | `auto` | `text` / `json` / `auto`. Auto: JSON when `RAILS_ENV`/`RACK_ENV` is `production`/`staging`, colored text on TTY, JSON otherwise. |
 | `--[no-]log-requests` | ON | Per-request access log. |
 | `--fiber-local-shim` | off | Patches `Thread#thread_variable_*` to fiber storage for older Rails idioms. |
+| `--[no-]yjit` | auto | Force YJIT on/off. Default: auto-on under `RAILS_ENV`/`RACK_ENV` = `production`/`staging`. |
+| `--[no-]async-io` | off | Run plain HTTP/1.1 connections under `Async::Scheduler`. Required for `hyperion-async-pg` on plain HTTP. TLS h1 / HTTP/2 always run under the scheduler regardless. |
+| `--max-body-bytes BYTES` | `16777216` (16 MiB) | Maximum request body size. |
+| `--max-header-bytes BYTES` | `65536` (64 KiB) | Maximum total request-header size. |
+| `--max-pending COUNT` | unbounded | Per-worker accept-queue cap before new connections are rejected with HTTP 503 + `Retry-After: 1`. |
+| `--max-request-read-seconds SECONDS` | `60` | Total wallclock budget for reading request line + headers + body for ONE request. Slowloris defence. |
+| `--admin-token TOKEN` | unset | Bearer token for `POST /-/quit` and `GET /-/metrics`. **Production: prefer `--admin-token-file` — argv is visible via `ps`.** |
+| `--admin-token-file PATH` | unset | Read the admin token from a file. Refuses to load if the file is missing or world-readable (mode must mask `0o007`). |
+| `--worker-max-rss-mb MB` | unset | Master gracefully recycles a worker once its RSS exceeds this many megabytes. nil = disabled. |
+| `--idle-keepalive SECONDS` | `5` | Keep-alive idle timeout. Connection closes after this many seconds of inactivity. |
+| `--graceful-timeout SECONDS` | `30` | Shutdown deadline before SIGKILL is delivered to a worker that hasn't drained. |
 
 ### Environment variables
 
@@ -363,6 +374,31 @@ Hyperion.stats
 # => {connections_accepted: 1234, connections_active: 7, requests_total: 8910, …}
 ```
 
+### Prometheus exporter
+
+When `admin_token` is set in your config, Hyperion mounts a `/-/metrics` endpoint that emits Prometheus text-format v0.0.4. Same token guards both `/-/metrics` (GET) and `/-/quit` (POST); auth is via the `X-Hyperion-Admin-Token` header.
+
+```sh
+$ curl -s -H 'X-Hyperion-Admin-Token: secret' http://127.0.0.1:9292/-/metrics
+# HELP hyperion_requests_total Total HTTP requests handled
+# TYPE hyperion_requests_total counter
+hyperion_requests_total 8910
+# HELP hyperion_bytes_written_total Total bytes written to response sockets
+# TYPE hyperion_bytes_written_total counter
+hyperion_bytes_written_total 2351023
+# HELP hyperion_responses_status_total Responses by HTTP status code
+# TYPE hyperion_responses_status_total counter
+hyperion_responses_status_total{status="200"} 8521
+hyperion_responses_status_total{status="404"} 12
+hyperion_responses_status_total{status="500"} 3
+# … and so on for sendfile_responses_total, rejected_connections_total,
+# slow_request_aborts_total, requests_async_dispatched_total, etc.
+```
+
+Any counter not in the known set (added by app middleware via `Hyperion.metrics.increment(:custom_thing)`) is auto-exported as `hyperion_custom_thing` with a generic HELP line — no Hyperion config change required.
+
+Point your scraper at it: in Prometheus' `scrape_configs`, set `metrics_path: /-/metrics` and `bearer_token` (or use a custom header relabel — Prometheus 2.42+ supports `authorization.credentials_file` paired with a custom `header` block). Network-isolate the admin endpoints if the listener is internet-facing — see [docs/REVERSE_PROXY.md](docs/REVERSE_PROXY.md) for the nginx `location /-/ { return 404; }` recipe.
+
 ## TLS + HTTP/2
 
 Provide a PEM cert + key:

data/lib/hyperion/adapter/rack.rb

@@ -138,7 +138,17 @@ module Hyperion
 
           if host_header.start_with?('[')
             close = host_header.index(']')
-            return [host_header, '80'] unless close
+            # Malformed bracketed IPv6 (no closing bracket): we used to return
+            # the raw garbage as SERVER_NAME, which then leaked into Rack env
+            # where downstream URL generators / loggers / SSRF allow-lists
+            # would trust attacker-controlled bytes. Fail closed to a safe
+            # default and bump a counter so operators can alert on volume.
+            # No raise — Rack apps don't expect Hyperion's adapter to throw
+            # on header-parse failures, so we degrade gracefully instead.
+            unless close
+              Hyperion.metrics.increment(:malformed_host_header)
+              return %w[localhost 80]
+            end
 
             name = host_header[0..close]
             rest = host_header[(close + 1)..]

data/lib/hyperion/cli.rb

@@ -11,6 +11,55 @@ module Hyperion
     DEFAULT_CONFIG_PATH = 'config/hyperion.rb'
 
     def self.run(argv)
+      cli_opts, config_path = parse_argv!(argv)
+
+      # Precedence: CLI > config file > built-in default. We auto-load
+      # config/hyperion.rb if present so operators can drop a file in their
+      # repo and have it take effect without having to remember -C.
+      config_path ||= DEFAULT_CONFIG_PATH if File.exist?(DEFAULT_CONFIG_PATH)
+      config = config_path ? Hyperion::Config.load(config_path) : Hyperion::Config.new
+      config.merge_cli!(cli_opts)
+
+      # Install logger early so every subsequent log call honours the operator's
+      # chosen format/level (config file or CLI) before anything else logs.
+      if config.log_level || config.log_format
+        Hyperion.logger = Hyperion::Logger.new(level: config.log_level, format: config.log_format)
+      end
+
+      # Propagate log_requests so every Connection picks it up via
+      # `Hyperion.log_requests?` without needing to thread it through
+      # Server/ThreadPool/Master plumbing. Default is ON; nil means "don't
+      # touch — fall through to the env/default chain in Hyperion.log_requests?".
+      Hyperion.log_requests = config.log_requests unless config.log_requests.nil?
+
+      # Enable YJIT before workers fork / connections start. Auto-on in
+      # production/staging gives operators the perf bump for free; explicit
+      # config.yjit (true/false) overrides the env-based default.
+      maybe_enable_yjit(config)
+
+      rackup = argv.first || 'config.ru'
+      abort("[hyperion] no such rackup file: #{rackup}") unless File.exist?(rackup)
+
+      if config.fiber_local_shim
+        Hyperion::FiberLocal.install!
+        Hyperion.logger.info { { message: 'FiberLocal shim installed' } }
+      end
+
+      app = load_rack_app(rackup)
+      app = wrap_admin_middleware(app, config)
+      workers = config.workers.zero? ? Etc.nprocessors : config.workers
+
+      if workers <= 1
+        run_single(config, app)
+      else
+        run_cluster(config, app, workers)
+      end
+    end
+
+    # Extracted from #run so the flag-to-cli_opts mapping can be unit-tested
+    # without booting a server. Returns [cli_opts, config_path]. Mutates argv
+    # in place (consumes flags, leaves the rackup path for the caller).
+    def self.parse_argv!(argv)
       cli_opts    = {}
       config_path = nil
 
@@ -61,6 +110,46 @@ module Hyperion
              'Run plain HTTP/1.1 connections under Async::Scheduler (required for hyperion-async-pg and other fiber-cooperative I/O; default off)') do |v|
           cli_opts[:async_io] = v
         end
+        o.on('--max-body-bytes BYTES', Integer,
+             'Maximum request body size in bytes (default 16777216 = 16 MiB)') do |n|
+          cli_opts[:max_body_bytes] = n
+        end
+        o.on('--max-header-bytes BYTES', Integer,
+             'Maximum total request-header size in bytes (default 65536 = 64 KiB)') do |n|
+          cli_opts[:max_header_bytes] = n
+        end
+        o.on('--max-pending COUNT', Integer,
+             'Maximum queued connections per worker before new accepts are rejected with 503 (default unbounded)') do |n|
+          cli_opts[:max_pending] = n
+        end
+        o.on('--max-request-read-seconds SECONDS', Float,
+             'Total wallclock budget for reading request line + headers + body (default 60.0; 0 disables)') do |n|
+          cli_opts[:max_request_read_seconds] = n
+        end
+        # Security-sensitive: read the token verbatim and never echo it back
+        # in any subsequent log/help line. argv is visible via `ps` on most
+        # systems; production deployments should prefer --admin-token-file.
+        o.on('--admin-token TOKEN',
+             "Bearer token for the /-/quit and /-/metrics admin endpoints. \
+WARNING: argv is visible via `ps`; prefer --admin-token-file PATH for production.") do |t|
+          cli_opts[:admin_token] = t
+        end
+        o.on('--admin-token-file PATH',
+             'Read the admin token from a file. File must NOT be world-readable (perms must mask 0o007).') do |p|
+          cli_opts[:admin_token] = read_admin_token_file(p)
+        end
+        o.on('--worker-max-rss-mb MB', Integer,
+             'Recycle a worker when its RSS exceeds MB megabytes (default unset; nil disables)') do |n|
+          cli_opts[:worker_max_rss_mb] = n
+        end
+        o.on('--idle-keepalive SECONDS', Float,
+             'Idle keep-alive timeout in seconds (default 5.0)') do |n|
+          cli_opts[:idle_keepalive] = n
+        end
+        o.on('--graceful-timeout SECONDS', Integer,
+             'Graceful shutdown deadline in seconds before SIGKILL (default 30)') do |n|
+          cli_opts[:graceful_timeout] = n
+        end
         o.on('-h', '--help', 'show help') do
           puts o
           exit 0
@@ -68,47 +157,7 @@ module Hyperion
       end
       parser.parse!(argv)
 
-      # Precedence: CLI > config file > built-in default. We auto-load
-      # config/hyperion.rb if present so operators can drop a file in their
-      # repo and have it take effect without having to remember -C.
-      config_path ||= DEFAULT_CONFIG_PATH if File.exist?(DEFAULT_CONFIG_PATH)
-      config = config_path ? Hyperion::Config.load(config_path) : Hyperion::Config.new
-      config.merge_cli!(cli_opts)
-
-      # Install logger early so every subsequent log call honours the operator's
-      # chosen format/level (config file or CLI) before anything else logs.
-      if config.log_level || config.log_format
-        Hyperion.logger = Hyperion::Logger.new(level: config.log_level, format: config.log_format)
-      end
-
-      # Propagate log_requests so every Connection picks it up via
-      # `Hyperion.log_requests?` without needing to thread it through
-      # Server/ThreadPool/Master plumbing. Default is ON; nil means "don't
-      # touch — fall through to the env/default chain in Hyperion.log_requests?".
-      Hyperion.log_requests = config.log_requests unless config.log_requests.nil?
-
-      # Enable YJIT before workers fork / connections start. Auto-on in
-      # production/staging gives operators the perf bump for free; explicit
-      # config.yjit (true/false) overrides the env-based default.
-      maybe_enable_yjit(config)
-
-      rackup = argv.first || 'config.ru'
-      abort("[hyperion] no such rackup file: #{rackup}") unless File.exist?(rackup)
-
-      if config.fiber_local_shim
-        Hyperion::FiberLocal.install!
-        Hyperion.logger.info { { message: 'FiberLocal shim installed' } }
-      end
-
-      app = load_rack_app(rackup)
-      app = wrap_admin_middleware(app, config)
-      workers = config.workers.zero? ? Etc.nprocessors : config.workers
-
-      if workers <= 1
-        run_single(config, app)
-      else
-        run_cluster(config, app, workers)
-      end
+      [cli_opts, config_path]
     end
 
     def self.run_single(config, app)
@@ -227,6 +276,28 @@ module Hyperion
     end
     private_class_method :wrap_admin_middleware
 
+    # Read the admin token from a file on disk. Refuses to load if the file
+    # is missing, unreadable, or world-readable — the whole point of using a
+    # file instead of `--admin-token` is to keep the token off argv (which
+    # `ps` exposes) and off other-user-readable storage. Trailing whitespace
+    # is stripped so operators can use `echo "$TOKEN" > /etc/hyperion-token`
+    # without inadvertently embedding a newline. Empty files abort.
+    def self.read_admin_token_file(path)
+      abort("[hyperion] admin token file not found: #{path}") unless File.file?(path)
+      abort("[hyperion] admin token file not readable: #{path}") unless File.readable?(path)
+
+      mode = File.stat(path).mode & 0o777
+      if (mode & 0o007).positive?
+        abort("[hyperion] admin token file #{path} is world-readable (mode #{format('%04o', mode)}); chmod 600")
+      end
+
+      token = File.read(path).strip
+      abort("[hyperion] admin token file is empty: #{path}") if token.empty?
+
+      token
+    end
+    private_class_method :read_admin_token_file
+
     # Warn loudly at boot if the C parser didn't load — operators running
     # production with the pure-Ruby fallback are paying ~2× CPU on parse-heavy
     # workloads and probably don't know it.

data/lib/hyperion/logger.rb

@@ -150,7 +150,7 @@ module Hyperion
                build_access_text(ts, method, path, query, status, duration_ms, remote_addr, http_version)
              end
 
-      buf = Thread.current[@buffer_key] || allocate_access_buffer
+      buf = Thread.current.thread_variable_get(@buffer_key) || allocate_access_buffer
       buf << line
       return if buf.bytesize < ACCESS_FLUSH_BYTES
 
@@ -164,7 +164,7 @@ module Hyperion
     # loop when a connection closes (so log lines from a closing keep-alive
     # session don't get stuck behind the buffer until the next connection).
     def flush_access_buffer
-      buf = Thread.current[@buffer_key]
+      buf = Thread.current.thread_variable_get(@buffer_key)
       return if buf.nil? || buf.empty?
 
       @out.write(buf)
@@ -215,7 +215,7 @@ module Hyperion
     # Mutex is taken once per thread (not per request).
     def allocate_access_buffer
       buf = +''
-      Thread.current[@buffer_key] = buf
+      Thread.current.thread_variable_set(@buffer_key, buf)
       @access_buffers_mutex.synchronize { @access_buffers << buf }
       buf
     end
@@ -229,11 +229,21 @@ module Hyperion
     end
 
     # Cached UTC iso8601(3) timestamp, refreshed at most once per millisecond
-    # per thread. At 24k r/s with 16 threads we render ~1500 r/s/thread; only
-    # ~1000 of those allocate a new String. The other 500 reuse the cached one.
+    # per OS thread. At 24k r/s with 16 threads we render ~1500 r/s/thread;
+    # only ~1000 of those allocate a new String. The other 500 reuse the
+    # cached one. Stored as a thread variable (truly thread-local across
+    # fibers) so under Async every fiber on this thread shares the same
+    # cache and the per-second amortisation actually fires; with the prior
+    # `Thread.current[:k]` storage each fiber would re-build the iso8601
+    # String on its first call after a millisecond tick.
     def cached_timestamp
       now_ms = Process.clock_gettime(Process::CLOCK_REALTIME, :millisecond)
-      cache = (Thread.current[:__hyperion_ts_cache__] ||= [-1, ''])
+      thread = Thread.current
+      cache = thread.thread_variable_get(:__hyperion_ts_cache__)
+      if cache.nil?
+        cache = [-1, '']
+        thread.thread_variable_set(:__hyperion_ts_cache__, cache)
+      end
       return cache[1] if cache[0] == now_ms
 
       cache[0] = now_ms

data/lib/hyperion/response_writer.rb

@@ -142,10 +142,19 @@ module Hyperion
 
     # Cached HTTP `Date:` header at second resolution. `Time.now.httpdate`
     # allocates several strings; at high r/s the cache reuses one String per
-    # second per thread instead of allocating per response.
+    # second per OS thread instead of allocating per response. Stored as a
+    # thread variable (truly thread-local across fibers) so under Async
+    # every fiber on this thread shares the same cache — otherwise each
+    # fiber would rebuild the httpdate String on its first response after
+    # a second tick.
     def cached_date
       now_s = Process.clock_gettime(Process::CLOCK_REALTIME, :second)
-      cache = (Thread.current[:__hyperion_date_cache__] ||= [-1, ''])
+      thread = Thread.current
+      cache = thread.thread_variable_get(:__hyperion_date_cache__)
+      if cache.nil?
+        cache = [-1, '']
+        thread.thread_variable_set(:__hyperion_date_cache__, cache)
+      end
       return cache[1] if cache[0] == now_s
 
       cache[0] = now_s

data/lib/hyperion/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Hyperion
-  VERSION = '1.4.1'
+  VERSION = '1.5.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: hyperion-rb
 version: !ruby/object:Gem::Version
-  version: 1.4.1
+  version: 1.5.0
 platform: ruby
 authors:
 - Andrey Lobanov