redhound 2.0.0.rc1 → 2.0.0.rc2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dd9c749a43481a575813ed4bacad00dbc54dfecc7701e63a0baeb6789a1d8680
-  data.tar.gz: 791efd1f3e810c7e67bcf5abfbbc3e02aa0db460bd8dca51b83beecd94797bae
+  metadata.gz: 33f3a971291b9d5c6be138098ab5d6072717b8ffc79f418830f3538d8b9c4500
+  data.tar.gz: 191cf625ab53877b304432e4cd0b9caa6725afd1e4929bebaba31ef2382c0108
 SHA512:
-  metadata.gz: 99e69fb1319c95448572dcfc2151da08987523b95301c1b50f1f1345b269752cd98e4ddcd3c24f711866bc6b7754793061b51f1d208f92428dfd337a007ac444
-  data.tar.gz: 1081108559a19f9f17b0c482dc91285f4950b1f7dda81c4f1221e2eac76206831839d265d50139182060c203e12dfd7752e434ecfae2fa5479310260b039fa7f
+  metadata.gz: 73fc49a4129a62cfdad32ee818adb4d1915f115f3bbe3f0a47b36d5d8a23c625d275d28a39e8307d4b8d1cc068b0d6be12409e2f99400f0135aa89383d58b0f9
+  data.tar.gz: 817303ae395d6f1b288bbab65105736d97f019cfb27dc6ba8edea787d7eab82fed9817d4a3fd0ee36e7e1407af17b48c50adab04412410b36f29044d778f0215

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## Unreleased
 
+## 2.0.0.rc2 - 2026-10-02
+
+### Features
+
+- Use the ring backend automatically on aarch64 Linux when mapping is available.
+
+### Bug fixes
+
+- Improve file reading and packet summary throughput while keeping stdin responsive.
+- Preserve readable fields in truncated headers and correct ICMP errors, IGMP reports, signed NTP precision, IPv6 extension boundaries and variable field locations.
+- Reject malformed TCP options and empty or conflicting HTTP Content-Length values; recognize HTTP when its first line spans segments on nonstandard ports.
+- Report incomplete IP/TCP data at EOF and flow eviction, distinguish retransmitted SYN/FIN packets, and handle IPv6 atomic fragments independently.
+- Account for retained buffer capacity when enforcing reassembly limits, bound pcapng interface metadata, and release analysis buffers even when output fails.
+- Keep capture timeouts and stopping responsive under rejected traffic and idle stdin, preserve partial input records across timeouts, and support replacing file-source filters.
+- Preserve wire lengths with attached filters, close capture files after write failures, and prevent metadata errors from corrupting pcapng interface numbering.
+- Correct macOS interface MAC/MTU and loopback link types; include interface and direction in packet output and show checksum/TCP options with verbose summaries.
+- Preserve negative and expanded-year timestamps and emit valid JSON for invalid UTF-8 diagnostics and interface names; reject invalid packet metadata before file serialization.
+- Include empty intervals in IO statistics and keep unknown IP protocol conversations separate.
+- Fall back to socket capture when Ruby 4.0 cannot map a packet ring, and explain unavailable explicit ring capture and receive-buffer/VLAN limitations.
+
 ## 2.0.0.rc1 - 2026-10-01
 
 ### Breaking changes

data/README.md

@@ -14,7 +14,7 @@ for remaining GA evaluation gates.
 gem install redhound --pre
 ```
 
-Or add `gem 'redhound', '~> 2.0.0.rc1'` to your Gemfile.
+Or add `gem 'redhound', '~> 2.0.0.rc2'` to your Gemfile.
 
 ## Usage
 

data/docs/API.md

@@ -26,6 +26,16 @@ provides `next_packet`, `stats`, `interfaces`, `stop`, and `close`; a block alwa
 closes it. `Redhound.capture` yields packets and closes the live source even if
 the block fails. Capture options include `snaplen`, `promiscuous`, `buffer_size`
 (bytes), `direction` (`:in`, `:out`, `:inout`), `backend`, and `filter`.
+On Linux Ruby 4.0, `backend: :auto` uses sockets because `IO::Buffer.map` cannot
+map packet sockets; explicit `:ring` raises `UnsupportedPlatform`. Ruby 3.3/3.4
+retain ring support.
+
+`next_packet(timeout:)` returns nil at its deadline without discarding a partial
+stdin/pipe record. `stop` interrupts a waiting read. `attach_filter(program)`
+replaces a source's capture filter; filtering preserves the original wire length.
+For pcapng input, retained interface metadata is limited to 4,096 interfaces and
+16 MiB across sections, with at most 4,096 options in a block. Excess input raises
+`FileFormatError` rather than growing memory without a bound.
 
 `Redhound.dissect` creates an immutable-byte Packet and decodes its layers lazily.
 Packet metadata includes `timestamp_ns`, `original_length`, `linktype`,
@@ -35,6 +45,23 @@ selects a decoded field value. `layers_of`, `innermost`, and `field_values` expo
 repeated or tunneled layers. Layer has `values`, `fields`, and `diagnostics`.
 `to_h` follows [json-schema.json](json-schema.json); bytes use hexadecimal strings,
 addresses use text, booleans remain booleans, repeated fields become arrays.
+`timestamp_ns` must be an Integer; calendar output preserves expanded years
+outside 0000..9999. `number` is positive, `direction` is `:in`, `:out`, or nil,
+and integer link types are restricted to 0..65535. Invalid UTF-8 field text,
+interface names, and diagnostic text use hexadecimal strings without discarding
+their original bytes.
+
+Each Field exposes its `name`, `type`, `value`, `raw_value`, `offset`, and
+`length`. For packet dissection, offsets are absolute captured byte positions;
+lengths describe the encoded source span. Complete fixed fields remain available
+when the rest of a header or its options are truncated. Derived values such as
+`tcp.len`, `data.len`, and `igmp.version` have zero source length. DNS compressed
+names refer to their encoded name/pointer span. Concatenated DHCP values and TLS
+handshakes crossing records span the first through last source byte, including
+intervening TLV/record headers; decoded values need not equal a direct byte slice.
+Likewise, chunked HTTP body spans include chunk framing. Stream-reassembled
+application fields use offsets in the logical PDU rather than the completing
+packet's `data`; `tcp.reassembled_from` identifies the contributing frames.
 
 Writer accepts `format: :pcap|:pcapng`, `linktype`, `snaplen`, `precision`,
 `packet_buffered`, `max_bytes`, `interval`, `file_count`, and
@@ -58,7 +85,8 @@ class Example < Redhound::Dissector
 end
 ```
 
-Header fields support unsigned integers, MAC/IPv4/IPv6 addresses and bit fields.
+Header fields support unsigned integers, signed `int8`, MAC/IPv4/IPv6 addresses
+and bit fields.
 Use `dissect(ctx, layer)` for bounded variable-length parsing through
 `ctx.cursor`; append fields with `layer.add`. `next_dissector(ctx, layer)` returns
 a registered dissector class, or nil. Child cursors cannot escape their packet

data/docs/MIGRATION.md

@@ -18,6 +18,6 @@ Interface indexes and Linux `any` are supported. Linux loopback duplicates are
 removed. Capture timestamps come from the kernel; captured and original packet
 lengths are recorded separately. pcapng supports multiple interfaces and metadata.
 
-Version 2.0.0.rc1 is a prerelease. The two-week RC evaluation and long capture
+Version 2 is currently a prerelease. The two-week RC evaluation and long capture
 soak gates remain prerequisites for GA; a 1.x maintenance deadline will be
 announced when GA is published.

data/docs/PROTOCOLS.md

@@ -17,7 +17,7 @@ ipv6: Internet Protocol v6 ipv6.version ipv6.tclass ipv6.flow ipv6.plen ipv6.nxt
 ipv6_ext: IPv6 extension
 udp: User Datagram Protocol udp.srcport udp.dstport udp.length udp.checksum
 tcp: Transmission Control Protocol tcp.srcport tcp.dstport tcp.seq tcp.ack tcp.hdr_len tcp.reserved tcp.flags tcp.window_size_value tcp.checksum tcp.urgent_pointer
-icmp: Internet Control Message Protocol icmp.type icmp.code icmp.checksum icmp.ident icmp.seq
+icmp: Internet Control Message Protocol icmp.type icmp.code icmp.checksum
 icmpv6: ICMPv6 / Neighbor Discovery icmpv6.type icmpv6.code icmpv6.checksum
 igmp: Internet Group Management Protocol igmp.type igmp.max_resp igmp.checksum
 gre: Generic Routing Encapsulation gre.flags gre.proto
@@ -37,9 +37,18 @@ ClientHello/ServerHello metadata including SNI, ALPN and supported versions.
 DNS includes mDNS/LLMNR and TCP framing. IPv6 supports Hop-by-Hop, Routing,
 Fragment, Destination and AH headers; ESP stops dissection. GRE and VXLAN
 encapsulations decode inner packet layers.
+ICMP identifiers/sequences are emitted for echo, timestamp, and address-mask
+messages; fragmentation-needed errors expose `icmp.mtu` instead. IGMP reserved
+report bytes are not labeled as maximum-response times. NTP timestamps such as
+`ntp.xmt` preserve the unsigned 32.32 wire integer; their era-dependent calendar
+interpretation is left to the caller.
 
 Dissectors use checked cursors and parent payload boundaries. Truncation,
 malformed lengths, checksum failures and reassembly gaps are available as
 structured diagnostics. Checksum verification is enabled by `-v` or
 `Engine.new(verify_checksums: true)`; outgoing/offloaded packets are marked
 unverified.
+IPv6 No Next Header terminates the chain, and the 16-extension limit applies
+independently to each encapsulated IPv6 header. Variable wire fields carry source
+ranges; [API.md](API.md) describes compression, concatenation, and reassembly
+span semantics.

data/docs/USAGE.md

@@ -71,11 +71,15 @@ success, 1 on capture/file failures and 2 on invalid arguments.
 
 ## Platforms and limits
 
-Linux `auto` uses TPACKET_V3 on x86_64 and falls back to the socket backend when
+Linux `auto` uses TPACKET_V3 on x86_64 and aarch64 and falls back to the socket backend when
 mapping is unavailable. Other Linux architectures default to sockets; `ring`
 can be selected explicitly. macOS uses BPF, with native timestamp precision
 reported by the device. Linux socket capture cannot recover NIC-stripped VLAN
 tags; choose the ring backend for that metadata.
+Ruby 4.0's `IO::Buffer.map` rejects packet sockets, so `auto` uses socket capture
+and explicit `ring` reports the mapping limitation. Use Ruby 3.3/3.4 for ring
+capture. If using sockets, disable receive VLAN offload on the receiving
+interface when VLAN tags are required (`ethtool -K IF rxvlan off`).
 
 Windows, Wi-Fi monitor mode, packet transmission, TLS decryption, complete
 HTTP/2/QUIC dissection, display-filter syntax and a TUI are outside v2's scope.

data/docs/VALIDATION.md

@@ -1,8 +1,126 @@
 # Release validation
 
+## Adversarial review after rc1
+
+The review checked the design, roadmap, work procedures and phase0 patch against
+the running implementation, rather than treating a green test suite as proof
+that every acceptance gate had passed. Reproduced failures included truncated
+headers losing readable fields, incorrect ICMP error fields and dynamic field
+locations, missed split HTTP identification, unreported reassembly loss at EOF
+and flow eviction, underestimated Ruby buffer capacity, unbounded pcapng
+metadata, source timeout violations and capture writer descriptor leaks.
+Regression checks accompany the fixes.
+
+The parser fuzz now retains each fixture's real link type during structured
+mutations. This prevents tests from stopping at an unrelated link header before
+reaching the mutated protocol. `ruby bench/analysis.rb` exercises default state
+limits with 60,000 conversations, 40,000 fragmented datagram identities and 80
+large TCP streams, checking every update and final buffer release. Its retained
+state accounting includes String capacity and object storage. Process RSS is
+reported separately: Ruby may retain allocator pages after state is released.
+
+The manual [Live acceptance workflow](../.github/workflows/acceptance.yml) and
+`bench/live.rb` record requested/sent/captured counts, gaps, byte digests, kernel
+timestamps, drops, RSS and descriptor counts. An insufficient sender rate fails
+the check. Controlled aarch64 runs passed 50,000 pps with simultaneous socket and
+ring capture and 80,000 pps with socket capture for ten seconds. These short
+checks do not replace the hour-long comparison or the x86_64 targets.
+The strengthened live harness checks complete expected wire bytes, every
+timestamp and the entire zero-based sequence. Its mixed TCP/UDP workload and
+capture-only CPU affinity are recorded alongside implementation and harness
+digests. The earlier long-running snapshot uses the original UDP workload and
+verifier; its comparison requires identical complete-frame digests and sampled
+timestamps, and does not establish the new mixed single-core throughput gate.
+
+Local adversarial runtime validation on 2026-10-02 passed all 244 non-live examples
+on Linux, with only the macOS-specific interface test skipped. After adding
+bounded file prefetch and live-harness regressions, the complete macOS Rake task
+passed its 257 examples (20 tshark comparisons are run on Linux), RBS generation
+and Steep, with 92.92% total line coverage. Coverage gates passed; the earlier
+Linux check measured 92.43% total line coverage. The strengthened one-million-case parser fuzz completed without an
+exception. Default state-pressure checks reached exactly 256 MiB accounted
+state without exceeding it and released IP/TCP buffers at EOF. Native RSS
+reached 308 MiB despite retaining only 107 MiB of Ruby heap before finalization;
+this is not reported as a 256 MiB process-RSS guarantee.
+The independent [manual CI and million-case fuzz run](https://github.com/ydah/redhound/actions/runs/36885293639)
+also passed every job on the adversarial runtime fixes.
+
+The GitHub maintenance configuration now includes weekly Bundler/Actions
+dependency updates, actionlint, codespell, strict yamllint, zizmor and Ruby/Actions
+CodeQL analysis. Action revisions are pinned and checkout credentials are not
+retained. All four lint workflows passed on main; the final CodeQL run succeeded
+with no open alerts. HTTP header trimming uses
+bounded byte ranges while preserving SP/HTAB-only whitespace semantics; controls
+remain visible to malformed-header diagnostics.
+After this final parser change, all 16 application-boundary/live-harness examples,
+the one-million-case native fuzz run (94.25 seconds), and Steep passed.
+All 5,013 binary boundary/random inputs preserved the previous trimming bytes
+and encoding. Linux tcpdump/tshark comparisons passed all 26 examples.
+
+Ruby 4.0 introduced an additional restriction in
+[`IO::Buffer.map`](https://docs.ruby-lang.org/en/4.0/IO/Buffer.html#method-c-map):
+it rejects files with zero size even when a mapping size is supplied. Linux
+AF_PACKET sockets have zero file size, so the pure-Ruby ring backend cannot map
+them on that runtime. Ruby 3.3/3.4 ring support remains available; Ruby 4.0 uses
+the socket backend. No C extension or Fiddle workaround is introduced. Ring
+availability on Ruby 4.0 remains an upstream API constraint on full acceptance.
+
+Long-running controlled tests started at 2026-10-01 14:55:57 UTC with a frozen
+capture implementation digest
+`084591da70d195146b37aff76de7ec9495085d045684decc26da6c34c4ebc20e`:
+
+| Check | Requested workload | Evidence directory | Earliest completion |
+| --- | --- | --- | --- |
+| aarch64 socket/ring comparison | 50,000 pps, 3,600 seconds | `tmp/acceptance-arm-hour-final` | 2026-10-01 15:55:58 UTC |
+| Bounded ring pcapng rotation | 100 pps, 259,200 seconds | `tmp/acceptance-rotation-72h-final` | 2026-10-04 14:55:58 UTC |
+
+These directories contain local private captures and are excluded from Git.
+Completion requires the final `result.json` to pass, inspection of the periodic
+resource samples, and successful external validation of rotated files. The
+controlled veth soak does not establish the separate busy physical-interface
+24-hour gate. Results and unfulfilled conditions are recorded explicitly below.
+
+The frozen rotation run first reported 1,178 kernel drops and three ring freezes
+at the 4,200-second sample, after zero drops at 3,600 seconds. The macOS power log
+records hibernation/wake in that interval with a 10.614-second recovery delay.
+This correlation does not isolate the cause, and the run is not loss-free
+acceptance evidence. Its periodic resource observations are retained. External
+capinfos/tshark inspection now runs after capture closes, validating the three
+retained rotation files rather than blocking packet reads at every rotation.
+A four-second rotation smoke captured and verified all 4,000 frames without
+drops, gaps or malformed records and passed both external tools.
+
+A fresh frozen-source rotation run started at 2026-10-01 16:22:04 UTC, using
+capture digest
+`7fd49cc4c24664c298c99dbf35276f64e67a1195093c3e1c4f9190fa9d234014`
+and harness digest
+`55b36abe852549836350d51ea9994b71ded5409794d853777f0634ca6830c760`.
+Its private evidence is in `tmp/acceptance-rotation-72h-rc2`; completion cannot
+be checked before 2026-10-04 16:22:06 UTC. macOS idle/system sleep assertions
+were confirmed on AC power with a 72-hour-plus-ten-minute `caffeinate` timer.
+This run is still in progress and does not count as completed acceptance.
+
+The one-hour aarch64 comparison completed successfully: each backend captured
+all 180,000,000 frames with zero drops/gaps and the same complete-byte digest.
+All 18,000 paired timestamp samples were identical. Ring final drain was
+47.638 ms; descriptor counts remained 10/11. Last-half-hour RSS ranged from
+31.2–40.2 MiB for sockets and 86.5–86.6 MiB for rings. P6-06 is complete, so
+aarch64/arm64 Linux now tries ring capture automatically, retaining socket
+fallback when mapping is unavailable. This does not complete the separate
+72-hour rotation soak or the mixed x86_64 throughput gates.
+
+The physical macOS en0 interface is active, but `/dev/bpf*` requires administrator
+access and `sudo -n` requires a password in this environment. The user could not
+start the privileged run, so the 24-hour physical-interface check has not begun.
+An independent three-second observation measured approximately 82 packets/s and
+21 kB/s; it does not establish a busy workload. `bench/soak.rb` is ready to record
+24-hour elapsed time, bounded private pcapng rotation, traffic rates, RSS and
+descriptor counts once capture privileges are available. Its one-second Linux
+smoke and interrupted-run checks passed, including capinfos/tshark file checks.
+
 ## 2.0.0.rc1
 
-The implementation covers the v2 roadmap: the packet/dissector API, network and
+The release candidate implements the functional parts of the v2 roadmap: the packet/dissector API, network and
 application protocols, Ruby cBPF compiler/VM, Linux socket/ring and macOS BPF
 backends, capture files/rotation/privilege drop, bounded stateful analysis,
 statistics, follow output and documentation. This is a release candidate,
@@ -46,8 +164,6 @@ for throughput figures and measurement limits.
 
 - Run a 24-hour continuous capture stability check.
 - Run a 72-hour rotation soak and record RSS, descriptor counts and drop rates.
-- Run an hour of high-load aarch64 socket/ring comparison before changing its
-  automatic backend default.
 - Confirm x86_64 single-core throughput and live drop targets on the specified
   workload; file benchmarks alone cannot establish loss-free live throughput.
 - Keep rc1 available for two weeks, classify reports, and fix critical/high

data/docs/json-schema.json

@@ -14,12 +14,12 @@
       "properties": {
         "number": { "type": "integer", "minimum": 1 },
         "time_epoch_ns": { "type": "integer" },
-        "time": { "type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\\.[0-9]{9}Z$" },
+        "time": { "type": "string", "description": "UTC calendar time with nine fractional digits; years outside 0000..9999 use expanded signed/long year notation. time_epoch_ns preserves the exact instant.", "pattern": "^-?[0-9]{4,}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\\.[0-9]{9}Z$" },
         "caplen": { "type": "integer", "minimum": 0 },
         "len": { "type": "integer", "minimum": 0 },
         "interface": { "type": ["string", "null"] },
         "direction": { "enum": ["in", "out", null] },
-        "linktype": { "type": "integer", "minimum": 0 }
+        "linktype": { "type": "integer", "minimum": 0, "maximum": 65535 }
       }
     },
     "layers": {

data/lib/redhound/analysis/flow_key.rb

@@ -27,7 +27,13 @@ module Redhound
                     layers.reverse.find { |item| %i[ipv4 ipv6].include?(item.protocol) }
                   end
         return nil unless network && network[:src] && network[:dst]
-        proto = type || (transport ? transport.protocol : network.protocol)
+        proto = type || transport&.protocol
+        unless proto
+          position = layers.index(network) #: Integer
+          extensions = layers.drop(position + 1).take_while { |item| item.protocol == :ipv6_ext }
+          number = network.protocol == :ipv4 ? network[:proto] : extensions.last&.[](:next) || network[:nxt]
+          proto = "ip_proto_#{number}".to_sym
+        end
         if %i[tcp udp].include?(proto)
           return nil unless transport && transport[:srcport] && transport[:dstport]
           ports = [transport[:srcport], transport[:dstport]]

data/lib/redhound/analysis/flow_table.rb

@@ -7,7 +7,7 @@ module Redhound
     # @api private
     class Flow
       BASE_BYTES = 4096
-      attr_reader :key, :id, :first_ns, :packets, :bytes, :streams, :applications, :acks, :keep_alives, :http_methods
+      attr_reader :key, :id, :first_ns, :packets, :bytes, :streams, :applications, :acks, :keep_alives, :http_methods, :probes, :probe_disabled
       attr_accessor :last_ns, :closed_ns, :syn_ns, :syn_direction, :synack_ns, :synack_seq, :initial_rtt, :accounted_bytes
       # @rbs (untyped key, Integer id, Integer timestamp_ns) -> void
       def initialize(key, id, timestamp_ns)
@@ -15,13 +15,15 @@ module Redhound
         @packets, @bytes, @streams, @applications, @acks = [0, 0], [0, 0], [nil, nil], [nil, nil], [nil, nil]
         @keep_alives = [false, false]
         @http_methods = [Array.new, Array.new] #: Array[untyped]
+        @probes = [Array.new, Array.new] #: Array[untyped]
+        @probe_disabled = [false, false]
         @closed_ns = @syn_ns = @syn_direction = @synack_ns = @synack_seq = @initial_rtt = nil # @rbs untyped
         @accounted_bytes = BASE_BYTES
       end
       # @rbs () -> bool
       def closed? = !@closed_ns.nil?
       # @rbs () -> Integer
-      def bytesize = BASE_BYTES + @streams.compact.sum(&:bytesize) + @applications.compact.sum(&:bytesize) + @http_methods.sum { |methods| methods.size * 16 }
+      def bytesize = BASE_BYTES + @streams.compact.sum(&:bytesize) + @applications.compact.sum(&:bytesize) + @http_methods.sum { |methods| methods.size * 16 } + @probes.sum { |parts| parts.sum { |data, _frame| data.bytesize * 2 + 128 } }
     end
 
     # @api private
@@ -82,9 +84,14 @@ module Redhound
           stale
         end
       end
-      # @rbs () -> Flow?
-      def evict
-        pair = @entries.shift
+      # @rbs (?Symbol? type) -> Flow?
+      def evict(type = nil)
+        pair = if type
+                 key = @entries.each_key.find { |item| item[0] == type }
+                 key ? [key, @entries.delete(key)] : nil
+               else
+                 @entries.shift
+               end
         return nil unless pair
         @evicted += 1
         discard(pair[1])

data/lib/redhound/analysis/ip_reassembler.rb

@@ -12,6 +12,7 @@ module Redhound
         raise ArgumentError, 'fragment limits must be positive' unless max_bytes.positive? && timeout_ns.positive?
         @max_bytes, @timeout_ns, @clock, @evicted, @expired = max_bytes, timeout_ns, 0, 0, 0
         @next_sweep = 0
+        @bytes = 0
         @entries, @discarded = {}, {} # @rbs untyped
       end
       # @rbs (Packet packet) -> untyped
@@ -21,6 +22,8 @@ module Redhound
           if layer.protocol == :ipv4 && layer[:src] && layer[:dst] && (layer[:mf] == 1 || layer[:frag_offset].to_i.positive?)
             return [layer, layer, [4, layer[:src], layer[:dst], layer[:proto], layer[:id]], layer[:frag_offset], layer[:mf] == 1]
           elsif layer.protocol == :ipv6_ext && layer[:type] == 44 && layer[:fragment_offset] && layer[:identification]
+            # RFC 6946: atomic fragments bypass queued fragments and tombstones.
+            next if layer[:fragment_offset].zero? && layer[:more] == 0
             network = layers.take(index).reverse.find { |item| item.protocol == :ipv6 }
             return [network, layer, [6, network[:src], network[:dst], layer[:identification]], layer[:fragment_offset], layer[:more] == 1] if network && network[:src] && network[:dst]
           end
@@ -38,7 +41,7 @@ module Redhound
         length = network.payload_end - fragment.payload_offset
         if offset + length > 65_535 || length.negative? || (more && (length.zero? || length % 8 != 0))
           fragment.diagnose(:error, :malformed, 'invalid IP fragment size or offset')
-          @entries.delete(key)
+          remove(key)
           return nil
         end
         if network.diagnostics.any? { |diagnostic| diagnostic.code == :truncated }
@@ -46,7 +49,7 @@ module Redhound
           return nil
         end
         data = packet.data.byteslice(fragment.payload_offset, length) #: String
-        state = @entries.delete(key) || { parts: Array.new, frames: Array.new, header: nil, next_offset: 6, next: fragment[:next], end: nil } #: untyped
+        state = remove(key) || { parts: Array.new, frames: Array.new, header: nil, next_offset: 6, next: fragment[:next], end: nil } #: untyped
         if key[0] == 6 && state[:next] != fragment[:next]
           fragment.diagnose(:error, :malformed, 'IPv6 fragments have inconsistent next headers')
           return nil
@@ -58,11 +61,13 @@ module Redhound
           fragment.diagnose(:warning, :fragment_overlap, 'overlapping IP fragments')
           if key[0] == 6
             @discarded[key] = @clock
+            @bytes += 128
             while @discarded.size > [4096, @max_bytes / 128].min
               @discarded.shift
+              @bytes -= 128
               @evicted += 1
             end
-            while bytesize > @max_bytes && @entries.shift
+            while bytesize > @max_bytes && remove_oldest
               @evicted += 1
             end
             return nil
@@ -103,12 +108,13 @@ module Redhound
           end
         end
         @entries[key] = state
+        @bytes += state_bytesize(state)
         if complete?(state)
-          @entries.delete(key)
+          remove(key)
           return virtual_packet(packet, state, key[0])
         end
         while bytesize > @max_bytes
-          removed = @entries.shift
+          removed = remove_oldest
           break unless removed
           @evicted += 1
           fragment.diagnose(:warning, :reassembly_gap, 'IP reassembly memory limit exceeded')
@@ -158,16 +164,43 @@ module Redhound
         @next_sweep = now_ns + 1_000_000_000
         @entries.delete_if do |_key, state|
           stale = now_ns - state[:time] >= @timeout_ns
-          @expired += 1 if stale
+          if stale
+            @expired += 1
+            @bytes -= state_bytesize(state)
+          end
+          stale
+        end
+        @discarded.delete_if do |_key, time|
+          stale = now_ns - time >= @timeout_ns
+          @bytes -= 128 if stale
           stale
         end
-        @discarded.delete_if { |_key, time| now_ns - time >= @timeout_ns }
       end
       # @rbs () -> Integer
-      def bytesize
-        @discarded.size * 128 + @entries.values.sum do |state|
-          512 + (state[:header]&.bytesize || 0) + state[:frames].size * 32 + state[:parts].sum { |_pos, bytes| bytes.bytesize + 128 }
-        end
+      def bytesize = @bytes
+      # @rbs (untyped state) -> Integer
+      def state_bytesize(state)
+        512 + (state[:header]&.bytesize || 0) * 2 + state[:frames].size * 32 + state[:parts].sum { |_pos, bytes| bytes.bytesize * 2 + 168 }
+      end
+      # @rbs (untyped key) -> untyped
+      def remove(key)
+        state = @entries.delete(key)
+        @bytes -= state_bytesize(state) if state
+        state
+      end
+      # @rbs () -> untyped
+      def remove_oldest
+        pair = @entries.shift
+        @bytes -= state_bytesize(pair[1]) if pair
+        pair
+      end
+      # @rbs () -> Integer
+      def finish
+        incomplete = @entries.size
+        @entries.clear
+        @discarded.clear
+        @bytes = 0
+        incomplete
       end
       # @rbs () -> Integer
       def size = @entries.size

data/lib/redhound/analysis/stats.rb

@@ -38,6 +38,7 @@ module Redhound
           result = to_h
           io.puts("#{result[:kind]}#{result[:type] ? ",#{result[:type]}" : ''}")
           result[:rows].each { |item| io.puts(item.map { |name, value| "#{name}=#{value.is_a?(Array) ? value.join(':') : value}" }.join(' ')) }
+          io.puts("#{result[:omitted_intervals]} earlier IO intervals omitted") if result[:omitted_intervals] && result[:omitted_intervals].positive?
           io.puts("#{@evicted} statistics rows evicted") if @evicted.positive?
         end
         # @rbs () -> Hash[Symbol, untyped]
@@ -58,12 +59,23 @@ module Redhound
           @start_ns ||= packet.timestamp_ns
           start = @start_ns #: Integer
           interval = [(packet.timestamp_ns - start) / @interval_ns, 0].max
-          item = row(interval) { { interval: interval, start_ns: start + interval * @interval_ns, end_ns: start + (interval + 1) * @interval_ns, packets: 0, bytes: 0 } }
+          item = row(interval) { interval_row(interval) }
           item[:packets] += 1
           item[:bytes] += packet.original_length
         end
         # @rbs () -> Hash[Symbol, untyped]
-        def to_h = { kind: :io, interval_ns: @interval_ns, rows: @rows.values.sort_by { |item| item[:interval] } }
+        def to_h
+          return { kind: :io, interval_ns: @interval_ns, rows: [] } if @rows.empty?
+          last = @rows.keys.max #: Integer
+          first = [@rows.keys.min, last - @max_rows + 1].max #: Integer
+          rows = (first..last).map { |interval| @rows[interval] || interval_row(interval) }
+          { kind: :io, interval_ns: @interval_ns, omitted_intervals: first, rows: rows }
+        end
+        # @rbs (Integer interval) -> Hash[Symbol, Integer]
+        def interval_row(interval)
+          start = @start_ns || 0
+          { interval: interval, start_ns: start + interval * @interval_ns, end_ns: start + (interval + 1) * @interval_ns, packets: 0, bytes: 0 }
+        end
       end
 
       # @api private