vanken 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6bfbb1b288a09cb5fb170bed462adb00f4ae86aa21ee62c247e9e2cefe2aaa97
-  data.tar.gz: 3f8f754621be04e6fc2a4dac72abae4c10f57a8d3524f018c67825abe18b4e8a
+  metadata.gz: 3ff6f270de34fbee27bb93839749a5105ece2667ef59e3281c28e37d5dee782a
+  data.tar.gz: cfb0118a6e6c0b1b6a026ca8254e58b2e20f260b11e75bedbbce21a6aa47755b
 SHA512:
-  metadata.gz: bc1c63f6df229fbb85db6e00cd69059faa16a69f91469f5cd8a3d7d6ad7b8e0bcce3068ed31ebb4533c77906188b3448026a6e5a1cfe581a2d1f5d4fb058cce0
-  data.tar.gz: 370e198b35f3c4bb080f773b026ae6005c3457bd4412d9be8b13e1ea868c182b0919cf8d4b041325231a44f1ec8bf2d556dc15eb01ba1bbff47893cde91c6d49
+  metadata.gz: d7913b548c7ce8fd8bb0531542b70d633c1c06525bb2be762230661ec2d2691a233c81116376ab3c6fa0857db76c8796c4e585271431fd6025af7c130fc55fab
+  data.tar.gz: 1de08c7960be93a43239e88564f3267622406597909a7bd18d7ca59f5bf77dab29526da4b2852405cf711a2066342bc332bad70121ad9f67bead285eb9b6812b

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 0.2.0
+
+- Add editable packet coloring, four search modes, marks, ignored packets, time references, and conversation navigation.
+- Add Follow TCP Stream with retransmission handling, missing-data indicators, searching, saving, and stream exclusion.
+- Add protocol hierarchy, conversations, endpoints, expert information, capture properties, and filtered I/O graphs.
+- Add typed custom columns, Decode As rules, trusted Ruby dissector plugins, packet-range exports, and JSON, NDJSON, text, and CSV exports.
+- Add the command palette, profile-based preferences, Japanese and English screens, high-contrast themes, and keyboard operation in the terminal UI.
+- Add automatic capture stopping, rotating pcapng files, interface traffic graphs, optional asynchronous address resolution, and interrupted-capture recovery.
+- Accelerate supported address and port display filters while preserving ordinary evaluation for other packets and dissector extensions.
+- Preserve captured packets awaiting analysis when saving, and retain filter results and current selection through reanalysis.
+
+Live acquisition can outpace analysis on slower systems. Captured packets remain available while queued analysis finishes; redraw latency on measured Linux systems remains above the design target. See [measured performance](https://github.com/ydah/vanken/blob/main/docs/performance.md).
+
+Requires Ruby 3.3 or newer. Live capture supports Linux and macOS; Windows supports file inspection. See [capture permissions](https://github.com/ydah/vanken/blob/main/packaging/README.md).
+
 ## 0.1.0
 
 Initial release.

data/README.md

@@ -10,21 +10,20 @@ Packet acquisition runs in a separate helper; ordered packet analysis runs in an
 - Linux with a desktop session, or macOS. File inspection is also available on Windows; live capture uses Linux packet sockets or macOS BPF.
 - Linux desktop dependencies: Vulkan loader and drivers, fonts, and `zenity` for native file dialogs. On Ubuntu: `sudo apt install libvulkan1 mesa-vulkan-drivers fonts-dejavu-core fonts-noto-cjk zenity`.
 
-## Run from source
+## Install
 
 ```sh
-git clone https://github.com/ydah/vanken.git
-cd vanken
-bundle install
-bundle exec exe/vanken capture.pcapng
+gem install vanken
+vanken capture.pcapng
 ```
 
-Initial RubyGems publication is pending. To install a locally built gem:
+For a source checkout:
 
 ```sh
-gem build --strict vanken.gemspec
-gem install ./vanken-0.1.0.gem
-vanken capture.pcapng
+git clone https://github.com/ydah/vanken.git
+cd vanken
+bundle install
+bundle exec exe/vanken capture.pcapng
 ```
 
 Strict gem builds require RubyGems 4.0.16 or newer; older versions reject the pinned redhound prerelease with a recommendation warning.
@@ -43,10 +42,24 @@ frame.len > 1000 && !udp
 
 Vanken display filters (VDF) and acquisition filters (BPF) are separate languages. See [display filters](docs/filters.md) for operators, types, and repeated fields.
 
+## Analysis
+
+Use the menus or the command palette (Ctrl+Shift+K; Command+Shift+K in the macOS desktop application) to find actions. Search by display filter, hexadecimal bytes, string, or regular expression. Mark and ignore packets, change the time reference, and move between packets in the same conversation. Packet and field context menus expose filter and analysis actions.
+
+Coloring rules can be edited, reordered, imported, and exported. Add a protocol field as a custom column from the details tree, then change its position, width, or visibility in the column editor. Custom columns sort by their field types.
+
+Follow TCP Stream reconstructs each direction, reports missing data, and supports ASCII, hexadecimal, and raw views, searching, saving, and excluding a stream. Statistics include protocol hierarchy, conversations, endpoints, and packet properties. I/O graphs accept display filters per series and intervals from 0.01 to 60 seconds. Expert information links diagnostics to their packets.
+
+Decode As rules and registered Ruby dissector plugins are applied to packet summaries, details, and filter workers. Plugins require an explicit trust confirmation before loading. Save captures as pcap or pcapng, export selected ranges, or export dissections as JSON, NDJSON, text, or CSV.
+
+Preferences include Japanese and English, dark/light/system/high-contrast themes, analysis limits, and optional asynchronous address resolution. Profiles keep preferences, columns, coloring, bookmarks, Decode As rules, and plugins separate; recent files and window geometry are shared. Stop capture before switching profiles or changing dissectors. Analysis preferences changed during loading or capture are applied when it finishes. Interrupted captures can be recovered or discarded on the next start.
+
 ## Capture
 
 Choose Start, select an interface, and optionally set a BPF acquisition filter. Stop preserves captured packets for inspection and saving. Vanken asks before discarding an unsaved capture.
 
+Capture options include automatic stopping by packet count, duration, or byte count, and rotating pcapng files by size or time with a bounded file count. The welcome screen shows interface traffic rates. Saving includes all durable packets even while analysis is still catching up.
+
 ```sh
 bundle exec exe/vanken-capture --list-interfaces
 bundle exec exe/vanken-capture --check --interface lo
@@ -64,6 +77,8 @@ bundle exec exe/vanken --headless --smoke
 
 `--no-yjit` disables default YJIT activation. `--debug` enables debug logging. Preferences use safe YAML in the platform's user configuration directory; logs rotate without retaining raw packet bytes.
 
+The terminal UI supports opening captures, selecting packets, applying filters, and starting/stopping capture with keyboard actions. Details of keyboard controls, settings files, and analysis limits are in the [usage guide](docs/usage.md).
+
 ## Development
 
 ```sh
@@ -75,9 +90,9 @@ bundle exec ruby script/benchmark.rb
 script/capture-ci.sh
 ```
 
-To develop with a sibling Zaniah checkout, set `VANKEN_ZANIAH_PATH=../zaniah` when running Bundler. Production dependencies are redhound `2.0.0.rc2` and Zaniah `~> 0.12.3`, including fixes for growing memory use and repeated style allocations during redraws.
+To develop with a sibling Zaniah checkout, set `VANKEN_ZANIAH_PATH=../zaniah` when running Bundler. Production dependencies are redhound `2.0.0.rc2` and Zaniah `~> 0.12.4`.
 
-See [upstream contracts](docs/upstream.md), [measured performance](docs/performance.md), and [release procedure](docs/releases.md). Version 0.1 implements the file inspection, acquisition, and display filter milestones. Statistics, stream following, coloring, profiles, and later extensions are scheduled for subsequent releases.
+See [upstream contracts](docs/upstream.md), [measured performance](docs/performance.md), and [release procedure](docs/releases.md). Live acquisition can outpace analysis, and redraw latency on slower Linux machines remains above the design target; captured packets are retained while queued analysis finishes.
 
 ## License
 

data/data/coloring_rules.yml

@@ -0,0 +1,67 @@
+schema_version: 1
+rules:
+  - name: Bad TCP
+    filter: tcp.analysis.retransmission || tcp.analysis.out_of_order || tcp.analysis.lost_segment || tcp.analysis.duplicate_ack || tcp.analysis.zero_window
+    light: { fg: "#F78787", bg: "#121212" }
+    dark: { fg: "#F78787", bg: "#1A1A1A" }
+    enabled: true
+  - name: Malformed
+    filter: expert.severity == error
+    light: { fg: "#FFFC9C", bg: "#A40000" }
+    dark: { fg: "#FFFC9C", bg: "#7A0000" }
+    enabled: true
+  - name: Checksum Errors
+    filter: expert.code == "bad_checksum"
+    light: { fg: "#F78787", bg: "#121212" }
+    dark: { fg: "#F78787", bg: "#1A1A1A" }
+    enabled: true
+  - name: TCP RST
+    filter: tcp.flags.rst
+    light: { fg: "#FFFC9C", bg: "#A40000" }
+    dark: { fg: "#FFFC9C", bg: "#7A0000" }
+    enabled: true
+  - name: TCP SYN/FIN
+    filter: tcp.flags.syn || tcp.flags.fin
+    light: { fg: "#12272E", bg: "#A0A0A0" }
+    dark: { fg: "#E0E0E0", bg: "#4A4A4A" }
+    enabled: true
+  - name: HTTP
+    filter: http
+    light: { fg: "#12272E", bg: "#E4FFC7" }
+    dark: { fg: "#E4FFC7", bg: "#25331A" }
+    enabled: true
+  - name: TLS
+    filter: tls
+    light: { fg: "#12272E", bg: "#D6E8FF" }
+    dark: { fg: "#D6E8FF", bg: "#1E2A3A" }
+    enabled: true
+  - name: DNS
+    filter: dns
+    light: { fg: "#12272E", bg: "#C9F0FF" }
+    dark: { fg: "#C9F0FF", bg: "#16303A" }
+    enabled: true
+  - name: ICMP
+    filter: icmp || icmpv6
+    light: { fg: "#12272E", bg: "#FCE0FF" }
+    dark: { fg: "#FCE0FF", bg: "#3A2440" }
+    enabled: true
+  - name: ARP
+    filter: arp
+    light: { fg: "#12272E", bg: "#FAF0D7" }
+    dark: { fg: "#FAF0D7", bg: "#3A331F" }
+    enabled: true
+  - name: Broadcast
+    filter: eth.dst == ff:ff:ff:ff:ff:ff
+    light: { fg: "#BABDB6", bg: "#FFFFFF" }
+    dark: { fg: "#9A9D96", bg: "#2A2A2A" }
+    enabled: true
+  - name: UDP
+    filter: udp
+    light: { fg: "#12272E", bg: "#DAEEFF" }
+    dark: { fg: "#DAEEFF", bg: "#1C2833" }
+    enabled: true
+  - name: TCP
+    filter: tcp
+    light: { fg: "#12272E", bg: "#E7E6FF" }
+    dark: { fg: "#E7E6FF", bg: "#24233A" }
+    enabled: true

data/docs/performance.md

@@ -111,3 +111,13 @@ The final single-run check with public Zaniah 0.12.3 and the same Ruby, platform
 Before visible-row batching, the same operation failed its first selection at 105.252 ms. A separate diagnostic reproduced a 107.234 ms background queue wait behind 11 row jobs, while actual detail analysis took 2.677 ms and byte reading 0.129 ms. Nine intermediate renders consumed most of that wait. PacketSource now collects visible-row requests after rendering, fetches them in one background job, and publishes their values in one foreground update. The passing run followed this runtime change; the earlier failure remains part of the evidence.
 
 The native and synthetic runs above do not substitute for this five-minute Linux capture test. Varied addresses, large TCP flow sets, reassembly, and plugins are outside the repeated-UDP benchmark's scope.
+
+## Analysis extensions and cBPF
+
+The address/port optimization uses the public verified capture-filter compiler for eligible complete Ethernet IPv4/IPv6 TCP/UDP packets. It falls back to VDF for unsupported expressions, truncation, fragmentation, extension headers, common UDP tunnels, other link types, Decode As, or plugins. This preserves the fields seen by the dissector.
+
+On 2026-10-02, Ruby 4.0.6 with YJIT on arm64 Darwin compared the same 20,000 complete Ethernet/IPv4/UDP frames and expression `ip.addr == 192.0.2.1 && udp.port == 54321`. Three sequential samples per evaluator all matched 20,000 packets. Median stateless VDF dissection/evaluation took 0.339384 s, and the eligible cBPF path took 0.020937 s: 16.21 times faster. This comparison excludes file IO and worker startup. Run `bundle exec ruby --yjit script/filter-performance.rb 20000` to repeat it.
+
+A separate one-million-packet run of the production document measured 0.5529 s for the existing fast filter, 6.2598 s for the four-worker `ip.ttl == 64` scan, and 2.7721 s for the four-worker address/port cBPF scan. Each matched all one million packets. Analysis took 115.6605 s (8,646 packets/s), with 67.2 B/frame retained parent heap before filtering and 81.08 B/frame after the worker scan. This run overlapped integration and UI tests and is not an isolated comparison with the earlier analysis-throughput result. RSS sampling was unavailable and is recorded as null; these heap figures do not establish the RSS target.
+
+The I/O graph check seeds actual frame metadata, packed columns, and annotations for 200,000 TCP frames, then times two series at each supported interval. It excludes packet parsing. The observed interval rebuilds took 0.30–0.33 s, each below the one-second target, with every packet counted in both series. Run `bundle exec ruby --yjit script/analysis-performance.rb` to repeat it. Nightly validation retains the one-million-packet benchmark, evaluator comparison, graph timings, and fuzz results as workflow artifacts.

data/docs/releases.md

@@ -4,12 +4,10 @@ Work is committed on main. Release notes and CHANGELOG entries include only user
 
 Before tagging, run `bundle exec rake`, verify generated RBS is current, run privileged capture and benchmarks, perform native smoke, and build the gem strictly. Version, tag, gemspec dependencies, and lockfile must agree. Exclude captures, spools, test artifacts, local dependencies, and credentials from the gem.
 
-Initial 0.1.0 notes are exactly `Initial release.`. Publication is paused until the owner publishes the first RubyGem and configures trusted publishing. No initial release tag is pushed automatically.
-
-For the first publication, run `gem push pkg/vanken-0.1.0.gem`, then configure the publisher identity below. Tagging `v0.1.0` afterwards runs validation and creates the GitHub release; the job skips uploading a version already present on RubyGems.
+Initial 0.1.0 notes are exactly `Initial release.`. Its owner publication and trusted publishing setup are complete. The job skips uploading a version already present on RubyGems.
 
 Subsequent `vX.Y.Z` tags trigger `.github/workflows/release.yml`, based on the Canopus release job. It verifies version, runs checks, builds the gem, publishes with RubyGems trusted publishing, and creates GitHub release notes from CHANGELOG.
 
 Trusted publisher identity: owner `ydah`, repository `vanken`, workflow `release.yml`, GitHub environment `release`.
 
-The design milestones are 0.1 for file inspection, capture, and filters; 0.2 for coloring, search, stream following, statistics, and analysis extensions; and 0.3 for profiles, localization, permission packaging, recovery UI, and additional capture conveniences. Work stops at the initial publication gate before the later milestones.
+The design milestones are 0.1 for file inspection, capture, and filters; 0.2 for coloring, search, stream following, statistics, and analysis extensions; and 0.3 for profiles, localization, permission packaging, recovery UI, and additional capture conveniences. Some 0.3 foundations are included with 0.2 so analysis extensions can share profiles and settings. The 0.3 release completes the administrator setup command and includes permission packaging as downloadable release assets.

data/docs/usage.md

@@ -0,0 +1,51 @@
+# Using Vanken
+
+## Keyboard and terminal operation
+
+The desktop and terminal views share the same commands. Use Tab and Shift+Tab to move between controls, Enter to activate, Escape to close dialogs, and arrow keys in lists. In the terminal and on Linux, use Ctrl for the shortcuts below; the native macOS window uses Command instead.
+
+| Action | Shortcut |
+| --- | --- |
+| Open capture | Ctrl+O |
+| Save as | Ctrl+Shift+S |
+| Start or stop capture | Ctrl+E |
+| Restart capture | Ctrl+Shift+R |
+| Preferences | Ctrl+Shift+P |
+| Command palette | Ctrl+Shift+K |
+| Find packet / next / previous | Ctrl+F / Ctrl+N / Ctrl+B |
+| Mark / ignore / time reference | Ctrl+M / Ctrl+D / Ctrl+T |
+| Previous / next packet | Alt+Up / Alt+Down |
+| Previous / next in conversation | Ctrl+, / Ctrl+. |
+| Go to packet | Ctrl+G |
+| Follow TCP Stream | Ctrl+Alt+Shift+T |
+| Decode As | Ctrl+Shift+U |
+| Reload plugins | Ctrl+Shift+L |
+| Quit | Ctrl+Q |
+
+Terminals differ in the keys they transmit. Use the command palette for a command whose shortcut is intercepted by the terminal. Type an action name and use Up/Down and Enter to run it. Column movement, width, and visibility have controls in the column editor, so they do not require dragging.
+
+## Profiles and settings
+
+Preferences change the appearance, packet list, acquisition defaults, analysis limits, and optional name resolution. The language defaults to Japanese when `LANG` begins with `ja`, and English otherwise. Switching language rebuilds the current screen. High contrast is available from the theme menu or Preferences.
+
+Create, duplicate, delete, and switch profiles from Preferences or the Profiles command. The default profile lives in the configuration directory; other profiles live under `profiles/NAME`. Each has `preferences.yml`, `columns.yml`, `coloring_rules.yml`, `filters.yml`, `decode_as.yml`, and `plugins.yml`. Configuration uses schema version 1, safe YAML, and atomic saves. Recent-file history and window geometry are shared outside profiles.
+
+Stop acquisition before changing profiles or dissectors. Changes to analysis preferences during acquisition or file loading wait until it finishes. Reanalysis keeps the original bytes, marks, ignored packets, time references, and current selection, then refreshes summaries, details, and filters. Registered Ruby plugins execute with your account's permissions and require a trust confirmation before their first load.
+
+## Exporting and following streams
+
+Export all analyzed packets, displayed packets, the selected packet, marked packets, the range between the first and last marks, or explicit ranges such as `1-10,15,20-`. An option excludes ignored packets. pcapng preserves interface metadata; pcap requires compatible link types. Ordinary Save includes all durable packets, including packets still awaiting analysis.
+
+JSON follows redhound's packet schema; NDJSON writes one packet per line. CSV exports visible columns, including custom fields. Text exports the protocol tree. Outputs are written to a temporary file and replace the destination only after completion.
+
+Follow TCP Stream removes duplicate retransmissions, handles sequence wrap and out-of-order segments, and displays missing-data markers. The display preview is limited to 16 MiB; saving reads the full reconstructed stream. Raw saving omits missing bytes rather than inventing data.
+
+## Limits and recovery
+
+I/O graphs offer intervals of 0.01, 0.1, 1, 10, and 60 seconds. They show at most the latest 100,000 intervals and report omitted earlier intervals; choose a longer interval for long captures. Custom-column cells show at most 16 values and 4,096 bytes; the details and byte panes retain the full packet.
+
+Acquisition can stop by packet count, duration, or byte count. Ring saving rotates pcapng files by size or time and replaces older slots after the configured file count. A size limit may be crossed by the packet that triggers rotation; packets are never split across files.
+
+After an interrupted capture, the next start offers recovery or discard for owned, inactive session directories. Recovery reconstructs the analysis and keeps the capture unsaved until you save it. Sessions still used by a running process are excluded. A failed recovery preserves the raw session for another attempt.
+
+See [capture permissions](../packaging/README.md) for administrator setup and [performance measurements](performance.md) for tested workloads and current limits.

data/lib/vanken/app/capture_controller.rb

@@ -4,11 +4,50 @@ require_relative "../errors"
 require_relative "../capture/launcher"
 require_relative "../capture/control_protocol"
 require_relative "../gateway/file_reader"
+require_relative "../gateway/file_writer"
 require_relative "document"
 
 module Vanken
   module App
     class CaptureController
+      class RingReader
+        def initialize(reader, writer, on_error:)
+          @reader, @writer, @on_error = reader, writer, on_error
+          @flushed_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        end
+
+        def next_frame(timeout:)
+          frame = @reader.next_frame(timeout: timeout)
+          @writer << frame if frame
+          now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          if !frame || now - @flushed_at >= 0.05
+            @writer.flush
+            @flushed_at = now
+          end
+          frame
+        rescue Vanken::FileError, IOError, SystemCallError
+          @on_error.call
+          raise
+        end
+
+        def eof? = @reader.eof?
+        def stop = @reader.stop
+        def close
+          return if @closed
+          @closed = true
+          begin
+            @writer.write_stats(@reader.stats)
+          ensure
+            begin
+              @writer.close
+            ensure
+              @reader.close
+            end
+          end
+        end
+      end
+      private_constant :RingReader
+
       class Error < Vanken::CaptureError
         attr_reader :code
 
@@ -20,6 +59,7 @@ module Vanken
 
       def initialize(launcher: nil, preferences: nil, on_update: nil, on_document: nil)
         @preferences, @on_update, @on_document = preferences, on_update, on_document
+        @custom_launcher = !launcher.nil?
         @launcher = launcher || Capture::Launcher.new(strategy: preferences&.get("capture.launcher") || :auto)
         @mutex = Mutex.new
         @state, @stats = :idle, {}
@@ -35,6 +75,14 @@ module Vanken
       def running? = @mutex.synchronize { !!@worker&.alive? }
       def snapshot = @mutex.synchronize { {state: @state, stats: @stats.dup, error: @error, warning: @warning, document: @document} }
 
+      def preferences=(value)
+        @mutex.synchronize do
+          raise Error, "cannot change capture preferences while capture is running" if @worker&.alive?
+          @preferences = value
+          @launcher = Capture::Launcher.new(strategy: value&.get("capture.launcher") || :auto) unless @custom_launcher
+        end
+      end
+
       def start(options)
         @mutex.synchronize do
           raise Error, "capture controller is closed" if @closed
@@ -89,7 +137,9 @@ module Vanken
       private
 
       def run
-        handle = @launcher.launch(@options)
+        helper_options = @options.is_a?(Hash) ? @options.reject { |key, _| key.start_with?("ring_") } : @options
+        ring_writer = open_ring
+        handle = @launcher.launch(helper_options)
         @mutex.synchronize do
           @handle = handle
           stop_helper if @stop_requested
@@ -99,6 +149,7 @@ module Vanken
         @mutex.synchronize { @document = doc }
         @on_document&.call(doc)
         reader = Gateway::FileReader.new(handle.stdout)
+        reader = RingReader.new(reader, ring_writer, on_error: -> { stop }) if ring_writer
         doc.ingest(reader, live: true)
         doc.wait
         unless control.join(3)
@@ -122,6 +173,7 @@ module Vanken
         control&.join
         doc&.wait
         reader&.close unless doc
+        ring_writer&.close
         handle&.close
         @mutex.synchronize do
           @handle = nil
@@ -130,6 +182,19 @@ module Vanken
         notify
       end
 
+      def open_ring
+        return unless @options.is_a?(Hash)
+        path = @options["ring_path"].to_s
+        return if path.empty?
+
+        values = %w[max_bytes interval file_count].to_h do |key|
+          value = @options["ring_#{key}"]
+          [key.to_sym, value && value != 0 ? value : nil]
+        end.compact
+        raise Error, "ring saving requires a size or time rotation condition" unless values[:max_bytes] || values[:interval]
+        Gateway::FileWriter.open(File.expand_path(path), format: :pcapng, **values)
+      end
+
       def consume_control(io)
         while (line = io.gets(Capture::ControlProtocol::MAX_LINE_BYTES + 1))
           event = Capture::ControlProtocol.parse(line)

data/lib/vanken/app/document.rb

@@ -9,27 +9,36 @@ require_relative "../capture/receiver"
 require_relative "../capture/analyzer"
 require_relative "../capture/analyzer_process"
 require_relative "document_jobs"
+require_relative "../config/analysis_settings"
+require_relative "document_analysis"
+require_relative "navigation"
+require_relative "../config/columns"
 
 module Vanken
   module App
     class Document
       include DocumentJobs
+      include DocumentAnalysis
+      include Navigation
       attr_reader :store, :columns, :annotations, :error, :path, :source, :progress, :filter,
-                  :marked, :ignored, :time_references, :catalog, :analysis_options
+                  :marked, :ignored, :time_references, :catalog, :analysis_options, :analysis_gateway_options
       attr_accessor :frame_latency, :scanner
 
-      def initialize(on_update: nil, preferences: nil, store: nil, scanner: nil, process_analysis: true)
+      def initialize(on_update: nil, preferences: nil, store: nil, scanner: nil, process_analysis: true, decode_as: nil, plugins: nil)
+        @verify_checksums = preferences&.get("analysis.verify_checksums") || false
+        settings = preferences && Config::AnalysisSettings.new(directory: preferences.directory)
+        @analysis_gateway_options = {verify_checksums: @verify_checksums, decode_as: decode_as || settings&.decode_as || [], plugins: plugins || settings&.plugins || []}.freeze
+        @dissector = Gateway::Dissector.new(**@analysis_gateway_options)
         @store = store || Core::FrameStore.new
         @columns = Core::ColumnStore.new
         @annotations = Core::AnnotationStore.new(@store.directory)
         @cache = Core::RowCache.new(limit: preferences&.get("packet_list.row_cache_rows") || 20_000)
-        @verify_checksums = preferences&.get("analysis.verify_checksums") || false
-        @dissector = Gateway::Dissector.new(verify_checksums: @verify_checksums)
         @process_analysis = process_analysis
         @analysis_options = {max_state_bytes: (preferences&.get("analysis.max_state_mib") || 256) << 20,
                              max_flows: preferences&.get("analysis.max_flows") || 100_000}.freeze
-        @catalog = Gateway::FieldCatalog.new
+        @catalog = Gateway::FieldCatalog.new(registry: @dissector.registry)
         @mutex, @condition = Mutex.new, ConditionVariable.new
+        @close_mutex = Mutex.new
         @jobs, @count, @generation = [], 0, 0
         @display = nil
         @marked, @ignored, @time_references = Set.new, Set.new, Set.new
@@ -38,15 +47,7 @@ module Vanken
         @scanner = scanner
         @scan_concurrency = preferences&.get("analysis.workers") || 4
         @filter_tasks = []
-        @details = File.open(File.join(@store.directory, "details.jsonl"), "w+b", 0o600)
-        @detail_reader = File.open(File.join(@store.directory, "details.jsonl"), "rb")
-        @detail_index = File.open(File.join(@store.directory, "reassembled.idx"), "w+b", 0o600)
-        @detail_offsets = {}
-        @summaries = File.open(File.join(@store.directory, "summaries.bin"), "w+b", 0o600)
-        @summary_index = File.open(File.join(@store.directory, "summaries.idx"), "w+b", 0o600)
-        @summary_reader = File.open(File.join(@store.directory, "summaries.bin"), "rb")
-        @summary_index_reader = File.open(File.join(@store.directory, "summaries.idx"), "rb")
-        @summary_count = 0
+        open_analysis_files
         @frame_latency = 0.0
       end
 
@@ -75,12 +76,18 @@ module Vanken
       def displayed_count = @mutex.synchronize { @display ? @display.size : @count }
       def number_at(index) = @mutex.synchronize { @display ? @display.fetch(index) : (index + 1).tap { |number| raise IndexError unless number.between?(1, @count) } }
       def display_numbers = @mutex.synchronize { @display ? @display.dup : (1..@count).to_a }
-      def dirty? = @dirty
+      def dirty? = @dirty || (@source == :live && @store.count > (@saved_count || 0))
       def loading? = !@analyzed && !@cancelled
       def cancelled? = @cancelled
       def received? = @received
       def complete? = @analyzed
       def closing? = !!@closing
+      def analysis_stopped? = closing? || !!@reanalysis_cancelled
+      def capture_stats
+        value = @reader.respond_to?(:stats) && @reader.stats
+        value.respond_to?(:to_h) ? value.to_h : {}
+      end
+      def expert_summary = @mutex.synchronize { {count: @annotations.expert_count, severity: @annotations.expert_max} }
       def stream_frames(stream) = @mutex.synchronize { @annotations.streams[stream].dup }
       def packet(number) = @dissector.dissect(@store.read(number))
       # @rbs (Integer number) -> Gateway::PacketSnapshot?
@@ -93,7 +100,33 @@ module Vanken
         base = @mutex.synchronize { @columns[number] }
         frame = @store.metadata(number)
         info = @cache.fetch(number) { persisted_summary(number) || persisted_details(number)&.fetch("info", nil) || packet(number).info }
-        base.merge(no: number, number: number, timestamp_ns: frame[:timestamp_ns], length: frame[:original_length], info: info)
+        base.merge(no: number, number: number, timestamp_ns: frame[:timestamp_ns], length: frame[:original_length], info: info).merge(custom_row(number))
+      end
+      def custom_columns = @custom_columns || []
+      def custom_columns=(columns)
+        raise Vanken::ConfigError, "invalid custom columns" unless columns.is_a?(Array) && columns.size <= 64 && columns.all? do |item|
+          item.is_a?(Hash) && Config::Columns.valid_field?(item["field"]) && item["key"] == "field:#{item['field']}" && [true, false].include?(item["visible"])
+        end
+        saved = columns.map { |item| item.slice("key", "field", "visible").transform_values { |value| value.is_a?(String) ? value.dup.freeze : value }.freeze }.freeze
+        @mutex.synchronize do
+          unless saved == @custom_columns
+            @custom_columns = saved
+            @sort_generation = (@sort_generation || 0) + 1
+          end
+        end
+      end
+      def custom_row(number)
+        fields = custom_columns.select { |item| item["visible"] }
+        return {} if fields.empty?
+        packet = view(number)
+        resolver = Core::DisplayFilter::FieldResolver.new(@catalog)
+        fields.to_h do |item|
+          field = item.fetch("field")
+          [item.fetch("key").to_sym, Config::Columns.display(packet.values(field), type: resolver.resolve(field).type)]
+        end
+      end
+      def custom_sort_value(number, field)
+        Config::Columns.sort_value(view(number).values(field).first, type: Core::DisplayFilter::FieldResolver.new(@catalog).resolve(field).type)
       end
       def details(number)
         saved = persisted_details(number)
@@ -112,7 +145,7 @@ module Vanken
             program, generation, snapshot = @mutex.synchronize { [@filter, @generation, @filter_context] }
             matched = !program || program.match?(view(number, packet: packet, snapshot: snapshot))
             committed = @mutex.synchronize do
-              next false if generation != @generation
+              next false if generation != @generation || !snapshot.equal?(@filter_context)
               committing = true
               @columns.append(columns)
               @annotations.append(number, annotation)
@@ -123,8 +156,8 @@ module Vanken
                 @detail_index.write([number, @detail_offsets[number]].pack("Q<Q<"))
                 @detail_index.flush
               end
+              record_live_predecessors(number, number, matched ? [number] : [], context: snapshot)
               @count = number
-              @dirty = true if @source == :live
               @display << number if @display && matched
               true
             end
@@ -142,6 +175,7 @@ module Vanken
           @columns.append(source: "", destination: "", protocol: "Malformed", src_port: -1, dst_port: -1, ip_proto: -1, layers: [])
           @annotations.append(number, tcp_stream: -1, seq_rel: -1, ack_rel: -1, analysis_flags: [], expert_max: 3,
             expert_items: [{severity: :error, code: "vanken.internal_error", protocol: "vanken", message: error.message}], extra: {})
+          record_live_predecessors(number, number, [])
           @count = number
           @display << number if @display && !@filter
         end
@@ -149,14 +183,20 @@ module Vanken
       end
       # Only immutable context for this unpublished range crosses the worker pipe.
       def analysis_configuration(first, last)
+        raise Vanken::Error, "invalid analysis range" unless first.is_a?(Integer) && last.is_a?(Integer) && first.positive? && (last - first + 1).between?(1, 256)
         @mutex.synchronize do
           historical = !!@filter_context
           current = @filter_context || {marked: @marked, ignored: @ignored, references: @time_references,
             predecessors: nil, limit: @count, last_displayed: @display ? @display.last : @count}
-          snapshot = current.merge(marked: current[:marked].select { |n| n.between?(first, last) }.to_set,
+          predecessors = if current[:live_predecessors]
+            (first..last).to_h { |number| [number, displayed_predecessor(number, current)] }
+          else
+            current[:predecessors]&.slice(*(first..last).to_a)
+          end
+          snapshot = current.except(:live_predecessors, :live_predecessor_start).merge(marked: current[:marked].select { |n| n.between?(first, last) }.to_set,
             ignored: current[:ignored].select { |n| n.between?(first, last) }.to_set,
-            references: current[:references].dup, predecessors: nil,
-            unfiltered_predecessor: historical && current[:predecessors].nil?)
+            references: current[:references].dup, predecessors: predecessors,
+            unfiltered_predecessor: historical && current[:predecessors].nil? && !current[:live_predecessors])
           {expression: @filter&.expression || "", generation: @generation, snapshot: snapshot,
            historical_context: historical, context_token: @filter_context&.object_id}
         end
@@ -185,8 +225,8 @@ module Vanken
             @summary_index.write(offsets)
             [@annotations, @details, @detail_index, @summaries, @summary_index].each(&:flush)
             @catalog.merge(batch[:fields])
+            record_live_predecessors(first, last, batch[:matches])
             @summary_count = @count = last
-            @dirty = true if @source == :live
             @display.concat(batch[:matches]) if @display
             true
           end
@@ -199,7 +239,18 @@ module Vanken
         notify
       end
       def receiving_done = (@received = true; signal)
-      def analyzing_done = (@annotations.flush; @analyzed = true; notify(force: true))
+      def analyzing_done
+        @annotations.flush
+        @mutex.synchronize do
+          @filter_context = nil if @reanalysis_context && @filter_context.equal?(@reanalysis_context)
+          @reanalysis_context = nil
+          @rebuilding_filter_basis = nil
+          @rebuilding = false
+          @reanalysis_cancelled = false
+          @analyzed = true
+        end
+        notify(force: true)
+      end
       def signal = @mutex.synchronize { @condition.broadcast }
       def wait_for_frames = @mutex.synchronize { @condition.wait(@mutex, 0.05) }
       def fail(error) = (@error = error; notify(force: true))
@@ -227,6 +278,15 @@ module Vanken
         signal
         self
       end
+      def cancel_reanalysis
+        @mutex.synchronize do
+          if @rebuilding
+            @reanalysis_cancelled = true
+            @condition.broadcast
+          end
+        end
+        self
+      end
       def wait(timeout = nil)
         deadline = timeout && (Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout)
         @jobs.each do |job|
@@ -237,22 +297,25 @@ module Vanken
         self
       end
       def close
-        return self if @closed
-        @closing = true
-        cancel
-        cancel_scan
-        wait(3)
-        @annotations.close
-        @details.close
-        @detail_reader.close
-        @detail_index.close
-        @summaries.close
-        @summary_index.close
-        @summary_reader.close
-        @summary_index_reader.close
-        @store.close
-        @closed = true
-        self
+        @close_mutex.synchronize do
+          return self if @closed
+          @closing = true
+          cancel_search
+          cancel
+          cancel_scan
+          wait(3)
+          @annotations.close
+          @details.close
+          @detail_reader.close
+          @detail_index.close
+          @summaries.close
+          @summary_index.close
+          @summary_reader.close
+          @summary_index_reader.close
+          @store.close
+          @closed = true
+          self
+        end
       end
 
       private