@appthreat/atom-parsetools 1.3.0 → 1.4.0

package/README.md

@@ -4,7 +4,7 @@ This package hosts a collection of parsing tools that complement the `@appthreat
 
 - astgen - Generates AST for JavaScript and TypeScript projects in JSON format
 - phpastgen - Generates AST for PHP projects using `php-parse` command from `nikic/php-parser`
-- rbastgen - Generates AST for Ruby projects using the AppThreat's `ruby_ast_gen` gem
+- rbastgen - Generates AST for Ruby projects using AppThreat's [`ruby_ast_gen`](https://github.com/AppThreat/ruby_ast_gen) gem (2.0.1)
 - scalasem - Generates a custom semantics slice for Scala Projects by utilising scalac command.
 
 ## Runtime support
@@ -31,13 +31,13 @@ Options:
 
 #### Environment variables
 
-| Variable                    | Default   | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
-| --------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `ASTGEN_TYPE_WORKERS`       | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
-| `ASTGEN_INCLUDE_TEST_FILES` | `false`   | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation.                                                                                                                                                                                                                                                                                                                              |
-| `ASTGEN_CONCURRENCY`        | `10`      | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
-| `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false` | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely.                                                                                                                                                                                                                                                       |
-| `ASTGEN_IGNORE_DIRS`        | unset     | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`).                                                                                                                                                                                                        |
+| Variable                              | Default   | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| ------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ASTGEN_TYPE_WORKERS`                 | `1` (off) | Number of worker threads for the TypeScript type-generation phase, or `auto` to derive it from the available CPUs. The TypeScript checker is single-threaded, so parallelism comes from sharding files across workers, each building its own program. **Opt-in:** sharding changes TypeScript's internal type-id ordering, which reorders the members of a small number of inferred union types (e.g. `A \| B` → `B \| A`; semantically identical). Leave unset for byte-identical output; set it (e.g. `auto` or `8`) to trade that cosmetic reordering for a large speedup on big projects. |
+| `ASTGEN_INCLUDE_TEST_FILES`           | `false`   | When `true`, do not exclude test files (`*.poku.js`, `*.test.*`, `*.spec.*`, `*.e2e.*`, `__tests__/`, `__mocks__/`) from AST and type generation. They are excluded by default because they are typically the heaviest, lowest-value inputs for type generation.                                                                                                                                                                                                                                                                                                                              |
+| `ASTGEN_CONCURRENCY`                  | `10`      | Chunk size for the in-thread file loop (bounds peak memory between `gc()` passes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES` | `false`   | When `true`, also parse bundled entrypoints inside `node_modules` (files matching `*.(bundle\|dist\|index\|min\|app).(js\|cjs\|mjs)`). Off by default; `node_modules` is otherwise skipped entirely.                                                                                                                                                                                                                                                                                                                                                                                          |
+| `ASTGEN_IGNORE_DIRS`                  | unset     | Comma/space-separated list of directories to ignore. As a side effect, when it is set and does **not** contain `node_modules`, the `node_modules` bundle entrypoints above are included (equivalent to `ASTGEN_INCLUDE_NODE_MODULES_BUNDLES=true`).                                                                                                                                                                                                                                                                                                                                           |
 
 ### phpastgen
 
@@ -64,17 +64,90 @@ Operations is a list of the following options (--dump by default):
 
 ### rbastgen
 
+Requires Ruby 3.4.x or 4.0.x on the `PATH`, or `ATOM_RUBY_HOME` pointing at an install. The gem and
+its pure-Ruby dependencies are bundled under `plugins/rubyastgen`, so nothing needs to be
+gem-installed, and one build of this package runs under every supported Ruby: the bundle is exposed
+to the interpreter through `GEM_PATH` rather than through bundler's standalone loader, which
+resolved its paths from the ABI of the Ruby that built it.
+
+`prism` and `racc` are deliberately **not** bundled. Both carry C extensions, which are built per
+ABI and per platform, and both are default gems in every supported Ruby, so the runtime's own copies
+are used. One consequence is visible: the newest grammar available follows the interpreter's prism,
+so Ruby 3.4 tops out lower than Ruby 4.0 (grammar 3.5 versus 4.1 at the time of writing). Installing
+a newer `prism` gem on the machine raises it, since the caller's `GEM_PATH` is preserved.
+
 ```text
 node rbastgen.js --help
 Usage:
-  -i, --input      The input file or directory (required)
-  -o, --output     The output directory (default: '.ast')
-  -e, --exclude    The exclusion regex (default: '^(tests?|vendor|spec)')
-  -d, --debug      Enable debug logging
-      --version    Print the version
-      --help       Print usage
+  -i, --input <path>       The input file or directory (required)
+  -o, --output <dir>       The output directory (default: '.ast')
+  -e, --exclude <regex>    The exclusion regex (default: '^(tests?|vendor|spec)')
+  -l, --log <level>        The logging level: debug, info, warn or error (default: info)
+  -d, --debug              Enable debug logging (same as --log debug)
+      --parser-target <x.y>
+                           Parse with a specific Ruby grammar (e.g. 3.4, 4.0) instead
+                           of the newest available
+      --max-depth <n>      Maximum AST depth before truncation (default: 250)
+      --threads <n>        Worker threads for a directory run (default: 10)
+      --fail-on-error      Exit non-zero when any input file failed to parse
+      --parser-info        Print parser/runtime capability information
+      --version            Print the version
+      --help               Print usage
+```
+
+Problems with individual files are reported and skipped, never fatal; only usage errors (a missing
+`-i`, an unusable `--exclude` regex, an invalid `--parser-target`) exit non-zero. Note that
+`rbastgen` does not propagate the generator's exit status, so `--fail-on-error` is reported in the
+log but the wrapper still exits 0 — call the gem directly if a CI job needs to fail on a parse
+error.
+
+#### What the bundled generator emits (ruby_ast_gen 2.0.1)
+
+- **Parsing is decoupled from the running Ruby.** When `prism` is available the newest grammar its
+  translation layer supports is used, so a 3.4 runtime parses Ruby 4.0/4.1 syntax. `--parser-target
+x.y` pins a grammar instead (down to 1.8 through the `parser` gem), and a file that fails under
+  the selected grammar is retried once with the newest one. Every JSON file records the backend
+  that produced it in `parser_backend`, alongside `generator_version` and `ruby_version`.
+- **Ruby DSL files are discovered, not just `.rb`.** `Rakefile`, `Gemfile`, `Capfile`,
+  `Vagrantfile`, `Fastfile` and friends are matched by basename, plus the `.gemspec`, `.rake`,
+  `.ru`, `.rbi`, `.thor`, `.jbuilder`, `.axlsx` and `.rabl` extensions. Vendor and tool
+  directories (`.git`, `.bundle`, `.venv`, …) are skipped.
+- **Non-UTF-8 sources survive.** `# coding:` magic comments are honoured and undecodable bytes are
+  scrubbed rather than dropping the file, marked with `encoding_scrubbed: true`.
+- **Deeply nested source is truncated, not dropped**, with `truncated: true` on the boundary node
+  and a `truncated_nodes` count at the top level (`--max-depth`).
+- **Semantic metadata for consumers**: a `magic_comments` array (Sorbet `typed:` levels,
+  `frozen_string_literal`, …), `call_operator`/`has_parentheses` on calls, `percent_array` and
+  regexp `options`, heredoc body offsets, and `has_sig: true` on a `def` preceded by a Sorbet `sig`
+  block.
+- **Two side-records per run, written inside the output directory**, both ending in `.jsonl` so a
+  consumer globbing `*.json` for ASTs never mistakes them for one:
+  `ruby_ast_gen_manifest.jsonl` (one object: input/output, backend, counts of parsed, failed,
+  skipped, excluded and truncated files, threads, max depth) and
+  `ruby_ast_gen_diagnostics.jsonl` (one object per failed file with message, line, column), the
+  latter written only when something failed and removed when a later run is clean.
+
+To check which backend a machine will use:
+
+```shell
+rbastgen --parser-info
 ```
 
+`Parser gem` and `Prism gem` name the versions actually loaded — the vendored `parser` and the
+runtime's `prism` — so they are the quickest way to tell which copy of each library a machine is
+parsing with. They read `unavailable` only when a library genuinely is not loaded.
+
+#### Environment variables
+
+| Variable                          | Default                | Purpose                                                                                                                                      |
+| --------------------------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ATOM_RUBY_HOME`                  | unset                  | Ruby install directory to use when a suitable `ruby` is not on the `PATH`; its `bin` is prepended to `PATH` for the child process.           |
+| `RUBY_CMD`                        | `ruby`                 | Ruby interpreter to invoke. Set this (or `ATOM_RUBY_HOME`) when the detected version is not 3.4.x/4.0.x.                                     |
+| `RUBY_ASTGEN_BIN`                 | bundled `ruby_ast_gen` | The generator script to run. Point it at a checkout's `exe/ruby_ast_gen` to test an unreleased `ruby_ast_gen` without touching this package. |
+| `ATOM_CWD`                        | `process.cwd()`        | Working directory for the generator, which is what relative `-i`/`-o` paths resolve against.                                                 |
+| `ATOM_TIMEOUT` / `ASTGEN_TIMEOUT` | unset (no timeout)     | Milliseconds before the generator process is killed. `ATOM_TIMEOUT` wins; a non-numeric value is ignored. Also honoured by `scalasem`.       |
+| `GEM_PATH`                        | unset                  | Preserved and appended to the vendored bundle, so gems installed on the machine (a newer `prism`, for instance) stay reachable.              |
+
 ### scalasem
 
 ```text
@@ -87,6 +160,30 @@ Example:
 scalasem $(pwd) slices.json
 ```
 
+## Testing
+
+`npm test` covers the JavaScript/TypeScript side and needs nothing but Node. The Ruby workflow tests
+run the `rbastgen` command end to end against `test-fixtures/projects/ruby-parsing`, so they need a
+built plugin bundle and a supported Ruby, and they skip themselves cleanly when either is missing:
+
+```shell
+bash build.sh --ruby-only
+npm run test:ruby
+```
+
+`ci/verify-packed-tarball.sh` is the release check: it packs the tarball, installs it into a scratch
+project and parses the fixture with the installed copy, asserting along the way that the Ruby bundle
+is present and free of compiled extensions. CI runs it, and it runs the same way locally.
+
+They assert what a consumer depends on rather than that the command ran: which files are discovered
+(including `Gemfile` and `Rakefile`, matched by basename), that `has_sig`, `magic_comments`,
+`percent_array` and regexp `options` are emitted, that the manifest's counts reconcile, that a
+failed file lands in the diagnostics record while the run still exits 0, that a clean run leaves no
+stale record behind, and that the bundle contains no compiled extension. CI runs them under both
+Ruby 3.4 and Ruby 4.0 against a bundle built with 3.4, and separately installs the packed tarball
+and parses with that, which is the check that catches a bundle usable only on the Ruby that built
+it.
+
 ## License
 
 MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@appthreat/atom-parsetools",
-  "version": "1.3.0",
+  "version": "1.4.0",
   "description": "Parsing tools that complement the @appthreat/atom project.",
   "main": "./index.js",
   "type": "module",
@@ -8,6 +8,7 @@
     "pretty": "prettier --write *.js --trailing-comma=none",
     "test": "node test-fixtures/astgen-type-regression.js && node test-fixtures/astgen-json-regression.js && node test-fixtures/astgen-vue-regression.js && node test-fixtures/astgen-shape-snapshot.js && node test-fixtures/evaluate-astgen.js",
     "test:evaluate": "node test-fixtures/evaluate-astgen.js",
+    "test:ruby": "node test-fixtures/rbastgen-regression.js",
     "test:shape": "node test-fixtures/astgen-shape-snapshot.js",
     "test:shape:update": "UPDATE_SHAPE_SNAPSHOT=1 node test-fixtures/astgen-shape-snapshot.js",
     "test:json": "node test-fixtures/astgen-json-regression.js",

package/plugins/composer/installed.php

@@ -1,9 +1,9 @@
 <?php return array(
     'root' => array(
         'name' => '__root__',
-        'pretty_version' => 'v1.3.0',
-        'version' => '1.3.0.0',
-        'reference' => '65d96106e47fe9607cec2ba1be69da7c9c4d574c',
+        'pretty_version' => 'v1.4.0',
+        'version' => '1.4.0.0',
+        'reference' => 'f0a5257f9414e759bc4a433b007003ebb6d530cb',
         'type' => 'library',
         'install_path' => __DIR__ . '/../../',
         'aliases' => array(),
@@ -11,9 +11,9 @@
     ),
     'versions' => array(
         '__root__' => array(
-            'pretty_version' => 'v1.3.0',
-            'version' => '1.3.0.0',
-            'reference' => '65d96106e47fe9607cec2ba1be69da7c9c4d574c',
+            'pretty_version' => 'v1.4.0',
+            'version' => '1.4.0.0',
+            'reference' => 'f0a5257f9414e759bc4a433b007003ebb6d530cb',
             'type' => 'library',
             'install_path' => __DIR__ . '/../../',
             'aliases' => array(),

package/plugins/rubyastgen/Gemfile

@@ -2,10 +2,12 @@
 
 source "https://rubygems.org"
 
+# Only pure-Ruby gems are vendored: rbastgen puts these on the load path directly, so the bundle
+# works under every supported Ruby ABI. prism and racc are deliberately absent — both are default
+# gems in Ruby 3.4+ and 4.0+, and their C extensions are built per ABI, which is what locked the
+# published bundle to the single Ruby version it was built with.
 group :frontend do
-    gem "ruby_ast_gen", github: 'appthreat/ruby_ast_gen', ref: 'cc312ebf06b4b5b1a8ef4c3c9c71ca98eb9fcd01'
-    gem "parser", "~> 3.3.11.1"
-    gem "prism", "~> 1.9"
+    gem "ruby_ast_gen", github: 'appthreat/ruby_ast_gen', ref: '4ac84327b3af4a8003bb9ece2566df6471cdd851'
+    gem "parser", "~> 3.3.12.0"
     gem "ostruct", "~> 0.6.3"
-    gem "racc", "~> 1.8.1"
 end

package/plugins/rubyastgen/Gemfile.lock

@@ -1,19 +1,18 @@
 GIT
   remote: https://github.com/appthreat/ruby_ast_gen.git
-  revision: cc312ebf06b4b5b1a8ef4c3c9c71ca98eb9fcd01
-  ref: cc312ebf06b4b5b1a8ef4c3c9c71ca98eb9fcd01
+  revision: 4ac84327b3af4a8003bb9ece2566df6471cdd851
+  ref: 4ac84327b3af4a8003bb9ece2566df6471cdd851
   specs:
-    ruby_ast_gen (1.3.0)
+    ruby_ast_gen (2.0.1)
 
 GEM
   remote: https://rubygems.org/
   specs:
     ast (2.4.3)
     ostruct (0.6.3)
-    parser (3.3.11.1)
+    parser (3.3.12.0)
       ast (~> 2.4.1)
       racc
-    prism (1.9.0)
     racc (1.8.1)
 
 PLATFORMS
@@ -22,9 +21,7 @@ PLATFORMS
 
 DEPENDENCIES
   ostruct (~> 0.6.3)
-  parser (~> 3.3.11.1)
-  prism (~> 1.9)
-  racc (~> 1.8.1)
+  parser (~> 3.3.12.0)
   ruby_ast_gen!
 
 BUNDLED WITH

package/plugins/rubyastgen/bundle/ruby/4.0.0/bundler/gems/ruby_ast_gen-4ac84327b3af/Gemfile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+gemspec
+
+gem "ostruct", "~> 0.6.3"
+gem "parser", "~> 3.3.12.0"
+gem "prism", "~> 1.9"
+gem "rake", "~> 13.4.2"
+gem "rspec", "~> 3.13.2"

package/plugins/rubyastgen/bundle/ruby/4.0.0/bundler/gems/ruby_ast_gen-4ac84327b3af/exe/ruby_ast_gen

@@ -0,0 +1,55 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# require "bundler/setup"
+# We do the below instead of the above, as the above can be flaky
+libs = File.expand_path("../vendor/bundle/ruby/*/gems/**/lib", __dir__)
+$LOAD_PATH.unshift(*Dir.glob(libs))
+
+require_relative "../lib/ruby_ast_gen"
+
+# Argument parsing and the usage text both come from RubyAstGen::CLI::FLAGS.
+result = RubyAstGen::CLI.parse(ARGV)
+options = result.options
+
+# Configure logging before emitting anything, so -l/--log applies to the argument warnings too.
+RubyAstGen::Logger.level = :debug if options[:debug]
+RubyAstGen::Logger.level = options[:log_level] if options[:log_level]
+result.warnings.each { |warning| RubyAstGen::Logger.warn warning }
+
+# Informational modes run after the whole argv is read, so `--parser-info` reflects a
+# `--parser-target` passed in any order.
+case result.mode
+when :parser_info
+  puts RubyAstGen.parser_info_text(parser_target: options[:parser_target])
+  exit
+when :version
+  puts RubyAstGen::VERSION
+  exit
+when :help
+  puts RubyAstGen::CLI.help_text
+  exit
+end
+
+# Usage errors are the only fatal cases: everything a run encounters per file is reported and
+# skipped instead, because a non-zero exit costs the caller every file (see the comment in
+# RubyAstGen::CLI.parse). --fail-on-error is the sole opt-out, for CI callers who would rather
+# fail than analyse a partial corpus; chen never passes it.
+begin
+  summary = RubyAstGen.parse(options)
+  if options[:fail_on_error] && summary[:files_failed] > 0
+    RubyAstGen::Logger.error "#{summary[:files_failed]} file(s) failed to parse; details in " \
+      "#{File.join(options[:output].to_s, RubyAstGen::DIAGNOSTICS_FILENAME)}"
+    exit 1
+  end
+rescue ArgumentError => e
+  RubyAstGen::Logger.error e.message
+  exit 1
+rescue StandardError => e
+  # Anything unexpected (an unwritable output directory, for example) still exits with a single
+  # readable line: the caller only sees our stdout, and a raw backtrace tells it nothing about
+  # which input it was invoked with. The backtrace is available with -d.
+  RubyAstGen::Logger.error "#{e.class}: #{e.message}"
+  RubyAstGen::Logger.debug e.backtrace.join("\n")
+  exit 1
+end

package/plugins/rubyastgen/bundle/ruby/4.0.0/bundler/gems/ruby_ast_gen-4ac84327b3af/lib/ruby_ast_gen/cli.rb

@@ -0,0 +1,133 @@
+# frozen_string_literal: true
+
+module RubyAstGen
+  # Command-line surface, kept in the library rather than in `exe/ruby_ast_gen` so it can be
+  # tested without shelling out.
+  #
+  # FLAGS is the single source of truth: it drives argument parsing *and* renders `--help`, so
+  # the two cannot drift. They used to be maintained separately, and the README documented
+  # `-l/--log` for a release in which the argv loop rejected it with a non-zero exit.
+  module CLI
+
+    Flag = Struct.new(:short, :long, :kind, :key, :placeholder, :description, keyword_init: true) do
+      def names
+        [short, long].compact
+      end
+
+      # "-i, --input <path>" / "    --parser-info"
+      def usage
+        head = short ? "#{short}, #{long}" : "    #{long}"
+        placeholder ? "#{head} #{placeholder}" : head
+      end
+    end
+
+    # Worker count for directory runs. Exposed because the ceiling is memory, not CPU: a JRuby
+    # container or a repo of very large files may need fewer (plan 03 §7).
+    DEFAULT_THREADS = 10
+
+    FLAGS = [
+      Flag.new(short: "-i", long: "--input", kind: :value, key: :input, placeholder: "<path>",
+               description: "The input file or directory (required)"),
+      Flag.new(short: "-o", long: "--output", kind: :value, key: :output, placeholder: "<dir>",
+               description: "The output directory (default: '.ast')"),
+      Flag.new(short: "-e", long: "--exclude", kind: :value, key: :exclude, placeholder: "<regex>",
+               description: "The exclusion regex (default: '^(tests?|vendor|spec)')"),
+      Flag.new(short: "-l", long: "--log", kind: :value, key: :log_level, placeholder: "<level>",
+               description: "The logging level: debug, info, warn or error (default: info)"),
+      Flag.new(short: "-d", long: "--debug", kind: :boolean, key: :debug,
+               description: "Enable debug logging (same as --log debug)"),
+      Flag.new(long: "--parser-target", kind: :value, key: :parser_target, placeholder: "<x.y>",
+               description: "Parse with a specific Ruby grammar (e.g. 3.4, 4.0) instead\nof the newest available"),
+      Flag.new(long: "--max-depth", kind: :value, key: :max_depth, placeholder: "<n>",
+               description: "Maximum AST depth before truncation " \
+                            "(default: #{NodeHandling::MAX_NESTING_DEPTH})"),
+      Flag.new(long: "--threads", kind: :value, key: :threads, placeholder: "<n>",
+               description: "Worker threads for a directory run " \
+                            "(default: #{DEFAULT_THREADS})"),
+      Flag.new(long: "--fail-on-error", kind: :boolean, key: :fail_on_error,
+               description: "Exit non-zero when any input file failed to parse"),
+      Flag.new(long: "--parser-info", kind: :mode, key: :parser_info,
+               description: "Print parser/runtime capability information"),
+      Flag.new(long: "--version", kind: :mode, key: :version,
+               description: "Print the version"),
+      Flag.new(long: "--help", kind: :mode, key: :help,
+               description: "Print usage")
+    ].freeze
+
+    BY_NAME = FLAGS.flat_map { |flag| flag.names.map { |name| [name, flag] } }.to_h.freeze
+
+    DEFAULTS = {
+      input: nil,
+      output: ".ast",
+      exclude: "^(tests?|vendor|spec)",
+      debug: false,
+      log_level: nil,
+      parser_target: nil,
+      max_depth: nil,
+      threads: nil,
+      fail_on_error: false
+    }.freeze
+
+    # Warnings are collected rather than printed so the caller can configure the log level
+    # first: `-l error` would otherwise be unable to silence them, since they are produced
+    # before the level is known.
+    Result = Struct.new(:options, :mode, :warnings, keyword_init: true)
+
+    def self.parse(argv)
+      options = DEFAULTS.dup
+      mode = :parse
+      warnings = []
+
+      index = 0
+      while index < argv.size
+        arg = argv[index]
+        flag = BY_NAME[arg]
+
+        case flag&.kind
+        when :value
+          value = argv[index + 1]
+          # Only a *known flag* counts as a missing value. Testing for a leading "-" would
+          # reject legitimate values such as an exclusion regex of "-vendor".
+          if value.nil? || BY_NAME.key?(value)
+            warnings << "Option #{arg} expects a value; got #{value.inspect}. Ignoring it."
+          else
+            options[flag.key] = value
+            index += 1
+          end
+        when :boolean
+          options[flag.key] = true
+        when :mode
+          mode = flag.key
+        else
+          # chen maps any non-zero exit to a failed command -> DefaultAstGenRunnerResult() ->
+          # zero parsed files, i.e. an empty CPG (chen consumer plan 04 §10.1). An unrecognized flag must not
+          # cost the user their whole atom, so warn and continue.
+          warnings << "Unknown option: #{arg} (ignoring)"
+        end
+
+        index += 1
+      end
+
+      Result.new(options: options, mode: mode, warnings: warnings)
+    end
+
+    DESCRIPTION_COLUMN = 25
+
+    def self.help_text
+      lines = ["Usage:"]
+      FLAGS.each do |flag|
+        usage = flag.usage
+        head, *rest = flag.description.split("\n")
+        if usage.length < DESCRIPTION_COLUMN
+          lines << "  #{usage.ljust(DESCRIPTION_COLUMN)}#{head}"
+        else
+          # No room on the flag's own line; start the description underneath it.
+          lines << "  #{usage}"
+          lines << "  #{' ' * DESCRIPTION_COLUMN}#{head}"
+        end
+        rest.each { |line| lines << "  #{' ' * DESCRIPTION_COLUMN}#{line}" }
+      end
+      lines.join("\n")
+    end
+  end
+end