sas-linter 0.1.0

data/README.md

@@ -0,0 +1,140 @@
+# sas-linter
+
+A configurable lint engine for SAS source files. Built on the [`sas-lexer`](https://github.com/mes-amis/sas-lexer-rb) gem (a Ruby FFI binding to Misha Perlov's Rust [`sas-lexer`](https://github.com/mishamsk/sas-lexer)) and ships with eleven pluggable rules covering structural defects, cosmetic issues, and source-header conventions.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "sas-linter"
+```
+
+Or install directly:
+
+```sh
+gem install sas-linter
+```
+
+## CLI usage
+
+```sh
+# Run every rule on a single file
+bin/sas_lint path/to/source.sas
+
+# List all registered rules with their description and autofix capability
+bin/sas_lint --list-rules
+
+# Run only specific rules
+bin/sas_lint --rules malformed_if_condition,identical_if_else_branches src/*.sas
+
+# Use a YAML config (default: config/lint.yaml)
+bin/sas_lint --config my-lint.yaml src/*.sas
+
+# Lint without applying any autofixes the config requested
+bin/sas_lint --no-autofix src/*.sas
+```
+
+Exit codes: `0` clean, `1` findings, `2` invalid args.
+
+## Library usage
+
+```ruby
+require "sas_linter"
+
+linter = SasLinter.new                                    # all registered rules
+linter = SasLinter.new(rules: [:malformed_if_condition])  # subset by rule id
+linter = SasLinter.from_config(YAML.load_file("lint.yaml"))
+
+findings = linter.lint(source_string, path: "demo.sas")
+findings.each { |f| puts f.to_s }   # path:line:col: [rule_id] message
+
+# Lint a file. If any rule has autofix enabled and changed the source,
+# the file is rewritten in place.
+findings = linter.lint_file("path/to/source.sas")
+```
+
+## Built-in rules
+
+| rule id | description |
+|---|---|
+| `unreachable_inner_branch_value` | Outer `if VAR in (S) then do;` guards an inner branch whose comparison values aren't all in `S`. |
+| `identical_if_else_branches` | `if COND then S; else S;` with identical bodies — almost always a copy-paste error. |
+| `commented_out_guard` | SAS line-comment `* if ... then do;` pattern indicating a disabled outer validity guard. |
+| `choose_one_template` | `** CHOOSE ONE OF THE BELOW STATEMENTS;` banner indicating a broken-by-default source. |
+| `trailing_whitespace` | Trailing spaces/tabs at end of line. |
+| `tab_expansion` | Tab characters that should be spaces (configurable width). |
+| `source_headers` | Restore the `**...**;` 90-char header convention to broken sources. |
+| `line_endings` | Mixed or non-CRLF line terminators (configurable target). |
+| `encoding_issues` | Smart-quote / em-dash / Win-1252 byte sequences that confuse downstream tooling. |
+| `malformed_if_condition` | Empty conditions, missing operators, orphan `then`, unbalanced parens, etc. |
+| `missing_assignment_semicolon` | Assignment statements followed by an inline `**` comment but no terminating `;`. |
+| `variable_value_out_of_known_range` | `if VAR = N` / `if VAR in (...)` literals fall outside the variable's documented acceptable values. Loads the catalog from one or more CSVs with configurable column names and column separator (`,`, `;`, tab). |
+
+`bin/sas_lint --list-rules` prints the same set with autofix capability.
+
+## Writing a custom rule
+
+Subclass `SasLinter::Rule`, declare an id, description, and severity, then implement `#check`:
+
+```ruby
+class MyRule < SasLinter::Rule
+  rule_id :my_rule
+  description "Flag occurrences of FOO in DATA steps."
+  severity :warning
+
+  def check(tokens, path:, all_tokens: nil, source: nil)
+    findings = []
+    tokens.each do |t|
+      next unless t[:text] == "FOO"
+      findings << finding(line: t[:start_line], column: t[:start_column],
+                          message: "FOO is forbidden", path: path)
+    end
+    findings
+  end
+end
+```
+
+Subclasses self-register on the rule registry via `rule_id` — once required, they're picked up by `SasLinter.new` (no rule list) and resolvable via `SasLinter::Rule.fetch(:my_rule)`.
+
+To support autofix, override `self.supports_autofix?` to return `true` and implement `#autofix(source)` to return the rewritten source.
+
+## YAML config
+
+```yaml
+rules:
+  malformed_if_condition:
+    enabled: true              # default
+  trailing_whitespace:
+    enabled: true
+    autofix: true
+  encoding_issues:
+    enabled: true
+    use_defaults: true
+    replacements:
+      "—": "--"
+  identical_if_else_branches:
+    enabled: false             # disable a rule
+```
+
+Rules omitted from the config default to enabled with no options, so adding a new rule to the gem won't silently disable it for users with existing configs.
+
+## Testing
+
+```sh
+bundle install
+bundle exec rake spec
+```
+
+## License
+
+[GNU Affero General Public License v3.0 or later](LICENSE) — chosen to match the upstream `sas-lexer` gem (which `sas-linter` requires at runtime). © Mon Ami, Inc.
+
+Practical implications:
+
+- **Internal / personal use** has no obligations beyond preserving notices.
+- **Redistribution** (shipping the gem inside a binary, container image, or product) requires offering the complete corresponding source under AGPL-3.0.
+- **Network use** (running `sas-linter` as a backend that users interact with remotely) triggers the AGPL's source-disclosure clause for those network users.
+- **Combined works** with `sas-linter` must be licensed under AGPL-compatible terms.
+
+If those terms don't fit your use case, run a standalone lint job (CLI / CI step) instead of embedding the linter in a redistributed product.

data/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: :spec

data/bin/sas_lint

@@ -0,0 +1,79 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "sas_linter"
+require "optparse"
+
+options = { config: SasLinter::DEFAULT_CONFIG_PATH, rules: nil, autofix: true }
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: sas_lint FILE [FILE ...] [options]"
+
+  opts.on("--config PATH", "YAML config file (default: #{SasLinter::DEFAULT_CONFIG_PATH}). " \
+                            "Missing file → run with built-in defaults.") do |path|
+    options[:config] = path
+  end
+
+  opts.on("--rules id1,id2,...", Array,
+          "Override config and run only the listed rules with default options.") do |ids|
+    options[:rules] = ids.map(&:to_sym)
+  end
+
+  opts.on("--no-autofix",
+          "Suppress autofix even if the config sets `autofix: true` for some rule. " \
+          "Findings are still reported but no file is rewritten.") do
+    options[:autofix] = false
+  end
+
+  opts.on("--list-rules", "Print every registered rule and exit") do
+    SasLinter::Rule.all.each do |klass|
+      mark = klass.supports_autofix? ? "  [autofix]" : ""
+      puts format("%-40s  %s%s", klass.rule_id, klass.description, mark)
+    end
+    exit 0
+  end
+
+  opts.on("-h", "--help", "Show this help message") do
+    puts opts
+    exit 0
+  end
+end
+
+parser.parse!
+
+if ARGV.empty?
+  warn parser.help
+  exit 2
+end
+
+linter =
+  if options[:rules]
+    SasLinter.new(rules: options[:rules])
+  else
+    config = SasLinter.load_config_file(options[:config])
+    # `--no-autofix` strips every rule's autofix flag from the loaded config
+    # before the linter is built, so a dry run can never rewrite a file.
+    if options[:autofix] == false && config.is_a?(Hash) && config["rules"].is_a?(Hash)
+      config["rules"].each_value do |opts_hash|
+        opts_hash["autofix"] = false if opts_hash.is_a?(Hash) && opts_hash["autofix"]
+      end
+    end
+    SasLinter.from_config(config)
+  end
+
+exit_code = 0
+
+ARGV.each do |path|
+  unless File.file?(path)
+    warn "sas_lint: #{path}: not a regular file"
+    exit_code = 2
+    next
+  end
+
+  findings = linter.lint_file(path)
+  next if findings.empty?
+
+  exit_code = 1
+  findings.each { |f| puts f.to_s }
+end
+
+exit exit_code

data/lib/sas_linter/rules/choose_one_template.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require_relative "../../sas_linter"
+require "sas_lexer"
+
+class SasLinter
+  module Rules
+    # Flag SAS sources that ship with the "CHOOSE ONE OF THE BELOW STATEMENTS"
+    # banner. The banner introduces a block of mutually-exclusive validity
+    # guards (typically `[USE FOR HC OR CHA WITH FS]`, `[USE FOR LTCF]`,
+    # `[USE FOR CHA WITHOUT FS]`, etc.), all commented out, and asks the
+    # downstream consumer to pick one before the source will work.
+    #
+    # Why it's an antipattern:
+    #   - The source is broken-by-default — every consumer must mutate it
+    #     before use.
+    #   - SAS won't error on the dangling `end;` of the (also-commented)
+    #     block, but the algorithm runs unguarded if no variant is picked.
+    #   - The deployment-context decision should belong to a config file or
+    #     a separate per-context source variant, not to a comment-toggle
+    #     buried in the middle of the algorithm.
+    #
+    # Companion rule: `commented_out_guard` flags the individual disabled
+    # guards. This rule flags the banner that introduces them, so we can
+    # find every file that ships in the multi-template state regardless
+    # of whether any variant has already been activated.
+    class ChooseOneTemplate < Rule
+      rule_id :choose_one_template
+      description "Source ships with a 'CHOOSE ONE OF THE BELOW STATEMENTS' " \
+                  "banner — broken-by-default; consumers must mutate the " \
+                  "source to pick a deployment-context guard."
+      severity :warning
+
+      TT = SasLexer::Lexer::TokenType
+      TC = SasLexer::Lexer::TokenChannel
+
+      BANNER = /CHOOSE\s+ONE\s+OF\s+THE\s+BELOW\s+STATEMENTS/i
+
+      def check(_tokens, path:, all_tokens: nil, source: nil) # rubocop:disable Lint/UnusedMethodArgument
+        return [] unless all_tokens
+
+        all_tokens.filter_map do |tok|
+          next unless tok[:channel] == TC::COMMENT
+          next unless tok[:type] == TT::COMMENT_STAT
+          next unless tok[:text] =~ BANNER
+
+          finding(
+            line: tok[:start_line],
+            column: tok[:start_column] + 1,
+            message: "'CHOOSE ONE OF THE BELOW STATEMENTS' banner — source is " \
+                     "broken-by-default; the alternative validity guards below " \
+                     "are all commented out so every consumer must edit this " \
+                     "file. Pick one variant, delete the others, and remove " \
+                     "this banner.",
+            path: path
+          )
+        end
+      end
+    end
+  end
+end

data/lib/sas_linter/rules/commented_out_guard.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require_relative "../../sas_linter"
+require "sas_lexer"
+
+class SasLinter
+  module Rules
+    # Flag SAS line-comments (`* ... ;`) whose body looks like a disabled
+    # validity guard — specifically, the body contains both an `IF` and a
+    # `THEN DO` (case-insensitive).
+    #
+    # Motivating shape: a source's outer `if ... then do;` validity guard
+    # is commented out by a leading `*`, leaving an orphan `end;` further
+    # down. The body then runs unguarded for inputs the guard would have
+    # rejected. Worth a human review on each finding — either the guard
+    # should be live, or the orphan `end;` should be removed.
+    class CommentedOutGuard < Rule
+      rule_id :commented_out_guard
+      description "SAS `* ... ;` line comment looks like a disabled `if " \
+                  "... then do` validity guard — review and either restore " \
+                  "the guard or remove the orphan `end;`."
+      severity :warning
+
+      TT = SasLexer::Lexer::TokenType
+      TC = SasLexer::Lexer::TokenChannel
+
+      # Match `if ... then do` anywhere in the comment body, case-insensitive.
+      # Look for `then` followed (after whitespace and possibly more tokens)
+      # by `do` — the SAS authoring style where the guard expression is
+      # spread across multiple lines.
+      GUARD_PATTERN = /\bif\b.*\bthen\b\s+do\b/im
+
+      def check(_tokens, path:, all_tokens: nil, source: nil) # rubocop:disable Lint/UnusedMethodArgument
+        return [] unless all_tokens
+
+        all_tokens.filter_map do |tok|
+          next unless tok[:channel] == TC::COMMENT
+          next unless tok[:type] == TT::COMMENT_STAT
+
+          body = tok[:text]
+          # Only flag SAS statement-comments that start with `*` (not `**`),
+          # since `** ... **;` is a header comment style and `* ...;` is
+          # the disable-this-statement style.
+          next unless body =~ /\A\s*\*(?!\*)/
+          next unless body =~ GUARD_PATTERN
+
+          finding(
+            line: tok[:start_line],
+            column: tok[:start_column] + 1,
+            message: "looks like a disabled validity guard (`* if ... then do; ...`); " \
+                     "review whether the guard should be live or whether the matching " \
+                     "`end;` is now orphaned.",
+            path: path
+          )
+        end
+      end
+    end
+  end
+end

data/lib/sas_linter/rules/encoding_issues.rb

@@ -0,0 +1,322 @@
+# frozen_string_literal: true
+
+require_relative "../../sas_linter"
+
+class SasLinter
+  module Rules
+    # Flag and (optionally) rewrite encoding issues in SAS source —
+    # smart quotes (`‘ ’ “ ”`), en/em dashes (`– —`), ellipsis (`…`),
+    # non-break space, U+2000–U+200A typographic spaces, line/para
+    # separators, and the Windows-1252 single-byte forms of the same
+    # characters (0x91/0x92, 0x93/0x94, 0x96/0x97, 0xA0, …) that
+    # bypass UTF-8 transcoding when source files arrive from Word /
+    # Outlook / a legacy Latin-1 round-trip.
+    #
+    # The rule has two layers, both off by default:
+    #
+    #   1. `use_defaults: true` enables the canonical fix
+    #      table — UTF8_REPLACEMENTS (multibyte UTF-8 smart
+    #      punctuation) plus BYTE_REPLACEMENTS (single Windows-1252
+    #      bytes the lexer can't make sense of). The fixer walks the
+    #      source as a byte stream and only touches a Win-1252 byte
+    #      when it's not already part of a valid UTF-8 sequence — so
+    #      names like `MÖLLER` (`\xC3\x96`) survive intact.
+    #
+    #   2. `replacements: { from => to }` adds project-specific
+    #      string-level substitutions. Use this for site-local
+    #      cleanups, stylistic preferences (em-dash → "--" instead
+    #      of "-"), to override default behavior on specific bytes,
+    #      or to add extra characters not in the default table.
+    #
+    # Findings carry a line:column position for every match. Autofix
+    # runs the user `replacements:` map FIRST and the canonical
+    # fixer SECOND — so a project-specific pattern can target byte
+    # sequences the canonical defaults would otherwise consume
+    # (e.g. catch a multi-byte ellipsis in a specific surname before
+    # the default `… → ...` rewrite hits it).
+    #
+    # Recognized config options:
+    #   use_defaults: true | false                          (default: false)
+    #   replacements: { String => String }                  (default: {})
+    #   autofix:      true | false                          (default: false)
+    class EncodingIssues < Rule
+      rule_id :encoding_issues
+      description "Source contains smart-punctuation / Win-1252 byte sequences."
+      severity :warning
+
+      # Map of UTF-8 (multi-byte) byte sequences to their ASCII
+      # replacement. Stored as raw byte strings so substitution
+      # doesn't depend on the encoding state of the source.
+      UTF8_REPLACEMENTS = {
+        "\xE2\x80\x98".b => "'",   # U+2018 LEFT SINGLE QUOTATION MARK
+        "\xE2\x80\x99".b => "'",   # U+2019 RIGHT SINGLE QUOTATION MARK
+        "\xE2\x80\x9A".b => "'",   # U+201A SINGLE LOW-9 QUOTATION MARK
+        "\xE2\x80\x9B".b => "'",   # U+201B SINGLE HIGH-REVERSED-9 QUOTATION MARK
+        "\xE2\x80\x9C".b => '"',   # U+201C LEFT DOUBLE QUOTATION MARK
+        "\xE2\x80\x9D".b => '"',   # U+201D RIGHT DOUBLE QUOTATION MARK
+        "\xE2\x80\x9E".b => '"',   # U+201E DOUBLE LOW-9 QUOTATION MARK
+        "\xE2\x80\x93".b => "-",   # U+2013 EN DASH
+        "\xE2\x80\x94".b => "-",   # U+2014 EM DASH
+        "\xE2\x80\x95".b => "-",   # U+2015 HORIZONTAL BAR
+        "\xE2\x80\xA6".b => "...", # U+2026 HORIZONTAL ELLIPSIS
+        "\xC2\xA0".b     => " ",   # U+00A0 NO-BREAK SPACE
+        # Typographic spaces in the U+2000–U+200A range plus the line/para
+        # separators. Word docs sprinkle these in liberally and SAS chokes
+        # with `ERROR 217-322: Invalid statement due to first character
+        # being unprintable`.
+        "\xE2\x80\x80".b => " ",   # U+2000 EN QUAD
+        "\xE2\x80\x81".b => " ",   # U+2001 EM QUAD
+        "\xE2\x80\x82".b => " ",   # U+2002 EN SPACE
+        "\xE2\x80\x83".b => " ",   # U+2003 EM SPACE
+        "\xE2\x80\x84".b => " ",   # U+2004 THREE-PER-EM SPACE
+        "\xE2\x80\x85".b => " ",   # U+2005 FOUR-PER-EM SPACE
+        "\xE2\x80\x86".b => " ",   # U+2006 SIX-PER-EM SPACE
+        "\xE2\x80\x87".b => " ",   # U+2007 FIGURE SPACE
+        "\xE2\x80\x88".b => " ",   # U+2008 PUNCTUATION SPACE
+        "\xE2\x80\x89".b => " ",   # U+2009 THIN SPACE
+        "\xE2\x80\x8A".b => " ",   # U+200A HAIR SPACE
+        "\xE2\x80\xA8".b => "\n",  # U+2028 LINE SEPARATOR
+        "\xE2\x80\xA9".b => "\n",  # U+2029 PARAGRAPH SEPARATOR
+        # Mac Roman 0xD0–0xD5 misread as Win-1252 → Latin-1 letters.
+        # Mac OS Roman uses these byte slots for smart punctuation; when
+        # `SasLinter#read_source` interprets a Mac-Roman-authored file
+        # as Win-1252 it produces these spurious Latin-1 letters in the
+        # post-transcode UTF-8. A typical SAS source corpus (English,
+        # with documents that round-tripped through Word on Mac) shows
+        # this as Latin-1 letters Ð / Ò / Ó / Ô / Õ standing in for
+        # smart-punctuation glyphs.
+        #
+        # Skipping U+00D1 (Ñ) since it has too much legitimate Spanish-
+        # name traffic to safely auto-replace.
+        "\xC3\x90".b     => "-",   # U+00D0 Ð (Mac Roman: en dash)
+        "\xC3\x92".b     => '"',   # U+00D2 Ò (Mac Roman: left double quote)
+        "\xC3\x93".b     => '"',   # U+00D3 Ó (Mac Roman: right double quote)
+        "\xC3\x94".b     => "'",   # U+00D4 Ô (Mac Roman: left single quote)
+        "\xC3\x95".b     => "'",   # U+00D5 Õ (Mac Roman: right single quote)
+      }.freeze
+
+      # Map of single Windows-1252 bytes (0x80-0x9F range, plus 0xA0)
+      # to their ASCII replacement. These bytes are invalid as
+      # standalone UTF-8 but are how Word documents on legacy Windows
+      # render the same characters covered by UTF8_REPLACEMENTS above.
+      # Only applied to bytes that are *not* part of a valid UTF-8
+      # sequence in context.
+      #
+      # 0x85 (HORIZONTAL ELLIPSIS) is intentionally absent. In real-world
+      # SAS source corpora a standalone 0x85 is overwhelmingly a corrupted
+      # Latin-1 letter inside a name (e.g. `\x85` standing in for `Ö`,
+      # `Á`, etc.), not a real ellipsis. Mapping it to `...` would corrupt
+      # those names. Recovering the proper letter is data-loss recovery
+      # and belongs in a separate, source-specific fix configured via the
+      # user `replacements:` map.
+      BYTE_REPLACEMENTS = {
+        0x82 => "'",   # SINGLE LOW-9 QUOTATION MARK
+        0x91 => "'",   # LEFT SINGLE QUOTATION MARK
+        0x92 => "'",   # RIGHT SINGLE QUOTATION MARK
+        0x93 => '"',   # LEFT DOUBLE QUOTATION MARK
+        0x94 => '"',   # RIGHT DOUBLE QUOTATION MARK
+        0x96 => "-",   # EN DASH
+        0x97 => "-",   # EM DASH
+        0xA0 => " ",   # NO-BREAK SPACE
+      }.freeze
+
+      def self.supports_autofix?
+        true
+      end
+
+      def self.from_config(opts = {})
+        opts = opts.transform_keys(&:to_s)
+        replacements = (opts["replacements"] || {}).to_h { |k, v| [k.to_s, v.to_s] }
+        new(
+          use_defaults: opts.fetch("use_defaults", false) ? true : false,
+          replacements: replacements,
+          autofix: opts["autofix"] ? true : false
+        )
+      end
+
+      attr_reader :replacements, :use_defaults
+
+      def initialize(use_defaults: false, replacements: {}, autofix: false)
+        super(autofix: autofix)
+        @use_defaults = use_defaults
+        @replacements = replacements
+      end
+
+      def check(_tokens, path:, all_tokens: nil, source: nil) # rubocop:disable Lint/UnusedMethodArgument
+        return [] unless source
+
+        findings = []
+        findings.concat(default_findings(source, path: path)) if @use_defaults
+        findings.concat(replacement_findings(source, path: path)) unless @replacements.empty?
+        findings
+      end
+
+      def autofix(source)
+        # User `replacements:` run FIRST so a project-specific pattern
+        # can target byte sequences the canonical defaults would
+        # otherwise consume. The motivating case: a SAS source corpus
+        # has stray `\x85` bytes inside surnames (a corrupted Latin-1
+        # letter) that `SasLinter#read_source` transcodes to
+        # `\xE2\x80\xA6` (the UTF-8 ellipsis); a project-specific
+        # `"M…LLER": "MÖLLER"` entry only fires if it sees those bytes
+        # BEFORE the canonical map rewrites them to `...`.
+        #
+        # `.b` is required on every side of the `gsub` so mismatched
+        # encodings can't blow up `String#gsub` with
+        # `Encoding::CompatibilityError`. `source` arrives from
+        # `lint_file` as UTF-8; `@replacements` keys/values from YAML
+        # are UTF-8; `apply_canonical_fix` returns ASCII-8BIT.
+        # Either combination raises when a multi-byte pattern
+        # actually matches. `.b` returns a binary-encoded duplicate
+        # without altering bytes, so the substitution stays byte-
+        # faithful regardless of which direction the mismatch goes.
+        step1 = @replacements.inject(source.b) { |s, (from, to)| s.gsub(from.b, to.b) }
+        @use_defaults ? apply_canonical_fix(step1) : step1
+      end
+
+      # Walk `source` as a byte stream applying UTF8_REPLACEMENTS and
+      # BYTE_REPLACEMENTS. A Win-1252 byte is replaced only when it's
+      # not already part of a valid UTF-8 multibyte sequence — so
+      # `M\xC3\x96LLER` (the Ö in `MÖLLER`) survives, but a standalone
+      # `\x96` becomes `-`. Returns ASCII-8BIT.
+      def apply_canonical_fix(source)
+        bytes = source.bytes
+        out = String.new(encoding: Encoding::BINARY)
+        i = 0
+        n = bytes.length
+
+        while i < n
+          b = bytes[i]
+
+          if b < 0x80
+            out << b
+            i += 1
+            next
+          end
+
+          seq_len = utf8_sequence_length(bytes, i, n)
+          if seq_len.positive?
+            seq = bytes[i, seq_len].pack("C*")
+            replacement = UTF8_REPLACEMENTS[seq]
+            out << (replacement ? replacement.b : seq)
+            i += seq_len
+          else
+            replacement = BYTE_REPLACEMENTS[b]
+            out << (replacement ? replacement.b : b)
+            i += 1
+          end
+        end
+
+        out
+      end
+
+      private
+
+      # Walk the source byte-by-byte. For every byte / sequence in
+      # the canonical replacement table, emit a finding tagged with
+      # its line:column.
+      def default_findings(source, path:)
+        findings = []
+        bytes = source.b.bytes
+        line = 1
+        col = 1
+        i = 0
+        n = bytes.length
+
+        while i < n
+          b = bytes[i]
+
+          if b < 0x80
+            if b == 0x0A
+              line += 1
+              col = 1
+            else
+              col += 1
+            end
+            i += 1
+            next
+          end
+
+          seq_len = utf8_sequence_length(bytes, i, n)
+          if seq_len.positive?
+            seq = bytes[i, seq_len].pack("C*")
+            if (replacement = UTF8_REPLACEMENTS[seq])
+              findings << finding(
+                line: line, column: col,
+                message: "UTF-8 #{format('U+%04X', codepoint(seq))} -> #{replacement.inspect}" \
+                         "#{autofix? ? ' (autofixed)' : ''}",
+                path: path
+              )
+            end
+            col += 1
+            i += seq_len
+          else
+            if (replacement = BYTE_REPLACEMENTS[b])
+              findings << finding(
+                line: line, column: col,
+                message: "Windows-1252 0x#{format('%02X', b)} -> #{replacement.inspect}" \
+                         "#{autofix? ? ' (autofixed)' : ''}",
+                path: path
+              )
+            end
+            col += 1
+            i += 1
+          end
+        end
+        findings
+      end
+
+      def replacement_findings(source, path:)
+        findings = []
+        source.each_line.with_index do |line, line_idx|
+          chomped = line.sub(/\r?\n\z/, "")
+          @replacements.each do |from, to|
+            search_from = 0
+            while (idx = chomped.index(from, search_from))
+              findings << finding(
+                line: line_idx + 1,
+                column: idx + 1,
+                message: "found #{from.inspect}#{autofix? ? " → #{to.inspect} (autofixed)" : ' (no autofix)'}",
+                path: path
+              )
+              search_from = idx + from.length
+            end
+          end
+        end
+        findings
+      end
+
+      def codepoint(seq)
+        seq.encode("UTF-8", invalid: :replace, undef: :replace).codepoints.first
+      end
+
+      # Returns the length (1-4) of a valid UTF-8 sequence starting
+      # at `bytes[i]`, or 0 if the bytes there aren't a valid UTF-8
+      # character. The 1-byte case (ASCII) is handled by the caller
+      # before this is invoked, so we only return ≥2 here.
+      def utf8_sequence_length(bytes, i, n)
+        b = bytes[i]
+        return 0 if b < 0xC2
+        return valid_continuation?(bytes, i + 1, n) ? 2 : 0 if b < 0xE0
+        if b < 0xF0
+          return valid_continuation?(bytes, i + 1, n) && valid_continuation?(bytes, i + 2, n) ? 3 : 0
+        end
+        if b < 0xF5 &&
+           valid_continuation?(bytes, i + 1, n) &&
+           valid_continuation?(bytes, i + 2, n) &&
+           valid_continuation?(bytes, i + 3, n)
+          return 4
+        end
+
+        0
+      end
+
+      def valid_continuation?(bytes, i, n)
+        return false if i >= n
+
+        b = bytes[i]
+        b >= 0x80 && b < 0xC0
+      end
+    end
+  end
+end