json-repair 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b50bffd203f06c7b7d2fa875802b66dd6fc944d01fe7c7f6c349233b2dc73d60
-  data.tar.gz: f018489c9572a61a72e9784af8f2b2fec335e215933ad0efea1265cff7d7be4e
+  metadata.gz: 1873dc4a32f871c8dc79227074438b0118837f32bc3f36d004510604b265de4a
+  data.tar.gz: 33e15f3df710287842b41aa0a7490b239e5022d7fa53433ae272a103dc34d242
 SHA512:
-  metadata.gz: 7b1047f154815fde7e587fac75c1316ccf799a3ce73c8d5da97ae94305cc8368ea745f7472972a552da4a9df61acb54730d39a2080eb5e61e9fc46d234bbcfd0
-  data.tar.gz: 25d80f6b35509da21cbf0932a67187bd8f69ccac2f06c15e93dbc74a45c0ae4018145051f6d601df5b651aa33036d8cd4dcafefe800f5533110271a38500b4d7
+  metadata.gz: a61ee9fa2a2220c494ded4bba09da73e5366a1af65e5cd6ae4ed08141c93db2ad445f5fe5571b7778e0693b5826e0c2506d042d0b3f8a91c8f78e9f6b213ca08
+  data.tar.gz: 8f1953f3959e5a571e0c6a02de0dab1ad775b007c4024ba21f7ae8b1d70e235b145a1de90bd1ddbf29d1ee16d36649dc68dceca6d536bc5f343ff4c150a7b2a6

data/CLAUDE.md

@@ -0,0 +1,67 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+- `bin/setup` — install dependencies via Bundler.
+- `bundle exec rake` — default task; runs both RSpec and RuboCop.
+- `bundle exec rspec` — run the test suite.
+- `bundle exec rspec spec/json_spec.rb:42` — run a single example by line number; nearly all behavioral specs live in `spec/json_spec.rb`.
+- `bundle exec rubocop` — lint. Project-specific exclusions in `.rubocop.yml` deliberately disable several `Metrics/*` cops for `lib/json/repairer.rb` and `lib/json/repair/string_utils.rb` because the parser is long by design — don't try to "fix" it by chopping methods up.
+- `bin/console` — IRB with the gem preloaded.
+- `bundle exec rake install` / `bundle exec rake release` — local install / publish to rubygems.org.
+- Type checking: `Steepfile` checks `lib/` against `sig/`. Run `bundle exec steep check` if/when steep is installed (not in `Gemfile` by default).
+
+Ruby `>= 3.0.0` is required (per gemspec). CI runs against Ruby 3.3.1.
+
+## Architecture
+
+This gem is a **Ruby port of the [josdejong/jsonrepair](https://github.com/josdejong/jsonrepair) TypeScript library**. The upstream version currently mirrored is tracked in `CHANGELOG.md` (presently v3.14.0). When syncing upstream changes, the goal is parity with the JS implementation, not idiomatic refactoring — keep method names, control flow, and repair heuristics aligned with the JS source so future syncs stay tractable.
+
+### Entry point
+
+`JSON.repair(str)` in `lib/json/repair.rb` is a thin wrapper that constructs `JSON::Repairer.new(str).repair`. `JSON::JSONRepairError` is the only error raised for unrecoverable inputs.
+
+### The parser (`lib/json/repairer.rb`)
+
+A single-pass, hand-written recursive-descent parser. State is three instance variables:
+
+- `@json` — the input string (read-only after init).
+- `@index` — the current cursor into `@json`.
+- `@output` — a mutable `+''` buffer that accumulates the *repaired* JSON. The parser writes directly to `@output` as it walks; it does not build an AST.
+
+Each `parse_*` method (`parse_value`, `parse_object`, `parse_array`, `parse_string`, `parse_number`, `parse_keywords`, `parse_unquoted_string`, `parse_regex`, `parse_comment`, `parse_markdown_code_block`, …) follows a contract:
+
+1. Returns truthy if it consumed something, falsy otherwise.
+2. On success, advances `@index` past the consumed input and appends the *valid* JSON form to `@output`.
+3. On a recoverable mismatch (missing quote, missing comma, trailing comma, wrong quote style, etc.) it performs an in-place repair on `@output` using helpers like `insert_before_last_whitespace`, `strip_last_occurrence`, or `remove_at_index`.
+4. On an unrecoverable error it calls one of the `throw_*` helpers, which raise `JSON::JSONRepairError`.
+
+Two patterns recur and are worth knowing before editing:
+
+- **Backtracking via snapshots.** Methods like `parse_string` capture `i_before = @index` and `o_before = @output.length` before tentatively consuming input. If a later check (e.g. "the end quote turned out not to be a real end quote") fails, they restore both and re-invoke themselves with different flags (e.g. `stop_at_delimiter: true`, `stop_at_index: …`). Preserve this pattern when modifying string/number parsing.
+- **Repair-by-rewriting-tail.** Helpers like `insert_before_last_whitespace(@output, ',')` and `@output = strip_last_occurrence(@output, ',')` patch the already-emitted output to fix things like missing or trailing commas. These run *after* the malformed input has been partially emitted — they are the mechanism for "I now realize that earlier token needed a comma after it."
+
+`repair` (the public method) drives `parse_value` then handles top-level concerns: stripping Markdown fences (` ```json ... ``` `), converting newline-delimited JSON at the root into an array, dropping redundant trailing braces/brackets, and rejecting any non-whitespace trailing garbage.
+
+### Shared helpers (`lib/json/repair/string_utils.rb`)
+
+`JSON::Repair::StringUtils` is a mixin included into `Repairer`. It holds:
+
+- Character constants (`OPENING_BRACE`, `BACKSLASH`, smart-quote variants, special whitespace code points, etc.) used in lieu of magic literals.
+- Character-class predicates (`digit?`, `hex?`, `quote?`, `single_quote_like?`, `delimiter?`, `whitespace?`, `special_whitespace?`, `start_of_value?`, …).
+- The keyword machinery — `parse_keywords` / `parse_keyword` — which converts Python `True`/`False`/`None` and Ruby `nil` into their JSON equivalents in addition to recognizing `true`/`false`/`null`.
+- Output-buffer surgery helpers: `strip_last_occurrence`, `insert_before_last_whitespace`, `remove_at_index`, `ends_with_comma_or_newline?`.
+
+Because the mixin reads `@json`, `@index`, and `@output` directly (notably inside `parse_keyword`), it is **not standalone** — it is coupled to `Repairer`'s state and should only be mixed into classes that own those ivars.
+
+### Type signatures (`sig/`)
+
+RBS signatures mirror the public surface of `JSON.repair`, `JSON::Repairer`, and `JSON::Repair::StringUtils`. Update them in lockstep with `lib/` changes; the `Steepfile` will surface drift.
+
+### Test layout
+
+- `spec/json_spec.rb` — the substantive behavioral suite (700+ examples covering every repair heuristic). New behavior — and every sync from upstream — belongs here.
+- `spec/json/repair_spec.rb` — sanity check on `JSON::Repair::VERSION` only.
+- `.rspec_status` is committed and tracks per-example pass/fail so `--only-failures` / `--next-failure` work across runs.

data/README.md

@@ -1,4 +1,4 @@
-# JSON::Repair [![Gem Version](https://badge.fury.io/rb/json-repair.svg)](https://badge.fury.io/rb/json-repair) [![Build Status](https://github.com/sashazykov/json-repair-rb/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/sashazykov/json-repair-rb/actions)
+# JSON::Repair [![Gem Version](https://badge.fury.io/rb/json-repair.svg)](https://badge.fury.io/rb/json-repair) [![Build Status](https://github.com/sashazykov/json-repair-rb/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/sashazykov/json-repair-rb/actions) [![Stand With Ukraine](https://raw.githubusercontent.com/vshymanskyy/StandWithUkraine/main/badges/StandWithUkraine.svg)](https://stand-with-ukraine.pp.ua)
 
 This is a Ruby gem designed to repair broken JSON strings. Inspired by and based on the [jsonrepair js library](https://github.com/josdejong/jsonrepair/). It efficiently handles and corrects malformed JSON data, making it especially useful in scenarios where JSON output from LLMs might not strictly adhere to JSON standards. Whether it's missing quotes, misplaced commas, or unexpected characters, it ensures that the JSON data is valid and can be parsed correctly.
 
@@ -31,6 +31,21 @@ puts repaired_json  # Outputs: {"name": "Alice", "age": 25}
 
 The `repair` method takes a string containing JSON data and returns a corrected version of this string, ensuring it is valid JSON.
 
+## Command line
+
+The gem ships a `json-repair` executable. It reads from stdin or a file and writes to stdout, `--output FILE`, or back over the input file with `--overwrite`.
+
+```bash
+$ echo '{a:1,}' | json-repair
+{"a":1}
+
+$ json-repair broken.json
+$ json-repair broken.json -o fixed.json
+$ json-repair broken.json --overwrite
+```
+
+Run `json-repair --help` for the full list of options.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/Steepfile

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+target :lib do
+  signature 'sig'
+  check 'lib'
+end

data/exe/json-repair

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'json/repair/cli'
+
+exit JSON::Repair::CLI.call(ARGV)

data/lib/json/repair/cli.rb

@@ -0,0 +1,133 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'optparse'
+require 'tempfile'
+require_relative '../repair'
+
+module JSON
+  module Repair
+    class CLI
+      def self.call(argv, stdin: $stdin, stdout: $stdout, stderr: $stderr)
+        new(stdin: stdin, stdout: stdout, stderr: stderr).call(argv)
+      end
+
+      def initialize(stdin: $stdin, stdout: $stdout, stderr: $stderr)
+        @stdin = stdin
+        @stdout = stdout
+        @stderr = stderr
+      end
+
+      # Reset per-invocation state so a single instance can be safely reused
+      # (e.g. `cli = CLI.new; cli.call(['-v']); cli.call(['x'])`).
+      def call(argv)
+        @output_path = @halt = nil
+        @overwrite = false
+        run(argv)
+      rescue OptionParser::ParseError, JSON::JSONRepairError, SystemCallError, IOError,
+             SystemStackError => e
+        @stderr.puts "json-repair: #{e.message}"
+        1
+      end
+
+      private
+
+      def run(argv)
+        positional = catch(:halt) { parser.parse(argv) }
+        return @halt if @halt
+
+        input_path = positional.first
+        return 1 unless validate(positional, input_path)
+
+        repaired = JSON.repair(read_input(input_path))
+        write_output(repaired, input_path)
+        0
+      end
+
+      def validate(positional, input_path)
+        error = validation_error(positional, input_path)
+        return true unless error
+
+        @stderr.puts "json-repair: #{error}"
+        false
+      end
+
+      def validation_error(positional, input_path)
+        return "unexpected argument: #{positional[1]}" if positional.length > 1
+        return '--overwrite requires a filename' if @overwrite && input_path.nil?
+        return '--overwrite and --output are mutually exclusive' if @overwrite && @output_path
+
+        nil
+      end
+
+      def read_input(input_path)
+        raw = input_path ? File.read(input_path) : @stdin.read
+        raw.force_encoding(Encoding::UTF_8)
+        raise JSON::JSONRepairError, 'input is not valid UTF-8' unless raw.valid_encoding?
+
+        raw
+      end
+
+      def write_output(repaired, input_path)
+        if @overwrite
+          replace_in_place(input_path, repaired)
+        elsif @output_path
+          File.write(@output_path, repaired)
+        else
+          @stdout.write(repaired)
+          @stdout.write("\n") unless repaired.end_with?("\n")
+        end
+      end
+
+      # Write to a uniquely-named tempfile alongside the input, then move it
+      # over the original. Tempfile.create uses O_EXCL + a random suffix, so
+      # the temp path is safe against symlink / clobber races; FileUtils.mv
+      # with force: true handles cross-device renames and Windows, where
+      # File.rename cannot overwrite an existing destination. The original
+      # file's mode is preserved (Tempfile defaults to 0600).
+      #
+      # Symlinks are followed via File.realpath so the underlying file is
+      # rewritten in place and the link is left pointing at it; otherwise
+      # the rename would replace the link itself with a regular file.
+      def replace_in_place(input_path, repaired)
+        real_path = File.realpath(input_path)
+        original_mode = File.stat(real_path).mode
+        Tempfile.create(['json-repair', '.tmp'], File.dirname(real_path)) do |tmp|
+          tmp.write(repaired)
+          tmp.close
+          File.chmod(original_mode, tmp.path)
+          FileUtils.mv(tmp.path, real_path, force: true)
+        end
+      end
+
+      def parser
+        OptionParser.new do |opts|
+          opts.banner = 'Usage: json-repair [filename] [options]'
+          opts.separator ''
+          opts.separator 'Repair a broken JSON document. Reads stdin when no filename is given.'
+          opts.separator ''
+          define_options(opts)
+        end
+      end
+
+      OVERWRITE_DESC = 'Replace the input file in place (requires filename; conflicts with --output)'
+      private_constant :OVERWRITE_DESC
+
+      def define_options(opts)
+        opts.on('-o', '--output FILE', 'Write repaired JSON to FILE') { |f| @output_path = f }
+        opts.on('--overwrite', OVERWRITE_DESC) { @overwrite = true }
+        opts.on('-v', '--version', 'Print version and exit') { halt_with(JSON::Repair::VERSION) }
+        opts.on('-h', '--help', 'Print this help and exit') { halt_with(opts.help) }
+      end
+
+      # Print to stdout and short-circuit `parser.parse` so trailing args
+      # after --version/--help do not raise OptionParser::ParseError and
+      # flip the exit code (the option text promises "...and exit").
+      def halt_with(message)
+        @stdout.puts message
+        @halt = 0
+        throw :halt
+      end
+    end
+  end
+end

data/lib/json/repair/string_utils.rb

@@ -35,21 +35,28 @@ module JSON
       LOWERCASE_E = 'e' # 0x65
       UPPERCASE_F = 'F' # 0x46
       LOWERCASE_F = 'f' # 0x66
-      NON_BREAKING_SPACE = "\u00a0" # 0xa0
-      EN_QUAD = "\u2000" # 0x2000
-      HAIR_SPACE = "\u200a" # 0x200a
-      NARROW_NO_BREAK_SPACE = "\u202f" # 0x202f
-      MEDIUM_MATHEMATICAL_SPACE = "\u205f" # 0x205f
-      IDEOGRAPHIC_SPACE = "\u3000" # 0x3000
-      DOUBLE_QUOTE_LEFT = "\u201c" # 0x201c
-      DOUBLE_QUOTE_RIGHT = "\u201d" # 0x201d
-      QUOTE_LEFT = "\u2018" # 0x2018
-      QUOTE_RIGHT = "\u2019" # 0x2019
+      NON_BREAKING_SPACE = ' ' # 0xa0
+      MONGOLIAN_VOWEL_SEPARATOR = '᠎' # 0x180e
+      EN_QUAD = ' ' # 0x2000
+      ZERO_WIDTH_SPACE = '​' # 0x200b
+      NARROW_NO_BREAK_SPACE = ' ' # 0x202f
+      MEDIUM_MATHEMATICAL_SPACE = ' ' # 0x205f
+      IDEOGRAPHIC_SPACE = '　' # 0x3000
+      ZERO_WIDTH_NO_BREAK_SPACE = '' # 0xfeff
+      DOUBLE_QUOTE_LEFT = '“' # 0x201c
+      DOUBLE_QUOTE_RIGHT = '”' # 0x201d
+      QUOTE_LEFT = '‘' # 0x2018
+      QUOTE_RIGHT = '’' # 0x2019
       GRAVE_ACCENT = '`' # 0x0060
-      ACUTE_ACCENT = "\u00b4" # 0x00b4
+      ACUTE_ACCENT = '´' # 0x00b4
 
       REGEX_DELIMITER = %r{^[,:\[\]/{}()\n+]+$}
+      REGEX_UNQUOTED_STRING_DELIMITER = %r{^[,\[\]/{}\n+]+$}
       REGEX_START_OF_VALUE = /^[\[{\w-]$/
+      # matches "https://" and other schemas
+      REGEX_URL_START = %r{^(http|https|ftp|mailto|file|data|irc)://$}
+      # matches all valid URL characters EXCEPT "[", "]", and "," (important JSON delimiters)
+      REGEX_URL_CHAR = %r{^[A-Za-z0-9\-._~:/?#@!$&'()*+;=]$}
 
       # Functions to check character chars
       def hex?(char)
@@ -70,8 +77,19 @@ module JSON
         REGEX_DELIMITER.match?(char)
       end
 
-      def delimiter_except_slash?(char)
-        delimiter?(char) && char != SLASH
+      def unquoted_string_delimiter?(char)
+        REGEX_UNQUOTED_STRING_DELIMITER.match?(char)
+      end
+
+      REGEX_FUNCTION_NAME_CHAR_START = /\A[a-zA-Z_$]\z/
+      REGEX_FUNCTION_NAME_CHAR = /\A[a-zA-Z0-9_$]\z/
+
+      def function_name_char_start?(char)
+        !char.nil? && REGEX_FUNCTION_NAME_CHAR_START.match?(char)
+      end
+
+      def function_name_char?(char)
+        !char.nil? && REGEX_FUNCTION_NAME_CHAR.match?(char)
       end
 
       def start_of_value?(char)
@@ -86,11 +104,22 @@ module JSON
         [SPACE, NEWLINE, TAB, RETURN].include?(char)
       end
 
+      def whitespace_except_newline?(char)
+        [SPACE, TAB, RETURN].include?(char)
+      end
+
       def special_whitespace?(char)
+        return false unless char
+
         [
-          NON_BREAKING_SPACE, NARROW_NO_BREAK_SPACE, MEDIUM_MATHEMATICAL_SPACE, IDEOGRAPHIC_SPACE
+          NON_BREAKING_SPACE,
+          MONGOLIAN_VOWEL_SEPARATOR,
+          NARROW_NO_BREAK_SPACE,
+          MEDIUM_MATHEMATICAL_SPACE,
+          IDEOGRAPHIC_SPACE,
+          ZERO_WIDTH_NO_BREAK_SPACE
         ].include?(char) ||
-          (char >= EN_QUAD && char <= HAIR_SPACE)
+          (char >= EN_QUAD && char <= ZERO_WIDTH_SPACE)
       end
 
       def quote?(char)
@@ -149,7 +178,7 @@ module JSON
 
       def parse_keyword(name, value)
         if @json[@index, name.length] == name
-          @output += value
+          @output << value
           @index += name.length
           true
         else
@@ -161,10 +190,6 @@ module JSON
         text[0...start] + text[start + count..]
       end
 
-      def function_name?(text)
-        /^\w+$/.match?(text)
-      end
-
       def ends_with_comma_or_newline?(text)
         /[,\n][ \t\r]*$/.match?(text)
       end

data/lib/json/repair/version.rb

@@ -2,6 +2,6 @@
 
 module JSON
   module Repair
-    VERSION = '0.2.0'
+    VERSION = '0.4.0'
   end
 end