testgenai 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d777fd0eeb2312efcfcf966ec80867bcd8eeb0d468b6d71038a8d855cb331cb0
+  data.tar.gz: 9a1aac24a5e315eb90b721990d23e7c409193b09863d83d57f99f20e41b1b27d
+SHA512:
+  metadata.gz: cddad336eb03ef1d9f577e0ecd494274124d612e7225188d77695a1b185c9d022912f6a4f0b967f86cca080e877a2b8dd360bf5c03a4fb4f5cab6bb250aced06
+  data.tar.gz: eb200c2862a0d37485f29cd070c92dd2cd132ed3e18ef1bd45683810a1c8c0cc520fddd46d84bb9b95c5ad78d9d42bf75ee3c82657737188d6444b62e568df3b

data/Gemfile

@@ -0,0 +1,10 @@
+source "https://rubygems.org"
+
+gemspec
+
+group :development, :test do
+  gem "rspec", "~> 3.13"
+  gem "standard", "~> 1.40"
+  gem "minitest"
+  gem "simplecov", require: false
+end

data/README.md

@@ -0,0 +1,155 @@
+# TestGenAI
+
+✨ A CLI gem that adds tests for untested methods in your code ✨
+
+![Ruby Version](https://img.shields.io/badge/Ruby-3.4.5-green?logo=Ruby&logoColor=red&label=Ruby%20version&color=green)
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/grymoire7/testgenai/blob/main/LICENSE)
+
+TestGenAI is a Ruby CLI gem for development environments that finds untested
+methods in your codebase and generates tests for them using an AI service.
+
+> [!CAUTION]
+> TestGenAI is a work in progress and should not be used in production
+> environments. While is it works locally for me, it has not been tested in a
+> wide variety of projects yet and may have many edge cases or bugs. Use with
+> caution and please report any issues you encounter.
+
+
+See `docs/article.md` for the concepts this is based on. The article code is
+illustrative only and has never been run; this repository is the working
+implementation.
+
+## Installation
+
+Add to your project's `Gemfile` in the development group, or install the gem directly:
+
+```bash
+gem install testgenai
+```
+
+## Setup
+
+TestGenAI needs an LLM provider to generate tests. Configure it via environment variables or CLI flags:
+
+| Env var                | Flag                     | Description                                         |
+| ---------------------- | ------------------------ | --------------------------------------------------- |
+| `TESTGENAI_PROVIDER`   | `--provider`             | LLM provider (e.g. `anthropic`, `openai`)           |
+| `TESTGENAI_MODEL`      | `--model`                | Model name (e.g. `claude-opus-4-7`)                 |
+| —                      | `--api-key`              | API key (overrides env var set by the provider SDK) |
+| `TESTGENAI_FRAMEWORK`  | `--test-framework`, `-t` | Test framework: `rspec` (default) or `minitest`     |
+| `TESTGENAI_OUTPUT_DIR` | `--output-dir`, `-o`     | Output directory for generated tests                |
+| `TESTGENAI_PAUSE`      | `--pause`, `-p`          | Seconds to pause between API calls (default: 1)     |
+
+## Commands
+
+```
+testgenai scan      Scan for untested methods and report them
+testgenai context   Scan and show the LLM context that would be sent (diagnostic)
+testgenai generate  Full pipeline: scan → context → generate → validate
+testgenai version   Show version
+testgenai help      Show help
+```
+
+### scan
+
+Finds untested methods and prints them. No API calls are made.
+
+```bash
+bin/testgenai scan
+bin/testgenai scan --test-framework minitest
+```
+
+### context
+
+Like `scan`, but also prints the context snippet that would be sent to the LLM
+for each method. Useful for inspecting what the generator will see before
+running a full `generate`.
+
+### generate
+
+Runs the full pipeline: scan → build context → call LLM → validate generated tests → report results.
+
+```bash
+bin/testgenai generate --provider anthropic --model claude-opus-4-7
+```
+
+Generated test files are written to `spec/` (rspec) or `test/` (minitest) by
+default, mirroring the source file's path under `lib/` or `app/`.
+
+## How scanning works
+
+TestGenAI uses one of two scanners depending on whether SimpleCov is available
+in your project.
+
+### SimpleCov scanner (accurate)
+
+If your `Gemfile` or gemspec contains `simplecov`, TestGenAI runs your test
+suite with `COVERAGE=true` to generate `coverage/.resultset.json`, then uses
+that data to find methods where every executable line has zero hits.
+
+This scanner correctly handles **partially-tested files** — it only reports
+methods that were never exercised, even if other methods in the same file are
+tested.
+
+To use the SimpleCov scanner, add SimpleCov to your project and configure it to
+start when `COVERAGE=true` is set:
+
+```ruby
+# Gemfile
+gem "simplecov", require: false, group: [:development, :test]
+```
+
+```ruby
+# spec/spec_helper.rb or test/test_helper.rb
+if ENV["COVERAGE"]
+  require "simplecov"
+  SimpleCov.start
+end
+```
+
+If `coverage/.resultset.json` already exists from a previous run, TestGenAI
+uses it directly without re-running the test suite.
+
+### File-existence scanner (fallback)
+
+If SimpleCov is not detected, TestGenAI falls back to checking whether a
+corresponding spec/test file exists for each source file. Any source file with
+no matching test file has all its methods reported as untested.
+
+This scanner **cannot detect untested methods in partially-tested files**. A
+file tested only through integration tests or through specs for its subclasses
+will appear fully untested even though its methods are exercised.
+
+## Troubleshooting
+
+### "SimpleCov not found in Gemfile or gemspec. Using file-existence scanner."
+
+SimpleCov is not declared in your `Gemfile` or gemspec. The fallback scanner is
+being used. See [Setup](#setup-simplecov) above to enable the accurate scanner.
+
+### The file-existence scanner reports many false positives
+
+Base classes, modules, and shared utilities that are only tested indirectly
+(through subclass specs or integration tests) will be reported as untested by
+the file-existence scanner. This is expected — it can only see whether a
+matching test file exists, not whether code was actually executed. Use the
+SimpleCov scanner for accurate results.
+
+### "Coverage generation failed. Using file-existence scanner."
+
+SimpleCov was found in the Gemfile, but running the test suite with `COVERAGE=true` did not produce `coverage/.resultset.json`. Check that:
+
+1. SimpleCov is configured to start when `COVERAGE=true` is set (see above)
+2. Your test suite runs successfully with `COVERAGE=true bundle exec rspec` (or equivalent for minitest)
+
+### Parser warning about ruby34
+
+```
+warning: parser/current is loading parser/ruby34, which recognizes 3.4.0-dev-compliant syntax,
+but you are running 3.4.5.
+```
+
+This warning comes from the `parser` gem (a dependency). It is harmless — the
+parser handles Ruby 3.4.x syntax correctly. A 3.4.x release of the parser gem
+is not yet available to silence it.
+

data/bin/testgenai

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift("#{File.dirname(__FILE__)}/../lib")
+
+require 'bundler/setup'
+require "testgenai"
+
+Testgenai::CLI.start(ARGV)

data/lib/testgenai/batch_pipeline.rb

@@ -0,0 +1,39 @@
+module Testgenai
+  class BatchPipeline
+    def initialize(config, context_builder, pipeline, reporter, conventions: nil)
+      @config = config
+      @context_builder = context_builder
+      @pipeline = pipeline
+      @reporter = reporter
+      @conventions = conventions
+    end
+
+    def run(untested_methods)
+      results = {successful: [], failed: [], skipped: []}
+
+      untested_methods.each_with_index do |method_info, i|
+        sleep @config.pause if i > 0
+
+        context = @context_builder.build(method_info)
+        context = context.merge(conventions: @conventions) if @conventions
+        result = @pipeline.run(method_info, context)
+
+        if result[:success]
+          results[:successful] << result
+          @reporter.success(method_info, result)
+        else
+          results[:failed] << result
+          @reporter.failure(method_info, result)
+        end
+      rescue ConfigurationError => e
+        @reporter.fatal_error(e)
+        raise
+      rescue => e
+        results[:skipped] << {method_info: method_info, error: e.message}
+        @reporter.skipped(method_info, e)
+      end
+
+      results
+    end
+  end
+end

data/lib/testgenai/cli.rb

@@ -0,0 +1,126 @@
+require "thor"
+
+module Testgenai
+  class CLI < Thor
+    package_name "testgenai"
+
+    class_option :output_dir, aliases: "-o", desc: "Output directory for generated tests"
+    class_option :test_framework, aliases: "-t", default: "rspec", desc: "Test framework: rspec or minitest"
+    class_option :provider, desc: "LLM provider (e.g. anthropic, openai)"
+    class_option :model, desc: "LLM model name"
+    class_option :api_key, desc: "API key (overrides env var)"
+    class_option :pause, aliases: "-p", type: :numeric, default: 1, desc: "Pause between API calls in seconds"
+
+    def self.exit_on_failure?
+      true
+    end
+
+    map "--version" => :version
+
+    desc "version", "Show version"
+    def version
+      puts Testgenai::VERSION
+    end
+
+    desc "scan", "Scan for untested code and report it"
+    def scan
+      config = build_config
+      scanner = build_scanner(config)
+      warn "Scanning source files..."
+      methods = scanner.scan
+      reporter.scan_results(methods, files_scanned: scanner.files_scanned)
+    end
+
+    desc "context", "Scan, build LLM context, and report it (diagnostic)"
+    def context
+      config = build_config
+      scanner = build_scanner(config)
+      methods = scanner.scan
+      ctx_builder = ContextBuilder.new
+      methods.each do |method_info|
+        ctx = ctx_builder.build(method_info)
+        reporter.context_result(method_info, ctx)
+      end
+    end
+
+    desc "generate", "Full pipeline: scan → context → generate → validate"
+    method_option :conventions, type: :boolean, default: false,
+                  desc: "Synthesize project conventions from existing tests (one LLM call; result cached to spec/conventions.md — add to .gitignore)"
+    def generate
+      config = build_config
+      scanner = build_scanner(config)
+      methods = scanner.scan
+
+      if methods.empty?
+        reporter.scan_results(methods)
+        exit 0
+      end
+
+      conventions = options[:conventions] ? ConventionsSynthesizer.new(config).synthesize : nil
+
+      generator = config.generator_class.new(config)
+      validator = config.validator_class.new
+      single_pipeline = Pipeline.new(generator, validator)
+      ctx_builder = ContextBuilder.new
+      batch = BatchPipeline.new(config, ctx_builder, single_pipeline, reporter, conventions: conventions)
+
+      results = batch.run(methods)
+      reporter.summary(results)
+      exit(results[:successful].empty? ? 1 : 0)
+    rescue ConfigurationError => e
+      warn "Error: #{e.message}"
+      exit 2
+    end
+
+    private
+
+    def build_config
+      Configuration.new(
+        provider: options[:provider],
+        model: options[:model],
+        api_key: options[:api_key],
+        framework: options[:test_framework],
+        output_dir: options[:output_dir],
+        pause: options[:pause]
+      )
+    rescue ConfigurationError => e
+      warn "Configuration error: #{e.message}"
+      exit 2
+    end
+
+    def build_scanner(config)
+      resultset = File.join(Dir.pwd, "coverage", ".resultset.json")
+
+      if File.exist?(resultset)
+        warn "Using SimpleCov coverage data..."
+        return Scanner::SimplecovScanner.new
+      end
+
+      unless simplecov_in_gemfile?
+        warn "Warning: SimpleCov not found in Gemfile or gemspec. Using file-existence scanner."
+        warn "Note: file-existence scanner cannot detect untested methods in partially-tested files."
+        return Scanner::FileExistenceScanner.new
+      end
+
+      warn "Running test suite to generate coverage data..."
+      cmd = (config.framework == "minitest") ? "bundle exec ruby -Itest test/**/*_test.rb" : "bundle exec rspec"
+      system({"COVERAGE" => "true"}, cmd, out: File::NULL, err: File::NULL)
+
+      if File.exist?(resultset)
+        Scanner::SimplecovScanner.new
+      else
+        warn "Warning: Coverage generation failed. Using file-existence scanner."
+        Scanner::FileExistenceScanner.new
+      end
+    end
+
+    def simplecov_in_gemfile?
+      gemfiles = Dir.glob(File.join(Dir.pwd, "{Gemfile,*.gemspec}"))
+      gemfiles.any? { |f| File.read(f).include?("simplecov") }
+    end
+
+    def reporter
+      @reporter ||= Reporter.new
+    end
+  end
+end

data/lib/testgenai/code_extractor.rb

@@ -0,0 +1,22 @@
+require_relative "parser_support"
+
+module Testgenai
+  class CodeExtractor
+    def self.extract(response)
+      if (match = response.match(/```ruby\n(.*?)```/m))
+        match[1]
+      elsif (match = response.match(/```\w*\n(.*?)```/m))
+        match[1]
+      else
+        response
+      end
+    end
+
+    def self.valid_ruby?(code)
+      Testgenai::CurrentParser.parse(code)
+      true
+    rescue Parser::SyntaxError
+      false
+    end
+  end
+end

data/lib/testgenai/configuration.rb

@@ -0,0 +1,30 @@
+module Testgenai
+  class Configuration
+    attr_reader :provider, :model, :api_key, :framework, :output_dir, :pause
+
+    def initialize(options = {})
+      @provider = options[:provider] || ENV["TESTGENAI_PROVIDER"]
+      @model = options[:model] || ENV["TESTGENAI_MODEL"]
+      @api_key = options[:api_key]
+      @framework = options[:framework] || ENV["TESTGENAI_FRAMEWORK"] || "rspec"
+      @output_dir = options[:output_dir] || ENV["TESTGENAI_OUTPUT_DIR"]
+      @pause = (options[:pause] || ENV["TESTGENAI_PAUSE"] || 1).to_f
+    end
+
+    def generator_class
+      case framework
+      when "rspec" then Generator::RspecGenerator
+      when "minitest" then Generator::MinitestGenerator
+      else raise ConfigurationError, "Unknown framework: #{framework}"
+      end
+    end
+
+    def validator_class
+      case framework
+      when "rspec" then Validator::RspecValidator
+      when "minitest" then Validator::MinitestValidator
+      else raise ConfigurationError, "Unknown framework: #{framework}"
+      end
+    end
+  end
+end

data/lib/testgenai/context_builder.rb

@@ -0,0 +1,91 @@
+require_relative "parser_support"
+
+module Testgenai
+  class ContextBuilder
+    def initialize(root: Dir.pwd)
+      @root = root
+    end
+
+    def build(method_info)
+      {
+        target_file: File.read(method_info[:file]),
+        dependencies: extract_dependencies(method_info[:file]),
+        example_usage: find_usages(method_info[:method]),
+        related_tests: find_related_tests(method_info[:file])
+      }
+    rescue => e
+      raise Error, "Could not build context for #{method_info[:file]}: #{e.message}"
+    end
+
+    private
+
+    def extract_dependencies(file)
+      source = File.read(file)
+      ast = Testgenai::CurrentParser.parse(source)
+      return [] unless ast
+      find_requires(ast, File.dirname(file))
+    rescue Parser::SyntaxError
+      []
+    end
+
+    def find_requires(node, dir, results = [])
+      return results unless node.is_a?(Parser::AST::Node)
+
+      if node.type == :send && node.children[0].nil? && node.children[2]&.type == :str
+        path = node.children[2].children[0]
+        case node.children[1]
+        when :require_relative
+          resolved = File.expand_path("#{path}.rb", dir)
+          results << resolved if File.exist?(resolved)
+        when :require
+          resolved = File.join(@root, "lib", "#{path}.rb")
+          results << resolved if File.exist?(resolved)
+        end
+      end
+
+      node.children.each { |c| find_requires(c, dir, results) }
+      results
+    end
+
+    def find_usages(method_name)
+      name = method_name.to_s.sub(/\Aself\./, "")
+      pattern = /\.#{Regexp.escape(name)}[\s(]/
+      usages = []
+
+      source_files.each do |file|
+        lines = File.readlines(file)
+        lines.each_with_index do |line, i|
+          next unless line.match?(pattern)
+          start = [0, i - 2].max
+          finish = [lines.size - 1, i + 2].min
+          usages << lines[start..finish].join
+          break if usages.size >= 3
+        end
+        break if usages.size >= 3
+      end
+
+      usages
+    end
+
+    def find_related_tests(source_file)
+      rel = source_file.sub("#{@root}/", "")
+      base = rel.sub(/\A(?:lib|app)\//, "").sub(/\.rb\z/, "")
+
+      spec_path = File.join(@root, "spec", "#{base}_spec.rb")
+      test_path = File.join(@root, "test", "#{base}_test.rb")
+
+      if File.exist?(spec_path)
+        File.read(spec_path)
+      elsif File.exist?(test_path)
+        File.read(test_path)
+      end
+    end
+
+    def source_files
+      %w[lib app].flat_map do |dir|
+        full = File.join(@root, dir)
+        Dir.exist?(full) ? Dir.glob(File.join(full, "**", "*.rb")) : []
+      end
+    end
+  end
+end

data/lib/testgenai/conventions_extractor.rb

@@ -0,0 +1,115 @@
+module Testgenai
+  class ConventionsExtractor
+    TEST_DIRS = %w[support controllers requests models jobs services].freeze
+
+    def initialize(root: Dir.pwd)
+      @root = root
+    end
+
+    def extract
+      file_pairs = test_files_with_paths
+      return nil if file_pairs.empty?
+
+      contents = file_pairs.map { |_, c| c }
+
+      facts = [
+        auth_pattern(contents),
+        unavailable_helpers,
+        factory_traits,
+        common_stubs(contents),
+        transactional_specs(file_pairs)
+      ].compact
+
+      facts.empty? ? nil : facts.join("\n")
+    end
+
+    private
+
+    def auth_pattern(contents)
+      auth_lines = contents.flat_map do |content|
+        content.scan(/^[^#\n]*(?:session\[|authenticate|login|sign_in)[^\n]+/).map(&:strip)
+      end
+      return nil if auth_lines.empty?
+
+      top, count = auth_lines.tally.max_by { |_, n| n }
+      return nil if count < 3
+
+      "Authentication setup: `#{top}` appears in #{count} test setup blocks."
+    end
+
+    def unavailable_helpers
+      return nil unless File.exist?(File.join(@root, "Gemfile"))
+
+      gemfile_content = gemfile_source
+      absent = []
+      unless gemfile_content.include?("rails-controller-testing")
+        absent << "`assigns` and `assert_template` (rails-controller-testing gem is not present -- use response body assertions instead)"
+      end
+      absent.empty? ? nil : "Unavailable helpers: #{absent.join("; ")}."
+    end
+
+    def factory_traits
+      factory_files = %w[spec/factories test/factories].flat_map do |dir|
+        Dir.glob("#{@root}/#{dir}/**/*.rb")
+      end
+      traits = factory_files.flat_map do |f|
+        File.read(f).scan(/trait\s+:(\w+)/).flatten
+      rescue
+        []
+      end
+      return nil if traits.empty?
+
+      "Factory traits available: #{traits.uniq.map { |t| ":#{t}" }.join(", ")}."
+    end
+
+    def common_stubs(contents)
+      stub_lines = contents.flat_map do |content|
+        content.scan(/allow\([^\n]+/)
+      end
+      frequent = stub_lines.tally.select { |_, n| n >= 3 }.keys.first(5)
+      return nil if frequent.empty?
+
+      "Commonly stubbed in tests:\n" + frequent.map { |l| "  #{l}" }.join("\n")
+    end
+
+    def transactional_specs(file_pairs)
+      disabling = file_pairs.select do |_path, content|
+        content.include?("use_transactional_tests = false") ||
+          content.include?("use_transactional_fixtures = false")
+      end
+      return nil if disabling.empty?
+
+      names = disabling.map { |path, _| path.sub("#{@root}/", "") }.join(", ")
+
+      cleanup_lines = disabling.flat_map do |_path, content|
+        content.scan(/\b\w[\w:.]*\.(?:destroy_all|delete_all|truncate)\b/).map(&:strip)
+      end.uniq
+
+      result = "Transactional fixtures are disabled in: #{names}."
+      if cleanup_lines.any?
+        result += " Cleanup patterns: #{cleanup_lines.map { |l| "`#{l}`" }.join(", ")}."
+      else
+        result += " These tests clean up manually in after blocks."
+      end
+      result
+    end
+
+    def test_files_with_paths
+      TEST_DIRS.flat_map do |sub|
+        %w[spec test].flat_map do |dir|
+          Dir.glob("#{@root}/#{dir}/#{sub}/**/*.rb")
+        end
+      end.uniq.filter_map do |f|
+        [f, File.read(f)]
+      rescue
+        nil
+      end
+    end
+
+    def gemfile_source
+      File.read(File.join(@root, "Gemfile"))
+    rescue
+      ""
+    end
+  end
+end

data/lib/testgenai/conventions_synthesizer.rb

@@ -0,0 +1,76 @@
+require "ruby_llm"
+require "fileutils"
+
+module Testgenai
+  class ConventionsSynthesizer
+    def initialize(config, root: Dir.pwd, cache_path: nil)
+      @config = config
+      @root = root
+      @cache_path = cache_path || default_cache_path
+    end
+
+    def synthesize
+      return File.read(@cache_path) if cache_valid?
+
+      facts = ConventionsExtractor.new(root: @root).extract
+      return nil if facts.nil?
+
+      conventions = call_llm(facts)
+      FileUtils.mkdir_p(File.dirname(@cache_path))
+      File.write(@cache_path, conventions)
+      conventions
+    end
+
+    private
+
+    def default_cache_path
+      test_dir = %w[spec test].find { |d| Dir.exist?(File.join(@root, d)) } || "spec"
+      File.join(@root, test_dir, "conventions.md")
+    end
+
+    def cache_valid?
+      return false unless File.exist?(@cache_path)
+
+      cache_mtime = File.mtime(@cache_path)
+      watched_files.none? { |f| File.mtime(f) > cache_mtime rescue false }
+    end
+
+    def watched_files
+      spec_files = Dir.glob("#{@root}/{spec,test}/**/*.rb")
+      gemfiles = Dir.glob("#{@root}/{Gemfile,Gemfile.lock,*.gemspec}")
+      spec_files + gemfiles
+    end
+
+    def call_llm(facts)
+      if @config.provider.nil? && @config.model.nil?
+        raise ConfigurationError,
+          "provider and model must be configured. " \
+          "Set TESTGENAI_PROVIDER and TESTGENAI_MODEL, or use --provider and --model flags."
+      end
+      raise ConfigurationError, "provider must be configured. Set TESTGENAI_PROVIDER or use --provider flag." if @config.provider.nil?
+      raise ConfigurationError, "model must be configured. Set TESTGENAI_MODEL or use --model flag." if @config.model.nil?
+
+      configure_llm
+      chat = RubyLLM.chat(model: @config.model)
+      chat.ask(synthesis_prompt(facts)).content
+    end
+
+    def configure_llm
+      return unless @config.api_key && @config.provider
+      RubyLLM.configure do |c|
+        c.public_send(:"#{@config.provider}_api_key=", @config.api_key)
+      end
+    end
+
+    def synthesis_prompt(facts)
+      <<~PROMPT
+        The following patterns were extracted mechanically from a Ruby test suite.
+
+        #{facts}
+
+        Write a concise conventions guide (under 300 words) explaining the rule behind each pattern, so a developer generating new tests knows what to do and why.
+        Return plain text, no code blocks.
+      PROMPT
+    end
+  end
+end