spec_ai 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 01a7c9667ce149d1f966604521278c69b09ecba63493b3763afd299604cf6814
+  data.tar.gz: d99d5769469e4b638cc9f592bfa90cc36658bbd3274dce88dd884c6593ef7ab3
+SHA512:
+  metadata.gz: d087b04a2b130b94b097e438d88fe70084287acbfe23753b6eca495d737ec4c9c51425065bcd9b231b54580a68347153f7d976fe0274a8d732ff77d81d9a7990
+  data.tar.gz: 74f124d98ab7fe42fb08bdf3da092d26d110d8cc8d9e9af938bfd22c996bb4f4e264c421c06c598bce5922b15ea5f84ab858d8eb1345282da3ca0f4032582c3b

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## 0.1.0
+
+- Initial release: generate model specs from Rails context, run RSpec, and repair failures with an RCrewAI crew.
+- Terminal UI: timed loaders, color, and a few phase emojis. RCrewAI/HTTP logs stay hidden unless `--verbose`.
+- Stop the repair loop when RSpec is missing from the application bundle and print install instructions.
+- Present RSpec failures in plain language (what broke, why, how to re-run) instead of dumping the raw log.
+- Create a missing FactoryBot factory (and add `factory_bot_rails` to the Gemfile when needed), then tell the user what changed.
+- Create missing `spec/rails_helper.rb` and `spec/spec_helper.rb` when the spec folder was deleted, and show the load error instead of a bare `1 error`.
+- If the LLM returns no JSON test plan, explain the reply and generate from the model schema and validations instead.
+- Replace the model spec on each generate (pass `--merge` to append). Drop examples that assert Devise behavior the model does not enforce.
+- Ruby, not the LLM, decides which examples to keep: a deterministic plan from schema/validations is the base, and invented rules (password special characters, extra email formats) are dropped.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Karthick Nagarajan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,212 @@
+# 🤖 Spec AI
+
+**AI-powered test generation for Ruby on Rails applications.**
+
+Spec AI analyzes your Rails application, understands the model, schema, factory, and routes context, generates test scenarios with an LLM, writes the RSpec file, and runs the generated tests.
+
+> Generate better Rails tests with AI — locally or using hosted LLMs.
+
+It is not a raw “LLM writes an RSpec file” wrapper. Ruby owns control flow. An [RCrewAI](https://github.com/gkosmo/rcrewAI) crew reasons about the app; a deterministic generator writes the spec.
+
+## ✨ Features
+
+- 🤖 AI-powered test case generation
+- 💎 Built specifically for Ruby on Rails
+- 🧠 Rails-aware application context
+- 📋 Automatic test scenario planning
+- 🧪 RSpec test generation
+- 🚀 Automatic test execution
+- 🔧 Local LLMs with Ollama
+- ☁️ Hosted LLMs such as OpenAI
+- 🔍 Analyzes models, schema, factories, associations, and routes
+- ⚡ Compare different AI models for test generation
+- 🔐 Local AI option for privacy-sensitive projects
+
+## 🚀 How it works
+
+Spec AI collects relevant context from your Rails application before asking the model to generate tests.
+
+![How Spec AI works](images/how-it-works.png)
+
+1. **Context collector** reads the model, `db/schema.rb`, associations, FactoryBot factory, existing spec, and matching routes.
+2. **RailsAnalyst** (RCrewAI) may fetch extra related files through sandboxed tools.
+3. **TestArchitect** returns a **JSON test plan** (not Ruby). Ruby then keeps only scenarios the model actually enforces and fills any gaps from the schema and validations. The LLM can suggest extras; it cannot invent columns, associations, or Devise rules.
+4. **RspecGenerator** (plain Ruby) writes `spec/models/user_spec.rb`.
+5. **RspecRunner** executes the file.
+6. On failure, Spec AI drops examples the model does not enforce, then **QADebugger** may return a corrected plan; Ruby regenerates and reruns, up to `max_repair_attempts`.
+
+Agents can only read `app/`, `spec/`, `db/`, `config/`, and `test/` inside the application root. They cannot execute application code or write files directly.
+
+## ⚡ Model comparison
+
+A single generate run against the same Devise `User` model, local Ollama vs hosted OpenAI:
+
+![Ollama llama3.1 vs OpenAI gpt-4o-mini](images/model-comparison.png)
+
+| Metric                   | Ollama / llama3.1 | OpenAI / gpt-4o-mini |
+| ------------------------ | ----------------: | -------------------: |
+| Provider                 |             Local |               Hosted |
+| Test scenarios planned   |                17 |                   21 |
+| RSpec examples generated |                17 |                   21 |
+| AI generation time       |             3m 8s |                 9.1s |
+| Test execution time      |              3.0s |                 1.4s |
+| Failures                 |                 0 |                    0 |
+| Result                   |          ✅ Passed |             ✅ Passed |
+
+Both models generated executable RSpec tests with **0 failures** for the tested `User` model.
+
+The hosted `gpt-4o-mini` run planned more scenarios in this test and finished generation substantially faster. The local `llama3.1` run shows that Rails test generation can also stay on your machine.
+
+> These numbers are a single comparison run, not a general benchmark. Results vary with the Rails app, model complexity, prompt, hardware, model version, and configuration.
+
+Generate with a local Ollama model:
+
+![Generate User with Ollama llama3.1](images/generate-ollama.png)
+
+The same flow with OpenAI:
+
+![Generate User with OpenAI](images/generate-openai.png)
+
+## Installation
+
+Add the gem to your Rails app:
+
+```ruby
+# Gemfile
+group :development, :test do
+  gem "spec_ai", github: "karthick965938/spec-ai"
+end
+```
+
+```bash
+bundle install
+bin/rails generate spec_ai:install
+```
+
+That creates `config/spec_ai.yml`.
+
+## Configuration
+
+```yaml
+# config/spec_ai.yml
+provider: ollama
+model: llama3.1
+temperature: 0.1
+
+testing:
+  framework: rspec
+
+generation:
+  include_edge_cases: true
+  include_security_cases: true
+  include_authorization: true
+
+execution:
+  run_after_generation: true
+  auto_fix: true
+  max_repair_attempts: 2
+```
+
+Set `verbose: true` (or pass `--verbose`) to print RCrewAI and HTTP logs.
+
+API keys come from the environment, not the YAML file:
+
+| Provider   | Environment variables                                      |
+| ---------- | ---------------------------------------------------------- |
+| OpenAI     | `OPENAI_API_KEY`                                           |
+| Anthropic  | `ANTHROPIC_API_KEY`                                        |
+| Gemini     | `GOOGLE_API_KEY` or `GEMINI_API_KEY`                       |
+| Azure      | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`            |
+| Ollama     | `OLLAMA_URL` (defaults to `http://localhost:11434`)        |
+
+RCrewAI is the only LLM layer. You can switch providers without changing the gem:
+
+```bash
+bundle exec spec-ai generate User --provider ollama --model llama3.1
+bundle exec spec-ai generate User --provider openai --model gpt-4o-mini
+```
+
+Local Ollama is the path to keep application source on the developer’s machine.
+
+## Usage
+
+```bash
+bundle exec spec-ai generate User
+bundle exec spec-ai generate app/models/user.rb
+bin/rails spec_ai:generate[User]
+```
+
+If FactoryBot or RSpec bootstrap files are missing, Spec AI creates them and stops when you need to `bundle install`:
+
+![Missing factory and spec helpers](images/setup-missing.png)
+
+```bash
+bundle install
+bundle exec spec-ai generate User
+```
+
+Inspect Rails context without calling an LLM:
+
+```bash
+bundle exec spec-ai analyze User
+```
+
+![Analyze User](images/analyze.png)
+
+Re-run the generated spec the usual way:
+
+```bash
+bundle exec rspec
+bundle exec rspec spec/models/user_spec.rb
+```
+
+![bundle exec rspec](images/rspec.png)
+
+RCrewAI HTTP logs are hidden by default. Pass `-v` to show them.
+
+Other commands:
+
+```bash
+spec-ai analyze User                          # print collected context, no LLM
+spec-ai fix spec/models/user_spec.rb          # run + repair
+spec-ai generate User --no-run                # write spec only
+spec-ai generate User --no-fix                # do not auto-repair
+spec-ai generate User --force                 # replace an existing spec (this is the default)
+spec-ai generate User --merge                 # add examples instead of replacing the spec
+spec-ai generate User --verbose               # show RCrewAI / HTTP logs
+```
+
+`spec-ai missing` and `spec-ai improve` are reserved for 0.3 / 0.5.
+
+## Requirements
+
+- Ruby 3.1+
+- Rails 7+
+- RSpec (`rspec-rails` in the Rails app Gemfile; this version generates model specs only)
+
+If FactoryBot is missing, Spec AI adds `factory_bot_rails` to the Gemfile, writes a factory from the schema, and asks you to `bundle install` before generating.
+
+If the `spec/` folder is missing (`rails_helper`, `spec_helper`), Spec AI creates those bootstrap files so generated specs can load.
+
+## Development
+
+```bash
+bundle install
+bundle exec rspec
+```
+
+Gem tests stub RCrewAI crews. They never call a live LLM provider.
+
+## Author
+
+Built by **[Karthick Nagarajan](https://www.linkedin.com/in/karthick-nagarajan-44800710b/)**.
+
+## Star the project
+
+If you find Spec AI useful, consider giving the project a ⭐ on GitHub.
+
+More features and AI-powered Rails testing capabilities are coming soon.
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).

data/exe/spec-ai

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+lib = File.expand_path("../lib", __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require "spec_ai"
+
+SpecAi::CLI.start(ARGV)

data/lib/generators/spec_ai/install/install_generator.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module SpecAi
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+      desc "Install Spec AI configuration"
+
+      def copy_config
+        template "spec_ai.yml", "config/spec_ai.yml"
+      end
+    end
+  end
+end

data/lib/generators/spec_ai/install/templates/spec_ai.yml

@@ -0,0 +1,23 @@
+# config/spec_ai.yml
+#
+# API keys are read from the environment:
+#   OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY / GEMINI_API_KEY,
+#   AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, OLLAMA_URL
+
+provider: openai
+model: gpt-4o
+temperature: 0.1
+# verbose: true   # show RCrewAI / HTTP logs
+
+testing:
+  framework: rspec
+
+generation:
+  include_edge_cases: true
+  include_security_cases: true
+  include_authorization: true
+
+execution:
+  run_after_generation: true
+  auto_fix: true
+  max_repair_attempts: 2

data/lib/spec_ai/analyzers/existing_specs.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module SpecAi
+  module Analyzers
+    class ExistingSpecs
+      EXAMPLE_RE = /\bit\s+(?:["']([^"']+)["']|%q[{\(]([^})\n]+))/
+
+      def initialize(target, sandbox: Sandbox.new)
+        @target = target
+        @sandbox = sandbox
+      end
+
+      def analyze
+        path = @target.spec_path
+        unless @sandbox.exist?(path)
+          return { path: path, present: false, excerpt: nil, example_names: [] }
+        end
+
+        excerpt = @sandbox.read(path)
+        {
+          path: path,
+          present: true,
+          excerpt: excerpt,
+          example_names: excerpt.scan(EXAMPLE_RE).flatten.compact
+        }
+      end
+    end
+  end
+end

data/lib/spec_ai/analyzers/factories.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module SpecAi
+  module Analyzers
+    class Factories
+      def initialize(target, sandbox: Sandbox.new)
+        @target = target
+        @sandbox = sandbox
+      end
+
+      def analyze
+        path = factory_path
+        return { path: nil, name: @target.factory_name, excerpt: nil, present: false } unless path
+
+        contents = @sandbox.read(path)
+        excerpt = extract_factory(contents, @target.factory_name) || contents
+        { path: path, name: @target.factory_name, excerpt: excerpt, present: true }
+      end
+
+      private
+
+      def factory_path
+        name = @target.factory_name
+        table = @target.table_name
+        candidates = [
+          "spec/factories/#{table}.rb",
+          "spec/factories/#{name}.rb",
+          "spec/factories.rb",
+          "test/factories/#{table}.rb",
+          "test/factories/#{name}.rb",
+          "test/factories.rb"
+        ]
+        globbed = @sandbox.glob("spec/factories/**/*.rb") + @sandbox.glob("test/factories/**/*.rb")
+        (candidates + globbed).find do |path|
+          next unless @sandbox.exist?(path)
+
+          contents = @sandbox.read(path)
+          contents.match?(/factory\s+:#{Regexp.escape(name)}\b/) || path.end_with?("/#{name}.rb", "/#{table}.rb")
+        end
+      end
+
+      def extract_factory(contents, name)
+        contents[/factory\s+:#{Regexp.escape(name)}\b.*?^  end/m]
+      end
+    end
+  end
+end

data/lib/spec_ai/analyzers/model.rb

@@ -0,0 +1,169 @@
+# frozen_string_literal: true
+
+module SpecAi
+  module Analyzers
+    class Model
+      ASSOCIATION_RE = /\b(has_many|has_one|belongs_to|has_and_belongs_to_many)\s+:(\w+)(.*)$/
+      VALIDATES_RE = /^\s*validates(?:_presence_of|_uniqueness_of|_length_of|_inclusion_of|_numericality_of)?\s+(.+)$/
+      SCOPE_RE = /^\s*scope\s+:(\w+)/
+      ENUM_RE = /^\s*enum\s+(?:(\w+):|:(\w+)\s*,|\s+(\w+)\s*:)/
+
+      def initialize(target, sandbox: Sandbox.new)
+        @target = target
+        @sandbox = sandbox
+      end
+
+      def analyze
+        source = @sandbox.read(@target.model_path)
+        live = live_analysis
+
+        {
+          class_name: @target.class_name,
+          path: @target.model_path,
+          source: source,
+          table_name: live[:table_name] || @target.table_name,
+          associations: live[:associations] || parse_associations(source),
+          validations: live[:validations] || parse_validations(source),
+          scopes: live[:scopes] || parse_scopes(source),
+          enums: live[:enums] || parse_enums(source)
+        }
+      end
+
+      private
+
+      def live_analysis
+        return {} unless SpecAi.rails_booted?
+
+        klass = constantize(@target.class_name)
+        return {} unless klass
+
+        {
+          table_name: klass.respond_to?(:table_name) ? klass.table_name : nil,
+          associations: live_associations(klass),
+          validations: live_validations(klass),
+          scopes: live_scopes(klass),
+          enums: live_enums(klass)
+        }
+      rescue StandardError
+        {}
+      end
+
+      def live_associations(klass)
+        return [] unless klass.respond_to?(:reflect_on_all_associations)
+
+        klass.reflect_on_all_associations.map do |reflection|
+          {
+            name: reflection.name.to_s,
+            macro: reflection.macro.to_s,
+            class_name: reflection.class_name.to_s,
+            options: compact_options(reflection.options)
+          }
+        end
+      end
+
+      def live_validations(klass)
+        return [] unless klass.respond_to?(:validators)
+
+        klass.validators.group_by { |validator| Array(validator.attributes).map(&:to_s) }.map do |attributes, validators|
+          {
+            attributes: attributes,
+            validators: validators.map { |validator| validator.kind.to_s }.uniq
+          }
+        end
+      end
+
+      def live_scopes(klass)
+        return [] unless klass.respond_to?(:send)
+
+        (klass.respond_to?(:defined_scopes) ? klass.defined_scopes.keys : [])
+      rescue StandardError
+        []
+      end
+
+      def live_enums(klass)
+        return [] unless klass.respond_to?(:defined_enums)
+
+        klass.defined_enums.keys
+      rescue StandardError
+        []
+      end
+
+      def parse_associations(source)
+        source.each_line.filter_map do |line|
+          match = line.match(ASSOCIATION_RE)
+          next unless match
+
+          options = {}
+          class_name_match = match[3].to_s.match(/class_name:\s*["']([^"']+)["']/)
+          options[:class_name] = class_name_match[1] if class_name_match
+          dependent_match = match[3].to_s.match(/dependent:\s*:(\w+)/)
+          options[:dependent] = dependent_match[1] if dependent_match
+
+          {
+            name: match[2],
+            macro: match[1],
+            class_name: options[:class_name] || Inflector.camelize(Inflector.singularize(match[2])),
+            options: options
+          }
+        end
+      end
+
+      def parse_validations(source)
+        grouped = Hash.new { |hash, key| hash[key] = [] }
+
+        source.each_line do |line|
+          match = line.match(VALIDATES_RE)
+          next unless match
+
+          rest = match[1]
+          attributes = rest.scan(/:(\w+)/).flatten
+          next if attributes.empty?
+
+          validators = []
+          validators << "presence" if line.include?("presence:") || line.include?("validates_presence_of")
+          validators << "uniqueness" if line.include?("uniqueness:") || line.include?("validates_uniqueness_of")
+          validators << "length" if line.include?("length:") || line.include?("validates_length_of")
+          validators << "format" if line.include?("format:")
+          validators << "inclusion" if line.include?("inclusion:") || line.include?("validates_inclusion_of")
+          validators << "numericality" if line.include?("numericality:") || line.include?("validates_numericality_of")
+          validators << "confirmation" if line.include?("confirmation:")
+          validators = ["presence"] if validators.empty? && line.include?("validates_presence_of")
+
+          key = attributes.sort
+          grouped[key].concat(validators)
+        end
+
+        grouped.map do |attributes, validators|
+          { attributes: attributes, validators: validators.uniq }
+        end
+      end
+
+      def parse_scopes(source)
+        source.each_line.filter_map { |line| line[SCOPE_RE, 1] }
+      end
+
+      def parse_enums(source)
+        source.each_line.filter_map do |line|
+          match = line.match(ENUM_RE)
+          next unless match
+
+          match.captures.compact.first
+        end
+      end
+
+      def constantize(name)
+        name.split("::").reduce(Object) { |mod, part| mod.const_get(part) }
+      rescue NameError
+        nil
+      end
+
+      def compact_options(options)
+        return {} unless options.respond_to?(:each_pair)
+
+        options.each_with_object({}) do |(key, value), memo|
+          memo[key] = value unless value.nil?
+        end
+      end
+    end
+  end
+end

data/lib/spec_ai/analyzers/routes.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module SpecAi
+  module Analyzers
+    class Routes
+      def initialize(target, sandbox: Sandbox.new)
+        @target = target
+        @sandbox = sandbox
+      end
+
+      def analyze
+        path = "config/routes.rb"
+        return { path: nil, excerpt: nil } unless @sandbox.exist?(path)
+
+        contents = @sandbox.read(path)
+        token = Inflector.pluralize(@target.factory_name)
+        matching = contents.each_line.grep(/#{Regexp.escape(token)}|#{Regexp.escape(@target.factory_name)}/i)
+        excerpt = matching.any? ? matching.join : contents.lines.first(40).join
+        { path: path, excerpt: excerpt.strip }
+      end
+    end
+  end
+end

data/lib/spec_ai/analyzers/schema.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module SpecAi
+  module Analyzers
+    class Schema
+      def initialize(target, sandbox: Sandbox.new)
+        @target = target
+        @sandbox = sandbox
+      end
+
+      def analyze
+        path = schema_path
+        return { path: nil, table: @target.table_name, excerpt: nil } unless path
+
+        contents = @sandbox.read(path)
+        excerpt = extract_table(contents, @target.table_name)
+        { path: path, table: @target.table_name, excerpt: excerpt, columns: parse_columns(excerpt) }
+      end
+
+      private
+
+      def schema_path
+        %w[db/schema.rb db/structure.sql].find { |path| @sandbox.exist?(path) }
+      end
+
+      def extract_table(contents, table_name)
+        ruby = contents[create_table_regex(table_name), 0]
+        return ruby if ruby
+
+        extract_sql_table(contents, table_name)
+      end
+
+      def create_table_regex(table_name)
+        /
+          create_table\s+["']#{Regexp.escape(table_name)}["'][^\n]*\s+do\s+\|t\|
+          .*?
+          ^\s*end
+        /mx
+      end
+
+      def extract_sql_table(contents, table_name)
+        match = contents.match(/CREATE TABLE(?: IF NOT EXISTS)? ["'`]?#{Regexp.escape(table_name)}["'`]?[^;]*;/i)
+        match && match[0]
+      end
+
+      def parse_columns(excerpt)
+        return [] if excerpt.to_s.empty?
+
+        excerpt.scan(/t\.\w+\s+"([^"]+)"/).flatten
+      end
+    end
+  end
+end