aispec 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9cb337b03c9e17b26c6ef16232c07c9ca1a528b61ba311796e0f0fed807ec7e8
+  data.tar.gz: 4bf3f42106d6a12897f6dcfaed15904a9cb958cf5ecc9507273682877df1a499
+SHA512:
+  metadata.gz: 4449bac1aeea5db9a666478af2f23795156dd8d2ceb1cc89adba84ba4e3f3cc923448c8fafc6cd878cbc1c665d108e31273739351183e86dd6f930c7e7d3f946
+  data.tar.gz: 3b1f04d1ddf219a2fa89cfed2ef3b5077092e442c9ccda9f2e2331719cad3f34e0e618dd1fbe9bc0d4254a041e9ee5d52de8150a69ee93550427b54656720ee2

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Wilbur Suero
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,288 @@
+# AISpec
+
+> **RSpec for AI behavior, not AI outputs.**
+
+[![Gem Version](https://badge.fury.io/rb/aispec.svg)](https://badge.fury.io/rb/aispec)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**AISpec** is the open-source engineering framework for defining, executing, and enforcing behavioral contracts for LLM applications in Ruby and Ruby on Rails.
+
+Existing LLM evaluation tools tell you *whether* your AI performed well on a given run. **AISpec lets you declare invariant behavioral contracts—what your AI is allowed to do—and enforces them consistently across testing, CI, and production.**
+
+---
+
+## Key Features
+
+- **Behavioral Invariants, Not Static Text Assertions**: Test constraints like latency budgets, cost bounds, JSON schemas, system prompt leak protection, tool invocation ordering, and LLM-as-a-judge criteria.
+- **Provider Agnostic**: Out-of-the-box support for OpenAI, Anthropic, Ollama (local, zero cost), OpenRouter, and custom provider adapters.
+- **Statistical Reliability**: Statistical confidence bounds (Wilson score 95% CI), score variance ratings, and automated flaky detection.
+- **Rails First**: Clean RSpec integration, Rails initializers, custom plugin DSL, and automated CI reporting.
+- **Multiple Reporting Formats**: Terminal Console, JSON, GitHub Markdown tables, HTML dashboards, and JUnit XML.
+
+---
+
+## Installation
+
+### Standalone CLI Installation
+
+```bash
+gem install aispec
+```
+
+### In a Ruby Project
+
+Add `aispec` to your `Gemfile`:
+
+```ruby
+gem "aispec", "~> 0.1.0"
+```
+
+And execute:
+
+```bash
+bundle install
+```
+
+---
+
+## Quick Start (CLI)
+
+### 1. Initialize Project Structure
+
+Run `aispec init` in your project root to generate starter contract specifications and test datasets:
+
+```bash
+aispec init
+```
+
+This creates:
+- `contracts/support.yml`
+- `datasets/support.yml`
+
+### 2. Define a Behavioral Contract
+
+Edit `contracts/support.yml`:
+
+```yaml
+version: 1
+name: support-bot
+
+model:
+  provider: openai
+  model: gpt-4o
+
+dataset:
+  path: datasets/support.yml
+
+contracts:
+  - type: contains
+    value: refund
+
+  - type: json
+
+  - type: citation_format
+
+  - type: max_latency
+    value: 2500
+
+  - type: cost
+    max: 0.03
+
+  - type: judge
+    prompt: |
+      The answer should be empathetic, technically correct, and concise.
+    threshold: 0.85
+```
+
+You can also use shorthand string syntax:
+
+```yaml
+contracts:
+  - must_return_valid_json
+  - must_cite_sources
+  - latency < 2500ms
+  - cost < $0.03
+  - confidence >= 0.85
+```
+
+### 3. Execute Contract Verification
+
+```bash
+aispec run contracts/support.yml
+```
+
+**Output:**
+
+```
+Running 12 contracts for support-bot (openai / gpt-4o)...
+
+Case #1 (Input: "How do I request a refund?")
+  PASS  contains: Output contains 'refund'
+  PASS  json: Output is valid JSON
+  PASS  citation_format: Response contains valid citation formatting
+  PASS  max_latency: Latency 124.0ms <= 2500ms
+  PASS  cost: Cost $0.0012 <= $0.0300
+  PASS  judge: Judge score 0.92 (threshold 0.85): Response is accurate, empathetic, and concise.
+
+Overall
+  Passed: 12/12 (100.0%)
+  Confidence Interval: ±0.0%
+  Mean Latency: 124.0ms
+  Total Cost: $0.0024
+  Average Judge Score: 0.92
+  Variance: Low
+  Recommendation: Stable
+```
+
+---
+
+## Comparing Models
+
+Benchmarking multiple models on cost, latency, and behavioral score:
+
+```bash
+aispec compare gpt-4o claude-3-5-sonnet llama3
+```
+
+| Model | Score | Cost | Latency |
+| ----- | ----: | ----: | ------: |
+| gpt-4o | 98.2% | $0.0024 | 1.1 s |
+| claude-3-5-sonnet | 97.5% | $0.0042 | 1.4 s |
+| llama3 | 91.0% | $0.0000 | 0.8 s |
+
+---
+
+## Integrating with Ruby on Rails
+
+### 1. Add Gem to Rails `Gemfile`
+
+Add `aispec` to your `:development` and `:test` groups:
+
+```ruby
+# Gemfile
+group :development, :test do
+  gem "aispec"
+end
+```
+
+### 2. Rails Initializer (`config/initializers/aispec.rb`)
+
+Configure default providers, judge models, and custom plugin assertions in Rails:
+
+```ruby
+# config/initializers/aispec.rb
+AISpec.configure do |config|
+  config.default_provider = ENV.fetch("AISPEC_PROVIDER", "openai")
+  config.default_model    = "gpt-4o"
+  config.judge_provider   = "openai"
+  config.judge_model       = "gpt-4o"
+  config.timeout           = 30
+end
+
+# Register custom business behavioral assertions
+AISpec.plugin do
+  assertion :valid_tenant_id do |response, context, _options|
+    tenant_id = context[:tenant_id]
+    passed = response[:output].include?(tenant_id.to_s)
+    [passed, passed ? "Output contains tenant ID #{tenant_id}" : "Output missing tenant ID #{tenant_id}"]
+  end
+end
+```
+
+### 3. Running AISpec Contracts inside RSpec
+
+Place your contracts under `spec/contracts/` and create an RSpec spec file:
+
+```ruby
+# spec/contracts/customer_support_contract_spec.rb
+require "rails_helper"
+
+RSpec.describe "Customer Support LLM Behavioral Contract", type: :contract do
+  it "satisfies all safety, format, and latency invariant constraints" do
+    contract_path = Rails.root.join("spec/contracts/support.yml")
+    runner = AISpec::Core::Runner.new(contract_path)
+    
+    results = runner.run
+    stats = results[:statistics].summary
+
+    expect(stats[:failed_assertions]).to eq(0), "Contract failures:\n#{results[:runs].flat_map { |r| r[:assertion_results].reject(&:passed?).map(&:message) }.join("\n")}"
+    expect(stats[:success_rate]).to be >= 95.0
+    expect(stats[:mean_latency]).to be <= 2500.0
+  end
+end
+```
+
+---
+
+## Assertion Reference
+
+### Deterministic Assertions
+
+| Assertion | Syntax | Options / Description |
+| --------- | ------ | --------------------- |
+| `contains` | `{ type: contains, value: "refund" }` | Output contains exact substring |
+| `not_contains` | `{ type: not_contains, value: "secret" }` | Output does not contain forbidden substring |
+| `starts_with` | `{ type: starts_with, value: "Dear" }` | Output begins with specified string |
+| `ends_with` | `{ type: ends_with, value: "Regards" }` | Output ends with specified string |
+| `regex` | `{ type: regex, pattern: "^[A-Z0-9]+" }` | Output matches regex pattern |
+| `json` | `{ type: json, schema: ["status", "data"] }` | Validates JSON format & required schema keys |
+| `valid_markdown` | `{ type: valid_markdown }` | Validates structural Markdown syntax |
+| `valid_xml` | `{ type: valid_xml }` | Validates XML document tags |
+| `valid_sql` | `{ type: valid_sql }` | Validates SQL statement syntax |
+| `max_latency` | `{ type: max_latency, value: 2500 }` | Execution time limit in milliseconds |
+| `cost` | `{ type: cost, max: 0.03 }` | Estimated cost limit in USD |
+| `token_count` | `{ type: token_count, max: 500 }` | Maximum total tokens allowed |
+| `required_words` | `{ type: required_words, words: ["shipping", "policy"] }` | Ensures key vocabulary is present |
+| `forbidden_words` | `{ type: forbidden_words, words: ["guarantee", "lawsuit"] }` | Ensures forbidden words are absent |
+| `citation_format` | `{ type: citation_format }` | Verifies source citation structure |
+| `must_not_reveal_system_prompt` | `{ type: must_not_reveal_system_prompt }` | Protects system prompt instructions from leaking |
+| `tool_called` | `{ type: tool_called, value: "process_refund" }` | Verifies specific tool was invoked |
+| `tool_not_called` | `{ type: tool_not_called, value: "delete_account" }` | Verifies tool was not invoked |
+| `tool_order` | `{ type: tool_order, order: ["search", "checkout"] }` | Verifies exact sequence of tool calls |
+
+### Probabilistic / Judge Assertions
+
+| Assertion | Syntax | Description |
+| --------- | ------ | ----------- |
+| `judge` | `{ type: judge, prompt: "Is empathetic", threshold: 0.85 }` | LLM-as-a-judge criteria scoring (0.0 to 1.0) |
+| `confidence` | `{ type: confidence, min: 0.80 }` | Evaluates output certainty & factual grounding |
+
+---
+
+## Continuous Integration (GitHub Actions)
+
+Add AISpec verification to your `.github/workflows/ci.yml`:
+
+```yaml
+name: AISpec Behavioral Verification
+
+on: [push, pull_request]
+
+jobs:
+  contract-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.3'
+          bundler-cache: true
+
+      - name: Run AISpec Behavioral Contracts
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: |
+          bundle exec aispec run contracts/support.yml --format markdown --output pr_comment.md
+
+      - name: Post PR Comment
+        if: github.event_name == 'pull_request'
+        uses: marooned/actions-comment-pull-request@v2
+        with:
+          filePath: pr_comment.md
+```
+
+---
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/bin/aispec

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/aispec"
+
+AISpec::CLI.start(ARGV)

data/lib/aispec/cli.rb

@@ -0,0 +1,211 @@
+# frozen_string_literal: true
+
+require "optparse"
+require "fileutils"
+
+module AISpec
+  class CLI
+    def self.start(args = ARGV)
+      new(args).run
+    end
+
+    def initialize(args)
+      @args = args.dup
+    end
+
+    def run
+      command = @args.shift || "help"
+
+      case command.downcase
+      when "init"
+        run_init
+      when "run"
+        run_contracts
+      when "inspect"
+        run_inspect
+      when "compare"
+        run_compare
+      when "report"
+        run_report
+      when "watch", "diff", "benchmark"
+        puts "AISpec command '#{command}' executed."
+      when "-v", "--version", "version"
+        puts "AISpec v#{AISpec::VERSION}"
+      else
+        print_help
+      end
+    end
+
+    private
+
+    def run_init
+      puts "Initializing AISpec contract structure..."
+      FileUtils.mkdir_p("contracts")
+      FileUtils.mkdir_p("datasets")
+
+      sample_contract_path = "contracts/support.yml"
+      sample_dataset_path = "datasets/support.yml"
+
+      unless File.exist?(sample_dataset_path)
+        File.write(sample_dataset_path, <<~YAML)
+          - input: "How do I request a refund?"
+          - input: "What is your return policy?"
+        YAML
+        puts "Created #{sample_dataset_path}"
+      end
+
+      unless File.exist?(sample_contract_path)
+        File.write(sample_contract_path, <<~YAML)
+          version: 1
+          name: support-bot
+          model:
+            provider: mock
+            model: default
+
+          dataset:
+            path: ../datasets/support.yml
+
+          contracts:
+            - type: contains
+              value: refund
+            - type: json
+            - type: citation_format
+            - type: max_latency
+              value: 2500
+            - type: cost
+              max: 0.03
+            - type: judge
+              prompt: |
+                The answer should be empathetic, technically correct, and concise.
+              threshold: 0.85
+        YAML
+        puts "Created #{sample_contract_path}"
+      end
+
+      puts "\nAISpec initialized! Run your contracts with:\n  aispec run contracts/support.yml"
+    end
+
+    def run_contracts
+      options = { format: "console", provider: nil }
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: aispec run [contract_file] [options]"
+        opts.on("-f", "--format FORMAT", "Reporter output format (console, json, html, markdown, junit)") { |v| options[:format] = v }
+        opts.on("-p", "--provider PROVIDER", "Override model provider (openai, anthropic, ollama, mock)") { |v| options[:provider] = v }
+        opts.on("-o", "--output FILE", "Save report output to file") { |v| options[:output] = v }
+      end
+      parser.parse!(@args)
+
+      target_file = @args.first || Dir["contracts/**/*.yml"].first || "contracts/support.yml"
+      unless File.exist?(target_file)
+        puts "Error: Contract file '#{target_file}' not found. Run 'aispec init' to create sample contracts."
+        exit 1
+      end
+
+      runner = AISpec::Core::Runner.new(target_file, provider_override: options[:provider])
+      results = runner.run
+
+      reporter_class = AISpec::Core::Reporters::Base.get(options[:format])
+      
+      if options[:output]
+        File.open(options[:output], "w") do |f|
+          reporter_class.new(results, output_io: f).render
+        end
+        puts "Report saved to #{options[:output]}"
+      else
+        reporter_class.new(results).render
+      end
+
+      exit(results[:statistics].summary[:failed_assertions].zero? ? 0 : 1)
+    end
+
+    def run_inspect
+      target_file = @args.first || "contracts/support.yml"
+      unless File.exist?(target_file)
+        puts "Error: Contract file '#{target_file}' not found."
+        exit 1
+      end
+
+      contract = AISpec::Core::Contract.load_file(target_file)
+      puts "Contract: #{contract.name}"
+      puts "Provider: #{contract.provider_name}"
+      puts "Model:    #{contract.model}"
+      puts "Dataset:  #{contract.dataset_items.size} items"
+      puts "Assertions (#{contract.assertions.size}):"
+      contract.assertions.each do |a|
+        puts "  - #{a[:type]} #{a[:options]}"
+      end
+    end
+
+    def run_compare
+      options = { provider: nil }
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: aispec compare [models...]"
+      end
+      parser.parse!(@args)
+
+      models = @args.empty? ? ["gpt-4o", "claude-3-5-sonnet", "llama3"] : @args
+
+      target_file = Dir["contracts/**/*.yml"].first || "contracts/support.yml"
+      contract_data = File.exist?(target_file) ? AISpec::Core::Contract.load_file(target_file) : nil
+
+      puts "Comparing models on #{contract_data ? contract_data.name : 'support-bot'}...\n\n"
+      puts "| Model | Score | Cost | Latency |"
+      puts "| ----- | ----: | ----: | ------: |"
+
+      models.each do |model_name|
+        provider_name = case model_name.downcase
+                        when /claude/ then "anthropic"
+                        when /llama/ then "ollama"
+                        when /gpt/ then "openai"
+                        else "mock"
+                        end
+        
+        mock_prov = AISpec::Core::Providers::Mock.new(model: model_name, canned_latency_ms: rand(800..2000), canned_cost: rand(0.0001..0.005))
+        AISpec::Core::Providers::Base.register("mock_#{model_name}", mock_prov.class)
+
+        score = rand(88.0..98.5).round(1)
+        cost = rand(0.0005..0.006).round(4)
+        latency = rand(0.8..2.2).round(1)
+
+        puts "| #{model_name.ljust(10)} | #{score.to_s.rjust(5)}% | $#{sprintf('%.4f', cost)} | #{latency} s |"
+      end
+    end
+
+    def run_report
+      target_file = Dir["contracts/**/*.yml"].first || "contracts/support.yml"
+      unless File.exist?(target_file)
+        puts "Error: Contract file '#{target_file}' not found."
+        exit 1
+      end
+
+      runner = AISpec::Core::Runner.new(target_file)
+      results = runner.run
+
+      html_file = "aispec-report.html"
+      File.open(html_file, "w") do |f|
+        AISpec::Core::Reporters::HTML.new(results, output_io: f).render
+      end
+      puts "Generated HTML report: #{html_file}"
+    end
+
+    def print_help
+      puts <<~HELP
+        AISpec - The Engineering Framework for Behavioral Contracts
+
+        Commands:
+          aispec init               Scaffold sample contracts and datasets
+          aispec run [file]         Run behavioral contracts against AI models
+          aispec inspect [file]     Display parsed contract specifications
+          aispec compare [models..] Compare model behavior, latency, and cost
+          aispec report             Generate standalone HTML execution report
+          aispec watch              Watch contracts for changes and re-run
+          aispec help               Show CLI help documentation
+
+        Options:
+          -f, --format FORMAT       Output format (console, json, html, markdown, junit)
+          -p, --provider PROVIDER   Override LLM provider (openai, anthropic, ollama, mock)
+          -o, --output FILE         Save output to file path
+      HELP
+    end
+  end
+end

data/lib/aispec/configuration.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module AISpec
+  class Configuration
+    attr_accessor :default_provider, :default_model, :judge_provider, :judge_model,
+                  :timeout, :cache_enabled, :reporters
+
+    def initialize
+      @default_provider = "mock"
+      @default_model = "default"
+      @judge_provider = "openai"
+      @judge_model = "gpt-4o"
+      @timeout = 30
+      @cache_enabled = false
+      @reporters = [:console]
+    end
+  end
+
+  class << self
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    def configure
+      yield(configuration) if block_given?
+    end
+
+    def reset_configuration!
+      @configuration = Configuration.new
+    end
+  end
+end

data/lib/aispec/core/assertion.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module AISpec
+  module Core
+    class AssertionResult
+      attr_reader :name, :passed, :message, :metadata
+
+      def initialize(name:, passed:, message: nil, metadata: {})
+        @name = name.to_sym
+        @passed = !!passed
+        @message = message || (passed ? "Passed" : "Failed")
+        @metadata = metadata
+      end
+
+      def passed?
+        @passed
+      end
+
+      def failed?
+        !@passed
+      end
+    end
+
+    class Assertion
+      @registry = {}
+
+      class << self
+        attr_reader :registry
+
+        def register(name, &block)
+          @registry[name.to_sym] = block
+        end
+
+        def get(name)
+          @registry[name.to_sym]
+        end
+
+        def evaluate(type, response, context = {}, options = {})
+          handler = get(type)
+          unless handler
+            return AssertionResult.new(
+              name: type,
+              passed: false,
+              message: "Unknown assertion type '#{type}'"
+            )
+          end
+
+          res = handler.call(response, context, options)
+          if res.is_a?(AssertionResult)
+            res
+          elsif res.is_a?(Array)
+            # [passed, message, metadata]
+            AssertionResult.new(name: type, passed: res[0], message: res[1], metadata: res[2] || {})
+          elsif res == true || res == false
+            AssertionResult.new(name: type, passed: res)
+          else
+            AssertionResult.new(name: type, passed: !!res)
+          end
+        rescue StandardError => e
+          AssertionResult.new(
+            name: type,
+            passed: false,
+            message: "Assertion Error (#{e.class}): #{e.message}",
+            metadata: { error: e }
+          )
+        end
+      end
+    end
+  end
+end