probatio_diabolica 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c53898870646f0712544d5bc8998715622cffb4981711737757c26d3f8e41ad7
+  data.tar.gz: e3a828ea56faac76e54443c1292235a849bbfd0aca1ff9745c94b3f53190650e
+SHA512:
+  metadata.gz: 38aa1da254d62abeed89a013e82a1e87b63970c4faedf0fb858d140e3a83a8c61a2fc16fff6f073e0cafd81260f762a7fa0ab294d7a879f20c27c204d9549d50
+  data.tar.gz: 5e108178e7da9d4ec7605959f96c6b782754a068bcec05faedac09bf93c709fc559cf1c4fb6e0f71aaa6110f396ec2a2f549092ab1f17581efd7d8a92a5dd8d9

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/README.md

@@ -0,0 +1,249 @@
+# Probatio Diabolica
+
+<p align="center">
+  <img src="diable.png" alt="Logo Probatio Diabolica" width="260" />
+</p>
+
+A Ruby DSL-based testing framework with classic matchers and LLM-powered matchers (text and image).
+
+This project is experimental and not production-ready.
+
+## What this project does
+
+`probatio_diabolica` runs `*_spec.rb` files through a custom runtime (`PrD::Runtime`) with an RSpec-like syntax:
+
+- `describe`, `context`, `it`, `pending`, `let`, `subject`
+- `expect(...).to(...)` and `expect(...).not_to(...)`
+- standard matchers (`eq`, `be`, `includes`, `have`, `all`)
+- LLM matcher `satisfy(...)` to validate natural-language conditions
+
+Tests are evaluated with `instance_eval` (not through RSpec).
+
+## Installation
+
+### From the gem
+
+```ruby
+gem 'probatio_diabolica'
+```
+
+Then:
+
+```bash
+bundle install
+```
+
+In Ruby code, you can load it with:
+
+```ruby
+require "probatio_diabolica"
+```
+
+### From this repository (local development)
+
+```bash
+bundle install
+bundle exec prd examples/basics_spec.rb
+```
+
+### Build and install as a gem
+
+```bash
+# build the package
+gem build probatio_diabolica.gemspec
+
+# install locally from the built gem
+gem install ./probatio_diabolica-*.gem
+```
+
+After installation, you can run:
+
+```bash
+prd examples/basics_spec.rb
+```
+
+If `prd` is not found, add your gem bin directory to `PATH`:
+
+```bash
+export PATH="$(ruby -e 'print Gem.user_dir')/bin:$PATH"
+```
+
+## Configuration LLM
+
+The runtime automatically loads `prd_helper.rb` if present (or a file passed with `-c`).
+
+Minimal example:
+
+```ruby
+# prd_helper.rb
+RubyLLM.configure do |config|
+  config.openrouter_api_key = ENV['OPENROUTER_API_KEY']
+end
+```
+
+Without valid configuration, tests using `satisfy(...)` will fail.
+
+## Running tests
+
+Command:
+
+```bash
+prd <file_or_directory> [options]
+```
+
+From source checkout (without gem install), use:
+
+```bash
+bundle exec prd <file_or_directory> [options]
+```
+
+Supported options:
+
+- `-o, --out DIR` writes output to `DIR/report.qd` (otherwise stdout)
+- `-c, --config FILE` Ruby config file to require
+- `-t, --type TYPE` formatter type (`simple` by default; supports `simple`, `html`, `json`, `pdf`)
+- `-m, --mode MODE` output verbosity mode (`verbose` by default; supports `verbose`, `synthetic`)
+
+Examples:
+
+```bash
+# single file
+bundle exec prd examples/basics_spec.rb
+
+# all *_spec.rb files in a directory
+bundle exec prd examples
+
+# HTML report
+bundle exec prd examples/image_spec.rb -t html -o ./tmp
+
+# compact synthetic output on console
+bundle exec prd examples/basics_spec.rb --mode synthetic
+```
+
+## Available DSL
+
+### Structure
+
+```ruby
+describe 'My domain' do
+  context 'my context' do
+    let(:value) { 5 }
+    subject { 'hello' }
+
+    it 'runs an assertion' do
+      expect(value).to eq(5)
+    end
+
+    pending 'test to implement later'
+  end
+end
+```
+
+### Assertions
+
+- `expect(actual).to matcher`
+- `expect(actual).not_to matcher`
+- `expect { |subject| ... }.to matcher`
+- `expect.to matcher` (uses `subject`)
+
+### Matchers
+
+- `eq(expected)` equality with `==`
+- `be(expected)` object identity (`equal?`)
+- `includes(expected)` inclusion for `String`, `Array`, `File`, `PDF::Reader`
+- `have(expected)` alias inclusion via `include?`
+- `all(proc)` checks all elements against a block
+- `satisfy(natural_language_condition)` LLM-based validation
+
+### Browser helpers (Ferrum)
+
+`PrD::Runtime` exposes helpers to test content loaded in Chrome:
+
+- `screen(at:, width:, height:, warmup_time:)` captures a PNG and returns a `File`
+- `text(at:, css:, warmup_time:)` extracts a CSS node into a `.txt` file and returns a `File`
+- `network(at:, warmup_time:)` returns Ferrum network traffic
+- `network_urls(at:, warmup_time:)` returns traffic URLs
+- `pdf(at:, warmup_time:)` generates a PDF and returns a `PDF::Reader`
+- `html(at:, warmup_time:)` returns HTML (`browser.body`)
+
+Prerequisites:
+
+- Chrome/Chromium must be installed.
+- The `ferrum` gem is optional and only required for these helpers.
+  - Add `gem 'ferrum'` to your Gemfile, or install it with `gem install ferrum`.
+  - If it is missing, an explicit `LoadError` is raised on the first browser helper call.
+
+Example:
+
+```ruby
+it 'checks dynamic content loaded in browser' do
+  page_text = text(at: 'https://example.com', css: 'main')
+  expect(page_text).to(includes('Example Domain'))
+end
+```
+
+### Source code helper (Prism)
+
+`source_code(...)` uses the `prism` gem to parse Ruby source and extract class/method code.
+
+Prerequisites:
+
+- The `prism` gem is optional and only required for `source_code(...)`.
+  - Add `gem 'prism'` to your Gemfile, or install it with `gem install prism`.
+  - If it is missing, an explicit `LoadError` is raised on the first `source_code(...)` call.
+
+## LLM models
+
+You can set a model at `context` or `it` level:
+
+```ruby
+context 'SQL checks', model: 'qwen/qwen-2.5-72b-instruct:free' do
+  it 'accepts a valid query' do
+    expect('SELECT * FROM users').to satisfy('This statement is valid SQL.')
+  end
+end
+```
+
+The runtime keeps a model stack (`it` can temporarily override the parent `context` model).
+
+## Formatters
+
+- `PrD::Formatters::SimpleFormatter` (text console output)
+- `PrD::Formatters::HtmlFormatter` (simple HTML output)
+- `PrD::Formatters::JsonFormatter` exists in code but is not currently exposed by `bin/prd`
+
+### Subject rendering policy (best effort)
+
+When you define a `subject`, each formatter tries to render it in the most useful way for its medium:
+
+- `SimpleFormatter`:
+  - renders readable text in terminal
+  - for files, prints a textual representation (for example path, file preview for `.txt`)
+- `HtmlFormatter`:
+  - renders text values directly
+  - for image files (`.png`, `.jpg`, `.jpeg`), embeds the image in the report
+  - for PDF subjects (`File` `.pdf` or `PDF::Reader`), embeds the PDF with a `data:application/pdf;base64,...` URI
+- `PdfFormatter`:
+  - renders text values as report lines
+  - for image files (`.png`, `.jpg`, `.jpeg`), inserts the image directly in the PDF report
+- `JsonFormatter`:
+  - keeps a structured representation for machine processing
+  - `File` values (images, PDFs, text files, etc.) are embedded as base64 payloads
+  - `PDF::Reader` values are also embedded as base64 (`application/pdf`)
+
+The goal is to preserve readability and report size while surfacing the richest representation each formatter can reasonably support.
+
+## Useful references in this repository
+
+- Basic example: `examples/basics_spec.rb`
+- Code analysis example: `examples/code_example_spec.rb`
+- Image example: `examples/image_spec.rb`
+- Browser example: `examples/browser_spec.rb`
+- CLI entrypoint: `bin/prd`
+- DSL runtime: `lib/pr_d.rb`
+
+## Current limitations
+
+- Work in progress, API may change.
+- Strong dependency on an LLM provider for `satisfy`.
+- `-o` output is always written to a file named `report.qd`.

data/bin/prd

@@ -0,0 +1,82 @@
+#!/usr/bin/env ruby
+
+begin
+  require 'pr_d'
+rescue LoadError
+  require_relative '../lib/pr_d'
+end
+require 'optparse'
+
+options = {}
+OptionParser
+  .new do |opts|
+    opts.banner = 'Usage: prd [options] FILE_OR_DIR'
+    opts.on('-o', '--out DIR', 'Specify the output directory') { |dir| options[:out] = dir }
+    opts.on('-c', '--config FILE', 'Specify the config file') { |file| options[:config] = file }
+    opts.on('-t', '--type TYPE', 'Specify the formatter type (simple, html, json, pdf)') do |type|
+      options[:formatter] = type
+    end
+    opts.on('-m', '--mode MODE', 'Specify output mode (verbose, synthetic)') do |mode|
+      options[:mode] = mode
+    end
+  end
+  .parse!
+
+input_path = ARGV.shift
+raise 'No tests found. Please specify a test file or directory (e.g. prd examples/basics_spec.rb).' if input_path.nil? || input_path.empty?
+raise "Unexpected arguments: #{ARGV.join(' ')}" unless ARGV.empty?
+raise "Path not found: #{input_path}" unless File.exist?(input_path)
+
+tests =
+  if File.directory?(input_path)
+    files = Dir[File.join(input_path, '**', '*_spec.rb')].sort
+    raise "No spec files found in directory: #{input_path}" if files.empty?
+    files.map { |file| File.read(file) }
+  elsif File.file?(input_path)
+    [File.read(input_path)]
+  end
+
+io =
+  if options[:out].nil?
+    raise 'PDF formatter requires --out to write a binary report file.' if options[:formatter] == 'pdf'
+    STDOUT
+  else
+    Dir.mkdir(options[:out]) unless Dir.exist?(options[:out])
+    report_filename = case options[:formatter]
+                      when 'html'
+                        'report.html'
+                      when 'pdf'
+                        'report.pdf'
+                      else
+                        'report.qd'
+                      end
+    mode = options[:formatter] == 'pdf' ? 'wb' : 'w'
+    File.open(File.join(options[:out], report_filename), mode)
+  end
+
+formatter_class = case options[:formatter]
+                  when nil, 'simple'
+                    PrD::Formatters::SimpleFormatter
+                  when 'html'
+                    PrD::Formatters::HtmlFormatter
+                  when 'json'
+                    PrD::Formatters::JsonFormatter
+                  when 'pdf'
+                    PrD::Formatters::PdfFormatter
+                  else
+                    raise "Unsupported formatter type: #{options[:formatter]}. Supported: simple, html, json, pdf"
+                  end
+
+mode = options[:mode] || 'verbose'
+supported_modes = %w[verbose synthetic]
+raise "Unsupported mode: #{mode}. Supported: verbose, synthetic" unless supported_modes.include?(mode)
+
+serializers = {
+  # Ferrum::Network::Exchange => (->(exchange) { exchange.url }),
+  # PDF::Reader => (->(pdf) { pdf.pages.map(&:text).join("\n") })
+}
+
+formatter = formatter_class.new(io:, serializers:, mode: mode.to_sym)
+runtime = PrD::Runtime.new(formatter:, output_dir: options[:out], config_file: options[:config])
+runtime.run(tests)
+io.close if io != STDOUT

data/lib/pr_d/formatters/formatter.rb

@@ -0,0 +1,105 @@
+module PrD
+  module Formatters
+    class Formatter
+      SUPPORTED_MODES = %i[verbose synthetic].freeze
+
+      def initialize(io: $stdout, serializers: {}, mode: :verbose)
+        @io = io
+        @serializers = serializers
+        @level = 0
+        @mode = normalize_mode(mode)
+        @current_test_title = nil
+      end
+
+      def title(message)
+        raise NotImplementedError, "#{self.class} must implement #title"
+      end
+
+      def context(message)
+        raise NotImplementedError, "#{self.class} must implement #context"
+      end
+
+      def success_result(message)
+        raise NotImplementedError, "#{self.class} must implement #success_result"
+      end
+
+      def failure_result(message)
+        raise NotImplementedError, "#{self.class} must implement #failure_result"
+      end
+
+      def it(description = nil, &block)
+        raise NotImplementedError, "#{self.class} must implement #it"
+      end
+
+      def end_it(description = nil, &block)
+        raise NotImplementedError, "#{self.class} must implement #end_it"
+      end
+
+      def justification(justification)
+        raise NotImplementedError, "#{self.class} must implement #justification"
+      end
+
+      def subject(subject)
+        raise NotImplementedError, "#{self.class} must implement #subject"
+      end
+
+      def pending(description = nil)
+        raise NotImplementedError, "#{self.class} must implement #pending"
+      end
+
+      def expect(expectation)
+        raise NotImplementedError, "#{self.class} must implement #expect"
+      end
+
+      def to
+        raise NotImplementedError, "#{self.class} must implement #to"
+      end
+
+      def not_to
+        raise NotImplementedError, "#{self.class} must implement #not_to"
+      end
+      def matcher(matcher, sources: nil)
+        raise NotImplementedError, "#{self.class} must implement #matcher"
+      end
+
+      def result(passed_count, failed_count)
+        raise NotImplementedError, "#{self.class} must implement #result"
+      end
+
+      def increment_level
+        @level += 1
+      end
+
+      def decrement_level
+        @level -= 1
+      end
+
+      def flush
+        @io.flush
+      end
+
+      private
+
+      def synthetic?
+        @mode == :synthetic
+      end
+
+      def normalize_mode(mode)
+        normalized = mode.to_sym
+        return normalized if SUPPORTED_MODES.include?(normalized)
+
+        raise ArgumentError, "Unsupported formatter mode: #{mode}. Supported: #{SUPPORTED_MODES.join(', ')}"
+      end
+
+      def serialize(value)
+        serializer = @serializers[value.class]
+        return serializer.call(value) if serializer
+        return value.path if value.is_a?(File)
+        return value.map { |v| serialize(v) } if value.is_a?(Array)
+        return value.transform_values { |v| serialize(v) } if value.is_a?(Hash)
+
+        value
+      end
+    end
+  end
+end