checkset 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 459566d1d7eef689dcd1c0cf625d4669470a55273895253fa1b7b129c57dab46
+  data.tar.gz: 9434ff8ea911baab9e50330dff0528fd31a9bc04c2a59c6cbd13152bcf31f265
+SHA512:
+  metadata.gz: 06cfbf3c011e045408b06b17b0bb08f21ba19b916c88ce27fc00acc3066870d6bfd1d33a753af2b18e5e7a8e89d18ecfc03f1f4d3f6b8275128d4b2d5d30fe26
+  data.tar.gz: 97fc2baa0e7305ad456f8c460b9dcb012cdd5f50ee00378789c68831c0878597325174018dcb38c789f05200ca6016b80bb802d5c7144faa15729ee52d7c57bc

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Andrea Fomera
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,428 @@
+# Checkset
+
+A browser-based verification suite built on [Playwright](https://playwright.dev/). Run checks against any URL — local dev, staging, or production — with automatic screenshots, structured JSON results, and a prep system for state management.
+
+Inspired by [Basecamp's Upright](https://github.com/basecamp/upright) Playwright probes, but focused on on-demand verification rather than synthetic monitoring.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "checkset"
+```
+
+Then install Playwright:
+
+```bash
+npm install playwright
+npx playwright install chromium
+```
+
+## Quick Start
+
+Scaffold a new project:
+
+```bash
+bundle exec checkset init
+```
+
+This creates a starter `checkset.yml` and `checks/homepage_loads.rb` to get you going.
+
+Create a check in `checks/`:
+
+```ruby
+# checks/homepage.rb
+class Checks::Homepage < Checkset::Check
+  description "Verifies the homepage loads correctly"
+
+  def call
+    visit "/"
+
+    verify "page has title" do
+      page.title.include?("My App")
+    end
+
+    verify "has main heading" do
+      page.get_by_role("heading", name: "Welcome").visible?
+    end
+  end
+end
+```
+
+Run it:
+
+```bash
+bundle exec checkset --target https://staging.myapp.com
+```
+
+## The DSL
+
+Checks are Ruby classes that subclass `Checkset::Check`. Inside `call`, you use two primitives:
+
+### `verify` — assertions that continue on failure
+
+```ruby
+verify "cart has items" do
+  page.get_by_test_id("cart-count").text_content.to_i > 0
+end
+```
+
+The block returns truthy (pass) or falsy (fail). On failure, the check **keeps running** to collect all failures in one run.
+
+### `step` — actions that halt on failure
+
+```ruby
+step "add product to cart" do
+  page.get_by_role("button", name: "Add to Cart").first.click
+end
+```
+
+Steps perform navigation or interaction. On failure, the check **stops** because subsequent steps depend on the state this one creates.
+
+### `visit` — navigate relative to the target URL
+
+```ruby
+visit "/products"  # goes to https://staging.myapp.com/products
+```
+
+### Full example
+
+```ruby
+class Checks::UserCanCheckout < Checkset::Check
+  prep :sign_in_as_customer
+  description "Verifies the full checkout flow"
+  tags :checkout, :critical
+
+  def call
+    visit "/products"
+
+    step "add product to cart" do
+      page.get_by_role("button", name: "Add to Cart").first.click
+    end
+
+    verify "cart has items" do
+      page.get_by_test_id("cart-count").text_content.to_i > 0
+    end
+
+    step "go to checkout" do
+      page.get_by_role("link", name: "Cart").click
+      page.get_by_role("button", name: "Checkout").click
+    end
+
+    verify "reached checkout page" do
+      page.url.include?("/checkout")
+    end
+  end
+end
+```
+
+## Preps
+
+Preps handle state setup before a check runs — signing in, seeding data, etc. They run in the **same browser context** as the check, so session state carries over.
+
+```ruby
+# preps/sign_in_as_admin.rb
+class Preps::SignInAsAdmin < Checkset::Prep
+  def satisfy(page)
+    page.goto("#{target_url}/login")
+    page.get_by_label("Email").fill(credentials[:admin_email])
+    page.get_by_label("Password").fill(credentials[:admin_password])
+    page.get_by_role("button", name: "Sign in").click
+    page.wait_for_url("**/dashboard")
+  end
+
+  # Optional: skip sign-in if already authenticated
+  def satisfied?(page)
+    page.goto("#{target_url}/dashboard")
+    !page.url.include?("/login")
+  rescue
+    false
+  end
+end
+```
+
+Declare preps on your check:
+
+```ruby
+class Checks::AdminDashboard < Checkset::Check
+  prep :sign_in_as_admin
+
+  def call
+    visit "/admin/dashboard"
+    verify("dashboard loads") { page.get_by_role("heading", name: "Dashboard").visible? }
+  end
+end
+```
+
+Multiple preps run in declaration order:
+
+```ruby
+class Checks::UserCanCheckout < Checkset::Check
+  prep :sign_in_as_customer
+  prep :test_customer_account
+  # ...
+end
+```
+
+## Suites
+
+When you need to run checks against multiple domains, create a `checkset.yml` to map suite names to target URLs. Checks are auto-discovered by folder — no need to list them manually.
+
+### Folder structure
+
+```
+checks/
+├── app/
+│   ├── user_can_sign_in.rb    → belongs to "app" suite
+│   └── user_can_checkout.rb   → belongs to "app" suite
+├── admin/
+│   └── admin_dashboard.rb     → belongs to "admin" suite
+└── homepage.rb                → top-level = runs in EVERY suite
+```
+
+### `checkset.yml`
+
+```yaml
+# Default domain for %{domain} interpolation
+base_domain: afomera.dev
+
+# Optional: override the checks directory (default: checks)
+# checks_dir: path/to/checks
+
+suites:
+  app:
+    target: https://app.%{domain}
+  admin:
+    target: https://admin.%{domain}
+```
+
+Targets support two kinds of interpolation:
+
+- **`%{domain}`** — replaced with `base_domain` from the yml (or `--domain` CLI flag)
+- **`${ENV_VAR}`** — replaced with the environment variable's value
+
+```yaml
+suites:
+  app:
+    target: https://app.%{domain}         # uses base_domain or --domain
+  monitoring:
+    target: https://${MONITORING_HOST}     # uses env var
+  marketing:
+    target: https://marketing.example.com  # hardcoded, no interpolation
+```
+
+### Running suites
+
+```bash
+# Run all suites (uses base_domain from yml)
+bundle exec checkset
+
+# Override the domain for staging/preview environments
+bundle exec checkset --domain staging.afomera.dev
+
+# Run just one suite
+bundle exec checkset --suite app
+
+# ENV vars work too
+MONITORING_HOST=status.example.com bundle exec checkset
+
+# Override with --target (ignores yml, loads all checks)
+bundle exec checkset --target https://staging.myapp.com
+```
+
+Each suite runs checks from its subfolder (`checks/app/**/*.rb`) plus any top-level checks (`checks/*.rb`). All suites run concurrently, sharing a single browser for efficiency.
+
+## CLI
+
+```bash
+# Basic usage
+bundle exec checkset --target https://staging.myapp.com
+
+# Run a specific check
+bundle exec checkset --target https://staging.myapp.com Checks::UserCanCheckout
+
+# Debug mode — headed browser with slow motion
+bundle exec checkset --target http://localhost:3000 --headed --slow-mo 500
+
+# Run checks in parallel
+bundle exec checkset --target https://staging.myapp.com --parallel 4
+
+# Run only checks tagged :critical
+bundle exec checkset --target https://staging.myapp.com --tag critical
+
+# Retry failed checks up to 2 times (useful for flaky browser checks)
+bundle exec checkset --target https://staging.myapp.com --retries 2
+
+# All options
+bundle exec checkset --help
+```
+
+### Options
+
+| Flag | Description | Default |
+|------|-------------|---------|
+| `--target URL` | Target URL to verify against | required (unless `checkset.yml` exists) |
+| `--suite NAME` | Run only the named suite from `checkset.yml` | all suites |
+| `--config FILE` | Path to config file | `checkset.yml` |
+| `--domain DOMAIN` | Base domain for `%{domain}` in suite targets | `base_domain` from yml |
+| `--tag TAG` | Only run checks tagged with TAG | all checks |
+| `--retries N` | Retry failed checks N times | `0` |
+| `--clean` | Remove all screenshots, traces, and results | |
+| `--headed` | Run browser in headed mode | headless |
+| `--browser TYPE` | `chromium`, `firefox`, or `webkit` | `chromium` |
+| `--parallel N` | Run N checks concurrently | `1` |
+| `--slow-mo MS` | Slow down actions by N ms | off |
+| `--timeout MS` | Default timeout in ms | `10000` |
+| `--screenshots-dir DIR` | Screenshot output directory | `tmp/checkset/screenshots` |
+| `--checks-dir DIR` | Directory containing checks | `checks` |
+| `--preps-dir DIR` | Directory containing preps | `preps` |
+| `--playwright-server URL` | WebSocket URL for remote Playwright | local |
+
+## Output
+
+### Terminal
+
+Single target:
+
+```
+Checkset — https://staging.myapp.com
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+   PASS   UserCanSignIn .................. 1.2s
+   PASS   UserCanCheckout ................ 3.4s
+   FAIL   AdminCanViewDashboard .......... 2.1s
+          ✗ verify "dashboard loads" — timed out waiting for selector
+          screenshot: tmp/checkset/screenshots/admin_can_view_dashboard/FAIL_...png
+          trace: tmp/checkset/traces/admin_can_view_dashboard_...zip
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+2 passed, 1 failed (6.7s)
+Results: tmp/checkset/results/results_20260218_103000.json
+```
+
+Suite mode (suites run concurrently, results printed per suite):
+
+```
+Checkset — app → https://app.afomera.dev
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+   PASS   PageLoads ........................................ 0.8s
+   PASS   UserCanSignIn .................................... 1.2s
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+2 passed (2.0s)
+Results: tmp/checkset/results/results_app_20260218_103000.json
+
+
+Checkset — admin → https://admin.afomera.dev
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+   PASS   PageLoads ........................................ 0.7s
+   PASS   AdminDashboard ................................... 1.5s
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+2 passed (2.2s)
+Results: tmp/checkset/results/results_admin_20260218_103000.json
+
+
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+All suites: 4 checks across 2 suites — 4 passed, 0 failed, 0 skipped (2.5s)
+```
+
+### JSON
+
+Every run writes structured results to `tmp/checkset/results/`. In suite mode, filenames include the suite name (e.g. `results_app_20260218_103000.json`).
+
+```json
+{
+  "run_at": "2026-02-18T10:30:00-07:00",
+  "target_url": "https://staging.myapp.com",
+  "duration": 6.7,
+  "summary": { "total": 3, "passed": 2, "failed": 1, "skipped": 0 },
+  "checks": [
+    {
+      "check": "Checks::UserCanCheckout",
+      "status": "passed",
+      "duration": 3.4,
+      "steps": [
+        { "name": "cart has items", "type": "verify", "status": "passed", "duration": 0.5, "screenshot_path": "..." }
+      ]
+    }
+  ]
+}
+```
+
+### Screenshots & Traces
+
+- **Screenshots** are captured on every `verify` and `step` (pass or fail)
+- **Full-page failure screenshots** are taken when a verify/step fails
+- **Playwright traces** (`.zip`) are saved only on failure — open with `npx playwright show-trace trace.zip`
+
+## Configuration
+
+```ruby
+# checkset.rb or anywhere before running
+Checkset.configure do |c|
+  c.target_url = "https://staging.myapp.com"
+  c.headless = true
+  c.browser_type = :chromium
+  c.default_timeout = 10_000
+  c.viewport_size = { width: 1280, height: 720 }
+  c.credentials_provider = :env  # or :rails_credentials
+end
+```
+
+## Credentials
+
+By default, credentials come from environment variables:
+
+```ruby
+credentials[:admin_email]  # reads ENV["ADMIN_EMAIL"]
+```
+
+With Rails, you can use encrypted credentials:
+
+```ruby
+Checkset.configure { |c| c.credentials_provider = :rails_credentials }
+# reads Rails.application.credentials.dig(:checkset, :admin_email)
+```
+
+## Try the Examples
+
+The `examples/` directory includes sample checks organized into suites. To try them out:
+
+```bash
+# Run the built-in smoke test against example.com
+bundle exec checkset --target https://example.com
+
+# Run example checks against a single target
+bundle exec checkset --target https://hey.com --checks-dir examples/checks
+
+# Run all example suites using the example checkset.yml
+bundle exec checkset --config examples/checkset.yml
+
+# Run just the "hey" suite from the examples
+bundle exec checkset --config examples/checkset.yml --suite hey
+
+# Run in headed mode to watch the browser
+bundle exec checkset --config examples/checkset.yml --suite basecamp --headed
+
+# Run all suites with checks running in parallel within each suite
+bundle exec checkset --config examples/checkset.yml --parallel 3
+
+# Run a specific check by name
+bundle exec checkset --target https://hey.com --checks-dir examples/checks HeyHomepage
+
+# Clean up all output (screenshots, traces, results)
+bundle exec checkset --clean
+```
+
+## Exit Codes
+
+- `0` — all checks passed
+- `1` — one or more checks failed
+
+## License
+
+MIT

data/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/test_*.rb"]
+end
+
+task default: :test

data/checks/example.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Checks
+  class ExampleDotCom < Checkset::Check
+    description "Verifies example.com loads correctly"
+
+    def call
+      visit "/"
+
+      verify "page has title" do
+        page.title.include?("Example")
+      end
+
+      verify "page has heading" do
+        page.get_by_role("heading", name: "Example Domain").visible?
+      end
+    end
+  end
+end

data/checkset.gemspec

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require_relative "lib/checkset/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "checkset"
+  spec.version = Checkset::VERSION
+  spec.authors = ["Andrea Fomera"]
+  spec.email = ["afomera@hey.com"]
+
+  spec.summary = "Browser-based verification suite built on Playwright"
+  spec.description = "A standalone Ruby gem for running browser-based verification checks against any URL. " \
+                     "Built on Playwright with automatic screenshots, structured JSON results, and a prep system " \
+                     "for state management."
+  spec.homepage = "https://github.com/afomera/checkset"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.1"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/main/CHANGELOG.md"
+
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = ["checkset"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "playwright-ruby-client", "~> 1.50"
+end

data/examples/checks/basecamp/homepage.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Checks
+  class BasecampHomepage < Checkset::Check
+    description "Verifies basecamp.com homepage loads correctly"
+    tags :basecamp, :homepage
+
+    def call
+      visit "/"
+
+      verify "page has title" do
+        page.title.downcase.include?("basecamp")
+      end
+
+      verify "has primary heading" do
+        page.get_by_role("heading").first.visible?
+      end
+
+      verify "has sign up call to action" do
+        page.get_by_role("link", name: /try|sign up|get started|free/i).first.visible?
+      end
+    end
+  end
+end

data/examples/checks/hey/homepage.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Checks
+  class HeyHomepage < Checkset::Check
+    description "Verifies hey.com homepage loads and key elements are present"
+    tags :hey, :homepage
+
+    def call
+      visit "/"
+
+      verify "page has title" do
+        page.title.downcase.include?("hey")
+      end
+
+      verify "has main heading" do
+        page.get_by_role("heading", name: /hey/i).first.visible?
+      end
+
+      verify "has sign up or login link" do
+        page.get_by_role("link", name: /sign up|sign in|log in/i).first.visible?
+      end
+    end
+  end
+end

data/examples/checks/hey/pricing.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Checks
+  class HeyPricing < Checkset::Check
+    description "Verifies hey.com pricing page loads with plan details"
+    tags :hey, :pricing
+
+    def call
+      visit "/"
+
+      step "navigate to pricing" do
+        page.get_by_role("link", name: /pricing/i).first.click
+      end
+
+      verify "pricing page loaded" do
+        page.url.include?("pricing") || page.title.downcase.include?("pricing")
+      end
+
+      verify "shows a price" do
+        page.locator("text=/\\$\\d+/").first.visible?
+      end
+    end
+  end
+end

data/examples/checks/homepage_loads.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Checks
+  class HomepageLoads < Checkset::Check
+    description "Verifies the target homepage loads without errors"
+
+    def call
+      visit "/"
+
+      verify "page returns a response" do
+        page.title.is_a?(String)
+      end
+
+      verify "page has visible content" do
+        page.locator("body").text_content.strip.length > 0
+      end
+
+      verify "no error status in title" do
+        !page.title.match?(/404|500|502|503|error/i)
+      end
+    end
+  end
+end

data/examples/checks/once/homepage.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Checks
+  class OnceHomepage < Checkset::Check
+    description "Verifies once.com homepage loads correctly"
+    tags :once, :homepage
+
+    def call
+      visit "/"
+
+      verify "page has title" do
+        page.title.length > 0
+      end
+
+      verify "has visible text content" do
+        page.locator("body").text_content.length > 50
+      end
+
+      verify "shows product offerings" do
+        page.get_by_role("link").count > 3
+      end
+    end
+  end
+end

data/examples/checks/ruby/ruby_lang.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Checks
+  class RubyLang < Checkset::Check
+    description "Verifies ruby-lang.org loads and has key content"
+    tags :ruby, :community
+
+    def call
+      visit "/"
+
+      verify "page title mentions Ruby" do
+        page.title.include?("Ruby")
+      end
+
+      verify "shows Ruby version info" do
+        page.locator("text=/Ruby \\d+\\.\\d+/").first.visible?
+      end
+
+      step "navigate to downloads" do
+        page.get_by_role("link", name: /download/i).first.click
+      end
+
+      verify "downloads page loaded" do
+        page.url.include?("download")
+      end
+    end
+  end
+end