reqcord 0.1.0 → 0.1.2

data/docs/configuration.md

@@ -0,0 +1,319 @@
+# `reqcord.yml` reference
+
+Reqcord reads one file, `reqcord.yml`, from the application root (`Rails.root`).
+`bin/rails reqcord:init` writes a starting point; every key is optional.
+
+```yaml
+version: 1
+
+test:
+  framework: minitest
+  paths:
+    - test/integration
+    - test/api
+
+routes:
+  prefix: /api
+
+output:
+  directory: docs/api
+  include_uncovered: false
+
+exporters:
+  - curl
+  - markdown
+  - postman
+
+variables:
+  base_url: http://localhost:3000
+
+sanitize:
+  headers:
+    Authorization: "Bearer {{token}}"
+    X-Api-Key: "{{api_key}}"
+  body:
+    password: "{{password}}"
+```
+
+## How values are resolved
+
+```text
+environment variable  >  reqcord.yml  >  Reqcord default
+```
+
+The file is **deep-merged** over the defaults: setting `sanitize.headers.X-Account-Id`
+keeps the default `Authorization` and `X-Api-Key` entries. Lists are replaced,
+not merged: an `exporters:` list is used exactly as written.
+
+The file must be a YAML mapping; anything else raises `Reqcord::ConfigurationError`
+before any test runs. YAML aliases are disabled.
+
+| Variable | Overrides |
+| --- | --- |
+| `REQCORD_TEST_FRAMEWORK` | `test.framework` |
+| `REQCORD_TEST_COMMAND` | `test.command` (and therefore `test.paths`) |
+| `REQCORD_OUTPUT` | `output.directory` |
+| `REQCORD_BASE_URL` | `variables.base_url` |
+| `RESOURCE`, `VERSION` | run-time filters, see [Filtering a run](#filtering-a-run) |
+
+`REQCORD_CAPTURE` and `REQCORD_CAPTURE_FILE` are set **by** Reqcord on the test
+process it starts; do not set them yourself. Without them an ordinary
+`bin/rails test` patches nothing and writes nothing.
+
+---
+
+## `version`
+
+```yaml
+version: 1
+```
+
+The configuration format version. Informational today; keep it at `1`.
+
+---
+
+## `test`
+
+Where the requests come from. `reqcord:generate` runs your suite in a
+subprocess with capture enabled; every request the integration tests make
+is captured, so point this at the tests that exercise the API.
+
+### `test.framework`
+
+`minitest` (default) or `rspec`. Decides two things: how the suite is run
+when only `paths` is given, and how captured examples are named — from the
+Minitest test method (`test "creates customer"`) or the RSpec example
+(`it "creates customer"`). Any other value is rejected before the run.
+
+### `test.paths`
+
+Directories, files or globs. Reqcord builds the runner:
+
+| Framework | Project has `bin/rails` | Command |
+| --- | --- | --- |
+| minitest | yes | `bin/rails test <paths…>` |
+| minitest | no | `ruby -Itest -e '<require each file>' <every *_test.rb beneath the paths>` |
+| rspec | — | `rspec <paths…>` |
+
+A directory means every `*_test.rb` (or `*_spec.rb`) beneath it; a glob such as
+`test/api/*_test.rb` is expanded by Reqcord.
+
+```yaml
+test:
+  framework: minitest
+  paths:
+    - test/api
+    - test/controllers/api
+    - test/integration
+```
+
+### `test.command`
+
+Spell the command out yourself. It wins over `paths`.
+
+```yaml
+test:
+  command: bin/rails test test/api test/integration/orders_test.rb
+```
+
+The command is split into words and executed **without a shell**: no pipes,
+`&&`, environment assignments or quoting tricks. Globs in the arguments are
+expanded by Reqcord (`test/api/*_test.rb` works). An empty string counts as
+absent.
+
+With neither `paths` nor `command`, Reqcord runs `bin/rails test`.
+
+Whatever runs must exit successfully. A failing suite aborts the run with
+`Reqcord::GenerationError` and nothing is written — documentation is only
+generated from a green suite.
+
+---
+
+## `routes`
+
+The documented surface is the application's route table (including mounted
+engines), not the captured traffic: every matching route becomes an endpoint,
+and captures are attached to it.
+
+### `routes.prefix`
+
+Default `/api`. Only routes whose path starts with the prefix are documented.
+For a mounted engine the prefix is matched against the full path
+(`/api/billing/invoices`). Set it to an empty string to document every route.
+
+```yaml
+routes:
+  prefix: /api/v2
+```
+
+Rails' own routes (`rails/…`, Active Storage, Action Mailbox, Turbo) are always
+left out. `redirect(...)` routes and plain Rack mounts cannot be documented from
+a test; they are counted as *skipped* in the report rather than dropped.
+
+### Filtering a run
+
+Filters are given per run, not in the file:
+
+```bash
+bin/rails reqcord:generate RESOURCE=customers
+bin/rails reqcord:generate RESOURCE=customers,cart
+bin/rails reqcord:generate VERSION=v2
+bin/rails reqcord:generate RESOURCE=orders VERSION=v1
+```
+
+`RESOURCE` matches the controller's last segment (`customers`), its singular
+(`cart` for `CartsController`) or the full controller path
+(`api/v2/customers`). `VERSION` matches a `v<number>` segment in the controller
+path or the route path.
+
+---
+
+## `output`
+
+### `output.directory`
+
+Default `docs/api`, relative to the application root (an absolute path is used
+as is). Override per run with `REQCORD_OUTPUT`. A run writes:
+
+```text
+<directory>/
+├── dataset.json          the canonical dataset every exporter reads
+├── README.md             index of resources and endpoints        (markdown)
+├── api/v1/customers/     one directory per controller path       (markdown)
+│   ├── index.md
+│   └── create.md
+├── curl/api/v1/customers/create.sh                                 (curl)
+└── postman/collection.json                                         (postman)
+```
+
+Directories follow the controller path, so `admin/customers` and
+`api/v1/customers` never collide.
+
+### `output.include_uncovered`
+
+Default `false`. A route no test reached with a `2xx` response is listed in
+the index under *No Successful Request Captured* and in `dataset.json` under
+`uncovered_routes`, but gets no page of its own. Set `true` to write those
+pages as well, each carrying a note that nothing was captured.
+
+---
+
+## `exporters`
+
+Which outputs to write. Default: all three.
+
+| Name | Writes |
+| --- | --- |
+| `markdown` | `README.md` plus one page per endpoint: headers, typed parameter tables, example request, cURL, one example and field table per response status |
+| `curl` | one runnable `.sh` per endpoint under `curl/`, built from the successful captured request |
+| `postman` | `postman/collection.json`, a Postman Collection v2.1 — folders per controller namespace, one request per endpoint, every captured status saved as an example, placeholders as collection variables, bearer auth at collection level. Hoppscotch imports the same file |
+
+`dataset.json` is always written. An unknown name raises
+`Reqcord::ConfigurationError` before any test runs.
+
+---
+
+## `variables`
+
+### `variables.base_url`
+
+Default `http://localhost:3000`. The host in every generated cURL and the value
+of the Postman `base_url` collection variable. A trailing slash is dropped.
+Override per run with `REQCORD_BASE_URL`.
+
+Other keys under `variables` are read but not used by the built-in exporters.
+
+---
+
+## `sanitize`
+
+Captured tests carry real credentials. Sanitization runs on every exchange
+before anything reaches the dataset, so generated files never contain them.
+
+### `sanitize.headers`
+
+Header name → replacement, matched case-insensitively, replaced **verbatim**:
+
+```yaml
+sanitize:
+  headers:
+    Authorization: "Bearer {{token}}"
+    X-Api-Key: "{{api_key}}"
+    X-Account-Id: "{{account_id}}"
+```
+
+Defaults: `Authorization: "Bearer {{token}}"` and `X-Api-Key: "{{api_key}}"`.
+
+Some headers are redacted **whether configured or not**: `Authorization`,
+`Proxy-Authorization`, `Cookie`, `Set-Cookie`, `X-Api-Key`, `X-Auth-Token`,
+`X-Csrf-Token`. Without a configured replacement the value becomes a
+placeholder named after the header (`{{cookie}}`, `{{x_auth_token}}`), and a
+`Bearer`, `Token` or `Basic` scheme is kept so the cURL stays runnable
+(`Basic {{authorization}}`).
+
+Two more things happen to headers, on requests and responses alike:
+
+* transport noise is dropped: `Host`, `User-Agent`, `Connection`, `Version`,
+  `Remote-Addr`, `Accept-Encoding`, `Cache-Control`, `Content-Length`, `Date`,
+  `ETag`, `Server-Timing`, `Transfer-Encoding`, `Vary`, `X-Request-Id`,
+  `X-Runtime`, `Referrer-Policy` and the `X-*-Options` security headers;
+* headers with an empty value are dropped.
+
+### `sanitize.body`
+
+Body key → replacement, matched case-insensitively **at any depth**, in
+request and response bodies alike. The whole value under the key is replaced,
+whatever its type.
+
+```yaml
+sanitize:
+  body:
+    password: "{{password}}"
+    payment_url: "{{payment_url}}"
+```
+
+Defaults: `password`, `password_confirmation`, `token`, `access_token`,
+`refresh_token`, `api_key`, `secret`, `client_secret` (→ `{{password}}`,
+`{{token}}`, `{{api_key}}`, `{{secret}}`).
+
+A sanitized value is what the documentation shows, so it also shapes the
+parameter tables: two logins whose passwords both became `{{password}}` are two
+examples of the same request, and the one that succeeded is the one used for
+the cURL.
+
+### Placeholders
+
+Every `{{name}}` you write is kept as-is in Markdown and cURL, listed in the
+index under *Placeholders*, and declared as a variable in the Postman
+collection — Postman's variable syntax is the same, so the collection runs as
+soon as `base_url` and `token` are filled in.
+
+---
+
+## Rake tasks
+
+| Task | Does |
+| --- | --- |
+| `bin/rails reqcord:init` | writes `reqcord.yml` (never overwrites) and creates `docs/api/` |
+| `bin/rails reqcord:generate` | collects routes, runs the suite with capture, writes every exporter, prints the report |
+| `bin/rails reqcord:routes` | lists the routes the current `prefix` (and `RESOURCE` / `VERSION`) would document |
+
+Every `generate` run ends with a reconciliation of the whole route table:
+
+```text
+[reqcord] captured 92 request(s), 87 matched a documented route
+[reqcord] captured a successful 2xx request for 14 of 16 endpoint(s)
+[reqcord] routes: 16 = 14 documented + 2 uncovered + 0 skipped
+```
+
+Each route is in exactly one bucket; a captured request that matched no
+documented route is listed, never dropped silently.
+
+## When the output looks thin
+
+| Symptom | Usual cause |
+| --- | --- |
+| `no request was captured` | the gem is not in the `:test` group of the Gemfile, or `test.paths` / `test.command` runs no integration tests |
+| many requests captured, few matched | `routes.prefix` does not cover them — the unmatched paths are printed |
+| routes documented but few covered | the tests that exercise them are not in `test.paths` (a `2xx` from a test is what makes an endpoint documented) |
+| `Test suite failed while generating` | the suite is red; fix the tests, documentation is only generated from a passing run |

data/examples/reqcord.yml

@@ -0,0 +1,58 @@
+# Reqcord configuration. Copy this file to your application root as
+# `reqcord.yml`, or generate it with `bin/rails reqcord:init`.
+#
+# Precedence: environment variables > reqcord.yml > Reqcord defaults.
+# Every key is documented in docs/configuration.md.
+
+version: 1
+
+test:
+  # minitest or rspec.
+  framework: minitest
+
+  # Where the tests that exercise your API live. Reqcord picks the runner:
+  # `bin/rails test <paths>` in a Rails app, `rspec <paths>` for request
+  # specs, a plain Ruby runner when there is no bin/rails.
+  paths:
+    - test/integration
+
+  # Or spell the command out yourself; it wins over `paths`. Globs expand.
+  # command: bin/rails test test/integration test/api
+
+routes:
+  # Only routes under this prefix are documented. Filter a single run with
+  # RESOURCE=customers,cart or VERSION=v2 instead of editing this file.
+  prefix: /api
+
+output:
+  # Relative to the application root. Override per run with REQCORD_OUTPUT.
+  directory: docs/api
+
+  # Routes no test reached with a 2xx are listed in the index either way;
+  # `true` also writes a page for each of them.
+  include_uncovered: false
+
+# markdown: pages under docs/api, curl: one runnable .sh per endpoint,
+# postman: postman/collection.json (import into Postman or Hoppscotch).
+exporters:
+  - curl
+  - markdown
+  - postman
+
+variables:
+  # Used as the host in every generated cURL and as the Postman `base_url`
+  # variable. Override with REQCORD_BASE_URL.
+  base_url: http://localhost:3000
+
+sanitize:
+  # Captured header values are replaced with these, verbatim. Authorization,
+  # Cookie and X-Api-Key are always redacted, configured here or not.
+  headers:
+    Authorization: "Bearer {{token}}"
+    X-Api-Key: "{{api_key}}"
+    X-Account-Id: "{{account_id}}"
+
+  # Body keys, matched at any depth, in requests and responses alike.
+  body:
+    password: "{{password}}"
+    access_token: "{{token}}"

data/gemfiles/rails_7.1.gemfile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# CI: BUNDLE_GEMFILE=gemfiles/rails_7.1.gemfile bundle exec rake test
+
+source "https://rubygems.org"
+
+gemspec path: ".."
+
+gem "railties", "~> 7.1.0"
+gem "actionpack", "~> 7.1.0"
+
+# activesupport calls JSON.parse(json, options); json 3.0 dropped that signature.
+gem "json", "< 3"

data/gemfiles/rails_7.2.gemfile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# CI: BUNDLE_GEMFILE=gemfiles/rails_7.2.gemfile bundle exec rake test
+
+source "https://rubygems.org"
+
+gemspec path: ".."
+
+gem "railties", "~> 7.2.0"
+gem "actionpack", "~> 7.2.0"
+
+# activesupport calls JSON.parse(json, options); json 3.0 dropped that signature.
+gem "json", "< 3"

data/gemfiles/rails_8.0.gemfile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# CI: BUNDLE_GEMFILE=gemfiles/rails_8.0.gemfile bundle exec rake test
+
+source "https://rubygems.org"
+
+gemspec path: ".."
+
+gem "railties", "~> 8.0.0"
+gem "actionpack", "~> 8.0.0"
+
+# activesupport calls JSON.parse(json, options); json 3.0 dropped that signature.
+gem "json", "< 3"

data/gemfiles/rails_8.1.gemfile

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+# CI: BUNDLE_GEMFILE=gemfiles/rails_8.1.gemfile bundle exec rake test
+
+source "https://rubygems.org"
+
+gemspec path: ".."
+
+gem "railties", "~> 8.1.0"
+gem "actionpack", "~> 8.1.0"
+
+# activesupport calls JSON.parse(json, options); json 3.0 dropped that signature.
+gem "json", "< 3"

data/lib/reqcord/capture/collector.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Reqcord
+  module Capture
+    # Appends one JSON line per exchange. The lock keeps parallel test workers
+    # from interleaving partial lines in the same file.
+    class Collector
+      class << self
+        def write(exchange)
+          return unless Capture.enabled?
+
+          path = Capture.capture_file
+
+          FileUtils.mkdir_p(File.dirname(path))
+
+          File.open(path, "a") do |file|
+            file.flock(File::LOCK_EX)
+
+            file.puts(
+              JSON.generate(exchange)
+            )
+
+            file.flush
+          ensure
+            file.flock(File::LOCK_UN)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/reqcord/capture/integration_patch.rb

@@ -0,0 +1,209 @@
+# frozen_string_literal: true
+
+module Reqcord
+  module Capture
+    # Captures exactly what the integration test passed to Rails. Reqcord does
+    # not reconstruct the payload from controller params: the test call is the
+    # source of truth for generated cURL examples.
+    module IntegrationPatch
+      RAILS_DEFAULT_ACCEPT =
+        "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
+
+      def process(method, path, **kwargs)
+        raw_params = kwargs[:params]
+        raw_headers = kwargs[:headers]
+        request_format = kwargs[:as]
+
+        result = super
+
+        if Reqcord::Capture.enabled?
+          Reqcord::Capture::Collector.write(
+            reqcord_exchange(
+              method: method,
+              path: path,
+              params: raw_params,
+              input_headers: raw_headers,
+              request_format: request_format
+            )
+          )
+        end
+
+        result
+      end
+
+      private
+
+      def reqcord_exchange(method:, path:, params:, input_headers:, request_format:)
+        verb = method.to_s.upcase
+        query_params, body = reqcord_split_params(verb, path, params)
+
+        {
+          request: {
+            method: verb,
+            path: reqcord_request_path(path),
+            path_params: reqcord_path_parameters,
+            query_params: query_params,
+            headers: reqcord_request_headers(input_headers, body, request_format),
+            body: body,
+            content_type: reqcord_content_type(body, request_format)
+          },
+          response: {
+            status: response&.status,
+            headers: reqcord_response_headers,
+            body: reqcord_response_body,
+            content_type: response&.media_type
+          },
+          source: TestContext.current
+        }
+      end
+
+      # Keep the concrete test URL (/customers/42), not the route pattern
+      # (/customers/:id). A generated cURL command must be runnable as-is.
+      def reqcord_request_path(path)
+        URI.parse(path.to_s).path
+      rescue URI::InvalidURIError
+        path.to_s.split("?").first
+      end
+
+      def reqcord_path_parameters
+        return {} unless request
+
+        request.path_parameters
+               .except(:controller, :action, :format)
+               .transform_keys(&:to_s)
+      end
+
+      # Rails integration tests use params as query parameters for GET/HEAD and
+      # as the request payload for mutating verbs. Read the original test input
+      # instead of trying to reverse-engineer it from ActionDispatch afterwards.
+      def reqcord_split_params(verb, path, params)
+        explicit_query = reqcord_query_from_path(path)
+        normalized = reqcord_normalize_value(params)
+
+        if %w[GET HEAD].include?(verb)
+          query = explicit_query
+          query = reqcord_deep_merge(query, normalized) if normalized.is_a?(Hash)
+          [query, nil]
+        else
+          [explicit_query, reqcord_meaningful?(normalized) ? normalized : nil]
+        end
+      end
+
+      def reqcord_query_from_path(path)
+        uri = URI.parse(path.to_s)
+        return {} if uri.query.nil? || uri.query.empty?
+
+        Rack::Utils.parse_nested_query(uri.query)
+      rescue URI::InvalidURIError
+        {}
+      end
+
+      def reqcord_request_headers(input_headers, body, request_format)
+        result = {}
+
+        (input_headers || {}).each do |key, value|
+          result[reqcord_header_name(key)] = reqcord_normalize_value(value)
+        end
+
+        result.delete("X-Http-Method-Override")
+
+        if body
+          content_type = reqcord_content_type(body, request_format)
+          result["Content-Type"] ||= content_type if content_type
+        end
+
+        if request
+          accept = request.headers["Accept"]
+          result["Accept"] ||= accept if accept.present? && accept != RAILS_DEFAULT_ACCEPT
+        end
+
+        result
+      end
+
+      def reqcord_content_type(body, request_format)
+        return nil unless body
+
+        return "application/json" if request_format.to_s == "json"
+
+        request&.content_type
+      end
+
+      def reqcord_header_name(key)
+        name = key.to_s
+
+        return name if name.include?("-")
+
+        name.delete_prefix("HTTP_")
+            .split("_")
+            .map { |part| part.empty? ? part : part.capitalize }
+            .join("-")
+      end
+
+      def reqcord_response_headers
+        return {} unless response
+
+        response.headers.to_h
+      end
+
+      def reqcord_response_body
+        return nil unless response
+
+        body = response.body
+        return nil if body.nil? || body.empty?
+
+        if response.media_type == "application/json"
+          JSON.parse(body)
+        else
+          body
+        end
+      rescue JSON::ParserError
+        body
+      end
+
+      def reqcord_normalize_value(value)
+        case value
+        when nil, true, false, Numeric, String
+          value
+        when Symbol
+          value.to_s
+        when Hash
+          value.each_with_object({}) do |(key, nested), result|
+            result[key.to_s] = reqcord_normalize_value(nested)
+          end
+        when Array
+          value.map { |item| reqcord_normalize_value(item) }
+        else
+          if value.respond_to?(:to_unsafe_h)
+            reqcord_normalize_value(value.to_unsafe_h)
+          elsif value.respond_to?(:to_h)
+            reqcord_normalize_value(value.to_h)
+          elsif value.respond_to?(:as_json)
+            reqcord_normalize_value(value.as_json)
+          else
+            value.to_s
+          end
+        end
+      end
+
+      def reqcord_deep_merge(left, right)
+        left.merge(right) do |_key, old_value, new_value|
+          if old_value.is_a?(Hash) && new_value.is_a?(Hash)
+            reqcord_deep_merge(old_value, new_value)
+          else
+            new_value
+          end
+        end
+      end
+
+      def reqcord_meaningful?(value)
+        case value
+        when nil then false
+        when Hash then value.any? { |_key, nested| reqcord_meaningful?(nested) }
+        when Array then value.any? { |item| reqcord_meaningful?(item) }
+        when String then !value.empty?
+        else true
+        end
+      end
+    end
+  end
+end