reqcord 0.2.0 → 0.3.0

data/docs/getting-started.md

@@ -0,0 +1,186 @@
+# Getting started
+
+Reqcord documents an API from the integration tests that already exercise it.
+Nothing in the tests changes; Reqcord runs them, watches the requests they
+make, and writes the documentation from what it saw.
+
+## 1. Install
+
+```ruby
+# Gemfile
+group :development, :test do
+  gem "reqcord"
+end
+```
+
+```bash
+bundle install
+```
+
+Both groups matter: `reqcord:generate` runs in development, the capture runs
+inside the test process.
+
+## 2. Create the configuration
+
+```bash
+bin/rails reqcord:init
+```
+
+This writes an annotated `reqcord.yml` (never overwriting one that exists)
+and creates `docs/api/`. The two keys worth checking before the first run:
+
+```yaml
+test:
+  framework: minitest     # or: rspec
+  paths:
+    - test/integration    # where the tests that call the API live
+
+routes:
+  prefix: /api            # only routes under this prefix are documented
+```
+
+Every key is described in [configuration.md](configuration.md).
+
+## 3. Generate
+
+```bash
+bin/rails reqcord:generate
+```
+
+Reqcord collects the route table, runs the tests under `test.paths` in a
+subprocess with capture enabled, sanitizes what was captured and writes every
+exporter. The run ends with a reconciliation of the whole route table:
+
+```text
+[reqcord] captured 87 request(s), 85 matched a documented route
+[reqcord] captured a successful 2xx request for 15 of 16 endpoint(s)
+[reqcord] routes: 18 = 15 documented + 1 uncovered + 2 skipped
+[reqcord] skipped 2 route(s) that cannot be documented: 1 redirect, 1 mount
+```
+
+Every route is in exactly one bucket:
+
+| Bucket | Meaning |
+| --- | --- |
+| documented | a test got a `2xx` from it — it has a page, a cURL script, a Postman request and an OpenAPI operation |
+| uncovered | the route exists, but no test reached it with a `2xx`; listed in the index and in `dataset.json` |
+| skipped | `redirect(...)` routes and plain Rack mounts, which no test can document |
+
+A red suite still produces documentation from what its green tests captured
+(see [`test.strict`](configuration.md#teststrict) to abort instead).
+
+## 4. Read the output
+
+```text
+docs/api/
+├── README.md                      index: placeholders, resources, endpoints, gaps
+├── dataset.json                   the canonical dataset every exporter reads
+├── api/v2/customers/
+│   ├── index.md                   one page per controller path
+│   └── create.md                  one page per endpoint
+├── curl/api/v2/customers/
+│   └── create.sh                  runnable, sanitized
+├── postman/collection.json        Postman v2.1, also for Hoppscotch
+└── openapi/openapi.json           OpenAPI 3.1
+```
+
+An endpoint page looks like this:
+
+````markdown
+# Create Customer
+
+`POST /api/v2/customers`
+
+## Headers
+
+| Header | Value |
+| --- | --- |
+| Authorization | `Bearer {{token}}` |
+| Content-Type | `application/json` |
+
+## Body Parameters
+
+| Field | Type | Required | Values |
+| --- | --- | --- | --- |
+| `customer.name` | string | yes | `"John Doe"` |
+| `customer.email` | string | yes | `"john@example.com"` |
+| `customer.status` | string | yes | `"active"` \| `"passive"` |
+
+## cURL
+
+```bash
+curl --request POST \
+  --url "http://localhost:3000/api/v2/customers" \
+  --header "Authorization: Bearer {{token}}" \
+  --header "Content-Type: application/json" \
+  --data '{"customer":{"name":"John Doe","email":"john@example.com","status":"active"}}'
+```
+
+## Responses
+
+### 201 Created
+### 401 Unauthorized
+### 422 Unprocessable Content
+````
+
+The parameter table comes from the requests the application **accepted**: two
+tests sent `"active"` and `"passive"`, a third sent `"inactive"` and got a
+`422`, so the page lists the two values that work and keeps the rejection as a
+response example. [capture.md](capture.md) explains the rules.
+
+## 5. Browse it in the app (optional)
+
+```ruby
+# config/routes.rb
+mount Reqcord::Web => "/api-docs" if Rails.env.development?
+```
+
+`/api-docs` renders the OpenAPI document with Scalar and serves every other
+generated file. See [web.md](web.md).
+
+## 6. Keep it current in CI
+
+The output is a function of the routes and the tests, so it can be checked
+like generated code:
+
+```bash
+bin/rails reqcord:check
+```
+
+It regenerates into a scratch directory, compares with `docs/api` file by
+file and exits `1` with a git-status style list when they differ:
+
+```text
+Reqcord: /app/docs/api is out of date.
+
+  M api/v2/customers/create.md
+  A api/v2/customers/destroy.md
+  D curl/api/v2/orders/cancel.sh
+
+Run `bin/rails reqcord:generate` and commit the result.
+```
+
+```yaml
+# .github/workflows/ci.yml
+- run: bin/rails reqcord:check
+```
+
+A PR that changes a request or a response without regenerating the docs goes
+red; `bin/rails reqcord:generate` and a commit fix it.
+
+## Narrowing a run
+
+```bash
+bin/rails reqcord:generate RESOURCE=customers
+bin/rails reqcord:generate RESOURCE=customers,cart
+bin/rails reqcord:generate VERSION=v2
+bin/rails reqcord:routes                  # what would be documented, without running anything
+```
+
+## Next
+
+* [configuration.md](configuration.md) — every key in `reqcord.yml`
+* [exporters.md](exporters.md) — what each output format contains
+* [route-coverage.md](route-coverage.md) — how the route table becomes endpoints
+* [troubleshooting.md](troubleshooting.md) — when the output looks thin
+* [`examples/`](../examples) — four runnable applications with their generated docs committed

data/docs/route-coverage.md

@@ -0,0 +1,91 @@
+# Route coverage
+
+The documented surface is the application's **route table**, not the traffic
+the tests happened to produce. Every route under `routes.prefix` becomes an
+endpoint; captured requests are attached to it. That is what lets the run end
+with an accounting nothing can slip through:
+
+```text
+routes = documented + uncovered + skipped
+```
+
+| Bucket | Rule |
+| --- | --- |
+| documented | at least one captured request returned a `2xx` |
+| uncovered | no `2xx` was captured; the route is listed under *No Successful Request Captured* and in `dataset.json` → `uncovered_routes` |
+| skipped | `redirect(...)` routes and plain Rack mounts: nothing a test could document; counted by reason in the report |
+
+A captured request that matches no documented route is reported with its
+path (the first five), never dropped silently — usually a sign that
+`routes.prefix` is too narrow.
+
+## What becomes an endpoint
+
+| Route | Documented as |
+| --- | --- |
+| `resources :customers` | one endpoint per action; `index` is titled *List Customers*, `show` *Get Customer*, `create`, `update`, `destroy`, `new`, `edit` likewise |
+| `post :activate, on: :member` | *Activate Customer* |
+| `resource :cart` | `GET /cart` (*Get Cart*), `PATCH /cart` (also `PUT`) — one record, no id |
+| `post "checkout", on: :collection` under `resource :cart` | *Checkout Cart* |
+| `post "auth/login", to: "auth#login"` | *Login* — the action is the page, the controller only where it lives |
+| `root to: "home#index"` | `GET /`, titled *Home* |
+| `match "/echo", via: [:get, :post]` | `GET /echo` and `POST /echo`, two endpoints |
+| `match "/anything", via: :all` | one endpoint per verb the tests actually used; one `ANY` entry when none did |
+| `patch` + `put` for `update` | folded into one endpoint, `also_methods: ["PUT"]`; the captured verb wins, `PATCH` preferred |
+| `get "/items(/:id)"` | one endpoint; the optional segment is kept in the path, `id` is a path parameter |
+| `get "/files/*path"` | `path` as a path parameter |
+| `resources :items, param: :sku` | `sku` as the path parameter |
+| `namespace :admin { resources :customers }` | `admin/customers`, a separate resource from `api/v2/customers` |
+| `mount Billing::Engine => "/billing"` | the engine's own routes, walked under `/billing`; the prefix filter sees the full path |
+| `scope module: :v2` / `namespace :v2` | `v2` detected as the API version (`VERSION=v2` filters on it) |
+
+Rails' own routes (`rails/…`, Active Storage, Action Mailbox, Turbo — anything
+flagged `internal`) are always left out.
+
+## Grouping
+
+Endpoints are grouped by the **full controller path**: `api/v2/customers`,
+`admin/customers`, `billing/invoices`. That path becomes the Markdown
+directory, the Postman folder chain (`Api › V2 › Customers`) and the OpenAPI
+tag, so two controllers with the same last segment never collide.
+
+## Naming
+
+Endpoint names come from the action and the resource, not from the test
+names — tests name the *examples*. The table above shows the patterns; a
+custom action on a plural resource is `<Action> <Resource>` (*Cancel Order*),
+on a singular resource `<Action> <Singular>` (*Checkout Cart*), and a custom
+action whose name is in the path of a singular controller is just the action
+(*Login*).
+
+## Filtering a run
+
+```bash
+bin/rails reqcord:generate RESOURCE=customers
+bin/rails reqcord:generate RESOURCE=customers,cart
+bin/rails reqcord:generate VERSION=v2
+bin/rails reqcord:generate RESOURCE=orders VERSION=v1
+```
+
+`RESOURCE` matches the controller's last segment (`customers`), its singular
+(`cart` for `CartsController`) or the full controller path
+(`api/v2/customers`). `VERSION` matches a `v<number>` segment in the
+controller path or the route path.
+
+`bin/rails reqcord:routes` prints what the current prefix and filters would
+document, without running any test.
+
+## Prefixes
+
+```yaml
+routes:
+  prefix: /api            # everything under /api, all versions
+```
+
+```yaml
+routes:
+  prefix: [/v1, /v2, /partner]   # APIs whose versions do not share a root
+```
+
+An empty prefix documents every route. For a mounted engine the prefix is
+matched against the full path (`/api/billing/invoices`).

data/docs/troubleshooting.md

@@ -0,0 +1,113 @@
+# Troubleshooting
+
+The generate report is the first place to look; it says how many requests
+were captured, how many matched a route, and where every route ended up.
+
+```text
+[reqcord] captured 87 request(s), 85 matched a documented route
+[reqcord] captured a successful 2xx request for 15 of 16 endpoint(s)
+[reqcord] routes: 18 = 15 documented + 1 uncovered + 2 skipped
+```
+
+## `no request was captured`
+
+Nothing reached the capture file. In order of likelihood:
+
+1. **The gem is not loaded in the test process.** It must be in the `:test`
+   group of the Gemfile, not only `:development`.
+2. **The run did not include the API tests.** `test.paths` (or
+   `test.command`) points somewhere else. Name the directories that hold the
+   integration tests / request specs:
+   ```yaml
+   test:
+     paths:
+       - test/integration
+       - test/controllers/api
+   ```
+3. **The tests are not integration tests.** Reqcord patches
+   `ActionDispatch::Integration::Session`; unit tests and controller tests
+   that stub the request never go through it.
+4. **The framework is wrong.** `test.framework: rspec` for request specs, or
+   the examples are not named and the runner is not `rspec`.
+
+## Many requests captured, few matched a route
+
+`routes.prefix` does not cover them. The report prints the first unmatched
+paths:
+
+```text
+[reqcord] 12 path(s) matched no documented route, for example:
+[reqcord]   /v1/customers
+```
+
+Widen the prefix or give a list (`prefix: [/api, /v1]`). `bin/rails
+reqcord:routes` shows what the current prefix documents.
+
+## Routes documented, few covered
+
+An endpoint is *documented* only when a test got a `2xx` from it. Routes that
+tests reach only with `401`/`422` stay uncovered — add a happy-path test, or
+check that the test which does succeed is under `test.paths`.
+
+## The suite is red
+
+By default Reqcord still writes the documentation and warns:
+
+```text
+[reqcord] test run exited with status 1; documenting what it captured anyway
+[reqcord] the test run failed: routes exercised only by failing tests are listed as uncovered
+```
+
+A failed assertion after a `2xx` still counts as a documented request. If you
+would rather get nothing from a red build, set `test.strict: true` (or
+`REQCORD_STRICT=1`); the run then aborts with `Reqcord::GenerationError`.
+
+## `unknown exporter "…"`
+
+The `exporters:` list is used exactly as written (lists are not merged with
+the defaults). Valid names: `curl`, `markdown`, `postman`, `openapi`.
+
+## The cURL does not run
+
+* `{{token}}`, `{{base_url}}` and other placeholders are meant to be replaced
+  before running — they are the documentation's variables.
+* The host is `variables.base_url` (`REQCORD_BASE_URL` per run).
+* The command replays the request a test made; if the test relied on
+  fixtures (an `id` that exists only in the test database) the real server
+  needs equivalent data.
+
+## `reqcord:check` fails but nothing changed
+
+The comparison is byte for byte, so the committed docs must come from the
+same code and configuration:
+
+* a different Reqcord version, `reqcord.yml`, `REQCORD_BASE_URL` or
+  `REQCORD_OUTPUT` between the commit and CI;
+* a `RESOURCE` / `VERSION` filter on one side only;
+* files added by hand under `docs/api` (listed as `D`), or stale files from
+  an endpoint that no longer exists — `generate` never deletes.
+
+Run `bin/rails reqcord:generate` locally with the same settings and commit
+what changes. Test order does not matter: the output is ordered by test
+file and line, not by execution.
+
+## `/api-docs` shows "No documentation generated yet"
+
+`Reqcord::Web` only serves files; run `bin/rails reqcord:generate` first. If
+you generated into a different directory (`REQCORD_OUTPUT`), mount
+`Reqcord::Web.new(root: …)` for that directory, see [web.md](web.md).
+
+## A value is listed as a closed set, or is not
+
+The rules are in [capture.md → Inference](capture.md#inference): 2–6 distinct
+values, not an identifier, and either repeated across requests or every value
+a token. Free-text and generated values (`e-00056197`) are shown as one
+example on purpose.
+
+## A route is missing entirely
+
+* `redirect(...)` routes and plain Rack mounts are *skipped*, and counted as
+  such in the report.
+* Rails' internal routes are always left out.
+* Engine routes are walked under their mount path; the prefix filter sees
+  the full path (`/api/billing/invoices`).

data/docs/web.md

@@ -0,0 +1,66 @@
+# Reqcord::Web
+
+`Reqcord::Web` is a Rack application that serves the generated documentation
+from inside the Rails app, the way `Sidekiq::Web` does. It renders the
+OpenAPI document with [Scalar](https://scalar.com) and serves every other
+generated file.
+
+## Mount it
+
+```ruby
+# config/routes.rb
+mount Reqcord::Web => "/api-docs" if Rails.env.development?
+```
+
+Then `bin/rails reqcord:generate` and open `http://localhost:3000/api-docs`.
+
+## What it serves
+
+| Path | Content |
+| --- | --- |
+| `/api-docs` | the Scalar reference for `openapi/openapi.json` — searchable, with a *Try it* client |
+| `/api-docs/openapi/openapi.json` | the OpenAPI document |
+| `/api-docs/dataset.json` | the canonical dataset |
+| `/api-docs/postman/collection.json` | the Postman collection |
+| `/api-docs/README.md`, `/api-docs/api/v2/customers/create.md` … | the Markdown pages (`text/markdown`) |
+| `/api-docs/curl/api/v2/customers/create.sh` | the cURL scripts (`text/plain`) |
+
+The mount point can be anything; the page computes its links from it.
+
+Before the first generate, `/api-docs` shows a page saying what to run.
+
+## Rules
+
+* **Read-only.** Nothing is generated on request. Run `reqcord:generate`
+  after the tests change; the page picks up the new files on reload.
+* **Only the output directory.** Every request is resolved inside
+  `output.directory`; a path that escapes it (`..`, encoded or not) is a
+  `404`, as is anything that is not a regular file.
+* **Scalar comes from a CDN.** The page loads
+  `https://cdn.jsdelivr.net/npm/@scalar/api-reference`. With a strict
+  Content-Security-Policy, allow that host for `script-src`.
+
+## Serving a different directory
+
+`Reqcord::Web` reads `output.directory` from `reqcord.yml` (and
+`REQCORD_OUTPUT`). To serve another directory, mount an instance:
+
+```ruby
+mount Reqcord::Web.new(root: Rails.root.join("public/api-docs")) => "/api-docs"
+```
+
+## Production
+
+The files are static and contain only sanitized data, so mounting in
+production is a deployment decision rather than a Reqcord one. Guard the
+route like any internal page:
+
+```ruby
+authenticate :admin do
+  mount Reqcord::Web => "/api-docs"
+end
+```
+
+or leave the mount behind `Rails.env.development?` and publish the generated
+`docs/api` directory (or the OpenAPI file alone) through whatever already
+hosts your documentation.

data/lib/reqcord/capture/integration_patch.rb

@@ -120,12 +120,15 @@ module Reqcord
         result
       end
 
+      # The media type without parameters: a multipart boundary is noise that
+      # would make every upload request look different.
       def reqcord_content_type(body, request_format)
         return nil unless body
 
         return "application/json" if request_format.to_s == "json"
+        return nil unless request
 
-        request&.content_type
+        request.respond_to?(:media_type) ? request.media_type : request.content_type
       end
 
       def reqcord_header_name(key)
@@ -173,7 +176,11 @@ module Reqcord
         when Array
           value.map { |item| reqcord_normalize_value(item) }
         else
-          if value.respond_to?(:to_unsafe_h)
+          # Rack::Test::UploadedFile / ActionDispatch::Http::UploadedFile: the
+          # bytes stay out of the documentation, the name and type go in.
+          if value.respond_to?(:original_filename)
+            Reqcord::FileValue.marker(value.original_filename, value.respond_to?(:content_type) ? value.content_type : nil)
+          elsif value.respond_to?(:to_unsafe_h)
             reqcord_normalize_value(value.to_unsafe_h)
           elsif value.respond_to?(:to_h)
             reqcord_normalize_value(value.to_h)

data/lib/reqcord/check.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+require "tmpdir"
+
+module Reqcord
+  # `reqcord:check`: generates the documentation into a scratch directory and
+  # compares it with the committed one, file by file. The output is a
+  # function of the routes and the tests, so any difference means the docs
+  # in the repository are behind the code — the CI signal a "living
+  # documentation" promise needs.
+  class Check
+    Result = Struct.new(:added, :removed, :changed, keyword_init: true) do
+      def clean?
+        added.empty? && removed.empty? && changed.empty?
+      end
+
+      # git-status style, one line per file.
+      def lines
+        added.map { |path| "A #{path}" } +
+          removed.map { |path| "D #{path}" } +
+          changed.map { |path| "M #{path}" }
+      end
+    end
+
+    def self.call(resources: [], version: nil, configuration: Reqcord.configuration)
+      new(resources: resources, version: version, configuration: configuration).call
+    end
+
+    # `expected` is what the repository holds, `actual` what the code produces.
+    def self.compare(expected:, actual:)
+      expected_files = files_under(expected)
+      actual_files = files_under(actual)
+
+      Result.new(
+        added: (actual_files - expected_files).sort,
+        removed: (expected_files - actual_files).sort,
+        changed: (expected_files & actual_files).sort.reject do |path|
+          FileUtils.identical?(File.join(expected, path), File.join(actual, path))
+        end
+      )
+    end
+
+    def self.files_under(directory)
+      return [] unless File.directory?(directory)
+
+      Dir.glob("**/*", File::FNM_DOTMATCH, base: directory).select do |path|
+        File.file?(File.join(directory, path))
+      end
+    end
+
+    def initialize(resources:, version:, configuration:)
+      @resources = resources
+      @version = version
+      @configuration = configuration
+    end
+
+    def call
+      Dir.mktmpdir("reqcord-check") do |scratch|
+        Generator.call(
+          resources: resources,
+          version: version,
+          configuration: configuration.with_output_directory(scratch)
+        )
+
+        self.class.compare(expected: configuration.output_directory.to_s, actual: scratch)
+      end
+    end
+
+    private
+
+    attr_reader :resources, :version, :configuration
+  end
+end

data/lib/reqcord/configuration.rb

@@ -145,6 +145,8 @@ module Reqcord
     end
 
     def output_directory
+      return Pathname(@output_override) if @output_override
+
       value =
         ENV["REQCORD_OUTPUT"] ||
         data.dig("output", "directory") ||
@@ -153,6 +155,13 @@ module Reqcord
       root.join(value)
     end
 
+    # The same configuration writing somewhere else — how `reqcord:check`
+    # generates into a scratch directory while the configured one stays the
+    # thing to compare against.
+    def with_output_directory(path)
+      dup.tap { |copy| copy.instance_variable_set(:@output_override, path.to_s) }
+    end
+
     def include_uncovered?
       data.dig("output", "include_uncovered") == true
     end

data/lib/reqcord/dataset.rb

@@ -142,7 +142,6 @@ module Reqcord
     def to_h
       {
         schema_version: schema_version,
-        generated_at: Time.now.utc.iso8601,
         endpoints: curl_ready_endpoints.map(&:to_h),
         uncovered_routes: uncovered_endpoints.map do |endpoint|
           {

data/lib/reqcord/exporters/markdown.rb

@@ -234,12 +234,15 @@ module Reqcord
 
         lines = ["## Example Request", ""]
 
-        unless Renderers::Payload.json?(example) || example.content_type.to_s.empty?
+        if Renderers::Payload.multipart?(example)
+          lines << "Sent as `multipart/form-data`; file parts are shown by name, the cURL below attaches them with `--form`."
+          lines << ""
+        elsif !Renderers::Payload.json?(example) && !example.content_type.to_s.empty?
           lines << "Sent as `#{example.content_type}`; the cURL below carries it in that encoding."
           lines << ""
         end
 
-        lines.concat([*code_block(example.body, example.content_type), ""])
+        lines.concat([*code_block(Renderers::Payload.display_body(example), example.content_type), ""])
       end
 
       def curl_section(example)

data/lib/reqcord/exporters/openapi.rb

@@ -153,12 +153,22 @@ module Reqcord
       def request_body(endpoint, example)
         return nil if endpoint.body_schema.empty?
 
-        content_type = Renderers::Payload.json?(example) ? "application/json" : "application/x-www-form-urlencoded"
+        content_type =
+          if Renderers::Payload.multipart?(example)
+            "multipart/form-data"
+          elsif Renderers::Payload.json?(example)
+            "application/json"
+          else
+            "application/x-www-form-urlencoded"
+          end
+
+        # A file part is shown by name in the example; the bytes are not data.
+        body = Renderers::Payload.display_body(example) { |file| FileValue.filename(file) }
 
         {
           required: true,
           content: {
-            content_type => { schema: json_schema(endpoint.body_schema), example: example.body }.compact
+            content_type => { schema: json_schema(endpoint.body_schema), example: body }.compact
           }
         }
       end
@@ -224,6 +234,8 @@ module Reqcord
 
       def scalar_schema(field)
         types = field.types.to_a.sort
+        return { type: "string", format: "binary" } if types == ["file"]
+
         schema = { type: types.size == 1 ? types.first : types }
         schema[:enum] = field.listed_values if field.enum?
         schema[:example] = field.example unless field.example.nil?

data/lib/reqcord/exporters/postman.rb

@@ -135,7 +135,17 @@ module Reqcord
       def body_object(example)
         return nil unless example.body?
 
-        if Renderers::Payload.json?(example)
+        if Renderers::Payload.multipart?(example)
+          parts = Renderers::Payload.form_pairs(example).map do |key, value|
+            if FileValue.file?(value)
+              { key: key, type: "file", src: FileValue.filename(value) }
+            else
+              { key: key, value: value.to_s, type: "text" }
+            end
+          end
+
+          { mode: "formdata", formdata: parts }
+        elsif Renderers::Payload.json?(example)
           { mode: "raw", raw: Renderers::Payload.raw_body(example), options: { raw: { language: "json" } } }
         elsif example.body.is_a?(Hash)
           pairs = Renderers::Payload.form_pairs(example).map { |key, value| { key: key, value: value.to_s, type: "text" } }