reqcord 0.2.0 → 0.3.0

data/docs/architecture.md

@@ -0,0 +1,134 @@
+# Architecture
+
+Reqcord separates **capturing API behaviour** from **rendering
+documentation**. Test execution produces a framework-independent dataset;
+exporters operate on that dataset and nothing else.
+
+```text
+                  Rails Routes
+                       │
+                       ▼
+                Route Collector ─────────────────────────────┐
+                       │                                     │
+Minitest ──────► Test Adapter ──► Capture file (NDJSON)      │
+RSpec    ──────►       │                                     │
+                       ▼                                     │
+                   Sanitizer                                 │
+                       │                                     │
+                       ▼                                     ▼
+                Reqcord Dataset  ◄──── uncovered routes, skipped counts
+                       │
+          ┌────────────┼────────────┬────────────┐
+          ▼            ▼            ▼            ▼
+      Markdown       cURL        Postman      OpenAPI ──► Scalar (Reqcord::Web)
+```
+
+## Modules
+
+```text
+Reqcord
+├── Configuration        reqcord.yml + ENV, defaults, deep merge
+├── RouteCollector       every route kind, engines walked, skips counted by reason
+│
+├── Capture
+│   ├── IntegrationPatch prepended to ActionDispatch::Integration::Session#process
+│   ├── Collector        appends one NDJSON line per exchange, flock'd
+│   ├── TestContext      thread-local "which test is running"
+│   └── MinitestContext / RSpecContext
+│
+├── Sanitizers::Sanitizer  headers and bodies, before the dataset
+│
+├── Dataset
+│   ├── Resource         one controller path → directory / folder / tag
+│   ├── Endpoint         route + examples + inferred schemas + responses
+│   ├── RequestExample / ResponseExample
+│   └── Schema           fields, types, required, closed value sets
+│
+├── Generator            validate → collect routes → run tests → read captures
+│                        → build dataset → report → write outputs
+├── Check                generate into a scratch dir, diff against the committed docs
+├── FileValue            the upload marker ({"$file": name, "content_type": type})
+│
+├── Renderers
+│   ├── Payload          JSON vs form, nested query flattening (shared by cURL and Postman)
+│   └── Curl
+│
+├── Exporters            registry: register("name", Klass)
+│   ├── Markdown
+│   ├── Curl
+│   ├── Postman
+│   └── Openapi
+│
+├── Web                  Rack app: Scalar page + generated files
+└── Railtie              rake tasks; installs the patch only when capture is on
+```
+
+## The pipeline, step by step
+
+1. **Validate** — framework and exporter names are checked before anything
+   runs.
+2. **Collect routes** — the route table is walked (mounted engines
+   recursively, with their mount prefix), filtered by `routes.prefix`,
+   `RESOURCE` and `VERSION`; redirects and Rack mounts are counted as
+   skipped; multi-verb routes become one route per verb.
+3. **Run the tests** — `test.command` or the runner built from `test.paths`,
+   in a subprocess with `REQCORD_CAPTURE=1` and a capture file path. Inside
+   the test process the Railtie prepends the integration patch and installs
+   the framework context, so each request becomes one exchange line. Output
+   is streamed. A red suite is a warning unless `test.strict`.
+4. **Read and sanitize** — each line is parsed, malformed lines are skipped
+   with a warning, and the sanitizer replaces credentials before anything is
+   kept.
+5. **Build the dataset** — exchanges are matched to routes; `via: :all`
+   routes are materialized per captured verb; PATCH/PUT twins fold; examples
+   dedupe by sanitized signature + status; schemas are inferred from accepted
+   requests.
+6. **Report** — `routes = documented + uncovered + skipped`, unmatched paths,
+   the test outcome.
+7. **Write** — `dataset.json`, then every configured exporter.
+
+## Design principles
+
+**Tests are the source of truth.** Reqcord observes existing tests instead
+of asking for a documentation DSL. It never reads models, serializers or
+contracts: what the application accepted and answered is the specification.
+
+**Capture once, export anywhere.** Adapters turn test execution into the
+canonical dataset; exporters know nothing about Minitest, RSpec or Rails
+internals. A new format is one class with `call(dataset:, output_dir:,
+configuration:)`.
+
+**Nothing is lost silently.** Every route lands in one bucket and the report
+reconciles the sum; unmatched captures are printed; malformed capture lines
+are warned about.
+
+**Generated documentation must be safe.** Sanitization runs before the
+dataset exists, so `dataset.json` is as clean as the pages built from it.
+
+**Useful without a server.** Markdown and cURL read fine on GitHub or in a
+checkout; `Reqcord::Web` is optional.
+
+## Scope
+
+Included today: Rails 7.1–8.1 on Ruby 3.2+, Minitest integration tests and
+RSpec request specs, the whole route table, JSON, form and multipart bodies,
+multiple responses per endpoint, sanitization, inferred parameter and
+response schemas, deterministic output, Markdown, cURL, Postman (Hoppscotch),
+OpenAPI 3.1 and Scalar, `reqcord:check` for CI.
+
+Not yet: capture outside Rails (Rack::Test for Sinatra, Roda, Hanami).
+Candidate exporters: Bruno, Insomnia, `llms.txt`, static HTML, JSON Schema.
+
+## Working on Reqcord
+
+```bash
+bundle install
+bundle exec rake test                        # 190+ tests, ~12 s
+BUNDLE_GEMFILE=gemfiles/rails_7.1.gemfile bundle exec rake test
+```
+
+The suite includes an out-of-process run of `test/dummy` (a small Rails app
+with every route kind), a live Puma replay of every generated cURL script and
+Postman request, Postman schema validation, and generation of the four
+[`examples/`](../examples) — whose committed `docs/api` must match what the
+code produces.

data/docs/capture.md

@@ -0,0 +1,169 @@
+# Capture, sanitization and inference
+
+How a request made by a test ends up as a row in a parameter table.
+
+```text
+integration test ──► capture ──► sanitize ──► dataset ──► exporters
+                    (in the test    (before       (schemas
+                     process)        anything      inferred
+                                     is stored)    per endpoint)
+```
+
+## Capture
+
+`reqcord:generate` runs the suite in a subprocess with `REQCORD_CAPTURE=1`
+and `REQCORD_CAPTURE_FILE=<path>`. Inside that process the Railtie prepends a
+patch to `ActionDispatch::Integration::Session#process`, so every `get`,
+`post`, `patch`, `put` and `delete` a test makes — through Minitest
+integration tests or RSpec request specs — appends one JSON line to the
+capture file. The file is append-only and locked per write, so parallel test
+workers share it. Without those variables an ordinary `bin/rails test`
+patches nothing and writes nothing.
+
+Each line is one **exchange**:
+
+| Field | Source |
+| --- | --- |
+| `request.method`, `request.path` | the verb and the concrete path the test called (`/api/v2/customers/42`), method-override forms normalized |
+| `request.path_params` | the dynamic segments the router filled in (`id: "42"`) |
+| `request.query_params` | the query string, nested (`filter: { category: "mugs" }`) |
+| `request.headers` | the headers the test passed, plus `Content-Type` / `Accept` when they carry information (Rails' default `Accept` is dropped) |
+| `request.body`, `request.content_type` | the params the test sent, as JSON, form fields or multipart parts; an uploaded file becomes `{"$file": "label.png", "content_type": "image/png"}` — name and type, never the bytes |
+| `response.status`, `response.headers`, `response.body`, `response.content_type` | the response; JSON bodies parsed, others kept as text |
+| `source` | the test method or example name, its class or group, file and line |
+
+Supported request bodies are JSON, form-encoded and multipart
+(`Rack::Test::UploadedFile` / `fixture_file_upload` in the params). The file
+marker is what every exporter reads: `file` in the parameter table,
+`--form name=@file` in cURL, a `formdata` file part in Postman,
+`format: binary` in OpenAPI.
+
+Exchanges are ordered by test file, line and name before the dataset is
+built, so the output does not depend on the (random) order the suite ran
+in; requests made inside one test keep their execution order.
+
+## Sanitization
+
+Sanitization runs on every exchange **before** it reaches the dataset, so no
+generated file — and not `dataset.json` either — contains a credential.
+
+**Headers** are matched case-insensitively and replaced verbatim with what
+you configure:
+
+```yaml
+sanitize:
+  headers:
+    Authorization: "Bearer {{token}}"     # default
+    X-Api-Key: "{{api_key}}"              # default
+    X-Account-Id: "{{account_id}}"
+```
+
+Some headers are redacted whether configured or not: `Authorization`,
+`Proxy-Authorization`, `Cookie`, `Set-Cookie`, `X-Api-Key`, `X-Auth-Token`,
+`X-Csrf-Token`. Without a configured replacement the value becomes a
+placeholder named after the header (`{{cookie}}`), and a `Bearer`, `Token` or
+`Basic` scheme is kept so the cURL stays runnable.
+
+Transport noise (`Host`, `User-Agent`, `Content-Length`, `X-Request-Id`,
+`X-Runtime`, `ETag`, the `X-*-Options` security headers …) and empty headers
+are dropped from requests and responses alike.
+
+**Body keys** are matched case-insensitively at any depth, in requests and
+responses:
+
+```yaml
+sanitize:
+  body:
+    password: "{{password}}"      # default, with password_confirmation
+    token: "{{token}}"            # default, with access_token, refresh_token
+    api_key: "{{api_key}}"        # default
+    secret: "{{secret}}"          # default, with client_secret
+    payment_url: "{{payment_url}}"
+```
+
+The whole value under the key is replaced, whatever its type.
+
+**Placeholders** are the documentation's variables: `{{name}}` stays as-is in
+Markdown and cURL, is listed in the index under *Placeholders*, becomes a
+collection variable in Postman and feeds the security schemes in OpenAPI.
+The full list of rules is in
+[configuration.md → sanitize](configuration.md#sanitize).
+
+## Building the dataset
+
+Each exchange is matched against the route table (concrete path → route
+pattern, engine mount prefixes included) and attached to that endpoint as a
+request example and a response example. Two captures that are identical
+after sanitization and produced the same status collapse into one example;
+a different status is always kept, so a right and a wrong password are two
+examples even though both read `{{password}}`.
+
+## Inference
+
+The parameter tables and response field tables are **inferred**, and only
+from the requests the application **accepted** (`2xx`). A request that was
+rejected tells you nothing reliable about what the endpoint takes, so its
+payload is kept as a response example and nothing more.
+
+For each endpoint, all accepted bodies, query hashes and path parameter
+hashes are flattened to field paths and merged:
+
+| Column | Rule |
+| --- | --- |
+| `Field` | the path: `customer.name`, `order.line_items[].sku`, `filter.category` |
+| `Type` | the JSON types seen (`string`, `integer`, `number`, `boolean`, `null`, `object`, `array`, `file` for an upload), joined with `\|` when they differ |
+| `Required` | `yes` only when **every** accepted request carried the field; a request accepted with no parameters at all counts, so `?status=` on a list endpoint is optional |
+| `Values` | a closed set (`"active"` \| `"passive"`) when the values look like a choice; otherwise one example |
+
+A field is shown as a closed set when it has 2–6 distinct scalar values, is
+not an identifier (`id`, `*_id`, `uuid`, `token`, `slug`, `*_key`), and
+either a value repeated across requests or every value reads like a token
+(`active`, `pending_review`, `USD`) rather than content (`Ada Lovelace`,
+`ada@example.com`, `e-00056197`).
+
+Response fields use the same inference over every body captured with that
+status, with one difference: repetition across responses does not make a
+set — fixture names that come back in every list response are content, not a
+choice — so only the token rule applies. The first body captured for a
+status is the example shown.
+
+## What the dataset holds
+
+```json
+{
+  "schema_version": 2,
+  "endpoints": [
+    {
+      "name": "Create Customer",
+      "method": "POST",
+      "path": "/api/v2/customers",
+      "controller": "api/v2/customers",
+      "action": "create",
+      "route_name": "api_v2_customers",
+      "also_methods": [],
+      "parameters": {
+        "path": [],
+        "query": [],
+        "body": [
+          { "path": "customer.name",   "type": "string", "required": true, "values": ["John Doe"] },
+          { "path": "customer.status", "type": "string", "required": true, "values": ["active", "passive"] }
+        ]
+      },
+      "responses": [
+        { "status": 201, "schema": [ { "path": "id", "type": "integer", "required": true, "values": [42] } ],
+          "example": { "id": 42, "name": "John Doe" } },
+        { "status": 422, "schema": [ "…" ], "example": { "errors": { "email": ["can't be blank"] } } }
+      ],
+      "request_examples":  [ "every captured request, sanitized, with its source test" ],
+      "response_examples": [ "every captured response, sanitized" ]
+    }
+  ],
+  "uncovered_routes": [
+    { "method": "DELETE", "path": "/api/v2/customers/:id", "controller": "api/v2/customers", "action": "destroy" }
+  ]
+}
+```
+
+`endpoints` holds the documented ones; everything else is under
+`uncovered_routes`. `schema_version` lets the format evolve — readers ignore
+keys they do not know.

data/docs/configuration.md

@@ -23,6 +23,7 @@ exporters:
   - curl
   - markdown
   - postman
+  - openapi
 
 variables:
   base_url: http://localhost:3000
@@ -212,7 +213,8 @@ as is). Override per run with `REQCORD_OUTPUT`. A run writes:
 │   ├── index.md
 │   └── create.md
 ├── curl/api/v1/customers/create.sh                                 (curl)
-└── postman/collection.json                                         (postman)
+├── postman/collection.json                                         (postman)
+└── openapi/openapi.json                                            (openapi)
 ```
 
 Directories follow the controller path, so `admin/customers` and
@@ -349,6 +351,7 @@ soon as `base_url` and `token` are filled in.
 | --- | --- |
 | `bin/rails reqcord:init` | writes `reqcord.yml` (never overwrites) and creates `docs/api/` |
 | `bin/rails reqcord:generate` | collects routes, runs the suite with capture, writes every exporter, prints the report |
+| `bin/rails reqcord:check` | generates into a scratch directory and compares with `output.directory`; exits 1 with an `A`/`D`/`M` file list when the committed docs are behind the tests. Honours `RESOURCE` / `VERSION` |
 | `bin/rails reqcord:routes` | lists the routes the current `prefix` (and `RESOURCE` / `VERSION`) would document |
 
 Every `generate` run ends with a reconciliation of the whole route table:
@@ -369,4 +372,6 @@ documented route is listed, never dropped silently.
 | `no request was captured` | the gem is not in the `:test` group of the Gemfile, or `test.paths` / `test.command` runs no integration tests |
 | many requests captured, few matched | `routes.prefix` does not cover them — the unmatched paths are printed |
 | routes documented but few covered | the tests that exercise them are not in `test.paths` (a `2xx` from a test is what makes an endpoint documented) |
-| `Test suite failed while generating` | the suite is red; fix the tests, documentation is only generated from a passing run |
+| `test run exited with status …` | the suite is red; the docs were still generated from what it captured. `test.strict: true` aborts instead |
+
+More in [troubleshooting.md](troubleshooting.md).

data/docs/exporters.md

@@ -0,0 +1,182 @@
+# Exporters
+
+Every exporter reads `dataset.json` and nothing else; none of them knows
+about Minitest, RSpec or Rails. Pick them in `reqcord.yml`:
+
+```yaml
+exporters:
+  - curl
+  - markdown
+  - postman
+  - openapi
+```
+
+All four are on by default. `dataset.json` is always written. An unknown name
+raises `Reqcord::ConfigurationError` before any test runs.
+
+Only **documented** endpoints — those a test reached with a `2xx` — get a
+Markdown page, a cURL script, a Postman request and an OpenAPI operation.
+Uncovered routes are listed in the Markdown index and in
+`dataset.json` (`uncovered_routes`); `output.include_uncovered: true` also
+writes a placeholder page for each.
+
+## What the canonical request is
+
+Every output that shows a request — the cURL, the Postman request, the OpenAPI
+example — uses the same one: the first captured request that returned a
+`2xx`, with its concrete URL, headers, query and body, sanitized. Reqcord
+never invents a payload. Requests that produced other statuses are kept as
+response examples, never promoted to the canonical request.
+
+## `markdown`
+
+```text
+docs/api/
+├── README.md
+├── api/v2/customers/
+│   ├── index.md
+│   ├── list.md
+│   ├── create.md
+│   ├── show.md
+│   └── update.md
+└── admin/customers/
+    └── …
+```
+
+* **`README.md`** — the base URL, a *Placeholders* table (every sanitized
+  header and its replacement), one section per resource with a
+  method / path / description table linking to the endpoint pages, and a *No
+  Successful Request Captured* section listing the uncovered routes.
+* **`<controller path>/index.md`** — the resource's endpoints.
+* **`<controller path>/<action>.md`** — the endpoint page: method and path
+  (with `(also PUT)` when PATCH/PUT twins were folded), the namespace, the
+  headers, *Path / Query / Body Parameters* tables (`Field | Type | Required |
+  Values`), the example request, the cURL, and one `### <status>` section per
+  captured status with a *Fields* table and an example body.
+
+Directories follow the full controller path, so `admin/customers` and
+`api/v2/customers` never collide. File names come from the action
+(`index` → `list.md`); when two endpoints of a resource would share a name the
+verb is appended.
+
+Form bodies are shown as they were sent, with a note that the request is
+`application/x-www-form-urlencoded`. In a multipart request a file part is
+shown by name and type (`"image": "label.png (image/png)"`) and typed `file`
+in the parameter table.
+
+## `curl`
+
+One runnable script per documented endpoint:
+
+```text
+docs/api/curl/api/v2/customers/create.sh
+```
+
+```bash
+curl --request POST \
+  --url "http://localhost:3000/api/v2/customers" \
+  --header "Authorization: Bearer {{token}}" \
+  --header "Content-Type: application/json" \
+  --data '{
+  "customer": {
+    "name": "John Doe",
+    "email": "john@example.com",
+    "status": "active"
+  }
+}'
+```
+
+* The host is `variables.base_url`.
+* JSON bodies are `--data '<json>'`; form bodies are `--data 'a=b'` pairs,
+  never re-encoded as JSON; query strings keep Rails' bracket notation
+  (`filter[category]=mugs`).
+* Uploads are one `--form` per part, the file as `@name`
+  (`--form 'image=@label.png;type=image/png'`) — run the script from the
+  directory that holds the file. curl sets the multipart `Content-Type` and
+  boundary itself, so none is written.
+* Placeholders (`{{token}}`) are left for you to substitute — the scripts
+  are meant to be copied into a terminal or a runbook.
+
+Reqcord's own test suite replays every generated script of its dummy
+application against a live Puma and asserts the documented status, so the
+scripts are known to run.
+
+## `postman`
+
+`docs/api/postman/collection.json`, a Postman Collection v2.1:
+
+* one folder per controller namespace (`Api › V2 › Customers`),
+* one request per documented endpoint, from the canonical request — JSON
+  bodies as `raw`, form bodies as `urlencoded`, uploads as `formdata` with
+  the file part's `src` set to the file name (pick the file in Postman),
+* every captured status saved as a response example on that request,
+* collection variables for `base_url` and every placeholder the sanitizer
+  wrote (`{{token}}`, `{{api_key}}` …) — Postman's variable syntax is the
+  same, so the collection runs as soon as the variables are filled in,
+* `Authorization: Bearer {{token}}` lifted to collection-level bearer auth;
+  requests made without credentials are marked `noauth`, so they replay
+  exactly as their tests did.
+
+The collection is validated against the published v2.1 JSON Schema in
+Reqcord's tests, and replayed against the dummy application.
+
+**Hoppscotch** imports Postman v2.1 collections directly: *Import → Postman*
+and point it at the same file.
+
+## `openapi`
+
+`docs/api/openapi/openapi.json`, an OpenAPI 3.1 document:
+
+* `info.title` is `<application directory> API`; `info.version` is the API
+  version when the dataset has exactly one (`v2`), `1.0.0` otherwise;
+  `servers` is `variables.base_url`.
+* One path item per documented route in OpenAPI notation: `/customers/:id`
+  → `/customers/{id}`, `/files/*path` → `/files/{path}`. An optional segment
+  becomes two paths: `/items(/:id)` → `/items` and `/items/{id}`.
+* One operation per verb: `operationId` from the verb and path
+  (`post_api_v2_customers`), `summary` is the endpoint name, `tags` is the
+  resource, `description` carries `controller#action`, the route name and
+  the folded verbs.
+* **Parameters**: path parameters (always required) and query parameters
+  from the inferred schemas, with `enum` for closed value sets and the
+  captured example. Nested query params keep Rails' bracket notation
+  (`filter[status]`).
+* **Request body**: `application/json`, `application/x-www-form-urlencoded`
+  or `multipart/form-data`, whichever the test sent. The JSON Schema is
+  rebuilt from the flattened field paths — `order.line_items[].sku` becomes
+  object → array → object — with `required` at every level from what every
+  accepted request carried, and the canonical request as `example`. A file
+  part is `type: string, format: binary`, its example the file name.
+* **Responses**: one per captured status, `description` from the status
+  text, schema and example from the captured bodies.
+* **Security**: `bearerAuth` (HTTP bearer) when a documented request carried
+  `Authorization: Bearer …`, `apiKeyAuth` (header) for the first sanitized
+  API-key header a request carried (`X-Api-Key` by default). Applied per
+  operation, so endpoints called without credentials stay public.
+
+Anything that reads OpenAPI consumes the file as is: Scalar (through
+[`Reqcord::Web`](web.md)), Swagger UI, Redoc, client generators.
+
+## Writing your own
+
+An exporter is a class with one class method, registered under a name:
+
+```ruby
+module Reqcord
+  module Exporters
+    class Bruno
+      def self.call(dataset:, output_dir:, configuration:)
+        # dataset: Reqcord::Dataset — resources, endpoints, examples, schemas
+        # returns the paths it wrote
+      end
+    end
+
+    register("bruno", Bruno)
+  end
+end
+```
+
+Once registered, `exporters: [bruno]` in `reqcord.yml` runs it after the
+tests. `dataset.json` on disk is the same structure, if you would rather
+post-process the file from another language. See
+[architecture.md](architecture.md) for the dataset.