nous 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c73c21d427c9bb99cc148e089ed5899e7aa9e3ca86a4825540380d41771354d2
-  data.tar.gz: 4b361a7aed3c0dfb28a6a650b0813d371622c82b910b8880502281047152a739
+  metadata.gz: e4371689843d5f5275e56afaaab2630ba2a703e57e29cfb79d6d47d19040d934
+  data.tar.gz: 1402e3ab54931a99bbad484002345514142a9ade10453938c8f2feae77ef7742
 SHA512:
-  metadata.gz: aacbc4777dc1e5bd66513ddc3bc5a1f667276ac2e89ff781f7b43762133ae640d68bc1191c11b61ee00ac7911a128fcf6ad80653f86d369d284223e830f09120
-  data.tar.gz: 90fc8f0cf3c30c6e06bf6aeebd2790539ff80d0d63469a367c4a09b008c9ceed3dedc0d18528e5f6fab3c4a02103b4d2c7c9b3788b1708c6a1a46790d9f5cbab
+  metadata.gz: 14a0487839b29fa46c277e341e4c33b631bb22380cbb2ee91954fcff61f264856baf981f2bc8a6e20b6da3e8a81c7c26c7178db5f96d96f5be0e2a53676d72f6
+  data.tar.gz: 8e2eeb9b3e790ea85f79d877c19176569918e3030fff096950c6de7771e3ebe9556cdeb82b16a67cb8d9c6ea166c6634c3fec6c40ea8f72ffa9168b0c4fa9370

data/.agents/skills/verify-nous/SKILL.md

@@ -0,0 +1,6 @@
+---
+name: verify-nous
+description: Verify Nous retained-HTML extraction and detailed fetch outcomes through offline Ruby recipes; use when changing extraction or assessing Substrate and Autodidact readiness.
+---
+
+Read [the verification guide](../../../docs/verification/README.md) from the repository root. Select the affected recipe, execute launch, doctor, drive, evidence capture and cleanup, and report every exit status. Follow the guide's source-identification and blocker rules. A draft or failed recipe does not certify readiness. Keep provisioning in the linked setup guide; do not add live network calls to these checks.

data/CHANGELOG.md

@@ -1,4 +1,26 @@
-## [Unreleased]
+## [0.5.0] - 2026-10-25
+
+### Added
+
+- `Nous::ExtractHtml.call` extracts retained HTML without network acquisition, with
+  Markdown or plain text, source metadata, explicit decoding errors and caller byte limits.
+
+### Changed
+
+- Defuddle is the default for URL and retained-HTML extraction. The gem bundles
+  its JavaScript dependencies and requires Node.js 22+ on macOS/Linux.
+- Readability remains available through `Extractor::Default` and `--readability`.
+  Extraction failures do not silently switch backends.
+
+### Removed
+
+- Jina Reader integration (`Extractor::Jina`, `--jina`, `JINA_API_KEY`). Neither
+  remaining backend renders JavaScript pages.
+
+### Fixed
+
+- Detailed fetch results retain extraction failures and all attempted failures from
+  concurrent crawl batches, with accurate request counts and unchanged page limits.
 
 ## [0.4.0] - 2026-04-11
 

data/CODING_STANDARDS.md

@@ -0,0 +1,203 @@
+# Coding standards
+
+Read these standards before implementation and include them in Anaphor's Standards
+review. They apply to this Ruby gem and its development tools. Existing code is not
+a blanket exception: when changing an affected boundary, address deviations or
+record a concrete compatibility reason in the PR. This document does not itself
+change existing public APIs.
+
+## Responsibilities and service objects
+
+Keep crawling, readable-content extraction, conversion, and serialization distinct.
+Commands perform operations; fetch records describe acquired originals; pages describe
+extracted content and provenance. Keep HTTP details in crawler and remote extractor
+adapters. Follow the existing Zeitwerk layout; `lib/nous/primitives/` is collapsed
+into the `Nous` namespace.
+
+A command exposes one public instance operation, `call`, invoked through inherited
+`.call`. Its initializer accepts explicit inputs and dependencies and stores state;
+construction must not launch processes, perform network requests, or do business
+work. Resolve environment-dependent defaults when the operation runs.
+
+Application-service instance `call` methods that describe business operations must
+contain only bareword calls to private methods named for business functions. Read
+them as prose: one named step per line, without arguments, explicit receivers,
+assignments, conditionals, loops, blocks, or explicit returns. A `rescue` clause
+is the only exception; it may map an expected exception to a failure result.
+Put decisions, resource scopes, collaborator calls, and result construction inside
+the private steps. The final step returns the service result. For example:
+
+```ruby
+def call
+  prepare_retained_content
+  extract_readable_content
+  build_extraction_result
+rescue ArgumentError => error
+  failure(error)
+end
+```
+
+A failed prerequisite must stop the sequence: raise a meaningful exception handled
+by the service or the existing command boundary. Returning a failure from a private
+step alone does not stop later steps. Keep resource acquisition, use and cleanup
+in one private scope rather than spreading temporary paths across orchestration
+steps. Do not add instance state merely to split an indivisible operation.
+
+This rule governs business-operation services, including extraction workflows. The shared class-level dispatcher and low-level value/protocol helpers
+are not business orchestration. Apply the rule when changing an existing service;
+this change does not claim that all historical commands have been migrated.
+
+Before extracting an object, name the policy, resource, or collaborator contract
+it will own. If no separate owner exists, keep the steps private in the current
+service. Do not extract forwarding layers to satisfy a line count: they add places
+to navigate without reducing the caller's obligations. Name an extraction for its
+operation, not Manager, Processor, or Utils. Keep duplication when sharing would
+require unrelated behavior flags; those flags couple operations that change apart.
+
+Use Sandi Metz's five-line methods, 100-line classes, and four-parameter guideline
+as review prompts, not mechanical limits. Count independent keyword options as
+parameters. A growing constructor suggests mixed responsibilities; hiding those
+options in a hash does not fix that. Introduce an input object only when its data
+has cohesive behavior or invariants. Explain justified exceptions locally in the PR.
+
+Apply tell, don't ask: put an operation with the object that owns its behavior.
+Use a known collaborator's interface instead of branching on `is_a?`, `kind_of?`,
+or `respond_to?`. At an untrusted provider boundary, validate the actual protocol
+and required data before producing normalized content. Do not spread speculative
+type checks through in-process callers.
+
+## Building and configuring objects
+
+Prefer `tap` when configuring and returning the same object, and
+`each_with_object` when accumulating into a mutable collection, instead of a
+temporary local used only to initialize, mutate, and return that object. This
+keeps construction and its result in one expression. Use `map` for one-to-one
+transformations; keep named locals when they explain an intermediate result or
+are reused. With `each_with_object`, mutate the accumulator rather than reassign
+the block variable. These blocks belong inside private service steps, preserving
+the bareword-only business `call` convention.
+
+## Arguments and defaults
+
+For each proposed argument, inspect its callers and choose in this order:
+
+1. If no caller or accepted use case needs the choice, do not add the argument.
+   A test convenience alone does not justify expanding the public API.
+2. If every caller must supply a value, require it.
+3. If omission selects a normal value, give it a concrete default or default
+   expression. Do not use `nil` merely to replace it with that value later.
+4. If absence changes the operation, document it and permit `nil`. For example,
+   `selector: nil` leaves extraction unscoped.
+5. If automatic selection differs from an explicit override, represent that mode
+   explicitly and test both paths. Discover environment dependencies during `call`,
+   not construction, so creating the service does not launch work.
+
+For an existing argument, record the behavior of omission, explicit `nil`, and
+explicit values before editing. Preserve supported cases and test the changed
+boundary. Do not silently make a strict executable override fall back to another
+installation. Add a sentinel or null object only when those behaviors must differ;
+otherwise it adds another state every caller must understand.
+
+## Results, failures, and resources
+
+Use the existing `Nous::Command` and `Nous::Command::Result` contract in
+[lib/nous/command.rb](lib/nous/command.rb): commands return `payload`, `error`, and
+`metadata`; callers inspect `success?` or `failure?`. Extractors expose `extract(record)`
+and return the same result contract; do not rename that protocol as a style change. Do not copy Rails `ApplicationService`, named
+result fields, HTTP error codes, or reporting infrastructure into this gem.
+
+The inherited class `.call` currently converts `StandardError` into a failure
+result, wrapping post-construction errors as `CommandError`. Preserve that behavior
+unless an explicit API change is accepted. Direct instance `call` and extractor
+`extract` do not receive that dispatcher protection. A failed child command does not
+short-circuit its parent automatically: inspect its result and stop before any
+step that depends on success. Keep stage-specific errors useful without including
+credentials or sensitive source content. Do not turn a failure into empty success.
+
+One owner acquires, uses, and releases each temporary resource. Keep cleanup in a
+block or `ensure` that covers success and failure, including parsing. Do not return
+a temporary path after its owner has released it. Never use the original input as
+an output destination. Invoke subprocesses with argument arrays, not interpolated
+shell strings. Make partial external effects explicit; cleanup cannot undo them all.
+
+## Development and verification commands
+
+Choose the command's owner before adding a file:
+
+1. For a repeatable project check, add a named Rake task. Put live checks under
+   `verify:` and document inputs in the owning recipe so agents can discover them
+   with `bundle exec rake -T`. Create these directories when a check needs them.
+2. Put verification assertions, fixture preparation, and cleanup in a focused
+   harness under `spec/verification/`. Keep `tasks/*.rake` limited to task inputs
+   and dispatch, so command plumbing does not obscure what the check proves.
+3. Use `bin/` for setup and console; the existing public command lives in `exe/nous`. Do not
+   add a generic scripts directory for verification commands; it creates a second
+   entry-point convention to discover and maintain.
+
+Keep production behavior in the library and exercise its public API from the
+harness. Never add downloads or live provider calls to default `bundle exec rake`;
+ordinary verification must run without external services or model provisioning.
+
+Document required binaries, models, credentials, supported runtime versions, and
+readiness checks in `docs/setup/`. Keep executable verification steps and evidence
+limits in `docs/verification/`. Link both from onboarding. Install dependencies through explicit setup
+steps, never silently during a check; readiness and behavior verification are separate results. Retain evidence with
+the tested source revision; never claim that simulated processes prove a local
+runtime works.
+
+## Tests and review
+
+Test observable outcomes through command `.call` or the owning source interface.
+Do not assert private helper order. When a workflow has prerequisites, cover failure before dependent steps. When it
+owns files or processes, cover cleanup and input preservation on success and
+failure. When calls can overlap, prove their resources stay isolated. Use
+controlled provider/process fixtures for ordinary tests and separate opt-in live
+checks for actual compatibility. Keep fixtures' provenance and limitations visible.
+
+For Ruby changes, run affected specs, the full RSpec suite, and Standard using
+the Development commands in [README.md](README.md). Include an affected verification
+recipe once one exists. Documentation-only changes need
+link, command, and contract checks; they do not require pretending runtime behavior
+changed. Anaphor reviews must distinguish documented rules, design heuristics, and
+accepted requirements. A green suite does not establish compliance with every
+principle or prove live-provider behavior.
+
+## Retained HTML convention
+
+`ExtractHtml` keeps six explicit inputs: original bytes, source URL, content type,
+output representation and two independent caller budgets. This is a deliberate
+exception to the parameter heuristic: a hash or options object without cohesive
+behavior would only hide the same choices. The command orchestrates decoding, readable
+selection, rendering and bounded output. `DecodeHtml` owns the byte interpretation
+policy shared with acquired Defuddle HTML. Within `ExtractHtml`, invoke `DecodeHtml`
+and `Defuddle::Client` with `new(...).call`: this deliberate exception to inherited
+`.call` preserves the original unexpected exception as `UnexpectedFailure#cause`.
+The shared dispatcher wraps those exceptions and loses their identity. Keep this
+exception local to the retained command's diagnostic contract and cover failures
+during dependency execution. Trust the client's normalized title and Markdown;
+the retained command owns final rendering, emptiness and combined-byte budgets.
+Legacy shared extractors must
+not acquire per-record mutable state: the fetch runner calls them concurrently.
+
+## Sources and adaptation
+
+Adapted from Substrate's `CODING_STANDARDS.md` on October 25, 2026, which was
+adapted from Autodidact's standards. Nous-specific responsibilities, result names,
+extractor protocol, executable locations, and verification commands replace their
+Substrate equivalents. Local audio conventions do not apply to Nous.
+Retained service orchestration, explicit inputs, failure sequencing, ownership,
+and public-interface testing. Rails controllers, Active Record, transactions,
+Jbuilder, Sentry, Next.js, and that application's result shape are outside this
+project's conventions. The source document provides context, not inherited
+instructions to provision those systems.
+
+- [Thoughtbot: Tell, Don't Ask](https://thoughtbot.com/blog/tell-dont-ask)
+  informs behavior ownership and collaborator interfaces.
+- [Thoughtbot: Sandi Metz' Rules for Developers](https://thoughtbot.com/blog/sandi-metz-rules-for-developers)
+  supplies the size and parameter heuristics; their use as review prompts here is
+  a local convention.
+- [Sandi Metz: The Wrong Abstraction](https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction)
+  informs the preference for cohesive objects over premature generalization.
+
+Nullable-default policy and Rake placement are project decisions, not rules
+attributed to these authors.

data/README.md

@@ -2,10 +2,14 @@
 
 Crawl websites and extract readable Markdown, optimized for LLM consumption. Inspired by [sitefetch](https://github.com/egoist/sitefetch).
 
-Nous fetches same-host pages starting from a seed URL, extracts readable content, and outputs clean Markdown as XML-tagged text or JSON. It supports concurrent crawling, glob-based URL filtering, and two extraction backends: a local parser (ruby-readability) and the Jina Reader API for JS-rendered sites.
+Nous fetches same-host pages starting from a seed URL, extracts readable content, and outputs clean Markdown as XML-tagged text or JSON. It supports concurrent crawling, glob-based URL filtering, and local extraction with Defuddle by default and an explicit ruby-readability backup. Both parse acquired HTML without rendering JavaScript. Defuddle decodes acquired bytes using the same header/BOM/meta/UTF-8 precedence as retained HTML; invalid bytes produce extraction failures.
 
 ## Installation
 
+Default extraction requires Node.js 22+ on PATH (macOS/Linux). The gem includes
+the bundled Defuddle adapter; npm is only needed to rebuild it during development.
+See [Defuddle setup](docs/setup/defuddle.md).
+
 Add to your Gemfile:
 
 ```ruby
@@ -39,8 +43,8 @@ nous https://example.com -m "/blog/*"
 # Scope extraction to a CSS selector
 nous https://example.com -s "article.post"
 
-# Use Jina Reader API for JS-rendered sites (Next.js, SPAs)
-nous https://example.com --jina
+# Use the Ruby-only Readability backup
+nous https://example.com --readability
 
 # Debug logging
 nous https://example.com -d
@@ -57,7 +61,7 @@ nous https://example.com -d
 | `-s`, `--selector SELECTOR` | CSS selector to scope extraction | none |
 | `-l`, `--limit N` | Maximum pages to fetch | `100` |
 | `--timeout N` | Per-request timeout in seconds | `15` |
-| `--jina` | Use Jina Reader API for extraction | off |
+| `--readability` | Use the Ruby-only Readability backup | off |
 | `-v`, `--version` | Print version and exit | off |
 | `-h`, `--help` | Print usage and exit | off |
 | `-d`, `--debug` | Debug logging to stderr | off |
@@ -84,9 +88,9 @@ text = Nous.serialize(pages, format: :text)
 # Serialize to JSON
 json = Nous.serialize(pages, format: :json)
 
-# Use the Jina extractor for JS-heavy sites
-pages = Nous.fetch("https://spa-site.com",
-  extractor: Nous::Extractor::Jina.new,
+# Select the Readability backup explicitly
+pages = Nous.fetch("https://example.com",
+  extractor: Nous::Extractor::Default.new,
   limit: 5
 )
 ```
@@ -99,12 +103,20 @@ Use the `details: true` option to receive full fetch results including failures:
 result = Nous.fetch("https://example.com", details: true)
 
 result.pages       # Array<Nous::Page> - successfully extracted pages
-result.failures    # Array<{requested_url:, error:}> - failed fetches
+result.failures    # Array<{requested_url:, error:}> - failed acquisition or extraction
 result.total_requested  # Integer - total URLs attempted
 result.all_succeeded?   # Boolean - true if no failures
 result.any_succeeded?   # Boolean - true if at least one page extracted
 ```
 
+Acquisition failures retain their existing messages; extraction failures retain their
+exception objects. Detailed results include failed recursive requests as well as
+failed extraction. Pages-only calls still return successful pages in source order.
+A concurrent batch may already have attempted more successful pages than the limit
+allows returning. `total_requested` counts every attempted request in that batch;
+failures remain visible, while excess successful pages are omitted from `pages`.
+The limit still governs retained successful pages, not failed attempts.
+
 This is useful when you need to handle failures explicitly:
 
 ```ruby
@@ -122,13 +134,69 @@ result.pages.each do |page|
 end
 ```
 
+### Retained HTML extraction
+
+Use saved bytes directly, without crawling or reacquiring a URL:
+
+```ruby
+result = Nous::ExtractHtml.call(
+  html: File.binread("saved.html"),
+  source_url: "https://example.com/article",
+  content_type: "text/html; charset=utf-8",
+  format: :text
+)
+raise result.error if result.failure?
+puts result.payload.title
+puts result.payload.content
+puts result.metadata
+```
+
+`html:` and `source_url:` are required. The URL is provenance only; this command
+never requests it. `content_type:` defaults to `"text/html"`; HTML and XHTML are
+supported. `format:` defaults to `:markdown`; `:text` returns readable text with
+collapsed whitespace and separated blocks. Defuddle is used for both formats. Legacy `Extractor::Default#extract`
+continues producing Markdown with its existing selector behavior.
+
+Character decoding uses a declared content-type charset first, then a UTF BOM,
+then an HTML meta charset in the first 1024 bytes, then UTF-8. Input is treated as
+bytes regardless of its Ruby encoding label and is never modified. Invalid byte
+sequences and decoded NULs fail; unsupported charset names fail explicitly. A
+conflicting meta declaration does not override a declared header charset.
+
+The result payload remains `Nous::ExtractedContent` (`title`, `content`). Metadata
+contains `source_url`, original `content_type`, selected `encoding`, `format`, and
+limitations `static_html_only` and `authentication_not_detected`. Static readable
+content does not prove that an authenticated or JavaScript-rendered page is complete.
+
+`max_input_bytes:` defaults to 25 MiB; `max_output_bytes:` defaults to 32 MiB.
+Both require positive integers. Input is checked before copying/parsing. Output
+counts the combined UTF-8 bytes of title and content, excluding caller-supplied
+provenance metadata. These are input/result guards, not peak-memory, CPU or wall-time
+limits; callers needing those protections must retain process supervision.
+
+Expected errors are nested under `Nous::ExtractHtml`:
+
+| Error | Meaning |
+| --- | --- |
+| `InvalidContent` | Invalid encoded bytes, NULs, or invalid decoded output |
+| `UnsupportedContent` | Unsupported media type, output format or charset |
+| `NoContent` | No readable content or empty converted output |
+| `InputLimitExceeded` / `OutputLimitExceeded` | Caller byte budget exceeded |
+| `InvalidLimits` (`ArgumentError`) | Nonpositive or noninteger byte limit |
+| `UnexpectedFailure` | Safe outward message; original exception in `cause` |
+
+Treat diagnostic causes as private debugging data; their messages can contain
+source content. Rails acquisition, storage, lifecycle and process supervision are
+outside this command. See the [verification map](docs/verification/README.md) for
+executed surfaces and remaining consumer acceptance.
+
 ### Page Structure
 
 Each extracted page contains:
 
 | Field | Type | Description |
 |-------|------|-------------|
-| `title` | String | Page title (fallback chain: readability → `<title>` tag → `<h1>`) |
+| `title` | String | Page title selected by the extraction backend |
 | `url` | String | Final URL after redirects |
 | `pathname` | String | URL path component |
 | `content` | String | Extracted content as Markdown |
@@ -138,7 +206,7 @@ Each extracted page contains:
 
 ```ruby
 page.metadata  # => {
-  #   extractor: "Nous::Extractor::Default",  # Which extractor was used
+  #   extractor: "Nous::Extractor::Defuddle",  # Which extractor was used
   #   requested_url: "https://example.com/blog", # Original URL before redirects
   #   content_type: "text/html; charset=utf-8",  # HTTP Content-Type header
   #   redirected: true                           # Whether redirects occurred
@@ -147,7 +215,19 @@ page.metadata  # => {
 
 ## Extraction Backends
 
-### Default (ruby-readability)
+### Defuddle (default)
+
+Runs the bundled Defuddle parser in Node.js against the acquired HTML and final
+URL. Returns Markdown directly, retaining structure such as code blocks, tables,
+and links. Supports `--selector`. Each extraction owns its process, with a
+15-second deadline, a 64 MiB JSON response limit and a 64 KiB diagnostic limit.
+These are separate from HTTP timeouts and retained-command byte budgets.
+Page scripts do not execute and third-party fallback fetching is disabled.
+Failures remain visible through `details: true`; there is no automatic switch
+to Readability. `ExtractHtml` uses this same engine while retaining its decoding,
+plain-text rendering and byte-budget contract.
+
+### Readability (explicit backup)
 
 Parses static HTML using [ruby-readability](https://github.com/cantino/ruby-readability), strips noisy elements (script, style, nav, footer), and converts to Markdown via [reverse_markdown](https://github.com/xijo/reverse_markdown). Fast and requires no external services, but cannot extract content from JS-rendered pages.
 
@@ -156,9 +236,13 @@ Title extraction uses a fallback chain:
 2. Original `<title>` tag from HTML
 3. First `<h1>` from extracted content
 
-### Jina Reader API
+### Migrating from Jina
 
-Uses the [Jina Reader API](https://jina.ai/reader/) which renders pages with headless Chrome. Handles Next.js App Router, React Server Components, SPAs, and other JS-heavy sites. Free tier allows 20 requests/minute without a key, or 500 RPM with a `JINA_API_KEY` environment variable.
+`Extractor::Jina` and `--jina` have been removed. Remove the extractor argument or
+flag to use Defuddle; `JINA_API_KEY` is no longer used. This retires remote browser
+rendering: pages requiring JavaScript may yield incomplete or no readable content.
+Use `Extractor::Default.new` or `--readability` for the Ruby-only backup.
+`ExtractHtml` always uses Defuddle; its result and error shapes remain unchanged.
 
 ## Output Formats
 
@@ -171,7 +255,7 @@ XML-tagged output designed for LLM context windows:
   <title>Page Title</title>
   <url>https://example.com/page</url>
   <pathname>/page</pathname>
-  <extractor>Nous::Extractor::Default</extractor>
+  <extractor>Nous::Extractor::Defuddle</extractor>
   <content>
 # Heading
 
@@ -190,7 +274,7 @@ Extracted markdown content...
     "pathname": "/page",
     "content": "# Heading\n\nExtracted markdown content...",
     "metadata": {
-      "extractor": "Nous::Extractor::Default",
+      "extractor": "Nous::Extractor::Defuddle",
       "requested_url": "https://example.com/page",
       "content_type": "text/html; charset=utf-8",
       "redirected": false
@@ -201,6 +285,11 @@ Extracted markdown content...
 
 ## Development
 
+Read [CODING_STANDARDS.md](CODING_STANDARDS.md) before implementation and include it
+in Anaphor Standards reviews. Use the [verification map](docs/verification/README.md)
+and project-local [verify-nous skill](.agents/skills/verify-nous/SKILL.md) for the
+retained-HTML and failure-propagation readiness checks.
+
 ```bash
 bin/setup               # Install dependencies
 bundle exec rspec       # Run tests

data/Rakefile

@@ -8,3 +8,5 @@ RSpec::Core::RakeTask.new(:spec)
 require "standard/rake"
 
 task default: %i[spec standard]
+
+Dir.glob("tasks/*.rake").sort.each { |path| load path }

data/docs/setup/defuddle.md

@@ -0,0 +1,38 @@
+# Defuddle runtime
+
+Default URL and retained-HTML extraction requires Node.js 22 or newer on PATH.
+The process adapter currently targets macOS/Linux (POSIX process groups). Install
+Node explicitly using your normal runtime manager. No API key or service is used.
+Check readiness from the repository root:
+
+```bash
+node --version
+node -e 'if (Number(process.versions.node.split(".")[0]) < 22) process.exit(1)'
+printf '%s' '{"html":"<html><head><title>Probe</title></head><body><p>Readable probe.</p></body></html>","url":"https://example.com"}' | node vendor/defuddle.mjs
+```
+
+Require exit zero and JSON containing title `Probe` and Markdown `Readable probe.`.
+The published gem includes `vendor/defuddle.mjs` and third-party license notices;
+users do not run npm. A missing Node binary produces an extraction failure.
+`Nous.fetch(..., extractor: Nous::Extractor::Default.new)` and `--readability`
+select the Ruby-only backup explicitly. Retained `ExtractHtml` requires Node.
+
+## Rebuilding the bundled adapter
+
+Maintainers use the pinned dependencies in `package-lock.json`. From the repository
+root, provision dependencies explicitly and rebuild:
+
+```bash
+npm ci --ignore-scripts
+npm run build
+```
+
+Commit the source, package manifest/lockfile, generated bundle and license notices
+together. `javascript/build.mjs` regenerates the bundle and gathers dependency
+license texts. Runtime extraction and verification never install dependencies.
+Run the [Defuddle recipe](../verification/defuddle.md) after rebuilding.
+
+The bridge uses a non-executing DOM parser, disables async extractors and rejects
+fetch calls. It is not a browser renderer. The adapter uses a 15-second process
+deadline and bounded output, but does not enforce a process memory limit. Callers
+with stronger isolation requirements still own external process supervision.

data/docs/setup/verification.md

@@ -0,0 +1,21 @@
+# Verification setup
+
+Work from the Nous repository root. Use the Ruby declared in `.ruby-version`
+(currently 4.0.1), the Bundler version in `Gemfile.lock` (currently 4.0.6), and
+that lockfile's gems. With rbenv, `rbenv exec ruby -v` must identify the intended
+Ruby; shell/system Ruby 2.6 is unsuitable. Put rbenv shims on PATH or prefix the
+commands in the verification guide with `rbenv exec`.
+
+Run `BUNDLE_FROZEN=true bundle check`. If dependencies are absent, run
+`BUNDLE_FROZEN=true bundle install` once, then repeat the check. Installation may
+need network and permission to write the gem installation. If Ruby/Bundler is
+missing, provision those declared versions using your runtime manager first.
+Do not update or regenerate the lockfile to make verification work. Frozen mode
+also prevents Bundler from opportunistically rewriting platform metadata.
+
+The recipes use RSpec, WebMock, Node.js 22+ and the bundled Defuddle adapter.
+Follow [Defuddle setup and readiness](defuddle.md) before running them. No server,
+browser, remote account or model is needed. HTTP in the failure recipe is a
+controlled response fixture. WebMock blocks Ruby HTTP; the Node adapter disables
+Defuddle async fetching and rejects fetch calls.
+No credentials or retained private content belong in evidence.

data/docs/verification/README.md

@@ -0,0 +1,77 @@
+# Nous verification
+
+The project-local [verify-nous skill](../../.agents/skills/verify-nous/SKILL.md)
+uses this guide. Initial scope is the two selected architecture candidates:
+retained-HTML extraction and visible extraction failures. This map does not
+certify all crawling, rendering, serialization or consumer behavior.
+
+| Recipe | State | Coverage |
+| --- | --- | --- |
+| [Defuddle](defuddle.md) | Implemented; run required | Bundled parser, selectors, process failures, concurrency and packaging |
+| [Retained HTML](retained-html.md) | Implemented; run required | Offline content/provenance, formats, decoding, byte budgets, errors and legacy compatibility |
+| [Failure propagation](failure-propagation.md) | Implemented; run required | Public detailed result must retain extraction failure |
+
+## Launch
+
+Follow [setup](../setup/verification.md). These are one-shot Ruby commands; no
+persistent process is launched. Start an evidence directory for this run:
+
+```bash
+mkdir -p tmp/verification
+receipt_dir="$(mktemp -d "$PWD/tmp/verification/run-XXXXXX")"
+export receipt_dir
+```
+
+Keep the same shell variable for the commands below. Do not pipe through `tee`
+without preserving the original exit status.
+
+## Doctor
+
+```bash
+ruby -v > "$receipt_dir/ruby.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/ruby.exit"
+BUNDLE_FROZEN=true bundle check > "$receipt_dir/bundle.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/bundle.exit"
+git rev-parse HEAD > "$receipt_dir/head.txt"
+git status --short > "$receipt_dir/status.txt"
+git diff --binary HEAD > "$receipt_dir/working-tree.patch"
+BUNDLE_FROZEN=true bundle exec ruby -Ilib -r nous -e 'puts Nous::VERSION; puts $LOADED_FEATURES.grep(%r{/lib/nous.rb$})' > "$receipt_dir/target.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/target.exit"
+git ls-files --others --exclude-standard -z | BUNDLE_FROZEN=true bundle exec ruby -rdigest -e 'STDIN.read.split("\0").each { |path| puts "#{Digest::SHA256.file(path).hexdigest}  #{path}" if File.file?(path) }' > "$receipt_dir/untracked.sha256"
+```
+
+Inspect each exit file and the target path before driving: Ruby, bundle and target
+must exit zero and target must resolve this checkout. Untracked hashes identify
+new recipes/harnesses; retain their contents with the change. For committed work,
+HEAD identifies the source; for dirty work retain the patch and hashes too.
+
+## Drive and capture
+
+Discover opt-in checks with `bundle exec rake -T`.
+Run each selected recipe's exact command, redirect its output to the named log,
+and immediately record `$?` in its exit file. A nonzero product assertion stays
+failed; do not weaken it to current behavior. For a prerequisite failure, attempt
+one documented setup recovery, retry once, then mark blocked if unavailable.
+
+Record a receipt table with criterion, repository/revision, command, expected,
+observed, evidence path and pass/fail/blocked/not-run status. Keep failed logs.
+Do not treat successful neighbors as readiness for a failed or unimplemented goal.
+
+For Ruby or harness changes also run `BUNDLE_FROZEN=true bundle exec rspec` and
+`BUNDLE_FROZEN=true bundle exec standardrb`, capturing output and exit status.
+The draft readiness harness is opt-in and is not part of ordinary specs.
+
+## Cleanup
+
+These checks leave no server, child process, storage record or fixture changes.
+RSpec may update the ignored `.rspec_status`; it is not evidence. Keep
+`$receipt_dir` and all logs after cleanup; do not delete the evidence directory.
+Never use private captures for the synthetic fixture checks.
+
+## Coverage limits
+
+Real-world readability quality, process CPU/memory/wall-time supervision,
+Substrate integration, locked Autodidact compatibility and hosted acceptance remain
+outside this map. Neither local backend renders JavaScript.
+The caller owns immutable storage and any application lifecycle. Passing library
+recipes alone does not certify downstream adoption.

data/docs/verification/defuddle.md

@@ -0,0 +1,45 @@
+# Defuddle extraction
+
+Status: implemented; execute on the source under review.
+
+Outcomes: Defuddle is the default for `Nous.fetch`, `Fetcher` and `ExtractHtml`.
+The CLI selects Defuddle unless `--readability` is supplied. Jina is removed.
+The extractor consumes already acquired HTML and the final URL, returns title
+and Markdown through the existing result contract, honors selectors, resolves
+relative links, and keeps concurrent calls isolated. Empty content and runtime
+failures remain failures; no automatic Readability fallback occurs.
+
+Follow [launch and doctor](README.md) and [Node readiness](../setup/defuddle.md).
+Then run:
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:defuddle > "$receipt_dir/defuddle.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/defuddle.exit"
+```
+
+Require exit zero. The extractor specs exercise the actual bundled parser with
+synthetic HTML. Client specs cover JSON/field validation, empty content and propagation of a
+failed process result. `RunProcess` specs independently exercise raw process
+output, nonzero exit, output limits, reaping and timeout cleanup with controlled
+Ruby subprocesses. Missing-runtime and safe-diagnostic errors are covered through `RunProcess`. CLI specs check backend selection and observable selector behavior. Acquired-byte
+specs cover BOM/meta decoding and meaningful invalid-byte/charset failures.
+Run the retained-HTML and failure-propagation recipes, full suite and Standard too.
+
+For installed-runtime and packaging compatibility, run the separate opt-in check:
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:defuddle_package > "$receipt_dir/defuddle-package.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/defuddle-package.exit"
+```
+
+Require exit zero. The harness in `spec/verification/defuddle_package.rb` builds
+and unpacks the gem in an isolated temporary directory, verifies the loaded Nous
+path, checks Jina is absent, and asserts title/content through `ExtractHtml`.
+It owns and cleans up the temporary package on success and failure. This verifies
+that extraction does not resolve npm dependencies from the checkout.
+
+Record source identity, commands, exit statuses and logs using the verification
+guide. Runtime checks establish compatibility only with the Node/platform tested;
+controlled subprocesses do not establish Node compatibility. Synthetic fixtures
+do not establish extraction quality across live sites, JavaScript completeness,
+authentication detection or downstream Substrate acceptance.