nous 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7636d207654dbf38a64aeec480164c0e57b3c8bf98ac8373e576f692896fb3a3
-  data.tar.gz: 62ae3b01ec837d71caf104710c42bde82df6d50e6c7acc50252f2902ef9b2046
+  metadata.gz: e4371689843d5f5275e56afaaab2630ba2a703e57e29cfb79d6d47d19040d934
+  data.tar.gz: 1402e3ab54931a99bbad484002345514142a9ade10453938c8f2feae77ef7742
 SHA512:
-  metadata.gz: af52f527a8720d46cd00f3a42814d432730d105aed05ebb2435f1546afb2140bd88fd1e2c6f4e75c0226afd5ef6c9072c6919518bae366047eb022f24b30ffcd
-  data.tar.gz: '049b133f406f694771617c34d3adfef2aa64ef3aa5608d89d98230b2f596e4cdef831e53f9ee039aa247a6c588900fc51cfd07a7b456dec3ee524e83abe08b93'
+  metadata.gz: 14a0487839b29fa46c277e341e4c33b631bb22380cbb2ee91954fcff61f264856baf981f2bc8a6e20b6da3e8a81c7c26c7178db5f96d96f5be0e2a53676d72f6
+  data.tar.gz: 8e2eeb9b3e790ea85f79d877c19176569918e3030fff096950c6de7771e3ebe9556cdeb82b16a67cb8d9c6ea166c6634c3fec6c40ea8f72ffa9168b0c4fa9370

data/.agents/skills/verify-nous/SKILL.md

@@ -0,0 +1,6 @@
+---
+name: verify-nous
+description: Verify Nous retained-HTML extraction and detailed fetch outcomes through offline Ruby recipes; use when changing extraction or assessing Substrate and Autodidact readiness.
+---
+
+Read [the verification guide](../../../docs/verification/README.md) from the repository root. Select the affected recipe, execute launch, doctor, drive, evidence capture and cleanup, and report every exit status. Follow the guide's source-identification and blocker rules. A draft or failed recipe does not certify readiness. Keep provisioning in the linked setup guide; do not add live network calls to these checks.

data/CHANGELOG.md

@@ -1,4 +1,72 @@
-## [Unreleased]
+## [0.5.0] - 2026-10-25
+
+### Added
+
+- `Nous::ExtractHtml.call` extracts retained HTML without network acquisition, with
+  Markdown or plain text, source metadata, explicit decoding errors and caller byte limits.
+
+### Changed
+
+- Defuddle is the default for URL and retained-HTML extraction. The gem bundles
+  its JavaScript dependencies and requires Node.js 22+ on macOS/Linux.
+- Readability remains available through `Extractor::Default` and `--readability`.
+  Extraction failures do not silently switch backends.
+
+### Removed
+
+- Jina Reader integration (`Extractor::Jina`, `--jina`, `JINA_API_KEY`). Neither
+  remaining backend renders JavaScript pages.
+
+### Fixed
+
+- Detailed fetch results retain extraction failures and all attempted failures from
+  concurrent crawl batches, with accurate request counts and unchanged page limits.
+
+## [0.4.0] - 2026-04-11
+
+### Added
+
+- **New `details: true` option for `Nous.fetch`** - Returns a `FetchResult` object containing both successful pages and failed fetch/extraction attempts. This enables explicit failure handling without exceptions.
+  ```ruby
+  result = Nous.fetch("https://example.com", details: true)
+  result.pages    # Array<Page> - successfully extracted
+  result.failures # [{requested_url:, error:}, ...]
+  ```
+
+- **Page metadata** - Every extracted page now includes provenance information:
+  - `extractor`: Which extractor backend was used (e.g., "Nous::Extractor::Default")
+  - `requested_url`: The original URL before any redirects
+  - `content_type`: HTTP Content-Type header from the response
+  - `redirected`: Boolean indicating if redirects occurred
+
+- **FetchRecord internal primitive** - Unified fetch result representation that captures both success and failure cases with full provenance tracking. Replaces the previous `RawPage` which only handled successful fetches.
+
+- **Configuration#single_page? helper** - Convenience predicate method for checking if the current configuration is in single-page (non-recursive) mode.
+
+### Changed
+
+- **Improved title extraction** - Title extraction now uses a fallback chain: readability extracted title → HTML `<title>` tag → first `<h1>` element. This significantly improves title reliability on pages where readability fails to identify the title.
+
+- **Reduced aggressive DOM stripping** - The default extractor now preserves more content before readability processing. Previously removed elements (`header`, `img`, `video`, `svg`, `link`) are now retained, providing better context for readability scoring and preserving useful content like captions and bylines.
+
+- **Unified fetch contract** - Both single-page and recursive crawling now use the same internal `FetchRecord` structure, ensuring consistent provenance tracking and failure handling across all fetch modes.
+
+- **Serializer schema updated** - Both text and JSON output formats now include:
+  - `pathname`: URL path component
+  - `extractor`: Which extractor processed the page
+  - Full metadata object (JSON only)
+
+### Fixed
+
+- **JSON serialization** - The JSON output now correctly includes the `pathname` field that was documented but missing in previous versions.
+
+- **Extraction failure visibility** - Previously, extraction failures were only visible with debug logging enabled. The new `FetchResult` structure makes failures programmatically accessible.
+
+### Internal Changes
+
+- **Duck-typed extractor interface** - Extractors now receive the full `FetchRecord` object and can access the fields they need (`Default` uses `record.html`, `Jina` uses `record.final_url`).
+
+- **Removed `RawPage` primitive** - Superseded by the richer `FetchRecord` which handles both success and failure uniformly.
 
 ## [0.3.0] - 2026-02-23
 
@@ -29,3 +97,9 @@
 ## [0.1.0] - 2026-02-21
 
 - Initial release
+
+[Unreleased]: https://github.com/danfrenette/nous/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/danfrenette/nous/compare/v0.3.0...v0.4.0
+[0.3.0]: https://github.com/danfrenette/nous/compare/v0.2.0...v0.3.0
+[0.2.0]: https://github.com/danfrenette/nous/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/danfrenette/nous/releases/tag/v0.1.0

data/CODING_STANDARDS.md

@@ -0,0 +1,203 @@
+# Coding standards
+
+Read these standards before implementation and include them in Anaphor's Standards
+review. They apply to this Ruby gem and its development tools. Existing code is not
+a blanket exception: when changing an affected boundary, address deviations or
+record a concrete compatibility reason in the PR. This document does not itself
+change existing public APIs.
+
+## Responsibilities and service objects
+
+Keep crawling, readable-content extraction, conversion, and serialization distinct.
+Commands perform operations; fetch records describe acquired originals; pages describe
+extracted content and provenance. Keep HTTP details in crawler and remote extractor
+adapters. Follow the existing Zeitwerk layout; `lib/nous/primitives/` is collapsed
+into the `Nous` namespace.
+
+A command exposes one public instance operation, `call`, invoked through inherited
+`.call`. Its initializer accepts explicit inputs and dependencies and stores state;
+construction must not launch processes, perform network requests, or do business
+work. Resolve environment-dependent defaults when the operation runs.
+
+Application-service instance `call` methods that describe business operations must
+contain only bareword calls to private methods named for business functions. Read
+them as prose: one named step per line, without arguments, explicit receivers,
+assignments, conditionals, loops, blocks, or explicit returns. A `rescue` clause
+is the only exception; it may map an expected exception to a failure result.
+Put decisions, resource scopes, collaborator calls, and result construction inside
+the private steps. The final step returns the service result. For example:
+
+```ruby
+def call
+  prepare_retained_content
+  extract_readable_content
+  build_extraction_result
+rescue ArgumentError => error
+  failure(error)
+end
+```
+
+A failed prerequisite must stop the sequence: raise a meaningful exception handled
+by the service or the existing command boundary. Returning a failure from a private
+step alone does not stop later steps. Keep resource acquisition, use and cleanup
+in one private scope rather than spreading temporary paths across orchestration
+steps. Do not add instance state merely to split an indivisible operation.
+
+This rule governs business-operation services, including extraction workflows. The shared class-level dispatcher and low-level value/protocol helpers
+are not business orchestration. Apply the rule when changing an existing service;
+this change does not claim that all historical commands have been migrated.
+
+Before extracting an object, name the policy, resource, or collaborator contract
+it will own. If no separate owner exists, keep the steps private in the current
+service. Do not extract forwarding layers to satisfy a line count: they add places
+to navigate without reducing the caller's obligations. Name an extraction for its
+operation, not Manager, Processor, or Utils. Keep duplication when sharing would
+require unrelated behavior flags; those flags couple operations that change apart.
+
+Use Sandi Metz's five-line methods, 100-line classes, and four-parameter guideline
+as review prompts, not mechanical limits. Count independent keyword options as
+parameters. A growing constructor suggests mixed responsibilities; hiding those
+options in a hash does not fix that. Introduce an input object only when its data
+has cohesive behavior or invariants. Explain justified exceptions locally in the PR.
+
+Apply tell, don't ask: put an operation with the object that owns its behavior.
+Use a known collaborator's interface instead of branching on `is_a?`, `kind_of?`,
+or `respond_to?`. At an untrusted provider boundary, validate the actual protocol
+and required data before producing normalized content. Do not spread speculative
+type checks through in-process callers.
+
+## Building and configuring objects
+
+Prefer `tap` when configuring and returning the same object, and
+`each_with_object` when accumulating into a mutable collection, instead of a
+temporary local used only to initialize, mutate, and return that object. This
+keeps construction and its result in one expression. Use `map` for one-to-one
+transformations; keep named locals when they explain an intermediate result or
+are reused. With `each_with_object`, mutate the accumulator rather than reassign
+the block variable. These blocks belong inside private service steps, preserving
+the bareword-only business `call` convention.
+
+## Arguments and defaults
+
+For each proposed argument, inspect its callers and choose in this order:
+
+1. If no caller or accepted use case needs the choice, do not add the argument.
+   A test convenience alone does not justify expanding the public API.
+2. If every caller must supply a value, require it.
+3. If omission selects a normal value, give it a concrete default or default
+   expression. Do not use `nil` merely to replace it with that value later.
+4. If absence changes the operation, document it and permit `nil`. For example,
+   `selector: nil` leaves extraction unscoped.
+5. If automatic selection differs from an explicit override, represent that mode
+   explicitly and test both paths. Discover environment dependencies during `call`,
+   not construction, so creating the service does not launch work.
+
+For an existing argument, record the behavior of omission, explicit `nil`, and
+explicit values before editing. Preserve supported cases and test the changed
+boundary. Do not silently make a strict executable override fall back to another
+installation. Add a sentinel or null object only when those behaviors must differ;
+otherwise it adds another state every caller must understand.
+
+## Results, failures, and resources
+
+Use the existing `Nous::Command` and `Nous::Command::Result` contract in
+[lib/nous/command.rb](lib/nous/command.rb): commands return `payload`, `error`, and
+`metadata`; callers inspect `success?` or `failure?`. Extractors expose `extract(record)`
+and return the same result contract; do not rename that protocol as a style change. Do not copy Rails `ApplicationService`, named
+result fields, HTTP error codes, or reporting infrastructure into this gem.
+
+The inherited class `.call` currently converts `StandardError` into a failure
+result, wrapping post-construction errors as `CommandError`. Preserve that behavior
+unless an explicit API change is accepted. Direct instance `call` and extractor
+`extract` do not receive that dispatcher protection. A failed child command does not
+short-circuit its parent automatically: inspect its result and stop before any
+step that depends on success. Keep stage-specific errors useful without including
+credentials or sensitive source content. Do not turn a failure into empty success.
+
+One owner acquires, uses, and releases each temporary resource. Keep cleanup in a
+block or `ensure` that covers success and failure, including parsing. Do not return
+a temporary path after its owner has released it. Never use the original input as
+an output destination. Invoke subprocesses with argument arrays, not interpolated
+shell strings. Make partial external effects explicit; cleanup cannot undo them all.
+
+## Development and verification commands
+
+Choose the command's owner before adding a file:
+
+1. For a repeatable project check, add a named Rake task. Put live checks under
+   `verify:` and document inputs in the owning recipe so agents can discover them
+   with `bundle exec rake -T`. Create these directories when a check needs them.
+2. Put verification assertions, fixture preparation, and cleanup in a focused
+   harness under `spec/verification/`. Keep `tasks/*.rake` limited to task inputs
+   and dispatch, so command plumbing does not obscure what the check proves.
+3. Use `bin/` for setup and console; the existing public command lives in `exe/nous`. Do not
+   add a generic scripts directory for verification commands; it creates a second
+   entry-point convention to discover and maintain.
+
+Keep production behavior in the library and exercise its public API from the
+harness. Never add downloads or live provider calls to default `bundle exec rake`;
+ordinary verification must run without external services or model provisioning.
+
+Document required binaries, models, credentials, supported runtime versions, and
+readiness checks in `docs/setup/`. Keep executable verification steps and evidence
+limits in `docs/verification/`. Link both from onboarding. Install dependencies through explicit setup
+steps, never silently during a check; readiness and behavior verification are separate results. Retain evidence with
+the tested source revision; never claim that simulated processes prove a local
+runtime works.
+
+## Tests and review
+
+Test observable outcomes through command `.call` or the owning source interface.
+Do not assert private helper order. When a workflow has prerequisites, cover failure before dependent steps. When it
+owns files or processes, cover cleanup and input preservation on success and
+failure. When calls can overlap, prove their resources stay isolated. Use
+controlled provider/process fixtures for ordinary tests and separate opt-in live
+checks for actual compatibility. Keep fixtures' provenance and limitations visible.
+
+For Ruby changes, run affected specs, the full RSpec suite, and Standard using
+the Development commands in [README.md](README.md). Include an affected verification
+recipe once one exists. Documentation-only changes need
+link, command, and contract checks; they do not require pretending runtime behavior
+changed. Anaphor reviews must distinguish documented rules, design heuristics, and
+accepted requirements. A green suite does not establish compliance with every
+principle or prove live-provider behavior.
+
+## Retained HTML convention
+
+`ExtractHtml` keeps six explicit inputs: original bytes, source URL, content type,
+output representation and two independent caller budgets. This is a deliberate
+exception to the parameter heuristic: a hash or options object without cohesive
+behavior would only hide the same choices. The command orchestrates decoding, readable
+selection, rendering and bounded output. `DecodeHtml` owns the byte interpretation
+policy shared with acquired Defuddle HTML. Within `ExtractHtml`, invoke `DecodeHtml`
+and `Defuddle::Client` with `new(...).call`: this deliberate exception to inherited
+`.call` preserves the original unexpected exception as `UnexpectedFailure#cause`.
+The shared dispatcher wraps those exceptions and loses their identity. Keep this
+exception local to the retained command's diagnostic contract and cover failures
+during dependency execution. Trust the client's normalized title and Markdown;
+the retained command owns final rendering, emptiness and combined-byte budgets.
+Legacy shared extractors must
+not acquire per-record mutable state: the fetch runner calls them concurrently.
+
+## Sources and adaptation
+
+Adapted from Substrate's `CODING_STANDARDS.md` on October 25, 2026, which was
+adapted from Autodidact's standards. Nous-specific responsibilities, result names,
+extractor protocol, executable locations, and verification commands replace their
+Substrate equivalents. Local audio conventions do not apply to Nous.
+Retained service orchestration, explicit inputs, failure sequencing, ownership,
+and public-interface testing. Rails controllers, Active Record, transactions,
+Jbuilder, Sentry, Next.js, and that application's result shape are outside this
+project's conventions. The source document provides context, not inherited
+instructions to provision those systems.
+
+- [Thoughtbot: Tell, Don't Ask](https://thoughtbot.com/blog/tell-dont-ask)
+  informs behavior ownership and collaborator interfaces.
+- [Thoughtbot: Sandi Metz' Rules for Developers](https://thoughtbot.com/blog/sandi-metz-rules-for-developers)
+  supplies the size and parameter heuristics; their use as review prompts here is
+  a local convention.
+- [Sandi Metz: The Wrong Abstraction](https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction)
+  informs the preference for cohesive objects over premature generalization.
+
+Nullable-default policy and Rake placement are project decisions, not rules
+attributed to these authors.

data/README.md

@@ -2,10 +2,14 @@
 
 Crawl websites and extract readable Markdown, optimized for LLM consumption. Inspired by [sitefetch](https://github.com/egoist/sitefetch).
 
-Nous fetches same-host pages starting from a seed URL, extracts readable content, and outputs clean Markdown as XML-tagged text or JSON. It supports concurrent crawling, glob-based URL filtering, and two extraction backends: a local parser (ruby-readability) and the Jina Reader API for JS-rendered sites.
+Nous fetches same-host pages starting from a seed URL, extracts readable content, and outputs clean Markdown as XML-tagged text or JSON. It supports concurrent crawling, glob-based URL filtering, and local extraction with Defuddle by default and an explicit ruby-readability backup. Both parse acquired HTML without rendering JavaScript. Defuddle decodes acquired bytes using the same header/BOM/meta/UTF-8 precedence as retained HTML; invalid bytes produce extraction failures.
 
 ## Installation
 
+Default extraction requires Node.js 22+ on PATH (macOS/Linux). The gem includes
+the bundled Defuddle adapter; npm is only needed to rebuild it during development.
+See [Defuddle setup](docs/setup/defuddle.md).
+
 Add to your Gemfile:
 
 ```ruby
@@ -39,8 +43,8 @@ nous https://example.com -m "/blog/*"
 # Scope extraction to a CSS selector
 nous https://example.com -s "article.post"
 
-# Use Jina Reader API for JS-rendered sites (Next.js, SPAs)
-nous https://example.com --jina
+# Use the Ruby-only Readability backup
+nous https://example.com --readability
 
 # Debug logging
 nous https://example.com -d
@@ -57,20 +61,22 @@ nous https://example.com -d
 | `-s`, `--selector SELECTOR` | CSS selector to scope extraction | none |
 | `-l`, `--limit N` | Maximum pages to fetch | `100` |
 | `--timeout N` | Per-request timeout in seconds | `15` |
-| `--jina` | Use Jina Reader API for extraction | off |
+| `--readability` | Use the Ruby-only Readability backup | off |
 | `-v`, `--version` | Print version and exit | off |
 | `-h`, `--help` | Print usage and exit | off |
 | `-d`, `--debug` | Debug logging to stderr | off |
 
 ## Ruby API
 
+### Basic Usage
+
 ```ruby
 require "nous"
 
 # Fetch pages with the default extractor
 pages = Nous.fetch("https://example.com", limit: 10, concurrency: 3)
 
-# Each page is a Nous::Page with title, url, pathname, content
+# Each page is a Nous::Page with title, url, pathname, content, metadata
 pages.each do |page|
   puts "#{page.title} (#{page.url})"
   puts page.content
@@ -82,22 +88,161 @@ text = Nous.serialize(pages, format: :text)
 # Serialize to JSON
 json = Nous.serialize(pages, format: :json)
 
-# Use the Jina extractor for JS-heavy sites
-pages = Nous.fetch("https://spa-site.com",
-  extractor: Nous::Extractor::Jina.new,
+# Select the Readability backup explicitly
+pages = Nous.fetch("https://example.com",
+  extractor: Nous::Extractor::Default.new,
   limit: 5
 )
 ```
 
+### Detailed Results
+
+Use the `details: true` option to receive full fetch results including failures:
+
+```ruby
+result = Nous.fetch("https://example.com", details: true)
+
+result.pages       # Array<Nous::Page> - successfully extracted pages
+result.failures    # Array<{requested_url:, error:}> - failed acquisition or extraction
+result.total_requested  # Integer - total URLs attempted
+result.all_succeeded?   # Boolean - true if no failures
+result.any_succeeded?   # Boolean - true if at least one page extracted
+```
+
+Acquisition failures retain their existing messages; extraction failures retain their
+exception objects. Detailed results include failed recursive requests as well as
+failed extraction. Pages-only calls still return successful pages in source order.
+A concurrent batch may already have attempted more successful pages than the limit
+allows returning. `total_requested` counts every attempted request in that batch;
+failures remain visible, while excess successful pages are omitted from `pages`.
+The limit still governs retained successful pages, not failed attempts.
+
+This is useful when you need to handle failures explicitly:
+
+```ruby
+result = Nous.fetch("https://example.com/api-docs", details: true)
+
+if result.failures.any?
+  puts "Failed to fetch:"
+  result.failures.each do |failure|
+    puts "  #{failure[:requested_url]}: #{failure[:error]}"
+  end
+end
+
+result.pages.each do |page|
+  puts "Successfully extracted: #{page.title}"
+end
+```
+
+### Retained HTML extraction
+
+Use saved bytes directly, without crawling or reacquiring a URL:
+
+```ruby
+result = Nous::ExtractHtml.call(
+  html: File.binread("saved.html"),
+  source_url: "https://example.com/article",
+  content_type: "text/html; charset=utf-8",
+  format: :text
+)
+raise result.error if result.failure?
+puts result.payload.title
+puts result.payload.content
+puts result.metadata
+```
+
+`html:` and `source_url:` are required. The URL is provenance only; this command
+never requests it. `content_type:` defaults to `"text/html"`; HTML and XHTML are
+supported. `format:` defaults to `:markdown`; `:text` returns readable text with
+collapsed whitespace and separated blocks. Defuddle is used for both formats. Legacy `Extractor::Default#extract`
+continues producing Markdown with its existing selector behavior.
+
+Character decoding uses a declared content-type charset first, then a UTF BOM,
+then an HTML meta charset in the first 1024 bytes, then UTF-8. Input is treated as
+bytes regardless of its Ruby encoding label and is never modified. Invalid byte
+sequences and decoded NULs fail; unsupported charset names fail explicitly. A
+conflicting meta declaration does not override a declared header charset.
+
+The result payload remains `Nous::ExtractedContent` (`title`, `content`). Metadata
+contains `source_url`, original `content_type`, selected `encoding`, `format`, and
+limitations `static_html_only` and `authentication_not_detected`. Static readable
+content does not prove that an authenticated or JavaScript-rendered page is complete.
+
+`max_input_bytes:` defaults to 25 MiB; `max_output_bytes:` defaults to 32 MiB.
+Both require positive integers. Input is checked before copying/parsing. Output
+counts the combined UTF-8 bytes of title and content, excluding caller-supplied
+provenance metadata. These are input/result guards, not peak-memory, CPU or wall-time
+limits; callers needing those protections must retain process supervision.
+
+Expected errors are nested under `Nous::ExtractHtml`:
+
+| Error | Meaning |
+| --- | --- |
+| `InvalidContent` | Invalid encoded bytes, NULs, or invalid decoded output |
+| `UnsupportedContent` | Unsupported media type, output format or charset |
+| `NoContent` | No readable content or empty converted output |
+| `InputLimitExceeded` / `OutputLimitExceeded` | Caller byte budget exceeded |
+| `InvalidLimits` (`ArgumentError`) | Nonpositive or noninteger byte limit |
+| `UnexpectedFailure` | Safe outward message; original exception in `cause` |
+
+Treat diagnostic causes as private debugging data; their messages can contain
+source content. Rails acquisition, storage, lifecycle and process supervision are
+outside this command. See the [verification map](docs/verification/README.md) for
+executed surfaces and remaining consumer acceptance.
+
+### Page Structure
+
+Each extracted page contains:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `title` | String | Page title selected by the extraction backend |
+| `url` | String | Final URL after redirects |
+| `pathname` | String | URL path component |
+| `content` | String | Extracted content as Markdown |
+| `metadata` | Hash | Provenance information (see below) |
+
+### Page Metadata
+
+```ruby
+page.metadata  # => {
+  #   extractor: "Nous::Extractor::Defuddle",  # Which extractor was used
+  #   requested_url: "https://example.com/blog", # Original URL before redirects
+  #   content_type: "text/html; charset=utf-8",  # HTTP Content-Type header
+  #   redirected: true                           # Whether redirects occurred
+  # }
+```
+
 ## Extraction Backends
 
-### Default (ruby-readability)
+### Defuddle (default)
+
+Runs the bundled Defuddle parser in Node.js against the acquired HTML and final
+URL. Returns Markdown directly, retaining structure such as code blocks, tables,
+and links. Supports `--selector`. Each extraction owns its process, with a
+15-second deadline, a 64 MiB JSON response limit and a 64 KiB diagnostic limit.
+These are separate from HTTP timeouts and retained-command byte budgets.
+Page scripts do not execute and third-party fallback fetching is disabled.
+Failures remain visible through `details: true`; there is no automatic switch
+to Readability. `ExtractHtml` uses this same engine while retaining its decoding,
+plain-text rendering and byte-budget contract.
 
-Parses static HTML using [ruby-readability](https://github.com/cantino/ruby-readability), strips noisy elements (nav, footer, script, header), and converts to Markdown via [reverse_markdown](https://github.com/xijo/reverse_markdown). Fast and requires no external services, but cannot extract content from JS-rendered pages.
+### Readability (explicit backup)
 
-### Jina Reader API
+Parses static HTML using [ruby-readability](https://github.com/cantino/ruby-readability), strips noisy elements (script, style, nav, footer), and converts to Markdown via [reverse_markdown](https://github.com/xijo/reverse_markdown). Fast and requires no external services, but cannot extract content from JS-rendered pages.
 
-Uses the [Jina Reader API](https://jina.ai/reader/) which renders pages with headless Chrome. Handles Next.js App Router, React Server Components, SPAs, and other JS-heavy sites. Free tier allows 20 requests/minute without a key, or 500 RPM with a `JINA_API_KEY` environment variable.
+Title extraction uses a fallback chain:
+1. Readability's extracted title
+2. Original `<title>` tag from HTML
+3. First `<h1>` from extracted content
+
+### Migrating from Jina
+
+`Extractor::Jina` and `--jina` have been removed. Remove the extractor argument or
+flag to use Defuddle; `JINA_API_KEY` is no longer used. This retires remote browser
+rendering: pages requiring JavaScript may yield incomplete or no readable content.
+Use `Extractor::Default.new` or `--readability` for the Ruby-only backup.
+`ExtractHtml` always uses Defuddle; its result and error shapes remain unchanged.
 
 ## Output Formats
 
@@ -107,13 +252,15 @@ XML-tagged output designed for LLM context windows:
 
 ```xml
 <page>
-<title>Page Title</title>
-<url>https://example.com/page</url>
-<content>
+  <title>Page Title</title>
+  <url>https://example.com/page</url>
+  <pathname>/page</pathname>
+  <extractor>Nous::Extractor::Defuddle</extractor>
+  <content>
 # Heading
 
 Extracted markdown content...
-</content>
+  </content>
 </page>
 ```
 
@@ -125,13 +272,24 @@ Extracted markdown content...
     "title": "Page Title",
     "url": "https://example.com/page",
     "pathname": "/page",
-    "content": "# Heading\n\nExtracted markdown content..."
+    "content": "# Heading\n\nExtracted markdown content...",
+    "metadata": {
+      "extractor": "Nous::Extractor::Defuddle",
+      "requested_url": "https://example.com/page",
+      "content_type": "text/html; charset=utf-8",
+      "redirected": false
+    }
   }
 ]
 ```
 
 ## Development
 
+Read [CODING_STANDARDS.md](CODING_STANDARDS.md) before implementation and include it
+in Anaphor Standards reviews. Use the [verification map](docs/verification/README.md)
+and project-local [verify-nous skill](.agents/skills/verify-nous/SKILL.md) for the
+retained-HTML and failure-propagation readiness checks.
+
 ```bash
 bin/setup               # Install dependencies
 bundle exec rspec       # Run tests

data/Rakefile

@@ -8,3 +8,5 @@ RSpec::Core::RakeTask.new(:spec)
 require "standard/rake"
 
 task default: %i[spec standard]
+
+Dir.glob("tasks/*.rake").sort.each { |path| load path }

data/docs/setup/defuddle.md

@@ -0,0 +1,38 @@
+# Defuddle runtime
+
+Default URL and retained-HTML extraction requires Node.js 22 or newer on PATH.
+The process adapter currently targets macOS/Linux (POSIX process groups). Install
+Node explicitly using your normal runtime manager. No API key or service is used.
+Check readiness from the repository root:
+
+```bash
+node --version
+node -e 'if (Number(process.versions.node.split(".")[0]) < 22) process.exit(1)'
+printf '%s' '{"html":"<html><head><title>Probe</title></head><body><p>Readable probe.</p></body></html>","url":"https://example.com"}' | node vendor/defuddle.mjs
+```
+
+Require exit zero and JSON containing title `Probe` and Markdown `Readable probe.`.
+The published gem includes `vendor/defuddle.mjs` and third-party license notices;
+users do not run npm. A missing Node binary produces an extraction failure.
+`Nous.fetch(..., extractor: Nous::Extractor::Default.new)` and `--readability`
+select the Ruby-only backup explicitly. Retained `ExtractHtml` requires Node.
+
+## Rebuilding the bundled adapter
+
+Maintainers use the pinned dependencies in `package-lock.json`. From the repository
+root, provision dependencies explicitly and rebuild:
+
+```bash
+npm ci --ignore-scripts
+npm run build
+```
+
+Commit the source, package manifest/lockfile, generated bundle and license notices
+together. `javascript/build.mjs` regenerates the bundle and gathers dependency
+license texts. Runtime extraction and verification never install dependencies.
+Run the [Defuddle recipe](../verification/defuddle.md) after rebuilding.
+
+The bridge uses a non-executing DOM parser, disables async extractors and rejects
+fetch calls. It is not a browser renderer. The adapter uses a 15-second process
+deadline and bounded output, but does not enforce a process memory limit. Callers
+with stronger isolation requirements still own external process supervision.

data/docs/setup/verification.md

@@ -0,0 +1,21 @@
+# Verification setup
+
+Work from the Nous repository root. Use the Ruby declared in `.ruby-version`
+(currently 4.0.1), the Bundler version in `Gemfile.lock` (currently 4.0.6), and
+that lockfile's gems. With rbenv, `rbenv exec ruby -v` must identify the intended
+Ruby; shell/system Ruby 2.6 is unsuitable. Put rbenv shims on PATH or prefix the
+commands in the verification guide with `rbenv exec`.
+
+Run `BUNDLE_FROZEN=true bundle check`. If dependencies are absent, run
+`BUNDLE_FROZEN=true bundle install` once, then repeat the check. Installation may
+need network and permission to write the gem installation. If Ruby/Bundler is
+missing, provision those declared versions using your runtime manager first.
+Do not update or regenerate the lockfile to make verification work. Frozen mode
+also prevents Bundler from opportunistically rewriting platform metadata.
+
+The recipes use RSpec, WebMock, Node.js 22+ and the bundled Defuddle adapter.
+Follow [Defuddle setup and readiness](defuddle.md) before running them. No server,
+browser, remote account or model is needed. HTTP in the failure recipe is a
+controlled response fixture. WebMock blocks Ruby HTTP; the Node adapter disables
+Defuddle async fetching and rejects fetch calls.
+No credentials or retained private content belong in evidence.