nous 0.3.0 → 0.5.0

data/docs/verification/README.md

@@ -0,0 +1,77 @@
+# Nous verification
+
+The project-local [verify-nous skill](../../.agents/skills/verify-nous/SKILL.md)
+uses this guide. Initial scope is the two selected architecture candidates:
+retained-HTML extraction and visible extraction failures. This map does not
+certify all crawling, rendering, serialization or consumer behavior.
+
+| Recipe | State | Coverage |
+| --- | --- | --- |
+| [Defuddle](defuddle.md) | Implemented; run required | Bundled parser, selectors, process failures, concurrency and packaging |
+| [Retained HTML](retained-html.md) | Implemented; run required | Offline content/provenance, formats, decoding, byte budgets, errors and legacy compatibility |
+| [Failure propagation](failure-propagation.md) | Implemented; run required | Public detailed result must retain extraction failure |
+
+## Launch
+
+Follow [setup](../setup/verification.md). These are one-shot Ruby commands; no
+persistent process is launched. Start an evidence directory for this run:
+
+```bash
+mkdir -p tmp/verification
+receipt_dir="$(mktemp -d "$PWD/tmp/verification/run-XXXXXX")"
+export receipt_dir
+```
+
+Keep the same shell variable for the commands below. Do not pipe through `tee`
+without preserving the original exit status.
+
+## Doctor
+
+```bash
+ruby -v > "$receipt_dir/ruby.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/ruby.exit"
+BUNDLE_FROZEN=true bundle check > "$receipt_dir/bundle.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/bundle.exit"
+git rev-parse HEAD > "$receipt_dir/head.txt"
+git status --short > "$receipt_dir/status.txt"
+git diff --binary HEAD > "$receipt_dir/working-tree.patch"
+BUNDLE_FROZEN=true bundle exec ruby -Ilib -r nous -e 'puts Nous::VERSION; puts $LOADED_FEATURES.grep(%r{/lib/nous.rb$})' > "$receipt_dir/target.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/target.exit"
+git ls-files --others --exclude-standard -z | BUNDLE_FROZEN=true bundle exec ruby -rdigest -e 'STDIN.read.split("\0").each { |path| puts "#{Digest::SHA256.file(path).hexdigest}  #{path}" if File.file?(path) }' > "$receipt_dir/untracked.sha256"
+```
+
+Inspect each exit file and the target path before driving: Ruby, bundle and target
+must exit zero and target must resolve this checkout. Untracked hashes identify
+new recipes/harnesses; retain their contents with the change. For committed work,
+HEAD identifies the source; for dirty work retain the patch and hashes too.
+
+## Drive and capture
+
+Discover opt-in checks with `bundle exec rake -T`.
+Run each selected recipe's exact command, redirect its output to the named log,
+and immediately record `$?` in its exit file. A nonzero product assertion stays
+failed; do not weaken it to current behavior. For a prerequisite failure, attempt
+one documented setup recovery, retry once, then mark blocked if unavailable.
+
+Record a receipt table with criterion, repository/revision, command, expected,
+observed, evidence path and pass/fail/blocked/not-run status. Keep failed logs.
+Do not treat successful neighbors as readiness for a failed or unimplemented goal.
+
+For Ruby or harness changes also run `BUNDLE_FROZEN=true bundle exec rspec` and
+`BUNDLE_FROZEN=true bundle exec standardrb`, capturing output and exit status.
+The draft readiness harness is opt-in and is not part of ordinary specs.
+
+## Cleanup
+
+These checks leave no server, child process, storage record or fixture changes.
+RSpec may update the ignored `.rspec_status`; it is not evidence. Keep
+`$receipt_dir` and all logs after cleanup; do not delete the evidence directory.
+Never use private captures for the synthetic fixture checks.
+
+## Coverage limits
+
+Real-world readability quality, process CPU/memory/wall-time supervision,
+Substrate integration, locked Autodidact compatibility and hosted acceptance remain
+outside this map. Neither local backend renders JavaScript.
+The caller owns immutable storage and any application lifecycle. Passing library
+recipes alone does not certify downstream adoption.

data/docs/verification/defuddle.md

@@ -0,0 +1,45 @@
+# Defuddle extraction
+
+Status: implemented; execute on the source under review.
+
+Outcomes: Defuddle is the default for `Nous.fetch`, `Fetcher` and `ExtractHtml`.
+The CLI selects Defuddle unless `--readability` is supplied. Jina is removed.
+The extractor consumes already acquired HTML and the final URL, returns title
+and Markdown through the existing result contract, honors selectors, resolves
+relative links, and keeps concurrent calls isolated. Empty content and runtime
+failures remain failures; no automatic Readability fallback occurs.
+
+Follow [launch and doctor](README.md) and [Node readiness](../setup/defuddle.md).
+Then run:
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:defuddle > "$receipt_dir/defuddle.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/defuddle.exit"
+```
+
+Require exit zero. The extractor specs exercise the actual bundled parser with
+synthetic HTML. Client specs cover JSON/field validation, empty content and propagation of a
+failed process result. `RunProcess` specs independently exercise raw process
+output, nonzero exit, output limits, reaping and timeout cleanup with controlled
+Ruby subprocesses. Missing-runtime and safe-diagnostic errors are covered through `RunProcess`. CLI specs check backend selection and observable selector behavior. Acquired-byte
+specs cover BOM/meta decoding and meaningful invalid-byte/charset failures.
+Run the retained-HTML and failure-propagation recipes, full suite and Standard too.
+
+For installed-runtime and packaging compatibility, run the separate opt-in check:
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:defuddle_package > "$receipt_dir/defuddle-package.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/defuddle-package.exit"
+```
+
+Require exit zero. The harness in `spec/verification/defuddle_package.rb` builds
+and unpacks the gem in an isolated temporary directory, verifies the loaded Nous
+path, checks Jina is absent, and asserts title/content through `ExtractHtml`.
+It owns and cleans up the temporary package on success and failure. This verifies
+that extraction does not resolve npm dependencies from the checkout.
+
+Record source identity, commands, exit statuses and logs using the verification
+guide. Runtime checks establish compatibility only with the Node/platform tested;
+controlled subprocesses do not establish Node compatibility. Synthetic fixtures
+do not establish extraction quality across live sites, JavaScript completeness,
+authentication detection or downstream Substrate acceptance.

data/docs/verification/failure-propagation.md

@@ -0,0 +1,37 @@
+# Visible extraction failures
+
+Status: implemented; run this recipe on the target revision before claiming verification.
+This is the accepted readiness requirement, with pages-only compatibility preserved.
+
+**Outcome FP-1:** A caller invokes `Nous.fetch(..., details: true)` after successful
+acquisition of unreadable HTML. The detailed result retains a visible failure for
+that original and must not report `all_succeeded?`. An empty successful page list
+alone is insufficient for Substrate/Autodidact's explicit-outcome handoff.
+
+**Entry point:** Public `Nous.fetch`, single-page mode (`limit: 1`), detailed results.
+Default pages-only behavior and legacy runner specs are not changed by this recipe.
+
+**Prerequisites:** Follow [launch and doctor](README.md). The opt-in harness at
+`spec/verification/failure_propagation.rb` stubs one HTTP response with WebMock;
+all real network is disabled. It exercises real crawling, parsing and collection.
+No account or persistent record is created.
+
+**Drive:**
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:failure_propagation > "$receipt_dir/failure-propagation.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/failure-propagation.exit"
+```
+
+**Assertions:** Failed original URL remains visible in `failures`; `all_succeeded?`
+is false and no page is fabricated. Require exit zero. The ordinary integration suite also covers mixed acquisition and
+extraction failures, readable page order, in-flight batch limits with complete attempt
+accounting, original unexpected extractor errors, recursive Windows-1252 response
+decoding with header precedence, and fetch-mode selection at execution after
+construction or configuration changes. Configuration remains global; these checks
+do not establish isolation between concurrent top-level fetch calls.
+
+**Evidence and cleanup:** Follow [capture and cleanup](README.md). Retain the failed
+log and exit status. WebMock and Nous configuration reset with the test process.
+This focused check alone does not prove stable error categories,
+unexpected diagnostic causes, resource exhaustion or consumer integration.

data/docs/verification/retained-html.md

@@ -0,0 +1,37 @@
+# Retained HTML
+
+Status: implemented; execute on the target source before claiming verification.
+
+**Outcomes:** RH-1 accepts retained HTML bytes through `Nous::ExtractHtml.call`
+without network or input mutation, returning readable content, title and source
+metadata. RH-2 selects `:text` or default `:markdown`. RH-3 applies header charset,
+then BOM, then meta in the first 1024 bytes, then UTF-8, rejecting invalid bytes,
+NULs and unsupported requests. RH-4 enforces positive input and combined title/content
+UTF-8 output budgets. RH-5 separates expected failures from safe unexpected failures
+with diagnostic causes; concurrent calls retain their own content and metadata.
+Legacy `Extractor::Default#extract` remains compatible.
+
+**Prerequisites:** Follow [launch and doctor](README.md). The synthetic existing
+fixtures and inline examples contain no private captures. No identity or durable
+state is involved. WebMock disables network. The bundled Defuddle parser runs locally under Node; process/dependency failures
+are controlled in diagnostic tests. No real websites are requested.
+
+**Drive:**
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:retained_html > "$receipt_dir/retained-html.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/retained-html.exit"
+```
+
+**Assertions:** Require exit zero. The task runs public retained-command tests and
+legacy default extractor tests. They check title/content, selector compatibility,
+unreadable failure, header/meta conflict, BOM/meta/default charset, invalid bytes,
+NULs, formats/types, real charset declarations versus examples, byte thresholds, title-inclusive multibyte budgets, safe
+unexpected causes during decoder parsing and client execution, empty final conversion and concurrent isolation. See
+[the public contract](../../README.md#retained-html-extraction) for supported policy.
+There is no durable application write to read back.
+
+**Evidence and cleanup:** Follow [capture and cleanup](README.md). Preserve log,
+exit status and source identity. This proves synthetic local parser behavior,
+not all real-world document quality, JavaScript rendering, authentication detection,
+process CPU/memory/wall-time limits or Substrate/Autodidact/hosted compatibility.

data/javascript/build.mjs

@@ -0,0 +1,24 @@
+import { build } from 'esbuild';
+import { readFile, readdir, writeFile } from 'node:fs/promises';
+
+await build({
+  entryPoints: ['javascript/defuddle.mjs'],
+  bundle: true,
+  minify: true,
+  platform: 'node',
+  format: 'esm',
+  target: 'node22',
+  outfile: 'vendor/defuddle.mjs',
+  legalComments: 'linked',
+});
+
+// Include upstream license texts, which minified packages may omit from comments.
+const lock = JSON.parse(await readFile('package-lock.json', 'utf8'));
+const notices = [];
+for (const [directory, info] of Object.entries(lock.packages)) {
+  if (!directory || info.dev) continue;
+  const files = (await readdir(directory)).filter(name => /^(license|licence|copying)(\.|$)/i.test(name));
+  if (!files.length) throw new Error(`Missing license text: ${directory}`);
+  notices.push(`${directory} (${info.version})\n${(await Promise.all(files.map(name => readFile(`${directory}/${name}`, 'utf8')))).join('\n')}`);
+}
+await writeFile('vendor/THIRD_PARTY_LICENSES.txt', notices.join('\n\n========================================\n\n'));

data/javascript/defuddle.mjs

@@ -0,0 +1,28 @@
+import { parseHTML } from 'linkedom';
+import { Defuddle } from 'defuddle/node';
+
+// Retained extraction must never reacquire content, including site-specific APIs.
+globalThis.fetch = async () => { throw new Error('Network disabled during extraction'); };
+console.log = console.warn = console.error = () => {};
+
+try {
+  if (Number(process.versions.node.split('.')[0]) < 22) throw new Error('Node.js 22+ required');
+  process.stdin.setEncoding('utf8');
+  let input = '';
+  for await (const chunk of process.stdin) input += chunk;
+  const { html, url, selector } = JSON.parse(input);
+  const { document } = parseHTML(html);
+  const result = await Defuddle(document, url, {
+    separateMarkdown: true,
+    useAsync: false,
+    contentSelector: selector || undefined,
+  });
+  process.stdout.write(JSON.stringify({
+    title: result.title || '',
+    content: result.content,
+    markdown: result.contentMarkdown,
+  }));
+} catch {
+  process.stderr.write('Defuddle extraction failed\n');
+  process.exitCode = 1;
+}

data/lib/nous/cli.rb

@@ -40,9 +40,9 @@ module Nous
     end
 
     def extractor
-      return Extractor::Jina.new if options[:jina]
+      return Extractor::Default.new(selector: options[:selector]) if options[:readability]
 
-      Extractor::Default.new(selector: options[:selector])
+      Extractor::Defuddle.new(selector: options[:selector])
     end
 
     def validate!
@@ -78,7 +78,7 @@ module Nous
         opts.on("-s", "--selector SELECTOR", "CSS selector to scope extraction") { |v| options[:selector] = v }
         opts.on("-l", "--limit N", Integer, "Maximum pages to fetch") { |v| options[:limit] = v }
         opts.on("--timeout N", Integer, "Per-request timeout in seconds (default: 15)") { |v| options[:timeout] = v }
-        opts.on("--jina", "Use Jina Reader API for extraction (handles JS-rendered sites)") { options[:jina] = true }
+        opts.on("--readability", "Use the Ruby Readability backup extractor") { options[:readability] = true }
         opts.on("-r", "--recursive", "Follow same-host links recursively") { options[:recursive] = true }
         opts.on("-d", "--debug", "Debug logging to stderr") { options[:debug] = true }
         opts.on("-v", "--version", "Show version") do

data/lib/nous/crawler/async_page_fetcher.rb

@@ -13,20 +13,30 @@ module Nous
       def fetch(url)
         Async::Task.current.with_timeout(config.timeout) do
           result = RedirectFollower.call(client:, seed_host:, url:)
-          return skip(url, result.error.message) if result.failure?
+          return build_failed_record(url, result.error.message) if result.failure?
 
           response, final_url = result.payload
-          return skip(url, "status #{response.status}") unless response.status == 200
-          return skip(url, "non-html content") unless html?(response)
+          content_type = response.headers["content-type"].to_s
+          redirected = final_url.to_s != url
 
-          RawPage.new(url: final_url.to_s, pathname: final_url.path, html: response.read)
+          return build_failed_record(url, "status #{response.status}") unless response.status == 200
+          return build_failed_record(url, "non-html content") unless html?(content_type)
+
+          build_success_record(
+            url: url,
+            final_url: final_url.to_s,
+            pathname: final_url.path,
+            html: response.read,
+            content_type: content_type,
+            redirected: redirected
+          )
         ensure
           response&.close
         end
       rescue Async::TimeoutError
-        skip(url, "timeout after #{config.timeout}s")
+        build_failed_record(url, "timeout after #{config.timeout}s")
       rescue IOError, SocketError, Errno::ECONNREFUSED => e
-        skip(url, e.message)
+        build_failed_record(url, e.message)
       end
 
       private
@@ -37,14 +47,36 @@ module Nous
         Nous.configuration
       end
 
-      def html?(response)
-        content_type = response.headers["content-type"].to_s
+      def html?(content_type)
         HTML_CONTENT_TYPES.any? { |type| content_type.include?(type) }
       end
 
-      def skip(url, reason)
-        warn("[nous] skip #{url}: #{reason}") if config.debug?
-        nil
+      def build_success_record(url:, final_url:, pathname:, html:, content_type:, redirected:)
+        FetchRecord.new(
+          requested_url: url,
+          final_url: final_url,
+          pathname: pathname,
+          html: html,
+          content_type: content_type,
+          ok: true,
+          error: nil,
+          redirected: redirected
+        )
+      end
+
+      def build_failed_record(url, error)
+        FetchRecord.new(
+          requested_url: url,
+          final_url: nil,
+          pathname: Url.new(url).path,
+          html: nil,
+          content_type: nil,
+          ok: false,
+          error: error,
+          redirected: false
+        ).tap do |record|
+          warn("[nous] skip #{url}: #{error}") if config.debug?
+        end
       end
     end
   end

data/lib/nous/crawler/recursive_page_fetcher.rb

@@ -9,24 +9,34 @@ module Nous
       def initialize(seed_url:, http_client: nil)
         @seed_uri = Url.new(seed_url)
         @http_client = http_client
-        @pages = []
+        @records = []
+        @successful_count = 0
+        @total_requested = 0
         @queue = [url_filter.canonicalize(seed_uri)]
         @seen = Set.new(queue)
       end
 
       def call
-        suppress_async_warnings unless config.debug?
+        prepare_crawl_logging
+        acquire_linked_records
+        report_acquired_records
+      end
 
-        open_connection do |client|
-          crawl(client)
-        end
+      private
 
-        success(payload: pages)
+      attr_reader :seed_uri, :http_client, :records, :queue, :seen
+
+      def prepare_crawl_logging
+        suppress_async_warnings unless config.debug?
       end
 
-      private
+      def acquire_linked_records
+        open_connection { |client| crawl(client) }
+      end
 
-      attr_reader :seed_uri, :http_client, :pages, :queue, :seen
+      def report_acquired_records
+        success(payload: records, metadata: {total_requested: @total_requested})
+      end
 
       def config
         Nous.configuration
@@ -37,30 +47,28 @@ module Nous
       end
 
       def fetch_and_enqueue(batch, client)
-        fetch_batch(batch, client).each do |page|
-          next unless page
-          break unless within_limit?
-
-          pages << page
-          seen << page.url
-          enqueue_links(page)
+        fetch_batch(batch, client).each do |record|
+          @total_requested += 1
+          next if record.ok && !within_limit?
+
+          records << record
+          next unless record.ok
+          @successful_count += 1
+          seen << record.final_url
+          enqueue_links(record)
         end
       end
 
       def fetch_batch(urls, client)
-        tasks = []
-
         Async do |task|
-          urls.each do |url|
-            tasks << task.async { page_fetcher(client).fetch(url) }
+          urls.map do |url|
+            task.async { page_fetcher(client).fetch(url) }
           end
-        end.wait
-
-        tasks.map(&:wait)
+        end.wait.map(&:wait)
       end
 
-      def enqueue_links(page)
-        link_extractor.extract(page.url, page.html).each do |url|
+      def enqueue_links(record)
+        link_extractor.extract(record.final_url, record.html).each do |url|
           next if seen.include?(url)
 
           seen << url
@@ -69,7 +77,7 @@ module Nous
       end
 
       def within_limit?
-        pages.length < config.limit
+        @successful_count < config.limit
       end
 
       def open_connection
@@ -91,7 +99,7 @@ module Nous
       end
 
       def link_extractor
-        @link_extractor ||= LinkExtractor.new(url_filter:)
+        @link_filter ||= LinkExtractor.new(url_filter:)
       end
 
       def suppress_async_warnings

data/lib/nous/crawler/single_page_fetcher.rb

@@ -20,16 +20,26 @@ module Nous
       def call
         response = connection.get(url)
         final_url = resolve_final_url(response)
+        content_type = response.headers["content-type"].to_s
+        redirected = final_url.to_s != url
+
+        record = build_record(
+          final_url: final_url.to_s,
+          pathname: final_url.path,
+          html: response.body,
+          content_type: content_type,
+          redirected: redirected
+        )
 
-        validate_host!(final_url)
-        validate_html!(response)
+        validate!(record)
 
-        raw_page = RawPage.new(url: final_url.to_s, pathname: final_url.path, html: response.body)
-        success(payload: [raw_page])
+        success(payload: [record])
       rescue FetchError => e
-        failure(e)
+        record = build_failed_record(error: e.message)
+        success(payload: [record])
       rescue Faraday::Error => e
-        failure(FetchError.new(e.message))
+        record = build_failed_record(error: e.message)
+        success(payload: [record])
       end
 
       private
@@ -40,19 +50,49 @@ module Nous
         Nous.configuration
       end
 
+      def build_record(final_url:, pathname:, html:, content_type:, redirected:)
+        FetchRecord.new(
+          requested_url: url,
+          final_url: final_url,
+          pathname: pathname,
+          html: html,
+          content_type: content_type,
+          ok: true,
+          error: nil,
+          redirected: redirected
+        )
+      end
+
+      def build_failed_record(error:)
+        FetchRecord.new(
+          requested_url: url,
+          final_url: nil,
+          pathname: Url.new(url).path,
+          html: nil,
+          content_type: nil,
+          ok: false,
+          error: error,
+          redirected: false
+        )
+      end
+
+      def validate!(record)
+        validate_host!(record.final_url)
+        validate_html!(record.content_type)
+      end
+
       def resolve_final_url(response)
         location = response.env.url.to_s
         Url.new(location)
       end
 
       def validate_host!(final_url)
-        return if final_url.host == seed_host
+        return if Url.new(final_url).host == seed_host
 
         raise FetchError, "redirected to #{final_url} outside #{seed_host}"
       end
 
-      def validate_html!(response)
-        content_type = response.headers["content-type"].to_s
+      def validate_html!(content_type)
         return if HTML_CONTENT_TYPES.any? { |type| content_type.include?(type) }
 
         raise FetchError, "non-html content: #{content_type}"

data/lib/nous/decode_html.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+module Nous
+  # Owns charset precedence and byte validation for both HTML entry points.
+  class DecodeHtml < Command
+    class Error < StandardError; end
+    class InvalidContent < Error; end
+    class UnsupportedContent < Error; end
+
+    def initialize(html:, content_type:)
+      @html = html
+      @content_type = content_type
+    end
+
+    def call
+      decode_bytes
+      report_decoding
+    rescue Error => error
+      failure(error)
+    end
+
+    private
+
+    attr_reader :html, :content_type
+
+    def select_encoding(name)
+      Encoding.find(name)
+    rescue ArgumentError
+      raise UnsupportedContent, "Unsupported HTML charset"
+    end
+
+    def decode_bytes
+      raise InvalidContent, "HTML must be a byte string" unless html.is_a?(String)
+      bytes = html.b
+      declared = charset_parameter(content_type.to_s)
+      @encoding = select_encoding(declared || bom_encoding(bytes) || meta_encoding(bytes) || "UTF-8")
+      encoded = bytes.dup.force_encoding(@encoding)
+      raise InvalidContent, "HTML contains invalid encoded bytes" unless encoded.valid_encoding?
+
+      @decoded = encoded.encode(Encoding::UTF_8)
+      @decoded = @decoded.delete_prefix("\uFEFF")
+      raise InvalidContent, "HTML contains NUL characters" if @decoded.include?("\0")
+    rescue EncodingError
+      raise InvalidContent, "HTML cannot be decoded as UTF-8"
+    end
+
+    def bom_encoding(bytes)
+      return "UTF-8" if bytes.start_with?("\xEF\xBB\xBF".b)
+      return "UTF-32BE" if bytes.start_with?("\x00\x00\xFE\xFF".b)
+      return "UTF-32LE" if bytes.start_with?("\xFF\xFE\x00\x00".b)
+      return "UTF-16BE" if bytes.start_with?("\xFE\xFF".b)
+      "UTF-16LE" if bytes.start_with?("\xFF\xFE".b)
+    end
+
+    def charset_parameter(value)
+      value.scan(/(?:\A|;)\s*([^;=\s]+)\s*=\s*(?:"([^"]*)"|'([^']*)'|([^;\s]*))/).each do |name, quoted, single_quoted, bare|
+        return quoted || single_quoted || bare if name.casecmp?("charset")
+      end
+      nil
+    end
+
+    def meta_encoding(bytes)
+      document = Nokogiri::HTML(bytes.byteslice(0, 1024), nil, "ISO-8859-1")
+      document.css("meta").each do |meta|
+        return meta["charset"] if meta["charset"]
+        if meta["http-equiv"].to_s.casecmp?("content-type")
+          declared = charset_parameter(meta["content"].to_s)
+          return declared if declared
+        end
+      end
+      nil
+    end
+
+    def report_decoding
+      success(payload: @decoded, metadata: {encoding: @encoding.name})
+    end
+  end
+end