nous 0.4.0 → 0.5.0

data/docs/verification/failure-propagation.md

@@ -0,0 +1,37 @@
+# Visible extraction failures
+
+Status: implemented; run this recipe on the target revision before claiming verification.
+This is the accepted readiness requirement, with pages-only compatibility preserved.
+
+**Outcome FP-1:** A caller invokes `Nous.fetch(..., details: true)` after successful
+acquisition of unreadable HTML. The detailed result retains a visible failure for
+that original and must not report `all_succeeded?`. An empty successful page list
+alone is insufficient for Substrate/Autodidact's explicit-outcome handoff.
+
+**Entry point:** Public `Nous.fetch`, single-page mode (`limit: 1`), detailed results.
+Default pages-only behavior and legacy runner specs are not changed by this recipe.
+
+**Prerequisites:** Follow [launch and doctor](README.md). The opt-in harness at
+`spec/verification/failure_propagation.rb` stubs one HTTP response with WebMock;
+all real network is disabled. It exercises real crawling, parsing and collection.
+No account or persistent record is created.
+
+**Drive:**
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:failure_propagation > "$receipt_dir/failure-propagation.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/failure-propagation.exit"
+```
+
+**Assertions:** Failed original URL remains visible in `failures`; `all_succeeded?`
+is false and no page is fabricated. Require exit zero. The ordinary integration suite also covers mixed acquisition and
+extraction failures, readable page order, in-flight batch limits with complete attempt
+accounting, original unexpected extractor errors, recursive Windows-1252 response
+decoding with header precedence, and fetch-mode selection at execution after
+construction or configuration changes. Configuration remains global; these checks
+do not establish isolation between concurrent top-level fetch calls.
+
+**Evidence and cleanup:** Follow [capture and cleanup](README.md). Retain the failed
+log and exit status. WebMock and Nous configuration reset with the test process.
+This focused check alone does not prove stable error categories,
+unexpected diagnostic causes, resource exhaustion or consumer integration.

data/docs/verification/retained-html.md

@@ -0,0 +1,37 @@
+# Retained HTML
+
+Status: implemented; execute on the target source before claiming verification.
+
+**Outcomes:** RH-1 accepts retained HTML bytes through `Nous::ExtractHtml.call`
+without network or input mutation, returning readable content, title and source
+metadata. RH-2 selects `:text` or default `:markdown`. RH-3 applies header charset,
+then BOM, then meta in the first 1024 bytes, then UTF-8, rejecting invalid bytes,
+NULs and unsupported requests. RH-4 enforces positive input and combined title/content
+UTF-8 output budgets. RH-5 separates expected failures from safe unexpected failures
+with diagnostic causes; concurrent calls retain their own content and metadata.
+Legacy `Extractor::Default#extract` remains compatible.
+
+**Prerequisites:** Follow [launch and doctor](README.md). The synthetic existing
+fixtures and inline examples contain no private captures. No identity or durable
+state is involved. WebMock disables network. The bundled Defuddle parser runs locally under Node; process/dependency failures
+are controlled in diagnostic tests. No real websites are requested.
+
+**Drive:**
+
+```bash
+BUNDLE_FROZEN=true bundle exec rake verify:retained_html > "$receipt_dir/retained-html.log" 2>&1
+printf '%s\n' "$?" > "$receipt_dir/retained-html.exit"
+```
+
+**Assertions:** Require exit zero. The task runs public retained-command tests and
+legacy default extractor tests. They check title/content, selector compatibility,
+unreadable failure, header/meta conflict, BOM/meta/default charset, invalid bytes,
+NULs, formats/types, real charset declarations versus examples, byte thresholds, title-inclusive multibyte budgets, safe
+unexpected causes during decoder parsing and client execution, empty final conversion and concurrent isolation. See
+[the public contract](../../README.md#retained-html-extraction) for supported policy.
+There is no durable application write to read back.
+
+**Evidence and cleanup:** Follow [capture and cleanup](README.md). Preserve log,
+exit status and source identity. This proves synthetic local parser behavior,
+not all real-world document quality, JavaScript rendering, authentication detection,
+process CPU/memory/wall-time limits or Substrate/Autodidact/hosted compatibility.

data/javascript/build.mjs

@@ -0,0 +1,24 @@
+import { build } from 'esbuild';
+import { readFile, readdir, writeFile } from 'node:fs/promises';
+
+await build({
+  entryPoints: ['javascript/defuddle.mjs'],
+  bundle: true,
+  minify: true,
+  platform: 'node',
+  format: 'esm',
+  target: 'node22',
+  outfile: 'vendor/defuddle.mjs',
+  legalComments: 'linked',
+});
+
+// Include upstream license texts, which minified packages may omit from comments.
+const lock = JSON.parse(await readFile('package-lock.json', 'utf8'));
+const notices = [];
+for (const [directory, info] of Object.entries(lock.packages)) {
+  if (!directory || info.dev) continue;
+  const files = (await readdir(directory)).filter(name => /^(license|licence|copying)(\.|$)/i.test(name));
+  if (!files.length) throw new Error(`Missing license text: ${directory}`);
+  notices.push(`${directory} (${info.version})\n${(await Promise.all(files.map(name => readFile(`${directory}/${name}`, 'utf8')))).join('\n')}`);
+}
+await writeFile('vendor/THIRD_PARTY_LICENSES.txt', notices.join('\n\n========================================\n\n'));

data/javascript/defuddle.mjs

@@ -0,0 +1,28 @@
+import { parseHTML } from 'linkedom';
+import { Defuddle } from 'defuddle/node';
+
+// Retained extraction must never reacquire content, including site-specific APIs.
+globalThis.fetch = async () => { throw new Error('Network disabled during extraction'); };
+console.log = console.warn = console.error = () => {};
+
+try {
+  if (Number(process.versions.node.split('.')[0]) < 22) throw new Error('Node.js 22+ required');
+  process.stdin.setEncoding('utf8');
+  let input = '';
+  for await (const chunk of process.stdin) input += chunk;
+  const { html, url, selector } = JSON.parse(input);
+  const { document } = parseHTML(html);
+  const result = await Defuddle(document, url, {
+    separateMarkdown: true,
+    useAsync: false,
+    contentSelector: selector || undefined,
+  });
+  process.stdout.write(JSON.stringify({
+    title: result.title || '',
+    content: result.content,
+    markdown: result.contentMarkdown,
+  }));
+} catch {
+  process.stderr.write('Defuddle extraction failed\n');
+  process.exitCode = 1;
+}

data/lib/nous/cli.rb

@@ -40,9 +40,9 @@ module Nous
     end
 
     def extractor
-      return Extractor::Jina.new if options[:jina]
+      return Extractor::Default.new(selector: options[:selector]) if options[:readability]
 
-      Extractor::Default.new(selector: options[:selector])
+      Extractor::Defuddle.new(selector: options[:selector])
     end
 
     def validate!
@@ -78,7 +78,7 @@ module Nous
         opts.on("-s", "--selector SELECTOR", "CSS selector to scope extraction") { |v| options[:selector] = v }
         opts.on("-l", "--limit N", Integer, "Maximum pages to fetch") { |v| options[:limit] = v }
         opts.on("--timeout N", Integer, "Per-request timeout in seconds (default: 15)") { |v| options[:timeout] = v }
-        opts.on("--jina", "Use Jina Reader API for extraction (handles JS-rendered sites)") { options[:jina] = true }
+        opts.on("--readability", "Use the Ruby Readability backup extractor") { options[:readability] = true }
         opts.on("-r", "--recursive", "Follow same-host links recursively") { options[:recursive] = true }
         opts.on("-d", "--debug", "Debug logging to stderr") { options[:debug] = true }
         opts.on("-v", "--version", "Show version") do

data/lib/nous/crawler/recursive_page_fetcher.rb

@@ -10,24 +10,34 @@ module Nous
         @seed_uri = Url.new(seed_url)
         @http_client = http_client
         @records = []
+        @successful_count = 0
+        @total_requested = 0
         @queue = [url_filter.canonicalize(seed_uri)]
         @seen = Set.new(queue)
       end
 
       def call
-        suppress_async_warnings unless config.debug?
-
-        open_connection do |client|
-          crawl(client)
-        end
-
-        success(payload: records)
+        prepare_crawl_logging
+        acquire_linked_records
+        report_acquired_records
       end
 
       private
 
       attr_reader :seed_uri, :http_client, :records, :queue, :seen
 
+      def prepare_crawl_logging
+        suppress_async_warnings unless config.debug?
+      end
+
+      def acquire_linked_records
+        open_connection { |client| crawl(client) }
+      end
+
+      def report_acquired_records
+        success(payload: records, metadata: {total_requested: @total_requested})
+      end
+
       def config
         Nous.configuration
       end
@@ -38,25 +48,23 @@ module Nous
 
       def fetch_and_enqueue(batch, client)
         fetch_batch(batch, client).each do |record|
-          next unless record.ok
-          break unless within_limit?
+          @total_requested += 1
+          next if record.ok && !within_limit?
 
           records << record
+          next unless record.ok
+          @successful_count += 1
           seen << record.final_url
           enqueue_links(record)
         end
       end
 
       def fetch_batch(urls, client)
-        tasks = []
-
         Async do |task|
-          urls.each do |url|
-            tasks << task.async { page_fetcher(client).fetch(url) }
+          urls.map do |url|
+            task.async { page_fetcher(client).fetch(url) }
           end
-        end.wait
-
-        tasks.map(&:wait)
+        end.wait.map(&:wait)
       end
 
       def enqueue_links(record)
@@ -69,7 +77,7 @@ module Nous
       end
 
       def within_limit?
-        records.count(&:ok) < config.limit
+        @successful_count < config.limit
       end
 
       def open_connection

data/lib/nous/decode_html.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+module Nous
+  # Owns charset precedence and byte validation for both HTML entry points.
+  class DecodeHtml < Command
+    class Error < StandardError; end
+    class InvalidContent < Error; end
+    class UnsupportedContent < Error; end
+
+    def initialize(html:, content_type:)
+      @html = html
+      @content_type = content_type
+    end
+
+    def call
+      decode_bytes
+      report_decoding
+    rescue Error => error
+      failure(error)
+    end
+
+    private
+
+    attr_reader :html, :content_type
+
+    def select_encoding(name)
+      Encoding.find(name)
+    rescue ArgumentError
+      raise UnsupportedContent, "Unsupported HTML charset"
+    end
+
+    def decode_bytes
+      raise InvalidContent, "HTML must be a byte string" unless html.is_a?(String)
+      bytes = html.b
+      declared = charset_parameter(content_type.to_s)
+      @encoding = select_encoding(declared || bom_encoding(bytes) || meta_encoding(bytes) || "UTF-8")
+      encoded = bytes.dup.force_encoding(@encoding)
+      raise InvalidContent, "HTML contains invalid encoded bytes" unless encoded.valid_encoding?
+
+      @decoded = encoded.encode(Encoding::UTF_8)
+      @decoded = @decoded.delete_prefix("\uFEFF")
+      raise InvalidContent, "HTML contains NUL characters" if @decoded.include?("\0")
+    rescue EncodingError
+      raise InvalidContent, "HTML cannot be decoded as UTF-8"
+    end
+
+    def bom_encoding(bytes)
+      return "UTF-8" if bytes.start_with?("\xEF\xBB\xBF".b)
+      return "UTF-32BE" if bytes.start_with?("\x00\x00\xFE\xFF".b)
+      return "UTF-32LE" if bytes.start_with?("\xFF\xFE\x00\x00".b)
+      return "UTF-16BE" if bytes.start_with?("\xFE\xFF".b)
+      "UTF-16LE" if bytes.start_with?("\xFF\xFE".b)
+    end
+
+    def charset_parameter(value)
+      value.scan(/(?:\A|;)\s*([^;=\s]+)\s*=\s*(?:"([^"]*)"|'([^']*)'|([^;\s]*))/).each do |name, quoted, single_quoted, bare|
+        return quoted || single_quoted || bare if name.casecmp?("charset")
+      end
+      nil
+    end
+
+    def meta_encoding(bytes)
+      document = Nokogiri::HTML(bytes.byteslice(0, 1024), nil, "ISO-8859-1")
+      document.css("meta").each do |meta|
+        return meta["charset"] if meta["charset"]
+        if meta["http-equiv"].to_s.casecmp?("content-type")
+          declared = charset_parameter(meta["content"].to_s)
+          return declared if declared
+        end
+      end
+      nil
+    end
+
+    def report_decoding
+      success(payload: @decoded, metadata: {encoding: @encoding.name})
+    end
+  end
+end

data/lib/nous/extract_html.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+module Nous
+  class ExtractHtml < Command
+    class Error < StandardError; end
+    class InvalidLimits < ArgumentError; end
+    class InvalidContent < Error; end
+    class NoContent < Error; end
+    class UnsupportedContent < Error; end
+    class InputLimitExceeded < Error; end
+    class OutputLimitExceeded < Error; end
+
+    class UnexpectedFailure < Error
+      attr_reader :cause
+
+      def initialize(cause)
+        @cause = cause
+        super("HTML extraction failed unexpectedly")
+      end
+    end
+
+    def initialize(html:, source_url:, content_type: "text/html", format: :markdown,
+      max_input_bytes: 25 * 1024 * 1024, max_output_bytes: 32 * 1024 * 1024)
+      @html = html
+      @source_url = source_url
+      @content_type = content_type
+      @format = format
+      @max_input_bytes = max_input_bytes
+      @max_output_bytes = max_output_bytes
+    end
+
+    def call
+      verify_extraction_request
+      decode_retained_html
+      extract_readable_content
+      render_content
+      verify_extracted_content
+      report_extraction
+    rescue Error, InvalidLimits => error
+      failure(error)
+    rescue => error
+      failure(UnexpectedFailure.new(error))
+    end
+
+    private
+
+    attr_reader :html, :source_url, :content_type, :format, :max_input_bytes, :max_output_bytes
+
+    def verify_extraction_request
+      unless [max_input_bytes, max_output_bytes].all? { |limit| limit.is_a?(Integer) && limit.positive? }
+        raise InvalidLimits, "Byte limits must be positive integers"
+      end
+      raise InvalidContent, "HTML must be a byte string" unless html.is_a?(String)
+      raise InputLimitExceeded, "HTML exceeds input byte limit" if html.bytesize > max_input_bytes
+      raise UnsupportedContent, "Unsupported output format" unless [:markdown, :text].include?(format)
+      unless content_type.is_a?(String) && ["text/html", "application/xhtml+xml"].include?(content_type.split(";", 2).first.to_s.strip.downcase)
+        raise UnsupportedContent, "Unsupported HTML content type"
+      end
+    end
+
+    def decode_retained_html
+      result = DecodeHtml.new(html: html, content_type: content_type).call
+      raise result.error if result.failure?
+
+      @decoded = result.payload
+      @encoding = result.metadata.fetch(:encoding)
+    rescue DecodeHtml::InvalidContent => error
+      raise InvalidContent, error.message
+    rescue DecodeHtml::UnsupportedContent => error
+      raise UnsupportedContent, error.message
+    end
+
+    def extract_readable_content
+      result = Extractor::Defuddle::Client.new(html: @decoded, url: source_url).call
+      raise result.error if result.failure?
+
+      @readable = result.payload
+      @title = @readable[:title]
+    rescue Extractor::Defuddle::Client::NoContent => error
+      raise NoContent.new("HTML has no readable content"), cause: error
+    end
+
+    def render_content
+      @content = (format == :text) ? render_plain_text : @readable[:markdown]
+    end
+
+    def render_plain_text
+      Nokogiri::HTML.fragment(@readable[:content]).tap do |fragment|
+        fragment.css("p, div, h1, h2, h3, h4, h5, h6, li, tr, br").each do |node|
+          node.add_next_sibling(Nokogiri::XML::Text.new(" ", fragment.document))
+        end
+      end.text.gsub(/\s+/, " ").strip
+    end
+
+    def verify_extracted_content
+      raise NoContent, "HTML has no readable content" if @content.strip.empty?
+      if @content.bytesize + @title.bytesize > max_output_bytes
+        raise OutputLimitExceeded, "Extracted content exceeds output byte limit"
+      end
+    end
+
+    def report_extraction
+      success(payload: ExtractedContent.new(title: @title, content: @content),
+        metadata: {source_url: source_url, content_type: content_type, encoding: @encoding, format: format, limitations: ["static_html_only", "authentication_not_detected"]})
+    end
+  end
+end

data/lib/nous/extractor/default.rb

@@ -2,8 +2,6 @@
 
 module Nous
   module Extractor
-    class ExtractionError < StandardError; end
-
     class Default < Command
       def initialize(selector: nil)
         @selector = selector

data/lib/nous/extractor/defuddle/client.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Nous
+  module Extractor
+    class Defuddle
+      class Client < Command
+        class NoContent < ExtractionError; end
+        ProcessError = RunProcess::Error
+
+        def initialize(html:, url:, selector: nil)
+          @html = html
+          @url = url
+          @selector = selector
+        end
+
+        def call
+          run_defuddle
+          validate_response
+          report_content
+        rescue NoContent, ProcessError => error
+          failure(error)
+        rescue JSON::JSONError, EncodingError
+          failure(ProcessError.new("Invalid Defuddle process response"))
+        end
+
+        private
+
+        def run_defuddle
+          request = JSON.generate(html: @html, url: @url, selector: @selector)
+          result = RunProcess.call(request: request)
+          raise result.error if result.failure?
+
+          @response = JSON.parse(result.payload, symbolize_names: true)
+        end
+
+        def validate_response
+          unless @response.is_a?(Hash) && [:title, :content, :markdown].all? { |key| valid_text?(@response[key]) }
+            raise ProcessError, "Invalid Defuddle response fields"
+          end
+          if @response[:markdown].strip.empty?
+            raise NoContent, "HTML has no readable content"
+          end
+        end
+
+        def valid_text?(value)
+          value.is_a?(String) && value.encoding == Encoding::UTF_8 && value.valid_encoding? && !value.include?("\0")
+        end
+
+        def report_content
+          success(payload: @response)
+        end
+      end
+    end
+  end
+end

data/lib/nous/extractor/defuddle/run_process.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+require "open3"
+require "timeout"
+
+module Nous
+  module Extractor
+    class Defuddle
+      class RunProcess < Command
+        class Error < ExtractionError; end
+
+        BRIDGE = File.expand_path("../../../../vendor/defuddle.mjs", __dir__)
+        OUTPUT_LIMIT = 64 * 1024 * 1024
+        TIMEOUT = 15
+
+        def initialize(request:)
+          @request = request
+        end
+
+        def call
+          execute_bridge
+          report_output
+        rescue Error => error
+          failure(error)
+        rescue Errno::ENOENT
+          failure(Error.new("Defuddle requires Node.js 22 or newer on PATH"))
+        rescue Timeout::Error
+          failure(Error.new("Defuddle extraction timed out"))
+        rescue IOError, SystemCallError
+          failure(Error.new("Invalid Defuddle process response"))
+        end
+
+        private
+
+        def execute_bridge
+          Open3.popen3("node", BRIDGE, pgroup: true) do |stdin, stdout, stderr, child|
+            workers = {}
+            begin
+              Timeout.timeout(TIMEOUT) do
+                @output = exchange_content(stdin, stdout, stderr, workers)
+                raise Error, "Defuddle extraction failed" unless child.value.success?
+              end
+            ensure
+              terminate(child.pid)
+              [stdin, stdout, stderr].each { |io| io.close unless io.closed? }
+              workers.each_value { |worker| finish_worker(worker) }
+            end
+          end
+        end
+
+        def exchange_content(stdin, stdout, stderr, workers)
+          workers[:input] = start_worker {
+            stdin.write(@request)
+            stdin.close
+          }
+          workers[:output] = start_worker { read_bounded(stdout, OUTPUT_LIMIT) }
+          workers[:diagnostics] = start_worker { read_bounded(stderr, 64 * 1024) }
+          collect_output(workers)
+        end
+
+        def collect_output(workers)
+          output = workers.fetch(:output).value
+          workers.fetch(:input).value
+          workers.fetch(:diagnostics).value
+          output
+        end
+
+        def start_worker(&operation)
+          Thread.new do
+            Thread.current.report_on_exception = false
+            operation.call
+          end
+        end
+
+        def finish_worker(worker)
+          worker.join
+        rescue
+          # Keep the original failure while joining every owned worker.
+          nil
+        end
+
+        def terminate(pid)
+          Process.kill("KILL", -pid)
+        rescue Errno::ESRCH
+          nil
+        end
+
+        def read_bounded(io, limit)
+          output = +""
+          loop do
+            output << io.readpartial(16 * 1024)
+            raise Error, "Defuddle process output exceeds byte limit" if output.bytesize > limit
+          end
+        rescue EOFError
+          output
+        end
+
+        def report_output
+          success(payload: @output)
+        end
+      end
+    end
+  end
+end

data/lib/nous/extractor/defuddle.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Nous
+  module Extractor
+    class Defuddle < Command
+      def initialize(selector: nil)
+        @selector = selector
+      end
+
+      def extract(record)
+        decoded = DecodeHtml.call(html: record.html, content_type: record.content_type)
+        raise decoded.error if decoded.failure?
+
+        result = Client.call(html: decoded.payload, url: record.final_url, selector: @selector)
+        return result if result.failure?
+
+        success(payload: ExtractedContent.new(title: result.payload[:title], content: result.payload[:markdown]))
+      rescue DecodeHtml::Error => error
+        failure(ExtractionError.new(error.message))
+      end
+    end
+  end
+end

data/lib/nous/extractor/extraction_error.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Nous
+  module Extractor
+    class ExtractionError < StandardError; end
+  end
+end

data/lib/nous/fetcher/extraction_runner.rb

@@ -11,21 +11,36 @@ module Nous
       end
 
       def call
-        pages = records.each_slice(Nous.configuration.concurrency).each_with_object([]) do |batch, results|
-          threads = batch.map { |record| Thread.new { PageExtractor.call(extractor:, record:) } }
+        extract_records
+        report_extraction_outcomes
+      end
+
+      private
+
+      attr_reader :records, :extractor, :pages, :failures
 
-          threads.each do |thread|
-            result = thread.value
-            results << result.payload if result.success?
+      def extract_records
+        @pages = []
+        @failures = []
+        records.each_slice(Nous.configuration.concurrency) do |batch|
+          threads = batch.map { |record| Thread.new { PageExtractor.call(extractor:, record:) } }
+          batch.zip(threads).each do |record, thread|
+            retain_outcome(record, thread.value)
           end
         end
-
-        success(payload: pages)
       end
 
-      private
+      def retain_outcome(record, result)
+        if result.success?
+          pages << result.payload
+        else
+          failures << {requested_url: record.requested_url, error: result.error}
+        end
+      end
 
-      attr_reader :records, :extractor
+      def report_extraction_outcomes
+        success(payload: pages, metadata: {failures: failures})
+      end
     end
   end
 end