nokodown 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3f7e9d1e1249f8e65e9402b110bdd9896558944949ed5d746b38c587c5ed3753
+  data.tar.gz: 2dba5af57c129f03a1915a29cd7f387917d115e0d430f72e5d81b41035eb006a
+SHA512:
+  metadata.gz: f3526fb214d5ae49f41b919a10712e35b7b931368fd2b2bf9aa7c37252b3b30e21b03709f16670c5cf1c60beba7d457d4aa95d5a65785b42098d9fd17aea9335
+  data.tar.gz: 2909458364fd34ad9bf42bacc19cc80a6c7a8f0b44688178bf0fdbe6826f00f38b664cbbf695784461358e9945ca57136945eed2f660ac76e2c2cf47b7f5ecb0

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 0.1.0 (2026-11-02)
+
+First release.
+
+- `Nokodown.extract` returns the main content of a web page as Markdown,
+  with Readability-style metadata (title, byline, excerpt, site name,
+  published time, language).
+- `Nokodown.to_markdown` returns only the Markdown.
+- Works on articles and on home, listing, product, documentation, forum and
+  service pages.

data/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/NOTICE

@@ -0,0 +1,10 @@
+nokodown
+Copyright 2026 SerpApi
+
+The content scoring, conditional cleaning, title and metadata heuristics are
+adapted from Readability.js (https://github.com/mozilla/readability),
+licensed under the Apache License, Version 2.0. Its NOTICE reads:
+
+  Readability
+  Copyright (c) 2010 Arc90 Inc
+  Copyright (c) 2010-2026 Mozilla and Contributors

data/README.md

@@ -0,0 +1,246 @@
+<p align="center">
+  <a href="https://serpapi.com">
+    <img src="https://serpapi.com/assets/media_kit/logo-with-wordmark-gradient.svg" alt="SerpApi" width="220">
+  </a>
+</p>
+
+# nokodown
+
+Any web page to main-content Markdown, at Nokolexbor speed.
+
+Extracts the main content of a web page as clean Markdown, together with the
+metadata Readability returns (title, byline, excerpt, site name, publish date,
+language). It works on articles and on everything else: home pages, search
+results, product and pricing pages, documentation, forums.
+
+Built on [Nokolexbor](https://github.com/serpapi/nokolexbor), for the
+`include_markdown` option of SerpApi's [Extract API][extract]. The name is
+Nokolexbor's "noko" plus Markdown's "down".
+
+> **Don't want to fetch the pages yourself?** SerpApi's [Extract API][extract]
+> takes a URL and returns the page's Markdown as JSON, along with its meta
+> tags, structured data, links and images.
+> [Try the Extract API][extract].
+
+## Installation
+
+```ruby
+gem "nokodown"
+```
+
+Ruby 2.7 or newer, nokolexbor 0.6 or a later 0.x release. nokolexbor 0.7 and
+later need Ruby 3.1. Bundler picks 0.6.4 on older Rubies, but `gem install`
+does not, so install that version first:
+
+```sh
+gem install nokolexbor -v 0.6.4
+gem install nokodown
+```
+
+## Usage
+
+```ruby
+require "nokodown"
+
+result = Nokodown.extract(html, url: "https://example.com/post")
+
+result.markdown       # "# Post title\n\nFirst paragraph..."
+result.title          # "Post title"
+result.byline         # "Jane Doe"
+result.excerpt        # "A one-line summary."
+result.site_name      # "Example"
+result.published_time # "2026-09-30T08:00:00Z"
+result.lang           # "en"
+result.mode           # :article, :page, :full or :none
+
+Nokodown.to_markdown(html, url: "https://example.com/post") # just the Markdown
+```
+
+`html` is a string in any encoding. Binary strings are decoded using their
+`<meta charset>`, and invalid bytes are dropped.
+
+`Nokodown.extract`, `Nokodown.to_markdown` and `Nokodown::Result` are the
+public API. Other classes are internal and can change in any release.
+
+| Option | Default | |
+| --- | --- | --- |
+| `url` | `nil` | Page URL. Relative links and images are resolved against it (and `<base>`), and links to the page itself are reduced to their text. |
+| `main_content` | `true` | Keep only the main content. `false` converts the whole page, dropping only scripts, styles, hidden elements and form controls. |
+| `title` | `true` | Start with the page title as a `#` heading when the content has no `<h1>`. When no content is found at all (a page rendered by scripts, a bot wall), `markdown` is empty and only `result.title` has the title. |
+| `images` | `true` | Include images. Emoji images are always kept as their text. |
+| `links` | `true` | Keep links. `false` keeps only their text. |
+
+`mode` says how the content was chosen: `:article` when the page was narrowed
+to one body of prose, `:page` when the whole main region was kept, `:full`
+with `main_content: false`, `:none` for a document without a body.
+
+## What it removes
+
+- Site navigation, headers, footers, sidebars, breadcrumbs, pagination and
+  tables of contents, found through HTML semantics (`nav`, page-level `header`
+  and `footer`, ARIA roles, schema.org types) and class/id names.
+- Cookie banners, ads, newsletter and sign-in prompts, share bars, modals,
+  chat widgets, "related", "recommended" and "most popular" boxes, and, next
+  to an article, its comment thread.
+- On product pages, customer reviews and Q&A widgets, with their review
+  summaries and rating histograms. An inline rating badge ("212 reviews" next
+  to the price), testimonials and editorial reviews stay.
+- Invisible elements (`hidden`, `aria-hidden`, inline `display: none`,
+  screen-reader-only and other hiding classes), keeping collapsed accordions,
+  tab panels and FAQ answers.
+- Tracking pixels, spacers, icons, placeholders, README badges, click-tracking
+  URL parameters (`utm_*`, `gclid`, `fbclid`, ...), permalink anchors,
+  "Read more" links, code line numbers, and repeated responsive copies of the
+  same image.
+- In the main content, images marked decorative (`role="presentation"` or
+  `role="none"`, and outside an `:article` also `alt=""`) unless they sit in a
+  figure, an image link or a text-free paragraph or table cell, or declare a
+  width or height of 300 pixels or more; and card thumbnails while the
+  Markdown still has a text link to the same page. "Read more" or
+  "3 comments" never names the page, and a label without letters ("$40") or
+  with only a date ("Oct 1, 2026") names it only for an image without alt
+  text. Any other text does, so in `:page` mode a thumbnail beside a
+  "Shop now" link goes too; in an `:article`, only one outside a figure whose
+  alt is a word or two, or is mostly the link text.
+- Later copies of an image that stands alone on its line: always for an
+  image without alt text, and for one with alt text only when nothing but
+  images came since the last copy, as in a logo strip shown twice. A line
+  with several copies of one image, such as a row of rating stars, keeps
+  them all.
+
+Every structural rule is guarded: an element that contains the `<main>`
+region, or most of the page's text, is never removed because of its class name.
+
+## How it works
+
+1. **Charset**: transcode to UTF-8; Lexbor itself ignores `<meta charset>`.
+2. **Metadata**: JSON-LD, Open Graph and other `<meta>` tags, then the
+   document, in Readability's order of preference.
+3. **Preprocessor**: restore lazy-loaded images (`data-src`, `srcset`,
+   `<noscript>` copies), drop scripts, invisible elements and form controls.
+4. **Boilerplate**: remove page chrome and noise widgets across the whole page.
+5. **ContentLocator**: score paragraphs with Readability's algorithm. When one
+   candidate holds most of the page's prose the content is narrowed to it
+   (`:article`); otherwise the whole main region is kept (`:page`), because a
+   listing or a product page spreads its content over many small blocks. The
+   share required depends on the page type its metadata declares. On forum
+   threads (schema.org types or the forum software's generator tag) replies
+   are content even in comment-named containers, and on forums and Q&A pages
+   where no single post holds the page, the whole main region is cleaned as an
+   article.
+6. **Pruner**: in articles, Readability's conditional cleaning plus removal of
+   bylines, datelines and inline boxes of links to other stories; on other
+   pages, removal of leftover link clusters, unless they hold most of the text
+   as on an index page. In both modes, short links and boxes styled as
+   buttons ("Add to cart", "Book a demo") are removed, except for disclosure
+   toggles, buttons inside a sentence and titles or labels styled as buttons.
+   If article cleaning removes almost everything, it is undone and light
+   cleaning is used instead.
+7. **Markdown**: headings, emphasis, links, images, nested and ordered lists,
+   task lists, block quotes, fenced code with the language from the
+   highlighter's classes, GFM tables (rowspans and colspans filled in) for data
+   tables while layout tables are unwrapped, definition lists, details, TeX
+   from MathML, superscript and subscript digits as Unicode (10⁶, H₂O) and
+   other scripts as 2^(n-1) and a_n. Text is escaped only where it would
+   otherwise change meaning.
+
+## Quality
+
+### All kinds of pages: WCXB
+
+[WCXB][wcxb] has 2,008 human-reviewed pages: 1,050 articles and 958 product,
+collection, listing, forum, documentation and service (SaaS, pricing) pages,
+split into a development set (1,497) and a test set (511). The test set was
+not held out: some of its pages were studied while tuning nokodown, and 139
+of them have the same HTML as a development page. Scores use
+its metric: word-level F1 against the reference main content, plus the share
+of "must include" snippets found and of boilerplate snippets leaked. All
+systems ran on the same pages with links, images and tables on; their Markdown
+was converted back to text for scoring, and tokens were counted on the Markdown
+an LLM would read (o200k tokenizer). Word F1 is identical to WCXB's
+`evaluate.py`. The snippet rates average only the pages that have snippets and
+compare text with whitespace collapsed, while `evaluate.py` counts a page
+without snippets as 100% and compares raw text, so its rates differ by about a
+point.
+
+| System | F1 (dev) | F1 (test) | Boilerplate leaked (test) | Content snippets found (test) | Tokens per page (test) | Tokens on URLs (test) |
+| --- | --- | --- | --- | --- | --- | --- |
+| nokodown | 0.836 | 0.874 | 8.3% | 83.6% | 2,718 | 21% |
+| rs-trafilatura 0.1.1 | 0.838 | 0.873 | 12.7% | 87.7% | 2,664 | 18% |
+| Trafilatura 2.3.1 | 0.816 | 0.858 | 8.1% | 81.8% | 2,552 | 17% |
+| Readability.js 0.6 + Turndown | 0.770 | 0.830 | 10.0% | 76.5% | 2,899 | 23% |
+| Main-region baseline | 0.782 | 0.802 | 43.2% | 90.1% | 4,356 | 35% |
+
+F1 by page type on the test set (257 articles; 28 to 59 pages of each other type):
+
+| System | Article | Product | Collection | Listing | Forum | Documentation | Service |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| nokodown | 0.949 | 0.806 | 0.736 | 0.724 | 0.783 | 0.922 | 0.807 |
+| rs-trafilatura 0.1.1 | 0.942 | 0.740 | 0.732 | 0.714 | 0.829 | 0.953 | 0.801 |
+| Trafilatura 2.3.1 | 0.934 | 0.719 | 0.589 | 0.757 | 0.786 | 0.941 | 0.817 |
+| Readability.js 0.6 + Turndown | 0.931 | 0.634 | 0.614 | 0.637 | 0.711 | 0.882 | 0.805 |
+| Main-region baseline | 0.874 | 0.666 | 0.647 | 0.727 | 0.679 | 0.906 | 0.727 |
+
+"Main-region baseline" converts the page's `<main>`, else its only
+`<article>`, else `<body>`, with nokodown's converter and no main-content
+detection. See [benchmark/compare](benchmark/compare/README.md) to reproduce
+the table.
+
+```sh
+bundle exec ruby benchmark/wcxb.rb tmp/corpora/wcxb test
+```
+
+### Articles
+
+Scored with the [Scrapinghub article extraction benchmark][benchmark]'s metric
+(precision and recall over 4-token shingles, averaged per page). The Markdown is
+converted back to text and the leading title is removed, since both references
+leave the headline out of the body.
+
+| Corpus | Pages | F1 | Precision | Recall |
+| --- | --- | --- | --- | --- |
+| [Scrapinghub][benchmark], human-labelled article bodies | 181 | 0.945 | 0.915 | 0.978 |
+| [Mozilla Readability test pages][mozilla], Readability.js output as reference | 130 | 0.941 | 0.953 | 0.929 |
+
+Published results on the same Scrapinghub corpus, for comparison:
+Trafilatura 0.958, newspaper4k 0.949, Readability.js 0.947, go-readability
+0.934, readability-lxml 0.922, goose3 0.896, jusText 0.804, html2text 0.662.
+
+```sh
+bundle exec rake benchmark:quality   # downloads the corpora at pinned commits into tmp/
+```
+
+## Speed
+
+Per page over the Scrapinghub corpus (pages of 20 KB to 1.4 MB, median
+111 KB), median of 5 runs, on an Apple Silicon laptop with nokolexbor 0.6.4:
+
+| Ruby | Median | p90 | p99 | Max |
+| --- | --- | --- | --- | --- |
+| 2.7.8 | 6.4 ms | 11.6 ms | 30.3 ms | 59.8 ms |
+| 3.4.5 | 7.0 ms | 12.8 ms | 34.4 ms | 63.1 ms |
+
+```sh
+bundle exec rake benchmark:speed
+ruby benchmark/speed.rb DIRECTORY --per-page   # any folder of .html or .html.gz files
+```
+
+## Development
+
+```sh
+bundle install
+bundle exec rspec
+NOKOLEXBOR_VERSION=0.6.4 bundle exec rspec   # the last nokolexbor for Ruby 2.7
+```
+
+Specs use RSpec's `should` syntax.
+
+## License
+
+Apache License 2.0. The scoring and cleaning heuristics are adapted from
+[Readability.js][mozilla]; see [NOTICE](NOTICE).
+
+[benchmark]: https://github.com/scrapinghub/article-extraction-benchmark
+[extract]: https://serpapi.com/extract-api
+[mozilla]: https://github.com/mozilla/readability
+[wcxb]: https://github.com/Murrough-Foley/web-content-extraction-benchmark