extract-webpage 1.2.420 → 1.2.421

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/README.md +7 -1
  2. package/dist/extract-webpage.cjs.js +1 -1
  3. package/dist/extract-webpage.cjs.js.map +1 -1
  4. package/dist/extract-webpage.es.js +2 -4
  5. package/dist/extract-webpage.es.js.map +1 -1
  6. package/dist/html-to-content/highlight.d.ts +8 -0
  7. package/dist/index.d.ts +3 -2
  8. package/package.json +5 -5
  9. package/src/html-to-content/highlight.ts +59 -0
  10. package/src/html-to-content/html-to-content.ts +1 -2
  11. package/src/html-to-content/html-utils.ts +11 -20
  12. package/src/index.ts +5 -2
  13. package/src/url-to-content/docx-to-content.ts +9 -5
  14. package/dist/html-to-cite/extract-author.d.ts +0 -11
  15. package/dist/html-to-cite/extract-cite.d.ts +0 -33
  16. package/dist/html-to-cite/extract-date/date-extractors.d.ts +0 -40
  17. package/dist/html-to-cite/extract-date/date-validators.d.ts +0 -15
  18. package/dist/html-to-cite/extract-date/extract-date-quick.d.ts +0 -8
  19. package/dist/html-to-cite/extract-date/extract-date.d.ts +0 -26
  20. package/dist/html-to-cite/extract-source.d.ts +0 -11
  21. package/dist/html-to-cite/extract-title.d.ts +0 -11
  22. package/dist/html-to-cite/human-names-recognize.d.ts +0 -16
  23. package/dist/html-to-cite/metadata-to-cite.d.ts +0 -16
  24. package/dist/html-to-cite/url-to-domain.d.ts +0 -20
  25. package/dist/html-to-content/prism-global.d.ts +0 -12
  26. package/src/html-to-cite/extract-author.ts +0 -125
  27. package/src/html-to-cite/extract-cite.ts +0 -97
  28. package/src/html-to-cite/extract-date/date-extractors.ts +0 -484
  29. package/src/html-to-cite/extract-date/date-validators.ts +0 -191
  30. package/src/html-to-cite/extract-date/extract-date-quick.ts +0 -184
  31. package/src/html-to-cite/extract-date/extract-date.ts +0 -1049
  32. package/src/html-to-cite/extract-source.ts +0 -35
  33. package/src/html-to-cite/extract-title.ts +0 -78
  34. package/src/html-to-cite/human-names-92k.json +0 -1
  35. package/src/html-to-cite/human-names-recognize.ts +0 -396
  36. package/src/html-to-cite/metadata-to-cite.ts +0 -77
  37. package/src/html-to-cite/url-to-domain.ts +0 -50
  38. package/src/html-to-content/prism-global.d.ts +0 -40
  39. package/src/html-to-content/prism-global.ts +0 -82
package/README.md CHANGED
@@ -24,6 +24,10 @@
24
24
  </p>
25
25
  <!-- template-git-repo:badges:end -->
26
26
 
27
+ <!-- skills:install:start -->
28
+ **🤖 Agent skill** — `npx skills@latest add https://github.com/OpenSourceAGI/qwksearch-research-agent --skill extract-webpage` ([what it covers](../../skills/extract-webpage/SKILL.md))
29
+ <!-- skills:install:end -->
30
+
27
31
  ## extract-webpage
28
32
 
29
33
  Search, extract, cite, and outline the web for a topic with AI Research Agent. This package provides the core pipeline for turning URLs into structured content: fetching pages, extracting readable text, identifying citations, summarizing keyphrases, and tokenizing for search.
@@ -32,6 +36,8 @@ Search, extract, cite, and outline the web for a topic with AI Research Agent. T
32
36
  npm install extract-webpage
33
37
  ```
34
38
 
39
+ Try it in the browser: the live demo for this package and [extract-pdf](../extract-pdf), with their docs, is one Cloudflare Worker built from [`packages/extract-pdf/site`](../extract-pdf/site) (open `/demo#webpage`).
40
+
35
41
  ---
36
42
 
37
43
  ### 🚜📜 Tractor the Text Extractor
@@ -47,7 +53,7 @@ npm install extract-webpage
47
53
  3. **YouTube Transcript Processing**: When a YouTube video URL is detected, retrieve the complete video transcript including both manual captions and auto-generated subtitles, maintaining proper timestamp synchronization.
48
54
  4. **DOCX Extraction**: Extracts text and structure from Word documents.
49
55
  5. **PDF to HTML**: Extracts formatted text from PDF with parsing of linebreaks, page headers, footnotes, and section headings. Supports fonts, links, bold, italics, lists, headings, headers, footnotes, Table of Contents, Quotes, and Code Blocks. Uses [pdfjs-serverless](https://github.com/johannschopplich/pdfjs-serverless) to work in Cloudflare workers, serverless, Node.js, and front-end environments.
50
- 6. **Cite**: Identify and extract citation metadata including author names, publication dates, sources, and titles using HTML meta tags and common class name patterns. Validates author names against a database of 90,000 first and last names to distinguish personal from organizational authors.
56
+ 6. **Cite** (now the `extract-cite` package, re-exported here): Identify and extract citation metadata including author names, publication dates, sources, and titles using HTML meta tags and common class name patterns. Validates author names against a database of 90,000 first and last names to distinguish personal from organizational authors.
51
57
 
52
58
  ---
53
59