html_to_plain_text 1.0.5 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/CHANGELOG.md +73 -0
- data/README.md +81 -0
- data/VERSION +1 -1
- data/html_to_plain_text.gemspec +45 -0
- data/lib/html_to_plain_text.rb +438 -77
- metadata +17 -49
- data/README.rdoc +0 -24
- data/Rakefile +0 -10
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: b4fd3fe718bfeabca2713b8c2241cc363001c0d38c9f1b415fd31267450b29d0
|
|
4
|
+
data.tar.gz: dd80dfeca05f0b8e4a18c06a6917fc882167a71156c90b5f4004359ed9125eb1
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 432942f6d7aa660ff77377e202ba756503ea98805721a65406e4a660ed23cae9c3ee4cf53432e5b9a5ae7a9e5a9fdfb1818e49cdac30dcc6d9e94c90e1304a31
|
|
7
|
+
data.tar.gz: 013afbd565c041c63d2954b256e45f7b04ca34af15a8fe41a382c76e757b64b96216b96419886254c03b40cb8f91039bf6c524e69d2b07184415fa20a83b8d03
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
All notable changes to this project will be documented in this file.
|
|
3
|
+
|
|
4
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
5
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## 1.1.0
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Markdown output can now be generated with the `markdown` method or by calling `plain_text` with the new `markdown: true` option. Headings, bold, italic, strikethrough, inline code, links, images, blockquotes, code blocks, lists, horizontal rules, and data tables are converted to Markdown syntax.
|
|
12
|
+
- New `all_tables` option to format all tables as data tables regardless of their markup.
|
|
13
|
+
- The `template`, `svg`, `math`, `canvas`, `audio`, `video`, `select`, and `textarea` tags are now stripped from the output.
|
|
14
|
+
- The `main`, `figcaption`, `caption`, `summary`, `details`, `form`, `fieldset`, and `hgroup` tags are now formatted as block elements.
|
|
15
|
+
- The `menu` tag is now formatted as a bulleted list.
|
|
16
|
+
- New `ignore_nav` option to suppress non-content elements: `header`, `footer`, and `nav` tags along with any elements that have a `role` attribute of `navigation`, `banner`, or `contentinfo`.
|
|
17
|
+
- New `selector` option to limit output to elements matching a CSS selector.
|
|
18
|
+
|
|
19
|
+
### Changed
|
|
20
|
+
|
|
21
|
+
- Data tables are now detected by the presence of a `thead` or `tbody` element in addition to a non-zero `border` attribute since the `border` attribute is deprecated in HTML5.
|
|
22
|
+
- `mailto:` and `tel:` link URLs are now included in the output when the link text differs from the address.
|
|
23
|
+
- Return an empty string instead of `nil` when the parsed document has no body.
|
|
24
|
+
- Line breaks are normalized and whitespace stripped on plain text input for consistency with HTML input.
|
|
25
|
+
- Minimum required Ruby version is now 2.7.
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
|
|
29
|
+
- Data tables with an explicit `tbody`, `thead`, or `tfoot` no longer lose their pipe separators.
|
|
30
|
+
- Trailing spaces are no longer removed from inside `pre` tag content.
|
|
31
|
+
- Fixed quadratic slowdown when converting large documents.
|
|
32
|
+
- Multiline `href` values are rejected so `javascript:` URLs can no longer leak into the output.
|
|
33
|
+
|
|
34
|
+
## 1.0.5
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
Only add pipes on tables if border attributes set to non-zero value.
|
|
39
|
+
|
|
40
|
+
## 1.0.4
|
|
41
|
+
|
|
42
|
+
### Changed
|
|
43
|
+
|
|
44
|
+
Small tweak to outputing link URLs when they don't make sense.
|
|
45
|
+
|
|
46
|
+
## 1.0.3
|
|
47
|
+
|
|
48
|
+
### Changed
|
|
49
|
+
|
|
50
|
+
- improve performance slightly by replacing runtime strings with constants
|
|
51
|
+
- testing on modern rubies (grosser)
|
|
52
|
+
- using gemspec in Gemfile (grosser)
|
|
53
|
+
- not shipping test files for smaller gem / faster installs / smaller cached gems (grosser)
|
|
54
|
+
- rake bump:patch -> increment version (grosser)
|
|
55
|
+
- rake release -> ship new version (grosser)
|
|
56
|
+
|
|
57
|
+
## 1.0.2
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
|
|
61
|
+
- remove trailing whitespace on converted text.
|
|
62
|
+
|
|
63
|
+
## 1.0.1
|
|
64
|
+
|
|
65
|
+
### Changed
|
|
66
|
+
|
|
67
|
+
- better handling of non-html or nil text
|
|
68
|
+
|
|
69
|
+
## 1.0.0
|
|
70
|
+
|
|
71
|
+
### Added
|
|
72
|
+
|
|
73
|
+
- initial release
|
data/README.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# HTML To Plain Text
|
|
2
|
+
|
|
3
|
+
[](https://github.com/bdurand/html_to_plain_text/actions/workflows/continuous_integration.yml)
|
|
4
|
+
[](https://github.com/testdouble/standard)
|
|
5
|
+
[](https://badge.fury.io/rb/html_to_plain_text)
|
|
6
|
+
|
|
7
|
+
A simple gem that provide code to convert HTML into a plain text alternative. Line breaks from HTML block level elements will be maintained. Lists and tables will also maintain a little bit of formatting.
|
|
8
|
+
|
|
9
|
+
* Line breaks will be approximated using the generally established default margins for HTML tags (i.e. <p> tag generates two line breaks, <div> generates one)
|
|
10
|
+
* Lists items will be numbered or bulleted with an asterisk
|
|
11
|
+
* <br> tags will add line breaks
|
|
12
|
+
* <hr> tags will add a string of hyphens to serve as a horizontal rule
|
|
13
|
+
* <table> elements will enclosed in "|" delimiters
|
|
14
|
+
* <a> tags will have the href URL appended to the text in parentheses
|
|
15
|
+
* Formatting tags like <strong> or <b> will be stripped
|
|
16
|
+
* Formatting inside <pre> or <plaintext> elements will be honored
|
|
17
|
+
* Code-like tags like <script> or <style> will be stripped
|
|
18
|
+
|
|
19
|
+
HTML can also be converted to a Markdown approximation instead. In Markdown mode:
|
|
20
|
+
|
|
21
|
+
* Headings are prefixed with # characters
|
|
22
|
+
* Formatting tags like <strong>, <em>, <del>, and <code> are converted to Markdown markers
|
|
23
|
+
* <a> tags become Markdown links and <img> tags become Markdown images
|
|
24
|
+
* <blockquote> content is prefixed with "> "
|
|
25
|
+
* <pre> blocks become fenced code blocks
|
|
26
|
+
* Data tables are formatted as Markdown tables with a header separator row
|
|
27
|
+
|
|
28
|
+
## Usage
|
|
29
|
+
|
|
30
|
+
```ruby
|
|
31
|
+
require 'html_to_plain_text'
|
|
32
|
+
|
|
33
|
+
html = "<h1>Hello</h1><p>world!</p>"
|
|
34
|
+
HtmlToPlainText.plain_text(html) # => "Hello\n\nworld!"
|
|
35
|
+
|
|
36
|
+
HtmlToPlainText.markdown(html) # => "# Hello\n\nworld!"
|
|
37
|
+
# equivalent to HtmlToPlainText.plain_text(html, markdown: true)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Options
|
|
41
|
+
|
|
42
|
+
* `show_links` (default `true`) - Include link URLs and image sources in the output.
|
|
43
|
+
* `markdown` (default `false`) - Format the output as Markdown instead of plain text.
|
|
44
|
+
* `all_tables` (default `false`) - Format all tables as data tables. By default only tables with a non-zero
|
|
45
|
+
`border` attribute or a `thead` or `tbody` element are formatted as data tables; other tables are assumed
|
|
46
|
+
to be used for layout only and their cells are separated with spaces instead of "|" delimiters.
|
|
47
|
+
* `ignore_nav` (default `false`) - Suppress non-content elements from the output. When set, `header`,
|
|
48
|
+
`footer`, and `nav` tags are omitted along with any elements that have a `role` attribute of
|
|
49
|
+
`navigation`, `banner`, or `contentinfo`.
|
|
50
|
+
* `selector` (default `nil`) - A CSS selector limiting the output to matching elements. Only the contents
|
|
51
|
+
of elements matching the selector are included in the output (e.g. `selector: "#main, article"`). Only
|
|
52
|
+
elements within the body of the document are matched. An `ArgumentError` is raised if the selector is
|
|
53
|
+
not a valid CSS selector.
|
|
54
|
+
|
|
55
|
+
## Installation
|
|
56
|
+
|
|
57
|
+
Add this line to your application's Gemfile:
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
gem "html_to_plain_text"
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
And then execute:
|
|
64
|
+
```bash
|
|
65
|
+
$ bundle install
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Or install it yourself as:
|
|
69
|
+
```bash
|
|
70
|
+
$ gem install html_to_plain_text
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
## Contributing
|
|
74
|
+
|
|
75
|
+
Open a pull request on GitHub.
|
|
76
|
+
|
|
77
|
+
Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
|
|
78
|
+
|
|
79
|
+
## License
|
|
80
|
+
|
|
81
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.0
|
|
1
|
+
1.1.0
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
Gem::Specification.new do |spec|
|
|
4
|
+
spec.name = "html_to_plain_text"
|
|
5
|
+
spec.version = File.read(File.expand_path("../VERSION", __FILE__)).strip
|
|
6
|
+
spec.authors = ["Brian Durand"]
|
|
7
|
+
spec.email = ["bbdurand@gmail.com"]
|
|
8
|
+
|
|
9
|
+
spec.summary = "A simple library for converting HTML into plain text."
|
|
10
|
+
|
|
11
|
+
spec.homepage = "https://github.com/bdurand/html_to_plain_text"
|
|
12
|
+
spec.license = "MIT"
|
|
13
|
+
|
|
14
|
+
spec.metadata = {
|
|
15
|
+
"homepage_uri" => spec.homepage,
|
|
16
|
+
"source_code_uri" => spec.homepage,
|
|
17
|
+
"changelog_uri" => "#{spec.homepage}/blob/master/CHANGELOG.md"
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
# Specify which files should be added to the gem when it is released.
|
|
21
|
+
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
|
22
|
+
ignore_files = %w[
|
|
23
|
+
.
|
|
24
|
+
AGENTS.md
|
|
25
|
+
Appraisals
|
|
26
|
+
Gemfile
|
|
27
|
+
Gemfile.lock
|
|
28
|
+
Rakefile
|
|
29
|
+
bin/
|
|
30
|
+
gemfiles/
|
|
31
|
+
spec/
|
|
32
|
+
test_app/
|
|
33
|
+
]
|
|
34
|
+
spec.files = Dir.chdir(File.expand_path("..", __FILE__)) do
|
|
35
|
+
`git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
spec.require_paths = ["lib"]
|
|
39
|
+
|
|
40
|
+
spec.required_ruby_version = ">= 2.7"
|
|
41
|
+
|
|
42
|
+
spec.add_dependency "nokogiri", ">=1.4.0"
|
|
43
|
+
|
|
44
|
+
spec.add_development_dependency "bundler"
|
|
45
|
+
end
|
data/lib/html_to_plain_text.rb
CHANGED
|
@@ -1,130 +1,453 @@
|
|
|
1
|
-
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "nokogiri"
|
|
2
4
|
|
|
3
5
|
# The main method on this module +plain_text+ will convert a string of HTML to a plain text approximation.
|
|
4
6
|
module HtmlToPlainText
|
|
5
|
-
IGNORE_TAGS = %w
|
|
6
|
-
|
|
7
|
-
|
|
7
|
+
IGNORE_TAGS = %w[script noscript style object applet iframe template svg math canvas audio video select option optgroup datalist textarea].each_with_object({}) { |t, h|
|
|
8
|
+
h[t] = true
|
|
9
|
+
}.freeze
|
|
10
|
+
PARAGRAPH_TAGS = %w[p h1 h2 h3 h4 h5 h6 table ol ul menu dl dd blockquote dialog figure aside section].each_with_object({}) { |t, h|
|
|
11
|
+
h[t] = true
|
|
12
|
+
}.freeze
|
|
13
|
+
BLOCK_TAGS = %w[div address li dt center del article header footer nav pre legend tr main figcaption caption summary details form fieldset hgroup].each_with_object({}) { |t, h|
|
|
14
|
+
h[t] = true
|
|
15
|
+
}.freeze
|
|
16
|
+
NAV_TAGS = %w[header footer nav].each_with_object({}) { |t, h|
|
|
17
|
+
h[t] = true
|
|
18
|
+
}.freeze
|
|
19
|
+
NAV_ROLES = %w[navigation banner contentinfo].each_with_object({}) { |t, h|
|
|
20
|
+
h[t] = true
|
|
21
|
+
}.freeze
|
|
22
|
+
MARKDOWN_INLINE_TAGS = {
|
|
23
|
+
"strong" => "**",
|
|
24
|
+
"b" => "**",
|
|
25
|
+
"em" => "*",
|
|
26
|
+
"i" => "*",
|
|
27
|
+
"del" => "~~",
|
|
28
|
+
"s" => "~~",
|
|
29
|
+
"strike" => "~~",
|
|
30
|
+
"code" => "`"
|
|
31
|
+
}.freeze
|
|
32
|
+
MARKDOWN_HEADING_TAGS = {
|
|
33
|
+
"h1" => "# ",
|
|
34
|
+
"h2" => "## ",
|
|
35
|
+
"h3" => "### ",
|
|
36
|
+
"h4" => "#### ",
|
|
37
|
+
"h5" => "##### ",
|
|
38
|
+
"h6" => "###### "
|
|
39
|
+
}.freeze
|
|
8
40
|
WHITESPACE = [" ", "\n", "\r"].freeze
|
|
9
|
-
PLAINTEXT = "plaintext"
|
|
10
|
-
PRE = "pre"
|
|
11
|
-
BR = "br"
|
|
12
|
-
HR = "hr"
|
|
13
|
-
TD = "td"
|
|
14
|
-
TH = "th"
|
|
15
|
-
TR = "tr"
|
|
16
|
-
OL = "ol"
|
|
17
|
-
UL = "ul"
|
|
18
|
-
LI = "li"
|
|
19
|
-
A = "a"
|
|
20
|
-
|
|
41
|
+
PLAINTEXT = "plaintext"
|
|
42
|
+
PRE = "pre"
|
|
43
|
+
BR = "br"
|
|
44
|
+
HR = "hr"
|
|
45
|
+
TD = "td"
|
|
46
|
+
TH = "th"
|
|
47
|
+
TR = "tr"
|
|
48
|
+
OL = "ol"
|
|
49
|
+
UL = "ul"
|
|
50
|
+
LI = "li"
|
|
51
|
+
A = "a"
|
|
52
|
+
IMG = "img"
|
|
53
|
+
MENU = "menu"
|
|
54
|
+
BLOCKQUOTE = "blockquote"
|
|
55
|
+
TABLE = "table"
|
|
56
|
+
THEAD = "thead"
|
|
57
|
+
TBODY = "tbody"
|
|
21
58
|
NUMBERS = ["1", "a"].freeze
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
59
|
+
LINK_URL_PATTERN = /\A(?:[a-z][a-z0-9.+-]*:\/\/[a-z0-9]|mailto:|tel:)/i
|
|
60
|
+
URI_SCHEME_PATTERN = /\A[a-z][a-z0-9.+-]*:/i
|
|
61
|
+
BRACKET_PATTERN = /[\[\]]/
|
|
62
|
+
BACKTICK_RUN_PATTERN = /`+/
|
|
63
|
+
HTML_PATTERN = /[<&]/
|
|
64
|
+
BODY_TAG_XPATH = "/html/body"
|
|
65
|
+
FIRST_TR_XPATH = ".//tr"
|
|
66
|
+
CARRIAGE_RETURN_PATTERN = /\r\n?/
|
|
67
|
+
LINE_BREAK_PATTERN = /[\n\r]/
|
|
68
|
+
NON_PROTOCOL_PATTERN = /:\/?\/?(.*)/
|
|
69
|
+
ALL_WHITESPACE_PATTERN = /[[:space:]]+/
|
|
70
|
+
NOT_WHITESPACE_PATTERN = /[^[:space:]]/
|
|
71
|
+
LEADING_WHITESPACE_PATTERN = /\A[[:space:]]*/
|
|
72
|
+
TRAILING_WHITESPACE_PATTERN = /[[:space:]]*\z/
|
|
73
|
+
LEADING_LINE_BREAK_PATTERN = /\A[\r\n]+/
|
|
74
|
+
SPACE = " "
|
|
75
|
+
TAB = "\t"
|
|
76
|
+
EMPTY = ""
|
|
77
|
+
NEWLINE = "\n"
|
|
78
|
+
PIPE = "|"
|
|
79
|
+
ESCAPED_PIPE = "\\|"
|
|
80
|
+
HREF = "href"
|
|
81
|
+
SRC = "src"
|
|
82
|
+
ALT = "alt"
|
|
83
|
+
BORDER = "border"
|
|
84
|
+
ROLE = "role"
|
|
85
|
+
TABLE_SEPARATOR = " | "
|
|
86
|
+
MARKDOWN_HR = "---\n"
|
|
87
|
+
# Hard line breaks are emitted as a marker character so that ones that would produce
|
|
88
|
+
# a stray backslash (before a blank line or at the end of the output) can be removed
|
|
89
|
+
# when the output is finalized. The null character cannot appear in parsed HTML text.
|
|
90
|
+
MARKDOWN_BR_MARKER = "\u0000"
|
|
91
|
+
MARKDOWN_BR = "#{MARKDOWN_BR_MARKER}\n"
|
|
92
|
+
MARKDOWN_HARD_BREAK = "\\"
|
|
93
|
+
MARKDOWN_BR_BEFORE_BLANK_PATTERN = /#{MARKDOWN_BR_MARKER}(?=\n[>[[:space:]]]*\n)/
|
|
94
|
+
MARKDOWN_FENCE = "```"
|
|
95
|
+
MARKDOWN_QUOTE = "> "
|
|
96
|
+
MARKDOWN_EMPTY_QUOTE = ">"
|
|
97
|
+
MARKDOWN_BULLET = "- "
|
|
98
|
+
BACKTICK = "`"
|
|
99
|
+
MARKDOWN_TABLE_SEPARATOR_CELL = " --- |"
|
|
35
100
|
|
|
36
101
|
# Helper instance method for converting HTML into plain text. This method simply calls HtmlToPlainText.plain_text.
|
|
37
|
-
|
|
38
|
-
|
|
102
|
+
#
|
|
103
|
+
# @param html [String] The HTML to convert into plain text.
|
|
104
|
+
# @param show_links [Boolean] Whether to include link URLs and image sources in the output.
|
|
105
|
+
# @param markdown [Boolean] Whether to format the output as Markdown.
|
|
106
|
+
# @param all_tables [Boolean] Whether to format all tables as data tables regardless of their markup.
|
|
107
|
+
# @param ignore_nav [Boolean] Whether to suppress navigational, header, and footer elements.
|
|
108
|
+
# @param selector [String, nil] A CSS selector limiting the output to matching elements.
|
|
109
|
+
# @return [String] The plain text approximation of the HTML.
|
|
110
|
+
# @raise [ArgumentError] If the selector is not a valid CSS selector.
|
|
111
|
+
def plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil)
|
|
112
|
+
HtmlToPlainText.plain_text(html, show_links: show_links, markdown: markdown, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
# Helper instance method for converting HTML into Markdown. This method simply calls HtmlToPlainText.markdown.
|
|
116
|
+
#
|
|
117
|
+
# @param html [String] The HTML to convert into Markdown.
|
|
118
|
+
# @param show_links [Boolean] Whether to include link URLs and image sources in the output.
|
|
119
|
+
# @param all_tables [Boolean] Whether to format all tables as data tables regardless of their markup.
|
|
120
|
+
# @param ignore_nav [Boolean] Whether to suppress navigational, header, and footer elements.
|
|
121
|
+
# @param selector [String, nil] A CSS selector limiting the output to matching elements.
|
|
122
|
+
# @return [String] The Markdown approximation of the HTML.
|
|
123
|
+
# @raise [ArgumentError] If the selector is not a valid CSS selector.
|
|
124
|
+
def markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil)
|
|
125
|
+
HtmlToPlainText.markdown(html, show_links: show_links, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
|
|
39
126
|
end
|
|
40
127
|
|
|
41
128
|
class << self
|
|
42
129
|
# Convert some HTML into a plain text approximation.
|
|
43
|
-
|
|
44
|
-
|
|
130
|
+
#
|
|
131
|
+
# @param html [String] The HTML to convert into plain text.
|
|
132
|
+
# @param show_links [Boolean] Whether to include link URLs and image sources in the output.
|
|
133
|
+
# @param markdown [Boolean] Whether to format the output as Markdown.
|
|
134
|
+
# @param all_tables [Boolean] Whether to format all tables as data tables regardless of their markup.
|
|
135
|
+
# By default only tables with a non-zero border attribute or a thead or tbody element are formatted
|
|
136
|
+
# as data tables; other tables are assumed to be for layout only.
|
|
137
|
+
# @param ignore_nav [Boolean] Whether to suppress navigational, header, and footer elements. When true,
|
|
138
|
+
# header, footer, and nav tags are omitted from the output along with any elements that have a
|
|
139
|
+
# role attribute of navigation, banner, or contentinfo.
|
|
140
|
+
# @param selector [String, nil] A CSS selector limiting the output to matching elements. Only the
|
|
141
|
+
# contents of elements matching the selector are included in the output. Only elements within
|
|
142
|
+
# the body of the document are matched.
|
|
143
|
+
# @return [String] The plain text approximation of the HTML.
|
|
144
|
+
# @raise [ArgumentError] If the selector is not a valid CSS selector.
|
|
145
|
+
def plain_text(html, show_links: true, markdown: false, all_tables: false, ignore_nav: false, selector: nil)
|
|
45
146
|
return nil if html.nil?
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
147
|
+
unless selector || HTML_PATTERN.match?(html)
|
|
148
|
+
return html.gsub(CARRIAGE_RETURN_PATTERN, NEWLINE).strip
|
|
149
|
+
end
|
|
150
|
+
document = Nokogiri::HTML::Document.parse(html)
|
|
151
|
+
options = {show_links: show_links, markdown: markdown, all_tables: all_tables, ignore_nav: ignore_nav}
|
|
152
|
+
out = +""
|
|
153
|
+
body = document.xpath(BODY_TAG_XPATH).first
|
|
154
|
+
return +"" unless body
|
|
155
|
+
if selector
|
|
156
|
+
begin
|
|
157
|
+
elements = document.css(selector)
|
|
158
|
+
rescue Nokogiri::CSS::SyntaxError => e
|
|
159
|
+
raise ArgumentError, "Invalid CSS selector: #{e.message}"
|
|
160
|
+
end
|
|
161
|
+
elements.each do |element|
|
|
162
|
+
ancestors = element.ancestors
|
|
163
|
+
# Only elements within the body are included, and nested matches are
|
|
164
|
+
# skipped since their content is already included by a matching ancestor.
|
|
165
|
+
next unless element == body || ancestors.include?(body)
|
|
166
|
+
next if ancestors.any? { |ancestor| elements.include?(ancestor) }
|
|
167
|
+
append_block_breaks(out)
|
|
168
|
+
convert_node_to_plain_text(element, out, options)
|
|
169
|
+
end
|
|
170
|
+
else
|
|
171
|
+
convert_node_to_plain_text(body, out, options)
|
|
172
|
+
end
|
|
173
|
+
# String#strip removes null characters as well as whitespace, so a hard break
|
|
174
|
+
# marker at the end of the output is removed here along with any trailing newline.
|
|
175
|
+
out = out.strip
|
|
176
|
+
if markdown
|
|
177
|
+
# Hard break markers followed by a blank line would leave a stray backslash,
|
|
178
|
+
# so they are removed; the rest become hard line break backslashes.
|
|
179
|
+
out.gsub!(MARKDOWN_BR_BEFORE_BLANK_PATTERN, EMPTY)
|
|
180
|
+
out.gsub!(MARKDOWN_BR_MARKER, MARKDOWN_HARD_BREAK)
|
|
181
|
+
end
|
|
182
|
+
out.gsub(CARRIAGE_RETURN_PATTERN, NEWLINE)
|
|
183
|
+
end
|
|
184
|
+
|
|
185
|
+
# Convert some HTML into a Markdown approximation. This is the same as calling plain_text
|
|
186
|
+
# with the markdown option set to true.
|
|
187
|
+
#
|
|
188
|
+
# @param html [String] The HTML to convert into Markdown.
|
|
189
|
+
# @param show_links [Boolean] Whether to include link URLs and image sources in the output.
|
|
190
|
+
# @param all_tables [Boolean] Whether to format all tables as data tables regardless of their markup.
|
|
191
|
+
# @param ignore_nav [Boolean] Whether to suppress navigational, header, and footer elements.
|
|
192
|
+
# @param selector [String, nil] A CSS selector limiting the output to matching elements.
|
|
193
|
+
# @return [String] The Markdown approximation of the HTML.
|
|
194
|
+
# @raise [ArgumentError] If the selector is not a valid CSS selector.
|
|
195
|
+
def markdown(html, show_links: true, all_tables: false, ignore_nav: false, selector: nil)
|
|
196
|
+
plain_text(html, show_links: show_links, markdown: true, all_tables: all_tables, ignore_nav: ignore_nav, selector: selector)
|
|
50
197
|
end
|
|
51
198
|
|
|
52
199
|
private
|
|
53
200
|
|
|
54
201
|
# Convert an HTML node to plain text. This method is called recursively with the output and
|
|
55
202
|
# formatting options for special tags.
|
|
56
|
-
def convert_node_to_plain_text(parent, out
|
|
57
|
-
if
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
203
|
+
def convert_node_to_plain_text(parent, out, options = {})
|
|
204
|
+
out = out.dup if out.frozen?
|
|
205
|
+
markdown = options[:markdown] && !options[:pre]
|
|
206
|
+
|
|
207
|
+
unless markdown && MARKDOWN_INLINE_TAGS.include?(parent.name)
|
|
208
|
+
# The buffer tail here was written before this node opened, so it is only
|
|
209
|
+
# preformatted content when an ancestor (not this node itself) is a pre tag.
|
|
210
|
+
trim = !options[:pre] || parent.name == PRE
|
|
211
|
+
if PARAGRAPH_TAGS.include?(parent.name)
|
|
212
|
+
append_paragraph_breaks(out, trim: trim)
|
|
213
|
+
heading = (MARKDOWN_HEADING_TAGS[parent.name] if markdown)
|
|
214
|
+
out << heading if heading
|
|
215
|
+
elsif BLOCK_TAGS.include?(parent.name)
|
|
216
|
+
append_block_breaks(out, trim: trim)
|
|
217
|
+
end
|
|
61
218
|
end
|
|
62
219
|
|
|
63
220
|
format_list_item(out, options) if parent.name == LI
|
|
64
|
-
out << "| " if parent.name == TR && data_table?(parent
|
|
221
|
+
out << "| " if parent.name == TR && data_table?(parent, options)
|
|
65
222
|
|
|
66
223
|
parent.children.each do |node|
|
|
67
224
|
if node.text? || node.cdata?
|
|
68
225
|
text = node.text
|
|
69
226
|
unless options[:pre]
|
|
70
|
-
text
|
|
227
|
+
text.gsub!(ALL_WHITESPACE_PATTERN, SPACE)
|
|
71
228
|
text.lstrip! if WHITESPACE.include?(out[-1, 1])
|
|
229
|
+
# Pipes in data table cells would be mistaken for column delimiters.
|
|
230
|
+
text.gsub!(PIPE, ESCAPED_PIPE) if markdown && options[:table_cell]
|
|
72
231
|
end
|
|
73
232
|
out << text
|
|
74
233
|
elsif node.name == PLAINTEXT
|
|
75
234
|
out << node.text
|
|
76
235
|
elsif node.element? && !IGNORE_TAGS.include?(node.name)
|
|
236
|
+
next if options[:ignore_nav] && ignore_nav_element?(node)
|
|
237
|
+
trim_trailing_blanks!(out) if markdown && (node.name == BLOCKQUOTE || node.name == PRE)
|
|
238
|
+
pos = out.length
|
|
77
239
|
convert_node_to_plain_text(node, out, child_options(node, options))
|
|
78
240
|
|
|
79
241
|
if node.name == BR
|
|
80
|
-
|
|
81
|
-
out << NEWLINE
|
|
242
|
+
trim_trailing_blanks!(out) unless options[:pre]
|
|
243
|
+
out << if markdown && !out.empty? && !out.end_with?(NEWLINE)
|
|
244
|
+
MARKDOWN_BR
|
|
245
|
+
else
|
|
246
|
+
NEWLINE
|
|
247
|
+
end
|
|
82
248
|
elsif node.name == HR
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
if href &&
|
|
91
|
-
href =~ ABSOLUTE_URL_PATTERN &&
|
|
92
|
-
node.text =~ NOT_WHITESPACE_PATTERN &&
|
|
93
|
-
node.text != href &&
|
|
94
|
-
node.text != href[NON_PROTOCOL_PATTERN, 1] # use only text for <a href="mailto:a@b.com">a@b.com</a>
|
|
95
|
-
out << " (#{href}) "
|
|
249
|
+
if markdown
|
|
250
|
+
append_paragraph_breaks(out)
|
|
251
|
+
out << MARKDOWN_HR
|
|
252
|
+
else
|
|
253
|
+
trim_trailing_blanks!(out)
|
|
254
|
+
out << NEWLINE unless out.end_with?(NEWLINE)
|
|
255
|
+
out << "-------------------------------\n"
|
|
96
256
|
end
|
|
257
|
+
elsif node.name == TD || node.name == TH
|
|
258
|
+
out << (data_table?(parent, options) ? TABLE_SEPARATOR : SPACE)
|
|
259
|
+
elsif node.name == A && options[:show_links]
|
|
260
|
+
format_link(out, pos, node, options)
|
|
261
|
+
elsif markdown && MARKDOWN_INLINE_TAGS.include?(node.name)
|
|
262
|
+
wrap_markdown_inline(out, pos, MARKDOWN_INLINE_TAGS[node.name])
|
|
263
|
+
elsif markdown && node.name == IMG
|
|
264
|
+
append_markdown_image(out, node) if options[:show_links]
|
|
265
|
+
elsif markdown && node.name == BLOCKQUOTE
|
|
266
|
+
format_markdown_blockquote(out, pos)
|
|
267
|
+
elsif markdown && node.name == PRE
|
|
268
|
+
format_markdown_code_block(out, pos)
|
|
269
|
+
elsif markdown && node.name == TR && data_table?(node, options)
|
|
270
|
+
flatten_markdown_table_row(out, pos)
|
|
271
|
+
append_block_breaks(out)
|
|
272
|
+
append_markdown_table_separator(out, node)
|
|
97
273
|
elsif PARAGRAPH_TAGS.include?(node.name)
|
|
98
|
-
append_paragraph_breaks(out)
|
|
274
|
+
append_paragraph_breaks(out, trim: !options[:pre])
|
|
99
275
|
elsif BLOCK_TAGS.include?(node.name)
|
|
100
|
-
|
|
276
|
+
# A closing pre tag leaves preformatted content at the buffer tail.
|
|
277
|
+
append_block_breaks(out, trim: !options[:pre] && node.name != PRE)
|
|
101
278
|
end
|
|
102
279
|
end
|
|
103
280
|
end
|
|
104
281
|
out
|
|
105
282
|
end
|
|
106
283
|
|
|
284
|
+
# Determine if an element is a navigational element suppressed by the ignore_nav option.
|
|
285
|
+
# The role attribute can contain a space delimited list of roles.
|
|
286
|
+
def ignore_nav_element?(node)
|
|
287
|
+
return true if NAV_TAGS.include?(node.name)
|
|
288
|
+
role = node[ROLE]
|
|
289
|
+
return false unless role
|
|
290
|
+
role.downcase.split(ALL_WHITESPACE_PATTERN).any? { |value| NAV_ROLES.include?(value) }
|
|
291
|
+
end
|
|
292
|
+
|
|
107
293
|
# Set formatting options that will be passed to child elements for a tag.
|
|
108
294
|
def child_options(node, options)
|
|
109
|
-
if node.name == UL
|
|
110
|
-
level = options[:ul] || -1
|
|
111
|
-
level
|
|
112
|
-
options.merge(:list => :ul, :ul => level)
|
|
295
|
+
if node.name == UL || node.name == MENU
|
|
296
|
+
level = (options[:ul] || -1) + 1
|
|
297
|
+
options.merge(list: :ul, ul: level, indent: options[:child_indent].to_s)
|
|
113
298
|
elsif node.name == OL
|
|
114
|
-
level = options[:ol] || -1
|
|
115
|
-
level
|
|
116
|
-
options.merge(:
|
|
299
|
+
level = (options[:ol] || -1) + 1
|
|
300
|
+
number = options[:markdown] ? NUMBERS[0] : NUMBERS[level % 2]
|
|
301
|
+
options.merge(list: :ol, ol: level, number: number, indent: options[:child_indent].to_s)
|
|
117
302
|
elsif node.name == PRE
|
|
118
|
-
options.merge(:
|
|
303
|
+
options.merge(pre: true)
|
|
304
|
+
elsif options[:markdown] && (node.name == TD || node.name == TH) && data_table?(node, options)
|
|
305
|
+
options.merge(table_cell: true)
|
|
119
306
|
else
|
|
120
307
|
options
|
|
121
308
|
end
|
|
122
309
|
end
|
|
123
310
|
|
|
311
|
+
# Append the URL for a link to the output. In markdown mode the link content is rewritten
|
|
312
|
+
# using markdown link syntax instead.
|
|
313
|
+
def format_link(out, pos, node, options)
|
|
314
|
+
href = node[HREF]
|
|
315
|
+
return unless href && href =~ LINK_URL_PATTERN
|
|
316
|
+
# A multiline href could smuggle a non-link protocol URL past the pattern check.
|
|
317
|
+
return if LINE_BREAK_PATTERN.match?(href)
|
|
318
|
+
|
|
319
|
+
if options[:markdown] && !options[:pre]
|
|
320
|
+
content = out.slice!(pos..).to_s
|
|
321
|
+
stripped = content.strip
|
|
322
|
+
if stripped.empty? || stripped == href || stripped == href[NON_PROTOCOL_PATTERN, 1]
|
|
323
|
+
out << content
|
|
324
|
+
else
|
|
325
|
+
out << content[LEADING_WHITESPACE_PATTERN] << "[" << escape_unbalanced_brackets(stripped) << "](" << href << ")" << content[TRAILING_WHITESPACE_PATTERN]
|
|
326
|
+
end
|
|
327
|
+
else
|
|
328
|
+
text = node.text
|
|
329
|
+
text.gsub!(ALL_WHITESPACE_PATTERN, SPACE)
|
|
330
|
+
text.strip!
|
|
331
|
+
if text.size > 0 &&
|
|
332
|
+
text != href &&
|
|
333
|
+
text != href[NON_PROTOCOL_PATTERN, 1] # use only text for <a href="mailto:a@b.com">a@b.com</a>
|
|
334
|
+
out << " (#{href}) "
|
|
335
|
+
end
|
|
336
|
+
end
|
|
337
|
+
end
|
|
338
|
+
|
|
339
|
+
# Wrap the output generated by an inline formatting element in markdown markers,
|
|
340
|
+
# keeping any surrounding whitespace outside of the markers.
|
|
341
|
+
def wrap_markdown_inline(out, pos, marker)
|
|
342
|
+
content = out.slice!(pos..).to_s
|
|
343
|
+
stripped = content.strip
|
|
344
|
+
if stripped.empty?
|
|
345
|
+
out << content
|
|
346
|
+
else
|
|
347
|
+
if marker == BACKTICK && stripped.include?(BACKTICK)
|
|
348
|
+
# A code span containing backticks needs a longer delimiter, padded with
|
|
349
|
+
# spaces if the content starts or ends with a backtick.
|
|
350
|
+
longest_run = stripped.scan(BACKTICK_RUN_PATTERN).max_by(&:length).length
|
|
351
|
+
marker = BACKTICK * (longest_run + 1)
|
|
352
|
+
stripped = "#{SPACE}#{stripped}#{SPACE}" if stripped.start_with?(BACKTICK) || stripped.end_with?(BACKTICK)
|
|
353
|
+
end
|
|
354
|
+
out << content[LEADING_WHITESPACE_PATTERN] << marker << stripped << marker << content[TRAILING_WHITESPACE_PATTERN]
|
|
355
|
+
end
|
|
356
|
+
end
|
|
357
|
+
|
|
358
|
+
# Append a markdown image for an img tag. Sources with a URI scheme are limited
|
|
359
|
+
# to the same protocols allowed for links.
|
|
360
|
+
def append_markdown_image(out, node)
|
|
361
|
+
src = node[SRC].to_s
|
|
362
|
+
return if src.empty? || LINE_BREAK_PATTERN.match?(src)
|
|
363
|
+
return if URI_SCHEME_PATTERN.match?(src) && !LINK_URL_PATTERN.match?(src)
|
|
364
|
+
alt = node[ALT].to_s.gsub(ALL_WHITESPACE_PATTERN, SPACE).strip
|
|
365
|
+
out << "![" << escape_unbalanced_brackets(alt) << "](" << src << ")"
|
|
366
|
+
end
|
|
367
|
+
|
|
368
|
+
# Escape square brackets that are not part of a balanced pair so that the text
|
|
369
|
+
# is safe to use inside a markdown link or image label. Balanced pairs are left
|
|
370
|
+
# alone since they are valid inside a label.
|
|
371
|
+
def escape_unbalanced_brackets(text)
|
|
372
|
+
return text unless BRACKET_PATTERN.match?(text)
|
|
373
|
+
unmatched = []
|
|
374
|
+
open_brackets = []
|
|
375
|
+
text.each_char.with_index do |char, i|
|
|
376
|
+
if char == "["
|
|
377
|
+
open_brackets << i
|
|
378
|
+
elsif char == "]"
|
|
379
|
+
open_brackets.empty? ? unmatched << i : open_brackets.pop
|
|
380
|
+
end
|
|
381
|
+
end
|
|
382
|
+
unmatched.concat(open_brackets)
|
|
383
|
+
return text if unmatched.empty?
|
|
384
|
+
escaped = text.dup
|
|
385
|
+
unmatched.sort.reverse_each { |i| escaped.insert(i, "\\") }
|
|
386
|
+
escaped
|
|
387
|
+
end
|
|
388
|
+
|
|
389
|
+
# Rewrite the output generated by a blockquote element with each line prefixed by "> ".
|
|
390
|
+
def format_markdown_blockquote(out, pos)
|
|
391
|
+
content = out.slice!(pos..).to_s
|
|
392
|
+
stripped = content.strip
|
|
393
|
+
if stripped.empty?
|
|
394
|
+
out << content
|
|
395
|
+
else
|
|
396
|
+
append_paragraph_breaks(out)
|
|
397
|
+
out << stripped.split(NEWLINE, -1).map { |line| line.empty? ? MARKDOWN_EMPTY_QUOTE : "#{MARKDOWN_QUOTE}#{line}" }.join(NEWLINE)
|
|
398
|
+
append_paragraph_breaks(out)
|
|
399
|
+
end
|
|
400
|
+
end
|
|
401
|
+
|
|
402
|
+
# Rewrite the output generated by a pre element as a fenced code block.
|
|
403
|
+
def format_markdown_code_block(out, pos)
|
|
404
|
+
content = out.slice!(pos..).to_s
|
|
405
|
+
stripped = content.sub(LEADING_LINE_BREAK_PATTERN, EMPTY).sub(TRAILING_WHITESPACE_PATTERN, EMPTY)
|
|
406
|
+
if stripped.empty?
|
|
407
|
+
out << content
|
|
408
|
+
else
|
|
409
|
+
# A hard line break immediately before a code block would leave a stray backslash.
|
|
410
|
+
out.slice!(-2, 1) if out.end_with?(MARKDOWN_BR)
|
|
411
|
+
append_block_breaks(out)
|
|
412
|
+
fence = markdown_code_fence(stripped)
|
|
413
|
+
out << fence << NEWLINE << stripped << NEWLINE << fence << NEWLINE
|
|
414
|
+
end
|
|
415
|
+
end
|
|
416
|
+
|
|
417
|
+
# Determine the fence needed to enclose code block content. The fence must be longer
|
|
418
|
+
# than any run of backticks in the content so the content cannot close the fence early.
|
|
419
|
+
def markdown_code_fence(content)
|
|
420
|
+
return MARKDOWN_FENCE unless content.include?(MARKDOWN_FENCE)
|
|
421
|
+
longest_run = content.scan(BACKTICK_RUN_PATTERN).max_by(&:length).length
|
|
422
|
+
BACKTICK * (longest_run + 1)
|
|
423
|
+
end
|
|
424
|
+
|
|
425
|
+
# Rewrite the output generated by a data table row in markdown so that it stays on a
|
|
426
|
+
# single line. Line breaks from br tags or block elements inside the cells would
|
|
427
|
+
# otherwise break the table syntax.
|
|
428
|
+
def flatten_markdown_table_row(out, pos)
|
|
429
|
+
row = out.slice!(pos..).to_s
|
|
430
|
+
row.gsub!(MARKDOWN_BR_MARKER, EMPTY)
|
|
431
|
+
row.gsub!(ALL_WHITESPACE_PATTERN, SPACE)
|
|
432
|
+
row.strip!
|
|
433
|
+
append_block_breaks(out)
|
|
434
|
+
out << row
|
|
435
|
+
end
|
|
436
|
+
|
|
437
|
+
# Append the markdown header separator row after the first row of a data table.
|
|
438
|
+
def append_markdown_table_separator(out, tr)
|
|
439
|
+
return unless first_table_row?(tr)
|
|
440
|
+
cells = tr.elements.count { |cell| cell.name == TD || cell.name == TH }
|
|
441
|
+
return if cells == 0
|
|
442
|
+
out << "|"
|
|
443
|
+
cells.times { out << MARKDOWN_TABLE_SEPARATOR_CELL }
|
|
444
|
+
out << NEWLINE
|
|
445
|
+
end
|
|
446
|
+
|
|
124
447
|
# Add double line breaks between paragraph elements. If line breaks already exist,
|
|
125
448
|
# new ones will only be added to get to two.
|
|
126
|
-
def append_paragraph_breaks(out)
|
|
127
|
-
|
|
449
|
+
def append_paragraph_breaks(out, trim: true)
|
|
450
|
+
trim_trailing_blanks!(out) if trim
|
|
128
451
|
if out.end_with?(NEWLINE)
|
|
129
452
|
out << NEWLINE unless out.end_with?("\n\n")
|
|
130
453
|
else
|
|
@@ -134,15 +457,32 @@ module HtmlToPlainText
|
|
|
134
457
|
|
|
135
458
|
# Add a single line break between block elements. If a line break already exists,
|
|
136
459
|
# none will be added.
|
|
137
|
-
def append_block_breaks(out)
|
|
138
|
-
|
|
460
|
+
def append_block_breaks(out, trim: true)
|
|
461
|
+
trim_trailing_blanks!(out) if trim
|
|
139
462
|
out << NEWLINE unless out.end_with?(NEWLINE)
|
|
140
463
|
end
|
|
141
464
|
|
|
465
|
+
# Remove spaces and tabs from the end of the output buffer.
|
|
466
|
+
def trim_trailing_blanks!(out)
|
|
467
|
+
out.slice!(-1) while out.end_with?(SPACE, TAB)
|
|
468
|
+
end
|
|
469
|
+
|
|
142
470
|
# Add an appropriate bullet or number to a list element.
|
|
143
471
|
def format_list_item(out, options)
|
|
144
|
-
if options[:
|
|
145
|
-
|
|
472
|
+
if options[:markdown]
|
|
473
|
+
indent = options[:indent].to_s
|
|
474
|
+
if options[:list] == :ul
|
|
475
|
+
out << indent << MARKDOWN_BULLET
|
|
476
|
+
options[:child_indent] = "#{indent} "
|
|
477
|
+
elsif options[:list] == :ol
|
|
478
|
+
number = options[:number]
|
|
479
|
+
options[:number] = number.next
|
|
480
|
+
marker = "#{number}. "
|
|
481
|
+
out << indent << marker
|
|
482
|
+
options[:child_indent] = indent + (SPACE * marker.length)
|
|
483
|
+
end
|
|
484
|
+
elsif options[:list] == :ul
|
|
485
|
+
out << "#{"*" * (options[:ul] + 1)} "
|
|
146
486
|
elsif options[:list] == :ol
|
|
147
487
|
number = options[:number]
|
|
148
488
|
options[:number] = number.next
|
|
@@ -150,8 +490,29 @@ module HtmlToPlainText
|
|
|
150
490
|
end
|
|
151
491
|
end
|
|
152
492
|
|
|
153
|
-
|
|
154
|
-
|
|
493
|
+
# Determine if a table is a data table rather than one used only for layout. A table
|
|
494
|
+
# is considered a data table if it has a non-zero border attribute or contains a
|
|
495
|
+
# thead or tbody element, or if the all_tables option is set.
|
|
496
|
+
def data_table?(tr, options)
|
|
497
|
+
return true if options[:all_tables]
|
|
498
|
+
table = containing_table(tr)
|
|
499
|
+
return false unless table
|
|
500
|
+
return true if table.attributes[BORDER].to_s.to_i > 0
|
|
501
|
+
table.elements.any? { |child| child.name == THEAD || child.name == TBODY }
|
|
502
|
+
end
|
|
503
|
+
|
|
504
|
+
# Determine if a tr is the first row in its table.
|
|
505
|
+
def first_table_row?(tr)
|
|
506
|
+
table = containing_table(tr)
|
|
507
|
+
return false unless table
|
|
508
|
+
table.at_xpath(FIRST_TR_XPATH) == tr
|
|
509
|
+
end
|
|
510
|
+
|
|
511
|
+
# Find the table element containing a table row.
|
|
512
|
+
def containing_table(tr)
|
|
513
|
+
table = tr.parent
|
|
514
|
+
table = table.parent while table && table.name != TABLE
|
|
515
|
+
table
|
|
155
516
|
end
|
|
156
517
|
end
|
|
157
518
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: html_to_plain_text
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.0
|
|
4
|
+
version: 1.1.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Brian Durand
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: bin
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: nokogiri
|
|
@@ -25,21 +24,7 @@ dependencies:
|
|
|
25
24
|
- !ruby/object:Gem::Version
|
|
26
25
|
version: 1.4.0
|
|
27
26
|
- !ruby/object:Gem::Dependency
|
|
28
|
-
name:
|
|
29
|
-
requirement: !ruby/object:Gem::Requirement
|
|
30
|
-
requirements:
|
|
31
|
-
- - ">"
|
|
32
|
-
- !ruby/object:Gem::Version
|
|
33
|
-
version: 2.6.0
|
|
34
|
-
type: :development
|
|
35
|
-
prerelease: false
|
|
36
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
37
|
-
requirements:
|
|
38
|
-
- - ">"
|
|
39
|
-
- !ruby/object:Gem::Version
|
|
40
|
-
version: 2.6.0
|
|
41
|
-
- !ruby/object:Gem::Dependency
|
|
42
|
-
name: rake
|
|
27
|
+
name: bundler
|
|
43
28
|
requirement: !ruby/object:Gem::Requirement
|
|
44
29
|
requirements:
|
|
45
30
|
- - ">="
|
|
@@ -52,57 +37,40 @@ dependencies:
|
|
|
52
37
|
- - ">="
|
|
53
38
|
- !ruby/object:Gem::Version
|
|
54
39
|
version: '0'
|
|
55
|
-
- !ruby/object:Gem::Dependency
|
|
56
|
-
name: bump
|
|
57
|
-
requirement: !ruby/object:Gem::Requirement
|
|
58
|
-
requirements:
|
|
59
|
-
- - ">="
|
|
60
|
-
- !ruby/object:Gem::Version
|
|
61
|
-
version: '0'
|
|
62
|
-
type: :development
|
|
63
|
-
prerelease: false
|
|
64
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
65
|
-
requirements:
|
|
66
|
-
- - ">="
|
|
67
|
-
- !ruby/object:Gem::Version
|
|
68
|
-
version: '0'
|
|
69
|
-
description: A simple library for converting HTML into an approximation in plain text.
|
|
70
40
|
email:
|
|
71
|
-
-
|
|
41
|
+
- bbdurand@gmail.com
|
|
72
42
|
executables: []
|
|
73
43
|
extensions: []
|
|
74
|
-
extra_rdoc_files:
|
|
75
|
-
- README.rdoc
|
|
44
|
+
extra_rdoc_files: []
|
|
76
45
|
files:
|
|
46
|
+
- CHANGELOG.md
|
|
77
47
|
- MIT_LICENSE
|
|
78
|
-
- README.
|
|
79
|
-
- Rakefile
|
|
48
|
+
- README.md
|
|
80
49
|
- VERSION
|
|
50
|
+
- html_to_plain_text.gemspec
|
|
81
51
|
- lib/html_to_plain_text.rb
|
|
82
52
|
homepage: https://github.com/bdurand/html_to_plain_text
|
|
83
|
-
licenses:
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
53
|
+
licenses:
|
|
54
|
+
- MIT
|
|
55
|
+
metadata:
|
|
56
|
+
homepage_uri: https://github.com/bdurand/html_to_plain_text
|
|
57
|
+
source_code_uri: https://github.com/bdurand/html_to_plain_text
|
|
58
|
+
changelog_uri: https://github.com/bdurand/html_to_plain_text/blob/master/CHANGELOG.md
|
|
59
|
+
rdoc_options: []
|
|
90
60
|
require_paths:
|
|
91
61
|
- lib
|
|
92
62
|
required_ruby_version: !ruby/object:Gem::Requirement
|
|
93
63
|
requirements:
|
|
94
64
|
- - ">="
|
|
95
65
|
- !ruby/object:Gem::Version
|
|
96
|
-
version: '
|
|
66
|
+
version: '2.7'
|
|
97
67
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
98
68
|
requirements:
|
|
99
69
|
- - ">="
|
|
100
70
|
- !ruby/object:Gem::Version
|
|
101
71
|
version: '0'
|
|
102
72
|
requirements: []
|
|
103
|
-
|
|
104
|
-
rubygems_version: 2.4.5
|
|
105
|
-
signing_key:
|
|
73
|
+
rubygems_version: 4.0.3
|
|
106
74
|
specification_version: 4
|
|
107
75
|
summary: A simple library for converting HTML into plain text.
|
|
108
76
|
test_files: []
|
data/README.rdoc
DELETED
|
@@ -1,24 +0,0 @@
|
|
|
1
|
-
= HTML To Plain Text
|
|
2
|
-
|
|
3
|
-
<code>gem install html_to_plain_text</code>
|
|
4
|
-
|
|
5
|
-
A simple gem that provide code to convert HTML into a plain text alternative. Line breaks from HTML block level elements will be maintained. Lists and tables will also maintain a little bit of formatting.
|
|
6
|
-
|
|
7
|
-
* Line breaks will be approximated using the generally established default margins for HTML tags (i.e. <p>
|
|
8
|
-
tag generates two line breaks, <div> generates one)
|
|
9
|
-
* Lists items will be numbered or bulleted with an asterisk
|
|
10
|
-
* <br> tags will add line breaks
|
|
11
|
-
* <hr> tags will add a string of hyphens to serve as a horizontal rule
|
|
12
|
-
* <table> elements will enclosed in "|" delimiters
|
|
13
|
-
* <a> tags will have the href URL appended to the text in parentheses
|
|
14
|
-
* Formatting tags like <strong> or <b> will be stripped
|
|
15
|
-
* Formatting inside <pre> or <plaintext> elements will be honored
|
|
16
|
-
* Code-like tags like <script> or <style> will be stripped
|
|
17
|
-
|
|
18
|
-
== Usage
|
|
19
|
-
|
|
20
|
-
require 'html_to_plain_text'
|
|
21
|
-
html = "<h1>Hello</h1><p>world!</p>"
|
|
22
|
-
HtmlToPlainText.plain_text(html)
|
|
23
|
-
=> "Hello\n\nworld!"
|
|
24
|
-
|