purehtml 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/README.md +31 -1
- data/lib/purehtml/backend.rb +119 -1
- data/lib/purehtml/version.rb +1 -1
- data/lib/purehtml.rb +31 -3
- data/purehtml.gemspec +5 -2
- metadata +25 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: '0874a4a517f3ccdc7513231cde4084e1ad7f81d8985ecdb44485d6b307ad81ca'
|
|
4
|
+
data.tar.gz: 13c64400abf0eb074dd09465d7b78288c678cccc3062be327653d1d57af7055f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fc7ed845c4c834faea208018a30234eeaa38525418652d39c3e3d4885b7d4f3ac69ec86421b09d8d2b379804ca91afe57a843f4860415bdf103d753712b1b20b
|
|
7
|
+
data.tar.gz: af30cd26d55ea3bd3b037b84c4f82cf23d20620397f9eb040745d9f739dbccb557a7f94e5646bbe7f6fd8c666aac235bbdb833aa53e20b614a991b9ba830b3e6
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to the PureHTML Ruby gem will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.2.0] - 2026-11-20
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- Support for `nokolexbor` backend (`PureHTML::NokolexborBackend`) for high-performance HTML5 parsing.
|
|
14
|
+
- Flexible backend configuration supporting symbols (`:nokogiri`, `:nokolexbor`), classes, or instances via `PureHTML.backend=` and `PureHTML.extract(..., backend: ...)`.
|
|
15
|
+
- Automatic detection of pre-parsed `Nokolexbor::Node` / `Nokolexbor::Document` and `Nokogiri::XML::Node` objects passed to `PureHTML.extract`.
|
|
16
|
+
- Dual-backend test suite running all PureHTML specification tests against both Nokogiri and Nokolexbor.
|
|
17
|
+
- Multi-backend benchmarking in `bench/bench.rb`.
|
|
18
|
+
|
|
19
|
+
### Changed
|
|
20
|
+
- Added `nokolexbor` development dependency and relaxed `rake` development dependency constraint to `>= 12.0`.
|
|
21
|
+
|
|
22
|
+
## [0.1.1] - 2026-11-19
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
- Constrained gem dependencies (`nokogiri`, `rake`, `minitest`) to silence open-ended dependency warnings.
|
|
26
|
+
- Cleaned up duplicate `documentation_uri` in gemspec metadata.
|
|
27
|
+
- Added repository metadata URIs (`changelog_uri`, `source_code_uri`, `bug_tracker_uri`).
|
|
28
|
+
|
|
29
|
+
## [0.1.0] - 2026-11-18
|
|
30
|
+
|
|
31
|
+
### Added
|
|
32
|
+
- Initial Ruby implementation of PureHTML specification.
|
|
33
|
+
- Built on top of Nokogiri for fast HTML/XML parsing.
|
|
34
|
+
- Support for declarative YAML configuration schemas (`PureHTML::ConfigFactory.from_yaml`).
|
|
35
|
+
- Support for primitives, nested objects, arrays, unions, constants, and default values.
|
|
36
|
+
- Built-in transformers (`trim`, `number`, `boolean`, `attr`, `html`, `json`, `jsonpath`, `split`, `join`, `replace`, `capitalize`, `upper`, `lower`, etc.).
|
|
37
|
+
- Comprehensive test suite matching the PureHTML multi-language test specifications.
|
|
38
|
+
|
|
39
|
+
[Unreleased]: https://github.com/purescraps/purehtml/compare/ruby-v0.2.0...HEAD
|
|
40
|
+
[0.2.0]: https://github.com/purescraps/purehtml/compare/ruby-v0.1.1...ruby-v0.2.0
|
|
41
|
+
[0.1.1]: https://github.com/purescraps/purehtml/compare/ruby-v0.1.0...ruby-v0.1.1
|
|
42
|
+
[0.1.0]: https://github.com/purescraps/purehtml/releases/tag/ruby-v0.1.0
|
data/README.md
CHANGED
|
@@ -7,7 +7,7 @@ PureHTML is a declarative HTML parsing and extraction library for Ruby. Extract
|
|
|
7
7
|
## Features
|
|
8
8
|
|
|
9
9
|
- **Declarative Schemas**: Simple YAML definitions for HTML data extraction.
|
|
10
|
-
- **Pluggable Backends**: Built on top of [Nokogiri](https://nokogiri.org).
|
|
10
|
+
- **Pluggable Backends**: Built on top of [Nokogiri](https://nokogiri.org) with optional high-performance [Nokolexbor](https://github.com/serpapi/nokolexbor) support.
|
|
11
11
|
- **Extensible Transformers**: Value transformation chain (e.g. `trim`, `number`, `attr`, `html`, `json`, `jsonpath`, `split`, `join`, `replace`, etc.).
|
|
12
12
|
- **Complex Structures**: Supports primitives, nested objects, arrays, unions, constants, and default values.
|
|
13
13
|
|
|
@@ -51,6 +51,36 @@ puts result.inspect
|
|
|
51
51
|
# => ["a", "b", "c"]
|
|
52
52
|
```
|
|
53
53
|
|
|
54
|
+
## Backends
|
|
55
|
+
|
|
56
|
+
PureHTML supports pluggable HTML parsing backends:
|
|
57
|
+
|
|
58
|
+
- **Nokogiri** (default): Robust and widely compatible HTML/XML parser.
|
|
59
|
+
- **Nokolexbor**: Ultra-fast HTML5 parser backed by Lexbor (significantly faster parsing and CSS selector execution).
|
|
60
|
+
|
|
61
|
+
### Using Nokolexbor
|
|
62
|
+
|
|
63
|
+
Add `nokolexbor` to your Gemfile:
|
|
64
|
+
|
|
65
|
+
```ruby
|
|
66
|
+
gem 'nokolexbor'
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
You can select the backend globally or pass it per extraction:
|
|
70
|
+
|
|
71
|
+
```ruby
|
|
72
|
+
# Set globally
|
|
73
|
+
PureHTML.backend = :nokolexbor
|
|
74
|
+
# or PureHTML.backend = PureHTML::NokolexborBackend.new
|
|
75
|
+
|
|
76
|
+
# Per extraction
|
|
77
|
+
PureHTML.extract(config, html_string, backend: :nokolexbor)
|
|
78
|
+
|
|
79
|
+
# Or pass a pre-parsed Nokolexbor document directly
|
|
80
|
+
doc = Nokolexbor::HTML(html_string)
|
|
81
|
+
PureHTML.extract(config, doc)
|
|
82
|
+
```
|
|
83
|
+
|
|
54
84
|
## Configuration Types
|
|
55
85
|
|
|
56
86
|
### Primitive Value
|
data/lib/purehtml/backend.rb
CHANGED
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
require "nokogiri"
|
|
4
4
|
|
|
5
|
+
begin
|
|
6
|
+
require "nokolexbor"
|
|
7
|
+
rescue LoadError
|
|
8
|
+
# nokolexbor is an optional backend
|
|
9
|
+
end
|
|
10
|
+
|
|
5
11
|
module PureHTML
|
|
6
12
|
class Backend
|
|
7
13
|
def load(html)
|
|
@@ -43,12 +49,19 @@ module PureHTML
|
|
|
43
49
|
|
|
44
50
|
class NokogiriBackend < Backend
|
|
45
51
|
def load(html)
|
|
46
|
-
|
|
52
|
+
if html.is_a?(NokogiriDocument)
|
|
53
|
+
html
|
|
54
|
+
elsif html.is_a?(::Nokogiri::XML::Node)
|
|
55
|
+
NokogiriDocument.new(html)
|
|
56
|
+
else
|
|
57
|
+
NokogiriDocument.new(Nokogiri::HTML(html))
|
|
58
|
+
end
|
|
47
59
|
end
|
|
48
60
|
end
|
|
49
61
|
|
|
50
62
|
class NokogiriDocument < Document
|
|
51
63
|
attr_reader :nokogiri_doc
|
|
64
|
+
alias_method :doc, :nokogiri_doc
|
|
52
65
|
|
|
53
66
|
def initialize(nokogiri_doc)
|
|
54
67
|
@nokogiri_doc = nokogiri_doc
|
|
@@ -117,4 +130,109 @@ module PureHTML
|
|
|
117
130
|
end
|
|
118
131
|
end
|
|
119
132
|
end
|
|
133
|
+
|
|
134
|
+
class NokolexborBackend < Backend
|
|
135
|
+
def initialize
|
|
136
|
+
require_nokolexbor!
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
def load(html)
|
|
140
|
+
require_nokolexbor!
|
|
141
|
+
if html.is_a?(NokolexborDocument)
|
|
142
|
+
html
|
|
143
|
+
elsif defined?(::Nokolexbor::Node) && html.is_a?(::Nokolexbor::Node)
|
|
144
|
+
NokolexborDocument.new(html)
|
|
145
|
+
else
|
|
146
|
+
NokolexborDocument.new(::Nokolexbor::HTML(html))
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
|
|
150
|
+
private
|
|
151
|
+
|
|
152
|
+
def require_nokolexbor!
|
|
153
|
+
return if defined?(::Nokolexbor)
|
|
154
|
+
|
|
155
|
+
begin
|
|
156
|
+
require "nokolexbor"
|
|
157
|
+
rescue LoadError => e
|
|
158
|
+
raise LoadError, "NokolexborBackend requires the 'nokolexbor' gem: #{e.message}"
|
|
159
|
+
end
|
|
160
|
+
end
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
class NokolexborDocument < Document
|
|
164
|
+
attr_reader :nokolexbor_doc
|
|
165
|
+
alias_method :doc, :nokolexbor_doc
|
|
166
|
+
|
|
167
|
+
def initialize(nokolexbor_doc)
|
|
168
|
+
@nokolexbor_doc = nokolexbor_doc
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def root
|
|
172
|
+
NokolexborNode.new(@nokolexbor_doc, @nokolexbor_doc)
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def find(selector)
|
|
176
|
+
@nokolexbor_doc.css(selector).map { |node| NokolexborNode.new(@nokolexbor_doc, node) }
|
|
177
|
+
end
|
|
178
|
+
end
|
|
179
|
+
|
|
180
|
+
class NokolexborNode < Node
|
|
181
|
+
attr_reader :doc, :node
|
|
182
|
+
|
|
183
|
+
def initialize(doc, node)
|
|
184
|
+
@doc = doc
|
|
185
|
+
@node = node
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def is_selector?(selector)
|
|
189
|
+
return false if @node.nil? || @node.document?
|
|
190
|
+
|
|
191
|
+
if @node.respond_to?(:matches?)
|
|
192
|
+
begin
|
|
193
|
+
return @node.matches?(selector)
|
|
194
|
+
rescue StandardError
|
|
195
|
+
# Fall through to doc.css check if matches? encounters an unsupported selector syntax
|
|
196
|
+
end
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
@doc.css(selector).include?(@node)
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def find(selector)
|
|
203
|
+
nodes = if @node.document?
|
|
204
|
+
@doc.css(selector)
|
|
205
|
+
else
|
|
206
|
+
@node.css(selector)
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
nodes.map { |n| NokolexborNode.new(@doc, n) }
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
def text
|
|
213
|
+
return "" if @node.nil?
|
|
214
|
+
|
|
215
|
+
if @node.document? && @node.respond_to?(:root) && @node.root
|
|
216
|
+
return @node.root.text
|
|
217
|
+
end
|
|
218
|
+
|
|
219
|
+
@node.text
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def html
|
|
223
|
+
return nil if @node.nil?
|
|
224
|
+
|
|
225
|
+
@node.inner_html
|
|
226
|
+
end
|
|
227
|
+
|
|
228
|
+
def attr(name = nil)
|
|
229
|
+
return nil if @node.nil? || @node.document?
|
|
230
|
+
|
|
231
|
+
if name
|
|
232
|
+
@node[name]
|
|
233
|
+
else
|
|
234
|
+
@node.respond_to?(:attributes) ? @node.attributes.transform_values(&:value) : {}
|
|
235
|
+
end
|
|
236
|
+
end
|
|
237
|
+
end
|
|
120
238
|
end
|
data/lib/purehtml/version.rb
CHANGED
data/lib/purehtml.rb
CHANGED
|
@@ -19,14 +19,25 @@ require_relative "purehtml/config_factory"
|
|
|
19
19
|
|
|
20
20
|
module PureHTML
|
|
21
21
|
class << self
|
|
22
|
-
attr_writer :backend
|
|
23
|
-
|
|
24
22
|
def backend
|
|
25
23
|
@backend ||= NokogiriBackend.new
|
|
26
24
|
end
|
|
27
25
|
|
|
26
|
+
def backend=(backend)
|
|
27
|
+
@backend = backend.nil? ? nil : resolve_backend(backend)
|
|
28
|
+
end
|
|
29
|
+
|
|
28
30
|
def extract(config, html, url = "", backend: nil)
|
|
29
|
-
active_backend =
|
|
31
|
+
active_backend = if backend
|
|
32
|
+
resolve_backend(backend)
|
|
33
|
+
elsif defined?(::Nokolexbor::Node) && html.is_a?(::Nokolexbor::Node)
|
|
34
|
+
NokolexborBackend.new
|
|
35
|
+
elsif defined?(::Nokogiri::XML::Node) && html.is_a?(::Nokogiri::XML::Node)
|
|
36
|
+
NokogiriBackend.new
|
|
37
|
+
else
|
|
38
|
+
self.backend
|
|
39
|
+
end
|
|
40
|
+
|
|
30
41
|
doc = html.is_a?(Document) ? html : active_backend.load(html)
|
|
31
42
|
params = ExtractParams.new(
|
|
32
43
|
doc: doc,
|
|
@@ -38,5 +49,22 @@ module PureHTML
|
|
|
38
49
|
)
|
|
39
50
|
config.extract(params)
|
|
40
51
|
end
|
|
52
|
+
|
|
53
|
+
def resolve_backend(backend)
|
|
54
|
+
case backend
|
|
55
|
+
when :nokogiri, "nokogiri"
|
|
56
|
+
NokogiriBackend.new
|
|
57
|
+
when :nokolexbor, "nokolexbor"
|
|
58
|
+
NokolexborBackend.new
|
|
59
|
+
when Class
|
|
60
|
+
backend.new
|
|
61
|
+
else
|
|
62
|
+
if backend.respond_to?(:load)
|
|
63
|
+
backend
|
|
64
|
+
else
|
|
65
|
+
raise ArgumentError, "Unsupported backend: #{backend.inspect}"
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
41
69
|
end
|
|
42
70
|
end
|
data/purehtml.gemspec
CHANGED
|
@@ -16,14 +16,17 @@ Gem::Specification.new do |spec|
|
|
|
16
16
|
|
|
17
17
|
spec.metadata["homepage_uri"] = spec.homepage
|
|
18
18
|
spec.metadata["source_code_uri"] = "https://github.com/purescraps/purehtml"
|
|
19
|
+
spec.metadata["changelog_uri"] = "https://github.com/purescraps/purehtml/blob/master/ruby/CHANGELOG.md"
|
|
20
|
+
spec.metadata["bug_tracker_uri"] = "https://github.com/purescraps/purehtml/issues"
|
|
19
21
|
|
|
20
22
|
spec.files = Dir.chdir(File.expand_path(__dir__)) do
|
|
21
|
-
Dir.glob("{lib}/**/*") + %w[README.md LICENSE purehtml.gemspec Rakefile]
|
|
23
|
+
Dir.glob("{lib}/**/*") + %w[README.md LICENSE CHANGELOG.md purehtml.gemspec Rakefile]
|
|
22
24
|
end
|
|
23
25
|
spec.require_paths = ["lib"]
|
|
24
26
|
|
|
25
|
-
spec.add_dependency "nokogiri", ">= 1.10.0"
|
|
27
|
+
spec.add_dependency "nokogiri", "~> 1.10", ">= 1.10.0"
|
|
26
28
|
|
|
27
29
|
spec.add_development_dependency "rake", ">= 12.0"
|
|
28
30
|
spec.add_development_dependency "minitest", ">= 5.0"
|
|
31
|
+
spec.add_development_dependency "nokolexbor", ">= 0.5.0"
|
|
29
32
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: purehtml
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Fatih Kaya
|
|
@@ -13,6 +13,9 @@ dependencies:
|
|
|
13
13
|
name: nokogiri
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
|
+
- - "~>"
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '1.10'
|
|
16
19
|
- - ">="
|
|
17
20
|
- !ruby/object:Gem::Version
|
|
18
21
|
version: 1.10.0
|
|
@@ -20,6 +23,9 @@ dependencies:
|
|
|
20
23
|
prerelease: false
|
|
21
24
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
25
|
requirements:
|
|
26
|
+
- - "~>"
|
|
27
|
+
- !ruby/object:Gem::Version
|
|
28
|
+
version: '1.10'
|
|
23
29
|
- - ">="
|
|
24
30
|
- !ruby/object:Gem::Version
|
|
25
31
|
version: 1.10.0
|
|
@@ -51,6 +57,20 @@ dependencies:
|
|
|
51
57
|
- - ">="
|
|
52
58
|
- !ruby/object:Gem::Version
|
|
53
59
|
version: '5.0'
|
|
60
|
+
- !ruby/object:Gem::Dependency
|
|
61
|
+
name: nokolexbor
|
|
62
|
+
requirement: !ruby/object:Gem::Requirement
|
|
63
|
+
requirements:
|
|
64
|
+
- - ">="
|
|
65
|
+
- !ruby/object:Gem::Version
|
|
66
|
+
version: 0.5.0
|
|
67
|
+
type: :development
|
|
68
|
+
prerelease: false
|
|
69
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
70
|
+
requirements:
|
|
71
|
+
- - ">="
|
|
72
|
+
- !ruby/object:Gem::Version
|
|
73
|
+
version: 0.5.0
|
|
54
74
|
description: PureHTML is a parsing specification and library for extracting JSON from
|
|
55
75
|
HTML documents using simple YAML schemas.
|
|
56
76
|
email:
|
|
@@ -59,6 +79,7 @@ executables: []
|
|
|
59
79
|
extensions: []
|
|
60
80
|
extra_rdoc_files: []
|
|
61
81
|
files:
|
|
82
|
+
- CHANGELOG.md
|
|
62
83
|
- LICENSE
|
|
63
84
|
- README.md
|
|
64
85
|
- Rakefile
|
|
@@ -106,6 +127,8 @@ licenses:
|
|
|
106
127
|
metadata:
|
|
107
128
|
homepage_uri: https://purescraps.github.io/purehtml/
|
|
108
129
|
source_code_uri: https://github.com/purescraps/purehtml
|
|
130
|
+
changelog_uri: https://github.com/purescraps/purehtml/blob/master/ruby/CHANGELOG.md
|
|
131
|
+
bug_tracker_uri: https://github.com/purescraps/purehtml/issues
|
|
109
132
|
rdoc_options: []
|
|
110
133
|
require_paths:
|
|
111
134
|
- lib
|
|
@@ -120,7 +143,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
120
143
|
- !ruby/object:Gem::Version
|
|
121
144
|
version: '0'
|
|
122
145
|
requirements: []
|
|
123
|
-
rubygems_version:
|
|
146
|
+
rubygems_version: 3.6.9
|
|
124
147
|
specification_version: 4
|
|
125
148
|
summary: Declarative HTML parsing and data extraction library for Ruby
|
|
126
149
|
test_files: []
|