render-url 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (17) hide show
  1. render_url-1.0.0/PKG-INFO +189 -0
  2. render_url-1.0.0/README.md +180 -0
  3. render_url-1.0.0/parse_html.py +432 -0
  4. render_url-1.0.0/pyproject.toml +22 -0
  5. render_url-1.0.0/render_and_parse.py +156 -0
  6. render_url-1.0.0/render_url.egg-info/PKG-INFO +189 -0
  7. render_url-1.0.0/render_url.egg-info/SOURCES.txt +15 -0
  8. render_url-1.0.0/render_url.egg-info/dependency_links.txt +1 -0
  9. render_url-1.0.0/render_url.egg-info/entry_points.txt +4 -0
  10. render_url-1.0.0/render_url.egg-info/requires.txt +2 -0
  11. render_url-1.0.0/render_url.egg-info/top_level.txt +3 -0
  12. render_url-1.0.0/render_url.py +322 -0
  13. render_url-1.0.0/setup.cfg +4 -0
  14. render_url-1.0.0/tests/test_integration.py +113 -0
  15. render_url-1.0.0/tests/test_parser.py +231 -0
  16. render_url-1.0.0/tests/test_parser_cli.py +150 -0
  17. render_url-1.0.0/tests/test_renderer.py +116 -0
render_url-1.0.0/PKG-INFO ADDED
@@ -0,0 +1,189 @@
1
+ Metadata-Version: 2.4
2
+ Name: render-url
3
+ Version: 1.0.0
4
+ Summary: Render a single URL with Chromium (via Playwright) and emit the post-JS DOM as JSON.
5
+ Requires-Python: >=3.8
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: playwright>=1.40.0
8
+ Requires-Dist: beautifulsoup4>=4.12.0
9
+
10
+ # PlaywrightURLJsonExtractor
11
+
12
+ Two independent, deterministic CLI tools — no LLM/AI, no crawling:
13
+
14
+ - **`render-url`** — renders one URL in headless Chromium (via Playwright) and outputs the post-JS DOM as JSON.
15
+ - **`parse-html`** — parses the `html` field from `render-url`'s output into structured JSON (via BeautifulSoup4).
16
+
17
+ ```
18
+ URL -> render-url -> rendered JSON (html field) -> parse-html -> structured JSON
19
+ ```
20
+
21
+ ## Install
22
+
23
+ ```bash
24
+ pip install -e .
25
+ playwright install chromium
26
+ ```
27
+
28
+ This puts `render-url` and `parse-html` on your `PATH`.
29
+
30
+ ## Commands
31
+
32
+ ### `render-url` — render a URL to JSON
33
+
34
+ ```bash
35
+ render-url "https://example.com"
36
+ ```
37
+
38
+ Prints one JSON object to stdout and writes it to an auto-incremented file (`rendered_page_1.json`, `rendered_page_2.json`, ...). Never overwrites existing files.
39
+
40
+ | Parameter | Flag | Default | Description |
41
+ |---|---|---|---|
42
+ | URL | positional, or `url` in config | *(none)* | The single URL to render. Required (from CLI or config). |
43
+ | Config file | `--config <path>` | `config.json` | Path to the JSON config file. |
44
+ | Timeout | `--timeout <ms>` | `30000` | Overall navigation timeout in milliseconds. |
45
+ | Stabilization | `--stabilization <ms>` | `1000` | Fixed settle time (ms) after load, before capturing the DOM. |
46
+ | Wait condition | `--wait-until <state>` | `load` | `load`, `domcontentloaded`, or `networkidle`. |
47
+ | Headless | `--headless` / `--no-headless` | `--headless` | Run Chromium headless or with a visible window. |
48
+ | Output prefix | `--output-prefix <name>` | `rendered_page` | Base name for the output JSON file. |
49
+ | Output dir | `--output-dir <dir>` | `.` | Directory the output file is written into. |
50
+ | Verbose | `--verbose` / `-v` | off | Log progress (navigation, waits, extraction, browser lifecycle) to stderr. Stdout still carries only the final JSON. |
51
+ | Log JSON | `--log-json` | off | Additionally pretty-print the final result JSON to stderr. |
52
+
53
+ ### `parse-html` — extract structured data from rendered HTML
54
+
55
+ ```bash
56
+ parse-html rendered_page_1.json
57
+ ```
58
+
59
+ Reads a `render-url` JSON file (or stdin via `-`), extracts fields from its `html`, prints one JSON object to stdout, and writes it to an auto-incremented file (`parsed_page_1.json`, `parsed_page_2.json`, ...).
60
+
61
+ | Parameter | Flag | Default | Description |
62
+ |---|---|---|---|
63
+ | Input file | positional, or `--input <path>` | *(none)* | `render-url` JSON file to read. Use `-` for stdin. |
64
+ | Config file | `--config <path>` | `config.json` | Path to the JSON config file (reads its `"parser"` section). |
65
+ | CSS selector | `--selector <css>` | `null` | Scopes `headings`/`links`/`images`/`text` to the first matching element. No match -> those fields come back empty, not an error. |
66
+ | Whitespace | `--no-strip-whitespace` | strip on | Disable whitespace collapsing in extracted text. |
67
+ | Output prefix | `--output-prefix <name>` | `parsed_page` | Base name for the output JSON file. |
68
+ | Output dir | `--output-dir <dir>` | `.` | Directory the output file is written into. |
69
+ | Verbose | `--verbose` / `-v` | off | Log progress (input loading, extraction steps, field counts) to stderr. Stdout still carries only the final JSON. |
70
+ | Log JSON | `--log-json` | off | Additionally pretty-print the final result JSON to stderr. |
71
+
72
+ ### Extracted fields
73
+
74
+ | Field | What it is | Fallback |
75
+ |---|---|---|
76
+ | `title` | `<title>` text | `null` if absent/empty |
77
+ | `headings` | `<h1>`–`<h6>`, in order | `[]` if none |
78
+ | `links` | `<a href>` text + href, in order (duplicates kept) | anchors without `href` excluded |
79
+ | `images` | `<img src>` + `alt`, in order | `alt` defaults to `""`; no `src` excluded |
80
+ | `meta` | `<meta name/property>` -> `content` | later tag wins on duplicate keys |
81
+ | `text` | visible body text (scripts/styles excluded) | `""` if none |
82
+
83
+ ### `render-and-parse` — do both in one command
84
+
85
+ ```bash
86
+ render-and-parse "https://example.com"
87
+ ```
88
+
89
+ Runs `render-url` then `parse-html` in a single process and prints the final structured JSON. Writes both `rendered_page_N.json` and `parsed_page_N.json`.
90
+
91
+ | Parameter | Flag | Default | Description |
92
+ |---|---|---|---|
93
+ | URL | positional | *(none)* | The URL to render and parse. |
94
+ | Config file | `--config <path>` | `config.json` | Shared config file for both stages. |
95
+ | Timeout | `--timeout <ms>` | `30000` | Render-stage navigation timeout. |
96
+ | Stabilization | `--stabilization <ms>` | `1000` | Render-stage settle time. |
97
+ | Wait condition | `--wait-until <state>` | `load` | Render-stage load state. |
98
+ | Headless | `--headless` / `--no-headless` | `--headless` | Render-stage Chromium visibility. |
99
+ | CSS selector | `--selector <css>` | `null` | Parse-stage extraction scope. |
100
+ | Whitespace | `--no-strip-whitespace` | strip on | Parse-stage whitespace handling. |
101
+ | Output dir | `--output-dir <dir>` | `.` | Directory for both output files. |
102
+ | Verbose | `--verbose` / `-v` | off | Log progress from both stages to stderr. |
103
+ | Log JSON | `--log-json` | off | Additionally pretty-print the render stage and final result JSON to stderr. |
104
+
105
+ ### Full pipeline manually (two commands)
106
+
107
+ ```bash
108
+ render-url "https://example.com"
109
+ parse-html rendered_page_1.json
110
+
111
+ # or piped:
112
+ render-url "https://example.com" | parse-html -
113
+ ```
114
+
115
+ ### Tests
116
+
117
+ ```bash
118
+ python -m pytest tests/ -v
119
+ ```
120
+
121
+ ### Publishing to PyPI
122
+
123
+ ```bash
124
+ pip install build twine
125
+
126
+ # bump "version" in pyproject.toml first, then:
127
+ rm -rf dist build render_url.egg-info # PowerShell: Remove-Item -Recurse -Force dist, build, render_url.egg-info -ErrorAction SilentlyContinue
128
+ python -m build # builds dist/*.whl and dist/*.tar.gz
129
+ python -m twine check dist/* # validates metadata before upload
130
+ python -m twine upload dist/* # uploads to PyPI (prompts for credentials/token)
131
+ ```
132
+
133
+ Use `python -m twine upload --repository testpypi dist/*` to publish to [TestPyPI](https://test.pypi.org/) first if you want to verify the package before a real release.
134
+
135
+ ## Configuration
136
+
137
+ Precedence: **built-in defaults < `config.json` < CLI flags**. Both tools share one `config.json` (`parse-html` only reads its `"parser"` key). Copy [config.example.json](config.example.json) to get started — `config.json` is gitignored (local/per-environment). Unknown keys are rejected as typos.
138
+
139
+ ```json
140
+ {
141
+ "url": "https://example.com",
142
+ "timeout_ms": 30000,
143
+ "stabilization_ms": 1000,
144
+ "wait_until": "load",
145
+ "headless": true,
146
+ "output_prefix": "rendered_page",
147
+ "output_dir": ".",
148
+ "verbose": false,
149
+ "log_json": false,
150
+ "parser": {
151
+ "selector": null,
152
+ "strip_whitespace": true,
153
+ "output_prefix": "parsed_page",
154
+ "output_dir": ".",
155
+ "verbose": false,
156
+ "log_json": false
157
+ }
158
+ }
159
+ ```
160
+
161
+ ## Output schemas
162
+
163
+ **`render-url` success:**
164
+ ```json
165
+ {"ok": true, "url": "...", "final_url": "...", "status_code": 200, "title": "...", "html": "...", "error": null}
166
+ ```
167
+ Errors: `invalid_url`, `navigation_error`, `navigation_timeout`, `browser_error`, `html_extraction_error`, `unknown_error`.
168
+
169
+ **`parse-html` success:**
170
+ ```json
171
+ {
172
+ "ok": true,
173
+ "source_url": "...",
174
+ "title": "...",
175
+ "data": {"headings": [...], "links": [...], "images": [...], "meta": {...}, "text": "..."},
176
+ "selector_matched": null,
177
+ "error": null
178
+ }
179
+ ```
180
+ Errors: `invalid_input`, `missing_html` (Step 1 failed or had no `html`), `invalid_config`, `parse_error`, `unknown_error`.
181
+
182
+ Both tools always print exactly one JSON object to stdout (success or failure) — diagnostics go to stderr only.
183
+
184
+ ## What this project intentionally does NOT do
185
+
186
+ - No crawling, link-following, or multi-URL batching.
187
+ - No DOM mutation, clicking, form submission, or authentication.
188
+ - No LLM, AI, embeddings, or semantic interpretation — structural extraction only.
189
+ - `parse-html` makes no network calls and never launches a browser.
render_url-1.0.0/README.md ADDED
@@ -0,0 +1,180 @@
1
+ # PlaywrightURLJsonExtractor
2
+
3
+ Two independent, deterministic CLI tools — no LLM/AI, no crawling:
4
+
5
+ - **`render-url`** — renders one URL in headless Chromium (via Playwright) and outputs the post-JS DOM as JSON.
6
+ - **`parse-html`** — parses the `html` field from `render-url`'s output into structured JSON (via BeautifulSoup4).
7
+
8
+ ```
9
+ URL -> render-url -> rendered JSON (html field) -> parse-html -> structured JSON
10
+ ```
11
+
12
+ ## Install
13
+
14
+ ```bash
15
+ pip install -e .
16
+ playwright install chromium
17
+ ```
18
+
19
+ This puts `render-url` and `parse-html` on your `PATH`.
20
+
21
+ ## Commands
22
+
23
+ ### `render-url` — render a URL to JSON
24
+
25
+ ```bash
26
+ render-url "https://example.com"
27
+ ```
28
+
29
+ Prints one JSON object to stdout and writes it to an auto-incremented file (`rendered_page_1.json`, `rendered_page_2.json`, ...). Never overwrites existing files.
30
+
31
+ | Parameter | Flag | Default | Description |
32
+ |---|---|---|---|
33
+ | URL | positional, or `url` in config | *(none)* | The single URL to render. Required (from CLI or config). |
34
+ | Config file | `--config <path>` | `config.json` | Path to the JSON config file. |
35
+ | Timeout | `--timeout <ms>` | `30000` | Overall navigation timeout in milliseconds. |
36
+ | Stabilization | `--stabilization <ms>` | `1000` | Fixed settle time (ms) after load, before capturing the DOM. |
37
+ | Wait condition | `--wait-until <state>` | `load` | `load`, `domcontentloaded`, or `networkidle`. |
38
+ | Headless | `--headless` / `--no-headless` | `--headless` | Run Chromium headless or with a visible window. |
39
+ | Output prefix | `--output-prefix <name>` | `rendered_page` | Base name for the output JSON file. |
40
+ | Output dir | `--output-dir <dir>` | `.` | Directory the output file is written into. |
41
+ | Verbose | `--verbose` / `-v` | off | Log progress (navigation, waits, extraction, browser lifecycle) to stderr. Stdout still carries only the final JSON. |
42
+ | Log JSON | `--log-json` | off | Additionally pretty-print the final result JSON to stderr. |
43
+
44
+ ### `parse-html` — extract structured data from rendered HTML
45
+
46
+ ```bash
47
+ parse-html rendered_page_1.json
48
+ ```
49
+
50
+ Reads a `render-url` JSON file (or stdin via `-`), extracts fields from its `html`, prints one JSON object to stdout, and writes it to an auto-incremented file (`parsed_page_1.json`, `parsed_page_2.json`, ...).
51
+
52
+ | Parameter | Flag | Default | Description |
53
+ |---|---|---|---|
54
+ | Input file | positional, or `--input <path>` | *(none)* | `render-url` JSON file to read. Use `-` for stdin. |
55
+ | Config file | `--config <path>` | `config.json` | Path to the JSON config file (reads its `"parser"` section). |
56
+ | CSS selector | `--selector <css>` | `null` | Scopes `headings`/`links`/`images`/`text` to the first matching element. No match -> those fields come back empty, not an error. |
57
+ | Whitespace | `--no-strip-whitespace` | strip on | Disable whitespace collapsing in extracted text. |
58
+ | Output prefix | `--output-prefix <name>` | `parsed_page` | Base name for the output JSON file. |
59
+ | Output dir | `--output-dir <dir>` | `.` | Directory the output file is written into. |
60
+ | Verbose | `--verbose` / `-v` | off | Log progress (input loading, extraction steps, field counts) to stderr. Stdout still carries only the final JSON. |
61
+ | Log JSON | `--log-json` | off | Additionally pretty-print the final result JSON to stderr. |
62
+
63
+ ### Extracted fields
64
+
65
+ | Field | What it is | Fallback |
66
+ |---|---|---|
67
+ | `title` | `<title>` text | `null` if absent/empty |
68
+ | `headings` | `<h1>`–`<h6>`, in order | `[]` if none |
69
+ | `links` | `<a href>` text + href, in order (duplicates kept) | anchors without `href` excluded |
70
+ | `images` | `<img src>` + `alt`, in order | `alt` defaults to `""`; no `src` excluded |
71
+ | `meta` | `<meta name/property>` -> `content` | later tag wins on duplicate keys |
72
+ | `text` | visible body text (scripts/styles excluded) | `""` if none |
73
+
74
+ ### `render-and-parse` — do both in one command
75
+
76
+ ```bash
77
+ render-and-parse "https://example.com"
78
+ ```
79
+
80
+ Runs `render-url` then `parse-html` in a single process and prints the final structured JSON. Writes both `rendered_page_N.json` and `parsed_page_N.json`.
81
+
82
+ | Parameter | Flag | Default | Description |
83
+ |---|---|---|---|
84
+ | URL | positional | *(none)* | The URL to render and parse. |
85
+ | Config file | `--config <path>` | `config.json` | Shared config file for both stages. |
86
+ | Timeout | `--timeout <ms>` | `30000` | Render-stage navigation timeout. |
87
+ | Stabilization | `--stabilization <ms>` | `1000` | Render-stage settle time. |
88
+ | Wait condition | `--wait-until <state>` | `load` | Render-stage load state. |
89
+ | Headless | `--headless` / `--no-headless` | `--headless` | Render-stage Chromium visibility. |
90
+ | CSS selector | `--selector <css>` | `null` | Parse-stage extraction scope. |
91
+ | Whitespace | `--no-strip-whitespace` | strip on | Parse-stage whitespace handling. |
92
+ | Output dir | `--output-dir <dir>` | `.` | Directory for both output files. |
93
+ | Verbose | `--verbose` / `-v` | off | Log progress from both stages to stderr. |
94
+ | Log JSON | `--log-json` | off | Additionally pretty-print the render stage and final result JSON to stderr. |
95
+
96
+ ### Full pipeline manually (two commands)
97
+
98
+ ```bash
99
+ render-url "https://example.com"
100
+ parse-html rendered_page_1.json
101
+
102
+ # or piped:
103
+ render-url "https://example.com" | parse-html -
104
+ ```
105
+
106
+ ### Tests
107
+
108
+ ```bash
109
+ python -m pytest tests/ -v
110
+ ```
111
+
112
+ ### Publishing to PyPI
113
+
114
+ ```bash
115
+ pip install build twine
116
+
117
+ # bump "version" in pyproject.toml first, then:
118
+ rm -rf dist build render_url.egg-info # PowerShell: Remove-Item -Recurse -Force dist, build, render_url.egg-info -ErrorAction SilentlyContinue
119
+ python -m build # builds dist/*.whl and dist/*.tar.gz
120
+ python -m twine check dist/* # validates metadata before upload
121
+ python -m twine upload dist/* # uploads to PyPI (prompts for credentials/token)
122
+ ```
123
+
124
+ Use `python -m twine upload --repository testpypi dist/*` to publish to [TestPyPI](https://test.pypi.org/) first if you want to verify the package before a real release.
125
+
126
+ ## Configuration
127
+
128
+ Precedence: **built-in defaults < `config.json` < CLI flags**. Both tools share one `config.json` (`parse-html` only reads its `"parser"` key). Copy [config.example.json](config.example.json) to get started — `config.json` is gitignored (local/per-environment). Unknown keys are rejected as typos.
129
+
130
+ ```json
131
+ {
132
+ "url": "https://example.com",
133
+ "timeout_ms": 30000,
134
+ "stabilization_ms": 1000,
135
+ "wait_until": "load",
136
+ "headless": true,
137
+ "output_prefix": "rendered_page",
138
+ "output_dir": ".",
139
+ "verbose": false,
140
+ "log_json": false,
141
+ "parser": {
142
+ "selector": null,
143
+ "strip_whitespace": true,
144
+ "output_prefix": "parsed_page",
145
+ "output_dir": ".",
146
+ "verbose": false,
147
+ "log_json": false
148
+ }
149
+ }
150
+ ```
151
+
152
+ ## Output schemas
153
+
154
+ **`render-url` success:**
155
+ ```json
156
+ {"ok": true, "url": "...", "final_url": "...", "status_code": 200, "title": "...", "html": "...", "error": null}
157
+ ```
158
+ Errors: `invalid_url`, `navigation_error`, `navigation_timeout`, `browser_error`, `html_extraction_error`, `unknown_error`.
159
+
160
+ **`parse-html` success:**
161
+ ```json
162
+ {
163
+ "ok": true,
164
+ "source_url": "...",
165
+ "title": "...",
166
+ "data": {"headings": [...], "links": [...], "images": [...], "meta": {...}, "text": "..."},
167
+ "selector_matched": null,
168
+ "error": null
169
+ }
170
+ ```
171
+ Errors: `invalid_input`, `missing_html` (Step 1 failed or had no `html`), `invalid_config`, `parse_error`, `unknown_error`.
172
+
173
+ Both tools always print exactly one JSON object to stdout (success or failure) — diagnostics go to stderr only.
174
+
175
+ ## What this project intentionally does NOT do
176
+
177
+ - No crawling, link-following, or multi-URL batching.
178
+ - No DOM mutation, clicking, form submission, or authentication.
179
+ - No LLM, AI, embeddings, or semantic interpretation — structural extraction only.
180
+ - `parse-html` makes no network calls and never launches a browser.