web-capture-python-examples 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- web_capture_python_examples-0.1.0/LICENSE +24 -0
- web_capture_python_examples-0.1.0/MANIFEST.in +2 -0
- web_capture_python_examples-0.1.0/PKG-INFO +52 -0
- web_capture_python_examples-0.1.0/README.md +346 -0
- web_capture_python_examples-0.1.0/docs/python-examples.md +37 -0
- web_capture_python_examples-0.1.0/js/examples/python/__init__.py +5 -0
- web_capture_python_examples-0.1.0/js/examples/python/capture_client.py +20 -0
- web_capture_python_examples-0.1.0/js/examples/python/html_download.py +6 -0
- web_capture_python_examples-0.1.0/js/examples/python/image_download.py +19 -0
- web_capture_python_examples-0.1.0/js/examples/python/markdown_download.py +6 -0
- web_capture_python_examples-0.1.0/js/examples/python/playwright_example.py +25 -0
- web_capture_python_examples-0.1.0/pyproject.toml +26 -0
- web_capture_python_examples-0.1.0/setup.cfg +4 -0
- web_capture_python_examples-0.1.0/web_capture_python_examples.egg-info/PKG-INFO +52 -0
- web_capture_python_examples-0.1.0/web_capture_python_examples.egg-info/SOURCES.txt +16 -0
- web_capture_python_examples-0.1.0/web_capture_python_examples.egg-info/dependency_links.txt +1 -0
- web_capture_python_examples-0.1.0/web_capture_python_examples.egg-info/requires.txt +1 -0
- web_capture_python_examples-0.1.0/web_capture_python_examples.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
This is free and unencumbered software released into the public domain.
|
|
2
|
+
|
|
3
|
+
Anyone is free to copy, modify, publish, use, compile, sell, or
|
|
4
|
+
distribute this software, either in source code form or as a compiled
|
|
5
|
+
binary, for any purpose, commercial or non-commercial, and by any
|
|
6
|
+
means.
|
|
7
|
+
|
|
8
|
+
In jurisdictions that recognize copyright laws, the author or authors
|
|
9
|
+
of this software dedicate any and all copyright interest in the
|
|
10
|
+
software to the public domain. We make this dedication for the benefit
|
|
11
|
+
of the public at large and to the detriment of our heirs and
|
|
12
|
+
successors. We intend this dedication to be an overt act of
|
|
13
|
+
relinquishment in perpetuity of all present and future rights to this
|
|
14
|
+
software under copyright law.
|
|
15
|
+
|
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
|
|
19
|
+
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
|
|
20
|
+
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
|
|
21
|
+
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
|
22
|
+
OTHER DEALINGS IN THE SOFTWARE.
|
|
23
|
+
|
|
24
|
+
For more information, please refer to <https://unlicense.org>
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: web-capture-python-examples
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Python HTTP client examples for the web-capture service
|
|
5
|
+
License-Expression: Unlicense
|
|
6
|
+
Project-URL: Homepage, https://github.com/link-assistant/web-capture
|
|
7
|
+
Project-URL: Repository, https://github.com/link-assistant/web-capture
|
|
8
|
+
Project-URL: Issues, https://github.com/link-assistant/web-capture/issues
|
|
9
|
+
Project-URL: Documentation, https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md
|
|
10
|
+
Requires-Python: >=3.14
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Requires-Dist: requests>=2.34.2
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# web-capture-python-examples
|
|
17
|
+
|
|
18
|
+
Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
|
|
19
|
+
Python 3.14 or later is required. The first public version is **0.1.0**;
|
|
20
|
+
`0.0.0` was a development placeholder.
|
|
21
|
+
|
|
22
|
+
The package connects to a running JavaScript or Rust web-capture HTTP service.
|
|
23
|
+
Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
|
|
24
|
+
|
|
25
|
+
After the first release is published:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
python -m pip install web-capture-python-examples
|
|
29
|
+
python -m web_capture_python_examples.html_download https://example.com
|
|
30
|
+
python -m web_capture_python_examples.markdown_download https://example.com
|
|
31
|
+
python -m web_capture_python_examples.image_download https://example.com
|
|
32
|
+
python -m web_capture_python_examples.playwright_example https://example.com
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
|
|
36
|
+
in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
|
|
37
|
+
or `output/` for the Playwright example), so the installed package need not be writable.
|
|
38
|
+
The Playwright example selects the server's browser engine; it needs no local browser.
|
|
39
|
+
The source scripts in `js/examples/python/` remain directly runnable.
|
|
40
|
+
|
|
41
|
+
The shared client is also importable:
|
|
42
|
+
|
|
43
|
+
```python
|
|
44
|
+
from web_capture_python_examples import capture
|
|
45
|
+
|
|
46
|
+
response = capture('markdown', 'https://example.com')
|
|
47
|
+
print(response.text)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Requests have a 30-second timeout and propagate HTTP and connection errors.
|
|
51
|
+
See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
|
|
52
|
+
for pending-publisher configuration and publication status.
|
|
@@ -0,0 +1,346 @@
|
|
|
1
|
+
# web-capture
|
|
2
|
+
|
|
3
|
+
**JavaScript:** [](https://www.npmjs.com/package/@link-assistant/web-capture) [](https://www.npmjs.com/package/@link-assistant/web-capture) [](https://github.com/link-assistant/web-capture/actions/workflows/js.yml)
|
|
4
|
+
|
|
5
|
+
**Rust:** [](https://crates.io/crates/web-capture) [](https://crates.io/crates/web-capture) [](https://docs.rs/web-capture) [](https://github.com/link-assistant/web-capture/actions/workflows/rust.yml)
|
|
6
|
+
|
|
7
|
+
**Release:** [](https://github.com/link-assistant/web-capture/releases) [](https://unlicense.org)
|
|
8
|
+
|
|
9
|
+
A CLI and microservice to fetch URLs and render them as:
|
|
10
|
+
|
|
11
|
+
- **Markdown**: Clean HTML-to-Markdown conversion with image extraction
|
|
12
|
+
- **HTML**: Rendered page content
|
|
13
|
+
- **Plain text**: Raw text downloads for paste-like URLs such as xpaste.pro
|
|
14
|
+
- **PNG/JPEG screenshot**: Viewport or full-page capture
|
|
15
|
+
- **ZIP archive**: Markdown/HTML + locally downloaded images
|
|
16
|
+
- **PDF**: Print-quality document
|
|
17
|
+
- **DOCX**: Word document
|
|
18
|
+
- **Shared AI dialog transcript**: ChatGPT share captures with structured diagnostics for unsupported providers
|
|
19
|
+
|
|
20
|
+
## Language Implementations
|
|
21
|
+
|
|
22
|
+
This repository contains two implementations with compatible APIs:
|
|
23
|
+
|
|
24
|
+
| Implementation | Directory | Package | Status |
|
|
25
|
+
| ---------------------- | ------------------ | ---------------------------------------------------------------------------------------- | ---------- |
|
|
26
|
+
| **JavaScript/Node.js** | [`./js`](./js) | [@link-assistant/web-capture](https://www.npmjs.com/package/@link-assistant/web-capture) | Production |
|
|
27
|
+
| **Rust** | [`./rust`](./rust) | [web-capture](https://crates.io/crates/web-capture) | Production |
|
|
28
|
+
|
|
29
|
+
Both implementations provide the same CLI interface and HTTP API endpoints, allowing you to choose based on your deployment preferences.
|
|
30
|
+
|
|
31
|
+
## Quick Start
|
|
32
|
+
|
|
33
|
+
### JavaScript
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
cd js
|
|
37
|
+
npm install
|
|
38
|
+
npm run dev
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
### Rust
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
cd rust
|
|
45
|
+
cargo run -- --serve
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
## API Endpoints
|
|
49
|
+
|
|
50
|
+
Both implementations expose the same API:
|
|
51
|
+
|
|
52
|
+
| Endpoint | Description |
|
|
53
|
+
| --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
|
|
54
|
+
| `GET /html?url=<URL>` | Get rendered HTML content |
|
|
55
|
+
| `GET /txt?url=<URL>` | Get raw text content, with xpaste.pro paste URLs normalized to `/raw` |
|
|
56
|
+
| `GET /markdown?url=<URL>` | Get Markdown conversion; xpaste.pro pastes include raw text inline under 1500 lines or as a ZIP when larger |
|
|
57
|
+
| `GET /markdown?url=<URL>&converter=kreuzberg` | High-performance Markdown conversion via [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) |
|
|
58
|
+
| `GET /markdown?url=<URL>&converter=kreuzberg&format=json` | Structured result with metadata, tables, images, and warnings |
|
|
59
|
+
| `GET /image?url=<URL>` | Get PNG screenshot |
|
|
60
|
+
| `GET /archive?url=<URL>` | Get a ZIP archive with markdown/HTML and images |
|
|
61
|
+
| `GET /fetch?url=<URL>` | Proxy fetch content |
|
|
62
|
+
| `GET /stream?url=<URL>` | Stream content |
|
|
63
|
+
| `GET /search?q=<QUERY>` | Capture structured search-provider results |
|
|
64
|
+
| `GET /shared-dialog?url=<URL>` | Capture shared AI dialog transcripts as JSON, Markdown, text, or Links Notation |
|
|
65
|
+
|
|
66
|
+
FormalAI integration should use the stable HTTP/CLI contract documented in
|
|
67
|
+
[`docs/formalai-contract.md`](docs/formalai-contract.md).
|
|
68
|
+
|
|
69
|
+
## CLI Usage
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
# Capture a URL as Markdown (default format, writes to ./data/web-capture/<host>/<path>/)
|
|
73
|
+
web-capture https://example.com
|
|
74
|
+
|
|
75
|
+
# Capture as Markdown and save to specific file
|
|
76
|
+
web-capture https://example.com -o page.md
|
|
77
|
+
|
|
78
|
+
# Write to stdout explicitly
|
|
79
|
+
web-capture https://example.com -o -
|
|
80
|
+
|
|
81
|
+
# Capture as HTML
|
|
82
|
+
web-capture https://example.com --format html
|
|
83
|
+
|
|
84
|
+
# Capture raw paste text
|
|
85
|
+
web-capture https://xpaste.pro/p/t4q0Lsp0 --format txt -o paste.txt
|
|
86
|
+
|
|
87
|
+
# Take a screenshot
|
|
88
|
+
web-capture https://example.com --format png -o screenshot.png
|
|
89
|
+
|
|
90
|
+
# Create a ZIP archive
|
|
91
|
+
web-capture https://example.com --archive
|
|
92
|
+
|
|
93
|
+
# Capture a shared AI dialog transcript as structured JSON
|
|
94
|
+
web-capture shared-dialog https://chatgpt.com/share/SHARE_ID
|
|
95
|
+
|
|
96
|
+
# Start as API server
|
|
97
|
+
web-capture --serve
|
|
98
|
+
|
|
99
|
+
# Start server on custom port
|
|
100
|
+
web-capture --serve --port 8080
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## CLI Options
|
|
104
|
+
|
|
105
|
+
| Option | Short | Description | Default |
|
|
106
|
+
| ------------------------ | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
|
|
107
|
+
| `--serve` | `-s` | Start as HTTP API server | - |
|
|
108
|
+
| `--port` | `-p` | Port to listen on | 3000 |
|
|
109
|
+
| `--format` | `-f` | Output format: `markdown`/`md`, `html`, `txt`/`text`, `image`/`png`, `jpeg`, `pdf`, `docx`, `archive`, `json`, `meta-language`, `demo-memory` | `markdown` |
|
|
110
|
+
| `--output` | `-o` | Output file path. Use `-o -` for stdout | auto-derived from URL |
|
|
111
|
+
| `--data-dir` | | Base directory for auto-derived output paths | `./data/web-capture` |
|
|
112
|
+
| `--engine` | `-e` | Browser engine (JS only): `puppeteer`, `playwright` | `puppeteer` |
|
|
113
|
+
| `--embed-images` | | Keep images inline as base64 data URIs (self-contained file) | `false` |
|
|
114
|
+
| `--no-extract-images` | | Alias for `--embed-images` | `false` |
|
|
115
|
+
| `--extract-images[=DIR]` | | Extract images to `DIR/images/` (or next to the output) and download remote images | - |
|
|
116
|
+
| `--keep-original-links` | | Keep remote image URLs as direct links (the default markdown behavior) | `false` |
|
|
117
|
+
| `--images-dir` | | Subdirectory name for extracted images | `images` |
|
|
118
|
+
| `--archive` | | Create archive: `zip` (default), `7z`, `tar.gz`, `tar` | - |
|
|
119
|
+
| `--extract-latex` | | Extract LaTeX formulas | `true` |
|
|
120
|
+
| `--extract-metadata` | | Extract article metadata | `true` |
|
|
121
|
+
| `--post-process` | | Apply post-processing | `true` |
|
|
122
|
+
| `--detect-code-language` | | Detect code block languages | `true` |
|
|
123
|
+
|
|
124
|
+
## Image Handling
|
|
125
|
+
|
|
126
|
+
Markdown output supports three image modes, and every capture path (browser or
|
|
127
|
+
API, CLI or server) routes through the same chokepoint so a flag behaves
|
|
128
|
+
identically regardless of how the page was captured:
|
|
129
|
+
|
|
130
|
+
| Mode | Flag | Result |
|
|
131
|
+
| -------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
132
|
+
| **Direct links** (default) | _none_ / `--keep-original-links` | Remote images stay as direct `https://…` URLs. Inline base64 (which has no remote URL to restore) is stripped to a placeholder with a warning — never silently kept as a multi-megabyte blob. No `images/` folder. |
|
|
133
|
+
| **Embed** | `--embed-images` | Base64 images are kept inline, producing a single self-contained file. |
|
|
134
|
+
| **Extract** | `--extract-images[=DIR]` | Inline base64 _and_ remote images are written to `DIR/images/` (defaults to next to the output file) and the markdown is rewritten to reference the local files. |
|
|
135
|
+
|
|
136
|
+
The `--archive` formats always bundle images into the archive's `images/`
|
|
137
|
+
folder regardless of these flags.
|
|
138
|
+
|
|
139
|
+
## Environment Variables
|
|
140
|
+
|
|
141
|
+
All flags can be controlled via environment variables:
|
|
142
|
+
|
|
143
|
+
| Variable | Description | Default |
|
|
144
|
+
| ---------------------------------- | ----------------------------------- | -------------------- |
|
|
145
|
+
| `WEB_CAPTURE_DATA_DIR` | Base directory for output | `./data/web-capture` |
|
|
146
|
+
| `WEB_CAPTURE_EMBED_IMAGES` | `0`/`1` — keep images inline | `0` |
|
|
147
|
+
| `WEB_CAPTURE_EXTRACT_IMAGES` | Directory to extract images into | - |
|
|
148
|
+
| `WEB_CAPTURE_KEEP_ORIGINAL_LINKS` | `0`/`1` — keep original remote URLs | `0` |
|
|
149
|
+
| `WEB_CAPTURE_IMAGES_DIR` | Subdirectory for extracted images | `images` |
|
|
150
|
+
| `WEB_CAPTURE_EXTRACT_LATEX` | `0`/`1` — extract LaTeX | `1` |
|
|
151
|
+
| `WEB_CAPTURE_EXTRACT_METADATA` | `0`/`1` — extract metadata | `1` |
|
|
152
|
+
| `WEB_CAPTURE_POST_PROCESS` | `0`/`1` — post-processing | `1` |
|
|
153
|
+
| `WEB_CAPTURE_DETECT_CODE_LANGUAGE` | `0`/`1` — detect code langs | `1` |
|
|
154
|
+
|
|
155
|
+
## API Endpoints
|
|
156
|
+
|
|
157
|
+
Both implementations expose the same API:
|
|
158
|
+
|
|
159
|
+
| Endpoint | Description |
|
|
160
|
+
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
|
161
|
+
| `GET /html?url=<URL>` | Get rendered HTML content |
|
|
162
|
+
| `GET /txt?url=<URL>` | Get raw text content, including normalized xpaste.pro raw paste text |
|
|
163
|
+
| `GET /markdown?url=<URL>` | Get Markdown; xpaste.pro pastes include raw text inline under 1500 lines or as a ZIP when larger |
|
|
164
|
+
| `GET /markdown?url=<URL>&converter=kreuzberg` | Get Markdown with the high-performance converter |
|
|
165
|
+
| `GET /markdown?url=<URL>&converter=kreuzberg&format=json` | Get structured Markdown conversion data |
|
|
166
|
+
| `GET /markdown?url=<URL>&embedImages=true` | Get Markdown with base64 images inline |
|
|
167
|
+
| `GET /markdown?url=<URL>&keepOriginalLinks=false` | Get Markdown with all images stripped |
|
|
168
|
+
| `GET /image?url=<URL>` | Get PNG screenshot |
|
|
169
|
+
| `GET /archive?url=<URL>` | ZIP archive with markdown + images extracted to `images/` |
|
|
170
|
+
| `GET /archive?url=<URL>&keepOriginalLinks=true` | ZIP archive keeping original remote image URLs |
|
|
171
|
+
| `GET /archive?url=<URL>&embedImages=true` | ZIP archive with base64 images inline |
|
|
172
|
+
| `GET /pdf?url=<URL>` | PDF with embedded images |
|
|
173
|
+
| `GET /docx?url=<URL>` | DOCX with embedded images |
|
|
174
|
+
| `GET /fetch?url=<URL>` | Proxy fetch content |
|
|
175
|
+
| `GET /stream?url=<URL>` | Stream content |
|
|
176
|
+
| `GET /shared-dialog?url=<URL>` | Shared AI dialog transcript capture with structured diagnostics |
|
|
177
|
+
|
|
178
|
+
## Docker
|
|
179
|
+
|
|
180
|
+
### JavaScript
|
|
181
|
+
|
|
182
|
+
```bash
|
|
183
|
+
cd js
|
|
184
|
+
docker build -t web-capture-js .
|
|
185
|
+
docker run -p 3000:3000 web-capture-js
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
### Rust
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
cd rust
|
|
192
|
+
docker build -t web-capture-rust .
|
|
193
|
+
docker run -p 3000:3000 web-capture-rust
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
## Project Structure
|
|
197
|
+
|
|
198
|
+
```
|
|
199
|
+
web-capture/
|
|
200
|
+
├── js/ # JavaScript/Node.js implementation
|
|
201
|
+
│ ├── src/ # Source code
|
|
202
|
+
│ ├── bin/ # CLI entry point
|
|
203
|
+
│ ├── tests/ # Test files
|
|
204
|
+
│ ├── examples/ # Usage examples
|
|
205
|
+
│ ├── package.json # npm package manifest
|
|
206
|
+
│ ├── Dockerfile # Docker build file
|
|
207
|
+
│ └── README.md # JavaScript-specific docs
|
|
208
|
+
│
|
|
209
|
+
├── rust/ # Rust implementation
|
|
210
|
+
│ ├── src/ # Source code
|
|
211
|
+
│ │ ├── lib.rs # Library exports
|
|
212
|
+
│ │ ├── main.rs # CLI/server entry point
|
|
213
|
+
│ │ ├── browser.rs # Browser automation
|
|
214
|
+
│ │ ├── html.rs # HTML processing
|
|
215
|
+
│ │ └── markdown.rs # Markdown conversion
|
|
216
|
+
│ ├── tests/ # Test files
|
|
217
|
+
│ ├── examples/ # Usage examples
|
|
218
|
+
│ ├── Cargo.toml # Cargo package manifest
|
|
219
|
+
│ ├── Dockerfile # Docker build file
|
|
220
|
+
│ └── README.md # Rust-specific docs
|
|
221
|
+
│
|
|
222
|
+
├── scripts/ # Shared build/release scripts
|
|
223
|
+
│ ├── *.mjs # JavaScript-specific scripts
|
|
224
|
+
│ ├── xpaste/ # xpaste fixture capture/regeneration helpers
|
|
225
|
+
│ └── rust-*.mjs # Rust-specific scripts
|
|
226
|
+
├── tests/xpaste/data/ # Shared xpaste HTML/text/markdown/screenshot fixtures
|
|
227
|
+
│
|
|
228
|
+
├── .github/workflows/
|
|
229
|
+
│ ├── js.yml # JavaScript CI/CD
|
|
230
|
+
│ └── rust.yml # Rust CI/CD
|
|
231
|
+
│
|
|
232
|
+
└── README.md # This file
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
## Development
|
|
236
|
+
|
|
237
|
+
Use Node.js 26.11.1 (`.node-version`), npm 12.2.0, Rust 1.99.0
|
|
238
|
+
(`rust-toolchain.toml`), and Python 3.14.8 (`.python-version`). The Docker
|
|
239
|
+
service supports Node.js 26.10.0, the latest published official image.
|
|
240
|
+
Install the shared, locked release tools from the repository root before
|
|
241
|
+
running scripts in `scripts/`:
|
|
242
|
+
|
|
243
|
+
```bash
|
|
244
|
+
node scripts/setup-npm.mjs
|
|
245
|
+
npm ci
|
|
246
|
+
npm test
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Release commands detect the published package in `js/`; the root npm package
|
|
250
|
+
contains private tooling. Weekly Dependabot updates cover both npm and Cargo
|
|
251
|
+
projects, Python, Docker, Compose, and GitHub Actions.
|
|
252
|
+
|
|
253
|
+
### JavaScript
|
|
254
|
+
|
|
255
|
+
```bash
|
|
256
|
+
cd js
|
|
257
|
+
npm ci
|
|
258
|
+
npm run dev # Start dev server
|
|
259
|
+
npm test # Run tests
|
|
260
|
+
npm run lint # Run linter
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
### Rust
|
|
264
|
+
|
|
265
|
+
```bash
|
|
266
|
+
cd rust
|
|
267
|
+
cargo build # Build
|
|
268
|
+
cargo test # Run tests
|
|
269
|
+
cargo clippy # Run linter
|
|
270
|
+
cargo fmt # Format code
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
### Python examples
|
|
274
|
+
|
|
275
|
+
The Python client examples are packaged as `web-capture-python-examples`.
|
|
276
|
+
See [installation and usage](docs/python-examples.md) and
|
|
277
|
+
[trusted publishing and first-release status](docs/releases/python-trusted-publishing.md).
|
|
278
|
+
|
|
279
|
+
Install uv 0.12.23, then run these commands from the repository root:
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
uv sync --locked
|
|
283
|
+
uv run --locked python -m unittest discover -s tests/python -v
|
|
284
|
+
uv export --locked --no-emit-project --format requirements-txt --output-file .venv/audit-requirements.txt
|
|
285
|
+
uv run --locked pip-audit --strict --require-hashes --disable-pip -r .venv/audit-requirements.txt
|
|
286
|
+
uv run --locked python js/examples/python/html_download.py https://example.com
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
Set `WEB_CAPTURE_URL` to use a service other than `http://localhost:3000`.
|
|
290
|
+
`requirements.txt` exports the hashed runtime dependencies for pip consumers;
|
|
291
|
+
`uv.lock` also locks the example test and audit tools.
|
|
292
|
+
|
|
293
|
+
## Features
|
|
294
|
+
|
|
295
|
+
- **Markdown Conversion**: Clean HTML-to-Markdown with LaTeX extraction, metadata, and code language detection
|
|
296
|
+
- **Plain Text Capture**: `/txt` endpoint and `--format txt` output for text resources and xpaste.pro raw paste URLs
|
|
297
|
+
- **Image Extraction**: Base64 data URI images extracted to files with content-hash filenames
|
|
298
|
+
- **HTML Rendering**: Fetch and render HTML with JavaScript support via headless browsers
|
|
299
|
+
- **High-Performance Conversion**: Optional [kreuzberg html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) backend with structured metadata, table, and image results
|
|
300
|
+
- **Screenshots**: Capture PNG/JPEG screenshots with theme and viewport control
|
|
301
|
+
- **Archives**: ZIP archives with markdown/HTML + locally downloaded images
|
|
302
|
+
- **Google Docs**: Public export, Google Docs REST API, and editor-model capture
|
|
303
|
+
- **URL Normalization**: Convert relative URLs to absolute
|
|
304
|
+
- **Encoding Detection**: Automatic charset detection and UTF-8 conversion
|
|
305
|
+
|
|
306
|
+
## Browser Engines
|
|
307
|
+
|
|
308
|
+
### JavaScript Version
|
|
309
|
+
|
|
310
|
+
- **Puppeteer** (default): Mature, well-tested Chrome automation
|
|
311
|
+
- **Playwright**: Cross-browser automation with similar capabilities
|
|
312
|
+
|
|
313
|
+
### Rust Version
|
|
314
|
+
|
|
315
|
+
- **browser-commander**: A Rust crate for browser automation using chromiumoxide
|
|
316
|
+
|
|
317
|
+
## License
|
|
318
|
+
|
|
319
|
+
[Unlicense](LICENSE) — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See [https://unlicense.org](https://unlicense.org) for details.
|
|
320
|
+
|
|
321
|
+
## Markdown Converters
|
|
322
|
+
|
|
323
|
+
web-capture supports three HTML-to-Markdown converter backends across the JavaScript and Rust implementations:
|
|
324
|
+
|
|
325
|
+
| Converter | Selection | Throughput | Structured Results | Used In |
|
|
326
|
+
| ---------------------- | --------------------- | ------------ | ------------------------------ | ------------------- |
|
|
327
|
+
| **Turndown** (default) | `converter=turndown` | ~5-10 MB/s | No | JS implementation |
|
|
328
|
+
| **html2md** (default) | `converter=html2md` | ~20-40 MB/s | No | Rust implementation |
|
|
329
|
+
| **kreuzberg** | `converter=kreuzberg` | 150-280 MB/s | Yes (metadata, tables, images) | Both JS and Rust |
|
|
330
|
+
|
|
331
|
+
The kreuzberg converter is powered by [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) and uses the same Rust core across both implementations, ensuring consistent output. See [integration analysis](docs/html-to-markdown-integration.md) for details.
|
|
332
|
+
|
|
333
|
+
## Related Projects
|
|
334
|
+
|
|
335
|
+
- [browser-commander](https://github.com/link-foundation/browser-commander) - Browser automation library used in Rust implementation
|
|
336
|
+
- [turndown](https://github.com/mixmark-io/turndown) - HTML to Markdown converter used in JS implementation
|
|
337
|
+
- [html2md](https://github.com/nickyc975/html2md-rs) - HTML to Markdown converter used in Rust implementation
|
|
338
|
+
- [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) - High-performance HTML to Markdown converter (kreuzberg), integrated as optional converter
|
|
339
|
+
|
|
340
|
+
## Capture cache
|
|
341
|
+
|
|
342
|
+
Both packages provide `CaptureStore` and `CachedTransport` over caller-owned transports. Captures keep the exact body bytes, final URL, status, response headers and diagnostics. Request keys include the exact URL, method and every request header in sorted order. Identical bodies share a SHA-256 file. The default TTL is 60 days; expiry is measured from fetch completion, in Unix milliseconds.
|
|
343
|
+
|
|
344
|
+
Online mode reuses fresh captures and refetches expired entries. Offline mode never invokes the transport: it returns cached captures with `cached: true` and a `stale` flag, or an `offline_cache_miss` error. Use `capture` to read these flags; the transport adapter returns the original receipt unchanged. Transport errors are not stored. Corrupt or unsupported records produce cache errors instead of returning unverified bytes.
|
|
345
|
+
|
|
346
|
+
`list` returns request metadata; `prune(now, ttl)` removes expired request records and unreferenced body files. `export(request)` and `import(json)` exchange version 1 JSON replay fixtures with byte arrays, shared between Rust and JavaScript. See [the cache format and examples](docs/capture-cache.md).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# web-capture-python-examples
|
|
2
|
+
|
|
3
|
+
Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
|
|
4
|
+
Python 3.14 or later is required. The first public version is **0.1.0**;
|
|
5
|
+
`0.0.0` was a development placeholder.
|
|
6
|
+
|
|
7
|
+
The package connects to a running JavaScript or Rust web-capture HTTP service.
|
|
8
|
+
Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
|
|
9
|
+
|
|
10
|
+
After the first release is published:
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
python -m pip install web-capture-python-examples
|
|
14
|
+
python -m web_capture_python_examples.html_download https://example.com
|
|
15
|
+
python -m web_capture_python_examples.markdown_download https://example.com
|
|
16
|
+
python -m web_capture_python_examples.image_download https://example.com
|
|
17
|
+
python -m web_capture_python_examples.playwright_example https://example.com
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
|
|
21
|
+
in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
|
|
22
|
+
or `output/` for the Playwright example), so the installed package need not be writable.
|
|
23
|
+
The Playwright example selects the server's browser engine; it needs no local browser.
|
|
24
|
+
The source scripts in `js/examples/python/` remain directly runnable.
|
|
25
|
+
|
|
26
|
+
The shared client is also importable:
|
|
27
|
+
|
|
28
|
+
```python
|
|
29
|
+
from web_capture_python_examples import capture
|
|
30
|
+
|
|
31
|
+
response = capture('markdown', 'https://example.com')
|
|
32
|
+
print(response.text)
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Requests have a 30-second timeout and propagate HTTP and connection errors.
|
|
36
|
+
See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
|
|
37
|
+
for pending-publisher configuration and publication status.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
"""Shared request policy for the capture-service examples."""
|
|
2
|
+
import os
|
|
3
|
+
from pathlib import Path
|
|
4
|
+
import sys
|
|
5
|
+
import requests
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
def capture(kind, url=None, **options):
|
|
9
|
+
if url is None:
|
|
10
|
+
url = sys.argv[1] if len(sys.argv) > 1 else 'https://example.com'
|
|
11
|
+
endpoint = os.environ.get('WEB_CAPTURE_URL', 'http://localhost:3000').rstrip('/')
|
|
12
|
+
response = requests.get(f'{endpoint}/{kind}', params={'url': url, **options}, timeout=30)
|
|
13
|
+
response.raise_for_status()
|
|
14
|
+
return response
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
def download_text(kind, filename):
|
|
18
|
+
output = Path.cwd() / filename if __package__ else Path(__file__).with_name(filename)
|
|
19
|
+
output.write_text(capture(kind).text, encoding='utf-8')
|
|
20
|
+
print(f'{kind.title()} saved to {output}')
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
if __package__:
|
|
2
|
+
from .capture_client import capture
|
|
3
|
+
else:
|
|
4
|
+
from capture_client import capture
|
|
5
|
+
import os
|
|
6
|
+
|
|
7
|
+
response = capture('image')
|
|
8
|
+
print('Status:', response.status_code)
|
|
9
|
+
print('Content-Type:', response.headers.get('content-type'))
|
|
10
|
+
|
|
11
|
+
# PNG signature: first 8 bytes
|
|
12
|
+
png_signature = b'\x89PNG\r\n\x1a\n'
|
|
13
|
+
if response.content[:8] == png_signature:
|
|
14
|
+
output_path = 'downloaded.png' if __package__ else os.path.join(os.path.dirname(__file__), 'downloaded.png')
|
|
15
|
+
with open(output_path, 'wb') as f:
|
|
16
|
+
f.write(response.content)
|
|
17
|
+
print(f'Image saved to {output_path} (valid PNG)')
|
|
18
|
+
else:
|
|
19
|
+
print('Response is not a valid PNG! First bytes:', response.content[:16])
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
import sys
|
|
2
|
+
if __package__:
|
|
3
|
+
from .capture_client import capture
|
|
4
|
+
else:
|
|
5
|
+
from capture_client import capture
|
|
6
|
+
import os
|
|
7
|
+
|
|
8
|
+
target_url = sys.argv[1] if len(sys.argv) > 1 else 'https://example.com'
|
|
9
|
+
|
|
10
|
+
# Create output directory if it doesn't exist
|
|
11
|
+
os.makedirs('output', exist_ok=True)
|
|
12
|
+
|
|
13
|
+
print('Fetching HTML using Playwright engine...')
|
|
14
|
+
html_response = capture('html', target_url, engine='playwright')
|
|
15
|
+
with open('output/playwright_html.html', 'w', encoding='utf-8') as f:
|
|
16
|
+
f.write(html_response.text)
|
|
17
|
+
print('HTML saved to output/playwright_html.html')
|
|
18
|
+
|
|
19
|
+
print('Fetching screenshot using Playwright engine...')
|
|
20
|
+
image_response = capture('image', target_url, engine='playwright')
|
|
21
|
+
with open('output/playwright_screenshot.png', 'wb') as f:
|
|
22
|
+
f.write(image_response.content)
|
|
23
|
+
print('Screenshot saved to output/playwright_screenshot.png')
|
|
24
|
+
|
|
25
|
+
print('Done! Both requests used Playwright engine.')
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=77,<83"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "web-capture-python-examples"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Python HTTP client examples for the web-capture service"
|
|
9
|
+
readme = "docs/python-examples.md"
|
|
10
|
+
license = "Unlicense"
|
|
11
|
+
license-files = ["LICENSE"]
|
|
12
|
+
requires-python = ">=3.14"
|
|
13
|
+
dependencies = ["requests>=2.34.2"]
|
|
14
|
+
|
|
15
|
+
[project.urls]
|
|
16
|
+
Homepage = "https://github.com/link-assistant/web-capture"
|
|
17
|
+
Repository = "https://github.com/link-assistant/web-capture"
|
|
18
|
+
Issues = "https://github.com/link-assistant/web-capture/issues"
|
|
19
|
+
Documentation = "https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md"
|
|
20
|
+
|
|
21
|
+
[dependency-groups]
|
|
22
|
+
dev = ["pip-audit>=2.10.1"]
|
|
23
|
+
|
|
24
|
+
[tool.setuptools]
|
|
25
|
+
packages = ["web_capture_python_examples"]
|
|
26
|
+
package-dir = { web_capture_python_examples = "js/examples/python" }
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: web-capture-python-examples
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Python HTTP client examples for the web-capture service
|
|
5
|
+
License-Expression: Unlicense
|
|
6
|
+
Project-URL: Homepage, https://github.com/link-assistant/web-capture
|
|
7
|
+
Project-URL: Repository, https://github.com/link-assistant/web-capture
|
|
8
|
+
Project-URL: Issues, https://github.com/link-assistant/web-capture/issues
|
|
9
|
+
Project-URL: Documentation, https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md
|
|
10
|
+
Requires-Python: >=3.14
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Requires-Dist: requests>=2.34.2
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# web-capture-python-examples
|
|
17
|
+
|
|
18
|
+
Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
|
|
19
|
+
Python 3.14 or later is required. The first public version is **0.1.0**;
|
|
20
|
+
`0.0.0` was a development placeholder.
|
|
21
|
+
|
|
22
|
+
The package connects to a running JavaScript or Rust web-capture HTTP service.
|
|
23
|
+
Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
|
|
24
|
+
|
|
25
|
+
After the first release is published:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
python -m pip install web-capture-python-examples
|
|
29
|
+
python -m web_capture_python_examples.html_download https://example.com
|
|
30
|
+
python -m web_capture_python_examples.markdown_download https://example.com
|
|
31
|
+
python -m web_capture_python_examples.image_download https://example.com
|
|
32
|
+
python -m web_capture_python_examples.playwright_example https://example.com
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
|
|
36
|
+
in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
|
|
37
|
+
or `output/` for the Playwright example), so the installed package need not be writable.
|
|
38
|
+
The Playwright example selects the server's browser engine; it needs no local browser.
|
|
39
|
+
The source scripts in `js/examples/python/` remain directly runnable.
|
|
40
|
+
|
|
41
|
+
The shared client is also importable:
|
|
42
|
+
|
|
43
|
+
```python
|
|
44
|
+
from web_capture_python_examples import capture
|
|
45
|
+
|
|
46
|
+
response = capture('markdown', 'https://example.com')
|
|
47
|
+
print(response.text)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Requests have a 30-second timeout and propagate HTTP and connection errors.
|
|
51
|
+
See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
|
|
52
|
+
for pending-publisher configuration and publication status.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
MANIFEST.in
|
|
3
|
+
README.md
|
|
4
|
+
pyproject.toml
|
|
5
|
+
docs/python-examples.md
|
|
6
|
+
js/examples/python/__init__.py
|
|
7
|
+
js/examples/python/capture_client.py
|
|
8
|
+
js/examples/python/html_download.py
|
|
9
|
+
js/examples/python/image_download.py
|
|
10
|
+
js/examples/python/markdown_download.py
|
|
11
|
+
js/examples/python/playwright_example.py
|
|
12
|
+
web_capture_python_examples.egg-info/PKG-INFO
|
|
13
|
+
web_capture_python_examples.egg-info/SOURCES.txt
|
|
14
|
+
web_capture_python_examples.egg-info/dependency_links.txt
|
|
15
|
+
web_capture_python_examples.egg-info/requires.txt
|
|
16
|
+
web_capture_python_examples.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
requests>=2.34.2
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
web_capture_python_examples
|