web-capture-python-examples 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,24 @@
1
+ This is free and unencumbered software released into the public domain.
2
+
3
+ Anyone is free to copy, modify, publish, use, compile, sell, or
4
+ distribute this software, either in source code form or as a compiled
5
+ binary, for any purpose, commercial or non-commercial, and by any
6
+ means.
7
+
8
+ In jurisdictions that recognize copyright laws, the author or authors
9
+ of this software dedicate any and all copyright interest in the
10
+ software to the public domain. We make this dedication for the benefit
11
+ of the public at large and to the detriment of our heirs and
12
+ successors. We intend this dedication to be an overt act of
13
+ relinquishment in perpetuity of all present and future rights to this
14
+ software under copyright law.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
19
+ IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
20
+ OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
21
+ ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22
+ OTHER DEALINGS IN THE SOFTWARE.
23
+
24
+ For more information, please refer to <https://unlicense.org>
@@ -0,0 +1,2 @@
1
+ include docs/python-examples.md
2
+ prune tests
@@ -0,0 +1,52 @@
1
+ Metadata-Version: 2.4
2
+ Name: web-capture-python-examples
3
+ Version: 0.1.0
4
+ Summary: Python HTTP client examples for the web-capture service
5
+ License-Expression: Unlicense
6
+ Project-URL: Homepage, https://github.com/link-assistant/web-capture
7
+ Project-URL: Repository, https://github.com/link-assistant/web-capture
8
+ Project-URL: Issues, https://github.com/link-assistant/web-capture/issues
9
+ Project-URL: Documentation, https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md
10
+ Requires-Python: >=3.14
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE
13
+ Requires-Dist: requests>=2.34.2
14
+ Dynamic: license-file
15
+
16
+ # web-capture-python-examples
17
+
18
+ Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
19
+ Python 3.14 or later is required. The first public version is **0.1.0**;
20
+ `0.0.0` was a development placeholder.
21
+
22
+ The package connects to a running JavaScript or Rust web-capture HTTP service.
23
+ Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
24
+
25
+ After the first release is published:
26
+
27
+ ```bash
28
+ python -m pip install web-capture-python-examples
29
+ python -m web_capture_python_examples.html_download https://example.com
30
+ python -m web_capture_python_examples.markdown_download https://example.com
31
+ python -m web_capture_python_examples.image_download https://example.com
32
+ python -m web_capture_python_examples.playwright_example https://example.com
33
+ ```
34
+
35
+ Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
36
+ in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
37
+ or `output/` for the Playwright example), so the installed package need not be writable.
38
+ The Playwright example selects the server's browser engine; it needs no local browser.
39
+ The source scripts in `js/examples/python/` remain directly runnable.
40
+
41
+ The shared client is also importable:
42
+
43
+ ```python
44
+ from web_capture_python_examples import capture
45
+
46
+ response = capture('markdown', 'https://example.com')
47
+ print(response.text)
48
+ ```
49
+
50
+ Requests have a 30-second timeout and propagate HTTP and connection errors.
51
+ See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
52
+ for pending-publisher configuration and publication status.
@@ -0,0 +1,346 @@
1
+ # web-capture
2
+
3
+ **JavaScript:** [![npm version](https://img.shields.io/npm/v/@link-assistant/web-capture?label=npm&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture) [![npm downloads](https://img.shields.io/npm/dm/@link-assistant/web-capture?label=downloads&color=blue)](https://www.npmjs.com/package/@link-assistant/web-capture) [![CI - JavaScript](https://img.shields.io/github/actions/workflow/status/link-assistant/web-capture/js.yml?branch=main&label=JS%20CI)](https://github.com/link-assistant/web-capture/actions/workflows/js.yml)
4
+
5
+ **Rust:** [![crates.io version](https://img.shields.io/crates/v/web-capture?label=crates.io&color=orange)](https://crates.io/crates/web-capture) [![crates.io downloads](https://img.shields.io/crates/d/web-capture?label=downloads&color=orange)](https://crates.io/crates/web-capture) [![docs.rs](https://img.shields.io/docsrs/web-capture?label=docs.rs)](https://docs.rs/web-capture) [![CI - Rust](https://img.shields.io/github/actions/workflow/status/link-assistant/web-capture/rust.yml?branch=main&label=Rust%20CI)](https://github.com/link-assistant/web-capture/actions/workflows/rust.yml)
6
+
7
+ **Release:** [![GitHub Release](https://img.shields.io/github/v/release/link-assistant/web-capture?label=GitHub%20Release)](https://github.com/link-assistant/web-capture/releases) [![License: Unlicense](https://img.shields.io/badge/license-Unlicense-green)](https://unlicense.org)
8
+
9
+ A CLI and microservice to fetch URLs and render them as:
10
+
11
+ - **Markdown**: Clean HTML-to-Markdown conversion with image extraction
12
+ - **HTML**: Rendered page content
13
+ - **Plain text**: Raw text downloads for paste-like URLs such as xpaste.pro
14
+ - **PNG/JPEG screenshot**: Viewport or full-page capture
15
+ - **ZIP archive**: Markdown/HTML + locally downloaded images
16
+ - **PDF**: Print-quality document
17
+ - **DOCX**: Word document
18
+ - **Shared AI dialog transcript**: ChatGPT share captures with structured diagnostics for unsupported providers
19
+
20
+ ## Language Implementations
21
+
22
+ This repository contains two implementations with compatible APIs:
23
+
24
+ | Implementation | Directory | Package | Status |
25
+ | ---------------------- | ------------------ | ---------------------------------------------------------------------------------------- | ---------- |
26
+ | **JavaScript/Node.js** | [`./js`](./js) | [@link-assistant/web-capture](https://www.npmjs.com/package/@link-assistant/web-capture) | Production |
27
+ | **Rust** | [`./rust`](./rust) | [web-capture](https://crates.io/crates/web-capture) | Production |
28
+
29
+ Both implementations provide the same CLI interface and HTTP API endpoints, allowing you to choose based on your deployment preferences.
30
+
31
+ ## Quick Start
32
+
33
+ ### JavaScript
34
+
35
+ ```bash
36
+ cd js
37
+ npm install
38
+ npm run dev
39
+ ```
40
+
41
+ ### Rust
42
+
43
+ ```bash
44
+ cd rust
45
+ cargo run -- --serve
46
+ ```
47
+
48
+ ## API Endpoints
49
+
50
+ Both implementations expose the same API:
51
+
52
+ | Endpoint | Description |
53
+ | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
54
+ | `GET /html?url=<URL>` | Get rendered HTML content |
55
+ | `GET /txt?url=<URL>` | Get raw text content, with xpaste.pro paste URLs normalized to `/raw` |
56
+ | `GET /markdown?url=<URL>` | Get Markdown conversion; xpaste.pro pastes include raw text inline under 1500 lines or as a ZIP when larger |
57
+ | `GET /markdown?url=<URL>&converter=kreuzberg` | High-performance Markdown conversion via [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) |
58
+ | `GET /markdown?url=<URL>&converter=kreuzberg&format=json` | Structured result with metadata, tables, images, and warnings |
59
+ | `GET /image?url=<URL>` | Get PNG screenshot |
60
+ | `GET /archive?url=<URL>` | Get a ZIP archive with markdown/HTML and images |
61
+ | `GET /fetch?url=<URL>` | Proxy fetch content |
62
+ | `GET /stream?url=<URL>` | Stream content |
63
+ | `GET /search?q=<QUERY>` | Capture structured search-provider results |
64
+ | `GET /shared-dialog?url=<URL>` | Capture shared AI dialog transcripts as JSON, Markdown, text, or Links Notation |
65
+
66
+ FormalAI integration should use the stable HTTP/CLI contract documented in
67
+ [`docs/formalai-contract.md`](docs/formalai-contract.md).
68
+
69
+ ## CLI Usage
70
+
71
+ ```bash
72
+ # Capture a URL as Markdown (default format, writes to ./data/web-capture/<host>/<path>/)
73
+ web-capture https://example.com
74
+
75
+ # Capture as Markdown and save to specific file
76
+ web-capture https://example.com -o page.md
77
+
78
+ # Write to stdout explicitly
79
+ web-capture https://example.com -o -
80
+
81
+ # Capture as HTML
82
+ web-capture https://example.com --format html
83
+
84
+ # Capture raw paste text
85
+ web-capture https://xpaste.pro/p/t4q0Lsp0 --format txt -o paste.txt
86
+
87
+ # Take a screenshot
88
+ web-capture https://example.com --format png -o screenshot.png
89
+
90
+ # Create a ZIP archive
91
+ web-capture https://example.com --archive
92
+
93
+ # Capture a shared AI dialog transcript as structured JSON
94
+ web-capture shared-dialog https://chatgpt.com/share/SHARE_ID
95
+
96
+ # Start as API server
97
+ web-capture --serve
98
+
99
+ # Start server on custom port
100
+ web-capture --serve --port 8080
101
+ ```
102
+
103
+ ## CLI Options
104
+
105
+ | Option | Short | Description | Default |
106
+ | ------------------------ | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
107
+ | `--serve` | `-s` | Start as HTTP API server | - |
108
+ | `--port` | `-p` | Port to listen on | 3000 |
109
+ | `--format` | `-f` | Output format: `markdown`/`md`, `html`, `txt`/`text`, `image`/`png`, `jpeg`, `pdf`, `docx`, `archive`, `json`, `meta-language`, `demo-memory` | `markdown` |
110
+ | `--output` | `-o` | Output file path. Use `-o -` for stdout | auto-derived from URL |
111
+ | `--data-dir` | | Base directory for auto-derived output paths | `./data/web-capture` |
112
+ | `--engine` | `-e` | Browser engine (JS only): `puppeteer`, `playwright` | `puppeteer` |
113
+ | `--embed-images` | | Keep images inline as base64 data URIs (self-contained file) | `false` |
114
+ | `--no-extract-images` | | Alias for `--embed-images` | `false` |
115
+ | `--extract-images[=DIR]` | | Extract images to `DIR/images/` (or next to the output) and download remote images | - |
116
+ | `--keep-original-links` | | Keep remote image URLs as direct links (the default markdown behavior) | `false` |
117
+ | `--images-dir` | | Subdirectory name for extracted images | `images` |
118
+ | `--archive` | | Create archive: `zip` (default), `7z`, `tar.gz`, `tar` | - |
119
+ | `--extract-latex` | | Extract LaTeX formulas | `true` |
120
+ | `--extract-metadata` | | Extract article metadata | `true` |
121
+ | `--post-process` | | Apply post-processing | `true` |
122
+ | `--detect-code-language` | | Detect code block languages | `true` |
123
+
124
+ ## Image Handling
125
+
126
+ Markdown output supports three image modes, and every capture path (browser or
127
+ API, CLI or server) routes through the same chokepoint so a flag behaves
128
+ identically regardless of how the page was captured:
129
+
130
+ | Mode | Flag | Result |
131
+ | -------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
132
+ | **Direct links** (default) | _none_ / `--keep-original-links` | Remote images stay as direct `https://…` URLs. Inline base64 (which has no remote URL to restore) is stripped to a placeholder with a warning — never silently kept as a multi-megabyte blob. No `images/` folder. |
133
+ | **Embed** | `--embed-images` | Base64 images are kept inline, producing a single self-contained file. |
134
+ | **Extract** | `--extract-images[=DIR]` | Inline base64 _and_ remote images are written to `DIR/images/` (defaults to next to the output file) and the markdown is rewritten to reference the local files. |
135
+
136
+ The `--archive` formats always bundle images into the archive's `images/`
137
+ folder regardless of these flags.
138
+
139
+ ## Environment Variables
140
+
141
+ All flags can be controlled via environment variables:
142
+
143
+ | Variable | Description | Default |
144
+ | ---------------------------------- | ----------------------------------- | -------------------- |
145
+ | `WEB_CAPTURE_DATA_DIR` | Base directory for output | `./data/web-capture` |
146
+ | `WEB_CAPTURE_EMBED_IMAGES` | `0`/`1` — keep images inline | `0` |
147
+ | `WEB_CAPTURE_EXTRACT_IMAGES` | Directory to extract images into | - |
148
+ | `WEB_CAPTURE_KEEP_ORIGINAL_LINKS` | `0`/`1` — keep original remote URLs | `0` |
149
+ | `WEB_CAPTURE_IMAGES_DIR` | Subdirectory for extracted images | `images` |
150
+ | `WEB_CAPTURE_EXTRACT_LATEX` | `0`/`1` — extract LaTeX | `1` |
151
+ | `WEB_CAPTURE_EXTRACT_METADATA` | `0`/`1` — extract metadata | `1` |
152
+ | `WEB_CAPTURE_POST_PROCESS` | `0`/`1` — post-processing | `1` |
153
+ | `WEB_CAPTURE_DETECT_CODE_LANGUAGE` | `0`/`1` — detect code langs | `1` |
154
+
155
+ ## API Endpoints
156
+
157
+ Both implementations expose the same API:
158
+
159
+ | Endpoint | Description |
160
+ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
161
+ | `GET /html?url=<URL>` | Get rendered HTML content |
162
+ | `GET /txt?url=<URL>` | Get raw text content, including normalized xpaste.pro raw paste text |
163
+ | `GET /markdown?url=<URL>` | Get Markdown; xpaste.pro pastes include raw text inline under 1500 lines or as a ZIP when larger |
164
+ | `GET /markdown?url=<URL>&converter=kreuzberg` | Get Markdown with the high-performance converter |
165
+ | `GET /markdown?url=<URL>&converter=kreuzberg&format=json` | Get structured Markdown conversion data |
166
+ | `GET /markdown?url=<URL>&embedImages=true` | Get Markdown with base64 images inline |
167
+ | `GET /markdown?url=<URL>&keepOriginalLinks=false` | Get Markdown with all images stripped |
168
+ | `GET /image?url=<URL>` | Get PNG screenshot |
169
+ | `GET /archive?url=<URL>` | ZIP archive with markdown + images extracted to `images/` |
170
+ | `GET /archive?url=<URL>&keepOriginalLinks=true` | ZIP archive keeping original remote image URLs |
171
+ | `GET /archive?url=<URL>&embedImages=true` | ZIP archive with base64 images inline |
172
+ | `GET /pdf?url=<URL>` | PDF with embedded images |
173
+ | `GET /docx?url=<URL>` | DOCX with embedded images |
174
+ | `GET /fetch?url=<URL>` | Proxy fetch content |
175
+ | `GET /stream?url=<URL>` | Stream content |
176
+ | `GET /shared-dialog?url=<URL>` | Shared AI dialog transcript capture with structured diagnostics |
177
+
178
+ ## Docker
179
+
180
+ ### JavaScript
181
+
182
+ ```bash
183
+ cd js
184
+ docker build -t web-capture-js .
185
+ docker run -p 3000:3000 web-capture-js
186
+ ```
187
+
188
+ ### Rust
189
+
190
+ ```bash
191
+ cd rust
192
+ docker build -t web-capture-rust .
193
+ docker run -p 3000:3000 web-capture-rust
194
+ ```
195
+
196
+ ## Project Structure
197
+
198
+ ```
199
+ web-capture/
200
+ ├── js/ # JavaScript/Node.js implementation
201
+ │ ├── src/ # Source code
202
+ │ ├── bin/ # CLI entry point
203
+ │ ├── tests/ # Test files
204
+ │ ├── examples/ # Usage examples
205
+ │ ├── package.json # npm package manifest
206
+ │ ├── Dockerfile # Docker build file
207
+ │ └── README.md # JavaScript-specific docs
208
+ │
209
+ ├── rust/ # Rust implementation
210
+ │ ├── src/ # Source code
211
+ │ │ ├── lib.rs # Library exports
212
+ │ │ ├── main.rs # CLI/server entry point
213
+ │ │ ├── browser.rs # Browser automation
214
+ │ │ ├── html.rs # HTML processing
215
+ │ │ └── markdown.rs # Markdown conversion
216
+ │ ├── tests/ # Test files
217
+ │ ├── examples/ # Usage examples
218
+ │ ├── Cargo.toml # Cargo package manifest
219
+ │ ├── Dockerfile # Docker build file
220
+ │ └── README.md # Rust-specific docs
221
+ │
222
+ ├── scripts/ # Shared build/release scripts
223
+ │ ├── *.mjs # JavaScript-specific scripts
224
+ │ ├── xpaste/ # xpaste fixture capture/regeneration helpers
225
+ │ └── rust-*.mjs # Rust-specific scripts
226
+ ├── tests/xpaste/data/ # Shared xpaste HTML/text/markdown/screenshot fixtures
227
+ │
228
+ ├── .github/workflows/
229
+ │ ├── js.yml # JavaScript CI/CD
230
+ │ └── rust.yml # Rust CI/CD
231
+ │
232
+ └── README.md # This file
233
+ ```
234
+
235
+ ## Development
236
+
237
+ Use Node.js 26.11.1 (`.node-version`), npm 12.2.0, Rust 1.99.0
238
+ (`rust-toolchain.toml`), and Python 3.14.8 (`.python-version`). The Docker
239
+ service supports Node.js 26.10.0, the latest published official image.
240
+ Install the shared, locked release tools from the repository root before
241
+ running scripts in `scripts/`:
242
+
243
+ ```bash
244
+ node scripts/setup-npm.mjs
245
+ npm ci
246
+ npm test
247
+ ```
248
+
249
+ Release commands detect the published package in `js/`; the root npm package
250
+ contains private tooling. Weekly Dependabot updates cover both npm and Cargo
251
+ projects, Python, Docker, Compose, and GitHub Actions.
252
+
253
+ ### JavaScript
254
+
255
+ ```bash
256
+ cd js
257
+ npm ci
258
+ npm run dev # Start dev server
259
+ npm test # Run tests
260
+ npm run lint # Run linter
261
+ ```
262
+
263
+ ### Rust
264
+
265
+ ```bash
266
+ cd rust
267
+ cargo build # Build
268
+ cargo test # Run tests
269
+ cargo clippy # Run linter
270
+ cargo fmt # Format code
271
+ ```
272
+
273
+ ### Python examples
274
+
275
+ The Python client examples are packaged as `web-capture-python-examples`.
276
+ See [installation and usage](docs/python-examples.md) and
277
+ [trusted publishing and first-release status](docs/releases/python-trusted-publishing.md).
278
+
279
+ Install uv 0.12.23, then run these commands from the repository root:
280
+
281
+ ```bash
282
+ uv sync --locked
283
+ uv run --locked python -m unittest discover -s tests/python -v
284
+ uv export --locked --no-emit-project --format requirements-txt --output-file .venv/audit-requirements.txt
285
+ uv run --locked pip-audit --strict --require-hashes --disable-pip -r .venv/audit-requirements.txt
286
+ uv run --locked python js/examples/python/html_download.py https://example.com
287
+ ```
288
+
289
+ Set `WEB_CAPTURE_URL` to use a service other than `http://localhost:3000`.
290
+ `requirements.txt` exports the hashed runtime dependencies for pip consumers;
291
+ `uv.lock` also locks the example test and audit tools.
292
+
293
+ ## Features
294
+
295
+ - **Markdown Conversion**: Clean HTML-to-Markdown with LaTeX extraction, metadata, and code language detection
296
+ - **Plain Text Capture**: `/txt` endpoint and `--format txt` output for text resources and xpaste.pro raw paste URLs
297
+ - **Image Extraction**: Base64 data URI images extracted to files with content-hash filenames
298
+ - **HTML Rendering**: Fetch and render HTML with JavaScript support via headless browsers
299
+ - **High-Performance Conversion**: Optional [kreuzberg html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) backend with structured metadata, table, and image results
300
+ - **Screenshots**: Capture PNG/JPEG screenshots with theme and viewport control
301
+ - **Archives**: ZIP archives with markdown/HTML + locally downloaded images
302
+ - **Google Docs**: Public export, Google Docs REST API, and editor-model capture
303
+ - **URL Normalization**: Convert relative URLs to absolute
304
+ - **Encoding Detection**: Automatic charset detection and UTF-8 conversion
305
+
306
+ ## Browser Engines
307
+
308
+ ### JavaScript Version
309
+
310
+ - **Puppeteer** (default): Mature, well-tested Chrome automation
311
+ - **Playwright**: Cross-browser automation with similar capabilities
312
+
313
+ ### Rust Version
314
+
315
+ - **browser-commander**: A Rust crate for browser automation using chromiumoxide
316
+
317
+ ## License
318
+
319
+ [Unlicense](LICENSE) — This is free and unencumbered software released into the public domain. You are free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means. See [https://unlicense.org](https://unlicense.org) for details.
320
+
321
+ ## Markdown Converters
322
+
323
+ web-capture supports three HTML-to-Markdown converter backends across the JavaScript and Rust implementations:
324
+
325
+ | Converter | Selection | Throughput | Structured Results | Used In |
326
+ | ---------------------- | --------------------- | ------------ | ------------------------------ | ------------------- |
327
+ | **Turndown** (default) | `converter=turndown` | ~5-10 MB/s | No | JS implementation |
328
+ | **html2md** (default) | `converter=html2md` | ~20-40 MB/s | No | Rust implementation |
329
+ | **kreuzberg** | `converter=kreuzberg` | 150-280 MB/s | Yes (metadata, tables, images) | Both JS and Rust |
330
+
331
+ The kreuzberg converter is powered by [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) and uses the same Rust core across both implementations, ensuring consistent output. See [integration analysis](docs/html-to-markdown-integration.md) for details.
332
+
333
+ ## Related Projects
334
+
335
+ - [browser-commander](https://github.com/link-foundation/browser-commander) - Browser automation library used in Rust implementation
336
+ - [turndown](https://github.com/mixmark-io/turndown) - HTML to Markdown converter used in JS implementation
337
+ - [html2md](https://github.com/nickyc975/html2md-rs) - HTML to Markdown converter used in Rust implementation
338
+ - [html-to-markdown](https://github.com/kreuzberg-dev/html-to-markdown) - High-performance HTML to Markdown converter (kreuzberg), integrated as optional converter
339
+
340
+ ## Capture cache
341
+
342
+ Both packages provide `CaptureStore` and `CachedTransport` over caller-owned transports. Captures keep the exact body bytes, final URL, status, response headers and diagnostics. Request keys include the exact URL, method and every request header in sorted order. Identical bodies share a SHA-256 file. The default TTL is 60 days; expiry is measured from fetch completion, in Unix milliseconds.
343
+
344
+ Online mode reuses fresh captures and refetches expired entries. Offline mode never invokes the transport: it returns cached captures with `cached: true` and a `stale` flag, or an `offline_cache_miss` error. Use `capture` to read these flags; the transport adapter returns the original receipt unchanged. Transport errors are not stored. Corrupt or unsupported records produce cache errors instead of returning unverified bytes.
345
+
346
+ `list` returns request metadata; `prune(now, ttl)` removes expired request records and unreferenced body files. `export(request)` and `import(json)` exchange version 1 JSON replay fixtures with byte arrays, shared between Rust and JavaScript. See [the cache format and examples](docs/capture-cache.md).
@@ -0,0 +1,37 @@
1
+ # web-capture-python-examples
2
+
3
+ Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
4
+ Python 3.14 or later is required. The first public version is **0.1.0**;
5
+ `0.0.0` was a development placeholder.
6
+
7
+ The package connects to a running JavaScript or Rust web-capture HTTP service.
8
+ Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
9
+
10
+ After the first release is published:
11
+
12
+ ```bash
13
+ python -m pip install web-capture-python-examples
14
+ python -m web_capture_python_examples.html_download https://example.com
15
+ python -m web_capture_python_examples.markdown_download https://example.com
16
+ python -m web_capture_python_examples.image_download https://example.com
17
+ python -m web_capture_python_examples.playwright_example https://example.com
18
+ ```
19
+
20
+ Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
21
+ in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
22
+ or `output/` for the Playwright example), so the installed package need not be writable.
23
+ The Playwright example selects the server's browser engine; it needs no local browser.
24
+ The source scripts in `js/examples/python/` remain directly runnable.
25
+
26
+ The shared client is also importable:
27
+
28
+ ```python
29
+ from web_capture_python_examples import capture
30
+
31
+ response = capture('markdown', 'https://example.com')
32
+ print(response.text)
33
+ ```
34
+
35
+ Requests have a 30-second timeout and propagate HTTP and connection errors.
36
+ See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
37
+ for pending-publisher configuration and publication status.
@@ -0,0 +1,5 @@
1
+ """HTTP client examples for a running web-capture service."""
2
+
3
+ from .capture_client import capture, download_text
4
+
5
+ __all__ = ["capture", "download_text"]
@@ -0,0 +1,20 @@
1
+ """Shared request policy for the capture-service examples."""
2
+ import os
3
+ from pathlib import Path
4
+ import sys
5
+ import requests
6
+
7
+
8
+ def capture(kind, url=None, **options):
9
+ if url is None:
10
+ url = sys.argv[1] if len(sys.argv) > 1 else 'https://example.com'
11
+ endpoint = os.environ.get('WEB_CAPTURE_URL', 'http://localhost:3000').rstrip('/')
12
+ response = requests.get(f'{endpoint}/{kind}', params={'url': url, **options}, timeout=30)
13
+ response.raise_for_status()
14
+ return response
15
+
16
+
17
+ def download_text(kind, filename):
18
+ output = Path.cwd() / filename if __package__ else Path(__file__).with_name(filename)
19
+ output.write_text(capture(kind).text, encoding='utf-8')
20
+ print(f'{kind.title()} saved to {output}')
@@ -0,0 +1,6 @@
1
+ if __package__:
2
+ from .capture_client import download_text
3
+ else:
4
+ from capture_client import download_text
5
+
6
+ download_text('html', 'downloaded.html')
@@ -0,0 +1,19 @@
1
+ if __package__:
2
+ from .capture_client import capture
3
+ else:
4
+ from capture_client import capture
5
+ import os
6
+
7
+ response = capture('image')
8
+ print('Status:', response.status_code)
9
+ print('Content-Type:', response.headers.get('content-type'))
10
+
11
+ # PNG signature: first 8 bytes
12
+ png_signature = b'\x89PNG\r\n\x1a\n'
13
+ if response.content[:8] == png_signature:
14
+ output_path = 'downloaded.png' if __package__ else os.path.join(os.path.dirname(__file__), 'downloaded.png')
15
+ with open(output_path, 'wb') as f:
16
+ f.write(response.content)
17
+ print(f'Image saved to {output_path} (valid PNG)')
18
+ else:
19
+ print('Response is not a valid PNG! First bytes:', response.content[:16])
@@ -0,0 +1,6 @@
1
+ if __package__:
2
+ from .capture_client import download_text
3
+ else:
4
+ from capture_client import download_text
5
+
6
+ download_text('markdown', 'downloaded.md')
@@ -0,0 +1,25 @@
1
+ import sys
2
+ if __package__:
3
+ from .capture_client import capture
4
+ else:
5
+ from capture_client import capture
6
+ import os
7
+
8
+ target_url = sys.argv[1] if len(sys.argv) > 1 else 'https://example.com'
9
+
10
+ # Create output directory if it doesn't exist
11
+ os.makedirs('output', exist_ok=True)
12
+
13
+ print('Fetching HTML using Playwright engine...')
14
+ html_response = capture('html', target_url, engine='playwright')
15
+ with open('output/playwright_html.html', 'w', encoding='utf-8') as f:
16
+ f.write(html_response.text)
17
+ print('HTML saved to output/playwright_html.html')
18
+
19
+ print('Fetching screenshot using Playwright engine...')
20
+ image_response = capture('image', target_url, engine='playwright')
21
+ with open('output/playwright_screenshot.png', 'wb') as f:
22
+ f.write(image_response.content)
23
+ print('Screenshot saved to output/playwright_screenshot.png')
24
+
25
+ print('Done! Both requests used Playwright engine.')
@@ -0,0 +1,26 @@
1
+ [build-system]
2
+ requires = ["setuptools>=77,<83"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "web-capture-python-examples"
7
+ version = "0.1.0"
8
+ description = "Python HTTP client examples for the web-capture service"
9
+ readme = "docs/python-examples.md"
10
+ license = "Unlicense"
11
+ license-files = ["LICENSE"]
12
+ requires-python = ">=3.14"
13
+ dependencies = ["requests>=2.34.2"]
14
+
15
+ [project.urls]
16
+ Homepage = "https://github.com/link-assistant/web-capture"
17
+ Repository = "https://github.com/link-assistant/web-capture"
18
+ Issues = "https://github.com/link-assistant/web-capture/issues"
19
+ Documentation = "https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md"
20
+
21
+ [dependency-groups]
22
+ dev = ["pip-audit>=2.10.1"]
23
+
24
+ [tool.setuptools]
25
+ packages = ["web_capture_python_examples"]
26
+ package-dir = { web_capture_python_examples = "js/examples/python" }
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,52 @@
1
+ Metadata-Version: 2.4
2
+ Name: web-capture-python-examples
3
+ Version: 0.1.0
4
+ Summary: Python HTTP client examples for the web-capture service
5
+ License-Expression: Unlicense
6
+ Project-URL: Homepage, https://github.com/link-assistant/web-capture
7
+ Project-URL: Repository, https://github.com/link-assistant/web-capture
8
+ Project-URL: Issues, https://github.com/link-assistant/web-capture/issues
9
+ Project-URL: Documentation, https://github.com/link-assistant/web-capture/blob/main/docs/python-examples.md
10
+ Requires-Python: >=3.14
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE
13
+ Requires-Dist: requests>=2.34.2
14
+ Dynamic: license-file
15
+
16
+ # web-capture-python-examples
17
+
18
+ Python HTTP client examples for [web-capture](https://github.com/link-assistant/web-capture).
19
+ Python 3.14 or later is required. The first public version is **0.1.0**;
20
+ `0.0.0` was a development placeholder.
21
+
22
+ The package connects to a running JavaScript or Rust web-capture HTTP service.
23
+ Start that service separately using the [repository quick start](https://github.com/link-assistant/web-capture#quick-start).
24
+
25
+ After the first release is published:
26
+
27
+ ```bash
28
+ python -m pip install web-capture-python-examples
29
+ python -m web_capture_python_examples.html_download https://example.com
30
+ python -m web_capture_python_examples.markdown_download https://example.com
31
+ python -m web_capture_python_examples.image_download https://example.com
32
+ python -m web_capture_python_examples.playwright_example https://example.com
33
+ ```
34
+
35
+ Set `WEB_CAPTURE_URL` to override `http://localhost:3000`. Downloads are saved
36
+ in the current directory (`downloaded.html`, `downloaded.md`, `downloaded.png`,
37
+ or `output/` for the Playwright example), so the installed package need not be writable.
38
+ The Playwright example selects the server's browser engine; it needs no local browser.
39
+ The source scripts in `js/examples/python/` remain directly runnable.
40
+
41
+ The shared client is also importable:
42
+
43
+ ```python
44
+ from web_capture_python_examples import capture
45
+
46
+ response = capture('markdown', 'https://example.com')
47
+ print(response.text)
48
+ ```
49
+
50
+ Requests have a 30-second timeout and propagate HTTP and connection errors.
51
+ See [the release instructions](https://github.com/link-assistant/web-capture/blob/main/docs/releases/python-trusted-publishing.md)
52
+ for pending-publisher configuration and publication status.
@@ -0,0 +1,16 @@
1
+ LICENSE
2
+ MANIFEST.in
3
+ README.md
4
+ pyproject.toml
5
+ docs/python-examples.md
6
+ js/examples/python/__init__.py
7
+ js/examples/python/capture_client.py
8
+ js/examples/python/html_download.py
9
+ js/examples/python/image_download.py
10
+ js/examples/python/markdown_download.py
11
+ js/examples/python/playwright_example.py
12
+ web_capture_python_examples.egg-info/PKG-INFO
13
+ web_capture_python_examples.egg-info/SOURCES.txt
14
+ web_capture_python_examples.egg-info/dependency_links.txt
15
+ web_capture_python_examples.egg-info/requires.txt
16
+ web_capture_python_examples.egg-info/top_level.txt
@@ -0,0 +1 @@
1
+ web_capture_python_examples