dala 2.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dala-2.4.0/.gitignore +32 -0
- dala-2.4.0/CHANGELOG.md +68 -0
- dala-2.4.0/LICENSE +21 -0
- dala-2.4.0/PKG-INFO +746 -0
- dala-2.4.0/PRIVACY.md +35 -0
- dala-2.4.0/dala/__init__.py +0 -0
- dala-2.4.0/dala/cli.py +500 -0
- dala-2.4.0/dala/core/__init__.py +0 -0
- dala-2.4.0/dala/core/browser.py +356 -0
- dala-2.4.0/dala/core/discovery.py +583 -0
- dala-2.4.0/dala/core/dispatcher.py +55 -0
- dala-2.4.0/dala/core/extractor.py +678 -0
- dala-2.4.0/dala/core/forum_image_processor.py +611 -0
- dala-2.4.0/dala/core/image_budget.py +198 -0
- dala-2.4.0/dala/core/image_processor.py +1662 -0
- dala-2.4.0/dala/core/profiles.py +56 -0
- dala-2.4.0/dala/core/server_jobs.py +161 -0
- dala-2.4.0/dala/core/session.py +99 -0
- dala-2.4.0/dala/core/translation.py +817 -0
- dala-2.4.0/dala/core/writer.py +1052 -0
- dala-2.4.0/dala/data/__init__.py +1 -0
- dala-2.4.0/dala/data/sites.yaml +28 -0
- dala-2.4.0/dala/drivers/__init__.py +0 -0
- dala-2.4.0/dala/drivers/base.py +8 -0
- dala-2.4.0/dala/drivers/forum.py +461 -0
- dala-2.4.0/dala/drivers/generic.py +361 -0
- dala-2.4.0/dala/drivers/hn.py +170 -0
- dala-2.4.0/dala/drivers/reddit.py +199 -0
- dala-2.4.0/dala/drivers/substack.py +255 -0
- dala-2.4.0/dala/drivers/wordpress.py +172 -0
- dala-2.4.0/dala/drivers/youtube.py +427 -0
- dala-2.4.0/dala/models.py +211 -0
- dala-2.4.0/dala/server.py +1616 -0
- dala-2.4.0/dala/setup_browser.py +98 -0
- dala-2.4.0/dala/utils/__init__.py +0 -0
- dala-2.4.0/dala/utils/formatting.py +98 -0
- dala-2.4.0/dala/utils/llm.py +213 -0
- dala-2.4.0/launchers/Start Dala Server.bat +3 -0
- dala-2.4.0/launchers/Start Dala Server.command +2 -0
- dala-2.4.0/launchers/dala-server.desktop +7 -0
- dala-2.4.0/main.py +7 -0
- dala-2.4.0/pyproject.toml +116 -0
- dala-2.4.0/readme.md +700 -0
- dala-2.4.0/scripts/install-dala.ps1 +50 -0
- dala-2.4.0/scripts/install-dala.sh +52 -0
- dala-2.4.0/server.py +5 -0
- dala-2.4.0/sites.yaml +28 -0
- dala-2.4.0/tests/helpers.py +54 -0
- dala-2.4.0/tests/test_browser.py +233 -0
- dala-2.4.0/tests/test_chrome_extension.py +302 -0
- dala-2.4.0/tests/test_discovery.py +275 -0
- dala-2.4.0/tests/test_drivers.py +1443 -0
- dala-2.4.0/tests/test_hn_integration.py +110 -0
- dala-2.4.0/tests/test_image_budget.py +115 -0
- dala-2.4.0/tests/test_server.py +581 -0
- dala-2.4.0/tests/test_server_saving.py +71 -0
- dala-2.4.0/tests/test_setup_browser.py +70 -0
- dala-2.4.0/tests/test_translation.py +648 -0
- dala-2.4.0/tests/test_units.py +320 -0
- dala-2.4.0/tests/test_writer.py +1022 -0
- dala-2.4.0/web_to_epub.py +28 -0
dala-2.4.0/.gitignore
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Environments
|
|
2
|
+
.env
|
|
3
|
+
.env.local
|
|
4
|
+
.env.*.local
|
|
5
|
+
|
|
6
|
+
# Python
|
|
7
|
+
__pycache__/
|
|
8
|
+
*.py[cod]
|
|
9
|
+
*$py.class
|
|
10
|
+
.venv/
|
|
11
|
+
.pytest_cache/
|
|
12
|
+
.uvcache/
|
|
13
|
+
|
|
14
|
+
# Project specific
|
|
15
|
+
dist/
|
|
16
|
+
dala-chrome-*.zip
|
|
17
|
+
dala-chrome-v*.zip
|
|
18
|
+
dala-firefox-v*.xpi
|
|
19
|
+
extension_chrome/web-ext-artifacts/
|
|
20
|
+
firefox_extension/web-ext-artifacts/
|
|
21
|
+
firefox_extension/.amo-upload-uuid
|
|
22
|
+
exports/
|
|
23
|
+
server.log
|
|
24
|
+
Screenshot_*.png
|
|
25
|
+
*.epub
|
|
26
|
+
release_notes.md
|
|
27
|
+
|
|
28
|
+
# Local third-party browser helpers
|
|
29
|
+
config/bpc/
|
|
30
|
+
|
|
31
|
+
# Local service exports
|
|
32
|
+
config/systemd/
|
dala-2.4.0/CHANGELOG.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## v2.3.6
|
|
4
|
+
- **Package:** Include missing Firefox extension XPI (v2.3.5) in the release.
|
|
5
|
+
|
|
6
|
+
## v2.3.5
|
|
7
|
+
|
|
8
|
+
### Keyboard Shortcuts & Feedback
|
|
9
|
+
- Fixed "Queue shortcut failed" error toast in Firefox by properly handling message responses.
|
|
10
|
+
- Fixed missing toast notifications for keyboard shortcuts in Chrome by implementing robust response handling.
|
|
11
|
+
- Added immediate visual feedback ("Starting EPUB download...", "Added to EPUB queue") for native keyboard shortcuts (Ctrl+Shift+E / Ctrl+Shift+Q) in both browsers, ensuring feedback even if the browser consumes the key event.
|
|
12
|
+
|
|
13
|
+
### YouTube Improvements
|
|
14
|
+
- **Periodic Thumbnails:** Added an option to embed 3 periodic thumbnails (at ~25%, 50%, 75% timestamps) into YouTube transcripts for visual context. Toggleable via the extension popup ("Video Thumbnails").
|
|
15
|
+
- **Language & Caption Control:** New settings in the Extension Options page (and CLI flags `--yt-lang`, `--yt-auto`) allow you to specify preferred transcript languages (e.g., `en,es`) and whether to prioritize Manual or Auto-generated captions. The driver will now attempt to find the best match or auto-translate if necessary.
|
|
16
|
+
|
|
17
|
+
### Extension Reliability & Pipeline Alignment
|
|
18
|
+
- **Unified Data Context:** Refactored `background.js` to ensure keyboard shortcuts and context menu actions invoke the same data-gathering logic as the popup UI. Specifically, they now check the `include_cookies` option and fetch `browser.cookies.getAll()` for the target URL, transmitting the session state to the backend. This enables authenticated scraping (e.g., paywalls) via shortcuts which previously failed due to missing cookies.
|
|
19
|
+
- **Mitigating Background Throttling:** Bundle downloads previously hung on background tabs due to browser resource throttling of `executeScript` callbacks. Implemented a `Promise.race` wrapper around script injection with a 5-second timeout. If a background tab is unresponsive, the extension now fails soft, logging a timeout and proceeding to the next URL (falling back to server-side scraping) instead of stalling the entire queue.
|
|
20
|
+
- **Zombie/Discarded Tab Handling:** Added explicit checks for `tab.discarded` in `popup.js`. The extension now skips client-side DOM injection for suspended/unloaded tabs immediately, avoiding API errors and deadlocks, and relying on the backend to fetch the content freshly.
|
|
21
|
+
|
|
22
|
+
### Download Reliability
|
|
23
|
+
- **Browser Download Retry:** If the browser fails to save the file with the specific filename (e.g. due to invalid characters), the extension now automatically retries with a safe, generic filename (`web_to_epub_export.epub`).
|
|
24
|
+
- **Error Notifications:** If download fails completely, a desktop notification now shows the exact error message from the browser API.
|
|
25
|
+
- **Server-Side Backup:** The server now logs the full path of the generated EPUB in `/tmp` (e.g., `✅ Generated EPUB at: /tmp/tmpAbCdEf.epub`). If the browser download fails, the file can still be recovered from the server's temporary directory.
|
|
26
|
+
|
|
27
|
+
### Recursive Comment Fetching
|
|
28
|
+
- **HN + Source Comments:** When a Hacker News post links to a supported site (e.g. Substack), the downloader now fetches *both* the HN comments AND the original article's native comments.
|
|
29
|
+
- **Unified TOC:** The Table of Contents is structured with the Article at the top, and both comment threads (dynamically labeled, e.g., "Substack Comments" and "HN Comments") nested as children for easy navigation.
|
|
30
|
+
|
|
31
|
+
### Washington Post images
|
|
32
|
+
- Extract origin URLs from `imrs.php` proxies and try those first, so WaPo images download reliably.
|
|
33
|
+
- If no images survive extraction, parse `__NEXT_DATA__` and inject the listed images into the article body.
|
|
34
|
+
- Safer image wrapping tolerates detached tags; `LOGLEVEL=DEBUG` now shows per-image candidates/fetches when needed.
|
|
35
|
+
- Added missing `parse_srcset_with_width`, configurable logging, and origin/`__NEXT_DATA__` handling to unblock image downloads and document the WaPo improvements.
|
|
36
|
+
|
|
37
|
+
### UI & queue polish
|
|
38
|
+
- Popup queue is now an editable textarea: paste/edit URLs directly; changes auto-save and the queue persists across downloads.
|
|
39
|
+
- Added context-menu action “Download Page to EPUB” for immediate conversion of the current page/link; queue remains untouched.
|
|
40
|
+
|
|
41
|
+
### Metadata consistency
|
|
42
|
+
- Unified article meta block across drivers (Article Source/Author/Date/Site; archive notice where applicable). HN/Reddit external articles now show consistent metadata alongside thread context.
|
|
43
|
+
|
|
44
|
+
### Image cleanup
|
|
45
|
+
- Generic image processing strips placeholders (grey-placeholder), flattens wrappers, dedupes captions, and enforces `<div class="img-block"><img class="epub-image">` with a single caption when present.
|
|
46
|
+
- Figures/spans are unwrapped; duplicate captions removed.
|
|
47
|
+
|
|
48
|
+
### Reddit media fixes
|
|
49
|
+
- Image-link posts (`i.redd.it/...png`) render as the article with embedded images.
|
|
50
|
+
- Comment image links are inlined and fetched (skipping non-file wiki pages).
|
|
51
|
+
|
|
52
|
+
### Wikimedia fetch tuning
|
|
53
|
+
- Wikimedia images fetched with Commons referer + project UA; targeted fallbacks added and logging when blocked (ongoing).
|
|
54
|
+
|
|
55
|
+
### Forum image HTML simplification
|
|
56
|
+
- **Problem:** XenForo lightbox markup (`lazyloadPreSize`, `lbContainer*`, zoomer stubs, `data-lb-*`, `data-zoom-target`, empty `title`) leaked into the EPUB, leaving nested wrappers and non-reader-safe attributes.
|
|
57
|
+
- **Fix:** Added forum-specific cleanup in `ForumImageProcessor` to strip lightbox attrs, unwrap XenForo containers, and remove zoomer divs while still running the same asset mapping/dedup logic.
|
|
58
|
+
- **Result:** EPUB now emits minimal image HTML (`<div class="img-block"><img class="epub-image" src="..."></div>` plus caption when found) without breaking filename mapping or preload reuse.
|
|
59
|
+
|
|
60
|
+
### Forum attachment reliability
|
|
61
|
+
- **What was broken:** Page fragments (`#replies`) kept every fetch on page 1; popup closed before fetch finished; discovery crashed on lightbox nodes without `closest`; “enough assets” skipped re-fetch when only avatars/1x1s were present; 409/redirect fetches aborted; attachments on page 2/3 never entered the preload map, causing misses/dupes.
|
|
62
|
+
- **Fixes applied:** Strip fragments before building `page-N` URLs so pages 2/3 load; move all asset fetching to background after the popup closes; always re-fetch/merge assets and dedupe by URL instead of early-skipping; filter strictly to `/attachments/`, skip avatars/1x1/data GIFs; expand lightbox selectors (`[data-lb-*]`, `.bbImage`, `a.attachment`, `[data-attachment-id]`) with guards for missing `closest`; binary fetch uses `cache: reload` and retries query-stripped URLs on 409/opaque redirects. Result: all post attachments across pages are discovered and reused without duplication.
|
|
63
|
+
|
|
64
|
+
### Reddit Driver
|
|
65
|
+
- Reddit/old.reddit/redd.it links now fetch via the JSON API (`raw_json=1`), render self-posts or linked articles, and include threaded comments with navigation. Works in both CLI and the Firefox extension via the existing FastAPI backend.
|
|
66
|
+
|
|
67
|
+
### Forum Driver & Attachments
|
|
68
|
+
- Forum threads (e.g., XenForo) now support page ranges, asset preloading from the browser, and external images. The Firefox extension can fetch gated attachments with your session cookies and embed them into EPUBs.
|
dala-2.4.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2024 dala contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|