abx-plugins 0.9.0__tar.gz

abx_plugins-0.9.0/.gitignore

@@ -0,0 +1,6 @@
+.DS_Store
+
+data/
+*.sqlite3*
+__pycache__/
+*.pyc

abx_plugins-0.9.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Nick Sweeting
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

abx_plugins-0.9.0/PKG-INFO

@@ -0,0 +1,118 @@
+Metadata-Version: 2.4
+Name: abx-plugins
+Version: 0.9.0
+Summary: ArchiveBox-compatible plugin suite (hooks, configs, binaries manifests)
+Project-URL: Homepage, https://github.com/ArchiveBox/abx-plugins
+Project-URL: Source, https://github.com/ArchiveBox/abx-plugins
+Project-URL: Documentation, https://github.com/ArchiveBox/ArchiveBox
+Project-URL: Bug Tracker, https://github.com/ArchiveBox/abx-plugins/issues
+Author-email: Nick Sweeting <pyproject.toml+abx-plugins@archivebox.io>
+License: MIT
+License-File: LICENSE
+Keywords: archivebox,hooks,plugins,scraping,web-archiving
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Requires-Dist: abx-pkg>=0.6.0
+Requires-Dist: rich-click>=1.9.7
+Provides-Extra: dev
+Requires-Dist: abx-pkg>=0.6.0; extra == 'dev'
+Requires-Dist: feedparser>=6.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: requests>=2.28.0; extra == 'dev'
+Requires-Dist: rich-click>=1.8.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# abx-plugins
+
+ArchiveBox-compatible plugin suite (hooks, config schemas, binaries manifests).
+
+This package contains only plugin assets and a tiny helper to locate them.
+It does **not** depend on Django or ArchiveBox.
+
+## Usage
+
+```python
+from abx_plugins import get_plugins_dir
+
+plugins_dir = get_plugins_dir()
+# scan plugins_dir for plugins/*/config.json, binaries.jsonl, on_* hooks
+```
+
+Tools like `abx-dl` and ArchiveBox can discover plugins from this package
+without symlinks or environment-variable tricks.
+
+## Plugin Contract
+
+### Directory layout
+
+Each plugin lives under `plugins/<name>/` and may include:
+
+- `config.json` (optional) - config schema
+- `binaries.jsonl` (optional) - binary manifests
+- `on_*` hook scripts (required to do work)
+
+Hooks run with:
+
+- **SNAP_DIR** = base snapshot directory (default: `.`)
+- **CRAWL_DIR** = base crawl directory (default: `.`)
+- **Snapshot hook output** = `SNAP_DIR/<plugin>/...`
+- **Crawl hook output** = `CRAWL_DIR/<plugin>/...`
+- **Other plugin outputs** can be read via `../<other-plugin>/...` from your own output dir
+
+### Key environment variables
+
+- `SNAP_DIR` - base snapshot directory (default: `.`)
+- `CRAWL_DIR` - base crawl directory (default: `.`)
+- `LIB_DIR` - binaries/tools root (default: `~/.config/abx/lib`)
+- `PERSONAS_DIR` - persona profiles root (default: `~/.config/abx/personas`)
+- `ACTIVE_PERSONA` - persona name (default: `Default`)
+
+### Event JSONL interface (bbus-style, no dependency)
+
+Hooks emit JSONL events to stdout. They do **not** need to import `bbus`.
+The event envelope matches the bbus style so higher layers can stream/replay.
+
+Minimal envelope:
+
+```json
+{
+  "event_id": "uuidv7",
+  "event_type": "SnapshotCreated",
+  "event_created_at": "2026-02-01T20:10:22Z",
+  "event_parent_id": "uuidv7-or-null",
+  "event_schema": "abx.events.v1",
+  "event_path": "abx-plugins",
+  "data": { "...": "event-specific fields" }
+}
+```
+
+Conventions:
+
+- Active verb names are **requests** (e.g. `BinaryInstall`, `ProcessLaunch`).
+- Past tense names are **facts** (e.g. `BinaryInstalled`, `ProcessExited`).
+- Plugins can emit additional fields inside `data` without coordination.
+
+Common event types emitted by hooks:
+
+- `ArchiveResultCreated` (status + output files)
+- `Binary` records (dependency detection/install)
+- `ProcessStarted` / `ProcessExited`
+
+Higher-level tools (abx-dl / ArchiveBox) can:
+
+- Parse these events from stdout
+- Persist or project them (SQLite/JSONL/Django) without plugins knowing
+
+Legacy note:
+
+Some hooks still emit a lightweight JSONL record with a top-level `type` field
+(e.g., `{"type": "ArchiveResult", ...}`). Runtimes should accept those and
+optionally translate them into the event envelope above.

abx_plugins-0.9.0/README.md

@@ -0,0 +1,86 @@
+# abx-plugins
+
+ArchiveBox-compatible plugin suite (hooks, config schemas, binaries manifests).
+
+This package contains only plugin assets and a tiny helper to locate them.
+It does **not** depend on Django or ArchiveBox.
+
+## Usage
+
+```python
+from abx_plugins import get_plugins_dir
+
+plugins_dir = get_plugins_dir()
+# scan plugins_dir for plugins/*/config.json, binaries.jsonl, on_* hooks
+```
+
+Tools like `abx-dl` and ArchiveBox can discover plugins from this package
+without symlinks or environment-variable tricks.
+
+## Plugin Contract
+
+### Directory layout
+
+Each plugin lives under `plugins/<name>/` and may include:
+
+- `config.json` (optional) - config schema
+- `binaries.jsonl` (optional) - binary manifests
+- `on_*` hook scripts (required to do work)
+
+Hooks run with:
+
+- **SNAP_DIR** = base snapshot directory (default: `.`)
+- **CRAWL_DIR** = base crawl directory (default: `.`)
+- **Snapshot hook output** = `SNAP_DIR/<plugin>/...`
+- **Crawl hook output** = `CRAWL_DIR/<plugin>/...`
+- **Other plugin outputs** can be read via `../<other-plugin>/...` from your own output dir
+
+### Key environment variables
+
+- `SNAP_DIR` - base snapshot directory (default: `.`)
+- `CRAWL_DIR` - base crawl directory (default: `.`)
+- `LIB_DIR` - binaries/tools root (default: `~/.config/abx/lib`)
+- `PERSONAS_DIR` - persona profiles root (default: `~/.config/abx/personas`)
+- `ACTIVE_PERSONA` - persona name (default: `Default`)
+
+### Event JSONL interface (bbus-style, no dependency)
+
+Hooks emit JSONL events to stdout. They do **not** need to import `bbus`.
+The event envelope matches the bbus style so higher layers can stream/replay.
+
+Minimal envelope:
+
+```json
+{
+  "event_id": "uuidv7",
+  "event_type": "SnapshotCreated",
+  "event_created_at": "2026-02-01T20:10:22Z",
+  "event_parent_id": "uuidv7-or-null",
+  "event_schema": "abx.events.v1",
+  "event_path": "abx-plugins",
+  "data": { "...": "event-specific fields" }
+}
+```
+
+Conventions:
+
+- Active verb names are **requests** (e.g. `BinaryInstall`, `ProcessLaunch`).
+- Past tense names are **facts** (e.g. `BinaryInstalled`, `ProcessExited`).
+- Plugins can emit additional fields inside `data` without coordination.
+
+Common event types emitted by hooks:
+
+- `ArchiveResultCreated` (status + output files)
+- `Binary` records (dependency detection/install)
+- `ProcessStarted` / `ProcessExited`
+
+Higher-level tools (abx-dl / ArchiveBox) can:
+
+- Parse these events from stdout
+- Persist or project them (SQLite/JSONL/Django) without plugins knowing
+
+Legacy note:
+
+Some hooks still emit a lightweight JSONL record with a top-level `type` field
+(e.g., `{"type": "ArchiveResult", ...}`). Runtimes should accept those and
+optionally translate them into the event envelope above.

abx_plugins-0.9.0/abx_plugins/__init__.py

@@ -0,0 +1,14 @@
+"""Plugin suite package for ArchiveBox-compatible tools."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from importlib import resources
+
+
+def get_plugins_dir() -> Path:
+    """Return the filesystem path to the bundled plugins directory."""
+    return Path(resources.files(__name__) / "plugins")
+
+
+__all__ = ["get_plugins_dir"]

abx_plugins-0.9.0/abx_plugins/plugins/__init__.py

@@ -0,0 +1 @@
+"""Plugin suite root package for abx-plugins."""

abx_plugins-0.9.0/abx_plugins/plugins/accessibility/config.json

@@ -0,0 +1,21 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "additionalProperties": false,
+  "required_plugins": ["chrome"],
+  "properties": {
+    "ACCESSIBILITY_ENABLED": {
+      "type": "boolean",
+      "default": true,
+      "x-aliases": ["SAVE_ACCESSIBILITY", "USE_ACCESSIBILITY"],
+      "description": "Enable accessibility tree capture"
+    },
+    "ACCESSIBILITY_TIMEOUT": {
+      "type": "integer",
+      "default": 30,
+      "minimum": 5,
+      "x-fallback": "TIMEOUT",
+      "description": "Timeout for accessibility capture in seconds"
+    }
+  }
+}

abx_plugins-0.9.0/abx_plugins/plugins/accessibility/on_Snapshot__39_accessibility.js

@@ -0,0 +1,294 @@
+#!/usr/bin/env node
+/**
+ * Extract accessibility tree and page outline from a URL.
+ *
+ * Extracts:
+ * - Page outline (headings h1-h6, sections, articles)
+ * - Iframe tree
+ * - Accessibility snapshot
+ * - ARIA labels and roles
+ *
+ * Usage: on_Snapshot__39_accessibility.js --url=<url> --snapshot-id=<uuid>
+ * Output: Writes accessibility/accessibility.json
+ *
+ * Environment variables:
+ *     SAVE_ACCESSIBILITY: Enable accessibility extraction (default: true)
+ */
+
+const fs = require('fs');
+const path = require('path');
+// Add NODE_MODULES_DIR to module resolution paths if set
+if (process.env.NODE_MODULES_DIR) module.paths.unshift(process.env.NODE_MODULES_DIR);
+const puppeteer = require('puppeteer-core');
+
+// Extractor metadata
+const PLUGIN_NAME = 'accessibility';
+const PLUGIN_DIR = path.basename(__dirname);
+const SNAP_DIR = path.resolve((process.env.SNAP_DIR || '.').trim());
+const OUTPUT_DIR = path.join(SNAP_DIR, PLUGIN_DIR);
+if (!fs.existsSync(OUTPUT_DIR)) {
+    fs.mkdirSync(OUTPUT_DIR, { recursive: true });
+}
+process.chdir(OUTPUT_DIR);
+const OUTPUT_FILE = 'accessibility.json';
+const CHROME_SESSION_DIR = '../chrome';
+const CHROME_SESSION_REQUIRED_ERROR = 'No Chrome session found (chrome plugin must run first)';
+
+// Parse command line arguments
+function parseArgs() {
+    const args = {};
+    process.argv.slice(2).forEach(arg => {
+        if (arg.startsWith('--')) {
+            const [key, ...valueParts] = arg.slice(2).split('=');
+            args[key.replace(/-/g, '_')] = valueParts.join('=') || true;
+        }
+    });
+    return args;
+}
+
+// Get environment variable with default
+function getEnv(name, defaultValue = '') {
+    return (process.env[name] || defaultValue).trim();
+}
+
+function getEnvBool(name, defaultValue = false) {
+    const val = getEnv(name, '').toLowerCase();
+    if (['true', '1', 'yes', 'on'].includes(val)) return true;
+    if (['false', '0', 'no', 'off'].includes(val)) return false;
+    return defaultValue;
+}
+
+// Wait for chrome tab to be fully loaded
+async function waitForChromeTabLoaded(timeoutMs = 60000) {
+    const navigationFile = path.join(CHROME_SESSION_DIR, 'navigation.json');
+    const startTime = Date.now();
+
+    while (Date.now() - startTime < timeoutMs) {
+        if (fs.existsSync(navigationFile)) {
+            return true;
+        }
+        // Wait 100ms before checking again
+        await new Promise(resolve => setTimeout(resolve, 100));
+    }
+
+    return false;
+}
+
+// Get CDP URL from chrome plugin
+function getCdpUrl() {
+    const cdpFile = path.join(CHROME_SESSION_DIR, 'cdp_url.txt');
+    if (fs.existsSync(cdpFile)) {
+        return fs.readFileSync(cdpFile, 'utf8').trim();
+    }
+    return null;
+}
+
+function assertChromeSession() {
+    const cdpFile = path.join(CHROME_SESSION_DIR, 'cdp_url.txt');
+    const targetIdFile = path.join(CHROME_SESSION_DIR, 'target_id.txt');
+    const pidFile = path.join(CHROME_SESSION_DIR, 'chrome.pid');
+    if (!fs.existsSync(cdpFile) || !fs.existsSync(targetIdFile) || !fs.existsSync(pidFile)) {
+        throw new Error(CHROME_SESSION_REQUIRED_ERROR);
+    }
+    try {
+        const pid = parseInt(fs.readFileSync(pidFile, 'utf8').trim(), 10);
+        if (!pid || Number.isNaN(pid)) throw new Error('Invalid pid');
+        process.kill(pid, 0);
+    } catch (e) {
+        throw new Error(CHROME_SESSION_REQUIRED_ERROR);
+    }
+    const cdpUrl = getCdpUrl();
+    if (!cdpUrl) {
+        throw new Error(CHROME_SESSION_REQUIRED_ERROR);
+    }
+    return cdpUrl;
+}
+
+// Extract accessibility info
+async function extractAccessibility(url) {
+    // Output directory is current directory (hook already runs in output dir)
+    const outputPath = path.join(OUTPUT_DIR, OUTPUT_FILE);
+
+    let browser = null;
+
+    try {
+        // Connect to existing Chrome session
+        const cdpUrl = assertChromeSession();
+
+        browser = await puppeteer.connect({
+            browserWSEndpoint: cdpUrl,
+        });
+
+        // Get the page
+        const pages = await browser.pages();
+        const page = pages.find(p => p.url().startsWith('http')) || pages[0];
+
+        if (!page) {
+            return { success: false, error: 'No page found in Chrome session' };
+        }
+
+        // Get accessibility snapshot
+        const accessibilityTree = await page.accessibility.snapshot({ interestingOnly: true });
+
+        // Extract page outline (headings, sections, etc.)
+        const outline = await page.evaluate(() => {
+            const headings = [];
+            const elements = document.querySelectorAll(
+                'h1, h2, h3, h4, h5, h6, a[name], header, footer, article, main, aside, nav, section, figure, summary, table, form, iframe'
+            );
+
+            elements.forEach(elem => {
+                // Skip unnamed anchors
+                if (elem.tagName.toLowerCase() === 'a' && !elem.name) return;
+
+                const tagName = elem.tagName.toLowerCase();
+                const elemId = elem.id || elem.name || elem.getAttribute('aria-label') || elem.role || '';
+                const elemClasses = (elem.className || '').toString().trim().split(/\s+/).slice(0, 3).join(' .');
+                const action = elem.action?.split('/').pop() || '';
+
+                let summary = (elem.innerText || '').slice(0, 128);
+                if (summary.length >= 128) summary += '...';
+
+                let prefix = '';
+                let title = '';
+
+                // Format headings with # prefix
+                const level = parseInt(tagName.replace('h', ''));
+                if (!isNaN(level)) {
+                    prefix = '#'.repeat(level);
+                    title = elem.innerText || elemId || elemClasses;
+                } else {
+                    // For other elements, create breadcrumb path
+                    const parents = [tagName];
+                    let node = elem.parentNode;
+                    while (node && parents.length < 5) {
+                        if (node.tagName) {
+                            const tag = node.tagName.toLowerCase();
+                            if (!['div', 'span', 'p', 'body', 'html'].includes(tag)) {
+                                parents.unshift(tag);
+                            } else {
+                                parents.unshift('');
+                            }
+                        }
+                        node = node.parentNode;
+                    }
+                    prefix = parents.join('>');
+
+                    title = elemId ? `#${elemId}` : '';
+                    if (!title && elemClasses) title = `.${elemClasses}`;
+                    if (action) title += ` /${action}`;
+                    if (summary && !title.includes(summary)) title += `: ${summary}`;
+                }
+
+                // Clean up title
+                title = title.replace(/\s+/g, ' ').trim();
+
+                if (prefix) {
+                    headings.push(`${prefix} ${title}`);
+                }
+            });
+
+            return headings;
+        });
+
+        // Get iframe tree
+        const iframes = [];
+        function dumpFrameTree(frame, indent = '>') {
+            iframes.push(indent + frame.url());
+            for (const child of frame.childFrames()) {
+                dumpFrameTree(child, indent + '>');
+            }
+        }
+        dumpFrameTree(page.mainFrame(), '');
+
+        const accessibilityData = {
+            url,
+            headings: outline,
+            iframes,
+            tree: accessibilityTree,
+        };
+
+        // Write output
+        fs.writeFileSync(outputPath, JSON.stringify(accessibilityData, null, 2));
+
+        return { success: true, output: outputPath, accessibilityData };
+
+    } catch (e) {
+        return { success: false, error: `${e.name}: ${e.message}` };
+    } finally {
+        if (browser) {
+            browser.disconnect();
+        }
+    }
+}
+
+async function main() {
+    const args = parseArgs();
+    const url = args.url;
+    const snapshotId = args.snapshot_id;
+
+    if (!url || !snapshotId) {
+        console.error('Usage: on_Snapshot__39_accessibility.js --url=<url> --snapshot-id=<uuid>');
+        process.exit(1);
+    }
+
+    const startTs = new Date();
+    let status = 'failed';
+    let output = null;
+    let error = '';
+
+    try {
+        // Check if enabled
+        if (!getEnvBool('ACCESSIBILITY_ENABLED', true)) {
+            console.log('Skipping accessibility (ACCESSIBILITY_ENABLED=False)');
+            // Output clean JSONL (no RESULT_JSON= prefix)
+            console.log(JSON.stringify({
+                type: 'ArchiveResult',
+                status: 'skipped',
+                output_str: 'ACCESSIBILITY_ENABLED=False',
+            }));
+            process.exit(0);
+        }
+
+        // Check if Chrome session exists, then wait for page load
+        assertChromeSession();
+        const pageLoaded = await waitForChromeTabLoaded(60000);
+        if (!pageLoaded) {
+            throw new Error('Page not loaded after 60s (chrome_navigate must complete first)');
+        }
+
+        const result = await extractAccessibility(url);
+
+        if (result.success) {
+            status = 'succeeded';
+            output = result.output;
+            const headingCount = result.accessibilityData.headings.length;
+            const iframeCount = result.accessibilityData.iframes.length;
+            console.log(`Accessibility extracted: ${headingCount} headings, ${iframeCount} iframes`);
+        } else {
+            status = 'failed';
+            error = result.error;
+        }
+    } catch (e) {
+        error = `${e.name}: ${e.message}`;
+        status = 'failed';
+    }
+
+    const endTs = new Date();
+
+    if (error) console.error(`ERROR: ${error}`);
+
+    // Output clean JSONL (no RESULT_JSON= prefix)
+    console.log(JSON.stringify({
+        type: 'ArchiveResult',
+        status,
+        output_str: output || error || '',
+    }));
+
+    process.exit(status === 'succeeded' ? 0 : 1);
+}
+
+main().catch(e => {
+    console.error(`Fatal error: ${e.message}`);
+    process.exit(1);
+});

abx_plugins-0.9.0/abx_plugins/plugins/accessibility/templates/icon.html

@@ -0,0 +1 @@
+<span class="abx-output-icon abx-output-icon--accessibility" title="Accessibility"><svg width="16" height="16" viewBox="0 0 24 24" aria-hidden="true" focusable="false" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="4.5" r="2" fill="currentColor" stroke="none"/><path d="M4 7.5h16"/><path d="M12 7.5v12"/><path d="M7 20l5-6 5 6"/></svg></span>