PyPI - scrape-cli - Versions diffs - 1.2.4__tar.gz → 1.3.0__tar.gz - Mend

scrape-cli 1.2.4tar.gz → 1.3.0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (13) hide show

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrape_cli
-Version: 1.2.4
+Version: 1.3.0
 Summary: It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
 Author-email: Andrea Borruso <aborruso@gmail.com>
 Project-URL: Homepage, https://github.com/aborruso/scrape-cli
@@ -11,6 +11,7 @@ Description-Content-Type: text/markdown
 Requires-Dist: cssselect
 Requires-Dist: lxml
 Requires-Dist: requests
+Requires-Dist: xmltodict
 [![PyPI version](https://img.shields.io/pypi/v/scrape-cli.svg?label=PyPI%20version)](https://pypi.org/project/scrape-cli/)
 [![Python Versions](https://img.shields.io/pypi/pyversions/scrape-cli.svg)](https://pypi.org/project/scrape-cli/)
@@ -230,13 +231,12 @@ scrape -te 'h1, h2, h3' resources/test.html
 The `-t` option automatically excludes text from `<script>` and `<style>` tags and cleans up whitespace for better readability.
-### JSON Output Integration
+### JSON Output
-You can integrate scrape-cli with [xq](https://github.com/kislyuk/yq) (part of yq) to convert HTML output to structured JSON:
+Use the `-j`/`--json` flag to get structured JSON natively, with no external tools:
 ```bash
-# Extract and convert to JSON (requires -b for complete HTML)
-scrape -be "a.external-link" resources/test.html | xq .
+scrape -je "a.external-link" resources/test.html
 ```
 Output:
@@ -258,7 +258,7 @@ Output:
 Table extraction example:
 ```bash
-scrape -be "table.data-table td" resources/test.html | xq .
+scrape -je "table.data-table td" resources/test.html
 ```
 Output:
@@ -268,19 +268,21 @@ Output:
   "html": {
     "body": {
       "td": [
-        "1",
-        "John Doe",
-        "john@example.com",
-        "2",
-        "Jane Smith",
-        "jane@example.com"
+        "Italy",
+        "Rome",
+        "59",
+        "France",
+        "Paris",
+        "68"
       ]
     }
   }
 }
 ```
-**Note**: The `-b` flag is mandatory to produce valid HTML with `<html>`, `<head>` and `<body>` tags.
+`-j` automatically wraps the result in `<html>/<body>` before conversion, so you don't need to pass `-b`. The output is the same you would get from `scrape -be ... | xq .` (the underlying converter is `xmltodict`, the same library used by `xq`).
+`-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.
@@ -288,6 +290,7 @@ Some notes on the commands:
 - `-e` to set the query
 - `-b` to add `<html>`, `<head>` and `<body>` tags to the HTML output
+- `-j` to output structured JSON (built-in)
 - `-t` to extract only text content (useful for LLMs and text processing)

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/README.md RENAMED Viewed

@@ -216,13 +216,12 @@ scrape -te 'h1, h2, h3' resources/test.html
 The `-t` option automatically excludes text from `<script>` and `<style>` tags and cleans up whitespace for better readability.
-### JSON Output Integration
+### JSON Output
-You can integrate scrape-cli with [xq](https://github.com/kislyuk/yq) (part of yq) to convert HTML output to structured JSON:
+Use the `-j`/`--json` flag to get structured JSON natively, with no external tools:
 ```bash
-# Extract and convert to JSON (requires -b for complete HTML)
-scrape -be "a.external-link" resources/test.html | xq .
+scrape -je "a.external-link" resources/test.html
 ```
 Output:
@@ -244,7 +243,7 @@ Output:
 Table extraction example:
 ```bash
-scrape -be "table.data-table td" resources/test.html | xq .
+scrape -je "table.data-table td" resources/test.html
 ```
 Output:
@@ -254,19 +253,21 @@ Output:
   "html": {
     "body": {
       "td": [
-        "1",
-        "John Doe",
-        "john@example.com",
-        "2",
-        "Jane Smith",
-        "jane@example.com"
+        "Italy",
+        "Rome",
+        "59",
+        "France",
+        "Paris",
+        "68"
       ]
     }
   }
 }
 ```
-**Note**: The `-b` flag is mandatory to produce valid HTML with `<html>`, `<head>` and `<body>` tags.
+`-j` automatically wraps the result in `<html>/<body>` before conversion, so you don't need to pass `-b`. The output is the same you would get from `scrape -be ... | xq .` (the underlying converter is `xmltodict`, the same library used by `xq`).
+`-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.
@@ -274,6 +275,7 @@ Some notes on the commands:
 - `-e` to set the query
 - `-b` to add `<html>`, `<head>` and `<body>` tags to the HTML output
+- `-j` to output structured JSON (built-in)
 - `-t` to extract only text content (useful for LLMs and text processing)

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/pyproject.toml RENAMED Viewed

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "scrape_cli"
-version = "1.2.4"
+version = "1.3.0"
 description = "It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector."
 readme = "README.md"
 authors = [
@@ -18,7 +18,8 @@ requires-python = ">=3.8"
 dependencies = [
     "cssselect",
     "lxml",
-    "requests"
+    "requests",
+    "xmltodict"
 ]
 [project.urls]

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli/__init__.py RENAMED Viewed

@@ -4,7 +4,7 @@ scrape-cli - A command-line tool to extract HTML elements using XPath or CSS3 se
 from scrape_cli.scrape import main
-__version__ = "1.2.4"
+__version__ = "1.3.0"
 __author__ = "Andrea Borruso"
 __author_email__ = "aborruso@gmail.com"

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli/scrape.py RENAMED Viewed

@@ -13,8 +13,10 @@
 import os
 import sys
 import re
+import json
 import argparse
 import requests
+import xmltodict
 from lxml import etree
 from cssselect import GenericTranslator
@@ -109,15 +111,20 @@ examples:
   cat file.html | scrape -e "//h1"                read from stdin
   scrape -e "//h1" -x file.html                   check existence (exit 0/1)
   scrape -be "//article" file.html                wrap output in <html><body>
+  scrape -je "//article" file.html                output as JSON (built-in)
 '''
     )
     parser.add_argument('--version', action='version', version=f'%(prog)s {__version__}')
-    # Check for incorrect argument order (-eb instead of -be)
-    if '-eb' in ' '.join(sys.argv):
+    # Check for incorrect argument order (-eb instead of -be, -ej instead of -je)
+    joined_argv = ' '.join(sys.argv)
+    if '-eb' in joined_argv:
         print("Error: use -be not -eb.\n  scrape -be \"//article\" file.html", file=sys.stderr)
         sys.exit(1)
+    if '-ej' in joined_argv:
+        print("Error: use -je not -ej.\n  scrape -je \"//article\" file.html", file=sys.stderr)
+        sys.exit(1)
     # Defines the HTML input argument (can be a file, URL or stdin)
     parser.add_argument('html', nargs='?', type=str, default='',
                         help="HTML input (file, URL or stdin, default: stdin)", metavar="HTML")
@@ -130,6 +137,9 @@ examples:
     # Option to extract only text content
     parser.add_argument('-t', '--text', action='store_true', default=False,
                         help="Extract only text content (useful for LLMs)")
+    # Option to output structured JSON (built-in, no external xq needed)
+    parser.add_argument('-j', '--json', action='store_true', default=False,
+                        help="Output result as structured JSON")
     # Allows to specify one or more XPath or CSS3 selector expressions
     parser.add_argument('-e', '--expression', default=[], action='append',
                         help="XPath query or CSS3 selector")
@@ -143,6 +153,17 @@ examples:
                         help="Custom User-Agent string for HTTP requests")
     args = parser.parse_args()
+    # JSON flag is mutually exclusive with -t, -x, -a
+    if args.json and args.text:
+        print("Error: --json and --text are mutually exclusive.", file=sys.stderr)
+        sys.exit(1)
+    if args.json and args.check_existence:
+        print("Error: --json and --check-existence are mutually exclusive.", file=sys.stderr)
+        sys.exit(1)
+    if args.json and args.argument:
+        print("Error: --json cannot be combined with -a/--argument.", file=sys.stderr)
+        sys.exit(1)
     # Check that at least one expression is provided by the user (unless using -t option)
     if not args.expression and not args.text:
         print(
@@ -281,7 +302,11 @@ examples:
         results = [final_text] if final_text else []
     # Output handling
-    if args.body and not args.text:
+    if args.json:
+        wrapped = "<!DOCTYPE html>\n<html>\n<body>\n" + "".join(r + "\n" for r in results) + "</body>\n</html>\n"
+        parsed = xmltodict.parse(wrapped)
+        sys.stdout.write(json.dumps(parsed, indent=2, ensure_ascii=False) + "\n")
+    elif args.body and not args.text:
         sys.stdout.write("<!DOCTYPE html>\n<html>\n<body>\n")
         for result in results:
             sys.stdout.write(result + "\n")

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrape_cli
-Version: 1.2.4
+Version: 1.3.0
 Summary: It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
 Author-email: Andrea Borruso <aborruso@gmail.com>
 Project-URL: Homepage, https://github.com/aborruso/scrape-cli
@@ -11,6 +11,7 @@ Description-Content-Type: text/markdown
 Requires-Dist: cssselect
 Requires-Dist: lxml
 Requires-Dist: requests
+Requires-Dist: xmltodict
 [![PyPI version](https://img.shields.io/pypi/v/scrape-cli.svg?label=PyPI%20version)](https://pypi.org/project/scrape-cli/)
 [![Python Versions](https://img.shields.io/pypi/pyversions/scrape-cli.svg)](https://pypi.org/project/scrape-cli/)
@@ -230,13 +231,12 @@ scrape -te 'h1, h2, h3' resources/test.html
 The `-t` option automatically excludes text from `<script>` and `<style>` tags and cleans up whitespace for better readability.
-### JSON Output Integration
+### JSON Output
-You can integrate scrape-cli with [xq](https://github.com/kislyuk/yq) (part of yq) to convert HTML output to structured JSON:
+Use the `-j`/`--json` flag to get structured JSON natively, with no external tools:
 ```bash
-# Extract and convert to JSON (requires -b for complete HTML)
-scrape -be "a.external-link" resources/test.html | xq .
+scrape -je "a.external-link" resources/test.html
 ```
 Output:
@@ -258,7 +258,7 @@ Output:
 Table extraction example:
 ```bash
-scrape -be "table.data-table td" resources/test.html | xq .
+scrape -je "table.data-table td" resources/test.html
 ```
 Output:
@@ -268,19 +268,21 @@ Output:
   "html": {
     "body": {
       "td": [
-        "1",
-        "John Doe",
-        "john@example.com",
-        "2",
-        "Jane Smith",
-        "jane@example.com"
+        "Italy",
+        "Rome",
+        "59",
+        "France",
+        "Paris",
+        "68"
       ]
     }
   }
 }
 ```
-**Note**: The `-b` flag is mandatory to produce valid HTML with `<html>`, `<head>` and `<body>` tags.
+`-j` automatically wraps the result in `<html>/<body>` before conversion, so you don't need to pass `-b`. The output is the same you would get from `scrape -be ... | xq .` (the underlying converter is `xmltodict`, the same library used by `xq`).
+`-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 Useful for JSON-based pipelines, APIs, databases, and processing with jq/DuckDB.
@@ -288,6 +290,7 @@ Some notes on the commands:
 - `-e` to set the query
 - `-b` to add `<html>`, `<head>` and `<body>` tags to the HTML output
+- `-j` to output structured JSON (built-in)
 - `-t` to extract only text content (useful for LLMs and text processing)

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/requires.txt RENAMED Viewed

@@ -1,3 +1,4 @@
 cssselect
 lxml
 requests
+xmltodict

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/tests/test_scrape.py RENAMED Viewed

@@ -1,3 +1,4 @@
+import json
 import subprocess
 import sys
 import threading
@@ -189,6 +190,51 @@ def test_invalid_css_selector_fails_conversion():
     assert "Error converting CSS selector to XPath" in result.stderr
+def test_json_flag_produces_valid_json_for_single_element():
+    result = run_scrape(str(TEST_HTML), "-je", "a.external-link")
+    assert result.returncode == 0
+    payload = json.loads(result.stdout)
+    a = payload["html"]["body"]["a"]
+    assert a["@href"] == "https://example.com"
+    assert a["@class"] == "external-link"
+    assert a["#text"] == "Example Link"
+def test_json_flag_produces_array_for_multiple_elements():
+    result = run_scrape(str(TEST_HTML), "-je", "table.data-table td")
+    assert result.returncode == 0
+    payload = json.loads(result.stdout)
+    tds = payload["html"]["body"]["td"]
+    assert isinstance(tds, list)
+    assert "Italy" in tds
+    assert "Rome" in tds
+def test_json_flag_works_without_b_flag():
+    only_j = run_scrape(str(TEST_HTML), "-j", "-e", "a.external-link")
+    with_bj = run_scrape(str(TEST_HTML), "-bj", "-e", "a.external-link")
+    assert only_j.returncode == 0
+    assert with_bj.returncode == 0
+    assert only_j.stdout == with_bj.stdout
+def test_json_flag_conflicts_with_text_check_existence_argument():
+    jt = run_scrape(str(TEST_HTML), "-j", "-t")
+    assert jt.returncode == 1
+    assert "mutually exclusive" in jt.stderr
+    jx = run_scrape(str(TEST_HTML), "-j", "-e", "//h1", "-x")
+    assert jx.returncode == 1
+    assert "mutually exclusive" in jx.stderr
+    ja = run_scrape(str(TEST_HTML), "-j", "-e", "//a", "-a", "href")
+    assert ja.returncode == 1
+    assert "cannot be combined" in ja.stderr
 def test_url_input_downloads_and_extracts_text():
     html_bytes = TEST_HTML.read_bytes()
     server, thread = run_test_server(html_bytes)

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/SOURCES.txt RENAMED Viewed

File without changes

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/dependency_links.txt RENAMED Viewed

File without changes

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/entry_points.txt RENAMED Viewed

File without changes

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/scrape_cli.egg-info/top_level.txt RENAMED Viewed

File without changes

{scrape_cli-1.2.4 → scrape_cli-1.3.0}/setup.cfg RENAMED Viewed

File without changes

scrape-cli 1.2.4__tar.gz → 1.3.0__tar.gz

scrape-cli 1.2.4tar.gz → 1.3.0tar.gz