scrape-cli 1.3.6__tar.gz → 1.3.7__tar.gz

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrape_cli
-Version: 1.3.6
+Version: 1.3.7
 Summary: It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
 Author-email: Andrea Borruso <aborruso@gmail.com>
 Project-URL: Homepage, https://github.com/aborruso/scrape-cli
@@ -287,6 +287,37 @@ Output:
 
 Use `--json-array` when a script needs one shape for any number of results. The output is a flat list with no `html`/`body` envelope, one item per result, in the order of the expressions. Strings stay strings and elements become objects with the tag as key, so `scrape --json-array -e "//a/@href" -e "b"` gives `["x", "y", {"b": "tre"}]`, and no results give `[]`. It implies `-j` and has the same limits.
 
+With mixed expressions, each result keeps its own form and its place in the order of the expressions. XPath function values are strings, so `count()` gives `"2"`, not `2`:
+
+```bash
+scrape --json-array -e "h1" -e "//a/@href" -e "count(//a)" page.html
+# [{"h1": "News"}, "/one", "/two", "2"]
+```
+
+The list shape makes `jq` pipelines simple, with the same code for zero, one or many results:
+
+```bash
+# Count the results
+scrape --json-array -e "a" page.html | jq 'length'
+
+# Loop over attribute values; with no results, the loop does not run
+scrape --json-array -e "//a/@href" page.html | jq -r '.[]' | while read -r href; do
+  echo "got $href"
+done
+
+# Read fields of each element, as TSV
+scrape --json-array -e "a" page.html | jq -r '.[].a | [.["@href"], .["#text"]] | @tsv'
+
+# Filter elements by attribute
+scrape --json-array -e "a" page.html | jq -c '.[] | select(.a["@class"] == "ext")'
+
+# With mixed expressions, keep only the strings
+scrape --json-array -e "h1" -e "//a/@href" page.html | jq -c 'map(strings)'
+
+# Turn a count() into a number
+scrape --json-array -e "count(//a)" page.html | jq '.[0] | tonumber'
+```
+
 `-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 
 `-j` exits with an error when a matched element has an attribute name that is not valid XML, such as `@click` or `:bind`.
@@ -301,6 +332,68 @@ Some notes on the commands:
 - `-t` to extract only text content (useful for LLMs and text processing)
 
 
+### Several fields per element
+
+Separate `-e` expressions lose the link between fields: if an article has no date, every following date ends up on the wrong article. Use `--row` to select the elements, and one `--field NAME=EXPR` for each value, relative to the row:
+
+```bash
+scrape --row "article" --field "title=h2" --field "url=.//a/@href" --field "date=time/@datetime" page.html
+```
+
+```json
+[
+  {"title": "First story", "url": "/one", "date": "2026-01-01"},
+  {"title": "Second story", "url": "/two", "date": null}
+]
+```
+
+- Every object has the same keys, and every value is a string or `null`. `null` means no match, `""` a match with no text.
+- A field gives only its first match. Elements give their cleaned text, and XPath functions a string.
+- Field expressions are XPath or CSS, as with `-e`. Write attributes as `.//a/@href`: `a/@href` looks among direct children only, and `//a` is refused because it would read the whole document. In functions, write `string(.//h2)`.
+- `--absolute-urls` and `--base-url` apply to `href` and `src` fields.
+- `--row` cannot be combined with `-e`, `-t`, `-x`, `-a`, `-b`, `-j` or `--json-array`.
+
+### JSON Lines
+
+Use `--jsonl` to get one JSON value per line: the objects of `--row`, or otherwise the items of `--json-array`. With no results the output is empty. It cannot be combined with `-t`, `-x`, `-a`, `-b` or `-j`.
+
+```bash
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html
+# {"title": "First story", "date": "2026-01-01"}
+# {"title": "Second story", "date": null}
+
+# Articles with no date, with jq
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | jq -r 'select(.date == null) | .title'
+
+# Query with DuckDB
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | duckdb -c "select title, date from read_json('/dev/stdin', format='newline_delimited')"
+```
+
+The whole page is read and the output built in memory: `--jsonl` changes the format, not the memory use.
+
+### Absolute URLs
+
+Links and images often have relative values. Use `--absolute-urls` to get absolute `href` and `src` values, in every output mode except `-t`:
+
+```bash
+scrape --absolute-urls -e "//a/@href" https://www.iana.org/help/example-domains
+```
+
+With stdin or a local file there is no page URL, so pass it with `--base-url`, which implies `--absolute-urls`:
+
+```bash
+curl -s https://www.iana.org/help/example-domains \
+  | scrape --base-url https://www.iana.org/help/example-domains -e "//a/@href"
+```
+
+- The base is the first `<base href>` of the page, resolved against the page URL, as in browsers. With a URL as input, the page URL is the final one after redirects.
+- With no absolute base, the command exits with code 1.
+- Selection uses the original values, so `a[href^="/"]` still matches.
+- Other attributes, such as `srcset`, and XPath functions, such as `string(//a/@href)`, are not changed. Use `//a/@href`.
+- `--absolute-urls` and `--base-url` cannot be combined with `-r`, and are ignored with `-t` and `-x`.
+
 ## License
 
 [MIT](LICENSE)

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/README.md

@@ -270,6 +270,37 @@ Output:
 
 Use `--json-array` when a script needs one shape for any number of results. The output is a flat list with no `html`/`body` envelope, one item per result, in the order of the expressions. Strings stay strings and elements become objects with the tag as key, so `scrape --json-array -e "//a/@href" -e "b"` gives `["x", "y", {"b": "tre"}]`, and no results give `[]`. It implies `-j` and has the same limits.
 
+With mixed expressions, each result keeps its own form and its place in the order of the expressions. XPath function values are strings, so `count()` gives `"2"`, not `2`:
+
+```bash
+scrape --json-array -e "h1" -e "//a/@href" -e "count(//a)" page.html
+# [{"h1": "News"}, "/one", "/two", "2"]
+```
+
+The list shape makes `jq` pipelines simple, with the same code for zero, one or many results:
+
+```bash
+# Count the results
+scrape --json-array -e "a" page.html | jq 'length'
+
+# Loop over attribute values; with no results, the loop does not run
+scrape --json-array -e "//a/@href" page.html | jq -r '.[]' | while read -r href; do
+  echo "got $href"
+done
+
+# Read fields of each element, as TSV
+scrape --json-array -e "a" page.html | jq -r '.[].a | [.["@href"], .["#text"]] | @tsv'
+
+# Filter elements by attribute
+scrape --json-array -e "a" page.html | jq -c '.[] | select(.a["@class"] == "ext")'
+
+# With mixed expressions, keep only the strings
+scrape --json-array -e "h1" -e "//a/@href" page.html | jq -c 'map(strings)'
+
+# Turn a count() into a number
+scrape --json-array -e "count(//a)" page.html | jq '.[0] | tonumber'
+```
+
 `-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 
 `-j` exits with an error when a matched element has an attribute name that is not valid XML, such as `@click` or `:bind`.
@@ -284,6 +315,68 @@ Some notes on the commands:
 - `-t` to extract only text content (useful for LLMs and text processing)
 
 
+### Several fields per element
+
+Separate `-e` expressions lose the link between fields: if an article has no date, every following date ends up on the wrong article. Use `--row` to select the elements, and one `--field NAME=EXPR` for each value, relative to the row:
+
+```bash
+scrape --row "article" --field "title=h2" --field "url=.//a/@href" --field "date=time/@datetime" page.html
+```
+
+```json
+[
+  {"title": "First story", "url": "/one", "date": "2026-01-01"},
+  {"title": "Second story", "url": "/two", "date": null}
+]
+```
+
+- Every object has the same keys, and every value is a string or `null`. `null` means no match, `""` a match with no text.
+- A field gives only its first match. Elements give their cleaned text, and XPath functions a string.
+- Field expressions are XPath or CSS, as with `-e`. Write attributes as `.//a/@href`: `a/@href` looks among direct children only, and `//a` is refused because it would read the whole document. In functions, write `string(.//h2)`.
+- `--absolute-urls` and `--base-url` apply to `href` and `src` fields.
+- `--row` cannot be combined with `-e`, `-t`, `-x`, `-a`, `-b`, `-j` or `--json-array`.
+
+### JSON Lines
+
+Use `--jsonl` to get one JSON value per line: the objects of `--row`, or otherwise the items of `--json-array`. With no results the output is empty. It cannot be combined with `-t`, `-x`, `-a`, `-b` or `-j`.
+
+```bash
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html
+# {"title": "First story", "date": "2026-01-01"}
+# {"title": "Second story", "date": null}
+
+# Articles with no date, with jq
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | jq -r 'select(.date == null) | .title'
+
+# Query with DuckDB
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | duckdb -c "select title, date from read_json('/dev/stdin', format='newline_delimited')"
+```
+
+The whole page is read and the output built in memory: `--jsonl` changes the format, not the memory use.
+
+### Absolute URLs
+
+Links and images often have relative values. Use `--absolute-urls` to get absolute `href` and `src` values, in every output mode except `-t`:
+
+```bash
+scrape --absolute-urls -e "//a/@href" https://www.iana.org/help/example-domains
+```
+
+With stdin or a local file there is no page URL, so pass it with `--base-url`, which implies `--absolute-urls`:
+
+```bash
+curl -s https://www.iana.org/help/example-domains \
+  | scrape --base-url https://www.iana.org/help/example-domains -e "//a/@href"
+```
+
+- The base is the first `<base href>` of the page, resolved against the page URL, as in browsers. With a URL as input, the page URL is the final one after redirects.
+- With no absolute base, the command exits with code 1.
+- Selection uses the original values, so `a[href^="/"]` still matches.
+- Other attributes, such as `srcset`, and XPath functions, such as `string(//a/@href)`, are not changed. Use `//a/@href`.
+- `--absolute-urls` and `--base-url` cannot be combined with `-r`, and are ignored with `-t` and `-x`.
+
 ## License
 
 [MIT](LICENSE)

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scrape_cli"
-version = "1.3.6"
+version = "1.3.7"
 description = "It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector."
 readme = "README.md"
 authors = [

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/scrape_cli/__init__.py

@@ -4,7 +4,7 @@ scrape-cli - A command-line tool to extract HTML elements using XPath or CSS3 se
 
 from scrape_cli.scrape import main
 
-__version__ = "1.3.6"
+__version__ = "1.3.7"
 __author__ = "Andrea Borruso"
 __author_email__ = "aborruso@gmail.com"
 

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/scrape_cli/scrape.py

@@ -15,10 +15,12 @@ import sys
 import re
 import json
 import argparse
+import copy
 import requests
 import xmltodict
 from lxml import etree
 from cssselect import GenericTranslator
+from urllib.parse import urljoin, urlsplit
 from xml.parsers.expat import ExpatError
 from xml.sax.saxutils import escape
 
@@ -67,6 +69,92 @@ def xpath_result_items(result):
     return [result] if result else []
 
 
+def is_absolute_url(url):
+    try:
+        parts = urlsplit(url)
+    except ValueError:
+        return False
+    return bool(parts.scheme and parts.netloc)
+
+
+def resolve_url(base, value):
+    """Return value as an absolute URL, or unchanged if it cannot be parsed."""
+    try:
+        return urljoin(base, value.strip())
+    except ValueError:
+        return value
+
+
+def document_base(document, page_url):
+    """Base as in the HTML standard: the first <base href>, against the page URL."""
+    hrefs = document.xpath('(//base[@href])[1]/@href')
+    if hrefs:
+        base = resolve_url(page_url or '', hrefs[0])
+        if is_absolute_url(base):
+            return base
+    return page_url
+
+
+def with_absolute_urls(el, base):
+    """Copy the element, so that later expressions still see the original values."""
+    el = copy.deepcopy(el)
+    for node in el.xpath('descendant-or-self::*[@href or @src]'):
+        for name in ('href', 'src'):
+            if name in node.attrib:
+                node.set(name, resolve_url(base, node.get(name)))
+    return el
+
+
+def evaluate(context, expression):
+    try:
+        return context.xpath(expression)
+    except etree.XPathError as err:
+        print(f"Error: invalid XPath expression '{expression}': {err}", file=sys.stderr)
+        sys.exit(1)
+
+
+def parse_fields(fields):
+    """Turn NAME=EXPR strings into {name: XPath}, exiting on invalid ones."""
+    parsed = {}
+    for field in fields:
+        name, _, expression = (part.strip() for part in field.partition('='))
+        if not name or not expression:
+            print(f"Error: --field must be NAME=EXPR, got '{field}'.", file=sys.stderr)
+            sys.exit(1)
+        if name in parsed:
+            print(f"Error: repeated --field name '{name}'.", file=sys.stderr)
+            sys.exit(1)
+        if expression.startswith(('/', '(/')):
+            print(f"Error: --field '{field}' reads the whole document on each row: "
+                  "start the expression with .// instead.", file=sys.stderr)
+            sys.exit(1)
+        parsed[name] = expression if is_xpath(expression) else convert_css_to_xpath(expression)
+    return parsed
+
+
+def field_value(row, expression, base):
+    """String form of the first match in the row, or None if there is none."""
+    for item in xpath_result_items(evaluate(row, expression)):
+        if isinstance(item, str):
+            is_attribute = getattr(item, 'is_attribute', False)
+            if base and is_attribute and item.attrname in ('href', 'src'):
+                return resolve_url(base, item)
+            if is_attribute or item.strip():
+                return item.strip()
+        elif isinstance(item.tag, str):  # not a comment or processing instruction
+            return clean_text(''.join(item.itertext()))
+    return None
+
+
+def write_json(value, lines):
+    """Print a JSON value, or with lines one item of the list per line."""
+    if lines:
+        for item in value:
+            sys.stdout.write(json.dumps(item, ensure_ascii=False) + "\n")
+    else:
+        sys.stdout.write(json.dumps(value, indent=2, ensure_ascii=False) + "\n")
+
+
 def wrap_in_body(results):
     return "<!DOCTYPE html>\n<html>\n<body>\n" + "".join(r + "\n" for r in results) + "</body>\n</html>\n"
 
@@ -84,7 +172,7 @@ def is_xpath(expression):
     expr = expression.strip()
 
     # Direct XPath patterns
-    if expr.startswith('/') or expr.startswith('//'):
+    if expr.startswith('/') or expr.startswith('//') or expr.startswith('./'):
         return True
     if '::' in expr:
         return True
@@ -134,6 +222,14 @@ examples:
   scrape -te "//article" file.html      text of the matching elements
   scrape -je "a" file.html              JSON
   scrape --json-array -e "a" file.html  JSON, always a flat list
+  scrape --absolute-urls -e "//a/@href" https://example.org/
+                                        absolute href and src values
+  curl -s https://example.org/ | scrape --base-url https://example.org/ -e "a"
+                                        same, for stdin or a local file
+  scrape --row "article" --field "title=h2" --field "url=.//a/@href" file.html
+                                        one JSON object per article
+  scrape --jsonl -e "//a/@href" file.html
+                                        one JSON value per line
   scrape -be "//article" file.html      wrap output in <html><body>
   scrape -x -e "//h1" file.html         test only: exit code, no output
   scrape -e "//h1" https://example.com  fetch from URL
@@ -149,7 +245,14 @@ output:
              strings   one is a string, several are a list
              nothing   null
   --json-array  always a flat list, one item per result, in expression order:
-             [] for nothing, strings as strings, elements as {"tag": ...}
+             [] for nothing, strings as strings, elements as {"tag": ...},
+             function values as strings: count() gives "2"
+  --row    a list with one object per row, one key per --field in the order
+           given, each value a string or null (no match); "" is a match with
+           no text. Elements give their cleaned text. Rows with no match in
+           any field stay in the list.
+  --jsonl  one JSON value per line: the objects of --row, otherwise the items
+           of --json-array. Nothing at all when there are no results.
 
 exit codes:
   0  success, also when nothing matches: empty output, or null body with -j
@@ -165,7 +268,17 @@ notes:
   - -t with -e includes the text of any script or style inside the matches.
   - -j fails with exit 1 on attribute names that are not valid XML, such as
     @click or :bind. Use -e without -j on those pages.
+  - --field is relative to the row and gives only the first match. Write
+    .//a/@href: a/@href looks among direct children only, and //a is refused
+    because it would read the whole document. In functions, write
+    string(.//h2): string(//h2) is not refused and reads the whole document.
+  - --row is not allowed with -e, -t, -x, -a, -b, -j, --json-array.
+    --jsonl is not allowed with -t, -x, -a, -b, -j.
   - URLs: 30 s timeout, HTTP errors (4xx, 5xx) exit 1.
+  - --absolute-urls changes href and src in the output, not in selection:
+    a[href^="/"] still matches. Base: the first <base href>, against the page
+    URL or --base-url. No absolute base: exit 1. XPath functions such as
+    string(//a/@href) are not changed. Not with -r, ignored with -t and -x.
   - Charset: from the meta tag, otherwise UTF-8. If the bytes are not valid
     UTF-8: Windows-1252, then ISO-8859-1. Other encodings need the meta tag.
 '''
@@ -198,6 +311,16 @@ notes:
                         help="Output structured JSON, see below. Not with -t, -x, -a")
     parser.add_argument('--json-array', action='store_true', default=False,
                         help="Output JSON as a flat list, one item per result, see below. Same limits as -j")
+    parser.add_argument('--absolute-urls', action='store_true', default=False,
+                        help="Output href and src values as absolute URLs, see notes")
+    parser.add_argument('--base-url', default=None, metavar="URL",
+                        help="Page URL for stdin and local files. Implies --absolute-urls")
+    parser.add_argument('--row', default=None, metavar="EXPR",
+                        help="Select the elements that give one JSON object each. Needs --field")
+    parser.add_argument('--field', default=[], action='append', metavar="NAME=EXPR",
+                        help="Value of a key in each --row object, relative to the row. Repeatable")
+    parser.add_argument('--jsonl', action='store_true', default=False,
+                        help="Output one JSON value per line, see below. Implies --json-array without --row")
     # Allows to specify one or more XPath or CSS3 selector expressions
     parser.add_argument('-e', '--expression', default=[], action='append',
                         help="XPath query or CSS3 selector, detected automatically. Repeatable")
@@ -212,21 +335,46 @@ notes:
                         help="User-Agent for URL requests")
     args = parser.parse_args()
 
+    if args.jsonl and (args.text or args.check_existence or args.argument or args.body or args.json):
+        print("Error: --jsonl cannot be combined with -t, -x, -a, -b or -j.", file=sys.stderr)
+        sys.exit(1)
+    if bool(args.row) != bool(args.field):
+        print("Error: --row and --field must be used together.", file=sys.stderr)
+        sys.exit(1)
+    fields = {}
+    if args.row:
+        if (args.expression or args.text or args.check_existence or args.argument
+                or args.body or args.json or args.json_array):
+            print("Error: --row cannot be combined with -e, -t, -x, -a, -b, -j or --json-array.", file=sys.stderr)
+            sys.exit(1)
+        fields = parse_fields(args.field)
+    elif args.jsonl:
+        args.json_array = True
+
     # JSON flag is mutually exclusive with -t, -x, -a
+    json_flag = '--json-array' if args.json_array else '--json'
     if args.json_array:
         args.json = True
     if args.json and args.text:
-        print("Error: --json and --text are mutually exclusive.", file=sys.stderr)
+        print(f"Error: {json_flag} and --text are mutually exclusive.", file=sys.stderr)
         sys.exit(1)
     if args.json and args.check_existence:
-        print("Error: --json and --check-existence are mutually exclusive.", file=sys.stderr)
+        print(f"Error: {json_flag} and --check-existence are mutually exclusive.", file=sys.stderr)
         sys.exit(1)
     if args.json and args.argument:
-        print("Error: --json cannot be combined with -a/--argument.", file=sys.stderr)
+        print(f"Error: {json_flag} cannot be combined with -a/--argument.", file=sys.stderr)
+        sys.exit(1)
+
+    if args.base_url and not is_absolute_url(args.base_url):
+        print(f"Error: --base-url must be an absolute URL, got '{args.base_url}'.", file=sys.stderr)
+        sys.exit(1)
+    absolute_urls = args.absolute_urls or bool(args.base_url)
+    if absolute_urls and args.rawinput:
+        print("Error: --absolute-urls and --base-url cannot be combined with -r/--rawinput.", file=sys.stderr)
         sys.exit(1)
 
     # Check that at least one expression is provided by the user (unless using -t option)
-    if not args.expression and not args.text:
+    if not args.expression and not args.text and not args.row:
         print(
             "Error: no expression specified. Use -e \"//selector\" or -t.\n"
             "  scrape -e \"//h1\" file.html\n"
@@ -235,6 +383,7 @@ notes:
         )
         sys.exit(1)
 
+    page_url = args.base_url
     # Determine the source of the input: URL, file, or stdin
     if args.html:
         if args.html.startswith('http://') or args.html.startswith('https://'):
@@ -245,6 +394,7 @@ notes:
                 response = requests.get(args.html, headers=headers, timeout=30)
                 response.raise_for_status()
                 inp = response.content
+                page_url = page_url or response.url
             except requests.RequestException as e:
                 print(f"Error downloading HTML: {e}", file=sys.stderr)
                 sys.exit(1)
@@ -330,17 +480,32 @@ notes:
         print("Error parsing HTML: no elements found in the input", file=sys.stderr)
         sys.exit(1)
 
+    # -t and -x output no attribute values
+    base = None
+    if absolute_urls and not args.text and not args.check_existence:
+        base = document_base(document, page_url)
+        if not base:
+            print("Error: no base for --absolute-urls: use --base-url URL, "
+                  "or a page with an absolute <base href>.", file=sys.stderr)
+            sys.exit(1)
+
+    if args.row:
+        row_expression = args.row if is_xpath(args.row) else convert_css_to_xpath(args.row)
+        rows = xpath_result_items(evaluate(document, row_expression))
+        if any(isinstance(row, str) or not isinstance(row.tag, str) for row in rows):
+            print(f"Error: --row must select elements, not values: '{args.row}'.", file=sys.stderr)
+            sys.exit(1)
+        write_json([{name: field_value(row, expression, base) for name, expression in fields.items()}
+                    for row in rows], args.jsonl)
+        return
+
     results = []
     string_results = []
     # --json-array items: str for strings, bytes for element markup
     array_items = []
     # For each expression, perform the search in the parsed HTML
     for e in expression:
-        try:
-            result = document.xpath(e)
-        except etree.XPathError as err:
-            print(f"Error: invalid XPath expression '{e}': {err}", file=sys.stderr)
-            sys.exit(1)
+        result = evaluate(document, e)
 
         # If check-existence is enabled, return 0 or 1 depending on the existence of elements
         if args.check_existence:
@@ -350,6 +515,12 @@ notes:
 
         # Extract the text or content of the found elements
         for el in els:
+            if base:
+                if isinstance(el, str):
+                    if getattr(el, 'is_attribute', False) and el.attrname in ('href', 'src'):
+                        el = resolve_url(base, el)
+                elif isinstance(el.tag, str):
+                    el = with_absolute_urls(el, base)
             if args.json_array:
                 if isinstance(el, str):
                     if el.strip():
@@ -403,9 +574,9 @@ notes:
                 parsed = xmltodict.parse(wrap_in_body(results))
         except ExpatError as e:
             print(f"Error: cannot convert result to JSON, it is not valid XML ({e}). "
-                  "Attribute names like @click or :bind are not supported by -j.", file=sys.stderr)
+                  f"Attribute names like @click or :bind are not supported by {'--json-array' if args.json_array else '-j'}.", file=sys.stderr)
             sys.exit(1)
-        sys.stdout.write(json.dumps(parsed, indent=2, ensure_ascii=False) + "\n")
+        write_json(parsed, args.jsonl)
     elif args.body and not args.text:
         sys.stdout.write(wrap_in_body(results))
     else:

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/scrape_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scrape_cli
-Version: 1.3.6
+Version: 1.3.7
 Summary: It's a command-line tool to extract HTML elements using an XPath query or CSS3 selector.
 Author-email: Andrea Borruso <aborruso@gmail.com>
 Project-URL: Homepage, https://github.com/aborruso/scrape-cli
@@ -287,6 +287,37 @@ Output:
 
 Use `--json-array` when a script needs one shape for any number of results. The output is a flat list with no `html`/`body` envelope, one item per result, in the order of the expressions. Strings stay strings and elements become objects with the tag as key, so `scrape --json-array -e "//a/@href" -e "b"` gives `["x", "y", {"b": "tre"}]`, and no results give `[]`. It implies `-j` and has the same limits.
 
+With mixed expressions, each result keeps its own form and its place in the order of the expressions. XPath function values are strings, so `count()` gives `"2"`, not `2`:
+
+```bash
+scrape --json-array -e "h1" -e "//a/@href" -e "count(//a)" page.html
+# [{"h1": "News"}, "/one", "/two", "2"]
+```
+
+The list shape makes `jq` pipelines simple, with the same code for zero, one or many results:
+
+```bash
+# Count the results
+scrape --json-array -e "a" page.html | jq 'length'
+
+# Loop over attribute values; with no results, the loop does not run
+scrape --json-array -e "//a/@href" page.html | jq -r '.[]' | while read -r href; do
+  echo "got $href"
+done
+
+# Read fields of each element, as TSV
+scrape --json-array -e "a" page.html | jq -r '.[].a | [.["@href"], .["#text"]] | @tsv'
+
+# Filter elements by attribute
+scrape --json-array -e "a" page.html | jq -c '.[] | select(.a["@class"] == "ext")'
+
+# With mixed expressions, keep only the strings
+scrape --json-array -e "h1" -e "//a/@href" page.html | jq -c 'map(strings)'
+
+# Turn a count() into a number
+scrape --json-array -e "count(//a)" page.html | jq '.[0] | tonumber'
+```
+
 `-j` is mutually exclusive with `-t`, `-x` (`--check-existence`) and `-a` (`--argument`).
 
 `-j` exits with an error when a matched element has an attribute name that is not valid XML, such as `@click` or `:bind`.
@@ -301,6 +332,68 @@ Some notes on the commands:
 - `-t` to extract only text content (useful for LLMs and text processing)
 
 
+### Several fields per element
+
+Separate `-e` expressions lose the link between fields: if an article has no date, every following date ends up on the wrong article. Use `--row` to select the elements, and one `--field NAME=EXPR` for each value, relative to the row:
+
+```bash
+scrape --row "article" --field "title=h2" --field "url=.//a/@href" --field "date=time/@datetime" page.html
+```
+
+```json
+[
+  {"title": "First story", "url": "/one", "date": "2026-01-01"},
+  {"title": "Second story", "url": "/two", "date": null}
+]
+```
+
+- Every object has the same keys, and every value is a string or `null`. `null` means no match, `""` a match with no text.
+- A field gives only its first match. Elements give their cleaned text, and XPath functions a string.
+- Field expressions are XPath or CSS, as with `-e`. Write attributes as `.//a/@href`: `a/@href` looks among direct children only, and `//a` is refused because it would read the whole document. In functions, write `string(.//h2)`.
+- `--absolute-urls` and `--base-url` apply to `href` and `src` fields.
+- `--row` cannot be combined with `-e`, `-t`, `-x`, `-a`, `-b`, `-j` or `--json-array`.
+
+### JSON Lines
+
+Use `--jsonl` to get one JSON value per line: the objects of `--row`, or otherwise the items of `--json-array`. With no results the output is empty. It cannot be combined with `-t`, `-x`, `-a`, `-b` or `-j`.
+
+```bash
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html
+# {"title": "First story", "date": "2026-01-01"}
+# {"title": "Second story", "date": null}
+
+# Articles with no date, with jq
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | jq -r 'select(.date == null) | .title'
+
+# Query with DuckDB
+scrape --jsonl --row "article" --field "title=h2" --field "date=time/@datetime" page.html \
+  | duckdb -c "select title, date from read_json('/dev/stdin', format='newline_delimited')"
+```
+
+The whole page is read and the output built in memory: `--jsonl` changes the format, not the memory use.
+
+### Absolute URLs
+
+Links and images often have relative values. Use `--absolute-urls` to get absolute `href` and `src` values, in every output mode except `-t`:
+
+```bash
+scrape --absolute-urls -e "//a/@href" https://www.iana.org/help/example-domains
+```
+
+With stdin or a local file there is no page URL, so pass it with `--base-url`, which implies `--absolute-urls`:
+
+```bash
+curl -s https://www.iana.org/help/example-domains \
+  | scrape --base-url https://www.iana.org/help/example-domains -e "//a/@href"
+```
+
+- The base is the first `<base href>` of the page, resolved against the page URL, as in browsers. With a URL as input, the page URL is the final one after redirects.
+- With no absolute base, the command exits with code 1.
+- Selection uses the original values, so `a[href^="/"]` still matches.
+- Other attributes, such as `srcset`, and XPath functions, such as `string(//a/@href)`, are not changed. Use `//a/@href`.
+- `--absolute-urls` and `--base-url` cannot be combined with `-r`, and are ignored with `-t` and `-x`.
+
 ## License
 
 [MIT](LICENSE)

{scrape_cli-1.3.6 → scrape_cli-1.3.7}/tests/test_scrape.py

@@ -278,7 +278,7 @@ def test_help_documents_the_contract_for_agents():
     result = run_scrape("--help")
 
     assert result.returncode == 0
-    for section in ("output:", "exit codes:", "notes:", "examples:", "--json-array"):
+    for section in ("output:", "exit codes:", "notes:", "examples:", "--json-array", "--base-url", "--row", "--field", "--jsonl"):
         assert section in result.stdout
 
 
@@ -504,6 +504,10 @@ def test_json_array_keeps_the_order_of_the_expressions():
     assert json_array("b", "//a/@href") == [{"b": "tre"}, "x", "y"]
 
 
+def test_json_array_mixed_expressions_keep_order_and_forms():
+    assert json_array("b", "//a/@href", "count(//p)") == [{"b": "tre"}, "x", "y", "2"]
+
+
 def test_json_array_element_item():
     assert json_array("a")[0] == {"a": {"@href": "x", "#text": "l"}}
 
@@ -525,7 +529,7 @@ def test_json_array_rejects_conflicting_flags():
     for flags in (["-t"], ["-x"], ["-a", "href"]):
         result = run_scrape("--json-array", *flags, "-e", "a", input_data=ARRAY_HTML)
         assert result.returncode == 1
-        assert "Error:" in result.stderr
+        assert "Error: --json-array " in result.stderr
 
 
 def test_json_array_reports_clean_error_on_non_xml_attribute_name():
@@ -534,6 +538,12 @@ def test_json_array_reports_clean_error_on_non_xml_attribute_name():
     assert result.stdout == ""
     assert "Error:" in result.stderr
     assert "Traceback" not in result.stderr
+    assert "not supported by --json-array" in result.stderr
+
+
+def test_json_array_conflict_does_not_change_json_messages():
+    result = run_scrape("-j", "-t", "-e", "a", input_data=ARRAY_HTML)
+    assert "Error: --json and --text" in result.stderr
 
 
 def test_html_input_with_no_elements_reports_clean_error():
@@ -544,3 +554,309 @@ def test_html_input_with_no_elements_reports_clean_error():
             assert result.stdout == ""
             assert "Error parsing HTML: no elements found" in result.stderr
             assert "Traceback" not in result.stderr
+
+
+URLS_HTML = (
+    '<html><body><a id="i" href="/x">l</a><a href="https://o.org/p">o</a>'
+    '<a href="mailto:a@b.it">m</a><a href="javascript:void(0)">j</a>'
+    '<a href="http://[bad">b</a><a href="  /y  ">s</a>'
+    '<div><img src="i.png"></div></body></html>'
+)
+PAGE_URL = "https://e.org/a/b.html"
+
+
+def absolute(*args, input_data=URLS_HTML):
+    result = run_scrape("--base-url", PAGE_URL, *args, input_data=input_data)
+    assert result.returncode == 0, result.stderr
+    return result.stdout.split()
+
+
+def test_absolute_urls_attribute_expression():
+    assert absolute("-e", "//a/@href")[0] == "https://e.org/x"
+    assert absolute("-e", "//img/@src") == ["https://e.org/a/i.png"]
+
+
+def test_absolute_urls_attribute_option():
+    assert absolute("-e", "//img", "-a", "src") == ["https://e.org/a/i.png"]
+
+
+def test_absolute_urls_element_and_descendants():
+    result = run_scrape("--base-url", PAGE_URL, "-e", "div", input_data=URLS_HTML)
+    assert 'src="https://e.org/a/i.png"' in result.stdout
+    result = run_scrape("--base-url", PAGE_URL, "-be", "(//a)[1]", input_data=URLS_HTML)
+    assert 'href="https://e.org/x"' in result.stdout
+
+
+def test_absolute_urls_json_outputs():
+    result = run_scrape("--base-url", PAGE_URL, "--json-array", "-e", "(//a)[1]", input_data=URLS_HTML)
+    assert json.loads(result.stdout) == [{"a": {"@id": "i", "@href": "https://e.org/x", "#text": "l"}}]
+    result = run_scrape("--base-url", PAGE_URL, "-j", "-e", "//img/@src", input_data=URLS_HTML)
+    assert json.loads(result.stdout) == {"html": {"body": "https://e.org/a/i.png"}}
+
+
+def test_absolute_urls_leaves_other_values_unchanged():
+    assert absolute("-e", "//a/@id") == ["i"]
+    hrefs = absolute("-e", "//a/@href")
+    for value in ("https://o.org/p", "mailto:a@b.it", "javascript:void(0)", "http://[bad"):
+        assert value in hrefs
+
+
+def test_absolute_urls_strips_spaces():
+    assert absolute("-e", "//a/@href")[-1] == "https://e.org/y"
+
+
+def test_no_flag_keeps_original_values():
+    result = run_scrape("-e", "//a/@href", input_data=URLS_HTML)
+    assert result.stdout.split()[0] == "/x"
+
+
+def test_absolute_urls_selection_uses_original_values():
+    result = run_scrape("--base-url", PAGE_URL, "-e", 'a[href^="/"]', input_data=URLS_HTML)
+    assert 'href="https://e.org/x"' in result.stdout
+
+
+def test_absolute_urls_element_matched_by_two_expressions():
+    output = absolute("-e", "//a[@id='i']/@href", "-e", "//a[@href='/x']/@href")
+    assert output == ["https://e.org/x", "https://e.org/x"]
+    output = run_scrape("--base-url", PAGE_URL, "-e", "//a[@id='i']", "-e", "//a[@href='/x']", input_data=URLS_HTML)
+    assert output.stdout.count('href="https://e.org/x"') == 2
+
+
+def test_absolute_urls_leaves_function_results_unchanged():
+    assert absolute("-e", "string(//a/@href)") == ["/x"]
+
+
+def test_absolute_urls_uses_final_url_after_redirect():
+    class Handler(BaseHTTPRequestHandler):
+        def do_GET(self):
+            if self.path == "/old":
+                self.send_response(301)
+                self.send_header("Location", "/new/page.html")
+                self.end_headers()
+                return
+            body = b'<a href="x.html">l</a>'
+            self.send_response(200)
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+        def log_message(self, *args):
+            pass
+
+    server = HTTPServer(("127.0.0.1", 0), Handler)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    try:
+        port = server.server_address[1]
+        result = run_scrape("--absolute-urls", "-e", "//a/@href", f"http://127.0.0.1:{port}/old")
+    finally:
+        server.shutdown()
+        server.server_close()
+        thread.join(timeout=2)
+    assert result.stdout.strip() == f"http://127.0.0.1:{port}/new/x.html"
+
+
+def test_base_element_relative_to_page_url():
+    html = '<head><base href="/sub/"><base href="/other/"></head><img src="i.png">'
+    assert absolute("-e", "//img/@src", input_data=html) == ["https://e.org/sub/i.png"]
+
+
+def test_base_url_does_not_override_base_element():
+    html = '<head><base href="/sub/"></head><img src="i.png">'
+    result = run_scrape("--base-url", "https://e.org/", "-e", "//img/@src", input_data=html)
+    assert result.stdout.strip() == "https://e.org/sub/i.png"
+
+
+def test_base_element_that_is_not_usable_falls_back_to_page_url():
+    html = '<head><base href="http://[bad"></head><img src="i.png">'
+    assert absolute("-e", "//img/@src", input_data=html) == ["https://e.org/a/i.png"]
+
+
+def test_absolute_base_element_with_no_page_url():
+    html = '<head><base href="https://e.org/"></head><a href="/x">l</a>'
+    result = run_scrape("--absolute-urls", "-e", "//a/@href", input_data=html)
+    assert result.stdout.strip() == "https://e.org/x"
+
+
+def test_absolute_urls_with_no_base_is_an_error():
+    for html in (URLS_HTML, '<head><base href="/sub/"></head><a href="/x">l</a>'):
+        result = run_scrape("--absolute-urls", "-e", "//a/@href", input_data=html)
+        assert result.returncode == 1
+        assert result.stdout == ""
+        assert "--base-url" in result.stderr
+
+
+def test_base_url_must_be_absolute():
+    for value in ("page.html", "/a/b", "mailto:a@b.it"):
+        result = run_scrape("--base-url", value, "-e", "//a/@href", input_data=URLS_HTML)
+        assert result.returncode == 1
+        assert result.stdout == ""
+        assert "Error: --base-url" in result.stderr
+
+
+def test_absolute_urls_rejected_with_rawinput():
+    for flags in (["--absolute-urls"], ["--base-url", PAGE_URL]):
+        result = run_scrape(*flags, "-r", "-e", "//a", input_data="<a/>")
+        assert result.returncode == 1
+        assert result.stdout == ""
+        assert "-r" in result.stderr
+
+
+def test_absolute_urls_ignored_with_text_and_check_existence():
+    result = run_scrape("--absolute-urls", "-t", input_data=URLS_HTML.replace("<body>", "<body>testo "))
+    assert result.returncode == 0
+    assert "testo" in result.stdout
+    assert run_scrape("--absolute-urls", "-x", "-e", "a", input_data=URLS_HTML).returncode == 0
+
+
+ROWS_HTML = (
+    '<html><body>'
+    '<article><h2><a href="/1">Uno è</a></h2> dopo <time datetime="2026-01-01">gen</time>'
+    '<a class="tag">t1</a><a class="tag">t2</a></article>'
+    '<article><h2><a href="/2">Due</a></h2></article>'
+    '<article><h2></h2></article>'
+    '<section>fuori</section>'
+    '</body></html>'
+)
+ROW_FIELDS = ["--row", "article", "--field", "title=h2", "--field", "url=.//a/@href", "--field", "date=time/@datetime"]
+
+
+def rows(*args, input_data=ROWS_HTML):
+    result = run_scrape(*args, input_data=input_data)
+    assert result.returncode == 0, result.stderr
+    return json.loads(result.stdout)
+
+
+def rows_error(*args, input_data=ROWS_HTML):
+    result = run_scrape(*args, input_data=input_data)
+    assert result.returncode == 1, result.stdout
+    assert result.stdout == ""
+    assert "Error" in result.stderr
+    assert "Traceback" not in result.stderr
+    return result.stderr
+
+
+def test_row_fields_stay_together_when_one_is_missing():
+    assert rows(*ROW_FIELDS) == [
+        {"title": "Uno è", "url": "/1", "date": "2026-01-01"},
+        {"title": "Due", "url": "/2", "date": None},
+        {"title": "", "url": None, "date": None},
+    ]
+
+
+def test_row_keys_follow_the_order_of_the_fields():
+    assert list(rows("--row", "article", "--field", "b=h2", "--field", "a=time")[0]) == ["b", "a"]
+
+
+def test_row_no_rows():
+    assert rows("--row", "aside", "--field", "t=h2") == []
+
+
+def test_row_with_no_matching_field_is_kept():
+    assert rows("--row", "section", "--field", "t=h2") == [{"t": None}]
+
+
+def test_row_field_takes_the_first_match():
+    assert rows("--row", "article", "--field", "t=a.tag")[0] == {"t": "t1"}
+
+
+def test_row_field_with_function_gives_string():
+    assert rows("--row", "article", "--field", "n=count(.//a)")[0] == {"n": "3"}
+
+
+def test_row_field_text_leaves_out_text_after_element():
+    assert rows("--row", "article", "--field", "t=h2")[0] == {"t": "Uno è"}
+
+
+def test_row_field_expression_with_equals_sign():
+    assert rows("--row", "article", "--field", 't=a[href="/2"]')[1] == {"t": "Due"}
+
+
+def test_row_field_skips_whitespace_only_text():
+    html = "<article>\n  <b>x</b> testo</article>"
+    assert rows("--row", "article", "--field", "t=text()", input_data=html) == [{"t": "testo"}]
+
+
+def test_row_field_empty_attribute_is_empty_string():
+    assert rows("--row", "p", "--field", "t=a/@href", input_data='<p><a href="">l</a></p>') == [{"t": ""}]
+
+
+def test_row_field_with_absolute_urls():
+    result = rows("--base-url", "https://e.org/", *ROW_FIELDS)
+    assert result[0]["url"] == "https://e.org/1"
+    assert result[0]["date"] == "2026-01-01"
+
+
+def test_row_with_rawinput():
+    xml = "<rss><item><title>a</title><link>l</link></item></rss>"
+    assert rows("-r", "--row", "item", "--field", "t=title", "--field", "l=link", input_data=xml) == [
+        {"t": "a", "l": "l"}
+    ]
+
+
+def test_jsonl_rows():
+    result = run_scrape("--jsonl", *ROW_FIELDS, input_data=ROWS_HTML)
+    lines = result.stdout.splitlines()
+    assert len(lines) == 3
+    assert json.loads(lines[1]) == {"title": "Due", "url": "/2", "date": None}
+    assert "è" in lines[0]
+
+
+def test_jsonl_items_without_row():
+    for flags in (["--jsonl"], ["--jsonl", "--json-array"]):
+        result = run_scrape(*flags, "-e", "//a/@href", input_data=ROWS_HTML)
+        assert [json.loads(line) for line in result.stdout.splitlines()] == ["/1", "/2"]
+
+
+def test_jsonl_no_results():
+    for args in (["--row", "aside", "--field", "t=h2"], ["-e", "aside"]):
+        result = run_scrape("--jsonl", *args, input_data=ROWS_HTML)
+        assert result.returncode == 0
+        assert result.stdout == ""
+
+
+def test_is_xpath_recognizes_leading_dot_slash():
+    assert is_xpath("./body/h1") is True
+    result = run_scrape("-e", "./body/h1", input_data="<html><body><h1>t</h1></body></html>")
+    assert result.returncode == 0
+    assert "<h1>t</h1>" in result.stdout
+
+
+def test_row_and_field_need_each_other():
+    rows_error("--field", "t=h2")
+    rows_error("--row", "article")
+
+
+def test_row_rejects_other_output_modes():
+    for flags in (["-e", "a"], ["-t"], ["-x"], ["-a", "href"], ["-b"], ["-j"], ["--json-array"]):
+        rows_error(*ROW_FIELDS, *flags)
+
+
+def test_row_rejects_malformed_fields():
+    for field in ("title", "=h2", "title=", " =h2"):
+        assert field.strip() in rows_error("--row", "article", "--field", field)
+
+
+def test_row_rejects_repeated_names():
+    assert "title" in rows_error("--row", "article", "--field", "title=h2", "--field", "title=h3")
+
+
+def test_row_rejects_absolute_xpath_in_field():
+    for expression in ("//h2", "/html", "(//h2)[1]"):
+        assert ".//" in rows_error("--row", "article", "--field", f"t={expression}")
+
+
+def test_row_must_select_elements():
+    rows_error("--row", "//a/@href", "--field", "t=h2")
+    rows_error("--row", "count(//a)", "--field", "t=h2")
+
+
+def test_row_rejects_invalid_expressions():
+    rows_error("--row", "//[", "--field", "t=h2")
+    rows_error("--row", "article", "--field", "t=.//[")
+    rows_error("--row", "article", "--field", "t=a..b!")
+
+
+def test_jsonl_rejects_other_output_modes():
+    for flags in (["-t"], ["-x"], ["-a", "href"], ["-b"], ["-j"]):
+        assert "--jsonl" in rows_error("--jsonl", "-e", "a", *flags)