ForageFacebook 2.2.0__tar.gz → 2.3.0__tar.gz

foragefacebook-2.3.0/.python-version

@@ -0,0 +1 @@
+3.14

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/AGENTS.md

@@ -14,6 +14,7 @@ uv run forage --help                 # Run CLI
 uv run ty check src/                 # Type check (Astral's ty)
 make check                           # Lint, type check, and build
 make test                            # Read-only live Facebook checks
+make pre-pr                          # Static and live checks; set FORAGE_E2E_GROUP
 uv run ruff check src/ scripts/       # Lint
 uv run ruff format src/ scripts/      # Format
 ```
@@ -60,6 +61,8 @@ Random viewport sizes, `human_delay()` with variance, real Chromium browser, ses
 
 Use `make test` for real CLI reads through the saved Facebook session. Install Chromium first.
 Set `FORAGE_E2E_GROUP` to test a group; otherwise the test searches Marketplace for computers.
+Before you create a PR, run `FORAGE_E2E_GROUP=<active-group> make pre-pr`.
+Set `FORAGE_E2E_UNTIL=YYYY-MM-DD` to also check a historical window with known posts.
 Keep unit tests, mock tests, and fake-server suites out of this repo. CI runs static checks and package builds.
 Live tests must not log in, change credentials, or write to Facebook.
 

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/CHANGELOG.md

@@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-11-03
+
+### Added
+
+- Add Marketplace sort choices and support more currency-symbol price formats, inspired by PR #50.
+- Add local `make pre-pr` checks for live Marketplace sorts, group comments, date windows, and saved exports.
+
+### Fixed
+
+- Preserve comment nesting and remove the fallback that treats arbitrary post text as comments.
+- Read relative comment-author links and restrict permalink comments to the requested post.
+- Continue historical searches past newer posts and include posts labeled "Just now" at the default cutoff.
+- Parse compact reaction totals and recognize `multi_permalinks` post links.
+- Skip repeated Marketplace candidates before text parsing and stop before an unnecessary final group scroll.
+- Convert the CLI's miles to Facebook's kilometer radius and use the verified nearest-first sort value.
+- Select Python 3.14 for local development, matching CI.
+
+### Thanks
+
+- Thanks to [@kha-faz](https://github.com/kha-faz) for the currency and Marketplace sort contributions in [PR #50](https://github.com/jwmoss/forage/pull/50).
+
 ## [2.2.0] - 2026-10-10
 
 ### Changed
@@ -247,7 +268,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Sensitive files excluded via `.gitignore`
 - Security guidelines in SECURITY.md
 
-[Unreleased]: https://github.com/jwmoss/forage/compare/v2.2.0...HEAD
+[Unreleased]: https://github.com/jwmoss/forage/compare/v2.3.0...HEAD
+[2.3.0]: https://github.com/jwmoss/forage/compare/v2.2.0...v2.3.0
 [2.2.0]: https://github.com/jwmoss/forage/compare/v2.1.0...v2.2.0
 [2.1.0]: https://github.com/jwmoss/forage/compare/v2.0.0...v2.1.0
 [2.0.0]: https://github.com/jwmoss/forage/compare/v1.2.1...v2.0.0

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/CONTRIBUTING.md

@@ -56,7 +56,12 @@ Example: `feat: add SQLite export option`
 1. Fork the repository
 2. Create a feature branch (`git checkout -b feat/my-feature`)
 3. Make your changes
-4. Ensure tests pass (`make check`; run `make test` with a saved session)
+4. Run the local PR checks with a saved session:
+   ```bash
+   FORAGE_E2E_GROUP=your-group-slug FORAGE_E2E_UNTIL=YYYY-MM-DD make pre-pr
+   ```
+   Select an active group with visible reply threads and a historical window with known posts.
+   `FORAGE_E2E_UNTIL` is optional. It tests the seven days before the selected inclusive end date.
 5. Ensure type checking passes (`uv run ty check src/forage/`)
 6. Commit with a descriptive message
 7. Push and open a pull request
@@ -66,8 +71,11 @@ Example: `feat: add SQLite export option`
 - Verify behavior through the real CLI against Facebook.
 - Keep live checks read-only and use the existing session.
 - Do not add unit tests, mock tests, or fake-server suites.
-- Mock external dependencies (Playwright, network)
-- Aim for good coverage of edge cases
+- Run `make test-marketplace` to check all five sort orders against Facebook's selected controls.
+- Run `FORAGE_E2E_GROUP=your-group-slug make test-group` to check posts, comments, and real-data exports.
+- Run `make pre-pr` with the group variables above before you create a PR.
+- Keep session files and private records out of commits and PR descriptions.
+- Live checks need suitable records. A passing run does not prove rare cases that the live pages do not contain.
 
 ## Reporting Issues
 

foragefacebook-2.3.0/Makefile

@@ -0,0 +1,25 @@
+.PHONY: check test test-marketplace test-group test-all pre-pr build
+
+check:
+	uv lock --check --no-config
+	uv run --locked --no-config ruff check src/ scripts/
+	uv run --locked --no-config ruff format --check src/ scripts/
+	uv run --locked --no-config ty check src/forage/
+	uv build --no-config
+
+test:
+	uv run --locked --no-config python scripts/live-e2e.py
+
+test-marketplace:
+	env -u FORAGE_E2E_GROUP uv run --locked --no-config python scripts/live-e2e.py
+
+test-group:
+	@test -n "$$FORAGE_E2E_GROUP" || { echo 'Set FORAGE_E2E_GROUP to an active group you can access.' >&2; exit 2; }
+	uv run --locked --no-config python scripts/live-e2e.py
+
+test-all: test-marketplace test-group
+
+pre-pr: check test-all
+
+build:
+	uv build --no-config

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ForageFacebook
-Version: 2.2.0
+Version: 2.3.0
 Summary: CLI tool to scrape posts, comments, and reactions from private Facebook groups
 Project-URL: Homepage, https://github.com/jwmoss/forage
 Project-URL: Repository, https://github.com/jwmoss/forage
@@ -177,11 +177,19 @@ Commands:
 
 ```bash
 uv run forage marketplace "rtx 4070" --city wilmington --radius 40 --limit 20 -o listings.json
+uv run forage marketplace "computer" --sort-by price_ascend
 ```
 
 `--limit` counts accepted listings. `--max-candidates` bounds the search cost, including rejected candidates; its default is 200.
 Forage excludes distant listings and listings whose coordinates it cannot verify. City is the search center, not a city-only filter.
-The current parser supports electronics with dollar prices or “Free” labels. Search relevance follows Facebook's results.
+The parser accepts “Free” and prices with `$`, `£`, `€`, `¥`, `₹`, `₩`, or `₽` symbols.
+It accepts prefixes such as `CA$` and `R$`, trailing symbols such as `80 €`, and space-separated digits.
+Prices remain display strings. Currency conversion, currency-code-only prices, and localized free labels are unsupported.
+Search relevance follows Facebook's results.
+
+`--sort-by` accepts `creation_time_descend` (default), `suggested`, `price_ascend`, `price_descend`, or `distance_ascend`.
+These values request Facebook's sort order. They do not sort the collected subset locally.
+The CLI radius uses miles; Forage converts it to kilometers for Facebook's search URL.
 
 ### Local diagnosis and saved exports
 
@@ -327,12 +335,21 @@ uv run playwright install chromium
 # Read Marketplace through the saved Facebook session
 make test
 
-# Read one post from a group you can access
+# Read posts and comments from an active group you can access
 FORAGE_E2E_GROUP=your-group-slug make test
+
+# Run static checks, all Marketplace sorts, group comments, and saved exports before a PR
+FORAGE_E2E_GROUP=your-group-slug make pre-pr
+
+# Also check a historical window with known posts (seven days through the inclusive end date)
+FORAGE_E2E_GROUP=your-group-slug FORAGE_E2E_UNTIL=2026-10-29 make pre-pr
 ```
 
 Live tests run the real CLI and Facebook pages. They do not use fixtures or mock servers.
 They need an existing login session. CI runs static checks and package build/install checks.
+Group checks require an active group with visible reply threads. The checks use temporary local files for exports.
+Marketplace checks confirm Facebook's selected sort control for each CLI result URL.
+Live data changes. These checks do not cover a rare parser case unless the selected pages contain it.
 
 
 ## Architecture
@@ -358,7 +375,7 @@ src/forage/
 
 ## Roadmap
 
-Prioritize sanitized DOM fixtures and verified selector updates. Add incremental collection only after identity and completeness checks support it.
+Prioritize read-only live regression checks and verified selector updates. Add incremental collection only after identity and completeness checks support it.
 Use shell loops for batches, system scheduling for recurring commands, and SQLite or jq for analysis.
 
 ### Contributing

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/README.md

@@ -143,11 +143,19 @@ Commands:
 
 ```bash
 uv run forage marketplace "rtx 4070" --city wilmington --radius 40 --limit 20 -o listings.json
+uv run forage marketplace "computer" --sort-by price_ascend
 ```
 
 `--limit` counts accepted listings. `--max-candidates` bounds the search cost, including rejected candidates; its default is 200.
 Forage excludes distant listings and listings whose coordinates it cannot verify. City is the search center, not a city-only filter.
-The current parser supports electronics with dollar prices or “Free” labels. Search relevance follows Facebook's results.
+The parser accepts “Free” and prices with `$`, `£`, `€`, `¥`, `₹`, `₩`, or `₽` symbols.
+It accepts prefixes such as `CA$` and `R$`, trailing symbols such as `80 €`, and space-separated digits.
+Prices remain display strings. Currency conversion, currency-code-only prices, and localized free labels are unsupported.
+Search relevance follows Facebook's results.
+
+`--sort-by` accepts `creation_time_descend` (default), `suggested`, `price_ascend`, `price_descend`, or `distance_ascend`.
+These values request Facebook's sort order. They do not sort the collected subset locally.
+The CLI radius uses miles; Forage converts it to kilometers for Facebook's search URL.
 
 ### Local diagnosis and saved exports
 
@@ -293,12 +301,21 @@ uv run playwright install chromium
 # Read Marketplace through the saved Facebook session
 make test
 
-# Read one post from a group you can access
+# Read posts and comments from an active group you can access
 FORAGE_E2E_GROUP=your-group-slug make test
+
+# Run static checks, all Marketplace sorts, group comments, and saved exports before a PR
+FORAGE_E2E_GROUP=your-group-slug make pre-pr
+
+# Also check a historical window with known posts (seven days through the inclusive end date)
+FORAGE_E2E_GROUP=your-group-slug FORAGE_E2E_UNTIL=2026-10-29 make pre-pr
 ```
 
 Live tests run the real CLI and Facebook pages. They do not use fixtures or mock servers.
 They need an existing login session. CI runs static checks and package build/install checks.
+Group checks require an active group with visible reply threads. The checks use temporary local files for exports.
+Marketplace checks confirm Facebook's selected sort control for each CLI result URL.
+Live data changes. These checks do not cover a rare parser case unless the selected pages contain it.
 
 
 ## Architecture
@@ -324,7 +341,7 @@ src/forage/
 
 ## Roadmap
 
-Prioritize sanitized DOM fixtures and verified selector updates. Add incremental collection only after identity and completeness checks support it.
+Prioritize read-only live regression checks and verified selector updates. Add incremental collection only after identity and completeness checks support it.
 Use shell loops for batches, system scheduling for recurring commands, and SQLite or jq for analysis.
 
 ### Contributing

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/docs/capabilities.md

@@ -1,16 +1,16 @@
 # Forage capability contract
 
-Updated: 2026-09-20. This file describes the tested application behavior, not exhaustive Facebook coverage.
+Updated: 2026-11-03. This file describes observed application behavior, not exhaustive Facebook coverage.
 
 | Surface | Verified locally | Known limits |
 | --- | --- | --- |
 | Login | Browser state storage with owner-only POSIX permissions | Manual Facebook login; session validity requires Facebook access |
 | Doctor | Browser executable, session existence, permission checks | No session contents read; no browser launch; no network check |
-| Group posts | Supported HTML fixtures, full visible paragraphs, nested-comment exclusion, supported expansion controls, permalinks | English labels; Facebook layout changes; media and attachment extraction absent |
+| Group posts | Live authenticated CLI reads with native post IDs and known timestamps | English labels; Facebook layout changes; media and attachment extraction absent |
 | Dates | Inclusive calendar end dates, reversed-range rejection, relative timestamp parsing | Local system/browser timezone; month/year relative times remain approximate |
-| Comments | Native IDs, fallback IDs scoped to posts, supported reply collection | All-comment coverage remains unverified; identical fallback comments within one post can collide; comment timestamps remain unknown |
+| Comments | Live native IDs, relative author links, and reply-parent relationships; fallback IDs scoped to posts | All-comment coverage remains unverified; identical fallback comments within one post can collide; comment timestamps remain unknown |
 | Reactions | Supported total-count labels | Missing and skipped counts retain legacy zero defaults; no promise of complete per-type counts |
-| Marketplace | Electronics, dollar/free prices, title/location, accepted result limit, strict radius check bound to the requested listing ID | Fixed category; fuzzy Facebook relevance; approximate coordinates; unbound or invalid coordinates are excluded; no exhaustive search guarantee |
+| Marketplace | Live listings and all five selected sort controls; mile-to-kilometer URL conversion; strict radius check bound to the requested listing ID | Fixed category; fuzzy Facebook relevance; approximate coordinates; unbound or invalid coordinates are excluded; no exhaustive search guarantee |
 | Export | Group JSON to JSON, LLM JSON, CSV, SQLite without login | Keep original JSON for diagnostics; old overwritten comments cannot be recovered automatically |
 
 ## Result diagnostics
@@ -30,8 +30,19 @@ Meta's [Content Library announcement](https://about.fb.com/news/2023/11/new-tool
 It does not establish private-group or general personal Marketplace API access.
 
 The [Graph API v19 changelog](https://developers.facebook.com/docs/graph-api/changelog/version19.0/) is the authority for historical Groups API removal.
-The current audit cannot retrieve that page or current [Marketplace documentation](https://developers.facebook.com/docs/marketplace/) because Meta returns 429 responses.
-Current replacement availability and exhaustive REST coverage therefore remain unknown. Do not infer private APIs from DOM fields.
+The September 2026 audit cannot retrieve that page or the [Marketplace documentation](https://developers.facebook.com/docs/marketplace/) because Meta returns 429 responses.
+This update does not recheck replacement API availability. Do not infer private APIs from DOM fields.
 
-All new selector checks use synthetic, network-blocked Chromium pages. No live authenticated Facebook session validates this change.
-Verify selector changes through read-only live CLI requests. Never publish session state or private group content.
+## Local regression checks
+
+Run `FORAGE_E2E_GROUP=<active-group> FORAGE_E2E_UNTIL=YYYY-MM-DD make pre-pr` before a PR.
+The date is optional. Select a historical window with known posts.
+The checks use the saved session and real CLI reads. They require visible reply threads and verify Facebook's selected sort controls,
+post/comment records, and CSV/SQLite exports of the collected data. CI remains static and does not use Facebook credentials.
+
+On November 3, the same historical request returns zero posts on the base revision after nine rejected newer posts.
+The fix reaches a matching post after 63 newer candidates. Live group, export, and all five Marketplace sort checks pass.
+Direct parser checks cover the default "Just now" cutoff, compact reaction counts, and supported currency-symbol forms.
+Those direct checks do not establish live coverage for every rare label or locale.
+
+Never publish session state or private group content.

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ForageFacebook"
-version = "2.2.0"
+version = "2.3.0"
 description = "CLI tool to scrape posts, comments, and reactions from private Facebook groups"
 readme = "README.md"
 requires-python = ">=3.10"

foragefacebook-2.3.0/scripts/live-e2e.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python3
+"""Check real Facebook reads and exports through the CLI and saved session."""
+
+from __future__ import annotations
+
+import csv
+import json
+import os
+import re
+import shlex
+import sqlite3
+import subprocess
+import sys
+import tempfile
+from datetime import date, datetime, timedelta
+from pathlib import Path
+
+from playwright.sync_api import sync_playwright
+
+SORT_LABELS = {
+    "creation_time_descend": "Date listed: Newest first",
+    "suggested": "Suggested",
+    "price_ascend": "Price: Lowest first",
+    "price_descend": "Price: Highest first",
+    "distance_ascend": "Distance: Nearest first",
+}
+
+
+def call(*args: str) -> dict:
+    """Run the real CLI without exposing private records on failure."""
+    cli = shlex.split(os.environ.get("FORAGE_BIN", "forage"))
+    result = subprocess.run(
+        [*cli, "--no-color", *args], capture_output=True, text=True, timeout=300
+    )
+    if result.returncode:
+        raise RuntimeError(
+            f"{args[0]} failed with exit {result.returncode}; run the command to inspect the error"
+        )
+    return json.loads(result.stdout) if result.stdout.strip() else {}
+
+
+def check_group(group: str) -> None:
+    """Check real post/comment reads, saved exports, and an optional date window."""
+    result = call("scrape", group, "--limit", "3", "--no-input")
+    posts = result["posts"]
+    assert posts and len(posts) <= 3, "No group posts or post limit exceeded"
+    assert len({post["id"] for post in posts}) == len(posts), "Duplicate posts"
+    comments = []
+    for post in posts:
+        pending = list(post["comments"])
+        seen = set()
+        while pending:
+            comment = pending.pop()
+            assert comment["id"] not in seen, "Duplicate comment or reply"
+            assert comment["content"], "Empty comment"
+            assert comment["content"] != post["content"], (
+                "Post text appears as a comment"
+            )
+            seen.add(comment["id"])
+            comments.append(comment)
+            pending.extend(comment["replies"])
+    assert comments, "Use an active group with visible comments for this check"
+    assert any(c["replies"] for c in comments), "Use a group with visible reply threads"
+    assert any(c["author"]["name"] != "Unknown" for c in comments), "No comment authors"
+
+    with tempfile.TemporaryDirectory(prefix="forage-live-") as directory:
+        root = Path(directory)
+        source = root / "posts.json"
+        source.write_text(json.dumps(result), encoding="utf-8")
+        call("export", str(source), "--format", "csv", "-o", str(root / "posts.csv"))
+        with (root / "posts.csv").open() as handle:
+            assert len(list(csv.DictReader(handle))) == len(posts), "CSV loses posts"
+        with (root / "posts.comments.csv").open() as handle:
+            assert len(list(csv.DictReader(handle))) == len(comments), (
+                "CSV loses comments"
+            )
+        call("export", str(source), "--format", "sqlite", "-o", str(root / "posts.db"))
+        with sqlite3.connect(root / "posts.db") as db:
+            assert db.execute("SELECT COUNT(*) FROM posts").fetchone()[0] == len(posts)
+            assert db.execute("SELECT COUNT(*) FROM comments").fetchone()[0] == len(
+                comments
+            )
+    print(
+        "PASS live group posts, comments, replies, and CSV/SQLite exports", flush=True
+    )
+
+    until = os.environ.get("FORAGE_E2E_UNTIL")
+    if until:
+        end = date.fromisoformat(until)
+        start = end - timedelta(days=7)
+        history = call(
+            "scrape",
+            group,
+            "--limit",
+            "1",
+            "--skip-comments",
+            "--no-input",
+            "--since",
+            start.isoformat(),
+            "--until",
+            until,
+        )
+        assert history["posts"], "No historical record; use a window with known posts"
+        for post in history["posts"]:
+            assert post["timestamp"], "Historical check needs a known timestamp"
+            assert start <= datetime.fromisoformat(post["timestamp"]).date() <= end
+        print("PASS live historical date window", flush=True)
+
+
+def check_marketplace(session_path: str) -> None:
+    """Check each sort through the CLI and Facebook's selected UI control."""
+    with sync_playwright() as playwright:
+        browser = playwright.chromium.launch()
+        page = browser.new_context(storage_state=session_path).new_page()
+        try:
+            for order, label in SORT_LABELS.items():
+                result = call(
+                    "marketplace",
+                    "computer",
+                    "--limit",
+                    "1",
+                    "--max-candidates",
+                    "10",
+                    "--sort-by",
+                    order,
+                )
+                assert len(result["listings"]) == 1, "No accepted Marketplace listing"
+                assert result["listings"][0]["id"], "Missing listing ID"
+                assert result["diagnostics"]["candidates_seen"] <= 10, (
+                    "Candidate limit exceeded"
+                )
+                page.goto(result["search_url"], wait_until="domcontentloaded")
+                page.wait_for_selector('a[href*="/marketplace/item/"]')
+                radio = page.get_by_role("radio", name=label, exact=True)
+                if not radio.count():
+                    page.get_by_role("button", name=re.compile(r"^Sort by")).click()
+                assert radio.get_attribute("aria-checked") == "true", (
+                    f"Facebook ignores {order}"
+                )
+                print(f"PASS live Marketplace {order}", flush=True)
+        finally:
+            browser.close()
+
+
+def main() -> None:
+    """Verify the local setup, then read the selected live Facebook surface."""
+    doctor = call("doctor")
+    if not doctor.get("browser_installed") or not doctor.get("session_exists"):
+        raise RuntimeError(
+            "Install Chromium and save a Facebook session before live tests"
+        )
+    group = os.environ.get("FORAGE_E2E_GROUP")
+    if group:
+        check_group(group)
+    else:
+        check_marketplace(doctor["session_path"])
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except Exception as error:
+        print(f"FAIL {type(error).__name__}: {error}", file=sys.stderr)
+        sys.exit(1)

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/src/forage/__init__.py

@@ -1,3 +1,3 @@
 """Forage: CLI tool to scrape private Facebook groups."""
 
-__version__ = "2.2.0"
+__version__ = "2.3.0"

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/src/forage/cli.py

@@ -121,6 +121,22 @@ def login(ctx: Context, browser: str, session_dir: Optional[Path]):
     help="Stop after this many candidates, including radius exclusions",
 )
 @click.option("-o", "--output", type=click.Path(path_type=Path), default=None)
+@click.option(
+    "--sort-by",
+    type=click.Choice(
+        [
+            "suggested",
+            "creation_time_descend",
+            "price_ascend",
+            "price_descend",
+            "distance_ascend",
+        ],
+        case_sensitive=False,
+    ),
+    default="creation_time_descend",
+    show_default=True,
+    help="Requested Marketplace sort order; Facebook controls the results",
+)
 @click.option("--session-dir", type=click.Path(path_type=Path), default=None)
 @click.option("--headless/--no-headless", default=True)
 @click.option(
@@ -137,12 +153,13 @@ def marketplace(
     radius: str,
     limit: int,
     max_candidates: int,
+    sort_by: str,
     output: Optional[Path],
     session_dir: Optional[Path],
     headless: bool,
     browser: str,
 ) -> None:
-    """Search Marketplace electronics listings, newest first."""
+    """Search Marketplace electronics listings (newest first by default)."""
     if not session_exists(session_dir):
         console.print("[red]Please run 'forage login' first.[/red]")
         raise SystemExit(3)
@@ -152,6 +169,7 @@ def marketplace(
         radius=int(radius),
         limit=limit,
         max_candidates=max_candidates,
+        sort_by=sort_by,
         headless=headless,
         verbose=ctx.verbose,
         session_dir=session_dir,

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/src/forage/parser.py

@@ -21,6 +21,14 @@ from forage.models import (
 
 console = Console(stderr=True)
 
+# Keep prices as displayed; this recognizes symbols, not currency identities.
+_PRICE_NUMBER = r"\d(?:[\d., \u00a0\u202f]*\d)?"
+_PRICE_SYMBOL = r"(?:[A-Z]{1,3})?[$£€¥₹₩₽]"
+_PRICE_RE = re.compile(
+    rf"^(?:free|{_PRICE_SYMBOL}\s*{_PRICE_NUMBER}|{_PRICE_NUMBER}\s*{_PRICE_SYMBOL})$",
+    re.IGNORECASE,
+)
+
 
 def _warn_parse_failure(kind: str, error: Exception, *, verbose: bool) -> None:
     """Surface a parse exception instead of silently dropping the element."""
@@ -227,6 +235,10 @@ def extract_post_id(url: str) -> Optional[str]:
 
     parsed = urlparse(url)
 
+    multi_id = parse_qs(parsed.query).get("multi_permalinks", [None])[0]
+    if multi_id and multi_id.isdigit():
+        return multi_id
+
     if "story_fbid" in url:
         params = parse_qs(parsed.query)
         story_fbid = params.get("story_fbid", [None])[0]
@@ -339,7 +351,7 @@ def extract_post_identity(
 ) -> tuple[Optional[str], Optional[str]]:
     """Return this article's reliable Facebook permalink identity, if available."""
     for link in article.query_selector_all(
-        'a[href*="/posts/"], a[href*="story_fbid="]'
+        'a[href*="/posts/"], a[href*="story_fbid="], a[href*="multi_permalinks="]'
     ):
         href = link.get_attribute("href") or ""
         if "comment_id=" in href:
@@ -465,7 +477,7 @@ def parse_modern_post(
 
         # Find timestamp - look for aria-label with time info or links with timestamps
         time_links = article.query_selector_all(
-            'a[href*="/posts/"], a[href*="?story_fbid"]'
+            'a[href*="/posts/"], a[href*="story_fbid="], a[href*="multi_permalinks="]'
         )
         timestamp = _parse_post_timestamp(time_links, now=now)
 
@@ -499,9 +511,9 @@ def parse_modern_post(
             # Try finding reaction count in text like "All reactions:\n44"
             if reactions.total == 0:
                 # Look for "All reactions:" followed by a number
-                match = re.search(r"All reactions:?\s*\n?(\d+)", all_text)
+                match = re.search(r"All reactions:?\s*([\d,.]+\s*[kKmM]?)", all_text)
                 if match:
-                    reactions = Reactions(total=int(match.group(1)))
+                    reactions = Reactions(total=_parse_compact_int(match.group(1)))
 
                 # Also try just standalone numbers near "reactions" or after names
                 if reactions.total == 0:
@@ -550,6 +562,25 @@ def parse_modern_post(
         return None
 
 
+def extract_comment_identity(
+    element: ElementHandle,
+) -> tuple[Optional[str], Optional[str]]:
+    """Return a comment's native ID and its native reply-parent ID."""
+    for link in element.query_selector_all('a[href*="comment_id="]'):
+        if not link.evaluate(
+            "(node, root) => node.closest('[role=article]') === root", element
+        ):
+            continue
+        params = parse_qs(urlparse(link.get_attribute("href") or "").query)
+        parent_id = params.get("comment_id", [None])[0]
+        reply_id = params.get("reply_comment_id", [None])[0]
+        if reply_id:
+            return reply_id, parent_id
+        if parent_id:
+            return parent_id, None
+    return None, None
+
+
 def parse_modern_comment(
     element: ElementHandle,
     *,
@@ -576,36 +607,30 @@ def parse_modern_comment(
         # Try to find profile link
         links = element.query_selector_all('a[role="link"]')
         for link in links:
+            if not link.evaluate(
+                "(node, root) => node.closest('[role=article]') === root", element
+            ):
+                continue
             href = link.get_attribute("href") or ""
             text = link.inner_text().strip()
+            profile = urlparse(urljoin("https://www.facebook.com", href))
             if (
                 text
                 and len(text) < 50
-                and "facebook.com/" in href
-                and "/groups/" not in href
+                and profile.hostname in {"facebook.com", "www.facebook.com"}
+                and ("/user/" in profile.path or "/groups/" not in profile.path)
+                and "comment_id=" not in href
             ):
                 if author_name == "Unknown":
                     author_name = text
-                profile_url = href
+                profile_url = profile.geturl()
                 break
 
         content, _ = _article_content(element, author_name)
         if not content:
             return None
 
-        comment_id = None
-        for link in element.query_selector_all('a[href*="comment_id="]'):
-            if not link.evaluate(
-                "(node, root) => node.closest('[role=article]') === root.closest('[role=article]')",
-                element,
-            ):
-                continue
-            params = parse_qs(urlparse(link.get_attribute("href") or "").query)
-            comment_id = (
-                params.get("reply_comment_id") or params.get("comment_id") or [None]
-            )[0]
-            if comment_id:
-                break
+        comment_id, _ = extract_comment_identity(element)
         comment_id = comment_id or _stable_id(
             "comment", post_id, author_name, profile_url or "", content
         )
@@ -659,18 +684,14 @@ def parse_marketplace_listing(
             line.strip() for line in element.inner_text().splitlines() if line.strip()
         ]
         price_index = next(
-            (
-                index
-                for index, line in enumerate(lines)
-                if line.lower() == "free" or line.startswith("$")
-            ),
+            (index for index, line in enumerate(lines) if _PRICE_RE.fullmatch(line)),
             None,
         )
         if price_index is None or price_index + 1 >= len(lines):
             return None
 
         details = lines[price_index + 1 :]
-        while details and details[0].startswith("$"):
+        while details and _PRICE_RE.fullmatch(details[0]):
             details.pop(0)
         if not details:
             return None

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/src/forage/scraper.py

@@ -38,6 +38,8 @@ from forage.models import (
 from forage.parser import (
     filter_comments,
     extract_post_identity,
+    extract_post_id,
+    extract_comment_identity,
     parse_marketplace_listing,
     parse_modern_post,
     parse_modern_comment,
@@ -45,6 +47,11 @@ from forage.parser import (
 
 console = Console(stderr=True)
 
+COMMENT_ARTICLES = (
+    '[role="article"][aria-label^="Comment by"], '
+    '[role="article"][aria-label^="Reply by"]'
+)
+
 MARKETPLACE_CENTER_PATTERN = re.compile(
     r'"buyLocation"\s*:\s*\{\s*"latitude"\s*:\s*(-?\d+(?:\.\d+)?),'
     r'\s*"longitude"\s*:\s*(-?\d+(?:\.\d+)?)'
@@ -121,6 +128,7 @@ class MarketplaceOptions:
     verbose: bool = False
     session_dir: Optional[Path] = None
     browser_type: str = "chromium"
+    sort_by: str = "creation_time_descend"
 
 
 def random_delay(base: float = 1.0, variance: float = 0.5) -> float:
@@ -166,14 +174,17 @@ def get_group_url(group_id: str) -> str:
     return f"https://www.facebook.com/groups/{group_id}?sorting_setting=CHRONOLOGICAL"
 
 
-def get_marketplace_url(query: str, city: str, radius: int) -> str:
-    """Build a newest-first Marketplace electronics search URL."""
+def get_marketplace_url(
+    query: str, city: str, radius: int, sort_by: str = "creation_time_descend"
+) -> str:
+    """Build a Marketplace electronics search URL with the requested order."""
     city_slug = city.strip().strip("/")
     params = urlencode(
         {
             "query": query,
-            "radius": radius,
-            "sortBy": "creation_time_descend",
+            # Facebook's URL uses kilometers; the CLI and distance check use miles.
+            "radius": round(radius * 1.609344),
+            "sortBy": sort_by,
             "category": "electronics",
         }
     )
@@ -264,7 +275,8 @@ def calculate_date_range(
             date.fromisoformat(options.until), datetime_time()
         ) + timedelta(days=1)
     else:
-        until_date = now
+        # Include posts labeled "Just now" in the default, exclusive cutoff.
+        until_date = now + timedelta(microseconds=1)
 
     if options.since:
         since_date = datetime.combine(
@@ -305,19 +317,8 @@ def create_browser_context(
     return context
 
 
-def _find_comment_articles(container: Page | ElementHandle) -> list[ElementHandle]:
-    """Find likely comment article elements inside a page or post container."""
-    comment_articles = container.query_selector_all(
-        '[role="article"][aria-label^="Comment by"]'
-    )
-    if comment_articles:
-        return comment_articles
-
-    return container.query_selector_all('[role="article"]')
-
-
 def _has_comment_aria(container: Page | ElementHandle) -> bool:
-    return bool(container.query_selector('[role="article"][aria-label^="Comment by"]'))
+    return bool(container.query_selector(COMMENT_ARTICLES))
 
 
 def _comment_article_depth(element: ElementHandle, selector: str) -> int:
@@ -345,105 +346,105 @@ def _comment_article_depth(element: ElementHandle, selector: str) -> int:
         return -1
 
 
-def scrape_post_comments(
-    page: Page,
-    article: ElementHandle,
-    options: ScrapeOptions,
-    *,
-    post_id: str = "",
+def _collect_comments(
+    container: Page | ElementHandle, options: ScrapeOptions, post_id: str
 ) -> list[Comment]:
-    """
-    Scrape comments from a post by clicking to expand them.
-
-    This works on the modern Facebook interface.
-    """
-    if options.skip_comments:
-        return []
-
-    comments: list[Comment] = []
-    seen_comment_ids: set[str] = set()
-
-    try:
-        # Find all comment containers
-        # Modern Facebook comments are in divs with specific structure
-        comment_elements = _find_comment_articles(article)
-
-        if not comment_elements:
-            # Try alternative selector for comments
-            # Comments often have a specific nesting pattern
-            all_text = article.inner_text()
-            if "Comment" in all_text or "Reply" in all_text:
-                # There might be comments but in a different structure
-                # Try to find elements that look like comments
-                potential_comments = article.query_selector_all('div[dir="auto"]')
-                for elem in potential_comments:
-                    comment = parse_modern_comment(
-                        elem,
-                        skip_reactions=options.skip_reactions,
-                        verbose=options.verbose,
-                        post_id=post_id,
-                    )
-                    if (
-                        comment
-                        and comment.content
-                        and comment.id not in seen_comment_ids
-                    ):
-                        comments.append(comment)
-                        seen_comment_ids.add(comment.id)
-
-        for elem in comment_elements:
-            comment = parse_modern_comment(
-                elem,
-                skip_reactions=options.skip_reactions,
-                verbose=options.verbose,
-                post_id=post_id,
-            )
-            if comment and comment.content and comment.id not in seen_comment_ids:
-                comments.append(comment)
-                seen_comment_ids.add(comment.id)
-
-        # Try clicking "View more comments" links
-        view_more_selectors = [
-            'span:has-text("View more comments")',
-            'span:has-text("View all")',
-            '[role="button"]:has-text("comments")',
-        ]
-
-        for selector in view_more_selectors:
-            try:
-                view_more = article.query_selector(selector)
-                if view_more:
-                    view_more.click()
-                    human_delay(page, options.delay, options.delay * 0.3)
-                    break
-            except Exception:
+    """Parse article-backed comments and preserve their nesting."""
+    labeled = _has_comment_aria(container)
+    selector = COMMENT_ARTICLES if labeled else '[role="article"]'
+    seen: dict[str, Comment] = {}
+    replies: list[tuple[Comment, str]] = []
+
+    def collect(parent: Page | ElementHandle, depth: int) -> list[Comment]:
+        comments: list[Comment] = []
+        for element in parent.query_selector_all(selector):
+            if _comment_article_depth(element, selector) != depth:
                 continue
-
-        # After expanding, try to get more comments
-        expanded_comments = _find_comment_articles(article)
-        for elem in expanded_comments:
             comment = parse_modern_comment(
-                elem,
+                element,
                 skip_reactions=options.skip_reactions,
                 verbose=options.verbose,
                 post_id=post_id,
             )
-            if comment and comment.content and comment.id not in seen_comment_ids:
+            if comment is None or comment.id in seen:
+                continue
+            seen[comment.id] = comment
+            comment.replies = collect(element, depth + 1)
+            _, parent_id = extract_comment_identity(element)
+            if parent_id:
+                replies.append((comment, parent_id))
+            else:
                 comments.append(comment)
-                seen_comment_ids.add(comment.id)
-
-    except Exception as e:
-        if options.verbose:
-            console.print(f"[yellow]Warning: Error scraping comments: {e}[/yellow]")
+        return comments
 
-    # Apply filters
-    comments = filter_comments(
+    comments = collect(container, 0 if labeled else 1)
+    for reply, parent_id in replies:
+        if parent_id in seen and parent_id != reply.id:
+            seen[parent_id].replies.append(reply)
+    return filter_comments(
         comments,
         min_reactions=options.min_reactions,
         top_n=options.top_comments,
     )
 
-    return comments
+
+def _find_post_container(page: Page, post_id: str) -> Optional[ElementHandle]:
+    """Find the requested post's dialog or article without unrelated feed posts."""
+    for dialog in page.query_selector_all('[role="dialog"]'):
+        for link in dialog.query_selector_all("a[href]"):
+            if extract_post_id(link.get_attribute("href") or "") == post_id:
+                return dialog
+    for article in page.query_selector_all('[role="article"]'):
+        if extract_post_identity(article)[0] == post_id:
+            return article
+    return None
+
+
+def scrape_post_comments(
+    page: Page,
+    article: ElementHandle,
+    options: ScrapeOptions,
+    *,
+    post_id: str = "",
+) -> list[Comment]:
+    """Expand visible feed comments, then parse their article hierarchy."""
+    if options.skip_comments:
+        return []
+
+    for selector in (
+        'span:has-text("View more comments")',
+        'span:has-text("View all")',
+        '[role="button"]:has-text("comments")',
+    ):
+        try:
+            view_more = article.query_selector(selector)
+            if view_more:
+                view_more.click(timeout=2000)
+                human_delay(page, options.delay, options.delay * 0.3)
+                break
+        except Exception:
+            continue
+
+    try:
+        container = _find_post_container(page, post_id) or article
+        for control in container.query_selector_all('[role="button"]'):
+            if re.fullmatch(
+                r"View (?:\d+|more) repl(?:y|ies)", control.inner_text().strip()
+            ):
+                control.click(timeout=2000)
+                human_delay(page, options.delay, options.delay * 0.3)
+                container = _find_post_container(page, post_id) or container
+                break
+        return _collect_comments(container, options, post_id)
+    except Exception as error:
+        if options.verbose:
+            console.print(
+                f"[yellow]Warning: cannot read feed comments: {error}[/yellow]"
+            )
+        return []
+    finally:
+        if page.query_selector('[role="dialog"]'):
+            page.keyboard.press("Escape")
 
 
 def scrape_comments_from_post_page(
@@ -462,7 +463,6 @@ def scrape_comments_from_post_page(
         return []
 
     comments: list[Comment] = []
-    seen_comment_ids: set[str] = set()
     comment_page = page.context.new_page()
 
     try:
@@ -499,51 +499,12 @@ def scrape_comments_from_post_page(
             except Exception:
                 break
 
-        # Find all comment elements
-        comment_elements = _find_comment_articles(comment_page)
-        use_comment_aria = _has_comment_aria(comment_page)
-        if use_comment_aria:
-            depth_selector = '[role="article"][aria-label^="Comment by"]'
-            top_level_depth = 0
-        else:
-            # Without aria-labels every article matches, including the post
-            # itself: skip it (depth 0) and treat its direct article children
-            # as top-level comments (depth 1).
-            depth_selector = '[role="article"]'
-            top_level_depth = 1
-
-        for elem in comment_elements:
-            if _comment_article_depth(elem, depth_selector) != top_level_depth:
-                continue
-
-            comment = parse_modern_comment(
-                elem,
-                skip_reactions=options.skip_reactions,
-                verbose=options.verbose,
-                post_id=post_id or post_url,
-            )
-            if comment and comment.content and comment.id not in seen_comment_ids:
-                reply_ids: set[str] = set()
-                for reply_elem in _find_comment_articles(elem):
-                    reply = parse_modern_comment(
-                        reply_elem,
-                        skip_reactions=options.skip_reactions,
-                        verbose=options.verbose,
-                        post_id=post_id or post_url,
-                    )
-                    if (
-                        reply
-                        and reply.content
-                        and reply.id != comment.id
-                        and reply.id not in seen_comment_ids
-                        and reply.id not in reply_ids
-                    ):
-                        comment.replies.append(reply)
-                        reply_ids.add(reply.id)
-                        seen_comment_ids.add(reply.id)
-
-                seen_comment_ids.add(comment.id)
-                comments.append(comment)
+        # A permalink may show a feed with unrelated posts beside the target.
+        target_id = post_id or extract_post_id(post_url)
+        if target_id:
+            container = _find_post_container(comment_page, target_id)
+            if container:
+                comments = _collect_comments(container, options, target_id)
 
     except Exception as e:
         if options.verbose:
@@ -557,13 +518,6 @@ def scrape_comments_from_post_page(
         except Exception:
             pass
 
-    # Apply filters
-    comments = filter_comments(
-        comments,
-        min_reactions=options.min_reactions,
-        top_n=options.top_comments,
-    )
-
     return comments
 
 
@@ -703,7 +657,7 @@ def scrape_group(group: str, options: ScrapeOptions) -> ScrapeResult:
                         f"{len(articles)} are posts"
                     )
 
-                new_posts_this_page = 0
+                new_candidates_this_page = 0
 
                 for i, article in enumerate(articles):
                     identity = extract_post_identity(article)
@@ -735,6 +689,7 @@ def scrape_group(group: str, options: ScrapeOptions) -> ScrapeResult:
 
                     seen_post_ids.add(post.id)
                     diagnostics.candidates_seen += 1
+                    new_candidates_this_page += 1
 
                     if post.timestamp:
                         if post.timestamp < since_date:
@@ -766,31 +721,25 @@ def scrape_group(group: str, options: ScrapeOptions) -> ScrapeResult:
                         )
 
                         # If no comments found and we have a post URL, try navigating to it
-                        if not post.comments:
-                            # Try to find permalink
-                            permalink = article.query_selector(
-                                'a[href*="/posts/"], a[href*="?story_fbid"]'
+                        if not post.comments and post.url:
+                            post.comments = scrape_comments_from_post_page(
+                                page, post.url, options, post_id=post.id
                             )
-                            if permalink:
-                                href = permalink.get_attribute("href")
-                                if href:
-                                    if not href.startswith("http"):
-                                        href = f"https://www.facebook.com{href}"
-                                    post.comments = scrape_comments_from_post_page(
-                                        page, href, options, post_id=post.id
-                                    )
 
                         human_delay(page, options.delay * 0.5, options.delay * 0.2)
 
                     posts.append(post)
-                    new_posts_this_page += 1
 
                     progress.update(task, description=f"Scraped {len(posts)} posts...")
 
                     if options.limit and len(posts) >= options.limit:
                         break
 
-                if new_posts_this_page == 0:
+                if options.limit and len(posts) >= options.limit:
+                    diagnostics.stop_reason = "limit"
+                    break
+
+                if new_candidates_this_page == 0:
                     pages_without_new_posts += 1
                 else:
                     pages_without_new_posts = 0
@@ -862,7 +811,9 @@ def search_marketplace(
     options: MarketplaceOptions,
 ) -> MarketplaceResult:
     """Search Facebook Marketplace for electronics listings."""
-    search_url = get_marketplace_url(query, options.city, options.radius)
+    search_url = get_marketplace_url(
+        query, options.city, options.radius, options.sort_by
+    )
     listings: list[MarketplaceListing] = []
     seen_ids: set[str] = set()
     diagnostics = CollectionDiagnostics(stop_reason="no_new_listings")
@@ -912,27 +863,31 @@ def search_marketplace(
         ):
             previous_count = len(seen_ids)
             for element in page.query_selector_all('a[href*="/marketplace/item/"]'):
+                href = element.get_attribute("href") or ""
+                identity = re.search(r"/marketplace/item/(\d+)", href)
+                if identity is None or identity.group(1) in seen_ids:
+                    continue
                 listing = parse_marketplace_listing(element, verbose=options.verbose)
                 if listing is None:
                     diagnostics.parse_failures += 1
-                if listing and listing.id not in seen_ids:
-                    seen_ids.add(listing.id)
-                    diagnostics.candidates_seen += 1
-                    if _listing_matches_radius(
-                        detail_page,
-                        listing,
-                        center,
-                        options.radius,
-                        options.verbose,
-                    ):
-                        listings.append(listing)
-                    else:
-                        diagnostics.rejected_count += 1
-                    if (
-                        len(listings) >= options.limit
-                        or len(seen_ids) >= options.max_candidates
-                    ):
-                        break
+                    continue
+                seen_ids.add(listing.id)
+                diagnostics.candidates_seen += 1
+                if _listing_matches_radius(
+                    detail_page,
+                    listing,
+                    center,
+                    options.radius,
+                    options.verbose,
+                ):
+                    listings.append(listing)
+                else:
+                    diagnostics.rejected_count += 1
+                if (
+                    len(listings) >= options.limit
+                    or len(seen_ids) >= options.max_candidates
+                ):
+                    break
 
             empty_scrolls = empty_scrolls + 1 if len(seen_ids) == previous_count else 0
             if len(listings) < options.limit and len(seen_ids) < options.max_candidates:

{foragefacebook-2.2.0 → foragefacebook-2.3.0}/uv.lock

@@ -34,7 +34,7 @@ wheels = [
 
 [[package]]
 name = "foragefacebook"
-version = "2.2.0"
+version = "2.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "click" },

foragefacebook-2.2.0/Makefile

@@ -1,14 +0,0 @@
-.PHONY: check test build
-
-check:
-	uv lock --check --no-config
-	uv run --locked --no-config ruff check src/ scripts/
-	uv run --locked --no-config ruff format --check src/ scripts/
-	uv run --locked --no-config ty check src/forage/
-	uv build --no-config
-
-test:
-	uv run --locked --no-config python scripts/live-e2e.py
-
-build:
-	uv build --no-config

foragefacebook-2.2.0/scripts/live-e2e.py

@@ -1,57 +0,0 @@
-#!/usr/bin/env python3
-"""Check real Facebook reads through the installed CLI and saved session."""
-
-import json
-import os
-import shlex
-import subprocess
-import sys
-
-
-def main():
-    cli = shlex.split(os.environ.get("FORAGE_BIN", "forage"))
-
-    def call(*args):
-        result = subprocess.run(
-            [*cli, "--no-color", *args], capture_output=True, text=True, timeout=120
-        )
-        if result.returncode:
-            raise RuntimeError(
-                f"{args[0]} failed with exit {result.returncode}; run the command to inspect the error"
-            )
-        return json.loads(result.stdout)
-
-    doctor = call("doctor")
-    if not doctor.get("browser_installed") or not doctor.get("session_exists"):
-        raise RuntimeError(
-            "Install Chromium and save a Facebook session before live tests"
-        )
-    group = os.environ.get("FORAGE_E2E_GROUP")
-    if group:
-        result = call("scrape", group, "--limit", "1", "--skip-comments", "--no-input")
-        rows = result.get("posts")
-    else:
-        result = call(
-            "marketplace", "computer", "--limit", "1", "--max-candidates", "3"
-        )
-        rows = result.get("listings")
-    if not isinstance(rows, list) or not rows or not rows[0].get("id"):
-        raise RuntimeError(
-            "Live read returned no verified record; inspect access and page layout"
-        )
-    print("PASS real Facebook CLI read")
-
-
-if __name__ == "__main__":
-    try:
-        main()
-    except (
-        RuntimeError,
-        ValueError,
-        KeyError,
-        TypeError,
-        OSError,
-        subprocess.TimeoutExpired,
-    ) as error:
-        print(f"FAIL {error}", file=sys.stderr)
-        sys.exit(1)