tutti-scraper 0.1.0__py3-none-any.whl

tutti_scraper-0.1.0.dist-info/METADATA

@@ -0,0 +1,510 @@
+Metadata-Version: 2.4
+Name: tutti-scraper
+Version: 0.1.0
+Summary: Free, keyless scraper and library for tutti.ch listings. Unofficial, not affiliated with Tamedia or tutti.ch.
+Project-URL: Homepage, https://github.com/danyk20/tutti-scraper
+Project-URL: Repository, https://github.com/danyk20/tutti-scraper
+Project-URL: Issues, https://github.com/danyk20/tutti-scraper/issues
+Project-URL: Changelog, https://github.com/danyk20/tutti-scraper/blob/master/CHANGELOG.md
+Author-email: Daniel Košč <maintainer@danielkosc.eu>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: classifieds,cli,scraper,switzerland,tutti,web-scraping
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: requests~=2.34
+Provides-Extra: dev
+Requires-Dist: mypy~=2.1; extra == 'dev'
+Requires-Dist: pytest-cov~=7.1; extra == 'dev'
+Requires-Dist: pytest~=9.1; extra == 'dev'
+Requires-Dist: responses~=0.26; extra == 'dev'
+Requires-Dist: ruff~=0.15; extra == 'dev'
+Requires-Dist: types-requests~=2.33; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# tutti.ch Scraper
+
+[![CI](https://github.com/danyk20/tutti-scraper/actions/workflows/ci.yml/badge.svg)](https://github.com/danyk20/tutti-scraper/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/tutti-scraper)](https://pypi.org/project/tutti-scraper/)
+[![Coverage](https://img.shields.io/badge/unit%20test%20coverage-100%25-brightgreen)](#testing)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Python 3.11 | 3.12](https://img.shields.io/badge/python-3.11%20%7C%203.12-blue)](https://www.python.org/)
+
+> Unofficial, independently developed project. Not affiliated with,
+> endorsed by, or sponsored by Tamedia or tutti.ch. "tutti.ch" is a
+> trademark of its respective owner.
+
+Fetches every listing matching a search phrase from tutti.ch, for free — no
+API key, no token, no paid scraping service.
+
+This project is the sibling of
+[`autoscout24-scraper`](https://github.com/danyk20/autoscout24-scraper) —
+same `scrape()`/`ScrapeResult`/CLI shape, deliberately kept interchangeable
+so switching between them (or writing code that works with either) means
+changing the import and the search arguments, not the whole integration.
+See [Interchangeability with autoscout24-scraper](#interchangeability-with-autoscout24-scraper)
+below for exactly what that means in practice.
+
+**🤖 This project is robot-friendly.** It is explicitly intended to be used
+by AI agents and bots exactly as a human developer would: to run it, read
+its output, import it into another project, or adapt its code. It's released
+under the very permissive [MIT license](LICENSE) specifically so there is no
+ambiguity about that — see [License](#license) below.
+
+## How it works
+
+tutti.ch's own frontend loads its data from a **GraphQL API** at
+`https://www.tutti.ch/api/v10/graphql` — not a published/documented schema,
+but the query documents this scraper sends were recovered from tutti.ch's
+compiled JS bundles and verified against the live API. It needs a specific
+header set (`X-Tutti-Hash` — any UUID, regenerated per request —
+`X-Tutti-Source`, `X-Tutti-Client-Identifier`), which `TuttiClient` sets
+automatically.
+
+| Operation | Purpose |
+|---|---|
+| `searchListingsByQuery(query, category, constraints)` | the search, paginated up to 100 results at a time, plus tutti.ch's own suggested sub-categories for the query |
+| `listing(listingID)` | full detail record for one listing — visited once per listing by default |
+
+**The pagination cap, and how this scraper works around it.** A single
+`(query, category, constraints, sort)` combination can only be paged up to
+an offset of about 3000 — larger offsets return a server error, not more
+results. For a query with more matches than that, this scraper recursively
+partitions the search:
+
+1. Search the phrase with no filters. If the total is small enough, page
+   through it directly.
+2. Otherwise, split by every category tutti.ch suggests for the query, and
+   repeat per category.
+3. If a single category is still too large, bisect it by price range
+   (binary search on min/max price) until each slice is small enough to
+   page through.
+4. A dedicated free-only pass and an ascending-sort sweep mop up listings
+   that a price filter or a single sort order might otherwise miss.
+
+Listings are de-duplicated by ID across all of this, so overlap between
+slices is harmless. In pathological cases (thousands of listings clustered
+at the exact same price, in the exact same category) full coverage still
+isn't guaranteed — see [Notes](#notes) below; the scraper logs a warning
+whenever a slice it visited was larger than what's reachable.
+
+**Two-phase scraping.** The search operation only returns a summary per
+listing. To get everything (GPS coordinates, full-resolution images,
+structured attributes, richer seller info), the scraper visits each
+listing individually, one by one, via its detail operation, after the
+search phase has collected every id. That's one extra HTTP request per
+listing, with a delay between requests — use `--no-detail` to skip this
+and keep only the fast summary fields.
+
+Every field the API returns for a listing is extracted — nested objects
+(seller, postcode/canton, category, images, structured attributes, ...)
+are flattened into columns, and lists are joined into a single
+semicolon-separated cell — so no data from the API response is dropped on
+the way into the CSV.
+
+## Locales
+
+Every function and the CLI accept a `lang` (default `"de"`) — `"fr"` and
+`"it"` are both live tutti.ch locales, used both for the `Accept-Language`
+header sent to the API and for which localized URL slug gets used to build
+each listing's `url`.
+
+## Setup
+
+Requires [pipenv](https://pipenv.pypa.io/) (`brew install pipenv` if you
+don't have it).
+
+```bash
+cd tutti-scraper
+pipenv install --dev
+```
+
+(`--dev` also installs the test/lint tooling — pytest, pytest-cov, responses,
+ruff, mypy. Leave it off if you only want to run the scraper.)
+
+```bash
+pipenv run ruff check .          # lint
+pipenv run ruff format --check . # formatting (drop --check to auto-format)
+pipenv run mypy tutti_scraper.py # type-check
+```
+
+These are exactly the checks the CI workflow (`.github/workflows/ci.yml`)
+runs on every push/PR, across Python 3.11 and 3.12.
+
+## Usage
+
+The scraper works two ways: as a standalone CLI script that writes files, or
+as a library you import into another project to get the data back directly.
+
+### As a CLI script
+
+```bash
+pipenv run python tutti_scraper.py "velo"
+```
+
+(If you installed the package via `pip install` instead, as described in the
+"as a library" section below, the same command is just
+`tutti-scraper "velo"` — no `pipenv run` needed.)
+
+This prints progress while paging through results, then visits every
+matching listing one by one to pull full details, and writes two output
+files in the current directory: `velo.csv` and `velo.json`.
+
+### Options
+
+| Flag | Description |
+|---|---|
+| `--version` | Print the installed version and exit |
+| `query` | Search phrase, e.g. `velo` or `"Tesla Roadster"` (required, positional) |
+| `--lang` | Listing/URL locale — `de` (default), `fr`, or `it` |
+| `--out` | Output file base name, without extension. Defaults to a slug of the search phrase |
+| `--no-detail` | Skip visiting each listing individually; keep only the summary fields from the search results (faster, fewer fields) |
+| `--sort` | Sort order tutti.ch searches with — `timestamp` (default), `price`, or `relevance` |
+| `--max` | Stop after N listings |
+| `--delay` | Seconds to wait between requests (default `1.0`) — raise this if you get rate-limited |
+| `-v` / `--verbose` | Also show debug-level detail, including every HTTP request made (mutually exclusive with `-q`) |
+| `-q` / `--quiet` | Suppress progress output; only warnings/errors are shown (mutually exclusive with `-v`) |
+
+### Examples
+
+```bash
+# Full run: search + visit every listing for full details (default)
+pipenv run python tutti_scraper.py "velo"
+
+# Custom output filename
+pipenv run python tutti_scraper.py "velo" --out my_search
+
+# Fast mode: search results only, skip visiting each listing
+pipenv run python tutti_scraper.py "velo" --no-detail
+
+# Stop after the first 50 listings found
+pipenv run python tutti_scraper.py "velo" --max 50
+
+# French locale
+pipenv run python tutti_scraper.py "velo" --lang fr
+
+# Any free-text phrase works, including multi-word ones
+pipenv run python tutti_scraper.py "Tesla Roadster"
+```
+
+### As a library, from another project
+
+Import `scrape()` and call it directly — it does the same work as the CLI
+(search, then visit every listing for full detail) but returns a
+`ScrapeResult` object instead of writing files. No files are written unless
+you explicitly ask for them.
+
+```python
+from tutti_scraper import scrape
+
+result = scrape("velo", max_results=50)
+
+result.rows       # list[dict]: one flattened dict per listing, CSV-ready
+result.listings   # list[dict]: raw (unflattened) API JSON per listing, each with a "url" field
+result.query, result.total_elements, result.lang
+
+for row in result.rows:
+    print(row["price"], row["title"], row["url"])
+
+# Optional: write to disk anyway, e.g. for a one-off export
+result.to_csv("velo.csv")
+result.to_json("velo.json")
+```
+
+This section is the authoritative reference for the return types — both for
+a human integrating this into another project, and for an AI agent that
+needs to know exactly what it's going to get back without having to read
+the whole source file.
+
+#### `scrape()` signature
+
+```python
+def scrape(
+    query: str,                      # free-text search phrase, e.g. "velo" or "Tesla Roadster"
+    *,
+    lang: str = "de",                # "de" (default), "fr", or "it"
+    detail: bool = True,             # visit every listing individually for full fields (slower)
+    sort: str = "timestamp",         # "timestamp" (default), "price", or "relevance"
+    max_results: int | None = None,  # stop after this many unique listings, if given
+    delay: float = 1.0,              # seconds between HTTP requests
+    verbose: bool = True,            # emit progress via the "tutti_scraper" logger at INFO level
+    client: TuttiClient | None = None,  # reuse a client across calls if given
+) -> ScrapeResult:
+    ...
+```
+
+Raises `TuttiError` on unrecoverable GraphQL/HTTP errors from tutti.ch after
+retries are exhausted, and `requests.RequestException` subclasses on
+unrecoverable network errors.
+
+**Logging.** Library code never configures logging itself (no
+`basicConfig`, no handlers) — it only emits through
+`logging.getLogger("tutti_scraper")`, same as any well-behaved library. That
+means if you call `scrape()` from your own script with no logging
+configuration of your own, `verbose=True`'s progress messages exist but
+won't be visible anywhere, by design — Python's standard "libraries don't
+talk unless you ask them to" behavior. To see them:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)  # now scrape()'s progress is visible
+```
+
+The CLI is the one place that *does* configure real handlers automatically
+(see `--verbose`/`--quiet` above) — that's the only difference between
+running this as a script versus importing it.
+
+#### `ScrapeResult` — the return value
+
+```python
+@dataclass
+class ScrapeResult:
+    query: str              # the search phrase, as requested
+    total_elements: int     # number of unique listings found by the search phase
+    listings: list[dict]    # raw API objects — see "Data structure" below
+    rows: list[dict]        # flattened dicts, one per listing, CSV-ready, sorted by price ascending
+    lang: str                # locale that was scraped, e.g. "de"
+
+    def to_csv(self, path: str) -> None: ...   # writes self.rows
+    def to_json(self, path: str) -> None: ...  # writes self.listings
+```
+
+`len(result.rows) == len(result.listings) == result.total_elements` always
+holds (barring `--no-detail`/`detail=False`, where they still match — detail
+mode only adds fields, it never drops or adds listings).
+
+Install it into your own project's environment with:
+
+```bash
+pip install tutti-scraper
+```
+
+(Not yet published? Install the latest unreleased code straight from GitHub
+instead: `pip install git+https://github.com/danyk20/tutti-scraper.git`.)
+
+Either way you also get a real `tutti-scraper` command (see `--version`
+above), not just the importable module — pipenv is only needed if you're
+working on this repo itself (running its CLI from source, or its test
+suite).
+
+## Interchangeability with autoscout24-scraper
+
+Both projects share the same shape on purpose, so code written against one
+transfers to the other with minimal changes:
+
+| Concept | `autoscout24-scraper` | `tutti-scraper` |
+|---|---|---|
+| Search call | `scrape(make, model, **filters)` | `scrape(query, **filters)` |
+| Return value | `ScrapeResult` (`.rows`, `.listings`, `.to_csv()`, `.to_json()`) | same |
+| Common row keys | `row["price"]`, `row["url"]` | same |
+| Detail toggle | `detail=True` / `--no-detail` | same |
+| Reusable transport | `session: requests.Session | None` | `client: TuttiClient | None` (see below) |
+| Locale/region | `domain: str = "ch"` | `lang: str = "de"` |
+| Logging | `logging.getLogger("autoscout24_scraper")`, `-v`/`-q` CLI flags | `logging.getLogger("tutti_scraper")`, same flags |
+| CLI entry point | `main(argv=None)` / `run_cli(argv=None)` | same |
+
+Two things are genuinely different, not just renamed, because the two sites'
+domains actually differ:
+
+- **`client` vs. `session`.** tutti.ch's API needs a specific header set
+  regenerated per request (see [How it works](#how-it-works)), so tutti's
+  reusable transport object is a small `TuttiClient` class wrapping a
+  session, not a bare `requests.Session`. If you're writing code that
+  should work against either scraper, treat this parameter as opaque
+  (pass `None` to let each library build its own) rather than constructing
+  it yourself.
+- **Search shape.** AutoScout24 lets you filter by price/mileage/year server
+  side, because it's a structured make/model catalog. tutti.ch is a general
+  classifieds site searched by free text, so those specific filters don't
+  have a tutti equivalent — `scrape()`'s `sort`/`max_results` options exist
+  instead, for the free-text case.
+
+If you're adapting this project to a *third* data source, following this
+same shape (`scrape()` → `ScrapeResult`, `flatten_listing()`/`save_csv()`/
+`save_json()`/`order_fieldnames()`, `main()`/`run_cli()`, logging via
+`logging.getLogger(...)`) is the recommended path — several of tutti's
+utility functions (`_scalarize()`, `order_fieldnames()`, `save_csv()`,
+`save_json()`) are generic enough to reuse close to verbatim.
+
+## Data structure
+
+This section documents exactly what's in the output — precisely enough that
+a developer or an AI agent can parse it without having to run the scraper
+first and reverse-engineer the shape themselves.
+
+### JSON (`result.listings` / the `.json` file)
+
+The JSON file (and `ScrapeResult.listings`) is a **JSON array of listing
+objects**, one per ad found. Every listing object always includes:
+
+| Field | Type | Description |
+|---|---|---|
+| `listingID` | `string` | tutti.ch's internal listing id |
+| `url` | `string \| null` | **Full URL of the original ad** on tutti.ch, e.g. `https://www.tutti.ch/de/vi/bern/velos/velo/76699338` — added by this scraper (the raw API response does not include it), so you can always click straight back to the source listing. `null` if tutti.ch didn't return a URL slug for the requested `lang` |
+| `title` | `string` | |
+| `body` | `string` | ad description text |
+| `timestamp` | `string` | ISO 8601 |
+| `formattedPrice` | `string \| null` | display price as tutti.ch renders it, e.g. `"550.-"` |
+| `postcodeInformation` | `object` | `{"postcode", "locationName", "canton": {"shortName", "name"}}` |
+| `primaryCategory` | `object` | `{"categoryID", ...}` (richer in detail shape, see below) |
+| `sellerInfo` | `object` | `{"alias", "logoURL", ...}` (richer in detail shape, see below) |
+
+There are two possible **shapes** for the rest of the object, depending on
+whether detail mode ran:
+
+- **Summary shape** (`detail=False` / `--no-detail`): the search operation's
+  fields only — includes a `thumbnail` (small renditions), but no GPS
+  coordinates, no full-resolution images, no structured attributes.
+- **Detail shape** (`detail=True`, the default): adds `coordinates`
+  (`{"latitude", "longitude"}`), `images` (`list[{"rendition": {"src"}}]`,
+  full resolution), `properties` (`list[{"listingPropertyID", "label",
+  "text"}]`, structured attributes specific to the listing's category),
+  `address`, `phoneInfo` (`{"isMobile", "phoneHash"}`), a richer
+  `sellerInfo` (adds `locationName`, `url`, `memberSince`,
+  `publicAccountID`), and a richer `primaryCategory` (adds `label`,
+  `parent`). `seoInformation` also gains `numericPrice` (`number \| null`)
+  — the raw numeric price the display-only `formattedPrice` string doesn't
+  give you, used to sort `result.rows` and promoted to a flat `price`
+  column in CSV output.
+
+There is no published/versioned schema for these objects — the tables above
+reflect the fields observed in practice as of this writing, recovered from
+tutti.ch's own compiled JS (see [How it works](#how-it-works)). Treat
+unknown/missing fields defensively (`.get(...)`, not `[...]`) since tutti.ch
+can add or omit fields per listing.
+
+### CSV (`result.rows` / the `.csv` file)
+
+The CSV is a **flattened** version of the same data — one row per listing,
+same rows/listings correspondence and order. Flattening rules (also
+available programmatically as `flatten_listing()`):
+
+- `seoInformation.numericPrice` is promoted to a top-level `price` column
+  (also present as `seoInformation_numericPrice`).
+- `sellerInfo` becomes `sellerAlias`, `sellerLocationName`,
+  `sellerMemberSince`.
+- `primaryCategory` becomes `category` (label if available, else the raw
+  category id) and `categoryKey` (always the raw id).
+- `postcodeInformation` becomes `postcode`, `locationName`, `canton`,
+  `cantonKey`.
+- `thumbnail` becomes `thumbnailURL`.
+- `images` is joined into one semicolon-separated cell of image URLs.
+- `properties` is joined into one semicolon-separated cell of
+  `"label: text"` pairs.
+- Any other nested object becomes `parent_child` columns, e.g.
+  `coordinates.latitude` → `coordinates_latitude`.
+- `url` is always present as its own column (same value as the JSON `url`
+  field described above).
+- Columns are the union of every field seen across all rows (heterogeneous
+  listings don't crash the writer — missing values are an empty string),
+  with `listingID, title, price, formattedPrice, category, postcode,
+  locationName, canton, timestamp, sellerAlias, url` pinned first and
+  everything else sorted alphabetically after them.
+
+## Testing
+
+The CI badge above is live (it reflects the actual state of the most recent
+GitHub Actions run). The coverage badge is a static snapshot of the last
+verified `pytest` run, not wired to a live coverage service — enforced
+locally and in CI via the `--cov-fail-under=95` gate described below, so it
+can't silently drop without the build going red.
+
+The test suite lives in `tests/` and is split into two kinds of tests:
+
+- **Unit tests** (`tests/test_*.py`, excluding `test_e2e.py`) — every
+  function is tested in isolation with HTTP mocked out (via the
+  [`responses`](https://github.com/getsentry/responses) library) or via an
+  in-memory `FakeClient` test double for the partitioning algorithm, so
+  they run in well under a second, need no network access, and never touch
+  the real site. This is the default `pytest` run.
+- **End-to-end tests** (`tests/test_e2e.py`) — make real calls against
+  tutti.ch. They're marked `@pytest.mark.e2e` and excluded by default; run
+  them explicitly when you want to confirm the scraper still works against
+  the live API (e.g. after tutti.ch changes something). They target "Tesla
+  Roadster" specifically because its inventory is small and stable, so the
+  full detail-visiting pipeline and a real CLI run both complete in a few
+  seconds without hammering the API.
+
+```bash
+# Unit tests only (fast, no network) — this is what `pytest` runs by default.
+# Also prints a coverage report and fails the run if coverage drops below 95%.
+pipenv run pytest
+
+# End-to-end tests only (real network calls, several seconds)
+pipenv run pytest -m e2e --no-cov
+
+# Everything
+pipenv run pytest -m "e2e or not e2e" --no-cov
+
+# HTML coverage report you can open in a browser
+pipenv run pytest --cov-report=html && open htmlcov/index.html
+```
+
+The unit suite covers 100% of `tutti_scraper.py` (the one line excluded via
+`# pragma: no cover` is the `if __name__ == "__main__":` guard itself,
+which is exercised for real by the e2e suite's CLI run instead).
+
+What's covered:
+
+| Area | Unit tests | E2E tests |
+|---|---|---|
+| `TuttiClient._post` | retry-then-succeed and exhausted-retries paths for GraphQL errors, 429/5xx, connection errors; no retry on 4xx; fresh hash per attempt | — |
+| `Scraper` (partitioning) | direct paging, category split (once, not recursively), price bisection, free-only pass, ascending sweep, dedup, depth/range safety valve + warning | — |
+| `search_listings` / `visit_all_listings` | max-results early stop, progress logging, detail-fetch-failure fallback | real search + detail fetch |
+| `flatten_listing` / `_scalarize` / `order_fieldnames` | every branch (nested dicts, lists, missing/unrecognized types) | implicitly, via real data |
+| `save_csv` / `save_json` / `ScrapeResult` | heterogeneous rows, unicode, empty input | round-trip against real files |
+| `scrape()` | orchestration, sorting, client reuse/construction, verbose logging | full real pipeline, with and without `detail` |
+| `main()` / `run_cli()` | every CLI flag, default vs. custom output filenames, all three exit-code paths | real subprocess-equivalent run, real error exit code |
+
+## Notes
+
+- Be a reasonable citizen: the default delay between requests is intentional.
+  Don't remove it or crank up concurrency — this is an undocumented API the
+  site's own frontend uses, not a public API with a stated rate limit.
+  tutti.ch's `robots.txt` disallows `/api/` (the search *pages* themselves,
+  e.g. `/de/q/...`, are allowed) — this scraper calls the API directly
+  because it's the only reliable way to get complete, structured data for
+  large result sets.
+- If tutti.ch changes their GraphQL schema or required headers, the
+  `TuttiClient`, `Scraper`, `search_listings`, and `visit_all_listings`
+  functions/classes are the places to look — the module docstring at the
+  top of `tutti_scraper.py` documents the endpoint shapes in more detail.
+  Run the e2e suite after any such change to confirm the fix.
+- Extremely large, heavily price-clustered result sets (e.g. thousands of
+  listings at the exact same price, in the exact same category) can still
+  exceed what's reachable even after bisection; `Scraper` logs a warning
+  whenever a slice it scraped was larger than it could fully cover.
+
+## License
+
+This project is released under the [MIT License](LICENSE) — one of the most
+permissive open-source licenses that exist. In plain terms: you can use,
+copy, modify, merge, publish, distribute, and even sell copies of this code,
+for free, for any purpose, commercial or not, as long as the license text
+stays attached. There is no warranty.
+
+**AI agents, LLM-based coding assistants, and other bots are explicitly
+welcome to use this project** — to run the scraper, to read and parse its
+output, to import `scrape()` into another project, or to read and adapt its
+source code — under exactly the same terms as a human would, with no
+additional restriction and no need to ask permission. That's the whole
+point of the fully-typed [`scrape()` signature and `ScrapeResult`
+reference](#as-a-library-from-another-project) and the [Data
+structure](#data-structure) section above: so a bot reading this file can
+integrate with the code correctly without a human in the loop, same as a
+person reading it would.
+
+The one thing this permissive license does *not* do is grant any rights to
+tutti.ch's own data or terms of service — this project only automates
+requests to the same GraphQL API their own website already calls from any
+visitor's browser; what you do with the results is between you and them.

tutti_scraper-0.1.0.dist-info/RECORD

@@ -0,0 +1,6 @@
+tutti_scraper.py,sha256=Z1U4YYMs0sQwvaDK9v7eFT4ksFZXlyPoeG6bB-ueMds,27304
+tutti_scraper-0.1.0.dist-info/METADATA,sha256=fphxHkAbDjbfFlx5lodhCf-zqf2iQyi_sQ59Gz5Ywmg,24960
+tutti_scraper-0.1.0.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+tutti_scraper-0.1.0.dist-info/entry_points.txt,sha256=UahFy3SEWESqR59o8suuX4ybaCfEBQZC3qtElWHKhVU,56
+tutti_scraper-0.1.0.dist-info/licenses/LICENSE,sha256=Z2X9MhMOvq79ywuoydtHNVmLydb6pxJ4JrahRxylfYU,1170
+tutti_scraper-0.1.0.dist-info/RECORD,,

tutti_scraper-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any

tutti_scraper-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+tutti-scraper = tutti_scraper:run_cli

tutti_scraper-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,23 @@
+MIT License
+
+Copyright (c) 2026 Daniel Košč
+
+Permission is hereby granted, free of charge, to any person or automated
+agent (including AI assistants, bots, and other software acting on behalf
+of a person) obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without restriction,
+including without limitation the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to the
+following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

tutti_scraper.py

@@ -0,0 +1,712 @@
+#!/usr/bin/env python3
+"""
+tutti.ch listing scraper.
+
+Talks directly to the GraphQL API tutti.ch's own website calls
+(https://www.tutti.ch/api/v10/graphql) rather than scraping rendered HTML.
+The query documents used here (SEARCH_QUERY, DETAIL_QUERY) were recovered
+from tutti.ch's compiled JS bundles and verified against the live API; the
+GraphQL schema itself isn't publicly documented.
+
+Discovered endpoints:
+  POST https://www.tutti.ch/api/v10/graphql
+       operation searchListingsByQuery(query, category, constraints)
+       -> paginated listing summaries + suggested sub-categories
+       operation listing(listingID)
+       -> full detail record for one listing
+
+Required headers (a normal `requests` client must set these explicitly;
+tutti.ch's frontend sets them via its own bundled JS):
+  X-Tutti-Hash: any UUID, regenerated per request
+  X-Tutti-Source: "web LIVE"
+  X-Tutti-Client-Identifier: "web/1.0.0+env-live.git-0000000"
+
+Pagination quirk: a single (query, category, constraints, sort) combination
+can only be paged up to an offset of ~3000 - larger offsets return a server
+error. To cover result sets bigger than that, this scraper recursively
+partitions the search: first by every category tutti.ch suggests for the
+query, then (if a single category is still too large) by binary-searched
+price range, until every slice is small enough to page through completely.
+A dedicated free-only pass and an ascending-sort sweep mop up listings a
+price filter or a single sort order might otherwise miss. Listings are
+de-duplicated by ID across all of this.
+
+After the search phase collects every listing id, the scraper (by default)
+visits each listing's detail operation one by one to extract fields the
+search summary doesn't return: GPS coordinates, full-resolution images,
+structured attributes, and richer seller info. This is slower (one extra
+request per listing) but gives full details.
+
+Language: every function that builds a URL or requests server-rendered
+slugs takes an optional `lang` (default "de"), matching tutti.ch/<lang>/....
+"fr" and "it" are also live tutti.ch locales.
+
+This module can be used two ways:
+
+1. As a standalone CLI script that writes a CSV + JSON file:
+
+    python3 tutti_scraper.py "velo"
+    python3 tutti_scraper.py "Tesla Roadster" --out tesla_roadster
+    python3 tutti_scraper.py "velo" --no-detail   # skip per-listing detail fetch
+
+2. As a library, imported from another project, returning data directly
+   instead of writing files:
+
+    from tutti_scraper import scrape
+
+    result = scrape("velo", max_results=50)
+    for row in result.rows:          # flattened dicts, one per listing
+        print(row["price"], row["url"])
+    result.listings                  # raw (unflattened) API JSON per listing
+    result.to_csv("velo.csv")        # optional, if you want a file after all
+    result.to_json("velo.json")
+"""
+
+import argparse
+import csv
+import json
+import logging
+import re
+import sys
+import time
+import uuid
+from collections.abc import Iterable
+from dataclasses import dataclass, field
+from typing import Any
+
+import requests
+
+__version__ = "0.1.0"
+
+API_URL = "https://www.tutti.ch/api/v10/graphql"
+
+USER_AGENT = (
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
+)
+
+SEARCH_QUERY = """
+query Search(
+  $q: String, $category: ID, $constraints: ListingSearchConstraints,
+  $first: Int!, $offset: Int!, $sort: ListingSortMode!, $direction: SortDirection!
+) {
+  searchListingsByQuery(query: $q, category: $category, constraints: $constraints) {
+    suggestedCategories { categoryID label }
+    listings(first: $first, offset: $offset, sort: $sort, direction: $direction) {
+      totalCount
+      edges {
+        node {
+          listingID
+          title
+          body
+          postcodeInformation { postcode locationName canton { shortName name } }
+          timestamp
+          formattedPrice
+          formattedSource
+          highlighted
+          primaryCategory { categoryID }
+          sellerInfo { alias logoURL }
+          thumbnail {
+            normalRendition: rendition(width: 235, height: 167) { src }
+            retinaRendition: rendition(width: 470, height: 334) { src }
+          }
+          seoInformation {
+            deSlug: slug(language: DE)
+            frSlug: slug(language: FR)
+            itSlug: slug(language: IT)
+          }
+        }
+      }
+    }
+  }
+}
+"""
+
+DETAIL_QUERY = """
+query Detail($id: ListingID!) {
+  listing(listingID: $id) {
+    listingID
+    title
+    body
+    language
+    externalURL
+    postcodeInformation { postcode locationName canton { shortName name } }
+    coordinates { latitude longitude }
+    timestamp
+    formattedPrice
+    formattedSource
+    highlighted
+    sellerInfo { alias logoURL locationName url memberSince publicAccountID }
+    images(first: 15) { rendition(width: 1024, height: 768) { src } }
+    primaryCategory {
+      categoryID
+      label
+      parent { categoryID label }
+    }
+    address
+    phoneInfo { isMobile phoneHash }
+    properties {
+      ... on ListingPropertyDescription { listingPropertyID label text }
+    }
+    seoInformation {
+      deSlug: slug(language: DE)
+      frSlug: slug(language: FR)
+      itSlug: slug(language: IT)
+      numericPrice
+    }
+  }
+}
+"""
+
+PAGE_SIZE = 100
+MAX_OFFSET = 3000  # highest offset the API accepts reliably (probed live)
+MAX_TOTAL = MAX_OFFSET + PAGE_SIZE  # items reachable in one linear pass
+MAX_PRICE = 100_000_000  # CHF ceiling for price bisection (covers real estate)
+MAX_BISECT_DEPTH = 40
+
+PRIORITY_FIELDS = [
+    "listingID",
+    "title",
+    "price",
+    "formattedPrice",
+    "category",
+    "postcode",
+    "locationName",
+    "canton",
+    "timestamp",
+    "sellerAlias",
+    "url",
+]
+
+# Library code only ever logs through this logger - it never calls
+# basicConfig or attaches handlers of its own (that would be rude to a host
+# application). The CLI (see _configure_cli_logging(), used by main()) is the
+# only place that sets up real handlers, so plain library use is silent
+# unless the caller configures logging themselves, e.g.:
+#     import logging; logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger("tutti_scraper")
+logger.addHandler(logging.NullHandler())
+
+
+class TuttiError(Exception):
+    pass
+
+
+class TuttiClient:
+    """Thin GraphQL client for tutti.ch's API: session + required headers
+    + retry/backoff. Roughly the tutti equivalent of a plain
+    requests.Session for a REST API - except tutti's API needs a specific
+    header set (see module docstring) and a fresh X-Tutti-Hash per request,
+    so it's a small class rather than a bare session."""
+
+    def __init__(self, lang="de", delay=1.0, max_retries=5):
+        self.lang = lang
+        self.delay = delay
+        self.max_retries = max_retries
+        self.session = requests.Session()
+        self.session.headers.update(
+            {
+                "Content-Type": "application/json",
+                "User-Agent": USER_AGENT,
+                "Accept-Language": f"{lang}-CH",
+                "X-Tutti-Source": "web LIVE",
+                "X-Tutti-Client-Identifier": "web/1.0.0+env-live.git-0000000",
+            }
+        )
+
+    def _post(self, query, variables):
+        body = {"query": query, "variables": variables}
+        backoff = self.delay
+        last_exc: Exception | None = None
+        for attempt in range(1, self.max_retries + 1):
+            self.session.headers["X-Tutti-Hash"] = str(uuid.uuid4())
+            try:
+                resp = self.session.post(API_URL, json=body, timeout=30)
+            except requests.RequestException as exc:
+                last_exc = exc
+                logger.warning("POST %s failed (%s); retry %d/%d", API_URL, exc, attempt, self.max_retries)
+                time.sleep(backoff)
+                backoff *= 2
+                continue
+            if resp.status_code == 200:
+                data = resp.json()
+                if data.get("errors"):
+                    last_exc = TuttiError(str(data["errors"]))
+                    logger.warning("GraphQL errors: %s; retry %d/%d", data["errors"], attempt, self.max_retries)
+                    time.sleep(backoff)
+                    backoff *= 2
+                    continue
+                time.sleep(self.delay)
+                return data["data"]
+            if resp.status_code in (429, 500, 502, 503, 504):
+                last_exc = TuttiError(f"HTTP {resp.status_code}")
+                logger.warning("POST %s -> %d; retry %d/%d", API_URL, resp.status_code, attempt, self.max_retries)
+                time.sleep(backoff)
+                backoff *= 2
+                continue
+            resp.raise_for_status()
+        raise TuttiError(f"giving up after {self.max_retries} attempts: {last_exc}")
+
+    def search(
+        self,
+        query,
+        category=None,
+        constraints=None,
+        offset=0,
+        first=PAGE_SIZE,
+        sort="TIMESTAMP",
+        direction="DESCENDING",
+    ):
+        variables = {
+            "q": query,
+            "category": category,
+            "constraints": constraints,
+            "first": first,
+            "offset": offset,
+            "sort": sort,
+            "direction": direction,
+        }
+        data = self._post(SEARCH_QUERY, variables)
+        return data["searchListingsByQuery"]
+
+    def fetch_detail(self, listing_id):
+        data = self._post(DETAIL_QUERY, {"id": listing_id})
+        return data["listing"]
+
+
+def price_constraint(pmin=None, pmax=None, free_only=False):
+    entry = {"key": "price", "freeOnly": free_only}
+    if pmin is not None:
+        entry["min"] = pmin
+    if pmax is not None:
+        entry["max"] = pmax
+    return {"prices": [entry]}
+
+
+class Scraper:
+    """Recursively partitions a search until every slice fits in one
+    linear pass, de-duplicating listings by ID across all slices. This is
+    tutti's equivalent of a simple paginated search - more involved than
+    that would otherwise be, only because of the ~3000-offset pagination
+    cap (see module docstring). search_listings(), below, is the plain
+    function wrapper most callers should use instead of this class directly."""
+
+    def __init__(self, client, query, sort="TIMESTAMP"):
+        self.client = client
+        self.query = query
+        self.sort = sort
+        self.seen = set()
+
+    def run(self):
+        yield from self._scrape(category=None, constraints=None, allow_category_split=True)
+        # Sweep from the opposite end too: for query/category combos that
+        # stayed too large even after splitting, DESCENDING and ASCENDING
+        # sample different items from the same offset-limited window.
+        yield from self._linear_page(category=None, constraints=None, direction="ASCENDING")
+
+    def _probe_total(self, category, constraints):
+        result = self.client.search(self.query, category=category, constraints=constraints, offset=0, first=1)
+        listings = result["listings"]
+        return listings["totalCount"], result.get("suggestedCategories") or []
+
+    def _linear_page(self, category, constraints, direction="DESCENDING"):
+        offset = 0
+        while True:
+            result = self.client.search(
+                self.query,
+                category=category,
+                constraints=constraints,
+                offset=offset,
+                first=PAGE_SIZE,
+                sort=self.sort,
+                direction=direction,
+            )
+            listings = result["listings"]
+            edges = listings["edges"]
+            if not edges:
+                break
+            for edge in edges:
+                node = edge["node"]
+                listing_id = node["listingID"]
+                if listing_id not in self.seen:
+                    self.seen.add(listing_id)
+                    yield node
+            offset += PAGE_SIZE
+            if offset > MAX_OFFSET or offset >= listings["totalCount"]:
+                break
+
+    def _scrape(self, category, constraints, allow_category_split):
+        total, suggested = self._probe_total(category, constraints)
+        if total == 0:
+            return
+        if total <= MAX_TOTAL:
+            yield from self._linear_page(category, constraints)
+            return
+        if allow_category_split and suggested:
+            for cat in suggested:
+                yield from self._scrape(cat["categoryID"], constraints, allow_category_split=False)
+            return
+        yield from self._bisect_price(category, constraints, 0, MAX_PRICE, depth=0)
+        # Price-bucketed constraints can exclude listings with no numeric
+        # price (e.g. "price on request"); a dedicated free-only pass
+        # recovers at least the free ones.
+        free_constraints = price_constraint(free_only=True)
+        yield from self._linear_page(category, free_constraints)
+
+    def _bisect_price(self, category, base_constraints, pmin, pmax, depth):
+        constraints = price_constraint(pmin, pmax)
+        total, _ = self._probe_total(category, constraints)
+        if total == 0:
+            return
+        if total <= MAX_TOTAL or depth >= MAX_BISECT_DEPTH or pmax - pmin <= 1:
+            yield from self._linear_page(category, constraints)
+            if total > MAX_TOTAL:
+                logger.warning(
+                    "category=%s price=[%s,%s] has %d listings but only %d are reachable; "
+                    "some listings in this slice may be missing",
+                    category,
+                    pmin,
+                    pmax,
+                    total,
+                    MAX_TOTAL,
+                )
+            return
+        mid = (pmin + pmax) // 2
+        yield from self._bisect_price(category, base_constraints, pmin, mid, depth + 1)
+        yield from self._bisect_price(category, base_constraints, mid, pmax, depth + 1)
+
+
+def search_listings(client, query, *, sort="TIMESTAMP", max_results=None, verbose=True):
+    """Search tutti.ch for `query` and return every reachable listing
+    summary as a list of raw node dicts (see SEARCH_QUERY for the shape).
+    Stops early once `max_results` unique listings have been collected, if
+    given."""
+    scraper = Scraper(client, query, sort=sort)
+    nodes = []
+    for node in scraper.run():
+        nodes.append(node)
+        if verbose and len(nodes) % PAGE_SIZE == 0:
+            logger.info("... %d listings found so far", len(nodes))
+        if max_results is not None and len(nodes) >= max_results:
+            break
+    return nodes
+
+
+def visit_all_listings(client, listings, *, verbose=True):
+    """Visit each listing's detail operation one by one, merging the
+    richer detail fields (coordinates, full-res images, structured
+    attributes, seller info) into a copy of its summary dict. A listing
+    whose detail fetch fails keeps its summary fields instead of being
+    dropped."""
+    detailed = []
+    total = len(listings)
+    for i, node in enumerate(listings, 1):
+        record = dict(node)
+        try:
+            detail = client.fetch_detail(node["listingID"])
+            record.update(detail)
+        except TuttiError as exc:
+            logger.warning("Detail fetch failed for %s: %s", node["listingID"], exc)
+        detailed.append(record)
+        if verbose:
+            logger.info("Visited %d/%d listings", i, total)
+    return detailed
+
+
+def listing_url(node, lang="de"):
+    slug = (node.get("seoInformation") or {}).get(f"{lang}Slug")
+    if not slug:
+        return None
+    return f"https://www.tutti.ch/{lang}/vi/{slug}/{node['listingID']}"
+
+
+def _scalarize(value: Any) -> Any:
+    """Turn a nested dict/list value into something that fits one CSV cell."""
+    if value is None:
+        return ""
+    if isinstance(value, (str, int, float, bool)):
+        return value
+    if isinstance(value, dict):
+        for key in ("name", "label", "src", "shortName"):
+            if key in value and not isinstance(value[key], (dict, list)):
+                return value[key]
+        return json.dumps(value, ensure_ascii=False, sort_keys=True)
+    if isinstance(value, list):
+        return "; ".join(str(_scalarize(v)) for v in value)
+    return str(value)
+
+
+def flatten_listing(item: dict[str, Any], lang: str = "de") -> dict[str, Any]:
+    """Flatten a listing (search-summary or full-detail shape) into one flat
+    dict covering every field tutti.ch returned for it, so nothing is lost."""
+    flat: dict[str, Any] = {}
+    for key, value in item.items():
+        if key == "seoInformation" and isinstance(value, dict):
+            flat["price"] = value.get("numericPrice")
+            for sub_key, sub_value in value.items():
+                flat[f"seoInformation_{sub_key}"] = _scalarize(sub_value)
+            continue
+        if key == "sellerInfo" and isinstance(value, dict):
+            flat["sellerAlias"] = value.get("alias")
+            flat["sellerLocationName"] = value.get("locationName")
+            flat["sellerMemberSince"] = value.get("memberSince")
+            continue
+        if key == "primaryCategory" and isinstance(value, dict):
+            flat["category"] = value.get("label") or value.get("categoryID")
+            flat["categoryKey"] = value.get("categoryID")
+            continue
+        if key == "postcodeInformation" and isinstance(value, dict):
+            flat["postcode"] = value.get("postcode")
+            flat["locationName"] = value.get("locationName")
+            canton = value.get("canton") or {}
+            flat["canton"] = canton.get("name")
+            flat["cantonKey"] = canton.get("shortName")
+            continue
+        if key == "thumbnail" and isinstance(value, dict):
+            flat["thumbnailURL"] = (value.get("normalRendition") or {}).get("src")
+            continue
+        if key == "images" and isinstance(value, list):
+            flat["images"] = "; ".join(
+                (img.get("rendition") or {}).get("src", "") for img in value if isinstance(img, dict)
+            )
+            continue
+        if key == "properties" and isinstance(value, list):
+            flat["properties"] = "; ".join(f"{p.get('label')}: {p.get('text')}" for p in value if isinstance(p, dict))
+            continue
+        if isinstance(value, dict):
+            for sub_key, sub_value in value.items():
+                flat[f"{key}_{sub_key}"] = _scalarize(sub_value)
+            continue
+        flat[key] = _scalarize(value)
+    # search_listings()/visit_all_listings() already embed a lang-correct
+    # "url" on the raw item; only fall back to computing a default one here
+    # for listings flattened without going through those (e.g. tests).
+    flat.setdefault("url", listing_url(item, lang))
+    return flat
+
+
+def order_fieldnames(all_keys: Iterable[str]) -> list[str]:
+    ordered = [f for f in PRIORITY_FIELDS if f in all_keys]
+    remaining = sorted(k for k in all_keys if k not in ordered)
+    return ordered + remaining
+
+
+def save_csv(rows: list[dict[str, Any]], path: str) -> None:
+    if not rows:
+        logger.warning("no rows to write")
+        return
+    all_keys: set[str] = set()
+    for row in rows:
+        all_keys.update(row.keys())
+    fieldnames = order_fieldnames(all_keys)
+    with open(path, "w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=fieldnames, restval="")
+        writer.writeheader()
+        writer.writerows(rows)
+
+
+def save_json(rows: list[dict[str, Any]], path: str) -> None:
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(rows, f, ensure_ascii=False, indent=2)
+
+
+@dataclass
+class ScrapeResult:
+    """Everything a scrape() call produced, ready to use in-memory or save to disk."""
+
+    query: str
+    total_elements: int
+    listings: list[dict[str, Any]] = field(default_factory=list)  # raw API objects (summary or full detail shape)
+    rows: list[dict[str, Any]] = field(default_factory=list)  # flattened dicts, one per listing, CSV-ready
+    lang: str = "de"  # locale that was scraped, e.g. "de"
+
+    def to_csv(self, path: str) -> None:
+        save_csv(self.rows, path)
+
+    def to_json(self, path: str) -> None:
+        save_json(self.listings, path)
+
+
+def scrape(
+    query: str,
+    *,
+    lang: str = "de",
+    detail: bool = True,
+    sort: str = "timestamp",
+    max_results: int | None = None,
+    delay: float = 1.0,
+    verbose: bool = True,
+    client: "TuttiClient | None" = None,
+) -> ScrapeResult:
+    """Search tutti.ch for `query` and return the results in memory.
+
+    This is the library entry point: it does the same work as the CLI but
+    returns a ScrapeResult instead of writing files. The CLI (main(), below)
+    is a thin wrapper around this function.
+
+    Args:
+        query: Free-text search phrase, e.g. "velo" or "Tesla Roadster".
+        lang: Listing/URL locale - "de" (default), "fr", or "it".
+        detail: If True (default), visit every listing's detail operation
+            one by one to extract every field tutti.ch returns for it
+            (coordinates, full-res images, attributes, richer seller info).
+            If False, keep only the summary fields from the search results
+            (much faster).
+        sort: Sort order tutti.ch searches with - "timestamp" (default),
+            "price", or "relevance".
+        max_results: Stop after this many unique listings, if given.
+        delay: Seconds to wait between requests.
+        verbose: If True, emit progress via the "tutti_scraper" logger at
+            INFO level.
+        client: Optional TuttiClient to reuse (e.g. across repeated calls).
+            A new one is created (using `lang` and `delay`) if not given -
+            if you do pass one, make sure its `lang` matches this `lang`
+            argument, since they aren't cross-checked.
+
+    Returns:
+        A ScrapeResult with `.listings` (raw API objects, each including a
+        "url" pointing at the original ad) and `.rows` (flattened dicts, one
+        per listing, sorted by price).
+    """
+    client = client or TuttiClient(lang=lang, delay=delay)
+
+    if verbose:
+        logger.info("Searching tutti.ch for %r ...", query)
+    nodes = search_listings(client, query, sort=sort.upper(), max_results=max_results, verbose=verbose)
+    total_elements = len(nodes)
+    for node in nodes:
+        node["url"] = listing_url(node, lang)
+
+    if detail:
+        if verbose:
+            logger.info("Visiting each of %d listings one by one to extract full details ...", len(nodes))
+        listings = visit_all_listings(client, nodes, verbose=verbose)
+    else:
+        listings = nodes
+
+    rows = [flatten_listing(item, lang) for item in listings]
+    rows.sort(key=lambda r: (r.get("price") in (None, ""), r.get("price")))
+
+    return ScrapeResult(
+        query=query,
+        total_elements=total_elements,
+        listings=listings,
+        rows=rows,
+        lang=lang,
+    )
+
+
+def _slugify(text: str) -> str:
+    slug = re.sub(r"[^a-z0-9]+", "-", text.strip().lower()).strip("-")
+    return slug or "listings"
+
+
+def build_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description="Scrape tutti.ch listings for a given search phrase.")
+    parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    parser.add_argument("query", help="Search phrase, e.g. 'velo' or 'Tesla Roadster'")
+    parser.add_argument("--lang", default="de", choices=["de", "fr", "it"], help="Listing/URL locale (default: de)")
+    parser.add_argument(
+        "--out",
+        default=None,
+        help="Output file base name (without extension). Defaults to a slug of the search phrase.",
+    )
+    parser.add_argument(
+        "--no-detail",
+        action="store_true",
+        help="Skip visiting each listing's detail operation; keep only the summary "
+        "fields from the search results (faster, fewer fields).",
+    )
+    parser.add_argument(
+        "--sort",
+        default="timestamp",
+        choices=["timestamp", "price", "relevance"],
+        help="Sort order tutti.ch searches with (default: timestamp)",
+    )
+    parser.add_argument("--max", type=int, default=None, help="Stop after N listings.")
+    parser.add_argument("--delay", type=float, default=1.0, help="Delay in seconds between requests.")
+    verbosity = parser.add_mutually_exclusive_group()
+    verbosity.add_argument(
+        "-v", "--verbose", action="store_true", help="Show debug-level detail, including every HTTP request made."
+    )
+    verbosity.add_argument(
+        "-q", "--quiet", action="store_true", help="Suppress progress output; only warnings/errors are shown."
+    )
+    return parser
+
+
+def _configure_cli_logging(*, verbose: bool, quiet: bool) -> None:
+    """Set up console logging for CLI use: progress (INFO, or DEBUG with -v)
+    goes to stdout, warnings/errors (-q still shows these) go to stderr.
+    Only main() calls this - plain library use of scrape() never touches
+    logging config, since that would be rude to whatever application
+    imported it."""
+    level = logging.DEBUG if verbose else logging.WARNING if quiet else logging.INFO
+    plain = logging.Formatter("%(message)s")
+
+    stdout_handler = logging.StreamHandler(sys.stdout)
+    stdout_handler.setLevel(level)
+    stdout_handler.addFilter(lambda record: record.levelno < logging.WARNING)
+    stdout_handler.setFormatter(plain)
+
+    stderr_handler = logging.StreamHandler(sys.stderr)
+    stderr_handler.setLevel(logging.WARNING)
+    stderr_handler.setFormatter(plain)
+
+    logger.handlers.clear()
+    logger.addHandler(stdout_handler)
+    logger.addHandler(stderr_handler)
+    logger.setLevel(level)
+    logger.propagate = False
+
+
+def main(argv: list[str] | None = None) -> int:
+    """CLI entry point. Parses argv (defaults to sys.argv[1:]), scrapes, and
+    writes CSV + JSON files. Returns 0 on success; lets exceptions propagate
+    (see run_cli() for the error-handling / exit-code wrapper used by the
+    __main__ guard below)."""
+    parser = build_arg_parser()
+    args = parser.parse_args(argv)
+    _configure_cli_logging(verbose=args.verbose, quiet=args.quiet)
+
+    result = scrape(
+        args.query,
+        lang=args.lang,
+        detail=not args.no_detail,
+        sort=args.sort,
+        max_results=args.max,
+        delay=args.delay,
+        verbose=True,
+    )
+
+    out_base = args.out or _slugify(args.query)
+    csv_path = f"{out_base}.csv"
+    json_path = f"{out_base}.json"
+    result.to_csv(csv_path)
+    result.to_json(json_path)
+
+    logger.info("\nDone. %d unique listings found.", len(result.rows))
+    logger.info("  CSV:  %s", csv_path)
+    logger.info("  JSON: %s", json_path)
+    return 0
+
+
+def run_cli(argv: list[str] | None = None) -> int:
+    """Run main() and translate exceptions into (message, exit code) the way
+    the command line expects. Factored out from the __main__ guard so it can
+    be unit-tested directly without spawning a subprocess."""
+    try:
+        return main(argv) or 0
+    except TuttiError as exc:
+        logger.error("Error talking to tutti.ch: %s", exc)
+        return 1
+    except requests.RequestException as exc:
+        logger.error("Network error talking to tutti.ch: %s", exc)
+        return 1
+    except KeyboardInterrupt:
+        logger.error("\nInterrupted.")
+        return 130
+
+
+if __name__ == "__main__":  # pragma: no cover - exercised via subprocess in test_e2e.py
+    sys.exit(run_cli())