themeparks 3.2.0__tar.gz → 4.0.0__tar.gz

{themeparks-3.2.0 → themeparks-4.0.0}/CHANGELOG.md

@@ -1,5 +1,184 @@
 # Changelog
 
+## [4.0.0] - 2026-09-28
+
+**3.3.0 was yanked: incomplete CSV export and a resume defect.**
+
+A major version because **the CSV header changed**: fifteen columns were added and
+the order is now the schema's, so a reader that takes columns by position gets the
+wrong ones rather than an error. Read by name. The library API is backward
+compatible.
+
+Everything here came out of porting `themeparks-backfill` to the JavaScript SDK
+and then diffing the two outputs over the same park, and out of six reviews of the
+result. Two independent implementations reading one API disagree in exactly the
+places one of them is wrong. Magic Kingdom's full archive now comes back
+**byte for byte identical** from both SDKs: 94,223 rows, 41 columns, the only
+differences being today's row, which grows as the day elapses.
+
+### Fixed
+
+- **`themeparks-backfill "magic kingdom"` wrote the wrong park name into every
+  row.** A name that matched one park by substring returned the formatted display
+  label, so `parkName` read `Magic Kingdom Park  (Walt Disney World® Resort)` for
+  all ~94,000 rows, and the resolution echo printed the destination twice. Four
+  live names reached it.
+
+- **The CSV was missing ten of the thirty-six fields the API sends, on every row.**
+  `unknownMinutes`, the whole `inParkHours` block (the day's numbers limited to the
+  park's published hours -- usually the ones you want, since a ride "down" at 2am
+  is not down), `extremeWaits` (how many readings of 480+ minutes are folded into
+  the statistics, which is how you spot a feed error), and three of `singleRider`'s
+  five percentiles while `standby` carried all five. On a five-year Magic Kingdom
+  export, 72,200 of 94,223 rows were missing their in-park statistics. **The column
+  list is now derived from the model**, so it cannot drift again.
+
+- **Vendored models were stale, and pydantic drops what it does not declare**, so
+  those three fields were deleted at parse time for every caller of `days()`, not
+  just for the CSV. Models regenerated, and every model now keeps fields the schema
+  does not declare (`themeparks._models_base.ApiModel`, `extra="allow"`), so a
+  field the API adds tomorrow survives parsing and reaches `model_dump()` and the
+  NDJSON output before this SDK knows it exists. It does not reach the CSV, whose
+  columns come from the schema.
+
+- **A resumed download duplicated a day.** The checkpoint was the newest row
+  written; the page it came from covered further, because an entity that stopped
+  reporting has no rows for the tail days. A rerun re-fetched a day already in the
+  file and appended every row of it again, breaking the `(entityId, date)` key --
+  on the exit-75 path, which is the ordinary path for a long back fill. The
+  checkpoint is now the day the server's own `next` URL starts on.
+
+- **A failure on a resumed run deleted everything already downloaded.** `written
+  == 0` means "this process wrote nothing", not "the file is empty". The state file
+  survived pointing mid-archive, so the next run appended only the tail and
+  recorded `complete: true`. Same for a window that closes under a resumed run --
+  a key rotated out of a scheduler's environment, a lapsed subscription -- which
+  additionally exited 0, so the scheduler logged success, and became a permanent
+  trap.
+
+- **Resuming across versions, formats or SDKs corrupted the file.** One state file
+  served both formats, so `ndjson` then `csv` then `ndjson` doubled every row in
+  the first file; and the state carried nothing about the header, so 3.3.0's
+  19-column file resumed under this build appended 41-field rows beneath it. The
+  state file is now `<parkId>.<format>.backfill-state.json` and records the SDK,
+  its version, a state version and a fingerprint of the exact header. Anything that
+  does not match is refused with a message saying why, never resumed.
+
+- **A network failure or timeout now exits 75, not 1**, so a scheduler retries
+  rather than alerting; anything the API actively rejected still exits 1. The
+  JavaScript SDK had these the other way round.
+
+- **A carriage return in an entity name was written unquoted on Python 3.9 and
+  3.10**, so one row parsed as two with every later column shifted. The `csv`
+  module's QUOTE_MINIMAL only quotes characters that appear in the line terminator,
+  and this command sets LF; 3.11 changed the module to always quote CR and LF, so
+  the defect was invisible on a modern interpreter and live on two supported ones.
+  The CSV writer now does its own minimal quoting, which also makes the output
+  byte-identical across Python versions rather than only within one.
+
+- **UTC timestamps are written `Z`, not `+00:00`**, and CSV line endings are LF.
+  Between them these accounted for 39,201 differing lines against the JavaScript
+  SDK's output for no difference in meaning.
+
+- **One park's failure no longer abandons the rest of a destination.** Every park
+  is tried, what failed is named at the end, and the exit code still says something
+  went wrong. A spent budget still stops everything, deliberately.
+
+- **A failed park no longer leaves a 0-byte file** that reads as "this park has no
+  history", including when the budget runs out before the first page.
+
+- **A network failure, a full disk or Ctrl-C is a sentence, not a traceback.**
+
+- **The user agent named neither version.** It was the literal
+  `themeparks-backfill/1`, and it replaced the SDK's own, so a support question had
+  no version to work from at either end.
+
+- **`--list <text>` reported the wrong total**, printing "all 1 parks" for a
+  destination with six -- on the one line whose whole job is that number.
+
+- **An ambiguous name listed the wrong candidates**, widening to substrings and
+  offering a third park that was not what was typed. It now lists the ids of the
+  parks that actually match, sorted by name.
+
+- **A collection of nested models would have produced phantom columns** and then an
+  `AttributeError` on the first row. Duplicate column names are now impossible at
+  import rather than a wrong number under a right-looking header.
+
+- **The NDJSON identity columns could be overwritten by the row** once models kept
+  undeclared fields.
+
+### Added
+
+- **The CSV carries a UTF-8 BOM**, so Excel on Windows stops rendering
+  `Walt Disney World® Resort` as mojibake.
+- **A cell a spreadsheet would execute is prefixed with an apostrophe** (`=`, `+`,
+  `-`, `@`, tab, CR). Numeric cells are left alone, so a negative number stays a
+  number.
+- **`on_page` on `days()` and `days_with_entities()`**, called once every row of a
+  page has been yielded, with a `HistoryPage` (`start`, `end`, `next_url`). The page
+  boundary is the server's own answer to "where do I carry on", and the rows cannot
+  tell you.
+- **`--version`.**
+- `EntityRef` and `HistoryPage` are exported from the package.
+- `tests/fixtures/csv_contract.json`, an identical copy of which lives in the
+  JavaScript SDK. Both suites assert their column list against it, because this is
+  one command with two implementations and a customer using both should get one
+  file format.
+
+### Changed
+
+- The `themeparks-backfill` entry point is `themeparks.backfill:cli`, which adds
+  the top-level error handling. `main()` is unchanged for anyone calling it.
+- Model equality and `model_json_schema()` reflect `extra="allow"`: two responses
+  differing only in an undeclared field now compare unequal, and dumps may contain
+  fields the schema does not list.
+
+## [3.3.0] - 2026-09-28
+
+### Added
+
+- **`themeparks-backfill`: the archive download as a command.** It was an example
+  to copy off GitHub. The first paying customer followed that link and had to
+  work out that the library needed installing, then what the arguments were,
+  then read a traceback. Now:
+
+  ```bash
+  pip install themeparks
+  themeparks-backfill "Disneyland Park"
+  ```
+
+  - Takes a park or a **destination**, by name or id. A destination back fills
+    every park in it, one file each. `"Walt Disney World Resort"` is the handle
+    people actually have; four park uuids is not.
+  - `--list [text]` prints destinations with their parks underneath, and **needs
+    no key**, so you can find your park before deciding whether to pay.
+  - Refuses to guess between two matches. Two parks are named exactly
+    "Disneyland Park" (Anaheim and Paris), so the candidate list names the
+    destination as well.
+  - **Runs without a key**, reading the 7 days anonymous access allows, and says
+    what a key would add. It used to refuse to start with a message that
+    mentioned anonymous access in the same breath.
+  - NDJSON by default, `--format csv` for one wide row per entity per day.
+  - Checkpoints against the hourly history budget and exits 75 (`EX_TEMPFAIL`),
+    so a cron or timer retries rather than alerting. Re-running continues.
+
+  `python -m themeparks.backfill` is the same thing. `examples/backfill.py`
+  remains as a shim so existing links keep working.
+
+### Fixed
+
+- **The history window recovery now actually works.** 3.2.0's `examples/backfill.py`
+  read `earliestAllowedDate` from the top level of the 403 body; the API nests it
+  under `error`. So the recovery shipped doing nothing and a Pro customer still
+  got a traceback on their first request. The tests passed because the fixture was
+  built from the formatted text in a traceback rather than a real response, so the
+  code and the test were wrong together. The fixture is now captured from
+  production and a test fails if anyone flattens it.
+
+  The underlying gap is in the API, not the client: `/history/coverage` reports
+  where the archive starts and where your window ends, and nothing about where
+  your window begins. Until it does, the 403 is the only place that date exists.
+
 ## [3.2.0] - 2026-09-26
 
 ### Added

{themeparks-3.2.0 → themeparks-4.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: themeparks
-Version: 3.2.0
+Version: 4.0.0
 Summary: Official SDK for the ThemeParks.wiki API
 Project-URL: Homepage, https://api.themeparks.wiki
 Project-URL: Source, https://github.com/ThemeParks/ThemeParks_Python
@@ -350,9 +350,25 @@ except BudgetExhaustedError as exc:
     print(f"resume in {exc.retry_after:.0f}s")
 ```
 
-A complete backfill script with resume and CSV output is in
-[`examples/backfill.py`](examples/backfill.py); it pulls Disneyland Resort's
-whole daily archive, 98,452 rows, in one run.
+### Or skip the code: there is a command
+
+Installing the library installs `themeparks-backfill`, which does all of the
+above and stops before the walls:
+
+```bash
+themeparks-backfill "Disneyland Park"           # a park, by name or id
+themeparks-backfill "Walt Disney World Resort"  # a destination: every park in it
+themeparks-backfill --list disney               # find an id. Needs no key.
+```
+
+It reads how far back your own key may ask and starts there, writes NDJSON or
+`--format csv`, names every row with the park and the entity, records what it
+has done so re-running never duplicates a file, and exits 75 when the hourly
+history budget runs out so a scheduler retries rather than alerts.
+
+`python -m themeparks.backfill` is the same thing, which is the one to use if
+`pip install --user` put the script somewhere off your PATH. `themeparks-backfill
+--help` has the rest.
 
 ## Low-level escape hatch
 

{themeparks-3.2.0 → themeparks-4.0.0}/README.md

@@ -311,9 +311,25 @@ except BudgetExhaustedError as exc:
     print(f"resume in {exc.retry_after:.0f}s")
 ```
 
-A complete backfill script with resume and CSV output is in
-[`examples/backfill.py`](examples/backfill.py); it pulls Disneyland Resort's
-whole daily archive, 98,452 rows, in one run.
+### Or skip the code: there is a command
+
+Installing the library installs `themeparks-backfill`, which does all of the
+above and stops before the walls:
+
+```bash
+themeparks-backfill "Disneyland Park"           # a park, by name or id
+themeparks-backfill "Walt Disney World Resort"  # a destination: every park in it
+themeparks-backfill --list disney               # find an id. Needs no key.
+```
+
+It reads how far back your own key may ask and starts there, writes NDJSON or
+`--format csv`, names every row with the park and the entity, records what it
+has done so re-running never duplicates a file, and exits 75 when the hourly
+history budget runs out so a scheduler retries rather than alerts.
+
+`python -m themeparks.backfill` is the same thing, which is the one to use if
+`pip install --user` put the script somewhere off your PATH. `themeparks-backfill
+--help` has the rest.
 
 ## Low-level escape hatch
 

{themeparks-3.2.0 → themeparks-4.0.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "themeparks"
-version = "3.2.0"
+version = "4.0.0"
 description = "Official SDK for the ThemeParks.wiki API"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -30,6 +30,12 @@ dependencies = [
   "eval-type-backport>=0.2; python_version < '3.10'",
 ]
 
+[project.scripts]
+# The archive backfill, as a command rather than a file to copy off GitHub.
+# `pip install themeparks` then `themeparks-backfill "Disneyland Park"` is the
+# whole path from nothing to a file of history.
+themeparks-backfill = "themeparks.backfill:cli"
+
 [project.urls]
 Homepage = "https://api.themeparks.wiki"
 Source = "https://github.com/ThemeParks/ThemeParks_Python"

themeparks-4.0.0/tests/fixtures/README.md

@@ -0,0 +1,50 @@
+# Fixtures captured from the live API
+
+Not hand-written. Each file here was taken from a real response, and the header
+below says which endpoint and when. That matters because of a specific failure
+this project keeps repeating.
+
+## Why this directory exists
+
+On 2026-09-28 a 403-recovery fix shipped doing nothing. Its tests passed because
+the fixture was built from the *formatted text of a traceback* rather than from a
+response body, so the code and the test were wrong together and agreed with each
+other. The memory note for that pattern is `self-confirming-harness`, and it was
+its fifth occurrence in one day.
+
+A fixture the author of the code also invented proves only that the two are
+consistent. A fixture taken from the real thing can disagree.
+
+## destinations_slice.json
+
+`GET https://api.themeparks.wiki/v1/destinations`, captured 2026-09-28, trimmed
+to 9 destinations and 20 parks and otherwise verbatim — every id, name and slug
+is exactly what the API returned.
+
+It is trimmed to keep, deliberately, every case that broke name resolution:
+
+| Case | Why it is here |
+|---|---|
+| `Walt Disney World® Resort` | U+00AE. The command's own documented example, `"Walt Disney World Resort"`, did not match it. |
+| `LEGOLAND® Korea` | the same, on a destination whose park shares the name |
+| `Walibi Rhône-Alpes` | U+00F4, so a normaliser must decompose accents |
+| `Knott's Berry Farm` + `Knott’s Soak City` | ASCII `'` and U+2019 **in one destination**. Nobody types the curly one. |
+| `Disneyland Park` ×2 | Anaheim and Paris, identical park names. The reason the candidate list names the destination. |
+| `Hurricane Harbor` + `Hurricane Harbor Chicago` | an exact match with a substring rival: the shape that silently downloaded the wrong park |
+| `Cedar Point` | destination name == park name, with a second park, so "exact destination beats exact park" is observable |
+
+**Do not edit these by hand.** Re-capture them. If a name upstream has drifted,
+that is a real change and the test should notice.
+
+## mk_park_daily_page1.json / mk_park_daily_page2.json
+
+Two consecutive pages of one real request, captured 2026-09-28:
+`GET /entity/75ea578a-adc8-4116-a54d-dccb60765ef9/history/daily?from=2026-08-01&to=2026-09-20`
+then its `next` followed verbatim. Trimmed to three entities (an attraction, a
+show, a restaurant); `range`, `next` and every row are the server's.
+
+They are the oracle for resumable paging. Page one covers through 2026-08-31 and
+the server says carry on at 2026-09-01, but two of its three entities have no
+rows after 2026-08-30 -- so a checkpoint taken from the newest ROW rewinds and
+re-downloads days already written. The same files are in the JavaScript SDK, so
+both ports are tested against identical bytes.

{themeparks-3.2.0 → themeparks-4.0.0}/themeparks/__init__.py

@@ -1,7 +1,12 @@
 from themeparks._cache import Cache, CacheConfig, InMemoryLRUCache
 from themeparks._client import AsyncThemeParks, ThemeParks
 from themeparks._ergonomic.dates import parse_api_datetime
-from themeparks._ergonomic.history import BudgetExhaustedError, HistorySpan
+from themeparks._ergonomic.history import (
+    BudgetExhaustedError,
+    EntityRef,
+    HistoryPage,
+    HistorySpan,
+)
 from themeparks._ergonomic.live import current_wait_time, iter_queues
 from themeparks._errors import (
     APIError,
@@ -17,6 +22,8 @@ __all__ = [
     "APIError",
     "AsyncThemeParks",
     "BudgetExhaustedError",
+    "EntityRef",
+    "HistoryPage",
     "HistorySpan",
     "RateLimit",
     "RateLimits",

{themeparks-3.2.0 → themeparks-4.0.0}/themeparks/_ergonomic/history.py

@@ -20,7 +20,7 @@ the cheap path here without having to know the expensive one exists.
 
 from __future__ import annotations
 
-from collections.abc import AsyncIterator, Iterator
+from collections.abc import AsyncIterator, Callable, Iterator
 from datetime import date as _date
 from typing import Any, NamedTuple, Union
 
@@ -111,17 +111,87 @@ def _reraise_if_too_long(exc: RateLimitError, max_wait: float) -> None:
     raise exc
 
 
-def _daily_rows(envelope: DailyEnvelope) -> Iterator[tuple[str, HistoryDailyRow]]:
-    """Yield (entity id, row). A park envelope carries many entities; an entity
-    envelope carries its own rows, so both flatten to the same stream."""
+class HistoryPage(NamedTuple):
+    """One page of daily history, as the server described it.
+
+    `start` and `end` are the park-local days the page ACTUALLY covered, which is
+    not the range you asked for: a park daily call serves at most 31 days, so a
+    50-day request comes back as 31 days plus a `next`. `next_url` is the URL of
+    the following page, or None on the last one.
+
+    This exists for resumable downloads. A checkpoint taken from the ROWS is
+    wrong in both directions: the newest row's date can be earlier than the page
+    covered, because an entity that stopped reporting has no rows for the tail
+    days, so resuming there re-fetches days already written and duplicates them;
+    and a half-written page is indistinguishable from a finished one. The page
+    boundary is the server's own answer to "where do I carry on", so it is the
+    only safe checkpoint.
+    """
+
+    start: str
+    end: str
+    next_url: str | None
+
+
+class EntityRef(NamedTuple):
+    """Who a history row belongs to, AS THE HISTORY RESPONSE REPORTS IT.
+
+    The name matters and the source of it matters more. A park's current
+    `/children` list gives today's name, which is the wrong label for a row
+    recorded years ago: rides are renamed, and stamping today's name on old data
+    quietly rewrites history. The history envelope carries its own `name` and
+    `entityType` per entity, and that is the name to use.
+    """
+
+    id: str
+    name: str
+    entity_type: str
+
+
+def _ref(entity: Any) -> EntityRef:
+    kind = getattr(entity, "entityType", None)
+    # The generated models use an enum, and str(EntityType.SHOW) is
+    # "EntityType.SHOW". `.value` is what the API sends.
+    inner = getattr(kind, "value", kind)
+    return EntityRef(
+        entity.id, getattr(entity, "name", "") or "", "" if inner is None else str(inner)
+    )
+
+
+def _page_of(envelope: DailyEnvelope) -> HistoryPage:
+    """The page an envelope represents, for :class:`HistoryPage`'s callers."""
+    rng = getattr(envelope, "range", None)
+    nxt = getattr(envelope, "next", None)
+    return HistoryPage(
+        getattr(rng, "from_", "") or "",
+        getattr(rng, "to", "") or "",
+        nxt or None,
+    )
+
+
+def _daily_entity_rows(envelope: DailyEnvelope) -> Iterator[tuple[EntityRef, HistoryDailyRow]]:
+    """Yield (entity ref, row), keeping the name the response gave.
+
+    `_daily_rows` below is the same walk with the ref flattened to its id, kept
+    because `days()` has yielded `(id, row)` since 3.0 and that shape is public.
+    """
     entities = getattr(envelope, "entities", None)
     if entities is not None:
         for entity in entities:
+            ref = _ref(entity)
             for row in entity.days or []:
-                yield (entity.id, row)
+                yield (ref, row)
         return
+    ref = _ref(envelope)
     for row in getattr(envelope, "days", None) or []:
-        yield (envelope.id, row)
+        yield (ref, row)
+
+
+def _daily_rows(envelope: DailyEnvelope) -> Iterator[tuple[str, HistoryDailyRow]]:
+    """Yield (entity id, row). A park envelope carries many entities; an entity
+    envelope carries its own rows, so both flatten to the same stream."""
+    for ref, row in _daily_entity_rows(envelope):
+        yield (ref.id, row)
 
 
 def _raw_rows(envelope: RawEnvelope) -> Iterator[tuple[str, HistoryRow]]:
@@ -154,21 +224,58 @@ class HistoryApi:
         """
         return _span(self.coverage())
 
+    def days_with_entities(
+        self,
+        start: str | _date | None = None,
+        end: str | _date | None = None,
+        *,
+        max_wait: float = DEFAULT_MAX_WAIT_SECONDS,
+        on_page: Callable[[HistoryPage], None] | None = None,
+    ) -> Iterator[tuple[EntityRef, HistoryDailyRow]]:
+        """`days()`, but each row arrives with the entity's name and type.
+
+        Use this when you are writing history to a file. The name comes from the
+        history response itself, so it is the label that response gives for those
+        rows rather than the park's current `/children` list -- rides get renamed,
+        and today's name on a row from three years ago is a quiet rewrite of the
+        record.
+
+        It also saves a request: the name is already in the payload, so nothing
+        needs to ask what an id refers to.
+
+        `on_page` is called once every row of a page has been yielded, with a
+        :class:`HistoryPage`. Checkpoint on that, never on the last row you saw.
+        """
+        envelope: DailyEnvelope | None = self._first_daily(start, end, max_wait)
+        while envelope is not None:
+            yield from _daily_entity_rows(envelope)
+            # AFTER the rows, never before: a caller checkpointing on this has to
+            # be able to trust that everything the page held is already written.
+            if on_page is not None:
+                on_page(_page_of(envelope))
+            envelope = self._next_daily(envelope, max_wait)
+
     def days(
         self,
         start: str | _date | None = None,
         end: str | _date | None = None,
         *,
         max_wait: float = DEFAULT_MAX_WAIT_SECONDS,
+        on_page: Callable[[HistoryPage], None] | None = None,
     ) -> Iterator[tuple[str, HistoryDailyRow]]:
         """One summary row per park-local day, as (entity id, row).
 
         Pages automatically. Given a park id this uses the park call, which
         answers every entity in the park in one request.
+
+        `days_with_entities()` is the same stream with the entity's name and type
+        attached; this shape is kept because it is public API from 3.0.
         """
         envelope: DailyEnvelope | None = self._first_daily(start, end, max_wait)
         while envelope is not None:
             yield from _daily_rows(envelope)
+            if on_page is not None:
+                on_page(_page_of(envelope))
             envelope = self._next_daily(envelope, max_wait)
 
     def _first_daily(
@@ -237,6 +344,7 @@ class AsyncHistoryApi:
         end: str | _date | None = None,
         *,
         max_wait: float = DEFAULT_MAX_WAIT_SECONDS,
+        on_page: Callable[[HistoryPage], None] | None = None,
     ) -> AsyncIterator[tuple[str, HistoryDailyRow]]:
         try:
             envelope: DailyEnvelope | None = await self._raw.get_entity_history_daily(
@@ -248,6 +356,8 @@ class AsyncHistoryApi:
         while envelope is not None:
             for pair in _daily_rows(envelope):
                 yield pair
+            if on_page is not None:
+                on_page(_page_of(envelope))
             nxt = getattr(envelope, "next", None)
             if not nxt:
                 return