hdx-python-country 4.2.9__tar.gz → 4.3.1__tar.gz

{hdx_python_country-4.2.9 → hdx_python_country-4.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hdx-python-country
-Version: 4.2.9
+Version: 4.3.1
 Summary: HDX Python country code and exchange rate (fx) utilities
 Project-URL: Homepage, https://github.com/OCHA-DAP/hdx-python-country
 Project-URL: Repository, https://github.com/OCHA-DAP/hdx-python-country

hdx_python_country-4.3.1/docs/decisions/0001-wfp-databridges-fetched-as-csv.md

@@ -0,0 +1,37 @@
+# 0001: Fetch WFP DataBridges data as CSV rather than paged JSON
+
+## Status
+
+Accepted — 2026-11-03
+
+## Context
+
+JSON paging of `market_prices_price_monthly_get` is unstable. Confirmed live on NGA: each run
+returned about 1,690 rows twice and left out the same number, so datasets built from it missed
+about 1.4% of rows. The endpoint has no sort parameter that would make offset paging
+deterministic.
+
+`format="CSV"` returns every row once in a single response and ignores `page` (NGA: 121,669
+distinct rows in about 10s, compared with about 800s paged). The other `WFPAPI` endpoints accept
+CSV too.
+
+## Decision
+
+Every `WFPAPI` endpoint is called with `format="CSV"` through the generated
+`*_without_preload_content` methods, and the JSON paging code is removed.
+
+- CSV headers are the JSON keys with the first letter capitalised, and every value is a string.
+  `_csv_converters` builds the int, float, bool and datetime converters from each model's pydantic
+  annotations.
+- `_get_csv` raises `ApiException` on a non-200 status, as the JSON methods do, so `_with_retry`
+  still refreshes the token and retries.
+- Saved data is the CSV that WFP returns, one file per call.
+
+## Consequences
+
+- Blank text fields are `""` where JSON gave `None`.
+- Commodity `qualities` and `processing` are always `None`, since CSV omits them.
+- Datetimes lose milliseconds.
+- Data saved in the old JSON format no longer loads.
+- A whole country comes back in one response (SYR: 516,907 rows, 103 MB), which can hit the
+  gateway timeout (see 0002) and needs care with memory (see 0003).

hdx_python_country-4.3.1/docs/decisions/0002-split-price-date-range-on-504.md

@@ -0,0 +1,28 @@
+# 0002: Split the monthly prices date range on a gateway timeout
+
+## Status
+
+Accepted — 2026-11-03
+
+## Context
+
+A single CSV request for a large country can exceed the API gateway's timeout of about 30s. SYR's
+first full request got a 504 after 30s, and the second returned 200 in 35s. Relying on
+`_with_retry` is not acceptable, because the country run uses `wait=1800`, i.e. 30 minutes per
+retry.
+
+`start_date` and `end_date` are inclusive and split the data cleanly. On BTN, 1990–2019 plus
+2020–2030 returned exactly the whole set (1,836 + 6,171 = 8,007 rows).
+
+## Decision
+
+`_get_prices_csv_range` requests the whole range (1990-01-01 to 31 December five years ahead,
+since WFP has forecast prices). On a 504 it splits the range at the midpoint and requests each
+half, recursively. A 504 on a single day, or any other error, is raised. `_get_prices_csv` joins
+the pieces, keeping only the first header row.
+
+## Consequences
+
+- Most countries take one request. Each 504 costs about 30s before the split.
+- On 2026-11-04 the server was slower, and IDN's range was split at least three times.
+- Saved data is still one file per country, whatever the number of requests.

hdx_python_country-4.3.1/docs/decisions/0003-memory-efficient-csv-parsing.md

@@ -0,0 +1,46 @@
+# 0003: Parse WFP CSV with shared values and fields sets
+
+## Status
+
+Accepted — 2026-11-03
+
+## Context
+
+Fetching and parsing IDN's prices (462,027 rows) peaked at 2.31 GB with the first CSV parser, and
+the country run was killed on Jenkins (exit code 137). Each row's
+`ViewExtendedMonthlyAggregatedPrice` took about 3.2 KB:
+
+| Part | Bytes |
+|---|---|
+| Object | 72 |
+| `__dict__` with 22 fields | 832 |
+| `__pydantic_fields_set__`, the 22 field names set on this row | 2,264 |
+| Empty `additional_properties` | 64 |
+
+Every row also held its own copies of repeated values such as market names, dates and units.
+`io.StringIO` copied the whole text into a buffer of 4 bytes per character.
+
+## Decision
+
+`_parse_csv`:
+
+- Splits lines lazily with `text.split("\n")` instead of `io.StringIO`, re-adding the `\n` to each
+  line so quoted fields containing line breaks parse the same.
+- Reuses one object per distinct value in each column.
+- Gives rows with the same fields one shared `__pydantic_fields_set__`. Pydantic only adds to it
+  when a field is assigned, and every field is already in it.
+
+## Consequences
+
+Measured on the saved IDN CSV:
+
+| | Peak memory | Parse time |
+|---|---|---|
+| `io.StringIO` | 2,310 MB | 9.5s |
+| Lazy split | 2,059 MB | |
+| Lazy split and reused values | 1,755 MB | |
+| Lazy split, reused values and shared fields set | 746 MB | 5.7s |
+
+- The parsed rows are identical to those from the `io.StringIO` parser.
+- Rows share value objects. They are immutable (str, int, float, datetime), so this is not visible
+  to callers.

hdx_python_country-4.3.1/docs/decisions/0004-pse-fetched-as-psw-and-psg.md

@@ -0,0 +1,21 @@
+# 0004: Fetch PSE as PSW and PSG
+
+## Status
+
+Accepted — 2024-10-31
+
+## Context
+
+WFP treats the State of Palestine as two areas, the West Bank (`PSW`) and the Gaza Strip (`PSG`).
+Asking WFP for `PSE` returns nothing. Checked live on 2026-11-04, the markets endpoint gave 0
+markets for `PSE`, 12 for `PSW` and 47 for `PSG`.
+
+## Decision
+
+`WFPAPI._countryiso3s` turns `PSE` into `["PSW", "PSG"]`, so every country-filtered endpoint makes
+one request per area and returns the combined rows.
+
+## Consequences
+
+- Callers use the ISO 3166 code `PSE` and get both areas' data.
+- Saved files are per area, e.g. `MarketPrices_PriceMonthly_PSW.csv`.

hdx_python_country-4.3.1/docs/decisions/0005-wfp-dates-assumed-utc.md

@@ -0,0 +1,21 @@
+# 0005: Treat WFP dates without a timezone as UTC
+
+## Status
+
+Accepted — 2026-07-20
+
+## Context
+
+WFP's dates have no timezone. Before the move to the DataBridges client (#88), they were parsed
+with `parse_date`, which returns UTC datetimes. `data_bridges_client` returns naive datetimes
+instead.
+
+## Decision
+
+`WFPExchangeRates.get_currency_historic_rates` sets UTC on any naive quotation date before
+converting it to a timestamp. hdx-scraper-wfp-foodprices does the same for price dates.
+
+## Consequences
+
+- Historic rate timestamps are unchanged from the `parse_date` behaviour.
+- If WFP starts returning dates with a timezone, they are used as given.

hdx_python_country-4.3.1/docs/decisions/README.md

@@ -0,0 +1,35 @@
+# docs/decisions/
+
+Decision records: one immutable file per non-trivial design/architecture
+decision — using Michael Nygard's minimal headers (Title, Status, Context,
+Decision, Consequences), plus a `Date:` line under Status. Distill the
+decision itself, not the full planning narrative behind it.
+
+A reversed decision gets a new record whose Status reads
+`Superseded by 00NN`; the old file stays as-is. Routine notes belong in
+`CLAUDE.md` instead of here; PR-description-only detail doesn't need a
+record at all.
+
+Numbered `NNNN-title.md`, sequential, never reused.
+
+## Format
+
+```markdown
+# NNNN: <Title>
+
+## Status
+
+Accepted — YYYY-MM-DD
+
+## Context
+
+...
+
+## Decision
+
+...
+
+## Consequences
+
+...
+```

{hdx_python_country-4.2.9 → hdx_python_country-4.3.1}/src/hdx/location/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '4.2.9'
-__version_tuple__ = version_tuple = (4, 2, 9)
+__version__ = version = '4.3.1'
+__version_tuple__ = version_tuple = (4, 3, 1)
 
 __commit_id__ = commit_id = None

{hdx_python_country-4.2.9 → hdx_python_country-4.3.1}/src/hdx/location/wfp_api.py

@@ -1,5 +1,4 @@
 import csv
-import io
 import logging
 import types
 from collections.abc import Callable
@@ -115,7 +114,9 @@ def _parse_csv(text: str, model_cls: type) -> list:
     """Parse WFP's CSV output into data_bridges_client model objects. CSV
     headers are the JSON keys with the first letter capitalised and every
     value is a string. Blank strings are kept as "" (as JSON returns them)
-    but blank values of non-string fields become None.
+    but blank values of non-string fields become None. Values and pydantic
+    fields sets are shared between rows to limit memory use (see
+    docs/decisions/0003).
 
     Args:
         text: CSV text
@@ -125,30 +126,39 @@ def _parse_csv(text: str, model_cls: type) -> list:
         List of model_cls objects
     """
     converters = _csv_converters(model_cls)
-    reader = csv.reader(io.StringIO(text))
+    reader = csv.reader(f"{line}\n" for line in text.split("\n"))
     keys = [header[0].lower() + header[1:] for header in next(reader, [])]
-    columns = [(key, converters.get(key)) for key in keys]
+    columns = [(key, converters.get(key), {}) for key in keys]
+    fields_sets = {}
     items = []
     for row in reader:
         if not row:
             continue
         obj = {}
-        for (key, converter), value in zip(columns, row):
-            if converter:
-                value = converter(value) if value else None
-            obj[key] = value
-        items.append(model_cls.from_dict(obj))
+        for (key, converter, values), value in zip(columns, row):
+            try:
+                obj[key] = values[value]
+            except KeyError:
+                if converter:
+                    converted = converter(value) if value else None
+                else:
+                    converted = value
+                values[value] = converted
+                obj[key] = converted
+        item = model_cls.from_dict(obj)
+        fields_set = frozenset(item.__pydantic_fields_set__)
+        fields_set = fields_sets.setdefault(fields_set, item.__pydantic_fields_set__)
+        object.__setattr__(item, "__pydantic_fields_set__", fields_set)
+        items.append(item)
     return items
 
 
 class WFPAPI:
     """Light wrapper around WFP's data-bridges-client library. All data is
-    requested as CSV, which returns every row in one response, rather than
-    JSON, whose paging returns some rows twice and leaves others out. It
-    needs a retriever used only to decide whether to load from disk or hit
-    the network and whether to persist the network result (as the CSV
-    returned by WFP). Token refreshing is delegated to data_bridges_client's
-    own WfpApiToken.
+    requested as CSV (see docs/decisions/0001). It needs a retriever used
+    only to decide whether to load from disk or hit the network and whether
+    to persist the network result (as the CSV returned by WFP). Token
+    refreshing is delegated to data_bridges_client's own WfpApiToken.
 
     Args:
         retriever: Retrieve object for interacting with WFP API
@@ -263,7 +273,7 @@ class WFPAPI:
 
     @staticmethod
     def _countryiso3s(countryiso3: str | None) -> list:
-        if countryiso3 == "PSE":  # hack as PSE is treated by WFP as 2 areas
+        if countryiso3 == "PSE":  # see docs/decisions/0004
             return ["PSW", "PSG"]
         return [countryiso3]
 
@@ -405,7 +415,7 @@ class WFPAPI:
     ) -> list[str]:
         """Get monthly aggregated market prices between two dates (inclusive)
         as CSV. If the API gateway times out (504), the date range is split in
-        two and each half is requested.
+        two and each half is requested (see docs/decisions/0002).
 
         Args:
             api_method: Bound market_prices_price_monthly_get_without_preload_content

{hdx_python_country-4.2.9 → hdx_python_country-4.3.1}/src/hdx/location/wfp_exchangerates.py

@@ -59,9 +59,7 @@ class WFPExchangeRates:
                 continue
             date = quote.var_date
             if date.tzinfo is None:
-                # data_bridges_client parses WFP's timezone-less date strings
-                # into naive datetimes; assume UTC to match previous
-                # parse_date-based behaviour
+                # See docs/decisions/0005
                 date = date.replace(tzinfo=UTC)
             timestamp = get_int_timestamp(date)
             historic_rates[timestamp] = quote.value