arkparser 0.5.3__tar.gz → 0.6.0__tar.gz

{arkparser-0.5.3 → arkparser-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: arkparser
-Version: 0.5.3
+Version: 0.6.0
 Summary: Pure Python parser for ARK: Survival Evolved and ARK: Survival Ascended save files
 Author: Vertyco
 License-Expression: MIT
@@ -58,6 +58,7 @@ A pure-Python library for parsing ARK: Survival Evolved (ASE) and ARK: Survival
 - **Dual format**: automatic ASE (v5-12) / ASA (v13-14+, SQLite) detection
 - **Legacy-parity export**: drop-in JSON output matching `ASVExport.exe` schema, plus parser-only extras under descriptive snake_case keys (no namespace prefix; never overloading a legacy key)
 - **Fast**: pure-Python `BinaryReader` (`int.from_bytes` + `struct.Struct` unpackers, slots-based dataclasses), a 30 MB ASE save (65k objects) loads in ~3s on CPython 3.14
+- **Memory-bounded**: streaming JSON export plus an opt-in lazy parse mode (`lazy_properties=True`, ASE and ASA) that materializes property blocks on demand and evicts them as records stream to disk; a 1.8 GB / 1.79M-object busy-PvE ASE save exports in 2.5 GB peak RSS instead of 7.1 GB, and a 217k-object ASA save in 307 MB instead of 853 MB (and slightly faster)
 
 ## Installation
 
@@ -132,6 +133,38 @@ print(f"Parse errors: {save.parse_error_count}")
 print(f"Is ASA: {save.is_asa}")
 ```
 
+### Low-memory parsing (large saves)
+
+By default every object's property block is parsed up front, which on a busy
+multi-GB PvE save means a multi-GB resident object graph. `lazy_properties=True`
+parses object headers eagerly but defers each property block until something
+reads it, then the export drivers evict it again once the record has been
+written:
+
+```python
+from arkparser import WorldSave, export_to_files
+from arkparser.common import get_map_config
+
+save = WorldSave.load("path/to/Fjordur.ark", lazy_properties=True)       # ASE
+save = WorldSave.load("path/to/TheIsland_WP.ark", lazy_properties=True)  # ASA
+export_to_files(save, "output/", get_map_config("fjordur.ark"))
+```
+
+Output is identical to eager mode (validated record-for-record). ASE retains
+the file reader and re-seeks each object's block on demand; ASA retains the
+SQLite connection (one held read transaction) and re-fetches row blobs by
+GUID. On ASA v14+ saves the export pipeline additionally uses partial decodes:
+a verified byte-exact skip walk that parses only the property names a record
+needs, with any out-of-whitelist read transparently upgrading to the full
+block.
+
+Measured load + export, same machine: a 1.8 GB Fjordur PvE ASE save (1.79M
+objects) drops from ~7.1 GB peak RSS / 348 s eager to ~2.5 GB / 287 s lazy; a
+233 MB ASA TheIsland save (217k objects) drops from 853 MB / 38 s eager to
+307 MB / ~36 s lazy. Property access on a lazy save is transparent: any
+`get_property_value` call materializes the block on demand, so all getters and
+exports work unchanged.
+
 ### Cryopod-stored Creatures
 
 Tamed creatures stored inside in-world cryopods aren't returned by `get_tamed_creatures()`. Iterate them via `WorldSave.iter_cryopod_creatures()`:
@@ -218,7 +251,7 @@ The legacy ASVExport.exe emitted only the visible 8 stats (`hp`, `stam`, `melee`
 | `isClone` | legacy | `bIsClone` or `bIsCloneDino` |
 | `tamedServer` | legacy | `TamedOnServerName` |
 | `uploadedServer` | legacy | `UploadedFromServerName` |
-| `maturation` | legacy | `str(int(BabyAge * 100))` — integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
+| `maturation` | legacy | `str(int(BabyAge * 100))`, integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
 | `traits` | legacy | `CreatureTraits` as a list of `{"trait": <class>}` objects (matches `ASVExport`'s shape, not a flat string list) |
 | `inventory` | legacy | items from `MyInventoryComponent.InventoryItems`. Each entry carries `itemId`, `qty`, `blueprint`, plus a full snake_case property dump flattened in at the top level (`id`, `rating`, `durability`, `quality`, `damage`, `armor`, `durability_max`, `hypo`, `hyper`, `clip_size`, `weight`, `crafter`, `crafter_tribe`, `skill_bonus`, `loaded_ammo`, `spoils_at`, `spoiled_at`, `c0`..`c5`, `egg_*`, etc). `item_stat_values` is unpacked into the universal 8-slot ARK map (slot 0 `gen_quality`, 1 `armor`, 2 `durability_max`, 3 `damage`, 4 `clip_size`, 5 `hypo`, 6 `weight`, 7 `hyperthermal_insulation`); raw uint16s scaled by the per-blueprint multiplier (which lives in the UE blueprint, not the save). Default / unset values are filtered (no `craft_queue=0`, `skin=-1`, `color_pre_skin=[0]*6`, NaN spoil timers, etc). When the item is a **cryopod / soultrap / vivarium / dinoball** with an embedded creature, the entry is enriched with `dino_id` (combined 64-bit id matching the corresponding `ASV_Tamed` record), `dino_creature` (species / class name), and `dino_name` (`TamedName` if set). Cryopods stored in containers (cryofridges, vaults, dedicated storage) get the same enrichment. |
 | `father_id`, `mother_id` | added | combined dino id from the first `DinoAncestors` entry (`null` when missing) |
@@ -322,13 +355,13 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 
 #### `ASV_Tribes` schema
 
-`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam` — deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
+`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam`, deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
 
 | Field | Origin | Source / formula |
 |---|---|---|
 | `tribeid` | legacy | `TribeID` / parser `Tribe.tribe_id`, or a synthesized stub/sentinel id (see above) |
 | `tribe` | legacy | tribe name (file `TribeName`; for stubs, `OwnerName`/`TamerString`; or `Tribe of <character>` for a solo) |
-| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count` — not the raw `.arktribe` member count |
+| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count`, not the raw `.arktribe` member count |
 | `members` | legacy | list of `{ign, lvl, playerid, playername, steamid}` built from the allocated players. `lvl` and `steamid` populate from the matching `.arkprofile`; members with no profile (back-filled from the tribe's member list) carry `lvl=0`, `steamid=""`. |
 | `tames`, `structures` | legacy | counts derived from `WorldSave` (creatures + structures whose `TargetingTeam` matches) |
 | `uploadedTames` | legacy | reserved (currently `0`) |
@@ -353,7 +386,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `id` | added | `GameObject.id`, internal numeric identifier. Cross-references the values in other records' `linked_structures` / `saddle_structures` / `attached_dino_id` lists. |
 | `tribeid` | legacy | `TargetingTeam` for player-owned structures (`>= 50000`). Unowned structures fall to the synthetic `[ASV Abandoned]` tribe id `-2147483648` (legacy `int.MinValue`), mirroring `ContentContainer.cs`. |
 | `tribe` | legacy | resolved owning-tribe name: the loaded `.arktribe` `TribeName`, else the structure's `OwnerName` / `TamerString`; `"[ASV Abandoned]"` for unowned. (Legacy emits the resolved tribe name, not the raw `OwnerName`.) |
-| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded — surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
+| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded; surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
 | `name` | legacy | `BoxName` (empty when it matches the class name, mirroring legacy's no-rename strip) |
 | `locked` | legacy | `bIsPinLocked` or `bIsLocked` |
 | `created` | legacy (richer) | ISO 8601 datetime with the local TZ of the parser machine, computed `save.file_mtime + (OriginalCreationTime - game_time)` (mirrors legacy `ContentContainer.GetApproxDateTimeOf`). `null` when the anchors are missing. |
@@ -412,7 +445,7 @@ data = export_all(save, map_config, cluster="path/to/cluster")
 #                       .arkprofile filename stem.
 ```
 
-Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem — the Steam id on ASE, the hex platform UUID on ASA — so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths — which sets `Profile.source_path` — is required for the ASA splice.)
+Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem (the Steam id on ASE, the hex platform UUID on ASA) so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths (which sets `Profile.source_path`) is required for the ASA splice.)
 
 Pre-loaded `CloudInventory` instances also work (`cluster=[inv1, inv2, ...]`).
 
@@ -530,7 +563,9 @@ Key methods:
 
 | Method | Returns | Description |
 |---|---|---|
-| `load(source, load_properties=True, max_objects=None)` | `WorldSave` | Load and parse a world save |
+| `load(source, load_properties=True, max_objects=None, lazy_properties=False)` | `WorldSave` | Load and parse a world save (`lazy_properties`: on-demand property parsing, ASE + ASA, see Quick Start) |
+| `materialize_object(obj, names=None)` | `None` | Parse one lazy object's deferred property block (called automatically on property access; `names` is an ASA v14+ partial-decode hint) |
+| `evict_materialized()` | `int` | Release every property block materialized since the last call (lazy saves; no-op eager) |
 | `get_creatures()` | `list[GameObject]` | All creatures |
 | `get_tamed_creatures()` / `get_wild_creatures()` | `list[GameObject]` | Filtered creature sets |
 | `get_structures()` | `list[GameObject]` | Tribe-owned placed structures |
@@ -558,7 +593,7 @@ Key methods:
 | `properties` | `list[Property]` | Parsed property list |
 | `parent` / `components` | `GameObject \| None` / `dict[str, GameObject]` | Relationships |
 
-Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`.
+Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`. On lazy saves every helper materializes the deferred property block transparently; `evict_properties()` releases it again (re-access re-parses, so eviction is always safe).
 
 #### GameObjectContainer (`arkparser.GameObjectContainer`)
 
@@ -674,6 +709,11 @@ python -m pytest tests/ -v
 
 Tests live in `tests/`. Byte-level layout tests (`test_v13_property_layouts.py`, `test_binary_reader_layouts.py`) pin the canonical v13/v14 property body byte sequences with no file-fixture dependency. Integration tests (`test_world_save.py`, `test_export.py`, etc.) skip cleanly when their referenced save files are not present under `references/examples/`.
 
+Two suites guard export output against drift:
+
+- **Golden manifests** (`tests/test_golden_exports.py` + `tests/golden/*.json`): a per-map fingerprint of `export_all` (record count, field-key union, order-independent sha256) for 17 real saves (10 ASE map dumps + 7 ASA). Any change to parsing or export that alters output fails the matching map. Regenerate intentionally with `ARKPARSER_UPDATE_GOLDEN=1`.
+- **Lazy parity** (`tests/test_lazy_properties.py`): proves `lazy_properties=True` yields property-for-property identical objects (ASE and ASA), that evict/re-materialize round-trips, and that full lazy exports match the committed eager goldens. The ASA partial-decode skip walk is additionally verified byte-exact against the full parser over every object of every local ASA fixture by `references/scripts/verify_partial_walk.py`.
+
 ## Credits
 
 Built by reverse-engineering ARK save formats with heavy reference to [ASV (Ark Save Visualizer)](https://github.com/miragedmuk/ASV) by **miragedmuk**. The C# implementation in ASV is the primary reference for porting binary parsing logic to Python.

{arkparser-0.5.3 → arkparser-0.6.0}/README.md

@@ -24,6 +24,7 @@ A pure-Python library for parsing ARK: Survival Evolved (ASE) and ARK: Survival
 - **Dual format**: automatic ASE (v5-12) / ASA (v13-14+, SQLite) detection
 - **Legacy-parity export**: drop-in JSON output matching `ASVExport.exe` schema, plus parser-only extras under descriptive snake_case keys (no namespace prefix; never overloading a legacy key)
 - **Fast**: pure-Python `BinaryReader` (`int.from_bytes` + `struct.Struct` unpackers, slots-based dataclasses), a 30 MB ASE save (65k objects) loads in ~3s on CPython 3.14
+- **Memory-bounded**: streaming JSON export plus an opt-in lazy parse mode (`lazy_properties=True`, ASE and ASA) that materializes property blocks on demand and evicts them as records stream to disk; a 1.8 GB / 1.79M-object busy-PvE ASE save exports in 2.5 GB peak RSS instead of 7.1 GB, and a 217k-object ASA save in 307 MB instead of 853 MB (and slightly faster)
 
 ## Installation
 
@@ -98,6 +99,38 @@ print(f"Parse errors: {save.parse_error_count}")
 print(f"Is ASA: {save.is_asa}")
 ```
 
+### Low-memory parsing (large saves)
+
+By default every object's property block is parsed up front, which on a busy
+multi-GB PvE save means a multi-GB resident object graph. `lazy_properties=True`
+parses object headers eagerly but defers each property block until something
+reads it, then the export drivers evict it again once the record has been
+written:
+
+```python
+from arkparser import WorldSave, export_to_files
+from arkparser.common import get_map_config
+
+save = WorldSave.load("path/to/Fjordur.ark", lazy_properties=True)       # ASE
+save = WorldSave.load("path/to/TheIsland_WP.ark", lazy_properties=True)  # ASA
+export_to_files(save, "output/", get_map_config("fjordur.ark"))
+```
+
+Output is identical to eager mode (validated record-for-record). ASE retains
+the file reader and re-seeks each object's block on demand; ASA retains the
+SQLite connection (one held read transaction) and re-fetches row blobs by
+GUID. On ASA v14+ saves the export pipeline additionally uses partial decodes:
+a verified byte-exact skip walk that parses only the property names a record
+needs, with any out-of-whitelist read transparently upgrading to the full
+block.
+
+Measured load + export, same machine: a 1.8 GB Fjordur PvE ASE save (1.79M
+objects) drops from ~7.1 GB peak RSS / 348 s eager to ~2.5 GB / 287 s lazy; a
+233 MB ASA TheIsland save (217k objects) drops from 853 MB / 38 s eager to
+307 MB / ~36 s lazy. Property access on a lazy save is transparent: any
+`get_property_value` call materializes the block on demand, so all getters and
+exports work unchanged.
+
 ### Cryopod-stored Creatures
 
 Tamed creatures stored inside in-world cryopods aren't returned by `get_tamed_creatures()`. Iterate them via `WorldSave.iter_cryopod_creatures()`:
@@ -184,7 +217,7 @@ The legacy ASVExport.exe emitted only the visible 8 stats (`hp`, `stam`, `melee`
 | `isClone` | legacy | `bIsClone` or `bIsCloneDino` |
 | `tamedServer` | legacy | `TamedOnServerName` |
 | `uploadedServer` | legacy | `UploadedFromServerName` |
-| `maturation` | legacy | `str(int(BabyAge * 100))` — integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
+| `maturation` | legacy | `str(int(BabyAge * 100))`, integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
 | `traits` | legacy | `CreatureTraits` as a list of `{"trait": <class>}` objects (matches `ASVExport`'s shape, not a flat string list) |
 | `inventory` | legacy | items from `MyInventoryComponent.InventoryItems`. Each entry carries `itemId`, `qty`, `blueprint`, plus a full snake_case property dump flattened in at the top level (`id`, `rating`, `durability`, `quality`, `damage`, `armor`, `durability_max`, `hypo`, `hyper`, `clip_size`, `weight`, `crafter`, `crafter_tribe`, `skill_bonus`, `loaded_ammo`, `spoils_at`, `spoiled_at`, `c0`..`c5`, `egg_*`, etc). `item_stat_values` is unpacked into the universal 8-slot ARK map (slot 0 `gen_quality`, 1 `armor`, 2 `durability_max`, 3 `damage`, 4 `clip_size`, 5 `hypo`, 6 `weight`, 7 `hyperthermal_insulation`); raw uint16s scaled by the per-blueprint multiplier (which lives in the UE blueprint, not the save). Default / unset values are filtered (no `craft_queue=0`, `skin=-1`, `color_pre_skin=[0]*6`, NaN spoil timers, etc). When the item is a **cryopod / soultrap / vivarium / dinoball** with an embedded creature, the entry is enriched with `dino_id` (combined 64-bit id matching the corresponding `ASV_Tamed` record), `dino_creature` (species / class name), and `dino_name` (`TamedName` if set). Cryopods stored in containers (cryofridges, vaults, dedicated storage) get the same enrichment. |
 | `father_id`, `mother_id` | added | combined dino id from the first `DinoAncestors` entry (`null` when missing) |
@@ -288,13 +321,13 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 
 #### `ASV_Tribes` schema
 
-`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam` — deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
+`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam`, deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
 
 | Field | Origin | Source / formula |
 |---|---|---|
 | `tribeid` | legacy | `TribeID` / parser `Tribe.tribe_id`, or a synthesized stub/sentinel id (see above) |
 | `tribe` | legacy | tribe name (file `TribeName`; for stubs, `OwnerName`/`TamerString`; or `Tribe of <character>` for a solo) |
-| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count` — not the raw `.arktribe` member count |
+| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count`, not the raw `.arktribe` member count |
 | `members` | legacy | list of `{ign, lvl, playerid, playername, steamid}` built from the allocated players. `lvl` and `steamid` populate from the matching `.arkprofile`; members with no profile (back-filled from the tribe's member list) carry `lvl=0`, `steamid=""`. |
 | `tames`, `structures` | legacy | counts derived from `WorldSave` (creatures + structures whose `TargetingTeam` matches) |
 | `uploadedTames` | legacy | reserved (currently `0`) |
@@ -319,7 +352,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `id` | added | `GameObject.id`, internal numeric identifier. Cross-references the values in other records' `linked_structures` / `saddle_structures` / `attached_dino_id` lists. |
 | `tribeid` | legacy | `TargetingTeam` for player-owned structures (`>= 50000`). Unowned structures fall to the synthetic `[ASV Abandoned]` tribe id `-2147483648` (legacy `int.MinValue`), mirroring `ContentContainer.cs`. |
 | `tribe` | legacy | resolved owning-tribe name: the loaded `.arktribe` `TribeName`, else the structure's `OwnerName` / `TamerString`; `"[ASV Abandoned]"` for unowned. (Legacy emits the resolved tribe name, not the raw `OwnerName`.) |
-| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded — surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
+| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded; surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
 | `name` | legacy | `BoxName` (empty when it matches the class name, mirroring legacy's no-rename strip) |
 | `locked` | legacy | `bIsPinLocked` or `bIsLocked` |
 | `created` | legacy (richer) | ISO 8601 datetime with the local TZ of the parser machine, computed `save.file_mtime + (OriginalCreationTime - game_time)` (mirrors legacy `ContentContainer.GetApproxDateTimeOf`). `null` when the anchors are missing. |
@@ -378,7 +411,7 @@ data = export_all(save, map_config, cluster="path/to/cluster")
 #                       .arkprofile filename stem.
 ```
 
-Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem — the Steam id on ASE, the hex platform UUID on ASA — so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths — which sets `Profile.source_path` — is required for the ASA splice.)
+Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem (the Steam id on ASE, the hex platform UUID on ASA) so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths (which sets `Profile.source_path`) is required for the ASA splice.)
 
 Pre-loaded `CloudInventory` instances also work (`cluster=[inv1, inv2, ...]`).
 
@@ -496,7 +529,9 @@ Key methods:
 
 | Method | Returns | Description |
 |---|---|---|
-| `load(source, load_properties=True, max_objects=None)` | `WorldSave` | Load and parse a world save |
+| `load(source, load_properties=True, max_objects=None, lazy_properties=False)` | `WorldSave` | Load and parse a world save (`lazy_properties`: on-demand property parsing, ASE + ASA, see Quick Start) |
+| `materialize_object(obj, names=None)` | `None` | Parse one lazy object's deferred property block (called automatically on property access; `names` is an ASA v14+ partial-decode hint) |
+| `evict_materialized()` | `int` | Release every property block materialized since the last call (lazy saves; no-op eager) |
 | `get_creatures()` | `list[GameObject]` | All creatures |
 | `get_tamed_creatures()` / `get_wild_creatures()` | `list[GameObject]` | Filtered creature sets |
 | `get_structures()` | `list[GameObject]` | Tribe-owned placed structures |
@@ -524,7 +559,7 @@ Key methods:
 | `properties` | `list[Property]` | Parsed property list |
 | `parent` / `components` | `GameObject \| None` / `dict[str, GameObject]` | Relationships |
 
-Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`.
+Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`. On lazy saves every helper materializes the deferred property block transparently; `evict_properties()` releases it again (re-access re-parses, so eviction is always safe).
 
 #### GameObjectContainer (`arkparser.GameObjectContainer`)
 
@@ -640,6 +675,11 @@ python -m pytest tests/ -v
 
 Tests live in `tests/`. Byte-level layout tests (`test_v13_property_layouts.py`, `test_binary_reader_layouts.py`) pin the canonical v13/v14 property body byte sequences with no file-fixture dependency. Integration tests (`test_world_save.py`, `test_export.py`, etc.) skip cleanly when their referenced save files are not present under `references/examples/`.
 
+Two suites guard export output against drift:
+
+- **Golden manifests** (`tests/test_golden_exports.py` + `tests/golden/*.json`): a per-map fingerprint of `export_all` (record count, field-key union, order-independent sha256) for 17 real saves (10 ASE map dumps + 7 ASA). Any change to parsing or export that alters output fails the matching map. Regenerate intentionally with `ARKPARSER_UPDATE_GOLDEN=1`.
+- **Lazy parity** (`tests/test_lazy_properties.py`): proves `lazy_properties=True` yields property-for-property identical objects (ASE and ASA), that evict/re-materialize round-trips, and that full lazy exports match the committed eager goldens. The ASA partial-decode skip walk is additionally verified byte-exact against the full parser over every object of every local ASA fixture by `references/scripts/verify_partial_walk.py`.
+
 ## Credits
 
 Built by reverse-engineering ARK save formats with heavy reference to [ASV (Ark Save Visualizer)](https://github.com/miragedmuk/ASV) by **miragedmuk**. The C# implementation in ASV is the primary reference for porting binary parsing logic to Python.

{arkparser-0.5.3 → arkparser-0.6.0}/arkparser/__init__.py

@@ -1,98 +1,98 @@
-"""
-ARK Save Parser - Parse ARK: Survival Evolved/Ascended save files.
-
-This package provides tools to parse various ARK save file formats:
-
-- Profile: Player profile data (.arkprofile)
-- Tribe: Tribe data (.arktribe)
-- CloudInventory: Obelisk/cloud inventory data (no extension)
-- WorldSave: World save data (.ark), auto-detects ASE binary and ASA SQLite
-
-Supports both ASE (ARK: Survival Evolved) and ASA (ARK: Survival Ascended)
-formats with automatic detection.
-
-Example usage:
-    >>> from arkparser import Profile, Tribe, CloudInventory, WorldSave
-    >>> profile = Profile.load("path/to/profile.arkprofile")
-    >>> tribe = Tribe.load("path/to/tribe.arktribe")
-    >>> inv = CloudInventory.load("path/to/obelisk_file")
-    >>> save = WorldSave.load("path/to/TheIsland.ark")    # ASE
-    >>> save = WorldSave.load("path/to/Extinction_WP.ark")  # ASA
-"""
-
-from arkparser.common.exceptions import ArkParseError
-from arkparser.common.map_config import MapConfig, get_map_config, get_map_config_by_name
-from arkparser.common.version_detection import (
-    ArkFileFormat,
-    ArkFileType,
-    detect_file_type,
-    detect_format,
-)
-from arkparser.data_models import (
-    CryopodCreature,
-    DinoStats,
-    UploadedCreature,
-    UploadedItem,
-)
-from arkparser.export import (
-    export_all,
-    export_cloud_inventory,
-    export_cluster_items,
-    export_cluster_uploads,
-    export_map_structures,
-    export_players,
-    export_structures,
-    export_tamed,
-    export_to_files,
-    export_tribe_logs,
-    export_tribes,
-    export_wild,
-)
-from arkparser.files import CloudInventory, Profile, Tribe, WorldSave
-from arkparser.game_objects import GameObject, GameObjectContainer, LocationData
-
-# Convenience alias - users may know this as "Obelisk" from the game
-Obelisk = CloudInventory
-
-__all__ = [
-    # File parsers
-    "Profile",
-    "Tribe",
-    "CloudInventory",
-    "Obelisk",
-    "WorldSave",
-    # Cloud-inventory data models
-    "UploadedCreature",
-    "UploadedItem",
-    "CryopodCreature",
-    "DinoStats",
-    # Game objects
-    "GameObject",
-    "GameObjectContainer",
-    "LocationData",
-    # Map config
-    "MapConfig",
-    "get_map_config",
-    "get_map_config_by_name",
-    # Export
-    "export_all",
-    "export_tamed",
-    "export_wild",
-    "export_players",
-    "export_tribes",
-    "export_structures",
-    "export_tribe_logs",
-    "export_map_structures",
-    "export_cluster_uploads",
-    "export_cluster_items",
-    "export_cloud_inventory",
-    "export_to_files",
-    # Utilities
-    "detect_format",
-    "detect_file_type",
-    "ArkFileFormat",
-    "ArkFileType",
-    "ArkParseError",
-]
-
-__version__ = "0.5.3"
+"""
+ARK Save Parser - Parse ARK: Survival Evolved/Ascended save files.
+
+This package provides tools to parse various ARK save file formats:
+
+- Profile: Player profile data (.arkprofile)
+- Tribe: Tribe data (.arktribe)
+- CloudInventory: Obelisk/cloud inventory data (no extension)
+- WorldSave: World save data (.ark), auto-detects ASE binary and ASA SQLite
+
+Supports both ASE (ARK: Survival Evolved) and ASA (ARK: Survival Ascended)
+formats with automatic detection.
+
+Example usage:
+    >>> from arkparser import Profile, Tribe, CloudInventory, WorldSave
+    >>> profile = Profile.load("path/to/profile.arkprofile")
+    >>> tribe = Tribe.load("path/to/tribe.arktribe")
+    >>> inv = CloudInventory.load("path/to/obelisk_file")
+    >>> save = WorldSave.load("path/to/TheIsland.ark")    # ASE
+    >>> save = WorldSave.load("path/to/Extinction_WP.ark")  # ASA
+"""
+
+from arkparser.common.exceptions import ArkParseError
+from arkparser.common.map_config import MapConfig, get_map_config, get_map_config_by_name
+from arkparser.common.version_detection import (
+    ArkFileFormat,
+    ArkFileType,
+    detect_file_type,
+    detect_format,
+)
+from arkparser.data_models import (
+    CryopodCreature,
+    DinoStats,
+    UploadedCreature,
+    UploadedItem,
+)
+from arkparser.export import (
+    export_all,
+    export_cloud_inventory,
+    export_cluster_items,
+    export_cluster_uploads,
+    export_map_structures,
+    export_players,
+    export_structures,
+    export_tamed,
+    export_to_files,
+    export_tribe_logs,
+    export_tribes,
+    export_wild,
+)
+from arkparser.files import CloudInventory, Profile, Tribe, WorldSave
+from arkparser.game_objects import GameObject, GameObjectContainer, LocationData
+
+# Convenience alias - users may know this as "Obelisk" from the game
+Obelisk = CloudInventory
+
+__all__ = [
+    # File parsers
+    "Profile",
+    "Tribe",
+    "CloudInventory",
+    "Obelisk",
+    "WorldSave",
+    # Cloud-inventory data models
+    "UploadedCreature",
+    "UploadedItem",
+    "CryopodCreature",
+    "DinoStats",
+    # Game objects
+    "GameObject",
+    "GameObjectContainer",
+    "LocationData",
+    # Map config
+    "MapConfig",
+    "get_map_config",
+    "get_map_config_by_name",
+    # Export
+    "export_all",
+    "export_tamed",
+    "export_wild",
+    "export_players",
+    "export_tribes",
+    "export_structures",
+    "export_tribe_logs",
+    "export_map_structures",
+    "export_cluster_uploads",
+    "export_cluster_items",
+    "export_cloud_inventory",
+    "export_to_files",
+    # Utilities
+    "detect_format",
+    "detect_file_type",
+    "ArkFileFormat",
+    "ArkFileType",
+    "ArkParseError",
+]
+
+__version__ = "0.6.0"

{arkparser-0.5.3 → arkparser-0.6.0}/arkparser/common/binary_reader.py

@@ -21,7 +21,10 @@ Example:
 
 from __future__ import annotations
 
+import ctypes
+import mmap
 import struct
+import sys
 from pathlib import Path
 from uuid import UUID
 
@@ -32,6 +35,25 @@ from .exceptions import EndOfDataError
 _S_FLOAT = struct.Struct("<f")
 _S_DOUBLE = struct.Struct("<d")
 _S_INT32_PAIR = struct.Struct("<ii")
+_S_INT32_X4 = struct.Struct("<4i")
+
+
+def guid_str_le(raw: bytes) -> str:
+    """Format a 16-byte little-endian GUID as its canonical string.
+
+    Byte-for-byte equivalent to ``str(uuid.UUID(bytes_le=raw))`` (asserted in
+    tests) at ~40% of the cost; ASA saves key every game row and actor
+    transform by GUID, so this runs hundreds of thousands of times per load.
+    """
+    assert len(raw) == 16, "GUID must be 16 bytes"
+    h = raw.hex()
+    return (
+        h[6:8] + h[4:6] + h[2:4] + h[0:2]
+        + "-" + h[10:12] + h[8:10]
+        + "-" + h[14:16] + h[12:14]
+        + "-" + h[16:20]
+        + "-" + h[20:32]
+    )
 
 
 class BinaryReader:
@@ -39,10 +61,14 @@ class BinaryReader:
 
     __slots__ = ("_buf", "_pos", "_size", "save_version")
 
-    def __init__(self, data: bytes | memoryview, save_version: int = 0) -> None:
-        # Materialize to bytes once. CPython 3.14 small-bytes slicing is
-        # significantly faster than memoryview slicing for the per-read
-        # ``int.from_bytes(buf[a:b], ...)`` hot path used here.
+    def __init__(self, data: bytes | memoryview | mmap.mmap, save_version: int = 0) -> None:
+        # Materialize memoryviews to bytes once. CPython 3.14 small-bytes
+        # slicing is significantly faster than memoryview slicing for the
+        # per-read ``int.from_bytes(buf[a:b], ...)`` hot path used here.
+        # ``mmap`` buffers are kept as-is: slicing an mmap returns plain
+        # ``bytes``, so every read path behaves identically while the file
+        # contents stay out of the Python heap (used by lazy world saves,
+        # which retain the reader for the whole export).
         if isinstance(data, memoryview):
             self._buf = bytes(data)
         else:
@@ -61,9 +87,46 @@ class BinaryReader:
         self.close()
 
     def close(self) -> None:
-        # Nothing to release; underlying buffer is plain bytes.
+        # Plain-bytes buffers have nothing to release; mmap buffers unmap.
+        if isinstance(self._buf, mmap.mmap):
+            self._buf.close()
         return None
 
+    def trim_working_set(self) -> bool:
+        """Drop this reader's clean mmap pages from the process working set.
+
+        A lazy world save touches every property page over the course of an
+        export. Those pages are clean and file-backed, but they accumulate in
+        the working set until peak RSS approaches heap + whole file size. On
+        POSIX, madvise(MADV_DONTNEED) releases just this mapping. Windows has
+        no per-range call usable on a read-only mapping, so EmptyWorkingSet
+        trims the whole process; evicted heap pages soft-fault back from the
+        standby list with no disk I/O.
+
+        Pre: none (safe on any reader). Post: returns True only when the
+        reader is mmap-backed and a trim was actually issued; plain-bytes
+        readers return False and do nothing.
+        """
+        if not isinstance(self._buf, mmap.mmap):
+            return False
+        if sys.platform == "win32":
+            kernel32 = ctypes.windll.kernel32
+            kernel32.GetCurrentProcess.restype = ctypes.c_void_p
+            handle = kernel32.GetCurrentProcess()
+            # EmptyWorkingSet lives in psapi.dll on older Windows and is
+            # forwarded from kernel32 as K32EmptyWorkingSet on newer builds.
+            empty = getattr(kernel32, "K32EmptyWorkingSet", None)
+            if empty is None:
+                empty = ctypes.windll.psapi.EmptyWorkingSet
+            empty.argtypes = [ctypes.c_void_p]
+            empty.restype = ctypes.c_int
+            return bool(empty(handle))
+        madvise_flag = getattr(mmap, "MADV_DONTNEED", None)
+        if madvise_flag is None:
+            return False
+        self._buf.madvise(madvise_flag)
+        return True
+
     # =========================================================================
     # Factory Methods
     # =========================================================================
@@ -72,6 +135,23 @@ class BinaryReader:
     def from_file(cls, path: str | Path) -> BinaryReader:
         return cls(Path(path).read_bytes())
 
+    @classmethod
+    def from_file_mmap(cls, path: str | Path) -> BinaryReader:
+        """Memory-map the file instead of reading it onto the heap.
+
+        For readers retained over a long export (lazy world saves): the OS
+        pages the contents in on demand and may reclaim them under pressure,
+        so the file bytes never count against the Python heap. Falls back to
+        :meth:`from_file` for empty files (Windows cannot mmap length 0).
+        """
+        p = Path(path)
+        if p.stat().st_size == 0:
+            return cls.from_file(p)
+        with open(p, "rb") as fh:
+            # mmap duplicates the OS handle; closing fh afterwards is safe.
+            buf = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+        return cls(buf)
+
     @classmethod
     def from_bytes(cls, data: bytes) -> BinaryReader:
         return cls(bytes(data) if not isinstance(data, bytes) else data)
@@ -200,6 +280,19 @@ class BinaryReader:
         self._pos += 8
         return a, b
 
+    def read_int32_x4(self) -> tuple[int, int, int, int]:
+        """Read four signed int32 values in a single struct unpack call.
+
+        Hot path for ASE property headers: after the name pair, the type
+        name-ref (index + instance) and data_size + index are 16 contiguous
+        bytes. One call beats two read_int32_pair calls.
+        """
+        if self._pos + 16 > self._size:
+            raise EndOfDataError(16, self._size - self._pos)
+        vals = _S_INT32_X4.unpack_from(self._buf, self._pos)
+        self._pos += 16
+        return vals
+
     # =========================================================================
     # Floating Point Types
     # =========================================================================

{arkparser-0.5.3 → arkparser-0.6.0}/arkparser/common/normalization.py

@@ -21,8 +21,17 @@ def normalize_indexed_data(value: t.Any) -> t.Any:
       and collapsed to a single value when only one indexed entry exists.
     - Other dicts are normalized value-by-value while keeping their keys.
     """
+    # Scalars are by far the most common input. Inline the recursion's leaf
+    # case at every call site below (``x if not container else recurse``) so a
+    # scalar element costs one ``isinstance`` instead of a full recursive call;
+    # this function was the hottest frame in the export profile (~2.5M calls).
     if isinstance(value, list):
-        return [normalize_indexed_data(item) for item in value]
+        return [
+            item
+            if not isinstance(item, (list, dict))
+            else normalize_indexed_data(item)
+            for item in value
+        ]
 
     if not isinstance(value, dict):
         return value
@@ -30,7 +39,11 @@ def normalize_indexed_data(value: t.Any) -> t.Any:
     all_int = True
     out: dict[t.Any, t.Any] = {}
     for key, item in value.items():
-        out[key] = normalize_indexed_data(item)
+        out[key] = (
+            item
+            if not isinstance(item, (list, dict))
+            else normalize_indexed_data(item)
+        )
         if all_int and not isinstance(key, int):
             all_int = False