arkparser 0.5.4__tar.gz → 0.7.0__tar.gz

{arkparser-0.5.4 → arkparser-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: arkparser
-Version: 0.5.4
+Version: 0.7.0
 Summary: Pure Python parser for ARK: Survival Evolved and ARK: Survival Ascended save files
 Author: Vertyco
 License-Expression: MIT
@@ -58,6 +58,7 @@ A pure-Python library for parsing ARK: Survival Evolved (ASE) and ARK: Survival
 - **Dual format**: automatic ASE (v5-12) / ASA (v13-14+, SQLite) detection
 - **Legacy-parity export**: drop-in JSON output matching `ASVExport.exe` schema, plus parser-only extras under descriptive snake_case keys (no namespace prefix; never overloading a legacy key)
 - **Fast**: pure-Python `BinaryReader` (`int.from_bytes` + `struct.Struct` unpackers, slots-based dataclasses), a 30 MB ASE save (65k objects) loads in ~3s on CPython 3.14
+- **Memory-bounded**: streaming JSON export plus an opt-in lazy parse mode (`lazy_properties=True`, ASE and ASA) that materializes property blocks on demand and evicts them as records stream to disk; a 1.8 GB / 1.79M-object busy-PvE ASE save exports in 2.5 GB peak RSS instead of 7.1 GB, and a 217k-object ASA save in 307 MB instead of 853 MB (and slightly faster)
 
 ## Installation
 
@@ -132,6 +133,38 @@ print(f"Parse errors: {save.parse_error_count}")
 print(f"Is ASA: {save.is_asa}")
 ```
 
+### Low-memory parsing (large saves)
+
+By default every object's property block is parsed up front, which on a busy
+multi-GB PvE save means a multi-GB resident object graph. `lazy_properties=True`
+parses object headers eagerly but defers each property block until something
+reads it, then the export drivers evict it again once the record has been
+written:
+
+```python
+from arkparser import WorldSave, export_to_files
+from arkparser.common import get_map_config
+
+save = WorldSave.load("path/to/Fjordur.ark", lazy_properties=True)       # ASE
+save = WorldSave.load("path/to/TheIsland_WP.ark", lazy_properties=True)  # ASA
+export_to_files(save, "output/", get_map_config("fjordur.ark"))
+```
+
+Output is identical to eager mode (validated record-for-record). ASE retains
+the file reader and re-seeks each object's block on demand; ASA retains the
+SQLite connection (one held read transaction) and re-fetches row blobs by
+GUID. On ASA v14+ saves the export pipeline additionally uses partial decodes:
+a verified byte-exact skip walk that parses only the property names a record
+needs, with any out-of-whitelist read transparently upgrading to the full
+block.
+
+Measured load + export, same machine: a 1.8 GB Fjordur PvE ASE save (1.79M
+objects) drops from ~7.1 GB peak RSS / 348 s eager to ~2.5 GB / 287 s lazy; a
+233 MB ASA TheIsland save (217k objects) drops from 853 MB / 38 s eager to
+307 MB / ~36 s lazy. Property access on a lazy save is transparent: any
+`get_property_value` call materializes the block on demand, so all getters and
+exports work unchanged.
+
 ### Cryopod-stored Creatures
 
 Tamed creatures stored inside in-world cryopods aren't returned by `get_tamed_creatures()`. Iterate them via `WorldSave.iter_cryopod_creatures()`:
@@ -218,7 +251,7 @@ The legacy ASVExport.exe emitted only the visible 8 stats (`hp`, `stam`, `melee`
 | `isClone` | legacy | `bIsClone` or `bIsCloneDino` |
 | `tamedServer` | legacy | `TamedOnServerName` |
 | `uploadedServer` | legacy | `UploadedFromServerName` |
-| `maturation` | legacy | `str(int(BabyAge * 100))` — integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
+| `maturation` | legacy | `str(int(BabyAge * 100))`, integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
 | `traits` | legacy | `CreatureTraits` as a list of `{"trait": <class>}` objects (matches `ASVExport`'s shape, not a flat string list) |
 | `inventory` | legacy | items from `MyInventoryComponent.InventoryItems`. Each entry carries `itemId`, `qty`, `blueprint`, plus a full snake_case property dump flattened in at the top level (`id`, `rating`, `durability`, `quality`, `damage`, `armor`, `durability_max`, `hypo`, `hyper`, `clip_size`, `weight`, `crafter`, `crafter_tribe`, `skill_bonus`, `loaded_ammo`, `spoils_at`, `spoiled_at`, `c0`..`c5`, `egg_*`, etc). `item_stat_values` is unpacked into the universal 8-slot ARK map (slot 0 `gen_quality`, 1 `armor`, 2 `durability_max`, 3 `damage`, 4 `clip_size`, 5 `hypo`, 6 `weight`, 7 `hyperthermal_insulation`); raw uint16s scaled by the per-blueprint multiplier (which lives in the UE blueprint, not the save). Default / unset values are filtered (no `craft_queue=0`, `skin=-1`, `color_pre_skin=[0]*6`, NaN spoil timers, etc). When the item is a **cryopod / soultrap / vivarium / dinoball** with an embedded creature, the entry is enriched with `dino_id` (combined 64-bit id matching the corresponding `ASV_Tamed` record), `dino_creature` (species / class name), and `dino_name` (`TamedName` if set). Cryopods stored in containers (cryofridges, vaults, dedicated storage) get the same enrichment. |
 | `father_id`, `mother_id` | added | combined dino id from the first `DinoAncestors` entry (`null` when missing) |
@@ -281,7 +314,7 @@ ARK keeps player data in **two distinct places**, and the parser builds `ASV_Pla
 
 The export pipeline (`export_players`) loops over `save.profiles` (a list the caller assembles). For each entry:
 
-- If it's a `Profile` instance → `_player_from_profile` runs: fills core identity (name, gender, level, stats, tribe id, engram count, experience, active datetime from `LastLoginTime`). Leaves `lat`/`lon`/`ccc` as zero, `inventory` empty, pawn-state flags (`is_sleeping`, `is_dead`, `chibi_levels`, …) absent.
+- If it's a `Profile` instance → `_player_from_profile` runs: fills core identity (name, gender, level, stats, tribe id, engram count, experience, active datetime from `LastLoginTime`). It joins `PlayerDataID == LinkedPlayerDataID` against the world save's player pawns, so when a live pawn exists `lat`/`lon`/`ccc` come from that pawn's `.location`; players with no live pawn (dead / logged out) keep `lat`/`lon`/`ccc` at zero. `inventory` is empty and pawn-state flags (`is_sleeping`, `is_dead`, `chibi_levels`, …) absent (the profile carries neither).
 - Otherwise it's treated as a wrapped `(profile, objects)` pair pointing at an in-world `PlayerPawn` → `_player_from_object` runs: fills location, inventory, pawn-state flags, body/hair cosmetics, death timestamps, and `active` from `SavedLastTimeHadController`.
 
 For the richest output, hand `export_players` **both**, assemble a wrapper for each player that carries the `Profile` (for offline identity) *and* the in-world pawn (when present) so the merged record gets identity + live state. The current validation script only passes `Profile` instances, which is why fields like `is_sleeping` / `body_colors` / `current_weapon` show up empty in the validation output even though the parser supports them.
@@ -303,7 +336,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `netAddress` | legacy (now populated) | Last client IP ARK persisted (`SavedNetworkAddress` in profile `MyData`). Legacy ASVExport reads the same field (ContentPlayer.cs:157 ASE / :341 ASA). `""` when the profile lacks it (e.g. never-played placeholders). ASA stores a clean IPv4/IPv6 string; some ASE saves store an engine-truncated value (e.g. `"[2001"`) reproduced verbatim, matching legacy. |
 | `steamid`, `dataFile` | legacy | platform net id and `{steamid}.arkprofile` filename |
 | `active` | legacy (now populated) | ISO 8601 datetime of last login, converted from profile `LastLoginTime` or in-world pawn `SavedLastTimeHadController`. Legacy schema reserved the field but the old C# exporter only filled it for in-world pawns; the parser fills it for profiles too. `null` when neither source is present. |
-| `lat`, `lon`, `ccc` | legacy (now populated) | In-world position when the record was built from a `PlayerPawn` GameObject. Records sourced from profile / cluster files keep the legacy `0` / `"0 0 0"` placeholders since profiles carry no world position. |
+| `lat`, `lon`, `ccc` | legacy (now populated) | In-world position. Built directly from a `PlayerPawn` GameObject when one is passed, and for profile-sourced records the parser joins `PlayerDataID == LinkedPlayerDataID` to the world save's player pawns and pulls the matched pawn's `.location`. Records whose player has no live pawn (dead / logged out / cluster-only) keep the legacy `0` / `"0 0 0"` placeholders since there is no world position to report. |
 | `inventory` | legacy (now populated) | Items from the pawn's `MyInventoryComponent` when built from an in-world pawn. Empty list otherwise. |
 | `engram_points` | added | `TotalEngramPoints` |
 | `experience` | added | `ExperiencePoints` (status), integer |
@@ -322,17 +355,17 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 
 #### `ASV_Tribes` schema
 
-`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam` — deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
+`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam`, deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
 
 | Field | Origin | Source / formula |
 |---|---|---|
 | `tribeid` | legacy | `TribeID` / parser `Tribe.tribe_id`, or a synthesized stub/sentinel id (see above) |
 | `tribe` | legacy | tribe name (file `TribeName`; for stubs, `OwnerName`/`TamerString`; or `Tribe of <character>` for a solo) |
-| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count` — not the raw `.arktribe` member count |
+| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count`, not the raw `.arktribe` member count |
 | `members` | legacy | list of `{ign, lvl, playerid, playername, steamid}` built from the allocated players. `lvl` and `steamid` populate from the matching `.arkprofile`; members with no profile (back-filled from the tribe's member list) carry `lvl=0`, `steamid=""`. |
 | `tames`, `structures` | legacy | counts derived from `WorldSave` (creatures + structures whose `TargetingTeam` matches) |
 | `uploadedTames` | legacy | reserved (currently `0`) |
-| `active` | legacy | ISO 8601 datetime of the most recent tribe log entry, converted from in-game "Day N, HH:MM:SS" via the save's anchor. `null` when no parseable log entries or anchors are missing. |
+| `active` | legacy | ISO 8601 datetime, legacy `ContentTribe.LastActive`: max of the tribe file's write time (`.arktribe` mtime; world-save mtime for in-save tribes) and the allocated members' last-active times (`LastLoginTime` via the save anchor), discarding future values. `null` when no past candidate exists. Tribe-log "Day N" stamps are game-calendar time and are deliberately not used. |
 | `dataFile` | legacy | `"{tribeid}.arktribe"` filename pattern. |
 | `owner_id` | added | `OwnerPlayerDataID` / parser `Tribe.owner_player_id` |
 | `owner_name` | added | `OwnerPlayerName` (object form only; parser-tribe form has no equivalent) |
@@ -353,7 +386,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `id` | added | `GameObject.id`, internal numeric identifier. Cross-references the values in other records' `linked_structures` / `saddle_structures` / `attached_dino_id` lists. |
 | `tribeid` | legacy | `TargetingTeam` for player-owned structures (`>= 50000`). Unowned structures fall to the synthetic `[ASV Abandoned]` tribe id `-2147483648` (legacy `int.MinValue`), mirroring `ContentContainer.cs`. |
 | `tribe` | legacy | resolved owning-tribe name: the loaded `.arktribe` `TribeName`, else the structure's `OwnerName` / `TamerString`; `"[ASV Abandoned]"` for unowned. (Legacy emits the resolved tribe name, not the raw `OwnerName`.) |
-| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded — surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
+| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded; surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
 | `name` | legacy | `BoxName` (empty when it matches the class name, mirroring legacy's no-rename strip) |
 | `locked` | legacy | `bIsPinLocked` or `bIsLocked` |
 | `created` | legacy (richer) | ISO 8601 datetime with the local TZ of the parser machine, computed `save.file_mtime + (OriginalCreationTime - game_time)` (mirrors legacy `ContentContainer.GetApproxDateTimeOf`). `null` when the anchors are missing. |
@@ -412,7 +445,7 @@ data = export_all(save, map_config, cluster="path/to/cluster")
 #                       .arkprofile filename stem.
 ```
 
-Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem — the Steam id on ASE, the hex platform UUID on ASA — so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths — which sets `Profile.source_path` — is required for the ASA splice.)
+Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem (the Steam id on ASE, the hex platform UUID on ASA) so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths (which sets `Profile.source_path`) is required for the ASA splice.)
 
 Pre-loaded `CloudInventory` instances also work (`cluster=[inv1, inv2, ...]`).
 
@@ -530,7 +563,9 @@ Key methods:
 
 | Method | Returns | Description |
 |---|---|---|
-| `load(source, load_properties=True, max_objects=None)` | `WorldSave` | Load and parse a world save |
+| `load(source, load_properties=True, max_objects=None, lazy_properties=False)` | `WorldSave` | Load and parse a world save (`lazy_properties`: on-demand property parsing, ASE + ASA, see Quick Start) |
+| `materialize_object(obj, names=None)` | `None` | Parse one lazy object's deferred property block (called automatically on property access; `names` is an ASA v14+ partial-decode hint) |
+| `evict_materialized()` | `int` | Release every property block materialized since the last call (lazy saves; no-op eager) |
 | `get_creatures()` | `list[GameObject]` | All creatures |
 | `get_tamed_creatures()` / `get_wild_creatures()` | `list[GameObject]` | Filtered creature sets |
 | `get_structures()` | `list[GameObject]` | Tribe-owned placed structures |
@@ -558,7 +593,7 @@ Key methods:
 | `properties` | `list[Property]` | Parsed property list |
 | `parent` / `components` | `GameObject \| None` / `dict[str, GameObject]` | Relationships |
 
-Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`.
+Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`. On lazy saves every helper materializes the deferred property block transparently; `evict_properties()` releases it again (re-access re-parses, so eviction is always safe).
 
 #### GameObjectContainer (`arkparser.GameObjectContainer`)
 
@@ -674,6 +709,11 @@ python -m pytest tests/ -v
 
 Tests live in `tests/`. Byte-level layout tests (`test_v13_property_layouts.py`, `test_binary_reader_layouts.py`) pin the canonical v13/v14 property body byte sequences with no file-fixture dependency. Integration tests (`test_world_save.py`, `test_export.py`, etc.) skip cleanly when their referenced save files are not present under `references/examples/`.
 
+Two suites guard export output against drift:
+
+- **Golden manifests** (`tests/test_golden_exports.py` + `tests/golden/*.json`): a per-map fingerprint of `export_all` (record count, field-key union, order-independent sha256) for 17 real saves (10 ASE map dumps + 7 ASA). Any change to parsing or export that alters output fails the matching map. Regenerate intentionally with `ARKPARSER_UPDATE_GOLDEN=1`.
+- **Lazy parity** (`tests/test_lazy_properties.py`): proves `lazy_properties=True` yields property-for-property identical objects (ASE and ASA), that evict/re-materialize round-trips, and that full lazy exports match the committed eager goldens. The ASA partial-decode skip walk is additionally verified byte-exact against the full parser over every object of every local ASA fixture by `references/scripts/verify_partial_walk.py`.
+
 ## Credits
 
 Built by reverse-engineering ARK save formats with heavy reference to [ASV (Ark Save Visualizer)](https://github.com/miragedmuk/ASV) by **miragedmuk**. The C# implementation in ASV is the primary reference for porting binary parsing logic to Python.

{arkparser-0.5.4 → arkparser-0.7.0}/README.md

@@ -24,6 +24,7 @@ A pure-Python library for parsing ARK: Survival Evolved (ASE) and ARK: Survival
 - **Dual format**: automatic ASE (v5-12) / ASA (v13-14+, SQLite) detection
 - **Legacy-parity export**: drop-in JSON output matching `ASVExport.exe` schema, plus parser-only extras under descriptive snake_case keys (no namespace prefix; never overloading a legacy key)
 - **Fast**: pure-Python `BinaryReader` (`int.from_bytes` + `struct.Struct` unpackers, slots-based dataclasses), a 30 MB ASE save (65k objects) loads in ~3s on CPython 3.14
+- **Memory-bounded**: streaming JSON export plus an opt-in lazy parse mode (`lazy_properties=True`, ASE and ASA) that materializes property blocks on demand and evicts them as records stream to disk; a 1.8 GB / 1.79M-object busy-PvE ASE save exports in 2.5 GB peak RSS instead of 7.1 GB, and a 217k-object ASA save in 307 MB instead of 853 MB (and slightly faster)
 
 ## Installation
 
@@ -98,6 +99,38 @@ print(f"Parse errors: {save.parse_error_count}")
 print(f"Is ASA: {save.is_asa}")
 ```
 
+### Low-memory parsing (large saves)
+
+By default every object's property block is parsed up front, which on a busy
+multi-GB PvE save means a multi-GB resident object graph. `lazy_properties=True`
+parses object headers eagerly but defers each property block until something
+reads it, then the export drivers evict it again once the record has been
+written:
+
+```python
+from arkparser import WorldSave, export_to_files
+from arkparser.common import get_map_config
+
+save = WorldSave.load("path/to/Fjordur.ark", lazy_properties=True)       # ASE
+save = WorldSave.load("path/to/TheIsland_WP.ark", lazy_properties=True)  # ASA
+export_to_files(save, "output/", get_map_config("fjordur.ark"))
+```
+
+Output is identical to eager mode (validated record-for-record). ASE retains
+the file reader and re-seeks each object's block on demand; ASA retains the
+SQLite connection (one held read transaction) and re-fetches row blobs by
+GUID. On ASA v14+ saves the export pipeline additionally uses partial decodes:
+a verified byte-exact skip walk that parses only the property names a record
+needs, with any out-of-whitelist read transparently upgrading to the full
+block.
+
+Measured load + export, same machine: a 1.8 GB Fjordur PvE ASE save (1.79M
+objects) drops from ~7.1 GB peak RSS / 348 s eager to ~2.5 GB / 287 s lazy; a
+233 MB ASA TheIsland save (217k objects) drops from 853 MB / 38 s eager to
+307 MB / ~36 s lazy. Property access on a lazy save is transparent: any
+`get_property_value` call materializes the block on demand, so all getters and
+exports work unchanged.
+
 ### Cryopod-stored Creatures
 
 Tamed creatures stored inside in-world cryopods aren't returned by `get_tamed_creatures()`. Iterate them via `WorldSave.iter_cryopod_creatures()`:
@@ -184,7 +217,7 @@ The legacy ASVExport.exe emitted only the visible 8 stats (`hp`, `stam`, `melee`
 | `isClone` | legacy | `bIsClone` or `bIsCloneDino` |
 | `tamedServer` | legacy | `TamedOnServerName` |
 | `uploadedServer` | legacy | `UploadedFromServerName` |
-| `maturation` | legacy | `str(int(BabyAge * 100))` — integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
+| `maturation` | legacy | `str(int(BabyAge * 100))`, integer maturation percent (a baby with no `BabyAge` is `0`, a newborn). Note: legacy emits the full float string (e.g. `"7.5131035"`); arkparser truncates to the integer percent. Semantically equal and the downstream consumer coerces to `int` either way. |
 | `traits` | legacy | `CreatureTraits` as a list of `{"trait": <class>}` objects (matches `ASVExport`'s shape, not a flat string list) |
 | `inventory` | legacy | items from `MyInventoryComponent.InventoryItems`. Each entry carries `itemId`, `qty`, `blueprint`, plus a full snake_case property dump flattened in at the top level (`id`, `rating`, `durability`, `quality`, `damage`, `armor`, `durability_max`, `hypo`, `hyper`, `clip_size`, `weight`, `crafter`, `crafter_tribe`, `skill_bonus`, `loaded_ammo`, `spoils_at`, `spoiled_at`, `c0`..`c5`, `egg_*`, etc). `item_stat_values` is unpacked into the universal 8-slot ARK map (slot 0 `gen_quality`, 1 `armor`, 2 `durability_max`, 3 `damage`, 4 `clip_size`, 5 `hypo`, 6 `weight`, 7 `hyperthermal_insulation`); raw uint16s scaled by the per-blueprint multiplier (which lives in the UE blueprint, not the save). Default / unset values are filtered (no `craft_queue=0`, `skin=-1`, `color_pre_skin=[0]*6`, NaN spoil timers, etc). When the item is a **cryopod / soultrap / vivarium / dinoball** with an embedded creature, the entry is enriched with `dino_id` (combined 64-bit id matching the corresponding `ASV_Tamed` record), `dino_creature` (species / class name), and `dino_name` (`TamedName` if set). Cryopods stored in containers (cryofridges, vaults, dedicated storage) get the same enrichment. |
 | `father_id`, `mother_id` | added | combined dino id from the first `DinoAncestors` entry (`null` when missing) |
@@ -247,7 +280,7 @@ ARK keeps player data in **two distinct places**, and the parser builds `ASV_Pla
 
 The export pipeline (`export_players`) loops over `save.profiles` (a list the caller assembles). For each entry:
 
-- If it's a `Profile` instance → `_player_from_profile` runs: fills core identity (name, gender, level, stats, tribe id, engram count, experience, active datetime from `LastLoginTime`). Leaves `lat`/`lon`/`ccc` as zero, `inventory` empty, pawn-state flags (`is_sleeping`, `is_dead`, `chibi_levels`, …) absent.
+- If it's a `Profile` instance → `_player_from_profile` runs: fills core identity (name, gender, level, stats, tribe id, engram count, experience, active datetime from `LastLoginTime`). It joins `PlayerDataID == LinkedPlayerDataID` against the world save's player pawns, so when a live pawn exists `lat`/`lon`/`ccc` come from that pawn's `.location`; players with no live pawn (dead / logged out) keep `lat`/`lon`/`ccc` at zero. `inventory` is empty and pawn-state flags (`is_sleeping`, `is_dead`, `chibi_levels`, …) absent (the profile carries neither).
 - Otherwise it's treated as a wrapped `(profile, objects)` pair pointing at an in-world `PlayerPawn` → `_player_from_object` runs: fills location, inventory, pawn-state flags, body/hair cosmetics, death timestamps, and `active` from `SavedLastTimeHadController`.
 
 For the richest output, hand `export_players` **both**, assemble a wrapper for each player that carries the `Profile` (for offline identity) *and* the in-world pawn (when present) so the merged record gets identity + live state. The current validation script only passes `Profile` instances, which is why fields like `is_sleeping` / `body_colors` / `current_weapon` show up empty in the validation output even though the parser supports them.
@@ -269,7 +302,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `netAddress` | legacy (now populated) | Last client IP ARK persisted (`SavedNetworkAddress` in profile `MyData`). Legacy ASVExport reads the same field (ContentPlayer.cs:157 ASE / :341 ASA). `""` when the profile lacks it (e.g. never-played placeholders). ASA stores a clean IPv4/IPv6 string; some ASE saves store an engine-truncated value (e.g. `"[2001"`) reproduced verbatim, matching legacy. |
 | `steamid`, `dataFile` | legacy | platform net id and `{steamid}.arkprofile` filename |
 | `active` | legacy (now populated) | ISO 8601 datetime of last login, converted from profile `LastLoginTime` or in-world pawn `SavedLastTimeHadController`. Legacy schema reserved the field but the old C# exporter only filled it for in-world pawns; the parser fills it for profiles too. `null` when neither source is present. |
-| `lat`, `lon`, `ccc` | legacy (now populated) | In-world position when the record was built from a `PlayerPawn` GameObject. Records sourced from profile / cluster files keep the legacy `0` / `"0 0 0"` placeholders since profiles carry no world position. |
+| `lat`, `lon`, `ccc` | legacy (now populated) | In-world position. Built directly from a `PlayerPawn` GameObject when one is passed, and for profile-sourced records the parser joins `PlayerDataID == LinkedPlayerDataID` to the world save's player pawns and pulls the matched pawn's `.location`. Records whose player has no live pawn (dead / logged out / cluster-only) keep the legacy `0` / `"0 0 0"` placeholders since there is no world position to report. |
 | `inventory` | legacy (now populated) | Items from the pawn's `MyInventoryComponent` when built from an in-world pawn. Empty list otherwise. |
 | `engram_points` | added | `TotalEngramPoints` |
 | `experience` | added | `ExperiencePoints` (status), integer |
@@ -288,17 +321,17 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 
 #### `ASV_Tribes` schema
 
-`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam` — deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
+`ASV_Tribes` (and `ASV_TribeLogs` / `ASV_Players`, which iterate the same list) is a **superset** of the loaded `.arktribe` files, mirroring legacy `ContentContainer`: it seeds two sentinels (`[ASV Unclaimed]` id `2000000000`, `[ASV Abandoned]` id `-2147483648`), adds every file-backed tribe, then synthesizes a stub tribe for each player profile not already in a tribe (`Tribe of <name>`, id = `PlayerDataID`), each distinct structure `TargetingTeam` (`>= 50000`), and each distinct in-world tame `TargetingTeam`, deduped by id. Cross-server tribes that exist **only** in cluster cloud-inventory files (no map presence) are not synthesized.
 
 | Field | Origin | Source / formula |
 |---|---|---|
 | `tribeid` | legacy | `TribeID` / parser `Tribe.tribe_id`, or a synthesized stub/sentinel id (see above) |
 | `tribe` | legacy | tribe name (file `TribeName`; for stubs, `OwnerName`/`TamerString`; or `Tribe of <character>` for a solo) |
-| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count` — not the raw `.arktribe` member count |
+| `players` | legacy | count of players **allocated** to this tribe (profiles whose explicit team / membership / solo id resolves here, plus member back-fill), matching legacy `Players.Count`, not the raw `.arktribe` member count |
 | `members` | legacy | list of `{ign, lvl, playerid, playername, steamid}` built from the allocated players. `lvl` and `steamid` populate from the matching `.arkprofile`; members with no profile (back-filled from the tribe's member list) carry `lvl=0`, `steamid=""`. |
 | `tames`, `structures` | legacy | counts derived from `WorldSave` (creatures + structures whose `TargetingTeam` matches) |
 | `uploadedTames` | legacy | reserved (currently `0`) |
-| `active` | legacy | ISO 8601 datetime of the most recent tribe log entry, converted from in-game "Day N, HH:MM:SS" via the save's anchor. `null` when no parseable log entries or anchors are missing. |
+| `active` | legacy | ISO 8601 datetime, legacy `ContentTribe.LastActive`: max of the tribe file's write time (`.arktribe` mtime; world-save mtime for in-save tribes) and the allocated members' last-active times (`LastLoginTime` via the save anchor), discarding future values. `null` when no past candidate exists. Tribe-log "Day N" stamps are game-calendar time and are deliberately not used. |
 | `dataFile` | legacy | `"{tribeid}.arktribe"` filename pattern. |
 | `owner_id` | added | `OwnerPlayerDataID` / parser `Tribe.owner_player_id` |
 | `owner_name` | added | `OwnerPlayerName` (object form only; parser-tribe form has no equivalent) |
@@ -319,7 +352,7 @@ For the richest output, hand `export_players` **both**, assemble a wrapper for e
 | `id` | added | `GameObject.id`, internal numeric identifier. Cross-references the values in other records' `linked_structures` / `saddle_structures` / `attached_dino_id` lists. |
 | `tribeid` | legacy | `TargetingTeam` for player-owned structures (`>= 50000`). Unowned structures fall to the synthetic `[ASV Abandoned]` tribe id `-2147483648` (legacy `int.MinValue`), mirroring `ContentContainer.cs`. |
 | `tribe` | legacy | resolved owning-tribe name: the loaded `.arktribe` `TribeName`, else the structure's `OwnerName` / `TamerString`; `"[ASV Abandoned]"` for unowned. (Legacy emits the resolved tribe name, not the raw `OwnerName`.) |
-| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded — surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
+| `struct` | legacy | `GameObject.class_name`. Unowned map elements / crates / debug actors (`Button_*`, `*Vein_*`, `*Nest_*`, `*Beaver*`, `BeeHive_C`, `ArtifactCrate_*`, `TributeTerminal_*`, `SupplyCrate_*`) are excluded; surfaced via `ASV_MapStructures` instead, matching legacy's abandoned-structure filter. |
 | `name` | legacy | `BoxName` (empty when it matches the class name, mirroring legacy's no-rename strip) |
 | `locked` | legacy | `bIsPinLocked` or `bIsLocked` |
 | `created` | legacy (richer) | ISO 8601 datetime with the local TZ of the parser machine, computed `save.file_mtime + (OriginalCreationTime - game_time)` (mirrors legacy `ContentContainer.GetApproxDateTimeOf`). `null` when the anchors are missing. |
@@ -378,7 +411,7 @@ data = export_all(save, map_config, cluster="path/to/cluster")
 #                       .arkprofile filename stem.
 ```
 
-Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem — the Steam id on ASE, the hex platform UUID on ASA — so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths — which sets `Profile.source_path` — is required for the ASA splice.)
+Cluster items are spliced into the owning player's `inventory`; no separate `ASV_ClusterItems` file is emitted. The match works because every cluster file and the player's `.arkprofile` share a stem (the Steam id on ASE, the hex platform UUID on ASA) so the join is keyed on the profile's source filename stem. (`Profile.unique_id` equals that stem only on ASE; on ASA it is the numeric net id, not the UUID filename, so loading profiles from real file paths (which sets `Profile.source_path`) is required for the ASA splice.)
 
 Pre-loaded `CloudInventory` instances also work (`cluster=[inv1, inv2, ...]`).
 
@@ -496,7 +529,9 @@ Key methods:
 
 | Method | Returns | Description |
 |---|---|---|
-| `load(source, load_properties=True, max_objects=None)` | `WorldSave` | Load and parse a world save |
+| `load(source, load_properties=True, max_objects=None, lazy_properties=False)` | `WorldSave` | Load and parse a world save (`lazy_properties`: on-demand property parsing, ASE + ASA, see Quick Start) |
+| `materialize_object(obj, names=None)` | `None` | Parse one lazy object's deferred property block (called automatically on property access; `names` is an ASA v14+ partial-decode hint) |
+| `evict_materialized()` | `int` | Release every property block materialized since the last call (lazy saves; no-op eager) |
 | `get_creatures()` | `list[GameObject]` | All creatures |
 | `get_tamed_creatures()` / `get_wild_creatures()` | `list[GameObject]` | Filtered creature sets |
 | `get_structures()` | `list[GameObject]` | Tribe-owned placed structures |
@@ -524,7 +559,7 @@ Key methods:
 | `properties` | `list[Property]` | Parsed property list |
 | `parent` / `components` | `GameObject \| None` / `dict[str, GameObject]` | Relationships |
 
-Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`.
+Lookup helpers: `get_property(name, index=None)`, `get_property_value(name, default=None, index=None)`, `get_properties_by_name(name)`, `has_property(name)`, `to_dict()`. On lazy saves every helper materializes the deferred property block transparently; `evict_properties()` releases it again (re-access re-parses, so eviction is always safe).
 
 #### GameObjectContainer (`arkparser.GameObjectContainer`)
 
@@ -640,6 +675,11 @@ python -m pytest tests/ -v
 
 Tests live in `tests/`. Byte-level layout tests (`test_v13_property_layouts.py`, `test_binary_reader_layouts.py`) pin the canonical v13/v14 property body byte sequences with no file-fixture dependency. Integration tests (`test_world_save.py`, `test_export.py`, etc.) skip cleanly when their referenced save files are not present under `references/examples/`.
 
+Two suites guard export output against drift:
+
+- **Golden manifests** (`tests/test_golden_exports.py` + `tests/golden/*.json`): a per-map fingerprint of `export_all` (record count, field-key union, order-independent sha256) for 17 real saves (10 ASE map dumps + 7 ASA). Any change to parsing or export that alters output fails the matching map. Regenerate intentionally with `ARKPARSER_UPDATE_GOLDEN=1`.
+- **Lazy parity** (`tests/test_lazy_properties.py`): proves `lazy_properties=True` yields property-for-property identical objects (ASE and ASA), that evict/re-materialize round-trips, and that full lazy exports match the committed eager goldens. The ASA partial-decode skip walk is additionally verified byte-exact against the full parser over every object of every local ASA fixture by `references/scripts/verify_partial_walk.py`.
+
 ## Credits
 
 Built by reverse-engineering ARK save formats with heavy reference to [ASV (Ark Save Visualizer)](https://github.com/miragedmuk/ASV) by **miragedmuk**. The C# implementation in ASV is the primary reference for porting binary parsing logic to Python.

{arkparser-0.5.4 → arkparser-0.7.0}/arkparser/__init__.py

@@ -1,98 +1,98 @@
-"""
-ARK Save Parser - Parse ARK: Survival Evolved/Ascended save files.
-
-This package provides tools to parse various ARK save file formats:
-
-- Profile: Player profile data (.arkprofile)
-- Tribe: Tribe data (.arktribe)
-- CloudInventory: Obelisk/cloud inventory data (no extension)
-- WorldSave: World save data (.ark), auto-detects ASE binary and ASA SQLite
-
-Supports both ASE (ARK: Survival Evolved) and ASA (ARK: Survival Ascended)
-formats with automatic detection.
-
-Example usage:
-    >>> from arkparser import Profile, Tribe, CloudInventory, WorldSave
-    >>> profile = Profile.load("path/to/profile.arkprofile")
-    >>> tribe = Tribe.load("path/to/tribe.arktribe")
-    >>> inv = CloudInventory.load("path/to/obelisk_file")
-    >>> save = WorldSave.load("path/to/TheIsland.ark")    # ASE
-    >>> save = WorldSave.load("path/to/Extinction_WP.ark")  # ASA
-"""
-
-from arkparser.common.exceptions import ArkParseError
-from arkparser.common.map_config import MapConfig, get_map_config, get_map_config_by_name
-from arkparser.common.version_detection import (
-    ArkFileFormat,
-    ArkFileType,
-    detect_file_type,
-    detect_format,
-)
-from arkparser.data_models import (
-    CryopodCreature,
-    DinoStats,
-    UploadedCreature,
-    UploadedItem,
-)
-from arkparser.export import (
-    export_all,
-    export_cloud_inventory,
-    export_cluster_items,
-    export_cluster_uploads,
-    export_map_structures,
-    export_players,
-    export_structures,
-    export_tamed,
-    export_to_files,
-    export_tribe_logs,
-    export_tribes,
-    export_wild,
-)
-from arkparser.files import CloudInventory, Profile, Tribe, WorldSave
-from arkparser.game_objects import GameObject, GameObjectContainer, LocationData
-
-# Convenience alias - users may know this as "Obelisk" from the game
-Obelisk = CloudInventory
-
-__all__ = [
-    # File parsers
-    "Profile",
-    "Tribe",
-    "CloudInventory",
-    "Obelisk",
-    "WorldSave",
-    # Cloud-inventory data models
-    "UploadedCreature",
-    "UploadedItem",
-    "CryopodCreature",
-    "DinoStats",
-    # Game objects
-    "GameObject",
-    "GameObjectContainer",
-    "LocationData",
-    # Map config
-    "MapConfig",
-    "get_map_config",
-    "get_map_config_by_name",
-    # Export
-    "export_all",
-    "export_tamed",
-    "export_wild",
-    "export_players",
-    "export_tribes",
-    "export_structures",
-    "export_tribe_logs",
-    "export_map_structures",
-    "export_cluster_uploads",
-    "export_cluster_items",
-    "export_cloud_inventory",
-    "export_to_files",
-    # Utilities
-    "detect_format",
-    "detect_file_type",
-    "ArkFileFormat",
-    "ArkFileType",
-    "ArkParseError",
-]
-
-__version__ = "0.5.4"
+"""
+ARK Save Parser - Parse ARK: Survival Evolved/Ascended save files.
+
+This package provides tools to parse various ARK save file formats:
+
+- Profile: Player profile data (.arkprofile)
+- Tribe: Tribe data (.arktribe)
+- CloudInventory: Obelisk/cloud inventory data (no extension)
+- WorldSave: World save data (.ark), auto-detects ASE binary and ASA SQLite
+
+Supports both ASE (ARK: Survival Evolved) and ASA (ARK: Survival Ascended)
+formats with automatic detection.
+
+Example usage:
+    >>> from arkparser import Profile, Tribe, CloudInventory, WorldSave
+    >>> profile = Profile.load("path/to/profile.arkprofile")
+    >>> tribe = Tribe.load("path/to/tribe.arktribe")
+    >>> inv = CloudInventory.load("path/to/obelisk_file")
+    >>> save = WorldSave.load("path/to/TheIsland.ark")    # ASE
+    >>> save = WorldSave.load("path/to/Extinction_WP.ark")  # ASA
+"""
+
+from arkparser.common.exceptions import ArkParseError
+from arkparser.common.map_config import MapConfig, get_map_config, get_map_config_by_name
+from arkparser.common.version_detection import (
+    ArkFileFormat,
+    ArkFileType,
+    detect_file_type,
+    detect_format,
+)
+from arkparser.data_models import (
+    CryopodCreature,
+    DinoStats,
+    UploadedCreature,
+    UploadedItem,
+)
+from arkparser.export import (
+    export_all,
+    export_cloud_inventory,
+    export_cluster_items,
+    export_cluster_uploads,
+    export_map_structures,
+    export_players,
+    export_structures,
+    export_tamed,
+    export_to_files,
+    export_tribe_logs,
+    export_tribes,
+    export_wild,
+)
+from arkparser.files import CloudInventory, Profile, Tribe, WorldSave
+from arkparser.game_objects import GameObject, GameObjectContainer, LocationData
+
+# Convenience alias - users may know this as "Obelisk" from the game
+Obelisk = CloudInventory
+
+__all__ = [
+    # File parsers
+    "Profile",
+    "Tribe",
+    "CloudInventory",
+    "Obelisk",
+    "WorldSave",
+    # Cloud-inventory data models
+    "UploadedCreature",
+    "UploadedItem",
+    "CryopodCreature",
+    "DinoStats",
+    # Game objects
+    "GameObject",
+    "GameObjectContainer",
+    "LocationData",
+    # Map config
+    "MapConfig",
+    "get_map_config",
+    "get_map_config_by_name",
+    # Export
+    "export_all",
+    "export_tamed",
+    "export_wild",
+    "export_players",
+    "export_tribes",
+    "export_structures",
+    "export_tribe_logs",
+    "export_map_structures",
+    "export_cluster_uploads",
+    "export_cluster_items",
+    "export_cloud_inventory",
+    "export_to_files",
+    # Utilities
+    "detect_format",
+    "detect_file_type",
+    "ArkFileFormat",
+    "ArkFileType",
+    "ArkParseError",
+]
+
+__version__ = "0.7.0"