crowdypy 0.29.0.dev1__tar.gz → 0.30.0.dev1__tar.gz

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/AGENTS.md

@@ -246,6 +246,13 @@ The replication system saturates the local network stack, so the binding
   16x16x16 one-byte grid cannot hold goes to `ChunkData::overlay`.
 - Do not write voxel types or states into the dense grid from Python. That bypasses the
   overlay, and the echo matching of local edits.
+- Since 0.30.0 (CrowdyCPP 0.82.0) the native store paces and undoes edits; the binding turns its
+  own I/O thread, reloads and reconnect catch-up off (`ioThread`, `reloadReplacedChunks`,
+  `catchUpOnReconnect` false), because the facade reads the durable store itself and runs
+  `catch_up` and the reload of a replaced chunk. A refused, unsent or timed-out edit reaches the
+  facade as an `EDIT_REFUSED` event row (`a` packs x/y/z as 16-bit fields, `c` the result,
+  `restored` and the error code, `epochMs` the edit id). Read a chunk through `forEachState`,
+  `forEachOverlay` and `grid()`: `ChunkData` keeps no public maps any more.
 - The voice helpers (`native/voice_binding.cpp`) bind CrowdyCPP's `voice_frames.hpp`.
   `crowdypy.media` checks every argument as CrowdyJS does first, because CrowdyCPP clamps the
   options CrowdyJS refuses.

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/MIGRATION.md

@@ -3,6 +3,31 @@
 CrowdyPy is pre-1.0. Within a minor line, patch releases keep source compatibility; each
 new minor may change the API. Read the section for every minor you skip.
 
+## 0.30.0
+
+CrowdyJS 18.30.0 and CrowdyCPP 0.82.0 (the ChunkStore rework). Box reads and `since` need ck-api
+v2.67.0. **Breaking** for code that keeps `set_voxel()`'s return value or relies on a realtime
+edit being written back.
+
+- **`ensure_around` reads only what the store does not hold**, as boxes (`lowChunk`/`highChunk`)
+  read in pages of the size the server applied; a chunk two pages return is applied once
+  (`duplicate_page_chunks`). A step along one axis at range 8 reads the 289 chunks it uncovers.
+  Range 0 reads one chunk. `CachedChunk` has `voxel_states_truncated` and `stored_at`.
+- **`set_voxel()` returns an edit id, is paced, and is not written back.** At most
+  `max_edits_in_flight` (128) are unanswered and `max_edits_per_second` (250) go out a second
+  (`chunk_options`); the rest wait, already shown (`queued_edits`, `edits_in_flight`). The server
+  records realtime edits, so the chunk is no longer queued for write-back (it was, whole).
+- **A refused edit is undone.** One the server refuses, the connection does not take, or whose
+  echo does not come back within `edit_timeout_ms` (10 s) puts the voxel back and reaches
+  `on_edit_refused` as a `VoxelEditRefusal` (`reason` `refused`, `not_sent` or `timeout`,
+  `restored`, `error_code`, `edit_id`).
+- **A replaced chunk is read again** (`chunk_changed`, `reload_replaced_chunks`), and a seed queued
+  for write-back gives way to it. **After the connection comes back**, `catch_up(since)` reads the
+  last area's rewritten chunks and those with edits since 30 s before the drop
+  (`catch_up_on_reconnect`).
+- A hydration replaces the chunk's states and overlay with the stored copy's and keeps this
+  client's unanswered edits over it (the native store's rule since CrowdyCPP 0.82.0).
+
 ## 0.29.0
 
 CrowdyJS 18.29.0 and CrowdyCPP 0.81.0. It needs nothing new on the tier; the bound frames and

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: crowdypy
-Version: 0.29.0.dev1
+Version: 0.30.0.dev1
 Summary: Official Python SDK for Crowded Kingdoms: typed GraphQL clients and a native UDP replication core
 Keywords: crowded-kingdoms,multiplayer,replication,graphql,sdk,udp
 Author: Crowded Kingdoms
@@ -51,6 +51,8 @@ CrowdyPy follows the [CrowdyJS](https://github.com/CrowdedKingdoms/CrowdyJS) API
 client, bound with [nanobind](https://github.com/wjakob/nanobind) and shipped inside the
 wheel. Python never touches a datagram.
 
+**v0.30.0: the ChunkStore rework.** `ensure_around` reads only what is missing, as boxes, paged; `set_voxel` is paced and returns an edit id, and an edit the server refuses or never echoes is undone (`on_edit_refused`); a replaced chunk is read again and a reconnect catches up; realtime edits are no longer written back. CrowdyJS 18.30.0 and CrowdyCPP 0.82.0. See [MIGRATION.md](MIGRATION.md).
+
 **v0.29.0: reconnect commands verify, bound control frames, and `chunk_changed`.** A server's `COMMAND_RECONNECT` verifies again; against replication servers v0.45.0 the reconnect and release frames carry a per-assignment nonce under their HMAC, so one captured earlier is refused (`stats()["stale_bound_control"]`); `chunk_changed` reports a chunk a hub's or mod's `world.set_chunk` replaced. CrowdyJS 18.29.0 and CrowdyCPP 0.81.0. See [MIGRATION.md](MIGRATION.md).
 
 **v0.28.0: region reads.** `chunks.by_distance()` and `voxels.list_by_distance()` take a box (`lowChunk`, `highChunk`) as well as a cube, pages are stable and answer the paging applied, `since` filters a chunk read, `voxels.list()` pages, and `game_apps.nearby_grids()` returns your own `permissionKeys`. Needs ck-api v2.67.0. CrowdyJS 18.28.0 and CrowdyCPP 0.80.0. See [MIGRATION.md](MIGRATION.md).

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/README.md

@@ -11,6 +11,8 @@ CrowdyPy follows the [CrowdyJS](https://github.com/CrowdedKingdoms/CrowdyJS) API
 client, bound with [nanobind](https://github.com/wjakob/nanobind) and shipped inside the
 wheel. Python never touches a datagram.
 
+**v0.30.0: the ChunkStore rework.** `ensure_around` reads only what is missing, as boxes, paged; `set_voxel` is paced and returns an edit id, and an edit the server refuses or never echoes is undone (`on_edit_refused`); a replaced chunk is read again and a reconnect catches up; realtime edits are no longer written back. CrowdyJS 18.30.0 and CrowdyCPP 0.82.0. See [MIGRATION.md](MIGRATION.md).
+
 **v0.29.0: reconnect commands verify, bound control frames, and `chunk_changed`.** A server's `COMMAND_RECONNECT` verifies again; against replication servers v0.45.0 the reconnect and release frames carry a per-assignment nonce under their HMAC, so one captured earlier is refused (`stats()["stale_bound_control"]`); `chunk_changed` reports a chunk a hub's or mod's `world.set_chunk` replaced. CrowdyJS 18.29.0 and CrowdyCPP 0.81.0. See [MIGRATION.md](MIGRATION.md).
 
 **v0.28.0: region reads.** `chunks.by_distance()` and `voxels.list_by_distance()` take a box (`lowChunk`, `highChunk`) as well as a cube, pages are stable and answer the paging applied, `since` filters a chunk read, `voxels.list()` pages, and `game_apps.nearby_grids()` returns your own `permissionKeys`. Needs ck-api v2.67.0. CrowdyJS 18.28.0 and CrowdyCPP 0.80.0. See [MIGRATION.md](MIGRATION.md).

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/docs/compatibility.md

@@ -5,7 +5,7 @@ are in `pyproject.toml` (`[project].version`, `[tool.crowdypy.crowdyjs]`,
 `[tool.crowdypy.crowdycpp]`); `tests/unit/test_release_versions.py` refuses this page when
 its version disagrees.
 
-- CrowdyPy `0.29.0`
+- CrowdyPy `0.30.0`
 - API surface: CrowdyJS (the pinned version and commit in `pyproject.toml`); the generated
   `docs/parity-matrix.md` lists every covered method, every native equivalent and every
   browser exclusion with its reason. The gate is strict: there are no portable gaps, and a

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/docs/parity-matrix.md

@@ -4,14 +4,14 @@ Generated by `tools/parity/parity.py`; do not edit by hand. Classifications live
 
 ## Target
 
-- CrowdyJS: `18.29.0` at `f698cfe05fda587d30195531ac896325f7d43ea2`
-- CrowdyCPP (native core): `0.81.0` at `6de05acdd79135d2b45a74c5abfc6777fc2de3fa`
+- CrowdyJS: `18.30.0` at `4840d76029ffb15a6de6232dab9336f2e9ab6514`
+- CrowdyCPP (native core): `0.82.0` at `c6875412bcea97120b4540b80d9b34504c29acf9`
 - schema.gql identical to CrowdyJS's: yes
 - Gate mode: strict portable parity (every portable gap is release-blocking)
 
 ## Summary
 
-- Covered: 1475
+- Covered: 1480
 - Portable gap: 0
 - Native equivalent: 47
 - Browser exclusion: 128
@@ -792,6 +792,9 @@ Generated by `tools/parity/parity.py`; do not edit by hand. Classifications live
 | `ChannelsAPI.unban` | covered |
 | `ChannelsAPI.update` | covered |
 | `ChannelsAPI.updateRole` | covered |
+| `ChunkStore.catchUp` | covered |
+| `ChunkStore.duplicatePageChunks` | covered |
+| `ChunkStore.editsInFlight` | covered |
 | `ChunkStore.ensureAround` | covered |
 | `ChunkStore.flush` | covered |
 | `ChunkStore.get` | covered |
@@ -799,9 +802,11 @@ Generated by `tools/parity/parity.py`; do not edit by hand. Classifications live
 | `ChunkStore.list` | covered |
 | `ChunkStore.markDirty` | covered |
 | `ChunkStore.onChunkChanged` | covered |
+| `ChunkStore.onEditRefused` | covered |
 | `ChunkStore.onWriteBackFailed` | covered |
 | `ChunkStore.pendingWriteBacks` | covered |
 | `ChunkStore.pruneBeyond` | covered |
+| `ChunkStore.queuedEdits` | covered |
 | `ChunkStore.revision` | covered |
 | `ChunkStore.seed` | covered |
 | `ChunkStore.setVoxel` | covered |

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/native/session_binding.cpp

@@ -51,6 +51,10 @@ struct SessionEvent {
     kDirectMessage,
     kEvent,
     kActorLeft,
+    /// An edit undone or not sent (CrowdyCPP 0.82.0): `chunk`; `a` the voxel (x, y, z as
+    /// 16-bit fields); `b` its type; `c` the result | restored << 8 | error code << 16;
+    /// `epochMs` the edit id.
+    kEditRefused,
   };
   int kind = 0;
   std::string lane;  // "" for the default lane
@@ -141,7 +145,8 @@ class PySession {
   PySession(PyConnection& conn, const std::string& appId, const std::string& actorUuid,
             int sendHz, std::int64_t keyframeIntervalMs, std::int64_t heartbeatIntervalMs,
             int distance, int decay, std::int64_t staleAfterMs, int historySize,
-            std::int64_t reapIntervalMs, int voxelSendDistance)
+            std::int64_t reapIntervalMs, int voxelSendDistance, int maxEditsInFlight,
+            int maxEditsPerSecond, std::int64_t editTimeoutMs)
       : conn_(conn) {
     session::WorldSessionConfig config;
     config.appId = appId;
@@ -155,6 +160,13 @@ class PySession {
     config.actors.historySize = historySize;
     config.chunks.writeBackIntervalMs = 0;  // durable write-back is the facade's
     config.chunks.voxelSendDistance = distance_arg(voxelSendDistance);
+    // The facade loads, re-reads and catches up itself; its staged source answers inline.
+    config.chunks.ioThread = false;
+    config.chunks.reloadReplacedChunks = false;
+    config.chunks.catchUpOnReconnect = false;
+    config.chunks.maxEditsInFlight = maxEditsInFlight;
+    config.chunks.maxEditsPerSecond = maxEditsPerSecond;
+    config.chunks.editTimeoutMs = editTimeoutMs;
     config.hostHeartbeatIntervalMs = 0;     // host election is the facade's
     config.reapIntervalMs = reapIntervalMs;
     auto forward = [this](const replication::SpatialNotification& n) {
@@ -194,6 +206,20 @@ class PySession {
     }
     conn_.attachSession(true);
     watchLane("");
+    session_->chunks().onEditRefused([this](const session::EditOutcome& o) {
+      SessionEvent e;
+      e.kind = SessionEvent::kEditRefused;
+      e.hasUuid = false;
+      e.chunk = o.coord;
+      e.a = (static_cast<std::int64_t>(static_cast<std::uint16_t>(o.x)) << 32) |
+            (static_cast<std::int64_t>(static_cast<std::uint16_t>(o.y)) << 16) |
+            static_cast<std::int64_t>(static_cast<std::uint16_t>(o.z));
+      e.b = o.voxelType;
+      e.c = static_cast<std::int64_t>(o.result) | (o.restored ? 1 << 8 : 0) |
+            (static_cast<std::int64_t>(o.errorCode) << 16);
+      e.epochMs = static_cast<std::int64_t>(o.editId);
+      events_.push_back(std::move(e));
+    });
     session_->chunks().onChunkChanged([this](const session::ChunkData& chunk) {
       if (!watchChunks_) return;
       SessionEvent e;
@@ -464,14 +490,15 @@ class PySession {
     const session::ChunkData* c = live().chunks().find(coord_of(x, y, z));
     if (!c) return std::nullopt;
     nb::dict states;
-    for (const auto& [index, state] : c->voxelStates)
+    c->forEachState([&](int index, const session::VoxelStateView& state) {
       states[nb::int_(index)] = nb::make_tuple(state.voxelType, blob(state.state));
+    });
     nb::list overlay;
-    for (const auto& [key, wide] : c->overlay)
-      overlay.append(
-          nb::make_tuple(wide.x, wide.y, wide.z, wide.voxel.voxelType, blob(wide.voxel.state)));
+    c->forEachOverlay([&](const session::OverlayVoxel& wide) {
+      overlay.append(nb::make_tuple(wide.x, wide.y, wide.z, wide.voxelType, blob(wide.state)));
+    });
     return nb::make_tuple(
-        nb::bytes(reinterpret_cast<const char*>(c->voxels.data()), c->voxels.size()), states,
+        nb::bytes(reinterpret_cast<const char*>(c->grid()), session::kChunkVolume), states,
         c->hydratedAtMs, overlay);
   }
 
@@ -519,25 +546,40 @@ class PySession {
   std::optional<nb::tuple> voxelStateAt(std::int64_t x, std::int64_t y, std::int64_t z, int vx,
                                         int vy, int vz) {
     std::lock_guard lock(conn_.consumerMutex());
-    const session::VoxelState* s = live().chunks().voxelStateAt(coord_of(x, y, z), vx, vy, vz);
+    const auto s = live().chunks().voxelStateAt(coord_of(x, y, z), vx, vy, vz);
     if (!s) return std::nullopt;
     return nb::make_tuple(s->voxelType, blob(s->state));
   }
 
-  int setVoxel(std::int64_t x, std::int64_t y, std::int64_t z, int vx, int vy, int vz,
-               int voxelType, nb::handle state, bool optimistic) {
+  /// Returns (edit id, the sequence number it went out with, or -1 while it waits its turn).
+  nb::tuple setVoxel(std::int64_t x, std::int64_t y, std::int64_t z, int vx, int vy, int vz,
+                     int voxelType, nb::handle state, bool optimistic) {
     BufferView body(state);
     const std::int16_t type = i16_arg(voxelType, "voxel type");
-    Result<std::uint8_t> result = Errc::InvalidArgument;
+    Result<std::uint64_t> result = Errc::InvalidArgument;
+    int sequence = -1;
     {
       nb::gil_scoped_release release;
       std::lock_guard lock(conn_.consumerMutex());
       auto& s = live();
       result = s.chunks().setVoxel(coord_of(x, y, z), vx, vy, vz, type, body.bytes(), s.actorUuid(),
                                    optimistic);
-      if (result.ok()) s.errors().recordSend(result.value(), session::SendKind::VoxelUpdate, s.actorUuid());
+      if (result.ok()) {
+        if (auto seq = s.chunks().sequenceOf(result.value())) sequence = *seq;
+      }
     }
-    return sent(result, "setVoxel");
+    if (!result.ok()) raise(result.error(), "setVoxel");
+    return nb::make_tuple(result.value(), sequence);
+  }
+
+  std::size_t editsInFlight() {
+    std::lock_guard lock(conn_.consumerMutex());
+    return live().chunks().editsInFlight();
+  }
+
+  std::size_t queuedEdits() {
+    std::lock_guard lock(conn_.consumerMutex());
+    return live().chunks().queuedEdits();
   }
 
   /// Insert a chunk the facade loaded from the durable store, or generated.
@@ -736,6 +778,7 @@ void register_session(nb::module_& m) {
   m.attr("DIRECT_MESSAGE") = static_cast<int>(SessionEvent::kDirectMessage);
   m.attr("EVENT") = static_cast<int>(SessionEvent::kEvent);
   m.attr("ACTOR_LEFT") = static_cast<int>(SessionEvent::kActorLeft);
+  m.attr("EDIT_REFUSED") = static_cast<int>(SessionEvent::kEditRefused);
   m.attr("MAX_STATE_BYTES") = session::kMaxStateBytes;
   m.attr("CHUNK_VOLUME") = session::kChunkVolume;
 
@@ -762,13 +805,15 @@ void register_session(nb::module_& m) {
 
   nb::class_<PySession>(m, "WorldSession")
       .def(nb::init<PyConnection&, const std::string&, const std::string&, int, std::int64_t,
-                    std::int64_t, int, int, std::int64_t, int, std::int64_t, int>(),
+                    std::int64_t, int, int, std::int64_t, int, std::int64_t, int, int, int,
+                    std::int64_t>(),
            nb::arg("connection"), nb::arg("app_id"), nb::arg("actor_uuid") = "",
            nb::arg("send_hz") = 5, nb::arg("keyframe_interval_ms") = 3000,
            nb::arg("heartbeat_interval_ms") = 2000, nb::arg("distance") = 8, nb::arg("decay") = 1,
            nb::arg("stale_after_ms") = 12000, nb::arg("history_size") = 2,
            nb::arg("reap_interval_ms") = 1000, nb::arg("voxel_send_distance") = 8,
-           nb::keep_alive<1, 2>())
+           nb::arg("max_edits_in_flight") = 128, nb::arg("max_edits_per_second") = 250,
+           nb::arg("edit_timeout_ms") = 10000, nb::keep_alive<1, 2>())
       .def("close", &PySession::close)
       .def("tick", &PySession::tick)
       .def("drain_events", &PySession::drainEvents)
@@ -815,6 +860,8 @@ void register_session(nb::module_& m) {
            nb::arg("distance"))
       .def("chunk_revision", &PySession::chunkRevision)
       .def("chunk_count", &PySession::chunkCount)
+      .def("edits_in_flight", &PySession::editsInFlight)
+      .def("queued_edits", &PySession::queuedEdits)
       .def("recent_errors", &PySession::recentErrors, nb::arg("limit") = SIZE_MAX)
       .def("error_total", &PySession::errorTotal)
       .def("clear_errors", &PySession::clearErrors)

{crowdypy-0.29.0.dev1 → crowdypy-0.30.0.dev1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "scikit_build_core.build"
 
 [project]
 name = "crowdypy"
-version = "0.29.0.dev1"
+version = "0.30.0.dev1"
 description = "Official Python SDK for Crowded Kingdoms: typed GraphQL clients and a native UDP replication core"
 readme = "README.md"
 license = "MIT AND Apache-2.0"
@@ -119,14 +119,14 @@ environment = { OPENSSL_ROOT_DIR = "C:/vcpkg/installed/x64-windows-static-md" }
 # --------------------------------------------------------------------------------------
 
 [tool.crowdypy.crowdyjs]
-version = "18.29.0"
-commit = "f698cfe05fda587d30195531ac896325f7d43ea2"
+version = "18.30.0"
+commit = "4840d76029ffb15a6de6232dab9336f2e9ab6514"
 
 [tool.crowdypy.crowdycpp]
-version = "0.81.0"
-tag = "dev/v0.81.0"
-commit = "6de05acdd79135d2b45a74c5abfc6777fc2de3fa"
-tree-sha256 = "d002d2175f4461232dafb819812c56bbd6e602b26ff5d561405d7da375ecdad8"
+version = "0.82.0"
+tag = "dev/v0.82.0"
+commit = "c6875412bcea97120b4540b80d9b34504c29acf9"
+tree-sha256 = "44dca0b0bd34d575808e5bf7a1ca6944ce779a790b9187894761541623a0f3ec"
 
 # schema.gql and operations/ are CrowdyJS's, copied from the commit pinned above by
 # scripts/schema_sync.py, which records these hashes and re-derives them in CI.