picopyn 2.0.0.dev1__tar.gz → 3.0.0__tar.gz

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/AUTHORS.md

@@ -2,4 +2,5 @@
 
 - Asya Lomakina <a.lomakina@picodata.io>
 - Igor Kuznetsov <i.kuznetsov@picodata.io>
-- Dmitriy Koltsov <dkoltsov@picodata.io>
+- Dmitriy Koltsov <dkoltsov@picodata.io>
+- Daniil Gorlyakov <d.gorlyakov@picodata.io>

{picopyn-2.0.0.dev1/picopyn.egg-info → picopyn-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: picopyn
-Version: 2.0.0.dev1
+Version: 3.0.0
 Summary: Python connector for working with the distributed Picodata database.
 Author-email: Asya Lomakina <a.lomakina@picodata.io>, Igor Kuznetsov <i.kuznetsov@picodata.io>, Dmitriy Koltsov <dkoltsov@picodata.io>, Daniil Gorlyakov <d.gorlyakov@picodata.io>
 License-Expression: BSD-2-Clause
@@ -55,17 +55,17 @@ Dynamic: license-file
 | >=25.2.1, <25.4.4 | 0.1.1             |
 | >=25.4.4, <25.5.1 | 0.2.0             |
 | >=25.5.1, <26.1.x | 1.0.0             |
-| >=26.1.1, <26.2.x | 2.0.0             |
+| >=26.1.1, <26.2.x | >=2.0.0, <4.0.0   |
 | 26.2.x            | ⚠️ Unreleased yet |
 
 ## Features
 * Connection pooling with configurable pool size
 * Optional automatic node discovery
 * Pluggable load-balancing strategies
+* Shard-aware routing
+* Automatic pool membership reconciliation, including rebalancing a full pool across the nodes
 * Asynchronous API based on [asyncpg](https://github.com/MagicStack/asyncpg)
-    * (async only) Shard-aware routing
-    * (async only) Automatic pool membership reconciliation
-* Synchronous [DPAPI-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
+* Synchronous [DB-API 2.0-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
 
 ## Navigation
 * [Installation](#installation)
@@ -97,24 +97,24 @@ Async:
 
 ```python
 import asyncio
-from picopyn.asynchronous import Client
+from picopyn.asynchronous import Pool
 
 async def main():
-    # create and connect client to the picodata cluster
-    client = Client(dsn="postgresql://admin:pass@localhost:5432")
-    await client.connect()
+    # create and open pool to the picodata cluster
+    pool = Pool(dsn="postgresql://admin:pass@localhost:5432", enable_discovery=True)
+    await pool.open()
 
     # execute DDL operations
-    await client.execute('''
+    await pool.execute('''
         CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
     ''')
 
     # execute DML/DQL operations
-    await client.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
-    rows = await client.fetch('SELECT * FROM "warehouse"')
+    await pool.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
+    rows = await pool.fetch('SELECT * FROM "warehouse"')
     print(rows)
 
-    await client.close()
+    await pool.close()
 
 asyncio.run(main())
 ```

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/README.md

@@ -12,17 +12,17 @@
 | >=25.2.1, <25.4.4 | 0.1.1             |
 | >=25.4.4, <25.5.1 | 0.2.0             |
 | >=25.5.1, <26.1.x | 1.0.0             |
-| >=26.1.1, <26.2.x | 2.0.0             |
+| >=26.1.1, <26.2.x | >=2.0.0, <4.0.0   |
 | 26.2.x            | ⚠️ Unreleased yet |
 
 ## Features
 * Connection pooling with configurable pool size
 * Optional automatic node discovery
 * Pluggable load-balancing strategies
+* Shard-aware routing
+* Automatic pool membership reconciliation, including rebalancing a full pool across the nodes
 * Asynchronous API based on [asyncpg](https://github.com/MagicStack/asyncpg)
-    * (async only) Shard-aware routing
-    * (async only) Automatic pool membership reconciliation
-* Synchronous [DPAPI-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
+* Synchronous [DB-API 2.0-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
 
 ## Navigation
 * [Installation](#installation)
@@ -54,24 +54,24 @@ Async:
 
 ```python
 import asyncio
-from picopyn.asynchronous import Client
+from picopyn.asynchronous import Pool
 
 async def main():
-    # create and connect client to the picodata cluster
-    client = Client(dsn="postgresql://admin:pass@localhost:5432")
-    await client.connect()
+    # create and open pool to the picodata cluster
+    pool = Pool(dsn="postgresql://admin:pass@localhost:5432", enable_discovery=True)
+    await pool.open()
 
     # execute DDL operations
-    await client.execute('''
+    await pool.execute('''
         CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
     ''')
 
     # execute DML/DQL operations
-    await client.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
-    rows = await client.fetch('SELECT * FROM "warehouse"')
+    await pool.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
+    rows = await pool.fetch('SELECT * FROM "warehouse"')
     print(rows)
 
-    await client.close()
+    await pool.close()
 
 asyncio.run(main())
 ```

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/benchmark/bench.py

@@ -8,7 +8,7 @@ from uuid import UUID
 import matplotlib
 import numpy as np
 
-from picopyn.asynchronous import Client
+from picopyn.asynchronous import Pool
 
 matplotlib.use("Agg")
 import os
@@ -121,17 +121,17 @@ def read_csv(input_file):
         exit(1)
 
 
-async def prepare_database(client):
+async def prepare_database(pool):
     """
     Prepare the database: drop and recreate the target table.
 
     Args:
-        client (Client): Picodata client connection.
+        pool (Pool): Picodata connection pool.
     """
     try:
-        await client.connect()
-        await client.execute("DROP TABLE IF EXISTS test_all_types;")
-        await client.execute(
+        await pool.open()
+        await pool.execute("DROP TABLE IF EXISTS test_all_types;")
+        await pool.execute(
             """
             CREATE TABLE test_all_types(
                 id INTEGER PRIMARY KEY,
@@ -176,13 +176,13 @@ def build_batch_insert_sql(batch_size: int) -> str:
     )
 
 
-async def worker(thread_id, client, batches, durations, errors, start_time, args):
+async def worker(thread_id, pool, batches, durations, errors, start_time, args):
     """
     Worker coroutine to perform batch inserts for assigned batches.
 
     Args:
         thread_id (int): Worker ID.
-        client (Client): Picodata client connection.
+        pool (Pool): Picodata connection pool.
         batches (list): List of batch rows to insert.
         durations (list): List to collect durations per batch.
         errors (list): List to collect timestamps of failed inserts.
@@ -194,7 +194,7 @@ async def worker(thread_id, client, batches, durations, errors, start_time, args
         try:
             sql = build_batch_insert_sql(len(batch))
             flat_args = [item for row in batch for item in row]
-            await client.execute(sql, *flat_args)
+            await pool.execute(sql, *flat_args)
         except Exception as e:
             if args.log_insert_errors:
                 print(f"[ERROR] Failed batch insert: {e}")
@@ -203,12 +203,12 @@ async def worker(thread_id, client, batches, durations, errors, start_time, args
         durations[thread_id].append(end - start)
 
 
-async def run_benchmark(client, rows, args):
+async def run_benchmark(pool, rows, args):
     """
     Run the entire insert benchmark using multiple workers.
 
     Args:
-        client (Client): Picodata client.
+        pool (Pool): Picodata connection pool.
         rows (list): Parsed rows to insert.
         args (Namespace): Parsed arguments.
 
@@ -229,7 +229,7 @@ async def run_benchmark(client, rows, args):
     errors = []
 
     tasks = [
-        worker(i, client, batches_per_thread[i], durations, errors, start_time, args)
+        worker(i, pool, batches_per_thread[i], durations, errors, start_time, args)
         for i in range(args.threads)
     ]
     await asyncio.gather(*tasks)
@@ -473,10 +473,13 @@ async def main():
 
     rows = read_csv(args.input)
 
-    client = Client(dsn=args.cluster, pool_size=args.threads)
-    await prepare_database(client)
+    pool = Pool(dsn=args.cluster, max_size=args.threads, enable_discovery=True)
+    try:
+        await prepare_database(pool)
 
-    durations, errors = await run_benchmark(client, rows, args)
+        durations, errors = await run_benchmark(pool, rows, args)
+    finally:
+        await pool.close()
 
     build_thread_graph(durations, args)
     build_overall_graph(durations, rows, args)

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_pool_init.md

@@ -16,7 +16,7 @@ ERROR([Initialization failed]):::error
 CFG[/"Pool settings<br>· DSN string<br>· pool size<br>· forbidden tiers<br>· cluster discovery mode"/]
 
 USAGE_DETAILS_NOTE>"Return to pool lifecycle details"]:::note
-click USAGE_DETAILS_NOTE "#picopyn.asynchronous.Pool" "Return to pool lifecycle details"
+click USAGE_DETAILS_NOTE "#picopyn.$MOD$.Pool" "Return to pool lifecycle details"
 
 %% === Data objects ===
 POOL[[Connection pool]]

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_pool_init_discovery_disabled.md

@@ -17,7 +17,7 @@ ERROR([Initialization failed]):::error
 CFG[/"Pool settings<br>· DSN string<br>· pool size<br>· forbidden tiers<br>· cluster discovery disabled"/]
 
 GENERAL_DETAILS_NOTE>"Return to general algorithm"]:::note
-click GENERAL_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Return to general algorithm"
+click GENERAL_DETAILS_NOTE "#picopyn.$MOD$.Pool.open" "Return to general algorithm"
 
 %% === Data objects ===
 B_DSN[[DSN node list]]
@@ -29,7 +29,7 @@ START --> CFG --> PARSE[Parse DSN string to DSN node list] --> B_PoolFilling
 PARSE <-. write .-> B_DSN
 B_DSN <-. read .-> B_CONNECT_NODE[Connect to DSN node]
 
-B_PoolFilling_NOTE>"iterate over DSN nodes until either:<br>- the list is empty<br>- pool_size connections<br>have been established"]:::note
+B_PoolFilling_NOTE>"iterate over DSN nodes until either:<br>- the list is empty<br>- max size connections<br>have been established"]:::note
 B_PoolFilling_NOTE -.-> B_PoolFilling
 subgraph B_PoolFilling["Pool filling"]
     direction TB

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_pool_init_discovery_enabled.md

@@ -15,10 +15,10 @@ START([Start]):::neutral
 DONE([Pool ready]):::success
 D_ERROR_CAND([Initialization failed]):::error
 ERROR([Initialization failed]):::error
-CFG[/"Pool settings<br>· DSN string<br>· pool_size<br>· forbidden tiers<br>· cluster discovery enabled"/]
+CFG[/"Pool settings<br>· DSN string<br>· max_size<br>· forbidden tiers<br>· cluster discovery enabled"/]
 
 GENERAL_DETAILS_NOTE>"Return to general algorithm"]:::note
-click GENERAL_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Return to general algorithm"
+click GENERAL_DETAILS_NOTE "#picopyn.$MOD$.Pool.open" "Return to general algorithm"
 
 %% === Data objects ===
 D_DSN[[DSN node list]]
@@ -40,7 +40,7 @@ subgraph DiscoverNodes["Discover cluster nodes"]
 end
 DiscoverNodes --> D_PoolFilling
 
-D_PoolFilling_NOTE>"iterate over candidate nodes<br>until either:<br>- the list is empty<br>- pool_size connections<br>have been established"]:::note
+D_PoolFilling_NOTE>"iterate over candidate nodes<br>until either:<br>- the list is empty<br>- max_size connections<br>have been established"]:::note
 D_PoolFilling_NOTE -.-> D_PoolFilling
 CAND <-. read .-> D_CONNECT_NODE[Connect to candidate node]
 subgraph D_PoolFilling["Pool filling"]

picopyn-2.0.0.dev1/docs/diagrams/_pool_lifecycle.md → picopyn-3.0.0/docs/diagrams/_pool_lifecycle_async.md

@@ -17,15 +17,15 @@ DONE([Pool closed]):::neutral
 
 CFG[/"Pool settings<br>· DSN<br>· pool size<br>· forbidden tiers<br>· cluster discovery mode"/]
 
-START --> CFG --> CREATE["Pool.__init__"] --> CONNECT["pool.connect()"] --> UsePool
+START --> CFG --> CREATE["Pool.__init__"] --> CONNECT["pool.open()"] --> UsePool
 CONNECT <-. write .-> POOL
 
-CONNECT_DETAILS_NOTE>"Connect algorithm details"]:::note
-click CONNECT_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Connect algorithm details"
+CONNECT_DETAILS_NOTE>"Open algorithm details"]:::note
+click CONNECT_DETAILS_NOTE "#picopyn.$MOD$.Pool.open" "Open algorithm details"
 CONNECT_DETAILS_NOTE -.-> CONNECT
 
-RECONCILE_NOTE>"(discovery mode only):<br>pool membership changes<br>in the background:<br>connections are opened/closed<br>to match topology changes"]:::note
-click RECONCILE_NOTE "#picopyn.asynchronous.Pool.topology" "Reconcile algorithm details"
+RECONCILE_NOTE>"pool membership changes<br>in the background:<br>connections are opened/closed<br>to match topology changes<br>(discovery mode: any cluster node,<br>bootstrap mode: DSN nodes only)"]:::note
+click RECONCILE_NOTE "#picopyn.$MOD$.Pool.topology" "Reconcile algorithm details"
 RECONCILE_NOTE -.-> POOL
 
 CYCLE_NOTE>"repeated for each<br>database operation"]:::note
@@ -33,12 +33,12 @@ CYCLE_NOTE -.-> UsePool
 
 subgraph UsePool["Pool usage"]
     direction TB
-    ACQUIRE["pool.acquire()"] --> USE["Execute queries<br>· pool.fetchrow()<br>· pool.fetch()<br>· pool.execute()"] --> RELEASE["pool.release(conn)"]
+    ACQUIRE["pool.acquire() -> conn"] --> USE["Execute queries<br>· conn.fetchrow()<br>· conn.fetch()<br>· conn.execute()"] --> RELEASE["pool.release(conn)"]
 end
 
 ACQUIRE <-. delete .-> POOL
 RELEASE <-. write .-> POOL
-CLOSE <-. delete .-> POOL
+CLOSE["pool.close()"] <-. delete .-> POOL
 UsePool --> CLOSE
 CLOSE --> DONE
 ```

picopyn-3.0.0/docs/diagrams/_pool_lifecycle_sync.md

@@ -0,0 +1,44 @@
+```mermaid
+---
+title: Pool lifecycle
+---
+
+flowchart TD
+%% === Style ===
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+%% === Data objects ===
+POOL[[Connection pool]]
+
+%% === Lifecycle ===
+START([Start]):::neutral
+DONE([Pool closed]):::neutral
+
+CFG[/"Pool settings<br>· DSN<br>· pool size<br>· forbidden tiers<br>· cluster discovery mode"/]
+
+START --> CFG --> CREATE["Pool.__init__"] --> CONNECT["pool.open()"] --> UsePool
+CONNECT <-. write .-> POOL
+
+CONNECT_DETAILS_NOTE>"Open algorithm details"]:::note
+click CONNECT_DETAILS_NOTE "#picopyn.$MOD$.Pool.open" "Open algorithm details"
+CONNECT_DETAILS_NOTE -.-> CONNECT
+
+RECONCILE_NOTE>"pool membership changes<br>in the background:<br>connections are opened/closed<br>to match topology changes<br>(discovery mode: any cluster node,<br>bootstrap mode: DSN nodes only)"]:::note
+click RECONCILE_NOTE "#picopyn.$MOD$.Pool.topology" "Reconcile algorithm details"
+RECONCILE_NOTE -.-> POOL
+
+CYCLE_NOTE>"repeated for each<br>database operation"]:::note
+CYCLE_NOTE -.-> UsePool
+
+subgraph UsePool["Pool usage"]
+    direction TB
+    ACQUIRE["with pool.connection() as conn"] --> CURSOR["conn.cursor() -> cursor"] --> USE["Execute queries<br>· cursor.fetchone()<br>· cursor.execute()<br>· ..."] --> RELEASE["leaving the with block<br>releases the connection"]
+end
+
+ACQUIRE <-. delete .-> POOL
+RELEASE <-. write .-> POOL
+CLOSE["pool.close()"] <-. delete .-> POOL
+UsePool --> CLOSE
+CLOSE --> DONE
+```

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_query_metadata_pool_init.md

@@ -10,13 +10,13 @@ classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
 
 INIT(["Pool init"]):::neutral
 
-INIT --> CACHE_EMPTY["Create empty query meta cache"] --> CONNECT["Fill the pool with ordinary<br>connections to picodata"] --> Pool1
+INIT --> CONNECT["Fill the pool with ordinary<br>connections to picodata"] --> Pool1
 
-NOTE_CONN>"connection options require<br>statement invalidation signal only"]:::note
+NOTE_CONN>"connection options require<br>statement invalidation<br>signal only"]:::note
 NOTE_CONN -.-> CONNECT
 
-CONNECT_DETAILS_NOTE>"Connect algorithm details"]:::note
-click CONNECT_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Connect algorithm details"
+CONNECT_DETAILS_NOTE>"Open algorithm details"]:::note
+click CONNECT_DETAILS_NOTE "#picopyn.$MOD$.Pool.open" "Open algorithm details"
 CONNECT_DETAILS_NOTE -.-> CONNECT
 
 subgraph Pool1["Pool"]
@@ -25,13 +25,14 @@ subgraph Pool1["Pool"]
     CONN11["conn to inst1"] --> PICO11[(Picodata)]
     CONN12["..."] --> PICO12[(Picodata)]
     CONN1N["conn to instN"] --> PICO1N[(Picodata)]
-
-    CACHE1[/"Query meta cache"/]
 end
 
-Pool1 --> CONNECT_SERV["create service connection<br>for meta listener"]
+Pool1 --> CONNECT_SERV["create empty query meta cache<br>and service connection<br>for meta listener"]
+
+NOTE_DISABLED>"skipped entirely when<br>service disabled: no cache,<br>no service connection"]:::note
+NOTE_DISABLED -.-> CONNECT_SERV
 
-NOTE_CONN_SERV>"connection options require statement invalidation signal<br>and distribution key metadata"]:::note
+NOTE_CONN_SERV>"connection options require<br>statement invalidation signal<br>and distribution key metadata"]:::note
 NOTE_CONN_SERV -.-> CONNECT_SERV
 
 CONNECT_SERV --> LIST["add picodata notice listener"]
@@ -45,7 +46,7 @@ subgraph Pool["Pool"]
     CONN2["..."] --> PICO2[(...)]
     CONNN["connN for<br>user usage"] --> PICON[(PicodataN)]
 
-    PIC(["can be any of"]) -.- CONNSERV["service connection with Notice listener"]
+    PIC(["can be any of"]) -.- CONNSERV["service connection<br>with Notice listener"]
     PICO1 -.- PIC
     PICO2 -.- PIC
     PICON -.- PIC

picopyn-3.0.0/docs/diagrams/_query_metadata_usage_lookup.md

@@ -0,0 +1,37 @@
+```mermaid
+---
+title: Query metadata -- usage lookup
+---
+
+sequenceDiagram
+    participant App as Application
+    box rgba(128,128,128,0.15) Picopyn
+        participant Driver as Core
+        participant Pool as Connection pool
+        participant Meta as Service connection
+    end
+    participant Server as Picodata
+
+    App->>Driver: pool.execute(query, params)
+
+    alt First call: cache miss
+        Driver->>Pool: Any free connection
+    else Repeated call: cache hit
+        Driver->>Driver: query meta + params -> bucket_id -> node
+        Driver->>Pool: Connection to that node
+    end
+    Pool-->>Driver: Connection
+
+    Driver->>Server: Bind + Execute
+    Server-->>Driver: Result
+    Driver-->>App: Result
+
+    rect rgba(128,128,128,0.15)
+        Note over Driver,Server: In the background, only after a cache miss
+        Driver->>Meta: query
+        Meta->>Server: Parse(query)
+        Server-->>Meta: NoticeResponse
+        Meta-->>Driver: query meta
+        Driver->>Driver: LRU cache
+    end
+```

picopyn-3.0.0/docs/diagrams/_rebalance_pool_after_reconcile.md

@@ -0,0 +1,50 @@
+```mermaid
+---
+title: Rebalance pool after reconcile
+---
+flowchart TD
+
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+classDef success fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+
+START(["START"]):::neutral --> COUNT
+
+COUNT_DETAILS_NOTE>"connections already marked<br>to evict are not counted:<br>they are closed on<br>release and refilled elsewhere"]:::note
+COUNT_DETAILS_NOTE -.-> COUNT
+
+COUNT_DETAILS_NOTE2>"candidate nodes are<br>online Picodata nodes<br>from allowed tiers"]:::note
+COUNT_DETAILS_NOTE2 -.-> COUNT
+
+COUNT["count free and in-use pool<br>connections per candidate uuid"] --> GAP
+
+GAP_DETAILS_NOTE>"a gap of one<br>is not an imbalance"]:::note
+GAP_DETAILS_NOTE -.-> GAP
+
+GAP{"gap between<br>max count and min count<br>(most overloaded and<br>most underloaded nodes)<br>>= 2?"}
+GAP -->|no| POOL_OK(["pool is balanced<br>(exit)"]):::success
+GAP -->|yes| FREE{"is there a free connection<br>to the overloaded uuid?"}
+
+FREE -->|no| EVICT_USED["mark one in-use connection<br>to the overloaded uuid to evict"]
+FREE -->|yes| MIGRATE["migrate one connection:<br>take the free one out, connect<br>to the underloaded uuid,<br>close the old one"]
+
+EVICT_DETAILS_NOTE>"release closes it, and the next<br>refresh refills the free slot to<br>the underloaded uuid"]:::note
+EVICT_DETAILS_NOTE -.-> EVICT_USED
+
+EVICT_USED --> POOL_OK3(["(exit)"]):::success
+
+MIGRATE -->|migrated| LIMIT
+MIGRATE -->|"could not connect to<br>the underloaded uuid"| EXCLUDE["put the connection back and<br>exclude the underloaded uuid<br>from the candidate list<br>in this reconcile"]
+EXCLUDE --> LIMIT
+
+LIMIT{"migration limit for<br>this reconcile reached?"}
+LIMIT -->|no| GAP
+LIMIT -->|yes| POOL_OK4(["(exit)"]):::success
+
+LIMIT_DETAILS_NOTE>"the limit is proportional to<br>the pool size, see<br>Pool.rebalance_pool_divisor"]:::note
+click LIMIT_DETAILS_NOTE "#picopyn.$MOD$.Pool" "Pool.rebalance_pool_divisor"
+LIMIT_DETAILS_NOTE -.-> LIMIT
+
+LIMIT_DETAILS_NOTE2>"the rest is left to<br>the next refresh"]:::note
+LIMIT_DETAILS_NOTE2 -.-> POOL_OK4
+```

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_reconcile_pool_with_topology.md

@@ -10,7 +10,6 @@ classDef success fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
 classDef error fill:#ffebee,stroke:#f44336,color:#b71c1c
 
 subgraph Pool["Pool"]
-    Settings[/"forbidden tiers=[...]<br>max size=N"/]
     FreeConns["free conns=[...]"]
     UsedConns["used conns=[...]"]
     Topology[/"Topology"/]
@@ -25,14 +24,16 @@ TOPOLOGY_DETAILS_NOTE>"Topology details"]:::note
 click TOPOLOGY_DETAILS_NOTE "../topology/#picopyn.topology.Topology" "Topology details"
 TOPOLOGY_DETAILS_NOTE -.-> TOPOLOGY_APPLY
 
-TOPOLOGY_APPLY --> GET_LIST["build candidate_list from Topology:<br>- uuids with allowed tiers<br>- online uuids"]
+TOPOLOGY_APPLY --> COMPLETE{"topology has both<br>instances and replicasets?"}
+COMPLETE -->|no| NOT_TRUSTED(["snapshot is not trustworthy:<br>pool is left as is<br>(exit)"]):::error
+COMPLETE -->|yes| GET_LIST["build candidate_list from Topology:<br>- uuids with allowed tiers<br>- online uuids<br>- (bootstrap mode) only<br>DSN node addresses"]
 GET_LIST --> REMOVE["close all connections whose<br>uuid not in candidate_list"] --> CHECK_SIZE_START
 
 REMOVE -.-> FreeConns
 REMOVE -.-> UsedConns
 
 CHECK_SIZE_START{"current pool size < max size?"}
-CHECK_SIZE_START -->|no| POOL_OK(["Pool okay<br>(exit)"]):::success
+CHECK_SIZE_START -->|no| REBALANCE
 CHECK_SIZE_START -->|yes| MIN["choose uuid from candidate_list<br>with min connections count"]
 
 MIN -->|got candidate| ADD_CONN["try to connect to candidate<br>and add connection to pool..."]
@@ -41,6 +42,13 @@ MIN -->|no candidate| NO_CANDIDATE2(["pool is temporarily not<br>full until next
 ADD_CONN -->|connection failed| EXCLUDE["exclude the candidate uuid<br/>from list in this loop"] --> CHECK_SIZE_LOOP
 ADD_CONN-->|connected| CHECK_SIZE_LOOP{"current size < max size?"}
 
-CHECK_SIZE_LOOP -->|no| POOL_OK2(["Pool okay<br>(exit)"]):::success
+CHECK_SIZE_LOOP -->|no| REBALANCE
 CHECK_SIZE_LOOP -->|yes| MIN
+
+REBALANCE["rebalance the pool:<br>move connections from overloaded<br>nodes to underloaded ones"]
+REBALANCE --> POOL_OK(["Pool okay<br>(exit)"]):::success
+
+REBALANCE_DETAILS_NOTE>"Rebalance details"]:::note
+click REBALANCE_DETAILS_NOTE "#picopyn.$MOD$.Pool.topology--rebalance" "Rebalance details"
+REBALANCE_DETAILS_NOTE -.-> REBALANCE
 ```

picopyn-3.0.0/docs/diagrams/_reconcile_rebalance.md

@@ -0,0 +1,45 @@
+```mermaid
+---
+title: (Rebalance) a pooled node returned to Online (pool is already full)
+---
+
+flowchart TD
+
+classDef offline fill:#ffebee,stroke:#f44336,color:#b71c1c
+classDef online fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+subgraph before["Before"]
+    subgraph Picodata_before["Picodata"]
+        p1[("ins 1")]:::online
+        p2[("ins 2")]:::online
+        p3[("ins 3")]:::offline
+    end
+
+    subgraph Pool_before["Pool: max_size=6"]
+        c1["conn x3"] --> p1
+        c2["conn x3"] --> p2
+    end
+end
+EVENT1[/"Event<br>Instance 3 back Online"/]:::neutral -.-> REBALANCE{{"reconcile pool"}}
+before -.-> REBALANCE
+NOTE>"Pool is full, so refill<br>does nothing.<br>Initial connection counts:<br>[3/3/0], gap=3<br>=> migrate: [2/3/1], gap=2<br>=> migrate: [2/2/2], gap=0<br>Pool rebalanced in 2 refreshes"]:::note
+NOTE -.-> REBALANCE
+REBALANCE --> after
+
+subgraph after["After (2 refreshes)"]
+    subgraph Picodata_after["Picodata"]
+        p21[("ins 1")]:::online
+        p22[("ins 2")]:::online
+        p23[("ins 3")]:::online
+    end
+
+    subgraph Pool_after["Pool: max_size=6"]
+        c21["conn x2"] --> p21
+        c22["conn x2"] --> p22
+        c23["conn x2"] --> p23
+    end
+end
+```

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_request_if_missing.md

@@ -9,10 +9,8 @@ classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
 classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
 
 QUERY(["parametrized query"]):::neutral
-HASH(["hash of the<br>query text"]):::neutral
 
 QUERY --> PARSE_QUERY
-HASH --> PARSE_QUERY
 PARSE_QUERY["ask Picodata to parse query"] -. Parse .-> PICO[("Picodata")] -. Notice message .-> GET
 
 GET["receive the notice"] --> PARSE_MSG{"is this query metadata?"}

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_topology_components.md

@@ -20,10 +20,10 @@ flowchart TB
     click TOPOLOGY_DETAILS_NOTE "../topology/#picopyn.topology.Topology" "See topology details"
 
     TRACKER_DETAILS_NOTE>"See topology tracker details"]:::note -.-> TRACKER
-    click TRACKER_DETAILS_NOTE "#picopyn.asynchronous.topology_tracker.TopologyTracker" "Topology tracker details"
+    click TRACKER_DETAILS_NOTE "#picopyn.$MOD$.topology_tracker.TopologyTracker" "Topology tracker details"
 
     SOURCE_DETAILS_NOTE>"See topology source details"]:::note -.-> SOURCE
-    click SOURCE_DETAILS_NOTE "#picopyn.asynchronous.topology_tracker.PollingTopologySource" "Topology source details"
+    click SOURCE_DETAILS_NOTE "#picopyn.$MOD$.topology_tracker.PollingTopologySource" "Topology source details"
 
     NOTE>"today: events are built by driver<br>polled Picodata's system<br>tables/functions.<br>planned: Picodata itself<br>is the source, pushing<br>events directly"]:::note
     NOTE -.-> SOURCE

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_topology_refresh.md

@@ -44,7 +44,7 @@ BUCKETS_DETAILS_NOTE>"Bucket event details"]:::note
 click BUCKETS_DETAILS_NOTE "#picopyn.topology.Topology.apply_bucket" "Bucket event details"
 BUCKETS_DETAILS_NOTE -.-> BUCKET_APPLY
 
-NOTE>"_pico_bucket(tier) may not exist<br>on older Picodata -- failure is logged<br>and skipped per tier, not fatal"]:::note
+NOTE>"_pico_bucket(tier) may not exist<br>on older Picodata -- failure<br>is logged and skipped per tier,<br>not fatal"]:::note
 NOTE -.-> BUCKET_FETCH
 BUCKET_APPLY <-. write .-> TOPOLOGY4[/Topology/]
 

{picopyn-2.0.0.dev1 → picopyn-3.0.0}/docs/diagrams/_topology_tracking.md

@@ -12,8 +12,8 @@ click ADR_DETAILS_NOTE "https://git.picodata.io/core/picodata/-/blob/master/doc/
 
 TOPOLOGY[[Topology<br>·instances<br>·replicasets<br>·buckets<br>·tiers]]
 
-CONNECT["Pool.connect()"] --> CFG[/"Topology tracker settings<br>· update interval"/] --> START["start topology tracker"] --> INITIAL["initial refresh<br>(sync, imitate ReadyForQuery)"] --> WAIT
-INITAL_DETAILS_NOTE>"Picodata doesn't send it yet,<br>but it's planned: Picodata will send<br>ReadyForQuery once the initial snapshot<br>is fully delivered -- meaning our<br>topology view is consistent"]:::note
+CONNECT["Pool.open()"] --> CFG[/"Topology tracker settings<br>· update interval"/] --> START["start topology tracker"] --> INITIAL["initial refresh<br>(sync, imitate ReadyForQuery)"] --> WAIT
+INITAL_DETAILS_NOTE>"Picodata doesn't send it yet,<br>but it's planned: Picodata will<br>send ReadyForQuery once the<br>initial snapshot is fully<br>delivered -- meaning our<br>topology view is consistent"]:::note
 
 INITAL_DETAILS_NOTE -.-> INITIAL
 

picopyn-3.0.0/docs/explain.md

@@ -0,0 +1,21 @@
+# Explain
+
+::: picopyn.explain
+    options:
+      show_root_heading: false
+      show_root_toc_entry: false
+      heading_level: 1
+      members: []
+
+## API Reference
+
+::: picopyn.explain
+    options:
+      show_root_heading: false
+      show_root_toc_entry: false
+      heading_level: 3
+      members_order: source
+      extra:
+        hide_docstring: true
+      filters:
+        - "!^_"