picopyn 0.3.0.dev1__tar.gz → 2.0.0.dev1__tar.gz

{picopyn-0.3.0.dev1/picopyn.egg-info → picopyn-2.0.0.dev1}/PKG-INFO

@@ -1,16 +1,16 @@
 Metadata-Version: 2.4
 Name: picopyn
-Version: 0.3.0.dev1
+Version: 2.0.0.dev1
 Summary: Python connector for working with the distributed Picodata database.
-Author-email: Asya Lomakina <a.lomakina@picodata.io>, Igor Kuznetsov <i.kuznetsov@picodata.io>, Dmitriy Кoltsov <dkoltsov@picodata.io>, Daniil Gorlyakov <d.gorlyakov@picodata.io>
+Author-email: Asya Lomakina <a.lomakina@picodata.io>, Igor Kuznetsov <i.kuznetsov@picodata.io>, Dmitriy Koltsov <dkoltsov@picodata.io>, Daniil Gorlyakov <d.gorlyakov@picodata.io>
 License-Expression: BSD-2-Clause
-Project-URL: Source, https://git.picodata.io/solution/picopyn
+Project-URL: Source, https://git.picodata.io/core/drivers/picopyn
 Project-URL: Documentation, https://picopyn.readthedocs.io/en/stable/
 Project-URL: Changelog, https://git.picodata.io/core/drivers/picopyn/-/blob/main/CHANGELOG.md
 Keywords: database,picodata
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Operating System :: OS Independent
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Database
 Classifier: Framework :: AsyncIO
@@ -18,10 +18,11 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 License-File: AUTHORS.md
-Requires-Dist: async-timeout==5.0.1
 Requires-Dist: asyncpg==0.30.0
-Requires-Dist: typing_extensions==4.12.2; python_version == "3.10"
+Requires-Dist: typing_extensions<5,>=4.12.2
 Requires-Dist: psycopg[binary]==3.3.4
+Requires-Dist: msgpack<2,>=1.0.0
+Requires-Dist: mmh3<6,>=5.2.1
 Provides-Extra: test
 Requires-Dist: pytest==8.3.5; extra == "test"
 Requires-Dist: pytest-asyncio==0.25.3; extra == "test"
@@ -29,7 +30,7 @@ Requires-Dist: ruff==0.11.13; extra == "test"
 Requires-Dist: mypy==1.16.0; extra == "test"
 Requires-Dist: black==25.1.0; extra == "test"
 Provides-Extra: docs
-Requires-Dist: mkdocs<2.0,>=1.0; extra == "docs"
+Requires-Dist: mkdocs<2.0,>=1.6; extra == "docs"
 Requires-Dist: mkdocs-material; extra == "docs"
 Requires-Dist: mkdocstrings[python]; extra == "docs"
 Provides-Extra: benchmark
@@ -48,11 +49,23 @@ Dynamic: license-file
 
 **Picopyn** is a Python package for working with the distributed [Picodata](https://picodata.io/en/) database.
 
+**Version compatibility**:
+| Picodata version  | Picopyn version   |
+| ----------------- | ----------------- |
+| >=25.2.1, <25.4.4 | 0.1.1             |
+| >=25.4.4, <25.5.1 | 0.2.0             |
+| >=25.5.1, <26.1.x | 1.0.0             |
+| >=26.1.1, <26.2.x | 2.0.0             |
+| 26.2.x            | ⚠️ Unreleased yet |
+
 ## Features
 * Connection pooling with configurable pool size
 * Optional automatic node discovery
 * Pluggable load-balancing strategies
-* Provides both asynchronous API (based on [asyncpg](https://github.com/MagicStack/asyncpg)) and synchronous API (based on [psycopg](https://www.psycopg.org/))
+* Asynchronous API based on [asyncpg](https://github.com/MagicStack/asyncpg)
+    * (async only) Shard-aware routing
+    * (async only) Automatic pool membership reconciliation
+* Synchronous [DPAPI-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
 
 ## Navigation
 * [Installation](#installation)
@@ -73,16 +86,18 @@ pip install picopyn
 Or from source:
 
 ```sh
-git clone https://git.picodata.io/solution/picopyn.git
+git clone https://git.picodata.io/core/drivers/picopyn.git
 cd picopyn
 pip install -e .
 ```
 
 ## Quickstart
 
+Async:
+
 ```python
 import asyncio
-from picopyn import Client
+from picopyn.asynchronous import Client
 
 async def main():
     # create and connect client to the picodata cluster
@@ -94,9 +109,9 @@ async def main():
         CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
     ''')
 
-    # execute DML operations
-    await client.execute('INSERT INTO \"warehouse\" VALUES ($1::int, $2::varchar)', 1, "test")
-    rows = await client.fetch('SELECT * FROM \"warehouse\"')
+    # execute DML/DQL operations
+    await client.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
+    rows = await client.fetch('SELECT * FROM "warehouse"')
     print(rows)
 
     await client.close()
@@ -104,6 +119,26 @@ async def main():
 asyncio.run(main())
 ```
 
+Sync:
+
+```python
+from picopyn.synchronous import connect
+
+# create and connect to the picodata cluster
+with connect("postgresql://admin:pass@localhost:5432") as conn:
+    cur = conn.cursor()
+
+    # execute DDL operations
+    cur.execute('''
+        CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
+    ''')
+
+    # execute DML/DQL operations
+    cur.execute('INSERT INTO "warehouse" VALUES (%s, %s)', (1, "test"))
+    cur.execute('SELECT * FROM "warehouse"')
+    print(cur.fetchall())
+```
+
 ## Development
 For development we use [uv](https://docs.astral.sh/uv/getting-started/installation/) as package manager and [docker compose](https://docs.docker.com/compose/install/) for test environment.
 

{picopyn-0.3.0.dev1 → picopyn-2.0.0.dev1}/README.md

@@ -6,11 +6,23 @@
 
 **Picopyn** is a Python package for working with the distributed [Picodata](https://picodata.io/en/) database.
 
+**Version compatibility**:
+| Picodata version  | Picopyn version   |
+| ----------------- | ----------------- |
+| >=25.2.1, <25.4.4 | 0.1.1             |
+| >=25.4.4, <25.5.1 | 0.2.0             |
+| >=25.5.1, <26.1.x | 1.0.0             |
+| >=26.1.1, <26.2.x | 2.0.0             |
+| 26.2.x            | ⚠️ Unreleased yet |
+
 ## Features
 * Connection pooling with configurable pool size
 * Optional automatic node discovery
 * Pluggable load-balancing strategies
-* Provides both asynchronous API (based on [asyncpg](https://github.com/MagicStack/asyncpg)) and synchronous API (based on [psycopg](https://www.psycopg.org/))
+* Asynchronous API based on [asyncpg](https://github.com/MagicStack/asyncpg)
+    * (async only) Shard-aware routing
+    * (async only) Automatic pool membership reconciliation
+* Synchronous [DPAPI-compatible](https://peps.python.org/pep-0249/) API based on [psycopg](https://www.psycopg.org/)
 
 ## Navigation
 * [Installation](#installation)
@@ -31,16 +43,18 @@ pip install picopyn
 Or from source:
 
 ```sh
-git clone https://git.picodata.io/solution/picopyn.git
+git clone https://git.picodata.io/core/drivers/picopyn.git
 cd picopyn
 pip install -e .
 ```
 
 ## Quickstart
 
+Async:
+
 ```python
 import asyncio
-from picopyn import Client
+from picopyn.asynchronous import Client
 
 async def main():
     # create and connect client to the picodata cluster
@@ -52,9 +66,9 @@ async def main():
         CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
     ''')
 
-    # execute DML operations
-    await client.execute('INSERT INTO \"warehouse\" VALUES ($1::int, $2::varchar)', 1, "test")
-    rows = await client.fetch('SELECT * FROM \"warehouse\"')
+    # execute DML/DQL operations
+    await client.execute('INSERT INTO "warehouse" VALUES ($1::int, $2::varchar)', 1, "test")
+    rows = await client.fetch('SELECT * FROM "warehouse"')
     print(rows)
 
     await client.close()
@@ -62,6 +76,26 @@ async def main():
 asyncio.run(main())
 ```
 
+Sync:
+
+```python
+from picopyn.synchronous import connect
+
+# create and connect to the picodata cluster
+with connect("postgresql://admin:pass@localhost:5432") as conn:
+    cur = conn.cursor()
+
+    # execute DDL operations
+    cur.execute('''
+        CREATE TABLE "warehouse" (id INTEGER NOT NULL, item TEXT NOT NULL, PRIMARY KEY (id)) USING memtx DISTRIBUTED BY (id) OPTION (TIMEOUT = 3.0);
+    ''')
+
+    # execute DML/DQL operations
+    cur.execute('INSERT INTO "warehouse" VALUES (%s, %s)', (1, "test"))
+    cur.execute('SELECT * FROM "warehouse"')
+    print(cur.fetchall())
+```
+
 ## Development
 For development we use [uv](https://docs.astral.sh/uv/getting-started/installation/) as package manager and [docker compose](https://docs.docker.com/compose/install/) for test environment.
 

{picopyn-0.3.0.dev1 → picopyn-2.0.0.dev1}/benchmark/bench.py

@@ -8,7 +8,7 @@ from uuid import UUID
 import matplotlib
 import numpy as np
 
-from picopyn import Client
+from picopyn.asynchronous import Client
 
 matplotlib.use("Agg")
 import os

picopyn-2.0.0.dev1/docs/async.md

@@ -0,0 +1,6 @@
+# Async
+
+::: picopyn.asynchronous
+    options:
+      filters:
+        - "!^_"

picopyn-2.0.0.dev1/docs/async_internal.md

@@ -0,0 +1,18 @@
+# Async: Internal
+
+!!! warning
+    Not public API. Driver wires this up and owns it; nothing here is meant
+    to be constructed or called directly. Documented anyway so you can see
+    how it actually works
+
+::: picopyn.asynchronous.topology_tracker
+    options:
+      show_root_heading: true
+      filters:
+        - "!^_"
+
+::: picopyn.asynchronous.metadata_service
+    options:
+      show_root_heading: true
+      filters:
+        - "!^_"

picopyn-2.0.0.dev1/docs/bucket.md

@@ -0,0 +1,7 @@
+# Bucket ID
+
+::: picopyn.bucket
+    options:
+      filters:
+        - "!^_"
+        - "^__init__$"

picopyn-2.0.0.dev1/docs/changelog.md

@@ -0,0 +1,3 @@
+# Changelog
+
+--8<-- "CHANGELOG.md"

picopyn-2.0.0.dev1/docs/contributing.md

@@ -0,0 +1,7 @@
+# Contributing
+
+Contributions are welcome! The project is hosted on GitLab:
+
+**[git.picodata.io/core/drivers/picopyn](https://git.picodata.io/core/drivers/picopyn)**
+
+To get started with the development environment, running tests, and code style requirements, see the [Development](https://git.picodata.io/core/drivers/picopyn#development) section in the project README.

picopyn-2.0.0.dev1/docs/diagrams/_pool_init.md

@@ -0,0 +1,64 @@
+```mermaid
+---
+title: Pool initialization
+---
+
+flowchart TD
+%% === Initialization ===
+classDef success fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+classDef error fill:#ffebee,stroke:#f44336,color:#b71c1c
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+START([Start]):::neutral
+DONE([Pool ready]):::success
+ERROR([Initialization failed]):::error
+CFG[/"Pool settings<br>· DSN string<br>· pool size<br>· forbidden tiers<br>· cluster discovery mode"/]
+
+USAGE_DETAILS_NOTE>"Return to pool lifecycle details"]:::note
+click USAGE_DETAILS_NOTE "#picopyn.asynchronous.Pool" "Return to pool lifecycle details"
+
+%% === Data objects ===
+POOL[[Connection pool]]
+B_DSN[[DSN node list]]
+B_SYS[(Picodata system tables)]
+D_DSN[[DSN node list]]
+D_SYS[(Picodata system tables)]
+CAND[[Candidate node list]]
+
+%% === Actions ===
+
+START --> CFG --> PARSE[Parse DSN string to DSN node list] --> DISCOVERY{Cluster discovery enabled?}
+
+DISCOVERY -->|yes| D_FILTER[Use DSN nodes to connect to<br>Picodata cluster and retrieve<br>online nodes with allowed tiers<br>as a candidate node list]
+DISCOVERY -->|no| B_FILTER[Filter DSN nodes: use only online<br>nodes with allowed tiers]
+
+subgraph DiscoverMode["Cluster discovery mode"]
+    D_FILTER --> D_FILL_POOL[Try to fill pool with connections<br>to candidates up to pool size]
+
+    D_DETAILS_NOTE>"Discovery mode details"]:::note
+    click D_DETAILS_NOTE "#pool-init-discovery-enabled" "Discovery mode details"
+
+    D_FILTER <-. write .-> CAND
+    CAND <-. read .-> D_FILL_POOL
+end
+
+subgraph BootstrapMode["Bootstrap mode"]
+    B_FILTER --> B_FILL_POOL[Try to fill pool with connections<br>to DSN nodes up to pool size]
+
+    B_DETAILS_NOTE>"Bootstrap mode details"]:::note
+    click B_DETAILS_NOTE "#pool-init-discovery-disabled" "Bootstrap mode details"
+end
+
+B_DSN <-. read .-> B_FILTER
+B_SYS <-. read .-> B_FILTER
+B_FILL_POOL <-. write .-> POOL
+
+D_DSN <-. read .-> D_FILTER
+D_SYS <-. read .-> D_FILTER
+D_FILL_POOL <-. write .-> POOL
+
+POOL <-.-> POOL_CHECK{Pool full?}
+POOL_CHECK -->|no| ERROR
+POOL_CHECK -->|yes| DONE
+```

picopyn-2.0.0.dev1/docs/diagrams/_pool_init_discovery_disabled.md

@@ -0,0 +1,46 @@
+```mermaid
+---
+title: Pool initialization with cluster discovery disabled
+---
+
+flowchart TD
+
+%% === Initialization ===
+classDef success fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+classDef error fill:#ffebee,stroke:#f44336,color:#b71c1c
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+START([Start]):::neutral
+DONE([Pool ready]):::success
+ERROR([Initialization failed]):::error
+CFG[/"Pool settings<br>· DSN string<br>· pool size<br>· forbidden tiers<br>· cluster discovery disabled"/]
+
+GENERAL_DETAILS_NOTE>"Return to general algorithm"]:::note
+click GENERAL_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Return to general algorithm"
+
+%% === Data objects ===
+B_DSN[[DSN node list]]
+BOOTSTRAP_POOL[[Connection pool]]
+SYS[(Picodata system tables)]
+
+%% === Bootstrap mode ===
+START --> CFG --> PARSE[Parse DSN string to DSN node list] --> B_PoolFilling
+PARSE <-. write .-> B_DSN
+B_DSN <-. read .-> B_CONNECT_NODE[Connect to DSN node]
+
+B_PoolFilling_NOTE>"iterate over DSN nodes until either:<br>- the list is empty<br>- pool_size connections<br>have been established"]:::note
+B_PoolFilling_NOTE -.-> B_PoolFilling
+subgraph B_PoolFilling["Pool filling"]
+    direction TB
+    SYS <-. read .-> B_CHECK_NODE_TIER[Check node tier]
+    B_CONNECT_NODE --> B_CHECK_NODE_TIER
+    B_CHECK_NODE_TIER -->|node offline or tier forbidden| B_EXCLUDE[Exclude DSN node]
+    B_CHECK_NODE_TIER -->|node online and tier allowed| B_SAVE[Save connection to pool]
+    B_SAVE <-. write .-> BOOTSTRAP_POOL
+end
+
+B_PoolFilling --> POOL_CHECK{Pool full?}
+POOL_CHECK -->|no| ERROR
+POOL_CHECK -->|yes| DONE
+```

picopyn-2.0.0.dev1/docs/diagrams/_pool_init_discovery_enabled.md

@@ -0,0 +1,55 @@
+```mermaid
+---
+title: Pool initialization with cluster discovery enabled
+---
+
+flowchart TD
+
+%% === Initialization ===
+classDef success fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+classDef error fill:#ffebee,stroke:#f44336,color:#b71c1c
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+START([Start]):::neutral
+DONE([Pool ready]):::success
+D_ERROR_CAND([Initialization failed]):::error
+ERROR([Initialization failed]):::error
+CFG[/"Pool settings<br>· DSN string<br>· pool_size<br>· forbidden tiers<br>· cluster discovery enabled"/]
+
+GENERAL_DETAILS_NOTE>"Return to general algorithm"]:::note
+click GENERAL_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Return to general algorithm"
+
+%% === Data objects ===
+D_DSN[[DSN node list]]
+CAND[[Candidate node list]]
+DISCOVERY_POOL[[Connection pool]]
+SYS[(Picodata system tables)]
+
+%% === Discovery mode ===
+START --> CFG --> PARSE[Parse DSN string to<br>DSN node list] --> DiscoverNodes
+PARSE <-. write .-> D_DSN
+
+D_DiscoverNodes_NOTE>"iterate over DSN nodes<br>to establish a temporary<br>connection and retrieve the<br>list of online nodes with<br>allowed tiers"]:::note
+D_DiscoverNodes_NOTE -.-> DiscoverNodes
+D_DSN <-. read .-> D_RECEIVE[Try to connect and receive cluster nodes]
+subgraph DiscoverNodes["Discover cluster nodes"]
+    SYS <-. read .-> D_RECEIVE
+    D_RECEIVE -->|could not select candidates| D_ERROR_CAND
+    D_RECEIVE <-. write .-> CAND
+end
+DiscoverNodes --> D_PoolFilling
+
+D_PoolFilling_NOTE>"iterate over candidate nodes<br>until either:<br>- the list is empty<br>- pool_size connections<br>have been established"]:::note
+D_PoolFilling_NOTE -.-> D_PoolFilling
+CAND <-. read .-> D_CONNECT_NODE[Connect to candidate node]
+subgraph D_PoolFilling["Pool filling"]
+    D_CONNECT_NODE -->|connection error| D_EXCLUDE[Exclude candidate node]
+    D_CONNECT_NODE -->|connected| D_SAVE[Save opened connection to pool]
+    D_SAVE <-. write .-> DISCOVERY_POOL
+end
+
+D_PoolFilling --> POOL_CHECK{Pool full?}
+POOL_CHECK -->|no| ERROR
+POOL_CHECK -->|yes| DONE
+```

picopyn-2.0.0.dev1/docs/diagrams/_pool_lifecycle.md

@@ -0,0 +1,44 @@
+```mermaid
+---
+title: Pool lifecycle
+---
+
+flowchart TD
+%% === Style ===
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+%% === Data objects ===
+POOL[[Connection pool]]
+
+%% === Lifecycle ===
+START([Start]):::neutral
+DONE([Pool closed]):::neutral
+
+CFG[/"Pool settings<br>· DSN<br>· pool size<br>· forbidden tiers<br>· cluster discovery mode"/]
+
+START --> CFG --> CREATE["Pool.__init__"] --> CONNECT["pool.connect()"] --> UsePool
+CONNECT <-. write .-> POOL
+
+CONNECT_DETAILS_NOTE>"Connect algorithm details"]:::note
+click CONNECT_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Connect algorithm details"
+CONNECT_DETAILS_NOTE -.-> CONNECT
+
+RECONCILE_NOTE>"(discovery mode only):<br>pool membership changes<br>in the background:<br>connections are opened/closed<br>to match topology changes"]:::note
+click RECONCILE_NOTE "#picopyn.asynchronous.Pool.topology" "Reconcile algorithm details"
+RECONCILE_NOTE -.-> POOL
+
+CYCLE_NOTE>"repeated for each<br>database operation"]:::note
+CYCLE_NOTE -.-> UsePool
+
+subgraph UsePool["Pool usage"]
+    direction TB
+    ACQUIRE["pool.acquire()"] --> USE["Execute queries<br>· pool.fetchrow()<br>· pool.fetch()<br>· pool.execute()"] --> RELEASE["pool.release(conn)"]
+end
+
+ACQUIRE <-. delete .-> POOL
+RELEASE <-. write .-> POOL
+CLOSE <-. delete .-> POOL
+UsePool --> CLOSE
+CLOSE --> DONE
+```

picopyn-2.0.0.dev1/docs/diagrams/_query_metadata_pool_init.md

@@ -0,0 +1,55 @@
+```mermaid
+---
+title: Query metadata -- pool initialization in cache meta context
+---
+
+flowchart TD
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+
+INIT(["Pool init"]):::neutral
+
+INIT --> CACHE_EMPTY["Create empty query meta cache"] --> CONNECT["Fill the pool with ordinary<br>connections to picodata"] --> Pool1
+
+NOTE_CONN>"connection options require<br>statement invalidation signal only"]:::note
+NOTE_CONN -.-> CONNECT
+
+CONNECT_DETAILS_NOTE>"Connect algorithm details"]:::note
+click CONNECT_DETAILS_NOTE "#picopyn.asynchronous.Pool.connect" "Connect algorithm details"
+CONNECT_DETAILS_NOTE -.-> CONNECT
+
+subgraph Pool1["Pool"]
+    direction TB
+
+    CONN11["conn to inst1"] --> PICO11[(Picodata)]
+    CONN12["..."] --> PICO12[(Picodata)]
+    CONN1N["conn to instN"] --> PICO1N[(Picodata)]
+
+    CACHE1[/"Query meta cache"/]
+end
+
+Pool1 --> CONNECT_SERV["create service connection<br>for meta listener"]
+
+NOTE_CONN_SERV>"connection options require statement invalidation signal<br>and distribution key metadata"]:::note
+NOTE_CONN_SERV -.-> CONNECT_SERV
+
+CONNECT_SERV --> LIST["add picodata notice listener"]
+
+LIST --> Pool
+
+subgraph Pool["Pool"]
+    direction TB
+
+    CONN1["conn1 for<br>user usage"] --> PICO1[(Picodata1)]
+    CONN2["..."] --> PICO2[(...)]
+    CONNN["connN for<br>user usage"] --> PICON[(PicodataN)]
+
+    PIC(["can be any of"]) -.- CONNSERV["service connection with Notice listener"]
+    PICO1 -.- PIC
+    PICO2 -.- PIC
+    PICON -.- PIC
+
+    CACHE[/"Query meta cache"/]
+end
+```

picopyn-2.0.0.dev1/docs/diagrams/_query_metadata_usage_lookup.md

@@ -0,0 +1,44 @@
+```mermaid
+---
+title: Query metadata -- usage lookup
+---
+
+flowchart TD
+
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+
+QUERY(["parametrized query"]):::neutral --> CALL(["Pool.execute"]) --> HASH
+
+HASH["hash the query text"] --> SEARCH_CACHE{"cached meta by hash?"}
+
+CACHE1[/"Pool query meta cache"/] -. read .- SEARCH_CACHE
+SEARCH_CACHE -->|yes| USE_CACHE["metadata found in cache"]
+SEARCH_CACHE -->|no| CACHE_MISS["no metadata yet --<br>schedule background prepare"] --> ACQUIRE_CONN
+
+subgraph Routing["BUCKET ROUTING"]
+    direction TB
+
+    CALC["calculate `bucket_id`<br>from dk_meta"]
+
+    CALC --> MAP_CONN["find bucket master<br>for this tier"]
+
+    MAP_CONN -->|routed| CALL_THE_CONN["run the query on<br>the chosen connection"]
+    MAP_CONN -->|cannot route| ACQUIRE_CONN
+
+    CALL_THE_CONN --> RESULT(["give result to the user"]):::neutral
+end
+
+USE_CACHE --> CALC
+
+subgraph Fallback["FALLBACK"]
+    direction TB
+
+    ACQUIRE_CONN["acquire pool connection<br>via the configured balance strategy"]
+
+    ACQUIRE_CONN --> CALL_CONN["run the query on<br>the chosen connection"]
+
+    CALL_CONN --> RESULT2(["give result to the user"]):::neutral
+end
+```

picopyn-2.0.0.dev1/docs/diagrams/_reconcile_node_outside_pool_died.md

@@ -0,0 +1,50 @@
+```mermaid
+---
+title: A node outside the pool died
+---
+
+flowchart TD
+
+classDef offline fill:#ffebee,stroke:#f44336,color:#b71c1c
+classDef online fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
+classDef neutral fill:#f5f5f5,stroke:#9e9e9e,color:#424242
+
+classDef note fill:#fffde7,stroke:#f9a825,stroke-dasharray:4,color:#555
+
+subgraph before["Before"]
+    subgraph Picodata_before["Picodata"]
+        p1[("ins 1")]:::online
+        p2[("ins 2")]:::online
+        p3[("ins 3")]:::online
+        p4[("ins 4")]:::online
+    end
+
+    subgraph Pool_before["Pool: max_size=3"]
+        c1["conn"] --> p1
+        c2["conn"] --> p2
+        c3["conn"] --> p3
+    end
+end
+EVENT1[/"Event<br>Instance 4 dead"/]:::neutral -.-> REFILL{{"reconcile pool"}}
+before -.-> REFILL
+
+NOTE>"Node outside the pool died<br>=> pool stays unchanged"]:::note
+NOTE-.->REFILL
+
+REFILL --> after
+
+subgraph after["After"]
+    subgraph Picodata_after["Picodata"]
+        p21[("ins 1")]:::online
+        p22[("ins 2")]:::online
+        p23[("ins 3")]:::online
+        p24[("ins 4")]:::offline
+    end
+
+    subgraph Pool_after["Pool: max_size=3"]
+        c21["conn"] --> p21
+        c22["conn"] --> p22
+        c23["conn"] --> p23
+    end
+end
+```