leangeo 3.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- leangeo-3.0.0/.cargo/config.toml +7 -0
- leangeo-3.0.0/.gitignore +39 -0
- leangeo-3.0.0/CLIENT.md +398 -0
- leangeo-3.0.0/Cargo.lock +737 -0
- leangeo-3.0.0/Cargo.toml +35 -0
- leangeo-3.0.0/LICENSE +21 -0
- leangeo-3.0.0/MANIFEST.in +8 -0
- leangeo-3.0.0/PKG-INFO +446 -0
- leangeo-3.0.0/README.md +408 -0
- leangeo-3.0.0/client.py +21 -0
- leangeo-3.0.0/leangeo_engine/__init__.py +721 -0
- leangeo-3.0.0/leangeo_engine/_compat.py +177 -0
- leangeo-3.0.0/leangeo_engine/_guard.py +782 -0
- leangeo-3.0.0/leangeo_pb2.py +20 -0
- leangeo-3.0.0/leangeo_pb2_grpc.py +20 -0
- leangeo-3.0.0/leangeo_server/__init__.py +13 -0
- leangeo-3.0.0/leangeo_server/cli.py +81 -0
- leangeo-3.0.0/leangeo_server/client.py +308 -0
- leangeo-3.0.0/leangeo_server/leangeo.proto +109 -0
- leangeo-3.0.0/leangeo_server/leangeo_pb2.py +92 -0
- leangeo-3.0.0/leangeo_server/leangeo_pb2_grpc.py +700 -0
- leangeo-3.0.0/leangeo_server/py.typed +0 -0
- leangeo-3.0.0/leangeo_server/serve.py +693 -0
- leangeo-3.0.0/leangeo_server/serve_common.py +189 -0
- leangeo-3.0.0/leangeo_server/serve_grpc.py +517 -0
- leangeo-3.0.0/leangeo_server/serve_guardrails.py +714 -0
- leangeo-3.0.0/pyproject.toml +79 -0
- leangeo-3.0.0/serve.py +20 -0
- leangeo-3.0.0/serve_grpc.py +20 -0
- leangeo-3.0.0/serve_guardrails.py +20 -0
- leangeo-3.0.0/src/bitslice.rs +488 -0
- leangeo-3.0.0/src/grid_index.rs +310 -0
- leangeo-3.0.0/src/integrity.rs +192 -0
- leangeo-3.0.0/src/inverted_index.rs +239 -0
- leangeo-3.0.0/src/lib.rs +1979 -0
- leangeo-3.0.0/src/point.rs +187 -0
- leangeo-3.0.0/src/python.rs +545 -0
- leangeo-3.0.0/src/query.rs +488 -0
- leangeo-3.0.0/src/storage.rs +2163 -0
- leangeo-3.0.0/src/sysutil.rs +474 -0
- leangeo-3.0.0/src/tests.rs +424 -0
- leangeo-3.0.0/src/wal.rs +384 -0
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Align loop starts to 32 bytes on x86_64: a hot loop of up to 32 bytes then never
|
|
2
|
+
# straddles a 64-byte line. Without it, where each loop landed was luck of the build, and
|
|
3
|
+
# a straddling loop (UTF-8 validation, roaring intersections) ran 10-20% slower on Zen 2
|
|
4
|
+
# (measured with perf, HISTORY.md 2026-10-22). Cargo applies this to the release wheels
|
|
5
|
+
# too (maturin runs cargo in this directory).
|
|
6
|
+
[target.'cfg(target_arch = "x86_64")']
|
|
7
|
+
rustflags = ["-C", "llvm-args=-align-loops=32"]
|
leangeo-3.0.0/.gitignore
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
/target
|
|
2
|
+
**/*.rs.bk
|
|
3
|
+
__pycache__/
|
|
4
|
+
*.py[cod]
|
|
5
|
+
*$py.class
|
|
6
|
+
*.so
|
|
7
|
+
.Python
|
|
8
|
+
build/
|
|
9
|
+
develop-eggs/
|
|
10
|
+
dist/
|
|
11
|
+
downloads/
|
|
12
|
+
eggs/
|
|
13
|
+
.eggs/
|
|
14
|
+
lib/
|
|
15
|
+
lib64/
|
|
16
|
+
parts/
|
|
17
|
+
sdist/
|
|
18
|
+
var/
|
|
19
|
+
wheels/
|
|
20
|
+
*.egg-info/
|
|
21
|
+
.installed.cfg
|
|
22
|
+
*.egg
|
|
23
|
+
*.mmap
|
|
24
|
+
*.db
|
|
25
|
+
.venv/
|
|
26
|
+
venv/
|
|
27
|
+
|
|
28
|
+
# test/debug artifacts
|
|
29
|
+
datetest/
|
|
30
|
+
*.commit
|
|
31
|
+
*.wal
|
|
32
|
+
*.lock
|
|
33
|
+
*.epoch
|
|
34
|
+
*.base
|
|
35
|
+
*.dels*.mmap
|
|
36
|
+
*.ids*.mmap
|
|
37
|
+
.pytest_cache/
|
|
38
|
+
!Cargo.lock
|
|
39
|
+
*.from2x
|
leangeo-3.0.0/CLIENT.md
ADDED
|
@@ -0,0 +1,398 @@
|
|
|
1
|
+
# LeanGeo API and Client Guide
|
|
2
|
+
|
|
3
|
+
LeanGeo can run as a network service over **gRPC** or **REST** (HTTP/JSON). Both servers
|
|
4
|
+
and a Python gRPC client ship in the wheel, in the `leangeo_server` package:
|
|
5
|
+
|
|
6
|
+
- gRPC server: `leangeo_server.serve_grpc` (command `leangeo-grpc`), service defined in
|
|
7
|
+
`leangeo_server/leangeo.proto`
|
|
8
|
+
- REST server: `leangeo_server.serve` (command `leangeo-serve`), FastAPI on uvicorn
|
|
9
|
+
- Python client: `leangeo_server.client.LeanGeoClient`
|
|
10
|
+
|
|
11
|
+
`import leangeo_engine` never imports any of this. In a source checkout, the root-level
|
|
12
|
+
`serve.py`, `serve_grpc.py`, `serve_guardrails.py`, `client.py`, `leangeo_pb2.py` and
|
|
13
|
+
`leangeo_pb2_grpc.py` are aliases of the packaged modules, so `uvicorn serve:app`,
|
|
14
|
+
`python serve_grpc.py` and `from client import LeanGeoClient` still work from the
|
|
15
|
+
repository root. The proto moved from `proto/leangeo.proto` to `leangeo_server/leangeo.proto`.
|
|
16
|
+
|
|
17
|
+
**Polygon vertex order changed meaning in 3.0 — read [Polygons](#polygons) if you used
|
|
18
|
+
the 2.x servers.**
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## 1. Install
|
|
23
|
+
|
|
24
|
+
The engine has no server dependencies. Install the extras into the environment where
|
|
25
|
+
`leangeo` is installed:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
pip install "leangeo[server]" # REST server: fastapi, uvicorn[standard], pydantic>=2.8, orjson
|
|
29
|
+
pip install "leangeo[grpc]" # gRPC server and client: grpcio>=1.82.1, protobuf>=7.35,<8
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The generated stubs (`leangeo_pb2.py`, `leangeo_pb2_grpc.py`) ship in the wheel, so
|
|
33
|
+
`grpcio-tools` is only needed to regenerate them (section 9).
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## 2. Run a server
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
leangeo-serve --path /data/geo # REST, default 127.0.0.1:8003
|
|
41
|
+
leangeo-grpc --path /data/geo # gRPC, default 127.0.0.1:50053
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Equivalent forms: `LEANGEO_PATH=/data/geo python -m leangeo_server.serve_grpc`, and
|
|
45
|
+
`LEANGEO_PATH=/data/geo uvicorn leangeo_server.serve:app --workers 1` for REST (uvicorn
|
|
46
|
+
binds 127.0.0.1 unless you pass `--host`; see section 3 for what a non-loopback `--host`
|
|
47
|
+
requires).
|
|
48
|
+
|
|
49
|
+
Command-line flags only set the environment variables below, so flags and environment
|
|
50
|
+
configure the same thing. `leangeo-serve` takes `--path --capacity --host --port
|
|
51
|
+
--threads --allow-insecure`; `leangeo-grpc` takes `--path --capacity --host --port
|
|
52
|
+
--workers --tls-cert --tls-key --allow-insecure`. The bearer token is read from
|
|
53
|
+
`LEANGEO_API_TOKEN` only, so it never shows up in `ps`.
|
|
54
|
+
|
|
55
|
+
Each server opens one `LeanGeoEngine` for its lifetime and runs every request on a worker
|
|
56
|
+
thread (the gRPC `ThreadPoolExecutor`; FastAPI's threadpool for the REST endpoints, so no
|
|
57
|
+
engine call runs on the event loop). The native core releases the GIL, so reads run in
|
|
58
|
+
parallel; writes to one collection serialize. Run one process per data directory; the
|
|
59
|
+
engine is also safe across processes if you run several.
|
|
60
|
+
|
|
61
|
+
### Configuration
|
|
62
|
+
|
|
63
|
+
| Variable | Default | Meaning |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| `LEANGEO_PATH` | `leangeo_data` | Data directory (one set of files per collection) |
|
|
66
|
+
| `LEANGEO_CAPACITY` | `100000` | Initial capacity hint for a new collection's files |
|
|
67
|
+
| `LEANGEO_HOST` / `LEANGEO_PORT` | `127.0.0.1` / `8003` | REST bind address (`HOST`/`PORT` are read when these are unset) |
|
|
68
|
+
| `LEANGEO_GRPC_HOST` / `LEANGEO_GRPC_PORT` | `127.0.0.1` / `50053` | gRPC bind address (`GRPC_HOST`/`GRPC_PORT` are read when these are unset) |
|
|
69
|
+
| `LEANGEO_THREADS` | `min(64, 2 x CPUs)` | REST worker threads |
|
|
70
|
+
| `LEANGEO_GRPC_WORKERS` | `min(64, 2 x CPUs)` | gRPC worker threads |
|
|
71
|
+
| `LEANGEO_API_TOKEN` | unset | Bearer token required on every endpoint/RPC except health |
|
|
72
|
+
| `LEANGEO_ALLOW_INSECURE` | unset | `1` permits non-loopback access without authentication |
|
|
73
|
+
| `LEANGEO_GRPC_TLS_CERT` / `LEANGEO_GRPC_TLS_KEY` | unset | PEM certificate chain and key; gRPC serves TLS when both are set |
|
|
74
|
+
| `LEANGEO_GRPC_MAX_CONCURRENT_RPCS` | unset (unbounded) | Running + queued RPCs; past it gRPC answers `RESOURCE_EXHAUSTED` immediately. Must be >= 1 |
|
|
75
|
+
| `LEANGEO_GRPC_KEEPALIVE_TIME_MS` / `..._TIMEOUT_MS` | `60000` / `20000` | gRPC keepalive pings; a peer that stops answering is dropped and its RPCs cancelled |
|
|
76
|
+
| `LEANGEO_WAL_SYNC_INTERVAL_MS` | unset | Unset: every acknowledged write is fsynced before the reply. N > 0: fsync within N ms (a crash may lose the last N ms) |
|
|
77
|
+
|
|
78
|
+
Guardrail variables are listed in section 7.
|
|
79
|
+
|
|
80
|
+
### Shutdown
|
|
81
|
+
|
|
82
|
+
On SIGTERM or SIGINT both servers stop accepting work, let in-flight requests finish, and
|
|
83
|
+
close the engine, which checkpoints every collection (the WAL is folded into the data
|
|
84
|
+
files and emptied). The gRPC server waits up to 5 s for in-flight RPCs; REST waits up to
|
|
85
|
+
30 s. From the moment shutdown begins, gRPC `Health` reports `ready: false` and other
|
|
86
|
+
RPCs answer `UNAVAILABLE`; REST `/health` reports `"ready": false` and other endpoints
|
|
87
|
+
answer 503.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## 3. Trust boundary, authentication, TLS
|
|
92
|
+
|
|
93
|
+
The API can read, delete and clear every collection and has no per-collection
|
|
94
|
+
authorization. Both servers therefore bind `127.0.0.1` by default, and the console
|
|
95
|
+
scripts refuse to start on any other address (exit with a message) unless one of these
|
|
96
|
+
holds:
|
|
97
|
+
|
|
98
|
+
- REST: `LEANGEO_API_TOKEN` is set. Every endpoint except `GET /health` then requires
|
|
99
|
+
`Authorization: Bearer <token>`; a missing or wrong token is 401. Tokens are compared
|
|
100
|
+
in constant time.
|
|
101
|
+
- gRPC: `LEANGEO_GRPC_TLS_CERT`, `LEANGEO_GRPC_TLS_KEY` **and** `LEANGEO_API_TOKEN` are all
|
|
102
|
+
set. Every RPC except `Health` then requires `authorization: Bearer <token>` metadata
|
|
103
|
+
(unary and streaming handlers alike); a missing or wrong token is `UNAUTHENTICATED`.
|
|
104
|
+
A token without TLS, or TLS without a token, is refused on a non-loopback address.
|
|
105
|
+
- `LEANGEO_ALLOW_INSECURE=1`: you state that the network is already trusted (for example,
|
|
106
|
+
an authenticating reverse proxy in front of a private listener).
|
|
107
|
+
|
|
108
|
+
The REST server also enforces this per request, on the address each connection actually
|
|
109
|
+
reached, because an ASGI app cannot see how it was bound: when started directly with
|
|
110
|
+
`uvicorn leangeo_server.serve:app --host 0.0.0.0` (or any non-loopback `--host`) and no
|
|
111
|
+
token, every request that arrives on a non-loopback address is refused with 403, except
|
|
112
|
+
`GET /health`; requests through `127.0.0.1` are served. Set `LEANGEO_API_TOKEN` (or
|
|
113
|
+
`LEANGEO_ALLOW_INSECURE=1`) to serve other addresses. A reverse proxy on the same host
|
|
114
|
+
that connects to `127.0.0.1` is served without a token.
|
|
115
|
+
|
|
116
|
+
The token also works on loopback: setting `LEANGEO_API_TOKEN` always enables the check.
|
|
117
|
+
IPv6 binds work for both servers (`LEANGEO_GRPC_HOST=::1`; the address is bracketed for
|
|
118
|
+
gRPC).
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## 4. REST API
|
|
123
|
+
|
|
124
|
+
All bodies are JSON. Request models reject unknown fields with 422 (a misspelled
|
|
125
|
+
`filter` must not run a query unfiltered), and so are unknown query parameters
|
|
126
|
+
(`GET /count?colection=x` is 422, not a count of `default`). Every `collection` defaults
|
|
127
|
+
to `"default"`. Collection names are letters, digits, `-`, `_` and `.`, not starting with
|
|
128
|
+
`.` (the engine enforces this; anything else is 400).
|
|
129
|
+
|
|
130
|
+
| Method and path | Body / query | Response |
|
|
131
|
+
|---|---|---|
|
|
132
|
+
| `GET /health` | | `{"status": "ok", "ready": bool, "limits": {...}}` |
|
|
133
|
+
| `GET /count` | `?collection=` | `{"count": int}` |
|
|
134
|
+
| `GET /collections` | | `{"collections": [str, ...]}` |
|
|
135
|
+
| `POST /insert` | `{"latitude", "longitude", "metadata"?, "collection"?}` | `{"id": int}` |
|
|
136
|
+
| `POST /insert_batch` | `[{"latitude", "longitude", "metadata"?, "collection"?}, ...]` or `{"items": [...], "collection"?}` | `{"ids": [int, ...], "count": int}` |
|
|
137
|
+
| `POST /query` | `{"filter"?, "limit"?, "collection"?}` | `{"results": [...], "truncated": bool}` |
|
|
138
|
+
| `POST /query_radius` | `{"center_lat", "center_lon", "radius_meters", "filter"?, "limit"?, "collection"?}` | same |
|
|
139
|
+
| `POST /query_bbox` | `{"min_lat", "min_lon", "max_lat", "max_lon", "filter"?, "limit"?, "collection"?}` | same |
|
|
140
|
+
| `POST /query_polygon` | `{"coordinates": [[lon, lat], ...], "filter"?, "limit"?, "collection"?}` | same |
|
|
141
|
+
| `POST /exists` | `{"filter"?, "collection"?}` | `{"exists": bool}` |
|
|
142
|
+
| `POST /delete` | `{"filter", "collection"?}` (filter required) | `{"deleted": int}` |
|
|
143
|
+
| `POST /clear` | `{"collection", "confirm": true}` (both required) | `{"deleted": int}` |
|
|
144
|
+
| `POST /compact` | `{"collection"?}` | `{"reclaimed": int}` |
|
|
145
|
+
| `POST /persist` | none | `{"status": "ok"}` |
|
|
146
|
+
| `GET /verify` | `?collection=&flush=false` | `{"ok": bool, "errors": [str, ...]}` |
|
|
147
|
+
|
|
148
|
+
Result rows are `{"id": int, "latitude": float, "longitude": float, "metadata": {...}}`,
|
|
149
|
+
the same dicts the library returns, in the same order.
|
|
150
|
+
|
|
151
|
+
Notes:
|
|
152
|
+
|
|
153
|
+
- **Ids** are integers per collection, stable, and never reused.
|
|
154
|
+
- **`/insert_batch`** is atomic per collection: it makes one `engine.insert_batch` call
|
|
155
|
+
(one durable WAL record) per collection in the request, so one invalid point means none
|
|
156
|
+
of the batch is written. The 2.x server inserted point by point (one fsync each, and a
|
|
157
|
+
bad point left the earlier ones written). A batch that spans several collections is
|
|
158
|
+
validated in full (names, coordinates) before the first collection is written; after
|
|
159
|
+
that each collection commits on its own, so only a storage failure between two
|
|
160
|
+
collections can leave the earlier ones written. In the `{"items": ...}` form, an item
|
|
161
|
+
without its own `collection` uses the batch's.
|
|
162
|
+
- **`limit`** is optional on every read. Omitted means "everything, up to the server
|
|
163
|
+
cap"; `0` returns no rows (as in the library, after validating the filter); negative
|
|
164
|
+
is 400. See section 7 for `truncated`.
|
|
165
|
+
- **`collection`** defaults to `"default"` when omitted; an explicit `""` is refused with
|
|
166
|
+
400, as the library refuses it.
|
|
167
|
+
- **`/query_bbox`** with `min_lon > max_lon` crosses the antimeridian.
|
|
168
|
+
- **`/delete`** refuses (400, nothing deleted) a filter that matches every point by its
|
|
169
|
+
shape (`{}`, `{"f": {"$nin": []}}`, ...) or because it tests a field no point has (a
|
|
170
|
+
misspelled name). `/clear` is the explicit wipe and needs `"confirm": true`.
|
|
171
|
+
- **`/compact`** reclaims the slots of deleted points now (it also runs in the background).
|
|
172
|
+
Ids do not change.
|
|
173
|
+
- **`/persist`** checkpoints every open collection; a failure is 500 with the detail.
|
|
174
|
+
- **`/verify`** recomputes the checksum of every committed region; `flush=true` checkpoints
|
|
175
|
+
first.
|
|
176
|
+
|
|
177
|
+
Example:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
curl -s localhost:8003/insert_batch -H 'content-type: application/json' -d '[
|
|
181
|
+
{"latitude": -23.55, "longitude": -46.63, "metadata": {"city": "SP"}},
|
|
182
|
+
{"latitude": -22.91, "longitude": -43.17, "metadata": {"city": "RJ"}}]'
|
|
183
|
+
curl -s localhost:8003/query_radius -H 'content-type: application/json' \
|
|
184
|
+
-d '{"center_lat": -23.0, "center_lon": -45.0, "radius_meters": 500000, "filter": {"city": "RJ"}}'
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
### Filters
|
|
188
|
+
|
|
189
|
+
Filters are the MongoDB-style contract shared by the Lean* engines (top-level metadata
|
|
190
|
+
fields; equality, `$eq $ne $gt $gte $lt $lte $in $nin $exists`, `$and $or $nor $not`, and
|
|
191
|
+
array membership). The engine validates them; an unknown operator, a
|
|
192
|
+
malformed shape, or a NaN/infinite number is 400 (`INVALID_ARGUMENT`), never silently
|
|
193
|
+
treated as "match all" or "match null". Over REST, `null` or an omitted `filter` means no
|
|
194
|
+
filter on reads.
|
|
195
|
+
|
|
196
|
+
---
|
|
197
|
+
|
|
198
|
+
## 5. Polygons
|
|
199
|
+
|
|
200
|
+
Polygon vertices are **(longitude, latitude)** pairs, GeoJSON order, everywhere in 3.0:
|
|
201
|
+
|
|
202
|
+
- library: `engine.query_polygon([(lon, lat), ...])`
|
|
203
|
+
- REST: `"coordinates": [[lon, lat], ...]`; objects `{"latitude": .., "longitude": ..}`
|
|
204
|
+
are accepted too and are unambiguous
|
|
205
|
+
- gRPC: `repeated LatLon coordinates`, a message with named `latitude` and `longitude`
|
|
206
|
+
fields; the server hands the engine `(longitude, latitude)`
|
|
207
|
+
- Python client: `query_polygon([(lon, lat), ...])`, or dicts with `"latitude"` and
|
|
208
|
+
`"longitude"` keys
|
|
209
|
+
|
|
210
|
+
**Migration from 2.x.** The 2.x `CLIENT.md` and `client.py` documented `(lat, lon)`
|
|
211
|
+
tuples, and the 2.x gRPC server built `(latitude, longitude)` tuples from `LatLon`
|
|
212
|
+
messages, which swapped the axes relative to the library. A 2.x Python caller that
|
|
213
|
+
passed `(lat, lon)` tuples to `LeanGeoClient.query_polygon` must swap them (or pass dicts).
|
|
214
|
+
A raw gRPC caller that filled `LatLon` by field name needs no change: those messages now
|
|
215
|
+
mean what their field names say. Containment is planar; split a polygon that crosses
|
|
216
|
+
the antimeridian yourself. Fewer than 3 vertices matches nothing. The vertex count is
|
|
217
|
+
capped (section 7).
|
|
218
|
+
|
|
219
|
+
---
|
|
220
|
+
|
|
221
|
+
## 6. gRPC API and the Python client
|
|
222
|
+
|
|
223
|
+
Service `leangeo.LeanGeo` (`leangeo_server/leangeo.proto`). Metadata, filters and result
|
|
224
|
+
rows travel as JSON strings (`metadata_json`, `filter_json`, `results_json`) so their JSON
|
|
225
|
+
types survive; coordinates are doubles. An empty `filter_json` means no filter on reads.
|
|
226
|
+
`collection` and `limit` are proto3 `optional` fields: unset `collection` means
|
|
227
|
+
`"default"` and an explicit `""` is `INVALID_ARGUMENT` (as in the library and REST);
|
|
228
|
+
unset `limit` means no limit of your own (the server cap still applies) and `limit = 0`
|
|
229
|
+
returns no rows after validating the request. All 2.x field numbers are unchanged and the
|
|
230
|
+
encoding is the same (a 2.x client never sends `""` or `0`, so its requests keep their
|
|
231
|
+
2.x meaning); 3.0 only adds fields and RPCs.
|
|
232
|
+
|
|
233
|
+
| RPC | Request | Reply |
|
|
234
|
+
|---|---|---|
|
|
235
|
+
| `Health` | | `ready`, `limits_json` |
|
|
236
|
+
| `Count` | `collection` | `count` |
|
|
237
|
+
| `ListCollections` (3.0) | | `collections` |
|
|
238
|
+
| `Insert` | `point {latitude, longitude, metadata_json, collection}` | `id` |
|
|
239
|
+
| `InsertBatch` | `repeated Point points` | `repeated ids` (atomic per collection, as REST) |
|
|
240
|
+
| `Query` | `filter_json, limit, collection` | `results_json, truncated` |
|
|
241
|
+
| `QueryRadius` | `center_lat, center_lon, radius_meters, filter_json, collection, limit` | same |
|
|
242
|
+
| `QueryBbox` | `min_lat, min_lon, max_lat, max_lon, filter_json, collection, limit` | same |
|
|
243
|
+
| `QueryPolygon` | `repeated LatLon coordinates, filter_json, collection, limit` | same |
|
|
244
|
+
| `Exists` (3.0) | `filter_json, collection` | `exists` |
|
|
245
|
+
| `Delete` | `filter_json` (required), `collection` | `deleted` |
|
|
246
|
+
| `Clear` (3.0) | `collection, confirm` (must be true) | `deleted` |
|
|
247
|
+
| `Compact` (3.0) | `collection` | `reclaimed` |
|
|
248
|
+
| `Persist` | | `status` |
|
|
249
|
+
| `Verify` | `collection, flush` | `json` (`{"ok", "errors"}`) |
|
|
250
|
+
|
|
251
|
+
A `filter_json` or `metadata_json` that is not a JSON object (`[]`, `0`, `"x"`) is
|
|
252
|
+
`INVALID_ARGUMENT`.
|
|
253
|
+
|
|
254
|
+
### LeanGeoClient
|
|
255
|
+
|
|
256
|
+
```python
|
|
257
|
+
from leangeo_server.client import LeanGeoClient
|
|
258
|
+
|
|
259
|
+
with LeanGeoClient("localhost:50053", timeout=30) as c:
|
|
260
|
+
c.wait_ready()
|
|
261
|
+
pid = c.insert(-23.55, -46.63, {"city": "SP"})
|
|
262
|
+
ids = c.insert_batch([{"latitude": -22.91, "longitude": -43.17, "metadata": {"city": "RJ"}},
|
|
263
|
+
{"latitude": 40.71, "longitude": -74.0, "collection": "us"}])
|
|
264
|
+
near = c.query_radius(-23.0, -45.0, 500_000, filter={"city": "RJ"})
|
|
265
|
+
box = c.query_bbox(-40, -80, 10, -30, limit=100)
|
|
266
|
+
inside = c.query_polygon([(-50, -30), (-40, -30), (-40, -20), (-50, -20)]) # (lon, lat)
|
|
267
|
+
rows, truncated = c.query({"city": {"$in": ["SP", "RJ"]}}, with_truncated=True)
|
|
268
|
+
c.delete({"city": "SP"})
|
|
269
|
+
c.clear("us", confirm=True)
|
|
270
|
+
c.compact(); c.persist(); print(c.verify(), c.count(), c.list_collections(), c.limits())
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
Constructor: `LeanGeoClient(target="localhost:50053", *, timeout=None, channel=None,
|
|
274
|
+
options=None, token=None, tls=False, allow_insecure_token=False)`.
|
|
275
|
+
|
|
276
|
+
- `token` defaults to `LEANGEO_API_TOKEN` from the environment and is sent as
|
|
277
|
+
`authorization: Bearer <token>`.
|
|
278
|
+
- `tls=True` uses the system roots; `tls=<PEM bytes>` trusts that root certificate (for a
|
|
279
|
+
self-signed server). The client refuses to send a token over a plaintext channel unless
|
|
280
|
+
`allow_insecure_token=True`.
|
|
281
|
+
- The channel is thread-safe and multiplexes concurrent RPCs: share one client across
|
|
282
|
+
threads.
|
|
283
|
+
|
|
284
|
+
Methods: `health()`, `limits()`, `count(collection)`, `list_collections()`,
|
|
285
|
+
`insert(lat, lon, metadata, collection)`, `insert_batch(points, collection)`,
|
|
286
|
+
`query(filter, limit, collection)`, `query_radius(center_lat, center_lon, radius_meters,
|
|
287
|
+
filter, collection, limit)`, `query_bbox(min_lat, min_lon, max_lat, max_lon, filter,
|
|
288
|
+
collection, limit)`, `query_polygon(coordinates, filter, collection, limit)`,
|
|
289
|
+
`exists(filter, collection)`, `delete(filter, collection)`, `clear(collection, *,
|
|
290
|
+
confirm)`, `compact(collection)`, `persist()`, `verify(collection, flush)`. The read
|
|
291
|
+
methods take `with_truncated=True` to return `(rows, truncated)`. Every method takes a
|
|
292
|
+
keyword `timeout` that overrides the default deadline.
|
|
293
|
+
|
|
294
|
+
Errors surface as `grpc.RpcError`:
|
|
295
|
+
|
|
296
|
+
```python
|
|
297
|
+
import grpc
|
|
298
|
+
try:
|
|
299
|
+
c.delete({})
|
|
300
|
+
except grpc.RpcError as e:
|
|
301
|
+
print(e.code(), e.details()) # StatusCode.INVALID_ARGUMENT, "delete_by_metadata requires ..."
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
---
|
|
305
|
+
|
|
306
|
+
## 7. Limits and guardrails
|
|
307
|
+
|
|
308
|
+
Static caps reject oversized requests before they reach the engine. They can be set
|
|
309
|
+
through the environment, but **only lowered**: a value above the default (or one that does
|
|
310
|
+
not parse) leaves the default in place.
|
|
311
|
+
|
|
312
|
+
| Variable | Default | Bounds |
|
|
313
|
+
|---|---|---|
|
|
314
|
+
| `LEANGEO_MAX_BODY_BYTES` | 128 MiB | One REST body (413 past it); the gRPC send/receive message size |
|
|
315
|
+
| `LEANGEO_MAX_BATCH` | 100000 | Points per `insert_batch` (429) |
|
|
316
|
+
| `LEANGEO_MAX_POLYGON_VERTICES` | 100000 | Vertices per polygon query (429) |
|
|
317
|
+
| `LEANGEO_HARD_MAX_K` | 1000000 | Absolute ceiling on rows per read reply |
|
|
318
|
+
| `LEANGEO_MIN_K` | 1000 | Floor of the dynamic cap while memory allows it |
|
|
319
|
+
| `LEANGEO_MEM_FRACTION` | 0.5 | Share of available RAM that read replies may use (0 < f <= 1) |
|
|
320
|
+
| `LEANGEO_BODY_MEM_FRACTION` | `LEANGEO_MEM_FRACTION` | Share of available RAM that request bodies being parsed may use |
|
|
321
|
+
| `LEANGEO_PER_RESULT_BYTES` | 4096 | Memory estimate per returned row |
|
|
322
|
+
| `LEANGEO_MEM_REFRESH_SECS` | 1.0 | How often the memory figures are re-read |
|
|
323
|
+
| `LEANGEO_MAX_OPEN_COLLECTIONS` | (soft fd limit - 256) / 8, at most 4096 | Collections held open at once; past it the least recently used idle ones are closed (checkpointed) and reopen on their next use |
|
|
324
|
+
|
|
325
|
+
**Dynamic result cap.** The rows one reply may hold are
|
|
326
|
+
`clamp(available_RAM * MEM_FRACTION / workers / PER_RESULT_BYTES)` between `MIN_K` and
|
|
327
|
+
`HARD_MAX_K`, where available RAM is the smaller of `MemAvailable` and the remaining
|
|
328
|
+
allowance of this process's cgroup (v1 or v2). Every read passes a limit to the engine,
|
|
329
|
+
so rows are bounded before they are materialized: when your `limit` is within the cap
|
|
330
|
+
it is used as is; otherwise the server asks the engine for `cap + 1` rows, returns
|
|
331
|
+
`cap`, and sets `truncated: true` if the extra row existed. The serialized rows must also
|
|
332
|
+
fit a byte budget (the same memory share, at most 8 x `MAX_BODY_BYTES`; for gRPC also at
|
|
333
|
+
most the message cap, `MAX_BODY_BYTES`); serialization stops at the first row that would
|
|
334
|
+
pass it and `truncated` is set. The byte budget is also passed to the engine, which stops
|
|
335
|
+
building rows once it is reached, so rows with large metadata never materialize the
|
|
336
|
+
whole row cap. `truncated: false` means you got every row
|
|
337
|
+
you asked for. To read more than the cap, narrow the query (a smaller box or radius, a
|
|
338
|
+
filter) or partition it. The current figures are in `GET /health` (`limits`) and gRPC
|
|
339
|
+
`Health.limits_json`.
|
|
340
|
+
|
|
341
|
+
**Body admission (REST).** A declared `Content-Length` past `MAX_BODY_BYTES` is refused
|
|
342
|
+
with 413 before any byte is read. A chunked body, or any body sent with
|
|
343
|
+
`Transfer-Encoding`, is counted as it arrives and refused once it passes the cap. Every
|
|
344
|
+
nonempty body also reserves its estimated parse memory (about 4x its size, more for
|
|
345
|
+
token-dense JSON such as `[0,0,0,...]`) against a budget the bodies in flight share; a
|
|
346
|
+
body that does not fit the memory free right now is refused with 429 and `Retry-After: 1`
|
|
347
|
+
before it is parsed. Refusals close the connection. Authentication runs before
|
|
348
|
+
admission.
|
|
349
|
+
|
|
350
|
+
**gRPC.** Messages larger than `MAX_BODY_BYTES` are refused by gRPC itself
|
|
351
|
+
(`RESOURCE_EXHAUSTED`). The JSON strings in a request (filters, metadata) reserve their
|
|
352
|
+
estimated parse memory from the same kind of budget while the RPC runs; one that does not
|
|
353
|
+
fit is `RESOURCE_EXHAUSTED`. `LEANGEO_GRPC_MAX_CONCURRENT_RPCS` optionally bounds running
|
|
354
|
+
plus queued RPCs.
|
|
355
|
+
|
|
356
|
+
**Open collections.** Each open collection holds about 6 to 8 file descriptors. At start
|
|
357
|
+
the servers raise the soft `RLIMIT_NOFILE` to the hard limit (at most 65536), and they keep
|
|
358
|
+
at most `LEANGEO_MAX_OPEN_COLLECTIONS` collections open, closing the least recently used
|
|
359
|
+
idle ones; a collection a request is using is never closed under it. Running out of
|
|
360
|
+
descriptors anyway is reported as 503 / `UNAVAILABLE`, not as corruption.
|
|
361
|
+
|
|
362
|
+
---
|
|
363
|
+
|
|
364
|
+
## 8. Errors
|
|
365
|
+
|
|
366
|
+
| Condition | REST | gRPC | Retry? |
|
|
367
|
+
|---|---|---|---|
|
|
368
|
+
| Bad input: coordinates out of range or non-finite, bad filter, bad collection name, `InvalidInputError`, `ValueError`, `TypeError`; guarded delete; `clear` without confirmation | 400 | `INVALID_ARGUMENT` | no |
|
|
369
|
+
| Missing or wrong bearer token | 401 | `UNAUTHENTICATED` | no |
|
|
370
|
+
| No token configured and the request reached a non-loopback address (REST, section 3) | 403 | n/a (the server refuses to start) | no |
|
|
371
|
+
| Unknown route, or the engine reports a missing item (`KeyError`) | 404 | `NOT_FOUND` | no |
|
|
372
|
+
| Body past `LEANGEO_MAX_BODY_BYTES` (also an invalid `Content-Length`: 400) | 413 | `RESOURCE_EXHAUSTED` | no |
|
|
373
|
+
| Unknown body field or query parameter, wrong JSON types | 422 | n/a | no |
|
|
374
|
+
| Over a guardrail (batch size, vertices, memory for parsing, `LEANGEO_GRPC_MAX_CONCURRENT_RPCS`) | 429 + `Retry-After: 1` | `RESOURCE_EXHAUSTED` | later |
|
|
375
|
+
| `StorageError` (I/O failure, corruption, a poisoned collection), or any unexpected error | 500 | `INTERNAL` | no, needs an operator |
|
|
376
|
+
| Engine not open, closed (`ClosedError`), or shutting down; out of file descriptors (EMFILE/ENFILE) | 503 + `Retry-After: 1` | `UNAVAILABLE` | yes |
|
|
377
|
+
|
|
378
|
+
Error details are bounded to 512 characters. REST 422 bodies list the location and
|
|
379
|
+
message of at most 16 validation errors and do not echo the submitted values.
|
|
380
|
+
|
|
381
|
+
---
|
|
382
|
+
|
|
383
|
+
## 9. Regenerating the gRPC stubs
|
|
384
|
+
|
|
385
|
+
Only needed after editing `leangeo_server/leangeo.proto` in a source checkout:
|
|
386
|
+
|
|
387
|
+
```bash
|
|
388
|
+
pip install "grpcio-tools==1.82.1" # in a throwaway environment
|
|
389
|
+
python scripts/gen_grpc.py # rewrite leangeo_server/leangeo_pb2*.py
|
|
390
|
+
python scripts/gen_grpc.py --check # exit 1 if the checked-in stubs are stale
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
The script refuses any other grpcio-tools version: newer tools stamp a newer minimum
|
|
394
|
+
runtime into the stubs. It compiles with the proto's own directory as the include root
|
|
395
|
+
(descriptor `leangeo.proto`, package `leangeo`), so the wire protocol does not depend on
|
|
396
|
+
where the package lives, and rewrites the generated `import leangeo_pb2` into a relative
|
|
397
|
+
import. `tests/test_server_packaging_lifecycle.py` runs the `--check` when grpcio-tools
|
|
398
|
+
1.82.1 is importable or `LEANGEO_GRPC_TOOLS_PYTHON` points at a Python that has it.
|