ydb-qdrant 8.1.1 → 9.1.0

package/README.md

@@ -16,7 +16,7 @@
 
 # YDB Qdrant-compatible Service
 
-Qdrant-compatible Node.js/TypeScript **service and npm library** that stores and searches vectors in YDB using a global one-table layout (`qdrant_all_points`) with exact KNN search (single-phase over `embedding`) by default and an optional approximate mode (two‑phase bit-quantized over `embedding_quantized` + `embedding`). Topics: ydb, vector-search, qdrant-compatible, nodejs, typescript, express, yql, ann, semantic-search, rag.
+Qdrant-compatible Node.js/TypeScript **service and npm library** that stores and searches vectors in YDB using a global one-table layout (`qdrant_all_points`) with exact KNN search over `embedding`. Topics: ydb, vector-search, qdrant-compatible, nodejs, typescript, express, yql, ann, semantic-search, rag.
 
 Modes:
 - **HTTP server**: Qdrant-compatible REST API (`/collections`, `/points/*`) on top of YDB.
@@ -54,7 +54,16 @@ npm install
 ```
 
 ## Configure credentials
-The server uses `getCredentialsFromEnv()` and supports these env vars (first match wins):
+The server supports explicit programmatic `authService` in the npm API. Without it, env-based credentials are used:
+
+- Static username/password credentials (durable local YDB)
+  ```bash
+  export YDB_STATIC_CREDENTIALS_USER=qdrantapp
+  export YDB_STATIC_CREDENTIALS_PASSWORD_FILE=/run/secrets/qdrantapp.password
+  ```
+  `YDB_STATIC_CREDENTIALS_PASSWORD` is supported only as a fallback when a password file is not set. `YDB_STATIC_CREDENTIALS_AUTH_ENDPOINT` can override the auth endpoint; otherwise the YDB endpoint is used. For private CA TLS, set `YDB_SSL_ROOT_CERTIFICATES_FILE`.
+
+The remaining YDB SDK env credentials are resolved by `getCredentialsFromEnv()`:
 
 - Service account key file (recommended)
   ```bash
@@ -79,8 +88,8 @@ The server uses `getCredentialsFromEnv()` and supports these env vars (first mat
 
 Also set endpoint and database:
 ```bash
-export YDB_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135
-export YDB_DATABASE=/ru-central1/<cloud>/<db>
+export YDB_QDRANT_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135
+export YDB_QDRANT_DATABASE=/ru-central1/<cloud>/<db>
 ```
 
 Optional env:
@@ -88,23 +97,25 @@ Optional env:
 # Server
 export PORT=8080
 export LOG_LEVEL=info
-# One-table search tuning (default is 'exact' when unset)
-export YDB_QDRANT_SEARCH_MODE=approximate               # approximate or exact (default: exact)
-export YDB_QDRANT_OVERFETCH_MULTIPLIER=10               # candidate multiplier in approximate mode
+# One-table tuning
+export YDB_QDRANT_UPSERT_BATCH_SIZE=100
+export YDB_QDRANT_SEARCH_TIMEOUT_MS=10000
 ```
 
 ## Use as a Node.js library (npm package)
 
 The package entrypoint exports a programmatic API that mirrors the Qdrant HTTP semantics.
 
+- `createYdbQdrantClient()` requires exactly one namespace/signing identifier:
+  pass either `apiKey` or `userUid`, but not both.
+
 - Import and initialize a client (reuses the same YDB env vars as the server):
   ```ts
   import { createYdbQdrantClient } from "ydb-qdrant";
 
   async function main() {
-    // defaultTenant is optional; defaults to "default"
     const client = await createYdbQdrantClient({
-      defaultTenant: "myapp",
+      apiKey: "my-stable-namespace-key",
       endpoint: "grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135",
       database: "/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h",
     });
@@ -133,20 +144,20 @@ The package entrypoint exports a programmatic API that mirrors the Qdrant HTTP s
   }
   ```
 
-- Multi-tenant usage with `forTenant`:
+- Explicit namespace usage with `userUid`:
   ```ts
   const client = await createYdbQdrantClient({
+    userUid: "team_a",
     endpoint: "grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135",
     database: "/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h",
   });
-  const tenantClient = client.forTenant("tenant-a");
 
-  await tenantClient.upsertPoints("sessions", {
+  await client.upsertPoints("sessions", {
     points: [{ id: "s1", vector: [/* ... */] }],
   });
   ```
 
-The request/response shapes follow the same schemas as the HTTP API (`CreateCollectionReq`, `UpsertPointsReq`, `SearchReq`, `DeletePointsReq`), so code written against the REST API can usually be translated directly to the library calls.
+The request/response shapes follow the same schemas as the HTTP API (`CreateCollectionReq`, `UpsertPointsReq`, `SearchReq`, `DeletePointsReq`, `RetrievePointsReq`), so code written against the REST API can usually be translated directly to the library calls. `createYdbQdrantClient` requires exactly one of `apiKey` or `userUid`.
 
 ### Example: in-process points search with a shared client
 
@@ -160,7 +171,7 @@ let clientPromise: ReturnType<typeof createYdbQdrantClient> | null = null;
 async function getClient() {
   if (!clientPromise) {
     clientPromise = createYdbQdrantClient({
-      defaultTenant: 'myapp',
+      apiKey: 'myapp-shared-key',
       endpoint: 'grpcs://lb.etn01g9tcilcon2mrt3h.ydb.mdb.yandexcloud.net:2135',
       database: '/ru-central1/b1ge4v9r1l3h1q4njclp/etn01g9tcilcon2mrt3h',
     });
@@ -193,9 +204,9 @@ For full tables of popular embedding models and their dimensions, see [docs/vect
 
 **Option 1: Public Demo (No setup required)**
 - Set Qdrant URL to `http://ydb-qdrant.tech:8080`
-- No API key needed
+- `api-key` recommended for stable isolation
 - Free to use for testing and development
-- Shared instance - use `X-Tenant-Id` header for isolation
+- If `api-key` is omitted, anonymous isolation requires request client metadata such as IP or User-Agent
 
 **Option 2: Self-hosted (Local)**
 - Set Qdrant URL to `http://localhost:8080`
@@ -274,8 +285,8 @@ Basic example:
 ```bash
 docker run -d --name ydb-qdrant \
   -p 8080:8080 \
-  -e YDB_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135 \
-  -e YDB_DATABASE=/ru-central1/<cloud>/<db> \
+  -e YDB_QDRANT_ENDPOINT=grpcs://ydb.serverless.yandexcloud.net:2135 \
+  -e YDB_QDRANT_DATABASE=/ru-central1/<cloud>/<db> \
   -e YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS=/sa-key.json \
   -v /abs/path/sa-key.json:/sa-key.json:ro \
   ghcr.io/astandrik/ydb-qdrant:latest
@@ -330,7 +341,7 @@ curl -X POST http://localhost:8080/collections/mycol/points/delete \
 
 ## Architecture and Storage
 
-For details on the YDB one-table storage layout, vector serialization (full-precision and bit‑quantized), approximate vs exact search modes, request normalization, and Qdrant compatibility semantics, see [docs/architecture-and-storage.md](docs/architecture-and-storage.md).
+For details on the YDB one-table storage layout, vector serialization, request normalization, and Qdrant compatibility semantics, see [docs/architecture-and-storage.md](docs/architecture-and-storage.md).
 
 ## Evaluation, CI, and Release
 

package/dist/SmokeTest.js

@@ -1,9 +1,9 @@
 import "dotenv/config";
 import { createYdbQdrantClient } from "./package/api.js";
 async function main() {
-    const tenant = process.env.SMOKE_TENANT ?? "smoke";
     const collection = process.env.SMOKE_COLLECTION ?? "demo";
-    const client = await createYdbQdrantClient({ defaultTenant: tenant });
+    const apiKey = process.env.SMOKE_API_KEY ?? "smoke-demo-api-key";
+    const client = await createYdbQdrantClient({ apiKey });
     await client.createCollection(collection, {
         vectors: {
             size: 4,

package/dist/compute/ComputePool.d.ts

@@ -0,0 +1,5 @@
+import type { PrepareUpsertBatchTask, PreparedUpsertRow, VerifySearchRowsResult, VerifySearchRowsTask } from "./ComputeWorker.js";
+export declare function isComputePoolEnabled(): boolean;
+export declare function isComputePoolQueueAtLimitError(err: unknown): boolean;
+export declare function runPrepareUpsertBatch(task: PrepareUpsertBatchTask): Promise<PreparedUpsertRow[]>;
+export declare function runVerifySearchRows(task: VerifySearchRowsTask): Promise<VerifySearchRowsResult>;

package/dist/compute/ComputePool.js

@@ -0,0 +1,64 @@
+import { Piscina } from "piscina";
+import { WORKERS_ENABLED, WORKERS_IDLE_TIMEOUT_MS, WORKERS_MAX_QUEUE, WORKERS_MAX_THREADS, WORKERS_MIN_THREADS, } from "../config/env.js";
+import { logger } from "../logging/logger.js";
+let pool;
+function buildWorkerFilename() {
+    const isTsSource = import.meta.url.endsWith(".ts");
+    const workerUrl = new URL(isTsSource ? "./ComputeWorker.ts" : "./ComputeWorker.js", import.meta.url);
+    return workerUrl.href;
+}
+function buildExecArgv() {
+    const isTsSource = import.meta.url.endsWith(".ts");
+    if (!isTsSource) {
+        return undefined;
+    }
+    // Enable TypeScript execution in worker threads while running under tsx (dev).
+    // See: https://tsx.is/dev-api/
+    return ["--import", "tsx"];
+}
+function createPool() {
+    const filename = buildWorkerFilename();
+    const execArgv = buildExecArgv();
+    const p = new Piscina({
+        filename,
+        ...(execArgv ? { execArgv } : {}),
+        minThreads: WORKERS_MIN_THREADS,
+        maxThreads: WORKERS_MAX_THREADS,
+        ...(WORKERS_IDLE_TIMEOUT_MS >= 0
+            ? { idleTimeout: WORKERS_IDLE_TIMEOUT_MS }
+            : {}),
+        ...(WORKERS_MAX_QUEUE !== undefined ? { maxQueue: WORKERS_MAX_QUEUE } : {}),
+    });
+    p.on("error", (err) => {
+        logger.error({ err }, "Compute worker pool emitted an error");
+    });
+    return p;
+}
+function getPool() {
+    if (!pool) {
+        pool = createPool();
+    }
+    return pool;
+}
+export function isComputePoolEnabled() {
+    return WORKERS_ENABLED === true;
+}
+export function isComputePoolQueueAtLimitError(err) {
+    if (!(err instanceof Error)) {
+        return false;
+    }
+    // Piscina uses these messages for backpressure / overload conditions.
+    // See piscina/src/errors.ts in the dependency for the canonical strings.
+    return (err.message === "Task queue is at limit" ||
+        err.message === "No task queue available and all Workers are busy");
+}
+export async function runPrepareUpsertBatch(task) {
+    return (await getPool().run(task, {
+        name: "prepareUpsertBatch",
+    }));
+}
+export async function runVerifySearchRows(task) {
+    return (await getPool().run(task, {
+        name: "verifySearchRows",
+    }));
+}

package/dist/compute/ComputeWorker.d.ts

@@ -0,0 +1,36 @@
+import type { UpsertPoint } from "../qdrant/Requests.js";
+import type { YdbQdrantScoredPoint } from "../qdrant/QdrantRestTypes.js";
+export type PrepareUpsertBatchTask = {
+    collection: string;
+    apiKey: string;
+    batch: Array<Pick<UpsertPoint, "id" | "vector" | "payload">>;
+};
+export type PreparedUpsertRow = {
+    collection: string;
+    point_id: string;
+    embedding: Buffer;
+    payload: string;
+    payload_sign: string;
+    path_prefix: string | null;
+};
+export declare function prepareUpsertBatch(task: PrepareUpsertBatchTask): PreparedUpsertRow[];
+export type VerifySearchRowsTask = {
+    collection: string;
+    apiKey: string;
+    withPayload: boolean | undefined;
+    rows: Array<{
+        pointId: string;
+        payloadText: string | undefined;
+        payloadSign: string | undefined;
+        scoreText: string | undefined;
+        scoreFloat: number | undefined;
+    }>;
+};
+export type VerifySearchRowsResult = {
+    points: YdbQdrantScoredPoint[];
+    dropped: Array<{
+        pointId: string;
+        reason: "missing_payload_or_signature" | "signature_mismatch";
+    }>;
+};
+export declare function verifySearchRows(task: VerifySearchRowsTask): VerifySearchRowsResult;

package/dist/compute/ComputeWorker.js

@@ -0,0 +1,84 @@
+import { buildVectorBinaryParams } from "../ydb/helpers.js";
+import { Piscina, move, transferableSymbol, valueSymbol } from "piscina";
+import { computePayloadSign } from "../utils/PayloadSign.js";
+import { extractPathPrefix } from "../utils/pathPrefix.js";
+export function prepareUpsertBatch(task) {
+    const apiKey = task.apiKey.trim();
+    if (apiKey.length === 0) {
+        throw new Error("prepareUpsertBatch: apiKey is empty");
+    }
+    const rows = task.batch.map((p) => {
+        const binaries = buildVectorBinaryParams(p.vector);
+        const payloadObj = p.payload ?? {};
+        return {
+            collection: task.collection,
+            point_id: String(p.id),
+            embedding: binaries.float,
+            payload: JSON.stringify(payloadObj),
+            payload_sign: computePayloadSign({ apiKey, payload: payloadObj }),
+            path_prefix: extractPathPrefix(payloadObj),
+        };
+    });
+    if (!Piscina.isWorkerThread) {
+        return rows;
+    }
+    // Transfer underlying ArrayBuffers to the main thread to avoid cloning large binary payloads.
+    // See Node worker_threads transferList semantics and Piscina's Transferable interface:
+    // https://nodejs.org/api/worker_threads.html
+    // https://www.npmjs.com/package/piscina
+    const transferable = {
+        get [transferableSymbol]() {
+            const out = [];
+            for (const r of rows) {
+                out.push(r.embedding.buffer);
+            }
+            return out;
+        },
+        get [valueSymbol]() {
+            return rows;
+        },
+    };
+    return move(transferable);
+}
+export function verifySearchRows(task) {
+    const apiKey = task.apiKey.trim();
+    if (apiKey.length === 0) {
+        throw new Error("verifySearchRows: apiKey is empty");
+    }
+    const shouldReturnPayload = task.withPayload === true;
+    const points = [];
+    const dropped = [];
+    for (const row of task.rows) {
+        let payload;
+        if (row.payloadText) {
+            try {
+                payload = JSON.parse(row.payloadText);
+            }
+            catch {
+                payload = undefined;
+            }
+        }
+        if (!payload || typeof row.payloadSign !== "string" || row.payloadSign === "") {
+            dropped.push({
+                pointId: row.pointId,
+                reason: "missing_payload_or_signature",
+            });
+            continue;
+        }
+        const expected = computePayloadSign({ apiKey, payload });
+        if (expected !== row.payloadSign) {
+            dropped.push({
+                pointId: row.pointId,
+                reason: "signature_mismatch",
+            });
+            continue;
+        }
+        const score = Number(row.scoreFloat ?? row.scoreText);
+        points.push({
+            id: row.pointId,
+            score,
+            ...(shouldReturnPayload ? { payload } : {}),
+        });
+    }
+    return { points, dropped };
+}

package/dist/config/env.d.ts

@@ -1,20 +1,42 @@
 import "dotenv/config";
+export declare const YDB_QDRANT_ENDPOINT: string;
+export declare const YDB_QDRANT_DATABASE: string;
+export declare const YDB_STATIC_CREDENTIALS_USER: string;
+export declare const YDB_STATIC_CREDENTIALS_PASSWORD_FILE: string;
+export declare const YDB_STATIC_CREDENTIALS_PASSWORD: string;
+export declare const YDB_STATIC_CREDENTIALS_AUTH_ENDPOINT: string;
+export declare const YDB_SSL_ROOT_CERTIFICATES_FILE: string;
+export declare const LEGACY_YDB_ENDPOINT: string;
+export declare const LEGACY_YDB_DATABASE: string;
 export declare const YDB_ENDPOINT: string;
 export declare const YDB_DATABASE: string;
+export declare function resolveYdbConnectionConfig(options?: {
+    endpoint?: string;
+    database?: string;
+    connectionString?: string;
+}): {
+    connectionString: string;
+} | {
+    endpoint: string;
+    database: string;
+};
 export declare const PORT: number;
 export declare const LOG_LEVEL: string;
-export declare enum SearchMode {
-    Exact = "exact",
-    Approximate = "approximate"
-}
-export declare function resolveSearchMode(raw: string | undefined): SearchMode;
-export declare const SEARCH_MODE: SearchMode;
-export declare const OVERFETCH_MULTIPLIER: number;
 export declare const UPSERT_BATCH_SIZE: number;
+export declare const DELETE_FILTER_SELECT_BATCH_SIZE: number;
 export declare const SESSION_POOL_MIN_SIZE: number;
 export declare const SESSION_POOL_MAX_SIZE: number;
 export declare const SESSION_KEEPALIVE_PERIOD_MS: number;
+export declare const YDB_SESSION_RETRY_MAX_RETRIES: number;
 export declare const STARTUP_PROBE_SESSION_TIMEOUT_MS: number;
 export declare const UPSERT_OPERATION_TIMEOUT_MS: number;
+export declare const UPSERT_BODY_TIMEOUT_MS: number;
+export declare const UPSERT_HTTP_TIMEOUT_MS: number;
 export declare const SEARCH_OPERATION_TIMEOUT_MS: number;
+export declare const TABLE_SESSION_TIMEOUT_MS: number;
 export declare const LAST_ACCESS_MIN_WRITE_INTERVAL_MS: number;
+export declare const WORKERS_ENABLED: boolean;
+export declare const WORKERS_MAX_THREADS: number;
+export declare const WORKERS_MIN_THREADS: number;
+export declare const WORKERS_IDLE_TIMEOUT_MS: number;
+export declare const WORKERS_MAX_QUEUE: number | "auto" | undefined;

package/dist/config/env.js

@@ -1,47 +1,59 @@
 import "dotenv/config";
-export const YDB_ENDPOINT = process.env.YDB_ENDPOINT ?? "";
-export const YDB_DATABASE = process.env.YDB_DATABASE ?? "";
-export const PORT = process.env.PORT ? Number(process.env.PORT) : 8080;
-export const LOG_LEVEL = process.env.LOG_LEVEL ?? "info";
-function parseIntegerEnv(value, defaultValue, opts) {
-    if (value === undefined) {
-        return defaultValue;
-    }
-    const parsed = Number.parseInt(value.trim(), 10);
-    if (!Number.isFinite(parsed)) {
-        return defaultValue;
+import { parseBooleanEnv, parseIntegerEnv } from "../utils/EnvParsers.js";
+export const YDB_QDRANT_ENDPOINT = process.env.YDB_QDRANT_ENDPOINT?.trim() ?? "";
+export const YDB_QDRANT_DATABASE = process.env.YDB_QDRANT_DATABASE?.trim() ?? "";
+export const YDB_STATIC_CREDENTIALS_USER = process.env.YDB_STATIC_CREDENTIALS_USER?.trim() ?? "";
+export const YDB_STATIC_CREDENTIALS_PASSWORD_FILE = process.env.YDB_STATIC_CREDENTIALS_PASSWORD_FILE?.trim() ?? "";
+export const YDB_STATIC_CREDENTIALS_PASSWORD = process.env.YDB_STATIC_CREDENTIALS_PASSWORD ?? "";
+export const YDB_STATIC_CREDENTIALS_AUTH_ENDPOINT = process.env.YDB_STATIC_CREDENTIALS_AUTH_ENDPOINT?.trim() ?? "";
+export const YDB_SSL_ROOT_CERTIFICATES_FILE = process.env.YDB_SSL_ROOT_CERTIFICATES_FILE?.trim() ?? "";
+export const LEGACY_YDB_ENDPOINT = process.env.YDB_ENDPOINT?.trim() ?? "";
+export const LEGACY_YDB_DATABASE = process.env.YDB_DATABASE?.trim() ?? "";
+export const YDB_ENDPOINT = YDB_QDRANT_ENDPOINT;
+export const YDB_DATABASE = YDB_QDRANT_DATABASE;
+const LEGACY_YDB_ENDPOINT_ERROR = [
+    "Legacy env var YDB_ENDPOINT is not supported by ydb-qdrant.",
+    "Reason: ydb-sdk uses process.env.YDB_ENDPOINT as a dev-only override that forces all discovered endpoints to that host, breaking discovery and potentially causing session hangs/timeouts.",
+    "Fix: set YDB_QDRANT_ENDPOINT instead.",
+].join(" ");
+const MISSING_YDB_CONNECTION_SETTINGS_ERROR = [
+    "Missing YDB connection settings.",
+    "Set YDB_QDRANT_ENDPOINT (grpc(s)://host:port) and YDB_QDRANT_DATABASE (/path/to/db).",
+    "Legacy YDB_ENDPOINT/YDB_DATABASE are not supported as application config.",
+].join(" ");
+export function resolveYdbConnectionConfig(options) {
+    if (LEGACY_YDB_ENDPOINT) {
+        throw new Error(LEGACY_YDB_ENDPOINT_ERROR);
     }
-    let result = parsed;
-    if (opts?.min !== undefined && result < opts.min) {
-        result = opts.min;
+    const connectionString = options?.connectionString?.trim();
+    if (connectionString) {
+        return { connectionString };
     }
-    if (opts?.max !== undefined && result > opts.max) {
-        result = opts.max;
+    const endpoint = options?.endpoint?.trim() || YDB_QDRANT_ENDPOINT;
+    const database = options?.database?.trim() || YDB_QDRANT_DATABASE;
+    if (!endpoint || !database) {
+        throw new Error(MISSING_YDB_CONNECTION_SETTINGS_ERROR);
     }
-    return result;
+    return { endpoint, database };
 }
-export var SearchMode;
-(function (SearchMode) {
-    SearchMode["Exact"] = "exact";
-    SearchMode["Approximate"] = "approximate";
-})(SearchMode || (SearchMode = {}));
-export function resolveSearchMode(raw) {
-    const normalized = raw?.trim().toLowerCase();
-    if (normalized === SearchMode.Exact) {
-        return SearchMode.Exact;
+function parseWorkersMaxQueue(value) {
+    const raw = value?.trim().toLowerCase();
+    if (!raw) {
+        return "auto";
     }
-    if (normalized === SearchMode.Approximate) {
-        return SearchMode.Approximate;
+    if (raw === "auto") {
+        return "auto";
     }
-    // Default: exact search (single-phase over full-precision embedding) for the one-table layout.
-    return SearchMode.Exact;
+    const n = parseIntegerEnv(raw, 0, { min: 1 });
+    return n > 0 ? n : undefined;
 }
-function resolveSearchModeEnv() {
-    return resolveSearchMode(process.env.YDB_QDRANT_SEARCH_MODE);
-}
-export const SEARCH_MODE = resolveSearchModeEnv();
-export const OVERFETCH_MULTIPLIER = parseIntegerEnv(process.env.YDB_QDRANT_OVERFETCH_MULTIPLIER, 10, { min: 1 });
+export const PORT = parseIntegerEnv(process.env.PORT, 8080, {
+    min: 1,
+    max: 65535,
+});
+export const LOG_LEVEL = process.env.LOG_LEVEL ?? "info";
 export const UPSERT_BATCH_SIZE = parseIntegerEnv(process.env.YDB_QDRANT_UPSERT_BATCH_SIZE, 100, { min: 1 });
+export const DELETE_FILTER_SELECT_BATCH_SIZE = parseIntegerEnv(process.env.YDB_QDRANT_DELETE_FILTER_SELECT_BATCH_SIZE, 10000, { min: 1 });
 // Session pool configuration
 const RAW_SESSION_POOL_MIN_SIZE = parseIntegerEnv(process.env.YDB_SESSION_POOL_MIN_SIZE, 5, { min: 1, max: 500 });
 const RAW_SESSION_POOL_MAX_SIZE = parseIntegerEnv(process.env.YDB_SESSION_POOL_MAX_SIZE, 100, { min: 1, max: 500 });
@@ -51,7 +63,30 @@ const NORMALIZED_SESSION_POOL_MIN_SIZE = RAW_SESSION_POOL_MIN_SIZE > RAW_SESSION
 export const SESSION_POOL_MIN_SIZE = NORMALIZED_SESSION_POOL_MIN_SIZE;
 export const SESSION_POOL_MAX_SIZE = RAW_SESSION_POOL_MAX_SIZE;
 export const SESSION_KEEPALIVE_PERIOD_MS = parseIntegerEnv(process.env.YDB_SESSION_KEEPALIVE_PERIOD_MS, 5000, { min: 1000, max: 60000 });
+export const YDB_SESSION_RETRY_MAX_RETRIES = parseIntegerEnv(process.env.YDB_SESSION_RETRY_MAX_RETRIES, 3, { min: 0, max: 50 });
 export const STARTUP_PROBE_SESSION_TIMEOUT_MS = parseIntegerEnv(process.env.YDB_QDRANT_STARTUP_PROBE_SESSION_TIMEOUT_MS, 5000, { min: 1000 });
 export const UPSERT_OPERATION_TIMEOUT_MS = parseIntegerEnv(process.env.YDB_QDRANT_UPSERT_TIMEOUT_MS, 5000, { min: 1000 });
+const RAW_UPSERT_BODY_TIMEOUT_MS = process.env.YDB_QDRANT_UPSERT_BODY_TIMEOUT_MS?.trim();
+export const UPSERT_BODY_TIMEOUT_MS = RAW_UPSERT_BODY_TIMEOUT_MS === "0"
+    ? 0
+    : parseIntegerEnv(RAW_UPSERT_BODY_TIMEOUT_MS, 60000, {
+        min: 1000,
+    });
+const RAW_UPSERT_HTTP_TIMEOUT_MS = process.env.YDB_QDRANT_UPSERT_HTTP_TIMEOUT_MS?.trim();
+export const UPSERT_HTTP_TIMEOUT_MS = RAW_UPSERT_HTTP_TIMEOUT_MS === "0"
+    ? 0
+    : parseIntegerEnv(RAW_UPSERT_HTTP_TIMEOUT_MS, 10000, {
+        min: 1000,
+    });
 export const SEARCH_OPERATION_TIMEOUT_MS = parseIntegerEnv(process.env.YDB_QDRANT_SEARCH_TIMEOUT_MS, 10000, { min: 1000 });
+export const TABLE_SESSION_TIMEOUT_MS = parseIntegerEnv(process.env.YDB_QDRANT_SESSION_TIMEOUT_MS, 30000, { min: 5000, max: 120000 });
 export const LAST_ACCESS_MIN_WRITE_INTERVAL_MS = parseIntegerEnv(process.env.YDB_QDRANT_LAST_ACCESS_MIN_WRITE_INTERVAL_MS, 60000, { min: 1000 });
+export const WORKERS_ENABLED = parseBooleanEnv(process.env.YDB_QDRANT_WORKERS_ENABLED, false);
+const RAW_WORKERS_MIN_THREADS = parseIntegerEnv(process.env.YDB_QDRANT_WORKERS_MIN_THREADS, 0, { min: 0, max: 512 });
+const RAW_WORKERS_MAX_THREADS = parseIntegerEnv(process.env.YDB_QDRANT_WORKERS_MAX_THREADS, 1, { min: 1, max: 512 });
+export const WORKERS_MAX_THREADS = RAW_WORKERS_MAX_THREADS;
+export const WORKERS_MIN_THREADS = RAW_WORKERS_MIN_THREADS > RAW_WORKERS_MAX_THREADS
+    ? RAW_WORKERS_MAX_THREADS
+    : RAW_WORKERS_MIN_THREADS;
+export const WORKERS_IDLE_TIMEOUT_MS = parseIntegerEnv(process.env.YDB_QDRANT_WORKERS_IDLE_TIMEOUT_MS, 10000, { min: 0, max: 600000 });
+export const WORKERS_MAX_QUEUE = parseWorkersMaxQueue(process.env.YDB_QDRANT_WORKERS_MAX_QUEUE);

package/dist/index.d.ts

@@ -1 +1,3 @@
 import "dotenv/config";
+export declare function installStderrAsErrorLogger(): void;
+export declare function __restoreStderrWriteForTests(): void;

package/dist/index.js

@@ -1,15 +1,95 @@
 import "dotenv/config";
 import { buildServer } from "./server.js";
-import { PORT } from "./config/env.js";
+import { PORT, UPSERT_BODY_TIMEOUT_MS, UPSERT_HTTP_TIMEOUT_MS, } from "./config/env.js";
 import { logger } from "./logging/logger.js";
 import { readyOrThrow, isCompilationTimeoutError } from "./ydb/client.js";
-import { ensureMetaTable, ensureGlobalPointsTable } from "./ydb/schema.js";
+import { GLOBAL_POINTS_TABLE, ensureMetaTable, ensureGlobalPointsTable, POINTS_BY_FILE_LOOKUP_TABLE, ensurePointsByFileTable, } from "./ydb/schema.js";
 import { verifyCollectionsQueryCompilationForStartup } from "./repositories/collectionsRepo.js";
+import { scheduleExit } from "./utils/exit.js";
+function describeUnknownError(err) {
+    if (err instanceof Error) {
+        return err.message;
+    }
+    if (typeof err === "string") {
+        return err;
+    }
+    try {
+        return JSON.stringify(err);
+    }
+    catch {
+        return String(err);
+    }
+}
+function isFatalStartupSchemaError(err) {
+    if (!(err instanceof Error)) {
+        return false;
+    }
+    return (err.message.includes(`Global points table ${GLOBAL_POINTS_TABLE}`) ||
+        err.message.includes(`Points-by-file lookup table ${POINTS_BY_FILE_LOOKUP_TABLE}`));
+}
+let originalStderrWrite;
+let stderrAsErrorLoggerInstalled = false;
+function writeToOriginalStderr(chunk, encoding, cb) {
+    if (!originalStderrWrite) {
+        return true;
+    }
+    if (typeof encoding === "function") {
+        return originalStderrWrite(chunk, encoding);
+    }
+    if (encoding !== undefined) {
+        return originalStderrWrite(chunk, encoding, cb);
+    }
+    return originalStderrWrite(chunk);
+}
+export function installStderrAsErrorLogger() {
+    if (stderrAsErrorLoggerInstalled) {
+        return;
+    }
+    // Some infra log collectors treat raw stderr lines as DEBUG by default.
+    // Mirror stderr writes into structured error logs while preserving the
+    // original stream so native/runtime diagnostics still reach stderr.
+    originalStderrWrite = process.stderr.write.bind(process.stderr);
+    process.stderr.write = ((chunk, encoding, cb) => {
+        const enc = typeof encoding === "string" ? encoding : undefined;
+        const text = typeof chunk === "string"
+            ? chunk
+            : Buffer.from(chunk).toString(enc ?? "utf8");
+        for (const line of text.split(/\r?\n/)) {
+            if (line.length > 0) {
+                logger.error({ stream: "stderr" }, line);
+            }
+        }
+        return writeToOriginalStderr(chunk, encoding, cb);
+    });
+    stderrAsErrorLoggerInstalled = true;
+}
+export function __restoreStderrWriteForTests() {
+    if (!stderrAsErrorLoggerInstalled || !originalStderrWrite) {
+        return;
+    }
+    process.stderr.write = originalStderrWrite;
+    stderrAsErrorLoggerInstalled = false;
+}
+installStderrAsErrorLogger();
+process.on("uncaughtException", (err) => {
+    logger.fatal({ err }, "Uncaught exception");
+    scheduleExit(1);
+});
+process.on("unhandledRejection", (reason) => {
+    if (reason instanceof Error) {
+        logger.fatal({ err: reason }, "Unhandled rejection");
+    }
+    else {
+        logger.fatal({ err: new Error(describeUnknownError(reason)), reason }, "Unhandled rejection");
+    }
+    scheduleExit(1);
+});
 async function start() {
     try {
         await readyOrThrow();
         await ensureMetaTable();
         await ensureGlobalPointsTable();
+        await ensurePointsByFileTable();
         await verifyCollectionsQueryCompilationForStartup();
         logger.info("YDB compilation startup probe for qdr__collections completed successfully");
     }
@@ -17,9 +97,19 @@ async function start() {
         if (isCompilationTimeoutError(err)) {
             logger.error({ err }, "Fatal YDB compilation timeout during startup probe; exiting so supervisor can restart the process");
             process.exit(1);
+            return;
+        }
+        if (isFatalStartupSchemaError(err)) {
+            logger.error({ err }, "Fatal YDB schema/startup check failure; exiting until required migrations are applied");
+            process.exit(1);
+            return;
         }
         logger.error({ err }, "YDB not ready; startup continues, requests may fail until configured.");
     }
+    logger.info({
+        upsertBodyTimeoutMs: UPSERT_BODY_TIMEOUT_MS,
+        upsertProcessingTimeoutMs: UPSERT_HTTP_TIMEOUT_MS,
+    }, "Resolved upsert timeout budgets");
     const app = buildServer();
     app.listen(PORT, () => {
         logger.info({ port: PORT }, "ydb-qdrant proxy listening");

package/dist/logging/DeployLogFormatter.d.ts

@@ -0,0 +1,2 @@
+import { Transform } from "node:stream";
+export declare function createDeployLogFormatter(): Transform;