agentic-server 0.21.2 → 0.22.0

package/README.md

@@ -9,29 +9,27 @@
     <img height="20" src="https://github.com/constructive-io/constructive/actions/workflows/run-tests.yaml/badge.svg" />
   </a>
    <a href="https://github.com/constructive-io/constructive/blob/main/LICENSE"><img height="20" src="https://img.shields.io/badge/license-MIT-blue.svg"/></a>
-   <a href="https://www.npmjs.com/package/agentic-server"><img height="20" src="https://img.shields.io/github/package-json/v/constructive-io/constructive?filename=packages%2Fagentic-server%2Fpackage.json"/></a>
+   <a href="https://www.npmjs.com/package/agentic-server"><img height="20" src="https://img.shields.io/github/package-json/v/constructive-io/constructive?filename=agentic%2Fagentic-server%2Fpackage.json"/></a>
 </p>
 
-Standalone Express LLM service — agent threads, chat streaming, billing metering, and inference logging via `@constructive-io/express-context`.
+Standalone, OpenAI-compatible LLM gateway — a stateless multi-provider proxy with pluggable inference metering.
 
 ## Overview
 
-`agentic-server` is the Express-only equivalent of what `graphile-llm` does inside PostGraphile. It provides the same capabilities (agent threads, chat, embeddings, billing, inference logging) but as a standalone Express router that uses `@constructive-io/express-context` for tenant-scoped database access.
+`agentic-server` is a small Express app that accepts OpenAI-compatible requests, routes them to a configured LLM provider (OpenAI, Anthropic, or Ollama), normalizes the response back to the OpenAI shape, and records usage through an injected sink. It is **backend-agnostic**: the gateway never imports a concrete telemetry/billing implementation — callers inject an `InferenceSink`, so the same gateway can meter to `compute_log`, a billing service, or nothing at all.
 
 ## Usage
 
 ```typescript
-import express from 'express';
-import { createContextMiddleware } from '@constructive-io/express-context';
-import { createAgenticRouter } from 'agentic-server';
-
-const app = express();
-
-// Tenant context middleware (domain resolution, JWT, pgSettings, withPgClient)
-app.use(createContextMiddleware());
-
-// Mount the agentic router
-app.use(createAgenticRouter());
+import { createAgenticServer } from 'agentic-server';
+
+const app = createAgenticServer({
+  providerType: 'ollama',
+  providerBaseUrl: 'http://localhost:11434',
+  defaultModel: 'llama3',
+  // Optional: record usage. Fire-and-forget; implementations must never throw.
+  inferenceSink: { logInference: (entry) => myBackend.record(entry) },
+});
 
 app.listen(3001, () => {
   console.log('agentic-server running on :3001');
@@ -42,46 +40,41 @@ app.listen(3001, () => {
 
 | Method | Path | Description |
 |--------|------|-------------|
-| POST | `/v1/threads` | Create a new conversation thread |
-| POST | `/v1/threads/:thread_id/messages` | Send messages + get AI response (streaming SSE) |
-| POST | `/v1/orgs/:entity_id/threads` | Create thread (entity-scoped) |
-| POST | `/v1/orgs/:entity_id/threads/:thread_id/messages` | Send message (entity-scoped) |
-| POST | `/v1/embed` | Generate embeddings |
-
-## Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CHAT_PROVIDER` | `ollama` | Chat LLM provider |
-| `CHAT_MODEL` | `llama3` | Default chat model |
-| `CHAT_BASE_URL` | `http://localhost:11434` | Chat provider URL |
-| `EMBEDDER_PROVIDER` | `ollama` | Embedding provider |
-| `EMBEDDER_MODEL` | `nomic-embed-text` | Default embedding model |
-| `EMBEDDER_BASE_URL` | `http://localhost:11434` | Embedding provider URL |
-
-## Features
-
-- **Thread management**: Create and manage conversation threads with system prompts
-- **Streaming chat**: SSE streaming responses (OpenAI-compatible format)
-- **Batch chat**: Non-streaming JSON responses
-- **Embeddings**: Generate vector embeddings via `/v1/embed`
-- **Billing integration**: Automatic quota checks and usage recording (when billing module is provisioned)
-- **Inference logging**: Automatic token usage logging (when inference_log module is provisioned)
-- **RLS enforcement**: All database operations run within tenant-scoped RLS transactions
+| POST | `/v1/chat/completions` | OpenAI-compatible chat completion (multi-provider) |
+| POST | `/v1/embeddings` | OpenAI-compatible embeddings (multi-provider) |
+| POST | `/v1/usage` | Submit an external usage report to the metering sink |
+| GET | `/v1/providers` | List configured providers |
+| GET | `/healthz` | Health check |
+
+## Identity & tenancy
+
+Requests carry tenant identity via headers: `X-Database-Id` (**required** — requests without it are rejected `400`), `X-Entity-Id`, and `X-Actor-Id`. Routing to a specific provider can be forced with `X-LLM-Provider`.
+
+`isPublic` controls trust: when `false` (default) the server sits on a private network and trusts these headers directly; when `true` the identity headers are stripped from incoming requests (external clients cannot set tenant context), so such requests fail loud rather than being mis-attributed.
+
+## Metering
+
+Metering is injected, not built in:
+
+```typescript
+interface InferenceSink {
+  logInference(entry: InferenceEntry): void; // fire-and-forget; must not throw
+}
+```
+
+The gateway calls `logInference` after each chat/embedding (and for `/v1/usage` reports). Consumers own the concrete implementation — e.g. `constructive-db` injects a `compute_log`-backed sink built on its `ModuleLoader`.
 
 ## Architecture
 
 ```
-Cloud Function → POST /v1/threads/:id/messages → agentic-server
-                                                      │
-                                  ┌───────────────────┼───────────────────┐
-                                  │                   │                   │
-                         express-context        OllamaAdapter        Billing
-                        (tenant DB context)    (LLM provider)    (quota + usage)
+Client → POST /v1/chat/completions → agentic-server
+                                          │
+                        ┌─────────────────┼─────────────────┐
+                        │                 │                 │
+                 resolveProvider     LLM provider      InferenceSink
+                 + transforms      (OpenAI/Ollama/…)   (injected metering)
 ```
 
-The cloud function doesn't need to know about databases, billing, or LLM providers. It just POSTs `{ messages, model }` and gets back a streamed response.
-
 ---
 
 ## Education and Tutorials

package/esm/index.js

@@ -1,26 +1,26 @@
 /**
- * agentic-server — Standalone Express LLM service
+ * agentic-server — Standalone OpenAI-compatible LLM gateway
  *
- * Express-only equivalent of graphile-llm: agent threads, chat streaming,
- * billing metering, and inference logging. Uses @constructive-io/express-context
- * for tenant-scoped database access.
- *
- * LLM provider config is resolved per-database via `ctx.useLlm()` (from the
- * llm_module table), falling back to env vars from @constructive-io/llm-env when the module
- * is not provisioned. Discovery and billing are handled by the shared loaders
- * in express-context — no custom caching here.
+ * A stateless multi-provider proxy that:
+ *   1. Accepts OpenAI-compatible requests (`/v1/chat/completions`, `/v1/embeddings`)
+ *   2. Enforces tenant isolation via X-Database-Id / X-Entity-Id / X-Actor-Id headers
+ *   3. Routes to configured LLM provider(s) (OpenAI, Anthropic, Ollama)
+ *   4. Records inference usage through an injected {@link InferenceSink}
+ *      (fire-and-forget) — the gateway is backend-agnostic and never imports a
+ *      concrete metering implementation.
  *
  * @example
  * ```typescript
- * import express from 'express';
- * import { createContextMiddleware } from '@constructive-io/express-context';
- * import { createAgenticRouter } from 'agentic-server';
+ * import { createAgenticServer } from 'agentic-server';
  *
- * const app = express();
- * app.use(createContextMiddleware());
- * app.use(createAgenticRouter());
+ * const app = createAgenticServer({
+ *   providerType: 'ollama',
+ *   providerBaseUrl: 'http://localhost:11434',
+ *   inferenceSink: { logInference: (entry) => myBackend.record(entry) },
+ * });
  * app.listen(3001);
  * ```
  */
-export { createAgenticRouter } from './router';
-export { getEnvVars as getLlmEnvVars, getEnvOptions as getLlmEnvOptions, llmDefaults } from '@constructive-io/llm-env';
+export { buildProviderHeaders, resolveProvider, resolveUpstreamUrl } from './providers';
+export { createRouter } from './router';
+export { createAgenticServer } from './server';

package/esm/providers.js

@@ -0,0 +1,77 @@
+/**
+ * Resolve which provider to use for a given request.
+ * Priority: model-based routing → explicit provider header → default.
+ */
+export function resolveProvider(options, requestModel, requestProvider) {
+    const providers = options.providers || [];
+    if (providers.length > 0) {
+        if (requestProvider) {
+            const match = providers.find((p) => p.type === requestProvider);
+            if (match)
+                return match;
+        }
+        // Route by model prefix: "anthropic/claude-3" → anthropic provider
+        if (requestModel && requestModel.includes('/')) {
+            const prefix = requestModel.split('/')[0];
+            const match = providers.find((p) => p.type === prefix);
+            if (match) {
+                return { ...match, defaultModel: requestModel.split('/').slice(1).join('/') };
+            }
+        }
+        // Route by known model patterns
+        if (requestModel) {
+            if (requestModel.startsWith('claude')) {
+                const match = providers.find((p) => p.type === 'anthropic');
+                if (match)
+                    return match;
+            }
+            if (requestModel.startsWith('gpt') || requestModel.startsWith('o1') || requestModel.startsWith('o3')) {
+                const match = providers.find((p) => p.type === 'openai');
+                if (match)
+                    return match;
+            }
+            if (requestModel.startsWith('llama') ||
+                requestModel.startsWith('mistral') ||
+                requestModel.startsWith('gemma') ||
+                requestModel.startsWith('nomic')) {
+                const match = providers.find((p) => p.type === 'ollama');
+                if (match)
+                    return match;
+            }
+        }
+        return providers[0];
+    }
+    // Legacy single-provider mode (default: ollama for local dev)
+    return {
+        type: options.providerType || 'ollama',
+        baseUrl: options.providerBaseUrl || 'http://localhost:11434',
+        apiKey: options.providerApiKey,
+        defaultModel: options.defaultModel
+    };
+}
+export function resolveUpstreamUrl(provider, path) {
+    if (provider.type === 'ollama') {
+        if (path === '/v1/chat/completions')
+            return `${provider.baseUrl}/api/chat`;
+        if (path === '/v1/embeddings')
+            return `${provider.baseUrl}/api/embed`;
+    }
+    if (provider.type === 'anthropic') {
+        if (path === '/v1/chat/completions')
+            return `${provider.baseUrl}/v1/messages`;
+    }
+    return `${provider.baseUrl}${path}`;
+}
+export function buildProviderHeaders(provider) {
+    const headers = { 'Content-Type': 'application/json' };
+    if (provider.apiKey) {
+        if (provider.type === 'anthropic') {
+            headers['x-api-key'] = provider.apiKey;
+            headers['anthropic-version'] = '2023-06-01';
+        }
+        else {
+            headers['Authorization'] = `Bearer ${provider.apiKey}`;
+        }
+    }
+    return headers;
+}