@hebo-ai/gateway 0.6.2 → 0.8.0-rc0

package/README.md

@@ -13,6 +13,7 @@ Learn more in our blog post: [Yet Another AI Gateway?](https://hebo.ai/blog/2601
 ## 🍌 Features
 
 - 🌐 OpenAI-compatible /chat/completions, /embeddings & /models endpoints.
+- 💬 /conversations endpoint built on top of the Responses API.
 - 🔌 Integrate into your existing Hono, Elysia, Next.js & TanStack apps.
 - 🧩 Provider registry compatible with Vercel AI SDK providers.
 - 🧭 Canonical model IDs and parameter naming across providers.
@@ -32,13 +33,13 @@ bun install @hebo-ai/gateway
 - Quickstart
   - [Setup A Gateway Instance](#setup-a-gateway-instance) | [Mount Route Handlers](#mount-route-handlers) | [Call the Gateway](#call-the-gateway)
 - Configuration Reference
-  - [Providers](#providers) | [Models](#models) | [Hooks](#hooks) | [Logger](#logger-settings) | [Observability](#observability)
+  - [Providers](#providers) | [Models](#models) | [Hooks](#hooks) | [Storage](#storage) | [Logger](#logger-settings) | [Observability](#observability) | [Timeouts](#timeout-settings)
 - Framework Support
   - [ElysiaJS](#elysiajs) | [Hono](#hono) | [Next.js](#nextjs) | [TanStack Start](#tanstack-start)
 - Runtime Support
   - [Vercel Edge](#vercel-edge) | [Cloudflare Workers](#cloudflare-workers) | [Deno Deploy](#deno-deploy) | [AWS Lambda](#aws-lambda)
 - OpenAI Extensions
-  - [Reasoning](#reasoning) | [Prompt Caching](#prompt-caching)
+  - [Reasoning](#reasoning) | [Service Tier](#service-tier) | [Prompt Caching](#prompt-caching)
 - Advanced Usage
   - [Passing Framework State to Hooks](#passing-framework-state-to-hooks) | [Selective Route Mounting](#selective-route-mounting) | [Low-level Schemas & Converters](#low-level-schemas--converters)
 
@@ -342,10 +343,8 @@ const gw = gateway({
      * @returns Modified result, or undefined to keep original.
      */
     after: async (ctx: {
-      result: ChatCompletions | ReadableStream<ChatCompletionsChunk | Error> | Embeddings;
-    }): Promise<
-      ChatCompletions | ReadableStream<ChatCompletionsChunk | Error> | Embeddings | void
-    > => {
+      result: ChatCompletions | ChatCompletionsStream | Embeddings;
+    }): Promise<ChatCompletions | ChatCompletionsStream | Embeddings | void> => {
       // Example Use Cases:
       // - Transform result
       // - Result logging
@@ -371,6 +370,52 @@ The `ctx` object is **readonly for core fields**. Use return values to override
 > [!TIP]
 > To pass data between hooks, use `ctx.state`. It’s a per-request mutable bag in which you can stash things like auth info, routing decisions, timers, or trace IDs and read them later again in any of the other hooks.
 
+### Storage
+
+The `/conversations` endpoint stores conversation history and associated items. By default, the gateway uses an in-memory storage, which is suitable for development but not for production as data is lost when the server restarts.
+
+#### In-Memory Storage
+
+You can configure the size of the in-memory storage (default is 256MB).
+
+```ts
+import { gateway } from "@hebo-ai/gateway";
+import { InMemoryStorage } from "@hebo-ai/gateway/storage/memory";
+
+const gw = gateway({
+  // ...
+  storage: new InMemoryStorage({
+    maxSize: 512 * 1024 * 1024, // 512MB
+  }),
+});
+```
+
+#### SQL Storage
+
+Hebo Gateway provides high-performance SQL adapters for **PostgreSQL**, **SQLite**, **MySQL**, and **GrepTimeDB**. It supports common drivers like `pg`, `postgres.js`, `mysql2`, `better-sqlite3`, `@libsql/client`, and `Bun.SQL`.
+
+```ts
+import { gateway } from "@hebo-ai/gateway";
+import { SqlStorage, PostgresDialect } from "@hebo-ai/gateway/storage/sql";
+import { Pool } from "pg";
+
+// 1. Setup dialect-specific client (e.g. pg, mysql2, sqlite, bun)
+const client = new Pool({ connectionString: process.env.DATABASE_URL });
+
+// 2. Setup storage with matching dialect (PostgresDialect, SqliteDialect, MysqlDialect, ...)
+const storage = new SqlStorage({
+  dialect: new PostgresDialect({ client }),
+});
+
+// 3. Run migrations
+await storage.migrate();
+
+const gw = gateway({ storage });
+```
+
+> [!TIP]
+> The `PostgresDialect` includes optimized `JSONB` storage and high-performance `BRIN` indexing for time-ordered data by default.
+
 ## 🧩 Framework Support
 
 Hebo Gateway exposes **WinterCG-compatible** handlers that integrate with almost any existing framework.
@@ -561,6 +606,57 @@ Advanced models (like Anthropic Claude 3.7 or Gemini 3) surface structured reaso
 
 For **Gemini 3** models, returning the thought signature via `extra_content` is mandatory to resume the chain-of-thought; failing to do so may result in errors or degraded performance.
 
+### Service Tier
+
+The chat completions endpoint accepts a provider-agnostic `service_tier` extension:
+
+- `auto`, `default`, `flex`, `priority`, `scale`
+
+Provider-specific mapping:
+
+- **OpenAI**: forwards as OpenAI `serviceTier` (no middleware remap).
+- **Groq**: maps to Groq `serviceTier` (`default` -> `on_demand`, `scale`/`priority` -> `performance`).
+- **Google Vertex**: maps to request headers via middleware:
+  - `default` -> `x-vertex-ai-llm-request-type: shared`
+  - `flex` -> `x-vertex-ai-llm-request-type: shared` + `x-vertex-ai-llm-shared-request-type: flex`
+  - `priority` -> `x-vertex-ai-llm-request-type: shared` + `x-vertex-ai-llm-shared-request-type: priority`
+  - `scale` -> `x-vertex-ai-llm-request-type: dedicated`
+- **Amazon Bedrock**: maps to Bedrock `serviceTier.type` (`default`, `flex`, `priority`, `reserved`; `scale` -> `reserved`, `auto` -> omitted/default).
+
+When available, the resolved value is echoed back on response as `service_tier`.
+
+### Conversations
+
+Hebo Gateway provides a dedicated `/conversations` endpoint for managing persistent conversation state. It is designed as an extension of the [OpenAI Conversations API](https://developers.openai.com/api/reference/typescript/resources/conversations) and supports standard CRUD operations alongside advanced listing with metadata filtering.
+
+#### List & Filter Conversations
+
+You can list conversations with standard cursor-based pagination and filter by any metadata key using the `metadata.KEY=VALUE` pattern.
+
+```bash
+# List conversations for a specific user
+curl "https://api.gateway.com/conversations?limit=10&metadata.user_id=123"
+```
+
+The response follows the standard OpenAI list object:
+
+```json
+{
+  "object": "list",
+  "data": [
+    {
+      "id": "conv_abc123",
+      "object": "conversation",
+      "created_at": 1678531200,
+      "metadata": { "user_id": "123" }
+    }
+  ],
+  "first_id": "conv_abc123",
+  "last_id": "conv_abc123",
+  "has_more": false
+}
+```
+
 ### Prompt Caching
 
 The chat completions endpoint supports both implicit (provider-managed) and explicit prompt caching across OpenAI-compatible providers.
@@ -737,6 +833,37 @@ const gw = gateway({
 
 Langfuse credentials are read from environment variables by the Langfuse OTel SDK (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_BASE_URL`).
 
+### Timeout Settings
+
+You can configure request timeouts via the `timeouts` field:
+
+```ts
+import { gateway } from "@hebo-ai/gateway";
+
+const gw = gateway({
+  // ...
+  // default timeout is 300_000 (5 minutes).
+  // You can set one timeout for all tiers...
+  timeouts: 60_000,
+  // ...disable timeouts completely:
+  // timeouts: null,
+  // ...or split by service tier:
+  // - normal: all non-flex tiers (set null to disable)
+  // - flex: defaults to 3x normal when omitted (set null to disable)
+  // timeouts: { normal: 30_000, flex: null },
+});
+```
+
+> [!NOTE]
+> **Runtime/engine timeout limits**
+> Runtime-level `fetch()` clients may enforce their own timeouts. Configure those runtime/platform limits in addition to gateway `timeouts`.
+>
+> - Node.js runtimes use Undici: https://github.com/nodejs/undici/issues/1373 (Node.js, Vercel Serverless Functions, AWS Lambda)
+> - Bun context: https://github.com/oven-sh/bun/issues/16682
+>
+> **Provider/service timeout limits**
+> Serverless platforms (e.g. Cloudflare Workers, Vercel Edge/Serverless, AWS Lambda) also enforce platform time limits (roughly ~25-100s on edge paths, ~300s for streaming, and up to ~900s configurable for some).
+
 ### Passing Framework State to Hooks
 
 You can pass per-request info from your framework into the gateway via the second `state` argument on the handler, then read it in hooks through `ctx.state`.
@@ -838,7 +965,7 @@ export async function handler(req: Request): Promise<Response> {
 }
 ```
 
-Non-streaming versions are available via `createChatCompletionsResponse`. Equivalent schemas and helpers are available in the `embeddings` and `models` endpoints.
+Non-streaming versions are available via `toChatCompletionsResponse`. Equivalent schemas and helpers are available in the `conversations`, `embeddings` and `models` endpoints.
 
 > [!TIP]
 > Since Zod v4.3 you can generate a JSON Schema from any zod object by calling `z.toJSONSchema(...)`. This is useful for producing OpenAPI documentation from the same source of truth.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hebo-ai/gateway",
-  "version": "0.6.2",
+  "version": "0.8.0-rc0",
   "description": "AI gateway as a framework. For full control over models, routing & lifecycle. OpenAI-compatible /chat/completions, /embeddings & /models.",
   "keywords": [
     "ai",
@@ -53,6 +53,11 @@
       "types": "./dist/endpoints/chat-completions/index.d.ts",
       "import": "./dist/endpoints/chat-completions/index.js"
     },
+    "./endpoints/conversations": {
+      "types": "./dist/endpoints/conversations/index.d.ts",
+      "import": "./dist/endpoints/conversations/index.js",
+      "dev-source": "./src/endpoints/conversations/index.ts"
+    },
     "./endpoints/embeddings": {
       "types": "./dist/endpoints/embeddings/index.d.ts",
       "import": "./dist/endpoints/embeddings/index.js"
@@ -61,6 +66,16 @@
       "types": "./dist/endpoints/models/index.d.ts",
       "import": "./dist/endpoints/models/index.js"
     },
+    "./storage/memory": {
+      "types": "./dist/endpoints/conversations/storage/memory.d.ts",
+      "import": "./dist/endpoints/conversations/storage/memory.js",
+      "dev-source": "./src/endpoints/conversations/storage/memory.ts"
+    },
+    "./storage/sql": {
+      "types": "./dist/endpoints/conversations/storage/sql.d.ts",
+      "import": "./dist/endpoints/conversations/storage/sql.js",
+      "dev-source": "./src/endpoints/conversations/storage/sql.ts"
+    },
     "./errors/openai": {
       "types": "./dist/errors/openai.d.ts",
       "import": "./dist/errors/openai.js"
@@ -142,6 +157,8 @@
   "dependencies": {
     "@ai-sdk/provider": "^3.0.8",
     "ai": "^6.0.116",
+    "lru-cache": "^11.2.6",
+    "uuid": "^13.0.0",
     "zod": "^4.3.6"
   },
   "devDependencies": {
@@ -153,23 +170,31 @@
     "@ai-sdk/openai": "^3.0.41",
     "@aws-sdk/credential-providers": "^3.1004.0",
     "@langfuse/otel": "^4.6.1",
+    "@libsql/client": "^0.17.0",
     "@mjackson/node-fetch-server": "^0.7.0",
     "@opentelemetry/api": "^1.9.0",
     "@opentelemetry/context-async-hooks": "^2.6.0",
     "@opentelemetry/sdk-trace-base": "^2.6.0",
     "@tanstack/react-router": "^1.166.3",
     "@tanstack/react-start": "^1.166.3",
+    "@types/better-sqlite3": "^7.6.13",
     "@types/bun": "latest",
+    "@types/pg": "^8.18.0",
     "@types/react": "^19.2.14",
     "@types/react-dom": "^19.2.3",
+    "@types/uuid": "^11.0.0",
+    "better-sqlite3": "^12.6.2",
     "elysia": "^1.4.27",
     "hono": "^4.12.5",
     "lefthook": "^2.1.3",
+    "mysql2": "^3.18.2",
     "next": "^16.1.6",
     "oxfmt": "^0.24.0",
     "oxlint": "^1.51.0",
     "oxlint-tsgolint": "^0.16.0",
+    "pg": "^8.19.0",
     "pino": "^10.3.1",
+    "postgres": "^3.4.8",
     "typescript": "^5.9.3",
     "vite": "^7.3.1",
     "vite-tsconfig-paths": "^6.1.1",
@@ -183,7 +208,12 @@
     "@ai-sdk/google-vertex": "^4.0.80",
     "@ai-sdk/groq": "^3.0.29",
     "@ai-sdk/openai": "^3.0.41",
+    "@libsql/client": "^0.14.0",
     "@opentelemetry/api": "^1.9.0",
+    "better-sqlite3": "^11.0.0",
+    "mysql2": "^3.11.0",
+    "pg": "^8.13.0",
+    "postgres": "^3.4.0",
     "typescript": "^5.9.3",
     "voyage-ai-provider": "^3.0.0"
   },
@@ -211,6 +241,21 @@
     },
     "voyage-ai-provider": {
       "optional": true
+    },
+    "pg": {
+      "optional": true
+    },
+    "postgres": {
+      "optional": true
+    },
+    "mysql2": {
+      "optional": true
+    },
+    "better-sqlite3": {
+      "optional": true
+    },
+    "@libsql/client": {
+      "optional": true
     }
   }
 }

package/dist/config.d.ts

@@ -1,2 +0,0 @@
-import { type GatewayConfig, type GatewayConfigParsed } from "./types";
-export declare const parseConfig: (config: GatewayConfig) => GatewayConfigParsed;

package/dist/config.js

@@ -1,81 +0,0 @@
-import { isLogger, logger, setLoggerInstance } from "./logger";
-import { createDefaultLogger } from "./logger/default";
-import { installAiSdkWarningLogger } from "./telemetry/ai-sdk";
-import { kParsed, } from "./types";
-export const parseConfig = (config) => {
-    // If it has been parsed before, just return.
-    if (kParsed in config)
-        return config;
-    const providers = config.providers ?? {};
-    const parsedProviders = {};
-    const models = config.models ?? {};
-    // Set the global logger instance.
-    if (config.logger === undefined) {
-        setLoggerInstance(createDefaultLogger({}));
-    }
-    else if (config.logger !== null) {
-        setLoggerInstance(isLogger(config.logger) ? config.logger : createDefaultLogger(config.logger));
-        logger.info(isLogger(config.logger)
-            ? `[logger] custom logger configured`
-            : `[logger] logger configured: level=${config.logger.level}`);
-    }
-    // Strip providers that are not configured.
-    for (const id in providers) {
-        const provider = providers[id];
-        if (provider === undefined) {
-            logger.warn(`[config] ${id} provider removed (undefined)`);
-            continue;
-        }
-        parsedProviders[id] = provider;
-    }
-    if (Object.keys(parsedProviders).length === 0) {
-        throw new Error("No providers configured (config.providers is empty)");
-    }
-    // Strip providers that are not configured from models.
-    const parsedModels = {};
-    const warnings = new Set();
-    for (const id in models) {
-        const model = models[id];
-        const kept = [];
-        for (const p of model.providers) {
-            if (p in parsedProviders)
-                kept.push(p);
-            else
-                warnings.add(p);
-        }
-        if (kept.length > 0)
-            parsedModels[id] = { ...model, providers: kept };
-    }
-    for (const warning of warnings) {
-        logger.warn(`[config] ${warning} provider removed (not configured)`);
-    }
-    if (Object.keys(parsedModels).length === 0) {
-        throw new Error("No models configured (config.models is empty)");
-    }
-    // Default for the telemetry settings.
-    const telemetryEnabled = config.telemetry?.enabled ?? false;
-    const telemetrySignals = telemetryEnabled
-        ? {
-            http: config.telemetry?.signals?.http ?? "recommended",
-            gen_ai: config.telemetry?.signals?.gen_ai ?? "full",
-            hebo: config.telemetry?.signals?.hebo ?? "off",
-        }
-        : {
-            http: "off",
-            gen_ai: "off",
-            hebo: "off",
-        };
-    installAiSdkWarningLogger(telemetrySignals.gen_ai);
-    // Return parsed config.
-    return {
-        ...config,
-        telemetry: {
-            ...config.telemetry,
-            enabled: telemetryEnabled,
-            signals: telemetrySignals,
-        },
-        providers: parsedProviders,
-        models: parsedModels,
-        [kParsed]: true,
-    };
-};

package/dist/endpoints/chat-completions/converters.d.ts

@@ -1,43 +0,0 @@
-import type { SharedV3ProviderOptions, SharedV3ProviderMetadata } from "@ai-sdk/provider";
-import type { GenerateTextResult, StreamTextResult, FinishReason, ToolChoice, ToolSet, ModelMessage, UserContent, LanguageModelUsage, TextStreamPart, ReasoningOutput, AssistantModelMessage, ToolModelMessage, UserModelMessage } from "ai";
-import { Output } from "ai";
-import type { ChatCompletionsToolCall, ChatCompletionsTool, ChatCompletionsToolChoice, ChatCompletionsContentPart, ChatCompletionsMessage, ChatCompletionsUserMessage, ChatCompletionsAssistantMessage, ChatCompletionsToolMessage, ChatCompletionsFinishReason, ChatCompletionsUsage, ChatCompletionsInputs, ChatCompletions, ChatCompletionsChunk, ChatCompletionsReasoningDetail } from "./schema";
-import { OpenAIError } from "../../errors/openai";
-export type TextCallOptions = {
-    messages: ModelMessage[];
-    tools?: ToolSet;
-    toolChoice?: ToolChoice<ToolSet>;
-    activeTools?: Array<keyof ToolSet>;
-    output?: Output.Output;
-    temperature?: number;
-    maxOutputTokens?: number;
-    frequencyPenalty?: number;
-    presencePenalty?: number;
-    seed?: number;
-    stopSequences?: string[];
-    topP?: number;
-    providerOptions: SharedV3ProviderOptions;
-};
-export declare function convertToTextCallOptions(params: ChatCompletionsInputs): TextCallOptions;
-export declare function convertToModelMessages(messages: ChatCompletionsMessage[]): ModelMessage[];
-export declare function fromChatCompletionsUserMessage(message: ChatCompletionsUserMessage): UserModelMessage;
-export declare function fromChatCompletionsAssistantMessage(message: ChatCompletionsAssistantMessage): AssistantModelMessage;
-export declare function fromChatCompletionsToolResultMessage(message: ChatCompletionsAssistantMessage, toolById: Map<string, ChatCompletionsToolMessage>): ToolModelMessage | undefined;
-export declare function fromChatCompletionsContent(content: ChatCompletionsContentPart[]): UserContent;
-export declare const convertToToolSet: (tools: ChatCompletionsTool[] | undefined) => ToolSet | undefined;
-export declare const convertToToolChoiceOptions: (toolChoice: ChatCompletionsToolChoice | undefined) => {
-    toolChoice?: ToolChoice<ToolSet>;
-    activeTools?: Array<keyof ToolSet>;
-};
-export declare function toChatCompletions(result: GenerateTextResult<ToolSet, Output.Output>, model: string): ChatCompletions;
-export declare function toChatCompletionsResponse(result: GenerateTextResult<ToolSet, Output.Output>, model: string, responseInit?: ResponseInit): Response;
-export declare function toChatCompletionsStream<E extends boolean = false>(result: StreamTextResult<ToolSet, Output.Output>, model: string, wrapErrors?: E): ReadableStream<ChatCompletionsChunk | (E extends true ? OpenAIError : Error)>;
-export declare function toChatCompletionsStreamResponse(result: StreamTextResult<ToolSet, Output.Output>, model: string, responseInit?: ResponseInit): Response;
-export declare class ChatCompletionsStream<E extends boolean = false> extends TransformStream<TextStreamPart<ToolSet>, ChatCompletionsChunk | (E extends true ? OpenAIError : Error)> {
-    constructor(model: string, wrapErrors?: E);
-}
-export declare const toChatCompletionsAssistantMessage: (result: GenerateTextResult<ToolSet, Output.Output>) => ChatCompletionsAssistantMessage;
-export declare function toReasoningDetail(reasoning: ReasoningOutput, id: string, index: number): ChatCompletionsReasoningDetail;
-export declare function toChatCompletionsUsage(usage: LanguageModelUsage): ChatCompletionsUsage;
-export declare function toChatCompletionsToolCall(id: string, name: string, args: unknown, providerMetadata?: SharedV3ProviderMetadata): ChatCompletionsToolCall;
-export declare const toChatCompletionsFinishReason: (finishReason: FinishReason) => ChatCompletionsFinishReason;