@juspay/neurolink 12.28.1 → 12.28.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -2
- package/README.md +65 -51
- package/dist/adapters/imageFormatSupport.d.ts +115 -5
- package/dist/adapters/imageFormatSupport.js +424 -12
- package/dist/browser/neurolink.min.js +367 -367
- package/dist/types/file.d.ts +39 -0
- package/dist/utils/messageBuilder.js +154 -16
- package/docs-site/static/search-index.json +16 -16
- package/package.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
## [12.28.
|
|
1
|
+
## [12.28.3](https://github.com/juspay/neurolink/compare/v12.28.2...v12.28.3) (2026-09-26)
|
|
2
2
|
|
|
3
3
|
### Bug Fixes
|
|
4
4
|
|
|
5
|
-
- **(
|
|
5
|
+
- **(docs):** stop recording source line numbers in generated API docs ([47c4d1c](https://github.com/juspay/neurolink/commit/47c4d1c7d249231a3e96754cac9d455da4a9e4d4)), closes [#1800](https://github.com/juspay/neurolink/issues/1800) [#1800](https://github.com/juspay/neurolink/issues/1800)
|
|
6
6
|
|
|
7
7
|
## [11.2.3](https://github.com/juspay/neurolink/compare/v11.2.2...v11.2.3) (2026-08-19)
|
|
8
8
|
|
package/README.md
CHANGED
|
@@ -47,7 +47,7 @@ const decision = await pipe.tryDecide({
|
|
|
47
47
|
|
|
48
48
|
**NeuroLink is the pipe layer of an AI nervous system.** Providers — OpenAI, Anthropic, Google, AWS, Azure, Mistral, local runtimes like Ollama, and dozens more — are the neurons: each generates a different kind of intelligence, at a different cost and latency. NeuroLink is the vascular layer that carries that intelligence, as a stream, to the applications — the organs — that consume it, across three inference types: `generate` and `stream` produce text, `decide` produces a calibrated `boolean`/`choice`/`score` judgment instead. A curated model registry (64 models, 132 aliases) backs metadata, routing, and context-window checks out of the box, and hundreds more models are reachable through aggregator providers — 100+ via LiteLLM, 300+ via OpenRouter.
|
|
49
49
|
|
|
50
|
-
Extracted from production systems at Juspay, NeuroLink provides a practical, TypeScript-first way to plug any application into that nervous system. Switch which neuron answers a request with a single parameter change — OpenAI, Anthropic, Google, AWS Bedrock, Azure, a local runtime, or any provider you add. `decide` is the third inference type — a typed, calibrated judgment instead of text — for the model-routing and gating decisions `generate`/`stream` were never meant to make, powered by a purpose-built decision model (TypeSafe Jev) rather than a general-purpose LLM: routing decisions land in ~400ms for about $0.00002, instead of a full generation call.
|
|
50
|
+
Extracted from production systems at Juspay, NeuroLink provides a practical, TypeScript-first way to plug any application into that nervous system. Switch which neuron answers a request with a single parameter change — OpenAI, Anthropic, Google, AWS Bedrock, Azure, a local runtime, or any provider you add. `decide` is the third inference type — a typed, calibrated judgment instead of text — for the model-routing and gating decisions `generate`/`stream` were never meant to make, powered by a purpose-built decision model (TypeSafe Jev, or the open-weights Laya) rather than a general-purpose LLM: with Jev, routing decisions land in ~400ms for about $0.00002, instead of a full generation call.
|
|
51
51
|
|
|
52
52
|
**Why NeuroLink?** Three genuine inference types, not one dressed up three ways — `generate` and `stream` produce text; `decide` produces a calibrated `boolean`/`choice`/`score` judgment, and which types a provider serves is declared per-provider via `inferenceKinds` rather than inferred from behavior. Every neuron plugs into the same pipe, including 3 fully local runtimes (Ollama, LM Studio, llama.cpp) with per-request credential overrides, and MCP support covers all 4 transports (stdio, HTTP, SSE, WebSocket). Every AI-driven optimization the pipe performs — model routing, context compaction, tool selection — fails open: no key configured behaves exactly like NeuroLink without it, and routing uses asymmetric confidence thresholds (upgrade at 0.3, downgrade at 0.6) rather than a single cutoff, because a wrong downgrade costs more than a wrong upgrade. Switch providers with a single parameter change, leverage built-in tools plus any MCP-compliant tool server, deploy with confidence using enterprise features like Redis memory and multi-provider failover, and optimize costs automatically with intelligent routing. Use it via our professional CLI or TypeScript SDK—whichever fits your workflow.
|
|
53
53
|
|
|
@@ -59,37 +59,37 @@ Extracted from production systems at Juspay, NeuroLink provides a practical, Typ
|
|
|
59
59
|
|
|
60
60
|
## What's New
|
|
61
61
|
|
|
62
|
-
| Feature | Version | Description
|
|
63
|
-
| ------------------------------------------------------- | ----------------- |
|
|
64
|
-
| **`decide` Inference Type + TypeSafe Jev**
|
|
65
|
-
| **7 More Catalog Providers** | v12.11.0–v12.16.0 | Baseten, GMI Cloud, Inception Labs, io.net Intelligence, Mancer, Upstage and API Route onboarded as Tier-2 catalog entries — one JSON file each, roster live-verified against the provider's own `/v1/models`.
|
|
66
|
-
| **Claude-on-Vertex Proxy Fallback** | v12.18.0 | The Anthropic proxy pool can fall back to Claude served on Google Vertex, so an agentic turn survives losing its primary backend mid-conversation instead of failing the turn.
|
|
67
|
-
| **Native-Loop V3 Conversation Reclaim** | v12.17.0 | Reclaims V3 conversations without splitting tool-call/tool-result pairs — the pairing a provider rejects the whole request over.
|
|
68
|
-
| **Multi-Modal Embeddings** | v12.15.0 | `embed()` / `embedMany()` accept images alongside text on providers whose embedding models are multi-modal, for cross-modal retrieval in RAG and custom vector search.
|
|
69
|
-
| **Grok Build Auto-Configuration** | v12.14.0 | The proxy configures Grok Build automatically, deriving context windows and backends from the model catalog rather than hardcoded values.
|
|
70
|
-
| **Anthropic Execution-Control Contract** | v12.13.0 | Truthful stream termination plus an opt-in execution-control contract, so a stream that stopped early reports why instead of looking like a clean finish.
|
|
71
|
-
| **Catalog Tool Declarations Honoured at Runtime** | v12.12.0 | A Tier-2 catalog entry declaring `tools: false` (e.g. Mancer) no longer has tools offered to it at runtime — the JSON declaration is enforced, not just documented.
|
|
72
|
-
| **Artifact Stores: Redis, Custom, Range Reads, Search** | v12.10.0 | Artifacts can be backed by Redis or a custom store, read by byte range, and searched — instead of being held only in process memory.
|
|
73
|
-
| **Local CLI Spend Reading** | v12.6.0–v12.9.0 | Reads token usage directly from other coding CLIs' own local stores — Cursor, Grok Build, Hermes Agent and three more — and names them in proxy traffic, so spend is attributed per client.
|
|
74
|
-
| **Native OpenAI Audio Streaming** | v12.7.0 | OpenAI TTS audio streams natively rather than being buffered to completion first.
|
|
75
|
-
| **HITL Pending-Confirmation State** | v12.5.0 | Exposes whether a human-in-the-loop confirmation is still outstanding, so a caller can distinguish 'waiting on a human' from 'finished'.
|
|
76
|
-
| **OpenCode + Gemini CLI Proxy Clients** | v12.4.0 | OpenCode's generated config is actually loadable, and Gemini CLI is onboarded as a proxy client.
|
|
77
|
-
| **SambaNova Provider** | v12.3.0 | RDU-accelerated open-weight flagships: Llama 3.3 70B (default), GPT-OSS 120B, DeepSeek V3.x, MiniMax, Gemma 4 (vision) — OpenAI-compatible Tier 2 catalog entry. Note: new SambaNova accounts require purchased credits.
|
|
78
|
-
| **Cerebras Provider** | v12.1.0 | Wafer-scale inference at ~3000 tok/s: GPT-OSS 120B (default) + Gemma 4 31B, OpenAI-compatible Tier 2 catalog entry, live-verified end to end (generate, stream, tools, structured output).
|
|
79
|
-
| **Avatar / Music Modalities + 12 Providers** | v9.65.0 | New `output: { mode: "avatar" \| "music" }` dispatch with handlers for D-ID, HeyGen, Replicate-MuseTalk (avatar) and Beatoven, ElevenLabs Music, Lyria, Replicate-MusicGen (music). Plus Fish Audio TTS, Kling/Runway/Replicate video, xAI/Groq/Cohere/Together/Fireworks/Perplexity/Cloudflare LLMs, Voyage/Jina embeddings, Stability/Ideogram/Recraft/Replicate image-gen.
|
|
80
|
-
| **Multi-Provider Voice (TTS/STT)** | v9.62.0 | 6 TTS providers (OpenAI TTS, ElevenLabs, Google TTS, Azure TTS, Fish Audio, Cartesia) + 4 STT providers (Whisper, Deepgram, Azure STT, Google STT) + 2 realtime APIs (OpenAI Realtime, Gemini Live).
|
|
81
|
-
| **4 New Providers** | v9.60.0 | DeepSeek (V3/R1), NVIDIA NIM (400+ catalog), LM Studio (local), llama.cpp (GGUF local).
|
|
82
|
-
| **ModelAccessDeniedError** | v9.59.0 | Typed `ModelAccessDeniedError` + `sdk.checkCredentials()` API for proactive credential validation before first call.
|
|
83
|
-
| **Provider Fallback Policy** | v9.58.0 | `providerFallback` callback + `modelChain` config for centralized multi-provider fallback logic.
|
|
84
|
-
| **Per-Request Credentials** | v9.52.0 | Pass credentials per-call or per-instance for all providers. Per-call overrides instance; instance overrides env vars.
|
|
85
|
-
| **AutoResearch** | v9.53.0 | Autonomous AI experiment engine: proposes code changes, runs experiments, evaluates metrics — unattended for hours.
|
|
86
|
-
| **Gemini 3 Multi-turn Tool Fix** | v9.49.0 | Fixed multi-step agentic tool calling on Vertex AI Gemini 3. Correct `thoughtSignature` replay, `stepIndex` grouping, `executionId` session isolation, 5-min timeout.
|
|
87
|
-
| **MCP Enhancements** | v9.16.0 | Tool routing (6 strategies), result caching (LRU/FIFO/LFU), request batching, annotations, elicitation protocol, multi-server management.
|
|
88
|
-
| **Memory** | v9.12.0 | Per-user condensed memory across conversations. LLM-powered condensation with S3, Redis, or SQLite.
|
|
89
|
-
| **Context Window Management** | v9.2.0 | 5-stage compaction pipeline with budget gate at 80% usage, per-provider token estimation.
|
|
90
|
-
| **Tool Execution Control** | v9.3.0 | `prepareStep` and `toolChoice` for per-step tool enforcement in multi-step agentic loops.
|
|
91
|
-
| **File Processor System** | v9.1.0 | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection.
|
|
92
|
-
| **RAG with generate()/stream()** | v9.2.0 | Pass `rag: { files }` for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking, and a choice of 4 vector stores (in-memory, Chroma, PgVector, Pinecone).
|
|
62
|
+
| Feature | Version | Description | Guide |
|
|
63
|
+
| ------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
|
|
64
|
+
| **`decide` Inference Type + TypeSafe Jev + Laya** | next | A third inference type alongside `generate`/`stream`: typed, calibrated judgments (`boolean`, `choice`, `score`) via `neurolink.decide()` / `tryDecide()`, one parallel pass (~400ms and ~$0.00002/decision on Jev). First provider is TypeSafe Jev (`TYPESAFE_API_KEY`, also reachable via the Vercel AI Gateway); [Laya](docs/getting-started/providers/laya.md) (`LAYA_API_KEY` + `LAYA_BASE_URL`), an open-weights model you run yourself, is the second — TypeSafe wins when both are configured. Used internally for model routing, context budgeting, relevance compaction and tool routing — fail-open and a no-op without a key. Per-query RAG planning is opt-in via `RAGPipeline`. | [Decide Guide](docs/features/decide-inference-type.md) |
|
|
65
|
+
| **7 More Catalog Providers** | v12.11.0–v12.16.0 | Baseten, GMI Cloud, Inception Labs, io.net Intelligence, Mancer, Upstage and API Route onboarded as Tier-2 catalog entries — one JSON file each, roster live-verified against the provider's own `/v1/models`. | [Tier 2 Onboarding](docs/provider-integration/tiers/tier-2-catalog-entry.md) |
|
|
66
|
+
| **Claude-on-Vertex Proxy Fallback** | v12.18.0 | The Anthropic proxy pool can fall back to Claude served on Google Vertex, so an agentic turn survives losing its primary backend mid-conversation instead of failing the turn. | [Claude Proxy](docs/features/claude-proxy.md) |
|
|
67
|
+
| **Native-Loop V3 Conversation Reclaim** | v12.17.0 | Reclaims V3 conversations without splitting tool-call/tool-result pairs — the pairing a provider rejects the whole request over. | [Claude Proxy Architecture](docs/features/claude-proxy-architecture.md) |
|
|
68
|
+
| **Multi-Modal Embeddings** | v12.15.0 | `embed()` / `embedMany()` accept images alongside text on providers whose embedding models are multi-modal, for cross-modal retrieval in RAG and custom vector search. | [Embeddings Guide](docs/features/embeddings.md) |
|
|
69
|
+
| **Grok Build Auto-Configuration** | v12.14.0 | The proxy configures Grok Build automatically, deriving context windows and backends from the model catalog rather than hardcoded values. | [Proxy CLI Onboarding](docs/features/proxy-cli-onboarding.md) |
|
|
70
|
+
| **Anthropic Execution-Control Contract** | v12.13.0 | Truthful stream termination plus an opt-in execution-control contract, so a stream that stopped early reports why instead of looking like a clean finish. | [Claude Proxy](docs/features/claude-proxy.md) |
|
|
71
|
+
| **Catalog Tool Declarations Honoured at Runtime** | v12.12.0 | A Tier-2 catalog entry declaring `tools: false` (e.g. Mancer) no longer has tools offered to it at runtime — the JSON declaration is enforced, not just documented. | [Tier 2 Onboarding](docs/provider-integration/tiers/tier-2-catalog-entry.md) |
|
|
72
|
+
| **Artifact Stores: Redis, Custom, Range Reads, Search** | v12.10.0 | Artifacts can be backed by Redis or a custom store, read by byte range, and searched — instead of being held only in process memory. | [Claude Proxy](docs/features/claude-proxy.md) |
|
|
73
|
+
| **Local CLI Spend Reading** | v12.6.0–v12.9.0 | Reads token usage directly from other coding CLIs' own local stores — Cursor, Grok Build, Hermes Agent and three more — and names them in proxy traffic, so spend is attributed per client. | [Proxy CLI Onboarding](docs/features/proxy-cli-onboarding.md) |
|
|
74
|
+
| **Native OpenAI Audio Streaming** | v12.7.0 | OpenAI TTS audio streams natively rather than being buffered to completion first. | [TTS Guide](docs/features/tts.md) |
|
|
75
|
+
| **HITL Pending-Confirmation State** | v12.5.0 | Exposes whether a human-in-the-loop confirmation is still outstanding, so a caller can distinguish 'waiting on a human' from 'finished'. | [Task Manager](docs/features/task-manager.md) |
|
|
76
|
+
| **OpenCode + Gemini CLI Proxy Clients** | v12.4.0 | OpenCode's generated config is actually loadable, and Gemini CLI is onboarded as a proxy client. | [OpenCode Proxy](docs/features/opencode-proxy-support.md) \| [Proxy CLI Onboarding](docs/features/proxy-cli-onboarding.md) |
|
|
77
|
+
| **SambaNova Provider** | v12.3.0 | RDU-accelerated open-weight flagships: Llama 3.3 70B (default), GPT-OSS 120B, DeepSeek V3.x, MiniMax, Gemma 4 (vision) — OpenAI-compatible Tier 2 catalog entry. Note: new SambaNova accounts require purchased credits. | [SambaNova Guide](docs/getting-started/providers/sambanova.md) |
|
|
78
|
+
| **Cerebras Provider** | v12.1.0 | Wafer-scale inference at ~3000 tok/s: GPT-OSS 120B (default) + Gemma 4 31B, OpenAI-compatible Tier 2 catalog entry, live-verified end to end (generate, stream, tools, structured output). | [Cerebras Guide](docs/getting-started/providers/cerebras.md) |
|
|
79
|
+
| **Avatar / Music Modalities + 12 Providers** | v9.65.0 | New `output: { mode: "avatar" \| "music" }` dispatch with handlers for D-ID, HeyGen, Replicate-MuseTalk (avatar) and Beatoven, ElevenLabs Music, Lyria, Replicate-MusicGen (music). Plus Fish Audio TTS, Kling/Runway/Replicate video, xAI/Groq/Cohere/Together/Fireworks/Perplexity/Cloudflare LLMs, Voyage/Jina embeddings, Stability/Ideogram/Recraft/Replicate image-gen. | [Provider Integration](docs/provider-integration/) |
|
|
80
|
+
| **Multi-Provider Voice (TTS/STT)** | v9.62.0 | 6 TTS providers (OpenAI TTS, ElevenLabs, Google TTS, Azure TTS, Fish Audio, Cartesia) + 4 STT providers (Whisper, Deepgram, Azure STT, Google STT) + 2 realtime APIs (OpenAI Realtime, Gemini Live). | [TTS Guide](docs/features/tts.md) \| [STT Guide](docs/features/audio-input.md) \| [Realtime Guide](docs/features/real-time-services.md) |
|
|
81
|
+
| **4 New Providers** | v9.60.0 | DeepSeek (V3/R1), NVIDIA NIM (400+ catalog), LM Studio (local), llama.cpp (GGUF local). | [Provider Setup](docs/getting-started/provider-setup.md) |
|
|
82
|
+
| **ModelAccessDeniedError** | v9.59.0 | Typed `ModelAccessDeniedError` + `sdk.checkCredentials()` API for proactive credential validation before first call. | [Error Reference](docs/reference/troubleshooting.md) |
|
|
83
|
+
| **Provider Fallback Policy** | v9.58.0 | `providerFallback` callback + `modelChain` config for centralized multi-provider fallback logic. | [Advanced Guide](docs/advanced/index.md) |
|
|
84
|
+
| **Per-Request Credentials** | v9.52.0 | Pass credentials per-call or per-instance for all providers. Per-call overrides instance; instance overrides env vars. | [Credentials Guide](docs/features/per-request-credentials.md) |
|
|
85
|
+
| **AutoResearch** | v9.53.0 | Autonomous AI experiment engine: proposes code changes, runs experiments, evaluates metrics — unattended for hours. | [AutoResearch Guide](docs/features/autoresearch.md) |
|
|
86
|
+
| **Gemini 3 Multi-turn Tool Fix** | v9.49.0 | Fixed multi-step agentic tool calling on Vertex AI Gemini 3. Correct `thoughtSignature` replay, `stepIndex` grouping, `executionId` session isolation, 5-min timeout. | [Vertex AI Guide](docs/getting-started/providers/google-vertex.md) |
|
|
87
|
+
| **MCP Enhancements** | v9.16.0 | Tool routing (6 strategies), result caching (LRU/FIFO/LFU), request batching, annotations, elicitation protocol, multi-server management. | [MCP Enhancements Guide](docs/features/mcp-enhancements.md) |
|
|
88
|
+
| **Memory** | v9.12.0 | Per-user condensed memory across conversations. LLM-powered condensation with S3, Redis, or SQLite. | [Memory Guide](docs/features/memory.md) |
|
|
89
|
+
| **Context Window Management** | v9.2.0 | 5-stage compaction pipeline with budget gate at 80% usage, per-provider token estimation. | [Context Compaction Guide](docs/features/context-compaction.md) |
|
|
90
|
+
| **Tool Execution Control** | v9.3.0 | `prepareStep` and `toolChoice` for per-step tool enforcement in multi-step agentic loops. | [API Reference](docs/api/type-aliases/GenerateOptions.md#preparestep) |
|
|
91
|
+
| **File Processor System** | v9.1.0 | 17+ file type processors with ProcessorRegistry, security sanitization, SVG text injection. | [File Processors Guide](docs/features/file-processors.md) |
|
|
92
|
+
| **RAG with generate()/stream()** | v9.2.0 | Pass `rag: { files }` for automatic document chunking, embedding, and AI-powered search. 10 chunking strategies, hybrid search, reranking, and a choice of 4 vector stores (in-memory, Chroma, PgVector, Pinecone). | [RAG Guide](docs/features/rag.md) |
|
|
93
93
|
|
|
94
94
|
```typescript
|
|
95
95
|
// decide() — a third inference type: calibrated judgments, not text (next)
|
|
@@ -221,8 +221,8 @@ const neurolink = new NeuroLink({
|
|
|
221
221
|
|
|
222
222
|
## Decide: Calibrated Judgments, Not Text
|
|
223
223
|
|
|
224
|
-
**NeuroLink now supports decision models — a third inference type,
|
|
225
|
-
|
|
224
|
+
**NeuroLink now supports decision models — a third inference type, with two
|
|
225
|
+
providers today: a hosted one and an open-weights one you can run yourself.**
|
|
226
226
|
|
|
227
227
|
`decide` sits alongside `generate` and `stream`. Instead of tokens, a decision
|
|
228
228
|
model takes one `state` plus a map of named typed questions and returns one
|
|
@@ -270,6 +270,20 @@ through the **Vercel AI Gateway** via `AI_GATEWAY_API_KEY`; force one transport
|
|
|
270
270
|
with `TYPESAFE_TRANSPORT=direct|gateway`. Per-request credentials work as they do
|
|
271
271
|
for every other provider.
|
|
272
272
|
|
|
273
|
+
[**Laya**](docs/getting-started/providers/laya.md) is the second decision
|
|
274
|
+
provider — Convai Innovations' Apache-2.0, open-weights "System One" model, for
|
|
275
|
+
when you'd rather run the decision model on your own infrastructure than call a
|
|
276
|
+
hosted one. There is no built-in endpoint: set `LAYA_API_KEY` and
|
|
277
|
+
`LAYA_BASE_URL` to point at a Laya server you run yourself, or a LiteLLM proxy
|
|
278
|
+
with a pass-through route to one (or pass `credentials.laya` to
|
|
279
|
+
`new NeuroLink({ credentials })`, or per call). Its encoders read a much
|
|
280
|
+
shorter state than TypeSafe's — about 768 tokens on the default
|
|
281
|
+
`typed-decisions` checkpoint (320 on `english`/`auto`), against TypeSafe's
|
|
282
|
+
~33,000 — so it fits short, structured decisions rather than long context.
|
|
283
|
+
Every built-in consumer below asks for the first configured decision provider,
|
|
284
|
+
and TypeSafe is listed first: with both configured, TypeSafe runs; with only
|
|
285
|
+
Laya's key and base URL set, Laya runs.
|
|
286
|
+
|
|
273
287
|
```typescript
|
|
274
288
|
import {
|
|
275
289
|
NeuroLink,
|
|
@@ -350,7 +364,7 @@ Measured against the live API — don't extrapolate past these:
|
|
|
350
364
|
- Two input ceilings: `state` + the longest single question ≈ 33,000 tokens; `state` + all questions ≈ 64,000 tokens.
|
|
351
365
|
- **Accuracy is the trade-off**: ~68% on TypeSafe's own 711-case benchmark vs. ~73% for a frontier model (TypeSafe's published figures, not our measurement). Use it for decisions that are **gated and reversible** — routing, dropping, budgeting — never for a final answer a user will see.
|
|
352
366
|
|
|
353
|
-
**[Decide Guide](docs/features/decide-inference-type.md)** · **[TypeSafe Provider Guide](docs/getting-started/providers/typesafe.md)**
|
|
367
|
+
**[Decide Guide](docs/features/decide-inference-type.md)** · **[TypeSafe Provider Guide](docs/getting-started/providers/typesafe.md)** · **[Laya Provider Guide](docs/getting-started/providers/laya.md)**
|
|
354
368
|
|
|
355
369
|
## Enterprise Security: Human-in-the-Loop (HITL)
|
|
356
370
|
|
|
@@ -629,7 +643,7 @@ NeuroLink is a comprehensive AI development platform. Every feature below is shi
|
|
|
629
643
|
|
|
630
644
|
### 🤖 AI Provider Integration
|
|
631
645
|
|
|
632
|
-
**Every provider neuron behind one API** - Switch providers with a single parameter change. Nearly all serve `generate`/`stream`; TypeSafe Jev
|
|
646
|
+
**Every provider neuron behind one API** - Switch providers with a single parameter change. Nearly all serve `generate`/`stream`; TypeSafe Jev and Laya serve `decide` instead. Tool support: 30 native tool-calling, 3 model-dependent, 9 that serve no tools at all (embedding-, media- and decision-only). 3 are fully local runtimes (Ollama, LM Studio, llama.cpp) and 4 need zero configuration to start (those three plus LiteLLM) — no cloud account, no API key. 9 providers (OpenAI, Google AI Studio, Google Vertex, Amazon Bedrock, Cohere, Ollama, LiteLLM, Voyage, Jina) expose `embed()`/`embedMany()` natively for RAG and custom vector search.
|
|
633
647
|
|
|
634
648
|
| Provider | Models | Free Tier | Tool Support | Status | Documentation |
|
|
635
649
|
| --------------------- | -------------------------------------------------------------------------- | --------------- | ------------ | ------------- | ----------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -663,9 +677,9 @@ NeuroLink is a comprehensive AI development platform. Every feature below is shi
|
|
|
663
677
|
|
|
664
678
|
**Media generation** — [Replicate](docs/getting-started/providers/replicate.md) (`REPLICATE_API_TOKEN`) · [Stability AI](docs/getting-started/providers/stability.md) (`STABILITY_API_KEY`) · [Ideogram](docs/getting-started/providers/ideogram.md) (`IDEOGRAM_API_KEY`) · [Recraft](docs/getting-started/providers/recraft.md) (`RECRAFT_API_KEY`)
|
|
665
679
|
|
|
666
|
-
**Decision** — [TypeSafe Jev](docs/getting-started/providers/typesafe.md) (`TYPESAFE_API_KEY`, or `AI_GATEWAY_API_KEY` via the Vercel AI Gateway) — the
|
|
680
|
+
**Decision** — [TypeSafe Jev](docs/getting-started/providers/typesafe.md) (`TYPESAFE_API_KEY`, or `AI_GATEWAY_API_KEY` via the Vercel AI Gateway) · [Laya](docs/getting-started/providers/laya.md) (`LAYA_API_KEY` + `LAYA_BASE_URL`, open-weights, self-hosted) — the two providers serving `decide` rather than `generate`/`stream`.
|
|
667
681
|
|
|
668
|
-
**Decision-only
|
|
682
|
+
**Decision-only providers:** **TypeSafe Jev** (`TYPESAFE_API_KEY`) and **Laya** (`LAYA_API_KEY` + `LAYA_BASE_URL`) do not appear in the table above because neither serves `generate`/`stream` — they are the two providers for the `decide` inference type, TypeSafe first when both are configured. See [Decide: Calibrated Judgments, Not Text](#decide-calibrated-judgments-not-text).
|
|
669
683
|
|
|
670
684
|
**[📖 Provider Comparison Guide](docs/reference/provider-comparison.md)** - Detailed feature matrix and selection criteria
|
|
671
685
|
**[🔬 Provider Feature Compatibility](docs/reference/provider-feature-compatibility.md)** - Test-based compatibility reference for all 19 features across every provider
|
|
@@ -1187,18 +1201,18 @@ Full command and API breakdown lives in [`docs/cli/commands.md`](docs/cli/comman
|
|
|
1187
1201
|
|
|
1188
1202
|
## Platform Capabilities at a Glance
|
|
1189
1203
|
|
|
1190
|
-
| Capability | Highlights
|
|
1191
|
-
| ------------------------ |
|
|
1192
|
-
| **Provider unification** | Every provider neuron behind one API, with automatic fallback, cost-aware routing, `providerFallback` policy, `modelChain` config.
|
|
1193
|
-
| **Decision inference** | Third inference type (`decide`) alongside generate/stream: calibrated `boolean`/`choice`/`score` judgments via TypeSafe Jev
|
|
1194
|
-
| **Multimodal pipeline** | Stream images + CSV data + PDF documents across providers with local/remote assets. Auto-detection for mixed file types.
|
|
1195
|
-
| **Voice pipeline** | TTS (6 providers: Google, OpenAI, ElevenLabs, Azure, Fish Audio, Cartesia) + STT (4 providers) + realtime voice APIs (OpenAI Realtime, Gemini Live).
|
|
1196
|
-
| **Quality & governance** | Auto-evaluation engine (14 scorers), guardrails middleware, HITL workflows, audit logging.
|
|
1197
|
-
| **Memory & context** | Per-user condensed memory (S3/Redis/SQLite), Redis session export, 5-stage context compaction.
|
|
1198
|
-
| **CLI tooling** | 34 commands: loop sessions, setup wizard, config validation, Redis auto-detect, JSON output, TTS/STT flags.
|
|
1199
|
-
| **Enterprise ops** | Claude proxy, OTLP observability, OpenObserve dashboard, regional routing, credential management.
|
|
1200
|
-
| **Tool ecosystem** | MCP auto discovery, HTTP/stdio/SSE/WebSocket transports, LiteLLM hub access, SageMaker custom deployment, web search.
|
|
1201
|
-
| **Engineering rigor** | 129 end-to-end test suites (every suite drives the public `generate`/`stream`/`decide`/CLI surface, never internals), 13 custom ESLint rules enforcing the architecture (no `interface`, unique type names, barrel-only type imports) — all AST-based, no regex heuristics.
|
|
1204
|
+
| Capability | Highlights |
|
|
1205
|
+
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1206
|
+
| **Provider unification** | Every provider neuron behind one API, with automatic fallback, cost-aware routing, `providerFallback` policy, `modelChain` config. |
|
|
1207
|
+
| **Decision inference** | Third inference type (`decide`) alongside generate/stream: calibrated `boolean`/`choice`/`score` judgments via TypeSafe Jev (~400ms flat, ~$0.00002/decision) or Laya, a self-hosted open-weights alternative. Used internally for model routing, context budgeting, relevance compaction and tool routing; per-query RAG planning is opt-in via `RAGPipeline`. |
|
|
1208
|
+
| **Multimodal pipeline** | Stream images + CSV data + PDF documents across providers with local/remote assets. Auto-detection for mixed file types. |
|
|
1209
|
+
| **Voice pipeline** | TTS (6 providers: Google, OpenAI, ElevenLabs, Azure, Fish Audio, Cartesia) + STT (4 providers) + realtime voice APIs (OpenAI Realtime, Gemini Live). |
|
|
1210
|
+
| **Quality & governance** | Auto-evaluation engine (14 scorers), guardrails middleware, HITL workflows, audit logging. |
|
|
1211
|
+
| **Memory & context** | Per-user condensed memory (S3/Redis/SQLite), Redis session export, 5-stage context compaction. |
|
|
1212
|
+
| **CLI tooling** | 34 commands: loop sessions, setup wizard, config validation, Redis auto-detect, JSON output, TTS/STT flags. |
|
|
1213
|
+
| **Enterprise ops** | Claude proxy, OTLP observability, OpenObserve dashboard, regional routing, credential management. |
|
|
1214
|
+
| **Tool ecosystem** | MCP auto discovery, HTTP/stdio/SSE/WebSocket transports, LiteLLM hub access, SageMaker custom deployment, web search. |
|
|
1215
|
+
| **Engineering rigor** | 129 end-to-end test suites (every suite drives the public `generate`/`stream`/`decide`/CLI surface, never internals), 13 custom ESLint rules enforcing the architecture (no `interface`, unique type names, barrel-only type imports) — all AST-based, no regex heuristics. |
|
|
1202
1216
|
|
|
1203
1217
|
## Documentation Map
|
|
1204
1218
|
|
|
@@ -1226,7 +1240,7 @@ Full command and API breakdown lives in [`docs/cli/commands.md`](docs/cli/comman
|
|
|
1226
1240
|
|
|
1227
1241
|
**Decision Inference:**
|
|
1228
1242
|
|
|
1229
|
-
- [Decide Guide](docs/features/decide-inference-type.md) - The `decide` inference type: boolean/choice/score primitives, TypeSafe Jev setup, measured latency/cost
|
|
1243
|
+
- [Decide Guide](docs/features/decide-inference-type.md) - The `decide` inference type: boolean/choice/score primitives, TypeSafe Jev / Laya setup, measured latency/cost
|
|
1230
1244
|
|
|
1231
1245
|
**Provider Intelligence:**
|
|
1232
1246
|
|
|
@@ -19,7 +19,7 @@
|
|
|
19
19
|
*
|
|
20
20
|
* @module adapters/imageFormatSupport
|
|
21
21
|
*/
|
|
22
|
-
import type { ImageWithAltText, VisionImageConversion,
|
|
22
|
+
import type { ExifOrientationProbe, ImageOrientationNormalization, ImageWithAltText, VisionImageConversion, VisionTranscodeOptions } from "../types/index.js";
|
|
23
23
|
/**
|
|
24
24
|
* MIME types every vision-capable provider accepts as-is.
|
|
25
25
|
*
|
|
@@ -45,6 +45,53 @@ export declare const SUPPORTED_INPUT_IMAGE_MIME_TYPES: ReadonlySet<string>;
|
|
|
45
45
|
* reading a file off disk that would not have been converted anyway.
|
|
46
46
|
*/
|
|
47
47
|
export declare function needsVisionTranscode(mimeType: string): boolean;
|
|
48
|
+
/**
|
|
49
|
+
* True when a MIME type's format can carry an orientation tag at all. Cheap
|
|
50
|
+
* enough to call on every image; callers use it to skip reading a file off
|
|
51
|
+
* disk (or invoking sharp) for a format that structurally cannot need it.
|
|
52
|
+
*/
|
|
53
|
+
export declare function mayCarryExifOrientation(mimeType: string): boolean;
|
|
54
|
+
/**
|
|
55
|
+
* Leading bytes {@link probeExifOrientation} needs to reach a verdict on the
|
|
56
|
+
* containers it parses.
|
|
57
|
+
*
|
|
58
|
+
* A JPEG's EXIF block lives in an APP1 segment whose length field is 16 bits
|
|
59
|
+
* wide, so the segment cannot exceed 64 KiB — and IFD0, which holds the
|
|
60
|
+
* orientation tag, sits at the very front of it. One window this size
|
|
61
|
+
* therefore spans the entire tag block of any file written EXIF-first, which
|
|
62
|
+
* is every file a camera or phone produces. WebP settles in 21 bytes; PNG
|
|
63
|
+
* walks complete chunks through the first IDAT and falls back if this window
|
|
64
|
+
* ends first.
|
|
65
|
+
*
|
|
66
|
+
* Sized deliberately in terms of what the format guarantees rather than what
|
|
67
|
+
* a sample of files happens to need: a window chosen by measurement would
|
|
68
|
+
* turn an unusual-but-legal layout into a silent wrong answer, whereas
|
|
69
|
+
* running out of this one returns `"inconclusive"` and costs only the read it
|
|
70
|
+
* was trying to avoid.
|
|
71
|
+
*/
|
|
72
|
+
export declare const EXIF_PROBE_PREFIX_BYTES: number;
|
|
73
|
+
/**
|
|
74
|
+
* Decide from a bounded header prefix whether an image carries an EXIF
|
|
75
|
+
* orientation worth acting on — without decoding it, and without needing all
|
|
76
|
+
* of its bytes.
|
|
77
|
+
*
|
|
78
|
+
* This exists because the orientation question is asked of every JPEG, PNG
|
|
79
|
+
* and WebP a caller supplies, while the answer is "no" for almost all of them.
|
|
80
|
+
* Answering it with a decoder means reading the whole file into memory and
|
|
81
|
+
* loading sharp to look at a header; answering it here costs
|
|
82
|
+
* {@link EXIF_PROBE_PREFIX_BYTES} at most, and usually far less.
|
|
83
|
+
*
|
|
84
|
+
* The verdict is deliberately three-valued. `"absent"` is a promise — the
|
|
85
|
+
* container was parsed far enough to rule an actionable tag out — and callers
|
|
86
|
+
* may skip the expensive path on it. Everything else, including every
|
|
87
|
+
* container this probe does not parse (HEIC, HEIF, AVIF: ISOBMFF, where
|
|
88
|
+
* orientation is an item property rather than a header field), comes back
|
|
89
|
+
* `"inconclusive"` and must fall through to the decoder.
|
|
90
|
+
*
|
|
91
|
+
* @param prefix - Leading bytes of the image; the whole image is also fine.
|
|
92
|
+
* @param mimeType - Detected MIME type of the image `prefix` came from.
|
|
93
|
+
*/
|
|
94
|
+
export declare function probeExifOrientation(prefix: Buffer, mimeType: string): ExifOrientationProbe;
|
|
48
95
|
/**
|
|
49
96
|
* Return image bytes every vision provider can read, transcoding to PNG when
|
|
50
97
|
* the source format is one no provider accepts.
|
|
@@ -55,13 +102,76 @@ export declare function needsVisionTranscode(mimeType: string): boolean;
|
|
|
55
102
|
* compatibility step becoming a new way for a previously working request to
|
|
56
103
|
* break.
|
|
57
104
|
*
|
|
105
|
+
* `options.autoOrient` folds an EXIF orientation correction into the same
|
|
106
|
+
* decode, for a format that needs both (HEIC, HEIF, TIFF, AVIF are in both
|
|
107
|
+
* sets). The sharp backend honours it explicitly, via `.rotate()`.
|
|
108
|
+
*
|
|
109
|
+
* The ffmpeg backends take no flag for it, and `-autorotate` would only
|
|
110
|
+
* restate a default rather than fix anything — but that is established for
|
|
111
|
+
* one orientation mechanism only, so read the scope before relying on it.
|
|
112
|
+
*
|
|
113
|
+
* **Measured — EXIF-IFD orientation.** A 300x150 JPEG tagged `Orientation=6`
|
|
114
|
+
* came back 150x300 with the tag consumed on system ffmpeg 8.1.1 and on the
|
|
115
|
+
* bundled `ffmpeg-static` 6.0, and stayed 300x150 on both only when
|
|
116
|
+
* `-noautorotate` was forced, which is the control showing the default did
|
|
117
|
+
* the work. TIFF stores orientation the same way.
|
|
118
|
+
*
|
|
119
|
+
* ⚠️ **Not measured — ISOBMFF `irot`/`imir`.** HEIC, HEIF and AVIF carry
|
|
120
|
+
* orientation as item properties, a structurally different mechanism the
|
|
121
|
+
* EXIF result says nothing about. That gap matters more than its size
|
|
122
|
+
* suggests: HEVC-coded HEIC is the main reason this fallback exists at all,
|
|
123
|
+
* since sharp cannot decode it, whereas the JPEG used above never reaches
|
|
124
|
+
* ffmpeg in production — JPEG is universal and is never transcoded, so the
|
|
125
|
+
* fixture demonstrates the binaries' behaviour rather than a path a request
|
|
126
|
+
* takes. Whether an `irot`-bearing HEIC is corrected here is untested in
|
|
127
|
+
* both directions; treat it as unknown rather than working.
|
|
128
|
+
*
|
|
129
|
+
* `autoOrient` is in any case a request rather than a guarantee, exactly like
|
|
130
|
+
* `converted`: a source no backend can decode is returned untouched.
|
|
131
|
+
*
|
|
132
|
+
* ⚠️ Two further things would quietly break the measured case, and neither
|
|
133
|
+
* fails a test, because the fused path reports `converted: true` either way.
|
|
134
|
+
* Passing `-noautorotate`, or an ffmpeg build defaulting it off, drops the
|
|
135
|
+
* rotation. And the sharp-first ordering in {@link transcodeBackends} is what
|
|
136
|
+
* keeps the two backends' orientation behaviour equivalent rather than merely
|
|
137
|
+
* similar — sharp strips the tag while ffmpeg's PNG output carries none to
|
|
138
|
+
* begin with.
|
|
139
|
+
*
|
|
140
|
+
* @param buffer - Raw image bytes.
|
|
141
|
+
* @param mimeType - Detected MIME type of `buffer`.
|
|
142
|
+
* @param options - See {@link VisionTranscodeOptions}. `outputFormat`
|
|
143
|
+
* defaults to `"png"` — see the module header for why that stays the
|
|
144
|
+
* default for every caller who doesn't opt into `"jpeg"` explicitly.
|
|
145
|
+
*/
|
|
146
|
+
export declare function toVisionCompatibleImage(buffer: Buffer, mimeType: string, options?: VisionTranscodeOptions): Promise<VisionImageConversion>;
|
|
147
|
+
/**
|
|
148
|
+
* Auto-orient an image from its EXIF/XMP tag and strip the tag, so a photo
|
|
149
|
+
* that was stored rotated (the common case for phone cameras, since the
|
|
150
|
+
* sensor is read in one fixed orientation and rotation is recorded as a flag
|
|
151
|
+
* rather than baked into the pixels) is sent upright instead of sideways.
|
|
152
|
+
*
|
|
153
|
+
* Three costs are guarded deliberately, matching how `toVisionCompatibleImage`
|
|
154
|
+
* guards its own:
|
|
155
|
+
* - **Cost**: `mayCarryExifOrientation` skips formats that cannot carry the
|
|
156
|
+
* tag; `probeExifOrientation` then settles the formats that can from their
|
|
157
|
+
* own header bytes, so the overwhelmingly common orientation-less JPEG or
|
|
158
|
+
* WebP never loads sharp at all. Only a container the probe cannot rule
|
|
159
|
+
* out reaches a `metadata()` call, and only a tag that is present and not
|
|
160
|
+
* the already-upright value `1` reaches the decode-and-re-encode.
|
|
161
|
+
* - **Availability**: sharp is an optional dependency (the ffmpeg fallback
|
|
162
|
+
* path above exists precisely because it can be absent). Any failure —
|
|
163
|
+
* missing package, malformed pixels, a timeout — is caught here and
|
|
164
|
+
* degrades to returning the original bytes unchanged, never throwing.
|
|
165
|
+
* - **Size**: re-encoding changes byte length. This function only returns the
|
|
166
|
+
* new bytes; it does not itself re-validate size limits. Callers that
|
|
167
|
+
* re-encode must re-run their size guard on the returned buffer.
|
|
168
|
+
*
|
|
169
|
+
* Never throws for image reasons — mirrors `toVisionCompatibleImage`.
|
|
170
|
+
*
|
|
58
171
|
* @param buffer - Raw image bytes.
|
|
59
172
|
* @param mimeType - Detected MIME type of `buffer`.
|
|
60
|
-
* @param outputFormat - Transcode target. Defaults to `"png"` — see the
|
|
61
|
-
* module header for why that stays the default for every caller who
|
|
62
|
-
* doesn't opt into `"jpeg"` explicitly.
|
|
63
173
|
*/
|
|
64
|
-
export declare function
|
|
174
|
+
export declare function normalizeImageOrientation(buffer: Buffer, mimeType: string): Promise<ImageOrientationNormalization>;
|
|
65
175
|
/**
|
|
66
176
|
* Unwrap an `input.images` entry to its payload.
|
|
67
177
|
*
|