@retrivora-ai/rag-engine 2.5.2 → 2.5.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,232 +1,232 @@
1
- # Retrivora Free Tier (MVP) Architecture & Implementation Guide
2
-
3
- This document outlines the architecture, constraints, and SDK integration patterns for the **Retrivora Free Tier (MVP)**. The objective of this design is to maximize functionality for multi-tenant users while maintaining near-zero fixed infrastructure costs per tenant.
4
-
5
- ---
6
-
7
- ## 1. Overview & Core Philosophy
8
-
9
- The Retrivora Free Tier is built around an intentionally small, highly cost-efficient scope:
10
-
11
- - **Storage Isolation**: Single shared vector database index with multi-tenancy via **Pinecone Namespaces**.
12
- - **Fixed Embedding Footprint**: Standardized on **768-dimension embeddings** (e.g. `nomic-embed-text` or `bge-base-en-v1.5`).
13
- - **Vendor-Agnostic LLM Routing**: Unified LLM proxy layer via **LiteLLM** (exposing an OpenAI-compatible REST interface).
14
- - **Simple Retrieval**: Standardized semantic RAG flow ($ \text{Search} \rightarrow \text{Top-K} \rightarrow \text{Rerank (optional)} \rightarrow \text{Prompt} \rightarrow \text{LLM} $).
15
- - **Document Support**: Core business formats only (`PDF`, `DOCX`, `TXT`, `Markdown`).
16
- - **Deterministic Limits**: Strict operational guardrails to prevent infrastructure cost overruns.
17
-
18
- ---
19
-
20
- ## 2. Component Specifications
21
-
22
- ### 2.1 Storage & Multi-Tenancy (Pinecone)
23
-
24
- Instead of instantiating dedicated Pinecone accounts or indexes per user, all free-tier tenants share **one Retrivora Pinecone index** (`retrivora-free`). Complete tenant data isolation is guaranteed using **Pinecone Namespaces**.
25
-
26
- ```
27
- Pinecone Index: retrivora-free (768 dimensions, Cosine metric)
28
- ├── Namespace: john@example.com
29
- ├── Namespace: abc-company
30
- └── Namespace: workspace-123
31
- ```
32
-
33
- - **Index Name**: `retrivora-free`
34
- - **Vector Dimension**: `768`
35
- - **Isolation Scope**: One namespace per project / user / workspace (`projectId`).
36
- - **Metadata Filtering**: Fully supported within each isolated namespace.
37
-
38
- ---
39
-
40
- ### 2.2 Embedding Model Strategy
41
-
42
- To standardize vector dimension sizes across the shared index:
43
-
44
- - **Dimensions**: `768`
45
- - **Recommended Models**:
46
- - `nomic-embed-text` (Ollama or REST, using task prefixes `search_query: ` and `search_document: `).
47
- - `BAAI/bge-base-en-v1.5` (768-dimension sentence transformer).
48
- - `BAAI/bge-small-en-v1.5` (384-dimension fallback if required).
49
-
50
- ---
51
-
52
- ### 2.3 LLM Routing Layer (LiteLLM Abstraction)
53
-
54
- Retrivora integrates with **LiteLLM** as a centralized model gateway. Host applications configure a single LiteLLM endpoint, giving end users access to OpenAI, Gemini, Claude, Groq, Ollama, Azure OpenAI, and OpenRouter without vendor lock-in.
55
-
56
- ```
57
- Retrivora SDK ──► LiteLLM Gateway (/v1) ──► OpenAI / Gemini / Claude / Groq / Ollama / OpenRouter
58
- ```
59
-
60
- ---
61
-
62
- ### 2.4 Retrieval Pipeline
63
-
64
- The Free Tier enforces a simple, high-speed semantic search pipeline:
65
-
66
- ```
67
- User Query ──► Embedding ──► Pinecone Top-K Search ──► (Optional Rerank) ──► Prompt Assembly ──► LLM Generation
68
- ```
69
-
70
- * **Disabled Features on Free Tier**:
71
- - Graph retrieval (`useGraphRetrieval: false`)
72
- - Agentic routing & workflows
73
- - Hybrid BM25 / dense search
74
-
75
- ---
76
-
77
- ### 2.5 Document Ingestion & Parsing & Global Metadata Schema
78
-
79
- Supported formats handled by `DocumentParser` & `FreeTierIngestor`:
80
- - `.txt` (Text files)
81
- - `.json` (Structured JSON documents)
82
- - `.pdf` (PDF documents via `pdf-parse`)
83
- - `.xlsx` / `.xls` (Excel spreadsheets via `xlsx` sheet-to-csv)
84
- - `.docx` / `.doc` (Word documents via `mammoth`)
85
- - `.md` (Markdown text)
86
- - `.csv` (Comma-separated values)
87
-
88
- #### Global Chunk Metadata Schema (`RetrivoraChunkMetadata`):
89
- All vectors upserted into Pinecone namespaces conform to a standardized schema:
90
- ```typescript
91
- interface RetrivoraChunkMetadata {
92
- docId: string;
93
- fileName: string;
94
- fileType: 'txt' | 'json' | 'pdf' | 'excel' | 'word' | 'md' | 'csv' | string;
95
- mimeType: string;
96
- chunkIndex: number;
97
- totalChunks: number;
98
- content: string;
99
- characterCount: number;
100
- workspaceId: string;
101
- pineconeNamespace: string;
102
- tier: 'free' | 'enterprise';
103
- ingestedAt: string;
104
- customMetadata?: Record<string, unknown>;
105
- }
106
- ```
107
-
108
- ---
109
-
110
- ### 2.6 Chunking Configuration
111
-
112
- - **Strategy**: `Recursive Character Splitter`
113
- - **Target Chunk Size**: `500–800 tokens` ($\approx 2,000–3,200$ characters)
114
- - **Chunk Overlap**: `100 tokens` ($\approx 400$ characters)
115
- - **Separators**: `['\n# ', '\n## ', '\n### ', '\n\n', '\n', ' ', '']`
116
-
117
- ---
118
-
119
- ### 2.7 Operational Free Tier Limits
120
-
121
- | Feature / Resource | Free Tier Limit |
122
- | :--- | :--- |
123
- | **Documents** | `20` max per workspace |
124
- | **Storage Size** | `50 MB` total file limit |
125
- | **Total Embeddings** | `10,000` vectors |
126
- | **Queries / Day** | `100` queries |
127
- | **Users / Workspaces** | `1` user, `1` namespace, `1` project |
128
-
129
- ---
130
-
131
- ## 3. Retrivora SDK Integration
132
-
133
- ### 3.1 Free Tier Configuration Code Example
134
-
135
- Host applications can configure Retrivora using `ConfigBuilder`:
136
-
137
- ```typescript
138
- import { ConfigBuilder, Retrivora } from '@retrivora-ai/rag-engine/server';
139
-
140
- // 1. Construct Free Tier Configuration
141
- const config = new ConfigBuilder()
142
- .projectId("workspace-123") // Mapped directly to Pinecone Namespace
143
- .vectorDb("pinecone", {
144
- indexName: "retrivora-free",
145
- apiKey: process.env.PINECONE_API_KEY,
146
- })
147
- .llm("universal_rest", "groq/llama-3.3-70b-versatile", process.env.LITELLM_API_KEY, {
148
- baseUrl: process.env.LITELLM_BASE_URL || "http://localhost:4000/v1",
149
- profile: "litellm",
150
- temperature: 0.7,
151
- maxTokens: 1024,
152
- })
153
- .embedding("ollama", "nomic-embed-text", undefined, {
154
- baseUrl: process.env.OLLAMA_BASE_URL || "http://localhost:11434",
155
- dimensions: 768,
156
- queryPrefix: "search_query: ",
157
- documentPrefix: "search_document: ",
158
- })
159
- .rag({
160
- architecture: "simple",
161
- topK: 5,
162
- chunkSize: 2500,
163
- chunkOverlap: 400,
164
- chunkingStrategy: "recursive",
165
- useGraphRetrieval: false,
166
- useQueryTransformation: false,
167
- })
168
- .build();
169
-
170
- // 2. Initialize Engine Instance
171
- const retrivora = new Retrivora(config);
172
- await retrivora.initialize();
173
- ```
174
-
175
- ---
176
-
177
- ### 3.2 Free Tier Preset Helper Function
178
-
179
- ```typescript
180
- export const FREE_TIER_LIMITS = {
181
- maxDocuments: 20,
182
- maxStorageBytes: 50 * 1024 * 1024, // 50 MB
183
- maxEmbeddings: 10000,
184
- maxQueriesPerDay: 100,
185
- allowedFormats: ['.pdf', '.docx', '.txt', '.md'],
186
- };
187
-
188
- export function createFreeTierConfig(params: {
189
- workspaceId: string;
190
- pineconeApiKey: string;
191
- liteLlmApiKey?: string;
192
- liteLlmBaseUrl?: string;
193
- }) {
194
- return new ConfigBuilder()
195
- .projectId(params.workspaceId)
196
- .vectorDb("pinecone", {
197
- indexName: "retrivora-free",
198
- apiKey: params.pineconeApiKey,
199
- })
200
- .llm("universal_rest", "gpt-4o-mini", params.liteLlmApiKey, {
201
- baseUrl: params.liteLlmBaseUrl || "http://localhost:4000/v1",
202
- profile: "litellm",
203
- })
204
- .embedding("ollama", "nomic-embed-text", undefined, {
205
- dimensions: 768,
206
- })
207
- .rag({
208
- architecture: "simple",
209
- topK: 5,
210
- chunkSize: 2500,
211
- chunkOverlap: 400,
212
- chunkingStrategy: "recursive",
213
- useGraphRetrieval: false,
214
- })
215
- .build();
216
- }
217
- ```
218
-
219
- ---
220
-
221
- ## 4. Feature Comparison Matrix
222
-
223
- | Feature | Free Tier (MVP) | Enterprise Tier |
224
- | :--- | :--- | :--- |
225
- | **Vector DB Index** | Shared (`retrivora-free`) | Dedicated index / cluster |
226
- | **Tenancy Isolation** | Namespace | Index / Database / Cloud Project |
227
- | **Embedding Model** | Fixed 768-dim (`nomic-embed-text`, `bge-base`) | Custom / Multimodal / Any dim |
228
- | **LLM Gateway** | LiteLLM Proxy | Direct API / LiteLLM / Enterprise VPC |
229
- | **Retrieval Mode** | Semantic Search (Top-K) | Hybrid + Graph RAG + Agentic Routing |
230
- | **Document Formats** | PDF, DOCX, TXT, MD | All formats + Web scraping + Audio |
231
- | **Document Limit** | 20 documents | Unlimited |
232
- | **Storage Limit** | 50 MB | Custom / Unlimited |
1
+ # Retrivora Free Tier (MVP) Architecture & Implementation Guide
2
+
3
+ This document outlines the architecture, constraints, and SDK integration patterns for the **Retrivora Free Tier (MVP)**. The objective of this design is to maximize functionality for multi-tenant users while maintaining near-zero fixed infrastructure costs per tenant.
4
+
5
+ ---
6
+
7
+ ## 1. Overview & Core Philosophy
8
+
9
+ The Retrivora Free Tier is built around an intentionally small, highly cost-efficient scope:
10
+
11
+ - **Storage Isolation**: Single shared vector database index with multi-tenancy via **Pinecone Namespaces**.
12
+ - **Fixed Embedding Footprint**: Standardized on **768-dimension embeddings** (e.g. `nomic-embed-text` or `bge-base-en-v1.5`).
13
+ - **Vendor-Agnostic LLM Routing**: Unified LLM proxy layer via **LiteLLM** (exposing an OpenAI-compatible REST interface).
14
+ - **Simple Retrieval**: Standardized semantic RAG flow ($ \text{Search} \rightarrow \text{Top-K} \rightarrow \text{Rerank (optional)} \rightarrow \text{Prompt} \rightarrow \text{LLM} $).
15
+ - **Document Support**: Core business formats only (`PDF`, `DOCX`, `TXT`, `Markdown`).
16
+ - **Deterministic Limits**: Strict operational guardrails to prevent infrastructure cost overruns.
17
+
18
+ ---
19
+
20
+ ## 2. Component Specifications
21
+
22
+ ### 2.1 Storage & Multi-Tenancy (Pinecone)
23
+
24
+ Instead of instantiating dedicated Pinecone accounts or indexes per user, all free-tier tenants share **one Retrivora Pinecone index** (`retrivora-free`). Complete tenant data isolation is guaranteed using **Pinecone Namespaces**.
25
+
26
+ ```
27
+ Pinecone Index: retrivora-free (768 dimensions, Cosine metric)
28
+ ├── Namespace: john@example.com
29
+ ├── Namespace: abc-company
30
+ └── Namespace: workspace-123
31
+ ```
32
+
33
+ - **Index Name**: `retrivora-free`
34
+ - **Vector Dimension**: `768`
35
+ - **Isolation Scope**: One namespace per project / user / workspace (`projectId`).
36
+ - **Metadata Filtering**: Fully supported within each isolated namespace.
37
+
38
+ ---
39
+
40
+ ### 2.2 Embedding Model Strategy
41
+
42
+ To standardize vector dimension sizes across the shared index:
43
+
44
+ - **Dimensions**: `768`
45
+ - **Recommended Models**:
46
+ - `nomic-embed-text` (Ollama or REST, using task prefixes `search_query: ` and `search_document: `).
47
+ - `BAAI/bge-base-en-v1.5` (768-dimension sentence transformer).
48
+ - `BAAI/bge-small-en-v1.5` (384-dimension fallback if required).
49
+
50
+ ---
51
+
52
+ ### 2.3 LLM Routing Layer (LiteLLM Abstraction)
53
+
54
+ Retrivora integrates with **LiteLLM** as a centralized model gateway. Host applications configure a single LiteLLM endpoint, giving end users access to OpenAI, Gemini, Claude, Groq, Ollama, Azure OpenAI, and OpenRouter without vendor lock-in.
55
+
56
+ ```
57
+ Retrivora SDK ──► LiteLLM Gateway (/v1) ──► OpenAI / Gemini / Claude / Groq / Ollama / OpenRouter
58
+ ```
59
+
60
+ ---
61
+
62
+ ### 2.4 Retrieval Pipeline
63
+
64
+ The Free Tier enforces a simple, high-speed semantic search pipeline:
65
+
66
+ ```
67
+ User Query ──► Embedding ──► Pinecone Top-K Search ──► (Optional Rerank) ──► Prompt Assembly ──► LLM Generation
68
+ ```
69
+
70
+ * **Disabled Features on Free Tier**:
71
+ - Graph retrieval (`useGraphRetrieval: false`)
72
+ - Agentic routing & workflows
73
+ - Hybrid BM25 / dense search
74
+
75
+ ---
76
+
77
+ ### 2.5 Document Ingestion & Parsing & Global Metadata Schema
78
+
79
+ Supported formats handled by `DocumentParser` & `FreeTierIngestor`:
80
+ - `.txt` (Text files)
81
+ - `.json` (Structured JSON documents)
82
+ - `.pdf` (PDF documents via `pdf-parse`)
83
+ - `.xlsx` / `.xls` (Excel spreadsheets via `xlsx` sheet-to-csv)
84
+ - `.docx` / `.doc` (Word documents via `mammoth`)
85
+ - `.md` (Markdown text)
86
+ - `.csv` (Comma-separated values)
87
+
88
+ #### Global Chunk Metadata Schema (`RetrivoraChunkMetadata`):
89
+ All vectors upserted into Pinecone namespaces conform to a standardized schema:
90
+ ```typescript
91
+ interface RetrivoraChunkMetadata {
92
+ docId: string;
93
+ fileName: string;
94
+ fileType: 'txt' | 'json' | 'pdf' | 'excel' | 'word' | 'md' | 'csv' | string;
95
+ mimeType: string;
96
+ chunkIndex: number;
97
+ totalChunks: number;
98
+ content: string;
99
+ characterCount: number;
100
+ workspaceId: string;
101
+ pineconeNamespace: string;
102
+ tier: 'free' | 'enterprise';
103
+ ingestedAt: string;
104
+ customMetadata?: Record<string, unknown>;
105
+ }
106
+ ```
107
+
108
+ ---
109
+
110
+ ### 2.6 Chunking Configuration
111
+
112
+ - **Strategy**: `Recursive Character Splitter`
113
+ - **Target Chunk Size**: `500–800 tokens` ($\approx 2,000–3,200$ characters)
114
+ - **Chunk Overlap**: `100 tokens` ($\approx 400$ characters)
115
+ - **Separators**: `['\n# ', '\n## ', '\n### ', '\n\n', '\n', ' ', '']`
116
+
117
+ ---
118
+
119
+ ### 2.7 Operational Free Tier Limits
120
+
121
+ | Feature / Resource | Free Tier Limit |
122
+ | :--- | :--- |
123
+ | **Documents** | `20` max per workspace |
124
+ | **Storage Size** | `50 MB` total file limit |
125
+ | **Total Embeddings** | `10,000` vectors |
126
+ | **Queries / Day** | `100` queries |
127
+ | **Users / Workspaces** | `1` user, `1` namespace, `1` project |
128
+
129
+ ---
130
+
131
+ ## 3. Retrivora SDK Integration
132
+
133
+ ### 3.1 Free Tier Configuration Code Example
134
+
135
+ Host applications can configure Retrivora using `ConfigBuilder`:
136
+
137
+ ```typescript
138
+ import { ConfigBuilder, Retrivora } from '@retrivora-ai/rag-engine/server';
139
+
140
+ // 1. Construct Free Tier Configuration
141
+ const config = new ConfigBuilder()
142
+ .projectId("workspace-123") // Mapped directly to Pinecone Namespace
143
+ .vectorDb("pinecone", {
144
+ indexName: "retrivora-free",
145
+ apiKey: process.env.PINECONE_API_KEY,
146
+ })
147
+ .llm("universal_rest", "groq/qwen-3.6-27b", process.env.LITELLM_API_KEY, {
148
+ baseUrl: process.env.LITELLM_BASE_URL || "http://localhost:4000/v1",
149
+ profile: "litellm",
150
+ temperature: 0.7,
151
+ maxTokens: 1024,
152
+ })
153
+ .embedding("ollama", "nomic-embed-text", undefined, {
154
+ baseUrl: process.env.OLLAMA_BASE_URL || "http://localhost:11434",
155
+ dimensions: 768,
156
+ queryPrefix: "search_query: ",
157
+ documentPrefix: "search_document: ",
158
+ })
159
+ .rag({
160
+ architecture: "simple",
161
+ topK: 5,
162
+ chunkSize: 2500,
163
+ chunkOverlap: 400,
164
+ chunkingStrategy: "recursive",
165
+ useGraphRetrieval: false,
166
+ useQueryTransformation: false,
167
+ })
168
+ .build();
169
+
170
+ // 2. Initialize Engine Instance
171
+ const retrivora = new Retrivora(config);
172
+ await retrivora.initialize();
173
+ ```
174
+
175
+ ---
176
+
177
+ ### 3.2 Free Tier Preset Helper Function
178
+
179
+ ```typescript
180
+ export const FREE_TIER_LIMITS = {
181
+ maxDocuments: 20,
182
+ maxStorageBytes: 50 * 1024 * 1024, // 50 MB
183
+ maxEmbeddings: 10000,
184
+ maxQueriesPerDay: 100,
185
+ allowedFormats: ['.pdf', '.docx', '.txt', '.md'],
186
+ };
187
+
188
+ export function createFreeTierConfig(params: {
189
+ workspaceId: string;
190
+ pineconeApiKey: string;
191
+ liteLlmApiKey?: string;
192
+ liteLlmBaseUrl?: string;
193
+ }) {
194
+ return new ConfigBuilder()
195
+ .projectId(params.workspaceId)
196
+ .vectorDb("pinecone", {
197
+ indexName: "retrivora-free",
198
+ apiKey: params.pineconeApiKey,
199
+ })
200
+ .llm("universal_rest", "gpt-4o-mini", params.liteLlmApiKey, {
201
+ baseUrl: params.liteLlmBaseUrl || "http://localhost:4000/v1",
202
+ profile: "litellm",
203
+ })
204
+ .embedding("ollama", "nomic-embed-text", undefined, {
205
+ dimensions: 768,
206
+ })
207
+ .rag({
208
+ architecture: "simple",
209
+ topK: 5,
210
+ chunkSize: 2500,
211
+ chunkOverlap: 400,
212
+ chunkingStrategy: "recursive",
213
+ useGraphRetrieval: false,
214
+ })
215
+ .build();
216
+ }
217
+ ```
218
+
219
+ ---
220
+
221
+ ## 4. Feature Comparison Matrix
222
+
223
+ | Feature | Free Tier (MVP) | Enterprise Tier |
224
+ | :--- | :--- | :--- |
225
+ | **Vector DB Index** | Shared (`retrivora-free`) | Dedicated index / cluster |
226
+ | **Tenancy Isolation** | Namespace | Index / Database / Cloud Project |
227
+ | **Embedding Model** | Fixed 768-dim (`nomic-embed-text`, `bge-base`) | Custom / Multimodal / Any dim |
228
+ | **LLM Gateway** | LiteLLM Proxy | Direct API / LiteLLM / Enterprise VPC |
229
+ | **Retrieval Mode** | Semantic Search (Top-K) | Hybrid + Graph RAG + Agentic Routing |
230
+ | **Document Formats** | PDF, DOCX, TXT, MD | All formats + Web scraping + Audio |
231
+ | **Document Limit** | 20 documents | Unlimited |
232
+ | **Storage Limit** | 50 MB | Custom / Unlimited |