@thinkbase/sdk 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ThinkBase
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,291 @@
+# ThinkBase SDK
+
+![enter image description here](https://olive-chemical-haddock-701.mypinata.cloud/ipfs/bafkreid4j732kjz6372omkb6fkonic64t34mgs3h2dxaahjg6jvrwr3mj4)
+
+Turn any data into structured, searchable, and AI-ready context with a single API.
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Node.js-339933?style=for-the-badge&logo=nodedotjs&logoColor=white" />
+  <img src="https://img.shields.io/badge/TypeScript-3178C6?style=for-the-badge&logo=typescript&logoColor=white" />
+  <img src="https://img.shields.io/badge/OpenAI-412991?style=for-the-badge&logo=openai&logoColor=white" />
+    <img src="https://img.shields.io/badge/PostgreSQL-4169E1?style=for-the-badge&logo=postgresql&logoColor=white" />
+  <img src="https://img.shields.io/badge/MySQL-4479A1?style=for-the-badge&logo=mysql&logoColor=white" />
+  <img src="https://img.shields.io/badge/MongoDB-47A248?style=for-the-badge&logo=mongodb&logoColor=white" />
+    <img src="https://img.shields.io/badge/REST_API-000000?style=for-the-badge&logo=fastapi&logoColor=white" />
+  <img src="https://img.shields.io/badge/JSON_Data-000000?style=for-the-badge&logo=json&logoColor=white" />
+  <img src="https://img.shields.io/badge/Vector_Search-FF6F00?style=for-the-badge&logo=semanticweb&logoColor=white" />
+</p>
+
+
+## Installation
+
+```bash
+npm i @thinkbase/sdk
+```
+
+## Quick Start
+
+```typescript
+import { ThinkBase } from '@thinkbase/sdk';
+
+const thinkbase = new ThinkBase();
+
+const dataset = thinkbase.createDataset({
+  source: 'postgres',
+  connection: process.env.DATABASE_URL
+});
+
+await dataset.sync({
+  embedding: 'openai:text-embedding-3-small'
+});
+
+const results = await dataset.search('top performing meme coins today');
+```
+
+## Core Features
+
+### 1. Universal Data Connector
+
+Connect to any data source instantly:
+
+#### PostgreSQL
+
+```typescript
+const dataset = thinkbase.createDataset({
+  source: 'postgres',
+  connection: process.env.DATABASE_URL,
+  query: 'SELECT * FROM products WHERE active = true'
+});
+```
+
+#### MySQL
+
+```typescript
+const dataset = thinkbase.createDataset({
+  source: 'mysql',
+  connection: process.env.MYSQL_URL,
+  table: 'users'
+});
+```
+
+#### MongoDB
+
+```typescript
+const dataset = thinkbase.createDataset({
+  source: 'mongodb',
+  connection: process.env.MONGO_URL,
+  collection: 'documents'
+});
+```
+
+#### REST API
+
+```typescript
+const dataset = thinkbase.createDataset({
+  source: 'rest',
+  url: 'https://api.example.com/data'
+});
+```
+
+#### JSON Data
+
+```typescript
+const dataset = thinkbase.createDataset({
+  source: 'json',
+  data: [
+    { id: 1, name: 'Product A', price: 100 },
+    { id: 2, name: 'Product B', price: 200 }
+  ]
+});
+```
+
+### 2. Auto Embedding Pipeline
+
+Automatically convert your data into AI-searchable vectors:
+
+```typescript
+await dataset.sync({
+  embedding: 'openai:text-embedding-3-small',
+  chunkSize: 1000,
+  chunkOverlap: 200,
+  batchSize: 100
+});
+```
+
+Features:
+- Smart text chunking with overlap
+- Multi-model support (OpenAI, and more)
+- Automatic background indexing
+- Batch processing for efficiency
+
+### 3. Semantic Search API
+
+Query your data using natural language:
+
+```typescript
+const results = await dataset.search('top performing meme coins today', {
+  limit: 10,
+  threshold: 0.7,
+  hybrid: false
+});
+
+results.forEach(result => {
+  console.log(`Score: ${result.score}`);
+  console.log(`Content: ${result.content}`);
+});
+```
+
+#### Hybrid Search
+
+Combine vector similarity with keyword matching:
+
+```typescript
+const results = await dataset.search('important updates', {
+  limit: 5,
+  hybrid: true
+});
+```
+
+### 4. AI Query Layer (Natural Language → Data)
+
+Ask questions in plain English:
+
+```typescript
+const result = await dataset.ask('Which users spent the most last month?');
+
+console.log('Generated SQL:', result.sql);
+console.log('Results:', result.data);
+```
+
+This translates natural language into SQL and executes it safely.
+
+### 5. Context Builder for LLM
+
+Prepare optimized context for AI models:
+
+```typescript
+const context = await dataset.context({
+  query: 'analyze this token',
+  limit: 5,
+  maxTokens: 4000
+});
+
+const response = await openai.chat.completions.create({
+  model: 'gpt-4',
+  messages: [
+    { role: 'system', content: context },
+    { role: 'user', content: 'What are the key insights?' }
+  ]
+});
+```
+
+#### Structured Context
+
+Get structured data instead of formatted text:
+
+```typescript
+const chunks = await dataset.contextStructured({
+  query: 'product analysis',
+  limit: 3
+});
+
+chunks.forEach(chunk => {
+  console.log(chunk.content);
+  console.log('Relevance:', chunk.score);
+});
+```
+
+## Complete Example
+
+```typescript
+import { ThinkBase } from '@thinkbase/sdk';
+
+async function main() {
+  const thinkbase = new ThinkBase();
+
+  const dataset = thinkbase.createDataset({
+    source: 'postgres',
+    connection: process.env.DATABASE_URL,
+    query: 'SELECT * FROM crypto_tokens'
+  });
+
+  await dataset.sync({
+    embedding: 'openai:text-embedding-3-small',
+    chunkSize: 1000
+  });
+
+  console.log('Stats:', dataset.getStats());
+
+  const searchResults = await dataset.search(
+    'best performing tokens today',
+    { limit: 5, threshold: 0.75 }
+  );
+
+  console.log('Search Results:', searchResults);
+
+  const queryResult = await dataset.ask(
+    'What are the top 5 tokens by market cap?'
+  );
+
+  console.log('Query Results:', queryResult.data);
+  console.log('Generated SQL:', queryResult.sql);
+
+  const context = await dataset.context({
+    query: 'token performance analysis',
+    limit: 5,
+    maxTokens: 3000
+  });
+
+  console.log('LLM Context:', context);
+
+  await dataset.disconnect();
+}
+
+main().catch(console.error);
+```
+
+## Environment Variables
+
+Set up your API keys:
+
+```bash
+OPENAI_API_KEY=your_openai_key
+DATABASE_URL=postgresql://user:pass@host:5432/db
+```
+
+## API Reference
+
+### ThinkBase
+
+Main client for creating datasets.
+
+```typescript
+const thinkbase = new ThinkBase(config?: ThinkBaseConfig);
+```
+
+### Dataset
+
+Main class for working with data.
+
+#### Methods
+
+- `sync(config: EmbeddingConfig)` - Process and embed data
+- `search(query: string, options?: SearchOptions)` - Semantic search
+- `ask(query: string)` - Natural language query
+- `context(options: ContextOptions)` - Build LLM context
+- `contextStructured(options: ContextOptions)` - Get structured context
+- `getStats()` - Get dataset statistics
+- `connect()` - Connect to data source
+- `disconnect()` - Disconnect from data source
+
+## TypeScript Support
+
+Full TypeScript definitions included:
+
+```typescript
+import { ThinkBase, Dataset, SearchResult, QueryResult } from '@thinkbase/sdk';
+```
+
+## License
+
+MIT

package/dist/client.d.ts

@@ -0,0 +1,8 @@
+import { Dataset } from './dataset';
+import { ThinkBaseConfig, DataSourceConfig } from './types';
+export declare class ThinkBase {
+    private config;
+    constructor(config?: ThinkBaseConfig);
+    createDataset(config: DataSourceConfig): Dataset;
+    getConfig(): ThinkBaseConfig;
+}

package/dist/client.js

@@ -0,0 +1,16 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ThinkBase = void 0;
+const dataset_1 = require("./dataset");
+class ThinkBase {
+    constructor(config = {}) {
+        this.config = config;
+    }
+    createDataset(config) {
+        return new dataset_1.Dataset(config);
+    }
+    getConfig() {
+        return this.config;
+    }
+}
+exports.ThinkBase = ThinkBase;

package/dist/connectors/base.d.ts

@@ -0,0 +1,9 @@
+import { DataSourceConfig } from '../types';
+export declare abstract class DataConnector {
+    protected config: DataSourceConfig;
+    constructor(config: DataSourceConfig);
+    abstract connect(): Promise<void>;
+    abstract disconnect(): Promise<void>;
+    abstract fetchData(): Promise<any[]>;
+    abstract getName(): string;
+}

package/dist/connectors/base.js

@@ -0,0 +1,9 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DataConnector = void 0;
+class DataConnector {
+    constructor(config) {
+        this.config = config;
+    }
+}
+exports.DataConnector = DataConnector;

package/dist/connectors/factory.d.ts

@@ -0,0 +1,5 @@
+import { DataConnector } from './base';
+import { DataSourceConfig } from '../types';
+export declare class ConnectorFactory {
+    static create(config: DataSourceConfig): DataConnector;
+}

package/dist/connectors/factory.js

@@ -0,0 +1,29 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ConnectorFactory = void 0;
+const postgres_1 = require("./postgres");
+const mysql_1 = require("./mysql");
+const mongodb_1 = require("./mongodb");
+const rest_1 = require("./rest");
+const json_1 = require("./json");
+class ConnectorFactory {
+    static create(config) {
+        switch (config.source) {
+            case 'postgres':
+                return new postgres_1.PostgresConnector(config);
+            case 'mysql':
+                return new mysql_1.MySQLConnector(config);
+            case 'mongodb':
+                return new mongodb_1.MongoDBConnector(config);
+            case 'rest':
+            case 'graphql':
+                return new rest_1.RESTConnector(config);
+            case 'json':
+            case 'csv':
+                return new json_1.JSONConnector(config);
+            default:
+                throw new Error(`Unsupported data source: ${config.source}`);
+        }
+    }
+}
+exports.ConnectorFactory = ConnectorFactory;

package/dist/connectors/json.d.ts

@@ -0,0 +1,7 @@
+import { DataConnector } from './base';
+export declare class JSONConnector extends DataConnector {
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    fetchData(): Promise<any[]>;
+    getName(): string;
+}

package/dist/connectors/json.js

@@ -0,0 +1,22 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.JSONConnector = void 0;
+const base_1 = require("./base");
+class JSONConnector extends base_1.DataConnector {
+    async connect() {
+        // No connection needed for in-memory data
+    }
+    async disconnect() {
+        // No connection to close
+    }
+    async fetchData() {
+        if (!this.config.data) {
+            throw new Error('JSON data is required');
+        }
+        return Array.isArray(this.config.data) ? this.config.data : [this.config.data];
+    }
+    getName() {
+        return 'JSON';
+    }
+}
+exports.JSONConnector = JSONConnector;

package/dist/connectors/mongodb.d.ts

@@ -0,0 +1,9 @@
+import { DataConnector } from './base';
+export declare class MongoDBConnector extends DataConnector {
+    private client;
+    private db;
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    fetchData(): Promise<any[]>;
+    getName(): string;
+}

package/dist/connectors/mongodb.js

@@ -0,0 +1,39 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MongoDBConnector = void 0;
+const mongodb_1 = require("mongodb");
+const base_1 = require("./base");
+class MongoDBConnector extends base_1.DataConnector {
+    constructor() {
+        super(...arguments);
+        this.client = null;
+        this.db = null;
+    }
+    async connect() {
+        if (!this.config.connection) {
+            throw new Error('MongoDB connection string is required');
+        }
+        this.client = new mongodb_1.MongoClient(this.config.connection);
+        await this.client.connect();
+        this.db = this.client.db();
+    }
+    async disconnect() {
+        if (this.client) {
+            await this.client.close();
+            this.client = null;
+            this.db = null;
+        }
+    }
+    async fetchData() {
+        if (!this.db) {
+            throw new Error('Not connected to MongoDB');
+        }
+        const collection = this.config.collection || 'data';
+        const documents = await this.db.collection(collection).find({}).toArray();
+        return documents;
+    }
+    getName() {
+        return 'MongoDB';
+    }
+}
+exports.MongoDBConnector = MongoDBConnector;

package/dist/connectors/mysql.d.ts

@@ -0,0 +1,8 @@
+import { DataConnector } from './base';
+export declare class MySQLConnector extends DataConnector {
+    private connection;
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    fetchData(): Promise<any[]>;
+    getName(): string;
+}

package/dist/connectors/mysql.js

@@ -0,0 +1,38 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MySQLConnector = void 0;
+const promise_1 = __importDefault(require("mysql2/promise"));
+const base_1 = require("./base");
+class MySQLConnector extends base_1.DataConnector {
+    constructor() {
+        super(...arguments);
+        this.connection = null;
+    }
+    async connect() {
+        if (!this.config.connection) {
+            throw new Error('MySQL connection string is required');
+        }
+        this.connection = await promise_1.default.createConnection(this.config.connection);
+    }
+    async disconnect() {
+        if (this.connection) {
+            await this.connection.end();
+            this.connection = null;
+        }
+    }
+    async fetchData() {
+        if (!this.connection) {
+            throw new Error('Not connected to MySQL');
+        }
+        const query = this.config.query || `SELECT * FROM ${this.config.table || 'data'}`;
+        const [rows] = await this.connection.execute(query);
+        return rows;
+    }
+    getName() {
+        return 'MySQL';
+    }
+}
+exports.MySQLConnector = MySQLConnector;

package/dist/connectors/postgres.d.ts

@@ -0,0 +1,8 @@
+import { DataConnector } from './base';
+export declare class PostgresConnector extends DataConnector {
+    private pool;
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    fetchData(): Promise<any[]>;
+    getName(): string;
+}

package/dist/connectors/postgres.js

@@ -0,0 +1,35 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.PostgresConnector = void 0;
+const pg_1 = require("pg");
+const base_1 = require("./base");
+class PostgresConnector extends base_1.DataConnector {
+    constructor() {
+        super(...arguments);
+        this.pool = null;
+    }
+    async connect() {
+        if (!this.config.connection) {
+            throw new Error('PostgreSQL connection string is required');
+        }
+        this.pool = new pg_1.Pool({ connectionString: this.config.connection });
+    }
+    async disconnect() {
+        if (this.pool) {
+            await this.pool.end();
+            this.pool = null;
+        }
+    }
+    async fetchData() {
+        if (!this.pool) {
+            throw new Error('Not connected to PostgreSQL');
+        }
+        const query = this.config.query || `SELECT * FROM ${this.config.table || 'data'}`;
+        const result = await this.pool.query(query);
+        return result.rows;
+    }
+    getName() {
+        return 'PostgreSQL';
+    }
+}
+exports.PostgresConnector = PostgresConnector;

package/dist/connectors/rest.d.ts

@@ -0,0 +1,7 @@
+import { DataConnector } from './base';
+export declare class RESTConnector extends DataConnector {
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    fetchData(): Promise<any[]>;
+    getName(): string;
+}

package/dist/connectors/rest.js

@@ -0,0 +1,27 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.RESTConnector = void 0;
+const axios_1 = __importDefault(require("axios"));
+const base_1 = require("./base");
+class RESTConnector extends base_1.DataConnector {
+    async connect() {
+        // No persistent connection needed for REST
+    }
+    async disconnect() {
+        // No persistent connection to close
+    }
+    async fetchData() {
+        if (!this.config.url) {
+            throw new Error('REST API URL is required');
+        }
+        const response = await axios_1.default.get(this.config.url);
+        return Array.isArray(response.data) ? response.data : [response.data];
+    }
+    getName() {
+        return 'REST API';
+    }
+}
+exports.RESTConnector = RESTConnector;

package/dist/context/builder.d.ts

@@ -0,0 +1,11 @@
+import { SearchResult, ContextOptions } from '../types';
+import { DataChunk } from '../types';
+export declare class ContextBuilder {
+    private search;
+    private embedder;
+    constructor(chunks: DataChunk[], embeddingModel: string);
+    build(options: ContextOptions): Promise<string>;
+    private formatContext;
+    private estimateTokens;
+    buildStructured(options: ContextOptions): Promise<SearchResult[]>;
+}

package/dist/context/builder.js

@@ -0,0 +1,40 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ContextBuilder = void 0;
+const embedder_1 = require("../embedding/embedder");
+const vector_1 = require("../search/vector");
+class ContextBuilder {
+    constructor(chunks, embeddingModel) {
+        this.search = new vector_1.VectorSearch(chunks);
+        this.embedder = new embedder_1.Embedder(embeddingModel);
+    }
+    async build(options) {
+        const queryEmbedding = await this.embedder.embed(options.query);
+        const results = this.search.search(queryEmbedding, options.limit || 5, 0.5);
+        return this.formatContext(results, options);
+    }
+    formatContext(results, options) {
+        const maxTokens = options.maxTokens || 4000;
+        let context = `Relevant context for: "${options.query}"\n\n`;
+        let currentTokens = this.estimateTokens(context);
+        for (let i = 0; i < results.length; i++) {
+            const result = results[i];
+            const chunk = `[${i + 1}] (score: ${result.score.toFixed(3)})\n${result.content}\n\n`;
+            const chunkTokens = this.estimateTokens(chunk);
+            if (currentTokens + chunkTokens > maxTokens) {
+                break;
+            }
+            context += chunk;
+            currentTokens += chunkTokens;
+        }
+        return context.trim();
+    }
+    estimateTokens(text) {
+        return Math.ceil(text.length / 4);
+    }
+    async buildStructured(options) {
+        const queryEmbedding = await this.embedder.embed(options.query);
+        return this.search.search(queryEmbedding, options.limit || 5, 0.5);
+    }
+}
+exports.ContextBuilder = ContextBuilder;

package/dist/dataset.d.ts

@@ -0,0 +1,21 @@
+import { DataSourceConfig, EmbeddingConfig, SearchOptions, ContextOptions, DataChunk, SearchResult, QueryResult } from './types';
+export declare class Dataset {
+    private connector;
+    private chunks;
+    private embeddingConfig?;
+    private nlQuery?;
+    private contextBuilder?;
+    constructor(config: DataSourceConfig);
+    connect(): Promise<void>;
+    disconnect(): Promise<void>;
+    sync(config: EmbeddingConfig): Promise<void>;
+    search(query: string, options?: SearchOptions): Promise<SearchResult[]>;
+    ask(query: string): Promise<QueryResult>;
+    context(options: ContextOptions): Promise<string>;
+    contextStructured(options: ContextOptions): Promise<SearchResult[]>;
+    getChunks(): DataChunk[];
+    getStats(): {
+        totalChunks: number;
+        totalDocuments: number;
+    };
+}

package/dist/dataset.js

@@ -0,0 +1,83 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Dataset = void 0;
+const factory_1 = require("./connectors/factory");
+const pipeline_1 = require("./embedding/pipeline");
+const vector_1 = require("./search/vector");
+const natural_language_1 = require("./query/natural-language");
+const builder_1 = require("./context/builder");
+const embedder_1 = require("./embedding/embedder");
+class Dataset {
+    constructor(config) {
+        this.chunks = [];
+        this.connector = factory_1.ConnectorFactory.create(config);
+    }
+    async connect() {
+        await this.connector.connect();
+    }
+    async disconnect() {
+        await this.connector.disconnect();
+    }
+    async sync(config) {
+        this.embeddingConfig = config;
+        await this.connector.connect();
+        const data = await this.connector.fetchData();
+        await this.connector.disconnect();
+        const pipeline = new pipeline_1.EmbeddingPipeline(config);
+        this.chunks = await pipeline.process(data);
+        this.contextBuilder = new builder_1.ContextBuilder(this.chunks, config.embedding);
+        this.nlQuery = new natural_language_1.NaturalLanguageQuery();
+    }
+    async search(query, options = {}) {
+        if (!this.embeddingConfig) {
+            throw new Error('Dataset not synced. Call sync() first.');
+        }
+        const embedder = new embedder_1.Embedder(this.embeddingConfig.embedding);
+        const queryEmbedding = await embedder.embed(query);
+        const search = new vector_1.VectorSearch(this.chunks);
+        if (options.hybrid) {
+            const keywords = query.toLowerCase().split(' ');
+            return search.hybridSearch(queryEmbedding, keywords, options.limit || 10, 0.7);
+        }
+        return search.search(queryEmbedding, options.limit || 10, options.threshold || 0.7);
+    }
+    async ask(query) {
+        if (!this.nlQuery) {
+            this.nlQuery = new natural_language_1.NaturalLanguageQuery();
+        }
+        await this.connector.connect();
+        try {
+            const result = await this.nlQuery.executeQuery(query, async (sql) => {
+                return await this.connector.fetchData();
+            });
+            await this.connector.disconnect();
+            return result;
+        }
+        catch (error) {
+            await this.connector.disconnect();
+            throw error;
+        }
+    }
+    async context(options) {
+        if (!this.contextBuilder) {
+            throw new Error('Dataset not synced. Call sync() first.');
+        }
+        return this.contextBuilder.build(options);
+    }
+    async contextStructured(options) {
+        if (!this.contextBuilder) {
+            throw new Error('Dataset not synced. Call sync() first.');
+        }
+        return this.contextBuilder.buildStructured(options);
+    }
+    getChunks() {
+        return this.chunks;
+    }
+    getStats() {
+        return {
+            totalChunks: this.chunks.length,
+            totalDocuments: this.chunks.length
+        };
+    }
+}
+exports.Dataset = Dataset;

package/dist/embedding/chunker.d.ts

@@ -0,0 +1,10 @@
+export declare class TextChunker {
+    private chunkSize;
+    private chunkOverlap;
+    constructor(chunkSize?: number, chunkOverlap?: number);
+    chunk(text: string): string[];
+    chunkDocuments(documents: any[]): Array<{
+        content: string;
+        metadata: any;
+    }>;
+}

package/dist/embedding/chunker.js

@@ -0,0 +1,41 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.TextChunker = void 0;
+class TextChunker {
+    constructor(chunkSize = 1000, chunkOverlap = 200) {
+        this.chunkSize = chunkSize;
+        this.chunkOverlap = chunkOverlap;
+    }
+    chunk(text) {
+        const chunks = [];
+        let start = 0;
+        while (start < text.length) {
+            const end = Math.min(start + this.chunkSize, text.length);
+            let chunk = text.slice(start, end);
+            if (end < text.length) {
+                const lastSpace = chunk.lastIndexOf(' ');
+                if (lastSpace > 0) {
+                    chunk = chunk.slice(0, lastSpace);
+                }
+            }
+            chunks.push(chunk.trim());
+            start += this.chunkSize - this.chunkOverlap;
+        }
+        return chunks.filter(c => c.length > 0);
+    }
+    chunkDocuments(documents) {
+        const chunkedDocs = [];
+        for (const doc of documents) {
+            const text = typeof doc === 'string' ? doc : JSON.stringify(doc);
+            const chunks = this.chunk(text);
+            for (const chunk of chunks) {
+                chunkedDocs.push({
+                    content: chunk,
+                    metadata: typeof doc === 'object' ? doc : {}
+                });
+            }
+        }
+        return chunkedDocs;
+    }
+}
+exports.TextChunker = TextChunker;

package/dist/embedding/embedder.d.ts

@@ -0,0 +1,9 @@
+export declare class Embedder {
+    private model;
+    private client;
+    constructor(model: string);
+    private isOpenAIModel;
+    private getModelName;
+    embed(text: string): Promise<number[]>;
+    embedBatch(texts: string[], batchSize?: number): Promise<number[][]>;
+}

package/dist/embedding/embedder.js

@@ -0,0 +1,49 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Embedder = void 0;
+const openai_1 = __importDefault(require("openai"));
+class Embedder {
+    constructor(model) {
+        this.client = null;
+        this.model = model;
+        if (this.isOpenAIModel()) {
+            this.client = new openai_1.default({
+                apiKey: process.env.OPENAI_API_KEY
+            });
+        }
+    }
+    isOpenAIModel() {
+        return this.model.startsWith('openai:');
+    }
+    getModelName() {
+        return this.model.replace('openai:', '');
+    }
+    async embed(text) {
+        if (this.isOpenAIModel() && this.client) {
+            const response = await this.client.embeddings.create({
+                model: this.getModelName(),
+                input: text
+            });
+            return response.data[0].embedding;
+        }
+        throw new Error(`Unsupported embedding model: ${this.model}`);
+    }
+    async embedBatch(texts, batchSize = 100) {
+        const embeddings = [];
+        for (let i = 0; i < texts.length; i += batchSize) {
+            const batch = texts.slice(i, i + batchSize);
+            if (this.isOpenAIModel() && this.client) {
+                const response = await this.client.embeddings.create({
+                    model: this.getModelName(),
+                    input: batch
+                });
+                embeddings.push(...response.data.map(d => d.embedding));
+            }
+        }
+        return embeddings;
+    }
+}
+exports.Embedder = Embedder;

package/dist/embedding/pipeline.d.ts

@@ -0,0 +1,8 @@
+import { DataChunk, EmbeddingConfig } from '../types';
+export declare class EmbeddingPipeline {
+    private chunker;
+    private embedder;
+    private config;
+    constructor(config: EmbeddingConfig);
+    process(documents: any[]): Promise<DataChunk[]>;
+}

package/dist/embedding/pipeline.js

@@ -0,0 +1,24 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.EmbeddingPipeline = void 0;
+const chunker_1 = require("./chunker");
+const embedder_1 = require("./embedder");
+class EmbeddingPipeline {
+    constructor(config) {
+        this.config = config;
+        this.chunker = new chunker_1.TextChunker(config.chunkSize || 1000, config.chunkOverlap || 200);
+        this.embedder = new embedder_1.Embedder(config.embedding);
+    }
+    async process(documents) {
+        const chunked = this.chunker.chunkDocuments(documents);
+        const texts = chunked.map(doc => doc.content);
+        const embeddings = await this.embedder.embedBatch(texts, this.config.batchSize || 100);
+        return chunked.map((doc, index) => ({
+            id: `chunk_${index}`,
+            content: doc.content,
+            embedding: embeddings[index],
+            metadata: doc.metadata
+        }));
+    }
+}
+exports.EmbeddingPipeline = EmbeddingPipeline;

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export { ThinkBase } from './client';
+export { Dataset } from './dataset';
+export * from './types';

package/dist/index.js

@@ -0,0 +1,22 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Dataset = exports.ThinkBase = void 0;
+var client_1 = require("./client");
+Object.defineProperty(exports, "ThinkBase", { enumerable: true, get: function () { return client_1.ThinkBase; } });
+var dataset_1 = require("./dataset");
+Object.defineProperty(exports, "Dataset", { enumerable: true, get: function () { return dataset_1.Dataset; } });
+__exportStar(require("./types"), exports);

package/dist/query/natural-language.d.ts

@@ -0,0 +1,9 @@
+import { QueryResult } from '../types';
+export declare class NaturalLanguageQuery {
+    private client;
+    private schema;
+    constructor(schema?: any);
+    setSchema(schema: any): void;
+    translateToSQL(query: string, dialect?: string): Promise<string>;
+    executeQuery(query: string, executor: (sql: string) => Promise<any[]>): Promise<QueryResult>;
+}

package/dist/query/natural-language.js

@@ -0,0 +1,44 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.NaturalLanguageQuery = void 0;
+const openai_1 = __importDefault(require("openai"));
+class NaturalLanguageQuery {
+    constructor(schema) {
+        this.client = new openai_1.default({
+            apiKey: process.env.OPENAI_API_KEY
+        });
+        this.schema = schema;
+    }
+    setSchema(schema) {
+        this.schema = schema;
+    }
+    async translateToSQL(query, dialect = 'postgres') {
+        const schemaContext = this.schema
+            ? `Database schema:\n${JSON.stringify(this.schema, null, 2)}\n\n`
+            : '';
+        const prompt = `${schemaContext}Convert this natural language query to ${dialect} SQL:
+"${query}"
+
+Return ONLY the SQL query, no explanations.`;
+        const response = await this.client.chat.completions.create({
+            model: 'gpt-4o-mini',
+            messages: [{ role: 'user', content: prompt }],
+            temperature: 0
+        });
+        return response.choices[0].message.content?.trim() || '';
+    }
+    async executeQuery(query, executor) {
+        const sql = await this.translateToSQL(query);
+        try {
+            const data = await executor(sql);
+            return { data, sql };
+        }
+        catch (error) {
+            throw new Error(`Query execution failed: ${error.message}`);
+        }
+    }
+}
+exports.NaturalLanguageQuery = NaturalLanguageQuery;

package/dist/search/vector.d.ts

@@ -0,0 +1,8 @@
+import { DataChunk, SearchResult } from '../types';
+export declare class VectorSearch {
+    private chunks;
+    constructor(chunks: DataChunk[]);
+    cosineSimilarity(a: number[], b: number[]): number;
+    search(queryEmbedding: number[], limit?: number, threshold?: number): SearchResult[];
+    hybridSearch(queryEmbedding: number[], keywords: string[], limit?: number, vectorWeight?: number): SearchResult[];
+}

package/dist/search/vector.js

@@ -0,0 +1,65 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.VectorSearch = void 0;
+class VectorSearch {
+    constructor(chunks) {
+        this.chunks = chunks;
+    }
+    cosineSimilarity(a, b) {
+        if (a.length !== b.length) {
+            throw new Error('Vectors must have the same length');
+        }
+        let dotProduct = 0;
+        let normA = 0;
+        let normB = 0;
+        for (let i = 0; i < a.length; i++) {
+            dotProduct += a[i] * b[i];
+            normA += a[i] * a[i];
+            normB += b[i] * b[i];
+        }
+        return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
+    }
+    search(queryEmbedding, limit = 5, threshold = 0.7) {
+        const results = [];
+        for (const chunk of this.chunks) {
+            if (!chunk.embedding)
+                continue;
+            const score = this.cosineSimilarity(queryEmbedding, chunk.embedding);
+            if (score >= threshold) {
+                results.push({
+                    id: chunk.id,
+                    content: chunk.content,
+                    score,
+                    metadata: chunk.metadata
+                });
+            }
+        }
+        return results
+            .sort((a, b) => b.score - a.score)
+            .slice(0, limit);
+    }
+    hybridSearch(queryEmbedding, keywords, limit = 5, vectorWeight = 0.7) {
+        const results = [];
+        const keywordWeight = 1 - vectorWeight;
+        for (const chunk of this.chunks) {
+            if (!chunk.embedding)
+                continue;
+            const vectorScore = this.cosineSimilarity(queryEmbedding, chunk.embedding);
+            const lowerContent = chunk.content.toLowerCase();
+            const keywordScore = keywords.reduce((score, keyword) => {
+                return score + (lowerContent.includes(keyword.toLowerCase()) ? 1 : 0);
+            }, 0) / keywords.length;
+            const combinedScore = (vectorScore * vectorWeight) + (keywordScore * keywordWeight);
+            results.push({
+                id: chunk.id,
+                content: chunk.content,
+                score: combinedScore,
+                metadata: chunk.metadata
+            });
+        }
+        return results
+            .sort((a, b) => b.score - a.score)
+            .slice(0, limit);
+    }
+}
+exports.VectorSearch = VectorSearch;

package/dist/types.d.ts

@@ -0,0 +1,48 @@
+export interface ThinkBaseConfig {
+    apiKey?: string;
+    apiUrl?: string;
+}
+export type DataSourceType = 'postgres' | 'mysql' | 'mongodb' | 'rest' | 'graphql' | 'csv' | 'json' | 'webscrape';
+export interface DataSourceConfig {
+    source: DataSourceType;
+    connection?: string;
+    url?: string;
+    data?: any;
+    query?: string;
+    collection?: string;
+    table?: string;
+}
+export interface EmbeddingConfig {
+    embedding: string;
+    chunkSize?: number;
+    chunkOverlap?: number;
+    batchSize?: number;
+}
+export interface SearchOptions {
+    limit?: number;
+    threshold?: number;
+    filter?: Record<string, any>;
+    hybrid?: boolean;
+}
+export interface ContextOptions {
+    query: string;
+    limit?: number;
+    maxTokens?: number;
+}
+export interface DataChunk {
+    id: string;
+    content: string;
+    embedding?: number[];
+    metadata?: Record<string, any>;
+}
+export interface SearchResult {
+    id: string;
+    content: string;
+    score: number;
+    metadata?: Record<string, any>;
+}
+export interface QueryResult {
+    data: any[];
+    sql?: string;
+    metadata?: Record<string, any>;
+}

package/dist/types.js

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "@thinkbase/sdk",
+  "version": "1.0.0",
+  "description": "ThinkBase SDK - Turn any data into structured, searchable, and AI-ready context with a single API",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "files": [
+    "dist",
+    "README.md"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "test": "echo \"Error: no test specified\" && exit 1",
+    "prepublishOnly": "npm run build"
+  },
+  "keywords": [
+    "ai",
+    "embeddings",
+    "semantic-search",
+    "vector-search",
+    "llm",
+    "data-layer",
+    "thinkbase"
+  ],
+  "author": "ThinkBase",
+  "license": "MIT",
+  "dependencies": {
+    "openai": "^4.68.4",
+    "pg": "^8.13.1",
+    "mysql2": "^3.11.5",
+    "mongodb": "^6.10.0",
+    "axios": "^1.7.9"
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.2",
+    "@types/pg": "^8.11.10",
+    "typescript": "^5.7.2"
+  }
+}