gs-search 0.1.0

package/README.ja.md

@@ -0,0 +1,126 @@
+# gs-search
+
+## 他の言語
+
+- [中文 README](README.zh-CN.md)
+- [English README](README.md)
+- [한국어 README](README.ko.md)
+
+JavaScript/TypeScriptアプリケーション向けの軽量、高速、メモリ効率の良い全文検索エンジンです。
+
+## 特徴
+
+- 🔍 **全文検索**（トークン化サポート付き）
+- 📦 **軽量**（外部依存関係なし）
+- ⚡ **高速**な検索パフォーマンス
+- 📱 **ブラウザ & Node.js** サポート
+- 🌐 **多言語**トークン化
+- 🗄️ **カスタムストレージ**サポート
+- 📊 **バッチ操作**（効率的なインデックス作成）
+
+## インストール
+
+```bash
+# Yarnを使用する場合
+yarn add gs-search
+
+# npmを使用する場合
+npm install gs-search
+```
+
+## クイックスタート
+
+```typescript
+import { SimpleSearch } from 'gs-search';
+
+// 検索エンジンインスタンスを作成
+const searchEngine = new SimpleSearch();
+
+// ドキュメントを追加
+await searchEngine.addDocuments([
+  { id: 1, text: 'Hello world!' },
+  { id: 2, text: 'This is a test document' },
+  { id: 3, text: 'Another document for testing' }
+]);
+
+// 検索
+const results = await searchEngine.search('test');
+console.log(results);
+// 出力: [{ id: 2, score: 1.5, tokens: ['test'] }, { id: 3, score: 1.5, tokens: ['test'] }]
+
+// ドキュメントを削除
+await searchEngine.deleteDocument(1);
+
+// 検索エンジンのステータスを取得
+const status = await searchEngine.getStatus();
+console.log(status);
+```
+
+## 高度な使用方法
+
+### SearchEngine
+
+より細かい制御と高度な機能には、`SearchEngine`を使用します：
+
+```typescript
+import { SearchEngine, NodeStorage } from 'gs-search';
+
+// カスタムストレージでエンジンを作成
+const engine = new SearchEngine({
+  storage: new NodeStorage('./search-data')
+});
+
+// エンジンを初期化
+await engine.init();
+
+// トランザクション内でドキュメントを追加
+await engine.beginTransaction();
+try {
+  await engine.addDocuments([
+    // ... ドキュメント
+  ]);
+  await engine.commit();
+} catch (error) {
+  await engine.rollback();
+}
+```
+
+## APIリファレンス
+
+### SimpleSearch
+
+- `constructor()`: 新しい検索エンジンインスタンスを作成
+- `addDocument(doc: IDocument): Promise<void>`: 単一のドキュメントを追加
+- `addDocuments(docs: IDocument[]): Promise<void>`: 複数のドキュメントを追加
+- `deleteDocument(id: number): Promise<void>`: ドキュメントを削除
+- `search(query: string, limit?: number): Promise<IResult[]>`: ドキュメントを検索
+- `getStatus(): Promise<IStatus>`: 検索エンジンのステータスを取得
+
+### CoreSearchEngine
+
+- `constructor(options: ICoreSearchOptions)`: 新しいコアエンジンインスタンスを作成
+- `init(): Promise<void>`: エンジンを初期化
+- `addDocument(doc: IDocument): Promise<void>`: 単一のドキュメントを追加
+- `addDocuments(docs: IDocument[]): Promise<void>`: 複数のドキュメントを追加
+- `deleteDocument(id: number): Promise<void>`: ドキュメントを削除
+- `search(query: string, limit?: number): Promise<IResult[]>`: ドキュメントを検索
+- `getStatus(): Promise<IStatus>`: 検索エンジンのステータスを取得
+- `beginTransaction(): void`: トランザクションを開始
+- `commit(): Promise<void>`: トランザクションをコミット
+- `rollback(): void`: トランザクションをロールバック
+
+## ストレージ
+
+検索エンジンはカスタムストレージ実装をサポートしています：
+
+- `BrowserStorage`: ブラウザ環境用（IndexedDBを使用）
+- `NodeStorage`: Node.js環境用（ファイルシステムを使用）
+
+## ライセンス
+
+MIT License
+
+## リンク
+
+- [GitHubリポジトリ](https://github.com/grain-sand/gs-search)
+- [npmパッケージ](https://www.npmjs.com/package/gs-search)

package/README.ko.md

@@ -0,0 +1,126 @@
+# gs-search
+
+## 다른 언어
+
+- [中文 README](README.zh-CN.md)
+- [English README](README.md)
+- [日本語 README](README.ja.md)
+
+JavaScript/TypeScript 애플리케이션을 위한 가볍고 빠르며 메모리 효율적인 전문 검색 엔진입니다.
+
+## 기능
+
+- 🔍 **전문 검색** 토큰화 지원
+ - 📦 **가볍고** 외부 의존성 없음
+- ⚡ **고속** 검색 성능
+- 📱 **브라우저 & Node.js** 지원
+- 🌐 **다국어** 토큰화
+- 🗄️ **커스텀 스토리지** 지원
+- 📊 **일괄 작업** 효율적인 인덱싱
+
+## 설치
+
+```bash
+# Yarn 사용
+yarn add gs-search
+
+# npm 사용
+npm install gs-search
+```
+
+## 빠른 시작
+
+```typescript
+import { SimpleSearch } from 'gs-search';
+
+// 검색 엔진 인스턴스 생성
+const searchEngine = new SimpleSearch();
+
+// 문서 추가
+await searchEngine.addDocuments([
+  { id: 1, text: 'Hello world!' },
+  { id: 2, text: 'This is a test document' },
+  { id: 3, text: 'Another document for testing' }
+]);
+
+// 검색
+const results = await searchEngine.search('test');
+console.log(results);
+// 출력: [{ id: 2, score: 1.5, tokens: ['test'] }, { id: 3, score: 1.5, tokens: ['test'] }]
+
+// 문서 삭제
+await searchEngine.deleteDocument(1);
+
+// 검색 엔진 상태 가져오기
+const status = await searchEngine.getStatus();
+console.log(status);
+```
+
+## 고급 사용법
+
+### SearchEngine
+
+보다 세밀한 제어와 고급 기능을 위해 `SearchEngine`를 사용하세요:
+
+```typescript
+import { SearchEngine, NodeStorage } from 'gs-search';
+
+// 커스텀 스토리지로 엔진 생성
+const engine = new SearchEngine({
+  storage: new NodeStorage('./search-data')
+});
+
+// 엔진 초기화
+await engine.init();
+
+// 트랜잭션 내에서 문서 추가
+await engine.beginTransaction();
+try {
+  await engine.addDocuments([
+    // ... 문서
+  ]);
+  await engine.commit();
+} catch (error) {
+  await engine.rollback();
+}
+```
+
+## API 참조
+
+### SimpleSearch
+
+- `constructor()`: 새로운 검색 엔진 인스턴스 생성
+- `addDocument(doc: IDocument): Promise<void>`: 단일 문서 추가
+- `addDocuments(docs: IDocument[]): Promise<void>`: 여러 문서 추가
+- `deleteDocument(id: number): Promise<void>`: 문서 삭제
+- `search(query: string, limit?: number): Promise<IResult[]>`: 문서 검색
+- `getStatus(): Promise<IStatus>`: 검색 엔진 상태 가져오기
+
+### CoreSearchEngine
+
+- `constructor(options: ICoreSearchOptions)`: 새로운 코어 엔진 인스턴스 생성
+- `init(): Promise<void>`: 엔진 초기화
+- `addDocument(doc: IDocument): Promise<void>`: 단일 문서 추가
+- `addDocuments(docs: IDocument[]): Promise<void>`: 여러 문서 추가
+- `deleteDocument(id: number): Promise<void>`: 문서 삭제
+- `search(query: string, limit?: number): Promise<IResult[]>`: 문서 검색
+- `getStatus(): Promise<IStatus>`: 검색 엔진 상태 가져오기
+- `beginTransaction(): void`: 트랜잭션 시작
+- `commit(): Promise<void>`: 트랜잭션 커밋
+- `rollback(): void`: 트랜잭션 롤백
+
+## 스토리지
+
+검색 엔진은 커스텀 스토리지 구현을 지원합니다:
+
+- `BrowserStorage`: 브라우저 환경용 (IndexedDB 사용)
+- `NodeStorage`: Node.js 환경용 (파일 시스템 사용)
+
+## 라이선스
+
+MIT License
+
+## 링크
+
+- [GitHub 리포지토리](https://github.com/grain-sand/gs-search)
+- [npm 패키지](https://www.npmjs.com/package/gs-search)

package/README.md

@@ -0,0 +1,128 @@
+# gs-search
+
+## Other Languages
+
+- [中文 README](README.zh-CN.md)
+- [日本語 README](README.ja.md)
+- [한국어 README](README.ko.md)
+
+A lightweight, fast, and memory-efficient full-text search engine for JavaScript/TypeScript applications.
+
+## Features
+
+- 🔍 **Full-text search** with tokenization support
+- 📦 **Lightweight** with zero external dependencies
+- ⚡ **Fast** search performance
+- 📱 **Browser & Node.js** support
+- 🌐 **Multi-language** tokenization
+- 🗄️ **Custom storage** support
+- 📊 **Batch operations** for efficient indexing
+
+## Installation
+
+```bash
+# Using yarn
+yarn add gs-search
+
+# Using npm
+npm install gs-search
+```
+
+## Quick Start
+
+```typescript
+import { SimpleSearch } from 'gs-search';
+
+// Add documents in batch
+await SimpleSearch.addDocuments([
+  { id: 1, text: 'Hello world!' },
+  { id: 2, text: 'This is a test document' },
+  { id: 3, text: 'Another document for testing' }
+]);
+
+// Add a single document
+await SimpleSearch.addDocument({ id: 4, text: 'Single document addition' });
+
+// Search
+const results = await SimpleSearch.search('test');
+console.log(results);
+// Output: [{ id: 2, score: 1.5, tokens: ['test'] }, { id: 3, score: 1.5, tokens: ['test'] }]
+
+// Delete a document
+await SimpleSearch.removeDocument(1);
+
+// Get search engine status
+const status = await SimpleSearch.getStatus();
+console.log(status);
+```
+
+## Advanced Usage
+
+### SearchEngine
+
+For more control and advanced features, use the `SearchEngine`:
+
+```typescript
+import { SearchEngine, NodeStorage } from 'gs-search';
+
+// Create engine with custom storage
+const engine = new SearchEngine({
+  storage: new NodeStorage('./search-data')
+});
+
+// Initialize engine
+await engine.init();
+
+// Add documents in batch
+await engine.startBatch();
+try {
+  await engine.addDocuments([
+    // ... documents
+  ]);
+  await engine.endBatch();
+} catch (error) {
+  // Handle error
+}
+```
+
+## API Reference
+
+### SimpleSearch
+
+**Static Methods (No instance creation required):**
+- `configure(config: Partial<ISearchEngineConfig>): void`: Configure the search engine
+- `addDocument(doc: IDocument): Promise<void>`: Add a single document
+- `addDocuments(docs: IDocument[]): Promise<void>`: Add multiple documents
+- `removeDocument(id: number): Promise<void>`: Delete a document
+- `search(query: string, limit?: number): Promise<IResult[]>`: Search for documents
+- `getStatus(): Promise<IStatus>`: Get search engine status
+- `startBatch(): void`: Start batch operations
+- `endBatch(): Promise<void>`: End batch operations
+
+### CoreSearchEngine
+
+- `constructor(options: ICoreSearchOptions)`: Create a new core engine instance
+- `init(): Promise<void>`: Initialize the engine
+- `addDocument(doc: IDocument): Promise<void>`: Add a single document
+- `addDocuments(docs: IDocument[]): Promise<void>`: Add multiple documents
+- `removeDocument(id: number): Promise<void>`: Delete a document
+- `search(query: string, limit?: number): Promise<IResult[]>`: Search for documents
+- `getStatus(): Promise<IStatus>`: Get search engine status
+- `startBatch(): void`: Start batch operations
+- `endBatch(): Promise<void>`: End batch operations
+
+## Storage
+
+The search engine supports custom storage implementations:
+
+- `BrowserStorage`: For browser environments (uses IndexedDB)
+- `NodeStorage`: For Node.js environments (uses file system)
+
+## License
+
+MIT License
+
+## Links
+
+- [GitHub Repository](https://github.com/grain-sand/gs-search)
+- [npm Package](https://www.npmjs.com/package/gs-search)

package/README.zh-CN.md

@@ -0,0 +1,224 @@
+# gs-search
+
+## 其他语言
+
+- [English README](README.md)
+- [日本語 README](README.ja.md)
+- [한국어 README](README.ko.md)
+
+可以在现代浏览器运行，且会自动存储索引的极小纯前端搜索库，可以配合其它强大的分词库使用效果更好
+
+## 特性
+
+- 🔍 **全文搜索** 支持多语言分词
+- 📦 **轻量级** 无任何第三方依赖，体积小
+- ⚡ **高性能** 快速搜索与索引构建
+- 📱 **浏览器兼容** 支持现代浏览器
+- 🌐 **多语言支持** 适配不同语言分词需求
+- 🗄️ **自定义存储** 支持灵活的存储实现
+- 📊 **批处理操作** 批量添加文档更高效
+
+## 安装
+
+```bash
+# 使用yarn
+yarn add gs-search
+
+# 使用npm
+npm install gs-search
+```
+
+## 快速开始
+
+```typescript
+import { SimpleSearch } from 'gs-search';
+
+// 批量添加文档
+await SimpleSearch.addDocuments([
+  { id: 1, text: 'Hello world!' },
+  { id: 2, text: '这是一个测试文档' },
+  { id: 3, text: '另一个用于测试的文档' }
+]);
+
+// 添加单个文档
+await SimpleSearch.addDocument({ id: 4, text: '单个文档添加示例' });
+
+// 搜索
+const results = await SimpleSearch.search('测试');
+console.log(results);
+// 输出: [{ id: 2, score: 1.5, tokens: ['测试'] }, { id: 3, score: 1.5, tokens: ['测试'] }]
+
+// 删除文档
+await SimpleSearch.removeDocument(1);
+
+// 获取搜索引擎状态
+const status = await SimpleSearch.getStatus();
+console.log(status);
+```
+
+## 高级用法
+
+### SearchEngine
+
+如需更多控制和高级功能，可以使用 `SearchEngine`：
+
+```typescript
+import { SearchEngine, NodeStorage } from 'gs-search';
+
+// 创建带有自定义存储的引擎
+const engine = new SearchEngine({
+  storage: new NodeStorage('./search-data')
+});
+
+// 初始化引擎
+await engine.init();
+
+// 批处理添加文档
+await engine.startBatch();
+try {
+  await engine.addDocuments([
+    // ... 文档
+  ]);
+  await engine.endBatch();
+} catch (error) {
+  // 处理错误
+}
+```
+
+### 自定义存储
+
+您可以实现自定义存储来持久化数据：
+
+```typescript
+import { StorageInterface } from 'gs-search';
+
+class CustomStorage implements StorageInterface {
+  async saveIndex(index: any): Promise<void> {
+    // 保存索引到自定义存储
+  }
+
+  async loadIndex(): Promise<any | null> {
+    // 从自定义存储加载索引
+    return null;
+  }
+
+  async clear(): Promise<void> {
+    // 清空存储
+  }
+}
+
+// 使用自定义存储
+const storage = new CustomStorage();
+const engine = new SearchEngine({ storage });
+```
+
+### 事务支持
+
+使用事务进行批量操作以提高性能：
+
+```typescript
+await engine.startTransaction();
+
+try {
+  // 批量添加文档
+  for (let i = 0; i < 1000; i++) {
+    await engine.addDocuments([{ id: i, text: `文档 ${i}` }]);
+  }
+  
+  // 提交事务
+  await engine.commitTransaction();
+} catch (error) {
+  // 回滚事务
+  await engine.rollbackTransaction();
+}
+```
+
+## 自定义分词器
+
+您可以通过配置自定义分词器来支持特定的语言或分词需求。以下是一个简单的正则分词器示例，按空格和字符分词，且最长token不超过5字符：
+
+```typescript
+import { SimpleSearch } from 'gs-search';
+
+// 自定义分词器：按空格和字符分词，最长token不超过5字符
+const customTokenizer = (text: string): string[] => {
+  // 按空格分词
+  const tokens: string[] = [];
+  const words = text.toLowerCase().split(/\s+/);
+  
+  // 对每个单词，按字符切分，最长不超过5字符
+  for (const word of words) {
+    if (word.length <= 5) {
+      tokens.push(word);
+    } else {
+      // 超过5字符的单词按字符切分
+      for (let i = 0; i < word.length; i++) {
+        tokens.push(word[i]);
+      }
+    }
+  }
+  
+  return tokens;
+};
+
+// 配置自定义分词器
+SimpleSearch.configure({
+  indexingTokenizer: customTokenizer,
+  searchTokenizer: customTokenizer
+});
+```
+
+## API参考
+
+### SimpleSearch
+
+**静态方法（无需创建实例）：**
+- `configure(config: Partial<ISearchEngineConfig>): void`: 配置搜索引擎
+- `addDocument(doc: IDocument): Promise<void>`: 添加单个文档
+- `addDocuments(docs: IDocument[]): Promise<void>`: 批量添加文档
+- `removeDocument(id: number): Promise<void>`: 删除文档
+- `search(query: string, limit?: number): Promise<IResult[]>`: 搜索文档
+- `getStatus(): Promise<IStatus>`: 获取搜索引擎状态
+- `startBatch(): void`: 开始批处理操作
+- `endBatch(): Promise<void>`: 结束批处理操作
+
+### CoreSearchEngine
+
+- `constructor(options: ICoreSearchOptions)`: 创建核心引擎实例
+- `init(): Promise<void>`: 初始化引擎
+- `addDocument(doc: IDocument): Promise<void>`: 添加单个文档
+- `addDocuments(docs: IDocument[]): Promise<void>`: 批量添加文档
+- `removeDocument(id: number): Promise<void>`: 删除文档
+- `search(query: string, limit?: number): Promise<IResult[]>`: 搜索文档
+- `getStatus(): Promise<IStatus>`: 获取搜索引擎状态
+- `startBatch(): void`: 开始批处理
+- `endBatch(): Promise<void>`: 结束批处理
+
+## 存储支持
+
+gs-search支持多种存储方式：
+
+- **InMemoryStorage**：内存存储（默认）
+- **LocalStorage**：浏览器本地存储
+- **IndexedDBStorage**：浏览器IndexedDB存储
+- **自定义存储**：实现StorageInterface接口
+
+## 浏览器支持
+
+- Chrome (最新)
+- Firefox (最新)
+- Safari (最新)
+- Edge (最新)
+
+## 许可证
+
+[MIT License](LICENSE)
+
+## 贡献
+
+欢迎贡献代码！请查看[GitHub仓库](https://github.com/grain-sand/gs-search)了解更多信息。
+
+## 联系方式
+
+- [GitHub Repository](https://github.com/grain-sand/gs-search)
+- [npm Package](https://www.npmjs.com/package/gs-search)

package/lib/index.cjs

@@ -0,0 +1 @@
+"use strict";Object.create,Object.defineProperty,Object.getOwnPropertyDescriptor,Object.getOwnPropertyNames,Object.getPrototypeOf,Object.prototype.hasOwnProperty;class t{#t;constructor(t){this.#t=t}async#e(){return await(await navigator.storage.getDirectory()).getDirectoryHandle(this.#t,{create:!0})}async write(t,e){const s=await(await(await this.#e()).getFileHandle(t,{create:!0})).createWritable();await s.write(e),await s.close()}async append(t,e){const s=await this.#e();let i;try{i=await s.getFileHandle(t,{create:!0})}catch{i=await s.getFileHandle(t,{create:!0})}const n=await i.getFile(),a=await i.createWritable({keepExistingData:!0});await a.seek(n.size),await a.write(e),await a.close()}async read(t){const e=await this.#e();try{return await(await(await e.getFileHandle(t)).getFile()).arrayBuffer()}catch{return null}}async readRange(t,e,s){const i=await this.#e();try{return await(await(await i.getFileHandle(t)).getFile()).slice(e,s).arrayBuffer()}catch{return null}}async remove(t){const e=await this.#e();try{await e.removeEntry(t)}catch{}}async listFiles(){const t=await this.#e(),e=[];for await(const s of t.keys())e.push(s);return e}async clearAll(){const t=await this.#e();for await(const e of t.keys())await t.removeEntry(e,{recursive:!0})}async getFileSize(t){const e=await this.#e();try{return(await(await e.getFileHandle(t)).getFile()).size}catch{return 0}}}class e{#s=null;#i=null;#t;#n="";constructor(t){this.#t=t}async#a(){if(this.#s)return;const t=await import("node:fs"),e=await import("node:path");this.#s=t.promises,this.#i=e.default||e,this.#n=this.#i.join(process.cwd(),this.#t);try{await this.#s.access(this.#n)}catch{await this.#s.mkdir(this.#n,{recursive:!0})}}#r(t){return this.#i.join(this.#n,t)}async write(t,e){await this.#a(),await this.#s.writeFile(this.#r(t),Buffer.from(e))}async append(t,e){await this.#a(),await this.#s.appendFile(this.#r(t),Buffer.from(e))}async read(t){await this.#a();try{const e=await this.#s.readFile(this.#r(t));return e.buffer.slice(e.byteOffset,e.byteOffset+e.byteLength)}catch{return null}}async readRange(t,e,s){await this.#a();try{const i=await this.#s.open(this.#r(t),"r"),n=s-e,a=Buffer.alloc(n);return await i.read(a,0,n,e),await i.close(),a.buffer.slice(a.byteOffset,a.byteOffset+a.byteLength)}catch{return null}}async remove(t){await this.#a();try{await this.#s.unlink(this.#r(t))}catch{}}async listFiles(){await this.#a();try{return await this.#s.readdir(this.#n)}catch{return[]}}async clearAll(){await this.#a();try{const t=await this.#s.readdir(this.#n);for(const e of t)await this.#s.unlink(this.#i.join(this.#n,e))}catch{}}async getFileSize(t){await this.#a();try{return(await this.#s.stat(this.#r(t))).size}catch{return 0}}}const s="search_meta.json",i="deleted_ids.bin",n="added_ids.bin";class a{#o;#h={wordSegments:[],charSegments:[]};#c=new Set;#d=new Set;constructor(t){this.#o=t}async load(){const t=await this.#o.read(s);if(t){const e=(new TextDecoder).decode(t);this.#h=JSON.parse(e)}else this.#h={wordSegments:[],charSegments:[]};const e=await this.#o.read(i);if(e){const t=new DataView(e);let s=0;const i=e.byteLength;for(;s<i&&!(s+4>i);){const e=t.getUint32(s,!0);this.#c.add(e),s+=4,s<i&&30===t.getUint8(s)&&(s+=1)}}const a=await this.#o.read(n);if(a){const t=new DataView(a);let e=0;const s=a.byteLength;for(;e<s&&!(e+4>s);){const i=t.getUint32(e,!0);this.#d.add(i),e+=4,e<s&&30===t.getUint8(e)&&(e+=1)}}}async save(){const t=JSON.stringify(this.#h);if(await this.#o.write(s,(new TextEncoder).encode(t).buffer),0===this.#c.size)await this.#o.remove(i);else{const t=4*this.#c.size+this.#c.size,e=new ArrayBuffer(t),s=new DataView(e);let n=0;for(const t of this.#c)s.setUint32(n,t,!0),n+=4,s.setUint8(n,30),n+=1;await this.#o.write(i,e)}if(0===this.#d.size)await this.#o.remove(n);else{const t=4*this.#d.size+this.#d.size,e=new ArrayBuffer(t),s=new DataView(e);let i=0;for(const t of this.#d)s.setUint32(i,t,!0),i+=4,s.setUint8(i,30),i+=1;await this.#o.write(n,e)}}getSegments(t){return"word"===t?this.#h.wordSegments:this.#h.charSegments}getDeletedIds(){return this.#c}addDeletedId(t){this.#c.add(t)}isDeleted(t){return this.#c.has(t)}addAddedId(t){this.#d.add(t)}removeAddedId(t){this.#d.delete(t)}isAdded(t){return this.#d.has(t)}getAddedIds(){return this.#d}getLastSegmentInfo(t){const e=this.getSegments(t);return 0===e.length?null:e[e.length-1]}updateSegment(t,e,s,i,n,a){const r="word"===t?this.#h.wordSegments:this.#h.charSegments;if(a)r.push({filename:e,start:s,end:i,tokenCount:n});else{const t=r[r.length-1];t&&t.filename===e&&(t.end=i,t.tokenCount=n)}}reset(){this.#h={wordSegments:[],charSegments:[]},this.#c.clear(),this.#d.clear()}}class r{static SEPARATOR=30;#o;constructor(t){this.#o=t}async appendBatch(t,e){if(0===e.length)return await this.#o.getFileSize(t);const s=new TextEncoder;let i=0;for(const t of e){i+=8;for(const e of t.tokens){i+=2+Math.min(s.encode(e).byteLength,65535)}i+=1}const n=new Uint8Array(i);let a=0;for(const t of e){const e=[];for(const i of t.tokens){const t=s.encode(i),n=t.byteLength>65535?t.slice(0,65535):t;e.push(n)}const i=new DataView(n.buffer,a);i.setUint32(0,t.id,!0),i.setUint32(4,e.length,!0),a+=8;for(const t of e)new DataView(n.buffer,a).setUint16(0,t.byteLength,!0),a+=2,n.set(t,a),a+=t.byteLength;n[a++]=r.SEPARATOR}return await this.#o.append(t,n.buffer),await this.#o.getFileSize(t)}async readRange(t,e,s){const i=await this.#o.readRange(t,e,s);if(!i||0===i.byteLength)return[];const n=new DataView(i),a=new Uint8Array(i),o=new TextDecoder,h=[];let c=0;const d=i.byteLength;for(;c<d&&!(c+8>d);){const t=n.getUint32(c,!0);c+=4;const e=n.getUint32(c,!0);c+=4;const s=[];for(let t=0;t<e&&!(c+2>d);t++){const t=n.getUint16(c,!0);if(c+=2,c+t>d)break;const e=new Uint8Array(i,c,t);s.push(o.decode(e)),c+=t}c<d&&a[c]===r.SEPARATOR&&(c+=1),h.push({id:t,tokens:s})}return h}async getCurrentSize(t){return await this.#o.getFileSize(t)}}class o{#g;#o;#l=null;#f=null;static hash(t){let e=5381;for(let s=0;s<t.length;s++)e=(e<<5)+e^t.charCodeAt(s);return e>>>0}constructor(t,e){this.#g=t,this.#o=e}async loadIndex(){return!!this.#l||(this.#l=await this.#o.read(this.#g),!!this.#l&&(this.#f=new DataView(this.#l),!0))}async buildAndSave(t){const e=new Map;for(const s of t){const t=new Map;for(const i of s.tokens)if(!t.has(i)){t.set(i,!0);const n=o.hash(i);e.has(n)||e.set(n,[]),e.get(n).push(s.id)}}const s=Array.from(e.keys()).sort((t,e)=>t-e);let i=0;const n=new Array(s.length);for(let t=0;t<s.length;t++){const a=s[t],r=e.get(a);n[t]=r,i+=r.length}const a=12*s.length,r=new ArrayBuffer(8+a+4*i),h=new DataView(r);h.setUint32(0,1229866072),h.setUint32(4,s.length);let c=8,d=8+a;for(let t=0;t<s.length;t++){const e=s[t],i=n[t];h.setUint32(c,e),h.setUint32(c+4,d),h.setUint32(c+8,i.length),c+=12;for(let t=0;t<i.length;t++)h.setUint32(d,i[t],!0),d+=4}await this.#o.write(this.#g,r),this.#l=r,this.#f=h}search(t){if(!this.#f||!this.#l)return[];const e=o.hash(t);let s=0,i=this.#f.getUint32(4)-1;for(;s<=i;){const t=s+i>>>1,n=8+12*t,a=this.#f.getUint32(n);if(a<e)s=t+1;else{if(!(a>e)){const t=this.#f.getUint32(n+4),e=this.#f.getUint32(n+8),s=[];for(let i=0;i<e;i++)s.push(this.#f.getUint32(t+4*i,!0));return s}i=t-1}}return[]}}const h="word_cache.bin",c="char_cache.bin";class d{#o;#h;#w;#u;#m=!1;#y;#p=!1;#S={word:0,char:0};constructor(s){if(!s.baseDir)throw new Error("SearchEngine requires 'baseDir' in config.");if(this.#y={wordSegmentTokenThreshold:1e5,charSegmentTokenThreshold:5e5,minWordTokenSave:0,minCharTokenSave:0,...s},(this.#y.minWordTokenSave||0)>=(this.#y.wordSegmentTokenThreshold||1e5))throw new Error("minWordTokenSave must be less than wordSegmentTokenThreshold");if((this.#y.minCharTokenSave||0)>=(this.#y.charSegmentTokenThreshold||5e5))throw new Error("minCharTokenSave must be less than charSegmentTokenThreshold");let i=null;if(this.#y.storage&&("object"==typeof this.#y.storage?i=this.#y.storage:"browser"===this.#y.storage?i=new t(this.#y.baseDir):"node"===this.#y.storage&&(i=new e(this.#y.baseDir))),!i){const s=typeof navigator<"u"&&navigator?.storage?.getDirectory instanceof Function,n=typeof process<"u"&&null!=process.versions&&null!=process.versions.node;s?i=new t(this.#y.baseDir):n&&(i=new e(this.#y.baseDir))}if(!i)throw new Error('Storage initialization failed. Please configure "storage" explicitly or ensure you are in a supported environment (Browser/Node).');this.#o=i,this.#h=new a(this.#o),this.#w=new r(this.#o),this.#u=new Map}async init(){if(this.#m)return;await this.#h.load();const t=[...this.#h.getSegments("word"),...this.#h.getSegments("char")];for(const e of t)this.#u.has(e.filename)||this.#u.set(e.filename,new o(e.filename,this.#o)),await this.#u.get(e.filename).loadIndex();this.#m=!0}startBatch(){this.#p=!0,this.#S={word:0,char:0}}async endBatch(){this.#p=!1,this.#S.word>0&&await this.#D("word",this.#S.word),this.#S.char>0&&await this.#D("char",this.#S.char),this.#S={word:0,char:0},await this.#h.save()}#k(t){if(typeof Intl<"u"&&Intl.Segmenter){const e=new Intl.Segmenter([],{granularity:"word"});return Array.from(e.segment(t)).filter(t=>t.isWordLike).map(t=>t.segment.toLowerCase())}return t.toLowerCase().split(/[^a-z0-9\u4e00-\u9fa5]+/g).filter(t=>t.length>0)}#b(t){return this.#y.indexingTokenizer?this.#y.indexingTokenizer(t):this.#k(t)}#T(t){return this.#y.searchTokenizer?this.#y.searchTokenizer(t):this.#y.indexingTokenizer?this.#y.indexingTokenizer(t):this.#k(t)}async addDocument(t){return this.addDocuments([t])}async addDocuments(t){if(this.#m||await this.init(),0===t.length)return;const e=this.#h.getDeletedIds(),s=[],i=[];for(const n of t){if(e.has(n.id))throw new Error(`Document ID ${n.id} has been deleted and cannot be re-added.`);if(this.#h.isAdded(n.id))throw new Error(`Document ID ${n.id} already exists.`);const t=this.#b(n.text),a=[],r=[];for(const e of t)e.length>1?a.push(e):1===e.length&&r.push(e);a.length>0&&s.push({id:n.id,tokens:a}),r.length>0&&i.push({id:n.id,tokens:r})}let n=0,a=0;if(s.length>0){await this.#w.appendBatch(h,s);for(const t of s)n+=t.tokens.length}if(i.length>0){await this.#w.appendBatch(c,i);for(const t of i)a+=t.tokens.length}for(const e of t)this.#h.addAddedId(e.id);this.#p?(this.#S.word+=n,this.#S.char+=a):(n>0&&await this.#D("word",n),a>0&&await this.#D("char",a),await this.#h.save())}async#D(t,e){const s="word"===t?h:c,i=await this.#w.getCurrentSize(s),n="word"===t?this.#y.wordSegmentTokenThreshold||1e5:this.#y.charSegmentTokenThreshold||5e5,a="word"===t?this.#y.minWordTokenSave||0:this.#y.minCharTokenSave||0,r=this.#h.getLastSegmentInfo(t);let d,g,l,f;const w=()=>{const e=this.#h.getSegments(t).length+1;return`${t}_seg_${e}.bin`};if(r){const t=r.tokenCount;t>=n||t+e>=n?(d=w(),l=!0,g=r.end,f=e):(d=r.filename,l=!1,g=r.start,f=t+e)}else d=w(),l=!0,g=0,f=e;if(f<a)return void this.#h.updateSegment(t,d,g,i,f,l);const u=await this.#w.readRange(s,g,i);let m=this.#u.get(d);m||(m=new o(d,this.#o),this.#u.set(d,m)),await m.buildAndSave(u),this.#h.updateSegment(t,d,g,i,f,l)}async search(t,e){this.#m||await this.init();const s=this.#T(t),i=s.filter(t=>t.length>1),n=s.filter(t=>1===t.length),a=this.#h.getDeletedIds(),r=new Map,h=new Map,c=t=>{const e=this.#h.getSegments(t);for(const t of e){const e=t.filename;!this.#u.has(e)&&!h.has(e)&&h.set(e,new o(e,this.#o))}};c("word"),c("char"),await Promise.all(Array.from(h.entries()).map(([t,e])=>e.loadIndex().then(s=>{s&&this.#u.set(t,e)})));const d=async(t,e)=>{if(0===e.length)return;const s=this.#h.getSegments(t);for(const t of s){const s=t.filename,i=this.#u.get(s);if(i)for(const t of e){const e=i.search(t),s=1+.1*t.length;for(const i of e)if(!a.has(i))if(r.has(i)){const e=r.get(i);e.score+=s,e.tokens.add(t)}else r.set(i,{score:0,tokens:new Set([t])})}}};await d("word",i),await d("char",n);const g=[];return r.forEach((t,e)=>{g.push({id:e,score:t.score,tokens:Array.from(t.tokens)})}),g.sort((t,e)=>e.score-t.score),"number"==typeof e&&e>0?g.slice(0,e):g}async removeDocument(t){this.#m||await this.init(),this.#h.addDeletedId(t),this.#h.removeAddedId(t),await this.#h.save()}async clearAll(){await this.#o.clearAll(),this.#u.clear(),this.#h.reset(),this.#m=!1,this.#p=!1,this.#S={word:0,char:0}}async getStatus(){return this.#m||await this.init(),{wordSegments:this.#h.getSegments("word").length,charSegments:this.#h.getSegments("char").length,deleted:this.#h.getDeletedIds().size,wordCacheSize:await this.#w.getCurrentSize(h),charCacheSize:await this.#w.getCurrentSize(c),inBatch:this.#p}}}exports.BrowserStorage=t,exports.NodeStorage=e,exports.SearchEngine=d,exports.SimpleSearch=class{static#I=null;static#v={baseDir:"simple_search_data",wordSegmentTokenThreshold:1e5,minWordTokenSave:0};static configure(t){const e={...this.#v,...t};this.#I=new d(e)}static#z(){return this.#I||(this.#I=new d(this.#v)),this.#I}static async startBatch(){this.#z().startBatch()}static async endBatch(){return this.#z().endBatch()}static async addDocument(t){return this.#z().addDocument(t)}static async addDocuments(t){return this.#z().addDocuments(t)}static async search(t,e){return this.#z().search(t,e)}static async removeDocument(t){return this.#z().removeDocument(t)}static async clearAll(){return this.#z().clearAll()}static async getStatus(){return this.#z().getStatus()}};

package/lib/index.d.ts

@@ -0,0 +1,199 @@
+/**
+ * 核心类型定义
+ */
+interface IDocument {
+    id: number;
+    text: string;
+}
+interface IResult {
+    id: number;
+    score: number;
+    tokens: string[];
+}
+interface ISegmentMeta {
+    filename: string;
+    start: number;
+    end: number;
+    tokenCount: number;
+}
+interface IIndexMeta {
+    wordSegments: ISegmentMeta[];
+    charSegments: ISegmentMeta[];
+}
+interface ITokenizedDoc {
+    id: number;
+    tokens: string[];
+}
+type IndexType = 'word' | 'char';
+/**
+ * 存储层接口 (外部化)
+ */
+interface IStorage {
+    write(filename: string, data: ArrayBuffer): Promise<void>;
+    append(filename: string, data: ArrayBuffer): Promise<void>;
+    read(filename: string): Promise<ArrayBuffer | null>;
+    readRange(filename: string, start: number, end: number): Promise<ArrayBuffer | null>;
+    remove(filename: string): Promise<void>;
+    listFiles(): Promise<string[]>;
+    clearAll(): Promise<void>;
+    getFileSize(filename: string): Promise<number>;
+}
+/**
+ * 核心搜索引擎配置
+ */
+interface ISearchEngineConfig {
+    /** * 数据存储的基础目录 (必填)
+     * 用于区分不同的搜索引擎实例
+     */
+    baseDir: string;
+    /**
+     * 存储实现配置 (可选)
+     * - 'browser': 强制使用 OPFS (BrowserStorage)
+     * - 'node': 强制使用 Node.js fs (NodeStorage)
+     * - IStorage: 传入自定义的存储实例
+     * - undefined: 自动检测环境
+     */
+    storage?: 'browser' | 'node' | IStorage;
+    /**
+     * 索引时使用的分词器 (算法核心配置)
+     * - 作用: 将文档文本转换为索引用的token序列
+     * - 算法: 自定义实现的分词逻辑，需满足返回字符串数组的格式要求
+     * - 建议: 针对不同语言(中文/英文/日文等)使用专门的分词实现
+     * - 影响: 直接决定索引的粒度和搜索的准确性
+     */
+    indexingTokenizer?: (text: string) => string[];
+    /**
+     * 搜索时使用的分词器 (算法核心配置)
+     * - 作用: 将查询文本转换为搜索用的token序列
+     * - 算法: 自定义实现的分词逻辑，需满足返回字符串数组的格式要求
+     * - 建议: 与indexingTokenizer保持一致的分词策略以确保搜索准确性
+     * - 影响: 直接决定搜索匹配的范围和结果的相关性
+     */
+    searchTokenizer?: (text: string) => string[];
+    /**
+     * 词索引分段阈值 (Token数) - 分段算法配置
+     * - 作用: 控制词索引文件的大小，超过阈值时创建新的索引段
+     * - 算法: 基于Token数量的分段策略，当新增Token数加上已有Token数超过阈值时触发分段
+     * - 默认值: 100000
+     * - 影响: 过小会导致索引文件过多，过大可能影响搜索性能
+     */
+    wordSegmentTokenThreshold?: number;
+    /**
+     * 字索引分段阈值 (Token数) - 分段算法配置
+     * - 作用: 控制字索引文件的大小，超过阈值时创建新的索引段
+     * - 算法: 基于Token数量的分段策略，当新增Token数加上已有Token数超过阈值时触发分段
+     * - 默认值: 500000
+     * - 影响: 过小会导致索引文件过多，过大可能影响搜索性能
+     */
+    charSegmentTokenThreshold?: number;
+    /**
+     * 词索引最小保存阈值 (Token数) - 缓存算法配置
+     * - 作用: 控制词索引是否立即写入磁盘，低于阈值时只保存在内存缓存中
+     * - 算法: 基于Token数量的缓存策略，当累计Token数达到阈值时才进行持久化
+     * - 默认值: 0
+     * - 影响: 适当设置可减少磁盘IO次数，提高索引性能
+     */
+    minWordTokenSave?: number;
+    /**
+     * 字索引最小保存阈值 (Token数) - 缓存算法配置
+     * - 作用: 控制字索引是否立即写入磁盘，低于阈值时只保存在内存缓存中
+     * - 算法: 基于Token数量的缓存策略，当累计Token数达到阈值时才进行持久化
+     * - 默认值: 0
+     * - 影响: 适当设置可减少磁盘IO次数，提高索引性能
+     */
+    minCharTokenSave?: number;
+}
+
+/**
+ * 核心搜索引擎类 (多实例支持)
+ */
+declare class SearchEngine {
+    #private;
+    constructor(config: ISearchEngineConfig);
+    init(): Promise<void>;
+    /**
+     * 开启批处理
+     * 批处理期间 addDocuments 只写入缓存，不触发索引段构建
+     */
+    startBatch(): void;
+    /**
+     * 结束批处理
+     * 触发索引构建检查并保存元数据
+     */
+    endBatch(): Promise<void>;
+    addDocument(doc: IDocument): Promise<void>;
+    addDocuments(docs: IDocument[]): Promise<void>;
+    search(query: string, limit?: number): Promise<IResult[]>;
+    removeDocument(id: number): Promise<void>;
+    clearAll(): Promise<void>;
+    getStatus(): Promise<{
+        wordSegments: number;
+        charSegments: number;
+        deleted: number;
+        wordCacheSize: number;
+        charCacheSize: number;
+        inBatch: boolean;
+    }>;
+}
+
+/**
+ * 快速使用封装
+ * 提供单例模式和默认配置
+ */
+declare class SimpleSearch {
+    #private;
+    /**
+     * 配置并初始化单例
+     */
+    static configure(config: Partial<ISearchEngineConfig>): void;
+    static startBatch(): Promise<void>;
+    static endBatch(): Promise<void>;
+    static addDocument(doc: IDocument): Promise<void>;
+    static addDocuments(docs: IDocument[]): Promise<void>;
+    static search(query: string, limit?: number): Promise<IResult[]>;
+    static removeDocument(id: number): Promise<void>;
+    static clearAll(): Promise<void>;
+    static getStatus(): Promise<{
+        wordSegments: number;
+        charSegments: number;
+        deleted: number;
+        wordCacheSize: number;
+        charCacheSize: number;
+        inBatch: boolean;
+    }>;
+}
+
+/**
+ * 浏览器实现 (OPFS - 子目录隔离)
+ * 支持 Main Thread 和 Web Worker
+ */
+declare class BrowserStorage implements IStorage {
+    #private;
+    constructor(baseDir: string);
+    write(filename: string, data: ArrayBuffer): Promise<void>;
+    append(filename: string, data: ArrayBuffer): Promise<void>;
+    read(filename: string): Promise<ArrayBuffer | null>;
+    readRange(filename: string, start: number, end: number): Promise<ArrayBuffer | null>;
+    remove(filename: string): Promise<void>;
+    listFiles(): Promise<string[]>;
+    clearAll(): Promise<void>;
+    getFileSize(filename: string): Promise<number>;
+}
+/**
+ * Node.js 实现
+ */
+declare class NodeStorage implements IStorage {
+    #private;
+    constructor(baseDir: string);
+    write(filename: string, data: ArrayBuffer): Promise<void>;
+    append(filename: string, data: ArrayBuffer): Promise<void>;
+    read(filename: string): Promise<ArrayBuffer | null>;
+    readRange(filename: string, start: number, end: number): Promise<ArrayBuffer | null>;
+    remove(filename: string): Promise<void>;
+    listFiles(): Promise<string[]>;
+    clearAll(): Promise<void>;
+    getFileSize(filename: string): Promise<number>;
+}
+
+export { BrowserStorage, NodeStorage, SearchEngine, SimpleSearch };
+export type { IDocument, IIndexMeta, IResult, ISearchEngineConfig, ISegmentMeta, IStorage, ITokenizedDoc, IndexType };

package/lib/index.js

@@ -0,0 +1 @@
+class t{#t;constructor(t){this.#t=t}async#e(){return await(await navigator.storage.getDirectory()).getDirectoryHandle(this.#t,{create:!0})}async write(t,e){const s=await(await(await this.#e()).getFileHandle(t,{create:!0})).createWritable();await s.write(e),await s.close()}async append(t,e){const s=await this.#e();let i;try{i=await s.getFileHandle(t,{create:!0})}catch{i=await s.getFileHandle(t,{create:!0})}const n=await i.getFile(),a=await i.createWritable({keepExistingData:!0});await a.seek(n.size),await a.write(e),await a.close()}async read(t){const e=await this.#e();try{return await(await(await e.getFileHandle(t)).getFile()).arrayBuffer()}catch{return null}}async readRange(t,e,s){const i=await this.#e();try{return await(await(await i.getFileHandle(t)).getFile()).slice(e,s).arrayBuffer()}catch{return null}}async remove(t){const e=await this.#e();try{await e.removeEntry(t)}catch{}}async listFiles(){const t=await this.#e(),e=[];for await(const s of t.keys())e.push(s);return e}async clearAll(){const t=await this.#e();for await(const e of t.keys())await t.removeEntry(e,{recursive:!0})}async getFileSize(t){const e=await this.#e();try{return(await(await e.getFileHandle(t)).getFile()).size}catch{return 0}}}class e{#s=null;#i=null;#t;#n="";constructor(t){this.#t=t}async#a(){if(this.#s)return;const t=await import("node:fs"),e=await import("node:path");this.#s=t.promises,this.#i=e.default||e,this.#n=this.#i.join(process.cwd(),this.#t);try{await this.#s.access(this.#n)}catch{await this.#s.mkdir(this.#n,{recursive:!0})}}#r(t){return this.#i.join(this.#n,t)}async write(t,e){await this.#a(),await this.#s.writeFile(this.#r(t),Buffer.from(e))}async append(t,e){await this.#a(),await this.#s.appendFile(this.#r(t),Buffer.from(e))}async read(t){await this.#a();try{const e=await this.#s.readFile(this.#r(t));return e.buffer.slice(e.byteOffset,e.byteOffset+e.byteLength)}catch{return null}}async readRange(t,e,s){await this.#a();try{const i=await this.#s.open(this.#r(t),"r"),n=s-e,a=Buffer.alloc(n);return await i.read(a,0,n,e),await i.close(),a.buffer.slice(a.byteOffset,a.byteOffset+a.byteLength)}catch{return null}}async remove(t){await this.#a();try{await this.#s.unlink(this.#r(t))}catch{}}async listFiles(){await this.#a();try{return await this.#s.readdir(this.#n)}catch{return[]}}async clearAll(){await this.#a();try{const t=await this.#s.readdir(this.#n);for(const e of t)await this.#s.unlink(this.#i.join(this.#n,e))}catch{}}async getFileSize(t){await this.#a();try{return(await this.#s.stat(this.#r(t))).size}catch{return 0}}}const s="search_meta.json",i="deleted_ids.bin",n="added_ids.bin";class a{#o;#h={wordSegments:[],charSegments:[]};#c=new Set;#d=new Set;constructor(t){this.#o=t}async load(){const t=await this.#o.read(s);if(t){const e=(new TextDecoder).decode(t);this.#h=JSON.parse(e)}else this.#h={wordSegments:[],charSegments:[]};const e=await this.#o.read(i);if(e){const t=new DataView(e);let s=0;const i=e.byteLength;for(;s<i&&!(s+4>i);){const e=t.getUint32(s,!0);this.#c.add(e),s+=4,s<i&&30===t.getUint8(s)&&(s+=1)}}const a=await this.#o.read(n);if(a){const t=new DataView(a);let e=0;const s=a.byteLength;for(;e<s&&!(e+4>s);){const i=t.getUint32(e,!0);this.#d.add(i),e+=4,e<s&&30===t.getUint8(e)&&(e+=1)}}}async save(){const t=JSON.stringify(this.#h);if(await this.#o.write(s,(new TextEncoder).encode(t).buffer),0===this.#c.size)await this.#o.remove(i);else{const t=4*this.#c.size+this.#c.size,e=new ArrayBuffer(t),s=new DataView(e);let n=0;for(const t of this.#c)s.setUint32(n,t,!0),n+=4,s.setUint8(n,30),n+=1;await this.#o.write(i,e)}if(0===this.#d.size)await this.#o.remove(n);else{const t=4*this.#d.size+this.#d.size,e=new ArrayBuffer(t),s=new DataView(e);let i=0;for(const t of this.#d)s.setUint32(i,t,!0),i+=4,s.setUint8(i,30),i+=1;await this.#o.write(n,e)}}getSegments(t){return"word"===t?this.#h.wordSegments:this.#h.charSegments}getDeletedIds(){return this.#c}addDeletedId(t){this.#c.add(t)}isDeleted(t){return this.#c.has(t)}addAddedId(t){this.#d.add(t)}removeAddedId(t){this.#d.delete(t)}isAdded(t){return this.#d.has(t)}getAddedIds(){return this.#d}getLastSegmentInfo(t){const e=this.getSegments(t);return 0===e.length?null:e[e.length-1]}updateSegment(t,e,s,i,n,a){const r="word"===t?this.#h.wordSegments:this.#h.charSegments;if(a)r.push({filename:e,start:s,end:i,tokenCount:n});else{const t=r[r.length-1];t&&t.filename===e&&(t.end=i,t.tokenCount=n)}}reset(){this.#h={wordSegments:[],charSegments:[]},this.#c.clear(),this.#d.clear()}}class r{static SEPARATOR=30;#o;constructor(t){this.#o=t}async appendBatch(t,e){if(0===e.length)return await this.#o.getFileSize(t);const s=new TextEncoder;let i=0;for(const t of e){i+=8;for(const e of t.tokens){i+=2+Math.min(s.encode(e).byteLength,65535)}i+=1}const n=new Uint8Array(i);let a=0;for(const t of e){const e=[];for(const i of t.tokens){const t=s.encode(i),n=t.byteLength>65535?t.slice(0,65535):t;e.push(n)}const i=new DataView(n.buffer,a);i.setUint32(0,t.id,!0),i.setUint32(4,e.length,!0),a+=8;for(const t of e)new DataView(n.buffer,a).setUint16(0,t.byteLength,!0),a+=2,n.set(t,a),a+=t.byteLength;n[a++]=r.SEPARATOR}return await this.#o.append(t,n.buffer),await this.#o.getFileSize(t)}async readRange(t,e,s){const i=await this.#o.readRange(t,e,s);if(!i||0===i.byteLength)return[];const n=new DataView(i),a=new Uint8Array(i),o=new TextDecoder,h=[];let c=0;const d=i.byteLength;for(;c<d&&!(c+8>d);){const t=n.getUint32(c,!0);c+=4;const e=n.getUint32(c,!0);c+=4;const s=[];for(let t=0;t<e&&!(c+2>d);t++){const t=n.getUint16(c,!0);if(c+=2,c+t>d)break;const e=new Uint8Array(i,c,t);s.push(o.decode(e)),c+=t}c<d&&a[c]===r.SEPARATOR&&(c+=1),h.push({id:t,tokens:s})}return h}async getCurrentSize(t){return await this.#o.getFileSize(t)}}class o{#g;#o;#l=null;#f=null;static hash(t){let e=5381;for(let s=0;s<t.length;s++)e=(e<<5)+e^t.charCodeAt(s);return e>>>0}constructor(t,e){this.#g=t,this.#o=e}async loadIndex(){return!!this.#l||(this.#l=await this.#o.read(this.#g),!!this.#l&&(this.#f=new DataView(this.#l),!0))}async buildAndSave(t){const e=new Map;for(const s of t){const t=new Map;for(const i of s.tokens)if(!t.has(i)){t.set(i,!0);const n=o.hash(i);e.has(n)||e.set(n,[]),e.get(n).push(s.id)}}const s=Array.from(e.keys()).sort((t,e)=>t-e);let i=0;const n=new Array(s.length);for(let t=0;t<s.length;t++){const a=s[t],r=e.get(a);n[t]=r,i+=r.length}const a=12*s.length,r=new ArrayBuffer(8+a+4*i),h=new DataView(r);h.setUint32(0,1229866072),h.setUint32(4,s.length);let c=8,d=8+a;for(let t=0;t<s.length;t++){const e=s[t],i=n[t];h.setUint32(c,e),h.setUint32(c+4,d),h.setUint32(c+8,i.length),c+=12;for(let t=0;t<i.length;t++)h.setUint32(d,i[t],!0),d+=4}await this.#o.write(this.#g,r),this.#l=r,this.#f=h}search(t){if(!this.#f||!this.#l)return[];const e=o.hash(t);let s=0,i=this.#f.getUint32(4)-1;for(;s<=i;){const t=s+i>>>1,n=8+12*t,a=this.#f.getUint32(n);if(a<e)s=t+1;else{if(!(a>e)){const t=this.#f.getUint32(n+4),e=this.#f.getUint32(n+8),s=[];for(let i=0;i<e;i++)s.push(this.#f.getUint32(t+4*i,!0));return s}i=t-1}}return[]}}const h="word_cache.bin",c="char_cache.bin";class d{#o;#h;#w;#u;#m=!1;#y;#p=!1;#S={word:0,char:0};constructor(s){if(!s.baseDir)throw new Error("SearchEngine requires 'baseDir' in config.");if(this.#y={wordSegmentTokenThreshold:1e5,charSegmentTokenThreshold:5e5,minWordTokenSave:0,minCharTokenSave:0,...s},(this.#y.minWordTokenSave||0)>=(this.#y.wordSegmentTokenThreshold||1e5))throw new Error("minWordTokenSave must be less than wordSegmentTokenThreshold");if((this.#y.minCharTokenSave||0)>=(this.#y.charSegmentTokenThreshold||5e5))throw new Error("minCharTokenSave must be less than charSegmentTokenThreshold");let i=null;if(this.#y.storage&&("object"==typeof this.#y.storage?i=this.#y.storage:"browser"===this.#y.storage?i=new t(this.#y.baseDir):"node"===this.#y.storage&&(i=new e(this.#y.baseDir))),!i){const s=typeof navigator<"u"&&navigator?.storage?.getDirectory instanceof Function,n=typeof process<"u"&&null!=process.versions&&null!=process.versions.node;s?i=new t(this.#y.baseDir):n&&(i=new e(this.#y.baseDir))}if(!i)throw new Error('Storage initialization failed. Please configure "storage" explicitly or ensure you are in a supported environment (Browser/Node).');this.#o=i,this.#h=new a(this.#o),this.#w=new r(this.#o),this.#u=new Map}async init(){if(this.#m)return;await this.#h.load();const t=[...this.#h.getSegments("word"),...this.#h.getSegments("char")];for(const e of t)this.#u.has(e.filename)||this.#u.set(e.filename,new o(e.filename,this.#o)),await this.#u.get(e.filename).loadIndex();this.#m=!0}startBatch(){this.#p=!0,this.#S={word:0,char:0}}async endBatch(){this.#p=!1,this.#S.word>0&&await this.#D("word",this.#S.word),this.#S.char>0&&await this.#D("char",this.#S.char),this.#S={word:0,char:0},await this.#h.save()}#k(t){if(typeof Intl<"u"&&Intl.Segmenter){const e=new Intl.Segmenter([],{granularity:"word"});return Array.from(e.segment(t)).filter(t=>t.isWordLike).map(t=>t.segment.toLowerCase())}return t.toLowerCase().split(/[^a-z0-9\u4e00-\u9fa5]+/g).filter(t=>t.length>0)}#b(t){return this.#y.indexingTokenizer?this.#y.indexingTokenizer(t):this.#k(t)}#T(t){return this.#y.searchTokenizer?this.#y.searchTokenizer(t):this.#y.indexingTokenizer?this.#y.indexingTokenizer(t):this.#k(t)}async addDocument(t){return this.addDocuments([t])}async addDocuments(t){if(this.#m||await this.init(),0===t.length)return;const e=this.#h.getDeletedIds(),s=[],i=[];for(const n of t){if(e.has(n.id))throw new Error(`Document ID ${n.id} has been deleted and cannot be re-added.`);if(this.#h.isAdded(n.id))throw new Error(`Document ID ${n.id} already exists.`);const t=this.#b(n.text),a=[],r=[];for(const e of t)e.length>1?a.push(e):1===e.length&&r.push(e);a.length>0&&s.push({id:n.id,tokens:a}),r.length>0&&i.push({id:n.id,tokens:r})}let n=0,a=0;if(s.length>0){await this.#w.appendBatch(h,s);for(const t of s)n+=t.tokens.length}if(i.length>0){await this.#w.appendBatch(c,i);for(const t of i)a+=t.tokens.length}for(const e of t)this.#h.addAddedId(e.id);this.#p?(this.#S.word+=n,this.#S.char+=a):(n>0&&await this.#D("word",n),a>0&&await this.#D("char",a),await this.#h.save())}async#D(t,e){const s="word"===t?h:c,i=await this.#w.getCurrentSize(s),n="word"===t?this.#y.wordSegmentTokenThreshold||1e5:this.#y.charSegmentTokenThreshold||5e5,a="word"===t?this.#y.minWordTokenSave||0:this.#y.minCharTokenSave||0,r=this.#h.getLastSegmentInfo(t);let d,g,l,f;const w=()=>{const e=this.#h.getSegments(t).length+1;return`${t}_seg_${e}.bin`};if(r){const t=r.tokenCount;t>=n||t+e>=n?(d=w(),l=!0,g=r.end,f=e):(d=r.filename,l=!1,g=r.start,f=t+e)}else d=w(),l=!0,g=0,f=e;if(f<a)return void this.#h.updateSegment(t,d,g,i,f,l);const u=await this.#w.readRange(s,g,i);let m=this.#u.get(d);m||(m=new o(d,this.#o),this.#u.set(d,m)),await m.buildAndSave(u),this.#h.updateSegment(t,d,g,i,f,l)}async search(t,e){this.#m||await this.init();const s=this.#T(t),i=s.filter(t=>t.length>1),n=s.filter(t=>1===t.length),a=this.#h.getDeletedIds(),r=new Map,h=new Map,c=t=>{const e=this.#h.getSegments(t);for(const t of e){const e=t.filename;!this.#u.has(e)&&!h.has(e)&&h.set(e,new o(e,this.#o))}};c("word"),c("char"),await Promise.all(Array.from(h.entries()).map(([t,e])=>e.loadIndex().then(s=>{s&&this.#u.set(t,e)})));const d=async(t,e)=>{if(0===e.length)return;const s=this.#h.getSegments(t);for(const t of s){const s=t.filename,i=this.#u.get(s);if(i)for(const t of e){const e=i.search(t),s=1+.1*t.length;for(const i of e)if(!a.has(i))if(r.has(i)){const e=r.get(i);e.score+=s,e.tokens.add(t)}else r.set(i,{score:0,tokens:new Set([t])})}}};await d("word",i),await d("char",n);const g=[];return r.forEach((t,e)=>{g.push({id:e,score:t.score,tokens:Array.from(t.tokens)})}),g.sort((t,e)=>e.score-t.score),"number"==typeof e&&e>0?g.slice(0,e):g}async removeDocument(t){this.#m||await this.init(),this.#h.addDeletedId(t),this.#h.removeAddedId(t),await this.#h.save()}async clearAll(){await this.#o.clearAll(),this.#u.clear(),this.#h.reset(),this.#m=!1,this.#p=!1,this.#S={word:0,char:0}}async getStatus(){return this.#m||await this.init(),{wordSegments:this.#h.getSegments("word").length,charSegments:this.#h.getSegments("char").length,deleted:this.#h.getDeletedIds().size,wordCacheSize:await this.#w.getCurrentSize(h),charCacheSize:await this.#w.getCurrentSize(c),inBatch:this.#p}}}class g{static#I=null;static#v={baseDir:"simple_search_data",wordSegmentTokenThreshold:1e5,minWordTokenSave:0};static configure(t){const e={...this.#v,...t};this.#I=new d(e)}static#z(){return this.#I||(this.#I=new d(this.#v)),this.#I}static async startBatch(){this.#z().startBatch()}static async endBatch(){return this.#z().endBatch()}static async addDocument(t){return this.#z().addDocument(t)}static async addDocuments(t){return this.#z().addDocuments(t)}static async search(t,e){return this.#z().search(t,e)}static async removeDocument(t){return this.#z().removeDocument(t)}static async clearAll(){return this.#z().clearAll()}static async getStatus(){return this.#z().getStatus()}}export{t as BrowserStorage,e as NodeStorage,d as SearchEngine,g as SimpleSearch};

package/package.json

@@ -0,0 +1,28 @@
+{
+  "name": "gs-search",
+  "version": "0.1.0",
+  "type": "module",
+  "main": "lib/index.cjs",
+  "module": "lib/index.js",
+  "exports": {
+    ".": {
+      "import": "./lib/index.js",
+      "require": "./lib/index.cjs",
+      "types": "./lib/index.d.ts"
+    }
+  },
+  "types": "lib/index.d.ts",
+  "keywords": [
+  ],
+  "homepage": "https://github.com/grain-sand/gs-search",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/grain-sand/gs-search.git"
+  },
+  "bugs": {
+    "url": "https://github.com/grain-sand/gs-search/issues"
+  },
+  "author": "grain-sand",
+  "license": "MIT",
+  "dependencies": {}
+}