smart_rag 0.1.0 → 0.2.1

data/Hybrid_Reranking.md

@@ -0,0 +1,171 @@
+# RAGFlow 多路召回与重排序（Hybrid Search & Reranking）技术介绍
+
+在 RAGFlow 中，多路召回（Multi-path Retrieval）与重排序（Reranking）技术共同构成了其高精度检索的核心架构，旨在解决单一检索方式在复杂查询下容易出现的“搜不准、搜不全”问题。以下是 2026 年初该技术的详细解读：
+
+1. 多路召回技术：确保“广而全”
+
+RAGFlow 采用双路混合检索架构，利用不同维度的“筛子”同时在海量文档中筛选候选结果。 
+全文检索（关键词通路）：
+核心算法：通常基于 BM25 算法。
+优势：擅长精准匹配缩写、产品型号、专业术语或姓名。例如搜索“R1-750”，全文检索能精准锁定包含该特定编号的文档。
+向量检索（语义通路）：
+核心算法：利用 Embedding 模型（如 BGE 或 OpenAI 兼容模型）将文本转化为高维向量。
+优势：理解用户意图，即使问题中没有原词，也能找到意思相近的内容。例如用户问“如何理财”，系统能检索到包含“资产配置”或“储蓄方案”的片段。
+混合融合（RRF）：
+系统使用 倒数排名融合（Reciprocal Rank Fusion, RRF） 等算法将两路结果合并，初步平衡关键词匹配和语义相关性的得分。 
+
+2. 重排序技术：确保“精而准”
+
+多路召回虽然覆盖面广，但往往会混入不相关的噪音。RAGFlow 引入重排序（Reranking）阶段，对初步选出的 Top-K 候选片断进行“二次打分”。 
+级联式重排序策略：
+交叉编码器（Cross-Encoder）：这是重排序的核心模型（如 bge-reranker-v2-m3）。与简单的向量相似度不同，它会将查询（Query）和文档（Document）同时输入模型，捕捉更细微的语义匹配关系。
+上下文长度优势：截至 2026 年，其主流重排序模型已支持最高 8192 tokens 的上下文，能够处理更长的文档片段而不会丢失关键信息。
+多模型集成：
+RAGFlow 支持集成多种顶级重排序器，包括 Cohere Rerank、Jina Reranker 以及开源的 BGE 系列。2026 年甚至支持通过 vLLM 托管这些重排序模型以获得更高的推理效率。 
+
+3. 技术核心优势
+
+首条命中率提升：通过先“广搜”再“精排”，显著提高了首个召回片段的相关性，这对于 LLM 减少幻觉至关重要。
+结构化数据亲和：针对 CSV/JSON 等缺乏自然语言语义的结构化数据，多路召回中的关键词通路能补足传统向量检索的短板。
+可追溯性：重排序后的高分片段会与原文位置绑定，在 UI 界面上直接展示为高亮引用，保证了 AI 回答的“有据可查”。 
+通过这种双引擎驱动 + 深度精排的模式，RAGFlow 能够将检索准确率从初级 RAG 的约 60% 提升至 90% 以上。
+
+# 多路召回与重排序设计方案
+
+  1) Hybrid Search：多路召回 + 融合
+
+  - 入口：rag\nlp\search.py 的 Dealer.search()
+      - 文本召回：FulltextQueryer.question() 生成 MatchTextExpr（BM25/查询字符串）并扩展同义词与细粒度 token。
+          - 参考：rag\nlp\query.py
+      - 向量召回：get_vector() 生成 MatchDenseExpr，向量字段命名 q_{dim}_vec。
+          - 参考：rag\nlp\search.py
+      - 融合：FusionExpr("weighted_sum", {"weights":"0.05,0.95"})，文本/向量权重融合。
+          - 参考：rag\nlp\search.py
+  - 数据库层执行：
+      - OpenSearch：使用 query_string + knn，并用 FusionExpr 的权重调整 boost。
+          - 参考：rag\utils\opensearch_conn.py
+      - OB/Infinity 等：由连接器实现融合查询与归一化，Infinity 会归一化两路得分，后续无需再 rerank。
+          - 参考：rag\nlp\search.py 中 settings.DOC_ENGINE_INFINITY 分支
+
+  2) Rerank：两种路径
+
+  - 内置 rerank（无外部模型）：
+      - Dealer.rerank() 调用 FulltextQueryer.hybrid_similarity()
+      - 得分 = token_similarity * tkweight + vector_similarity * vtweight + rank_feature
+      - token_similarity 对内容 tokens + 标题/重要关键词加权（title2，important5，question*6）。
+      - 参考：rag\nlp\search.py, rag\nlp\query.py
+  - 外部 rerank 模型：
+      - Dealer.rerank_by_model() 使用 reranker 输出向量得分，混合 token 相似度。
+      - rerank 模型适配统一接口：similarity(query, texts)。
+      - 参考：rag\nlp\search.py, rag\llm\rerank_model.py
+
+  3) 排序增强：rank_feature（Pagerank / 标签）
+
+  - rank_feature 引入 pagerank 与标签相关性加权。
+  - _rank_feature_scores() 对标签向量与 query 标签做相似度，叠加 pagerank。
+  - 参考：rag\nlp\search.py, rag\utils\opensearch_conn.py
+
+  4) 流程细节
+
+  - 再排序池大小：RERANK_LIMIT 固定到 64 的倍数分页，以扩大 rerank 范围。
+      - 参考：rag\nlp\search.py
+  - 失败回退：如果融合查询结果为空，降低 min_match、提高 similarity threshold 再试。
+      - 参考：rag\nlp\search.py
+
+  Ruby 复刻设计方案（详细）
+
+  A. 模块划分
+
+  - HybridSearch::QueryBuilder
+      - 构造全文检索查询 + 同义词扩展 + token 权重。
+      - 接口：build_text_query(question, min_match) → MatchTextExpr + keywords。
+      - 参考：rag\nlp\query.py
+  - HybridSearch::Embedding
+      - encode_queries(text) → vector
+      - 统一向量字段名：q_#{dim}_vec。
+  - HybridSearch::Fusion
+      - 表示融合策略：weighted_sum，保存权重。
+      - 参考：common\doc_store\doc_store_base.py
+  - HybridSearch::DocStoreAdapter
+      - search(select_fields, filters, match_exprs, order_by, limit, offset, rank_feature)
+      - 提供 OpenSearch/PG/OB/自研引擎适配。
+  - HybridSearch::Reranker
+      - rerank_by_model：外部模型返回相似度。
+      - rerank_by_hybrid：token+vector 混合。
+  - HybridSearch::Retriever
+      - orchestrator：负责 recall → rerank → filtering → pagination。
+
+  B. 核心数据结构
+
+  - MatchTextExpr, MatchDenseExpr, FusionExpr（对齐 common\doc_store\doc_store_base.py）
+  - SearchResult：total, ids, fields, query_vector, highlight, aggs
+
+  C. 召回策略（Hybrid Search）
+
+  1. 构建全文查询：
+      - 英文：词权重 + 词邻近短语（bigram boost）。
+      - 中文：分词 + 同义词 + fine-grained token。
+  2. 构建向量查询：q_{dim}_vec + topk + similarity_threshold
+  3. 组装 FusionExpr：默认 "0.05,0.95"（文本/向量）
+  4. 交给 DocStoreAdapter 执行。
+
+  D. Rerank 策略
+
+  - 如果配置 rerank_model：
+      - score = tkweight * token_similarity + vtweight * model_score + rank_feature
+  - 否则：
+      - score = tkweight * token_similarity + vtweight * vector_similarity + rank_feature
+  - tkweight = 1 - vector_similarity_weight（配置默认 0.3）
+  - rank_feature 依赖 pagerank / tag_vector（若存在）。
+
+  E. 评分细节复刻
+
+  - token_similarity：
+      - 使用 term-weight 计算 query tokens 与 doc tokens 的重合度。
+      - doc tokens= content_ltks + title_tks*2 + important_kwd*5 + question_tks*6
+  - vector_similarity：
+      - cosine similarity of query_vector vs doc_vector。
+  - rank_feature：
+      - pagerank + 标签向量相似度（可按需求保留）。
+
+  F. 分页与 rerank pool
+
+  - 先取大范围 RERANK_LIMIT（推荐 64 的倍数）
+  - rerank 后再分页
+  - 避免直接分页导致 rerank “局部最优”。
+
+  G. Ruby 伪代码
+
+  def retrieve(question, page, page_size, topk:, similarity:, vec_weight:, rerank_model: nil)
+    text_expr, keywords = QueryBuilder.build_text_query(question, min_match: 0.3)
+    dense_expr = Embedding.match_dense(question, topk: topk, similarity: similarity)
+
+    fusion = FusionExpr.new("weighted_sum", topk, weights: "0.05,0.95")
+    match_exprs = [text_expr, dense_expr, fusion]
+
+    pool = docstore.search(fields, filters, match_exprs, limit: rerank_limit, offset: page_offset)
+
+    scores = if rerank_model
+      Reranker.rerank_by_model(rerank_model, pool, question, tkweight: 1-vec_weight, vtweight: vec_weight)
+    else
+      Reranker.rerank_by_hybrid(pool, question, tkweight: 1-vec_weight, vtweight: vec_weight)
+    end
+
+    ranked = pool.sort_by { |doc| -scores[doc.id] }
+    paginate(ranked, page, page_size)
+  end
+
+  H. 配置参数建议
+
+  - vector_similarity_weight（默认 0.3）
+  - topk（召回池大小）
+  - similarity_threshold（向量召回阈值）
+  - rerank_model_id（可选）
+  - rank_feature（pagerank/tag_fea 权重）
+
+  复刻重点与注意事项
+
+  - 若底层引擎能做融合归一化（类似 Infinity），可跳过自定义 rerank。
+      - 参考：rag\nlp\search.py 对 Infinity 的分支判断。
+  - 不同引擎的融合实现差异较大（OpenSearch 用 knn + query_string + boost），建议先实现一个“逻辑融合 + 本地 rerank”的通用路径，再做引擎级融合优化。
+  - rank_feature（pagerank/tag_fea）是 RAGFlow 的额外增益项，若没有对应特征可直接忽略或留接口。

data/README.en.md

@@ -2,21 +2,65 @@
 
 [中文 README](README.md)
 
-SmartRAG is a Ruby-based hybrid RAG library that combines vector retrieval, full-text search, and topic/tag organization for document intelligence workflows.
+SmartRAG is a Ruby-based hybrid Retrieval-Augmented Generation (RAG) library that combines vector retrieval, full-text search, LLM-powered Q&A summarization, and topic/tag management — covering the full pipeline from document ingestion to intelligent question answering.
 
 ## Overview
 
-- Hybrid retrieval: vector + full-text + weighted fusion
-- Document ingestion from local files and URLs
-- Topic and tag management APIs
-- Search logs and system statistics
-- Runnable example scripts for quick onboarding
+SmartRAG covers the complete RAG lifecycle:
+
+- **Document Processing**: local files and URL import with automatic format conversion (PDF/DOCX/HTML → Markdown)
+- **Smart Chunking**: Markdown heading-based splitting + adaptive size splitting + structure detection
+- **Vector Embedding**: text embeddings via Ollama, stored in PostgreSQL pgvector
+- **Full-Text Indexing**: PostgreSQL tsvector search, Chinese segmented with pg_jieba
+- **Hybrid Retrieval**: vector semantic search + full-text keyword search → RRF fusion → reranking
+- **LLM Q&A**: structured answers from search results (supports EN/ZH/JA/Traditional Chinese)
+- **Topics & Tags**: LLM-generated tags, topic organization and association recommendations
+- **Operations Tooling**: index rebuild, deduplication, backfill, search logs, system statistics
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────┐
+│            Public API                        │
+│   SmartRAG::SmartRAG  (lib/smart_rag.rb)     │
+├─────────────────────────────────────────────┤
+│        Retrieve Structured Retrieval         │
+│   SmartRAG::Retrieve  (retrieve.rb)          │
+├──────────────┬──────────────┬────────────────┤
+│  Core Layer  │ Services Layer│ Chunking Layer │
+│              │               │                │
+│ Query        │ Hybrid        │ Markdown       │
+│ Processor    │ Search Svc    │ Chunker        │
+│              │               │                │
+│ Document     │ Embedding     │ Smart          │
+│ Processor    │ Service       │ Chunking       │
+│              │               │                │
+│ Embedding    │ Fulltext      │                │
+│ (Core)       │ Search Svc    │                │
+│              │               │                │
+│ Fulltext     │ Tag           │                │
+│ Manager      │ Service       │                │
+│              │               │                │
+│              │ Summarization │                │
+│              │ Service       │                │
+├──────────────┴──────────────┴────────────────┤
+│              Models Layer (Sequel ORM)        │
+│ SourceDocument / SourceSection / Embedding    │
+│ Tag / ResearchTopic / SearchLog / SectionFts  │
+├─────────────────────────────────────────────┤
+│         Config Layer (YAML + ERB)             │
+│   smart_rag.yml / database.yml / llm_config   │
+├─────────────────────────────────────────────┤
+│         Workers Layer (SmartPrompt)           │
+│   get_embedding / analyze_content             │
+└─────────────────────────────────────────────┘
+```
 
 ## Default Model Setup
 
 Current defaults use local Ollama-compatible endpoints:
 
-- Embedding model: `qwen3-embedding`
+- Embedding model: `qwen3-embedding` (1024 dimensions)
 - Text LLM model: `qwen3`
 - Embedding endpoint: `http://localhost:11434/v1/embeddings`
 - LLM endpoint: `http://localhost:11434/v1/chat/completions`
@@ -66,49 +110,397 @@ ruby examples/01_quick_start.rb
 ruby examples/03_search_operations.rb
 ```
 
-## Minimal Usage
+## API Reference
+
+### Initialization
 
 ```ruby
 require "smart_rag"
 
-config = SmartRAG::Config.load("config/smart_rag.yml")
-client = SmartRAG::SmartRAG.new(config)
+# From a config file
+client = SmartRAG::SmartRAG.new("config/smart_rag.yml")
+
+# Or from a Hash
+client = SmartRAG::SmartRAG.new({
+  database: {
+    adapter: "postgresql",
+    host: "localhost",
+    database: "smart_rag_development",
+    user: "rag_user",
+    password: "your_password"
+  }
+})
+```
+
+### Knowledge Base Management
+
+| Method | Description | Returns |
+|---|---|---|
+| `add_document(path, options)` | Import a document (local file or URL) | `{ document_id:, section_count:, status: }` |
+| `remove_document(id)` | Delete a document with all sections/embeddings | `{ success:, deleted_sections:, deleted_embeddings: }` |
+| `get_document(id)` | Get document details | `{ id:, title:, description:, section_count:, metadata: }` |
+| `list_documents(options)` | Paginated listing with title search | `{ documents:, total_count:, page:, per_page:, total_pages: }` |
+
+```ruby
+# Import and auto-generate embeddings
+client.add_document("docs/report.md", generate_embeddings: true)
 
-client.add_document("test/python_basics.md", generate_embeddings: true)
-results = client.search("What is machine learning?", search_type: "hybrid", limit: 5)
+# Import from URL
+client.add_document("https://example.com/article.pdf")
 
-puts results[:results].map { |r| r[:section_title] }
+# Paginated listing
+client.list_documents(page: 1, per_page: 10, search: "Python")
 ```
 
-## Development Commands
+### Search (Core)
+
+SmartRAG provides three search modes: **hybrid** (default), **vector**, and **fulltext**.
+
+```ruby
+# Unified search entry point
+results = client.search("What is machine learning?",
+  search_type: "hybrid",    # hybrid | vector | fulltext
+  limit: 5,
+  language: :en,             # :en | :zh_cn | :ja | auto-detect
+  alpha: 0.7,                # vector weight (0.0-1.0), hybrid only
+  include_content: true,
+  include_metadata: true,
+  generate_tags: false,      # use LLM to generate tags from query
+  document_ids: [1, 2],      # restrict to specific documents
+  tags: ["AI", "ML"]         # tag filtering/boosting
+)
+
+# Individual search modes
+client.vector_search("neural network architectures", limit: 5)
+client.fulltext_search('"deep reinforcement learning"', limit: 5)
+client.hybrid_search("AI applications", language: :en)
+```
+
+#### Hybrid Search Pipeline
+
+```
+User Query
+    ↓
+① Language Detection (EN/ZH/JA)
+    ↓
+② Optional: LLM query tag generation
+    ↓
+③ Query vector embedding
+    ↓
+④ Parallel: vector search + full-text search
+    ↓
+⑤ RRF Fusion (Reciprocal Rank Fusion, k=60)
+    ↓
+⑥ Reranking (rerank_limit=64)
+    ↓
+⑦ Domain boosting + category diversity
+    ↓
+⑧ Return results
+```
+
+**RRF weights**: vector 0.6 / fulltext 0.4 (adjustable in `config/fulltext_search.yml`).
+
+#### Multilingual Support
+
+| Language | FTS Configuration | Tokenizer |
+|---|---|---|
+| English | `pg_catalog.english` | stemming |
+| Chinese | `jieba` | pg_jieba segmentation |
+| Japanese | `pg_catalog.simple` | basic |
+| Korean | `pg_catalog.simple` | basic |
+
+### Structured Retrieval (SmartBrain Integration)
+
+```ruby
+plan = {
+  queries: [
+    { text: "machine learning basics", mode: "semantic", weight: 1.0 },
+    { text: "deep neural networks", mode: "keyword", weight: 0.8 }
+  ],
+  budget: {
+    candidate_k: 200,
+    per_mode_k: { semantic: 30, keyword: 20 }
+  },
+  ranking: {
+    rerank: { enabled: true }
+  }
+}
+
+evidence_pack = client.retrieve(plan: plan)
+# Returns EvidencePack: { evidences:, stats:, explain:, warnings: }
+```
+
+### Tag Management
+
+```ruby
+# LLM auto-generate tags
+result = client.generate_tags("This is a text about deep learning and neural networks...",
+  max_tags: 10
+)
+# => { content_tags: ["deep learning", "neural networks"], category_tags: ["AI"] }
+
+# Paginated tag listing
+client.list_tags(page: 1, per_page: 20, search: "AI")
+```
+
+### Topic Management
+
+```ruby
+# Create topic
+client.create_topic("AI Research",
+  description: "Artificial intelligence research topics",
+  tags: ["AI", "Machine Learning"],
+  document_ids: [1, 3]
+)
+
+# Query topics
+client.get_topic(1)
+client.list_topics(page: 1, per_page: 20, search: "AI")
+
+# Update topic
+client.update_topic(1, title: "Artificial Intelligence Research", tags: ["AI", "DL"])
+
+# Delete topic
+client.delete_topic(1)
+
+# Document-topic association
+client.add_document_to_topic(topic_id: 1, document_id: 5)
+client.remove_document_from_topic(topic_id: 1, document_id: 5)
+
+# Topic recommendations (based on tag co-occurrence)
+client.get_topic_recommendations(1, limit: 5)
+```
+
+### System Operations
+
+| Method | Description |
+|---|---|
+| `statistics` | System stats (documents/sections/topics/tags/embeddings count) |
+| `search_logs(limit:, search_type:)` | Query search history |
+| `rebuild_fts(document_id)` | Rebuild full-text indexes (omit for all) |
+| `rebuild_embeddings(document_id)` | Rebuild vector embeddings (omit for all) |
+| `reindex(document_id)` | Rebuild both FTS + embeddings |
+| `dedupe_by_content_hash` | Deduplicate documents by content hash |
+| `backfill_source_fields(dry_run:)` | Backfill source_uri/source_type/content_hash fields |
+| `prepare_release_indexes(dry_run:)` | Pre-release pipeline: backfill → dedupe → reindex |
 
-- `bundle exec rspec`: run RSpec tests
-- `ruby test/test_rag.rb`: run E2E script
-- `bundle exec rake db:reset`: recreate database
-- `gem build smart_rag.gemspec`: build gem package
+```ruby
+# System statistics
+stats = client.statistics
+# => { document_count:, section_count:, topic_count:, tag_count:, embedding_count: }
+
+# Search logs
+client.search_logs(limit: 50, search_type: "hybrid")
+
+# Dry run release prep
+client.prepare_release_indexes(dry_run: true)
+
+# Execute release prep
+client.prepare_release_indexes
+```
+
+## Document Processing Pipeline
+
+```
+URL / file path
+    ↓
+① Download (with 301/302 redirect support)
+    ↓
+② Extract metadata (size, type, timestamp)
+    ↓
+③ Format conversion (Markitdown: PDF/DOCX/HTML → Markdown)
+    ↓
+④ Create SourceDocument record
+    ↓
+⑤ Smart chunking (heading-based → size-based fallback)
+    ↓
+⑥ Store SourceSection records
+    ↓
+⑦ Optional: generate Embedding + Tag
+    ↓
+⑧ Mark document status as completed
+```
+
+Supported input formats: `.md` / `.txt` / `.pdf` / `.docx` / `.html`
+
+### Chunking Strategies
+
+- **MarkdownChunker** (default): split by H1-H3 headings, oversized chunks split further by character count
+- **SmartChunking** (advanced): structure detection + document type awareness (laws/books/papers/manuals), token-based merging
+
+Configuration (`config/smart_rag.yml`):
+
+```yaml
+chunking:
+  max_chars: 4000        # max characters per chunk
+  overlap: 100           # character overlap between chunks
+  split_by_headers: true # split by markdown headers first
+  min_chunk_size: 100    # discard chunks smaller than this
+```
+
+## Data Model
+
+| Model | Table | Purpose |
+|---|---|---|
+| `SourceDocument` | `source_documents` | Document metadata (title, author, source type, state) |
+| `SourceSection` | `source_sections` | Document sections (title, number, content, language) |
+| `Embedding` | `embeddings` | pgvector vector storage |
+| `Tag` | `tags` | Tags (supports hierarchy via parent_id) |
+| `SectionTag` | `section_tags` | Many-to-many: section ↔ tag |
+| `ResearchTopic` | `research_topics` | Research topics |
+| `ResearchTopicSection` | `research_topic_sections` | Topic ↔ section association |
+| `ResearchTopicTag` | `research_topic_tags` | Topic ↔ tag association |
+| `SearchLog` | `search_logs` | Search records (query, duration, result count) |
+| `SectionFts` | `section_fts` | Full-text search materialized view |
+
+## Dependencies
+
+- **Ruby** >= 2.7
+- **PostgreSQL** + `pgvector` extension + `pg_jieba` extension
+- **Sequel** ORM
+- **SmartPrompt** gem (LLM abstraction layer)
+- **Nokogiri** / **Markitdown** (document format conversion)
+
+## Configuration Reference
+
+Main configuration files:
+
+| File | Purpose |
+|---|---|
+| `config/smart_rag.yml` | Main config (database, embedding, search, chunking, LLM, logging) |
+| `config/database.yml` | Multi-environment database config |
+| `config/llm_config.yml` | LLM adapter config (Ollama / SiliconFlow etc.) |
+| `config/fulltext_search.yml` | Full-text search details (languages, indexes, performance) |
+
+Key environment variables:
+
+| Variable | Default | Description |
+|---|---|---|
+| `SMARTRAG_DB_NAME` | `smart_rag_development` | Database name |
+| `SMARTRAG_DB_USER` | `rag_user` | Database user |
+| `SMARTRAG_DB_PASSWORD` | - | Database password |
+| `EMBEDDING_MODEL` | `qwen3-embedding` | Embedding model |
+| `EMBEDDING_DIMENSIONS` | `1024` | Vector dimensions |
+| `LLM_MODEL` | `qwen3` | LLM model |
+| `DEFAULT_LANGUAGE` | `en` | Default language |
+| `ENABLE_JIEBA` | `true` | Enable Chinese segmentation |
 
 ## Project Structure
 
 ```text
 lib/
-  smart_rag.rb                 # Main API entry
-  smart_rag/core/              # Core processing logic
-  smart_rag/services/          # Search/tag/embedding services
-config/                        # Runtime config files
-db/                            # Migrations and seed SQL
-examples/                      # Example programs
-test/                          # Manual/E2E scripts + sample docs
-spec/                          # RSpec tests
+  smart_rag.rb                     # Main API entry (SmartRAG::SmartRAG class)
+  smart_rag/config.rb              # Config loading (YAML + ERB)
+  smart_rag/version.rb             # Version
+  smart_rag/errors.rb              # Custom error classes
+  smart_rag/retrieve.rb            # Structured retrieval (RetrievalPlan → EvidencePack)
+  smart_rag/models.rb              # Model loading and connection management
+  smart_rag/models/                # Sequel ORM models (10 tables)
+  smart_rag/core/                  # Core processing logic
+    query_processor.rb             # Query processor (search + Q&A)
+    document_processor.rb          # Document processor (import + chunk + store)
+    embedding.rb                   # Low-level embedding operations
+    fulltext_manager.rb            # Full-text index management
+    markitdown_bridge.rb           # Document format conversion bridge
+  smart_rag/services/              # Service layer
+    embedding_service.rb           # Embedding service (CRUD + batch)
+    vector_search_service.rb       # Vector search service
+    fulltext_search_service.rb     # Full-text search service
+    hybrid_search_service.rb       # Hybrid search service (RRF + reranking)
+    summarization_service.rb       # LLM Q&A summarization service
+    tag_service.rb                 # Tag generation and management service
+  smart_rag/chunker/               # Chunkers
+    markdown_chunker.rb            # Markdown heading-based chunker
+  smart_rag/smart_chunking/        # Advanced smart chunking
+    pipeline.rb / parser.rb        # Structure detection + type-aware chunking
+    merger.rb / tokenizer.rb       # Token merging strategies
+  smart_rag/parsers/               # Parsers
+    query_parser.rb                # Query parser
+config/                            # Runtime configuration
+db/                                # Database migrations and seed SQL
+examples/                          # Example scripts (6 scenarios)
+test/                              # E2E test scripts + sample documents
+spec/                              # RSpec tests
+workers/                           # SmartPrompt worker definitions
+```
+
+## Development Commands
+
+```bash
+# Run tests
+bundle exec rspec                      # RSpec unit/integration tests
+SMARTRAG_LIVE_SPECS=1 bundle exec rspec spec/documentation # Live-model documentation examples
+ruby test/test_rag.rb                  # E2E test script
+
+# Database operations
+bundle exec rake db:create             # Create database
+bundle exec rake db:migrate            # Run migrations
+bundle exec rake db:seed               # Seed data
+bundle exec rake db:reset              # Recreate database
+
+# Operations
+bundle exec rake db:backfill_source_fields  # Backfill fields
+bundle exec rake db:prepare_release         # Pre-release pipeline
+
+# Build
+gem build smart_rag.gemspec            # Build gem package
+
+# Import test documents
+ruby test/import_doc.rb import
+
+# Rebuild embeddings
+ruby test/reembed_all.rb
+```
+
+## Media Storage, Tenant Isolation, and Idempotency
+
+Run database migrations through `017_add_media_job_request_fingerprint` before deploying the current asynchronous media queue. Migrations 015 and 016 add leases, object references, authenticated document ownership, and explicit staging-object protection. Migration 017 backfills and requires a canonical SHA-256 fingerprint for every queued request.
+
+For asynchronous `POST /v1/media` requests, send `Idempotency-Key`. Repeating the same operation, source, and canonicalized options for the same authenticated principal returns the original job with `deduplicated: true`. Reusing the key with a different payload returns HTTP `409` and `code: "idempotency_conflict"`. Different principals may use the same key independently.
+
+Configure AWS S3 or MinIO with the `MEDIA_CONTENT_STORE_*` and `MEDIA_S3_*` variables documented in `.env.example`. MinIO normally requires `MEDIA_S3_FORCE_PATH_STYLE=true`. The `aws-sdk-s3` runtime dependency is loaded only when the S3 provider is selected. Retained queued, processing, and failed jobs protect their staging objects from garbage collection.
+
+When HTTP authentication is enabled, every endpoint except `/healthz` requires a Bearer token. Retrieval is scoped by the authenticated principal at both boundaries: owned document IDs are pushed into search, and returned candidates are rechecked against PostgreSQL before evidence is emitted. Job reads and mutations are scoped the same way.
+
+The real MinIO suite is opt-in because it writes and deletes actual objects:
+
+```bash
+SMARTRAG_MINIO_E2E=1 \
+SMARTRAG_MINIO_ENDPOINT=http://127.0.0.1:19000 \
+SMARTRAG_MINIO_BUCKET=smart-rag-e2e \
+SMARTRAG_MINIO_ACCESS_KEY=smart-rag-e2e \
+SMARTRAG_MINIO_SECRET_KEY=smart-rag-e2e-secret \
+bundle exec rspec spec/integration/minio_content_store_spec.rb
+```
+
+It verifies cross-instance storage, asynchronous worker materialization, reference lifecycle, real deletion, and failed-job GC protection. Run the real PostgreSQL HTTP isolation and idempotency checks with:
+
+```bash
+bundle exec rspec \
+  spec/integration/media_tenant_isolation_spec.rb \
+  spec/integration/media_p3_spec.rb
 ```
 
 ## Documentation Map
 
-See `docs/DOCUMENTATION_INDEX.en.md` for a curated map of all docs, reading order, and maintenance notes.  
+See `docs/DOCUMENTATION_INDEX.en.md` for a curated map of all docs, reading order, and maintenance notes.
 Chinese version: `docs/DOCUMENTATION_INDEX.md`.
 
+Other key documents:
+
+- `docs/design.md` — System design
+- `docs/API_DOCUMENTATION.md` — Detailed API docs
+- `docs/SETUP_GUIDE.md` — Environment setup guide
+- `docs/USAGE_EXAMPLES.md` — Usage examples
+- `docs/Hybrid_Reranking.md` — Hybrid search & reranking details
+- `docs/SmartChunking.md` — Smart chunking details
+- `docs/MIGRATION_GUIDE.md` — Migration guide
+- `docs/PERFORMANCE_GUIDE.md` — Performance optimization guide
+- `ER-diagram.mmd` — ER diagram
+
 ## Notes
 
-- Some legacy docs still contain older defaults (for example OpenAI references). Runtime truth is `config/smart_rag.yml`.
+- Some legacy docs still contain older defaults (e.g. OpenAI references). Runtime truth is `config/smart_rag.yml`.
 
 ## License