smart_rag 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 19e9abcf3817e98fad15499879a25db6fbb563bc189f5a8f0c58fbea2145cbf8
+  data.tar.gz: 88f7e4a0f1929b83c9346e8261928bdd72690de4c9da3689a3ca7882a548a6b1
+SHA512:
+  metadata.gz: f3c7c945032e61c816f383ea6ae1ea87b0be9d4fb2a794175543489e91763825f73fbb925c2a52b37d786af2e1dd126180b4da3cb5c31ded4130e1fa5bc4ebda
+  data.tar.gz: 6de0148938c32c3842d3d3e2a8f31641c508be193d70e9fa11d01c93dd413e3e805f2fec3afb348981455566989654436cb5e7f70c0351b2b94e1dc21978a94a

data/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+## Unreleased
+
+### Added
+- 新增 `retrieve(plan:)` 结构化检索入口（`RetrievalPlan -> EvidencePack`）。
+- 新增 `SmartRAG::Retrieve` 执行器：支持多 query、mode 映射、signals、provenance、stats、explain。
+- 新增索引治理接口：
+  - `rebuild_fts(document_id=nil)`
+  - `rebuild_embeddings(document_id=nil)`
+  - `reindex(document_id=nil)`
+  - `dedupe_by_content_hash`
+- 新增 `source_documents` 字段与索引：
+  - `source_type`
+  - `source_uri`
+  - `content_hash`
+- 新增轻量单测入口 `spec/unit_spec_helper.rb`（不依赖数据库连接）。
+- 新增回填任务：`rake db:backfill_source_fields`（历史数据回填新字段）。
+- 新增一键发布任务：`rake db:prepare_release`（backfill -> dedupe -> reindex）。
+- 新增 API：
+  - `backfill_source_fields(limit: nil, dry_run: false)`
+  - `prepare_release_indexes(document_id: nil, dry_run: false)`
+
+### Changed
+- `retrieve` 现支持 `global_filters.source_type` 与 `global_filters.source_uri_prefix` 的执行过滤。
+- `retrieve` 新增 `global_filters.topic_ids` 的执行过滤（按 section-topic 关系过滤）。
+- `retrieve` 新增 `budget.diversity.by_source` 执行约束。
+- `dedupe_by_content_hash` 从“仅 content_hash”升级为“`source_uri + content_hash`”去重。
+- 检索日志 (`search_logs.filters`) 新增保存 `plan/stats/explain/warnings`，用于回放与调试。
+
+### Compatibility
+- 保留 `search(...)` 旧接口，不破坏现有调用。
+- 对未支持字段通过 `explain.ignored_fields` 明确返回，不做静默忽略。

data/README.en.md

@@ -0,0 +1,115 @@
+# SmartRAG
+
+[中文 README](README.md)
+
+SmartRAG is a Ruby-based hybrid RAG library that combines vector retrieval, full-text search, and topic/tag organization for document intelligence workflows.
+
+## Overview
+
+- Hybrid retrieval: vector + full-text + weighted fusion
+- Document ingestion from local files and URLs
+- Topic and tag management APIs
+- Search logs and system statistics
+- Runnable example scripts for quick onboarding
+
+## Default Model Setup
+
+Current defaults use local Ollama-compatible endpoints:
+
+- Embedding model: `qwen3-embedding`
+- Text LLM model: `qwen3`
+- Embedding endpoint: `http://localhost:11434/v1/embeddings`
+- LLM endpoint: `http://localhost:11434/v1/chat/completions`
+
+You can override these via `.env` or `config/smart_rag.yml`.
+
+## Quick Start
+
+### 1) Install dependencies
+
+```bash
+bundle install
+```
+
+### 2) Configure environment
+
+```bash
+cp .env.example .env
+```
+
+Required DB variables:
+
+- `SMARTRAG_DB_HOST`
+- `SMARTRAG_DB_PORT`
+- `SMARTRAG_DB_NAME`
+- `SMARTRAG_DB_USER`
+- `SMARTRAG_DB_PASSWORD`
+
+### 3) Setup database
+
+```bash
+bundle exec rake db:create
+bundle exec rake db:migrate
+bundle exec rake db:seed
+```
+
+### 4) Import test docs (optional)
+
+```bash
+ruby test/import_doc.rb import
+```
+
+### 5) Run sample scripts
+
+```bash
+ruby examples/01_quick_start.rb
+ruby examples/03_search_operations.rb
+```
+
+## Minimal Usage
+
+```ruby
+require "smart_rag"
+
+config = SmartRAG::Config.load("config/smart_rag.yml")
+client = SmartRAG::SmartRAG.new(config)
+
+client.add_document("test/python_basics.md", generate_embeddings: true)
+results = client.search("What is machine learning?", search_type: "hybrid", limit: 5)
+
+puts results[:results].map { |r| r[:section_title] }
+```
+
+## Development Commands
+
+- `bundle exec rspec`: run RSpec tests
+- `ruby test/test_rag.rb`: run E2E script
+- `bundle exec rake db:reset`: recreate database
+- `gem build smart_rag.gemspec`: build gem package
+
+## Project Structure
+
+```text
+lib/
+  smart_rag.rb                 # Main API entry
+  smart_rag/core/              # Core processing logic
+  smart_rag/services/          # Search/tag/embedding services
+config/                        # Runtime config files
+db/                            # Migrations and seed SQL
+examples/                      # Example programs
+test/                          # Manual/E2E scripts + sample docs
+spec/                          # RSpec tests
+```
+
+## Documentation Map
+
+See `docs/DOCUMENTATION_INDEX.en.md` for a curated map of all docs, reading order, and maintenance notes.  
+Chinese version: `docs/DOCUMENTATION_INDEX.md`.
+
+## Notes
+
+- Some legacy docs still contain older defaults (for example OpenAI references). Runtime truth is `config/smart_rag.yml`.
+
+## License
+
+MIT

data/README.md

@@ -0,0 +1,144 @@
+# SmartRAG
+
+[English README](README.en.md)
+
+SmartRAG 是一个 Ruby 混合检索增强生成（RAG）库，结合向量检索、全文检索与主题/标签管理，用于文档智能检索与问答场景。
+
+## 项目概览
+
+- 混合检索：向量检索 + 全文检索 + 权重融合
+- 支持本地文件与 URL 文档导入
+- 提供主题与标签管理 API
+- 提供搜索日志与系统统计能力
+- 内置示例脚本便于快速上手
+
+## 默认模型配置
+
+当前默认配置为本地 Ollama 兼容端点：
+
+- Embedding 模型：`qwen3-embedding`
+- 文本 LLM 模型：`qwen3`
+- Embedding 端点：`http://localhost:11434/v1/embeddings`
+- LLM 端点：`http://localhost:11434/v1/chat/completions`
+
+可通过 `.env` 或 `config/smart_rag.yml` 覆盖以上默认值。
+
+## 快速开始
+
+### 1) 安装依赖
+
+```bash
+bundle install
+```
+
+### 2) 配置环境变量
+
+```bash
+cp .env.example .env
+```
+
+必填数据库变量：
+
+- `SMARTRAG_DB_HOST`
+- `SMARTRAG_DB_PORT`
+- `SMARTRAG_DB_NAME`
+- `SMARTRAG_DB_USER`
+- `SMARTRAG_DB_PASSWORD`
+
+### 3) 初始化数据库
+
+```bash
+bundle exec rake db:create
+bundle exec rake db:migrate
+bundle exec rake db:seed
+```
+
+### 4) 导入测试文档（可选）
+
+```bash
+ruby test/import_doc.rb import
+```
+
+### 5) 运行示例程序
+
+```bash
+ruby examples/01_quick_start.rb
+ruby examples/03_search_operations.rb
+```
+
+## 最小调用示例
+
+```ruby
+require "smart_rag"
+
+config = SmartRAG::Config.load("config/smart_rag.yml")
+client = SmartRAG::SmartRAG.new(config)
+
+client.add_document("test/python_basics.md", generate_embeddings: true)
+results = client.search("机器学习是什么？", search_type: "hybrid", limit: 5)
+
+puts results[:results].map { |r| r[:section_title] }
+```
+
+## 开发常用命令
+
+- `bundle exec rspec`：运行 RSpec 测试
+- `ruby test/test_rag.rb`：运行端到端测试脚本
+- `bundle exec rake db:reset`：重建数据库
+- `gem build smart_rag.gemspec`：构建 gem 包
+
+## 运维命令（发布前）
+
+标准发布前步骤（推荐）：
+
+1. 迁移数据库
+
+```bash
+bundle exec rake db:migrate
+```
+
+2. 回填历史数据字段（`source_type/source_uri/content_hash`）
+
+```bash
+bundle exec rake db:backfill_source_fields
+```
+
+3. 执行一键检索发布准备（backfill -> dedupe -> reindex）
+
+```bash
+bundle exec rake db:prepare_release
+```
+
+仅预演（不写入）：
+
+```bash
+DRY_RUN=1 bundle exec rake db:backfill_source_fields
+DRY_RUN=1 bundle exec rake db:prepare_release
+```
+
+## 目录结构
+
+```text
+lib/
+  smart_rag.rb                 # 主 API 入口
+  smart_rag/core/              # 核心处理逻辑
+  smart_rag/services/          # 搜索/标签/嵌入服务
+config/                        # 运行时配置
+db/                            # 迁移与种子 SQL
+examples/                      # 示例代码
+test/                          # 手工/E2E 脚本与样例文档
+spec/                          # RSpec 测试
+```
+
+## 文档导航
+
+完整文档清单、阅读顺序和维护建议见 `docs/DOCUMENTATION_INDEX.md`。  
+英文版见 `docs/DOCUMENTATION_INDEX.en.md`。
+
+## 说明
+
+- 部分历史文档仍保留旧默认值（如 OpenAI 示例）。运行时配置以 `config/smart_rag.yml` 为准。
+
+## 许可证
+
+MIT

data/config/database.yml

@@ -0,0 +1,42 @@
+# Database configuration for different environments
+# This is a sample configuration file
+# Copy and modify for your specific environment
+
+default: &default
+  adapter: postgresql
+  encoding: unicode
+  pool: 5
+  timeout: 5000
+
+development:
+  <<: *default
+  host: <%= ENV['SMARTRAG_DB_HOST'] || 'localhost' %>
+  port: <%= ENV['SMARTRAG_DB_PORT'] || 5432 %>
+  database: <%= ENV['SMARTRAG_DB_NAME'] || 'smart_rag_development' %>
+  username: <%= ENV['SMARTRAG_DB_USER'] || 'rag_user' %>
+  password: <%= ENV['SMARTRAG_DB_PASSWORD'] || 'rag_pwd' %>
+
+test:
+  <<: *default
+  host: <%= ENV['SMARTRAG_TEST_DB_HOST'] || 'localhost' %>
+  port: <%= ENV['SMARTRAG_TEST_DB_PORT'] || 5432 %>
+  database: <%= ENV['SMARTRAG_TEST_DB_NAME'] || 'smart_rag_test' %>
+  username: <%= ENV['SMARTRAG_TEST_DB_USER'] || 'postgres' %>
+  password: <%= ENV['SMARTRAG_TEST_DB_PASSWORD'] %>
+
+staging:
+  <<: *default
+  host: <%= ENV['SMARTRAG_DB_HOST'] %>
+  port: <%= ENV['SMARTRAG_DB_PORT'] || 5432 %>
+  database: <%= ENV['SMARTRAG_DB_NAME'] || 'smart_rag_staging' %>
+  username: <%= ENV['SMARTRAG_DB_USER'] %>
+  password: <%= ENV['SMARTRAG_DB_PASSWORD'] %>
+
+production:
+  <<: *default
+  host: <%= ENV['SMARTRAG_DB_HOST'] %>
+  port: <%= ENV['SMARTRAG_DB_PORT'] || 5432 %>
+  database: <%= ENV['SMARTRAG_DB_NAME'] || 'smart_rag_production' %>
+  username: <%= ENV['SMARTRAG_DB_USER'] %>
+  password: <%= ENV['SMARTRAG_DB_PASSWORD'] %>
+  pool: 10

data/config/fulltext_search.yml

@@ -0,0 +1,111 @@
+# Full-text search configuration
+# This file contains language-specific settings for full-text search
+
+default: &default
+  # Default language for text search
+  default_language: en
+
+  # Maximum number of search results to return
+  max_results: 100
+
+  # Enable Chinese text segmentation with pg_jieba
+  enable_jieba: true
+
+  # Hybrid search weight configuration
+  # These weights determine how results are combined in hybrid searches
+  hybrid_weight:
+    fulltext: 0.4  # Weight for full-text search results
+    vector: 0.6    # Weight for vector search results
+
+  # RRF (Reciprocal Rank Fusion) algorithm parameter
+  # Higher k values give more weight to lower-ranked results
+  rrf_k: 60
+
+  # Search result caching
+  cache:
+    enabled: false  # Disabled by default, can be enabled with Redis
+    ttl: 3600       # Cache TTL in seconds
+
+  # Highlighting configuration
+  highlighting:
+    enabled: true
+    max_length: 200  # Maximum length of highlighted snippets
+    pre_tags:
+      - '<mark>'
+    post_tags:
+      - '</mark>'
+
+  # Language-specific configurations
+  languages:
+    en:
+      config_name: pg_catalog.english
+      stop_words: true  # Remove common words
+      stemming: true    # Apply stemming
+    zh:
+      config_name: jieba
+      custom_dict: null  # Path to custom dictionary
+      stop_words: true
+      stemming: false
+    ja:
+      config_name: pg_catalog.simple
+      stop_words: true
+      stemming: false
+    ko:
+      config_name: pg_catalog.simple
+      stop_words: true
+      stemming: false
+
+  # Query configuration
+  queries:
+    # Minimum word length to include in search
+    min_word_length: 2
+    # Maximum number of words in a query
+    max_words: 20
+    # Allow phrase queries with quotes
+    phrase_queries: true
+    # Allow boolean operators (AND, OR, NOT)
+    boolean_operators: true
+
+  # Index configuration
+  indexes:
+    # Use GIN indexes for tsvector columns (recommended)
+    use_gin: true
+    # Create language-specific partitioned indexes
+    partitioned_indexes: true
+    # Automatic index maintenance
+    auto_rebuild: false
+    rebuild_schedule: null  # Cron format, e.g., '0 2 * * 0' for weekly on Sunday 2am
+
+  # Synonyms and thesaurus
+  synonyms:
+    enabled: false
+    # Path to synonym dictionary file
+    # Format: word1 word2 word3 (one group per line)
+    dictionary_path: null
+
+  # Performance settings
+  performance:
+    # Enable query planning with text search
+    enable_query_planning: true
+    # Work memory for sorting (increase for large result sets)
+    work_mem: '4MB'
+
+  # Monitoring and logging
+  monitoring:
+    # Log slow queries for analysis
+    log_slow_queries: true
+    # Threshold for slow query logging (ms)
+    slow_query_threshold: 100
+    # Track query performance over time
+    track_performance: true
+    # Retention period for query logs (days)
+    log_retention_days: 30
+
+  # Advanced settings
+  advanced:
+    # Consider word proximity in ranking
+    rank_by_proximity: true
+    # Weight for title matches vs content
+    title_weight: 1.5
+    # Normalize rankings
+    normalize_rank: true

data/config/llm_config.yml

@@ -0,0 +1,15 @@
+adapters:
+  openai: OpenAIAdapter
+logger_file: ./log/prompt.log
+llms:
+  OllamaEmbedding:
+    adapter: openai
+    url: http://localhost:11434/v1/
+    # Ollama OpenAI-compatible endpoint typically ignores api_key, keep a default placeholder.
+    api_key: ollama-local
+  SiliconFlow:
+    adapter: openai
+    url: https://api.siliconflow.cn/v1/
+    api_key: ""
+worker_path: "./workers"
+template_path: "./templates"

data/config/smart_rag.yml

@@ -0,0 +1,156 @@
+# SmartRAG Configuration File
+# Copy this file and modify for your environment
+#
+# Production: config/smart_rag.production.yml
+# Development: config/smart_rag.development.yml
+# Test: config/smart_rag.test.yml
+
+database:
+  adapter: postgresql
+  host: localhost
+  port: 5432
+  # Use environment variables for sensitive data
+  database: <%= ENV['SMARTRAG_DB_NAME'] || 'smart_rag_development' %>
+  username: <%= ENV['SMARTRAG_DB_USER'] || 'rag_user' %>
+  password: <%= ENV['SMARTRAG_DB_PASSWORD'] || 'rag_pwd' %>
+  pool: 5
+  encoding: unicode
+  # Connection timeout in seconds
+  timeout: 5000
+  # Extensions to enable
+  extensions:
+    - pgvector
+    - pg_jieba  # For Chinese text segmentation
+
+# Embedding Configuration
+embedding:
+  # Provider can be: openai, azure_openai, local, or custom
+  provider: <%= ENV['EMBEDDING_PROVIDER'] || 'local' %>
+  api_key: <%= ENV['EMBEDDING_API_KEY'] || 'ollama-local' %>
+  endpoint: <%= ENV['EMBEDDING_ENDPOINT'] || 'http://localhost:11434/v1/embeddings' %>
+  model: <%= ENV['EMBEDDING_MODEL'] || 'qwen3-embedding' %>
+  dimensions: <%= ENV['EMBEDDING_DIMENSIONS'] || 1024 %>
+  # Request timeout in seconds
+  timeout: 30
+  # Batch size for embedding generation
+  batch_size: 10
+
+# Full-Text Search Configuration
+fulltext_search:
+  # Default language for text search
+  default_language: <%= ENV['DEFAULT_LANGUAGE'] || 'en' %>
+
+  # Maximum number of search results to return
+  max_results: <%= ENV['MAX_SEARCH_RESULTS'] || 100 %>
+
+  # Enable Chinese text segmentation with pg_jieba
+  enable_jieba: <%= ENV['ENABLE_JIEBA'] != 'false' %>
+
+  # Path to custom dictionary for jieba (optional)
+  custom_dict_path: <%= ENV['JIEBA_DICT_PATH'] %>
+
+  # Hybrid search weight configuration (0.0 to 1.0)
+  hybrid_weight:
+    fulltext: <%= ENV['FULLTEXT_WEIGHT'] || 0.4 %>
+    vector: <%= ENV['VECTOR_WEIGHT'] || 0.6 %>
+
+  # RRF (Reciprocal Rank Fusion) algorithm parameter
+  # Higher k values give more weight to lower-ranked results
+  rrf_k: <%= ENV['RRF_K'] || 60 %>
+
+  # Search result caching configuration
+  cache:
+    enabled: <%= ENV['CACHE_ENABLED'] != 'false' %>
+    ttl: <%= ENV['CACHE_TTL'] || 3600 %>  # Cache TTL in seconds
+    # Optional Redis configuration
+    redis_url: <%= ENV['REDIS_URL'] %>
+
+  # Query logging and monitoring
+  monitoring:
+    log_slow_queries: <%= ENV['LOG_SLOW_QUERIES'] != 'false' %>
+    slow_query_threshold_ms: <%= ENV['SLOW_QUERY_THRESHOLD'] || 100 %>
+
+  # Index maintenance
+  index:
+    enable_partition: <%= ENV['ENABLE_PARTITION'] != 'false' %>
+    auto_vacuum: <%= ENV['AUTO_VACUUM'] != 'false' %>
+    # Schedule for automatic index rebuild (cron format)
+    rebuild_schedule: <%= ENV['INDEX_REBUILD_SCHEDULE'] %>
+
+# Document Processing Configuration
+chunking:
+  # Maximum characters per chunk
+  max_chars: <%= ENV['CHUNK_MAX_CHARS'] || 4000 %>
+
+  # Overlap between chunks to preserve context
+  overlap: <%= ENV['CHUNK_OVERLAP'] || 100 %>
+
+  # Enable splitting by Markdown headers first
+  split_by_headers: <%= ENV['SPLIT_BY_HEADERS'] != 'false' %>
+
+  # Minimum chunk size (discard chunks smaller than this)
+  min_chunk_size: <%= ENV['MIN_CHUNK_SIZE'] || 100 %>
+
+# Search Configuration
+search:
+  # Default number of results to return
+  default_limit: <%= ENV['DEFAULT_SEARCH_LIMIT'] || 5 %>
+
+  # Boost weight for tag matches in vector search (0.0 to 1.0)
+  tag_boost_weight: <%= ENV['TAG_BOOST_WEIGHT'] || 0.5 %>
+
+  # Maximum distance threshold for vector search results
+  max_distance_threshold: <%= ENV['MAX_DISTANCE_THRESHOLD'] || 0.3 %>
+
+# LLM Configuration for tags and summarization
+llm:
+  provider: <%= ENV['LLM_PROVIDER'] || 'openai' %>
+  api_key: <%= ENV['LLM_API_KEY'] || 'ollama-local' %>
+  endpoint: <%= ENV['LLM_ENDPOINT'] || 'http://localhost:11434/v1/chat/completions' %>
+  model: <%= ENV['LLM_MODEL'] || 'qwen3' %>
+
+  # Generation parameters
+  temperature: <%= ENV['LLM_TEMPERATURE'] || 0.3 %>
+  max_tokens: <%= ENV['LLM_MAX_TOKENS'] || 2000 %>
+  timeout: <%= ENV['LLM_TIMEOUT'] || 60 %>
+
+# Document Processing
+document:
+  # Supported file types for upload/processing
+  supported_types:
+    - .md
+    - .txt
+    - .pdf
+    - .docx
+    - .html
+
+  # Maximum file size in MB
+  max_file_size: <%= ENV['MAX_FILE_SIZE'] || 50 %>
+
+  # Download timeout for URLs
+  download_timeout: <%= ENV['DOWNLOAD_TIMEOUT'] || 30 %>
+
+# Logging Configuration
+logging:
+  # Log level: debug, info, warn, error, fatal
+  level: <%= ENV['LOG_LEVEL'] || 'info' %>
+
+  # Log format: json, plain
+  format: <%= ENV['LOG_FORMAT'] || 'json' %>
+
+  # Log file path (null for stdout)
+  file_path: <%= ENV['LOG_FILE_PATH'] %>
+
+  # Enable query logging
+  enable_query_log: <%= ENV['ENABLE_QUERY_LOG'] != 'false' %>
+
+# Performance Settings
+performance:
+  # Thread pool size for parallel processing
+  thread_pool_size: <%= ENV['THREAD_POOL_SIZE'] || 4 %>
+
+  # Batch size for database operations
+  db_batch_size: <%= ENV['DB_BATCH_SIZE'] || 100 %>
+
+  # Enable connection pooling
+  connection_pooling: <%= ENV['CONNECTION_POOLING'] != 'false' %>