smart_rag 0.1.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.env.example +252 -0
- data/.rspec +2 -0
- data/AGENTS.md +33 -0
- data/API_DOCUMENTATION.md +828 -0
- data/CHANGELOG.md +16 -1
- data/ER-diagram.mmd +144 -0
- data/Gemfile +50 -0
- data/Gemfile.lock +398 -0
- data/Hybrid_Reranking.md +171 -0
- data/README.en.md +420 -28
- data/README.md +534 -63
- data/Rakefile +268 -0
- data/SETUP_GUIDE.md +650 -0
- data/SmartChunking.md +180 -0
- data/USAGE_EXAMPLES.md +1002 -0
- data/config/llm_config.yml +4 -2
- data/config/smart_rag.yml +45 -1
- data/config.ru +15 -0
- data/db/migrations/006_create_text_search_configs.rb +3 -2
- data/db/migrations/008_create_embeddings.rb +5 -4
- data/db/migrations/012_add_metadata_to_source_sections.rb +11 -0
- data/db/migrations/013_create_media_jobs.rb +25 -0
- data/db/migrations/014_add_media_job_operations_indexes.rb +11 -0
- data/db/migrations/015_add_media_leases_and_objects.rb +80 -0
- data/db/migrations/016_add_document_principals_and_staging_references.rb +38 -0
- data/db/migrations/017_add_media_job_request_fingerprint.rb +48 -0
- data/db/seeds/text_search_configs.sql +3 -3
- data/design.md +1057 -0
- data/docs/API_DOCUMENTATION.md +838 -0
- data/docs/DOCUMENTATION_INDEX.en.md +60 -0
- data/docs/DOCUMENTATION_INDEX.md +65 -0
- data/docs/FIX_SUMMARY.md +256 -0
- data/docs/FIX_SUMMARY_COMPLETE.md +273 -0
- data/docs/Hybrid_Reranking.md +171 -0
- data/docs/MIGRATION_GUIDE.md +151 -0
- data/docs/PERFORMANCE_GUIDE.md +58 -0
- data/docs/SETUP_GUIDE.md +659 -0
- data/docs/SmartChunking.md +180 -0
- data/docs/USAGE_EXAMPLES.md +1008 -0
- data/docs/design.md +1057 -0
- data/docs/evidence_pack.md +211 -0
- data/docs/requirements.md +376 -0
- data/docs/retrieval_plan.md +251 -0
- data/docs/smartrag_improvement_plan.md +201 -0
- data/docs/smartrag_refactor.md +216 -0
- data/docs/todo.md +931 -0
- data/examples/common.rb +1 -1
- data/exe/smart-rag-db +163 -0
- data/exe/smart-rag-media-worker +34 -0
- data/lib/smart_rag/config.rb +12 -0
- data/lib/smart_rag/core/document_processor.rb +80 -16
- data/lib/smart_rag/core/local_content_store.rb +51 -0
- data/lib/smart_rag/core/media_extractors.rb +140 -0
- data/lib/smart_rag/core/media_job_queue.rb +353 -0
- data/lib/smart_rag/core/media_metadata_extractor.rb +188 -0
- data/lib/smart_rag/core/media_object_registry.rb +79 -0
- data/lib/smart_rag/core/media_processor.rb +228 -0
- data/lib/smart_rag/core/media_safety_policy.rb +61 -0
- data/lib/smart_rag/core/s3_content_store.rb +78 -0
- data/lib/smart_rag/core/transcript_normalizer.rb +44 -0
- data/lib/smart_rag/core/video_semantic_extractor.rb +130 -0
- data/lib/smart_rag/http_access_policy.rb +86 -0
- data/lib/smart_rag/http_app.rb +188 -0
- data/lib/smart_rag/models/embedding.rb +1 -1
- data/lib/smart_rag/models/research_topic.rb +1 -1
- data/lib/smart_rag/models/research_topic_section.rb +5 -0
- data/lib/smart_rag/models/research_topic_tag.rb +5 -0
- data/lib/smart_rag/models/search_log.rb +1 -1
- data/lib/smart_rag/models/section_fts.rb +5 -0
- data/lib/smart_rag/models/section_tag.rb +5 -0
- data/lib/smart_rag/models/source_document.rb +1 -1
- data/lib/smart_rag/models/source_section.rb +1 -1
- data/lib/smart_rag/models/tag.rb +1 -1
- data/lib/smart_rag/models/text_search_config.rb +5 -0
- data/lib/smart_rag/retrieve.rb +72 -1
- data/lib/smart_rag/services/embedding_service.rb +1 -1
- data/lib/smart_rag/services/fulltext_search_service.rb +11 -13
- data/lib/smart_rag/services/hybrid_search_service.rb +15 -11
- data/lib/smart_rag/services/summarization_service.rb +1 -1
- data/lib/smart_rag/services/tag_service.rb +1 -1
- data/lib/smart_rag/version.rb +1 -1
- data/lib/smart_rag.rb +264 -30
- data/patch_language.rb +27 -0
- data/requirements.md +376 -0
- data/source_documents_export.json +11072 -0
- data/todo.md +931 -0
- data/workers/analyze_content.rb +6 -2
- data/workers/get_embedding.rb +1 -1
- metadata +151 -38
data/USAGE_EXAMPLES.md
ADDED
|
@@ -0,0 +1,1002 @@
|
|
|
1
|
+
# SmartRAG Usage Examples and Best Practices
|
|
2
|
+
|
|
3
|
+
This guide provides practical examples and best practices for using SmartRAG in your applications.
|
|
4
|
+
|
|
5
|
+
## Table of Contents
|
|
6
|
+
|
|
7
|
+
1. [Quick Start](#quick-start)
|
|
8
|
+
2. [Document Management](#document-management)
|
|
9
|
+
3. [Search Operations](#search-operations)
|
|
10
|
+
4. [Research Topic Management](#research-topic-management)
|
|
11
|
+
5. [Tag Management](#tag-management)
|
|
12
|
+
6. [Advanced Usage Patterns](#advanced-usage-patterns)
|
|
13
|
+
7. [Performance Best Practices](#performance-best-practices)
|
|
14
|
+
8. [Error Handling](#error-handling)
|
|
15
|
+
9. [Common Patterns](#common-patterns)
|
|
16
|
+
|
|
17
|
+
## Quick Start
|
|
18
|
+
|
|
19
|
+
### Basic Setup
|
|
20
|
+
|
|
21
|
+
```ruby
|
|
22
|
+
require 'smart_rag'
|
|
23
|
+
require 'logger'
|
|
24
|
+
|
|
25
|
+
# Initialize configuration
|
|
26
|
+
config = {
|
|
27
|
+
database: {
|
|
28
|
+
adapter: 'postgresql',
|
|
29
|
+
host: ENV['SMARTRAG_DB_HOST'] || 'localhost',
|
|
30
|
+
database: ENV['SMARTRAG_DB_NAME'] || 'smart_rag_development',
|
|
31
|
+
user: ENV['SMARTRAG_DB_USER'] || 'smart_rag_user',
|
|
32
|
+
password: ENV['SMARTRAG_DB_PASSWORD']
|
|
33
|
+
},
|
|
34
|
+
llm: {
|
|
35
|
+
provider: 'openai',
|
|
36
|
+
api_key: ENV['OPENAI_API_KEY']
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
# Create SmartRAG instance
|
|
41
|
+
smart_rag = SmartRAG::SmartRAG.new(config)
|
|
42
|
+
smart_rag.logger = Logger.new(STDOUT)
|
|
43
|
+
smart_rag.logger.level = Logger::INFO
|
|
44
|
+
|
|
45
|
+
# Test the connection
|
|
46
|
+
puts "✓ SmartRAG initialized successfully"
|
|
47
|
+
stats = smart_rag.statistics
|
|
48
|
+
puts "✓ Database connected: #{stats[:document_count]} documents"
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### First Document Addition
|
|
52
|
+
|
|
53
|
+
```ruby
|
|
54
|
+
# Add your first document
|
|
55
|
+
result = smart_rag.add_document(
|
|
56
|
+
'https://example.com/ai-article.pdf',
|
|
57
|
+
title: 'Introduction to AI',
|
|
58
|
+
generate_embeddings: true,
|
|
59
|
+
generate_tags: true
|
|
60
|
+
)
|
|
61
|
+
|
|
62
|
+
puts "✓ Document added: ID #{result[:document_id]}"
|
|
63
|
+
puts "✓ Sections created: #{result[:section_count]}"
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
### First Search
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
# Perform your first search
|
|
70
|
+
results = smart_rag.search(
|
|
71
|
+
'machine learning algorithms',
|
|
72
|
+
search_type: 'hybrid',
|
|
73
|
+
limit: 5,
|
|
74
|
+
include_content: true
|
|
75
|
+
)
|
|
76
|
+
|
|
77
|
+
puts "\nSearch Results:"
|
|
78
|
+
results[:results].each_with_index do |result, i|
|
|
79
|
+
puts "#{i + 1}. #{result[:section_title]} (score: #{result[:combined_score].round(3)})"
|
|
80
|
+
puts " #{result[:content][0..150]}..."
|
|
81
|
+
end
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
## Document Management
|
|
85
|
+
|
|
86
|
+
### Adding Documents from Various Sources
|
|
87
|
+
|
|
88
|
+
```ruby
|
|
89
|
+
# From a local file
|
|
90
|
+
smart_rag.add_document(
|
|
91
|
+
'/path/to/document.pdf',
|
|
92
|
+
title: 'Research Paper 2024',
|
|
93
|
+
generate_embeddings: true
|
|
94
|
+
)
|
|
95
|
+
|
|
96
|
+
# From a URL
|
|
97
|
+
smart_rag.add_document(
|
|
98
|
+
'https://arxiv.org/abs/2301.00001',
|
|
99
|
+
generate_embeddings: true,
|
|
100
|
+
generate_tags: true,
|
|
101
|
+
tags: ['research', 'AI']
|
|
102
|
+
)
|
|
103
|
+
|
|
104
|
+
# With custom metadata
|
|
105
|
+
smart_rag.add_document(
|
|
106
|
+
'/path/to/report.docx',
|
|
107
|
+
title: 'Q3 Financial Report',
|
|
108
|
+
generate_embeddings: false, # Skip embeddings for non-technical docs
|
|
109
|
+
metadata: {
|
|
110
|
+
department: 'Finance',
|
|
111
|
+
year: 2024,
|
|
112
|
+
confidential: true
|
|
113
|
+
}
|
|
114
|
+
)
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
### Batch Document Processing
|
|
118
|
+
|
|
119
|
+
```ruby
|
|
120
|
+
# Process multiple documents efficiently
|
|
121
|
+
documents = [
|
|
122
|
+
{ path: '/docs/paper1.pdf', tags: ['AI'] },
|
|
123
|
+
{ path: '/docs/paper2.pdf', tags: ['ML'] },
|
|
124
|
+
{ path: '/docs/paper3.pdf', tags: ['NLP'] }
|
|
125
|
+
]
|
|
126
|
+
|
|
127
|
+
# Sequential processing
|
|
128
|
+
documents.each do |doc|
|
|
129
|
+
begin
|
|
130
|
+
result = smart_rag.add_document(
|
|
131
|
+
doc[:path],
|
|
132
|
+
generate_embeddings: true,
|
|
133
|
+
tags: doc[:tags]
|
|
134
|
+
)
|
|
135
|
+
puts "✓ Processed: #{doc[:path]}"
|
|
136
|
+
rescue => e
|
|
137
|
+
puts "✗ Failed: #{doc[:path]} - #{e.message}"
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
# Parallel processing for better performance
|
|
142
|
+
require 'concurrent'
|
|
143
|
+
|
|
144
|
+
pool = Concurrent::FixedThreadPool.new(5)
|
|
145
|
+
documents.each do |doc|
|
|
146
|
+
pool.post do
|
|
147
|
+
begin
|
|
148
|
+
smart_rag.add_document(
|
|
149
|
+
doc[:path],
|
|
150
|
+
generate_embeddings: true,
|
|
151
|
+
tags: doc[:tags]
|
|
152
|
+
)
|
|
153
|
+
puts "✓ Processed: #{doc[:path]}"
|
|
154
|
+
rescue => e
|
|
155
|
+
puts "✗ Failed: #{doc[:path]} - #{e.message}"
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
pool.shutdown
|
|
161
|
+
pool.wait_for_termination
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### Document Management Operations
|
|
165
|
+
|
|
166
|
+
```ruby
|
|
167
|
+
# List documents with pagination
|
|
168
|
+
docs_page_1 = smart_rag.list_documents(page: 1, per_page: 20)
|
|
169
|
+
docs_page_2 = smart_rag.list_documents(page: 2, per_page: 20)
|
|
170
|
+
|
|
171
|
+
# Search for specific documents
|
|
172
|
+
ml_docs = smart_rag.list_documents(search: 'machine learning')
|
|
173
|
+
|
|
174
|
+
# Get document details
|
|
175
|
+
doc = smart_rag.get_document(123)
|
|
176
|
+
puts "Title: #{doc[:title]}"
|
|
177
|
+
puts "Sections: #{doc[:section_count]}"
|
|
178
|
+
puts "Created: #{doc[:created_at]}"
|
|
179
|
+
|
|
180
|
+
# Remove a document
|
|
181
|
+
result = smart_rag.remove_document(123)
|
|
182
|
+
puts "Deleted sections: #{result[:deleted_sections]}"
|
|
183
|
+
puts "Deleted embeddings: #{result[:deleted_embeddings]}"
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## Search Operations
|
|
187
|
+
|
|
188
|
+
### Hybrid Search (Default)
|
|
189
|
+
|
|
190
|
+
Hybrid search combines vector and full-text search for optimal results.
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
# Basic hybrid search
|
|
194
|
+
results = smart_rag.search(
|
|
195
|
+
'deep learning applications in healthcare',
|
|
196
|
+
search_type: 'hybrid',
|
|
197
|
+
limit: 10,
|
|
198
|
+
alpha: 0.7 # Weight for vector search (0.0 = pure text, 1.0 = pure vector)
|
|
199
|
+
)
|
|
200
|
+
|
|
201
|
+
# With content and metadata
|
|
202
|
+
results = smart_rag.search(
|
|
203
|
+
'natural language processing',
|
|
204
|
+
search_type: 'hybrid',
|
|
205
|
+
limit: 5,
|
|
206
|
+
include_content: true,
|
|
207
|
+
include_metadata: true
|
|
208
|
+
)
|
|
209
|
+
|
|
210
|
+
# Search with filters
|
|
211
|
+
results = smart_rag.search(
|
|
212
|
+
'artificial intelligence',
|
|
213
|
+
search_type: 'hybrid',
|
|
214
|
+
limit: 10,
|
|
215
|
+
filters: {
|
|
216
|
+
document_ids: [1, 2, 3], # Search only in specific documents
|
|
217
|
+
tag_ids: [4, 5] # Filter by tags
|
|
218
|
+
}
|
|
219
|
+
)
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Vector Search
|
|
223
|
+
|
|
224
|
+
Useful for semantic similarity searches.
|
|
225
|
+
|
|
226
|
+
```ruby
|
|
227
|
+
# Pure vector search
|
|
228
|
+
results = smart_rag.vector_search(
|
|
229
|
+
'neural network architectures',
|
|
230
|
+
limit: 5
|
|
231
|
+
)
|
|
232
|
+
|
|
233
|
+
# Vector search with tag boosting
|
|
234
|
+
results = smart_rag.vector_search(
|
|
235
|
+
'transformer models',
|
|
236
|
+
limit: 10,
|
|
237
|
+
tag_boost_weight: 0.1 # Boost results with matching tags
|
|
238
|
+
)
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
### Full-Text Search
|
|
242
|
+
|
|
243
|
+
Best for exact keyword matching and boolean queries.
|
|
244
|
+
|
|
245
|
+
```ruby
|
|
246
|
+
# Basic full-text search
|
|
247
|
+
results = smart_rag.fulltext_search(
|
|
248
|
+
'convolutional neural networks',
|
|
249
|
+
limit: 5
|
|
250
|
+
)
|
|
251
|
+
|
|
252
|
+
# Advanced boolean query
|
|
253
|
+
results = smart_rag.fulltext_search(
|
|
254
|
+
'artificial AND (intelligence OR learning) AND NOT robotics',
|
|
255
|
+
limit: 10
|
|
256
|
+
)
|
|
257
|
+
|
|
258
|
+
# Phrase search
|
|
259
|
+
results = smart_rag.fulltext_search(
|
|
260
|
+
'"deep reinforcement learning"',
|
|
261
|
+
limit: 5
|
|
262
|
+
)
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
### Multi-language Search
|
|
266
|
+
|
|
267
|
+
SmartRAG automatically detects and handles multiple languages.
|
|
268
|
+
|
|
269
|
+
```ruby
|
|
270
|
+
# Chinese search
|
|
271
|
+
results = smart_rag.search('人工智能应用', language: 'zh_cn')
|
|
272
|
+
|
|
273
|
+
# Japanese search
|
|
274
|
+
results = smart_rag.search('機械学習アルゴリズム', language: 'ja')
|
|
275
|
+
|
|
276
|
+
# Korean search
|
|
277
|
+
results = smart_rag.search('딥러닝 모델', language: 'ko')
|
|
278
|
+
|
|
279
|
+
# Mixed language search (auto-detect)
|
|
280
|
+
results = smart_rag.search('AI和机器学习的发展', language: 'auto')
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
## Research Topic Management
|
|
284
|
+
|
|
285
|
+
### Creating and Organizing Topics
|
|
286
|
+
|
|
287
|
+
```ruby
|
|
288
|
+
# Create a research topic
|
|
289
|
+
topic = smart_rag.create_topic(
|
|
290
|
+
'AI in Healthcare',
|
|
291
|
+
'Applications of artificial intelligence in medical diagnosis and treatment',
|
|
292
|
+
tags: ['AI', 'healthcare', 'medicine', 'diagnosis'],
|
|
293
|
+
document_ids: [1, 2, 3] # Associate existing documents
|
|
294
|
+
)
|
|
295
|
+
|
|
296
|
+
# Create nested topic structure
|
|
297
|
+
parent_topic = smart_rag.create_topic(
|
|
298
|
+
'Machine Learning',
|
|
299
|
+
'Fundamental ML concepts and algorithms'
|
|
300
|
+
)
|
|
301
|
+
|
|
302
|
+
child_topic = smart_rag.create_topic(
|
|
303
|
+
'Deep Learning',
|
|
304
|
+
'Neural network based learning',
|
|
305
|
+
tags: ['neural_networks', 'deep_learning']
|
|
306
|
+
# Could link to parent if hierarchical topics are supported
|
|
307
|
+
)
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
### Managing Topic Content
|
|
311
|
+
|
|
312
|
+
```ruby
|
|
313
|
+
# Add documents to a topic
|
|
314
|
+
topic_id = 456
|
|
315
|
+
document_id = 123
|
|
316
|
+
|
|
317
|
+
result = smart_rag.add_document_to_topic(topic_id, document_id)
|
|
318
|
+
puts "Added #{result[:added_sections]} sections to topic"
|
|
319
|
+
|
|
320
|
+
# Get topic recommendations
|
|
321
|
+
recommendations = smart_rag.get_topic_recommendations(topic_id, limit: 10)
|
|
322
|
+
|
|
323
|
+
recommendations[:recommendations].each do |rec|
|
|
324
|
+
puts "Recommended: #{rec[:section_title]}"
|
|
325
|
+
puts " Matching tags: #{rec[:matching_tags]}"
|
|
326
|
+
puts " Score: #{rec[:relevance_score]}"
|
|
327
|
+
end
|
|
328
|
+
|
|
329
|
+
# List all topics
|
|
330
|
+
topics = smart_rag.list_topics(page: 1, per_page: 20)
|
|
331
|
+
topics[:topics].each do |topic|
|
|
332
|
+
puts "#{topic[:title]} (#{topic[:document_count]} documents)"
|
|
333
|
+
end
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
## Tag Management
|
|
337
|
+
|
|
338
|
+
### Automatic Tag Generation
|
|
339
|
+
|
|
340
|
+
```ruby
|
|
341
|
+
# Generate tags for text
|
|
342
|
+
text = """
|
|
343
|
+
Machine learning is a subset of artificial intelligence that enables systems
|
|
344
|
+
to learn and improve from experience without being explicitly programmed.
|
|
345
|
+
It focuses on developing computer programs that can access data and use it
|
|
346
|
+
to learn for themselves.
|
|
347
|
+
"""
|
|
348
|
+
|
|
349
|
+
tags = smart_rag.generate_tags(text, topic: 'AI Introduction')
|
|
350
|
+
puts "Categories: #{tags[:categories].join(', ')}"
|
|
351
|
+
puts "Content tags: #{tags[:content_tags].join(', ')}"
|
|
352
|
+
|
|
353
|
+
# Batch generate tags for document sections
|
|
354
|
+
document = smart_rag.get_document(1)
|
|
355
|
+
sections = document[:sections]
|
|
356
|
+
|
|
357
|
+
tags_by_section = {}
|
|
358
|
+
sections.each do |section|
|
|
359
|
+
tags = smart_rag.generate_tags(
|
|
360
|
+
section[:content],
|
|
361
|
+
topic: document[:title],
|
|
362
|
+
max_tags: 5
|
|
363
|
+
)
|
|
364
|
+
tags_by_section[section[:id]] = tags
|
|
365
|
+
end
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
### Manual Tag Management
|
|
369
|
+
|
|
370
|
+
```ruby
|
|
371
|
+
# Create hierarchical tags
|
|
372
|
+
tag_service = SmartRAG::Services::TagService.new
|
|
373
|
+
|
|
374
|
+
hierarchy = {
|
|
375
|
+
"Technology" => {
|
|
376
|
+
"AI" => ["Machine Learning", "Deep Learning", "Neural Networks"],
|
|
377
|
+
"Programming" => ["Python", "Ruby", "JavaScript"]
|
|
378
|
+
}
|
|
379
|
+
}
|
|
380
|
+
|
|
381
|
+
created_tags = tag_service.create_hierarchy(hierarchy)
|
|
382
|
+
|
|
383
|
+
# Associate tags with content
|
|
384
|
+
tag = SmartRAG::Models::Tag.find_or_create("machine_learning")
|
|
385
|
+
section = SmartRAG::Models::SourceSection[1]
|
|
386
|
+
|
|
387
|
+
# Add tag to section
|
|
388
|
+
section.add_tag(tag)
|
|
389
|
+
|
|
390
|
+
# Find content by tag
|
|
391
|
+
ml_sections = tag.sections
|
|
392
|
+
ml_sections.each do |section|
|
|
393
|
+
puts "#{section.section_title}: #{section.content[0..100]}..."
|
|
394
|
+
end
|
|
395
|
+
|
|
396
|
+
# Search for tags
|
|
397
|
+
tags = tag_service.search_tags('learn', limit: 10)
|
|
398
|
+
tags.each { |tag| puts "#{tag.name} (#{tag.section_count} sections)" }
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
## Advanced Usage Patterns
|
|
402
|
+
|
|
403
|
+
### Context-Aware Search
|
|
404
|
+
|
|
405
|
+
```ruby
|
|
406
|
+
class ContextualSearch
|
|
407
|
+
def initialize(smart_rag)
|
|
408
|
+
@smart_rag = smart_rag
|
|
409
|
+
@search_history = []
|
|
410
|
+
end
|
|
411
|
+
|
|
412
|
+
def search_with_context(query, user_context = {})
|
|
413
|
+
# Enhance query with context
|
|
414
|
+
enhanced_query = enhance_query(query, user_context)
|
|
415
|
+
|
|
416
|
+
# Perform search
|
|
417
|
+
results = @smart_rag.search(
|
|
418
|
+
enhanced_query,
|
|
419
|
+
search_type: 'hybrid',
|
|
420
|
+
limit: 10,
|
|
421
|
+
filters: build_filters(user_context)
|
|
422
|
+
)
|
|
423
|
+
|
|
424
|
+
# Store in history
|
|
425
|
+
@search_history << { query: query, context: user_context, results: results }
|
|
426
|
+
|
|
427
|
+
results
|
|
428
|
+
end
|
|
429
|
+
|
|
430
|
+
private
|
|
431
|
+
|
|
432
|
+
def enhance_query(query, context)
|
|
433
|
+
# Add context-specific terms
|
|
434
|
+
case context[:domain]
|
|
435
|
+
when 'healthcare'
|
|
436
|
+
"#{query} medical health clinical"
|
|
437
|
+
when 'finance'
|
|
438
|
+
"#{query} financial economic banking"
|
|
439
|
+
else
|
|
440
|
+
query
|
|
441
|
+
end
|
|
442
|
+
end
|
|
443
|
+
|
|
444
|
+
def build_filters(context)
|
|
445
|
+
filters = {}
|
|
446
|
+
filters[:document_ids] = context[:document_ids] if context[:document_ids]
|
|
447
|
+
filters[:tag_ids] = context[:preferred_tags] if context[:preferred_tags]
|
|
448
|
+
filters
|
|
449
|
+
end
|
|
450
|
+
end
|
|
451
|
+
|
|
452
|
+
# Usage
|
|
453
|
+
contextual_search = ContextualSearch.new(smart_rag)
|
|
454
|
+
results = contextual_search.search_with_context(
|
|
455
|
+
'risk assessment',
|
|
456
|
+
user_context: {
|
|
457
|
+
domain: 'finance',
|
|
458
|
+
document_ids: [1, 2, 3],
|
|
459
|
+
preferred_tags: [4, 5]
|
|
460
|
+
}
|
|
461
|
+
)
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
### Search Result Processing Pipeline
|
|
465
|
+
|
|
466
|
+
```ruby
|
|
467
|
+
class SearchPipeline
|
|
468
|
+
def initialize(smart_rag)
|
|
469
|
+
@smart_rag = smart_rag
|
|
470
|
+
@processors = []
|
|
471
|
+
end
|
|
472
|
+
|
|
473
|
+
def add_processor(&block)
|
|
474
|
+
@processors << block
|
|
475
|
+
self
|
|
476
|
+
end
|
|
477
|
+
|
|
478
|
+
def search(query, options = {})
|
|
479
|
+
# Initial search
|
|
480
|
+
results = @smart_rag.search(query, options)
|
|
481
|
+
|
|
482
|
+
# Process through pipeline
|
|
483
|
+
@processors.each do |processor|
|
|
484
|
+
results = processor.call(results, query, options)
|
|
485
|
+
end
|
|
486
|
+
|
|
487
|
+
results
|
|
488
|
+
end
|
|
489
|
+
end
|
|
490
|
+
|
|
491
|
+
# Create pipeline with processors
|
|
492
|
+
pipeline = SearchPipeline.new(smart_rag)
|
|
493
|
+
|
|
494
|
+
# Add relevance scoring
|
|
495
|
+
pipeline.add_processor do |results, query, options|
|
|
496
|
+
results[:results].each do |result|
|
|
497
|
+
result[:relevance_score] = calculate_relevance(result, query)
|
|
498
|
+
end
|
|
499
|
+
results
|
|
500
|
+
end
|
|
501
|
+
|
|
502
|
+
# Add result filtering
|
|
503
|
+
pipeline.add_processor do |results, query, options|
|
|
504
|
+
min_score = options[:min_score] || 0.5
|
|
505
|
+
results[:results].select! { |r| r[:relevance_score] >= min_score }
|
|
506
|
+
results[:metadata][:filtered_count] = results[:results].length
|
|
507
|
+
results
|
|
508
|
+
end
|
|
509
|
+
|
|
510
|
+
# Use pipeline
|
|
511
|
+
results = pipeline.search(
|
|
512
|
+
'neural networks',
|
|
513
|
+
min_score: 0.7,
|
|
514
|
+
limit: 20
|
|
515
|
+
)
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
### Caching Search Results
|
|
519
|
+
|
|
520
|
+
```ruby
|
|
521
|
+
require 'redis'
|
|
522
|
+
|
|
523
|
+
class CachedSmartRAG
|
|
524
|
+
def initialize(smart_rag, redis_client)
|
|
525
|
+
@smart_rag = smart_rag
|
|
526
|
+
@redis = redis_client
|
|
527
|
+
@cache_ttl = 3600 # 1 hour
|
|
528
|
+
end
|
|
529
|
+
|
|
530
|
+
def search(query, options = {})
|
|
531
|
+
# Create cache key
|
|
532
|
+
cache_key = create_cache_key(query, options)
|
|
533
|
+
|
|
534
|
+
# Try to get from cache
|
|
535
|
+
cached = @redis.get(cache_key)
|
|
536
|
+
if cached
|
|
537
|
+
puts "Cache hit for: #{query}"
|
|
538
|
+
return JSON.parse(cached, symbolize_names: true)
|
|
539
|
+
end
|
|
540
|
+
|
|
541
|
+
# Perform search
|
|
542
|
+
results = @smart_rag.search(query, options)
|
|
543
|
+
|
|
544
|
+
# Store in cache
|
|
545
|
+
@redis.setex(cache_key, @cache_ttl, results.to_json)
|
|
546
|
+
|
|
547
|
+
puts "Cache miss for: #{query}"
|
|
548
|
+
results
|
|
549
|
+
end
|
|
550
|
+
|
|
551
|
+
private
|
|
552
|
+
|
|
553
|
+
def create_cache_key(query, options)
|
|
554
|
+
key_parts = [query, options.sort].flatten.join(':')
|
|
555
|
+
"search:#{Digest::MD5.hexdigest(key_parts)}"
|
|
556
|
+
end
|
|
557
|
+
end
|
|
558
|
+
|
|
559
|
+
# Usage
|
|
560
|
+
redis = Redis.new
|
|
561
|
+
$cached_rag = CachedSmartRAG.new(smart_rag, redis)
|
|
562
|
+
|
|
563
|
+
# First search - cache miss
|
|
564
|
+
results1 = $cached_rag.search('deep learning', limit: 10)
|
|
565
|
+
|
|
566
|
+
# Second search - cache hit
|
|
567
|
+
results2 = $cached_rag.search('deep learning', limit: 10)
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
### Building a Q&A System
|
|
571
|
+
|
|
572
|
+
```ruby
|
|
573
|
+
class QA_system
|
|
574
|
+
def initialize(smart_rag)
|
|
575
|
+
@smart_rag = smart_rag
|
|
576
|
+
end
|
|
577
|
+
|
|
578
|
+
def answer(question, options = {})
|
|
579
|
+
# Search for relevant information
|
|
580
|
+
search_results = @smart_rag.search(
|
|
581
|
+
question,
|
|
582
|
+
search_type: 'hybrid',
|
|
583
|
+
limit: options[:context_limit] || 5,
|
|
584
|
+
include_content: true
|
|
585
|
+
)
|
|
586
|
+
|
|
587
|
+
# Generate answer based on search results (requires LLM integration)
|
|
588
|
+
answer = generate_answer(question, search_results[:results])
|
|
589
|
+
|
|
590
|
+
{
|
|
591
|
+
question: question,
|
|
592
|
+
answer: answer,
|
|
593
|
+
sources: extract_sources(search_results[:results]),
|
|
594
|
+
confidence: calculate_confidence(search_results[:results])
|
|
595
|
+
}
|
|
596
|
+
end
|
|
597
|
+
|
|
598
|
+
private
|
|
599
|
+
|
|
600
|
+
def generate_answer(question, results)
|
|
601
|
+
return "I don't have enough information to answer this question." if results.empty?
|
|
602
|
+
|
|
603
|
+
# Combine relevant content
|
|
604
|
+
context = results.map { |r| r[:content] }.join("\n\n---\n\n")
|
|
605
|
+
|
|
606
|
+
# Here you would call an LLM API to generate the answer
|
|
607
|
+
# This is a simplified version
|
|
608
|
+
"Based on the available information: #{context[0..500]}..."
|
|
609
|
+
end
|
|
610
|
+
|
|
611
|
+
def extract_sources(results)
|
|
612
|
+
results.map do |result|
|
|
613
|
+
{
|
|
614
|
+
section_id: result[:section_id],
|
|
615
|
+
title: result[:section_title],
|
|
616
|
+
score: result[:combined_score]
|
|
617
|
+
}
|
|
618
|
+
end
|
|
619
|
+
end
|
|
620
|
+
|
|
621
|
+
def calculate_confidence(results)
|
|
622
|
+
return 0.0 if results.empty?
|
|
623
|
+
|
|
624
|
+
# Simple confidence based on top result score
|
|
625
|
+
[results.first[:combined_score], 1.0].min
|
|
626
|
+
end
|
|
627
|
+
end
|
|
628
|
+
|
|
629
|
+
# Usage
|
|
630
|
+
qa = QA_system.new(smart_rag)
|
|
631
|
+
response = qa.answer(
|
|
632
|
+
'What are the applications of transformers in NLP?',
|
|
633
|
+
context_limit: 3
|
|
634
|
+
)
|
|
635
|
+
|
|
636
|
+
puts "Answer: #{response[:answer]}"
|
|
637
|
+
puts "Confidence: #{(response[:confidence] * 100).round(1)}%"
|
|
638
|
+
puts "Sources:"
|
|
639
|
+
response[:sources].each do |source|
|
|
640
|
+
puts " - #{source[:title]} (ID: #{source[:section_id]})"
|
|
641
|
+
end
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
## Performance Best Practices
|
|
645
|
+
|
|
646
|
+
### 1. Batch Operations
|
|
647
|
+
|
|
648
|
+
```ruby
|
|
649
|
+
# Instead of individual operations
|
|
650
|
+
bad_practice = documents.map do |doc|
|
|
651
|
+
smart_rag.add_document(doc[:path], generate_embeddings: true)
|
|
652
|
+
end
|
|
653
|
+
|
|
654
|
+
# Use batch processing
|
|
655
|
+
good_practice = documents.each_slice(10) do |batch|
|
|
656
|
+
# Process batch in parallel
|
|
657
|
+
batch.map do |doc|
|
|
658
|
+
Concurrent::Promises.future do
|
|
659
|
+
smart_rag.add_document(doc[:path], generate_embeddings: true)
|
|
660
|
+
end
|
|
661
|
+
end.map(&:value)
|
|
662
|
+
end
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
### 2. Connection Pooling
|
|
666
|
+
|
|
667
|
+
```ruby
|
|
668
|
+
# Configure database connection pool
|
|
669
|
+
config = {
|
|
670
|
+
database: {
|
|
671
|
+
adapter: 'postgresql',
|
|
672
|
+
host: 'localhost',
|
|
673
|
+
database: 'smart_rag',
|
|
674
|
+
user: 'user',
|
|
675
|
+
password: 'pass',
|
|
676
|
+
pool: 25, # Increase pool size
|
|
677
|
+
timeout: 5000, # Connection timeout
|
|
678
|
+
max_connections: 100 # Maximum connections
|
|
679
|
+
}
|
|
680
|
+
}
|
|
681
|
+
```
|
|
682
|
+
|
|
683
|
+
### 3. Efficient Searching
|
|
684
|
+
|
|
685
|
+
```ruby
|
|
686
|
+
# Use appropriate search types for queries
|
|
687
|
+
# For exact matches
|
|
688
|
+
text_results = smart_rag.fulltext_search('error code 404', limit: 5)
|
|
689
|
+
|
|
690
|
+
# For conceptual similarity
|
|
691
|
+
vector_results = smart_rag.vector_search('debugging techniques', limit: 5)
|
|
692
|
+
|
|
693
|
+
# For general queries
|
|
694
|
+
hybrid_results = smart_rag.search('how to fix bugs', search_type: 'hybrid')
|
|
695
|
+
|
|
696
|
+
# Adjust alpha based on use case
|
|
697
|
+
# - Technical/keyword-heavy: lower alpha (0.3-0.5)
|
|
698
|
+
# - Conceptual/exploratory: higher alpha (0.7-0.9)
|
|
699
|
+
```
|
|
700
|
+
|
|
701
|
+
### 4. Caching Strategies
|
|
702
|
+
|
|
703
|
+
```ruby
|
|
704
|
+
# Cache embeddings for repeated content
|
|
705
|
+
class EmbeddingCache
|
|
706
|
+
def initialize
|
|
707
|
+
@cache = {}
|
|
708
|
+
end
|
|
709
|
+
|
|
710
|
+
def get_embedding(text)
|
|
711
|
+
hash = Digest::MD5.hexdigest(text)
|
|
712
|
+
@cache[hash] ||= generate_embedding(text)
|
|
713
|
+
end
|
|
714
|
+
end
|
|
715
|
+
|
|
716
|
+
# Cache search results
|
|
717
|
+
class SearchCache
|
|
718
|
+
def initialize(redis, ttl: 3600)
|
|
719
|
+
@redis = redis
|
|
720
|
+
@ttl = ttl
|
|
721
|
+
end
|
|
722
|
+
|
|
723
|
+
def fetch(query, options = {}, &block)
|
|
724
|
+
key = cache_key(query, options)
|
|
725
|
+
|
|
726
|
+
if result = @redis.get(key)
|
|
727
|
+
JSON.parse(result, symbolize_names: true)
|
|
728
|
+
else
|
|
729
|
+
result = block.call
|
|
730
|
+
@redis.setex(key, @ttl, result.to_json)
|
|
731
|
+
result
|
|
732
|
+
end
|
|
733
|
+
end
|
|
734
|
+
end
|
|
735
|
+
```
|
|
736
|
+
|
|
737
|
+
### 5. Database Optimization
|
|
738
|
+
|
|
739
|
+
```sql
|
|
740
|
+
-- Create optimized indexes
|
|
741
|
+
CREATE INDEX CONCURRENTLY idx_section_fts_content
|
|
742
|
+
ON section_fts USING gin(to_tsvector('english', content));
|
|
743
|
+
|
|
744
|
+
CREATE INDEX CONCURRENTLY idx_embeddings_vector
|
|
745
|
+
ON embeddings USING ivfflat (vector vector_cosine_ops)
|
|
746
|
+
WITH (lists = 100);
|
|
747
|
+
|
|
748
|
+
-- Monitor and optimize slow queries
|
|
749
|
+
EXPLAIN ANALYZE
|
|
750
|
+
SELECT * FROM hybrid_search('machine learning', 10);
|
|
751
|
+
```
|
|
752
|
+
|
|
753
|
+
## Error Handling
|
|
754
|
+
|
|
755
|
+
### Comprehensive Error Handling
|
|
756
|
+
|
|
757
|
+
```ruby
|
|
758
|
+
begin
|
|
759
|
+
result = smart_rag.add_document(
|
|
760
|
+
'/path/to/document.pdf',
|
|
761
|
+
generate_embeddings: true
|
|
762
|
+
)
|
|
763
|
+
rescue SmartRAG::Errors::ArgumentError => e
|
|
764
|
+
puts "Invalid arguments: #{e.message}"
|
|
765
|
+
rescue SmartRAG::Errors::DatabaseError => e
|
|
766
|
+
puts "Database error: #{e.message}"
|
|
767
|
+
# Attempt to reconnect or use fallback
|
|
768
|
+
rescue SmartRAG::Errors::EmbeddingGenerationError => e
|
|
769
|
+
puts "Embedding generation failed: #{e.message}"
|
|
770
|
+
# Retry or skip embeddings
|
|
771
|
+
rescue SmartRAG::Errors::DocumentProcessingError => e
|
|
772
|
+
puts "Document processing failed: #{e.message}"
|
|
773
|
+
# Log and continue with next document
|
|
774
|
+
rescue => e
|
|
775
|
+
puts "Unexpected error: #{e.message}"
|
|
776
|
+
# Log for investigation
|
|
777
|
+
end
|
|
778
|
+
```
|
|
779
|
+
|
|
780
|
+
### Retry Logic
|
|
781
|
+
|
|
782
|
+
```ruby
|
|
783
|
+
require 'retriable'
|
|
784
|
+
|
|
785
|
+
class RetryableSmartRAG
|
|
786
|
+
def initialize(smart_rag)
|
|
787
|
+
@smart_rag = smart_rag
|
|
788
|
+
end
|
|
789
|
+
|
|
790
|
+
def add_document(path, options = {})
|
|
791
|
+
Retriable.retriable(
|
|
792
|
+
on: [SmartRAG::Errors::EmbeddingGenerationError],
|
|
793
|
+
tries: 3,
|
|
794
|
+
base_interval: 1,
|
|
795
|
+
multiplier: 2
|
|
796
|
+
) do
|
|
797
|
+
@smart_rag.add_document(path, options)
|
|
798
|
+
end
|
|
799
|
+
end
|
|
800
|
+
|
|
801
|
+
def search(query, options = {})
|
|
802
|
+
Retriable.retriable(
|
|
803
|
+
on: [SmartRAG::Errors::DatabaseError],
|
|
804
|
+
tries: 3,
|
|
805
|
+
base_interval: 0.5
|
|
806
|
+
) do
|
|
807
|
+
@smart_rag.search(query, options)
|
|
808
|
+
end
|
|
809
|
+
end
|
|
810
|
+
end
|
|
811
|
+
```
|
|
812
|
+
|
|
813
|
+
## Common Patterns
|
|
814
|
+
|
|
815
|
+
### Pattern 1: Document Processing Pipeline
|
|
816
|
+
|
|
817
|
+
```ruby
|
|
818
|
+
class DocumentPipeline
|
|
819
|
+
def initialize(smart_rag)
|
|
820
|
+
@smart_rag = smart_rag
|
|
821
|
+
end
|
|
822
|
+
|
|
823
|
+
def process(files, options = {})
|
|
824
|
+
results = []
|
|
825
|
+
|
|
826
|
+
files.each do |file|
|
|
827
|
+
begin
|
|
828
|
+
# Step 1: Add document
|
|
829
|
+
doc_result = @smart_rag.add_document(
|
|
830
|
+
file,
|
|
831
|
+
generate_embeddings: false # Delay embedding generation
|
|
832
|
+
)
|
|
833
|
+
|
|
834
|
+
# Step 2: Generate tags
|
|
835
|
+
tags = @smart_rag.generate_tags(
|
|
836
|
+
extract_text(file),
|
|
837
|
+
topic: options[:topic]
|
|
838
|
+
)
|
|
839
|
+
|
|
840
|
+
# Step 3: Apply tags
|
|
841
|
+
if doc_result[:document_id]
|
|
842
|
+
document = SmartRAG::Models::SourceDocument[doc_result[:document_id]]
|
|
843
|
+
sections = document.sections
|
|
844
|
+
|
|
845
|
+
sections.each do |section|
|
|
846
|
+
tag_objects = tags[:content_tags].map do |tag_name|
|
|
847
|
+
SmartRAG::Models::Tag.find_or_create(tag_name)
|
|
848
|
+
end
|
|
849
|
+
|
|
850
|
+
section.add_tag(*tag_objects)
|
|
851
|
+
end
|
|
852
|
+
|
|
853
|
+
# Step 4: Generate embeddings (batch)
|
|
854
|
+
document.sections.each do |section|
|
|
855
|
+
embedding = @smart_rag.generate_embedding(section.content)
|
|
856
|
+
store_embedding(section.id, embedding)
|
|
857
|
+
end
|
|
858
|
+
end
|
|
859
|
+
|
|
860
|
+
results << { success: true, file: file, document_id: doc_result[:document_id] }
|
|
861
|
+
rescue => e
|
|
862
|
+
results << { success: false, file: file, error: e.message }
|
|
863
|
+
end
|
|
864
|
+
end
|
|
865
|
+
|
|
866
|
+
results
|
|
867
|
+
end
|
|
868
|
+
end
|
|
869
|
+
```
|
|
870
|
+
|
|
871
|
+
### Pattern 2: Incremental Indexing
|
|
872
|
+
|
|
873
|
+
```ruby
|
|
874
|
+
class IncrementalIndexer
|
|
875
|
+
def initialize(smart_rag)
|
|
876
|
+
@smart_rag = smart_rag
|
|
877
|
+
end
|
|
878
|
+
|
|
879
|
+
def index_new_documents(source_dir, last_check = nil)
|
|
880
|
+
# Find new or modified documents
|
|
881
|
+
pattern = File.join(source_dir, '**/*.pdf')
|
|
882
|
+
documents = Dir.glob(pattern)
|
|
883
|
+
|
|
884
|
+
if last_check
|
|
885
|
+
documents.select! { |doc| File.mtime(doc) > last_check }
|
|
886
|
+
end
|
|
887
|
+
|
|
888
|
+
# Process in batches
|
|
889
|
+
documents.each_slice(10) do |batch|
|
|
890
|
+
batch_results = process_batch(batch)
|
|
891
|
+
log_results(batch_results)
|
|
892
|
+
end
|
|
893
|
+
end
|
|
894
|
+
|
|
895
|
+
private
|
|
896
|
+
|
|
897
|
+
def process_batch(files)
|
|
898
|
+
files.map do |file|
|
|
899
|
+
begin
|
|
900
|
+
result = @smart_rag.add_document(file, generate_embeddings: true)
|
|
901
|
+
{ file: file, success: true, document_id: result[:document_id] }
|
|
902
|
+
rescue => e
|
|
903
|
+
{ file: file, success: false, error: e.message }
|
|
904
|
+
end
|
|
905
|
+
end
|
|
906
|
+
end
|
|
907
|
+
end
|
|
908
|
+
```
|
|
909
|
+
|
|
910
|
+
### Pattern 3: Search Analytics
|
|
911
|
+
|
|
912
|
+
```ruby
|
|
913
|
+
class SearchAnalytics
|
|
914
|
+
def initialize(smart_rag)
|
|
915
|
+
@smart_rag = smart_rag
|
|
916
|
+
end
|
|
917
|
+
|
|
918
|
+
def analyze_search_patterns(days = 30)
|
|
919
|
+
logs = smart_rag.search_logs(limit: 1000)
|
|
920
|
+
|
|
921
|
+
analytics = {
|
|
922
|
+
total_searches: logs.length,
|
|
923
|
+
avg_execution_time: logs.sum { |l| l[:execution_time_ms] } / logs.length,
|
|
924
|
+
popular_queries: popular_queries(logs),
|
|
925
|
+
failed_searches: logs.count { |l| l[:results_count] == 0 },
|
|
926
|
+
trend_analysis: trend_analysis(logs, days)
|
|
927
|
+
}
|
|
928
|
+
end
|
|
929
|
+
|
|
930
|
+
def identify_content_gaps(logs)
|
|
931
|
+
no_result_queries = logs.select { |l| l[:results_count] == 0 }
|
|
932
|
+
|
|
933
|
+
# Group similar queries
|
|
934
|
+
clusters = cluster_queries(no_result_queries)
|
|
935
|
+
|
|
936
|
+
# Identify topics needing more content
|
|
937
|
+
clusters.map do |cluster|
|
|
938
|
+
{
|
|
939
|
+
topic: cluster[:topic],
|
|
940
|
+
query_count: cluster[:queries].length,
|
|
941
|
+
sample_queries: cluster[:queries].first(3)
|
|
942
|
+
}
|
|
943
|
+
end
|
|
944
|
+
end
|
|
945
|
+
end
|
|
946
|
+
```
|
|
947
|
+
|
|
948
|
+
### Pattern 4: Multi-tenant Applications
|
|
949
|
+
|
|
950
|
+
```ruby
|
|
951
|
+
class MultiTenantSmartRAG
|
|
952
|
+
def initialize(smart_rag)
|
|
953
|
+
@smart_rag = smart_rag
|
|
954
|
+
end
|
|
955
|
+
|
|
956
|
+
def add_document(tenant_id, path, options = {})
|
|
957
|
+
# Add tenant isolation
|
|
958
|
+
options[:metadata] ||= {}
|
|
959
|
+
options[:metadata][:tenant_id] = tenant_id
|
|
960
|
+
|
|
961
|
+
@smart_rag.add_document(path, options)
|
|
962
|
+
end
|
|
963
|
+
|
|
964
|
+
def search(tenant_id, query, options = {})
|
|
965
|
+
# Filter by tenant
|
|
966
|
+
options[:filters] ||= {}
|
|
967
|
+
options[:filters][:metadata] = { tenant_id: tenant_id }
|
|
968
|
+
|
|
969
|
+
@smart_rag.search(query, options)
|
|
970
|
+
end
|
|
971
|
+
|
|
972
|
+
def get_statistics(tenant_id)
|
|
973
|
+
# Get tenant-specific stats
|
|
974
|
+
@smart_rag.statistics(tenant_id: tenant_id)
|
|
975
|
+
end
|
|
976
|
+
end
|
|
977
|
+
```
|
|
978
|
+
|
|
979
|
+
## Summary
|
|
980
|
+
|
|
981
|
+
This guide has covered:
|
|
982
|
+
|
|
983
|
+
1. **Quick Start** - Basic setup and first operations
|
|
984
|
+
2. **Document Management** - Adding, organizing, and managing documents
|
|
985
|
+
3. **Search Operations** - Various search types and advanced filtering
|
|
986
|
+
4. **Research Topics** - Organizing content into thematic collections
|
|
987
|
+
5. **Tag Management** - Automatic and manual tagging strategies
|
|
988
|
+
6. **Advanced Patterns** - Production-ready implementations
|
|
989
|
+
7. **Performance** - Best practices for optimal performance
|
|
990
|
+
8. **Error Handling** - Robust error management strategies
|
|
991
|
+
9. **Common Patterns** - Reusable solutions for typical scenarios
|
|
992
|
+
|
|
993
|
+
For more information, see:
|
|
994
|
+
- [API Documentation](API_DOCUMENTATION.md) - Complete API reference
|
|
995
|
+
- [Performance Guide](PERFORMANCE_GUIDE.md) - Performance optimization details
|
|
996
|
+
- [Migration Guide](MIGRATION_GUIDE.md) - Version upgrade instructions
|
|
997
|
+
|
|
998
|
+
## Support
|
|
999
|
+
|
|
1000
|
+
- GitHub Issues: https://github.com/your-org/smart_rag/issues
|
|
1001
|
+
- Documentation Issues: Report any errors or inconsistencies in examples
|
|
1002
|
+
- Community Forum: Share your usage patterns and learn from others
|