smart_rag 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (90) hide show
  1. checksums.yaml +4 -4
  2. data/.env.example +252 -0
  3. data/.rspec +2 -0
  4. data/AGENTS.md +33 -0
  5. data/API_DOCUMENTATION.md +828 -0
  6. data/CHANGELOG.md +16 -1
  7. data/ER-diagram.mmd +144 -0
  8. data/Gemfile +50 -0
  9. data/Gemfile.lock +398 -0
  10. data/Hybrid_Reranking.md +171 -0
  11. data/README.en.md +420 -28
  12. data/README.md +534 -63
  13. data/Rakefile +268 -0
  14. data/SETUP_GUIDE.md +650 -0
  15. data/SmartChunking.md +180 -0
  16. data/USAGE_EXAMPLES.md +1002 -0
  17. data/config/llm_config.yml +4 -2
  18. data/config/smart_rag.yml +45 -1
  19. data/config.ru +15 -0
  20. data/db/migrations/006_create_text_search_configs.rb +3 -2
  21. data/db/migrations/008_create_embeddings.rb +5 -4
  22. data/db/migrations/012_add_metadata_to_source_sections.rb +11 -0
  23. data/db/migrations/013_create_media_jobs.rb +25 -0
  24. data/db/migrations/014_add_media_job_operations_indexes.rb +11 -0
  25. data/db/migrations/015_add_media_leases_and_objects.rb +80 -0
  26. data/db/migrations/016_add_document_principals_and_staging_references.rb +38 -0
  27. data/db/migrations/017_add_media_job_request_fingerprint.rb +48 -0
  28. data/db/seeds/text_search_configs.sql +3 -3
  29. data/design.md +1057 -0
  30. data/docs/API_DOCUMENTATION.md +838 -0
  31. data/docs/DOCUMENTATION_INDEX.en.md +60 -0
  32. data/docs/DOCUMENTATION_INDEX.md +65 -0
  33. data/docs/FIX_SUMMARY.md +256 -0
  34. data/docs/FIX_SUMMARY_COMPLETE.md +273 -0
  35. data/docs/Hybrid_Reranking.md +171 -0
  36. data/docs/MIGRATION_GUIDE.md +151 -0
  37. data/docs/PERFORMANCE_GUIDE.md +58 -0
  38. data/docs/SETUP_GUIDE.md +659 -0
  39. data/docs/SmartChunking.md +180 -0
  40. data/docs/USAGE_EXAMPLES.md +1008 -0
  41. data/docs/design.md +1057 -0
  42. data/docs/evidence_pack.md +211 -0
  43. data/docs/requirements.md +376 -0
  44. data/docs/retrieval_plan.md +251 -0
  45. data/docs/smartrag_improvement_plan.md +201 -0
  46. data/docs/smartrag_refactor.md +216 -0
  47. data/docs/todo.md +931 -0
  48. data/examples/common.rb +1 -1
  49. data/exe/smart-rag-db +163 -0
  50. data/exe/smart-rag-media-worker +34 -0
  51. data/lib/smart_rag/config.rb +12 -0
  52. data/lib/smart_rag/core/document_processor.rb +80 -16
  53. data/lib/smart_rag/core/local_content_store.rb +51 -0
  54. data/lib/smart_rag/core/media_extractors.rb +140 -0
  55. data/lib/smart_rag/core/media_job_queue.rb +353 -0
  56. data/lib/smart_rag/core/media_metadata_extractor.rb +188 -0
  57. data/lib/smart_rag/core/media_object_registry.rb +79 -0
  58. data/lib/smart_rag/core/media_processor.rb +228 -0
  59. data/lib/smart_rag/core/media_safety_policy.rb +61 -0
  60. data/lib/smart_rag/core/s3_content_store.rb +78 -0
  61. data/lib/smart_rag/core/transcript_normalizer.rb +44 -0
  62. data/lib/smart_rag/core/video_semantic_extractor.rb +130 -0
  63. data/lib/smart_rag/http_access_policy.rb +86 -0
  64. data/lib/smart_rag/http_app.rb +188 -0
  65. data/lib/smart_rag/models/embedding.rb +1 -1
  66. data/lib/smart_rag/models/research_topic.rb +1 -1
  67. data/lib/smart_rag/models/research_topic_section.rb +5 -0
  68. data/lib/smart_rag/models/research_topic_tag.rb +5 -0
  69. data/lib/smart_rag/models/search_log.rb +1 -1
  70. data/lib/smart_rag/models/section_fts.rb +5 -0
  71. data/lib/smart_rag/models/section_tag.rb +5 -0
  72. data/lib/smart_rag/models/source_document.rb +1 -1
  73. data/lib/smart_rag/models/source_section.rb +1 -1
  74. data/lib/smart_rag/models/tag.rb +1 -1
  75. data/lib/smart_rag/models/text_search_config.rb +5 -0
  76. data/lib/smart_rag/retrieve.rb +72 -1
  77. data/lib/smart_rag/services/embedding_service.rb +1 -1
  78. data/lib/smart_rag/services/fulltext_search_service.rb +11 -13
  79. data/lib/smart_rag/services/hybrid_search_service.rb +15 -11
  80. data/lib/smart_rag/services/summarization_service.rb +1 -1
  81. data/lib/smart_rag/services/tag_service.rb +1 -1
  82. data/lib/smart_rag/version.rb +1 -1
  83. data/lib/smart_rag.rb +264 -30
  84. data/patch_language.rb +27 -0
  85. data/requirements.md +376 -0
  86. data/source_documents_export.json +11072 -0
  87. data/todo.md +931 -0
  88. data/workers/analyze_content.rb +6 -2
  89. data/workers/get_embedding.rb +1 -1
  90. metadata +151 -38
data/USAGE_EXAMPLES.md ADDED
@@ -0,0 +1,1002 @@
1
+ # SmartRAG Usage Examples and Best Practices
2
+
3
+ This guide provides practical examples and best practices for using SmartRAG in your applications.
4
+
5
+ ## Table of Contents
6
+
7
+ 1. [Quick Start](#quick-start)
8
+ 2. [Document Management](#document-management)
9
+ 3. [Search Operations](#search-operations)
10
+ 4. [Research Topic Management](#research-topic-management)
11
+ 5. [Tag Management](#tag-management)
12
+ 6. [Advanced Usage Patterns](#advanced-usage-patterns)
13
+ 7. [Performance Best Practices](#performance-best-practices)
14
+ 8. [Error Handling](#error-handling)
15
+ 9. [Common Patterns](#common-patterns)
16
+
17
+ ## Quick Start
18
+
19
+ ### Basic Setup
20
+
21
+ ```ruby
22
+ require 'smart_rag'
23
+ require 'logger'
24
+
25
+ # Initialize configuration
26
+ config = {
27
+ database: {
28
+ adapter: 'postgresql',
29
+ host: ENV['SMARTRAG_DB_HOST'] || 'localhost',
30
+ database: ENV['SMARTRAG_DB_NAME'] || 'smart_rag_development',
31
+ user: ENV['SMARTRAG_DB_USER'] || 'smart_rag_user',
32
+ password: ENV['SMARTRAG_DB_PASSWORD']
33
+ },
34
+ llm: {
35
+ provider: 'openai',
36
+ api_key: ENV['OPENAI_API_KEY']
37
+ }
38
+ }
39
+
40
+ # Create SmartRAG instance
41
+ smart_rag = SmartRAG::SmartRAG.new(config)
42
+ smart_rag.logger = Logger.new(STDOUT)
43
+ smart_rag.logger.level = Logger::INFO
44
+
45
+ # Test the connection
46
+ puts "✓ SmartRAG initialized successfully"
47
+ stats = smart_rag.statistics
48
+ puts "✓ Database connected: #{stats[:document_count]} documents"
49
+ ```
50
+
51
+ ### First Document Addition
52
+
53
+ ```ruby
54
+ # Add your first document
55
+ result = smart_rag.add_document(
56
+ 'https://example.com/ai-article.pdf',
57
+ title: 'Introduction to AI',
58
+ generate_embeddings: true,
59
+ generate_tags: true
60
+ )
61
+
62
+ puts "✓ Document added: ID #{result[:document_id]}"
63
+ puts "✓ Sections created: #{result[:section_count]}"
64
+ ```
65
+
66
+ ### First Search
67
+
68
+ ```ruby
69
+ # Perform your first search
70
+ results = smart_rag.search(
71
+ 'machine learning algorithms',
72
+ search_type: 'hybrid',
73
+ limit: 5,
74
+ include_content: true
75
+ )
76
+
77
+ puts "\nSearch Results:"
78
+ results[:results].each_with_index do |result, i|
79
+ puts "#{i + 1}. #{result[:section_title]} (score: #{result[:combined_score].round(3)})"
80
+ puts " #{result[:content][0..150]}..."
81
+ end
82
+ ```
83
+
84
+ ## Document Management
85
+
86
+ ### Adding Documents from Various Sources
87
+
88
+ ```ruby
89
+ # From a local file
90
+ smart_rag.add_document(
91
+ '/path/to/document.pdf',
92
+ title: 'Research Paper 2024',
93
+ generate_embeddings: true
94
+ )
95
+
96
+ # From a URL
97
+ smart_rag.add_document(
98
+ 'https://arxiv.org/abs/2301.00001',
99
+ generate_embeddings: true,
100
+ generate_tags: true,
101
+ tags: ['research', 'AI']
102
+ )
103
+
104
+ # With custom metadata
105
+ smart_rag.add_document(
106
+ '/path/to/report.docx',
107
+ title: 'Q3 Financial Report',
108
+ generate_embeddings: false, # Skip embeddings for non-technical docs
109
+ metadata: {
110
+ department: 'Finance',
111
+ year: 2024,
112
+ confidential: true
113
+ }
114
+ )
115
+ ```
116
+
117
+ ### Batch Document Processing
118
+
119
+ ```ruby
120
+ # Process multiple documents efficiently
121
+ documents = [
122
+ { path: '/docs/paper1.pdf', tags: ['AI'] },
123
+ { path: '/docs/paper2.pdf', tags: ['ML'] },
124
+ { path: '/docs/paper3.pdf', tags: ['NLP'] }
125
+ ]
126
+
127
+ # Sequential processing
128
+ documents.each do |doc|
129
+ begin
130
+ result = smart_rag.add_document(
131
+ doc[:path],
132
+ generate_embeddings: true,
133
+ tags: doc[:tags]
134
+ )
135
+ puts "✓ Processed: #{doc[:path]}"
136
+ rescue => e
137
+ puts "✗ Failed: #{doc[:path]} - #{e.message}"
138
+ end
139
+ end
140
+
141
+ # Parallel processing for better performance
142
+ require 'concurrent'
143
+
144
+ pool = Concurrent::FixedThreadPool.new(5)
145
+ documents.each do |doc|
146
+ pool.post do
147
+ begin
148
+ smart_rag.add_document(
149
+ doc[:path],
150
+ generate_embeddings: true,
151
+ tags: doc[:tags]
152
+ )
153
+ puts "✓ Processed: #{doc[:path]}"
154
+ rescue => e
155
+ puts "✗ Failed: #{doc[:path]} - #{e.message}"
156
+ end
157
+ end
158
+ end
159
+
160
+ pool.shutdown
161
+ pool.wait_for_termination
162
+ ```
163
+
164
+ ### Document Management Operations
165
+
166
+ ```ruby
167
+ # List documents with pagination
168
+ docs_page_1 = smart_rag.list_documents(page: 1, per_page: 20)
169
+ docs_page_2 = smart_rag.list_documents(page: 2, per_page: 20)
170
+
171
+ # Search for specific documents
172
+ ml_docs = smart_rag.list_documents(search: 'machine learning')
173
+
174
+ # Get document details
175
+ doc = smart_rag.get_document(123)
176
+ puts "Title: #{doc[:title]}"
177
+ puts "Sections: #{doc[:section_count]}"
178
+ puts "Created: #{doc[:created_at]}"
179
+
180
+ # Remove a document
181
+ result = smart_rag.remove_document(123)
182
+ puts "Deleted sections: #{result[:deleted_sections]}"
183
+ puts "Deleted embeddings: #{result[:deleted_embeddings]}"
184
+ ```
185
+
186
+ ## Search Operations
187
+
188
+ ### Hybrid Search (Default)
189
+
190
+ Hybrid search combines vector and full-text search for optimal results.
191
+
192
+ ```ruby
193
+ # Basic hybrid search
194
+ results = smart_rag.search(
195
+ 'deep learning applications in healthcare',
196
+ search_type: 'hybrid',
197
+ limit: 10,
198
+ alpha: 0.7 # Weight for vector search (0.0 = pure text, 1.0 = pure vector)
199
+ )
200
+
201
+ # With content and metadata
202
+ results = smart_rag.search(
203
+ 'natural language processing',
204
+ search_type: 'hybrid',
205
+ limit: 5,
206
+ include_content: true,
207
+ include_metadata: true
208
+ )
209
+
210
+ # Search with filters
211
+ results = smart_rag.search(
212
+ 'artificial intelligence',
213
+ search_type: 'hybrid',
214
+ limit: 10,
215
+ filters: {
216
+ document_ids: [1, 2, 3], # Search only in specific documents
217
+ tag_ids: [4, 5] # Filter by tags
218
+ }
219
+ )
220
+ ```
221
+
222
+ ### Vector Search
223
+
224
+ Useful for semantic similarity searches.
225
+
226
+ ```ruby
227
+ # Pure vector search
228
+ results = smart_rag.vector_search(
229
+ 'neural network architectures',
230
+ limit: 5
231
+ )
232
+
233
+ # Vector search with tag boosting
234
+ results = smart_rag.vector_search(
235
+ 'transformer models',
236
+ limit: 10,
237
+ tag_boost_weight: 0.1 # Boost results with matching tags
238
+ )
239
+ ```
240
+
241
+ ### Full-Text Search
242
+
243
+ Best for exact keyword matching and boolean queries.
244
+
245
+ ```ruby
246
+ # Basic full-text search
247
+ results = smart_rag.fulltext_search(
248
+ 'convolutional neural networks',
249
+ limit: 5
250
+ )
251
+
252
+ # Advanced boolean query
253
+ results = smart_rag.fulltext_search(
254
+ 'artificial AND (intelligence OR learning) AND NOT robotics',
255
+ limit: 10
256
+ )
257
+
258
+ # Phrase search
259
+ results = smart_rag.fulltext_search(
260
+ '"deep reinforcement learning"',
261
+ limit: 5
262
+ )
263
+ ```
264
+
265
+ ### Multi-language Search
266
+
267
+ SmartRAG automatically detects and handles multiple languages.
268
+
269
+ ```ruby
270
+ # Chinese search
271
+ results = smart_rag.search('人工智能应用', language: 'zh_cn')
272
+
273
+ # Japanese search
274
+ results = smart_rag.search('機械学習アルゴリズム', language: 'ja')
275
+
276
+ # Korean search
277
+ results = smart_rag.search('딥러닝 모델', language: 'ko')
278
+
279
+ # Mixed language search (auto-detect)
280
+ results = smart_rag.search('AI和机器学习的发展', language: 'auto')
281
+ ```
282
+
283
+ ## Research Topic Management
284
+
285
+ ### Creating and Organizing Topics
286
+
287
+ ```ruby
288
+ # Create a research topic
289
+ topic = smart_rag.create_topic(
290
+ 'AI in Healthcare',
291
+ 'Applications of artificial intelligence in medical diagnosis and treatment',
292
+ tags: ['AI', 'healthcare', 'medicine', 'diagnosis'],
293
+ document_ids: [1, 2, 3] # Associate existing documents
294
+ )
295
+
296
+ # Create nested topic structure
297
+ parent_topic = smart_rag.create_topic(
298
+ 'Machine Learning',
299
+ 'Fundamental ML concepts and algorithms'
300
+ )
301
+
302
+ child_topic = smart_rag.create_topic(
303
+ 'Deep Learning',
304
+ 'Neural network based learning',
305
+ tags: ['neural_networks', 'deep_learning']
306
+ # Could link to parent if hierarchical topics are supported
307
+ )
308
+ ```
309
+
310
+ ### Managing Topic Content
311
+
312
+ ```ruby
313
+ # Add documents to a topic
314
+ topic_id = 456
315
+ document_id = 123
316
+
317
+ result = smart_rag.add_document_to_topic(topic_id, document_id)
318
+ puts "Added #{result[:added_sections]} sections to topic"
319
+
320
+ # Get topic recommendations
321
+ recommendations = smart_rag.get_topic_recommendations(topic_id, limit: 10)
322
+
323
+ recommendations[:recommendations].each do |rec|
324
+ puts "Recommended: #{rec[:section_title]}"
325
+ puts " Matching tags: #{rec[:matching_tags]}"
326
+ puts " Score: #{rec[:relevance_score]}"
327
+ end
328
+
329
+ # List all topics
330
+ topics = smart_rag.list_topics(page: 1, per_page: 20)
331
+ topics[:topics].each do |topic|
332
+ puts "#{topic[:title]} (#{topic[:document_count]} documents)"
333
+ end
334
+ ```
335
+
336
+ ## Tag Management
337
+
338
+ ### Automatic Tag Generation
339
+
340
+ ```ruby
341
+ # Generate tags for text
342
+ text = """
343
+ Machine learning is a subset of artificial intelligence that enables systems
344
+ to learn and improve from experience without being explicitly programmed.
345
+ It focuses on developing computer programs that can access data and use it
346
+ to learn for themselves.
347
+ """
348
+
349
+ tags = smart_rag.generate_tags(text, topic: 'AI Introduction')
350
+ puts "Categories: #{tags[:categories].join(', ')}"
351
+ puts "Content tags: #{tags[:content_tags].join(', ')}"
352
+
353
+ # Batch generate tags for document sections
354
+ document = smart_rag.get_document(1)
355
+ sections = document[:sections]
356
+
357
+ tags_by_section = {}
358
+ sections.each do |section|
359
+ tags = smart_rag.generate_tags(
360
+ section[:content],
361
+ topic: document[:title],
362
+ max_tags: 5
363
+ )
364
+ tags_by_section[section[:id]] = tags
365
+ end
366
+ ```
367
+
368
+ ### Manual Tag Management
369
+
370
+ ```ruby
371
+ # Create hierarchical tags
372
+ tag_service = SmartRAG::Services::TagService.new
373
+
374
+ hierarchy = {
375
+ "Technology" => {
376
+ "AI" => ["Machine Learning", "Deep Learning", "Neural Networks"],
377
+ "Programming" => ["Python", "Ruby", "JavaScript"]
378
+ }
379
+ }
380
+
381
+ created_tags = tag_service.create_hierarchy(hierarchy)
382
+
383
+ # Associate tags with content
384
+ tag = SmartRAG::Models::Tag.find_or_create("machine_learning")
385
+ section = SmartRAG::Models::SourceSection[1]
386
+
387
+ # Add tag to section
388
+ section.add_tag(tag)
389
+
390
+ # Find content by tag
391
+ ml_sections = tag.sections
392
+ ml_sections.each do |section|
393
+ puts "#{section.section_title}: #{section.content[0..100]}..."
394
+ end
395
+
396
+ # Search for tags
397
+ tags = tag_service.search_tags('learn', limit: 10)
398
+ tags.each { |tag| puts "#{tag.name} (#{tag.section_count} sections)" }
399
+ ```
400
+
401
+ ## Advanced Usage Patterns
402
+
403
+ ### Context-Aware Search
404
+
405
+ ```ruby
406
+ class ContextualSearch
407
+ def initialize(smart_rag)
408
+ @smart_rag = smart_rag
409
+ @search_history = []
410
+ end
411
+
412
+ def search_with_context(query, user_context = {})
413
+ # Enhance query with context
414
+ enhanced_query = enhance_query(query, user_context)
415
+
416
+ # Perform search
417
+ results = @smart_rag.search(
418
+ enhanced_query,
419
+ search_type: 'hybrid',
420
+ limit: 10,
421
+ filters: build_filters(user_context)
422
+ )
423
+
424
+ # Store in history
425
+ @search_history << { query: query, context: user_context, results: results }
426
+
427
+ results
428
+ end
429
+
430
+ private
431
+
432
+ def enhance_query(query, context)
433
+ # Add context-specific terms
434
+ case context[:domain]
435
+ when 'healthcare'
436
+ "#{query} medical health clinical"
437
+ when 'finance'
438
+ "#{query} financial economic banking"
439
+ else
440
+ query
441
+ end
442
+ end
443
+
444
+ def build_filters(context)
445
+ filters = {}
446
+ filters[:document_ids] = context[:document_ids] if context[:document_ids]
447
+ filters[:tag_ids] = context[:preferred_tags] if context[:preferred_tags]
448
+ filters
449
+ end
450
+ end
451
+
452
+ # Usage
453
+ contextual_search = ContextualSearch.new(smart_rag)
454
+ results = contextual_search.search_with_context(
455
+ 'risk assessment',
456
+ user_context: {
457
+ domain: 'finance',
458
+ document_ids: [1, 2, 3],
459
+ preferred_tags: [4, 5]
460
+ }
461
+ )
462
+ ```
463
+
464
+ ### Search Result Processing Pipeline
465
+
466
+ ```ruby
467
+ class SearchPipeline
468
+ def initialize(smart_rag)
469
+ @smart_rag = smart_rag
470
+ @processors = []
471
+ end
472
+
473
+ def add_processor(&block)
474
+ @processors << block
475
+ self
476
+ end
477
+
478
+ def search(query, options = {})
479
+ # Initial search
480
+ results = @smart_rag.search(query, options)
481
+
482
+ # Process through pipeline
483
+ @processors.each do |processor|
484
+ results = processor.call(results, query, options)
485
+ end
486
+
487
+ results
488
+ end
489
+ end
490
+
491
+ # Create pipeline with processors
492
+ pipeline = SearchPipeline.new(smart_rag)
493
+
494
+ # Add relevance scoring
495
+ pipeline.add_processor do |results, query, options|
496
+ results[:results].each do |result|
497
+ result[:relevance_score] = calculate_relevance(result, query)
498
+ end
499
+ results
500
+ end
501
+
502
+ # Add result filtering
503
+ pipeline.add_processor do |results, query, options|
504
+ min_score = options[:min_score] || 0.5
505
+ results[:results].select! { |r| r[:relevance_score] >= min_score }
506
+ results[:metadata][:filtered_count] = results[:results].length
507
+ results
508
+ end
509
+
510
+ # Use pipeline
511
+ results = pipeline.search(
512
+ 'neural networks',
513
+ min_score: 0.7,
514
+ limit: 20
515
+ )
516
+ ```
517
+
518
+ ### Caching Search Results
519
+
520
+ ```ruby
521
+ require 'redis'
522
+
523
+ class CachedSmartRAG
524
+ def initialize(smart_rag, redis_client)
525
+ @smart_rag = smart_rag
526
+ @redis = redis_client
527
+ @cache_ttl = 3600 # 1 hour
528
+ end
529
+
530
+ def search(query, options = {})
531
+ # Create cache key
532
+ cache_key = create_cache_key(query, options)
533
+
534
+ # Try to get from cache
535
+ cached = @redis.get(cache_key)
536
+ if cached
537
+ puts "Cache hit for: #{query}"
538
+ return JSON.parse(cached, symbolize_names: true)
539
+ end
540
+
541
+ # Perform search
542
+ results = @smart_rag.search(query, options)
543
+
544
+ # Store in cache
545
+ @redis.setex(cache_key, @cache_ttl, results.to_json)
546
+
547
+ puts "Cache miss for: #{query}"
548
+ results
549
+ end
550
+
551
+ private
552
+
553
+ def create_cache_key(query, options)
554
+ key_parts = [query, options.sort].flatten.join(':')
555
+ "search:#{Digest::MD5.hexdigest(key_parts)}"
556
+ end
557
+ end
558
+
559
+ # Usage
560
+ redis = Redis.new
561
+ $cached_rag = CachedSmartRAG.new(smart_rag, redis)
562
+
563
+ # First search - cache miss
564
+ results1 = $cached_rag.search('deep learning', limit: 10)
565
+
566
+ # Second search - cache hit
567
+ results2 = $cached_rag.search('deep learning', limit: 10)
568
+ ```
569
+
570
+ ### Building a Q&A System
571
+
572
+ ```ruby
573
+ class QA_system
574
+ def initialize(smart_rag)
575
+ @smart_rag = smart_rag
576
+ end
577
+
578
+ def answer(question, options = {})
579
+ # Search for relevant information
580
+ search_results = @smart_rag.search(
581
+ question,
582
+ search_type: 'hybrid',
583
+ limit: options[:context_limit] || 5,
584
+ include_content: true
585
+ )
586
+
587
+ # Generate answer based on search results (requires LLM integration)
588
+ answer = generate_answer(question, search_results[:results])
589
+
590
+ {
591
+ question: question,
592
+ answer: answer,
593
+ sources: extract_sources(search_results[:results]),
594
+ confidence: calculate_confidence(search_results[:results])
595
+ }
596
+ end
597
+
598
+ private
599
+
600
+ def generate_answer(question, results)
601
+ return "I don't have enough information to answer this question." if results.empty?
602
+
603
+ # Combine relevant content
604
+ context = results.map { |r| r[:content] }.join("\n\n---\n\n")
605
+
606
+ # Here you would call an LLM API to generate the answer
607
+ # This is a simplified version
608
+ "Based on the available information: #{context[0..500]}..."
609
+ end
610
+
611
+ def extract_sources(results)
612
+ results.map do |result|
613
+ {
614
+ section_id: result[:section_id],
615
+ title: result[:section_title],
616
+ score: result[:combined_score]
617
+ }
618
+ end
619
+ end
620
+
621
+ def calculate_confidence(results)
622
+ return 0.0 if results.empty?
623
+
624
+ # Simple confidence based on top result score
625
+ [results.first[:combined_score], 1.0].min
626
+ end
627
+ end
628
+
629
+ # Usage
630
+ qa = QA_system.new(smart_rag)
631
+ response = qa.answer(
632
+ 'What are the applications of transformers in NLP?',
633
+ context_limit: 3
634
+ )
635
+
636
+ puts "Answer: #{response[:answer]}"
637
+ puts "Confidence: #{(response[:confidence] * 100).round(1)}%"
638
+ puts "Sources:"
639
+ response[:sources].each do |source|
640
+ puts " - #{source[:title]} (ID: #{source[:section_id]})"
641
+ end
642
+ ```
643
+
644
+ ## Performance Best Practices
645
+
646
+ ### 1. Batch Operations
647
+
648
+ ```ruby
649
+ # Instead of individual operations
650
+ bad_practice = documents.map do |doc|
651
+ smart_rag.add_document(doc[:path], generate_embeddings: true)
652
+ end
653
+
654
+ # Use batch processing
655
+ good_practice = documents.each_slice(10) do |batch|
656
+ # Process batch in parallel
657
+ batch.map do |doc|
658
+ Concurrent::Promises.future do
659
+ smart_rag.add_document(doc[:path], generate_embeddings: true)
660
+ end
661
+ end.map(&:value)
662
+ end
663
+ ```
664
+
665
+ ### 2. Connection Pooling
666
+
667
+ ```ruby
668
+ # Configure database connection pool
669
+ config = {
670
+ database: {
671
+ adapter: 'postgresql',
672
+ host: 'localhost',
673
+ database: 'smart_rag',
674
+ user: 'user',
675
+ password: 'pass',
676
+ pool: 25, # Increase pool size
677
+ timeout: 5000, # Connection timeout
678
+ max_connections: 100 # Maximum connections
679
+ }
680
+ }
681
+ ```
682
+
683
+ ### 3. Efficient Searching
684
+
685
+ ```ruby
686
+ # Use appropriate search types for queries
687
+ # For exact matches
688
+ text_results = smart_rag.fulltext_search('error code 404', limit: 5)
689
+
690
+ # For conceptual similarity
691
+ vector_results = smart_rag.vector_search('debugging techniques', limit: 5)
692
+
693
+ # For general queries
694
+ hybrid_results = smart_rag.search('how to fix bugs', search_type: 'hybrid')
695
+
696
+ # Adjust alpha based on use case
697
+ # - Technical/keyword-heavy: lower alpha (0.3-0.5)
698
+ # - Conceptual/exploratory: higher alpha (0.7-0.9)
699
+ ```
700
+
701
+ ### 4. Caching Strategies
702
+
703
+ ```ruby
704
+ # Cache embeddings for repeated content
705
+ class EmbeddingCache
706
+ def initialize
707
+ @cache = {}
708
+ end
709
+
710
+ def get_embedding(text)
711
+ hash = Digest::MD5.hexdigest(text)
712
+ @cache[hash] ||= generate_embedding(text)
713
+ end
714
+ end
715
+
716
+ # Cache search results
717
+ class SearchCache
718
+ def initialize(redis, ttl: 3600)
719
+ @redis = redis
720
+ @ttl = ttl
721
+ end
722
+
723
+ def fetch(query, options = {}, &block)
724
+ key = cache_key(query, options)
725
+
726
+ if result = @redis.get(key)
727
+ JSON.parse(result, symbolize_names: true)
728
+ else
729
+ result = block.call
730
+ @redis.setex(key, @ttl, result.to_json)
731
+ result
732
+ end
733
+ end
734
+ end
735
+ ```
736
+
737
+ ### 5. Database Optimization
738
+
739
+ ```sql
740
+ -- Create optimized indexes
741
+ CREATE INDEX CONCURRENTLY idx_section_fts_content
742
+ ON section_fts USING gin(to_tsvector('english', content));
743
+
744
+ CREATE INDEX CONCURRENTLY idx_embeddings_vector
745
+ ON embeddings USING ivfflat (vector vector_cosine_ops)
746
+ WITH (lists = 100);
747
+
748
+ -- Monitor and optimize slow queries
749
+ EXPLAIN ANALYZE
750
+ SELECT * FROM hybrid_search('machine learning', 10);
751
+ ```
752
+
753
+ ## Error Handling
754
+
755
+ ### Comprehensive Error Handling
756
+
757
+ ```ruby
758
+ begin
759
+ result = smart_rag.add_document(
760
+ '/path/to/document.pdf',
761
+ generate_embeddings: true
762
+ )
763
+ rescue SmartRAG::Errors::ArgumentError => e
764
+ puts "Invalid arguments: #{e.message}"
765
+ rescue SmartRAG::Errors::DatabaseError => e
766
+ puts "Database error: #{e.message}"
767
+ # Attempt to reconnect or use fallback
768
+ rescue SmartRAG::Errors::EmbeddingGenerationError => e
769
+ puts "Embedding generation failed: #{e.message}"
770
+ # Retry or skip embeddings
771
+ rescue SmartRAG::Errors::DocumentProcessingError => e
772
+ puts "Document processing failed: #{e.message}"
773
+ # Log and continue with next document
774
+ rescue => e
775
+ puts "Unexpected error: #{e.message}"
776
+ # Log for investigation
777
+ end
778
+ ```
779
+
780
+ ### Retry Logic
781
+
782
+ ```ruby
783
+ require 'retriable'
784
+
785
+ class RetryableSmartRAG
786
+ def initialize(smart_rag)
787
+ @smart_rag = smart_rag
788
+ end
789
+
790
+ def add_document(path, options = {})
791
+ Retriable.retriable(
792
+ on: [SmartRAG::Errors::EmbeddingGenerationError],
793
+ tries: 3,
794
+ base_interval: 1,
795
+ multiplier: 2
796
+ ) do
797
+ @smart_rag.add_document(path, options)
798
+ end
799
+ end
800
+
801
+ def search(query, options = {})
802
+ Retriable.retriable(
803
+ on: [SmartRAG::Errors::DatabaseError],
804
+ tries: 3,
805
+ base_interval: 0.5
806
+ ) do
807
+ @smart_rag.search(query, options)
808
+ end
809
+ end
810
+ end
811
+ ```
812
+
813
+ ## Common Patterns
814
+
815
+ ### Pattern 1: Document Processing Pipeline
816
+
817
+ ```ruby
818
+ class DocumentPipeline
819
+ def initialize(smart_rag)
820
+ @smart_rag = smart_rag
821
+ end
822
+
823
+ def process(files, options = {})
824
+ results = []
825
+
826
+ files.each do |file|
827
+ begin
828
+ # Step 1: Add document
829
+ doc_result = @smart_rag.add_document(
830
+ file,
831
+ generate_embeddings: false # Delay embedding generation
832
+ )
833
+
834
+ # Step 2: Generate tags
835
+ tags = @smart_rag.generate_tags(
836
+ extract_text(file),
837
+ topic: options[:topic]
838
+ )
839
+
840
+ # Step 3: Apply tags
841
+ if doc_result[:document_id]
842
+ document = SmartRAG::Models::SourceDocument[doc_result[:document_id]]
843
+ sections = document.sections
844
+
845
+ sections.each do |section|
846
+ tag_objects = tags[:content_tags].map do |tag_name|
847
+ SmartRAG::Models::Tag.find_or_create(tag_name)
848
+ end
849
+
850
+ section.add_tag(*tag_objects)
851
+ end
852
+
853
+ # Step 4: Generate embeddings (batch)
854
+ document.sections.each do |section|
855
+ embedding = @smart_rag.generate_embedding(section.content)
856
+ store_embedding(section.id, embedding)
857
+ end
858
+ end
859
+
860
+ results << { success: true, file: file, document_id: doc_result[:document_id] }
861
+ rescue => e
862
+ results << { success: false, file: file, error: e.message }
863
+ end
864
+ end
865
+
866
+ results
867
+ end
868
+ end
869
+ ```
870
+
871
+ ### Pattern 2: Incremental Indexing
872
+
873
+ ```ruby
874
+ class IncrementalIndexer
875
+ def initialize(smart_rag)
876
+ @smart_rag = smart_rag
877
+ end
878
+
879
+ def index_new_documents(source_dir, last_check = nil)
880
+ # Find new or modified documents
881
+ pattern = File.join(source_dir, '**/*.pdf')
882
+ documents = Dir.glob(pattern)
883
+
884
+ if last_check
885
+ documents.select! { |doc| File.mtime(doc) > last_check }
886
+ end
887
+
888
+ # Process in batches
889
+ documents.each_slice(10) do |batch|
890
+ batch_results = process_batch(batch)
891
+ log_results(batch_results)
892
+ end
893
+ end
894
+
895
+ private
896
+
897
+ def process_batch(files)
898
+ files.map do |file|
899
+ begin
900
+ result = @smart_rag.add_document(file, generate_embeddings: true)
901
+ { file: file, success: true, document_id: result[:document_id] }
902
+ rescue => e
903
+ { file: file, success: false, error: e.message }
904
+ end
905
+ end
906
+ end
907
+ end
908
+ ```
909
+
910
+ ### Pattern 3: Search Analytics
911
+
912
+ ```ruby
913
+ class SearchAnalytics
914
+ def initialize(smart_rag)
915
+ @smart_rag = smart_rag
916
+ end
917
+
918
+ def analyze_search_patterns(days = 30)
919
+ logs = smart_rag.search_logs(limit: 1000)
920
+
921
+ analytics = {
922
+ total_searches: logs.length,
923
+ avg_execution_time: logs.sum { |l| l[:execution_time_ms] } / logs.length,
924
+ popular_queries: popular_queries(logs),
925
+ failed_searches: logs.count { |l| l[:results_count] == 0 },
926
+ trend_analysis: trend_analysis(logs, days)
927
+ }
928
+ end
929
+
930
+ def identify_content_gaps(logs)
931
+ no_result_queries = logs.select { |l| l[:results_count] == 0 }
932
+
933
+ # Group similar queries
934
+ clusters = cluster_queries(no_result_queries)
935
+
936
+ # Identify topics needing more content
937
+ clusters.map do |cluster|
938
+ {
939
+ topic: cluster[:topic],
940
+ query_count: cluster[:queries].length,
941
+ sample_queries: cluster[:queries].first(3)
942
+ }
943
+ end
944
+ end
945
+ end
946
+ ```
947
+
948
+ ### Pattern 4: Multi-tenant Applications
949
+
950
+ ```ruby
951
+ class MultiTenantSmartRAG
952
+ def initialize(smart_rag)
953
+ @smart_rag = smart_rag
954
+ end
955
+
956
+ def add_document(tenant_id, path, options = {})
957
+ # Add tenant isolation
958
+ options[:metadata] ||= {}
959
+ options[:metadata][:tenant_id] = tenant_id
960
+
961
+ @smart_rag.add_document(path, options)
962
+ end
963
+
964
+ def search(tenant_id, query, options = {})
965
+ # Filter by tenant
966
+ options[:filters] ||= {}
967
+ options[:filters][:metadata] = { tenant_id: tenant_id }
968
+
969
+ @smart_rag.search(query, options)
970
+ end
971
+
972
+ def get_statistics(tenant_id)
973
+ # Get tenant-specific stats
974
+ @smart_rag.statistics(tenant_id: tenant_id)
975
+ end
976
+ end
977
+ ```
978
+
979
+ ## Summary
980
+
981
+ This guide has covered:
982
+
983
+ 1. **Quick Start** - Basic setup and first operations
984
+ 2. **Document Management** - Adding, organizing, and managing documents
985
+ 3. **Search Operations** - Various search types and advanced filtering
986
+ 4. **Research Topics** - Organizing content into thematic collections
987
+ 5. **Tag Management** - Automatic and manual tagging strategies
988
+ 6. **Advanced Patterns** - Production-ready implementations
989
+ 7. **Performance** - Best practices for optimal performance
990
+ 8. **Error Handling** - Robust error management strategies
991
+ 9. **Common Patterns** - Reusable solutions for typical scenarios
992
+
993
+ For more information, see:
994
+ - [API Documentation](API_DOCUMENTATION.md) - Complete API reference
995
+ - [Performance Guide](PERFORMANCE_GUIDE.md) - Performance optimization details
996
+ - [Migration Guide](MIGRATION_GUIDE.md) - Version upgrade instructions
997
+
998
+ ## Support
999
+
1000
+ - GitHub Issues: https://github.com/your-org/smart_rag/issues
1001
+ - Documentation Issues: Report any errors or inconsistencies in examples
1002
+ - Community Forum: Share your usage patterns and learn from others