smart_rag 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (90) hide show
  1. checksums.yaml +4 -4
  2. data/.env.example +252 -0
  3. data/.rspec +2 -0
  4. data/AGENTS.md +33 -0
  5. data/API_DOCUMENTATION.md +828 -0
  6. data/CHANGELOG.md +16 -1
  7. data/ER-diagram.mmd +144 -0
  8. data/Gemfile +50 -0
  9. data/Gemfile.lock +398 -0
  10. data/Hybrid_Reranking.md +171 -0
  11. data/README.en.md +420 -28
  12. data/README.md +534 -63
  13. data/Rakefile +268 -0
  14. data/SETUP_GUIDE.md +650 -0
  15. data/SmartChunking.md +180 -0
  16. data/USAGE_EXAMPLES.md +1002 -0
  17. data/config/llm_config.yml +4 -2
  18. data/config/smart_rag.yml +45 -1
  19. data/config.ru +15 -0
  20. data/db/migrations/006_create_text_search_configs.rb +3 -2
  21. data/db/migrations/008_create_embeddings.rb +5 -4
  22. data/db/migrations/012_add_metadata_to_source_sections.rb +11 -0
  23. data/db/migrations/013_create_media_jobs.rb +25 -0
  24. data/db/migrations/014_add_media_job_operations_indexes.rb +11 -0
  25. data/db/migrations/015_add_media_leases_and_objects.rb +80 -0
  26. data/db/migrations/016_add_document_principals_and_staging_references.rb +38 -0
  27. data/db/migrations/017_add_media_job_request_fingerprint.rb +48 -0
  28. data/db/seeds/text_search_configs.sql +3 -3
  29. data/design.md +1057 -0
  30. data/docs/API_DOCUMENTATION.md +838 -0
  31. data/docs/DOCUMENTATION_INDEX.en.md +60 -0
  32. data/docs/DOCUMENTATION_INDEX.md +65 -0
  33. data/docs/FIX_SUMMARY.md +256 -0
  34. data/docs/FIX_SUMMARY_COMPLETE.md +273 -0
  35. data/docs/Hybrid_Reranking.md +171 -0
  36. data/docs/MIGRATION_GUIDE.md +151 -0
  37. data/docs/PERFORMANCE_GUIDE.md +58 -0
  38. data/docs/SETUP_GUIDE.md +659 -0
  39. data/docs/SmartChunking.md +180 -0
  40. data/docs/USAGE_EXAMPLES.md +1008 -0
  41. data/docs/design.md +1057 -0
  42. data/docs/evidence_pack.md +211 -0
  43. data/docs/requirements.md +376 -0
  44. data/docs/retrieval_plan.md +251 -0
  45. data/docs/smartrag_improvement_plan.md +201 -0
  46. data/docs/smartrag_refactor.md +216 -0
  47. data/docs/todo.md +931 -0
  48. data/examples/common.rb +1 -1
  49. data/exe/smart-rag-db +163 -0
  50. data/exe/smart-rag-media-worker +34 -0
  51. data/lib/smart_rag/config.rb +12 -0
  52. data/lib/smart_rag/core/document_processor.rb +80 -16
  53. data/lib/smart_rag/core/local_content_store.rb +51 -0
  54. data/lib/smart_rag/core/media_extractors.rb +140 -0
  55. data/lib/smart_rag/core/media_job_queue.rb +353 -0
  56. data/lib/smart_rag/core/media_metadata_extractor.rb +188 -0
  57. data/lib/smart_rag/core/media_object_registry.rb +79 -0
  58. data/lib/smart_rag/core/media_processor.rb +228 -0
  59. data/lib/smart_rag/core/media_safety_policy.rb +61 -0
  60. data/lib/smart_rag/core/s3_content_store.rb +78 -0
  61. data/lib/smart_rag/core/transcript_normalizer.rb +44 -0
  62. data/lib/smart_rag/core/video_semantic_extractor.rb +130 -0
  63. data/lib/smart_rag/http_access_policy.rb +86 -0
  64. data/lib/smart_rag/http_app.rb +188 -0
  65. data/lib/smart_rag/models/embedding.rb +1 -1
  66. data/lib/smart_rag/models/research_topic.rb +1 -1
  67. data/lib/smart_rag/models/research_topic_section.rb +5 -0
  68. data/lib/smart_rag/models/research_topic_tag.rb +5 -0
  69. data/lib/smart_rag/models/search_log.rb +1 -1
  70. data/lib/smart_rag/models/section_fts.rb +5 -0
  71. data/lib/smart_rag/models/section_tag.rb +5 -0
  72. data/lib/smart_rag/models/source_document.rb +1 -1
  73. data/lib/smart_rag/models/source_section.rb +1 -1
  74. data/lib/smart_rag/models/tag.rb +1 -1
  75. data/lib/smart_rag/models/text_search_config.rb +5 -0
  76. data/lib/smart_rag/retrieve.rb +72 -1
  77. data/lib/smart_rag/services/embedding_service.rb +1 -1
  78. data/lib/smart_rag/services/fulltext_search_service.rb +11 -13
  79. data/lib/smart_rag/services/hybrid_search_service.rb +15 -11
  80. data/lib/smart_rag/services/summarization_service.rb +1 -1
  81. data/lib/smart_rag/services/tag_service.rb +1 -1
  82. data/lib/smart_rag/version.rb +1 -1
  83. data/lib/smart_rag.rb +264 -30
  84. data/patch_language.rb +27 -0
  85. data/requirements.md +376 -0
  86. data/source_documents_export.json +11072 -0
  87. data/todo.md +931 -0
  88. data/workers/analyze_content.rb +6 -2
  89. data/workers/get_embedding.rb +1 -1
  90. metadata +151 -38
@@ -0,0 +1,1008 @@
1
+ # SmartRAG Usage Examples and Best Practices
2
+
3
+ This guide provides practical examples and best practices for using SmartRAG in your applications.
4
+
5
+ ## Table of Contents
6
+
7
+ 1. [Quick Start](#quick-start)
8
+ 2. [Document Management](#document-management)
9
+ 3. [Search Operations](#search-operations)
10
+ 4. [Research Topic Management](#research-topic-management)
11
+ 5. [Tag Management](#tag-management)
12
+ 6. [Advanced Usage Patterns](#advanced-usage-patterns)
13
+ 7. [Performance Best Practices](#performance-best-practices)
14
+ 8. [Error Handling](#error-handling)
15
+ 9. [Common Patterns](#common-patterns)
16
+
17
+ ## Quick Start
18
+
19
+ ### Basic Setup
20
+
21
+ ```ruby
22
+ require 'smart_rag'
23
+ require 'logger'
24
+
25
+ # Initialize configuration
26
+ config = {
27
+ database: {
28
+ adapter: 'postgresql',
29
+ host: ENV['SMARTRAG_DB_HOST'] || 'localhost',
30
+ database: ENV['SMARTRAG_DB_NAME'] || 'smart_rag_development',
31
+ user: ENV['SMARTRAG_DB_USER'] || 'smart_rag_user',
32
+ password: ENV['SMARTRAG_DB_PASSWORD']
33
+ },
34
+ llm: {
35
+ provider: 'openai',
36
+ api_key: ENV['OPENAI_API_KEY']
37
+ }
38
+ }
39
+
40
+ # Create SmartRAG instance
41
+ smart_rag = SmartRAG::SmartRAG.new(config)
42
+ smart_rag.logger = Logger.new(STDOUT)
43
+ smart_rag.logger.level = Logger::INFO
44
+
45
+ # Test the connection
46
+ puts "✓ SmartRAG initialized successfully"
47
+ stats = smart_rag.statistics
48
+ puts "✓ Database connected: #{stats[:document_count]} documents"
49
+ ```
50
+
51
+ ### First Document Addition
52
+
53
+ ```ruby
54
+ # Add your first document
55
+ result = smart_rag.add_document(
56
+ 'https://example.com/ai-article.pdf',
57
+ title: 'Introduction to AI',
58
+ generate_embeddings: true,
59
+ generate_tags: true
60
+ )
61
+
62
+ puts "✓ Document added: ID #{result[:document_id]}"
63
+ puts "✓ Sections created: #{result[:section_count]}"
64
+ ```
65
+
66
+ ### First Search
67
+
68
+ ```ruby
69
+ # Perform your first search
70
+ results = smart_rag.search(
71
+ 'machine learning algorithms',
72
+ search_type: 'hybrid',
73
+ limit: 5,
74
+ include_content: true
75
+ )
76
+
77
+ puts "\nSearch Results:"
78
+ results[:results].each_with_index do |result, i|
79
+ puts "#{i + 1}. #{result[:section_title]} (score: #{result[:combined_score].round(3)})"
80
+ puts " #{result[:content][0..150]}..."
81
+ end
82
+ ```
83
+
84
+ ## Document Management
85
+
86
+ ### Adding Documents from Various Sources
87
+
88
+ ```ruby
89
+ # From a local file
90
+ smart_rag.add_document(
91
+ '/path/to/document.pdf',
92
+ title: 'Research Paper 2024',
93
+ generate_embeddings: true
94
+ )
95
+
96
+ # From a URL
97
+ smart_rag.add_document(
98
+ 'https://arxiv.org/abs/2301.00001',
99
+ generate_embeddings: true,
100
+ generate_tags: true,
101
+ tags: ['research', 'AI']
102
+ )
103
+
104
+ # With custom metadata
105
+ smart_rag.add_document(
106
+ '/path/to/report.docx',
107
+ title: 'Q3 Financial Report',
108
+ generate_embeddings: false, # Skip embeddings for non-technical docs
109
+ metadata: {
110
+ department: 'Finance',
111
+ year: 2024,
112
+ confidential: true
113
+ }
114
+ )
115
+ ```
116
+
117
+ ### Batch Document Processing
118
+
119
+ ```ruby
120
+ # Process multiple documents efficiently
121
+ documents = [
122
+ { path: '/docs/paper1.pdf', tags: ['AI'] },
123
+ { path: '/docs/paper2.pdf', tags: ['ML'] },
124
+ { path: '/docs/paper3.pdf', tags: ['NLP'] }
125
+ ]
126
+
127
+ # Sequential processing
128
+ documents.each do |doc|
129
+ begin
130
+ result = smart_rag.add_document(
131
+ doc[:path],
132
+ generate_embeddings: true,
133
+ tags: doc[:tags]
134
+ )
135
+ puts "✓ Processed: #{doc[:path]}"
136
+ rescue => e
137
+ puts "✗ Failed: #{doc[:path]} - #{e.message}"
138
+ end
139
+ end
140
+
141
+ # Parallel processing for better performance
142
+ require 'concurrent'
143
+
144
+ pool = Concurrent::FixedThreadPool.new(5)
145
+ documents.each do |doc|
146
+ pool.post do
147
+ begin
148
+ smart_rag.add_document(
149
+ doc[:path],
150
+ generate_embeddings: true,
151
+ tags: doc[:tags]
152
+ )
153
+ puts "✓ Processed: #{doc[:path]}"
154
+ rescue => e
155
+ puts "✗ Failed: #{doc[:path]} - #{e.message}"
156
+ end
157
+ end
158
+ end
159
+
160
+ pool.shutdown
161
+ pool.wait_for_termination
162
+ ```
163
+
164
+ ### Document Management Operations
165
+
166
+ ```ruby
167
+ # List documents with pagination
168
+ docs_page_1 = smart_rag.list_documents(page: 1, per_page: 20)
169
+ docs_page_2 = smart_rag.list_documents(page: 2, per_page: 20)
170
+
171
+ # Search for specific documents
172
+ ml_docs = smart_rag.list_documents(search: 'machine learning')
173
+
174
+ # Get document details
175
+ doc = smart_rag.get_document(123)
176
+ puts "Title: #{doc[:title]}"
177
+ puts "Sections: #{doc[:section_count]}"
178
+ puts "Created: #{doc[:created_at]}"
179
+
180
+ # Remove a document
181
+ result = smart_rag.remove_document(123)
182
+ puts "Deleted sections: #{result[:deleted_sections]}"
183
+ puts "Deleted embeddings: #{result[:deleted_embeddings]}"
184
+ ```
185
+
186
+ ## Search Operations
187
+
188
+ ### Hybrid Search (Default)
189
+
190
+ Hybrid search combines vector and full-text search for optimal results.
191
+
192
+ ```ruby
193
+ # Basic hybrid search
194
+ results = smart_rag.search(
195
+ 'deep learning applications in healthcare',
196
+ search_type: 'hybrid',
197
+ limit: 10,
198
+ alpha: 0.7 # Weight for vector search (0.0 = pure text, 1.0 = pure vector)
199
+ )
200
+
201
+ # With content and metadata
202
+ results = smart_rag.search(
203
+ 'natural language processing',
204
+ search_type: 'hybrid',
205
+ limit: 5,
206
+ include_content: true,
207
+ include_metadata: true
208
+ )
209
+
210
+ # Search with filters
211
+ results = smart_rag.search(
212
+ 'artificial intelligence',
213
+ search_type: 'hybrid',
214
+ limit: 10,
215
+ filters: {
216
+ document_ids: [1, 2, 3], # Search only in specific documents
217
+ tag_ids: [4, 5] # Filter by tags
218
+ }
219
+ )
220
+ ```
221
+
222
+ ### Vector Search
223
+
224
+ Useful for semantic similarity searches.
225
+
226
+ ```ruby
227
+ # Pure vector search
228
+ results = smart_rag.vector_search(
229
+ 'neural network architectures',
230
+ limit: 5
231
+ )
232
+
233
+ # Vector search with tag boosting
234
+ results = smart_rag.vector_search(
235
+ 'transformer models',
236
+ limit: 10,
237
+ tag_boost_weight: 0.1 # Boost results with matching tags
238
+ )
239
+ ```
240
+
241
+ ### Full-Text Search
242
+
243
+ Best for exact keyword matching and boolean queries.
244
+
245
+ ```ruby
246
+ # Basic full-text search
247
+ results = smart_rag.fulltext_search(
248
+ 'convolutional neural networks',
249
+ limit: 5
250
+ )
251
+
252
+ # Advanced boolean query
253
+ results = smart_rag.fulltext_search(
254
+ 'artificial AND (intelligence OR learning) AND NOT robotics',
255
+ limit: 10
256
+ )
257
+
258
+ # Phrase search
259
+ results = smart_rag.fulltext_search(
260
+ '"deep reinforcement learning"',
261
+ limit: 5
262
+ )
263
+ ```
264
+
265
+ ### Multi-language Search
266
+
267
+ SmartRAG automatically detects and handles multiple languages.
268
+
269
+ ```ruby
270
+ # Chinese search
271
+ results = smart_rag.search('人工智能应用', language: 'zh_cn')
272
+
273
+ # Japanese search
274
+ results = smart_rag.search('機械学習アルゴリズム', language: 'ja')
275
+
276
+ # Korean search
277
+ results = smart_rag.search('딥러닝 모델', language: 'ko')
278
+
279
+ # Mixed language search (auto-detect)
280
+ results = smart_rag.search('AI和机器学习的发展', language: 'auto')
281
+ ```
282
+
283
+ ## Research Topic Management
284
+
285
+ ### Creating and Organizing Topics
286
+
287
+ ```ruby
288
+ # Create a research topic
289
+ topic = smart_rag.create_topic(
290
+ 'AI in Healthcare',
291
+ 'Applications of artificial intelligence in medical diagnosis and treatment',
292
+ tags: ['AI', 'healthcare', 'medicine', 'diagnosis'],
293
+ document_ids: [1, 2, 3] # Associate existing documents
294
+ )
295
+
296
+ # Create nested topic structure
297
+ parent_topic = smart_rag.create_topic(
298
+ 'Machine Learning',
299
+ 'Fundamental ML concepts and algorithms'
300
+ )
301
+
302
+ child_topic = smart_rag.create_topic(
303
+ 'Deep Learning',
304
+ 'Neural network based learning',
305
+ tags: ['neural_networks', 'deep_learning']
306
+ # Could link to parent if hierarchical topics are supported
307
+ )
308
+ ```
309
+
310
+ ### Managing Topic Content
311
+
312
+ ```ruby
313
+ # Add documents to a topic
314
+ topic_id = 456
315
+ document_id = 123
316
+
317
+ result = smart_rag.add_document_to_topic(topic_id, document_id)
318
+ puts "Added #{result[:added_sections]} sections to topic"
319
+
320
+ # Get topic recommendations
321
+ recommendations = smart_rag.get_topic_recommendations(topic_id, limit: 10)
322
+
323
+ recommendations[:recommendations].each do |rec|
324
+ puts "Recommended: #{rec[:section_title]}"
325
+ puts " Matching tags: #{rec[:matching_tags]}"
326
+ puts " Score: #{rec[:relevance_score]}"
327
+ end
328
+
329
+ # List all topics
330
+ topics = smart_rag.list_topics(page: 1, per_page: 20)
331
+ topics[:topics].each do |topic|
332
+ puts "#{topic[:title]} (#{topic[:document_count]} documents)"
333
+ end
334
+ ```
335
+
336
+ ## Tag Management
337
+
338
+ ### Automatic Tag Generation
339
+
340
+ ```ruby
341
+ # Generate tags for text
342
+ text = """
343
+ Machine learning is a subset of artificial intelligence that enables systems
344
+ to learn and improve from experience without being explicitly programmed.
345
+ It focuses on developing computer programs that can access data and use it
346
+ to learn for themselves.
347
+ """
348
+
349
+ tags = smart_rag.generate_tags(text, topic: 'AI Introduction')
350
+ puts "Categories: #{tags[:categories].join(', ')}"
351
+ puts "Content tags: #{tags[:content_tags].join(', ')}"
352
+
353
+ # Batch generate tags for document sections
354
+ document = smart_rag.get_document(1)
355
+ sections = document[:sections]
356
+
357
+ tags_by_section = {}
358
+ sections.each do |section|
359
+ tags = smart_rag.generate_tags(
360
+ section[:content],
361
+ topic: document[:title],
362
+ max_tags: 5
363
+ )
364
+ tags_by_section[section[:id]] = tags
365
+ end
366
+ ```
367
+
368
+ ### Manual Tag Management
369
+
370
+ ```ruby
371
+ # Create hierarchical tags
372
+ tag_service = SmartRAG::Services::TagService.new
373
+
374
+ hierarchy = {
375
+ "Technology" => {
376
+ "AI" => ["Machine Learning", "Deep Learning", "Neural Networks"],
377
+ "Programming" => ["Python", "Ruby", "JavaScript"]
378
+ }
379
+ }
380
+
381
+ created_tags = tag_service.create_hierarchy(hierarchy)
382
+
383
+ # Associate tags with content
384
+ tag = SmartRAG::Models::Tag.find_or_create("machine_learning")
385
+ section = SmartRAG::Models::SourceSection[1]
386
+
387
+ # Add tag to section
388
+ section.add_tag(tag)
389
+
390
+ # Find content by tag
391
+ ml_sections = tag.sections
392
+ ml_sections.each do |section|
393
+ puts "#{section.section_title}: #{section.content[0..100]}..."
394
+ end
395
+
396
+ # Search for tags
397
+ tags = tag_service.search_tags('learn', limit: 10)
398
+ tags.each { |tag| puts "#{tag.name} (#{tag.section_count} sections)" }
399
+ ```
400
+
401
+ ## Advanced Usage Patterns
402
+
403
+ ### Context-Aware Search
404
+
405
+ ```ruby
406
+ class ContextualSearch
407
+ def initialize(smart_rag)
408
+ @smart_rag = smart_rag
409
+ @search_history = []
410
+ end
411
+
412
+ def search_with_context(query, user_context = {})
413
+ # Enhance query with context
414
+ enhanced_query = enhance_query(query, user_context)
415
+
416
+ # Perform search
417
+ results = @smart_rag.search(
418
+ enhanced_query,
419
+ search_type: 'hybrid',
420
+ limit: 10,
421
+ filters: build_filters(user_context)
422
+ )
423
+
424
+ # Store in history
425
+ @search_history << { query: query, context: user_context, results: results }
426
+
427
+ results
428
+ end
429
+
430
+ private
431
+
432
+ def enhance_query(query, context)
433
+ # Add context-specific terms
434
+ case context[:domain]
435
+ when 'healthcare'
436
+ "#{query} medical health clinical"
437
+ when 'finance'
438
+ "#{query} financial economic banking"
439
+ else
440
+ query
441
+ end
442
+ end
443
+
444
+ def build_filters(context)
445
+ filters = {}
446
+ filters[:document_ids] = context[:document_ids] if context[:document_ids]
447
+ filters[:tag_ids] = context[:preferred_tags] if context[:preferred_tags]
448
+ filters
449
+ end
450
+ end
451
+
452
+ # Usage
453
+ contextual_search = ContextualSearch.new(smart_rag)
454
+ results = contextual_search.search_with_context(
455
+ 'risk assessment',
456
+ user_context: {
457
+ domain: 'finance',
458
+ document_ids: [1, 2, 3],
459
+ preferred_tags: [4, 5]
460
+ }
461
+ )
462
+ ```
463
+
464
+ ### Search Result Processing Pipeline
465
+
466
+ ```ruby
467
+ class SearchPipeline
468
+ def initialize(smart_rag)
469
+ @smart_rag = smart_rag
470
+ @processors = []
471
+ end
472
+
473
+ def add_processor(&block)
474
+ @processors << block
475
+ self
476
+ end
477
+
478
+ def search(query, options = {})
479
+ # Initial search
480
+ results = @smart_rag.search(query, options)
481
+
482
+ # Process through pipeline
483
+ @processors.each do |processor|
484
+ results = processor.call(results, query, options)
485
+ end
486
+
487
+ results
488
+ end
489
+ end
490
+
491
+ # Create pipeline with processors
492
+ pipeline = SearchPipeline.new(smart_rag)
493
+
494
+ # Add relevance scoring
495
+ pipeline.add_processor do |results, query, options|
496
+ results[:results].each do |result|
497
+ result[:relevance_score] = calculate_relevance(result, query)
498
+ end
499
+ results
500
+ end
501
+
502
+ # Add result filtering
503
+ pipeline.add_processor do |results, query, options|
504
+ min_score = options[:min_score] || 0.5
505
+ results[:results].select! { |r| r[:relevance_score] >= min_score }
506
+ results[:metadata][:filtered_count] = results[:results].length
507
+ results
508
+ end
509
+
510
+ # Use pipeline
511
+ results = pipeline.search(
512
+ 'neural networks',
513
+ min_score: 0.7,
514
+ limit: 20
515
+ )
516
+ ```
517
+
518
+ ### Caching Search Results
519
+
520
+ ```ruby
521
+ require 'redis'
522
+
523
+ class CachedSmartRAG
524
+ def initialize(smart_rag, redis_client)
525
+ @smart_rag = smart_rag
526
+ @redis = redis_client
527
+ @cache_ttl = 3600 # 1 hour
528
+ end
529
+
530
+ def search(query, options = {})
531
+ # Create cache key
532
+ cache_key = create_cache_key(query, options)
533
+
534
+ # Try to get from cache
535
+ cached = @redis.get(cache_key)
536
+ if cached
537
+ puts "Cache hit for: #{query}"
538
+ return JSON.parse(cached, symbolize_names: true)
539
+ end
540
+
541
+ # Perform search
542
+ results = @smart_rag.search(query, options)
543
+
544
+ # Store in cache
545
+ @redis.setex(cache_key, @cache_ttl, results.to_json)
546
+
547
+ puts "Cache miss for: #{query}"
548
+ results
549
+ end
550
+
551
+ private
552
+
553
+ def create_cache_key(query, options)
554
+ key_parts = [query, options.sort].flatten.join(':')
555
+ "search:#{Digest::MD5.hexdigest(key_parts)}"
556
+ end
557
+ end
558
+
559
+ # Usage
560
+ redis = Redis.new
561
+ $cached_rag = CachedSmartRAG.new(smart_rag, redis)
562
+
563
+ # First search - cache miss
564
+ results1 = $cached_rag.search('deep learning', limit: 10)
565
+
566
+ # Second search - cache hit
567
+ results2 = $cached_rag.search('deep learning', limit: 10)
568
+ ```
569
+
570
+ ### Building a Q&A System
571
+
572
+ ```ruby
573
+ class QA_system
574
+ def initialize(smart_rag)
575
+ @smart_rag = smart_rag
576
+ end
577
+
578
+ def answer(question, options = {})
579
+ # Search for relevant information
580
+ search_results = @smart_rag.search(
581
+ question,
582
+ search_type: 'hybrid',
583
+ limit: options[:context_limit] || 5,
584
+ include_content: true
585
+ )
586
+
587
+ # Generate answer based on search results (requires LLM integration)
588
+ answer = generate_answer(question, search_results[:results])
589
+
590
+ {
591
+ question: question,
592
+ answer: answer,
593
+ sources: extract_sources(search_results[:results]),
594
+ confidence: calculate_confidence(search_results[:results])
595
+ }
596
+ end
597
+
598
+ private
599
+
600
+ def generate_answer(question, results)
601
+ return "I don't have enough information to answer this question." if results.empty?
602
+
603
+ # Combine relevant content
604
+ context = results.map { |r| r[:content] }.join("\n\n---\n\n")
605
+
606
+ # Here you would call an LLM API to generate the answer
607
+ # This is a simplified version
608
+ "Based on the available information: #{context[0..500]}..."
609
+ end
610
+
611
+ def extract_sources(results)
612
+ results.map do |result|
613
+ {
614
+ section_id: result[:section_id],
615
+ title: result[:section_title],
616
+ score: result[:combined_score]
617
+ }
618
+ end
619
+ end
620
+
621
+ def calculate_confidence(results)
622
+ return 0.0 if results.empty?
623
+
624
+ # Simple confidence based on top result score
625
+ [results.first[:combined_score], 1.0].min
626
+ end
627
+ end
628
+
629
+ # Usage
630
+ qa = QA_system.new(smart_rag)
631
+ response = qa.answer(
632
+ 'What are the applications of transformers in NLP?',
633
+ context_limit: 3
634
+ )
635
+
636
+ puts "Answer: #{response[:answer]}"
637
+ puts "Confidence: #{(response[:confidence] * 100).round(1)}%"
638
+ puts "Sources:"
639
+ response[:sources].each do |source|
640
+ puts " - #{source[:title]} (ID: #{source[:section_id]})"
641
+ end
642
+ ```
643
+
644
+ ## Performance Best Practices
645
+
646
+ ### 1. Batch Operations
647
+
648
+ ```ruby
649
+ # Instead of individual operations
650
+ bad_practice = documents.map do |doc|
651
+ smart_rag.add_document(doc[:path], generate_embeddings: true)
652
+ end
653
+
654
+ # Use batch processing
655
+ good_practice = documents.each_slice(10) do |batch|
656
+ # Process batch in parallel
657
+ batch.map do |doc|
658
+ Concurrent::Promises.future do
659
+ smart_rag.add_document(doc[:path], generate_embeddings: true)
660
+ end
661
+ end.map(&:value)
662
+ end
663
+ ```
664
+
665
+ ### 2. Connection Pooling
666
+
667
+ ```ruby
668
+ # Configure database connection pool
669
+ config = {
670
+ database: {
671
+ adapter: 'postgresql',
672
+ host: 'localhost',
673
+ database: 'smart_rag',
674
+ user: 'user',
675
+ password: 'pass',
676
+ pool: 25, # Increase pool size
677
+ timeout: 5000, # Connection timeout
678
+ max_connections: 100 # Maximum connections
679
+ }
680
+ }
681
+ ```
682
+
683
+ Equivalent YAML key:
684
+
685
+ ```yaml
686
+ adapter: postgresql
687
+ ```
688
+
689
+ ### 3. Efficient Searching
690
+
691
+ ```ruby
692
+ # Use appropriate search types for queries
693
+ # For exact matches
694
+ text_results = smart_rag.fulltext_search('error code 404', limit: 5)
695
+
696
+ # For conceptual similarity
697
+ vector_results = smart_rag.vector_search('debugging techniques', limit: 5)
698
+
699
+ # For general queries
700
+ hybrid_results = smart_rag.search('how to fix bugs', search_type: 'hybrid')
701
+
702
+ # Adjust alpha based on use case
703
+ # - Technical/keyword-heavy: lower alpha (0.3-0.5)
704
+ # - Conceptual/exploratory: higher alpha (0.7-0.9)
705
+ ```
706
+
707
+ ### 4. Caching Strategies
708
+
709
+ ```ruby
710
+ # Cache embeddings for repeated content
711
+ class EmbeddingCache
712
+ def initialize
713
+ @cache = {}
714
+ end
715
+
716
+ def get_embedding(text)
717
+ hash = Digest::MD5.hexdigest(text)
718
+ @cache[hash] ||= generate_embedding(text)
719
+ end
720
+ end
721
+
722
+ # Cache search results
723
+ class SearchCache
724
+ def initialize(redis, ttl: 3600)
725
+ @redis = redis
726
+ @ttl = ttl
727
+ end
728
+
729
+ def fetch(query, options = {}, &block)
730
+ key = cache_key(query, options)
731
+
732
+ if result = @redis.get(key)
733
+ JSON.parse(result, symbolize_names: true)
734
+ else
735
+ result = block.call
736
+ @redis.setex(key, @ttl, result.to_json)
737
+ result
738
+ end
739
+ end
740
+ end
741
+ ```
742
+
743
+ ### 5. Database Optimization
744
+
745
+ ```sql
746
+ -- Create optimized indexes
747
+ CREATE INDEX CONCURRENTLY idx_section_fts_content
748
+ ON section_fts USING gin(to_tsvector('english', content));
749
+
750
+ CREATE INDEX CONCURRENTLY idx_embeddings_vector
751
+ ON embeddings USING ivfflat (vector vector_cosine_ops)
752
+ WITH (lists = 100);
753
+
754
+ -- Monitor and optimize slow queries
755
+ EXPLAIN ANALYZE
756
+ SELECT * FROM hybrid_search('machine learning', 10);
757
+ ```
758
+
759
+ ## Error Handling
760
+
761
+ ### Comprehensive Error Handling
762
+
763
+ ```ruby
764
+ begin
765
+ result = smart_rag.add_document(
766
+ '/path/to/document.pdf',
767
+ generate_embeddings: true
768
+ )
769
+ rescue SmartRAG::Errors::ArgumentError => e
770
+ puts "Invalid arguments: #{e.message}"
771
+ rescue SmartRAG::Errors::DatabaseError => e
772
+ puts "Database error: #{e.message}"
773
+ # Attempt to reconnect or use fallback
774
+ rescue SmartRAG::Errors::EmbeddingGenerationError => e
775
+ puts "Embedding generation failed: #{e.message}"
776
+ # Retry or skip embeddings
777
+ rescue SmartRAG::Errors::DocumentProcessingError => e
778
+ puts "Document processing failed: #{e.message}"
779
+ # Log and continue with next document
780
+ rescue => e
781
+ puts "Unexpected error: #{e.message}"
782
+ # Log for investigation
783
+ end
784
+ ```
785
+
786
+ ### Retry Logic
787
+
788
+ ```ruby
789
+ require 'retriable'
790
+
791
+ class RetryableSmartRAG
792
+ def initialize(smart_rag)
793
+ @smart_rag = smart_rag
794
+ end
795
+
796
+ def add_document(path, options = {})
797
+ Retriable.retriable(
798
+ on: [SmartRAG::Errors::EmbeddingGenerationError],
799
+ tries: 3,
800
+ base_interval: 1,
801
+ multiplier: 2
802
+ ) do
803
+ @smart_rag.add_document(path, options)
804
+ end
805
+ end
806
+
807
+ def search(query, options = {})
808
+ Retriable.retriable(
809
+ on: [SmartRAG::Errors::DatabaseError],
810
+ tries: 3,
811
+ base_interval: 0.5
812
+ ) do
813
+ @smart_rag.search(query, options)
814
+ end
815
+ end
816
+ end
817
+ ```
818
+
819
+ ## Common Patterns
820
+
821
+ ### Pattern 1: Document Processing Pipeline
822
+
823
+ ```ruby
824
+ class DocumentPipeline
825
+ def initialize(smart_rag)
826
+ @smart_rag = smart_rag
827
+ end
828
+
829
+ def process(files, options = {})
830
+ results = []
831
+
832
+ files.each do |file|
833
+ begin
834
+ # Step 1: Add document
835
+ doc_result = @smart_rag.add_document(
836
+ file,
837
+ generate_embeddings: false # Delay embedding generation
838
+ )
839
+
840
+ # Step 2: Generate tags
841
+ tags = @smart_rag.generate_tags(
842
+ extract_text(file),
843
+ topic: options[:topic]
844
+ )
845
+
846
+ # Step 3: Apply tags
847
+ if doc_result[:document_id]
848
+ document = SmartRAG::Models::SourceDocument[doc_result[:document_id]]
849
+ sections = document.sections
850
+
851
+ sections.each do |section|
852
+ tag_objects = tags[:content_tags].map do |tag_name|
853
+ SmartRAG::Models::Tag.find_or_create(tag_name)
854
+ end
855
+
856
+ section.add_tag(*tag_objects)
857
+ end
858
+
859
+ # Step 4: Generate embeddings (batch)
860
+ document.sections.each do |section|
861
+ embedding = @smart_rag.generate_embedding(section.content)
862
+ store_embedding(section.id, embedding)
863
+ end
864
+ end
865
+
866
+ results << { success: true, file: file, document_id: doc_result[:document_id] }
867
+ rescue => e
868
+ results << { success: false, file: file, error: e.message }
869
+ end
870
+ end
871
+
872
+ results
873
+ end
874
+ end
875
+ ```
876
+
877
+ ### Pattern 2: Incremental Indexing
878
+
879
+ ```ruby
880
+ class IncrementalIndexer
881
+ def initialize(smart_rag)
882
+ @smart_rag = smart_rag
883
+ end
884
+
885
+ def index_new_documents(source_dir, last_check = nil)
886
+ # Find new or modified documents
887
+ pattern = File.join(source_dir, '**/*.pdf')
888
+ documents = Dir.glob(pattern)
889
+
890
+ if last_check
891
+ documents.select! { |doc| File.mtime(doc) > last_check }
892
+ end
893
+
894
+ # Process in batches
895
+ documents.each_slice(10) do |batch|
896
+ batch_results = process_batch(batch)
897
+ log_results(batch_results)
898
+ end
899
+ end
900
+
901
+ private
902
+
903
+ def process_batch(files)
904
+ files.map do |file|
905
+ begin
906
+ result = @smart_rag.add_document(file, generate_embeddings: true)
907
+ { file: file, success: true, document_id: result[:document_id] }
908
+ rescue => e
909
+ { file: file, success: false, error: e.message }
910
+ end
911
+ end
912
+ end
913
+ end
914
+ ```
915
+
916
+ ### Pattern 3: Search Analytics
917
+
918
+ ```ruby
919
+ class SearchAnalytics
920
+ def initialize(smart_rag)
921
+ @smart_rag = smart_rag
922
+ end
923
+
924
+ def analyze_search_patterns(days = 30)
925
+ logs = smart_rag.search_logs(limit: 1000)
926
+
927
+ analytics = {
928
+ total_searches: logs.length,
929
+ avg_execution_time: logs.sum { |l| l[:execution_time_ms] } / logs.length,
930
+ popular_queries: popular_queries(logs),
931
+ failed_searches: logs.count { |l| l[:results_count] == 0 },
932
+ trend_analysis: trend_analysis(logs, days)
933
+ }
934
+ end
935
+
936
+ def identify_content_gaps(logs)
937
+ no_result_queries = logs.select { |l| l[:results_count] == 0 }
938
+
939
+ # Group similar queries
940
+ clusters = cluster_queries(no_result_queries)
941
+
942
+ # Identify topics needing more content
943
+ clusters.map do |cluster|
944
+ {
945
+ topic: cluster[:topic],
946
+ query_count: cluster[:queries].length,
947
+ sample_queries: cluster[:queries].first(3)
948
+ }
949
+ end
950
+ end
951
+ end
952
+ ```
953
+
954
+ ### Pattern 4: Multi-tenant Applications
955
+
956
+ ```ruby
957
+ class MultiTenantSmartRAG
958
+ def initialize(smart_rag)
959
+ @smart_rag = smart_rag
960
+ end
961
+
962
+ def add_document(tenant_id, path, options = {})
963
+ # Add tenant isolation
964
+ options[:metadata] ||= {}
965
+ options[:metadata][:tenant_id] = tenant_id
966
+
967
+ @smart_rag.add_document(path, options)
968
+ end
969
+
970
+ def search(tenant_id, query, options = {})
971
+ # Filter by tenant
972
+ options[:filters] ||= {}
973
+ options[:filters][:metadata] = { tenant_id: tenant_id }
974
+
975
+ @smart_rag.search(query, options)
976
+ end
977
+
978
+ def get_statistics(tenant_id)
979
+ # Get tenant-specific stats
980
+ @smart_rag.statistics(tenant_id: tenant_id)
981
+ end
982
+ end
983
+ ```
984
+
985
+ ## Summary
986
+
987
+ This guide has covered:
988
+
989
+ 1. **Quick Start** - Basic setup and first operations
990
+ 2. **Document Management** - Adding, organizing, and managing documents
991
+ 3. **Search Operations** - Various search types and advanced filtering
992
+ 4. **Research Topics** - Organizing content into thematic collections
993
+ 5. **Tag Management** - Automatic and manual tagging strategies
994
+ 6. **Advanced Patterns** - Production-ready implementations
995
+ 7. **Performance** - Best practices for optimal performance
996
+ 8. **Error Handling** - Robust error management strategies
997
+ 9. **Common Patterns** - Reusable solutions for typical scenarios
998
+
999
+ For more information, see:
1000
+ - [API Documentation](API_DOCUMENTATION.md) - Complete API reference
1001
+ - [Performance Guide](PERFORMANCE_GUIDE.md) - Performance optimization details
1002
+ - [Migration Guide](MIGRATION_GUIDE.md) - Version upgrade instructions
1003
+
1004
+ ## Support
1005
+
1006
+ - GitHub Issues: https://github.com/your-org/smart_rag/issues
1007
+ - Documentation Issues: Report any errors or inconsistencies in examples
1008
+ - Community Forum: Share your usage patterns and learn from others