opensearch-sugar 1.0.0

data/old_docs/EXPLANATION.md

@@ -0,0 +1,685 @@
+# Understanding OpenSearch::Sugar
+
+*(Documentation written by GitHub Copilot, powered by Claude Sonnet 4.5)*
+
+This document provides conceptual explanations and discussions about OpenSearch::Sugar's design, architecture, and key concepts. Rather than showing you what to do, it helps you understand why things work the way they do.
+
+## Table of Contents
+
+- [Design Philosophy](#design-philosophy)
+- [Architecture and Patterns](#architecture-and-patterns)
+- [When to Use OpenSearch::Sugar](#when-to-use-opensearchsugar)
+- [Understanding Text Analysis](#understanding-text-analysis)
+- [Index Management Concepts](#index-management-concepts)
+- [ML Models and Embeddings](#ml-models-and-embeddings)
+- [Performance Considerations](#performance-considerations)
+- [Security and Best Practices](#security-and-best-practices)
+
+---
+
+## Design Philosophy
+
+### The "Sugar" Metaphor
+
+OpenSearch::Sugar is intentionally named to reflect its purpose: it adds sweetness (convenience) on top of the official OpenSearch Ruby client without hiding or replacing the underlying functionality. Just as you can enjoy coffee with or without sugar, you can use OpenSearch with or without this wrapper.
+
+### Core Principles
+
+#### 1. Convention Over Configuration
+
+OpenSearch::Sugar provides sensible defaults that work out of the box:
+
+```ruby
+# Just works - uses environment variables for connection
+client = OpenSearch::Sugar.new
+
+# Opens or creates - handles existence checks automatically
+index = client.open_or_create('my_index')
+```
+
+The gem assumes common use cases and handles the boilerplate, but everything can be customized when needed.
+
+#### 2. Progressive Disclosure
+
+Simple things should be simple, complex things should be possible:
+
+```ruby
+# Simple: Get document count
+count = index.count
+
+# Complex: Use full OpenSearch query DSL
+results = client.search(
+  index: 'my_index',
+  body: {
+    query: { ... complex query ... },
+    aggs: { ... complex aggregations ... }
+  }
+)
+```
+
+You don't need to learn the entire OpenSearch API to do basic tasks, but you can access the full API when needed.
+
+#### 3. Transparency Through Delegation
+
+OpenSearch::Sugar::Client uses Ruby's `SimpleDelegator` to wrap `OpenSearch::Client`. This design choice has important implications:
+
+- **No hidden functionality**: Every method available in the official client is available here
+- **No magic**: You can read the opensearch-ruby documentation and apply it directly
+- **Easy migration**: Code written with opensearch-ruby works with OpenSearch::Sugar
+- **Best of both worlds**: Use sugar methods for common tasks, raw client for everything else
+
+```ruby
+# Sugar method
+client.has_index?('my_index')
+
+# Delegated to raw client
+client.indices.exists?(index: 'my_index')
+
+# Both work! Use whichever feels more natural
+```
+
+---
+
+## Architecture and Patterns
+
+### The Delegation Pattern
+
+The `Client` class inherits from `SimpleDelegator`, which forwards all method calls to the wrapped object:
+
+```ruby
+class Client < SimpleDelegator
+  def initialize(**kwargs)
+    @raw_client = OpenSearch::Client.new(**kwargs)
+    __setobj__(@raw_client)  # Set up delegation
+    # Now all raw_client methods are available
+  end
+end
+```
+
+**Why this matters:**
+- You're never "trapped" by the abstraction
+- The gem doesn't need to keep up with every OpenSearch API change
+- Documentation from the official client applies directly
+
+### The Facade Pattern
+
+The `Index` class acts as a facade, simplifying complex sequences of operations:
+
+```ruby
+def update_settings(settings)
+  # Behind the scenes:
+  # 1. Close the index
+  # 2. Apply settings
+  # 3. Reopen the index
+  # 4. Handle errors gracefully
+  client.update_settings(settings, name)
+end
+```
+
+**Benefits:**
+- Error-prone sequences become single method calls
+- Consistent error handling
+- Easier testing and maintenance
+
+### The Repository Pattern
+
+The `Models` class acts as a repository for ML models:
+
+```ruby
+models = client.models
+
+# Find by various identifiers
+model = models['name']
+model = models['id']
+model = models['partial_name_match']
+
+# List all
+all_models = models.list
+```
+
+**Why this works well:**
+- ML models are treated as first-class resources
+- Complex queries (by name, ID, or nickname) are unified
+- The interface hides OpenSearch's internal model storage structure
+
+---
+
+## When to Use OpenSearch::Sugar
+
+### Perfect Use Cases
+
+**1. Application Development**
+- Building Ruby applications that use OpenSearch
+- Creating search features in Rails apps
+- Developing internal tools and scripts
+
+**2. Index Management**
+- Creating and configuring indexes
+- Managing settings and mappings
+- Working with aliases and index lifecycle
+
+**3. ML Integration**
+- Deploying and managing ML models
+- Creating embedding pipelines
+- Vector search applications
+
+**4. Prototyping and Exploration**
+- Quickly testing OpenSearch features
+- Learning OpenSearch concepts
+- Building proof-of-concepts
+
+### When to Use the Raw Client Instead
+
+**1. Complex Queries**
+- Advanced search DSL with nested aggregations
+- Specialized query types not wrapped by sugar methods
+- Performance-critical search operations where you need fine control
+
+**2. Bulk Operations at Scale**
+- Processing millions of documents
+- Custom bulk error handling
+- Streaming data ingestion
+
+**3. Low-Level Operations**
+- Cluster management
+- Shard allocation
+- Snapshot and restore
+
+**The good news:** You don't have to choose! Use sugar methods where they help, drop to the raw client where they don't:
+
+```ruby
+# Mix and match freely
+index = client.open_or_create('my_index')  # Sugar
+count = index.count                         # Sugar
+
+response = client.search(                   # Raw client
+  index: index.name,
+  body: { query: { ... } }
+)
+```
+
+---
+
+## Understanding Text Analysis
+
+### Why Text Analysis Matters
+
+Text analysis is the process of converting text into searchable tokens. It's one of the most important concepts in OpenSearch:
+
+```ruby
+# Original text
+"The Quick Brown Fox"
+
+# After standard analyzer
+["the", "quick", "brown", "fox"]
+```
+
+Without proper analysis, searches won't work as expected.
+
+### The Analysis Pipeline
+
+Every field goes through these stages:
+
+1. **Character filters** - Modify the text (e.g., strip HTML)
+2. **Tokenizer** - Split text into tokens
+3. **Token filters** - Modify tokens (lowercase, stemming, stop words)
+
+```ruby
+settings = {
+  settings: {
+    analysis: {
+      analyzer: {
+        my_analyzer: {
+          type: 'custom',
+          char_filter: ['html_strip'],      # 1. Remove HTML
+          tokenizer: 'standard',            # 2. Split on whitespace/punctuation
+          filter: ['lowercase', 'stop']    # 3. Lowercase + remove stop words
+        }
+      }
+    }
+  }
+}
+```
+
+### Index-Time vs Search-Time Analysis
+
+A common pattern is using different analyzers for indexing and searching:
+
+```ruby
+mappings = {
+  mappings: {
+    properties: {
+      title: {
+        type: 'text',
+        analyzer: 'strict_analyzer',       # Index-time: aggressive filtering
+        search_analyzer: 'lenient_analyzer' # Search-time: keep more terms
+      }
+    }
+  }
+}
+```
+
+**Why?**
+- **Index-time**: Be strict, remove noise, normalize heavily
+- **Search-time**: Be lenient, match user's exact input
+
+### Testing Your Analyzers
+
+OpenSearch::Sugar makes it easy to test analysis:
+
+```ruby
+tokens = index.analyze_text(
+  analyzer: 'my_analyzer',
+  text: 'Sample text'
+)
+
+# See exactly what gets indexed!
+puts tokens
+```
+
+**Always test your analyzers** before indexing production data.
+
+---
+
+## Index Management Concepts
+
+### The Index Lifecycle
+
+Indexes in OpenSearch typically follow this lifecycle:
+
+1. **Create** - Define structure
+2. **Configure** - Set analyzers, shards, replicas
+3. **Map** - Define field types
+4. **Populate** - Add documents
+5. **Query** - Search and retrieve
+6. **Maintain** - Update settings, reindex
+7. **Archive/Delete** - Remove when no longer needed
+
+OpenSearch::Sugar provides methods for each stage.
+
+### Settings vs Mappings
+
+**Settings** control how the index behaves:
+- Number of shards and replicas
+- Refresh intervals
+- Analysis configuration (analyzers, tokenizers, filters)
+
+**Mappings** define the document structure:
+- Field names and types
+- Which analyzer each field uses
+- Whether fields are indexed, stored, or both
+
+```ruby
+# Settings: HOW the index works
+index.update_settings(
+  settings: {
+    number_of_replicas: 2,
+    analysis: { ... }
+  }
+)
+
+# Mappings: WHAT the documents contain
+index.update_mappings(
+  mappings: {
+    properties: {
+      title: { type: 'text' },
+      date: { type: 'date' }
+    }
+  }
+)
+```
+
+### Why Indexes Need to Close for Updates
+
+Some settings and mappings can't be changed on an open index because:
+- They affect how data is stored on disk
+- Changing them would require rewriting existing data
+- OpenSearch needs to ensure consistency
+
+OpenSearch::Sugar handles this automatically:
+
+```ruby
+# Behind the scenes:
+# indices.close(index: 'my_index')
+# indices.put_settings(...)
+# indices.open(index: 'my_index')
+
+index.update_settings(settings)
+```
+
+### The Alias Pattern
+
+Aliases are pointers to indexes. They enable:
+
+1. **Zero-downtime reindexing**
+2. **Multiple indexes in one query**
+3. **Versioned indexes**
+
+```ruby
+# Create new index with updated mappings
+new_index = client.create('products_v2')
+new_index.update_mappings(improved_mappings)
+
+# Copy data from old to new
+# ... reindex operation ...
+
+# Atomic switch
+client.indices.update_aliases(
+  body: {
+    actions: [
+      { remove: { index: 'products_v1', alias: 'products' } },
+      { add: { index: 'products_v2', alias: 'products' } }
+    ]
+  }
+)
+
+# Applications using 'products' alias are unaffected!
+```
+
+---
+
+## ML Models and Embeddings
+
+### What Are Embeddings?
+
+Embeddings are numerical representations of text (or other data) that capture semantic meaning:
+
+```
+"cat"     → [0.2, 0.8, 0.1, ...]
+"kitten"  → [0.3, 0.7, 0.2, ...]  # Similar to "cat"
+"car"     → [0.9, 0.1, 0.3, ...]  # Different from "cat"
+```
+
+This enables:
+- Semantic search (find documents by meaning, not just keywords)
+- Recommendations (find similar documents)
+- Classification and clustering
+
+### The ML Model Workflow
+
+OpenSearch::Sugar simplifies the ML workflow:
+
+```ruby
+# 1. Register and deploy a model
+model = client.models.register(
+  name: 'huggingface/sentence-transformers/all-MiniLM-L12-v2',
+  version: '1.0.1'
+)
+
+# 2. Create an ingest pipeline
+client.models.create_pipeline(
+  name: 'embedding_pipeline',
+  model: model.name,
+  field_map: { 'text' => 'text_embedding' }
+)
+
+# 3. Index documents with the pipeline
+client.index(
+  index: 'my_index',
+  pipeline: 'embedding_pipeline',
+  body: { text: 'My document text' }
+)
+# The pipeline automatically adds 'text_embedding' field
+
+# 4. Search using k-NN
+results = client.search(
+  index: 'my_index',
+  body: {
+    query: {
+      knn: {
+        text_embedding: {
+          vector: query_embedding,
+          k: 10
+        }
+      }
+    }
+  }
+)
+```
+
+### Why Models Are Complex
+
+ML models in OpenSearch:
+- Must be registered before use
+- Take time to deploy
+- Have specific format requirements
+- Need to be undeployed before deletion
+
+OpenSearch::Sugar handles these complexities:
+
+```ruby
+# Simple interface
+model = models.register(...)  # Waits for deployment
+models.delete!(model)         # Undeploys first, then deletes
+```
+
+---
+
+## Performance Considerations
+
+### Indexing Performance
+
+**Bulk operations are essential** for high-throughput indexing:
+
+```ruby
+# Slow: Individual requests
+documents.each do |doc|
+  client.index(index: 'my_index', body: doc)
+end
+
+# Fast: Bulk request
+operations = documents.flat_map do |doc|
+  [
+    { index: { _index: 'my_index', _id: doc[:id] } },
+    doc
+  ]
+end
+client.bulk(body: operations)
+```
+
+**Why?**
+- Network overhead: 1 request vs. 1000 requests
+- OpenSearch optimization: Processes batches more efficiently
+- Refresh timing: Fewer refresh operations
+
+### Refresh Interval
+
+By default, OpenSearch makes documents searchable every second:
+
+```ruby
+# For bulk indexing, disable automatic refresh
+index.update_settings(
+  settings: { refresh_interval: '-1' }
+)
+
+# Index lots of documents
+# ...
+
+# Manual refresh when done
+client.indices.refresh(index: index.name)
+
+# Re-enable automatic refresh
+index.update_settings(
+  settings: { refresh_interval: '1s' }
+)
+```
+
+**Tradeoff:**
+- Fast indexing but delayed search visibility
+- vs.
+- Slower indexing but immediate search visibility
+
+### Replica Strategy
+
+Replicas improve read performance but slow down indexing:
+
+```ruby
+# For initial bulk load
+index.update_settings(settings: { number_of_replicas: 0 })
+
+# Index documents
+# ...
+
+# Add replicas after indexing
+index.update_settings(settings: { number_of_replicas: 2 })
+```
+
+### Search Performance
+
+**Use filters when possible:**
+
+```ruby
+# Faster: filter (cacheable)
+{
+  query: {
+    bool: {
+      filter: [
+        { term: { status: 'published' } }
+      ]
+    }
+  }
+}
+
+# Slower: query (scoring required)
+{
+  query: {
+    match: { status: 'published' }
+  }
+}
+```
+
+Filters don't calculate relevance scores and can be cached.
+
+---
+
+## Security and Best Practices
+
+### Connection Security
+
+**Development:**
+```ruby
+client = OpenSearch::Sugar.new(
+  host: 'https://localhost:9200',
+  transport_options: {
+    ssl: { verify: false }  # OK for dev
+  }
+)
+```
+
+**Production:**
+```ruby
+client = OpenSearch::Sugar.new(
+  host: 'https://search.production.com:9200',
+  user: ENV['OPENSEARCH_USER'],
+  password: ENV['OPENSEARCH_PASSWORD'],
+  transport_options: {
+    ssl: {
+      verify: true,
+      ca_file: '/path/to/ca.pem'
+    }
+  }
+)
+```
+
+**Never:**
+- Hardcode credentials in source code
+- Disable SSL verification in production
+- Use default passwords
+- Expose OpenSearch directly to the internet
+
+### Environment Variables
+
+Use environment variables for configuration:
+
+```ruby
+# .env file
+OPENSEARCH_URL=https://localhost:9200
+OPENSEARCH_USER=admin
+OPENSEARCH_PASSWORD=secret_password
+
+# Load with dotenv gem
+require 'dotenv/load'
+
+# Client automatically uses these
+client = OpenSearch::Sugar.new
+```
+
+**Benefits:**
+- Different settings per environment
+- No secrets in version control
+- Easy configuration management
+
+### Index Design Best Practices
+
+**1. Use appropriate field types:**
+```ruby
+{
+  title: { type: 'text' },      # Full-text search
+  status: { type: 'keyword' },  # Exact match, aggregations
+  price: { type: 'float' },     # Numeric range queries
+  created_at: { type: 'date' }  # Date range queries
+}
+```
+
+**2. Include keyword subfields for text:**
+```ruby
+{
+  title: {
+    type: 'text',
+    fields: {
+      keyword: { type: 'keyword' }
+    }
+  }
+}
+
+# Now you can:
+# - Search: match on 'title'
+# - Sort/aggregate: use 'title.keyword'
+```
+
+**3. Design for your query patterns:**
+- If you only filter, use keyword type
+- If you search text, use text type with appropriate analyzer
+- If you do both, use text with keyword subfield
+
+### Error Handling
+
+Always handle errors in production:
+
+```ruby
+begin
+  index = client.open_or_create('my_index')
+  index.count
+rescue OpenSearch::Transport::Transport::Error => e
+  logger.error "OpenSearch error: #{e.message}"
+  # Handle error appropriately
+rescue ArgumentError => e
+  logger.error "Invalid argument: #{e.message}"
+  # Handle error appropriately
+end
+```
+
+**Don't:**
+- Silently ignore errors
+- Use broad rescue clauses
+- Let OpenSearch exceptions crash your application
+
+---
+
+## Conclusion
+
+OpenSearch::Sugar is designed to make working with OpenSearch more enjoyable and productive while never hiding the power of the underlying system. Understanding these concepts will help you make better decisions about:
+
+- When to use sugar methods vs. raw client methods
+- How to design efficient indexes
+- How to optimize for your specific use case
+- How to build maintainable, secure applications
+
+## Further Reading
+
+- **[Tutorial](TUTORIAL.md)** - Hands-on learning experience
+- **[How-to Guides](HOWTO.md)** - Solve specific problems
+- **[Reference](REFERENCE.md)** - Complete API documentation
+- **[OpenSearch Documentation](https://opensearch.org/docs/latest/)** - Official OpenSearch docs
+- **[OpenSearch Best Practices](https://opensearch.org/docs/latest/tuning-your-cluster/)** - Performance tuning
+