stretchy-model 0.6.5 → 0.6.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5de2e134e27640a01cc802b24b139510dacd84b633fe35795ba5215bf620bd5e
-  data.tar.gz: ee207f2854bbca7c03dfb104bf8bfe2934072ea578e23e58852f6842bd1e41b6
+  metadata.gz: ff8d7515eb6f795cf92989ba14d3dd404bb716378896ca2da4054bca98e467f4
+  data.tar.gz: 2361d39fcacb927f7d62bfe2105054c0923cd6b20af3cd7724c2bc9d784d42e5
 SHA512:
-  metadata.gz: a4937d127058d0bdaa2eb8d2e7ff6802096aa592ec75672d175475ea317d6794380c072934e0479dc45ca6baf864715be20392b796274bbc07ce31c8b474b2de
-  data.tar.gz: f0210926d21762429885e37126d17e7b6d718b4b6cb3ee5bfd2332842a35fb1f446c833854d60062fa423748c3696a8d16da6dcbe9b58fd8c254287ef281661a
+  metadata.gz: 410eb69810533c4a12fe12b1bdd5ea6df9b320f9c1953a32f924487b02f1825296765e5a5451abf1c22491a6d9e8a1741e4e7ef01789acbac0026e1db423602d
+  data.tar.gz: 9afdb5b058a307c3b64f8043f1e4f2348973050b6b5c802726031f060a44ac11003633ba816ab278ba204c1a4946a12f0c0c7171fd72cf0188493c43b3e0d4df

data/.yardopts

@@ -1 +1,2 @@
---markup markdown
+#  --markup markdown
+# --format=markdown

data/README.md

@@ -1,34 +1,43 @@
 stretchy-model
 ===
 <p>
-    <a href="https://stretchy.io/" target="_blank"><img src="./stretchy.logo.png" alt="Gum Image" width="450" /></a>
+    <a href="https://stretchy.io/" target="_blank"><img src="./stretchy.logo.png" alt="Stretchy Image" width="450" /></a>
     <br><br>
     <a href="https://github.com/theablefew/stretchy/releases"><img src="https://img.shields.io/github/v/release/theablefew/stretchy?sort=semver&color=blue"></a>
     <a href="https://github.com/theablefew/stretchy/actions"><img src="https://github.com/theablefew/stretchy/actions/workflows/spec.yml/badge.svg"></a>
 
 </p>
 
+Stretchy provides Elasticsearch/Opensearch models in Rails applications with an Rails-like model interface.
 
 ## Features
 Stretchy simplifies the process of querying, aggregating, and managing Elasticsearch-backed models, allowing Rails developers to work with search indices as comfortably as they would with traditional Rails models.
 
-* Model fully back by Elasticsearch/Opensearch
-* Chain queries, scopes and aggregations
+* Models fully back by Elasticsearch/Opensearch
+* Chain `queries`, `scopes` and `aggregations`
 * Reduce Elasticsearch query complexity
 * Support for time-based indices and aliases
-* Associations to both ActiveRecord models and Stretchy::Record
+* Associations to both ActiveRecord models and `StretchyModel`
 * Bulk Operations made easy
+* Ingest and Search Pipelines
+* Machine Learning
+* Vector and Neural search 
+* Integrated RAG and LLM connectors
 * Validations, custom attributes, and more...
 
 Follow the guides to learn more about:
-
 * [Models](https://theablefew.github.io/stretchy/#/guides/models?id=models)
 * [Querying](https://theablefew.github.io/stretchy/#/guides/querying?id=querying)
 * [Aggregations](https://theablefew.github.io/stretchy/#/guides/aggregations?id=aggregations)
 * [Scopes](https://theablefew.github.io/stretchy/#/guides/scopes?id=scopes)
+* [Pipelines](https://theablefew.github.io/stretchy/#/guides/pipelines?id=pipelines)
+* [Machine Learning](https://theablefew.github.io/stretchy/#/guides/machine-learning?id=machine-learning)
 
+[Read the Documentation](https://theablefew.github.io/stretchy/#/) or follow the examples below:
 
-[Read the Documentation](https://theablefew.github.io/stretchy/#/) or walk through of a simple [Data Analysis](https://theablefew.github.io/stretchy/#/examples/data_analysis?id=data-analysis) example.
+ **Examples**
+ - [Data Analysis](https://theablefew.github.io/stretchy/#/examples/data_analysis?id=data-analysis) example.
+ - [Simple Ingest Pipeline](https://theablefew.github.io/stretchy/#/examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
 
 
 
@@ -45,11 +54,15 @@ If bundler is not being used to manage dependencies, install the gem by executin
   gem install stretchy-model
 ```
 
+>[!TIP]
+> If using OpenSearch make sure to add the gem to your Gemfile.
+>
+> `bundle add opensearch-ruby`
+
 <details>
 <summary>Rails Configuration</summary>
 
 
-
 ```sh
 rails credentials:edit
 ```
@@ -59,19 +72,24 @@ rails credentials:edit
 elasticsearch:
    url: localhost:9200
 
-# or opensearch
+# or if using opensearch
 # opensearch:
 #    host: https://localhost:9200
 #    user: admin
 #    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
 ```
 
 #### Create an initializer 
 <p><sub><em>config/initializers/stretchy.rb</em></sub></p>
 
-```ruby {file=config/initializers/stretchy.rb}
+```ruby
 Stretchy.configure do |config|
-    config.client = Elasticsearch::Client.new url: Rails.application.credentials.elasticsearch.url, log: true
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using OpenSearch
+    # config.client = OpenSearch::Client.new Rails.application.credentials.opensearch
 end
 ```
 </details>

data/Rakefile

@@ -94,3 +94,59 @@ namespace :publish do
     Rake::Task['publish:release'].invoke('patch')
   end
 end
+
+
+namespace :documentation do
+
+  desc "Generate documentation"
+  task :generate do
+    system('rdoc --format=markdown --markup=markdown -o docs/doc --force-output -O')
+    # generate sidebar
+    Rake::Task['documentation:build_sidebar'].invoke
+    
+  end
+
+  desc "Build sidebar"
+  task :build_sidebar do
+    # Get all directories in docs/doc
+    directories = Dir.glob('docs/doc/**/*').select { |f| File.directory?(f) }
+  
+    # For each directory
+    directories.each do |dir|
+      puts "Building sidebar for #{dir}"
+      # Open _sidebar.md in write mode
+      File.open("#{dir}/_sidebar.md", 'w') do |file|
+        # Get all .md files in the directory
+        md_files = Dir.glob("#{dir}/*.md")
+  
+        # Add top level header with link to the directory
+        file.puts("- [#{File.basename(dir)}](#{dir.gsub('docs/', '')}.md)")
+        # For each .md file
+        md_files.each do |md_file|
+          next if md_file.include?('_sidebar.md')
+          # Write a markdown link to the file
+          md_file_path = md_file.gsub('docs/', '')
+          file_name = File.basename(md_file, '.md')
+          file.puts("  - [#{file_name}](#{md_file_path})")
+  
+
+          # Check for a directory with the same name as the file
+          if Dir.exist?("#{dir}/#{file_name}")
+            # Get all .md files in the subdirectory
+            sub_md_files = Dir.glob("#{dir}/#{file_name}/*.md")
+            # For each .md file in the subdirectory
+            # Put the parent directory name as a header
+            sub_md_files.each do |sub_md_file|
+              # Write a markdown link to the file
+              sub_md_file_path = sub_md_file.gsub('docs/', '')
+              sub_file_name = File.basename(sub_md_file, '.md')
+              next if sub_md_file.include?('_sidebar.md')
+              file.puts("    - [#{sub_file_name}](#{sub_md_file_path})")
+            end
+          end
+        end
+      end
+    end
+  end
+
+end

data/docs/README.md

@@ -0,0 +1,147 @@
+stretchy-model
+===
+Stretchy provides Elasticsearch/Opensearch models in Rails applications with an Rails-like model interface.
+<p >
+    <a href="https://github.com/theablefew/stretchy/releases"><img src="https://img.shields.io/github/v/release/theablefew/stretchy?sort=semver&color=blue"></a>
+    <a href="https://github.com/theablefew/stretchy/actions"><img src="https://github.com/theablefew/stretchy/actions/workflows/spec.yml/badge.svg"></a>
+
+</p>
+
+## Features
+Stretchy simplifies the process of querying, aggregating, and managing Elasticsearch-backed models, allowing Rails developers to work with search indices as comfortably as they would with traditional Rails models.
+
+* Models fully back by Elasticsearch/Opensearch
+* Chain `queries`, `scopes` and `aggregations`
+* Reduce Elasticsearch query complexity
+* Support for time-based indices and aliases
+* Associations to both ActiveRecord models and `StretchyModel`
+* Bulk Operations made easy
+* Ingest and Search Pipelines
+* Machine Learning
+* Vector and Neural search 
+* Integrated RAG and LLM connectors
+* Validations, custom attributes, and more...
+
+Follow the guides to learn more about:
+* [Models](guides/models?id=models)
+* [Querying](guides/querying?id=querying)
+* [Aggregations](guides/aggregations?id=aggregations)
+* [Scopes](guides/scopes?id=scopes) 
+* [Pipelines](guides/pipelines?id=pipelines)
+* [Machine Learning](guides/machine-learning?id=machine-learning)
+
+**Examples**
+ - [Data Analysis](examples/data_analysis?id=data-analysis)
+ - [Simple Ingest Pipeline](examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
+
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+```sh
+bundle add stretchy-model
+```
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+```sh
+gem install stretchy-model
+```
+
+>[!INFO|style:flat]
+> If using OpenSearch add the gem to your Gemfile:
+>
+> ```
+> bundle add opensearch-ruby
+> ```
+
+#### Add credentials
+
+```sh
+rails credentials:edit
+```
+
+```yaml
+elasticsearch:
+   url: localhost:9200
+
+# or if using opensearch
+# opensearch:
+#    host: http://localhost:9200
+#    user: admin
+#    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
+```
+
+#### Create an initializer 
+*config/initializers/stretchy.rb*
+
+```ruby
+Stretchy.configure do |config|
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using OpenSearch
+    # config.client = OpenSearch::Client.new Rails.application.credentials.opensearch
+end
+```
+
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+
+## Testing
+<details>
+<summary>Act</summary>
+
+Run github action workflow locally
+
+```sh
+brew install act --HEAD
+```
+
+```sh
+act -P ubuntu-latest=ghcr.io/catthehacker/ubuntu:runner-latest
+```
+
+</details>
+
+<details>
+<summary>Elasticsearch</summary>
+
+
+```
+docker-compose up elasticsearch
+```
+
+```
+bundle exec rspec
+```
+
+</details>
+
+<details>
+<summary>Opensearch</summary>
+
+
+```
+docker-compose up opensearch
+```
+
+```
+ENV['BACKEND']=opensearch bundle rspec 
+```
+</details>
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/theablefew/stretchy. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/theablefew/stretchy/blob/master/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+

data/docs/_coverpage.md

@@ -0,0 +1,14 @@
+
+<img src="./stretchy.cover.png" alt="stretchy" width="650" />
+
+
+> Elasticsearch Models for Ruby on Rails
+
+- Chainable Queries and Aggregations
+- Integrated RAG and LLM Agents
+- Ingest Pipelines
+
+[GitHub](https://github.com/theablefew/stretchy)
+[Get Started](#stretchy-model)
+
+![color](#D0B6E1)

data/docs/_sidebar.md

@@ -0,0 +1,14 @@
+* [__Readme__](/)
+
+* __Guides__ 
+    * [Quick Start](guides/quick-start?id=quick-start)
+    * [Models](guides/models?id=models)
+    * [Querying](guides/querying?id=querying)
+    * [Scopes](guides/scopes?id=scopes)
+    * [Aggregations](guides/aggregations?id=aggregations)
+    * [Pipelines](guides/pipelines?id=pipelines)
+    * [Machine Learning](guides/machine-learning?id=machine-learning)
+
+* __Examples__
+    * [Data Analysis](examples/data_analysis)
+    * [Simple Ingest Pipeline](examples/simple-ingest-pipeline)

data/docs/examples/_sidebar.md

@@ -0,0 +1,15 @@
+* [__Readme__](/)
+
+* __Guides__ 
+    * [Quick Start](guides/quick-start?id=quick-start)
+    * [Models](guides/models?id=models)
+    * [Querying](guides/querying?id=querying)
+    * [Scopes](guides/scopes?id=scopes)
+    * [Aggregations](guides/aggregations?id=aggregations)
+    * [Pipelines](guides/pipelines?id=pipelines)
+    * [Machine Learning](guides/machine-learning?id=machine-learning)
+
+* __Examples__
+    * [Data Analysis](examples/data_analysis)
+    * [Simple Ingest Pipeline](examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
+    * [Semantic Search with LLMs](examples/semantic_search_with_llm)

data/docs/examples/data_analysis.md

@@ -0,0 +1,216 @@
+# Data Analysis
+
+This guide will walk you through setting up a basic Rails application using Stretchy and Elasticsearch to ingest and visualize time series data.
+
+## Prerequisites
+
+- Ruby and Rails installed on your machine
+- Elasticsearch installed and running
+
+
+
+__Start Elasticsearch:__
+
+```sh
+docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.0
+```
+
+## Step 1: Create a New Rails Application
+
+```sh
+rails new time_series_app
+cd time_series_app
+```
+
+## Step 2: Add `stretchy-model` to Your Gemfile
+```sh
+bundle add stretchy-model   
+```
+## Step 3: Configure Stretchy
+#### Add credentials
+```sh
+rails credentials:edit
+```
+
+```yaml
+elasticsearch:
+   url: localhost:9200
+
+# or opensearch
+# opensearch:
+#    host: https://localhost:9200
+#    user: admin
+#    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
+```
+
+> __NOTE__ If using Opensearch be sure to run:
+> ```sh
+>   bundle add opensearch
+> ```
+
+Create an initializer file config/initializers/stretchy.rb and add the following:
+```ruby
+Stretchy.configure do |config|
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using Opensearch
+    # config.client = Openseaerch::Client.new Rails.application.credentials.opensearch
+end
+```
+## Identify a Data Set
+
+Let's use "Historical Plane Crashes Since 1908" for our dataset. The following will download the csv, transform the headers and output a json file. 
+
+Open a Rails console:
+```sh
+rails c
+```
+
+```ruby
+require 'open-uri'
+require 'csv'
+
+url = 'https://huggingface.co/datasets/nateraw/airplane-crashes-and-fatalities/resolve/main/Airplane_Crashes_and_Fatalities_Since_1908.csv?download=true'
+download = URI.open(url)
+scrub_headers = lambda { |header| header.parameterize.underscore }
+csv = CSV.parse(download.read, headers: true, header_converters: scrub_headers)
+data = csv.map(&:to_hash)
+data.each { |d| d.merge!({'occurred_on': "#{d.delete('date')} #{d.delete('time')}"}); d.delete('index') }
+
+File.open('airplane_crashes.json', 'w') do |f|
+  f.write(JSON.pretty_generate(data))
+end
+```
+
+Now, each entry has the following schema:
+```json
+{
+        "location": "Fort Myer, Virginia",
+        "operator": "Military - U.S. Army",
+          "flight": null,
+           "route": "Demonstration",
+            "type": "Wright Flyer III",
+    "registration": null,
+           "cn_in": "1",
+          "aboard": "2.0",
+      "fatalities": "1.0",
+          "ground": "0.0",
+         "summary": "During a demonstration flight, a U.S. Army flyer flown by Orville Wright nose-dived into the ground from a height of approximately 75 feet, killing Lt. Thomas E. Selfridge who was a passenger. This was the first recorded airplane fatality in history.  One of two propellers separated in flight, tearing loose the wires bracing the rudder and causing the loss of control of the aircraft.  Orville Wright suffered broken ribs, pelvis and a leg.  Selfridge suffered a crushed skull and died a short time later.",
+      "occured_on": "09/17/1908 17:18"
+}
+```
+
+## Understanding Performance Benefits
+## Define a Index Strategy
+
+Since we have a date field, we can store this data across multiple indexes. `stretchy-model` makes it easy to facilitate this strategy, which is useful when you have a rolling window of data. 
+
+## Determine Routing Strategy
+
+Routing determines which shard in the cluster the document will be written routed to. For large data sets it's a good idea to choose a routing key that groups similar queried data together. It all depends on your needs and your data distribution. For example purposes we'll route based on the operator.
+
+## Define CrashEvent Model
+Create a file for `CrashEvent` at `app/models/crash_event.rb`
+
+```ruby
+class CrashEvent < StretchyModel
+
+  attribute :occurred_on, :datetime, model_format: '%m/%d/%Y'
+  attribute :location, :keyword
+  attribute :operator, :keyword
+  attribute :flight, :keyword
+  attribute :route, :keyword
+  attribute :type, :keyword
+  attribute :registration, :keyword
+  attribute :cn_in, :keyword
+  attribute :aboard, :float
+  attribute :fatalities, :float
+  attribute :ground, :float
+  attribute :summary, :text
+
+  scope :operator_fatalities, -> { aggregation(:operator_fatalities, 
+      {
+        terms: {
+          field: :operator
+        }, 
+        aggs: {
+          fatalities: {
+            sum: {
+              field: :fatalities
+            }
+          }
+        }
+      }
+    ) 
+  }
+
+  scope :date_histogram, ->(name, options, *aggs) do
+    # requires field and calendar_interval or interval
+    aggregation(name, {date_histogram: options}.merge(*aggs))
+  end
+
+end
+
+```
+
+
+## Bulk Index Documents
+
+
+Reload your Rails console and start the bulk index operation:
+
+```ruby
+data = JSON.parse(File.read('/Users/spencer/Downloads/airplane_crashes.json'))
+CrashEvent.bulk_in_batches(data, size: 100) do |batch|
+  puts "Processing batch: #{batch.length}"
+  batch.map! do |event|
+    CrashEvent.new(event.symbolize_keys).to_bulk
+  end
+end
+```
+
+You should see a response showing that each batch was indexed successfully and we should have entries in Elasticsearch!
+
+```ruby
+CrashEvent.count
+#=> 5268
+```
+
+Let's use our scope to apply the aggregation and set size to 0 since we're only performing aggregations and don't want any document source
+```ruby
+response = CrashEvent.operator_fatalities.size(0)
+ap response.aggregations.operator_fatalities.buckets
+```
+
+```ruby
+=> [
+  {
+    "key"        => "Aeroflot",
+    "doc_count"  => 179,
+    "fatalities" => {
+      "value" => 7156.0
+    }
+  },
+  {
+    "key"        => "Military - U.S. Air Force",
+    "doc_count"  => 176,
+    "fatalities" => {
+      "value" => 3717.0
+    }
+  },
+  {
+    "key"        => "Air France",
+    "doc_count"  => 70,
+    "fatalities" => {
+      "value" => 1734.0
+    }
+  },
+  ...
+]
+```
+
+
+## Visualization
+_Coming soon.._

data/docs/examples/semantic_search_with_llm.md

@@ -0,0 +1,83 @@
+# Semantic Search with LLMs
+
+>[!INFO|style:flat|label:OpenSearch Only]
+> This guide only works with features found in Opensearch 2.12+
+
+**Prerequisites:**
+
+- Opensearch installed and running
+- Ruby on Rails or stretchy-model's `bin/console`
+
+Follow the [Quick Start](guides/quick-start) for detailed steps. 
+
+## Text Embeddings
+
+* OpenAI embeddings
+
+## Ingest Pipeline
+
+_Set up ingest pipeline with inference model_
+
+## Define Models
+```ruby
+class Repo < StretchyModel
+	attribute :name, :string
+	attribute :meta, :hash
+	attribute :files, :nested
+	has_many :files
+end
+```
+
+```ruby
+class RepoFile < StretchyModel
+	attribute :name, :string
+    attribute :path, :string
+	attribute :content, :text
+	attribute :file_embeddings, :dense_vector
+	attribute :method_definitions, :array
+	attribute :language, :string
+end
+```
+
+## Controller
+```ruby
+def search
+  query_text = params[:query_text] 
+
+  response = RepoFile.where(content: query_text)
+        .exists?(field: :file_embeddings)
+        .knn(
+          field: :file_embeddings,
+          k: 1,
+          num_candidates: 20,
+          query_vector_builder: {
+            text_embedding: {
+                model_id: :sentence_transformers,
+                model_text: query_text
+            }
+          },
+          boost: 24
+        )
+        .fields(:content, :name, :path)
+
+
+  negative_response = "Unable to find the answer to your question given the provided context."
+
+  prompt = "Answer this question: #{query_text}\nUsing only the information from this Elastic Doc: #{response}\nIf the answer is not contained in the supplied doc reply '#{negative_response}' and nothing else"
+
+  @answer = LLM.chat(prompt)
+end
+```
+
+## Views
+
+```slim
+
+div[data-controller="search"]
+  = simple_form :search do |f|
+    = f.input :query_text
+    = f.submit
+
+#responses data-controller="response"
+
+```