stretchy-model 0.6.0 → 0.6.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f23580fe0c8761ced02da75cbc44f16ab7893a559d30d9a5e0de23aa96ed790a
-  data.tar.gz: 821381d3e9e92822d71aac637d98f8b04a76736a80e421ca7d2c4fd05fd108d8
+  metadata.gz: ff8d7515eb6f795cf92989ba14d3dd404bb716378896ca2da4054bca98e467f4
+  data.tar.gz: 2361d39fcacb927f7d62bfe2105054c0923cd6b20af3cd7724c2bc9d784d42e5
 SHA512:
-  metadata.gz: 99dc36ee6cce021e869a8e1f3e2f005f546b1c50bf973434c624574c1dd601757baf4a1d39562f7ddd05bf4a1e640512111b821d8a70b8d5416bf73701b38f03
-  data.tar.gz: 4a58307bd7ae5a5781e7462df9c95438f66eecab5d0ae43238878d228e2ca6939a8a7643b70ac3c2586dcae537935206bfb15339415f6b3c3a70ef2518e401b4
+  metadata.gz: 410eb69810533c4a12fe12b1bdd5ea6df9b320f9c1953a32f924487b02f1825296765e5a5451abf1c22491a6d9e8a1741e4e7ef01789acbac0026e1db423602d
+  data.tar.gz: 9afdb5b058a307c3b64f8043f1e4f2348973050b6b5c802726031f060a44ac11003633ba816ab278ba204c1a4946a12f0c0c7171fd72cf0188493c43b3e0d4df

data/.yardopts

@@ -1 +1,2 @@
---markup markdown
+#  --markup markdown
+# --format=markdown

data/README.md

@@ -1,34 +1,43 @@
 stretchy-model
 ===
 <p>
-    <a href="https://stretchy.io/" target="_blank"><img src="./stretchy.logo.png" alt="Gum Image" width="450" /></a>
+    <a href="https://stretchy.io/" target="_blank"><img src="./stretchy.logo.png" alt="Stretchy Image" width="450" /></a>
     <br><br>
     <a href="https://github.com/theablefew/stretchy/releases"><img src="https://img.shields.io/github/v/release/theablefew/stretchy?sort=semver&color=blue"></a>
     <a href="https://github.com/theablefew/stretchy/actions"><img src="https://github.com/theablefew/stretchy/actions/workflows/spec.yml/badge.svg"></a>
 
 </p>
 
+Stretchy provides Elasticsearch/Opensearch models in Rails applications with an Rails-like model interface.
 
 ## Features
 Stretchy simplifies the process of querying, aggregating, and managing Elasticsearch-backed models, allowing Rails developers to work with search indices as comfortably as they would with traditional Rails models.
 
-* Model fully back by Elasticsearch/Opensearch
-* Chain queries, scopes and aggregations
+* Models fully back by Elasticsearch/Opensearch
+* Chain `queries`, `scopes` and `aggregations`
 * Reduce Elasticsearch query complexity
 * Support for time-based indices and aliases
-* Associations to both ActiveRecord models and Stretchy::Record
+* Associations to both ActiveRecord models and `StretchyModel`
 * Bulk Operations made easy
+* Ingest and Search Pipelines
+* Machine Learning
+* Vector and Neural search 
+* Integrated RAG and LLM connectors
 * Validations, custom attributes, and more...
 
 Follow the guides to learn more about:
-
 * [Models](https://theablefew.github.io/stretchy/#/guides/models?id=models)
 * [Querying](https://theablefew.github.io/stretchy/#/guides/querying?id=querying)
 * [Aggregations](https://theablefew.github.io/stretchy/#/guides/aggregations?id=aggregations)
 * [Scopes](https://theablefew.github.io/stretchy/#/guides/scopes?id=scopes)
+* [Pipelines](https://theablefew.github.io/stretchy/#/guides/pipelines?id=pipelines)
+* [Machine Learning](https://theablefew.github.io/stretchy/#/guides/machine-learning?id=machine-learning)
 
+[Read the Documentation](https://theablefew.github.io/stretchy/#/) or follow the examples below:
 
-[Read the Documentation](https://theablefew.github.io/stretchy/#/) or walk through of a simple [Data Analysis](https://theablefew.github.io/stretchy/#/examples/data_analysis?id=data-analysis) example.
+ **Examples**
+ - [Data Analysis](https://theablefew.github.io/stretchy/#/examples/data_analysis?id=data-analysis) example.
+ - [Simple Ingest Pipeline](https://theablefew.github.io/stretchy/#/examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
 
 
 
@@ -45,11 +54,15 @@ If bundler is not being used to manage dependencies, install the gem by executin
   gem install stretchy-model
 ```
 
+>[!TIP]
+> If using OpenSearch make sure to add the gem to your Gemfile.
+>
+> `bundle add opensearch-ruby`
+
 <details>
 <summary>Rails Configuration</summary>
 
 
-
 ```sh
 rails credentials:edit
 ```
@@ -59,19 +72,24 @@ rails credentials:edit
 elasticsearch:
    url: localhost:9200
 
-# or opensearch
+# or if using opensearch
 # opensearch:
 #    host: https://localhost:9200
 #    user: admin
 #    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
 ```
 
 #### Create an initializer 
 <p><sub><em>config/initializers/stretchy.rb</em></sub></p>
 
-```ruby {file=config/initializers/stretchy.rb}
+```ruby
 Stretchy.configure do |config|
-    config.client = Elasticsearch::Client.new url: Rails.application.credentials.elasticsearch.url, log: true
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using OpenSearch
+    # config.client = OpenSearch::Client.new Rails.application.credentials.opensearch
 end
 ```
 </details>

data/Rakefile

@@ -94,3 +94,59 @@ namespace :publish do
     Rake::Task['publish:release'].invoke('patch')
   end
 end
+
+
+namespace :documentation do
+
+  desc "Generate documentation"
+  task :generate do
+    system('rdoc --format=markdown --markup=markdown -o docs/doc --force-output -O')
+    # generate sidebar
+    Rake::Task['documentation:build_sidebar'].invoke
+    
+  end
+
+  desc "Build sidebar"
+  task :build_sidebar do
+    # Get all directories in docs/doc
+    directories = Dir.glob('docs/doc/**/*').select { |f| File.directory?(f) }
+  
+    # For each directory
+    directories.each do |dir|
+      puts "Building sidebar for #{dir}"
+      # Open _sidebar.md in write mode
+      File.open("#{dir}/_sidebar.md", 'w') do |file|
+        # Get all .md files in the directory
+        md_files = Dir.glob("#{dir}/*.md")
+  
+        # Add top level header with link to the directory
+        file.puts("- [#{File.basename(dir)}](#{dir.gsub('docs/', '')}.md)")
+        # For each .md file
+        md_files.each do |md_file|
+          next if md_file.include?('_sidebar.md')
+          # Write a markdown link to the file
+          md_file_path = md_file.gsub('docs/', '')
+          file_name = File.basename(md_file, '.md')
+          file.puts("  - [#{file_name}](#{md_file_path})")
+  
+
+          # Check for a directory with the same name as the file
+          if Dir.exist?("#{dir}/#{file_name}")
+            # Get all .md files in the subdirectory
+            sub_md_files = Dir.glob("#{dir}/#{file_name}/*.md")
+            # For each .md file in the subdirectory
+            # Put the parent directory name as a header
+            sub_md_files.each do |sub_md_file|
+              # Write a markdown link to the file
+              sub_md_file_path = sub_md_file.gsub('docs/', '')
+              sub_file_name = File.basename(sub_md_file, '.md')
+              next if sub_md_file.include?('_sidebar.md')
+              file.puts("    - [#{sub_file_name}](#{sub_md_file_path})")
+            end
+          end
+        end
+      end
+    end
+  end
+
+end

data/containers/Dockerfile.opensearch

@@ -2,7 +2,6 @@
 
 # Base image
 FROM opensearchproject/opensearch:2.12.0
-
 # Environment variables
 # disables bootstrap checks that are enabled when network.host is set to a non-loopback address
 ENV discovery.type=single-node
@@ -12,8 +11,10 @@ ENV plugins.security.disabled=true
 # along with the memlock settings below, disables swapping
 ENV bootstrap.memory_lock=true
 # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
-ENV OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m"
+ENV OPENSEARCH_JAVA_OPTS="-Xms3078m -Xmx3078m"
 # disables execution of install_demo_configuration.sh bundled with security plugin, which installs demo certificates and security configurations to OpenSearch
 ENV DISABLE_INSTALL_DEMO_CONFIG=true
 # disables security plugin entirely in OpenSearch by setting plugins.security.disabled: true in opensearch.yml
-ENV DISABLE_SECURITY_PLUGIN=true
+# ENV DISABLE_SECURITY_PLUGIN=true
+
+

data/docker-compose.yml

@@ -10,11 +10,16 @@ services:
       - "9200:9200"
     environment:
       - discovery.type=single-node
+    networks:
+      - elasticsearch-net
+
 
   opensearch:
-    build:
-      context: .
-      dockerfile: containers/Dockerfile.opensearch
+    image: opensearchproject/opensearch:latest
+    container_name: opensearch-node1
+    # build:
+    #   context: .
+    #   dockerfile: containers/Dockerfile.opensearch
     ulimits:
       memlock:
         soft: -1
@@ -24,29 +29,37 @@ services:
         hard: 65536
     environment:
       - discovery.type=single-node
+      - node.name=opensearch-node1
+      - plugins.security.disabled=true
+      - bootstrap.memory_lock=true
+      - "OPENSEARCH_JAVA_OPTS=-Xms3078m -Xmx3078m"
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=A3s0p3nS3cUr1tY
     # volumes:
     #   - opensearch-data1:/usr/share/opensearch/data
     ports:
       - 9200:9200
       - 9600:9600 # required for Performance Analyzer
-    # networks:
-    #   - opensearch-net
+    networks:
+      - opensearch-net
+
+  opensearch-dashboards:
+    image: opensearch-dashboards-no-security # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
+    container_name: opensearch-dashboards
+    ports:
+      - 5601:5601 # Map host port 5601 to container port 5601
+    expose:
+      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
+    environment:
+      - OPENSEARCH_HOSTS="http://opensearch-node1:9200"
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=A3s0p3nS3cUr1tY
+      - plugins.security.disabled=true
+    networks:
+      - opensearch-net
 
-  # opensearch-dashboards:
-  #   image: opensearchproject/opensearch-dashboards:1.2.0
-  #   container_name: opensearch-dashboards
-  #   ports:
-  #     - 5601:5601
-  #   expose:
-  #     - "5601"
-  #   environment:
-  #     - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]'
-  #     - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
-  #   networks:
-  #     - opensearch-net
 
 # volumes:
   # opensearch-data1:
 
-# networks:
-  # opensearch-net:
+networks:
+  opensearch-net:
+  elasticsearch-net:

data/docs/README.md

@@ -0,0 +1,147 @@
+stretchy-model
+===
+Stretchy provides Elasticsearch/Opensearch models in Rails applications with an Rails-like model interface.
+<p >
+    <a href="https://github.com/theablefew/stretchy/releases"><img src="https://img.shields.io/github/v/release/theablefew/stretchy?sort=semver&color=blue"></a>
+    <a href="https://github.com/theablefew/stretchy/actions"><img src="https://github.com/theablefew/stretchy/actions/workflows/spec.yml/badge.svg"></a>
+
+</p>
+
+## Features
+Stretchy simplifies the process of querying, aggregating, and managing Elasticsearch-backed models, allowing Rails developers to work with search indices as comfortably as they would with traditional Rails models.
+
+* Models fully back by Elasticsearch/Opensearch
+* Chain `queries`, `scopes` and `aggregations`
+* Reduce Elasticsearch query complexity
+* Support for time-based indices and aliases
+* Associations to both ActiveRecord models and `StretchyModel`
+* Bulk Operations made easy
+* Ingest and Search Pipelines
+* Machine Learning
+* Vector and Neural search 
+* Integrated RAG and LLM connectors
+* Validations, custom attributes, and more...
+
+Follow the guides to learn more about:
+* [Models](guides/models?id=models)
+* [Querying](guides/querying?id=querying)
+* [Aggregations](guides/aggregations?id=aggregations)
+* [Scopes](guides/scopes?id=scopes) 
+* [Pipelines](guides/pipelines?id=pipelines)
+* [Machine Learning](guides/machine-learning?id=machine-learning)
+
+**Examples**
+ - [Data Analysis](examples/data_analysis?id=data-analysis)
+ - [Simple Ingest Pipeline](examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
+
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+```sh
+bundle add stretchy-model
+```
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+```sh
+gem install stretchy-model
+```
+
+>[!INFO|style:flat]
+> If using OpenSearch add the gem to your Gemfile:
+>
+> ```
+> bundle add opensearch-ruby
+> ```
+
+#### Add credentials
+
+```sh
+rails credentials:edit
+```
+
+```yaml
+elasticsearch:
+   url: localhost:9200
+
+# or if using opensearch
+# opensearch:
+#    host: http://localhost:9200
+#    user: admin
+#    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
+```
+
+#### Create an initializer 
+*config/initializers/stretchy.rb*
+
+```ruby
+Stretchy.configure do |config|
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using OpenSearch
+    # config.client = OpenSearch::Client.new Rails.application.credentials.opensearch
+end
+```
+
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+
+## Testing
+<details>
+<summary>Act</summary>
+
+Run github action workflow locally
+
+```sh
+brew install act --HEAD
+```
+
+```sh
+act -P ubuntu-latest=ghcr.io/catthehacker/ubuntu:runner-latest
+```
+
+</details>
+
+<details>
+<summary>Elasticsearch</summary>
+
+
+```
+docker-compose up elasticsearch
+```
+
+```
+bundle exec rspec
+```
+
+</details>
+
+<details>
+<summary>Opensearch</summary>
+
+
+```
+docker-compose up opensearch
+```
+
+```
+ENV['BACKEND']=opensearch bundle rspec 
+```
+</details>
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/theablefew/stretchy. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/theablefew/stretchy/blob/master/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+

data/docs/_coverpage.md

@@ -0,0 +1,14 @@
+
+<img src="./stretchy.cover.png" alt="stretchy" width="650" />
+
+
+> Elasticsearch Models for Ruby on Rails
+
+- Chainable Queries and Aggregations
+- Integrated RAG and LLM Agents
+- Ingest Pipelines
+
+[GitHub](https://github.com/theablefew/stretchy)
+[Get Started](#stretchy-model)
+
+![color](#D0B6E1)

data/docs/_sidebar.md

@@ -0,0 +1,14 @@
+* [__Readme__](/)
+
+* __Guides__ 
+    * [Quick Start](guides/quick-start?id=quick-start)
+    * [Models](guides/models?id=models)
+    * [Querying](guides/querying?id=querying)
+    * [Scopes](guides/scopes?id=scopes)
+    * [Aggregations](guides/aggregations?id=aggregations)
+    * [Pipelines](guides/pipelines?id=pipelines)
+    * [Machine Learning](guides/machine-learning?id=machine-learning)
+
+* __Examples__
+    * [Data Analysis](examples/data_analysis)
+    * [Simple Ingest Pipeline](examples/simple-ingest-pipeline)

data/docs/examples/_sidebar.md

@@ -0,0 +1,15 @@
+* [__Readme__](/)
+
+* __Guides__ 
+    * [Quick Start](guides/quick-start?id=quick-start)
+    * [Models](guides/models?id=models)
+    * [Querying](guides/querying?id=querying)
+    * [Scopes](guides/scopes?id=scopes)
+    * [Aggregations](guides/aggregations?id=aggregations)
+    * [Pipelines](guides/pipelines?id=pipelines)
+    * [Machine Learning](guides/machine-learning?id=machine-learning)
+
+* __Examples__
+    * [Data Analysis](examples/data_analysis)
+    * [Simple Ingest Pipeline](examples/simple-ingest-pipeline?id=simple-ingest-pipeline)
+    * [Semantic Search with LLMs](examples/semantic_search_with_llm)

data/docs/examples/data_analysis.md

@@ -0,0 +1,216 @@
+# Data Analysis
+
+This guide will walk you through setting up a basic Rails application using Stretchy and Elasticsearch to ingest and visualize time series data.
+
+## Prerequisites
+
+- Ruby and Rails installed on your machine
+- Elasticsearch installed and running
+
+
+
+__Start Elasticsearch:__
+
+```sh
+docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.0
+```
+
+## Step 1: Create a New Rails Application
+
+```sh
+rails new time_series_app
+cd time_series_app
+```
+
+## Step 2: Add `stretchy-model` to Your Gemfile
+```sh
+bundle add stretchy-model   
+```
+## Step 3: Configure Stretchy
+#### Add credentials
+```sh
+rails credentials:edit
+```
+
+```yaml
+elasticsearch:
+   url: localhost:9200
+
+# or opensearch
+# opensearch:
+#    host: https://localhost:9200
+#    user: admin
+#    password: admin
+#    transport_options:
+#       ssl:
+#         verify: false
+```
+
+> __NOTE__ If using Opensearch be sure to run:
+> ```sh
+>   bundle add opensearch
+> ```
+
+Create an initializer file config/initializers/stretchy.rb and add the following:
+```ruby
+Stretchy.configure do |config|
+    config.client = Elasticsearch::Client.new Rails.application.credentials.elasticsearch
+    # or if using Opensearch
+    # config.client = Openseaerch::Client.new Rails.application.credentials.opensearch
+end
+```
+## Identify a Data Set
+
+Let's use "Historical Plane Crashes Since 1908" for our dataset. The following will download the csv, transform the headers and output a json file. 
+
+Open a Rails console:
+```sh
+rails c
+```
+
+```ruby
+require 'open-uri'
+require 'csv'
+
+url = 'https://huggingface.co/datasets/nateraw/airplane-crashes-and-fatalities/resolve/main/Airplane_Crashes_and_Fatalities_Since_1908.csv?download=true'
+download = URI.open(url)
+scrub_headers = lambda { |header| header.parameterize.underscore }
+csv = CSV.parse(download.read, headers: true, header_converters: scrub_headers)
+data = csv.map(&:to_hash)
+data.each { |d| d.merge!({'occurred_on': "#{d.delete('date')} #{d.delete('time')}"}); d.delete('index') }
+
+File.open('airplane_crashes.json', 'w') do |f|
+  f.write(JSON.pretty_generate(data))
+end
+```
+
+Now, each entry has the following schema:
+```json
+{
+        "location": "Fort Myer, Virginia",
+        "operator": "Military - U.S. Army",
+          "flight": null,
+           "route": "Demonstration",
+            "type": "Wright Flyer III",
+    "registration": null,
+           "cn_in": "1",
+          "aboard": "2.0",
+      "fatalities": "1.0",
+          "ground": "0.0",
+         "summary": "During a demonstration flight, a U.S. Army flyer flown by Orville Wright nose-dived into the ground from a height of approximately 75 feet, killing Lt. Thomas E. Selfridge who was a passenger. This was the first recorded airplane fatality in history.  One of two propellers separated in flight, tearing loose the wires bracing the rudder and causing the loss of control of the aircraft.  Orville Wright suffered broken ribs, pelvis and a leg.  Selfridge suffered a crushed skull and died a short time later.",
+      "occured_on": "09/17/1908 17:18"
+}
+```
+
+## Understanding Performance Benefits
+## Define a Index Strategy
+
+Since we have a date field, we can store this data across multiple indexes. `stretchy-model` makes it easy to facilitate this strategy, which is useful when you have a rolling window of data. 
+
+## Determine Routing Strategy
+
+Routing determines which shard in the cluster the document will be written routed to. For large data sets it's a good idea to choose a routing key that groups similar queried data together. It all depends on your needs and your data distribution. For example purposes we'll route based on the operator.
+
+## Define CrashEvent Model
+Create a file for `CrashEvent` at `app/models/crash_event.rb`
+
+```ruby
+class CrashEvent < StretchyModel
+
+  attribute :occurred_on, :datetime, model_format: '%m/%d/%Y'
+  attribute :location, :keyword
+  attribute :operator, :keyword
+  attribute :flight, :keyword
+  attribute :route, :keyword
+  attribute :type, :keyword
+  attribute :registration, :keyword
+  attribute :cn_in, :keyword
+  attribute :aboard, :float
+  attribute :fatalities, :float
+  attribute :ground, :float
+  attribute :summary, :text
+
+  scope :operator_fatalities, -> { aggregation(:operator_fatalities, 
+      {
+        terms: {
+          field: :operator
+        }, 
+        aggs: {
+          fatalities: {
+            sum: {
+              field: :fatalities
+            }
+          }
+        }
+      }
+    ) 
+  }
+
+  scope :date_histogram, ->(name, options, *aggs) do
+    # requires field and calendar_interval or interval
+    aggregation(name, {date_histogram: options}.merge(*aggs))
+  end
+
+end
+
+```
+
+
+## Bulk Index Documents
+
+
+Reload your Rails console and start the bulk index operation:
+
+```ruby
+data = JSON.parse(File.read('/Users/spencer/Downloads/airplane_crashes.json'))
+CrashEvent.bulk_in_batches(data, size: 100) do |batch|
+  puts "Processing batch: #{batch.length}"
+  batch.map! do |event|
+    CrashEvent.new(event.symbolize_keys).to_bulk
+  end
+end
+```
+
+You should see a response showing that each batch was indexed successfully and we should have entries in Elasticsearch!
+
+```ruby
+CrashEvent.count
+#=> 5268
+```
+
+Let's use our scope to apply the aggregation and set size to 0 since we're only performing aggregations and don't want any document source
+```ruby
+response = CrashEvent.operator_fatalities.size(0)
+ap response.aggregations.operator_fatalities.buckets
+```
+
+```ruby
+=> [
+  {
+    "key"        => "Aeroflot",
+    "doc_count"  => 179,
+    "fatalities" => {
+      "value" => 7156.0
+    }
+  },
+  {
+    "key"        => "Military - U.S. Air Force",
+    "doc_count"  => 176,
+    "fatalities" => {
+      "value" => 3717.0
+    }
+  },
+  {
+    "key"        => "Air France",
+    "doc_count"  => 70,
+    "fatalities" => {
+      "value" => 1734.0
+    }
+  },
+  ...
+]
+```
+
+
+## Visualization
+_Coming soon.._