stretchy-model 0.6.5 → 0.6.6

data/docs/examples/simple-ingest-pipeline.md

@@ -0,0 +1,326 @@
+# Simple Ingest Pipeline
+
+>[!INFO|style:flat:label:Prerequisites]
+>
+>- Opensearch 2.12+ installed and running
+>- Ruby on Rails
+>
+>Follow the [Quick Start](guides/quick-start) for detailed steps. 
+
+## Data Source
+
+Our data source is a JSON data representing vitals and patient information scraped from the dark web (jk). We have an id, vitals as a CSV, full name, age and a SSN with HTML tags. What a mess!
+
+
+| id      | vitals     | name                       | age | ssn                |
+| ------- | ---------- | -------------------------- | --- | ------------------ |
+| ta0j288 | 700,120,72 | Gov. Candy Williams        | 30  | <b>547-93-4227</b> |
+| ta0j288 | 56,120,72  | Romana Prohaska            | 30  | <b>547-93-4227</b> |
+| pnunl70 | 114,136,43 | Tristan Waelchi            | 81  | <b>323-23-5997</b> |
+| 8lhscax | 105,66,56  | Antoine Hauck              | 46  | <b>381-54-5352</b> |
+| impcbo9 | 119,78,60  | Dewayne Stark              | 39  | <b>816-86-6698</b> |
+| jxr8h3v | 81,69,58   | Shelton Powlowski          | 77  | <b>810-63-7478</b> |
+| d7lwaln | 103,140,93 | Sage Medhurst              | 19  | <b>470-43-3841</b> |
+| ryrtjb5 | 57,118,86  | Tobias Strosin             | 76  | <b>197-25-4397</b> |
+| ox227l3 | 82,103,98  | Jessi Barton               | 41  | <b>700-41-0042</b> |
+| c7vyqu2 | 103,73,90  | Eliseo Feest               | 53  | <b>153-01-6678</b> |
+| i8lbviz | 81,120,91  | The Hon. Zandra Dibbert MD | 55  | <b>881-10-7835</b> |
+
+Our goal is to create an ingest pipeline using `stretchy-model` to process this data. The pipeline will transform the vitals from a CSV into an array, remove the HTML tags from the SSN, and split the full name into first and last names.
+
+## Define the Pipeline
+
+An ingest pipeline in Elasticsearch allows us to pre-process documents before the actual document indexing occurs. It's a way to transform and enrich the data, making it more useful and easier to work with.
+
+```mermaid
+flowchart LR
+
+CSV --> SCRIPT
+
+SCRIPT --> HTML_STRIP
+
+HTML_STRIP --> CONVERT
+
+CONVERT --> REMOVE
+
+REMOVE --> INDEX
+
+```
+By doing these transformations as part of the ingest process, we ensure that the data is in the right format and structure for our needs right from the moment it enters Elasticsearch. This makes our subsequent data handling and analysis tasks much easier and more efficient.
+
+_app/pipelines/intake_form_pipeline.rb_
+
+```ruby
+class IntakeFormPipeline < Stretchy::Pipeline
+
+	description "Ingests intake forms and scrubs ssn of html"
+	  
+	processor :csv, 
+		field: :vitals, 
+		target_fields: [:heart_rate, :systolic, :diastolic],
+		trim: true
+		
+	processor :script, 
+		description: "Extracts first and last name from name field",
+		lang: "painless",
+		source: <<~PAINLESS
+			ctx['name'] = /^[\\w\\s]+\\.\\s/.matcher(ctx['name']).replaceAll("");
+			String[] parts = /\\s+/.split(ctx['name']);
+			ctx['first_name'] = parts[0];
+			if (parts.length > 1) {
+			  ctx['last_name'] = parts[1];
+			}
+		PAINLESS
+		
+	processor :html_strip, field: :ssn
+	processor :convert, field: :systolic, type: :integer
+	processor :convert, field: :diastolic, type: :integer
+	processor :convert, field: :heart_rate, type: :integer
+ 
+	processor :remove, field: :name
+	processor :remove, field: :vitals
+    
+end
+```
+
+The `IntakeFormPipeline` will preprocess documents that are sent to be indexed. We have a `description` and a series of `processor` statements, each performing a specific transformation on the data:
+
+- **csv** - parse `vitals` and map them to `heart_rate`, `systolic` and `diastolic` fields
+- **script** - split `name` into `first_name` and `last_name`, removing any titles like Dr., Rev. etc. 
+- **html_strip** - scrub `ssn` of any HTML tags
+- **convert** - ensure vitals are all integers
+- **remove** - remove the fields we no longer need
+
+**Create the pipeline:**
+
+This command sends a request to the Elasticsearch server to create a new ingest pipeline with the specifications defined in the IntakeFormPipeline class.
+
+Once the pipeline is created, it's ready to preprocess any documents that are sent to be indexed. The transformations defined in the pipeline (such as parsing CSVs, splitting names, stripping HTML tags, and converting fields to integers) will be applied to each document before it's indexed.
+
+```ruby
+IntakeFormPipeline.create!
+```
+
+**Response:**
+```ruby
+#=> {"acknowledged"=>true} 
+```
+
+Remember, the pipeline only needs to be created once. After it's created, it will be used automatically whenever documents are indexed in Elasticsearch. If you need to change the pipeline, you can remove it with the `IntakeFormPipeline.delete! command.
+
+## Describe the Model
+
+The `IntakeForm` model represents the index were we’ll store our intake forms. 
+
+*app/models/intake_form.rb*
+
+```ruby
+class IntakeForm < StretchyModel
+  attribute :first_name, :keyword
+  attribute :last_name, :keyword
+  attribute :ssn, :keyword
+  attribute :age, :integer
+  attribute :heart_rate, :integer
+  attribute :systolic, :integer
+  attribute :diastolic, :integer
+
+  default_pipeline :intake_form_pipeline
+end
+```
+The IntakeForm model inherits from `StretchyModel`, which means it gets all the functionality provided by `StretchyModel`.
+
+The `attribute` method is used to define the fields of the IntakeForm model. Each `attribute` has a name and a type. The type corresponds to the Elasticsearch field type.
+
+The `default_pipeline` method sets the default ingest pipeline for the model. In this case, it's set to `:intake_form_pipeline`, which means that the intake_form_pipeline will be used to preprocess documents before they are indexed.
+
+
+**Create the index:**
+
+```ruby
+IntakeForm.create_index!
+```
+
+**Response:**
+```ruby
+#=> {"acknowledged"=>true, "shards_acknowledged"=>true, "index"=>"intake_forms"} 
+```
+
+## Run the pipeline
+
+To run the pipeline, you'll need to index documents using the `IntakeForm` model. The `default_pipeline` you set earlier will automatically preprocess the documents before they are indexed.
+
+```ruby
+initial_data = [
+    {"id": "ta0j288", "vitals": "700,120,72", "name": "Gov. Candy Williams", "age": 30, "ssn": "<b>547-93-4227</b>"},
+    {"id": "ta0j288", "vitals": "56,120,72", "name": "Romana Prohaska", "age": 30, "ssn": "<b>547-93-4227</b>"},
+    {"id": "pnunl70", "vitals": "114,136,43", "name": "Tristan Waelchi", "age": 81, "ssn": "<b>323-23-5997</b>"},
+    {"id": "8lhscax", "vitals": "105,66,56", "name": "Antoine Hauck", "age": 46, "ssn": "<b>381-54-5352</b>"},
+    {"id": "impcbo9", "vitals": "119,78,60", "name": "Dewayne Stark", "age": 39, "ssn": "<b>816-86-6698</b>"},
+    {"id": "jxr8h3v", "vitals": "81,69,58", "name": "Shelton Powlowski", "age": 77, "ssn": "<b>810-63-7478</b>"},
+    {"id": "d7lwaln", "vitals": "103,140,93", "name": "Sage Medhurst", "age": 19, "ssn": "<b>470-43-3841</b>"},
+    {"id": "ryrtjb5", "vitals": "57,118,86", "name": "Tobias Strosin", "age": 76, "ssn": "<b>197-25-4397</b>"},
+    {"id": "ox227l3", "vitals": "82,103,98", "name": "Jessi Barton", "age": 41, "ssn": "<b>700-41-0042</b>"},
+    {"id": "c7vyqu2", "vitals": "103,73,90", "name": "Eliseo Feest", "age": 53, "ssn": "<b>153-01-6678</b>"},
+    {"id": "i8lbviz", "vitals": "81,120,91", "name": "The Hon. Zandra Dibbert MD", "age": 55, "ssn": "<b>881-10-7835</b>"}
+]
+```
+
+#### Simulate
+We can simulate `IntakeFormPipeline` to make sure it works as expected. 
+
+```ruby
+docs = initial_data.map {|doc| {_source: doc} }
+```
+
+We prepare the initial data by making sure each entry has a `_source` field with the document as the value. This is slightly different than how we'll prepare the data for actual indexing. 
+
+**Simulate the pipeline:**
+
+```ruby
+IntakeFormPipeline.simulate(docs)
+```
+
+**Response:**
+```ruby
+#=> {"docs"=>
+#  [{"processor_results"=>
+#     [{"processor_type"=>"csv",
+#       "status"=>"success",
+#       "doc"=>
+#        {"_index"=>"_index",
+#         "_type"=>"_doc",
+#         "_id"=>"_id",
+#         "_source"=>{"systolic"=>"120", "diastolic"=>"72", "name"=>"Gov. Candy Williams", #"heart_rate"=>"700", "id"=>"ta0j288", "vitals"=>"700,120,72", "age"=>30, "ssn"=>"<b>547-93-4227</#b>"},
+#         "_ingest"=>{"pipeline"=>"intake_form_pipeline", "timestamp"=>"2024-03-20T13:06:17.661745464Z"}}},
+#      {"processor_type"=>"script",
+#       "status"=>"success",
+#       "description"=>"Extracts first and last name from name field",
+#       "doc"=>
+#        {"_index"=>"_index",
+#         "_type"=>"_doc",
+#         "_id"=>"_id",
+#         "_source"=>{"heart_rate"=>"700", "last_name"=>"Williams", "ssn"=>"<b>547-93-4227</b>", #"systolic"=>"120", "diastolic"=>"72", "name"=>"Candy Williams", "id"=>"ta0j288", #"first_name"=>"Candy", "vitals"=>"700,120,72", "age"=>30},
+#         "_ingest"=>{"pipeline"=>"intake_form_pipeline", "timestamp"=>"2024-03-20T13:06:17.661745464Z"}}},
+# ...
+```
+
+The response should show the results of simulation, with each processor step and it’s status.  
+
+#### Ingest
+
+Now, let’s ingest the data into the index. We’ll use a bulk request to index our documents:
+
+```ruby 
+bulk_records = initial_data.map do |data|
+	{ index: { _index: IntakeForm.index_name, data: data } }
+end
+
+IntakeForm.bulk(bulk_records)
+```
+
+**Response:**
+```ruby
+ => 
+{"took"=>3,
+ "ingest_took"=>1,
+ "errors"=>false,
+ "items"=>
+  [{"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"vzz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>0, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"wDz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>1, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"wTz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>2, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"wjz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>3, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"wzz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>4, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"xDz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>5, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"xTz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>6, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"xjz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>7, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"xzz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>8, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"yDz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>9, "_primary_term"=>1, "status"=>201}},
+   {"index"=>{"_index"=>"intake_forms", "_type"=>"_doc", "_id"=>"yTz8W44BuORXSU88oIRr", "_version"=>1, "result"=>"created", "_shards"=>{"total"=>2, "successful"=>1, "failed"=>0}, "_seq_no"=>10, "_primary_term"=>1, "status"=>201}}]} 
+
+```
+
+Our ingest pipeline will perform all of the operations we defined as `processors` in the `IntakeFormPipeline` and index the resulting document. 
+
+Let's see how it did:
+
+```ruby
+IntakeForm.count
+#=> 11
+
+IntakeForm.first.heart_rate
+#=> 700
+```
+
+Wow! The Gov. must be having as much fun as us with a heart rate like that. 
+
+Let's get the average heart rate per age group:
+
+```ruby
+results = IntakeForm.range(:ages, {
+        field: :age, 
+        ranges: [
+	        {from: 19, to: 39}, 
+	        {from: 40, to: 59}, 
+	        {from: 60, to: 79}, 
+	        {from: 80}
+	    ],
+        keyed: true
+      }, 
+      aggs: {avg_heart_rate: {avg: {field: :heart_rate}}}).size(0)
+
+ap results.aggregations.ages.buckets
+```
+
+**Response:**
+```ruby
+
+{
+  "19.0-39.0" => {
+    "from"           => 19.0,
+    "to"             => 39.0,
+    "doc_count"      => 3,
+    "avg_heart_rate" => {
+      "value" => 286.3333333333333
+    }
+  },
+  "40.0-59.0" => {
+    "from"           => 40.0,
+    "to"             => 59.0,
+    "doc_count"      => 4,
+    "avg_heart_rate" => {
+      "value" => 92.75
+    }
+  },
+  "60.0-79.0" => {
+    "from"           => 60.0,
+    "to"             => 79.0,
+    "doc_count"      => 2,
+    "avg_heart_rate" => {
+      "value" => 69.0
+    }
+  },
+  "80.0-*"    => {
+    "from"           => 80.0,
+    "doc_count"      => 1,
+    "avg_heart_rate" => {
+      "value" => 114.0
+    }
+  }
+}
+```
+
+In this guide, we've walked through the process of creating an ingest pipeline with Elasticsearch using `stretchy-model`.
+
+We started with a dataset of patient information, which included fields that needed preprocessing before indexing. We defined an ingest pipeline, `IntakeFormPipeline`, that transformed the data into a more useful format, including parsing CSVs, splitting names, removing HTML tags, and converting fields to integers.
+
+We then used the `IntakeForm` model, which inherits from `StretchyModel`, to index the preprocessed data in Elasticsearch. We also demonstrated how to run aggregations on the indexed data to get insights, such as the average heart rate per age group.
+
+This is a simple example, but ingest pipelines can be much more complex and powerful, allowing you to preprocess your data in many different ways before indexing. With `stretchy-model`, you can leverage the full power of Elasticsearch's ingest pipelines while writing Ruby code that feels familiar and idiomatic.
+
+## Cleaning up
+
+```ruby
+IntakeForm.delete_index!
+IntakeFormPipeline.delete!
+```
+

data/docs/guides/_sidebar.md

@@ -0,0 +1,14 @@
+* [__Readme__](/)
+
+* __Guides__
+    * [Quick Start](guides/quick-start?id=quick-start)
+    * [Models](guides/models?id=models)
+    * [Querying](guides/querying?id=querying)
+    * [Scopes](guides/scopes?id=scopes)
+    * [Aggregations](guides/aggregations?id=aggregations)
+    * [Pipelines](guides/pipelines?id=pipelines)
+    * [Machine Learning](guides/machine-learning?id=machine-learning)
+
+* __Examples__
+    * [Data Analysis](examples/data_analysis)
+    * [Simple Ingest Pipeline](examples/simple-ingest-pipeline)

data/docs/guides/aggregations.md

@@ -0,0 +1,142 @@
+# Aggregations
+
+Aggregations in Elasticsearch allow you to get summary information about your data. For example, you can use aggregations to count the number of records that match certain criteria, calculate the average value of a field, find the minimum or maximum value, and more.
+
+
+When performing aggregations it's good practice to set `size(0)` if you don't need the source documents.
+
+```ruby
+results = Profile.aggregation(:flagged_counts, terms: {field: :flagged}).size(0)
+```
+
+Aggregation results are available on the results aggregations object by the name provided to the aggregation:
+
+```ruby
+results.aggregations.flagged_counts
+```
+
+returns:
+```ruby
+{
+  "doc_count_error_upper_bound"=>0,
+  "sum_other_doc_count"=>0,
+  "buckets"=>[
+    {"key"=>"true", "doc_count"=>123},
+    {"key"=>"false", "doc_count"=>456}
+  ]
+}
+```
+
+>[!TIP|label:Accessing Aggregation Results] 
+>You can access the entire structure through dot notation. 
+>
+>`aggregations.flagged_counts.buckets.first.doc_count` => `123`
+
+
+--- 
+
+In Stretchy, you use the `aggregation` method to define aggregations. Here are some examples:
+
+### Count by status 
+
+if you have a `status` field and you want to count how many records there are for each status, you can use a terms aggregation:
+```ruby
+profile.aggregation(:status_count, terms: { field: :status })
+```
+
+
+### Average Age
+
+If you have an age field and you want to calculate the average age, you can do this:
+
+```ruby
+Profile.aggregation(:average_age, avg: { field: :age })
+```
+
+### Minimum and Maximum Age
+
+If you want to find the minimum and maximum age, you can do this:
+
+```ruby
+Profile.aggregation(:min_age, min: { field: :age })
+Profile.aggregation(:max_age, max: { field: :age })
+```
+
+### Date Histogram
+
+If you have a `created_at` field and you want to count how many profiles were created in each month, you can do this:
+```ruby
+Profile.aggregation(:profiles_over_time, date_histogram: { field: :created_at, interval: 'month' })
+```
+
+In these examples, the first argument to the aggregation method is the name of the aggregation, and the second argument is a hash that defines the aggregation. The key of the hash is the type of the aggregation (terms, avg, min, max, or date_histogram), and the value is another hash that specifies the field to aggregate on and other options.
+
+## Named Aggregation Helpers
+
+The above shows how to use the `aggregation` method directly, but Stretchy makes working with named aggregations even easier. Named aggregation helpers make calling the aggregation you want a breeze.
+
+The documentation goes into depth for all available [aggregation types](/doc/stretchy/relations/AggregationMethods)
+
+### Percentiles
+
+The percentiles aggregation method calculates the percentiles of a numeric field. For example, if you want to calculate the 25th, 50th, and 75th percentiles of the age field, you can do this:
+```ruby
+Profile.percentiles(:age_percentiles, field: :age, percents: [25, 50, 75])
+```
+
+### Extended Stats
+
+The extended_stats aggregation method calculates several statistical measures of a numeric field, including the count, min, max, sum, average, sum of squares, variance, standard deviation, and bounds. For example, if you want to calculate these measures for the age field, you can do this:
+
+```ruby
+Profile.extended_stats(:age_stats, field: :age)
+```
+
+### Date Range
+
+The date_range aggregation method groups documents by whether their date field falls within specified ranges. For example, if you want to count how many profiles were created before and after a certain date, you can do this:
+```ruby
+Profile.date_range(:created_at_range, field: :created_at, ranges: [{ to: '2022-01-01' }, { from: '2022-01-01' }])
+```
+
+### Significant Terms
+
+The significant_terms aggregation method finds the terms that appear more often in the documents that match your query than in the documents that don't. For example, if you want to find the tags that are significantly associated with profiles that have a status of "active", you can do this:
+```ruby
+Profile.where(status: 'active').significant_terms(:significant_tags, field: :tags)
+```
+In these examples, the first argument to the aggregation method is the name of the aggregation, and the second argument is a hash that specifies the field to aggregate on and other options. The exact options depend on the aggregation method.
+
+## Nested Aggregations
+
+Elasticsearch supports complex aggregations by allowing you to nest sub-aggregations within top-level aggregations. These sub-aggregations operate within the context of the parent aggregation, allowing you to refine and group your data in various ways.
+
+There are three main types of aggregations in Elasticsearch: bucket, metric, and pipeline aggregations.
+
+#### Bucket Aggregations 
+These aggregations create buckets or sets of documents based on certain criteria. Examples include `terms`, `date_histogram`, `range`, and `significant_terms` aggregations. Each bucket effectively defines a document set, and any sub-aggregations operate within the context of that set.
+
+For example, you could use a terms aggregation to group documents by the status field, and then use a sub-aggregation to calculate the average age within each status group:
+```ruby
+Profile.aggregation(:status_avg_age, terms: { field: :status }, aggs: { avg_age: { avg: { field: :age } } })
+```
+
+#### Metric Aggregations
+ These aggregations perform calculations on the documents in each bucket, producing a single numeric result. Examples include `avg`, `sum`, `min`, `max`, and `extended_stats`.
+
+For example, you could use a terms aggregation to group documents by the status field, and then use a max sub-aggregation to find the maximum age within each status group:
+```ruby
+Profile.aggregation(:status_max_age, terms: { field: :status }, aggs: { max_age: { max: { field: :age } } })
+```
+
+#### Pipeline Aggregations
+These aggregations perform calculations on the results of other aggregations, allowing you to create complex summaries of your data. Examples include `avg_bucket`, `sum_bucket`, `min_bucket`, `max_bucket`, and `stats_bucket`.
+
+For example, you could use a date_histogram aggregation to count documents by month, and then use a sum_bucket sub-aggregation to calculate the total count over all months:
+```ruby
+Profile.aggregation(:total_count_over_time, date_histogram: { field: :created_at, interval: 'month' }, aggs: { total_count: { sum_bucket: { buckets_path: '_count' } } })
+# or
+Profile.date_histogram(:total_count_over_time, {field: :created_at, interval: :month}, aggs: {total_count: { sum_bucket: { buckets_path: '_count' } } })
+```
+
+In these examples, the aggs option is used to define sub-aggregations. The key is the name of the sub-aggregation, and the value is a hash that defines the sub-aggregation.

data/docs/guides/machine-learning.md

@@ -0,0 +1,154 @@
+# Machine Learning
+>[!NOTE|style:flat|label:OpenSearch Compatability]
+> OpenSearch and Elasticsearch diverge in how they handle machine learning APIs. These features are in active development and subject to change.
+>
+> This guide largely covers OpenSearch Machine Learning unless otherwise stated. 
+
+>[!WARNING|label:Machine Learning on Elasticsearch]
+> Elasticsearch requires a license to enable ML capabilities
+
+## Models
+Machine Learning models follow a specific convention for storing model definitions. This helps us keep our code organized and easy to navigate.
+
+- *app/machine_learning/models/example_machine_learning_model.rb*
+
+A `MachineLearningModel` consists of the following components:
+
+```ruby
+class SparseEncodingModel < Stretchy::MachineLearning::Model
+  model: :neural_sparse_encoding,
+          version: '1.0.1',
+          model_format: 'TORCH_SCRIPT',
+          description: 'Creates sparse embedding for onboarding docs'
+end
+```
+- `model:` This is the name of the model. It should match one of the pre-trained models available in your application. In this case, it's :neural_sparse_encoding.
+
+  - `version:` This is the version of the model. It's important to specify this, as different versions of the same model may have different behaviors or requirements.
+
+  - `model_format:` This is the format of the model. It tells Stretchy how to interpret the model file. In this case, it's 'TORCH_SCRIPT', which means the model is a TorchScript file. TorchScript is a way to serialize PyTorch models.
+
+  - `description:` This is a brief description of what the model does. It's a good practice to provide a meaningful description so that others can understand the purpose of your model at a glance. In this case, the description is 'Creates sparse embedding for onboarding docs'.
+
+
+
+## Managing Models
+
+>[!TIP|label:Machine Learning Settings]
+> When running development or single-node clusters you may need to adjust your cluster settings to allow Machine Learning models to run on all nodes instead of dedicated machine learning nodes.
+> Add `Stretchy::MachineLearning::Model.ml_on_all_nodes!` to your *config/environments/development.rb* file to enable machine learning on all nodes. 
+
+### register
+Registers the machine learning model. 
+
+```ruby
+MyMachineLearningModel.register
+```
+The `register` operation is asynchronous and can take some time to complete. To wait until the operation is complete use the helper method `wait_until_complete` in combination with the `registered?` method:
+```ruby
+MyMachineLearningModel.register do |model|
+  model.wait_until_complete do
+    model.registered?
+  end
+end
+```
+
+### registered?
+Checks the model status and returns true if `model_id` is present and `state` is `COMPLETED`
+
+```ruby
+MyMachineLearningModel.registered?
+```
+
+### status
+Returns the status of the model registration
+
+```ruby
+MyMachineLearningModel.status
+```
+
+### deploy
+Deploys the model making it available for use. Requires the model to be registered. 
+
+```ruby
+MyMachineLearningModel.deploy
+```
+
+The `deploy` operation is asynchronous and can take some time to complete. Use the `wait_until_complete` method in combination with `deployed?` to wait until the model is deployed. 
+
+```ruby
+MyMachineLearningModel.deploy do |model|
+  model.wait_until_complete(sleep_time: 5) do
+    model.deployed?
+  end
+end
+```
+
+### undeploy
+Undeploys the model. 
+
+```ruby
+MyMachineLearningModel.undeploy
+```
+
+### deployed?
+Gets the model and checks if `model_state` is `DEPLOYED`
+
+```ruby
+MyMachineLearningModel.deployed?
+```
+
+### delete
+Deletes the model. The model must be undeployed before it can be deleted. 
+
+```ruby
+MyMachineLearningModel.delete
+```
+
+### wait_until_complete
+A helper to provide wait for completion of async tasks. Accepts `max_attempts` and `sleep_time`. 
+
+```ruby
+MyMLModel.register do |model|
+  model.wait_until_complete(max_attempts: 20, sleep_time: 4) do
+    # finish waiting if last statement is true
+    model.registered?
+  end
+end
+```
+
+### all
+Returns all registered models.
+
+```ruby
+MyMLModel.all
+```
+
+## Pre-trained models
+OpenSearch provides a variety of pre-trained models for different tasks:
+
+### Neural Sparse Models
+- `:neural_sparse_encoding` - 'amazon/neural-sparse/opensearch-neural-sparse-encoding-v1'
+- `:neural_sparse_encoding_doc` - 'amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1'
+- `:neural_sparse_tokenizer` - 'amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1'
+
+### Cross Encoder Models
+- `:cross_encoder_minilm_6` - 'huggingface/cross-encoders/ms-marco-MiniLM-L-6-v2'
+- `:cross_encoder_minilm_12` - 'huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2'
+
+### Sentence Transformer Models
+- `:sentence_transformers_roberta_all` - 'huggingface/sentence-transformers/all-distilroberta-v1'
+- `:sentence_transformers_msmarco` - 'huggingface/sentence-transformers/msmarco-distilroberta-base-v2'
+- `:sentence_transformers_minilm_6` - 'huggingface/sentence-transformers/all-MiniLM-L6-v2'
+- `:sentence_transformers_minilm_12` - 'huggingface/sentence-transformers/all-MiniLM-L12-v2'
+- `:sentence_transformers_mpnet` - 'huggingface/sentence-transformers/all-mpnet-base-v'
+- `:sentence_transformers_multi_qa_minilm_6` - 'huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1'
+- `:sentence_transformers_multi_qa_mpnet` - 'huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1'
+- `:sentence_transformers_paraphrase_minilm_3` - 'huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2'
+- `:sentence_transformers_paraphrase_multilingual_minilm_12` - 'huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2'
+- `:sentence_transformers_paraphrase_mpnet` - 'huggingface/sentence-transformers/paraphrase-mpnet-base-v2'
+- `:sentence_transformers_multilingual_distiluse_cased` - 'huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1'
+
+## Custom Models
+
+Refer to the OpenSearch documentation on [deploying custom local models](https://opensearch.org/docs/latest/ml-commons-plugin/custom-local-models/)