stretchy-model 0.6.5 → 0.7.0

data/docs/guides/models.md

@@ -0,0 +1,372 @@
+# Models
+
+
+## Creating Stretchy Documents
+
+Create your models in app/models and subclass the `StretchyModel` class. 
+
+```ruby
+class Profile < StretchyModel
+end
+```
+
+The index name `profiles` will be inferred from the `Profile` model. 
+
+### Overriding Naming Conventions
+
+If you need a different naming convention for your indexes, you can override the default conventions.
+
+```ruby
+class Profile < StretchyModel
+    index_name 'user_profiles'
+end
+```
+
+### Default Sorting
+
+By default Stretchy Models are sorted by `created_at`. If you have another field you'd like to sort by you can easily change it. 
+
+
+```ruby
+class Profile < StretchyModel
+     default_sort_key :updated_at
+end
+```
+
+
+## Attributes
+
+
+Attributes are defined using `Stretchy::Attributes::Type` attributes. 
+
+* `:array` - Used to store multiple values in a single field.
+* `:binary` - Used to store binary data as a `Base64` encoded string.
+* `:boolean` - Used to store true or false values.
+* `:constant_keyword` - Used for fields that will contain the same value across all documents.
+* `:datetime` - Used to store date and time.
+* `:flattened` - Used for indexing object hierarchies as a flat list.
+* `:geo_point` - Used to store geographical locations as latitude/longitude points.
+* `:geo_shape` - Used to store complex shapes like polygons.
+* `:histogram` - Used to support aggregations on numerical data.
+* `:hash` - Used for structured JSON objects. Each field can be of any data type, including another object.
+* `:ip` - Used to store IP addresses.
+* `:join` - Used to create parent/child relation within documents.
+* `:keyword` - Used for exact match searches, sorting, and aggregations. This field is not analyzed.
+* `:match_only_text` - Used for full-text search.
+* `:nested` - Used to index arrays of objects as separate documents.
+* `:percolator` - Used to store queries for matching documents.
+* `:point` - Used to store points in space.
+* `:rank_feature` - Used to record a numeric feature to boost hits at query time.
+* `:rank_features` - Used to record many numeric features to boost hits at query time.
+* `:text` - Used to store text. This field is analyzed, which means that its value is broken down into separate searchable terms.
+* `:token_count` - Used to count the number of tokens in a string.
+* `:dense_vector` - Used to store dense vectors of float values.
+* `:search_as_you_type` - Used for full-text search, especially for auto-complete functionality.
+* `:sparse_vector` - Used to store sparse vectors of float values.
+* `:string` - Used to store text. This field is analyzed, which means that its value is broken down into separate searchable terms. _an alias of `:text`_
+* `:version` - Used to store version numbers.
+* `:wildcard` - Used for fields that will contain arbitrary strings.
+
+#### Numeric Types
+* `:long` - Used to store long integer numbers.
+* `:integer` - Used to store integer numbers.
+* `:short` - Used to store short integer numbers.
+* `:byte` - Used to store byte data.
+* `:double` - Used to store double-precision floating point numbers.
+* `:float` - Used to store floating point numbers.
+* `:half_float` - Used to store half-precision floating point numbers.
+* `:scaled_float` - Used to store floating point numbers that are scaled by a specific factor.
+* `:unsigned_long` - Used to store long integer numbers that are always positive.
+
+#### Range Types
+* `:integer_range` - Used to store ranges of integer numbers.
+* `:float_range` - Used to store ranges of floating point numbers.
+* `:long_range` - Used to store ranges of long integer numbers.
+* `:double_range` - Used to store ranges of double-precision floating point numbers.
+* `:date_range` - Used to store ranges of dates.
+* `:ip_range` - Used to store ranges of IP addresses.
+
+In Elasticsearch, `:string` and `:keyword` are two seemlingly similar data types that are used for different purposes.
+
+A `:string` field is analyzed, which means that its value is broken down into separate searchable terms. For example, the string "quick brown fox" might be broken down into the terms "quick", "brown", and "fox". This makes `:string` fields suitable for full-text search where you want to find partial matches or search for individual words within a field.
+
+On the other hand, a `:keyword` field is not analyzed and is used for exact match searches, sorting, and aggregations. The entire value of the field is used as a single term. For example, the string "quick brown fox" is a single term. This makes `:keyword` fields suitable for filtering, sorting, and aggregations where you need the exact value of the field.
+
+In Stretchy, when you define a field as `:keyword`, it's like defining a `:string` or `:text` field but with the `.keyword` notation automatically added to the field in queries, aggregations, and filters. This tells Elasticsearch to treat the field as a `:keyword` field and use the entire field value as a single term.
+
+Avoid using `:keyword` fields for full-text search. Use the `:text` or `:string` type instead.
+
+
+```ruby
+class Profile < StretchyModel
+
+    attribute :first_name,  :keyword
+    attribute :last_name,   :keyword
+    attribute :geo,         :hash
+    attribute :bio,         :text
+    attribute :age,         :integer
+    attribute :score,       :float
+    attribute :joined,      :datetime
+    attribute :fav_flowers, :array
+    attribute :visible,     :boolean, default: true
+
+end
+
+```
+
+>[!TIP]
+>
+> `created_at`, `updated_at` and `id` are automatically included
+
+
+All of the attribute types can receive additional parameters that are used to configure the mapping for the field. These parameters can be used to customize how the data is indexed and searched. For example, you can specify whether a field should be stored, whether it should be indexed, the data type of the field, and more.
+
+Here's an example of how to use additional parameters with the :keyword attribute type:
+
+```ruby
+class Product < StretchyModel
+    attribute :tag, :keyword, ignore_above: 256, time_series_dimension: true
+end
+```
+In this example, ignore_above: 256 means that strings longer than 256 characters will not be indexed, and time_series_dimension: true indicates that the field is a time series dimension.
+
+Refer to the documentation for each attribute type for a full list of available parameters and their meanings.
+
+When working with `:hash` data types, it's often useful to specify the fields within that object for fine-tuned control over mappings. 
+
+The `:hash` data type is used for structured JSON objects, where each field can be of any data type, including another object. This provides a flexible way to store complex data structures in a single field.
+
+However, this flexibility can sometimes lead to less than optimal search performance or unexpected search results. For example, if a field within the hash object contains text, it might be analyzed and tokenized in a way that's not suitable for your use case.
+
+To overcome this, you can specify the `:properties` within the hash object and their data types. This allows you to control how each field is indexed and searched. For example, you can specify that a field should be of type `:keyword` to ensure that it's not analyzed and can be used for exact match searches.
+
+Here's an example of how to specify fields within a hash object:
+
+```ruby
+attribute :metadata, :hash, properties: {
+  title: { type: :text },
+  tags: { type: :keyword }
+  checkins: { type: :array }
+}
+```
+In this example, the metadata attribute is a hash object with fields: title, tags and checkins. The title field is of type `:text`, which means it will be analyzed and can be used for full-text search. The tags field is of type `:keyword`, which means it will not be analyzed and can be used for exact match searches. The checkins field is of type `:array` which means it can contain zero or more values.
+
+> [!NOTE]
+>
+>When adding a field dynamically, the first value in the array determines the field type. All subsequent values must be of the same data type or it must at least be possible to coerce subsequent values to the same data type.
+>
+>Arrays with a mixture of data types are not supported: `[ 10, "some string" ]`
+
+By specifying the fields within the hash object, you have fine-tuned control over the mappings and can optimize the search performance and accuracy for your specific use case. 
+
+### Reading and Writing Data
+
+
+#### Create
+
+The `new` method will return a new object while `create` with return the object and save it to the index. 
+
+For example, given a model `Profile` with attributes `first_name`, `last_name`, and `age`, the `create` method call will create and save a new record to the index:
+
+```ruby
+profile = Profile.new(first_name: "Candy", last_name: "Mu", age: 33)
+```
+
+Using the `new` method, an object can be instatiated without being saved:
+
+```ruby
+profile = Profile.new
+profile.first_name = "Candy"
+profile.last_name = "Mu"
+profile.age = 33
+```
+
+Calling `profile.save` will index the record. 
+
+#### Read
+
+Following the ActiveRecord API, Stretchy models have the familiar methods defined: 
+
+```ruby
+# return a collection with all profiles 
+profiles = Profile.all
+```
+
+```ruby
+# return the first profile
+profile = Profile.first
+```
+
+```ruby
+# find all profiles with age of 24 named Lori and sort by :created_at in descending order
+profile = Profile.where(first_name: "Lori", age: 24).sort(created_at: :desc)
+```
+
+The full [Query Interface](guides/querying) guide goes into further depth on the API available for interacting with Stretchy models. 
+
+
+#### Update
+
+```ruby
+profile = Profile.where(first_name: "Candy").first
+profile.update(first_name: "Lilly")
+```
+
+
+#### Delete
+
+```ruby
+profile = Profile.where(first_name: "Lori")
+profile.destroy
+```
+
+### Bulk Operations
+
+Bulk operations in Elasticsearch are a way to perform multiple operations in a single API call. This can greatly increase the speed of indexing and updating documents in Elasticsearch.
+
+The bulk API makes it possible to perform many `index`, `update`, `create`, or `delete` operations in a single API call. This can greatly improve the indexing speed.
+
+In the context of Stretchy, the `Model.bulk` method can be used to perform bulk operations. You can pass an array of records to this method, and each record will be processed according to the operation specified in its `to_bulk` method.
+
+The `to_bulk` method is used to generate the structure for the bulk operation. By default, it performs an index operation, but you can also specify `:delete` or `:update` operations.
+
+For large datasets, you can use the `Model.bulk_in_batches` method to perform bulk operations in batches. This method divides the records into batches of a specified size and processes each batch separately. This can be more memory-efficient than processing all records at once, especially for large datasets.
+
+Here's an example of how you can use these methods:
+
+```ruby
+Model.bulk(records_as_bulk_operations)
+```
+
+#### Bulk helper
+Generates structure for the bulk operation
+```ruby
+record.to_bulk # default to_bulk(:index)
+record.to_bulk(:delete)
+record.to_bulk(:update)
+```
+
+#### In batches
+Run bulk operations in batches specified by `size`
+```ruby
+Model.bulk_in_batches(records, size: 100) do |batch|
+    batch.map! { |record| Model.new(record).to_bulk }
+end
+```
+
+
+### Validations
+
+```ruby
+class Profile < StretchyModel
+    attribute :first_name, :string
+    validates :first_name, presence: true
+end
+```
+
+```irb
+profile = Profile.new
+profile.save
+#=> First name can't be blank 
+```
+
+### Callbacks
+
+* `:before_save`
+* `:after_save`
+* `:before_create`
+* `:after_create`
+* `:before_update`
+* `:after_update`
+* `:before_destroy`
+* `:after_destroy`
+
+### Associations
+Associations can be made between models. While Elasticsearch is not a relational database, it is sometimes useful to have a link between records. 
+
+```ruby
+class Animal < StretchyModel
+    attribute :name, :keyword
+    attribute :zoo_id, :keyword
+
+    belongs_to :zoo
+end
+```
+
+```ruby
+class Zoo < StretchyModel
+    has_many :animals
+end
+```
+
+```ruby (console)
+zoo.animals
+=> [#<Animal id: 1, name: "Panda", zoo_id: 1, created_at: 2024-03-15T01:03:38.395Z, updated_at: 2024-03-15T01:03:38.395Z>, 
+#<Animal id: 2, name: "Lemur", zoo_id: 1, created_at: 2024-03-15T01:03:38.395Z, updated_at: 2024-03-15T01:03:38.395Z>]
+
+```
+
+Associations largely work the same as their Rails counterparts. The following association types are supported: 
+
+* `belongs_to`
+* `has_many`
+* `has_one`
+
+
+>[!WARNING|label:Associations in Elasticsearch]
+>
+> Because Elasticsearch is not a relational database, there are no join statements. This means associations will generate additional queries to fetch data.
+
+
+## Mappings
+
+In Elasticsearch, mappings define how documents and their fields are stored and indexed. When using the attribute method you can specify additional mapping options as parameters.
+
+```ruby
+class Report < StretchyModel
+    attribute :title, :text
+    attribute :created_by, :keyword
+    attribute :body, :text, term_vector: :with_positions_offsets
+end
+```
+The attribute method is used to define the fields of the Report model. The first argument is the field name, the second argument is the field type, and any subsequent arguments are additional mapping options for the field.
+
+In the case of the body field, `term_vector: :with_positions_offsets` is an additional mapping option. This option configures Elasticsearch to store term vectors for the body field, including position and character offset information. Term vectors are used to speed up full-text search operations.
+
+So, when you're defining mappings for your Elasticsearch models, you can use the attribute method to specify not only the field names and types, but also any additional mapping options that you need.
+
+If you're curious and want to dive deeper, check out the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). It's a treasure trove of information on all the supported options for each field type.
+
+```ruby
+Report.mappings
+=> {
+    "properties" => {
+      "id"         => {
+        "type" => "keyword"
+      },
+      "created_at" => {
+        "type" => "date"
+      },
+      "updated_at" => {
+        "type" => "date"
+      },
+      "title"      => {
+        "type" => "text"
+      },
+      "created_by" => {
+        "type" => "keyword"
+      },
+      "body"       => {
+        "type"        => "text",
+        "term_vector" => "with_positions_offsets",
+        "fields"      => {
+          "keyword" => {
+            "type"         => "keyword",
+            "ignore_above" => 256
+          }
+        }
+      }
+    },
+    :dynamic     => true
+  } 
+```

data/docs/guides/pipelines.md

@@ -0,0 +1,151 @@
+# Pipelines
+
+Pipelines follow a specific convention for storing pipeline definitions. This helps us keep our code organized and easy to navigate.
+
+Generally, pipelines can be stored in `app/pipelines`. However, if you have a mix of ingest and search pipelines it's good practice to break them out into their own namespace.
+
+For ingest pipelines, we store each pipeline in its own Ruby file inside the `app/pipelines/ingest` directory. The file name matches the pipeline name. For example, the definition for an ingest pipeline named `example_pipeline` would be stored in `app/pipelines/ingest/example_pipeline.rb`.
+
+Similarly, for search pipelines, we store each pipeline in its own Ruby file inside the `app/pipelines/search` directory. Again, the file name matches the pipeline name. So, a search pipeline named `example_pipeline` would be stored in `app/pipelines/search/example_pipeline.rb`.
+
+This convention makes it easy to find the definition for a specific pipeline, whether it's an ingest pipeline or a search pipeline.
+
+- *app/pipelines/ingest/example_ingest_pipeline.rb*
+- *app/pipelines/search/example_search_pipeline.rb*
+
+--- 
+
+## Defining a Pipeline
+
+A pipeline in `stretchy-model` is defined as a Ruby class that inherits from `Stretchy::Pipeline`. It has a specific structure and includes several key components:
+
+- `pipeline_name`: This is the ID of your pipeline. By default, it's inferred from the class name. 
+- `description`: This is a brief description of what your pipeline does. It's a good practice to provide a meaningful description so that others can understand the purpose of your pipeline at a glance.
+- `processor`: These are the processors that your pipeline will run. A pipeline can have one or more processors. Each processor is defined with a type (like `sparse_encoding`) and a set of options. The processors are run in the order they are defined. Refer to [Ingest Processors](/guides/pipelines?id=ingest-processors) for available options.
+
+
+Here's an example of a pipeline with these components:
+
+```ruby
+class NLPSparsePipeline < Stretchy::Pipeline
+
+	#inferred from class by default
+	pipeline_name 'nlp-sparse-pipeline' 
+	
+	description "Sparse encoding pipeline"
+	
+	processor :sparse_encoding, 
+				model_id: 'q32Pw02BJ3squ3VZa',
+				field_map: {
+					body: :embedding
+				}
+end
+```
+
+[^1]: https://opensearch.org/docs/latest/ingest-pipelines/
+
+[^2]: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
+
+
+## Managing Pipelines
+
+### create!
+This method creates a new pipeline in Elasticsearch. It uses the pipeline definition provided in the class where it's called. If a pipeline with the same name already exists, it will be overwritten.
+
+Example:
+
+```ruby
+MyPipeline.create!
+```
+
+### delete!
+This method deletes the pipeline from Elasticsearch. Be careful when using this method, as it will permanently remove the pipeline.
+
+Example:
+
+```ruby
+MyPipeline.delete!
+```
+
+### exists?
+This method checks if the pipeline exists in Elasticsearch. It returns true if the pipeline exists, and false otherwise.
+
+Example:
+
+```ruby
+MyPipeline.exists?
+```
+
+### find
+This method retrieves a pipeline from Elasticsearch using its ID. If the pipeline exists, it returns the pipeline definition. If the pipeline doesn't exist, it returns nil.
+
+
+Example:
+```ruby
+# Find another pipeline id
+MyPipeline.find('another-pipeline-id')
+
+# Find this pipeline
+MyPipeline.find
+```
+### all
+This method returns all pipelines that currently exist in Elasticsearch. It's useful when you want to see all your pipelines at once.
+
+Example:
+```ruby
+MyPipeline.all
+```
+
+### simulate
+
+This method is used to simulate the execution of the pipeline on a set of documents. It takes two parameters: `docs`, which is an array of documents to be processed, and `verbose`, which is a boolean that controls whether detailed information about each step is included in the simulation output.
+
+Example:
+
+```ruby
+MyPipeline.simulate([{ '_source' => { 'message' => 'hello world' } }])
+```
+
+---
+
+## Ingest Processors
+- `:append` - Adds one or more values to a field in a document.
+- `:bytes` - Converts a human-readable byte value to its value in bytes.
+- `:convert` - Changes the data type of a field in a document.
+- `:copy` - Copies an entire object in an existing field to another field.
+- `:csv` - Extracts CSVs and stores them as individual fields in a document.
+- `:date` - Parses dates from fields and then uses the date or timestamp as the timestamp for a document.
+- `:date_index_name` - Indexes documents into time-based indexes based on a date or timestamp field in a document.
+- `:dissect` - Extracts structured fields from a text field using a defined pattern.
+- `:dot_expander` - Expands a field with dots into an object field.
+- `:drop` - Drops a document without indexing it or raising any errors.
+- `:fail` - Raises an exception and stops the execution of a pipeline.
+- `:foreach` - Allows for another processor to be applied to each element of an array or an object field in a document.
+- `:geoip` - Adds information about the geographical location of an IP address.
+- `:geojson-feature` - Indexes GeoJSON data into a geospatial field.
+- `:grok` - Parses and structures unstructured data using pattern matching.
+- `:gsub` - Replaces or deletes substrings within a string field of a document.
+- `:html_strip` - Removes HTML tags from a text field and returns the plain text content.
+- `:ip2geo` - Adds information about the geographical location of an IPv4 or IPv6 address.
+- `:join` - Concatenates each element of an array into a single string using a separator character between each element.
+- `:json` - Converts a JSON string into a structured JSON object.
+- `:kv` - Automatically parses key-value pairs in a field.
+- `:lowercase` - Converts text in a specific field to lowercase letters.
+- `:pipeline` - Runs an inner pipeline.
+- `:remove` - Removes fields from a document.
+- `:script` - Runs an inline or stored script on incoming documents.
+- `:set` - Sets the value of a field to a specified value.
+- `:sort` - Sorts the elements of an array in ascending or descending order.
+- `:sparse_encoding` - Generates a sparse vector/token and weights from text fields for neural sparse search using sparse retrieval.
+- `:split` - Splits a field into an array using a separator character.
+- `:text_embedding` - Generates vector embeddings from text fields for semantic search.
+- `:text_image_embedding` - Generates combined vector embeddings from text and image fields for multimodal neural search.
+- `:trim` - Removes leading and trailing white space from a string field.
+- `:uppercase` - Converts text in a specific field to uppercase letters.
+- `:urldecode` - Decodes a string from URL-encoded format.
+- `:user_agent` - Extracts details from the user agent sent by a browser to its web requests.
+
+Please note that this is not an exhaustive list. There are many more ingest processors available in Elasticsearch and OpenSearch. For a complete list and their available options, refer to the official documentation: 
+
+- [Elasticsearch Ingest Processors](https://www.elastic.co/guide/en/elasticsearch/reference/current/processors.html)
+- [OpenSearch Ingest Processors](https://opensearch.org/docs/latest/ingest-pipelines/processors/index-processors/#supported-processors)