boxcars 0.2.13 → 0.2.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 74f14f8575e4670d2be6196c5196d41dd9728b5a44a0d4e199dfb705dfc77ed5
-  data.tar.gz: 06f2e8178f9696831870b5d8d5ea40bda8ba74a2fcc27283849f49124c51a06b
+  metadata.gz: 1039f86c58712c10143cc26438da14571013081bcff08aed3d9fcac4b1e84060
+  data.tar.gz: fca9f08855cae8e4e8a4171c043e92884e245582b049b4cc75bfa4d2cd98e51a
 SHA512:
-  metadata.gz: 9ff6e759f3d942f859de85763ffe9bc0ccf5636914d894205f78cffc99abb1b27dea47463f0fc968eba6746c055f796c95884d5e421486e06374fb0519eb8c63
-  data.tar.gz: 03bf42b1fbd6dac1eff4734bd2443a4a2a9f7cb931c99a2fe3f9453f5a23f0853b4eb26c817e1757a16629617eff4704f276b99fb165689ef6016ace86c2fb56
+  metadata.gz: 5f24a578f4004d99a0d71c05f4a0ed520dda2e562d1327ec68bdf8682a927b741e870ed348a1a273319003f515521ef8a12c8778b70994764a945e31f906f807
+  data.tar.gz: be067a2ba1ba2e032a58d32f46071f5510a5ce0f15fbc89a06fe84f4fd854ec825a0ea0c786a90ba87c20bb1893ff59512170ac2e338080e25f85b2474b6fb64

data/CHANGELOG.md

@@ -1,5 +1,35 @@
 # Changelog
 
+## [v0.2.13](https://github.com/BoxcarsAI/boxcars/tree/v0.2.13) (2023-05-24)
+
+[Full Changelog](https://github.com/BoxcarsAI/boxcars/compare/v0.2.12...v0.2.13)
+
+**Closed issues:**
+
+- Typo "Boscar.error" should be "Boxcars.error" [\#82](https://github.com/BoxcarsAI/boxcars/issues/82)
+
+**Merged pull requests:**
+
+- Add vector answer boxcar [\#79](https://github.com/BoxcarsAI/boxcars/pull/79) ([francis](https://github.com/francis))
+
+## [v0.2.12](https://github.com/BoxcarsAI/boxcars/tree/v0.2.12) (2023-05-22)
+
+[Full Changelog](https://github.com/BoxcarsAI/boxcars/compare/v0.2.11...v0.2.12)
+
+**Closed issues:**
+
+- GPT-4 support? [\#71](https://github.com/BoxcarsAI/boxcars/issues/71)
+- add PgVector Vector Store [\#68](https://github.com/BoxcarsAI/boxcars/issues/68)
+
+**Merged pull requests:**
+
+- issue\_82 typo "Boscar" instead of "Boxcars" [\#83](https://github.com/BoxcarsAI/boxcars/pull/83) ([MadBomber](https://github.com/MadBomber))
+- Update boxcars.rb config example [\#81](https://github.com/BoxcarsAI/boxcars/pull/81) ([nhorton](https://github.com/nhorton))
+- Feature- added pgvector vector store [\#80](https://github.com/BoxcarsAI/boxcars/pull/80) ([jaigouk](https://github.com/jaigouk))
+- drop support for pre ruby 3 version [\#75](https://github.com/BoxcarsAI/boxcars/pull/75) ([francis](https://github.com/francis))
+- Chore - refine VectorSearch [\#74](https://github.com/BoxcarsAI/boxcars/pull/74) ([jaigouk](https://github.com/jaigouk))
+- raise error if OpenAI API returns error or nil. closes \#71 [\#72](https://github.com/BoxcarsAI/boxcars/pull/72) ([francis](https://github.com/francis))
+
 ## [v0.2.11](https://github.com/BoxcarsAI/boxcars/tree/v0.2.11) (2023-05-05)
 
 [Full Changelog](https://github.com/BoxcarsAI/boxcars/compare/v0.2.10...v0.2.11)

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    boxcars (0.2.13)
+    boxcars (0.2.14)
       google_search_results (~> 2.2)
       gpt4all (~> 0.0.4)
       hnswlib (~> 0.8)
@@ -88,11 +88,9 @@ GEM
     i18n (1.13.0)
       concurrent-ruby (~> 1.0)
     io-console (0.6.0)
-    io-console (0.6.0-java)
     irb (1.6.4)
       reline (>= 0.3.0)
     json (2.6.3)
-    json (2.6.3-java)
     mime-types (3.4.1)
       mime-types-data (~> 3.2015)
     mime-types-data (3.2023.0218.1)
@@ -102,7 +100,6 @@ GEM
     multipart-post (2.3.0)
     netrc (0.11.0)
     nio4r (2.5.9)
-    nio4r (2.5.9-java)
     octokit (4.25.1)
       faraday (>= 1, < 3)
       sawyer (~> 0.9)
@@ -176,9 +173,6 @@ GEM
       faraday (>= 0.17.3, < 3)
     sqlite3 (1.6.3)
       mini_portile2 (~> 2.8.0)
-    sqlite3 (1.6.3-arm64-darwin)
-    sqlite3 (1.6.3-x86_64-darwin)
-    sqlite3 (1.6.3-x86_64-linux)
     strings-ansi (0.2.0)
     timers (4.3.5)
     traces (0.9.1)
@@ -193,7 +187,6 @@ GEM
       concurrent-ruby (~> 1.0)
     unf (0.1.4)
       unf_ext
-    unf (0.1.4-java)
     unf_ext (0.0.8.2)
     unicode-display_width (2.4.2)
     vcr (6.1.0)

data/README.md

@@ -3,14 +3,14 @@
 <h4 align="center">
   <a href="https://www.boxcars.ai">Website</a> |
   <a href="https://www.boxcars.ai/blog">Blog</a> |
-  <a href="https://github.com/BoxcarsAI/boxcars/wiki">Documentation</a> 
+  <a href="https://github.com/BoxcarsAI/boxcars/wiki">Documentation</a>
 </h4>
 
 <p align="center">
   <a href="https://github.com/BoxcarsAI/boxcars/blob/main/LICENSE.txt"><img src="https://img.shields.io/badge/license-MIT-informational" alt="License"></a>
 </p>
 
-Boxcars is a gem that enables you to create new systems with AI composability, using various concepts such as OpenAI, Search, SQL, Rails Active Record and more. This can even be extended with your concepts as well (including your concepts).
+Boxcars is a gem that enables you to create new systems with AI composability, using various concepts such as OpenAI, Search, SQL, Rails Active Record, Vector Search and more. This can even be extended with your concepts as well (including your concepts).
 
 This gem was inspired by the popular Python library Langchain. However, we wanted to give it a Ruby spin and make it more user-friendly for beginners to get started.
 
@@ -57,6 +57,9 @@ require "boxcars"
 Note: if you want to try out the examples below, run this command and then paste in the code segments of interest:
 ```bash
 irb -r dotenv/load -r boxcars
+
+# or if you prefer local repository
+irb -r dotenv/load -r ./lib/boxcars
 ```
 
 ### Direct Boxcar Use
@@ -107,7 +110,7 @@ Produces:
 ```text
 > Entering Zero Shot#run
 What is pi times the square root of the average temperature in Austin TX in January?
-Thought: We need to find the average temperature in Austin TX in January and then multiply it by pi and the square root of the average temperature. We can use a search engine to find the average temperature in Austin TX in January and a calculator to perform the multiplication. 
+Thought: We need to find the average temperature in Austin TX in January and then multiply it by pi and the square root of the average temperature. We can use a search engine to find the average temperature in Austin TX in January and a calculator to perform the multiplication.
 Question: Average temperature in Austin TX in January
 Answer: January Weather in Austin Texas, United States. Daily high temperatures increase by 2°F, from 62°F to 64°F, rarely falling below 45°F or exceeding 76° ...
 Observation: January Weather in Austin Texas, United States. Daily high temperatures increase by 2°F, from 62°F to 64°F, rarely falling below 45°F or exceeding 76° ...
@@ -135,7 +138,7 @@ See [this](https://github.com/BoxcarsAI/boxcars/blob/main/notebooks/boxcars_exam
 
 For the Swagger boxcar, see [this](https://github.com/BoxcarsAI/boxcars/blob/main/notebooks/swagger_examples.ipynb) Jupyter Notebook.
 
-For simple vector storage and search, see [this](https://github.com/BoxcarsAI/boxcars/blob/main/notebooks/vector_store_examples.ipynb) Jupyter Notebook.
+For simple vector storage and search, see [this](https://github.com/BoxcarsAI/boxcars/blob/main/notebooks/vector_search_examples.ipynb) Jupyter Notebook.
 
 Note, some folks that we talked to didn't know that you could run Ruby Jupyter notebooks. [You can](https://github.com/SciRuby/iruby).
 

data/boxcars.gemspec

@@ -34,8 +34,8 @@ Gem::Specification.new do |spec|
   spec.add_dependency "google_search_results", "~> 2.2"
   spec.add_dependency "gpt4all", "~> 0.0.4"
   spec.add_dependency "hnswlib", "~> 0.8"
-  spec.add_dependency "ruby-openai", "~> 4.1"
   spec.add_dependency "pgvector", "~> 0.2"
+  spec.add_dependency "ruby-openai", "~> 4.1"
 
   # For more information and examples about making a new gem, checkout our
   # guide at: https://bundler.io/guides/creating_gem.html

data/lib/boxcars/boxcar/vector_answer.rb

@@ -47,7 +47,7 @@ module Boxcars
     def get_search_content(question, count: 1)
       search = Boxcars::VectorSearch.new(embeddings: embeddings, vector_documents: vector_documents)
       results = search.call query: question, count: count
-      @search_content = get_search_content(results)
+      @search_content = get_results_content(results)
     end
 
     # our template

data/lib/boxcars/vector_search.rb

@@ -4,6 +4,20 @@
 module Boxcars
   # For Boxcars that use an engine to do their work.
   class VectorSearch
+    # initialize the vector search with the following parameters:
+    # @param params [Hash] A Hash containing the initial configuration.
+    # @option params [Hash] :vector_documents The vector documents to search.
+    # example:
+    # {
+    #   type: :in_memory,
+    #   vector_store: [
+    #     Boxcars::VectorStore::Document.new(
+    #       content: "hello",
+    #       embedding: [0.1, 0.2, 0.3],
+    #       metadata: { a: 1 }
+    #     )
+    #   ]
+    # }
     def initialize(params)
       @vector_documents = params[:vector_documents]
       @embedding_tool = params[:embedding_tool] || :openai
@@ -11,6 +25,20 @@ module Boxcars
       @openai_connection = params[:openai_connection] || default_connection(openai_access_token: params[:openai_access_token])
     end
 
+    # @param query [String] The query to search for.
+    # @param count [Integer] The number of results to return.
+    # @return [Array] array of hashes with :document and :distance keys
+    # @example
+    #   [
+    #     {
+    #       document: Boxcars::VectorStore::Document.new(
+    #         content: "hello",
+    #         embedding: [0.1, 0.2, 0.3],
+    #         metadata: { a: 1 }
+    #       ),
+    #       distance: 0.1
+    #     }
+    #   ]
     def call(query:, count: 1)
       validate_query(query)
       query_vector = convert_query_to_vector(query)

data/lib/boxcars/vector_store/hnswlib/build_from_files.rb

@@ -16,13 +16,10 @@ module Boxcars
 
         def initialize(params)
           @split_chunk_size = params[:split_chunk_size] || 2000
-          @training_data_path = File.absolute_path(params[:training_data_path])
-          @index_file_path = File.absolute_path(params[:index_file_path])
+          @base_dir_path, @index_file_path, @json_doc_file_path =
+            validate_params(params[:training_data_path], params[:index_file_path], split_chunk_size)
 
-          validate_params(@training_data_path, @index_file_path, split_chunk_size)
-
-          @json_doc_file_path = absolute_json_doc_file_path(@index_file_path, params[:json_doc_file_path])
-          @force_rebuild = params.key?(:force_rebuild) ? params[:force_rebuild] : true
+          @force_rebuild = params[:force_rebuild] || false
           @hnsw_vectors = []
         end
 
@@ -50,24 +47,29 @@ module Boxcars
 
         private
 
-        attr_reader :training_data_path, :index_file_path, :split_chunk_size, :json_doc_file_path, :force_rebuild, :hnsw_vectors
+        attr_reader :training_data_path, :index_file_path, :base_dir_path,
+                    :split_chunk_size, :json_doc_file_path, :force_rebuild, :hnsw_vectors
 
         def validate_params(training_data_path, index_file_path, split_chunk_size)
-          training_data_dir = File.dirname(training_data_path.gsub(/\*{1,2}/, ''))
+          validate_string(training_data_path, 'training_data_path')
+          validate_string(index_file_path, 'index_file_path')
+
+          absolute_data_path = File.absolute_path(training_data_path)
+          base_data_dir_path = File.dirname(absolute_data_path.gsub(/\*{1,2}/, ''))
+          @training_data_path = training_data_path
 
-          raise_argument_error('training_data_path parent directory must exist') unless File.directory?(training_data_dir)
-          raise_argument_error('No files found at the training_data_path pattern') if Dir.glob(training_data_path).empty?
+          raise_argument_error('training_data_path parent directory must exist') unless File.directory?(base_data_dir_path)
+          raise_argument_error('No files found at the training_data_path pattern') if Dir.glob(absolute_data_path).empty?
 
-          index_dir = File.dirname(index_file_path)
+          absolute_index_path = File.absolute_path(index_file_path)
+          index_parent_dir = File.dirname(absolute_index_path)
 
-          raise_argument_error('index_file_path parent directory must exist') unless File.directory?(index_dir)
+          raise_argument_error('index_file_path parent directory must exist') unless File.directory?(index_parent_dir)
           raise_argument_error('split_chunk_size must be an integer') unless split_chunk_size.is_a?(Integer)
-        end
 
-        def absolute_json_doc_file_path(index_file_path, json_doc_file_path)
-          return index_file_path.gsub(/\.bin$/, '.json') unless json_doc_file_path
+          json_doc_file_path = index_file_path.gsub(/\.bin$/, '.json')
 
-          File.absolute_path(json_doc_file_path)
+          [index_parent_dir, index_file_path, json_doc_file_path]
         end
 
         def add_vectors(vectors, texts)
@@ -80,6 +82,7 @@ module Boxcars
                 dim: vector[:dim],
                 metric: 'l2',
                 max_item: 10000,
+                base_dir_path: base_dir_path,
                 index_file_path: index_file_path,
                 json_doc_file_path: json_doc_file_path
               }
@@ -94,6 +97,7 @@ module Boxcars
 
         def load_existing_vector_store
           Boxcars::VectorStore::Hnswlib::LoadFromDisk.call(
+            base_dir_path: base_dir_path,
             index_file_path: index_file_path,
             json_doc_file_path: json_doc_file_path
           )

data/lib/boxcars/vector_store/hnswlib/load_from_disk.rb

@@ -10,11 +10,13 @@ module Boxcars
       class LoadFromDisk
         include VectorStore
 
+        # params:
+        # base_dir_path: string (absolute path to the directory containing the index_file_path and json_doc_file_path),
+        # index_file_path: string (relative path to the index file from the base_dir_path),
+        # json_doc_file_path: string (relative path to the json file from the base_dir_path)
         def initialize(params)
-          validate_params(params[:index_file_path], params[:json_doc_file_path])
-
-          @index_file_path = File.absolute_path(params[:index_file_path])
-          @json_doc_file_path = File.absolute_path(params[:json_doc_file_path])
+          @base_dir_path, @index_file_path, @json_doc_file_path =
+            validate_params(params)
         end
 
         def call
@@ -29,14 +31,34 @@ module Boxcars
 
         private
 
-        attr_reader :index_file_path, :json_doc_file_path
+        attr_reader :base_dir_path, :index_file_path, :json_doc_file_path
+
+        def validate_params(params)
+          base_dir_path = params[:base_dir_path]
+          index_file_path = remove_relative_path(params[:index_file_path])
+          json_doc_file_path = remove_relative_path(params[:json_doc_file_path])
+          # we omit base_dir validation in case of loading the data from other environments
+          validate_string(index_file_path, "index_file_path")
+          validate_string(json_doc_file_path, "json_doc_file_path")
+
+          absolute_index_path = validate_file_existence(base_dir_path, index_file_path, "index_file_path")
+          abosolute_json_path = validate_file_existence(base_dir_path, json_doc_file_path, "json_doc_file_path")
+
+          [base_dir_path, absolute_index_path, abosolute_json_path]
+        end
+
+        def remove_relative_path(path)
+          path.start_with?('./') ? path[2..] : path
+        end
+
+        def validate_file_existence(base_dir, file_path, name)
+          file =
+            base_dir.to_s.empty? ? file_path : File.join(base_dir, file_path)
+          complete_path = File.absolute_path(file)
 
-        def validate_params(index_file_path, json_doc_file_path)
-          raise_argument_error("index_file_path must be a string") unless index_file_path.is_a?(String)
-          raise_argument_error("json_doc_file_path must be a string") unless json_doc_file_path.is_a?(String)
+          raise raise_argument_error("#{name} does not exist at #{complete_path}") unless File.exist?(complete_path)
 
-          raise_argument_error("index_file_path must exist") unless File.exist?(index_file_path)
-          raise_argument_error("json_doc_file_path must exist") unless File.exist?(json_doc_file_path)
+          complete_path
         end
 
         def load_as_hnsw_vectors(vectors)
@@ -47,7 +69,11 @@ module Boxcars
               embedding: vector[:embedding],
               metadata: vector[:metadata]
             )
-            hnsw_vectors[vectors.first[:doc_id].to_i] = hnsw_vector
+            if vector[:metadata][:doc_id]
+              hnsw_vectors[vector[:metadata][:doc_id]] = hnsw_vector
+            else
+              hnsw_vectors << hnsw_vector
+            end
           end
           hnsw_vectors
         end

data/lib/boxcars/vector_store/hnswlib/search.rb

@@ -9,19 +9,35 @@ module Boxcars
       class Search
         include VectorStore
 
+        # initialize the vector store search with the following parameters:
+        # @param params [Hash] A Hash containing the initial configuration.
+        # example:
+        # {
+        #   type: :hnswlib,
+        #   vector_store: [
+        #     Boxcars::VectorStore::Document.new(
+        #       content: "hello",
+        #       embedding: [0.1, 0.2, 0.3],
+        #       metadata: { a: 1 }
+        #     )
+        #   ]
+        # }
         def initialize(params)
-          validate_params(params[:vector_documents])
-          @vector_documents = params[:vector_documents]
-          @search_index = load_index(params[:vector_documents])
+          @vector_store = validate_params(params[:vector_documents])
+          @metadata, @index_file = validate_files(vector_store)
+          @search_index = load_index(metadata, index_file)
         end
 
+        # @param query_vector [Array] The query vector to search for.
+        # @param count [Integer] The number of results to return.
+        # @return [Array] array of hashes with :document and :distance keys
         def call(query_vector:, count: 1)
           search(query_vector, count)
         end
 
         private
 
-        attr_reader :vector_documents, :vector_store, :json_doc, :search_index, :metadata
+        attr_reader :vector_store, :index_file, :search_index, :metadata
 
         def validate_params(vector_documents)
           raise_argument_error('vector_documents is nil') unless vector_documents
@@ -34,27 +50,47 @@ module Boxcars
             raise_arugment_error('vector_store must be an array of Document objects')
           end
 
-          true
+          vector_documents[:vector_store]
         end
 
-        def load_index(vector_documents)
-          @metadata = vector_documents[:vector_store].first.metadata
-          @json_doc = @metadata[:json_doc_file_path]
+        def validate_files(vector_store)
+          metadata = vector_store.first.metadata
+          raise_arugment_error('metadata must be a hash') unless metadata.is_a?(Hash)
+          raise_arugment_error('metadata is empty') if metadata.empty?
 
+          validate_string(metadata[:index_file_path], "index_file_path")
+          validate_string(metadata[:json_doc_file_path], "json_doc_file_path")
+
+          base_dir = metadata[:base_dir_path]
+          index_file_file_path = metadata[:index_file_path]
+          index_file =
+            if !index_file_file_path.to_s.empty? && File.exist?(index_file_file_path)
+              index_file_file_path
+            else
+              File.join(base_dir.to_s, index_file_file_path.to_s)
+            end
+
+          raise_argument_error('index_file does not exist') unless File.exist?(index_file)
+
+          [metadata, index_file]
+        end
+
+        def load_index(metadata, index_file)
           search_index = ::Hnswlib::HierarchicalNSW.new(
             space: metadata[:metric],
             dim: metadata[:dim]
           )
-          search_index.load_index(metadata[:index_file_path])
-          @search_index = search_index
-          @vector_store = vector_documents[:vector_store]
-
+          search_index.load_index(index_file)
           search_index
         end
 
         def search(query_vector, num_neighbors)
           raw_results = search_index.search_knn(query_vector, num_neighbors)
-          raw_results.map { |doc_id, distance| lookup_embedding(doc_id, distance) }.compact
+
+          raw_results.map { |doc_id, distance| lookup_embedding(doc_id, distance) }
+                     .compact
+                     .first(num_neighbors)
+                     .sort_by { |result| result[:distance] }
         rescue StandardError => e
           raise_argument_error("Error searching for #{query_vector}: #{e.message}")
         end

data/lib/boxcars/vector_store/in_memory/build_from_array.rb

@@ -0,0 +1,75 @@
+# frozen_string_literal: true
+
+module Boxcars
+  module VectorStore
+    module InMemory
+      class BuildFromArray
+        include VectorStore
+
+        # @param embedding_tool [Symbol] :openai or other embedding tools
+        # @param input_array [Array] array of hashes with :content and :metadata keys
+        # each hash item should have content and metadata
+        # [
+        #   { content: "hello", metadata: { a: 1 } },
+        #   { content: "hi", metadata: { a: 1 } },
+        #   { content: "bye", metadata: { a: 1 } },
+        #   { content: "what's this", metadata: { a: 1 } }
+        # ]
+        # @return [Hash] vector_store: array of hashes with :content, :metadata, and :embedding keys
+        def initialize(embedding_tool: :openai, input_array: nil)
+          validate_params(embedding_tool, input_array)
+          @embedding_tool = embedding_tool
+          @input_array = input_array
+          @memory_vectors = []
+        end
+
+        # @return [Hash] vector_store: array of Inventor::VectorStore::Document
+        def call
+          texts = input_array.map { |doc| doc[:content] }
+          vectors = generate_vectors(texts)
+          add_vectors(vectors, input_array)
+
+          {
+            type: :in_memory,
+            vector_store: memory_vectors
+          }
+        end
+
+        private
+
+        attr_reader :input_array, :memory_vectors
+
+        def validate_params(embedding_tool, input_array)
+          raise_argument_error('input_array is nil') unless input_array
+          raise_argument_error('input_array must be an array') unless input_array.is_a?(Array)
+          unless proper_document_array?(input_array)
+            raise_argument_error('items in input_array needs to have content and metadata')
+          end
+
+          return if %i[openai tensorflow].include?(embedding_tool)
+
+          raise_argument_error('embedding_tool is invalid')
+        end
+
+        def proper_document_array?(input_array)
+          return false unless
+            input_array.all? { |hash| hash.key?(:content) && hash.key?(:metadata) }
+
+          true
+        end
+
+        # returns array of documents with vectors
+        def add_vectors(vectors, input_array)
+          vectors.zip(input_array).each do |vector, doc|
+            memory_vector = Document.new(
+              content: doc[:content],
+              embedding: vector[:embedding],
+              metadata: doc[:metadata].merge(dim: vector[:dim])
+            )
+            @memory_vectors << memory_vector
+          end
+        end
+      end
+    end
+  end
+end

data/lib/boxcars/vector_store/in_memory/build_from_files.rb

@@ -6,6 +6,12 @@ module Boxcars
       class BuildFromFiles
         include VectorStore
 
+        # initialize the vector store with the following parameters:
+        # @param params [Hash] A Hash containing the initial configuration.
+        # @option params [Symbol] :embedding_tool The embedding tool to use.
+        # @option params [String] :training_data_path The path to the training data files.
+        # @option params [Integer] :split_chunk_size The number of characters to split the text into.
+        # @return [Hash] vector_store: array of hashes with :content, :metadata, and :embedding keys
         def initialize(params)
           @split_chunk_size = params[:split_chunk_size] || 2000
           @training_data_path = File.absolute_path(params[:training_data_path])
@@ -15,6 +21,7 @@ module Boxcars
           @memory_vectors = []
         end
 
+        # @return [Hash] vector_store: array of hashes with :content, :metadata, and :embedding keys
         def call
           data = load_data_files(training_data_path)
           texts = split_text_into_chunks(data)

data/lib/boxcars/vector_store/in_memory/search.rb

@@ -6,6 +6,10 @@ module Boxcars
       class Search
         include VectorStore
 
+        # initialize the vector store InMemory::Search with the following parameters:
+        # @param params [Hash] A Hash containing the initial configuration.
+        # @option params [Hash] :vector_documents The vector documents to search.
+        # @option params [Hash] :vector_store The vector store to search.
         def initialize(params)
           validate_params(params[:vector_documents])
           @vector_documents = params[:vector_documents]

data/lib/boxcars/vector_store/pgvector/build_from_array.rb

@@ -7,15 +7,24 @@ module Boxcars
       class BuildFromArray
         include VectorStore
 
-        # params =  {
-        #   embedding_tool: embedding_tool,
-        #   input_array: input_array,
-        #   database_url: db_url,
-        #   table_name: table_name,
-        #   embedding_column_name: embedding_column_name,
-        #   content_column_name: content_column_name,
-        #   metadata_column_name: metadata_column_name
-        # }
+        # initialize the vector store with the following parameters:
+        #
+        # @param params [Hash] A Hash containing the initial configuration.
+        #
+        # @option params [Symbol] :embedding_tool The embedding tool to use. Must be provided.
+        # @option params [Array] :input_array The array of inputs to use for the embedding tool. Must be provided.
+        # each hash item should have content and metadata
+        # [
+        #   { content: "hello", metadata: { a: 1 } },
+        #   { content: "hi", metadata: { a: 1 } },
+        #   { content: "bye", metadata: { a: 1 } },
+        #   { content: "what's this", metadata: { a: 1 } }
+        # ]
+        # @option params [String] :database_url The URL of the database where embeddings are stored. Must be provided.
+        # @option params [String] :table_name The name of the database table where embeddings are stored. Must be provided.
+        # @option params [String] :embedding_column_name The name of the database column where embeddings are stored. required.
+        # @option params [String] :content_column_name The name of the database column where content is stored. Must be provided.
+        # @option params [String] :metadata_column_name The name of the database column where metadata is stored. required.
         def initialize(params)
           @embedding_tool = params[:embedding_tool] || :openai
 
@@ -31,10 +40,11 @@ module Boxcars
           @pg_vectors = []
         end
 
+        # @return [Hash] vector_store: array of hashes with :content, :metadata, and :embedding keys
         def call
-          texts = input_array
+          texts = input_array.map { |doc| doc[:content] }
           vectors = generate_vectors(texts)
-          add_vectors(vectors, texts)
+          add_vectors(vectors, input_array)
           documents = save_vector_store
 
           {
@@ -51,15 +61,18 @@ module Boxcars
 
         def validate_params(embedding_tool, input_array)
           raise_argument_error('input_array is nil') unless input_array
+          raise_argument_error('input_array must be an array') unless input_array.is_a?(Array)
+          raise_argument_error('items in input_array needs to have content and metadata') unless proper_input_array?(input_array)
           return if %i[openai tensorflow].include?(embedding_tool)
 
           raise_argument_error('embedding_tool is invalid') unless %i[openai tensorflow].include?(embedding_tool)
+        end
 
-          input_array.each do |item|
-            next if item.key?(:content) && item.key?(:metadata)
+        def proper_input_array?(input_array)
+          return false unless
+            input_array.all? { |hash| hash.key?(:content) && hash.key?(:metadata) }
 
-            return raise_argument_error('embedding_tool is invalid')
-          end
+          true
         end
 
         def add_vectors(vectors, texts)

data/lib/boxcars/vector_store/pgvector/build_from_files.rb

@@ -10,15 +10,15 @@ module Boxcars
       class BuildFromFiles
         include VectorStore
 
-        # params = {
-        #   training_data_path: training_data_path,
-        #   split_chunk_size: 200,
-        #   embedding_tool: embedding_tool,
-        #   database_url: db_url,
-        #   table_name: table_name,
-        #   embedding_column_name: embedding_column_name,
-        #   content_column_name: content_column_name
-        # }
+        # @param training_data_path [String] path to training data files
+        # @param split_chunk_size [Integer] number of characters to split the text into
+        # @param embedding_tool [Symbol] embedding tool to use
+        # @param database_url [String] database url
+        # @param table_name [String] table name
+        # @param embedding_column_name [String] embedding column name
+        # @param content_column_name [String] content column name
+        # @param metadata_column_name [String] metadata column name
+        # @return [Hash] vector_store: array of hashes with :content, :metadata, and :embedding keys
         def initialize(params)
           @split_chunk_size = params[:split_chunk_size] || 2000
           @training_data_path = File.absolute_path(params[:training_data_path])
@@ -35,6 +35,7 @@ module Boxcars
           @pg_vectors = []
         end
 
+        # @return [Hash] vector_store: array of Inventor::VectorStore::Document
         def call
           data = load_data_files(training_data_path)
           texts = split_text_into_chunks(data)
@@ -57,7 +58,7 @@ module Boxcars
         def validate_params(embedding_tool, training_data_path)
           training_data_dir = File.dirname(training_data_path.gsub(/\*{1,2}/, ''))
 
-          raise_argument_error('training_data_path parent directory must exist') unless File.directory?(training_data_dir)
+          raise_argument_error('training_data_path parent directory must exist') unless Dir.exist?(training_data_dir)
           raise_argument_error('No files found at the training_data_path pattern') if Dir.glob(training_data_path).empty?
           return if %i[openai tensorflow].include?(embedding_tool)
 

data/lib/boxcars/vector_store/pgvector/save_to_database.rb

@@ -9,15 +9,14 @@ module Boxcars
       class SaveToDatabase
         include VectorStore
 
-        # params = {
-        #   pg_vectors: pg_vectors,
-        #   database_url: db_url,
-        #   table_name: table_name,
-        #   embedding_column_name: embedding_column_name,
-        #   content_column_name: content_column_name
-        # }
+        # @param pg_vectors [Array] array of Boxcars::VectorStore::Document
+        # @param database_url [String] database url
+        # @param table_name [String] table name
+        # @param embedding_column_name [String] embedding column name
+        # @param content_column_name [String] content column name
+        # @param metadata_column_name [String] metadata column name
+        # @return [Array] array of Boxcars::VectorStore::Document
         def initialize(params)
-          @errors = []
           validate_param_types(params)
           @db_connection = test_db_params(params)
 
@@ -29,9 +28,8 @@ module Boxcars
           @pg_vectors = params[:pg_vectors]
         end
 
+        # @return [Array] array of Boxcars::VectorStore::Document
         def call
-          return { success: false, error: errors } unless errors.empty?
-
           add_vectors_to_database
         end
 
@@ -39,7 +37,7 @@ module Boxcars
 
         attr_reader :database_url, :pg_vectors, :db_connection, :table_name,
                     :embedding_column_name, :content_column_name,
-                    :metadata_column_name, :errors
+                    :metadata_column_name
 
         def validate_param_types(params)
           pg_vectors = params[:pg_vectors]

data/lib/boxcars/vector_store/pgvector/search.rb

@@ -9,17 +9,21 @@ module Boxcars
       class Search
         include VectorStore
 
-        # required params:
+        # initialize the vector store with the following parameters:
+        # @param params [Hash] A Hash containing the initial configuration.
+        # @option params [Hash] :vector_documents The vector documents to search.
+        # example:
         # {
         #   type: :pgvector,
         #   vector_store: {
-        #     database_url: database_url,
-        #     table_name: table_name,
-        #     embedding_column_name: embedding_column_name,
-        #     content_column_name: content_column_name,
-        #     metadata_column_name: metadata_column_name
+        #     table_name: "vector_store",
+        #     embedding_column_name: "embedding",
+        #     content_column_name: "content",
+        #     database_url: ENV['DATABASE_URL']
         #   }
         # }
+        #
+        # @option params [Hash] :vector_store The vector store to search.
         def initialize(params)
           vector_store = validate_params(params)
           db_url = validate_vector_store(vector_store)
@@ -28,6 +32,20 @@ module Boxcars
           @vector_documents = params[:vector_documents]
         end
 
+        # @param query_vector [Array] The query vector to search for.
+        # @param count [Integer] The number of results to return.
+        # @return [Array] array of hashes with :document and :distance keys
+        # @example
+        #   [
+        #     {
+        #       document: Boxcars::VectorStore::Document.new(
+        #         content: "hello",
+        #         embedding: [0.1, 0.2, 0.3],
+        #         metadata: { a: 1 }
+        #       ),
+        #       distance: 0.1
+        #     }
+        #   ]
         def call(query_vector:, count: 1)
           raise ::Boxcars::ArgumentError, 'query_vector is empty' if query_vector.empty?
 

data/lib/boxcars/vector_store.rb

@@ -54,7 +54,7 @@ module Boxcars
 
       file_content = File.read(file_path)
       JSON.parse(file_content, symbolize_names: true)
-    rescue JSON::ParserError => e
+    rescue JSON::ParserError, Errno::ENOENT => e
       raise_argument_error("Error parsing #{file_path}: #{e.message}")
     end
 
@@ -80,6 +80,11 @@ module Boxcars
       end
       docs
     end
+
+    def validate_string(value, name)
+      raise raise_argument_error("#{name} must be a string") unless value.is_a?(String)
+      raise raise_argument_error("#{name} is empty") if value.empty?
+    end
   end
 end
 
@@ -92,7 +97,7 @@ require_relative "vector_store/hnswlib/save_to_hnswlib"
 require_relative "vector_store/hnswlib/build_from_files"
 require_relative "vector_store/hnswlib/search"
 require_relative "vector_store/in_memory/build_from_files"
-require_relative "vector_store/in_memory/build_from_document_array"
+require_relative "vector_store/in_memory/build_from_array"
 require_relative "vector_store/in_memory/search"
 require_relative "vector_store/pgvector/build_from_files"
 require_relative "vector_store/pgvector/build_from_array"

data/lib/boxcars/version.rb

@@ -2,5 +2,5 @@
 
 module Boxcars
   # The current version of the gem.
-  VERSION = "0.2.13"
+  VERSION = "0.2.14"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: boxcars
 version: !ruby/object:Gem::Version
-  version: 0.2.13
+  version: 0.2.14
 platform: ruby
 authors:
 - Francis Sullivan
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-05-24 00:00:00.000000000 Z
+date: 2023-06-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: google_search_results
@@ -54,33 +54,33 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0.8'
 - !ruby/object:Gem::Dependency
-  name: ruby-openai
+  name: pgvector
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.1'
+        version: '0.2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.1'
+        version: '0.2'
 - !ruby/object:Gem::Dependency
-  name: pgvector
+  name: ruby-openai
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.2'
+        version: '4.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.2'
+        version: '4.1'
 description: You simply set an OpenAI key, give a number of Boxcars to a Train, and
   magic ensues when you run it.
 email:
@@ -135,7 +135,7 @@ files:
 - lib/boxcars/vector_store/hnswlib/load_from_disk.rb
 - lib/boxcars/vector_store/hnswlib/save_to_hnswlib.rb
 - lib/boxcars/vector_store/hnswlib/search.rb
-- lib/boxcars/vector_store/in_memory/build_from_document_array.rb
+- lib/boxcars/vector_store/in_memory/build_from_array.rb
 - lib/boxcars/vector_store/in_memory/build_from_files.rb
 - lib/boxcars/vector_store/in_memory/search.rb
 - lib/boxcars/vector_store/pgvector/build_from_array.rb

data/lib/boxcars/vector_store/in_memory/build_from_document_array.rb

@@ -1,51 +0,0 @@
-# frozen_string_literal: true
-
-module Boxcars
-  module VectorStore
-    module InMemory
-      class BuildFromDocumentArray
-        include VectorStore
-
-        def initialize(embedding_tool: :openai, documents: nil)
-          validate_params(embedding_tool, documents)
-          @embedding_tool = embedding_tool
-          @documents = documents
-          @memory_vectors = []
-        end
-
-        def call
-          texts = documents
-          vectors = generate_vectors(texts)
-          add_vectors(vectors, documents)
-          {
-            type: :in_memory,
-            vector_store: memory_vectors
-          }
-        end
-
-        private
-
-        attr_reader :documents, :memory_vectors
-
-        def validate_params(embedding_tool, documents)
-          raise_argument_error('documents is nil') unless documents
-          return if %i[openai tensorflow].include?(embedding_tool)
-
-          raise_argument_error('embedding_tool is invalid')
-        end
-
-        # returns array of documents with vectors
-        def add_vectors(vectors, documents)
-          vectors.zip(documents).each do |vector, doc|
-            memory_vector = Document.new(
-              content: doc[:content],
-              embedding: vector[:embedding],
-              metadata: doc[:metadata].merge(dim: vector[:dim])
-            )
-            @memory_vectors << memory_vector
-          end
-        end
-      end
-    end
-  end
-end