neighbor 0.2.1 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8a71d9aff5133b9551dc47cfe490d5d6a0ee63d2b77ac1d3d2f381f58aef67d6
-  data.tar.gz: d22cab3140b979ee0d13ddd2c99f064282e896d49c265bf9b1946dafbda7783a
+  metadata.gz: 48a9bcfda91ac7ed0af8c593216b9a84a2b488d15e9375d897f063e44bd0f5c8
+  data.tar.gz: c4cf3ca35811336d7574eff4733078cc08e4b1cb7140cf5c5a12ad06a34506e7
 SHA512:
-  metadata.gz: 3e24a71c6ace693525fc4866aaa21ee7c75f7251d5d6e0820da3b6abc0bd6fd5921b20c2edbb7fc656b75991e1fc5607357dc02870072900e7b6ef972c840280
-  data.tar.gz: 5b75200355c62c549c2d61820e74e1cc71e4ebce68a1e390af60dbe76b6e1b7a69e71d73c8121db42ea05fc296db4ecc1f5776c71ad988685f13f71fc00fec66
+  metadata.gz: 83d90c764613158ca0a753796c9d1ccbe19127792ac8a73c32eddcc2c6537ad919c21e32c43be5b88833eae92fc5d723e3275e07307e17c0951e8436faed27b5
+  data.tar.gz: 990ad6a45d982e7ababbb57112a65d1fcf8a37f3a37a6459cf4b0483aee407cfc3eda2575e6ae3ecd8389daaca8925b045d8df50c8787f0d1102e9e913b4269f

data/CHANGELOG.md

@@ -1,3 +1,12 @@
+## 0.2.3 (2023-04-02)
+
+- Added support for dimensions to model generator
+
+## 0.2.2 (2022-07-13)
+
+- Added support for configurable attribute name
+- Added support for multiple attributes per model
+
 ## 0.2.1 (2021-12-15)
 
 - Added support for Active Record 7

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2021 Andrew Kane
+Copyright (c) 2021-2023 Andrew Kane
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -9,12 +9,12 @@ Nearest neighbor search for Rails and Postgres
 Add this line to your application’s Gemfile:
 
 ```ruby
-gem 'neighbor'
+gem "neighbor"
 ```
 
 ## Choose An Extension
 
-Neighbor supports two extensions: [cube](https://www.postgresql.org/docs/current/cube.html) and [vector](https://github.com/ankane/pgvector). cube ships with Postgres, while vector supports approximate nearest neighbor search.
+Neighbor supports two extensions: [cube](https://www.postgresql.org/docs/current/cube.html) and [vector](https://github.com/pgvector/pgvector). cube ships with Postgres, while vector supports approximate nearest neighbor search.
 
 For cube, run:
 
@@ -23,7 +23,7 @@ rails generate neighbor:cube
 rails db:migrate
 ```
 
-For vector, [install pgvector](https://github.com/ankane/pgvector#installation) and run:
+For vector, [install pgvector](https://github.com/pgvector/pgvector#installation) and run:
 
 ```sh
 rails generate neighbor:vector
@@ -35,11 +35,11 @@ rails db:migrate
 Create a migration
 
 ```ruby
-class AddNeighborVectorToItems < ActiveRecord::Migration[6.1]
+class AddNeighborVectorToItems < ActiveRecord::Migration[7.0]
   def change
-    add_column :items, :neighbor_vector, :cube
+    add_column :items, :embedding, :cube
     # or
-    add_column :items, :neighbor_vector, :vector, limit: 3 # dimensions
+    add_column :items, :embedding, :vector, limit: 3 # dimensions
   end
 end
 ```
@@ -48,26 +48,26 @@ Add to your model
 
 ```ruby
 class Item < ApplicationRecord
-  has_neighbors
+  has_neighbors :embedding
 end
 ```
 
 Update the vectors
 
 ```ruby
-item.update(neighbor_vector: [1.0, 1.2, 0.5])
+item.update(embedding: [1.0, 1.2, 0.5])
 ```
 
 Get the nearest neighbors to a record
 
 ```ruby
-item.nearest_neighbors(distance: "euclidean").first(5)
+item.nearest_neighbors(:embedding, distance: "euclidean").first(5)
 ```
 
 Get the nearest neighbors to a vector
 
 ```ruby
-Item.nearest_neighbors([0.9, 1.3, 1.1], distance: "euclidean").first(5)
+Item.nearest_neighbors(:embedding, [0.9, 1.3, 1.1], distance: "euclidean").first(5)
 ```
 
 ## Distance
@@ -84,7 +84,7 @@ For cosine distance with cube, vectors must be normalized before being stored.
 
 ```ruby
 class Item < ApplicationRecord
-  has_neighbors normalize: true
+  has_neighbors :embedding, normalize: true
 end
 ```
 
@@ -93,19 +93,19 @@ For inner product with cube, see [this example](examples/disco_user_recs_cube.rb
 Records returned from `nearest_neighbors` will have a `neighbor_distance` attribute
 
 ```ruby
-nearest_item = item.nearest_neighbors(distance: "euclidean").first
+nearest_item = item.nearest_neighbors(:embedding, distance: "euclidean").first
 nearest_item.neighbor_distance
 ```
 
 ## Dimensions
 
-The cube data type is limited 100 dimensions by default. See the [Postgres docs](https://www.postgresql.org/docs/current/cube.html) for how to increase this. The vector data type is limited to 1024 dimensions.
+The cube data type can have up to 100 dimensions by default. See the [Postgres docs](https://www.postgresql.org/docs/current/cube.html) for how to increase this. The vector data type can have up to 16,000 dimensions, and vectors with up to 2,000 dimensions can be indexed.
 
 For cube, it’s a good idea to specify the number of dimensions to ensure all records have the same number.
 
 ```ruby
-class Movie < ApplicationRecord
-  has_neighbors dimensions: 3
+class Item < ApplicationRecord
+  has_neighbors :embedding, dimensions: 3
 end
 ```
 
@@ -114,9 +114,9 @@ end
 For vector, add an approximate index to speed up queries. Create a migration with:
 
 ```ruby
-class AddIndexToItemsNeighborVector < ActiveRecord::Migration[6.1]
+class AddIndexToItemsNeighborVector < ActiveRecord::Migration[7.0]
   def change
-    add_index :items, :neighbor_vector, using: :ivfflat, opclass: :vector_l2_ops
+    add_index :items, :embedding, using: :ivfflat, opclass: :vector_l2_ops
   end
 end
 ```
@@ -129,14 +129,85 @@ Set the number of probes
 Item.connection.execute("SET ivfflat.probes = 3")
 ```
 
-## Example
+## Examples
+
+- [OpenAI Embeddings](#openai-embeddings)
+- [Disco Recommendations](#disco-recommendations)
+
+### OpenAI Embeddings
+
+Generate a model
+
+```sh
+rails generate model Article content:text embedding:vector{1536}
+rails db:migrate
+```
+
+And add `has_neighbors`
+
+```ruby
+class Article < ApplicationRecord
+  has_neighbors :embedding
+end
+```
+
+Create a method to call the [embeddings API](https://platform.openai.com/docs/guides/embeddings)
+
+```ruby
+def fetch_embeddings(input)
+  url = "https://api.openai.com/v1/embeddings"
+  headers = {
+    "Authorization" => "Bearer #{ENV.fetch("OPENAI_API_KEY")}",
+    "Content-Type" => "application/json"
+  }
+  data = {
+    input: input,
+    model: "text-embedding-ada-002"
+  }
+
+  response = Net::HTTP.post(URI(url), data.to_json, headers)
+  JSON.parse(response.body)["data"].map { |v| v["embedding"] }
+end
+```
+
+Pass your input
+
+```ruby
+input = [
+  "The dog is barking",
+  "The cat is purring",
+  "The bear is growling"
+]
+embeddings = fetch_embeddings(input)
+```
+
+Store the embeddings
+
+```ruby
+articles = []
+input.zip(embeddings) do |content, embedding|
+  articles << {content: content, embedding: embedding}
+end
+Article.insert_all!(articles) # use create! for Active Record < 6
+```
+
+And get similar articles
+
+```ruby
+article = Article.first
+article.nearest_neighbors(:embedding, distance: "inner_product").first(5).map(&:content)
+```
+
+See the [complete code](examples/openai_embeddings.rb)
+
+### Disco Recommendations
 
 You can use Neighbor for online item-based recommendations with [Disco](https://github.com/ankane/disco). We’ll use MovieLens data for this example.
 
 Generate a model
 
 ```sh
-rails generate model Movie name:string neighbor_vector:cube
+rails generate model Movie name:string factors:cube
 rails db:migrate
 ```
 
@@ -144,7 +215,7 @@ And add `has_neighbors`
 
 ```ruby
 class Movie < ApplicationRecord
-  has_neighbors dimensions: 20, normalize: true
+  has_neighbors :factors, dimensions: 20, normalize: true
 end
 ```
 
@@ -156,12 +227,12 @@ recommender = Disco::Recommender.new(factors: 20)
 recommender.fit(data)
 ```
 
-Use item factors for the neighbor vector
+Store the item factors
 
 ```ruby
 movies = []
 recommender.item_ids.each do |item_id|
-  movies << {name: item_id, neighbor_vector: recommender.item_factors(item_id)}
+  movies << {name: item_id, factors: recommender.item_factors(item_id)}
 end
 Movie.insert_all!(movies) # use create! for Active Record < 6
 ```
@@ -170,7 +241,7 @@ And get similar movies
 
 ```ruby
 movie = Movie.find_by(name: "Star Wars (1977)")
-movie.nearest_neighbors(distance: "cosine").first(5).map(&:name)
+movie.nearest_neighbors(:factors, distance: "cosine").first(5).map(&:name)
 ```
 
 See the complete code for [cube](examples/disco_item_recs_cube.rb) and [vector](examples/disco_item_recs_vector.rb)

data/lib/neighbor/model.rb

@@ -1,16 +1,43 @@
 module Neighbor
   module Model
-    def has_neighbors(dimensions: nil, normalize: nil)
-      # TODO make configurable
-      # likely use argument
-      attribute_name = :neighbor_vector
+    def has_neighbors(attribute_name = :neighbor_vector, dimensions: nil, normalize: nil)
+      attribute_name = attribute_name.to_sym
 
       class_eval do
-        raise Error, "nearest_neighbors already defined" if method_defined?(:nearest_neighbors)
+        @neighbor_attributes ||= {}
+
+        if @neighbor_attributes.empty?
+          def self.neighbor_attributes
+            parent_attributes =
+              if superclass.respond_to?(:neighbor_attributes)
+                superclass.neighbor_attributes
+              else
+                {}
+              end
+
+            parent_attributes.merge(@neighbor_attributes || {})
+          end
+        end
+
+        raise Error, "has_neighbors already called for #{attribute_name.inspect}" if neighbor_attributes[attribute_name]
+        @neighbor_attributes[attribute_name] = {dimensions: dimensions, normalize: normalize}
 
         attribute attribute_name, Neighbor::Vector.new(dimensions: dimensions, normalize: normalize, model: self, attribute_name: attribute_name)
 
-        scope :nearest_neighbors, ->(vector, distance:) {
+        return if @neighbor_attributes.size != 1
+
+        scope :nearest_neighbors, ->(attribute_name, vector = nil, distance:) {
+          if vector.nil? && !attribute_name.nil? && attribute_name.respond_to?(:to_a)
+            vector = attribute_name
+            attribute_name = :neighbor_vector
+          end
+          attribute_name = attribute_name.to_sym
+
+          options = neighbor_attributes[attribute_name]
+          raise ArgumentError, "Invalid attribute" unless options
+          normalize = options[:normalize]
+          dimensions = options[:dimensions]
+
           return none if vector.nil?
 
           distance = distance.to_s
@@ -80,10 +107,14 @@ module Neighbor
             .order(Arel.sql(order))
         }
 
-        define_method :nearest_neighbors do |**options|
+        def nearest_neighbors(attribute_name = :neighbor_vector, **options)
+          attribute_name = attribute_name.to_sym
+          # important! check if neighbor attribute before calling send
+          raise ArgumentError, "Invalid attribute" unless self.class.neighbor_attributes[attribute_name]
+
           self.class
             .where.not(self.class.primary_key => send(self.class.primary_key))
-            .nearest_neighbors(send(attribute_name), **options)
+            .nearest_neighbors(attribute_name, send(attribute_name), **options)
         end
       end
     end

data/lib/neighbor/railtie.rb

@@ -0,0 +1,19 @@
+module Neighbor
+  class Railtie < Rails::Railtie
+    generators do
+      # rails generate model Item embedding:vector{3}
+      if defined?(Rails::Generators::GeneratedAttribute)
+        Rails::Generators::GeneratedAttribute.singleton_class.prepend(Neighbor::GeneratedAttribute)
+      end
+    end
+  end
+
+  module GeneratedAttribute
+    def parse_type_and_options(type, *, **)
+      if type =~ /\A(vector)\{(\d+)\}\z/
+        return $1, limit: $2.to_i
+      end
+      super
+    end
+  end
+end

data/lib/neighbor/version.rb

@@ -1,3 +1,3 @@
 module Neighbor
-  VERSION = "0.2.1"
+  VERSION = "0.2.3"
 end

data/lib/neighbor.rb

@@ -50,3 +50,5 @@ ActiveSupport.on_load(:active_record) do
     ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend(Neighbor::RegisterTypes)
   end
 end
+
+require "neighbor/railtie" if defined?(Rails::Railtie)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: neighbor
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.3
 platform: ruby
 authors:
 - Andrew Kane
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-12-16 00:00:00.000000000 Z
+date: 2023-04-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -39,6 +39,7 @@ files:
 - lib/generators/neighbor/vector_generator.rb
 - lib/neighbor.rb
 - lib/neighbor/model.rb
+- lib/neighbor/railtie.rb
 - lib/neighbor/vector.rb
 - lib/neighbor/version.rb
 homepage: https://github.com/ankane/neighbor
@@ -60,7 +61,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.32
+rubygems_version: 3.4.10
 signing_key:
 specification_version: 4
 summary: Nearest neighbor search for Rails and Postgres