dynamic-records-meritfront 2.0.21 → 3.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0145c6915704e699ab557a6e45348a1555a774e79f4a49925be7f41d88877a6d
-  data.tar.gz: ebed51c113a9f94b9f4bc481069da98b8771c5b6aa24af2403b9a1283869fed2
+  metadata.gz: 6a2496e72da8096b231edb9fab7d9a469926bbcd9a698d0224a78ee7ed5e25a5
+  data.tar.gz: 23100e7e24dad43bdfeab2f1291d4616c72e3726eba69a1c3cb2be99cf73df19
 SHA512:
-  metadata.gz: cc6978059ea84ee5584a967bbf0c465ebd81fdaee751dd94c745811f2704216b5992d433ed4168f67187fd18e485d586ca220cc468e617b93da3798e5ed8b475
-  data.tar.gz: 6efa88c7bb6f60789301660d77d874b2252fd69fe0b4de66d223d1df863df9518dce2c89bd60d6f6a30ff167535d7adf9392ea9320979a59780a2b9e130737d7
+  metadata.gz: a2861fed694119175940f5153f429491cc25210611da866e55b5a52720610655a89b8291dd7bf1a1042fa155ce8c7dbeacbb846470e01645dda9381d868c6b42
+  data.tar.gz: 84110301448a31b997b6acbd89b99dfbe3d5b3c6432e7ba7cafc20b78bfa92a80818a4291da6a30090643fc7eec3da2028f8a30030339e0ac7cdd90f7c12d49a

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    dynamic-records-meritfront (2.0.21)
+    dynamic-records-meritfront (3.0.1)
       hashid-rails
 
 GEM

data/README.md

@@ -1,10 +1,33 @@
 # Dynamic Records Meritfront
 
-Dyanmic Records Meritfront contains some helpers methods for active record. These methods have the goal of allowing one to
+Dyanmic Records Meritfront helps extend active record functionality to make it more dynamic. These methods have the goal of allowing one to
 1. communicate with the frontend quicker and more effectively through Global HashIds
-2. communicate with the backend more effectively with raw sql queries. This becomes especially relevant when you hit the limits of Active Record Relations and the usual way of querying in rails. For instance, if you have a page-long dynamic sql query.
+2. communicate with the backend more effectively with sql queries. This becomes especially relevant when you hit the limits of Active Record Relations and the usual way of querying in rails. For instance, if you have dynamic sql queries that are hard to convert properly into ruby.
+3. add other helper methods to work with your database, such as checking if relations exist, or if a migration has been run.
 
-Note that postgres is a requirement for this gem. I dont tend to get much feedback, so any given would be appreciated.
+Note that postgres is currently a requirement for this gem.
+
+## Basic Examples
+```ruby
+# returns a json-like hash list of user data
+users = ApplicationRecord.dynamic_sql(
+	'get_5_users',
+	'select * from users limit :our_limit',
+	our_limit: 5
+)
+
+#returns a list of users (each an instance of User)
+users = User.dynamic_sql(
+	'get_users_from_ids',
+	'select * from users where id = ANY (:ids)',
+	ids: [1,2,3]
+)
+
+uhgid = users.first.hgid		#returns a hashed global id like: 'gid://appname/User/K9YI4K'
+user = User.locate_hgid(uhgid)		#returns that user
+
+#... and much more!
+```
 
 ## Installation
 
@@ -38,58 +61,21 @@ end
 
 These are methods written for easier sql usage.
 
-#### has_run_migration?(nm)
-
-put in a string name of the migration's class and it will say if it has allready run the migration.
-good during enum migrations as the code to migrate wont run if enumerate is there 
-as it is not yet enumerated (causing an error when it loads the class that will have the
-enumeration in it). This can lead it to being impossible to commit clean code.
-
-<details><summary>example usage</summary>
-only load relationa if it exists in the database
-
-```ruby
-if ApplicationRecord.has_run_migration?('UserImageRelationsTwo')
-    class UserImageRelation < ApplicationRecord
-        belongs_to :imageable, polymorphic: true
-        belongs_to :image
-    end
-else
-    class UserImageRelation; end
-end
-
-```
-</details>
-
-#### has_association?(*args)
-
-accepts a list, checks if the model contains those associations
-
-<details><summary>example usage</summary>
-Check if object is a votable class
-
-```ruby
-obj = Comment.first
-obj.has_association?(:votes) #true
-obj = User.first
-obj.has_association?(:votes) #false
-```
-</details>
-
 #### self.dynamic_sql(name, sql, opts = { })
 A better and safer way to write sql. Can return either a Hash, ActiveRecord::Response object, or an instantiated model.
+
 with options: 
-- instantiate_class: returns User, Post, etc objects instead of straight sql output.
-    I prefer doing the alterantive
-        ```User.headache_sql(...)```
-        which is also supported
-- prepare: sets whether the db will preprocess the strategy for lookup (defaults true) (have not verified the prepared-ness)
-- name_modifiers: allows one to change the preprocess associated name, useful in cases of dynamic sql.
+- options not stated below: considered sql arguments, and will replace their ":option_name" with a sql argument. Always use sql arguments to avoid sql injection. Lists are converted into a format such as ```{1,2,3,4}```. Lists of lists are converted into ```(1,2,3), (4,5,6), (7,8,9)``` etc. So as to allow easy inserts/upserts.
+- raw: whether to return a ActiveRecord::Response object or a hash when called on an abstract class (like ApplicationRecord). Default can be switched with DYNAMIC_SQL_RAW variable on the class level.
+- instantiate_class: determines what format to return. Can return ActiveRecord objects (User, Post, etc), or whatever raw is set to. I prefer doing the alterantive ```User.dynamic_sql(...)``` which is also supported. For example, ```User.dynamic_sql(...)``` will return User records. ```ApplicationRecord.dynamic_sql(..., raw: false)``` will return a List of Hashes with the column names as keys. ```ApplicationRecord.dynamic_sql(..., raw: true)``` will return an ActiveRecord::Response.
+
+other options:
+
+- prepare: Defaults to true. Gets passed to ActiveRecord::Base.connection.exec_query as a parameter. Should change whether the command will be prepared, which means that on subsequent calls the command will be faster. Downsides are when, for example, the sql query has hard-coded arguments, the query always changes, causing technical issues as the number of prepared statements stack up.
+- name_modifiers: allows one to change the associated name dynamically.
 - multi_query: allows more than one query (you can seperate an insert and an update with ';' I dont know how else to say it.)
-    this disables other options including arguments (except name_modifiers). Not sure how it effects prepared statements.
-- async: Gets passed to ActiveRecord::Base.connection.exec_query as a parameter. See that methods documentation for more. I was looking through the source code, and I think it only effects how it logs to the logfile?
-- raw: whether to return a ActiveRecord::Response object or a hash
-- other options: considered sql arguments
+    this disables other options including arguments (except name_modifiers). Not sure how it effects prepared statements. Not super useful.
+- async: Defaults to false. Gets passed to ActiveRecord::Base.connection.exec_query as a parameter. See that methods documentation for more. I was looking through the source code, and I think it only effects how it logs to the logfile?
 	
 <details>
 <summary>example usage</summary>
@@ -150,7 +136,7 @@ Get users who match a list of ids. Uses a postgresql Array, see the potential is
 
 ```ruby
 	id_list = [1,2,3]
-	return User.headache_sql('get_usrs', %Q{
+	return User.dynamic_sql('get_usrs', %Q{
 		SELECT * FROM users WHERE id = ANY (:id_list)
 	}, id_list: id_list)
 ```
@@ -175,7 +161,7 @@ Do an upsert
 		DO UPDATE SET updated_at = :time
 	}, rows: rows, time: t)
 ```
-This will output sql similar to below. Note this can be done for multiple conversation_participants. Also note that it only sent one time variable as an argument as headache_sql detected that we were sending duplicate information.
+This will output sql similar to below. Note this can be done for multiple conversation_participants. Also note that it only sent one time variable as an argument as dynamic_sql detected that we were sending duplicate information.
 ```sql
 	INSERT INTO conversation_participants (user_id, conversation_id, invited_by, created_at, updated_at)
 	VALUES ($1,$2,$3,$4,$4)
@@ -187,16 +173,17 @@ This will output sql similar to below. Note this can be done for multiple conver
 	
 
 #### self.dynamic_preload(records, associations)
-Preloads from a list of records, and not from a ActiveRecord_Relation. This will be useful when using the above headache_sql method (as it returns a list of records, and not a record relation). This is basically the same as a normal relation preload but it works on a list. 
+Preloads from a list of records, and not from a ActiveRecord_Relation. This will be useful when using the above dynamic_sql method (as it returns a list of records, and not a record relation). This is basically the same as a normal relation preload but it works on a list. 
 
 <details>
 <summary>example usage</summary>
 Preload :votes on some comments. :votes is an active record has_many relation.
 
 ```ruby
-    comments = Comment.headache_sql('get_comments', %Q{
+    comments = Comment.dynamic_sql('get_comments', %Q{
         SELECT * FROM comments LIMIT 4
     })
+    comments.class.to_s # 'Array' note: not a relation.
     ApplicationRecord.headache_preload(comments, [:votes])
     puts comments[0].votes #this line should be preloaded and hence not call the database
 
@@ -204,12 +191,50 @@ Preload :votes on some comments. :votes is an active record has_many relation.
     user.comments.preload(:votes)
 ```
 </details>
-	
+
+#### has_run_migration?(nm)
+
+put in a string name of the migration's class and it will say if it has allready run the migration.
+good during enum migrations as the code to migrate wont run if enumerate is there 
+as it is not yet enumerated (causing an error when it loads the class that will have the
+enumeration in it). This can lead it to being impossible to commit clean code.
+
+<details><summary>example usage</summary>
+only load relationa if it exists in the database
+
+```ruby
+if ApplicationRecord.has_run_migration?('UserImageRelationsTwo')
+    class UserImageRelation < ApplicationRecord
+        belongs_to :imageable, polymorphic: true
+        belongs_to :image
+    end
+else
+    class UserImageRelation; end
+end
+
+```
+</details>
+
+#### has_association?(*args)
+
+accepts a list of association names, checks if the model has those associations
+
+<details><summary>example usage</summary>
+Check if object is a votable class
+
+```ruby
+obj = Comment.first
+obj.has_association?(:votes) #true
+obj = User.first
+obj.has_association?(:votes) #false
+```
+</details>
+
 #### self.dynamic_instaload_sql(name, insta_array, opts = { })
-*instaloads* a bunch of diffrent models at the same time by casting them to json before returning them. Kinda cool. Seems to be more efficient to preloading when i tested it.
+*instaloads* a bunch of diffrent models at the same time by casting them to json before returning them. Kinda cool. Maybe a bit overcomplicated. Seems to be more efficient to preloading when i tested it.
 - name is passed to dynamic_sql and is the name of the sql request
 - opts are passed to dynamic_sql (except for the raw option which is set to true. Raw output is not allowed on this request)
-- insta-array is an array of instaload method outputs. See examples for more.
+- requires a list of instaload method output which provides information for how to treat each sql block.
 	
 <details>
 <summary>example usage</summary>
@@ -274,11 +299,37 @@ the output:
    ...]}
 
 
+```
+</details>
+	
+#### self.instaload(sql, table_name: nil, relied_on: false, dont_return: false)
+A method used to prepare data for the dynamic_instaload_sql method. It returns a hash of options.
+- klass called on: if called on an abstract class (ApplicationRecord) it will return a list of hashes with the data. Otherwise returns a list of the classes records.
+- table_name: sets the name of the temporary postgresql table. This can then be used in further instaload sql snippets.
+- relied_on: will make it so other instaload sql snippets can reference this table (it makes it use posrgresql's WITH operator)
+- dont_return: when used with relied_on makes it so that this data is not returned to rails from the database.
+
+note that the order of the instaload methods matter depending on how they reference eachother.
+<details>
+<summary> format data </summary>
+
+```ruby
+
+User.instaload('SELECT id FROM users WHERE users.id = ANY (:user_ids) AND users.created_at > :time', table_name: 'limited_users', relied_on: true)
+#output:
+{
+    table_name: "limited_users",
+    klass: "User",
+    sql: "\tSELECT id FROM users WHERE users.id = ANY (:user_ids) AND users.created_at > :time",
+    relied_on: true,
+    dont_return: false
+}
+
 ```
 </details>
 
 #### self.dynamic_attach(instaload_sql_output, base_name, attach_name, base_on: nil, attach_on: nil, one_to_one: false)
-taking the output of the dynamic_instaload_sql, this method attaches the models together so they have relations.
+taking the output of the dynamic_instaload_sql, this method attaches the models together so they are attached.
 - base_name: the name of the table we will be attaching to
 - attach_name: the name of the table that will be attached
 - base_on: put a proc here to override the matching key for the base table. Default is, for a user and post type, {|user| user.id}
@@ -337,7 +388,8 @@ See the hashid-rails gem for more (https://github.com/jcypret/hashid-rails). Als
 
 ## Potential Issues
 
-This gem was made with a postgresql database. This could cause a lot of issues with the sql-related methods. I dont have the bandwidth to help switch it elsewhere.
+- This gem was made with a postgresql database. This could cause a lot of issues with the sql-related methods if you do not. I dont have the bandwidth to help switch it elsewhere, but if you want to take charge of that, I would be more than happy to assist by answering questions an pointing out any areas that need transitioning.
+- If you return a password column (for example) as pwd, this gem will accept that. That would mean that the password could me accessed as model.pwd. This is cool - until all passwords are getting logged in production servers. So be wary of accessing, storing, and logging of sensative information. Active Record has in built solutions for this type of data, as long as you dont change the column name. This gem is a sharp knife, its very versitile, but its also, you know, sharp.
 	
 ## Changelog
 	
@@ -363,6 +415,34 @@ This gem was made with a postgresql database. This could cause a lot of issues w
 - changed dynamic_attach so that it now uses the model.dynamic attribute, instead of using singleton classes. This is better practice, and also contains all the moving parts of this gem in one place.
 - added the dynamic_print method to easier see the objects one is working with.
 
+2.0.21
+- figured out how to add to a model's @attributes, so .dynamic OpenStruct no longer needed, no longer need dynamic_print, singletons are out aswell. unexpected columns are now usable as just regular attributes.
+- overrode inspect to show the dynamic attributes aswell, warning about passwords printed to logs etc.
+
+2.0.24
+- added error logging in dynamic_sql method for the sql query when and if that fails. So just look at log file to see exactly what sql was running and what the args are.
+- added a dont_return option to the instaload method which works with the relied_on option to have a normal WITH statement that is not returned.
+
+3.0.1
+- Previous versions will break when two sql attributes unexpectantly share the same value. Yeah my bad, was trying to be fancy and decrease sql argument count.
+- People using symbols as sql values (and expecting them to be turned to strings) may have breakages. (aka a sql option like "insert_list: 
+[[:a, 1], [:b, 2]]" will break)
+- setting DYNAMIC_SQL_RAW apparently did nothing, you actually need to set DynamicRecordsMeritfront::DYNAMIC_SQL_RAW. Changed default to false, which may break things. But
+since things may be broken already, it seemed like a good time to do this.
+- Went to new version due to 1. a large functionality improvement, 2. the fact that previous versions are broken as explained above.
+- more on breaking error
+  - got this error: ActiveRecord::StatementInvalid (PG::ProtocolViolation: ERROR:  bind message supplies 3 parameters, but prepared statement "a27" requires 4)
+  - this tells me that names are not actually required to be unique for prepared statement identification, which was a bad assumption on my part
+  - this also tells me that uniq'ing variables to decrease the number of them was a bad idea which could cause random failures.
+- functionality improvements
+  - The biggest change is that names are now optional! name_modifiers is now depreciated functionality as it serves no useful purpose. Will leave in for compatibility but take out of documentation. Used to think the name was related to prepared statements. This will lead simpler ruby code.
+  - If name is left out, the name will be set to the location in your app which called the method. For example, when dynamic_sql was called from irb, the name was: "(irb):45:in `irb_binding'". This is done using stack trace functionality.
+  - dynamic_instaload_sql is now just instaload_sql. dynamic_instaload_sql has been aliased.
+  - Name is optional on instaload_sql aswell
+  - MultiAttributeArrays (array's of arrays) which can be passed into dynamic_sql largely for inserts/upserts will now treat symbols as an attribute name. This leads to more consise sql without running into above error.
+  - When dynamic_sql errors out, it now posts some helpful information to the log.
+  - Added a test script. No experience testing, so its just a method you pass a model, and then it does a rollback to reverse any changes.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/LukeClancy/dynamic-records-meritfront. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/LukeClancy/dynamic-records-meritfront/blob/master/CODE_OF_CONDUCT.md).

data/lib/dynamic-records-meritfront/version.rb

@@ -1,5 +1,5 @@
 
 	module DynamicRecordsMeritfront
-	    VERSION = '2.0.21'
+	    VERSION = '3.0.1'
 	end
 	#this file gets overwritten automatically on minor updates, major ones need to be manually changed

data/lib/dynamic-records-meritfront.rb

@@ -11,15 +11,102 @@ module DynamicRecordsMeritfront
 	module Hashid::Rails::ClassMethods
 		alias hfind find_by_hashid
 	end
-
 	included do
 		# include hash id gem
 		include Hashid::Rails
 		#should work, probably able to override by redefining in ApplicationRecord class.
 		#Note we defined here as it breaks early on as Rails.application returns nil
 		PROJECT_NAME = Rails.application.class.to_s.split("::").first.to_s.downcase
-		DYNAMIC_SQL_RAW = true
+        DYNAMIC_SQL_RAW = false
 	end
+    class DynamicSqlVariables
+        attr_accessor :sql_hash
+        attr_accessor :params
+        def initialize(params)
+            @sql_hash = {}
+            self.params = params
+        end
+
+        def add_key_value(key, value = nil)
+            value = params[key] if value.nil?
+            #tracks the variable and returns the keys sql variable number
+            sql_hash[key] ||= convert_to_query_attribute(key, value)
+            return sql_hash.keys.index(key) + 1
+        end
+
+        def next_sql_num
+            #gets the next sql variable number
+            sql_hash.keys.length + 1
+        end
+
+        def get_array_for_exec_query
+            sql_hash.values
+        end
+
+        #thank god for some stack overflow people are pretty awesome https://stackoverflow.com/questions/64894375/executing-a-raw-sql-query-in-rails-with-an-array-parameter-against-postgresql
+		#BigIntArray = ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(ActiveModel::Type::BigInteger.new).freeze
+		#IntegerArray = ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(ActiveModel::Type::Integer.new).freeze
+
+		#https://api.rubyonrails.org/files/activemodel/lib/active_model/type_rb.html
+		# active_model/type/helpers
+		# active_model/type/value
+		# active_model/type/big_integer
+		# active_model/type/binary
+		# active_model/type/boolean
+		# active_model/type/date
+		# active_model/type/date_time
+		# active_model/type/decimal
+		# active_model/type/float
+		# active_model/type/immutable_string
+		# active_model/type/integer
+		# active_model/type/string
+		# active_model/type/time
+		# active_model
+
+		DB_TYPE_MAPS = {
+			String => ActiveModel::Type::String,
+			Symbol => ActiveModel::Type::String,
+			Integer => ActiveModel::Type::BigInteger,
+			BigDecimal => ActiveRecord::Type::Decimal,
+			TrueClass => ActiveModel::Type::Boolean,
+			FalseClass => ActiveModel::Type::Boolean,
+			Date => ActiveModel::Type::Date,
+			DateTime => ActiveModel::Type::DateTime,
+			Time => ActiveModel::Type::Time,
+			Float => ActiveModel::Type::Float,
+			Array =>  Proc.new{ |first_el_class| ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(DB_TYPE_MAPS[first_el_class].new) }
+		}
+
+        def convert_to_query_attribute(name, v)
+            #yes its dumb I know dont look at me look at rails
+
+            # https://stackoverflow.com/questions/40407700/rails-exec-query-bindings-ignored
+            # binds = [ ActiveRecord::Relation::QueryAttribute.new(
+            # 	"id", 6, ActiveRecord::Type::Integer.new
+            # )]
+            # ApplicationRecord.connection.exec_query(
+            # 	'SELECT * FROM users WHERE id = $1', 'sql', binds
+            # )
+
+            return v if v.kind_of? ActiveRecord::Relation::QueryAttribute	#so users can have fine-grained control if they are trying to do something
+                #that we didn't handle properly.
+
+            type = DB_TYPE_MAPS[v.class]
+            if type.nil?
+                raise StandardError.new("#{name} (of value: #{v}, class: #{v.class}) unsupported class for ApplicationRecord#headache_sql")
+            elsif type.class == Proc
+                a = v[0]
+                # if a.nil?
+                # 	a = Integer
+                # elsif a.class == Array
+                a = a.nil? ? Integer : a.class
+                type = type.call(a)
+            else
+                type = type.new
+            end	
+            ActiveRecord::Relation::QueryAttribute.new( name, v, type )
+        end
+    end
 
 	class MultiRowExpression
 		#this class is meant to be used in congunction with headache_sql method
@@ -42,26 +129,23 @@ module DynamicRecordsMeritfront
 			#assuming we are putting in an array of arrays.
 			self.val = val
 		end
-		def for_query(x = 0, unique_value_hash:)
+		def for_query(key, var_track)
 			#accepts x = current number of variables previously processed
 			#returns ["sql string with $# location information", variables themselves in order, new x]
-			db_val = val.map{|attribute_array| "(#{
-				attribute_index = 0
+			x = -1
+            db_val = val.map{|attribute_array| "(#{
 				attribute_array.map{|attribute|
-					prexist_num = unique_value_hash[attribute]
-					if prexist_num
-						attribute_array[attribute_index] = nil
-						ret = "$#{prexist_num}"
-					else
-						unique_value_hash[attribute] = x
-						ret = "$#{x}"
-						x += 1
-					end
-					attribute_index += 1
-					next ret
+                    if attribute.kind_of? Symbol
+                        #allow pointers to other more explicit variables through symbols
+                        x = var_track.add_key_value(attribute, nil)
+                    else
+                        k = "#{key}_#{var_track.next_sql_num.to_s}"
+                        x = var_track.add_key_value(k, attribute)
+                    end
+                    next "$" + x.to_s
 				}.join(",")
 			})"}.join(",")
-			return db_val, val.flatten.select{|a| not a.nil?}, x
+			return db_val
 		end
 	end
 
@@ -230,70 +314,6 @@ module DynamicRecordsMeritfront
 			end
 		end
 
-		#thank god for some stack overflow people are pretty awesome https://stackoverflow.com/questions/64894375/executing-a-raw-sql-query-in-rails-with-an-array-parameter-against-postgresql
-		#BigIntArray = ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(ActiveModel::Type::BigInteger.new).freeze
-		#IntegerArray = ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(ActiveModel::Type::Integer.new).freeze
-
-		#https://api.rubyonrails.org/files/activemodel/lib/active_model/type_rb.html
-		# active_model/type/helpers
-		# active_model/type/value
-		# active_model/type/big_integer
-		# active_model/type/binary
-		# active_model/type/boolean
-		# active_model/type/date
-		# active_model/type/date_time
-		# active_model/type/decimal
-		# active_model/type/float
-		# active_model/type/immutable_string
-		# active_model/type/integer
-		# active_model/type/string
-		# active_model/type/time
-		# active_model
-
-		DB_TYPE_MAPS = {
-			String => ActiveModel::Type::String,
-			Symbol => ActiveModel::Type::String,
-			Integer => ActiveModel::Type::BigInteger,
-			BigDecimal => ActiveRecord::Type::Decimal,
-			TrueClass => ActiveModel::Type::Boolean,
-			FalseClass => ActiveModel::Type::Boolean,
-			Date => ActiveModel::Type::Date,
-			DateTime => ActiveModel::Type::DateTime,
-			Time => ActiveModel::Type::Time,
-			Float => ActiveModel::Type::Float,
-			Array =>  Proc.new{ |first_el_class| ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Array.new(DB_TYPE_MAPS[first_el_class].new) }
-		}
-
-		def convert_to_query_attribute(name, v)
-			#yes its dumb I know dont look at me look at rails
-	
-			# https://stackoverflow.com/questions/40407700/rails-exec-query-bindings-ignored
-			# binds = [ ActiveRecord::Relation::QueryAttribute.new(
-			# 	"id", 6, ActiveRecord::Type::Integer.new
-			# )]
-			# ApplicationRecord.connection.exec_query(
-			# 	'SELECT * FROM users WHERE id = $1', 'sql', binds
-			# )
-	
-			return v if v.kind_of? ActiveRecord::Relation::QueryAttribute	#so users can have fine-grained control if they are trying to do something
-				#that we didn't handle properly.
-	
-			type = DB_TYPE_MAPS[v.class]
-			if type.nil?
-				raise StandardError.new("#{v}'s class #{v.class} unsupported type right now for ApplicationRecord#headache_sql")
-			elsif type.class == Proc
-				a = v[0]
-				# if a.nil?
-				# 	a = Integer
-				# elsif a.class == Array
-				a = a.nil? ? Integer : a.class
-				type = type.call(a)
-			else
-				type = type.new
-			end
-			
-			ActiveRecord::Relation::QueryAttribute.new( name, v, type )
-		end
 		#allows us to preload on a list and not a active record relation. So basically from the output of headache_sql
 		def dynamic_preload(records, associations)
 			ActiveRecord::Associations::Preloader.new(records: records, associations: associations).call
@@ -301,122 +321,137 @@ module DynamicRecordsMeritfront
 
 		alias headache_preload dynamic_preload
 		
-		def dynamic_sql(name, sql, opts = { }) #see below for opts
-			# - instantiate_class - returns User, Post, etc objects instead of straight sql output.
-			#		I prefer doing the alterantive
-			#			User.headache_class(...)
-			#		which is also supported
-			# - prepare sets whether the db will preprocess the strategy for lookup (defaults true) (I dont think turning this off works...)
-			# - name_modifiers allows one to change the preprocess associated name, useful in cases of dynamic sql.
-			# - multi_query allows more than one query (you can seperate an insert and an update with ';' I dont know how else to say it.)
-			#		this disables other options (except name_modifiers). Not sure how it effects prepared statements. Its a fairly useless
-			#		command as you can do multiple queries anyway with 'WITH' statements and also gain the other options.
-			# - async does what it says but I haven't used it yet so. Probabably doesn't work
-			# - raw switches between using a Hash or a ActiveRecord::Response object when used on a abstract class
-
-			#
-			# Any other option is assumed to be a sql argument (see other examples in code base)
-
-			#grab options from the opts hash
-			instantiate_class = opts.delete(:instantiate_class)
-			name_modifiers = opts.delete(:name_modifiers)
-			raw = opts.delete(:raw)
-			raw = DYNAMIC_SQL_RAW if raw.nil?
-			name_modifiers ||= []
-			prepare = opts.delete(:prepare) != false
-			multi_query = opts.delete(:multi_query) == true
-			async = opts.delete(:async) == true
-			params = opts
-
-			#unique value hash cuts down on the number of repeated arguments like in an update or insert statement
-			#by checking if there is an equal existing argument and then using that argument number instead.
-			#If this functionality is used at a lower level we should probably remove this.
-			unique_value_hash = {}
-
-			#allows dynamic sql prepared statements.
-			for mod in name_modifiers
-				name << "_#{mod.to_s}" unless mod.nil?
-			end
-
-			unless multi_query
-				#https://stackoverflow.com/questions/49947990/can-i-execute-a-raw-sql-query-leverage-prepared-statements-and-not-use-activer/67442353#67442353
-				#change the keys to $1, $2 etc. this step is needed for ex. {id: 1, id_user: 2}.
-				#doing the longer ones first prevents id replacing :id_user -> 1_user
-				keys = params.keys.sort{|a,b| b.to_s.length <=> a.to_s.length}
-				sql_vals = []
-				x = 1
-				for key in keys
-					#replace the key with $1, $2 etc
-					v = params[key]
-
-					#this is where we guess what it is
-					looks_like_multi_attribute_array = ((v.class == Array) and (not v.first.nil?) and (v.first.class == Array))
-
-					if v.class == MultiRowExpression or looks_like_multi_attribute_array
-					#it looks like or is a multi-row expression (like those in an insert statement)
-						v = MultiRowExpression.new(v) if looks_like_multi_attribute_array
-						#process into usable information
-						sql_for_replace, mat_vars, new_x = v.for_query(x, unique_value_hash: unique_value_hash)
-						#replace the key with the sql
-						if sql.gsub!(":#{key}", sql_for_replace) != nil
-						#if successful set the new x number and append variables to our sql variables
-							x = new_x
-							name_num = 0
-							mat_vars.each{|mat_var|
-								name_num += 1
-								sql_vals << convert_to_query_attribute("#{key}_#{name_num}", mat_var)
-							}
-						end
-					else
-						prexist_arg_num = unique_value_hash[v]
-						if prexist_arg_num
-							sql.gsub!(":#{key}", "$#{prexist_arg_num}")
-						else
-							if sql.gsub!(":#{key}", "$#{x}") == nil
-								#nothing changed, param not used, delete it
-								params.delete key
-							else
-								unique_value_hash[v] = x
-								sql_vals << convert_to_query_attribute(key, v)
-								x += 1
-							end
-						end
-					end
-				end
-				ret = ActiveRecord::Base.connection.exec_query sql, name, sql_vals, prepare: prepare, async: async
-			else
-				ret = ActiveRecord::Base.connection.execute sql, name
-			end
-
-			#this returns a PG::Result object, which is pretty basic. To make this into User/Post/etc objects we do
-					#the following
-			if instantiate_class or not self.abstract_class
-				instantiate_class = self if not instantiate_class
-				#no I am not actually this cool see https://stackoverflow.com/questions/30826015/convert-pgresult-to-an-active-record-model
-				ret = zip_ar_result(ret)
-				return ret.map{|r| dynamic_init(instantiate_class, r)}
-				# fields = ret.columns
-				# vals = ret.rows
-				# ret = vals.map { |v|
-				# 	dynamic_init()
-				# 	instantiate_class.instantiate(Hash[fields.zip(v)])
-				# }
-			else
-				if raw
-					return ret
-				else
-					return zip_ar_result(ret)
-				end
-			end
-		end
+        def dynamic_sql(*args) #see below for opts
+        # call like: dynamic_sql(name, sql, option_1: 1, option_2: 2)
+        #   or like: dynamic_sql(sql, {option: 1, option_2: 2})
+        #   or like: dynamic_sql(sql, option: 1, option_2: 2)
+        #	or just: dynamic_sql(sql)
+        #
+        # Options: (options not listed will be sql arguments)
+        # - instantiate_class - returns User, Post, etc objects instead of straight sql output.
+        #		I prefer doing the alterantive
+        #			User.headache_class(...)
+        #		which is also supported
+        # - prepare sets whether the db will preprocess the strategy for lookup (defaults true) (I dont think turning this off works...)
+        # - name_modifiers allows one to change the preprocess associated name, useful in cases of dynamic sql.
+        # - multi_query allows more than one query (you can seperate an insert and an update with ';' I dont know how else to say it.)
+        #		this disables other options (except name_modifiers). Not sure how it effects prepared statements. Its a fairly useless
+        #		command as you can do multiple queries anyway with 'WITH' statements and also gain the other options.
+        # - async does what it says but I haven't used it yet so. Probabably doesn't work
+        # - raw switches between using a Hash or a ActiveRecord::Response object when used on a abstract class
+            args << {} unless args[-1].kind_of? Hash
+            if args.length == 3
+                name, sql, opts = args
+            elsif args.length == 2
+                sql, opts = args
+                #give default name functionality as a pointer to source code location
+                #of the method that called this. Love ruby. Meta up the a$$
+                first_app_stack_trace = caller[0...3].select{|str| not str.include?('dynamic_records_meritfront.rb')}.first
+                shorter_source_loc = first_app_stack_trace.split('/')[-1]
+                name = shorter_source_loc
+            else
+                raise StandardError.new("bad input to DynamicRecordsMeritfront#dynamic_sql method.")
+            end
+
+            #grab options from the opts hash
+            instantiate_class = opts.delete(:instantiate_class)
+            name_modifiers = opts.delete(:name_modifiers)
+            raw = opts.delete(:raw)
+            raw = DYNAMIC_SQL_RAW if raw.nil?
+            name_modifiers ||= []
+            prepare = opts.delete(:prepare) != false
+            multi_query = opts.delete(:multi_query) == true
+            async = opts.delete(:async) == true
+            params = opts
+
+                #unique value hash cuts down on the number of repeated arguments like in an update or insert statement
+                #by checking if there is an equal existing argument and then using that argument number instead.
+                #If this functionality is used at a lower level we should probably remove this.
+                #________________________________
+                #got this error: ActiveRecord::StatementInvalid (PG::ProtocolViolation: ERROR:  bind message supplies 3 parameters, but prepared statement "a27" requires 4)
+                #this error tells me two things
+                #   1. the name of a sql statement actually has no effect on prepared statements (whoops).
+                #       This means we should accept queries with no name.
+                #   2. Need to get rid of the unique variable name functionality which uniques all the variables
+                #       to decrease the amount sent to database
+
+            #name_modifiers are super unnecessary now I realize the given name is not actually related
+            #to prepped statements. But will keep it as it is backwards compatitable and sorta useful maybe.
+            for mod in name_modifiers
+                name << "_#{mod.to_s}" unless mod.nil?
+            end
+            begin
+                var_track = DynamicSqlVariables.new(params)
+                unless multi_query
+                    #https://stackoverflow.com/questions/49947990/can-i-execute-a-raw-sql-query-leverage-prepared-statements-and-not-use-activer/67442353#67442353
+                    #change the keys to $1, $2 etc. this step is needed for ex. {id: 1, id_user: 2}.
+                    #doing the longer ones first prevents id replacing :id_user -> $1_user
+                    keys = params.keys.sort{|a,b| b.to_s.length <=> a.to_s.length}
+
+                    for key in keys
+                        #replace MultiRowExpressions
+                        v = params[key]
+                        #check if it looks like one
+                        looks_like_multi_attribute_array = ((v.class == Array) and (not v.first.nil?) and (v.first.class == Array))
+                        if v.class == MultiRowExpression or looks_like_multi_attribute_array
+                        #we need to substitute with the correct sql now.
+                            v = MultiRowExpression.new(v) if looks_like_multi_attribute_array #standardize
+                            #process into appropriate sql while keeping track of variables
+                            sql_for_replace = v.for_query(key, var_track)
+                            #replace the key with the sql
+                            sql.gsub!(":#{key}", sql_for_replace)
+                        else
+                            x = var_track.next_sql_num
+                            if sql.gsub!(":#{key}", "$#{x}")
+                                var_track.add_key_value(key, v)
+                            end
+                        end
+                    end
+                    sql_vals = var_track.get_array_for_exec_query
+                    ret = ActiveRecord::Base.connection.exec_query sql, name, sql_vals, prepare: prepare, async: async
+                else
+                    ret = ActiveRecord::Base.connection.execute sql, name
+                end
+            rescue Exception => e
+                #its ok if some of these are empty, just dont want the error
+                name ||= ''
+                sql ||= ''
+                sql_vals ||= ''
+                prepare ||= ''
+                async ||= ''
+                Rails.logger.error(%Q{
+    DynamicRecords#dynamic_sql debug info.
+    name: #{name.to_s}
+    sql: #{sql.to_s}
+    sql_vals: #{sql_vals.to_s}
+    prepare: #{prepare.to_s}
+    async: #{async.to_s}
+    })
+                raise e
+            end
+
+            #this returns a PG::Result object, which is pretty basic. To make this into User/Post/etc objects we do
+                    #the following
+            if instantiate_class or not self.abstract_class
+                instantiate_class = self if not instantiate_class
+                #no I am not actually this cool see https://stackoverflow.com/questions/30826015/convert-pgresult-to-an-active-record-model
+                ret = ret.to_a
+                return ret.map{|r| dynamic_init(instantiate_class, r)}
+                # fields = ret.columns
+                # vals = ret.rows
+                # ret = vals.map { |v|
+                # 	dynamic_init()
+                # 	instantiate_class.instantiate(Hash[fields.zip(v)])
+                # }
+            else
+                if raw
+                    return ret
+                else
+                    return ret.to_a
+                end
+            end
+        end
 		alias headache_sql dynamic_sql
-		
-		def instaload(sql, table_name: nil, relied_on: false)
-			table_name ||= "_" + self.to_s.underscore.downcase.pluralize
-			klass = self.to_s
-			sql = "\t" + sql.strip
-			return {table_name: table_name, klass: klass, sql: sql, relied_on: relied_on}
-		end
 
 		def _dynamic_instaload_handle_with_statements(with_statements)
 			%Q{WITH #{
@@ -427,7 +462,9 @@ module DynamicRecordsMeritfront
 		end
 
 		def _dynamic_instaload_union(insta_array)
-			insta_array.map{|insta|
+			insta_array.select{|insta|
+                not insta[:dont_return]
+            }.map{|insta|
 				start = "SELECT row_to_json(#{insta[:table_name]}.*) AS row, '#{insta[:klass]}' AS _klass, '#{insta[:table_name]}' AS _table_name FROM "
 				if insta[:relied_on]
 					ending = "#{insta[:table_name]}\n"
@@ -439,13 +476,31 @@ module DynamicRecordsMeritfront
 			#{ other_statements.map{|os| "SELECT row_to_json(#{os[:table_name]}.*) AS row, '#{os[:klass]}' AS _klass FROM (\n#{os[:sql]}\n)) AS #{os[:table_name]}\n" }.join(' UNION ALL ')}
 		end
 
-		def dynamic_instaload_sql(name, insta_array, opts = { })
+        def instaload(sql, table_name: nil, relied_on: false, dont_return: false)
+            table_name ||= "_" + self.to_s.underscore.downcase.pluralize
+            klass = self.to_s
+            sql = "\t" + sql.strip
+            return {table_name: table_name, klass: klass, sql: sql, relied_on: relied_on, dont_return: dont_return}
+        end
+
+		def instaload_sql(*args) #name, insta_array, opts = { })
+            args << {} unless args[-1].kind_of? Hash
+            if args.length == 3
+                name, insta_array, opts = args
+            elsif args.length == 2
+                insta_array, opts = args
+                name = nil
+            else
+                raise StandardError.new("bad input to DynamicRecordsMeritfront#instaload_sql method.")
+            end
+
 			with_statements = insta_array.select{|a| a[:relied_on]}
 			sql = %Q{
 #{ _dynamic_instaload_handle_with_statements(with_statements) if with_statements.any? }
 #{ _dynamic_instaload_union(insta_array)}
 }
-			ret_hash = insta_array.map{|ar| [ar[:table_name].to_s, []]}.to_h
+            returned_arrays = insta_array.select{|ar| not ar[:dont_return]}
+			ret_hash = returned_arrays.map{|ar| [ar[:table_name].to_s, []]}.to_h
 			opts[:raw] = true
 			ApplicationRecord.headache_sql(name, sql, opts).rows.each{|row|
 				#need to pre-parsed as it has a non-normal output.
@@ -454,13 +509,94 @@ module DynamicRecordsMeritfront
 				json = row[0]
 				parsed = JSON.parse(json)
 
-				ret_hash[table_name ].push dynamic_init(klass, parsed)
+				ret_hash[table_name].push dynamic_init(klass, parsed)
 			}
 			return ret_hash
 		end
-		alias swiss_instaload_sql dynamic_instaload_sql
+		alias swiss_instaload_sql instaload_sql
+        alias dynamic_instaload_sql instaload_sql
+
+        def test_drmf(model_with_an_id_column_and_timestamps)
+			m = model_with_an_id_column_and_timestamps
+			ar = m.superclass
+			mtname = m.table_name
+			ApplicationRecord.transaction do
+				puts 'test recieving columns not normally in the record.'
+				rec = m.dynamic_sql(%Q{
+					SELECT id, 5 AS random_column from #{mtname} LIMIT 10
+				}).first
+				raise StandardError.new('no id') unless rec.id
+				raise StandardError.new('no dynamic column') unless rec.random_column
+				puts 'pass 1'
+				
+				puts 'test raw off with a custom name'
+				recs = ar.dynamic_sql('test_2', %Q{
+					SELECT id, 5 AS random_column from #{mtname} LIMIT 10
+				}, raw: false)
+				raise StandardError.new('not array of hashes') unless recs.first.class == Hash and recs.class == Array
+				rec = recs.first
+				raise StandardError.new('no id [raw off]') unless rec['id']
+				raise StandardError.new('no dynamic column [raw off]') unless rec['random_column']
+				puts 'pass 2'
+
+				puts 'test raw on'
+				recs = ar.dynamic_sql('test_3', %Q{
+					SELECT id, 5 AS random_column from #{mtname} LIMIT 10
+				}, raw: true)
+				raise StandardError.new('not raw') unless recs.class == ActiveRecord::Result
+				rec = recs.first
+				raise StandardError.new('no id [raw]') unless rec['id']
+				raise StandardError.new('no dynamic column [raw]') unless rec['random_column']
+				puts 'pass 3'
+
+				puts 'test when some of the variables are diffrent then the same (#see version 3.0.1 notes)'
+				x = Proc.new { |a, b|
+					recs = ar.dynamic_sql('test_4', %Q{
+						SELECT id, 5 AS random_column from #{mtname} WHERE id > :a LIMIT :b
+					}, a: a, b: b)
+				}
+				x.call(1, 2)
+				x.call(1, 1)
+				puts 'pass 4'
+
+				puts 'test MultiAttributeArrays, including symbols and duplicate values.'
+				time = DateTime.now
+				ids = m.limit(5).pluck(:id)
+				values = ids.map{|id|
+					[id, :time, time]
+				}
+				ar.dynamic_sql(%Q{
+					INSERT INTO #{mtname} (id, created_at, updated_at)
+					VALUES :values
+					ON CONFLICT (id) DO NOTHING
+				}, values: values, time: time)
+				puts 'pass 5'
+				
+				puts 'test arrays'
+				recs = ar.dynamic_sql(%Q{
+					SELECT id from #{mtname} where id = ANY(:idz)
+				}, idz: ids, raw: false)
+				puts recs
+				raise StandardError.new('wrong length') if recs.length != 5
+				puts 'pass 6'
 
-		
+				
+				puts 'test instaload_sql'
+				out = ar.instaload_sql([
+					ar.instaload("SELECT id FROM users", relied_on: true, dont_return: true, table_name: "users_2"),
+					ar.instaload("SELECT id FROM users_2 WHERE id % 2 != 0 LIMIT :limit", table_name: 'a'),
+					m.instaload("SELECT id FROM users_2 WHERE id % 2 != 1 LIMIT :limit", table_name: 'b')
+				], limit: 2)
+				puts out
+				raise StandardError.new('Bad return') if out["users_2"]
+				raise StandardError.new('Bad return') unless out["a"]
+				raise StandardError.new('Bad return') unless out["b"]
+				puts 'pass 7'
+
+				raise ActiveRecord::Rollback
+				#ApplicationRecord.dynamic_sql("SELECT * FROM")
+			end
+		end
 
 		def dynamic_attach(instaload_sql_output, base_name, attach_name, base_on: nil, attach_on: nil, one_to_one: false)
 			base_arr = instaload_sql_output[base_name]
@@ -563,11 +699,7 @@ module DynamicRecordsMeritfront
 		alias swiss_attach dynamic_attach
 
 		def zip_ar_result(x)
-			fields = x.columns
-			vals = x.rows
-			vals.map { |v|
-				Hash[fields.zip(v)]
-			}
+			x.to_a
 		end
 
 		def dynamic_init(klass, input)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dynamic-records-meritfront
 version: !ruby/object:Gem::Version
-  version: 2.0.21
+  version: 3.0.1
 platform: ruby
 authors:
 - Luke Clancy
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-10-26 00:00:00.000000000 Z
+date: 2022-12-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: hashid-rails