sequel 4.49.0 → 5.0.0

data/doc/security.rdoc

@@ -16,14 +16,11 @@ as it never calls eval on a string that is derived from user input.
 However, some Sequel methods used for creating methods via metaprogramming
 could conceivably be abused to do so:
 
-* Sequel::Schema::CreateTableGenerator.add_type_method
-* Sequel::Dataset.def_mutation_method
 * Sequel::Dataset.def_sql_method
-* Sequel::Model::Plugins.def_dataset_methods
-* Sequel.def_adapter_method (private)
-* Sequel::SQL::Expression.to_s_method (private)
-* Sequel::Plugins::HookClassMethods::ClassMethods#add_hook_type
+* Sequel::JDBC.load_driver
 * Sequel::Plugins.def_dataset_methods
+* Sequel::Dataset.prepared_statements_module (private)
+* Sequel::SQL::Expression.to_s_method (private)
 
 As long as you don't call those with user input, you should not be
 vulnerable to code execution.
@@ -48,6 +45,9 @@ There are basically two kinds of possible SQL injections in Sequel:
 Some Sequel methods are designed to execute raw SQL strings, including:
 
 * Sequel::Database#execute
+* Sequel::Database#execute_ddl
+* Sequel::Database#execute_dui
+* Sequel::Database#execute_insert
 * Sequel::Database#run
 * Sequel::Database#<<
 * Sequel::Dataset#fetch_rows
@@ -61,9 +61,12 @@ Some Sequel methods are designed to execute raw SQL strings, including:
 
 Here are some examples of use:
 
+  DB.execute 'SQL'
+  DB.execute_ddl 'SQL'
+  DB.execute_dui 'SQL'
+  DB.execute_insert 'SQL'
   DB.run 'SQL'
   DB << 'SQL'
-  DB.execute 'SQL'
   DB.fetch_rows('SQL'){|row| }
   DB.dataset.with_sql_all('SQL')
   DB.dataset.with_sql_delete('SQL')
@@ -102,19 +105,16 @@ With these methods you should use placeholders, in which case Sequel automatical
 
 Sequel generally treats ruby strings as SQL strings (escaping them correctly), and
 not as raw SQL.  However, you can convert a ruby string to a literal string, and
-Sequel will then treat it as raw SQL.  This is typically done through String#lit
-if the {core_extensions}[rdoc-ref:doc/core_extensions.rdoc] are in use,
-or Sequel.lit[rdoc-ref:Sequel::SQL::Builders#lit] if they are not in use.
+Sequel will then treat it as raw SQL.  This is typically done through
+Sequel.lit[rdoc-ref:Sequel::SQL::Builders#lit].
 
-  'a'.lit
   Sequel.lit('a')
 
-Using String#lit or Sequel.lit[rdoc-ref:Sequel::SQL::Builders#lit] to turn a ruby string into a literal string results
+Using Sequel.lit[rdoc-ref:Sequel::SQL::Builders#lit] to turn a ruby string into a literal string results
 in SQL injection if the string is derived from user input.  With both of these
 methods, the strings can contain placeholders, which you can use to safely include
 user input inside a literal string:
 
-  'a = ?'.lit(params[:user_id].to_s)
   Sequel.lit('a = ?', params[:user_id].to_s)
 
 Even though they have similar names, note that Sequel::Database#literal operates very differently from
@@ -132,91 +132,107 @@ a ruby string as raw SQL.  For example:
 
 ==== SQL Filter Fragments
 
-The most common way to use raw SQL with Sequel is in filters:
+Starting in Sequel 5, Sequel does not automatically convert plain strings to
+literal strings in typical code.  Instead, you can use Sequel.lit to
+create literal strings:
 
-  DB[:table].where("name > 'M'")
+  Sequel.lit("name > 'A'")
 
-If a filter method is passed a string as the first argument, it treats the rest of
-the arguments (if any) as placeholders to the string.  So you should never do:
+To safely include user input as part of an SQL filter fragment, use Sequel.lit
+with placeholders:
 
-  DB[:table].where("name > #{params[:id].to_s}") # SQL Injection!
+  DB[:table].where(Sequel.lit("name > ?", params[:id].to_s)) # Safe
 
-Instead, you should use a placeholder:
+Be careful to never call Sequel.lit where the first argument is derived from
+user input.
 
-  DB[:table].where("name > ?", params[:id].to_s) # Safe
- 
-Note that for that type of query, Sequel generally encourages the following form:
+There are a few uncommon cases where Sequel will still convert
+plain strings to literal strings.
 
-  DB[:table].where{|o| o.name > params[:id].to_s} # Safe
+==== SQL Fragment passed to Dataset#lock_style and Model#lock!
 
-Sequel's DSL supports a wide variety of SQL concepts, so it's possible to
-code most applications without ever using raw SQL.
+The Sequel::Dataset#lock_style and Sequel::Model#lock! methods also treat
+an input string as SQL code. This method should not be called with user input.
 
-A large number of dataset methods ultimately pass down their arguments to a filter
-method, even some you may not expect, so you should be careful.  At least the
-following methods pass their arguments to a filter method:
+  DB[:table].lock_style(params[:id]) # SQL injection!
+  Album.first.lock!(params[:id]) # SQL injection!
+
+==== SQL Type Names
+
+In general, most places where Sequel needs to use an SQL type that should
+be specified by the user, it allows you to use a ruby string, and that
+string is used verbatim as the SQL type.  You should not use user input
+for type strings.
+
+  DB[:table].select(Sequel.cast(:a, params[:id])) # SQL injection!
+
+==== SQL Function Names
+
+In most cases, Sequel does not quote SQL function names.  You should not use
+user input for function names.
+
+  DB[:table].select(Sequel.function(params[:id])) # SQL injection!
+
+==== auto_literal_strings extension
+
+If the auto_literal_strings extension is used for backwards compatibility,
+then Sequel will treat plain strings as literal strings if they are used
+as the first argument to a filtering method.  This can lead to SQL
+injection:
+
+  DB[:table].where("name > #{params[:id].to_s}")
+  # SQL injection when using auto_literal_strings extension
+
+If you are using the auto_literal_strings extension, you need to be very careful,
+as the following methods will treat a plain string given as the first arguments
+as a literal strings:
 
 * Sequel::Dataset#where
 * Sequel::Dataset#having
 * Sequel::Dataset#filter
 * Sequel::Dataset#exclude
-* Sequel::Dataset#exclude_where
 * Sequel::Dataset#exclude_having
-* Sequel::Dataset#and
 * Sequel::Dataset#or
 * Sequel::Dataset#first
 * Sequel::Dataset#last
 * Sequel::Dataset#[]
 
-The Model.find[rdoc-ref:Sequel::Model::ClassMethods#find] and Model.find_or_create[rdoc-ref:Sequel::Model::ClassMethods#find_or_create]
-class methods also call down to the filter methods.
+Even stuff that looks like it may be safe isn't:
 
-The no_auto_string_literals extension can be used to remove the default support 
-for plain strings as literal strings in filter methods.
+  DB[:table].first(params[:num_rows])
+  # SQL injection when using auto_literal_strings extension
 
-==== SQL Fragment passed to Dataset#update
+The Model.find[rdoc-ref:Sequel::Model::ClassMethods#find] and
+Model.find_or_create[rdoc-ref:Sequel::Model::ClassMethods#find_or_create]
+class methods also will treat string arguments as literal strings if the
+auto_literal_strings extension is used:
 
-Similar to the filter methods, Sequel::Dataset#update also treats a
-string argument as raw SQL:
+  Album.find(params[:id])
+  # SQL injection when using auto_literal_strings extension
+
+Similar to the filter methods, the auto_literal_strings extension
+also makes Sequel::Dataset#update treats a string argument as raw SQL:
 
   DB[:table].update("column = 1")
 
 So you should not do:
 
-  DB[:table].update("column = #{params[:value].to_s}") # SQL Injection!
+  DB[:table].update(params[:changes])
+  # SQL injection when using auto_literal_strings extension
 
-Instead, you should do:
+or:
 
-  DB[:table].update(:column => params[:value].to_s) # Safe
+  DB[:table].update("column = #{params[:value].to_s}")
+  # SQL injection when using auto_literal_strings extension
 
-The no_auto_string_literals extension can also be used to remove the default support
-for plain strings as literal strings in update methods.
-
-==== SQL Fragment passed to Dataset#lock_style and Model#lock!
-
-The Sequel::Dataset#lock_style and Sequel::Model#lock! methods also treat
-an input string as SQL code. This method should not be called with user input.
-
-==== SQL Fragment passed to Virtual Row #` method
-
-Virtual row blocks currently support a #` method for using literal SQL:
-
-  DB[:table].where{a > `some SQL`}
-
-This method should not be called with user input.
-
-==== SQL Type Names
-
-In general, most places where Sequel needs to use an SQL type that should
-be specified by the user, it allows you to use a ruby string, and that
-string is used verbatim as the SQL type.  You should not use user input
-for type strings.
-
-==== SQL Function Names
+Instead, you should do:
 
-In most cases, Sequel does not quote SQL function names.  You should not use
-user input for function names.
+  DB[:table].update(:column => params[:value].to_s) # Safe
 
+Because using the auto_literal_strings extension makes SQL injection
+so much eaiser, it is recommended to not use it, and instead
+use Sequel.lit with placeholders.
+ 
 === SQL Identifier Injections
 
 Usually, Sequel treats ruby symbols as SQL identifiers, and ruby
@@ -236,7 +252,7 @@ the Sequel::Dataset#insert and Sequel::Dataset#update methods:
   DB[:t].insert('b'=>1) # INSERT INTO "t" ("b") VALUES (1)
 
 Note how the identifier is still quoted in these cases.  Sequel quotes identifiers by default
-on most databases.  However, it does not quote identifiers by default on DB2 and Informix.
+on most databases.  However, it does not quote identifiers by default on DB2.
 On those databases using an identifier derived from user input can lead to SQL injection.
 Similarly, if you turn off identifier quoting manually on other databases, you open yourself
 up to SQL injection if you use identifiers derived from user input.
@@ -262,6 +278,10 @@ uses symbols as identifiers.  However, if you are creating symbols from user inp
 you at least have a denial of service vulnerability in ruby <2.2, and possibly a
 more serious vulnerability.
 
+Note that many Database schema modification methods (e.g. create_table, add_column)
+also allow for SQL identifier injections, and possibly also SQL code injections.
+These methods should never be called with user input.
+
 == Denial of Service
 
 Sequel converts some strings to symbols.  Because symbols in ruby <2.2 are not
@@ -293,7 +313,7 @@ if you allow the user to control the alias name:
 
   DB[:table].select(:column.as(params[:alias]))
 
-Then you have a denial of service vulnerability.  In general, such a vulnerability
+Then you can have a denial of service vulnerability.  In general, such a vulnerability
 is unlikely, because you are probably indexing into the returned hash(es) by name,
 and if an alias was used and you didn't expect it, your application wouldn't work.
 
@@ -353,28 +373,17 @@ These two methods iterate over the second argument (+:name+ and +:copies_sold+ i
 this example) instead of iterating over the entries in the first argument
 (<tt>params[:album]</tt> in this example).
 
-In addition to these two methods, you can also use 
-Model#set_only[rdoc-ref:Sequel::Model::InstanceMethods#set_only] or
-Model#update_only[rdoc-ref:Sequel::Model::InstanceMethods#update_only],
-which are similar but iterate over the entries in the first argument, checking
-the second argument to see if setting the entries is allowed.
-
-  album.set_only(params[:album], [:name, :copies_sold])
-  album.update_only(params[:album], [:name, :copies_sold])
-
-If you expect all entries in the second argument to be present in the first
-argument, use +set_fields+ or +update_fields+.  If you are not sure if all
-arguments in the second argument will be present in the first argument, but
-do not want to allow setting any column other than the ones listed in the
-second argument, use +set_only+ or +update_only+.
-
-You can override the columns to allow by default during mass assignment via
-the Model.set_allowed_columns[rdoc-ref:Sequel::Model::ClassMethods#set_allowed_columns] class method.  This is a good
-practice, though being explicit on a per-call basis is still recommended:
+If you want to override the columns that Model#set[rdoc-ref:Sequel::Model::InstanceMethods#set]
+allows by default during mass assignment, you can use the whitelist_security plugin, then call
+the set_allowed_columns class method.
 
+  Album.plugin :whitelist_security
   Album.set_allowed_columns(:name, :copies_sold)
   Album.create(params[:album]) # Only name and copies_sold set
 
+Being explicit on a per-call basis using the set_fields and update_fields methods is recommended
+instead of using the whitelist_security plugin and setting a global whitelist.
+
 For more details on the mass assignment methods, see the {Mass Assignment Guide}[rdoc-ref:doc/mass_assignment.rdoc].
 
 == General Parameter Handling
@@ -386,7 +395,7 @@ their type.  For example:
   Album.where(:id=>params[:id])
 
 is probably a bad idea.  Assuming you are using a web framework, <tt>params[:id]</tt> could
-be a string, an array, a hash, or nil.
+be a string, an array, a hash, nil, or potentially something else.
 
 Assuming that +id+ is an integer field, you probably want to do:
 
@@ -400,7 +409,7 @@ a string:
 If you are trying to use an IN clause with a list of id values based on input provided
 on a web form:
 
-  Album.where(:id=>params[:ids].to_a.map{|i| i.to_i})
+  Album.where(:id=>params[:ids].to_a.map(&:to_i))
 
 Basically, be as explicit as possible. While there aren't any known security issues
 in Sequel when you do:

data/doc/sharding.rdoc

@@ -13,7 +13,7 @@ option.  Using the :servers database option makes Sequel use a connection pool
 class that supports sharding, and the minimum required to enable sharding
 support is to use the empty hash:
 
-  DB=Sequel.connect('postgres://master_server/database', :servers=>{})
+  DB=Sequel.connect('postgres://master_server/database', servers: {})
 
 In most cases, you are probably not going to want to use an empty hash.  Keys in the server hash are
 not restricted to type, but the general recommendation is to use a symbol
@@ -35,8 +35,8 @@ tables you are accessing, unless you really know what you are doing.
 To use a single, read-only slave that handles SELECT queries, the following
 is the simplest configuration:
 
-  DB=Sequel.connect('postgres://master_server/database', \
-    :servers=>{:read_only=>{:host=>'slave_server'}})
+  DB=Sequel.connect('postgres://master_server/database', 
+    servers: {read_only: {host: 'slave_server'}})
 
 This will use the slave_server for SELECT queries and master_server for
 other queries.
@@ -48,8 +48,8 @@ the symbol name defined in the connect options. For example:
   # Force the SELECT to run on the master
   DB[:users].server(:default).all
 
-  # Force the SELECT to run on the read-only slave
-  DB[:users].server(:read_only).all
+  # Force the DELETE to run on the read-only slave
+  DB[:users].server(:read_only).delete
 
 === Multiple Read-Only Slaves, Single Master
 
@@ -59,13 +59,13 @@ slave_server1, slave_server2, and slave_server3.
   num_read_only = 4
   read_only_host = rand(num_read_only)
   read_only_proc = proc do |db|
-    {:host=>"slave_server#{(read_only_host+=1) % num_read_only}"}
+    {host: "slave_server#{(read_only_host+=1) % num_read_only}"}
   end
-  DB=Sequel.connect('postgres://master_server/database', \
-    :servers=>{:read_only=>read_only_proc})
+  DB=Sequel.connect('postgres://master_server/database',
+    servers: {read_only: read_only_proc})
 
 This will use one of the slave servers for SELECT queries and use the
-master_server for other queries.  It's also possible to pick a random host
+master server for other queries.  It's also possible to pick a random host
 instead of using the round robin approach presented above, but that can result
 in less optimal resource usage.
 
@@ -78,15 +78,15 @@ it shows that the master database is named :default.  So for 4 masters and
   num_read_only = 4
   read_only_host = rand(num_read_only)
   read_only_proc = proc do |db|
-    {:host=>"slave_server#{(read_only_host+=1) % num_read_only}"}
+    {host: "slave_server#{(read_only_host+=1) % num_read_only}"}
   end
   num_default = 4
   default_host = rand(num_default)
   default_proc = proc do |db|
-    {:host=>"master_server#{(default_host+=1) % num_default}"}
+    {host: "master_server#{(default_host+=1) % num_default}"}
   end
-  DB=Sequel.connect('postgres://master_server/database', \
-    :servers=>{:default=>default_proc, :read_only=>read_only_proc})
+  DB=Sequel.connect('postgres://master_server/database',
+    servers: {default: default_proc, read_only: read_only_proc})
   
 == Sharding
 
@@ -100,16 +100,16 @@ of 16 shards).  First, you need to configure the database:
 
   servers = {}
   (('0'..'9').to_a + ('a'..'f').to_a).each do |hex|
-    servers[hex.to_sym] = {:host=>"hash_host_#{hex}"}
+    servers[hex.to_sym] = {host: "hash_host_#{hex}"}
   end
-  DB=Sequel.connect('postgres://hash_host/hashes', :servers=>servers)
+  DB=Sequel.connect('postgres://hash_host/hashes', servers: servers)
   
 This configures 17 servers, the 16 shard servers (/hash_host_[0-9a-f]/), and 1
 default server which will be used if no shard is specified ("hash_host").  If
 you want the default server to be one of the shard servers (e.g. hash_host_a),
 it's easiest to do:
 
-  DB=Sequel.connect('postgres://hash_host_a/hashes', :servers=>servers)
+  DB=Sequel.connect('postgres://hash_host_a/hashes', servers: servers)
 
 That will still set up a second pool of connections for the default server,
 since it considers the default server and shard servers independent.  Note that
@@ -120,7 +120,7 @@ schemas, so you should always have a default server that works.
 
 To set the shard for a given query, you use the Dataset#server method:
 
-  DB[:hashes].server(:a).where(:hash=>/31337/)
+  DB[:hashes].server(:a).where(hash: /31337/)
   
 That will return all matching rows on the hash_host_a shard that have a hash
 column that contains 31337.
@@ -133,7 +133,7 @@ the shard to use.  This is fairly easy using a Sequel::Model:
     dataset_module do
       def plaintext_for_hash(hash)
         raise(ArgumentError, 'Invalid SHA-1 Hash') unless /\A[0-9a-f]{40}\z/.match(hash)
-        server(hash[0...1].to_sym).where(:hash=>hash).get(:plaintext)
+        server(hash[0...1].to_sym).where(hash: hash).get(:plaintext)
       end
     end
   end
@@ -146,24 +146,24 @@ to assume the :default shard.  However, you can specify a
 different shard using the :servers_hash option when connecting
 to the database:
 
-  DB = Sequel.connect('postgres://...', :servers_hash=>Hash.new(:some_shard))
+  DB = Sequel.connect('postgres://...', servers_hash: Hash.new(:some_shard))
 
 You can also use this feature to raise an exception if an
 unconfigured shard is used:
 
-  DB = Sequel.connect('postgres://...', :servers_hash=>Hash.new{raise 'foo'})
+  DB = Sequel.connect('postgres://...', servers_hash: Hash.new{raise 'foo'})
 
 If you specify a :servers_hash option to raise an exception for non configured
 shards you should also explicitly specify a :read_only entry in your :servers option
 for the case where a shard is not specified. In most cases it is sufficient
 to make the :read_only entry the same as the :default shard:
 
-  servers = {:read_only => {}}
+  servers = {read_only: {}}
   (('0'..'9').to_a + ('a'..'f').to_a).each do |hex|
-    servers[hex.to_sym] = {:host=>"hash_host_#{hex}"}
+    servers[hex.to_sym] = {host: "hash_host_#{hex}"}
   end
-  DB=Sequel.connect('postgres://hash_host/hashes', :servers=>servers, 
-    :servers_hash=>Hash.new{raise Exception.new("Invalid Server")}) 
+  DB=Sequel.connect('postgres://hash_host/hashes', servers: servers, 
+    servers_hash: Hash.new{raise "Invalid Server"}) 
 
 === Sharding Plugin
 
@@ -176,7 +176,7 @@ work well with shards.  You just need to remember to set to model to use the plu
     plugin :sharding
   end
 
-  Rainbow.server(:a).first(:id=>1).update(:plaintext=>'VGM')
+  Rainbow.server(:a).first(id: 1).update(plaintext: 'VGM')
 
 If all of your models are sharded, you can set all models to use the plugin via:
 
@@ -186,7 +186,7 @@ If all of your models are sharded, you can set all models to use the plugin via:
 
 By default, you must specify the server/shard you want to use for every dataset/action,
 or Sequel will use the default shard.  If you have a group of queries that should use the
-same shard, it can get a bit redundent to specify the same shard for all of them.
+same shard, it can get a bit redundant to specify the same shard for all of them.
 
 The server_block extension adds a Database#with_server method that scopes all database
 access inside the block to the given shard by default:
@@ -194,7 +194,7 @@ access inside the block to the given shard by default:
   DB.extension :server_block
   DB.with_server(:a) do
     # this SELECT query uses the "a" shard
-    if r = Rainbow.first(:hash=>/31337/)
+    if r = Rainbow.first(hash: /31337/)
       r.count += 1
       # this UPDATE query also uses the "a" shard
       r.save
@@ -209,6 +209,19 @@ you retrieve the models inside the block and save them outside of the block.  If
 need to do that, call the server method explicitly on the dataset used to retrieve the
 model objects.
 
+The with_server method also supports a second argument for the default read_only server
+to use, which can be useful if you are mixing sharding and master/slave servers:
+
+  DB.extension :server_block
+  DB.with_server(:a, :a_read_only) do
+    # this SELECT query uses the "a_read_only" shard
+    if r = Rainbow.first(hash: /31337/)
+      r.count += 1
+      # this UPDATE query also uses the "a" shard
+      r.save
+    end
+  end
+
 === arbitrary_servers Extension
 
 By default, Sequel's sharding support is designed to work with predefined shards.  It ships
@@ -220,13 +233,13 @@ The arbitrary_servers extension allows you to pass a server/shard options hash a
 server to use, and those options will be merged directly into the database's default options:
 
   DB.extension :arbitrary_servers
-  DB[:rainbows].server(:host=>'hash_host_a').all
+  DB[:rainbows].server(host: 'hash_host_a').all
   # or
-  DB[:rainbows].server(:host=>'hash_host_b', :database=>'backup').all
+  DB[:rainbows].server(host: 'hash_host_b', database: 'backup').all
 
 arbitrary_servers is designed to work well in conjunction with the server_block extension:
 
-  DB.with_server(:host=>'hash_host_b', :database=>'backup') do
+  DB.with_server(host: 'hash_host_b', database: 'backup') do
     DB.synchronize do
       # All queries here default to the backup database on hash_host_b
     end