bmg 0.17.1 → 0.17.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84d8703a8a6bf90c7bae5dfec7c5c862023900a25208234a5a5a1c9b5e5cb3c1
-  data.tar.gz: 258c5319ebf5c78dfbd14424aa5736ae716f05948c39379375d73796cc223e78
+  metadata.gz: e67d11656619195bfaa20a9d55fff49d593ccaa3ad558d0876c82534977d4e04
+  data.tar.gz: 7ecd9b926d4289feb0fec3efcafcf7b14c483b70f8c4e47b87e3ba038db64774
 SHA512:
-  metadata.gz: f68bfb8b7d247e2a8d9eb19d7a08f23df91856b693428389ab4f87d91708889c201095eab981c5a06810fe3246a8878767c5f1249b2696734b126e0774ae2f82
-  data.tar.gz: 740971c2005d9aca7c910aca170878671444bb7133af57c2bd415a84548d1ac7828b606398ea3b1012ed8b19eda993ca90cbe6192a35264f45bd93ffd9a78d98
+  metadata.gz: 4d99c96391c73c096e3e981155cbe53e637c909854a0e3fe92bae991eb7da1be7cf78188c87f9fb53cdbb90e5dd0d92b4fec0501f8d1fcf7f1cdb1900828d575
+  data.tar.gz: f8ff33a830852b1e0c5648596ef5f11cae7fc5f7bd817c1ed30a73ce3da81a92bfb42871eecd928a3ce7b717499fb9a975ee8f174a1b013054af506ed056b825

data/README.md

@@ -14,7 +14,7 @@ also much simpler, and make its easier to implement user-defined relations.
 
 ## Example
 
-```
+```ruby
 require 'bmg'
 require 'json'
 
@@ -32,13 +32,14 @@ by_city = suppliers
   .group([:sid, :name, :status], :suppliers_in)
 
 puts JSON.pretty_generate(by_city)
+# [{...},...]
 ```
 
 ## Connecting to a SQL database
 
 Bmg requires `sequel >= 3.0` to connect to SQL databases.
 
-```
+```ruby
 require 'sqlite3'
 require 'bmg'
 require 'bmg/sequel'
@@ -47,16 +48,67 @@ DB = Sequel.connect("sqlite://suppliers-and-parts.db")
 
 suppliers = Bmg.sequel(:suppliers, DB)
 
-puts suppliers
+big_suppliers = suppliers
   .restrict(Predicate.neq(status: 30))
-  .to_sql
 
+puts big_suppliers.to_sql
 # SELECT `t1`.`sid`, `t1`.`name`, `t1`.`status`, `t1`.`city` FROM `suppliers` AS 't1' WHERE (`t1`.`status` != 30)
+
+puts JSON.pretty_generate(big_suppliers)
+# [{...},...]
 ```
 
+## How is this different from similar libraries?
+
+1. The libraries you probably know (Sequel, Arel, SQLAlchemy, Korma, jOOQ,
+   etc.) do not implement a genuine relational algebra: their support for
+   chaining relational operators is limited (yielding errors or wrong SQL
+   queries). Bmg **always** allows chaining operators. If it does not, it's
+   a bug. In other words, the following query is 100% valid:
+
+       relation
+         .restrict(...)    # aka where
+         .union(...)
+         .summarize(...)   # aka group by
+         .restrict(...)
+
+2. Bmg supports in memory relations, json relations, csv relations, SQL
+   relations and so on. It's not tight to SQL generation, and supports
+   queries accross multiple data sources.
+
+3. Bmg makes a best effort to optimize queries, simplifying both generated
+   SQL code (low-level accesses to datasources) and in-memory operations.
+
+4. Bmg supports various *structuring* operators (group, image, autowrap,
+   autosummarize, etc.) and allows building 'non flat' relations.
+
+## How is this different from Alf?
+
+1. Bmg's implementation is much simpler than Alf, and uses no ruby core
+   extention.
+
+2. We are confident using Bmg in production. Systematic inspection of query
+   plans is suggested though. Alf was a bit too experimental to be used on
+   (critical) production systems.
+
+2. Alf exposes a functional syntax, command line tool, restful tools and
+   many more. Bmg is limited to the core algebra, main Relation abstraction
+   and SQL generation.
+
+3. Bmg is less strict regarding conformance to relational theory, and
+   may actually expose non relational features (such as support for null,
+   left_join operator, etc.). Sharp tools hurt, use them with great care.
+
+4. Bmg does not yet implement all operators documented on try-alf.org, even
+   if we plan to eventually support them all.
+
+5. Bmg has a few additional operators that prove very useful on real
+   production use cases: prefix, suffix, autowrap, autosummarize, left_join,
+   rxmatch, etc.
+
 ## Supported operators
 
-```
+```ruby
 r.allbut([:a, :b, ...])                      # remove specified attributes
 r.autowrap(split: '_')                       # structure a flat relation, split: '_' is the default
 r.autosummarize([:a, :b, ...], x: :sum)      # (experimental) usual summarizers supported
@@ -66,6 +118,8 @@ r.group([:a, :b, ...], :x)                   # relation-valued attribute from at
 r.image(right, :x, [:a, :b, ...])            # relation-valued attribute from another relation
 r.join(right, [:a, :b, ...])                 # natural join on a join key
 r.join(right, :a => :x, :b => :y, ...)       # natural join after right reversed renaming
+r.left_join(right, [:a, :b, ...], {...})     # left join with optional default right tuple
+r.left_join(right, {:a => :x, ...}, {...})   # left join after right reversed renaming
 r.matching(right, [:a, :b, ...])             # semi join, aka where exists
 r.matching(right, :a => :x, :b => :y, ...)   # semi join, after right reversed renaming
 r.not_matching(right, [:a, :b, ...])         # inverse semi join, aka where not exists
@@ -78,5 +132,19 @@ r.restrict(a: "foo", b: "bar", ...)          # relational restriction, aka where
 r.rxmatch([:a, :b, ...], /xxx/)              # regex match kind of restriction
 r.summarize([:a, :b, ...], x: :sum)          # relational summarization
 r.suffix(:_foo, but: [:a, ...])              # suffix kind of renaming
+t.transform(:to_s)                           # all-attrs transformation
+t.transform(&:to_s)                          # similar, but Proc-driven
+t.transform(:foo => :upcase, ...)            # specific-attrs tranformation
+t.transform([:to_s, :upcase])                # chain-transformation
 r.union(right)                               # relational union
 ```
+
+## Who is behind Bmg?
+
+Bernard Lambeau (bernard@klaro.cards) is Alf & Bmg main engineer & maintainer.
+
+Enspirit (https://enspirit.be) and Klaro App (https://klaro.cards) are both
+actively using and contributing to the library.
+
+Feel free to contact us for help, ideas and/or contributions. Please use github
+issues and pull requests if possible if code is involved.

data/lib/bmg.rb

@@ -38,11 +38,13 @@ module Bmg
   require_relative 'bmg/operator'
 
   require_relative 'bmg/reader'
+  require_relative 'bmg/writer'
 
   require_relative 'bmg/relation/empty'
   require_relative 'bmg/relation/in_memory'
   require_relative 'bmg/relation/spied'
   require_relative 'bmg/relation/materialized'
+  require_relative 'bmg/relation/proxy'
 
   # Deprecated
   Leaf = Relation::InMemory

data/lib/bmg/algebra.rb

@@ -80,6 +80,19 @@ module Bmg
     end
     protected :_joined_with
 
+    def left_join(right, on = [], default_right_tuple = {})
+      drt = default_right_tuple
+      _left_join self.type.left_join(right.type, on, drt), right, on, drt
+    end
+
+    def _left_join(type, right, on, default_right_tuple)
+      Operator::Join.new(type, self, right, on, {
+        variant: :left,
+        default_right_tuple: default_right_tuple
+      })
+    end
+    protected :_left_join
+
     def matching(right, on = [])
       _matching self.type.matching(right.type, on), right, on
     end
@@ -159,6 +172,16 @@ module Bmg
     end
     protected :_summarize
 
+    def transform(transformation = nil, options = {}, &proc)
+      transformation, options = proc, (transformation || {}) unless proc.nil?
+      _transform(self.type.transform(transformation, options), transformation, options)
+    end
+
+    def _transform(type, transformation, options)
+      Operator::Transform.new(type, self, transformation, options)
+    end
+    protected :_transform
+
     def union(other, options = {})
       return self if other.is_a?(Relation::Empty)
       _union self.type.union(other.type), other, options

data/lib/bmg/algebra/shortcuts.rb

@@ -37,6 +37,12 @@ module Bmg
         self.join(right.rename(renaming), on.keys)
       end
 
+      def left_join(right, on = [], *args)
+        return super unless on.is_a?(Hash)
+        renaming = Hash[on.map{|k,v| [v,k] }]
+        self.left_join(right.rename(renaming), on.keys, *args)
+      end
+
       def matching(right, on = [])
         return super unless on.is_a?(Hash)
         renaming = Hash[on.map{|k,v| [v,k] }]

data/lib/bmg/operator.rb

@@ -46,4 +46,5 @@ require_relative 'operator/rename'
 require_relative 'operator/restrict'
 require_relative 'operator/rxmatch'
 require_relative 'operator/summarize'
+require_relative 'operator/transform'
 require_relative 'operator/union'

data/lib/bmg/operator/join.rb

@@ -8,16 +8,19 @@ module Bmg
     class Join
       include Operator::Binary
 
-      def initialize(type, left, right, on)
+      DEFAULT_OPTIONS = {}
+
+      def initialize(type, left, right, on, options = {})
         @type = type
         @left = left
         @right = right
         @on = on
+        @options = DEFAULT_OPTIONS.merge(options)
       end
 
     private
 
-      attr_reader :on
+      attr_reader :on, :options
 
     public
 
@@ -34,12 +37,24 @@ module Bmg
             to_join.each do |right|
               yield right.merge(tuple)
             end
+          elsif left_join?
+            yield(tuple.merge(default_right_tuple))
           end
         end
       end
 
       def to_ast
-        [ :join, left.to_ast, right.to_ast, on ]
+        [ :join, left.to_ast, right.to_ast, on, extra_opts ].compact
+      end
+
+    protected
+
+      def left_join?
+        options[:variant] == :left
+      end
+
+      def default_right_tuple
+        options[:default_right_tuple]
       end
 
     protected ### optimization
@@ -63,8 +78,13 @@ module Bmg
 
     protected ### inspect
 
+      def extra_opts
+        extra = options.dup.delete_if{|k,v| DEFAULT_OPTIONS[k] == v }
+        extra.empty? ? nil : extra
+      end
+
       def args
-        [ on ]
+        [ on, extra_opts ].compact
       end
 
     private

data/lib/bmg/operator/transform.rb

@@ -0,0 +1,57 @@
+module Bmg
+  module Operator
+    #
+    # Transform operator.
+    #
+    # Transforms existing attributes through computations
+    #
+    # Example:
+    #
+    #   [{ a: 1 }] transform { a: ->(t){ t[:a]*2 } } => [{ a: 4 }]
+    #
+    class Transform
+      include Operator::Unary
+
+      DEFAULT_OPTIONS = {}
+
+      def initialize(type, operand, transformation, options = {})
+        @type = type
+        @operand = operand
+        @transformation = transformation
+        @options = DEFAULT_OPTIONS.merge(options)
+      end
+
+    protected
+
+      attr_reader :transformation
+
+    public
+
+      def each
+        t = transformer
+        @operand.each do |tuple|
+          yield t.call(tuple)
+        end
+      end
+
+      def to_ast
+        [ :transform, operand.to_ast, transformation.dup ]
+      end
+
+    protected ### optimization
+
+    protected ### inspect
+
+      def args
+        [ transformation ]
+      end
+
+    private
+
+      def transformer
+        @transformer ||= TupleTransformer.new(transformation)
+      end
+
+    end # class Transform
+  end # module Operator
+end # module Bmg

data/lib/bmg/relation.rb

@@ -105,6 +105,19 @@ module Bmg
       to_a.to_json(*args, &bl)
     end
 
+    # Writes the relation data to CSV.
+    #
+    # `string_or_io` and `options` are what CSV::new itself
+    # recognizes, default options are CSV's.
+    #
+    # When no string_or_io is used, the method uses a string.
+    #
+    # The method always returns the string_or_io.
+    def to_csv(options = {}, string_or_io = nil)
+      options, string_or_io = {}, options unless options.is_a?(Hash)
+      Writer::Csv.new(options).call(self, string_or_io)
+    end
+
     # Converts to an sexpr expression.
     def to_ast
       raise "Bmg is missing a feature!"

data/lib/bmg/relation/proxy.rb

@@ -0,0 +1,63 @@
+module Bmg
+  module Relation
+    #
+    # This module can be used to create typed collection on top
+    # of Bmg relations. Algebra methods will be delegated to the
+    # decorated relation, and results wrapped in a new instance
+    # of the class.
+    #
+    module Proxy
+
+      def initialize(relation)
+        @relation = relation
+      end
+
+      def method_missing(name, *args, &bl)
+        if @relation.respond_to?(name)
+          res = @relation.send(name, *args, &bl)
+          res.is_a?(Relation) ? _proxy(res) : res
+        else
+          super
+        end
+      end
+
+      def respond_to?(name, *args)
+        @relation.respond_to?(name) || super
+      end
+
+      [
+        :extend
+      ].each do |name|
+        define_method(name) do |*args, &bl|
+          res = @relation.send(name, *args, &bl)
+          res.is_a?(Relation) ? _proxy(res) : res
+        end
+      end
+
+      [
+        :one,
+        :one_or_nil
+      ].each do |meth|
+        define_method(meth) do |*args, &bl|
+          res = @relation.send(meth, *args, &bl)
+          res.nil? ? nil : _proxy_tuple(res)
+        end
+      end
+
+      def to_json(*args, &bl)
+        @relation.to_json(*args, &bl)
+      end
+
+    protected
+
+      def _proxy(relation)
+        self.class.new(relation)
+      end
+
+      def _proxy_tuple(tuple)
+        tuple
+      end
+
+    end # module Proxy
+  end # class Relation
+end # module Bmg

data/lib/bmg/sequel/translator.rb

@@ -41,7 +41,7 @@ module Bmg
         dataset   = apply(sexpr.from_clause)     if sexpr.from_clause
         #
         selection = apply(sexpr.select_list)
-        predicate = apply(sexpr.predicate)       if sexpr.predicate
+        predicate = compile_predicate(sexpr.predicate) if sexpr.predicate
         grouping  = apply(sexpr.group_by_clause) if sexpr.group_by_clause
         order     = apply(sexpr.order_by_clause) if sexpr.order_by_clause
         limit     = apply(sexpr.limit_clause)    if sexpr.limit_clause
@@ -50,7 +50,7 @@ module Bmg
         dataset   = dataset.select(*selection)
         dataset   = dataset.distinct             if sexpr.distinct?
         dataset   = dataset.where(predicate)     if predicate
-        dataset   = dataset.group(grouping)      if grouping
+        dataset   = dataset.group(*grouping)     if grouping
         dataset   = dataset.order_by(*order)     if order
         dataset   = dataset.limit(limit, offset == 0 ? nil : offset) if limit or offset
         dataset
@@ -70,13 +70,18 @@ module Bmg
         case kind = sexpr.left.first
         when :qualified_name
           left.column == right.value ? left : ::Sequel.as(left, right)
-        when :literal, :summarizer
+        when :literal, :summarizer, :func_call
           ::Sequel.as(left, right)
         else
           raise NotImplementedError, "Unexpected select item `#{kind}`"
         end
       end
 
+      def on_func_call(sexpr)
+        args = sexpr.func_args.map{|fa| apply(fa) }
+        ::Sequel.function(sexpr.func_name, *args)
+      end
+
       def on_summarizer(sexpr)
         if sexpr.summary_expr
           ::Sequel.function(sexpr.summary_func, apply(sexpr.summary_expr))
@@ -104,8 +109,15 @@ module Bmg
           elsif kind == :inner_join
             options = { qualify: false, table_alias: false }
             ds.join_table(:inner, apply(table), nil, options){|*args|
-              apply(on)
+              compile_predicate(on)
+            }
+          elsif kind == :left_join
+            options = { qualify: false, table_alias: false }
+            ds.join_table(:left, apply(table), nil, options){|*args|
+              compile_predicate(on)
             }
+          else
+            raise IllegalArgumentError, "Unrecognized from clause: `#{sexpr}`"
           end
         end
       end
@@ -153,22 +165,35 @@ module Bmg
         sexpr.last
       end
 
+    private
+
+      def dataset(expr)
+        return expr if ::Sequel::Dataset===expr
+        sequel_db[expr]
+      end
+
+      def compile_predicate(predicate)
+        PredicateTranslator.new(self).call(predicate)
+      end
+
+    class PredicateTranslator < Sexpr::Processor
+      include ::Predicate::ToSequel::Methods
+
+      def initialize(parent)
+        @parent = parent
+      end
+
     public ### Predicate hack
 
       def on_opaque(sexpr)
-        apply(sexpr.last)
+        @parent.apply(sexpr.last)
       end
 
       def on_exists(sexpr)
-        apply(sexpr.last).exists
+        @parent.apply(sexpr.last).exists
       end
 
-    private
-
-      def dataset(expr)
-        return expr if ::Sequel::Dataset===expr
-        sequel_db[expr]
-      end
+    end
 
     end # class Translator
   end # module Sequel