bmg 0.17.2 → 0.17.7

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 189526b3a2d6324a3ac380a7d1eb14f7b1559fc56ca59b72c34abba57f160584
4
- data.tar.gz: 62f44c0fe86139983cf4aecc3407e2d07ae19ebf3c23c5d4a65a39f53d7c6d9a
3
+ metadata.gz: 41671cac00088b48c072b6e276538989ce92f3583faef903748265558ad7d1d3
4
+ data.tar.gz: 32003d306f1d5ea608efea4c2dd875ac935fd205ff4e016e8d8af4afbc7ee1c5
5
5
  SHA512:
6
- metadata.gz: 03aa4daed6097ba7a680b352bf1fa4eba97635c3830fa3e375b41dba05df71489dc40d22379bed397f7797f57992a0c1c240275ec8d16a828226b0d3e0c27b64
7
- data.tar.gz: 3a025b7d36cf8316f01f54b73b76a5d74622124a88f0dd75160039a0e374966ca8d4021bc4dddebb3786ecb6add65742b69f7d1a0e8242816b58f993f37c9cb9
6
+ metadata.gz: 5c0fa25d89eefaaed383a8d675e046894ed8cdbc705ffc8a283a523d53d7c89b0879118d6a8163eedb5e60add7fe4dd2594238722946652c93d298c2cfd7f2e2
7
+ data.tar.gz: 7b0181f9d426d856f66c51ca010009c834cd4c09f3fa9563013ab06a67181576b13f469a1e4209edc82eff9d0100f3f26fdaafb4e8e21efc667cfaa4b5878371
data/README.md CHANGED
@@ -14,7 +14,7 @@ also much simpler, and make its easier to implement user-defined relations.
14
14
 
15
15
  ## Example
16
16
 
17
- ```
17
+ ```ruby
18
18
  require 'bmg'
19
19
  require 'json'
20
20
 
@@ -32,13 +32,14 @@ by_city = suppliers
32
32
  .group([:sid, :name, :status], :suppliers_in)
33
33
 
34
34
  puts JSON.pretty_generate(by_city)
35
+ # [{...},...]
35
36
  ```
36
37
 
37
38
  ## Connecting to a SQL database
38
39
 
39
40
  Bmg requires `sequel >= 3.0` to connect to SQL databases.
40
41
 
41
- ```
42
+ ```ruby
42
43
  require 'sqlite3'
43
44
  require 'bmg'
44
45
  require 'bmg/sequel'
@@ -47,16 +48,67 @@ DB = Sequel.connect("sqlite://suppliers-and-parts.db")
47
48
 
48
49
  suppliers = Bmg.sequel(:suppliers, DB)
49
50
 
50
- puts suppliers
51
+ big_suppliers = suppliers
51
52
  .restrict(Predicate.neq(status: 30))
52
- .to_sql
53
53
 
54
+ puts big_suppliers.to_sql
54
55
  # SELECT `t1`.`sid`, `t1`.`name`, `t1`.`status`, `t1`.`city` FROM `suppliers` AS 't1' WHERE (`t1`.`status` != 30)
56
+
57
+ puts JSON.pretty_generate(big_suppliers)
58
+ # [{...},...]
55
59
  ```
56
60
 
61
+ ## How is this different from similar libraries?
62
+
63
+ 1. The libraries you probably know (Sequel, Arel, SQLAlchemy, Korma, jOOQ,
64
+ etc.) do not implement a genuine relational algebra: their support for
65
+ chaining relational operators is limited (yielding errors or wrong SQL
66
+ queries). Bmg **always** allows chaining operators. If it does not, it's
67
+ a bug. In other words, the following query is 100% valid:
68
+
69
+ relation
70
+ .restrict(...) # aka where
71
+ .union(...)
72
+ .summarize(...) # aka group by
73
+ .restrict(...)
74
+
75
+ 2. Bmg supports in memory relations, json relations, csv relations, SQL
76
+ relations and so on. It's not tight to SQL generation, and supports
77
+ queries accross multiple data sources.
78
+
79
+ 3. Bmg makes a best effort to optimize queries, simplifying both generated
80
+ SQL code (low-level accesses to datasources) and in-memory operations.
81
+
82
+ 4. Bmg supports various *structuring* operators (group, image, autowrap,
83
+ autosummarize, etc.) and allows building 'non flat' relations.
84
+
85
+ ## How is this different from Alf?
86
+
87
+ 1. Bmg's implementation is much simpler than Alf, and uses no ruby core
88
+ extention.
89
+
90
+ 2. We are confident using Bmg in production. Systematic inspection of query
91
+ plans is suggested though. Alf was a bit too experimental to be used on
92
+ (critical) production systems.
93
+
94
+ 2. Alf exposes a functional syntax, command line tool, restful tools and
95
+ many more. Bmg is limited to the core algebra, main Relation abstraction
96
+ and SQL generation.
97
+
98
+ 3. Bmg is less strict regarding conformance to relational theory, and
99
+ may actually expose non relational features (such as support for null,
100
+ left_join operator, etc.). Sharp tools hurt, use them with great care.
101
+
102
+ 4. Bmg does not yet implement all operators documented on try-alf.org, even
103
+ if we plan to eventually support them all.
104
+
105
+ 5. Bmg has a few additional operators that prove very useful on real
106
+ production use cases: prefix, suffix, autowrap, autosummarize, left_join,
107
+ rxmatch, etc.
108
+
57
109
  ## Supported operators
58
110
 
59
- ```
111
+ ```ruby
60
112
  r.allbut([:a, :b, ...]) # remove specified attributes
61
113
  r.autowrap(split: '_') # structure a flat relation, split: '_' is the default
62
114
  r.autosummarize([:a, :b, ...], x: :sum) # (experimental) usual summarizers supported
@@ -66,6 +118,8 @@ r.group([:a, :b, ...], :x) # relation-valued attribute from at
66
118
  r.image(right, :x, [:a, :b, ...]) # relation-valued attribute from another relation
67
119
  r.join(right, [:a, :b, ...]) # natural join on a join key
68
120
  r.join(right, :a => :x, :b => :y, ...) # natural join after right reversed renaming
121
+ r.left_join(right, [:a, :b, ...], {...}) # left join with optional default right tuple
122
+ r.left_join(right, {:a => :x, ...}, {...}) # left join after right reversed renaming
69
123
  r.matching(right, [:a, :b, ...]) # semi join, aka where exists
70
124
  r.matching(right, :a => :x, :b => :y, ...) # semi join, after right reversed renaming
71
125
  r.not_matching(right, [:a, :b, ...]) # inverse semi join, aka where not exists
@@ -78,5 +132,19 @@ r.restrict(a: "foo", b: "bar", ...) # relational restriction, aka where
78
132
  r.rxmatch([:a, :b, ...], /xxx/) # regex match kind of restriction
79
133
  r.summarize([:a, :b, ...], x: :sum) # relational summarization
80
134
  r.suffix(:_foo, but: [:a, ...]) # suffix kind of renaming
135
+ t.transform(:to_s) # all-attrs transformation
136
+ t.transform(&:to_s) # similar, but Proc-driven
137
+ t.transform(:foo => :upcase, ...) # specific-attrs tranformation
138
+ t.transform([:to_s, :upcase]) # chain-transformation
81
139
  r.union(right) # relational union
82
140
  ```
141
+
142
+ ## Who is behind Bmg?
143
+
144
+ Bernard Lambeau (bernard@klaro.cards) is Alf & Bmg main engineer & maintainer.
145
+
146
+ Enspirit (https://enspirit.be) and Klaro App (https://klaro.cards) are both
147
+ actively using and contributing to the library.
148
+
149
+ Feel free to contact us for help, ideas and/or contributions. Please use github
150
+ issues and pull requests if possible if code is involved.
data/lib/bmg.rb CHANGED
@@ -38,11 +38,13 @@ module Bmg
38
38
  require_relative 'bmg/operator'
39
39
 
40
40
  require_relative 'bmg/reader'
41
+ require_relative 'bmg/writer'
41
42
 
42
43
  require_relative 'bmg/relation/empty'
43
44
  require_relative 'bmg/relation/in_memory'
44
45
  require_relative 'bmg/relation/spied'
45
46
  require_relative 'bmg/relation/materialized'
47
+ require_relative 'bmg/relation/proxy'
46
48
 
47
49
  # Deprecated
48
50
  Leaf = Relation::InMemory
@@ -80,6 +80,19 @@ module Bmg
80
80
  end
81
81
  protected :_joined_with
82
82
 
83
+ def left_join(right, on = [], default_right_tuple = {})
84
+ drt = default_right_tuple
85
+ _left_join self.type.left_join(right.type, on, drt), right, on, drt
86
+ end
87
+
88
+ def _left_join(type, right, on, default_right_tuple)
89
+ Operator::Join.new(type, self, right, on, {
90
+ variant: :left,
91
+ default_right_tuple: default_right_tuple
92
+ })
93
+ end
94
+ protected :_left_join
95
+
83
96
  def matching(right, on = [])
84
97
  _matching self.type.matching(right.type, on), right, on
85
98
  end
@@ -159,6 +172,16 @@ module Bmg
159
172
  end
160
173
  protected :_summarize
161
174
 
175
+ def transform(transformation = nil, options = {}, &proc)
176
+ transformation, options = proc, (transformation || {}) unless proc.nil?
177
+ _transform(self.type.transform(transformation, options), transformation, options)
178
+ end
179
+
180
+ def _transform(type, transformation, options)
181
+ Operator::Transform.new(type, self, transformation, options)
182
+ end
183
+ protected :_transform
184
+
162
185
  def union(other, options = {})
163
186
  return self if other.is_a?(Relation::Empty)
164
187
  _union self.type.union(other.type), other, options
@@ -37,6 +37,12 @@ module Bmg
37
37
  self.join(right.rename(renaming), on.keys)
38
38
  end
39
39
 
40
+ def left_join(right, on = [], *args)
41
+ return super unless on.is_a?(Hash)
42
+ renaming = Hash[on.map{|k,v| [v,k] }]
43
+ self.left_join(right.rename(renaming), on.keys, *args)
44
+ end
45
+
40
46
  def matching(right, on = [])
41
47
  return super unless on.is_a?(Hash)
42
48
  renaming = Hash[on.map{|k,v| [v,k] }]
@@ -46,4 +46,5 @@ require_relative 'operator/rename'
46
46
  require_relative 'operator/restrict'
47
47
  require_relative 'operator/rxmatch'
48
48
  require_relative 'operator/summarize'
49
+ require_relative 'operator/transform'
49
50
  require_relative 'operator/union'
@@ -8,16 +8,19 @@ module Bmg
8
8
  class Join
9
9
  include Operator::Binary
10
10
 
11
- def initialize(type, left, right, on)
11
+ DEFAULT_OPTIONS = {}
12
+
13
+ def initialize(type, left, right, on, options = {})
12
14
  @type = type
13
15
  @left = left
14
16
  @right = right
15
17
  @on = on
18
+ @options = DEFAULT_OPTIONS.merge(options)
16
19
  end
17
20
 
18
21
  private
19
22
 
20
- attr_reader :on
23
+ attr_reader :on, :options
21
24
 
22
25
  public
23
26
 
@@ -34,12 +37,24 @@ module Bmg
34
37
  to_join.each do |right|
35
38
  yield right.merge(tuple)
36
39
  end
40
+ elsif left_join?
41
+ yield(tuple.merge(default_right_tuple))
37
42
  end
38
43
  end
39
44
  end
40
45
 
41
46
  def to_ast
42
- [ :join, left.to_ast, right.to_ast, on ]
47
+ [ :join, left.to_ast, right.to_ast, on, extra_opts ].compact
48
+ end
49
+
50
+ protected
51
+
52
+ def left_join?
53
+ options[:variant] == :left
54
+ end
55
+
56
+ def default_right_tuple
57
+ options[:default_right_tuple]
43
58
  end
44
59
 
45
60
  protected ### optimization
@@ -63,8 +78,13 @@ module Bmg
63
78
 
64
79
  protected ### inspect
65
80
 
81
+ def extra_opts
82
+ extra = options.dup.delete_if{|k,v| DEFAULT_OPTIONS[k] == v }
83
+ extra.empty? ? nil : extra
84
+ end
85
+
66
86
  def args
67
- [ on ]
87
+ [ on, extra_opts ].compact
68
88
  end
69
89
 
70
90
  private
@@ -0,0 +1,57 @@
1
+ module Bmg
2
+ module Operator
3
+ #
4
+ # Transform operator.
5
+ #
6
+ # Transforms existing attributes through computations
7
+ #
8
+ # Example:
9
+ #
10
+ # [{ a: 1 }] transform { a: ->(t){ t[:a]*2 } } => [{ a: 4 }]
11
+ #
12
+ class Transform
13
+ include Operator::Unary
14
+
15
+ DEFAULT_OPTIONS = {}
16
+
17
+ def initialize(type, operand, transformation, options = {})
18
+ @type = type
19
+ @operand = operand
20
+ @transformation = transformation
21
+ @options = DEFAULT_OPTIONS.merge(options)
22
+ end
23
+
24
+ protected
25
+
26
+ attr_reader :transformation
27
+
28
+ public
29
+
30
+ def each
31
+ t = transformer
32
+ @operand.each do |tuple|
33
+ yield t.call(tuple)
34
+ end
35
+ end
36
+
37
+ def to_ast
38
+ [ :transform, operand.to_ast, transformation.dup ]
39
+ end
40
+
41
+ protected ### optimization
42
+
43
+ protected ### inspect
44
+
45
+ def args
46
+ [ transformation ]
47
+ end
48
+
49
+ private
50
+
51
+ def transformer
52
+ @transformer ||= TupleTransformer.new(transformation)
53
+ end
54
+
55
+ end # class Transform
56
+ end # module Operator
57
+ end # module Bmg
@@ -105,6 +105,19 @@ module Bmg
105
105
  to_a.to_json(*args, &bl)
106
106
  end
107
107
 
108
+ # Writes the relation data to CSV.
109
+ #
110
+ # `string_or_io` and `options` are what CSV::new itself
111
+ # recognizes, default options are CSV's.
112
+ #
113
+ # When no string_or_io is used, the method uses a string.
114
+ #
115
+ # The method always returns the string_or_io.
116
+ def to_csv(options = {}, string_or_io = nil)
117
+ options, string_or_io = {}, options unless options.is_a?(Hash)
118
+ Writer::Csv.new(options).call(self, string_or_io)
119
+ end
120
+
108
121
  # Converts to an sexpr expression.
109
122
  def to_ast
110
123
  raise "Bmg is missing a feature!"
@@ -0,0 +1,63 @@
1
+ module Bmg
2
+ module Relation
3
+ #
4
+ # This module can be used to create typed collection on top
5
+ # of Bmg relations. Algebra methods will be delegated to the
6
+ # decorated relation, and results wrapped in a new instance
7
+ # of the class.
8
+ #
9
+ module Proxy
10
+
11
+ def initialize(relation)
12
+ @relation = relation
13
+ end
14
+
15
+ def method_missing(name, *args, &bl)
16
+ if @relation.respond_to?(name)
17
+ res = @relation.send(name, *args, &bl)
18
+ res.is_a?(Relation) ? _proxy(res) : res
19
+ else
20
+ super
21
+ end
22
+ end
23
+
24
+ def respond_to?(name, *args)
25
+ @relation.respond_to?(name) || super
26
+ end
27
+
28
+ [
29
+ :extend
30
+ ].each do |name|
31
+ define_method(name) do |*args, &bl|
32
+ res = @relation.send(name, *args, &bl)
33
+ res.is_a?(Relation) ? _proxy(res) : res
34
+ end
35
+ end
36
+
37
+ [
38
+ :one,
39
+ :one_or_nil
40
+ ].each do |meth|
41
+ define_method(meth) do |*args, &bl|
42
+ res = @relation.send(meth, *args, &bl)
43
+ res.nil? ? nil : _proxy_tuple(res)
44
+ end
45
+ end
46
+
47
+ def to_json(*args, &bl)
48
+ @relation.to_json(*args, &bl)
49
+ end
50
+
51
+ protected
52
+
53
+ def _proxy(relation)
54
+ self.class.new(relation)
55
+ end
56
+
57
+ def _proxy_tuple(tuple)
58
+ tuple
59
+ end
60
+
61
+ end # module Proxy
62
+ end # class Relation
63
+ end # module Bmg
@@ -41,7 +41,7 @@ module Bmg
41
41
  dataset = apply(sexpr.from_clause) if sexpr.from_clause
42
42
  #
43
43
  selection = apply(sexpr.select_list)
44
- predicate = apply(sexpr.predicate) if sexpr.predicate
44
+ predicate = compile_predicate(sexpr.predicate) if sexpr.predicate
45
45
  grouping = apply(sexpr.group_by_clause) if sexpr.group_by_clause
46
46
  order = apply(sexpr.order_by_clause) if sexpr.order_by_clause
47
47
  limit = apply(sexpr.limit_clause) if sexpr.limit_clause
@@ -70,13 +70,18 @@ module Bmg
70
70
  case kind = sexpr.left.first
71
71
  when :qualified_name
72
72
  left.column == right.value ? left : ::Sequel.as(left, right)
73
- when :literal, :summarizer
73
+ when :literal, :summarizer, :func_call
74
74
  ::Sequel.as(left, right)
75
75
  else
76
76
  raise NotImplementedError, "Unexpected select item `#{kind}`"
77
77
  end
78
78
  end
79
79
 
80
+ def on_func_call(sexpr)
81
+ args = sexpr.func_args.map{|fa| apply(fa) }
82
+ ::Sequel.function(sexpr.func_name, *args)
83
+ end
84
+
80
85
  def on_summarizer(sexpr)
81
86
  if sexpr.summary_expr
82
87
  ::Sequel.function(sexpr.summary_func, apply(sexpr.summary_expr))
@@ -104,8 +109,15 @@ module Bmg
104
109
  elsif kind == :inner_join
105
110
  options = { qualify: false, table_alias: false }
106
111
  ds.join_table(:inner, apply(table), nil, options){|*args|
107
- apply(on)
112
+ compile_predicate(on)
113
+ }
114
+ elsif kind == :left_join
115
+ options = { qualify: false, table_alias: false }
116
+ ds.join_table(:left, apply(table), nil, options){|*args|
117
+ compile_predicate(on)
108
118
  }
119
+ else
120
+ raise IllegalArgumentError, "Unrecognized from clause: `#{sexpr}`"
109
121
  end
110
122
  end
111
123
  end
@@ -153,22 +165,35 @@ module Bmg
153
165
  sexpr.last
154
166
  end
155
167
 
168
+ private
169
+
170
+ def dataset(expr)
171
+ return expr if ::Sequel::Dataset===expr
172
+ sequel_db[expr]
173
+ end
174
+
175
+ def compile_predicate(predicate)
176
+ PredicateTranslator.new(self).call(predicate)
177
+ end
178
+
179
+ class PredicateTranslator < Sexpr::Processor
180
+ include ::Predicate::ToSequel::Methods
181
+
182
+ def initialize(parent)
183
+ @parent = parent
184
+ end
185
+
156
186
  public ### Predicate hack
157
187
 
158
188
  def on_opaque(sexpr)
159
- apply(sexpr.last)
189
+ @parent.apply(sexpr.last)
160
190
  end
161
191
 
162
192
  def on_exists(sexpr)
163
- apply(sexpr.last).exists
193
+ @parent.apply(sexpr.last).exists
164
194
  end
165
195
 
166
- private
167
-
168
- def dataset(expr)
169
- return expr if ::Sequel::Dataset===expr
170
- sequel_db[expr]
171
- end
196
+ end
172
197
 
173
198
  end # class Translator
174
199
  end # module Sequel