bmg 0.17.8 → 0.18.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 600739b827185e0d02252824b7b5616662a378b1c907a632ba695536e76d631e
-  data.tar.gz: f327e31860ef19e0ab8177ebd0e3d0c927c003cb25b8e48af202145cdf1ceece
+  metadata.gz: 1e208e1eb958222f5ffac079296fc7279f221dddf6ed8165f5004b0a90b08c4b
+  data.tar.gz: 6e9e2509ae9ce0d277d226d55d29274b895a6b31dabfe9ac387d4be7fd97be4a
 SHA512:
-  metadata.gz: 2cd09c0006211c6db5d220ee2371669bd3e29bf32c71eae9fb9ba2c4698392ed172412bbd71c9eded5df34172ab469aa6d20a4117ef263955ed73e8a128cf856
-  data.tar.gz: 8777307569c78001741a597b3ecd9974aa1ee94780495d86f97af38313dcd185c9516d43b9c1635cae5e57606c7422d3e6b3f4889be0b2adcdfeb9f2c30dee24
+  metadata.gz: d142fd93326193529359a02c9e14936ffc43dd52ceab4b44e7d890b82cc0df4561e555002f0e6c12e53f6956a4541140be5214a40745ac54e1dc046bdbd4b735
+  data.tar.gz: 6fbbd1ad533a0beee8faf783e07c7cedcc2535ef9edf1b601f973a30a513e5ee6d00f0031c8486f4460ccb912acdbad623c86ca649d4664fdc24c63c3f5de693

data/Gemfile

@@ -1,5 +1,2 @@
 source "https://rubygems.org"
 gemspec
-
-# gem "predicate", github: "enspirit/predicate", branch: "placeholders"
-# gem "predicate", path: "../predicate"

data/README.md

@@ -1,16 +1,30 @@
 # Bmg, a relational algebra (Alf's successor)!
 
+[![Build Status](https://travis-ci.com/enspirit/bmg.svg?branch=master)](https://travis-ci.com/enspirit/bmg)
+
 Bmg is a relational algebra implemented as a ruby library. It implements the
 [Relation as First-Class Citizen](http://www.try-alf.org/blog/2013-10-21-relations-as-first-class-citizen)
-paradigm contributed with Alf a few years ago.
-
-Like Alf, Bmg can be used to query relations in memory, from various files,
-SQL databases, and any data sources that can be seen as serving relations.
-Cross data-sources joins are supported, as with Alf.
-
-Unlike Alf, Bmg does not make any core ruby extension and exposes the
-object-oriented syntax only (not Alf's functional one). Bmg implementation is
-also much simpler, and make its easier to implement user-defined relations.
+paradigm contributed with [Alf](http://www.try-alf.org/) a few years ago.
+
+Bmg can be used to query relations in memory, from various files, SQL databases,
+and any data source that can be seen as serving relations. Cross data-sources
+joins are supported, as with Alf. For differences with Alf, see a section
+further down this README.
+
+## Outline
+
+* [Example](#example)
+* [Where are base relations coming from?](#where-are-base-relations-coming-from)
+  * [Memory relations](#memory-relations)
+  * [Connecting to SQL databases](#connecting-to-sql-databases)
+  * [Reading files (csv, excel, text)](#reading-files-csv-excel-text)
+  * [Your own relations](#your-own-relations)
+* [List of supported operators](#supported-operators)
+* [How is this different?](#how-is-this-different)
+  * [... from similar libraries](#-from-similar-libraries)
+  * [... from Alf](#-from-alf)
+* [Contribute](#contribute)
+* [License](#license)
 
 ## Example
 
@@ -27,7 +41,7 @@ suppliers = Bmg::Relation.new([
 ])
 
 by_city = suppliers
-  .restrict(Predicate.neq(status: 30))
+  .exclude(status: 30)
   .extend(upname: ->(t){ t[:name].upcase })
   .group([:sid, :name, :status], :suppliers_in)
 
@@ -35,76 +49,158 @@ puts JSON.pretty_generate(by_city)
 # [{...},...]
 ```
 
-## Connecting to a SQL database
+## Where are base relations coming from?
+
+Bmg sees relations as sets/enumerable of symbolized Ruby hashes. The following
+sections show you how to get them in the first place, to enter Relationland.
+
+### Memory relations
+
+If you have an Array of Hashes -- in fact any Enumerable -- you can easily get
+a Relation using either `Bmg::Relation.new` or `Bmg.in_memory`.
+
+```ruby
+# this...
+r = Bmg::Relation.new [{id: 1}, {id: 2}]
+
+# is the same as this...
+r = Bmg.in_memory [{id: 1}, {id: 2}]
+
+# entire algebra is available on `r`
+```
+
+### Connecting to SQL databases
 
-Bmg requires `sequel >= 3.0` to connect to SQL databases.
+Bmg currently requires `sequel >= 3.0` to connect to SQL databases. You also
+need to require `bmg/sequel`.
 
 ```ruby
 require 'sqlite3'
 require 'bmg'
 require 'bmg/sequel'
+```
 
-DB = Sequel.connect("sqlite://suppliers-and-parts.db")
+Then `Bmg.sequel` serves relations for tables of your SQL database:
 
+```ruby
+DB = Sequel.connect("sqlite://suppliers-and-parts.db")
 suppliers = Bmg.sequel(:suppliers, DB)
+```
+
+The entire algebra is available on those relations. As long as you keep using
+operators that can be translated to SQL, results remain SQL-able:
 
+```ruby
 big_suppliers = suppliers
-  .restrict(Predicate.neq(status: 30))
+  .exclude(status: 30)
+  .project([:sid, :name])
 
 puts big_suppliers.to_sql
-# SELECT `t1`.`sid`, `t1`.`name`, `t1`.`status`, `t1`.`city` FROM `suppliers` AS 't1' WHERE (`t1`.`status` != 30)
+# SELECT `t1`.`sid`, `t1`.`name` FROM `suppliers` AS 't1' WHERE (`t1`.`status` != 30)
+```
 
-puts JSON.pretty_generate(big_suppliers)
-# [{...},...]
+Operators not translatable to SQL are available too (such as `group` below).
+Bmg fallbacks to memory operators for them, but remains capable of pushing some
+operators down the tree as illustrated below (the restriction on `:city` is
+pushed to the SQL server):
+
+```ruby
+Bmg.sequel(:suppliers, sequel_db)
+  .project([:sid, :name, :city])
+  .group([:sid, :name], :suppliers_in)
+  .restrict(city: ["Paris", "London"])
+  .debug
+
+# (group
+#   (sequel SELECT `t1`.`sid`, `t1`.`name`, `t1`.`city` FROM `suppliers` AS 't1' WHERE (`t1`.`city` IN ('Paris', 'London')))
+#   [:sid, :name, :status]
+#   :suppliers_in
+#   {:array=>false})
 ```
 
-## How is this different from similar libraries?
+### Reading files (csv, excel, text)
 
-1. The libraries you probably know (Sequel, Arel, SQLAlchemy, Korma, jOOQ,
-   etc.) do not implement a genuine relational algebra: their support for
-   chaining relational operators is limited (yielding errors or wrong SQL
-   queries). Bmg **always** allows chaining operators. If it does not, it's
-   a bug. In other words, the following query is 100% valid:
+Bmg provides simple adapters to read files and reach Relationland as soon as
+possible.
 
-       relation
-         .restrict(...)    # aka where
-         .union(...)
-         .summarize(...)   # aka group by
-         .restrict(...)
+#### CSV files
 
-2. Bmg supports in memory relations, json relations, csv relations, SQL
-   relations and so on. It's not tight to SQL generation, and supports
-   queries accross multiple data sources.
+```ruby
+csv_options = { col_sep: ",", quote_char: '"' }
+r = Bmg.csv("path/to/a/file.csv", csv_options)
+```
 
-3. Bmg makes a best effort to optimize queries, simplifying both generated
-   SQL code (low-level accesses to datasources) and in-memory operations.
+Options are directly transmitted to `::CSV.new`, check ruby's standard
+library.
 
-4. Bmg supports various *structuring* operators (group, image, autowrap,
-   autosummarize, etc.) and allows building 'non flat' relations.
+#### Excel files
 
-## How is this different from Alf?
+You will need to add [`roo`](https://github.com/roo-rb/roo) to your Gemfile to
+read `.xls` and `.xlsx` files with Bmg.
 
-1. Bmg's implementation is much simpler than Alf, and uses no ruby core
-   extention.
+```ruby
+roo_options = { skip: 1 }
+r = Bmg.excel("path/to/a/file.xls", roo_options)
+```
 
-2. We are confident using Bmg in production. Systematic inspection of query
-   plans is suggested though. Alf was a bit too experimental to be used on
-   (critical) production systems.
+Options are directly transmitted to `Roo::Spreadsheet.open`, check roo's
+documentation.
 
-2. Alf exposes a functional syntax, command line tool, restful tools and
-   many more. Bmg is limited to the core algebra, main Relation abstraction
-   and SQL generation.
+#### Text files
 
-3. Bmg is less strict regarding conformance to relational theory, and
-   may actually expose non relational features (such as support for null,
-   left_join operator, etc.). Sharp tools hurt, use them with great care.
+There is also a straightforward way to read text files and convert lines to
+tuples.
 
-4. Bmg does not yet implement all operators documented on try-alf.org, even
-   if we plan to eventually support them all.
+```ruby
+r = Bmg.text_file("path/to/a/file.txt")
+r.type.attrlist
+# => [:line, :text]
+```
 
-5. Bmg has a few additional operators that prove very useful on real
-   production use cases: prefix, suffix, autowrap, autosummarize, left_join,
-   rxmatch, etc.
+Without options tuples will have `:line` and `:text` attributes, the former
+being the line number (starting at 1) and the latter being the line itself
+(stripped).
+
+The are a couple of options (see `Bmg::Reader::Textfile`). The most useful one
+is the use a of a Regexp with named captures to automatically extract
+attributes:
+
+```ruby
+r = Bmg.text_file("path/to/a/file.txt", parse: /GET (?<url>([^\s]+))/)
+r.type.attrlist
+# => [:line, :url]
+```
+
+In this scenario, non matching lines are skipped. The `:line` attribute keeps
+being used to have at least one candidate key (so to speak).
+
+### Your own relations
+
+As noted earlier, Bmg has a simple relation interface where you only have to
+provide an iteration of symbolized tuples.
+
+```ruby
+class MyRelation
+  include Bmg::Relation
+
+  def each
+    yield(id: 1, name: "Alf", year: 2014)
+    yield(id: 2, name: "Bmg", year: 2018)
+  end
+end
+
+MyRelation.new
+  .restrict(Predicate.gt(:year, 2015))
+  .allbut([:year])
+```
+
+As shown, creating adapters on top of various data source is straighforward.
+Adapters can also participate to query optimization (such as pushing
+restrictions down the tree) by overriding the underscored version of operators
+(e.g. `_restrict`).
+
+Have a look at `Bmg::Algebra` for the protocol and `Bmg::Sql::Relation` for an
+example. Keep in touch with the team if you need some help.
 
 ## Supported operators
 
@@ -114,8 +210,10 @@ r.autowrap(split: '_')                       # structure a flat relation, split:
 r.autosummarize([:a, :b, ...], x: :sum)      # (experimental) usual summarizers supported
 r.constants(x: 12, ...)                      # add constant attributes (sometimes useful in unions)
 r.extend(x: ->(t){ ... }, ...)               # add computed attributes
+r.exclude(predicate)                         # shortcut for restrict(!predicate)
 r.group([:a, :b, ...], :x)                   # relation-valued attribute from attributes
 r.image(right, :x, [:a, :b, ...])            # relation-valued attribute from another relation
+r.images({:x => r1, :y => r2}, [:a, ...])    # shortcut over image(r1, :x, ...).image(r2, :y, ...)
 r.join(right, [:a, :b, ...])                 # natural join on a join key
 r.join(right, :a => :x, :b => :y, ...)       # natural join after right reversed renaming
 r.left_join(right, [:a, :b, ...], {...})     # left join with optional default right tuple
@@ -137,14 +235,95 @@ t.transform(&:to_s)                          # similar, but Proc-driven
 t.transform(:foo => :upcase, ...)            # specific-attrs tranformation
 t.transform([:to_s, :upcase])                # chain-transformation
 r.union(right)                               # relational union
+r.where(predicate)                           # alias for restrict(predicate)
 ```
 
-## Who is behind Bmg?
+## How is this different?
+
+### ... from similar libraries?
+
+1. The libraries you probably know (Sequel, Arel, SQLAlchemy, Korma, jOOQ,
+   etc.) do not implement a genuine relational algebra. Their support for
+   chaining relational operators is thus limited (restricting your expression
+   power and/or raising errors and/or outputting wrong or counterintuitive
+   SQL code). Bmg **always** allows chaining operators. If it does not, it's
+   a bug.
+
+   For instance the expression below is 100% valid in Bmg. The last where
+   clause applies to the result of the summarize (while SQL requires a `HAVING`
+   clause, or a `SELECT ... FROM (SELECT ...) r`).
+
+      ```ruby
+      relation
+        .where(...)
+        .union(...)
+        .summarize(...)   # aka group by
+        .where(...)
+      ```
+
+2. Bmg supports in memory relations, json relations, csv relations, SQL
+   relations and so on. It's not tight to SQL generation, and supports
+   queries accross multiple data sources.
+
+3. Bmg makes a best effort to optimize queries, simplifying both generated
+   SQL code (low-level accesses to datasources) and in-memory operations.
+
+4. Bmg supports various *structuring* operators (group, image, autowrap,
+   autosummarize, etc.) and allows building 'non flat' relations.
+
+5. Bmg can use full ruby power when that helps (e.g. regular expressions in
+   WHERE clauses or ruby code in EXTEND clauses). This may prevent Bmg from
+   delegating work to underlying data sources (e.g. SQL server) and should
+   therefore be used with care though.
+
+### ... from Alf?
+
+If you use Alf (or used it in the past), below are the main differences between
+Bmg and Alf. Bmg has NOT been written to be API-compatible with Alf and will
+probably never be.
+
+1. Bmg's implementation is much simpler than Alf and uses no ruby core
+   extention.
+
+2. We are confident using Bmg in production. Systematic inspection of query
+   plans is advised though. Alf was a bit too experimental to be used on
+   (critical) production systems.
+
+3. Alf exposes a functional syntax, command line tool, restful tools and
+   many more. Bmg is limited to the core algebra, main Relation abstraction
+   and SQL generation.
 
-Bernard Lambeau (bernard@klaro.cards) is Alf & Bmg main engineer & maintainer.
+4. Bmg is less strict regarding conformance to relational theory, and
+   may actually expose non relational features (such as support for null,
+   left_join operator, etc.). Sharp tools hurt, use them with care.
+
+5. Unlike Alf::Relation instances of Bmg::Relation capture query-trees, not
+   values. Currently two instances `r1` and `r2` are not equal even if they
+   define the same mathematical relation. As a consequence joining on
+   relation-valued attributes does not work as expected in Bmg until further
+   notice.
+
+6. Bmg does not implement all operators documented on try-alf.org, even if
+   we plan to eventually support most of them.
+
+7. Bmg has a few additional operators that prove very useful on real
+   production use cases: prefix, suffix, autowrap, autosummarize, left_join,
+   rxmatch, etc.
+
+8. Bmg optimizes queries and compiles them to SQL on the fly, while Alf was
+   building an AST internally first. Strictly speaking this makes Bmg less
+   powerful than Alf since optimizations cannot be turned off for now.
+
+## Contribute
+
+Please use github issues and pull requests for all questions, bug reports,
+and contributions. Don't hesitate to get in touch with us with an early code
+spike if you plan to add non trivial features.
+
+## Licence
+
+This software is distributed by Enspirit SRL under a MIT Licence. Please
+contact Bernard Lambeau (blambeau@gmail.com) with any question.
 
 Enspirit (https://enspirit.be) and Klaro App (https://klaro.cards) are both
 actively using and contributing to the library.
-
-Feel free to contact us for help, ideas and/or contributions. Please use github
-issues and pull requests if possible if code is involved.

data/lib/bmg.rb

@@ -1,6 +1,7 @@
 require 'path'
 require 'predicate'
 require 'forwardable'
+require 'set'
 module Bmg
 
   def in_memory(enumerable, type = Type::ANY)
@@ -8,6 +9,11 @@ module Bmg
   end
   module_function :in_memory
 
+  def text_file(path, options = {}, type = Type::ANY)
+    Reader::TextFile.new(type, path, options).spied(main_spy)
+  end
+  module_function :text_file
+
   def csv(path, options = {}, type = Type::ANY)
     Reader::Csv.new(type, path, options).spied(main_spy)
   end

data/lib/bmg/algebra.rb

@@ -174,6 +174,7 @@ module Bmg
 
     def transform(transformation = nil, options = {}, &proc)
       transformation, options = proc, (transformation || {}) unless proc.nil?
+      return self if transformation.is_a?(Hash) && transformation.empty?
       _transform(self.type.transform(transformation, options), transformation, options)
     end
 

data/lib/bmg/algebra/shortcuts.rb

@@ -2,6 +2,14 @@ module Bmg
   module Algebra
     module Shortcuts
 
+      def where(predicate)
+        restrict(predicate)
+      end
+
+      def exclude(predicate)
+        restrict(!Predicate.coerce(predicate))
+      end
+
       def rxmatch(attrs, matcher, options = {})
         predicate = attrs.inject(Predicate.contradiction){|p,a|
           p | Predicate.match(a, matcher, options)
@@ -31,6 +39,12 @@ module Bmg
         self.image(right.rename(renaming), as, on.keys, options)
       end
 
+      def images(rights, on = [], options = {})
+        rights.each_pair.inject(self){|memo,(as,right)|
+          memo.image(right, as, on, options)
+        }
+      end
+
       def join(right, on = [])
         return super unless on.is_a?(Hash)
         renaming = Hash[on.map{|k,v| [v,k] }]

data/lib/bmg/operator/allbut.rb

@@ -30,6 +30,7 @@ module Bmg
     public
 
       def each
+        return to_enum unless block_given?
         seen = {}
         @operand.each do |tuple|
           allbuted = tuple_allbut(tuple)
@@ -63,12 +64,38 @@ module Bmg
 
     protected ### optimization
 
+      def _allbut(type, butlist)
+        operand.allbut(self.butlist|butlist)
+      end
+
+      def _matching(type, right, on)
+        # Always possible to push the matching, since by construction
+        # `on` can only use attributes that have not been trown away,
+        # hence they exist on `operand` too.
+        operand.matching(right, on).allbut(butlist)
+      end
+
+      def _page(type, ordering, page_index, options)
+        return super unless self.preserving_key?
+        operand.page(ordering, page_index, options).allbut(butlist)
+      end
+
+      def _project(type, attrlist)
+        operand.project(attrlist)
+      end
+
       def _restrict(type, predicate)
         operand.restrict(predicate).allbut(butlist)
       end
 
     protected ### inspect
 
+      def preserving_key?
+        operand.type.knows_keys? && operand.type.keys.find{|k|
+          (k & butlist).empty?
+        }
+      end
+
       def args
         [ butlist ]
       end