bmg 0.24.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ff9e5fa48bb225feaeee07ddbfb15f8616e7943be2304113d727d7b8fd71dda2
-  data.tar.gz: b8924512fc6e1b4d8cdb6d86ab40e5e5f317850839ae6d8bc3f02ddd8534e578
+  metadata.gz: b73ed9026efe140b3d55eb59a1949db5fad0ac04ff401f3c44199d39274108ca
+  data.tar.gz: 1d375d5ca7638e43d21c6bc27a34b3df47f6947d7d853158fa0bdc90db0754cc
 SHA512:
-  metadata.gz: dc7c8dde953a4792a17177abe5a3decd4a3aac156cda87c7ad8854382d2248b03a272a6d318a348fbcba255ba5302e5adbaa4f3764c6db04b61924e9d11e50f4
-  data.tar.gz: 5148e32c981959c62eedae0f8b15cd8816d9afc01f7e1e00e94cc4d7086892e22dc4b6da0a3bb07a57ec138e91060b3c2a839c9f6e4ef5b6af053df08382c20e
+  metadata.gz: f55dba1dc2db2ef68beaf1d0f58c0c84fd2ff7c4bb2c1b883ad5726fcdd6b44588529f602876783d6d14823e3cb6494bb9335490c6b214cc725a1e55d93d4181
+  data.tar.gz: ba1fe4d29f70002410ddb0d7afe7fd63555a87746cbfac7c85bc1ca0c3af5c796a3f4245928e8c1f841dfc847ad375518728c1f3ad225c434bb2c8420f8e4a2b

data/README.md

@@ -31,6 +31,7 @@ further down this README.
 * [How is this different?](#how-is-this-different)
   * [... from similar libraries](#-from-similar-libraries)
   * [... from Alf](#-from-alf)
+* [Stability and versioning](#stability-and-versioning)
 * [Contribute](#contribute)
 * [License](#license)
 
@@ -202,7 +203,7 @@ documentation.
 
 ### Connecting to Redis databases
 
-Bmg currently requires `bmg-redis` and `redis >= 4.6` to connect
+Bmg currently requires `bmg-redis` and `redis >= 4.0` to connect
 to Redis databases. You also need to require `bmg/redis`.
 
 ```Gemfile
@@ -360,7 +361,7 @@ db.to_xlsx('path/to/file.xlsx')
 ```ruby
 r.allbut([:a, :b, ...])                      # remove specified attributes
 r.autowrap(split: '_')                       # structure a flat relation, split: '_' is the default
-r.autosummarize([:a, :b, ...], x: :sum)      # (experimental) usual summarizers supported
+r.autosummarize([:a, :b, ...], x: :sum)      # usual summarizers supported
 r.constants(x: 12, ...)                      # add constant attributes (sometimes useful in unions)
 r.cross_product(right)                       # cross product, alias `cross_join`
 r.extend(x: ->(t){ ... }, ...)               # add computed attributes
@@ -536,12 +537,54 @@ probably never be.
    building an AST internally first. Strictly speaking this makes Bmg less
    powerful than Alf since optimizations cannot be turned off for now.
 
+## Stability and versioning
+
+As of 1.0, Bmg follows [semantic versioning](https://semver.org). Breaking
+changes to the API described below only happen on a major version.
+
+The following is public, and covered by that promise:
+
+* the factory methods on `Bmg` and `Bmg::Relation` (`in_memory`, `sequel`,
+  `csv`, `json`, `yaml`, `excel`, `text_file`, `generate`, `mutable`, ...);
+* the relational operators listed under [Supported
+  operators](#supported-operators), their arguments and their documented
+  options;
+* the predicates and summarizers listed under [Supported
+  predicates](#supported-predicates) and [Supported
+  summaries](#supported-summaries);
+* `Bmg::Relation` as an interface to include, with `each` yielding symbolized
+  tuples, as shown in [Your own relations](#your-own-relations);
+* `Bmg::Type`, `Bmg::Database`, and the reader/writer output preferences.
+
+The following is explicitly *not* covered, and may change in a minor version:
+
+* the underscored optimization hooks (`_restrict`, `_project`, ...). They are
+  documented in [Your own relations](#your-own-relations) because adapters
+  need them, but the protocol evolves as new optimizations are added. If you
+  override them, pin a minor version and read the CHANGELOG.
+* the shape of the ASTs returned by `to_ast`, and the SQL actually generated
+  for a given query tree. Both are observable, and both are expected to
+  change as the compiler improves; only the *semantics* of a relation is
+  guaranteed.
+* anything under `Bmg::Sql`, `Bmg::Sequel::Translator` and the sexpr grammar,
+  which are compiler internals.
+
 ## Contribute
 
 Please use github issues and pull requests for all questions, bug reports,
 and contributions. Don't hesitate to get in touch with us with an early code
 spike if you plan to add non trivial features.
 
+The `bench/` folder holds benchmarks for the in-memory operators and for the
+SQL compiler. Run one with `bundle exec ruby bench/operators.rb`. To compare
+two versions of the code, point `BMG_LIB` at another checkout and alternate
+between the two — benchmarking both in one batch, then the other, mostly
+measures how busy your machine was:
+
+```
+BMG_LIB=/path/to/other/lib bundle exec ruby bench/operators.rb
+```
+
 ## Licence
 
 This software is distributed by Enspirit SRL under a MIT Licence. Please

data/lib/bmg/operator/allbut.rb

@@ -103,7 +103,7 @@ module Bmg
     private
 
       def tuple_allbut(tuple)
-        TupleAlgebra.allbut(tuple, @butlist)
+        TupleAlgebra.allbut(tuple, @butlist_index ||= TupleAlgebra.membership(@butlist))
       end
 
       def valid_tuple!(tuple)

data/lib/bmg/operator/autosummarize.rb

@@ -5,9 +5,6 @@ module Bmg
     #
     # Autosummarize helps structuring the results of a big flat join.
     #
-    # This operator is still largely experimental and should be used with
-    # care...
-    #
     class Autosummarize
       include Operator::Unary
 
@@ -294,10 +291,11 @@ module Bmg
           v = v.reject{|tuple| tuple[@x].nil? }
           v = v.sort(&@sorter) if @sorter
           v.each do |tuple|
-            h[tuple[@x]] ||= []
-            h[tuple[@x]] << tuple[@y]
-            h[tuple[@x]].uniq!
+            (h[tuple[@x]] ||= []) << tuple[@y]
           end
+          # deduplicating once at the end, instead of on every append,
+          # keeps this linear in the number of observed `y`s
+          h.each_value(&:uniq!)
           h
         end
 

data/lib/bmg/operator/group.rb

@@ -34,6 +34,7 @@ module Bmg
       def each(&bl)
         return to_enum unless block_given?
         index = Hash.new{|h,k| h[k] = k.merge(as => empty_group) }
+        attrs = attrs_index
         operand.each do |tuple|
           key = TupleAlgebra.allbut(tuple, attrs)
           sub = TupleAlgebra.project(tuple, attrs)
@@ -76,6 +77,10 @@ module Bmg
 
     private
 
+      def attrs_index
+        @attrs_index ||= TupleAlgebra.membership(attrs)
+      end
+
       def empty_group
         Relation::InMemory.new(group_type, Set.new)
       end

data/lib/bmg/operator/image.rb

@@ -34,6 +34,8 @@ module Bmg
 
       }
 
+      EMPTY_BUTLIST = [].freeze
+
       def initialize(type, left, right, as, on, options = {})
         @type = type
         @left = left
@@ -91,8 +93,9 @@ module Bmg
         index = build_right_index(right_rel)
 
         # each left with image from right index
+        on = on_index
         left_rel.each do |tuple|
-          key = tuple_project(tuple, on)
+          key = TupleAlgebra.project(tuple, on)
           image = index[key] || (options[:array] ? [] : empty_image)
           yield tuple.merge(as => image)
         end
@@ -100,10 +103,11 @@ module Bmg
 
       def build_right_index(right)
         index = Hash.new{|h,k| h[k] = empty_image }
-        butlist = options[:preserve] ? [] : on
+        on = on_index
+        butlist = options[:preserve] ? EMPTY_BUTLIST : on
         right.each_with_object(index) do |t, index|
-          key = tuple_project(t, on)
-          index[key].operand << tuple_allbut(t, butlist)
+          key = TupleAlgebra.project(t, on)
+          index[key].operand << TupleAlgebra.allbut(t, butlist)
         end
         if opt = options[:array]
           sorter = to_sorter(opt)
@@ -252,12 +256,8 @@ module Bmg
 
     private
 
-      def tuple_project(tuple, on)
-        TupleAlgebra.project(tuple, on)
-      end
-
-      def tuple_allbut(tuple, butlist)
-        TupleAlgebra.allbut(tuple, butlist)
+      def on_index
+        @on_index ||= TupleAlgebra.membership(on)
       end
 
       def image_type

data/lib/bmg/operator/join.rb

@@ -27,13 +27,14 @@ module Bmg
       def each
         return to_enum unless block_given?
         index = Hash.new
+        on = on_index
         right.each_with_object(index) do |t, index|
-          key = tuple_project(t, on)
+          key = TupleAlgebra.project(t, on)
           index[key] ||= []
           index[key] << t
         end
         left.each do |tuple|
-          key = tuple_project(tuple, on)
+          key = TupleAlgebra.project(tuple, on)
           if to_join = index[key]
             to_join.each do |right|
               yield right.merge(tuple)
@@ -113,8 +114,8 @@ module Bmg
 
     private
 
-      def tuple_project(tuple, on)
-        TupleAlgebra.project(tuple, on)
+      def on_index
+        @on_index ||= TupleAlgebra.membership(on)
       end
 
     end # class Join

data/lib/bmg/operator/matching.rb

@@ -25,12 +25,13 @@ module Bmg
       def each
         return to_enum unless block_given?
         index = Hash.new
+        on = on_index
         right.each_with_object(index) do |t, index|
-          key = tuple_project(t, on)
+          key = TupleAlgebra.project(t, on)
           index[key] = true
         end
         left.each do |tuple|
-          key = tuple_project(tuple, on)
+          key = TupleAlgebra.project(tuple, on)
           yield tuple if index.has_key?(key)
         end
       end
@@ -73,8 +74,8 @@ module Bmg
 
     private
 
-      def tuple_project(tuple, on)
-        TupleAlgebra.project(tuple, on)
+      def on_index
+        @on_index ||= TupleAlgebra.membership(on)
       end
 
     end # class Matching

data/lib/bmg/operator/not_matching.rb

@@ -25,12 +25,13 @@ module Bmg
       def each
         return to_enum unless block_given?
         index = Hash.new
+        on = on_index
         right.each_with_object(index) do |t, index|
-          key = tuple_project(t, on)
+          key = TupleAlgebra.project(t, on)
           index[key] = true
         end
         left.each do |tuple|
-          key = tuple_project(tuple, on)
+          key = TupleAlgebra.project(tuple, on)
           yield tuple unless index.has_key?(key)
         end
       end
@@ -57,8 +58,8 @@ module Bmg
 
     private
 
-      def tuple_project(tuple, on)
-        TupleAlgebra.project(tuple, on)
+      def on_index
+        @on_index ||= TupleAlgebra.membership(on)
       end
 
     end # class NotMatching

data/lib/bmg/operator/project.rb

@@ -80,7 +80,8 @@ module Bmg
     private
 
       def tuple_project(tuple)
-        tuple.dup.delete_if{|k,_| !@attrlist.include?(k) }
+        attrs = (@attrs_index ||= TupleAlgebra.membership(@attrlist))
+        tuple.dup.delete_if{|k,_| !attrs.include?(k) }
       end
 
       def valid_tuple!(tuple)

data/lib/bmg/operator/summarize.rb

@@ -51,8 +51,9 @@ module Bmg
           }]
         }
         # iterate each tuple
+        by_index = TupleAlgebra.membership(@by)
         @operand.each do |tuple|
-          key = TupleAlgebra.project(tuple, @by)
+          key = TupleAlgebra.project(tuple, by_index)
           # apply them all and create a new memo
           result[key] = Hash[@summarization.map{|k,v|
             [ k, v.happens(result[key][k], tuple) ]

data/lib/bmg/sql/processor/join.rb

@@ -15,7 +15,7 @@ module Bmg
         def call(sexpr)
           if unjoinable?(sexpr)
             call(builder.from_self(sexpr))
-          elsif unjoinable?(right) || left_join_over_restricted_right?
+          elsif unjoinable?(right)
             Join.new(builder.from_self(right), on, options, builder).call(sexpr)
           else
             super(sexpr)
@@ -37,20 +37,6 @@ module Bmg
           sexpr.set_operator? or sexpr.limit_or_offset? or sexpr.group_by?
         end
 
-        # A restriction carried by the right operand of a left join may not be
-        # merged into the WHERE of the whole expression: there it would also
-        # have to hold of the tuples the left join fills with nulls, which is an
-        # inner join and loses exactly what the left join is for. Kept inside a
-        # derived table instead, where it applies to the right operand alone.
-        #
-        # The join condition would hold it just as well, and with less SQL, but
-        # it is the one place a restriction may be anything -- `>`, `IN`, a
-        # negation -- while the from clause orderer models a join condition as
-        # equalities between two qualified names.
-        def left_join_over_restricted_right?
-          left_join? && !right.select_exp.predicate.nil?
-        end
-
         def join_set_quantifiers(left, right)
           left_q, right_q = left.set_quantifier, right.set_quantifier
           left_q == right_q ? left_q : builder.distinct

data/lib/bmg/support/tuple_algebra.rb

@@ -1,6 +1,24 @@
 module Bmg
+  #
+  # Tuple-level counterparts of the relational operators.
+  #
   module TupleAlgebra
 
+    # `allbut` and `project` below test every attribute of every tuple for
+    # membership in an attribute list, so the cost of that lookup is paid
+    # `tuples x attributes` times. Converting the list to a Set is worth it,
+    # but only past a few elements: below that an Array scan wins, on every
+    # ruby (measured in bench/set_vs_array.rb). Join keys, in particular,
+    # are usually one or two attributes and are better left alone.
+    #
+    # Callers that iterate are expected to hoist this out of their loop.
+    SET_FROM = 3
+
+    def membership(attrlist)
+      attrlist.size < SET_FROM ? attrlist : attrlist.to_set
+    end
+    module_function :membership
+
     def allbut(tuple, butlist)
       tuple.reject{|k,v| butlist.include?(k) }
     end

data/lib/bmg/version.rb

@@ -1,8 +1,8 @@
 module Bmg
   module Version
-    MAJOR = 0
-    MINOR = 24
-    TINY  = 1
+    MAJOR = 1
+    MINOR = 0
+    TINY  = 0
   end
   VERSION = "#{Version::MAJOR}.#{Version::MINOR}.#{Version::TINY}"
 end

data/lib/bmg.rb

@@ -36,7 +36,4 @@ module Bmg
 
   # Add all factory methods
   extend Factory
-
-  # Deprecated
-  Leaf = Relation::InMemory
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: bmg
 version: !ruby/object:Gem::Version
-  version: 0.24.1
+  version: 1.0.0
 platform: ruby
 authors:
 - Bernard Lambeau
 bindir: bin
 cert_chain: []
-date: 2026-10-03 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: predicate
@@ -36,6 +36,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -43,6 +46,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -78,6 +84,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '4.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -85,6 +94,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '4.0'
 - !ruby/object:Gem::Dependency
   name: write_xlsx
   requirement: !ruby/object:Gem::Requirement
@@ -103,44 +115,90 @@ dependencies:
   name: sequel
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.0'
 - !ruby/object:Gem::Dependency
   name: sqlite3
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.4'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.4'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: benchmark-ips
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.14'
 description: Bmg is Alf's relational algebra for ruby, but much simpler and lighter
   than Alf itself
 email: blambeau@gmail.com
@@ -328,10 +386,14 @@ files:
 - lib/bmg/xlsx.rb
 - tasks/gem.rake
 - tasks/test.rake
-homepage: http://github.com/enspirit/bmg
+homepage: https://github.com/enspirit/bmg
 licenses:
 - MIT
-metadata: {}
+metadata:
+  source_code_uri: https://github.com/enspirit/bmg
+  changelog_uri: https://github.com/enspirit/bmg/blob/master/CHANGELOG.md
+  bug_tracker_uri: https://github.com/enspirit/bmg/issues
+  documentation_uri: https://www.relational-algebra.dev/
 rdoc_options: []
 require_paths:
 - lib
@@ -339,14 +401,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.2
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Bmg is Alf's successor.
 test_files: []