alf 0.9.2 → 0.9.3

data/CHANGELOG.md

@@ -1,3 +1,92 @@
+# 0.9.3 / FIX ME
+
+* New operators (available both in shell and in Lispy DSL)
+
+  * Added MATCHING and NOT MATCHING operators. These operators are useful 
+    shortcuts for the following expressions.
+    
+        (matching     l, r) := (project (join l, r), [l's attributes])
+        (not_matching l, r) := (minus l, (matching l, r))
+      
+    For example: 
+    
+        # Give suppliers that supply at least one part
+        (matching suppliers, supplies)
+
+        # Give suppliers that don't supply any part
+        (not_matching suppliers, supplies)
+
+  * Added RANK operator. The RANK operator is useful for for computing quota 
+    queries, as shown below. See 'alf help rank' for details.
+    
+        # Give the three heaviest parts
+        (allbut (restrict (rank :parts, [[:weight, :desc]], :pos), lambda{ pos < 3 }), [:pos])
+
+* Enhancements when using Alf in shell
+
+  * added alf's -r option, that mimics ruby's one (require library before run)
+
+  * When alf is invoked in shell using bin/alf (and only in this case), 
+    ENV['ALF_OPTS'] is used a global options to apply as they were specified
+    inline: 
+    
+        % export ALF_OPTS="--env=. --yaml" 
+        % alf show suppliers
+        
+    is the same as 
+    
+        % alf --env=. --yaml show suppliers
+        
+  * 'alf --help' now distinguish experimental operators from those coming from
+    the (much more stable) TUTORIAL D specification. The former should be used
+    with care as they specification may change at any time.  
+
+* Enhancements when using Alf in Ruby 
+
+  * Alf.lispy now accepts any argument recognized by Environment.autodetect; it
+    obtains its working Environment this way. Among others:
+    
+        Alf.lispy(Alf::Environment.folder("path/to/an/existing/folder"))
+        
+    is the same as: 
+    
+        Alf.lispy("path/to/an/existing/folder")
+
+  * Added Relation::DUM and Relation::DEE constants (relations of empty heading
+    with no and one tuple, respectively). They are also available as DUM and DEE 
+    in Lispy functional expressions.
+  
+  * Added a Heading abstraction, as a set of attribute (name, type) pairs
+
+* Internal enhancements (extension points)
+
+  * The Reader and Renderer classes now accept a Hash of options as third 
+    argument of the constructor (friendly varargs applies there). These options
+    can be used by extension points. 
+    
+  * The Environment class now provides a class-based registering mechanism 'ala'
+    Reader and Renderer. This allows auto-detecting the target environment when
+    --env=... is used in shell. See Environment.autodetect and 
+    Environment#recognizes? for contributing to this extension point. 
+    
+  * Internals now rely on Myrrha for code generation. This means that all 
+    datatypes can now be safely used in relations and dumped to .rash files in
+    particular.  
+
+* Bug fixes
+
+  * Added Relation#allbut, forgotten in two previous releases 
+  * Fixed (join xxx, DEE) and (join xxx, DUM)
+  * Fixed scoping bug when using attributes named :path, :expr or :block in
+    Lispy compiled expressions (coming from .alf files)
+  * Fixed 'alf --yaml show suppliers' that renderer a --text table instead of
+    a yaml output
+  * Fixed bugs when using Date and Time attributes with .rash files
+  * Fixed bugs when using Date and Time attributes in restrict expressions 
+    compiled from the commandline
+  * Fixed a few bugs when using attribute names that are ruby keywords 
+    (restrict & extend)  
+
 # 0.9.2 / 2011.07.13
 
 * Bug fixes

data/Gemfile.lock

@@ -1,7 +1,8 @@
 PATH
   remote: .
   specs:
-    alf (0.9.2)
+    alf (0.9.3)
+      myrrha (~> 1.0.0)
       quickl (~> 0.2.2)
 
 GEM
@@ -10,12 +11,15 @@ GEM
     bluecloth (2.0.11)
     diff-lcs (1.1.2)
     highline (1.6.2)
+    myrrha (1.0.0)
     noe (1.3.0)
       highline (~> 1.6.0)
       quickl (~> 0.2.0)
       wlang (~> 0.10.1)
     quickl (0.2.2)
     rake (0.9.2)
+    rcov (0.9.9)
+    rcov (0.9.9-java)
     rspec (2.6.0)
       rspec-core (~> 2.6.0)
       rspec-expectations (~> 2.6.0)
@@ -37,6 +41,7 @@ DEPENDENCIES
   bundler (~> 1.0)
   noe (~> 1.3.0)
   rake (~> 0.9.2)
+  rcov (~> 0.9.9)
   rspec (~> 2.6.0)
   wlang (~> 0.10.1)
   yard (~> 0.7.2)

data/README.md

@@ -1,4 +1,4 @@
-# Alf - Relational Algebra at your fingertips (version 0.9.1)
+# Alf - Relational Algebra at your fingertips (version 0.9.3)
 
 ## Description
 
@@ -15,6 +15,11 @@ _relations_... Let's stop the segregation ;-)
     % [sudo] gem install alf
     % alf --help
 
+### Bundler & Require 
+
+    # API is not considered stable enough for now, please use
+    gem "alf", "= 0.9.2"
+
 ### Links
     
 * {http://rubydoc.info/github/blambeau/alf/master/frames} (read this file there!)
@@ -31,7 +36,7 @@ of a truly relational algebra approach. Objectives behind Alf are manifold:
   as (the physical encoding of) a relation**. See 'alf --help' for the list of 
   available commands and implemented relational operators.
   
-      % alf restrict suppliers -- "city == 'London'" | alf join cities 
+        % alf restrict suppliers -- "city == 'London'" | alf join cities 
   
 * Alf is also a 100% Ruby relational algebra implementation shipped with a simple 
   to use, powerful, functional DSL for compiling and evaluating relational queries. 
@@ -40,9 +45,9 @@ of a truly relational algebra approach. Objectives behind Alf are manifold:
   section). See 'alf --help' as well as .alf files in the examples directory 
   for syntactic examples.
   
-      Alf.lispy.evaluate { 
-        (join (restrict :suppliers, lambda{ city == 'London' }), :cities)
-      }
+        Alf.lispy.evaluate { 
+          (join (restrict :suppliers, lambda{ city == 'London' }), :cities)
+        }
       
   In addition to this functional syntax, Alf comes bundled with an in-memory 
   Relation data structure that provides an object-oriented way of manipulating
@@ -71,7 +76,7 @@ of a truly relational algebra approach. Objectives behind Alf are manifold:
   the following query for the kind of things that you'll never ever have in SQL 
   (see also 'alf help quota', 'alf help wrap', 'alf help group', ...):
   
-      % alf --text summarize supplies --by=sid -- total "sum(:qty)" -- which "group(:pid)"
+        % alf --text summarize supplies --by=sid -- total "sum(:qty)" -- which "group(:pid)"
   
 * Last, but not least, Alf is an attempt to help me test some research ideas and 
   communicate about them with people that already know (all or part) of the TTM 
@@ -82,7 +87,7 @@ of a truly relational algebra approach. Objectives behind Alf are manifold:
   'research work in progress', and used with care because not necessarily in 
   conformance with the TTM.
   
-      % alf --text quota supplies --by=sid --order=qty -- pos "count()"  
+        % alf --text quota supplies --by=sid --order=qty -- pos "count()"  
 
 ## Overview of relational theory
 
@@ -695,8 +700,7 @@ An environment built that way will look for .rash and .alf files in the specifie
 folder and sub-folders. I'll of course strongly consider any contribution 
 implementing the Environment contract on top of SQL or NoSQL databases or anything 
 that can be useful to manipulate with relational algebra. Such contributions can 
-be added to the project directly, in the lib/alf/environment folder, for example. 
-A base template would look like:
+be added to the project directly. A base template would look like:
 
     class Foo < Alf::Environment
     
@@ -710,6 +714,9 @@ A base template would look like:
     
     end
 
+Read more about Environment's API so as to let your environment be recognized
+in shell (--env=...) on rubydoc.info
+
 ### Adding file decoders, aka Readers
 
 Environments should not be confused with Readers (see Reader class and its 
@@ -837,18 +844,25 @@ as my own wish list, while I would love hearing yours instead.
 ### Versioning policy
 
 Alf respects {http://semver.org/ semantic versioning}, which means that it has
-a X.Y.Z version number and follows a few rules:
-
-- The public API is made of both the commandline tool as well as the Lispy 
-  dialect and will become stable with version 1.0.0 in a near future.
-- Backward compatible bug fixes will increase Z.  
-- New features and enhancements that do not break backward compatibility of the 
-  public API will increase the Y number.
-- Non backward compatible changes of the public API will increase the X number.
-
-All classes and modules but the Alf module itself and the Lispy DSL are part of
-the private API and may change at any time. A best-effort strategy is followed
-to avoid breaking internals on tiny (Z) version increases. 
+a X.Y.Z version number and follows a few rules.
+
+- The public API is made of the commandline tool, the Lispy dialect and the 
+  Relation datastructure. This API will become stable with version 1.0.0 in a 
+  near future.
+- Currently, version 1.0.0 **has not been reached**. It means that **anything 
+  may change at any time**. Best effort will be done to upgrade Y when backward 
+  incompatible changes occur.
+- Once 1.0.0 will be reached, the following rules will be followed:
+  - Backward compatible bug fixes will increase Z.  
+  - New features and enhancements that do not break backward compatibility of 
+    the public API will increase the Y number.
+  - Non backward compatible changes of the public API will increase the X 
+    number.
+
+All classes and modules but Alf module, the Lispy DSL and Alf::Relation are part 
+of the private API and may change at any time. A best-effort strategy is followed
+to avoid breaking internals on tiny (Z) version increases, especially extension
+points like Reader and Renderer. 
 
 ## Enjoy Alf!
 

data/TODO.md

@@ -18,9 +18,4 @@
   
 * Add PIVOT and UNPIVOT operators
 
-* Add MATCHING, NOT_MATCHING
-
-* Add a to_ruby abstraction and replace inspect usages in TupleHandle and
-  Rash renderer
-
 * Find a way to complete the description of Quota...  

data/alf.gemspec

@@ -126,11 +126,13 @@ Gem::Specification.new do |s|
   s.add_development_dependency("rake", "~> 0.9.2")
   s.add_development_dependency("bundler", "~> 1.0")
   s.add_development_dependency("rspec", "~> 2.6.0")
+  s.add_development_dependency("rcov", "~> 0.9.9")
   s.add_development_dependency("yard", "~> 0.7.2")
   s.add_development_dependency("bluecloth", "~> 2.0.9")
   s.add_development_dependency("wlang", "~> 0.10.1")
   s.add_development_dependency("noe", "~> 1.3.0")
   s.add_dependency("quickl", "~> 0.2.2")
+  s.add_dependency("myrrha", "~> 1.0.0")
 
   # The version of ruby required by this gem
   #

data/alf.noespec

@@ -1,13 +1,16 @@
 template-info:
   name: "ruby"
   version: 1.3.0
+  manifest:
+    tasks/unit_test.rake: 
+      safe-override: false
 variables:
   lower: 
     alf
   upper:
     Alf
   version:
-    0.9.2
+    0.9.3
   summary: |-
     Relational Algebra at your fingertips
   description: |-
@@ -26,11 +29,10 @@ variables:
     - {name: rake,         version: "~> 0.9.2",  groups: [development]}
     - {name: bundler,      version: "~> 1.0",    groups: [development]}
     - {name: rspec,        version: "~> 2.6.0",  groups: [development]}
+    - {name: rcov,         version: "~> 0.9.9",  groups: [development]}
     - {name: yard,         version: "~> 0.7.2",  groups: [development]}
     - {name: bluecloth,    version: "~> 2.0.9",  groups: [development]}
     - {name: wlang,        version: "~> 0.10.1", groups: [development]}
     - {name: noe,          version: "~> 1.3.0",  groups: [development]}
     - {name: quickl,       version: "~> 0.2.2",  groups: [runtime]}
-  rake_tasks:
-    spec_test:
-      pattern: "spec/**/test_*.rb"
+    - {name: myrrha,       version: "~> 1.0.0",  groups: [runtime]}    

data/bin/alf

@@ -1,9 +1,13 @@
 #!/usr/bin/env ruby
 module AlfLauncher
 
+  def self.lib
+    File.expand_path('../../lib', __FILE__)
+  end
+  
   def self.load
     begin
-      $LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
+      $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 
       require "alf"
     rescue LoadError => ex
       require "rubygems"
@@ -11,20 +15,12 @@ module AlfLauncher
     end
   end
   
-  def self.normalize(args)
-    opts = []
-    while !args.empty? && (args.first =~ /^\-/)
-      opts << args.shift 
-    end
-    if args.empty? or (args.size == 1 && File.exists?(args.first))
-      opts << "exec"
-    end
-    opts += args
-  end
-  
   def self.start(argv)
     load
-    Alf::Command::Main.run(normalize(argv), __FILE__)
+    if ENV["ALF_OPTS"]
+      argv = Alf::Tools::parse_commandline_args(ENV["ALF_OPTS"]) + argv
+    end
+    Alf::Command::Main.run(argv, __FILE__)
   end
 
 end # module AlfLaucher

data/examples/{database.alf → operators/database.alf}

@@ -1,4 +1,4 @@
-[{
+Alf::Relation[{
   :suppliers => (dataset :suppliers),
   :parts     => (dataset :parts),
   :cities    => (dataset :cities),

data/examples/operators/matching.alf

@@ -0,0 +1,2 @@
+#!/usr/bin/env alf
+(matching :suppliers, :supplies)

data/examples/operators/not_matching.alf

@@ -0,0 +1,2 @@
+#!/usr/bin/env alf
+(not_matching :suppliers, :supplies)

data/examples/operators/rank.alf

@@ -0,0 +1,4 @@
+#!/usr/bin/env alf
+
+# Get the three heaviest parts
+(allbut (restrict (rank :parts, [[:weight, :desc]], :pos), lambda{ pos < 3 }), [:pos])

data/lib/alf.rb

@@ -1,8 +1,12 @@
+require "alf/version"
+require "alf/loader"
+
 require "enumerator"
 require "stringio"
 require "set"
-require "alf/version"
-require "alf/loader"
+
+require 'myrrha/to_ruby_literal'
+require 'myrrha/coerce'
 
 #
 # Classy data-manipulation dressed in a DSL (+ commandline)
@@ -15,6 +19,72 @@ module Alf
   module Tools
     
     #
+    # Parse a string with commandline arguments and returns an array.
+    #
+    # Example:
+    # 
+    #   parse_commandline_args("--text --size=10") # => ['--text', '--size=10']
+    #
+    def parse_commandline_args(args)
+      args = args.split(/\s+/)
+      result = []
+      until args.empty?
+        if args.first[0,1] == '"'
+          if args.first[-1,1] == '"'
+            result << args.shift[1...-1]
+          else
+            block = [ args.shift[1..-1] ]
+            while args.first[-1,1] != '"'
+              block << args.shift
+            end 
+            block << args.shift[0...-1]
+            result << block.join(" ")
+          end
+        elsif args.first[0,1] == "'"
+          if args.first[-1,1] == "'"
+            result << args.shift[1...-1]
+          else
+            block = [ args.shift[1..-1] ]
+            while args.first[-1,1] != "'"
+              block << args.shift
+            end 
+            block << args.shift[0...-1]
+            result << block.join(" ")
+          end
+        else
+          result << args.shift
+        end  
+      end
+      result
+    end
+
+    # Helper to define methods with multiple signatures. 
+    #
+    # Example:
+    #
+    #   varargs([1, "hello"], [Integer, String]) # => [1, "hello"]
+    #   varargs(["hello"],    [Integer, String]) # => [nil, "hello"]
+    # 
+    def varargs(args, types)
+      types.collect{|t| t===args.first ? args.shift : nil}
+    end
+    
+    #
+    # Attempt to require(who) the most friendly way as possible.
+    #
+    def friendly_require(who, dep = nil, retried = false)
+      gem(who, dep) if dep && defined?(Gem)
+      require who
+    rescue LoadError => ex
+      if retried
+        raise "Unable to require #{who}, which is now needed\n"\
+              "Try 'gem install #{who}'"
+      else
+        require 'rubygems' unless defined?(Gem)
+        friendly_require(who, dep, true)
+      end
+    end
+
     # Returns the unqualified name of a ruby class or module
     #
     # Example
@@ -99,9 +169,8 @@ module Alf
           if expr.empty?
             compile(nil)
           else
-            # TODO: replace inspect by to_ruby
-            compile expr.each_pair.collect{|k,v| 
-              "(#{k} == #{v.inspect})"
+            compile expr.each_pair.collect{|k,v|
+              "(self.#{k} == #{Myrrha.to_ruby_literal(v)})"
             }.join(" && ")
           end
         when Array
@@ -207,13 +276,31 @@ module Alf
         @sorter = nil
       end
     
+      # 
+      # Coerces `arg` to an ordering key. 
+      #
+      # Implemented coercions are:
+      # * Array of symbols (all attributes in ascending order)
+      # * Array of [Symbol, :asc|:desc] pairs (obvious semantics)
+      # * ProjectionKey (all its attributes in ascending order)
+      # * OrderingKey (self)
+      #
+      # @return [OrderingKey]
+      # @raises [ArgumentError] when `arg` is not recognized
+      #
       def self.coerce(arg)
         case arg
           when Array
-            if arg.all?{|a| a.is_a?(Symbol)}
-              arg = arg.collect{|a| [a, :asc]}
+            if arg.all?{|a| a.is_a?(Array)}
+              OrderingKey.new(arg)
+            elsif arg.all?{|a| a.is_a?(Symbol)}
+              sliced = arg.each_slice(2) 
+              if sliced.all?{|a,o| [:asc,:desc].include?(o)}
+                OrderingKey.new sliced.to_a
+              else
+                OrderingKey.new arg.collect{|a| [a, :asc]}
+              end
             end
-            OrderingKey.new(arg)
           when ProjectionKey
             arg.to_ordering_key
           when OrderingKey
@@ -261,26 +348,6 @@ module Alf
     extend Tools
   end # module Tools
   
-  #
-  # Builds and returns a lispy engine on a specific environment.
-  #
-  # Example(s):
-  #
-  #   # Returns a lispy instance on the default environment
-  #   lispy = Alf.lispy
-  #
-  #   # Returns a lispy instance on the examples' environment
-  #   lispy = Alf.lispy(Alf::Environment.examples)
-  #
-  #   # Returns a lispy instance on a folder environment of your choice
-  #   lispy = Alf.lispy(Alf::Environment.folder('path/to/a/folder'))
-  #
-  # @see Alf::Environment about available environments and their contract
-  #
-  def self.lispy(env = Alf::Environment.default)
-    Command::Main.new(env)
-  end
-
   #
   # Encapsulates the interface with the outside world, providing base iterators
   # for named datasets, among others.
@@ -301,9 +368,81 @@ module Alf
   # You can implement your own environment by subclassing this class and 
   # implementing the {#dataset} method. As additional support is implemented 
   # in the base class, Environment should never be mimiced.
+  #
+  # This class provides an extension point allowing to participate to auto 
+  # detection and resolving of the --env=... option when alf is used in shell.
+  # See Environment.register, Environment.autodetect and Environment.recognizes?
+  # for details. 
   # 
   class Environment
     
+    # Registered environments
+    @@environments = []
+    
+    #
+    # Register an environment class under a specific name. 
+    #
+    # Registered class must implement a recognizes? method that takes an array
+    # of arguments; it must returns true if an environment instance can be built
+    # using those arguments, false otherwise. Please be very specific in the 
+    # implementation for returning true. See also autodetect and recognizes?
+    #
+    # @param [Symbol] name name of the environment kind
+    # @param [Class] clazz class that implemented the environment
+    #
+    def self.register(name, clazz)
+      @@environments << [name, clazz]
+      (class << self; self; end).
+        send(:define_method, name) do |*args|
+          clazz.new(*args)
+        end
+    end
+    
+    #
+    # Auto-detect the environment to use for specific arguments.
+    #
+    # This method returns an instance of the first registered Environment class 
+    # that returns true to an invocation of recognizes?(args). It raises an 
+    # ArgumentError if no such class can be found.    
+    #
+    # @return [Environment] an environment instance
+    # @raise [ArgumentError] when no registered class recognizes the arguments
+    #
+    def self.autodetect(*args)
+      if (args.size == 1) && args.first.is_a?(Environment)
+        return args.first
+      else
+        @@environments.each do |name,clazz|
+          return clazz.new(*args) if clazz.recognizes?(args)
+        end
+      end
+      raise ArgumentError, "Unable to auto-detect Environment with #{args.inspect}"
+    end
+    
+    #
+    # (see Environment.autodetect)
+    #
+    def self.coerce(*args)
+      autodetect(*args)
+    end
+    
+    #
+    # Returns true _args_ can be used for building an environment instance,
+    # false otherwise.
+    #
+    # When returning true, an immediate invocation of new(*args) should 
+    # succeed. While runtime exception are admitted (no such database, for 
+    # example), argument errors should not occur (missing argument, wrong 
+    # typing, etc.).
+    #
+    # Please be specific in the implementation of this extension point, as 
+    # registered environments for a chain and each of them should have a 
+    # chance of being selected.
+    #
+    def self.recognizes?(args)
+      false
+    end
+    
     #
     # Returns a dataset whose name is provided.
     #
@@ -378,6 +517,18 @@ module Alf
     #
     class Folder < Environment
       
+      # 
+      # (see Environment.recognizes?)
+      #
+      # Returns true if args contains onely a String which is an existing
+      # folder.
+      #
+      def self.recognizes?(args)
+        (args.size == 1) && 
+        args.first.is_a?(String) && 
+        File.directory?(args.first.to_s)
+      end
+      
       #
       # Creates an environment instance, wired to the specified folder.
       #
@@ -412,15 +563,9 @@ module Alf
         end
       end
       
+      Environment.register(:folder, self)
     end # class Folder
     
-    #
-    # Factors a Folder environment on a specific path
-    #
-    def self.folder(path)
-      Folder.new(path)
-    end
-    
     #
     # Returns the default environment
     #
@@ -432,7 +577,7 @@ module Alf
     # Returns the examples environment
     #
     def self.examples
-      folder File.expand_path('../../examples', __FILE__)
+      folder File.expand_path('../../examples/operators', __FILE__)
     end
     
   end # class Environment
@@ -572,8 +717,10 @@ module Alf
         else
           raise "No registered reader for #{ext} (#{filepath})"
         end
-      else
+      elsif args.empty? 
         coerce(filepath)
+      else
+        raise ArgumentError, "Unable to return a reader for #{filepath} and #{args}" 
       end
     end
     
@@ -603,21 +750,33 @@ module Alf
       end
     end
     
+    # Default reader options
+    DEFAULT_OPTIONS = {}
+    
     # @return [Environment] Wired environment 
     attr_accessor :environment
   
     # @return [String or IO] Input IO, or file name
     attr_accessor :input
+    
+    # @return [Hash] Reader's options
+    attr_accessor :options
   
     #
-    # Creates a reader instance, with an optional input and environment wiring.
+    # Creates a reader instance. 
     #
     # @param [String or IO] path to a file or IO object for input
     # @param [Environment] environment wired environment, serving this reader
+    # @param [Hash] options Reader's options (see doc of subclasses) 
     #
-    def initialize(input = nil, environment = nil)
-      @input = input
-      @environment = environment 
+    def initialize(*args)
+      @input, @environment, @options = case args.first
+      when String, IO, StringIO
+        Tools.varargs(args, [args.first.class, Environment, Hash])
+      else
+        Tools.varargs(args, [String, Environment, Hash])
+      end
+      @options = self.class.const_get(:DEFAULT_OPTIONS).merge(@options || {}) 
     end
   
     #
@@ -821,17 +980,33 @@ module Alf
       @@renderers.each(&Proc.new)
     end
     
+    # Default renderer options
+    DEFAULT_OPTIONS = {}
+
     # Renderer input (typically an Iterator)
     attr_accessor :input
     
     # @return [Environment] Optional wired environment
     attr_accessor :environment
 
+    # @return [Hash] Renderer's options
+    attr_accessor :options
+    
     #
-    # Creates a renderer instance, optionally wired to an input
+    # Creates a reader instance. 
     #
-    def initialize(input = nil)
-      @input = input
+    # @param [Iterator] iterator an Iterator of tuples to render
+    # @param [Environment] environment wired environment, serving this reader
+    # @param [Hash] options Reader's options (see doc of subclasses) 
+    #
+    def initialize(*args)
+      @input, @environment, @options = case args.first
+      when Array
+        Tools.varargs(args, [Array, Environment, Hash])
+      else
+        Tools.varargs(args, [Iterator, Environment, Hash])
+      end
+      @options = self.class.const_get(:DEFAULT_OPTIONS).merge(@options || {}) 
     end
     
     # 
@@ -877,7 +1052,7 @@ module Alf
       # (see Renderer#render)
       def render(input, output)
         input.each do |tuple|
-          output << tuple.inspect << "\n"
+          output << Myrrha.to_ruby_literal(tuple) << "\n"
         end
         output
       end
@@ -885,8 +1060,6 @@ module Alf
       Renderer.register(:rash, "as ruby hashes", self)
     end # class Rash
 
-    require "alf/renderer/text"
-    require "alf/renderer/yaml"
   end # module Renderer
 
   #
@@ -937,7 +1110,14 @@ module Alf
     #
     # RELATIONAL COMMANDS
     # #{summarized_subcommands subcommands.select{|cmd| 
-    #    cmd.include?(Alf::Operator::Relational)
+    #    cmd.include?(Alf::Operator::Relational) &&
+    #    !cmd.include?(Alf::Operator::Experimental)
+    # }}
+    #
+    # EXPERIMENTAL OPERATORS
+    # #{summarized_subcommands subcommands.select{|cmd| 
+    #    cmd.include?(Alf::Operator::Relational) &&
+    #    cmd.include?(Alf::Operator::Experimental)
     # }}
     #
     # NON-RELATIONAL COMMANDS
@@ -964,7 +1144,6 @@ module Alf
       # Creates a command instance
       def initialize(env = Environment.default)
         @environment = env
-        extend(Lispy)
       end
       
       # Install options
@@ -974,16 +1153,20 @@ module Alf
           @execute = true
         end
         
-        @renderer = Renderer::Rash.new
+        @renderer = nil
         Renderer.each_renderer do |name,descr,clazz|
           opt.on("--#{name}", "Render output #{descr}"){ 
             @renderer = clazz.new 
           }
         end
         
-        opt.on('--env=FOLDER', 
-               "Set the environment folder to use") do |value|
-          @environment = Environment.folder(value)
+        opt.on('--env=ENV', 
+               "Set the environment to use") do |value|
+          @environment = Environment.autodetect(value)
+        end
+        
+        opt.on('-rlibrary', "require the library, before executing alf") do |value|
+          require(value)
         end
         
         opt.on_tail('-h', "--help", "Show help") do
@@ -991,16 +1174,29 @@ module Alf
         end
         
         opt.on_tail('-v', "--version", "Show version") do
-          raise Quickl::Exit, "#{program_name} #{Alf::VERSION}"\
+          raise Quickl::Exit, "alf #{Alf::VERSION}"\
                               " (c) 2011, Bernard Lambeau"
         end
       end # Alf's options
       
+      #
+      def _normalize(args)
+        opts = []
+        while !args.empty? && (args.first =~ /^\-/)
+          opts << args.shift 
+        end
+        if args.empty? or (args.size == 1 && File.exists?(args.first))
+          opts << "exec"
+        end
+        opts += args
+      end
+      
       #
       # Overrided because Quickl only keep --options but modifying it there 
       # should probably be considered a broken API.
       #
       def _run(argv = [])
+        argv = _normalize(argv)
         
         # 1) Extract my options and parse them
         my_argv = []
@@ -1011,7 +1207,7 @@ module Alf
         
         # 2) build the operator according to -e option
         operator = if @execute
-          instance_eval(argv.first)
+          Alf.lispy(environment).compile(argv.first)
         else
           super
         end
@@ -1019,6 +1215,7 @@ module Alf
         # 3) if there is a requester, then we do the job (assuming bin/alf)
         # with the renderer to use. Otherwise, we simply return built operator
         if operator && requester
+          renderer = self.renderer ||= Renderer::Rash.new
           renderer.pipe(operator, environment).execute($stdout)
         else
           operator
@@ -1048,7 +1245,7 @@ module Alf
       include Command
     
       options do |opt|
-        @renderer = Renderer::Text.new
+        @renderer = nil
         Renderer.each_renderer do |name,descr,clazz|
           opt.on("--#{name}", "Render output #{descr}"){ 
             @renderer = clazz.new 
@@ -1057,7 +1254,7 @@ module Alf
       end
         
       def execute(args)
-        requester.renderer = @renderer
+        requester.renderer = (@renderer || requester.renderer || Text::Renderer.new)
         args = [ $stdin ] if args.empty?
         args.first
       end
@@ -1456,6 +1653,9 @@ module Alf
       
     end # module Shortcut
 
+    # Marker for experimental operators
+    module Experimental; end
+    
   end # module Operator
 
   #
@@ -2089,40 +2289,80 @@ module Alf
     class Join < Factory::Operator(__FILE__, __LINE__)
       include Operator::Relational, Operator::Shortcut, Operator::Binary
       
+      #
+      # Performs a Join of two relations through a Hash buffer on the right
+      # one.
+      #
       class HashBased
         include Operator::Binary
       
+        #
+        # Implements a special Buffer for join-based relational operators.
+        #
+        # Example:
+        #
+        #   buffer = Buffer::Join.new(...) # pass the right part of the join
+        #   left.each do |left_tuple|
+        #     key, rest = buffer.split(tuple)
+        #     buffer.each(key) do |right_tuple|
+        #       #
+        #       # do whatever you want with left and right tuples
+        #       #
+        #     end
+        #   end 
+        #
         class JoinBuffer
           
+          #
+          # Creates a buffer instance with the right part of the join.
+          #
+          # @param [Iterator] enum a tuple iterator, right part of the join. 
+          #
           def initialize(enum)
             @buffer = nil
             @key = nil
             @enum = enum
           end
           
+          #
+          # Splits a left tuple according to the common key.
+          #
+          # @param [Hash] tuple a left tuple of the join
+          # @return [Array] an array of two elements, the key and the rest
+          # @see ProjectionKey#split
+          #
           def split(tuple)
             _init(tuple) unless @key
             @key.split(tuple)
           end
           
+          #
+          # Yields each right tuple that matches a given key value.
+          #
+          # @param [Hash] key a tuple that matches elements of the common key
+          #        (typically the first element returned by #split) 
+          #
           def each(key)
             @buffer[key].each(&Proc.new) if @buffer.has_key?(key)
           end
           
           private
           
+          # Initialize the buffer with a right tuple
           def _init(right)
             @buffer = Hash.new{|h,k| h[k] = []}
             @enum.each do |left|
               @key = Tools::ProjectionKey.coerce(left.keys & right.keys) unless @key
               @buffer[@key.project(left)] << left
             end
+            @key = Tools::ProjectionKey.coerce([]) unless @key
           end
           
-        end
+        end # class JoinBuffer
         
         protected
         
+        # (see Operator#_each)
         def _each
           buffer = JoinBuffer.new(right)
           left.each do |left_tuple|
@@ -2291,6 +2531,128 @@ module Alf
       end
       
     end # class Union
+
+    # 
+    # Relational matching
+    #
+    # SYNOPSIS
+    #   #{program_name} #{command_name} [LEFT] RIGHT
+    #
+    # API & EXAMPLE
+    #
+    #   (matching :suppliers, :supplies)
+    #
+    # DESCRIPTION
+    #
+    # This operator restricts left tuples to those for which there exists at 
+    # least one right tuple that joins. This is a shortcut operator for the
+    # longer expression:
+    #
+    #   (project (join xxx, yyy), [xxx's attributes])
+    #
+    # In shell:
+    #
+    #   alf matching suppliers supplies 
+    #  
+    class Matching < Factory::Operator(__FILE__, __LINE__)
+      include Operator::Relational, Operator::Shortcut, Operator::Binary
+      
+      #
+      # Performs a Matching of two relations through a Hash buffer on the right
+      # one.
+      #
+      class HashBased
+        include Operator::Binary
+      
+        # (see Operator#_each)
+        def _each
+          seen, key = nil, nil
+          left.each do |left_tuple|
+            seen ||= begin
+              h = Hash.new
+              right.each do |right_tuple|
+                key ||= Tools::ProjectionKey.coerce(left_tuple.keys & right_tuple.keys)
+                h[key.project(right_tuple)] = true
+              end
+              key ||= Tools::ProjectionKey.coerce([])
+              h
+            end
+            yield(left_tuple) if seen.has_key?(key.project(left_tuple))
+          end
+        end
+        
+      end # class HashBased
+      
+      protected
+      
+      # (see Shortcut#longexpr)
+      def longexpr
+        chain HashBased.new,
+              datasets 
+      end
+      
+    end # class Matching
+        
+    # 
+    # Relational not matching
+    #
+    # SYNOPSIS
+    #   #{program_name} #{command_name} [LEFT] RIGHT
+    #
+    # API & EXAMPLE
+    #
+    #   (not_matching :suppliers, :supplies)
+    #
+    # DESCRIPTION
+    #
+    # This operator restricts left tuples to those for which there does not 
+    # exist any right tuple that joins. This is a shortcut operator for the
+    # longer expression: 
+    #
+    #         (minus xxx, (matching xxx, yyy))
+    # 
+    # In shell:
+    #
+    #   alf not-matching suppliers supplies 
+    #  
+    class NotMatching < Factory::Operator(__FILE__, __LINE__)
+      include Operator::Relational, Operator::Shortcut, Operator::Binary
+      
+      #
+      # Performs a NotMatching of two relations through a Hash buffer on the 
+      # right one.
+      #
+      class HashBased
+        include Operator::Binary
+      
+        # (see Operator#_each)
+        def _each
+          seen, key = nil, nil
+          left.each do |left_tuple|
+            seen ||= begin
+              h = Hash.new
+              right.each do |right_tuple|
+                key ||= Tools::ProjectionKey.coerce(left_tuple.keys & right_tuple.keys)
+                h[key.project(right_tuple)] = true
+              end
+              key ||= Tools::ProjectionKey.coerce([])
+              h
+            end
+            yield(left_tuple) unless seen.has_key?(key.project(left_tuple))
+          end
+        end
+        
+      end # class HashBased
+      
+      protected
+      
+      # (see Shortcut#longexpr)
+      def longexpr
+        chain HashBased.new,
+              datasets 
+      end
+      
+    end # class NotMatching
     
     # 
     # Relational wraping (tuple-valued attributes)
@@ -2670,6 +3032,117 @@ module Alf
   
     end # class Summarize
   
+    # 
+    # Relational ranking (explicit tuple positions)
+    #
+    # SYNOPSIS
+    #   #{program_name} #{command_name} [OPERAND] --order=OR1... -- [RANKNAME]
+    #
+    # OPTIONS
+    # #{summarized_options}
+    #
+    # API & EXAMPLE
+    #
+    #   # Position attribute => # of tuples with smaller weight 
+    #   (rank :parts, [:weight], :position)
+    #    
+    #   # Position attribute => # of tuples with greater weight 
+    #   (rank :parts, [[:weight, :desc]], :position)
+    #
+    # DESCRIPTION
+    #
+    # This operator computes the ranking of input tuples, according to an order
+    # relation. Precisely, it extends the input tuples with a RANKNAME attribute
+    # whose value is the number of tuples which are considered strictly less
+    # according to the specified order. For the two examples above:
+    #
+    #   alf rank parts --order=weight -- position
+    #   alf rank parts --order=weight,desc -- position
+    #
+    # Note that, unless the ordering key includes a candidate key for the input
+    # relation, the newly RANKNAME attribute is not necessarily a candidate key
+    # for the output one. In the example above, adding the :pid attribute 
+    # ensured that position will contain all different values: 
+    #
+    #   alf rank parts --order=weight,pid -- position
+    # 
+    # Or even:
+    #
+    #   alf rank parts --order=weight,desc,pid,asc -- position
+    #
+    class Rank < Factory::Operator(__FILE__, __LINE__)
+      include Operator::Relational, Operator::Shortcut, Operator::Unary
+  
+      # Ranking order
+      attr_accessor :order
+      
+      # Ranking attribute name
+      attr_accessor :ranking_name
+  
+      def initialize(order = [], ranking_name = :rank)
+        @order, @ranking_name = order, ranking_name
+      end
+  
+      options do |opt|
+        opt.on('--order=x,y,z', 'Specify ranking order', Array) do |args|
+          @order = args.collect{|a| a.to_sym}
+        end
+      end
+  
+      class SortBased
+        include Operator::Cesure
+        
+        def initialize(order, ranking_name)
+          @order, @ranking_name = order, ranking_name
+        end
+        
+        def ordering_key
+          OrderingKey.coerce @order
+        end
+        
+        def cesure_key
+          ProjectionKey.coerce(ordering_key)
+        end
+    
+        def start_cesure(key, receiver)
+          @rank ||= 0
+          @last_block = 0
+        end
+  
+        def accumulate_cesure(tuple, receiver)
+          receiver.call tuple.merge(@ranking_name => @rank)
+          @last_block += 1
+        end
+        
+        def flush_cesure(key, receiver)
+          @rank += @last_block
+        end
+  
+      end # class SortBased
+  
+      protected
+      
+      # (see Operator::CommandMethods#set_args)
+      def set_args(args)
+        unless args.empty?
+          self.ranking_name = args.first.to_sym
+        end
+        self
+      end
+  
+      def ordering_key
+        OrderingKey.coerce @order
+      end
+  
+      def longexpr
+        sort_key = ordering_key
+        chain SortBased.new(sort_key, @ranking_name),
+              Operator::NonRelational::Sort.new(sort_key),
+              datasets
+      end 
+  
+    end # class Rank
+    
     # 
     # Relational quota-queries (position, sum progression, etc.)
     #
@@ -2692,7 +3165,8 @@ module Alf
     #   alf quota supplies --by=sid --order=qty -- position count sum_qty "sum(:qty)"
     #
     class Quota < Factory::Operator(__FILE__, __LINE__)
-      include Operator::Relational, Operator::Shortcut, Operator::Unary
+      include Operator::Relational, Operator::Experimental,
+              Operator::Shortcut, Operator::Unary
   
       # Quota by
       attr_accessor :by
@@ -2991,24 +3465,42 @@ module Alf
     # 
     # Keeps tuples ordered on a specific key
     #
+    # Example:
+    #
+    #   sorted = Buffer::Sorted.new OrderingKey.new(...)
+    #   sorted.add_all(...)
+    #   sorted.each do |tuple|
+    #     # tuples are ordered here 
+    #   end
+    #
     class Sorted < Buffer
   
+      #
+      # Creates a buffer instance with an ordering key
+      #
       def initialize(ordering_key)
         @ordering_key = ordering_key
         @buffer = []
       end
   
+      #
+      # Adds all elements of an iterator to the buffer
+      #
       def add_all(enum)
         sorter = @ordering_key.sorter
         @buffer = merge_sort(@buffer, enum.to_a.sort(&sorter), sorter)
       end
   
+      #
+      # (see Buffer#each)
+      #
       def each
         @buffer.each(&Proc.new)
       end
   
       private
     
+      # Implements a merge sort between two iterators s1 and s2
       def merge_sort(s1, s2, sorter)
         (s1 + s2).sort(&sorter)
       end
@@ -3017,7 +3509,245 @@ module Alf
     
   end # class Buffer
 
-  # 
+  #
+  # Defines a Heading, that is, a set of attribute (name,domain) pairs.
+  #
+  class Heading
+    
+    #
+    # Creates a Heading instance
+    #
+    # @param [Hash] a hash of attribute (name, type) pairs where name is
+    #        a Symbol and type is a Class
+    #
+    def self.[](attributes)
+      Heading.new(attributes) 
+    end
+
+    # @return [Hash] a (freezed) hash of (name, type) pairs  
+    attr_reader :attributes
+    
+    #
+    # Creates a Heading instance
+    #
+    # @param [Hash] a hash of attribute (name, type) pairs where name is
+    #        a Symbol and type is a Class
+    #
+    def initialize(attributes)
+      @attributes = attributes.dup.freeze
+    end
+    
+    #
+    # Returns heading's cardinality
+    #
+    def cardinality
+      attributes.size
+    end
+    alias :size  :cardinality
+    alias :count :cardinality
+    
+    #
+    # Returns heading's hash code
+    # 
+    def hash
+      @hash ||= attributes.hash
+    end
+    
+    #
+    # Checks equality with other heading
+    #
+    def ==(other)
+      other.is_a?(Heading) && (other.attributes == attributes) 
+    end
+    alias :eql? :==
+    
+    #
+    # Converts this heading to a Hash of (name,type) pairs
+    # 
+    def to_hash
+      attributes.dup
+    end
+    
+    # 
+    # Returns a Heading literal
+    #
+    def to_ruby_literal
+      attributes.empty? ?
+        "Alf::Heading::EMPTY" :
+        "Alf::Heading[#{Myrrha.to_ruby_literal(attributes)[1...-1]}]"
+    end
+    alias :inspect :to_ruby_literal
+    
+    EMPTY = Alf::Heading.new({})
+  end # class Heading
+  
+  #
+  # Defines an in-memory relation data structure.
+  #
+  # A relation is a set of tuples; a tuple is a set of attribute (name, value)
+  # pairs. The class implements such a data structure with full relational
+  # algebra installed as instance methods.
+  #
+  # Relation values can be obtained in various ways, for example by invoking
+  # a relational operator on an existing relation. Relation literals are simply
+  # constructed as follows:
+  #
+  #     Alf::Relation[
+  #       # ... a comma list of ruby hashes ...
+  #     ]
+  #
+  # See main Alf documentation about relational operators.
+  #
+  class Relation
+    include Iterator
+    
+    protected
+    
+    # @return [Set] the set of tuples
+    attr_reader :tuples
+    
+    public
+    
+    #
+    # Creates a Relation instance.
+    #
+    # @param [Set] tuples a set of tuples
+    #
+    def initialize(tuples)
+      raise ArgumentError unless tuples.is_a?(Set)
+      @tuples = tuples
+    end
+    
+    #
+    # Coerces `val` to a relation.
+    #
+    # Recognized arguments are: Relation (identity coercion), Set of ruby hashes, 
+    # Array of ruby hashes, Alf::Iterator.
+    #
+    # @return [Relation] a relation instance for the given set of tuples
+    # @raise [ArgumentError] when `val` is not recognized
+    #
+    def self.coerce(val)
+      case val
+      when Relation
+        val
+      when Set
+        Relation.new(val)
+      when Array
+        Relation.new val.to_set
+      when Iterator
+        Relation.new val.to_set
+      else
+        raise ArgumentError, "Unable to coerce #{val} to a Relation"
+      end
+    end
+    
+    # (see Relation.coerce)
+    def self.[](*tuples)
+      coerce(tuples)
+    end
+    
+    #
+    # (see Iterator#each)
+    #
+    def each(&block)
+      tuples.each(&block)
+    end
+    
+    #
+    # Returns relation's cardinality (number of tuples).
+    #
+    # @return [Integer] relation's cardinality 
+    # 
+    def cardinality
+      tuples.size
+    end
+    alias :size :cardinality
+    alias :count :cardinality
+    
+    # Returns true if this relation is empty
+    def empty?
+      cardinality == 0
+    end
+    
+    # 
+    # Install the DSL through iteration over defined operators
+    #
+    Operator::each do |op_class|
+      meth_name = Tools.ruby_case(Tools.class_name(op_class)).to_sym
+      if op_class.unary?
+        define_method(meth_name) do |*args|
+          op = op_class.new(*args).pipe(self)
+          Relation.coerce(op)
+        end
+      elsif op_class.binary?
+        define_method(meth_name) do |right, *args|
+          op = op_class.new(*args).pipe([self, Iterator.coerce(right)])
+          Relation.coerce(op)
+        end
+      else
+        raise "Unexpected operator #{op_class}"
+      end
+    end # Operators::each
+      
+    alias :+ :union
+    alias :- :minus
+
+    # Shortcut for project(attributes, true)
+    def allbut(attributes)
+      project(attributes, true)
+    end
+        
+    #
+    # (see Object#hash)
+    #
+    def hash
+      @tuples.hash
+    end
+    
+    #
+    # (see Object#==)
+    #
+    def ==(other)
+      return nil unless other.is_a?(Relation)
+      other.tuples == self.tuples
+    end
+    alias :eql? :==
+    
+    #
+    # Returns a textual representation of this relation
+    #
+    def to_s
+      Alf::Renderer.text(self).execute("")
+    end
+    
+    #
+    # Returns an array with all tuples in this relation.
+    #
+    # @param [Tools::OrderingKey] an optional ordering key (any argument 
+    #        recognized by OrderingKey.coerce is supported here). 
+    # @return [Array] an array of hashes, in requested order (if specified)
+    #
+    def to_a(okey = nil)
+      okey = Tools::OrderingKey.coerce(okey) if okey
+      ary = tuples.to_a
+      ary.sort!(&okey.sorter) if okey
+      ary
+    end
+    
+    #
+    # Returns a  literal representation of this relation
+    #
+    def to_ruby_literal
+      "Alf::Relation[" +
+        tuples.collect{|t| Myrrha.to_ruby_literal(t)}.join(', ') + "]"
+    end
+    alias :inspect :to_ruby_literal
+  
+    DEE = Relation.coerce([{}])
+    DUM = Relation.coerce([])
+  end # class Relation
+
   # Implements a small LISP-like DSL on top of Alf.
   #
   # The lispy dialect is the functional one used in .alf files and in compiled
@@ -3061,7 +3791,8 @@ module Alf
       if expr.nil? 
         instance_eval(&block)
       else 
-        (path ? Kernel.eval(expr, binding, path) : Kernel.eval(expr, binding))
+        b = _clean_binding
+        (path ? Kernel.eval(expr, b, path) : Kernel.eval(expr, b))
       end
     end
   
@@ -3128,8 +3859,52 @@ module Alf
       (project child, attributes, true)
     end
   
+    # 
+    # Runs a command as in shell.
+    #
+    # Example:
+    #
+    #     lispy = Alf.lispy(Alf::Environment.examples)
+    #     op = lispy.run(['restrict', 'suppliers', '--', "city == 'Paris'"])
+    #
+    def run(argv, requester = nil)
+      Alf::Command::Main.new(environment).run(argv, requester)
+    end
+    
     Agg = Alf::Aggregator
+    DUM = Relation::DUM
+    DEE = Relation::DEE
+    
+    private 
+    
+    def _clean_binding
+      binding
+    end
+    
   end # module Lispy
 
+  #
+  # Builds and returns a lispy engine on a specific environment.
+  #
+  # Example(s):
+  #
+  #   # Returns a lispy instance on the default environment
+  #   lispy = Alf.lispy
+  #
+  #   # Returns a lispy instance on the examples' environment
+  #   lispy = Alf.lispy(Alf::Environment.examples)
+  #
+  #   # Returns a lispy instance on a folder environment of your choice
+  #   lispy = Alf.lispy(Alf::Environment.folder('path/to/a/folder'))
+  #
+  # @see Alf::Environment about available environments and their contract
+  #
+  def self.lispy(env = Environment.default)
+    lispy = Object.new.extend(Lispy)
+    lispy.environment = Environment.coerce(env)
+    lispy
+  end
+  
 end # module Alf
-require "alf/relation"
+require "alf/text"
+require "alf/yaml"