alf 0.9.3 → 0.10.0

data/CHANGELOG.md

@@ -1,164 +1,290 @@
-# 0.9.3 / FIX ME
+# 0.10.0 / 2011-08-15
 
-* New operators (available both in shell and in Lispy DSL)
+## New recognized data sources
 
-  * Added MATCHING and NOT MATCHING operators. These operators are useful 
-    shortcuts for the following expressions.
-    
-        (matching     l, r) := (project (join l, r), [l's attributes])
-        (not_matching l, r) := (minus l, (matching l, r))
+* Alf now provides an Environment implementation on top of a SQL database. 
+  This means that SQL tables can now be used as data-sources. This feature 
+  relies on the sequel gem ('gem install sequel' is required), that drives 
+  recognized SQL servers. Then (don't forget that ALF_OPTS also exists):
+
+      % alf --env=postgres://user:password@host/database show table
+
+* Alf now recognizes and allows manipulating .csv files as first-class data
+  sources. CSV output is also supported of course. Under ruby <= 1.9, the
+  fastercsv gem is required ('gem install fastercsv' is required). Then:
+
+      % alf restrict suppliers.csv -- "city == 'Paris'"    (input)
+      % alf show suppliers --csv                           (output)
+  
+* Alf now recognizes and allows manipulating .log files as first-class data
+  sources. This feature relies on request-log-analyzer gem that provides the
+  parsers that Alf uses, and the log formats it recognizes 
+  ('gem install request-log-analyzer' is required). See examples/logs.
+  
+## New operators and enhancements
+
+* A GENERATOR operator is introduced. It allows generating a relation with one
+  auto-number attribute, up to a given size.
+
+* A COERCE operator is introduced. It provides a quick way to obtain type-safe
+  relations from type-unsafe sources like .csv files. For example:
+  
+      % alf coerce mydirtyfile.csv -- name String  price Float  at Time
       
-    For example: 
-    
-        # Give suppliers that supply at least one part
-        (matching suppliers, supplies)
+  The coerce operator is of course available in Ruby as well:
+  
+      (coerce "mydirtyfile.csv", :name => String, :price => Float, :at => Time)
+      
+* The DEFAULTS (non-relational) operator now accepts default values as tuple
+  expressions. When used in shell, provided default values are now evaluated
+  that way. This allows specifying default values as being computed on the 
+  current tuple.
 
-        # Give suppliers that don't supply any part
-        (not_matching suppliers, supplies)
+* Aggregations in the Lispy DSL must not be prefixed by Agg:: anymore.
 
-  * Added RANK operator. The RANK operator is useful for for computing quota 
-    queries, as shown below. See 'alf help rank' for details.
-    
-        # Give the three heaviest parts
-        (allbut (restrict (rank :parts, [[:weight, :desc]], :pos), lambda{ pos < 3 }), [:pos])
+## Miscellaneous enhancements
 
-* Enhancements when using Alf in shell
+* Added 'alf --input-reader' to specify $stdin format (csv, rash, etc.)
+* Added 'alf -Ipath' that mimics ruby's -I (add path to $LOAD_PATH before run)
+* Lispy#run supports command arguments to be passed as a string
+* Lispy#run supports piped commands, with '|' as in shell
 
-  * added alf's -r option, that mimics ruby's one (require library before run)
+## Hurting changes to Lispy DSL (and therefore to Relation)
 
-  * When alf is invoked in shell using bin/alf (and only in this case), 
-    ENV['ALF_OPTS'] is used a global options to apply as they were specified
-    inline: 
-    
-        % export ALF_OPTS="--env=. --yaml" 
-        % alf show suppliers
-        
-    is the same as 
-    
-        % alf --env=. --yaml show suppliers
-        
-  * 'alf --help' now distinguish experimental operators from those coming from
-    the (much more stable) TUTORIAL D specification. The former should be used
-    with care as they specification may change at any time.  
+* The attribute-name syntax of aggregation operators has been removed. The Agg::
+  prefix must not be specified anymore.
 
-* Enhancements when using Alf in Ruby 
+      Agg::sum(:qty)    # !! error !!
+      Agg::sum{ qty }   # !! error !!
+      sum{ qty }        # simply, and only!
 
-  * Alf.lispy now accepts any argument recognized by Environment.autodetect; it
-    obtains its working Environment this way. Among others:
-    
-        Alf.lispy(Alf::Environment.folder("path/to/an/existing/folder"))
-        
-    is the same as: 
+* The group aggregation operator has been removed. It will probably be replaced 
+  in a future version. In the meantime, the GROUP relational operator allows 
+  obtaining similar results.
+
+* Lispy syntax of CLIP has changed (when used with --allbut option)
+
+      (clip :suppliers, [:name, :city], true)                           (before)
+      (clip :suppliers, [:name, :city], :allbut => true)                (after)
+
+* Lispy syntax of DEFAULTS has changed (when used with --strict option)
+
+      (defaults :suppliers, {:country => 'Belgium'}, true)              (before)
+      (defaults :suppliers, {:country => 'Belgium'}, :strict => true)   (after)
+
+* Lispy syntax of GROUP has changed (when used with --allbut option)
+
+      (group :supplies, [:sid], :supplying, true)                       (before)
+      (group :supplies, [:sid], :supplying, :allbut => true)            (after)
+
+* Lispy syntax of PROJECT has changed (when used with --allbut option)
+
+      (project :suppliers, [:name, :city], true)                        (before)
+      (project :suppliers, [:name, :city], :allbut => true)             (after)
+
+* Lispy syntax of SUMMARIZE has changed (when used with --allbut option)
+
+      (summarize :supplies, [:qty, :pid], {...}, true)                  (before)
+      (summarize :supplies, [:qty, :pid], {...}, :allbut => true)       (after)
+
+## Hurting changes in shell
+
+* The attribute-name syntax of aggregation operators has been removed
+
+      sum(:qty)   # !! error !!
+      sum{ qty }  # works
+
+* Shell syntax of GROUP has changed (option separator before introduced name)
+
+      % alf --text group supplies  -- pid qty supplying                 (before)
+      % alf --text group supplies  -- pid qty -- supplying              (after)
+
+* Shell syntax of WRAP has changed (option separator before introduced name)
+
+      % alf --text wrap suppliers -- city status loc_and_status         (before) 
+      % alf --text wrap suppliers -- city status -- loc_and_status      (after) 
+
+* Shell syntax of QUOTA has changed (--by and --order become pure arguments)
+
+      % alf quota supplies --by=sid --order=qty -- position count sum_qty "sum{ qty }" (before)
+      % alf quota supplies -- sid -- qty -- position count sum_qty "sum{ qty }"        (after)
+
+* Shell syntax of RANK has changed (--order becomes a pure argument)
+
+      % alf rank parts --order=weight,desc,pid,asc -- position          (before)
+      % alf rank parts -- weight desc pid asc -- position               (after)
+
+* Shell syntax of SUMMARIZE has changed (--by becomes a pure argument)
+
+      % alf summarize supplies --by=sid -- total_qty "sum{ qty }"        (before)
+      % alf summarize supplies -- sid -- total_qty "sum{ qty }"          (after)
+
+## Bug fixes 
+
+* [In shell] Options are now correctly parsed in presence of option separators. 
+  That is, every argument after a '--' separator is considered a non-option 
+  argument. 
+
+# 0.9.3 / 2011-07-23
+
+## New operators (available both in shell and in Lispy DSL)
+
+* Added MATCHING and NOT MATCHING operators. These operators are useful 
+  shortcuts for the following expressions.
+  
+      (matching     l, r) := (project (join l, r), [l's attributes])
+      (not_matching l, r) := (minus l, (matching l, r))
     
-        Alf.lispy("path/to/an/existing/folder")
+  For example: 
+  
+      # Give suppliers who supply at least one part
+      (matching suppliers, supplies)
 
-  * Added Relation::DUM and Relation::DEE constants (relations of empty heading
-    with no and one tuple, respectively). They are also available as DUM and DEE 
-    in Lispy functional expressions.
+      # Give suppliers who don't supply any part
+      (not_matching suppliers, supplies)
+
+* Added RANK operator, which is useful for for computing quota queries as 
+  illustrated below. See 'alf help rank' for details.
   
-  * Added a Heading abstraction, as a set of attribute (name, type) pairs
+      # Give the three heaviest parts
+      (restrict (rank :parts, [[:weight, :desc]], :pos), lambda{ pos < 3 })
 
-* Internal enhancements (extension points)
+## Enhancements when using Alf in shell
 
-  * The Reader and Renderer classes now accept a Hash of options as third 
-    argument of the constructor (friendly varargs applies there). These options
-    can be used by extension points. 
-    
-  * The Environment class now provides a class-based registering mechanism 'ala'
-    Reader and Renderer. This allows auto-detecting the target environment when
-    --env=... is used in shell. See Environment.autodetect and 
-    Environment#recognizes? for contributing to this extension point. 
-    
-  * Internals now rely on Myrrha for code generation. This means that all 
-    datatypes can now be safely used in relations and dumped to .rash files in
-    particular.  
-
-* Bug fixes
-
-  * Added Relation#allbut, forgotten in two previous releases 
-  * Fixed (join xxx, DEE) and (join xxx, DUM)
-  * Fixed scoping bug when using attributes named :path, :expr or :block in
-    Lispy compiled expressions (coming from .alf files)
-  * Fixed 'alf --yaml show suppliers' that renderer a --text table instead of
-    a yaml output
-  * Fixed bugs when using Date and Time attributes with .rash files
-  * Fixed bugs when using Date and Time attributes in restrict expressions 
-    compiled from the commandline
-  * Fixed a few bugs when using attribute names that are ruby keywords 
-    (restrict & extend)  
+* added 'alf -r', that mimics 'ruby -r' (require library before run)
+
+* When alf is invoked in shell (and only in this case), ALF_OPTS is used as 
+  global options to apply as if they were specified inline: 
+  
+      % export ALF_OPTS="--env=. --yaml" 
+      % alf show suppliers
+      
+  is the same as 
+  
+      % alf --env=. --yaml show suppliers
+      
+* 'alf --help' now distinguishes experimental operators (quota in particular) 
+  from those coming from the (much more stable) **Tutorial D** specification. The 
+  former should be used with care as their specification may change at any 
+  time.  
+
+## Enhancements when using Alf in Ruby 
+
+* Alf.lispy now accepts any argument recognized by Environment.autodetect; it
+  obtains its working Environment that way. Among others:
+  
+      Alf.lispy(Alf::Environment.folder("path/to/an/existing/folder"))
+      
+  is the same as: 
+  
+      Alf.lispy("path/to/an/existing/folder")
+
+* Added Relation::DUM and Relation::DEE constants (relations of empty heading
+  with no and one tuple, respectively). They are also available as DUM and DEE 
+  in Lispy functional expressions.
+
+* Added a Heading abstraction, as a set of attribute (name, type) pairs.
+
+## Internal enhancements (extension points)
+
+* The Reader and Renderer classes accept a Hash of options as third 
+  constructor argument. These options can be used by extension points. 
+  
+* The Environment class now provides a class-based registering mechanism 'ala'
+  Reader and Renderer. This allows auto-detecting the target environment when
+  --env=... is used in shell. See Environment.autodetect and 
+  Environment#recognizes? for contributing to this extension point. 
+  
+* Internals now rely on Myrrha for code generation. This means that all 
+  datatypes can now be safely used in relations and dumped to .rash files in
+  particular.  
+
+## Bug fixes
+
+* Added Relation#allbut, forgotten in two previous releases 
+* Fixed (join xxx, DEE) and (join xxx, DUM)
+* Fixed scoping bug when using attributes named :path, :expr or :block in
+  Lispy compiled expressions (coming from .alf files)
+* Fixed 'alf --yaml show suppliers' that renderer a --text table instead of
+  a yaml output
+* Fixed bugs when using Date and Time attributes with .rash files
+* Fixed bugs when using Date and Time attributes in restrict expressions 
+  compiled from the commandline
+* Fixed a few bugs when using attribute names that are ruby keywords 
+  (restrict & extend)  
 
 # 0.9.2 / 2011.07.13
 
-* Bug fixes
+# Bug fixes
 
-  * Fixed the "alf show" command (undefined method `chain') 
+* Fixed the "alf show" command (undefined method `chain') 
 
 # 0.9.1 / 2011.07.13
 
-* Enhancements (public APIs)
+## Enhancements (public APIs)
 
-  * Added the in-memory Alf::Relation data structure and associated tooling.
-    This allows using Alf in a object-oriented usual way, in addition to the
-    functional DSL:
-    
-        Alf.lispy.evaluate {
-          (join (restrict :suppliers, lambda{ status > 10 }), :cities)
-        }
-    
-    is equivalent to
-    
-        suppliers, cities = [...], [...] 
-        suppliers.restrict(lambda{ status > 10 }).join(cities)
-        
-    see README about how to obtain suppliers and cities relations in the first 
-    place.
+* Added the in-memory Alf::Relation data structure and associated tooling.
+  This allows using Alf in a object-oriented usual way, in addition to the
+  functional DSL:
+  
+      Alf.lispy.evaluate {
+        (join (restrict :suppliers, lambda{ status > 10 }), :cities)
+      }
   
-  * Summarize now accepts a --allbut option, to specify 'by' attributes from an
-    exclusion perspective
+  is equivalent to
+  
+      suppliers, cities = [...], [...] 
+      suppliers.restrict(lambda{ status > 10 }).join(cities)
+      
+  see README about how to obtain suppliers and cities relations in the first 
+  place.
 
-  * .alf files are now evaluated in such a way that backtraces are "traceability
-    friendly"
+* Summarize now accepts a --allbut option, to specify 'by' attributes from an
+  exclusion perspective
 
-* Non backward-compatible changes to public APIs
+* .alf files are now evaluated in such a way that backtraces are "traceability
+  friendly"
 
-  * Lispy#with has been removed because not being stable enough. The clean way 
-    of reusing sub-queries is as follows (non purely functional, so far)
-    
-        kept_suppliers = (restrict :suppliers, lambda{ status > 10 })
-        with_countries = (join kept_suppliers, :cities)
-        supplying      = (join with_countries, :supplies)
-        (summarize supplying,
-                   [:country],
-                   :which => Agg::group(:pid),
-                   :total => Agg::sum{ qty })
-                   
-  * As a consequence, named data sources (Symbols, like :suppliers above) are 
-    now resolved at compile time, which is less powerful, yet much simpler and
-    sound.
-
-  * Nest and Unnest have been renamed to Wrap and Unwrap respectively. This is
-    to better conform to TUTORIAL D's terminology.
-    
-  * Lispy#chain was kept public in 0.9.0 by error and has been entirely removed 
-    from the DSL.
+## Non backward-compatible changes to public APIs
+
+* Lispy#with has been removed because not being stable enough. The clean way 
+  of reusing sub-queries is as follows (non purely functional, so far)
+  
+      kept_suppliers = (restrict :suppliers, lambda{ status > 10 })
+      with_countries = (join kept_suppliers, :cities)
+      supplying      = (join with_countries, :supplies)
+      (summarize supplying,
+                 [:country],
+                 :which => Agg::group(:pid),
+                 :total => Agg::sum{ qty })
+                 
+* As a consequence, named data sources (Symbols, like :suppliers above) are 
+  now resolved at compile time, which is less powerful, yet much simpler and
+  sound.
 
-* Enhancements (internals)
+* Nest and Unnest have been renamed to Wrap and Unwrap respectively. This is
+  to better conform to **Tutorial D**'s terminology.
+  
+* Lispy#chain was kept public in 0.9.0 by error and has been entirely removed 
+  from the DSL.
 
-  * Reader.reader delegates to Reader.coerce when its first argument is not 
-    a String. This allows calling Reader.reader(args.first || $stdin) in quickl
-    commands for example.
-    
-  * Operator, Operator::Relational and Operator::NonRelational have a .each 
-    class method that yields operator classes
+## Enhancements (internals)
+
+* Reader.reader delegates to Reader.coerce when its first argument is not 
+  a String. This allows calling Reader.reader(args.first || $stdin) in quickl
+  commands for example.
+  
+* Operator, Operator::Relational and Operator::NonRelational have a .each 
+  class method that yields operator classes
     
-* Bug fixes
+## Bug fixes
 
-  * Fixed a bug that led to an Nil error when using unary operators on $stdin
-  * Fixed a bug when summarizing or sorting on Symbol attributes with ruby 1.8
-  * Fixed numerous crashes under rubinius
+* Fixed a bug that led to an Nil error when using unary operators on $stdin
+* Fixed a bug when summarizing or sorting on Symbol attributes with ruby 1.8
+* Fixed numerous crashes under rubinius
 
 # 0.9.0 / 2011.06.19
 
-* Enhancements
+## Enhancements
 
-  * Birthday!
+* Birthday!

data/Gemfile

@@ -1,2 +1,32 @@
 source 'http://rubygems.org'
-gemspec :name => "alf"
+
+group :runtime do
+  gem "quickl", "~> 0.4.0"
+  gem "myrrha", "~> 1.2.0"
+end
+
+group :test do
+  gem "rake", "~> 0.9.2"
+  gem "rspec", "~> 2.6.0"
+end
+
+group :release do
+  gem "rake", "~> 0.9.2"
+  gem "rspec", "~> 2.6.0"
+  gem "wlang", "~> 0.10.2"
+end
+
+group :doc do
+  gem "rcov", "~> 0.9.9"
+  gem "yard", "~> 0.7.2"
+  gem "bluecloth", "~> 2.1.0"
+  gem "redcarpet", "~> 1.17.2"
+end
+
+group :extra do
+  gem "fastercsv", "~> 1.5.4"
+  gem "request-log-analyzer", "~> 1.11.0"
+  gem "sequel", "~> 3.25.0"
+  gem "sqlite3", "~> 1.3.0"
+end
+

data/Gemfile.lock

@@ -1,25 +1,16 @@
-PATH
-  remote: .
-  specs:
-    alf (0.9.3)
-      myrrha (~> 1.0.0)
-      quickl (~> 0.2.2)
-
 GEM
   remote: http://rubygems.org/
   specs:
-    bluecloth (2.0.11)
+    bluecloth (2.1.0)
     diff-lcs (1.1.2)
-    highline (1.6.2)
-    myrrha (1.0.0)
-    noe (1.3.0)
-      highline (~> 1.6.0)
-      quickl (~> 0.2.0)
-      wlang (~> 0.10.1)
-    quickl (0.2.2)
+    fastercsv (1.5.4)
+    myrrha (1.2.0)
+    quickl (0.4.0)
     rake (0.9.2)
     rcov (0.9.9)
     rcov (0.9.9-java)
+    redcarpet (1.17.2)
+    request-log-analyzer (1.11.0)
     rspec (2.6.0)
       rspec-core (~> 2.6.0)
       rspec-expectations (~> 2.6.0)
@@ -28,6 +19,8 @@ GEM
     rspec-expectations (2.6.0)
       diff-lcs (~> 1.1.2)
     rspec-mocks (2.6.0)
+    sequel (3.25.0)
+    sqlite3 (1.3.3)
     wlang (0.10.2)
     yard (0.7.2)
 
@@ -36,12 +29,16 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
-  alf!
-  bluecloth (~> 2.0.9)
-  bundler (~> 1.0)
-  noe (~> 1.3.0)
+  bluecloth (~> 2.1.0)
+  fastercsv (~> 1.5.4)
+  myrrha (~> 1.2.0)
+  quickl (~> 0.4.0)
   rake (~> 0.9.2)
   rcov (~> 0.9.9)
+  redcarpet (~> 1.17.2)
+  request-log-analyzer (~> 1.11.0)
   rspec (~> 2.6.0)
-  wlang (~> 0.10.1)
+  sequel (~> 3.25.0)
+  sqlite3 (~> 1.3.0)
+  wlang (~> 0.10.2)
   yard (~> 0.7.2)

data/LICENCE.md

@@ -1,6 +1,6 @@
 # The MIT Licence
 
-    Copyright (c) 2011 - Bernard Lambeau
+    Copyright (c) 2011 - Bernard Lambeau & The University of Louvain (Belgium)
 
     Permission is hereby granted, free of charge, to any person obtaining
     a copy of this software and associated documentation files (the

data/Manifest.txt

@@ -1,4 +1,6 @@
 bin/**/*
+doc/commands/**/*
+doc/operators/**/*
 examples/**/*
 lib/**/*
 spec/**/*