binder_core 0.1.0

data/.gitignore

@@ -0,0 +1,13 @@
+*.gem
+.bundle
+db/*.sqlite3*
+log/*.log
+*.log
+/tmp/
+doc/
+*.swp
+*~
+.project
+.DS_Store
+Gemfile.lock
+pkg/*

data/.rspec

@@ -0,0 +1,2 @@
+--colour
+--drb

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in binder_core.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2012 Ruaidhri Devery
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,86 @@
+Binder is an experimental framework for writing basic data compilers. Binder provides hooks for custom parsers which control how filesystem data is converted to native Ruby objects. 
+
+Binder began as an attempt to create a light weight filesystem based CMS that works with GIT. It progressed into a more generic framework to accommodate other ideas for how compiling data from the filesystem could be useful. The system is designed to make it as easy as possible to prototype those ideas.
+
+n.b. Please keep in mind, the term 'file' can refer to either a single file or a directory
+
+##How it works##
+The compiler takes care of traversing the files system, all you have to do is set rules for how it should parse specific kinds of files. You do this by defining a Parser. The definition requires two parameters, first a reference to your parser class and second a test block which will run against a file context. If the test block is run and returns true then your parser class is applied to the file and no further parser definitions are tested on that file. If no parser is matched then the file is effectively skipped. 
+
+Once a Parser has been applied its job is to obtain, format and output data from that file. If the parser does nothing, again the file is essentially skipped. Binder provides a small but powerful set of functionality to your parser. You have access to the file to read it into memory whatever way you like. 
+
+###descend###
+If the file is a directory you can instruct the compiler to step in by calling 'descend'. It returns the full output data from that operation.
+
+###create_asset###
+You can also choose to add a file to your output as a reference rather than actual data by calling 'new_asset'. 
+
+###add###
+Once you are done and have your output data you must 'add' it. From that point it is the responsibility of the the parent (folder) parser to pass that data down the chain.
+
+Binder comes with some very basic default parsers defined which you are free to extend. These handle reading utf text files, descending folders and adding images files as assets. You can/should be overriding these by adding your own.
+
+Tip: A parser that descends into a folder can itself create a whole set of local parser definitions only to be applied within the that folder and its children... powerful stuff ;)
+
+##Configure the compiler##
+You can configure the compiler with the following options:
+
+	BinderCore::Compiler.configure do
+	  config.content_folder =  "file/path/to/input/root/folder/"
+	  config.base_asset_folder =  "file/path/to/asset/output/folder/"
+	  config.base_asset_url = "/public/asset/base/path/"
+	  # size limit is a safety valve, 100 by default
+	  config.folder_size_limit_mb = 100
+	end
+
+##Define a Binder and add Parsers##
+Parsers are added with a test block. That test block is passed a FileContext during the compile process. If it returns true then this parser (and only this parser) is applied to that file.
+
+	BinderCore.define :my_feed do |config|
+	  config.add_parser MyCustomParsers::FeedRootFolderParser do |context|
+	    context.route == []
+	  end
+	end
+
+##Create a custom Parser##
+
+	module MyCustomParsers
+	  class FeedRootFolderParser < BinderCore::Parser
+	    # implement a parse method
+	    def parse
+	      # this is a directory so tell the compiler to descend into it
+	      # it returns the compiled data
+	      data = descend do |config|
+	        # you can add parsers within parsers
+	        # these are only applied to descendent files/folder
+	        config.add_parser MyCustomMarkdownParser do |context|
+	          context.file.ext.downcase == ".markdown"
+	        end
+	      end
+	      # add the data object to the feed
+	      add data
+	    end
+	  end
+	end
+
+##Assets##
+You can add a file as an asset to the data object. After compiling an optional asset linking phase will replace all the asset references with a url string. Here is how to add an asset in your parse method:
+
+	def parse
+	  add new_asset
+	end
+
+Your binder object provides you with an array of all the assets and you can handle those whatever way you want.
+
+##Compile a feed##
+
+	feed_binder = BinderCore.compile :my_feed
+	feed_binder.link_assets
+	data = feed_binder.data
+	assets = feed_binder.assets
+
+If you want more information about how this works, its all there in the source code. Any questions or feedback give me a shout.
+
+##Idea Chalkboard:##
+
+This is a TODO...

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/binder_core.gemspec

@@ -0,0 +1,24 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "binder_core/version"
+
+Gem::Specification.new do |s|
+  s.name        = "binder_core"
+  s.version     = BinderCore::VERSION
+  s.authors     = ["Ruaidhri Devery"]
+  s.email       = ["info@fluid.ie"]
+  s.homepage    = "https://github.com/rur/Binder-Core"
+  s.summary     = %q{Folder content compiler}
+  s.description = %q{Ruby framework for compiling data from a file system}
+
+  s.rubyforge_project = "binder_core"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # specify any dependencies here; for example:
+  s.add_development_dependency "rspec"
+  # s.add_runtime_dependency "rest-client"
+end

data/lib/binder_core/asset.rb

@@ -0,0 +1,45 @@
+module BinderCore
+  class Asset
+    attr_accessor :file, :name, :base_url, :asset_url, :route
+    
+    def initialize(path)
+      @file   = BinderCore::FileRef.new path
+      self.name = file.name
+      @route = []
+      @base_url = ""
+    end
+    
+    def with_name(value)
+      self.name = value
+      self
+    end
+    
+    def name=(value)
+      raise ArgumentError, "Asset name for '#{file.path}' cannot be empty" if value.empty?
+      @name = value
+    end
+    
+    def feed_path
+      [ *route, name ].join("/")
+    end
+    
+    def set_route(route)
+      @route = route
+      @route.freeze
+      refresh_asset_url
+      route
+    end
+    
+    def base_url=(url)
+      @base_url = url || ""
+      refresh_asset_url
+      url
+    end
+    
+    private
+    
+    def refresh_asset_url
+      @asset_url = File.join( @base_url, [*@route,@file.full_name].join("/"))
+    end
+  end
+end

data/lib/binder_core/binder.rb

@@ -0,0 +1,26 @@
+module BinderCore
+  class Binder
+    attr_reader :name, :config, :console
+    attr_accessor :raw
+    
+    def initialize(name, user_config = nil)
+      @name = name.is_a?(Symbol) ? name : name.to_s
+      @config = lambda do |config|
+        user_config.call(config) if user_config
+      end  
+    end
+    
+    def link_assets
+      data    = @raw[:data]||{}
+      assets  = @raw[:assets] || []
+      data    = Compiler.link_assets(data)
+      
+      {:data => data, :assets => assets.dup }
+    end
+    
+    # :NODOC:
+    def init_console(console)
+      @console = console
+    end
+  end
+end

data/lib/binder_core/compiler.rb

@@ -0,0 +1,97 @@
+module BinderCore
+  class Compiler
+    # Compile a configured Binder object data by scanning files and folders using parsers
+    def self.compile(binder)
+      binder.init_console Console.new Settings.propegate
+      # assemble pre, default and user config procs into single compiler config proc
+      pre_config   = lambda do |config|
+        config.set_console binder.console
+        config.set_path File.join( binder.console.defaults( :content_folder ), binder.name.to_s )
+        config.set_route []
+        config.set_name binder.name
+      end
+      default_config    = BinderCore::Default.config
+      user_config       = binder.config
+      compiler_config   = lambda do |config|
+        pre_config.call(config)
+        default_config.call(config)
+        user_config.call(config)
+        config.verify_context
+      end
+      # use that compiler proc to trigger the parsing process
+      compiled = Scanner.scan compiler_config
+      binder.raw = { :data => compiled.data, :assets => compiled.assets  }
+    end
+    
+    # Replace asset object values in the supplied data collection with their asset url
+    def self.link_assets(raw_data)
+       linker raw_data
+    end
+    
+    # Configure the global compiler settings, all Binder compiler
+    # process will inherit from these settings.
+    # 
+    # note: If you want to apply compiler settings on an individual Binder Definition
+    # you can do so in the BinderCore#define method using the config object
+    def self.configure(&block)
+      Default.settings
+      Settings.configure &block
+    end
+    
+    private
+    
+    def self.linker(collection) #:nodoc:
+      dest = collection.dup
+      itr = Iterator.new dest
+      while itr.has_next?
+         value = itr.next
+         if value.is_a? Asset
+           itr.replace value.asset_url
+         elsif value.is_a? Array or value.is_a? Hash
+           itr.replace linker value
+         end
+      end
+      dest
+    end
+    
+    class Iterator #:nodoc:
+      def initialize(collection=nil)
+        init collection unless collection.nil?
+      end
+      
+      def init(collection)
+        @collection = collection
+        @keys = []
+        @que = []
+        @step = -1
+        
+        if collection.is_a? Array
+          collection.length.times { |n| @keys << n }
+          @que = collection.dup
+        elsif collection.is_a? Hash
+          collection.keys.each do |key|
+            @keys << key
+            @que << collection[key]
+          end
+        end
+      end
+      
+      def has_next?
+        @step < count-1
+      end
+      
+      def count
+        @que.length
+      end
+      
+      def next
+        @step += 1
+        @que[@step]
+      end
+      
+      def replace(new_val)
+        @collection[@keys[@step]] = new_val
+      end
+    end
+  end
+end

data/lib/binder_core/config.rb

@@ -0,0 +1,98 @@
+module BinderCore
+  # This is used to configure both file and folder context objects, interchangeably
+  class Config
+    def initialize(context)
+      @context = context
+    end
+    
+    def set_console(console)
+      @context.console = console
+      @console = console
+    end
+    
+    def set_path(path)
+      @context.file = BinderCore::FileRef.new path
+    end
+    
+    def set_rules(rules)
+      @context.rules += rules
+    end
+    
+    def set_parsers(parsers)
+      @context.parsers += parsers
+    end
+    
+    def add_parser( klass, &test )
+      defn = BinderCore::Definition.new(klass, test)
+      @context.parsers.unshift(defn)
+      defn
+    end
+        
+    def add_rule( klass, &test )
+      defn = BinderCore::Definition.new(klass, test)
+      @context.rules.unshift(defn)
+      defn
+    end
+    
+    def set_name(name)
+      @context.name = name if(@context.respond_to?(:name))
+    end
+    
+    def set_route(route)
+      @context.route = route
+    end
+    
+    def set_data(data)
+      @context.data = data
+    end
+    
+    def set_assets(assets)
+      @context.assets = assets
+    end
+    
+    def add_params(params)
+      @context.params.merge!(params)
+    end
+    
+    def params
+      @context.params
+    end
+    
+    def verify_context
+      unless valid_file?(@context.file.path) then raise InvalidContextError, "Invalid file at path:#{@context.file.path}" end
+    end
+    
+    private
+    
+    def valid_file?(path)
+      size_limit_mb = @console.defaults(:folder_size_limit_mb)
+      size_limit_bt = size_limit_mb.to_f*1024*1024
+      
+      if      !File.exists?(path) then error_message = "File path:#{path} does not exist"
+      elsif   !File.readable?(path) then error_message = "You do not have permission to access the file at path:#{path}"
+      elsif   !file_within_size_limit? size_limit_bt, path then error_message = "File contents at path:#{path} is larger then the limit of #{size_limit_mb}MB"
+      end
+      
+      return true if error_message.nil?
+      # otherwise there was an error so...
+      @console.error error_message
+      false
+    end
+    
+    def file_within_size_limit?( limit, path )
+      def calculate_size(limit, file)
+        if File.directory? file
+          i = 0
+          Dir.open(file).each do |f|
+            i += calculate_size(limit, file + "/" + f).to_i unless f == '.' || f == '..'
+            break if i > limit
+          end
+          return i
+        else
+          return File.size?(file)
+        end
+      end
+      calculate_size(limit, path) < limit
+    end
+  end
+end

data/lib/binder_core/console.rb

@@ -0,0 +1,63 @@
+require "binder_core/debug/stack"
+
+module BinderCore
+  # TODO: Document this
+  class Console
+    attr_reader :stack
+    
+    def initialize(settings)
+      @settings = settings
+      @continue = true
+      @stack = Debug::Stack.new
+    end
+    
+    def defaults(param)
+      if param.is_a? String or param.is_a? Symbol then
+        return @settings.send(param) if @settings.prop_list.include? param.to_sym
+      end
+      param.keys.each do |key|
+        @settings.send "#{key}=", param[key] if @settings.prop_list.include? "#{key}=".to_sym
+      end
+    end
+    
+    def log(message = nil)
+      @log ||= []
+      return @log if message.nil?
+      @log.insert 0, message
+    end
+    
+    def warn(message)
+      self.warnings.insert 0, message
+    end
+    
+    def warnings
+      @warnings ||= []
+    end
+    
+    def error(message)
+      self.errors.insert 0, :message => message, :stack => stack.trace 
+      halt
+    end
+    
+    def errors
+      @errors ||= []
+    end
+    
+    def continue?
+      @continue
+    end
+    
+    #:NODOC: 
+    # Possible future feature: Figure out a way to allow the compilation process to be restarted
+    # This might be useful for some kind of debug-mode
+    # def continue
+    #   @continue = true
+    # end
+    
+    # Stop the compilation process
+    # NB: it cannot be restarted
+    def halt
+      @continue = false
+    end
+  end
+end

data/lib/binder_core/debug/stack.rb

@@ -0,0 +1,86 @@
+# This handles a stack record during the compile phase
+module BinderCore
+  module Debug
+    class Stack
+      attr_reader :route
+      
+      def initialize
+        @entry_hash = {}
+        @route = []
+      end
+      
+      # FIXME: it will not properly register the root file context and parser
+      
+      def add_folder(route,folder)
+        entry = create_entry route, folder.file.path, folder
+        @entry_hash[route.join("~")] = [entry]
+        self.route = route
+        entry
+      end
+      
+      def add_file(route,file)
+        entry = create_entry route, file.file.path, file
+        @entry_hash[route.join("~")] ||= []
+        @entry_hash[route.join("~")].insert 1, entry
+        self.route = route
+        entry
+      end
+      
+      def add_parser(route,parser)
+        entry = create_entry route, parser.file.path, parser
+        @entry_hash[route.join("~")] ||= []
+        @entry_hash[route.join("~")].insert 2, entry
+        self.route = route
+        entry
+      end
+      
+      def trace
+        output = []
+        tmp = []
+        route.each do |step|
+          tmp << step
+          ky = tmp.join("~").to_s
+          output.concat @entry_hash[ky]
+        end
+        output.reverse
+      end
+      
+      private
+      def route=(route)
+        prune_entry_hash route
+        @route = route.dup
+        @route.freeze 
+      end
+      
+      def create_entry(route,path,inst) #:NODOC:
+        entry = Entry.new
+        entry.route = route.dup
+        entry.file_path = path
+        entry.class_name = inst.class.to_s
+        entry.freeze
+        entry
+      end
+      
+      def prune_entry_hash( route ) #:NODOC: to hide my shame ;)
+        pruned_hash = {}
+        tmp = []
+        route.each do |step|
+          tmp << step
+          ky = tmp.join("~").to_s
+          pruned_hash[ky] = @entry_hash[ky]
+        end
+        @entry_hash = pruned_hash
+      end
+      
+      # This is created for every entry in the stack
+      # It holds the feed route, the file or folder path 
+      # and the class that is dealing with it when the entry is created
+      class Entry
+        attr_accessor :route, :file_path, :class_name
+        def to_s
+          return "#{@class_name} at route:(#{@route.join(',')}) to file:#{@file_path}"
+        end
+      end
+    end
+  end
+end

data/lib/binder_core/defaults.rb

@@ -0,0 +1,42 @@
+require "binder_core/parsers/null_parser"
+require "binder_core/parsers/text_parser"
+require "binder_core/parsers/folder_parser"
+require "binder_core/parsers/asset_parser"
+
+module BinderCore
+  module Default
+    def self.settings
+      Settings.configure do
+        config.base_asset_folder = ""
+        config.base_asset_url = ""
+        config.content_folder = ""
+        config.folder_size_limit_mb = 100
+      end
+    end
+    # this is the default config proc
+    def self.config
+      lambda do |config|
+        config.add_parser BinderCore::NullParser do |cxt|
+          true
+        end
+
+        config.add_parser BinderCore::TextParser do |cxt|
+          %w{.txt .html .htm .json}.include? cxt.file.ext.downcase
+        end
+
+        config.add_parser BinderCore::FolderParser do |cxt|
+          cxt.file.dir?
+        end
+      
+        config.add_parser BinderCore::AssetParser do |cxt|
+          %w{.jpg .jpeg .png .gif}.include? cxt.file.ext.downcase
+        end
+        
+        config.add_rule BinderCore::NullParser do |cxt|
+          cxt.file.name.start_with?(".","_")
+        end
+      end
+    end
+  end
+  
+end

data/lib/binder_core/definition.rb

@@ -0,0 +1,12 @@
+module BinderCore
+  class Definition
+    def initialize klass, test
+      @klass = klass
+      @test = test
+    end
+    
+    def test(context)
+      return @klass if @test.call(context) == true
+    end
+  end
+end

data/lib/binder_core/file_context.rb

@@ -0,0 +1,46 @@
+module BinderCore
+  # this object is passed to parser definition test proc
+  # it is then wrapped by the obtained parser instance
+  class FileContext
+    
+    attr_accessor :name, :file, :data, :parsers, :rules, :route, :assets, :console, :params
+     
+     def initialize
+       @rules    = []
+       @parsers  = []
+       @route    = []
+       @assets   = []
+       @params   = {}
+     end
+     
+     def file=(file)
+      @file = file
+      @name = file.name
+     end
+     
+    def register_asset( new_asset )
+      new_asset.route = @route.dup
+      
+      new_asset.base_url = console.defaults( :base_asset_url ) 
+      
+      @assets << new_asset
+      new_asset
+    end
+    
+    def descend(&user_config)
+      feed_config = lambda{ |config|
+        config.set_rules    @rules
+        config.set_parsers  @parsers
+        config.set_assets   @assets
+        config.set_console  console
+        config.add_params   @params.dup
+        config.set_path     file.path
+        config.set_route    route.dup << self.name
+        user_config.call    config
+        config.verify_context
+      }
+      
+      BinderCore::Scanner.descend( feed_config )
+    end
+  end
+end