rake-pipeline-fork 0.8.0

data/Rakefile

@@ -0,0 +1,21 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+
+directory "doc"
+
+desc "generate documentation"
+task :docs => Dir["lib/**"] do
+  sh "devbin/yard doc --readme README.yard --hide-void-return"
+end
+
+desc "generate a dependency graph from the documentation"
+task :graph => ["doc", :docs] do
+  sh "devbin/yard graph --dependencies | dot -Tpng -o doc/arch.png"
+end
+
+desc "run the specs"
+task :spec do
+  sh "rspec"
+end
+
+task :default => :spec

data/bin/rakep

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require "rake-pipeline"
+Rake::Pipeline::CLI.start

data/examples/copying_files.md

@@ -0,0 +1,12 @@
+The `copy` method also accepts an array. You can use this to copy the
+file into multiple locations. Here's an example `Assetfile`:
+
+```
+input "source" do
+  match "*.js" do
+    copy do |input|
+      ["/staging/#{input.path}", "/production/#{input.path}"]
+    end
+  end
+end
+```

data/examples/minifying_files.md

@@ -0,0 +1,37 @@
+Minifying files is a very common task for web developers. There are two
+common uses cases: 
+
+1. Create an unminified and minified file
+2. Generate a minified file from an unminifed file.
+
+Doing #2 is very easy with rake pipeline. Doing #1 is slightly more
+complicated. For these examples assume there is a `MinifyFilter` that
+actually does the work.
+
+Doing the first use case creates a problem. Filters are destructive.
+That means if you change one file (move it, rewrite) it will not be
+available for the next filters. For example, taking `app.js` and
+minifying it will remove the unminfied source from the pipeline. 
+
+You must first duplicate the unminified file then minify it. Here's an
+`Assetfile`
+
+```ruby
+input "source" do
+  match "application.js" do
+    # Duplicate the source file for future filters (application.js)
+    # and provide duplicate in "application.min.js" for minifcation
+    copy ["application.js", "application.min.js"]
+  end
+
+  match "application.min.js" do
+    filter MinifyFilter
+  end
+end
+
+output "compiled"
+```
+
+Now you have two files: `application.js` and `application.min.js`. You
+can use this same technique everytime you need to transform a file and
+keep the source around for future filters.

data/examples/modifying_pipelines.md

@@ -0,0 +1,67 @@
+You may want to do some trickery with your input files. Here is a use
+case: you need to sort your files in a custom way. The easiest way to do
+this is to insert a new pipeline into your pipeline. This pipeline must
+act like a filter because it will be used as such. Let's start out by
+describing the most basic pipeline:
+
+```ruby
+class PassThroughPipeline < Rake::Pipeline
+  # this need to act like a filter
+  attr_accessor :pipeline
+
+  # simply return the original input_files
+  def output_files
+    input_files
+  end
+
+  # this is very imporant! define this method
+  # to do nothing and files will not be copied 
+  # to the output directory
+  def finalize
+  end
+end
+```
+
+At this point you can insert it into your pipeline:
+
+```ruby
+input "**/*.js" do
+  # stick our pass through
+  pass_through = pipeline.copy PassThroughPipeline
+  pipeline.add_filter pass_through
+  
+  # now continue on with your life
+  concat "application.js"
+end
+```
+
+Now we can begin to do all sorts of crazyness in this pass through
+pipeline. You could expand directories to groups of files or you could
+collapse then. You could even skip files if you wanted to. Hell, you can
+even sort them--and that's what we're going to do. So let's get going
+
+```ruby
+class SortedPipeline < PassThroughPipeline
+  def output_files
+    super.sort do |f1, f2|
+      # just an easy example of reverse sorting
+      f2.fullpath <=> f1.fullpath
+    end
+  end
+end
+```
+
+Now add it to the pipeline:
+
+```ruby
+input "**/*.js" do
+  # stick our pass through
+  pass_through = pipeline.copy SortedPipeline
+  pipeline.add_filter pass_through
+
+  # now continue on with your life
+  concat "application.js"
+end
+```
+
+Voila! You can sort stuff. Let your mind run wild with possibilities!

data/examples/multiple_pipelines.md

@@ -0,0 +1,77 @@
+Your build process may become to complex to express in terms of a single
+`input` block. You can break your `Assetfile` into multiple input
+blocks. You can also use an `input`'s output as the input for a new
+`input` block.
+
+Say you have different pipelines for different types of files. There is
+one for JS, CSS, and other assets. The output of these 3 different build
+steps needs be the input for the next build step. You can express that
+rather easily with `Rake::Pipeline`. Here are the pipelines for the
+different types. The internals are left out because they are not
+relavant to this example.
+
+```ruby
+# Stage 1
+input "js" do
+  # do JS stuff
+end
+
+input "css" do
+  # do css stuff
+end
+
+input "assets" do
+  # do asset stuff
+end
+```
+
+Now let's add a line at the top of the `Assetfile`. Let's stay that all
+the pipelines should output into one directory.
+
+```ruby
+# All `input` blocks will output to `tmp/stage1` unless output is
+# set again
+output "tmp/stage1"
+
+input "js" do
+  # do JS stuff
+end
+
+input "css" do
+  # do css stuff
+end
+
+input "assets" do
+  # do asset stuff
+end
+```
+
+Now let's hookup stage 2.
+
+```ruby
+# Stage 1
+output "tmp/stage1"
+input "js" do
+  # do JS stuff
+end
+
+input "css" do
+  # do css stuff
+end
+
+input "assets" do
+  # do asset stuff
+end
+
+# Stage 2
+# output of next input block should go to the real output directory
+output "compiled"
+input "tmp/stage1" do
+  # do stage 2 stuff
+end
+```
+
+You can repeat this process over and over again for as many stages as
+you like. Just remember that the final input should output to where you
+want the final files. Also, keep the intermediate build steps inside
+temp directories so they are ignored by source control.

data/lib/generators/rake/pipeline/install/install_generator.rb

@@ -0,0 +1,70 @@
+module Rake
+  class Pipeline
+    class InstallGenerator < Rails::Generators::Base
+
+      desc "Install Rake::Pipeline in this Rails app"
+
+      def disable_asset_pipeline_railtie
+        say_status :config, "Updating configuration to remove asset pipeline"
+        gsub_file app, "require 'rails/all'", <<-RUBY.strip_heredoc
+          # Pick the frameworks you want:
+          require "active_record/railtie"
+          require "action_controller/railtie"
+          require "action_mailer/railtie"
+          require "active_resource/railtie"
+          require "rails/test_unit/railtie"
+        RUBY
+      end
+
+      # TODO: Support sprockets API
+      def disable_asset_pipeline_config
+        regex = /^\n?\s*#.*\n\s*(#\s*)?config\.assets.*\n/
+        gsub_file app, regex, ''
+        gsub_file Rails.root.join("config/environments/development.rb"), regex, ''
+        gsub_file Rails.root.join("config/environments/production.rb"), regex, ''
+      end
+
+      def remove_assets_group
+        regex = /^\n(#.*\n)+group :assets.*\n(.*\n)*?end\n/
+
+        gsub_file "Gemfile", regex, ''
+      end
+
+      def enable_assets_in_development
+        gsub_file "config/environments/development.rb", /^end/, "\n  config.rake_pipeline_enabled = true\nend"
+      end
+
+      # TODO: Support asset-pipeline like API
+      def add_assetfile
+        create_file "Assetfile", <<-RUBY.strip_heredoc
+          # NOTE: The Assetfile will eventually be replaced with an asset-pipeline
+          # compatible API. This is mostly important so that plugins can easily
+          # inject into the pipeline.
+          #
+          # Depending on demand and how the API shakes out, we may retain the
+          # Assetfile API but pull in the information from the Rails API.
+
+          input "app/assets"
+          output "public"
+
+          match "*.js" do
+            concat "application.js"
+          end
+
+          match "*.css" do
+            concat "application.css"
+          end
+
+          # copy any remaining files
+          concat
+        RUBY
+      end
+
+    private
+      def app
+        @app ||= Rails.root.join("config/application.rb")
+      end
+    end
+  end
+end
+

data/lib/rake-pipeline.rb

@@ -0,0 +1,462 @@
+require "rake-pipeline/file_wrapper"
+require "rake-pipeline/filter"
+require "rake-pipeline/manifest_entry"
+require "rake-pipeline/manifest"
+require "rake-pipeline/dynamic_file_task"
+require "rake-pipeline/filters"
+require "rake-pipeline/dsl"
+require "rake-pipeline/matcher"
+require "rake-pipeline/reject_matcher"
+require "rake-pipeline/sorted_pipeline"
+require "rake-pipeline/error"
+require "rake-pipeline/project"
+require "rake-pipeline/cli"
+require "rake-pipeline/graph"
+
+if defined?(Rails::Railtie)
+  require "rake-pipeline/railtie"
+elsif defined?(Rails)
+  require "rake-pipeline/rails_plugin"
+end
+
+require "thread"
+
+# Use the Rake namespace
+module Rake
+  # Override Rake::Task to support recursively re-enabling
+  # a task and its dependencies.
+  class Task
+
+    # @param [Rake::Application] app a Rake Application
+    # @return [void]
+    def recursively_reenable(app)
+      reenable
+
+      prerequisites.each do |dep|
+        app[dep].recursively_reenable(app)
+      end
+    end
+  end
+
+  # Override Rake::FileTask to make it sortable
+  class FileTask
+    # implement Ruby protocol for sorting
+    #
+    # @return [Fixnum]
+    def <=>(other)
+      [name, prerequisites] <=> [other.name, other.prerequisites]
+    end
+  end
+
+  # A Pipeline is responsible for taking a directory of input
+  # files, applying a number of filters to the inputs, and
+  # outputting them into an output directory.
+  #
+  # The normal way to build and configure a pipeline is by
+  # using {.build}. Inside the block passed to {.build}, all
+  # methods of {DSL} are available.
+  #
+  # @see DSL Rake::Pipeline::DSL for information on the methods
+  #   available inside the block.
+  #
+  # @example
+  #   !!!ruby
+  #   Rake::Pipeline.build do
+  #     # process all js, css and html files in app/assets
+  #     input "app/assets", "**/*.{js,coffee,css,scss,html}"
+  #
+  #     # processed files should be outputted to public
+  #     output "public"
+  #
+  #     # process all coffee files
+  #     match "*.coffee" do
+  #       # compile all CoffeeScript files. the output file
+  #       # for the compilation should be the input name
+  #       # with the .coffee extension replaced with .js
+  #       filter(CoffeeCompiler) do |input|
+  #         input.sub(/\.coffee$/, '.js')
+  #       end
+  #     end
+  #
+  #     # specify filters for js files. this includes the
+  #     # output of the previous step, which converted
+  #     # coffee files to js files
+  #     match "*.js" do
+  #       # first, wrap all JS files in a custom filter
+  #       filter ClosureFilter
+  #       # then, concatenate all JS files into a single file
+  #       concat "application.js"
+  #     end
+  #
+  #     # specify filters for css and scss files
+  #     match "*.{css,scss}" do
+  #       # compile CSS and SCSS files using the SCSS
+  #       # compiler. if an input file has the extension
+  #       # scss, replace it with css
+  #       filter(ScssCompiler) do |input|
+  #         input.sub(/\.scss$/, 'css')
+  #       end
+  #       # then, concatenate all CSS files into a single file
+  #       concat "application.css"
+  #     end
+  #
+  #     # the remaining files not specified by a matcher (the
+  #     # HTML files) are simply copied over.
+  #
+  #     # you can also specify filters here that will apply to
+  #     # all processed files (application.js and application.css)
+  #     # up until this point, as well as the HTML files.
+  #   end
+  class Pipeline
+    class Error < StandardError ; end
+    class TmpInputError < Error
+      def initialize(file)
+        @file = file
+      end
+
+      def to_s
+        "Temporary files cannot be input! #{@file} is inside a pipeline's tmp directory"
+      end
+    end
+
+    # @return [Hash[String, String]] the directory paths for the input files
+    #   and their matching globs.
+    attr_accessor :inputs
+
+    # @return [String] the directory path for the output files.
+    attr_reader   :output_root
+
+    # @return [String] the directory path for temporary files
+    attr_accessor :tmpdir
+
+    # @return [Array] an Array of Rake::Task objects. This
+    #   property is populated by the #generate_rake_tasks
+    #   method.
+    attr_reader   :rake_tasks
+
+    # @return [String] a list of files that will be outputted
+    #   to the output directory when the pipeline is invoked
+    attr_reader   :output_files
+
+    # @return [Array] this pipeline's filters.
+    attr_reader   :filters
+
+    attr_writer :input_files
+
+    # @return [Project] the Project that created this pipeline
+    attr_accessor :project
+
+    # @param [Hash] options
+    # @option options [Hash] :inputs
+    #   set the pipeline's {#inputs}.
+    # @option options [String] :tmpdir
+    #   set the pipeline's {#tmpdir}.
+    # @option options [String] :output_root
+    #   set the pipeline's {#output_root}.
+    # @option options [Rake::Application] :rake_application
+    #   set the pipeline's {#rake_application}.
+    def initialize(options={})
+      @filters         = []
+      @invoke_mutex    = Mutex.new
+      @clean_mutex     = Mutex.new
+      @tmp_id          = 0
+      @inputs          = options[:inputs] || {}
+      @tmpdir          = options[:tmpdir] || File.expand_path("tmp")
+      @project         = options[:project]
+
+      if options[:output_root]
+        self.output_root = options[:output_root]
+      end
+
+      if options[:rake_application]
+        self.rake_application = options[:rake_application]
+      end
+    end
+
+    # Build a new pipeline taking a block. The block will
+    # be evaluated by the Rake::Pipeline::DSL class.
+    #
+    # @see Rake::Pipeline::Filter Rake::Pipeline::Filter
+    #
+    # @example
+    #   Rake::Pipeline.build do
+    #     input "app/assets"
+    #     output "public"
+    #
+    #     concat "app.js"
+    #   end
+    #
+    # @see DSL the Rake::Pipeline::DSL documentation.
+    #   All instance methods of DSL are available inside
+    #   the build block.
+    #
+    # @return [Rake::Pipeline] the newly configured pipeline
+    def self.build(options={}, &block)
+      pipeline = new(options)
+      pipeline.build(options, &block)
+    end
+
+    # Evaluate a block using the Rake::Pipeline DSL against an
+    # existing pipeline.
+    #
+    # @see Rake::Pipeline.build
+    #
+    # @return [Rake::Pipeline] this pipeline with any modifications
+    #   made by the given block.
+    def build(options={}, &block)
+      DSL::PipelineDSL.evaluate(self, options, &block) if block
+      self
+    end
+
+    # Copy the current pipeline's attributes over.
+    #
+    # @param [Class] target_class the class to create a new
+    #   instance of. Defaults to the class of the current
+    #   pipeline. Is overridden in {Matcher}
+    # @param [Proc] block a block to pass to the {DSL DSL}
+    # @return [Pipeline] the new pipeline
+    # @api private
+    def copy(target_class=self.class, &block)
+      pipeline = target_class.new
+      pipeline.inputs = inputs
+      pipeline.tmpdir = tmpdir
+      pipeline.rake_application = rake_application
+      pipeline.project = project
+      pipeline.build &block
+      pipeline
+    end
+
+    # Set the output root of this pipeline and expand its path.
+    #
+    # @param [String] root this pipeline's output root
+    def output_root=(root)
+      @output_root = File.expand_path(root)
+    end
+
+    # Set the temporary directory for this pipeline and expand its path.
+    #
+    # @param [String] root this pipeline's temporary directory
+    def tmpdir=(dir)
+      @tmpdir = File.expand_path(dir)
+    end
+
+    # Add an input directory, optionally filtering which files within
+    # the input directory are included.
+    #
+    # @param [String] root the input root directory; required
+    # @param [String] pattern a pattern to match within +root+;
+    #   optional; defaults to "**/*"
+    def add_input(root, pattern = nil)
+      pattern ||= "**/*"
+      @inputs[root] = pattern
+    end
+
+    # If you specify #inputs, this method will
+    # calculate the input files for the directory. If you supply
+    # input_files directly, this method will simply return the
+    # input_files you supplied.
+    #
+    # @return [Array<FileWrapper>] An Array of file wrappers
+    #   that represent the inputs for the current pipeline.
+    def input_files
+      return @input_files if @input_files
+
+      assert_input_provided
+
+      result = []
+
+      @inputs.each do |root, glob|
+        expanded_root = File.expand_path(root)
+        files = Dir[File.join(expanded_root, glob)].sort.select { |f| File.file?(f) }
+
+        files.each do |file|
+          relative_path = file.sub(%r{^#{Regexp.escape(expanded_root)}/}, '')
+          wrapped_file = FileWrapper.new(expanded_root, relative_path)
+
+          raise TmpInputError, file if wrapped_file.in_directory?(tmpdir)
+
+          result << wrapped_file
+        end
+      end
+
+      result.sort
+    end
+
+    # for Pipelines, this is every file, but it may be overridden
+    # by subclasses
+    alias eligible_input_files input_files
+
+    # @return [Rake::Application] The Rake::Application to install
+    #   rake tasks onto. Defaults to Rake.application
+    def rake_application
+      @rake_application || Rake.application
+    end
+
+    # Set the rake_application on the pipeline and apply it to filters.
+    #
+    # @return [void]
+    def rake_application=(rake_application)
+      @rake_application = rake_application
+      @filters.each { |filter| filter.rake_application = rake_application }
+      @rake_tasks = nil
+    end
+
+    # Add one or more filters to the current pipeline.
+    #
+    # @param [Array<Filter>] filters a list of filters
+    # @return [void]
+    def add_filters(*filters)
+      filters.each do |filter|
+        filter.rake_application = rake_application
+        filter.pipeline = self
+      end
+      @filters.concat(filters)
+    end
+    alias add_filter add_filters
+
+    # Invoke the pipeline, processing the inputs into the output.
+    #
+    # @return [void]
+    def invoke
+      @invoke_mutex.synchronize do
+        @tmp_id = 0
+
+        self.rake_application = Rake::Application.new
+
+        setup
+
+        @rake_tasks.each { |task| task.invoke }
+      end
+    end
+
+    # Set up the filters and generate rake tasks. In general, this method
+    # is called by invoke.
+    #
+    # @return [void]
+    # @api private
+    def setup
+      setup_filters
+      generate_rake_tasks
+      record_input_files
+    end
+
+    # Set up the filters. This will loop through all of the filters for
+    # the current pipeline and wire up their input_files and output_files.
+    #
+    # Because matchers implement the filter API, matchers will also be
+    # set up as part of this process.
+    #
+    # @return [void]
+    # @api private
+    def setup_filters
+      last = @filters.last
+
+      @filters.inject(eligible_input_files) do |current_inputs, filter|
+        filter.input_files = current_inputs
+
+        # if filters are being reinvoked, they should keep their roots but
+        # get updated with new files.
+        filter.output_root ||= begin
+          output = if filter == last
+            output_root
+          else
+            generate_tmpdir
+          end
+
+          File.expand_path(output)
+        end
+
+        filter.setup_filters if filter.respond_to?(:setup_filters)
+
+        filter.output_files
+      end
+    end
+
+    # A list of the output files that invoking this pipeline will
+    # generate.
+    #
+    # @return [Array<FileWrapper>]
+    def output_files
+      @filters.last.output_files unless @filters.empty?
+    end
+
+    # Add a final filter to the pipeline that will copy the
+    # pipeline's generated files to the output.
+    #
+    # @return [void]
+    # @api private
+    def finalize
+      add_filter(Rake::Pipeline::PipelineFinalizingFilter.new)
+    end
+
+    # A unique fingerprint. It's used to generate unique temporary
+    # directory names. It must be unique to the pipeline. It must be
+    # the same across processes.
+    #
+    # @return [String]
+    # @api private
+    def fingerprint
+      if project 
+        project.pipelines.index self
+      else
+        1
+      end
+    end
+
+    # the Manifest used in this pipeline
+    def manifest
+      project.manifest
+    end
+
+    # the Manifest used in this pipeline
+    def last_manifest
+      project.last_manifest
+    end
+
+  protected
+    # Generate a new temporary directory name.
+    #
+    # @return [String] a unique temporary directory name
+    def generate_tmpname
+      "rake-pipeline-#{fingerprint}-tmp-#{@tmp_id += 1}"
+    end
+
+    # Generate a new temporary directory name under the main tmpdir.
+    #
+    # @return [void]
+    def generate_tmpdir
+      File.join(tmpdir, self.generate_tmpname)
+    end
+
+    # Generate all of the rake tasks for this pipeline.
+    #
+    # @return [void]
+    def generate_rake_tasks
+      @rake_tasks = filters.collect { |f| f.generate_rake_tasks }.flatten
+    end
+
+    # Assert that an input root and glob were both provided.
+    #
+    # @raise Rake::Pipeline::Error if input root or glob were missing.
+    # @return [void]
+    def assert_input_provided
+      if inputs.empty?
+        raise Rake::Pipeline::Error, "You cannot get input files without " \
+                                     "first providing input files and an input root"
+      end
+    end
+
+    # This is needed to ensure that every file procesed in the pipeline
+    # has an entry in the manifest. This is used to compare input files
+    # for the global dirty check
+    def record_input_files
+      input_files.each do |file|
+        full_path = file.fullpath
+
+        if File.exists?(full_path) && !manifest[full_path]
+          manifest[full_path] ||= ManifestEntry.new({}, File.mtime(full_path).to_i)
+        end
+      end
+    end
+  end
+end