skypager 0.0.4 → 0.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dd2176972c399bbfec3b8fde460f7592ecfca5a7
-  data.tar.gz: 08290cdac67c958f31a5e9ca96c2ede7a8827fd8
+  metadata.gz: 02aa518232f9553e53c8ecdddbdb55d1118e3697
+  data.tar.gz: 269856d146d98cf16044c83874af17ddf705b054
 SHA512:
-  metadata.gz: 661f07c8c3beaaf3e466ba68e7fa432c6ec1d62852f02e4bf09762dd97b9db9edd1fd4c3101266462b07bae83097ddf90f7420f7b6fb7f64ccba350c7189e2df
-  data.tar.gz: 09c044b5e5cdc7554047ad4e1c0827619455edd9b7d9bfd58b8d513f174835ed6c97d67a19845a020d76607367b179197448f23076fc31396cc20c80ec739160
+  metadata.gz: c8cd34b2ad8b1bf61a9637094e2bed9fe95c061efb398051783536be1b58cc76590b290791b2a52d7274e9f0f04e8891d9e7298d22624ff5ba8a109f38ef9ff3
+  data.tar.gz: 530be140aba47770a87067f8eef4f761d607e5d7c93c9c7a78926420a5a5c66bd4078a8fc1e9431ba67f95a2c9491d29b888ac90e7f7b6123ecd1a10c8c2a94f

data/.gitignore

@@ -18,3 +18,4 @@ tmp
 log/*.log
 examples/build
 skypager.json
+spec/skypager-test-config.rb

data/ARCHITECTURE.md

@@ -0,0 +1,34 @@
+### Middleman Extension
+
+The middleman extension provides configuration to do things like
+declare mappings between dropbox folders and your local folders, or to
+define an external data source based on a google spreadsheet.
+
+the extension takes care of keeping everything in sync for you between
+builds, or more frequently in development
+
+### Skypager CLI
+
+Creates new data sources, shared folders, etc.  Walks you through
+setting up AWS, DNSimple, etc.  Can be used to deploy.
+
+### Skypager Site 
+
+This represents a single middleman application which uses the skypager
+etension.  
+
+### Syncable Folders
+
+syncable folders (folders which are tied to external file hosting
+services, such as Dropbox or Google Drive)
+
+### Data Sources
+
+data sources .external services such as Google Drive, or Dropox, will
+hold spreadsheets or csv files, which are basically just editable
+tables. These get pulled in to the `data` folder and stored as json
+files.
+
+middleman provides that json files stored in the data folder, will be
+available to the config.rb as well as the template helpers.  this is
+how skypager data sources get used by the middleman project.

data/CONTRIBUTING.md

@@ -0,0 +1,7 @@
+### To work on external integrations
+
+You will need to register a few apps with Dropbox, Google, and Github.  
+
+You will need AWS credentials (secret access key, access key id)
+
+You will need a dnsimple username and api token.

data/README.md

@@ -1,49 +1,107 @@
-# Automate all the boring things
+# Static websites are awesome. 
 
-Skypager is a set of tools and API integrations that aims to make
-developing and deploying static HTML websites a blast.  Static websites
-are awesome, and perform far better than dynamically generated websites.
-Especially when deployed to a CDN and cached.  
+This is a fact: Static websites with no backend server are the cheapest, fastest,
+and most secure websites out there.  You can compress things to the max,
+cache them on a CDN distributed across the entire world, and guarantee
+the fastest page loads time for your users.
 
-A fast page load time is considered to be the single most important factor that effects
-conversions.
+If you are starting out developing a website or a web based application, constraining yourself to
+keeping as much of your user facing web assets static will pay off. 
 
-Of course, people build dynamic websites because they need to be powered
-by a Content Management System and a database, and supported by a version
-control system, among many other completely valid reasons.  
+### Dynamic sites are practical 
 
-Nobody wants to hand code HTML pages, so we need some form of templates which we can fill with our own
-data based on what URL the browser requests.
+But a static HTML only website with no backend or content management just doesn't suit the needs of most people, so people
+turn to hosted solutions like Wordpress or Drupal, maybe they develop their own Rails app.  
 
-This is where Skypager comes in. Skypager is a system for reacting to
-changes in a variety of data sources, file sharing APIs, or version control systems, and then
-automatically triggering builds in your site build tool (Middleman, for
-example.)  Skypager then takes the static build, and deploys it to an Amazon S3
-Bucket, and triggers expirations of your Cloudfront CDN or other Caches. 
+The problem with dynamic sites powered by servers and application code and databases is you need to scale them, 
+secure them, host them, or hire people to do those things for you.  Or you can pay for hosting for some generic 
+content management platform, but you'll be constrained by these platforms and limited in the ways you can customize them.
 
-Skypager can automate setting up your S3 Buckets, Cloudfront CDNs, and
-DNS configuration.
+### Skypager is the best of both worlds
 
-With Skypager, all of the following setups are possible:
+If we eliminate the sites that developers make for themselves and for
+other developers, most dynamic websites are going to be relying on content that
+comes from people who aren't developers.  Whether that is blog posts,
+marketing copy, landing pages, images, videos, or a large dataset which
+contains records that will be represented by their own URL.
 
-- building a photographer website, and allowing the photographer to
-  store all of her galleries in a Dropbox folder.  Every time they add a
-  new folder or a new file to the folder, Skypager regenerates the
-  static pages and pushes the assets to S3.
+What tools are our non-developer teammates already using? They're using Dropbox, Google
+Drive, Excel Spreadsheets, Word Documents, Photos, Videos, and things of
+that nature.
 
-- building a product catalog website powered by a spreadsheet on Google
-  Drive. Any time your customer adds or edits a row in the spreadsheet,
-  Skypager generates the product page and deploys it for you.
+Skypager provides ways to sync up with these other tools, and regenerate
+your site's content whenever changes are made to these file systems.  
 
-- building a blog for your creative writer friend by giving them a
-  Dropbox folder to write markdown. Any time they edit the markdown, or
-  add a new one? you guessed it.
+This opens up a lot of interesting possibilities for a wide range of
+different kinds of websites, and ways of collaborating with people to
+build them.
 
-- Building a Single page javascript app powered by JSON API? 
-  Do you want a static marketing site powered by data in your API or database? Having Rails
-  generate these pages on the fly and caching them it turns out can be a
-  terrible approach.  Skypager can watch a REST endpoint in your app for
-  changes, or you can POST to the Skypager Build Server's Webhooks and
-  trigger a build on your own.
+Best of all, you get the complete control and freedom provided to
+you by the Ruby programming language and the many great site template authoring
+tools provided by the community.
 
+### What about user generated content?
 
+Serverless apps which rely on storing and user generated content will need to get
+creative with how they handle these requirements. When you are constraining yourself to
+using a static site, this is one of the tradeoffs.    
+
+We have a number of guides and examples for how you can accomplish this.  Coming soon.
+
+### Hassle free deployment, DNS Configuration, and CDN hosting setup
+
+Skypager automates all of the boring but necessary steps that somebody
+needs to take if they wanted to configure their own Static website
+hosting using Amazon S3, Cloudfront, and it even takes care of the DNS configuration
+you'll need to do to make this setup work on your own domain.
+
+### Sync with Dropbox, Google Drive, and Github
+
+Store your source code on Github so your developer can work on it.  
+
+Store your photo library on Dropbox or Google Drive so your designer 
+and photographer can manage those assets using the tools they work with. 
+
+Store an excel spreadsheet containing all of your product information on Dropbox, 
+or in a Google spreadsheet, so people from your business team can manage
+that content.
+
+Skypager will make sure these external sources of content are always
+available, and incorporate them into the latest build of the site.
+
+### Powered by Middleman
+
+Skypager is an extension to the [Middleman](http://middlemanapp.com) static site generator.
+Middleman allows for you to develop your site using templates, and
+enhanced CSS and Javascript authoring languages like Sass, Less,
+or Coffeescript.
+
+### Getting Started: Standalone Mode
+
+Skypager in standlone mode is what you would want to use if you are
+working on a single site, and you are comfortable with the command line
+and ruby.  Instructions for doing this can be found in our [Getting
+Started Guide](https://github.com/architects/skypager)
+
+### The Skypager Build Server
+
+If you are managing a large number of Skypager sites, and would like to set it
+up so that all of your sites stay up to date and in sync, you can run your
+own instance of the Skypager Build Server.
+
+The build server will do things like: accept webhooks from Github,
+Dropbox, or Google Drive, which will happen whenever content changes.  
+
+This will trigger a re-build of your site, and the updated content gets automatically pushed
+to the hosting environment and to the CDN.
+
+### Skypager.io: Hosted option with a Web UI
+
+The easiest way to use Skypager in production is to subscribe to the Skypager.io
+hosting service, which provides you with a web based front end for
+managing all of your sites and handles all of the hosting service
+configuration and such for you.  
+
+Just point us to the source code, whether it is on Dropbox or Github, 
+and use the rest of the features which you can configure using our webfront end, or our
+Command line interface (CLI).

data/Rakefile

@@ -16,6 +16,6 @@ require 'rspec/core/rake_task'
 
 desc "Run all specs in spec directory (excluding plugin specs)"
 
-RSpec::Core::RakeTask.new(:spec => 'app:db:test:prepare')
+RSpec::Core::RakeTask.new(:spec)
 
 task :default => :spec

data/lib/skypager.rb

@@ -59,6 +59,11 @@ module Skypager
     Skypager::Sync::Dropbox.client
   end
 
+  def self.github
+    require 'octokit' unless defined?(Octokit)
+    Skypager::Sync::Github.client
+  end
+
   def self.dns
     require 'dnsimple'
     require 'skypager/dns'
@@ -105,5 +110,6 @@ require 'skypager/sync/dropbox/delta'
 require 'skypager/sync/google'
 require 'skypager/sync/folder'
 require 'skypager/sync/amazon'
+require 'skypager/sync/github'
 require 'skypager/dns'
 require 'skypager/extension'

data/lib/skypager/builder.rb

@@ -0,0 +1,158 @@
+module Skypager
+  class Builder
+
+    attr_reader :app,
+                :source,
+                :logger,
+                :options
+
+    def initialize(app, options={})
+      @app = app
+      @options = options.reverse_merge(clean: true, verbose: true)
+      @source_dir = app.source_path
+      @build_dir  = app.build_path
+      @to_clean   = Set.new
+
+      @logger = app.logger
+      @rack = ::Rack::Test::Session.new(app.class.to_rack_app)
+    end
+
+    def build(force=false)
+      return unless force || needs_build?
+
+      queue_current_paths if should_clean?
+      execute!
+      clean! if should_clean?
+
+      app.site.requires_build!(false)
+      app.deploy! if app.auto_deploy?
+    end
+
+    protected
+
+    def clean!
+      @to_clean.each do |f|
+        FileUtils.rm_rf(f)
+      end
+
+      Dir[@build_dir.join('**', '*')].select { |d| File.directory?(d) }.each do |d|
+        FileUtils.rm_rf(d) if Pathname(d).children.empty?
+      end
+    end
+
+    def should_clean?
+      options[:clean]
+    end
+
+    def queue_current_paths
+      return unless app.build_path.exist?
+
+      paths = ::Middleman::Util.all_files_under(app.build_path).map(&:realpath).select(&:file?)
+
+      @to_clean += paths.select do |path|
+        path.to_s !~ /\/\./ || path.to_s =~ /\.(htaccess|htpasswd)/
+      end
+
+      return unless RUBY_PLATFORM =~ /darwin/
+
+      # handle UTF-8-MAC filename on MacOS
+      @to_clean = @to_clean.map { |path| path.to_s.encode('UTF-8', 'UTF-8-MAC') }
+    end
+
+    def needs_build?
+      @needs_build = app.data_sources.values.any? do |ds|
+        ds.fresh_on_server?
+      end
+
+      # TODO
+      # Add support for google drive folders, github folder changes
+      @needs_build = @needs_build || app.syncables.any? do |syncable|
+        syncable.dropbox? && syncable.has_remote_changes?
+      end
+
+      @needs_build
+    end
+
+    def execute!
+      sort_order = %w(.png .jpeg .jpg .gif .bmp .svg .svgz .ico .webp .woff .otf .ttf .eot .js .css)
+
+      app.sitemap.resources.select do |resource|
+        resource.ext == '.css'
+      end.each(&method(:build_resource))
+
+      app.files.find_new_files((@source_dir + app.images_dir).relative_path_from(app.root_path))
+      app.sitemap.ensure_resource_list_updated!
+
+      resources = app.sitemap.resources.sort_by do |r|
+        sort_order.index(r.ext) || 100
+      end
+
+      if @build_dir.expand_path.relative_path_from(@source_dir).to_s =~ /\A[.\/]+\Z/
+        raise ":build_dir (#{@build_dir}) cannot be a parent of :source_dir (#{@source_dir})"
+      end
+
+      resources.reject do |resource|
+        resource.ext == '.css'
+      end.each(&method(:build_resource))
+
+    end
+
+    def build_resource(resource)
+      return if options[:glob] && !File.fnmatch(options[:glob], resource.destination_path)
+
+      output_path = render_to_file(resource)
+
+      return unless should_clean? && output_path.exist?
+
+      if RUBY_PLATFORM =~ /darwin/
+        # handle UTF-8-MAC filename on MacOS
+
+        @to_clean.delete(output_path.realpath.to_s.encode('UTF-8', 'UTF-8-MAC'))
+      else
+        @to_clean.delete(output_path.realpath)
+      end
+    end
+
+    def render_to_file(resource)
+      output_file = @build_dir + resource.destination_path.gsub('%20', ' ')
+
+      if resource.binary?
+        if !output_file.exist?
+          #base.say_status :create, output_file, :green
+        elsif FileUtils.compare_file(resource.source_file, output_file)
+          #base.say_status :identical, output_file, :blue
+          return output_file
+        else
+          #base.say_status :update, output_file, :yellow
+        end
+
+        output_file.dirname.mkpath
+        FileUtils.cp(resource.source_file, output_file)
+      else
+        begin
+          response = @rack.get(URI.escape(resource.request_path))
+
+          if response.status == 200
+            Pathname(output_file).open("w+") {|fh| fh.write binary_encode(response.body) }
+          else
+            handle_error(output_file, response.body)
+          end
+        rescue => e
+          handle_error(output_file, "#{e}\n#{e.backtrace.join("\n")}", e)
+        end
+      end
+
+      output_file
+    end
+
+    def handle_error(file_name, response, *args)
+
+    end
+
+    def binary_encode(string)
+      string.force_encoding('ascii-8bit') if string.respond_to?(:force_encoding)
+      string
+    end
+
+  end
+end

data/lib/skypager/builder/server.rb

@@ -0,0 +1,63 @@
+# Skypager::Builder::Server
+#
+# A Rack mountable server which can receive Webhook notifications
+# from services like Dropbox, Google Drive, or our own Rest APIs
+# and trigger builds for the various Skypager::Site that are referenced
+# in the Skypager::Site.directory
+#
+# Currently, the Builder::Server just finds sites and marks them as requiring
+# a build. Some external process will poll the sites directory to find the sites
+# which require a build, and run that for us. (Cron, perhaps)
+module Skypager
+  class Builder
+    class Server
+      attr_reader :options
+
+      def initialize(*args)
+        @options = args.extract_options!
+      end
+
+      def call(env)
+        request = Rack::Request.new(env)
+        respond_to(request)
+      rescue => e
+        [500, {}, [e.message]]
+      end
+
+      private
+
+      def webhooks
+        require "skypager/builder/webhook_handler" unless defined?(Skypager::Builder::WebhookHandler)
+        Skypager::Builder::WebhookHandler
+      end
+
+      def respond_to_webhook(request)
+        case
+        when request.path.match(/dropbox/)
+          puts "== Found dropbox webhook"
+          webhooks.handle(:dropbox, request)
+        when request.path.match(/github/)
+          puts "== Found github webhook"
+          webhooks.handle(:github, request)
+        when request.path.match(/google/)
+          puts "== Found google webhook"
+          webhooks.handle(:google, request)
+        else
+          puts "== Found custom webhook"
+          webhooks.handle(:custom, request)
+        end
+      end
+
+      def respond_to(request)
+        puts "== Responding to: #{ request.path }"
+        case
+        when request.path.match(/hooks/)
+          puts "== Found hooks path"
+          respond_to_webhook(request)
+        else
+          [404, {}, [""]]
+        end
+      end
+    end
+  end
+end