goat 0.2.12 → 0.3.0

data/README.md

@@ -0,0 +1,45 @@
+Goat
+===
+
+<http://goatweb.org>
+
+Description
+-----------
+
+Goat makes it easier to build complex, data-heavy websites. It's written in Ruby, and aims to be a good tool for building applications in the class of, say, Facebook: data-heavy, interactive, and real-time.
+
+Web development presents a fairly subtle set of problems:
+
+- Distributing application logic between the client and the server. (You usually can't put it all on the client, but it's unpleasant to wait twenty seconds for a page to refresh before you find out the value of a form field is invalid.)
+- Figuring out how to make changes without interruption -- live data migrations, and code upgrades that (ideally) don't interrupt users.
+- Varying browsers.
+- Multiple programming languages (CSS, JavaScript, HTML, server-side languages).
+- Multiple input devices (touch, mouse, keyboard).
+- May have to scale massively at short notice
+
+Goat
+----
+
+Goat is page-oriented. Pages are the fundamental building block of the web. Complex data often naturally breaks down into discrete pages. In the URL, pages come with one of the most universally used and understood collaboration tools.
+
+Figuring out how to combine presenting a significant amount of data presentation with complex user interaction is tough. It's not easy to imagine what Facebook, Wikipedia, Twitter, Gmail or Quora would look like as desktop applications -- and it's unlikely that they'd be better.
+
+Some applications, such as spreadsheets, probably aren't a good fit for a page-based model, and these applications are not likely to be a good fit for Goat.
+
+Goat avoids HTML templates, and replaces them with programmatic HTML generation and components: a page is composed of one or more components. Components may persist on the server, if you'd like to be able to update the client-side state of a component in real time. Components also group together the code that may be required for a single UI element: client-side JavaScript, server-side Ruby, and CSS.
+
+Goat tries to:
+
+- Make it easy to build pages that don't require refreshing (i.e., live updating of data).
+- Optimize for the development cycle (all things being equal, the application that's easier to change will end up better).
+- Make writing client-side JavaScript easier. (Intelligent scoping, easy RPC.)
+- Make programmatic HTML generation easy.
+- Not be all-encompassing, but to make sensible assumptions about most parts of the web application stack.
+
+Design decisions
+----------------
+
+- Goat should store JS for some set of components in an external file so that JS can be cached across pages.
+- You can't have to load 2mb of JS and have a loading bar when a user visits the site for the first time. URLs should look decent.
+- Make site work well for search engines.
+- Goat doesn't try to abstract away the fact that the web is built with CSS/HTML/JavaScript.

data/bin/channel-srv

@@ -0,0 +1,150 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'eventmachine'
+require 'json'
+require 'thin'
+require 'rack'
+
+%w{extn net-common state-srv}.each {|f| require File.join(File.dirname(__FILE__), "../lib/goat/#{f}")}
+
+$verbose = ARGV.include?('-v')
+
+$chsrv_instance = 'chsrv_' + String.random(10)
+
+module Goat
+  module ChannelSrv
+    class ChannelStateConnection < StateSrvConnection
+      def connection_completed
+        $stderr.puts :connection_completed
+        super
+
+        send_message('register_chsrv', 'chsrv' => $chsrv_instance)
+      end
+
+      def message_received(msg)
+        pusher = ChannelPusher.pusher_for(msg['pgid'])
+        if pusher
+          pusher.statesrv_message_received(msg)
+        else
+          logw "Page #{msg['pgid'].inspect} doesn't appear to be connected to this chsrv"
+        end
+      end
+    end
+
+    class ChannelPusher
+      include EM::Deferrable
+
+      @pushers = {}
+      class << self
+        attr_reader :pushers
+
+        def pusher_for(pgid)
+          @pushers[pgid]
+        end
+      end
+
+      attr_reader :pgid
+
+      def initialize(pgid, jsonp, version)
+        @pgid = pgid
+        @jsonp = jsonp
+        @version = version
+        @messages = []
+
+        self.class.pushers[pgid] = self
+
+        self.errback { cleanup }
+        self.callback { cleanup }
+
+        ChannelStateConnection.send_message('page_connected',
+          'pgid' => pgid,
+          'chsrv' => $chsrv_instance,
+          'version' => @version
+        )
+      end
+
+      def statesrv_message_received(msg)
+        #TODO hack below for batched messages
+        Log.log_message(msg)
+        if msg['type'] == 'page_updated'
+          @messages += msg['messages']
+          # send on next tick so we get a batch of messages together, rather than a single message
+          EM.next_tick { self.send_and_finish }
+        elsif msg['type'] == 'page_expired'
+          @messages << msg
+          EM.next_tick { self.send_and_finish }
+        else
+          raise "Don't understand this message: #{msg.inspect}"
+        end
+      end
+
+      def cleanup
+        logw "connection for #{@pgid}/#{@jsonp} closed"
+        @finished = true
+        self.class.pushers.delete(pgid)
+        @channel.unsubscribe(@sub) if @sub
+      end
+
+      def send_and_finish
+        return if @finished
+        @body_callback.call("#{@jsonp}(#{{'messages' => @messages}.to_json})")
+        succeed
+      end
+
+      def each(&blk)
+        @body_callback = blk
+      end
+    end
+
+    module Srv
+      def self.request_failed; [500, {}, 'request failed']; end
+
+      def self.handle_channel(req)
+        pgid = req['_id']
+        jsonp = req['jsonp']
+        ver = req['version']
+
+        return request_failed unless pgid && jsonp && ver
+
+        body = ChannelPusher.new(pgid, jsonp, ver)
+
+        EM.next_tick do
+          req.env['async.callback'].call([200, {
+            'Connection' => 'Keep-Alive',
+            'Content-Type' => 'application/javascript'
+          }, body])
+        end
+
+        [-1, {}, []]
+
+      rescue NoStateSrvConnectionError
+        $stderr.puts "Couldn't register with StateSrv to service a /channel request"
+        request_failed
+      end
+
+      def self.call(env)
+        req = Rack::Request.new(env)
+
+        if req.path == '/channel'
+          handle_channel(req)
+        else
+          [400, {}, 'bad request']
+        end
+      end
+    end
+  end
+end
+
+if $0 == __FILE__
+  app = Rack::Builder.app do |builder|
+    use Rack::CommonLogger
+    use Rack::ShowExceptions
+    run Goat::ChannelSrv::Srv
+  end
+
+  EM.run do
+    Goat::ChannelSrv::ChannelStateConnection.connect
+    Rack::Handler::Thin.run(app, :Port => (ARGV.first ? ARGV.first.to_i : 8050))
+  end
+end

data/bin/press

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'rubygems'
+require File.join(File.dirname(__FILE__), '../lib/goat')
+
+OptionParser.new do |opts|
+  opts.on('-fOPTIONAL', String) {|f| $file = f}
+  opts.on('-cOPTIONAL', String) {|c| $classes = c.split(',')}
+  opts.on('-pOPTIONAL', String) {|p| $classes = File.read(p).strip.split(',')}
+  opts.on('-oOPTIONAL', String) {|w| $specfile = w }
+  opts.on('-j') { $emitjs = true}
+  opts.on('-s') { $emitstyle = true}
+  opts.on('-v') {} # suppress optparse wanting to use -v as version
+  opts.on('-h') {usage; exit 1}
+end.parse!(ARGV)
+
+require $file
+$classes.each do |c|
+  cls = Kernel.fetch_class(c)
+  puts cls.__script if $emitjs && cls.__script
+  puts cls.scoped_css if $emitstyle && cls.__css
+end
+
+if $specfile
+  File.open($specfile, 'w') do |f|
+    f.write({:classes => $classes, :created => Time.now.to_i, :created_human => Time.now.to_s}.to_yaml)
+  end
+end

data/bin/state-srv

@@ -0,0 +1,390 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'eventmachine'
+require 'json'
+require 'set'
+require 'term/ansicolor'
+
+$verbose = ARGV.include?('-v')
+
+$:.unshift(File.join(File.dirname(__FILE__), '../lib'))
+
+require 'goat/common'
+require 'goat/net-common'
+
+module Goat
+  module StateSrv
+    NotifChannel = EM::Channel.new
+
+    class PageDeleted < RuntimeError; end
+
+    # written this clunky way because it'll be manually-memory-managed c++ some day soon
+    class Registry
+      def initialize
+        @components_by_class = {}
+        @components_by_id = {}
+        @components_by_page = {}
+        @last_connected = {}
+        @page_connections = {}
+        @channelsrvs = {}
+        @page_versions = {}
+        @page_updates = {}
+      end
+
+      def add_component(pgid, skel)
+        self.find_class(skel.cls) << skel.id
+        self.find_page(pgid) << skel.id
+        @components_by_id[skel.id] = skel
+      end
+
+      def find_class(cls)
+        @components_by_class[cls] ||= Set.new
+      end
+
+      def components_by_class(cls)
+        find_class(cls).map{|cid| find_id(cid)}
+      end
+
+      def find_id(cid)
+        @components_by_id[cid]
+      end
+
+      def initialize_page(pgid)
+        @page_versions[pgid] = 0
+      end
+
+      def incr_page_version(pgid)
+        @page_versions[pgid] += 1
+      end
+
+      def page_version(pgid)
+        @page_versions[pgid]
+      end
+
+      def find_page(pgid)
+        @components_by_page[pgid] ||= Set.new
+      end
+
+      def page_connected(pgid, chsrv, ver)
+        if @page_versions[pgid]
+          @last_connected[pgid] = Time.now
+          @page_connections[pgid] = {:chsrv => chsrv, :version => ver}
+        else
+          logw "Page #{pgid} has been deleted"
+          raise PageDeleted
+        end
+      end
+
+      def page_disconnected(pgid)
+        @page_connections.delete(pgid)
+      end
+
+      def last_connected_delta(pgid)
+        if last = @last_connected[pgid] 
+          Time.now - last
+        else
+          Time.now - Time.at(0)
+        end
+      end
+
+      def connected_pages
+        @page_connections.keys
+      end
+
+      def connected?(pgid)
+        @page_connections[pgid]
+      end
+
+      def delete_component(cid)
+        c = @components_by_id[cid]
+        if c
+          if cs = @components_by_class[c.cls]
+            cs.delete(cid)
+          end
+        end
+        @components_by_id.delete(cid)
+      end
+
+      def delete_page(pgid)
+        cs = @components_by_page[pgid]
+        if cs
+          cs.each {|c| delete_component(c)}
+        else
+          logw "(No components for #{pgid})"
+        end
+
+        @page_updates.delete(pgid)
+        @last_connected.delete(pgid)
+        @page_connections.delete(pgid)
+        @page_versions.delete(pgid)
+        @components_by_page.delete(pgid)
+      end
+
+      def register_chsrv(id, conn)
+        @channelsrvs[id] = conn
+      end
+
+      def unregister_chsrv(id, conn)
+        @channelsrvs.delete(id)
+      end
+
+      def chsrv_connection(id)
+        @channelsrvs[id]
+      end
+
+      def page_connection(id)
+        @page_connections[id]
+      end
+
+      def page_updates(pgid)
+        @page_updates[pgid] ||= []
+      end
+
+      def page_updates_for_txn(pgid, txn)
+        if txn
+          if ups = page_updates(pgid).detect{|u| u[:txn] == txn}
+            ups
+          else
+            txnu = {:txn => txn, :ups => []}
+            page_updates(pgid) << txnu
+            txnu
+          end
+        else
+          txnu = {:txn => nil, :ups => []}
+          page_updates(pgid) << txnu
+          txnu
+        end
+      end
+
+      def add_update(pgid, txn, up)
+        page_updates_for_txn(pgid, txn)[:ups] << up
+      end
+
+      def memusage
+        $stderr.puts "Components in memory: #{@components_by_id.values.count}"
+      end
+    end
+
+    class Delegate
+      @registry = Registry.new
+      
+      def self.registry; @registry; end
+
+      class << self
+        def ensure_keys(hash, ks)
+          ks.each do |k|
+            unless hash.include?(k)
+              raise IndexError.new("Hash omits key #{k.inspect}")
+            end
+          end
+        end
+
+        def register_page(msg)
+          ensure_keys(msg, %w{pgid components})
+
+          pgid = msg['pgid']
+
+          msg['components'].each do |c|
+            @registry.add_component(pgid, ComponentSkeleton.from_hash(c))
+          end
+
+          logd("registered #{msg['components'].count} components")
+
+          @registry.initialize_page(pgid)
+
+          nil
+        end
+
+        def live_components(msg)
+          ensure_keys(msg, %w{class spec})
+
+          @registry.components_by_class(msg['class']).\
+            select{|c| c.spec == msg['spec']}.\
+            select{|c| @registry.connected?(c.pgid) || @registry.last_connected_delta(c.pgid) < 120}.\
+            map(&:to_hash)
+        end
+
+        def fetch_component(msg)
+          ensure_keys(msg, %w{id})
+
+          @registry.find_id(msg['id']).to_hash
+        end
+
+        def send_message(chsrv_id, type, msg)
+          chsrv = @registry.chsrv_connection(chsrv_id)
+
+          if chsrv
+            chsrv.send_message(type, msg)
+          else
+            logw "send_message fail: #{chsrv_id} no longer connected"
+          end
+        end
+
+        def send_updates(pgid)
+          if c = @registry.page_connection(pgid)
+            cur = c[:version]
+            chsrv_id = c[:chsrv]
+            txn_ups = @registry.page_updates(pgid)
+            txn_ups = txn_ups.size > 5 ? txn_ups[-5..-1] : txn_ups
+            ups = txn_ups.map{|x| x[:ups]}.flatten
+            vers = {}
+            txn_ups.each {|txn_up| txn_up[:ups].each{|up| vers[up.version] = txn_up[:txn]}}
+
+            need = ups.select{|u| u.version > cur}
+
+            msgs = []
+
+            need.each do |u|
+              msgs << {
+                'pgid' => pgid,
+                'class' => u.skel.cls,
+                'id' => u.skel.id,
+                'type' => 'component_updated',
+                'updates' => u.mutations,
+                'version' => u.version
+              }
+
+              txn = vers[u.version]
+              msgs << {'type' => 'txn_complete', 'txn' => txn} if txn
+            end
+
+            send_message(chsrv_id, 'page_updated', {'pgid' => pgid, 'messages' => msgs}) unless msgs.empty?
+          else
+            logw "No chsrv connection for page #{pgid}"
+          end
+        end
+
+        def send_page_deleted_error(pgid, chsrv_id)
+          send_message(chsrv_id, 'page_expired', {'pgid' => pgid})
+        end
+
+        def handle_component_update(txn, update)
+          comp = @registry.find_id(update.skel.id)
+
+          if comp
+            comp.dom = update.skel.dom
+            comp.state = update.skel.state
+
+            @registry.incr_page_version(comp.pgid)
+
+            update.version = @registry.page_version(comp.pgid)
+
+            @registry.add_update(comp.pgid, txn, update)
+          else
+            $stderr.puts "Couldn't find component #{id}"
+          end
+        end
+
+        def components_updated(msg)
+          ensure_keys(msg, %w{txn pgid updates})
+
+          txn = msg['txn']
+          msg['updates'].each{|u| handle_component_update(txn, ComponentUpdate.from_hash(u))}
+          send_updates(msg['pgid'])
+
+          nil
+        end
+
+        def page_connected(msg)
+          ensure_keys(msg, %w{pgid chsrv version})
+
+          pgid = msg['pgid']
+
+          @registry.page_connected(pgid, msg['chsrv'], msg['version'].to_i)
+          send_updates(pgid)
+          nil
+        rescue PageDeleted
+          send_page_deleted_error(pgid, msg['chsrv'])
+          nil
+        end
+
+        def register_chsrv(conn, msg)
+          ensure_keys(msg, %w{chsrv})
+
+          @registry.register_chsrv(msg['chsrv'], conn)
+          nil
+        end
+
+        def unregister_chsrv(conn, msg)
+          @registry.unregister_chsrv(msg['chsrv'], conn)
+          nil
+        end
+      end
+    end
+
+    class Sweeper
+      def self.start
+        sweeper = self.new
+
+        EM::PeriodicTimer.new(5) do
+          sweeper.run
+        end
+      end
+
+      def initialize
+        @registry = Delegate.registry
+      end
+
+      def run
+        @registry.connected_pages.each do |pgid|
+          if @registry.last_connected_delta(pgid) > 300
+            logd "Deleting page #{pgid}"
+            @registry.delete_page(pgid)
+          end
+        end
+      end
+    end
+
+    class RecvServer < EM::Connection
+      include EM::P::LineText2
+      include Goat::JSONMessages
+
+      def self.start(host='127.0.0.1', port=8011)
+        EM.start_server(host, port, self)
+      end
+
+      def receive_line(line)
+        msg = JSON.load(line)
+        Log.log_message(msg) if $verbose
+
+        type = msg['type']
+        if type == 'register_chsrv'
+          Delegate.register_chsrv(self, msg)
+          @chsrv = msg
+          logw "#{@chsrv['chsrv']} connected"
+        elsif Delegate.respond_to?(type)
+          if resp = Delegate.send(type, msg)
+            send_message('response', 'response' => resp)
+          end
+        else
+          $stderr.puts "Delegate doesn't respond to #{type.inspect}"
+        end
+
+        maybe_print_statistics
+      end
+
+      def maybe_print_statistics
+        @last_printed ||= Time.now
+        if Time.now - @last_printed > 10
+          Delegate.registry.memusage
+        end
+        @last_printed = Time.now
+      end
+
+      def unbind
+        if @chsrv
+          logw "#{@chsrv['chsrv']} disconnected"
+          Delegate.unregister_chsrv(self, @chsrv)
+        end
+      end
+    end
+  end
+end
+
+if $0 == __FILE__
+  EM.run do 
+    Goat::StateSrv::Sweeper.start
+    Goat::StateSrv::RecvServer.start 
+  end
+end