mingle_events 0.0.7 → 0.1.0

data/README.textile

@@ -4,7 +4,9 @@ h3. Overview
 
 Mingle 3.3 introduced a new Events API in the form of an "Atom feed":http://www.thoughtworks-studios.com/mingle/3.3/help/mingle_api_events.html. The Mingle team and ThoughtWorks Studios are big believers in the use of Atom for exposing events. Atom is a widely used standard, and this event API style puts the issue of robust event delivery in the hands of the consumer, where it belongs. In fact, we'd argue this is the only feasible means of robust, scalable event delivery, short of spending hundreds of thousands or millions of dollars on enterprise buses and such. Atom-delivered events are cheap, scalable, standards-based, and robust.
 
-However, we do accept that asking integrators wishing to consume events to implement polling is not ideal. Writing polling consumers can be tedious. And this tedium gets in the way of writing sweet Mingle integrations. We are addressing this by publishing libraries such as this, which if effective, fully hide the mechanics of event polling from the consumer. The consumer only need worry about the processing of events. Said processing is modeled in the style of 'pipes and filters.'
+However, we do accept that asking integrators wishing to consume events to implement polling is not ideal. Writing polling consumers can be tedious. And this tedium gets in the way of writing sweet Mingle integrations. We are addressing this by publishing libraries such as this, which if effective, fully hide the mechanics of event polling from the consumer. The consumer only need worry about the processing of events. 
+
+The library supports both basic event analysis as well as polling for purposes of filtering and processing, e.g., re-publishing. The polling and processing portion is modeled in the style of 'pipes and filters.'
 
 h3. Installation
 
@@ -20,35 +22,68 @@ h3. Source
 git clone git://github.com/ThoughtWorksStudios/mingle_events.git
 </pre>
   
-h2. Quick example
+h2. Quick examples
+
+h3. Get the latest events, from time zero
+
+The event fetcher will manage it's own state, so fetch_latest can be called repeatedly to fetch only new events. The first fetch will crawl all the way back to the very first event in the project's history.
 
 <pre>
-# configure access to mingle
 mingle_access = MingleEvents::MingleBasicAuthAccess.new('https://localhost:7071', 'david', 'p')
+event_fetcher = MingleEvents::ProjectEventFetcher.new('my_project', mingle_access)
+latest_events = event_fetcher.fetch_latest
+</pre>
+
+h3. Get the latest events, from now
+
+Similar to the previous example, but a call to reset_to_now will tell the fetcher only to pull new stuff, starting now. If reset_now has been previously called, it will be ignored, and events will be fetched to the last seen event. If you have called reset_now previously, but really do wish to reset to now, you can call reset followed by reset_to_now.
+
+<pre>
+mingle_access = MingleEvents::MingleBasicAuthAccess.new('https://localhost:7071', 'david', 'p')
+event_fetcher = MingleEvents::ProjectEventFetcher.new('my_project', mingle_access)
+latest_events = event_fetcher.reset_to_now
+latest_events = event_fetcher.fetch_latest
+</pre>
+
+h3. Historical analysis, via event playback
+
+One of the great usages of event playback is historical analysis. E.g., if you wanted to see the number of stories added each day to your project, you could write a processor that counts new stories and takes a snapshot at the end of each day, where the end of the day is the first time you see an event for the next day. If you are doing this sort of analysis you will not want to poll the entire project's event history every time as that is quite time consuming. 
+
+The assumption is that you've previously fetched the latest events. Or at least you've fetched the events in which you are interested. Use all_fetched_entries to load previously fetched events, they're all cached on disk so you don't pay the prices of retrieving from Mingle.
+
+<pre>
+event_fetcher = MingleEvents::ProjectEventFetcher.new('my_project', mingle_access)
+event_fetcher.all_fetched_entries.each do |e|
+  # do something interesting with each event (see examples folder for real analysis examples)
+end  
+</pre>
+
+h3. Polling and processing events via a pipeline
+
+You can poll Mingle for new events at regular intervals and process those events to do things such as filter to events you are interested in and post them to other systems with which you wish to integrate Mingle. This example posts all new comments to an HTTP end point. You'd need to use cron or a similar scheduler to run it at regular intervals. 
 
-# assemble processing pipeline
+<pre>
 post_comments_to_another_service = MingleEvents::Processors::Pipeline.new([
   MingleEvents::Processors::CategoryFilter.new([MingleEvents::Feed::Category::COMMENT_ADDITION]),
   MingleEvents::Processors::HttpPostPublisher.new('http://localhost:4567/')
 ])
-
-# poll once
 MingleEvents::Poller.new(mingle_access, {'test_project' => [post_comments_to_another_service]}).run_once  
 </pre>
 
-h2. High level design
+A more detailed dive into the polling and processing design follows.
+
+h2. Polling and processing
 
-This library pumps the stream of a Mingle project's events through a pipeline of processors that you specify. The processors can do things such as "filter out any events that are not sourced from a Story" or "post an event to an HTTP end-point." 
+h3. High level design
+
+The Poller class can pump the stream of a Mingle project's events through a pipeline of processors that you specify. The processors can do things such as "filter out any events that are not sourced from a Story" or "post an event to an HTTP end-point." 
 
 !http://thoughtworksstudios.github.com/mingle_events_design.png!
 
 As stated in the opening paragraph, the aim of this library is to hide the mechanics of event polling, making the user's focus solely the definition of the processing pipeline. This library supplies fundamental event processors, such as card type filters, atom category filters, and http publishers. This library should also make it easy for you to write custom processors.
 
-h2. Events and entries
-
-You might get confused looking at the source code as to what's an Atom entry and what's a Mingle event. We're still trying to clean that up a bit, but for all intents and purposes, they are the same thing. The Atom feed represents Mingle events in the form of Atom entries. For the most part we try to use the word 'entry' in the context of the feed and 'event' in the context of processing.
 
-h2. Processors, filters, and pipelines
+h3. Processors, filters, and pipelines
 
 Processors, filters, and pipelines are all processors with the same interface. The fundamental model for pipes and filters, or pipelining, is that there is a single, common interface for processing input and returning output. In this context of Mingle event processing, the interface is basically "events in, events out" where "events in" is the list of unprocessed events and "events out" are the processed events.  Processed events might be enriched, filtered, untouched but emailed, etc. 
 
@@ -56,10 +91,11 @@ This library ships the following processors:
 * MingleEvents::Processors::CardData -- loads data for each card that sourced an event in the current stream. This processor requires some special handling (see next section).
 * MingleEvents::Processors::CardTypeFilter -- filters events to those sourced by cards of specific card type(s)
 * MingleEvents::Processors::CategoryFilter -- filters events to those with specified Atom categories. Mingle's Atom Categories are specified in MingleEvents::Category
+* MingleEvents::Processors::CustomPropertyFilter -- filters events to those with the specified value of a single project-level custom property
 * MingleEvents::Processors::HttpPostPublisher -- posts event's raw XML to an HTTP endpoint
 * MingleEvents::Processors::Pipeline -- manages to processing of events by a sequence of processors
 
-h2. Card Data
+h3. Card Data
 
 CardData is a special processor in that it implements a second interface, beyond event processing. This interface is one that allows the lookup of data for the card that sourced the event (if the event was actually sourced by a card). As looking up card data requires accessing additional Mingle server resources, you want to take special care that you don't make repeated requests for the same resources. If you have multiple processors requiring CardData, be sure to use a single instance of CardData across your entire pipeline.
 
@@ -76,9 +112,7 @@ post_commenting_on_high_priority_bugs_and_stories = MingleEvents::Processors::Pi
 
 Note that CardData will provide data for the version of the card that was created by the event you are processing and *not* the current version of the card.  
 
-(For users familiar with earlier versions of this library, you will notice that the bulk loading of card data is currently not being taken advantage of. This is because I needed to change some internals of event broadcasting such that processing a big project's entire history did not cause OOM issues. I will very soon restore the bulk loading optimization for when the poller is simply reading the last handful of events. CardData still works, you just might see your client making a few more calls to Mingle.)
-
-h2. Writing your own processor
+h3. Writing your own processor
 
 In ruby code, the processing interface is a single method named 'process_events' that has a single parameter, the list of unprocessed events' and returns a list of the processed events. 
 
@@ -110,24 +144,11 @@ Be absolutely sure that any processor you write returns a list of events. If you
 
 Each event that is passed to the processor is an instance of type MingleEvents::Entry which is a Ruby wrapper around an Atom event. The Entry class makes it easy to access information such as author, Atom categories, whether the event was sourced by a card, etc. As the model is not yet complete, the Entry class also exposes the raw XML of the entry.
 
-(If you are wondering why the interface for processing is for a list of events rather than a single event, it is because of the above mentioned bulk loading of card data. This may change in the future. CardData may get its own special interface to perform a pre-processing bulk load, and the event processing interface will move to a single event at a time.)
-
-h2. Retry & Error handling
+h3. Retry & Error handling
 
 As of now, retry is not implemented. If an error occurs during event processing, the error will be logged and processing will stop. The next run will re-start at the point of the last error.
 
-h2. Historical analysis
-
-One of the great usages of event playback is historical analysis. E.g., if you wanted to see the number of stories added each day to your project, you could write a processor that counts new stories and takes a snapshot at the end of each day, where the end of the day is the first time you see an event for the next day. If you are doing this sort of analysis you will not want to poll the entire project's event history every time as that is quite time consuming. 
-
-It is possible to repeatedly run analysis against previously fetched events using the ProjectEventFetcher class that is only used in the internals of the standard polling mechanism.  ProjectEventFetcher does most of the real work of reading new events from the server and making them available for local processing, so it's not a bad class to get to know.
-
-<pre>
-event_fetcher = MingleEvents::ProjectEventFetcher.new('my_project', mingle_access)
-event_fetcher.all_fetched_entries.each do |e|
-  # do something interesting with each event
-end  
-</pre>
+h2. Events and entries
 
-Until there's time for further documentation take a look at card_count_by_day.rb and story_count_by_day.rb in the examples folder.
+You might get confused looking at the source code, documentation, etc. as to what's an Atom entry and what's a Mingle event. We're still trying to clean that up a bit, but for all intents and purposes, they are the same thing. The Atom feed represents Mingle events in the form of Atom entries. For the most part we try to use the word 'entry' in the context of the feed and 'event' in the context of processing, but there's still cleanup to be done.
 

data/lib/mingle_events/poller.rb

@@ -17,10 +17,8 @@ module MingleEvents
       @processors_by_project_identifier.each do |project_identifier, processors|
         fetcher = ProjectEventFetcher.new(project_identifier, @mingle_access)
         fetcher.reset if options[:clean]
-        fetcher.fetch_latest.each do |entry|
-          MingleEvents.log.info("About to process event #{entry.entry_id}...")
-          processors.each{|p| p.process_events([entry])}
-        end
+        latest_events = fetcher.fetch_latest.to_a
+        processors.each{|p| p.process_events(latest_events)}        
       end
     end
     

data/lib/mingle_events/processors.rb

@@ -1,5 +1,7 @@
 require 'cgi'
 
+require File.expand_path(File.join(File.dirname(__FILE__), 'processors', 'filter'))
+require File.expand_path(File.join(File.dirname(__FILE__), 'processors', 'processor'))
 require File.expand_path(File.join(File.dirname(__FILE__), 'processors', 'card_data'))
 require File.expand_path(File.join(File.dirname(__FILE__), 'processors', 'category_filter'))
 require File.expand_path(File.join(File.dirname(__FILE__), 'processors', 'card_type_filter'))

data/lib/mingle_events/processors/author_filter.rb

@@ -2,7 +2,7 @@ module MingleEvents
   module Processors
     
     # Removes all events from stream not triggered by the specified author
-    class AuthorFilter
+    class AuthorFilter < Filter
     
       def initialize(spec, mingle_access, project_identifier)
         unless spec.size == 1
@@ -12,10 +12,10 @@ module MingleEvents
         @author_spec = AuthorSpec.new(spec, mingle_access, project_identifier)
       end
     
-      def process_events(events)
-        events.select{|event| @author_spec.event_triggered_by?(event)}
+      def match?(event)
+        @author_spec.event_triggered_by?(event)
       end
-      
+          
       class AuthorSpec
         
         def initialize(spec, mingle_access, project_identifier)

data/lib/mingle_events/processors/card_data.rb

@@ -42,13 +42,35 @@ module MingleEvents
       
       def load_bulk_card_data
         @card_data_by_number_and_version = {}
-        
-        # TODO: figure out max number of card numbers before we have to chunk this up.  or does mingle
-        # figure out how to handle a too-large IN clause?
+                
         card_numbers = @card_events.map(&:card_number).uniq
         path = "/api/v2/projects/#{@project_identifier}/cards/execute_mql.xml?mql=WHERE number IN (#{card_numbers.join(',')})"
         
-        raw_xml = @mingle_access.fetch_page(URI.escape(path))
+        # TODO: figure out whether it makes sense to chunk a large count of card numbers
+        # into multiple requests so that the MQL "IN" clause doesn't explode. For now, we'll
+        # just punt by logging the error and letting the individual card data load explode
+        # if there's a real problem. In most polling scenarios, this is a highly unlikely
+        # problem as there will usually be 1 or a few events.
+        begin
+          raw_xml = @mingle_access.fetch_page(URI.escape(path))
+        rescue
+          msg = %{
+
+There was an error while attempting bulk load of card data. 
+Individual data loads for each card will still be attempted.
+
+Root cause: 
+
+#{$!.message}
+
+Stack Trace: 
+
+#{($!.backtrace || []).join("\n")}
+
+}
+          MingleEvents.log.info(msg)
+          return
+        end
         doc = Nokogiri::XML(raw_xml)
         
         doc.search('/results/result').map do |card_result|

data/lib/mingle_events/processors/card_type_filter.rb

@@ -8,21 +8,19 @@ module MingleEvents
     # and this filtering, the event will be filtered as there is no means
     # to determine its type. Therefore, it's recommended to also
     # subscribe a 'CardDeleted' processor to the same project.    
-    class CardTypeFilter
+    class CardTypeFilter < Filter
     
       def initialize(card_types, card_data)
         @card_types = card_types
         @card_data = card_data
       end
-    
-      def process_events(events)
-        events.select do |event|
-          event.card? && 
-            @card_data.for_card_event(event) &&
-            @card_types.include?(@card_data.for_card_event(event)[:card_type_name])
-        end
+      
+      def match?(event)
+        event.card? && 
+          @card_data.for_card_event(event) &&
+          @card_types.include?(@card_data.for_card_event(event)[:card_type_name])
       end
-
+     
     end
   end
 end

data/lib/mingle_events/processors/category_filter.rb

@@ -2,18 +2,16 @@ module MingleEvents
   module Processors
     
     # Removes events from the stream that do not match all of the specified categories
-    class CategoryFilter
+    class CategoryFilter < Filter
     
       def initialize(categories)
         @categories = categories
       end
-    
-      def process_events(events)
-        events.select do |event|
-          @categories.all?{|c| event.categories.include?(c)}
-        end
+      
+      def match?(event)
+        @categories.all?{|c| event.categories.include?(c)}
       end
-    
+        
     end
   end
 end

data/lib/mingle_events/processors/custom_property_filter.rb

@@ -9,22 +9,20 @@ module MingleEvents
     # and this filtering, the event will be filtered as there is no means
     # to determine its type. Therefore, it's recommended to also
     # subscribe a 'CardDeleted' processor to the same project.    
-    class CustomPropertyFilter
+    class CustomPropertyFilter < Filter
     
       def initialize(property_name, property_value, card_data)
         @property_name = property_name
         @property_value = property_value
         @card_data = card_data
       end
-    
-      def process_events(events)
-        events.select do |event|
-          event.card? && 
-            @card_data.for_card_event(event) &&
-            @property_value == @card_data.for_card_event(event)[:custom_properties][@property_name]
-        end
+      
+      def match?(event)
+        event.card? && 
+          @card_data.for_card_event(event) &&
+          @property_value == @card_data.for_card_event(event)[:custom_properties][@property_name]
       end
-
+    
     end
   end
 end

data/lib/mingle_events/processors/filter.rb

@@ -0,0 +1,17 @@
+module MingleEvents
+  module Processors
+
+    class Filter
+    
+      def process_events(events)
+        events.select{|e| match?(e)}
+      end
+      
+      def match?(event)
+        raise "Subclass responsibility!"
+      end
+      
+    end
+    
+  end
+end

data/lib/mingle_events/processors/http_post_publisher.rb

@@ -1,17 +1,13 @@
 module MingleEvents
   module Processors
     
-    class HttpPostPublisher
+    class HttpPostPublisher < Processor
 
       def initialize(url)
         @url = url
       end
       
-      def process_events(events)
-        events.map{|e| process_event(e)}
-      end
-
-      def process_event(event) 
+      def process(event) 
         Net::HTTP.post_form(URI.parse(@url), {'event' => event.raw_xml}).body
       end
     

data/lib/mingle_events/processors/pipeline.rb

@@ -17,4 +17,5 @@ module MingleEvents
       end
     end
   end
+  
 end

data/lib/mingle_events/processors/processor.rb

@@ -0,0 +1,17 @@
+module MingleEvents
+  module Processors
+
+    class Processor
+    
+      def process_events(events)
+        events.map{|e| process(e)}
+      end
+      
+      def process(event)
+        raise "Subclass responsibility!"
+      end
+      
+    end
+    
+  end
+end

data/lib/mingle_events/processors/puts_publisher.rb

@@ -2,13 +2,9 @@ module MingleEvents
   module Processors
     
     # Writes each event in stream to stdout, mostly for demonstration purposes
-    class PutsPublisher
+    class PutsPublisher < Processor
       
-      def process_events(events)
-        events.map{|e| process_event(e)}
-      end
-
-      def process_event(event) 
+      def process(event) 
         puts "Processing event #{event}"
       end
     

data/lib/mingle_events/project_event_fetcher.rb

@@ -1,6 +1,10 @@
 module MingleEvents
   
   # fetch all unseen events and write them to disk for future processing
+  # 
+  # this class is messy and needs some cleanup, but some things are in here
+  # for a reason. specifically, for historical analysis, we can process each event,
+  # one at at time, reading it off disk, to avoid massive memory consumption.
   class ProjectEventFetcher
     
     def initialize(project_identifier, mingle_access, state_dir=nil)
@@ -16,9 +20,19 @@ module MingleEvents
       FileUtils.rm_rf(@state_dir)
     end
     
+    # setup fetcher to only fetch new events, occuring beyond "right now"
+    def reset_to_now
+      return if last_entry_fetched
+      
+      latest_event = page_with_latest_entries.entries.first
+      return if latest_event.nil?
+      write_entry_to_disk(latest_event, nil)
+      update_current_state(latest_event, latest_event)
+    end
+    
     # fetch the latest events from mingle, i.e., the ones not previously seen
     def fetch_latest
-      page = Feed::Page.new("/api/v2/projects/#{@project_identifier}/feeds/events.xml", @mingle_access)
+      page = page_with_latest_entries
       most_recent_new_entry = page.entries.first
       last_fetched_entry = load_last_fetched_entry
       last_fetched_entry_seen = false      
@@ -79,6 +93,10 @@ module MingleEvents
            
     private
     
+    def page_with_latest_entries
+      Feed::Page.new("/api/v2/projects/#{@project_identifier}/feeds/events.xml", @mingle_access)
+    end
+    
     def current_state_entry(info_file_key)
       if info_file = load_current_state[info_file_key]
         Feed::Entry.from_snippet(YAML.load(File.new(info_file))[:entry_xml])

data/test/mingle_events/feed/changes_test.rb

@@ -270,75 +270,6 @@ module MingleEvents
         assert_nil(change[:new_value])
       end  
       
-      def test_foo
-        foo = %{
-          <entry>
-              <id>https://mingle09.thoughtworks.com/projects/mingle/events/index/1344945</id>
-              <title>story #67 CRUD Project created</title>
-              <updated>2006-11-13T05:45:06Z</updated>
-              <author>
-                <name>Jon Tirsen</name>
-                <email>jtirsen@thoughtworks.com</email>
-                <uri>https://mingle09.thoughtworks.com/api/v2/users/10040.xml</uri>
-              </author>
-              <link href="https://mingle09.thoughtworks.com/api/v2/projects/mingle/cards/67.xml" rel="http://www.thoughtworks-studios.com/ns/mingle#event-source" type="application/vnd.mingle+xml" title="story #67"/>
-              <link href="https://mingle09.thoughtworks.com/projects/mingle/cards/67" rel="http://www.thoughtworks-studios.com/ns/mingle#event-source" type="text/html" title="story #67"/>
-              <link href="https://mingle09.thoughtworks.com/api/v2/projects/mingle/cards/67.xml?version=1" rel="http://www.thoughtworks-studios.com/ns/mingle#version" type="application/vnd.mingle+xml" title="story #67 (v1)"/>
-              <link href="https://mingle09.thoughtworks.com/projects/mingle/cards/67?version=1" rel="http://www.thoughtworks-studios.com/ns/mingle#version" type="text/html" title="story #67 (v1)"/>
-              <category term="card" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <category term="card-creation" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <category term="card-type-change" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <category term="description-change" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <category term="name-change" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <category term="property-change" scheme="http://www.thoughtworks-studios.com/ns/mingle#categories"/>
-              <content type="application/vnd.mingle+xml">
-                <changes xmlns="http://www.thoughtworks-studios.com/ns/mingle">
-                  <change type="card-creation"/>
-                  <change type="card-type-change">
-                    <old_value nil="true"/>
-                    <new_value>
-                      <card_type url="https://mingle09.thoughtworks.com/api/v2/projects/mingle/card_types/10134.xml">
-                        <name>story</name>
-                      </card_type>
-                    </new_value>
-                  </change>
-                  <change type="description-change">
-                  </change>
-                  <change type="name-change">
-                    <old_value nil="true"/>
-                    <new_value>CRUD Project</new_value>
-                  </change>
-                  <change type="property-change">
-                    <property_definition url="https://mingle09.thoughtworks.com/api/v2/projects/mingle/property_definitions/10380.xml">
-                      <name>release</name>
-                      <position nil="true"/>
-                      <data_type>string</data_type>
-                      <is_numeric type="boolean">false</is_numeric>
-                    </property_definition>
-                    <old_value nil="true"/>
-                    <new_value>0.92</new_value>
-                  </change>
-                  <change type="property-change">
-                    <property_definition url="https://mingle09.thoughtworks.com/api/v2/projects/mingle/property_definitions/10381.xml">
-                      <name>Priority</name>
-                      <position nil="true"/>
-                      <data_type>string</data_type>
-                      <is_numeric type="boolean">false</is_numeric>
-                    </property_definition>
-                    <old_value nil="true"/>
-                    <new_value>Should</new_value>
-                  </change>
-                </changes>
-              </content>
-            </entry>
-        }
-        
-        entry = Entry.from_snippet(foo)
-        entry.changes.each do |c|
-          puts c
-        end
-      end 
-      
     end        
   end
 end

data/test/mingle_events/processors/author_filter_test.rb

@@ -33,9 +33,8 @@ module MingleEvents
         @event_1 = stub_event(1, {:uri => "http://example.com/users/10.xml", :login => 'ctester'})
         @event_2 = stub_event(2, {:uri => "http://example.com/users/17.xml", :login => 'jdeveloper'})
         @event_3 = stub_event(3, {:uri => "http://example.com/users/10.xml", :login => 'ctester'})
-        @unprocessed_events = [@event_1, @event_2, @event_3]
       end
-      
+            
       def test_filter_can_only_be_constructed_with_a_single_criteria
         begin
           AuthorFilter.new({:url => 'foo', :email => 'bar'}, nil, nil)
@@ -45,30 +44,24 @@ module MingleEvents
                end
       end
   
-      def test_filter_by_author_url
+      def test_match_on_author_url
         author_filter = AuthorFilter.new({:url => 'http://example.com/users/10.xml'}, @dummy_mingle_access, 'atlas')
-        filtered_events = author_filter.process_events(@unprocessed_events)
-        assert_equal([@event_1, @event_3], filtered_events)
+        assert author_filter.match?(@event_1)
+        assert !author_filter.match?(@event_2)
       end
       
-      def test_filter_by_author_login
+      def test_match_on_author_login
         author_filter = AuthorFilter.new({:login => 'ctester'}, @dummy_mingle_access, 'atlas')
-        filtered_events = author_filter.process_events(@unprocessed_events)
-        assert_equal([@event_1, @event_3], filtered_events)
+        assert author_filter.match?(@event_1)
+        assert !author_filter.match?(@event_2)
       end
       
-      def test_filter_by_author_email
+      def test_match_on_author_email
         author_filter = AuthorFilter.new({:email => 'joe.developer@example.com'}, @dummy_mingle_access, 'atlas')
-        filtered_events = author_filter.process_events(@unprocessed_events)
-        assert_equal([@event_2], filtered_events)
-      end
-      
-      def test_filter_returns_empty_list_when_no_match
-        author_filter = AuthorFilter.new({:email => 'sammy.soso@example.com'}, @dummy_mingle_access, 'atlas')
-        filtered_events = author_filter.process_events(@unprocessed_events)
-        assert_equal([], filtered_events)
+        assert !author_filter.match?(@event_1)
+        assert author_filter.match?(@event_2)
       end
-      
+            
       private
       
       def stub_event(entry_id, author)

data/test/mingle_events/processors/card_data_test.rb

@@ -188,6 +188,29 @@ module MingleEvents
         
         assert_nil(card_data.for_card_event(event_1))
       end
+      
+      def test_survives_bulk_load_exploding
+        event = stub_event(4, 103, 13, ['card', 'property-change'])
+    
+        dummy_mingle_access = StubMingleAccess.new
+        dummy_mingle_access.register_explosion(URI.escape('/api/v2/projects/atlas/cards/execute_mql.xml?mql=WHERE number IN (103)'))
+
+        dummy_mingle_access.register_page_content('http://example.com?version=13',%{
+          <card>
+            <number type="integer">103</number> 
+            <card_type url="https://localhost:7071/api/v2/projects/atlas/card_types/21.xml"> 
+              <name>epic</name> 
+            </card_type> 
+            <version type="integer">13</version> 
+          </card>
+        })
+    
+        card_data = CardData.new(dummy_mingle_access, 'atlas')
+        card_data.process_events([event])
+        
+        assert_correct_basic_card_data_for_event({:number => 103, :card_type_name => 'epic', :version => 13}, card_data, event)
+      end
+      
             
       private 
   

data/test/mingle_events/processors/card_type_filter_test.rb

@@ -4,40 +4,38 @@ module MingleEvents
   module Processors
     
     class CardTypeFilterTest < Test::Unit::TestCase
-  
-      def test_filters_events_on_card_type
-        event_1 = stub_event(true)
-        event_2 = stub_event(false)
-        event_3 = stub_event(true)
-        event_4 = stub_event(true)
-        event_5 = stub_event(true)
+      
+      def setup      
+        @story_event = stub_event(true)
+        @page_event = stub_event(false)
+        @bug_event = stub_event(true)
+        @issue_event = stub_event(true)
         
-        card_data = {
-          event_1 => {:card_type_name => 'story'},
-          event_3 => {:card_type_name => 'bug'},
-          event_4 => {:card_type_name => 'story'},
-          event_5 => {:card_type_name => 'issue'}
+        @card_data = {
+          @story_event => {:card_type_name => 'story'},
+          @bug_event => {:card_type_name => 'bug'},
+          @issue_event => {:card_type_name => 'issue'}
         }
-        def card_data.for_card_event(card_event)
+        def @card_data.for_card_event(card_event)
           self[card_event]
         end
         
-        filter = CardTypeFilter.new(['story', 'issue'], card_data)
-        filtered_events = filter.process_events([event_1, event_2, event_3, event_4, event_5])
-        assert_equal([event_1, event_4, event_5], filtered_events)
+        @filter = CardTypeFilter.new(['story', 'issue'], @card_data)
       end
       
-      def test_drops_events_for_deleted_cards
-        event_1 = stub_event(true)
-        
-        card_data = {}
-        def card_data.for_card_event(card_event)
-          self[card_event]
-        end
-        
-        filter = CardTypeFilter.new(['story', 'issue'], card_data)
-        filtered_events = filter.process_events([event_1])
-        assert_equal([], filtered_events)
+      def test_does_not_match_non_card_events
+        assert !@filter.match?(@page_event)
+      end
+  
+      def test_match_on_card_type
+        assert @filter.match?(@story_event)
+        assert @filter.match?(@issue_event)
+        assert !@filter.match?(@bug_event)
+      end
+      
+      def test_does_not_match_deleted_cards
+        @card_data[@story_event] = nil
+        assert !@filter.match?(@story_event)
       end
       
       private

data/test/mingle_events/processors/category_filter_test.rb

@@ -3,17 +3,20 @@ require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'test_hel
 module MingleEvents
   module Processors
     class CategoryFilterTest < Test::Unit::TestCase
-  
-      def test_removes_events_without_matching_categories
-        event_1 = stub_event(1, [Feed::Category::CARD, Feed::Category::COMMENT_ADDITION])
-        event_2 = stub_event(2, [Feed::Category::CARD, Feed::Category::PROPERTY_CHANGE])
-        event_3 = stub_event(3, [Feed::Category::REVISION_COMMIT])
-        event_4 = stub_event(4, [Feed::Category::CARD, Feed::Category::PROPERTY_CHANGE])
-        event_5 = stub_event(5, [])
-        events = [event_1, event_2, event_3, event_4, event_5]
-    
-        filter = CategoryFilter.new([Feed::Category::CARD, Feed::Category::PROPERTY_CHANGE])
-        assert_equal([event_2, event_4], filter.process_events(events))
+        
+      def test_match_against_one_category
+        filter = CategoryFilter.new([Feed::Category::CARD])
+        assert filter.match?(stub_event(1, [Feed::Category::CARD, Feed::Category::COMMENT_ADDITION]))
+        assert filter.match?(stub_event(1, [Feed::Category::CARD]))
+        assert !filter.match?(stub_event(1, [Feed::Category::COMMENT_ADDITION]))
+        assert !filter.match?(stub_event(1, [Feed::Category::REVISION_COMMIT, Feed::Category::COMMENT_ADDITION]))
+      end
+      
+      def test_match_against_multiple_categories
+        filter = CategoryFilter.new([Feed::Category::CARD, Feed::Category::COMMENT_ADDITION])
+        assert filter.match?(stub_event(1, [Feed::Category::CARD, Feed::Category::COMMENT_ADDITION]))
+        assert !filter.match?(stub_event(1, [Feed::Category::CARD]))
+        assert !filter.match?(stub_event(1, [Feed::Category::REVISION_COMMIT, Feed::Category::COMMENT_ADDITION]))
       end
       
       private 

data/test/mingle_events/processors/custom_property_filter_test.rb

@@ -5,41 +5,35 @@ module MingleEvents
     
     class CustomPropertyFilterTest < Test::Unit::TestCase
   
-      def test_filters_events_on_custom_property
-        event_1 = stub_event(true)
-        event_2 = stub_event(false)
-        event_3 = stub_event(true)
-        event_4 = stub_event(true)
-        event_5 = stub_event(true)
+      def setup
+        @high_priority_event = stub_event(true)
+        @page_event = stub_event(false)
+        @low_priority_event = stub_event(true)
+        @high_severity_event = stub_event(true)
         
-        card_data = {
-          event_1 => {:custom_properties => {'Priority' => 'High'}},
-          event_3 => {:custom_properties => {'Priority' => 'Low'}},
-          event_4 => {:custom_properties => {'Priority' => 'High'}},
-          event_5 => {:custom_properties => {'Severity' => 'High'}}
+        @card_data = {
+          @high_priority_event => {:custom_properties => {'Priority' => 'High'}},
+          @low_priority_event => {:custom_properties => {'Priority' => 'Low'}},
+          @high_severity_event => {:custom_properties => {'Severity' => 'High'}}
         }
-        def card_data.for_card_event(card_event)
+        def @card_data.for_card_event(card_event)
           self[card_event]
         end
         
-        filter = CustomPropertyFilter.new('Priority', 'High', card_data)
-        filtered_events = filter.process_events([event_1, event_2, event_3, event_4, event_5])
-        assert_equal([event_1, event_4], filtered_events)
+        @filter = CustomPropertyFilter.new('Priority', 'High', @card_data)
       end
-      
-      def test_drops_events_for_deleted_cards
-        event_1 = stub_event(true)
-        
-        card_data = {}
-        def card_data.for_card_event(card_event)
-          self[card_event]
-        end
-        
-        filter = CustomPropertyFilter.new('Priority', 'High', card_data)
-        filtered_events = filter.process_events([event_1])
-        assert_equal([], filtered_events)
+
+      def test_match_on_property_value
+        assert @filter.match?(@high_priority_event)
+        assert !@filter.match?(@low_priority_event)
+        assert !@filter.match?(@high_severity_event)
       end
       
+      def test_does_not_match_delete_card
+        @card_data[@high_priority_event] = nil
+        assert !@filter.match?(@high_priority_event)
+      end
+              
       private
       
       def stub_event(is_card)

data/test/mingle_events/processors/filter_test.rb

@@ -0,0 +1,19 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'test_helper'))
+
+module MingleEvents
+  module Processors
+    class FilterTest < Test::Unit::TestCase  
+      
+      def test_returns_events_that_match
+        assert_equal([0,2,4], MatchEvenFilter.new.process_events([0,1,2,3,4,5]))
+      end
+      
+      class MatchEvenFilter < Filter
+        def match?(event)
+          event % 2 == 0
+        end
+      end
+
+    end
+  end
+end

data/test/mingle_events/processors/processor_test.rb

@@ -0,0 +1,19 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'test_helper'))
+
+module MingleEvents
+  module Processors
+    class ProcessorTest < Test::Unit::TestCase  
+      
+      def test_returns_events_that_match
+        assert_equal([0,2,4], DoubleProcessor.new.process_events([0,1,2]))
+      end
+      
+      class DoubleProcessor < Processor
+        def process(event)
+          event * 2
+        end
+      end
+
+    end
+  end
+end

data/test/mingle_events/project_event_fetcher_test.rb

@@ -56,6 +56,107 @@ module MingleEvents
       assert_nil fetcher.last_entry_fetched
     end
     
+    def test_reset_to_now_when_project_has_previous_history
+      state_dir = temp_dir
+      mingle_access = stub_mingle_access
+      fetcher = ProjectEventFetcher.new('atlas', mingle_access, state_dir)
+
+      fetcher.reset_to_now
+      assert fetcher.fetch_latest.to_a.empty?
+      
+      mingle_access.register_page_content('/api/v2/projects/atlas/feeds/events.xml',%{
+        <feed xmlns="http://www.w3.org/2005/Atom" xmlns:mingle="http://www.thoughtworks-studios.com/ns/mingle">
+
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="current"/>
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="self"/>
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml?page=2" rel="next"/>
+
+          <entry>
+            <id>https://mingle.example.com/projects/atlas/events/index/104</id>
+            <title>entry 104</title>
+            <updated>2011-02-03T08:14:42Z</updated>
+            <author><name>Bob</name></author>
+          </entry>
+          <entry>
+            <id>https://mingle.example.com/projects/atlas/events/index/103</id>
+            <title>entry 103</title>
+            <updated>2011-02-03T08:12:42Z</updated>
+            <author><name>Bob</name></author>
+          </entry>
+        </feed>
+      })
+            
+      assert_equal([entry(104)], fetcher.fetch_latest.to_a)
+    end
+    
+    def test_reset_to_now_is_ignored_if_there_is_already_local_current_state
+      state_dir = temp_dir
+      mingle_access = stub_mingle_access
+      fetcher = ProjectEventFetcher.new('atlas', mingle_access, state_dir)
+      fetcher.fetch_latest  # bring current state up to 103
+            
+      mingle_access.register_page_content('/api/v2/projects/atlas/feeds/events.xml',%{
+        <feed xmlns="http://www.w3.org/2005/Atom" xmlns:mingle="http://www.thoughtworks-studios.com/ns/mingle">
+
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="current"/>
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="self"/>
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml?page=2" rel="next"/>
+
+          <entry>
+            <id>https://mingle.example.com/projects/atlas/events/index/104</id>
+            <title>entry 104</title>
+            <updated>2011-02-03T08:14:42Z</updated>
+            <author><name>Bob</name></author>
+          </entry>
+          <entry>
+            <id>https://mingle.example.com/projects/atlas/events/index/103</id>
+            <title>entry 103</title>
+            <updated>2011-02-03T08:12:42Z</updated>
+            <author><name>Bob</name></author>
+          </entry>
+        </feed>
+      })
+            
+      fetcher.reset_to_now # if not ignored, next call would return no events rather than 104
+      assert_equal([entry(104)], fetcher.fetch_latest.to_a)
+    end
+    
+    def test_reset_to_now_when_project_has_no_previous_history
+      state_dir = temp_dir
+      mingle_access = StubMingleAccess.new
+      mingle_access.register_page_content('/api/v2/projects/atlas/feeds/events.xml',%{
+<?xml version="1.0" encoding="UTF-8"?>
+<feed xmlns="http://www.w3.org/2005/Atom" xmlns:mingle="http://www.thoughtworks-studios.com/ns/mingle">
+  <title>Mingle Events: Blank Project</title>
+  <id>https://mingle.example.com/api/v2/projects/blank_project/feeds/events.xml</id>
+  <link href="https://mingle.example.com/api/v2/projects/blank_project/feeds/events.xml" rel="current"/>
+  <link href="https://mingle.example.com/api/v2/projects/blank_project/feeds/events.xml" rel="self"/>
+  <updated>2011-08-04T19:42:04Z</updated>
+</feed>})
+      fetcher = ProjectEventFetcher.new('atlas', mingle_access, state_dir)
+      fetcher.fetch_latest
+      
+      fetcher.reset_to_now
+      assert fetcher.fetch_latest.to_a.empty?
+      
+      mingle_access.register_page_content('/api/v2/projects/atlas/feeds/events.xml',%{
+        <feed xmlns="http://www.w3.org/2005/Atom" xmlns:mingle="http://www.thoughtworks-studios.com/ns/mingle">
+
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="current"/>
+          <link href="https://mingle.example.com/api/v2/projects/atlas/feeds/events.xml" rel="self"/>
+
+          <entry>
+            <id>https://mingle.example.com/projects/atlas/events/index/104</id>
+            <title>entry 104</title>
+            <updated>2011-02-03T08:14:42Z</updated>
+            <author><name>Bob</name></author>
+          </entry>
+        </feed>
+      })
+            
+      assert_equal([entry(104)], fetcher.fetch_latest.to_a)
+    end
+    
     private
     
     def setup_current_state(first_entry_id, second_entry_id, last_entry_id, fetcher)    

data/test/test_helper.rb

@@ -119,6 +119,7 @@ class Test::Unit::TestCase
     def initialize
       @pages_by_path = {}
       @not_found_pages = []
+      @exploding_pages = []
     end
     
     def base_url
@@ -132,6 +133,10 @@ class Test::Unit::TestCase
     def register_page_not_found(path)
       @not_found_pages << path
     end
+    
+    def register_explosion(path)
+      @exploding_pages << path
+    end
 
     def fetch_page(path)
       if @not_found_pages.include?(path)
@@ -141,6 +146,14 @@ class Test::Unit::TestCase
         end
         raise MingleEvents::HttpError.new(rsp, path)
       end
+      
+      if @exploding_pages.include?(path)
+        rsp = Net::HTTPNotFound.new(nil, '500', 'Server exploded!')
+        def rsp.body
+          "500!!!!!"
+        end
+        raise MingleEvents::HttpError.new(rsp, path)
+      end
 
       raise "Attempting to fetch page at #{path}, but your test has not registered content for this path! Registered paths: #{@pages_by_path.keys.inspect}" unless @pages_by_path.key?(path)
       @pages_by_path[path]

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: mingle_events
 version: !ruby/object:Gem::Version 
-  hash: 17
+  hash: 27
   prerelease: 
   segments: 
   - 0
+  - 1
   - 0
-  - 7
-  version: 0.0.7
+  version: 0.1.0
 platform: ruby
 authors: 
 - David Rice
@@ -72,8 +72,10 @@ files:
 - lib/mingle_events/processors/card_type_filter.rb
 - lib/mingle_events/processors/category_filter.rb
 - lib/mingle_events/processors/custom_property_filter.rb
+- lib/mingle_events/processors/filter.rb
 - lib/mingle_events/processors/http_post_publisher.rb
 - lib/mingle_events/processors/pipeline.rb
+- lib/mingle_events/processors/processor.rb
 - lib/mingle_events/processors/puts_publisher.rb
 - lib/mingle_events/processors.rb
 - lib/mingle_events/project_custom_properties.rb
@@ -92,7 +94,9 @@ files:
 - test/mingle_events/processors/card_type_filter_test.rb
 - test/mingle_events/processors/category_filter_test.rb
 - test/mingle_events/processors/custom_property_filter_test.rb
+- test/mingle_events/processors/filter_test.rb
 - test/mingle_events/processors/pipeline_test.rb
+- test/mingle_events/processors/processor_test.rb
 - test/mingle_events/project_custom_properties_test.rb
 - test/mingle_events/project_event_fetcher_test.rb
 - test/test_helper.rb