conveyor 0.2.0 → 0.2.1

data/History.txt

@@ -1,3 +1,6 @@
+== 0.2.1 / 2008-02-29
+* added get by timestamp and rewind to timestamp (for groups, too)
+
 == 0.2.0 / 2008-02-26
 
 * switched from using Mongrel to Thin. this adds a dependency on thin, which depends on Event Machine

data/Manifest.txt

@@ -5,8 +5,8 @@ README.txt
 Rakefile
 bin/conveyor
 bin/conveyor-upgrade
-docs/file-formats.mkd
-docs/protocol.mkd
+docs/file-formats.rdoc
+docs/protocol.rdoc
 lib/conveyor.rb
 lib/conveyor/base_channel.rb
 lib/conveyor/channel.rb

data/Rakefile

@@ -14,4 +14,4 @@ Hoe.new('conveyor', Conveyor::VERSION) do |p|
   p.extra_deps << ['daemons']
 end
 
-# vim: syntax=Ruby
+# vim: syntax=Ruby

data/docs/{file-formats.mkd → file-formats.rdoc}

@@ -1,15 +1,16 @@
-= FILE FORMATS
-==  DATA FILES
+= File Formats
+== Data Files
 
-id time offset length hash
-content
-...
+ id timestamp offset length hash flags
+ content
+ ...
 
 contrived example:
 
-1213124 2008-01-05T13:35:32 1234 11 asdfasdfasdfasdfasdfasdfasdfa
-foo bar bam
+ q01w 3mp0a6g ya 11 asdfasdfasdfasdfasdfasdfasdfa 0
+ foo bar bam
 
+* all integers encoded in base-36
 * space separated line of metadata followed by content
 * delimiter might be useful for sanity checking, but the hash could probably suffice for ensuring that the offset was calculated and persisted properly. We should look at what ARC does here.
 * offset is to beginning of metadata line
@@ -17,15 +18,15 @@ foo bar bam
 
 === INDEX FILES
 
-id time offset length hash file
+id time offset length hash flags file
 
 contrived example:
 
-1213124 2008-01-05T13:35:32 1234 11 asdfasdfasdfasdfasdfasdfasdfa 1
+ q01w 3mp0a6g ya 11 asdfasdfasdfasdfasdfasdfasdfa 0 1
 
 notes:
-* 1 is the filename
-* assuming a lucene-style directory of datafiles + ToC/index
+* '1' is the filename
+* lucene-style directory of datafiles + ToC/index
 * given that the files are written sequentially we can avoid writing every entry to the index file (as long as you write the first and last entry to the index). At most this means you have to read n entries, where n is the gap between index entries. Given that most clients will have persistent connections and be reading sequentially, we can do some clever things on the server side to make this really efficient (basically meaning you'll only have to pay that penalty on the first item you read).
 
 == LOG FILES

data/docs/protocol.rdoc

@@ -0,0 +1,63 @@
+= Conveyor protocol
+
+== Create a channel
+
+[Request] PUT /channels/{channel name}
+
+[Response] success: 201, failure: ? 
+
+The simple explanation is that to create a channel you do a PUT operation on the url you want for the channel, which must conform to /channels/\A[a-zA-Z0-9\-]+\Z. In other words, the channel name may only have letters numbers and dashes.
+
+== Post to a channel
+
+[Request] POST /channels/{channel name}, body is the message
+[Response] success: 202, failure: ?
+
+A post to a channel URL with the message in the body.
+
+== Get from channel
+
+=== Get by id
+
+[Request] GET /channels/{channel name}/{id}
+[Response] success: 200, failure: 404
+
+=== Get by timestamp
+
+NOT IMPLEMENTED YET!
+
+[Request] GET /channels/{channel name}?after={timestamp}
+[Response] success: 200
+
+Will return the first entry *after* that timestamp.
+
+=== Get Next (Queue-like semantics)
+
+[Request] GET /channels/{channel name}?next
+[Response] success: 200
+
+If this is called for the first time, it will return the first item in the channel. Otherwise it will return the next item.
+
+=== Get Next by Group (Multi-consumer queue)
+
+[Request] GET /channels/{channel name}?next&group={group name}
+[Response] success: 200
+
+If this is called for the first time, it will return the first item in the channel. Otherwise it will return the next item.
+
+=== Rewinding to id
+
+[Request] POST /channels/{channel name}?rewind_id={id}
+[Response] success: 200
+
+=== Get next n
+
+[Request] GET /channels/{channel name}?next&n={n}
+[Response] JSON array of objects, which have the keys 'id', 'hash' and 'data'
+
+
+=== Get next n for group
+
+[Request] GET /channels/{channel name}?next&n={n}&group={group}
+[Response] JSON array of objects, which have the keys 'id', 'hash' and 'data'
+

data/lib/conveyor.rb

@@ -1,4 +1,4 @@
 module Conveyor
-  VERSION = '0.2.0'
+  VERSION = '0.2.1'
   QUALITY = 'alpha'
 end

data/lib/conveyor/base_channel.rb

@@ -65,7 +65,7 @@ module Conveyor
       end
     end
 
-    def commit data, time = nil
+    def commit data, time=nil
       l = nil
       gzip = data.length >= 256
       if gzip
@@ -105,7 +105,7 @@ module Conveyor
       end
     end
 
-    def get id, stream = false
+    def get id, stream=false
       return nil unless id <= @last_id && id > 0
       i = @index[id-1]
       headers, content, compressed_content, g = nil
@@ -127,6 +127,12 @@ module Conveyor
       end
     end
 
+    def get_nearest_after_timestamp timestamp, stream=false
+      # i = binary search to find nearest item at or after timestamp
+      i = nearest_after(timestamp)
+      get(i) if i
+    end
+
     def self.parse_headers str, index_file=false
       pattern =  '\A([a-z\d]+) ([a-z\d]+) ([a-z\d]+) ([a-z\d]+) ([a-f0-9]+) ([a-z\d]+)'
       pattern += ' (\d+)' if index_file
@@ -196,5 +202,24 @@ module Conveyor
       File.join(@directory, 'version')
     end
 
+    def nearest_after(timestamp)
+      low = 0
+      high = @index.length
+      while low < high
+        mid = (low + high) / 2
+        if (@index[mid][:time].to_i > timestamp)
+          high = mid - 1
+        elsif (@index[mid][:time].to_i < timestamp)
+          low = mid + 1
+        else
+          return mid
+        end
+      end
+      if timestamp <= @index[mid][:time].to_i
+        @index[mid][:id]
+      else
+        nil
+      end
+    end
   end
 end

data/lib/conveyor/channel.rb

@@ -97,7 +97,7 @@ module Conveyor
     end
 
     def rewind *opts
-      opts = opts.first
+      opts = opts.inject{|h, m| m.merge(h)}
       if opts.key?(:id)
         if opts.key?(:group)
           group_iterator_lock(opts[:group]) do
@@ -112,6 +112,12 @@ module Conveyor
             @iterator_file.write("#{@iterator.to_s(36)}\n")
           end
         end
+      elsif opts.key?(:time)
+        if opts.key?(:group)
+          rewind :id => nearest_after(opts[:time]), :group => opts[:group]
+        else
+          rewind :id => nearest_after(opts[:time])
+        end
       end
     end
 

data/lib/conveyor/client.rb

@@ -45,12 +45,21 @@ module Conveyor
       end
     end
     
-    def rewind id, group=nil
-      if group
-        @conn.post("/channels/#{@channel}?rewind_id=#{id}&group=#{group}", nil)
-      else
-        @conn.post("/channels/#{@channel}?rewind_id=#{id}", nil)
+    def rewind *opts
+      opts = opts.inject{|h,m| m.merge(h)}
+      if opts.key?(:id) && opts.key?(:group)
+        @conn.post("/channels/#{@channel}?rewind_id=#{opts[:id]}&group=#{opts[:group]}", nil)
+      elsif opts.key?(:id)
+        @conn.post("/channels/#{@channel}?rewind_id=#{opts[:id]}", nil)
+      elsif opts.key?(:group) && opts.key?(:time)
+        @conn.post("/channels/#{@channel}?rewind_time=#{opts[:time].to_i}&group=#{opts[:group]}", nil)
+      elsif opts.key?(:time)
+        @conn.post("/channels/#{@channel}?rewind_time=#{opts[:time].to_i}", nil)
       end
     end
+
+    def get_nearest_after_timestamp timestamp
+      @conn.get("/channels/#{@channel}?after=#{timestamp.to_i}").body
+    end
   end
 end

data/lib/conveyor/server.rb

@@ -68,6 +68,14 @@ module Conveyor
             @channels[m.captures[0]].rewind(:id => params['rewind_id']).to_i # TODO make sure this is an integer
             [200, {}, "iterator rewound to #{params['rewind_id']}"]
           end
+        elsif params.key?('rewind_time')
+          if params['group']
+            @channels[m.captures[0]].rewind(:time => params['rewind_time'].to_i, :group => params['group']).to_i # TODO make sure this is an integer
+            [200, {}, "iterator rewound to #{params['rewind_id']}"]
+          else
+            @channels[m.captures[0]].rewind(:time => params['rewind_time'].to_i) # TODO make sure this is an integer
+            [200, {}, "iterator rewound to #{params['rewind_time']}"]
+          end
         else
           if env.key?('HTTP_DATE') && d = Time.parse(env['HTTP_DATE'])
             id = @channels[m.captures[0]].post(env['rack.input'].read)
@@ -111,6 +119,8 @@ module Conveyor
                 headers, content = @channels[m.captures[0]].get_next
               end
             end
+          elsif params.key? 'after'
+            headers, content = @channels[m.captures[0]].get_nearest_after_timestamp(params['after'].to_i)
           else
             return [200, {}, @channels[m.captures[0]].status.to_json]
           end

data/test/test_channel.rb

@@ -162,7 +162,7 @@ class TestConveyorChannel < Test::Unit::TestCase
     d = Channel.new('/tmp/bar')
     assert_equal 'foo', d.get_next[1]
   end
-  
+
   def test_group_rewind
     FileUtils.rm_r('/tmp/bar') rescue nil
     c = Channel.new('/tmp/bar')
@@ -176,7 +176,7 @@ class TestConveyorChannel < Test::Unit::TestCase
     d = Channel.new('/tmp/bar')
     assert_equal 'foo', d.get_next_by_group('bar')[1]
   end
-  
+
   def test_valid_name
     assert BaseChannel.valid_channel_name?(('a'..'z').to_a.join)
     assert BaseChannel.valid_channel_name?(('A'..'Z').to_a.join)
@@ -237,7 +237,7 @@ class TestConveyorChannel < Test::Unit::TestCase
     end
     assert_equal [], c.get_next_n_by_group(10, 'bar')
   end
-  
+
   def test_delete
     chan = 'test_delete'
     FileUtils.rm_r "/tmp/#{chan}" rescue nil
@@ -250,5 +250,52 @@ class TestConveyorChannel < Test::Unit::TestCase
     d = Channel.new("/tmp/#{chan}")
     assert_equal nil, d.get(1)
   end
+
+  def test_get_by_timestamp
+    chan = 'test_get_by_timestamp'
+    FileUtils.rm_r "/tmp/#{chan}" rescue nil
+    c = Channel.new("/tmp/#{chan}")
+
+    10.times{|i| c.post(i.to_s)}
+    assert_equal '0', c.get_nearest_after_timestamp(0)[1]
+    assert_equal nil, c.get_nearest_after_timestamp(2**32)
+
+    t0 = Time.now.to_i
+    10.times{|i| c.post((10 + i).to_s)}
+    assert_equal '9', c.get_nearest_after_timestamp(t0)[1]
+  end
+  
+  def test_rewind_to_timestamp
+    chan = 'test_rewind_to_timestamp'
+    FileUtils.rm_r "/tmp/#{chan}" rescue nil
+    c = Channel.new("/tmp/#{chan}")
+
+    10.times{|i| c.post(i.to_s)}
+    10.times{|i| assert_equal i.to_s, c.get_next[1]}
+
+    c.rewind :time => 0
+    10.times{|i| assert_equal i.to_s, c.get_next[1]}
+
+    t0 = Time.now.to_i + 1
+    c.rewind :time => t0
+    assert_equal nil, c.get_next
+  end
+  
+  def test_rewind_group_to_timestamp
+    chan = 'test_rewind_group_to_timestamp'
+    FileUtils.rm_r "/tmp/#{chan}" rescue nil
+    c = Channel.new("/tmp/#{chan}")
+
+    group = 'foo'
+    10.times{|i| c.post(i.to_s)}
+    10.times{|i| assert_equal i.to_s, c.get_next_by_group(group)[1]}
+
+    c.rewind :time => 0, :group => group
+    10.times{|i| assert_equal i.to_s, c.get_next_by_group(group)[1]}
+
+    t0 = Time.now.to_i + 1
+    c.rewind :time => t0, :group => group
+    assert_equal nil, c.get_next_by_group(group)
+  end
   
 end

data/test/test_server.rb

@@ -49,7 +49,8 @@ class TestConveyorServer < Test::Unit::TestCase
         "WUSYY2dCBdDdZEiGWtyfC5yGKVMgDhzBhyNLwcefxa49fED1Sf05f8MlgXOBx6n5I6Ae2Wy3Mds", 
         "uAlUDvngWqDl3PaRVl1i9RcwDIvJlNp6yMy9RQgVsucwNvKaSOQlJMarWItKy8zT2ON08ElKkZ2aQJlb45Z8FwfE0xh8sA", 
         "NxWmEBmJp0uiNRhyxa26frQjfFaNERmZbConrytNQKnHfilFsZWAo0Qy8eVKgq", "ajq3i5ksiBovQYfvj", 
-        "yY3vhjeq","2IDeF0ccG8tRZIZSekz6fUii29"]
+        "yY3vhjeq","2IDeF0ccG8tRZIZSekz6fUii29"
+      ]
         
       data.each do |d|
         req = h.post("/channels/#{chan}", d, {'Content-Type' => 'application/octet-stream', 'Date' => Time.now.to_s})
@@ -62,7 +63,7 @@ class TestConveyorServer < Test::Unit::TestCase
       end
     end
   end
-  
+
   def test_invalid_channel
     Net::HTTP.start('localhost', 8011) do |h|
       req = h.put('/channels/|', '', {'Content-Type' => 'application/octet-stream'})
@@ -167,9 +168,9 @@ class TestConveyorServer < Test::Unit::TestCase
     c.post 'foo'
     
     assert_equal 'foo', c.get_next('bar')
-    c.rewind(1, 'bar')
+    c.rewind(:id => 1, :group => 'bar')
     assert_equal 'foo', c.get_next('bar')
-    c.rewind(1, 'bar')
+    c.rewind(:id => 1, :group => 'bar')
   end
 
   def test_get_next_by_group
@@ -275,5 +276,48 @@ class TestConveyorServer < Test::Unit::TestCase
     assert_equal 'foo', c.get_next
   end
 
+  def test_get_by_timestamp
+    chan = 'test_get_by_timestamp'
+    c = Client.new('localhost', chan)
+
+    10.times{|i| c.post(i.to_s)}
+    assert_equal '0', c.get_nearest_after_timestamp(0)
+    assert_equal '', c.get_nearest_after_timestamp(2**32)
+
+    t0 = Time.now.to_i
+    10.times{|i| c.post((10 + i).to_s)}
+    assert_equal '9', c.get_nearest_after_timestamp(t0)
+  end
+
+  def test_rewind_to_timestamp
+    chan = 'test_rewind_to_timestamp'
+    c = Client.new('localhost', chan)
+
+    10.times{|i| c.post(i.to_s)}
+    10.times{|i| assert_equal i.to_s, c.get_next}
+
+    c.rewind :time => 0
+    10.times{|i| assert_equal i.to_s, c.get_next}
+
+    t0 = Time.now.to_i + 1
+    c.rewind :time => t0
+    assert_equal '', c.get_next
+  end
+  
+  def test_rewind_group_to_timestamp
+    chan = 'test_rewind_group_to_timestamp'
+    group = 'foo'
+    c = Client.new('localhost', chan)
+
+    10.times{|i| c.post(i.to_s)}
+    10.times{|i| assert_equal i.to_s, c.get_next(group)}
+
+    c.rewind :time => 0, :group => group
+    10.times{|i| assert_equal i.to_s, c.get_next(group)}
+
+    t0 = Time.now.to_i + 1
+    c.rewind :time => t0, :group => group
+    assert_equal '', c.get_next(group)
+  end
 end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: conveyor
 version: !ruby/object:Gem::Version 
-  version: 0.2.0
+  version: 0.2.1
 platform: ruby
 authors: 
 - Ryan King
@@ -30,7 +30,7 @@ cert_chain:
   Zls3y84CmyAEGg==
   -----END CERTIFICATE-----
 
-date: 2008-02-26 00:00:00 -08:00
+date: 2008-02-29 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -88,8 +88,8 @@ files:
 - Rakefile
 - bin/conveyor
 - bin/conveyor-upgrade
-- docs/file-formats.mkd
-- docs/protocol.mkd
+- docs/file-formats.rdoc
+- docs/protocol.rdoc
 - lib/conveyor.rb
 - lib/conveyor/base_channel.rb
 - lib/conveyor/channel.rb

data/docs/protocol.mkd

@@ -1,89 +0,0 @@
-# Conveyor protocol #
-
-## Create a channel ##
-
-Request
-: PUT /channels/{channel name}
-
-Response
-: success: 201, failure: ? 
-
-The simple explanation is that to create a channel you do a PUT operation on the url you want for the channel, which must conform to /channels/\A[a-zA-Z0-9\-]+\Z. In other words, the channel name may only have letters numbers and dashes.
-
-## Post to a channel ##
-Request
-: POST /channels/{channel name}
-: body is the message
-
-Response
-: success: 202, failure: ?
-
-A post to a channel URL with the message in the body.
-
-## Get from channel ##
-
-### Get by id ###
-
-Request
-: GET /channels/{channel name}/{id}
-
-Response
-: success: 200, failure: 404
-
-### Get by datetime ###
-
-NOT IMPLEMENTED YET!
-
-Request
-: GET /channels/{channel name}?at={ISO datetime like 2008-01-11T17:53:59}
-
-Response
-: success: 200
-
-Will return the first entry *after* that datetime.
-
-### Get Next (Queue-like semantics) ###
-
-Request
-: GET /channels/{channel name}?next
-
-Response
-: success: 200
-
-If this is called for the first time, it will return the first item in the channel. Otherwise it will return the next item.
-
-### Get Next by Group (Multi-consumer queue) ###
-
-Request
-: GET /channels/{channel name}?next&group={group name}
-
-Response
-: success: 200
-
-If this is called for the first time, it will return the first item in the channel. Otherwise it will return the next item.
-
-### Rewinding to id ###
-
-Request
-: POST /channels/{channel name}?rewind_id={id}
-
-Response
-:success: 200
-
-### Get next n ###
-
-Request
-: GET /channels/{channel name}?next&n={n}
-
-Response
-: JSON array of objects, which have the keys 'id', 'hash' and 'data'
-
-
-### Get next n for group ###
-
-Request
-: GET /channels/{channel name}?next&n={n}&group={group}
-
-Response
-: JSON array of objects, which have the keys 'id', 'hash' and 'data'
-