mongoid-scroll 0.1.0 → 0.2.0

data/CHANGELOG.md

@@ -1,5 +1,11 @@
-0.1.0 (TBD)
-===========
+0.2.0 (3/14/2013)
+=================
 
-* Initial public release - [@dblock](https://github.com/dblock).
+* Extended `Moped::Query` with `scroll` - [@dblock](https://github.com/dblock).
+* `Mongoid::Scroll::Cursor.from_record` can now be called with either a Mongoid field or `field_type` and `field_name` in the `options` hash - [@dblock](https://github.com/dblock).
+
+0.1.0 (2/14/2013)
+=================
+
+* Initial public release, extends `Mongoid::Criteria` with `scroll` - [@dblock](https://github.com/dblock).
 

data/Gemfile

@@ -6,4 +6,5 @@ group :development, :test do
   gem "rake"
   gem "bundler"
   gem "rspec"
+  gem "faker"
 end

data/README.md

@@ -1,44 +1,57 @@
 Mongoid::Scroll [![Build Status](https://travis-ci.org/dblock/mongoid-scroll.png?branch=master)](https://travis-ci.org/dblock/mongoid-scroll)
 ===============
 
-Mongoid extension that enable infinite scrolling.
+Mongoid extension that enables infinite scrolling for `Mongoid::Criteria` and `Moped::Query`.
+
+Demo
+----
+
+Check out [artsy.net](http://artsy.net) homepage. Scroll down.
+
+There're also two code samples for Mongoid and Moped in [examples](examples). Run `bundle exec ruby examples/mongoid_scroll_feed.rb`.
 
 The Problem
 -----------
 
 Traditional pagination does not work when data changes between paginated requests, which makes it unsuitable for infinite scroll behaviors.
 
-* If a record is inserted before the current page limit, the collection will shift to the right, and the returned result will include a duplicate from a previous page.
-* If a record is removed before the current page limit, the collection will shift to the left, and the returned result will be missing a record.
+* If a record is inserted before the current page limit, items will shift right, and the next page will include a duplicate.
+* If a record is removed before the current page limit, items will shift left, and the next page will be missing a record.
 
 The solution implemented by the `scroll` extension paginates data using a cursor, giving you the ability to restart pagination where you left it off. This is a non-trivial problem when combined with sorting over non-unique record fields, such as timestamps.
 
-Usage
------
+Installation
+------------
 
-Add `mongoid-scroll` to Gemfile.
+Add the gem to your Gemfile and run `bundle install`.
 
 ```ruby
 gem 'mongoid-scroll'
 ```
 
+Usage
+-----
+
+### Mongoid
+
 A sample model.
 
 ```ruby
 module Feed
   class Item
     include Mongoid::Document
-    field :content, type: String
-    field :created_at, type: DateTime
+    field :title, type: String
+    field :position, type: Integer
+    index({ position: 1, _id: 1 })
   end
 end
 ```
 
-Scroll and save a cursor to the last item.
+Scroll by `:position` and save a cursor to the last item.
 
 ```ruby
 saved_cursor = nil
-Feed::Item.desc(:created_at).limit(5).scroll do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll do |record, next_cursor|
   # each record, one-by-one
   saved_cursor = next_cursor
 end
@@ -47,7 +60,7 @@ end
 Resume iterating using the previously saved cursor.
 
 ```ruby
-Feed::Item.desc(:created_at).limit(5).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).limit(5).scroll(saved_cursor) do |record, next_cursor|
   # each record, one-by-one
   saved_cursor = next_cursor
 end
@@ -56,23 +69,81 @@ end
 The iteration finishes when no more records are available. You can also finish iterating over the remaining records by omitting the query limit.
 
 ```ruby
-Feed::Item.desc(:created_at).scroll(saved_cursor) do |record, next_cursor|
+Feed::Item.desc(:position).scroll(saved_cursor) do |record, next_cursor|
   # each record, one-by-one
 end
 ```
 
+### Moped
+
+Scroll and save a cursor to the last item. You must also supply a `field_type` of the sort criteria.
+
+```ruby
+saved_cursor = nil
+session[:feed_items].find.sort(position: -1).limit(5).scroll(nil, { field_type: DateTime }) do |record, next_cursor|
+  # each record, one-by-one
+  saved_cursor = next_cursor
+end
+```
+
+Resume iterating using the previously saved cursor.
+
+```ruby
+session[:feed_items].find.sort(position: -1).limit(5).scroll(saved_cursor, { field_type: DateTime }) do |record, next_cursor|
+  # each record, one-by-one
+  saved_cursor = next_cursor
+end
+```
+
+Indexes and Performance
+-----------------------
+
+A query without a cursor is identical to a query without a scroll.
+
+``` ruby
+# db.feed_items.find().sort({ position: 1 }).limit(5)
+Feed::Item.desc(:position).limit(5).scroll
+```
+
+Subsequent queries use an `$or` to avoid skipping items with the same value as the one at the current cursor position.
+
+``` ruby
+# db.feed_items.find({ "$or" : [
+#   { "position" : { "$gt" : 13 }},
+#   { "position" : 13, "_id": { "$gt" : ObjectId("511d7c7c3b5552c92400000e") }}
+# ]}).sort({ position: 1 })
+Feed:Item.desc(:position).limit(5).scroll(cursor)
+```
+
+This means you need to hit an index on `position` and `_id`.
+
+``` ruby
+# db.feed_items.ensureIndex({ position: 1, _id: 1 })
+
+module Feed
+  class Item
+    ...
+    index({ position: 1, _id: 1 })
+  end
+end
+```
+
 Cursors
 -------
 
-You can use `Mongoid::Scroll::Cursor.from_record` to generate a cursor. This can be useful when you just want to return a collection of results and the cursor pointing to after the last item.
+You can use `Mongoid::Scroll::Cursor.from_record` to generate a cursor. A cursor points at the last record of the previous iteration and unlike MongoDB cursors will not expire.
 
 ```ruby
-record = Feed::Item.desc(:created_at).limit(3).last
-cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["created_at"] })
+record = Feed::Item.desc(:position).limit(3).last
+cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["position"] })
 # cursor or cursor.to_s can be returned to a client and passed into .scroll(cursor)
 ```
 
-Note that unlike MongoDB cursors, `Mongoid::Scroll::Cursor` values don't expire.
+You can also a `field_name` and `field_type` instead of a Mongoid field.
+
+```ruby
+cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: DateTime, field_name: "position" })
+```
 
 Contributing
 ------------

data/examples/mongoid_scroll_feed.rb

@@ -0,0 +1,51 @@
+require 'bundler'
+Bundler.setup(:default, :development)
+
+require 'mongoid-scroll'
+require 'faker'
+
+Mongoid.connect_to "mongoid_scroll_demo"
+Mongoid.purge!
+
+module Feed
+  class Item
+    include Mongoid::Document
+    field :title, type: String
+    field :position, type: Integer
+    index({ position: 1, _id: 1 })
+  end
+end
+
+# total items to insert
+total_items = 20
+# a MongoDB query will be executed every scroll_by items
+scroll_by = 7
+
+# insert items with a position out-of-order
+rands = (0..total_items).to_a.sort { rand }[0..total_items]
+total_items.times do |i|
+  Feed::Item.create! title: Faker::Lorem.sentence, position: rands.pop
+end
+
+Moped.logger = Logger.new($stdout)
+Moped.logger.level = Logger::DEBUG
+
+Feed::Item.create_indexes
+
+total_shown = 0
+next_cursor = nil
+while true
+  current_cursor = next_cursor
+  next_cursor = nil
+  Feed::Item.asc(:position).limit(scroll_by).scroll(current_cursor) do |item, cursor|
+    puts "#{item.position}: #{item.title}"
+    next_cursor = cursor
+    total_shown += 1
+  end
+  break unless next_cursor
+  # destroy an item just for the heck of it, scroll is not affected
+  Feed::Item.asc(:position).first.destroy
+end
+
+# this will be 20
+puts "Shown #{total_shown} items."

data/examples/moped_scroll_feed.rb

@@ -0,0 +1,43 @@
+require 'bundler'
+Bundler.setup(:default, :development)
+
+require 'mongoid-scroll'
+require 'faker'
+
+Mongoid.connect_to "mongoid_scroll_demo"
+Mongoid.purge!
+
+# total items to insert
+total_items = 20
+# a MongoDB query will be executed every scroll_by items
+scroll_by = 7
+
+# insert items with a position out-of-order
+rands = (0..total_items).to_a.sort { rand }[0..total_items]
+total_items.times do |i|
+  Mongoid.default_session['feed_items'].insert(title: Faker::Lorem.sentence, position: rands.pop)
+end
+
+Mongoid.default_session['feed_items'].indexes.create(position: 1, _id: 1)
+
+Moped.logger = Logger.new($stdout)
+Moped.logger.level = Logger::DEBUG
+
+total_shown = 0
+next_cursor = nil
+while true
+  current_cursor = next_cursor
+  next_cursor = nil
+  Mongoid.default_session['feed_items'].find.limit(scroll_by).sort(position: 1).scroll(current_cursor, { field_type: Integer, field_name: 'position' }) do |item, cursor|
+    puts "#{item['position']}: #{item['title']}"
+    next_cursor = cursor
+    total_shown += 1
+  end
+  break unless next_cursor
+  # destroy an item just for the heck of it, scroll is not affected
+  item = Mongoid.default_session['feed_items'].find.sort(position: 1).first
+  Mongoid.default_session['feed_items'].find(_id: item["_id"]).remove
+end
+
+# this will be 20
+puts "Shown #{total_shown} items."

data/lib/mongoid/criterion/scrollable.rb

@@ -3,28 +3,28 @@ module Mongoid
     module Scrollable
 
       def scroll(cursor = nil, &block)
-        c = self
+        criteria = self
         # we don't support scrolling over a criteria with multiple fields
-        if c.options[:sort] && c.options[:sort].keys.count != 1
-          sort = c.options[:sort].keys.join(", ")
-          raise Mongoid::Scroll::Errors::MultipleSortFieldsError.new(sort: sort)
+        if criteria.options[:sort] && criteria.options[:sort].keys.size != 1
+          raise Mongoid::Scroll::Errors::MultipleSortFieldsError.new(sort: criteria.options[:sort])
+        elsif ! criteria.options.has_key?(:sort) || criteria.options[:sort].empty?
+          # introduce a default sort order if there's none
+          criteria = criteria.asc(:_id)
         end
-        # introduce a default sort order if there's none
-        c = c.asc(:_id) if (! c.options[:sort]) || c.options[:sort].empty?
         # scroll field and direction
-        scroll_field = c.options[:sort].keys.first
-        scroll_direction = c.options[:sort].values.first.to_i == 1 ? '$gt' : '$lt'
+        scroll_field = criteria.options[:sort].keys.first
+        scroll_direction = criteria.options[:sort].values.first.to_i == 1 ? '$gt' : '$lt'
         # scroll cursor from the parameter, with value and tiebreak_id
-        field = c.klass.fields[scroll_field.to_s]
-        cursor_options = { field: field, direction: scroll_direction }
+        field = criteria.klass.fields[scroll_field.to_s]
+        cursor_options = { field_type: field.type, field_name: scroll_field, direction: scroll_direction }
         cursor = cursor.is_a?(Mongoid::Scroll::Cursor) ? cursor : Mongoid::Scroll::Cursor.new(cursor, cursor_options)
         # scroll
         if block_given?
-          c.where(cursor.criteria).each do |record|
+          criteria.where(cursor.criteria).each do |record|
             yield record, Mongoid::Scroll::Cursor.from_record(record, cursor_options)
           end
         else
-          c
+          criteria
         end
       end
 

data/lib/mongoid/scroll/cursor.rb

@@ -2,72 +2,85 @@ module Mongoid
   module Scroll
     class Cursor
 
-      attr_accessor :value, :tiebreak_id, :field, :direction
+      attr_accessor :value, :tiebreak_id, :field_type, :field_name, :direction
 
       def initialize(value = nil, options = {})
-        unless options && (@field = options[:field])
-          raise ArgumentError.new "Missing options[:field]."
-        end
+        @field_type, @field_name = Mongoid::Scroll::Cursor.extract_field_options(options)
         @direction = options[:direction] || '$gt'
-        @value, @tiebreak_id = Mongoid::Scroll::Cursor.parse(value, options)
+        parse(value)
       end
 
       def criteria
-        cursor_criteria = { field.name => { direction => value } } if value
-        tiebreak_criteria = { field.name => value, :_id => { '$gt' => tiebreak_id } } if value && tiebreak_id
+        mongo_value = value.class.mongoize(value) if value
+        cursor_criteria = { field_name => { direction => mongo_value } } if mongo_value
+        tiebreak_criteria = { field_name => mongo_value, :_id => { '$gt' => tiebreak_id } } if mongo_value && tiebreak_id
         (cursor_criteria || tiebreak_criteria) ? { '$or' => [ cursor_criteria, tiebreak_criteria].compact } : {}
       end
 
       class << self
         def from_record(record, options)
-          unless options && (field = options[:field])
-            raise ArgumentError.new "Missing options[:field]."
-          end
-          Mongoid::Scroll::Cursor.new("#{transform_field_value(field, record.send(field.name))}:#{record.id}", options)
+          cursor = Mongoid::Scroll::Cursor.new(nil, options)
+          value = record.respond_to?(cursor.field_name) ? record.send(cursor.field_name) : record[cursor.field_name]
+          cursor.value = Mongoid::Scroll::Cursor.parse_field_value(cursor.field_type, cursor.field_name, value)
+          cursor.tiebreak_id = record["_id"]
+          cursor
         end
       end
 
       def to_s
-        tiebreak_id ? [ Mongoid::Scroll::Cursor.transform_field_value(field, value), tiebreak_id ].join(":") : nil
+        tiebreak_id ? [ Mongoid::Scroll::Cursor.transform_field_value(field_type, field_name, value), tiebreak_id ].join(":") : nil
       end
 
       private
 
+        def parse(value)
+          return unless value
+          parts = value.split(":")
+          unless parts.length >= 2
+            raise Mongoid::Scroll::Errors::InvalidCursorError.new({ cursor: value })
+          end
+          id = parts[-1]
+          value = parts[0...-1].join(":")
+          @value = Mongoid::Scroll::Cursor.parse_field_value(field_type, field_name, value)
+          @tiebreak_id = Moped::BSON::ObjectId(id)
+        end
+
         class << self
 
-          def parse(value, options)
-            return [ nil, nil ] unless value
-            parts = value.split(":")
-            unless parts.length >= 2
-              raise Mongoid::Scroll::Errors::InvalidCursorError.new({ cursor: value })
+          def extract_field_options(options)
+            if options && (field_name = options[:field_name]) && (field_type = options[:field_type])
+              [ field_type.to_s, field_name.to_s ]
+            elsif options && (field = options[:field])
+              [ field.type.to_s, field.name.to_s ]
+            else
+              raise ArgumentError.new "Missing options[:field_name] and/or options[:field_type]."
             end
-            id = parts[-1]
-            value = parts[0...-1].join(":")
-            [ parse_field_value(options[:field], value), Moped::BSON::ObjectId(id) ]
           end
 
-          def parse_field_value(field, value)
-            case field.type.to_s
+          def parse_field_value(field_type, field_name, value)
+            case field_type.to_s
+            when "Moped::BSON::ObjectId" then value
             when "String" then value.to_s
-            when "DateTime" then Time.at(value.to_i).to_datetime
-            when "Time" then Time.at(value.to_i)
-            when "Date" then Time.at(value.to_i).utc.to_date
+            when "DateTime" then value.is_a?(DateTime) ? value : Time.at(value.to_i).to_datetime
+            when "Time" then value.is_a?(Time) ? value : Time.at(value.to_i)
+            when "Date" then value.is_a?(Date) ? value : Time.at(value.to_i).utc.to_date
             when "Float" then value.to_f
             when "Integer" then value.to_i
             else
-              raise Mongoid::Scroll::Errors::UnsupportedFieldTypeError.new(field: field.name, type: field.type)
+              raise Mongoid::Scroll::Errors::UnsupportedFieldTypeError.new(field: field_name, type: field_type)
             end
           end
 
-          def transform_field_value(field, value)
-            case field.type.to_s
+          def transform_field_value(field_type, field_name, value)
+            case field_type.to_s
+            when "Moped::BSON::ObjectId" then value
             when "String" then value.to_s
-            when "Date" then value.to_time(:utc).to_i
+            when "Date" then Time.utc_time(value.year, value.month, value.day).to_i
             when "DateTime", "Time" then value.to_i
             when "Float" then value.to_f
             when "Integer" then value.to_i
             else
-              raise Mongoid::Scroll::Errors::UnsupportedFieldTypeError.new(field: field.name, type: field.type)
+              raise Mongoid::Scroll::Errors::UnsupportedFieldTypeError.new(field: field_name, type: field_type)
             end
           end
 

data/lib/mongoid/scroll/errors/multiple_sort_fields_error.rb

@@ -4,6 +4,9 @@ module Mongoid
       class MultipleSortFieldsError < Mongoid::Scroll::Errors::Base
 
         def initialize(opts = {})
+          if opts[:sort] && opts[:sort].is_a?(Hash)
+            opts = opts.merge(sort: opts[:sort].keys.join(", "))
+          end
           super(compose_message("multiple_sort_fields", opts))
         end
 

data/lib/mongoid/scroll/version.rb

@@ -1,5 +1,5 @@
 module Mongoid
   module Scroll
-    VERSION = '0.1.0'
+    VERSION = '0.2.0'
   end
 end

data/lib/mongoid-scroll.rb

@@ -6,6 +6,9 @@ require 'mongoid'
 require 'mongoid/scroll/version'
 require 'mongoid/scroll/errors'
 require 'mongoid/scroll/cursor'
+
+require 'moped/scrollable'
 require 'mongoid/criterion/scrollable'
 
+Moped::Query.send(:include, Moped::Scrollable)
 Mongoid::Criteria.send(:include, Mongoid::Criterion::Scrollable)

data/lib/moped/scrollable.rb

@@ -0,0 +1,30 @@
+module Moped
+  module Scrollable
+
+    def scroll(cursor = nil, options = { field_type: Moped::BSON::ObjectId }, &block)
+      # we don't support scrolling over a criteria with multiple fields
+      if operation.selector["$orderby"] && operation.selector["$orderby"].keys.size != 1
+        raise Mongoid::Scroll::Errors::MultipleSortFieldsError.new(sort: operation.selector["$orderby"])
+      elsif ! operation.selector.has_key?("$orderby") || operation.selector["$orderby"].empty?
+        # introduce a default sort order if there's none
+        sort("_id" => 1)
+      end
+      # scroll field and direction
+      scroll_field = operation.selector["$orderby"].keys.first
+      scroll_direction = operation.selector["$orderby"].values.first.to_i == 1 ? '$gt' : '$lt'
+      # scroll cursor from the parameter, with value and tiebreak_id
+      cursor_options = { field_name: scroll_field, field_type: options[:field_type], direction: scroll_direction }
+      cursor = cursor.is_a?(Mongoid::Scroll::Cursor) ? cursor : Mongoid::Scroll::Cursor.new(cursor, cursor_options)
+      operation.selector["$query"].merge!(cursor.criteria)
+      # scroll
+      if block_given?
+        each do |record|
+          yield record, Mongoid::Scroll::Cursor.from_record(record, cursor_options)
+        end
+      else
+        self
+      end
+    end
+
+  end
+end

data/spec/mongoid/criteria_spec.rb

@@ -32,47 +32,64 @@ describe Mongoid::Criteria do
         Feed::Item.create!(
           a_string: i.to_s,
           a_integer: i,
-          a_datetime: DateTime.new(2013, i + 1, 21, 1, 42, 3)
+          a_datetime: DateTime.new(2013, i + 1, 21, 1, 42, 3),
+          a_date: Date.new(2013, i + 1, 21),
+          a_time: Time.at(Time.now.to_i + i)
         )
       end
     end
-    context "integer" do
-      it "scrolls all with a block" do
+    context "default" do
+      it "scrolls all" do
         records = []
-        Feed::Item.asc(:a_integer).scroll do |record, next_cursor|
+        Feed::Item.all.scroll do |record, next_cursor|
           records << record
         end
         records.size.should == 10
         records.should eq Feed::Item.all.to_a
       end
-      it "scrolls all with a break" do
-        records = []
-        cursor = nil
-        Feed::Item.asc(:a_integer).limit(5).scroll do |record, next_cursor|
-          records << record
-          cursor = next_cursor
+    end
+    { a_string: String, a_integer: Integer, a_date: Date, a_datetime: DateTime }.each_pair do |field_name, field_type|
+      context field_type do
+        it "scrolls all with a block" do
+          records = []
+          Feed::Item.asc(field_name).scroll do |record, next_cursor|
+            records << record
+          end
+          records.size.should == 10
+          records.should eq Feed::Item.all.to_a
         end
-        records.size.should == 5
-        Feed::Item.asc(:a_integer).scroll(cursor) do |record, next_cursor|
-          records << record
-          cursor = next_cursor
+        it "scrolls all with a break" do
+          records = []
+          cursor = nil
+          Feed::Item.asc(field_name).limit(5).scroll do |record, next_cursor|
+            records << record
+            cursor = next_cursor
+          end
+          records.size.should == 5
+          Feed::Item.asc(field_name).scroll(cursor) do |record, next_cursor|
+            records << record
+            cursor = next_cursor
+          end
+          records.size.should == 10
+          records.should eq Feed::Item.all.to_a
         end
-        records.size.should == 10
-        records.should eq Feed::Item.all.to_a
-      end
-      it "scrolls in descending order" do
-        records = []
-        Feed::Item.desc(:a_integer).limit(3).scroll do |record, next_cursor|
-          records << record
+        it "scrolls in descending order" do
+          records = []
+          Feed::Item.desc(field_name).limit(3).scroll do |record, next_cursor|
+            records << record
+          end
+          records.size.should == 3
+          records.should eq Feed::Item.desc(field_name).limit(3).to_a
+        end
+        it "map" do
+          record = Feed::Item.desc(field_name).limit(3).scroll.map { |record, cursor| record }.last
+          cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: field_type, field_name: field_name })
+          cursor.should_not be_nil
+          cursor.to_s.split(":").should == [
+            Mongoid::Scroll::Cursor.transform_field_value(field_type, field_name, record.send(field_name)).to_s,
+            record.id.to_s
+          ]
         end
-        records.size.should == 3
-        records.should eq Feed::Item.desc(:a_integer).limit(3).to_a
-      end
-      it "map" do
-        record = Feed::Item.desc(:a_integer).limit(3).scroll.map { |record, cursor| record }.last
-        cursor = Mongoid::Scroll::Cursor.from_record(record, { field: Feed::Item.fields["a_integer"] })
-        cursor.should_not be_nil
-        cursor.to_s.split(":").should == [ record.a_integer.to_s, record.id.to_s ]
       end
     end
   end

data/spec/mongoid/scroll_cursor_spec.rb

@@ -3,7 +3,7 @@ require 'spec_helper'
 describe Mongoid::Scroll::Cursor do
   context "an empty cursor" do
     subject do
-      Mongoid::Scroll::Cursor.new(nil, { field: Feed::Item.fields["a_string"]})
+      Mongoid::Scroll::Cursor.new nil, field_name: "a_string", field_type: String
     end
     its(:tiebreak_id) { should be_nil }
     its(:value) { should be_nil }
@@ -11,14 +11,14 @@ describe Mongoid::Scroll::Cursor do
   end
   context "an invalid cursor" do
     it "raises InvalidCursorError" do
-      expect { Mongoid::Scroll::Cursor.new "invalid", field: Feed::Item.fields["a_string"] }.to raise_error Mongoid::Scroll::Errors::InvalidCursorError,
+      expect { Mongoid::Scroll::Cursor.new "invalid", field_name: "a_string", field_type: String }.to raise_error Mongoid::Scroll::Errors::InvalidCursorError,
         /The cursor supplied is invalid: invalid./
     end
   end
   context "a string field cursor" do
     let(:feed_item) { Feed::Item.create!(a_string: "astring") }
     subject do
-      Mongoid::Scroll::Cursor.new "#{feed_item.a_string}:#{feed_item.id}", field: Feed::Item.fields["a_string"]
+      Mongoid::Scroll::Cursor.new "#{feed_item.a_string}:#{feed_item.id}", field_name: "a_string", field_type: String
     end
     its(:value) { should eq feed_item.a_string }
     its(:tiebreak_id) { should eq feed_item.id }
@@ -32,7 +32,7 @@ describe Mongoid::Scroll::Cursor do
   context "an integer field cursor" do
     let(:feed_item) { Feed::Item.create!(a_integer: 10) }
     subject do
-      Mongoid::Scroll::Cursor.new "#{feed_item.a_integer}:#{feed_item.id}", field: Feed::Item.fields["a_integer"]
+      Mongoid::Scroll::Cursor.new "#{feed_item.a_integer}:#{feed_item.id}", field_name: "a_integer", field_type: Integer
     end
     its(:value) { should eq feed_item.a_integer }
     its(:tiebreak_id) { should eq feed_item.id }
@@ -46,7 +46,7 @@ describe Mongoid::Scroll::Cursor do
   context "a date/time field cursor" do
     let(:feed_item) { Feed::Item.create!(a_datetime: DateTime.new(2013, 12, 21, 1, 42, 3)) }
     subject do
-      Mongoid::Scroll::Cursor.new "#{feed_item.a_datetime.to_i}:#{feed_item.id}", field: Feed::Item.fields["a_datetime"]
+      Mongoid::Scroll::Cursor.new "#{feed_item.a_datetime.to_i}:#{feed_item.id}", field_name: "a_datetime", field_type: DateTime
     end
     its(:value) { should eq feed_item.a_datetime }
     its(:tiebreak_id) { should eq feed_item.id }
@@ -61,7 +61,7 @@ describe Mongoid::Scroll::Cursor do
   context "a date field cursor" do
     let(:feed_item) { Feed::Item.create!(a_date: Date.new(2013, 12, 21)) }
     subject do
-      Mongoid::Scroll::Cursor.new "#{feed_item.a_date.to_datetime.to_i}:#{feed_item.id}", field: Feed::Item.fields["a_date"]
+      Mongoid::Scroll::Cursor.new "#{feed_item.a_date.to_datetime.to_i}:#{feed_item.id}", field_name: "a_date", field_type: Date
     end
     its(:value) { should eq feed_item.a_date }
     its(:tiebreak_id) { should eq feed_item.id }
@@ -74,6 +74,21 @@ describe Mongoid::Scroll::Cursor do
     }
   end
   context "a time field cursor" do
+    let(:feed_item) { Feed::Item.create!(a_time: Time.new(2013, 12, 21, 1, 2, 3)) }
+    subject do
+      Mongoid::Scroll::Cursor.new "#{feed_item.a_time.to_i}:#{feed_item.id}", field_name: "a_time", field_type: Time
+    end
+    its(:value) { should eq feed_item.a_time }
+    its(:tiebreak_id) { should eq feed_item.id }
+    its(:to_s) { should eq "#{feed_item.a_time.to_i}:#{feed_item.id}" }
+    its(:criteria) {
+      should eq({ "$or" => [
+        { "a_time" => { "$gt" => feed_item.a_time }},
+        { "a_time" => feed_item.a_time, :_id => { "$gt" => feed_item.id }}
+      ]})
+    }
+  end
+  context "a time field cursor with a field option" do
     let(:feed_item) { Feed::Item.create!(a_time: Time.new(2013, 12, 21, 1, 2, 3)) }
     subject do
       Mongoid::Scroll::Cursor.new "#{feed_item.a_time.to_i}:#{feed_item.id}", field: Feed::Item.fields["a_time"]
@@ -92,7 +107,7 @@ describe Mongoid::Scroll::Cursor do
     let(:feed_item) { Feed::Item.create!(a_array: [ "x", "y" ]) }
     it "is not supported" do
       expect {
-        Mongoid::Scroll::Cursor.from_record(feed_item, field: Feed::Item.fields["a_array"])
+        Mongoid::Scroll::Cursor.from_record feed_item, field_name: "a_array", field_type: Array
       }.to raise_error Mongoid::Scroll::Errors::UnsupportedFieldTypeError, /The type of the field 'a_array' is not supported: Array./
     end
   end

data/spec/moped/query_spec.rb

@@ -0,0 +1,117 @@
+require 'spec_helper'
+
+describe Moped::Query do
+  context "scrollable" do
+    subject do
+      Mongoid.default_session['feed_items'].find
+    end
+    it ":scroll" do
+      subject.should.respond_to? :scroll
+    end
+  end
+  context "with multiple sort fields" do
+    subject do
+      Mongoid.default_session['feed_items'].find.sort(name: 1, value: -1)
+    end
+    it "raises Mongoid::Scroll::Errors::MultipleSortFieldsError" do
+      expect { subject.scroll }.to raise_error Mongoid::Scroll::Errors::MultipleSortFieldsError,
+        /You're attempting to scroll over data with a sort order that includes multiple fields: name, value./
+    end
+  end
+  context "with no sort" do
+    subject do
+      Mongoid.default_session['feed_items'].find
+    end
+    it "adds a default sort by _id" do
+      subject.scroll.operation.selector["$orderby"].should == { "_id" => 1 }
+    end
+  end
+  context "with data" do
+    before :each do
+      10.times do |i|
+        Mongoid.default_session['feed_items'].insert(
+          a_string: i.to_s,
+          a_integer: i,
+          a_datetime: DateTime.mongoize(DateTime.new(2013, i + 1, 21, 1, 42, 3)),
+          a_date: Date.mongoize(Date.new(2013, i + 1, 21)),
+          a_time: Time.mongoize(Time.at(Time.now.to_i + i))
+        )
+      end
+    end
+    context "default" do
+      it "scrolls all" do
+        records = []
+        Mongoid.default_session['feed_items'].find.scroll do |record, next_cursor|
+          records << record
+        end
+        records.size.should == 10
+        records.should eq Mongoid.default_session['feed_items'].find.to_a
+      end
+    end
+    { a_string: String, a_integer: Integer, a_date: Date, a_datetime: DateTime }.each_pair do |field_name, field_type|
+      context field_type do
+        it "scrolls all with a block" do
+          records = []
+          Mongoid.default_session['feed_items'].find.sort(field_name => 1).scroll(nil, { field_type: field_type }) do |record, next_cursor|
+            records << record
+          end
+          records.size.should == 10
+          records.should eq Mongoid.default_session['feed_items'].find.to_a
+        end
+        it "scrolls all with a break" do
+          records = []
+          cursor = nil
+          Mongoid.default_session['feed_items'].find.sort(field_name => 1).limit(5).scroll(nil, { field_type: field_type }) do |record, next_cursor|
+            records << record
+            cursor = next_cursor
+          end
+          records.size.should == 5
+          Mongoid.default_session['feed_items'].find.sort(field_name => 1).scroll(cursor, { field_type: field_type }) do |record, next_cursor|
+            records << record
+            cursor = next_cursor
+          end
+          records.size.should == 10
+          records.should eq Mongoid.default_session['feed_items'].find.to_a
+        end
+        it "scrolls in descending order" do
+          records = []
+          Mongoid.default_session['feed_items'].find.sort(field_name => -1).limit(3).scroll(nil, { field_type: field_type, field_name: field_name }) do |record, next_cursor|
+            records << record
+          end
+          records.size.should == 3
+          records.should eq Mongoid.default_session['feed_items'].find.sort(field_name => -1).limit(3).to_a
+        end
+        it "map" do
+          record = Mongoid.default_session['feed_items'].find.limit(3).scroll(nil, { field_type: field_type, field_name: field_name }).map {
+            |record, cursor| record
+          }.last
+          cursor = Mongoid::Scroll::Cursor.from_record(record, { field_type: field_type, field_name: field_name })
+          cursor.should_not be_nil
+          cursor.to_s.split(":").should == [
+            Mongoid::Scroll::Cursor.transform_field_value(field_type, field_name, record[field_name.to_s]).to_s,
+            record["_id"].to_s
+          ]
+        end
+      end
+    end
+  end
+  context "with overlapping data" do
+    before :each do
+      3.times { Feed::Item.create! a_integer: 5 }
+    end
+    it "scrolls" do
+      records = []
+      cursor = nil
+      Mongoid.default_session['feed_items'].find.sort(a_integer: -1).limit(2).scroll do |record, next_cursor|
+        records << record
+        cursor = next_cursor
+      end
+      records.size.should == 2
+      Mongoid.default_session['feed_items'].find.sort(a_integer: -1).scroll(cursor) do |record, next_cursor|
+        records << record
+      end
+      records.size.should == 3
+      records.should eq Mongoid.default_session['feed_items'].find.to_a
+    end
+  end
+end

data/spec/support/feed/item.rb

@@ -2,7 +2,6 @@ module Feed
   class Item
     include Mongoid::Document
 
-    field :a_field
     field :a_integer, type: Integer
     field :a_string, type: String
     field :a_datetime, type: DateTime

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mongoid-scroll
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-02-14 00:00:00.000000000 Z
+date: 2013-02-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mongoid
@@ -58,6 +58,8 @@ files:
 - LICENSE.md
 - README.md
 - Rakefile
+- examples/mongoid_scroll_feed.rb
+- examples/moped_scroll_feed.rb
 - lib/config/locales/en.yml
 - lib/mongoid-scroll.rb
 - lib/mongoid/criterion/scrollable.rb
@@ -70,10 +72,12 @@ files:
 - lib/mongoid/scroll/errors/unsupported_field_type_error.rb
 - lib/mongoid/scroll/version.rb
 - lib/mongoid_scroll.rb
+- lib/moped/scrollable.rb
 - mongoid-scroll.gemspec
 - spec/mongoid/criteria_spec.rb
 - spec/mongoid/scroll_cursor_spec.rb
 - spec/mongoid/scroll_spec.rb
+- spec/moped/query_spec.rb
 - spec/spec_helper.rb
 - spec/support/feed/item.rb
 homepage: http://github.com/dblock/mongoid-scroll
@@ -91,7 +95,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 1370092332583330753
+      hash: 4314128693864897181
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements: