simple-feed 2.1.0 → 3.2.0

data/README.adoc

@@ -3,82 +3,130 @@
 :toc:
 :toclevels: 5
 :sectnums:
+:icons: font
 
-== Scalable, Easy to Use Activity Feed Implementation.
+====
 
-image:https://img.shields.io/gem/v/simple-feed.svg[Gem Version,link=https://rubygems.org/gems/simple-feed]
-image:https://img.shields.io/badge/license-MIT-blue.svg[MIT licensed,link=https://github.com/kigster/simple-feed/master/LICENSE.txt]
+https://liberapay.com/kigster/donate[image:https://liberapay.com/assets/widgets/donate.svg[Donate using Liberapay,height=30]]
+
+https://liberapay.com/kigster/donate[image:https://img.shields.io/liberapay/goal/kigster.svg?logo=liberapay[https://img.shields.io/liberapay/goal/kigster,height=50]]
+====
+
+== Scalable, Easy to Use Activity (aka "News") Feed
 
-image:https://github.com/kigster/simple-feed/workflows/Ruby/badge.svg?branch=master[Ruby,link=https://github.com/kigster/simple-feed/actions?query=workflow%3ARuby]
-image:https://github.com/kigster/simple-feed/workflows/Rubocop/badge.svg?branch=master[Ruby,link=https://github.com/kigster/simple-feed/actions?query=workflow%3ARubocop]
+=== What is this gem? — A Quick Summary
 
-image:https://travis-ci.org/kigster/simple-feed.svg?branch=master[Build Status,link=https://travis-ci.org/kigster/simple-feed]
-image:https://api.codeclimate.com/v1/badges/a11061820895fcde635e/maintainability[Maintainability,link=https://codeclimate.com/github/kigster/simple-feed/maintainability]
-image:https://api.codeclimate.com/v1/badges/a11061820895fcde635e/test_coverage[Test Coverage,link=https://codeclimate.com/github/kigster/simple-feed/test_coverage]
+NOTE: This gem is the Redis-backed backend implementation for a reverse chronological order news feed for a typical social network similar in some ways to Twitter: where users have followers, but also topics or a categories can have followers. When an event is posted by a user, it is also pushed to various individual feeds, and the gem then offers `O(1)` speed pagination for reading any page of the feed. The limitation is that the number of feed items must be capped (the default is 4000 items). The implementation can be scaled horizontally by sharding Redis backend, which can be implemented using a Redis Proxy layer such as https://github.com/twitter/twemproxy[Twemproxy] or https://github.com/envoyproxy/envoy/issues/19436[Envoy]).
 
-image:http://inch-ci.org/github/kigster/simple-feed.svg?branch=master[Inline docs,link=http://inch-ci.org/github/kigster/simple-feed]
-image:https://img.shields.io/gitter/room/gitterHQ/gitter.svg[Talk on Gitter,link=https://gitter.im/kigster/simple-feed]
+=== Build & Gem Status
+
+image:https://img.shields.io/badge/license-MIT-blue.svg[MIT licensed,link=https://github.com/kigster/simple-feed/main/LICENSE.txt]
+image:https://github.com/kigster/simple-feed/actions/workflows/ruby.yml/badge.svg[RSpec,link=https://github.com/kigster/simple-feed/actions/workflows/ruby.yml]
+image:https://github.com/kigster/simple-feed/actions/workflows/rubocop.yml/badge.svg[Rubocop,link=https://github.com/kigster/simple-feed/actions/workflows/rubocop.yml]
+
+image:https://img.shields.io/gem/v/simple-feed.svg[Gem Version,link=https://rubygems.org/gems/simple-feed]
+image:https://codecov.io/gh/kigster/simple-feed/branch/master/graph/badge.svg?token=RvDkGqoahz[Coverage,link=https://codecov.io/gh/kigster/simple-feed]
+
+=== Test Coverage Map
+
+image:https://codecov.io/gh/kigster/simple-feed/graphs/sunburst.svg?token=RvDkGqoahz[Coverage Map,link=https://codecov.io/gh/kigster/simple-feed/branch/master]
 
 IMPORTANT: Please read the (somewhat outdated) blog post http://kig.re/2017/02/19/feeding-frenzy-with-simple-feed-activity-feed-ruby-gem.html[Feeding Frenzy with SimpleFeed] launching this library. Please leave comments or questions in the discussion thread at the bottom of that post. Thanks!
 
 If you like to see this project grow, your donation of any amount is much appreciated.
 
 image::https://www.paypalobjects.com/en_US/i/btn/btn_donate_SM.gif[Donate,link=https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=FSFYYNEQ8RKWU]
-'''
 
-This is a fast, pure-ruby implementation of an activity feed concept commonly used in social networking applications. The implementation is optimized for *read-time performance* and high concurrency (lots of users), and can be extended with custom backend providers. Two providers come bundled: the production-ready Redis provider, and a naive Ruby Hash-based provider.
+This is a fast, pure-ruby implementation of an activity feed concept commonly used in social networking applications. The implementation is optimized for *read-time performance* and high concurrency (lots of users), and can be extended with custom backend providers. One data provider come bundled: the production-ready Redis provider.
 
 *Important Notes and Acknowledgements:*
 
 * SimpleFeed _does not depend on Ruby on Rails_ and is a *pure-ruby* implementation
-* SimpleFeed requires ruby 2.3 or later
+* SimpleFeed requires MRI Ruby 2.3 or later
 * SimpleFeed is currently live in production
+* SimpleFeed can use https://github.com/redis/hiredis[hiredis] if you install it separately and load it before loading the gem.
 * SimpleFeed is open source thanks to the generosity of *http://simbi.com[Simbi, Inc]*.
 
 == Features
 
-SimpleFeed offers the following features:
+SimpleFeed is a Ruby Library that can be plugged into any application to power a fast, Redis-based activity feed implementation so common on social networking sites. SimpleFeed offers the following features:
+
+ * Modelled after graph-relationships similar to those on Twitter (bi-directional independent follow relationships):
+ ** Feed maintains a reverse-chronological order for heterogeneous events for each user.
+ ** It offers a constant time lookup for user's feed, avoiding complex SQL joins to render it.
+ ** An API to read/paginate the feed for a given user
+ ** As well as to *query the total unread items in the feed* since it was last read by the user (typically shown on App icons).
+ * Scalable and well performing Redis-based activity feed —
+ ** Scales to millions of users (will need to use Twemproxy to shard across several Redis instances)
+ ** Stores a fixed number of events for each unique "user" — the default is 1000. When the feed reaches 1001 events, the oldest event is offloaded from the activity.
+ * Implementation properties:
+ ** Fully thread-safe implementation, writing events can be done in eg. Sidekiq.
+ ** Zero assumptions about what you are storing: the "data" is just a string. Serialize it with JSON, Marshall, YAML, or whatever.
+ ** You can create as many different types of feeds per application as you like (no Ruby Singletons used).
+ ** Customize mapping from `user_id` to the activity id based on your business logic (more on this later).
+
+=== Publishing Events
+
+Pushing events to the feed requires the following:
 
- * Highly performant Redis-based activity feed
+ * An `Event` consisting of:
+ ** String `data` that, most commonly, is a foreign key to a database table, but can really be anything you like.
+ ** Float `at` (typically, the timestamp, but can be any `float` number)
+ * One or more user IDs, or event consumers: basically — who should see the event being published in their feed.
 
- * Scales to millions of users (may need to use Twemproxy to shard across several Redis instances)
+You publish an event by choosing a set of users whose feed should be updated. For example, were you re-implementing Twitter, your array of `user_ids` when publishing an event would be all followers of the Tweet's author. While the `data` would probably be the Tweet ID.
 
- * Stores a fixed number of events per unique activity ID (eg, a user) — the default is 2000. When the full list is used, the oldest events are bumped off the feed (and are effectively trashed).
 
- * Store user_id as either a string or an integer (the latter is base62 encoded when serialized, so very large users_ids are OK)
+NOTE: Publishing an event to the feeds of N users is roughly a O(N * log(N)) operation
 
- * Updating feeds of N users is roughly a O(N * log(N)) operation
+=== Consuming Events (Reading / Rendering the Feed)
 
- * Reading feed for one user (or one type of user) is a O(1) operation
+You can fetch the chronologically ordered events for a particular user, using:
 
- * For each activity (user) read the number of total and new items in the feed, computed since the user last saw their feed.
+ * Methods on the `activity` such as `paginate`, `fetch`.
 
- * Delete items from user's feed selectively (for instance, if a user unfollows someone they shouldn't see their events anymore).
+ ** Reading feed for one user (or one type of user) is a `O(1)` operation
 
- * Automatically reset the timestamp when the user last read their feed when reading from the feed (or not, it's an argument). As soon as the user reads their feed (you call the `paginate` method), the "unread counter" is reset to 0.
+ * For each activity (user) you can fetch the `total_count` and the `unread_count` — the number of total and new items in the feed, where `unread_count` is computed since the user last reset their `read status`.
 
- * You can create as many different types of feeds per application as you like. No Singletons are used.
+ ** Note: `total_count` can never exceed the maximum size of the feed that you configured. The default is 1000 items.
 
- * Thread-safe implementation.
+ ** The `last_read` timestamp can be automatically reset when the user is shown the feed via `paginate` method (whether or not its reset is controlled via a method argument).
 
- * Zero assumptions about what you are storing: the "data" is just a string. Serialize it with JSON, Marshall, YAML, or whatever.
+=== Modifying User's Feed
+
+For any given user, you can:
+
+ * Wipe their feed with `wipe`
+
+ * Selectively remove items from the feed with `delete_if`.
+ ** For instance, if a user un-follows someone they shouldn't see their events anymore, so you'd have to call `delete_if` and remove any events published by the unfollowed user.
+
+=== Aggregating Events
+
+This is a feature planned for future versions.
+
+Help us much appreciated, even if you are not a developer, but have a clear idea about how it should work.
+
+== Commercial & Enterprise Support
+
+Commercial Support plans are available for SimpleFeed through author's https://reinvent.one[ReinventONE Inc] consulting company. Please reach out to kig AT reinvent.one for more information.
+
+== Usage
 
-== Background, Examples, Serialization, etc
+=== Example
 
 Please read the additional documentation, including the examples, on the https://github.com/kigster/simple-feed/wiki[project's Github Wiki].
 
 Below is a screen shot of an actual activity feed powered by this library.
 
-image::https://raw.githubusercontent.com/kigster/simple-feed/master/man/activity-feed-action.png[usage]
+image::https://raw.githubusercontent.com/kigster/simple-feed/main/man/activity-feed-action.png[usage]
 
-== Usage
+=== Providers
 
 A key concept to understanding SimpleFeed gem, is that of a _provider_, which is effectively a persistence implementation for the events belonging to each user.
 
-Two providers are supplied with this gem:
-
-* The production-ready `:redis` provider, which uses the https://redislabs.com/ebook/redis-in-action/part-2-core-concepts-2/chapter-3-commands-in-redis/3-5-sorted-sets[sorted set Redis data type] to store and fetch the events, scored by time (but not necessarily).
-* The naïve `:hash` provider based on the ruby `Hash` class, that can be useful in unit tests, or in simple simulations.
+One providers are supplied with this gem: the production-ready `:redis` provider, which uses the https://redislabs.com/ebook/redis-in-action/part-2-core-concepts-2/chapter-3-commands-in-redis/3-5-sorted-sets[sorted set Redis data type] to store and fetch the events, scored by time (but not necessarily).
 
 You initialize a provider by using the `SimpleFeed.provider([Symbol])` method.
 
@@ -91,6 +139,7 @@ Below we configure a feed called `:newsfeed`, which in this example will be popu
 require 'simplefeed'
 
 # Let's define a Redis-based feed, and wrap Redis in a in a ConnectionPool.
+
 SimpleFeed.define(:newsfeed) do |f|
   f.provider   = SimpleFeed.provider(:redis,
                                       redis: -> { ::Redis.new },
@@ -108,55 +157,130 @@ After the feed is defined, the gem creates a similarly named method under the `S
 
 You can also get a full list of currently defined feeds with `SimpleFeed.feed_names` method.
 
+=== Reading from and writing to the feed
+
+For the impatient, here is a quick way to get started with the `SimpleFeed`.
+
+[source,ruby]
+----
+# Let's use the feed we defined earlier and create activity for all followers of the current user
+publish_activity = SimpleFeed.newsfeed.activity(@current_user.followers.map(&:id))
+
+# Store directly the value and the optional time stamp
+publish_activity.store(value: 'hello', at: Time.now)
+# => true  # indicates that value 'hello' was not yet in the feed (all events must be unique)
+
+# Or, using the event form:
+publish_activity.store(event: SimpleFeed::Event.new('good bye', Time.now))
+# => true
+----
+
+As we've added the two events for these users, we can now read them back, sorted by
+the time and paginated:
+
+[source,ruby]
+----
+# Let's grab the first follower
+user_activity = SimpleFeed.newsfeed.activity(@current_user.followers.first.id)
+
+# Now we can paginate the events, while resetting this user's last-read timestamp:
+user_activity.paginate(page: 1, reset_last_read: true)
+# [
+#     [0] #<SimpleFeed::Event: value=hello, at=1480475294.0579991>,
+#     [1] #<SimpleFeed::Event: value=good bye, at=1480472342.8979871>,
+# ]
+----
+
+IMPORTANT: Note that we stored the activity by passing an array of users, but read the activity for just one user. This is how you'd use SimpleFeed most of the time, with the exception of the alternative mapping described below.
+
 === User IDs
 
-In the following section you will see the examples of reading and writing activity for users based on their ID.
+In the previous section you saw the examples of publishing events to many feeds, and then reading the activity for a given user.
+
+SimpleFeed supports user IDs that are either numeric (integer) or string-based (eg, UUID). Numeric IDs are best for simplest cases, and are the most compact. String keys offer the most flexibility.
+
+==== Activity Keys
 
-SimpleFeed supports user IDs that are either numeric (integer) or string-based (eg, UUID).
+In the next section we'll talk about generating `keys` from user_ids. We mean — Redis Hash keys that uniquely map a user (or a set of users) to the activity feed they should see.
 
-If your User IDs are numeric, they generate redis keys using Base62 encoding (which makes them shorter, and more compact).
+There are up to two keys that are computed depending on the situation:
 
-For string User IDs, the only transformation performed is the basic link:https://en.wikipedia.org/wiki/ROT13[`rot13`] in case user ids are semi-sensitive.
+ * `data_key` is used to store the actual feed events
+ * `meta_key` is used to store user's `last_read` status
 
 ==== Partitioning Schema
 
-You can take advantage of string user IDs for situations where your feed requires a composite keys for instance. Just remember that SimpleFeed does not care about what's in your user ID, and even what you call "a user". It's convenient to think of the activities in terms of users, because typically each user has a unique feed that only they see.
+NOTE: This feature is only available in **SimpleFeed Version 3+**.
+
+You can take advantage of string user IDs for situations where your feed requires keys to be composite for instance. Just remember that SimpleFeed does not care about what's in your user ID, and even what you call "a user". It's convenient to think of the activities in terms of users, because typically each user has a unique feed that only they see.
 
 But you can just as easily use zip code as the unique activity ID, and create one feed of events per geographical location, that all folks living in that zip code share. But what about other countries?
 
-Now you use partioning scheme: make "user_id" a combination `iso_country_code.postal_code`, eg for San Francisco, you'd use `us.94107`, but for Australia you could use, eg `au.3148`.
+Now you use partitioning scheme: make the "user_id" argument a combination `iso_country_code.postal_code`, eg for San Francisco, you'd use `us.94107`, but for Australia you could use, eg `au.3148`.
 
-=== Reading from and writing to the feed
+==== Relationship between an Activity and a User
 
-For the impatient, here is a quick way to get started with the `SimpleFeed`.
+===== One to One
+
+In the most common case, you will have one activity per user.
+
+For instance, in the Twitter example, each Twitter user has a unique tweeter feed that only they see.
+
+The events are published when someone posts a tweet, to the array of all users that follow the Tweet author.
+
+===== One to Many
+
+However, SimpleFeed supports one additional use-case, where you might have one activity shared among many users.
+
+Imagine a service that notifies residents of important announcements based on user's zip code of residence.
+
+We want this feed to work as follows:
+
+ * All users that share a zip-code should see the same exact feed.
+ * However, all users should never share the individual's `last_read` status: so if two people read the same activity from the same zip code, their `unread_count` should change independently.
+
+In terms of the activity keys, this means:
+
+ * `data_key` should be based on the zip-code of each user, and be one to many with users.
+ * `meta_key` should be based on the user ID as we want it to be 1-1 with users.
+
+To support this use-case, SimpleFeed supports two optional transformer lambdas that can be applied to each user object when computing their activity feed hash key:
 
 [source,ruby]
 ----
-# This assumes we have previously defined a feed named :newsfeed (see above)
-activity = SimpleFeed.newsfeed.activity(@current_user.id)
-# Store directly the value and the optional time stamp
-activity.store(value: 'hello')
-# => true
-
-# or equivalent:
-@event = SimpleFeed::Event.new('hello', Time.now)
-activity.store(event: @event)
-# => false # false indicates that the same event is already in the feed.
+SimpleFeed.define(:zipcode_alerts) do |f|
+  f.provider   = SimpleFeed.provider(:redis, redis: -> { ::Redis.new }, pool_size: 10)
+  f.namespace  = 'zc'
+  f.data_key_transformer = ->(user) { user.zip_code }  # actual feed data is stored once per zip code
+  f.meta_key_transformer = ->(user) { user.id }        # last_read status is stored once per user
+end
 ----
 
-As we've added events for this user, we can request them back, sorted by
-the time and paginated. If you are using a distributed provider, such as
-`Redis`, the events can be retrieved by any ruby process in your
-application, not just the one that published the event (which is the
-case for the "toy" `Hash::Provider`.
+When you publish events into this feed, you would need to provide `User` objects that all respond to `.zip_code` method (based on the above configuration). Since the data is only defined by Zip Code, you probably don't want to be publishing it via a giant array of users. Most likely, you'll want to publish event based on the zip code, and consume them based on the user ID.
+
+To support this user-case, let's modify our transformer lambda (only the `data` one) as follows — so that it can support both the consuming read by a user case, and the publishing a feed by zip code case:
+
+Alternatively, you could do something like this:
 
 [source,ruby]
 ----
-activity.paginate(page: 1, reset_last_read: true)
-# => [ <SimpleFeed::Event#0x2134afa value='hello' at='2016-11-20 23:32:56 -0800'> ]
+  f.data_key_transformer = ->(entity) do
+    case entity
+      when User
+        entity.zip_code.to_i
+      when String # UUIDs
+        User.find(entity)&.zip_code.to_i
+      when ZipCode, Numeric
+        entity.to_i
+      else
+        raise ArgumentError, "Invalid type #{entity.class.name}"
+    end
+  end
 ----
 
-=== The Two Forms of the API
+Just make sure that your users always have `.zip_code` defined, and that `ZipCode.new(94107).to_i` returns exactly the same thing as `@user.zip_code.to_i` or your users won't see the feeds they are supposed to see.
+
+=== The Two Forms of the Feed API
 
 The feed API is offered in two forms:
 
@@ -170,9 +294,8 @@ The method names and signatures are the same. The only difference is in what the
 
 Please see further below the details about the <<batch-api,Batch API>>.
 
-+++<a name="single-user-api">++++++</a>+++
-
 [discrete]
+
 ===== Single-User API
 
 In the examples below we show responses based on a single-user usage. As previously mentioned, the multi-user usage is the same, except what the response values are, and is discussed further down below.
@@ -183,10 +306,10 @@ Let's take a look at a ruby session, which demonstrates return values of the fee
 ----
 require 'simplefeed'
 
-# Define the feed using an in-memory Hash provider, which uses
+# Define the feed using Redis provider, which uses
 # SortedSet to keep user's events sorted.
 SimpleFeed.define(:followers) do |f|
-  f.provider = SimpleFeed.provider(:hash)
+  f.provider = SimpleFeed.provider(:redis)
   f.per_page = 50
   f.per_page = 2
 end
@@ -316,7 +439,7 @@ The DSL context has access to two additional methods:
 
 Below is an example output of `color_dump` method, which is intended for the debugging purposes.
 
-[image:https://raw.githubusercontent.com/kigster/simple-feed/master/man/sf-color-dump.png[color_dump output,450]](https://raw.githubusercontent.com/kigster/simple-feed/master/man/sf-color-dump.png)
+image::https://raw.githubusercontent.com/kigster/simple-feed/main/man/sf-color-dump.png[title=#color_dump method output, width=659, link=https://raw.githubusercontent.com/kigster/simple-feed/main/man/sf-color-dump.png]
 
 +++<a name="api">++++++</a>+++
 
@@ -393,9 +516,7 @@ It is the intention of this gem that:
 * it should be easy to write new providers
 * it should be easy to swap out providers
 
-To create a new provider please use `SimpleFeed::Providers::Hash::Provider` class as a starting point.
-
-Two providers are available with this gem:
+One provider is included with this gem:
 
 === `SimpleFeed::Providers::Redis::Provider`
 
@@ -403,36 +524,43 @@ Redis Provider is a production-ready persistence adapter that uses the https://r
 
 This provider is optimized for large writes and can use either a single Redis instance for all users of your application, or any number of Redis https://en.wikipedia.org/wiki/Shard_(database_architecture)[shards] by using a https://github.com/twitter/twemproxy[_Twemproxy_] in front of the Redis shards.
 
-=== `SimpleFeed::Providers::HashProvider`
-
-This is a pure Hash-like implementation of a provider that can be useful in unit tests of a host application. This provider could be used to write and read events within a single ruby process, can be serialized to and from a YAML file, and is therefore intended primarily for Feed emulations in automated tests.
-
-== Redis Provider
-
 If you set environment variable `REDIS_DEBUG` to `true` and run the example (see below) you will see every operation redis performs. This could be useful in debugging an issue or submitting a bug report.
 
-== Running the Examples
+== Running the Examples and Specs
 
 Source code for the gem contains the `examples` folder with an example file that can be used to test out the providers, and see what they do under the hood.
 
+Both the specs and the example requires a local redis instance to be available.
+
 To run it, checkout the source of the library, and then:
 
-[source,bash]
 ----
 git clone https://github.com/kigster/simple-feed.git
 cd simple-feed
-bundle
-be rspec  # make sure tests are passing
+
+# on OSX with HomeBrew:
+brew install redis
+brew services start redis
+
+# check that your redis is up:
+redis-cli info
+
+# install bundler and other dependencies
+gem install bundler --version 2.1.4
+bundle install
+bundle exec rspec  # make sure tests are passing
+
+# run the example:
 ruby examples/redis_provider_example.rb
 ----
 
 The above command will help you download, setup all dependencies, and run the examples for a single user:
 
-image::https://raw.githubusercontent.com/kigster/simple-feed/master/man/running-example.png[Example,link=https://raw.githubusercontent.com/kigster/simple-feed/master/man/running-example.png]
+image::https://raw.githubusercontent.com/kigster/simple-feed/main/man/running-example.png[title=Running Redis Example in a Terminal, width=663, link=https://raw.githubusercontent.com/kigster/simple-feed/main/man/running-example.png]
 
 If you set `REDIS_DEBUG` variable prior to running the example, you will be able to see every single Redis command executed as the example works its way through. Below is a sample output:
 
-image::https://raw.githubusercontent.com/kigster/simple-feed/master/man/running-example-redis-debug.png[Example with Debugging,link=https://raw.githubusercontent.com/kigster/simple-feed/master/man/running-example-redis-debug.png]
+image::https://raw.githubusercontent.com/kigster/simple-feed/main/man/running-example-redis-debug.png[title=Running Redis Example with REDIS_DEBUG set, width=918, link=https://raw.githubusercontent.com/kigster/simple-feed/main/man/running-example-redis-debug.png]
 
 === Generating Ruby API Documentation
 
@@ -474,6 +602,8 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/kigste
 
 The gem is available as open source under the terms of the http://opensource.org/licenses/MIT[MIT License].
 
+image:https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkigster%2Fsimple-feed.svg?type=large[FOSSA Scan Status, link=https://app.fossa.com/projects/git%2Bgithub.com%2Fkigster%2Fsimple-feed?ref=badge_large]
+
 === Acknowledgements
 
 * This project is conceived and sponsored by https://simbi.com[Simbi, Inc.].