json-emitter 0.0.3 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d39d9a362efddd2cb39412926290328feb042228099e2b737d6d8450fad28cdd
-  data.tar.gz: ff8a418af3119910c46d78320e8b4dcac3cd6c0282202e1bae970c56fadf3e12
+  metadata.gz: 5ffd5b387bbbf0ce7f40a9b725dbaaea83a479ac53ada2ede1f01231c5be28e1
+  data.tar.gz: c0af5bb60b80c67da129f2ba14fea69a99c97ba69a47e508de165ebb12bf11e7
 SHA512:
-  metadata.gz: b5cc83b5e2018d7f9b05f98f3cbfa9ab3f45176ff96e121f4336303466abdfb1836db673c5b567ae3579de7cdf9bf3d95b7879488ef35a25d34422d11177c1ed
-  data.tar.gz: 33cc0cf57bedd84fda214c19859a32289c31d08e5e1d587d55485e73b5dab81030cd4fc12b14ed0c6e8234238821492fc694e603ad4d7ee8193f479de867ad6c
+  metadata.gz: 6ffb66b5cae058582b8d0904b6ae55272b7065ea6f412a2ddc7ab6db89a030f1e13c7801894e672f182c2da0b72d9e7211b9751f500f05fa91c8356679416403
+  data.tar.gz: e7a855f40029fbd7343cd4bfebf0e59594d81257fd2d0a232b8bf9cee4d0ac82210c8fa8907f6c5ae0eeaf4ee3acb68ca5b58adaa9a7525c1c094ac8f67e6d35

data/README.md

@@ -1,106 +1,99 @@
-## JsonEmitter
+# JsonEmitter
 
 JsonEmitter is a library for efficiently generating very large bits of JSON in Ruby. Need to generate a JSON array of 10,000 database records without eating up all your RAM? No problem! Objects? Nested structures? JsonEmitter has you covered.
 
 Use JsonEmitter in your Rack/Rails/Sinatra/Grape API to stream large JSON responses without worrying about RAM or HTTP timeouts. Use it to write large JSON objects to your filesystem, S3, or ~~a 3D printer~~ anywhere else!
 
-**Stream a JSON array from ActiveRecord**
+# HTTP Chunked Responses
+
+These examples will use the Order enumerator to generate chunks of JSON and send them to the client as more chunks are generated. No more than 500 orders will be in memory at a time, regardless of how many orders there are. And only small portions of the JSON will be in memory at once, no matter how much we're generating.
 
 ```ruby
-order_query = Order.limit(10_000).find_each(batch_size: 500)
-stream = JsonEmitter.array(order_query) { |order|
-  {
-    number: order.id,
-    desc: order.description,
-    ...
-  }
-}
+enumerator = Order.
+  where("created_at >= ?", 1.year.ago).
+  find_each(batch_size: 500)
 ```
 
-**Stream a JSON object**
+**Rails**
 
 ```ruby
-order_query = Order.limit(10_000).find_each(batch_size: 500)
-stream = JsonEmitter.object({
-  tuesday: false,
-
-  orders: order_query.lazy.map { |order|
-    {id: order.id, desc: order.description}
-  }
-
-  big_text_1: ->() {
-    load_tons_of_text
-  },
-
-  big_text_2: ->() {
-    load_tons_of_text
-  },
-})
+class OrdersController < ApplicationController
+  def index
+    headers["Content-Type"] = "application/json"
+    headers["Last-Modified"] = Time.now.ctime.to_s
+    self.response_body = JsonEmitter.array(enumerator) { |order|
+      order.to_h
+    }
+  end
+end
 ```
 
-**Generate the JSON and put it somewhere**
+**Sinatra**
 
 ```ruby
-# write to a file or any IO
-File.open("/tmp/foo.json", "w+") { |file|
-  stream.write file
-}
-
-# get chunks and do something with them
-stream.each { |json_chunk|
-  ...
-}
+get "/orders" do
+  content_type :json
+  JsonEmitter.array(enumerator) { |order|
+    order.to_h
+  }
+end
 ```
 
-# HTTP Chunked Transfer (a.k.a streaming)
-
-In HTTP 1.0 the entire response is normally sent all at once. Usually this is fine, but it can cause problems when very large responses must be generated and sent. These problems usually manifest as spikes in memory usage and/or responses that take so long to send that the client (or an in-between proxy) times out the request.
-
-The solution to this in HTTP 1.1 is chunked transfer encoding. The response body can be split up and sent in a series of separate "chunks" for the client to receive and automatically put back together. Ruby's Rack specification supports chunking, as do most frameworks based on it (e.g. Rails, Sinatra, Grape, etc).
-
-The following examples all show the same streaming API in various Rack-based frameworks. Without streaming, the examples could eat up tons of memory, take too long, and time out on the client. With streaming, the following improvements are possible without your client-side code needing any changes:
-
-1. Only 500 orders will ever be in memory at once.
-2. Only one `ApiV1::Entities::Order` will ever be in memory at once.
-3. Only 16kb (roughly) of JSON will ever be in memory at once.
-5. That 16kb of JSON will be sent to the client while the next 16kb of JSON is generating.
-
-**IMPORTANT** Not every Ruby application server supports HTTP chunking. Puma definitely supports it and WEBrick definitely does not. Phusion Passenger claims to but I have not tried it.
-
-## Rails
-
-TODO
-
-## Sinatra
-
-TODO
-
-## Grape
+**Grape**
 
 ```ruby
 get :orders do
-  enumerator = Order.
-    where("created_at >= ?", 1.year.ago).
-    find_each(batch_size: 500)
-
   stream JsonEmitter.array(enumerator) { |order|
     ApiV1::Entities::Order.new(order)
   }
 end
 ```
 
-## Rack
+**Rack**
 
 ```ruby
 app = ->(env) {
-  enumerator = Order.
-    where("created_at >= ?", 1.year.ago).
-    find_each(batch_size: 500)
-
   stream = JsonEmitter.array(enumerator) { |order|
     order.to_h
   }
-
   [200, {"Content-Type" => "application/json"}, stream]
 }
 ```
+
+# Other uses
+
+`JsonEmitter.array` takes an `Enumerable` and returns a stream that generates chunks of JSON.
+
+```ruby
+JsonEmitter.array(enumerator).each { |json_chunk|
+  # write json_chunk somewhere
+}
+```
+
+`JsonEmitter.object` takes a `Hash` and returns a stream that generates chunks of JSON.
+
+```ruby
+JsonEmitter.object({
+  orders: Order.find_each.lazy.map { |order|
+    {id: order.id, desc: order.description}
+  },
+
+  big_text_1: ->() {
+    load_tons_of_text
+  },
+
+  big_text_2: ->() {
+    load_tons_of_text
+  },
+}).each { |json_chunk|
+  # write json_chunk somewhere
+}
+```
+
+Streams have a `#write` method for writing directly to a `File` or `IO` object.
+
+```ruby
+File.open("~/out.json", "w+") { |f|
+  JsonEmitter.array(enumerator).write f
+}
+```

data/lib/json-emitter.rb

@@ -1,6 +1,7 @@
 require 'multi_json'
 
 require 'json-emitter/version'
+require 'json-emitter/context'
 require 'json-emitter/emitter'
 require 'json-emitter/stream'
 require 'json-emitter/buffered_stream'
@@ -24,7 +25,7 @@ module JsonEmitter
   @error_handlers = []
 
   #
-  # Generates an stream that will output a JSON array. The input can be any Enumerable, such as an Array or an Enumerator.
+  # Generates a stream that will output a JSON array. The input can be any Enumerable, such as an Array or an Enumerator.
   #
   # The following example uses minumum memory to genrate a very large JSON array string from an ActiveRecord query.
   # Only 500 Order records will ever by in memory at once. The JSON will be generated in small chunks so that
@@ -49,7 +50,7 @@ module JsonEmitter
   #
   # @param enum [Enumerable] Something that can be enumerated over, like an Array or Enumerator. Each element should be something that can be rendered as JSON (e.g. a number, string, boolean, Array, or Hash).
   # @param buffer_size [Integer] The buffer size in kb. This is a size *hint*, not a hard limit.
-  # @param unit [Symbol] :bytes | :kb (default) | :mb
+  # @param buffer_unit [Symbol] :bytes | :kb (default) | :mb
   # @yield If a block is given, it will be yielded each value in the array. The return value from the block will be converted to JSON instead of the original value.
   # @return [JsonEmitter::BufferedStream]
   #
@@ -59,7 +60,7 @@ module JsonEmitter
   end
 
   #
-  # Generates an stream that will output a JSON object.
+  # Generates a stream that will output a JSON object.
   #
   # If some of the values will be large arrays, use Enumerators or lazy Enumerators to build each element on demand
   # (to potentially save lots of RAM).
@@ -93,7 +94,7 @@ module JsonEmitter
   #
   # @param hash [Hash] Keys should be Strings or Symbols and values should be any JSON-compatible value like a number, string, boolean, Array, or Hash.
   # @param buffer_size [Integer] The buffer size in kb. This is a size *hint*, not a hard limit.
-  # @param unit [Symbol] :bytes | :kb (default) | :mb
+  # @param buffer_unit [Symbol] :bytes | :kb (default) | :mb
   # @return [JsonEmitter::BufferedStream]
   #
   def self.object(hash, buffer_size: 16, buffer_unit: :kb)

data/lib/json-emitter/context.rb

@@ -0,0 +1,45 @@
+module JsonEmitter
+  #
+  # By using JsonEmitter.wrap and JsonEmitter.error you can wrap your streams within a special "context".
+  #
+  # This is probably most useful when your stream is being consumed by Rack/Rails/Sinatra/Grape/etc. Your
+  # app is probably depending on certain Rack middlewars to provide before/after behavior and error handling.
+  # All those middlewares will over by the time Rack consumes your stream, but you can use JsonEmitter.wrap
+  # and JsonEmitter.error to add critical behavior back in.
+  #
+  class Context
+    def initialize
+      @wrappers = JsonEmitter.wrappers.map(&:call).compact
+      @error_handlers = JsonEmitter.error_handlers
+      @pass_through_errors = []
+      @pass_through_errors << Puma::ConnectionError if defined? Puma::ConnectionError
+    end
+
+    # Wrap the enumeration in a block. It will be passed a callback which it must call to continue.
+    # TODO better docs and examples.
+    def wrap(&block)
+      if (wrapper = block.call)
+        @wrappers.unshift wrapper
+      end
+    end
+
+    # Add an error handler.
+    # TODO better docs and examples.
+    def error(&handler)
+      @error_handlers += [handler]
+    end
+
+    # Execute a block within this context.
+    def execute(&inner)
+      @wrappers.reduce(inner) { |f, outer_wrapper|
+        ->() { outer_wrapper.call(f) }
+      }.call
+
+    rescue *@pass_through_errors => e
+      raise e
+    rescue => e
+      @error_handlers.each { |h| h.call(e) }
+      raise e
+    end
+  end
+end

data/lib/json-emitter/emitter.rb

@@ -3,11 +3,11 @@ module JsonEmitter
   # Builds Enumerators that yield JSON from Ruby Arrays or Hashes.
   #
   class Emitter
+    # @return [JsonEmitter::Context]
+    attr_reader :context
+
     def initialize
-      @wrappers = JsonEmitter.wrappers.map(&:call).compact
-      @error_handlers = JsonEmitter.error_handlers
-      @pass_through_errors = []
-      @pass_through_errors << Puma::ConnectionError if defined? Puma::ConnectionError
+      @context = Context.new
     end
 
     #
@@ -19,7 +19,7 @@ module JsonEmitter
     #
     def array(enum, &mapper)
       Enumerator.new { |y|
-        wrapped {
+        context.execute {
           array_generator(enum, &mapper).each { |json_val|
             y << json_val
           }
@@ -35,7 +35,7 @@ module JsonEmitter
     #
     def object(hash)
       Enumerator.new { |y|
-        wrapped {
+        context.execute {
           object_generator(hash).each { |json_val|
             y << json_val
           }
@@ -43,20 +43,6 @@ module JsonEmitter
       }
     end
 
-    # Wrap the enumeration in a block. It will be passed a callback which it must call to continue.
-    # TODO better docs and examples.
-    def wrap(&block)
-      if (wrapper = block.call)
-        @wrappers.unshift wrapper
-      end
-    end
-
-    # Add an error handler.
-    # TODO better docs and examples.
-    def error(&handler)
-      @error_handlers += [handler]
-    end
-
     private
 
     def array_generator(enum, &mapper)
@@ -112,17 +98,5 @@ module JsonEmitter
         [MultiJson.dump(x)]
       end
     end
-
-    def wrapped(&final)
-      @wrappers.reduce(final) { |f, outer_wrapper|
-        ->() { outer_wrapper.call(f) }
-      }.call
-
-    rescue *@pass_through_errors => e
-      raise e
-    rescue => e
-      @error_handlers.each { |h| h.call(e) }
-      raise e
-    end
   end
 end

data/lib/json-emitter/version.rb

@@ -1,4 +1,4 @@
 module JsonEmitter
   # Library version
-  VERSION = "0.0.3".freeze
+  VERSION = "0.0.4".freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: json-emitter
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.4
 platform: ruby
 authors:
 - Jordan Hollinger
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-01-30 00:00:00.000000000 Z
+date: 2019-01-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: multi_json
@@ -33,6 +33,7 @@ files:
 - README.md
 - lib/json-emitter.rb
 - lib/json-emitter/buffered_stream.rb
+- lib/json-emitter/context.rb
 - lib/json-emitter/emitter.rb
 - lib/json-emitter/stream.rb
 - lib/json-emitter/version.rb