json-emitter 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 82b399ea5929c610e5f113a35c72b9df9f6cf9957292dc8e2dde5837472484c3
+  data.tar.gz: 9925436be580fa8b772a18e4bad918ffcb3938f638cf0e633db4788c4e1df52c
+SHA512:
+  metadata.gz: 85048433a36d8d985d57099c2d0dacc0f5ddb31eb128ef8f030c8136224bd37d101db41ed14aff3f8a59b9c5958aa4e391b4b28598242ff1978d6c65d114f206
+  data.tar.gz: e058dd052d8604acbf8c8516a887dd1672a0018faf4d25c874feacf273655fbd83f10c587982fa028c98b16dd67bccd32d51421ba5a8bb49395096672df74026

data/README.md

@@ -0,0 +1,111 @@
+## JsonEmitter
+
+JsonEmitter is a library for efficiently generating very large bits of JSON in Ruby. Need to generate a JSON array of 10,000 database records without eating up all your RAM? No problem! Objects? Nested structures? JsonEmitter has you covered.
+
+Use JsonEmitter in your Rack/Rails/Sinatra/Grape API to stream large JSON responses without worrying about RAM or HTTP timeouts. Use it to write large JSON objects to your filesystem, S3, or ~~a 3D printer~~ anywhere else!
+
+**Stream a JSON array from ActiveRecord**
+
+```ruby
+order_query = Order.limit(10_000).find_each(batch_size: 500)
+stream = JsonEmitter.array(order_query) { |order|
+  {
+    number: order.id,
+    desc: order.description,
+    ...
+  }
+}
+```
+
+**Stream a JSON object**
+
+```ruby
+order_query = Order.limit(10_000).find_each(batch_size: 500)
+stream = JsonEmitter.object({
+  tuesday: false,
+
+  orders: order_query.lazy.map { |order|
+    {id: order.id, desc: order.description}
+  }
+
+  big_text_1: ->() {
+    load_tons_of_text
+  },
+
+  big_text_2: ->() {
+    load_tons_of_text
+  },
+})
+```
+
+**Generate the JSON and put it somewhere**
+
+```ruby
+# write to a file or any IO
+File.open("/tmp/foo.json", "w+") { |file|
+  stream.write file
+}
+
+# get chunks and do something with them
+stream.each { |json_chunk|
+  ...
+}
+
+# this will buffer the JSON into roughly 8k chunks
+stream.buffered(8).each { |json_8k_chunk|
+  ...
+}
+```
+
+# HTTP Chunked Transfer (a.k.a streaming)
+
+In HTTP 1.0 the entire response is normally sent all at once. Usually this is fine, but it can cause problems when very large responses must be generated and sent. These problems usually manifest as spikes in memory usage and/or responses that take so long to send that the client (or an in-between proxy) times out the request.
+
+The solution to this in HTTP 1.1 is chunked transfer encoding. The response body can be split up and sent in a series of separate "chunks" for the client to receive and automatically put back together. Ruby's Rack specification supports chunking, as do most frameworks based on it (e.g. Rails, Sinatra, Grape, etc).
+
+The following examples all show the same streaming API in various Rack-based frameworks. Without streaming, the examples could eat up tons of memory, take too long, and time out on the client. With streaming, the following improvements are possible without your client-side code needing any changes:
+
+1. Only 500 orders will ever be in memory at once.
+2. Only one `ApiV1::Entities::Order` will ever be in memory at once.
+3. Only 16kb (roughly) of JSON will ever be in memory at once.
+5. That 16kb of JSON will be sent to the client while the next 16kb of JSON is generating.
+
+**IMPORTANT** Not every Ruby application server supports HTTP chunking. Puma definitely supports it and WEBrick definitely does not. Phusion Passenger claims to but I have not tried it.
+
+## Rails
+
+TODO
+
+## Sinatra
+
+TODO
+
+## Grape
+
+```ruby
+get :orders do
+  enumerator = Order.
+    where("created_at >= ?", 1.year.ago).
+    find_each(batch_size: 500)
+
+  stream JsonEmitter.array(enumerator) { |order|
+    ApiV1::Entities::Order.new(order)
+  }.buffered(16)
+end
+```
+
+## Rack
+
+```ruby
+app = ->(env) {
+  enumerator = Order.
+    where("created_at >= ?", 1.year.ago).
+    find_each(batch_size: 500)
+
+  stream = JsonEmitter.array(enumerator) { |order|
+    order.to_h
+  }.buffered(16)
+
+  [200, {"Content-Type" => "application/json"}, stream]
+}
+```

data/lib/json-emitter/buffered_stream.rb

@@ -0,0 +1,68 @@
+module JsonEmitter
+  #
+  # Represents a stream of JSON to be generated and yielded. It can be treated like any Enumerable.
+  # Unlike JsonEmitter::Stream, the yielded output is buffered into (roughly) equally sized chunks.
+  #
+  class BufferedStream
+    include Enumerable
+
+    #
+    # Initialize a new buffered stream.
+    #
+    # @param enum [Enumerator] An enumerator that yields pieces of JSON.
+    # @param buffer_size [Integer] The buffer size in kb. This is a size *hint*, not a hard limit.
+    # @param unit [Symbol] :bytes | :kb (default) | :mb
+    #
+    def initialize(enum, buffer_size, unit: :kb)
+      @enum = enum
+      @buffer_size = case unit
+                     when :bytes then buffer_size
+                     when :kb then buffer_size * 1024
+                     when :mb then buffer_size * 1024 * 1024
+                     else raise ArgumentError, "unknown buffer size unit ':#{unit}'"
+                     end
+    end
+
+    #
+    # Write the stream to the specified IO object.
+    #
+    # @param io [IO]
+    #
+    def write(io)
+      buffer.each { |str|
+        io << str
+      }
+    end
+
+    #
+    # If a block is given, each chunk of JSON is yielded to it. If not block is given, an Enumerator is returned.
+    #
+    # @return [Enumerator]
+    #
+    def each
+      if block_given?
+        buffer.each { |str|
+          yield str
+        }
+      else
+        buffer
+      end
+    end
+
+    private
+
+    def buffer
+      Enumerator.new { |y|
+        buff = ""
+        @enum.each { |str|
+          buff << str
+          if buff.bytesize >= @buffer_size
+            y << buff
+            buff = ""
+          end
+        }
+        y << buff unless buff.empty?
+      }
+    end
+  end
+end

data/lib/json-emitter/emitter.rb

@@ -0,0 +1,72 @@
+module JsonEmitter
+  #
+  # Builds Enumerators that yield JSON from Ruby Arrays or Hashes.
+  #
+  class Emitter
+    #
+    # Generates an Enumerator that will stream out a JSON array.
+    #
+    # @param enum [Enumerable] Something that can be enumerated over, like an Array or Enumerator. Each element should be something that can be rendered as JSON (e.g. a number, string, boolean, Array, or Hash).
+    # @yield If a block is given, it will be yielded each value in the array. The return value from the block will be converted to JSON instead of the original value.
+    # @return [Enumerator]
+    #
+    def array(enum, &mapper)
+      Enumerator.new { |y|
+        y << "[".freeze
+
+        first = true
+        enum.each { |val|
+          y << ",".freeze unless first
+          first = false if first
+
+          mapped_val = mapper ? mapper.call(val) : val
+          json_values(mapped_val).each { |json_val|
+            y << json_val
+          }
+        }
+
+        y << "]".freeze
+      }
+    end
+
+    #
+    # Generates an Enumerator that will stream out a JSON object.
+    #
+    # @param hash [Hash] Keys should be Strings or Symbols and values should be any JSON-compatible value like a number, string, boolean, Array, or Hash.
+    # @return [Enumerator]
+    #
+    def object(hash)
+      Enumerator.new { |y|
+        y << "{".freeze
+
+        first = true
+        hash.each { |key, val|
+          y << ",".freeze unless first
+          first = false if first
+
+          json_key = MultiJson.dump(key.to_s)
+          y << "#{json_key}:"
+
+          json_values(val).each { |json_val|
+            y << json_val
+          }
+        }
+
+        y << "}".freeze
+      }
+    end
+
+    private
+
+    def json_values(x)
+      case x
+      when Hash then object x
+      when Enumerable then array x
+      when Proc
+        y = x.call
+        json_values y
+      else [MultiJson.dump(x)]
+      end
+    end
+  end
+end

data/lib/json-emitter/stream.rb

@@ -0,0 +1,55 @@
+module JsonEmitter
+  #
+  # Represents a stream of JSON to be generated and yielded. It can be treated like any Enumerable.
+  # The size of the yielded strings can vary from 1 to 1000's. If that's a problem, call JsonEmitter::Stream.buffer.
+  #
+  class Stream
+    include Enumerable
+
+    #
+    # Initialize a new stream.
+    #
+    # @param enum [Enumerator] An enumerator that yields pieces of JSON.
+    #
+    def initialize(enum)
+      @enum = enum
+    end
+
+    #
+    # Returns a new stream that will buffer the output. You can perform the same "write" or "each" operations
+    # on the new stream, but the chunks of output will be (roughly) uniform in size.
+    #
+    # @param buffer_size [Integer] The buffer size in kb. This is a size *hint*, not a hard limit.
+    # @return [JsonEmitter::BufferedStream]
+    #
+    def buffered(buffer_size = 16, unit: :kb)
+      BufferedStream.new(@enum, buffer_size, unit: unit)
+    end
+
+    #
+    # Write the stream to the specified IO object.
+    #
+    # @param io [IO]
+    #
+    def write(io)
+      each { |str|
+        io << str
+      }
+    end
+
+    #
+    # If a block is given, each chunk of JSON is yielded to it. If not block is given, an Enumerator is returned.
+    #
+    # @return [Enumerator]
+    #
+    def each
+      if block_given?
+        @enum.each { |str|
+          yield str
+        }
+      else
+        @enum
+      end
+    end
+  end
+end

data/lib/json-emitter/version.rb

@@ -0,0 +1,4 @@
+module JsonEmitter
+  # Library version
+  VERSION = "0.0.1".freeze
+end

data/lib/json-emitter.rb

@@ -0,0 +1,100 @@
+require 'multi_json'
+
+require 'json-emitter/version'
+require 'json-emitter/emitter'
+require 'json-emitter/stream'
+require 'json-emitter/buffered_stream'
+
+#
+# Efficiently generate very large strings of JSON from Ruby objects.
+#
+# Complex values like arrays and objects may be as large and nested as you need without compromising efficiency.
+# Primitive values will be serialized to JSON using MultiJson.dump. MultiJson finds and uses the most efficient
+# JSON generator you have on your system (e.g. oj) and falls back to the stdlib JSON library.
+#
+# The emitter can be used to output to anything (files, network sockets, etc), and the output can optionally be
+# buffered. This works very well with so-called "HTTP chunked responses" in Rack/Rails/Sinatra/Grape/etc.
+#
+module JsonEmitter
+  #
+  # Generates an stream that will output a JSON array. The input can be any Enumerable, such as an Array or an Enumerator.
+  #
+  # The following example uses minumum memory to genrate a very large JSON array string from an ActiveRecord query.
+  # Only 500 Order records will ever by in memory at once. The JSON will be generated in small chunks so that
+  # the whole string is never in all memory at once.
+  #
+  #   enumerator = Order.limit(10_000).find_each(batch_size: 500)
+  #   stream = JsonEmitter.array(enumerator) { |order|
+  #     {
+  #       number: order.id,
+  #       desc: order.description,
+  #       ...
+  #     }
+  #   }
+  #
+  #   # generate the JSON in chunks and write them to STDOUT
+  #   stream.write($stdout)
+  #
+  #   # generate chunks of JSON and do something with them
+  #   stream.each do |json_chunk|
+  #     # do something with each json chunk
+  #   end
+  #
+  #   # if you need the outputted chunks to be (roughly) equal in size, call "buffered"
+  #   # and pass in the buffer size in kb.
+  #   buffered_stream = stream.buffered(16)
+  #
+  # @param enum [Enumerable] Something that can be enumerated over, like an Array or Enumerator. Each element should be something that can be rendered as JSON (e.g. a number, string, boolean, Array, or Hash).
+  # @yield If a block is given, it will be yielded each value in the array. The return value from the block will be converted to JSON instead of the original value.
+  # @return [JsonEmitter::Stream]
+  #
+  def self.array(enum, &mapper)
+    emitter = Emitter.new.array(enum, &mapper)
+    Stream.new(emitter)
+  end
+
+  #
+  # Generates an stream that will output a JSON object.
+  #
+  # If some of the values will be large arrays, use Enumerators or lazy Enumerators to build each element on demand
+  # (to potentially save lots of RAM).
+  #
+  # You can also use Procs to generate large arrays, objects, blocks of text, etc. They'll only be used one at
+  # a time, which can potentially save lots of RAM.
+  #
+  # The following example generates a very large JSON object string from several components.
+  #
+  #   stream = JsonEmitter.object({
+  #     time: Time.now.iso8601,
+  #     is_true: true,
+  #     orders: Order.limit(10_000).find_each(batch_size: 500).lazy.map { |order|
+  #       {number: order.id, desc: order.description}
+  #     },
+  #     high_mem_thing_1: ->() {
+  #       get_high_mem_thing1()
+  #     },
+  #     high_mem_thing_2: ->() {
+  #       get_high_mem_thing2()
+  #     },
+  #   })
+  #
+  #   # generate the JSON in chunks and write them to STDOUT
+  #   stream.write($stdout)
+  #
+  #   # generate chunks of JSON and do something with them
+  #   stream.each do |json_chunk|
+  #     # do something with each json chunk
+  #   end
+  #
+  #   # if you need the outputted chunks to be (roughly) equal in size, call "buffered"
+  #   # and pass in the buffer size in kb.
+  #   buffered_stream = stream.buffered(16)
+  #
+  # @param hash [Hash] Keys should be Strings or Symbols and values should be any JSON-compatible value like a number, string, boolean, Array, or Hash.
+  # @return [JsonEmitter::Stream]
+  #
+  def self.object(hash)
+    emitter = Emitter.new.object(hash)
+    Stream.new(emitter)
+  end
+end

metadata

@@ -0,0 +1,63 @@
+--- !ruby/object:Gem::Specification
+name: json-emitter
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Jordan Hollinger
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2019-01-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: multi_json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+description: Generates and outputs JSON in well-sized chunks
+email: jordan.hollinger@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/json-emitter.rb
+- lib/json-emitter/buffered_stream.rb
+- lib/json-emitter/emitter.rb
+- lib/json-emitter/stream.rb
+- lib/json-emitter/version.rb
+homepage: https://jhollinger.github.io/json-emitter/
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.3.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.7.6
+signing_key: 
+specification_version: 4
+summary: Efficiently generate tons of JSON
+test_files: []