json-streamer 2.0.1 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 15f8ae34f74e2fa49f528a81adde9fc20ad85a46
-  data.tar.gz: b856eaaf477457e4a396b1941ca5ff9a7da2b6cf
+SHA256:
+  metadata.gz: 8a3dd5e19a0ad214f6121458dd0de84db79a45ba2f321a83eadfbb26de2f575c
+  data.tar.gz: 55099a1cddf21b0392eae5306b74d9e7e7fd2665d12fd5c995a972786afc2d71
 SHA512:
-  metadata.gz: 204b4cf22c6f0e3bac4f41624af13ee858565393dc98d0b9b257711aff7c254a11bbbde4dc2c9effe6343bc0faf98774fad9c836440e310f4302acddf10645fd
-  data.tar.gz: 2ade58734aca4cd8adef60c563db8ae5748f11167f3e251aef82bad545d250b5a735cb7d41c599ebaae41cb4327aecea2e3f9637649508a173d0758cfc74ef94
+  metadata.gz: 87c6bbe06080168b7711c6620aa8522f6d86976db1dc826e889b1ca02b29ca467c89e054aba36063b392fb8a9052945fc60c08009853cd89ea6ec1481d23ca9a
+  data.tar.gz: 3335826bd0f377b72db28fbbdeb882067f84203f03fc68eb6e957bda0a19c82790ffe743117442cee49bcff4fde285675e054d4b3ff05ecb103c864c52035124

data/.travis.yml

@@ -1,5 +1,9 @@
 sudo: false
 language: ruby
+addons:
+  apt:
+    packages:
+    - libyajl2
 rvm:
   - 2.3.3
 before_install: gem install bundler -v 1.14.6

data/README.md

@@ -1,18 +1,27 @@
 # Json::Streamer
 
-#### Ruby utility that supports JSON streaming allowing you to get data based on various criteria (key, nesting level, etc).
+#### Ruby gem for getting data from JSON streams based on various criteria (key, nesting level, etc).
+
+Status and support
+
+- ✔ stable
+- ✔ supported
+- ✖ no ongoing development
+
+<!--- Version informartion -->
+*You are viewing the README of version [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0). You can find other releases [here](https://github.com/thisismydesign/json-streamer/releases).*
+<!--- Version informartion end -->
 
 | Branch | Status |
 | ------ | ------ |
 | Release | [![Build Status](https://travis-ci.org/thisismydesign/json-streamer.svg?branch=release)](https://travis-ci.org/thisismydesign/json-streamer)   [![Coverage Status](https://coveralls.io/repos/github/thisismydesign/json-streamer/badge.svg?branch=release)](https://coveralls.io/github/thisismydesign/json-streamer?branch=release)   [![Gem Version](https://badge.fury.io/rb/json-streamer.svg)](https://badge.fury.io/rb/json-streamer)   [![Total Downloads](http://ruby-gem-downloads-badge.herokuapp.com/json-streamer?type=total)](https://rubygems.org/gems/json-streamer) |
 | Development | [![Build Status](https://travis-ci.org/thisismydesign/json-streamer.svg?branch=master)](https://travis-ci.org/thisismydesign/json-streamer)   [![Coverage Status](https://coveralls.io/repos/github/thisismydesign/json-streamer/badge.svg?branch=master)](https://coveralls.io/github/thisismydesign/json-streamer?branch=master) |
 
-*If you've tried JSON streaming with other Ruby libraries before (e.g. [JSON::Stream](https://github.com/dgraham/json-stream), [Yajl::FFI](https://github.com/dgraham/yajl-ffi)):*
+####  If you've tried JSON streaming with other Ruby libraries before (e.g. [JSON::Stream](https://github.com/dgraham/json-stream), [Yajl::FFI](https://github.com/dgraham/yajl-ffi))
 
 This gem will basically spare you the need to define your own callbacks (i.e. implement an actual JSON parser using `start_object`, `end_object`, `key`, `value`, etc.).
 
-
-*If you're new to this:*
+#### If you're new to this
 
 Streaming is useful for
 - big files that do not fit in the memory (or you'd rather avoid the risk)
@@ -21,12 +30,16 @@ Streaming is useful for
 
 This gem is aimed at making streaming as easy and convenient as possible.
 
-*Performance:*
+#### Performance
+
+Highly depends on the event generator. Out of the box the gem uses [JSON::Stream](https://github.com/dgraham/json-stream). It was chosen because it's a pure Ruby parser with no runtime dependencies. You can use any custom event generator, such as [Yajl::FFI](https://github.com/dgraham/yajl-ffi) which is dependent on the native YAJL library and is [~10 times faster](https://github.com/dgraham/yajl-ffi#performance). See the [Custom event generators](#custom-event-generators) chapter.
 
-The gem uses JSON::Stream's events in the background. It was chosen because it's a pure Ruby parser.
-A similar implementation can be done using the ~10 times faster Yajl::FFI gem that is dependent on the native YAJL library.
 I did not measure the performance of my implementation on top of these libraries.
 
+#### Dependencies
+
+The gem's single runtime dependency is [JSON::Stream](https://github.com/dgraham/json-stream). It is only loaded if the default event generator is used.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -45,18 +58,14 @@ Or install it yourself as:
 
 ## Usage
 
-Check the unit tests for more examples ([spec/streamer_spec.rb](spec/json/streamer/json_streamer_spec.rb)).
-
-One `streamer` object handles one set of conditions. For multiple conditions create multiple streamers. For more details see [this discussion](https://github.com/thisismydesign/json-streamer/issues/9).
-
 ```ruby
 require 'json/streamer'
 ```
 
-### v1.2 (and above) API
-
 #### Passing IO upfront
 
+Since [v1.2.0](https://github.com/thisismydesign/json-streamer/releases/tag/v1.2.0)
+
 ```ruby
 file_stream = File.open('data.json', 'r')
 chunk_size = 500 # defaults to 1000
@@ -181,9 +190,23 @@ def receive_data(data)
 end
 ```
 
+#### Custom event generators
+
+Since [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0)
+
+```ruby
+require "yajl/ffi"
+
+Json::Streamer.parser(event_generator: Yajl::FFI::Parser.new)
+```
+
+Any parser can be used that provides the right events. The gem is tested with [Yajl::FFI](https://github.com/dgraham/yajl-ffi) and [JSON::Stream](https://github.com/dgraham/json-stream).
+
 #### Custom yield conditions
 
-[v2.0.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.0.0) introduces custom conditions which provide ultimate control over what to yield.
+Since [v2.0.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.0.0)
+
+Custom conditions provide ultimate control over what to yield.
 
 The Conditions API exposes 3 callbacks:
 - `yield_value`
@@ -246,7 +269,61 @@ Output:
 {"key2"=>"value"}
 ```
 
-### Legacy API (pre-v1.2)
+#### Get an Enumerable when not passing a block
+
+Since [v2.1.0](https://github.com/thisismydesign/json-streamer/releases/tag/v2.1.0)
+
+When _not_ passed a block both `get` and `get_with_conditions` return an enumerator of the requested objects. When passed a block they return an empty enumerator. This means that **when _not_ passed a block the requested objects will accumulate in memory**.
+
+Without block
+
+```ruby
+objects = streamer.get(nesting_level:1)
+p objects
+```
+
+Input:
+```json
+{
+    "object1": "first_level_value",
+    "object2": {}
+}
+```
+
+Output:
+```ruby
+["first_level_value", {}]
+```
+
+With block
+
+```ruby
+unyielded_objects = streamer.get(nesting_level:1) { |object| do_something(object) }
+p unyielded_objects
+```
+
+Input:
+```json
+{
+    "object1": "first_level_value",
+    "object2": {}
+}
+```
+
+Output:
+```ruby
+[]
+```
+
+#### Other usage information
+
+Check the unit tests for more examples ([spec/streamer_spec.rb](spec/json/streamer/json_streamer_spec.rb)).
+
+One `streamer` object handles one set of conditions. For multiple conditions create multiple streamers. For more details see [this discussion](https://github.com/thisismydesign/json-streamer/issues/9).
+
+#### Deprecated API
+
+Pre [v1.2.0](https://github.com/thisismydesign/json-streamer/releases/tag/v1.2.0)
 
 This functionality is deprecated but kept for compatibility reasons.
 
@@ -260,13 +337,11 @@ streamer = Json::Streamer::JsonStreamer.new
 streamer.parser << data
 ```
 
-## Feedback
-
-Any feedback is much appreciated.
+## Contribution and feedback
 
-I can only tailor this project to fit use-cases I know about - which are usually my own ones. If you find that this might be the right direction to solve your problem too but you find that it's suboptimal or lacks features don't hesitate to contact me.
+This project is built around known use-cases. If have one that isn't covered don't hesitate to open an issue and start a discussion.
 
-Please let me know if you make use of this project so that I can prioritize further efforts.
+Bug reports and pull requests are welcome on GitHub at https://github.com/thisismydesign/json-streamer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 ## Conventions
 
@@ -278,10 +353,6 @@ This gem is developed using the following conventions:
 - [RFC memo about key words used to Indicate Requirement Levels](https://tools.ietf.org/html/rfc2119)
 - [Bundler improvements](https://github.com/thisismydesign/bundler-improvements)
 
-## Contributing
-
-Bug reports and pull requests are welcome on GitHub at https://github.com/thisismydesign/json-streamer. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
-
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/json-streamer.gemspec

@@ -30,6 +30,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "guard"
   spec.add_development_dependency "guard-bundler"
   spec.add_development_dependency "guard-rspec"
+  spec.add_development_dependency "yajl-ffi"
 
   spec.add_dependency "json-stream"
 end

data/lib/json/streamer.rb

@@ -2,8 +2,8 @@ require_relative "streamer/json_streamer"
 
 module Json
   module Streamer
-    def self.parser(file_io: nil, chunk_size: 1000)
-      JsonStreamer.new(file_io, chunk_size)
+    def self.parser(file_io: nil, chunk_size: 1000, event_generator: :default)
+      JsonStreamer.new(file_io, chunk_size, event_generator)
     end
   end
 end

data/lib/json/streamer/json_streamer.rb

@@ -1,5 +1,3 @@
-require "json/stream"
-
 require_relative 'conditions'
 require_relative 'parser'
 
@@ -9,8 +7,8 @@ module Json
 
       attr_reader :parser
 
-      def initialize(file_io = nil, chunk_size = 1000)
-        @event_generator = JSON::Stream::Parser.new
+      def initialize(file_io = nil, chunk_size = 1000, event_generator = :default)
+        @event_generator = make_event_generator(event_generator)
 
         @file_io = file_io
         @chunk_size = chunk_size
@@ -24,24 +22,39 @@ module Json
         conditions = Conditions.new(yield_level: nesting_level, yield_key: key)
         conditions.yield_value = ->(aggregator:, value:) { false } unless yield_values
 
-        # TODO: deprecate symbolize_keys and move to initialize
         @parser = Parser.new(@event_generator, symbolize_keys: symbolize_keys)
+        unyielded_items = []
 
         parser.get(conditions) do |obj|
-          yield obj
+          if block_given?
+            yield obj
+          else
+            unyielded_items.push(obj)
+          end
+
+          obj
         end
 
         process_io
+        
+        unyielded_items
       end
 
       def get_with_conditions(conditions, options = {})
         @parser = Parser.new(@event_generator, symbolize_keys: options[:symbolize_keys])
+        unyielded_items = []
 
         parser.get(conditions) do |obj|
-          yield obj
+          if block_given?
+            yield obj
+          else
+            unyielded_items.push(obj)
+          end
         end
 
         process_io
+
+        unyielded_items
       end
 
       def aggregator
@@ -53,6 +66,16 @@ module Json
       def process_io
         @file_io.each(@chunk_size) { |chunk| parser << chunk } if @file_io
       end
+
+      def make_event_generator(generator)
+        case generator
+        when :default
+          require 'json/stream'
+          JSON::Stream::Parser.new
+        else
+          generator
+        end
+      end
     end
   end
 end

data/lib/json/streamer/version.rb

@@ -1,5 +1,5 @@
 module Json
   module Streamer
-    VERSION = "2.0.1"
+    VERSION = "2.1.0"
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: json-streamer
 version: !ruby/object:Gem::Version
-  version: 2.0.1
+  version: 2.1.0
 platform: ruby
 authors:
 - thisismydesign
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-09-29 00:00:00.000000000 Z
+date: 2018-11-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -136,6 +136,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: yajl-ffi
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: json-stream
   requirement: !ruby/object:Gem::Requirement
@@ -197,7 +211,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.11
+rubygems_version: 2.7.7
 signing_key: 
 specification_version: 4
 summary: Utility to support JSON streaming allowing you to get data based on various