data_list_converter 0.3.10 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 85c9410c6b8c64a1af696f42485e098f43f432c2
-  data.tar.gz: 930b7bb74e9d441e183451d4329e639c06aa3cdf
+  metadata.gz: 7862bf6ab71eb4187b03ba25e7bd199ea9d3eb51
+  data.tar.gz: f095138e55e9cc72c1691628c782a9c7ec5a203d
 SHA512:
-  metadata.gz: b53cb6e80ac57b256b74100421b12d14c9e1f57bee172807bb0f6437efe27e2c2c8915705d52b0e1ff01855188a6e4a3ac274138f7623cbec648a187ba349631
-  data.tar.gz: c87df4a93536be36f7aec7d88c3c93325b39fe03ab09414d4d1533024569e6d550f46cfff724df48bd39612bd6b52727473acbe20b1fdd273b92194b4f9286fc
+  metadata.gz: dfaa0989bafc3ae7e8d8675827b5728dd0476e1f62f79021b121407af2fa05ee4b47456f1db22aa46eb3aadfb4549a4ca3d0cf0c85bd0f8e6334c5105b36f1aa
+  data.tar.gz: f9056d920a5164dc109490c809fd8d57174ffcc30d6851f52cb676937befce297e42ff89c702b60b2d2b825c3edc55b8b744538744296bb6c83c36e4a1bcf9ad

data/README.md

@@ -22,61 +22,73 @@ DataListConverter.convert(:xls_file, :multi_sheet_item_data, {filename: 'result.
 You can also add filter to this process:
 
 ```ruby
-filters = [
-  # filter with default options
-  :limit,
-  # filter with options
-  {limit: {size: 2}},
-  # multiple filters
-  [{limit: {size: 12}},
-   {count: {size: 4}}],
-]
-filters.map do |filter|
-  data = [{name: "james"}] * 10
-  DataListConverter.convert(:item_data, :table_data, data, table_iterator: {filter: filter})
-end
+data = (1..20).map{|i| {name: "user-#{i}", age: i+20}}
+# filter with default options (limit 10)
+DataListConverter.convert(:item_data, :table_data, data, item_iterator: {filter: :limit})
+# filter with options
+DataListConverter.convert(:item_data, :table_data, data, item_iterator: {filter: {limit: {size: 2}}})
+# multiple filters
+DataListConverter.convert(:item_data, :table_data, data, item_iterator: {filter: [{limit: {size: 12}}, {count: {size: 4}}]})
 ```
 
-Please read [the source code](https://github.com/halida/data_list_converter/blob/master/lib/data_list_converter/) for more information,
-also you can check [test examples](https://github.com/halida/data_list_converter/blob/master/test/).
-
 ## Data Types
 
+Default data types:
+
 - **item_data** like: `[{name: 'James', age: '22'}, ...]`, keys should be symbol.
 - **item_iterator** iterator for item_data, used like: iter.call{|item| out << item}
 - **table_data** like: `[["name", "age"], ["James", "22"], ["Bob", "33"], ...]`
 - **table_iterator** iterator for table_data
-- **csv_file** file in csv format
-- **xls_file** file in excel format, should install `spreadsheet` gem, and `require 'data_list_converter/types/xls_file'`
-- **xlsx_file** file in excel xml format, should install `rubyXL` gem, and `require 'data_list_converter/types/xlsx_file'`
 - **multi_sheet** Contains several data with sheets:
     - **multi_sheet_table_iterator**: like: `{sheet1: table_iterator1, sheet2: table_iterator2}`
     - **multi_sheet_table_data**: like: `{sheet1: [['name', 'age'], ...], sheet2: ...}`
     - **multi_sheet_item_iterator**: like: `{sheet1: item_iterator1, sheet2: item_iterator2}`
     - **multi_sheet_item_data**: like: `{sheet1: [{name: 'James', age: 32}], sheet2: ...}`
-- **records** ActiveRecord records, usage: `DataListConverter.convert(:records, :item_data, Users.where(condition), item_iterator: {columns: [:name, :age]})`
 
+Plugin data types, should required first by `require 'data_list_converter/types/#{type}'`
+
+- **csv_file** file in csv format
+- **xls_file** file in excel format, should install `spreadsheet` gem first
+- **xlsx_file** file in excel xml format, should install `rubyXL` gem first
+- **records** ActiveRecord records
+
+Please check [test examples](https://github.com/halida/data_list_converter/blob/master/test/types_test.rb) to see how to use those types.
 
 ## Filters
 
-**item_iterator/table_iterator limit**: limit item_iterator result counts, usage: `DataListConverter.convert(:item_data, :table_data, item_data, item_iterator: {filter: {limit: {size: 2}}})`
+**item_iterator/table_iterator limit**: limit item_iterator result counts, usage: `DataListConverter.convert(:item_data, :table_data, item_data, item_iterator: {filter: {limit: {size: 2}}})`, default limit size is 10.
 
-**item_iterator count**: count item_iterator items, usage: `DataListConverter.convert(:xls_file, :item_data, {filename: 'result.xls'}, item_iterator: {filter: {count: {size: 10}}})`, it will print current item counts every `size`.
+**item_iterator count**: count item_iterator items, usage: `DataListConverter.convert(:xls_file, :item_data, {filename: 'result.xls'}, item_iterator: {filter: {count: {size: 10}}})`, it will print current item counts every `size`, please [see here](https://github.com/halida/data_list_converter/blob/master/lib/data_list_converter/filters/count.rb) for more options.
 
-Please check more [test examples](https://github.com/halida/data_list_converter/blob/master/test/filters_test.rb)
+Please see [test examples](https://github.com/halida/data_list_converter/blob/master/test/filters_test.rb) to learn how to use filter.
+
+## helpers
+
+- **types**: Get current valid types.
+- **routes**: Get current valid routes.
+- **file_types**: Get current file types, which is the types has `_file` suffix.
+- **get_file_format**: Get file type by filename, which compare file extension with `file_types`. `DataListConverter.get_file_format('xxx.xls') == :xls_file`
+- **save_to_file**: Save data to file, it will use `get_file_format` to find proper file format.  `DataListConverter.save_to_file(filename, data, data_format=:item_data)`
+- **load_from_file**: Get data from file, it will use `get_file_format` to find proper file format. `DataListConverter.load_from_file(filename, data_format=:item_data)`
+- **unify_item_data_keys**: Sometimes in the `item_data` list, each data keys don't exactly same, so use this function to fix it, example: `DataListConverter.unify_item_data_keys([{a: 12}, {b: 11}]) == [{a: 12, b: nil}, {a: nil, b: 11}]`.
+- **flatten**: Flatten multi level item_data list into one level, example: `DataListConverter.flatten({a: {b: 12}, c: {d: {e: 11}}}) == {:"a:b"=>12, :"c:d:e"=>11}`, can change seperator: `DataListConverter.flatten(data, '_')`
 
 ## Extend
 
 You can add your own data types and filters, example:
 
 ```ruby
-DataListConverter.register_converter(:records, :item_iterator) do |records, options|
-  columns = options[:columns]
+DataListConverter.register_converter(:records, :item_iterator) do |input, options|
+  query = self.parameter(input, :query, :input)
+  columns = self.parameter(input, :columns, :input)
+  display = input[:display] || columns
+
   lambda { |&block|
-    records.find_each do |record|
-      item = columns.map do |column|
-        [column.first.to_sym, record.send(column[1])]
-      end.to_h
+    query.pluck(*columns).each do |data|
+      item = {}
+      data.each_with_index do |d, i|
+        item[display[i]] = d
+      end
       block.call(item)
     end
   }
@@ -94,14 +106,3 @@ DataListConverter.register_filter(:item_iterator, :limit) do |proc, options|
   }
 end
 ```
-
-
-## Todo
-
-- check type exists
-- route not found, list all routes
-- register can bind method again
-- better API
-- better error message
-- load file/save file with auto format by extname
-- report error when cannot find type

data/changelog.md

@@ -1,3 +1,7 @@
+## 0.4
+
+Add parameter check, type check, change record type API.
+
 ## 0.3
 
 Refactor.

data/data_list_converter.gemspec

@@ -17,6 +17,8 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'spreadsheet', '~> 1.0'
   s.add_development_dependency 'rubyXL', '~> 3.3'
   s.add_development_dependency 'pry', '~> 0.10.1'
+  s.add_development_dependency 'sqlite3', '~> 1.3'
+  s.add_development_dependency 'activerecord', '~> 4.2'
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")

data/lib/data_list_converter/base.rb

@@ -21,6 +21,7 @@ class DataListConverter
       puts "#{Time.now.strftime('%Y-%m-%d %H:%M:%S')}\t#{msg}"
     end
 
+    # register_converter(:item_data, :item_iterator){ |data, options| ... }
     def register_converter(from_type, to_type, &block)
       @route_map = nil # clear cache
       CONVERTERS[[from_type, to_type]] = block
@@ -91,7 +92,11 @@ class DataListConverter
     # If we want to convert between types, like: convert item_data into csv_file,
     # we need find all the intermidate data type, like: [:item_data, :item_iterator, :table_iterator, :csv_file]
     def find_route(from_type, to_type)
+      [from_type, to_type].each do |type|
+        raise Exception, "cannot find type: #{type}" unless self.types.include?(type)
+      end
       raise Exception, "from_type should not equal to to_type: #{from_type}" if from_type == to_type
+
       # map wide search
       checked = Set.new
       checking = Set.new([from_type])
@@ -124,7 +129,10 @@ class DataListConverter
           checking += Set.new(next_nodes) - checked
         end
       end
-      raise Exception, "Route not found: #{from_type} -> #{to_type}"
+
+      log = ["Route not found: #{from_type} -> #{to_type}", "Current routes:"]
+      log += self.routes.map{|from, to| "#{from} -> #{to}"}
+      raise Exception, log.join("\n")
     end
 
     # convert adjacency list into quick lookup hash

data/lib/data_list_converter/filters/count.rb

@@ -5,6 +5,7 @@ class DataListConverter
     msg_format = options[:msg] || "on %{count}"
     total_format = options[:total] || "total: %{total}"
     out = options[:out] || STDOUT
+
     total = 1
     lambda { |&block|
       proc.call do |item|

data/lib/data_list_converter/helper.rb

@@ -23,6 +23,10 @@ class DataListConverter
     CONVERTERS.keys.flatten.uniq.sort
   end
 
+  def self.routes
+    CONVERTERS.keys
+  end
+
   def self.file_types
     matcher = /(.*)_file$/
     DataListConverter.types.select do |type|
@@ -69,4 +73,10 @@ class DataListConverter
     end
   end
 
+  def self.parameter(data, key, type)
+    raise Exception, "`#{type}` should be hash, not `#{data.class}`: #{data}" unless data.kind_of?(Hash)
+    raise Exception, "Need `#{key}` for `#{type}`, current: #{data}" if not data.has_key?(key)
+    data.fetch(key)
+  end
+
 end

data/lib/data_list_converter/types/csv_file.rb

@@ -3,7 +3,8 @@ require 'csv'
 class DataListConverter
   self.register_converter(:csv_file, :table_iterator) do |input, options|
     lambda { |&block|
-      CSV.open(input[:filename]) do |csv|
+      filename = self.parameter(input, :filename, :input)
+      CSV.open(filename) do |csv|
         csv.each do |row|
           block.call(row)
         end
@@ -12,7 +13,8 @@ class DataListConverter
   end
 
   self.register_converter(:table_iterator, :csv_file) do |proc, options|
-    CSV.open(options[:filename], 'wb', force_quotes: true) do |csv|
+    filename = self.parameter(options, :filename, :csv_file)
+    CSV.open(filename, 'wb', force_quotes: true) do |csv|
       proc.call do |row|
         csv << row
       end

data/lib/data_list_converter/types/marshal.rb

@@ -1,13 +1,15 @@
 class DataListConverter
 
   def self.marshal_file_to_data(input, options=nil)
-    File.open(input[:filename]) do |f|
+    filename = self.parameter(input, :filename, :input)
+    File.open(filename) do |f|
       Marshal.load(f)
     end
   end
 
   def self.marshal_data_to_file(data, options)
-    File.open(options[:filename], 'w+') do |f|
+    filename = self.parameter(options, :filename, :marshal)
+    File.open(filename, 'w+') do |f|
       Marshal.dump(data, f)
     end
     options[:filename]

data/lib/data_list_converter/types/records.rb

@@ -1,13 +1,20 @@
 # ActiveRecords
 class DataListConverter
 
-  self.register_converter(:records, :item_iterator) do |records, options|
-    columns = options[:columns]
+  # columns = [:table_column1, :table_column2, ...]
+  # display = [:display_name1, :display_name2, ...]
+  # convert(:records, :item_data, query: query, columns: columns, display: display)
+  self.register_converter(:records, :item_iterator) do |input, options|
+    query = self.parameter(input, :query, :input)
+    columns = self.parameter(input, :columns, :input)
+    display = input[:display] || columns
+
     lambda { |&block|
-      records.find_each do |record|
-        item = columns.map do |column|
-          [column[0].to_sym, record.send(column[1])]
-        end.to_h
+      query.pluck(*columns).each do |data|
+        item = {}
+        data.each_with_index do |d, i|
+          item[display[i]] = d
+        end
         block.call(item)
       end
     }

data/lib/data_list_converter/types/xls_file.rb

@@ -5,7 +5,8 @@ class DataListConverter
 
   self.register_converter(:xls_file, :table_iterator) do |input, options|
     lambda { |&block|
-      book = Spreadsheet.open(input[:filename])
+      filename = self.parameter(input, :filename, :input)
+      book = Spreadsheet.open(filename)
       sheet = book.worksheet input[:sheet] || 0
       sheet.each do |row|
         block.call(row.to_a)
@@ -21,8 +22,9 @@ class DataListConverter
       sheet.row(i).push *row
       i += 1
     end
-    book.write(options[:filename])
-    options[:filename]
+    filename = self.parameter(options, :filename, :xls_file)
+    book.write(filename)
+    filename
   end
 
   self.register_converter(:multi_sheet_table_iterator, :xls_file) do |data, options|
@@ -35,13 +37,14 @@ class DataListConverter
         i += 1
       end
     end
-    filename = options[:filename]
+    filename = self.parameter(options, :filename, :xls_file)
     book.write(filename)
     filename
   end
 
-  self.register_converter(:xls_file, :multi_sheet_table_iterator) do |data, options|
-    book = Spreadsheet.open(data[:filename])
+  self.register_converter(:xls_file, :multi_sheet_table_iterator) do |input, options|
+    filename = self.parameter(input, :filename, :input)
+    book = Spreadsheet.open(filename)
     book.worksheets.map do |sheet|
       iterator = lambda { |&block|
         sheet.each do |row| 

data/lib/data_list_converter/types/xlsx_file.rb

@@ -5,7 +5,8 @@ class DataListConverter
 
   self.register_converter(:xlsx_file, :table_iterator) do |input, options|
     lambda { |&block|
-      book = RubyXL::Parser.parse(input[:filename])
+      filename = self.parameter(input, :filename, :input)
+      book = RubyXL::Parser.parse(filename)
       sheet = book.worksheets[input[:sheet] || 0]
       sheet.each do |row|
         next unless row
@@ -26,7 +27,7 @@ class DataListConverter
       end
       i += 1
     end
-    filename = options[:filename]
+    filename = self.parameter(options, :filename, :xlsx_file)
     book.write(filename)
     filename
   end
@@ -55,13 +56,14 @@ class DataListConverter
         i += 1
       end
     end
-    filename = options[:filename]
+    filename = self.parameter(options, :filename, :xlsx_file)
     book.write(filename)
     filename
   end
 
-  self.register_converter(:xlsx_file, :multi_sheet_table_iterator) do |data, options|
-    book = RubyXL::Parser.parse(data[:filename])
+  self.register_converter(:xlsx_file, :multi_sheet_table_iterator) do |input, options|
+    filename = self.parameter(input, :filename, :input)
+    book = RubyXL::Parser.parse(filename)
     book.worksheets.map do |sheet|
       iterator = lambda { |&block|
         sheet.each do |row|

data/lib/data_list_converter/version.rb

@@ -1,3 +1,3 @@
 class DataListConverter
-  VERSION = "0.3.10".freeze
+  VERSION = "0.4.0".freeze
 end

data/test/base_test.rb

@@ -40,6 +40,12 @@ describe DataListConverter do
       string.string.must_equal ".4\n.8\n.12\n"
     end
 
+    it 'test data format exists' do
+      -> {
+        DataListConverter.convert(:item_data, :ppp, {a: 12})
+      }.must_raise Exception
+    end
+
   end
 
 end

data/test/types_test.rb

@@ -99,5 +99,36 @@ describe DataListConverter do
       end
     end
   end
+
+  describe :records do
+    specify do
+      require 'sqlite3'
+      require 'active_record'
+
+      ActiveRecord::Base.establish_connection(
+        adapter: 'sqlite3',
+        database: ':memory:'
+      )
+
+      ActiveRecord::Schema.define do
+        create_table :users, force: true do |t|
+          t.string :name
+          t.integer :age
+        end
+      end
+
+      class User < ActiveRecord::Base; end
+      (1..10).each{ |i| User.create(name: "user-#{i}", age: i+20) }
+
+      query = User.where("age > 25 and age < 27")
+      columns = [:name, :age]
+      @c.convert(:records, :item_data, query: query, columns: columns).
+        must_equal([{name: "user-6", age: 26}])
+
+      display = [:n, :a]
+      @c.convert(:records, :item_data, query: query, columns: columns, display: display).
+        must_equal([{n: "user-6", a: 26}])
+    end
+  end
   
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_list_converter
 version: !ruby/object:Gem::Version
-  version: 0.3.10
+  version: 0.4.0
 platform: ruby
 authors:
 - linjunhalida
@@ -80,6 +80,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.10.1
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.2'
 description: Data List Converter is a tool to convert data between different formats.
 email:
 - linjunhalida@gmail.com