embulk-plugin-input-roo-excel 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 39175d88b536db9345ddc1f516ab03dd1660a1db
+  data.tar.gz: 7745e6d72fd68c9f173378bfd98d5c2228e2aa71
+SHA512:
+  metadata.gz: b42ca65857ec1dd48977bc63b78d5c5934361c6fbe5cb8e97d42e4d9e65a18d4c33e38e6350ff396a5d2111f601c8eb29b31368a7b768d72fd6cd482db9639b8
+  data.tar.gz: 9a8f84401eaa961656fade25a3d7b33befaa22d5e4ee996f0c115e5aa21099310903a5de7d6e658d873db9f83df795ed914ca2394e05fe22d425908a870e89bc

data/.gitignore

@@ -0,0 +1,14 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in embulk-plugin-input-roo-excel.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2015 Hiroyuki Sato
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.ja.md

@@ -0,0 +1,134 @@
+# embulk-plugin-input-roo-excel
+
+[embulk-plugin-input-roo-excel](https://github.com/hiroyuki-sato/embulk-plugin-input-roo-excel)は、[Embulk](https://github.com/embulk/embulk)で、xlsxファイルを読込みするための入力プラグインです。
+
+## インストール方法
+
+Embulkのgemインストール方法にならって次のようにしてパッケージを導入します。
+
+
+```
+java -jar embulk.jar gem intall embulk-plugin-input-roo-excel
+```
+
+また本プラグインは、xlsxファイルの読込みにrooを利用しているので、自動的にrooが導入されない場合は次のようにしてrooもインストールしてください。
+
+```
+java -jar ~/embulk.jar gem install roo
+Fetching: ruby-ole-1.2.11.8.gem (100%)
+Successfully installed ruby-ole-1.2.11.8
+Fetching: spreadsheet-1.0.1.gem (100%)
+Successfully installed spreadsheet-1.0.1
+Fetching: nokogiri-1.6.6.2-java.gem (100%)
+Successfully installed nokogiri-1.6.6.2-java
+Fetching: rubyzip-1.1.7.gem (100%)
+Successfully installed rubyzip-1.1.7
+Fetching: roo-1.13.2.gem (100%)
+Successfully installed roo-1.13.2
+5 gems installed
+```
+
+
+導入したパッケージは次のコマンドで確認をすることができます。(~/.embulk以下に導入されます。)
+
+```
+java -jar ~/embulk.jar gem list
+
+*** LOCAL GEMS ***
+
+ffi (1.9.3 java)
+jar-dependencies (0.1.2)
+jruby-openssl (0.9.5 java)
+json (1.8.0 java)
+krypt (0.0.2)
+krypt-core (0.0.2 universal-java)
+krypt-provider-jdk (0.0.2)
+nokogiri (1.6.6.2 java)
+rake (10.1.0)
+rdoc (4.0.1)
+roo (1.13.2)
+ruby-ole (1.2.11.8)
+rubyzip (1.1.7)
+spreadsheet (1.0.1)
+```
+
+設定ファイル
+
+
+## 設定
+
+設定
+
+| 項目名   | 説明                         | 未指定時    |
+|----------|------------------------------|-------------|
+| data_pos | データが何行目から開始するか | 1           |
+| sheet    | 読み込みたいシート名         | 最初のシート|
+| paths    | xlsxを保存したディレクトリ   |             |
+| colums   | Embulkに取り込むカラム名     |             |
+
+
+カラム名は次のようなパラメータがあります。
+
+
+| 項目名   | 設定                         |
+|----------|------------------------------|
+| name     | カラムの名前                 |
+| type     | 型                           |
+
+型は、Embulkの型にあわせてください
+
+* booealn (未テスト)
+* long: 整数
+* double: 浮動小数点
+* string: 文字列
+* timestamp: 日時
+
+## 設定例
+
+シート名: "The Beatles"に次のようなデータが格納されている場合
+
+| No | first_name  | first_name | nickname | birthday   |
+|----|-------------|------------|----------|------------|
+| 1  | John        | Lennon     | John     | 1940/10/09 |
+| 2  | Paul        | McCartney  | Paul     | 1942/06/18 |
+| 3  | George      | Harrison   | George   | 1943/02/25 |
+| 4  | Ringo       | Starr      | Ringo    | 1940/07/07 |
+
+
+設定は次のように記述します。
+
+```
+in:
+  type: roo_excel
+  sheet: "The Beatles"
+  data_pos: 2
+  paths: ["/path/to/beatles"]
+  columns:
+    - { name: no, type: long }
+    - { name: first_name, type: string }
+    - { name: last_name,  type: string }
+    - { name: nick_name,  type: string }
+    - { name: birthday,   type:timestamp }
+out:
+  type: stdout
+```
+
+## 実行例
+
+```
+java -jar embulk.jar preview config.yml
+java -jar embulk.jar run config.yml
+```
+
+## 既存の問題点
+
+* doneはまだ動きません。
+* '1:00'など時刻を記載するとExcel上はTimeになりますが良い変換方法が思いつかないので型をdoubleにして、3600.0と秒に変換してください。
+
+## Contributing
+
+1. Fork it ( https://github.com/hiroyuki-sato/embulk-plugin-input-roo-excel/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/README.md

@@ -0,0 +1,124 @@
+# embulk-plugin-input-roo-excel
+
+[Japanese Page](README.ja.md)
+
+This is a input plugin for [Embulk](https://github.com/embulk/embulk) to read xlsx documents.
+
+## Installation
+
+```
+java -jar embulk.jar gem intall embulk-plugin-input-roo-excel
+```
+
+You also need roo gem for read xlsx documents. If those package not install automatically, install too.
+
+```
+java -jar ~/embulk.jar gem install roo
+Fetching: ruby-ole-1.2.11.8.gem (100%)
+Successfully installed ruby-ole-1.2.11.8
+Fetching: spreadsheet-1.0.1.gem (100%)
+Successfully installed spreadsheet-1.0.1
+Fetching: nokogiri-1.6.6.2-java.gem (100%)
+Successfully installed nokogiri-1.6.6.2-java
+Fetching: rubyzip-1.1.7.gem (100%)
+Successfully installed rubyzip-1.1.7
+Fetching: roo-1.13.2.gem (100%)
+Successfully installed roo-1.13.2
+5 gems installed
+```
+
+You can check package list.
+
+```
+java -jar ~/embulk.jar gem list
+
+*** LOCAL GEMS ***
+
+ffi (1.9.3 java)
+jar-dependencies (0.1.2)
+jruby-openssl (0.9.5 java)
+json (1.8.0 java)
+krypt (0.0.2)
+krypt-core (0.0.2 universal-java)
+krypt-provider-jdk (0.0.2)
+nokogiri (1.6.6.2 java)
+rake (10.1.0)
+rdoc (4.0.1)
+roo (1.13.2)
+ruby-ole (1.2.11.8)
+rubyzip (1.1.7)
+spreadsheet (1.0.1)
+```
+
+## configuration
+
+data example.
+
+
+設定
+
+| key      | description   | default     |
+|----------|-----------------------------|
+| data_pos | data position | 1           |
+| sheet    | sheet name    | first sheet |
+| paths    | file path     | []          |
+| colums   | column names  |             |
+
+column name
+
+| key    | description |
+|--------|-------------|
+| name   | colum name  |
+| type   | type        |
+
+Type is one of the following value.
+
+* booealn
+* long
+* double
+* string
+* timestamp
+
+## Usage
+
+Example data. The sheet name is "The Beatles".
+
+| No | first_name  | first_name | nickname | birthday   |
+|----|-------------|------------|----------|------------|
+| 1  | John        | Lennon     | John    | 1940/10/09 |
+| 2  | Paul        | McCartney  | Paul     | 1942/06/18 |
+| 3  | George      | Harrison   | George   | 1943/02/25 |
+| 4  | Ringo       | Starr      | Ringo    | 1940/07/07 |
+
+configuration file
+
+```
+in:
+  type: roo_excel
+  sheet: "The Beatles"
+  data_pos: 2
+  paths: ["/path/to/beatles"]
+  columns:
+    - { name: no, type: long }
+    - { name: first_name, type: string }
+    - { name: last_name,  type: string }
+    - { name: nick_name,  type: string }
+    - { name: birthday,   type:timestamp, format:"%Y/%m/%d" }
+out:
+  type: stdout
+```
+
+## execution
+
+```
+java -jar embulk.jar preview config.yml
+java -jar embulk.jar run config.yml
+```
+
+## Contributing
+
+1. Fork it ( https://github.com/hiroyuki-sato/embulk-plugin-input-roo-excel/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+require "bundler/gem_tasks"
+

data/embulk-plugin-input-roo-excel.gemspec

@@ -0,0 +1,23 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |spec|
+  spec.name          = "embulk-plugin-input-roo-excel"
+  spec.version       = "0.1.0"
+  spec.authors       = ["Hiroyuki Sato"]
+  spec.email         = ["hiroysato@gmail.com"]
+  spec.summary       = %q{Embulk input plugin to read xlsx files}
+  spec.description   = %q{Embulk input plugin to read xlsx files}
+  spec.homepage      = "https://github.com/hiroyuki-sato/embulk-plugin-input-roo-excel"
+  spec.license       = "Apache 2.0"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.7"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "roo"
+end

data/lib/embulk/input_roo_excel.rb

@@ -0,0 +1,116 @@
+require 'roo'
+require 'time'
+
+module Embulk
+  module Plugin
+
+    class InputRooExcel < InputPlugin
+      # input plugin file name must be: embulk/input_<name>.rb
+      Plugin.register_input('roo_excel', self)
+
+      def self.transaction(config, &control)
+
+        task = {
+          'columns'  => config.param('columns', :array, default: []),
+          'done'     => config.param('done', :array, default: []),
+          'sheet'    => config.param('sheet', :string, default: nil),
+          'data_pos' => config.param('data_pos', :integer, default: 1),
+        }
+        task['files'] = config.param('paths', :array, default: []).map{ |path|
+          next [] unless Dir.exists?(path)
+          Dir.entries(path).sort.select { |entry| entry.match(/\.xlsx\Z/) }.map{ |entry|
+              File.join(path,entry)
+          }
+         }.flatten
+
+        files = task['files'] - task['done']
+        if files.empty?
+          raise "no valid xlsx file found"
+        end
+
+        columns = []
+        task['columns'].each_with_index do |c,i|
+          columns << Column.new(i, c['name'], c['type'].to_sym)
+        end
+
+        resume(task, columns, files.length, &control)
+      end
+
+      def self.resume(task, columns, count, &control)
+        puts "InputRooExcel input started."
+        commit_reports = yield(task, columns, count)
+        puts "InputRooExcel input finished. Commit reports = #{commit_reports.to_json}"
+
+        next_config_diff = {}
+        return next_config_diff
+      end
+
+      def initialize(task, schema, index, page_builder)
+        super
+        @file = task['files'][index]
+      end
+
+      def run
+        puts "InputRooExcel input thread #{@index}..."
+
+        columns = @task['columns']
+        ncol = columns.size
+        data_pos = @task['data_pos']
+
+        sheet = @task['sheet']
+        xlsx = Roo::Excelx.new(@file)
+        if( sheet )
+          xlsx.default_sheet = sheet
+        else
+          xlsx.default_sheet = xlsx.sheets.first
+        end
+
+        data_pos.upto(xlsx.last_row) do |row|
+          data = []
+          1.upto(ncol) do |col|
+            column = columns[col-1]
+            data << convert_cell(column,xlsx,row,col)
+          end
+          @page_builder.add(data)
+        end
+
+        @page_builder.finish  # don't forget to call finish :-)
+
+        commit_report = {}
+        return commit_report
+      end
+
+      # MEMO roo celltype
+      # returns the type of a cell: * :float * :string, * :date * :percentage * :formula * :time * :datetime.
+      #
+      def convert_cell(column,xlsx,nrow,ncol)
+        d = xlsx.cell(nrow,ncol)
+        type = column['type'] || 'string'
+
+        case type
+        when 'long'
+          d.to_i
+        when 'double'
+          d.to_f
+        when 'string'
+          d.to_s
+        when 'timestamp'
+          convert_time(d)
+        else # TODO
+          d.to_s
+        end
+      end
+      def convert_time(t)
+        if( t.kind_of?(Date) or t.kind_of?(DateTime) )
+          t.to_time
+        elsif( t.kind_of?(Time) )
+          t
+        elsif( t.kind_of?(String) )
+          Time.parse(t)
+        else
+          raise ArgumentError,"Can't convert time:#{t}"
+        end
+      end
+    end # InputRooExcel
+  end # Plugin
+end # Embulk

metadata

@@ -0,0 +1,95 @@
+--- !ruby/object:Gem::Specification
+name: embulk-plugin-input-roo-excel
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Hiroyuki Sato
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-02-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.7'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: roo
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Embulk input plugin to read xlsx files
+email:
+- hiroysato@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- Gemfile
+- LICENSE.txt
+- README.ja.md
+- README.md
+- Rakefile
+- embulk-plugin-input-roo-excel.gemspec
+- lib/embulk/input_roo_excel.rb
+homepage: https://github.com/hiroyuki-sato/embulk-plugin-input-roo-excel
+licenses:
+- Apache 2.0
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: Embulk input plugin to read xlsx files
+test_files: []
+has_rdoc: