embulk 0.7.0-java

data/embulk-docs/src/recipe.rst

@@ -0,0 +1,8 @@
+Recipes
+==================================
+
+.. toctree::
+    :maxdepth: 1
+
+    recipe/scheduled-csv-load-to-elasticsearch-kibana4
+

data/embulk-docs/src/recipe/scheduled-csv-load-to-elasticsearch-kibana4.rst

@@ -0,0 +1,153 @@
+Scheduled bulk data loading to Elasticsearch + Kibana 4 from CSV files
+==================================
+
+.. contents::
+   :local:
+   :depth: 2
+
+This article shows how to:
+
+* Bulk load CSV files to Elasticsearch.
+* Visualize the data with Kibana interactively.
+* Schedule the data loading every hour using cron.
+
+This guide assumes you are using Ubuntu 12.04 Precise or Mac OS X.
+
+Setup Elasticsearch and Kibana 4
+------------------
+
+Step 1. Download and start Elasticsearch.
+~~~~~~~~~~~~~~~~~~
+
+You can find releases from the `Elasticsearch website <http://www.elasticsearch.org/download/>`_.
+For the smallest setup, you can unzip the package and run `./bin/elasticsearch` command:
+
+.. code-block:: console
+
+    $ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.zip
+    $ unzip elasticsearch-1.4.4.zip
+    $ cd elasticsearch-1.4.4
+    $ ./bin/elasticsearch
+
+Step 2. Download and unzip Kibana:
+~~~~~~~~~~~~~~~~~~
+
+You can find releases from the `Kibana website <http://www.elasticsearch.org/overview/kibana/installation/>`_. Open a new console and run following commands:
+
+.. code-block:: console
+
+    $ wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-linux-x64.tar.gz
+    $ tar zxvf kibana-4.0.0-linux-x64.tar.gz
+    $ cd kibana-4.0.0-linux-x64
+    $ ./bin/kibana
+
+Note: If you're using Mac OS X, https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-darwin-x64.tar.gz is the URL to download.
+
+Now Elasticsearch and Kibana started. Open http://localhost:5601/ using your browser to see the Kibana's graphical interface.
+
+
+Setup Embulk
+------------------
+
+Step 1. Download Embulk binary:
+~~~~~~~~~~~~~~~~~~
+
+You can find the latest embulk binary from the `releases <https://bintray.com/embulk/maven/embulk/view#files>`_. Because Embulk is a single executable binary, you can simply download it to /usr/local/bin directory and set executable flag as following:
+
+.. code-block:: console
+
+    $ sudo wget http://dl.embulk.org/embulk-latest.jar -O /usr/local/bin/embulk
+    $ sudo chmod +x /usr/local/bin/embulk
+
+Step 2. Install Elasticsearch plugin
+~~~~~~~~~~~~~~~~~~
+
+You also need Elasticsearch plugin for Embulk. You can install the plugin with this command:
+
+.. code-block:: console
+
+    $ embulk gem install embulk-output-elasticsearch
+
+Embulk includes CSV file reader in itself. Now everything is ready to use.
+
+Loading a CSV file
+------------------
+
+Assuming you have a CSV files at ``./mydata/csv/`` directory. If you don't have CSV files, you can create ones using ``embulk example ./mydata`` command.
+
+Create this configuration file and save as ``config.yml``:
+
+.. code-block:: yaml
+
+    in:
+      type: file
+      path_prefix: ./mydata/csv/
+    out:
+      type: elasticsearch
+      index: embulk
+      index_type: embulk
+      nodes:
+        - host: localhost
+
+In fact, this configuration lacks some important information. However, embulk guesses the other information. So, next step is to order embulk to guess them:
+
+.. code-block:: console
+
+    $ embulk guess config.yml -o config-complete.yml
+
+The generated config-complete.yml file should include complete information as following:
+
+.. code-block:: yaml
+
+    in:
+      type: file
+      path_prefix: ./mydata/csv/
+      decoders:
+      - {type: gzip}
+      parser:
+        charset: UTF-8
+        newline: CRLF
+        type: csv
+        delimiter: ','
+        quote: '"'
+        escape: ''
+        null_string: 'NULL'
+        skip_header_lines: 1
+        columns:
+        - {name: id, type: long}
+        - {name: account, type: long}
+        - {name: time, type: timestamp, format: '%Y-%m-%d %H:%M:%S'}
+        - {name: purchase, type: timestamp, format: '%Y%m%d'}
+        - {name: comment, type: string}
+    out:
+      type: elasticsearch
+      index: embulk
+      index_type: embulk
+      nodes:
+      - {host: localhost}
+
+Now, you can run the bulk loading:
+
+.. code-block:: console
+
+    $ embulk run config-complete.yml -o next-config.yml
+
+Scheduling loading by cron
+------------------
+
+At the last step, you ran embulk command with ``-o next-config.yml`` file. The ``next-config.yml`` file should include a parameter named ``last_path``:
+
+.. code-block:: yaml
+
+    last_path: mydata/csv/sample_01.csv.gz
+
+With this configuration, embulk loads the files newer than this file in alphabetical order.
+
+For example, if you create ``./mydata/csv/sample_02.csv.gz`` file, embulk skips ``sample_01.csv.gz`` file and loads ``sample_02.csv.gz`` only next time. And the next next-config.yml file has ``last_path: mydata/csv/sample_02.csv.gz`` for the next next execution.
+
+So, if you want to loads newly created files every day, you can setup this cron schedule:
+
+.. code-block:: cron
+
+    0 * * * * embulk run /path/to/next-config.yml -o /path/to/next-config.yml
+

data/embulk-docs/src/release.rst

@@ -0,0 +1,57 @@
+Release Notes
+==================================
+
+.. toctree::
+    :maxdepth: 1
+
+    release/release-0.7.0
+    release/release-0.6.25
+    release/release-0.6.24
+    release/release-0.6.23
+    release/release-0.6.22
+    release/release-0.6.21
+    release/release-0.6.20
+    release/release-0.6.19
+    release/release-0.6.18
+    release/release-0.6.17
+    release/release-0.6.16
+    release/release-0.6.15
+    release/release-0.6.14
+    release/release-0.6.13
+    release/release-0.6.12
+    release/release-0.6.11
+    release/release-0.6.10
+    release/release-0.6.9
+    release/release-0.6.8
+    release/release-0.6.7
+    release/release-0.6.6
+    release/release-0.6.5
+    release/release-0.6.4
+    release/release-0.6.3
+    release/release-0.6.2
+    release/release-0.6.1
+    release/release-0.6.0
+    release/release-0.5.5
+    release/release-0.5.4
+    release/release-0.5.3
+    release/release-0.5.2
+    release/release-0.5.1
+    release/release-0.5.0
+    release/release-0.4.10
+    release/release-0.4.9
+    release/release-0.4.8
+    release/release-0.4.7
+    release/release-0.4.6
+    release/release-0.4.5
+    release/release-0.4.4
+    release/release-0.4.3
+    release/release-0.4.2
+    release/release-0.4.1
+    release/release-0.4.0
+    release/release-0.3.2
+    release/release-0.3.1
+    release/release-0.3.0
+    release/release-0.2.1
+    release/release-0.2.0
+    release/release-0.1.0
+

data/embulk-docs/src/release/release-0.1.0.rst

@@ -0,0 +1,8 @@
+Release 0.1.0
+==================================
+
+The first release.
+
+Release Date
+---------------
+2015-01-26

data/embulk-docs/src/release/release-0.2.0.rst

@@ -0,0 +1,16 @@
+Release 0.2.0
+==================================
+
+Plugin API
+------------------
+
+* Changed JRuby InputPlugin API to use #run instead of .run
+
+CLI
+------------------
+
+* Fixed ``example`` subcommand to quote file path for Windows environment
+
+Release Date
+------------------
+2015-01-26

data/embulk-docs/src/release/release-0.2.1.rst

@@ -0,0 +1,19 @@
+Release 0.2.1
+==================================
+
+Plugin Utility
+------------------
+
+* Fixed ``spi.util.LineEncoder#finish`` to flush all remaining buffer (reported by @aibou)
+* ``Embulk::PageBuilder#add`` accepts nil value
+
+General Changes
+------------------
+
+* Fixed NextConfig to be merged to in: or out: rather than the top-level
+  (reported by enukane) [#41]
+* ./bin/embulk shows warns to run `rake` if ./classpath doesn't exist
+
+Release Date
+------------------
+2015-01-29

data/embulk-docs/src/release/release-0.3.0.rst

@@ -0,0 +1,34 @@
+Release 0.3.0
+==================================
+
+Resume Feature
+------------------
+
+* Added resume functionality.
+
+Filter Feature
+------------------
+
+* Added new filter plugin type.
+
+Plugin SPI
+------------------
+
+* ``spi.InputPlugin`` and ``spi.OutputPlugin`` need to implement ``resume`` and ``cleanup`` methods.
+* Added ``spi.FilterInputPlugin`` Java API.
+* Added ``Embulk::FilterInputPlugin`` JRuby API.
+
+CLI
+------------------
+
+* ``run`` subcommand supports ``-r, --resume-state PATH`` option.
+
+General Changes
+------------------
+
+* Added gradle-versions-plugin to build.gradle (@seratch++)
+* Fixed broken dependencies at build.gradle (@thagikura++)
+
+Release Date
+------------------
+2015-02-03

data/embulk-docs/src/release/release-0.3.1.rst

@@ -0,0 +1,11 @@
+Release 0.3.1
+==================================
+
+CLI
+------------------
+
+* Added ``-C, --classpath option`` to add java classpath
+
+Release Date
+------------------
+2015-02-04

data/embulk-docs/src/release/release-0.3.2.rst

@@ -0,0 +1,15 @@
+Release 0.3.2
+==================================
+
+General Changes
+------------------
+
+* Fixed a problem where ruby input plugins can't use timestamp type (reported
+  by @shun0102)
+* ``Embulk::Page`` includes Enumerable to include map, each_with_index, and other
+   a lot of convenient methods (@niku++)
+* Fixed ``spi.time.TimestampType::DEFAULT_FORMAT`` to use ':' as the separator of times
+
+Release Date
+------------------
+2015-02-04

data/embulk-docs/src/release/release-0.4.0.rst

@@ -0,0 +1,74 @@
+Release 0.4.0
+==================================
+
+Plugin Template Generator
+------------------
+
+A new CLI subcommand ``new`` generate a plugin template.
+
+For example, ``new ruby-output <name>`` generates a template of output plugin written in Ruby::
+
+    $ embulk new ruby-output foo
+    Creating embulk-output-foo/
+      Creating embulk-output-foo/README.md
+      Creating embulk-output-foo/LICENSE.txt
+      Creating embulk-output-foo/.gitignore
+      Creating embulk-output-foo/Rakefile
+      Creating embulk-output-foo/Gemfile
+      Creating embulk-output-foo/embulk-output-foo.gemspec
+      Creating embulk-output-foo/lib/embulk/output/foo.rb
+
+The template generate supports most of plugin types. ruby-file-input and output, ruby-parser and formatter, ruby-decoder and encoder are not available yet because plugin SPI is not implemented yet.
+
+
+Java Plugin Loader
+------------------
+
+Now, we can write plugins in Java.
+
+The plugin loader creates a dedicated classloader for each plugin so that plugins can depend on different versions of libraries. To develop a Java plugin, you can use use ``new java-<category> <name>`` command which generates a template. The generated directory has ``build.gradle`` file so that you can run ``./gradlew gem`` command to build a gem package.
+
+
+Plugin SPI
+------------------
+
+* **IMPORTANT**: Changed plugin path name from ``lib/embulk/<caetgory>_<name>.rb`` to ``lib/embulk/<category>/<name>.rb``.
+* Added ``@ConfigInject`` annotation to set resources injected by Guice to a Task.
+* Removed support for ``@JacksonInject`` annotation on a Task. ``@ConfigInject`` is the drop-in replacement.
+* Changed class name ``spi.NextConfig`` to ``spi.ConfigDiff``.
+
+
+
+Built-in plugins
+------------------
+
+* ``csv`` parser plugin and its guess plugin assume "true", "yes", "Y" and "on" as a boolean true value.
+
+* ``file`` input plugin changed parameters.
+
+  * Added ``path_prefix`` parameter (string, required).
+  * Added ``last_path`` parameter (string, optional) which skips files older than or same with the specified file path.
+  * Removed ``paths`` parameter.
+
+* ``file`` output plugin changed parameters.
+
+  * Added ``sequence_format`` parameter.
+  * Removed ``directory`` parameters.
+  * Removed ``file_name`` parameters.
+  * The actual file name is ``${path_prefix}${sequence_format % [processorIndex, sequenceNumber}${ext_name}``.
+
+* Removed ``s3`` file input plugin. It is now embulk-input-s3 gem.
+
+
+General Changes
+------------------
+
+* PluginManager reports the cause of exceptions when it failed to load a plugin.
+* Removed pom.xml.
+* Set locale = 'en_US' when gradle builds javadoc.
+* Added documentation scripts using Sphinx, YARD and JavaDoc.
+
+
+Release Date
+------------------
+2015-02-15

data/embulk-docs/src/release/release-0.4.1.rst

@@ -0,0 +1,18 @@
+Release 0.4.1
+==================================
+
+CLI
+------------------
+
+* ``--version`` exists with code=0 (@yuskesh++)
+
+General Changes
+------------------
+
+* Fixed a build script problem where it uploads corrupted CLI jar file
+* Updated document build script to not depend on externally installed "yardoc" command
+
+
+Release Date
+------------------
+2015-02-16

data/embulk-docs/src/release/release-0.4.10.rst

@@ -0,0 +1,17 @@
+Release 0.4.10
+==================================
+
+Plugin API Changes
+------------------
+
+* Classloader of Java-based plugins search classes from the embulk's classloader by default. This solves mismatches of the same class loaded by 2 classloaders.
+* Added ``spi.PluginClassLoader.addPath(Path)`` method so that Java-based plugins can load jar files dynamically.
+
+General Changes
+------------------
+
+* Rakefile generated by plugin template generator has default task aliased to ``build`` task (@takei-yuya++)
+
+Release Date
+------------------
+2015-02-26