embulk 0.7.0-java

data/embulk-docs/src/release/release-0.4.2.rst

@@ -0,0 +1,18 @@
+Release 0.4.2
+==================================
+
+CLI
+------------------
+
+* Removed support for setting ``-`` to ``-o, --output PATH`` option
+
+General Changes
+------------------
+
+* Fixed guess plugin paths (@takei-yuya++)
+* Fixed syntax error in generated gemspec file
+* Fixed syntax error in generated generated ruby filter and input plugins (@takei-yuya++, also reported by hiroyuki-sato)
+
+Release Date
+------------------
+2015-02-16

data/embulk-docs/src/release/release-0.4.3.rst

@@ -0,0 +1,34 @@
+Release 0.4.3
+==================================
+
+CLI
+------------------
+
+* All subcommands show current time with timezone and embulk's version number at the beginning.
+
+Plugin API Changes
+------------------
+
+* ``Thread.currentThread().getContextClassLoader()`` no longer returns JRuby's classloader. It returns null so that dependent libraries fallback to appropriate ``this.getClass().getContextClassLoader()`` call.
+
+Built-in plugins
+------------------
+
+* ``guess/csv`` guesses ``escape`` and ``null_string`` options.
+* Fixed ``guess/csv`` fails if the csv file includes a timestamp value with timezone (@kinyuka++).
+* Fixed memory leak at ``output/file`` (@akirakw++).
+* Fixed ``input/file`` loads unnecessary files when it lists files from ``.``.
+
+General Changes
+------------------
+
+* embulk-cli artifact is no longer released.
+* embulk-standards artifact doesn't directly depend on dependencies of embulk-core.
+* Updated the build script.
+
+  * ``bintrayUpload`` task uploads embulk-<version>.jar.
+  * ``release`` task actually releases gem to RubyGems and publishes jar files to Bintray.
+
+Release Date
+------------------
+2015-02-17

data/embulk-docs/src/release/release-0.4.4.rst

@@ -0,0 +1,39 @@
+Release 0.4.4
+==================================
+
+CLI
+------------------
+
+* Supports ``-l, --log-level LEVEL`` option to specify log level. (@ykubota++)
+
+Plugin API Changes
+------------------
+
+* Added ``Exec.isPreview()`` method which returns true if the transaction is running in in preview mode.
+* Added ``util.OutputStreamFileOutput`` class.
+* ``util.FileOutputOutputStream`` requires ``CloseMode`` at the constructor.
+* When ``Embulk::DataSource#prop`` returns a String, Array, or Hash value, it returns a copy of the value.
+* When ``Embulk::DataSource#prop`` returns a Hash value, it returns ``DataSource`` instance.
+
+Plugin SPI
+------------------
+
+* Changed local variable name ``processorIndex`` to ``taskIndex``.
+* Changed local variable name ``processorCount`` to ``taskCount``.
+
+Built-in plugins
+------------------
+
+* Added ``gzip`` encoder plugin.
+
+General Changes
+------------------
+
+* ``preview`` runs filter plugins in addition to input plugins.
+* Fixed a problem of ``guess`` where guess plugins ignore formatter properties (such as ``charset``) guessed by other guess plugins (@hiroyuki-sato++).
+* Log messages include "transaction", "resume", "cleanup", or task index at the header.
+* Timestamp in log messages include time zone.
+
+Release Date
+------------------
+2015-02-18

data/embulk-docs/src/release/release-0.4.5.rst

@@ -0,0 +1,24 @@
+Release 0.4.5
+==================================
+
+CLI
+------------------
+
+* Fixed a problem where ``embulk gem install foo --version x.y.z`` shows embulk's version.
+
+Plugin SPI
+------------------
+
+* Added ParserPlugin SPI for JRuby. You can write parser plugins in Ruby.
+* Added FormatterPlugin SPI for JRuby. You can write formatter plugins in Ruby.
+
+
+Plugin API Changes
+------------------
+
+* Added ``Embulk::FileInput`` API used mainly by parser plugins.
+* Added ``Embulk::FileOutput`` API used mainly by formatter plugins.
+
+Release Date
+------------------
+2015-02-19

data/embulk-docs/src/release/release-0.4.6.rst

@@ -0,0 +1,30 @@
+Release 0.4.6
+==================================
+
+CLI
+------------------
+
+* Updated installation script to install ``embulk`` binary to ~/.embulk/bin directory.
+* Updated boot script to set ``-XX:+TieredCompilation`` and ``-XX:TieredStopAtLevel=1`` JVM options excepting ``run`` mode.
+  * Those change improves startup time of embulk command 1.5 ~ 2x faster.
+* Updated boot script to set ``-XX:+UseConcMarkSweepGC`` JVM option to ``run`` mode.
+* Usage message shows embulk version.
+
+Built-in plugins
+------------------
+
+* ``input/file`` plugin sets ``last_path`` option to the next configuration.
+  * This enables scheduled execution where next execution loads files created after the last execution.
+* ``guess/csv`` fixed guess result when the ordr of dates is not year, month day.
+* ``guess/csv`` guesses dd/mm/yyyy time format if the input data includes at least one day larger than 12.
+* ``guess/csv`` guesses RFC 2822 date and time formats
+* ``guess/csv`` supports "." as the delimiter of year, month or day.
+
+General Changes
+------------------
+
+* embulk command raises a special message if HOME environment variable is not set.
+
+Release Date
+------------------
+2015-02-20

data/embulk-docs/src/release/release-0.4.7.rst

@@ -0,0 +1,16 @@
+Release 0.4.7
+==================================
+
+CLI
+------------------
+
+* Added ``-G, --vertical`` option to ``preview`` subcommand. It shows records in vertical style like MySQL's ``\G`` or PostgreSQL's ``\x`` (@yaggytter++).
+
+General Changes
+------------------
+
+* Fixed a problem on Windows where classloader raises (@hito4_t++)
+
+Release Date
+------------------
+2015-02-23

data/embulk-docs/src/release/release-0.4.8.rst

@@ -0,0 +1,15 @@
+Release 0.4.8
+==================================
+
+General Changes
+------------------
+
+* ``spi.time.TimestampParser`` supports microseconds.
+* ``spi.time.TimestampFormatter`` supports milliseconds, microseconds and nanoseconds.
+* Fixed a bug at ``spi.time.TimestampParser`` that returns wrong timestamp if format string is ``%s`` which means epoch seconds (@hiroyuki-sato++)
+* ``embulk`` command installed by ``gem install embulk`` sets JVM optimization flags. This behavior is same with embulk.jar updated since 0.4.6.
+* ``preview`` subcommand with ``-G, --vertical`` option uses CRLF on Windows
+
+Release Date
+------------------
+2015-02-24

data/embulk-docs/src/release/release-0.4.9.rst

@@ -0,0 +1,23 @@
+Release 0.4.9
+==================================
+
+CLI
+------------------
+
+* Fixed execution script for Windows
+
+Built-in plugins
+------------------
+
+* Fixed a problem that ``encoder/gzip`` plugin didn't flush last buffer in memory.
+* ``encoder/gzip`` plugin supports ``level`` option (@yaggytter++).
+
+General Changes
+------------------
+
+* Fixed a problem of timestamp parser that ignores milliseconds if format is ``%s.%N``.
+* Plugin template generator generates shorter description to fit in a table at the new plugin list page.
+
+Release Date
+------------------
+2015-02-25

data/embulk-docs/src/release/release-0.5.0.rst

@@ -0,0 +1,89 @@
+Release 0.5.0
+==================================
+
+New Guess Plugin Architecture
+------------------
+
+Embulk v0.5.0 supports dynamically loadable guess plugins.
+
+* For parser and decoder plugins:
+
+  * CLI subcommand ``guess`` accepts new ``-g, --guess NAMES`` argument to load parser and decoder guess plugins.
+
+  * Plugin template generator creates stub code of guess plugin for parser and decoder plugins.
+
+* For input plugins:
+
+  * Added ``Embulk::InputPlugin.guess(config)`` and ``spi.InputPlugin#guess(config)`` methods for Ruby and Java plugins.
+
+  * ``guess`` subcommand executes the new guess method and takes the return value of the method.
+
+Plugins can use new ``Guess::SchemaGuess`` utility class for ease of implementation.
+
+For example, if you write a parser plugin named ``myparser``, you can use this configuration file first:
+
+.. code-block:: yaml
+
+    in:
+      type: file
+      path_prefix: path/to/myfiles
+    out:
+      type: stdout
+
+The ``embulk guess`` command with ``-g myparser`` argument calls the guess plugin bundled in the plugin:
+
+.. code-block:: console
+
+    $ embulk gem install embulk-parser-myparser
+    $ embulk guess config.yml -o guessed.yml -g myparser
+
+On the other hand, if the plugin type is input, you don't need additional command-line arguments. For example, if the input plugin name is ``myinput``, you can use this configuration file:
+
+.. code-block:: yaml
+
+    in:
+      type: myinput
+    out:
+      type: stdout
+
+The ``embulk guess`` command finds the ``InputPlugin.guess`` of the input plugin and calls it:
+
+.. code-block:: console
+
+    $ embulk gem install embulk-input-myinput
+    $ embulk guess config.yml -o guessed.yml
+
+Plugin API
+------------------
+
+* Added ``Guess::SchemaGuess`` class. This utility class inputs array of hash objects or array of array objects and returns schema.
+
+
+Plugin SPI
+------------------
+
+* Added ``Embulk::InputPlugin.guess(config)`` method for Ruby input plugins.
+
+  * Backward compatibility: existent plugins don't have to implement the method. The default behavior is raising ``NotImplementedError``.
+
+* Added ``spi.InputPlugin#guess(ConfigSource config)`` method for Java input plugins.
+
+  * Backward compatibility: existent plugins don't have to implement the method. Mehtod linkage errors are handled at embulk-core. The default behavior is raising ``UnsupportedOperationException``.
+
+Built-in plugins
+------------------
+
+* ``csv`` parser plugin implements ``max_quoted_size_limit`` option. Default value is 131072 (128KB).
+
+  * This option is useful when a column value includes a quote character accidentally. If the plugin detects those values, it skips the line and continues parsing from the next line.
+
+General Changes
+------------------
+
+* ``spi.util.FileInputInputStream#skip`` method never returns -1 to follow the Java API standard (@hata++).
+* Plugin template generator creates appropriate "Overview" section in README.md file depending on the plugin type.
+
+
+Release Date
+------------------
+2015-03-02

data/embulk-docs/src/release/release-0.5.1.rst

@@ -0,0 +1,13 @@
+Release 0.5.1
+==================================
+
+General Changes
+------------------
+
+* Fixed a problem where embulk wrongly shows "unknown plugin" when a plugin is found but caused LoadError.
+* Preview, guess and run subcommands show causes of the exception and suppressed exceptions in addition to the stacktrace.
+* PluginManager includes messages of exceptions from all plugin sources.
+
+Release Date
+------------------
+2015-03-04

data/embulk-docs/src/release/release-0.5.2.rst

@@ -0,0 +1,30 @@
+Release 0.5.2
+==================================
+
+Built-in plugins
+------------------
+
+* ``parser-csv`` plugin supports ``skip_header_lines`` parameter to skip first some lines.
+
+  * ``header_line`` parameter is obsoleted. Although the parameter still works for backward compatibility, setting both ``header_line`` and ``skip_header_lines`` becomes configuration error.
+
+* ``guess-csv`` plugin guesses first ignorable lines and sets ``skip_header_lines`` parameter automatically.
+
+* ``guess-csv`` plugin guesses quoted column names correctly.
+
+* ``formatter-csv`` pugin supports ``delimiter`` parameter (@hiroyuki-sato++).
+
+* ``output-stdout`` fixed warning messages due to double-release of buffers.
+
+
+General Changes
+------------------
+
+* Improved error message when double-release of a ``spi.Buffer`` is detected.
+* Fixed ``preview`` when a filter plugin changes schema (@llibra++).
+* Fixed infinite loop at ``Embulk::FileOutput#flush`` (@goronao++). It happened if a formatter plugin written in Ruby writes more than 32KB of data.
+
+
+Release Date
+------------------
+2015-03-11

data/embulk-docs/src/release/release-0.5.3.rst

@@ -0,0 +1,22 @@
+Release 0.5.3
+==================================
+
+Built-in plugins
+------------------
+
+* ``guess`` guesses boolean types (@hata++)
+
+
+General Changes
+------------------
+
+* ``guess`` stops reading the file after first 32KB and ignores remaining data. Execution time of ``guess`` improves significantly especially when the first file is large.
+
+* ``guess`` ignores decoders when it reads sample data. This change fixes the problem where guess fails if config file includes ``decoder-gzip`` (@hata++).
+
+* Releases ``embulk-core-VERSION-tests.jar`` in addition to ``embulk-core-VERSION-sources.jar``.
+
+
+Release Date
+------------------
+2015-03-17

data/embulk-docs/src/release/release-0.5.4.rst

@@ -0,0 +1,24 @@
+Release 0.5.4
+==================================
+
+Built-in plugins
+------------------
+
+* ``parser-csv`` supports ``allow_optional_columns`` option. With this option set to ``true``, the parser sets null to insufficient columns rather than skipping the entire row (@kamatama41++)
+
+* Fixed exception handling of ``parser-csv`` so that the transaction properly fails with underlaying exceptions such as IOException
+
+
+General Changes
+------------------
+
+* Increased buffer size from 256 bytes to 32 KB. This improves performance significantly. (@hito4t++)
+
+* If plugin type is null, suggest to use ``{type: "null"}`` (@hito4t++)
+
+* Embulk logo is available! See the orca: https://github.com/embulk/embulk/issues/12
+
+
+Release Date
+------------------
+2015-03-23

data/embulk-docs/src/release/release-0.5.5.rst

@@ -0,0 +1,18 @@
+Release 0.5.5
+==================================
+
+Plugin API
+------------------
+
+* Added ``spi.Exec.getModelManager()`` method.
+
+General Changes
+------------------
+
+* ``guess-csv`` does not add config parameters if ``type`` is already set by other guess plugins (@shun0102++)
+* Fixed double double-quoting in build.gradle file generated by the plugin template generator
+
+
+Release Date
+------------------
+2015-04-07

data/embulk-docs/src/release/release-0.6.0.rst

@@ -0,0 +1,34 @@
+Release 0.6.0
+==================================
+
+Executor Plugin Mechanism
+------------------
+
+Now executor of Embulk is fully extensible using plugins. Executor plugins get input, filter and output plugins from the Embulk framework and runs them using multiple threads, processes, or servers. While input, filter and output plugins are response for data processing, executor plugins are responsible for scheduling the processing tasks and managing parallelism for performance.
+
+The built-in executor plugin is ``LocalExecutorPlugin`` that runs tasks using multiple threads. It has a shared thread pool and schedules tasks at most ``(number of available CPU cores) * 2`` tasks in parallel. Number of threads is configurable using ``max_threads`` system parameter.
+
+Another available executor is `embulk-executor-mapreduce <https://github.com/embulk/embulk-executor-mapreduce>`_ plugin. This executor plugin runs tasks on Hadoop, a distributed computing environment. It is suitable for processing TBs of data. An unique functionality is that it supports partitioning data by a certain column before passing them to output plugins. An example use case is that the MapReduce executor partitions data by time so that files on destination storage are partitioned for each day.
+
+Plugin API
+------------------
+
+* ``exec.LocalExecutor`` class is separated into ``exec.BulkLoader`` class for interface definition and ``exec.LocalExecutorPlugin`` for implementation. If you're application is Embulk through ``LocalExecutor`` class, you need to replace it with ``BulkLoader``.
+* ``spi.ExecAction#run`` can throw ``Exception`` and ``Exec.doWith(ExecSession, ExecAction<T>)`` throws ``ExecutionException``.
+* ``spi.Buffer`` implements ``#equals`` and ``hashCode`` methods.
+
+Plugin SPI
+------------------
+
+* Added ``spi.ExecutorPlugin`` interface.
+* Added ``Embulk::ExecutorPlugin`` class.
+
+General Changes
+------------------
+
+* If there are no input tasks, the transaction is committed successfully rather than making it failed.
+
+
+Release Date
+------------------
+2015-04-07