embulk 0.7.0-java

data/embulk-docs/src/release/release-0.6.4.rst

@@ -0,0 +1,13 @@
+Release 0.6.4
+==================================
+
+Built-in plugins
+------------------
+
+* ``guess-csv`` keeps existent configurations such as ``null_string`` if they are already set.
+* ``guess-csv`` skips line if a CSV file is broken rather than falling back to ad-hoc implementation which can't deal with quoted values.
+* Fixed a problem where ``guess-csv`` can't handle quoted values (@y-ken++)
+
+Release Date
+------------------
+2015-04-21

data/embulk-docs/src/release/release-0.6.5.rst

@@ -0,0 +1,17 @@
+Release 0.6.5
+==================================
+
+Built-in plugins
+------------------
+
+* ``guess-csv`` allows a value looks like a boolean as a column name (@y-ken++)
+* Fixed a problem where ``guess-csv`` can't handle character encodings other than UTF-8 character encodings (@y-ken++)
+
+General Changes
+------------------
+
+* Fixed 'warning: already initialized constant' warning messages.
+
+Release Date
+------------------
+2015-04-21

data/embulk-docs/src/release/release-0.6.6.rst

@@ -0,0 +1,17 @@
+Release 0.6.6
+==================================
+
+General Changes
+------------------
+
+* Fixed a problem that timestamp format guess code can't guess format if the string includes day of 30 and 31.
+* ``guess`` and ``preview`` throw NoSampleException with appropriate error message if there are no input tasks.
+
+Plugin API
+------------------
+
+* ``spi.util.InputStreamFileInput#close`` closes currently opened InputStream.
+
+Release Date
+------------------
+2015-05-04

data/embulk-docs/src/release/release-0.6.7.rst

@@ -0,0 +1,17 @@
+Release 0.6.7
+==================================
+
+Built-in plugins
+------------------
+
+* ``guess-csv`` guesses "t" and "f" as a boolean value.
+
+General Changes
+------------------
+
+* Added ``selfupdate`` subcommand. This command updates embulk itself to the released latest version.
+* Upgraded library dependency of Guice from 3.0 to 4.0.
+
+Release Date
+------------------
+2015-05-06

data/embulk-docs/src/release/release-0.6.8.rst

@@ -0,0 +1,24 @@
+Release 0.6.8
+==================================
+
+Plugin API
+------------------
+
+* Added utility class ``spi.util.ResumableInputStream``
+* Added utility class ``spi.util.RetryExecutor``
+
+Built-in plugins
+------------------
+
+* ``parser-csv`` rejects rows if one includes too many columns by default. Setting ``allow extra_columns`` option to ``true`` will make the behavior same with before.
+* ``guess-csv`` guesses ``columns`` option every time.
+
+General Changes
+------------------
+
+* Fixed a problem that IntelliJ IDEA causes problem when it imports embulk source code.
+* Fixed a problem that transaction silently succeeds when an exception happens after all taks succeeded.
+
+Release Date
+------------------
+2015-05-12

data/embulk-docs/src/release/release-0.6.9.rst

@@ -0,0 +1,24 @@
+Release 0.6.9
+==================================
+
+Built-in plugins
+------------------
+
+* ``formatter-csv`` supports ``quote``, ``quote_policy``, ``escape``, ``newline_in_field``, and ``null_string`` options (@sakama++)
+
+  * ``quote_policy`` controls how to quote values. It can be either of ``ALL`` (quote all values), ``MINIMAL`` (quote if a value includes delimiter or quote character), or ``NONE`` (never quotes).
+
+  * ``escape`` controls how to escape quote character in a quoted string. The default is ``"`` (``"`` will be ``""``). Some applications may set it to ``\`` (``"`` will be ``\"``)
+
+  * ``null_string`` controls how to write NULL values. The default is ``""`` (empty string). You can use any strings such as ``\N`` or ``#N/A``.
+
+* ``guess-csv`` guesses columns which contain only 0 and 1 in first 32KB as long type rather than boolean type.
+
+General Changes
+------------------
+
+* ``spi.util.LineEncoder`` uses buffered writer. This improves performance of ``formatter-csv`` upto 10%.
+
+Release Date
+------------------
+2015-05-14

data/embulk-docs/src/release/release-0.7.0.rst

@@ -0,0 +1,96 @@
+Release 0.7.0
+==================================
+
+General Changes
+------------------
+
+* Upraded JRuby version to 9.0.0.0. Ruby scripting is compatible with Ruby 2.2 syntax.
+
+* Added ``embulk migrate <plugin directory>`` subcommand. This command modifies plugin code to use the latest Embulk API.
+
+* Enambed Liquid template engine. If configuration file name ends with ``.yml.liquid``, embulk embeds environment variables using Liquid template engine.
+
+* Embulk gem package for JRuby doesn't include JRuby itself any more. Size of a gem package is reduced from 38MB to 7.6MB.
+
+* Embulk gem is also released for CRuby. This enables us to install ``embulk`` command using ``gem install embulk``.
+
+* **IMPORTANT**: ``embulk bundle`` command runs bundler. To create a new plugin bundle, use ``embulk bundle new <directory>`` command. To update gems, use ``embulk bundle`` command at the directory. Instructions are written at generated Gemfile file.
+
+
+Ruby Plugin API
+------------------
+
+* Added experimental ``Embulk.setup`` and ``Embulk::Runner``.
+
+  This enables ruby scripts to run embulk easily. This is also good for test code. For example, you can use this code:
+
+.. code-block:: ruby
+
+    require 'embulk'
+    Embulk.setup
+    Embulk::Runner.run(YAML.load_file("config.yml"))
+
+* ``embulk new`` generates .ruby-version file with jruby-9.0.0.0 for ruby-based plugins.
+
+  This makes plugin development easy as following:
+
+.. code-block:: console
+
+    # 1. Create plugin template
+    $ embulk new ruby-parser awesome
+    $ cd embulk-parser-awesome
+    # or upgrade existent plugin: embulk migrate embulk-parser-awesome
+   
+    # 2. Install dependency gems including embulk itself at vendor/bundle directory
+    $ bundle install --path vendor/bundle
+   
+    # 3. Create an example configuration file
+    $ vi config.yml
+   
+    # 4. You can run embulk without building & installing gem
+    $ bundle exec embulk run config.yml
+
+* Constants defined at ``Embulk::Java`` are deprecated. They're still kept for backward compatibility but will be removed at a future release.
+
+* Added ``Embulk::Java::Config`` and ``Embulk::SPI`` namespaces to access Java classes.
+
+
+Java Plugin API
+------------------
+
+* **IMPORTANT**: Renamed CommitReport class to TaskReport. Binary backward compatibility is kept so that old plugins built with embulk 0.6.x can run with embulk 0.7.0. But this backward compatibility code will be removed at future release.
+
+  To upgrade your plugin code, you can use ``embulk migrate <plugin directory>`` command.
+
+* Upgraded gradle version to 2.6. This version supports ``./gradlew -t <task>`` command that watches changes of files and rebuild continuously.
+
+  This makes plugin development easy as following:
+
+.. code-block:: console
+
+    # 1. Create plugin template
+    $ embulk new java-input awesome
+    $ cd embulk-input-awesome
+    # or upgrade existent plugin: embulk migrate embulk-input-awesome
+   
+    # 2. Build code continously
+    $ ./gradlew -t package
+   
+    # 3. Create an example configuration file
+    $ vi config.yml
+   
+    # 4. Run embulk with -L option
+    $ embulk -L . run config.yml
+
+* Added ``EmbulkEmbed.Bootstrap`` class to build ``EmbulkEmbed`` instance.
+
+* Added ``ConfigLoader.fromJsonString(String)`` and ``ConfigLoader.fromYamlString(String)`` methods.
+
+* Added guess, preview, and run methods at ``EmbulkEmbed`` don't need ExecSession instance any more.
+
+* EmbulkService is now deprecated. Replacement is EmbulkEmbed.
+
+
+Release Date
+------------------
+2015-08-18

data/embulk-standards/build.gradle

@@ -0,0 +1,5 @@
+dependencies {
+    compile project(':embulk-core')
+
+    testCompile project(':embulk-core').sourceSets.test.output
+}

data/embulk-standards/src/main/java/org/embulk/standards/CsvFormatterPlugin.java

@@ -0,0 +1,284 @@
+package org.embulk.standards;
+
+import com.google.common.base.Optional;
+import org.embulk.config.Config;
+import org.embulk.config.ConfigDefault;
+import org.embulk.spi.time.Timestamp;
+import org.embulk.spi.time.TimestampFormatter;
+import org.embulk.config.Task;
+import org.embulk.config.TaskSource;
+import org.embulk.config.ConfigSource;
+import org.embulk.spi.Column;
+import org.embulk.spi.Schema;
+import org.embulk.spi.ColumnVisitor;
+import org.embulk.spi.FormatterPlugin;
+import org.embulk.spi.Page;
+import org.embulk.spi.PageOutput;
+import org.embulk.spi.PageReader;
+import org.embulk.spi.FileOutput;
+import org.embulk.spi.util.LineEncoder;
+import org.embulk.spi.util.Timestamps;
+
+import org.embulk.spi.util.Newline;
+import java.util.Map;
+
+public class CsvFormatterPlugin
+        implements FormatterPlugin
+{
+    public enum QuotePolicy
+    {
+        ALL("ALL"),
+        MINIMAL("MINIMAL"),
+        NONE("NONE");
+
+        private final String string;
+
+        private QuotePolicy(String string)
+        {
+            this.string = string;
+        }
+
+        public String getString()
+        {
+            return string;
+        }
+    }
+
+    public interface PluginTask
+            extends Task, LineEncoder.EncoderTask, TimestampFormatter.Task
+    {
+        @Config("header_line")
+        @ConfigDefault("true")
+        boolean getHeaderLine();
+
+        @Config("delimiter")
+        @ConfigDefault("\",\"")
+        char getDelimiterChar();
+
+        @Config("quote")
+        @ConfigDefault("\"\\\"\"")
+        char getQuoteChar();
+
+        @Config("quote_policy")
+        @ConfigDefault("\"MINIMAL\"")
+        QuotePolicy getQuotePolicy();
+
+        @Config("escape")
+        @ConfigDefault("null")
+        Optional<Character> getEscapeChar();
+
+        @Config("null_string")
+        @ConfigDefault("\"\"")
+        String getNullString();
+
+        @Config("newline_in_field")
+        @ConfigDefault("\"LF\"")
+        Newline getNewlineInField();
+
+        @Config("column_options")
+        @ConfigDefault("{}")
+        Map<String, TimestampColumnOption> getColumnOptions();
+    }
+
+    public interface TimestampColumnOption
+            extends Task, TimestampFormatter.TimestampColumnOption
+    { }
+
+    @Override
+    public void transaction(ConfigSource config, Schema schema,
+            FormatterPlugin.Control control)
+    {
+        PluginTask task = config.loadConfig(PluginTask.class);
+
+        // validate column_options
+        for (String columnName : task.getColumnOptions().keySet()) {
+            schema.lookupColumn(columnName);  // throws SchemaConfigException
+        }
+
+        control.run(task.dump());
+    }
+
+    @Override
+    public PageOutput open(TaskSource taskSource, final Schema schema,
+            FileOutput output)
+    {
+        final PluginTask task = taskSource.loadTask(PluginTask.class);
+        final LineEncoder encoder = new LineEncoder(output, task);
+        final TimestampFormatter[] timestampFormatters = Timestamps.newTimestampColumnFormatters(task, schema, task.getColumnOptions());
+        final char delimiter = task.getDelimiterChar();
+        final QuotePolicy quotePolicy = task.getQuotePolicy();
+        final char quote = task.getQuoteChar() != '\0' ? task.getQuoteChar() : '"';
+        final char escape = task.getEscapeChar().or(quotePolicy == QuotePolicy.NONE ? '\\' : '\"');
+        final String newlineInField = task.getNewlineInField().getString();
+        final String nullString = task.getNullString();
+
+        // create a file
+        encoder.nextFile();
+
+        // write header
+        if (task.getHeaderLine()) {
+            writeHeader(schema, encoder, delimiter, quotePolicy, quote, escape, newlineInField, nullString);
+        }
+
+        return new PageOutput() {
+            private final PageReader pageReader = new PageReader(schema);
+            private final String delimiterString = String.valueOf(delimiter);
+
+            public void add(Page page)
+            {
+                pageReader.setPage(page);
+                while (pageReader.nextRecord()) {
+                    schema.visitColumns(new ColumnVisitor() {
+                        public void booleanColumn(Column column)
+                        {
+                            addDelimiter(column);
+                            if (!pageReader.isNull(column)) {
+                                addValue(Boolean.toString(pageReader.getBoolean(column)));
+                            } else {
+                                addNullString();
+                            }
+                        }
+
+                        public void longColumn(Column column)
+                        {
+                            addDelimiter(column);
+                            if (!pageReader.isNull(column)) {
+                                addValue(Long.toString(pageReader.getLong(column)));
+                            } else {
+                                addNullString();
+                            }
+                        }
+
+                        public void doubleColumn(Column column)
+                        {
+                            addDelimiter(column);
+                            if (!pageReader.isNull(column)) {
+                                addValue(Double.toString(pageReader.getDouble(column)));
+                            } else {
+                                addNullString();
+                            }
+                        }
+
+                        public void stringColumn(Column column)
+                        {
+                            addDelimiter(column);
+                            if (!pageReader.isNull(column)) {
+                                addValue(pageReader.getString(column));
+                            } else {
+                                addNullString();
+                            }
+                        }
+
+                        public void timestampColumn(Column column)
+                        {
+                            addDelimiter(column);
+                            if (!pageReader.isNull(column)) {
+                                Timestamp value = pageReader.getTimestamp(column);
+                                addValue(timestampFormatters[column.getIndex()].format(value));
+                            } else {
+                                addNullString();
+                            }
+                        }
+
+                        private void addDelimiter(Column column)
+                        {
+                            if (column.getIndex() != 0) {
+                                encoder.addText(delimiterString);
+                            }
+                        }
+
+                        private void addValue(String v)
+                        {
+                            encoder.addText(setEscapeAndQuoteValue(v, delimiter, quotePolicy, quote, escape, newlineInField, nullString));
+                        }
+
+                        private void addNullString()
+                        {
+                            encoder.addText(nullString);
+                        }
+                    });
+                    encoder.addNewLine();
+                }
+            }
+
+            public void finish()
+            {
+                encoder.finish();
+            }
+
+            public void close()
+            {
+                encoder.close();
+            }
+        };
+    }
+
+    private void writeHeader(Schema schema, LineEncoder encoder, char delimiter, QuotePolicy policy, char quote, char escape, String newline, String nullString)
+    {
+        String delimiterString = String.valueOf(delimiter);
+        for (Column column : schema.getColumns()) {
+            if (column.getIndex() != 0) {
+                encoder.addText(delimiterString);
+            }
+            encoder.addText(setEscapeAndQuoteValue(column.getName(), delimiter, policy, quote, escape, newline, nullString));
+        }
+        encoder.addNewLine();
+    }
+
+    private String setEscapeAndQuoteValue(String v, char delimiter, QuotePolicy policy, char quote, char escape, String newline, String nullString)
+    {
+        StringBuilder escapedValue = new StringBuilder();
+        char previousChar = ' ';
+
+        boolean isRequireQuote = (policy == QuotePolicy.ALL || policy == QuotePolicy.MINIMAL && v.equals(nullString)) ? true : false;
+
+        for (int i = 0; i < v.length(); i++) {
+            char c = v.charAt(i);
+
+            if (c == quote) {
+                escapedValue.append(escape);
+                escapedValue.append(c);
+                isRequireQuote = true;
+            } else if (c == '\r') {
+                if (policy == QuotePolicy.NONE) {
+                    escapedValue.append(escape);
+                }
+                escapedValue.append(newline);
+                isRequireQuote = true;
+            } else if (c == '\n') {
+                if (previousChar != '\r') {
+                    if (policy == QuotePolicy.NONE) {
+                        escapedValue.append(escape);
+                    }
+                    escapedValue.append(newline);
+                    isRequireQuote = true;
+                }
+            } else if (c == delimiter) {
+                if (policy == QuotePolicy.NONE) {
+                    escapedValue.append(escape);
+                }
+                escapedValue.append(c);
+                isRequireQuote = true;
+            } else {
+                escapedValue.append(c);
+            }
+            previousChar = c;
+        }
+
+        if (policy != QuotePolicy.NONE && isRequireQuote) {
+            return setQuoteValue(escapedValue.toString(), quote);
+        } else {
+            return escapedValue.toString();
+        }
+    }
+
+    private String setQuoteValue(String v, char quote)
+    {
+        StringBuilder sb = new StringBuilder();
+        sb.append(quote);
+        sb.append(v);
+        sb.append(quote);
+
+        return sb.toString();
+    }
+}