embulk-filter-base64 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 920673effb3067f179e366a12a6b1ea760a9c199
-  data.tar.gz: a77a241890b994b2fabc95d1ae175f314f70f6ce
+  metadata.gz: 076a19622912f88e9f4c0446a5ff1b656de12cad
+  data.tar.gz: 62fe25a76158dfacafc3af6b57b34bb76cd1f34c
 SHA512:
-  metadata.gz: 85586eecb14aeba081c8706353c393c11d7b3592b1ef3d3f3a5a90bcd8bce8268adaad14c044c35cab95a2da1c94eb6ebdadfb899ffcb2d982a7e287ef635e80
-  data.tar.gz: db625af1af9e2ba4a3b4f38de4c23d5f1f48276be36b45b89edd0408b6eb7cc69f7f88f699edc96f373baa66ec06a0bc283f23cb9fd36b53fa8832bc67e31d3c
+  metadata.gz: 04e9a03179c8c906dda682eaa3db724a83fbf321f9559f0de6dd1797518ddc9be7382a0440e43c93ec440d574c79c23a3c1927e82b4e18d49a82162a26eb749c
+  data.tar.gz: a6b749e4f88042449bd4e0ee3b8cfa2fbfd937fe5a2370a6f10f833b818fa4a2305c7e0d811344010685b45d0171fde3e3707cc659c1f7ee7dc306bcd79ed877

data/README.md

@@ -1,6 +1,6 @@
 # Base64 filter plugin for Embulk
 
-An Embulk filter plugin to encode/decode string by Base64.
+An Embulk filter plugin to encode/decode string by Base64, Base32 and Base16.
 
 ## Overview
 
@@ -8,23 +8,21 @@ An Embulk filter plugin to encode/decode string by Base64.
 
 ## Configuration
 
-* columns: Input columns to encode/decode (array of hash, required)
+* **columns**: Input columns to encode/decode (array of hash, required)
   - **name**
     + name of input column (string, required)
   - **decode**, **encode**
     + whether to encode or decode the value (boolean, default: `false`)
-    + either one must be `true` and exception is thrown if both are `true` or both are `false`
-
-## Limitation
-
-* Java8 environment is needed because this plugin uses [java.util.Base64](https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html) of Java8
-* Type of input value to be encoded must be string.
-  - encoded value is string and is is needed to align the type of input and output value
-    + e.g. 1234(string) is encoded into MTIzNA==(string)
-* Type of output decoded value is also string
-  - this plugin does nothing like type casting
-    + e.g. MTIzNA==(string) is decoded into 1234(string)
-  - if you want type casting, use [embulk-filter-typecast](https://github.com/sonots/embulk-filter-typecast)
+    + either one must be `true` and exception is thrown when both are `true` or both are `false`
+  - **encording**
+    + encording type (string, default: 'Base64')
+    + must be one of the follwing, Base64, Base32, Base16
+  - **urlsafe**
+    + whether to use urlsafe character in encoded string
+    + works only when `encording: Base64`
+  - **hex**
+    + whether to maintain sort order of encoded string
+    + works only when `encording: Base32`
 
 ## Example
 
@@ -32,79 +30,53 @@ An Embulk filter plugin to encode/decode string by Base64.
 
 See [example_encode.yml](./example/example_encode.yml) and [example_encode.csv](./example/example_encode.csv).
 
+input:
+
+```csv
+100,A0?B1>,A0?B1>,A0?B1>,A0?B1>,A0?B1>
+101,ab?cd~,ab?cd~,ab?cd~,ab?cd~,ab?cd~
+```
+
+output:
+
 ```bash
-$ embulk preview -G example/example_encode.yml
-*************************** 1 ***************************
-                 id (  long) : 10
-   string to encode (string) : Sm9obg==
-     long to encode (string) : MTIzNDU2
-   double to encode (string) : MzYuMjg=
-  boolean to encode (string) : dHJ1ZQ==
-timestamp to encode (string) : MjAxNi0xMi0zMSAyMzo1OTo1OQ==
-     json to encode (string) : eyJhZ2UiOiAyM30=
-*************************** 2 ***************************
-                 id (  long) : 11
-   string to encode (string) : RGF2aWQ=
-     long to encode (string) : MjM0NTY3
-   double to encode (string) : MTI1LjE=
-  boolean to encode (string) : ZmFsc2U=
-timestamp to encode (string) : MjAxNy0wMS0wMSAwMDowMDowMA==
-     json to encode (string) : eyJhZ2UiOiAzNH0=
+$ embulk preview example/example_encode.yml
++---------+---------------+------------------+------------------+------------------+---------------+
+| id:long | Base64:string | Base64Url:string |    Base32:string | Base32Hex:string | Base16:string |
++---------+---------------+------------------+------------------+------------------+---------------+
+|     100 |      QTA/QjE+ |         QTA_QjE- | IEYD6QRRHY====== | 84O3UGHH7O====== |  41303F42313E |
+|     101 |      YWI/Y2R+ |         YWI_Y2R- | MFRD6Y3EPY====== | C5H3UOR4FO====== |  61623F63647E |
++---------+---------------+------------------+------------------+------------------+---------------+
 ```
 
 ### decode
 
 See [example_decode.yml](./example/example_decode.yml) and [example_decode.csv](./example/example_decode.csv).
 
-```bash
-$ embulk preview -G example/example_decode.yml
-*************************** 1 ***************************
-                 id (  long) : 10
-   string to decode (string) : John
-     long to decode (string) : 123456
-   double to decode (string) : 36.28
-  boolean to decode (string) : true
-timestamp to decode (string) : 2016-12-31 23:59:59
-     json to decode (string) : {"age": 23}
-*************************** 2 ***************************
-                 id (  long) : 11
-   string to decode (string) : David
-     long to decode (string) : 234567
-   double to decode (string) : 125.01
-  boolean to decode (string) : false
-timestamp to decode (string) : 2017-01-01 00:00:00
-     json to decode (string) : {"age": 34}
-```
+input:
 
-### decode and cast type
+```csv
+100,QTA/QjE+,QTA_QjE-,IEYD6QRRHY======,84O3UGHH7O======,41303F42313E
+101,YWI/Y2R+,YWI_Y2R-,MFRD6Y3EPY======,C5H3UOR4FO======,61623F63647E
+```
 
-See [example_decode_with_typecast.yml](./example/example_decode_with_typecast.yml) and [example_decode.csv](./example/example_decode.csv).
+output:
 
 ```bash
-$ embulk preview -G example/example_decode_with_typecast.yml
-*************************** 1 ***************************
-                 id (     long) : 10
-   string to decode (   string) : John
-     long to decode (     long) : 123,456
-   double to decode (   double) : 36.28
-  boolean to decode (  boolean) : true
-timestamp to decode (timestamp) : 2016-12-31 23:59:00.590 UTC
-     json to decode (     json) : {"age":23}
-*************************** 2 ***************************
-                 id (     long) : 11
-   string to decode (   string) : David
-     long to decode (     long) : 234,567
-   double to decode (   double) : 125.01
-  boolean to decode (  boolean) : false
-timestamp to decode (timestamp) : 2017-01-01 00:00:00 UTC
-     json to decode (     json) : {"age":34}
+$ embulk preview example/example_decode.yml
++---------+---------------+------------------+---------------+------------------+---------------+
+| id:long | Base64:string | Base64Url:string | Base32:string | Base32Hex:string | Base16:string |
++---------+---------------+------------------+---------------+------------------+---------------+
+|     100 |        A0?B1> |           A0?B1> |        A0?B1> |           A0?B1> |        A0?B1> |
+|     101 |        ab?cd~ |           ab?cd~ |        ab?cd~ |           ab?cd~ |        ab?cd~ |
++---------+---------------+------------------+---------------+------------------+---------------+
 ```
 
+## Limitation
 
-### Todo
-
-* [Support base64 in apache commons codec](https://github.com/ysk24ok/embulk-filter-base64/issues/1)
-* [Support encoder/decoder of URL and MIME](https://github.com/ysk24ok/embulk-filter-base64/issues/2)
+* Type of input value to be encoded must be string.
+* Type of decoded output value will be string
+  - if you want type casting, use [embulk-filter-typecast](https://github.com/sonots/embulk-filter-typecast)
 
 ## Build
 

data/build.gradle

@@ -13,7 +13,7 @@ configurations {
     provided
 }
 
-version = "0.1.1"
+version = "0.2.0"
 
 sourceCompatibility = 1.7
 targetCompatibility = 1.7
@@ -23,6 +23,7 @@ dependencies {
     provided "org.embulk:embulk-core:0.8.15"
     // compile "YOUR_JAR_DEPENDENCY_GROUP:YOUR_JAR_DEPENDENCY_MODULE:YOUR_JAR_DEPENDENCY_VERSION"
     testCompile "junit:junit:4.+"
+    testCompile "org.hamcrest:hamcrest-library:1.+"
     testCompile "org.embulk:embulk-core:0.8.15:tests"
 }
 
@@ -76,7 +77,7 @@ Gem::Specification.new do |spec|
   spec.version       = "${project.version}"
   spec.authors       = ["Yusuke NISHIOKA"]
   spec.summary       = %[Base64 filter plugin for Embulk]
-  spec.description   = %[An Embulk filter plugin to encode/decode string by Base64.]
+  spec.description   = %[An Embulk filter plugin to encode/decode string by Base64, Base32 and Base16.]
   spec.email         = ["yusuke.nishioka.0713@gmail.com"]
   spec.licenses      = ["MIT"]
   spec.homepage      = "https://github.com/ysk24ok/embulk-filter-base64"

data/example/example_decode.csv

@@ -1,2 +1,2 @@
-10,Sm9obg==,MTIzNDU2,MzYuMjg=,dHJ1ZQ==,MjAxNi0xMi0zMSAyMzo1OTo1OQ==,eyJhZ2UiOiAyM30=
-11,RGF2aWQ=,MjM0NTY3,MTI1LjAx,ZmFsc2U=,MjAxNy0wMS0wMSAwMDowMDowMA==,eyJhZ2UiOiAzNH0=
+100,QTA/QjE+,QTA_QjE-,IEYD6QRRHY======,84O3UGHH7O======,41303F42313E
+101,YWI/Y2R+,YWI_Y2R-,MFRD6Y3EPY======,C5H3UOR4FO======,61623F63647E

data/example/example_decode.yml

@@ -4,21 +4,19 @@ in:
   parser:
     type: csv
     columns:
-      - {name: id, type: long}
-      - {name: string to decode, type: string}
-      - {name: long to decode, type: string}
-      - {name: double to decode, type: string}
-      - {name: boolean to decode, type: string}
-      - {name: timestamp to decode, type: string}
-      - {name: json to decode, type: string}
+      - {name: id,        type: long}
+      - {name: Base64,    type: string}
+      - {name: Base64Url, type: string}
+      - {name: Base32,    type: string}
+      - {name: Base32Hex, type: string}
+      - {name: Base16,    type: string}
 filters:
   - type: base64
     columns:
-      - {name: string to decode, decode: true}
-      - {name: long to decode, decode: true}
-      - {name: double to decode, decode: true}
-      - {name: boolean to decode, decode: true}
-      - {name: timestamp to decode, decode: true}
-      - {name: json to decode, decode: true}
+      - {name: Base64,    decode: true}
+      - {name: Base64Url, decode: true, urlsafe: true}
+      - {name: Base32,    decode: true, encoding: Base32}
+      - {name: Base32Hex, decode: true, encoding: Base32, hex: true}
+      - {name: Base16,    decode: true, encoding: Base16}
 out:
   type: stdout

data/example/example_encode.csv

@@ -1,2 +1,2 @@
-10,John,123456,36.28,true,2016-12-31 23:59:59,{"age": 23}
-11,David,234567,125.1,false,2017-01-01 00:00:00,{"age": 34}
+100,A0?B1>,A0?B1>,A0?B1>,A0?B1>,A0?B1>
+101,ab?cd~,ab?cd~,ab?cd~,ab?cd~,ab?cd~

data/example/example_encode.yml

@@ -4,21 +4,19 @@ in:
   parser:
     type: csv
     columns:
-      - {name: id, type: long}
-      - {name: string to encode, type: string}
-      - {name: long to encode, type: string}
-      - {name: double to encode, type: string}
-      - {name: boolean to encode, type: string}
-      - {name: timestamp to encode, type: string}
-      - {name: json to encode, type: string}
+      - {name: id,        type: long}
+      - {name: Base64,    type: string}
+      - {name: Base64Url, type: string}
+      - {name: Base32,    type: string}
+      - {name: Base32Hex, type: string}
+      - {name: Base16,    type: string}
 filters:
   - type: base64
     columns:
-      - {name: string to encode, encode: true}
-      - {name: long to encode, encode: true}
-      - {name: double to encode, encode: true}
-      - {name: boolean to encode, encode: true}
-      - {name: timestamp to encode, encode: true}
-      - {name: json to encode, encode: true}
+      - {name: Base64,    encode: true}
+      - {name: Base64Url, encode: true, urlsafe: true}
+      - {name: Base32,    encode: true, encoding: Base32}
+      - {name: Base32Hex, encode: true, encoding: Base32, hex: true}
+      - {name: Base16,    encode: true, encoding: Base16}
 out:
   type: stdout

data/src/main/java/org/embulk/filter/base64/Base64Filter.java

@@ -0,0 +1,100 @@
+package org.embulk.filter.base64;
+
+import com.google.common.io.BaseEncoding;
+
+import org.embulk.filter.base64.Base64FilterPlugin.ColumnTask;
+import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
+import org.embulk.spi.Column;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+public class Base64Filter
+{
+    private PluginTask task;
+    private final Map<String, ColumnTask> columnTaskMap;
+
+    Base64Filter(PluginTask task)
+    {
+        this.task = task;
+        this.columnTaskMap = getColumnTaskMap(task.getColumns());
+    }
+
+    private Map<String, ColumnTask> getColumnTaskMap(
+            List<ColumnTask> columnTasks)
+    {
+        Map<String, ColumnTask> m = new HashMap<>();
+        for (ColumnTask columnTask : columnTasks) {
+            m.put(columnTask.getName(), columnTask);
+        }
+        return m;
+    }
+
+    private ColumnTask getTask(Column column)
+    {
+        String colName = column.getName();
+        return columnTaskMap.get(colName);
+    }
+
+    public String doFilter(Column column, String inputValue)
+    {
+        ColumnTask colTask = getTask(column);
+        if (colTask == null) {
+            return inputValue;
+        }
+        String encoding = colTask.getEncoding();
+        boolean doEncode = colTask.getDoEncode().get();
+        boolean doDecode = colTask.getDoDecode().get();
+        // encode
+        if (doEncode && ! doDecode) {
+            byte[] inputAsBytes = inputValue.getBytes();
+            String encoded = null;
+            if (Objects.equals(encoding, "Base64")) {
+                if (colTask.getDoUrlsafe().get()) {
+                    encoded = BaseEncoding.base64Url().encode(inputAsBytes);
+                }
+                else {
+                    encoded = BaseEncoding.base64().encode(inputAsBytes);
+                }
+            }
+            else if (Objects.equals(encoding, "Base32")) {
+                if (colTask.getUseHex().get()) {
+                    encoded = BaseEncoding.base32Hex().encode(inputAsBytes);
+                }
+                else {
+                    encoded = BaseEncoding.base32().encode(inputAsBytes);
+                }
+            }
+            else if (Objects.equals(encoding, "Base16")) {
+                encoded = BaseEncoding.base16().encode(inputAsBytes);
+            }
+            return encoded;
+        }
+        // decode
+        else {
+            byte[] decodedAsBytes = null;
+            if (Objects.equals(encoding, "Base64")) {
+                if (colTask.getDoUrlsafe().get()) {
+                    decodedAsBytes = BaseEncoding.base64Url().decode(inputValue);
+                }
+                else {
+                    decodedAsBytes = BaseEncoding.base64().decode(inputValue);
+                }
+            }
+            else if (Objects.equals(encoding, "Base32")) {
+                if (colTask.getUseHex().get()) {
+                    decodedAsBytes = BaseEncoding.base32Hex().decode(inputValue);
+                }
+                else {
+                    decodedAsBytes = BaseEncoding.base32().decode(inputValue);
+                }
+            }
+            else if (Objects.equals(encoding, "Base16")) {
+                decodedAsBytes = BaseEncoding.base16().decode(inputValue);
+            }
+            return new String(decodedAsBytes);
+        }
+    }
+}

data/src/main/java/org/embulk/filter/base64/Base64FilterPlugin.java

@@ -1,29 +1,27 @@
 package org.embulk.filter.base64;
 
-import java.util.List;
-import java.util.Map;
-import java.util.HashMap;
-import java.util.Base64;
-
 import com.google.common.base.Optional;
+
 import org.embulk.config.Config;
 import org.embulk.config.ConfigDefault;
+import org.embulk.config.ConfigException;
 import org.embulk.config.ConfigSource;
 import org.embulk.config.Task;
 import org.embulk.config.TaskSource;
 import org.embulk.spi.Column;
-import org.embulk.spi.ColumnVisitor;
-import org.embulk.spi.DataException;
 import org.embulk.spi.Exec;
 import org.embulk.spi.FilterPlugin;
 import org.embulk.spi.Page;
 import org.embulk.spi.PageBuilder;
-import org.embulk.spi.PageReader;
 import org.embulk.spi.PageOutput;
+import org.embulk.spi.PageReader;
 import org.embulk.spi.Schema;
 import org.embulk.spi.type.Type;
 import org.embulk.spi.type.Types;
 
+import java.util.Arrays;
+import java.util.List;
+
 public class Base64FilterPlugin
         implements FilterPlugin
 {
@@ -31,17 +29,15 @@ public class Base64FilterPlugin
             extends Task
     {
         @Config("columns")
-        public List<Base64ColumnTask> getColumns();
+        public List<ColumnTask> getColumns();
     }
 
-    public interface Base64ColumnTask
+    public interface ColumnTask
             extends Task
     {
         @Config("name")
         public String getName();
 
-        // TODO: getType
-
         @Config("encode")
         @ConfigDefault("false")
         public Optional<Boolean> getDoEncode();
@@ -49,27 +45,46 @@ public class Base64FilterPlugin
         @Config("decode")
         @ConfigDefault("false")
         public Optional<Boolean> getDoDecode();
+
+        @Config("encoding")
+        @ConfigDefault("\"Base64\"")
+        public String getEncoding();
+
+        @Config("urlsafe")
+        @ConfigDefault("false")
+        public Optional<Boolean> getDoUrlsafe();
+
+        @Config("hex")
+        @ConfigDefault("false")
+        public Optional<Boolean> getUseHex();
     }
 
     public void validate(PluginTask pluginTask, Schema inputSchema)
     {
-        for (Base64ColumnTask task : pluginTask.getColumns()) {
-            // throws exception if the column name does not exist
-            inputSchema.lookupColumn(task.getName());
+        for (ColumnTask task : pluginTask.getColumns()) {
+            // throws exception when the column does not exist
+            Column column = inputSchema.lookupColumn(task.getName());
+            // check 'encode: true' or 'decode: true' in ColumnTask
             boolean doEncode = task.getDoEncode().get();
             boolean doDecode = task.getDoDecode().get();
             boolean bothTrue = doEncode && doDecode;
             boolean bothFalse = !doEncode && !doDecode;
             if (bothTrue || bothFalse) {
                 String errMsg = "Specify either 'encode: true' or 'decode: true'";
-                throw new DataException(errMsg);
+                throw new ConfigException(errMsg);
             }
-        }
-        for (Column column : inputSchema.getColumns()) {
+            // check 'encoding' option in ColumnTask
+            String[] allowedEncordings = {"Base16", "Base32", "Base64"};
+            String encoding = task.getEncoding();
+            if (!Arrays.asList(allowedEncordings).contains(encoding)) {
+                String errMsg = "Encording must be one of the following: Base16, Base32, Base64";
+                throw new ConfigException(errMsg);
+            }
+            // check type of input cloumn
             Type colType = column.getType();
             if (!Types.STRING.equals(colType)) {
                 String errMsg = "Type of input columns must be string";
-                throw new DataException(errMsg);
+                throw new ConfigException(errMsg);
             }
         }
     }
@@ -92,8 +107,9 @@ public class Base64FilterPlugin
         PageBuilder pageBuilder = new PageBuilder(
             Exec.getBufferAllocator(), outputSchema, output);
         PageReader pageReader = new PageReader(inputSchema);
+        Base64Filter filter = new Base64Filter(task);
         ColumnVisitorImpl visitor = new ColumnVisitorImpl(
-            task, pageReader, pageBuilder);
+            pageReader, pageBuilder, filter);
 
         return new PageOutputImpl(
             pageReader, pageBuilder, outputSchema, visitor);

data/src/main/java/org/embulk/filter/base64/ColumnVisitorImpl.java

@@ -1,68 +1,21 @@
 package org.embulk.filter.base64;
 
-import java.util.List;
-import java.util.Map;
-import java.util.HashMap;
-import java.util.Base64;
-
-import org.embulk.config.Config;
-import org.embulk.config.ConfigDefault;
-import org.embulk.config.Task;
 import org.embulk.spi.Column;
 import org.embulk.spi.ColumnVisitor;
-import org.embulk.spi.DataException;
 import org.embulk.spi.PageBuilder;
 import org.embulk.spi.PageReader;
-import org.embulk.spi.Schema;
-
-import org.embulk.filter.base64.Base64FilterPlugin.Base64ColumnTask;
-import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
 
-public class ColumnVisitorImpl
-        implements ColumnVisitor
+public class ColumnVisitorImpl implements ColumnVisitor
 {
     private final PageReader pageReader;
     private final PageBuilder pageBuilder;
-    private final Map<String, Base64ColumnTask> base64ColumnMap;
+    private final Base64Filter filter;
 
-    ColumnVisitorImpl(PluginTask task, PageReader reader, PageBuilder builder)
+    ColumnVisitorImpl(PageReader reader, PageBuilder builder, Base64Filter filter)
     {
         this.pageReader = reader;
         this.pageBuilder = builder;
-        this.base64ColumnMap = getBase64ColumnMap(task.getColumns());
-    }
-
-    private static Map<String, Base64ColumnTask> getBase64ColumnMap(
-            List<Base64ColumnTask> columnTasks)
-    {
-        Map<String, Base64ColumnTask> m = new HashMap<>();
-        for (Base64ColumnTask columnTask : columnTasks) {
-            m.put(columnTask.getName(), columnTask);
-        }
-        return m;
-    }
-
-    private Base64ColumnTask getTask(Column column)
-    {
-        String colName = column.getName();
-        return base64ColumnMap.get(colName);
-    }
-
-    private String executeTask(Base64ColumnTask task, Column column)
-    {
-        boolean doEncode = task.getDoEncode().get();
-        boolean doDecode = task.getDoDecode().get();
-        // encode
-        if (doEncode) {
-            String raw = pageReader.getString(column);
-            return Base64.getEncoder().encodeToString(raw.getBytes());
-        }
-        // decode
-        //else if (doDecode) {
-        else {
-            String encoded = pageReader.getString(column);
-            return new String(Base64.getDecoder().decode(encoded));
-        }
+        this.filter = filter;
     }
 
     @Override
@@ -108,17 +61,9 @@ public class ColumnVisitorImpl
             pageBuilder.setNull(outputColumn);
         }
         else {
-            Base64ColumnTask task = getTask(outputColumn);
-            // when there is no task executed on this column
-            if (task == null) {
-                pageBuilder.setString(
-                    outputColumn, pageReader.getString(outputColumn));
-            // when there is a task
-            }
-            else {
-                String str = executeTask(task, outputColumn);
-                pageBuilder.setString(outputColumn, str);
-            }
+            String outputValue = filter.doFilter(
+                outputColumn, pageReader.getString(outputColumn));
+            pageBuilder.setString(outputColumn, outputValue);
         }
     }
 

data/src/test/java/org/embulk/filter/base64/TestBase64FilterPlugin.java

@@ -1,14 +1,14 @@
 package org.embulk.filter.base64;
 
 import org.embulk.EmbulkTestRuntime;
+import org.embulk.config.ConfigException;
 import org.embulk.config.ConfigLoader;
 import org.embulk.config.ConfigSource;
-import org.embulk.spi.DataException;
+import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
 import org.embulk.spi.Exec;
 import org.embulk.spi.Schema;
 import org.embulk.spi.SchemaConfigException;
 import org.embulk.spi.type.Types;
-import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
 
 import org.junit.Rule;
 import org.junit.Test;
@@ -39,8 +39,8 @@ public class TestBase64FilterPlugin
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to encode, type: string, encode: true}",
-            "  - {name: to decode, type: string, decode: true}"
+            "  - {name: to encode, encode: true}",
+            "  - {name: to decode, decode: true}"
         );
         Schema inputSchema = Schema.builder()
             .add("to encode", Types.STRING)
@@ -48,13 +48,13 @@ public class TestBase64FilterPlugin
         plugin.validate(task, inputSchema);
     }
 
-    @Test(expected = DataException.class)
+    @Test(expected = ConfigException.class)
     public void testValidate_bothSpecified()
     {
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to encode, type: string, encode: true, decode: true}"
+            "  - {name: to encode, encode: true, decode: true}"
         );
         Schema inputSchema = Schema.builder()
             .add("to encode", Types.STRING)
@@ -62,13 +62,13 @@ public class TestBase64FilterPlugin
         plugin.validate(task, inputSchema);
     }
 
-    @Test(expected = DataException.class)
+    @Test(expected = ConfigException.class)
     public void testValidate_bothNotSpecified()
     {
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to encode, type: string}"
+            "  - {name: to encode}"
         );
         Schema inputSchema = Schema.builder()
             .add("to encode", Types.STRING)
@@ -76,13 +76,13 @@ public class TestBase64FilterPlugin
         plugin.validate(task, inputSchema);
     }
 
-    @Test(expected = DataException.class)
+    @Test(expected = ConfigException.class)
     public void testValidate_invalidInputType()
     {
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to encode, type: string, type: double}"
+            "  - {name: to encode, encode: true}"
         );
         Schema inputSchema = Schema.builder()
             .add("to encode", Types.DOUBLE)

data/src/test/java/org/embulk/filter/base64/TestColumnVisitorImpl.java

@@ -1,11 +1,8 @@
 package org.embulk.filter.base64;
 
-import java.util.List;
-
 import org.embulk.EmbulkTestRuntime;
-import org.embulk.config.ConfigLoader;
-import org.embulk.config.ConfigSource;
-import org.embulk.spi.Exec;
+import org.embulk.filter.base64.Base64FilterPlugin.PageOutputImpl;
+import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
 import org.embulk.spi.Page;
 import org.embulk.spi.PageBuilder;
 import org.embulk.spi.PageOutput;
@@ -15,20 +12,37 @@ import org.embulk.spi.Schema;
 import org.embulk.spi.TestPageBuilderReader.MockPageOutput;
 import org.embulk.spi.type.Types;
 import org.embulk.spi.util.Pages;
-import org.embulk.filter.base64.Base64FilterPlugin.PluginTask;
-import org.embulk.filter.base64.Base64FilterPlugin.PageOutputImpl;
-import org.embulk.filter.base64.ColumnVisitorImpl;
-import static org.embulk.filter.base64.TestBase64FilterPlugin.taskFromYamlString;
 
+import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
-import static org.junit.Assert.*;
+
+import static org.embulk.filter.base64.TestBase64FilterPlugin.taskFromYamlString;
+import static org.junit.Assert.assertEquals;
+
+import java.util.List;
 
 public class TestColumnVisitorImpl
 {
     @Rule
     public EmbulkTestRuntime runtime = new EmbulkTestRuntime();
 
+    private Schema inputSchemaToEncode;
+    private Schema inputSchemaToDecode;
+
+    @Before
+    public void createResource()
+    {
+        inputSchemaToEncode = Schema.builder()
+            .add("id", Types.LONG)
+            .add("to encode", Types.STRING)
+            .build();
+        inputSchemaToDecode = Schema.builder()
+            .add("id", Types.LONG)
+            .add("to decode", Types.STRING)
+            .build();
+    }
+
     private List<Object[]> filter(PluginTask task, Schema inputSchema, Object... objects)
     {
         MockPageOutput output = new MockPageOutput();
@@ -36,8 +50,9 @@ public class TestColumnVisitorImpl
         PageBuilder pageBuilder = new PageBuilder(
             runtime.getBufferAllocator(), outputSchema, output);
         PageReader pageReader = new PageReader(inputSchema);
+        Base64Filter filter = new Base64Filter(task);
         ColumnVisitorImpl visitor = new ColumnVisitorImpl(
-            task, pageReader, pageBuilder);
+            pageReader, pageBuilder, filter);
 
         List<Page> pages = PageTestUtils.buildPage(
             runtime.getBufferAllocator(), inputSchema, objects);
@@ -52,42 +67,270 @@ public class TestColumnVisitorImpl
     }
 
     @Test
-    public void testExecuteTask_encode()
+    public void testExecuteTask_encodeByBase64()
     {
+        // implicit
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to encode, type: string, encode: true}"
+            "  - {name: id}",
+            "  - {name: to encode, encode: true}"
         );
-        Schema inputSchema = Schema.builder()
-            .add("to encode", Types.STRING)
-            .build();
-        List<Object[]> records = filter(task, inputSchema,
-                "John",
-                "David"
+        List<Object[]> records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
         );
         assertEquals(2, records.size());
-        assertEquals("Sm9obg==", records.get(0)[0]);
-        assertEquals("RGF2aWQ=", records.get(1)[0]);
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("QTA/QjE+", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("YWI/Y2R+", records.get(1)[1]);
+        // explicit
+        task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, encoding: Base64}"
+        );
+        records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("QTA/QjE+", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("YWI/Y2R+", records.get(1)[1]);
     }
 
     @Test
-    public void testExecuteTask_decode()
+    public void testExecuteTask_encodeByBase64Url()
     {
+        // implicit
         PluginTask task = taskFromYamlString(
             "type: base64",
             "columns:",
-            "  - {name: to decode, type: string, decode: true}"
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, urlsafe: true}"
         );
-        Schema inputSchema = Schema.builder()
-            .add("to decode", Types.STRING)
-            .build();
-        List<Object[]> records = filter(task, inputSchema,
-                "Sm9obg==",
-                "RGF2aWQ="
+        List<Object[]> records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("QTA_QjE-", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("YWI_Y2R-", records.get(1)[1]);
+        // explicit
+        task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, encoding: Base64, urlsafe: true}"
+        );
+        records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("QTA_QjE-", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("YWI_Y2R-", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_encodeByBase32()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, encoding: Base32}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("IEYD6QRRHY======", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("MFRD6Y3EPY======", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_encodeByBase32Hex()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, encoding: Base32, hex: true}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("84O3UGHH7O======", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("C5H3UOR4FO======", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_encodeByBase16()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to encode, encode: true, encoding: Base16}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToEncode,
+            Long.valueOf(100), "A0?B1>",
+            Long.valueOf(101), "ab?cd~"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("41303F42313E", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("61623F63647E", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_decodeByBase64()
+    {
+        // implicit
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "QTA/QjE+",
+            Long.valueOf(101), "YWI/Y2R+"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+        // explicit
+        task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, encoding: Base64}"
+        );
+        records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "QTA/QjE+",
+            Long.valueOf(101), "YWI/Y2R+"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_decodeByBase64Url()
+    {
+        // implicit
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, urlsafe: true}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "QTA_QjE-",
+            Long.valueOf(101), "YWI_Y2R-"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+        // explicit
+        task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, encoding: Base64, urlsafe: true}"
+        );
+        records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "QTA_QjE-",
+            Long.valueOf(101), "YWI_Y2R-"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_decodeByBase32()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, encoding: Base32}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "IEYD6QRRHY======",
+            Long.valueOf(101), "MFRD6Y3EPY======"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_decodeByBase32Hex()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, encoding: Base32, hex: true}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "84O3UGHH7O======",
+            Long.valueOf(101), "C5H3UOR4FO======"
+        );
+        assertEquals(2, records.size());
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
+    }
+
+    @Test
+    public void testExecuteTask_decodeByBase16()
+    {
+        PluginTask task = taskFromYamlString(
+            "type: base64",
+            "columns:",
+            "  - {name: id}",
+            "  - {name: to decode, decode: true, encoding: Base16}"
+        );
+        List<Object[]> records = filter(task, inputSchemaToDecode,
+            Long.valueOf(100), "41303F42313E",
+            Long.valueOf(101), "61623F63647E"
         );
         assertEquals(2, records.size());
-        assertEquals("John", records.get(0)[0]);
-        assertEquals("David", records.get(1)[0]);
+        assertEquals(Long.valueOf(100), records.get(0)[0]);
+        assertEquals("A0?B1>", records.get(0)[1]);
+        assertEquals(Long.valueOf(101), records.get(1)[0]);
+        assertEquals("ab?cd~", records.get(1)[1]);
     }
 }

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: embulk-filter-base64
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Yusuke NISHIOKA
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2017-01-01 00:00:00.000000000 Z
+date: 2017-01-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -38,7 +38,7 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '10.0'
-description: An Embulk filter plugin to encode/decode string by Base64.
+description: An Embulk filter plugin to encode/decode string by Base64, Base32 and Base16.
 email:
 - yusuke.nishioka.0713@gmail.com
 executables: []
@@ -53,7 +53,6 @@ files:
 - config/checkstyle/default.xml
 - example/example_decode.csv
 - example/example_decode.yml
-- example/example_decode_with_typecast.yml
 - example/example_encode.csv
 - example/example_encode.yml
 - example/get_string.py
@@ -62,11 +61,12 @@ files:
 - gradlew
 - gradlew.bat
 - lib/embulk/filter/base64.rb
+- src/main/java/org/embulk/filter/base64/Base64Filter.java
 - src/main/java/org/embulk/filter/base64/Base64FilterPlugin.java
 - src/main/java/org/embulk/filter/base64/ColumnVisitorImpl.java
 - src/test/java/org/embulk/filter/base64/TestBase64FilterPlugin.java
 - src/test/java/org/embulk/filter/base64/TestColumnVisitorImpl.java
-- classpath/embulk-filter-base64-0.1.1.jar
+- classpath/embulk-filter-base64-0.2.0.jar
 homepage: https://github.com/ysk24ok/embulk-filter-base64
 licenses:
 - MIT

data/example/example_decode_with_typecast.yml

@@ -1,32 +0,0 @@
-in:
-  type: file
-  path_prefix: example/example_decode.csv
-  parser:
-    type: csv
-    columns:
-      - {name: id, type: long}
-      - {name: string to decode, type: string}
-      - {name: long to decode, type: string}
-      - {name: double to decode, type: string}
-      - {name: boolean to decode, type: string}
-      - {name: timestamp to decode, type: string}
-      - {name: json to decode, type: string}
-filters:
-  - type: base64
-    columns:
-      - {name: string to decode, decode: true}
-      - {name: long to decode, decode: true}
-      - {name: double to decode, decode: true}
-      - {name: boolean to decode, decode: true}
-      - {name: timestamp to decode, decode: true}
-      - {name: json to decode, decode: true}
-  - type: typecast
-    columns:
-      - {name: string to decode, type: string}
-      - {name: long to decode, type: long}
-      - {name: double to decode, type: double}
-      - {name: boolean to decode, type: boolean}
-      - {name: timestamp to decode, type: timestamp, format: "%Y-%m-%d %H:%M:%N", timezone: "UTC"}
-      - {name: json to decode, type: json}
-out:
-  type: stdout