logstash-codec-protobuf 1.2.1 → 1.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 48d78608a993ce87e16cefbf92995c27ea0f95b3a9af82f030d3bd7fc22a1e6d
-  data.tar.gz: 5f1e2aab280322aab0878eae12ee51f98f30257025ce8d50b0bac39a4edd793e
+  metadata.gz: a4a059736022035d5e326d6cb27ddec13cd54da37dcb1d2f37a8076cb0179e37
+  data.tar.gz: 3402775465ce05bbbbc87a7367acc482cd7d2468c84e27ad07ef0ac4383e8a97
 SHA512:
-  metadata.gz: 932aaff952bf4982d5192701eb3a3997416d76dc750516d53e48f08689f16f93bef0c734a2813f9d71f36ac8a3566fc8916af0c3e3b33b58714b33f40848b1f0
-  data.tar.gz: 76562195f5f05b10dc34695447ace2bd74081e1fdee281c4f5b28e04722f518f6fff0fac789b462c7a7be2877c03236922063d7bb86d16676c853bf1506cac26
+  metadata.gz: 4b1dea1ed7701b7e6e26e8226991bc718fab06302071f60b7c8b8b0b4b9a2b45d1d0809be6fb7a76a30af0c291181a5a77cb8a51cdf2a4c5e9ddc2596c0afc1e
+  data.tar.gz: 778c28f5ed7cfa1ed5bd54f175f8efec4dc74e0d93e1d47dc3ded5671a2e61c6baf4e2f9b6bf9e87e896324f3b2a3e22c3a3692195b62fcb23a2bc37815f26aa

data/CHANGELOG.md

@@ -1,3 +1,6 @@
+## 1.2.2
+  - Add type conversion feature to encoder
+
 ## 1.2.1
   - Keep original data in case of parsing errors
 

data/README.md

@@ -3,34 +3,46 @@
 This is a codec plugin for [Logstash](https://github.com/elastic/logstash) to parse protobuf messages.
 
 # Prerequisites and Installation
- 
+
 * prepare your Ruby versions of the Protobuf definitions:
-  * For protobuf 2 use the [ruby-protoc compiler](https://github.com/codekitchen/ruby-protocol-buffers).  
+  * For protobuf 2 use the [ruby-protoc compiler](https://github.com/codekitchen/ruby-protocol-buffers).
   * For protobuf 3 use the [official google protobuf compiler](https://developers.google.com/protocol-buffers/docs/reference/ruby-generated).
 * install the codec: `bin/logstash-plugin install logstash-codec-protobuf`
 * use the codec in your Logstash config file. See details below.
 
 ## Configuration
 
-`include_path`  (required): an array of strings with filenames where logstash can find your protobuf definitions. Requires absolute paths. Please note that protobuf v2 files have the ending `.pb.rb` whereas files compiled for protobuf v3 end in `_pb.rb`.
+There are two ways to specify the locations of the ruby protobuf definitions:
+* specify each class and their loading order using the configurations `include_path`. This option will soon be deprecated in favour of the autoloader.
+* specify the path to the main protobuf class, and a folder from which to load its dependencies, using `class_file`  and `protobuf_root_directory`. The codec will detect the dependencies of each file and load them automatically.
+
+`include_path`  (optional): an array of strings with filenames where logstash can find your protobuf definitions. Requires absolute paths. Please note that protobuf v2 files have the ending `.pb.rb` whereas files compiled for protobuf v3 end in `_pb.rb`.  Cannot be used together with `protobuf_root_directory` or `class_file`.
+
+`protobuf_root_directory` (optional): Only to be used in combination with `class_file`. Absolute path to the directory that contains all compiled protobuf files. Cannot be used together with `include_path`.
+
+`class_file`  (optional): Relative path to the ruby file that contains class_name. Only to be used in combination with `protobuf_root_directory`. Cannot be used together with `include_path`.
 
-`class_name`    (required): the name of the protobuf class that is to be decoded or encoded. For protobuf 2 separate the modules with ::. For protobuf 3 use single dots. 
+`class_name`    (required): the name of the protobuf class that is to be decoded or encoded. For protobuf 2 separate the modules with ::. For protobuf 3 use single dots.
 
 `protobuf_version` (optional): set this to 3 if you want to use protobuf 3 definitions. Defaults to 2.
 
+`stop_on_error` (optional): Decoder only: will stop the entire pipeline upon discovery of a non decodable message. Deactivated by default.
+
+`pb3_encoder_autoconvert_types` (optional): Encoder only: will try to fix type mismatches between the protobuf definition and the actual data. Available for protobuf 3 only. Activated by default.
+
 ## Usage example: decoder
 
 Use this as a codec in any logstash input. Just provide the name of the class that your incoming objects will be encoded in, and specify the path to the compiled definition.
 Here's an example for a kafka input with protobuf 2:
 
 ```ruby
-kafka 
+kafka
 {
   topic_id => "..."
   key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
   value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
 
-  codec => protobuf 
+  codec => protobuf
   {
     class_name => "Animals::Mammals::Unicorn"
     include_path => ['/path/to/pb_definitions/Animal.pb.rb', '/path/to/pb_definitions/Unicorn.pb.rb']
@@ -38,15 +50,15 @@ kafka
 }
 ```
 
-Example for protobuf 3:
+Example for protobuf 3, manual class loading specification (deprecated):
 
 ```ruby
-kafka 
+kafka
 {
   topic_id => "..."
   key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
   value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
-  codec => protobuf 
+  codec => protobuf
   {
     class_name => "Animals.Mammals.Unicorn"
     include_path => ['/path/to/pb_definitions/Animal_pb.rb', '/path/to/pb_definitions/Unicorn_pb.rb']
@@ -55,6 +67,25 @@ kafka
 }
 ```
 
+Example for protobuf 3, automatic class loading specification:
+
+```ruby
+kafka
+{
+  topic_id => "..."
+  key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+  value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+  codec => protobuf
+  {
+    class_name => "Animals.Mammals.Unicorn"
+    class_file => '/path/to/pb_definitions/some_folder/Unicorn_pb.rb'
+    protobuf_root_directory => "/path/to/pb_definitions/"
+    protobuf_version => 3
+  }
+}
+```
+In this example, all protobuf files must live in a subfolder of `/path/to/pb_definitions/`.
+
 For version 3 class names check the bottom of the generated protobuf ruby file. It contains lines like this:
 
 ```ruby
@@ -67,7 +98,7 @@ If you're using a kafka input please also set the deserializer classes as shown
 
 ### Class loading order
 
-Imagine you have the following protobuf version 2 relationship: class Unicorn lives in namespace Animal::Horse and uses another class Wings. 
+Imagine you have the following protobuf version 2 relationship: class Unicorn lives in namespace Animal::Horse and uses another class Wings.
 
 ```ruby
 module Animal
@@ -91,7 +122,7 @@ Set the class name to the parent class:
 class_name => "Animal::Mammal::Unicorn"
 ```
 
-for protobuf 2. For protobuf 3 use 
+for protobuf 2. For protobuf 3 use
 
 ```ruby
 class_name => "Animal.Mammal.Unicorn"
@@ -99,14 +130,35 @@ class_name => "Animal.Mammal.Unicorn"
 
 ## Usage example: encoder
 
-The configuration of the codec for encoding logstash events for a protobuf output is pretty much the same as for the decoder input usage as demonstrated above. There are some constraints though that you need to be aware of:
+The configuration of the codec for encoding logstash events for a protobuf output is pretty much the same as for the decoder input usage as demonstrated above, with the following exception: when writing to the Kafka output,
+* do not set the `value_deserializer_class` or the `key_deserializer_class`.
+* do set the serializer class like so: `value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"`.
+
+Please be aware of the following:
 * the protobuf definition needs to contain all the fields that logstash typically adds to an event, in the corrent data type. Examples for this are `@timestamp` (string), `@version` (string), `host`, `path`, all of which depend on your input sources and filters aswell. If you do not want to add those fields to your protobuf definition then please use a [modify filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html) to [remove](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field) the undesired fields.
 * object members starting with `@` are somewhat problematic in protobuf definitions. Therefore those fields will automatically be renamed to remove the at character. This also effects the important `@timestamp` field. Please name it just "timestamp" in your definition.
+* fields with a nil value will automatically be removed from the event. Empty fields will not be removed.
+* it is recommended to set the config option `pb3_encoder_autoconvert_types` to true. Otherwise any type mismatch between your data and the protobuf definition will cause an event to be lost. The auto typeconversion does not alter your data. It just tries to convert obviously identical data into the expected datatype, such as converting integers to floats where floats are expected, or "true" / "false" strings into booleans where booleans are expected.
 
+```ruby
+  kafka
+    {
+      codec => protobuf
+      {
+        class_name => "Animals.Mammals.Unicorn"
+        class_file => '/path/to/pb_definitions/some_folder/Unicorn_pb.rb'
+        protobuf_root_directory => "/path/to/pb_definitions/"
+        protobuf_version => 3
+      }
+      ...
+      value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+    }
+  }
+```
 
 ## Troubleshooting
 
-### Protobuf 2 
+### Decoder: Protobuf 2
 #### "uninitialized constant SOME_CLASS_NAME"
 
 If you include more than one definition class, consider the order of inclusion. This is especially relevant if you include whole directories. A definition might refer to another definition that is not loaded yet. In this case, please specify the files in the `include_path` variable in reverse order of reference. See 'Example with referenced definitions' above.
@@ -115,12 +167,16 @@ If you include more than one definition class, consider the order of inclusion.
 
 Maybe your protobuf definition does not fullfill the requirements and needs additional fields. Run logstash with the `--debug` flag and search for error messages.
 
-### Protobuf 3
+### Decoder: Protobuf 3
+
+#### NullPointerException
+
+Check for missing imports. There's a high probability that one of the imported classes has dependencies of its own and those are not being fully satisfied. To avoid this, consider using the autoloader feature by setting the configurations for `protobuf_root_directory` and `class_file`.
 
-Tba.
+### Encoder: Protobuf 3
 
-## Limitations and roadmap
+#### NullPointerException
 
-* maybe add support for setting undefined fields from default values in the decoder
+Check for missing imports. There's a high probability that one of the imported classes has dependencies of its own and those are not being fully satisfied. To avoid this, consider using the autoloader feature by setting the configurations for `protobuf_root_directory` and `class_file`.
 
 

data/docs/index.asciidoc

@@ -28,29 +28,31 @@ For protobuf 3 use the https://developers.google.com/protocol-buffers/docs/refer
 
 The following shows a usage example (protobuf v2) for decoding events from a kafka stream:
 [source,ruby]
-kafka 
+kafka
 {
- topic_id => "..."
- key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
- value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
- codec => protobuf 
- {
-   class_name => "Animals::Mammals::Unicorn"
-   include_path => ['/path/to/protobuf/definitions/UnicornProtobuf.pb.rb']
- }
+  topic_id => "..."
+  key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+  value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+  codec => protobuf
+  {
+    class_name => "Animals::Mammals::Unicorn"
+    class_file => '/path/to/pb_definitions/some_folder/Unicorn.pb.rb'
+    protobuf_root_directory => "/path/to/pb_definitions/"
+  }
 }
 
-Usage example for protobuf v3:
+Decoder usage example for protobuf v3:
 [source,ruby]
-kafka 
+kafka
 {
   topic_id => "..."
   key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
   value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
-  codec => protobuf 
+  codec => protobuf
   {
     class_name => "Animals.Mammals.Unicorn"
-    include_path => ['/path/to/pb_definitions/Animal_pb.rb', '/path/to/pb_definitions/Unicorn_pb.rb']
+    class_file => '/path/to/pb_definitions/some_folder/Unicorn_pb.rb'
+    protobuf_root_directory => "/path/to/pb_definitions/"
     protobuf_version => 3
   }
 }
@@ -60,10 +62,29 @@ The codec can be used in input and output plugins. +
 When using the codec in the kafka input plugin please set the deserializer classes as shown above. +
 When using the codec in an output plugin:
 
-* make sure to include all the desired fields in the protobuf definition, including timestamp. 
-  Remove fields that are not part of the protobuf definition from the event by using the mutate filter.
+* make sure to include all the desired fields in the protobuf definition, including timestamp.
+  Remove fields that are not part of the protobuf definition from the event by using the mutate filter. Encoding will fail if the event has fields which are not in the protobuf definition.
 * the `@` symbol is currently not supported in field names when loading the protobuf definitions for encoding. Make sure to call the timestamp field `timestamp`
   instead of `@timestamp` in the protobuf file. Logstash event fields will be stripped of the leading `@` before conversion.
+* fields with a nil value will automatically be removed from the event. Empty fields will not be removed.
+* it is recommended to set the config option `pb3_encoder_autoconvert_types` to true. Otherwise any type mismatch between your data and the protobuf definition will cause an event to be lost. The auto typeconversion does not alter your data. It just tries to convert obviously identical data into the expected datatype, such as converting integers to floats where floats are expected, or "true" / "false" strings into booleans where booleans are expected.
+* When writing to Kafka: set the serializer class: `value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"`
+
+Encoder usage example (protobufg v3):
+
+[source,ruby]
+kafka
+  {
+    codec => protobuf
+    {
+      class_name => "Animals.Mammals.Unicorn"
+      class_file => '/path/to/pb_definitions/some_folder/Unicorn_pb.rb'
+      protobuf_root_directory => "/path/to/pb_definitions/"
+      protobuf_version => 3
+    }
+    value_serializer => "org.apache.kafka.common.serialization.ByteArraySerializer"
+  }
+}
 
 
 [id="plugins-{type}s-{plugin}-options"]
@@ -73,14 +94,18 @@ When using the codec in an output plugin:
 |=======================================================================
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-class_name>> |<<string,string>>|Yes
-| <<plugins-{type}s-{plugin}-include_path>> |<<array,array>>|Yes
+| <<plugins-{type}s-{plugin}-class_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-protobuf_root_directory>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-include_path>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-protobuf_version>> |<<number,number>>|Yes
+| <<plugins-{type}s-{plugin}-stop_on_error>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-pb3_encoder_autoconvert_types>> |<<boolean,boolean>>|No
 |=======================================================================
 
 &nbsp;
 
 [id="plugins-{type}s-{plugin}-class_name"]
-===== `class_name` 
+===== `class_name`
 
   * This is a required setting.
   * Value type is <<string,string>>
@@ -99,18 +124,45 @@ For protobuf v3, you can copy the class name from the Descriptorpool registratio
 [source,ruby]
 Animals.Mammals.Unicorn = Google::Protobuf::DescriptorPool.generated_pool.lookup("Animals.Mammals.Unicorn").msgclass
 
-
 If your class references other definitions: you only have to add the name of the main class here.
 
+[id="plugins-{type}s-{plugin}-class_file"]
+===== `class_file`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Absolute path to the directory that contains all compiled protobuf files. If the protobuf definitions are spread across multiple folders, this needs to point to the folder containing all those folders.
+
+[id="plugins-{type}s-{plugin}-protobuf_root_directory"]
+===== `protobuf_root_directory`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+Absolute path to the root directory that contains all referenced/used dependencies of the main class (`class_name`) or any of its dependencies. Must be used in combination with the `class_file` setting, and can not be used in combination with the legacy loading mechanism `include_path`.
+
+Example:
+
+[source]
+ pb3
+   ├── header
+   │   └── header_pb.rb
+   ├── messageA_pb.rb
+
+In this case `messageA_pb.rb` has an embedded message from `header/header_pb.rb`.
+If `class_file` is set to `messageA_pb.rb`, and `class_name` to `MessageA`, `protobuf_root_directory` must be set to `/path/to/pb3`, which includes both definitions.
+
+
 [id="plugins-{type}s-{plugin}-include_path"]
-===== `include_path` 
+===== `include_path`
 
-  * This is a required setting.
   * Value type is <<array,array>>
   * There is no default value for this setting.
 
-List of absolute pathes to files with protobuf definitions. 
-When using more than one file, make sure to arrange the files in reverse order of dependency so that each class is loaded before it is 
+Legacy protobuf definition loading mechanism for backwards compatibility:
+List of absolute pathes to files with protobuf definitions.
+When using more than one file, make sure to arrange the files in reverse order of dependency so that each class is loaded before it is
 refered to by another.
 
 Example: a class _Unicorn_ referencing another protobuf class _Wings_
@@ -129,13 +181,34 @@ include_path => ['/path/to/pb_definitions/wings.pb.rb','/path/to/pb_definitions/
 
 Please note that protobuf v2 files have the ending `.pb.rb` whereas files compiled for protobuf v3 end in `_pb.rb`.
 
+Cannot be used together with `protobuf_root_directory` or `class_file`.
+
 [id="plugins-{type}s-{plugin}-protobuf_version"]
-===== `protobuf_version` 
+===== `protobuf_version`
 
   * Value type is <<number,number>>
   * Default value is 2
 
 Protocol buffers version. Valid settings are 2, 3.
 
+[id="plugins-{type}s-{plugin}-stop_on_error"]
+===== `stop_on_error`
+
+  * Value type is <<boolean,boolean>>
+  * Default value is false
+
+Stop entire pipeline when encountering a non decodable message.
+
+[id="plugins-{type}s-{plugin}-pb3_encoder_autoconvert_types"]
+===== `pb3_encoder_autoconvert_types`
+
+  * Value type is <<boolean,boolean>>
+  * Default value is true
 
+Convert data types to match the protobuf definition (if possible).
+The protobuf encoder library is very strict with regards to data types. Example: an event has an integer field but the protobuf definition expects a float. This would lead to an exception and the event would be lost.
+This feature tries to convert the datatypes to the expectations of the protobuf definitions, without modifying the data whatsoever. Examples of conversions it might attempt:
+"true" string => true boolean
+17 int => 17.0 float
+12345 number => "12345" string
 

data/lib/logstash/codecs/protobuf.rb

@@ -92,8 +92,6 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
   # `class_file` and `include_path` cannot be used at the same time.
   config :class_file, :validate => :string, :default => '', :required => false
 
-  # Absolute path to the directory that contains all compiled protobuf files.
-  #
   # Absolute path to the root directory that contains all referenced/used dependencies
   # of the main class (`class_name`) or any of its dependencies.
   #
@@ -101,12 +99,12 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
   #
   # pb3
   #   ├── header
-  #   │   └── header_pb.rb
+  #   │   └── header_pb.rb
   #   ├── messageA_pb.rb
   #
   # In this case `messageA_pb.rb` has an embedded message from `header/header_pb.rb`.
   # If `class_file` is set to `messageA_pb.rb`, and `class_name` to
-  # `MessageA`, `protobuf_root_directory` must be set to `/path/to/pb3`. Which includes
+  # `MessageA`, `protobuf_root_directory` must be set to `/path/to/pb3`, which includes
   #  both definitions.
   config :protobuf_root_directory, :validate => :string, :required => false
 
@@ -128,12 +126,6 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
   # [source,ruby]
   # include_path => ['/path/to/protobuf/definitions/Wings.pb.rb','/path/to/protobuf/definitions/Unicorn.pb.rb']
   #
-  # When using the codec in an output plugin:
-  # * make sure to include all the desired fields in the protobuf definition, including timestamp.
-  #   Remove fields that are not part of the protobuf definition from the event by using the mutate filter.
-  # * the @ symbol is currently not supported in field names when loading the protobuf definitions for encoding. Make sure to call the timestamp field "timestamp"
-  #   instead of "@timestamp" in the protobuf file. Logstash event fields will be stripped of the leading @ before conversion.
-  #
   # `class_file` and `include_path` cannot be used at the same time.
   config :include_path, :validate => :array, :default => [], :required => false
 
@@ -145,6 +137,11 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
   # To tolerate faulty messages that cannot be decoded, set this to false. Otherwise the pipeline will stop upon encountering a non decipherable message.
   config :stop_on_error, :validate => :boolean, :default => false, :required => false
 
+  # Instruct the encoder to attempt converting data types to match the protobuf definitions. Available only for protobuf version 3.
+  config :pb3_encoder_autoconvert_types, :validate => :boolean, :default => true, :required => false
+
+
+
   attr_reader :execution_context
 
   # id of the pipeline whose events you want to read from.
@@ -156,6 +153,8 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
     @metainfo_messageclasses = {}
     @metainfo_enumclasses = {}
     @metainfo_pb2_enumlist = []
+    @pb3_typeconversion_tag = "_protobuf_type_converted"
+
 
     if @include_path.length > 0 and not class_file.strip.empty?
       raise LogStash::ConfigurationError, "Cannot use `include_path` and `class_file` at the same time"
@@ -174,7 +173,6 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
     end
 
     @class_file = "#{@protobuf_root_directory}/#{@class_file}" unless (Pathname.new @class_file).absolute? or @class_file.empty?
-
     # exclusive access while loading protobuf definitions
     Google::Protobuf::DescriptorPool.with_lock.synchronize do
       # load from `class_file`
@@ -184,6 +182,7 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
 
       if @protobuf_version == 3
         @pb_builder = Google::Protobuf::DescriptorPool.generated_pool.lookup(class_name).msgclass
+
       else
         @pb_builder = pb2_create_instance(class_name)
       end
@@ -222,11 +221,11 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
 
   def encode(event)
     if @protobuf_version == 3
-      protobytes = pb3_encode_wrapper(event)
+      protobytes = pb3_encode(event)
     else
-      protobytes = pb2_encode_wrapper(event)
+      protobytes = pb2_encode(event)
     end
-     @on_event.call(event, protobytes)
+    @on_event.call(event, protobytes)
   end # def encode
 
 
@@ -256,101 +255,278 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
     result
   end
 
-  def pb3_encode_wrapper(event)
-    data = pb3_encode(event.to_hash, @class_name)
-    pb_obj = @pb_builder.new(data)
+  def pb3_encode(event)
+
+    datahash = event.to_hash
+
+    is_recursive_call = !event.get('tags').nil? and event.get('tags').include? @pb3_typeconversion_tag
+    if is_recursive_call
+      datahash = pb3_remove_typeconversion_tag(datahash)
+    end
+    datahash = pb3_prepare_for_encoding(datahash)
+    if datahash.nil?
+      @logger.warn("Protobuf encoding error 4: empty data for event #{event.to_hash}")
+    end
+    if @pb_builder.nil?
+      @logger.warn("Protobuf encoding error 5: empty protobuf builder for class #{@class_name}")
+    end
+    pb_obj = @pb_builder.new(datahash)
     @pb_builder.encode(pb_obj)
+
   rescue ArgumentError => e
     k = event.to_hash.keys.join(", ")
-    @logger.debug("Encoding error 2. Probably mismatching protobuf definition. Required fields in the protobuf definition are: #{k} and the timestamp field name must not include an @.")
-    raise e
+    @logger.warn("Protobuf encoding error 1: Argument error (#{e.inspect}). Reason: probably mismatching protobuf definition. \
+      Required fields in the protobuf definition are: #{k} and fields must not begin with @ sign. The event has been discarded.")
+  rescue TypeError => e
+    pb3_handle_type_errors(event, e, is_recursive_call, datahash)
   rescue => e
-    @logger.debug("Couldn't generate protobuf: #{e.inspect}")
-    raise e
+    @logger.warn("Protobuf encoding error 3: #{e.inspect}. Event discarded. Input data: #{datahash}. The event has been discarded. Backtrace: #{e.backtrace}")
   end
 
 
-  def pb3_encode(datahash, class_name)
-    if datahash.is_a?(::Hash)
 
 
+  def pb3_handle_type_errors(event, e, is_recursive_call, datahash)
+    begin
+      if is_recursive_call
+        @logger.warn("Protobuf encoding error 2.1: Type error (#{e.inspect}). Some types could not be converted. The event has been discarded. Type mismatches: #{mismatches}.")
+      else
+        if @pb3_encoder_autoconvert_types
 
-      # Preparation: the data cannot be encoded until certain criteria are met:
-      # 1) remove @ signs from keys.
-      # 2) convert timestamps and other objects to strings
-      datahash = datahash.inject({}){|x,(k,v)| x[k.gsub(/@/,'').to_sym] = (should_convert_to_string?(v) ? v.to_s : v); x}
+          msg = "Protobuf encoding error 2.2: Type error (#{e.inspect}). Will try to convert the data types. Original data: #{datahash}"
+          @logger.warn(msg)
+          mismatches = pb3_get_type_mismatches(datahash, "", @class_name)
 
-      # Check if any of the fields in this hash are protobuf classes and if so, create a builder for them.
-      meta = @metainfo_messageclasses[class_name]
-      if meta
-        meta.map do | (field_name,class_name) |
-          key = field_name.to_sym
-          if datahash.include?(key)
-            original_value = datahash[key]
-            datahash[key] =
-              if original_value.is_a?(::Array)
-                # make this field an array/list of protobuf objects
-                # value is a list of hashed complex objects, each of which needs to be protobuffed and
-                # put back into the list.
-                original_value.map { |x| pb3_encode(x, class_name) }
-                original_value
-              else
-                r = pb3_encode(original_value, class_name)
-                builder = Google::Protobuf::DescriptorPool.generated_pool.lookup(class_name).msgclass
-                builder.new(r)
-              end # if is array
-          end # if datahash_include
+          msg = "Protobuf encoding info 2.2: Type mismatches found: #{mismatches}." # TODO remove
+          @logger.warn(msg)
+
+          event = pb3_convert_mismatched_types(event, mismatches)
+          # Add a (temporary) tag to handle the recursion stop
+          pb3_add_tag(event, @pb3_typeconversion_tag )
+          pb3_encode(event)
+        else
+          @logger.warn("Protobuf encoding error 2.3: Type error (#{e.inspect}). The event has been discarded. Try setting pb3_encoder_autoconvert_types => true for automatic type conversion.")
+        end
+      end
+    rescue TypeError => e
+      if @pb3_encoder_autoconvert_types
+        @logger.warn("Protobuf encoding error 2.4.1: (#{e.inspect}). Failed to convert data types. The event has been discarded. original data: #{datahash}")
+      else
+        @logger.warn("Protobuf encoding error 2.4.2: (#{e.inspect}). The event has been discarded.")
+      end
+    rescue => ex
+      @logger.warn("Protobuf encoding error 2.5: (#{e.inspect}). The event has been discarded. Auto-typecasting was on: #{@pb3_encoder_autoconvert_types}")
+    end
+  end # pb3_handle_type_errors
+
+  def pb3_get_type_mismatches(data, key_prefix, pb_class)
+    mismatches = []
+    data.to_hash.each do |key, value|
+        expected_type = pb3_get_expected_type(key, pb_class)
+        r = pb3_compare_datatypes(value, key, key_prefix, pb_class, expected_type)
+        mismatches.concat(r)
+    end # data.each
+    mismatches
+  end
+
+
+  def pb3_get_expected_type(key, pb_class)
+    pb_descriptor = Google::Protobuf::DescriptorPool.generated_pool.lookup(pb_class)
+
+    if !pb_descriptor.nil?
+      pb_builder = pb_descriptor.msgclass
+      pb_obj = pb_builder.new({})
+      v = pb_obj.send(key)
+
+      if !v.nil?
+        v.class
+      else
+        nil
+      end
+    end
+  end
+
+  def pb3_compare_datatypes(value, key, key_prefix, pb_class, expected_type)
+    mismatches = []
+
+    if value.nil?
+      is_mismatch = false
+    else
+      case value
+      when ::Hash, Google::Protobuf::MessageExts
+
+        is_mismatch = false
+        descriptor = Google::Protobuf::DescriptorPool.generated_pool.lookup(pb_class).lookup(key)
+        if descriptor.subtype != nil
+          class_of_nested_object = pb3_get_descriptorpool_name(descriptor.subtype.msgclass)
+          new_prefix = "#{key}."
+          recursive_mismatches = pb3_get_type_mismatches(value, new_prefix, class_of_nested_object)
+          mismatches.concat(recursive_mismatches)
+        end
+      when ::Array
+
+        expected_type = pb3_get_expected_type(key, pb_class)
+        is_mismatch = (expected_type != Google::Protobuf::RepeatedField)
+        child_type = Google::Protobuf::DescriptorPool.generated_pool.lookup(pb_class).lookup(key).type
+        value.each_with_index  do | v, i |
+          new_prefix = "#{key}."
+          recursive_mismatches = pb3_compare_datatypes(v, i.to_s, new_prefix, pb_class, child_type)
+          mismatches.concat(recursive_mismatches)
+          is_mismatch |= recursive_mismatches.any?
         end # do
-      end # if meta
+      else # is scalar data type
 
-      # Check if any of the fields in this hash are enum classes and if so, create a builder for them.
-      meta = @metainfo_enumclasses[class_name]
-      if meta
-        meta.map do | (field_name,class_name) |
-          key = field_name.to_sym
-          if datahash.include?(key)
-            original_value = datahash[key]
-            datahash[key] = case original_value
-            when ::Array
-              original_value.map { |x| pb3_encode(x, class_name) }
-              original_value
-            when Fixnum
-              original_value # integers will be automatically converted into enum
-            # else
-              # feature request: support for providing integers as strings or symbols.
-              # not fully tested yet:
-              # begin
-              #   enum_lookup_name = "#{class_name}::#{original_value}"
-              #   enum_lookup_name.split('::').inject(Object) do |mod, class_name|
-              #     mod.const_get(class_name)
-              #   end # do
-              # rescue => e
-              #   @logger.debug("Encoding error 3: could not translate #{original_value} into enum. #{e}")
-              #   raise e
-              # end
+        is_mismatch = ! pb3_is_scalar_datatype_match(expected_type, value.class)
+      end # if
+    end # if value.nil?
+
+    if is_mismatch
+      mismatches << {"key" => "#{key_prefix}#{key}", "actual_type" => value.class, "expected_type" => expected_type, "value" => value}
+    end
+    mismatches
+  end
+
+  def pb3_remove_typeconversion_tag(data)
+    # remove the tag that we added to the event because
+    # the protobuf definition might not have a field for tags
+    data['tags'].delete(@pb3_typeconversion_tag)
+    if data['tags'].length == 0
+      data.delete('tags')
+    end
+    data
+  end
+
+  def pb3_get_descriptorpool_name(child_class)
+    # make instance
+    inst = child_class.new
+    # get the lookup name for the Descriptorpool
+    inst.class.descriptor.name
+  end
+
+  def pb3_is_scalar_datatype_match(expected_type, actual_type)
+    if expected_type == actual_type
+      true
+    else
+      e = expected_type.to_s.downcase.to_sym
+      a = actual_type.to_s.downcase.to_sym
+      case e
+      # when :string, :integer
+      when :string
+          a == e
+      when :integer
+          a == e
+      when :float
+          a == :float || a == :integer
+      end
+    end
+  end
+
+
+  def pb3_convert_mismatched_types_getter(struct, key)
+    if struct.is_a? ::Hash
+      struct[key]
+    else
+      struct.get(key)
+    end
+  end
+
+  def pb3_convert_mismatched_types_setter(struct, key, value)
+    if struct.is_a? ::Hash
+      struct[key] = value
+    else
+      struct.set(key, value)
+    end
+    struct
+  end
+
+  def pb3_add_tag(event, tag )
+    if event.get('tags').nil?
+        event.set('tags', [tag])
+    else
+      existing_tags = event.get('tags')
+      event.set("tags", existing_tags << tag)
+    end
+  end
+
+  # Due to recursion on nested fields in the event object this method might be given an event (1st call) or a hash (2nd .. nth call)
+  # First call will be the event object, child objects will be hashes.
+  def pb3_convert_mismatched_types(struct, mismatches)
+    mismatches.each do | m |
+        key = m['key']
+        expected_type = m['expected_type']
+        actual_type = m['actual_type']
+        if key.include? "." # the mismatch is in a child object
+            levels = key.split(/\./) # key is something like http_user_agent.minor_version and needs to be splitted.
+            key = levels[0]
+            sub_levels = levels.drop(1).join(".")
+            new_mismatches = [{"key"=>sub_levels, "actual_type"=>m["actual_type"], "expected_type"=>m["expected_type"]}]
+            value = pb3_convert_mismatched_types_getter(struct, key)
+            new_value = pb3_convert_mismatched_types(value, new_mismatches)
+            struct = pb3_convert_mismatched_types_setter(struct, key, new_value )
+        else
+            value = pb3_convert_mismatched_types_getter(struct, key)
+            begin
+                case expected_type.to_s
+                when "Integer"
+                    case actual_type.to_s
+                    when "String"
+                        new_value = value.to_i
+                    when "Float"
+                        if value.floor == value # convert values like 2.0 to 2, but not 2.1
+                          new_value = value.to_i
+                        end
+                    end
+                when "String"
+                    new_value = value.to_s
+                when "Float"
+                    new_value = value.to_f
+                when "Boolean","TrueClass", "FalseClass"
+                    new_value = value.to_s.downcase == "true"
+                end
+                if !new_value.nil?
+                  struct = pb3_convert_mismatched_types_setter(struct, key, new_value )
+                end
+            rescue Exception => ex
+                @logger.debug("Protobuf encoding error 5: Could not convert types for protobuf encoding: #{ex}")
             end
-          end # if datahash_include
-        end # do
-      end # if meta
+        end # if key contains .
+    end # mismatches.each
+    struct
+  end
+
+  def pb3_prepare_for_encoding(datahash)
+    # 0) Remove empty fields.
+    datahash = datahash.select { |key, value| !value.nil? }
+
+    # Preparation: the data cannot be encoded until certain criteria are met:
+    # 1) remove @ signs from keys.
+    # 2) convert timestamps and other objects to strings
+    datahash = datahash.inject({}){|x,(k,v)| x[k.gsub(/@/,'').to_sym] = (should_convert_to_string?(v) ? v.to_s : v); x}
+
+    datahash.each do |key, value|
+      datahash[key] = pb3_prepare_for_encoding(value) if value.is_a?(Hash)
     end
+
     datahash
   end
 
-  def pb2_encode_wrapper(event)
-    data = pb2_encode(event.to_hash, @class_name)
+
+
+  def pb2_encode(event)
+    data = pb2_prepare_for_encoding(event.to_hash, @class_name)
     msg = @pb_builder.new(data)
     msg.serialize_to_string
   rescue NoMethodError => e
-    @logger.debug("Encoding error 2. Probably mismatching protobuf definition. Required fields in the protobuf definition are: " + event.to_hash.keys.join(", ") + " and the timestamp field name must not include a @. ")
+    @logger.warn("Encoding error 2. Probably mismatching protobuf definition. Required fields in the protobuf definition are: " + event.to_hash.keys.join(", ") + " and the timestamp field name must not include a @. ")
     raise e
   rescue => e
-    @logger.debug("Encoding error 1: #{e.inspect}")
+    @logger.warn("Encoding error 1: #{e.inspect}")
     raise e
   end
 
 
 
-  def pb2_encode(datahash, class_name)
+  def pb2_prepare_for_encoding(datahash, class_name)
     if datahash.is_a?(::Hash)
       # Preparation: the data cannot be encoded until certain criteria are met:
       # 1) remove @ signs from keys.
@@ -368,11 +544,11 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
                 # make this field an array/list of protobuf objects
                 # value is a list of hashed complex objects, each of which needs to be protobuffed and
                 # put back into the list.
-                original_value.map { |x| pb2_encode(x, c) }
+                original_value.map { |x| pb2_prepare_for_encoding(x, c) }
                 original_value
               else
                 proto_obj = pb2_create_instance(c)
-                proto_obj.new(pb2_encode(original_value, c)) # this line is reached in the colourtest for an enum. Enums should not be instantiated. Should enums even be in the messageclasses? I dont think so! TODO bug
+                proto_obj.new(pb2_prepare_for_encoding(original_value, c)) # this line is reached in the colourtest for an enum. Enums should not be instantiated. Should enums even be in the messageclasses? I dont think so! TODO bug
               end # if is array
           end # if datahash_include
         end # do
@@ -383,7 +559,7 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
 
 
   def should_convert_to_string?(v)
-    !(v.is_a?(Fixnum) || v.is_a?(::Hash) || v.is_a?(::Array) || [true, false].include?(v))
+    !(v.is_a?(Integer) || v.is_a?(Float) || v.is_a?(::Hash) || v.is_a?(::Array) || [true, false].include?(v))
   end
 
 
@@ -394,6 +570,7 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
 
 
   def pb3_metadata_analyis(filename)
+
     regex_class_name = /\s*add_message "(?<name>.+?)" do\s+/ # TODO optimize both regexes for speed (negative lookahead)
     regex_pbdefs = /\s*(optional|repeated)(\s*):(?<name>.+),(\s*):(?<type>\w+),(\s*)(?<position>\d+)(, \"(?<enum_class>.*?)\")?/
     class_name = ""
@@ -493,6 +670,7 @@ class LogStash::Codecs::Protobuf < LogStash::Codecs::Base
           require filename
         rescue Exception => e
           @logger.error("Unable to load file: #{filename}. Reason: #{e.inspect}")
+          raise e
         end
       end