logstash-codec-netflow 3.8.0 → 3.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0d51a96934f6215fa848ef5826f1da03ee4bf555
-  data.tar.gz: 01285e9dad3666666178a98dd50af9275fedb43b
+  metadata.gz: 5b85cd8c1849c1914c49727e17a1775257aee259
+  data.tar.gz: fb97641f2b79c06cc380c27eba5933f40dfc8c87
 SHA512:
-  metadata.gz: a9e15da9958d4435051ba14da84768bd3db6b96ce0c8d7c80de3848ac2bfbcc5205f8d0cfd0415668447f3481fdedae3b5cea15bd9747124ac4aecbd648cc692
-  data.tar.gz: 1064815fd03ec5d7439f2fd3df9a9f50421742b8af83d002e62a3553005eb8cb1fa00ad1a43a8b19754fc2dd019bd963c35bb200a7b37ef6dbd8eea8a579c43f
+  metadata.gz: 2ee8918200546ef2e9d454eadf84bce73abab9fc24c268ce2c90808ddc1c4dba70d8503bad0213b774e71385df5a00a986c738e0abea1105afa9f842e5ed66ce
+  data.tar.gz: 578e1d35533e5053a4b43189bc2581a3e511ae0d152974fa4f3d8bb7aa80c35bc2f6dad837b433b905a7de8c7d644bfb9d81e6efd39b760713bcb82683aaf882

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 3.8.1
+
+  - Prevent Netflow and IPFIX templates from being modified concurrently
+  - Improved Palo Alto support and added rspec test
+
 ## 3.8.0
 
   - Added initial YAF support with applabel and silk (but without DPI plugins because of complex data types)

data/RFC_COMPLIANCE_IPFIX.md

@@ -71,53 +71,6 @@ Summary of collector-relevant requirements implemented versus the total collecto
 | subTemplateList       | No      |     |
 | subTemplateMultiList  | No      |     |
 
-## RFC7012 Information Elements support details
-
-IE 1-433 are supported
-
-These are not yet supported:
-
-|id | name                | data type
-|---|---------------------|-------------------------
-|434|mibObjectValueInteger|signed32
-|435|mibObjectValueOctetString|octetArray
-|436|mibObjectValueOID|octetArray
-|437|mibObjectValueBits|octetArray
-|438|mibObjectValueIPAddress|ipv4Address
-|439|mibObjectValueCounter|unsigned64
-|440|mibObjectValueGauge|unsigned32
-|441|mibObjectValueTimeTicks|unsigned32
-|442|mibObjectValueUnsigned|unsigned32
-|443|mibObjectValueTable|subTemplateList
-|444|mibObjectValueRow|subTemplateList
-|445|mibObjectIdentifier|octetArray
-|446|mibSubIdentifier|unsigned32
-|447|mibIndexIndicator|unsigned64
-|448|mibCaptureTimeSemantics|unsigned8
-|449|mibContextEngineID|octetArray
-|450|mibContextName|string
-|451|mibObjectName|string
-|452|mibObjectDescription|string
-|453|mibObjectSyntax|string
-|454|mibModuleName|string
-|455|mobileIMSI|string
-|456|mobileMSISDN|string
-|457|httpStatusCode|unsigned16
-|458|sourceTransportPortsLimit|unsigned16
-|459|httpRequestMethod|string
-|460|httpRequestHost|string
-|461|httpRequestTarget|string
-|462|httpMessageVersion|string
-|463|natInstanceID|unsigned32
-|464|internalAddressRealm|octetArray
-|465|externalAddressRealm|octetArray
-|466|natQuotaExceededEvent|unsigned32
-|467|natThresholdEvent|unsigned32
-|468|httpUserAgent|string
-|469|httpContentType|string
-|470|httpReasonPhrase|string
-
- 
 ## RFC 7011 collector compliance details
 
 The tables below detail the collector-relevant requirements, and whether or not they are implemented:
@@ -228,3 +181,51 @@ The tables below detail the collector-relevant requirements, and whether or not
 | 11.7 As IPFIX uses length-prefix encodings, Collector implementors should take care to ensure the detection of inconsistent values that could impact IPFIX Message decoding, and proper operation in the presence of such inconsistent values. | | | YES |
 | 11.7 Specifically, IPFIX Message, Set, and variable-length Information  Element lengths must be checked for consistency to avoid buffer-sizing vulnerabilities. | | | YES |
 
+
+## RFC7012 Information Elements support details
+
+IE 1-433 are supported
+
+These are not yet supported:
+
+|id | name                | data type
+|---|---------------------|-------------------------
+|434|mibObjectValueInteger|signed32
+|435|mibObjectValueOctetString|octetArray
+|436|mibObjectValueOID|octetArray
+|437|mibObjectValueBits|octetArray
+|438|mibObjectValueIPAddress|ipv4Address
+|439|mibObjectValueCounter|unsigned64
+|440|mibObjectValueGauge|unsigned32
+|441|mibObjectValueTimeTicks|unsigned32
+|442|mibObjectValueUnsigned|unsigned32
+|443|mibObjectValueTable|subTemplateList
+|444|mibObjectValueRow|subTemplateList
+|445|mibObjectIdentifier|octetArray
+|446|mibSubIdentifier|unsigned32
+|447|mibIndexIndicator|unsigned64
+|448|mibCaptureTimeSemantics|unsigned8
+|449|mibContextEngineID|octetArray
+|450|mibContextName|string
+|451|mibObjectName|string
+|452|mibObjectDescription|string
+|453|mibObjectSyntax|string
+|454|mibModuleName|string
+|455|mobileIMSI|string
+|456|mobileMSISDN|string
+|457|httpStatusCode|unsigned16
+|458|sourceTransportPortsLimit|unsigned16
+|459|httpRequestMethod|string
+|460|httpRequestHost|string
+|461|httpRequestTarget|string
+|462|httpMessageVersion|string
+|463|natInstanceID|unsigned32
+|464|internalAddressRealm|octetArray
+|465|externalAddressRealm|octetArray
+|466|natQuotaExceededEvent|unsigned32
+|467|natThresholdEvent|unsigned32
+|468|httpUserAgent|string
+|469|httpContentType|string
+|470|httpReasonPhrase|string
+
+ 

data/RFC_COMPLIANCE_NETFLOW_v9.md

@@ -26,6 +26,47 @@ Summary of collector-relevant requirements implemented versus the total collecto
 | 9. The collector side                        | 5/5 | 0/3 |     |
 | 10. Security considerations                  |     |     |     |
 
+## RFC 3954 collector compliance details
+
+The tables below detail the collector-relevant requirements, and whether or not they are implemented:
+
+### 5. Export packet format
+
+| Requirement                           |MUST |SHOULD| MAY| 
+|---------------------------------------|-----|-----|-----|
+| 5.1 Incremental sequence counter of all Export Packets sent from the current Observation Domain by the Exporter.  This value MUST be cumulative, and SHOULD be used by the Collector to identify whether any Export Packets have been missed. | | NO |  |
+| 5.1 NetFlow Collectors SHOULD use the combination of the source IP  address and the Source ID field to separate different export streams originating from the same Exporter. | | NO | |
+| 5.3 The Collector MUST use the FlowSet ID to find the corresponding Template Record and decode the Flow Records from the FlowSet. | YES | | |
+
+### 6. Options
+
+| Requirement                           |MUST |SHOULD| MAY| 
+|---------------------------------------|-----|-----|-----|
+| 6.2 The Collector MUST use the FlowSet ID to map the appropriate type and length to any field values that follow. | YES | | |
+
+### 7. Template management
+
+| Requirement                           |MUST |SHOULD| MAY| 
+|---------------------------------------|-----|-----|-----|
+| 7. the NetFlow Collector MUST store the Template Record to interpret the corresponding Flow Data Records that are received in subsequent data packets. | YES | | |
+| 7.  A NetFlow Collector that receives Export Packets from several Observation Domains from the same Exporter MUST be aware that the uniqueness of the Template ID is not guaranteed across Observation Domains. | YES | | | 
+| 7.  If a Collector should receive a new definition for an already existing Template ID, it MUST discard the previous template definition and use the new one. | YES | | |
+
+### 9. The collector side
+
+| Requirement                           |MUST |SHOULD| MAY| 
+|---------------------------------------|-----|-----|-----|
+| 9. If the Template Records have not been received at the time Flow Data Records (or Options Data Records) are received, the Collector SHOULD store the Flow Data Records (or Options Data Records) and decode them after the Template Records are received. | | NO | |
+| 9. A Collector device MUST NOT assume that the Data FlowSet and the associated Template FlowSet (or Options Template FlowSet) are exported in the same Export Packet. | YES | | | 
+| 9. The Collector MUST NOT assume that one and only one Template FlowSet is present in an Export Packet. | YES | | | 
+| 9. The Collector MUST NOT attempt to decode the Flow or Options Data Records with an expired Template. | YES | | | 
+| 9. At any given time the Collector SHOULD maintain the following for all the current Template Records and Options Template Records: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
+| 9. In the event of a clock configuration change on the Exporter, the  Collector SHOULD discard all Template Records and Options Template  Records associated with that Exporter, in order for Collector to learn the new set of fields: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
+| 9. If the Collector receives a new Template Record (for example, in the case of an Exporter restart) it MUST immediately override the existing Template Record. | YES | | |
+| 9. Finally, note that the Collector MUST accept padding in the Data  FlowSet and Options Template FlowSet, which means for the Flow Data Records, the Options Data Records and the Template Records. | YES | | |
+
+
+
 ## RFC 3954 Information Elements support details
 
 From the IEs 1-127, these are not yet supported:
@@ -244,7 +285,6 @@ From the IEs 128-, these are not yet supported:
 |343|informationElementRangeEnd|unsigned64
 |344|informationElementSemantics|unsigned8
 |345|informationElementUnits|unsigned16
-|346|privateEnterpriseNumber|unsigned32
 |347|virtualStationInterfaceId|octetArray
 |348|virtualStationInterfaceName|string
 |349|virtualStationUUID|octetArray
@@ -365,43 +405,3 @@ From the IEs 128-, these are not yet supported:
 |469|httpContentType|string
 |470|httpReasonPhrase|string
 
-## RFC 3954 collector compliance details
-
-The tables below detail the collector-relevant requirements, and whether or not they are implemented:
-
-### 5. Export packet format
-
-| Requirement                           |MUST |SHOULD| MAY| 
-|---------------------------------------|-----|-----|-----|
-| 5.1 Incremental sequence counter of all Export Packets sent from the current Observation Domain by the Exporter.  This value MUST be cumulative, and SHOULD be used by the Collector to identify whether any Export Packets have been missed. | | NO |  |
-| 5.1 NetFlow Collectors SHOULD use the combination of the source IP  address and the Source ID field to separate different export streams originating from the same Exporter. | | NO | |
-| 5.3 The Collector MUST use the FlowSet ID to find the corresponding Template Record and decode the Flow Records from the FlowSet. | YES | | |
-
-### 6. Options
-
-| Requirement                           |MUST |SHOULD| MAY| 
-|---------------------------------------|-----|-----|-----|
-| 6.2 The Collector MUST use the FlowSet ID to map the appropriate type and length to any field values that follow. | YES | | |
-
-### 7. Template management
-
-| Requirement                           |MUST |SHOULD| MAY| 
-|---------------------------------------|-----|-----|-----|
-| 7. the NetFlow Collector MUST store the Template Record to interpret the corresponding Flow Data Records that are received in subsequent data packets. | YES | | |
-| 7.  A NetFlow Collector that receives Export Packets from several Observation Domains from the same Exporter MUST be aware that the uniqueness of the Template ID is not guaranteed across Observation Domains. | YES | | | 
-| 7.  If a Collector should receive a new definition for an already existing Template ID, it MUST discard the previous template definition and use the new one. | YES | | |
-
-### 9. The collector side
-
-| Requirement                           |MUST |SHOULD| MAY| 
-|---------------------------------------|-----|-----|-----|
-| 9. If the Template Records have not been received at the time Flow Data Records (or Options Data Records) are received, the Collector SHOULD store the Flow Data Records (or Options Data Records) and decode them after the Template Records are received. | | NO | |
-| 9. A Collector device MUST NOT assume that the Data FlowSet and the associated Template FlowSet (or Options Template FlowSet) are exported in the same Export Packet. | YES | | | 
-| 9. The Collector MUST NOT assume that one and only one Template FlowSet is present in an Export Packet. | YES | | | 
-| 9. The Collector MUST NOT attempt to decode the Flow or Options Data Records with an expired Template. | YES | | | 
-| 9. At any given time the Collector SHOULD maintain the following for all the current Template Records and Options Template Records: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
-| 9. In the event of a clock configuration change on the Exporter, the  Collector SHOULD discard all Template Records and Options Template  Records associated with that Exporter, in order for Collector to learn the new set of fields: Exporter, Observation Domain, Template ID, Template Definition, Last Received. | | NO | |
-| 9. If the Collector receives a new Template Record (for example, in the case of an Exporter restart) it MUST immediately override the existing Template Record. | YES | | |
-| 9. Finally, note that the Collector MUST accept padding in the Data  FlowSet and Options Template FlowSet, which means for the Flow Data Records, the Options Data Records and the Template Records. | YES | | |
-
-

data/docs/index.asciidoc

@@ -51,8 +51,10 @@ The following Netflow/IPFIX exporters are known to work with the most recent ver
 |OpenBSD pflow            |  y | n  |   y   | http://man.openbsd.org/OpenBSD-current/man4/pflow.4
 |Softflowd                |  y | y  |   y   | IPFIX supported in https://github.com/djmdjm/softflowd
 |Streamcore Streamgroomer |    | y  |       |
+|Palo Alto PAN-OS         |    | y  |       |
 |Ubiquiti Edgerouter X    |    | y  |       | With MPLS labels
 |VMware VDS               |    |    |   y   | Still some unknown fields
+|YAF                      |    |    |   y   | With silk and applabel, but no DPI plugin support
 |===========================================================================================
 
 ==== Usage

data/lib/logstash/codecs/netflow/netflow.yaml

@@ -331,6 +331,9 @@
 323:
 - 8
 - :event_time_msec
+346:
+- :uint32
+- :privateEnterpriseNumber
 361:
 - :uint16
 - :postNATPortBlockStart

data/lib/logstash/codecs/netflow.rb

@@ -51,7 +51,9 @@ class LogStash::Codecs::Netflow < LogStash::Codecs::Base
 
   def initialize(params = {})
     super(params)
-    @threadsafe = false
+    @threadsafe = true
+    @decode_mutex_netflow = Mutex.new
+    @decode_mutex_ipfix = Mutex.new
   end
 
   def register
@@ -212,17 +214,20 @@ class LogStash::Codecs::Netflow < LogStash::Codecs::Base
           else
             key = "#{flowset.source_id}|#{template.template_id}"
           end
-          @netflow_templates[key, @cache_ttl] = BinData::Struct.new(:endian => :big, :fields => fields)
-          @logger.debug("Received template #{template.template_id} with fields #{fields.inspect}")
-          @logger.debug("Received template #{template.template_id} of size #{template_length} bytes. Representing in #{@netflow_templates[key].num_bytes} BinData bytes")
-          if template_length != @netflow_templates[key].num_bytes
-            @logger.warn("Received template #{template.template_id} of size #{template_length} bytes doesn't match BinData representation we built (#{@netflow_templates[key].num_bytes} bytes)")
-          end
-          # Purge any expired templates
-          @netflow_templates.cleanup!
-          if @cache_save_path
-            @netflow_templates_cache[key] = fields
-            save_templates_cache(@netflow_templates_cache, "#{@cache_save_path}/netflow_templates.cache")
+          # Prevent netflow_templates array from being concurrently modified
+          @decode_mutex_netflow.synchronize do
+            @netflow_templates[key, @cache_ttl] = BinData::Struct.new(:endian => :big, :fields => fields)
+            @logger.debug("Received template #{template.template_id} with fields #{fields.inspect}")
+            @logger.debug("Received template #{template.template_id} of size #{template_length} bytes. Representing in #{@netflow_templates[key].num_bytes} BinData bytes")
+            if template_length != @netflow_templates[key].num_bytes
+              @logger.warn("Received template #{template.template_id} of size #{template_length} bytes doesn't match BinData representation we built (#{@netflow_templates[key].num_bytes} bytes)")
+            end
+            # Purge any expired templates
+            @netflow_templates.cleanup!
+            if @cache_save_path
+              @netflow_templates_cache[key] = fields
+              save_templates_cache(@netflow_templates_cache, "#{@cache_save_path}/netflow_templates.cache")
+            end
           end
         end
       end
@@ -316,12 +321,15 @@ class LogStash::Codecs::Netflow < LogStash::Codecs::Base
           end
           # FIXME Source IP address required in key
           key = "#{flowset.observation_domain_id}|#{template.template_id}"
-          @ipfix_templates[key, @cache_ttl] = BinData::Struct.new(:endian => :big, :fields => fields)
-          # Purge any expired templates
-          @ipfix_templates.cleanup!
-          if @cache_save_path
-            @ipfix_templates_cache[key] = fields
-            save_templates_cache(@ipfix_templates_cache, "#{@cache_save_path}/ipfix_templates.cache")
+          # Prevent ipfix_templates array from being concurrently modified
+          @decode_mutex_ipfix.synchronize do
+            @ipfix_templates[key, @cache_ttl] = BinData::Struct.new(:endian => :big, :fields => fields)
+            # Purge any expired templates
+            @ipfix_templates.cleanup!
+            if @cache_save_path
+              @ipfix_templates_cache[key] = fields
+              save_templates_cache(@ipfix_templates_cache, "#{@cache_save_path}/ipfix_templates.cache")
+            end
           end
         end
       end

data/logstash-codec-netflow.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-codec-netflow'
-  s.version         = '3.8.0'
+  s.version         = '3.8.1'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Reads Netflow v5 and Netflow v9 data"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

data/spec/codecs/netflow_spec.rb

@@ -1967,6 +1967,60 @@ describe LogStash::Codecs::Netflow do
     end
   end
 
+  context "Netflow 9 Palo Alto PAN-OS with app-id" do
+    let(:data) do
+      packets = []
+      packets << IO.read(File.join(File.dirname(__FILE__), "netflow9_test_paloalto_panos_tpl.dat"), :mode => "rb")
+      packets << IO.read(File.join(File.dirname(__FILE__), "netflow9_test_paloalto_panos_data.dat"), :mode => "rb")
+    end
+
+    let(:json_events) do
+      events = []
+      events << <<-END
+        {
+          "netflow": {
+            "output_snmp": 23,
+            "icmp_type": 0,
+            "in_pkts": 1,
+            "src_tos": 0,
+            "ipv4_dst_addr": "162.115.24.30",
+            "first_switched": "2017-11-13T14:33:53.000Z",
+            "flowset_id": 257,
+            "l4_src_port": 39702,
+            "fw_event": 5,
+            "version": 9,
+            "flow_seq_num": 207392627,
+            "ipv4_src_addr": "10.32.105.103",
+            "in_bytes": 111,
+            "protocol": 6,
+            "tcp_flags": 26,
+            "input_snmp": 24,
+            "last_switched": "2017-11-13T14:39:32.000Z",
+            "user_id": "",
+            "conn_id": 415347,
+            "privateEnterpriseNumber": 25461,
+            "l4_dst_port": 443,
+            "app_id": "ssl",
+            "direction": 0
+          },
+          "@version": "1",
+          "@timestamp": "2017-11-13T14:39:31.000Z"
+        }
+        END
+      events.map{|event| event.gsub(/\s+/, "")}
+    end
+
+    it "should decode raw data" do
+      expect(decode.size).to eq(8)
+      expect(decode[7].get("[netflow][app_id]")).to eq("incomplete")
+    end
+
+    it "should serialize to json" do
+      expect(JSON.parse(decode[1].to_json)).to eq(JSON.parse(json_events[0]))
+    end
+  end
+
+
   context "IPFIX Barracuda firewall" do
     let(:data) do
       packets = []

data/spec/codecs/netflow_stress.py

@@ -0,0 +1,30 @@
+import socket
+import sys
+import time
+import random
+
+## Standalone Netflow v9 stressor
+## Used to reproduce issue 91 https://github.com/logstash-plugins/logstash-codec-netflow/issues/91
+
+host = 'host02'  
+port = 2055
+
+tpl = '\x00\t\x00\x01e\x9c\xc0_XF\x8eU\x01u\xc7\x03\x00\x00\x08\x81\x00\x00\x00d\x01\x04\x00\x17\x00\x02\x00\x04\x00\x01\x00\x04\x00\x08\x00\x04\x00\x0c\x00\x04\x00\n\x00\x04\x00\x0e\x00\x04\x00\x15\x00\x04\x00\x16\x00\x04\x00\x07\x00\x02\x00\x0b\x00\x02\x00\x10\x00\x04\x00\x11\x00\x04\x00\x12\x00\x04\x00\t\x00\x01\x00\r\x00\x01\x00\x04\x00\x01\x00\x06\x00\x01\x00\x05\x00\x01\x00=\x00\x01\x00Y\x00\x01\x000\x00\x02\x00\xea\x00\x04\x00\xeb\x00\x04'
+
+# 21 flows:
+data = '\x00\t\x00\x15e\x9c\xbcqXF\x8eT\x01u\xc6\xa1\x00\x00\x08\x81\x01\x04\x05\\\x00\x00\x00\x01\x00\x00\x00(\n\x00\t\x92\n\x00\x1fQ\x00\x00\x00n\x00\x00\x00\x9ee\x9cG\x05e\x9cG\x05\xd3\x01\x01\xbb\x00\x00\x00\x00\x00\x00\xfb\xf0\n\x00\x0e!\x10\x14\x06\x10\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00h\n\x00\x11*\n\x00#\x04\x00\x00\x00W\x00\x00\x00\x9ee\x9cI\x88e\x9cG\x07\x8e\x84\x01\xbb\x00\x00\x00\x00\x00\x00\xfb\xf0\n\x00\x0e!\x15\x10\x06\x10\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x004\n\x00\x16o\n\x00"\x8d\x00\x00\x00h\x00\x00\x00\x9ee\x9cG\ne\x9cG\nA\xae\x01\xbb\x00\x00\x00\x00\x00\x00\xfb\xf0\n\x00\x0e!\x18\x10\x06\x11\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x01\xb3\n\x00\x17;\n\x00$\xaa\x00\x00\x00V\x00\x00\x00\x9ee\x9cG\x0ce\x9cG\x0c\x005\xfd,\x00\x00\x00\x00\x00\x00\xfb\xf1\n\x00\x0e\x1f\x19\x13\x11\x00\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x03\xc9\n\x00"G\n\x00\x14\xf2\x00\x00\x00\x9e\x00\x00\x00je\x9cG\re\x9cG\r\x01\xbb\x07\xdd\x00\x00\xfb\xf0\x00\x00\xff\xa2\n\x00\x12\x05\x10\x15\x06\x18\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00h\n\x00\n\x85\n\x00\x1ef\x00\x00\x00n\x00\x00\x00\x9ee\x9cG\re\x9cF\xba\x89\xc9\x00P\x00\x00\x00\x00\x00\x00\xfb\xf0\n\x00\x0e!\x10\x10\x06\x10\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x004\n\x00%\x1d\n\x00\x06\x18\x00\x00\x00f\x00\x00\x00\xa2e\x9cG\x10e\x9cG\x10\x00P\xdd\xc3\x00\x00;\x1d\x00\x00\xff\x97\n\x00\x00\xf2\x18\x10\x06\x10 \x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x02f\n\x00 \xb0\n\x00\x0bq\x00\x00\x00\x9e\x00\x00\x00.e\x9cG\x10e\x9cG\x10\x01\xbb\xdd\xfe\x00\x00\xfb\xf0\x00\x00\xff\x98\n\x00\x12i\x14\x10\x06\x18\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x03\x00\x00\x10\xfe\n\x00\x0c\x15\n\x00\x0f&\x00\x00\x00W\x00\x00\x00\x9ee\x9cG\x11e\x9c1\xe7\x01\xbb\x9c\x8e\x00\x00\x80\xa6\x00\x00\xfb\xf2\n\x00\x0e\x1b\x18\x18\x06\x10\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x02\x00\x00\x02\x15\n\x00\x04\xd4\n\x00\x03n\x00\x00\x00\xa2\x00\x00\x00fe\x9cT\x07e\x9cG\x12\xc6\x03\x01\xbb\x00\x00\xff\x97\x00\x00\x00F\n\x00\x10e\x10\x11\x06\x18\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x01E\x00\x005\\\n\x00!z\n\x00\x01\x88\x00\x00\x00\x9e\x00\x00\x00he\x9co\xd0e\x9c"\x1a\xe5\xbe\x00P\x00\x00\xfb\xf1\x00\x00\x00\x00\x00\x00\x00\x00\x15\x1b\x06\x10\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00Y\n\x00\x14\xf2\n\x00"G\x00\x00\x00j\x00\x00\x00\x9ee\x9cG\x14e\x9cG\x14\x07\xdd\x01\xbb\x00\x00\xff\xa2\x00\x00\xfb\xf0\n\x00\x0e!\x15\x10\x06\x18`\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x03A\n\x00\r\x19\n\x00\x0f&\x00\x00\x00W\x00\x00\x00\x9ee\x9cG\x16e\x9cG\x16\x01\xbb\xc9\xa5\x00\x00\x80\xa6\x00\x00\xfb\xf2\n\x00\x0e\x1b\x18\x18\x06\x18\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x02\x00\x00\x06Y\n\x00\x19;\n\x00\x02\x12\x00\x00\x00\x9e\x00\x00\x00ne\x9cG\x18e\x9cF\xbf\x01\xbb\xf4\x00\x00\x00\xfb\xf0\x00\x00\xff\x9d\n\x00\x12~\x10\x10\x06\x18\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00a\x00\x02+h\n\x00\x07I\n\x00\x1b\xa8\x00\x00\x00V\x00\x00\x00\x9ee\x9cu\xabe\x9c1\xfe\xeb\x98\x01\xd1\x00\x00\xff\x9c\x00\x00\xfb\xf0\n\x00\x0e!\x10\x10\x06\x18\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00:\x00\x00\x0b\xc8\n\x00\x132\n\x00\x1b\xa9\x00\x00\x00j\x00\x00\x00\x9ee\x9cO\xcbe\x9cE:\x86\x94\x03\xe3\x00\x00\xff\xb7\x00\x00\xfb\xf0\n\x00\x0e!\x12\x10\x06\x10\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x15\x00\x00{\x0c\n\x00\x1c\x96\n\x00\x18\r\x00\x00\x00\x9e\x00\x00\x00he\x9cHYe\x9cF\xf0\x01\xbb\xc2\xfd\x00\x00\xfb\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x10\x19\x06\x10\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x03\x00\x00\x0bg\n\x00\x1a\xbc\n\x00\x15\xc8\x00\x00\x00\x9e\x00\x00\x00We\x9cGfe\x9cE\xec\x03\xe1\xc4N\x00\x00\xfb\xf0\x00\x00\x00\x00\x00\x00\x00\x00\x10\x19\x06\x18\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x05\x00\x00\x11\xa2\n\x00\x1d"\n\x00\x0f&\x00\x00\x00K\x00\x00\x00\x9ee\x9cm`e\x9cA\xfe\x01\xbb\x8c\x8f\x00\x00;A\x00\x00\xfb\xf2\n\x00\x0e\x1b\x18\x18\x06\x18\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x01\x00\x00\x01F\n\x00\x08\xc8\n\x00\x05\xe0\x00\x00\x00f\x00\x00\x00\xa2e\x9cG\x1de\x9cG\x1dZX\xc9\xd7\x00\x00\x03\x15\x00\x00\xff\x97\n\x00\x00\xf2\x10\x10\x06\x18\x00\x00@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00p\n\x00\x1d.\n\x00\x0f&\x00\x00\x00K\x00\x00\x00\x9ee\x9cG\x1de\x9c@\xea\x01\xbb\xcc\x8c\x00\x00;A\x00\x00\xfb\xf2\n\x00\x0e\x1b\x18\x18\x06\x12\x00\x01@\x00\x01`\x00\x00\x00`\x00\x00\x00\x00\x00\x00'
+
+
+sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) 
+
+print("NETFLOW v9: sending 1 template 1 data packet in an infinite loop")
+
+duration = 0.0
+while True:
+    for i in range(0,400):
+        sock.sendto(tpl, (host, port))
+        sock.sendto(data, (host, port))
+    sys.stdout.write('.')
+    sys.stdout.flush()
+    time.sleep(random.random())
+print

metadata

@@ -1,16 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: logstash-codec-netflow
 version: !ruby/object:Gem::Version
-  version: 3.8.0
+  version: 3.8.1
 platform: ruby
 authors:
 - Elastic
-autorequire:
+autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-11-11 00:00:00.000000000 Z
+date: 2017-11-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -19,9 +20,8 @@ dependencies:
     - - <=
       - !ruby/object:Gem::Version
         version: '2.99'
-  name: logstash-core-plugin-api
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -31,86 +31,81 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.99'
 - !ruby/object:Gem::Dependency
+  name: bindata
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: 1.5.0
-  name: bindata
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: 1.5.0
 - !ruby/object:Gem::Dependency
+  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: 1.0.0
-  name: logstash-devutils
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: 1.0.0
-description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
+description: This gem is a Logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
+  gem is not a stand-alone program
 email: info@elastic.co
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- CHANGELOG.md
-- CONTRIBUTORS
-- Gemfile
-- LICENSE
-- NOTICE.TXT
-- README.md
-- RFC_COMPLIANCE_IPFIX.md
-- RFC_COMPLIANCE_NETFLOW_v9.md
-- docs/index.asciidoc
-- lib/logstash/codecs/netflow.rb
 - lib/logstash/codecs/netflow/iana2yaml.rb
 - lib/logstash/codecs/netflow/ipfix.yaml
 - lib/logstash/codecs/netflow/netflow.yaml
 - lib/logstash/codecs/netflow/util.rb
-- logstash-codec-netflow.gemspec
+- lib/logstash/codecs/netflow.rb
 - spec/codecs/ipfix.dat
-- spec/codecs/ipfix_test_barracuda_data256.dat
-- spec/codecs/ipfix_test_barracuda_tpl.dat
-- spec/codecs/ipfix_test_mikrotik_data258.dat
-- spec/codecs/ipfix_test_mikrotik_data259.dat
-- spec/codecs/ipfix_test_mikrotik_tpl.dat
-- spec/codecs/ipfix_test_netscaler_data.dat
-- spec/codecs/ipfix_test_netscaler_tpl.dat
 - spec/codecs/ipfix_test_openbsd_pflow_data.dat
 - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
-- spec/codecs/ipfix_test_vmware_vds_data264.dat
-- spec/codecs/ipfix_test_vmware_vds_data266.dat
-- spec/codecs/ipfix_test_vmware_vds_data266_267.dat
-- spec/codecs/ipfix_test_vmware_vds_tpl.dat
-- spec/codecs/ipfix_test_yaf_data45841.dat
-- spec/codecs/ipfix_test_yaf_data45873.dat
-- spec/codecs/ipfix_test_yaf_data53248.dat
-- spec/codecs/ipfix_test_yaf_tpl45841.dat
-- spec/codecs/ipfix_test_yaf_tpls_option_tpl.dat
 - spec/codecs/netflow5.dat
 - spec/codecs/netflow5_test_invalid01.dat
 - spec/codecs/netflow5_test_invalid02.dat
 - spec/codecs/netflow5_test_juniper_mx80.dat
 - spec/codecs/netflow5_test_microtik.dat
-- spec/codecs/netflow9_cisco_asr1001x_tpl259.dat
-- spec/codecs/netflow9_test_0length_fields_tpl_data.dat
-- spec/codecs/netflow9_test_cisco_1941K9.dat
 - spec/codecs/netflow9_test_cisco_asa_1_data.dat
 - spec/codecs/netflow9_test_cisco_asa_1_tpl.dat
 - spec/codecs/netflow9_test_cisco_asa_2_data.dat
 - spec/codecs/netflow9_test_cisco_asa_2_tpl_26x.dat
 - spec/codecs/netflow9_test_cisco_asa_2_tpl_27x.dat
+- spec/codecs/netflow9_test_invalid01.dat
+- spec/codecs/netflow9_test_macaddr_data.dat
+- spec/codecs/netflow9_test_macaddr_tpl.dat
+- spec/codecs/netflow9_test_nprobe_data.dat
+- spec/codecs/netflow9_test_nprobe_tpl.dat
+- spec/codecs/netflow9_test_softflowd_tpl_data.dat
+- spec/codecs/netflow9_test_valid01.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
+- spec/codecs/ipfix_test_netscaler_data.dat
+- spec/codecs/ipfix_test_netscaler_tpl.dat
+- spec/codecs/ipfix_test_vmware_vds_data264.dat
+- spec/codecs/ipfix_test_vmware_vds_data266.dat
+- spec/codecs/ipfix_test_vmware_vds_data266_267.dat
+- spec/codecs/ipfix_test_vmware_vds_tpl.dat
+- spec/codecs/ipfix_test_barracuda_data256.dat
+- spec/codecs/ipfix_test_barracuda_tpl.dat
+- spec/codecs/ipfix_test_mikrotik_data258.dat
+- spec/codecs/ipfix_test_mikrotik_data259.dat
+- spec/codecs/ipfix_test_mikrotik_tpl.dat
+- spec/codecs/netflow9_test_0length_fields_tpl_data.dat
 - spec/codecs/netflow9_test_cisco_asr9k_data256.dat
 - spec/codecs/netflow9_test_cisco_asr9k_data260.dat
 - spec/codecs/netflow9_test_cisco_asr9k_opttpl256.dat
@@ -121,34 +116,44 @@ files:
 - spec/codecs/netflow9_test_cisco_nbar_data262.dat
 - spec/codecs/netflow9_test_cisco_nbar_opttpl260.dat
 - spec/codecs/netflow9_test_cisco_nbar_tpl262.dat
-- spec/codecs/netflow9_test_cisco_wlc_8510_tpl_262.dat
 - spec/codecs/netflow9_test_cisco_wlc_data261.dat
 - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_data257.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_tpl.dat
-- spec/codecs/netflow9_test_invalid01.dat
 - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
-- spec/codecs/netflow9_test_macaddr_data.dat
-- spec/codecs/netflow9_test_macaddr_tpl.dat
-- spec/codecs/netflow9_test_nprobe_data.dat
 - spec/codecs/netflow9_test_nprobe_dpi.dat
-- spec/codecs/netflow9_test_nprobe_tpl.dat
-- spec/codecs/netflow9_test_softflowd_tpl_data.dat
 - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
 - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
-- spec/codecs/netflow9_test_valid01.dat
+- spec/codecs/ipfix_test_yaf_data45841.dat
+- spec/codecs/ipfix_test_yaf_data45873.dat
+- spec/codecs/ipfix_test_yaf_data53248.dat
+- spec/codecs/ipfix_test_yaf_tpl45841.dat
+- spec/codecs/ipfix_test_yaf_tpls_option_tpl.dat
+- spec/codecs/netflow9_cisco_asr1001x_tpl259.dat
+- spec/codecs/netflow9_test_cisco_1941K9.dat
+- spec/codecs/netflow9_test_cisco_wlc_8510_tpl_262.dat
+- spec/codecs/netflow9_test_paloalto_panos_data.dat
+- spec/codecs/netflow9_test_paloalto_panos_tpl.dat
 - spec/codecs/netflow_spec.rb
+- spec/codecs/netflow_stress.py
+- logstash-codec-netflow.gemspec
+- RFC_COMPLIANCE_NETFLOW_v9.md
+- README.md
+- RFC_COMPLIANCE_IPFIX.md
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- docs/index.asciidoc
 homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
 metadata:
   logstash_plugin: 'true'
   logstash_group: codec
-post_install_message:
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -163,44 +168,47 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.4.8
-signing_key:
+rubyforge_project: 
+rubygems_version: 2.0.14.1
+signing_key: 
 specification_version: 4
 summary: Reads Netflow v5 and Netflow v9 data
 test_files:
 - spec/codecs/ipfix.dat
-- spec/codecs/ipfix_test_barracuda_data256.dat
-- spec/codecs/ipfix_test_barracuda_tpl.dat
-- spec/codecs/ipfix_test_mikrotik_data258.dat
-- spec/codecs/ipfix_test_mikrotik_data259.dat
-- spec/codecs/ipfix_test_mikrotik_tpl.dat
-- spec/codecs/ipfix_test_netscaler_data.dat
-- spec/codecs/ipfix_test_netscaler_tpl.dat
 - spec/codecs/ipfix_test_openbsd_pflow_data.dat
 - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
-- spec/codecs/ipfix_test_vmware_vds_data264.dat
-- spec/codecs/ipfix_test_vmware_vds_data266.dat
-- spec/codecs/ipfix_test_vmware_vds_data266_267.dat
-- spec/codecs/ipfix_test_vmware_vds_tpl.dat
-- spec/codecs/ipfix_test_yaf_data45841.dat
-- spec/codecs/ipfix_test_yaf_data45873.dat
-- spec/codecs/ipfix_test_yaf_data53248.dat
-- spec/codecs/ipfix_test_yaf_tpl45841.dat
-- spec/codecs/ipfix_test_yaf_tpls_option_tpl.dat
 - spec/codecs/netflow5.dat
 - spec/codecs/netflow5_test_invalid01.dat
 - spec/codecs/netflow5_test_invalid02.dat
 - spec/codecs/netflow5_test_juniper_mx80.dat
 - spec/codecs/netflow5_test_microtik.dat
-- spec/codecs/netflow9_cisco_asr1001x_tpl259.dat
-- spec/codecs/netflow9_test_0length_fields_tpl_data.dat
-- spec/codecs/netflow9_test_cisco_1941K9.dat
 - spec/codecs/netflow9_test_cisco_asa_1_data.dat
 - spec/codecs/netflow9_test_cisco_asa_1_tpl.dat
 - spec/codecs/netflow9_test_cisco_asa_2_data.dat
 - spec/codecs/netflow9_test_cisco_asa_2_tpl_26x.dat
 - spec/codecs/netflow9_test_cisco_asa_2_tpl_27x.dat
+- spec/codecs/netflow9_test_invalid01.dat
+- spec/codecs/netflow9_test_macaddr_data.dat
+- spec/codecs/netflow9_test_macaddr_tpl.dat
+- spec/codecs/netflow9_test_nprobe_data.dat
+- spec/codecs/netflow9_test_nprobe_tpl.dat
+- spec/codecs/netflow9_test_softflowd_tpl_data.dat
+- spec/codecs/netflow9_test_valid01.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
+- spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
+- spec/codecs/ipfix_test_netscaler_data.dat
+- spec/codecs/ipfix_test_netscaler_tpl.dat
+- spec/codecs/ipfix_test_vmware_vds_data264.dat
+- spec/codecs/ipfix_test_vmware_vds_data266.dat
+- spec/codecs/ipfix_test_vmware_vds_data266_267.dat
+- spec/codecs/ipfix_test_vmware_vds_tpl.dat
+- spec/codecs/ipfix_test_barracuda_data256.dat
+- spec/codecs/ipfix_test_barracuda_tpl.dat
+- spec/codecs/ipfix_test_mikrotik_data258.dat
+- spec/codecs/ipfix_test_mikrotik_data259.dat
+- spec/codecs/ipfix_test_mikrotik_tpl.dat
+- spec/codecs/netflow9_test_0length_fields_tpl_data.dat
 - spec/codecs/netflow9_test_cisco_asr9k_data256.dat
 - spec/codecs/netflow9_test_cisco_asr9k_data260.dat
 - spec/codecs/netflow9_test_cisco_asr9k_opttpl256.dat
@@ -211,24 +219,24 @@ test_files:
 - spec/codecs/netflow9_test_cisco_nbar_data262.dat
 - spec/codecs/netflow9_test_cisco_nbar_opttpl260.dat
 - spec/codecs/netflow9_test_cisco_nbar_tpl262.dat
-- spec/codecs/netflow9_test_cisco_wlc_8510_tpl_262.dat
 - spec/codecs/netflow9_test_cisco_wlc_data261.dat
 - spec/codecs/netflow9_test_cisco_wlc_tpl.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_data256.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_data257.dat
 - spec/codecs/netflow9_test_fortigate_fortios_521_tpl.dat
-- spec/codecs/netflow9_test_invalid01.dat
 - spec/codecs/netflow9_test_juniper_srx_tplopt.dat
-- spec/codecs/netflow9_test_macaddr_data.dat
-- spec/codecs/netflow9_test_macaddr_tpl.dat
-- spec/codecs/netflow9_test_nprobe_data.dat
 - spec/codecs/netflow9_test_nprobe_dpi.dat
-- spec/codecs/netflow9_test_nprobe_tpl.dat
-- spec/codecs/netflow9_test_softflowd_tpl_data.dat
 - spec/codecs/netflow9_test_streamcore_tpl_data256.dat
 - spec/codecs/netflow9_test_streamcore_tpl_data260.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_data1024.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_data1025.dat
-- spec/codecs/netflow9_test_ubnt_edgerouter_tpl.dat
-- spec/codecs/netflow9_test_valid01.dat
+- spec/codecs/ipfix_test_yaf_data45841.dat
+- spec/codecs/ipfix_test_yaf_data45873.dat
+- spec/codecs/ipfix_test_yaf_data53248.dat
+- spec/codecs/ipfix_test_yaf_tpl45841.dat
+- spec/codecs/ipfix_test_yaf_tpls_option_tpl.dat
+- spec/codecs/netflow9_cisco_asr1001x_tpl259.dat
+- spec/codecs/netflow9_test_cisco_1941K9.dat
+- spec/codecs/netflow9_test_cisco_wlc_8510_tpl_262.dat
+- spec/codecs/netflow9_test_paloalto_panos_data.dat
+- spec/codecs/netflow9_test_paloalto_panos_tpl.dat
 - spec/codecs/netflow_spec.rb
+- spec/codecs/netflow_stress.py