logstash-codec-netflow 3.11.4 → 3.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 41949ce8555c62015bf5cd4ce6420227f2a1e604
-  data.tar.gz: 26eee9cbe124fc7f4279a3012535b74104db6f97
+  metadata.gz: 23d7c5f0b11a7d4e1f1ec188639527e182eb0361
+  data.tar.gz: 40ff5c0a2e481c785649b7cac4df8f94f7bf2aff
 SHA512:
-  metadata.gz: a7fa1165d5db730ced811a519ab2f1ddaf80501b0dd6fb71f3d1749b73631aca935fa7fae26156b1e68748494fa7b53d711c3d9b33635e6d4bdb01c6b6603d5e
-  data.tar.gz: fd17268f0517a8a34771708e52d5a63d29720ef46e648d662d47d50db7e4574a1fe173747060cda63e10c0c5b272b7dfec6e697e9ae1e6ba9d428956e7e23105
+  metadata.gz: c07ea4ed3c53ff4147ac122cd065c002ce6e4361817de65122b129a6eee159754322e93b4b4f3bd109435a049d59221a58757f225c621cfebe1db5744b101d90
+  data.tar.gz: f2ea2d25ef1f77e7ad3e29ae4f343630124ec209f5adb7e1cdc676f6a6583fea172ccfc3157611915a8e3f4d85c2edc2ac14e96bba5180e09420a6bd88683a39

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 3.12.0
+
+  - Added support for IPFIX from Procera/NetIntact/Sandvine 15.1
+
 ## 3.11.4
 
   - Workaround for breaking change in Netflow-Input-UDP > 3.2.0, see issue #122

data/CONTRIBUTORS

@@ -5,6 +5,7 @@ Contributors:
 * Aaron Mildenstein (untergeek)
 * Adam Kaminski (thimslugga)
 * Andrew Cholakian (andrewvc)
+* Ayden Beeson (abeeson)
 * Bjørn Ruberg (bruberg)
 * Colin Surprenant (colinsurprenant)
 * Daniel Nägele (analogbyte)

data/RFC_COMPLIANCE_IPFIX.md

@@ -4,7 +4,7 @@ The level of RFC compliance reached for collector-relevant requirements:
 
 | RFC       | Level                                        |
 |-----------|----------------------------------------------|
-| RFC 7011  | 47% of RFC "MUST" requirements implemented   |
+| RFC 7011  | 42% of RFC "MUST" requirements implemented   |
 | RFC 7011  | 19% of RFC "SHOULD" requirements implemented |  
 | RFC 7012  | 83% of IE data types supported
 | RFC 7012  | 90% of IEs supported
@@ -20,9 +20,9 @@ Summary of collector-relevant requirements implemented versus the total collecto
 | 3. IPFIX message format               | 2/2 | 0/2 |     |
 | 4. Specific reporting requirements    | 0/1 |     |     |
 | 5. Timing considerations              |     | 0/2 |     |
-| 6. Linkage with the Information Model |     | 0/1 |     |
+| 6. Linkage with the Information Model |     | 0/1 | 0/1 |
 | 7. Variable Length IE                 |     |     |     |
-| 8. Template management                | 4/8 | 1/5 | 1/2 |
+| 8. Template management                | 3/9 | 1/5 | 1/2 |
 | 9. The collecting process's side      | 4/5 | 1/3 | 0/4 |
 | 10. Transport protocol                | 5/8 | 1/3 | 3/3 |
 | 11. Security considerations           | 0/8 | 1/5 | 2/3 |
@@ -102,6 +102,7 @@ The tables below detail the collector-relevant requirements, and whether or not
 | Requirement                           |MUST |SHOULD| MAY| 
 |---------------------------------------|-----|-----|-----|
 | 6.1.6 Collecting Processes SHOULD detect and ignore IPFIX Messages containing ill-formed UTF-8 string values for Information Elements | | NO | | 
+| 6.2. Reduced-size encoding of signed, unsigned, or float data types | | | NO |
 
 ### 8. Template Management
 
@@ -110,7 +111,7 @@ The tables below detail the collector-relevant requirements, and whether or not
 |8. The Collecting Process MUST store all received Template Record information for the duration of each Transport Session until reuse or withdrawal as described in Section 8.1, or expiry over UDP as described in Section 8.4, so that it can interpret the corresponding Data Records.| YES | | | 
 |8. The Collecting Process MUST NOT assume that the Template IDs from a given Exporting Process refer to the same Templates as they did in previous Transport Sessions from the same Exporting Process| NO | | |
 |8. Collecting Process MUST NOT use Templates from one Transport Session to decode Data Sets in a subsequent Transport Session.| NO | | |
-|8. Collecting Processes MUST properly handle Templates with multiple identical Information Elements.| ? | | |
+|8. Collecting Processes MUST properly handle Templates with multiple identical Information Elements.| NO | | |
 |8. a Collecting Process MUST NOT assume that the Data Set and the associated Template Set (or Options Template Set) are exported in the same IPFIX Message| YES | | |
 |8. Though a Collecting Process normally receives Template Records from  the Exporting Process before receiving Data Records, this is not always the case, e.g., in the case of reordering or Collecting Process restart over UDP.  In these cases, the Collecting Process MAY buffer Data Records for which it has no Templates, to wait for Template Records describing them; however, note that in the presence of Template withdrawal and redefinition (Section 8.1) this may lead to incorrect interpretation of Data Records.| | | NO |
 | 8.Different Observation Domains within a Transport Session MAY use the same Template ID value to refer to different Templates; Collecting Processes MUST properly handle this case.| NO | | |

data/docs/index.asciidoc

@@ -52,6 +52,7 @@ The following Netflow/IPFIX exporters are known to work with the most recent ver
 |nProbe                   |  y | y  |   y   | L7 DPI fields now also supported
 |Nokia BRAS               |    |    |   y   |
 |OpenBSD pflow            |  y | n  |   y   | http://man.openbsd.org/OpenBSD-current/man4/pflow.4
+|Sandvine Procera PacketLogic| |    |   y   | v15.1
 |Softflowd                |  y | y  |   y   | IPFIX supported in https://github.com/djmdjm/softflowd
 |Streamcore Streamgroomer |    | y  |       |
 |Palo Alto PAN-OS         |    | y  |       |

data/lib/logstash/codecs/netflow/ipfix.yaml

@@ -3632,3 +3632,142 @@
   4321:
   - :uint64
   - :viptelaVPNId
+# List below taken from Procera PacketLogic product guide 15.1 - Not publicly available AFAIK
+# Further updates / additional fields may be present with versions 16/17+
+15397:
+  1:
+  - :string
+  - :proceraService
+  2:
+  - :string
+  - :proceraBaseService
+  3:
+  - :uint64
+  - :proceraIncomingOctets
+  4:
+  - :uint64
+  - :proceraOutgoingOctets
+  5:
+  - :uint64
+  - :proceraIncomingPackets
+  6:
+  - :uint64
+  - :proceraOutgoingPackets
+  7:
+  - :uint16
+  - :proceraIncomingShapingLatency
+  8:
+  - :uint16
+  - :proceraOutgoingShapingLatency
+  9:
+  - :uint32
+  - :proceraIncomingShapingDrops
+  10:
+  - :uint32
+  - :proceraOutgoingShapingDrops
+  11:
+  - :int32
+  - :proceraInternalRtt
+  12:
+  - :int32
+  - :proceraExternalRtt
+  15:
+  - :string
+  - :proceraFlowBehavior
+  16:
+  - :string
+  - :proceraContentCategories
+  17:
+  - :string
+  - :proceraProperty
+  18:
+  - :string
+  - :proceraServerHostname
+  19:
+  - :string
+  - :proceraHttpRequestMethod
+  20:
+  - :string
+  - :proceraHttpUserAgent
+  21:
+  - :string
+  - :proceraHttpContentType
+  22:
+  - :string
+  - :proceraHttpUrl
+  23:
+  - :string
+  - :proceraHttpReferer
+  24:
+  - :uint16
+  - :proceraHttpResponseStatus
+  25:
+  - :uint32
+  - :proceraHttpFileLength
+  26:
+  - :string
+  - :proceraHttpLocation
+  27:
+  - :string
+  - :proceraHttpLanguage
+  28:
+  - :string
+  - :proceraSubscriberIdentifier
+  29:
+  - :uint64
+  - :proceraMsisdn
+  30:
+  - :uint64
+  - :proceraImsi
+  31:
+  - :string
+  - :proceraRat
+  32:
+  - :uint64
+  - :proceraDeviceId
+  33:
+  - :string
+  - :proceraSgsn
+  34:
+  - :uint16
+  - :proceraRnc
+  35:
+  - :string
+  - :proceraApn
+  36:
+  - :string
+  - :proceraUserLocationInformation
+  37:
+  - :string
+  - :proceraGgsn
+  38:
+  - :float32
+  - :proceraQoeIncomingInternal
+  39:
+  - :float32
+  - :proceraQoeIncomingExternal
+  40:
+  - :float32
+  - :proceraQoeOutgoingInternal
+  41:
+  - :float32
+  - :proceraQoeOutgoingExternal
+  42:
+  - :ip4_addr
+  - :proceraLocalIPv4Host
+  43:
+  - :ip6_addr
+  - :proceraLocalIPv6Host
+  44:
+  - :ip4_addr
+  - :proceraRemoteIPv4Host
+  45:
+  - :ip6_addr
+  - :proceraRemoteIPv6Host
+  46:
+  - :string
+  - :proceraHttpRequestVersion
+  47:
+  - :string
+  - :proceraTemplateName
+

data/logstash-codec-netflow.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-codec-netflow'
-  s.version         = '3.11.4'
+  s.version         = '3.12.0'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Reads Netflow v5, Netflow v9 and IPFIX data"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

data/spec/codecs/netflow_spec.rb

@@ -1027,6 +1027,64 @@ describe LogStash::Codecs::Netflow do
 
   end
 
+  context "IPFIX Procera" do
+    let(:data) do
+      packets = []
+      packets << IO.read(File.join(File.dirname(__FILE__), "ipfix_test_procera_tpl52935.dat"), :mode => "rb")
+      packets << IO.read(File.join(File.dirname(__FILE__), "ipfix_test_procera_data52935.dat"), :mode => "rb")
+    end
+
+    let(:json_events) do
+      events = []
+      events << <<-END
+        {
+          "@timestamp": "2018-04-15T03:30:00.000Z",
+          "@version": "1",
+          "netflow": {
+            "proceraFlowBehavior": "INTERACTIVE,CLIENT_IS_LOCAL,INBOUND,ESTABLISHED,ACTIVE",
+            "sourceIPv6Address": "::",
+            "proceraOutgoingOctets": 3310,
+            "sourceTransportPort": 33689,
+            "destinationIPv6Address": "::",
+            "destinationTransportPort": 179,
+            "flowStartSeconds": "2018-04-15T03:25:00.000Z",
+            "proceraHttpContentType": "",
+            "proceraContentCategories": "",
+            "proceraSubscriberIdentifier": "",
+            "proceraTemplateName": "IPFIX",
+            "proceraHttpLocation": "",
+            "protocolIdentifier": 6,
+            "sourceIPv4Address": "138.44.161.14",
+            "flowEndSeconds": "2018-04-15T03:30:00.000Z",
+            "version": 10,
+            "proceraBaseService": "BGP-4",
+            "bgpSourceAsNumber": 7575,
+            "proceraIncomingOctets": 7076,
+            "bgpDestinationAsNumber": 7575,
+            "proceraHttpUrl": "",
+            "proceraService": "BGP-4",
+            "proceraHttpFileLength": 0,
+            "destinationIPv4Address": "138.44.161.13"
+          }
+        }
+      END
+
+      events.map{|event| event.gsub(/\s+/, "")}
+    end
+
+    it "should decode raw data" do
+      expect(decode.size).to eq(8)
+      expect(decode[7].get("[netflow][sourceIPv4Address]")).to eq("138.44.161.14")
+      expect(decode[7].get("[netflow][proceraBaseService]")).to eq("BGP-4")
+      expect(decode[7].get("[netflow][proceraFlowBehavior]")).to eq("INTERACTIVE,CLIENT_IS_LOCAL,INBOUND,ESTABLISHED,ACTIVE")
+    end
+
+    it "should serialize to json" do
+      expect(JSON.parse(decode[7].to_json)).to eq(JSON.parse(json_events[0]))
+    end
+
+  end
+
 
 
   context "Netflow 9 Ubiquiti Edgerouter with MPLS labels" do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-codec-netflow
 version: !ruby/object:Gem::Version
-  version: 3.11.4
+  version: 3.12.0
 platform: ruby
 authors:
 - Elastic
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2018-03-26 00:00:00.000000000 Z
+date: 2018-04-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -91,6 +91,8 @@ files:
 - spec/codecs/ipfix_test_nokia_bras_tpl.dat
 - spec/codecs/ipfix_test_openbsd_pflow_data.dat
 - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
+- spec/codecs/ipfix_test_procera_data52935.dat
+- spec/codecs/ipfix_test_procera_tpl52935.dat
 - spec/codecs/ipfix_test_viptela_data257.dat
 - spec/codecs/ipfix_test_viptela_tpl257.dat
 - spec/codecs/ipfix_test_vmware_vds_data264.dat
@@ -199,6 +201,8 @@ test_files:
 - spec/codecs/ipfix_test_nokia_bras_tpl.dat
 - spec/codecs/ipfix_test_openbsd_pflow_data.dat
 - spec/codecs/ipfix_test_openbsd_pflow_tpl.dat
+- spec/codecs/ipfix_test_procera_data52935.dat
+- spec/codecs/ipfix_test_procera_tpl52935.dat
 - spec/codecs/ipfix_test_viptela_data257.dat
 - spec/codecs/ipfix_test_viptela_tpl257.dat
 - spec/codecs/ipfix_test_vmware_vds_data264.dat