dap 1.2.1 → 1.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +60 -17
- data/dap.gemspec +1 -2
- data/lib/dap/version.rb +1 -1
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3bd85239af6fa41370c3842a05c38ebe5f6326bf
|
4
|
+
data.tar.gz: 94f2fd35a90332fcf95d297c46a076beba82d17c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d399dd8304d8b6d44349b93efb1255104145ea4d9d8181f28aeea51196ce8ab215c3844991d9b84a4798b4d0b238c5382a15012911cc632fb6b68dc44ed87868
|
7
|
+
data.tar.gz: 8fa091707e3b5920a6aea17d394f0cfb7d1adc24e02cb1e086643a20596f2c869b00e5ec23ed3da45afb7fd863fea700c20d696edcabcb89fac3f308280df7d2
|
data/README.md
CHANGED
@@ -15,21 +15,46 @@ DAP was written to process terabyte-sized public scan datasets, such as those pr
|
|
15
15
|
|
16
16
|
### Prerequisites
|
17
17
|
|
18
|
-
DAP requires Ruby
|
18
|
+
DAP requires Ruby and is best suited for systems with a relatively current version with 2.4.x being the minimum requirement.
|
19
19
|
Ideally, this will be managed with either
|
20
20
|
[`rbenv`](https://github.com/rbenv/rbenv) or [`rvm`](https://rvm.io/) with the bundler gem also installed and up to date.
|
21
21
|
Using system managed/installed Rubies is possible but fraught with peril.
|
22
22
|
|
23
|
-
|
23
|
+
#### Maxmind IP Location Databases
|
24
24
|
|
25
|
-
|
26
|
-
|
25
|
+
If you intend on using any of the `geo_ip*` or `geo_ip2*` filters, you must
|
26
|
+
install the databases that provide the data for these filters. If you do not
|
27
|
+
intend on using these filters, you can skip this step.
|
27
28
|
|
28
|
-
|
29
|
-
|
29
|
+
`dap` versions 1.4.x and later depend on [Maxmind's geoip2/geolite2
|
30
|
+
databases](https://dev.maxmind.com/geoip/geoip2/geolite2/) to be able to append
|
31
|
+
geographic and related metadata to analyzed datasets. In order to use this
|
32
|
+
functionality you must put your copy of the relevant Maxmind databases in the
|
33
|
+
correct location in `/var/lib/geoip2` or the `data` directory of your `dap`
|
34
|
+
installation or override with an environment variable that specifies the full
|
35
|
+
path to the database in question:
|
36
|
+
|
37
|
+
* ASN: `GeoLite2-ASN.mmdb` (environment override: `GEOIP2_ASN_DATABASE_PATH`)
|
38
|
+
* City: `GeoLite2-City.mmdb` (environment override: `GEOIP2_CITY_DATABASE_PATH`)
|
39
|
+
* ISP: `GeoIP2-ISP.mmdb` (environment override: `GEOIP2_ISP_DATABASE_PATH`)
|
40
|
+
|
41
|
+
*NOTE*: Prior to `dap` version 1.4.x there was a dependency on [Maxmind's geoip
|
42
|
+
database](http://dev.maxmind.com/geoip/legacy/downloadable/)
|
43
|
+
to be able to append geographic metadata to analyzed datasets. However, since
|
44
|
+
that time Maxmind has dropped support for these legacy databases. If you
|
45
|
+
intend to continue using this deprecated functionality, you must put your copy
|
46
|
+
of the relevant Maxmind legacy databases in the correct location in
|
47
|
+
`/var/lib/geoip` or the `data` directory of your `dap` installation or override
|
48
|
+
with an environment variable that specifies the full path to the database in question:
|
49
|
+
|
50
|
+
* ASN: `GeoIPASNum.dat` (environment override in 1.4.x+: `GEOIP_ASN_DATABASE_PATH`)
|
51
|
+
* City: `geoip_city.dat` (environment override in 1.4.x+: `GEOIP_CITY_DATABASE_PATH`)
|
52
|
+
* Org: `geoip_org.dat` (environment override in 1.4.x+: `GEOIP_ORG_DATABASE_PATH`)
|
53
|
+
|
54
|
+
### Ubuntu 16.04+
|
30
55
|
|
31
56
|
```bash
|
32
|
-
sudo apt-get install
|
57
|
+
sudo apt-get install zlib1g-dev ruby ruby-dev gcc make ruby-bundler
|
33
58
|
gem install dap
|
34
59
|
```
|
35
60
|
|
@@ -37,7 +62,6 @@ gem install dap
|
|
37
62
|
|
38
63
|
```bash
|
39
64
|
brew update
|
40
|
-
brew install geoip
|
41
65
|
gem install dap
|
42
66
|
```
|
43
67
|
|
@@ -57,15 +81,34 @@ To see which input/output formats are supported and what filters are available,
|
|
57
81
|
This example reads as input a single IP address from `STDIN` in line form, applies geo-ip transformations as a filter on that line, and then returns the output as JSON:
|
58
82
|
|
59
83
|
```
|
60
|
-
$
|
61
|
-
{
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
84
|
+
$ echo 8.8.8.8 | bin/dap + lines + geo_ip2_city line + json | jq .
|
85
|
+
{
|
86
|
+
"line": "8.8.8.8",
|
87
|
+
"line.geoip2.city.city.geoname_id": "0",
|
88
|
+
"line.geoip2.city.continent.code": "NA",
|
89
|
+
"line.geoip2.city.continent.geoname_id": "6255149",
|
90
|
+
"line.geoip2.city.country.geoname_id": "6252001",
|
91
|
+
"line.geoip2.city.country.iso_code": "US",
|
92
|
+
"line.geoip2.city.country.is_in_european_union": "false",
|
93
|
+
"line.geoip2.city.location.accuracy_radius": "1000",
|
94
|
+
"line.geoip2.city.location.latitude": "37.751",
|
95
|
+
"line.geoip2.city.location.longitude": "-97.822",
|
96
|
+
"line.geoip2.city.location.metro_code": "0",
|
97
|
+
"line.geoip2.city.location.time_zone": "America/Chicago",
|
98
|
+
"line.geoip2.city.postal.code": "",
|
99
|
+
"line.geoip2.city.registered_country.geoname_id": "6252001",
|
100
|
+
"line.geoip2.city.registered_country.iso_code": "US",
|
101
|
+
"line.geoip2.city.registered_country.is_in_european_union": "false",
|
102
|
+
"line.geoip2.city.represented_country.geoname_id": "0",
|
103
|
+
"line.geoip2.city.represented_country.iso_code": "",
|
104
|
+
"line.geoip2.city.represented_country.is_in_european_union": "false",
|
105
|
+
"line.geoip2.city.represented_country.type": "",
|
106
|
+
"line.geoip2.city.traits.is_anonymous_proxy": "false",
|
107
|
+
"line.geoip2.city.traits.is_satellite_provider": "false",
|
108
|
+
"line.geoip2.city.continent.name": "North America",
|
109
|
+
"line.geoip2.city.country.name": "United States",
|
110
|
+
"line.geoip2.city.registered_country.name": "United States"
|
111
|
+
}
|
69
112
|
```
|
70
113
|
|
71
114
|
There are also several examples of how to use DAP along with sample datasets [here](samples).
|
data/dap.gemspec
CHANGED
@@ -5,7 +5,7 @@ require 'dap/version'
|
|
5
5
|
Gem::Specification.new do |s|
|
6
6
|
s.name = 'dap'
|
7
7
|
s.version = Dap::VERSION
|
8
|
-
s.required_ruby_version = '>= 2.
|
8
|
+
s.required_ruby_version = '>= 2.4'
|
9
9
|
s.authors = [
|
10
10
|
'Rapid7 Research'
|
11
11
|
]
|
@@ -40,5 +40,4 @@ Gem::Specification.new do |s|
|
|
40
40
|
s.add_runtime_dependency 'geoip-c'
|
41
41
|
s.add_runtime_dependency 'recog'
|
42
42
|
s.add_runtime_dependency 'maxmind-db'
|
43
|
-
|
44
43
|
end
|
data/lib/dap/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: dap
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.2.
|
4
|
+
version: 1.2.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Rapid7 Research
|
@@ -275,7 +275,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
275
275
|
requirements:
|
276
276
|
- - ">="
|
277
277
|
- !ruby/object:Gem::Version
|
278
|
-
version: '2.
|
278
|
+
version: '2.4'
|
279
279
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
280
280
|
requirements:
|
281
281
|
- - ">="
|
@@ -283,7 +283,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
283
283
|
version: '0'
|
284
284
|
requirements: []
|
285
285
|
rubyforge_project:
|
286
|
-
rubygems_version: 2.6.
|
286
|
+
rubygems_version: 2.6.11
|
287
287
|
signing_key:
|
288
288
|
specification_version: 4
|
289
289
|
summary: 'DAP: The Data Analysis Pipeline'
|