dap 0.0.5 → 0.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.travis.yml +9 -6
- data/CONTRIBUTING.md +140 -0
- data/Gemfile.lock +8 -5
- data/README.md +3 -0
- data/dap.gemspec +4 -3
- data/lib/dap/version.rb +1 -1
- metadata +5 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4fd7055de4b574f9e058515a7dbe5cf506029073
|
|
4
|
+
data.tar.gz: e05fa913dfccc81b3c34183baf2981b73b60fd14
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 534ece88a8a298ada25c24b621e877312a28407a091387133c65a11589f0d11bd99efc838c102ff99794bb4466436aa070fa05d769d058b7d55dbde36202694c
|
|
7
|
+
data.tar.gz: 8554e0dcb3e9fcecd353eadc31baa290d96183faa21e61cc1e31d62edcd96ccd41eadeacd587abb153c9d942329b031de711f34b46162aec3f2767b455472e3d
|
data/.travis.yml
CHANGED
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
language: ruby
|
|
2
|
+
cache: bundler
|
|
3
|
+
sudo: false
|
|
2
4
|
rvm:
|
|
3
|
-
- 2.
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
- 2.1.7
|
|
6
|
+
addons:
|
|
7
|
+
apt:
|
|
8
|
+
packages:
|
|
9
|
+
- libgeoip-dev
|
|
10
|
+
- libgeoip1
|
|
11
|
+
script: bundle exec rspec spec
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Contributing to dap
|
|
2
|
+
|
|
3
|
+
The users and maintainers of dap would greatly appreciate any contributions
|
|
4
|
+
you can make to the project. These contributions typically come in the form of
|
|
5
|
+
filed bugs/issues or pull requests (PRs). These contributions routinely result
|
|
6
|
+
in new versions of the [dap gem](https://rubygems.org/gems/dap) to be
|
|
7
|
+
released. The process for everything is described below.
|
|
8
|
+
|
|
9
|
+
## Contributing Issues / Bug Reports
|
|
10
|
+
|
|
11
|
+
If you encounter any bugs or problems with dap, please file them
|
|
12
|
+
[here](https://github.com/rapid7/dap/issues/new), providing as much detail as
|
|
13
|
+
possible. If the bug is straight-forward enough and you understand the fix for
|
|
14
|
+
the bug well enough, you may take the simpler, less-paperwork route and simply
|
|
15
|
+
fill a PR with the fix and the necessary details.
|
|
16
|
+
|
|
17
|
+
## Contributing Code
|
|
18
|
+
|
|
19
|
+
dap uses a model nearly identical to that of
|
|
20
|
+
[Metasploit](https://github.com/rapid7/metasploit-framework) as outlined
|
|
21
|
+
[here](https://github.com/rapid7/metasploit-framework/wiki/Setting-Up-a-Metasploit-Development-Environment),
|
|
22
|
+
at least from a ```git``` perspective. If you've been through that process
|
|
23
|
+
(or, even better, you've been through it many times with many people), you can
|
|
24
|
+
do exactly what you did for Metasploit but with dap and ignore the rest of
|
|
25
|
+
this document.
|
|
26
|
+
|
|
27
|
+
On the other hand, if you haven't, read on!
|
|
28
|
+
|
|
29
|
+
### Fork and Clone
|
|
30
|
+
|
|
31
|
+
Generally, this should only need to be done once, or if you need to start over.
|
|
32
|
+
|
|
33
|
+
1. Fork dap: Visit https://github.com/rapid7/dap and click Fork,
|
|
34
|
+
selecting your github account if prompted
|
|
35
|
+
2. Clone ```git@github.com:<your-github-username>/dap.git```, replacing
|
|
36
|
+
```<your-github-username>``` with, you guessed it, your Github username.
|
|
37
|
+
3. Add the master dap repository as your upstream:
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
git remote add upstream git://github.com/rapid7/dap.git
|
|
41
|
+
```
|
|
42
|
+
4. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream. It should look something like the following, in particular the second `fetch` option:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
[remote "upstream"]
|
|
46
|
+
url = git@github.com:rapid7/dap.git
|
|
47
|
+
fetch = +refs/heads/*:refs/remotes/upstream/*
|
|
48
|
+
fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
|
|
49
|
+
```
|
|
50
|
+
5. Fetch the latest revisions, including PRs:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
git fetch --all
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Branch and Improve
|
|
57
|
+
|
|
58
|
+
If you have a contribution to make, first create a branch to contain your
|
|
59
|
+
work. The name is yours to choose, however generally it should roughly
|
|
60
|
+
describe what you are doing. In this example, and from here on out, the
|
|
61
|
+
branch will be FOO, but you should obviously change this:
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
git fetch --all
|
|
65
|
+
git checkout master
|
|
66
|
+
git rebase upstream/master
|
|
67
|
+
git checkout -b FOO
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Now, make your changes, commit as necessary with useful commit messages.
|
|
71
|
+
|
|
72
|
+
Please note that changes to [lib/dap/version.rb](https://github.com/rapid7/dap/blob/master/lib/dap/version.rb) in PRs are almost never necessary.
|
|
73
|
+
|
|
74
|
+
Now push your changes to your fork:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
git push origin FOO
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Finally, submit the PR. Navigate to ```https://github.com/<your-github-username>/dap/compare/FOO```, fill in the details and submit.
|
|
81
|
+
|
|
82
|
+
### Testing
|
|
83
|
+
|
|
84
|
+
When your PR is submitted, it will be automatically subjected to the full run of tests in [Travis](https://travis-ci.org/rapid7/dap/), however you are encourage to perform testing _before_ submitting the PR. To do this, simply run `rake tests`.
|
|
85
|
+
|
|
86
|
+
## Landing PRs
|
|
87
|
+
|
|
88
|
+
(Note: this portion is a work-in-progress. Please update it as things change)
|
|
89
|
+
|
|
90
|
+
Much like with the process of submitting PRs, dap's process for landing PRs
|
|
91
|
+
is very similar to [Metasploit's process for landing
|
|
92
|
+
PRs](https://github.com/rapid7/metasploit-framework/wiki/Landing-Pull-Requests).
|
|
93
|
+
In short:
|
|
94
|
+
|
|
95
|
+
1. Follow the "Fork and Clone" steps from above
|
|
96
|
+
2. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream. It should look something like the following, in particular the second `fetch` option:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
[remote "upstream"]
|
|
100
|
+
url = git@github.com:rapid7/dap.git
|
|
101
|
+
fetch = +refs/heads/*:refs/remotes/upstream/*
|
|
102
|
+
fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
|
|
103
|
+
```
|
|
104
|
+
3. Fetch the latest revisions, including PRs:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
git fetch --all
|
|
108
|
+
```
|
|
109
|
+
4. Checkout and branch the PR for testing. Replace ```PR``` below with the actual PR # in question:
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
git checkout -b landing-PR upstream/pr/PR
|
|
113
|
+
```
|
|
114
|
+
5. Test the PR (see the Testing section above)
|
|
115
|
+
6. Merge with master, re-test, validate and push:
|
|
116
|
+
|
|
117
|
+
```
|
|
118
|
+
git checkout -b upstream-master --track upstream/master
|
|
119
|
+
git merge -S --no-ff --edit landing-PR # merge the PR into upstream-master
|
|
120
|
+
# re-test if/as necessary
|
|
121
|
+
git push upstream upstream-master:master --dry-run # confirm you are pushing what you expect
|
|
122
|
+
git push upstream upstream-master:master # push upstream-master to upstream:master
|
|
123
|
+
```
|
|
124
|
+
7. If applicable, release a new version (see next section)
|
|
125
|
+
|
|
126
|
+
## Releasing New Versions
|
|
127
|
+
|
|
128
|
+
When dap's critical parts are modified, for example its decoding or underlying supporting code, a new version _must_ eventually be released. Releases for non-functional updates such as updates to documentation are not necessary.
|
|
129
|
+
|
|
130
|
+
When a new version of dap is to be released, you _must_ follow the instructions below.
|
|
131
|
+
|
|
132
|
+
1. If are not already a dap project contributor for the dap gem (you'd be listed [here under OWNERS](https://rubygems.org/gems/dap)), become one:
|
|
133
|
+
1. Get an account on [Rubygems](https://rubygems.org)
|
|
134
|
+
2. Contact one of the dap project contributors (listed [here under OWNERS](https://rubygems.org/gems/dap) and have them add you to the dap gem. They'll need to run:
|
|
135
|
+
```
|
|
136
|
+
gem owner dap -a EMAIL
|
|
137
|
+
```
|
|
138
|
+
2. Edit [lib/dap/version.rb](https://github.com/rapid7/dap/blob/master/lib/dap/version.rb) and increment ```VERSION```. Commit and push to rapid7/dap master.
|
|
139
|
+
3. Run `rake release`. Among other things, this creates the new gem, uploads it to Rubygems and tags the release with a tag like `v<VERSION>`, where `<VERSION>` is replaced with the version from `version.rb`. For example, if you release version 1.2.3 of the gem, the tag will be `v1.2.3`.
|
|
140
|
+
4. If your default remote repository is not `rapid7/dap`, you must ensure that the tags created in the previous step are also pushed to the right location(s). For example, if `origin` is your fork of dap and `upstream` is `rapid7/master`, you should run `git push --tags --dry-run upstream` to confirm what tags will be pushed and then `git push --tags upstream` to push the tags.
|
data/Gemfile.lock
CHANGED
|
@@ -20,15 +20,15 @@ GEM
|
|
|
20
20
|
geoip-c (0.9.1)
|
|
21
21
|
gherkin (2.12.2)
|
|
22
22
|
multi_json (~> 1.3)
|
|
23
|
-
htmlentities (4.3.
|
|
24
|
-
mini_portile (0.6.
|
|
23
|
+
htmlentities (4.3.4)
|
|
24
|
+
mini_portile (0.6.2)
|
|
25
25
|
multi_json (1.10.1)
|
|
26
26
|
multi_test (0.1.1)
|
|
27
27
|
net-dns (0.8.0)
|
|
28
|
-
nokogiri (1.6.
|
|
29
|
-
mini_portile (
|
|
28
|
+
nokogiri (1.6.6.2)
|
|
29
|
+
mini_portile (~> 0.6.0)
|
|
30
30
|
oj (2.10.2)
|
|
31
|
-
recog (2.0.
|
|
31
|
+
recog (2.0.15)
|
|
32
32
|
nokogiri
|
|
33
33
|
rspec (3.1.0)
|
|
34
34
|
rspec-core (~> 3.1.0)
|
|
@@ -56,3 +56,6 @@ DEPENDENCIES
|
|
|
56
56
|
oj
|
|
57
57
|
recog (>= 2.0)
|
|
58
58
|
rspec (~> 3.1.0)
|
|
59
|
+
|
|
60
|
+
BUNDLED WITH
|
|
61
|
+
1.10.6
|
data/README.md
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# DAP: The Data Analysis Pipeline
|
|
2
2
|
|
|
3
|
+
[](http://badge.fury.io/rb/dap)
|
|
4
|
+
[](https://travis-ci.org/rapid7/dap)
|
|
5
|
+
|
|
3
6
|
DAP was created to transform text-based data on the command-line, specializing in transforms that are annoying or difficult to do with existing tools.
|
|
4
7
|
|
|
5
8
|
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between sed, awk, grep, csvtool, and jq, with map/reduce capabilities.
|
data/dap.gemspec
CHANGED
|
@@ -5,6 +5,7 @@ require 'dap/version'
|
|
|
5
5
|
Gem::Specification.new do |s|
|
|
6
6
|
s.name = 'dap'
|
|
7
7
|
s.version = Dap::VERSION
|
|
8
|
+
s.required_ruby_version = '>= 2.1'
|
|
8
9
|
s.authors = [
|
|
9
10
|
'Rapid7 Research'
|
|
10
11
|
]
|
|
@@ -14,9 +15,9 @@ Gem::Specification.new do |s|
|
|
|
14
15
|
s.homepage = "https://www.github.com/rapid7/dap"
|
|
15
16
|
s.summary = %q{DAP: The Data Analysis Pipeline}
|
|
16
17
|
s.description = %q{
|
|
17
|
-
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again
|
|
18
|
-
using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to
|
|
19
|
-
reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between
|
|
18
|
+
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again
|
|
19
|
+
using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to
|
|
20
|
+
reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between
|
|
20
21
|
sed, awk, grep, csvtool, and jq, with map/reduce capabilities.
|
|
21
22
|
}.gsub(/\s+/, ' ').strip
|
|
22
23
|
|
data/lib/dap/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dap
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.0.
|
|
4
|
+
version: 0.0.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Rapid7 Research
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2015-
|
|
11
|
+
date: 2015-11-07 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rspec
|
|
@@ -165,6 +165,7 @@ files:
|
|
|
165
165
|
- ".gitignore"
|
|
166
166
|
- ".rspec"
|
|
167
167
|
- ".travis.yml"
|
|
168
|
+
- CONTRIBUTING.md
|
|
168
169
|
- Gemfile
|
|
169
170
|
- Gemfile.lock
|
|
170
171
|
- LICENSE
|
|
@@ -233,7 +234,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
233
234
|
requirements:
|
|
234
235
|
- - ">="
|
|
235
236
|
- !ruby/object:Gem::Version
|
|
236
|
-
version: '
|
|
237
|
+
version: '2.1'
|
|
237
238
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
238
239
|
requirements:
|
|
239
240
|
- - ">="
|
|
@@ -241,7 +242,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
241
242
|
version: '0'
|
|
242
243
|
requirements: []
|
|
243
244
|
rubyforge_project:
|
|
244
|
-
rubygems_version: 2.4.
|
|
245
|
+
rubygems_version: 2.4.5.1
|
|
245
246
|
signing_key:
|
|
246
247
|
specification_version: 4
|
|
247
248
|
summary: 'DAP: The Data Analysis Pipeline'
|