dap 0.0.5 → 0.0.6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +9 -6
- data/CONTRIBUTING.md +140 -0
- data/Gemfile.lock +8 -5
- data/README.md +3 -0
- data/dap.gemspec +4 -3
- data/lib/dap/version.rb +1 -1
- metadata +5 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4fd7055de4b574f9e058515a7dbe5cf506029073
|
|
4
|
+
data.tar.gz: e05fa913dfccc81b3c34183baf2981b73b60fd14
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 534ece88a8a298ada25c24b621e877312a28407a091387133c65a11589f0d11bd99efc838c102ff99794bb4466436aa070fa05d769d058b7d55dbde36202694c
|
|
7
|
+
data.tar.gz: 8554e0dcb3e9fcecd353eadc31baa290d96183faa21e61cc1e31d62edcd96ccd41eadeacd587abb153c9d942329b031de711f34b46162aec3f2767b455472e3d
|
data/.travis.yml
CHANGED
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
language: ruby
|
|
2
|
+
cache: bundler
|
|
3
|
+
sudo: false
|
|
2
4
|
rvm:
|
|
3
|
-
- 2.
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
- 2.1.7
|
|
6
|
+
addons:
|
|
7
|
+
apt:
|
|
8
|
+
packages:
|
|
9
|
+
- libgeoip-dev
|
|
10
|
+
- libgeoip1
|
|
11
|
+
script: bundle exec rspec spec
|
data/CONTRIBUTING.md
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Contributing to dap
|
|
2
|
+
|
|
3
|
+
The users and maintainers of dap would greatly appreciate any contributions
|
|
4
|
+
you can make to the project. These contributions typically come in the form of
|
|
5
|
+
filed bugs/issues or pull requests (PRs). These contributions routinely result
|
|
6
|
+
in new versions of the [dap gem](https://rubygems.org/gems/dap) to be
|
|
7
|
+
released. The process for everything is described below.
|
|
8
|
+
|
|
9
|
+
## Contributing Issues / Bug Reports
|
|
10
|
+
|
|
11
|
+
If you encounter any bugs or problems with dap, please file them
|
|
12
|
+
[here](https://github.com/rapid7/dap/issues/new), providing as much detail as
|
|
13
|
+
possible. If the bug is straight-forward enough and you understand the fix for
|
|
14
|
+
the bug well enough, you may take the simpler, less-paperwork route and simply
|
|
15
|
+
fill a PR with the fix and the necessary details.
|
|
16
|
+
|
|
17
|
+
## Contributing Code
|
|
18
|
+
|
|
19
|
+
dap uses a model nearly identical to that of
|
|
20
|
+
[Metasploit](https://github.com/rapid7/metasploit-framework) as outlined
|
|
21
|
+
[here](https://github.com/rapid7/metasploit-framework/wiki/Setting-Up-a-Metasploit-Development-Environment),
|
|
22
|
+
at least from a ```git``` perspective. If you've been through that process
|
|
23
|
+
(or, even better, you've been through it many times with many people), you can
|
|
24
|
+
do exactly what you did for Metasploit but with dap and ignore the rest of
|
|
25
|
+
this document.
|
|
26
|
+
|
|
27
|
+
On the other hand, if you haven't, read on!
|
|
28
|
+
|
|
29
|
+
### Fork and Clone
|
|
30
|
+
|
|
31
|
+
Generally, this should only need to be done once, or if you need to start over.
|
|
32
|
+
|
|
33
|
+
1. Fork dap: Visit https://github.com/rapid7/dap and click Fork,
|
|
34
|
+
selecting your github account if prompted
|
|
35
|
+
2. Clone ```git@github.com:<your-github-username>/dap.git```, replacing
|
|
36
|
+
```<your-github-username>``` with, you guessed it, your Github username.
|
|
37
|
+
3. Add the master dap repository as your upstream:
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
git remote add upstream git://github.com/rapid7/dap.git
|
|
41
|
+
```
|
|
42
|
+
4. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream. It should look something like the following, in particular the second `fetch` option:
|
|
43
|
+
|
|
44
|
+
```
|
|
45
|
+
[remote "upstream"]
|
|
46
|
+
url = git@github.com:rapid7/dap.git
|
|
47
|
+
fetch = +refs/heads/*:refs/remotes/upstream/*
|
|
48
|
+
fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
|
|
49
|
+
```
|
|
50
|
+
5. Fetch the latest revisions, including PRs:
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
git fetch --all
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Branch and Improve
|
|
57
|
+
|
|
58
|
+
If you have a contribution to make, first create a branch to contain your
|
|
59
|
+
work. The name is yours to choose, however generally it should roughly
|
|
60
|
+
describe what you are doing. In this example, and from here on out, the
|
|
61
|
+
branch will be FOO, but you should obviously change this:
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
git fetch --all
|
|
65
|
+
git checkout master
|
|
66
|
+
git rebase upstream/master
|
|
67
|
+
git checkout -b FOO
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Now, make your changes, commit as necessary with useful commit messages.
|
|
71
|
+
|
|
72
|
+
Please note that changes to [lib/dap/version.rb](https://github.com/rapid7/dap/blob/master/lib/dap/version.rb) in PRs are almost never necessary.
|
|
73
|
+
|
|
74
|
+
Now push your changes to your fork:
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
git push origin FOO
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Finally, submit the PR. Navigate to ```https://github.com/<your-github-username>/dap/compare/FOO```, fill in the details and submit.
|
|
81
|
+
|
|
82
|
+
### Testing
|
|
83
|
+
|
|
84
|
+
When your PR is submitted, it will be automatically subjected to the full run of tests in [Travis](https://travis-ci.org/rapid7/dap/), however you are encourage to perform testing _before_ submitting the PR. To do this, simply run `rake tests`.
|
|
85
|
+
|
|
86
|
+
## Landing PRs
|
|
87
|
+
|
|
88
|
+
(Note: this portion is a work-in-progress. Please update it as things change)
|
|
89
|
+
|
|
90
|
+
Much like with the process of submitting PRs, dap's process for landing PRs
|
|
91
|
+
is very similar to [Metasploit's process for landing
|
|
92
|
+
PRs](https://github.com/rapid7/metasploit-framework/wiki/Landing-Pull-Requests).
|
|
93
|
+
In short:
|
|
94
|
+
|
|
95
|
+
1. Follow the "Fork and Clone" steps from above
|
|
96
|
+
2. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream. It should look something like the following, in particular the second `fetch` option:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
[remote "upstream"]
|
|
100
|
+
url = git@github.com:rapid7/dap.git
|
|
101
|
+
fetch = +refs/heads/*:refs/remotes/upstream/*
|
|
102
|
+
fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
|
|
103
|
+
```
|
|
104
|
+
3. Fetch the latest revisions, including PRs:
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
git fetch --all
|
|
108
|
+
```
|
|
109
|
+
4. Checkout and branch the PR for testing. Replace ```PR``` below with the actual PR # in question:
|
|
110
|
+
|
|
111
|
+
```
|
|
112
|
+
git checkout -b landing-PR upstream/pr/PR
|
|
113
|
+
```
|
|
114
|
+
5. Test the PR (see the Testing section above)
|
|
115
|
+
6. Merge with master, re-test, validate and push:
|
|
116
|
+
|
|
117
|
+
```
|
|
118
|
+
git checkout -b upstream-master --track upstream/master
|
|
119
|
+
git merge -S --no-ff --edit landing-PR # merge the PR into upstream-master
|
|
120
|
+
# re-test if/as necessary
|
|
121
|
+
git push upstream upstream-master:master --dry-run # confirm you are pushing what you expect
|
|
122
|
+
git push upstream upstream-master:master # push upstream-master to upstream:master
|
|
123
|
+
```
|
|
124
|
+
7. If applicable, release a new version (see next section)
|
|
125
|
+
|
|
126
|
+
## Releasing New Versions
|
|
127
|
+
|
|
128
|
+
When dap's critical parts are modified, for example its decoding or underlying supporting code, a new version _must_ eventually be released. Releases for non-functional updates such as updates to documentation are not necessary.
|
|
129
|
+
|
|
130
|
+
When a new version of dap is to be released, you _must_ follow the instructions below.
|
|
131
|
+
|
|
132
|
+
1. If are not already a dap project contributor for the dap gem (you'd be listed [here under OWNERS](https://rubygems.org/gems/dap)), become one:
|
|
133
|
+
1. Get an account on [Rubygems](https://rubygems.org)
|
|
134
|
+
2. Contact one of the dap project contributors (listed [here under OWNERS](https://rubygems.org/gems/dap) and have them add you to the dap gem. They'll need to run:
|
|
135
|
+
```
|
|
136
|
+
gem owner dap -a EMAIL
|
|
137
|
+
```
|
|
138
|
+
2. Edit [lib/dap/version.rb](https://github.com/rapid7/dap/blob/master/lib/dap/version.rb) and increment ```VERSION```. Commit and push to rapid7/dap master.
|
|
139
|
+
3. Run `rake release`. Among other things, this creates the new gem, uploads it to Rubygems and tags the release with a tag like `v<VERSION>`, where `<VERSION>` is replaced with the version from `version.rb`. For example, if you release version 1.2.3 of the gem, the tag will be `v1.2.3`.
|
|
140
|
+
4. If your default remote repository is not `rapid7/dap`, you must ensure that the tags created in the previous step are also pushed to the right location(s). For example, if `origin` is your fork of dap and `upstream` is `rapid7/master`, you should run `git push --tags --dry-run upstream` to confirm what tags will be pushed and then `git push --tags upstream` to push the tags.
|
data/Gemfile.lock
CHANGED
|
@@ -20,15 +20,15 @@ GEM
|
|
|
20
20
|
geoip-c (0.9.1)
|
|
21
21
|
gherkin (2.12.2)
|
|
22
22
|
multi_json (~> 1.3)
|
|
23
|
-
htmlentities (4.3.
|
|
24
|
-
mini_portile (0.6.
|
|
23
|
+
htmlentities (4.3.4)
|
|
24
|
+
mini_portile (0.6.2)
|
|
25
25
|
multi_json (1.10.1)
|
|
26
26
|
multi_test (0.1.1)
|
|
27
27
|
net-dns (0.8.0)
|
|
28
|
-
nokogiri (1.6.
|
|
29
|
-
mini_portile (
|
|
28
|
+
nokogiri (1.6.6.2)
|
|
29
|
+
mini_portile (~> 0.6.0)
|
|
30
30
|
oj (2.10.2)
|
|
31
|
-
recog (2.0.
|
|
31
|
+
recog (2.0.15)
|
|
32
32
|
nokogiri
|
|
33
33
|
rspec (3.1.0)
|
|
34
34
|
rspec-core (~> 3.1.0)
|
|
@@ -56,3 +56,6 @@ DEPENDENCIES
|
|
|
56
56
|
oj
|
|
57
57
|
recog (>= 2.0)
|
|
58
58
|
rspec (~> 3.1.0)
|
|
59
|
+
|
|
60
|
+
BUNDLED WITH
|
|
61
|
+
1.10.6
|
data/README.md
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
# DAP: The Data Analysis Pipeline
|
|
2
2
|
|
|
3
|
+
[](http://badge.fury.io/rb/dap)
|
|
4
|
+
[](https://travis-ci.org/rapid7/dap)
|
|
5
|
+
|
|
3
6
|
DAP was created to transform text-based data on the command-line, specializing in transforms that are annoying or difficult to do with existing tools.
|
|
4
7
|
|
|
5
8
|
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between sed, awk, grep, csvtool, and jq, with map/reduce capabilities.
|
data/dap.gemspec
CHANGED
|
@@ -5,6 +5,7 @@ require 'dap/version'
|
|
|
5
5
|
Gem::Specification.new do |s|
|
|
6
6
|
s.name = 'dap'
|
|
7
7
|
s.version = Dap::VERSION
|
|
8
|
+
s.required_ruby_version = '>= 2.1'
|
|
8
9
|
s.authors = [
|
|
9
10
|
'Rapid7 Research'
|
|
10
11
|
]
|
|
@@ -14,9 +15,9 @@ Gem::Specification.new do |s|
|
|
|
14
15
|
s.homepage = "https://www.github.com/rapid7/dap"
|
|
15
16
|
s.summary = %q{DAP: The Data Analysis Pipeline}
|
|
16
17
|
s.description = %q{
|
|
17
|
-
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again
|
|
18
|
-
using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to
|
|
19
|
-
reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between
|
|
18
|
+
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again
|
|
19
|
+
using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to
|
|
20
|
+
reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between
|
|
20
21
|
sed, awk, grep, csvtool, and jq, with map/reduce capabilities.
|
|
21
22
|
}.gsub(/\s+/, ' ').strip
|
|
22
23
|
|
data/lib/dap/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dap
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.0.
|
|
4
|
+
version: 0.0.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Rapid7 Research
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2015-
|
|
11
|
+
date: 2015-11-07 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rspec
|
|
@@ -165,6 +165,7 @@ files:
|
|
|
165
165
|
- ".gitignore"
|
|
166
166
|
- ".rspec"
|
|
167
167
|
- ".travis.yml"
|
|
168
|
+
- CONTRIBUTING.md
|
|
168
169
|
- Gemfile
|
|
169
170
|
- Gemfile.lock
|
|
170
171
|
- LICENSE
|
|
@@ -233,7 +234,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
233
234
|
requirements:
|
|
234
235
|
- - ">="
|
|
235
236
|
- !ruby/object:Gem::Version
|
|
236
|
-
version: '
|
|
237
|
+
version: '2.1'
|
|
237
238
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
238
239
|
requirements:
|
|
239
240
|
- - ">="
|
|
@@ -241,7 +242,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
241
242
|
version: '0'
|
|
242
243
|
requirements: []
|
|
243
244
|
rubyforge_project:
|
|
244
|
-
rubygems_version: 2.4.
|
|
245
|
+
rubygems_version: 2.4.5.1
|
|
245
246
|
signing_key:
|
|
246
247
|
specification_version: 4
|
|
247
248
|
summary: 'DAP: The Data Analysis Pipeline'
|