owasp-glue 0.9.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (54) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGES +27 -0
  3. data/FEATURES +19 -0
  4. data/README.md +117 -0
  5. data/bin/glue +67 -0
  6. data/lib/glue.rb +317 -0
  7. data/lib/glue/event.rb +14 -0
  8. data/lib/glue/filters.rb +41 -0
  9. data/lib/glue/filters/base_filter.rb +19 -0
  10. data/lib/glue/filters/jira_one_time_filter.rb +57 -0
  11. data/lib/glue/filters/remove_all_filter.rb +16 -0
  12. data/lib/glue/filters/zap_consdensing_filter.rb +76 -0
  13. data/lib/glue/finding.rb +52 -0
  14. data/lib/glue/mounters.rb +55 -0
  15. data/lib/glue/mounters/base_mounter.rb +31 -0
  16. data/lib/glue/mounters/docker_mounter.rb +44 -0
  17. data/lib/glue/mounters/filesystem_mounter.rb +20 -0
  18. data/lib/glue/mounters/git_mounter.rb +52 -0
  19. data/lib/glue/mounters/iso_mounter.rb +42 -0
  20. data/lib/glue/mounters/url_mounter.rb +28 -0
  21. data/lib/glue/options.rb +269 -0
  22. data/lib/glue/reporters.rb +50 -0
  23. data/lib/glue/reporters/base_reporter.rb +21 -0
  24. data/lib/glue/reporters/csv_reporter.rb +19 -0
  25. data/lib/glue/reporters/jira_reporter.rb +59 -0
  26. data/lib/glue/reporters/json_reporter.rb +20 -0
  27. data/lib/glue/reporters/text_reporter.rb +19 -0
  28. data/lib/glue/scanner.rb +28 -0
  29. data/lib/glue/tasks.rb +124 -0
  30. data/lib/glue/tasks/av.rb +42 -0
  31. data/lib/glue/tasks/base_task.rb +80 -0
  32. data/lib/glue/tasks/brakeman.rb +58 -0
  33. data/lib/glue/tasks/bundle-audit.rb +95 -0
  34. data/lib/glue/tasks/checkmarx.rb +60 -0
  35. data/lib/glue/tasks/dawnscanner.rb +55 -0
  36. data/lib/glue/tasks/eslint.rb +69 -0
  37. data/lib/glue/tasks/fim.rb +60 -0
  38. data/lib/glue/tasks/findsecbugs.rb +90 -0
  39. data/lib/glue/tasks/npm.rb +58 -0
  40. data/lib/glue/tasks/nsp.rb +65 -0
  41. data/lib/glue/tasks/owasp-dep-check.rb +117 -0
  42. data/lib/glue/tasks/patterns.json +394 -0
  43. data/lib/glue/tasks/pmd.rb +63 -0
  44. data/lib/glue/tasks/retirejs.rb +107 -0
  45. data/lib/glue/tasks/scanjs-eslintrc +106 -0
  46. data/lib/glue/tasks/scanjs.rb +31 -0
  47. data/lib/glue/tasks/sfl.rb +67 -0
  48. data/lib/glue/tasks/snyk.rb +81 -0
  49. data/lib/glue/tasks/test.rb +47 -0
  50. data/lib/glue/tasks/zap.rb +99 -0
  51. data/lib/glue/tracker.rb +47 -0
  52. data/lib/glue/util.rb +36 -0
  53. data/lib/glue/version.rb +3 -0
  54. metadata +294 -0
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: f5e7eb728b9c905eb0a61adef5fb2aaf7c427ef4
4
+ data.tar.gz: 755da45bbc21effa03b77ca3466a8e3286c30790
5
+ SHA512:
6
+ metadata.gz: e56539ab4dfcbfd383d0d39765dee5e560a723ab49187dd68df7b9dbd20733649d67cab07ab33d6075a01970334b65bdc4ad55a1873a432b13b67c5036aefe45
7
+ data.tar.gz: bba3c2f4f09ab3735417ff76ab4c3fec5d93a2c5a0e822dad839865cc1f12017e2a7ba9192b1416be5b1c4435aab308ed97efd85340b30f1a6f877efa8acf306
data/CHANGES ADDED
@@ -0,0 +1,27 @@
1
+ ## 0.9.0
2
+ * Rename to Glue
3
+ ## 0.8.0
4
+ * Java Tooling
5
+ ## 0.6.0
6
+ * Docker image.
7
+ * JavaScript tools (retire.js, nodesecurity, eslint)
8
+ * Java tools (SonarCube, Findbugs)
9
+ ## 0.5.3
10
+ * Checkmarx
11
+ * JIRA Integration
12
+ * Search for secrets (a la gitrob)
13
+ * OWASP dependency-check
14
+ ## 0.1.0
15
+ * Structure for pipeline and initial tool series.
16
+ * gem packaging
17
+ * Finding / reporting
18
+ * docker mounting
19
+ * Support for task labels. Run with 'pipeline -l label' and only tasks that identify with that label will run.
20
+ * Move to only keep options in tracker.
21
+
22
+ ## FUTURE
23
+ * TODO: add scap
24
+ * TODO: Devtools: pmd, brakeman, dependency checker, codescan, scanjs, bandit, etc.
25
+ * TODO: Misc OS checks
26
+ * TODO: Active: ZAP, gauntlt
27
+ * TODO: .NET
@@ -0,0 +1,19 @@
1
+ # Capabilities
2
+ * AV
3
+ * FIM
4
+
5
+ FUTURE:
6
+ * oscap - https://www.redhat.com/archives/open-scap-list/2013-May/msg00007.html
7
+ * pmd
8
+ * brakeman
9
+ * dependency checker
10
+ * codescan
11
+
12
+ # Images
13
+ * Raw directory on file system.
14
+ * git
15
+ * docker
16
+
17
+ FUTURE:
18
+ * iso
19
+ * vmdk, vdi, ami
@@ -0,0 +1,117 @@
1
+ <img src="./glue.png" width="120"/>
2
+
3
+ # Glue
4
+
5
+ Glue is a framework for running a series of tools. Generally, it is intended as a backbone for automating a security analysis pipeline of tools.
6
+
7
+ # Recommended Usage
8
+
9
+ For those wishing to run Glue, we recommend using the docker image because
10
+ it should have the other tools it uses available already and configured.
11
+ See the documentation for more info. [Glue Docker Documentation](./DOCKER.md)
12
+
13
+ For those interested in how to use Glue in a DevOps context, see
14
+ [Glue DevOps Integration Options](./DEVOPS.md)
15
+
16
+ # Installation
17
+
18
+ gem install Glue
19
+
20
+ or
21
+
22
+ docker run owasp/glue
23
+
24
+ # Installation for Development
25
+
26
+ git clone https://github.com/owasp/glue
27
+ cd glue -- RVM will set to 2.3.1 with Gemset Glue
28
+ gem install bundler
29
+ bundle install
30
+
31
+ ## Running in Development
32
+
33
+ cd lib
34
+ ../bin/glue -h
35
+
36
+ # Extending Glue
37
+
38
+ Glue is intended to be extended through added "tasks". To add a new tool,
39
+ copy an existing task and tweak to make it work for the tool in question.
40
+
41
+ # Usage
42
+
43
+ Glue <options> <target>
44
+
45
+ ## Options
46
+
47
+ Common options include:
48
+ -d for debug
49
+ -f for format (takes "json", "csv", "jira")
50
+
51
+ For a full list of options, use `Glue --help` or see the [OPTIONS.md](./OPTIONS.md) file.
52
+
53
+ ## Target
54
+
55
+ The target can be:
56
+ * Filesystem (which is analyzed in place)
57
+ * Git repo (which is cloned for analysis)
58
+ * Other types of images (.iso, docker, etc. are experimental)
59
+
60
+
61
+ # Dependencies
62
+
63
+ * clamav
64
+ * hashdeep
65
+ * rm (*nix)
66
+ * git
67
+ * mount (*nix)
68
+ * docker
69
+
70
+ # Development
71
+
72
+ To run the code, run the following from the root directory:
73
+ >ruby bin/Glue <options> target
74
+
75
+ To build a gem, just run:
76
+ gem build Glue.gemspec
77
+
78
+
79
+ # Integration
80
+
81
+ ## Git Hooks
82
+
83
+ First, grab the hook from the code.
84
+ ```
85
+ meditation:hooks mk$ cp /area53/owasp/Glue/hooks/pre-commit .
86
+ ```
87
+
88
+ Then make it executable.
89
+ ```
90
+ meditation:hooks mk$ chmod +x pre-commit
91
+ ```
92
+
93
+ Make sure the shell you are committing in can see docker.
94
+ ```
95
+ meditation:hooks mk$ eval "$(docker-machine env default)"
96
+ ```
97
+
98
+ Now go test and make a change and commit a file.
99
+ The result should be that Glue runs against your
100
+ code and will not allow commits unless the results
101
+ are clean. (Which is not necessarily a reasonable
102
+ expectation)
103
+
104
+
105
+ # Configuration files
106
+
107
+ For advanced usage scenarios, you can save your configuration and use it at runtime.
108
+
109
+ # Authors
110
+
111
+ Matt Konda
112
+ Alex Lock
113
+ Rafa Perez
114
+
115
+ # License
116
+
117
+ Apache 2: http://www.apache.org/licenses/LICENSE-2.0
@@ -0,0 +1,67 @@
1
+ #!/usr/bin/env ruby
2
+ #Adjust path in case called directly and not through gem
3
+ $:.unshift "#{File.expand_path(File.dirname(__FILE__))}/../lib"
4
+
5
+ require 'glue'
6
+ require 'glue/options'
7
+ require 'glue/version'
8
+
9
+ #Parse options
10
+ begin
11
+ options, parser = Glue::Options.parse! ARGV
12
+ rescue OptionParser::ParseError => e
13
+ $stderr.puts e.message.capitalize
14
+ $stderr.puts "Please see `glue --help` for valid options"
15
+ exit -1
16
+ end
17
+
18
+ #Exit early for these options
19
+ if options[:list_checks] or options[:list_optional_checks]
20
+ Glue.list_checks options
21
+ exit
22
+ elsif options[:create_config]
23
+ glue.dump_config options
24
+ exit
25
+ elsif options[:show_help]
26
+ puts parser
27
+ exit
28
+ elsif options[:show_version]
29
+ puts "Glue #{Glue::Version}"
30
+ exit
31
+ end
32
+
33
+ #Set application path according to the commandline arguments
34
+ unless options[:target]
35
+ if ARGV[-1].nil?
36
+ options[:target] = "."
37
+ else
38
+ options[:target] = ARGV[-1]
39
+ end
40
+ end
41
+
42
+ trap("INT") do
43
+ $stderr.puts "\nInterrupted - exiting."
44
+
45
+ if options[:debug]
46
+ $stderr.puts caller
47
+ end
48
+
49
+ exit!
50
+ end
51
+
52
+ if options[:quiet].nil?
53
+ options[:quiet] = :command_line
54
+ end
55
+
56
+ begin
57
+ #Run scan and output a report
58
+ tracker = Glue.run options.merge(:print_report => true, :quiet => options[:quiet])
59
+
60
+ #Return error code if --exit-on-warn is used and warnings were found
61
+ if options[:exit_on_warn] and not tracker.findings.empty?
62
+ exit Glue::Warnings_Found_Exit_Code
63
+ end
64
+ rescue Glue::NoTargetError => e
65
+ $stderr.puts e.message
66
+ exit 1
67
+ end
@@ -0,0 +1,317 @@
1
+ require 'rubygems'
2
+ require 'yaml'
3
+ require 'set'
4
+ require 'tempfile'
5
+
6
+ module Glue
7
+
8
+ #This exit code is used when warnings are found and the --exit-on-warn
9
+ #option is set
10
+ Warnings_Found_Exit_Code = 3
11
+
12
+ @debug = false
13
+ @quiet = false
14
+ @loaded_dependencies = []
15
+
16
+ #Run Glue.
17
+ #
18
+ #Options:
19
+ #
20
+ # * :config_file - configuration file
21
+ # * :exit_on_warn - return false if warnings found, true otherwise. Not recommended for library use (default: false)
22
+ # * :output_files - files for output
23
+ # * :output_formats - formats for output (:to_s, :to_tabs, :to_csv, :to_html)
24
+ # * :parallel_checks - run checks in parallel (default: true)
25
+ # * :print_report - if no output file specified, print to stdout (default: false)
26
+ # * :quiet - suppress most messages (default: true)
27
+ def self.run options
28
+ options = set_options options
29
+
30
+ @quiet = !!options[:quiet]
31
+ @debug = !!options[:debug]
32
+
33
+ if @quiet
34
+ options[:report_progress] = false
35
+ end
36
+
37
+ unless options[:logfile].nil?
38
+ if options[:logfile].is_a? File
39
+ $logfile = options[:logfile]
40
+ else
41
+ $logfile = File.open(options[:logfile], 'a')
42
+ end
43
+
44
+ begin
45
+ scan options
46
+ ensure
47
+ $logfile.close unless options[:logfile].is_a? File
48
+ end
49
+ end
50
+ end
51
+
52
+ #Sets up options for run, checks given application path
53
+ def self.set_options options
54
+ if options.is_a? String
55
+ options = { :target => options }
56
+ end
57
+
58
+ if options[:quiet] == :command_line
59
+ command_line = true
60
+ options.delete :quiet
61
+ end
62
+
63
+ options = default_options.merge(load_options(options[:config_file], options[:quiet])).merge(options)
64
+
65
+ if options[:quiet].nil? and not command_line
66
+ options[:quiet] = true
67
+ end
68
+
69
+ options[:output_format] = get_output_format options
70
+
71
+ if options[:appname].nil?
72
+ path = options[:target]
73
+ options[:appname] = File.split(path).last
74
+ end
75
+ options
76
+ end
77
+
78
+ CONFIG_FILES = [
79
+ File.expand_path("./config/glue.yml"),
80
+ File.expand_path("~/.glue/config.yml"),
81
+ File.expand_path("/etc/glue/config.yml")
82
+ ]
83
+
84
+ #Load options from YAML file
85
+ def self.load_options custom_location, quiet
86
+ #Load configuration file
87
+ if config = config_file(custom_location)
88
+ options = YAML.load_file config
89
+
90
+ if options
91
+ options.each { |k, v| options[k] = Set.new v if v.is_a? Array }
92
+
93
+ # notify if options[:quiet] and quiet is nil||false
94
+ notify "[Notice] Using configuration in #{config}" unless (options[:quiet] || quiet)
95
+ options
96
+ else
97
+ notify "[Notice] Empty configuration file: #{config}" unless quiet
98
+ {}
99
+ end
100
+ else
101
+ {}
102
+ end
103
+ end
104
+
105
+ def self.config_file custom_location = nil
106
+ supported_locations = [File.expand_path(custom_location || "")] + CONFIG_FILES
107
+ supported_locations.detect {|f| File.file?(f) }
108
+ end
109
+
110
+ #Default set of options
111
+ def self.default_options
112
+ {
113
+ :parallel_tasks => true,
114
+ :skip_tasks => Set.new(),
115
+ :exit_on_warn => true,
116
+ :output_format => :text,
117
+ :working_dir => "~/line/tmp/",
118
+ :zap_host => "http://localhost",
119
+ :zap_port => "9999",
120
+ :labels => Set.new() << "filesystem" << "code" # Defaults to run.
121
+ }
122
+ end
123
+
124
+ #Determine output formats based on options[:output_formats]
125
+ #or options[:output_files]
126
+ def self.get_output_format options
127
+ if options[:output_file]
128
+ get_format_from_output_file options[:output_file]
129
+ elsif options[:output_format]
130
+ get_format_from_output_format options[:output_format]
131
+ else
132
+ begin
133
+ require 'terminal-table'
134
+ return [:to_s]
135
+ rescue LoadError
136
+ return [:to_json]
137
+ end
138
+ end
139
+ end
140
+
141
+ def self.get_format_from_output_format output_format
142
+ case output_format
143
+ when :html, :to_html
144
+ [:to_html]
145
+ when :csv, :to_csv
146
+ [:to_csv]
147
+ when :pdf, :to_pdf
148
+ [:to_pdf]
149
+ when :tabs, :to_tabs
150
+ [:to_tabs]
151
+ when :json, :to_json
152
+ [:to_json]
153
+ when :jira, :to_jira
154
+ [:to_jira]
155
+ when :markdown, :to_markdown
156
+ [:to_markdown]
157
+ else
158
+ [:to_s]
159
+ end
160
+ end
161
+ private_class_method :get_format_from_output_format
162
+
163
+ def self.get_format_from_output_file output_file
164
+ case output_file
165
+ when /\.html$/i
166
+ :to_html
167
+ when /\.csv$/i
168
+ :to_csv
169
+ when /\.pdf$/i
170
+ :to_pdf
171
+ when /\.tabs$/i
172
+ :to_tabs
173
+ when /\.json$/i
174
+ :to_json
175
+ when /\.md$/i
176
+ :to_markdown
177
+ else
178
+ :to_s
179
+ end
180
+ end
181
+ private_class_method :get_format_from_output_file
182
+
183
+ #Output list of tasks (for `-k` option)
184
+ def self.list_checks options
185
+ require 'glue/scanner'
186
+
187
+ add_external_tasks options
188
+
189
+ if options[:list_optional_tasks]
190
+ $stderr.puts "Optional Tasks:"
191
+ tasks = Tasks.optional_tasks
192
+ else
193
+ $stderr.puts "Available tasks:"
194
+ tasks = Tasks.tasks
195
+ end
196
+
197
+ format_length = 30
198
+
199
+ $stderr.puts "-" * format_length
200
+ tasks.each do |task|
201
+ $stderr.printf("%-#{format_length}s\n", task.name)
202
+ end
203
+ end
204
+
205
+ #Output configuration to YAML
206
+ def self.dump_config options
207
+ if options[:create_config].is_a? String
208
+ file = options[:create_config]
209
+ else
210
+ file = nil
211
+ end
212
+
213
+ options.delete :create_config
214
+
215
+ options.each do |k,v|
216
+ if v.is_a? Set
217
+ options[k] = v.to_a
218
+ end
219
+ end
220
+
221
+ if file
222
+ File.open file, "w" do |f|
223
+ YAML.dump options, f
224
+ end
225
+ puts "Output configuration to #{file}"
226
+ else
227
+ puts YAML.dump(options)
228
+ end
229
+ exit
230
+ end
231
+
232
+ #Run a scan. Generally called from Glue.run instead of directly.
233
+ def self.scan options
234
+ #Load scanner
235
+ notify "Loading scanner..."
236
+
237
+ begin
238
+ require 'glue/scanner'
239
+ require 'glue/tracker'
240
+ require 'glue/mounters'
241
+ require 'glue/filters'
242
+ require 'glue/reporters'
243
+
244
+ rescue LoadError => e
245
+ $stderr.puts e.message
246
+ raise NoGlueError, "Cannot find lib/ directory or load the key glue."
247
+ end
248
+
249
+ # debug "API: #{options[:jira_api_url.to_s]}"
250
+ # debug "Project: #{options[:jira_project.to_s]}"
251
+ # debug "Cookie: #{options[:jira_cookie.to_s]}"
252
+
253
+ add_external_tasks options
254
+
255
+ tracker = Tracker.new options
256
+ debug "Mounting ... #{options[:target]}"
257
+ # Make the target accessible.
258
+ target = Glue::Mounters.mount tracker
259
+
260
+ #Start scanning
261
+ scanner = Scanner.new
262
+ notify "Processing target...#{options[:target]}"
263
+ scanner.process target, tracker
264
+
265
+ # Filter the results (Don't report anything that has been reported before)
266
+ Glue::Filters.filter tracker
267
+
268
+ # Generate Report
269
+ notify "Generating report...#{options[:output_format]}"
270
+ Glue::Reporters.run_report tracker
271
+
272
+ tracker
273
+ end
274
+
275
+ def self.error message
276
+ $stderr.puts message
277
+ $logfile.puts "[#{Time.now}] #{message}" if $logfile
278
+ end
279
+
280
+ def self.warn message
281
+ $stderr.puts message unless @quiet
282
+ $logfile.puts "[#{Time.now}] #{message}" if $logfile
283
+ end
284
+
285
+ def self.notify message
286
+ $stderr.puts message #unless @debug
287
+ $logfile.puts "[#{Time.now}] #{message}" if $logfile
288
+ end
289
+
290
+ def self.debug message
291
+ $stderr.puts message if @debug
292
+ $logfile.puts "[#{Time.now}] #{message}" if $logfile
293
+ end
294
+
295
+ def self.load_glue_dependency name
296
+ return if @loaded_dependencies.include? name
297
+
298
+ begin
299
+ require name
300
+ rescue LoadError => e
301
+ $stderr.puts e.message
302
+ $stderr.puts "Please install the appropriate dependency."
303
+ exit! -1
304
+ end
305
+ end
306
+
307
+ def self.add_external_tasks options
308
+ options[:additional_tasks_path].each do |path|
309
+ Glue::Tasks.initialize_tasks path
310
+ end if options[:additional_tasks_path]
311
+ end
312
+
313
+ class DependencyError < RuntimeError; end
314
+ class NoGlueError < RuntimeError; end
315
+ class NoTargetError < RuntimeError; end
316
+ class JiraConfigError < RuntimeError; end
317
+ end