nexpose_ticketing 1.2.1 → 1.3.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 03778e753df032041d1c6a46bcef5f89df66d705
4
- data.tar.gz: d163c3488d78c12ade992381b06479ba8ad40be9
3
+ metadata.gz: 6d4ec39b1ff5df7eea58ca9a4ed4509a96c98025
4
+ data.tar.gz: ba214982d93d1c97b6d05afdb4932aeefeba860d
5
5
  SHA512:
6
- metadata.gz: 958e7e85146d2c2010b10723b06570203effb135017b3f280d8132d60fe5903d73b713a4d3bf9356f9dc63881067b52f8dde20e72546bad1565857bedb2284ae
7
- data.tar.gz: ebaaa9c06a62105e3cb00425f1cc7e2ed93f67d2103f8224488da3f673b226dfd73abce6a72bb389869b6aa5bb4fcafad25167356167c0ef044b7f8018c61e08
6
+ metadata.gz: 99a4eb817dff4d247dbd05dafb16069dd617ae03977b2ed059df1e288b572d456997abfb4deb448ef1697f2ad4047e30446e25246b3bd77ffcc35d2657f7a061
7
+ data.tar.gz: 6ad2f56344d10e2770b6fe5d343363666ffda12300e453b0e3523364a0880f989c9769ed0d449dc6375c076f795ffedd27ca90b434579956d8082e29d69ef472
data/Gemfile.lock ADDED
@@ -0,0 +1,69 @@
1
+ PATH
2
+ remote: .
3
+ specs:
4
+ nexpose_ticketing (1.2.1)
5
+ nexpose (~> 3.1, >= 3.1.0)
6
+ nokogiri (~> 1.6)
7
+ savon (~> 2.1)
8
+
9
+ GEM
10
+ remote: https://rubygems.org/
11
+ specs:
12
+ akami (1.2.2)
13
+ gyoku (>= 0.4.0)
14
+ nokogiri
15
+ builder (3.2.2)
16
+ diff-lcs (1.2.5)
17
+ gyoku (1.1.1)
18
+ builder (>= 2.1.2)
19
+ httpi (2.1.1)
20
+ rack
21
+ rubyntlm (~> 0.3.2)
22
+ macaddr (1.7.1)
23
+ systemu (~> 2.6.2)
24
+ mini_portile2 (2.0.0)
25
+ nexpose (3.3.0)
26
+ nokogiri (1.6.7.2)
27
+ mini_portile2 (~> 2.0.0.rc2)
28
+ nori (2.4.0)
29
+ rack (1.6.4)
30
+ rspec (3.4.0)
31
+ rspec-core (~> 3.4.0)
32
+ rspec-expectations (~> 3.4.0)
33
+ rspec-mocks (~> 3.4.0)
34
+ rspec-core (3.4.4)
35
+ rspec-support (~> 3.4.0)
36
+ rspec-expectations (3.4.0)
37
+ diff-lcs (>= 1.2.0, < 2.0)
38
+ rspec-support (~> 3.4.0)
39
+ rspec-mocks (3.4.1)
40
+ diff-lcs (>= 1.2.0, < 2.0)
41
+ rspec-support (~> 3.4.0)
42
+ rspec-support (3.4.1)
43
+ rubyntlm (0.3.4)
44
+ savon (2.5.1)
45
+ akami (~> 1.2.0)
46
+ builder (>= 2.1.2)
47
+ gyoku (~> 1.1.0)
48
+ httpi (~> 2.1.0)
49
+ nokogiri (>= 1.4.0)
50
+ nori (~> 2.4.0)
51
+ uuid (~> 2.3.7)
52
+ wasabi (~> 3.3.0)
53
+ systemu (2.6.5)
54
+ uuid (2.3.8)
55
+ macaddr (~> 1.0)
56
+ wasabi (3.3.1)
57
+ httpi (~> 2.0)
58
+ nokogiri (>= 1.4.0)
59
+
60
+ PLATFORMS
61
+ ruby
62
+
63
+ DEPENDENCIES
64
+ nexpose_ticketing!
65
+ rspec (~> 3.2, >= 3.2.0)
66
+ rspec-mocks (~> 3.2, >= 3.2.0)
67
+
68
+ BUNDLED WITH
69
+ 1.12.5
data/README.md CHANGED
@@ -4,11 +4,11 @@ This is the official gem package for the Ruby Nexpose Ticketing engine.
4
4
 
5
5
  To share your scripts, or to discuss different approaches, please visit the Rapid7 forums for Nexpose: https://community.rapid7.com/community/nexpose
6
6
 
7
- For assistance with using the gem please email the Rapid7 integrations support team at integrations_support@rapid7.com.
7
+ For assistance with using the gem please email the Rapid7 integrations support team at support@rapid7.com.
8
8
 
9
9
  ## About
10
10
 
11
- The Nexpose Ticketing integration allows customers to create incident tickets based upon vulnerabilities found across their systems. The integration runs a report for a chosen site or tag group in Nexpose and then creates tickets based on the report, either for each machine or vulnerability, as specified by the ticketing mode selected. On subsequent scans, new tickets are created, existing tickets are updated (and potentially closed if resolved) based on any differences since the scan was performed.
11
+ The Nexpose Ticketing integration allows customers to create incident tickets based upon vulnerabilities found across their systems. The integration runs a report for a chosen site or tag group in Nexpose and then creates tickets based on the report, either for each machine or vulnerability, as specified by the ticketing mode selected. On subsequent scans, new tickets are created, existing tickets are updated (and potentially closed if resolved) based on any differences since the previous scan was performed.
12
12
 
13
13
  The integration has three ticket generation modes:
14
14
  * Default mode: This mode will create one ticket per instance of a vulnerability i.e. a vulnerability present on three machines will have three tickets. This mode makes for smaller, more actionable incidents but has the potential to generate a large number of tickets. This mode can only create and close tickets. It does not update any information in existing tickets.
@@ -19,7 +19,7 @@ The integration has three ticket generation modes:
19
19
 
20
20
  `Supported Ticketing systems: JIRA; Remedy ITSM; ServiceNow; ServiceDesk`
21
21
 
22
- For more information, as well as service specific information, please see the documentation attached to the integration.
22
+ For more information, as well as service specific information, please refer to the integration documentation which can be requested from the Rapid7 support team.
23
23
 
24
24
  ## Installation
25
25
 
@@ -31,26 +31,28 @@ $ gem install nexpose_ticketing
31
31
  ```
32
32
  ## Usage
33
33
 
34
- Documentation for setting up each integration can be found attached to the gem.
34
+ Documentation for setting up each integration can be requested from support@rapid7.com
35
35
 
36
36
  To use the JIRA implementation please follow these steps:
37
37
  * Edit the jira.config file under the gem config folder and add the necessary data.
38
38
  * Edit the ticket_service.config under the gem config folder and add the necessary data.
39
- * Run the nexpose_jira file under the bin folder. If installed with gem the command `console> nexpose_jira` should suffice.
39
+ * Run the nexpose_ticketing file under the bin folder. If installed with gem the command `console> nexpose_ticketing jira` should suffice. Replace 'jira' with your chosen helper for other implementations
40
40
 
41
41
  Note: Gem is usually installed under
42
42
  * Windows: C:\Ruby\<version\>\lib\ruby\gems\version\gems
43
- * Linux: /var/lib/gems/\<version\>/gems/
43
+ * Linux: /var/lib/gems/\<version\>/gems/ or /home/\<user\>/.rvm/gems/\<version\>/gems/
44
+
44
45
  Please refer to your particular Ruby documentation for actual installation folder.
45
46
 
46
- A logger is also implemented by default, and the log can be found under `/var/lib/nexpose_ticketing/logs/`; please refer to the log file in case of an error.
47
+ A logger is also implemented by default, and the log can be found under `<install_location>/lib/nexpose_ticketing/logs/`
48
+ Please refer to the log file in case of an error.
47
49
 
48
50
  ## Contributions
49
51
 
50
52
  To develop your own implementation for Ticketing service 'foo':
51
53
 
52
54
  1. Create a helper class that implements the following methods:
53
- * Initialize: This is the constructor that will take the implementation options and the service options.
55
+ * Initialize: This is the constructor that will take the implementation options and the service options. It should inherit from the base_helper class.
54
56
  * create_ticket(tickets) - This method should implement the transport class for the 'foo' service (https, smtp, SOAP, etc).
55
57
  * prepare_create_tickets(vulnerability_list, nexpose_identifier_id) - This method will take the vulnerability_list in CSV format and transform it into 'foo' accepted data (JSON, XML, etc). The implemented helpers group data into a single ticket according to the current ticketing mode: Per IP in IP mode and per vulnerability in Vulnerability mode.
56
58
 
@@ -60,18 +62,53 @@ To develop your own implementation for Ticketing service 'foo':
60
62
  * close_tickets(tickets) - This method should implement the transport class for the 'foo' service (https, smtp, SOAP, etc), to send closure messages to the service for a specific exisiting ticket.
61
63
  * prepare_close_tickets(vulnerability_list, nexpose_identifier_id) - This method will take the vulnerability_list in CSV format and transform it into 'foo' accepted data (JSON, XML, etc) containing information about the tickets to close.
62
64
 
63
- 3. Create your 'foo' caller under bin. See the file 'jira' for reference.
65
+ 3. A configuration file will be needed in the config folder for service specific options. This is loaded at the start of operation. Please refer to the existing configuration files, as certain options are common to all services.
64
66
 
65
- Please see jira_helper.rb under helpers for an helper example, and two_vulns_report.csv under the test folder for a sample CSV report. For more information about developing a new helper, including implementing the different ticketing modes, please see the 'Developer Guide for Nexpose Ticketing'.
67
+ Please see jira\_helper.rb under helpers for an helper example, and two\_vulns\_report.csv under the test folder for a sample CSV report. For more information about developing a new helper, including implementing the different ticketing modes, please see the 'Developer Guide for Nexpose Ticketing' document.
66
68
 
67
69
  We welcome contributions to this package. We ask only that pull requests and patches adhere to our coding standards.
68
70
 
69
71
  * Favor returning classes over key-value maps. Classes tend to be easier for users to manipulate and use.
70
72
  * Unless otherwise noted, code should adhere to the [Ruby Style Guide] (https://github.com/bbatsov/ruby-style-guide).
71
73
  * Use YARDoc comment style to improve the API documentation of the gem.
74
+ * Pull requests may not be accepted for user specific use-cases.
72
75
 
73
76
  ##Changelog
74
77
 
78
+ ###1.3.0
79
+ 25-01-17
80
+
81
+ #### JIRA Helper
82
+ Improved error logging. The helper now logs meaningful data returned for each error from JIRA.
83
+
84
+ #### Historical Tracking
85
+ Previously, the last\_scan\_data file was not updated until the integration was complete. If the integration failed mid-operation, it would attempt to create tickets for all sites, even if this was previously done before the failure. Each sites' data is now updated after tickets are generated.
86
+
87
+ #### Bug Fixes
88
+ General bug fixes for most classes. Notable listed below
89
+
90
+ ###### Ticket Service
91
+ - Not correctly logging errors when the integration failed to load helper and mode classes.
92
+ - Missing the site / tag id option when scanning a new asset during a non-initial scan, causing the integration to fail
93
+ - Ticket service was calling the incorrect query when closing tickets in Default mode.
94
+
95
+ ###### Ticket Repository
96
+ - Method to generate the report in Nexpose was incorreclty applying the site id from the ticket\_service.config file, rather than the value passed in. This may have resulted in an asset in a tag group not being correctly scanned.
97
+
98
+ ###### JIRA Helper
99
+ - 'Code' error: The create\_tickets method was trying to parse the reponse code from the HTTP response from JIRA. This was causing the integration to fail on success, as the response was not returned from the send\_tickets method on success.
100
+
101
+ ###### ServiceNow Helper
102
+ - Helper now retries retrieving sys_id for tickets from ServiceNow and skips if it cannot retrieve it.
103
+
104
+ ###### Ticketing Modes
105
+ - References to individual ticketing modes were removed from the ticket\_service and ticket\_repository class. Any reference now occurs in the chosen helper class
106
+
107
+ ###### Queries
108
+ - Default mode query issue: The default mode query on a non-initial scan was previously only returning new vulnerabilities from the previous scan - if a site had been scanned more than once since the integration was run, the vulnerabilities would not have had tickets created. This now correctly returns the number of vulnerabilities.
109
+
110
+
111
+
75
112
  ###1.2.0
76
113
 
77
114
  ####Configuration Options
@@ -79,16 +116,22 @@ Ticketing mode must be specified using the entire title, rather than a single ch
79
116
  Added the following configuration option:
80
117
  - log_console - NXLogger also gets printed to the console.
81
118
 
82
- ####Ticketing Modes
83
- Ticketing modes (Default, IP, Vulnerability) are now abstracted into their own classes.
84
- CommonHelper has been replaced with BaseMode from which other modes are extended.
119
+ #### Extensibility
120
+ Code for the ticket\_service, ticket\_repository and helpers has been refactored to make it easier for the end-user to modify. Classes listed below now provide common functionality across different implementations.
121
+
122
+ ###### Ticketing Modes
123
+ - Ticketing modes (Default, IP, Vulnerability) are now abstracted into their own classes.
124
+ - CommonHelper has been replaced with BaseMode from which other modes are extended.
85
125
 
86
- ####Ticketing Helpers
87
- Ticketing helpers extend a BaseHelper class.
88
- Ticketing helpers now log the number of tickets that are opened/closed/updated.
126
+ ###### Ticketing Helpers
127
+ - Ticketing helpers extend a BaseHelper class.
128
+ - Ticketing helpers now log the number of tickets that are opened/closed/updated.
129
+ - Helpers now support all ticketing modes
89
130
 
90
131
  ###1.1.0
91
132
  10-02-2016
133
+
134
+ ##### Configuration
92
135
  Added the following configuration options:
93
136
  - max_ticket_length - Specifies a maximum length for the description field of a ticket.
94
137
  - max_title_length - Specifies a maximum length for the title of a ticket.
@@ -96,62 +139,63 @@ Added the following configuration options:
96
139
 
97
140
  ###1.0.2
98
141
  08-02-2016
99
- - Encoding is now enforced as UTF-8 when parsing CSV files - fixes environment-specific errors.
100
142
 
101
- Jira Helper:
143
+ Encoding is now enforced as UTF-8 when parsing CSV files - fixes environment-specific errors.
144
+
145
+ ##### Jira Helper:
102
146
  - Non-200 return codes are now logged when creating or updating tickets.
103
147
 
104
- NX Logger:
148
+ ##### NX Logger:
105
149
  - Fixed Windows-specific, input-related errors.
106
150
 
107
- ServiceNow:
151
+ ##### ServiceNow:
108
152
  - No longer queries for existing incident if ticket is new.
109
153
 
110
154
  ###1.0.1
111
155
  19-01-2016
112
156
 
113
- ServiceNow Helper:
157
+ ##### ServiceNow Helper:
114
158
  - New update set.
115
159
  - Not backward-compatible with previous update set.
116
160
  - Incident table now has NXID and Rapid7 ID columns.
117
161
  - Data is queried from the incident table.
118
162
  - Updates/creates entries based on a coalesce value rather than sysparm_query.
119
163
 
120
- NX Logger:
164
+ ##### NX Logger:
121
165
  - Log level set as 'info' for default value.
122
166
 
123
- Ticket Service:
167
+ ##### Ticket Service:
124
168
  - Ticket batching ensures a single ticket is not spread across multiple batches.
125
169
 
126
- Report Helper:
170
+ ##### Report Helper:
127
171
  - Temp report is flushed before being returned (preventing some timing-based issues).
128
172
 
129
173
  ###1.0.0
130
174
  10-12-2015
131
175
 
132
- Queries:
176
+ ##### Queries:
133
177
  - Fixed issue where only first source/reference pair was returned.
134
178
  - Fixed issue where only single solution may be returned.
135
179
  - Fixed issue where results would be omitted if they lacked references.
136
180
  - Added hostname information.
137
181
 
138
- - old_vulns_since_scan
139
- old_tickets_by_ip
182
+ - old\_vulns\_since\_scan
183
+ old\_tickets\_by\_ip
140
184
  - Filters for distinct IP address and vulnerability ID pairs.
141
185
 
142
- - all_new_vulns
143
- all_vulns_since_scan
186
+ - all\_new\_vulns
187
+ all\_vulns\_since\_scan
144
188
  - Filters for distinct IP address and vulnerability ID pairs.
145
189
  - Summary row replaced with "solutions" row containing all solutions to each row's vulnerability.
146
190
 
147
- - all_new_vulns_by_vuln_id
148
- new_vulns_since_scan
149
- new_vulns_by_vuln_id_since_scan
150
- all_vulns_by_vuln_id_since_scan
191
+ - all\_new\_vulns\_by\_vuln\_id
192
+ new\_vulns\_since\_scan
193
+ new\_vulns\_by\_vuln\_id\_since\_scan
194
+ all\_vulns\_by\_vuln\_id\_since\_scan
151
195
  - Vulnerabilities condensed into a single row per vuln ID and comparison status with aggregated columns for assets and solutions.
152
196
  - Summary row replaced with "solutions" row containing all solutions to each row's vulnerability.
153
197
 
154
- Remedy Helper:
198
+ ##### Remedy Helper:
155
199
  - NXID and ticket description generation moved to CommonHelper class.
156
200
  - Savon client generation refactored into method.
157
201
  - Update/close/create ticket sending refactored into new method.
@@ -165,7 +209,7 @@ Remedy Helper:
165
209
  - Method "ticket_from_queried_incident" introduced to consolidate filling out ticket information (from "extract_queried_incident" and "prepare_close_tickets")
166
210
  - Hostname information added to vulnerability mode tickets.
167
211
 
168
- ServiceNow Helper:
212
+ ##### ServiceNow Helper:
169
213
  - Vulnerability mode implemented.
170
214
  - NXID and ticket description generation moved to CommonHelper class.
171
215
  - Logic for create/update methods consolidated into single prepare_tickets method.
@@ -173,18 +217,18 @@ ServiceNow Helper:
173
217
  - Queries for tickets updated so that only active tickets are returned.
174
218
  - Generates valid tickets for new machines which appear in noninitial scans.
175
219
 
176
- ServiceDesk Helper:
220
+ ##### ServiceDesk Helper:
177
221
  - Vulnerability mode implemented.
178
222
  - NXID and ticket description generation moved to CommonHelper class.
179
223
  - Added extra information to tickets (parity with Jira)
180
224
  - Logic for create/update methods consolidated into single prepare_tickets method.
181
225
  - Ticket update/creation logic updated to account for ticket being built up over multiple rows.
182
226
 
183
- JiraHelper:
227
+ ##### JiraHelper:
184
228
  - Vulnerability mode implemented.
185
229
  - NXID and ticket description generation moved to CommonHelper class.
186
230
  - Logic for create/update methods consolidated into single prepare_tickets method.
187
231
  - Fixed issue where fields could be emptied on ticket update.
188
232
 
189
- CommonHelper:
233
+ ##### CommonHelper:
190
234
  - New class for NXID generation and ticket formatting.
@@ -32,8 +32,8 @@ rescue ArgumentError => e
32
32
  end
33
33
 
34
34
  log = NexposeTicketing::NxLogger.instance
35
- log.setup_statistics_collection(service_options["vendor"],
36
- service_options["product"],
35
+ log.setup_statistics_collection(service_options[:vendor],
36
+ service_options[:product],
37
37
  NexposeTicketing::VERSION)
38
38
  log.setup_logging(true, 'info')
39
39
 
@@ -42,4 +42,4 @@ current_encoding = Encoding.default_external=Encoding.find("UTF-8")
42
42
  log.log_message("Current Encoding set to: #{current_encoding}")
43
43
 
44
44
  # Initialize Ticket Service using JIRA.
45
- NexposeTicketing.start(service_options)
45
+ NexposeTicketing.start(service_options)
@@ -186,7 +186,8 @@ class JiraHelper < BaseHelper
186
186
  req = Net::HTTP::Post.new(uri, headers)
187
187
  req.body = ticket
188
188
 
189
- code = send_ticket(uri, req).code.to_i
189
+ response = send_ticket(uri, req)
190
+ code = response.nil? ? 1 : response.code.to_i
190
191
  break if code.between?(400, 499)
191
192
 
192
193
  created_tickets += 1
@@ -283,14 +284,14 @@ class JiraHelper < BaseHelper
283
284
  @log.log_message("Closing ticket <#{ticket}> requires a field I know nothing about! Transition details are <#{transition}>. Ignoring this field.")
284
285
  next
285
286
  end
286
-
287
- field = "{\"id\" : \"#{field[1]['allowedValues'][0]['id']}\"}"
288
- field = "[#{field}]" if field[1]['schema']['type'] == 'array'
289
- required_fields << "\"#{field[0]}\" : #{field}"
287
+ val = "{\"id\" : \"#{field[1]['allowedValues'][0]['id']}\"}"
288
+ val = "[#{field}]" if field[1]['schema']['type'] == 'array'
289
+ required_fields << "\"#{field[0]}\" : #{val}"
290
290
  end
291
291
 
292
292
  req.body = "{\"transition\" : {\"id\" : #{transition['id']}}, \"fields\" : { #{required_fields.join(",")}}}"
293
- code = send_ticket(uri, req).code.to_i
293
+ response = send_ticket(uri, req)
294
+ code = response.nil? ? 1 : response.code.to_i
294
295
  break if code.between?(400, 499)
295
296
 
296
297
  closed_count += 1
@@ -354,7 +355,8 @@ class JiraHelper < BaseHelper
354
355
  req.body = {'update' => {'description' => [{'set' => "#{JSON.parse(ticket_details[1])['fields']['description']}"}]}}.to_json
355
356
  end
356
357
 
357
- code = send_ticket(URI.parse(url), req).code.to_i
358
+ response = send_ticket(URI.parse(url), req)
359
+ code = response.nil? ? 1 : response.code.to_i
358
360
  break if code.between?(400, 499)
359
361
 
360
362
  if create_new_ticket
@@ -53,12 +53,12 @@ class ServiceNowHelper < BaseHelper
53
53
  # - +tickets+ - List of JSON-formatted ticket closures.
54
54
  #
55
55
  def close_tickets(tickets)
56
- @metrics.closed tickets.count
57
-
58
56
  if tickets.nil? || tickets.empty?
59
57
  @log.log_message('No tickets to close.')
60
58
  return
61
59
  end
60
+ @metrics.closed tickets.count
61
+
62
62
  tickets.each do |ticket|
63
63
  send_ticket(ticket, @service_data[:servicenow_url], @service_data[:redirect_limit])
64
64
  end
@@ -90,9 +90,12 @@ class ServiceNowHelper < BaseHelper
90
90
  end
91
91
 
92
92
  begin
93
+ retries ||= 0
93
94
  response = resp.request(req)
94
95
  rescue Exception => e
95
- @log.log_error_message("Request failed for NXID #{nxid}.\n#{e}")
96
+ @log.log_error_message("Request failed for NXID #{nxid}.\n#{e}. Retry #{retries}")
97
+ retry if (retries += 1) < 3
98
+ return
96
99
  end
97
100
 
98
101
  tickets = JSON.parse(response.body)
@@ -178,7 +181,7 @@ class ServiceNowHelper < BaseHelper
178
181
  matching_fields = @mode_helper.get_matching_fields
179
182
  @ticket = Hash.new(-1)
180
183
 
181
- @log.log_message("Preparing tickets in #{options[:ticket_mode]} address.")
184
+ @log.log_message("Preparing tickets in #{options[:ticket_mode]} mode.")
182
185
  tickets = []
183
186
  previous_row = nil
184
187
  description = nil
@@ -220,7 +223,7 @@ class ServiceNowHelper < BaseHelper
220
223
  else
221
224
  unless row['comparison'].nil? || row['comparison'] == 'New'
222
225
  @ticket['sysparm_action'] = 'update'
223
- end
226
+ end
224
227
  description = @mode_helper.update_description(description, row)
225
228
  end
226
229
  end
@@ -288,4 +291,4 @@ class ServiceNowHelper < BaseHelper
288
291
  end
289
292
  tickets
290
293
  end
291
- end
294
+ end
@@ -146,37 +146,61 @@ module NexposeTicketing
146
146
  # |url| |summary| |fix|
147
147
  #
148
148
  def self.new_vulns_since_scan_by_ip(options = {})
149
- "SELECT DISTINCT on (da.ip_address, subs.vulnerability_id) subs.asset_id, da.ip_address, da.host_name, subs.current_scan,
150
- subs.vulnerability_id, dv.nexpose_id as vuln_nexpose_id,
151
- string_agg(DISTINCT 'Summary: ' || coalesce(ds.summary, 'None') ||
152
- '|Nexpose ID: ' || ds.nexpose_id ||
153
- '|Fix: ' || coalesce(proofAsText(ds.fix), 'None') ||
154
- '|URL: ' || coalesce(ds.url, 'None'), '~') as solutions,
155
- fa.riskscore, dv.cvss_score,
156
- string_agg(DISTINCT dvr.source || ': ' || dvr.reference, ', ') as references,
157
- fasva.first_discovered, fasva.most_recently_discovered
158
- FROM (SELECT fasv.asset_id, fasv.vulnerability_id, s.current_scan
159
- FROM fact_asset_scan_vulnerability_finding fasv
160
- JOIN
161
- (
162
- SELECT asset_id, previousScan(asset_id) AS baseline_scan, lastScan(asset_id) AS current_scan
163
- FROM dim_asset #{createAssetString(options)}) s
164
- ON s.asset_id = fasv.asset_id AND (fasv.scan_id >= #{options[:scan_id]} OR fasv.scan_id = s.current_scan)
165
- GROUP BY fasv.asset_id, fasv.vulnerability_id, s.current_scan
166
- HAVING baselineComparison(fasv.scan_id, current_scan) = 'New'
167
- ) subs
168
- JOIN dim_vulnerability dv USING (vulnerability_id)
169
- LEFT JOIN dim_vulnerability_reference dvr USING (vulnerability_id)
149
+ "SELECT fin.asset_id, fin.ip_address, fin.host_name, fin.current_scan,
150
+ fin.vulnerability_id, fin.vuln_nexpose_id,
151
+ string_agg(DISTINCT 'Summary: ' || coalesce(ds.summary, 'None') ||
152
+ '|Nexpose ID: ' || ds.nexpose_id ||
153
+ '|Fix: ' || coalesce(proofAsText(ds.fix), 'None') ||
154
+ '|URL: ' || coalesce(ds.url, 'None'), '~') as solutions,
155
+ fin.riskscore, fin.cvss_score, fin.references,
156
+ fin.first_discovered, fin.most_recently_discovered
157
+ FROM(
158
+ SELECT scns.asset_id, scns.ip_address, scns.host_name, scns.current_scan,
159
+ scns.scan_id, scns.vulnerability_id, dv.nexpose_id as vuln_nexpose_id,
160
+ fa.riskscore, dv.cvss_score,
161
+ string_agg(DISTINCT dvr.source || ': ' || dvr.reference, ', ') as references,
162
+ fasva.first_discovered, fasva.most_recently_discovered
163
+ FROM(
164
+ SELECT subs.asset_id, subs.ip_address, subs.host_name,
165
+ subs.vulnerability_id, subs.scan_id, subs.current_scan
166
+ FROM(
167
+ SELECT DISTINCT on (fasv.asset_id, fasv.vulnerability_id)
168
+ fasv.asset_id, fasv.vulnerability_id, s.ip_address, s.host_name,
169
+ fasv.scan_id, s.current_scan
170
+ FROM fact_asset_scan_vulnerability_finding fasv
171
+ JOIN (
172
+ SELECT asset_id, ip_address, host_name,
173
+ lastScan(asset_id) AS current_scan
174
+ FROM dim_asset #{createAssetString(options)}
175
+ ) s
176
+ ON s.asset_id = fasv.asset_id AND
177
+ (fasv.scan_id >= #{options[:scan_id]} OR fasv.scan_id = s.current_scan)
178
+ WHERE s.current_scan > #{options[:scan_id]}
179
+ GROUP BY fasv.asset_id, fasv.vulnerability_id, s.ip_address,
180
+ s.host_name, fasv.scan_id, s.current_scan
181
+ ORDER BY fasv.asset_id, fasv.vulnerability_id, s.ip_address,
182
+ s.host_name, fasv.scan_id
183
+ )subs
184
+ WHERE subs.scan_id > #{options[:scan_id]}
185
+ )scns
186
+ JOIN dim_vulnerability dv USING (vulnerability_id)
187
+ LEFT JOIN dim_vulnerability_reference dvr USING (vulnerability_id)
188
+ JOIN fact_asset_vulnerability_age fasva ON
189
+ scns.vulnerability_id = fasva.vulnerability_id AND
190
+ scns.asset_id = fasva.asset_id
191
+ JOIN fact_asset fa ON fa.asset_id = scns.asset_id
192
+ #{createRiskString(options[:riskScore])}
193
+ GROUP BY scns.asset_id, scns.ip_address, scns.host_name,
194
+ scns.current_scan, scns.vulnerability_id, dv.nexpose_id, fa.riskscore,
195
+ dv.cvss_score, scns.scan_id, fasva.first_discovered,
196
+ fasva.most_recently_discovered
197
+ )fin
170
198
  JOIN dim_asset_vulnerability_solution davs USING (vulnerability_id)
171
- JOIN fact_asset_vulnerability_age fasva ON subs.vulnerability_id = fasva.vulnerability_id AND subs.asset_id = fasva.asset_id
172
199
  JOIN dim_solution ds USING (solution_id)
173
- JOIN dim_asset da ON subs.asset_id = da.asset_id
174
- AND subs.current_scan > #{options[:scan_id]}
175
- JOIN fact_asset fa ON fa.asset_id = da.asset_id
176
- #{createRiskString(options[:riskScore])}
177
- GROUP BY subs.asset_id, da.ip_address, da.host_name, subs.current_scan, subs.vulnerability_id, dv.nexpose_id,
178
- fa.riskscore, dv.cvss_score, fasva.first_discovered, fasva.most_recently_discovered
179
- ORDER BY da.ip_address, subs.vulnerability_id"
200
+ GROUP BY fin.asset_id, fin.ip_address, fin.host_name, fin.current_scan,
201
+ fin.vulnerability_id, fin.vuln_nexpose_id, fin.riskscore, fin.cvss_score,
202
+ fin.scan_id, fin.references, fin.first_discovered, fin.most_recently_discovered
203
+ ORDER BY fin.ip_address, fin.vulnerability_id"
180
204
  end
181
205
 
182
206
 
@@ -227,6 +251,7 @@ module NexposeTicketing
227
251
 
228
252
 
229
253
  # Gets all old vulnerabilities happening after a reported scan id.
254
+ # Used in default mode to return tickets to close
230
255
  #
231
256
  # * *Args* :
232
257
  # - +reported_scan+ - Last reported scan id.
@@ -235,7 +260,7 @@ module NexposeTicketing
235
260
  # - Returns |asset_id| |ip_address| |current_scan| |vulnerability_id| |solution_id| |nexpose_id|
236
261
  # |url| |summary| |fix|
237
262
  #
238
- def self.old_vulns_since_scan(options = {})
263
+ def self.old_vulns_since_scan_by_ip(options = {})
239
264
  "SELECT DISTINCT on (da.ip_address, subs.vulnerability_id) subs.asset_id, da.ip_address, da.host_name, subs.current_scan, subs.vulnerability_id, dvs.solution_id, ds.nexpose_id, ds.url,
240
265
  proofAsText(ds.summary) as summary, proofAsText(ds.fix) as fix, subs.comparison, fa.riskscore
241
266
  FROM (
@@ -227,8 +227,13 @@ module NexposeTicketing
227
227
  end
228
228
 
229
229
  def request_query(query_name, options = {}, nexpose_items = nil)
230
- items = Array(nexpose_items || options["#{options[:scan_mode]}s".intern])
231
-
230
+ items =
231
+ if nexpose_items
232
+ Array(nexpose_items)
233
+ else
234
+ options[:nexpose_item] ? nil : options["#{options[:scan_mode]}s".intern]
235
+ end
236
+
232
237
  report_config = generate_config(query_name, options, items)
233
238
  end
234
239
 
@@ -160,8 +160,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
160
160
  log_message('Preparing tickets.')
161
161
 
162
162
  ticket_rate_limiter(all_vulns_file, 'create', item)
163
-
164
- post_scan item
163
+
164
+ post_scan(item_id: item, generate_asset_list: true)
165
165
  end
166
166
 
167
167
  log_message('Finished processing all vulnerabilities.')
@@ -172,18 +172,19 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
172
172
  # Compares the scan information from file && Nexpose.
173
173
  no_processing = true
174
174
  @latest_scans.each do |item_id, last_scan_id|
175
- # There's no entry in the file, so it's either a new item in Nexpose or a new item we have to monitor.
176
175
  prev_scan_id = scan_histories[item_id]
177
176
 
177
+ # There's no entry in the file, so it's either a new item in Nexpose or a new item we have to monitor.
178
178
  if prev_scan_id.nil? || prev_scan_id == -1
179
179
  options[:nexpose_item] = item_id
180
180
  full_new_site_report(item_id, ticket_repository, options, helper)
181
181
  options[:nexpose_item] = nil
182
- post_scan item_id
182
+ post_scan(item_id: item_id, generate_asset_list: true)
183
183
  no_processing = false
184
184
  # Site has been scanned since last seen according to the file.
185
185
  elsif prev_scan_id.to_s != @latest_scans[item_id].to_s
186
186
  delta_new_scan(item_id, options, helper, scan_histories)
187
+ post_scan item_id: item_id
187
188
  no_processing = false
188
189
  end
189
190
  end
@@ -191,9 +192,9 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
191
192
  log_name = @options["#{@options[:scan_mode]}_file_name".to_sym]
192
193
  # Done processing, update the CSV to the latest scan info.
193
194
  if no_processing
194
- log_message("Nothing new to process, updating historical CSV file #{log_name}.")
195
+ log_message("Nothing new to process, historical CSV file has not been updated: #{options[:file_name]}.")
195
196
  else
196
- log_message("Done processing, updating historical CSV file #{log_name}.")
197
+ log_message("Done processing, historical CSV file has been updated: #{options[:file_name]}.")
197
198
  end
198
199
  no_processing
199
200
  end
@@ -201,6 +202,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
201
202
  # There's a new site we haven't seen before.
202
203
  def full_new_site_report(nexpose_item, ticket_repository, options, helper)
203
204
  log_message("New nexpose id: #{nexpose_item} detected. Generating report.")
205
+ options[:scan_id] = 0
204
206
  new_item_vuln_file = ticket_repository.all_new_vulns(options, nexpose_item)
205
207
  log_message('Report generated, preparing tickets.')
206
208
 
@@ -301,13 +303,11 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
301
303
 
302
304
  def full_scan
303
305
  log_message('Storing current scan state before obtaining all vulnerabilities.')
304
- current_scan_state = ticket_repository.load_last_scans(@options)
306
+ @current_scan_state = ticket_repository.load_last_scans(@options)
305
307
 
306
308
  all_site_report(@ticket_repository, @options)
307
309
 
308
310
  #Generate historical CSV file after completing the fist query.
309
- log_message('No historical CSV file found. Generating.')
310
- @ticket_repository.save_to_file(@historical_file, current_scan_state)
311
311
  log_message('Historical CSV file generated.')
312
312
  end
313
313
 
@@ -319,14 +319,13 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
319
319
  # about to process and move this to the historical file if we
320
320
  # successfully process.
321
321
  log_message('Calculated deltas, storing current scan state.')
322
- current_scan_state = ticket_repository.load_last_scans(@options)
322
+ @current_scan_state = ticket_repository.load_last_scans(@options)
323
323
 
324
324
  # Only run if a scan has been ran ever in Nexpose.
325
325
  return if @latest_scans.empty?
326
326
 
327
327
  delta_site_report(@ticket_repository, @options, @helper, scan_histories)
328
- # Processing completed successfully. Update historical scan file.
329
- @ticket_repository.save_to_file(@historical_file, current_scan_state)
328
+ log_message('Historical CSV file updated.')
330
329
  end
331
330
 
332
331
  # Performs a delta scan
@@ -355,18 +354,20 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
355
354
 
356
355
  if @mode.updates_supported?
357
356
  helper_method = 'update'
358
- query = 'all_vulns_since_scan'
357
+ new_vulns_query = 'all_vulns_since_scan'
358
+ old_vulns_query = 'old_tickets'
359
359
  else
360
360
  helper_method = 'create'
361
- query = 'new_vulns_since_scan'
361
+ new_vulns_query = 'new_vulns_since_scan'
362
+ old_vulns_query = 'old_vulns_since_scan'
362
363
  end
363
364
 
364
- all_scan_vuln_file = ticket_repository.send(query, scan_options)
365
+ all_scan_vuln_file = ticket_repository.send(new_vulns_query, scan_options)
365
366
  ticket_rate_limiter(all_scan_vuln_file, helper_method, nexpose_id)
366
367
 
367
368
  return unless options[:close_old_tickets_on_update] == 'Y'
368
369
 
369
- tickets_to_close_file = ticket_repository.old_tickets(scan_options)
370
+ tickets_to_close_file = ticket_repository.send(old_vulns_query, scan_options)
370
371
  ticket_rate_limiter(tickets_to_close_file, 'close', nexpose_id)
371
372
  end
372
373
 
@@ -411,20 +412,59 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
411
412
  end
412
413
 
413
414
  # Methods to run after a scan
414
- def post_scan(item_id)
415
- self.send("post_#{@options[:scan_mode]}_scan", item_id)
415
+ def post_scan(**modifiers)
416
+ self.send("post_#{@options[:scan_mode]}_scan", modifiers)
417
+ scan_history = self.send("get_#{@options[:scan_mode]}_file_header")
418
+
419
+ item_id = modifiers[:item_id]
420
+ historic_data = nil
421
+ if File.exists?(@historical_file)
422
+ log_message("Updating historical CSV file: #{@historical_file}.")
423
+ historic_data = []
424
+ CSV.foreach(@historical_file, headers: true) { |r| historic_data << r }
425
+ end
426
+
427
+ updated_row = [@current_scan_state.find { |row| row[0].eql?(item_id) }]
428
+
429
+ if historic_data.nil?
430
+ log_message('No historical CSV file found. Generating.')
431
+ scan_history.concat(updated_row)
432
+ else
433
+ index = historic_data.find_index { |id| id[0] == item_id }
434
+ if index.nil?
435
+ historic_data.concat(updated_row)
436
+ historic_data.sort! { |x,y| x[0].to_i <=> y[0].to_i }
437
+ else
438
+ historic_data[index] = updated_row
439
+ historic_data.flatten!
440
+ end
441
+ scan_history.concat(historic_data)
442
+ end
443
+
444
+ log_message('Updated historical CSV file for ' \
445
+ "#{@options[:scan_mode]}: #{item_id}.")
446
+ @ticket_repository.save_to_file(@historical_file, scan_history)
416
447
  end
417
448
 
418
- def post_site_scan(item_id)
449
+ def post_site_scan(**modifiers)
419
450
  end
420
451
 
421
- def post_tag_scan(item_id)
422
- tag_assets_historic_file = File.join(File.dirname(__FILE__),
423
- 'tag_assets',
424
- "#{options[:tag_file_name]}_#{item_id}.csv")
425
-
426
- ticket_repository.generate_tag_asset_list(tags: item_id,
427
- csv_file: tag_assets_historic_file)
452
+ def post_tag_scan(**modifiers)
453
+ return unless modifiers[:generate_asset_list]
454
+ file_name = "#{@options[:tag_file_name]}_#{modifiers[:item_id]}.csv"
455
+ historic_file = File.join(File.dirname(__FILE__), 'tag_assets', file_name)
456
+
457
+ log_message("Generating current tag asset file: #{historic_file}.")
458
+ ticket_repository.generate_tag_asset_list(tags: modifiers[:item_id],
459
+ csv_file: historic_file)
460
+ end
461
+
462
+ def get_site_file_header
463
+ ['site_id,last_scan_id,finished']
464
+ end
465
+
466
+ def get_tag_file_header
467
+ ['tag_id,last_scan_fingerprint']
428
468
  end
429
469
 
430
470
  # Formats the Nexpose item ID according to the asset grouping mode
@@ -1,3 +1,3 @@
1
1
  module NexposeTicketing
2
- VERSION = "1.2.1"
2
+ VERSION = "1.3.0"
3
3
  end
metadata CHANGED
@@ -1,16 +1,17 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: nexpose_ticketing
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.1
4
+ version: 1.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Damian Finol
8
8
  - JJ Cassidy
9
9
  - David Valente
10
+ - Adam Robinson
10
11
  autorequire:
11
12
  bindir: bin
12
13
  cert_chain: []
13
- date: 2016-07-20 00:00:00.000000000 Z
14
+ date: 2017-01-26 00:00:00.000000000 Z
14
15
  dependencies:
15
16
  - !ruby/object:Gem::Dependency
16
17
  name: nexpose
@@ -103,7 +104,7 @@ dependencies:
103
104
  description: This gem provides a Ruby implementation of different integrations with
104
105
  ticketing services for Nexpose.
105
106
  email:
106
- - integrations_support@rapid7.com
107
+ - support@rapid7.com
107
108
  executables:
108
109
  - nexpose_ticketing
109
110
  extensions: []
@@ -111,6 +112,7 @@ extra_rdoc_files:
111
112
  - README.md
112
113
  files:
113
114
  - Gemfile
115
+ - Gemfile.lock
114
116
  - README.md
115
117
  - bin/nexpose_ticketing
116
118
  - lib/nexpose_ticketing.rb
@@ -158,7 +160,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
158
160
  version: '0'
159
161
  requirements: []
160
162
  rubyforge_project:
161
- rubygems_version: 2.4.8
163
+ rubygems_version: 2.5.1
162
164
  signing_key:
163
165
  specification_version: 4
164
166
  summary: Ruby Nexpose Ticketing Engine.