heimdall_tools 1.2.0 → 1.3.20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 16faf735e553864867f8e551ea92b96621eeb415
-  data.tar.gz: 2baa94079e6561d38c5c8511e39691a15d360619
+SHA256:
+  metadata.gz: 83950f4cf536e2df5b1fc18b6a5a910623c80ac1a064be6b7b1281b6dec61b74
+  data.tar.gz: cca52d5d8bf483a372029578277039303017bd02b73273d01e106b41e772fbfe
 SHA512:
-  metadata.gz: ba635075511c4299786f38a763625197da934bf3aa72d992c9da464bdfc8f992f7734d26907aaf67aa71c40c9f37f22232e32173e44e7302d06254a788c7a618
-  data.tar.gz: 7dbe62ad30fef7ed3a6eef4570ad5e4500345947013b2d152842c024fbffb4a5bf640f431861443ad49d1d5d99c429db7d39d8149c05c97a2b0e3b69480c2563
+  metadata.gz: bed11c34a71d7c8e893e1a4402abdb95b5ed5d6b959d7ebe117e0b3ef08cef650c2903da6074752202e7b603b3511eb3084dcf92729f603e6e97fa26a1b12556
+  data.tar.gz: 964c873a24db4ec4620435f3e3a08f48f1aaa1e54b04b1eca93dbbd3a9a49c6c7fe15a5d45c0e2d06d969daac99c43ad906cb023c232be12dabfdeae318308ee

data/CHANGELOG.md

@@ -1,7 +1,154 @@
-# Change Log
+# Changelog
 
-All notable changes to this project will be documented in this file.
-This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
+## [Unreleased](https://github.com/mitre/heimdall_tools/tree/HEAD)
 
-## [1.0.1]
-- Initial internal release.
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.19...HEAD)
+
+**Fixed bugs:**
+
+- Unable to Convert Fortify 19.2.0 FVDL file to HDF [\#25](https://github.com/mitre/heimdall_tools/issues/25)
+
+## [v1.3.19](https://github.com/mitre/heimdall_tools/tree/v1.3.19) (2020-03-30)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.18...v1.3.19)
+
+**Merged pull requests:**
+
+- Remove all gems from Gemfile and declare them properly in the gemspec [\#33](https://github.com/mitre/heimdall_tools/pull/33) ([rbclark](https://github.com/rbclark))
+
+## [v1.3.18](https://github.com/mitre/heimdall_tools/tree/v1.3.18) (2020-03-28)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.17...v1.3.18)
+
+## [v1.3.17](https://github.com/mitre/heimdall_tools/tree/v1.3.17) (2020-03-26)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.16...v1.3.17)
+
+**Closed issues:**
+
+- Request New converters [\#23](https://github.com/mitre/heimdall_tools/issues/23)
+
+## [v1.3.16](https://github.com/mitre/heimdall_tools/tree/v1.3.16) (2020-03-25)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.15...v1.3.16)
+
+## [v1.3.15](https://github.com/mitre/heimdall_tools/tree/v1.3.15) (2020-03-25)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.14...v1.3.15)
+
+## [v1.3.14](https://github.com/mitre/heimdall_tools/tree/v1.3.14) (2020-03-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.13...v1.3.14)
+
+## [v1.3.13](https://github.com/mitre/heimdall_tools/tree/v1.3.13) (2020-03-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.12...v1.3.13)
+
+## [v1.3.12](https://github.com/mitre/heimdall_tools/tree/v1.3.12) (2020-03-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.11...v1.3.12)
+
+## [v1.3.11](https://github.com/mitre/heimdall_tools/tree/v1.3.11) (2020-03-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.10...v1.3.11)
+
+## [v1.3.10](https://github.com/mitre/heimdall_tools/tree/v1.3.10) (2020-03-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.9...v1.3.10)
+
+## [v1.3.9](https://github.com/mitre/heimdall_tools/tree/v1.3.9) (2020-03-23)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.8...v1.3.9)
+
+**Closed issues:**
+
+- Update XML parser [\#26](https://github.com/mitre/heimdall_tools/issues/26)
+
+**Merged pull requests:**
+
+- Update XML parser [\#27](https://github.com/mitre/heimdall_tools/pull/27) ([rx294](https://github.com/rx294))
+
+## [v1.3.8](https://github.com/mitre/heimdall_tools/tree/v1.3.8) (2020-03-09)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.7...v1.3.8)
+
+**Closed issues:**
+
+- \[BUG\] | sonarqube\_mapper is not handling NIST mapping correctly [\#21](https://github.com/mitre/heimdall_tools/issues/21)
+
+**Merged pull requests:**
+
+- Fixes \#21 \[BUG\] | sonarqube\_mapper is not handling NIST mapping correctly [\#22](https://github.com/mitre/heimdall_tools/pull/22) ([rx294](https://github.com/rx294))
+
+## [v1.3.7](https://github.com/mitre/heimdall_tools/tree/v1.3.7) (2020-03-06)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.6...v1.3.7)
+
+## [v1.3.6](https://github.com/mitre/heimdall_tools/tree/v1.3.6) (2020-03-05)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.5...v1.3.6)
+
+## [v1.3.5](https://github.com/mitre/heimdall_tools/tree/v1.3.5) (2020-03-05)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.4...v1.3.5)
+
+## [v1.3.4](https://github.com/mitre/heimdall_tools/tree/v1.3.4) (2020-03-04)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.3...v1.3.4)
+
+**Closed issues:**
+
+- Support Authenticated Sonarqube API  for sonarqube\_mapper [\#18](https://github.com/mitre/heimdall_tools/issues/18)
+
+## [v1.3.3](https://github.com/mitre/heimdall_tools/tree/v1.3.3) (2020-03-04)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.2...v1.3.3)
+
+**Merged pull requests:**
+
+- Sonarqube authentication option [\#20](https://github.com/mitre/heimdall_tools/pull/20) ([rx294](https://github.com/rx294))
+
+## [v1.3.2](https://github.com/mitre/heimdall_tools/tree/v1.3.2) (2019-12-27)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.1...v1.3.2)
+
+**Merged pull requests:**
+
+- Adding dockerfile for heimdall tools [\#15](https://github.com/mitre/heimdall_tools/pull/15) ([rx294](https://github.com/rx294))
+
+## [v1.3.1](https://github.com/mitre/heimdall_tools/tree/v1.3.1) (2019-12-27)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/v1.3.0...v1.3.1)
+
+**Closed issues:**
+
+- Update HDF format generate jsons in Inspec results style [\#10](https://github.com/mitre/heimdall_tools/issues/10)
+
+**Merged pull requests:**
+
+- Updating required nori gem version [\#16](https://github.com/mitre/heimdall_tools/pull/16) ([rx294](https://github.com/rx294))
+- Populate shasum and runtime field  [\#14](https://github.com/mitre/heimdall_tools/pull/14) ([rx294](https://github.com/rx294))
+- Updates as per feedback [\#13](https://github.com/mitre/heimdall_tools/pull/13) ([rx294](https://github.com/rx294))
+- updating samples  [\#12](https://github.com/mitre/heimdall_tools/pull/12) ([rx294](https://github.com/rx294))
+- Change to results view on heimdall [\#11](https://github.com/mitre/heimdall_tools/pull/11) ([rx294](https://github.com/rx294))
+
+## [v1.3.0](https://github.com/mitre/heimdall_tools/tree/v1.3.0) (2019-09-24)
+
+[Full Changelog](https://github.com/mitre/heimdall_tools/compare/c9c08305796eaf12d7abb2535c285a4acd2f5a91...v1.3.0)
+
+**Closed issues:**
+
+- README needs authors [\#9](https://github.com/mitre/heimdall_tools/issues/9)
+- Get NIST rev version from CSV [\#4](https://github.com/mitre/heimdall_tools/issues/4)
+- Output in evaluation format, not profile  [\#2](https://github.com/mitre/heimdall_tools/issues/2)
+
+**Merged pull requests:**
+
+- Fixes to PR \#6 [\#8](https://github.com/mitre/heimdall_tools/pull/8) ([rx294](https://github.com/rx294))
+- Update README fortify-fvdl flag to fvdl as in usage [\#7](https://github.com/mitre/heimdall_tools/pull/7) ([mirskiy](https://github.com/mirskiy))
+- Add SonarQube Mapper and OWASP NIST mappings [\#6](https://github.com/mitre/heimdall_tools/pull/6) ([mirskiy](https://github.com/mirskiy))
+- OWASP ZAP Mapper PR [\#3](https://github.com/mitre/heimdall_tools/pull/3) ([rx294](https://github.com/rx294))
+
+
+
+\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*

data/README.md

@@ -1,6 +1,8 @@
-# WIP - ALPHA
+# Heimdall Tools
 
-# HeimdallTools
+![Overall Status](https://github.com/mitre/heimdall_tools/workflows/heimdall_tools/badge.svg)  
+
+![Heimdall Tools Build](https://github.com/mitre/heimdall_tools/workflows/Build%20and%20run%20heimdall_tools/badge.svg)
 
 HeimdallTools supplies several methods to convert output from various tools to "Heimdall Data Format"(HDF) format to be viewable in Heimdall. The converters in version 1.1.1 are from:
 
@@ -41,12 +43,17 @@ sonarqube_mapper pulls SonarQube results, for the specified project, from the AP
 USAGE: heimdall_tools sonarqube_mapper [OPTIONS] -n <project-name> -u <api-url> -o <scan-results.json>
 
 FLAGS:
-    -n --name <project-name>         : name of the project in SonarQube, aka Project Key
+    -n --name <project-key>         : Project Key of the project in SonarQube
     -u --api_url <api-url>           : url of the SonarQube Server API. Typically ends with /api.
+    --auth <credentials>              : username:password or token [optional].
     -o --output <scan-results>       : path to output scan-results json.
     -V --verbose                     : verbose run [optional].
 
-example: heimdall_tools sonarqube_mapper -n sonar_project -u http://sonar:9000/api -o scan_results.json
+example: 
+
+heimdall_tools sonarqube_mapper -n sonar_project_key -u http://sonar:9000/api -o scan_results.json
+
+heimdall_tools sonarqube_mapper -n sonar_project_key -u http://sonar:9000/api --auth admin:admin -o scan_results.json
 ```
 
 ## fortify_mapper
@@ -90,6 +97,42 @@ USAGE: heimdall_tools version
 
 # Development
 
+## Submitting a PR  
+
+### A complete PR should include 7 core elements:  
+
+- A signed PR ( aka `git commit -a -s` )
+- Code for the new functionality
+- Updates to the CLI
+- New unit tests for the functionality
+- Updates to the docs and examples in `README.md` and `./docs/*`
+- (if needed) Example / Template files ( `metadata.yml`,`example.yml`, etc )
+  - Scripts / Scaffolding code for the Example / Template files ( `generate_map` is an example )
+- Example Output of the new functionality if it produces an artifact
+
+### Overview of our PR process 
+
+1. open an issue on the main inspec_tools website noting the issues your PR will address
+2. fork the repo
+3. checkout your repo
+4. cd to the repo
+5. git co -b `<your_branch>`
+6. bundle install
+7. `hack as you will`
+8. test via rake
+9. ensure unit tests still function and add unit tests for your new feature
+10. add new docs to the `README.md` and to `./docs/examples`
+11. update the CLI as needed and add in `usage` example
+12. (if needed) create and document any example or templates
+13. (if needed) create any supporing scripts
+14. (opt) gem build inspec_tools.gemspec
+15. (opt) gem install inspec_tools
+16. (opt) test via the installed gem
+17. git commit -a -s `<your_branch>`
+18. Open a PRs aginst the MITRE inspec_tools repo
+
+# Testing
+
 This gem was developed using the [CLI Template](https://github.com/tongueroo/cli-template), a generator tool that builds a starter CLI project.
 
 There are a set of unit tests. Run `rake test` to run the tests.

data/exe/heimdall_tools

@@ -1,4 +1,4 @@
-#!/usr/bin/env ruby
+#!/usr/bin/env -Sruby -EUTF-8
 
 # Trap ^C
 Signal.trap('INT') {

data/lib/heimdall_tools/cli.rb

@@ -27,10 +27,11 @@ module HeimdallTools
     long_desc Help.text(:sonarqube_mapper)
     option :name, required: true, aliases: '-n'
     option :api_url, required: true, aliases: '-u'
+    option :auth, type: :string, required: false
     option :output, required: true, aliases: '-o'
     option :verbose, type: :boolean, aliases: '-V'
     def sonarqube_mapper
-      hdf = HeimdallTools::SonarQubeMapper.new(options[:name], options[:api_url]).to_hdf
+      hdf = HeimdallTools::SonarQubeMapper.new(options[:name], options[:api_url], options[:auth]).to_hdf
       File.write(options[:output], hdf)
     end
 

data/lib/heimdall_tools/fortify_mapper.rb

@@ -1,6 +1,6 @@
 require 'json'
-require 'nokogiri'
-require 'nori'
+require 'heimdall_tools/hdf'
+require 'utilities/xml_to_hash'
 
 NIST_REFERENCE_NAME = 'Standards Mapping - NIST Special Publication 800-53 Revision 4'.freeze
 
@@ -11,21 +11,26 @@ module HeimdallTools
       @verbose = verbose
 
       begin
-        data = Nori.new(empty_tag_value: true).parse(fvdl)
+        data = xml_to_hash(fvdl)
         @timestamp = data['FVDL']['CreatedTS']
         @vulns = data['FVDL']['Vulnerabilities']['Vulnerability']
         @snippets = data['FVDL']['Snippets']['Snippet']
         @rules = data['FVDL']['Description']
+        @uuid = data['FVDL']['UUID']
+        @fortify_version = data['FVDL']['EngineData']['EngineVersion']
+
       rescue StandardError => e
         raise "Invalid Fortify FVDL file provided Exception: #{e}"
       end
     end
 
     def process_entry(entry)
-      snippetid = entry['Node']['SourceLocation']['@snippet']
+      snippetid = entry['Node']['SourceLocation']['snippet']
       finding = {}
       finding['status'] = 'failed'
       finding['code_desc'] = snippet(snippetid)
+      finding['run_time'] = NA_FLOAT
+      finding['start_time'] = [@timestamp['date'], @timestamp['time']].join(' ')
       finding
     end
 
@@ -48,11 +53,11 @@ module HeimdallTools
     end
 
     def snippet(snippetid)
-      snippet = @snippets.select { |x| x['@id'].eql?(snippetid) }.first
+      snippet = @snippets.select { |x| x['id'].eql?(snippetid) }.first
       "\nPath: #{snippet['File']}\n" \
       "StartLine: #{snippet['StartLine']}, " \
       "EndLine: #{snippet['EndLine']}\n" \
-      "Code:\n#{snippet['Text'].strip}" \
+      "Code:\n#{snippet['Text']['#cdata-section'].strip}" \
     end
 
     def nist_tag(rule)
@@ -68,26 +73,29 @@ module HeimdallTools
     end
 
     def to_hdf
-      inpsec_json = {}
-
-      inpsec_json['name'] = 'Fortify Static Analyzer Scan'
-      inpsec_json['version'] = [@timestamp['@date'], @timestamp['@time']].join(' ')
-      inpsec_json['controls'] = []
-
+      controls = []
       @rules.each do |rule|
         @item = {}
-        @item['id']           = rule['@classID']
-        @item['desc']         = rule['Explanation']
-        @item['title']        = rule['Abstract']
-        @item['impact']       = impact(rule['@classID'])
-        @item['code']         = ''
-        @item['results']      = []
-        @item['results']      = primaries(@item['id'])
-        @item['tags']         = {}
-        @item['tags']['nist'] = [nist_tag(rule).to_s, 'Rev_4']
-        inpsec_json['controls'] << @item
+        @item['id']              = rule['classID']
+        @item['desc']            = rule['Explanation']
+        @item['title']           = rule['Abstract']
+        @item['impact']          = impact(rule['classID'])
+        @item['descriptions']    = NA_ARRAY
+        @item['refs']            = NA_ARRAY
+        @item['source_location'] = NA_HASH
+        @item['code']            = NA_TAG
+        @item['results']         = []
+        @item['results']         = primaries(@item['id'])
+        @item['tags']            = {}
+        @item['tags']['nist']    = [nist_tag(rule).to_s, 'Rev_4']
+        controls << @item
       end
-      inpsec_json.to_json
+      results = HeimdallDataFormat.new(profile_name: 'Fortify Static Analyzer Scan',
+                                       version: @fortify_version,
+                                       title: 'Fortify Static Analyzer Scan',
+                                       summary: "Fortify Static Analyzer Scan of UUID: #{@uuid}",
+                                       controls: controls)
+      results.to_hdf
     end
   end
 end

data/lib/heimdall_tools/hdf.rb

@@ -0,0 +1,66 @@
+require 'json'
+require 'heimdall_tools/version'
+require 'openssl'
+
+NA_TAG = nil.freeze
+NA_ARRAY = [].freeze
+NA_HASH = {}.freeze
+NA_FLOAT = 0.0.freeze
+
+PLATFORM_NAME = 'Heimdall Tools'.freeze
+
+
+module HeimdallTools
+  class HeimdallDataFormat
+    def initialize(profile_name: NA_TAG,
+                   version: NA_TAG,
+                   duration: NA_TAG,
+                   sha256: NA_TAG,
+                   title: NA_TAG,
+                   maintainer: NA_TAG,
+                   summary: NA_TAG,
+                   license: NA_TAG,
+                   copyright: NA_TAG,
+                   copyright_email: NA_TAG,
+                   supports: NA_ARRAY,
+                   attributes: NA_ARRAY,
+                   depends: NA_ARRAY,
+                   groups: NA_ARRAY,
+                   status: 'loaded',
+                   controls: NA_TAG)
+
+      @results_json = {}
+      @results_json['platform'] = {}
+      @results_json['platform']['name'] = 'Heimdall Tools'
+      @results_json['platform']['release'] = HeimdallTools::VERSION
+      @results_json['version'] = HeimdallTools::VERSION
+
+      @results_json['statistics'] = {}
+      @results_json['statistics']['duration'] = duration || NA_TAG
+
+      @results_json['profiles'] = []
+
+      profile_block = {}
+      profile_block['name']            = profile_name
+      profile_block['version']         = version
+      profile_block['title']           = title
+      profile_block['maintainer']      = maintainer
+      profile_block['summary']         = summary
+      profile_block['license']         = license
+      profile_block['copyright']       = copyright
+      profile_block['copyright_email'] = copyright_email
+      profile_block['supports']        = supports
+      profile_block['attributes']      = attributes
+      profile_block['depends']         = depends
+      profile_block['groups']          = groups
+      profile_block['status']          = status
+      profile_block['controls']        = controls
+      profile_block['sha256']          = OpenSSL::Digest::SHA256.digest(profile_block.to_s).unpack("H*")[0]
+      @results_json['profiles'] << profile_block
+    end
+
+    def to_hdf
+      @results_json.to_json
+    end
+  end
+end

data/lib/heimdall_tools/help/sonarqube_mapper.md

@@ -2,4 +2,6 @@
 
 Examples:
 
-  heimdall_tools sonarqube_mapper -n sonar_project -u http://sonar:9000/api -o scan_results.json
+  heimdall_tools sonarqube_mapper -n sonar_project_key -u http://sonar:9000/api -o scan_results.json
+  
+  heimdall_tools sonarqube_mapper -n sonar_project_key -u http://sonar:9000/api --auth admin:admin -o scan_results.json

data/lib/heimdall_tools/sonarqube_mapper.rb

@@ -1,6 +1,7 @@
 require 'httparty'
 require 'json'
 require 'csv'
+require 'heimdall_tools/hdf'
 
 MAPPING_FILES = {
   cwe: './lib/data/cwe-nist-mapping.csv'.freeze,
@@ -16,7 +17,7 @@ IMPACT_MAPPING = {
 }.freeze
 
 def check_response(response)
-  raise "API Error: #{response.status}\n#{response.body}" unless response.ok?
+  raise "API Error: #{response.response}\n#{response.body}" unless response.ok?
 end
 
 class SonarQubeApi
@@ -28,8 +29,20 @@ class SonarQubeApi
 
   PAGE_SIZE = 100
 
-  def initialize(api_url)
+  def initialize(api_url, auth=nil)
     @api_url = api_url
+    @auth = auth
+  end
+
+  def query_api(endpoint, params={})
+    creds = {
+              username: @auth.split(':')[0],
+              password: @auth.split(':')[1]
+    } unless @auth.nil?
+
+    response = HTTParty.get(@api_url + endpoint, { query: params, basic_auth: creds })
+    check_response response
+    response
   end
 
   # Query issues endpoint, get all vulnerabilities
@@ -45,8 +58,7 @@ class SonarQubeApi
     }
 
     loop do # Get all pages
-      response = HTTParty.get(@api_url + ISSUES_ENDPOINT, { query: params })
-      check_response response
+      response = query_api(ISSUES_ENDPOINT, params)
       issues += response['issues']
 
       if params[:p] * PAGE_SIZE >= response['paging']['total']
@@ -64,8 +76,7 @@ class SonarQubeApi
     params = {
       key: rule
     }
-    response = HTTParty.get(@api_url + RULE_ENDPOINT, { query: params })
-    check_response response
+    response = query_api(RULE_ENDPOINT, params)
     response['rule']
   end
 
@@ -80,15 +91,13 @@ class SonarQubeApi
     params = {
       key: component
     }
-    response = HTTParty.get(@api_url + SOURCE_ENDPOINT, { query: params })
-    check_response response
+    response = query_api(SOURCE_ENDPOINT, params)
     response.body.split("\n")[start_line..end_line].join("\n")
   end
 
   # Query the version of the SonarQube server
   def query_version
-    response = HTTParty.get(@api_url + VERSION_ENDPOINT)
-    check_response response
+    response = query_api(VERSION_ENDPOINT)
     response.body
   end
 end
@@ -96,9 +105,9 @@ end
 module HeimdallTools
   class SonarQubeMapper
     # Fetches the necessary data from the API and builds report
-    def initialize(project_name, sonarqube_url)
+    def initialize(project_name, sonarqube_url, auth=nil)
       @project_name = project_name
-      @api = SonarQubeApi.new(sonarqube_url)
+      @api = SonarQubeApi.new(sonarqube_url,auth)
 
       @mappings = load_nist_mappings
       @findings = @api.query_issues(@project_name).map { |x| Finding.new(x, @api) }
@@ -119,7 +128,7 @@ module HeimdallTools
                                             headers: true,
                                             header_converters: :symbol,
                                             converters: :all })
-        mappings[mapping_type] = Hash[csv_data.map { |row|
+        mappings[mapping_type] = Hash[csv_data.reject{ |row| row[:nistid].nil? }.map { |row|
           [row[(mapping_type.to_s.downcase + 'id').to_sym].to_s, [row[:nistid], "Rev_#{row[:rev]}"]]
         }]
       end
@@ -128,13 +137,12 @@ module HeimdallTools
 
     # Returns a report in HDF format
     def to_hdf
-      {
-        controls: @controls.map(&:hdf),
-          # currently on heimdall version tag is displayed as time on profile view
-          # this wil be updated after heimdal update to fix this
-          version: Time.now.strftime("%a,%d %b %Y %X"),
-          name: "#{@project_name} SonarQube Scan"
-      }.to_json
+      results = HeimdallDataFormat.new(profile_name: "SonarQube Scan",
+                                       version: @api.query_version,
+                                       title: "SonarQube Scan of Project: #{@project_name}",
+                                       summary: "SonarQube Scan of Project: #{@project_name}",
+                                       controls: @controls.map(&:hdf))
+      results.to_hdf
     end
   end
 end
@@ -148,17 +156,20 @@ class Control
   TAG_DATA[:cwe] = {
     # Some rules with cwe tag don't have cwe number in description!
     # Currently only squid:S2658, but it has OWASP tag so we can use that.
-    regex: 'cwe.mitre.org/data/definitions/([^\.]*)' # Sometimes the "http://" is not part of the url
+    regex: 'cwe.mitre.org/data/definitions/(\d*)' # Sometimes the "http://" is not part of the url
   }
   TAG_DATA[:owasp] = {
     # Many (19 currently) owasp have don't cwe (ex. squid:S3355)
   }
-  TAG_DATA[:cert] = {
-    # Some rules only have cert tag (ex. kotlin:S1313)
-    # Some rules with cert tag don't actually have cert in description!
-    # Currently only squid:S4434, but it has OWASP tag so we can use that.
-    regex: 'CERT,?\n? ([^<]*)\.?<'
-  }
+
+  # CERT data mapping is deactivated for now until CERT -> NIST 800-53 mapping is available.
+  # TAG_DATA[:cert] = {
+  #   # Some rules only have cert tag (ex. kotlin:S1313)
+  #   # Some rules with cert tag don't actually have cert in description!
+  #   # Currently only squid:S4434, but it has OWASP tag so we can use that.
+  #   regex: 'CERT,?\n? ([^<]*)\.?<'
+  # }
+
   # All sans-tagged rules have CWE number, so no need to map SANS
   # There some tags which we can map directly (ex. denial-of-service)
   # But there are currently no rules with such a tag that don't have a better tag (ex. cwe)
@@ -237,8 +248,11 @@ class Control
           nist: get_nist_tags
         },
         results: @findings.map(&:get_result),
-        code: '', # This should be the inspec code for the control, which we don't have
-        id: @key
+        code: NA_TAG, # This should be the inspec code for the control, which we don't have
+        id: @key,
+        descriptions: NA_ARRAY,
+        refs: NA_ARRAY,
+        source_location: NA_HASH,
     }
   end
 end
@@ -266,8 +280,10 @@ class Finding
 
     snip_html = "StartLine: #{snip_start}, EndLine: #{snip_end}<br>Code:<pre>#{snip}</pre>"
     {
-      status: 'failed',
-        code_desc: "Path:#{component}:#{vuln_start}:#{vuln_end} #{snip_html}"
+        status: 'failed',
+        code_desc: "Path:#{component}:#{vuln_start}:#{vuln_end} #{snip_html}",
+        run_time:  NA_FLOAT,
+        start_time: Time.now.strftime("%a,%d %b %Y %X")
     }
   end
 end

data/lib/heimdall_tools/version.rb

@@ -1,3 +1,4 @@
 module HeimdallTools
-  VERSION = '1.2.0'.freeze
+  VERSION = '1.3.20'.freeze
 end
+

data/lib/heimdall_tools/zap_mapper.rb

@@ -1,6 +1,8 @@
 require 'json'
 require 'nokogiri'
 require 'csv'
+require 'heimdall_tools/hdf'
+
 
 CWE_NIST_MAPPING_FILE = './lib/data/cwe-nist-mapping.csv'.freeze
 
@@ -46,6 +48,8 @@ module HeimdallTools
       finding = {}
       finding['status'] = 'failed'
       finding['code_desc'] = format_code_desc(instance)
+      finding['run_time'] = NA_FLOAT
+      finding['start_time'] = @timestamp
       finding
     end
 
@@ -98,20 +102,17 @@ module HeimdallTools
     end
 
     def to_hdf
-      inpsec_profile = {}
-
-      inpsec_profile['name'] = "#{@host} OWASP ZAP Scan"
-      inpsec_profile['version'] = @timestamp
-
-      inpsec_profile['controls'] = []
-
+      controls = []
       @alerts.each do |alert|
         @item = {}
         @item['id']                 = alert[:pluginid].to_s
         @item['title']              = alert[:name].to_s
         @item['desc']               = Nokogiri::HTML(alert[:desc]).text
-        @item['impact']             = impact(alert[:riskcode]).to_s
+        @item['impact']             = impact(alert[:riskcode])
         @item['tags']               = {}
+        @item['descriptions']       = NA_ARRAY
+        @item['refs']               = NA_ARRAY
+        @item['source_location']    = NA_HASH
         @item['tags']['nist']       = nist_tag(alert[:cweid])
         @item['tags']['cweid']      = alert[:cweid].to_s
         @item['tags']['wascid']     = alert[:wascid].to_s
@@ -122,10 +123,16 @@ module HeimdallTools
         @item['code']               = ''
         @item['results']            = process_instances(alert[:instances])
 
-        inpsec_profile['controls'] << @item
+        controls << @item
       end
-      fix_duplicates(inpsec_profile['controls'])
-      inpsec_profile.to_json
+      fix_duplicates(controls)
+
+      results = HeimdallDataFormat.new(profile_name: 'OWASP ZAP Scan',
+                                       version: @zap_verison,
+                                       title: "OWASP ZAP Scan of Host: #{@host}",
+                                       summary: "OWASP ZAP Scan of Host: #{@host}",
+                                       controls: controls)
+      results.to_hdf
     end
   end
 end

data/lib/utilities/xml_to_hash.rb

@@ -0,0 +1,55 @@
+require 'nokogiri'
+
+def xml_node_to_hash(node)
+  # If we are at the root of the document, start the hash
+  if node.element?
+    result_hash = {}
+    if node.attributes != {}
+      attributes = {}
+      node.attributes.keys.each do |key|
+        attributes[node.attributes[key].name] = node.attributes[key].value
+      end
+    end
+    if !node.children.empty?
+      node.children.each do |child|
+        result = xml_node_to_hash(child)
+
+        if child.name == 'text'
+          unless child.next_sibling || child.previous_sibling
+            return result unless attributes
+
+            result_hash[child.name] = result
+          end
+        elsif result_hash[child.name]
+
+          if result_hash[child.name].is_a?(Object::Array)
+            result_hash[child.name] << result
+          else
+            result_hash[child.name] = [result_hash[child.name]] << result
+          end
+        else
+          result_hash[child.name] = result
+        end
+      end
+      if attributes
+        # add code to remove non-data attributes e.g. xml schema, namespace here
+        # if there is a collision then node content supersets attributes
+        result_hash = attributes.merge(result_hash)
+      end
+      return result_hash
+    else
+      return attributes
+    end
+  else
+    node.content.to_s
+  end
+end
+
+def xml_to_hash(xml)
+  begin
+    data = Nokogiri::XML(xml) { |config| config.strict }
+  rescue Nokogiri::XML::SyntaxError => e
+    puts "XML Parsing caught exception: #{e}"
+  end
+  { data.root.name => xml_node_to_hash(data.root) }
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: heimdall_tools
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.20
 platform: ruby
 authors:
 - Robert Thew
@@ -10,92 +10,106 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2019-02-04 00:00:00.000000000 Z
+date: 2020-03-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: activesupport
+  name: nokogiri
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.2.3
+        version: 1.10.9
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.2.3
+        version: 1.10.9
 - !ruby/object:Gem::Dependency
-  name: colorize
+  name: thor
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.19'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.19'
 - !ruby/object:Gem::Dependency
-  name: nokogiri
+  name: json
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.8'
+        version: '2.3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.8'
+        version: '2.3'
 - !ruby/object:Gem::Dependency
-  name: nori
+  name: csv
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '3.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '3.1'
 - !ruby/object:Gem::Dependency
-  name: OptionParser
+  name: httparty
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.18.0
 - !ruby/object:Gem::Dependency
-  name: thor
+  name: openssl
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.19'
+        version: '2.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.19'
+        version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: nori
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.6'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -128,30 +142,44 @@ dependencies:
   name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: codeclimate-test-reporter
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
-  type: :runtime
+        version: '0'
+  type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '0'
 description: Converter utils that can be included as a gem or used from the command
   line
 email:
@@ -174,6 +202,7 @@ files:
 - lib/heimdall_tools/cli.rb
 - lib/heimdall_tools/command.rb
 - lib/heimdall_tools/fortify_mapper.rb
+- lib/heimdall_tools/hdf.rb
 - lib/heimdall_tools/help.rb
 - lib/heimdall_tools/help/fortify_mapper.md
 - lib/heimdall_tools/help/sonarqube_mapper.md
@@ -181,7 +210,7 @@ files:
 - lib/heimdall_tools/sonarqube_mapper.rb
 - lib/heimdall_tools/version.rb
 - lib/heimdall_tools/zap_mapper.rb
-- lib/utilities/gitkeep
+- lib/utilities/xml_to_hash.rb
 homepage: https://github.com/mitre/heimdall_tools
 licenses:
 - Apache-2.0
@@ -202,8 +231,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.14
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: Convert Forify, Openzap and Sonarqube results to HDF