heimdall_tools 1.3.42 → 1.3.47

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 938eedc5f080d2ffca3623fd3f25c8996244009cc65aa4b78981ee8310063052
-  data.tar.gz: '0588206839c8f454134354721e3d850859e59c0fab8d964c4eb40a9987930443'
+  metadata.gz: 0c1758366de13bb1966a1ffeccbab60c7b1c2d91c221d5ee604dd2d4f1c96dcb
+  data.tar.gz: 2ba1491cb04b569b216c966724b9b8e4f06430a3dc7913ce8bf5e0611ed6896a
 SHA512:
-  metadata.gz: f6b1a7071a33b8d54d2336ce715c67bff26fb1a02306612cb63d752553cd9758ce05274b37c62c4887a0746772c100772fba9d38abded9099ed14862726d5f90
-  data.tar.gz: 1c752004ea7dde9a7695bb68d4e4a71839821afdf4032e6561bf0d119b2ee108dc131ace3b78382cab517afdfa19699f69a34b155faa546f4effa0c9fb658933
+  metadata.gz: 38d66cf5d8653a1a859b7254b9b2ba05518ce5c750b73fb6e638b24fca08ea574aaf1e89bbf560f98fa5cdd7d56d49ca5dedf23b20845d56dd9ecb8c41210ce1
+  data.tar.gz: b9db7f5c78640e37795493f36e2b119e4c15d5184d3b830e57bf92ad41709a0fa8345946e249afef7f1adb0ba02a89183c756ce8e81eeafb5cd766232518b197

data/README.md

@@ -9,13 +9,30 @@ HeimdallTools supplies several methods to convert output from various tools to "
 - **fortify_mapper** - commercial static code analysis tool
 - **zap_mapper** - OWASP ZAP - open-source dynamic code analysis tool
 - **burpsuite_mapper** - commercial dynamic analysis tool
-- **nessus_mapper** - commercial vulnerability scanner
+- **nessus_mapper** - commercial security scanner (supports compliance and vulnerability scans from Tenable.sc and Tenable.io)
 - **snyk_mapper** - commercial package vulnerability scanner
 - **nikto_mapper** - open-source web server scanner 
 - **jfrog_xray_mapper** - package vulnerability scanner
 - **dbprotect_mapper** - database vulnerability scanner
 - **aws_config_mapper** - assess, audit, and evaluate AWS resources
 - **netsparker_mapper** - web application security scanner
+- **sarif_mapper** - static analysis results interchange format
+- **scoutsuite_mapper** - multi-cloud security auditing tool
+
+## Want to recommend a mapper for another tool? Please use these steps:
+  1. Create an [issue](https://github.com/mitre/heimdall_tools/issues/new), and email saf@groups.mitre.org citing the issue link so we can help
+  2. Provide a sample output, preferably the most detailed the tool can provide, and also preferably in a machine-readable format, such as xml, json, or csv - whichever is natively available. If it is sensitive we'll work that in #3. (If it's an API only, we'll also just talk about it in #3)
+  3. Let's arrange a time to take a close look at the data it provides to get an idea of all it has to offer. We'll suggest an initial mapping of the HDF core elements. (see https://saf.mitre.org/#/normalize)
+  4. Note: if the tool doesn't provide a NIST SP 800-53 reference, we've worked on mappings to other references such as CWE or OWASP Top 10:  
+    https://github.com/mitre/heimdall_tools/tree/master/lib/data  
+    https://github.com/mitre/heimdall_tools/blob/master/lib/data/cwe-nist-mapping.csv  
+    https://github.com/mitre/heimdall_tools/blob/master/lib/data/owasp-nist-mapping.csv  
+  5. If the tool doesn't provide something for #4, or another core element such as impact, we'll help you identify a custom mapping approach.
+  6. We'll help you decide how to preserve any other information (non-core elements) the tool provides to ensure that all of the original tool's intent comes through for the user when the data is viewed in Heimdall.
+  7. Finally, We'll provide final peer review and support merging your pull request.
+We appreciate your contributions, but we're here to help!
+
+## How to Install Heimdall Tools:
 
 Ruby 2.4 or higher (check using "ruby -v")
 
@@ -136,7 +153,8 @@ example: heimdall_tools burpsuite_mapper -x burpsuite_results.xml -o scan_result
 
 ## nessus_mapper
 
-nessus_mapper translates a Nessus-exported XML results file into HDF format json to be viewable in Heimdall
+nessus_mapper translates a Nessus-exported XML results file into HDF format json to be viewable in Heimdall.
+Supports compliance and vulnerability scans from Tenable.sc and Tenable.io.
 
 Note: A separate HDF JSON file is generated for each host reported in the Nessus Report.
 
@@ -185,6 +203,22 @@ FLAGS:
 example: heimdall_tools nikto_mapper -j nikto_results.json -o nikto_results.json
 ```
 
+## scoutsuite_mapper
+
+scoutsuite_mapper translates Scout Suite results from Javascript to HDF-formatted JSON so as to be viewable on Heimdall
+
+Note: Currently this mapper only supports AWS.
+
+```
+USAGE: heimdall_tools scoutsuite_mapper -i <scoutsuite-results-js> -o <hdf-scan-results-json>
+
+FLAGS:
+    -i --input -j --javascript <scoutsuite-results-js> : path to Scout Suite results Javascript file.
+    -o --output <hdf-scan-results-json>                : path to output scan-results json.
+
+example: heimdall_tools scoutsuite_mapper -i scoutsuite_results.js -o scoutsuite_hdf.json
+```
+
 ## jfrog_xray_mapper
 
 jfrog_xray_mapper translates an JFrog Xray results JSON file into HDF format JSON to be viewable in Heimdall
@@ -252,6 +286,21 @@ FLAGS:
 example: heimdall_tools netsparker_mapper -x netsparker_results.xml -o netsparker_hdf.json
 ```
 
+## sarif_mapper
+
+sarif_mapper translates a SARIF JSON file into HDF format JSON to be viewable in Heimdall
+
+```
+USAGE: heimdall_tools sarif_mapper [OPTIONS] -j <sarif-results-json> -o <hdf-scan-results.json>
+
+FLAGS:
+    -j <sarif_results_json>          : path to SARIF results JSON file.
+    -o --output_prefix <prefix>      : path to output scan-results json.
+    -V --verbose                     : verbose run [optional].
+
+example: heimdall_tools sarif_mapper -j sarif_results.json -o sarif_results_hdf.json
+```
+
 ## version  
 
 Prints out the gem version

data/lib/data/scoutsuite-nist-mapping.csv

@@ -0,0 +1,140 @@
+rule,nistid
+acm-certificate-with-close-expiration-date,SC-12
+acm-certificate-with-transparency-logging-disabled,SC-12
+cloudformation-stack-with-role,AC-6
+cloudtrail-duplicated-global-services-logging,AU-6
+cloudtrail-no-cloudwatch-integration,AU-12|SI-4(2)
+cloudtrail-no-data-logging,AU-12
+cloudtrail-no-encryption-with-kms,AU-6
+cloudtrail-no-global-services-logging,AU-12
+cloudtrail-no-log-file-validation,AU-6
+cloudtrail-no-logging,AU-12
+cloudtrail-not-configured,AU-12
+cloudwatch-alarm-without-actions,AU-12
+config-recorder-not-configured,CM-8|CM-8(2)|CM-8(6)
+ec2-ami-public,AC-3
+ec2-default-security-group-in-use,AC-3(3)
+ec2-default-security-group-with-rules,AC-3(3)
+ec2-ebs-snapshot-not-encrypted,SC-28
+ec2-ebs-snapshot-public,AC-3
+ec2-ebs-volume-not-encrypted,SC-28
+ec2-instance-in-security-group,CM-7(1)
+ec2-instance-type,CM-2
+ec2-instance-types,CM-2
+ec2-instance-with-public-ip,AC-3
+ec2-instance-with-user-data-secrets,AC-3
+ec2-security-group-opens-all-ports,CM-7(1)
+ec2-security-group-opens-all-ports-to-all,CM-7(1)
+ec2-security-group-opens-all-ports-to-self,CM-7(1)
+ec2-security-group-opens-icmp-to-all,CM-7(1)
+ec2-security-group-opens-known-port-to-all,CM-7(1)
+ec2-security-group-opens-plaintext-port,CM-7(1)
+ec2-security-group-opens-port-range,CM-7(1)
+ec2-security-group-opens-port-to-all,CM-7(1)
+ec2-security-group-whitelists-aws,CM-7(1)
+ec2-security-group-whitelists-aws-ip-from-banned-region,CM-7(1)
+ec2-security-group-whitelists-non-elastic-ips,CM-7(1)
+ec2-security-group-whitelists-unknown-aws,CM-7(1)
+ec2-security-group-whitelists-unknown-cidrs,CM-7(1)
+ec2-unused-security-group,CM-7(1)
+elb-listener-allowing-cleartext,SC-8
+elb-no-access-logs,AU-12
+elb-older-ssl-policy,SC-8
+elbv2-http-request-smuggling,SC-8
+elbv2-listener-allowing-cleartext,SC-8
+elbv2-no-access-logs,AU-12
+elbv2-no-deletion-protection,SI-7
+elbv2-older-ssl-policy,SC-8
+iam-assume-role-lacks-external-id-and-mfa,AC-17
+iam-assume-role-no-mfa,AC-6
+iam-assume-role-policy-allows-all,AC-6
+iam-ec2-role-without-instances,AC-6
+iam-group-with-inline-policies,AC-6
+iam-group-with-no-users,AC-6
+iam-human-user-with-policies,AC-6
+iam-inline-policy-allows-non-sts-action,AC-6
+iam-inline-policy-allows-NotActions,AC-6
+iam-inline-policy-for-role,AC-6
+iam-managed-policy-allows-full-privileges,AC-6
+iam-managed-policy-allows-non-sts-action,AC-6
+iam-managed-policy-allows-NotActions,AC-6
+iam-managed-policy-for-role,AC-6
+iam-managed-policy-no-attachments,AC-6
+iam-no-support-role,IR-7
+iam-password-policy-expiration-threshold,AC-2
+iam-password-policy-minimum-length,AC-2
+iam-password-policy-no-expiration,AC-2
+iam-password-policy-no-lowercase-required,AC-2
+iam-password-policy-no-number-required,AC-2
+iam-password-policy-no-symbol-required,AC-2
+iam-password-policy-no-uppercase-required,AC-2
+iam-password-policy-reuse-enabled,IA-5(1)
+iam-role-with-inline-policies,AC-6
+iam-root-account-no-hardware-mfa,IA-2(1)
+iam-root-account-no-mfa,IA-2(1)
+iam-root-account-used-recently,AC-6(9)
+iam-root-account-with-active-certs,AC-6(9)
+iam-root-account-with-active-keys,AC-6(9)
+iam-service-user-with-password,AC-2
+iam-unused-credentials-not-disabled,AC-2
+iam-user-no-key-rotation,AC-2
+iam-user-not-in-category-group,AC-2
+iam-user-not-in-common-group,AC-2
+iam-user-unused-access-key-initial-setup,AC-2
+iam-user-with-multiple-access-keys,IA-2
+iam-user-without-mfa,IA-2(1)
+iam-user-with-password-and-key,IA-2
+iam-user-with-policies,AC-2
+kms-cmk-rotation-disabled,SC-12
+logs-no-alarm-aws-configuration-changes,CM-8|CM-8(2)|CM-8(6)
+logs-no-alarm-cloudtrail-configuration-changes,AU-6
+logs-no-alarm-cmk-deletion,AC-2
+logs-no-alarm-console-authentication-failures,AC-2
+logs-no-alarm-iam-policy-changes,AC-2
+logs-no-alarm-nacl-changes,CM-6(2)
+logs-no-alarm-network-gateways-changes,AU-12|CM-6(2)
+logs-no-alarm-root-usage,AU-2
+logs-no-alarm-route-table-changes,AU-12|CM-6(2)
+logs-no-alarm-s3-policy-changes,AC-6|AU-12
+logs-no-alarm-security-group-changes,AC-2(4)
+logs-no-alarm-signin-without-mfa,AC-2
+logs-no-alarm-unauthorized-api-calls,AU-6|SI-4(2)
+logs-no-alarm-vpc-changes,CM-6(1)
+rds-instance-backup-disabled,CP-9
+rds-instance-ca-certificate-deprecated,SC-12
+rds-instance-no-minor-upgrade,SI-2
+rds-instance-short-backup-retention-period,CP-9
+rds-instance-single-az,CP-7
+rds-instance-storage-not-encrypted,SC-28
+rds-postgres-instance-with-invalid-certificate,SC-12
+rds-security-group-allows-all,CM-7(1)
+rds-snapshot-public,SC-28
+redshift-cluster-database-not-encrypted,SC-28
+redshift-cluster-no-version-upgrade,SI-2
+redshift-cluster-publicly-accessible,AC-3
+redshift-parameter-group-logging-disabled,AU-12
+redshift-parameter-group-ssl-not-required,SC-8
+redshift-security-group-whitelists-all,CM-7(1)
+route53-domain-no-autorenew,SC-2
+route53-domain-no-transferlock,SC-2
+route53-domain-transferlock-not-authorized,SC-2
+s3-bucket-allowing-cleartext,SC-28
+s3-bucket-no-default-encryption,SC-28
+s3-bucket-no-logging,AU-2|AU-12
+s3-bucket-no-mfa-delete,SI-7
+s3-bucket-no-versioning,SI-7
+s3-bucket-world-acl,AC-3(3)
+s3-bucket-world-policy-arg,AC-3(3)
+s3-bucket-world-policy-star,AC-3(3)
+ses-identity-dkim-not-enabled,SC-23
+ses-identity-dkim-not-verified,SC-23
+ses-identity-world-policy,AC-6
+sns-topic-world-policy,AC-6
+sqs-queue-world-policy,AC-6
+vpc-custom-network-acls-allow-all,SC-7
+vpc-default-network-acls-allow-all,SC-7
+vpc-network-acl-not-used,SC-7
+vpc-routing-tables-with-peering,AC-3(3)
+vpc-subnet-with-bad-acls,SC-7
+vpc-subnet-with-default-acls,SC-7
+vpc-subnet-without-flow-log,AU-12

data/lib/heimdall_tools.rb

@@ -16,4 +16,6 @@ module HeimdallTools
   autoload :DBProtectMapper, 'heimdall_tools/dbprotect_mapper'
   autoload :AwsConfigMapper, 'heimdall_tools/aws_config_mapper'
   autoload :NetsparkerMapper, 'heimdall_tools/netsparker_mapper'
+  autoload :SarifMapper, 'heimdall_tools/sarif_mapper'
+  autoload :ScoutSuiteMapper, 'heimdall_tools/scoutsuite_mapper'
 end

data/lib/heimdall_tools/aws_config_mapper.rb

@@ -18,8 +18,7 @@ INSUFFICIENT_DATA_MSG = 'Not enough data has been collectd to determine complian
 #
 module HeimdallTools
   class AwsConfigMapper
-    def initialize(custom_mapping, endpoint = nil, verbose = false)
-      @verbose = verbose
+    def initialize(custom_mapping, endpoint = nil)
       @default_mapping = get_rule_mapping(AWS_CONFIG_MAPPING_FILE)
       @custom_mapping = custom_mapping.nil? ? {} : get_rule_mapping(custom_mapping)
       if endpoint.nil?
@@ -38,8 +37,8 @@ module HeimdallTools
     def to_hdf
       controls = @issues.map do |issue|
         @item = {}
-        @item['id']              = issue[:config_rule_name]
-        @item['title']           = issue[:config_rule_name]
+        @item['id']              = issue[:config_rule_id]
+        @item['title']           = "#{get_account_id(issue[:config_rule_arn])} - #{issue[:config_rule_name]}"
         @item['desc']            = issue[:description]
         @item['impact']          = 0.5
         @item['tags']            = hdf_tags(issue)
@@ -55,18 +54,33 @@ module HeimdallTools
           @item
         end
       end
+
       results = HeimdallDataFormat.new(
         profile_name: 'AWS Config',
-         title: 'AWS Config',
-         summary: 'AWS Config',
-         controls: controls,
-         statistics: { aws_config_sdk_version: Aws::ConfigService::GEM_VERSION },
+        title: 'AWS Config',
+        summary: 'AWS Config',
+        controls: controls,
+        statistics: { aws_config_sdk_version: Aws::ConfigService::GEM_VERSION },
       )
       results.to_hdf
     end
 
     private
 
+    ##
+    # Gets the account ID from a config rule ARN
+    #
+    # https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
+    # https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html
+    #
+    # Params:
+    # - arn: The ARN of the config rule
+    #
+    # Returns: The account ID portion of the ARN
+    def get_account_id(arn)
+      /:(\d{12}):config-rule/.match(arn)&.captures&.first || 'no-account-id'
+    end
+
     ##
     # Read in a config rule -> 800-53 control mapping CSV.
     #
@@ -263,7 +277,8 @@ module HeimdallTools
       # If no input parameters, then provide an empty JSON array to the JSON
       # parser because passing nil to JSON.parse throws an exception.
       params = (JSON.parse(config_rule[:input_parameters] || '[]').map { |key, value| "#{key}: #{value}" }).join('<br/>')
-      check_text = config_rule[:config_rule_arn] || ''
+      check_text = "ARN: #{config_rule[:config_rule_arn] || 'N/A'}"
+      check_text += "<br/>Source Identifier: #{config_rule.dig(:source, :source_identifier) || 'N/A'}"
       check_text += "<br/>#{params}" unless params.empty?
       check_text
     end

data/lib/heimdall_tools/burpsuite_mapper.rb

@@ -20,9 +20,8 @@ DEFAULT_NIST_TAG = %w{SA-11 RA-5 Rev_4}.freeze
 
 module HeimdallTools
   class BurpSuiteMapper
-    def initialize(burps_xml, _name = nil, verbose = false)
+    def initialize(burps_xml, _name = nil)
       @burps_xml = burps_xml
-      @verbose = verbose
 
       begin
         @cwe_nist_mapping = parse_mapper

data/lib/heimdall_tools/cli.rb

@@ -6,7 +6,6 @@ module HeimdallTools
     long_desc Help.text(:fortify_mapper)
     option :fvdl, required: true, aliases: '-f'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def fortify_mapper
       hdf = HeimdallTools::FortifyMapper.new(File.read(options[:fvdl])).to_hdf
       File.write(options[:output], hdf)
@@ -17,7 +16,6 @@ module HeimdallTools
     option :json, required: true, aliases: '-j'
     option :name, required: true, aliases: '-n'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def zap_mapper
       hdf = HeimdallTools::ZapMapper.new(File.read(options[:json]), options[:name]).to_hdf
       File.write(options[:output], hdf)
@@ -29,7 +27,6 @@ module HeimdallTools
     option :api_url, required: true, aliases: '-u'
     option :auth, type: :string, required: false
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def sonarqube_mapper
       hdf = HeimdallTools::SonarQubeMapper.new(options[:name], options[:api_url], options[:auth]).to_hdf
       File.write(options[:output], hdf)
@@ -39,7 +36,6 @@ module HeimdallTools
     long_desc Help.text(:burpsuite_mapper)
     option :xml, required: true, aliases: '-x'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def burpsuite_mapper
       hdf = HeimdallTools::BurpSuiteMapper.new(File.read(options[:xml])).to_hdf
       File.write(options[:output], hdf)
@@ -49,7 +45,6 @@ module HeimdallTools
     long_desc Help.text(:nessus_mapper)
     option :xml, required: true, aliases: '-x'
     option :output_prefix, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def nessus_mapper
       hdfs = HeimdallTools::NessusMapper.new(File.read(options[:xml])).to_hdf
 
@@ -64,7 +59,6 @@ module HeimdallTools
     long_desc Help.text(:snyk_mapper)
     option :json, required: true, aliases: '-j'
     option :output_prefix, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def snyk_mapper
       hdfs = HeimdallTools::SnykMapper.new(File.read(options[:json]), options[:name]).to_hdf
       puts "\r\HDF Generated:\n"
@@ -78,7 +72,6 @@ module HeimdallTools
     long_desc Help.text(:nikto_mapper)
     option :json, required: true, aliases: '-j'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def nikto_mapper
       hdf = HeimdallTools::NiktoMapper.new(File.read(options[:json])).to_hdf
       File.write(options[:output], hdf)
@@ -90,7 +83,6 @@ module HeimdallTools
     long_desc Help.text(:jfrog_xray_mapper)
     option :json, required: true, aliases: '-j'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def jfrog_xray_mapper
       hdf = HeimdallTools::JfrogXrayMapper.new(File.read(options[:json])).to_hdf
       File.write(options[:output], hdf)
@@ -102,7 +94,6 @@ module HeimdallTools
     long_desc Help.text(:dbprotect_mapper)
     option :xml, required: true, aliases: '-x'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def dbprotect_mapper
       hdf = HeimdallTools::DBProtectMapper.new(File.read(options[:xml])).to_hdf
       File.write(options[:output], hdf)
@@ -114,7 +105,6 @@ module HeimdallTools
     long_desc Help.text(:aws_config_mapper)
     # option :custom_mapping, required: false, aliases: '-m'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def aws_config_mapper
       hdf = HeimdallTools::AwsConfigMapper.new(options[:custom_mapping]).to_hdf
       File.write(options[:output], hdf)
@@ -126,7 +116,6 @@ module HeimdallTools
     long_desc Help.text(:netsparker_mapper)
     option :xml, required: true, aliases: '-x'
     option :output, required: true, aliases: '-o'
-    option :verbose, type: :boolean, aliases: '-V'
     def netsparker_mapper
       hdf = HeimdallTools::NetsparkerMapper.new(File.read(options[:xml])).to_hdf
       File.write(options[:output], hdf)
@@ -134,6 +123,29 @@ module HeimdallTools
       puts options[:output].to_s
     end
 
+    desc 'sarif_mapper', 'sarif_mapper translates a SARIF JSON file into HDF format JSON to be viewable in Heimdall'
+    long_desc Help.text(:sarif_mapper)
+    option :json, required: true, aliases: '-j'
+    option :output, required: true, aliases: '-o'
+    option :verbose, type: :boolean, aliases: '-V'
+    def sarif_mapper
+      hdf = HeimdallTools::SarifMapper.new(File.read(options[:json])).to_hdf
+      File.write(options[:output], hdf)
+      puts "\r\HDF Generated:\n"
+      puts options[:output].to_s
+    end
+
+    desc 'scoutsuite_mapper', 'scoutsuite_mapper translates Scout Suite results from Javascript to HDF-formatted JSON so as to be viewable on Heimdall'
+    long_desc Help.text(:scoutsuite_mapper)
+    option :javascript, required: true, banner: 'SCOUTSUITE-RESULTS-JS', aliases: ['-i', '--input', '-j']
+    option :output, required: true, banner: 'HDF-SCAN-RESULTS-JSON', aliases: '-o'
+    def scoutsuite_mapper
+      hdf = HeimdallTools::ScoutSuiteMapper.new(File.read(options[:javascript])).to_hdf
+      File.write(options[:output], hdf)
+      puts "\rHDF Generated:\n"
+      puts options[:output].to_s
+    end
+
     desc 'version', 'prints version'
     def version
       puts VERSION

data/lib/heimdall_tools/dbprotect_mapper.rb

@@ -12,15 +12,11 @@ IMPACT_MAPPING = {
 
 module HeimdallTools
   class DBProtectMapper
-    def initialize(xml, _name = nil, verbose = false)
-      @verbose = verbose
-
-      begin
-        dataset = xml_to_hash(xml)
-        @entries = compile_findings(dataset['dataset'])
-      rescue StandardError => e
-        raise "Invalid DBProtect XML file provided Exception: #{e};\nNote that XML must be of kind `Check Results Details`."
-      end
+    def initialize(xml, _name = nil)
+      dataset = xml_to_hash(xml)
+      @entries = compile_findings(dataset['dataset'])
+    rescue StandardError => e
+      raise "Invalid DBProtect XML file provided Exception: #{e};\nNote that XML must be of kind `Check Results Details`."
     end
 
     def to_hdf

data/lib/heimdall_tools/fortify_mapper.rb

@@ -7,9 +7,8 @@ DEFAULT_NIST_TAG = %w{SA-11 RA-5}.freeze
 
 module HeimdallTools
   class FortifyMapper
-    def initialize(fvdl, verbose = false)
+    def initialize(fvdl)
       @fvdl = fvdl
-      @verbose = verbose
 
       begin
         data = xml_to_hash(fvdl)

data/lib/heimdall_tools/help/sarif_mapper.md

@@ -0,0 +1,12 @@
+  sarif_mapper translates a SARIF JSON file into HDF format JSON to be viewable in Heimdall
+
+SARIF level to HDF impact Mapping:
+  SARIF level error -> HDF impact 0.7
+  SARIF level warning -> HDF impact 0.5
+  SARIF level note -> HDF impact 0.3
+  SARIF level none -> HDF impact 0.1
+  SARIF level not provided -> HDF impact 0.1 as default
+
+Examples:
+
+  heimdall_tools sarif_mapper [OPTIONS] -j <sarif-results-json> -o <hdf-scan-results.json>

data/lib/heimdall_tools/help/scoutsuite_mapper.md

@@ -0,0 +1,7 @@
+  scoutsuite_mapper translates Scout Suite results from Javascript to HDF-formatted JSON so as to be viewable on Heimdall
+
+  Note: Currently this mapper only supports AWS.
+
+Examples:
+
+  heimdall_tools scoutsuite_mapper -i <scoutsuite-results-js> -o <hdf-scan-results-json>

data/lib/heimdall_tools/jfrog_xray_mapper.rb

@@ -27,9 +27,8 @@ end
 
 module HeimdallTools
   class JfrogXrayMapper
-    def initialize(xray_json, _name = nil, verbose = false)
+    def initialize(xray_json, _name = nil)
       @xray_json = xray_json
-      @verbose = verbose
 
       begin
         @cwe_nist_mapping = parse_mapper

data/lib/heimdall_tools/nessus_mapper.rb

@@ -39,9 +39,8 @@ end
 
 module HeimdallTools
   class NessusMapper
-    def initialize(nessus_xml, verbose = false)
+    def initialize(nessus_xml)
       @nessus_xml = nessus_xml
-      @verbose = verbose
       read_cci_xml
       begin
         @cwe_nist_mapping = parse_mapper
@@ -72,7 +71,8 @@ module HeimdallTools
       info = {}
 
       info['policyName'] = policy['policyName']
-      info['version'] = policy['Preferences']['ServerPreferences']['preference'].select { |x| x['name'].eql? 'sc_version' }.first['value']
+      scanner_version = policy['Preferences']['ServerPreferences']['preference'].select { |x| x['name'].eql? 'sc_version' }
+      info['version'] = scanner_version.empty? ? NA_STRING : scanner_version.first['value']
       info
     rescue StandardError => e
       raise "Invalid Nessus XML file provided Exception: #{e}"
@@ -221,8 +221,12 @@ module HeimdallTools
           end
           if item['compliance-reference']
             @item['tags']['nist']     = cci_nist_tag(parse_refs(item['compliance-reference'], 'CCI'))
+            @item['tags']['cci']      = parse_refs(item['compliance-reference'], 'CCI')
+            @item['tags']['rid']      = parse_refs(item['compliance-reference'], 'Rule-ID').join(',')
+            @item['tags']['stig_id']  = parse_refs(item['compliance-reference'], 'STIG-ID').join(',')
           else
             @item['tags']['nist']     = plugin_nist_tag(item['pluginFamily'], item['pluginID'])
+            @item['tags']['rid']      = item['pluginID'].to_s
           end
           if item['compliance-solution']
             @item['descriptions']       << desc_tags(item['compliance-solution'], 'check')

data/lib/heimdall_tools/netsparker_mapper.rb

@@ -21,19 +21,15 @@ DEFAULT_NIST_TAG = %w{SA-11 RA-5}.freeze
 
 module HeimdallTools
   class NetsparkerMapper
-    def initialize(xml, _name = nil, verbose = false)
-      @verbose = verbose
-
-      begin
-        @cwe_nist_mapping = parse_mapper(CWE_NIST_MAPPING_FILE)
-        @owasp_nist_mapping = parse_mapper(OWASP_NIST_MAPPING_FILE)
-        data = xml_to_hash(xml)
-
-        @vulnerabilities = data['netsparker-enterprise']['vulnerabilities']['vulnerability']
-        @scan_info = data['netsparker-enterprise']['target']
-      rescue StandardError => e
-        raise "Invalid Netsparker XML file provided Exception: #{e}"
-      end
+    def initialize(xml, _name = nil)
+      @cwe_nist_mapping = parse_mapper(CWE_NIST_MAPPING_FILE)
+      @owasp_nist_mapping = parse_mapper(OWASP_NIST_MAPPING_FILE)
+      data = xml_to_hash(xml)
+
+      @vulnerabilities = data['netsparker-enterprise']['vulnerabilities']['vulnerability']
+      @scan_info = data['netsparker-enterprise']['target']
+    rescue StandardError => e
+      raise "Invalid Netsparker XML file provided Exception: #{e}"
     end
 
     def to_hdf

data/lib/heimdall_tools/nikto_mapper.rb

@@ -26,9 +26,8 @@ end
 
 module HeimdallTools
   class NiktoMapper
-    def initialize(nikto_json, _name = nil, verbose = false)
+    def initialize(nikto_json, _name = nil)
       @nikto_json = nikto_json
-      @verbose = verbose
 
       begin
         @nikto_nist_mapping = parse_mapper

data/lib/heimdall_tools/sarif_mapper.rb

@@ -0,0 +1,198 @@
+require 'json'
+require 'csv'
+require 'heimdall_tools/hdf'
+
+RESOURCE_DIR = Pathname.new(__FILE__).join('../../data')
+
+CWE_NIST_MAPPING_FILE = File.join(RESOURCE_DIR, 'cwe-nist-mapping.csv')
+
+IMPACT_MAPPING = {
+  error: 0.7,
+  warning: 0.5,
+  note: 0.3,
+  none: 0.0
+}.freeze
+
+DEFAULT_NIST_TAG = %w{SA-11 RA-5}.freeze
+
+# Loading spinner sign
+$spinner = Enumerator.new do |e|
+  loop do
+    e.yield '|'
+    e.yield '/'
+    e.yield '-'
+    e.yield '\\'
+  end
+end
+
+module HeimdallTools
+  class SarifMapper
+    def initialize(sarif_json, _name = nil, verbose = false)
+      @sarif_json = sarif_json
+      @verbose = verbose
+      begin
+        @cwe_nist_mapping = parse_mapper
+        @sarif_log = JSON.parse(@sarif_json)
+      rescue StandardError => e
+        raise "Invalid SARIF JSON file provided\n\nException: #{e}"
+      end
+    end
+
+    def extract_scaninfo(sarif_log)
+      info = {}
+      begin
+        info['policy'] = 'SARIF'
+        info['version'] = sarif_log['version']
+        info['projectName'] = 'Static Analysis Results Interchange Format'
+        info['summary'] = NA_STRING
+        info
+      rescue StandardError => e
+        raise "Error extracting project info from SARIF JSON file provided Exception: #{e}"
+      end
+    end
+
+    def finding(result)
+      finding = {}
+      finding['status'] = 'failed'
+      finding['code_desc'] = ''
+      if get_location(result)['uri']
+        finding['code_desc'] += " URL : #{get_location(result)['uri']}"
+      end
+      if get_location(result)['start_line']
+        finding['code_desc'] += " LINE : #{get_location(result)['start_line']}"
+      end
+      if get_location(result)['start_column']
+        finding['code_desc'] += " COLUMN : #{get_location(result)['start_column']}"
+      end
+      finding['code_desc'].strip!
+      finding['run_time'] = NA_FLOAT
+      finding['start_time'] = NA_STRING
+      finding
+    end
+
+    def add_nist_tag_from_cwe(cweid, taxonomy_name, tags_node)
+      entries = @cwe_nist_mapping.select { |x| cweid.include?(x[:cweid].to_s) && !x[:nistid].nil? }
+      tags = entries.map { |x| x[:nistid] }
+      result_tags = tags.empty? ? DEFAULT_NIST_TAG : tags.flatten.uniq
+      if result_tags.count.positive?
+        if !tags_node
+          tags_node = {}
+        end
+        if !tags_node.key?(taxonomy_name)
+          tags_node[taxonomy_name] = []
+        end
+        result_tags.each do |t|
+          tags_node[taxonomy_name] |= [t]
+        end
+      end
+      tags_node
+    end
+
+    def get_location(result)
+      location_info = {}
+      location_info['uri'] = result.dig('locations', 0, 'physicalLocation', 'artifactLocation', 'uri')
+      location_info['start_line'] = result.dig('locations', 0, 'physicalLocation', 'region', 'startLine')
+      location_info['start_column'] = result.dig('locations', 0, 'physicalLocation', 'region', 'startColumn')
+      location_info
+    end
+
+    def get_rule_info(run, result, rule_id)
+      finding = {}
+      driver = run.dig('tool', 'driver')
+      finding['driver_name'] = driver['name']
+      finding['driver_version'] = driver['version']
+      rules = driver['rules']
+      if rules
+        rule = rules.find { |x| x['id'].eql?(rule_id) }
+        if rule
+          finding['rule_name'] = rule&.[]('name')
+          finding['rule_short_description'] = rule&.[]('shortDescription')&.[]('text')
+          finding['rule_tags'] = get_tags(rule)
+          finding['rule_name'] = rule&.[]('messageStrings')&.[]('default')&.[]('text') unless finding['rule_name']
+        end
+      end
+      finding['rule_name'] = result&.[]('message')&.[]('text') unless finding['rule_name']
+      finding
+    end
+
+    def get_tags(rule)
+      result = {}
+      Array(rule&.[]('relationships')).each do |relationship|
+        taxonomy_name = relationship['target']['toolComponent']['name'].downcase
+        taxonomy_id = relationship['target']['id']
+        if !result.key?(taxonomy_name)
+          result[taxonomy_name] = []
+        end
+        result[taxonomy_name] |= [taxonomy_id]
+      end
+      result
+    end
+
+    def parse_identifiers(rule_tags, ref)
+      # Extracting id number from reference style CWE-297
+      rule_tags[ref.downcase].map { |e| e.downcase.split("#{ref.downcase}-")[1] }
+    rescue StandardError
+      []
+    end
+
+    def impact(severity)
+      severity_mapping = IMPACT_MAPPING[severity.to_sym]
+      severity_mapping.nil? ? 0.1 : severity_mapping
+    end
+
+    def parse_mapper
+      csv_data = CSV.read(CWE_NIST_MAPPING_FILE, **{ encoding: 'UTF-8',
+                                                   headers: true,
+                                                   header_converters: :symbol,
+                                                   converters: :all })
+      csv_data.map(&:to_hash)
+    end
+
+    def desc_tags(data, label)
+      { data: data || NA_STRING, label: label || NA_STRING }
+    end
+
+    def process_item(run, result, controls)
+      printf("\rProcessing: %s", $spinner.next)
+      control = controls.find { |x| x['id'].eql?(result['ruleId']) }
+
+      if control
+        control['results'] << finding(result)
+      else
+        rule_info = get_rule_info(run, result, result['ruleId'])
+        item = {}
+        item['tags']               = rule_info['rule_tags']
+        item['descriptions']       = []
+        item['refs']               = NA_ARRAY
+        item['source_location']    = { ref: get_location(result)['uri'], line: get_location(result)['start_line'] }
+        item['descriptions']       = NA_ARRAY
+        item['title']              = rule_info['rule_name'].to_s
+        item['id']                 = result['ruleId'].to_s
+        item['desc']               = rule_info['rule_short_description'].to_s
+        item['impact']             = impact(result['level'].to_s)
+        item['code']               = NA_STRING
+        item['results']            = [finding(result)]
+        item['tags']               = add_nist_tag_from_cwe(parse_identifiers(rule_info['rule_tags'], 'CWE'), 'nist', item['tags'])
+        controls << item
+      end
+    end
+
+    def to_hdf
+      controls = []
+      @sarif_log['runs'].each do |run|
+        run['results'].each do |result|
+          process_item(run, result, controls)
+        end
+      end
+
+      scaninfo = extract_scaninfo(@sarif_log)
+      results = HeimdallDataFormat.new(profile_name: scaninfo['policy'],
+                                       version: scaninfo['version'],
+                                       title: scaninfo['projectName'],
+                                       summary: scaninfo['summary'],
+                                       controls: controls,
+                                       target_id: scaninfo['projectName'])
+      results.to_hdf
+    end
+  end
+end

data/lib/heimdall_tools/scoutsuite_mapper.rb

@@ -0,0 +1,180 @@
+require 'json'
+require 'csv'
+require 'heimdall_tools/hdf'
+
+RESOURCE_DIR = Pathname.new(__FILE__).join('../../data')
+
+SCOUTSUITE_NIST_MAPPING_FILE = File.join(RESOURCE_DIR, 'scoutsuite-nist-mapping.csv')
+
+IMPACT_MAPPING = {
+  danger: 0.7,
+  warning: 0.5
+}.freeze
+
+DEFAULT_NIST_TAG = %w{SA-11 RA-5}.freeze
+
+INSPEC_INPUTS_MAPPING = {
+  string: 'String',
+  numeric: 'Numeric',
+  regexp: 'Regexp',
+  array: 'Array',
+  hash: 'Hash',
+  boolean: 'Boolean',
+  any: 'Any'
+}.freeze
+
+# Loading spinner sign
+$spinner = Enumerator.new do |e|
+  loop do
+    e.yield '|'
+    e.yield '/'
+    e.yield '-'
+    e.yield '\\'
+  end
+end
+
+module HeimdallTools
+  # currently only tested against an AWS based result, but ScoutSuite supports many other cloud providers such as Azure
+  class ScoutSuiteMapper
+    def initialize(scoutsuite_js)
+      begin
+        @scoutsuite_nist_mapping = parse_mapper
+      rescue StandardError => e
+        raise "Invalid Scout Suite to NIST mapping file:\nException: #{e}"
+      end
+
+      begin
+        @scoutsuite_json = scoutsuite_js.lines[1] # first line is `scoutsuite_results =\n` and second line is json
+        @report = JSON.parse(@scoutsuite_json)
+      rescue StandardError => e
+        raise "Invalid Scout Suite JavaScript file provided:\nException: #{e}"
+      end
+    end
+
+    def parse_mapper
+      csv_data = CSV.read(SCOUTSUITE_NIST_MAPPING_FILE, { encoding: 'UTF-8', headers: true, header_converters: :symbol })
+      csv_data.map(&:to_hash)
+    end
+
+    def create_attribute(name, value, required = nil, sensitive = nil, type = nil)
+      { name: name, options: { value: value, required: required, sensitive: sensitive, type: type }.compact }
+    end
+
+    def extract_scaninfo(report)
+      info = {}
+      begin
+        info['name'] = 'Scout Suite Multi-Cloud Security Auditing Tool'
+        info['version'] = report['last_run']['version']
+        info['title'] = "Scout Suite Report using #{report['last_run']['ruleset_name']} ruleset on #{report['provider_name']} with account #{report['account_id']}"
+        info['target_id'] = "#{report['last_run']['ruleset_name']} ruleset:#{report['provider_name']}:#{report['account_id']}"
+        info['summary'] = report['last_run']['ruleset_about']
+        info['attributes'] = [
+          create_attribute('account_id', report['account_id'], true, false, INSPEC_INPUTS_MAPPING[:string]),
+          create_attribute('environment', report['environment']),
+          create_attribute('ruleset', report['ruleset_name']),
+          # think at least these run_parameters are aws only
+          create_attribute('run_parameters_excluded_regions', report['last_run']['run_parameters']['excluded_regions'].join(', ')),
+          create_attribute('run_parameters_regions', report['last_run']['run_parameters']['regions'].join(', ')),
+          create_attribute('run_parameters_services', report['last_run']['run_parameters']['services'].join(', ')),
+          create_attribute('run_parameters_skipped_services', report['last_run']['run_parameters']['skipped_services'].join(', ')),
+          create_attribute('time', report['last_run']['time']),
+          create_attribute('partition', report['partition']), # think this is aws only
+          create_attribute('provider_code', report['provider_code']),
+          create_attribute('provider_name', report['provider_name']),
+        ]
+
+        info
+      rescue StandardError => e
+        raise "Error extracting report info from Scout Suite JS->JSON file:\nException: #{e}"
+      end
+    end
+
+    def nist_tag(rule)
+      entries = @scoutsuite_nist_mapping.select { |x| rule.eql?(x[:rule].to_s) && !x[:nistid].nil? }
+      tags = entries.map { |x| x[:nistid].split('|') }
+      tags.empty? ? DEFAULT_NIST_TAG : tags.flatten.uniq
+    end
+
+    def impact(severity)
+      IMPACT_MAPPING[severity.to_sym]
+    end
+
+    def desc_tags(data, label)
+      { data: data || NA_STRING, label: label || NA_STRING }
+    end
+
+    def findings(details)
+      finding = {}
+      if (details['checked_items']).zero?
+        finding['status'] = 'skipped'
+        finding['skip_message'] = 'Skipped because no items were checked'
+      elsif (details['flagged_items']).zero?
+        finding['status'] = 'passed'
+        finding['message'] = "0 flagged items out of #{details['checked_items']} checked items"
+      else # there are checked items and things were flagged
+        finding['status'] = 'failed'
+        finding['message'] = "#{details['flagged_items']} flagged items out of #{details['checked_items']} checked items:\n#{details['items'].join("\n")}"
+      end
+      finding['code_desc'] = details['description']
+      finding['start_time'] = @report['last_run']['time']
+      [finding]
+    end
+
+    def compliance(arr)
+      str = 'Compliant with '
+      arr.map do |val|
+        info = "#{val['name']}, reference #{val['reference']}, version #{val['version']}"
+        str + info
+      end.join("\n")
+    end
+
+    def to_hdf
+      controls = []
+      @report['services'].each_key do |service|
+        @report['services'][service]['findings'].each_key do |finding|
+          printf("\rProcessing: %s", $spinner.next)
+
+          finding_id = finding
+          finding_details = @report['services'][service]['findings'][finding]
+
+          item = {}
+          item['id']                 = finding_id
+          item['title']              = finding_details['description']
+
+          item['tags']               = { nist: nist_tag(finding_id) }
+
+          item['impact']             = impact(finding_details['level'])
+
+          item['desc']               = finding_details['rationale']
+
+          item['descriptions']       = []
+          item['descriptions']       << desc_tags(finding_details['remediation'], 'fix') unless finding_details['remediation'].nil?
+          item['descriptions']       << desc_tags(finding_details['service'], 'service')
+          item['descriptions']       << desc_tags(finding_details['path'], 'path')
+          item['descriptions']       << desc_tags(finding_details['id_suffix'], 'id_suffix')
+
+          item['refs']               = []
+          item['refs']               += finding_details['references'].map { |link| { url: link } } unless finding_details['references'].nil? || finding_details['references'].empty?
+          item['refs']               << { ref: compliance(finding_details['compliance']) } unless finding_details['compliance'].nil?
+
+          item['source_location']    = NA_HASH
+          item['code']               = NA_STRING
+
+          item['results']            = findings(finding_details)
+
+          controls << item
+        end
+      end
+
+      scaninfo = extract_scaninfo(@report)
+      results = HeimdallDataFormat.new(profile_name: scaninfo['name'],
+                                       version: scaninfo['version'],
+                                       title: scaninfo['title'],
+                                       summary: scaninfo['summary'],
+                                       controls: controls,
+                                       target_id: scaninfo['target_id'],
+                                       attributes: scaninfo['attributes'])
+      results.to_hdf
+    end
+  end
+end

data/lib/heimdall_tools/snyk_mapper.rb

@@ -29,9 +29,8 @@ end
 
 module HeimdallTools
   class SnykMapper
-    def initialize(synk_json, _name = nil, verbose = false)
+    def initialize(synk_json, _name = nil)
       @synk_json = synk_json
-      @verbose = verbose
 
       begin
         @cwe_nist_mapping = parse_mapper

data/lib/heimdall_tools/zap_mapper.rb

@@ -8,13 +8,10 @@ RESOURCE_DIR = Pathname.new(__FILE__).join('../../data')
 CWE_NIST_MAPPING_FILE = File.join(RESOURCE_DIR, 'cwe-nist-mapping.csv')
 DEFAULT_NIST_TAG = %w{SA-11 RA-5}.freeze
 
-# rubocop:disable Metrics/AbcSize
-
 module HeimdallTools
   class ZapMapper
-    def initialize(zap_json, name, verbose = false)
+    def initialize(zap_json, name)
       @zap_json = zap_json
-      @verbose = verbose
 
       begin
         data = JSON.parse(zap_json, symbolize_names: true)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: heimdall_tools
 version: !ruby/object:Gem::Version
-  version: 1.3.42
+  version: 1.3.47
 platform: ruby
 authors:
 - Robert Thew
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-04-08 00:00:00.000000000 Z
+date: 2021-06-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk-configservice
@@ -88,14 +88,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.10.9
+        version: '1.11'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 1.10.9
+        version: '1.11'
 - !ruby/object:Gem::Dependency
   name: openssl
   requirement: !ruby/object:Gem::Requirement
@@ -214,6 +214,7 @@ files:
 - lib/data/nessus-plugins-nist-mapping.csv
 - lib/data/nikto-nist-mapping.csv
 - lib/data/owasp-nist-mapping.csv
+- lib/data/scoutsuite-nist-mapping.csv
 - lib/heimdall_tools.rb
 - lib/heimdall_tools/aws_config_mapper.rb
 - lib/heimdall_tools/burpsuite_mapper.rb
@@ -231,6 +232,8 @@ files:
 - lib/heimdall_tools/help/nessus_mapper.md
 - lib/heimdall_tools/help/netsparker_mapper.md
 - lib/heimdall_tools/help/nikto_mapper.md
+- lib/heimdall_tools/help/sarif_mapper.md
+- lib/heimdall_tools/help/scoutsuite_mapper.md
 - lib/heimdall_tools/help/snyk_mapper.md
 - lib/heimdall_tools/help/sonarqube_mapper.md
 - lib/heimdall_tools/help/zap_mapper.md
@@ -238,6 +241,8 @@ files:
 - lib/heimdall_tools/nessus_mapper.rb
 - lib/heimdall_tools/netsparker_mapper.rb
 - lib/heimdall_tools/nikto_mapper.rb
+- lib/heimdall_tools/sarif_mapper.rb
+- lib/heimdall_tools/scoutsuite_mapper.rb
 - lib/heimdall_tools/snyk_mapper.rb
 - lib/heimdall_tools/sonarqube_mapper.rb
 - lib/heimdall_tools/version.rb
@@ -262,7 +267,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.3
+rubygems_version: 3.2.15
 signing_key: 
 specification_version: 4
 summary: Convert Forify, Openzap and Sonarqube results to HDF