miteru 0.14.6 → 1.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4e33e427231064c47dd4e1358e71e009ebc9418af15350e1f9c7f833ea02205f
4
- data.tar.gz: cf3e3f1608a6be63b7b5b93f1e12a8264935da1e059bfbda7437f91a8009bf63
3
+ metadata.gz: 2217ef5ac9ed2060203df9f5edebbac779fbebc53e4418f54de6d4b080ee81fc
4
+ data.tar.gz: 45c934a406c619b1d1cc9d03a00103ae2b732460ca70877953aaac2d36d3b0e0
5
5
  SHA512:
6
- metadata.gz: e26651ad45bfc20c22b6584b3ea6f8419f99801f36e6673835034c4a77da3ea5ec2502273265fae8f89590551c4bd3eb42f33bc66a6fb3b873422b57e5136d40
7
- data.tar.gz: f582e8422d5d33a97b8f9d3ae7aa58fe1346fc68284b867c0139f7bb908107b3b8eb8081f1bcdc3bbfb678427d318e6eaf269ce1d28647d317ec7dd8ed49a7d8
6
+ metadata.gz: 03ebfc42660abedfa61a7e931e0d863b03fc50d615cfe0c81db10c0473ae8e0908619c5bcbc656e48784d40b5ccc5b98d616efe467d2a68ecca1bb57415bacfb
7
+ data.tar.gz: a74accbf1d3949e8211792e02c68e6e6911acbe2cc0cd113459eb812bad29ae0dc4c434f56009a1e49fa255d58bfc5485b9cbe8e3d6e936fef68e23d671d7d86
@@ -0,0 +1,68 @@
1
+ name: Ruby CI
2
+
3
+ on: [push, pull_request]
4
+
5
+ jobs:
6
+ build:
7
+ runs-on: ubuntu-latest
8
+
9
+ services:
10
+ postgres:
11
+ image: postgres:12
12
+ env:
13
+ POSTGRES_USER: postgres
14
+ POSTGRES_PASSWORD: postgres
15
+ POSTGRES_DB: test
16
+ options: >-
17
+ --health-cmd pg_isready
18
+ --health-interval 10s
19
+ --health-timeout 5s
20
+ --health-retries 5
21
+ ports:
22
+ - 5432:5432
23
+
24
+ mysql:
25
+ image: mysql:8.0
26
+ env:
27
+ MYSQL_USER: mysql
28
+ MYSQL_PASSWORD: mysql
29
+ MYSQL_DATABASE: test
30
+ MYSQL_ROOT_PASSWORD: rootpassword
31
+ ports:
32
+ - 3306:3306
33
+ options: >-
34
+ --health-cmd="mysqladmin ping"
35
+ --health-interval=10s
36
+ --health-timeout=5s
37
+ --health-retries=3
38
+
39
+ strategy:
40
+ fail-fast: false
41
+ matrix:
42
+ ruby: [2.7, "3.0"]
43
+
44
+ steps:
45
+ - uses: actions/checkout@v2
46
+ - name: Set up Ruby
47
+ uses: ruby/setup-ruby@v1
48
+ with:
49
+ ruby-version: ${{ matrix.ruby }}
50
+ bundler-cache: true
51
+
52
+ - name: Install dependencies
53
+ run: |
54
+ sudo apt-get -yqq install libpq-dev libmysqlclient-dev
55
+ gem install bundler
56
+ bundle install
57
+
58
+ - name: Test with PostgreSQL
59
+ env:
60
+ MITERU_DATABASE: postgresql://postgres:postgres@localhost:5432/test
61
+ run: |
62
+ bundle exec rake
63
+
64
+ - name: Test with MySQL
65
+ env:
66
+ MITERU_DATABASE: mysql2://mysql:mysql@127.0.0.1:3306/test
67
+ run: |
68
+ bundle exec rake
data/.gitignore CHANGED
@@ -51,3 +51,6 @@ Gemfile.lock
51
51
 
52
52
  ## RSpec
53
53
  .rspec_status
54
+
55
+ # SQLite database
56
+ *.db
data/.overcommit.yml ADDED
@@ -0,0 +1,12 @@
1
+ PreCommit:
2
+ BundleCheck:
3
+ enabled: true
4
+
5
+ RuboCop:
6
+ enabled: true
7
+ required_executable: bundle
8
+ command: ["bundle", "exec", "standardrb"]
9
+ on_warn: fail
10
+
11
+ YamlSyntax:
12
+ enabled: true
data/.standard.yml ADDED
@@ -0,0 +1,4 @@
1
+ ignore:
2
+ - "**/*":
3
+ - Layout/SpaceInsideHashLiteralBraces
4
+ - Style/RescueStandardError
data/README.md CHANGED
@@ -1,109 +1,40 @@
1
1
  # Miteru
2
2
 
3
3
  [![Gem Version](https://badge.fury.io/rb/miteru.svg)](https://badge.fury.io/rb/miteru)
4
- [![Build Status](https://travis-ci.com/ninoseki/miteru.svg?branch=master)](https://travis-ci.com/ninoseki/miteru)
5
- [![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/ninoseki/miteru)](https://hub.docker.com/repository/docker/ninoseki/miteru)
4
+ [![Ruby CI](https://github.com/ninoseki/miteru/actions/workflows/test.yml/badge.svg)](https://github.com/ninoseki/miteru/actions/workflows/test.yml)
6
5
  [![CodeFactor](https://www.codefactor.io/repository/github/ninoseki/miteru/badge)](https://www.codefactor.io/repository/github/ninoseki/miteru)
7
6
  [![Coverage Status](https://coveralls.io/repos/github/ninoseki/miteru/badge.svg?branch=master)](https://coveralls.io/github/ninoseki/miteru?branch=master)
8
7
 
9
8
  Miteru is an experimental phishing kit detection tool.
10
9
 
10
+ ## Disclaimer
11
+
12
+ This tool is for research purposes only. The use of this tool is your responsibility.
13
+ I take no responsibility and/or liability for how you choose to use this tool.
14
+
11
15
  ## How it works
12
16
 
13
17
  - It collects phishy URLs from the following feeds:
14
- - [CertStream-Suspicious feed via urlscan.io](https://urlscan.io/search/#certstream-suspicious)
15
- - [OpenPhish feed via urlscan.io](https://urlscan.io/search/#OpenPhish)
16
- - [PhishTank feed via urlscan.io](https://urlscan.io/search/#PhishTank)
17
- - [URLhaus feed via urlscan.io](https://urlscan.io/search/#URLHaus)
18
+ - [CertStream-Suspicious feed via urlscan.io](https://urlscan.io/search/#task.source%3Acertstream-suspicious)
19
+ - [OpenPhish feed via urlscan.io](https://urlscan.io/search/#task.source%3Aopenphish)
20
+ - [PhishTank feed via urlscan.io](https://urlscan.io/search/#task.source%3Aphishtank)
21
+ - [URLhaus feed via urlscan.io](https://urlscan.io/search/#task.source%3Aurlhaus)
18
22
  - urlscan.io phish feed (available for Pro users)
19
23
  - [Ayashige feed](https://github.com/ninoseki/ayashige)
20
24
  - [Phishing Database feed](https://github.com/mitchellkrogza/Phishing.Database)
21
25
  - [PhishStats feed](https://phishstats.info/)
22
26
  - It checks each phishy URL whether it enables directory listing and contains a phishing kit (compressed file) or not.
23
- - Note: compressed file = `*.zip`, `*.rar`, `*.7z`, `*.tar` and `*.gz`.
27
+ - Note: Supported compressed files are: `*.zip`, `*.rar`, `*.7z`, `*.tar` and `*.gz`.
24
28
 
25
29
  ## Features
26
30
 
27
- - [x] Phishing kit detection & collection.
28
- - [x] Slack notification.
29
- - [x] Threading.
30
-
31
- ## Installation
32
-
33
- ```bash
34
- gem install miteru
35
- ```
36
-
37
- ## Usage
38
-
39
- ```bash
40
- $ miteru
41
- Commands:
42
- miteru execute # Execute the crawler
43
- miteru help [COMMAND] # Describe available commands or one specific command
44
- ```
45
-
46
- ```bash
47
- $ miteru help execute
48
- Usage:
49
- miteru execute
50
-
51
- Options:
52
- [--auto-download], [--no-auto-download] # Enable or disable auto-download of phishing kits
53
- [--ayashige], [--no-ayashige] # Enable or disable ayashige(ninoseki/ayashige) feed
54
- [--directory-traveling], [--no-directory-traveling] # Enable or disable directory traveling
55
- [--download-to=DOWNLOAD_TO] # Directory to download file(s)
56
- # Default: /tmp
57
- [--post-to-slack], [--no-post-to-slack] # Post a message to Slack if it detects a phishing kit
58
- [--size=N] # Number of urlscan.io's results. (Max: 10,000)
59
- # Default: 100
60
- [--threads=N] # Number of threads to use
61
- [--verbose], [--no-verbose]
62
- # Default: true
63
-
64
- Execute the crawler
65
- ```
66
-
67
- ```bash
68
- $ miteru execute
69
- ...
70
- https://dummy1.com: it doesn't contain a phishing kit.
71
- https://dummy2.com: it doesn't contain a phishing kit.
72
- https://dummy3.com: it doesn't contain a phishing kit.
73
- https://dummy4.com: it might contain a phishing kit (dummy.zip).
74
- ```
75
-
76
- ## Using Docker (alternative if you don't install Ruby)
77
-
78
- ```bash
79
- $ docker pull ninoseki/miteru
80
- # ex. auto-download detected phishing kit(s) into host machines's /tmp directory
81
- $ docker run --rm -v /tmp:/tmp ninoseki/miteru execute --auto-download
82
- ```
83
-
84
- ## Configuration
85
-
86
- For using `--post-to-slack` feature, you should set the following environment variables:
87
-
88
- - `SLACK_WEBHOOK_URL`: Your Slack Webhook URL.
89
- - `SLACK_CHANNEL`: Slack channel to post a message (default: "#general").
90
-
91
- If you are a urlscan.io Pro user, set your API key as an environment variable `URLSCAN_API_KEY`.
92
-
93
- It enables you to subscribe the urlscan.io phish feed.
94
-
95
- ## Examples
96
-
97
- ### Aasciinema cast
98
-
99
- [![asciicast](https://asciinema.org/a/hHpkHhMLiiv17gmdRhVMtZWwM.svg)](https://asciinema.org/a/hHpkHhMLiiv17gmdRhVMtZWwM)
100
-
101
- ### Slack notification
102
-
103
- ![img](./screenshots/slack.png)
31
+ - [x] Phishing kit detection & collection
32
+ - [x] Slack notification
33
+ - [x] Threading
104
34
 
105
- ## Alternatives
35
+ ## Docs
106
36
 
107
- - [t4d/StalkPhish](https://github.com/t4d/StalkPhish): The Phishing kits stalker, harvesting phishing kits for investigations.
108
- - [duo-labs/phish-collect](https://github.com/duo-labs/phish-collect): Python script to hunt phishing kits.
109
- - [leunammejii/analyst_arsenal](https://github.com/leunammejii/analyst_arsenal): A tool belt for analysts to continue fighting the good fight.
37
+ - [Requirements & Installation](https://github.com/ninoseki/miteru/wiki/Requirements-&-Installation)
38
+ - [Usage](https://github.com/ninoseki/miteru/wiki/Usage)
39
+ - [Configuration](https://github.com/ninoseki/miteru/wiki/Configuration)
40
+ - [Alternatives](https://github.com/ninoseki/miteru/wiki/Alternatives)
data/docker/Dockerfile CHANGED
@@ -1,10 +1,13 @@
1
- FROM ruby:2.7-alpine3.10
2
- RUN apk --no-cache add git build-base ruby-dev \
1
+ FROM ruby:3-alpine3.13
2
+
3
+ RUN apk --no-cache add git build-base ruby-dev mysql-client mysql-dev sqlite-dev postgresql-client postgresql-dev \
3
4
  && cd /tmp/ \
4
5
  && git clone https://github.com/ninoseki/miteru.git \
5
6
  && cd miteru \
6
7
  && gem build miteru.gemspec -o miteru.gem \
7
8
  && gem install miteru.gem \
9
+ && gem install mysql2 \
10
+ && gem install pg \
8
11
  && rm -rf /tmp/miteru \
9
12
  && apk del --purge git build-base ruby-dev
10
13
 
data/exe/miteru CHANGED
@@ -5,4 +5,5 @@ $LOAD_PATH.unshift("#{__dir__}/../lib")
5
5
 
6
6
  require "miteru"
7
7
 
8
- Miteru::CLI.start
8
+ ARGV.unshift(Miteru::CLI.default_task) unless Miteru::CLI.all_tasks.key?(ARGV[0])
9
+ Miteru::CLI.start(ARGV)
@@ -5,6 +5,7 @@ require "uri"
5
5
  module Miteru
6
6
  class Attachement
7
7
  attr_reader :url
8
+
8
9
  def initialize(url)
9
10
  @url = url
10
11
  end
@@ -31,7 +32,7 @@ module Miteru
31
32
  {
32
33
  type: "button",
33
34
  text: "Lookup on VirusTotal",
34
- url: _vt_link,
35
+ url: _vt_link
35
36
  }
36
37
  end
37
38
 
@@ -41,7 +42,7 @@ module Miteru
41
42
  {
42
43
  type: "button",
43
44
  text: "Lookup on urlscan.io",
44
- url: _urlscan_link,
45
+ url: _urlscan_link
45
46
  }
46
47
  end
47
48
 
data/lib/miteru/cli.rb CHANGED
@@ -5,11 +5,11 @@ require "thor"
5
5
  module Miteru
6
6
  class CLI < Thor
7
7
  method_option :auto_download, type: :boolean, default: false, desc: "Enable or disable auto-download of phishing kits"
8
- method_option :ayashige, type: :boolean, default: false, desc: "Enable or disable ayashige(ninoseki/ayashige) feed"
8
+ method_option :ayashige, type: :boolean, default: false, desc: "Enable or disable Ayashige(ninoseki/ayashige) feed"
9
9
  method_option :directory_traveling, type: :boolean, default: false, desc: "Enable or disable directory traveling"
10
- method_option :download_to, type: :string, default: "/tmp", desc: "Directory to download file(s)"
11
- method_option :post_to_slack, type: :boolean, default: false, desc: "Post a message to Slack if it detects a phishing kit"
12
- method_option :size, type: :numeric, default: 100, desc: "Number of urlscan.io's results. (Max: 10,000)"
10
+ method_option :download_to, type: :string, default: "/tmp", desc: "Directory to download phishing kits"
11
+ method_option :post_to_slack, type: :boolean, default: false, desc: "Enable or disable Slack notification"
12
+ method_option :size, type: :numeric, default: 100, desc: "Number of urlscan.io's results to fetch. (Max: 10,000)"
13
13
  method_option :threads, type: :numeric, desc: "Number of threads to use"
14
14
  method_option :verbose, type: :boolean, default: true
15
15
  desc "execute", "Execute the crawler"
@@ -29,5 +29,13 @@ module Miteru
29
29
 
30
30
  Crawler.execute
31
31
  end
32
+
33
+ default_command :execute
34
+
35
+ class << self
36
+ def exit_on_failure?
37
+ true
38
+ end
39
+ end
32
40
  end
33
41
  end
@@ -28,8 +28,19 @@ module Miteru
28
28
  # @return [Boolean]
29
29
  attr_accessor :verbose
30
30
 
31
+ # @return [String]
32
+ attr_accessor :database
33
+
34
+ # @return [String, nil]
35
+ attr_accessor :slack_webhook_url
36
+
37
+ # @return [String]
38
+ attr_accessor :slack_channel
39
+
40
+ # @return [Array<String>]
31
41
  attr_reader :valid_extensions
32
42
 
43
+ # @return [Array<String>]
33
44
  attr_reader :valid_mime_types
34
45
 
35
46
  def initialize
@@ -41,6 +52,10 @@ module Miteru
41
52
  @size = 100
42
53
  @threads = Parallel.processor_count
43
54
  @verbose = false
55
+ @database = ENV["MITERU_DATABASE"] || "miteru.db"
56
+
57
+ @slack_webhook_url = ENV["SLACK_WEBHOOK_URL"]
58
+ @slack_channel = ENV["SLACK_CHANNEL"] || "#general"
44
59
 
45
60
  @valid_extensions = [".zip", ".rar", ".7z", ".tar", ".gz"].freeze
46
61
  @valid_mime_types = ["application/zip", "application/vnd.rar", "application/x-7z-compressed", "application/x-tar", "application/gzip"]
@@ -65,6 +80,10 @@ module Miteru
65
80
  def verbose?
66
81
  @verbose
67
82
  end
83
+
84
+ def slack_webhook_url?
85
+ @slack_webhook_url
86
+ end
68
87
  end
69
88
 
70
89
  class << self
@@ -6,8 +6,7 @@ require "uri"
6
6
 
7
7
  module Miteru
8
8
  class Crawler
9
- attr_reader :downloader
10
- attr_reader :feeds
9
+ attr_reader :downloader, :feeds
11
10
 
12
11
  def initialize
13
12
  @downloader = Downloader.new(Miteru.configuration.download_to)
@@ -0,0 +1,65 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "active_record"
4
+
5
+ class InitialSchema < ActiveRecord::Migration[6.1]
6
+ def change
7
+ create_table :records, if_not_exists: true do |t|
8
+ t.string :hash, null: false, index: { unique: true }
9
+ t.string :hostname, null: false
10
+ t.json :headers, null: false
11
+ t.text :filename, null: false
12
+ t.string :downloaded_as, null: false
13
+ t.integer :filesize, null: false
14
+ t.string :mime_type, null: false
15
+ t.text :url, null: false
16
+
17
+ t.timestamps
18
+ end
19
+ end
20
+ end
21
+
22
+ def adapter
23
+ return "postgresql" if Miteru.configuration.database.start_with?("postgresql://", "postgres://")
24
+ return "mysql2" if Miteru.configuration.database.start_with?("mysql2://")
25
+
26
+ "sqlite3"
27
+ end
28
+
29
+ module Miteru
30
+ class Database
31
+ class << self
32
+ def connect
33
+ case adapter
34
+ when "postgresql", "mysql2"
35
+ ActiveRecord::Base.establish_connection(Miteru.configuration.database)
36
+ else
37
+ ActiveRecord::Base.establish_connection(
38
+ adapter: adapter,
39
+ database: Miteru.configuration.database
40
+ )
41
+ end
42
+
43
+ # ActiveRecord::Base.logger = Logger.new STDOUT
44
+ ActiveRecord::Migration.verbose = false
45
+
46
+ InitialSchema.migrate(:up)
47
+ rescue StandardError => _e
48
+ # Do nothing
49
+ end
50
+
51
+ def close
52
+ ActiveRecord::Base.clear_active_connections!
53
+ ActiveRecord::Base.connection.close
54
+ end
55
+
56
+ def destroy!
57
+ return unless ActiveRecord::Base.connected?
58
+
59
+ InitialSchema.migrate(:down)
60
+ end
61
+ end
62
+ end
63
+ end
64
+
65
+ Miteru::Database.connect
@@ -6,13 +6,12 @@ require "uri"
6
6
 
7
7
  module Miteru
8
8
  class Downloader
9
- attr_reader :base_dir
10
- attr_reader :memo
9
+ attr_reader :base_dir, :memo
11
10
 
12
11
  def initialize(base_dir = "/tmp")
13
12
  @base_dir = base_dir
14
13
  @memo = {}
15
- raise ArgumentError, "#{base_dir} is not exist." unless Dir.exist?(base_dir)
14
+ raise ArgumentError, "#{base_dir} doesn't exist." unless Dir.exist?(base_dir)
16
15
  end
17
16
 
18
17
  def download_kits(kits)
@@ -22,23 +21,29 @@ module Miteru
22
21
  private
23
22
 
24
23
  def download_kit(kit)
25
- destination = kit.download_filepath
24
+ destination = kit.filepath_to_download
25
+
26
26
  begin
27
- downloaded_filepath = HTTPClient.download(kit.url, destination)
28
- hash = sha256(downloaded_filepath)
29
- if duplicated?(hash)
30
- puts "Do not download #{kit.url} because there is a duplicate file in the directory (SHA256: #{hash})."
31
- FileUtils.rm downloaded_filepath
32
- else
33
- puts "Download #{kit.url} as #{downloaded_filepath}"
34
- end
27
+ downloaded_as = HTTPClient.download(kit.url, destination)
35
28
  rescue Down::Error => e
36
29
  puts "Failed to download: #{kit.url} (#{e})"
30
+ return
37
31
  end
38
- end
39
32
 
40
- def filepath_to_download(filename)
41
- "#{base_dir}/#{filename}"
33
+ hash = sha256(downloaded_as)
34
+
35
+ ActiveRecord::Base.connection_pool.with_connection do
36
+ # Remove a downloaded file if it is not unique
37
+ unless Record.unique_hash?(hash)
38
+ puts "Don't download #{kit.url}. The same hash is already recorded. (SHA256: #{hash})."
39
+ FileUtils.rm downloaded_as
40
+ return
41
+ end
42
+
43
+ # Record a kit in DB
44
+ Record.create_by_kit_and_hash(kit, hash)
45
+ puts "Download #{kit.url} as #{downloaded_as}"
46
+ end
42
47
  end
43
48
 
44
49
  def sha256(path)
@@ -49,13 +54,5 @@ module Miteru
49
54
  memo[path] = hash
50
55
  hash
51
56
  end
52
-
53
- def sha256s
54
- Dir.glob("#{base_dir}/*.{zip,rar,7z,tar,gz}").map { |path| sha256(path) }
55
- end
56
-
57
- def duplicated?(hash)
58
- sha256s.count(hash) > 1
59
- end
60
57
  end
61
58
  end
data/lib/miteru/error.rb CHANGED
@@ -2,5 +2,6 @@
2
2
 
3
3
  module Miteru
4
4
  class HTTPResponseError < StandardError; end
5
+
5
6
  class DownloadError < StandardError; end
6
7
  end
@@ -10,10 +10,10 @@ module Miteru
10
10
  URL = "https://#{HOST}"
11
11
 
12
12
  def urls
13
- url = url_for("/feed")
13
+ url = url_for("/api/v1/domains/")
14
14
  res = JSON.parse(get(url))
15
15
 
16
- domains = res.map { |item| item["domain"] }
16
+ domains = res.map { |item| item["fqdn"] }
17
17
  domains.map do |domain|
18
18
  [
19
19
  "https://#{domain}",
@@ -6,7 +6,7 @@ require "uri"
6
6
  module Miteru
7
7
  class Feeds
8
8
  class PhishingDatabase < Feed
9
- URL = "https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-links-NEW-today.txt"
9
+ URL = "https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-links-ACTIVE-NOW.txt"
10
10
 
11
11
  def urls
12
12
  body = get(URL)
@@ -11,7 +11,7 @@ module Miteru
11
11
  def urls
12
12
  json = JSON.parse(get(URL))
13
13
  json.map do |entry|
14
- entry.dig("url")
14
+ entry["url"]
15
15
  end
16
16
  rescue HTTPResponseError, HTTP::Error, JSON::ParserError => e
17
17
  puts "Failed to load PhishStats feed (#{e})"
@@ -27,7 +27,7 @@ module Miteru
27
27
 
28
28
  res = api.pro.phishfeed
29
29
  results = res["results"] || []
30
- results.map { |result| result.dig("page_url") }
30
+ results.map { |result| result["page_url"] }
31
31
  rescue ArgumentError => _e
32
32
  []
33
33
  end
data/lib/miteru/feeds.rb CHANGED
@@ -9,7 +9,7 @@ require_relative "./feeds/urlscan_pro"
9
9
 
10
10
  module Miteru
11
11
  class Feeds
12
- IGNORE_EXTENSIONS = %w(.htm .html .php .asp .aspx .exe .txt).freeze
12
+ IGNORE_EXTENSIONS = %w[.htm .html .php .asp .aspx .exe .txt].freeze
13
13
 
14
14
  def initialize
15
15
  @feeds = [
@@ -48,7 +48,7 @@ module Miteru
48
48
  segments = uri.path.split("/")
49
49
  return [base] if segments.length.zero?
50
50
 
51
- urls = (0...segments.length).map { |idx| "#{base}#{segments[0..idx].join('/')}" }
51
+ urls = (0...segments.length).map { |idx| "#{base}#{segments[0..idx].join("/")}" }
52
52
 
53
53
  urls.reject do |breakdowned_url|
54
54
  # Reject a url which ends with specific extension names
@@ -7,7 +7,7 @@ require "uri"
7
7
  module Miteru
8
8
  class HTTPClient
9
9
  DEFAULT_UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
10
- URLSCAN_UA = "miteru/#{Miteru::VERSION}"
10
+ URLSCAN_UA = "miteru/#{Miteru::VERSION}".freeze
11
11
 
12
12
  attr_reader :ssl_context
13
13
 
@@ -27,18 +27,18 @@ module Miteru
27
27
  options = options.merge default_options
28
28
 
29
29
  HTTP.follow
30
- .timeout(3)
31
- .headers(urlscan_url?(url) ? urlscan_headers : default_headers)
32
- .head(url, options)
30
+ .timeout(3)
31
+ .headers(urlscan_url?(url) ? urlscan_headers : default_headers)
32
+ .head(url, options)
33
33
  end
34
34
 
35
35
  def get(url, options = {})
36
36
  options = options.merge default_options
37
37
 
38
38
  HTTP.follow
39
- .timeout(write: 2, connect: 5, read: 10)
40
- .headers(urlscan_url?(url) ? urlscan_headers : default_headers)
41
- .get(url, options)
39
+ .timeout(write: 2, connect: 5, read: 10)
40
+ .headers(urlscan_url?(url) ? urlscan_headers : default_headers)
41
+ .get(url, options)
42
42
  end
43
43
 
44
44
  def post(url, options = {})
data/lib/miteru/kit.rb CHANGED
@@ -1,18 +1,15 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "cgi"
4
- require "securerandom"
4
+ require "uuidtools"
5
+ require "uri"
5
6
 
6
7
  module Miteru
7
8
  class Kit
8
9
  VALID_EXTENSIONS = Miteru.configuration.valid_extensions
9
10
  VALID_MIME_TYPES = Miteru.configuration.valid_mime_types
10
11
 
11
- attr_reader :url
12
-
13
- attr_reader :status
14
- attr_reader :content_length
15
- attr_reader :mime_type
12
+ attr_reader :url, :status, :content_length, :mime_type, :headers
16
13
 
17
14
  def initialize(url)
18
15
  @url = url
@@ -20,9 +17,10 @@ module Miteru
20
17
  @content_length = nil
21
18
  @mime_type = nil
22
19
  @status = nil
20
+ @headers = nil
23
21
  end
24
22
 
25
- def valid?;
23
+ def valid?
26
24
  # make a HEAD request for the validation
27
25
  before_validation
28
26
 
@@ -36,41 +34,50 @@ module Miteru
36
34
  end
37
35
 
38
36
  def basename
39
- File.basename(url)
37
+ @basename ||= File.basename(url)
40
38
  end
41
39
 
42
40
  def filename
43
- CGI.unescape basename
41
+ @filename ||= CGI.unescape(basename)
42
+ end
43
+
44
+ def filepath_to_download
45
+ "#{base_dir}/#{filename_to_download}"
44
46
  end
45
47
 
46
- def download_filepath
47
- "#{base_dir}/#{download_filename}"
48
+ def downloaded?
49
+ File.exist?(filepath_to_download)
48
50
  end
49
51
 
50
52
  def filesize
51
- return nil unless File.exist?(download_filepath)
53
+ return nil unless downloaded?
52
54
 
53
- File.size download_filepath
55
+ File.size filepath_to_download
54
56
  end
55
57
 
56
58
  def filename_with_size
57
59
  return filename unless filesize
58
60
 
59
- "#{filename}(#{filesize / 1024}KB)"
61
+ kb = (filesize.to_f / 1024.0).ceil
62
+ "#{filename}(#{kb}KB)"
60
63
  end
61
64
 
62
- private
63
-
64
65
  def id
65
- @id ||= SecureRandom.hex(10)
66
+ @id ||= UUIDTools::UUID.random_create.to_s
66
67
  end
67
68
 
68
69
  def hostname
69
- URI(url).hostname
70
+ @hostname ||= URI(url).hostname
70
71
  end
71
72
 
72
- def download_filename
73
- "#{hostname}_#{filename}_#{id}#{extname}"
73
+ def decoded_url
74
+ @decoded_url ||= URI.decode_www_form_component(url)
75
+ end
76
+
77
+ private
78
+
79
+ def filename_to_download
80
+ "#{id}#{extname}"
74
81
  end
75
82
 
76
83
  def base_dir
@@ -86,6 +93,7 @@ module Miteru
86
93
  @content_length = res.content_length
87
94
  @mime_type = res.content_type.mime_type.to_s
88
95
  @status = res.status
96
+ @headers = res.headers.to_h
89
97
  rescue StandardError
90
98
  # do nothing
91
99
  end
@@ -7,31 +7,19 @@ module Miteru
7
7
  class Notifier
8
8
  def notify(url:, kits:, message:)
9
9
  attachement = Attachement.new(url)
10
- kits = kits.select(&:filesize)
10
+ kits = kits.select(&:downloaded?)
11
11
 
12
- if post_to_slack? && kits.any?
13
- notifier = Slack::Notifier.new(slack_webhook_url, channel: slack_channel)
14
- notifier.post(text: message, attachments: attachement.to_a)
12
+ if notifiable? && kits.any?
13
+ notifier = Slack::Notifier.new(Miteru.configuration.slack_webhook_url, channel: Miteru.configuration.slack_channel)
14
+ notifier.post(text: message.capitalize, attachments: attachement.to_a)
15
15
  end
16
16
 
17
17
  message = message.colorize(:light_red) if kits.any?
18
18
  puts "#{url}: #{message}"
19
19
  end
20
20
 
21
- def post_to_slack?
22
- slack_webhook_url? && Miteru.configuration.post_to_slack?
23
- end
24
-
25
- def slack_webhook_url
26
- ENV.fetch "SLACK_WEBHOOK_URL"
27
- end
28
-
29
- def slack_channel
30
- ENV.fetch "SLACK_CHANNEL", "#general"
31
- end
32
-
33
- def slack_webhook_url?
34
- ENV.key? "SLACK_WEBHOOK_URL"
21
+ def notifiable?
22
+ Miteru.configuration.slack_webhook_url? && Miteru.configuration.post_to_slack?
35
23
  end
36
24
  end
37
25
  end
@@ -0,0 +1,48 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "active_record"
4
+
5
+ module Miteru
6
+ class Record < ActiveRecord::Base
7
+ class << self
8
+ #
9
+ # Check uniqueness of a record by a hash
10
+ #
11
+ # @param [String] hash
12
+ #
13
+ # @return [Boolean] true if it is unique. Otherwise false.
14
+ #
15
+ def unique_hash?(hash)
16
+ record = find_by(hash: hash)
17
+ return true if record.nil?
18
+
19
+ false
20
+ end
21
+
22
+ #
23
+ # Create a new record based on a kit
24
+ #
25
+ # @param [Miteru::Kit] kit
26
+ # @param [String] hash
27
+ #
28
+ # @return [Miteru::Record]
29
+ #
30
+ def create_by_kit_and_hash(kit, hash)
31
+ record = new(
32
+ hash: hash,
33
+ hostname: kit.hostname,
34
+ url: kit.decoded_url,
35
+ headers: kit.headers,
36
+ filename: kit.filename,
37
+ filesize: kit.filesize,
38
+ mime_type: kit.mime_type,
39
+ downloaded_as: kit.filepath_to_download
40
+ )
41
+ record.save
42
+ record
43
+ rescue TypeError, ActiveRecord::RecordNotUnique => _e
44
+ nil
45
+ end
46
+ end
47
+ end
48
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Miteru
4
- VERSION = "0.14.6"
4
+ VERSION = "1.0.2"
5
5
  end
@@ -17,10 +17,10 @@ module Miteru
17
17
  end
18
18
 
19
19
  def kits
20
- @kits ||= links.map do |link|
20
+ @kits ||= links.filter_map do |link|
21
21
  kit = Kit.new(link)
22
22
  kit.valid? ? kit : nil
23
- end.compact
23
+ end
24
24
  end
25
25
 
26
26
  def ok?
@@ -42,11 +42,11 @@ module Miteru
42
42
  end
43
43
 
44
44
  def message
45
- return "It doesn't contain a phishing kit." unless kits?
45
+ return "it doesn't contain a phishing kit." unless kits?
46
46
 
47
47
  filename_with_sizes = kits.map(&:filename_with_size).join(", ")
48
48
  noun = kits.length == 1 ? "a phishing kit" : "phishing kits"
49
- "It might contain #{noun}: #{filename_with_sizes}."
49
+ "it might contain #{noun}: #{filename_with_sizes}."
50
50
  end
51
51
 
52
52
  def links
@@ -75,7 +75,7 @@ module Miteru
75
75
 
76
76
  def href_links
77
77
  if doc && ok? && index?
78
- doc.css("a").map { |a| a.get("href") }.compact.map do |href|
78
+ doc.css("a").filter_map { |a| a.get("href") }.map do |href|
79
79
  href = href.start_with?("/") ? href : "/#{href}"
80
80
  url + href
81
81
  end
data/lib/miteru.rb CHANGED
@@ -3,6 +3,10 @@
3
3
  require "miteru/version"
4
4
 
5
5
  require "miteru/configuration"
6
+ require "miteru/database"
7
+
8
+ require "miteru/record"
9
+
6
10
  require "miteru/error"
7
11
  require "miteru/http_client"
8
12
  require "miteru/kit"
@@ -14,6 +18,4 @@ require "miteru/notifier"
14
18
  require "miteru/crawler"
15
19
  require "miteru/cli"
16
20
 
17
- module Miteru
18
- # Your code goes here...
19
- end
21
+ module Miteru; end
data/miteru.gemspec CHANGED
@@ -1,43 +1,51 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- lib = File.expand_path('lib', __dir__)
3
+ lib = File.expand_path("lib", __dir__)
4
4
  $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
5
5
  require "miteru/version"
6
6
 
7
7
  Gem::Specification.new do |spec|
8
- spec.name = "miteru"
9
- spec.version = Miteru::VERSION
10
- spec.authors = ["Manabu Niseki"]
11
- spec.email = ["manabu.niseki@gmail.com"]
8
+ spec.name = "miteru"
9
+ spec.version = Miteru::VERSION
10
+ spec.authors = ["Manabu Niseki"]
11
+ spec.email = ["manabu.niseki@gmail.com"]
12
12
 
13
- spec.summary = "An experimental phishing kit detector"
14
- spec.description = "An experimental phishing kit detector"
15
- spec.homepage = "https://github.com/ninoseki/miteru"
16
- spec.license = "MIT"
13
+ spec.summary = "An experimental phishing kit detector"
14
+ spec.description = "An experimental phishing kit detector"
15
+ spec.homepage = "https://github.com/ninoseki/miteru"
16
+ spec.license = "MIT"
17
17
 
18
18
  # Specify which files should be added to the gem when it is released.
19
19
  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
20
20
  spec.files = Dir.chdir(File.expand_path(__dir__)) do
21
21
  `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
22
22
  end
23
- spec.bindir = "exe"
24
- spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
23
+ spec.bindir = "exe"
24
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
25
25
  spec.require_paths = ["lib"]
26
26
 
27
- spec.add_development_dependency "bundler", "~> 2.1"
28
- spec.add_development_dependency "coveralls", "~> 0.8"
27
+ spec.add_development_dependency "bundler", "~> 2.2"
28
+ spec.add_development_dependency "coveralls_reborn", "~> 0.22"
29
29
  spec.add_development_dependency "glint", "~> 0.1"
30
+ spec.add_development_dependency "mysql2", "~> 0.5"
31
+ spec.add_development_dependency "overcommit", "~> 0.58"
32
+ spec.add_development_dependency "pg", "~> 1.2"
30
33
  spec.add_development_dependency "rake", "~> 13.0"
31
- spec.add_development_dependency "rspec", "~> 3.9"
34
+ spec.add_development_dependency "rspec", "~> 3.10"
35
+ spec.add_development_dependency "standard", "~> 1.3"
32
36
  spec.add_development_dependency "vcr", "~> 6.0"
33
- spec.add_development_dependency "webmock", "~> 3.9"
37
+ spec.add_development_dependency "webmock", "~> 3.14"
38
+ spec.add_development_dependency "webrick", "~> 1.7.0"
34
39
 
40
+ spec.add_dependency "activerecord", "~> 6.1"
35
41
  spec.add_dependency "colorize", "~> 0.8"
36
42
  spec.add_dependency "down", "~> 5.2"
37
- spec.add_dependency "http", "~> 4.4"
43
+ spec.add_dependency "http", "~> 5.0"
38
44
  spec.add_dependency "oga", "~> 3.3"
39
- spec.add_dependency "parallel", "~> 1.19"
40
- spec.add_dependency "slack-notifier", "~> 2.3"
41
- spec.add_dependency "thor", "~> 1.0"
42
- spec.add_dependency "urlscan", "~> 0.6"
45
+ spec.add_dependency "parallel", "~> 1.20"
46
+ spec.add_dependency "slack-notifier", "~> 2.4"
47
+ spec.add_dependency "sqlite3", "~> 1.4"
48
+ spec.add_dependency "thor", "~> 1.1"
49
+ spec.add_dependency "urlscan", "~> 0.7"
50
+ spec.add_dependency "uuidtools", "~> 2.2"
43
51
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: miteru
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.14.6
4
+ version: 1.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Manabu Niseki
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2020-10-04 00:00:00.000000000 Z
11
+ date: 2021-09-10 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -16,28 +16,28 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '2.1'
19
+ version: '2.2'
20
20
  type: :development
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '2.1'
26
+ version: '2.2'
27
27
  - !ruby/object:Gem::Dependency
28
- name: coveralls
28
+ name: coveralls_reborn
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
31
  - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: '0.8'
33
+ version: '0.22'
34
34
  type: :development
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '0.8'
40
+ version: '0.22'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: glint
43
43
  requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,48 @@ dependencies:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0.1'
55
+ - !ruby/object:Gem::Dependency
56
+ name: mysql2
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '0.5'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '0.5'
69
+ - !ruby/object:Gem::Dependency
70
+ name: overcommit
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - "~>"
74
+ - !ruby/object:Gem::Version
75
+ version: '0.58'
76
+ type: :development
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - "~>"
81
+ - !ruby/object:Gem::Version
82
+ version: '0.58'
83
+ - !ruby/object:Gem::Dependency
84
+ name: pg
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - "~>"
88
+ - !ruby/object:Gem::Version
89
+ version: '1.2'
90
+ type: :development
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - "~>"
95
+ - !ruby/object:Gem::Version
96
+ version: '1.2'
55
97
  - !ruby/object:Gem::Dependency
56
98
  name: rake
57
99
  requirement: !ruby/object:Gem::Requirement
@@ -72,14 +114,28 @@ dependencies:
72
114
  requirements:
73
115
  - - "~>"
74
116
  - !ruby/object:Gem::Version
75
- version: '3.9'
117
+ version: '3.10'
118
+ type: :development
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - "~>"
123
+ - !ruby/object:Gem::Version
124
+ version: '3.10'
125
+ - !ruby/object:Gem::Dependency
126
+ name: standard
127
+ requirement: !ruby/object:Gem::Requirement
128
+ requirements:
129
+ - - "~>"
130
+ - !ruby/object:Gem::Version
131
+ version: '1.3'
76
132
  type: :development
77
133
  prerelease: false
78
134
  version_requirements: !ruby/object:Gem::Requirement
79
135
  requirements:
80
136
  - - "~>"
81
137
  - !ruby/object:Gem::Version
82
- version: '3.9'
138
+ version: '1.3'
83
139
  - !ruby/object:Gem::Dependency
84
140
  name: vcr
85
141
  requirement: !ruby/object:Gem::Requirement
@@ -100,14 +156,42 @@ dependencies:
100
156
  requirements:
101
157
  - - "~>"
102
158
  - !ruby/object:Gem::Version
103
- version: '3.9'
159
+ version: '3.14'
160
+ type: :development
161
+ prerelease: false
162
+ version_requirements: !ruby/object:Gem::Requirement
163
+ requirements:
164
+ - - "~>"
165
+ - !ruby/object:Gem::Version
166
+ version: '3.14'
167
+ - !ruby/object:Gem::Dependency
168
+ name: webrick
169
+ requirement: !ruby/object:Gem::Requirement
170
+ requirements:
171
+ - - "~>"
172
+ - !ruby/object:Gem::Version
173
+ version: 1.7.0
104
174
  type: :development
105
175
  prerelease: false
106
176
  version_requirements: !ruby/object:Gem::Requirement
107
177
  requirements:
108
178
  - - "~>"
109
179
  - !ruby/object:Gem::Version
110
- version: '3.9'
180
+ version: 1.7.0
181
+ - !ruby/object:Gem::Dependency
182
+ name: activerecord
183
+ requirement: !ruby/object:Gem::Requirement
184
+ requirements:
185
+ - - "~>"
186
+ - !ruby/object:Gem::Version
187
+ version: '6.1'
188
+ type: :runtime
189
+ prerelease: false
190
+ version_requirements: !ruby/object:Gem::Requirement
191
+ requirements:
192
+ - - "~>"
193
+ - !ruby/object:Gem::Version
194
+ version: '6.1'
111
195
  - !ruby/object:Gem::Dependency
112
196
  name: colorize
113
197
  requirement: !ruby/object:Gem::Requirement
@@ -142,14 +226,14 @@ dependencies:
142
226
  requirements:
143
227
  - - "~>"
144
228
  - !ruby/object:Gem::Version
145
- version: '4.4'
229
+ version: '5.0'
146
230
  type: :runtime
147
231
  prerelease: false
148
232
  version_requirements: !ruby/object:Gem::Requirement
149
233
  requirements:
150
234
  - - "~>"
151
235
  - !ruby/object:Gem::Version
152
- version: '4.4'
236
+ version: '5.0'
153
237
  - !ruby/object:Gem::Dependency
154
238
  name: oga
155
239
  requirement: !ruby/object:Gem::Requirement
@@ -170,56 +254,84 @@ dependencies:
170
254
  requirements:
171
255
  - - "~>"
172
256
  - !ruby/object:Gem::Version
173
- version: '1.19'
257
+ version: '1.20'
174
258
  type: :runtime
175
259
  prerelease: false
176
260
  version_requirements: !ruby/object:Gem::Requirement
177
261
  requirements:
178
262
  - - "~>"
179
263
  - !ruby/object:Gem::Version
180
- version: '1.19'
264
+ version: '1.20'
181
265
  - !ruby/object:Gem::Dependency
182
266
  name: slack-notifier
183
267
  requirement: !ruby/object:Gem::Requirement
184
268
  requirements:
185
269
  - - "~>"
186
270
  - !ruby/object:Gem::Version
187
- version: '2.3'
271
+ version: '2.4'
188
272
  type: :runtime
189
273
  prerelease: false
190
274
  version_requirements: !ruby/object:Gem::Requirement
191
275
  requirements:
192
276
  - - "~>"
193
277
  - !ruby/object:Gem::Version
194
- version: '2.3'
278
+ version: '2.4'
279
+ - !ruby/object:Gem::Dependency
280
+ name: sqlite3
281
+ requirement: !ruby/object:Gem::Requirement
282
+ requirements:
283
+ - - "~>"
284
+ - !ruby/object:Gem::Version
285
+ version: '1.4'
286
+ type: :runtime
287
+ prerelease: false
288
+ version_requirements: !ruby/object:Gem::Requirement
289
+ requirements:
290
+ - - "~>"
291
+ - !ruby/object:Gem::Version
292
+ version: '1.4'
195
293
  - !ruby/object:Gem::Dependency
196
294
  name: thor
197
295
  requirement: !ruby/object:Gem::Requirement
198
296
  requirements:
199
297
  - - "~>"
200
298
  - !ruby/object:Gem::Version
201
- version: '1.0'
299
+ version: '1.1'
202
300
  type: :runtime
203
301
  prerelease: false
204
302
  version_requirements: !ruby/object:Gem::Requirement
205
303
  requirements:
206
304
  - - "~>"
207
305
  - !ruby/object:Gem::Version
208
- version: '1.0'
306
+ version: '1.1'
209
307
  - !ruby/object:Gem::Dependency
210
308
  name: urlscan
211
309
  requirement: !ruby/object:Gem::Requirement
212
310
  requirements:
213
311
  - - "~>"
214
312
  - !ruby/object:Gem::Version
215
- version: '0.6'
313
+ version: '0.7'
314
+ type: :runtime
315
+ prerelease: false
316
+ version_requirements: !ruby/object:Gem::Requirement
317
+ requirements:
318
+ - - "~>"
319
+ - !ruby/object:Gem::Version
320
+ version: '0.7'
321
+ - !ruby/object:Gem::Dependency
322
+ name: uuidtools
323
+ requirement: !ruby/object:Gem::Requirement
324
+ requirements:
325
+ - - "~>"
326
+ - !ruby/object:Gem::Version
327
+ version: '2.2'
216
328
  type: :runtime
217
329
  prerelease: false
218
330
  version_requirements: !ruby/object:Gem::Requirement
219
331
  requirements:
220
332
  - - "~>"
221
333
  - !ruby/object:Gem::Version
222
- version: '0.6'
334
+ version: '2.2'
223
335
  description: An experimental phishing kit detector
224
336
  email:
225
337
  - manabu.niseki@gmail.com
@@ -228,9 +340,11 @@ executables:
228
340
  extensions: []
229
341
  extra_rdoc_files: []
230
342
  files:
343
+ - ".github/workflows/test.yml"
231
344
  - ".gitignore"
345
+ - ".overcommit.yml"
232
346
  - ".rspec"
233
- - ".travis.yml"
347
+ - ".standard.yml"
234
348
  - Gemfile
235
349
  - LICENSE
236
350
  - README.md
@@ -244,6 +358,7 @@ files:
244
358
  - lib/miteru/cli.rb
245
359
  - lib/miteru/configuration.rb
246
360
  - lib/miteru/crawler.rb
361
+ - lib/miteru/database.rb
247
362
  - lib/miteru/downloader.rb
248
363
  - lib/miteru/error.rb
249
364
  - lib/miteru/feeds.rb
@@ -256,6 +371,7 @@ files:
256
371
  - lib/miteru/http_client.rb
257
372
  - lib/miteru/kit.rb
258
373
  - lib/miteru/notifier.rb
374
+ - lib/miteru/record.rb
259
375
  - lib/miteru/version.rb
260
376
  - lib/miteru/website.rb
261
377
  - miteru.gemspec
@@ -280,7 +396,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
280
396
  - !ruby/object:Gem::Version
281
397
  version: '0'
282
398
  requirements: []
283
- rubygems_version: 3.1.2
399
+ rubygems_version: 3.2.22
284
400
  signing_key:
285
401
  specification_version: 4
286
402
  summary: An experimental phishing kit detector
data/.travis.yml DELETED
@@ -1,7 +0,0 @@
1
- sudo: false
2
- language: ruby
3
- cache: bundler
4
- rvm:
5
- - 2.6
6
- - 2.7
7
- before_install: gem install bundler -v 2.1