Icarus-Mod-Tools 3.1.0 → 3.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1ce21bb90f71e739bad403f4f8c30a29d4f1c10ecd374ecd4012d2273a49d583
4
- data.tar.gz: ff4adcb3bf12dae9bee5c0570c1c9b93c0ce062888014ee916ebf52fcf66e3ff
3
+ metadata.gz: de06ee7fa7b4384bd296cd428c7946edc03c9dde4d8313f8cc1b9550095d89a0
4
+ data.tar.gz: cc5488cad0c60bc7c52eb535ae241d4ee3457446802affb70f2adf60d368829e
5
5
  SHA512:
6
- metadata.gz: '08bb649e40e165fa13072434fb0f70bb62d7f67a8309f02b7ad06b2b40986510a7176771f711dcbaec0c0b727643b31a3e6d95e877e0e53cfc37f67a761153fb'
7
- data.tar.gz: 765209817e047371f78c2df064a6294c0ffbb30c40454ad1771cdabba0c93bf5d0c2384d446c38aafd3f4d6eaae587b2f12af09e42a1ccc1f7ea6d19721d2a23
6
+ metadata.gz: f3ad937f990a1b765c033af08dbef572c0e02db86138c95400ebf06b38c07c94ef41e3bd2bc692c816a906b302d2ad64da9b6a39cbdeb4f167afea315a8dcc9b
7
+ data.tar.gz: e2a1a5ede7f1733ef712d24e0c62f119757e382c27dbb31a5ef39a5ab85fdf582723c7578f5940894d42cf02d1ce9750efb37e22591152216f4eff772974970c
data/CHANGELOG.md CHANGED
@@ -4,6 +4,11 @@ All notable changes to this project will be documented in this file.
4
4
 
5
5
  ## History (reverse chronological order)
6
6
 
7
+ ### v3.2.0 - 2026-10-30
8
+
9
+ - Add `imt validate repo OWNER/REPO [--json]`, a read-only pre-check for a repo before it is added to `meta/repos` (repo state, info files, entry validation and link checks; exits 0/1/2)
10
+ - Link checks never request loopback, private, link-local or other non-public addresses (checked on every redirect, and the request goes to the checked address), since the links come from the repo being vetted
11
+
7
12
  ### v3.1.0 - 2026-10-30
8
13
 
9
14
  - `imt sync mods`/`tools --dry-run` now lists the entries a real run would delete, and warns when the mass-delete brake would refuse (a real run would exit 1 without `--force`)
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- Icarus-Mod-Tools (3.1.0)
4
+ Icarus-Mod-Tools (3.2.0)
5
5
  google-cloud-firestore (~> 3.3)
6
6
  octokit (~> 10.0)
7
7
  ostruct (~> 0.6)
data/README.md CHANGED
@@ -160,6 +160,20 @@ Options:
160
160
 
161
161
  **Note**: Sync operations automatically convert GitHub URLs to the correct `raw.githubusercontent.com` format for direct downloads. If you're using GitHub URLs in `modinfo.json` or `toolinfo.json` files, you can use either `/blob/` or `/raw/` formats and they will be automatically converted. Warnings will be displayed when URLs are auto-fixed so you can update your source files.
162
162
 
163
+ #### `imt validate`
164
+
165
+ ```sh
166
+ Commands:
167
+ imt validate help [COMMAND] # Describe subcommands or one specific subcommand
168
+ imt validate modinfo # Reads modinfo data from 'meta/modinfo/list' and Validates syntax of modfiles
169
+ imt validate repo OWNER/REPO # Vets a GitHub repo before it is added to 'meta/repos/list'
170
+ imt validate toolinfo # Reads Toolinfo data from 'meta/toolinfo/list' and Validates syntax of toolfiles
171
+ ```
172
+
173
+ `imt validate repo OWNER/REPO [--json]` (a `github.com` URL also works) vets a repo someone asked to have added, before you add it. It is read-only, never touches Firestore, and needs only the GitHub token from the config. It checks that the repo exists, is public and is not archived; finds `modinfo.json` and/or `toolinfo.json` (at least one is required); validates every entry; and requests every file, image and readme link (following up to 5 redirects) to confirm it resolves. A broken download link is an error, a broken image or readme link is a warning, and a download hosted outside GitHub or a tool shipping an `.exe` is a warning. Use `-v`/`-vv` for more detail, or `--json` to print the machine-readable result (`repo`, `exists`, `private`, `archived`, `default_branch`, `info_files`, `problems`, `entries`, `valid`) and nothing else on stdout.
174
+
175
+ **Exit status**: 0 when the repo is valid, 1 when it is not, 2 on usage, config or GitHub API errors (for example a rate limit or bad token).
176
+
163
177
  ## Development
164
178
 
165
179
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -1,6 +1,8 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require "tools/validator"
4
+ require "tools/repo_check"
5
+ require "tools/repo_report"
4
6
 
5
7
  module Icarus
6
8
  module Mod
@@ -17,6 +19,26 @@ module Icarus
17
19
  validate(:toolinfo)
18
20
  end
19
21
 
22
+ desc "repo OWNER/REPO", "Vets a GitHub repo (and its modinfo/toolinfo links) before it is added to 'meta/repos/list'; read-only"
23
+ method_option :json, type: :boolean, default: false, desc: "Print the result as JSON"
24
+ long_desc <<~DESC
25
+ Exits 0 when the repo is valid, 1 when it is not, and 2 on usage, config or GitHub API errors.
26
+ Never writes to Firestore and does not need it; only the GitHub token from the config is used.
27
+ DESC
28
+ def repo(name)
29
+ result = Icarus::Mod::Tools::RepoCheck.new(name).call
30
+
31
+ if options[:json]
32
+ puts JSON.generate(result)
33
+ else
34
+ Icarus::Mod::Tools::RepoReport.new(result, verbose:).print
35
+ end
36
+ exit(result[:valid] ? 0 : 1)
37
+ rescue Icarus::Mod::Tools::RepoCheck::Error => e
38
+ warn "Error: #{e.message}"
39
+ exit 2
40
+ end
41
+
20
42
  no_commands do
21
43
  def validate(type)
22
44
  exit_code = 0
@@ -0,0 +1,32 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "uri"
4
+ require "tools/modinfo"
5
+
6
+ module Icarus
7
+ module Mod
8
+ module Tools
9
+ # The links inside a Modinfo/Toolinfo entry that RepoCheck probes
10
+ module EntryLinks
11
+ module_function
12
+
13
+ # [field, url, download?] for every non-blank string link in the entry
14
+ def for(info)
15
+ links = info.is_a?(Modinfo) ? mod_links(info) : [["fileURL", info.fileURL, true]]
16
+ links += [["imageURL", info.imageURL, false], ["readmeURL", info.readmeURL, false]]
17
+ links.select { |_, url, _| url.is_a?(String) && !url.strip.empty? }
18
+ end
19
+
20
+ def mod_links(info)
21
+ info.file_urls.zip(info.file_types).map { |url, name| ["files.#{name}", url, true] }
22
+ end
23
+
24
+ def windows_tool?(info)
25
+ info.fileType.to_s.match?(/\A\.?exe\z/i) || URI.parse(info.fileURL.to_s).path.to_s.match?(/\.exe\z/i)
26
+ rescue URI::InvalidURIError
27
+ false
28
+ end
29
+ end
30
+ end
31
+ end
32
+ end
@@ -0,0 +1,116 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "ipaddr"
4
+ require "resolv"
5
+ require "uri"
6
+ require "net/http"
7
+ require "tools/sync/helpers"
8
+
9
+ module Icarus
10
+ module Mod
11
+ module Tools
12
+ # Requests a URL (following redirects) without downloading it, to see where it ends up
13
+ class LinkProbe
14
+ MAX_REDIRECTS = 5
15
+ REDIRECT_CODES = [301, 302, 303, 307, 308].freeze
16
+ # Statuses that mean "HEAD is not allowed here", so try a ranged GET instead
17
+ HEAD_REFUSED = [403, 405, 501].freeze
18
+ OK_STATUSES = [200, 206].freeze
19
+
20
+ # Links come from a stranger's modinfo.json and are probed from the maintainer's network, and the status
21
+ # ends up in a public comment: never request loopback, private, link-local or other non-public addresses
22
+ NON_PUBLIC = %w[0.0.0.0/8 100.64.0.0/10 192.0.0.0/24 198.18.0.0/15 224.0.0.0/3 ::/128 64:ff9b::/96 ff00::/8]
23
+ .map { |range| IPAddr.new(range) }.freeze
24
+
25
+ # http: a callable (uri, verb, ip) -> [status, location]; resolver: a callable (host) -> [addresses].
26
+ # Both are replaceable so specs need no network.
27
+ def initialize(http: nil, resolver: nil)
28
+ @http = http || method(:net_request)
29
+ @resolver = resolver || Resolv.method(:getaddresses)
30
+ end
31
+
32
+ def self.github_host?(host)
33
+ host == "github.com" || host.to_s.end_with?(".github.com") || host == "githubusercontent.com" ||
34
+ host.to_s.end_with?(".githubusercontent.com")
35
+ end
36
+
37
+ # Follows up to MAX_REDIRECTS redirects; returns [status, final_host, problem]
38
+ def call(url)
39
+ uri = URI.parse(url)
40
+ status = nil
41
+ (MAX_REDIRECTS + 1).times do
42
+ status, location, refused = hop(uri)
43
+ return [status, uri.host, refused] if refused || !(REDIRECT_CODES.include?(status) && location)
44
+
45
+ uri = URI.join(uri, location)
46
+ end
47
+ [status, uri.host, "too many redirects (more than #{MAX_REDIRECTS})"]
48
+ rescue URI::Error, *Sync::Helpers::NETWORK_ERRORS => e
49
+ [nil, nil, "request failed (#{e.message})"]
50
+ end
51
+
52
+ private
53
+
54
+ # One request, after checking where it would go: [status, location] or [nil, nil, refusal]
55
+ def hop(uri)
56
+ ips, refused = public_addresses(uri)
57
+ refused ? [nil, nil, refused] : request(uri, ips)
58
+ end
59
+
60
+ # The addresses to connect to (IPv4 first), or a reason to refuse. Every address the host resolves to must be
61
+ # public, and the request then goes to a checked address, so a DNS answer that changes in between cannot
62
+ # redirect it.
63
+ def public_addresses(uri)
64
+ return [nil, "refused: only http and https links are checked"] unless %w[http https].include?(uri.scheme)
65
+
66
+ host = uri.hostname
67
+ addresses = ip_literal?(host) ? [host] : @resolver.call(host)
68
+ return [nil, "refused: #{uri.host} does not resolve"] if addresses.empty?
69
+ return [nil, "refused: #{uri.host} resolves to a non-public address"] unless addresses.all? { |a| public_ip?(a) }
70
+
71
+ [addresses.partition { |a| IPAddr.new(a).ipv4? }.flatten, nil]
72
+ end
73
+
74
+ def ip_literal?(host)
75
+ IPAddr.new(host)
76
+ true
77
+ rescue IPAddr::InvalidAddressError
78
+ false
79
+ end
80
+
81
+ def public_ip?(address)
82
+ ip = IPAddr.new(address)
83
+ ip = ip.native if ip.ipv4_mapped?
84
+ !(ip.private? || ip.loopback? || ip.link_local? || NON_PUBLIC.any? { |range| range.include?(ip) })
85
+ rescue IPAddr::InvalidAddressError
86
+ false
87
+ end
88
+
89
+ # HEAD, then a one-byte ranged GET if HEAD is refused. An unreachable address (no IPv6 route, say) falls back
90
+ # to the next checked one; the last failure propagates.
91
+ def request(uri, ips)
92
+ ips.each_with_index do |ip, index|
93
+ result = @http.call(uri, "HEAD", ip)
94
+ return HEAD_REFUSED.include?(result.first) ? @http.call(uri, "GET", ip) : result
95
+ rescue *Sync::Helpers::NETWORK_ERRORS
96
+ raise if index == ips.size - 1
97
+ end
98
+ end
99
+
100
+ def net_request(uri, verb, ip)
101
+ req = verb == "GET" ? Net::HTTP::Get.new(uri.request_uri, "Range" => "bytes=0-0") : Net::HTTP::Head.new(uri.request_uri)
102
+ # Yield from the block so a server that ignores Range never has its whole body downloaded
103
+ connection(uri, ip).start { |conn| conn.request(req) { |res| return [res.code.to_i, res["location"]] } }
104
+ end
105
+
106
+ def connection(uri, ip)
107
+ Net::HTTP.new(uri.host, uri.port).tap do |http|
108
+ http.ipaddr = ip
109
+ http.use_ssl = uri.scheme == "https"
110
+ http.open_timeout = http.read_timeout = 15
111
+ end
112
+ end
113
+ end
114
+ end
115
+ end
116
+ end
@@ -0,0 +1,140 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "github"
4
+ require "tools/modinfo"
5
+ require "tools/toolinfo"
6
+ require "tools/link_probe"
7
+ require "tools/entry_links"
8
+ require "tools/sync/helpers"
9
+
10
+ module Icarus
11
+ module Mod
12
+ module Tools
13
+ # Read-only vetting of a GitHub repo before it is added to meta/repos: checks the repo itself, its
14
+ # modinfo.json/toolinfo.json and every link they contain. Needs only the GitHub token, never Firestore.
15
+ class RepoCheck
16
+ include Sync::Helpers
17
+
18
+ # Usage or GitHub API failure (as opposed to a repo that simply fails validation)
19
+ class Error < StandardError; end
20
+
21
+ TYPES = {
22
+ modinfo: { file: "modinfo.json", type: "mod", key: :mods, entity: Modinfo },
23
+ toolinfo: { file: "toolinfo.json", type: "tool", key: :tools, entity: Toolinfo }
24
+ }.freeze
25
+
26
+ # http/resolver: see LinkProbe; replaceable so specs need no network
27
+ def initialize(repo, github: nil, http: nil, resolver: nil)
28
+ @github = github || Github.new
29
+ @github.repository = repo
30
+ @probe = LinkProbe.new(http:, resolver:)
31
+ end
32
+
33
+ def call
34
+ result = blank_result(repo_name)
35
+ check_repo(result[:repo], result)
36
+ result[:valid] = valid?(result)
37
+ result
38
+ rescue Octokit::Error => e
39
+ raise Error, "GitHub API error: #{e.class.name.split("::").last}: #{e.message.lines.first&.strip}"
40
+ end
41
+
42
+ private
43
+
44
+ def repo_name
45
+ repo = @github.repository.to_s.delete_suffix("/").delete_suffix(".git")
46
+ raise Error, "Repository must be given as OWNER/REPO" unless %r{\A[\w.-]+/[\w.-]+\z}.match?(repo)
47
+
48
+ repo
49
+ end
50
+
51
+ def blank_result(repo)
52
+ { repo:, exists: false, private: false, archived: false, default_branch: nil,
53
+ info_files: { modinfo: nil, toolinfo: nil }, problems: [], entries: [], valid: false }
54
+ end
55
+
56
+ def check_repo(repo, result)
57
+ record_repo(@github.client.repository(repo), result)
58
+ check_info_files(result) unless result[:private] || result[:archived]
59
+ rescue Octokit::NotFound
60
+ result[:problems] << "repository not found"
61
+ end
62
+
63
+ def record_repo(data, result)
64
+ result.merge!(exists: true, private: data[:private] ? true : false, archived: data[:archived] ? true : false,
65
+ default_branch: data[:default_branch])
66
+ result[:problems] << "repository is private" if result[:private]
67
+ result[:problems] << "repository is archived" if result[:archived]
68
+ end
69
+
70
+ def check_info_files(result)
71
+ TYPES.each do |kind, meta|
72
+ file = @github.find(meta[:file])
73
+ next unless file
74
+
75
+ result[:info_files][kind] = file[:download_url]
76
+ check_info_file(meta, file[:download_url], result)
77
+ end
78
+
79
+ result[:problems] << "no modinfo.json or toolinfo.json found" if result[:info_files].values.none?
80
+ end
81
+
82
+ def check_info_file(meta, url, result)
83
+ entries = entries_from(meta, retrieve_from_url(url))
84
+ entries.each_with_index do |data, index|
85
+ if usable_name?(data)
86
+ result[:entries] << check_entry(meta, data)
87
+ else
88
+ result[:problems] << "#{meta[:file]}: entry #{index + 1} has no usable name (sync would skip it)"
89
+ end
90
+ end
91
+ rescue Sync::RequestFailed, JSON::ParserError => e
92
+ result[:problems] << "#{meta[:file]}: #{e.message}"
93
+ end
94
+
95
+ # Same rules as Sync::Entities#entries_from
96
+ def entries_from(meta, body)
97
+ entries = body[meta[:key]] if body.is_a?(Hash)
98
+ raise JSON::ParserError, "no '#{meta[:key]}' array" unless entries.is_a?(Array)
99
+ raise JSON::ParserError, "'#{meta[:key]}' entries must be objects" unless entries.all?(Hash)
100
+
101
+ entries
102
+ end
103
+
104
+ def usable_name?(data)
105
+ data[:name].is_a?(String) && /[a-z0-9]+/i.match?(data[:name])
106
+ end
107
+
108
+ def check_entry(meta, data)
109
+ info = meta[:entity].new(data)
110
+ entry = entry_for(meta[:type], info)
111
+
112
+ entry[:warnings] << "tool ships a Windows executable" if meta[:type] == "tool" && EntryLinks.windows_tool?(info)
113
+ EntryLinks.for(info).each { |field, url, download| check_link(entry, field, url, download) }
114
+ entry
115
+ end
116
+
117
+ def entry_for(type, info)
118
+ { type:, name: info.name, author: info.author, version: info.version,
119
+ errors: info.errors.dup, warnings: info.warnings.dup, links: [] }
120
+ end
121
+
122
+ def check_link(entry, field, url, download)
123
+ status, host, problem = @probe.call(url)
124
+ entry[:links] << { field:, url:, status:, final_host: host }
125
+
126
+ if problem || !LinkProbe::OK_STATUSES.include?(status)
127
+ entry[download ? :errors : :warnings] << "#{field}: #{problem || "HTTP #{status}"} (#{url})"
128
+ elsif download && !LinkProbe.github_host?(host)
129
+ entry[:warnings] << "download hosted off GitHub: #{host}"
130
+ end
131
+ end
132
+
133
+ def valid?(result)
134
+ result[:exists] && result[:info_files].values.any? && result[:problems].empty? &&
135
+ result[:entries].none? { |entry| entry[:errors].any? }
136
+ end
137
+ end
138
+ end
139
+ end
140
+ end
@@ -0,0 +1,51 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "paint"
4
+
5
+ module Icarus
6
+ module Mod
7
+ module Tools
8
+ # Human-readable output for a RepoCheck result, in the style of `imt validate modinfo`
9
+ class RepoReport
10
+ def initialize(result, verbose: 1)
11
+ @result = result
12
+ @verbose = verbose
13
+ end
14
+
15
+ def print
16
+ print_header
17
+ @result[:problems].each { |problem| puts Paint["#{problem} in #{@result[:repo]}", :red] }
18
+ @result[:entries].each { |entry| print_entry(entry) }
19
+ print_verdict if @verbose.positive?
20
+ end
21
+
22
+ private
23
+
24
+ def print_header
25
+ puts Paint["Validating #{@result[:repo]}", :cyan, :bright] if @verbose.positive?
26
+ return unless @verbose > 1
27
+
28
+ puts Paint["Default branch: #{@result[:default_branch]}", :cyan]
29
+ @result[:info_files].each { |kind, url| puts "#{kind}: #{url || "not found"}" }
30
+ end
31
+
32
+ def print_verdict
33
+ valid = @result[:valid]
34
+ puts Paint[valid ? "VALID" : "NOT VALID", valid ? :green : :red, :bright]
35
+ end
36
+
37
+ def print_links(entry, name)
38
+ puts Paint["Checked #{entry[:type]} #{name}", :cyan, :bright]
39
+ entry[:links].each { |link| puts " #{link[:field]}: #{link[:status] || "failed"} #{link[:final_host]}" }
40
+ end
41
+
42
+ def print_entry(entry)
43
+ name = "#{entry[:author].to_s.strip}/#{entry[:name].to_s.strip}"
44
+ print_links(entry, name) if @verbose > 1
45
+ entry[:errors].each { |error| puts Paint["#{error} in #{name}", :red] }
46
+ entry[:warnings].each { |warning| puts Paint["#{warning} in #{name}", :yellow] }
47
+ end
48
+ end
49
+ end
50
+ end
51
+ end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Icarus
4
4
  module Mod
5
- VERSION = "3.1.0"
5
+ VERSION = "3.2.0"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: Icarus-Mod-Tools
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.1.0
4
+ version: 3.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Donovan Young
@@ -116,7 +116,11 @@ files:
116
116
  - lib/icarus/mod/github.rb
117
117
  - lib/icarus/mod/tools.rb
118
118
  - lib/icarus/mod/tools/baseinfo.rb
119
+ - lib/icarus/mod/tools/entry_links.rb
120
+ - lib/icarus/mod/tools/link_probe.rb
119
121
  - lib/icarus/mod/tools/modinfo.rb
122
+ - lib/icarus/mod/tools/repo_check.rb
123
+ - lib/icarus/mod/tools/repo_report.rb
120
124
  - lib/icarus/mod/tools/sync/entities.rb
121
125
  - lib/icarus/mod/tools/sync/helpers.rb
122
126
  - lib/icarus/mod/tools/sync/modinfo_list.rb