Icarus-Mod-Tools 2.6.0 → 2.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: f02e414cd84c6117e8ff650a1048c16eac751dae9cfe99a52a17a3e58682d1b0
4
- data.tar.gz: f14a5bf74a4fcd74be9c71aba1debf378d45af6bc54290b35c2d20236e17d3af
3
+ metadata.gz: 0c707c742cfcc5417eeb13440063a6ea79d95a41404f4c5c69d6bc20fcdbec72
4
+ data.tar.gz: 8c0bed84ae140dd24d20d1cf7c8e02f2d2a3f6310a43bdf98c5232eb63df701c
5
5
  SHA512:
6
- metadata.gz: 636dfd397c992d7dd15d464584734f7e599417e1145f0cf139ffcf2f8eec9e409b473c4deae9abae453a42abbfe1f04138b36440669ead6db5c4386b6e33b4ca
7
- data.tar.gz: b5085429a592dfc1bd8bbf0fde991ade384afcd88aedd50c6f0969a8a786b4c3e98c740fda3053aa48c52ea734dfe74381d2d8797dde0b0e563abc467ec3e75d
6
+ metadata.gz: fc3600b85a17b9d00a9a8bdb287553f0da946a2528e827b2758c4e6d86d03823f0b4fb989ff71a2a225db0cad5bb8de0d7e0b6aead9a074ae731e45906c7b204
7
+ data.tar.gz: 27157226ad92b847a1692d6a17891a19fb8408642e3eb5120752e92f684dca40bb602bdd5b6f2a2227c68cfc966e33a50118bc9ee3f9336c8434c353b9ca0640
data/CHANGELOG.md CHANGED
@@ -4,6 +4,20 @@ All notable changes to this project will be documented in this file.
4
4
 
5
5
  ## History (reverse chronological order)
6
6
 
7
+ ### v2.7.0 - 2026-10-30
8
+
9
+ - Fix sync deleting a source's mods and tools when its `modinfo.json`/`toolinfo.json` could not be read (#16)
10
+ - Authors still control removal: if a file is removed or renamed (HTTP 404 or 410), its entries are deleted as before
11
+ - A transient failure (timeout, network/DNS/TLS error, 5xx, 403/429 rate limit, any other unexpected status) or invalid JSON no longer deletes that source's entries; deletions are blocked per source, so other authors' removals still go through, and a warning lists every kept entry and why
12
+ - Sync records which file each mod and tool came from in new internal Firestore docs, `meta/mod_sources` and `meta/tool_sources` (the mods and tools docs are unchanged). An entry with no recorded source yet, such as on the first run after upgrading, is kept while any source fails transiently
13
+ - Invalid JSON is reported with a loud warning so an author mid-edit is not wiped
14
+ - Fix a repository whose GitHub search fails transiently (rate limit, 5xx, network error) aborting `imt sync modinfo`/`toolinfo`; it is now skipped with a warning and its existing list entries are kept (#16)
15
+ - Change `imt sync cleanup` to keep the oldest duplicate, whose ID clients already store, instead of the most recently updated one (#16)
16
+ - The most recently updated duplicate's data is copied onto the kept entry before the others are deleted
17
+ - `--dry-run` shows what would be kept, copied and deleted
18
+ - Give new mods and tools deterministic document IDs derived from type, author and name, so an entry that is deleted and re-added keeps its ID (#16)
19
+ - Existing documents keep their current IDs
20
+
7
21
  ### v2.6.0 - 2026-03-04
8
22
 
9
23
  - Add `imt sync cleanup` command to remove duplicate entries from mods and tools collections
data/CLAUDE.md CHANGED
@@ -45,11 +45,13 @@ imt [command] # Run installed gem
45
45
 
46
46
  The tool implements a two-stage sync pipeline:
47
47
 
48
- 1. **GitHub → Meta Collections**: Scan repositories for metadata files and sync to `meta/modinfo`, `meta/toolinfo`, `meta/repos`
48
+ 1. **GitHub → Meta Collections**: Scan repositories for metadata files and sync to `meta/modinfo`, `meta/toolinfo`, `meta/repos` (`meta/mod_sources` and `meta/tool_sources` are internal, written by stage 2)
49
49
  2. **Meta → Entity Collections**: Process meta lists to populate/update `mods` and `tools` collections
50
50
 
51
51
  All operations support `--dry-run` for safe testing.
52
52
 
53
+ Deletion contract (#16): stage 2 deletes a doc only when no info file still lists it. `Sync::Helpers#retrieve_from_url` raises `RequestGone` for 404/410 (author removed the file: delete) and `RequestTransient` for everything else; invalid JSON also counts as transient. Docs do not record their source URL; instead `meta/mod_sources` / `meta/tool_sources` (Firestore, shared across sync hosts) map doc ID => source URL, written by `Sync::Mods/Tools#update` + `save_sources` and pruned by `Firestore#delete`. A transient failure protects only the docs mapped to that URL; unmapped docs are kept while any URL failed. Stage 1 never drops list entries (`Firestore#update` unions), and a repo whose GitHub search fails transiently is skipped. New docs get a stable ID (`Firestore#stable_id`); `imt sync cleanup` keeps the oldest duplicate and copies the newest data onto it.
54
+
53
55
  ### Core Components
54
56
 
55
57
  **CLI Layer** (`lib/icarus/mod/cli/`)
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- Icarus-Mod-Tools (2.5.3)
4
+ Icarus-Mod-Tools (2.7.0)
5
5
  google-cloud-firestore (~> 2.7)
6
6
  octokit (~> 6.0)
7
7
  paint (~> 2.3)
data/README.md CHANGED
@@ -132,6 +132,7 @@ Options:
132
132
  ```sh
133
133
  Commands:
134
134
  imt sync all # Run all sync jobs
135
+ imt sync cleanup # Remove duplicate entries from mods and tools collections
135
136
  imt sync help [COMMAND] # Describe subcommands or one specific subcommand
136
137
  imt sync modinfo # Reads from 'meta/repos/list' and Syncs any modinfo files we find (github only for now)
137
138
  imt sync mods # Reads from 'meta/modinfo/list' and updates the 'mods' database accordingly
@@ -147,6 +148,10 @@ Options:
147
148
  [--dry-run], [--no-dry-run] # Dry run (no changes will be made)
148
149
  ```
149
150
 
151
+ **Removal**: Mod and tool authors control their own entries. If a `modinfo.json` or `toolinfo.json` is removed or renamed (so it returns HTTP 404 or 410), `imt sync mods`/`tools` deletes its entries. If a file cannot be read for any other reason (timeout, network error, server error, rate limit, or invalid JSON), that file's entries are kept on that run and a warning names them; other sources are unaffected. Sync tracks which file each entry came from in the internal `meta/mod_sources` and `meta/tool_sources` documents (override with `collections.meta.mod_sources` / `tool_sources` in the config). An entry with no recorded source yet is kept while any source fails.
152
+
153
+ **IDs**: New mods and tools get a document ID derived from their type, author and name, so an entry that is deleted and later re-added gets the same ID. `imt sync cleanup` keeps the oldest of any duplicates (the ID clients already have), copies the newest duplicate's data onto it, and deletes the rest.
154
+
150
155
  **Note**: Sync operations automatically convert GitHub URLs to the correct `raw.githubusercontent.com` format for direct downloads. If you're using GitHub URLs in `modinfo.json` or `toolinfo.json` files, you can use either `/blob/` or `/raw/` formats and they will be automatically converted. Warnings will be displayed when URLs are auto-fixed so you can update your source files.
151
156
 
152
157
  ## Development
@@ -15,8 +15,10 @@
15
15
  "collections": {
16
16
  "meta": {
17
17
  "modinfo": "meta/modinfo",
18
+ "mod_sources": "meta/mod_sources",
18
19
  "repositories": "meta/repos",
19
20
  "status": "meta/status",
21
+ "tool_sources": "meta/tool_sources",
20
22
  "toolinfo": "meta/toolinfo"
21
23
  },
22
24
  "mods": "mods",
@@ -223,7 +223,8 @@ module Icarus
223
223
  track_delete(type, "#{entity.name} (#{entity.id})") { firestore.delete(type, entity) }
224
224
  end
225
225
  end
226
- rescue SocketError, IOError, SystemCallError, Timeout::Error, JSON::ParserError => e
226
+ rescue Tools::Sync::RequestFailed, SocketError, IOError, SystemCallError, Timeout::Error,
227
+ JSON::ParserError => e
227
228
  @failed_entity_fetches ||= []
228
229
  @failed_entity_fetches << { url: url, error: e.class.name, message: e.message }
229
230
  warn Paint["Warning: Could not fetch #{url} to remove entities: #{e.message}", :yellow]
@@ -267,7 +268,7 @@ module Icarus
267
268
  end
268
269
 
269
270
  entities
270
- rescue SocketError, IOError, SystemCallError, Timeout::Error, JSON::ParserError => e
271
+ rescue Tools::Sync::RequestFailed, SocketError, IOError, SystemCallError, Timeout::Error, JSON::ParserError => e
271
272
  warn Paint["Warning: Could not fetch #{url}: #{e.message}", :yellow] if verbose?
272
273
  nil
273
274
  end
@@ -80,28 +80,13 @@ module Icarus
80
80
 
81
81
  puts "Found #{duplicates.length} duplicate #{singular_type}(s) to clean up:" if verbose?
82
82
 
83
+ deleted_count = 0
83
84
  duplicates.each do |key, items|
84
85
  name, author = key
85
- # Sort by updated_at descending, keep the most recent
86
- sorted = items.sort_by { |item| item.updated_at || Time.at(0) }.reverse
87
- keeper = sorted.first
88
- to_delete = sorted[1..]
89
-
90
86
  puts " #{author}/#{name}: #{items.length} entries" if verbose?
91
- puts " Keeping: #{keeper.id} (updated: #{keeper.updated_at})" if verbose?
92
-
93
- to_delete.each do |item|
94
- if options[:dry_run]
95
- puts Paint[" Would delete: #{item.id} (updated: #{item.updated_at})", :yellow] if verbose?
96
- else
97
- puts " Deleting: #{item.id} (updated: #{item.updated_at})" if verbose?
98
- response = firestore.delete(singular_type, item)
99
- puts " #{success_or_failure(response)}" if verbose > 1
100
- end
101
- end
87
+ deleted_count += merge_duplicates(singular_type, items)
102
88
  end
103
89
 
104
- deleted_count = duplicates.values.sum { |items| items.length - 1 }
105
90
  if options[:dry_run]
106
91
  puts Paint["Dry run; no changes made. Would have deleted #{deleted_count} duplicate #{type}.", :yellow] if verbose?
107
92
  else
@@ -109,6 +94,90 @@ module Icarus
109
94
  end
110
95
  end
111
96
 
97
+ # Keeps the oldest entry (the ID clients already hold), copies the most recently updated entry's
98
+ # data onto it, then deletes the rest. Returns how many entries were (or would be) deleted.
99
+ def merge_duplicates(singular_type, items)
100
+ keeper = items.min_by { |item| oldest_first_key(item) }
101
+ newest = items.min_by { |item| newest_first_key(item) }
102
+ to_delete = items.reject { |item| item.equal?(keeper) }
103
+
104
+ puts " Keeping: #{keeper.id} (created: #{keeper.created_at}, updated: #{keeper.updated_at})" if verbose?
105
+ return 0 unless copy_onto_keeper(singular_type, keeper, newest)
106
+
107
+ to_delete.each do |item|
108
+ if options[:dry_run]
109
+ puts Paint[" Would delete: #{item.id} (updated: #{item.updated_at})", :yellow] if verbose?
110
+ else
111
+ puts " Deleting: #{item.id} (updated: #{item.updated_at})" if verbose?
112
+ response = firestore.delete(singular_type, item)
113
+ puts " #{success_or_failure(response)}" if verbose > 1
114
+ end
115
+ end
116
+
117
+ to_delete.length
118
+ end
119
+
120
+ def copy_onto_keeper(singular_type, keeper, newest)
121
+ return true if newest.equal?(keeper)
122
+
123
+ copying = "data from #{newest.id} (updated: #{newest.updated_at}) onto #{keeper.id}"
124
+ if options[:dry_run]
125
+ puts Paint[" Would copy #{copying}", :yellow] if verbose?
126
+ return true
127
+ end
128
+
129
+ puts " Copying #{copying}" if verbose?
130
+ if firestore.replace_data(singular_type, id: keeper.id, data: newest.data)
131
+ firestore.move_source(singular_type, from: newest.id, to: keeper.id)
132
+ return true
133
+ end
134
+
135
+ warn Paint[" Failed to copy #{newest.id} onto #{keeper.id} for #{keeper.author}/#{keeper.name}; " \
136
+ "nothing deleted for this entry", :red]
137
+ false
138
+ end
139
+
140
+ # Earliest created_at, then updated_at, then id: deterministic even with missing timestamps
141
+ def oldest_first_key(item)
142
+ time = item.created_at || item.updated_at
143
+ [time ? time.to_r : Float::INFINITY, item.id.to_s]
144
+ end
145
+
146
+ # Latest updated_at, then created_at; ties go to the lowest id
147
+ def newest_first_key(item)
148
+ time = item.updated_at || item.created_at
149
+ [time ? -time.to_r : Float::INFINITY, item.id.to_s]
150
+ end
151
+
152
+ # Drops from +delete_array+ every doc that may only be missing because its source could not be read
153
+ # this run (#16): a doc mapped to a transiently failed URL, or, while any URL failed, a doc with no
154
+ # recorded source yet. A source that is gone (404/410) is not a transient failure, so authors still
155
+ # control removal, and docs from every other source are deleted as normal.
156
+ def protect_transient_sources(sync, delete_array)
157
+ failed_urls = sync.transient_failures
158
+ return delete_array if failed_urls.empty?
159
+
160
+ sources = sync.sources
161
+ kept, deletable = delete_array.partition do |list|
162
+ url = sources[list.id.to_s]
163
+ url.nil? || failed_urls.include?(url)
164
+ end
165
+ warn_kept(failed_urls, kept, sources) if kept.any?
166
+
167
+ deletable
168
+ end
169
+
170
+ def warn_kept(failed_urls, kept, sources)
171
+ warn Paint["WARNING: #{failed_urls.count} source(s) could not be read this run:", :yellow, :bright]
172
+ failed_urls.each { |url| warn Paint[" #{url}", :yellow] }
173
+ warn Paint["WARNING: Kept #{kept.count} outdated item(s) instead of deleting them:", :yellow, :bright]
174
+ kept.each do |list|
175
+ url = sources[list.id.to_s]
176
+ reason = url ? "its source could not be read (#{url})" : "no recorded source yet, and a source failed"
177
+ warn Paint[" #{list.author || 'NoOne'}/#{list.name || 'Unnamed'} (#{list.id}): #{reason}", :yellow]
178
+ end
179
+ end
180
+
112
181
  def success_or_failure(status)
113
182
  format("%<status>10s", status: status ? Paint["Success", :green] : Paint["Failure", :red])
114
183
  end
@@ -124,6 +193,11 @@ module Icarus
124
193
  puts "Retrieving Info Array..." if verbose?
125
194
  info_array = sync.data(repositories, verbose: verbose > 1)&.map(&:download_url)&.compact
126
195
 
196
+ # The list update only ever adds URLs, so a repository we could not search keeps its entries (#16)
197
+ sync.transient_failures.each do |repo|
198
+ warn Paint["WARNING: Could not search #{repo} this run; its existing #{type} entries were kept", :yellow]
199
+ end
200
+
127
201
  raise Icarus::Mod::Tools::Error, "no .json files found for #{type}" unless info_array&.any?
128
202
 
129
203
  if options[:dry_run]
@@ -192,8 +266,9 @@ module Icarus
192
266
  end
193
267
 
194
268
  puts "Created/Updated #{info_array.count} Items" if verbose?
269
+ sync.save_sources
195
270
 
196
- delete_array = list_array.filter { |list| sync.find_info(list).nil? }
271
+ delete_array = protect_transient_sources(sync, list_array.filter { |list| sync.find_info(list).nil? })
197
272
 
198
273
  return unless delete_array.any?
199
274
 
@@ -1,5 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "digest"
4
+ require "set"
3
5
  require "google/cloud/firestore"
4
6
  require "tools/modinfo"
5
7
  require "tools/toolinfo"
@@ -40,10 +42,62 @@ module Icarus
40
42
  @tools ||= list(:tools)
41
43
  end
42
44
 
45
+ # Internal map of mod/tool doc ID => the modinfo/toolinfo URL it was last synced from, kept in
46
+ # meta/mod_sources and meta/tool_sources ({ sources: { "<doc id>" => "<url>" } }). Sync uses it to protect
47
+ # only the docs of a source that failed transiently (#16). Consumers never read it; mods/tools docs are unchanged.
48
+ def sources(type)
49
+ key = pluralize(type)
50
+ @sources ||= {}
51
+ @sources[key] ||= begin
52
+ data = @client.doc(sources_path(key)).get&.data
53
+ (data&.dig(:sources) || {}).transform_keys(&:to_s)
54
+ end
55
+ end
56
+
57
+ # Records (in memory) that doc +id+ came from +url+; #save_sources writes it
58
+ def record_source(type, id, url)
59
+ return if id.nil? || url.nil?
60
+
61
+ map = sources(type)
62
+ return if map[id.to_s] == url
63
+
64
+ map[id.to_s] = url
65
+ sources_changed << pluralize(type)
66
+ end
67
+
68
+ # Writes the source map if it changed since it was read or last saved
69
+ def save_sources(type)
70
+ key = pluralize(type)
71
+ return true unless sources_changed.include?(key)
72
+
73
+ response = @client.doc(sources_path(key)).set({ sources: sources(key) }, merge: false)
74
+ success = response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
75
+ sources_changed.delete(key) if success
76
+
77
+ success
78
+ end
79
+
80
+ # Gives doc +to+ the source of doc +from+ (cleanup copies +from+'s data onto +to+), and saves
81
+ def move_source(type, from:, to:)
82
+ url = sources(type)[from.to_s]
83
+ return true unless url
84
+
85
+ record_source(type, to, url)
86
+ save_sources(type)
87
+ end
88
+
43
89
  def find_by_type(type:, name:, author:)
44
90
  instance_variable_get(:"@#{type}").find { |obj| obj.name == name && obj.author == author }
45
91
  end
46
92
 
93
+ # A deterministic document id for a new :mods or :tools document, so a mod that is deleted and later
94
+ # re-added gets the same id back (#16). Author and name are case- and whitespace-normalised, and the
95
+ # collection is part of the key so a mod and a tool with the same author and name do not collide.
96
+ def stable_id(type, author:, name:)
97
+ key = [pluralize(type), author.to_s.strip.downcase, name.to_s.strip.downcase].join("\0")
98
+ Digest::SHA256.hexdigest(key)[0, 20]
99
+ end
100
+
47
101
  def update(type, payload, merge: false)
48
102
  raise "You must specify a payload to update" if payload&.empty? || payload.nil?
49
103
 
@@ -62,6 +116,19 @@ module Icarus
62
116
  response.is_a?(Google::Cloud::Firestore::DocumentReference) || response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
63
117
  end
64
118
 
119
+ # Overwrites the :mod or :tool document at +id+ with +data+ (merge: false), keeping the document's id.
120
+ # Used by `imt sync cleanup` to copy the newest duplicate's data onto the oldest one (#16).
121
+ def replace_data(type, id:, data:)
122
+ raise "Invalid type: #{type}" unless %i[mod tool].include?(type.to_sym)
123
+
124
+ collection = pluralize(type)
125
+ response = @client.doc("#{collections.send(collection)}/#{id}").set(data, merge: false)
126
+ success = response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
127
+ replace_cached(collection, id, data) if success
128
+
129
+ success
130
+ end
131
+
65
132
  def delete(type, payload)
66
133
  case type.to_sym
67
134
  when :mod, :tool
@@ -71,6 +138,7 @@ module Icarus
71
138
  cache_var = type == :mod ? :@mods : :@tools
72
139
  cached_collection = instance_variable_get(cache_var)
73
140
  cached_collection&.delete_if { |item| item.id == payload.id }
141
+ forget_source(type, payload.id)
74
142
  end
75
143
  when :modinfo, :toolinfo, :repositories
76
144
  update_array = (send(type) - [payload]).flatten.uniq
@@ -106,15 +174,47 @@ module Icarus
106
174
 
107
175
  return @client.doc("#{collections.send(type)}/#{doc_id}").set(payload.to_h, merge:) if doc_id
108
176
 
109
- doc_ref = @client.col(collections.send(type)).add(payload.to_h)
177
+ # New document: write it at its stable id (set creates it, or updates a doc a previous re-add left there).
178
+ # Only if that id is already held by a different mod (same author/name in another case) fall back to add.
179
+ cached_collection = instance_variable_get(:"@#{type}")
180
+ new_id = stable_id(type, author: payload.author, name: payload.name)
181
+ if cached_collection&.any? { |item| item.id == new_id }
182
+ response = @client.col(collections.send(type)).add(payload.to_h)
183
+ new_id = response.document_id
184
+ else
185
+ response = @client.doc("#{collections.send(type)}/#{new_id}").set(payload.to_h, merge:)
186
+ end
110
187
 
111
188
  # Update cache to prevent duplicate creation within the same sync run
112
- payload.id = doc_ref.document_id
113
- cache_var = :"@#{type}"
114
- cached_collection = instance_variable_get(cache_var)
189
+ payload.id = new_id
115
190
  cached_collection&.push(payload)
116
191
 
117
- doc_ref
192
+ response
193
+ end
194
+
195
+ def forget_source(type, id)
196
+ return unless sources(type).delete(id.to_s)
197
+
198
+ sources_changed << pluralize(type)
199
+ save_sources(type)
200
+ end
201
+
202
+ def sources_changed
203
+ @sources_changed ||= Set.new
204
+ end
205
+
206
+ def sources_path(collection)
207
+ name = :"#{collection.chomp('s')}_sources"
208
+ collections.meta&.send(name) || "meta/#{name}"
209
+ end
210
+
211
+ def replace_cached(collection, id, data)
212
+ cached_collection = instance_variable_get(:"@#{collection}")
213
+ index = cached_collection&.index { |item| item.id == id }
214
+ return unless index
215
+
216
+ old = cached_collection[index]
217
+ cached_collection[index] = old.class.new(data, id:, created: old.created_at, updated: Time.now)
118
218
  end
119
219
 
120
220
  def pluralize(type)
@@ -7,6 +7,8 @@ module Icarus
7
7
  class Baseinfo
8
8
  attr_reader :data
9
9
  attr_accessor :id, :created_at, :updated_at
10
+ # The modinfo/toolinfo URL this entry was read from during sync; never written to the entry's doc
11
+ attr_accessor :source_url
10
12
 
11
13
  HASHKEYS = %i[name author version compatibility description files imageURL readmeURL].freeze
12
14
 
@@ -9,10 +9,33 @@ module Icarus
9
9
  module Mod
10
10
  module Tools
11
11
  module Sync
12
- class RequestFailed < StandardError; end
12
+ # Raised when a source URL cannot be read. Subclasses say whether the failure is permanent
13
+ # (RequestGone: the author removed or renamed the file) or may clear up on a later run
14
+ # (RequestTransient). Sync only deletes catalog entries on a permanent failure (#16).
15
+ class RequestFailed < StandardError
16
+ def transient?
17
+ false
18
+ end
19
+ end
20
+
21
+ # The source is gone (HTTP 404 or 410): its entries may be removed from the catalog
22
+ class RequestGone < RequestFailed; end
23
+
24
+ # The source could not be read this run (timeout, network/DNS/TLS error, 5xx, 403/429 rate limits,
25
+ # or any other unexpected status): its entries must be kept until a run can read it again
26
+ class RequestTransient < RequestFailed
27
+ def transient?
28
+ true
29
+ end
30
+ end
13
31
 
14
32
  # Sync helper methods
15
33
  module Helpers
34
+ GONE_STATUS_CODES = %w[404 410].freeze
35
+
36
+ # Errors raised by Net::HTTP for a request that never got a usable response
37
+ NETWORK_ERRORS = [Timeout::Error, SocketError, SystemCallError, IOError, OpenSSL::SSL::SSLError, Net::ProtocolError].freeze
38
+
16
39
  def retrieve_from_url(url)
17
40
  raise RequestFailed, "Invalid URI: '#{url}'" unless url && url =~ URI::DEFAULT_PARSER.make_regexp
18
41
 
@@ -32,9 +55,16 @@ module Icarus
32
55
  }
33
56
  end
34
57
 
35
- res = http.get(uri.path.empty? ? "/" : uri.path)
58
+ res = begin
59
+ http.get(uri.path.empty? ? "/" : uri.path)
60
+ rescue *NETWORK_ERRORS => e
61
+ raise RequestTransient, "HTTP Request failed for #{url} (#{e.class}): #{e.message}"
62
+ end
36
63
 
37
- raise RequestFailed, "HTTP Request failed for #{url} (#{res.code}): #{res.message}" unless res&.code == "200"
64
+ unless res&.code == "200"
65
+ error_class = GONE_STATUS_CODES.include?(res&.code) ? RequestGone : RequestTransient
66
+ raise error_class, "HTTP Request failed for #{url} (#{res&.code}): #{res&.message}"
67
+ end
38
68
 
39
69
  JSON.parse(res.body, symbolize_names: true)
40
70
  end
@@ -16,8 +16,13 @@ module Icarus
16
16
  @firestore = client || Firestore.new
17
17
  @github = Github.new
18
18
  @repositories = []
19
+ @transient_failures = []
19
20
  end
20
21
 
22
+ # Repositories whose GitHub lookup failed this run for a reason that may clear up (rate limit,
23
+ # 5xx, network error). Their entries already in the list are kept: #update only ever adds (#16).
24
+ attr_reader :transient_failures
25
+
21
26
  def repositories
22
27
  @firestore.repositories
23
28
  end
@@ -31,26 +36,41 @@ module Icarus
31
36
  end
32
37
 
33
38
  def data(repositories, verbose: false)
39
+ @transient_failures = []
40
+
34
41
  repositories.map do |repo|
35
42
  print "searching #{repo}..." if verbose
36
43
 
37
44
  case repo
38
45
  when /github/
39
- @github.repository = repo
40
- modinfo_url = @github.find("modinfo.json")
41
-
42
- unless modinfo_url
43
- puts "Skipped...no modinfo.json" if verbose
44
- next
45
- end
46
-
47
- puts "Found!" if verbose
48
- modinfo_url
46
+ find_modinfo(repo, verbose:)
49
47
  else
50
48
  puts "Skipped...repository type not supported yet" if verbose
51
49
  end
52
50
  end.compact
53
51
  end
52
+
53
+ private
54
+
55
+ def find_modinfo(repo, verbose: false)
56
+ @github.repository = repo
57
+ modinfo_url = @github.find("modinfo.json")
58
+
59
+ unless modinfo_url
60
+ puts "Skipped...no modinfo.json" if verbose
61
+ return
62
+ end
63
+
64
+ puts "Found!" if verbose
65
+ modinfo_url
66
+ rescue Octokit::NotFound
67
+ puts "Skipped...repository not found" if verbose
68
+ nil
69
+ rescue Octokit::Error, Faraday::Error => e
70
+ @transient_failures << repo
71
+ warn "Skipped; could not search #{repo} (#{e.class}); its existing modinfo entries are kept"
72
+ nil
73
+ end
54
74
  end
55
75
  end
56
76
  end
@@ -1,5 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "paint"
3
4
  require "firestore"
4
5
  require "tools/sync/helpers"
5
6
 
@@ -19,18 +20,34 @@ module Icarus
19
20
  @firestore.mods
20
21
  end
21
22
 
23
+ # Every mod we could read from the modinfo URLs this run. URLs that failed transiently
24
+ # (see #transient_failures) contribute nothing here, so their mods must not be deleted.
22
25
  def info_array
23
- @info_array ||= @firestore.modinfo.map do |url|
24
- retrieve_from_url(url)[:mods].map { |mod| Icarus::Mod::Tools::Modinfo.new(mod) if /[a-z0-9]+/i.match?(mod[:name]) }
25
- rescue Icarus::Mod::Tools::Sync::RequestFailed
26
- warn "Skipped; Failed to retrieve #{url}"
26
+ return @info_array if @info_array
27
+
28
+ @transient_failures = []
29
+ @info_array = @firestore.modinfo.map do |url|
30
+ mods_from(url)
31
+ rescue Icarus::Mod::Tools::Sync::RequestFailed => e
32
+ @transient_failures << url if e.transient?
33
+ warn "Skipped; Failed to retrieve #{url}#{' (transient; its existing entries are kept)' if e.transient?}"
27
34
  next
28
35
  rescue JSON::ParserError => e
29
- warn "Skipped; Invalid JSON in #{url}: #{e.message}"
36
+ @transient_failures << url
37
+ warn Paint["WARNING: Skipped; Invalid JSON in #{url}: #{e.message}", :red, :bright]
38
+ warn Paint["WARNING: Existing entries from #{url} are kept until it parses again; " \
39
+ "remove or rename the file to remove them", :red, :bright]
30
40
  next
31
41
  end.flatten.compact.uniq { |mod| [mod.name, mod.author] }
32
42
  end
33
43
 
44
+ # The modinfo URLs that could not be read this run for a reason that may clear up
45
+ # (network error, 5xx, rate limit, invalid JSON). Gone (404/410) URLs are not included.
46
+ def transient_failures
47
+ info_array
48
+ @transient_failures || []
49
+ end
50
+
34
51
  def find(modinfo)
35
52
  @firestore.find_by_type(type: "mods", name: modinfo.name, author: modinfo.author)&.id
36
53
  end
@@ -40,12 +57,37 @@ module Icarus
40
57
  end
41
58
 
42
59
  def update(modinfo)
43
- @firestore.update(:mod, modinfo, merge: false)
60
+ success = @firestore.update(:mod, modinfo, merge: false)
61
+ @firestore.record_source(:mods, modinfo.id, modinfo.source_url) if success && modinfo.source_url
62
+
63
+ success
64
+ end
65
+
66
+ # doc ID => modinfo URL it was last synced from (see Firestore#sources)
67
+ def sources
68
+ @firestore.sources(:mods)
69
+ end
70
+
71
+ def save_sources
72
+ @firestore.save_sources(:mods)
44
73
  end
45
74
 
46
75
  def delete(modinfo)
47
76
  @firestore.delete(:mod, modinfo)
48
77
  end
78
+
79
+ private
80
+
81
+ def mods_from(url)
82
+ mods = retrieve_from_url(url)[:mods]
83
+ raise JSON::ParserError, "no 'mods' array" unless mods.is_a?(Array)
84
+
85
+ mods.filter_map do |data|
86
+ next unless /[a-z0-9]+/i.match?(data[:name])
87
+
88
+ Icarus::Mod::Tools::Modinfo.new(data).tap { |mod| mod.source_url = url }
89
+ end
90
+ end
49
91
  end
50
92
  end
51
93
  end
@@ -16,8 +16,13 @@ module Icarus
16
16
  @firestore = client || Firestore.new
17
17
  @github = Github.new
18
18
  @repositories = []
19
+ @transient_failures = []
19
20
  end
20
21
 
22
+ # Repositories whose GitHub lookup failed this run for a reason that may clear up (rate limit,
23
+ # 5xx, network error). Their entries already in the list are kept: #update only ever adds (#16).
24
+ attr_reader :transient_failures
25
+
21
26
  def repositories
22
27
  @firestore.repositories
23
28
  end
@@ -31,26 +36,41 @@ module Icarus
31
36
  end
32
37
 
33
38
  def data(repositories, verbose: false)
39
+ @transient_failures = []
40
+
34
41
  repositories.map do |repo|
35
42
  print "searching #{repo}..." if verbose
36
43
 
37
44
  case repo
38
45
  when /github/
39
- @github.repository = repo
40
- toolinfo_url = @github.find("toolinfo.json")
41
-
42
- unless toolinfo_url
43
- puts "Skipped...no toolinfo.json" if verbose
44
- next
45
- end
46
-
47
- puts "Found!" if verbose
48
- toolinfo_url
46
+ find_toolinfo(repo, verbose:)
49
47
  else
50
48
  puts "Skipped...repository type not supported yet" if verbose
51
49
  end
52
50
  end.compact
53
51
  end
52
+
53
+ private
54
+
55
+ def find_toolinfo(repo, verbose: false)
56
+ @github.repository = repo
57
+ toolinfo_url = @github.find("toolinfo.json")
58
+
59
+ unless toolinfo_url
60
+ puts "Skipped...no toolinfo.json" if verbose
61
+ return
62
+ end
63
+
64
+ puts "Found!" if verbose
65
+ toolinfo_url
66
+ rescue Octokit::NotFound
67
+ puts "Skipped...repository not found" if verbose
68
+ nil
69
+ rescue Octokit::Error, Faraday::Error => e
70
+ @transient_failures << repo
71
+ warn "Skipped; could not search #{repo} (#{e.class}); its existing toolinfo entries are kept"
72
+ nil
73
+ end
54
74
  end
55
75
  end
56
76
  end
@@ -1,5 +1,6 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require "paint"
3
4
  require "firestore"
4
5
  require "tools/sync/helpers"
5
6
 
@@ -19,20 +20,36 @@ module Icarus
19
20
  @firestore.tools
20
21
  end
21
22
 
23
+ # Every tool we could read from the toolinfo URLs this run. URLs that failed transiently
24
+ # (see #transient_failures) contribute nothing here, so their tools must not be deleted.
22
25
  def info_array
23
- @info_array ||= @firestore.toolinfo.map do |url|
26
+ return @info_array if @info_array
27
+
28
+ @transient_failures = []
29
+ @info_array = @firestore.toolinfo.map do |url|
24
30
  next unless url
25
31
 
26
- retrieve_from_url(url)[:tools].map { |tool| Icarus::Mod::Tools::Toolinfo.new(tool) if /[a-z0-9]+/i.match?(tool[:name]) }
27
- rescue Icarus::Mod::Tools::Sync::RequestFailed
28
- warn "Skipped; Failed to retrieve #{url}"
32
+ tools_from(url)
33
+ rescue Icarus::Mod::Tools::Sync::RequestFailed => e
34
+ @transient_failures << url if e.transient?
35
+ warn "Skipped; Failed to retrieve #{url}#{' (transient; its existing entries are kept)' if e.transient?}"
29
36
  next
30
37
  rescue JSON::ParserError => e
31
- warn "Skipped; Invalid JSON in #{url}: #{e.message}"
38
+ @transient_failures << url
39
+ warn Paint["WARNING: Skipped; Invalid JSON in #{url}: #{e.message}", :red, :bright]
40
+ warn Paint["WARNING: Existing entries from #{url} are kept until it parses again; " \
41
+ "remove or rename the file to remove them", :red, :bright]
32
42
  next
33
43
  end.flatten.compact.uniq { |tool| [tool.name, tool.author] }
34
44
  end
35
45
 
46
+ # The toolinfo URLs that could not be read this run for a reason that may clear up
47
+ # (network error, 5xx, rate limit, invalid JSON). Gone (404/410) URLs are not included.
48
+ def transient_failures
49
+ info_array
50
+ @transient_failures || []
51
+ end
52
+
36
53
  def find(toolinfo)
37
54
  @firestore.find_by_type(type: "tools", name: toolinfo.name, author: toolinfo.author)&.id
38
55
  end
@@ -42,12 +59,37 @@ module Icarus
42
59
  end
43
60
 
44
61
  def update(toolinfo)
45
- @firestore.update(:tool, toolinfo, merge: false)
62
+ success = @firestore.update(:tool, toolinfo, merge: false)
63
+ @firestore.record_source(:tools, toolinfo.id, toolinfo.source_url) if success && toolinfo.source_url
64
+
65
+ success
66
+ end
67
+
68
+ # doc ID => toolinfo URL it was last synced from (see Firestore#sources)
69
+ def sources
70
+ @firestore.sources(:tools)
71
+ end
72
+
73
+ def save_sources
74
+ @firestore.save_sources(:tools)
46
75
  end
47
76
 
48
77
  def delete(toolinfo)
49
78
  @firestore.delete(:tool, toolinfo)
50
79
  end
80
+
81
+ private
82
+
83
+ def tools_from(url)
84
+ tools = retrieve_from_url(url)[:tools]
85
+ raise JSON::ParserError, "no 'tools' array" unless tools.is_a?(Array)
86
+
87
+ tools.filter_map do |data|
88
+ next unless /[a-z0-9]+/i.match?(data[:name])
89
+
90
+ Icarus::Mod::Tools::Toolinfo.new(data).tap { |tool| tool.source_url = url }
91
+ end
92
+ end
51
93
  end
52
94
  end
53
95
  end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Icarus
4
4
  module Mod
5
- VERSION = "2.6.0"
5
+ VERSION = "2.7.0"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: Icarus-Mod-Tools
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.6.0
4
+ version: 2.7.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Donovan Young
@@ -132,7 +132,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
132
132
  - !ruby/object:Gem::Version
133
133
  version: '0'
134
134
  requirements: []
135
- rubygems_version: 4.0.3
135
+ rubygems_version: 3.6.9
136
136
  specification_version: 4
137
137
  summary: Various tools for Icarus Modding
138
138
  test_files: []