Icarus-Mod-Tools 2.6.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +14 -0
- data/CLAUDE.md +3 -1
- data/Gemfile.lock +1 -1
- data/README.md +5 -0
- data/imtconfig.sample.json +2 -0
- data/lib/icarus/mod/cli/remove.rb +3 -2
- data/lib/icarus/mod/cli/sync.rb +93 -18
- data/lib/icarus/mod/firestore.rb +105 -5
- data/lib/icarus/mod/tools/baseinfo.rb +2 -0
- data/lib/icarus/mod/tools/sync/helpers.rb +33 -3
- data/lib/icarus/mod/tools/sync/modinfo_list.rb +30 -10
- data/lib/icarus/mod/tools/sync/mods.rb +48 -6
- data/lib/icarus/mod/tools/sync/toolinfo_list.rb +30 -10
- data/lib/icarus/mod/tools/sync/tools.rb +48 -6
- data/lib/icarus/mod/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0c707c742cfcc5417eeb13440063a6ea79d95a41404f4c5c69d6bc20fcdbec72
|
|
4
|
+
data.tar.gz: 8c0bed84ae140dd24d20d1cf7c8e02f2d2a3f6310a43bdf98c5232eb63df701c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fc3600b85a17b9d00a9a8bdb287553f0da946a2528e827b2758c4e6d86d03823f0b4fb989ff71a2a225db0cad5bb8de0d7e0b6aead9a074ae731e45906c7b204
|
|
7
|
+
data.tar.gz: 27157226ad92b847a1692d6a17891a19fb8408642e3eb5120752e92f684dca40bb602bdd5b6f2a2227c68cfc966e33a50118bc9ee3f9336c8434c353b9ca0640
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,20 @@ All notable changes to this project will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
## History (reverse chronological order)
|
|
6
6
|
|
|
7
|
+
### v2.7.0 - 2026-10-30
|
|
8
|
+
|
|
9
|
+
- Fix sync deleting a source's mods and tools when its `modinfo.json`/`toolinfo.json` could not be read (#16)
|
|
10
|
+
- Authors still control removal: if a file is removed or renamed (HTTP 404 or 410), its entries are deleted as before
|
|
11
|
+
- A transient failure (timeout, network/DNS/TLS error, 5xx, 403/429 rate limit, any other unexpected status) or invalid JSON no longer deletes that source's entries; deletions are blocked per source, so other authors' removals still go through, and a warning lists every kept entry and why
|
|
12
|
+
- Sync records which file each mod and tool came from in new internal Firestore docs, `meta/mod_sources` and `meta/tool_sources` (the mods and tools docs are unchanged). An entry with no recorded source yet, such as on the first run after upgrading, is kept while any source fails transiently
|
|
13
|
+
- Invalid JSON is reported with a loud warning so an author mid-edit is not wiped
|
|
14
|
+
- Fix a repository whose GitHub search fails transiently (rate limit, 5xx, network error) aborting `imt sync modinfo`/`toolinfo`; it is now skipped with a warning and its existing list entries are kept (#16)
|
|
15
|
+
- Change `imt sync cleanup` to keep the oldest duplicate, whose ID clients already store, instead of the most recently updated one (#16)
|
|
16
|
+
- The most recently updated duplicate's data is copied onto the kept entry before the others are deleted
|
|
17
|
+
- `--dry-run` shows what would be kept, copied and deleted
|
|
18
|
+
- Give new mods and tools deterministic document IDs derived from type, author and name, so an entry that is deleted and re-added keeps its ID (#16)
|
|
19
|
+
- Existing documents keep their current IDs
|
|
20
|
+
|
|
7
21
|
### v2.6.0 - 2026-03-04
|
|
8
22
|
|
|
9
23
|
- Add `imt sync cleanup` command to remove duplicate entries from mods and tools collections
|
data/CLAUDE.md
CHANGED
|
@@ -45,11 +45,13 @@ imt [command] # Run installed gem
|
|
|
45
45
|
|
|
46
46
|
The tool implements a two-stage sync pipeline:
|
|
47
47
|
|
|
48
|
-
1. **GitHub → Meta Collections**: Scan repositories for metadata files and sync to `meta/modinfo`, `meta/toolinfo`, `meta/repos`
|
|
48
|
+
1. **GitHub → Meta Collections**: Scan repositories for metadata files and sync to `meta/modinfo`, `meta/toolinfo`, `meta/repos` (`meta/mod_sources` and `meta/tool_sources` are internal, written by stage 2)
|
|
49
49
|
2. **Meta → Entity Collections**: Process meta lists to populate/update `mods` and `tools` collections
|
|
50
50
|
|
|
51
51
|
All operations support `--dry-run` for safe testing.
|
|
52
52
|
|
|
53
|
+
Deletion contract (#16): stage 2 deletes a doc only when no info file still lists it. `Sync::Helpers#retrieve_from_url` raises `RequestGone` for 404/410 (author removed the file: delete) and `RequestTransient` for everything else; invalid JSON also counts as transient. Docs do not record their source URL; instead `meta/mod_sources` / `meta/tool_sources` (Firestore, shared across sync hosts) map doc ID => source URL, written by `Sync::Mods/Tools#update` + `save_sources` and pruned by `Firestore#delete`. A transient failure protects only the docs mapped to that URL; unmapped docs are kept while any URL failed. Stage 1 never drops list entries (`Firestore#update` unions), and a repo whose GitHub search fails transiently is skipped. New docs get a stable ID (`Firestore#stable_id`); `imt sync cleanup` keeps the oldest duplicate and copies the newest data onto it.
|
|
54
|
+
|
|
53
55
|
### Core Components
|
|
54
56
|
|
|
55
57
|
**CLI Layer** (`lib/icarus/mod/cli/`)
|
data/Gemfile.lock
CHANGED
data/README.md
CHANGED
|
@@ -132,6 +132,7 @@ Options:
|
|
|
132
132
|
```sh
|
|
133
133
|
Commands:
|
|
134
134
|
imt sync all # Run all sync jobs
|
|
135
|
+
imt sync cleanup # Remove duplicate entries from mods and tools collections
|
|
135
136
|
imt sync help [COMMAND] # Describe subcommands or one specific subcommand
|
|
136
137
|
imt sync modinfo # Reads from 'meta/repos/list' and Syncs any modinfo files we find (github only for now)
|
|
137
138
|
imt sync mods # Reads from 'meta/modinfo/list' and updates the 'mods' database accordingly
|
|
@@ -147,6 +148,10 @@ Options:
|
|
|
147
148
|
[--dry-run], [--no-dry-run] # Dry run (no changes will be made)
|
|
148
149
|
```
|
|
149
150
|
|
|
151
|
+
**Removal**: Mod and tool authors control their own entries. If a `modinfo.json` or `toolinfo.json` is removed or renamed (so it returns HTTP 404 or 410), `imt sync mods`/`tools` deletes its entries. If a file cannot be read for any other reason (timeout, network error, server error, rate limit, or invalid JSON), that file's entries are kept on that run and a warning names them; other sources are unaffected. Sync tracks which file each entry came from in the internal `meta/mod_sources` and `meta/tool_sources` documents (override with `collections.meta.mod_sources` / `tool_sources` in the config). An entry with no recorded source yet is kept while any source fails.
|
|
152
|
+
|
|
153
|
+
**IDs**: New mods and tools get a document ID derived from their type, author and name, so an entry that is deleted and later re-added gets the same ID. `imt sync cleanup` keeps the oldest of any duplicates (the ID clients already have), copies the newest duplicate's data onto it, and deletes the rest.
|
|
154
|
+
|
|
150
155
|
**Note**: Sync operations automatically convert GitHub URLs to the correct `raw.githubusercontent.com` format for direct downloads. If you're using GitHub URLs in `modinfo.json` or `toolinfo.json` files, you can use either `/blob/` or `/raw/` formats and they will be automatically converted. Warnings will be displayed when URLs are auto-fixed so you can update your source files.
|
|
151
156
|
|
|
152
157
|
## Development
|
data/imtconfig.sample.json
CHANGED
|
@@ -15,8 +15,10 @@
|
|
|
15
15
|
"collections": {
|
|
16
16
|
"meta": {
|
|
17
17
|
"modinfo": "meta/modinfo",
|
|
18
|
+
"mod_sources": "meta/mod_sources",
|
|
18
19
|
"repositories": "meta/repos",
|
|
19
20
|
"status": "meta/status",
|
|
21
|
+
"tool_sources": "meta/tool_sources",
|
|
20
22
|
"toolinfo": "meta/toolinfo"
|
|
21
23
|
},
|
|
22
24
|
"mods": "mods",
|
|
@@ -223,7 +223,8 @@ module Icarus
|
|
|
223
223
|
track_delete(type, "#{entity.name} (#{entity.id})") { firestore.delete(type, entity) }
|
|
224
224
|
end
|
|
225
225
|
end
|
|
226
|
-
rescue SocketError, IOError, SystemCallError, Timeout::Error,
|
|
226
|
+
rescue Tools::Sync::RequestFailed, SocketError, IOError, SystemCallError, Timeout::Error,
|
|
227
|
+
JSON::ParserError => e
|
|
227
228
|
@failed_entity_fetches ||= []
|
|
228
229
|
@failed_entity_fetches << { url: url, error: e.class.name, message: e.message }
|
|
229
230
|
warn Paint["Warning: Could not fetch #{url} to remove entities: #{e.message}", :yellow]
|
|
@@ -267,7 +268,7 @@ module Icarus
|
|
|
267
268
|
end
|
|
268
269
|
|
|
269
270
|
entities
|
|
270
|
-
rescue SocketError, IOError, SystemCallError, Timeout::Error, JSON::ParserError => e
|
|
271
|
+
rescue Tools::Sync::RequestFailed, SocketError, IOError, SystemCallError, Timeout::Error, JSON::ParserError => e
|
|
271
272
|
warn Paint["Warning: Could not fetch #{url}: #{e.message}", :yellow] if verbose?
|
|
272
273
|
nil
|
|
273
274
|
end
|
data/lib/icarus/mod/cli/sync.rb
CHANGED
|
@@ -80,28 +80,13 @@ module Icarus
|
|
|
80
80
|
|
|
81
81
|
puts "Found #{duplicates.length} duplicate #{singular_type}(s) to clean up:" if verbose?
|
|
82
82
|
|
|
83
|
+
deleted_count = 0
|
|
83
84
|
duplicates.each do |key, items|
|
|
84
85
|
name, author = key
|
|
85
|
-
# Sort by updated_at descending, keep the most recent
|
|
86
|
-
sorted = items.sort_by { |item| item.updated_at || Time.at(0) }.reverse
|
|
87
|
-
keeper = sorted.first
|
|
88
|
-
to_delete = sorted[1..]
|
|
89
|
-
|
|
90
86
|
puts " #{author}/#{name}: #{items.length} entries" if verbose?
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
to_delete.each do |item|
|
|
94
|
-
if options[:dry_run]
|
|
95
|
-
puts Paint[" Would delete: #{item.id} (updated: #{item.updated_at})", :yellow] if verbose?
|
|
96
|
-
else
|
|
97
|
-
puts " Deleting: #{item.id} (updated: #{item.updated_at})" if verbose?
|
|
98
|
-
response = firestore.delete(singular_type, item)
|
|
99
|
-
puts " #{success_or_failure(response)}" if verbose > 1
|
|
100
|
-
end
|
|
101
|
-
end
|
|
87
|
+
deleted_count += merge_duplicates(singular_type, items)
|
|
102
88
|
end
|
|
103
89
|
|
|
104
|
-
deleted_count = duplicates.values.sum { |items| items.length - 1 }
|
|
105
90
|
if options[:dry_run]
|
|
106
91
|
puts Paint["Dry run; no changes made. Would have deleted #{deleted_count} duplicate #{type}.", :yellow] if verbose?
|
|
107
92
|
else
|
|
@@ -109,6 +94,90 @@ module Icarus
|
|
|
109
94
|
end
|
|
110
95
|
end
|
|
111
96
|
|
|
97
|
+
# Keeps the oldest entry (the ID clients already hold), copies the most recently updated entry's
|
|
98
|
+
# data onto it, then deletes the rest. Returns how many entries were (or would be) deleted.
|
|
99
|
+
def merge_duplicates(singular_type, items)
|
|
100
|
+
keeper = items.min_by { |item| oldest_first_key(item) }
|
|
101
|
+
newest = items.min_by { |item| newest_first_key(item) }
|
|
102
|
+
to_delete = items.reject { |item| item.equal?(keeper) }
|
|
103
|
+
|
|
104
|
+
puts " Keeping: #{keeper.id} (created: #{keeper.created_at}, updated: #{keeper.updated_at})" if verbose?
|
|
105
|
+
return 0 unless copy_onto_keeper(singular_type, keeper, newest)
|
|
106
|
+
|
|
107
|
+
to_delete.each do |item|
|
|
108
|
+
if options[:dry_run]
|
|
109
|
+
puts Paint[" Would delete: #{item.id} (updated: #{item.updated_at})", :yellow] if verbose?
|
|
110
|
+
else
|
|
111
|
+
puts " Deleting: #{item.id} (updated: #{item.updated_at})" if verbose?
|
|
112
|
+
response = firestore.delete(singular_type, item)
|
|
113
|
+
puts " #{success_or_failure(response)}" if verbose > 1
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
to_delete.length
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def copy_onto_keeper(singular_type, keeper, newest)
|
|
121
|
+
return true if newest.equal?(keeper)
|
|
122
|
+
|
|
123
|
+
copying = "data from #{newest.id} (updated: #{newest.updated_at}) onto #{keeper.id}"
|
|
124
|
+
if options[:dry_run]
|
|
125
|
+
puts Paint[" Would copy #{copying}", :yellow] if verbose?
|
|
126
|
+
return true
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
puts " Copying #{copying}" if verbose?
|
|
130
|
+
if firestore.replace_data(singular_type, id: keeper.id, data: newest.data)
|
|
131
|
+
firestore.move_source(singular_type, from: newest.id, to: keeper.id)
|
|
132
|
+
return true
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
warn Paint[" Failed to copy #{newest.id} onto #{keeper.id} for #{keeper.author}/#{keeper.name}; " \
|
|
136
|
+
"nothing deleted for this entry", :red]
|
|
137
|
+
false
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Earliest created_at, then updated_at, then id: deterministic even with missing timestamps
|
|
141
|
+
def oldest_first_key(item)
|
|
142
|
+
time = item.created_at || item.updated_at
|
|
143
|
+
[time ? time.to_r : Float::INFINITY, item.id.to_s]
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
# Latest updated_at, then created_at; ties go to the lowest id
|
|
147
|
+
def newest_first_key(item)
|
|
148
|
+
time = item.updated_at || item.created_at
|
|
149
|
+
[time ? -time.to_r : Float::INFINITY, item.id.to_s]
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# Drops from +delete_array+ every doc that may only be missing because its source could not be read
|
|
153
|
+
# this run (#16): a doc mapped to a transiently failed URL, or, while any URL failed, a doc with no
|
|
154
|
+
# recorded source yet. A source that is gone (404/410) is not a transient failure, so authors still
|
|
155
|
+
# control removal, and docs from every other source are deleted as normal.
|
|
156
|
+
def protect_transient_sources(sync, delete_array)
|
|
157
|
+
failed_urls = sync.transient_failures
|
|
158
|
+
return delete_array if failed_urls.empty?
|
|
159
|
+
|
|
160
|
+
sources = sync.sources
|
|
161
|
+
kept, deletable = delete_array.partition do |list|
|
|
162
|
+
url = sources[list.id.to_s]
|
|
163
|
+
url.nil? || failed_urls.include?(url)
|
|
164
|
+
end
|
|
165
|
+
warn_kept(failed_urls, kept, sources) if kept.any?
|
|
166
|
+
|
|
167
|
+
deletable
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
def warn_kept(failed_urls, kept, sources)
|
|
171
|
+
warn Paint["WARNING: #{failed_urls.count} source(s) could not be read this run:", :yellow, :bright]
|
|
172
|
+
failed_urls.each { |url| warn Paint[" #{url}", :yellow] }
|
|
173
|
+
warn Paint["WARNING: Kept #{kept.count} outdated item(s) instead of deleting them:", :yellow, :bright]
|
|
174
|
+
kept.each do |list|
|
|
175
|
+
url = sources[list.id.to_s]
|
|
176
|
+
reason = url ? "its source could not be read (#{url})" : "no recorded source yet, and a source failed"
|
|
177
|
+
warn Paint[" #{list.author || 'NoOne'}/#{list.name || 'Unnamed'} (#{list.id}): #{reason}", :yellow]
|
|
178
|
+
end
|
|
179
|
+
end
|
|
180
|
+
|
|
112
181
|
def success_or_failure(status)
|
|
113
182
|
format("%<status>10s", status: status ? Paint["Success", :green] : Paint["Failure", :red])
|
|
114
183
|
end
|
|
@@ -124,6 +193,11 @@ module Icarus
|
|
|
124
193
|
puts "Retrieving Info Array..." if verbose?
|
|
125
194
|
info_array = sync.data(repositories, verbose: verbose > 1)&.map(&:download_url)&.compact
|
|
126
195
|
|
|
196
|
+
# The list update only ever adds URLs, so a repository we could not search keeps its entries (#16)
|
|
197
|
+
sync.transient_failures.each do |repo|
|
|
198
|
+
warn Paint["WARNING: Could not search #{repo} this run; its existing #{type} entries were kept", :yellow]
|
|
199
|
+
end
|
|
200
|
+
|
|
127
201
|
raise Icarus::Mod::Tools::Error, "no .json files found for #{type}" unless info_array&.any?
|
|
128
202
|
|
|
129
203
|
if options[:dry_run]
|
|
@@ -192,8 +266,9 @@ module Icarus
|
|
|
192
266
|
end
|
|
193
267
|
|
|
194
268
|
puts "Created/Updated #{info_array.count} Items" if verbose?
|
|
269
|
+
sync.save_sources
|
|
195
270
|
|
|
196
|
-
delete_array = list_array.filter { |list| sync.find_info(list).nil? }
|
|
271
|
+
delete_array = protect_transient_sources(sync, list_array.filter { |list| sync.find_info(list).nil? })
|
|
197
272
|
|
|
198
273
|
return unless delete_array.any?
|
|
199
274
|
|
data/lib/icarus/mod/firestore.rb
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "digest"
|
|
4
|
+
require "set"
|
|
3
5
|
require "google/cloud/firestore"
|
|
4
6
|
require "tools/modinfo"
|
|
5
7
|
require "tools/toolinfo"
|
|
@@ -40,10 +42,62 @@ module Icarus
|
|
|
40
42
|
@tools ||= list(:tools)
|
|
41
43
|
end
|
|
42
44
|
|
|
45
|
+
# Internal map of mod/tool doc ID => the modinfo/toolinfo URL it was last synced from, kept in
|
|
46
|
+
# meta/mod_sources and meta/tool_sources ({ sources: { "<doc id>" => "<url>" } }). Sync uses it to protect
|
|
47
|
+
# only the docs of a source that failed transiently (#16). Consumers never read it; mods/tools docs are unchanged.
|
|
48
|
+
def sources(type)
|
|
49
|
+
key = pluralize(type)
|
|
50
|
+
@sources ||= {}
|
|
51
|
+
@sources[key] ||= begin
|
|
52
|
+
data = @client.doc(sources_path(key)).get&.data
|
|
53
|
+
(data&.dig(:sources) || {}).transform_keys(&:to_s)
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Records (in memory) that doc +id+ came from +url+; #save_sources writes it
|
|
58
|
+
def record_source(type, id, url)
|
|
59
|
+
return if id.nil? || url.nil?
|
|
60
|
+
|
|
61
|
+
map = sources(type)
|
|
62
|
+
return if map[id.to_s] == url
|
|
63
|
+
|
|
64
|
+
map[id.to_s] = url
|
|
65
|
+
sources_changed << pluralize(type)
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Writes the source map if it changed since it was read or last saved
|
|
69
|
+
def save_sources(type)
|
|
70
|
+
key = pluralize(type)
|
|
71
|
+
return true unless sources_changed.include?(key)
|
|
72
|
+
|
|
73
|
+
response = @client.doc(sources_path(key)).set({ sources: sources(key) }, merge: false)
|
|
74
|
+
success = response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
|
|
75
|
+
sources_changed.delete(key) if success
|
|
76
|
+
|
|
77
|
+
success
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# Gives doc +to+ the source of doc +from+ (cleanup copies +from+'s data onto +to+), and saves
|
|
81
|
+
def move_source(type, from:, to:)
|
|
82
|
+
url = sources(type)[from.to_s]
|
|
83
|
+
return true unless url
|
|
84
|
+
|
|
85
|
+
record_source(type, to, url)
|
|
86
|
+
save_sources(type)
|
|
87
|
+
end
|
|
88
|
+
|
|
43
89
|
def find_by_type(type:, name:, author:)
|
|
44
90
|
instance_variable_get(:"@#{type}").find { |obj| obj.name == name && obj.author == author }
|
|
45
91
|
end
|
|
46
92
|
|
|
93
|
+
# A deterministic document id for a new :mods or :tools document, so a mod that is deleted and later
|
|
94
|
+
# re-added gets the same id back (#16). Author and name are case- and whitespace-normalised, and the
|
|
95
|
+
# collection is part of the key so a mod and a tool with the same author and name do not collide.
|
|
96
|
+
def stable_id(type, author:, name:)
|
|
97
|
+
key = [pluralize(type), author.to_s.strip.downcase, name.to_s.strip.downcase].join("\0")
|
|
98
|
+
Digest::SHA256.hexdigest(key)[0, 20]
|
|
99
|
+
end
|
|
100
|
+
|
|
47
101
|
def update(type, payload, merge: false)
|
|
48
102
|
raise "You must specify a payload to update" if payload&.empty? || payload.nil?
|
|
49
103
|
|
|
@@ -62,6 +116,19 @@ module Icarus
|
|
|
62
116
|
response.is_a?(Google::Cloud::Firestore::DocumentReference) || response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
|
|
63
117
|
end
|
|
64
118
|
|
|
119
|
+
# Overwrites the :mod or :tool document at +id+ with +data+ (merge: false), keeping the document's id.
|
|
120
|
+
# Used by `imt sync cleanup` to copy the newest duplicate's data onto the oldest one (#16).
|
|
121
|
+
def replace_data(type, id:, data:)
|
|
122
|
+
raise "Invalid type: #{type}" unless %i[mod tool].include?(type.to_sym)
|
|
123
|
+
|
|
124
|
+
collection = pluralize(type)
|
|
125
|
+
response = @client.doc("#{collections.send(collection)}/#{id}").set(data, merge: false)
|
|
126
|
+
success = response.is_a?(Google::Cloud::Firestore::CommitResponse::WriteResult)
|
|
127
|
+
replace_cached(collection, id, data) if success
|
|
128
|
+
|
|
129
|
+
success
|
|
130
|
+
end
|
|
131
|
+
|
|
65
132
|
def delete(type, payload)
|
|
66
133
|
case type.to_sym
|
|
67
134
|
when :mod, :tool
|
|
@@ -71,6 +138,7 @@ module Icarus
|
|
|
71
138
|
cache_var = type == :mod ? :@mods : :@tools
|
|
72
139
|
cached_collection = instance_variable_get(cache_var)
|
|
73
140
|
cached_collection&.delete_if { |item| item.id == payload.id }
|
|
141
|
+
forget_source(type, payload.id)
|
|
74
142
|
end
|
|
75
143
|
when :modinfo, :toolinfo, :repositories
|
|
76
144
|
update_array = (send(type) - [payload]).flatten.uniq
|
|
@@ -106,15 +174,47 @@ module Icarus
|
|
|
106
174
|
|
|
107
175
|
return @client.doc("#{collections.send(type)}/#{doc_id}").set(payload.to_h, merge:) if doc_id
|
|
108
176
|
|
|
109
|
-
|
|
177
|
+
# New document: write it at its stable id (set creates it, or updates a doc a previous re-add left there).
|
|
178
|
+
# Only if that id is already held by a different mod (same author/name in another case) fall back to add.
|
|
179
|
+
cached_collection = instance_variable_get(:"@#{type}")
|
|
180
|
+
new_id = stable_id(type, author: payload.author, name: payload.name)
|
|
181
|
+
if cached_collection&.any? { |item| item.id == new_id }
|
|
182
|
+
response = @client.col(collections.send(type)).add(payload.to_h)
|
|
183
|
+
new_id = response.document_id
|
|
184
|
+
else
|
|
185
|
+
response = @client.doc("#{collections.send(type)}/#{new_id}").set(payload.to_h, merge:)
|
|
186
|
+
end
|
|
110
187
|
|
|
111
188
|
# Update cache to prevent duplicate creation within the same sync run
|
|
112
|
-
payload.id =
|
|
113
|
-
cache_var = :"@#{type}"
|
|
114
|
-
cached_collection = instance_variable_get(cache_var)
|
|
189
|
+
payload.id = new_id
|
|
115
190
|
cached_collection&.push(payload)
|
|
116
191
|
|
|
117
|
-
|
|
192
|
+
response
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
def forget_source(type, id)
|
|
196
|
+
return unless sources(type).delete(id.to_s)
|
|
197
|
+
|
|
198
|
+
sources_changed << pluralize(type)
|
|
199
|
+
save_sources(type)
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def sources_changed
|
|
203
|
+
@sources_changed ||= Set.new
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
def sources_path(collection)
|
|
207
|
+
name = :"#{collection.chomp('s')}_sources"
|
|
208
|
+
collections.meta&.send(name) || "meta/#{name}"
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
def replace_cached(collection, id, data)
|
|
212
|
+
cached_collection = instance_variable_get(:"@#{collection}")
|
|
213
|
+
index = cached_collection&.index { |item| item.id == id }
|
|
214
|
+
return unless index
|
|
215
|
+
|
|
216
|
+
old = cached_collection[index]
|
|
217
|
+
cached_collection[index] = old.class.new(data, id:, created: old.created_at, updated: Time.now)
|
|
118
218
|
end
|
|
119
219
|
|
|
120
220
|
def pluralize(type)
|
|
@@ -7,6 +7,8 @@ module Icarus
|
|
|
7
7
|
class Baseinfo
|
|
8
8
|
attr_reader :data
|
|
9
9
|
attr_accessor :id, :created_at, :updated_at
|
|
10
|
+
# The modinfo/toolinfo URL this entry was read from during sync; never written to the entry's doc
|
|
11
|
+
attr_accessor :source_url
|
|
10
12
|
|
|
11
13
|
HASHKEYS = %i[name author version compatibility description files imageURL readmeURL].freeze
|
|
12
14
|
|
|
@@ -9,10 +9,33 @@ module Icarus
|
|
|
9
9
|
module Mod
|
|
10
10
|
module Tools
|
|
11
11
|
module Sync
|
|
12
|
-
|
|
12
|
+
# Raised when a source URL cannot be read. Subclasses say whether the failure is permanent
|
|
13
|
+
# (RequestGone: the author removed or renamed the file) or may clear up on a later run
|
|
14
|
+
# (RequestTransient). Sync only deletes catalog entries on a permanent failure (#16).
|
|
15
|
+
class RequestFailed < StandardError
|
|
16
|
+
def transient?
|
|
17
|
+
false
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# The source is gone (HTTP 404 or 410): its entries may be removed from the catalog
|
|
22
|
+
class RequestGone < RequestFailed; end
|
|
23
|
+
|
|
24
|
+
# The source could not be read this run (timeout, network/DNS/TLS error, 5xx, 403/429 rate limits,
|
|
25
|
+
# or any other unexpected status): its entries must be kept until a run can read it again
|
|
26
|
+
class RequestTransient < RequestFailed
|
|
27
|
+
def transient?
|
|
28
|
+
true
|
|
29
|
+
end
|
|
30
|
+
end
|
|
13
31
|
|
|
14
32
|
# Sync helper methods
|
|
15
33
|
module Helpers
|
|
34
|
+
GONE_STATUS_CODES = %w[404 410].freeze
|
|
35
|
+
|
|
36
|
+
# Errors raised by Net::HTTP for a request that never got a usable response
|
|
37
|
+
NETWORK_ERRORS = [Timeout::Error, SocketError, SystemCallError, IOError, OpenSSL::SSL::SSLError, Net::ProtocolError].freeze
|
|
38
|
+
|
|
16
39
|
def retrieve_from_url(url)
|
|
17
40
|
raise RequestFailed, "Invalid URI: '#{url}'" unless url && url =~ URI::DEFAULT_PARSER.make_regexp
|
|
18
41
|
|
|
@@ -32,9 +55,16 @@ module Icarus
|
|
|
32
55
|
}
|
|
33
56
|
end
|
|
34
57
|
|
|
35
|
-
res =
|
|
58
|
+
res = begin
|
|
59
|
+
http.get(uri.path.empty? ? "/" : uri.path)
|
|
60
|
+
rescue *NETWORK_ERRORS => e
|
|
61
|
+
raise RequestTransient, "HTTP Request failed for #{url} (#{e.class}): #{e.message}"
|
|
62
|
+
end
|
|
36
63
|
|
|
37
|
-
|
|
64
|
+
unless res&.code == "200"
|
|
65
|
+
error_class = GONE_STATUS_CODES.include?(res&.code) ? RequestGone : RequestTransient
|
|
66
|
+
raise error_class, "HTTP Request failed for #{url} (#{res&.code}): #{res&.message}"
|
|
67
|
+
end
|
|
38
68
|
|
|
39
69
|
JSON.parse(res.body, symbolize_names: true)
|
|
40
70
|
end
|
|
@@ -16,8 +16,13 @@ module Icarus
|
|
|
16
16
|
@firestore = client || Firestore.new
|
|
17
17
|
@github = Github.new
|
|
18
18
|
@repositories = []
|
|
19
|
+
@transient_failures = []
|
|
19
20
|
end
|
|
20
21
|
|
|
22
|
+
# Repositories whose GitHub lookup failed this run for a reason that may clear up (rate limit,
|
|
23
|
+
# 5xx, network error). Their entries already in the list are kept: #update only ever adds (#16).
|
|
24
|
+
attr_reader :transient_failures
|
|
25
|
+
|
|
21
26
|
def repositories
|
|
22
27
|
@firestore.repositories
|
|
23
28
|
end
|
|
@@ -31,26 +36,41 @@ module Icarus
|
|
|
31
36
|
end
|
|
32
37
|
|
|
33
38
|
def data(repositories, verbose: false)
|
|
39
|
+
@transient_failures = []
|
|
40
|
+
|
|
34
41
|
repositories.map do |repo|
|
|
35
42
|
print "searching #{repo}..." if verbose
|
|
36
43
|
|
|
37
44
|
case repo
|
|
38
45
|
when /github/
|
|
39
|
-
|
|
40
|
-
modinfo_url = @github.find("modinfo.json")
|
|
41
|
-
|
|
42
|
-
unless modinfo_url
|
|
43
|
-
puts "Skipped...no modinfo.json" if verbose
|
|
44
|
-
next
|
|
45
|
-
end
|
|
46
|
-
|
|
47
|
-
puts "Found!" if verbose
|
|
48
|
-
modinfo_url
|
|
46
|
+
find_modinfo(repo, verbose:)
|
|
49
47
|
else
|
|
50
48
|
puts "Skipped...repository type not supported yet" if verbose
|
|
51
49
|
end
|
|
52
50
|
end.compact
|
|
53
51
|
end
|
|
52
|
+
|
|
53
|
+
private
|
|
54
|
+
|
|
55
|
+
def find_modinfo(repo, verbose: false)
|
|
56
|
+
@github.repository = repo
|
|
57
|
+
modinfo_url = @github.find("modinfo.json")
|
|
58
|
+
|
|
59
|
+
unless modinfo_url
|
|
60
|
+
puts "Skipped...no modinfo.json" if verbose
|
|
61
|
+
return
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
puts "Found!" if verbose
|
|
65
|
+
modinfo_url
|
|
66
|
+
rescue Octokit::NotFound
|
|
67
|
+
puts "Skipped...repository not found" if verbose
|
|
68
|
+
nil
|
|
69
|
+
rescue Octokit::Error, Faraday::Error => e
|
|
70
|
+
@transient_failures << repo
|
|
71
|
+
warn "Skipped; could not search #{repo} (#{e.class}); its existing modinfo entries are kept"
|
|
72
|
+
nil
|
|
73
|
+
end
|
|
54
74
|
end
|
|
55
75
|
end
|
|
56
76
|
end
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "paint"
|
|
3
4
|
require "firestore"
|
|
4
5
|
require "tools/sync/helpers"
|
|
5
6
|
|
|
@@ -19,18 +20,34 @@ module Icarus
|
|
|
19
20
|
@firestore.mods
|
|
20
21
|
end
|
|
21
22
|
|
|
23
|
+
# Every mod we could read from the modinfo URLs this run. URLs that failed transiently
|
|
24
|
+
# (see #transient_failures) contribute nothing here, so their mods must not be deleted.
|
|
22
25
|
def info_array
|
|
23
|
-
@info_array
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
26
|
+
return @info_array if @info_array
|
|
27
|
+
|
|
28
|
+
@transient_failures = []
|
|
29
|
+
@info_array = @firestore.modinfo.map do |url|
|
|
30
|
+
mods_from(url)
|
|
31
|
+
rescue Icarus::Mod::Tools::Sync::RequestFailed => e
|
|
32
|
+
@transient_failures << url if e.transient?
|
|
33
|
+
warn "Skipped; Failed to retrieve #{url}#{' (transient; its existing entries are kept)' if e.transient?}"
|
|
27
34
|
next
|
|
28
35
|
rescue JSON::ParserError => e
|
|
29
|
-
|
|
36
|
+
@transient_failures << url
|
|
37
|
+
warn Paint["WARNING: Skipped; Invalid JSON in #{url}: #{e.message}", :red, :bright]
|
|
38
|
+
warn Paint["WARNING: Existing entries from #{url} are kept until it parses again; " \
|
|
39
|
+
"remove or rename the file to remove them", :red, :bright]
|
|
30
40
|
next
|
|
31
41
|
end.flatten.compact.uniq { |mod| [mod.name, mod.author] }
|
|
32
42
|
end
|
|
33
43
|
|
|
44
|
+
# The modinfo URLs that could not be read this run for a reason that may clear up
|
|
45
|
+
# (network error, 5xx, rate limit, invalid JSON). Gone (404/410) URLs are not included.
|
|
46
|
+
def transient_failures
|
|
47
|
+
info_array
|
|
48
|
+
@transient_failures || []
|
|
49
|
+
end
|
|
50
|
+
|
|
34
51
|
def find(modinfo)
|
|
35
52
|
@firestore.find_by_type(type: "mods", name: modinfo.name, author: modinfo.author)&.id
|
|
36
53
|
end
|
|
@@ -40,12 +57,37 @@ module Icarus
|
|
|
40
57
|
end
|
|
41
58
|
|
|
42
59
|
def update(modinfo)
|
|
43
|
-
@firestore.update(:mod, modinfo, merge: false)
|
|
60
|
+
success = @firestore.update(:mod, modinfo, merge: false)
|
|
61
|
+
@firestore.record_source(:mods, modinfo.id, modinfo.source_url) if success && modinfo.source_url
|
|
62
|
+
|
|
63
|
+
success
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# doc ID => modinfo URL it was last synced from (see Firestore#sources)
|
|
67
|
+
def sources
|
|
68
|
+
@firestore.sources(:mods)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
def save_sources
|
|
72
|
+
@firestore.save_sources(:mods)
|
|
44
73
|
end
|
|
45
74
|
|
|
46
75
|
def delete(modinfo)
|
|
47
76
|
@firestore.delete(:mod, modinfo)
|
|
48
77
|
end
|
|
78
|
+
|
|
79
|
+
private
|
|
80
|
+
|
|
81
|
+
def mods_from(url)
|
|
82
|
+
mods = retrieve_from_url(url)[:mods]
|
|
83
|
+
raise JSON::ParserError, "no 'mods' array" unless mods.is_a?(Array)
|
|
84
|
+
|
|
85
|
+
mods.filter_map do |data|
|
|
86
|
+
next unless /[a-z0-9]+/i.match?(data[:name])
|
|
87
|
+
|
|
88
|
+
Icarus::Mod::Tools::Modinfo.new(data).tap { |mod| mod.source_url = url }
|
|
89
|
+
end
|
|
90
|
+
end
|
|
49
91
|
end
|
|
50
92
|
end
|
|
51
93
|
end
|
|
@@ -16,8 +16,13 @@ module Icarus
|
|
|
16
16
|
@firestore = client || Firestore.new
|
|
17
17
|
@github = Github.new
|
|
18
18
|
@repositories = []
|
|
19
|
+
@transient_failures = []
|
|
19
20
|
end
|
|
20
21
|
|
|
22
|
+
# Repositories whose GitHub lookup failed this run for a reason that may clear up (rate limit,
|
|
23
|
+
# 5xx, network error). Their entries already in the list are kept: #update only ever adds (#16).
|
|
24
|
+
attr_reader :transient_failures
|
|
25
|
+
|
|
21
26
|
def repositories
|
|
22
27
|
@firestore.repositories
|
|
23
28
|
end
|
|
@@ -31,26 +36,41 @@ module Icarus
|
|
|
31
36
|
end
|
|
32
37
|
|
|
33
38
|
def data(repositories, verbose: false)
|
|
39
|
+
@transient_failures = []
|
|
40
|
+
|
|
34
41
|
repositories.map do |repo|
|
|
35
42
|
print "searching #{repo}..." if verbose
|
|
36
43
|
|
|
37
44
|
case repo
|
|
38
45
|
when /github/
|
|
39
|
-
|
|
40
|
-
toolinfo_url = @github.find("toolinfo.json")
|
|
41
|
-
|
|
42
|
-
unless toolinfo_url
|
|
43
|
-
puts "Skipped...no toolinfo.json" if verbose
|
|
44
|
-
next
|
|
45
|
-
end
|
|
46
|
-
|
|
47
|
-
puts "Found!" if verbose
|
|
48
|
-
toolinfo_url
|
|
46
|
+
find_toolinfo(repo, verbose:)
|
|
49
47
|
else
|
|
50
48
|
puts "Skipped...repository type not supported yet" if verbose
|
|
51
49
|
end
|
|
52
50
|
end.compact
|
|
53
51
|
end
|
|
52
|
+
|
|
53
|
+
private
|
|
54
|
+
|
|
55
|
+
def find_toolinfo(repo, verbose: false)
|
|
56
|
+
@github.repository = repo
|
|
57
|
+
toolinfo_url = @github.find("toolinfo.json")
|
|
58
|
+
|
|
59
|
+
unless toolinfo_url
|
|
60
|
+
puts "Skipped...no toolinfo.json" if verbose
|
|
61
|
+
return
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
puts "Found!" if verbose
|
|
65
|
+
toolinfo_url
|
|
66
|
+
rescue Octokit::NotFound
|
|
67
|
+
puts "Skipped...repository not found" if verbose
|
|
68
|
+
nil
|
|
69
|
+
rescue Octokit::Error, Faraday::Error => e
|
|
70
|
+
@transient_failures << repo
|
|
71
|
+
warn "Skipped; could not search #{repo} (#{e.class}); its existing toolinfo entries are kept"
|
|
72
|
+
nil
|
|
73
|
+
end
|
|
54
74
|
end
|
|
55
75
|
end
|
|
56
76
|
end
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "paint"
|
|
3
4
|
require "firestore"
|
|
4
5
|
require "tools/sync/helpers"
|
|
5
6
|
|
|
@@ -19,20 +20,36 @@ module Icarus
|
|
|
19
20
|
@firestore.tools
|
|
20
21
|
end
|
|
21
22
|
|
|
23
|
+
# Every tool we could read from the toolinfo URLs this run. URLs that failed transiently
|
|
24
|
+
# (see #transient_failures) contribute nothing here, so their tools must not be deleted.
|
|
22
25
|
def info_array
|
|
23
|
-
@info_array
|
|
26
|
+
return @info_array if @info_array
|
|
27
|
+
|
|
28
|
+
@transient_failures = []
|
|
29
|
+
@info_array = @firestore.toolinfo.map do |url|
|
|
24
30
|
next unless url
|
|
25
31
|
|
|
26
|
-
|
|
27
|
-
rescue Icarus::Mod::Tools::Sync::RequestFailed
|
|
28
|
-
|
|
32
|
+
tools_from(url)
|
|
33
|
+
rescue Icarus::Mod::Tools::Sync::RequestFailed => e
|
|
34
|
+
@transient_failures << url if e.transient?
|
|
35
|
+
warn "Skipped; Failed to retrieve #{url}#{' (transient; its existing entries are kept)' if e.transient?}"
|
|
29
36
|
next
|
|
30
37
|
rescue JSON::ParserError => e
|
|
31
|
-
|
|
38
|
+
@transient_failures << url
|
|
39
|
+
warn Paint["WARNING: Skipped; Invalid JSON in #{url}: #{e.message}", :red, :bright]
|
|
40
|
+
warn Paint["WARNING: Existing entries from #{url} are kept until it parses again; " \
|
|
41
|
+
"remove or rename the file to remove them", :red, :bright]
|
|
32
42
|
next
|
|
33
43
|
end.flatten.compact.uniq { |tool| [tool.name, tool.author] }
|
|
34
44
|
end
|
|
35
45
|
|
|
46
|
+
# The toolinfo URLs that could not be read this run for a reason that may clear up
|
|
47
|
+
# (network error, 5xx, rate limit, invalid JSON). Gone (404/410) URLs are not included.
|
|
48
|
+
def transient_failures
|
|
49
|
+
info_array
|
|
50
|
+
@transient_failures || []
|
|
51
|
+
end
|
|
52
|
+
|
|
36
53
|
def find(toolinfo)
|
|
37
54
|
@firestore.find_by_type(type: "tools", name: toolinfo.name, author: toolinfo.author)&.id
|
|
38
55
|
end
|
|
@@ -42,12 +59,37 @@ module Icarus
|
|
|
42
59
|
end
|
|
43
60
|
|
|
44
61
|
def update(toolinfo)
|
|
45
|
-
@firestore.update(:tool, toolinfo, merge: false)
|
|
62
|
+
success = @firestore.update(:tool, toolinfo, merge: false)
|
|
63
|
+
@firestore.record_source(:tools, toolinfo.id, toolinfo.source_url) if success && toolinfo.source_url
|
|
64
|
+
|
|
65
|
+
success
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# doc ID => toolinfo URL it was last synced from (see Firestore#sources)
|
|
69
|
+
def sources
|
|
70
|
+
@firestore.sources(:tools)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def save_sources
|
|
74
|
+
@firestore.save_sources(:tools)
|
|
46
75
|
end
|
|
47
76
|
|
|
48
77
|
def delete(toolinfo)
|
|
49
78
|
@firestore.delete(:tool, toolinfo)
|
|
50
79
|
end
|
|
80
|
+
|
|
81
|
+
private
|
|
82
|
+
|
|
83
|
+
def tools_from(url)
|
|
84
|
+
tools = retrieve_from_url(url)[:tools]
|
|
85
|
+
raise JSON::ParserError, "no 'tools' array" unless tools.is_a?(Array)
|
|
86
|
+
|
|
87
|
+
tools.filter_map do |data|
|
|
88
|
+
next unless /[a-z0-9]+/i.match?(data[:name])
|
|
89
|
+
|
|
90
|
+
Icarus::Mod::Tools::Toolinfo.new(data).tap { |tool| tool.source_url = url }
|
|
91
|
+
end
|
|
92
|
+
end
|
|
51
93
|
end
|
|
52
94
|
end
|
|
53
95
|
end
|
data/lib/icarus/mod/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: Icarus-Mod-Tools
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.
|
|
4
|
+
version: 2.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Donovan Young
|
|
@@ -132,7 +132,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
132
132
|
- !ruby/object:Gem::Version
|
|
133
133
|
version: '0'
|
|
134
134
|
requirements: []
|
|
135
|
-
rubygems_version:
|
|
135
|
+
rubygems_version: 3.6.9
|
|
136
136
|
specification_version: 4
|
|
137
137
|
summary: Various tools for Icarus Modding
|
|
138
138
|
test_files: []
|