net-connector 0.18.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/README.md +19 -0
- data/docs/architecture.md +9 -1
- data/lib/net/connector/credentials.rb +39 -0
- data/lib/net/connector/engine/error/metadata.rb +2 -1
- data/lib/net/connector/engine/known_hosts.rb +19 -6
- data/lib/net/connector/engine/transports.rb +1 -1
- data/lib/net/connector/netdisco/catalog.rb +53 -0
- data/lib/net/connector/netdisco/fleet.rb +5 -28
- data/lib/net/connector/netdisco/settings.rb +1 -7
- data/lib/net/connector/netdisco/ssh/archive/client.rb +105 -75
- data/lib/net/connector/netdisco/ssh/archive/protocol.rb +107 -0
- data/lib/net/connector/netdisco/ssh/archive/worker.py +35 -19
- data/lib/net/connector/storage/atomic_file.rb +60 -3
- data/lib/net/connector/version.rb +1 -1
- data/lib/net/connector.rb +1 -0
- metadata +11 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: fe3f1f3bc71cf02e0007ec892699ea03fb13f30b697f5e318ca256159e77a466
|
|
4
|
+
data.tar.gz: 7f2ed60b91541c0eb362b1dc1c063661f5f22eac07719f1ed9d56f552a7d175b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 03ea333dc6fdc10956b58478de7b67d8e3061345b3d96af8601a847dd393b158cd0bdbd7e4ca3ebc69c7cdca03d58ced76e34964b0370d9b2b0dad27c4d1f9dc
|
|
7
|
+
data.tar.gz: 38784dee207a4b5161add61183ebb49cf7b4a7a843171a29da2effe393105cd9c4bd00f2a0d3780ab296c430149a43792c85b9054375cb1c97269e89b1ad7f96
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
## Unreleased
|
|
4
4
|
|
|
5
|
+
## 0.19.0 - 2026-10-29
|
|
6
|
+
|
|
7
|
+
- Extract `Netdisco::Catalog#devices(policy:)` for immutable inventory without credentials or operations; Fleet delegates inventory access. Add live, injected `Credentials#resolve(platform:)` with platform-specific username/password/enable-password precedence and independent secret snapshots.
|
|
8
|
+
|
|
9
|
+
## 0.18.1 - 2026-10-24
|
|
10
|
+
|
|
11
|
+
- Stream SSH archive bodies through bounded binary frames and incremental hashes instead of whole-file base64 copies; preserve deadlines, verified server receipts and committed local paths.
|
|
12
|
+
- Retain cleanup ownership across cancellation, failed closes and forked KnownHosts snapshots; reject inherited trust commits before file or lock access.
|
|
13
|
+
- Preserve the primary exception or nonlocal exit when atomic-file and archive-process cleanup also fails.
|
|
14
|
+
- Expose JSON Pointer field provenance through configuration previews and leave runtime dependency versions to host lockfiles.
|
|
15
|
+
|
|
5
16
|
## 0.18.0 - 2026-10-23
|
|
6
17
|
|
|
7
18
|
- Add on_receipt completion callbacks to command, script, save and custom operations, retaining completed responses after later logging, processing, budget or cancellation failures.
|
data/README.md
CHANGED
|
@@ -66,3 +66,22 @@ Licensed under [MIT](LICENSE).
|
|
|
66
66
|
## ctx_cli 公共入口
|
|
67
67
|
|
|
68
68
|
宿主配置文件读取、私有文件初始化、公共参数与终端交互由 ctx_cli 维护。原 `Net::Connector::Config`、配置路径、优先级和 backup/TFTP/topology 策略继续保留。源码联调使用 `CTX_CLI_PATH=../ctx_cli`;隔离安装可通过 `CTX_CLI_GEM=/absolute/path/ctx_cli-0.1.0.gem` 传入精确候选。
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
## Inventory and credentials without device operations
|
|
72
|
+
|
|
73
|
+
```ruby
|
|
74
|
+
require "net/connector/netdisco/catalog"
|
|
75
|
+
require "net/connector/credentials"
|
|
76
|
+
|
|
77
|
+
settings = Net::Connector::Netdisco::Settings.new(env: ENV)
|
|
78
|
+
policy = settings.snapshot(mode: :inventory)
|
|
79
|
+
catalog = Net::Connector::Netdisco::Catalog.new(settings: settings)
|
|
80
|
+
devices = catalog.devices(policy: policy)
|
|
81
|
+
# Inspect device.issue before selecting targets; duplicate rows are retained.
|
|
82
|
+
credentials = Net::Connector::Credentials.new(env: ENV).resolve(platform: :cisco_nxos)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Catalog owns inventory reading, normalization, duplicate marking and cache status (`#status`). It does not resolve device credentials, log into devices or create operation reports. Pass one policy snapshot throughout a run. Fleet delegates to Catalog and retains operation orchestration.
|
|
86
|
+
|
|
87
|
+
Credentials reads the injected environment on every `resolve`; it returns an independent frozen hash, or nil when no username is available. Platform-specific `NC_<PLATFORM>_USERNAME`, `_PASSWORD`, and `_ENABLE_PASSWORD` override `NC_DEVICE_USERNAME`, `NC_DEVICE_PASSWORD`, and `NC_ENABLE_PASSWORD` respectively. Blank values fall back. Secrets are never cached or included in object inspection.
|
data/docs/architecture.md
CHANGED
|
@@ -88,12 +88,20 @@ Fleet 在规范目标校验、凭据解析及连接器构造之前加同一把
|
|
|
88
88
|
|
|
89
89
|
`KnownHosts` 在跨进程锁内重读和合并已认证的会话密钥,条目索引保留原顺序及已有重复行,新增行只登记一次。提交复用 `AtomicFile`;目录同步失败保留已替换文件并报告 `file_persistence_unconfirmed`,不能触发再次登记。不安全的非普通共享信任文件报告 `known_hosts_unsafe`,strict 模式仍由 OpenSSH 管理。写入器仅在实际提交时加载,纯会话引擎加载不引入存储层。
|
|
90
90
|
|
|
91
|
+
信任快照及其 `.old` 文件由创建进程拥有。fork 副本关闭时只释放继承描述符;提交继承快照(即使没有新增密钥)在读取和加锁前报告 `known_hosts_inherited`。子进程需要重新打开 SSH 连接,取得自己的快照后再认证和登记;父进程的快照与连接继续有效。快照关闭失败时,KnownHosts 和 Ssh 保留清理责任,后续 close/open 重试,清理成功后才释放引用。
|
|
92
|
+
|
|
91
93
|
提交后的 `BackupPersistenceError` 只携带原 `Backup` 元数据和受控写入回执,Fleet 保留产物并使用已有 saved_with_error 分类。只接受库定义的具体完成错误及匹配产物类型,第三方异常的 backup 属性或自定义 WriteError 子类不作为完成证据。报告保留已提交位置,导出错误说明 committed;设备命令不会自动重放。
|
|
92
94
|
|
|
93
95
|
直接备份在路径锁退出前保存完成元数据,锁关闭的普通故障转换为 `BackupFinalizeError`,其中 `backup` 保留结果,`underlying_type` 经过类型白名单。该错误只确认备份操作已完成,不制造文件同步回执;未变文件的历史持久性仍未补验。Fleet 的外层路径锁及 TFTP 归档锁同样保存最新结果,退出故障转为 saved_with_error 或 reported_with_error,已有设备诊断优先,中断继续传播。
|
|
94
96
|
|
|
95
97
|
TFTP 归档分别保留本地批次副本与服务器归档副本的提交位置;写入器的原生回执必须与本次目标路径一致。目录同步或收尾失败仍保留已提交路径,成功上传转为 `reported_with_error`,诊断记录归档失败及持久化阶段;设备任务此前已有错误时保留原诊断。报告失败后的 JSON 补写同样保留已提交位置,不覆盖首次报告故障,也不重新执行设备任务。
|
|
96
98
|
|
|
99
|
+
SSH 归档的 Client 持有事务进程和管道,Protocol 只借用 IO,维护 16 KiB 控制帧、64 KiB 正文片段和阶段截止时间。`archived` 元数据、声明长度的二进制正文与 `delivered` 确认共用 280 秒期限;文件上限仍为 64 MiB,并发槽位仍为 4。Python 工作进程分片读取、复制和摘要核验,复制及交付前后均检查文件身份、大小与目录/锁边界,不保留整份正文或 base64 副本。
|
|
100
|
+
|
|
101
|
+
`AtomicFile.write_stream` 借出私有临时文件,只有正文长度、SHA-256 和交付确认全部通过后,才允许同步和原子替换。本地准备或写盘失败时,Client 尽可能继续有界读取并核验服务器副本,保留 `server_verified`,随后发送 abort,保持 staging;失败写入和设备上传均不重试。原子替换之后的同步故障保留可信回执中的 `local_path`。启动器只返回管道与等待线程,Client 是唯一清理者,取消和非局部退出不会被普通关闭错误覆盖。
|
|
102
|
+
|
|
103
|
+
AtomicFile 同样独占临时文件及父目录句柄的清理。取得句柄和登记临时目录项身份共用取消保护边界;清理仍尝试关闭及移除属于本次写入的临时目录项。生产者或目录同步期间的异常、取消及非局部退出优先于普通关闭错误;正常完成后的清理错误继续按原写入回执报告。
|
|
104
|
+
|
|
97
105
|
TFTP 只确认设备报告的上传结果:策略去掉命令、应答和提示符回显后,检查明确的完成行;文件名或表示“即将上传”的进度文字不算成功。原始输出或终端渲染文本中的失败证据优先于成功文字。
|
|
98
106
|
|
|
99
107
|
失败证据核对与完成核对共用精确回显过滤。静态应答从已完成步骤的 `Command#interactions` 读取,`Interaction#static_response` 对动态回调返回 nil,核对不会重新执行挑战回调。失败核对逐步读取原文、回车历史和终端画面,不累积全部画面副本;颜色样式只影响显示,不能拆开诊断词或借光标擦除隐藏已经收到的失败。命令诊断及拓扑保存确认同样保留这类失败历史。
|
|
@@ -208,7 +216,7 @@ Fleet 可把回执与本地普通文件的时间、稳定性、摘要及目录
|
|
|
208
216
|
|
|
209
217
|
传输关闭先撤下活跃信道,使连接不可用;尚未回收的信道由内部 `@closing_channel` 保留。`hard_close` 正常返回且当前进程拥有的子进程已退出后才释放引用;异常、非局部退出及有界等待后仍存活的子进程都保留清理责任,后者明确报告 IOError。后续 `close` 重试原信道,完成后不重复清理。fork 副本仅关闭继承句柄,不要求回收父进程拥有的子进程。
|
|
210
218
|
|
|
211
|
-
重新打开会先完成旧资源回收,失败时不创建新进程或 SSH 信任快照。私有 `close_resources` 供关闭与打开共用;公开 `close` 停止日志交付,而打开时保留新连接已配置的日志目标。SSH
|
|
219
|
+
重新打开会先完成旧资源回收,失败时不创建新进程或 SSH 信任快照。私有 `close_resources` 供关闭与打开共用;公开 `close` 停止日志交付,而打开时保留新连接已配置的日志目标。SSH 临时信任快照在成功关闭后释放引用,失败时保留供重试。连接恢复只有在旧传输及日志清理成功后才能替换传输对象;失败仍报告原连接错误,保留旧所有者供后续关闭,不重放设备命令。注入信道工厂须遵守现有 Expect 的 `hard_close` / `alive?` 生命周期协议。
|
|
212
220
|
|
|
213
221
|
PTY 输出上限直接读取信道公开的 `buffer_bytesize` 标量,避免正常分片反复取得累计正文副本;只有实际超限时才获取错误正文,关闭日志也不会关闭上限检查。传输与 SSH 归档认证统一使用当前 `Session`、`transcript`、`spawn(raw: true)` 和 `expect` 契约,不探测或回退到旧接口。运行要求为 expect-pty 0.9.4+、Ruby 3.4+。
|
|
214
222
|
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Net
|
|
4
|
+
module Connector
|
|
5
|
+
# 设备凭据的字段优先级;借用来源,每次解析独立快照,不缓存秘密。
|
|
6
|
+
class Credentials
|
|
7
|
+
def initialize(env: ENV)
|
|
8
|
+
raise ArgumentError, "Credential source must support key lookup" unless env.respond_to?(:[])
|
|
9
|
+
@env = env
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
# 平台字段优先于通用字段;缺少用户名时由宿主决定跳过或拒绝。
|
|
13
|
+
def resolve(platform:)
|
|
14
|
+
unless (platform.is_a?(String) || platform.is_a?(Symbol)) && /\A[a-z][a-z0-9_]*\z/i.match?(platform.to_s)
|
|
15
|
+
raise ArgumentError, "Credential platform must be a name"
|
|
16
|
+
end
|
|
17
|
+
prefix = "NC_#{platform.to_s.upcase}_"
|
|
18
|
+
username = value(prefix + "USERNAME") || value("NC_DEVICE_USERNAME")
|
|
19
|
+
return nil unless username
|
|
20
|
+
|
|
21
|
+
{ username: username, password: value(prefix + "PASSWORD") || value("NC_DEVICE_PASSWORD"),
|
|
22
|
+
enable_password: value(prefix + "ENABLE_PASSWORD") || value("NC_ENABLE_PASSWORD") }.freeze
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def inspect = "#<#{self.class}>"
|
|
26
|
+
alias to_s inspect
|
|
27
|
+
def pretty_print(printer) = printer.text(inspect)
|
|
28
|
+
|
|
29
|
+
private
|
|
30
|
+
|
|
31
|
+
# 拒绝非法值而不回显内容,也不冻结调用方的字符串。
|
|
32
|
+
def value(key)
|
|
33
|
+
result = @env[key]
|
|
34
|
+
raise ArgumentError, "#{key} must be a String" unless result.nil? || result.is_a?(String)
|
|
35
|
+
result.dup.freeze unless result.nil? || result.strip.empty?
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
@@ -25,7 +25,8 @@ module Net
|
|
|
25
25
|
Net::Connector::Netdisco::Tftp::Archive::Unavailable Net::Connector::Netdisco::Tftp::Archive::Failure]).freeze
|
|
26
26
|
CODES = (CONNECTOR_TYPES.map { |name| name.gsub(/([a-z])([A-Z])/, '\1_\2').downcase.to_sym } +
|
|
27
27
|
%i[authentication_failed connection_failed connection_refused no_route connection_reset
|
|
28
|
-
connection_timeout host_key_changed host_key_untrusted known_hosts_busy known_hosts_unsafe
|
|
28
|
+
connection_timeout host_key_changed host_key_untrusted known_hosts_busy known_hosts_unsafe known_hosts_inherited
|
|
29
|
+
rsa_too_small unsupported_cipher
|
|
29
30
|
ambiguous_description ambiguous_neighbor confirmation_required description_stages_unsupported
|
|
30
31
|
description_unconfirmed description_unsupported incomplete_configuration interface_missing
|
|
31
32
|
neighbor_discovery_unsupported parse_failed invalid_output_encoding persistence_unconfirmed stale_plan startup_config_missing
|
|
@@ -14,6 +14,7 @@ module Net
|
|
|
14
14
|
|
|
15
15
|
# 建立本次连接的私有主机密钥快照,初始化失败时清理已取得的资源。
|
|
16
16
|
def initialize(configuration, replace: false)
|
|
17
|
+
@owner_pid = Process.pid
|
|
17
18
|
destination = configuration.known_hosts || File.join(Dir.home, ".ssh", "known_hosts")
|
|
18
19
|
FileUtils.mkdir_p(File.dirname(destination), mode: 0o700)
|
|
19
20
|
@destination = File.join(File.realpath(File.dirname(destination)), File.basename(destination))
|
|
@@ -35,6 +36,11 @@ module Net
|
|
|
35
36
|
# 在目标文件锁内合并新增信任;检测并发密钥变化后原子写回。
|
|
36
37
|
# 会话只写私有快照;登记时重读共享信任并检查冲突,避免并发连接覆盖已有主机密钥。
|
|
37
38
|
def commit
|
|
39
|
+
unless @owner_pid == Process.pid
|
|
40
|
+
raise ConnectionError.new("inherited known_hosts snapshot cannot be committed; reopen the SSH connection",
|
|
41
|
+
code: :known_hosts_inherited, host: @host, phase: :login), cause: nil
|
|
42
|
+
end
|
|
43
|
+
|
|
38
44
|
# 纯会话引擎加载不触及文件业务;实际登记信任时才加载持久化协议。
|
|
39
45
|
require_relative "../storage/atomic_file"
|
|
40
46
|
begin
|
|
@@ -60,10 +66,10 @@ module Net
|
|
|
60
66
|
end
|
|
61
67
|
end
|
|
62
68
|
|
|
63
|
-
#
|
|
69
|
+
# 关闭私有密钥快照,创建进程同时删除文件;延迟取消直至清理完成。
|
|
64
70
|
def close
|
|
65
71
|
failure = nil
|
|
66
|
-
#
|
|
72
|
+
# 关闭句柄和删除快照期间延后异步取消,完成清理后再释放引用。
|
|
67
73
|
Thread.handle_interrupt(Exception => :never) do
|
|
68
74
|
close_snapshot
|
|
69
75
|
rescue Exception => error # rubocop:disable Lint/RescueException -- 保存同步清理故障,退出屏蔽范围后继续传播。
|
|
@@ -104,17 +110,24 @@ module Net
|
|
|
104
110
|
raise failure, cause: nil if failure
|
|
105
111
|
end
|
|
106
112
|
|
|
107
|
-
#
|
|
113
|
+
# 创建进程负责删除文件;fork 副本只关闭继承句柄,失败时保留清理引用。
|
|
108
114
|
def close_snapshot
|
|
109
115
|
file = @file
|
|
110
|
-
|
|
111
|
-
File.unlink(@path + ".old") if @path && File.exist?(@path + ".old")
|
|
116
|
+
owned = @owner_pid == Process.pid
|
|
117
|
+
File.unlink(@path + ".old") if owned && @path && File.exist?(@path + ".old")
|
|
112
118
|
rescue Exception # rubocop:disable Lint/RescueException -- 删除附属文件失败或中断后仍须关闭私有快照。
|
|
113
119
|
failed = true
|
|
114
120
|
raise
|
|
115
121
|
ensure
|
|
116
122
|
begin
|
|
117
|
-
|
|
123
|
+
if owned
|
|
124
|
+
file&.close!
|
|
125
|
+
# Tempfile 会吞掉删除时的权限错误;路径仍存在表示删除责任尚未完成。
|
|
126
|
+
raise IOError, "known_hosts snapshot removal is incomplete" if file&.path
|
|
127
|
+
else
|
|
128
|
+
file&.close
|
|
129
|
+
end
|
|
130
|
+
@file = nil
|
|
118
131
|
rescue Exception # rubocop:disable Lint/RescueException -- 私有快照清理不能覆盖先发生的删除故障。
|
|
119
132
|
raise unless failed
|
|
120
133
|
end
|
|
@@ -234,7 +234,6 @@ module Net
|
|
|
234
234
|
# 一次取消须等待信道和信任快照均完成回收;同步故障仍保留首个。
|
|
235
235
|
Thread.handle_interrupt(Exception => :never) do
|
|
236
236
|
known_hosts = @known_hosts
|
|
237
|
-
@known_hosts = nil
|
|
238
237
|
begin
|
|
239
238
|
super
|
|
240
239
|
rescue Exception => error # rubocop:disable Lint/RescueException -- 全部资源均须尝试关闭,保留首个失败或中断。
|
|
@@ -243,6 +242,7 @@ module Net
|
|
|
243
242
|
ensure
|
|
244
243
|
begin
|
|
245
244
|
known_hosts&.close
|
|
245
|
+
@known_hosts = nil
|
|
246
246
|
rescue Exception => error # rubocop:disable Lint/RescueException -- 信任文件收尾不能覆盖传输关闭的原始异常。
|
|
247
247
|
unless failure
|
|
248
248
|
failure = error
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "settings"
|
|
4
|
+
require_relative "device"
|
|
5
|
+
|
|
6
|
+
module Net
|
|
7
|
+
module Connector
|
|
8
|
+
module Netdisco
|
|
9
|
+
# 只读取、归一化及标记清单;凭据、连接、操作和报告由宿主拥有。
|
|
10
|
+
class Catalog
|
|
11
|
+
attr_reader :status
|
|
12
|
+
|
|
13
|
+
def initialize(settings: Settings.new, inventory: nil, rules: nil)
|
|
14
|
+
@settings, @inventory, @rules = settings, inventory, rules
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
# 使用调用方固定的同批策略,保留被过滤、无效及重复的记录。
|
|
18
|
+
def devices(policy:, force_refresh: false)
|
|
19
|
+
@status = nil
|
|
20
|
+
raise ArgumentError, "force_refresh must be true or false" unless [true, false].include?(force_refresh)
|
|
21
|
+
|
|
22
|
+
inventory = @inventory || @settings.inventory(policy: policy)
|
|
23
|
+
cache_mode = policy.inventory_cache_mode
|
|
24
|
+
raise ArgumentError, "cache-only cannot force refresh" if cache_mode == :only && force_refresh
|
|
25
|
+
if cache_mode == :only && !inventory.is_a?(Inventory::Cache)
|
|
26
|
+
raise Inventory::Error.new("validated inventory cache is unavailable", code: :cache_unavailable), cause: nil
|
|
27
|
+
end
|
|
28
|
+
rows = if inventory.is_a?(Inventory::Cache)
|
|
29
|
+
inventory.devices(force_refresh: force_refresh, cache_mode: cache_mode)
|
|
30
|
+
else
|
|
31
|
+
inventory.devices
|
|
32
|
+
end
|
|
33
|
+
@status = if inventory.is_a?(Inventory::Cache)
|
|
34
|
+
{ source: inventory.status, fetched_at: inventory.fetched_at,
|
|
35
|
+
refresh_error_code: inventory.refresh_error_code, write_error_type: inventory.write_error_type }.freeze
|
|
36
|
+
else
|
|
37
|
+
{ source: :uncached }.freeze
|
|
38
|
+
end
|
|
39
|
+
raise Inventory::Error, "Netdisco returned an invalid device inventory" unless rows.is_a?(Array)
|
|
40
|
+
|
|
41
|
+
rules = @rules || policy.rules
|
|
42
|
+
devices = rows.map { |row| Device.from_row(row, rules: rules) }
|
|
43
|
+
host_counts = devices.each_with_object(Hash.new(0)) { |device, counts| counts[device.host] += 1 if device.host }
|
|
44
|
+
devices.map { |device| host_counts[device.host] > 1 ? device.with(issue: :duplicate_host) : device }.freeze
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def inspect = "#<#{self.class}>"
|
|
48
|
+
alias to_s inspect
|
|
49
|
+
def pretty_print(printer) = printer.text(inspect)
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
@@ -4,6 +4,7 @@ require "fileutils"
|
|
|
4
4
|
require_relative "../storage"
|
|
5
5
|
require_relative "../device/scope"
|
|
6
6
|
require_relative "inventory"
|
|
7
|
+
require_relative "catalog"
|
|
7
8
|
require_relative "settings"
|
|
8
9
|
require_relative "device"
|
|
9
10
|
require_relative "device/access"
|
|
@@ -35,8 +36,7 @@ module Net
|
|
|
35
36
|
def initialize(inventory: nil, settings: Settings.new, rules: nil, credentials: nil, connector_factory: nil,
|
|
36
37
|
report_store: Report::Store::Json.new)
|
|
37
38
|
@settings = settings
|
|
38
|
-
@
|
|
39
|
-
@rules = rules
|
|
39
|
+
@catalog = Catalog.new(settings: settings, inventory: inventory, rules: rules)
|
|
40
40
|
@credentials = credentials
|
|
41
41
|
@connector_factory = connector_factory || ->(device, access_options) { device.build_connector(**access_options) }
|
|
42
42
|
@report_store = report_store
|
|
@@ -176,32 +176,9 @@ module Net
|
|
|
176
176
|
|
|
177
177
|
# 策略在清单请求之前固定;注入的清单来源仍由调用方负责其自身生命周期。
|
|
178
178
|
def inventory(settings_snapshot, force_refresh: false)
|
|
179
|
-
@
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
inventory = @inventory || @settings.inventory(policy: settings_snapshot)
|
|
183
|
-
cache_mode = settings_snapshot.inventory_cache_mode
|
|
184
|
-
raise ArgumentError, "cache-only cannot force refresh" if cache_mode == :only && force_refresh
|
|
185
|
-
if cache_mode == :only && !inventory.is_a?(Inventory::Cache)
|
|
186
|
-
raise Inventory::Error.new("validated inventory cache is unavailable", code: :cache_unavailable), cause: nil
|
|
187
|
-
end
|
|
188
|
-
rows = if inventory.is_a?(Inventory::Cache)
|
|
189
|
-
inventory.devices(force_refresh: force_refresh, cache_mode: cache_mode)
|
|
190
|
-
else
|
|
191
|
-
inventory.devices
|
|
192
|
-
end
|
|
193
|
-
@inventory_status = if inventory.is_a?(Inventory::Cache)
|
|
194
|
-
{ source: inventory.status, fetched_at: inventory.fetched_at,
|
|
195
|
-
refresh_error_code: inventory.refresh_error_code, write_error_type: inventory.write_error_type }.freeze
|
|
196
|
-
else
|
|
197
|
-
{ source: :uncached }.freeze
|
|
198
|
-
end
|
|
199
|
-
raise Inventory::Error, "Netdisco returned an invalid device inventory" unless rows.is_a?(Array)
|
|
200
|
-
|
|
201
|
-
rules = @rules || settings_snapshot.rules
|
|
202
|
-
devices = rows.map { |row| Device.from_row(row, rules: rules) }
|
|
203
|
-
host_counts = devices.each_with_object(Hash.new(0)) { |device, counts| counts[device.host] += 1 if device.host }
|
|
204
|
-
devices.map { |device| host_counts[device.host] > 1 ? device.with(issue: :duplicate_host) : device }.freeze
|
|
179
|
+
@catalog.devices(policy: settings_snapshot, force_refresh: force_refresh)
|
|
180
|
+
ensure
|
|
181
|
+
@inventory_status = @catalog.status
|
|
205
182
|
end
|
|
206
183
|
|
|
207
184
|
# 在固定清单快照上生成模式对应的计划,规划阶段不读取设备凭据。
|
|
@@ -346,13 +346,7 @@ module Net
|
|
|
346
346
|
|
|
347
347
|
# 每台任务读取最新厂商凭据,缺失字段回退到通用凭据;无用户名时跳过设备。
|
|
348
348
|
def device_credentials_for(device)
|
|
349
|
-
|
|
350
|
-
username = secret_value(prefix + "USERNAME") || secret_value("NC_DEVICE_USERNAME")
|
|
351
|
-
return nil unless username
|
|
352
|
-
|
|
353
|
-
{ username: username,
|
|
354
|
-
password: secret_value(prefix + "PASSWORD") || secret_value("NC_DEVICE_PASSWORD"),
|
|
355
|
-
enable_password: secret_value(prefix + "ENABLE_PASSWORD") || secret_value("NC_ENABLE_PASSWORD") }
|
|
349
|
+
Net::Connector::Credentials.new(env: @env).resolve(platform: device.vendor)
|
|
356
350
|
end
|
|
357
351
|
|
|
358
352
|
# 会话日志目录属于公开策略;调用方和批次入口负责实际创建。
|
|
@@ -3,7 +3,6 @@
|
|
|
3
3
|
require_relative "../../../storage/directory"
|
|
4
4
|
|
|
5
5
|
require "digest"
|
|
6
|
-
require "json"
|
|
7
6
|
require "open3"
|
|
8
7
|
require "securerandom"
|
|
9
8
|
require "shellwords"
|
|
@@ -11,6 +10,7 @@ require "tmpdir"
|
|
|
11
10
|
require_relative "../../tftp/root"
|
|
12
11
|
require_relative "../authentication"
|
|
13
12
|
require_relative "../../tftp/archive"
|
|
13
|
+
require_relative "protocol"
|
|
14
14
|
|
|
15
15
|
module Net
|
|
16
16
|
module Connector
|
|
@@ -41,12 +41,11 @@ module Net
|
|
|
41
41
|
end
|
|
42
42
|
end
|
|
43
43
|
|
|
44
|
-
MAX_FRAME_BYTES = 90 * 1024 * 1024
|
|
45
44
|
SESSION_LIMIT = 4
|
|
46
45
|
WORKER = File.expand_path("worker.py", __dir__)
|
|
47
46
|
ACTIVE_CLIENTS = {}
|
|
48
47
|
ACTIVE_CLIENTS_MUTEX = Mutex.new
|
|
49
|
-
private_constant :
|
|
48
|
+
private_constant :WORKER, :ACTIVE_CLIENTS, :ACTIVE_CLIENTS_MUTEX
|
|
50
49
|
|
|
51
50
|
# 退出处理只持有仍需清理的连接;已关闭的批次与凭据可由 GC 回收。
|
|
52
51
|
at_exit do
|
|
@@ -94,30 +93,26 @@ module Net
|
|
|
94
93
|
acquired = @slots.pop
|
|
95
94
|
filename = TftpTarget.validate_path!(upload_filename(device))
|
|
96
95
|
archive_name = "#{device.backup_filename(style: :hostname_ip).delete_suffix(".txt")}#{File.extname(filename)}"
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
write_frame(input, action: "abort")
|
|
116
|
-
end
|
|
117
|
-
outcome
|
|
118
|
-
ensure
|
|
119
|
-
stop_process(input, output, error, process)
|
|
96
|
+
with_process do |protocol|
|
|
97
|
+
protocol.send_frame(root: @root, filename: filename, archive_name: archive_name, batch: @batch)
|
|
98
|
+
protocol.expect_phase("ready")
|
|
99
|
+
outcome = yield filename
|
|
100
|
+
receipt = outcome.backup
|
|
101
|
+
unless receipt.is_a?(TftpReceipt) && receipt.path == filename
|
|
102
|
+
protocol.send_frame(action: "abort")
|
|
103
|
+
next outcome
|
|
104
|
+
end
|
|
105
|
+
outcome = outcome.with(started_at: started_at)
|
|
106
|
+
protocol.send_frame(action: "verify")
|
|
107
|
+
proof = protocol.expect_phase("archived")
|
|
108
|
+
outcome = retain_upload(outcome, protocol, proof, archive_name)
|
|
109
|
+
if outcome.success?
|
|
110
|
+
protocol.send_frame(action: "cleanup", sha256: outcome.backup.server_sha256)
|
|
111
|
+
protocol.expect_phase("complete")
|
|
112
|
+
else
|
|
113
|
+
protocol.send_frame(action: "abort")
|
|
120
114
|
end
|
|
115
|
+
outcome
|
|
121
116
|
end
|
|
122
117
|
rescue StandardError
|
|
123
118
|
raise Tftp::Archive::Failure.new(host: device.host), cause: nil unless outcome
|
|
@@ -158,33 +153,26 @@ module Net
|
|
|
158
153
|
|
|
159
154
|
# 用实际事务进程预检目录与写入边界,不读取或上传任何设备文件。
|
|
160
155
|
def prepare_archive
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
expect_phase(output, "ready")
|
|
165
|
-
ensure
|
|
166
|
-
stop_process(input, output, error, process)
|
|
167
|
-
end
|
|
156
|
+
with_process do |protocol|
|
|
157
|
+
protocol.send_frame(root: @root, filename: "preflight.cfg", archive_name: "preflight.cfg", batch: @batch, action: "preflight")
|
|
158
|
+
protocol.expect_phase("ready")
|
|
168
159
|
end
|
|
169
160
|
rescue StandardError
|
|
170
161
|
raise PreparationUnavailable, cause: nil
|
|
171
162
|
end
|
|
172
163
|
|
|
173
164
|
# 先核对服务器正文与摘要,再保留远端归档路径;后续本地写入失败仍保留已有证据。
|
|
174
|
-
def retain_upload(outcome, proof, archive_name)
|
|
175
|
-
|
|
176
|
-
digest = Digest::SHA256.hexdigest(bytes)
|
|
177
|
-
unless bytes.bytesize.positive? && bytes.bytesize == proof.fetch("bytes") && digest == proof.fetch("sha256")
|
|
178
|
-
raise Tftp::Archive::Failure
|
|
179
|
-
end
|
|
180
|
-
receipt = outcome.backup.with(verification: :server_verified, server_sha256: digest, server_bytes: bytes.bytesize,
|
|
165
|
+
def retain_upload(outcome, protocol, proof, archive_name)
|
|
166
|
+
receipt = outcome.backup.with(verification: :server_verified, server_sha256: proof.fetch("sha256"), server_bytes: proof.fetch("bytes"),
|
|
181
167
|
archive_path: remote_path(proof.fetch("archive_path")))
|
|
182
|
-
outcome = outcome.with(backup: receipt)
|
|
183
168
|
path = File.join(@report_directory, "tftp", archive_name)
|
|
184
|
-
|
|
185
|
-
|
|
169
|
+
persist_upload(path) do |file|
|
|
170
|
+
failure = receive_upload(protocol, receipt, file)
|
|
171
|
+
outcome = outcome.with(backup: receipt)
|
|
172
|
+
raise failure if failure
|
|
173
|
+
end
|
|
186
174
|
outcome = outcome.with(backup: receipt.with(local_path: path))
|
|
187
|
-
raise Tftp::Archive::Failure unless Storage::FileReader.fingerprint(path).sha256 ==
|
|
175
|
+
raise Tftp::Archive::Failure unless Storage::FileReader.fingerprint(path).sha256 == receipt.server_sha256
|
|
188
176
|
|
|
189
177
|
outcome
|
|
190
178
|
rescue StandardError => error
|
|
@@ -194,6 +182,38 @@ module Net
|
|
|
194
182
|
outcome.success? ? archive_failure(outcome) : outcome
|
|
195
183
|
end
|
|
196
184
|
|
|
185
|
+
# 本地准备失败仍接收并核验服务器正文;已经开始接收的失败事务不能重读或重写。
|
|
186
|
+
def persist_upload(path)
|
|
187
|
+
received = false
|
|
188
|
+
Storage::Directory.prepare(File.dirname(path))
|
|
189
|
+
Storage::AtomicFile.write_stream(path) do |file|
|
|
190
|
+
received = true
|
|
191
|
+
yield file
|
|
192
|
+
end
|
|
193
|
+
rescue StandardError
|
|
194
|
+
yield nil unless received
|
|
195
|
+
raise
|
|
196
|
+
end
|
|
197
|
+
|
|
198
|
+
# 逐块核对摘要,首次写盘错误后只继续读取;完整交付确认后才返回写盘故障。
|
|
199
|
+
def receive_upload(protocol, receipt, file)
|
|
200
|
+
digest = Digest::SHA256.new
|
|
201
|
+
failure = nil
|
|
202
|
+
protocol.read_payload(receipt.server_bytes) do |chunk|
|
|
203
|
+
digest << chunk
|
|
204
|
+
next unless file && !failure
|
|
205
|
+
|
|
206
|
+
begin
|
|
207
|
+
raise IOError, "incomplete archive write" unless file.write(chunk) == chunk.bytesize
|
|
208
|
+
rescue StandardError => error
|
|
209
|
+
failure = error
|
|
210
|
+
end
|
|
211
|
+
end
|
|
212
|
+
raise Tftp::Archive::Failure unless digest.hexdigest == receipt.server_sha256
|
|
213
|
+
|
|
214
|
+
failure
|
|
215
|
+
end
|
|
216
|
+
|
|
197
217
|
# 保留设备和服务器回执,将归档收尾异常转换为可诊断的部分成功。
|
|
198
218
|
def archive_failure(outcome)
|
|
199
219
|
failure = Tftp::Archive::Failure.new(host: outcome.device.host)
|
|
@@ -251,11 +271,10 @@ module Net
|
|
|
251
271
|
false
|
|
252
272
|
end
|
|
253
273
|
|
|
254
|
-
#
|
|
255
|
-
def open_ssh
|
|
256
|
-
prepare_master
|
|
274
|
+
# 返回事务管道和等待线程,由调用作用域唯一负责关闭、终止及回收。
|
|
275
|
+
def open_ssh
|
|
257
276
|
command = "python3 -u -c #{Shellwords.escape(File.read(WORKER))}"
|
|
258
|
-
Open3.popen3(*ssh_arguments, @uri.host, command
|
|
277
|
+
Open3.popen3(*ssh_arguments, @uri.host, command)
|
|
259
278
|
end
|
|
260
279
|
|
|
261
280
|
# 将已校验端点组装为独立 SSH 参数,固定严格主机密钥检查与连接超时。
|
|
@@ -265,32 +284,24 @@ module Net
|
|
|
265
284
|
"-p", (@uri.port || 22).to_s, "-l", @uri.user]
|
|
266
285
|
end
|
|
267
286
|
|
|
268
|
-
#
|
|
269
|
-
def
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
frame << chunk
|
|
287
|
-
raise Tftp::Archive::Failure if frame.bytesize > MAX_FRAME_BYTES
|
|
288
|
-
next unless frame.end_with?("\n")
|
|
289
|
-
|
|
290
|
-
value = JSON.parse(frame)
|
|
291
|
-
raise Tftp::Archive::Failure unless value["ok"] == true && value["phase"] == phase
|
|
292
|
-
|
|
293
|
-
return value
|
|
287
|
+
# 每次事务独占进程与协议状态,主故障或非局部退出不能被普通清理错误覆盖。
|
|
288
|
+
def with_process
|
|
289
|
+
handles = nil
|
|
290
|
+
prepare_master unless @custom_launcher
|
|
291
|
+
completed = false
|
|
292
|
+
# 取得资源与登记所有权之间暂缓取消;事务本身仍可立即中断。
|
|
293
|
+
Thread.handle_interrupt(Exception => :never) { handles = @launcher.call }
|
|
294
|
+
result = yield Protocol.new(handles[0], handles[1])
|
|
295
|
+
completed = true
|
|
296
|
+
result
|
|
297
|
+
rescue Exception # rubocop:disable Lint/RescueException -- 返回前递送的取消仍属于事务主控制流。
|
|
298
|
+
completed = false
|
|
299
|
+
raise
|
|
300
|
+
ensure
|
|
301
|
+
begin
|
|
302
|
+
stop_process(*handles) if handles
|
|
303
|
+
rescue Exception # rubocop:disable Lint/RescueException -- 取消和非局部退出均保留原始控制流。
|
|
304
|
+
raise if completed
|
|
294
305
|
end
|
|
295
306
|
end
|
|
296
307
|
|
|
@@ -303,7 +314,26 @@ module Net
|
|
|
303
314
|
|
|
304
315
|
# 先关闭事务管道并等待子进程退出,超时后分级终止,促使服务器释放锁。
|
|
305
316
|
def stop_process(input, output, error, process)
|
|
306
|
-
|
|
317
|
+
Thread.handle_interrupt(Exception => :never) do
|
|
318
|
+
failure = nil
|
|
319
|
+
[input, output, error].each do |io|
|
|
320
|
+
begin
|
|
321
|
+
io.close unless io.closed?
|
|
322
|
+
rescue Exception => caught # rubocop:disable Lint/RescueException -- 单个关闭失败仍须回收其余管道和进程。
|
|
323
|
+
failure ||= caught
|
|
324
|
+
end
|
|
325
|
+
end
|
|
326
|
+
begin
|
|
327
|
+
reap_process(process)
|
|
328
|
+
rescue Exception => caught # rubocop:disable Lint/RescueException -- 保留先发生的清理故障。
|
|
329
|
+
failure ||= caught
|
|
330
|
+
end
|
|
331
|
+
raise failure if failure
|
|
332
|
+
end
|
|
333
|
+
end
|
|
334
|
+
|
|
335
|
+
# 为正常退出保留短暂宽限,随后终止仍未退出的事务进程并回收等待线程。
|
|
336
|
+
def reap_process(process)
|
|
307
337
|
return if process.join(2)
|
|
308
338
|
|
|
309
339
|
Process.kill("TERM", process.pid)
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require_relative "../../tftp/archive"
|
|
5
|
+
|
|
6
|
+
module Net
|
|
7
|
+
module Connector
|
|
8
|
+
module Netdisco
|
|
9
|
+
module SSH
|
|
10
|
+
module Archive
|
|
11
|
+
# 仅借用事务管道,维护控制帧与二进制正文的分界、缓冲和绝对期限。
|
|
12
|
+
class Protocol
|
|
13
|
+
READ_BYTES = 65_536
|
|
14
|
+
CONTROL_BYTES = 16_384
|
|
15
|
+
PAYLOAD_BYTES = 64 * 1024 * 1024
|
|
16
|
+
private_constant :READ_BYTES, :CONTROL_BYTES, :PAYLOAD_BYTES
|
|
17
|
+
|
|
18
|
+
# 每个事务独占自己的协议状态;管道关闭和进程回收仍由客户端负责。
|
|
19
|
+
def initialize(input, output, timeout: 280)
|
|
20
|
+
@input, @output = input, output
|
|
21
|
+
@timeout = timeout
|
|
22
|
+
@buffer = +"".b
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# 发送有界单行控制帧,写入不完整不能当作命令已经交付。
|
|
26
|
+
def send_frame(**values)
|
|
27
|
+
frame = JSON.generate(values) + "\n"
|
|
28
|
+
raise Tftp::Archive::Failure if frame.bytesize > CONTROL_BYTES || @input.write(frame) != frame.bytesize
|
|
29
|
+
|
|
30
|
+
@input.flush
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
# 开始一个控制阶段;归档元数据取得的期限同时覆盖后续正文及交付确认。
|
|
34
|
+
def expect_phase(phase)
|
|
35
|
+
@deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @timeout
|
|
36
|
+
read_phase(phase)
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# 按声明长度分片交付二进制正文,随后在同一期限内确认服务器完成交付。
|
|
40
|
+
def read_payload(length)
|
|
41
|
+
unless @phase == "archived" && length.is_a?(Integer) && (1..PAYLOAD_BYTES).cover?(length)
|
|
42
|
+
raise Tftp::Archive::Failure
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
remaining = length
|
|
46
|
+
while remaining.positive?
|
|
47
|
+
remaining_time
|
|
48
|
+
@buffer << read_chunk if @buffer.empty?
|
|
49
|
+
chunk = @buffer.slice!(0, [remaining, READ_BYTES, @buffer.bytesize].min)
|
|
50
|
+
remaining -= chunk.bytesize
|
|
51
|
+
yield chunk
|
|
52
|
+
end
|
|
53
|
+
read_phase("delivered")
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
private
|
|
57
|
+
|
|
58
|
+
# 缓冲中可能同时含有控制帧和正文,只消费当前行并限制控制帧长度。
|
|
59
|
+
def read_phase(phase)
|
|
60
|
+
loop do
|
|
61
|
+
remaining_time
|
|
62
|
+
ending = @buffer.index("\n")
|
|
63
|
+
if ending
|
|
64
|
+
raise Tftp::Archive::Failure if ending + 1 > CONTROL_BYTES
|
|
65
|
+
|
|
66
|
+
value = JSON.parse(@buffer.slice!(0, ending + 1))
|
|
67
|
+
unless value.is_a?(Hash) && value["ok"] == true && value["phase"] == phase
|
|
68
|
+
raise Tftp::Archive::Failure
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
@phase = phase
|
|
72
|
+
return value
|
|
73
|
+
end
|
|
74
|
+
raise Tftp::Archive::Failure if @buffer.bytesize >= CONTROL_BYTES
|
|
75
|
+
|
|
76
|
+
@buffer << read_chunk
|
|
77
|
+
end
|
|
78
|
+
rescue JSON::ParserError
|
|
79
|
+
raise Tftp::Archive::Failure, cause: nil
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# 每次非阻塞读取使用剩余总预算,片段到达不能重新开始计时。
|
|
83
|
+
def read_chunk
|
|
84
|
+
loop do
|
|
85
|
+
raise Tftp::Archive::Failure unless @output.wait_readable(remaining_time)
|
|
86
|
+
|
|
87
|
+
chunk = @output.read_nonblock(READ_BYTES, exception: false)
|
|
88
|
+
next if chunk == :wait_readable
|
|
89
|
+
raise Tftp::Archive::Failure if chunk.nil?
|
|
90
|
+
|
|
91
|
+
return chunk
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# 即使数据已经缓冲,也检查绝对期限,避免消费者或确认阶段偷偷延长预算。
|
|
96
|
+
def remaining_time
|
|
97
|
+
remaining = @deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
98
|
+
raise Tftp::Archive::Failure unless remaining.positive?
|
|
99
|
+
|
|
100
|
+
remaining
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
end
|
|
107
|
+
end
|
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
# 服务器端单次归档事务:锁覆盖上传与核验,客户端断开后退出且不重放设备命令。
|
|
2
2
|
"""在一个目标锁内处理单次上传事务,客户端断开后退出且不重放设备操作。"""
|
|
3
3
|
|
|
4
|
-
import base64
|
|
5
4
|
from contextlib import ExitStack
|
|
6
5
|
import fcntl
|
|
7
6
|
import hashlib
|
|
@@ -14,6 +13,7 @@ import sys
|
|
|
14
13
|
import time
|
|
15
14
|
|
|
16
15
|
MAX_BYTES = 64 * 1024 * 1024
|
|
16
|
+
CHUNK_BYTES = 64 * 1024
|
|
17
17
|
DIRECTORY_FLAGS = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
|
|
18
18
|
|
|
19
19
|
|
|
@@ -107,7 +107,7 @@ def fingerprint(s):
|
|
|
107
107
|
|
|
108
108
|
|
|
109
109
|
# 从固定目录安全读取有界普通文件,读前、读后和路径身份必须一致。
|
|
110
|
-
def read_file(parent, name, missing=False):
|
|
110
|
+
def read_file(parent, name, missing=False, consume=None):
|
|
111
111
|
try:
|
|
112
112
|
fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent)
|
|
113
113
|
except FileNotFoundError:
|
|
@@ -118,13 +118,24 @@ def read_file(parent, name, missing=False):
|
|
|
118
118
|
before = os.fstat(f.fileno())
|
|
119
119
|
if not stat.S_ISREG(before.st_mode) or before.st_size > MAX_BYTES:
|
|
120
120
|
raise ValueError("unsafe or oversized file")
|
|
121
|
-
|
|
121
|
+
digest = hashlib.sha256()
|
|
122
|
+
size = 0
|
|
123
|
+
while True:
|
|
124
|
+
chunk = f.read(CHUNK_BYTES)
|
|
125
|
+
if not chunk:
|
|
126
|
+
break
|
|
127
|
+
size += len(chunk)
|
|
128
|
+
if size > MAX_BYTES:
|
|
129
|
+
raise ValueError("oversized file")
|
|
130
|
+
digest.update(chunk)
|
|
131
|
+
if consume is not None and consume(chunk) != len(chunk):
|
|
132
|
+
raise IOError("incomplete file write")
|
|
122
133
|
after = os.fstat(f.fileno())
|
|
123
|
-
if
|
|
134
|
+
if size != after.st_size or fingerprint(before) != fingerprint(after):
|
|
124
135
|
raise ValueError("file changed while reading")
|
|
125
136
|
if fingerprint(after) != fingerprint(os.stat(name, dir_fd=parent, follow_symlinks=False)):
|
|
126
137
|
raise ValueError("file replaced while reading")
|
|
127
|
-
return
|
|
138
|
+
return digest.hexdigest(), fingerprint(after)
|
|
128
139
|
|
|
129
140
|
|
|
130
141
|
# 在固定目录中以随机名称独占创建私有临时文件,碰撞时重新选择名称。
|
|
@@ -139,11 +150,13 @@ def temporary(parent, prefix):
|
|
|
139
150
|
|
|
140
151
|
|
|
141
152
|
# 先写全并同步临时文件,再独占链接到归档名并同步目录,已有归档不会被覆盖。
|
|
142
|
-
def persist(parent, name,
|
|
153
|
+
def persist(parent, name, source_parent, source_name, expected, digest):
|
|
143
154
|
fd, pending = temporary(parent, ".net-connector-")
|
|
144
155
|
try:
|
|
145
156
|
with os.fdopen(fd, "wb") as f:
|
|
146
|
-
f.write
|
|
157
|
+
copied, current = read_file(source_parent, source_name, consume=f.write)
|
|
158
|
+
if current != expected or copied != digest:
|
|
159
|
+
raise ValueError("source changed before archive commit")
|
|
147
160
|
f.flush()
|
|
148
161
|
os.fsync(f.fileno())
|
|
149
162
|
# 临时名与最终名均相对固定目录解析,父路径改名也不能将写入重定向。
|
|
@@ -218,9 +231,9 @@ def run():
|
|
|
218
231
|
# 上传前先保留同名历史文件;只有服务器确认就绪后,客户端才允许设备导出。
|
|
219
232
|
verify_boundary()
|
|
220
233
|
old, previous = read_file(root_fd, name, missing=True)
|
|
221
|
-
if
|
|
234
|
+
if previous is not None:
|
|
222
235
|
previous_fd = directories.private(batch_fd, "previous")
|
|
223
|
-
persist(previous_fd, archive_name, old)
|
|
236
|
+
persist(previous_fd, archive_name, root_fd, name, previous, old)
|
|
224
237
|
fd, marker = temporary(root_fd, ".net-connector-upload-")
|
|
225
238
|
try:
|
|
226
239
|
started = os.fstat(fd).st_mtime_ns
|
|
@@ -233,26 +246,29 @@ def run():
|
|
|
233
246
|
return
|
|
234
247
|
verify_boundary()
|
|
235
248
|
# 新文件必须非空、区别于旧基线且不早于本次上传边界,旧文件不能冒充新结果。
|
|
236
|
-
|
|
237
|
-
if not
|
|
249
|
+
digest, current = read_file(root_fd, name)
|
|
250
|
+
if not current[2] or current == previous or current[3] < started:
|
|
238
251
|
raise ValueError("no new nonempty upload")
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
archived
|
|
242
|
-
if hashlib.sha256(archived).hexdigest() != digest:
|
|
252
|
+
persist(batch_fd, archive_name, root_fd, name, current, digest)
|
|
253
|
+
archived, saved = read_file(batch_fd, archive_name)
|
|
254
|
+
if archived != digest or saved[2] != current[2]:
|
|
243
255
|
raise ValueError("archive verification failed")
|
|
244
256
|
verify_boundary()
|
|
245
257
|
destination = os.path.join(root, "archive", batch, archive_name)
|
|
246
|
-
send(phase="archived", bytes=
|
|
247
|
-
|
|
258
|
+
send(phase="archived", bytes=current[2], sha256=digest, archive_path=destination)
|
|
259
|
+
delivered, check = read_file(batch_fd, archive_name, consume=sys.stdout.buffer.write)
|
|
260
|
+
if delivered != digest or check != saved:
|
|
261
|
+
raise ValueError("archive changed during delivery")
|
|
262
|
+
verify_boundary()
|
|
263
|
+
send(phase="delivered")
|
|
248
264
|
# 远端归档及正文已交付客户端;只有收到客户端核验后的明确清理指令才删除 staging。
|
|
249
265
|
action = receive()
|
|
250
266
|
if action.get("action") == "cleanup":
|
|
251
267
|
if action.get("sha256") != digest:
|
|
252
268
|
raise ValueError("cleanup digest mismatch")
|
|
253
269
|
verify_boundary()
|
|
254
|
-
|
|
255
|
-
if check != current or
|
|
270
|
+
uploaded, check = read_file(root_fd, name)
|
|
271
|
+
if check != current or uploaded != digest:
|
|
256
272
|
raise ValueError("upload changed before cleanup")
|
|
257
273
|
verify_boundary()
|
|
258
274
|
os.unlink(name, dir_fd=root_fd)
|
|
@@ -79,24 +79,30 @@ module Net
|
|
|
79
79
|
|
|
80
80
|
# 统一返回写入回执;失败时从异常 receipt 判断目录项是否已经替换。
|
|
81
81
|
def self.write(path, contents)
|
|
82
|
+
write_stream(path) { |file| file.write(contents) }
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
# 借出私有临时文件供分片写入,生产者正常返回后才同步并替换目标。
|
|
86
|
+
def self.write_stream(path)
|
|
82
87
|
raise ArgumentError, "path must be a nonempty String" unless path.is_a?(String) && !path.empty?
|
|
88
|
+
raise ArgumentError, "a streaming writer block is required" unless block_given?
|
|
83
89
|
|
|
84
90
|
receipt = Receipt.new(path: File.expand_path(path), state: :not_committed, phase: :resolve_parent)
|
|
85
91
|
directory = File.realpath(File.dirname(receipt.path))
|
|
86
92
|
destination = File.join(directory, File.basename(receipt.path))
|
|
87
93
|
receipt = receipt.with(phase: :temporary_file)
|
|
88
|
-
Tempfile.create([".net-connector-", ".tmp"], directory) do |file|
|
|
94
|
+
with_owned_file(-> { Tempfile.create([".net-connector-", ".tmp"], directory) }, temporary: true) do |file|
|
|
89
95
|
file.binmode
|
|
90
96
|
file.chmod(0o600)
|
|
91
97
|
receipt = receipt.with(phase: :write)
|
|
92
|
-
file
|
|
98
|
+
yield file
|
|
93
99
|
receipt = receipt.with(phase: :file_sync)
|
|
94
100
|
file.flush
|
|
95
101
|
file.fsync
|
|
96
102
|
receipt = receipt.with(phase: :rename)
|
|
97
103
|
File.rename(file.path, destination)
|
|
98
104
|
receipt = receipt.with(state: :committed, phase: :directory_open)
|
|
99
|
-
File.open(directory, File::RDONLY | File::NOFOLLOW) do |parent|
|
|
105
|
+
with_owned_file(-> { File.open(directory, File::RDONLY | File::NOFOLLOW) }) do |parent|
|
|
100
106
|
raise IOError, "parent is not a directory" unless parent.stat.directory?
|
|
101
107
|
|
|
102
108
|
receipt = receipt.with(phase: :directory_sync)
|
|
@@ -118,6 +124,57 @@ module Net
|
|
|
118
124
|
end
|
|
119
125
|
raise klass.new(receipt: receipt, underlying_type: error.class.name.to_s), cause: nil
|
|
120
126
|
end
|
|
127
|
+
|
|
128
|
+
# 唯一持有本次写入的文件句柄;临时目录项身份在取消递送前登记,退出时保留主控制流。
|
|
129
|
+
def self.with_owned_file(acquire, temporary: false)
|
|
130
|
+
file = path = identity = nil
|
|
131
|
+
completed = false
|
|
132
|
+
Thread.handle_interrupt(Exception => :never) do
|
|
133
|
+
file = acquire.call
|
|
134
|
+
if temporary
|
|
135
|
+
path = file.path
|
|
136
|
+
identity = file.stat
|
|
137
|
+
end
|
|
138
|
+
end
|
|
139
|
+
result = yield file
|
|
140
|
+
completed = true
|
|
141
|
+
result
|
|
142
|
+
rescue Exception # rubocop:disable Lint/RescueException -- 正常返回标记之后递送的取消也必须先于清理故障。
|
|
143
|
+
completed = false
|
|
144
|
+
raise
|
|
145
|
+
ensure
|
|
146
|
+
begin
|
|
147
|
+
cleanup_file(file, path, identity) if file
|
|
148
|
+
rescue StandardError, NotImplementedError
|
|
149
|
+
raise if completed
|
|
150
|
+
end
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
# 关闭失败仍尝试删除属于本次写入的临时目录项,已替换的同名文件不归本作用域所有。
|
|
154
|
+
def self.cleanup_file(file, path, identity)
|
|
155
|
+
Thread.handle_interrupt(Exception => :never) do
|
|
156
|
+
failure = nil
|
|
157
|
+
begin
|
|
158
|
+
file.close unless file.closed?
|
|
159
|
+
rescue Exception => error # rubocop:disable Lint/RescueException -- 关闭故障不能跳过临时目录项清理。
|
|
160
|
+
failure = error
|
|
161
|
+
end
|
|
162
|
+
begin
|
|
163
|
+
if path && identity
|
|
164
|
+
current = File.lstat(path)
|
|
165
|
+
File.unlink(path) if current.dev == identity.dev && current.ino == identity.ino
|
|
166
|
+
end
|
|
167
|
+
rescue Errno::ENOENT
|
|
168
|
+
# 原子替换已经移走临时目录项,或其他清理已将其删除。
|
|
169
|
+
nil
|
|
170
|
+
rescue Exception => error # rubocop:disable Lint/RescueException -- 保留最先发生的清理故障。
|
|
171
|
+
failure ||= error
|
|
172
|
+
end
|
|
173
|
+
raise failure if failure
|
|
174
|
+
end
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
private_class_method :with_owned_file, :cleanup_file
|
|
121
178
|
end
|
|
122
179
|
end
|
|
123
180
|
end
|
data/lib/net/connector.rb
CHANGED
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: net-connector
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.19.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- net-connector contributors
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date: 2026-10-
|
|
10
|
+
date: 2026-10-29 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: ctx_cli
|
|
@@ -15,28 +15,28 @@ dependencies:
|
|
|
15
15
|
requirements:
|
|
16
16
|
- - ">="
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version: 0
|
|
18
|
+
version: '0'
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
23
|
- - ">="
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version: 0
|
|
25
|
+
version: '0'
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: expect-pty
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
30
|
- - ">="
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: 0
|
|
32
|
+
version: '0'
|
|
33
33
|
type: :runtime
|
|
34
34
|
prerelease: false
|
|
35
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
36
|
requirements:
|
|
37
37
|
- - ">="
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: 0
|
|
39
|
+
version: '0'
|
|
40
40
|
- !ruby/object:Gem::Dependency
|
|
41
41
|
name: textfsm
|
|
42
42
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -71,14 +71,14 @@ dependencies:
|
|
|
71
71
|
requirements:
|
|
72
72
|
- - ">="
|
|
73
73
|
- !ruby/object:Gem::Version
|
|
74
|
-
version: '
|
|
74
|
+
version: '0'
|
|
75
75
|
type: :runtime
|
|
76
76
|
prerelease: false
|
|
77
77
|
version_requirements: !ruby/object:Gem::Requirement
|
|
78
78
|
requirements:
|
|
79
79
|
- - ">="
|
|
80
80
|
- !ruby/object:Gem::Version
|
|
81
|
-
version: '
|
|
81
|
+
version: '0'
|
|
82
82
|
- !ruby/object:Gem::Dependency
|
|
83
83
|
name: digest
|
|
84
84
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -375,6 +375,7 @@ files:
|
|
|
375
375
|
- lib/net/connector.rb
|
|
376
376
|
- lib/net/connector/collect/progress.rb
|
|
377
377
|
- lib/net/connector/config.rb
|
|
378
|
+
- lib/net/connector/credentials.rb
|
|
378
379
|
- lib/net/connector/device/base.rb
|
|
379
380
|
- lib/net/connector/device/interfaces.rb
|
|
380
381
|
- lib/net/connector/device/interfaces/audit.rb
|
|
@@ -436,6 +437,7 @@ files:
|
|
|
436
437
|
- lib/net/connector/inventory/selection.rb
|
|
437
438
|
- lib/net/connector/netdisco.rb
|
|
438
439
|
- lib/net/connector/netdisco/batch.rb
|
|
440
|
+
- lib/net/connector/netdisco/catalog.rb
|
|
439
441
|
- lib/net/connector/netdisco/cli.rb
|
|
440
442
|
- lib/net/connector/netdisco/cli/backup.rb
|
|
441
443
|
- lib/net/connector/netdisco/cli/credentials.rb
|
|
@@ -472,6 +474,7 @@ files:
|
|
|
472
474
|
- lib/net/connector/netdisco/ssh/archive.rb
|
|
473
475
|
- lib/net/connector/netdisco/ssh/archive/client.rb
|
|
474
476
|
- lib/net/connector/netdisco/ssh/archive/pool.rb
|
|
477
|
+
- lib/net/connector/netdisco/ssh/archive/protocol.rb
|
|
475
478
|
- lib/net/connector/netdisco/ssh/archive/worker.py
|
|
476
479
|
- lib/net/connector/netdisco/ssh/authentication.rb
|
|
477
480
|
- lib/net/connector/netdisco/tftp/archive.rb
|