net-connector 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +50 -1
- data/docs/architecture.md +6 -0
- data/examples/netdisco_database.yml +16 -0
- data/lib/net/connector/engine/authentication.rb +1 -1
- data/lib/net/connector/engine/dialogue.rb +4 -4
- data/lib/net/connector/engine/execution.rb +2 -2
- data/lib/net/connector/engine/session.rb +1 -1
- data/lib/net/connector/netdisco/cli.rb +11 -2
- data/lib/net/connector/netdisco/client.rb +2 -1
- data/lib/net/connector/netdisco/config_file.rb +6 -3
- data/lib/net/connector/netdisco/database_client.rb +197 -0
- data/lib/net/connector/netdisco/settings.rb +26 -3
- data/lib/net/connector/netdisco.rb +1 -0
- data/lib/net/connector/version.rb +1 -1
- metadata +23 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b9f2c5dfacb73a1c1aecf350bd4d1fe93755e0be25dbd1cc5c964a5676beaee7
|
|
4
|
+
data.tar.gz: 3afcf973ef1a25d848b1a91feb5803348eea6881f9d4b2721525378f9611d0d9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5b168ca118a7323f1b698908b7e6db2e627c97b52f57a4817f8b932456862ba9097491f4f879b23ffcdd9d8ea1c6f287b870354a9c279cf04963295b9cec815f
|
|
7
|
+
data.tar.gz: 1d5cdcedba90facbc57a9a33438c66a244c482ea580399e901f9b42572a5b6401f0cbb68f01e1bf106eb53e0562f4f213742de1cc2dc9dff80f0bae47697eb81
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# 更新记录
|
|
2
2
|
|
|
3
|
+
## 0.6.0 - 2026-09-27
|
|
4
|
+
|
|
5
|
+
- 支持直接查询 PostgreSQL 获取 Netdisco 设备清单,SQL 和绑定参数可通过 YAML、环境变量或 CLI 设置,查询结果继续使用已有的厂商映射、计划预览和批量备份流程。
|
|
6
|
+
- 数据库连接信息只从 `NETDISCO_DB_*` 环境变量读取,查询在只读事务中执行并限制时间和容量,失败时停止整批任务,错误输出不包含数据库凭据或原始响应。
|
|
7
|
+
- 增加可编辑的数据库查询示例并随 gem 分发,安装时包含所需的 `pg` 驱动,原有 HTTP 清单及离线配置导出保持可用。
|
|
8
|
+
|
|
3
9
|
## 0.5.0 - 2026-09-27
|
|
4
10
|
|
|
5
11
|
本版包含不兼容的接口调整,不保留旧路径或方法别名;升级时请按[当前接口说明](https://github.com/gatework/net-connector/blob/v0.5.0/docs/architecture.md#当前命名与接口调整)更新调用方。
|
data/README.md
CHANGED
|
@@ -274,6 +274,47 @@ net-connector-backup --config config.yml --tftp --all
|
|
|
274
274
|
net-connector-backup --config config.yml --export 192.0.2.7 --output ./exports/device.cfg
|
|
275
275
|
```
|
|
276
276
|
|
|
277
|
+
### PostgreSQL 联机查询
|
|
278
|
+
|
|
279
|
+
设置 `netdisco.source: postgres` 可以直接从数据库查询清单,继续使用同一套 Fleet、规则、计划和备份流程。库不内置表名、SQL 或业务筛选条件;必须提供查询,可直接修改 [YAML 示例](examples/netdisco_database.yml):
|
|
280
|
+
|
|
281
|
+
```yaml
|
|
282
|
+
netdisco:
|
|
283
|
+
source: postgres
|
|
284
|
+
query: |
|
|
285
|
+
SELECT host(ip) AS ip, name, dns, vendor, os, model, os_ver, serial
|
|
286
|
+
FROM device
|
|
287
|
+
WHERE vendor = $1
|
|
288
|
+
ORDER BY ip
|
|
289
|
+
query_params: [H3C]
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
查询必须返回唯一命名的 `ip` 列;可选列为 `name`、`dns`、`vendor`、`os`、`model`、`os_ver`、`serial`,其他列会丢弃。自定义表、视图、JOIN 和只读 CTE 均可用列别名适配这一契约。PostgreSQL 的 `inet` 字段建议用 `host(ip) AS ip` 去除前缀长度;筛选及排序由 SQL 决定。参数按数组顺序绑定 `$1`、`$2`,支持字符串、数字、布尔和 null,不做字符串插值或环境变量展开。
|
|
293
|
+
|
|
294
|
+
数据库连接信息只从进程环境读取,沿用 [Netdisco 官方环境变量命名](https://github.com/netdisco/netdisco/wiki/Environment-Variables)。`HOST`、`NAME`、`USER`、`PASS` 必填;HTTP 地址、API key 和设备登录凭据不是清单查询的前置条件:
|
|
295
|
+
|
|
296
|
+
```sh
|
|
297
|
+
export NETDISCO_DB_HOST=database.example
|
|
298
|
+
export NETDISCO_DB_NAME=netdisco
|
|
299
|
+
export NETDISCO_DB_USER=inventory-reader
|
|
300
|
+
export NETDISCO_DB_PASS='replace-me'
|
|
301
|
+
export NETDISCO_DB_SSLMODE=verify-full
|
|
302
|
+
export NETDISCO_DB_SSLROOTCERT=/etc/net-connector/database-ca.crt
|
|
303
|
+
|
|
304
|
+
net-connector-backup --config examples/netdisco_database.yml --show-config
|
|
305
|
+
net-connector-backup --config examples/netdisco_database.yml --plan
|
|
306
|
+
# 覆盖查询参数;实际备份仍需设置 NET_CONNECTOR_DEVICE_* 凭据。
|
|
307
|
+
net-connector-backup --config examples/netdisco_database.yml --plan --query-params '["Cisco"]'
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
SQL 也可通过 `NETDISCO_QUERY` / `--query SQL` 提供,参数通过 `NETDISCO_QUERY_PARAMS` / `--query-params JSON` 提供;来源对应 `NETDISCO_SOURCE` / `--source postgres`。优先级均为 CLI > ENV > YAML。SQL 和参数属于可公开配置,会出现在 `--show-config` 中;数据库密码只放在连接环境变量中。连接信息不进入策略快照、计划、报告或 `inspect`。已有 Fleet 每次重新查询时读取最新连接凭据;传入已有 `plan:` 执行时不会重新查询。
|
|
311
|
+
|
|
312
|
+
客户端通过 `pg` 驱动执行只读事务,使用参数化游标分批取数,并在每批启用单行读取。PostgreSQL 原生解析拒绝多条语句,写入和锁定查询会失败;查询账户应仅授予所需表/视图的 SELECT 权限,只读事务不能替代账户权限隔离。驱动只在实际查询时加载,HTTP 和离线导出路径不加载它。
|
|
313
|
+
|
|
314
|
+
数据库查询复用现有清单预算:`page_size` 控制每次 FETCH 的行数,`max_pages` 限制 FETCH 次数,`max_devices` 在去重前计数;字节预算统计返回列名与文本值,包含最终丢弃的列。字节检查发生在 libpq 解码一行之后,不能限制单个超大字段在驱动内部的瞬时内存。连接、查询和全部 FETCH 共用总期限,同时设置数据库 statement_timeout。任一查询错误、超时或超限均关闭连接并丢弃整份清单,不连接设备;错误消息不输出原始 SQL、参数或数据库响应。建议只选所需字段,并提供明确的 ORDER BY 保持采样顺序稳定。
|
|
315
|
+
|
|
316
|
+
### 计划与执行
|
|
317
|
+
|
|
277
318
|
`--plan` 只拉取并验证清单;`--host` 选择一个管理地址;`--tftp` 默认每厂商最多选择五台,`--all` 选择所有就绪设备。本地备份默认选择所有就绪设备,可用 `--limit-per-vendor` 限制。`--show-config` 只输出有效的非敏感设置,不访问 Netdisco。CLI 会拒绝未知 YAML 字段、Ruby 对象标签及配置中的凭据。`--export IP` 只离线读取规范化管理地址对应的 `<IP>.txt`;默认原样写到标准输出,指定 `--output` 后以 `0600` 权限原子写文件。导出的配置仍是敏感数据。
|
|
278
319
|
|
|
279
320
|
CLI 的计划与批次摘要使用 JSON。默认 `--success-policy strict` 使用严格规则:非空清单且全部成功、回调及报告正常时为 `0`;空清单或有跳过、部分成功、失败时为 `1`;清单或配置错误为 `2`。`--host` 未在清单中找到也返回 `2`。由于其他清单记录会标记为过滤,默认单主机备份成功时批次退出码仍可能是 `1`;应查看 JSON 中的 `succeeded`、`skipped` 和逐台 `status`。
|
|
@@ -291,10 +332,18 @@ CLI 的计划与批次摘要使用 JSON。默认 `--success-policy strict` 使
|
|
|
291
332
|
|
|
292
333
|
| 环境变量 | 默认值 | 用途 |
|
|
293
334
|
| --- | --- | --- |
|
|
294
|
-
| `
|
|
335
|
+
| `NETDISCO_SOURCE` | `http` | 清单来源,支持 `http`、`postgres` |
|
|
336
|
+
| `NETDISCO_URL` | HTTP 模式必填 | Netdisco 服务根地址,可包含租户路径 |
|
|
295
337
|
| `NET_CONNECTOR_CONFIG` | 未设置 | CLI 的非敏感 YAML 配置文件 |
|
|
296
338
|
| `NETDISCO_USERNAME`, `NETDISCO_PASSWORD` | 未提供 API 密钥时必填 | 清单 API 登录 |
|
|
297
339
|
| `NETDISCO_API_KEY` | 未设置 | 直接使用已有 API 密钥 |
|
|
340
|
+
| `NETDISCO_QUERY` | PostgreSQL 模式必填 | 用户提供的单条清单 SQL |
|
|
341
|
+
| `NETDISCO_QUERY_PARAMS` | `[]` | SQL 参数的 JSON 标量数组 |
|
|
342
|
+
| `NETDISCO_DB_HOST`, `NETDISCO_DB_NAME` | PostgreSQL 模式必填 | 数据库主机或 Unix socket 目录、数据库名 |
|
|
343
|
+
| `NETDISCO_DB_USER`, `NETDISCO_DB_PASS` | PostgreSQL 模式必填 | 仅从环境注入的数据库用户名、密码 |
|
|
344
|
+
| `NETDISCO_DB_PORT` | libpq 默认 `5432` | PostgreSQL 端口 |
|
|
345
|
+
| `NETDISCO_DB_SSLMODE`, `NETDISCO_DB_SSLROOTCERT` | libpq 默认 | TLS 模式、CA 文件;远程连接建议 `verify-full` |
|
|
346
|
+
| `NETDISCO_DB_CONNECT_TIMEOUT` | 未单独设置 | 可选正整数秒;连接始终受清单总期限限制 |
|
|
298
347
|
| `NETDISCO_PAGE_SIZE` | `500` | 清单分页大小 |
|
|
299
348
|
| `NETDISCO_MAX_PAGES` | `10000` | 最大分页次数 |
|
|
300
349
|
| `NETDISCO_MAX_RESPONSE_BYTES` | `16777216` | 单次响应正文上限,认证和错误正文也计数 |
|
data/docs/architecture.md
CHANGED
|
@@ -112,6 +112,12 @@ Diagnostic 只保存固定词表中的码、类型、阶段和受控产物状态
|
|
|
112
112
|
|
|
113
113
|
注入的 `requester` 接收 URI 和 request。它返回后才接受长度和期限检查,不在任意用户 Ruby 回调中注入异步异常。解析和集合操作完成后也检查期限,但这不构成任意 CPU 回调的抢占保证。预算失败仅报告错误码与安全类型,没有部分清单、响应正文或底层 cause。HTTPS 的标准证书检查不变;默认允许明文 HTTP,可通过显式策略禁止。
|
|
114
114
|
|
|
115
|
+
`Settings#inventory_source` 选择 HTTP `Client` 或 PostgreSQL `DatabaseClient`,两者只向 Fleet 提供 `devices`,不改变设备映射、计划和执行接口。数据库 SQL 与标量参数由用户配置,不内置业务表名或筛选;结果用列别名适配 `Client::FIELDS`,只保留这份现有字段白名单。连接参数由 `DatabaseClient::CONNECTION_ENV` 集中映射到 Netdisco 的 `NETDISCO_DB_*` 环境变量,全部排除在纯策略快照外;每次新建清单客户端时读取最新值。
|
|
116
|
+
|
|
117
|
+
`DatabaseClient` 每次调用独占 PostgreSQL 连接,在只读事务中用扩展查询协议声明游标,原生解析器拒绝多语句和非查询输入。FETCH 大小及次数复用清单预算,单行模式使 libpq 不缓存整页;每行在追加前计数,额外列也消耗字节预算。驱动必须先解码单行,所以单个超大字段仍可超过 Ruby 预算的瞬时内存。总 deadline 覆盖连接和所有语句,每次 FETCH 前缩短 statement_timeout;正常完成回滚只读事务,异常关闭连接,任何失败均不交付部分清单。NOTICE、原始数据库异常及 cause 不进入日志或 CLI;稳定错误码区分连接失败、查询失败、无效清单和预算超限。SQL/参数可在 show-config 中查看,不能用于传递凭据。
|
|
118
|
+
|
|
119
|
+
数据库集成测试运行 `bundle exec rake test:postgres`,通过 `pg_config --bindir`(或 `NET_CONNECTOR_TEST_PG_BINDIR`)定位服务端工具。测试只创建临时 SCRAM 数据库和私有 Unix socket,退出时停止并删除;不会读取真实 Netdisco 凭据或连接已有服务。覆盖 SQL/参数、只读限制、真实认证失败、查询中途失败、各项预算、连接回收以及 CLI 计划。
|
|
120
|
+
|
|
115
121
|
## 迁移与尚未启用的能力
|
|
116
122
|
|
|
117
123
|
| 变化 | 调用方需要保留的约定 |
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# 数据库连接只从 NETDISCO_DB_* 环境变量读取;此文件可直接编辑并纳入版本控制。
|
|
2
|
+
netdisco:
|
|
3
|
+
source: postgres
|
|
4
|
+
query: |
|
|
5
|
+
SELECT host(ip) AS ip, name, dns, vendor, os, model, os_ver, serial
|
|
6
|
+
FROM device
|
|
7
|
+
WHERE vendor = $1
|
|
8
|
+
ORDER BY ip
|
|
9
|
+
query_params: [H3C]
|
|
10
|
+
page_size: 500
|
|
11
|
+
max_devices: 100000
|
|
12
|
+
inventory_timeout: 300
|
|
13
|
+
|
|
14
|
+
backup:
|
|
15
|
+
directory: ./backups
|
|
16
|
+
concurrency: 4
|
|
@@ -62,7 +62,7 @@ module Net
|
|
|
62
62
|
host_key = Interaction.new(SSH_CONFIRMATION, lambda { |_prompt|
|
|
63
63
|
if @config.host_key_policy == :strict
|
|
64
64
|
raise @session.build_error(ConnectionError, "host key confirmation rejected",
|
|
65
|
-
|
|
65
|
+
phase: :login, code: :host_key_untrusted), cause: nil
|
|
66
66
|
end
|
|
67
67
|
"yes\n"
|
|
68
68
|
}, limit: 1)
|
|
@@ -172,7 +172,7 @@ module Net
|
|
|
172
172
|
def validate_event!(event, bytes, raw, phase, command)
|
|
173
173
|
if bytes > @session.configuration.max_output_bytes
|
|
174
174
|
raise @session.build_error(OutputLimitExceeded, "device output exceeded max_output_bytes",
|
|
175
|
-
|
|
175
|
+
phase: phase, command: command, output: (raw + [event.match]).join), cause: nil
|
|
176
176
|
end
|
|
177
177
|
raise build_read_error(event, phase, command, raw.join), cause: nil unless event.matched?
|
|
178
178
|
end
|
|
@@ -181,7 +181,7 @@ module Net
|
|
|
181
181
|
def finish_response(prompt, raw, output, phase, command)
|
|
182
182
|
if prompt.empty?
|
|
183
183
|
raise @session.build_error(PromptError, "prompt pattern did not consume any output",
|
|
184
|
-
|
|
184
|
+
phase: phase, command: command, output: raw.join), cause: nil
|
|
185
185
|
end
|
|
186
186
|
|
|
187
187
|
raw << prompt
|
|
@@ -229,7 +229,7 @@ module Net
|
|
|
229
229
|
def build_read_error(event, phase, command, output)
|
|
230
230
|
if event&.error.is_a?(Exception)
|
|
231
231
|
return @session.build_error(TransportError, "transport read failed", phase: phase, command: command,
|
|
232
|
-
|
|
232
|
+
underlying: event.error, output: output)
|
|
233
233
|
end
|
|
234
234
|
klass = if event&.error == :eof
|
|
235
235
|
ConnectionClosed
|
|
@@ -241,7 +241,7 @@ module Net
|
|
|
241
241
|
CommandTimeout
|
|
242
242
|
end
|
|
243
243
|
@session.build_error(klass, "device response #{event&.error || :timeout}", phase: phase,
|
|
244
|
-
|
|
244
|
+
command: command, output: output)
|
|
245
245
|
end
|
|
246
246
|
end
|
|
247
247
|
end
|
|
@@ -90,8 +90,8 @@ module Net
|
|
|
90
90
|
return unless @output_limit && (completed ? @output_bytes > @output_limit : @output_bytes >= @output_limit)
|
|
91
91
|
|
|
92
92
|
raise @session.build_error(ScriptOutputLimitExceeded,
|
|
93
|
-
|
|
94
|
-
|
|
93
|
+
"script output reached max_script_output_bytes; commands already sent may have executed",
|
|
94
|
+
phase: :script, command: command), cause: nil
|
|
95
95
|
end
|
|
96
96
|
end
|
|
97
97
|
end
|
|
@@ -350,7 +350,7 @@ module Net
|
|
|
350
350
|
redactor.reset
|
|
351
351
|
@log.open(transport)
|
|
352
352
|
@log.log_event("connect", host: configuration.host, username: configuration.username,
|
|
353
|
-
|
|
353
|
+
phase: :connect, protocol: transport.respond_to?(:protocol) ? transport.protocol : configuration.protocol)
|
|
354
354
|
transport.open
|
|
355
355
|
@state = :authenticating
|
|
356
356
|
@log.log_event("login_start", level: :debug, phase: :login)
|
|
@@ -69,6 +69,9 @@ module Net
|
|
|
69
69
|
def settings_for(options)
|
|
70
70
|
defaults = options[:config] ? ConfigFile.load(options[:config]) : {}
|
|
71
71
|
values = {}
|
|
72
|
+
values["NETDISCO_SOURCE"] = options[:source] if options[:source]
|
|
73
|
+
values["NETDISCO_QUERY"] = options[:query] if options[:query]
|
|
74
|
+
values["NETDISCO_QUERY_PARAMS"] = options[:query_params] if options[:query_params]
|
|
72
75
|
values["NET_CONNECTOR_BACKUP_DIRECTORY"] = options[:directory] if options[:directory]
|
|
73
76
|
values["NET_CONNECTOR_CONCURRENCY"] = options[:concurrency].to_s if options[:concurrency]
|
|
74
77
|
values["NET_CONNECTOR_MAX_SCRIPT_OUTPUT_BYTES"] = options[:max_script_output_bytes].to_s if options[:max_script_output_bytes]
|
|
@@ -81,7 +84,7 @@ module Net
|
|
|
81
84
|
# 导出已有配置不创建清单或设备连接。
|
|
82
85
|
def export_config(settings, options)
|
|
83
86
|
path = Storage::SavedConfig.new(directory: settings.backup_directory)
|
|
84
|
-
|
|
87
|
+
.export(host: options[:export], output: options[:output], io: @output)
|
|
85
88
|
@output.puts JSON.generate(host: options[:export], output: path) if path
|
|
86
89
|
end
|
|
87
90
|
|
|
@@ -120,6 +123,9 @@ module Net
|
|
|
120
123
|
parser = OptionParser.new do |args|
|
|
121
124
|
args.banner = "用法:net-connector-backup [--config FILE] [--plan | --show-config | --export IP] [--tftp]"
|
|
122
125
|
args.on("--config FILE", "读取不含凭据的 YAML 设置") { |value| options[:config] = value }
|
|
126
|
+
args.on("--source SOURCE", %w[http postgres], "清单来源:http(默认)或 postgres") { |value| options[:source] = value }
|
|
127
|
+
args.on("--query SQL", "PostgreSQL 清单查询,使用 $1、$2 绑定参数") { |value| options[:query] = value }
|
|
128
|
+
args.on("--query-params JSON", "查询参数的 JSON 数组") { |value| options[:query_params] = value }
|
|
123
129
|
args.on("--show-config", "显示生效的非敏感设置") { options[:show_config] = true }
|
|
124
130
|
args.on("--plan", "预览设备清单,不连接设备") { options[:plan] = true }
|
|
125
131
|
args.on("--tftp", "由设备发起 TFTP 备份") { options[:tftp] = true }
|
|
@@ -158,8 +164,11 @@ module Net
|
|
|
158
164
|
raise ArgumentError, "--show-config、--plan 和 --export 只能选择一项" if modes > 1
|
|
159
165
|
raise ArgumentError, "--output 需要同时指定 --export" if options[:output] && !options[:export]
|
|
160
166
|
raise ArgumentError, "--all 不能与 --limit-per-vendor 同时使用" if options[:all] && options[:limit]
|
|
167
|
+
if options[:export] && (options[:source] || options[:query] || options[:query_params])
|
|
168
|
+
raise ArgumentError, "--export 不能与清单查询选项同时使用"
|
|
169
|
+
end
|
|
161
170
|
if options[:export] && (options[:tftp] || options[:all] || options[:limit] || options[:concurrency] || options[:host] ||
|
|
162
|
-
|
|
171
|
+
options[:success_policy] || options[:max_script_output_bytes])
|
|
163
172
|
raise ArgumentError, "--export 不能与备份选项同时使用"
|
|
164
173
|
end
|
|
165
174
|
end
|
|
@@ -96,7 +96,7 @@ module Net
|
|
|
96
96
|
@options.fetch(:max_pages).times do |index|
|
|
97
97
|
uri = endpoint("api/v1/search/device")
|
|
98
98
|
uri.query = URI.encode_www_form(fields: FIELDS.join(","), limit: @options.fetch(:page_size),
|
|
99
|
-
|
|
99
|
+
offset: index * @options.fetch(:page_size))
|
|
100
100
|
page = request_json(uri, authorized_get(uri, token), budget, allow_empty: true, query_fallback: true)
|
|
101
101
|
validate_rows!(page, budget)
|
|
102
102
|
return rows.uniq if page.empty?
|
|
@@ -274,6 +274,7 @@ module Net
|
|
|
274
274
|
end
|
|
275
275
|
|
|
276
276
|
def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
277
|
+
|
|
277
278
|
def present?(value) = value.is_a?(String) && !value.strip.empty?
|
|
278
279
|
end
|
|
279
280
|
end
|
|
@@ -9,7 +9,10 @@ module Net
|
|
|
9
9
|
# 命令行使用的无凭据 YAML 配置,环境变量优先。
|
|
10
10
|
class ConfigFile
|
|
11
11
|
FIELDS = {
|
|
12
|
-
"netdisco" => { "
|
|
12
|
+
"netdisco" => { "source" => ["NETDISCO_SOURCE", :string],
|
|
13
|
+
"query" => ["NETDISCO_QUERY", :string],
|
|
14
|
+
"query_params" => ["NETDISCO_QUERY_PARAMS", :json_array],
|
|
15
|
+
"url" => ["NETDISCO_URL", :string],
|
|
13
16
|
"page_size" => ["NETDISCO_PAGE_SIZE", :integer],
|
|
14
17
|
"max_pages" => ["NETDISCO_MAX_PAGES", :integer],
|
|
15
18
|
"max_response_bytes" => ["NETDISCO_MAX_RESPONSE_BYTES", :integer],
|
|
@@ -94,8 +97,8 @@ module Net
|
|
|
94
97
|
end
|
|
95
98
|
|
|
96
99
|
value.join(",")
|
|
97
|
-
when :json
|
|
98
|
-
expected = name == "device_rules" ? Array : Hash
|
|
100
|
+
when :json, :json_array
|
|
101
|
+
expected = type == :json_array || name == "device_rules" ? Array : Hash
|
|
99
102
|
raise ArgumentError, "#{section}.#{name} must be a #{expected}" unless value.is_a?(expected)
|
|
100
103
|
|
|
101
104
|
JSON.generate(value)
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "timeout"
|
|
4
|
+
require_relative "client"
|
|
5
|
+
|
|
6
|
+
module Net
|
|
7
|
+
module Connector
|
|
8
|
+
module Netdisco
|
|
9
|
+
# SQL 负责业务选择和列别名;适配器只管理只读查询、清单预算与连接生命周期。
|
|
10
|
+
class DatabaseClient
|
|
11
|
+
CONNECTION_ENV = {
|
|
12
|
+
host: "NETDISCO_DB_HOST", port: "NETDISCO_DB_PORT", dbname: "NETDISCO_DB_NAME",
|
|
13
|
+
user: "NETDISCO_DB_USER", password: "NETDISCO_DB_PASS",
|
|
14
|
+
sslmode: "NETDISCO_DB_SSLMODE", sslrootcert: "NETDISCO_DB_SSLROOTCERT",
|
|
15
|
+
connect_timeout: "NETDISCO_DB_CONNECT_TIMEOUT"
|
|
16
|
+
}.freeze
|
|
17
|
+
REQUIRED_CONNECTION_KEYS = %i[host dbname user password].freeze
|
|
18
|
+
DEFAULTS = Client::DEFAULTS.except(:allow_insecure_http).freeze
|
|
19
|
+
|
|
20
|
+
def self.options(query:, query_params: [], **values)
|
|
21
|
+
unless query.is_a?(String) && !query.strip.empty? && !query.include?("\0")
|
|
22
|
+
raise ArgumentError, "query must be a nonempty SQL string without NUL"
|
|
23
|
+
end
|
|
24
|
+
unless query_params.is_a?(Array) && query_params.all? { |value|
|
|
25
|
+
value.nil? || value == true || value == false || value.is_a?(Integer) ||
|
|
26
|
+
(value.is_a?(Float) && value.finite?) || (value.is_a?(String) && !value.include?("\0"))
|
|
27
|
+
}
|
|
28
|
+
raise ArgumentError, "query_params must be an array of JSON scalar values without NUL"
|
|
29
|
+
end
|
|
30
|
+
raise ArgumentError, "unknown database inventory options" unless (values.keys - DEFAULTS.keys).empty?
|
|
31
|
+
|
|
32
|
+
Client.options(**values).slice(*DEFAULTS.keys).merge(
|
|
33
|
+
query: query.dup.freeze,
|
|
34
|
+
query_params: query_params.map { |value| value.is_a?(String) ? value.dup.freeze : value }.freeze
|
|
35
|
+
).freeze
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def initialize(connection_options:, **options)
|
|
39
|
+
@options = self.class.options(**options)
|
|
40
|
+
@connection_options = validate_connection_options(connection_options)
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# 每次查询独占连接和游标;只有全部结果验证成功后才把清单交给 Fleet。
|
|
44
|
+
def devices
|
|
45
|
+
require "pg"
|
|
46
|
+
budget = InventoryBudget.new(@options, clock: method(:monotonic))
|
|
47
|
+
connection = nil
|
|
48
|
+
connected = false
|
|
49
|
+
begin
|
|
50
|
+
Timeout.timeout(budget.remaining, Client::InventoryTimeout) do
|
|
51
|
+
# 从握手开始持有连接;PG.connect 尚未返回时被超时中断,调用方无法负责关闭它。
|
|
52
|
+
connection = PG::Connection.connect_start(@connection_options)
|
|
53
|
+
# PostgreSQL NOTICE 可能包含 SQL 或数据,不交给 libpq 默认的 stderr 输出器。
|
|
54
|
+
connection.set_notice_processor { |_notice| nil }
|
|
55
|
+
wait_for_connection(connection, budget)
|
|
56
|
+
connected = true
|
|
57
|
+
# 使用 pg 可中断的 Ruby 等待接口;底层 socket 仍以非阻塞方式工作。
|
|
58
|
+
connection.setnonblocking(false)
|
|
59
|
+
connection.set_client_encoding("UTF8")
|
|
60
|
+
connection.exec("BEGIN READ ONLY").clear
|
|
61
|
+
set_statement_timeout(connection, budget)
|
|
62
|
+
# 即使没有参数也走扩展查询协议,由 PostgreSQL 拒绝多语句和非查询命令。
|
|
63
|
+
connection.exec_params("DECLARE net_connector_inventory NO SCROLL CURSOR FOR\n#{@options.fetch(:query)}",
|
|
64
|
+
@options.fetch(:query_params)).clear
|
|
65
|
+
rows = inventory(connection, budget)
|
|
66
|
+
connection.exec("ROLLBACK").clear
|
|
67
|
+
budget.remaining
|
|
68
|
+
rows
|
|
69
|
+
end
|
|
70
|
+
rescue Client::Error => error
|
|
71
|
+
raise error, cause: nil
|
|
72
|
+
rescue PG::QueryCanceled
|
|
73
|
+
raise Client::InventoryTimeout, cause: nil
|
|
74
|
+
rescue PG::Error, IOError, SystemCallError
|
|
75
|
+
code = connected ? :query_failed : :connection_failed
|
|
76
|
+
raise Client::Error.new("Netdisco database #{code.to_s.tr("_", " ")}", code: code), cause: nil
|
|
77
|
+
ensure
|
|
78
|
+
connection&.finish
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def inspect = "#<#{self.class}>"
|
|
83
|
+
|
|
84
|
+
private
|
|
85
|
+
|
|
86
|
+
def wait_for_connection(connection, budget)
|
|
87
|
+
deadline = monotonic + @connection_options.fetch(:connect_timeout, budget.remaining).to_f
|
|
88
|
+
loop do
|
|
89
|
+
status = connection.connect_poll
|
|
90
|
+
return if status == PG::PGRES_POLLING_OK
|
|
91
|
+
raise PG::ConnectionBad unless [PG::PGRES_POLLING_READING, PG::PGRES_POLLING_WRITING].include?(status)
|
|
92
|
+
|
|
93
|
+
remaining = [budget.remaining, deadline - monotonic].min
|
|
94
|
+
raise PG::ConnectionBad if remaining <= 0
|
|
95
|
+
|
|
96
|
+
socket = connection.socket_io
|
|
97
|
+
readers = status == PG::PGRES_POLLING_READING ? [socket] : nil
|
|
98
|
+
writers = status == PG::PGRES_POLLING_WRITING ? [socket] : nil
|
|
99
|
+
next if IO.select(readers, writers, [socket], remaining)
|
|
100
|
+
|
|
101
|
+
budget.remaining
|
|
102
|
+
raise PG::ConnectionBad
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def validate_connection_options(options)
|
|
107
|
+
unless options.is_a?(Hash) && (options.keys - CONNECTION_ENV.keys).empty?
|
|
108
|
+
raise ArgumentError, "connection_options must contain supported PostgreSQL connection keys"
|
|
109
|
+
end
|
|
110
|
+
REQUIRED_CONNECTION_KEYS.each do |key|
|
|
111
|
+
value = options[key]
|
|
112
|
+
unless value.is_a?(String) && !value.strip.empty?
|
|
113
|
+
raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} is required"
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
options.each do |key, value|
|
|
117
|
+
valid = value.is_a?(String) && !value.empty? && !value.include?("\0")
|
|
118
|
+
valid ||= %i[port connect_timeout].include?(key) && value.is_a?(Integer)
|
|
119
|
+
raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} is invalid" unless valid
|
|
120
|
+
end
|
|
121
|
+
%i[port connect_timeout].each do |key|
|
|
122
|
+
next unless options.key?(key)
|
|
123
|
+
|
|
124
|
+
value = options.fetch(key).to_s
|
|
125
|
+
maximum = key == :port ? 65_535 : 2_147_483_647
|
|
126
|
+
unless value.match?(/\A\d+\z/) && value.to_i.between?(1, maximum)
|
|
127
|
+
raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} must be a positive integer within range"
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
if options[:sslmode] && !%w[disable allow prefer require verify-ca verify-full].include?(options[:sslmode])
|
|
131
|
+
raise ArgumentError, "NETDISCO_DB_SSLMODE is invalid"
|
|
132
|
+
end
|
|
133
|
+
# libpq 会把 dbname 中的 URI/conninfo 再解释为连接参数,不能借此绕过凭据来源。
|
|
134
|
+
if options.fetch(:dbname).match?(/=|\Apostgres(?:ql)?:\/\//)
|
|
135
|
+
raise ArgumentError, "NETDISCO_DB_NAME must be a database name, not a connection string"
|
|
136
|
+
end
|
|
137
|
+
options.transform_values { |value| value.is_a?(String) ? value.dup.freeze : value }.freeze
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
def set_statement_timeout(connection, budget)
|
|
141
|
+
milliseconds = [(budget.remaining * 1000).ceil, 2_147_483_647].min
|
|
142
|
+
connection.exec_params("SELECT set_config('statement_timeout', $1, true)", [milliseconds.to_s]).clear
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
def inventory(connection, budget)
|
|
146
|
+
rows = []
|
|
147
|
+
@options.fetch(:max_pages).times do
|
|
148
|
+
set_statement_timeout(connection, budget)
|
|
149
|
+
count = fetch_page(connection, rows, budget)
|
|
150
|
+
return rows if count < @options.fetch(:page_size)
|
|
151
|
+
end
|
|
152
|
+
raise Client::Error.new("Netdisco inventory exceeded max_pages", code: :max_pages), cause: nil
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
# 单行模式避免 libpq 先缓存整页;Ruby 只积累已经通过预算检查的白名单字段。
|
|
156
|
+
def fetch_page(connection, rows, budget)
|
|
157
|
+
connection.send_query_params("FETCH FORWARD #{@options.fetch(:page_size)} FROM net_connector_inventory", [])
|
|
158
|
+
connection.set_single_row_mode
|
|
159
|
+
count = 0
|
|
160
|
+
response_bytes = 0
|
|
161
|
+
while (result = connection.get_result)
|
|
162
|
+
begin
|
|
163
|
+
result.check
|
|
164
|
+
response_bytes = append_rows(result, rows, budget, response_bytes)
|
|
165
|
+
count += result.ntuples
|
|
166
|
+
ensure
|
|
167
|
+
result.clear
|
|
168
|
+
end
|
|
169
|
+
end
|
|
170
|
+
count
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
def append_rows(result, rows, budget, response_bytes)
|
|
174
|
+
fields = result.fields
|
|
175
|
+
unless fields.include?("ip") && fields.uniq == fields
|
|
176
|
+
raise Client::Error.new("Netdisco database query must return unique column names including ip",
|
|
177
|
+
code: :invalid_inventory), cause: nil
|
|
178
|
+
end
|
|
179
|
+
budget.consume_devices(result.ntuples)
|
|
180
|
+
result.each do |row|
|
|
181
|
+
# 未选用的列也占传输预算,但不会进入清单、计划或报告。
|
|
182
|
+
size = row.sum { |key, value| key.bytesize + (value&.bytesize || 0) }
|
|
183
|
+
budget.consume_bytes(size, response_bytes: response_bytes)
|
|
184
|
+
response_bytes += size
|
|
185
|
+
unless row["ip"].is_a?(String) && !row["ip"].strip.empty?
|
|
186
|
+
raise Client::Error.new("Netdisco returned an invalid device inventory", code: :invalid_inventory), cause: nil
|
|
187
|
+
end
|
|
188
|
+
rows << row.slice(*Client::FIELDS)
|
|
189
|
+
end
|
|
190
|
+
response_bytes
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
194
|
+
end
|
|
195
|
+
end
|
|
196
|
+
end
|
|
197
|
+
end
|
|
@@ -63,6 +63,13 @@ module Net
|
|
|
63
63
|
|
|
64
64
|
def client(policy: snapshot(mode: :inventory))
|
|
65
65
|
options = policy.client_options
|
|
66
|
+
if policy.inventory_source == :postgres
|
|
67
|
+
connection = DatabaseClient::CONNECTION_ENV.each_with_object({}) do |(key, name), values|
|
|
68
|
+
item = secret_value(name)
|
|
69
|
+
values[key] = item if item
|
|
70
|
+
end
|
|
71
|
+
return DatabaseClient.new(connection_options: connection, **options)
|
|
72
|
+
end
|
|
66
73
|
raise ArgumentError, "NETDISCO_URL is required" unless options.fetch(:url)
|
|
67
74
|
|
|
68
75
|
authentication = if (token = secret_value("NETDISCO_API_KEY"))
|
|
@@ -73,9 +80,17 @@ module Net
|
|
|
73
80
|
Client.new(**options, **authentication)
|
|
74
81
|
end
|
|
75
82
|
|
|
76
|
-
|
|
83
|
+
def inventory_source
|
|
84
|
+
source = value("NETDISCO_SOURCE") || "http"
|
|
85
|
+
raise ArgumentError, "NETDISCO_SOURCE must be http or postgres" unless %w[http postgres].include?(source)
|
|
86
|
+
|
|
87
|
+
source.to_sym
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
# 原生客户端和 --show-config 共享预算校验;只验证所选来源的查询设置。
|
|
77
91
|
def client_options
|
|
78
|
-
|
|
92
|
+
defaults = inventory_source == :postgres ? DatabaseClient::DEFAULTS : Client::DEFAULTS
|
|
93
|
+
options = defaults.to_h do |name, default|
|
|
79
94
|
key = "NETDISCO_#{name.to_s.upcase}"
|
|
80
95
|
parsed = case name
|
|
81
96
|
when :inventory_timeout then number(key, default)
|
|
@@ -84,6 +99,13 @@ module Net
|
|
|
84
99
|
end
|
|
85
100
|
[name, parsed]
|
|
86
101
|
end
|
|
102
|
+
if inventory_source == :postgres
|
|
103
|
+
query = value("NETDISCO_QUERY") || (raise ArgumentError, "NETDISCO_QUERY is required for postgres")
|
|
104
|
+
return DatabaseClient.options(query: query, query_params: json("NETDISCO_QUERY_PARAMS", []), **options)
|
|
105
|
+
end
|
|
106
|
+
if value("NETDISCO_QUERY") || value("NETDISCO_QUERY_PARAMS")
|
|
107
|
+
raise ArgumentError, "SQL query settings require NETDISCO_SOURCE=postgres"
|
|
108
|
+
end
|
|
87
109
|
options = Client.options(**options)
|
|
88
110
|
url = value("NETDISCO_URL")
|
|
89
111
|
Client.validate_url!(url, allow_insecure_http: options.fetch(:allow_insecure_http)) if url
|
|
@@ -189,7 +211,7 @@ module Net
|
|
|
189
211
|
|
|
190
212
|
def config_hash
|
|
191
213
|
{
|
|
192
|
-
netdisco: client_options,
|
|
214
|
+
netdisco: client_options.merge(source: inventory_source),
|
|
193
215
|
backup: { directory: backup_directory, concurrency: concurrency, limit_per_vendor: limit_per_vendor },
|
|
194
216
|
inventory: { include_hosts: list("NET_CONNECTOR_INCLUDE_HOSTS"),
|
|
195
217
|
exclude_hosts: list("NET_CONNECTOR_EXCLUDE_HOSTS"),
|
|
@@ -279,6 +301,7 @@ module Net
|
|
|
279
301
|
end
|
|
280
302
|
|
|
281
303
|
def required_secret(key) = secret_value(key) || (raise ArgumentError, "#{key} is required")
|
|
304
|
+
|
|
282
305
|
def present?(value) = value.is_a?(String) && !value.strip.empty?
|
|
283
306
|
end
|
|
284
307
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: net-connector
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- net-connector contributors
|
|
@@ -37,6 +37,26 @@ dependencies:
|
|
|
37
37
|
- - "~>"
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
39
|
version: 0.2.0
|
|
40
|
+
- !ruby/object:Gem::Dependency
|
|
41
|
+
name: pg
|
|
42
|
+
requirement: !ruby/object:Gem::Requirement
|
|
43
|
+
requirements:
|
|
44
|
+
- - ">="
|
|
45
|
+
- !ruby/object:Gem::Version
|
|
46
|
+
version: '1.5'
|
|
47
|
+
- - "<"
|
|
48
|
+
- !ruby/object:Gem::Version
|
|
49
|
+
version: '2.0'
|
|
50
|
+
type: :runtime
|
|
51
|
+
prerelease: false
|
|
52
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
53
|
+
requirements:
|
|
54
|
+
- - ">="
|
|
55
|
+
- !ruby/object:Gem::Version
|
|
56
|
+
version: '1.5'
|
|
57
|
+
- - "<"
|
|
58
|
+
- !ruby/object:Gem::Version
|
|
59
|
+
version: '2.0'
|
|
40
60
|
- !ruby/object:Gem::Dependency
|
|
41
61
|
name: digest
|
|
42
62
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -407,6 +427,7 @@ files:
|
|
|
407
427
|
- docs/RELEASING.md
|
|
408
428
|
- docs/VERIFICATION.md
|
|
409
429
|
- docs/architecture.md
|
|
430
|
+
- examples/netdisco_database.yml
|
|
410
431
|
- exe/net-connector-backup
|
|
411
432
|
- lib/net-connector.rb
|
|
412
433
|
- lib/net/connector.rb
|
|
@@ -451,6 +472,7 @@ files:
|
|
|
451
472
|
- lib/net/connector/netdisco/cli.rb
|
|
452
473
|
- lib/net/connector/netdisco/client.rb
|
|
453
474
|
- lib/net/connector/netdisco/config_file.rb
|
|
475
|
+
- lib/net/connector/netdisco/database_client.rb
|
|
454
476
|
- lib/net/connector/netdisco/device.rb
|
|
455
477
|
- lib/net/connector/netdisco/diagnostic.rb
|
|
456
478
|
- lib/net/connector/netdisco/fleet.rb
|