net-connector 0.5.0 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fdd886424e85114a4773396098117ed0e89ddbfe4e4ad3b1dc43dfc003dbc589
4
- data.tar.gz: f29240ec971ecddb62f9d8d31bd45db027ae0e799c5c14872e8dfdf55ad2f0be
3
+ metadata.gz: b9f2c5dfacb73a1c1aecf350bd4d1fe93755e0be25dbd1cc5c964a5676beaee7
4
+ data.tar.gz: 3afcf973ef1a25d848b1a91feb5803348eea6881f9d4b2721525378f9611d0d9
5
5
  SHA512:
6
- metadata.gz: 97f0d16f83f74c24caf69d3beaa1dbb0c01b3fed6eb58a1609c385eb48eba9acfaa9bb47250d18d91274b06412aada14d1baed6b8e4531603987a4a84505657d
7
- data.tar.gz: 343cf7298e695ea38fab06a4e838fbb643de8393a4c9ed87ddaee77401eccd5aeeaead086cb9408cb5321a861be423e43b525488a5d06d9014ced050fa3942bd
6
+ metadata.gz: 5b168ca118a7323f1b698908b7e6db2e627c97b52f57a4817f8b932456862ba9097491f4f879b23ffcdd9d8ea1c6f287b870354a9c279cf04963295b9cec815f
7
+ data.tar.gz: 1d5cdcedba90facbc57a9a33438c66a244c482ea580399e901f9b42572a5b6401f0cbb68f01e1bf106eb53e0562f4f213742de1cc2dc9dff80f0bae47697eb81
data/CHANGELOG.md CHANGED
@@ -1,5 +1,11 @@
1
1
  # 更新记录
2
2
 
3
+ ## 0.6.0 - 2026-09-27
4
+
5
+ - 支持直接查询 PostgreSQL 获取 Netdisco 设备清单,SQL 和绑定参数可通过 YAML、环境变量或 CLI 设置,查询结果继续使用已有的厂商映射、计划预览和批量备份流程。
6
+ - 数据库连接信息只从 `NETDISCO_DB_*` 环境变量读取,查询在只读事务中执行并限制时间和容量,失败时停止整批任务,错误输出不包含数据库凭据或原始响应。
7
+ - 增加可编辑的数据库查询示例并随 gem 分发,安装时包含所需的 `pg` 驱动,原有 HTTP 清单及离线配置导出保持可用。
8
+
3
9
  ## 0.5.0 - 2026-09-27
4
10
 
5
11
  本版包含不兼容的接口调整,不保留旧路径或方法别名;升级时请按[当前接口说明](https://github.com/gatework/net-connector/blob/v0.5.0/docs/architecture.md#当前命名与接口调整)更新调用方。
data/README.md CHANGED
@@ -274,6 +274,47 @@ net-connector-backup --config config.yml --tftp --all
274
274
  net-connector-backup --config config.yml --export 192.0.2.7 --output ./exports/device.cfg
275
275
  ```
276
276
 
277
+ ### PostgreSQL 联机查询
278
+
279
+ 设置 `netdisco.source: postgres` 可以直接从数据库查询清单,继续使用同一套 Fleet、规则、计划和备份流程。库不内置表名、SQL 或业务筛选条件;必须提供查询,可直接修改 [YAML 示例](examples/netdisco_database.yml):
280
+
281
+ ```yaml
282
+ netdisco:
283
+ source: postgres
284
+ query: |
285
+ SELECT host(ip) AS ip, name, dns, vendor, os, model, os_ver, serial
286
+ FROM device
287
+ WHERE vendor = $1
288
+ ORDER BY ip
289
+ query_params: [H3C]
290
+ ```
291
+
292
+ 查询必须返回唯一命名的 `ip` 列;可选列为 `name`、`dns`、`vendor`、`os`、`model`、`os_ver`、`serial`,其他列会丢弃。自定义表、视图、JOIN 和只读 CTE 均可用列别名适配这一契约。PostgreSQL 的 `inet` 字段建议用 `host(ip) AS ip` 去除前缀长度;筛选及排序由 SQL 决定。参数按数组顺序绑定 `$1`、`$2`,支持字符串、数字、布尔和 null,不做字符串插值或环境变量展开。
293
+
294
+ 数据库连接信息只从进程环境读取,沿用 [Netdisco 官方环境变量命名](https://github.com/netdisco/netdisco/wiki/Environment-Variables)。`HOST`、`NAME`、`USER`、`PASS` 必填;HTTP 地址、API key 和设备登录凭据不是清单查询的前置条件:
295
+
296
+ ```sh
297
+ export NETDISCO_DB_HOST=database.example
298
+ export NETDISCO_DB_NAME=netdisco
299
+ export NETDISCO_DB_USER=inventory-reader
300
+ export NETDISCO_DB_PASS='replace-me'
301
+ export NETDISCO_DB_SSLMODE=verify-full
302
+ export NETDISCO_DB_SSLROOTCERT=/etc/net-connector/database-ca.crt
303
+
304
+ net-connector-backup --config examples/netdisco_database.yml --show-config
305
+ net-connector-backup --config examples/netdisco_database.yml --plan
306
+ # 覆盖查询参数;实际备份仍需设置 NET_CONNECTOR_DEVICE_* 凭据。
307
+ net-connector-backup --config examples/netdisco_database.yml --plan --query-params '["Cisco"]'
308
+ ```
309
+
310
+ SQL 也可通过 `NETDISCO_QUERY` / `--query SQL` 提供,参数通过 `NETDISCO_QUERY_PARAMS` / `--query-params JSON` 提供;来源对应 `NETDISCO_SOURCE` / `--source postgres`。优先级均为 CLI > ENV > YAML。SQL 和参数属于可公开配置,会出现在 `--show-config` 中;数据库密码只放在连接环境变量中。连接信息不进入策略快照、计划、报告或 `inspect`。已有 Fleet 每次重新查询时读取最新连接凭据;传入已有 `plan:` 执行时不会重新查询。
311
+
312
+ 客户端通过 `pg` 驱动执行只读事务,使用参数化游标分批取数,并在每批启用单行读取。PostgreSQL 原生解析拒绝多条语句,写入和锁定查询会失败;查询账户应仅授予所需表/视图的 SELECT 权限,只读事务不能替代账户权限隔离。驱动只在实际查询时加载,HTTP 和离线导出路径不加载它。
313
+
314
+ 数据库查询复用现有清单预算:`page_size` 控制每次 FETCH 的行数,`max_pages` 限制 FETCH 次数,`max_devices` 在去重前计数;字节预算统计返回列名与文本值,包含最终丢弃的列。字节检查发生在 libpq 解码一行之后,不能限制单个超大字段在驱动内部的瞬时内存。连接、查询和全部 FETCH 共用总期限,同时设置数据库 statement_timeout。任一查询错误、超时或超限均关闭连接并丢弃整份清单,不连接设备;错误消息不输出原始 SQL、参数或数据库响应。建议只选所需字段,并提供明确的 ORDER BY 保持采样顺序稳定。
315
+
316
+ ### 计划与执行
317
+
277
318
  `--plan` 只拉取并验证清单;`--host` 选择一个管理地址;`--tftp` 默认每厂商最多选择五台,`--all` 选择所有就绪设备。本地备份默认选择所有就绪设备,可用 `--limit-per-vendor` 限制。`--show-config` 只输出有效的非敏感设置,不访问 Netdisco。CLI 会拒绝未知 YAML 字段、Ruby 对象标签及配置中的凭据。`--export IP` 只离线读取规范化管理地址对应的 `<IP>.txt`;默认原样写到标准输出,指定 `--output` 后以 `0600` 权限原子写文件。导出的配置仍是敏感数据。
278
319
 
279
320
  CLI 的计划与批次摘要使用 JSON。默认 `--success-policy strict` 使用严格规则:非空清单且全部成功、回调及报告正常时为 `0`;空清单或有跳过、部分成功、失败时为 `1`;清单或配置错误为 `2`。`--host` 未在清单中找到也返回 `2`。由于其他清单记录会标记为过滤,默认单主机备份成功时批次退出码仍可能是 `1`;应查看 JSON 中的 `succeeded`、`skipped` 和逐台 `status`。
@@ -291,10 +332,18 @@ CLI 的计划与批次摘要使用 JSON。默认 `--success-policy strict` 使
291
332
 
292
333
  | 环境变量 | 默认值 | 用途 |
293
334
  | --- | --- | --- |
294
- | `NETDISCO_URL` | 必填 | Netdisco 服务根地址,可包含租户路径 |
335
+ | `NETDISCO_SOURCE` | `http` | 清单来源,支持 `http`、`postgres` |
336
+ | `NETDISCO_URL` | HTTP 模式必填 | Netdisco 服务根地址,可包含租户路径 |
295
337
  | `NET_CONNECTOR_CONFIG` | 未设置 | CLI 的非敏感 YAML 配置文件 |
296
338
  | `NETDISCO_USERNAME`, `NETDISCO_PASSWORD` | 未提供 API 密钥时必填 | 清单 API 登录 |
297
339
  | `NETDISCO_API_KEY` | 未设置 | 直接使用已有 API 密钥 |
340
+ | `NETDISCO_QUERY` | PostgreSQL 模式必填 | 用户提供的单条清单 SQL |
341
+ | `NETDISCO_QUERY_PARAMS` | `[]` | SQL 参数的 JSON 标量数组 |
342
+ | `NETDISCO_DB_HOST`, `NETDISCO_DB_NAME` | PostgreSQL 模式必填 | 数据库主机或 Unix socket 目录、数据库名 |
343
+ | `NETDISCO_DB_USER`, `NETDISCO_DB_PASS` | PostgreSQL 模式必填 | 仅从环境注入的数据库用户名、密码 |
344
+ | `NETDISCO_DB_PORT` | libpq 默认 `5432` | PostgreSQL 端口 |
345
+ | `NETDISCO_DB_SSLMODE`, `NETDISCO_DB_SSLROOTCERT` | libpq 默认 | TLS 模式、CA 文件;远程连接建议 `verify-full` |
346
+ | `NETDISCO_DB_CONNECT_TIMEOUT` | 未单独设置 | 可选正整数秒;连接始终受清单总期限限制 |
298
347
  | `NETDISCO_PAGE_SIZE` | `500` | 清单分页大小 |
299
348
  | `NETDISCO_MAX_PAGES` | `10000` | 最大分页次数 |
300
349
  | `NETDISCO_MAX_RESPONSE_BYTES` | `16777216` | 单次响应正文上限,认证和错误正文也计数 |
data/docs/architecture.md CHANGED
@@ -112,6 +112,12 @@ Diagnostic 只保存固定词表中的码、类型、阶段和受控产物状态
112
112
 
113
113
  注入的 `requester` 接收 URI 和 request。它返回后才接受长度和期限检查,不在任意用户 Ruby 回调中注入异步异常。解析和集合操作完成后也检查期限,但这不构成任意 CPU 回调的抢占保证。预算失败仅报告错误码与安全类型,没有部分清单、响应正文或底层 cause。HTTPS 的标准证书检查不变;默认允许明文 HTTP,可通过显式策略禁止。
114
114
 
115
+ `Settings#inventory_source` 选择 HTTP `Client` 或 PostgreSQL `DatabaseClient`,两者只向 Fleet 提供 `devices`,不改变设备映射、计划和执行接口。数据库 SQL 与标量参数由用户配置,不内置业务表名或筛选;结果用列别名适配 `Client::FIELDS`,只保留这份现有字段白名单。连接参数由 `DatabaseClient::CONNECTION_ENV` 集中映射到 Netdisco 的 `NETDISCO_DB_*` 环境变量,全部排除在纯策略快照外;每次新建清单客户端时读取最新值。
116
+
117
+ `DatabaseClient` 每次调用独占 PostgreSQL 连接,在只读事务中用扩展查询协议声明游标,原生解析器拒绝多语句和非查询输入。FETCH 大小及次数复用清单预算,单行模式使 libpq 不缓存整页;每行在追加前计数,额外列也消耗字节预算。驱动必须先解码单行,所以单个超大字段仍可超过 Ruby 预算的瞬时内存。总 deadline 覆盖连接和所有语句,每次 FETCH 前缩短 statement_timeout;正常完成回滚只读事务,异常关闭连接,任何失败均不交付部分清单。NOTICE、原始数据库异常及 cause 不进入日志或 CLI;稳定错误码区分连接失败、查询失败、无效清单和预算超限。SQL/参数可在 show-config 中查看,不能用于传递凭据。
118
+
119
+ 数据库集成测试运行 `bundle exec rake test:postgres`,通过 `pg_config --bindir`(或 `NET_CONNECTOR_TEST_PG_BINDIR`)定位服务端工具。测试只创建临时 SCRAM 数据库和私有 Unix socket,退出时停止并删除;不会读取真实 Netdisco 凭据或连接已有服务。覆盖 SQL/参数、只读限制、真实认证失败、查询中途失败、各项预算、连接回收以及 CLI 计划。
120
+
115
121
  ## 迁移与尚未启用的能力
116
122
 
117
123
  | 变化 | 调用方需要保留的约定 |
@@ -0,0 +1,16 @@
1
+ # 数据库连接只从 NETDISCO_DB_* 环境变量读取;此文件可直接编辑并纳入版本控制。
2
+ netdisco:
3
+ source: postgres
4
+ query: |
5
+ SELECT host(ip) AS ip, name, dns, vendor, os, model, os_ver, serial
6
+ FROM device
7
+ WHERE vendor = $1
8
+ ORDER BY ip
9
+ query_params: [H3C]
10
+ page_size: 500
11
+ max_devices: 100000
12
+ inventory_timeout: 300
13
+
14
+ backup:
15
+ directory: ./backups
16
+ concurrency: 4
@@ -62,7 +62,7 @@ module Net
62
62
  host_key = Interaction.new(SSH_CONFIRMATION, lambda { |_prompt|
63
63
  if @config.host_key_policy == :strict
64
64
  raise @session.build_error(ConnectionError, "host key confirmation rejected",
65
- phase: :login, code: :host_key_untrusted), cause: nil
65
+ phase: :login, code: :host_key_untrusted), cause: nil
66
66
  end
67
67
  "yes\n"
68
68
  }, limit: 1)
@@ -172,7 +172,7 @@ module Net
172
172
  def validate_event!(event, bytes, raw, phase, command)
173
173
  if bytes > @session.configuration.max_output_bytes
174
174
  raise @session.build_error(OutputLimitExceeded, "device output exceeded max_output_bytes",
175
- phase: phase, command: command, output: (raw + [event.match]).join), cause: nil
175
+ phase: phase, command: command, output: (raw + [event.match]).join), cause: nil
176
176
  end
177
177
  raise build_read_error(event, phase, command, raw.join), cause: nil unless event.matched?
178
178
  end
@@ -181,7 +181,7 @@ module Net
181
181
  def finish_response(prompt, raw, output, phase, command)
182
182
  if prompt.empty?
183
183
  raise @session.build_error(PromptError, "prompt pattern did not consume any output",
184
- phase: phase, command: command, output: raw.join), cause: nil
184
+ phase: phase, command: command, output: raw.join), cause: nil
185
185
  end
186
186
 
187
187
  raw << prompt
@@ -229,7 +229,7 @@ module Net
229
229
  def build_read_error(event, phase, command, output)
230
230
  if event&.error.is_a?(Exception)
231
231
  return @session.build_error(TransportError, "transport read failed", phase: phase, command: command,
232
- underlying: event.error, output: output)
232
+ underlying: event.error, output: output)
233
233
  end
234
234
  klass = if event&.error == :eof
235
235
  ConnectionClosed
@@ -241,7 +241,7 @@ module Net
241
241
  CommandTimeout
242
242
  end
243
243
  @session.build_error(klass, "device response #{event&.error || :timeout}", phase: phase,
244
- command: command, output: output)
244
+ command: command, output: output)
245
245
  end
246
246
  end
247
247
  end
@@ -90,8 +90,8 @@ module Net
90
90
  return unless @output_limit && (completed ? @output_bytes > @output_limit : @output_bytes >= @output_limit)
91
91
 
92
92
  raise @session.build_error(ScriptOutputLimitExceeded,
93
- "script output reached max_script_output_bytes; commands already sent may have executed",
94
- phase: :script, command: command), cause: nil
93
+ "script output reached max_script_output_bytes; commands already sent may have executed",
94
+ phase: :script, command: command), cause: nil
95
95
  end
96
96
  end
97
97
  end
@@ -350,7 +350,7 @@ module Net
350
350
  redactor.reset
351
351
  @log.open(transport)
352
352
  @log.log_event("connect", host: configuration.host, username: configuration.username,
353
- phase: :connect, protocol: transport.respond_to?(:protocol) ? transport.protocol : configuration.protocol)
353
+ phase: :connect, protocol: transport.respond_to?(:protocol) ? transport.protocol : configuration.protocol)
354
354
  transport.open
355
355
  @state = :authenticating
356
356
  @log.log_event("login_start", level: :debug, phase: :login)
@@ -69,6 +69,9 @@ module Net
69
69
  def settings_for(options)
70
70
  defaults = options[:config] ? ConfigFile.load(options[:config]) : {}
71
71
  values = {}
72
+ values["NETDISCO_SOURCE"] = options[:source] if options[:source]
73
+ values["NETDISCO_QUERY"] = options[:query] if options[:query]
74
+ values["NETDISCO_QUERY_PARAMS"] = options[:query_params] if options[:query_params]
72
75
  values["NET_CONNECTOR_BACKUP_DIRECTORY"] = options[:directory] if options[:directory]
73
76
  values["NET_CONNECTOR_CONCURRENCY"] = options[:concurrency].to_s if options[:concurrency]
74
77
  values["NET_CONNECTOR_MAX_SCRIPT_OUTPUT_BYTES"] = options[:max_script_output_bytes].to_s if options[:max_script_output_bytes]
@@ -81,7 +84,7 @@ module Net
81
84
  # 导出已有配置不创建清单或设备连接。
82
85
  def export_config(settings, options)
83
86
  path = Storage::SavedConfig.new(directory: settings.backup_directory)
84
- .export(host: options[:export], output: options[:output], io: @output)
87
+ .export(host: options[:export], output: options[:output], io: @output)
85
88
  @output.puts JSON.generate(host: options[:export], output: path) if path
86
89
  end
87
90
 
@@ -120,6 +123,9 @@ module Net
120
123
  parser = OptionParser.new do |args|
121
124
  args.banner = "用法:net-connector-backup [--config FILE] [--plan | --show-config | --export IP] [--tftp]"
122
125
  args.on("--config FILE", "读取不含凭据的 YAML 设置") { |value| options[:config] = value }
126
+ args.on("--source SOURCE", %w[http postgres], "清单来源:http(默认)或 postgres") { |value| options[:source] = value }
127
+ args.on("--query SQL", "PostgreSQL 清单查询,使用 $1、$2 绑定参数") { |value| options[:query] = value }
128
+ args.on("--query-params JSON", "查询参数的 JSON 数组") { |value| options[:query_params] = value }
123
129
  args.on("--show-config", "显示生效的非敏感设置") { options[:show_config] = true }
124
130
  args.on("--plan", "预览设备清单,不连接设备") { options[:plan] = true }
125
131
  args.on("--tftp", "由设备发起 TFTP 备份") { options[:tftp] = true }
@@ -158,8 +164,11 @@ module Net
158
164
  raise ArgumentError, "--show-config、--plan 和 --export 只能选择一项" if modes > 1
159
165
  raise ArgumentError, "--output 需要同时指定 --export" if options[:output] && !options[:export]
160
166
  raise ArgumentError, "--all 不能与 --limit-per-vendor 同时使用" if options[:all] && options[:limit]
167
+ if options[:export] && (options[:source] || options[:query] || options[:query_params])
168
+ raise ArgumentError, "--export 不能与清单查询选项同时使用"
169
+ end
161
170
  if options[:export] && (options[:tftp] || options[:all] || options[:limit] || options[:concurrency] || options[:host] ||
162
- options[:success_policy] || options[:max_script_output_bytes])
171
+ options[:success_policy] || options[:max_script_output_bytes])
163
172
  raise ArgumentError, "--export 不能与备份选项同时使用"
164
173
  end
165
174
  end
@@ -96,7 +96,7 @@ module Net
96
96
  @options.fetch(:max_pages).times do |index|
97
97
  uri = endpoint("api/v1/search/device")
98
98
  uri.query = URI.encode_www_form(fields: FIELDS.join(","), limit: @options.fetch(:page_size),
99
- offset: index * @options.fetch(:page_size))
99
+ offset: index * @options.fetch(:page_size))
100
100
  page = request_json(uri, authorized_get(uri, token), budget, allow_empty: true, query_fallback: true)
101
101
  validate_rows!(page, budget)
102
102
  return rows.uniq if page.empty?
@@ -274,6 +274,7 @@ module Net
274
274
  end
275
275
 
276
276
  def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
277
+
277
278
  def present?(value) = value.is_a?(String) && !value.strip.empty?
278
279
  end
279
280
  end
@@ -9,7 +9,10 @@ module Net
9
9
  # 命令行使用的无凭据 YAML 配置,环境变量优先。
10
10
  class ConfigFile
11
11
  FIELDS = {
12
- "netdisco" => { "url" => ["NETDISCO_URL", :string],
12
+ "netdisco" => { "source" => ["NETDISCO_SOURCE", :string],
13
+ "query" => ["NETDISCO_QUERY", :string],
14
+ "query_params" => ["NETDISCO_QUERY_PARAMS", :json_array],
15
+ "url" => ["NETDISCO_URL", :string],
13
16
  "page_size" => ["NETDISCO_PAGE_SIZE", :integer],
14
17
  "max_pages" => ["NETDISCO_MAX_PAGES", :integer],
15
18
  "max_response_bytes" => ["NETDISCO_MAX_RESPONSE_BYTES", :integer],
@@ -94,8 +97,8 @@ module Net
94
97
  end
95
98
 
96
99
  value.join(",")
97
- when :json
98
- expected = name == "device_rules" ? Array : Hash
100
+ when :json, :json_array
101
+ expected = type == :json_array || name == "device_rules" ? Array : Hash
99
102
  raise ArgumentError, "#{section}.#{name} must be a #{expected}" unless value.is_a?(expected)
100
103
 
101
104
  JSON.generate(value)
@@ -0,0 +1,197 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "timeout"
4
+ require_relative "client"
5
+
6
+ module Net
7
+ module Connector
8
+ module Netdisco
9
+ # SQL 负责业务选择和列别名;适配器只管理只读查询、清单预算与连接生命周期。
10
+ class DatabaseClient
11
+ CONNECTION_ENV = {
12
+ host: "NETDISCO_DB_HOST", port: "NETDISCO_DB_PORT", dbname: "NETDISCO_DB_NAME",
13
+ user: "NETDISCO_DB_USER", password: "NETDISCO_DB_PASS",
14
+ sslmode: "NETDISCO_DB_SSLMODE", sslrootcert: "NETDISCO_DB_SSLROOTCERT",
15
+ connect_timeout: "NETDISCO_DB_CONNECT_TIMEOUT"
16
+ }.freeze
17
+ REQUIRED_CONNECTION_KEYS = %i[host dbname user password].freeze
18
+ DEFAULTS = Client::DEFAULTS.except(:allow_insecure_http).freeze
19
+
20
+ def self.options(query:, query_params: [], **values)
21
+ unless query.is_a?(String) && !query.strip.empty? && !query.include?("\0")
22
+ raise ArgumentError, "query must be a nonempty SQL string without NUL"
23
+ end
24
+ unless query_params.is_a?(Array) && query_params.all? { |value|
25
+ value.nil? || value == true || value == false || value.is_a?(Integer) ||
26
+ (value.is_a?(Float) && value.finite?) || (value.is_a?(String) && !value.include?("\0"))
27
+ }
28
+ raise ArgumentError, "query_params must be an array of JSON scalar values without NUL"
29
+ end
30
+ raise ArgumentError, "unknown database inventory options" unless (values.keys - DEFAULTS.keys).empty?
31
+
32
+ Client.options(**values).slice(*DEFAULTS.keys).merge(
33
+ query: query.dup.freeze,
34
+ query_params: query_params.map { |value| value.is_a?(String) ? value.dup.freeze : value }.freeze
35
+ ).freeze
36
+ end
37
+
38
+ def initialize(connection_options:, **options)
39
+ @options = self.class.options(**options)
40
+ @connection_options = validate_connection_options(connection_options)
41
+ end
42
+
43
+ # 每次查询独占连接和游标;只有全部结果验证成功后才把清单交给 Fleet。
44
+ def devices
45
+ require "pg"
46
+ budget = InventoryBudget.new(@options, clock: method(:monotonic))
47
+ connection = nil
48
+ connected = false
49
+ begin
50
+ Timeout.timeout(budget.remaining, Client::InventoryTimeout) do
51
+ # 从握手开始持有连接;PG.connect 尚未返回时被超时中断,调用方无法负责关闭它。
52
+ connection = PG::Connection.connect_start(@connection_options)
53
+ # PostgreSQL NOTICE 可能包含 SQL 或数据,不交给 libpq 默认的 stderr 输出器。
54
+ connection.set_notice_processor { |_notice| nil }
55
+ wait_for_connection(connection, budget)
56
+ connected = true
57
+ # 使用 pg 可中断的 Ruby 等待接口;底层 socket 仍以非阻塞方式工作。
58
+ connection.setnonblocking(false)
59
+ connection.set_client_encoding("UTF8")
60
+ connection.exec("BEGIN READ ONLY").clear
61
+ set_statement_timeout(connection, budget)
62
+ # 即使没有参数也走扩展查询协议,由 PostgreSQL 拒绝多语句和非查询命令。
63
+ connection.exec_params("DECLARE net_connector_inventory NO SCROLL CURSOR FOR\n#{@options.fetch(:query)}",
64
+ @options.fetch(:query_params)).clear
65
+ rows = inventory(connection, budget)
66
+ connection.exec("ROLLBACK").clear
67
+ budget.remaining
68
+ rows
69
+ end
70
+ rescue Client::Error => error
71
+ raise error, cause: nil
72
+ rescue PG::QueryCanceled
73
+ raise Client::InventoryTimeout, cause: nil
74
+ rescue PG::Error, IOError, SystemCallError
75
+ code = connected ? :query_failed : :connection_failed
76
+ raise Client::Error.new("Netdisco database #{code.to_s.tr("_", " ")}", code: code), cause: nil
77
+ ensure
78
+ connection&.finish
79
+ end
80
+ end
81
+
82
+ def inspect = "#<#{self.class}>"
83
+
84
+ private
85
+
86
+ def wait_for_connection(connection, budget)
87
+ deadline = monotonic + @connection_options.fetch(:connect_timeout, budget.remaining).to_f
88
+ loop do
89
+ status = connection.connect_poll
90
+ return if status == PG::PGRES_POLLING_OK
91
+ raise PG::ConnectionBad unless [PG::PGRES_POLLING_READING, PG::PGRES_POLLING_WRITING].include?(status)
92
+
93
+ remaining = [budget.remaining, deadline - monotonic].min
94
+ raise PG::ConnectionBad if remaining <= 0
95
+
96
+ socket = connection.socket_io
97
+ readers = status == PG::PGRES_POLLING_READING ? [socket] : nil
98
+ writers = status == PG::PGRES_POLLING_WRITING ? [socket] : nil
99
+ next if IO.select(readers, writers, [socket], remaining)
100
+
101
+ budget.remaining
102
+ raise PG::ConnectionBad
103
+ end
104
+ end
105
+
106
+ def validate_connection_options(options)
107
+ unless options.is_a?(Hash) && (options.keys - CONNECTION_ENV.keys).empty?
108
+ raise ArgumentError, "connection_options must contain supported PostgreSQL connection keys"
109
+ end
110
+ REQUIRED_CONNECTION_KEYS.each do |key|
111
+ value = options[key]
112
+ unless value.is_a?(String) && !value.strip.empty?
113
+ raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} is required"
114
+ end
115
+ end
116
+ options.each do |key, value|
117
+ valid = value.is_a?(String) && !value.empty? && !value.include?("\0")
118
+ valid ||= %i[port connect_timeout].include?(key) && value.is_a?(Integer)
119
+ raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} is invalid" unless valid
120
+ end
121
+ %i[port connect_timeout].each do |key|
122
+ next unless options.key?(key)
123
+
124
+ value = options.fetch(key).to_s
125
+ maximum = key == :port ? 65_535 : 2_147_483_647
126
+ unless value.match?(/\A\d+\z/) && value.to_i.between?(1, maximum)
127
+ raise ArgumentError, "#{CONNECTION_ENV.fetch(key)} must be a positive integer within range"
128
+ end
129
+ end
130
+ if options[:sslmode] && !%w[disable allow prefer require verify-ca verify-full].include?(options[:sslmode])
131
+ raise ArgumentError, "NETDISCO_DB_SSLMODE is invalid"
132
+ end
133
+ # libpq 会把 dbname 中的 URI/conninfo 再解释为连接参数,不能借此绕过凭据来源。
134
+ if options.fetch(:dbname).match?(/=|\Apostgres(?:ql)?:\/\//)
135
+ raise ArgumentError, "NETDISCO_DB_NAME must be a database name, not a connection string"
136
+ end
137
+ options.transform_values { |value| value.is_a?(String) ? value.dup.freeze : value }.freeze
138
+ end
139
+
140
+ def set_statement_timeout(connection, budget)
141
+ milliseconds = [(budget.remaining * 1000).ceil, 2_147_483_647].min
142
+ connection.exec_params("SELECT set_config('statement_timeout', $1, true)", [milliseconds.to_s]).clear
143
+ end
144
+
145
+ def inventory(connection, budget)
146
+ rows = []
147
+ @options.fetch(:max_pages).times do
148
+ set_statement_timeout(connection, budget)
149
+ count = fetch_page(connection, rows, budget)
150
+ return rows if count < @options.fetch(:page_size)
151
+ end
152
+ raise Client::Error.new("Netdisco inventory exceeded max_pages", code: :max_pages), cause: nil
153
+ end
154
+
155
+ # 单行模式避免 libpq 先缓存整页;Ruby 只积累已经通过预算检查的白名单字段。
156
+ def fetch_page(connection, rows, budget)
157
+ connection.send_query_params("FETCH FORWARD #{@options.fetch(:page_size)} FROM net_connector_inventory", [])
158
+ connection.set_single_row_mode
159
+ count = 0
160
+ response_bytes = 0
161
+ while (result = connection.get_result)
162
+ begin
163
+ result.check
164
+ response_bytes = append_rows(result, rows, budget, response_bytes)
165
+ count += result.ntuples
166
+ ensure
167
+ result.clear
168
+ end
169
+ end
170
+ count
171
+ end
172
+
173
+ def append_rows(result, rows, budget, response_bytes)
174
+ fields = result.fields
175
+ unless fields.include?("ip") && fields.uniq == fields
176
+ raise Client::Error.new("Netdisco database query must return unique column names including ip",
177
+ code: :invalid_inventory), cause: nil
178
+ end
179
+ budget.consume_devices(result.ntuples)
180
+ result.each do |row|
181
+ # 未选用的列也占传输预算,但不会进入清单、计划或报告。
182
+ size = row.sum { |key, value| key.bytesize + (value&.bytesize || 0) }
183
+ budget.consume_bytes(size, response_bytes: response_bytes)
184
+ response_bytes += size
185
+ unless row["ip"].is_a?(String) && !row["ip"].strip.empty?
186
+ raise Client::Error.new("Netdisco returned an invalid device inventory", code: :invalid_inventory), cause: nil
187
+ end
188
+ rows << row.slice(*Client::FIELDS)
189
+ end
190
+ response_bytes
191
+ end
192
+
193
+ def monotonic = Process.clock_gettime(Process::CLOCK_MONOTONIC)
194
+ end
195
+ end
196
+ end
197
+ end
@@ -63,6 +63,13 @@ module Net
63
63
 
64
64
  def client(policy: snapshot(mode: :inventory))
65
65
  options = policy.client_options
66
+ if policy.inventory_source == :postgres
67
+ connection = DatabaseClient::CONNECTION_ENV.each_with_object({}) do |(key, name), values|
68
+ item = secret_value(name)
69
+ values[key] = item if item
70
+ end
71
+ return DatabaseClient.new(connection_options: connection, **options)
72
+ end
66
73
  raise ArgumentError, "NETDISCO_URL is required" unless options.fetch(:url)
67
74
 
68
75
  authentication = if (token = secret_value("NETDISCO_API_KEY"))
@@ -73,9 +80,17 @@ module Net
73
80
  Client.new(**options, **authentication)
74
81
  end
75
82
 
76
- # 原生客户端和 --show-config 共享同一预算、地址与 HTTP 策略校验。
83
+ def inventory_source
84
+ source = value("NETDISCO_SOURCE") || "http"
85
+ raise ArgumentError, "NETDISCO_SOURCE must be http or postgres" unless %w[http postgres].include?(source)
86
+
87
+ source.to_sym
88
+ end
89
+
90
+ # 原生客户端和 --show-config 共享预算校验;只验证所选来源的查询设置。
77
91
  def client_options
78
- options = Client::DEFAULTS.to_h do |name, default|
92
+ defaults = inventory_source == :postgres ? DatabaseClient::DEFAULTS : Client::DEFAULTS
93
+ options = defaults.to_h do |name, default|
79
94
  key = "NETDISCO_#{name.to_s.upcase}"
80
95
  parsed = case name
81
96
  when :inventory_timeout then number(key, default)
@@ -84,6 +99,13 @@ module Net
84
99
  end
85
100
  [name, parsed]
86
101
  end
102
+ if inventory_source == :postgres
103
+ query = value("NETDISCO_QUERY") || (raise ArgumentError, "NETDISCO_QUERY is required for postgres")
104
+ return DatabaseClient.options(query: query, query_params: json("NETDISCO_QUERY_PARAMS", []), **options)
105
+ end
106
+ if value("NETDISCO_QUERY") || value("NETDISCO_QUERY_PARAMS")
107
+ raise ArgumentError, "SQL query settings require NETDISCO_SOURCE=postgres"
108
+ end
87
109
  options = Client.options(**options)
88
110
  url = value("NETDISCO_URL")
89
111
  Client.validate_url!(url, allow_insecure_http: options.fetch(:allow_insecure_http)) if url
@@ -189,7 +211,7 @@ module Net
189
211
 
190
212
  def config_hash
191
213
  {
192
- netdisco: client_options,
214
+ netdisco: client_options.merge(source: inventory_source),
193
215
  backup: { directory: backup_directory, concurrency: concurrency, limit_per_vendor: limit_per_vendor },
194
216
  inventory: { include_hosts: list("NET_CONNECTOR_INCLUDE_HOSTS"),
195
217
  exclude_hosts: list("NET_CONNECTOR_EXCLUDE_HOSTS"),
@@ -279,6 +301,7 @@ module Net
279
301
  end
280
302
 
281
303
  def required_secret(key) = secret_value(key) || (raise ArgumentError, "#{key} is required")
304
+
282
305
  def present?(value) = value.is_a?(String) && !value.strip.empty?
283
306
  end
284
307
  end
@@ -2,6 +2,7 @@
2
2
 
3
3
  require_relative "../connector"
4
4
  require_relative "netdisco/client"
5
+ require_relative "netdisco/database_client"
5
6
  require_relative "netdisco/rules"
6
7
  require_relative "netdisco/device"
7
8
  require_relative "netdisco/config_file"
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Net
4
4
  module Connector
5
- VERSION = "0.5.0"
5
+ VERSION = "0.6.0"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: net-connector
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.0
4
+ version: 0.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - net-connector contributors
@@ -37,6 +37,26 @@ dependencies:
37
37
  - - "~>"
38
38
  - !ruby/object:Gem::Version
39
39
  version: 0.2.0
40
+ - !ruby/object:Gem::Dependency
41
+ name: pg
42
+ requirement: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - ">="
45
+ - !ruby/object:Gem::Version
46
+ version: '1.5'
47
+ - - "<"
48
+ - !ruby/object:Gem::Version
49
+ version: '2.0'
50
+ type: :runtime
51
+ prerelease: false
52
+ version_requirements: !ruby/object:Gem::Requirement
53
+ requirements:
54
+ - - ">="
55
+ - !ruby/object:Gem::Version
56
+ version: '1.5'
57
+ - - "<"
58
+ - !ruby/object:Gem::Version
59
+ version: '2.0'
40
60
  - !ruby/object:Gem::Dependency
41
61
  name: digest
42
62
  requirement: !ruby/object:Gem::Requirement
@@ -407,6 +427,7 @@ files:
407
427
  - docs/RELEASING.md
408
428
  - docs/VERIFICATION.md
409
429
  - docs/architecture.md
430
+ - examples/netdisco_database.yml
410
431
  - exe/net-connector-backup
411
432
  - lib/net-connector.rb
412
433
  - lib/net/connector.rb
@@ -451,6 +472,7 @@ files:
451
472
  - lib/net/connector/netdisco/cli.rb
452
473
  - lib/net/connector/netdisco/client.rb
453
474
  - lib/net/connector/netdisco/config_file.rb
475
+ - lib/net/connector/netdisco/database_client.rb
454
476
  - lib/net/connector/netdisco/device.rb
455
477
  - lib/net/connector/netdisco/diagnostic.rb
456
478
  - lib/net/connector/netdisco/fleet.rb