tinet-data-foundry-cli 0.1.5__tar.gz → 0.1.8__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (20) hide show
  1. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/PKG-INFO +25 -3
  2. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/README.md +24 -2
  3. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/pyproject.toml +1 -1
  4. tinet_data_foundry_cli-0.1.8/src/tinet_data_foundry_cli/__init__.py +1 -0
  5. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli/cli.py +67 -4
  6. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli/docs.py +117 -0
  7. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli.egg-info/PKG-INFO +25 -3
  8. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/tests/test_cli.py +8 -0
  9. tinet_data_foundry_cli-0.1.8/tests/test_docs.py +101 -0
  10. tinet_data_foundry_cli-0.1.5/src/tinet_data_foundry_cli/__init__.py +0 -1
  11. tinet_data_foundry_cli-0.1.5/tests/test_docs.py +0 -52
  12. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/setup.cfg +0 -0
  13. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli/__main__.py +0 -0
  14. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli/auth.py +0 -0
  15. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli/client.py +0 -0
  16. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli.egg-info/SOURCES.txt +0 -0
  17. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli.egg-info/dependency_links.txt +0 -0
  18. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli.egg-info/entry_points.txt +0 -0
  19. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/src/tinet_data_foundry_cli.egg-info/top_level.txt +0 -0
  20. {tinet_data_foundry_cli-0.1.5 → tinet_data_foundry_cli-0.1.8}/tests/test_sign.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: tinet-data-foundry-cli
3
- Version: 0.1.5
3
+ Version: 0.1.8
4
4
  Summary: Tinet Data Foundry 取数 CLI(docs / catalog / query / schema)
5
5
  Author: Tinet
6
6
  License: MIT
@@ -55,7 +55,16 @@ tdf query \
55
55
  | `ends_with` | 左模糊,匹配后缀 | `{"op":"ends_with","field":"name","value":"公司"}` |
56
56
  | `contains` | 左右模糊 | `{"op":"contains","field":"name","value":"科技"}` |
57
57
 
58
- 分区表可以不传分区字段。服务端按上海时区补默认闭区间:按日为当天,按月为当月,按年为当年,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 时,按日不超过 30 天,按月不超过 12 个月,按年不超过 3 年,否则 422 `PARTITION_RANGE_INVALID`。
58
+ 仅按日分区表必须带分区条件。未传分区字段时,服务端按上海时区补当天闭区间,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 不超过 30 天,否则 422 `PARTITION_RANGE_INVALID`。按月、按年分区字段不是必填条件。
59
+
60
+ ODS 可以查询和过滤 `_source_ds`,值是来源数据源编码。编码清单:
61
+
62
+ ```bash
63
+ tdf datasource-codes
64
+ tdf datasource-codes --code nboss
65
+ ```
66
+
67
+ `_df_pt` 是通用查询字段,表示分区查询时间,写在 `fields` 或 `filter` 里,只有分区表有这一列。建议范围:按日查当天,`between` 不超过 30 天;按月的值是每月 1 日,建议查当月 1 日,跨月不超过 12 个月;按年查当年。列上的 `suggestedRange` 给出当前建议的 `from`/`to`。
59
68
 
60
69
  若 pip 提示 scripts 目录不在 PATH,把该目录加入 PATH,或用 `python3 -m tinet_data_foundry_cli`。
61
70
 
@@ -87,12 +96,25 @@ tdf catalog --insecure
87
96
  | `tdf docs rev` | `GET /docs/credentials/{ak}?format=rev` |
88
97
  | `tdf docs pull` | 按 `copy.catalog.policy` 只拉取变化的 `part=public` 或 `part=policy` |
89
98
  | `tdf catalog` | `GET /data-api/v1/catalog` |
99
+ | `tdf datasource-codes` | `GET /data-api/v1/datasource-codes` |
90
100
  | `tdf query` | `POST /data-api/v1/datasets/{id}/query` |
91
101
  | `tdf schema` | `GET /data-api/v1/datasets/{id}/schema` |
92
102
 
93
103
  文档站点默认 `https://data-foundry.tinetcloud.com`,可用 `DF_DOCS_BASE` 覆盖。`tdf docs` 只需要 `DF_AK`。缓存位于 `~/.cache/tinet-data-foundry/docs/<AK 哈希>/`,可用 `DF_DOCS_CACHE` 改根目录。stdout 非终端时 `tdf catalog` 改为 JSON。
94
104
 
95
- Agent / Skill 传查询体用文件,避免 shell 转义:
105
+ ### 业务 Skill 锁定数据集
106
+
107
+ `tdf docs pull` 把公共目录与授权策略缓存到本机。业务 Skill 自行检索和阅读字段,不要把 `public.json` 或 `policy.json` 读进上下文:
108
+
109
+ ```bash
110
+ tdf docs pull
111
+ tdf docs search --query "<检索词>"
112
+ tdf docs describe --dataset "DATASET_ID"
113
+ ```
114
+
115
+ `search` 默认只返回有表权限的数据集,表名或中文名精确匹配排在前面,最多 20 条。`describe` 只返回该表已授权字段、分区粒度和脱敏规则。需要与网关当前结构核对时再用 `tdf schema --dataset "DATASET_ID"`。不要用不带 `--keyword` 的 `tdf catalog` 拉取整份目录。
116
+
117
+ DataFoundry 查询 Skill 只执行已经写好的查询。查询体用文件传入,避免 shell 转义:
96
118
 
97
119
  ```bash
98
120
  tdf query --dataset "DATASET_ID" --body-file query.json
@@ -38,7 +38,16 @@ tdf query \
38
38
  | `ends_with` | 左模糊,匹配后缀 | `{"op":"ends_with","field":"name","value":"公司"}` |
39
39
  | `contains` | 左右模糊 | `{"op":"contains","field":"name","value":"科技"}` |
40
40
 
41
- 分区表可以不传分区字段。服务端按上海时区补默认闭区间:按日为当天,按月为当月,按年为当年,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 时,按日不超过 30 天,按月不超过 12 个月,按年不超过 3 年,否则 422 `PARTITION_RANGE_INVALID`。
41
+ 仅按日分区表必须带分区条件。未传分区字段时,服务端按上海时区补当天闭区间,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 不超过 30 天,否则 422 `PARTITION_RANGE_INVALID`。按月、按年分区字段不是必填条件。
42
+
43
+ ODS 可以查询和过滤 `_source_ds`,值是来源数据源编码。编码清单:
44
+
45
+ ```bash
46
+ tdf datasource-codes
47
+ tdf datasource-codes --code nboss
48
+ ```
49
+
50
+ `_df_pt` 是通用查询字段,表示分区查询时间,写在 `fields` 或 `filter` 里,只有分区表有这一列。建议范围:按日查当天,`between` 不超过 30 天;按月的值是每月 1 日,建议查当月 1 日,跨月不超过 12 个月;按年查当年。列上的 `suggestedRange` 给出当前建议的 `from`/`to`。
42
51
 
43
52
  若 pip 提示 scripts 目录不在 PATH,把该目录加入 PATH,或用 `python3 -m tinet_data_foundry_cli`。
44
53
 
@@ -70,12 +79,25 @@ tdf catalog --insecure
70
79
  | `tdf docs rev` | `GET /docs/credentials/{ak}?format=rev` |
71
80
  | `tdf docs pull` | 按 `copy.catalog.policy` 只拉取变化的 `part=public` 或 `part=policy` |
72
81
  | `tdf catalog` | `GET /data-api/v1/catalog` |
82
+ | `tdf datasource-codes` | `GET /data-api/v1/datasource-codes` |
73
83
  | `tdf query` | `POST /data-api/v1/datasets/{id}/query` |
74
84
  | `tdf schema` | `GET /data-api/v1/datasets/{id}/schema` |
75
85
 
76
86
  文档站点默认 `https://data-foundry.tinetcloud.com`,可用 `DF_DOCS_BASE` 覆盖。`tdf docs` 只需要 `DF_AK`。缓存位于 `~/.cache/tinet-data-foundry/docs/<AK 哈希>/`,可用 `DF_DOCS_CACHE` 改根目录。stdout 非终端时 `tdf catalog` 改为 JSON。
77
87
 
78
- Agent / Skill 传查询体用文件,避免 shell 转义:
88
+ ### 业务 Skill 锁定数据集
89
+
90
+ `tdf docs pull` 把公共目录与授权策略缓存到本机。业务 Skill 自行检索和阅读字段,不要把 `public.json` 或 `policy.json` 读进上下文:
91
+
92
+ ```bash
93
+ tdf docs pull
94
+ tdf docs search --query "<检索词>"
95
+ tdf docs describe --dataset "DATASET_ID"
96
+ ```
97
+
98
+ `search` 默认只返回有表权限的数据集,表名或中文名精确匹配排在前面,最多 20 条。`describe` 只返回该表已授权字段、分区粒度和脱敏规则。需要与网关当前结构核对时再用 `tdf schema --dataset "DATASET_ID"`。不要用不带 `--keyword` 的 `tdf catalog` 拉取整份目录。
99
+
100
+ DataFoundry 查询 Skill 只执行已经写好的查询。查询体用文件传入,避免 shell 转义:
79
101
 
80
102
  ```bash
81
103
  tdf query --dataset "DATASET_ID" --body-file query.json
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "tinet-data-foundry-cli"
7
- version = "0.1.5"
7
+ version = "0.1.8"
8
8
  description = "Tinet Data Foundry 取数 CLI(docs / catalog / query / schema)"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -0,0 +1 @@
1
+ __version__ = "0.1.8"
@@ -82,6 +82,56 @@ def _ok_data(parsed: Any) -> Any:
82
82
  return parsed.get("data") if isinstance(parsed, dict) else parsed
83
83
 
84
84
 
85
+ def _print_datasource_codes(rows: List[Mapping[str, Any]]) -> None:
86
+ if not rows:
87
+ print("当前凭证没有可查询的数据源编码。")
88
+ return
89
+ cols = [
90
+ ("code", "编码", 24),
91
+ ("name", "名称", 28),
92
+ ("id", "id", 36),
93
+ ]
94
+ header = " ".join(_cell(title, w) for _, title, w in cols)
95
+ print(header)
96
+ print("-" * len(header))
97
+ for row in rows:
98
+ print(" ".join(_cell(row.get(key), w) for key, _, w in cols))
99
+
100
+
101
+ def cmd_datasource_codes(args: argparse.Namespace) -> int:
102
+ base, ak, sk = credentials(args.endpoint)
103
+ query = {
104
+ "code": args.code or "",
105
+ "keyword": args.keyword or "",
106
+ }
107
+ try:
108
+ _, parsed, _ = request(
109
+ "GET",
110
+ base,
111
+ "/data-api/v1/datasource-codes",
112
+ ak,
113
+ sk,
114
+ query=query,
115
+ timeout=args.timeout,
116
+ insecure=args.insecure,
117
+ dry_run=args.dry_run,
118
+ )
119
+ except ApiError as ex:
120
+ _exit_api(ex)
121
+ except TransportError as ex:
122
+ _exit_transport(ex)
123
+ fmt = args.format
124
+ if fmt == "table" and not sys.stdout.isatty():
125
+ fmt = "json"
126
+ if args.dry_run or fmt == "json":
127
+ _print_json(parsed)
128
+ return 0
129
+ data = _ok_data(parsed) or {}
130
+ rows = data.get("list") if isinstance(data, dict) else []
131
+ _print_datasource_codes(list(rows or []))
132
+ return 0
133
+
134
+
85
135
  def cmd_catalog(args: argparse.Namespace) -> int:
86
136
  base, ak, sk = credentials(args.endpoint)
87
137
  query = {
@@ -249,6 +299,13 @@ def build_parser() -> argparse.ArgumentParser:
249
299
  c.add_argument("--format", choices=["table", "json"], default="table")
250
300
  c.set_defaults(func=cmd_catalog)
251
301
 
302
+ codes = sub.add_parser("datasource-codes", help="列出当前凭证可查询数据集关联的数据源编码")
303
+ _add_insecure(codes, overlay=True)
304
+ codes.add_argument("--code", help="按编码精确匹配,忽略大小写")
305
+ codes.add_argument("--keyword", help="按编码或名称包含匹配")
306
+ codes.add_argument("--format", choices=["table", "json"], default="table")
307
+ codes.set_defaults(func=cmd_datasource_codes)
308
+
252
309
  q = sub.add_parser("query", help="查询 Dataset")
253
310
  _add_insecure(q, overlay=True)
254
311
  q.add_argument("--dataset", required=True)
@@ -259,8 +316,10 @@ def build_parser() -> argparse.ArgumentParser:
259
316
  "--filter",
260
317
  help=(
261
318
  "filter AST JSON。字符串字段可 starts_with(右模糊)、ends_with(左模糊)、contains(左右模糊)。"
262
- "分区表省略分区字段时,按日补当天、按月补当月、按年补当年;"
263
- "显式 between 按日不超过 30 天、按月不超过 12 个月、按年不超过 3 年"
319
+ "仅按日分区必填:省略分区字段时按上海时区补当天,显式 between 不超过 30 天。"
320
+ "按月、按年分区字段不是必填条件。"
321
+ "_df_pt 是通用查询字段,表示分区查询时间:按日建议当天(不超过 30 天),"
322
+ "按月建议当月 1 日(不超过 12 个月),按年建议当年"
264
323
  ),
265
324
  )
266
325
  q.add_argument("--sort", help="sort JSON 数组")
@@ -277,9 +336,13 @@ def build_parser() -> argparse.ArgumentParser:
277
336
  )
278
337
  q.set_defaults(func=cmd_query)
279
338
 
280
- d = sub.add_parser("docs", help="核对数据集开发文档修订号并缓存 public/policy")
339
+ d = sub.add_parser("docs", help="核对数据集开发文档修订号,并在本地缓存中检索数据集")
281
340
  _add_insecure(d, overlay=True)
282
- d.add_argument("action", choices=["rev", "pull"])
341
+ d.add_argument("action", choices=["rev", "pull", "search", "describe"])
342
+ d.add_argument("--query", help="search:按名称、描述、表名或业务域检索本地缓存。表名精确匹配优先")
343
+ d.add_argument("--dataset", help="describe:输出指定数据集的已授权字段")
344
+ d.add_argument("--all", action="store_true", help="search:包含无表权限的数据集")
345
+ d.add_argument("--limit", type=int, default=20, help="search:最多返回条数,默认 20")
283
346
  d.add_argument("--docs-base", help="覆盖文档站点。默认 DF_DOCS_BASE 或 https://data-foundry.tinetcloud.com")
284
347
  d.set_defaults(func=cmd_docs)
285
348
 
@@ -47,6 +47,15 @@ def run_docs(args: Any) -> int:
47
47
  ak = access_key()
48
48
  base = docs_base(getattr(args, "docs_base", None))
49
49
  folder = cache_dir(ak)
50
+ if args.action == "search":
51
+ return _search(
52
+ folder,
53
+ str(getattr(args, "query", "") or ""),
54
+ bool(getattr(args, "all", False)),
55
+ int(getattr(args, "limit", None) or 20),
56
+ )
57
+ if args.action == "describe":
58
+ return _describe(folder, str(getattr(args, "dataset", "") or ""))
50
59
  url = docs_url(base, ak, "rev")
51
60
  if args.dry_run:
52
61
  _print({
@@ -64,6 +73,114 @@ def run_docs(args: Any) -> int:
64
73
  return _pull(folder, base, ak, args.timeout, args.insecure)
65
74
 
66
75
 
76
+ def _cached_docs(folder: Path) -> tuple[Dict[str, Any], Dict[str, Any]]:
77
+ public = _read_json(folder / "public.json")
78
+ policy = _read_json(folder / "policy.json")
79
+ if not public or not policy:
80
+ raise SystemExit("本地文档缓存不存在,请先运行 tdf docs pull")
81
+ return public, policy
82
+
83
+
84
+ def _search(folder: Path, query: str, include_denied: bool, limit: int = 20) -> int:
85
+ public, policy = _cached_docs(folder)
86
+ needle = query.strip().casefold()
87
+ if not needle:
88
+ raise SystemExit("docs search 需要 --query")
89
+ if limit < 1:
90
+ raise SystemExit("docs search --limit 必须大于 0")
91
+ grants = _index(policy)
92
+ ranked: List[tuple] = []
93
+ for dataset in public.get("datasets") or []:
94
+ if not isinstance(dataset, dict):
95
+ continue
96
+ grant = grants.get(str(dataset.get("id") or ""), {})
97
+ if not include_denied and not grant.get("tableGranted"):
98
+ continue
99
+ rank = _match_rank(dataset, needle)
100
+ if rank is None:
101
+ continue
102
+ ranked.append((rank, str(dataset.get("name") or ""), {
103
+ "id": dataset.get("id"),
104
+ "name": dataset.get("name"),
105
+ "chineseName": dataset.get("chineseName"),
106
+ "description": dataset.get("description"),
107
+ "businessDomainName": dataset.get("businessDomainName"),
108
+ "layer": dataset.get("layer"),
109
+ "schemaVersion": dataset.get("schemaVersion"),
110
+ "tableGranted": bool(grant.get("tableGranted")),
111
+ }))
112
+ ranked.sort(key=lambda row: (row[0], row[1]))
113
+ shown = ranked[:limit]
114
+ _print({
115
+ "query": query,
116
+ "total": len(ranked),
117
+ "truncated": len(ranked) > limit,
118
+ "datasets": [row[2] for row in shown],
119
+ })
120
+ return 0
121
+
122
+
123
+ def _match_rank(dataset: Dict[str, Any], needle: str) -> Optional[int]:
124
+ identity = [str(dataset.get(key) or "").casefold() for key in ("id", "name", "chineseName")]
125
+ if needle in identity:
126
+ return 0
127
+ if any(value.startswith(needle) for value in identity if value):
128
+ return 1
129
+ if any(needle in value for value in identity if value):
130
+ return 2
131
+ extra = " ".join(str(dataset.get(key) or "") for key in (
132
+ "tableComment", "description", "businessDomainName",
133
+ )).casefold()
134
+ if needle in extra:
135
+ return 3
136
+ return None
137
+
138
+
139
+ def _describe(folder: Path, dataset_id: str) -> int:
140
+ public, policy = _cached_docs(folder)
141
+ if not dataset_id.strip():
142
+ raise SystemExit("docs describe 需要 --dataset")
143
+ datasets = _index(public)
144
+ dataset = datasets.get(dataset_id)
145
+ if not dataset:
146
+ raise SystemExit(f"缓存中没有数据集 {dataset_id};先执行 tdf docs search --query <关键词>")
147
+ grant = _index(policy).get(dataset_id, {})
148
+ rules = {
149
+ str(col.get("name")): col
150
+ for col in grant.get("columns") or []
151
+ if isinstance(col, dict) and col.get("granted") and col.get("name")
152
+ }
153
+ columns = []
154
+ for column in dataset.get("columns") or []:
155
+ if not isinstance(column, dict) or str(column.get("name")) not in rules:
156
+ continue
157
+ name = str(column["name"])
158
+ columns.append({
159
+ "name": name,
160
+ "type": column.get("type"),
161
+ "comment": column.get("comment"),
162
+ "pk": column.get("pk"),
163
+ "partition": column.get("partition"),
164
+ "partitionGrain": column.get("partitionGrain"),
165
+ "partitionRequired": column.get("partitionRequired"),
166
+ "suggestedRange": column.get("suggestedRange"),
167
+ "outputRule": rules[name].get("outputRule"),
168
+ })
169
+ _print({
170
+ "id": dataset.get("id"),
171
+ "name": dataset.get("name"),
172
+ "chineseName": dataset.get("chineseName"),
173
+ "description": dataset.get("description"),
174
+ "businessDomainName": dataset.get("businessDomainName"),
175
+ "schemaVersion": dataset.get("schemaVersion"),
176
+ "primaryKey": dataset.get("primaryKey"),
177
+ "partitionFields": dataset.get("partitionFields"),
178
+ "tableGranted": bool(grant.get("tableGranted")),
179
+ "columns": columns,
180
+ })
181
+ return 0
182
+
183
+
67
184
  def _pull(folder: Path, base: str, ak: str, timeout: float, insecure: bool) -> int:
68
185
  local = _read_json(folder / "rev.json")
69
186
  local_etag = str(local.get("etag") or "")
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: tinet-data-foundry-cli
3
- Version: 0.1.5
3
+ Version: 0.1.8
4
4
  Summary: Tinet Data Foundry 取数 CLI(docs / catalog / query / schema)
5
5
  Author: Tinet
6
6
  License: MIT
@@ -55,7 +55,16 @@ tdf query \
55
55
  | `ends_with` | 左模糊,匹配后缀 | `{"op":"ends_with","field":"name","value":"公司"}` |
56
56
  | `contains` | 左右模糊 | `{"op":"contains","field":"name","value":"科技"}` |
57
57
 
58
- 分区表可以不传分区字段。服务端按上海时区补默认闭区间:按日为当天,按月为当月,按年为当年,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 时,按日不超过 30 天,按月不超过 12 个月,按年不超过 3 年,否则 422 `PARTITION_RANGE_INVALID`。
58
+ 仅按日分区表必须带分区条件。未传分区字段时,服务端按上海时区补当天闭区间,并在响应 `data.partitionWindow` 里返回实际 `field/from/to`。显式 `between` 不超过 30 天,否则 422 `PARTITION_RANGE_INVALID`。按月、按年分区字段不是必填条件。
59
+
60
+ ODS 可以查询和过滤 `_source_ds`,值是来源数据源编码。编码清单:
61
+
62
+ ```bash
63
+ tdf datasource-codes
64
+ tdf datasource-codes --code nboss
65
+ ```
66
+
67
+ `_df_pt` 是通用查询字段,表示分区查询时间,写在 `fields` 或 `filter` 里,只有分区表有这一列。建议范围:按日查当天,`between` 不超过 30 天;按月的值是每月 1 日,建议查当月 1 日,跨月不超过 12 个月;按年查当年。列上的 `suggestedRange` 给出当前建议的 `from`/`to`。
59
68
 
60
69
  若 pip 提示 scripts 目录不在 PATH,把该目录加入 PATH,或用 `python3 -m tinet_data_foundry_cli`。
61
70
 
@@ -87,12 +96,25 @@ tdf catalog --insecure
87
96
  | `tdf docs rev` | `GET /docs/credentials/{ak}?format=rev` |
88
97
  | `tdf docs pull` | 按 `copy.catalog.policy` 只拉取变化的 `part=public` 或 `part=policy` |
89
98
  | `tdf catalog` | `GET /data-api/v1/catalog` |
99
+ | `tdf datasource-codes` | `GET /data-api/v1/datasource-codes` |
90
100
  | `tdf query` | `POST /data-api/v1/datasets/{id}/query` |
91
101
  | `tdf schema` | `GET /data-api/v1/datasets/{id}/schema` |
92
102
 
93
103
  文档站点默认 `https://data-foundry.tinetcloud.com`,可用 `DF_DOCS_BASE` 覆盖。`tdf docs` 只需要 `DF_AK`。缓存位于 `~/.cache/tinet-data-foundry/docs/<AK 哈希>/`,可用 `DF_DOCS_CACHE` 改根目录。stdout 非终端时 `tdf catalog` 改为 JSON。
94
104
 
95
- Agent / Skill 传查询体用文件,避免 shell 转义:
105
+ ### 业务 Skill 锁定数据集
106
+
107
+ `tdf docs pull` 把公共目录与授权策略缓存到本机。业务 Skill 自行检索和阅读字段,不要把 `public.json` 或 `policy.json` 读进上下文:
108
+
109
+ ```bash
110
+ tdf docs pull
111
+ tdf docs search --query "<检索词>"
112
+ tdf docs describe --dataset "DATASET_ID"
113
+ ```
114
+
115
+ `search` 默认只返回有表权限的数据集,表名或中文名精确匹配排在前面,最多 20 条。`describe` 只返回该表已授权字段、分区粒度和脱敏规则。需要与网关当前结构核对时再用 `tdf schema --dataset "DATASET_ID"`。不要用不带 `--keyword` 的 `tdf catalog` 拉取整份目录。
116
+
117
+ DataFoundry 查询 Skill 只执行已经写好的查询。查询体用文件传入,避免 shell 转义:
96
118
 
97
119
  ```bash
98
120
  tdf query --dataset "DATASET_ID" --body-file query.json
@@ -17,13 +17,21 @@ class CliParserTest(unittest.TestCase):
17
17
  self.assertIn("starts_with", help_text)
18
18
  self.assertIn("ends_with", help_text)
19
19
  self.assertIn("contains", help_text)
20
+ self.assertIn("仅按日分区必填", help_text)
20
21
  self.assertIn("30", help_text)
21
22
  self.assertIn("当天", help_text)
23
+ self.assertIn("按月、按年分区字段不是必填条件", help_text)
24
+ self.assertIn("_df_pt", help_text)
25
+ self.assertIn("分区查询时间", help_text)
26
+ self.assertNotIn("按月补当月", help_text)
22
27
 
23
28
  def test_subcommands(self):
24
29
  parser = build_parser()
25
30
  catalog = parser.parse_args(["catalog", "--format", "json"])
26
31
  self.assertEqual(catalog.cmd, "catalog")
32
+ codes = parser.parse_args(["datasource-codes", "--code", "nboss"])
33
+ self.assertEqual(codes.cmd, "datasource-codes")
34
+ self.assertEqual(codes.code, "nboss")
27
35
  query = parser.parse_args(["query", "--dataset", "ds_1", "--page-no", "1", "--page-size", "10"])
28
36
  self.assertEqual(query.cmd, "query")
29
37
  catalog_k = parser.parse_args(["catalog", "--insecure"])
@@ -0,0 +1,101 @@
1
+ import json
2
+ import tempfile
3
+ import unittest
4
+ from io import StringIO
5
+ from pathlib import Path
6
+ from unittest.mock import patch
7
+
8
+ from tinet_data_foundry_cli.docs import _describe, _search, summarize
9
+
10
+
11
+ class DocsSummaryTest(unittest.TestCase):
12
+ def test_search_and_describe_only_print_authorized_dataset_content(self):
13
+ with tempfile.TemporaryDirectory() as tmp, patch("sys.stdout", new_callable=StringIO) as out:
14
+ folder = Path(tmp)
15
+ (folder / "public.json").write_text(json.dumps({"datasets": [{
16
+ "id": "ds_bill", "name": "monthly_bill", "chineseName": "月账单",
17
+ "description": "京东云消费", "columns": [
18
+ {"name": "amount", "type": "decimal", "comment": "金额"},
19
+ {"name": "secret", "type": "text", "comment": "无权限"},
20
+ {"name": "_df_pt", "type": "date", "comment": "分区查询时间",
21
+ "suggestedRange": {"from": "2026-10-01", "to": "2026-10-01", "grain": "month"}},
22
+ ],
23
+ }]}), encoding="utf-8")
24
+ (folder / "policy.json").write_text(json.dumps({"datasets": [{
25
+ "id": "ds_bill", "tableGranted": True,
26
+ "columns": [
27
+ {"name": "amount", "granted": True, "outputRule": "PLAIN"},
28
+ {"name": "_df_pt", "granted": True, "outputRule": "PLAIN"},
29
+ ],
30
+ }]}), encoding="utf-8")
31
+ self.assertEqual(_search(folder, "京东云", False), 0)
32
+ search = json.loads(out.getvalue())
33
+ self.assertEqual(search["datasets"][0]["id"], "ds_bill")
34
+ self.assertEqual(search["total"], 1)
35
+ self.assertFalse(search["truncated"])
36
+ out.seek(0)
37
+ out.truncate(0)
38
+ self.assertEqual(_describe(folder, "ds_bill"), 0)
39
+ described = json.loads(out.getvalue())
40
+ self.assertEqual([col["name"] for col in described["columns"]], ["amount", "_df_pt"])
41
+ self.assertEqual(described["columns"][1]["suggestedRange"]["from"], "2026-10-01")
42
+
43
+ def test_search_prefers_exact_name_and_limits_rows(self):
44
+ with tempfile.TemporaryDirectory() as tmp, patch("sys.stdout", new_callable=StringIO) as out:
45
+ folder = Path(tmp)
46
+ datasets = [
47
+ {"id": "ds_desc", "name": "other", "description": "bill detail"},
48
+ {"id": "ds_exact", "name": "bill", "chineseName": "账单"},
49
+ {"id": "ds_more", "name": "bill_extra", "description": "bill"},
50
+ ]
51
+ grants = [{"id": row["id"], "tableGranted": True, "columns": []} for row in datasets]
52
+ (folder / "public.json").write_text(json.dumps({"datasets": datasets}), encoding="utf-8")
53
+ (folder / "policy.json").write_text(json.dumps({"datasets": grants}), encoding="utf-8")
54
+ self.assertEqual(_search(folder, "bill", False, 2), 0)
55
+ search = json.loads(out.getvalue())
56
+ self.assertEqual([row["id"] for row in search["datasets"]], ["ds_exact", "ds_more"])
57
+ self.assertEqual(search["total"], 3)
58
+ self.assertTrue(search["truncated"])
59
+
60
+ def test_summarize_catalog_and_policy(self):
61
+ old_public = {"datasets": [{"id": "a", "schemaVersion": "1"}, {"id": "b", "schemaVersion": "1"}]}
62
+ new_public = {"datasets": [{"id": "b", "schemaVersion": "2"}, {"id": "c", "schemaVersion": "1"}]}
63
+ old_policy = {"datasets": [{"id": "a", "tableGranted": True}, {"id": "b", "tableGranted": False}]}
64
+ new_policy = {"datasets": [{"id": "b", "tableGranted": True}, {"id": "c", "tableGranted": False}]}
65
+ lines = summarize(old_public, new_public, old_policy, new_policy)
66
+ self.assertIn("新增数据集 c", lines)
67
+ self.assertIn("下线数据集 a", lines)
68
+ self.assertIn("schemaVersion 变化 b", lines)
69
+ self.assertIn("表权限变化 b 有", lines)
70
+
71
+ def test_pull_fetches_only_policy_when_catalog_unchanged(self):
72
+ from tinet_data_foundry_cli import docs as docs_mod
73
+
74
+ rev = {"copy": "c", "catalog": "k", "policy": "p2", "etag": "c.k.p2"}
75
+ calls = []
76
+
77
+ def fake_get(url, etag, timeout, insecure):
78
+ calls.append(url)
79
+ if "format=rev" in url:
80
+ return 200, rev
81
+ if "part=policy" in url:
82
+ return 200, {"credentialType": "PERSONAL", "datasets": [{"id": "a", "tableGranted": True, "columns": []}]}
83
+ raise AssertionError(url)
84
+
85
+ with tempfile.TemporaryDirectory() as tmp:
86
+ folder = Path(tmp)
87
+ (folder / "rev.json").write_text(json.dumps({
88
+ "copy": "c", "catalog": "k", "policy": "p1", "etag": "c.k.p1",
89
+ }), encoding="utf-8")
90
+ (folder / "public.json").write_text(json.dumps({"datasets": [{"id": "a"}]}), encoding="utf-8")
91
+ with patch.object(docs_mod, "_get_json", fake_get):
92
+ code = docs_mod._pull(folder, "https://docs.example", "DFAK_x", 5, False)
93
+ self.assertEqual(code, 0)
94
+ self.assertTrue(any("part=policy" in url for url in calls))
95
+ self.assertFalse(any("part=public" in url for url in calls))
96
+ saved = json.loads((folder / "rev.json").read_text(encoding="utf-8"))
97
+ self.assertEqual(saved["policy"], "p2")
98
+
99
+
100
+ if __name__ == "__main__":
101
+ unittest.main()
@@ -1 +0,0 @@
1
- __version__ = "0.1.5"
@@ -1,52 +0,0 @@
1
- import json
2
- import tempfile
3
- import unittest
4
- from pathlib import Path
5
- from unittest.mock import patch
6
-
7
- from tinet_data_foundry_cli.docs import summarize
8
-
9
-
10
- class DocsSummaryTest(unittest.TestCase):
11
- def test_summarize_catalog_and_policy(self):
12
- old_public = {"datasets": [{"id": "a", "schemaVersion": "1"}, {"id": "b", "schemaVersion": "1"}]}
13
- new_public = {"datasets": [{"id": "b", "schemaVersion": "2"}, {"id": "c", "schemaVersion": "1"}]}
14
- old_policy = {"datasets": [{"id": "a", "tableGranted": True}, {"id": "b", "tableGranted": False}]}
15
- new_policy = {"datasets": [{"id": "b", "tableGranted": True}, {"id": "c", "tableGranted": False}]}
16
- lines = summarize(old_public, new_public, old_policy, new_policy)
17
- self.assertIn("新增数据集 c", lines)
18
- self.assertIn("下线数据集 a", lines)
19
- self.assertIn("schemaVersion 变化 b", lines)
20
- self.assertIn("表权限变化 b 有", lines)
21
-
22
- def test_pull_fetches_only_policy_when_catalog_unchanged(self):
23
- from tinet_data_foundry_cli import docs as docs_mod
24
-
25
- rev = {"copy": "c", "catalog": "k", "policy": "p2", "etag": "c.k.p2"}
26
- calls = []
27
-
28
- def fake_get(url, etag, timeout, insecure):
29
- calls.append(url)
30
- if "format=rev" in url:
31
- return 200, rev
32
- if "part=policy" in url:
33
- return 200, {"credentialType": "PERSONAL", "datasets": [{"id": "a", "tableGranted": True, "columns": []}]}
34
- raise AssertionError(url)
35
-
36
- with tempfile.TemporaryDirectory() as tmp:
37
- folder = Path(tmp)
38
- (folder / "rev.json").write_text(json.dumps({
39
- "copy": "c", "catalog": "k", "policy": "p1", "etag": "c.k.p1",
40
- }), encoding="utf-8")
41
- (folder / "public.json").write_text(json.dumps({"datasets": [{"id": "a"}]}), encoding="utf-8")
42
- with patch.object(docs_mod, "_get_json", fake_get):
43
- code = docs_mod._pull(folder, "https://docs.example", "DFAK_x", 5, False)
44
- self.assertEqual(code, 0)
45
- self.assertTrue(any("part=policy" in url for url in calls))
46
- self.assertFalse(any("part=public" in url for url in calls))
47
- saved = json.loads((folder / "rev.json").read_text(encoding="utf-8"))
48
- self.assertEqual(saved["policy"], "p2")
49
-
50
-
51
- if __name__ == "__main__":
52
- unittest.main()