@josephyan/qingflow-app-user-mcp 1.0.7 → 1.0.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/package.json +1 -1
- package/pyproject.toml +1 -1
- package/skills/qingflow-app-user/references/public-surface-sync.md +1 -0
- package/skills/qingflow-record-analysis/SKILL.md +5 -1
- package/skills/qingflow-record-analysis/references/analysis-gotchas.md +2 -0
- package/skills/qingflow-record-analysis/references/analysis-patterns.md +13 -9
- package/skills/qingflow-record-analysis/references/confidence-reporting.md +2 -0
- package/src/qingflow_mcp/cli/formatters.py +31 -0
- package/src/qingflow_mcp/response_trim.py +1 -0
- package/src/qingflow_mcp/server_app_user.py +4 -1
- package/src/qingflow_mcp/tools/record_tools.py +473 -109
package/README.md
CHANGED
|
@@ -3,13 +3,13 @@
|
|
|
3
3
|
Install:
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
|
-
npm install @josephyan/qingflow-app-user-mcp@1.0.
|
|
6
|
+
npm install @josephyan/qingflow-app-user-mcp@1.0.9
|
|
7
7
|
```
|
|
8
8
|
|
|
9
9
|
Run:
|
|
10
10
|
|
|
11
11
|
```bash
|
|
12
|
-
npx -y -p @josephyan/qingflow-app-user-mcp@1.0.
|
|
12
|
+
npx -y -p @josephyan/qingflow-app-user-mcp@1.0.9 qingflow-app-user-mcp
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
Environment:
|
package/package.json
CHANGED
package/pyproject.toml
CHANGED
|
@@ -8,6 +8,7 @@ It is not a user-facing product spec. It exists to prevent skill drift.
|
|
|
8
8
|
### User data
|
|
9
9
|
|
|
10
10
|
- Read range first with `app_get`, then `record_browse_schema_get(view_id=...)`
|
|
11
|
+
- Treat `record_browse_schema_get.fields` as the selected Qingflow table view header schema; `record_access.fields` and `schema.json` must stay aligned with it.
|
|
11
12
|
- Standard flows:
|
|
12
13
|
- analyze: `app_get -> record_browse_schema_get -> record_access -> Python`
|
|
13
14
|
- browse detail: `app_get -> record_browse_schema_get -> record_list / record_get`
|
|
@@ -9,7 +9,7 @@ metadata:
|
|
|
9
9
|
|
|
10
10
|
This skill is for final statistical conclusions only.
|
|
11
11
|
Assumes MCP is connected, authenticated, and on the correct workspace.
|
|
12
|
-
Analysis tasks must start with `app_get`, then `record_browse_schema_get(view_id=...)`. Read top-level `fields` and `suggested_*`, then choose field_id-based columns and filters only.
|
|
12
|
+
Analysis tasks must start with `app_get`, then `record_browse_schema_get(view_id=...)`. `record_browse_schema_get.fields` is the selected Qingflow table view's readable header schema and is the same field source used by `record_access.fields` / `schema.json`. Read top-level `fields` and `suggested_*`, then choose field_id-based columns and filters only.
|
|
13
13
|
If `app_get.accessible_views` marks a view with `analysis_supported=false`, do not use that view for `record_access`, `record_list`, or `record_analyze`. `boardView` and `ganttView` are special UI views, not data-access targets.
|
|
14
14
|
|
|
15
15
|
## Step 1: `app_get` → Step 2: `record_browse_schema_get(view_id=...)` → Step 3: `record_access` → Step 4: Python
|
|
@@ -17,6 +17,7 @@ If `app_get.accessible_views` marks a view with `analysis_supported=false`, do n
|
|
|
17
17
|
This is the default execution order. Never skip `app_get` when the browse range is unclear. Never call `record_access` without a browse schema.
|
|
18
18
|
|
|
19
19
|
Core tools: `app_get`, `record_browse_schema_get`, `record_access`, Python. Use field_id-based DSLs only for columns, filters, sort clauses, and any optional lightweight `record_analyze` helper. Use `record_list`/`record_get` only for browse samples. Use `record_analyze` only as a lightweight non-default statistics helper when a compact grouped result is enough. Task/comment work stays in [$qingflow-task-ops](../qingflow-task-ops/SKILL.md).
|
|
20
|
+
For analysis-style tasks, prefer a concrete time range or business filter. If the user did not give one and the data could be large, ask for scope instead of trying an unbounded historical scan.
|
|
20
21
|
|
|
21
22
|
## Execution Modes
|
|
22
23
|
|
|
@@ -76,7 +77,10 @@ Use `record_access` to fetch detail rows into local CSV shards. It does not anal
|
|
|
76
77
|
Then run Python against every `files[].local_path`. CSV columns are stable: `record_id`, then `field_<field_id>`. Use `fields[]` plus `metadata_files.schema` / `metadata_files.readme` in the same output directory to map titles and types, especially when reusing the CSV later.
|
|
77
78
|
|
|
78
79
|
- Never ask for `page`, `page_size`, `limit`, or `max_rows`; `record_access` owns paging internally and follows the backend's native paging capability.
|
|
80
|
+
- Never treat backend `searchQueIds` as column selection; it is only a full-text search scope.
|
|
79
81
|
- If multiple CSV files are returned, read them all.
|
|
82
|
+
- If `status=needs_scope`, no CSV was written; use `scope.suggested_time_fields` / `scope.recommended_where_examples` to ask for or apply a time/business range, then call `record_access` again.
|
|
83
|
+
- If `status=partial`, read the returned files only as a limited subset; state the limitation and do not present a full-population final conclusion.
|
|
80
84
|
- If `complete=false` or `safe_for_final_conclusion=false`, downgrade the answer and disclose the limitation.
|
|
81
85
|
- `record_export_direct` is only for explicit export/download/Excel requests, not default analysis.
|
|
82
86
|
- QingBI/report reads are only for user-provided report URLs or `chart_id`; do not create or use reports as the default analysis path.
|
|
@@ -125,6 +125,8 @@ If the user asked for several outputs and only part of them is stable:
|
|
|
125
125
|
- say which parts are still unresolved
|
|
126
126
|
- do not present the answer as fully finished
|
|
127
127
|
|
|
128
|
+
If `record_access.status=needs_scope`, stop and ask for a time/business range; no CSV was written. If `record_access.status=partial`, read the returned CSV only as a partial subset and name the limitation before any numbers.
|
|
129
|
+
|
|
128
130
|
## Do not send unsupported formula or div-style metrics into `record_analyze`.
|
|
129
131
|
|
|
130
132
|
Examples to avoid:
|
|
@@ -25,15 +25,19 @@ Use this skill when the user asks for:
|
|
|
25
25
|
|
|
26
26
|
Result reading order:
|
|
27
27
|
|
|
28
|
-
1. `record_access.
|
|
29
|
-
2. `record_access.
|
|
30
|
-
3. `record_access.
|
|
31
|
-
4. `record_access.
|
|
32
|
-
5.
|
|
33
|
-
6.
|
|
34
|
-
7. `record_access.
|
|
35
|
-
|
|
36
|
-
|
|
28
|
+
1. `record_access.status`
|
|
29
|
+
2. `record_access.complete`
|
|
30
|
+
3. `record_access.safe_for_final_conclusion`
|
|
31
|
+
4. `record_access.files[].local_path`
|
|
32
|
+
5. `record_access.metadata_files.schema`
|
|
33
|
+
6. Python outputs
|
|
34
|
+
7. `record_access.fields`
|
|
35
|
+
8. `record_access.warnings`
|
|
36
|
+
9. `record_access.scope`
|
|
37
|
+
|
|
38
|
+
If `record_access.status=needs_scope`, no CSV was written; ask for a time/business range or retry with a user-provided period from `scope.suggested_time_fields`. If `status=partial`, use the CSV files only as a limited subset and do not make a full-population conclusion.
|
|
39
|
+
|
|
40
|
+
Treat `record_browse_schema_get` as the browse-schema source of truth. It matches the selected Qingflow table view header and is the same schema source used by `record_access.fields` and `schema.json`. Missing fields are permission boundaries, not invitations to guess hidden ids.
|
|
37
41
|
|
|
38
42
|
## Lightweight `record_analyze` helper sequence
|
|
39
43
|
|
|
@@ -27,6 +27,8 @@ Put evidence into `样本观察` when:
|
|
|
27
27
|
- the tool reports `row_cap_hit`
|
|
28
28
|
- the tool reports `sample_only`
|
|
29
29
|
- the result is compact/capped and not complete
|
|
30
|
+
- `record_access.status=needs_scope`
|
|
31
|
+
- `record_access.status=partial`
|
|
30
32
|
- `record_access.complete=false` or `record_access.truncated=true`
|
|
31
33
|
|
|
32
34
|
## Downgrade rule
|
|
@@ -194,6 +194,36 @@ def _format_record_list(result: dict[str, Any]) -> str:
|
|
|
194
194
|
return "\n".join(lines) + "\n"
|
|
195
195
|
|
|
196
196
|
|
|
197
|
+
def _format_record_access(result: dict[str, Any]) -> str:
|
|
198
|
+
status = result.get("status") or "-"
|
|
199
|
+
lines = [
|
|
200
|
+
f"Status: {status}",
|
|
201
|
+
f"Rows: {result.get('row_count')}",
|
|
202
|
+
f"Complete: {result.get('complete')}",
|
|
203
|
+
f"Safe for final conclusion: {result.get('safe_for_final_conclusion')}",
|
|
204
|
+
]
|
|
205
|
+
if result.get("local_dir"):
|
|
206
|
+
lines.append(f"Local dir: {result.get('local_dir')}")
|
|
207
|
+
files = result.get("files") if isinstance(result.get("files"), list) else []
|
|
208
|
+
if files:
|
|
209
|
+
lines.append("Files:")
|
|
210
|
+
for item in files:
|
|
211
|
+
if isinstance(item, dict):
|
|
212
|
+
lines.append(f"- part {item.get('part')}: {item.get('local_path')} ({item.get('row_count')} rows)")
|
|
213
|
+
scope = result.get("scope") if isinstance(result.get("scope"), dict) else {}
|
|
214
|
+
if status == "needs_scope" and scope:
|
|
215
|
+
lines.append("Scope required:")
|
|
216
|
+
lines.append(f"- reported_total: {scope.get('reported_total')}")
|
|
217
|
+
lines.append(f"- estimated_pages: {scope.get('estimated_pages')}")
|
|
218
|
+
suggested = scope.get("suggested_time_fields") if isinstance(scope.get("suggested_time_fields"), list) else []
|
|
219
|
+
if suggested:
|
|
220
|
+
names = ", ".join(str(item.get("title") or item.get("field_id")) for item in suggested if isinstance(item, dict))
|
|
221
|
+
lines.append(f"- suggested_time_fields: {names}")
|
|
222
|
+
_append_warnings(lines, result.get("warnings"))
|
|
223
|
+
_append_verification(lines, result.get("verification"))
|
|
224
|
+
return "\n".join(lines) + "\n"
|
|
225
|
+
|
|
226
|
+
|
|
197
227
|
def _format_task_list(result: dict[str, Any]) -> str:
|
|
198
228
|
data = result.get("data") if isinstance(result.get("data"), dict) else {}
|
|
199
229
|
items = data.get("items") if isinstance(data.get("items"), list) else []
|
|
@@ -693,6 +723,7 @@ _FORMATTERS = {
|
|
|
693
723
|
"app_search": _format_app_items,
|
|
694
724
|
"app_get": _format_app_get,
|
|
695
725
|
"record_list": _format_record_list,
|
|
726
|
+
"record_access": _format_record_access,
|
|
696
727
|
"task_list": _format_task_list,
|
|
697
728
|
"task_workbench": _format_task_workbench,
|
|
698
729
|
"task_get": _format_task_get,
|
|
@@ -60,7 +60,9 @@ Call `record_import_schema_get` when the import field mapping is unclear before
|
|
|
60
60
|
`record_insert_schema_get` returns the current user's insert-ready applicant schema; read `required_fields`, `optional_fields`, `runtime_linked_required_fields`, and `payload_template`.
|
|
61
61
|
Inside `optional_fields`, any field with `may_become_required=true` is still writable, but may become required when linked visibility or option-driven runtime rules activate.
|
|
62
62
|
`record_update_schema_get` returns the current record's overall update-ready writable field set across matched accessible views; read `writable_fields` and `payload_template`.
|
|
63
|
-
`record_browse_schema_get(view_id=...)` returns
|
|
63
|
+
`record_browse_schema_get(view_id=...)` returns the same readable fields shown in the selected Qingflow table view header.
|
|
64
|
+
`record_access.fields` and its `schema.json` use that exact same view schema; a missing field means it is not readable in that view.
|
|
65
|
+
`searchQueIds` is a backend full-text search scope, not an output-column/projection mechanism.
|
|
64
66
|
`record_code_block_schema_get` returns code-block-ready schema for exact code block field selection.
|
|
65
67
|
`record_import_schema_get` returns import-ready column metadata.
|
|
66
68
|
|
|
@@ -75,6 +77,7 @@ Inside `optional_fields`, any field with `may_become_required=true` is still wri
|
|
|
75
77
|
Prefer `view_id` entries from `accessible_views` where `analysis_supported=true`.
|
|
76
78
|
|
|
77
79
|
Use `record_access` to write local CSV shard files, then use Python to compute counts, rankings, ratios, trends, and final conclusions. `record_access` does not return bulk `items`; read `files[].local_path`. Use `fields[]` and the same-directory `metadata_files.schema` / `metadata_files.readme` to map stable `field_<id>` CSV columns back to titles and types.
|
|
80
|
+
For analysis-style tasks, prefer an explicit time range or business filter. If `record_access.status == "needs_scope"`, do not treat it as a failure; ask for a time/business scope or retry with a user-provided period using `scope.suggested_time_fields` / `scope.recommended_where_examples`. If `record_access.status == "partial"`, read the returned files only as a limited subset and do not give a final full-population conclusion.
|
|
78
81
|
Use `chart_get` only when the user provides a report URL / chart_id or explicitly asks to read an existing report. Do not use QingBI as the default analysis route.
|
|
79
82
|
|
|
80
83
|
Use this data-access DSL shape:
|
|
@@ -7,7 +7,7 @@ import re
|
|
|
7
7
|
import time
|
|
8
8
|
from copy import deepcopy
|
|
9
9
|
from dataclasses import dataclass
|
|
10
|
-
from datetime import UTC, datetime
|
|
10
|
+
from datetime import UTC, datetime, timedelta
|
|
11
11
|
from decimal import Decimal, InvalidOperation
|
|
12
12
|
from pathlib import Path
|
|
13
13
|
from typing import Any, cast
|
|
@@ -34,6 +34,9 @@ DEFAULT_QUERY_PAGE_SIZE = 50
|
|
|
34
34
|
DEFAULT_LIST_PAGE_SIZE = 200
|
|
35
35
|
BACKEND_RECORD_ACCESS_PAGE_SIZE = 1000
|
|
36
36
|
DEFAULT_RECORD_ACCESS_SHARD_ROWS = 5000
|
|
37
|
+
RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD = 20_000
|
|
38
|
+
RECORD_ACCESS_TIME_BUDGET_SECONDS = 55.0
|
|
39
|
+
RECORD_ACCESS_MIN_REMAINING_SECONDS = 8.0
|
|
37
40
|
DEFAULT_ANALYSIS_PAGE_SIZE = 1000
|
|
38
41
|
DEFAULT_SCAN_MAX_PAGES = 10
|
|
39
42
|
DEFAULT_ANALYSIS_SCAN_MAX_PAGES = 100
|
|
@@ -602,7 +605,7 @@ class RecordTools(ToolBase):
|
|
|
602
605
|
),
|
|
603
606
|
}
|
|
604
607
|
)
|
|
605
|
-
browse_scope = self.
|
|
608
|
+
browse_scope = self._build_browse_read_scope(
|
|
606
609
|
profile,
|
|
607
610
|
context,
|
|
608
611
|
app_key,
|
|
@@ -1848,25 +1851,21 @@ class RecordTools(ToolBase):
|
|
|
1848
1851
|
sorts = self._normalize_record_list_order_by(order_by)
|
|
1849
1852
|
|
|
1850
1853
|
def runner(session_profile, context):
|
|
1851
|
-
|
|
1852
|
-
|
|
1854
|
+
created_at = datetime.now(UTC).isoformat()
|
|
1855
|
+
browse_scope = self._build_browse_read_scope(
|
|
1856
|
+
profile,
|
|
1857
|
+
context,
|
|
1858
|
+
app_key,
|
|
1859
|
+
view_route,
|
|
1860
|
+
force_refresh=False,
|
|
1861
|
+
)
|
|
1862
|
+
index = cast(FieldIndex, browse_scope["index"])
|
|
1863
|
+
selected_fields = self._resolve_record_access_columns(
|
|
1853
1864
|
normalized_columns,
|
|
1854
1865
|
index,
|
|
1855
|
-
|
|
1856
|
-
default_limit=MAX_LIST_COLUMN_LIMIT,
|
|
1866
|
+
view_route=view_route,
|
|
1857
1867
|
)
|
|
1858
|
-
selected_field_batches = _chunk_fields(selected_fields, BACKEND_LIST_SEARCH_FIELD_LIMIT)
|
|
1859
|
-
primary_search_que_ids: list[int] | None = [field.que_id for field in selected_field_batches[0]]
|
|
1860
1868
|
view_selection = view_route.view_selection
|
|
1861
|
-
if view_selection is not None and not _view_selection_supported_by_search_ids(
|
|
1862
|
-
view_selection,
|
|
1863
|
-
primary_search_que_ids,
|
|
1864
|
-
):
|
|
1865
|
-
primary_search_que_ids = None
|
|
1866
|
-
remaining_field_batches: list[list[FormField]] = []
|
|
1867
|
-
else:
|
|
1868
|
-
remaining_field_batches = selected_field_batches[1:]
|
|
1869
|
-
primary_search_que_ids = primary_search_que_ids or None
|
|
1870
1869
|
match_rules = self._resolve_match_rules(context, filters, index)
|
|
1871
1870
|
sort_rules = self._resolve_sorts(sorts, index)
|
|
1872
1871
|
used_list_type: int | None = None
|
|
@@ -1875,6 +1874,106 @@ class RecordTools(ToolBase):
|
|
|
1875
1874
|
if view_selection is not None or view_route.list_type is not None
|
|
1876
1875
|
else [DEFAULT_RECORD_LIST_TYPE, 14, 1, 2, 12]
|
|
1877
1876
|
)
|
|
1877
|
+
fields_payload = [_record_access_field_payload(field) for field in selected_fields]
|
|
1878
|
+
scope_status = _record_access_scope_status(filters, view_route, index)
|
|
1879
|
+
|
|
1880
|
+
def fetch_page(page_num: int, page_size: int) -> JSONObject:
|
|
1881
|
+
nonlocal used_list_type
|
|
1882
|
+
if used_list_type is None:
|
|
1883
|
+
last_error: QingflowApiError | None = None
|
|
1884
|
+
page: JSONObject | None = None
|
|
1885
|
+
for candidate_list_type in fallback_list_types:
|
|
1886
|
+
try:
|
|
1887
|
+
page = self._search_page(
|
|
1888
|
+
context,
|
|
1889
|
+
app_key=app_key,
|
|
1890
|
+
view_selection=view_selection,
|
|
1891
|
+
page_num=page_num,
|
|
1892
|
+
page_size=page_size,
|
|
1893
|
+
query_key=None,
|
|
1894
|
+
match_rules=match_rules,
|
|
1895
|
+
sorts=sort_rules,
|
|
1896
|
+
search_que_ids=None,
|
|
1897
|
+
list_type=candidate_list_type,
|
|
1898
|
+
)
|
|
1899
|
+
used_list_type = None if view_selection is not None else candidate_list_type
|
|
1900
|
+
break
|
|
1901
|
+
except QingflowApiError as exc:
|
|
1902
|
+
last_error = exc
|
|
1903
|
+
if self._should_retry_list_type_fallback(exc) and candidate_list_type != fallback_list_types[-1]:
|
|
1904
|
+
continue
|
|
1905
|
+
raise
|
|
1906
|
+
if page is None:
|
|
1907
|
+
if last_error is not None:
|
|
1908
|
+
raise last_error
|
|
1909
|
+
raise_tool_error(QingflowApiError.config_error("record_access failed: no accessible listType"))
|
|
1910
|
+
return page
|
|
1911
|
+
return self._search_page(
|
|
1912
|
+
context,
|
|
1913
|
+
app_key=app_key,
|
|
1914
|
+
view_selection=view_selection,
|
|
1915
|
+
page_num=page_num,
|
|
1916
|
+
page_size=page_size,
|
|
1917
|
+
query_key=None,
|
|
1918
|
+
match_rules=match_rules,
|
|
1919
|
+
sorts=sort_rules,
|
|
1920
|
+
search_que_ids=None,
|
|
1921
|
+
list_type=used_list_type,
|
|
1922
|
+
)
|
|
1923
|
+
|
|
1924
|
+
probe_page_size = 1 if _record_access_light_probe_recommended(scope_status) else BACKEND_RECORD_ACCESS_PAGE_SIZE
|
|
1925
|
+
probe_page = fetch_page(1, probe_page_size)
|
|
1926
|
+
reported_total = _effective_total(probe_page, probe_page_size)
|
|
1927
|
+
estimated_pages = _record_access_estimated_pages(probe_page, reported_total, page_size=probe_page_size)
|
|
1928
|
+
scope_payload = _record_access_scope_payload(
|
|
1929
|
+
scope_status=scope_status,
|
|
1930
|
+
reported_total=reported_total,
|
|
1931
|
+
estimated_pages=estimated_pages,
|
|
1932
|
+
index=index,
|
|
1933
|
+
)
|
|
1934
|
+
warnings: list[JSONObject] = []
|
|
1935
|
+
warnings.extend(legacy_warnings)
|
|
1936
|
+
warnings.extend(compatibility_warnings)
|
|
1937
|
+
warnings.extend(_view_filter_trust_warnings(view_route))
|
|
1938
|
+
if used_list_type is not None and used_list_type != DEFAULT_RECORD_LIST_TYPE:
|
|
1939
|
+
warnings.append(
|
|
1940
|
+
{
|
|
1941
|
+
"code": "LIST_TYPE_FALLBACK",
|
|
1942
|
+
"message": (
|
|
1943
|
+
f"record_access not accessible via listType={DEFAULT_RECORD_LIST_TYPE}; "
|
|
1944
|
+
f"fell back to listType={used_list_type} ({get_record_list_type_label(used_list_type)})."
|
|
1945
|
+
),
|
|
1946
|
+
}
|
|
1947
|
+
)
|
|
1948
|
+
if _record_access_needs_scope(scope_status=scope_status, reported_total=reported_total):
|
|
1949
|
+
warnings.append(_record_access_unbounded_scan_warning(reported_total=reported_total, estimated_pages=estimated_pages))
|
|
1950
|
+
verification_payload: JSONObject = {
|
|
1951
|
+
**_view_filter_verification_payload(view_route),
|
|
1952
|
+
"reported_total": reported_total,
|
|
1953
|
+
"estimated_pages": estimated_pages,
|
|
1954
|
+
"fetched_pages": 1,
|
|
1955
|
+
"stopped_reason": "UNBOUNDED_SCAN_TOO_LARGE",
|
|
1956
|
+
"list_type_used": used_list_type,
|
|
1957
|
+
}
|
|
1958
|
+
return {
|
|
1959
|
+
"profile": profile,
|
|
1960
|
+
"ws_id": session_profile.selected_ws_id,
|
|
1961
|
+
"ok": True,
|
|
1962
|
+
"status": "needs_scope",
|
|
1963
|
+
"app_key": app_key,
|
|
1964
|
+
"view_id": view_route.view_id,
|
|
1965
|
+
"format": "csv",
|
|
1966
|
+
"row_count": 0,
|
|
1967
|
+
"complete": False,
|
|
1968
|
+
"truncated": True,
|
|
1969
|
+
"safe_for_final_conclusion": False,
|
|
1970
|
+
"files": [],
|
|
1971
|
+
"fields": fields_payload,
|
|
1972
|
+
"warnings": warnings,
|
|
1973
|
+
"verification": verification_payload,
|
|
1974
|
+
"scope": scope_payload,
|
|
1975
|
+
"request_route": self._request_route_payload(context),
|
|
1976
|
+
}
|
|
1878
1977
|
|
|
1879
1978
|
run_dir = _record_access_run_dir()
|
|
1880
1979
|
run_dir.mkdir(parents=True, exist_ok=True)
|
|
@@ -1931,58 +2030,14 @@ class RecordTools(ToolBase):
|
|
|
1931
2030
|
|
|
1932
2031
|
current_page = 1
|
|
1933
2032
|
has_more = False
|
|
1934
|
-
|
|
1935
|
-
|
|
2033
|
+
fetched_pages = 0
|
|
2034
|
+
stopped_reason: str | None = None
|
|
2035
|
+
deadline = time.monotonic() + RECORD_ACCESS_TIME_BUDGET_SECONDS
|
|
2036
|
+
page = probe_page if probe_page_size == BACKEND_RECORD_ACCESS_PAGE_SIZE else fetch_page(1, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
1936
2037
|
try:
|
|
1937
2038
|
while True:
|
|
1938
|
-
if used_list_type is None:
|
|
1939
|
-
last_error: QingflowApiError | None = None
|
|
1940
|
-
page: JSONObject | None = None
|
|
1941
|
-
for candidate_list_type in fallback_list_types:
|
|
1942
|
-
try:
|
|
1943
|
-
page = self._search_page(
|
|
1944
|
-
context,
|
|
1945
|
-
app_key=app_key,
|
|
1946
|
-
view_selection=view_selection,
|
|
1947
|
-
page_num=current_page,
|
|
1948
|
-
page_size=BACKEND_RECORD_ACCESS_PAGE_SIZE,
|
|
1949
|
-
query_key=None,
|
|
1950
|
-
match_rules=match_rules,
|
|
1951
|
-
sorts=sort_rules,
|
|
1952
|
-
search_que_ids=primary_search_que_ids,
|
|
1953
|
-
list_type=candidate_list_type,
|
|
1954
|
-
)
|
|
1955
|
-
used_list_type = None if view_selection is not None else candidate_list_type
|
|
1956
|
-
break
|
|
1957
|
-
except QingflowApiError as exc:
|
|
1958
|
-
last_error = exc
|
|
1959
|
-
if (
|
|
1960
|
-
self._should_retry_list_type_fallback(exc)
|
|
1961
|
-
and candidate_list_type != fallback_list_types[-1]
|
|
1962
|
-
):
|
|
1963
|
-
continue
|
|
1964
|
-
raise
|
|
1965
|
-
if page is None:
|
|
1966
|
-
if last_error is not None:
|
|
1967
|
-
raise last_error
|
|
1968
|
-
raise_tool_error(QingflowApiError.config_error("record_access failed: no accessible listType"))
|
|
1969
|
-
else:
|
|
1970
|
-
page = self._search_page(
|
|
1971
|
-
context,
|
|
1972
|
-
app_key=app_key,
|
|
1973
|
-
view_selection=view_selection,
|
|
1974
|
-
page_num=current_page,
|
|
1975
|
-
page_size=BACKEND_RECORD_ACCESS_PAGE_SIZE,
|
|
1976
|
-
query_key=None,
|
|
1977
|
-
match_rules=match_rules,
|
|
1978
|
-
sorts=sort_rules,
|
|
1979
|
-
search_que_ids=primary_search_que_ids,
|
|
1980
|
-
list_type=used_list_type,
|
|
1981
|
-
)
|
|
1982
2039
|
page_rows = page.get("list")
|
|
1983
2040
|
items = page_rows if isinstance(page_rows, list) else []
|
|
1984
|
-
if reported_total is None:
|
|
1985
|
-
reported_total = _effective_total(page, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
1986
2041
|
has_more = _page_has_more(page, current_page, BACKEND_RECORD_ACCESS_PAGE_SIZE, len(items))
|
|
1987
2042
|
page_apply_order: list[int] = []
|
|
1988
2043
|
page_answer_map: dict[int, list[JSONValue]] = {}
|
|
@@ -1996,64 +2051,35 @@ class RecordTools(ToolBase):
|
|
|
1996
2051
|
continue
|
|
1997
2052
|
page_apply_order.append(apply_id)
|
|
1998
2053
|
page_answer_map[apply_id] = cast(list[JSONValue], answer_list)
|
|
1999
|
-
if page_apply_order and remaining_field_batches:
|
|
2000
|
-
for batch in remaining_field_batches:
|
|
2001
|
-
extra_page = self._search_page(
|
|
2002
|
-
context,
|
|
2003
|
-
app_key=app_key,
|
|
2004
|
-
view_selection=view_selection,
|
|
2005
|
-
page_num=current_page,
|
|
2006
|
-
page_size=BACKEND_RECORD_ACCESS_PAGE_SIZE,
|
|
2007
|
-
query_key=None,
|
|
2008
|
-
match_rules=match_rules,
|
|
2009
|
-
sorts=sort_rules,
|
|
2010
|
-
search_que_ids=[field.que_id for field in batch],
|
|
2011
|
-
list_type=used_list_type or DEFAULT_RECORD_LIST_TYPE,
|
|
2012
|
-
)
|
|
2013
|
-
extra_rows = extra_page.get("list")
|
|
2014
|
-
extra_items = extra_rows if isinstance(extra_rows, list) else []
|
|
2015
|
-
for extra_item in extra_items:
|
|
2016
|
-
if not isinstance(extra_item, dict):
|
|
2017
|
-
continue
|
|
2018
|
-
apply_id = _coerce_count(extra_item.get("applyId")) or _coerce_count(extra_item.get("id"))
|
|
2019
|
-
if apply_id is None or apply_id not in page_answer_map:
|
|
2020
|
-
continue
|
|
2021
|
-
extra_answers = extra_item.get("answers")
|
|
2022
|
-
extra_answer_list = extra_answers if isinstance(extra_answers, list) else []
|
|
2023
|
-
page_answer_map[apply_id] = _merge_answer_lists_by_field_id(
|
|
2024
|
-
page_answer_map.get(apply_id, []),
|
|
2025
|
-
cast(list[JSONValue], extra_answer_list),
|
|
2026
|
-
)
|
|
2027
2054
|
for apply_id in page_apply_order:
|
|
2028
2055
|
write_record(apply_id, page_answer_map.get(apply_id, []))
|
|
2056
|
+
fetched_pages += 1
|
|
2029
2057
|
if not has_more:
|
|
2030
2058
|
break
|
|
2031
2059
|
current_page += 1
|
|
2060
|
+
if _record_access_time_budget_exceeded(
|
|
2061
|
+
deadline,
|
|
2062
|
+
fetched_pages=fetched_pages,
|
|
2063
|
+
estimated_pages=estimated_pages,
|
|
2064
|
+
):
|
|
2065
|
+
stopped_reason = "TIME_BUDGET_EXCEEDED"
|
|
2066
|
+
break
|
|
2067
|
+
page = fetch_page(current_page, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
2032
2068
|
finally:
|
|
2033
2069
|
close_shard()
|
|
2034
2070
|
|
|
2035
|
-
|
|
2036
|
-
|
|
2037
|
-
|
|
2038
|
-
warnings.extend(_view_filter_trust_warnings(view_route))
|
|
2039
|
-
if used_list_type is not None and used_list_type != DEFAULT_RECORD_LIST_TYPE:
|
|
2040
|
-
warnings.append(
|
|
2041
|
-
{
|
|
2042
|
-
"code": "LIST_TYPE_FALLBACK",
|
|
2043
|
-
"message": (
|
|
2044
|
-
f"record_access not accessible via listType={DEFAULT_RECORD_LIST_TYPE}; "
|
|
2045
|
-
f"fell back to listType={used_list_type} ({get_record_list_type_label(used_list_type)})."
|
|
2046
|
-
),
|
|
2047
|
-
}
|
|
2048
|
-
)
|
|
2049
|
-
complete = not has_more
|
|
2071
|
+
if stopped_reason == "TIME_BUDGET_EXCEEDED":
|
|
2072
|
+
warnings.append(_record_access_time_budget_warning(reported_total=reported_total, fetched_pages=fetched_pages))
|
|
2073
|
+
complete = stopped_reason is None and not has_more
|
|
2050
2074
|
safe_for_final_conclusion = complete and not any(
|
|
2051
2075
|
warning.get("code") == "CUSTOM_VIEW_FILTER_UNVERIFIED" for warning in warnings
|
|
2052
2076
|
)
|
|
2053
|
-
fields_payload = [_record_access_field_payload(field) for field in selected_fields]
|
|
2054
2077
|
verification_payload: JSONObject = {
|
|
2055
2078
|
**_view_filter_verification_payload(view_route),
|
|
2056
2079
|
"reported_total": reported_total,
|
|
2080
|
+
"estimated_pages": estimated_pages,
|
|
2081
|
+
"fetched_pages": fetched_pages,
|
|
2082
|
+
"stopped_reason": stopped_reason,
|
|
2057
2083
|
"list_type_used": used_list_type,
|
|
2058
2084
|
}
|
|
2059
2085
|
metadata_files = _write_record_access_metadata_files(
|
|
@@ -2068,6 +2094,7 @@ class RecordTools(ToolBase):
|
|
|
2068
2094
|
fields=fields_payload,
|
|
2069
2095
|
warnings=warnings,
|
|
2070
2096
|
verification=verification_payload,
|
|
2097
|
+
scope=scope_payload,
|
|
2071
2098
|
)
|
|
2072
2099
|
return {
|
|
2073
2100
|
"profile": profile,
|
|
@@ -2087,6 +2114,7 @@ class RecordTools(ToolBase):
|
|
|
2087
2114
|
"fields": fields_payload,
|
|
2088
2115
|
"warnings": warnings,
|
|
2089
2116
|
"verification": verification_payload,
|
|
2117
|
+
"scope": scope_payload,
|
|
2090
2118
|
"request_route": self._request_route_payload(context),
|
|
2091
2119
|
}
|
|
2092
2120
|
|
|
@@ -7427,6 +7455,39 @@ class RecordTools(ToolBase):
|
|
|
7427
7455
|
self._form_cache[cache_key] = normalized
|
|
7428
7456
|
return normalized
|
|
7429
7457
|
|
|
7458
|
+
def _get_system_browse_base_info_schema(
|
|
7459
|
+
self,
|
|
7460
|
+
profile: str,
|
|
7461
|
+
context, # type: ignore[no-untyped-def]
|
|
7462
|
+
app_key: str,
|
|
7463
|
+
*,
|
|
7464
|
+
list_type: int,
|
|
7465
|
+
force_refresh: bool,
|
|
7466
|
+
) -> JSONObject:
|
|
7467
|
+
"""Return system view apply/baseInfo without falling back to app form metadata."""
|
|
7468
|
+
cache_key = (profile, app_key, "browse_system_base_info", list_type)
|
|
7469
|
+
if not force_refresh and cache_key in self._form_cache:
|
|
7470
|
+
return self._form_cache[cache_key]
|
|
7471
|
+
payload = self.backend.request(
|
|
7472
|
+
"GET",
|
|
7473
|
+
context,
|
|
7474
|
+
f"/app/{app_key}/apply/baseInfo",
|
|
7475
|
+
params={"type": list_type},
|
|
7476
|
+
)
|
|
7477
|
+
normalized = _normalize_data_list_base_info_schema(payload)
|
|
7478
|
+
self._form_cache[cache_key] = normalized
|
|
7479
|
+
return normalized
|
|
7480
|
+
|
|
7481
|
+
def _get_custom_view_browse_schema(self, profile: str, context, view_key: str, *, force_refresh: bool) -> JSONObject: # type: ignore[no-untyped-def]
|
|
7482
|
+
"""Return the same baseInfo schema used by the Qingflow table view UI."""
|
|
7483
|
+
cache_key = (profile, f"view:{view_key}", "browse_view_base_info", None)
|
|
7484
|
+
if not force_refresh and cache_key in self._form_cache:
|
|
7485
|
+
return self._form_cache[cache_key]
|
|
7486
|
+
payload = self.backend.request("GET", context, f"/view/{view_key}/apply/baseInfo")
|
|
7487
|
+
normalized = _normalize_data_list_base_info_schema(payload)
|
|
7488
|
+
self._form_cache[cache_key] = normalized
|
|
7489
|
+
return normalized
|
|
7490
|
+
|
|
7430
7491
|
def _get_view_field_index(self, profile: str, context, view_key: str, *, force_refresh: bool) -> FieldIndex: # type: ignore[no-untyped-def]
|
|
7431
7492
|
"""执行内部辅助逻辑。"""
|
|
7432
7493
|
return _build_field_index(self._get_view_form_schema(profile, context, view_key, force_refresh=force_refresh))
|
|
@@ -7469,6 +7530,80 @@ class RecordTools(ToolBase):
|
|
|
7469
7530
|
force_refresh=force_refresh,
|
|
7470
7531
|
)["index"]
|
|
7471
7532
|
|
|
7533
|
+
def _build_browse_read_scope(
|
|
7534
|
+
self,
|
|
7535
|
+
profile: str,
|
|
7536
|
+
context, # type: ignore[no-untyped-def]
|
|
7537
|
+
app_key: str,
|
|
7538
|
+
resolved_view: AccessibleViewRoute | None,
|
|
7539
|
+
*,
|
|
7540
|
+
force_refresh: bool,
|
|
7541
|
+
) -> JSONObject:
|
|
7542
|
+
"""Build the UI/table-view readable field scope from apply/baseInfo."""
|
|
7543
|
+
applicant_index: FieldIndex | None
|
|
7544
|
+
applicant_writable_field_ids: set[int]
|
|
7545
|
+
try:
|
|
7546
|
+
applicant_index = self._get_applicant_top_level_field_index(profile, context, app_key, force_refresh=force_refresh)
|
|
7547
|
+
except QingflowApiError as exc:
|
|
7548
|
+
if exc.backend_code != 40002:
|
|
7549
|
+
raise
|
|
7550
|
+
applicant_index = None
|
|
7551
|
+
applicant_writable_field_ids = set()
|
|
7552
|
+
else:
|
|
7553
|
+
applicant_writable_field_ids = {
|
|
7554
|
+
field.que_id
|
|
7555
|
+
for field in applicant_index.by_id.values()
|
|
7556
|
+
if bool(self._schema_write_hints(field)["writable"])
|
|
7557
|
+
}
|
|
7558
|
+
|
|
7559
|
+
if resolved_view is not None and resolved_view.kind == "custom" and resolved_view.view_selection is not None:
|
|
7560
|
+
schema = self._get_custom_view_browse_schema(
|
|
7561
|
+
profile,
|
|
7562
|
+
context,
|
|
7563
|
+
resolved_view.view_selection.view_key,
|
|
7564
|
+
force_refresh=force_refresh,
|
|
7565
|
+
)
|
|
7566
|
+
index = _build_top_level_field_index(schema)
|
|
7567
|
+
elif resolved_view is not None and resolved_view.kind == "system" and resolved_view.list_type is not None:
|
|
7568
|
+
schema = self._get_system_browse_base_info_schema(
|
|
7569
|
+
profile,
|
|
7570
|
+
context,
|
|
7571
|
+
app_key,
|
|
7572
|
+
list_type=resolved_view.list_type,
|
|
7573
|
+
force_refresh=force_refresh,
|
|
7574
|
+
)
|
|
7575
|
+
index = _build_top_level_field_index(schema)
|
|
7576
|
+
else:
|
|
7577
|
+
index = applicant_index or _build_top_level_field_index(
|
|
7578
|
+
self._get_form_schema(profile, context, app_key, force_refresh=force_refresh)
|
|
7579
|
+
)
|
|
7580
|
+
|
|
7581
|
+
if applicant_index is not None and index.by_id:
|
|
7582
|
+
enriched_fields = [
|
|
7583
|
+
_enrich_read_field_from_applicant(field, applicant_index.by_id.get(str(field.que_id)))
|
|
7584
|
+
for field in index.by_id.values()
|
|
7585
|
+
]
|
|
7586
|
+
index = _build_top_level_field_index({"formQues": [[_form_field_to_question(field) for field in enriched_fields]]})
|
|
7587
|
+
|
|
7588
|
+
visible_question_ids = {field.que_id for field in index.by_id.values()}
|
|
7589
|
+
if applicant_index is None:
|
|
7590
|
+
writable_field_ids = {
|
|
7591
|
+
field.que_id
|
|
7592
|
+
for field in index.by_id.values()
|
|
7593
|
+
if bool(self._schema_write_hints(field)["writable"])
|
|
7594
|
+
}
|
|
7595
|
+
else:
|
|
7596
|
+
writable_field_ids = {
|
|
7597
|
+
field_id
|
|
7598
|
+
for field_id in visible_question_ids
|
|
7599
|
+
if field_id in applicant_writable_field_ids
|
|
7600
|
+
}
|
|
7601
|
+
return {
|
|
7602
|
+
"index": index,
|
|
7603
|
+
"writable_field_ids": writable_field_ids,
|
|
7604
|
+
"visible_question_ids": visible_question_ids,
|
|
7605
|
+
}
|
|
7606
|
+
|
|
7472
7607
|
def _build_browse_write_scope(
|
|
7473
7608
|
self,
|
|
7474
7609
|
profile: str,
|
|
@@ -9538,6 +9673,44 @@ class RecordTools(ToolBase):
|
|
|
9538
9673
|
break
|
|
9539
9674
|
return fields
|
|
9540
9675
|
|
|
9676
|
+
def _resolve_record_access_columns(
|
|
9677
|
+
self,
|
|
9678
|
+
selectors: list[int],
|
|
9679
|
+
index: FieldIndex,
|
|
9680
|
+
*,
|
|
9681
|
+
view_route: AccessibleViewRoute,
|
|
9682
|
+
) -> list[FormField]:
|
|
9683
|
+
"""Resolve record_access columns against the selected view's baseInfo schema."""
|
|
9684
|
+
if not selectors:
|
|
9685
|
+
raise_tool_error(QingflowApiError.config_error("columns is required"))
|
|
9686
|
+
fields: list[FormField] = []
|
|
9687
|
+
seen: set[int] = set()
|
|
9688
|
+
for selector in selectors:
|
|
9689
|
+
try:
|
|
9690
|
+
field = self._resolve_field_selector(selector, index, location="record_access.columns")
|
|
9691
|
+
except RecordInputError as exc:
|
|
9692
|
+
if exc.error_code == "FIELD_NOT_FOUND":
|
|
9693
|
+
raise RecordInputError(
|
|
9694
|
+
message=(
|
|
9695
|
+
f"record_access column field_id '{selector}' is not in the selected view schema "
|
|
9696
|
+
f"({view_route.view_id})."
|
|
9697
|
+
),
|
|
9698
|
+
error_code="FIELD_NOT_IN_VIEW_SCHEMA",
|
|
9699
|
+
fix_hint="Call record_browse_schema_get for this exact view_id and pass only field_id values from its fields[].",
|
|
9700
|
+
details={
|
|
9701
|
+
"location": "record_access.columns",
|
|
9702
|
+
"requested": selector,
|
|
9703
|
+
"view_id": view_route.view_id,
|
|
9704
|
+
"view_name": view_route.name,
|
|
9705
|
+
},
|
|
9706
|
+
) from exc
|
|
9707
|
+
raise
|
|
9708
|
+
if field.que_id in seen:
|
|
9709
|
+
continue
|
|
9710
|
+
fields.append(field)
|
|
9711
|
+
seen.add(field.que_id)
|
|
9712
|
+
return fields
|
|
9713
|
+
|
|
9541
9714
|
def _resolve_summary_preview_fields(
|
|
9542
9715
|
self,
|
|
9543
9716
|
selectors: list[str | int],
|
|
@@ -11295,6 +11468,38 @@ def _clone_form_field(field: FormField, *, readonly: bool | None = None) -> Form
|
|
|
11295
11468
|
)
|
|
11296
11469
|
|
|
11297
11470
|
|
|
11471
|
+
def _enrich_read_field_from_applicant(field: FormField, applicant_field: FormField | None) -> FormField:
|
|
11472
|
+
if applicant_field is None:
|
|
11473
|
+
return field
|
|
11474
|
+
raw = dict(applicant_field.raw) if isinstance(applicant_field.raw, dict) else {}
|
|
11475
|
+
raw.update(dict(field.raw) if isinstance(field.raw, dict) else {})
|
|
11476
|
+
raw["queId"] = field.que_id
|
|
11477
|
+
raw["queTitle"] = field.que_title
|
|
11478
|
+
que_type = field.que_type if field.que_type is not None else applicant_field.que_type
|
|
11479
|
+
if que_type is not None:
|
|
11480
|
+
raw["queType"] = que_type
|
|
11481
|
+
readonly_source = field if any(key in field.raw for key in ("readonly", "beingReadonly", "canEdit")) else applicant_field
|
|
11482
|
+
enriched = FormField(
|
|
11483
|
+
que_id=field.que_id,
|
|
11484
|
+
que_title=field.que_title,
|
|
11485
|
+
que_type=que_type,
|
|
11486
|
+
required=field.required or applicant_field.required,
|
|
11487
|
+
readonly=readonly_source.readonly,
|
|
11488
|
+
system=field.system or applicant_field.system,
|
|
11489
|
+
options=list(field.options or applicant_field.options),
|
|
11490
|
+
aliases=[],
|
|
11491
|
+
target_app_key=field.target_app_key or applicant_field.target_app_key,
|
|
11492
|
+
target_app_name_hint=field.target_app_name_hint or applicant_field.target_app_name_hint,
|
|
11493
|
+
member_select_scope_type=field.member_select_scope_type if field.member_select_scope_type is not None else applicant_field.member_select_scope_type,
|
|
11494
|
+
member_select_scope=field.member_select_scope or applicant_field.member_select_scope,
|
|
11495
|
+
dept_select_scope_type=field.dept_select_scope_type if field.dept_select_scope_type is not None else applicant_field.dept_select_scope_type,
|
|
11496
|
+
dept_select_scope=field.dept_select_scope or applicant_field.dept_select_scope,
|
|
11497
|
+
raw=raw,
|
|
11498
|
+
)
|
|
11499
|
+
enriched.aliases = sorted(_field_alias_candidates(enriched))
|
|
11500
|
+
return enriched
|
|
11501
|
+
|
|
11502
|
+
|
|
11298
11503
|
def _form_field_to_question(field: FormField) -> JSONObject:
|
|
11299
11504
|
question = dict(field.raw) if isinstance(field.raw, dict) else {}
|
|
11300
11505
|
question.setdefault("queId", field.que_id)
|
|
@@ -11521,6 +11726,148 @@ def _record_access_field_payload(field: FormField) -> JSONObject:
|
|
|
11521
11726
|
}
|
|
11522
11727
|
|
|
11523
11728
|
|
|
11729
|
+
def _record_access_scope_status(filters: list[JSONObject], view_route: AccessibleViewRoute, index: FieldIndex) -> JSONObject:
|
|
11730
|
+
has_time_filter = False
|
|
11731
|
+
has_explicit_business_filter = False
|
|
11732
|
+
for item in filters:
|
|
11733
|
+
if not isinstance(item, dict):
|
|
11734
|
+
continue
|
|
11735
|
+
field_id = _coerce_count(item.get("field_id", item.get("fieldId")))
|
|
11736
|
+
field = index.by_id.get(str(field_id)) if field_id is not None else None
|
|
11737
|
+
if field is not None and field.que_type in DATE_QUE_TYPES:
|
|
11738
|
+
has_time_filter = True
|
|
11739
|
+
else:
|
|
11740
|
+
has_explicit_business_filter = True
|
|
11741
|
+
view_selection = view_route.view_selection
|
|
11742
|
+
has_saved_view_filter = bool(view_selection is not None and view_selection.conditions)
|
|
11743
|
+
return {
|
|
11744
|
+
"filter_count": len(filters),
|
|
11745
|
+
"has_explicit_where": bool(filters),
|
|
11746
|
+
"has_time_filter": has_time_filter,
|
|
11747
|
+
"has_business_filter": bool(has_explicit_business_filter or has_saved_view_filter),
|
|
11748
|
+
"has_saved_view_filter": has_saved_view_filter,
|
|
11749
|
+
}
|
|
11750
|
+
|
|
11751
|
+
|
|
11752
|
+
def _record_access_light_probe_recommended(scope_status: JSONObject) -> bool:
|
|
11753
|
+
return not bool(scope_status.get("has_time_filter")) and not bool(scope_status.get("has_business_filter"))
|
|
11754
|
+
|
|
11755
|
+
|
|
11756
|
+
def _record_access_estimated_pages(page: JSONObject, reported_total: int | None, *, page_size: int) -> int | None:
|
|
11757
|
+
page_amount = _coerce_count(page.get("pageAmount"))
|
|
11758
|
+
if page_size == BACKEND_RECORD_ACCESS_PAGE_SIZE and page_amount is not None:
|
|
11759
|
+
return page_amount
|
|
11760
|
+
if reported_total is None or reported_total <= 0:
|
|
11761
|
+
return page_amount
|
|
11762
|
+
return (reported_total + BACKEND_RECORD_ACCESS_PAGE_SIZE - 1) // BACKEND_RECORD_ACCESS_PAGE_SIZE
|
|
11763
|
+
|
|
11764
|
+
|
|
11765
|
+
def _record_access_suggested_time_fields(index: FieldIndex) -> list[JSONObject]:
|
|
11766
|
+
fields: list[JSONObject] = []
|
|
11767
|
+
for field in index.by_id.values():
|
|
11768
|
+
if field.que_type in DATE_QUE_TYPES:
|
|
11769
|
+
fields.append(_record_access_field_payload(field))
|
|
11770
|
+
return fields
|
|
11771
|
+
|
|
11772
|
+
|
|
11773
|
+
def _record_access_recommended_where_examples(suggested_time_fields: list[JSONObject]) -> list[JSONObject]:
|
|
11774
|
+
if not suggested_time_fields:
|
|
11775
|
+
return []
|
|
11776
|
+
now = datetime.now(UTC)
|
|
11777
|
+
month_start = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
|
|
11778
|
+
if month_start.month == 12:
|
|
11779
|
+
next_month = month_start.replace(year=month_start.year + 1, month=1)
|
|
11780
|
+
else:
|
|
11781
|
+
next_month = month_start.replace(month=month_start.month + 1)
|
|
11782
|
+
month_end = next_month - timedelta(seconds=1)
|
|
11783
|
+
examples: list[JSONObject] = []
|
|
11784
|
+
for field in suggested_time_fields[:3]:
|
|
11785
|
+
field_id = field.get("field_id")
|
|
11786
|
+
examples.append(
|
|
11787
|
+
{
|
|
11788
|
+
"field_id": field_id,
|
|
11789
|
+
"title": field.get("title"),
|
|
11790
|
+
"where": [
|
|
11791
|
+
{
|
|
11792
|
+
"field_id": field_id,
|
|
11793
|
+
"op": "between",
|
|
11794
|
+
"value": [
|
|
11795
|
+
month_start.strftime("%Y-%m-%d"),
|
|
11796
|
+
month_end.strftime("%Y-%m-%d %H:%M:%S"),
|
|
11797
|
+
],
|
|
11798
|
+
}
|
|
11799
|
+
],
|
|
11800
|
+
}
|
|
11801
|
+
)
|
|
11802
|
+
return examples
|
|
11803
|
+
|
|
11804
|
+
|
|
11805
|
+
def _record_access_scope_payload(
|
|
11806
|
+
*,
|
|
11807
|
+
scope_status: JSONObject,
|
|
11808
|
+
reported_total: int | None,
|
|
11809
|
+
estimated_pages: int | None,
|
|
11810
|
+
index: FieldIndex,
|
|
11811
|
+
) -> JSONObject:
|
|
11812
|
+
suggested_time_fields = _record_access_suggested_time_fields(index)
|
|
11813
|
+
return {
|
|
11814
|
+
"reported_total": reported_total,
|
|
11815
|
+
"estimated_pages": estimated_pages,
|
|
11816
|
+
"has_time_filter": bool(scope_status.get("has_time_filter")),
|
|
11817
|
+
"has_business_filter": bool(scope_status.get("has_business_filter")),
|
|
11818
|
+
"suggested_time_fields": suggested_time_fields,
|
|
11819
|
+
"recommended_where_examples": _record_access_recommended_where_examples(suggested_time_fields),
|
|
11820
|
+
}
|
|
11821
|
+
|
|
11822
|
+
|
|
11823
|
+
def _record_access_needs_scope(*, scope_status: JSONObject, reported_total: int | None) -> bool:
|
|
11824
|
+
if reported_total is None or reported_total <= RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD:
|
|
11825
|
+
return False
|
|
11826
|
+
return not bool(scope_status.get("has_time_filter")) and not bool(scope_status.get("has_business_filter"))
|
|
11827
|
+
|
|
11828
|
+
|
|
11829
|
+
def _record_access_unbounded_scan_warning(*, reported_total: int | None, estimated_pages: int | None) -> JSONObject:
|
|
11830
|
+
page_text = f" across about {estimated_pages} pages" if estimated_pages else ""
|
|
11831
|
+
return {
|
|
11832
|
+
"code": "UNBOUNDED_SCAN_TOO_LARGE",
|
|
11833
|
+
"message": (
|
|
11834
|
+
"record_access stopped before writing CSV because this query has no time/business boundary "
|
|
11835
|
+
f"and the backend reports {reported_total or 'unknown'} rows{page_text}. "
|
|
11836
|
+
"Add a where filter, preferably on a time field, then retry."
|
|
11837
|
+
),
|
|
11838
|
+
"row_threshold": RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD,
|
|
11839
|
+
"reported_total": reported_total,
|
|
11840
|
+
"estimated_pages": estimated_pages,
|
|
11841
|
+
}
|
|
11842
|
+
|
|
11843
|
+
|
|
11844
|
+
def _record_access_time_budget_exceeded(
|
|
11845
|
+
deadline: float,
|
|
11846
|
+
*,
|
|
11847
|
+
fetched_pages: int,
|
|
11848
|
+
estimated_pages: int | None,
|
|
11849
|
+
) -> bool:
|
|
11850
|
+
now = time.monotonic()
|
|
11851
|
+
if now >= deadline:
|
|
11852
|
+
return True
|
|
11853
|
+
if estimated_pages is not None and estimated_pages - fetched_pages <= 1:
|
|
11854
|
+
return False
|
|
11855
|
+
return now + RECORD_ACCESS_MIN_REMAINING_SECONDS >= deadline
|
|
11856
|
+
|
|
11857
|
+
|
|
11858
|
+
def _record_access_time_budget_warning(*, reported_total: int | None, fetched_pages: int) -> JSONObject:
|
|
11859
|
+
return {
|
|
11860
|
+
"code": "TIME_BUDGET_EXCEEDED",
|
|
11861
|
+
"message": (
|
|
11862
|
+
"record_access stopped early to return partial CSV files before the caller timeout. "
|
|
11863
|
+
"Narrow the query with a time or business filter for a complete result."
|
|
11864
|
+
),
|
|
11865
|
+
"reported_total": reported_total,
|
|
11866
|
+
"fetched_pages": fetched_pages,
|
|
11867
|
+
"time_budget_seconds": RECORD_ACCESS_TIME_BUDGET_SECONDS,
|
|
11868
|
+
}
|
|
11869
|
+
|
|
11870
|
+
|
|
11524
11871
|
def _write_record_access_metadata_files(
|
|
11525
11872
|
*,
|
|
11526
11873
|
run_dir: Path,
|
|
@@ -11534,6 +11881,7 @@ def _write_record_access_metadata_files(
|
|
|
11534
11881
|
fields: list[JSONObject],
|
|
11535
11882
|
warnings: list[JSONObject],
|
|
11536
11883
|
verification: JSONObject,
|
|
11884
|
+
scope: JSONObject | None = None,
|
|
11537
11885
|
) -> JSONObject:
|
|
11538
11886
|
schema_path = run_dir / "schema.json"
|
|
11539
11887
|
readme_path = run_dir / "README.md"
|
|
@@ -11554,6 +11902,8 @@ def _write_record_access_metadata_files(
|
|
|
11554
11902
|
"verification": verification,
|
|
11555
11903
|
"csv_columns": ["record_id"] + [str(field["column_name"]) for field in fields if field.get("column_name")],
|
|
11556
11904
|
}
|
|
11905
|
+
if scope is not None:
|
|
11906
|
+
schema_payload["scope"] = scope
|
|
11557
11907
|
schema_path.write_text(json.dumps(schema_payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
|
11558
11908
|
readme_path.write_text(
|
|
11559
11909
|
_record_access_readme(
|
|
@@ -11566,6 +11916,7 @@ def _write_record_access_metadata_files(
|
|
|
11566
11916
|
files=files,
|
|
11567
11917
|
fields=fields,
|
|
11568
11918
|
warnings=warnings,
|
|
11919
|
+
scope=scope,
|
|
11569
11920
|
),
|
|
11570
11921
|
encoding="utf-8",
|
|
11571
11922
|
)
|
|
@@ -11586,6 +11937,7 @@ def _record_access_readme(
|
|
|
11586
11937
|
files: list[JSONObject],
|
|
11587
11938
|
fields: list[JSONObject],
|
|
11588
11939
|
warnings: list[JSONObject],
|
|
11940
|
+
scope: JSONObject | None = None,
|
|
11589
11941
|
) -> str:
|
|
11590
11942
|
lines = [
|
|
11591
11943
|
"# Qingflow Record Access",
|
|
@@ -11631,6 +11983,18 @@ def _record_access_readme(
|
|
|
11631
11983
|
code = _normalize_optional_text(warning.get("code")) or "WARNING"
|
|
11632
11984
|
message = _normalize_optional_text(warning.get("message")) or ""
|
|
11633
11985
|
lines.append(f"- `{code}`: {message}")
|
|
11986
|
+
if scope:
|
|
11987
|
+
lines.extend(
|
|
11988
|
+
[
|
|
11989
|
+
"",
|
|
11990
|
+
"## Scope",
|
|
11991
|
+
"",
|
|
11992
|
+
f"- Reported total: {scope.get('reported_total')}",
|
|
11993
|
+
f"- Estimated pages: {scope.get('estimated_pages')}",
|
|
11994
|
+
f"- Has time filter: {str(bool(scope.get('has_time_filter'))).lower()}",
|
|
11995
|
+
f"- Has business filter: {str(bool(scope.get('has_business_filter'))).lower()}",
|
|
11996
|
+
]
|
|
11997
|
+
)
|
|
11634
11998
|
lines.extend(
|
|
11635
11999
|
[
|
|
11636
12000
|
"",
|