@josephyan/qingflow-app-user-mcp 1.0.8 → 1.0.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/package.json +1 -1
- package/pyproject.toml +1 -1
- package/skills/qingflow-record-analysis/SKILL.md +3 -0
- package/skills/qingflow-record-analysis/references/analysis-gotchas.md +2 -0
- package/skills/qingflow-record-analysis/references/analysis-patterns.md +11 -7
- package/skills/qingflow-record-analysis/references/confidence-reporting.md +2 -0
- package/src/qingflow_mcp/cli/formatters.py +31 -0
- package/src/qingflow_mcp/response_trim.py +1 -0
- package/src/qingflow_mcp/server_app_user.py +1 -0
- package/src/qingflow_mcp/tools/record_tools.py +285 -65
package/README.md
CHANGED
|
@@ -3,13 +3,13 @@
|
|
|
3
3
|
Install:
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
|
-
npm install @josephyan/qingflow-app-user-mcp@1.0.
|
|
6
|
+
npm install @josephyan/qingflow-app-user-mcp@1.0.9
|
|
7
7
|
```
|
|
8
8
|
|
|
9
9
|
Run:
|
|
10
10
|
|
|
11
11
|
```bash
|
|
12
|
-
npx -y -p @josephyan/qingflow-app-user-mcp@1.0.
|
|
12
|
+
npx -y -p @josephyan/qingflow-app-user-mcp@1.0.9 qingflow-app-user-mcp
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
Environment:
|
package/package.json
CHANGED
package/pyproject.toml
CHANGED
|
@@ -17,6 +17,7 @@ If `app_get.accessible_views` marks a view with `analysis_supported=false`, do n
|
|
|
17
17
|
This is the default execution order. Never skip `app_get` when the browse range is unclear. Never call `record_access` without a browse schema.
|
|
18
18
|
|
|
19
19
|
Core tools: `app_get`, `record_browse_schema_get`, `record_access`, Python. Use field_id-based DSLs only for columns, filters, sort clauses, and any optional lightweight `record_analyze` helper. Use `record_list`/`record_get` only for browse samples. Use `record_analyze` only as a lightweight non-default statistics helper when a compact grouped result is enough. Task/comment work stays in [$qingflow-task-ops](../qingflow-task-ops/SKILL.md).
|
|
20
|
+
For analysis-style tasks, prefer a concrete time range or business filter. If the user did not give one and the data could be large, ask for scope instead of trying an unbounded historical scan.
|
|
20
21
|
|
|
21
22
|
## Execution Modes
|
|
22
23
|
|
|
@@ -78,6 +79,8 @@ Then run Python against every `files[].local_path`. CSV columns are stable: `rec
|
|
|
78
79
|
- Never ask for `page`, `page_size`, `limit`, or `max_rows`; `record_access` owns paging internally and follows the backend's native paging capability.
|
|
79
80
|
- Never treat backend `searchQueIds` as column selection; it is only a full-text search scope.
|
|
80
81
|
- If multiple CSV files are returned, read them all.
|
|
82
|
+
- If `status=needs_scope`, no CSV was written; use `scope.suggested_time_fields` / `scope.recommended_where_examples` to ask for or apply a time/business range, then call `record_access` again.
|
|
83
|
+
- If `status=partial`, read the returned files only as a limited subset; state the limitation and do not present a full-population final conclusion.
|
|
81
84
|
- If `complete=false` or `safe_for_final_conclusion=false`, downgrade the answer and disclose the limitation.
|
|
82
85
|
- `record_export_direct` is only for explicit export/download/Excel requests, not default analysis.
|
|
83
86
|
- QingBI/report reads are only for user-provided report URLs or `chart_id`; do not create or use reports as the default analysis path.
|
|
@@ -125,6 +125,8 @@ If the user asked for several outputs and only part of them is stable:
|
|
|
125
125
|
- say which parts are still unresolved
|
|
126
126
|
- do not present the answer as fully finished
|
|
127
127
|
|
|
128
|
+
If `record_access.status=needs_scope`, stop and ask for a time/business range; no CSV was written. If `record_access.status=partial`, read the returned CSV only as a partial subset and name the limitation before any numbers.
|
|
129
|
+
|
|
128
130
|
## Do not send unsupported formula or div-style metrics into `record_analyze`.
|
|
129
131
|
|
|
130
132
|
Examples to avoid:
|
|
@@ -25,13 +25,17 @@ Use this skill when the user asks for:
|
|
|
25
25
|
|
|
26
26
|
Result reading order:
|
|
27
27
|
|
|
28
|
-
1. `record_access.
|
|
29
|
-
2. `record_access.
|
|
30
|
-
3. `record_access.
|
|
31
|
-
4. `record_access.
|
|
32
|
-
5.
|
|
33
|
-
6.
|
|
34
|
-
7. `record_access.
|
|
28
|
+
1. `record_access.status`
|
|
29
|
+
2. `record_access.complete`
|
|
30
|
+
3. `record_access.safe_for_final_conclusion`
|
|
31
|
+
4. `record_access.files[].local_path`
|
|
32
|
+
5. `record_access.metadata_files.schema`
|
|
33
|
+
6. Python outputs
|
|
34
|
+
7. `record_access.fields`
|
|
35
|
+
8. `record_access.warnings`
|
|
36
|
+
9. `record_access.scope`
|
|
37
|
+
|
|
38
|
+
If `record_access.status=needs_scope`, no CSV was written; ask for a time/business range or retry with a user-provided period from `scope.suggested_time_fields`. If `status=partial`, use the CSV files only as a limited subset and do not make a full-population conclusion.
|
|
35
39
|
|
|
36
40
|
Treat `record_browse_schema_get` as the browse-schema source of truth. It matches the selected Qingflow table view header and is the same schema source used by `record_access.fields` and `schema.json`. Missing fields are permission boundaries, not invitations to guess hidden ids.
|
|
37
41
|
|
|
@@ -27,6 +27,8 @@ Put evidence into `样本观察` when:
|
|
|
27
27
|
- the tool reports `row_cap_hit`
|
|
28
28
|
- the tool reports `sample_only`
|
|
29
29
|
- the result is compact/capped and not complete
|
|
30
|
+
- `record_access.status=needs_scope`
|
|
31
|
+
- `record_access.status=partial`
|
|
30
32
|
- `record_access.complete=false` or `record_access.truncated=true`
|
|
31
33
|
|
|
32
34
|
## Downgrade rule
|
|
@@ -194,6 +194,36 @@ def _format_record_list(result: dict[str, Any]) -> str:
|
|
|
194
194
|
return "\n".join(lines) + "\n"
|
|
195
195
|
|
|
196
196
|
|
|
197
|
+
def _format_record_access(result: dict[str, Any]) -> str:
|
|
198
|
+
status = result.get("status") or "-"
|
|
199
|
+
lines = [
|
|
200
|
+
f"Status: {status}",
|
|
201
|
+
f"Rows: {result.get('row_count')}",
|
|
202
|
+
f"Complete: {result.get('complete')}",
|
|
203
|
+
f"Safe for final conclusion: {result.get('safe_for_final_conclusion')}",
|
|
204
|
+
]
|
|
205
|
+
if result.get("local_dir"):
|
|
206
|
+
lines.append(f"Local dir: {result.get('local_dir')}")
|
|
207
|
+
files = result.get("files") if isinstance(result.get("files"), list) else []
|
|
208
|
+
if files:
|
|
209
|
+
lines.append("Files:")
|
|
210
|
+
for item in files:
|
|
211
|
+
if isinstance(item, dict):
|
|
212
|
+
lines.append(f"- part {item.get('part')}: {item.get('local_path')} ({item.get('row_count')} rows)")
|
|
213
|
+
scope = result.get("scope") if isinstance(result.get("scope"), dict) else {}
|
|
214
|
+
if status == "needs_scope" and scope:
|
|
215
|
+
lines.append("Scope required:")
|
|
216
|
+
lines.append(f"- reported_total: {scope.get('reported_total')}")
|
|
217
|
+
lines.append(f"- estimated_pages: {scope.get('estimated_pages')}")
|
|
218
|
+
suggested = scope.get("suggested_time_fields") if isinstance(scope.get("suggested_time_fields"), list) else []
|
|
219
|
+
if suggested:
|
|
220
|
+
names = ", ".join(str(item.get("title") or item.get("field_id")) for item in suggested if isinstance(item, dict))
|
|
221
|
+
lines.append(f"- suggested_time_fields: {names}")
|
|
222
|
+
_append_warnings(lines, result.get("warnings"))
|
|
223
|
+
_append_verification(lines, result.get("verification"))
|
|
224
|
+
return "\n".join(lines) + "\n"
|
|
225
|
+
|
|
226
|
+
|
|
197
227
|
def _format_task_list(result: dict[str, Any]) -> str:
|
|
198
228
|
data = result.get("data") if isinstance(result.get("data"), dict) else {}
|
|
199
229
|
items = data.get("items") if isinstance(data.get("items"), list) else []
|
|
@@ -693,6 +723,7 @@ _FORMATTERS = {
|
|
|
693
723
|
"app_search": _format_app_items,
|
|
694
724
|
"app_get": _format_app_get,
|
|
695
725
|
"record_list": _format_record_list,
|
|
726
|
+
"record_access": _format_record_access,
|
|
696
727
|
"task_list": _format_task_list,
|
|
697
728
|
"task_workbench": _format_task_workbench,
|
|
698
729
|
"task_get": _format_task_get,
|
|
@@ -77,6 +77,7 @@ Inside `optional_fields`, any field with `may_become_required=true` is still wri
|
|
|
77
77
|
Prefer `view_id` entries from `accessible_views` where `analysis_supported=true`.
|
|
78
78
|
|
|
79
79
|
Use `record_access` to write local CSV shard files, then use Python to compute counts, rankings, ratios, trends, and final conclusions. `record_access` does not return bulk `items`; read `files[].local_path`. Use `fields[]` and the same-directory `metadata_files.schema` / `metadata_files.readme` to map stable `field_<id>` CSV columns back to titles and types.
|
|
80
|
+
For analysis-style tasks, prefer an explicit time range or business filter. If `record_access.status == "needs_scope"`, do not treat it as a failure; ask for a time/business scope or retry with a user-provided period using `scope.suggested_time_fields` / `scope.recommended_where_examples`. If `record_access.status == "partial"`, read the returned files only as a limited subset and do not give a final full-population conclusion.
|
|
80
81
|
Use `chart_get` only when the user provides a report URL / chart_id or explicitly asks to read an existing report. Do not use QingBI as the default analysis route.
|
|
81
82
|
|
|
82
83
|
Use this data-access DSL shape:
|
|
@@ -7,7 +7,7 @@ import re
|
|
|
7
7
|
import time
|
|
8
8
|
from copy import deepcopy
|
|
9
9
|
from dataclasses import dataclass
|
|
10
|
-
from datetime import UTC, datetime
|
|
10
|
+
from datetime import UTC, datetime, timedelta
|
|
11
11
|
from decimal import Decimal, InvalidOperation
|
|
12
12
|
from pathlib import Path
|
|
13
13
|
from typing import Any, cast
|
|
@@ -34,6 +34,9 @@ DEFAULT_QUERY_PAGE_SIZE = 50
|
|
|
34
34
|
DEFAULT_LIST_PAGE_SIZE = 200
|
|
35
35
|
BACKEND_RECORD_ACCESS_PAGE_SIZE = 1000
|
|
36
36
|
DEFAULT_RECORD_ACCESS_SHARD_ROWS = 5000
|
|
37
|
+
RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD = 20_000
|
|
38
|
+
RECORD_ACCESS_TIME_BUDGET_SECONDS = 55.0
|
|
39
|
+
RECORD_ACCESS_MIN_REMAINING_SECONDS = 8.0
|
|
37
40
|
DEFAULT_ANALYSIS_PAGE_SIZE = 1000
|
|
38
41
|
DEFAULT_SCAN_MAX_PAGES = 10
|
|
39
42
|
DEFAULT_ANALYSIS_SCAN_MAX_PAGES = 100
|
|
@@ -1848,6 +1851,7 @@ class RecordTools(ToolBase):
|
|
|
1848
1851
|
sorts = self._normalize_record_list_order_by(order_by)
|
|
1849
1852
|
|
|
1850
1853
|
def runner(session_profile, context):
|
|
1854
|
+
created_at = datetime.now(UTC).isoformat()
|
|
1851
1855
|
browse_scope = self._build_browse_read_scope(
|
|
1852
1856
|
profile,
|
|
1853
1857
|
context,
|
|
@@ -1870,6 +1874,106 @@ class RecordTools(ToolBase):
|
|
|
1870
1874
|
if view_selection is not None or view_route.list_type is not None
|
|
1871
1875
|
else [DEFAULT_RECORD_LIST_TYPE, 14, 1, 2, 12]
|
|
1872
1876
|
)
|
|
1877
|
+
fields_payload = [_record_access_field_payload(field) for field in selected_fields]
|
|
1878
|
+
scope_status = _record_access_scope_status(filters, view_route, index)
|
|
1879
|
+
|
|
1880
|
+
def fetch_page(page_num: int, page_size: int) -> JSONObject:
|
|
1881
|
+
nonlocal used_list_type
|
|
1882
|
+
if used_list_type is None:
|
|
1883
|
+
last_error: QingflowApiError | None = None
|
|
1884
|
+
page: JSONObject | None = None
|
|
1885
|
+
for candidate_list_type in fallback_list_types:
|
|
1886
|
+
try:
|
|
1887
|
+
page = self._search_page(
|
|
1888
|
+
context,
|
|
1889
|
+
app_key=app_key,
|
|
1890
|
+
view_selection=view_selection,
|
|
1891
|
+
page_num=page_num,
|
|
1892
|
+
page_size=page_size,
|
|
1893
|
+
query_key=None,
|
|
1894
|
+
match_rules=match_rules,
|
|
1895
|
+
sorts=sort_rules,
|
|
1896
|
+
search_que_ids=None,
|
|
1897
|
+
list_type=candidate_list_type,
|
|
1898
|
+
)
|
|
1899
|
+
used_list_type = None if view_selection is not None else candidate_list_type
|
|
1900
|
+
break
|
|
1901
|
+
except QingflowApiError as exc:
|
|
1902
|
+
last_error = exc
|
|
1903
|
+
if self._should_retry_list_type_fallback(exc) and candidate_list_type != fallback_list_types[-1]:
|
|
1904
|
+
continue
|
|
1905
|
+
raise
|
|
1906
|
+
if page is None:
|
|
1907
|
+
if last_error is not None:
|
|
1908
|
+
raise last_error
|
|
1909
|
+
raise_tool_error(QingflowApiError.config_error("record_access failed: no accessible listType"))
|
|
1910
|
+
return page
|
|
1911
|
+
return self._search_page(
|
|
1912
|
+
context,
|
|
1913
|
+
app_key=app_key,
|
|
1914
|
+
view_selection=view_selection,
|
|
1915
|
+
page_num=page_num,
|
|
1916
|
+
page_size=page_size,
|
|
1917
|
+
query_key=None,
|
|
1918
|
+
match_rules=match_rules,
|
|
1919
|
+
sorts=sort_rules,
|
|
1920
|
+
search_que_ids=None,
|
|
1921
|
+
list_type=used_list_type,
|
|
1922
|
+
)
|
|
1923
|
+
|
|
1924
|
+
probe_page_size = 1 if _record_access_light_probe_recommended(scope_status) else BACKEND_RECORD_ACCESS_PAGE_SIZE
|
|
1925
|
+
probe_page = fetch_page(1, probe_page_size)
|
|
1926
|
+
reported_total = _effective_total(probe_page, probe_page_size)
|
|
1927
|
+
estimated_pages = _record_access_estimated_pages(probe_page, reported_total, page_size=probe_page_size)
|
|
1928
|
+
scope_payload = _record_access_scope_payload(
|
|
1929
|
+
scope_status=scope_status,
|
|
1930
|
+
reported_total=reported_total,
|
|
1931
|
+
estimated_pages=estimated_pages,
|
|
1932
|
+
index=index,
|
|
1933
|
+
)
|
|
1934
|
+
warnings: list[JSONObject] = []
|
|
1935
|
+
warnings.extend(legacy_warnings)
|
|
1936
|
+
warnings.extend(compatibility_warnings)
|
|
1937
|
+
warnings.extend(_view_filter_trust_warnings(view_route))
|
|
1938
|
+
if used_list_type is not None and used_list_type != DEFAULT_RECORD_LIST_TYPE:
|
|
1939
|
+
warnings.append(
|
|
1940
|
+
{
|
|
1941
|
+
"code": "LIST_TYPE_FALLBACK",
|
|
1942
|
+
"message": (
|
|
1943
|
+
f"record_access not accessible via listType={DEFAULT_RECORD_LIST_TYPE}; "
|
|
1944
|
+
f"fell back to listType={used_list_type} ({get_record_list_type_label(used_list_type)})."
|
|
1945
|
+
),
|
|
1946
|
+
}
|
|
1947
|
+
)
|
|
1948
|
+
if _record_access_needs_scope(scope_status=scope_status, reported_total=reported_total):
|
|
1949
|
+
warnings.append(_record_access_unbounded_scan_warning(reported_total=reported_total, estimated_pages=estimated_pages))
|
|
1950
|
+
verification_payload: JSONObject = {
|
|
1951
|
+
**_view_filter_verification_payload(view_route),
|
|
1952
|
+
"reported_total": reported_total,
|
|
1953
|
+
"estimated_pages": estimated_pages,
|
|
1954
|
+
"fetched_pages": 1,
|
|
1955
|
+
"stopped_reason": "UNBOUNDED_SCAN_TOO_LARGE",
|
|
1956
|
+
"list_type_used": used_list_type,
|
|
1957
|
+
}
|
|
1958
|
+
return {
|
|
1959
|
+
"profile": profile,
|
|
1960
|
+
"ws_id": session_profile.selected_ws_id,
|
|
1961
|
+
"ok": True,
|
|
1962
|
+
"status": "needs_scope",
|
|
1963
|
+
"app_key": app_key,
|
|
1964
|
+
"view_id": view_route.view_id,
|
|
1965
|
+
"format": "csv",
|
|
1966
|
+
"row_count": 0,
|
|
1967
|
+
"complete": False,
|
|
1968
|
+
"truncated": True,
|
|
1969
|
+
"safe_for_final_conclusion": False,
|
|
1970
|
+
"files": [],
|
|
1971
|
+
"fields": fields_payload,
|
|
1972
|
+
"warnings": warnings,
|
|
1973
|
+
"verification": verification_payload,
|
|
1974
|
+
"scope": scope_payload,
|
|
1975
|
+
"request_route": self._request_route_payload(context),
|
|
1976
|
+
}
|
|
1873
1977
|
|
|
1874
1978
|
run_dir = _record_access_run_dir()
|
|
1875
1979
|
run_dir.mkdir(parents=True, exist_ok=True)
|
|
@@ -1926,58 +2030,14 @@ class RecordTools(ToolBase):
|
|
|
1926
2030
|
|
|
1927
2031
|
current_page = 1
|
|
1928
2032
|
has_more = False
|
|
1929
|
-
|
|
1930
|
-
|
|
2033
|
+
fetched_pages = 0
|
|
2034
|
+
stopped_reason: str | None = None
|
|
2035
|
+
deadline = time.monotonic() + RECORD_ACCESS_TIME_BUDGET_SECONDS
|
|
2036
|
+
page = probe_page if probe_page_size == BACKEND_RECORD_ACCESS_PAGE_SIZE else fetch_page(1, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
1931
2037
|
try:
|
|
1932
2038
|
while True:
|
|
1933
|
-
if used_list_type is None:
|
|
1934
|
-
last_error: QingflowApiError | None = None
|
|
1935
|
-
page: JSONObject | None = None
|
|
1936
|
-
for candidate_list_type in fallback_list_types:
|
|
1937
|
-
try:
|
|
1938
|
-
page = self._search_page(
|
|
1939
|
-
context,
|
|
1940
|
-
app_key=app_key,
|
|
1941
|
-
view_selection=view_selection,
|
|
1942
|
-
page_num=current_page,
|
|
1943
|
-
page_size=BACKEND_RECORD_ACCESS_PAGE_SIZE,
|
|
1944
|
-
query_key=None,
|
|
1945
|
-
match_rules=match_rules,
|
|
1946
|
-
sorts=sort_rules,
|
|
1947
|
-
search_que_ids=None,
|
|
1948
|
-
list_type=candidate_list_type,
|
|
1949
|
-
)
|
|
1950
|
-
used_list_type = None if view_selection is not None else candidate_list_type
|
|
1951
|
-
break
|
|
1952
|
-
except QingflowApiError as exc:
|
|
1953
|
-
last_error = exc
|
|
1954
|
-
if (
|
|
1955
|
-
self._should_retry_list_type_fallback(exc)
|
|
1956
|
-
and candidate_list_type != fallback_list_types[-1]
|
|
1957
|
-
):
|
|
1958
|
-
continue
|
|
1959
|
-
raise
|
|
1960
|
-
if page is None:
|
|
1961
|
-
if last_error is not None:
|
|
1962
|
-
raise last_error
|
|
1963
|
-
raise_tool_error(QingflowApiError.config_error("record_access failed: no accessible listType"))
|
|
1964
|
-
else:
|
|
1965
|
-
page = self._search_page(
|
|
1966
|
-
context,
|
|
1967
|
-
app_key=app_key,
|
|
1968
|
-
view_selection=view_selection,
|
|
1969
|
-
page_num=current_page,
|
|
1970
|
-
page_size=BACKEND_RECORD_ACCESS_PAGE_SIZE,
|
|
1971
|
-
query_key=None,
|
|
1972
|
-
match_rules=match_rules,
|
|
1973
|
-
sorts=sort_rules,
|
|
1974
|
-
search_que_ids=None,
|
|
1975
|
-
list_type=used_list_type,
|
|
1976
|
-
)
|
|
1977
2039
|
page_rows = page.get("list")
|
|
1978
2040
|
items = page_rows if isinstance(page_rows, list) else []
|
|
1979
|
-
if reported_total is None:
|
|
1980
|
-
reported_total = _effective_total(page, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
1981
2041
|
has_more = _page_has_more(page, current_page, BACKEND_RECORD_ACCESS_PAGE_SIZE, len(items))
|
|
1982
2042
|
page_apply_order: list[int] = []
|
|
1983
2043
|
page_answer_map: dict[int, list[JSONValue]] = {}
|
|
@@ -1993,34 +2053,33 @@ class RecordTools(ToolBase):
|
|
|
1993
2053
|
page_answer_map[apply_id] = cast(list[JSONValue], answer_list)
|
|
1994
2054
|
for apply_id in page_apply_order:
|
|
1995
2055
|
write_record(apply_id, page_answer_map.get(apply_id, []))
|
|
2056
|
+
fetched_pages += 1
|
|
1996
2057
|
if not has_more:
|
|
1997
2058
|
break
|
|
1998
2059
|
current_page += 1
|
|
2060
|
+
if _record_access_time_budget_exceeded(
|
|
2061
|
+
deadline,
|
|
2062
|
+
fetched_pages=fetched_pages,
|
|
2063
|
+
estimated_pages=estimated_pages,
|
|
2064
|
+
):
|
|
2065
|
+
stopped_reason = "TIME_BUDGET_EXCEEDED"
|
|
2066
|
+
break
|
|
2067
|
+
page = fetch_page(current_page, BACKEND_RECORD_ACCESS_PAGE_SIZE)
|
|
1999
2068
|
finally:
|
|
2000
2069
|
close_shard()
|
|
2001
2070
|
|
|
2002
|
-
|
|
2003
|
-
|
|
2004
|
-
|
|
2005
|
-
warnings.extend(_view_filter_trust_warnings(view_route))
|
|
2006
|
-
if used_list_type is not None and used_list_type != DEFAULT_RECORD_LIST_TYPE:
|
|
2007
|
-
warnings.append(
|
|
2008
|
-
{
|
|
2009
|
-
"code": "LIST_TYPE_FALLBACK",
|
|
2010
|
-
"message": (
|
|
2011
|
-
f"record_access not accessible via listType={DEFAULT_RECORD_LIST_TYPE}; "
|
|
2012
|
-
f"fell back to listType={used_list_type} ({get_record_list_type_label(used_list_type)})."
|
|
2013
|
-
),
|
|
2014
|
-
}
|
|
2015
|
-
)
|
|
2016
|
-
complete = not has_more
|
|
2071
|
+
if stopped_reason == "TIME_BUDGET_EXCEEDED":
|
|
2072
|
+
warnings.append(_record_access_time_budget_warning(reported_total=reported_total, fetched_pages=fetched_pages))
|
|
2073
|
+
complete = stopped_reason is None and not has_more
|
|
2017
2074
|
safe_for_final_conclusion = complete and not any(
|
|
2018
2075
|
warning.get("code") == "CUSTOM_VIEW_FILTER_UNVERIFIED" for warning in warnings
|
|
2019
2076
|
)
|
|
2020
|
-
fields_payload = [_record_access_field_payload(field) for field in selected_fields]
|
|
2021
2077
|
verification_payload: JSONObject = {
|
|
2022
2078
|
**_view_filter_verification_payload(view_route),
|
|
2023
2079
|
"reported_total": reported_total,
|
|
2080
|
+
"estimated_pages": estimated_pages,
|
|
2081
|
+
"fetched_pages": fetched_pages,
|
|
2082
|
+
"stopped_reason": stopped_reason,
|
|
2024
2083
|
"list_type_used": used_list_type,
|
|
2025
2084
|
}
|
|
2026
2085
|
metadata_files = _write_record_access_metadata_files(
|
|
@@ -2035,6 +2094,7 @@ class RecordTools(ToolBase):
|
|
|
2035
2094
|
fields=fields_payload,
|
|
2036
2095
|
warnings=warnings,
|
|
2037
2096
|
verification=verification_payload,
|
|
2097
|
+
scope=scope_payload,
|
|
2038
2098
|
)
|
|
2039
2099
|
return {
|
|
2040
2100
|
"profile": profile,
|
|
@@ -2054,6 +2114,7 @@ class RecordTools(ToolBase):
|
|
|
2054
2114
|
"fields": fields_payload,
|
|
2055
2115
|
"warnings": warnings,
|
|
2056
2116
|
"verification": verification_payload,
|
|
2117
|
+
"scope": scope_payload,
|
|
2057
2118
|
"request_route": self._request_route_payload(context),
|
|
2058
2119
|
}
|
|
2059
2120
|
|
|
@@ -11665,6 +11726,148 @@ def _record_access_field_payload(field: FormField) -> JSONObject:
|
|
|
11665
11726
|
}
|
|
11666
11727
|
|
|
11667
11728
|
|
|
11729
|
+
def _record_access_scope_status(filters: list[JSONObject], view_route: AccessibleViewRoute, index: FieldIndex) -> JSONObject:
|
|
11730
|
+
has_time_filter = False
|
|
11731
|
+
has_explicit_business_filter = False
|
|
11732
|
+
for item in filters:
|
|
11733
|
+
if not isinstance(item, dict):
|
|
11734
|
+
continue
|
|
11735
|
+
field_id = _coerce_count(item.get("field_id", item.get("fieldId")))
|
|
11736
|
+
field = index.by_id.get(str(field_id)) if field_id is not None else None
|
|
11737
|
+
if field is not None and field.que_type in DATE_QUE_TYPES:
|
|
11738
|
+
has_time_filter = True
|
|
11739
|
+
else:
|
|
11740
|
+
has_explicit_business_filter = True
|
|
11741
|
+
view_selection = view_route.view_selection
|
|
11742
|
+
has_saved_view_filter = bool(view_selection is not None and view_selection.conditions)
|
|
11743
|
+
return {
|
|
11744
|
+
"filter_count": len(filters),
|
|
11745
|
+
"has_explicit_where": bool(filters),
|
|
11746
|
+
"has_time_filter": has_time_filter,
|
|
11747
|
+
"has_business_filter": bool(has_explicit_business_filter or has_saved_view_filter),
|
|
11748
|
+
"has_saved_view_filter": has_saved_view_filter,
|
|
11749
|
+
}
|
|
11750
|
+
|
|
11751
|
+
|
|
11752
|
+
def _record_access_light_probe_recommended(scope_status: JSONObject) -> bool:
|
|
11753
|
+
return not bool(scope_status.get("has_time_filter")) and not bool(scope_status.get("has_business_filter"))
|
|
11754
|
+
|
|
11755
|
+
|
|
11756
|
+
def _record_access_estimated_pages(page: JSONObject, reported_total: int | None, *, page_size: int) -> int | None:
|
|
11757
|
+
page_amount = _coerce_count(page.get("pageAmount"))
|
|
11758
|
+
if page_size == BACKEND_RECORD_ACCESS_PAGE_SIZE and page_amount is not None:
|
|
11759
|
+
return page_amount
|
|
11760
|
+
if reported_total is None or reported_total <= 0:
|
|
11761
|
+
return page_amount
|
|
11762
|
+
return (reported_total + BACKEND_RECORD_ACCESS_PAGE_SIZE - 1) // BACKEND_RECORD_ACCESS_PAGE_SIZE
|
|
11763
|
+
|
|
11764
|
+
|
|
11765
|
+
def _record_access_suggested_time_fields(index: FieldIndex) -> list[JSONObject]:
|
|
11766
|
+
fields: list[JSONObject] = []
|
|
11767
|
+
for field in index.by_id.values():
|
|
11768
|
+
if field.que_type in DATE_QUE_TYPES:
|
|
11769
|
+
fields.append(_record_access_field_payload(field))
|
|
11770
|
+
return fields
|
|
11771
|
+
|
|
11772
|
+
|
|
11773
|
+
def _record_access_recommended_where_examples(suggested_time_fields: list[JSONObject]) -> list[JSONObject]:
|
|
11774
|
+
if not suggested_time_fields:
|
|
11775
|
+
return []
|
|
11776
|
+
now = datetime.now(UTC)
|
|
11777
|
+
month_start = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
|
|
11778
|
+
if month_start.month == 12:
|
|
11779
|
+
next_month = month_start.replace(year=month_start.year + 1, month=1)
|
|
11780
|
+
else:
|
|
11781
|
+
next_month = month_start.replace(month=month_start.month + 1)
|
|
11782
|
+
month_end = next_month - timedelta(seconds=1)
|
|
11783
|
+
examples: list[JSONObject] = []
|
|
11784
|
+
for field in suggested_time_fields[:3]:
|
|
11785
|
+
field_id = field.get("field_id")
|
|
11786
|
+
examples.append(
|
|
11787
|
+
{
|
|
11788
|
+
"field_id": field_id,
|
|
11789
|
+
"title": field.get("title"),
|
|
11790
|
+
"where": [
|
|
11791
|
+
{
|
|
11792
|
+
"field_id": field_id,
|
|
11793
|
+
"op": "between",
|
|
11794
|
+
"value": [
|
|
11795
|
+
month_start.strftime("%Y-%m-%d"),
|
|
11796
|
+
month_end.strftime("%Y-%m-%d %H:%M:%S"),
|
|
11797
|
+
],
|
|
11798
|
+
}
|
|
11799
|
+
],
|
|
11800
|
+
}
|
|
11801
|
+
)
|
|
11802
|
+
return examples
|
|
11803
|
+
|
|
11804
|
+
|
|
11805
|
+
def _record_access_scope_payload(
|
|
11806
|
+
*,
|
|
11807
|
+
scope_status: JSONObject,
|
|
11808
|
+
reported_total: int | None,
|
|
11809
|
+
estimated_pages: int | None,
|
|
11810
|
+
index: FieldIndex,
|
|
11811
|
+
) -> JSONObject:
|
|
11812
|
+
suggested_time_fields = _record_access_suggested_time_fields(index)
|
|
11813
|
+
return {
|
|
11814
|
+
"reported_total": reported_total,
|
|
11815
|
+
"estimated_pages": estimated_pages,
|
|
11816
|
+
"has_time_filter": bool(scope_status.get("has_time_filter")),
|
|
11817
|
+
"has_business_filter": bool(scope_status.get("has_business_filter")),
|
|
11818
|
+
"suggested_time_fields": suggested_time_fields,
|
|
11819
|
+
"recommended_where_examples": _record_access_recommended_where_examples(suggested_time_fields),
|
|
11820
|
+
}
|
|
11821
|
+
|
|
11822
|
+
|
|
11823
|
+
def _record_access_needs_scope(*, scope_status: JSONObject, reported_total: int | None) -> bool:
|
|
11824
|
+
if reported_total is None or reported_total <= RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD:
|
|
11825
|
+
return False
|
|
11826
|
+
return not bool(scope_status.get("has_time_filter")) and not bool(scope_status.get("has_business_filter"))
|
|
11827
|
+
|
|
11828
|
+
|
|
11829
|
+
def _record_access_unbounded_scan_warning(*, reported_total: int | None, estimated_pages: int | None) -> JSONObject:
|
|
11830
|
+
page_text = f" across about {estimated_pages} pages" if estimated_pages else ""
|
|
11831
|
+
return {
|
|
11832
|
+
"code": "UNBOUNDED_SCAN_TOO_LARGE",
|
|
11833
|
+
"message": (
|
|
11834
|
+
"record_access stopped before writing CSV because this query has no time/business boundary "
|
|
11835
|
+
f"and the backend reports {reported_total or 'unknown'} rows{page_text}. "
|
|
11836
|
+
"Add a where filter, preferably on a time field, then retry."
|
|
11837
|
+
),
|
|
11838
|
+
"row_threshold": RECORD_ACCESS_UNBOUNDED_ROW_THRESHOLD,
|
|
11839
|
+
"reported_total": reported_total,
|
|
11840
|
+
"estimated_pages": estimated_pages,
|
|
11841
|
+
}
|
|
11842
|
+
|
|
11843
|
+
|
|
11844
|
+
def _record_access_time_budget_exceeded(
|
|
11845
|
+
deadline: float,
|
|
11846
|
+
*,
|
|
11847
|
+
fetched_pages: int,
|
|
11848
|
+
estimated_pages: int | None,
|
|
11849
|
+
) -> bool:
|
|
11850
|
+
now = time.monotonic()
|
|
11851
|
+
if now >= deadline:
|
|
11852
|
+
return True
|
|
11853
|
+
if estimated_pages is not None and estimated_pages - fetched_pages <= 1:
|
|
11854
|
+
return False
|
|
11855
|
+
return now + RECORD_ACCESS_MIN_REMAINING_SECONDS >= deadline
|
|
11856
|
+
|
|
11857
|
+
|
|
11858
|
+
def _record_access_time_budget_warning(*, reported_total: int | None, fetched_pages: int) -> JSONObject:
|
|
11859
|
+
return {
|
|
11860
|
+
"code": "TIME_BUDGET_EXCEEDED",
|
|
11861
|
+
"message": (
|
|
11862
|
+
"record_access stopped early to return partial CSV files before the caller timeout. "
|
|
11863
|
+
"Narrow the query with a time or business filter for a complete result."
|
|
11864
|
+
),
|
|
11865
|
+
"reported_total": reported_total,
|
|
11866
|
+
"fetched_pages": fetched_pages,
|
|
11867
|
+
"time_budget_seconds": RECORD_ACCESS_TIME_BUDGET_SECONDS,
|
|
11868
|
+
}
|
|
11869
|
+
|
|
11870
|
+
|
|
11668
11871
|
def _write_record_access_metadata_files(
|
|
11669
11872
|
*,
|
|
11670
11873
|
run_dir: Path,
|
|
@@ -11678,6 +11881,7 @@ def _write_record_access_metadata_files(
|
|
|
11678
11881
|
fields: list[JSONObject],
|
|
11679
11882
|
warnings: list[JSONObject],
|
|
11680
11883
|
verification: JSONObject,
|
|
11884
|
+
scope: JSONObject | None = None,
|
|
11681
11885
|
) -> JSONObject:
|
|
11682
11886
|
schema_path = run_dir / "schema.json"
|
|
11683
11887
|
readme_path = run_dir / "README.md"
|
|
@@ -11698,6 +11902,8 @@ def _write_record_access_metadata_files(
|
|
|
11698
11902
|
"verification": verification,
|
|
11699
11903
|
"csv_columns": ["record_id"] + [str(field["column_name"]) for field in fields if field.get("column_name")],
|
|
11700
11904
|
}
|
|
11905
|
+
if scope is not None:
|
|
11906
|
+
schema_payload["scope"] = scope
|
|
11701
11907
|
schema_path.write_text(json.dumps(schema_payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
|
11702
11908
|
readme_path.write_text(
|
|
11703
11909
|
_record_access_readme(
|
|
@@ -11710,6 +11916,7 @@ def _write_record_access_metadata_files(
|
|
|
11710
11916
|
files=files,
|
|
11711
11917
|
fields=fields,
|
|
11712
11918
|
warnings=warnings,
|
|
11919
|
+
scope=scope,
|
|
11713
11920
|
),
|
|
11714
11921
|
encoding="utf-8",
|
|
11715
11922
|
)
|
|
@@ -11730,6 +11937,7 @@ def _record_access_readme(
|
|
|
11730
11937
|
files: list[JSONObject],
|
|
11731
11938
|
fields: list[JSONObject],
|
|
11732
11939
|
warnings: list[JSONObject],
|
|
11940
|
+
scope: JSONObject | None = None,
|
|
11733
11941
|
) -> str:
|
|
11734
11942
|
lines = [
|
|
11735
11943
|
"# Qingflow Record Access",
|
|
@@ -11775,6 +11983,18 @@ def _record_access_readme(
|
|
|
11775
11983
|
code = _normalize_optional_text(warning.get("code")) or "WARNING"
|
|
11776
11984
|
message = _normalize_optional_text(warning.get("message")) or ""
|
|
11777
11985
|
lines.append(f"- `{code}`: {message}")
|
|
11986
|
+
if scope:
|
|
11987
|
+
lines.extend(
|
|
11988
|
+
[
|
|
11989
|
+
"",
|
|
11990
|
+
"## Scope",
|
|
11991
|
+
"",
|
|
11992
|
+
f"- Reported total: {scope.get('reported_total')}",
|
|
11993
|
+
f"- Estimated pages: {scope.get('estimated_pages')}",
|
|
11994
|
+
f"- Has time filter: {str(bool(scope.get('has_time_filter'))).lower()}",
|
|
11995
|
+
f"- Has business filter: {str(bool(scope.get('has_business_filter'))).lower()}",
|
|
11996
|
+
]
|
|
11997
|
+
)
|
|
11778
11998
|
lines.extend(
|
|
11779
11999
|
[
|
|
11780
12000
|
"",
|