dsh-wrenai 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +121 -2
- package/README.zh-CN.md +110 -0
- package/ROADMAP.md +54 -0
- package/SKILL.md +28 -0
- package/bin/dsh-wrenai.js +15 -0
- package/cordis.patch.yml +7 -0
- package/package.json +52 -4
- package/scripts/ask.py +274 -0
- package/scripts/pgwire.py +224 -0
- package/semantic/models.yaml +14 -0
- package/semantic/weknora.yaml +39 -0
- package/skills/dsh-wrenai/SKILL.md +30 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 vicfei
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,122 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-WrenAI — a self-hosted semantic layer for DeepSeek Harness
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Ask data questions in natural language; get numbers with their metric
|
|
4
|
+
calibers attached.** The semantic model (dimensions/measures/metrics +
|
|
5
|
+
calibers) rides between the question and the SQL — that is the line between
|
|
6
|
+
governed analytics and raw text-to-SQL.
|
|
7
|
+
|
|
8
|
+
[English](README.md) | [简体中文](README.zh-CN.md)
|
|
9
|
+
|
|
10
|
+
[](https://github.com/vicfei/dsh-WrenAI/actions/workflows/ci.yml)
|
|
11
|
+
[](https://www.npmjs.com/package/dsh-wrenai)
|
|
12
|
+
[](LICENSE)
|
|
13
|
+
[](#install)
|
|
14
|
+
[](https://github.com/Canner/WrenAI)
|
|
15
|
+
|
|
16
|
+
> Implements the differentiated core of [WrenAI](https://github.com/Canner/WrenAI)
|
|
17
|
+
> (18.3k★ — GenBI / governed text-to-SQL) natively for
|
|
18
|
+
> [DeepSeek Harness](https://github.com/deepseek-ai): the **self-hosted
|
|
19
|
+
> semantic layer**. Clean-room, zero upstream code. Not affiliated.
|
|
20
|
+
> Family: the 12-repo quant stack — [dsh-TradingAgents](https://github.com/vicfei/dsh-TradingAgents) ·
|
|
21
|
+
> [dsh-OpenBB](https://github.com/vicfei/dsh-OpenBB) ·
|
|
22
|
+
> [dsh-CCXT](https://github.com/vicfei/dsh-CCXT) ·
|
|
23
|
+
> [dsh-quantstats](https://github.com/vicfei/dsh-quantstats) ·
|
|
24
|
+
> [dsh-freqtrade](https://github.com/vicfei/dsh-freqtrade) …
|
|
25
|
+
|
|
26
|
+
## The headline: ask your WeKnora anything
|
|
27
|
+
|
|
28
|
+
[WeKnora](https://github.com/WeKnora/WeKnora) (self-hosted RAG platform)
|
|
29
|
+
ships no self-observability: its audit log records **write-side actions
|
|
30
|
+
only**, so "how often is this knowledge base actually used?" is unanswerable
|
|
31
|
+
from the UI. dsh-WrenAI connects straight to the deployment's PostgreSQL —
|
|
32
|
+
zero config, DSN resolved from `~/WeKnora/.env` — with a preset governed
|
|
33
|
+
model and answers with the honest caliber attached:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
dsh-wrenai --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
```text
|
|
40
|
+
## Ask: how many times was the test knowledge base queried in the past month?
|
|
41
|
+
| kb_query_events |
|
|
42
|
+
|-----------------|
|
|
43
|
+
| 2 |
|
|
44
|
+
## Calibers riding with this result (metric definitions used)
|
|
45
|
+
- kb_query_events: DISTINCT COUNT of assistant messages citing at least one
|
|
46
|
+
chunk of the KB in the time window … HONEST LIMIT: WeKnora audit_logs record
|
|
47
|
+
write-side actions only, so retrieval usage is measured by citations
|
|
48
|
+
actually served — this counts questions whose answers cited the KB, not raw
|
|
49
|
+
retrieval calls.
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
(Live dev deployment, 2026-11-19; verified against hand-written SQL.)
|
|
53
|
+
`kb_query_events` (distinct questions) and `kb_citations` (citation
|
|
54
|
+
occurrences) are two calibers on the same join path — "which KBs were cited
|
|
55
|
+
most this quarter?" ranks both side by side (hotpot-pool-500: 1287 events /
|
|
56
|
+
34948 citations on the live dev box).
|
|
57
|
+
|
|
58
|
+
## Why (NL2SQL plugins exist — what's missing is the semantic layer)
|
|
59
|
+
|
|
60
|
+
| | raw NL2SQL plugins | **dsh-WrenAI** |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| Model sees | table schemas | **semantic catalog** — metrics with calibers, dimensions with business meaning |
|
|
63
|
+
| Trust | hope | **calibers ride with results**: every result cites the metric definitions it used |
|
|
64
|
+
| SQL safety | varies | **single-SELECT guard** + self-heal retry, deterministic execution |
|
|
65
|
+
| Backends | one, usually | **SQLite (stdlib) or PostgreSQL** — via a zero-dependency wire client |
|
|
66
|
+
| Self-hosted | — | fully local (YAML models + your DB), no cloud service, no pip installs |
|
|
67
|
+
|
|
68
|
+
## Install
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
npm i -g dsh-wrenai # or: dsh plugin add dsh-wrenai
|
|
72
|
+
dsh-wrenai --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Requires `python3` on PATH — the engine is **stdlib-only** (no pip installs);
|
|
76
|
+
the npm package is a thin launcher plus the DSH plugin manifest and skill.
|
|
77
|
+
From source: clone and call `python3 scripts/ask.py` directly.
|
|
78
|
+
|
|
79
|
+
## Quick start (any dataset)
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
git clone https://github.com/vicfei/dsh-WrenAI && cd dsh-wrenai
|
|
83
|
+
python3 scripts/ask.py --init <klines.json> # load data + build semantic views
|
|
84
|
+
python3 scripts/ask.py --ask "average close and annualized volatility over the last 30 days?"
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Real smoke (binance BTC/USDT 365d, 2026-11-17): *avg close 75,870 · vol 36.7%*
|
|
88
|
+
— with the vol metric's caliber formula (`sqrt(252×mean(ret²))`, 30d rolling,
|
|
89
|
+
non-centered simplification stated) attached to the answer. The first generated
|
|
90
|
+
SQL had a syntax error; the engine self-healed on the error feedback.
|
|
91
|
+
|
|
92
|
+
Output is English-first; `--lang zh` switches the user-facing labels to Chinese.
|
|
93
|
+
|
|
94
|
+
## How it works
|
|
95
|
+
|
|
96
|
+
```
|
|
97
|
+
semantic model (YAML: dimensions / measures / metrics + calibers)
|
|
98
|
+
│ the LLM sees ONLY this catalog — never raw schemas
|
|
99
|
+
▼
|
|
100
|
+
single SELECT ── deterministic guard (one statement, SELECT-only)
|
|
101
|
+
│ └─ executor error → self-heal retry with the error fed back
|
|
102
|
+
▼
|
|
103
|
+
numbers + calibers together (a result without its definitions is a bug)
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Execution is pluggable: SQLite (default, stdlib) or PostgreSQL via
|
|
107
|
+
`--pg "postgresql://…"` / `--weknora`. `scripts/pgwire.py` is a
|
|
108
|
+
**zero-dependency** PostgreSQL wire-protocol client (SCRAM-SHA-256 with
|
|
109
|
+
server-signature verification, MD5, cleartext, optional TLS) — clean-room from
|
|
110
|
+
the public protocol docs, so the whole kit stays stdlib-only Python plus a
|
|
111
|
+
Node shim.
|
|
112
|
+
|
|
113
|
+
## Status & roadmap
|
|
114
|
+
|
|
115
|
+
v0.1 — semantic engine, PG backend, WeKnora bridge; hermetic tests + CI green.
|
|
116
|
+
[ROADMAP](ROADMAP.md): M1 metric versioning + caliber drift alarms; M2
|
|
117
|
+
MDL-style model compatibility + DuckDB backend.
|
|
118
|
+
|
|
119
|
+
## License & disclaimer
|
|
120
|
+
|
|
121
|
+
MIT; concept credit to WrenAI (multi-licensed upstream, core Apache-2.0) —
|
|
122
|
+
zero upstream code used. **Not investment advice.**
|
package/README.zh-CN.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# dsh-WrenAI — DeepSeek Harness 的自托管语义层
|
|
2
|
+
|
|
3
|
+
**自然语言问数,数字带着指标口径一起回来。** 语义模型(维度/度量/指标+口径)
|
|
4
|
+
站在问题与 SQL 之间——这就是受治理的数据问数与裸 text-to-SQL 的分界线。
|
|
5
|
+
|
|
6
|
+
[English](README.md) | [简体中文](README.zh-CN.md)
|
|
7
|
+
|
|
8
|
+
[](https://github.com/vicfei/dsh-WrenAI/actions/workflows/ci.yml)
|
|
9
|
+
[](https://www.npmjs.com/package/dsh-wrenai)
|
|
10
|
+
[](LICENSE)
|
|
11
|
+
[](README.zh-CN.md#安装)
|
|
12
|
+
[](https://github.com/Canner/WrenAI)
|
|
13
|
+
|
|
14
|
+
> 原生实现 [WrenAI](https://github.com/Canner/WrenAI)(18.3k★ —— GenBI / 受治理
|
|
15
|
+
> text-to-SQL)的差异核心,面向 [DeepSeek Harness](https://github.com/deepseek-ai):
|
|
16
|
+
> **自托管语义层**。净室实现,零上游代码关系,无隶属。
|
|
17
|
+
> 家族:12 仓量化栈 —— [dsh-TradingAgents](https://github.com/vicfei/dsh-TradingAgents) ·
|
|
18
|
+
> [dsh-OpenBB](https://github.com/vicfei/dsh-OpenBB) ·
|
|
19
|
+
> [dsh-CCXT](https://github.com/vicfei/dsh-CCXT) ·
|
|
20
|
+
> [dsh-quantstats](https://github.com/vicfei/dsh-quantstats) ·
|
|
21
|
+
> [dsh-freqtrade](https://github.com/vicfei/dsh-freqtrade) …
|
|
22
|
+
|
|
23
|
+
## 头条特性:问你的 WeKnora 任何事
|
|
24
|
+
|
|
25
|
+
[WeKnora](https://github.com/WeKnora/WeKnora)(自托管 RAG 平台)没有自我可观测性:
|
|
26
|
+
审计日志**只记写侧动作**,"这个知识库到底被用了多少次?"在 UI 里无法回答。
|
|
27
|
+
dsh-WrenAI 直连部署的 PostgreSQL——零配置,DSN 从 `~/WeKnora/.env` 解析——
|
|
28
|
+
用预置的受治理模型作答,并附上诚实口径:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
dsh-wrenai --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
```text
|
|
35
|
+
## Ask: how many times was the test knowledge base queried in the past month?
|
|
36
|
+
| kb_query_events |
|
|
37
|
+
|-----------------|
|
|
38
|
+
| 2 |
|
|
39
|
+
## Calibers riding with this result (metric definitions used)
|
|
40
|
+
- kb_query_events: DISTINCT COUNT of assistant messages citing at least one
|
|
41
|
+
chunk of the KB in the time window … HONEST LIMIT: WeKnora audit_logs record
|
|
42
|
+
write-side actions only, so retrieval usage is measured by citations
|
|
43
|
+
actually served — this counts questions whose answers cited the KB, not raw
|
|
44
|
+
retrieval calls.
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
(真实 dev 部署,2026-11-19;已与手写 SQL 对账。)`kb_query_events`(去重问题数)
|
|
48
|
+
与 `kb_citations`(引用出现次数)是同一 join 路径上的两个口径——"本季度哪些库被引用
|
|
49
|
+
最多?"会把两者并排列出(live dev 机上 hotpot-pool-500:1287 事件 / 34948 引用)。
|
|
50
|
+
|
|
51
|
+
## 为什么(NL2SQL 插件已有——缺的是语义层)
|
|
52
|
+
|
|
53
|
+
| | 裸 NL2SQL 插件 | **dsh-WrenAI** |
|
|
54
|
+
|---|---|---|
|
|
55
|
+
| 模型看到什么 | 表结构 | **语义目录**——带口径的指标、带业务含义的维度 |
|
|
56
|
+
| 信任 | 靠信念 | **口径随行**:每个结果引用它实际使用的指标定义 |
|
|
57
|
+
| SQL 安全 | 参差 | **单条 SELECT 护栏**+自愈重试,确定性执行 |
|
|
58
|
+
| 后端 | 通常一个 | **SQLite(stdlib)或 PostgreSQL**——零依赖 wire 客户端 |
|
|
59
|
+
| 自托管 | — | 完全本地(YAML 模型+你的库),无云服务,无 pip 安装 |
|
|
60
|
+
|
|
61
|
+
## 安装
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
npm i -g dsh-wrenai # 或:dsh plugin add dsh-wrenai
|
|
65
|
+
dsh-wrenai --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
需要 PATH 上有 `python3`——引擎**纯 stdlib**(无需 pip 安装);npm 包是一个薄启动器
|
|
69
|
+
加 DSH 插件清单与技能。源码方式:clone 后直接 `python3 scripts/ask.py`。
|
|
70
|
+
|
|
71
|
+
## 快速开始(任意数据集)
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
git clone https://github.com/vicfei/dsh-WrenAI && cd dsh-wrenai
|
|
75
|
+
python3 scripts/ask.py --init <klines.json> # 装载数据 + 建语义视图
|
|
76
|
+
python3 scripts/ask.py --ask "average close and annualized volatility over the last 30 days?"
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
真实冒烟(binance BTC/USDT 365d,2026-11-17):*30 天均价 75,870 · 年化波动 36.7%*
|
|
80
|
+
——答案附带波动率指标的口径公式(`sqrt(252×mean(ret²))`,30 日滚动,声明了无均值
|
|
81
|
+
中心化的简化)。首次生成的 SQL 有语法错误,引擎靠错误反馈自愈重试通过。
|
|
82
|
+
|
|
83
|
+
输出英文优先;`--lang zh` 把用户可见标签切为中文。
|
|
84
|
+
|
|
85
|
+
## 工作原理
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
语义模型(YAML:维度/度量/指标 + 口径)
|
|
89
|
+
│ LLM 只看这个目录——绝不看裸表结构
|
|
90
|
+
▼
|
|
91
|
+
单条 SELECT ── 确定性护栏(仅一条语句、仅 SELECT)
|
|
92
|
+
│ └─ 执行错误 → 错误回带自愈重试一次
|
|
93
|
+
▼
|
|
94
|
+
数字 + 口径一起出(没有定义的数字即为 bug)
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
执行层可插拔:SQLite(默认,stdlib)或 PostgreSQL(`--pg "postgresql://…"` /
|
|
98
|
+
`--weknora`)。`scripts/pgwire.py` 是**零依赖**的 PostgreSQL wire 协议客户端
|
|
99
|
+
(SCRAM-SHA-256 含服务端签名校验、MD5、cleartext、可选 TLS)——依公开协议文档
|
|
100
|
+
净室实现,整套工具保持纯 stdlib Python 加一个 Node 启动器。
|
|
101
|
+
|
|
102
|
+
## 状态与路线图
|
|
103
|
+
|
|
104
|
+
v0.1——语义引擎、PG 后端、WeKnora 桥;hermetic 测试 + CI 绿。
|
|
105
|
+
[ROADMAP](ROADMAP.md):M1 指标版本化+口径漂移告警;M2 MDL 风格模型兼容+DuckDB 后端。
|
|
106
|
+
|
|
107
|
+
## 许可与免责
|
|
108
|
+
|
|
109
|
+
MIT;概念致谢 WrenAI(上游多许可,core Apache-2.0)——零上游代码。
|
|
110
|
+
**非投资建议。**
|
package/ROADMAP.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# ROADMAP — dsh-WrenAI
|
|
2
|
+
|
|
3
|
+
North star: the family's semantic-layer seat, riding WrenAI's 18.3k★ gravity.
|
|
4
|
+
Naming on all surfaces: GitHub = dsh-WrenAI (mirror CamelCase) / npm = dsh-wrenai.
|
|
5
|
+
Positioning (research-verified): DSH has raw NL2SQL seats, **nobody ships a
|
|
6
|
+
self-hosted semantic layer** — this repo takes that gap, and "calibers ride with
|
|
7
|
+
results" is the product soul.
|
|
8
|
+
|
|
9
|
+
## Milestones
|
|
10
|
+
|
|
11
|
+
### M0 ✅ (2026-11-17)
|
|
12
|
+
Semantic model (dimensions/measures/metrics + calibers in YAML) + SQLite semantic
|
|
13
|
+
view (metric windows precomputed) + ask engine (LLM sees the catalog, emits a
|
|
14
|
+
single SELECT; guard + self-heal retry; calibers ride with the result).
|
|
15
|
+
Live smoke: BTC 365d ask (30d avg close / annualized vol + caliber formulas),
|
|
16
|
+
one SQL syntax error self-healed on error feedback.
|
|
17
|
+
|
|
18
|
+
### M0.5 · Release packaging — partially done
|
|
19
|
+
- [x] README polish + npm 0.0.1 placeholder (owner-published)
|
|
20
|
+
- [ ] ask screenshots (with the calibers section) + marketplace + channels
|
|
21
|
+
|
|
22
|
+
### W1 · PostgreSQL backend + WeKnora ops bridge (in flight, 2026-11-18)
|
|
23
|
+
WrenAI connects directly to WeKnora's PostgreSQL: ops ask natural-language
|
|
24
|
+
questions about WeKnora's own operational data ("how many times was the test KB
|
|
25
|
+
queried in the past week?"). WeKnora lacks self-observability; WrenAI lacked a
|
|
26
|
+
real data source — each solves the other's pain.
|
|
27
|
+
- [x] W1.1 EN-first conversion with `--lang zh` fallback (family rule)
|
|
28
|
+
- [x] W1.2 PostgreSQL execution backend — pluggable executor behind the same
|
|
29
|
+
single-SELECT guard; zero-dependency pure-stdlib PG wire client
|
|
30
|
+
(SCRAM-SHA-256 auth, server signature verified); `--pg "postgresql://…"` DSN
|
|
31
|
+
- [x] W1.3 `--weknora` preset semantic model — zero-config (reads
|
|
32
|
+
`~/WeKnora/.env`), governed model over `audit_logs` / `knowledge_bases` /
|
|
33
|
+
`knowledges` / `chunks` / `sessions` / `messages`, honest calibers included
|
|
34
|
+
("KB queried" = retrieval-cited assistant messages via chunk refs, because
|
|
35
|
+
`audit_logs` records write-side actions only; kb_query_events vs
|
|
36
|
+
kb_citations are two calibers). Live smoke hit hand-verified ground truth
|
|
37
|
+
(test KB: 2 events / 30d; hotpot-pool-500: 1287 events / 34948 citations / 90d)
|
|
38
|
+
- [x] W1.4 npm 0.1.0 release train — root package (bin `dsh-wrenai`, DSH bundle
|
|
39
|
+
manifest + skill, files whitelist), hermetic test suite + CI, submission
|
|
40
|
+
kit staged in `docs/marketplace-submission/` (fires after owner publishes
|
|
41
|
+
npm 0.1.0 — 2FA). awesome-dsh-plugin PR follows the #16227 precedent.
|
|
42
|
+
|
|
43
|
+
### M1 · Governance depth (week 2-3)
|
|
44
|
+
- [ ] joins / multi-table models + metric versioning + caliber drift alarms
|
|
45
|
+
(same question across versions, side by side)
|
|
46
|
+
- [ ] result cache and a query audit log (what was asked, which caliber version)
|
|
47
|
+
|
|
48
|
+
### M2 · Compatibility & multi-DB (week 4-6)
|
|
49
|
+
- [ ] MDL-style model file compatibility layer (WrenAI ecosystem models import directly)
|
|
50
|
+
- [ ] DuckDB execution backend + typed tools for the harness plugin surface
|
|
51
|
+
|
|
52
|
+
## Discipline
|
|
53
|
+
- Calibers ride with results — never numbers alone; single-SELECT guard is
|
|
54
|
+
non-negotiable; metric definition changes are version bumps, not edits-in-place.
|
package/SKILL.md
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dsh-wrenai
|
|
3
|
+
description: Self-hosted semantic layer for DeepSeek Harness — natural-language data questions over a governed model (dimensions/measures/metrics with calibers); single-SELECT guard, self-heal SQL, calibers ride with every result.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Use this skill for "问数" over any dataset with a semantic model. Calibers are
|
|
7
|
+
the contract: numbers never travel without their definitions.
|
|
8
|
+
|
|
9
|
+
## Run
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
KIT=<this-directory>
|
|
13
|
+
python3 $KIT/scripts/ask.py --init <klines.json>
|
|
14
|
+
python3 $KIT/scripts/ask.py --ask "<question>" # English-first output
|
|
15
|
+
python3 $KIT/scripts/ask.py --ask "<question>" --lang zh # Chinese fallback
|
|
16
|
+
# WeKnora ops: zero-config bridge to a WeKnora deployment's own PostgreSQL
|
|
17
|
+
python3 $KIT/scripts/ask.py --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
18
|
+
# any PostgreSQL: --pg "postgresql://user:pass@host:5432/db" --model <yaml>
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Semantic model lives in `semantic/models.yaml` (edit for your dataset; the view
|
|
22
|
+
SQL builds from it). Data: family klines JSON or any tabular load you wire in.
|
|
23
|
+
|
|
24
|
+
## Discipline
|
|
25
|
+
|
|
26
|
+
- Always quote the 口径 section with results — never numbers alone.
|
|
27
|
+
- Single-SELECT guard is non-negotiable; report self-heal events honestly.
|
|
28
|
+
- Metric definition changes are version bumps, not edits-in-place.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// dsh-wrenai CLI shim — spawns the stdlib-only Python engine with args passed through.
|
|
3
|
+
const { spawnSync } = require("child_process");
|
|
4
|
+
const path = require("path");
|
|
5
|
+
|
|
6
|
+
const engine = path.join(__dirname, "..", "scripts", "ask.py");
|
|
7
|
+
const args = process.argv.slice(2);
|
|
8
|
+
const candidates = process.platform === "win32" ? ["python", "python3"] : ["python3", "python"];
|
|
9
|
+
|
|
10
|
+
for (const py of candidates) {
|
|
11
|
+
const r = spawnSync(py, [engine, ...args], { stdio: "inherit" });
|
|
12
|
+
if (!r.error) process.exit(r.status ?? 1);
|
|
13
|
+
}
|
|
14
|
+
console.error("dsh-wrenai: python3 not found on PATH (the engine is stdlib-only Python 3, no pip installs needed)");
|
|
15
|
+
process.exit(127);
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# dsh-wrenai bundle layer: inserts the plugin into the profile's layer stack.
|
|
2
|
+
# The skill (skills/dsh-wrenai/SKILL.md) is discovered by dsh-skill-filesystem;
|
|
3
|
+
# the CLI stays on PATH via the npm bin. The engine itself is stdlib-only Python
|
|
4
|
+
# (scripts/ask.py + scripts/pgwire.py) — nothing to bundle beyond discovery.
|
|
5
|
+
- insert:
|
|
6
|
+
- id: dsh-wrenai
|
|
7
|
+
name: 'dsh-wrenai'
|
package/package.json
CHANGED
|
@@ -1,6 +1,54 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-wrenai",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Self-hosted semantic layer for DeepSeek Harness — natural-language data questions where metric calibers ride with every result. Single-SELECT guard, self-heal SQL, SQLite/PostgreSQL execution (zero-dependency wire client, SCRAM-SHA-256 verified), and a zero-config --weknora bridge: ask your WeKnora deployment anything.",
|
|
5
|
+
"bin": {
|
|
6
|
+
"dsh-wrenai": "bin/dsh-wrenai.js"
|
|
7
|
+
},
|
|
8
|
+
"dsh": {
|
|
9
|
+
"bundle": {
|
|
10
|
+
"patch": "./cordis.patch.yml"
|
|
11
|
+
},
|
|
12
|
+
"manifestVersion": 1
|
|
13
|
+
},
|
|
14
|
+
"files": [
|
|
15
|
+
"bin",
|
|
16
|
+
"scripts/*.py",
|
|
17
|
+
"skills",
|
|
18
|
+
"semantic/models.yaml",
|
|
19
|
+
"semantic/weknora.yaml",
|
|
20
|
+
"cordis.patch.yml",
|
|
21
|
+
"SKILL.md",
|
|
22
|
+
"README.md",
|
|
23
|
+
"README.zh-CN.md",
|
|
24
|
+
"ROADMAP.md",
|
|
25
|
+
"LICENSE"
|
|
26
|
+
],
|
|
27
|
+
"scripts": {
|
|
28
|
+
"test": "python3 tests/test_engine.py && node bin/dsh-wrenai.js --help"
|
|
29
|
+
},
|
|
30
|
+
"engines": {
|
|
31
|
+
"node": ">=16"
|
|
32
|
+
},
|
|
33
|
+
"keywords": [
|
|
34
|
+
"deepseek-harness",
|
|
35
|
+
"dsh",
|
|
36
|
+
"dsh-plugin",
|
|
37
|
+
"wrenai",
|
|
38
|
+
"semantic-layer",
|
|
39
|
+
"text-to-sql",
|
|
40
|
+
"data-analytics",
|
|
41
|
+
"metrics-governance",
|
|
42
|
+
"weknora",
|
|
43
|
+
"postgresql",
|
|
44
|
+
"scram-sha-256",
|
|
45
|
+
"observability"
|
|
46
|
+
],
|
|
47
|
+
"repository": {
|
|
48
|
+
"type": "git",
|
|
49
|
+
"url": "git+https://github.com/vicfei/dsh-WrenAI.git"
|
|
50
|
+
},
|
|
51
|
+
"homepage": "https://github.com/vicfei/dsh-WrenAI#readme",
|
|
52
|
+
"license": "MIT",
|
|
53
|
+
"author": "vicfei"
|
|
54
|
+
}
|
package/scripts/ask.py
ADDED
|
@@ -0,0 +1,274 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Self-hosted semantic layer + calibers-ride-with-results ask-data engine (WrenAI concept seat, M0)
|
|
3
|
+
|
|
4
|
+
Upstream WrenAI (18.3k★, multi-licensed: core Apache-2.0) = GenBI / governed text-to-SQL
|
|
5
|
+
with a semantic layer. This plugin natively implements the differentiated core — the
|
|
6
|
+
**self-hosted semantic layer**: metrics carry versioned calibers, generated SQL runs
|
|
7
|
+
against semantic views, and every result ships with the caliber statements of the
|
|
8
|
+
metrics it used ("calibers ride with results" — the line vs raw NL2SQL, and the
|
|
9
|
+
data-asking edition of the factor-authentication caliber discipline). Clean-room,
|
|
10
|
+
zero upstream code.
|
|
11
|
+
|
|
12
|
+
Pipeline: --init loads klines into SQLite and builds the semantic view (metric windows
|
|
13
|
+
precomputed) → on ask, the LLM sees only the semantic catalog (dimensions/metrics/
|
|
14
|
+
calibers) and emits one SELECT → deterministic guard (single SELECT only) → execute →
|
|
15
|
+
numbers + calibers together. Execution is pluggable: SQLite (default, stdlib) or
|
|
16
|
+
PostgreSQL via --pg (pgwire, a zero-dependency wire-protocol client) — the semantic
|
|
17
|
+
layer above (catalog → guarded SELECT → calibers ride) is identical either way.
|
|
18
|
+
|
|
19
|
+
Usage:
|
|
20
|
+
python3 scripts/ask.py --init /tmp/btc_365.json
|
|
21
|
+
python3 scripts/ask.py --ask "average close and annualized volatility over the last 30 days?"
|
|
22
|
+
python3 scripts/ask.py --ask "最近30天的平均收盘价和年化波动率是多少?" --lang zh
|
|
23
|
+
python3 scripts/ask.py --ask "..." --pg "postgresql://user:pass@host:5432/db" --model semantic/ops.yaml
|
|
24
|
+
"""
|
|
25
|
+
import argparse, datetime as dt, json, math, os, re, sqlite3, subprocess, sys
|
|
26
|
+
|
|
27
|
+
import pgwire
|
|
28
|
+
|
|
29
|
+
KIT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
|
30
|
+
DB = os.path.join(KIT, "semantic", "semantic.db")
|
|
31
|
+
MODEL = os.path.join(KIT, "semantic", "models.yaml")
|
|
32
|
+
|
|
33
|
+
# English is the first language (family rule); Chinese is the --lang zh fallback.
|
|
34
|
+
LANG = "en"
|
|
35
|
+
STR = {
|
|
36
|
+
"en": {
|
|
37
|
+
"ask": "## Ask: {q}",
|
|
38
|
+
"calibers": "## Calibers riding with this result (metric definitions used)",
|
|
39
|
+
"guard": "guard: single SELECT only — rejected: {sql}",
|
|
40
|
+
"init": "init → {db} · {symbol} · {n} bars · view v_metrics (calibers in semantic/models.yaml)",
|
|
41
|
+
"retry_hint": "\n\nThe previously generated SQL failed to execute with error: {err}\nFix it and output again.",
|
|
42
|
+
"footer": "---\n*engine: semantic-ask v0 · single-SELECT guard · self-heal retry · backend: {backend} · calibers versioned in semantic/models.yaml · {ts}*",
|
|
43
|
+
},
|
|
44
|
+
"zh": {
|
|
45
|
+
"ask": "## 问数:{q}",
|
|
46
|
+
"calibers": "## 口径随行(本次引用的指标定义)",
|
|
47
|
+
"guard": "护栏:仅允许单条 SELECT,拒绝:{sql}",
|
|
48
|
+
"init": "init → {db} · {symbol} · {n} bars · 视图 v_metrics(口径见 semantic/models.yaml)",
|
|
49
|
+
"retry_hint": "\n\n上次生成的 SQL 执行报错:{err}\n请修正后重新输出。",
|
|
50
|
+
"footer": "---\n*engine: semantic-ask v0 · 单条 SELECT 护栏 · 自愈重试 · 后端:{backend} · 口径版本随 semantic/models.yaml · {ts}*",
|
|
51
|
+
},
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
def t(key, **kw):
|
|
56
|
+
return STR[LANG].get(key, key).format(**kw)
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
def _split_inline(body):
|
|
60
|
+
# split an inline {k: v, ...} body on commas, but never inside double quotes
|
|
61
|
+
parts, buf, inq = [], [], False
|
|
62
|
+
for ch in body:
|
|
63
|
+
if ch == '"':
|
|
64
|
+
inq = not inq
|
|
65
|
+
elif ch == "," and not inq:
|
|
66
|
+
parts.append("".join(buf)); buf = []; continue
|
|
67
|
+
buf.append(ch)
|
|
68
|
+
parts.append("".join(buf))
|
|
69
|
+
return [p.strip() for p in parts if p.strip()]
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
def load_yaml(path):
|
|
73
|
+
out, cur, section = {}, None, None
|
|
74
|
+
for line in open(path):
|
|
75
|
+
line = line.rstrip()
|
|
76
|
+
if not line or line.lstrip().startswith("#"):
|
|
77
|
+
continue
|
|
78
|
+
m = re.match(r"^(\w+):\s*(.*)$", line)
|
|
79
|
+
if m and not line.startswith(" "):
|
|
80
|
+
cur, section = m.group(1), m.group(2) or None
|
|
81
|
+
out[cur] = section if section else {}
|
|
82
|
+
if section is None:
|
|
83
|
+
out[cur] = {}
|
|
84
|
+
else:
|
|
85
|
+
m2 = re.match(r"^\s+(\w+):\s*\{(.+)\}\s*$", line)
|
|
86
|
+
if m2 and cur in ("dimensions", "measures", "metrics", "tables"):
|
|
87
|
+
name, body = m2.group(1), m2.group(2)
|
|
88
|
+
d = {}
|
|
89
|
+
for part in _split_inline(body):
|
|
90
|
+
k, _, v = part.partition(":")
|
|
91
|
+
d[k.strip()] = v.strip().strip('"')
|
|
92
|
+
out[cur][name] = d
|
|
93
|
+
else:
|
|
94
|
+
m3 = re.match(r"^\s+(\w+):\s*(.+)$", line)
|
|
95
|
+
if m3 and cur == "joins":
|
|
96
|
+
out.setdefault(cur, {})[m3.group(1)] = m3.group(2).strip().strip('"')
|
|
97
|
+
return out
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def weknora_dsn():
|
|
101
|
+
"""Zero-config DSN: read the WeKnora deployment's own .env and map the
|
|
102
|
+
container-internal DB host to the host's loopback (published port)."""
|
|
103
|
+
from urllib.parse import quote
|
|
104
|
+
wk = os.environ.get("WEKNORA_HOME") or os.path.expanduser("~/WeKnora")
|
|
105
|
+
envf = os.path.join(wk, ".env")
|
|
106
|
+
if not os.path.exists(envf):
|
|
107
|
+
sys.exit("--weknora: %s not found (set WREN_PG_DSN or WEKNORA_HOME)" % envf)
|
|
108
|
+
env = {}
|
|
109
|
+
for line in open(envf):
|
|
110
|
+
line = line.strip()
|
|
111
|
+
if line and not line.startswith("#") and "=" in line:
|
|
112
|
+
k, _, v = line.partition("=")
|
|
113
|
+
env[k.strip()] = v.strip()
|
|
114
|
+
host = env.get("DB_HOST", "postgres")
|
|
115
|
+
if host in ("postgres", "db", "app"): # container-internal name → host loopback
|
|
116
|
+
host = "127.0.0.1"
|
|
117
|
+
return "postgresql://%s:%s@%s:%s/%s" % (
|
|
118
|
+
quote(env.get("DB_USER") or "postgres", safe=""),
|
|
119
|
+
quote(env.get("DB_PASSWORD") or "", safe=""),
|
|
120
|
+
host, env.get("DB_PORT") or "5432", env.get("DB_NAME") or "WeKnora")
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
def init_db(klines_json):
|
|
124
|
+
data = json.load(open(klines_json))
|
|
125
|
+
symbol, bars = next(iter(data.items()))
|
|
126
|
+
con = sqlite3.connect(DB)
|
|
127
|
+
con.execute("DROP TABLE IF EXISTS bars")
|
|
128
|
+
con.execute("CREATE TABLE bars (symbol TEXT, date TEXT, open REAL, close REAL, high REAL, low REAL, volume REAL)")
|
|
129
|
+
con.executemany("INSERT INTO bars VALUES (?,?,?,?,?,?,?)",
|
|
130
|
+
[(symbol, b[0], b[1], b[2], b[3], b[4], b[5]) for b in bars])
|
|
131
|
+
con.execute("""
|
|
132
|
+
CREATE VIEW v_metrics AS
|
|
133
|
+
WITH r AS (
|
|
134
|
+
SELECT symbol, date, close,
|
|
135
|
+
close / LAG(close, 1) OVER w - 1 AS ret_1d,
|
|
136
|
+
close / LAG(close, 30) OVER w - 1 AS ret_30d,
|
|
137
|
+
close / MAX(close) OVER w2 - 1 AS dd_120
|
|
138
|
+
FROM bars
|
|
139
|
+
WINDOW w AS (PARTITION BY symbol ORDER BY date),
|
|
140
|
+
w2 AS (PARTITION BY symbol ORDER BY date ROWS BETWEEN 119 PRECEDING AND CURRENT ROW)
|
|
141
|
+
)
|
|
142
|
+
SELECT symbol, date, close, ret_1d, ret_30d, dd_120,
|
|
143
|
+
SQRT(252 * AVG(ret_1d * ret_1d) OVER (PARTITION BY symbol ORDER BY date ROWS BETWEEN 29 PRECEDING AND CURRENT ROW)) AS vol_30d_ann
|
|
144
|
+
FROM r
|
|
145
|
+
""")
|
|
146
|
+
con.commit()
|
|
147
|
+
n = con.execute("SELECT COUNT(*) FROM bars").fetchone()[0]
|
|
148
|
+
print(t("init", db=DB, symbol=symbol, n=n))
|
|
149
|
+
con.close()
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def llm(catalog, question, dialect, target):
|
|
153
|
+
key = subprocess.run(["bash", "-c", 'grep -m1 "^export DEEPSEEK_API_KEY=" ~/.bashrc | cut -d= -f2'],
|
|
154
|
+
capture_output=True, text=True).stdout.strip().strip('"')
|
|
155
|
+
sysmsg = (f"You are a semantic-layer SQL generator. Write a SINGLE SELECT against {target} "
|
|
156
|
+
f"({dialect}; aggregate/window functions allowed). "
|
|
157
|
+
"INSERT/UPDATE/DELETE/DDL/multiple statements are forbidden. "
|
|
158
|
+
"Metric calibers are binding definitions — follow the join paths they prescribe, "
|
|
159
|
+
"and prefer metric/dimension names from the catalog. "
|
|
160
|
+
"Output JSON {\"sql\":\"...\",\"metrics_used\":[metric names]}. "
|
|
161
|
+
"Metric caliber catalog (the semantic source of any result):\n" + catalog)
|
|
162
|
+
body = {"model": "deepseek-chat", "temperature": 0, "response_format": {"type": "json_object"},
|
|
163
|
+
"messages": [{"role": "system", "content": sysmsg}, {"role": "user", "content": question}]}
|
|
164
|
+
r = subprocess.run(["curl", "-s", "-m", "120", "https://api.deepseek.com/chat/completions",
|
|
165
|
+
"-H", f"Authorization: Bearer {key}", "-H", "Content-Type: application/json",
|
|
166
|
+
"-d", json.dumps(body)], capture_output=True, text=True)
|
|
167
|
+
return json.loads(json.loads(r.stdout)["choices"][0]["message"]["content"])
|
|
168
|
+
|
|
169
|
+
|
|
170
|
+
def guarded(sql):
|
|
171
|
+
sql = sql.strip().rstrip(";").strip()
|
|
172
|
+
if not re.match(r"(?is)^SELECT\s.+$", sql) or ";" in sql:
|
|
173
|
+
sys.exit(t("guard", sql=sql[:120]))
|
|
174
|
+
return sql
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
def run_sql(dsn, sql):
|
|
178
|
+
"""Execute a guarded SELECT on the selected backend → (cols, rows).
|
|
179
|
+
Raises sqlite3.OperationalError / pgwire.PGError with the executor's message."""
|
|
180
|
+
if dsn:
|
|
181
|
+
con = pgwire.connect(dsn)
|
|
182
|
+
try:
|
|
183
|
+
return con.query(sql)
|
|
184
|
+
finally:
|
|
185
|
+
con.close()
|
|
186
|
+
con = sqlite3.connect(DB)
|
|
187
|
+
try:
|
|
188
|
+
cur = con.execute(sql)
|
|
189
|
+
rows = cur.fetchall()
|
|
190
|
+
return [d[0] for d in cur.description], rows
|
|
191
|
+
finally:
|
|
192
|
+
con.close()
|
|
193
|
+
|
|
194
|
+
|
|
195
|
+
def fmt(v):
|
|
196
|
+
if v is None:
|
|
197
|
+
return ""
|
|
198
|
+
if isinstance(v, float):
|
|
199
|
+
return f"{v:.6g}"
|
|
200
|
+
if not isinstance(v, int): # PG text format → compress numerics for display
|
|
201
|
+
try:
|
|
202
|
+
return f"{float(v):.6g}"
|
|
203
|
+
except ValueError:
|
|
204
|
+
return str(v)
|
|
205
|
+
return str(v)
|
|
206
|
+
|
|
207
|
+
|
|
208
|
+
def main():
|
|
209
|
+
global LANG
|
|
210
|
+
ap = argparse.ArgumentParser()
|
|
211
|
+
ap.add_argument("--init")
|
|
212
|
+
ap.add_argument("--ask")
|
|
213
|
+
ap.add_argument("--pg", metavar="DSN", help='PostgreSQL backend, e.g. "postgresql://user:pass@host:5432/db"')
|
|
214
|
+
ap.add_argument("--weknora", action="store_true",
|
|
215
|
+
help="zero-config WeKnora preset: PG DSN from ~/WeKnora/.env (WREN_PG_DSN/WEKNORA_HOME override) + semantic/weknora.yaml")
|
|
216
|
+
ap.add_argument("--model", metavar="PATH", help="semantic model YAML (default semantic/models.yaml; --weknora defaults to semantic/weknora.yaml)")
|
|
217
|
+
ap.add_argument("--lang", choices=("en", "zh"), default="en",
|
|
218
|
+
help="output language (default en; zh is the fallback)")
|
|
219
|
+
args = ap.parse_args()
|
|
220
|
+
LANG = args.lang
|
|
221
|
+
pg = args.pg
|
|
222
|
+
if args.weknora:
|
|
223
|
+
pg = pg or os.environ.get("WREN_PG_DSN") or weknora_dsn()
|
|
224
|
+
model_path = args.model or (os.path.join(KIT, "semantic", "weknora.yaml") if args.weknora else MODEL)
|
|
225
|
+
model = load_yaml(model_path)
|
|
226
|
+
if args.init:
|
|
227
|
+
if pg:
|
|
228
|
+
sys.exit("guard: --init is the SQLite demo loader; --pg backends attach to existing data")
|
|
229
|
+
init_db(args.init)
|
|
230
|
+
return
|
|
231
|
+
if not args.ask:
|
|
232
|
+
print(__doc__); sys.exit(2)
|
|
233
|
+
dialect = "PostgreSQL" if pg else "SQLite"
|
|
234
|
+
if model.get("tables"):
|
|
235
|
+
target = "the PostgreSQL tables and join paths listed in the catalog"
|
|
236
|
+
catalog = json.dumps({k: model[k] for k in ("tables", "joins", "dimensions", "measures", "metrics") if model.get(k)},
|
|
237
|
+
ensure_ascii=False, indent=1)
|
|
238
|
+
else:
|
|
239
|
+
target = "the %s view %s" % (dialect, model.get("view") or model.get("table") or "v_metrics")
|
|
240
|
+
catalog = json.dumps({k: {n: v.get("caliber", v.get("desc")) for n, v in model.get(k, {}).items()}
|
|
241
|
+
for k in ("dimensions", "measures", "metrics")}, ensure_ascii=False, indent=1)
|
|
242
|
+
plan = llm(catalog, args.ask, dialect, target)
|
|
243
|
+
sql = guarded(plan["sql"])
|
|
244
|
+
try:
|
|
245
|
+
cols, rows = run_sql(pg, sql)
|
|
246
|
+
except (sqlite3.Error, pgwire.PGError) as e:
|
|
247
|
+
# executor error → retry once with the error fed back (semantic self-heal)
|
|
248
|
+
plan = llm(catalog, args.ask + t("retry_hint", err=e), dialect, target)
|
|
249
|
+
sql = guarded(plan["sql"])
|
|
250
|
+
try:
|
|
251
|
+
cols, rows = run_sql(pg, sql)
|
|
252
|
+
except (sqlite3.Error, pgwire.PGError) as e:
|
|
253
|
+
# honest failure: show the SQL and the executor's error, no bare traceback
|
|
254
|
+
print(f"```sql\n{sql}\n```\n")
|
|
255
|
+
sys.exit(f"self-heal exhausted — executor error persists: {e}")
|
|
256
|
+
print(t("ask", q=args.ask) + "\n")
|
|
257
|
+
print(f"```sql\n{sql}\n```\n")
|
|
258
|
+
print("| " + " | ".join(cols) + " |")
|
|
259
|
+
print("|" + "---|" * len(cols))
|
|
260
|
+
for r in rows[:12]:
|
|
261
|
+
print("| " + " | ".join(fmt(v) for v in r) + " |")
|
|
262
|
+
print("\n" + t("calibers"))
|
|
263
|
+
for m in plan.get("metrics_used", []):
|
|
264
|
+
meta = model.get("metrics", {}).get(m) or model.get("measures", {}).get(m)
|
|
265
|
+
if meta:
|
|
266
|
+
print(f"- **{m}**: {meta.get('caliber') or meta.get('desc')}")
|
|
267
|
+
backend = "SQLite (stdlib)"
|
|
268
|
+
if pg:
|
|
269
|
+
backend = "WeKnora PostgreSQL via pgwire" if args.weknora else "PostgreSQL via pgwire (stdlib wire client)"
|
|
270
|
+
print("\n" + t("footer", ts=dt.datetime.now().isoformat(timespec='seconds'), backend=backend))
|
|
271
|
+
|
|
272
|
+
|
|
273
|
+
if __name__ == "__main__":
|
|
274
|
+
main()
|
|
@@ -0,0 +1,224 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""pgwire — a minimal PostgreSQL frontend (wire protocol v3) client, stdlib only.
|
|
3
|
+
|
|
4
|
+
Scope: exactly what a read-only analytics engine needs — connect, authenticate
|
|
5
|
+
(trust / cleartext / MD5 / SCRAM-SHA-256), optionally upgrade to TLS
|
|
6
|
+
(sslmode=require), run one simple query, return text-format rows. Clean-room
|
|
7
|
+
from the public PostgreSQL frontend/backend protocol documentation; zero
|
|
8
|
+
dependencies, zero upstream code.
|
|
9
|
+
|
|
10
|
+
DSN: postgresql://user:password@host:port/database?sslmode=require|disable
|
|
11
|
+
Special characters in user/password must be percent-encoded (e.g. @ → %40).
|
|
12
|
+
|
|
13
|
+
Usage:
|
|
14
|
+
cols, rows = pgwire.connect("postgresql://postgres:p%40ss@localhost:5432/db").query("SELECT 1 AS one")
|
|
15
|
+
"""
|
|
16
|
+
import base64, hashlib, hmac, secrets, socket, ssl, struct
|
|
17
|
+
from urllib.parse import unquote, urlparse, parse_qs
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
class PGError(Exception):
|
|
21
|
+
"""Server-reported error (ErrorResponse) or client-side protocol failure."""
|
|
22
|
+
|
|
23
|
+
def __init__(self, fields):
|
|
24
|
+
self.fields = fields or {}
|
|
25
|
+
super().__init__(self.fields.get("M") or "PostgreSQL error")
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def parse_dsn(dsn):
|
|
29
|
+
u = urlparse(dsn)
|
|
30
|
+
if u.scheme not in ("postgres", "postgresql"):
|
|
31
|
+
raise PGError({"M": "not a postgresql:// DSN: %s" % dsn[:60]})
|
|
32
|
+
sslmode = (parse_qs(u.query).get("sslmode") or ["disable"])[0]
|
|
33
|
+
if sslmode not in ("disable", "require"):
|
|
34
|
+
raise PGError({"M": "unsupported sslmode %r (disable|require)" % sslmode})
|
|
35
|
+
return {"user": unquote(u.username or "") or "postgres",
|
|
36
|
+
"password": unquote(u.password or ""),
|
|
37
|
+
"host": u.hostname or "localhost",
|
|
38
|
+
"port": u.port or 5432,
|
|
39
|
+
"database": (u.path or "/").lstrip("/") or "postgres",
|
|
40
|
+
"sslmode": sslmode}
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
class Connection:
|
|
44
|
+
def __init__(self, dsn, timeout=30):
|
|
45
|
+
self.p = parse_dsn(dsn)
|
|
46
|
+
self.sock = socket.create_connection((self.p["host"], self.p["port"]), timeout=timeout)
|
|
47
|
+
self.sock.settimeout(timeout)
|
|
48
|
+
if self.p["sslmode"] == "require":
|
|
49
|
+
self._start_tls()
|
|
50
|
+
self.tag = ""
|
|
51
|
+
self._startup()
|
|
52
|
+
|
|
53
|
+
# -- wire helpers ------------------------------------------------------
|
|
54
|
+
|
|
55
|
+
def _send(self, buf):
|
|
56
|
+
self.sock.sendall(buf)
|
|
57
|
+
|
|
58
|
+
def _msg(self, mtype, payload):
|
|
59
|
+
self._send(mtype + struct.pack("!I", len(payload) + 4) + payload)
|
|
60
|
+
|
|
61
|
+
def _recv_exact(self, n):
|
|
62
|
+
buf = b""
|
|
63
|
+
while len(buf) < n:
|
|
64
|
+
chunk = self.sock.recv(n - len(buf))
|
|
65
|
+
if not chunk:
|
|
66
|
+
raise PGError({"M": "connection closed by server"})
|
|
67
|
+
buf += chunk
|
|
68
|
+
return buf
|
|
69
|
+
|
|
70
|
+
def _recv_msg(self):
|
|
71
|
+
mtype = self._recv_exact(1)
|
|
72
|
+
(ln,) = struct.unpack("!I", self._recv_exact(4))
|
|
73
|
+
return mtype, self._recv_exact(ln - 4)
|
|
74
|
+
|
|
75
|
+
@staticmethod
|
|
76
|
+
def _fields(body):
|
|
77
|
+
# ErrorResponse/NoticeResponse: (1-byte code + c-string)* terminated by NUL
|
|
78
|
+
out, i = {}, 0
|
|
79
|
+
while i < len(body) and body[i:i + 1] != b"\x00":
|
|
80
|
+
j = body.index(b"\x00", i + 1)
|
|
81
|
+
out[body[i:i + 1].decode()] = body[i + 1:j].decode("utf-8", "replace")
|
|
82
|
+
i = j + 1
|
|
83
|
+
return out
|
|
84
|
+
|
|
85
|
+
# -- session setup -----------------------------------------------------
|
|
86
|
+
|
|
87
|
+
def _start_tls(self):
|
|
88
|
+
self._send(struct.pack("!II", 8, 80877103)) # SSLRequest
|
|
89
|
+
if self._recv_exact(1) != b"S":
|
|
90
|
+
raise PGError({"M": "server refused TLS (sslmode=require)"})
|
|
91
|
+
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
|
92
|
+
ctx.check_hostname = False
|
|
93
|
+
ctx.verify_mode = ssl.CERT_NONE # local/LAN analytics; not a trust root
|
|
94
|
+
self.sock = ctx.wrap_socket(self.sock)
|
|
95
|
+
|
|
96
|
+
def _startup(self):
|
|
97
|
+
params = [("user", self.p["user"]), ("database", self.p["database"]),
|
|
98
|
+
("application_name", "dsh-wrenai")]
|
|
99
|
+
payload = struct.pack("!I", 196608) # protocol 3.0
|
|
100
|
+
payload += b"".join(k.encode() + b"\x00" + v.encode() + b"\x00" for k, v in params) + b"\x00"
|
|
101
|
+
self._send(struct.pack("!I", len(payload) + 4) + payload)
|
|
102
|
+
while True:
|
|
103
|
+
mtype, body = self._recv_msg()
|
|
104
|
+
if mtype == b"E":
|
|
105
|
+
raise PGError(self._fields(body))
|
|
106
|
+
elif mtype == b"R":
|
|
107
|
+
self._auth(body)
|
|
108
|
+
elif mtype == b"Z": # ReadyForQuery — authenticated, ready
|
|
109
|
+
return
|
|
110
|
+
# ParameterStatus 'S' / BackendKeyData 'K' / NoticeResponse 'N' ignored
|
|
111
|
+
|
|
112
|
+
def _auth(self, body):
|
|
113
|
+
(code,) = struct.unpack("!i", body[:4])
|
|
114
|
+
if code == 0: # AuthenticationOk
|
|
115
|
+
return
|
|
116
|
+
if code == 3: # cleartext password
|
|
117
|
+
self._msg(b"p", self.p["password"].encode() + b"\x00")
|
|
118
|
+
elif code == 5: # md5 (user+password, then +salt)
|
|
119
|
+
inner = hashlib.md5((self.p["password"] + self.p["user"]).encode()).hexdigest().encode()
|
|
120
|
+
self._msg(b"p", (b"md5" + hashlib.md5(inner + body[4:8]).hexdigest()).encode() + b"\x00")
|
|
121
|
+
elif code == 10: # SASL — SCRAM-SHA-256
|
|
122
|
+
mechs = [m.decode() for m in body[4:].split(b"\x00") if m]
|
|
123
|
+
if "SCRAM-SHA-256" not in mechs:
|
|
124
|
+
raise PGError({"M": "no supported SASL mechanism; server offers %s" % mechs})
|
|
125
|
+
self._scram()
|
|
126
|
+
else:
|
|
127
|
+
raise PGError({"M": "unsupported auth request code %d" % code})
|
|
128
|
+
|
|
129
|
+
def _expect_auth(self):
|
|
130
|
+
while True:
|
|
131
|
+
mtype, body = self._recv_msg()
|
|
132
|
+
if mtype == b"E":
|
|
133
|
+
raise PGError(self._fields(body))
|
|
134
|
+
if mtype == b"R":
|
|
135
|
+
(code,) = struct.unpack("!i", body[:4])
|
|
136
|
+
return code, body[4:].decode("utf-8", "replace")
|
|
137
|
+
|
|
138
|
+
def _scram(self):
|
|
139
|
+
pw = self.p["password"].encode()
|
|
140
|
+
cnonce = base64.b64encode(secrets.token_bytes(18)).decode()
|
|
141
|
+
cfirst_bare = "n=%s,r=%s" % (self.p["user"], cnonce)
|
|
142
|
+
init = b"n,," + cfirst_bare.encode()
|
|
143
|
+
self._msg(b"p", b"SCRAM-SHA-256\x00" + struct.pack("!i", len(init)) + init)
|
|
144
|
+
code, sfirst = self._expect_auth()
|
|
145
|
+
if code != 11: # AuthenticationSASLContinue
|
|
146
|
+
raise PGError({"M": "expected SASLContinue, got auth code %d" % code})
|
|
147
|
+
attrs = dict(kv.split("=", 1) for kv in sfirst.split(","))
|
|
148
|
+
snonce, salt, iters = attrs["r"], base64.b64decode(attrs["s"]), int(attrs["i"])
|
|
149
|
+
if not snonce.startswith(cnonce):
|
|
150
|
+
raise PGError({"M": "server nonce does not extend client nonce"})
|
|
151
|
+
cfinal_wo = "c=biws,r=%s" % snonce
|
|
152
|
+
salted = hashlib.pbkdf2_hmac("sha256", pw, salt, iters)
|
|
153
|
+
ckey = hmac.new(salted, b"Client Key", hashlib.sha256).digest()
|
|
154
|
+
skey = hashlib.sha256(ckey).digest()
|
|
155
|
+
authmsg = ("%s,%s,%s" % (cfirst_bare, sfirst, cfinal_wo)).encode()
|
|
156
|
+
csig = hmac.new(skey, authmsg, hashlib.sha256).digest()
|
|
157
|
+
proof = bytes(a ^ b for a, b in zip(ckey, csig))
|
|
158
|
+
self._msg(b"p", ("%s,p=%s" % (cfinal_wo, base64.b64encode(proof).decode())).encode())
|
|
159
|
+
code, sfinal = self._expect_auth()
|
|
160
|
+
if code != 12: # AuthenticationSASLFinal
|
|
161
|
+
raise PGError({"M": "expected SASLFinal, got auth code %d" % code})
|
|
162
|
+
v = dict(kv.split("=", 1) for kv in sfinal.split(",")).get("v")
|
|
163
|
+
srvkey = hmac.new(salted, b"Server Key", hashlib.sha256).digest()
|
|
164
|
+
if v != base64.b64encode(hmac.new(srvkey, authmsg, hashlib.sha256).digest()).decode():
|
|
165
|
+
raise PGError({"M": "SCRAM server signature mismatch (channel not authenticated)"})
|
|
166
|
+
# AuthenticationOk follows in the _startup loop
|
|
167
|
+
|
|
168
|
+
# -- simple query ------------------------------------------------------
|
|
169
|
+
|
|
170
|
+
def query(self, sql):
|
|
171
|
+
"""Run one SQL statement; return (column_names, rows of str/None)."""
|
|
172
|
+
self._msg(b"Q", sql.encode() + b"\x00")
|
|
173
|
+
cols, rows, err = [], [], None
|
|
174
|
+
while True:
|
|
175
|
+
mtype, body = self._recv_msg()
|
|
176
|
+
if mtype == b"T": # RowDescription
|
|
177
|
+
(n,) = struct.unpack("!h", body[:2])
|
|
178
|
+
i = 2
|
|
179
|
+
for _ in range(n):
|
|
180
|
+
j = body.index(b"\x00", i)
|
|
181
|
+
cols.append(body[i:j].decode())
|
|
182
|
+
i = j + 1 + 4 + 2 + 4 + 2 + 4 + 2 # table oid, attnum, typoid, typlen, typmod, format
|
|
183
|
+
elif mtype == b"D": # DataRow (text format)
|
|
184
|
+
(n,) = struct.unpack("!h", body[:2])
|
|
185
|
+
i, row = 2, []
|
|
186
|
+
for _ in range(n):
|
|
187
|
+
(ln,) = struct.unpack("!i", body[i:i + 4])
|
|
188
|
+
i += 4
|
|
189
|
+
row.append(None if ln == -1 else body[i:i + ln].decode("utf-8", "replace"))
|
|
190
|
+
i += max(ln, 0)
|
|
191
|
+
rows.append(row)
|
|
192
|
+
elif mtype == b"C": # CommandComplete
|
|
193
|
+
self.tag = body[:-1].decode()
|
|
194
|
+
elif mtype == b"E": # ErrorResponse — surface after ReadyForQuery
|
|
195
|
+
err = PGError(self._fields(body))
|
|
196
|
+
elif mtype == b"Z": # ReadyForQuery
|
|
197
|
+
if err:
|
|
198
|
+
raise err
|
|
199
|
+
return cols, rows
|
|
200
|
+
# NoticeResponse 'N' / ParameterStatus 'S' ignored
|
|
201
|
+
|
|
202
|
+
def close(self):
|
|
203
|
+
try:
|
|
204
|
+
self._msg(b"X", b"")
|
|
205
|
+
self.sock.close()
|
|
206
|
+
except OSError:
|
|
207
|
+
pass
|
|
208
|
+
|
|
209
|
+
|
|
210
|
+
def connect(dsn, timeout=30):
|
|
211
|
+
return Connection(dsn, timeout)
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
if __name__ == "__main__": # self-test: pgwire.py <dsn> <sql>
|
|
215
|
+
import sys
|
|
216
|
+
con = connect(sys.argv[1])
|
|
217
|
+
try:
|
|
218
|
+
cols, rows = con.query(sys.argv[2])
|
|
219
|
+
print("\t".join(cols))
|
|
220
|
+
for r in rows:
|
|
221
|
+
print("\t".join("" if v is None else v for v in r))
|
|
222
|
+
print("(%s)" % con.tag, file=sys.stderr)
|
|
223
|
+
finally:
|
|
224
|
+
con.close()
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
dataset: crypto_daily
|
|
2
|
+
description: Daily crypto market semantic layer (demo model)
|
|
3
|
+
table: bars
|
|
4
|
+
dimensions:
|
|
5
|
+
date: {type: date, desc: trading day}
|
|
6
|
+
symbol: {type: string, desc: asset symbol}
|
|
7
|
+
measures:
|
|
8
|
+
close: {expr: close, unit: quote_ccy, desc: daily close price}
|
|
9
|
+
metrics:
|
|
10
|
+
ret_1d: {expr: ret_1d, caliber: "close_t / close_{t-1} − 1 (daily return)"}
|
|
11
|
+
ret_30d: {expr: ret_30d, caliber: "close_t / close_{t-30} − 1 (30-day cumulative return, NOT annualized)"}
|
|
12
|
+
vol_30d_ann: {expr: vol_30d_ann, caliber: "sqrt(252 × mean(ret_1d²)) — 30-day rolling window, annualized, simplified: no mean-centering"}
|
|
13
|
+
dd_120: {expr: dd_120, caliber: "close_t / max(close_{t-119..t}) − 1 (120-day rolling drawdown, ≤0)"}
|
|
14
|
+
view: v_metrics
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
dataset: weknora_ops
|
|
2
|
+
description: WeKnora self-observability — ask natural-language questions about the deployment itself (preset model, --weknora)
|
|
3
|
+
tables:
|
|
4
|
+
knowledge_bases: {key: id, cols: "id, tenant_id, name, created_at, deleted_at"}
|
|
5
|
+
knowledges: {key: id, cols: "id, knowledge_base_id, tenant_id, title, type, file_type, parse_status, enable_status, created_at, deleted_at"}
|
|
6
|
+
chunks: {key: id, cols: "id, knowledge_base_id, knowledge_id, chunk_index, is_enabled"}
|
|
7
|
+
sessions: {key: id, cols: "id, tenant_id, user_id, kind, created_at, deleted_at"}
|
|
8
|
+
messages: {key: id, cols: "id, session_id, tenant_id, role, model_id, is_fallback, agent_duration_ms, knowledge_references, created_at, deleted_at"}
|
|
9
|
+
audit_logs: {key: id, cols: "id, tenant_id, actor_user_id, actor_role, action, target_type, target_id, request_path, request_method, outcome, created_at"}
|
|
10
|
+
joins:
|
|
11
|
+
id_types: "ALL id columns (knowledge_bases.id, knowledges.id, chunks.id, sessions.id, messages.id) are varchar(36)/text, NOT uuid — never cast ids to ::uuid; ref->>'id' from jsonb is text and compares directly"
|
|
12
|
+
kb_documents: "knowledges.knowledge_base_id = knowledge_bases.id"
|
|
13
|
+
document_chunks: "chunks.knowledge_id = knowledges.id (chunks.knowledge_base_id also carries the KB directly)"
|
|
14
|
+
session_messages: "messages.session_id = sessions.id"
|
|
15
|
+
kb_retrieval_usage: "messages.knowledge_references is a jsonb array of {id: chunk_id, score, content}; explode with JOIN LATERAL jsonb_array_elements(messages.knowledge_references) ref ON TRUE, then chunks.id = ref->>'id' (text = varchar, no cast); KB name resolves via knowledge_bases.name (deleted_at IS NULL)"
|
|
16
|
+
dimensions:
|
|
17
|
+
day: {desc: "event day = created_at::date on the driving table"}
|
|
18
|
+
kb_name: {desc: knowledge base name (knowledge_bases.name)}
|
|
19
|
+
tenant_id: {desc: tenant (multi-tenant isolation dimension)}
|
|
20
|
+
action: {desc: "audit action vocabulary: knowledge.created, kb.created, kb.updated, kb.deleted, knowledge.reparse_started, rbac.member_added, rbac.access_denied, system.api_key_created, ..."}
|
|
21
|
+
outcome: {desc: "audit outcome: success | failure"}
|
|
22
|
+
parse_status: {desc: "document parse state: parsed | parsing | unprocessed | failed"}
|
|
23
|
+
file_type: {desc: document source file type (pdf, docx, md, ...)}
|
|
24
|
+
doc_type: {desc: "knowledge type: document | web link | faq ..."}
|
|
25
|
+
role: {desc: "message role: user | assistant"}
|
|
26
|
+
is_fallback: {desc: "answer fell back (retrieval/LLM unavailable)"}
|
|
27
|
+
measures:
|
|
28
|
+
doc_count: {desc: "documents (knowledges rows, deleted_at IS NULL)"}
|
|
29
|
+
kb_count: {desc: "knowledge bases (knowledge_bases rows, deleted_at IS NULL)"}
|
|
30
|
+
message_count: {desc: "chat messages (messages rows, deleted_at IS NULL)"}
|
|
31
|
+
session_count: {desc: "chat sessions (sessions rows, deleted_at IS NULL)"}
|
|
32
|
+
audit_event_count: {desc: "audit_logs rows"}
|
|
33
|
+
metrics:
|
|
34
|
+
kb_query_events: {caliber: "DISTINCT COUNT of assistant messages citing at least one chunk of the KB in the time window (messages.role='assistant', messages.deleted_at IS NULL, knowledge_references <> '[]'), joined ref->>'id' = chunks.id → chunks.knowledge_base_id. HONEST LIMIT: WeKnora audit_logs record write-side actions only, so retrieval usage is measured by citations actually served — this counts questions whose answers cited the KB, not raw retrieval calls. Windows: 'past week' = created_at > now() - interval '7 days', 'past month' = '30 days', 'past quarter' = '90 days'."}
|
|
35
|
+
kb_citations: {caliber: "TOTAL citation occurrences (messages × references) for the KB via the same chunk join path — counts repeats within one answer, unlike kb_query_events which counts distinct messages; a single answer citing 6 chunks counts once in kb_query_events and 6 times here."}
|
|
36
|
+
doc_parse_success_rate: {caliber: "share of the KB's non-deleted knowledges with parse_status='parsed' — documents still parsing or failed are in the denominator (honest denominator: ALL non-deleted docs, not only finished ones)"}
|
|
37
|
+
audit_event_count_by_action: {caliber: "audit_logs rows per action in the window — write-side operations ONLY (knowledge.created, kb.*, rbac.*, system.*); retrieval and chat queries are NOT audited by WeKnora, do not look for them here"}
|
|
38
|
+
fallback_rate: {caliber: "share of assistant messages with is_fallback=true (deleted_at IS NULL) in the window — the honest service-quality signal for answer availability"}
|
|
39
|
+
avg_answer_latency_ms: {caliber: "AVG(agent_duration_ms) over non-deleted assistant messages in the window — engine-side duration incl. retrieval, excludes client/network time"}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dsh-wrenai
|
|
3
|
+
description: Self-hosted semantic layer — natural-language data questions with metric calibers riding on every result; single-SELECT guard, self-heal SQL, SQLite/PostgreSQL backends, and a zero-config --weknora bridge (ask your WeKnora deployment anything).
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Use this skill for governed "ask the data" over any dataset with a semantic
|
|
7
|
+
model. Calibers are the contract: numbers never travel without their
|
|
8
|
+
definitions. The `dsh-wrenai` CLI must be on PATH (installed with the package;
|
|
9
|
+
requires python3, stdlib only).
|
|
10
|
+
|
|
11
|
+
## Run
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
dsh-wrenai --init <klines.json> # SQLite demo: load data + views
|
|
15
|
+
dsh-wrenai --ask "average close and volatility over the last 30 days?"
|
|
16
|
+
dsh-wrenai --ask "<问题>" --lang zh # Chinese fallback
|
|
17
|
+
# WeKnora ops: zero-config bridge to a WeKnora deployment's own PostgreSQL
|
|
18
|
+
dsh-wrenai --weknora --ask "how many times was the test knowledge base queried in the past month?"
|
|
19
|
+
# any PostgreSQL: --pg "postgresql://user:pass@host:5432/db" --model <yaml>
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Semantic models live in the kit's `semantic/` directory (`models.yaml` demo,
|
|
23
|
+
`weknora.yaml` preset); copy and edit for your dataset. LLM key: the engine
|
|
24
|
+
reads `DEEPSEEK_API_KEY` from ~/.bashrc export lines.
|
|
25
|
+
|
|
26
|
+
## Discipline
|
|
27
|
+
|
|
28
|
+
- Always quote the calibers section with results — never numbers alone.
|
|
29
|
+
- Single-SELECT guard is non-negotiable; report self-heal events honestly.
|
|
30
|
+
- Metric definition changes are version bumps, not edits-in-place.
|