sql-harness 0.2.4__tar.gz → 0.3.1__tar.gz

{sql_harness-0.2.4 → sql_harness-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-harness
-Version: 0.2.4
+Version: 0.3.1
 Summary: Single-process SQL + SSH CLI for LLM agents. PostgreSQL/MySQL/SSH via SQLAlchemy + paramiko. Plaintext credentials in one TOML file. Helpers auto-injected into the heredoc namespace.
 Project-URL: Source, https://github.com/zhaoliuxue/much_bigpy/tree/master/lab/sql_harness
 Project-URL: Issues, https://github.com/zhaoliuxue/much_bigpy/issues

{sql_harness-0.2.4 → sql_harness-0.3.1}/SKILL.md

@@ -33,31 +33,44 @@ PY
 Stuck-point mechanics — one file per topic, bare filenames only:
 
 - `interaction-skills/aggregations.md`
-- `interaction-skills/docker-via-ssh.md`
+- `interaction-skills/aggregations-advanced.md`
 - `interaction-skills/encoding-and-charset.md`
-- `interaction-skills/indexes-and-explain.md`
 - `interaction-skills/joins.md`
-- `interaction-skills/json-columns.md`
+- `interaction-skills/joins-advanced.md`
 - `interaction-skills/large-result-sets.md`
 - `interaction-skills/migrations.md`
 - `interaction-skills/pooling.md`
 - `interaction-skills/recursive-ctes.md`
+- `interaction-skills/recursive-ctes-advanced.md`
 - `interaction-skills/save-run-cycle.md`
 - `interaction-skills/schema-introspection.md`
-- `interaction-skills/ssh.md`
 - `interaction-skills/timeouts-and-cancellation.md`
 - `interaction-skills/transactions.md`
 - `interaction-skills/window-functions.md`
+- `interaction-skills/window-functions-advanced.md`
 - `interaction-skills/zone-skill-auto-surface.md`
 
-PG optimization depth — `interaction-skills/postgres/`:
+PG-specific depth — `interaction-skills/postgres/`:
 
 - `interaction-skills/postgres/btree-indexes.md`
+- `interaction-skills/postgres/indexes-and-explain.md`
+- `interaction-skills/postgres/json-columns.md`
+- `interaction-skills/postgres/migrations.md`
 - `interaction-skills/postgres/plan-reading.md`
 - `interaction-skills/postgres/slow-queries-joins.md`
 - `interaction-skills/postgres/specialized-indexes.md`
 - `interaction-skills/postgres/table-optimization.md`
 
+MySQL-specific — `interaction-skills/mysql/`:
+
+- `interaction-skills/mysql/json-columns.md`
+
+SSH-specific — `interaction-skills/ssh/`:
+
+- `interaction-skills/ssh/auth-and-tunnels.md`
+- `interaction-skills/ssh/docker-via-ssh.md`
+- `interaction-skills/ssh/ssh.md`
+
 ## Cross-DSN strategy skills
 
 Read these when working across multiple connections, tuning pool behavior, or operating remote hosts. Reached via `apply_skill("pool")` etc. from any active zone — `zones/meta/skills/` is the passive fallback layer (the legacy global `agent-workspace/skills/` was collapsed into a regular zone so `apply_skill()` has a single shape: active zone or meta).

sql_harness-0.3.1/interaction-skills/mysql/json-columns.md

@@ -0,0 +1,43 @@
+# JSON columns — MySQL (`JSON`)
+
+MySQL 5.7+ has a native `JSON` type. PyMySQL serializes Python `dict`/`list` to `JSON`.
+
+## Insert
+
+SQLAlchemy handles the conversion when passing a dict as a parameter:
+
+```python
+execute("INSERT INTO events (data) VALUES (:d)", {"d": {"event": "click"}})
+```
+
+## Query
+
+Use `JSON_EXTRACT(col, '$.path')` or the `->` / `->>` shorthand (MySQL 8.0+):
+
+```python
+rows = query("SELECT data->>'$.event' AS event FROM events WHERE JSON_EXTRACT(data, '$.event') = :e",
+             {"e": "click"})
+```
+
+Note `->>` in MySQL returns `TEXT` (unquoted). In PostgreSQL `->>` returns `text` of a jsonb path. The `'$.event'` path syntax is MySQL-specific.
+
+## Indexing
+
+MySQL 8.0+ supports functional indexes on JSON expressions:
+
+```sql
+CREATE INDEX idx_events_event ON events ((CAST(data->>'$.event' AS CHAR(64))));
+```
+
+Pre-8.0: index a generated column instead.
+
+## Gotchas
+
+- `JSON_EXTRACT` returns JSON; `->>` returns the unquoted scalar. For comparisons, match the operator.
+- `->>` shorthand requires MySQL 8.0+. On 5.7 use `JSON_UNQUOTE(JSON_EXTRACT(...))`.
+- MySQL `JSON` does NOT auto-compress (unlike PG `jsonb` TOAST). Large docs blow up row size.
+- Whitespace handling: MySQL normalizes on input but may reformat on output (PG normalizes the same way).
+
+## See also
+
+- `interaction-skills/postgres/json-columns.md` — PostgreSQL `jsonb` syntax (`@>`, `?`, no `$.path`)

sql_harness-0.3.1/interaction-skills/postgres/json-columns.md

@@ -0,0 +1,46 @@
+# JSON columns — PostgreSQL (`jsonb`)
+
+Use PostgreSQL's binary `jsonb` type (not `json` — `jsonb` is binary-comparable and indexable).
+
+## Insert
+
+`psycopg3` auto-serializes Python `dict`/`list` to `jsonb`:
+
+```python
+execute("INSERT INTO events (data) VALUES (:d)", {"d": {"event": "click", "ts": 1234}})
+```
+
+## Query
+
+Use `->` (returns `jsonb`), `->>` (returns `text`), and the containment operator `@>`:
+
+```python
+rows = query("SELECT data->>'event' AS event FROM events WHERE data @> :filter",
+             {"filter": {"event": "click"}})
+```
+
+## Indexing
+
+`jsonb` supports GIN indexes on the whole document (fast for `@>` / `?` / `?&` / `?|`):
+
+```sql
+CREATE INDEX idx_events_data ON events USING GIN (data);
+```
+
+For paths you hit often, a btree expression index:
+
+```sql
+CREATE INDEX idx_events_event ON events ((data->>'event'));
+```
+
+## Gotchas
+
+- Prefer `jsonb` over `json` — `json` is text-stored, no index support beyond expression indexes, whitespace-preserved.
+- Comparing `data = '{"event":"click"}'::jsonb` works (SQLAlchemy normalizes JSON whitespace).
+- `JSONB` aggregates (`jsonb_agg`, `jsonb_object_agg`) exist; for plain text JSON use `string_agg`.
+- GIN on `jsonb` is large — measure before adding to high-write tables.
+
+## See also
+
+- `interaction-skills/postgres/specialized-indexes.md` — GIN/GiST/BRIN tradeoffs
+- `interaction-skills/mysql/json-columns.md` — MySQL JSON syntax (different operators)

sql_harness-0.3.1/interaction-skills/postgres/migrations.md

@@ -0,0 +1,105 @@
+# PG Data Movement
+
+Move data in/out/around PostgreSQL: logical dump + restore, bulk COPY, and
+logical replication. (Schema migrations -- ALTER / CREATE under transaction
+-- live in `migrations.md`; this file is for moving data across databases or
+in/out of tables.)
+
+## Detection
+
+- **Move a whole database** to a new server -> `pg_dump` + `pg_restore`.
+- **Bulk-load** a big table (>1M rows) -> `COPY` (10-100x faster than `INSERT`).
+- **Live replication** of changes from one PG to another (zero-downtime
+  migration, read replica, aggregation) -> logical replication (publication
+  + subscription).
+- **One-shot export** for analysis / downstream pipeline -> `COPY` /
+  `psql \copy`.
+
+## Approach
+
+### 1. Logical dump + restore (server-side CLI)
+
+`pg_dump` and `pg_restore` live on the PG host, not in SQL. Run them via
+the SSH workspace that hosts your PG:
+
+```python
+use_workspace("2026aliyun")   # SSH workspace on the PG host
+
+ssh_exec("pg_dump -Fc -d sourcedb -f /tmp/sourcedb.dump")     # custom format, compressed
+ssh_exec("createdb targetdb")                                  # empty target
+ssh_exec("pg_restore -d targetdb --no-owner --jobs=4 /tmp/sourcedb.dump")
+```
+
+Flags worth knowing:
+
+- `-Fc` custom format (compressed, parallel-restoreable). `-Fp` plain SQL.
+- `-s` schema-only / `-a` data-only. `-t public.orders` one table.
+- `--no-owner` strips `OWNED BY` from objects (safer cross-cluster).
+
+Cross-version note: `pg_dump` from the **target's** major is safest.
+`pg_dump` 16 can read PG 15 / 14; restore with the target's own
+`pg_restore`. Going major-down (PG 16 dump -> PG 14) is **not supported**.
+
+### 2. Bulk COPY
+
+`COPY` bypasses per-row parse/plan/execute -- it's the right tool for
+anything >10k rows. `sql-harness`'s `execute` covers server-side file
+reads; client-streaming needs psql's `\copy` via SSH.
+
+```python
+use_workspace("2026aliyun")
+
+# (a) COPY TO -- export a query to a server-side CSV. psql \copy wraps
+#     COPY and lets a client feed/collect the stream.
+ssh_exec(r"""psql -U $PG_USER sourcedb -c "\copy (SELECT id,email FROM users WHERE created_at>'2025-01-01') TO '/tmp/users.csv' WITH CSV HEADER" """)
+ssh_download("/tmp/users.csv", "./users.csv")     # pull it local
+
+# (b) COPY FROM file -- server reads directly. Needs superuser OR
+#     `GRANT pg_read_server_files TO <role>` (PG 14+).
+execute("COPY users (id, email) FROM '/var/lib/pg/imports/users.csv' WITH CSV HEADER")
+
+# (c) COPY FROM STDIN -- client streams. `execute` can't do streaming;
+#     psql \copy from a *local* file is the practical path:
+ssh_exec(r"""psql -h $PG_HOST -U $PG_USER targetdb -c "\copy users (id,email) FROM '/local/users.csv' WITH CSV HEADER" """)
+```
+
+### 3. Logical replication (publication + subscription)
+
+For live, ongoing PG-to-PG replication of row changes. SQL-only on both
+sides; the only server config is `wal_level = logical` on the source.
+
+```python
+use_workspace("source_pg")
+# Source: declare what to publish. Default = all DML on the named tables.
+execute("CREATE PUBLICATION pub_users FOR TABLE users")
+# Add `WITH (publish = 'insert,update,delete,truncate')` to include TRUNCATEs.
+
+use_workspace("target_pg")
+# Target: subscribe. Initial copy runs, then changes stream.
+execute("""CREATE SUBSCRIPTION sub_users
+           CONNECTION 'host=src.db port=5432 user=repl password=... dbname=source'
+           PUBLICATION pub_users""")
+
+# Monitor catch-up from the TARGET:
+for r in query("SELECT subname, status, received_lsn, latest_end_lsn FROM pg_stat_subscription"):
+    print(r)      # status='streaming' + latest_end_lsn advancing = caught up
+```
+
+## Gotchas
+
+- **Logical replication does NOT replicate**: schema (DDL), sequence state
+  (manually sync with `pg_dump --section=pre-data` on the source +
+  `pg_restore --section=post-data` on target), or `TRUNCATE` unless the
+  publication includes `publish='truncate'`.
+- **Tables need a PK** (or explicit `REPLICA IDENTITY`) -- without it,
+  `UPDATE`/`DELETE` can't replicate.
+- **`COPY FROM` file path** is server-side (`SHOW data_directory`).
+  Permissions: superuser OR `pg_read_server_files` (PG 14+).
+- **`pg_dump` cross-major** only goes *up* (or equal). PG 14 dump -> PG 16
+  is fine; PG 16 dump -> PG 14 is rejected.
+- **`pg_restore --jobs=N`** uses N cores but each job = one transaction
+  per table -- mid-restore failures leave the DB half-loaded. For
+  restartability, restore one table at a time.
+- **Logical replication initial copy** puts write load on the source; for
+  very large DBs, stop writes briefly or use `ALTER SUBSCRIPTION ...
+  DISABLE` then re-enable after catching up.

sql_harness-0.3.1/interaction-skills/ssh/auth-and-tunnels.md

@@ -0,0 +1,145 @@
+# SSH — auth 选型与端口转发
+
+`ssh.md` 写了怎么用 helpers（`ssh_exec / upload / download`）和 CLI。本篇覆盖两块它没写的：(1) 选哪种 auth（key / password / ssh-agent / `ssh+key` / env fallback），(2) 用 SSH workspace 做端口转发把"躲在堡垒机后面的 DB"暴露成本地端口。
+
+> 全文代码块已注入的 helper：`ssh_exec / ssh_upload / ssh_download / use_workspace / ssh_info`。
+
+## Detection
+
+- `ssh_exec` 弹 `RuntimeError("...no tables...")` 或 paramiko `Auth failed` / `No valid authenticators` → 跳 [§1](#1-auth-选型)
+- PG/MySQL workspace 直连 `internal-db-1.corp:5432` 超时，但 SSH workspace 能登堡垒机 → 跳 [§2](#2-端口转发tunneling)
+- 远程主机有多个 SSH key 想用 ssh-agent 托管 → §1.3
+
+---
+
+## 1. Auth 选型
+
+### 1.1 三种 scheme 对照
+
+`drivers/ssh.py:_parse_url` 支持的三种 URL 形式：
+
+| URL 形式 | 何时用 | 取 key 顺序 |
+|---|---|---|
+| `ssh://user@host:port` | 默认 key auth | `?key=...` → `$BH_SSH_KEY` → `~/.ssh/id_ed25519` → `id_rsa` → `id_ecdsa`（`KEY_FALLBACK`） |
+| `ssh+key://user@host:port?key=/abs/path` | key 不在默认路径 | URL 里的 `?key=` 单独覆盖 |
+| `ssh+password://user:pw@host:port` | 跳板 / 临时主机，没配 key | URL 嵌入密码；也可用 `$BH_SSH_PASSWORD` |
+
+**注意**：`ssh+password` 要求密码嵌入 URL。不想明文落 TOML，把密码放 `$BH_SSH_PASSWORD` 并用 `ssh://user@host`，但驱动**不会**自动回退到 env 密码（仅 key 走 env fallback）。这是常见混淆点。
+
+### 1.2 用哪个连接串
+
+```
+有 ed25519 key 在 ~/.ssh/ 且 authorized_keys 已加？
+  ├─ 是 → ssh://user@host                                (默认 key path)
+  └─ 否：
+      有密码但没 key？ → ssh+password://user:pw@host
+      key 在非默认路径？ → ssh://user@host?key=/path/to/k
+      key 在 ssh-agent？ → §1.3
+```
+
+### 1.3 ssh-agent
+
+paramiko 默认不会主动连 ssh-agent。两条路：
+
+```python
+# 路径 A（最常用）：导出私钥文件路径，给驱动
+# connections.toml:
+#   url = "ssh://app@bastion.corp?key=/home/me/.ssh/bastion_ed25519"
+# 这是 ssh.md 默认推荐的。
+
+# 路径 B（agent 转发）：用 ssh-agent 把 key 放进 socket，paramiko 通过
+# SSHAgentKey 适配。当前 ssh-harness 没适配；如需要 agent 转发，得
+# 自己 patch driver。阶段性建议 → 用路径 A。
+```
+
+### 1.4 key 不被接受（`Auth failed` / `No valid authenticators`）
+
+8 成是这几个原因，按频率排：
+
+1. **key 文件权限太开放**（paramiko 严格遵循 OpenSSH 规则）
+   ```
+   chmod 600 ~/.ssh/id_ed25519
+   ```
+2. **authorized_keys 里这 key 被加过 passphrase**——paramiko 不弹 UI。
+   解决：`ssh-keygen -p -f ~/.ssh/id_ed25519` 清掉 passphrase，或 `ssh-add ~/.ssh/id_ed25519`。
+3. **`?key=` 路径拼错**：相对路径相对 pwd——绝对路径最稳。
+4. **用户名错了**：`ssh://root@host` vs `ssh://ubuntu@host`，老 OpenSSH 默认禁 root。
+
+排查：
+
+```python
+use_workspace("bastion")
+print(ssh_info())          # 看 user/host/port/key_path 解析对不对
+ssh_exec("whoami")         # 验证通了
+```
+
+### 1.5 env fallback 默认 key 优先级
+
+看 `drivers/ssh.py:KEY_FALLBACK = ("~/.ssh/id_ed25519", "~/.ssh/id_rsa", "~/.ssh/id_ecdsa")`：
+
+```python
+r = ssh_exec("ls -l ~/.ssh/id_ed25519 ~/.ssh/id_rsa 2>&1")
+print(r["stdout"])
+# 第一个存在的就是 driver 选的
+```
+
+---
+
+## 2. 端口转发（Tunneling）
+
+业务 DB 不对公网开、只对堡垒机的内网暴露时。三条路：
+
+### 2.1 远程 `ssh -L`（堡垒机侧启端口）
+
+```python
+use_workspace("bastion")
+ssh_exec(
+    "ssh -fN -L 0.0.0.0:15432:internal-db.corp:5432 jump@internal-db.corp",
+    timeout=5,
+)
+r = ssh_exec("ss -lntp | grep 15432", timeout=5)
+assert "15432" in r["stdout"]
+```
+
+> 局限：转发启在堡垒机上，**你本机** psql 连不到堡垒机的 `15432`（除非堡垒机做了 reverse tunnel）。适合堡垒机里跑 docker-compose；不适合"我本地 psql 直连"。
+
+### 2.2 paramiko `open_channel` 反向隧道
+
+让堡垒机把 internal-db 的 5432 反向暴露到堡垒机的 15432，然后用堡垒机侧 `psql` / `mysql` 客户端工具调试：
+
+```python
+use_workspace("bastion")
+import paramiko
+ws = current_workspace()
+transport = ws.engine.client.get_transport()
+reverse = transport.request_port_forward("0.0.0.0", 15432,
+    ("internal-db.corp", 5432))
+try:
+    r = ssh_exec("PGPASSWORD=... psql -h 127.0.0.1 -p 15432 -U app -c 'select 1'")
+finally:
+    transport.cancel_port_forward("0.0.0.0", 15432)
+```
+
+### 2.3 真要让你本机 SQL 客户端连
+
+sql-harness 不内置 SSH SOCKS 代理；本机 `psql`/`TablePlus` 直连躲在堡垒机后的 DB，最干净是用 OpenSSH 客户端 `ssh -L 15432:internal-db.corp:5432 bastion`（sql-harness 之外的工具）。
+
+如果坚持在 sql-harness 里搞，用 §2.2 + 堡垒机启端口 + 跑 `psql`。
+
+---
+
+## Gotchas
+
+- **paramiko 不会自动用 ssh-agent**——见 §1.3。
+- **`ssh+password://` 必须嵌入密码**——`BH_SSH_PASSWORD` env 不会回退（仅 key 走 env fallback）。
+- **堡垒机转发时 `ssh -fN` 在某些堡垒机受限**——`AllowTcpForwarding no`。失败就 §2.2 走 paramiko `request_port_forward`。
+- **`request_port_forward` 返回的对象必须在 finally 里 cancel**，否则下次 `use_workspace` 复用 transport 报端口占用。
+- **workspace disconnect 不自动清 tunnel**——硬规矩：tunnel 用 try/finally 包起来。
+- **Windows → Linux 堡垒机的 key 权限**：`scp` 把 600 丢没的话 OpenSSH 服务端会拒。`scp` 后 `chmod 600` 或用 `ssh-copy-id`（用 sftp，保留权限）。
+- **堡垒机禁用 SFTP subsystem**：`ssh_upload/ssh_download` 报 `RuntimeError("SFTP subsystem not available")`。回退 `tar -czf - | ssh host tar -xzf -` 经 `ssh_exec` 流式传。
+
+## See also
+
+- `interaction-skills/ssh/ssh.md` — mechanic（helpers / CLI / 基础 gotchas）
+- `interaction-skills/ssh/docker-via-ssh.md` — 通过 SSH workspace 把 docker-compose 部署到远程
+- `docs/connections-toml-example.toml` — `ssh://` / `ssh+password://` / `ssh+key://` 三种真实示例

{sql_harness-0.2.4/interaction-skills → sql_harness-0.3.1/interaction-skills/ssh}/docker-via-ssh.md

@@ -1,9 +1,10 @@
 # Docker-via-SSH — deploy a docker-compose service to a REMOTE host
 
-> **The root logic for all docker operations in sql-harness.** Templates live
-> locally (shipped with the skill); `sql-harness ssh` uploads them to the remote
-> host and runs `docker compose`. Everything runs **on the remote**, not locally.
-> Mirrors how you'd deploy by hand over SSH, but scripted + repeatable.
+> **The root logic for all docker operations in sql-harness.** You bring the
+> `docker-compose.yml` (and config files) locally; `sql-harness ssh` uploads
+> them to the remote host and runs `docker compose`. Everything runs **on the
+> remote**, not locally. Mirrors how you'd deploy by hand over SSH, but
+> scripted + repeatable.
 
 ## Detection
 
@@ -34,11 +35,12 @@ self-contained (compose + config + data + logs all relative to that dir):
 
 ## The 4-step deploy loop
 
-### Step 1 — write the compose locally (template shipped with the skill)
+### Step 1 — write the compose locally
 
-Templates live in `lab/sql_harness/docker-services/<service>/` (local repo).
-Each ships a `docker-compose.yml` + config files, all mounts relative to the
-service dir (`./data`, `./config` — NOT `./<service>/data`, which double-nests).
+This mechanic assumes you have a local `docker-compose.yml` + any config files
+you want to ship. The shipped template dir is gone in this build — bring your
+own. All mounts must be **relative** to the service dir (`./data`, `./config`
+— NOT `./<service>/data`, which double-nests).
 
 ### Step 2 — upload to the remote
 
@@ -122,15 +124,39 @@ the save→run cycle applied to infrastructure.
   yaml parse errors on the remote. Always strip CRLF before upload
   (`sed -i 's/\r$//' file.yml` or set `git config core.autocrlf false`).
 
-## Service-specific gotchas
+## Service-specific gotchas — coturn
 
-coturn has its own war stories (NAT `external-ip`, 49152-65535 UDP port-range
-iptables OOM / exit-137, `network_mode: host`, config-as-dir vs config-as-file).
-Those live in `docker-services/coturn/README.md` — read before deploying coturn.
+coturn (TURN/STUN relay) has war stories that bit production. All live here in the mechanic — not duplicated in the template dir.
+
+**Key config knobs** (`turnserver.conf`):
+- `realm` — your domain / public hostname (clients see this in their URL).
+- `external-ip` — server's PUBLIC ip when behind NAT (Aliyun ECS is). Set as `AUTO` in shipped template; deploy script substitutes real public ip before `up -d`.
+- `listening-ip=0.0.0.0` — bind interface (private is fine; only `external-ip` needs to be public).
+- `listening-port=3478` (TURN/STUN), `tls-listening-port=5349` (TURNS over TLS).
+- `min-port=49152` / `max-port=65535` — relay UDP range. Must match firewall.
+- `user=<u>:<p>` + `lt-cred-mech` — long-term credential mechanism.
+
+Test creds after deploy:
+```bash
+turnutils_uclient -u webrtc -w webrtc <server-ip> 3478
+```
+
+**Gotchas**:
+
+- **`external-ip` behind NAT**: Aliyun ECS is behind NAT — coturn's `turnserver.conf` `external-ip=` must be the **public** ip, not the private one `listening-ip=0.0.0.0` binds. Templated as `AUTO`; substitute at deploy time (`sed -i "s/external-ip=AUTO/external-ip=$PUBIP/"`).
+
+- **Firewall / security group**: docker publishes ports on the host, but Aliyun security group + host firewall (ufw/firewalld) must ALSO allow them. The coturn relay UDP range (49152-65535) is the easy one to forget. Open in security group, NOT via docker `ports:`.
+
+- **Don't publish the relay port range via `ports:`** — publishing coturn's relay range `49152-65535` (~16k ports) makes Docker create ~16k iptables DNAT rules, which **OOM-kills the container start** (exit 137) on small hosts (Aliyun ECS). Use `network_mode: host` instead — coturn binds the range natively, zero iptables overhead. This is the official coturn/docker recommendation.
+
+- **`network_mode: host` disables `ports:`** — the container binds host ports directly; don't also list `ports:` (compose rejects the combo). All port management moves to the host firewall + cloud security group.
+
+- **Config-as-dir gotcha**: if `./turnserver.conf` doesn't exist locally when docker tries to mount it, Docker creates a **directory** at that path (not a file) → coturn fails to read config. Always upload the file BEFORE `up -d`, or pre-`touch` it on the remote.
+
+**Why host networking (rationale)**: coturn's relay UDP range (49152-65535 = ~16k ports) must NOT be published via docker `-p` — Docker would create ~16k iptables DNAT rules, OOM-killing the daemon on small hosts. Host networking lets coturn bind the range natively, zero iptables overhead. Ports to open in host firewall / cloud security group: `3478/tcp+udp`, `5349/tcp+udp`, `49152-65535/udp` (relay range — open in security group, NOT in docker).
 
 ## See also
 
 - This file's "Detection" / "Approach" sections (decision P: the legacy strategy stub in `agent-workspace/zones/meta/skills/docker-deploy.md` was collapsed into the mechanic; no separate doc).
-- `interaction-skills/ssh.md` — the raw ssh_exec/upload/download helpers
-- `docker-services/<svc>/` — shipped templates (coturn, mysql, redis, ...)
-- `docker-services/<svc>/README.md` — service-specific gotchas (e.g. coturn)
+- `interaction-skills/ssh/ssh.md` — the raw ssh_exec/upload/download helpers
+- `interaction-skills/ssh/auth-and-tunnels.md` — SSH auth scheme 选型 + 端口转发

{sql_harness-0.2.4 → sql_harness-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sql-harness"
-version = "0.2.4"
+version = "0.3.1"
 description = "Single-process SQL + SSH CLI for LLM agents. PostgreSQL/MySQL/SSH via SQLAlchemy + paramiko. Plaintext credentials in one TOML file. Helpers auto-injected into the heredoc namespace."
 readme = "README.md"
 license = { text = "MIT" }
@@ -42,7 +42,6 @@ build-backend = "hatchling.build"
 # Mirrors browser-harness's setuptools src-layout. The package source is
 # under `src/sql_harness/`; the package name (top-level import) is `sql_harness`.
 packages = ["src/sql_harness"]
-artifacts = ["src/sql_harness/SKILL.md"]
 
 [tool.hatch.build.targets.wheel.force-include]
 # Bundle the full skill docs into the wheel so `sql-harness skill install`
@@ -53,9 +52,6 @@ artifacts = ["src/sql_harness/SKILL.md"]
 "SKILL.md" = "sql_harness/_skills/SKILL.md"
 "interaction-skills" = "sql_harness/_skills/interaction-skills"
 "agent-workspace/zones/meta/skills" = "sql_harness/_skills/zones-meta-skills"
-# Docker service templates (coturn/mysql/redis/...) — the docker-via-ssh mechanic
-# uploads these to ~/sql-harness/<svc>/ on a remote host.
-"docker-services" = "sql_harness/_skills/docker-services"
 
 [tool.pytest.ini_options]
 # Mirrors browser-harness's `pythonpath = ["src"]` — tests import `sql_harness`

{sql_harness-0.2.4 → sql_harness-0.3.1}/src/sql_harness/cli.py

@@ -343,17 +343,17 @@ def _packaged_skill_body() -> str | None:
 
     Mirrors browser-harness's run.py:_print_skill, which reads the packaged
     SKILL.md via importlib.resources so it works from both a source checkout
-    and a built wheel. Falls back to the repo-level skills/sql-harness/SKILL.md.
+    and a built wheel. Falls back to the repo-level SKILL.md (root).
     """
-    # 1. in-package: src/sql_harness/SKILL.md (force-included in the wheel)
+    # 1. in-package: sql_harness/_skills/SKILL.md (force-included in the wheel)
     try:
-        ref = importlib_resources.files("sql_harness").joinpath("SKILL.md")
+        ref = importlib_resources.files("sql_harness").joinpath("_skills", "SKILL.md")
         if ref.is_file():
             return ref.read_text(encoding="utf-8")
     except (ModuleNotFoundError, FileNotFoundError):
         pass
-    # 2. repo-level fallback (source checkout)
-    repo = _package_root_dir() / "skills" / "sql-harness" / "SKILL.md"
+    # 2. repo-level fallback (source checkout): lab/sql_harness/SKILL.md
+    repo = _package_root_dir() / "SKILL.md"
     if repo.is_file():
         return repo.read_text(encoding="utf-8")
     return None

{sql_harness-0.2.4 → sql_harness-0.3.1}/uv.lock

@@ -446,7 +446,7 @@ wheels = [
 
 [[package]]
 name = "sql-harness"
-version = "0.2.3"
+version = "0.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "paramiko" },

sql_harness-0.2.4/docker-services/coturn/README.md

@@ -1,69 +0,0 @@
-# coturn (TURN/STUN server) — docker-services template
-
-Self-contained TURN/STUN relay for WebRTC clients. Deploy via
-`../../interaction-skills/docker-via-ssh.md`. Service dir on the remote:
-`~/sql-harness/coturn/`.
-
-## Key config knobs (`turnserver.conf`)
-
-- `realm` — your domain / public hostname (clients see this in their URL).
-- `external-ip` — the server's PUBLIC ip when behind NAT (Aliyun ECS is).
-  Set as `AUTO` in the shipped template; the deploy script substitutes the
-  real public ip before `up -d`.
-- `listening-ip=0.0.0.0` — bind interface (private is fine; only `external-ip`
-  needs to be public).
-- `listening-port=3478` (TURN/STUN), `tls-listening-port=5349` (TURNS over TLS).
-- `min-port=49152` / `max-port=65535` — relay UDP range. Must match the firewall.
-- `user=<u>:<p>` + `lt-cred-mech` — long-term credential mechanism.
-
-Test creds after deploy:
-```bash
-turnutils_uclient -u webrtc -w webrtc <server-ip> 3478
-```
-
-## Service-specific gotchas (from production)
-
-- **`external-ip` behind NAT**: Aliyun ECS is behind NAT — coturn's
-  `turnserver.conf` `external-ip=` must be the **public** ip, not the private
-  one `listening-ip=0.0.0.0` binds. Templated as `AUTO`; substitute at deploy
-  time (`sed -i "s/external-ip=AUTO/external-ip=$PUBIP/"`).
-
-- **Firewall / security group**: docker-publishes ports on the host, but Aliyun
-  security group + host firewall (ufw/firewalld) must ALSO allow them. The
-  coturn relay UDP range (49152-65535) is the easy one to forget. Open in
-  security group, NOT via docker `ports:`.
-
-- **Don't publish the relay port range via `ports:`** — publishing coturn's
-  relay range `49152-65535` (16k ports) makes Docker create ~16k iptables
-  DNAT rules, which **OOM-kills the container start** (exit 137) on small
-  hosts (Aliyun ECS). Use `network_mode: host` instead — coturn binds the
-  range natively, zero iptables overhead. This is the official
-  coturn/docker recommendation, and the shipped `docker-compose.yml` already
-  does this.
-
-- **`network_mode: host` disables `ports:`** — the container binds host ports
-  directly; don't also list `ports:` (compose rejects the combo). All port
-  management moves to the host firewall + cloud security group.
-
-- **Config-as-dir gotcha**: if `./turnserver.conf` doesn't exist locally when
-  docker tries to mount it, Docker creates a **directory** at that path
-  (not a file) → coturn fails to read config. Always upload the file BEFORE
-  `up -d`, or pre-`touch` it on the remote.
-
-## Why host networking (rationale)
-
-coturn's relay UDP range (49152-65535 = ~16k ports) must NOT be published via
-docker `-p` — Docker would create ~16k iptables DNAT rules, OOM-killing the
-daemon on small hosts (Aliyun ECS etc.). Host networking lets coturn bind
-the range natively, zero iptables overhead. This is the official
-coturn/docker recommendation.
-
-Ports to open in the host firewall / cloud security group:
-- 3478/tcp + 3478/udp   (TURN/STUN)
-- 5349/tcp + 5349/udp   (TURNS over TLS)
-- 49152-65535/udp       (relay range — open in security group, NOT in docker)
-
-## Files in this template
-
-- `docker-compose.yml` — host-networked coturn container.
-- `turnserver.conf` — minimal config; edit `realm`/`external-ip`/`user` before deploy.

sql_harness-0.2.4/docker-services/coturn/docker-compose.yml

@@ -1,29 +0,0 @@
-# coturn via docker-compose — HOST NETWORKING (production-correct)
-#
-# Why host networking: coturn's relay UDP range (49152-65535 = ~16k ports)
-# must NOT be published via docker `-p` — Docker would create ~16k iptables
-# DNAT rules, OOM-killing the daemon on small hosts (Aliyun ECS etc.).
-# Host networking lets coturn bind the range natively, zero iptables overhead.
-# This is the official coturn/docker recommendation.
-#
-# Ports to open in the host firewall / cloud security group:
-#   3478/tcp + 3478/udp   (TURN/STUN)
-#   5349/tcp + 5349/udp   (TURNS over TLS)
-#   49152-65535/udp       (relay range — open in security group, NOT in docker)
-
-services:
-  coturn:
-    image: coturn/coturn:latest
-    container_name: coturn
-    restart: unless-stopped
-    network_mode: host           # ← binds directly to host ports; no `ports:`
-    volumes:
-      # Relative to THIS dir (~/sql-harness/coturn/) — data + config cohesive.
-      - ./turnserver.conf:/etc/turnserver.conf:ro
-      - ./data:/var/lib/coturn
-    command: ["-c", "/etc/turnserver.conf"]
-    logging:
-      driver: json-file
-      options:
-        max-size: "10m"
-        max-file: "3"

sql_harness-0.2.4/docker-services/coturn/turnserver.conf

@@ -1,36 +0,0 @@
-# coturn (TURN/STUN server) — minimal config
-# Deploy: see ../../interaction-skills/docker-via-ssh.md
-#
-# EDIT before deploy:
-#   - realm: your domain or public hostname
-#   - external-ip: the server's PUBLIC ip (needed when behind NAT)
-#   - user: <user>:<pass> for TURN credentials (used by WebRTC clients)
-#
-# Test credentials after deploy:
-#   turnutils_uclient -u webrtc -w webrtc <server-ip> 3478
-
-realm=turn.zhlx.dev
-listening-ip=0.0.0.0
-# external-ip must be the PUBLIC ip when the host is behind NAT (Aliyun ECS is).
-# Replace with the real public ip at deploy time (the deploy script does this).
-external-ip=AUTO
-listening-port=3478
-tls-listening-port=5349
-
-# Credentials (long-term mechanism)
-user=webrtc:webrtc
-lt-cred-mech
-
-# Relay port range — must match the UDP ports opened in docker-compose + firewall
-min-port=49152
-max-port=65535
-
-# Fingerprint + no-loopback-prevention niceties
-fingerprint
-no-cli
-no-tls
-no-dtls
-
-# Logging
-log-file=stdout
-simple-log

sql_harness-0.2.4/docker-services/mysql/docker-compose.yml

@@ -1,35 +0,0 @@
-# MySQL 8.0 — self-contained service template
-# Deploy: see ../../interaction-skills/docker-via-ssh.md
-# All mounts are relative to THIS dir (~/sql-harness/mysql/) — cohesive.
-# (Adapted from the upstream `dr` repo with the double-nesting bug fixed:
-#  mounts are ./data not ./mysql/data.)
-
-services:
-  mysql:
-    image: mysql:8.0
-    container_name: mysql
-    restart: unless-stopped
-    ports:
-      - "3306:3306"
-    environment:
-      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-223456}
-      MYSQL_DATABASE: ${MYSQL_DATABASE:-app}
-      MYSQL_USER: ${MYSQL_USER:-app}
-      MYSQL_PASSWORD: ${MYSQL_PASSWORD:-223456}
-      TZ: Asia/Shanghai
-    volumes:
-      - ./data:/var/lib/mysql
-      - ./conf:/etc/mysql/conf.d
-      - ./init:/docker-entrypoint-initdb.d
-      - ./logs:/var/log/mysql
-    command:
-      - --default-authentication-plugin=mysql_native_password
-      - --character-set-server=utf8mb4
-      - --collation-server=utf8mb4_unicode_ci
-      - --explicit_defaults_for_timestamp=1
-      - --lower_case_table_names=1
-    healthcheck:
-      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-p${MYSQL_ROOT_PASSWORD:-223456}"]
-      timeout: 10s
-      retries: 3
-      start_period: 30s

sql_harness-0.2.4/docker-services/redis/docker-compose.yml

@@ -1,24 +0,0 @@
-# Redis 7 — self-contained service template
-# Deploy: see ../../interaction-skills/docker-via-ssh.md
-# Mounts relative to THIS dir (~/sql-harness/redis/) — cohesive.
-# (Adapted from `dr` with double-nesting fixed: ./data not ./redis/data.)
-
-services:
-  redis:
-    image: redis:7-alpine
-    container_name: redis
-    restart: unless-stopped
-    ports:
-      - "6379:6379"
-    volumes:
-      - ./data:/data
-      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
-    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
-    sysctls:
-      - net.core.somaxconn=511
-    healthcheck:
-      test: ["CMD", "redis-cli", "ping"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 10s

sql_harness-0.2.4/interaction-skills/json-columns.md

@@ -1,30 +0,0 @@
-# JSON columns
-
-## PostgreSQL (`jsonb`)
-
-psycopg3 auto-serializes Python `dict`/`list` to `jsonb`. Query with `->`, `->>`, `@>`:
-
-```python
-execute("INSERT INTO events (data) VALUES (:d)", {"d": {"event": "click", "ts": 1234}})
-rows = query("SELECT data->>'event' AS event FROM events WHERE data @> :filter",
-             {"filter": {"event": "click"}})
-```
-
-## MySQL (`JSON`)
-
-PyMySQL auto-serializes Python `dict`/`list` to `JSON`. Query with `->`, `->>`, `JSON_EXTRACT`:
-
-```python
-execute("INSERT INTO events (data) VALUES (%s)", {"event": "click"})
-# Wait — execute uses :name style. SQLAlchemy handles conversion:
-rows = query("SELECT data->>'$.event' AS event FROM events WHERE JSON_EXTRACT(data, '$.event') = :e",
-             {"e": "click"})
-```
-
-## Gotchas
-
-- PostgreSQL `jsonb` is binary-comparable and indexable; `json` (text) is not. Prefer `jsonb`.
-- On PostgreSQL, comparing `data = '{"event": "click"}'::jsonb` works; comparing `data = '{"event":"click"}'` (without spaces) is the canonical form. SQLAlchemy normalizes.
-- MySQL `JSON` stores in a binary internal format; output may differ from input whitespace.
-- For nested-path queries on MySQL 5.7+, use `JSON_EXTRACT`/`->>`. The `->>` shorthand requires MySQL 8.0+.
-- Large JSON docs: store compressed (PG `TOAST` handles it; MySQL `JSON` does not auto-compress).

sql_harness-0.2.4/practice/pgexercises/README.md

@@ -1,63 +0,0 @@
-# PG Exercises Practice
-
-71 real [pgexercises](https://pgexercises.com/) problems organized as one `.sql` file per problem, with `cd`-schema fixtures and `run_sql_file` replay.
-
-## File Format
-
-Each problem lives in a single `.sql` file. Conventions:
-
-- **Header line** = the question (human-readable, free text).
-- **Body lines** = the SQL answer.
-- **Comment lines** starting with `--` are ignored at replay time (use them to annotate).
-- File naming: `<category>_<NN>_<slug>.sql` — e.g. `basic_01_retrieve_everything.sql`, `joins_01.sql`. Flat directory, prefixed by category.
-
-The replay splitter (`_split_sql`) respects single-quoted strings, double-quoted identifiers, `--` and `/* */` comments, and PG `$tag$` dollar-quoting.
-
-## Replay
-
-Always use `run_sql_file` to replay an answer. The helper splits the file on top-level `;` and runs every statement in one transaction on the active workspace; the final statement's result set comes back as `list[dict]`.
-
-1. **Load schema + seed once** (PG only — uses `cd` schema with `facilities`, `members`, `bookings`):
-
-   ```bash
-   uv run sql-harness <<'PY'
-   use_workspace("local_pg")
-   run_sql_file("lab/sql_harness/practice/pgexercises/schema.sql")
-   run_sql_file("lab/sql_harness/practice/pgexercises/seed.sql")
-   PY
-   ```
-
-   Expected seed counts: `cd.facilities=9`, `cd.members=15`, `cd.bookings=35`.
-
-2. **Replay an answer** from the heredoc, or from the shell:
-
-   ```bash
-   uv run sql-harness run_sql_file \
-     practice/pgexercises/basic_01_retrieve_everything.sql -c local_pg
-   ```
-
-## Performance Tuning
-
-After a correct answer, wrap the body in `EXPLAIN (ANALYZE, BUFFERS) ...` via `explain_analyze` to get a real plan + actual timings:
-
-```bash
-uv run sql-harness <<'PY'
-use_workspace("local_pg")
-explain_analyze("SELECT ... FROM cd.bookings WHERE ...")
-PY
-```
-
-`explain_analyze` is PG-only. It returns one row per plan line (`{'QUERY PLAN': '<line>'}`). Supports `format='text'|'json'|'xml'|'yaml'`.
-
-For tuning depth, read:
-
-- `interaction-skills/postgres/plan-reading.md` — how to read an `EXPLAIN ANALYZE` plan.
-- `interaction-skills/postgres/btree-indexes.md` — the index to add when a scan shows up.
-- `interaction-skills/postgres/slow-queries-joins.md` — join-order and predicate pushdown.
-- `interaction-skills/postgres/table-optimization.md` — vacuum/analyze, stats targets.
-
-## Adding Problems
-
-To add a new pgexercises problem, drop a new `<category>_<NN>_<slug>.sql` file into `practice/pgexercises/`, following the same header/body convention. Pick the next free `<NN>` in its category prefix (`basic_`, `joins_`, `aggregations`, `modifying`, `date`, `string`, `recursive`, …).
-
-Re-run `schema.sql` + `seed.sql` only when you change fixtures; existing problem files don't need re-loading.

sql_harness-0.2.4/skills/sql-harness/SKILL.md

@@ -1 +0,0 @@
-../../SKILL.md

sql_harness-0.2.4/src/sql_harness/SKILL.md

@@ -1 +0,0 @@
-../../SKILL.md