dsh-plugin-dbgraph 0.0.0-stage → 0.1.0

package/README.md

@@ -1,3 +1,179 @@
-# Temporary Holding Version
+# dsh-plugin-dbgraph
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+**Database structure the agent already knows, so it answers with one tool call instead of five.**
+
+A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) plugin that indexes a remote
+database's *catalog* — every table, column, comment, key, index, and a bounded sample of enum
+literals — into a local SQLite index under `~/.dsh/dbgraph`. The model reads that index (and a digest
+of it placed directly in the system prompt) instead of spending round-trips on `SHOW TABLES` and
+`DESCRIBE`, then fetches the actual values with exactly one network call.
+
+## The problem it removes
+
+Without an index, answering *"what is 张三's password?"* looks like this:
+
+```
+1. SHOW TABLES                      ← network round-trip
+2. DESCRIBE sys_user                ← network round-trip
+3. SELECT ... WHERE status = 1      ← wrong: status is a varchar
+4. SELECT ... WHERE status = 'paid' ← guess corrected
+5. 2,000 rows into the transcript   ← context burned
+```
+
+Four model round-trips, four network round-trips, two wrong guesses. **The expensive part is not the
+database — it is the model's turns.** Each one costs seconds; a 100 ms query does not.
+
+With dbgraph:
+
+```
+1. dbgraph find {entity:'用户', fields:['密码'], where:{name:'张三'}}
+← one row, and the answer says which table and column it used
+```
+
+## Structure local, data online
+
+| | Where | Why |
+|---|---|---|
+| Tables, columns, comments, keys, indexes, enum samples, row estimates | **Local index** | Zero network, usable offline, no credential needed to ask what a column means |
+| Actual row values | **Always the server** | A cached password is not a fast password, it is a wrong one |
+
+Nothing derived from business data is persisted. The one data-shaped field is `enum_values_json`, a
+`SELECT DISTINCT … LIMIT n` sample that teaches the model whether `status` holds `'paid'` or `1`.
+
+## Layout
+
+```
+packages/
+  bundle     cordis.patch.yml — mounts everything as one profile layer
+  service    ctx.dbgraph: the vocabulary, the dialect registry, the index protocol
+  catalog    one DatabaseDialect per engine (SQLite, MySQL; PostgreSQL/MongoDB/Redis to follow)
+  store      ~/.dsh/dbgraph: connections.json, the index writer, the reader, the resolver
+  tool       the two model tools, /dbgraph, and the prompt digest
+crates/
+  dbgraph-indexer   the index write path in Rust (normalize + terms + SQLite)
+```
+
+### Why the write path is Rust
+
+`node:sqlite` is a synchronous API. Writing ten thousand columns through it blocks the event loop for
+as long as the write takes — this is the same wall `dsh-plugin-codegraph` and
+`dsh-plugin-sessiongraph` hit, and both moved their kernels behind napi `AsyncTask` to get past it.
+`crates/dbgraph-indexer` does the same: normalization, term generation, and the SQLite write run on
+libuv's thread pool, so a rebuild never stalls the host.
+
+Measured on one machine, same catalog, same fixed clock — 500 tables / 10,000 columns:
+
+| writer | write | index file | longest event-loop block |
+|---|---|---|---|
+| JavaScript | 1992 ms | 8.46 MiB | **1992 ms** (the whole write) |
+| Rust (`AsyncTask`) | 218 ms | 8.41 MiB | **33.6 ms** |
+
+**9.1× faster, and the blocking cost drops 59×** — which is the number that matters, because a
+two-second stall is two seconds of the host doing nothing else.
+
+The two writers are checked against each other rather than trusted: given the same catalog and the
+same fixed `now`, `objects`, `columns`, `edges`, `terms`, and `search` (rowids included) must match
+row for row, and only `meta.writer` may differ. Fixture expectations are generated by *running* the
+JavaScript writer, never hand-written, so a change on either side that is not mirrored fails the
+build.
+
+The JavaScript writer is therefore not an optimization fallback — it is the **parity reference** and
+the **degradation path** (no Rust toolchain, no addon, a throwing call, an unparsable answer all fall
+back to it, once and quietly).
+
+### Two things that had to be measured, not assumed
+
+- **`node:sqlite` contentless FTS5 stores nothing you can read back.** Every column — `UNINDEXED`
+  ones included — returns `NULL`; only `rowid` and `rank` survive. The index therefore uses a
+  *contentful* table, where `object_id UNINDEXED` reads back verbatim. (The FTS5 column grammar is
+  also `column-name [UNINDEXED]`: a type name in front is a parse error.)
+- **`unicode61` treats a run of Chinese as one token.** Store `广告点击` verbatim and `MATCH '广告'`
+  finds nothing. So `text` holds a pre-tokenized stream — camel/digit boundaries for ASCII, bigrams
+  for CJK — and queries go through the same tokenizer, joined with `OR` (a space means *phrase*, not
+  *any of these*).
+
+### Picking a writer
+
+`dbgraph_index` takes `writer: 'auto' | 'js' | 'rust'`:
+
+- `js` — the JavaScript writer, always. Never touches the addon.
+- `rust` — the native writer only; a missing artifact or a throwing call fails the run rather than
+  quietly producing a slower index.
+- `auto` (default) — native when available; a missing artifact, a rejected call, or an unreadable
+  report logs **once per process** and falls back to JavaScript.
+
+`DSH_DBGRAPH_RUST_ADDON` points at an artifact explicitly: a test hook for the fallback path, and an
+escape hatch for a deployment that keeps the addon outside `crates/`. It defaults to
+`crates/dbgraph-indexer/artifacts/dbgraph-indexer.<platform>-<arch>.node`.
+
+One acknowledged difference: the JavaScript writer counts dangling edges (an edge naming a table the
+snapshot does not contain) into `report.warnings`; the native writer drops them silently. The stored
+data is identical — only that one warning line differs.
+
+## Index format
+
+One SQLite file per connection, replaced wholesale on every run (write to a staging file named
+`<index>.<pid>-<nonce>.tmp`, then rename). The nonce is per run, so two runs never share the staging
+file. That single decision removes incremental diffing, deletes, and half-updated indexes.
+
+```
+meta      key/value: engine, database, schemaVersion, l1IndexedAt, l2IndexedAt
+objects   one row per table / view / collection / key pattern
+columns   ordinal, type, type_class, nullability, default, comment, pk/unique, enum sample
+edges     fk | index | view_ref | code_ref
+terms     normalized words + Chinese bigrams + alias expansion, for semantic lookup
+search    FTS5 (contentless) over object and column names and comments
+```
+
+`status.json` beside the indexes carries a digest the system-prompt section reads **synchronously** —
+a section is assembled on every model request, so opening SQLite there would be a per-step cost the
+plugin refuses to pay.
+
+## Two levels
+
+| Level | Content | Where |
+|---|---|---|
+| **L1** | object and column names, declared types | inline, seconds — usable immediately |
+| **L2** | comments, foreign keys, indexes, row estimates, enum samples | background job |
+
+An L1 failure to enrich never discards the structural snapshot: an index that answers "which columns
+exist" is most of the value.
+
+## Tools
+
+| Tool | Timeout | What it does |
+|---|---|---|
+| `dbgraph` | 30 s | `connections` `schema` `objects` `columns` `find` `sql` `impact` `refs` |
+| `dbgraph_index` | 300 s | Builds or refreshes one connection (`l1` / `l2` / `full`); hands a large run to a background job |
+
+`find` is the one that matters: it resolves words to a table and columns **offline**, then runs one
+parameterized `SELECT`. It answers `ambiguous` with ranked candidates rather than guessing, and
+`missing_predicate` rather than scanning a table because the user said "this user" without saying
+which.
+
+## Masking
+
+Sensitive columns (`password`, `passwd`, `pwd`, `secret`, `token`, `salt`, `id_card`, …) come back as
+`[REDACTED]` by default. Without that, one careless question puts a password into a transcript — and
+this harness persists transcripts and indexes them. `sensitiveColumns: 'show'` per connection turns it
+off.
+
+Credentials never live in `connections.json`: a connection stores a **credential reference**, and the
+password is resolved through the host's `ctx.credentials` seam on each operation.
+
+## Commands
+
+```sh
+pnpm install
+pnpm run build          # tsc -b across the four TS packages
+pnpm test               # vitest run
+pnpm run test:coverage  # per-file 100% thresholds
+pnpm run rust:build     # cargo build --release for crates/dbgraph-indexer
+```
+
+Tests resolve to live `src/`, never to built `lib/` (`tsconfig.base.json`'s `paths` map is the
+facade), so a stale build can never load a second copy of a module singleton.
+
+## License
+
+MIT

package/cordis.patch.yml

@@ -0,0 +1,29 @@
+# The dsh-plugin-dbgraph bundle patch: mounts the whole database-structure capability
+# as ONE layer over an existing dsh profile.
+#
+# Row order carries no load semantics — activation is service-availability driven — so the
+# seam may appear before or after the providers that fill it. The user's own profile patch
+# layer can address any row below by `id` to retune it (last write per row wins), e.g.
+# disabling a dialect's `engines` list or lowering the tool's `maxRows`.
+
+- insert:
+    # Service Definition: publishes ctx.dbgraph — the dialect registry, the single index-store
+    # reservation, and the nine normalized operations. Holds no store and reaches no server.
+    - id: dbgraph
+      name: 'dsh-plugin-dbgraph-service'
+
+    # Service Provider (dialects): one DatabaseDialect per engine. Each reaches one server family
+    # for a probe, an L1 snapshot, an L2 enrichment, and one execute. Registers dialects only.
+    - id: dbgraph-catalog
+      name: 'dsh-plugin-dbgraph-catalog'
+
+    # Service Provider (offline): the on-disk index under ~/.dsh/dbgraph — reader, JavaScript
+    # write path (the parity reference and the fallback), and the prompt-digest sidecar.
+    - id: dbgraph-store
+      name: 'dsh-plugin-dbgraph-store'
+
+    # Consumer: the model-facing `dbgraph` and `dbgraph_index` tools, their system-prompt
+    # section (the schema digest that removes the discovery round-trips), and the `/dbgraph`
+    # command.
+    - id: dbgraph-tool
+      name: 'dsh-plugin-dbgraph-tool'

package/package.json

@@ -1,6 +1,51 @@
 {
   "name": "dsh-plugin-dbgraph",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
+  "version": "0.1.0",
+  "description": "Database structure for DeepSeek Harness — indexes a remote database catalog into a local index so the agent knows every table and column before it writes a query, then answers with one tool call",
+  "keywords": [
+    "dsh",
+    "dsh-plugin",
+    "deepseek-harness",
+    "cordis",
+    "cordis-plugin",
+    "database",
+    "sql",
+    "mysql",
+    "postgresql",
+    "sqlite",
+    "mongodb",
+    "redis",
+    "schema",
+    "ai-agent",
+    "llm-tools",
+    "agent-tools"
+  ],
+  "license": "MIT",
+  "author": "CC ZHAO",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/CC19990113/dsh-plugin-dbgraph.git",
+    "directory": "packages/bundle"
+  },
+  "homepage": "https://github.com/CC19990113/dsh-plugin-dbgraph#readme",
+  "bugs": "https://github.com/CC19990113/dsh-plugin-dbgraph/issues",
+  "type": "module",
+  "files": [
+    "cordis.patch.yml"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  },
+  "dsh": {
+    "bundle": {
+      "patch": "./cordis.patch.yml"
+    }
+  },
+  "dependencies": {
+    "dsh-plugin-dbgraph-tool": "^0.1.0",
+    "dsh-plugin-dbgraph-store": "^0.1.0",
+    "dsh-plugin-dbgraph-catalog": "^0.1.0",
+    "dsh-plugin-dbgraph-service": "^0.1.0"
+  }
 }