@foggy-projects/deepseek-harness-plugin 0.4.0-beta.9 → 0.4.0-rc.2

package/skills/foggy-deepseek-onboarding/references/onboarding-workflow.md

@@ -1,156 +1,168 @@
-# Resumable database onboarding
-
-Use the thin `scripts/onboard.ps1` or `scripts/onboard.sh` entrypoint. Every command emits one JSON
-object. Preserve the returned `next` instruction and use `onboard-status` or `onboard-resume` after an
-interruption.
+# Development datasource and semantic-model workflow
+
+Use this workflow after the Foggy plugin is initialized. It is designed for local experience and
+model development in DeepSeek Harness, not for production deployment.
+
+## Connection input
+
+The simplest local connection contract may contain a direct password:
+
+```json
+{
+  "schemaVersion": "foggy-deepseek-connection/v1",
+  "profile": "business",
+  "name": "business-db",
+  "type": "mysql",
+  "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/business",
+  "username": "business_dev",
+  "password": "local-development-password",
+  "namespace": "business",
+  "modelsDir": "models",
+  "evidenceDir": ".foggy/onboarding-command-evidence/business"
+}
+```
 
-The wrapper is mandatory for the entire sequence below. Until `onboard-status` returns
-`next.status=completed`, do not substitute direct `foggy-runtime datasources`, `tables`, `sql`, `models`,
-`bundles`, or `query` calls. A second Skill may help write draft TM/QM files, but must not take over
-orchestration. Do not inspect sample rows to author the semantic layer; use schema metadata plus the
-business definitions confirmed by the user.
+The wrapper uses the direct password only while submitting the datasource to the public Runtime API.
+It does not copy it into onboarding state, command evidence, TM/QM files, or conversational output.
+The source connection file is user-managed; do not commit it. The local Runtime currently stores a
+direct password in its private datasource registry, so use `passwordRef`-style production credential
+management when moving beyond local development.
+
+An environment variable is an optional alternative:
+
+```json
+{
+  "schemaVersion": "foggy-deepseek-connection/v1",
+  "profile": "business",
+  "name": "business-db",
+  "type": "mysql",
+  "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/business",
+  "username": "business_dev",
+  "passwordEnv": "FOGGY_BUSINESS_DB_PASSWORD",
+  "namespace": "business",
+  "modelsDir": "models"
+}
+```
 
-## Conversation contract
+Here `passwordEnv` is read by the onboarding wrapper and submitted online to the already-running
+Runtime. Do not restart Runtime merely to make the variable part of the Java process environment.
+Opaque CLI profiles remain supported for users who already prefer them, but are not required for a
+normal DSH experience.
 
-Collect these non-secret values before creating a plan:
+## Normal sequence
 
-1. an opaque Foggy profile ID and its exact reviewed revision;
-2. datasource name, database type, and namespace from the CLI's public profile summary;
-3. optional schemas, the project-relative semantic model directory, and a project-relative evidence directory.
+Run the platform-specific wrapper from this Skill directory. `onboard-datasource-run` performs the
+online datasource setup, namespace binding, and schema discovery against the healthy Runtime:
 
-Create the private profile in a trusted operator session outside DeepSeek Harness. Do not ask the user
-to paste a password and do not put a JDBC URL, username, password environment-variable name, or password in the JSON plan, command
-line, logs, or evidence. For non-SQLite databases, ask the user to set the named environment variable
-before Runtime starts so the Java process inherits it. Recommend a read-only database account.
+```text
+onboard-datasource-run --project-root <current-workspace> \
+  --connection-file <connection-json> \
+  --approve-configure --approve-bind --include-indexes
+```
 
-Use the current DSH session workspace as `projectRoot`. Store approved non-secret contracts at
-`.foggy/onboarding-contracts/<profile>/`, semantic drafts at `.foggy/onboarding-drafts/<profile>/`, and
-command evidence at `.foggy/onboarding-command-evidence/<profile>/`. Do not split these files across the
-session workspace and a separate example repository. The wrapper rejects a query payload outside the
-recorded project root before any semantic validation or publication mutation occurs.
+If the user already requested the complete connection experience, include both approval flags in the
+first call. They refer only to adding the named development datasource and binding the requested
+namespace. Do not add `--replace` unless replacement was explicitly requested.
 
-The wrapper defaults `FOGGY_RUNTIME_PROFILE_STORE` to the private persistent directory
-`<dataRoot>/cli-profiles`. An explicit operator-provided value still wins. Do not use `/tmp` for a profile
-that must survive a WSL or Harness restart.
+The command performs:
 
-If plugin settings detect profiles in the legacy temporary store, use **Move to persistent store**.
-Migration validates the opaque profile schema, rejects embedded passwords and conflicts, writes the
-destination with private permissions, verifies it, and moves the legacy JSON to a recoverable private
-backup below the Foggy data root.
-Do not manually copy or edit opaque profile JSON.
+```text
+datasource add -> datasource test -> namespace bind -> diagnostics
+tables list -> bounded schema inspection
+```
 
-Treat every user-supplied identifier and bound as immutable input: profile, datasource, namespace,
-models directory, bundle name, TM/QM name, query fields, and limit. Do not swap in demo names, add
-fields, raise the limit, or introduce replacement flags. If one of these inputs is missing, pause for
-that input rather than inventing it.
+It does not stop or restart Runtime. Repeated calls reuse completed checkpoints when the public
+connection contract is unchanged. Before reusing a datasource checkpoint, the wrapper verifies that
+the named datasource still exists with the expected public type in the live Runtime. If it was
+removed outside the wrapper, datasource configuration, verification, and schema discovery resume
+automatically from the first affected step. Inline password values are excluded from the comparison.
+Transient connection-pool readiness timeouts are retried a small, bounded number of times; invalid
+credentials, invalid URLs, and other configuration errors fail immediately.
 
-For composite Harness execution, persist `profile` and `evidenceDir` inside both approved contract
-files. The wrapper derives them from the files and rejects conflicting command-line values. Present the
-normalized contracts to the user before accepting mutation approvals; do not combine contract drafting
-and approval into an unseen one-turn mutation.
+When the user's request clearly identifies a business subject or likely tables, add `--pattern` or
+one or more `--table` arguments so discovery remains focused. Do not enumerate CLI help, inspect the
+wrapper source, load unrelated examples, or fetch online documentation on the successful normal path.
+Keep full command payloads in the numbered evidence files and use only their concise result fields in
+the conversation.
 
-Write the non-secret input using `assets/connection.schema.json`; `assets/datasource.example.json` is a
-template. Then execute the sequence below.
+After discovery, use `foggy-ai-analysis` to create TM/QM drafts from actual tables, columns, keys,
+small read-only samples when useful, and the user's business questions. Keep the files in the current
+workspace, normally:
 
-## Deterministic sequence
+```text
+models/
+  model/
+    <Name>.tm
+  query/
+    <Name>QueryModel.qm
+```
 
-For Harness-driven onboarding, use the two composite commands below. Each writes the numbered JSON
-evidence internally and refuses to cross an unapproved mutation gate:
+Then run the semantic composite:
 
 ```text
-onboard-datasource-run --project-root <current-session-workspace> \
-  --connection-file <approved-json> \
-  --approve-configure --approve-bind --include-indexes
-
-# After authoring the registered TM/QM draft from schema metadata:
-onboard-semantic-run --project-root <current-session-workspace> \
-  --semantic-plan <approved-json> --query-payload <approved-json> \
+onboard-semantic-run --project-root <current-workspace> \
+  --semantic-plan <semantic-json> \
+  --query-payload <bounded-query-json> \
   --approve-validate --approve-publish --approve-execute
 ```
 
-Include an approval flag only after the user approves that exact action. Without it, the composite
-command stops after the corresponding dry-run and returns `phaseStatus=awaiting-...-approval`.
+For an explicitly requested end-to-end local experience, these flags may be used together. The command
+validates the model files, copies them to the approved model directory when needed, registers or
+updates the local Runtime Bundle, refreshes/describes the declared QueryModels, validates the bounded
+query, and executes it. It never means production publication.
 
-Both composite commands are resumable. When the approved contract is unchanged, they skip completed
-checkpoints instead of re-adding an existing datasource, revalidating an already published draft, or
-registering the same bundle twice. If query validation fails after publication, correct the project-local
-payload and rerun `onboard-semantic-run`; it resumes at query verification and leaves the active bundle in
-place. A same-name datasource is accepted idempotently only when its public name and database type match
-the approved plan; otherwise replacement still requires explicit approval.
+The composite result keeps `success=true` when the Runtime safely continues with warnings and exposes
+`warningCount`, `warningCodes`, and `warnings` to the Agent. The full Runtime responses remain in
+`query-validate.json` and `query-execute.json`. Read the structured warning facts rather than relying
+on free-text remediation alone: compare `path`, `suggestedNextAction`, `safeToAutoRepair`,
+`normalizedFragment`, `docsRef`, and `details.allowedProperties` with the described
+model, correct and revalidate when an equivalent supported query preserves the user's intent, or
+clearly report the degraded interpretation. The wrapper accepts Runtime 0.1.21
+`queryInputWarnings` and legacy `warnings` as separate warning kinds. In `IGNORE`, ordinary unknown
+properties continue without `queryInputWarnings`; in `STRICT`, the Runtime rejects the query with the
+complete violations. Protected and governance fields remain fail-closed in every mode.
 
-When a completed profile already belongs to another workspace, the datasource composite may add the
-current workspace as a binding only if the approved connection contract is identical and the datasource,
-schema, and semantic publication checkpoints are complete. The semantic composite then accepts the same
-published draft digest from the bound workspace and performs a workspace-specific bounded verification
-query. It refuses semantic replacement from the secondary workspace.
+Use `--watch` only when the user wants the local Runtime to follow model-file edits. Use `--prune` or
+bundle replacement only when the user clearly asks for those changes.
 
-The granular commands below remain available for manual troubleshooting and resumption. Do not expand
-the composite commands into this list during a normal Harness turn.
+## Iteration and troubleshooting
+
+The granular commands remain available when a specific stage needs repair:
 
 ```text
 onboard-plan --connection-file <json> --profile <profile>
-datasource-configure --profile <profile>
-datasource-configure --profile <profile> --apply
-datasource-verify --profile <profile>
-datasource-verify --profile <profile> --bind
-schema-discover --profile <profile> --include-indexes
+datasource-configure --profile <profile> [--apply]
+datasource-verify --profile <profile> [--bind]
+schema-discover --profile <profile> [--include-indexes]
 semantic-draft --profile <profile> --semantic-plan <json>
-semantic-validate --profile <profile>
-semantic-validate --profile <profile> --apply
-semantic-publish --profile <profile>
-semantic-publish --profile <profile> --apply
-semantic-verify --profile <profile> --project-root <current-session-workspace> --query-payload <json>
-semantic-verify --profile <profile> --project-root <current-session-workspace> --query-payload <json> --execute
+semantic-validate --profile <profile> [--apply]
+semantic-publish --profile <profile> [--apply]
+semantic-verify --profile <profile> --query-payload <json> [--execute]
 onboard-status --profile <profile>
+onboard-resume --profile <profile>
 ```
 
-The first `datasource-configure` call is a dry run. Explain the concrete datasource mutation and get
-approval before adding `--apply`. `datasource-verify` tests connectivity without binding; explain the
-namespace binding mutation and get approval before adding `--bind`. Add `--replace` only when the user
-explicitly approves replacing an existing datasource definition.
-
-Schema discovery is metadata-only and inspects at most 25 tables by default. Use repeated `--schema` or
-`--table` arguments to narrow scope, `--list-only` for inventory only, and raise `--max-tables` only when
-the user needs broader inspection. The resulting evidence remains under the private Runtime data root.
-
-## Semantic authoring and publishing
-
-After `schemaDiscovered=completed`, review table names, columns, keys, and relationships with the user.
-Use `foggy-ai-analysis` for TM/QM authoring in a separate project-local draft directory. Do not invent
-business definitions, joins, units, enum meanings, or date semantics. Create a plan using
-`assets/semantic-plan.schema.json`; its declared query-model names must exist in the draft `.qm` files.
-
-`semantic-draft` records TM/QM hashes. Any later file change invalidates the recorded validation and must
-be registered and validated again. `semantic-validate` is a dry run until `--apply`; applying it may
-replace the Runtime validation catalog, so explain that mutation before approval.
-
-`semantic-publish` shows added, updated, unchanged, preserved, and optionally removed files. Applying it:
+A standalone `datasource-configure --apply` can resolve `passwordEnv` from its current Agent process.
+For a direct `password`, use the composite datasource command with the original connection file so the
+secret stays ephemeral to that invocation.
 
-1. backs up every overwritten or pruned project model file;
-2. copies the registered draft;
-3. validates the published directory;
-4. registers the bundle;
-5. refreshes only the declared query models.
+When a failure occurs, repair only the failed input and rerun the same composite command. Do not remove
+a successfully registered Bundle merely because later query validation failed. Keep SQL samples small,
+read-only, and relevant to semantic authoring.
 
-Use `--prune` only when the user approves removing target-only TM/QM files. Use `--replace-bundle` only
-when the user approves replacing an existing Runtime bundle, and `--watch` only when file watching is
-desired. A failure before refresh restores project files; a refresh failure is reported as a partial
-Runtime publication and must be diagnosed before retrying.
+## Git handoff
 
-For verification, build a project-local query payload only from `models describe` field names. The helper
-requires an integer `limit` from 1 through 100. The first `semantic-verify` call lists/describes models and
-validates the query; only `--execute` reads business data. Full results stay in evidence and the command
-returns only counts and paths.
+When validation and representative questions pass, recommend placing the model directory under the
+user's existing Git repository. Git is the source of truth for model history; Runtime Bundle
+registration only makes a selected local directory active. Do not create commits, remotes, tags, or
+pushes without the user's request.
 
-Before the first semantic mutation, the composite command checks that the query payload is inside
-`projectRoot`, has a bounded limit, and targets a query model declared in the approved semantic plan.
-After publication, `semantic-verify` always describes the live model before validation. If a field is
-rejected, update the payload from those described names and rerun the same composite command.
+## Production handoff
 
-Do not open, summarize, or quote `query-execute.json` or the Runtime's generated SQL after execution.
-The conversational result may contain only `queryValidated`, `queryExecuted`, `rowCount`, the model
-name, and the evidence path. A successful direct CLI command is not proof that onboarding completed;
-only the wrapper's persisted state with all steps `completed` is authoritative.
+Stop the local onboarding flow when the user asks to publish or update a formal environment. A separate
+manual or dedicated deployment workflow should collect the exact target, Launcher version, model Git
+commit/tag, namespace, production datasource procedure, access credentials, verification plan, and
+rollback point. It should then install/start Runtime if needed, prepare the target datasource, register
+the versioned model directory, refresh, and run a narrow smoke check.
 
-Do not bypass a failed step. If a command fails, report its structured error, fix only the relevant input
-or environment, and run `onboard-resume` to continue from the first incomplete step.
+Local onboarding credentials and approvals do not authorize production access or publication.

package/skills/foggy-deepseek-onboarding/scripts/invoke-onboarding.ps1

@@ -26,5 +26,7 @@ if (-not $pythonCommand -or -not (Test-Path -LiteralPath $pythonCommand)) {
     exit 1
 }
 
+$env:PYTHONUTF8 = '1'
+$env:PYTHONIOENCODING = 'utf-8'
 & $pythonCommand (Join-Path $PSScriptRoot 'onboarding.py') @OnboardingArgs
 exit $LASTEXITCODE