@n8r/surface 0.0.0-stage → 0.1.0-experimental.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (96) hide show
  1. package/LICENSE +202 -0
  2. package/LICENSE-DOCS +395 -0
  3. package/LICENSING.md +36 -0
  4. package/README.md +181 -2
  5. package/dist/activity.d.ts +16 -0
  6. package/dist/activity.js +44 -0
  7. package/dist/cli.d.ts +2 -0
  8. package/dist/cli.js +260 -0
  9. package/dist/codec.d.ts +4 -0
  10. package/dist/codec.js +41 -0
  11. package/dist/codex.d.ts +72 -0
  12. package/dist/codex.js +510 -0
  13. package/dist/contracts.d.ts +129 -0
  14. package/dist/contracts.js +1 -0
  15. package/dist/html.d.ts +5 -0
  16. package/dist/html.js +50 -0
  17. package/dist/index.d.ts +7 -0
  18. package/dist/index.js +5 -0
  19. package/dist/notes.d.ts +52 -0
  20. package/dist/notes.js +184 -0
  21. package/dist/portable.d.ts +10 -0
  22. package/dist/portable.js +71 -0
  23. package/dist/revisions.d.ts +2 -0
  24. package/dist/revisions.js +18 -0
  25. package/dist/settings.d.ts +23 -0
  26. package/dist/settings.js +100 -0
  27. package/dist/setup.d.ts +9 -0
  28. package/dist/setup.js +161 -0
  29. package/dist/source-tools.d.ts +102 -0
  30. package/dist/source-tools.js +201 -0
  31. package/dist/startup.d.ts +7 -0
  32. package/dist/startup.js +74 -0
  33. package/dist/surface-draft.d.ts +19 -0
  34. package/dist/surface-draft.js +95 -0
  35. package/dist/surface-instructions.d.ts +2 -0
  36. package/dist/surface-instructions.js +23 -0
  37. package/dist/surface-store.d.ts +36 -0
  38. package/dist/surface-store.js +181 -0
  39. package/dist/validate.d.ts +4 -0
  40. package/dist/validate.js +184 -0
  41. package/docs/n8r-surface/README.md +133 -0
  42. package/docs/n8r-surface/codex-research-transcript.md +51 -0
  43. package/docs/n8r-surface/codex-research-verification.md +176 -0
  44. package/docs/n8r-surface/command-outcomes.md +564 -0
  45. package/docs/n8r-surface/contracts.md +210 -0
  46. package/docs/n8r-surface/critical-path.md +163 -0
  47. package/docs/n8r-surface/design.md +289 -0
  48. package/docs/n8r-surface/getting-started.md +107 -0
  49. package/docs/n8r-surface/host-contract.md +176 -0
  50. package/docs/n8r-surface/implementation-decisions.md +96 -0
  51. package/docs/n8r-surface/npm-distribution.md +137 -0
  52. package/docs/n8r-surface/npm-release.md +188 -0
  53. package/docs/n8r-surface/recreation-handoff.md +117 -0
  54. package/docs/n8r-surface/researcher-discovery.md +157 -0
  55. package/docs/n8r-surface/researcher-prepared-check.md +117 -0
  56. package/docs/n8r-surface/researcher-reconciliation-trial.md +149 -0
  57. package/docs/n8r-surface/researcher-repository-check.md +95 -0
  58. package/docs/n8r-surface/researcher-skeleton-check.md +84 -0
  59. package/docs/n8r-surface/researcher-skill-verification.md +725 -0
  60. package/docs/n8r-surface/researcher-skill.md +255 -0
  61. package/docs/n8r-surface/researcher-vault.md +134 -0
  62. package/docs/n8r-surface/save-reopen-transcript.md +211 -0
  63. package/docs/n8r-surface/save-reopen-verification.md +152 -0
  64. package/docs/n8r-surface/startup-human-check.md +172 -0
  65. package/docs/n8r-surface/startup-verification.md +161 -0
  66. package/docs/n8r-surface/startup.md +151 -0
  67. package/docs/n8r-surface/surface-revisions-transcript.md +552 -0
  68. package/docs/n8r-surface/surface-revisions-verification.md +161 -0
  69. package/docs/n8r-surface/vault-setup-verification.md +210 -0
  70. package/docs/n8r-surface/verification.md +117 -0
  71. package/docs/n8r-surface/versioning.md +183 -0
  72. package/examples/researcher/README.md +92 -0
  73. package/examples/researcher/composed.surface.json +430 -0
  74. package/examples/researcher/composed.surface.yaml +283 -0
  75. package/examples/researcher/harbor-r1.research.html +193 -0
  76. package/examples/researcher/harbor-r2.research.html +226 -0
  77. package/examples/researcher/lumen-manual-edit.pending.html +271 -0
  78. package/examples/researcher/lumen-r1.research.html +271 -0
  79. package/examples/researcher/lumen-r2.research.html +280 -0
  80. package/examples/researcher/sources/harbor-api.md +16 -0
  81. package/examples/researcher/sources/harbor-auth.md +8 -0
  82. package/examples/researcher/sources/lumen-guide.md +28 -0
  83. package/examples/researcher/sources/lumen-settings.md +9 -0
  84. package/package.json +45 -4
  85. package/plugins/n8r-surface/.codex-plugin/plugin.json +31 -0
  86. package/plugins/n8r-surface/LICENSE +202 -0
  87. package/plugins/n8r-surface/LICENSE-DOCS +395 -0
  88. package/plugins/n8r-surface/LICENSING.md +36 -0
  89. package/plugins/n8r-surface/assets/icon.svg +5 -0
  90. package/plugins/n8r-surface/plugin.json +27 -0
  91. package/plugins/n8r-surface/skills/research/SKILL.md +122 -0
  92. package/plugins/n8r-surface/skills/research/agents/openai.yaml +4 -0
  93. package/plugins/n8r-surface/skills/research/references/discovery-map.md +160 -0
  94. package/plugins/n8r-surface/skills/research/references/model-meanings.md +55 -0
  95. package/plugins/n8r-surface/skills/research/references/skeleton-surface.md +105 -0
  96. package/plugins/n8r-surface/skills/research/references/working-notes.md +116 -0
@@ -0,0 +1,210 @@
1
+ # Experimental Researcher artefact contracts 0.1
2
+
3
+ Issue [#15](https://github.com/injectionator/n8r-surface/issues/15) introduces a
4
+ bounded experimental profile of the **AI Surface Model**, not a released universal
5
+ schema. Existing [LibreChat examples](../../examples/README.md) remain illustrative
6
+ and use a different shape. No deployment, interoperability or recreation claim
7
+ follows from passing these checks.
8
+
9
+ The TypeScript [types](../../src/contracts.ts), runtime
10
+ [checks](../../src/validate.ts) and [synthetic cases](../../examples/researcher/README.md)
11
+ make this profile concrete. Other languages can implement the same data contract.
12
+ The library reads supplied strings and returns values/strings; it does not read
13
+ files, fetch references, call inference or write a corpus.
14
+
15
+ ## Common envelope and identity
16
+
17
+ Every artefact has these required fields. Objects are closed except the explicit
18
+ `extensions` object; misspelled or future core fields are rejected, never stripped.
19
+
20
+ | Field | Meaning |
21
+ | --- | --- |
22
+ | `contract` | Exactly `n8r.research/0.1` or `n8r.surface/0.1`. |
23
+ | `id` | Stable namespaced identifier for this artefact across revisions. |
24
+ | `title`, `scope` | Nonempty human text; scope states coverage limits. |
25
+ | `revision` | `{id, parent, createdAt, author, reason}`. |
26
+ | `extensions` | Map of namespaced identifiers to portable values; `{}` is valid. |
27
+
28
+ Identifiers have a URI-style scheme and non-whitespace remainder. They are opaque
29
+ and case-sensitive; the checker does not fetch them or claim URI normalization.
30
+ Revision IDs are nonempty alphanumeric/dot/underscore/hyphen tokens, starting with
31
+ an alphanumeric character. They identify snapshots, not product release versions
32
+ or timestamps. Times use real UTC dates with seconds: `2026-11-04T00:00:00Z`.
33
+ Names and locations are labels, never identity matching rules.
34
+
35
+ `parent` is null for an initial revision, or `{id, revision, digest}` for the
36
+ previous snapshot of this same artefact. Revision changes retain the artefact ID
37
+ and existing component/finding/relationship IDs when their identity is unchanged.
38
+ A fork gets a new artefact ID and may record its origin in an extension. The
39
+ library does not infer whether two similarly named entities are the same thing.
40
+
41
+ `verifySuccessor(previous, next)` checks matching contract and identity, a fresh
42
+ revision token and an exact parent revision/digest. It rejects changing the type
43
+ or endpoints of a retained relationship ID; assign a new ID for a different
44
+ connection. Historical snapshots remain immutable. The caller must preserve old
45
+ bytes and prevent ID reuse across its complete history; a two-snapshot check
46
+ cannot enforce corpus-wide immutability. A snapshot ID is distinct from a later
47
+ Lab's deliberate release/version label.
48
+
49
+ ## Research record
50
+
51
+ `n8r.research/0.1` carries these required arrays (empty arrays are allowed unless
52
+ specified). IDs are unique across the record and all identified entries.
53
+
54
+ | Array | Entry fields |
55
+ | --- | --- |
56
+ | `sources` | `id`, `location`, `version`, `digest`, `inspectedAt`, `coverage`. |
57
+ | `findings` | `id`, `subject`, `property`, `state`, `note`, `claims`. |
58
+ | `alternatives` | `id`, `title`, `findings` (finding IDs), `requirements` (text). |
59
+ | `decisions` | `id`, `alternative` (option ID), `choice`, `reason`, `by`, `at`. |
60
+ | `questions` | `id`, `text`, `findings` (possibly empty related finding IDs). |
61
+ | `notes` | `by`, `at`, `text`; human commentary and corrections. |
62
+
63
+ A source identifies the supplied snapshot. `digest` is SHA-256 over its exact
64
+ UTF-8 bytes; `version` records an applicable document revision or checkout state.
65
+ A dirty checkout must not be labeled only with a clean commit. `location` is a
66
+ portable relative path, HTTP(S) URL without credentials, or URN; it conveys no
67
+ permission to access it. Absolute local paths and traversal are rejected. Coverage
68
+ contains at least one `{path, status, note}` entry; status is `inspected`, `missing`,
69
+ `unreadable`, `excluded`, `truncated` or `unsupported`. If a source's bytes are
70
+ unavailable and cannot be hashed, do not invent a digest: record the missing
71
+ material in scope/questions or coverage of the supplied source bundle.
72
+
73
+ A finding's `subject` is a stable namespaced identifier for the described thing;
74
+ it need not already be a selected graph node. `property` names the question being
75
+ answered. A claim is `{value, evidence}`. Evidence is a nonempty array of
76
+ `{source, locator}` referring to a local source and a human-readable section or
77
+ range. `note` states applicability, limitations and the basis for the conclusion.
78
+ The contract checks reference existence, not whether prose supports a claim.
79
+
80
+ States have distinct meanings:
81
+
82
+ - `known`: at least one sourced claim; all claim values agree.
83
+ - `absent`: a scoped, sourced absence assertion; at least one agreed claim value
84
+ describing what is absent. Absence outside that scope is not implied.
85
+ - `conflicting`: at least two differing sourced values, with no selected answer.
86
+ - `unknown`, `omitted`, `redacted`, `out-of-scope`: no claim values. The note explains
87
+ the distinction. Do not place withheld values in a redacted field's extension.
88
+
89
+ Decisions use `selected`, `rejected` or `deferred`, at most one current decision
90
+ per alternative. An unanswered option has no decision. A changed decision belongs
91
+ in a successor record; earlier decisions remain in its predecessor. Research
92
+ options are possibilities; selecting one does not prove a working integration.
93
+
94
+ ## Selected Surface definition
95
+
96
+ `n8r.surface/0.1` has `kind: "definition"`. It contains:
97
+
98
+ | Field | Shape and interpretation |
99
+ | --- | --- |
100
+ | `records` | Exact research `{id, revision, digest}` references. |
101
+ | `vocabularies` | Unique `{id, version}` references; version is a token. |
102
+ | `nodes` | Identified components, each with a human `label`. |
103
+ | `relationships` | Identified connections with explicit `from`, `to` and `claim: "declared"`. |
104
+ | `requirements` | `{id, kind, description, state}`; kind is `secret`, `resource` or `configuration`. |
105
+ | `coverage`, `gaps` | Arrays of human text about what is described and still needed. |
106
+
107
+ Every node/relationship carries `id`, `type: {vocabulary, term}`, `attributes`,
108
+ `basis` and `requirements` (local requirement IDs). Type vocabulary references
109
+ must resolve locally; unknown terms/vocabularies remain opaque and preserved.
110
+ Their presence does not establish consumer understanding. Graph IDs and requirement
111
+ IDs are unique across the Surface. Edges reference node IDs; shared nodes, cycles,
112
+ self-loops and distinct parallel edges are legal. Degree is relative to a start
113
+ node; no semantic two-hop limit exists.
114
+
115
+ `attributes` maps property names to `{state, note, value?}`. `known` requires a
116
+ portable value (including null); all other states prohibit one. Supported states
117
+ are the six draft states above, excluding `conflicting`: a selected definition
118
+ can leave an attribute unknown and cite the competing research findings. An
119
+ omitted attribute is unspecified, never automatically absent.
120
+
121
+ `basis` entries are `{record, revision, items}` with nonempty lists of research
122
+ finding, alternative, decision or question IDs. Empty basis is allowed for an
123
+ explicit author declaration; the envelope's author/reason and attribute notes
124
+ remain its attribution. `validateArtefact` checks local structure and references.
125
+ `verifyReferences(surface, records)` additionally requires the supplied research
126
+ revisions, verifies their logical digests and resolves each basis item. Missing
127
+ external context is an error from that explicit check, not evidence of absence.
128
+ No reference is fetched. A valid link is not proof that a selection is justified.
129
+
130
+ Secret requirements name a need, never contain a secret or a recipient's credential
131
+ binding. Requirements describe the selected composition; a Lab separately decides
132
+ how to satisfy them. Reachability, exercised connections, deployment bindings,
133
+ effective settings and Experiment Runs need separate linked records. Those record
134
+ formats are outside this profile. Later evidence cannot mutate this definition.
135
+
136
+ ## Portable values and JSON/YAML
137
+
138
+ JSON is the default; YAML is an alternative representation of the same logical
139
+ data. The profile supports null, booleans, well-formed Unicode strings, signed safe
140
+ integers from `-9007199254740991` through `9007199254740991`, arrays and string-keyed
141
+ objects. JavaScript arrays must have the ordinary `Array.prototype`; subclasses
142
+ and arrays with custom prototypes are rejected before inherited methods or
143
+ serialization hooks can run. Numeric tokens use decimal integer notation; no
144
+ negative zero, fraction, exponent, leading zero, hexadecimal, infinity or NaN.
145
+ Use an explicit string plus
146
+ unit for decimal quantities or larger identifiers. This deliberately small subset
147
+ prevents number precision loss between runtimes.
148
+
149
+ Reject duplicate keys, non-string mapping keys, custom or explicit YAML tags,
150
+ anchors, aliases, unquoted merge keys, directives and multiple YAML documents.
151
+ Quoted literal `<<` keys are ordinary extension data and are preserved. YAML comments
152
+ and formatting are not semantic and are not preserved by re-encoding. Strings
153
+ that resemble booleans, dates or numbers stay strings when emitted. JSON object
154
+ order is insignificant; array order is preserved. JavaScript inputs must be plain
155
+ data: no accessors, sparse arrays, extra array properties, cycles, class instances,
156
+ undefined, symbols or non-enumerable fields.
157
+
158
+ Unknown `extensions` values are preserved through conversion, revision checks and
159
+ HTML. No tool may treat opaque extensions as understood semantics or discard them
160
+ silently. Unknown core fields and contract versions fail closed; a later migration
161
+ must retain the original and report losses. This profile has no required-extension
162
+ mechanism: a consumer must separately report which extension meanings it supports.
163
+
164
+ The reference implementation bounds each encoded input/output and canonical
165
+ logical document to 2 MiB and nested values to depth 64. These are defensive
166
+ codec limits, not graph depth semantics. Oversized or unsupported input fails
167
+ without producing a replacement document.
168
+
169
+ ## Logical digest and byte digest
170
+
171
+ `digest(value)` returns `sha256:` plus lowercase hexadecimal SHA-256 over UTF-8
172
+ canonical JSON: recursively sort object keys by UTF-16 code units, preserve array
173
+ order, emit compact JSON strings/booleans/null/decimal integers, and add no trailing
174
+ newline. All fields, including extensions and parent references, participate.
175
+ No Unicode normalization occurs. This is this profile's algorithm, not a claim
176
+ of implementing a general canonicalization standard.
177
+
178
+ For example, both key orders of `{"a":1,"b":2}` hash to
179
+ `sha256:43258cff783fe7036d8a43033f830adfc60ec037382473548ac742b888292777`.
180
+ Research/Surface references use this logical digest. Source digests and future
181
+ storage compare-and-swap tokens instead hash exact bytes, including formatting.
182
+ A logical digest neither proves authorship nor describes what was running.
183
+
184
+ ## HTML ownership and manual editing
185
+
186
+ The durable research record is a complete UTF-8 HTML document, readable offline.
187
+ Its single inert `script` data block (`type="application/json"`, `id="n8r-record"`)
188
+ contains the authoritative research record. All visible facts and notes are a
189
+ deterministic projection of that block; there is no second narrative authority.
190
+ Source text is escaped as text, including closing-script sequences. Generated
191
+ HTML contains no executable code, external assets, active source links or forms.
192
+
193
+ `readResearchHtml` validates the data and compares the entire supplied document
194
+ with `renderResearchHtml(data)`. Any edit to the visible projection, embedded data,
195
+ styles or surrounding markup is a conflict. Even benign reformatting requires
196
+ review. This strict first profile favors detecting edits over accepting arbitrary
197
+ HTML. It is not a general HTML sanitizer; never render arbitrary imported HTML
198
+ in a privileged host before validation.
199
+
200
+ To reconcile an edit, preserve the original bytes and the manually edited copy,
201
+ review both data and visible changes with the human, and prepare a corrected
202
+ successor data object. Put human prose in `notes` or the relevant structured field;
203
+ retain unknown extensions. Check its parent against the last accepted revision,
204
+ render a new HTML file, and show what changed. Never regenerate over an edited
205
+ file merely because its embedded data still parses. Guarded atomic persistence,
206
+ backup retention and byte-conflict tokens are #16's responsibility; this library
207
+ returns strings and cannot overwrite a file.
208
+
209
+ See the [manual-edit fixture](../../examples/researcher/README.md#manual-edit-case)
210
+ and [host continuation contract](host-contract.md).
@@ -0,0 +1,163 @@
1
+ # n8r-surface critical path
2
+
3
+ The user updated delivery order on 5 November 2026: standalone Codex research,
4
+ Markdown save/reopen, then partial Surface generation and a complete fresh-session
5
+ revision proof. npm distribution ([#27](https://github.com/injectionator/n8r-surface/issues/27))
6
+ is separate and last. Lab integration stays in the n8r-labs repository.
7
+
8
+ The standalone conversation and Markdown continuation are merged in #33 and #34.
9
+ The current #17 increment adds paired structured research HTML and Surface JSON;
10
+ see the [short human check](getting-started.md) and
11
+ [versioned evidence](surface-revisions-verification.md). Broader host lifecycle,
12
+ live observations and consumer compatibility remain separate acceptance work.
13
+ The unfinished #16 source-overview branch is paused; no status subcommand is planned.
14
+
15
+ ```mermaid
16
+ flowchart LR
17
+ Contract[Artefact contracts] --> CLI[Standalone Codex research]
18
+ CLI --> Notes[Markdown save and reopen]
19
+ Notes --> Surface[Partial Surface and fresh-session revision]
20
+ Surface --> Distribution[Separate npm distribution decision]
21
+ ```
22
+
23
+ The sections below retain the wider requirements; their numbering is historical,
24
+ not the current delivery order. The
25
+ [command outcome specification](command-outcomes.md) governs implementation and
26
+ requires actual installed-runner proof beyond contract fixtures.
27
+
28
+ ## 1. Define artefacts with concrete examples
29
+
30
+ **Issue:** [#15](https://github.com/injectionator/n8r-surface/issues/15).
31
+
32
+ Define a versioned experimental HTML research record and partial Surface graph.
33
+ Use synthetic sources for two unrelated products and show how selected findings
34
+ compose into one description. Establish which representation owns each fact in
35
+ the HTML, how manual edits are retained, and what a revision identifies.
36
+
37
+ Specify JSON/YAML preservation rules and examples with attributed nodes and
38
+ relationships, shared components, cycles, multiple connections, unknowns and
39
+ conflicting evidence. The [implementation decisions](implementation-decisions.md) select TypeScript,
40
+ Node.js 22/24/26, the `n8r-surface` npm package and npm/Node testing. The original
41
+ Responses API runner choice has been superseded by installed Codex for later CLI work. Runner implementation and credentials are not
42
+ needed by these offline contract checks.
43
+
44
+ Review the [compatibility contract](design.md#compatibility-contract) against
45
+ n8r-labs requirements: source access, human questions, artefact persistence,
46
+ version handling and continuation. Record gaps and supported boundaries using
47
+ public, synthetic examples. This review does not implement the application.
48
+
49
+ **Advance when:** the examples and checks give #16 and #17 concrete contracts,
50
+ including a documented route for n8r-labs to consume the shared core later.
51
+ Do not relabel existing illustrative fixtures as schema-conforming without
52
+ specifying and checking them.
53
+
54
+ The [command outcome spec](command-outcomes.md) must also give each subsequent
55
+ increment a user task, initial state, actual input and independently verifiable
56
+ result. Contract fixtures alone do not pass a command's acceptance case.
57
+
58
+ ## 2. Prove the shared skill in the current harness
59
+
60
+ **Issue:** [#17](https://github.com/injectionator/n8r-surface/issues/17).
61
+
62
+ Use the [share/install and lifecycle procedure](researcher-skill.md) and record
63
+ the actual host/version, installed instruction identity, source access, output
64
+ and readback. The first package has no new runner or Lab dependency. A passing
65
+ manifest check or CLI installation does not establish a Work-mode invocation.
66
+
67
+ Lab adaptation remains separate work in n8r-labs. The standalone foundation and
68
+ research criteria below now precede it under the current user-selected order.
69
+
70
+ ## 3. Resume the local CLI foundation
71
+
72
+ **Issue:** [#16](https://github.com/injectionator/n8r-surface/issues/16).
73
+ **Depends on:** #15.
74
+
75
+ Implement informational startup, missing-setup guidance, current-directory and explicit
76
+ source selection, and the configured artefact folder. Provide bounded tools to
77
+ list, read, create and revise records. Updates check the version read earlier,
78
+ retain a recoverable previous revision and complete atomically.
79
+
80
+ Implement in TypeScript and package the built CLI and required assets for npm.
81
+ Verify the installed tarball on supported Node.js versions independently of the
82
+ source checkout. Bun may support development/builds but is not an end-user
83
+ requirement. Package tests do not require publishing a release.
84
+
85
+ STARTUP-01 selects `~/.n8r/n8r-surface.json` for settings and
86
+ `~/n8r/artefacts` as the default corpus; `artefactRoot` in that file overrides it.
87
+ The overview displays the effective path; see [settings decisions](startup.md).
88
+ Keep AI selection separate from an optional Lab destination. Preserve invalid
89
+ configuration for repair; non-interactive commands must not wait on a prompt.
90
+
91
+ **Advance when:** synthetic tests cover first run, missing/unwritable paths,
92
+ unavailable runners, interrupted saves and human edits between read and write.
93
+ The executable is a local foundation at this point; it must not present an
94
+ unconnected discovery command as working research.
95
+
96
+ Start with [STARTUP-01](command-outcomes.md#startup-01-run-the-command-and-see-what-is-here):
97
+ given a configuration file and one saved artefact, running `n8r-surface` alone
98
+ shows readable non-private settings and their locations, one line for that artefact,
99
+ and known process status or an honest unknown. Verify this simple terminal example
100
+ before the more involved research paths; it requires no AI or Lab connection.
101
+
102
+ Exercise those behaviors through the installed command as well as internal
103
+ handlers. Record the applicable SETUP, INPUT, SCOPE, SAVE, HELP and
104
+ NONINTERACTIVE cases from [the outcome spec](command-outcomes.md#other-required-paths-and-failure-outcomes),
105
+ including actual before/after state and independent readback.
106
+
107
+ ## 4. Demonstrate the standalone Researcher workflow
108
+
109
+ **Issue:** [#17](https://github.com/injectionator/n8r-surface/issues/17).
110
+ **Depends on:** #15 and #16.
111
+
112
+ Implement and version the shared Researcher skill in this repository, connecting
113
+ one supported AI runner and the local tools. No application extraction is required.
114
+ Support a human question, source inspection, saved findings, consequential
115
+ follow-up questions, corrections, continuation and JSON/YAML model output.
116
+ Reuse the skill already tried in ChatGPT with supplied documents. Host adapters provide access; they do not maintain separate
117
+ versions of discovery knowledge.
118
+
119
+ **Advance when:** a real supported-runner session completes the workflow on
120
+ permitted examples, the records survive save/reopen, and a second product works
121
+ without product-specific code. Compare facts, selections and gaps across the
122
+ CLI and external skill; generated wording need not match. Record any host
123
+ limitation and do not claim untested compatibility.
124
+
125
+ This completes the first standalone milestone. It does not establish that a Lab
126
+ importer accepts the output; that needs a separate named-consumer trial.
127
+
128
+ Use [E2E-01](command-outcomes.md#e2e-01-research-a-repository-and-produce-a-durable-record)
129
+ as the minimum complete path: locate the repo, load and follow the identified
130
+ Researcher instructions, interact with the human, save and reopen the actual
131
+ artefact. Exercise [E2E-02](command-outcomes.md#e2e-02-apply-the-same-path-to-librechat)
132
+ before claiming LibreChat research. Record actual tool/runner evidence and saved
133
+ state; do not substitute developer-authored findings or a fake runner.
134
+
135
+ ## Work after the first milestone
136
+
137
+ The application adaptation preceding CLI completion belongs in the n8r-labs project: use an identified
138
+ Researcher revision with the Lab's own source access, AI, conversation and
139
+ storage adapters. Use the shared examples to verify research behaviour and
140
+ import an incomplete standalone record/definition, then continue it in the Lab.
141
+ Preserve identities, evidence, gaps and user decisions; report unsupported data.
142
+ Record the tested application and contract versions before claiming compatibility.
143
+
144
+ The [capability map](README.md) also retains remote Researcher execution,
145
+ additional AI runners, MessagePack and standalone executables. Create their
146
+ delivery issues when their consumer contracts and acceptance cases are concrete.
147
+
148
+ Keep browser editing, a general database, automatic two-way synchronisation,
149
+ advanced indexing and deployment automation outside this first path. Users can
150
+ choose their own editor; safe Researcher-assisted updates are sufficient here.
151
+
152
+ ## Review and evidence
153
+
154
+ Each issue uses its own branch and draft PR in this repository. Link the primary
155
+ issue with `Closes #N` only when complete; use `Related to #N` for a partial
156
+ increment and retain its remaining scope. Link the overview with `Related to #13`.
157
+ Keep draft status
158
+ until review readiness is authorised; merging and releasing are separate steps.
159
+
160
+ Run the relevant [repository checks](../../CONTRIBUTING.md)
161
+ and the issue's acceptance cases. Distinguish file syntax, reference integrity,
162
+ model behaviour and actual consumer compatibility. Update the map and this path
163
+ when scope or dependencies change so the next task remains explicit.
@@ -0,0 +1,289 @@
1
+ # Researcher and n8r-surface design reference
2
+
3
+ This records the agreed direction and remaining choices for the standalone tool.
4
+ It consolidates [issue #13](https://github.com/injectionator/n8r-surface/issues/13).
5
+ The [capability map](README.md) shows the intended scope;
6
+ the [critical path](critical-path.md) determines what comes next.
7
+
8
+ **Status:** the [experimental #15 contract](contracts.md), codecs, HTML checks and
9
+ synthetic cases now provide concrete inputs to the next milestones.
10
+ The bare-startup overview is implemented; its [bounded behavior](startup.md)
11
+ does not include setup writes, conversation or other commands.
12
+ A [first skill package](researcher-skill.md) supplies Researcher instructions for
13
+ working notes; the standalone runner and structured-output workflow remain unimplemented. Existing LibreChat JSON
14
+ examples remain illustrative; this bounded profile is not a universal schema.
15
+
16
+ ## Purpose and roles
17
+
18
+ Researcher helps a human learn about software from repositories or documents
19
+ and develop a useful partial AI Surface Model description. It understands the
20
+ model's concepts and the requirements relevant to starting a Lab. Product
21
+ knowledge comes from supplied evidence, with general LLM knowledge treated as
22
+ hypotheses until supported.
23
+
24
+ `n8r-surface` is the working CLI name. Its standalone workflow requires a
25
+ supported AI runner and accessible sources, but no Lab installation, account
26
+ or running target. ChatGPT with supplied documents is the first external skill
27
+ experience. Other harnesses need their own demonstrated support.
28
+
29
+ Maintain one versioned core discovery skill, including its reference material
30
+ and output rules, in this repository. Prove the skill in the existing Codex/ChatGPT harness first; adapt the same core
31
+ in n8r-labs next, then complete the standalone CLI. Each host supplies
32
+ source access, inference, conversation and persistence. Existing application
33
+ needs inform the contract, but extracting or rebuilding its assistant is not a
34
+ prerequisite for standalone delivery. Do not hand-maintain divergent copies or
35
+ copy private application content.
36
+
37
+ ## Sources and scope
38
+
39
+ The [command outcome specification](command-outcomes.md) owns the complete initial
40
+ command inventory, conversational intents and State/Input/Outcome acceptance cases.
41
+ It is the product contract for implementation; the detailed artefact contracts
42
+ support it. Development starts with a real user task and returns to that same
43
+ interaction for verification. No command scenario is currently verified.
44
+
45
+ Core proposed research entry points:
46
+
47
+ ```sh
48
+ n8r-surface
49
+ n8r-surface discover
50
+ n8r-surface discover ./product
51
+ n8r-surface discover ./documents/architecture.md
52
+ ```
53
+
54
+ Bare startup first shows readable non-private configuration and its file locations,
55
+ local artefacts one per line, and process status where known. This local overview
56
+ works without an AI or Lab connection; see the concrete
57
+ [startup example](command-outcomes.md#startup-01-run-the-command-and-see-what-is-here).
58
+ The next CLI revision will make bare startup informational and exit without a
59
+ prompt; there will be no status subcommand. That work is paused while the skill
60
+ is developed. The existing STARTUP-01 guidance prompt remains the implemented
61
+ baseline. Research requires an explicit interaction; no implicit source upload
62
+ or task resumption is introduced.
63
+
64
+ `discover` without a source argument uses the process's current directory.
65
+ Explicit relative paths resolve from that directory; absolute paths are accepted
66
+ locally. Do not silently change to the tool installation or an ancestor repo.
67
+ Directories may contain source code or documentation; Git metadata is optional.
68
+ The destination and any existing record to extend are separate inputs.
69
+
70
+ A chat harness receives the documents made available to that conversation or
71
+ project. A named local path alone does not supply content. Likewise, a remote
72
+ Lab cannot automatically read a path on the CLI user's machine.
73
+
74
+ Retain source identity, applicable revision, inspected paths/document references
75
+ and coverage. A checkout can differ from its commit; preserve that distinction.
76
+ Portable output must not expose installation-specific absolute paths or secrets.
77
+ Report missing, unreadable, unsupported, excluded and truncated material. A
78
+ document-only investigation does not imply whole-repository coverage.
79
+
80
+ Reading sources does not execute their scripts, install dependencies, start
81
+ services or prove integration. Imported instructions remain source data.
82
+
83
+ ## Research records and Surface descriptions
84
+
85
+ The research record is the durable intermediate artefact between raw sources
86
+ and a selected Surface definition. HTML is the preferred research-record format.
87
+ It can contain findings, source references, possible configurations and
88
+ integrations, requirements, diagrams, decisions and unresolved questions.
89
+
90
+ A Surface description captures a particular selected composition and its gaps.
91
+ One research record can support several descriptions. Preserve the research
92
+ when generating a model; later research must not silently rewrite a definition
93
+ referenced by an earlier result.
94
+
95
+ The HTML should be portable and readable without a running n8r service. A
96
+ versioned inert data block may carry precise metadata or graph data. Decide
97
+ which representation owns each fact so rendered content, human edits and
98
+ embedded data cannot silently disagree. Source markup must remain inert.
99
+ An embedded research graph is not automatically a selected Surface definition.
100
+
101
+ Users choose their editor. Researcher-assisted editing is preferred; a built-in
102
+ browser editor is outside the initial scope. A host without storage tools returns
103
+ downloadable artefacts and must not claim it saved them into the user's corpus.
104
+
105
+ ## Model encodings and graph meaning
106
+
107
+ There is one AI Surface Model. JSON is the default encoding; YAML is an
108
+ alternative. MessagePack and other encodings are later candidates based on
109
+ consumer needs. Encoding choice does not define the ontology.
110
+
111
+ The proposed logical contract must preserve:
112
+
113
+ - Stable node identities, types and attributes.
114
+ - Identified relationships with explicit endpoints, types and attributes;
115
+ several distinct relationships may connect the same pair of nodes.
116
+ - Shared components, cycles, arbitrary depth and explicit references.
117
+ - Versioned vocabulary/type references and supported extension data.
118
+ - Provenance, applicability, uncertainty and coverage.
119
+
120
+ Keep supported product possibilities distinct from user-selected composition.
121
+ Keep definitions distinct from bindings, effective settings and dated runtime
122
+ observations. Preserve known, unknown, omitted, redacted, absent and out-of-scope
123
+ meanings. Matching product names alone do not establish shared instance identity.
124
+
125
+ Define a portable value subset and codec-specific handling of unsupported values.
126
+ Core identities must not depend on YAML aliases/custom tags or MessagePack
127
+ extension types. Demonstrate semantic round trips for the supported subset,
128
+ including edge attributes and unknowns. Reject or report losses; preserving
129
+ YAML comments or original formatting is a separate editing concern.
130
+
131
+ The [HTML data-block specification](https://html.spec.whatwg.org/multipage/scripting.html#data-block),
132
+ [YAML 1.2.2](https://yaml.org/spec/1.2.2/) and
133
+ [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
134
+ are format references. They do not establish model conformance or consumer
135
+ compatibility. See the repository's [mapping requirements](../interoperability.md).
136
+
137
+ ## Local settings and artefact tools
138
+
139
+ The first #16 increment selects the settings path and corpus default below.
140
+ See [the implemented settings subset](startup.md); setup and storage tools in
141
+ this section remain planned.
142
+
143
+ Selected defaults and planned outputs:
144
+
145
+ | Setting | Choice |
146
+ | --- | --- |
147
+ | Tool settings | `~/.n8r/n8r-surface.json` |
148
+ | Visible corpus | `~/n8r/artefacts`, overridable with a project or vault folder |
149
+ | Research record | HTML |
150
+ | Surface output | JSON by default; YAML available |
151
+ | Lab connection | Optional and independent of the AI runner |
152
+
153
+ The configuration holds a format version, selected runner and nonsecret options,
154
+ credential references, artefact root, output preference and optional Lab connection.
155
+ Preserve other n8r settings. Keep credentials with the runner or appropriate
156
+ credential storage; do not copy another application's tokens into this file.
157
+
158
+ Startup deterministically inspects configuration, runner availability and folder
159
+ access. Distinguish configured, missing, unavailable and not yet verified; a
160
+ configured runner does not establish a successful model request. A small non-LLM
161
+ bootstrap flow can guide missing setup before inference is available. Remember
162
+ choices, show the effective settings and allow changes.
163
+
164
+ Create new settings/directories without replacing existing files. Retain invalid
165
+ configuration for repair. Non-interactive runs return actionable status instead
166
+ of waiting for answers.
167
+
168
+ Keep artefact operations small: list/find/read, create and revise records,
169
+ change relevant settings, and produce a definition. The LLM interprets user
170
+ requests; deterministic handlers enforce identity/path scope and report actual
171
+ results. A selected default destination is sufficient; users need not choose
172
+ the folder on every save.
173
+
174
+ Updates check the revision/content hash previously read, re-read or report a
175
+ conflict when it changed, retain a recoverable prior revision and save atomically.
176
+ Human changes and unknown metadata must survive revision. Archiving and explicit
177
+ deletion can be added when their reference/retention behaviour is defined;
178
+ automatic destructive cleanup is unnecessary for the first workflow.
179
+
180
+ ## Implementation and distribution
181
+
182
+ The agreed first implementation uses TypeScript and ships a Node.js-compatible
183
+ npm package exposing the `n8r-surface` command. Support installation for regular
184
+ use and versioned invocation through npm/npx. Ship built JavaScript and required
185
+ skill/template assets; users should not need the source checkout, a TypeScript
186
+ compiler or Bun to run the published CLI.
187
+
188
+ Bun may be used for development and building, but verification must exercise
189
+ the packaged CLI on supported Node.js versions. Test an installed package tarball
190
+ in a temporary directory, including its bundled assets and command entry point.
191
+ These checks need no registry publication. The [#15 decisions](implementation-decisions.md) choose package `n8r-surface`,
192
+ Node.js 22/24/26, npm and the Node test runner. The experimental library tarball
193
+ ships built code and fixtures; the CLI entry point remains #16 work.
194
+
195
+ The Researcher instructions, artefact contracts and shared examples remain
196
+ independent of the implementation language. Chat-harness use does not require
197
+ Node.js, and a Lab may consume the shared core without launching the CLI.
198
+ Standalone executables are a later distribution option, with separate platform
199
+ and packaging checks; they are outside the first milestone.
200
+
201
+ ## AI execution and continuation
202
+
203
+ Inside a chat harness, its AI follows the Researcher skill. The CLI needs a
204
+ supported installed agent or configured model connection. One verified runner
205
+ is enough for the first milestone. The [selected first adapter](implementation-decisions.md#first-cli-ai-runner)
206
+ uses installed Codex with ChatGPT sign-in; its integration follows the host-native
207
+ skill and Lab trials.
208
+ The TypeScript/Node.js decision does not select an AI provider. A CLI that only
209
+ prints a prompt is not completed authoring.
210
+
211
+ Document authentication, network/model use, source transmission and filesystem
212
+ effects. Do not silently change runner, billing account or source destination.
213
+ Human questions and corrections require continuation with the current record,
214
+ source scope, decisions and unresolved work. Preserve evidence without requiring
215
+ the entire corpus or conversation history in every model context.
216
+
217
+ ## Optional coordination with n8r-labs
218
+
219
+ Users can produce a file independently, send selected work to a Lab, or later
220
+ choose a supported Lab Researcher endpoint as their runner. Runner selection and
221
+ result destination stay independent. Missing integration must not disable
222
+ standalone authoring.
223
+
224
+ Begin with explicit exchange of selected descriptions, source references and
225
+ unresolved decisions. Preserve identity and revision; do not import unrelated
226
+ Lab history or silently merge changes. A source path requires an actual attachment
227
+ or explicit transfer before a remote runner can inspect it.
228
+
229
+ This repository owns consumer-side integration. The separate application project
230
+ owns its assistant, source attachments, sessions, permissions, Lab persistence
231
+ and API/MCP endpoints. A research-execution endpoint is distinct from file import.
232
+ MCP is a possible transport, not inference or the discovery instructions.
233
+
234
+ Claim compatibility only after testing an identified consumer version and subset,
235
+ including an intentionally incomplete description. Automatic synchronisation,
236
+ deployment and operational Lab management remain outside the first path.
237
+
238
+ ### Compatibility contract
239
+
240
+ Define the boundary in #15 and keep it with the shared core as it develops:
241
+
242
+ - Version the Researcher instructions, research-record contract, Surface contract
243
+ and vocabulary references. A host identifies the versions it supports and
244
+ reports unsupported versions or losses explicitly.
245
+ - Describe the capabilities a host supplies: scoped source inspection, inference,
246
+ human questions, artefact read/write and continuation. Keep the core independent
247
+ of CLI paths, a specific provider, application sessions and MCP transport.
248
+ - Preserve record/component identity, revisions, evidence references, coverage,
249
+ selected possibilities, human decisions and unresolved questions across hosts.
250
+ A Lab can store artefacts differently while preserving these meanings.
251
+ - Keep portable continuation context in the record: what was inspected, decided
252
+ and left unresolved. A new host must establish source access and permissions;
253
+ a saved local path or an old chat transcript is not sufficient.
254
+ - Maintain shared synthetic acceptance examples for two unrelated products,
255
+ incomplete/conflicting evidence, manual edits and continuation with a second
256
+ source. CLI, external skill and later application tests use these cases.
257
+
258
+ The #15 review considered this boundary against n8r-labs requirements and
259
+ recorded gaps and decisions without a parallel application build.
260
+ Changes to the boundary update the contract and examples in the same PR; identify
261
+ any later application adaptation or migration they require.
262
+
263
+ After the first skill trial, the application project implements its adapters
264
+ and uses the shared instruction revision. Verify both shared discovery behaviour
265
+ and import/continuation of standalone artefacts against a named application
266
+ version. Compare supported facts, choices and gaps, not generated wording.
267
+ The standalone milestone and the verified application integration are separate
268
+ completion points.
269
+
270
+ ## Open choices and verification
271
+
272
+ [#15](https://github.com/injectionator/n8r-surface/issues/15) now specifies the
273
+ [HTML/data ownership, revision, graph and codec contracts](contracts.md),
274
+ [host capabilities and compatibility gaps](host-contract.md), and
275
+ [runtime, tooling and first-runner decisions](implementation-decisions.md).
276
+ Exact tool names, corpus layout and source-reader limits follow in #16; real AI
277
+ behavior, instruction packaging and model verification follow in #17.
278
+
279
+ Use synthetic unrelated products, incomplete/conflicting evidence, human edits
280
+ and a second source. Check local integrity, safe persistence and actual AI
281
+ behaviour separately. Compare supported facts, selections and gaps across hosts,
282
+ rather than requiring identical generated wording. A live consumer trial is
283
+ separate from file parsing, fixture checks and a successful standalone run.
284
+
285
+ Before implementing command behavior, define and attempt the corresponding
286
+ [user-outcome case](command-outcomes.md#governing-development-loop), or record why
287
+ the current product cannot run it. After implementation, repeat the actual command
288
+ and human interaction, then inspect the persisted output. Do not use the developing
289
+ agent's own research or a prewritten artefact as proof of Researcher behavior.