@zsailer/jupyter-server-side-documents-conformance 0.0.3 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -7
- package/bin/args.mjs +29 -23
- package/bin/conformance.mjs +8 -8
- package/bin/picker.mjs +1 -1
- package/bin/welcome.mjs +6 -6
- package/package.json +2 -2
- package/runtime/reporter.ts +1 -1
package/README.md
CHANGED
|
@@ -58,10 +58,10 @@ The suite takes nothing from a backend's repo. For jupyter-server-documents it b
|
|
|
58
58
|
## Running it
|
|
59
59
|
|
|
60
60
|
```sh
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
61
|
+
jupyter conformance ssd run # start jupyter-server-documents, run every story, stop it
|
|
62
|
+
jupyter conformance ssd run Z3 C2 # just these
|
|
63
|
+
jupyter conformance ssd run --url http://localhost:8888 # a server that is already running
|
|
64
|
+
jupyter conformance ssd run --start "my-server --port {port}" # your own command that starts one
|
|
65
65
|
```
|
|
66
66
|
|
|
67
67
|
The command is `bin/conformance.mjs` (`node compliance/bin/conformance.mjs` in a checkout); see
|
|
@@ -93,11 +93,11 @@ the backend's side of the contract.
|
|
|
93
93
|
|
|
94
94
|
## Reading a run
|
|
95
95
|
|
|
96
|
-
`
|
|
96
|
+
`jupyter conformance ssd run` ends with a summary; the same is available after any run with `python3 -I compliance/scripts/summarize.py`: one line per story (passed, or failed with the first line of why), what was skipped and what each skip lacked, and, for every story that is still an open question, **which of the storyboard's options the backend does**. A failing step names the storyboard step it was in, and every open tab is photographed at that moment.
|
|
97
97
|
|
|
98
98
|
## Watching a story
|
|
99
99
|
|
|
100
|
-
A story happens in several browsers at once, and Playwright films only the page it makes itself. So the tests film every tab (`runtime/recording.ts`) and write, next to the results of each story, **`watch.html`**: the videos of all the tabs side by side, on one clock, with the storyboard step shown underneath and a list of the steps. It is also attached to the report. `
|
|
100
|
+
A story happens in several browsers at once, and Playwright films only the page it makes itself. So the tests film every tab (`runtime/recording.ts`) and write, next to the results of each story, **`watch.html`**: the videos of all the tabs side by side, on one clock, with the storyboard step shown underneath and a list of the steps. It is also attached to the report. `jupyter conformance ssd run` prints where the pages of failed stories are, and `--open` opens them.
|
|
101
101
|
|
|
102
102
|
## Going offline
|
|
103
103
|
|
|
@@ -132,7 +132,7 @@ compliance/
|
|
|
132
132
|
runtime/ Tab, Story, the matcher, the adapter interface
|
|
133
133
|
adapters/ http (the default: speaks the control contract), generic
|
|
134
134
|
server/ what the tests bring to a Jupyter Server backend: its config, and the jsd side of the contract
|
|
135
|
-
bin/ the `
|
|
135
|
+
bin/ the `jupyter conformance ssd` command
|
|
136
136
|
known-gaps/ accepted stories a backend is known not to meet yet
|
|
137
137
|
tests/generated/ GENERATED: one spec per group of stories. Do not edit.
|
|
138
138
|
unit/ tests of the matcher (no browser)
|
package/bin/args.mjs
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
|
-
// Argument parsing and help for the `
|
|
1
|
+
// Argument parsing and help for the `jupyter conformance ssd` command. No dependencies, and nothing here touches the machine.
|
|
2
2
|
|
|
3
3
|
// Help is written to fit an ordinary terminal: no line is longer than 78 characters (a test keeps it so).
|
|
4
|
-
const MAIN = `
|
|
4
|
+
const MAIN = `jupyter conformance ssd: hold a backend to what people should see
|
|
5
5
|
|
|
6
6
|
What it does
|
|
7
7
|
Starts your server (or uses one that is running), opens real browsers and
|
|
@@ -11,9 +11,9 @@ What it does
|
|
|
11
11
|
doesn't.
|
|
12
12
|
|
|
13
13
|
Start here
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
14
|
+
jupyter conformance ssd pick choose stories from a list, then run them
|
|
15
|
+
jupyter conformance ssd run Z3 run one story
|
|
16
|
+
jupyter conformance ssd run run every story (takes a while)
|
|
17
17
|
|
|
18
18
|
Commands
|
|
19
19
|
run [story ...] run the stories named, or all of them
|
|
@@ -22,21 +22,21 @@ Commands
|
|
|
22
22
|
help [command] more about a command, with examples
|
|
23
23
|
|
|
24
24
|
Learn more
|
|
25
|
-
|
|
25
|
+
jupyter conformance ssd help run every option for running, and examples
|
|
26
26
|
docs/running-against-a-backend.md a longer guide
|
|
27
27
|
`;
|
|
28
28
|
|
|
29
|
-
const RUN = `
|
|
29
|
+
const RUN = `jupyter conformance ssd run: run the stories against a backend
|
|
30
30
|
|
|
31
31
|
Usage
|
|
32
|
-
|
|
32
|
+
jupyter conformance ssd run [story ...] [options]
|
|
33
33
|
|
|
34
34
|
Which stories
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
35
|
+
jupyter conformance ssd run every story
|
|
36
|
+
jupyter conformance ssd run Z3 C2 these two
|
|
37
|
+
jupyter conformance ssd run 'AI*' every story whose id starts AI
|
|
38
|
+
jupyter conformance ssd run --group edges one group (see \`list\`)
|
|
39
|
+
jupyter conformance ssd run -i pick them from a list first
|
|
40
40
|
|
|
41
41
|
Which server
|
|
42
42
|
--backend <name> how to start the server: jsd (default), the built-in
|
|
@@ -62,10 +62,10 @@ What you see
|
|
|
62
62
|
(default ./conformance-known-gaps.json)
|
|
63
63
|
|
|
64
64
|
Examples
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
65
|
+
jupyter conformance ssd run Z3 --env ~/envs/jsd
|
|
66
|
+
jupyter conformance ssd run --group handshake --headed
|
|
67
|
+
jupyter conformance ssd run --url http://localhost:8888
|
|
68
|
+
jupyter conformance ssd run --start "my-server --port {port}"
|
|
69
69
|
|
|
70
70
|
A run ends with a line per story and a summary. The exit code is 0 when no
|
|
71
71
|
story failed, so it works as a CI step.
|
|
@@ -77,11 +77,11 @@ story failed, so it works as a CI step.
|
|
|
77
77
|
what
|
|
78
78
|
`;
|
|
79
79
|
|
|
80
|
-
const PICK = `
|
|
80
|
+
const PICK = `jupyter conformance ssd pick: choose stories from a list, then run them
|
|
81
81
|
|
|
82
82
|
Usage
|
|
83
|
-
|
|
84
|
-
It takes the same options as \`
|
|
83
|
+
jupyter conformance ssd pick [options]
|
|
84
|
+
It takes the same options as \`jupyter conformance ssd run\`.
|
|
85
85
|
|
|
86
86
|
In the list
|
|
87
87
|
↑ ↓ (or j k) move
|
|
@@ -96,13 +96,13 @@ In the list
|
|
|
96
96
|
q or escape leave without running
|
|
97
97
|
|
|
98
98
|
Needs a terminal. In a script, name the stories:
|
|
99
|
-
|
|
99
|
+
jupyter conformance ssd run Z3 C2
|
|
100
100
|
`;
|
|
101
101
|
|
|
102
|
-
const LIST = `
|
|
102
|
+
const LIST = `jupyter conformance ssd list: every story, and what each needs
|
|
103
103
|
|
|
104
104
|
Usage
|
|
105
|
-
|
|
105
|
+
jupyter conformance ssd list [--group <name>] [--filter <text>]
|
|
106
106
|
|
|
107
107
|
--group <name> only the stories of one group
|
|
108
108
|
--filter <text> only the stories whose id, title or needs contain the text
|
|
@@ -113,6 +113,8 @@ server let go of a notebook.
|
|
|
113
113
|
`;
|
|
114
114
|
|
|
115
115
|
const TOPICS = { main: MAIN, run: RUN, pick: PICK, list: LIST };
|
|
116
|
+
/** The suite this command runs: the real-time collaboration of server-side documents. */
|
|
117
|
+
export const SUITE = 'ssd';
|
|
116
118
|
export const COMMANDS = ['run', 'pick', 'list', 'help'];
|
|
117
119
|
|
|
118
120
|
export function helpFor(topic = 'main') {
|
|
@@ -169,6 +171,10 @@ export function parse(argv) {
|
|
|
169
171
|
positional.push(arg);
|
|
170
172
|
}
|
|
171
173
|
}
|
|
174
|
+
// `jupyter conformance ssd run`: the first word names the suite (the only one so far). `jupyter conformance run` is the same
|
|
175
|
+
if (positional[0] === SUITE) {
|
|
176
|
+
positional.shift();
|
|
177
|
+
}
|
|
172
178
|
if (positional.length === 0 && !options.help && !options.version) {
|
|
173
179
|
return { command: 'welcome', stories: [], options, topic: 'main' };
|
|
174
180
|
}
|
package/bin/conformance.mjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
// `
|
|
2
|
+
// `jupyter conformance ssd`: run the real-time collaboration conformance tests against a backend. See --help.
|
|
3
3
|
import { spawn, spawnSync } from 'node:child_process';
|
|
4
4
|
import fs from 'node:fs';
|
|
5
5
|
import os from 'node:os';
|
|
@@ -92,13 +92,13 @@ function list(suite, options) {
|
|
|
92
92
|
}
|
|
93
93
|
const free = stories.filter(s => !s.internal && !s.requires.length).length;
|
|
94
94
|
console.log(`\n ${dim(`${stories.length} stories · ${free} need nothing special from a backend`)}`);
|
|
95
|
-
console.log(` ${dim('
|
|
95
|
+
console.log(` ${dim('jupyter conformance ssd pick to choose some · jupyter conformance ssd run <id> to run one')}\n`);
|
|
96
96
|
}
|
|
97
97
|
|
|
98
98
|
/** The interactive way in: a list to choose from. Resolves with the ids chosen, or exits if the person leaves. */
|
|
99
99
|
async function choose(suite) {
|
|
100
100
|
if (!process.stdin.isTTY || !process.stdout.isTTY) {
|
|
101
|
-
fail('Picking stories needs a terminal', 'Name the stories instead:\n
|
|
101
|
+
fail('Picking stories needs a terminal', 'Name the stories instead:\n jupyter conformance ssd run Z3 C2\nor see them with:\n jupyter conformance ssd list');
|
|
102
102
|
}
|
|
103
103
|
const chosen = await pick(loadStories(suite));
|
|
104
104
|
if (!chosen) {
|
|
@@ -141,7 +141,7 @@ async function startServer(suite, options, port, out) {
|
|
|
141
141
|
} else {
|
|
142
142
|
const pythonEnv = options.env ?? process.env.JSD_ENV ?? findPythonEnv();
|
|
143
143
|
if (!pythonEnv) {
|
|
144
|
-
fail("Can't find a Python environment with jupyter_server_documents", 'Run this inside the environment you develop it in, or point at one:\n
|
|
144
|
+
fail("Can't find a Python environment with jupyter_server_documents", 'Run this inside the environment you develop it in, or point at one:\n jupyter conformance ssd run --env /path/to/env');
|
|
145
145
|
}
|
|
146
146
|
rootDir ??= path.join(os.tmpdir(), 'conformance-root');
|
|
147
147
|
fs.rmSync(rootDir, { recursive: true, force: true });
|
|
@@ -215,17 +215,17 @@ async function run(suite, stories, options) {
|
|
|
215
215
|
const port = Number(process.env.COMPLIANCE_PORT ?? 8899);
|
|
216
216
|
let baseURL = options.url?.replace(/\/$/, '');
|
|
217
217
|
|
|
218
|
-
console.log(`\n ${bold('
|
|
218
|
+
console.log(`\n ${bold('jupyter conformance ssd')} ${dim(`v${VERSION}`)}`);
|
|
219
219
|
|
|
220
220
|
let server;
|
|
221
221
|
let rootDir = options['root-dir'];
|
|
222
222
|
if (!baseURL) {
|
|
223
223
|
if (!options.start && backend !== 'jsd') {
|
|
224
|
-
fail(`Don't know how to start "${backend}"`, 'Give a command that starts it, or a server that is already running:\n
|
|
224
|
+
fail(`Don't know how to start "${backend}"`, 'Give a command that starts it, or a server that is already running:\n jupyter conformance ssd run --start "<command that serves JupyterLab on {port}>"\n jupyter conformance ssd run --url http://localhost:8888');
|
|
225
225
|
}
|
|
226
226
|
baseURL = `http://localhost:${port}`;
|
|
227
227
|
if (await serving(baseURL)) {
|
|
228
|
-
fail(`Something is already serving ${baseURL}`, 'Stop it, or run the tests against it:\n
|
|
228
|
+
fail(`Something is already serving ${baseURL}`, 'Stop it, or run the tests against it:\n jupyter conformance ssd run --url ' + baseURL);
|
|
229
229
|
}
|
|
230
230
|
console.log(` ${dim(options.start ? 'your server' : backend)}\n`);
|
|
231
231
|
const started = await startServer(suite, options, port, out);
|
|
@@ -284,7 +284,7 @@ let parsed;
|
|
|
284
284
|
try {
|
|
285
285
|
parsed = parse(process.argv.slice(2));
|
|
286
286
|
} catch (error) {
|
|
287
|
-
fail(error.message, '
|
|
287
|
+
fail(error.message, 'jupyter conformance ssd help');
|
|
288
288
|
}
|
|
289
289
|
const { command, stories, options, topic } = parsed;
|
|
290
290
|
if (command === 'welcome') {
|
package/bin/picker.mjs
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// The interactive story picker of `
|
|
1
|
+
// The interactive story picker of `jupyter conformance ssd pick`: a list grouped by topic, in which a person selects several stories.
|
|
2
2
|
// The behaviour is pure functions (createPicker, reduce, render), so it is tested without a terminal; `pick` is the thin
|
|
3
3
|
// part that reads keys and draws.
|
|
4
4
|
import readline from 'node:readline';
|
package/bin/welcome.mjs
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// The welcome screen of `
|
|
1
|
+
// The welcome screen of `jupyter conformance ssd`: what is shown when it is run with nothing to do. A pure function of the
|
|
2
2
|
// version and whether to use colour, so it is tested without a terminal.
|
|
3
3
|
|
|
4
4
|
// The Jupyter logo as an outline in braille (two crescents and three small circles), drawn by scripts/draw-logo.py:
|
|
@@ -68,14 +68,14 @@ export function welcome({ version, color = true } = {}) {
|
|
|
68
68
|
// the name sits inside the arcs, like a wordmark
|
|
69
69
|
const art = logo({
|
|
70
70
|
color,
|
|
71
|
-
labels: { 5: { text: '
|
|
71
|
+
labels: { 5: { text: 'conformance', paint: bold }, 6: { text: `v${version}`, paint: dim } }
|
|
72
72
|
}).map(row => ` ${row}`);
|
|
73
73
|
|
|
74
74
|
const commands = [
|
|
75
|
-
['
|
|
76
|
-
['
|
|
77
|
-
['
|
|
78
|
-
['
|
|
75
|
+
['jupyter conformance ssd pick', 'choose stories from a list, then run them'],
|
|
76
|
+
['jupyter conformance ssd run Z3', 'run one story'],
|
|
77
|
+
['jupyter conformance ssd list', 'see every story, and what each needs'],
|
|
78
|
+
['jupyter conformance ssd help', 'everything it can do']
|
|
79
79
|
];
|
|
80
80
|
const width = Math.max(...commands.map(([c]) => c.length)) + 2;
|
|
81
81
|
|
package/package.json
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zsailer/jupyter-server-side-documents-conformance",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.5",
|
|
4
4
|
"description": "Compliance tests for real-time collaboration backends in JupyterLab, generated from the storyboards.",
|
|
5
5
|
"bin": {
|
|
6
|
-
"
|
|
6
|
+
"jupyter-conformance": "bin/conformance.mjs"
|
|
7
7
|
},
|
|
8
8
|
"license": "BSD-3-Clause",
|
|
9
9
|
"repository": {
|
package/runtime/reporter.ts
CHANGED
|
@@ -2,7 +2,7 @@ import type { FullResult, Reporter, TestCase, TestResult } from '@playwright/tes
|
|
|
2
2
|
import { rowLines, rowOf, Row, Style, summaryLines } from './report';
|
|
3
3
|
|
|
4
4
|
/**
|
|
5
|
-
* The reporter of the `
|
|
5
|
+
* The reporter of the `jupyter conformance ssd` command: a line per story as it finishes (skipped ones are only counted), a spinner
|
|
6
6
|
* line for the story running now, and a summary. It is not for CI logs that want the full detail: use Playwright's own
|
|
7
7
|
* reporters for that.
|
|
8
8
|
*/
|