solvewright 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {solvewright-0.1.0 → solvewright-0.1.2}/PKG-INFO +14 -6
- {solvewright-0.1.0 → solvewright-0.1.2}/README.md +13 -5
- {solvewright-0.1.0 → solvewright-0.1.2}/pyproject.toml +1 -1
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/__init__.py +1 -1
- solvewright-0.1.2/src/solvewright/activity.py +180 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/agents.py +77 -18
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/cli.py +2 -2
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/connection.py +27 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/interviewer.txt +5 -3
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/references/case-generation.txt +4 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/runtime.py +227 -24
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/worker.py +8 -3
- solvewright-0.1.2/tests/test_activity.py +159 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/tests/test_connection.py +17 -0
- solvewright-0.1.2/tests/test_turns.py +149 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/.gitignore +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/LICENSE +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/__main__.py +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/cases/portals/brief.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/cases/portals/interviewer.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/cases/portals/reported-round.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-01.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-02.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-03.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-04.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-05.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-06.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-07.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/data/portal-samples/sample-08.json +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/references/curriculum.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/references/scoring.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/references/session-format.txt +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/tools/portal_lab.py +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/service.py +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/settings.py +0 -0
- {solvewright-0.1.0 → solvewright-0.1.2}/tests/test_runtime.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: solvewright
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Connect your own Claude Code or Codex to Solvewright Interview Lab for mock technical interviews.
|
|
5
5
|
Project-URL: Homepage, https://solvewright.app/coding/interviews
|
|
6
6
|
Project-URL: Source, https://github.com/fus3r/solvewright-connector
|
|
@@ -44,8 +44,10 @@ then `solvewright`. Python 3.10 or newer. The package has no dependencies.
|
|
|
44
44
|
line for each thing it does (a round prepared, a reply, tests recorded).
|
|
45
45
|
4. Press Ctrl+C to stop. Nothing keeps running in the background, and nothing starts again by itself.
|
|
46
46
|
|
|
47
|
-
Next time,
|
|
48
|
-
|
|
47
|
+
Next time, run the same command again: it reconnects without a new code, and the terminal reminds you
|
|
48
|
+
which command that is. `uvx` and `pipx run` keep the package in a cache and add no `solvewright`
|
|
49
|
+
command to your shell, so with them, put `uvx` or `pipx run` in front of the commands below (for
|
|
50
|
+
example `uvx solvewright status`). Removing this computer from Interview Lab stops its key at once.
|
|
49
51
|
|
|
50
52
|
| Command | Does |
|
|
51
53
|
| --- | --- |
|
|
@@ -67,8 +69,14 @@ stop the interviewer, and sync your history. There is no request that runs a com
|
|
|
67
69
|
keeps a hash of it.
|
|
68
70
|
- **It never sees your Claude or ChatGPT credentials.** It starts the official `claude` or `codex`
|
|
69
71
|
program, unmodified, which signs in the way it always does. Usage counts against your own plan.
|
|
70
|
-
- **It never runs your code.** Public tests run in your browser,
|
|
71
|
-
|
|
72
|
+
- **It never runs your code.** Public tests and the interviewer's private checks run in your browser,
|
|
73
|
+
in Pyodide (Python compiled to WebAssembly); the page shows each check's label and result, never its
|
|
74
|
+
code. The connector only stores the files you tested and the results.
|
|
75
|
+
- **It reports progress, not content.** While the interviewer works, the page learns which kind of
|
|
76
|
+
step it is on (reading your code, writing its notes, running its checks), never what it reads,
|
|
77
|
+
writes or says. It also passes on your plan's usage when the agent reports it (Codex's usage
|
|
78
|
+
windows, Claude Code's rate-limit events), read without a model call, so replies can use less
|
|
79
|
+
effort when your window is nearly spent.
|
|
72
80
|
- **It does not update itself.** What runs is the version you installed from PyPI. The interviewer's
|
|
73
81
|
instructions ship inside the package (`src/solvewright/interviewer/`), so you can read exactly what
|
|
74
82
|
your agent is told.
|
|
@@ -81,7 +89,7 @@ It does that inside its own agent's sandbox, confined to the folder of the curre
|
|
|
81
89
|
| | Commands | Files | Network |
|
|
82
90
|
| --- | --- | --- | --- |
|
|
83
91
|
| Claude Code, macOS and Linux | Claude Code's OS sandbox | Read and write only in the round folder; the rest of your home folder is unreadable | None |
|
|
84
|
-
| Codex, macOS and Linux | Codex's sandbox with a Solvewright permission profile | Read and write in your interview folders; the rest of your home folder is unreadable | None |
|
|
92
|
+
| Codex, macOS and Linux | Codex's sandbox with a Solvewright permission profile | Read and write in your interview folders; read the folders Codex itself is installed in; the rest of your home folder is unreadable | None |
|
|
85
93
|
| Claude Code, native Windows | No command tool at all: its sandbox needs macOS, Linux or WSL2 | The file tools stay in the round folder | None |
|
|
86
94
|
|
|
87
95
|
Claude Code runs with `--restricted --safe-mode --strict-mcp-config`: your own CLAUDE.md, hooks,
|
|
@@ -21,8 +21,10 @@ then `solvewright`. Python 3.10 or newer. The package has no dependencies.
|
|
|
21
21
|
line for each thing it does (a round prepared, a reply, tests recorded).
|
|
22
22
|
4. Press Ctrl+C to stop. Nothing keeps running in the background, and nothing starts again by itself.
|
|
23
23
|
|
|
24
|
-
Next time,
|
|
25
|
-
|
|
24
|
+
Next time, run the same command again: it reconnects without a new code, and the terminal reminds you
|
|
25
|
+
which command that is. `uvx` and `pipx run` keep the package in a cache and add no `solvewright`
|
|
26
|
+
command to your shell, so with them, put `uvx` or `pipx run` in front of the commands below (for
|
|
27
|
+
example `uvx solvewright status`). Removing this computer from Interview Lab stops its key at once.
|
|
26
28
|
|
|
27
29
|
| Command | Does |
|
|
28
30
|
| --- | --- |
|
|
@@ -44,8 +46,14 @@ stop the interviewer, and sync your history. There is no request that runs a com
|
|
|
44
46
|
keeps a hash of it.
|
|
45
47
|
- **It never sees your Claude or ChatGPT credentials.** It starts the official `claude` or `codex`
|
|
46
48
|
program, unmodified, which signs in the way it always does. Usage counts against your own plan.
|
|
47
|
-
- **It never runs your code.** Public tests run in your browser,
|
|
48
|
-
|
|
49
|
+
- **It never runs your code.** Public tests and the interviewer's private checks run in your browser,
|
|
50
|
+
in Pyodide (Python compiled to WebAssembly); the page shows each check's label and result, never its
|
|
51
|
+
code. The connector only stores the files you tested and the results.
|
|
52
|
+
- **It reports progress, not content.** While the interviewer works, the page learns which kind of
|
|
53
|
+
step it is on (reading your code, writing its notes, running its checks), never what it reads,
|
|
54
|
+
writes or says. It also passes on your plan's usage when the agent reports it (Codex's usage
|
|
55
|
+
windows, Claude Code's rate-limit events), read without a model call, so replies can use less
|
|
56
|
+
effort when your window is nearly spent.
|
|
49
57
|
- **It does not update itself.** What runs is the version you installed from PyPI. The interviewer's
|
|
50
58
|
instructions ship inside the package (`src/solvewright/interviewer/`), so you can read exactly what
|
|
51
59
|
your agent is told.
|
|
@@ -58,7 +66,7 @@ It does that inside its own agent's sandbox, confined to the folder of the curre
|
|
|
58
66
|
| | Commands | Files | Network |
|
|
59
67
|
| --- | --- | --- | --- |
|
|
60
68
|
| Claude Code, macOS and Linux | Claude Code's OS sandbox | Read and write only in the round folder; the rest of your home folder is unreadable | None |
|
|
61
|
-
| Codex, macOS and Linux | Codex's sandbox with a Solvewright permission profile | Read and write in your interview folders; the rest of your home folder is unreadable | None |
|
|
69
|
+
| Codex, macOS and Linux | Codex's sandbox with a Solvewright permission profile | Read and write in your interview folders; read the folders Codex itself is installed in; the rest of your home folder is unreadable | None |
|
|
62
70
|
| Claude Code, native Windows | No command tool at all: its sandbox needs macOS, Linux or WSL2 | The file tools stay in the round folder | None |
|
|
63
71
|
|
|
64
72
|
Claude Code runs with `--restricted --safe-mode --strict-mcp-config`: your own CLAUDE.md, hooks,
|
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "solvewright"
|
|
7
|
-
version = "0.1.
|
|
7
|
+
version = "0.1.2"
|
|
8
8
|
description = "Connect your own Claude Code or Codex to Solvewright Interview Lab for mock technical interviews."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = "MIT"
|
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
"""What the examiner is doing during a turn, read from the agent's own event stream.
|
|
2
|
+
|
|
3
|
+
The page shows it while the candidate waits. Only the kind of action and the
|
|
4
|
+
folder it touches are kept: reading the brief, reading the candidate's code,
|
|
5
|
+
writing private notes, running checks, writing the candidate's files. File
|
|
6
|
+
contents, command text and the examiner's own words never leave this module,
|
|
7
|
+
so nothing private (a reference, a later part) can reach the candidate.
|
|
8
|
+
"""
|
|
9
|
+
from __future__ import annotations
|
|
10
|
+
|
|
11
|
+
import re
|
|
12
|
+
import time
|
|
13
|
+
from datetime import datetime, timezone
|
|
14
|
+
|
|
15
|
+
# The steps each kind of turn is expected to pass through, in order.
|
|
16
|
+
STEPS = {
|
|
17
|
+
"prepare": ("brief", "design", "reference", "checks", "files", "reply"),
|
|
18
|
+
"reply": ("read", "checks", "reply"),
|
|
19
|
+
"mark": ("read", "checks", "report", "reply"),
|
|
20
|
+
}
|
|
21
|
+
LINES = {
|
|
22
|
+
"brief": "Reading the brief and your past rounds",
|
|
23
|
+
"design": "Designing the problem",
|
|
24
|
+
"reference": "Writing a private reference solution",
|
|
25
|
+
"checks": "Running its checks",
|
|
26
|
+
"files": "Writing your files",
|
|
27
|
+
"read": "Reading your code",
|
|
28
|
+
"notes": "Updating its notes",
|
|
29
|
+
"report": "Writing your marks",
|
|
30
|
+
"reply": "Writing the reply",
|
|
31
|
+
"think": "Thinking it through",
|
|
32
|
+
}
|
|
33
|
+
FIRST_LINE = {"prepare": LINES["brief"], "reply": LINES["read"], "mark": LINES["read"]}
|
|
34
|
+
|
|
35
|
+
RUNS_PYTHON = re.compile(r"(?:^|[\s;&|(`'\"/])(?:python(?:3(?:\.\d+)?)?|py|pytest|unittest)(?=$|[\s'\";)])")
|
|
36
|
+
READS = re.compile(r"(?:^|[;&|]\s*|\n)\s*(?:cat|sed|head|tail|less|nl|rg|grep|ls|find|tree|wc|stat|diff|pwd)\b(?![^;&|\n]*>)")
|
|
37
|
+
BRIEF = re.compile(r"(?:^|/)(?:history-context\.json|session-settings\.json|context/interviewer(?:/|$))")
|
|
38
|
+
# Where a shell command writes: `> path`, `>> path`, `tee path`. Not 2>&1 or /dev/null.
|
|
39
|
+
WRITES = re.compile(r"(?:(?<![0-9&])>{1,2}|\btee(?:\s+-a)?)\s*['\"]?([A-Za-z0-9_./-]+)")
|
|
40
|
+
MENTIONS = re.compile(r"[A-Za-z0-9_./-]*(?:workspace/|\.examiner/|context/interviewer|history-context\.json|session-settings\.json)[A-Za-z0-9_./-]*")
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def stamp() -> str:
|
|
44
|
+
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def where(path: str, root: str = "") -> str:
|
|
48
|
+
"""The part of the round a path belongs to: brief, workspace, report, private or other."""
|
|
49
|
+
path = str(path or "").replace("\\", "/")
|
|
50
|
+
root = str(root or "").replace("\\", "/").rstrip("/")
|
|
51
|
+
if root and path.startswith(root + "/"):
|
|
52
|
+
path = path[len(root) + 1:]
|
|
53
|
+
while path.startswith("./"):
|
|
54
|
+
path = path[2:]
|
|
55
|
+
if BRIEF.search(path):
|
|
56
|
+
return "brief"
|
|
57
|
+
if path.startswith("workspace/") or path == "workspace":
|
|
58
|
+
return "workspace"
|
|
59
|
+
if path.endswith(".examiner/report.json") or path == ".examiner/report.json":
|
|
60
|
+
return "report"
|
|
61
|
+
if path.startswith(".examiner/") or "/.examiner/" in path:
|
|
62
|
+
return "private"
|
|
63
|
+
return "other"
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
Action = tuple[str, list[str], str]
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
def command_actions(command: str, listed=()) -> list[Action]:
|
|
70
|
+
"""What one shell command does, in order: reads, writes (by their targets), then a Python run.
|
|
71
|
+
|
|
72
|
+
Examiners write files with heredocs and scripts as often as with patch tools,
|
|
73
|
+
and read several files in one compound command, so the command's own text is
|
|
74
|
+
read for paths and redirections. Only their folders are kept.
|
|
75
|
+
"""
|
|
76
|
+
command = str(command or "")
|
|
77
|
+
listed = [action for action in listed or [] if isinstance(action, dict)]
|
|
78
|
+
targets = [target for target in WRITES.findall(command) if target != "/dev/null"]
|
|
79
|
+
mentioned = [path for path in MENTIONS.findall(command) if path not in targets]
|
|
80
|
+
actions: list[Action] = []
|
|
81
|
+
if READS.search(command) or any(action.get("type") in ("read", "listFiles", "search") for action in listed):
|
|
82
|
+
actions.append(("read", [str(action.get("path") or "") for action in listed] + mentioned, command))
|
|
83
|
+
if targets:
|
|
84
|
+
actions.append(("write", targets, command))
|
|
85
|
+
if RUNS_PYTHON.search(command):
|
|
86
|
+
actions.append(("run", mentioned, command))
|
|
87
|
+
return actions
|
|
88
|
+
|
|
89
|
+
|
|
90
|
+
def codex_action(item: dict) -> list[Action]:
|
|
91
|
+
"""The actions in one Codex app-server ThreadItem; an empty list to ignore it."""
|
|
92
|
+
kind = item.get("type")
|
|
93
|
+
if kind == "reasoning":
|
|
94
|
+
return [("think", [], "")]
|
|
95
|
+
if kind == "agentMessage":
|
|
96
|
+
return [("answer", [], "")] if item.get("phase") == "final_answer" else []
|
|
97
|
+
if kind == "fileChange":
|
|
98
|
+
return [("write", [change.get("path", "") for change in item.get("changes") or [] if isinstance(change, dict)], "")]
|
|
99
|
+
if kind == "commandExecution":
|
|
100
|
+
return command_actions(item.get("command"), item.get("commandActions"))
|
|
101
|
+
return []
|
|
102
|
+
|
|
103
|
+
|
|
104
|
+
def claude_action(block: dict) -> list[Action]:
|
|
105
|
+
"""The actions in one content block of a Claude Code stream-json assistant event."""
|
|
106
|
+
kind = block.get("type")
|
|
107
|
+
if kind in ("thinking", "redacted_thinking"):
|
|
108
|
+
return [("think", [], "")]
|
|
109
|
+
if kind != "tool_use":
|
|
110
|
+
return []
|
|
111
|
+
name, given = block.get("name"), block.get("input") or {}
|
|
112
|
+
if name == "StructuredOutput":
|
|
113
|
+
return [("answer", [], "")]
|
|
114
|
+
if name in ("Write", "Edit", "MultiEdit", "NotebookEdit"):
|
|
115
|
+
return [("write", [str(given.get("file_path") or given.get("notebook_path") or "")], "")]
|
|
116
|
+
if name in ("Read", "Glob", "Grep", "LS"):
|
|
117
|
+
return [("read", [str(given.get("file_path") or given.get("path") or "")], "")]
|
|
118
|
+
if name == "Bash":
|
|
119
|
+
return command_actions(given.get("command"))
|
|
120
|
+
return []
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
class TurnProgress:
|
|
124
|
+
"""The steps of one turn as the page sees them: `state["turn"]`, and `state["activity"]` for older pages."""
|
|
125
|
+
|
|
126
|
+
def __init__(self, kind: str, state: dict, save, root: str = "", clock=time.monotonic, min_gap: float = 0.5):
|
|
127
|
+
self.kind, self.state, self.save, self.root, self.clock, self.min_gap = kind, state, save, root, clock, min_gap
|
|
128
|
+
self.data = {"kind": kind, "since": stamp(), "steps": [], "checks": 0}
|
|
129
|
+
self.last_save = None
|
|
130
|
+
state["turn"] = self.data
|
|
131
|
+
state["activity"] = FIRST_LINE[kind]
|
|
132
|
+
|
|
133
|
+
def step_for(self, action: str, places: set[str]) -> str | None:
|
|
134
|
+
if action == "answer":
|
|
135
|
+
return "reply"
|
|
136
|
+
if action == "run":
|
|
137
|
+
return "checks"
|
|
138
|
+
if action == "think":
|
|
139
|
+
# Before it has written anything, a preparing examiner is designing the problem.
|
|
140
|
+
return "design" if self.kind == "prepare" and not ({"reference", "files"} & self.seen()) else None
|
|
141
|
+
if action == "read":
|
|
142
|
+
if self.kind == "prepare":
|
|
143
|
+
return "brief" if ("brief" in places or not self.seen()) else None
|
|
144
|
+
return "read"
|
|
145
|
+
if action == "write":
|
|
146
|
+
if "report" in places:
|
|
147
|
+
return "report"
|
|
148
|
+
if "workspace" in places:
|
|
149
|
+
return "files"
|
|
150
|
+
if "private" in places:
|
|
151
|
+
return "reference" if self.kind == "prepare" else "notes"
|
|
152
|
+
return None
|
|
153
|
+
|
|
154
|
+
def seen(self) -> set[str]:
|
|
155
|
+
return {step["id"] for step in self.data["steps"]}
|
|
156
|
+
|
|
157
|
+
def observe(self, actions) -> None:
|
|
158
|
+
for action in actions or []:
|
|
159
|
+
self.observe_one(action)
|
|
160
|
+
|
|
161
|
+
def observe_one(self, action: Action) -> None:
|
|
162
|
+
kind, paths, _command = action
|
|
163
|
+
step = self.step_for(kind, {where(path, self.root) for path in paths})
|
|
164
|
+
if not step:
|
|
165
|
+
return
|
|
166
|
+
new = step not in self.seen()
|
|
167
|
+
if new:
|
|
168
|
+
self.data["steps"].append({"id": step, "at": stamp()})
|
|
169
|
+
if step == "checks":
|
|
170
|
+
self.data["checks"] += 1
|
|
171
|
+
line = LINES[step]
|
|
172
|
+
changed = new or line != self.state.get("activity") or step == "checks"
|
|
173
|
+
self.state["activity"] = line
|
|
174
|
+
now = self.clock()
|
|
175
|
+
if changed and (new or self.last_save is None or now - self.last_save >= self.min_gap):
|
|
176
|
+
self.last_save = now
|
|
177
|
+
self.save()
|
|
178
|
+
|
|
179
|
+
def finish(self) -> None:
|
|
180
|
+
self.state.pop("turn", None)
|
|
@@ -173,7 +173,7 @@ def interrupt_process(process) -> None:
|
|
|
173
173
|
process.terminate()
|
|
174
174
|
|
|
175
175
|
|
|
176
|
-
def claude_command(binary: str, directory: Path, state: dict, sandbox: bool, read_roots: list[str]) -> list[str]:
|
|
176
|
+
def claude_command(binary: str, directory: Path, state: dict, sandbox: bool, read_roots: list[str], effort: str | None = None) -> list[str]:
|
|
177
177
|
command = [binary, "--print", "--output-format", "stream-json", "--verbose", "--json-schema", json.dumps(REPLY_SCHEMA),
|
|
178
178
|
"--restricted", "--safe-mode", "--strict-mcp-config",
|
|
179
179
|
"--permission-mode", "acceptEdits", "--permission-prompts", "none"]
|
|
@@ -190,7 +190,7 @@ def claude_command(binary: str, directory: Path, state: dict, sandbox: bool, rea
|
|
|
190
190
|
model = state["settings"].get("model", "").strip()
|
|
191
191
|
if model:
|
|
192
192
|
command += ["--model", model]
|
|
193
|
-
effort = state["settings"].get("effort", "").strip()
|
|
193
|
+
effort = (state["settings"].get("effort", "") if effort is None else effort).strip()
|
|
194
194
|
if effort:
|
|
195
195
|
command += ["--effort", effort]
|
|
196
196
|
return command
|
|
@@ -207,6 +207,27 @@ def claude_preamble(sandbox: bool) -> str:
|
|
|
207
207
|
".examiner/manifest.json that the reference was not executed, and say so when it matters.\n\n")
|
|
208
208
|
|
|
209
209
|
|
|
210
|
+
def link_folders(path: Path, limit: int = 40) -> set[Path]:
|
|
211
|
+
"""The folder of every symbolic link met while resolving `path`, one component at a time."""
|
|
212
|
+
folders: set[Path] = set()
|
|
213
|
+
pending = list(Path(os.path.abspath(path)).parts)
|
|
214
|
+
current = Path(pending.pop(0))
|
|
215
|
+
hops = 0
|
|
216
|
+
while pending and hops < limit:
|
|
217
|
+
candidate = current / pending.pop(0)
|
|
218
|
+
if candidate.is_symlink():
|
|
219
|
+
hops += 1
|
|
220
|
+
folders.add(candidate.parent)
|
|
221
|
+
target = Path(os.readlink(candidate))
|
|
222
|
+
if target.is_absolute():
|
|
223
|
+
current, pending = Path(target.parts[0]), list(target.parts[1:]) + pending
|
|
224
|
+
else:
|
|
225
|
+
pending = list(target.parts) + pending
|
|
226
|
+
else:
|
|
227
|
+
current = candidate
|
|
228
|
+
return folders
|
|
229
|
+
|
|
230
|
+
|
|
210
231
|
class CodexClient:
|
|
211
232
|
"""Newline JSON-RPC client for Codex's documented app-server, isolated."""
|
|
212
233
|
|
|
@@ -218,6 +239,10 @@ class CodexClient:
|
|
|
218
239
|
self.lock = threading.Lock()
|
|
219
240
|
self.condition = threading.Condition()
|
|
220
241
|
self.pending, self.finished, self.messages = {}, {}, {}
|
|
242
|
+
# Per thread: what the examiner starts doing, for the page's progress line. Never its content.
|
|
243
|
+
self.watchers: dict = {}
|
|
244
|
+
self.limits: dict | None = None
|
|
245
|
+
self.limits_at = 0.0
|
|
221
246
|
self.sequence = 0
|
|
222
247
|
self.loaded: set[str] = set()
|
|
223
248
|
self.closed = threading.Event()
|
|
@@ -271,7 +296,12 @@ class CodexClient:
|
|
|
271
296
|
if parent.resolve().is_relative_to(home):
|
|
272
297
|
# The release folder, e.g. ~/.codex/packages/standalone/releases/<version>/.
|
|
273
298
|
roots.append(str(parent.parent.resolve() if parent.name == "bin" else parent.resolve()))
|
|
274
|
-
|
|
299
|
+
# Codex starts itself again inside the sandbox (to load a resumed thread's instructions), so every link on
|
|
300
|
+
# the way to the binary must be readable too: ~/.local/bin/codex -> .../standalone/current/bin/codex, and
|
|
301
|
+
# `current` -> releases/<version>. Without the folder holding `current`, resuming a round after the
|
|
302
|
+
# connector restarts fails with "execvp() ... Operation not permitted".
|
|
303
|
+
roots += [str(folder) for folder in link_folders(Path(self.binary)) if folder.resolve().is_relative_to(home)]
|
|
304
|
+
return sorted(set(roots))
|
|
275
305
|
|
|
276
306
|
def _launch(self, overrides: list[str]) -> None:
|
|
277
307
|
self.closed.clear()
|
|
@@ -340,6 +370,18 @@ class CodexClient:
|
|
|
340
370
|
elif "id" in data and "method" in data:
|
|
341
371
|
# No approval is ever granted on the person's behalf.
|
|
342
372
|
self.send({"id": data["id"], "error": {"code": -32601, "message": "Approval is unavailable in Interview Lab."}})
|
|
373
|
+
elif data.get("method") == "item/started":
|
|
374
|
+
params = data["params"]
|
|
375
|
+
watcher = self.watchers.get(params.get("threadId"))
|
|
376
|
+
if watcher:
|
|
377
|
+
try:
|
|
378
|
+
watcher(params.get("item") or {})
|
|
379
|
+
except Exception: # A progress line must never break the turn.
|
|
380
|
+
pass
|
|
381
|
+
elif data.get("method") == "account/rateLimits/updated":
|
|
382
|
+
snapshot = (data.get("params") or {}).get("rateLimits")
|
|
383
|
+
if isinstance(snapshot, dict):
|
|
384
|
+
self.limits, self.limits_at = snapshot, time.monotonic()
|
|
343
385
|
elif data.get("method") == "item/completed":
|
|
344
386
|
params = data["params"]
|
|
345
387
|
item = params.get("item", {})
|
|
@@ -379,7 +421,19 @@ class CodexClient:
|
|
|
379
421
|
params["sandbox"] = "workspace-write"
|
|
380
422
|
return self.rpc("thread/start", params)["thread"]["id"]
|
|
381
423
|
|
|
382
|
-
def
|
|
424
|
+
def rate_limits(self, max_age: float = 60) -> dict | None:
|
|
425
|
+
"""The account's Codex usage windows (5-hour and weekly), read without a model call."""
|
|
426
|
+
if self.limits is not None and time.monotonic() - self.limits_at < max_age:
|
|
427
|
+
return self.limits
|
|
428
|
+
try:
|
|
429
|
+
snapshot = self.rpc("account/rateLimits/read", {}, timeout=10).get("rateLimits")
|
|
430
|
+
except (ValueError, OSError):
|
|
431
|
+
return self.limits
|
|
432
|
+
if isinstance(snapshot, dict):
|
|
433
|
+
self.limits, self.limits_at = snapshot, time.monotonic()
|
|
434
|
+
return self.limits
|
|
435
|
+
|
|
436
|
+
def turn(self, state: dict, directory: Path, prompt: str, callback, cancelled=None, effort=None, watch=None) -> dict:
|
|
383
437
|
if cancelled and cancelled.is_set():
|
|
384
438
|
raise ValueError("The interviewer turn was stopped. Send a message to continue.")
|
|
385
439
|
model = state["settings"].get("model", "").strip()
|
|
@@ -398,24 +452,29 @@ class CodexClient:
|
|
|
398
452
|
raise ValueError("The interviewer turn was stopped. Send a message to continue.")
|
|
399
453
|
params = {"threadId": thread, "cwd": str(directory), "input": [{"type": "text", "text": prompt}],
|
|
400
454
|
"outputSchema": REPLY_SCHEMA, "approvalPolicy": "never"}
|
|
401
|
-
effort = state["settings"].get("effort", "").strip()
|
|
455
|
+
effort = (state["settings"].get("effort", "") if effort is None else effort).strip()
|
|
402
456
|
if effort:
|
|
403
457
|
params["effort"] = effort
|
|
404
458
|
if not self.profile:
|
|
405
459
|
params["sandboxPolicy"] = {"type": "workspaceWrite", "writableRoots": [str(directory)], "networkAccess": False}
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
self.
|
|
418
|
-
|
|
460
|
+
if watch:
|
|
461
|
+
self.watchers[thread] = watch
|
|
462
|
+
try:
|
|
463
|
+
started = self.rpc("turn/start", params)
|
|
464
|
+
turn = started["turn"]["id"]
|
|
465
|
+
state["activeTurn"] = turn
|
|
466
|
+
callback()
|
|
467
|
+
if cancelled and cancelled.is_set():
|
|
468
|
+
self.interrupt(state)
|
|
469
|
+
key = (thread, turn)
|
|
470
|
+
with self.condition:
|
|
471
|
+
while key not in self.finished:
|
|
472
|
+
if self.closed.is_set() or self.process.poll() is not None:
|
|
473
|
+
raise ValueError("Codex stopped. Send another message to resume this round.")
|
|
474
|
+
self.condition.wait(timeout=5)
|
|
475
|
+
completed = self.finished.pop(key)
|
|
476
|
+
finally:
|
|
477
|
+
self.watchers.pop(thread, None)
|
|
419
478
|
if completed["status"] != "completed":
|
|
420
479
|
error = completed.get("error") or {}
|
|
421
480
|
raise ValueError(error.get("message", f"Codex turn {completed['status']}. Your files and conversation were kept."))
|
|
@@ -93,7 +93,7 @@ def header(out: Out, config, label: str, described: dict, account: str | None =
|
|
|
93
93
|
|
|
94
94
|
def approve(out: Out, config, service: Service, label: str, open_browser: bool) -> dict:
|
|
95
95
|
described = agents.describe()
|
|
96
|
-
details = {"connector": __version__, "os": connection.operating_system(),
|
|
96
|
+
details = {"connector": __version__, "os": connection.operating_system(), "launcher": connection.launcher(),
|
|
97
97
|
"python": ".".join(str(part) for part in sys.version_info[:3]),
|
|
98
98
|
"agents": {name: info["version"] for name, info in described.items() if info["available"]}}
|
|
99
99
|
pending = connection.request(service, label, details)
|
|
@@ -133,7 +133,7 @@ def serve(out: Out, config, service: Service, record: dict) -> int:
|
|
|
133
133
|
out.line()
|
|
134
134
|
out.line("Interview Lab is ready: " + out.bold(f"{config.site_url}/coding/interviews"))
|
|
135
135
|
out.line("Keep this window open while you practise. Closing it stops the connector;")
|
|
136
|
-
out.line("nothing keeps running in the background.")
|
|
136
|
+
out.line("nothing keeps running in the background. Next time, run " + out.bold(connection.next_command()) + ": no new code is needed.")
|
|
137
137
|
out.line()
|
|
138
138
|
out.event("Waiting for the Interview Lab")
|
|
139
139
|
if sys.platform == "win32":
|
|
@@ -12,6 +12,7 @@ import json
|
|
|
12
12
|
import os
|
|
13
13
|
import platform
|
|
14
14
|
import re
|
|
15
|
+
import shutil
|
|
15
16
|
import subprocess
|
|
16
17
|
import sys
|
|
17
18
|
import tempfile
|
|
@@ -47,6 +48,32 @@ def computer_name() -> str:
|
|
|
47
48
|
return name[:64] or "This computer"
|
|
48
49
|
|
|
49
50
|
|
|
51
|
+
def launcher() -> str:
|
|
52
|
+
"""How this copy was started, so the terminal and Interview Lab can say what to run next time.
|
|
53
|
+
|
|
54
|
+
uvx and pipx run keep the package in a cache and put no command on the PATH, which is why
|
|
55
|
+
a plain `solvewright` would not be found afterwards."""
|
|
56
|
+
prefix = Path(sys.prefix).as_posix().lower()
|
|
57
|
+
if "/uv/" in prefix and "/archive-v" in prefix:
|
|
58
|
+
return "uvx"
|
|
59
|
+
# `pipx run` keeps throwaway environments in a cache whose place changed between pipx versions;
|
|
60
|
+
# `pipx install` keeps them under venvs/ and puts the command on the PATH.
|
|
61
|
+
if "/pipx/" in prefix and "/venvs/" not in prefix:
|
|
62
|
+
return "pipx"
|
|
63
|
+
return "command" if shutil.which("solvewright") else "module"
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def next_command(kind: str | None = None) -> str:
|
|
67
|
+
kind = kind or launcher()
|
|
68
|
+
if kind == "uvx":
|
|
69
|
+
return "uvx solvewright"
|
|
70
|
+
if kind == "pipx":
|
|
71
|
+
return "pipx run solvewright"
|
|
72
|
+
if kind == "command":
|
|
73
|
+
return "solvewright"
|
|
74
|
+
return ("py" if sys.platform == "win32" else "python3") + " -m solvewright"
|
|
75
|
+
|
|
76
|
+
|
|
50
77
|
def operating_system() -> str:
|
|
51
78
|
if sys.platform == "darwin":
|
|
52
79
|
return "macOS " + (platform.mac_ver()[0] or "").strip()
|
|
@@ -13,7 +13,7 @@ This folder is portable instruction material. Use it in the candidate's chosen a
|
|
|
13
13
|
|
|
14
14
|
Honor settings already supplied. Defaults: **hybrid track, realistic mock, 50 minutes, English, Python, standard difficulty, AI off for the candidate**. Offer 45–60-minute mocks, 10–25-minute drills, and untimed coaching. Infer the programming language from existing code when appropriate.
|
|
15
15
|
|
|
16
|
-
If the user says “start,” begin with these defaults. If the request is substantially unclear, ask at most two short setup questions together. Do not require a diagnostic before every mock.
|
|
16
|
+
If the user says “start,” begin with these defaults. If the request is substantially unclear, ask at most two short setup questions together. Do not require a diagnostic before every mock. Open with the current prompt itself: its first words are the problem. In the live runtime the page already shows the settings, the clock and the `hint`, `skip`, `reveal this part` and `end` controls, so do not restate them. In a standalone session, follow the prompt with one short line naming the settings and those four commands.
|
|
17
17
|
|
|
18
18
|
Read [references/scoring.txt](references/scoring.txt). If the candidate supplies history or a history path, read it before choosing a case; otherwise start with unknown levels. Do not search unrelated personal files for history. A plain “I practised before” is not performance evidence.
|
|
19
19
|
|
|
@@ -59,6 +59,8 @@ New stages may deliberately change a requirement. Say “new requirement,” pre
|
|
|
59
59
|
|
|
60
60
|
Keep ordinary turns short, usually under 180 words excluding a necessary prompt or data. Let the candidate submit full code or a longer explanation. Do not answer your own question, add leading examples after it, or praise vague statements as demonstrated insight.
|
|
61
61
|
|
|
62
|
+
**A passing submission ends the part.** When the candidate's code meets the current part's contract (your checks pass and nothing in the contract is violated), accept it in a sentence or two and move on: release the next part, or, after the last part or in a drill, close the round with the report. Do not ask the candidate to explain or justify code that already works; working, tested code is the evidence for implementation and validation. Ask one short question only when the evidence cannot settle a mark you must give (for example, code that passes but whose approach you cannot tell from reading it), and say plainly that the candidate may skip it. Leave communication unmarked rather than demand an explanation.
|
|
63
|
+
|
|
62
64
|
For an error, first locate a concrete contradiction or a small counterexample; ask the candidate to reason through it. Do not reject correct boolean-only code because a hypothetical future device might have more states. Classify issues as **required for this contract**, **optional improvement**, or **outside this round**. Spend time on material behavior, not naming, design-pattern compliance, or imaginary production risks.
|
|
63
65
|
|
|
64
66
|
Accept a simple scan when sizes make it sufficient. Ask about scale only when scale is part of the active task. An executable design round should reach code and behavior; do not let it dissolve into a survey of distributed-system components.
|
|
@@ -94,9 +96,9 @@ Protect temporal boundaries. Training samples may be inspected completely. Durin
|
|
|
94
96
|
|
|
95
97
|
## Timing and stages
|
|
96
98
|
|
|
97
|
-
For a 50-minute mock, aim for roughly 5 minutes of clarification, 10 of a baseline/design, 20 of coding or investigation, 10 of an extension, and 5 of a defense. These are budget guides, not a script. Reach a usable baseline before chasing sophistication. A 15-minute drill should contain one task, a real check, and a short debrief.
|
|
99
|
+
For a 50-minute mock, aim for roughly 5 minutes of clarification, 10 of a baseline/design, 20 of coding or investigation, 10 of an extension, and 5 of a defense: one short, optional question about a tradeoff or a limit, asked only with time left. These are budget guides, not a script. Reach a usable baseline before chasing sophistication. A 15-minute drill should contain one task, a real check, and a short debrief.
|
|
98
100
|
|
|
99
|
-
Use actual timestamps only when available. Do not treat every chat turn as a fixed number of minutes. If real timing is unavailable, use milestone pacing and say timing is approximate. Pauses and tool waits need not count as candidate work. At the limit, stop adding requirements, accept the current artifact, and debrief. Do not make completion depend on “mastering everything.”
|
|
101
|
+
Use actual timestamps only when available. Do not treat every chat turn as a fixed number of minutes. If real timing is unavailable, use milestone pacing and say timing is approximate. Pauses and tool waits need not count as candidate work. In the live runtime, each message states the candidate's active time so far with your working time left out, because the page pauses their clock while you work; use that figure for pacing and in the report. At the limit, stop adding requirements, accept the current artifact, and debrief. Do not make completion depend on “mastering everything.”
|
|
100
102
|
|
|
101
103
|
In mock mode, keep evaluative feedback for the end unless needed to disambiguate the interface or prevent work under a mistaken rule. In coached mode, explain briefly after an attempt and then check transfer with a different example. Never grade immediate repetition of a revealed answer as independent mastery.
|
|
102
104
|
|
{solvewright-0.1.0 → solvewright-0.1.2}/src/solvewright/interviewer/references/case-generation.txt
RENAMED
|
@@ -43,6 +43,10 @@ workspace/ only candidate-visible files
|
|
|
43
43
|
|
|
44
44
|
Create `.examiner/manifest.json` with `title`, `caseFamily`, `concepts` (a short string list), `mechanism`, the fixed `contract`, stage plan, and validation commands/results. Maintain it as the round evolves. For Python, put the candidate's initial implementation skeleton in `workspace/solution.py` and unittest checks in `workspace/tests/test_public.py`. **Run tests executes in the candidate's browser**, under Pyodide (CPython compiled to WebAssembly), with pytest collecting `workspace/tests/test_*.py` from the `workspace/` directory, so `import solution` works. Public tests must therefore use only the standard library, plus numpy when the task needs it; no threads, subprocesses, sockets, network, wall-clock timing or files outside `workspace/`. Verify your reference and the public tests yourself with the local Python before publishing, under the same constraints. For pseudocode, use `workspace/solution.txt`, give a precise callable/operation contract and hand-trace examples, and do not pretend executable tests have run. Additional candidate data/notes may be created there. Do not put solutions, future-stage prompts, private datasets or an answer-bearing test name in candidate files.
|
|
45
45
|
|
|
46
|
+
**Private checks the page can run.** Besides the public tests, write private checks in `.examiner/checks/` (`test_*.py`, pytest or unittest, importing from `solution`) and `.examiner/checks/checks.json` with `{"checks": [{"id": "test_file.py::test_name", "part": 1, "label": "...", "note": "..."}]}`. The page runs them in the candidate's browser under Pyodide when they submit, so they follow the public tests' constraints and must not import your reference: put expected values in the checks themselves. The candidate sees each label (a plain name of the case, such as "cancel releases the cash") and, when a check fails, its note (one sentence that points at the case without giving the fix); never the check's code or expected values. Verify them on your reference and on one plausible wrong solution. When you release a later part, add its checks the same way.
|
|
47
|
+
|
|
48
|
+
**Keep later turns to one step.** The candidate's clock is paused while you work, and every step spends the candidate's own plan. Each later message arrives with the changes to the candidate's files since your last reply and the latest results of the public tests and your private checks, run in the browser. Reply from them. Run code yourself only for evidence the checks cannot give. Do not list or re-read unchanged files, re-read these instructions, or write checkpoints, hashes or review logs: the runtime snapshots every message's files and results.
|
|
49
|
+
|
|
46
50
|
The first turn should produce and validate the setup, then ask only the opening task. Later turns read candidate files from `workspace/` and test evidence from `.examiner/test-runs.json`. Use `.examiner/progress.json` for hints/reveals and continuity if needed. The runtime saves the chat and snapshots; do not ask the candidate to copy/import JSON.
|
|
47
51
|
|
|
48
52
|
At the end write `.examiner/report.json` in the exact `interview-lab-session-v1` format from [session-format.txt](session-format.txt). Use the runtime's session ID and starting timestamp supplied in settings. Score with [scoring.txt](scoring.txt); leave unobserved dimensions null. Never fabricate elapsed active time. The runtime will save and display the report automatically.
|