oquay 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- oquay-0.1.0/.gitignore +20 -0
- oquay-0.1.0/COMPETITIVE_ANALYSIS.md +352 -0
- oquay-0.1.0/LICENSE +21 -0
- oquay-0.1.0/PKG-INFO +350 -0
- oquay-0.1.0/README.md +324 -0
- oquay-0.1.0/docs/ARCHITECTURE.md +222 -0
- oquay-0.1.0/docs/MIGRATION.md +129 -0
- oquay-0.1.0/docs/PROTOCOL.md +169 -0
- oquay-0.1.0/pyproject.toml +62 -0
- oquay-0.1.0/run_demo.sh +196 -0
- oquay-0.1.0/src/quay/__init__.py +20 -0
- oquay-0.1.0/src/quay/bench.py +424 -0
- oquay-0.1.0/src/quay/canonical.py +160 -0
- oquay-0.1.0/src/quay/cli.py +821 -0
- oquay-0.1.0/src/quay/client.py +383 -0
- oquay-0.1.0/src/quay/daemon.py +279 -0
- oquay-0.1.0/src/quay/migrate.py +421 -0
- oquay-0.1.0/src/quay/protocol.py +249 -0
- oquay-0.1.0/src/quay/schemas.py +374 -0
- oquay-0.1.0/src/quay/server.py +901 -0
- oquay-0.1.0/src/quay/store.py +1217 -0
- oquay-0.1.0/src/quay/tui.py +656 -0
- oquay-0.1.0/src/quay/workspace.py +394 -0
- oquay-0.1.0/tests/conftest.py +104 -0
- oquay-0.1.0/tests/test_bench.py +173 -0
- oquay-0.1.0/tests/test_canonical.py +115 -0
- oquay-0.1.0/tests/test_cli.py +438 -0
- oquay-0.1.0/tests/test_daemon.py +257 -0
- oquay-0.1.0/tests/test_leases.py +177 -0
- oquay-0.1.0/tests/test_migrate.py +373 -0
- oquay-0.1.0/tests/test_protocol.py +277 -0
- oquay-0.1.0/tests/test_schemas.py +244 -0
- oquay-0.1.0/tests/test_server.py +500 -0
- oquay-0.1.0/tests/test_store.py +569 -0
- oquay-0.1.0/tests/test_tui.py +470 -0
oquay-0.1.0/.gitignore
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Quay build and runtime artifacts
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
build/
|
|
6
|
+
dist/
|
|
7
|
+
.venv/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.ruff_cache/
|
|
10
|
+
.coverage
|
|
11
|
+
htmlcov/
|
|
12
|
+
node_modules/
|
|
13
|
+
package-lock.json
|
|
14
|
+
|
|
15
|
+
# NPM tarballs
|
|
16
|
+
*.tgz
|
|
17
|
+
|
|
18
|
+
# Local workspaces created while trying the tool out
|
|
19
|
+
.quay/
|
|
20
|
+
.changi/
|
|
@@ -0,0 +1,352 @@
|
|
|
1
|
+
# Quay — Competitive Analysis & Product Strategy
|
|
2
|
+
|
|
3
|
+
**Mission:** `mission-factory-competitor-creation-20261004`
|
|
4
|
+
**Author:** Factory (Droid autonomous engineering swarm)
|
|
5
|
+
**Date:** 2026-10-04
|
|
6
|
+
**Status:** Milestone 1 deliverable (selection + strategy)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Executive decision
|
|
11
|
+
|
|
12
|
+
**Quay competes against Changi.**
|
|
13
|
+
|
|
14
|
+
Changi (`projects/changi/`, `@agentbus/changi` 0.1.0) is a local-first, single-workspace
|
|
15
|
+
terminal event companion: a zero-dependency Python daemon holds a SQLite ledger, a Unix
|
|
16
|
+
socket serves a one-shot JSON request/response control API, and a 1 Hz ANSI reprint loop
|
|
17
|
+
shows three planes (Record / Signal / Work).
|
|
18
|
+
|
|
19
|
+
Quay rebuilds that product category as a **streaming local event ledger with a real
|
|
20
|
+
terminal cockpit**. It keeps Changi's best property — no cloud, no external runtime
|
|
21
|
+
dependencies, one workspace, one daemon — and removes every structural weakness that makes
|
|
22
|
+
Changi a status printer rather than an observability tool.
|
|
23
|
+
|
|
24
|
+
AgentBus was rejected as a target. Section 3 records why.
|
|
25
|
+
|
|
26
|
+
**One-line positioning:**
|
|
27
|
+
|
|
28
|
+
> Changi records events and reprints a counter. Quay streams them, indexes them, proves
|
|
29
|
+
> they were not tampered with, and lets you watch and query them live.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## 2. Incumbent audit — Changi
|
|
34
|
+
|
|
35
|
+
### 2.1 What it is
|
|
36
|
+
|
|
37
|
+
| Aspect | Observation (from source) |
|
|
38
|
+
|---|---|
|
|
39
|
+
| Runtime | `changi_runtime.py` — 907 lines, single module, Python 3.10+, stdlib only |
|
|
40
|
+
| Storage | SQLite in `<workspace>/.changi/events.db`, `journal_mode=WAL`, `synchronous=FULL`, `busy_timeout=5000` |
|
|
41
|
+
| Schema | One table: `events(event_id, topic, producer_id, timestamp, schema_version, payload, causation_id)`; `payload` is an opaque JSON TEXT |
|
|
42
|
+
| Control plane | `AF_UNIX` SOCK_STREAM, newline-delimited JSON, one request per connection, `MAX_MESSAGE_BYTES = 1 MiB` |
|
|
43
|
+
| Operations | `health`, `status`, `log`, `emit`, `stop` |
|
|
44
|
+
| Planes | Record = any row. Signal = `payload.changi.kind == "signal"`. Work = `payload.changi.kind == "work_receipt"` with receipt fields validated into `receipt_valid` |
|
|
45
|
+
| Projection | `ledger_status()` scans the **entire** table on every `status` call, rebuilding signal-root chains in Python |
|
|
46
|
+
| UI | `monitor()` — `\x1b[?1049h`, clear screen, poll `status` each second, three `json.dumps` lines, `q` to quit |
|
|
47
|
+
| Lifecycle | Auto-spawn daemon on first CLI use, idle shutdown after 4 h (`CHANGI_IDLE_SECONDS`), `/proc/<pid>/stat` start-tick identity, stale-state reclaim, global registry in `~/.config/changi/workspaces.json` |
|
|
48
|
+
| Safety | Symlink refusal, `0o600`/`0o700` modes, `O_NOFOLLOW`, flock bootstrap lock, refusal to signal a live-but-unresponsive PID |
|
|
49
|
+
| Tests | `tests/test_changi.py` — 391 lines, 18 `unittest` cases |
|
|
50
|
+
| Platform | Linux only (`"os": ["linux"]` in `package.json`) |
|
|
51
|
+
|
|
52
|
+
### 2.2 Strengths worth preserving
|
|
53
|
+
|
|
54
|
+
1. Zero external dependencies; installs and runs anywhere with Python.
|
|
55
|
+
2. Real durability discipline: WAL + `synchronous=FULL`, owner-only modes, symlink refusal.
|
|
56
|
+
3. Careful daemon lifecycle: immutable process identity, stale-state reclaim, no signal-blast.
|
|
57
|
+
4. Honest plane semantics: a Signal never silently counts as completed Work.
|
|
58
|
+
5. Small, auditable surface; no network egress.
|
|
59
|
+
|
|
60
|
+
### 2.3 Weaknesses Quay attacks
|
|
61
|
+
|
|
62
|
+
| # | Weakness | Evidence | Consequence | Quay's answer |
|
|
63
|
+
|---|---|---|---|---|
|
|
64
|
+
| W1 | UI is a clear-and-reprint loop, not a TUI | `monitor()` reprints three JSON blobs each second | No event stream, no scroll, no detail view, no color; useless the moment activity arrives | Incremental-diff truecolor TUI: live stream, scroll, filter, detail pane, per-plane dashboards, byte-budgeted redraws |
|
|
65
|
+
| W2 | Request/response only; no streaming, no subscriptions | `socket_request()` opens a socket per CLI call and closes it | No push, no tail, no cross-process live view; latency is one round trip per poll | Framed multiplexed protocol with `subscribe`/`unsubscribe`, server-initiated `event` frames, heartbeats |
|
|
66
|
+
| W3 | `status` is O(rows) per call | `ledger_status()` scans every row and re-derives signal roots | Monitor cost grows with history; 1 Hz poll degrades on large ledgers | Materialized plane projections in dedicated tables, updated transactionally on append; status is O(1) |
|
|
67
|
+
| W4 | No query capability | CLI offers only `log --limit` | Cannot answer "which producer failed last hour" | Filter engine: topic glob, producer, plane, state, time range, substring, causation; aggregates; JSONL/CSV export |
|
|
68
|
+
| W5 | Payloads are unvalidated JSON | Only `work_receipt` gets a heuristic check; everything else is opaque | Typos and malformed events enter the ledger permanently | Declarative schema registry, versioned envelopes, strict reject-at-boundary validation |
|
|
69
|
+
| W6 | No retention or growth control | Ledger grows forever; no prune command | Unbounded disk growth | Retention policy + `quay prune`, plus log compaction of superseded plane updates |
|
|
70
|
+
| W7 | No tamper evidence or integrity check | Plain rows, no chain | Silent corruption or edit goes unnoticed | SHA-256 hash chain over canonical event bytes + `quay verify` |
|
|
71
|
+
| W8 | Head-of-line blocking | `serve()` handles one connection synchronously inside the accept loop | One slow client stalls all others | Worker-thread connection handling with a serialized write path and bounded per-connection queues |
|
|
72
|
+
| W9 | No backpressure | Frames are written synchronously; a slow subscriber has no policy | Unbounded memory or stalled daemon | Bounded outbound queues; explicit `lagged` frame with drop count instead of blocking |
|
|
73
|
+
| W10 | Single-threaded, one event per commit | `emit` inserts with autocommit `with connection:` | Throughput ceilings on bursty input | Batched append path with single-transaction multi-insert and `quay bench` to prove it |
|
|
74
|
+
| W11 | Plane logic is an ad-hoc `payload.changi` convention | Signal/Work semantics read out of extension dict | Core record plane coupled to app conventions | First-class typed event kinds and a declared projection model, not magic keys |
|
|
75
|
+
| W12 | Work has no lifecycle | A receipt is valid or not; nothing tracks ownership | Cannot tell "who is working on this now" | Work leases: `claim` / `heartbeat` / `complete` / `abandon` with expiry |
|
|
76
|
+
| W13 | Linux-only, `/proc` identity | `process_start_identity()` reads `/proc`; npm `os: linux` | No macOS support | Pluggable process-identity probe with `ps`-based fallback and documented platform matrix |
|
|
77
|
+
| W14 | Monitor cannot survive output pressure | Whole-frame reprint | Screen tearing, flicker under load | Double-buffered frame diffing; only changed cells written |
|
|
78
|
+
|
|
79
|
+
### 2.4 Measured baselines (to be reproduced by `quay bench`)
|
|
80
|
+
|
|
81
|
+
These are the incumbent behaviors Quay must beat; Milestone 5 publishes the measured numbers.
|
|
82
|
+
|
|
83
|
+
| Metric | Changi baseline (structural) | Quay target |
|
|
84
|
+
|---|---|---|
|
|
85
|
+
| Live update latency | 1 Hz poll interval + one round trip | Server-push, sub-10 ms p99 on localhost |
|
|
86
|
+
| `status` cost vs. ledger size | O(n) full table scan | O(1) projection read |
|
|
87
|
+
| Append throughput | 1 event per transaction | ≥ 20× Changi single-event path on batch append |
|
|
88
|
+
| Slow-consumer behavior | Blocks writer | Drops with accounting, never blocks |
|
|
89
|
+
| Integrity | none | Hash chain verified end-to-end |
|
|
90
|
+
|
|
91
|
+
Numbers are filled in by `quay bench` during Milestone 5; targets are hypotheses until measured.
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## 3. Why not AgentBus
|
|
96
|
+
|
|
97
|
+
AgentBus (`projects/agentbus/`, `okf-agentbus` 0.24.1) is not a tractable target for a
|
|
98
|
+
from-scratch competitor inside this mission, and competing with it is not the best use of
|
|
99
|
+
the available effort:
|
|
100
|
+
|
|
101
|
+
- **Scope.** ~6,500 lines in `src/agentbus/` alone across 55 modules, plus a Go control
|
|
102
|
+
plane (`go-core/`), a TypeScript client (`packages/js/agentbus-client`), an MCP server,
|
|
103
|
+
RBAC, cryptographic identity/ceremony, HITL approvals, SLA tracking, leases, wake
|
|
104
|
+
ingress, pricing, PostHog telemetry, and a Textual God View TUI. 58 test files.
|
|
105
|
+
- **Maturity.** It already ships multi-channel releases (PyPI + npm + platform wheels) with
|
|
106
|
+
a Pi QA gate, CodeRabbit review, and a versioned changelog at 0.24.1. A competitor built
|
|
107
|
+
in five milestones would either be a narrow slice or a shallow copy.
|
|
108
|
+
- **The honest comparison.** Any slice worth building (say, a fast single-binary broker)
|
|
109
|
+
is a *different product*, not a competitor to the whole system; the mission brief itself
|
|
110
|
+
frames AgentBus's only viable angles as "ultra-high throughput" or "single binary", both
|
|
111
|
+
of which are performance plays rather than a usability gap.
|
|
112
|
+
- **Changi has the real, exploitable gap.** Its category — one engineer, one directory,
|
|
113
|
+
one durable local ledger — has a clear definition of "better": stream instead of poll,
|
|
114
|
+
index instead of scan, validate instead of trust, prove instead of hope, and give the
|
|
115
|
+
human an actual cockpit. Every one of those is deliverable and testable.
|
|
116
|
+
|
|
117
|
+
AgentBus therefore serves as an **architecture reference** (framed protocol, hash-stable
|
|
118
|
+
canonical encoding, append-only event model), not as the target.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## 4. Target user and jobs to be done
|
|
123
|
+
|
|
124
|
+
**Primary user:** a solo engineer or a small local agent fleet working in one repository,
|
|
125
|
+
who wants durable local evidence of what happened and who wants to *watch* it live.
|
|
126
|
+
|
|
127
|
+
**Jobs to be done:**
|
|
128
|
+
|
|
129
|
+
1. "Record this event durably and cheaply, from a script, an agent, or my shell."
|
|
130
|
+
2. "Show me what is happening right now, live, without me polling or refreshing."
|
|
131
|
+
3. "Tell me which work is claimed, in flight, blocked, or finished — and by whom."
|
|
132
|
+
4. "Let me query history precisely: by topic, producer, plane, state, and time."
|
|
133
|
+
5. "Prove the ledger was not silently edited or corrupted."
|
|
134
|
+
6. "Do not grow forever, do not phone home, do not require Docker or a cloud account."
|
|
135
|
+
|
|
136
|
+
**Non-goals:** multi-host coordination, cross-workspace federation, agent execution,
|
|
137
|
+
network services, LLM orchestration, dashboards in a browser. Quay is single-workspace and
|
|
138
|
+
local by design; that is the product, not a limitation to apologize for.
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## 5. Product requirements
|
|
143
|
+
|
|
144
|
+
Requirements below are the acceptance contract for Milestones 2–5. Each has an ID used in
|
|
145
|
+
the test suite and the final report.
|
|
146
|
+
|
|
147
|
+
### Functional
|
|
148
|
+
|
|
149
|
+
| ID | Requirement | Milestone |
|
|
150
|
+
|---|---|---|
|
|
151
|
+
| F1 | Append-only event ledger in `<workspace>/.quay/ledger.db`, SQLite WAL, ACID, crash-safe | M2 |
|
|
152
|
+
| F2 | Each event carries a monotonic `seq`, canonical byte encoding, and a SHA-256 hash chained to its predecessor | M2 |
|
|
153
|
+
| F3 | Typed, versioned event envelopes with declared `kind`; unknown kinds accepted as `generic`, malformed ones rejected before write | M2 |
|
|
154
|
+
| F4 | Declarative schema registry: `signal`, `work`, `note`, `metric`, `generic`; unknown fields rejected in strict mode | M2 |
|
|
155
|
+
| F5 | Plane projections (record/signal/work) maintained transactionally; status is O(1) | M2 |
|
|
156
|
+
| F6 | Work lifecycle with leases: `claim`, `heartbeat`, `complete`, `abandon`; leases expire and are reclaimable | M2 |
|
|
157
|
+
| F7 | `verify` recomputes the hash chain and reports the first divergence with seq number | M2 |
|
|
158
|
+
| F8 | Retention + `prune` with a dry-run preview; never deletes without explicit confirmation | M2 |
|
|
159
|
+
| F9 | Framed IPC protocol (length-prefixed, versioned) over `AF_UNIX` with graceful shutdown | M3 |
|
|
160
|
+
| F10 | Operations: `health`, `status`, `emit`, `emit_batch`, `query`, `log`, `tail`, `subscribe`, `unsubscribe`, `work_*`, `verify`, `prune`, `stop` | M3 |
|
|
161
|
+
| F11 | Server-push `event` frames to subscribers; heartbeats; explicit `lagged` frame with drop count on backpressure | M3 |
|
|
162
|
+
| F12 | Strict frame validation with structured error envelopes (`code`, `message`, `detail`) | M3 |
|
|
163
|
+
| F13 | Query filters: topic glob, producer, kind, plane, state, time range, substring, causation, limit/offset | M3 |
|
|
164
|
+
| F14 | CLI subcommands: `init`, `start`, `stop`, `restart`, `status`, `emit`, `work`, `query`, `log`, `tail`, `watch`, `verify`, `prune`, `daemons`, `doctor`, `bench` | M4 |
|
|
165
|
+
| F15 | Non-interactive JSON output for every command (`--json` or default when piped) | M4 |
|
|
166
|
+
| F16 | Live TUI: incremental truecolor redraw, scrollable stream, filter bar, detail pane, plane dashboard, `q` detach, `?` help | M4 |
|
|
167
|
+
| F17 | Daemon lifecycle: `start/stop/restart`, PID + start identity, stale reclaim, idle shutdown, `--all` registry ops | M4 |
|
|
168
|
+
| F18 | `run_demo.sh`: unattended end-to-end demonstration, exit code 0 | M5 |
|
|
169
|
+
| F19 | `bench` suite: append throughput, query latency, status latency, fan-out to N subscribers | M5 |
|
|
170
|
+
| F20 | Packaging: `pip install .` yields `quay` and `quayd` entry points; zero runtime dependencies | M5 |
|
|
171
|
+
|
|
172
|
+
### Non-functional
|
|
173
|
+
|
|
174
|
+
| ID | Requirement |
|
|
175
|
+
|---|---|
|
|
176
|
+
| N1 | Zero runtime third-party dependencies (stdlib only) |
|
|
177
|
+
| N2 | No network egress; `AF_UNIX` only |
|
|
178
|
+
| N3 | Owner-only file modes (`0o600`/`0o700`), symlink refusal, `O_NOFOLLOW` on owned files |
|
|
179
|
+
| N4 | All owned state lives under `<workspace>/.quay/`; user registry under `$XDG_CONFIG_HOME/quay/` |
|
|
180
|
+
| N5 | Python ≥ 3.11; Linux and macOS supported paths for process identity and sockets |
|
|
181
|
+
| N6 | Test suite runs with one command and passes with no skips on a normal Linux host |
|
|
182
|
+
| N7 | No test may mock the storage engine; storage and transport are exercised for real |
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
## 6. Architectural principles
|
|
187
|
+
|
|
188
|
+
1. **The log is the truth.** Every derived view (planes, leases, counts) is a projection of
|
|
189
|
+
the append-only event log and can be rebuilt from it.
|
|
190
|
+
2. **Canonical bytes or it did not happen.** Events are serialized with a deterministic
|
|
191
|
+
canonical encoder (sorted keys, no insignificant whitespace), so hashes are stable
|
|
192
|
+
across processes, platforms, and Python versions.
|
|
193
|
+
3. **Validate at the boundary, never trust the caller.** Unknown kinds degrade to
|
|
194
|
+
`generic`; malformed fields are rejected with a code, not normalized silently.
|
|
195
|
+
4. **Push, do not poll.** The daemon owns change notification. Clients subscribe; the TUI
|
|
196
|
+
renders what arrives.
|
|
197
|
+
5. **Never block the writer.** Backpressure is accounted for and made visible (`lagged`),
|
|
198
|
+
never absorbed as unbounded memory or allowed to stall other clients.
|
|
199
|
+
6. **Cheap reads.** Projections are materialized; status and dashboards are index lookups,
|
|
200
|
+
not scans.
|
|
201
|
+
7. **Durable by default, explicit by exception.** `synchronous=FULL` by default, with a
|
|
202
|
+
documented fast mode for benchmarks.
|
|
203
|
+
8. **Honest planes.** A signal is not work; an expired lease is not progress. The UI says
|
|
204
|
+
so plainly.
|
|
205
|
+
9. **Small and auditable.** Stdlib only; a reader can understand the whole system.
|
|
206
|
+
10. **Every claim is testable.** Each requirement above maps to a test or a benchmark.
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## 7. Chosen tech stack
|
|
211
|
+
|
|
212
|
+
| Layer | Choice | Rationale |
|
|
213
|
+
|---|---|---|
|
|
214
|
+
| Language | Python ≥ 3.11, stdlib only | Matches Changi's zero-dependency install story while allowing threads, `selectors`, `sqlite3`, `hashlib`, `fcntl`/`msvcrt` |
|
|
215
|
+
| Storage | SQLite (`sqlite3`) in WAL mode, `synchronous=FULL` default, `busy_timeout` | Proven, crash-safe, single-file, no server |
|
|
216
|
+
| Encoding | Deterministic canonical JSON (sorted keys, compact separators, UTF-8, `\n`-free) | Stable hashes; mirrors AgentBus's JCS-style discipline without a third-party dep |
|
|
217
|
+
| Integrity | SHA-256 chain `hash_n = sha256(hash_{n-1} \|\| canonical_bytes_n)`, genesis constant | Tamper evidence with no dependency |
|
|
218
|
+
| Transport | `AF_UNIX` SOCK_STREAM, 4-byte big-endian length-prefixed JSON frames | Bounded framing, cheap multiplexing, mirrors AgentBus broker protocol |
|
|
219
|
+
| Concurrency | One writer thread + bounded queue; a worker thread per connection; `selectors` accept loop | Serialized writes, no head-of-line blocking |
|
|
220
|
+
| CLI | `argparse` subcommands, JSON-first output | No dependency; scriptable |
|
|
221
|
+
| TUI | Hand-rolled ANSI with double-buffered diffs, truecolor, alternate screen, raw-mode input via `termios` | Real cockpit without `textual`/`curses` weight |
|
|
222
|
+
| Tests | `pytest` (dev-only) with real subprocess daemons and real sockets | Honest integration coverage per N7 |
|
|
223
|
+
| Packaging | `hatchling` + `pyproject.toml`, entry points `quay` / `quayd` | Standard, no runtime deps |
|
|
224
|
+
|
|
225
|
+
Deliberate rejection: no `textual` (heavy, contradicts W-free install), no `msgpack`/`protobuf`
|
|
226
|
+
(external deps for a local socket), no ORM (SQLite is the right abstraction level), no
|
|
227
|
+
daemon supervisor library (we implement lifecycle explicitly and test it).
|
|
228
|
+
|
|
229
|
+
---
|
|
230
|
+
|
|
231
|
+
## 8. Project layout
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
projects/quay/
|
|
235
|
+
├── COMPETITIVE_ANALYSIS.md # this document (M1)
|
|
236
|
+
├── README.md # install, quickstart, CLI reference, architecture (M5)
|
|
237
|
+
├── LICENSE
|
|
238
|
+
├── pyproject.toml # hatchling, entry points quay/quayd (M5)
|
|
239
|
+
├── run_demo.sh # unattended end-to-end demo, exit 0 (M5)
|
|
240
|
+
├── src/quay/
|
|
241
|
+
│ ├── __init__.py # version
|
|
242
|
+
│ ├── canonical.py # deterministic encoding + hashing
|
|
243
|
+
│ ├── schemas.py # envelope + schema registry + validation
|
|
244
|
+
│ ├── store.py # SQLite ledger, projections, leases, verify, prune
|
|
245
|
+
│ ├── protocol.py # frame encode/decode, error envelopes, validation
|
|
246
|
+
│ ├── server.py # daemon: accept loop, writer queue, subscriptions
|
|
247
|
+
│ ├── client.py # blocking + streaming client
|
|
248
|
+
│ ├── daemon.py # lifecycle: spawn, health, stale reclaim, registry, idle
|
|
249
|
+
│ ├── cli.py # argparse subcommands, JSON output
|
|
250
|
+
│ ├── tui.py # incremental truecolor monitor
|
|
251
|
+
│ ├── bench.py # benchmark suite
|
|
252
|
+
│ └── migrate.py # Changi ledger importer
|
|
253
|
+
├── tests/
|
|
254
|
+
│ ├── test_canonical.py
|
|
255
|
+
│ ├── test_schemas.py
|
|
256
|
+
│ ├── test_store.py # durability, rollback, concurrency, verify, prune
|
|
257
|
+
│ ├── test_leases.py
|
|
258
|
+
│ ├── test_protocol.py
|
|
259
|
+
│ ├── test_server.py # handshake, round-trips, timeouts, cleanup, backpressure
|
|
260
|
+
│ ├── test_daemon.py # lifecycle, stale reclaim, registry, idle shutdown
|
|
261
|
+
│ ├── test_cli.py # every command, TTY and non-TTY
|
|
262
|
+
│ ├── test_tui.py # frame rendering/diff logic, headless
|
|
263
|
+
│ ├── test_bench.py # benchmarks run and report
|
|
264
|
+
│ └── test_migrate.py # Changi import fidelity and failure handling
|
|
265
|
+
└── docs/
|
|
266
|
+
├── ARCHITECTURE.md
|
|
267
|
+
├── PROTOCOL.md
|
|
268
|
+
└── MIGRATION.md # from Changi: import a .changi/events.db ledger
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
---
|
|
272
|
+
|
|
273
|
+
## 9. Milestone gates
|
|
274
|
+
|
|
275
|
+
| Gate | Criterion |
|
|
276
|
+
|---|---|
|
|
277
|
+
| M1 | This document exists; target chosen; requirements, stack, layout specified |
|
|
278
|
+
| M2 | 100% passing storage/state tests, including crash recovery, rollback, concurrent writes, chain verification |
|
|
279
|
+
| M3 | Transport suite green; daemon accepts framed clients, streams events, shuts down cleanly, cleans sockets |
|
|
280
|
+
| M4 | Every CLI command exercised in tests; `--json` valid; TUI renders and diffs correctly headless |
|
|
281
|
+
| M5 | Full suite + benchmarks green in one command; `run_demo.sh` exits 0 unattended; README complete; packaging installs; handoff published |
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
## 10. Handoff note
|
|
286
|
+
|
|
287
|
+
Milestone 1 is complete with this document. Milestone 2 begins with project scaffolding
|
|
288
|
+
(F9-adjacent package manifests), the canonical encoder, the schema registry, and the SQLite
|
|
289
|
+
ledger with projections, leases, verification, and pruning, each with real tests.
|
|
290
|
+
|
|
291
|
+
---
|
|
292
|
+
|
|
293
|
+
## 11. Gate results (recorded at completion)
|
|
294
|
+
|
|
295
|
+
Each gate is recorded with the evidence that closed it. Numbers are from the reference
|
|
296
|
+
machine (Linux 6.6, WSL2, Python 3.12.3).
|
|
297
|
+
|
|
298
|
+
| Gate | Status | Evidence |
|
|
299
|
+
|---|---|---|
|
|
300
|
+
| M1 | Met | This document: target chosen (Changi), 14 weaknesses, F1–F20 / N1–N7, stack, layout |
|
|
301
|
+
| M2 | Met | `tests/test_canonical.py`, `test_schemas.py`, `test_store.py`, `test_leases.py` — rollback, crash recovery, multi-process concurrent writes, tamper detection, chain verification |
|
|
302
|
+
| M3 | Met | `tests/test_protocol.py`, `test_server.py`, `test_daemon.py` — framing, round-trips, timeouts, backpressure, `lagged` accounting, socket cleanup, stale-state reclaim, idle shutdown |
|
|
303
|
+
| M4 | Met | `tests/test_cli.py` (every command, TTY and non-TTY, JSON default when piped), `tests/test_tui.py` (rendering, diffs, keys, live socket loop) |
|
|
304
|
+
| M5 | Met | 381 tests green in one command (`python3 -m pytest -q`), `run_demo.sh` exits 0 unattended, README + three docs, wheel and sdist install cleanly, handoff published to `agy` and `codex` |
|
|
305
|
+
|
|
306
|
+
### Requirements coverage
|
|
307
|
+
|
|
308
|
+
| Requirement | Where it is implemented and tested |
|
|
309
|
+
|---|---|
|
|
310
|
+
| F1–F4 ledger, envelope, kinds, hashing | `canonical.py`, `schemas.py`, `store.py`; `test_canonical.py`, `test_schemas.py` |
|
|
311
|
+
| F5–F8 projections, O(1) status, verify, prune | `store.py`; `test_store.py`, `bench.py` |
|
|
312
|
+
| F9–F11 leases, reap, work plane | `store.py`, `cli.py`; `test_leases.py` |
|
|
313
|
+
| F12–F15 protocol, daemon, subscriptions, CLI | `protocol.py`, `server.py`, `client.py`, `daemon.py`, `cli.py`; `test_protocol.py`, `test_server.py`, `test_daemon.py`, `test_cli.py` |
|
|
314
|
+
| F16–F20 TUI, doctor, JSON output, exports, import | `tui.py`, `cli.py`, `migrate.py`; `test_tui.py`, `test_cli.py`, `test_migrate.py` |
|
|
315
|
+
| N1 no runtime dependencies | `pyproject.toml` (`dependencies = []`), wheel install verified |
|
|
316
|
+
| N2 stdlib-only internals | imports are stdlib only across `src/quay/` |
|
|
317
|
+
| N3 real storage and transport in tests | `tests/conftest.py`; no mocks of SQLite or sockets |
|
|
318
|
+
| N4 bounded memory | bounded outbox, `SO_SNDBUF` cap, `MAX_FRAME_BYTES`; backpressure tests |
|
|
319
|
+
| N5 crash resilience | WAL + `synchronous=FULL`, transactional batches, checkpointed prune; recovery tests |
|
|
320
|
+
| N6 integrity verifiable | `verify()` over the SHA-256 chain; tamper tests |
|
|
321
|
+
| N7 no skipped tests | 381 collected, 381 run, 0 skipped |
|
|
322
|
+
|
|
323
|
+
### Measured outcomes
|
|
324
|
+
|
|
325
|
+
| Measurement | Quay | Baseline |
|
|
326
|
+
|---|---|---|
|
|
327
|
+
| Append throughput (batched, 500/txn) | 3,008 events/s | 893 events/s single-append |
|
|
328
|
+
| `status()` at 10,000 rows | 0.24 ms | 49.5 ms full-table scan |
|
|
329
|
+
| Push fan-out p50 / p99 | 0.0 ms / 8.4 ms | – |
|
|
330
|
+
| Fan-out end-to-end p50 / p99 | 10.7 ms / 22.2 ms | – |
|
|
331
|
+
| Fan-out deliveries | 1,200 / 1,200, 0 dropped | – |
|
|
332
|
+
|
|
333
|
+
### Deviations from the M1 plan, and why
|
|
334
|
+
|
|
335
|
+
1. **`import` was added as a 17th CLI command** and a daemon op (`migrate.py`). The M1 layout
|
|
336
|
+
listed `docs/MIGRATION.md` but not an importer; a migration guide without a working
|
|
337
|
+
importer would have been documentation of an intention, so the importer was built and
|
|
338
|
+
tested (`tests/test_migrate.py`, 24 tests).
|
|
339
|
+
2. **Changi's plane records cannot be copied verbatim.** Changi stores terminal records with
|
|
340
|
+
no history, while Quay's state machines require a chain root. The importer reconstructs a
|
|
341
|
+
root, marks it in the title and correlation id, and counts it separately in the report
|
|
342
|
+
rather than fabricating an unmarked past. See `docs/MIGRATION.md`.
|
|
343
|
+
3. **Two teardown-order bugs were found by the test suite, not by review.** Registration
|
|
344
|
+
happened after the socket was bound, and teardown removed the socket before the registry
|
|
345
|
+
row. Both made "socket present/absent" an unreliable signal, which is exactly the class of
|
|
346
|
+
defect this project exists to fix, so both were corrected (start publishes identity first;
|
|
347
|
+
teardown removes the socket last) and the daemon suite was run six times consecutively to
|
|
348
|
+
confirm the flakiness was gone.
|
|
349
|
+
4. **The fan-out benchmark reports two latencies.** An early version measured arrival after
|
|
350
|
+
the write acknowledgement and labelled it end-to-end, which flattered the number. It now
|
|
351
|
+
reports `push_*` and `end_to_end_*` separately, with the collector contention stated.
|
|
352
|
+
|
oquay-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 quay contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|