dreamteamer 0.13.4 → 0.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1020 @@
1
+ # data modeling — from a requirement to a model that stays good
2
+
3
+ The user states a requirement — "track the clinic's visits", "stop losing lab results" — and is not
4
+ a data architect. This reference is what stands between that sentence and a model: it turns the
5
+ requirement into collections, fields and relations that are searchable, filterable, legible in any
6
+ surface, cheap to keep, and still right a year in. Method and judgment live here; mechanics (the
7
+ meta verbs, namespaces, `templates:`, registering an existing folder) live in `collections.md`.
8
+
9
+ It is long on purpose. It is the reference for the single highest-leverage act in a dreamteamer
10
+ workspace — a model outlives every skill and command written against it — and the reader is usually
11
+ an agent mid-conversation, which cannot go read a shelf of data-modeling books. Sections are
12
+ self-contained; when loaded for one question, read the part that answers it:
13
+
14
+ | the question | read |
15
+ |---|---|
16
+ | what kind of database is this, really | Part I |
17
+ | a new capability needs modeling | Part II (the interview), then Part XI (a worked example) |
18
+ | which module / namespace does it go in | Part III |
19
+ | is this a collection, a field, an enum, tags…? | Part IV–V |
20
+ | how do these two things point at each other | Part VI |
21
+ | will it be usable — lists, forms, pickers, views | Part VII |
22
+ | will it stay fast | Part VIII |
23
+ | the model exists and is wrong | Part IX (change it) and Part X (what "wrong" looks like) |
24
+
25
+ ---
26
+
27
+ ## Part I — first principles: what kind of database this is
28
+
29
+ ### 1. Records are plain files, and the reader is a person, an agent, and git
30
+
31
+ A record is one file (`<id>.<suffix>.<ext>`), its fields are frontmatter, its prose is a markdown
32
+ body, and the whole store is a git repository. There is no query planner between the reader and the
33
+ data — the primary read paths are `cat`, `grep`, `git show`, `git log -p`, and the engine's own
34
+ one-pass filters. **Every modeling choice should be tested against those readers**, because they are
35
+ what the model is FOR:
36
+
37
+ - A record must be understandable **alone**, with no schema open beside it. This is why references
38
+ are fully qualified (`health/doctors/dana-levi`, never a bare `dana-levi`): the prefix is a type
39
+ annotation that survives `grep`, a hand-edit, and a `git show` five years later. A DB row is never
40
+ read without its schema; a record is designed to be.
41
+ - A record's **diff must mean something**. One logical change should touch few lines in few files,
42
+ because `git log -p` on a record is its audit trail and a noisy diff destroys it.
43
+ - A record's **name must sort and file itself**. Ids are paths; a time-prefixed id
44
+ (`2026/07/2026-07-02--intake--dana-levi`) makes `ls` a timeline and keeps any one folder small.
45
+
46
+ When two designs are otherwise equal, pick the one that reads better in a terminal. That single
47
+ tie-breaker decides more below than any other principle — it is why the far side of a relation is
48
+ materialised into the file instead of resolved at query time, why enums beat magic numbers, why
49
+ descriptions are load-bearing, and why a body is markdown rather than a `content` field of escaped
50
+ text.
51
+
52
+ ### 2. The schema serves questions, not data
53
+
54
+ A model is not a mirror of the domain; it is a machine for answering the questions the workspace
55
+ will actually ask. "What is a visit?" is philosophy; "which visits this month have no follow-up
56
+ booked?" is a filter — and only the second tells you what the fields are. So every collection is
57
+ designed backwards from its questions:
58
+
59
+ - A question asked by **filtering or sorting** needs a **field**.
60
+ - A question asked by **reading one record** is answered by the **body**.
61
+ - A question **never asked** is a `description:` on something else, or nothing.
62
+
63
+ Write the collection's defining question into its `description`. When you cannot name the question,
64
+ you have not found a collection — you have found a pile, and a pile is fine as a folder; it does not
65
+ need a schema.
66
+
67
+ ### 3. Denormalized on purpose — and exactly two licenses to copy
68
+
69
+ Classic normalization exists to make writes safe on a system where reads can join. Here reads
70
+ cannot join (filters resolve **one hop, outbound only** — see Part VI), the write rate is human-and-
71
+ agent scale, and git makes every copy auditable. So the model leans denormalized: a record should
72
+ answer its own questions without a join. But "denormalized" licenses exactly two kinds of copy, and
73
+ nothing else:
74
+
75
+ 1. **A generated mirror** — the far side of a relation. The engine writes it, maintains it in the
76
+ same write as every change to the owning side, refuses hand-edits to it, and `check` reports one
77
+ that has fallen behind. Cost-free to keep; declare it freely (Part VI).
78
+ 2. **A denormalized key** — one scalar (almost always a date) copied from a related record so a
79
+ list sorts or an id generates without a join. Manual, so it MUST carry its maintenance contract:
80
+ the field's `description` names the **source** and the **writer** ("denormalized from
81
+ `visit.date` by the intake command"), or it will rot silently.
82
+
83
+ Everything else — copying a parent's fields onto children, restating a body, duplicating a
84
+ vocabulary across collections — is not denormalization, it is a second copy with no keeper. The
85
+ test: **who updates this when the source changes, and how would anyone notice if they didn't?** No
86
+ answer, no copy.
87
+
88
+ ### 4. Object-oriented in the composition sense
89
+
90
+ The useful half of object orientation maps cleanly onto a workspace; the inheritance half does not.
91
+
92
+ | OO idea | its shape here | its non-shape |
93
+ |---|---|---|
94
+ | an object owns its state | a record owns its fields; the collection is the class | fields about X scattered on Y "for convenience" |
95
+ | methods | **command-bindings**: verbs bound to a collection, gated on field state (`can-enter`/`can-exit`) — `dt commands <ref>` lists what applies to a record right now | a workflow engine; procedures copied into descriptions |
96
+ | interfaces / mixins | **`templates:`** — a shared field set stamped onto consumers, merged at compile | copy-pasting the same four fields into six descriptors |
97
+ | encapsulation | **module ownership** — a concept's fields live with the module that owns the concept | the module that happened to need the field first |
98
+ | polymorphism | a **union reference** (`x-reference: [meetings, visits]`) or the open-world `'*'` for evidence/source fields | a `type` field plus fields that only apply to some rows |
99
+ | subclassing | **don't.** Two collections + a shared template, or one collection + an enum — decided by the triage test below | `extends:` as taxonomy — it exists for workspace overlays, not for "is-a" |
100
+
101
+ **The triage test, for "is this one collection with a `kind`, or two collections":** do the members
102
+ get triaged on the same questions, by the same person, in the same pass? A lead being sold and a
103
+ client being delivered are the same record at two funnel stages — one shape, two collections,
104
+ because the *questions* differ ("which should I chase" vs "which is at risk") even though the
105
+ fields barely do; a shared template carries the common fields. Conversely, prescriptions for
106
+ tablets and for physiotherapy are one collection with a `kind`, because every one gets the same
107
+ review. A `kind` enum whose values route to different reviews, different states, or different
108
+ required fields is two collections wearing one name — split it.
109
+
110
+ ### 5. The surface is downstream of the descriptor
111
+
112
+ The engine names no component, no route, no pixel. It projects each descriptor into a presentation
113
+ contract — types, specials, options — and any surface that honours the contract renders any model
114
+ this method produces (the full table is Part VII §29). The consequence for modeling: **you design
115
+ the UI by choosing field shapes, not by asking for widgets.** "I want a dropdown" is not a request
116
+ to a UI team; it is `enum:` — or better, nothing, because a low-cardinality free string already
117
+ becomes a dropdown through `dt values`. "I want chips" is an array. "I want a page" is `x-body`.
118
+ When the rendering is wrong, the fix is almost always in the descriptor, and a fix in the
119
+ descriptor fixes every surface at once.
120
+
121
+ ---
122
+
123
+ ## Part II — the method
124
+
125
+ ### 6. The interview — twelve questions, asked of the requirement
126
+
127
+ Answer from the requirement first; put a question to the user only where the requirement is silent
128
+ and the answers genuinely diverge. Most requirements answer eight of the twelve unprompted.
129
+
130
+ 1. **Nouns.** Every noun the user would *open, list, or point at* is a collection candidate; a noun
131
+ that only ever appears inside another is a field or a nested object. *A clinic: `visits` is a
132
+ collection; the vitals taken at a visit nest on the visit — until someone asks "blood pressure
133
+ over time", at which point `lab-values` becomes a collection with one value per record, because
134
+ a nested object cannot be filtered across records (Part V §19).*
135
+ 2. **The question.** Name the question each collection exists to answer and write it into
136
+ `description`. The **grain** — what one record IS — is the largest unit that answers the
137
+ question without unpacking (Part IV §13).
138
+ 3. **One record.** What is its identity? `id.generate` builds it from creation-time values the
139
+ record OWNS — the domain's own date, never `created` (a back-dated import would file a June
140
+ visit under November); a `YYYY/MM/` prefix when growth is unbounded, and `id.pattern` must then
141
+ admit `/`. And which field says two records are the *same* thing — that is what `x-unique` is
142
+ for, on the reference that must not be claimed twice.
143
+ 4. **Lifecycle.** What states, what moves a record between them, and what "done" is. `status` is
144
+ the field a board groups by and a command-binding gates on. End of life is a **state**
145
+ (`archived`, `dropped`), not `rm` — `rm` refuses while anything points at the record, and a
146
+ deleted record takes its history's legibility with it.
147
+ 5. **Stable vs volatile.** A thing and the churning activity about it are two collections
148
+ (`patients` and `visits`; `products` and `issues`). A new *version* of the same thing is **git
149
+ history, never a second record** — one dossier per patient, superseded text visible in
150
+ `git log`, not `dossier-v2`.
151
+ 6. **Who points at whom, from which side.** The foreign key lives on the many side; scalar or
152
+ array follows cardinality; the far side is one generated keyword away, never a hand-maintained
153
+ field. Declare the inverse when "which X have no Y" will be asked; skip it when only "this Y's
154
+ X" will (Part VI §25).
155
+ 7. **The forcing field.** One required field whose absence makes the record write-only: `evidence`
156
+ on a defect, `done_when` on a goal, `dose` on a prescription. It is the difference between a
157
+ record and a wish. A collection with no conceivable forcing field is a list — model it as one
158
+ (a `tags` array, a body bullet) and say so in the proposal.
159
+ 8. **Units and time.** A number without its unit is a latent bug: `amount` + `currency`, `value` +
160
+ `unit`, or the unit in the name (`duration_min`). An instant is `date-time` with its offset; a
161
+ day is `date`; never store an instant in a day field because "it sorts fine today". One
162
+ operator over git: **ownership and permissions are not modelled** — invent no `owner`,
163
+ `created_by`, `assignee` unless multiple humans genuinely write the workspace, and even then
164
+ model the *people* first.
165
+ 9. **Volume and write rate.** Roughly how many records in a year, and what writes them — a human
166
+ occasionally, a command per event, a sync every fifteen minutes? Ten records need no id prefix
167
+ and no namespace; ten thousand machine-written records need a time-sharded id, a lean
168
+ frontmatter, and a hard look at every inverse that would concentrate writes into one hot file
169
+ (Part VIII §36).
170
+ 10. **The list.** What columns would the user scan — which four to six fields identify, order, and
171
+ triage a record at a glance? That is `list_fields`, and if you cannot fill it, the fields are
172
+ wrong (Part VII §30).
173
+ 11. **The module.** Which module owns the *concept* (Part III §8)? In the first hour the answer is
174
+ "the workspace module" and that is correct; the question still gets asked, because the answer
175
+ decides where a later extraction cuts.
176
+ 12. **What is deliberately NOT modelled.** Every requirement contains entities you should refuse —
177
+ the one-off, the derivable, the thing git already records, the report that is a filter over an
178
+ existing collection. Naming them in the proposal is what makes the model's boundary a decision
179
+ instead of an accident.
180
+
181
+ ### 7. The proposal — the output contract
182
+
183
+ The interview's output is **not prose**. Show, before anything is written:
184
+
185
+ 1. The **descriptor YAML** per collection, complete — YAML in the proposal cannot drift from what
186
+ gets written.
187
+ 2. **One sample record each**, as the `dt add` command that creates it, run for real or shown
188
+ verbatim. Seeding one real record before declaring the schema catches half the field mistakes —
189
+ the missing unit, the enum value the domain actually spells differently, the id that comes out
190
+ wrong.
191
+ 3. The **`dt schema` commands** (or the hand-written descriptor when the collection is
192
+ module-owned or carries comments worth keeping).
193
+ 4. The **"deliberately not modelled"** list, each with its one-line reason.
194
+ 5. For each relation: which side owns, whether an inverse is declared, and the answer to "which X
195
+ have no Y" that justifies it.
196
+
197
+ Then stop. The operator decides what enters their workspace (`before-you-build.md`). After a yes:
198
+ write, `compile`, `check`, seed the sample, and only then build the skills and commands that write
199
+ the collection — behaviour follows shape.
200
+
201
+ ---
202
+
203
+ ## Part III — architecture: modules, namespaces, boundaries
204
+
205
+ ### 8. What makes a module
206
+
207
+ A module is the unit of **concept ownership**, and its acceptance test is mechanical: **it compiles
208
+ alone in a bare workspace.** Everything else follows from that.
209
+
210
+ - **A field or collection belongs to the module that owns the CONCEPT, not the module that happens
211
+ to need it first.** A company's LinkedIn page is generic CRM and belongs with the CRM
212
+ collections; a field encoding one workspace's enrichment convention belongs in that workspace's
213
+ own module as an overlay. Asking "who owns the concept" per field is what keeps a later
214
+ extraction a `git mv` instead of a surgery.
215
+ - **Self-containment is structural, not aspirational.** No `templates:` and no `extends:` may
216
+ reach into a sibling module — a shared field set is duplicated per module under a scoped name
217
+ (`crm-provenance`, `rnd-provenance`) precisely so each module stays copyable. This is the single
218
+ most expensive boundary mistake known: a module whose descriptors referenced a template living in
219
+ the *consuming* workspace could not compile anywhere else, and nothing said so until someone
220
+ tried.
221
+ - **A cross-module relation is a declared dependency.** An inverse stamps a generated field onto
222
+ another module's collection, so compile refuses it unless the declaring module lists the
223
+ target's module in its dependencies — a module may not grow fields on a stranger silently. (The
224
+ workspace module is exempt; it overlays everything by design.)
225
+ - **Start in the workspace module; extract on the second consumer, not the first hunch.** The meta
226
+ verbs write the workspace module, and that is the right first home for everything. A module is
227
+ worth extracting when its collections form a closed reference graph, when a second workspace
228
+ wants it, or when its vocabulary has stabilised — not before. Premature extraction buys a
229
+ boundary you will immediately need to breach.
230
+ - **A module ships behaviour with its shape.** The skills that write a collection, the commands
231
+ bound to it, and its default views belong in the same module as the descriptor. A collection in
232
+ one module written only by a skill in another is a hidden dependency.
233
+ - **Module-shipped entities must not name a person, an account, or a machine path.** Per-install
234
+ values come from `.env` via declared vars; who-did-what lives in a collection the workspace owns.
235
+ A hard-coded `contacts/<someone>` resolves in exactly one workspace on earth.
236
+
237
+ ### 9. Namespaces — one prefix per module, and the empty one
238
+
239
+ A namespace is a folder-shaped prefix on collection names (`health/doctors` →
240
+ `data/health/doctors/`, referenced as `health/doctors/dana-levi`). Use one namespace per domain
241
+ module, and give the *commons* — the collections everything else points at — the **empty** prefix,
242
+ so the things used daily are spelled shortest (`contacts`, `tasks`, `meetings`). Rules that keep it
243
+ sane:
244
+
245
+ - **`ns == module`.** A namespace that spans modules, or a module that scatters across namespaces,
246
+ makes "who owns this" a lookup instead of a glance.
247
+ - A collection drops its module prefix from its bare name **iff** the prefix equals the namespace
248
+ (`health-visits` → `health/visits`); a prefix that names the *subject* rather than the box stays
249
+ in the name (`meeting-summaries` keeps `meeting-` even in the commons — it is about meetings, not
250
+ owned by a "meeting" namespace).
251
+ - Declare the namespace before the first collection compiles — an id is also a slash path, so an
252
+ undeclared prefix is ambiguous and compile refuses it rather than guessing.
253
+ - Namespacing an existing collection later is `dt schema rename-collection <old> --namespace <ns>`
254
+ — one commit, every inbound reference rewritten. Cheap, so do not agonise up front.
255
+
256
+ ### 10. Templates vs extends vs copy
257
+
258
+ Three ways to share shape, in strictly descending order of preference:
259
+
260
+ | mechanism | what it is | use when | never for |
261
+ |---|---|---|---|
262
+ | `templates:` | a live shared field set, merged at every compile; precedence template < base < overlay | the same field group on several collections in ONE module — provenance quads, funnel fields, address blocks | reaching across modules |
263
+ | `extends:` | a workspace/overlay descriptor adding fields to another module's collection | a workspace tightening or extending a module it installed | taxonomy ("a lead is-a engagement"); removing inherited fields (impossible by design) |
264
+ | copy | duplicating fields into a second descriptor | crossing a module boundary (each side owns its copy, scoped name), or when two shapes are about to diverge | "saving time" inside one module — that is `templates:` |
265
+
266
+ The instinct to resist is building a type hierarchy. When two collections share most fields, the
267
+ model wants a shared **template** plus two thin descriptors — or a reference between them — never a
268
+ base collection that exists only to be extended. An abstraction with one concrete consumer is a
269
+ roadmap, not a model; wait for the second consumer.
270
+
271
+ ### 11. Where records live: `data/` vs `state/`, codecs, shapes
272
+
273
+ - **`data/` holds content; `state/` holds operational records** — cursors, run markers, cached
274
+ readings; things a human never edits and would not mourn. The split is a `storage.path` choice
275
+ and it matters because `data/` is what gets read, searched and reasoned over; polluting it with
276
+ machinery makes every list noisier.
277
+ - **`codec: md` whenever a human reads a body; `yaml` when nobody does.** A record that is all
278
+ fields and no prose (a lab value, a transaction, a cursor) is `yaml` — the body would only ever
279
+ be empty. `json` exists for tool-written records.
280
+ - **`codec: file` makes the record an opaque file** — an icon, a logo, an image the UI draws.
281
+ Fields are derived (`ext`, `bytes`), writes go through `add --from <path>`, and `check` guards
282
+ `max_bytes` (default 200 KB). A big binary is not a record: it lives outside the vault under a
283
+ declared var, with an ordinary record carrying the `${env:...}` template that points at it.
284
+ - **`shape: folder` when a record is intrinsically several files** (a skill with references beside
285
+ it). Rare; prefer one file until the record itself demands companions.
286
+ - **Machine-specific paths are templates, never absolute paths.** `${env:FILES_FOLDER}/…` is inert
287
+ data rendered per machine by `dt resolve`; an absolute path in a record is wrong on every other
288
+ machine, silently. **A files folder is named after the collection or field that indexes it, and
289
+ the path below it is the record id** — `<FILES_FOLDER>/visit-recordings/<record id>.m4a` needs no
290
+ lookup table, which is the point.
291
+
292
+ ### 12. Graduating a module — strip the identity, keep the measurement
293
+
294
+ A module built inside one workspace accumulates that workspace's identity: names in examples, env
295
+ conventions, decision references. Making it reusable is mostly deletion. The rule for what
296
+ survives: **a measurement keeps its numbers and loses its source's name** ("a 4,000-record
297
+ workspace checks in half a second", not "workspace X does"). Verify the graduation the same way a
298
+ module is defined: compile it alone in a virgin workspace, with a synthetic cast in every example.
299
+
300
+ ---
301
+
302
+ ## Part IV — collections
303
+
304
+ ### 13. Grain — what one record is
305
+
306
+ The grain is the single most consequential choice; everything else is adjustable later, the grain
307
+ only by rewriting every record. Choose the **largest unit that answers the collection's defining
308
+ question without unpacking** — and test it with the two-question drill:
309
+
310
+ > For each question the collection exists to answer: can a *filter* answer it, or does something
311
+ > have to open records and parse?
312
+
313
+ *The clinic's lab results.* "Show LDL over ten years" and "everyone's latest vitamin D" are the
314
+ questions. A panel-shaped record (one visit's bloods as a nested table) answers neither without
315
+ unpacking — so the grain is **one measured value**: one analyte, one patient, one date. The panel
316
+ is reconstructed by filtering patient+date; the timeline by patient+analyte. Twenty records where
317
+ one "document" would have been — and both questions are now one filter each. The reverse also
318
+ holds: a `visits` record is NOT split into one-record-per-symptom, because no question filters
319
+ across symptoms — they are read within one visit, so they stay inside it.
320
+
321
+ Grain heuristics:
322
+
323
+ - A record should be **claimable by one sentence**: "the LDL measured for Dana on 2026-07-02". If
324
+ the sentence needs "and", the grain may be too coarse.
325
+ - If two questions need two grains, the finer grain wins and the coarser one becomes a filter —
326
+ never model both (that is a copy with no keeper, §3).
327
+ - **Events are finer than you think; entities are coarser.** A payment, a measurement, a message
328
+ is one record per occurrence. A person, a company, a product is one record per identity — with
329
+ the churn pushed into an events collection beside it (§16).
330
+
331
+ ### 14. Identity — ids that stay true
332
+
333
+ The id is the record's address in every reference, filename, and URL. It cannot be casually changed
334
+ (rename rewrites every inbound reference — supported, but a commit-sized event), so derive it from
335
+ what is true at creation and stays true:
336
+
337
+ - **Creation-time values the record owns.** The domain's own date plus a slug of the name is the
338
+ workhorse: `{{ date | date }}--{{ name | slug }}`. Never write-time (`created`) for domain
339
+ events, never a mutable field (a title that gets edited), never an external id that might be
340
+ re-keyed.
341
+ - **A time prefix for unbounded growth**: `{{ date | date:YYYY/MM }}/…` shards the folder by
342
+ month, keeps `ls` fast and scannable, and files the record where a human would look. The
343
+ `id.pattern` must then admit `/`.
344
+ - **A subject prefix when the collection partitions by an anchor**: `<patient>/<date>--<analyte>`
345
+ puts everything about one patient under one folder — right when the dominant access is "all of X
346
+ for this person", wrong when it is time-global.
347
+ - **`id.pattern` must accept everything `id.generate` can produce.** Non-latin names slug to a
348
+ deterministic short hash, so `[a-z0-9-]` holds; test with a real awkward title before shipping.
349
+ - **Sameness is a relation property, not an id property.** "One summary per meeting" is
350
+ `x-unique: true` on the summary's `meeting` reference — enforced by `check`, naming both records
351
+ on a collision — not a convention about ids that nothing enforces.
352
+
353
+ ### 15. Lifecycle — states as data
354
+
355
+ A `status` enum is the collection's spine: the board groups by it, bindings gate on it, and "what
356
+ needs attention" is a filter over it. Design it deliberately:
357
+
358
+ - **States are observations, not aspirations.** Each value should be assignable by looking at the
359
+ record and the world — `identified / first-contact / active-discussions / converted / dropped`
360
+ can each be verified; `almost-done` cannot.
361
+ - **Separate axes stay separate fields.** Funnel stage and temperature ("is it moving right now"),
362
+ readiness and whether-a-human-acted — collapsing two judgments into one enum makes both
363
+ unreadable. Two enums, each honest, beat one that lies.
364
+ - **Terminal states, not deletion.** `archived`/`dropped` keep the record filterable and its
365
+ inbound references valid. `rm` is for mistakes, not lifecycle — and it refuses while anything
366
+ points at the record (`x-on-delete: restrict` is the default posture for exactly this reason).
367
+ - **The transition is a verb.** When moving between states has steps, that is a command bound to
368
+ the collection with `can-enter`/`can-exit` gates on the fields — not a fatter enum, and not
369
+ prose in the description hoping to be followed.
370
+ - **Never enum against data that already violates it.** Declaring `enum:` over a field whose
371
+ records hold other values makes `check` fail on every one; clean first, then narrow (Part IX).
372
+
373
+ ### 16. Stable vs volatile, singletons, documents vs entities
374
+
375
+ Three recurring shapes, each with a rule:
376
+
377
+ - **The thing / the activity about the thing.** `patients` (stable identity, edited in place) and
378
+ `visits` (append-only events pointing at a patient). Mixing them — visit notes accumulating on
379
+ the patient record — destroys both: the entity's diff becomes noise and the events lose their
380
+ grain. The stable record carries only what is *currently true*; everything dated hangs off it.
381
+ - **The singleton-per-subject document.** One living dossier per patient, one page per engagement:
382
+ enforce it with `x-unique` on the subject reference, edit in place, and let git hold every
383
+ superseded version. The alternative — dated versions as records — is right only when versions
384
+ are *compared* as data ("what changed between Q1 and Q2 reports"), which is rare.
385
+ - **Documents vs entities.** A brief, a report, an analysis is a *document*: its value is the
386
+ body, its fields exist to file and find it (date, subject references, status). An entity's value
387
+ is its fields. Do not force documents into entity shapes — a document collection with fifteen
388
+ required fields will simply stop being written — and do not let an entity's facts hide in a
389
+ body where no filter can reach them.
390
+
391
+ ### 17. The forcing field
392
+
393
+ One required field whose absence makes the record pointless: `evidence` on a defect (a claim
394
+ without a repro is a rumour), `done_when` on a goal (a goal without a test is a mood), `dose` on a
395
+ prescription. It does the work review would: the writer must have the thing, not the intention.
396
+ Requiredness is for the forcing field and identity inputs — **not** for everything that is "nice
397
+ to have", because every extra `required` is a record that cannot be captured quickly and therefore
398
+ will not be captured at all. Capture-fast collections (inbox-like: raw ideas, unfiled notes) may
399
+ have *no* required field beyond a name — deliberately, stated in the description — and a digestion
400
+ step downstream.
401
+
402
+ ### 18. Descriptions and `use_when` — the model is also the prompt
403
+
404
+ In an agent-operated workspace, descriptor prose is not documentation; it is **retrieval surface**.
405
+ Collection descriptions are compiled into the orientation block every session loads, and field
406
+ descriptions are what an agent reads before writing a value. So:
407
+
408
+ - A collection's `description` carries the question it answers **and the neighbour it is NOT**:
409
+ "the person, never the org — that is `companies`". Confusable pairs each point at the other.
410
+ - A field's `description` says what the value MEANS, names the source when the value is copied
411
+ from elsewhere, and states the convention an agent must follow ("empty means unmatched — the
412
+ matching command's queue").
413
+ - `use_when` is authored **only** when an agent that fully understood the description would still
414
+ not reach for the collection — a search-here-first trigger, a write-here-when situation. It is
415
+ prose; nothing fires on it; and a `use_when` restating the description costs every session
416
+ tokens while diluting the few that carry real signal.
417
+ - Descriptions are the cheapest UX in the system: the same line is the tooltip in every surface,
418
+ the agent's guidance, and the future maintainer's note. Budget a real sentence per field.
419
+
420
+ ---
421
+
422
+ ## Part V — fields
423
+
424
+ ### 19. The ladder: collection · reference · enum · vocabulary · tags · nested — measured, not guessed
425
+
426
+ For every value, walk down until a rung holds. The measurements are two commands, not taste.
427
+
428
+ | the value… | model as | the measure |
429
+ |---|---|---|
430
+ | has fields of its own, a lifecycle, or is opened alone | a **collection**, pointed at with `x-reference` | — |
431
+ | is one of a small set the domain itself defines and closes | **`enum`** | `dt values <c> <f>` shows ≤ ~10 distinct values AND their counts sum to ≥ 80 % of the records — `values` reports distinct counts (never fill; compute fill against the record count), and echoes a declared enum verbatim without counts |
432
+ | is a vocabulary the workspace grows as it goes | **free string** — `dt values` derives the dropdown from the data, so a moving set needs no schema change and `check` never fails on a new value | the set is still moving, or predates any enum |
433
+ | is a loose label with no attributes | **`tags`** — an array of free strings. The moment labels read like `key:value`, that is a field wanting to exist: promote it | — |
434
+ | repeats inside one record and is never referenced from outside | **nested** `array` of `object`, with `x-title-template` on `items` so rows get a label | the costs are real: not filterable across records (a non-operator filter key means a reference hop, so a nested key narrows to nothing), skipped by `dt values`, unusable as a list column — the form and the body are its only readers |
435
+
436
+ Two hard "nevers": never declare an `enum` against records that already violate it (§15), and
437
+ never create a `tags` *collection* — a label with no attributes does not earn records, and the day
438
+ it grows attributes it becomes a real collection with a real name.
439
+
440
+ The promotion paths all run upward and are all cheap except the last: a free string becomes an
441
+ enum by declaration once the vocabulary settles; a tag becomes a field; a nested object becomes a
442
+ collection **only by a one-shot migration script**, which is why question 1 of the interview errs
443
+ toward collections whenever cross-record questions are conceivable.
444
+
445
+ ### 20. Every field attribute, and what it does downstream
446
+
447
+ Property order matters: **it is the form order**, and the `x-body` field goes last.
448
+
449
+ | attribute | decide by | what it drives downstream |
450
+ |---|---|---|
451
+ | `type` + `format` | the JSON type; `date` vs `date-time` honestly (§21) | the control every surface picks: date / datetime widgets, toggle, number, textarea for `format: markdown` |
452
+ | `title` | authored only when title-casing the name reads wrong (`ui-views` → `UI Views`) | the label everywhere; no surface invents its own casing |
453
+ | `description` | §18 | tooltip + agent guidance |
454
+ | `required` | the forcing field and identity inputs, nothing else (§17) | the asterisk; a write missing it is refused before disk |
455
+ | `enum` | the ladder | a dropdown; `check` enforcement |
456
+ | `default` | what is true when unstated — not what is common | prefilled on create; also documents the neutral value |
457
+ | `x-reference` | Part VI | the record picker, labelled by the target's `title_template`; `check` and `rename` follow it |
458
+ | `x-inverse` / `x-inverse-of` / `x-unique` / `x-on-delete` | Part VI | the generated mirror; cardinality; delete behaviour |
459
+ | `x-body: true` | the ONE long prose field | the markdown body; rendered as the page, last |
460
+ | `x-title-template` (on `items` of nested arrays) | give repeating rows a human label | the row label in the list editor |
461
+ | `templates:` (collection level) | a field set shared within the module | identical rows across collections, maintained in one file |
462
+
463
+ What is deliberately absent: computed fields (a skill or command keeps a denormalized key, §3),
464
+ field-level permissions (one operator over git), validation beyond JSON Schema plus the `x-`
465
+ keywords (a rule the schema cannot express belongs in the writing command's gate, where it can
466
+ refuse with a sentence).
467
+
468
+ ### 21. Time, money, measures
469
+
470
+ - **`date-time` for instants, `date` for days, never one in the other.** The engine compares
471
+ date-times across offsets correctly; a day stored as midnight lies about precision and breaks
472
+ "same day" filters across timezones.
473
+ - **Denormalized date keys are the sanctioned copy** (§3): a child record carrying the parent's
474
+ date so lists sort and ids generate without a join — with the source and writer named in the
475
+ description.
476
+ - **Money is `amount` + `currency`**, always, even when "it's all one currency today" — the first
477
+ foreign invoice is not the moment to migrate. Same for measures: `value` + `unit`, or the unit
478
+ in the field name (`duration_min`, `weight_kg`) when it is genuinely fixed.
479
+ - **Timestamps the engine already keeps are not fields.** `last-modified` is injected from git
480
+ into every list; creation is derivable from history. Author a date field only when it carries
481
+ *domain* time (the visit's date, not the record's).
482
+
483
+ ### 22. Naming
484
+
485
+ Names are read hundreds of times per write; optimize for the reader:
486
+
487
+ - **Collections: plural, hyphenated, the word the user says** — `lab-values`, `visits`. The
488
+ record suffix derives as the singular (`<id>.lab-value.md`). A collection is for what *recurs*;
489
+ a one-off is a record in some existing collection, not a new box.
490
+ - **Fields: `snake_case`; a reference named for the target's singular** (`visit`, `doctor`), the
491
+ mirror for the owner's plural (`visits`). Name the **role, not the instance**: `person`, not
492
+ `staff-member`; `source_account`, not the bank's name.
493
+ - **No abbreviations the user does not say aloud.** `qty` saves three characters and costs every
494
+ future reader a beat.
495
+ - **Confusable pairs get contrasting names AND contrasting descriptions** — `expense_transactions`
496
+ vs `reimbursement_transactions`, each saying which it is not.
497
+ - **Rename early.** A wrong name compounds daily, and `rename-collection` rewrites every inbound
498
+ reference in one commit. There is no field rename verb, so field renames cost a script — one
499
+ more reason to spend a minute on the name now.
500
+
501
+ ---
502
+
503
+ ## Part VI — relations
504
+
505
+ ### 23. The model: one owner, a generated mirror, declared from either side
506
+
507
+ A relation has exactly one **owning field** — the foreign key, on the many side — and optionally a
508
+ **generated mirror** on the far side. The engine writes the mirror into the compiled descriptor
509
+ (`readOnly`, with a hint naming the owner), maintains its values in the same write as every change
510
+ to the owner, **refuses direct writes to it** naming the field to set instead, and `check` reports
511
+ one fallen behind as stale with the repair command. You author one side; the other is kept.
512
+
513
+ Two source spellings, byte-identical once compiled — choose by where the sentence reads best:
514
+
515
+ ```yaml
516
+ # A — on the owning side (the foreign key):
517
+ # collections/visits.collection.yaml
518
+ doctor: { type: string, x-reference: health/doctors, x-inverse: visits }
519
+
520
+ # B — on the far side, as an authored field that names whose mirror it is:
521
+ # collections/health/doctors.collection.yaml
522
+ visits:
523
+ type: array
524
+ x-inverse-of: visits.doctor # <owner collection>.<owner field> — split at the LAST dot
525
+ items: { type: string, x-reference: visits }
526
+ description: Every visit this doctor has seen — generated; set `doctor` on the visit.
527
+ ```
528
+
529
+ Spelling B's one real advantage: **the mirror keeps a hand-written `description`** in the file a
530
+ reader opens, where spelling A's generated mirror gets a mechanical one. In a workspace where
531
+ descriptions feed the agents' orientation block, that is reason enough to prefer B. B also
532
+ composes with shared templates: when the owning field comes from a `templates:` set used by
533
+ several collections, an `x-inverse` on the template would generate the same mirror name from each
534
+ consumer and collide — B declares each mirror on its own target, touching no shared file.
535
+
536
+ Cardinality closes from the shapes, and one-to-many is **never authored** — it is always the
537
+ mirror of a scalar:
538
+
539
+ | owning field | far side | kind |
540
+ |---|---|---|
541
+ | scalar `x-reference` | array mirror | many-to-one |
542
+ | scalar + `x-unique: true` | scalar mirror | one-to-one |
543
+ | array `x-reference` (+ inverse) | array mirror | many-to-many |
544
+
545
+ A **scalar spelling-B mirror closes backwards**: declaring the far side as a scalar means the
546
+ foreign key is one-to-one, and compile stamps `x-unique: true` onto the owner for you.
547
+
548
+ `x-on-delete` says what removing a *target* does to records pointing at it: `restrict` (default)
549
+ refuses the `rm` and names them; `set-null` clears the foreign keys — refused at compile when the
550
+ key is `required` or a list with a floor above one, because a delete must not manufacture invalid
551
+ records.
552
+
553
+ ### 24. Choosing the shape — a decision table
554
+
555
+ | the relationship, in a sentence | model |
556
+ |---|---|
557
+ | each visit has one doctor; a doctor has many visits | scalar `doctor` on visits (+ inverse when §25 says yes) |
558
+ | each meeting has at most one summary, ever | scalar `meeting` on summaries + `x-unique` — sameness enforced, collisions named by `check` |
559
+ | an analysis covers several meetings; a meeting may be analysed repeatedly | array on the analysis (+ inverse) — many-to-many with no junction |
560
+ | the link itself has fields (a dosage, a role, a start date) | a **junction collection**: an ordinary collection with two references and the edge's own fields. Only when the edge genuinely carries data |
561
+ | the field may point at one of several collections | a union: `x-reference: [meetings, visits]` — the stored value's prefix disambiguates |
562
+ | evidence/source pointing anywhere | `x-reference: '*'` — open world, `check` verifies existence only, **no inverse possible** |
563
+ | an org's parent org | self-reference — legal, mirrors and all (`parent` → `subsidiaries`) |
564
+ | A and B reference each other with different meanings | two independent relations — never one field doing double duty |
565
+
566
+ ### 25. Direction — when to declare the inverse, when to skip it
567
+
568
+ The inverse is cost-free to *maintain* (the engine keeps it) but not free to *have*: it is lines in
569
+ every target record, churn in the target's diff whenever an owner changes, and a write to the
570
+ target's file on every link change. Decide per relation:
571
+
572
+ **Declare it when:**
573
+ - "which X have no Y" will be asked — the empty mirror is the only thing that makes absence
574
+ filterable (`--where '{"summary":{"_empty":true}}'`);
575
+ - the target record is *read* with its children in mind (a doctor's page should list the visits);
576
+ - a surface will show the far side as chips or a count.
577
+
578
+ **Skip it when:**
579
+ - only the outbound direction is ever asked ("this visit's doctor", never "doctors with no
580
+ visits");
581
+ - **fan-in would swamp the record** — measure it before declaring:
582
+ `dt list visits --where '{"doctor":{"_eq":"health/doctors/dana-levi"}}'` piped to a count. A
583
+ mirror of 40 refs on a record with 12 lines of its own frontmatter has inverted the record's
584
+ purpose; past a few hundred the file is majority-mirror and every reader pays for it;
585
+ - **the owning side is machine-written on every sync** — each sync write also touches the target,
586
+ making it a hot file under concurrent sessions (Part VIII §36);
587
+ - the target is a `codec: file`, runtime-generated, or bodyless-`md` collection — compile refuses
588
+ the mirror anyway, with the reason.
589
+
590
+ Leaving a reference one-way is not a compromise; for evidence fields, `'*'` fields, and
591
+ high-fan-in event streams it is the design.
592
+
593
+ ### 26. Denormalized keys vs mirrors
594
+
595
+ Both put related information on this side of a hop; they answer different needs and do not
596
+ substitute:
597
+
598
+ | | generated mirror | denormalized key |
599
+ |---|---|---|
600
+ | holds | references to the far side | one scalar copied from the far side |
601
+ | maintained by | the engine, same write | whichever skill/command writes the record — named in the description |
602
+ | exists for | "which have none", chips, the far list | sorting and id generation without a join |
603
+ | staleness | `check` reports; `relations rebuild` repairs | rots silently unless the writer is disciplined |
604
+
605
+ The classic pairing on an event record: a reference to the parent **and** the parent's date as a
606
+ denormalized sort key — the reference for identity and filters, the date because a list of ten
607
+ thousand events cannot resolve ten thousand hops to sort. Copy *keys*, never *content*: the parent
608
+ body readable from the child is what the reference is for.
609
+
610
+ ### 27. Integrity — what the machine holds, so the model can rely on it
611
+
612
+ Design against the enforcement that exists, not the enforcement you wish existed:
613
+
614
+ - `check` verifies: every reference resolves; mirrors agree with owners (else `stale`, naming
615
+ `dreamteamer relations rebuild <collection>`); `x-unique` collisions (naming both claimants);
616
+ enums, requireds, id patterns.
617
+ - The store maintains mirrors on `add`/`set`/`rm`/`revert`; refuses writes to mirrors; `rm`
618
+ honours `x-on-delete`, detaches its own mirrors, and refuses on unmanaged inbound references.
619
+ - `dt commit <collection>/<id>` sweeps the relation partners whose edge to the named record
620
+ changed — one logical change, one commit — and refuses when a partner also carries *another*
621
+ session's edge changes, naming everything, rather than publishing someone else's work.
622
+ - Schema ops clean up after themselves: dropping an inverse clears the generated values it
623
+ orphans; removing a populated field clears its values, reporting the count, with the previous
624
+ version in git.
625
+
626
+ What no machine holds: the discipline of a denormalized key (§26), the honesty of a status enum
627
+ (§15), and the grain (§13). Those live in descriptions and review.
628
+
629
+ ---
630
+
631
+ ## Part VII — UX: designing for the surface without naming one
632
+
633
+ ### 28. The contract
634
+
635
+ The engine emits, per field, a presentation row — type, label, description, options, and `special`
636
+ markers; per collection — title, icon, order, list fields, per-record title template. A surface
637
+ renders the contract; the model *is* the UI design. This is what keeps the engine detached from
638
+ any particular extension while still letting a modeler decide how things will look.
639
+
640
+ ### 29. Shape → projection — the full table
641
+
642
+ | descriptor shape | the engine emits | any surface shows |
643
+ |---|---|---|
644
+ | `x-body: true` | type `text`, special `dt-body`, edit `input-rich-text-md` | the record's page, rendered last |
645
+ | scalar `x-reference` | special `dt-relation-path` | a record picker, values labelled through the target's `title_template` |
646
+ | array `x-reference` | type `json`, specials `dt-relation-path`, `dt-relation-list` | a multi-picker; chips |
647
+ | generated mirror | `readonly: true`, `inverse_of`, the hint, special `dt-relation-mirror` | read-only chips + the hint; **never an editable control, the create form included** |
648
+ | `enum` | `edit_options.choices` | a dropdown |
649
+ | plain array of strings | type `json`, edit/view `tags` | chips |
650
+ | array of objects | type `json`, edit/view `list`, fields + `items.x-title-template` | a repeating-row editor |
651
+ | object with properties | type `json`, edit/view `nested` | a sub-form |
652
+ | `boolean` / `integer` / `number` | `boolean` / `integer` / `float` | toggle / number inputs |
653
+ | `format: date` / `date-time` | type `date` / `timestamp` | date / datetime controls |
654
+ | `format: markdown` (not the body) | type `text` | a textarea |
655
+ | `description` | `meta.description` | the tooltip |
656
+ | low-cardinality free string | *(no schema mark)* — `dt values` supplies the vocabulary | a dropdown of observed values, most-used first |
657
+
658
+ Three consequences worth designing with: a **mirror is knowable by its special**, so a surface can
659
+ disable the control with a reason instead of letting a write bounce; a **free string is already a
660
+ dropdown** wherever the surface asks `values`, so enums are for *closed* sets, not for getting a
661
+ picker; and everything here degrades to plain text gracefully, because the record is plain text —
662
+ the projection adds affordances, never meaning.
663
+
664
+ ### 30. Lists — the record's row
665
+
666
+ `list_fields` is the model's answer to "what does a scanning human need": **four to six columns** —
667
+ the name, the sort key, the status, the relation you filter by. The engine injects `last-modified`
668
+ itself. More columns is not more information; it is a horizontal scrollbar. A field that would
669
+ usually be empty in a list, and any body or nested table, does not go in one. A ui-view's
670
+ `columns` **replaces** `list_fields` — and a column naming a field the schema lacks is dropped
671
+ silently, so check the descriptor's real names when writing views.
672
+
673
+ ### 31. The record's face: `title_template`, icons, order
674
+
675
+ - `title_template` is how a record is labelled *everywhere it is referenced* — pickers, chips,
676
+ lists, links. Authored once on the collection, inherited by every reference field pointing at
677
+ it (a union inherits only when every member agrees). Compile derives one from
678
+ `title`/`name`/`subject`; **a derivation to `{{ id }}` is a smell** meaning no name-like field
679
+ exists — add one, because ids make terrible chips.
680
+ - `icon` (one material-symbols name) and `order` place the collection in any nav; the nav groups
681
+ by owning module. These are cheap and worth setting — an unnamed generic glyph in a tree of
682
+ twenty collections costs a glance every time.
683
+
684
+ ### 32. Forms — property order is the design
685
+
686
+ The form is the schema, top to bottom. So: identity first (name, date, the references that file
687
+ the record), then state (status and its axes), then detail, then the body last — which
688
+ `templates:` respects by inserting shared fields before `x-body`. Progressive disclosure is
689
+ achieved by what you *don't* make a field (§2: read-once information belongs in the body) and by
690
+ defaults (a field with a good default is a field nobody has to touch). Required-field discipline
691
+ (§17) is form design: every asterisk is friction at capture time.
692
+
693
+ ### 33. Views — when a `ui-view` earns existence
694
+
695
+ The default rendering — a table of `list_fields` over the whole collection — is itself an ordinary
696
+ default view; a named ui-view exists to be *different* from it. One earns its keep when it encodes
697
+ a recurring question (`/views/visits/unbilled`), a different layout genuinely fits the data
698
+ (kanban by `status`, calendar by a date field, map by a location), or a filtered slice is
699
+ someone's daily surface. A view that restates the fallback is a record to maintain for nothing.
700
+ Layouts come from the registered set (`table`, `cards`, `kanban`, `calendar`, `map`, plus
701
+ module-declared); filters are operator objects (`{status: {_eq: todo}}`), and a saved view is an
702
+ ordinary record — diffable, agent-writable, one per recurring question rather than one per mood.
703
+
704
+ ### 34. Searchable and filterable — the model side of "find it"
705
+
706
+ - **Filters reach one hop outbound** — any field on the record, and through a reference to the
707
+ target's fields (`{"doctor": {"specialty": {"_eq": "cardiology"}}}`), arrays with any-match
708
+ semantics. They do not reach inbound (that is what mirrors are for) and never two hops (that is
709
+ what denormalized keys are for). Model so every recurring question lands within one hop.
710
+ - The operator set is rich (`_eq`…`_between`, `_contains`, `_starts_with`, case-insensitive
711
+ variants, `_empty`/`_nempty`, `_regex`, `_and`/`_or`) — but **an unknown key or a dangling
712
+ reference narrows to nothing** rather than erroring, so a filter's field names deserve the same
713
+ care as code.
714
+ - **Grep is the other search engine.** Qualified references make every relation greppable
715
+ (`grep -r 'health/doctors/dana-levi'` finds every record pointing at her); prose wikilinks
716
+ (`[[collection/id|label]]`) keep body mentions findable too. A model whose links are bare names
717
+ has opted out of both.
718
+
719
+ ---
720
+
721
+ ## Part VIII — performance: what actually costs
722
+
723
+ ### 35. The cost model
724
+
725
+ Reads walk files. To first order: **`get` by id is O(1)** (the id is the path); **`list`, `check`,
726
+ `values`, and any filter are O(records in the collection)** — every record parsed, one pass; a
727
+ one-hop filter adds resolution of the referenced records it actually touches. Writes are O(1)
728
+ files touched (the record, plus its relation partners), and a scoped commit is O(what changed).
729
+ Orders of magnitude, measured on a real workspace: ~4,000 records across ~70 collections **checks
730
+ in about half a second** and compiles in a third of one; a single paired write (both sides of a
731
+ relation maintained) lands in ~0.2 s; regenerating ~100 mirror values is ~0.2 s. The practical
732
+ meaning: at personal-workspace scale, **model for legibility first — the performance budget is
733
+ fat**. The shapes below are the ones that actually spend it.
734
+
735
+ ### 36. Hot files and fan-in — the real concurrency cost
736
+
737
+ Git is the transaction log, and its unit is the file. Two writers touching *different* files merge
738
+ trivially; two writers touching the *same* file are a conflict, and the engine's own commits are
739
+ pathspec-scoped precisely to keep strangers out of each other's commits. The modeling
740
+ consequences:
741
+
742
+ - **A mirror concentrates writes.** Every link change on an owner also writes the target's file. A
743
+ target with hundreds of inbound links, or owners written by a fifteen-minute sync, makes that
744
+ one target file a contention point *and* a churn magnet in history. That is the §25 rule from
745
+ the performance side: skip the inverse on machine-written high-fan-in relations, or hang a small
746
+ stats record beside the anchor instead.
747
+ - **Event streams append; entities update.** An append-only collection (one new file per event)
748
+ has no hot files at all — which is why the stable/volatile split (§16) is also the concurrency
749
+ design.
750
+ - **Do not renumber.** Manual order uses a fractional index per moved record (`sort_field`, moved
751
+ by `dt <collection> move`) because renumbering a list is a multi-file commit against git for no
752
+ information.
753
+
754
+ ### 37. Growth — sharding and splitting
755
+
756
+ - **Shard folders by time in the id** (`YYYY/MM/`) once a collection will outgrow a few hundred
757
+ records — for the humans and the tooling both; a 10,000-entry directory helps nobody.
758
+ - **Split a collection when the questions split**, not when the count grows: an archive collection
759
+ is almost always wrong (one more place to look, and filters on `status`/date already answer
760
+ "active"). The exception is a *hot working set* pattern where a sync rewrites recent records
761
+ constantly — then a stable/volatile split of the collection itself can be justified, and should
762
+ be written up as such.
763
+ - **Frontmatter lean, body fat.** Every list and filter parses frontmatter; nothing scans bodies
764
+ unless asked. A machine-written collection with 40 frontmatter fields pays 40 fields × N records
765
+ on every list — push the read-once payload into the body or an attached file.
766
+
767
+ ### 38. What not to optimize
768
+
769
+ No indexes to design, no query plans to hint, no caches to invalidate — do not invent them. A
770
+ "summary table" collection maintained beside the real one is a copy with no keeper (§3); a
771
+ mirrored count field is a denormalized key nobody asked for; a nightly "rebuild" script is a smell
772
+ that the model, not the machinery, is wrong. When something is actually slow, measure which pass
773
+ is slow (`time` the command) before modeling around it — the honest fixes are usually "shard the
774
+ folder", "lean the frontmatter", or "drop the unread inverse", in that order.
775
+
776
+ ---
777
+
778
+ ## Part IX — evolution: changing a model that already holds records
779
+
780
+ ### 39. Additive first
781
+
782
+ Widening is always safe and needs no ceremony: a new optional field, a new enum value, a new
783
+ collection, a new inverse on an existing reference (declare it, then `relations rebuild` backfills
784
+ every mirror in one command — no script). Ship the widening, backfill opportunistically, narrow
785
+ later if ever.
786
+
787
+ ### 40. Narrowing — clean, then declare
788
+
789
+ A new `required`, a tightened `enum`, a pattern change: `check` will flood on every pre-existing
790
+ violation, so the order is fixed — measure (`dt values`, a filter for the outliers), clean the
791
+ data, *then* narrow the schema. Narrowing first "to see what breaks" makes every later `check`
792
+ useless until the flood is drained.
793
+
794
+ ### 41. Renames
795
+
796
+ - **Collections**: `dt schema rename-collection <old> <new>` — descriptor, records, filenames and
797
+ every inbound reference in one commit. Cheap; do it the day the name is wrong.
798
+ - **Fields**: there is no rename verb, deliberately (a rename that rewrites every record is a
799
+ migration, and pretending otherwise invites half-renames). The honest sequence: add the new
800
+ field; a one-shot script moving the values (committed with the records it rewrote, the commit
801
+ message being the ledger); `schema remove-field` the old one — which clears any leftovers and
802
+ reports the count.
803
+ - **Values** (an id, a reference target): `dt rename <collection>/<old> <new>` rewrites inbound
804
+ references; qualified prose wikilinks are followed, bare-name prose is not — one more reason
805
+ references in bodies are written qualified.
806
+
807
+ ### 42. Migrations are scripts, run once, committed with their effects
808
+
809
+ There is no migration framework — one shipped, went unused by every real schema change, and was
810
+ removed. A shape change across existing records is a one-shot script: written, run, verified with
811
+ `check`, committed together with the records it rewrote and a commit message saying what it did.
812
+ The message is the only ledger, so write it like one. Before any of it: does the change actually
813
+ need a migration, or is it additive (§39) plus a `relations rebuild`?
814
+
815
+ ---
816
+
817
+ ## Part X — anti-patterns: what wrong looks like
818
+
819
+ Each smell, with its fix. These are lint-shaped on purpose.
820
+
821
+ | smell | what it means | fix |
822
+ |---|---|---|
823
+ | a string field with ≤ 10 distinct values, ≥ 80 % fill, no enum | a vocabulary that has settled | declare the enum (after §40's cleaning) |
824
+ | a field at 0 % fill after fifty records | modelling an intention, not a practice | delete it, or find why writers skip it |
825
+ | an array whose values look like `<collection>/<id>` with no `x-reference` | a relation the machine cannot see | add `x-reference` — `check` and `rename` start working |
826
+ | `title_template` derived to `{{ id }}` | no name-like field | add one; ids make terrible chips |
827
+ | two collections sharing most fields, no reference between them | a hidden relation or a false split | reference, template, or merge — decided by the triage test (§4) |
828
+ | a `kind` enum whose members are triaged on different questions | two collections wearing one name | split |
829
+ | a scalar reference with fan-in > 1, no inverse, and a saved view asking "which have none" | the question exists, the mirror doesn't | declare the inverse |
830
+ | a body used as a list column | prose where a field belongs | extract the field the column actually wanted |
831
+ | a number with no unit beside it | a latent unit bug | `value` + `unit`, or the unit in the name |
832
+ | a nested object someone tries to filter on | grain too coarse (§13) | promote it to a collection |
833
+ | a `tags` value shaped `key:value` recurring | a field wanting to exist | promote it |
834
+ | a required field writers routinely fake (`unknown`, `-`) | requiredness as wish (§17) | make it optional and let emptiness be the signal |
835
+ | `status: almost-done` | aspiration in an enum (§15) | states you can observe |
836
+ | a second record that is "v2" of an existing one | versioning by copy | edit in place; git holds the old version |
837
+ | a collection nothing points at and no view lists | a model without questions (§2) | delete it, or find its question |
838
+ | an `owner`/`assignee` field in a single-operator workspace | imported enterprise reflex | delete; git already says who wrote what |
839
+ | a copied field with no source named in its description | a copy with no keeper (§3) | name source + writer, or drop the copy |
840
+ | an absolute machine path in a record | breaks on every other machine | a `${env:VAR}` template + a declared var |
841
+ | a "misc"/"notes" collection growing unrelated shapes | a pile wearing a schema | leave piles as capture collections *by design* (§17), and digest into real ones |
842
+
843
+ ---
844
+
845
+ ## Part XI — worked example: the clinic, end to end
846
+
847
+ The requirement, verbatim: *"track the clinic's visits and what gets prescribed, and stop losing
848
+ lab results."*
849
+
850
+ **The interview, compressed.** Nouns: patients, doctors, visits, prescriptions, lab results — five
851
+ candidates; "symptoms" appear only inside a visit, and nobody filters on them → body prose.
852
+ Questions: "what happened with this patient" (visits, by patient, newest first); "what is Dana
853
+ currently on" (prescriptions filtered by patient + status); "LDL over time" (lab-values by patient
854
+ + analyte — the grain question, settled as one value per record). Identity: a visit is
855
+ `date + patient`; a lab value is `patient + date + analyte`, and its id leads with the patient
856
+ because the dominant access is per-person. Lifecycle: prescriptions are the only state-bearing
857
+ collection (`active / stopped / completed`); visits are events, done the moment they are written.
858
+ Stable/volatile: `patients` stable, everything else events pointing at it. Relations: all the
859
+ event collections point at `patients`; visits also at `health/doctors`; prescriptions at the
860
+ visit that issued them. Inverses: `patients.visits` yes ("patients not seen this year" is a real
861
+ question) — but **not** `patients.lab_values`: fan-in measured in the hundreds per patient would
862
+ drown the record, and the per-person question is already answered by the lab-values id prefix and
863
+ a filter. Forcing fields: `dose` on a prescription; `value` + `unit` on a lab value. Volume: tens
864
+ of visits a week — `YYYY/MM` sharding on visits and prescriptions, patient-prefix on lab-values,
865
+ none on patients. Module: `clinic`, namespace `health`, commons untouched.
866
+
867
+ **The proposal** (two of the five descriptors — the ones that carry the lessons):
868
+
869
+ ```yaml
870
+ # modules/clinic/collections/health/lab-values.collection.yaml
871
+ name: health/lab-values
872
+ description: >-
873
+ ONE measured value — one analyte, one patient, one date. The grain is deliberately a single
874
+ value, because the questions are "LDL over time" and "everyone's latest vitamin D", and a
875
+ panel-shaped record answers neither without unpacking. A panel is a filter on patient+date.
876
+ storage: { path: data/health/lab-values, codec: yaml, shape: file, suffix: lab-value }
877
+ id:
878
+ generate: "{{ patient | basename }}/{{ date | date }}--{{ analyte | slug }}"
879
+ pattern: "^[a-z0-9-]+/\\d{4}-\\d{2}-\\d{2}--[a-z0-9-]+$"
880
+ schema:
881
+ type: object
882
+ required: [patient, date, analyte, value, unit]
883
+ properties:
884
+ patient:
885
+ type: string
886
+ x-reference: health/patients
887
+ description: Whose blood. First id segment, so one folder holds one person's history.
888
+ date:
889
+ type: string
890
+ format: date
891
+ description: The draw date — the lab's own, never the filing date.
892
+ analyte:
893
+ type: string
894
+ description: >-
895
+ What was measured, lowercase (ldl, hdl, vitamin-d) — a vocabulary, not an enum: labs keep
896
+ inventing panels, and `dt values` is the dropdown.
897
+ value:
898
+ type: number
899
+ description: The number alone — the unit is beside it, never inside it.
900
+ unit:
901
+ type: string
902
+ description: mg/dL, mmol/L, % — from the lab report, verbatim.
903
+ flag:
904
+ type: string
905
+ enum: [low, normal, high]
906
+ description: The lab's own flag when printed; empty when the report shows none.
907
+ source:
908
+ type: string
909
+ description: >-
910
+ Where this value came from — a report file template or a visit reference — so a
911
+ surprising number can be re-checked.
912
+ order: 33
913
+ list_fields: [patient, date, analyte, value, unit, flag]
914
+ icon: labs
915
+ ```
916
+
917
+ ```yaml
918
+ # modules/clinic/collections/health/prescriptions.collection.yaml (the relation lessons)
919
+ name: health/prescriptions
920
+ description: >-
921
+ One drug prescribed once — what, how much, and whether it is still running. "What is this
922
+ patient currently on" is a filter (patient + status=active), which is why status is a field and
923
+ not a sentence in the body.
924
+ storage: { path: data/health/prescriptions, codec: md, shape: file, suffix: prescription }
925
+ id:
926
+ generate: "{{ date | date:YYYY/MM }}/{{ date | date }}--{{ patient | basename }}--{{ drug | slug }}"
927
+ pattern: "^\\d{4}/\\d{2}/\\d{4}-\\d{2}-\\d{2}--[a-z0-9-]+--[a-z0-9-]+$"
928
+ schema:
929
+ type: object
930
+ required: [patient, drug, dose, date]
931
+ properties:
932
+ patient:
933
+ type: string
934
+ x-reference: health/patients
935
+ description: Who takes it.
936
+ visit:
937
+ type: string
938
+ x-reference: health/visits
939
+ description: >-
940
+ The visit that issued it — empty for a renewal issued between visits, which is itself the
941
+ signal that a renewal happened outside a consult.
942
+ drug:
943
+ type: string
944
+ description: Generic name, lowercase — a vocabulary.
945
+ dose:
946
+ type: string
947
+ description: 'The forcing field: a prescription without a dose is a rumour. As written — "20mg once daily".'
948
+ date:
949
+ type: string
950
+ format: date
951
+ description: Issued when — also the id's filing date.
952
+ status:
953
+ type: string
954
+ enum: [active, stopped, completed]
955
+ default: active
956
+ description: Observable states only — stopped is a decision, completed is the course run out.
957
+ notes:
958
+ type: string
959
+ format: markdown
960
+ x-body: true
961
+ description: Why prescribed, reactions, the story — read within one record, never filtered.
962
+ order: 32
963
+ list_fields: [patient, drug, dose, status, date]
964
+ icon: pill
965
+ ```
966
+
967
+ ```yaml
968
+ # and on health/patients — the far sides, spelling B so each mirror documents itself:
969
+ visits:
970
+ type: array
971
+ x-inverse-of: health/visits.patient
972
+ items: { type: string, x-reference: health/visits }
973
+ description: Every consult, generated — set `patient` on the visit. Empty means never seen.
974
+ prescriptions:
975
+ type: array
976
+ x-inverse-of: health/prescriptions.patient
977
+ items: { type: string, x-reference: health/prescriptions }
978
+ description: Everything ever prescribed, generated. "Currently on" is this filtered to active.
979
+ ```
980
+
981
+ **One sample record, seeded before declaring** — the step that catches the unit nobody mentioned:
982
+
983
+ ```
984
+ dt add health/lab-values --patient health/patients/dana-levi --date 2026-07-02 --analyte ldl \
985
+ --value 138 --unit mg/dL --flag high --source "health/visits/2026/07/2026-07-02--intake--dana-levi"
986
+ → data/health/lab-values/dana-levi/2026-07-02--ldl.lab-value.yaml
987
+ ```
988
+
989
+ **Deliberately not modelled:** appointments (the calendar owns scheduling; a visit is written when
990
+ it *happens*) · symptoms as fields (read within one visit — body prose) · a `patients.lab_values`
991
+ inverse (fan-in; the id prefix already files them per person) · a `doctors.patients` relation (it
992
+ is derivable through visits and would be a copy with no keeper) · invoices (a different module's
993
+ concept — declared out of scope rather than half-modelled).
994
+
995
+ Why this model holds up, checked against the parts: every recurring question is a one-hop filter
996
+ (I §2, VII §34) · the grains differ per collection and each was chosen by its question (IV §13) ·
997
+ the one risky inverse was measured and refused (VI §25) · every number carries its unit (V §21) ·
998
+ the statuses are observable (IV §15) · the module compiles alone and touches no commons (III §8).
999
+
1000
+ ---
1001
+
1002
+ ## Appendix — the modeler's checklists
1003
+
1004
+ **Proposing (before writing anything):**
1005
+
1006
+ - [ ] every collection's `description` names its question and its confusable neighbour
1007
+ - [ ] grain chosen per collection by the two-question drill, stated in the description when non-obvious
1008
+ - [ ] ids from creation-time owned values; patterns admit what generate produces; time-sharded where unbounded
1009
+ - [ ] each relation: owner side chosen, inverse declared-or-refused *with the reason*, on-delete stated
1010
+ - [ ] forcing field per collection — or the collection declared a capture pile on purpose
1011
+ - [ ] units beside every number; `date` vs `date-time` honest
1012
+ - [ ] `list_fields` scannable at 4–6 columns; `title_template` never `{{ id }}`
1013
+ - [ ] module ownership stated; no cross-module template/extends; namespace == module
1014
+ - [ ] the deliberately-not-modelled list exists
1015
+ - [ ] one sample record per collection, shown as its `dt add`
1016
+
1017
+ **Reviewing an existing model:** run the smells table (Part X) top to bottom; measure before
1018
+ declaring (`dt values`, fan-in counts, fill rates); prefer the additive fix; and when a change
1019
+ rewrites records, it is a one-shot script committed with its effects, whose commit message is the
1020
+ ledger.