@malloy-publisher/skills 0.1.40 → 0.1.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/eval-answer/SKILL.md +1 -1
- package/skills/eval-diagnose/SKILL.md +2 -2
- package/skills/eval-import/SKILL.md +1 -1
- package/skills/eval-import/reference/case-format.md +1 -1
- package/skills/eval-improve/SKILL.md +10 -6
- package/skills/eval-judge/SKILL.md +2 -2
- package/skills/eval-loop/SKILL.md +1 -1
- package/skills/eval-loop/templates/eval-run-package/README.md +2 -2
- package/skills/eval-loop/templates/eval-run-package/notebooks/eval_run.malloy +58 -50
- package/skills/eval-report/SKILL.md +1 -1
- package/skills/malloy-analysis/SKILL.md +1 -1
- package/skills/malloy-analysis-report/SKILL.md +88 -60
- package/skills/malloy-charts/SKILL.md +1 -1
- package/skills/malloy-dashboards/SKILL.md +1 -1
- package/skills/malloy-define/SKILL.md +2 -2
- package/skills/malloy-discover/SKILL.md +1 -1
- package/skills/malloy-document/SKILL.md +1 -1
- package/skills/malloy-getting-started/SKILL.md +1 -1
- package/skills/malloy-gotchas-modeling/SKILL.md +1 -1
- package/skills/malloy-html-data-apps/SKILL.md +1 -1
- package/skills/malloy-lookml-review/SKILL.md +1 -1
- package/skills/malloy-materialization/SKILL.md +1 -1
- package/skills/malloy-model/SKILL.md +21 -13
- package/skills/malloy-model/reference/analysis-to-model.md +1 -1
- package/skills/malloy-model-as-you-go/SKILL.md +1 -1
- package/skills/malloy-modeling/SKILL.md +1 -1
- package/skills/malloy-notebooks/SKILL.md +1 -1
- package/skills/malloy-phrase-detection/SKILL.md +1 -1
- package/skills/malloy-powerbi-review/SKILL.md +1 -1
- package/skills/malloy-publish/SKILL.md +2 -2
- package/skills/malloy-queries/SKILL.md +1 -1
- package/skills/malloy-reports/SKILL.md +89 -24
- package/skills/malloy-reports/reference/linear.md +32 -106
- package/skills/malloy-review/SKILL.md +1 -1
- package/skills/malloy-review/reference/rubric-structure.md +1 -1
- package/skills/malloy-run-agent/SKILL.md +1 -1
- package/skills/malloy-source-unreachable/SKILL.md +1 -2
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-answer
|
|
3
3
|
description: 'Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer. Run the contamination checklist, re-execute the submitted query yourself, then spawn a judge subagent per skill:eval-judge. Append attempt, tool_call, score, and retrieval_score events to the file ledger (reference/ledger-schema.md). Never explain the failure (eval-diagnose) or edit the model (eval-improve). Use when asked whether an answer was correct, to score a run, or to baseline a model.'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Evaluate One Answer
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-diagnose
|
|
3
3
|
description: 'Diagnose why a scored answer failed and who owns the fix, then cluster the failures by shared root cause. Walk dataset, agent-call, get_context/model, get_context/retrieval, construction, then model-definition. Append issue events to the file ledger, linked by traceId, one per cluster. Use after eval-answer, when triaging a run, or before changing a model. Does not edit the model (eval-improve).'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Diagnose One Answer
|
|
@@ -282,7 +282,7 @@ with both queries. Otherwise it is still `NOT-RETURNED` / `LOW-RANK`.
|
|
|
282
282
|
Use when the entity was found and used, and the definition or the data behind it
|
|
283
283
|
is wrong (bad grain, wrong join key, inverted filter). Owner: model. A doc whose
|
|
284
284
|
factual claim the data contradicts (a population statement, a grain claim) is
|
|
285
|
-
also model-definition: the
|
|
285
|
+
also model-definition: the field may compute correctly while its doc is false,
|
|
286
286
|
and an agent that trusts the doc answers wrongly without ever failing a query.
|
|
287
287
|
Probe the claim before writing the issue.
|
|
288
288
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-import
|
|
3
3
|
description: 'Turn a list of questions into an eval set, whatever shape it arrived in: a JSONL a customer sent, a CSV, a spreadsheet export, a markdown doc, an email thread, or a pull from production logs. Classify each item by what came WITH the question (a query, a number, prose criteria, or nothing), write cases.jsonl and set.json per reference/case-format.md, keep the file as it arrived, and seal each question so a later edit is detectable. Never marks a value-bearing golden verified and never invents a question. Use when questions arrive from outside and need to become a set, or before the first run against a set nobody here authored.'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Import questions into an eval set
|
|
@@ -60,7 +60,7 @@ trusting a name that looks familiar, and write down the mapping you used in
|
|
|
60
60
|
|---|---|---|
|
|
61
61
|
| `prompt`, `query`, `input`, `nl_question` | the question | `query` is the trap: in half of these it is the SQL or Malloy they ran, not the question |
|
|
62
62
|
| `answer`, `expected`, `output`, `ground_truth` | a golden value, prose, or both mixed in one string | a paragraph under `answer` is criteria, not a value. Classify it, do not store it as a number |
|
|
63
|
-
| `sql`, `malloy`, `gold_query` | their derivation | run it (`SKILL.md` step 3).
|
|
63
|
+
| `sql`, `malloy`, `gold_query` | their derivation | Malloy: run it (`SKILL.md` step 3). SQL: keep it as evidence of how they got the number, but do not run it; derive the number in Malloy over the truth package instead. Either way, this is the field that upgrades a typed number into something checkable |
|
|
64
64
|
| `criteria`, `rubric`, `assertions`, `must_include` | prose criteria | apply the query test in `SKILL.md` step 4 to EACH one, not to the field |
|
|
65
65
|
| `id`, `case_id`, `uuid` | their id | keep it, prefixed. Two sets from one customer collide otherwise |
|
|
66
66
|
| `difficulty`, `category`, `tags` | their own labels | keep as-is on the case for slicing. Never let one become `split` |
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-improve
|
|
3
3
|
description: 'Make the smallest safe edit that closes a diagnosed gap the model repo owns: a Malloy model edit, or a package skill under skills/. Every factual claim carries a probe receipt. Use after eval-diagnose, or when asked to fix a model so an agent can discover the right answer. Never accepts its own edit; the acceptance check belongs to eval-loop. Does not decide whether an answer was wrong (eval-answer) or why (eval-diagnose).'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Improve the Model
|
|
@@ -68,7 +68,7 @@ data with no question in front of them?* Reject:
|
|
|
68
68
|
|
|
69
69
|
- a field that hard-codes this question's filter and serves no other question
|
|
70
70
|
- this question's text, qid, or expected numbers in a doc, comment, or name
|
|
71
|
-
- a join copied from
|
|
71
|
+
- a join copied from a golden's query that you have not probed as a real relationship
|
|
72
72
|
|
|
73
73
|
The golden is a hypothesis source. The data is still the verifier.
|
|
74
74
|
|
|
@@ -77,12 +77,16 @@ The golden is a hypothesis source. The data is still the verifier.
|
|
|
77
77
|
Every structural claim needs a query you ran: join key, primary key, filter
|
|
78
78
|
value existence, snapshot assumption, value space, cardinality.
|
|
79
79
|
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
80
|
+
```malloy
|
|
81
|
+
run: t -> { aggregate: row_count is count(), distinct_keys is count(col) }
|
|
82
|
+
run: b -> { group_by: k; aggregate: n is count(); order_by: n desc; limit: 5 }
|
|
83
|
+
run: t -> { group_by: col; aggregate: n is count(); order_by: n desc; limit: 5 }
|
|
84
84
|
```
|
|
85
85
|
|
|
86
|
+
Run probes in Malloy, against the raw tables (the truth package, or a bare
|
|
87
|
+
`connection.table()` source), never as SQL. The second line is the join probe:
|
|
88
|
+
any `n` above 1 means `k` is not a key on `b`, so a `join_one` on it fans out.
|
|
89
|
+
|
|
86
90
|
A false `primary_key` compiles and silently corrupts every aggregate. Of one
|
|
87
91
|
pilot's 11 accepted edits, 4 of 5 wrong ones died to a single
|
|
88
92
|
`COUNT(*)` vs `COUNT(DISTINCT …)` probe that was never run.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-judge
|
|
3
3
|
description: 'Decide whether ONE answer matches its golden, and say whether you believe the golden. Read this before emitting any verdict. Covers containment, column pairing, near_match, refusals, and the gold_status judgement. Use when scoring an attempt in an evaluation run; never to conduct a run (eval-loop), diagnose a failure (eval-diagnose) or edit a model (eval-improve).'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# The judge
|
|
@@ -135,7 +135,7 @@ Output, exactly this shape:
|
|
|
135
135
|
plainly the answer explained itself, and however the rubric is worded. Say
|
|
136
136
|
in `why` that you are overriding a rubric clause, so the clause gets fixed.
|
|
137
137
|
This rule exists because a set shipped one: a question asked for sales over
|
|
138
|
-
the company's season, the answer gave the meteorological window
|
|
138
|
+
the company's season, the answer gave the meteorological window 20% lower
|
|
139
139
|
and said which window it used, and a clause granting `near_match` for a
|
|
140
140
|
stated window kept a materially wrong answer out of the pass rate
|
|
141
141
|
entirely -- the arm reported 100%.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-loop
|
|
3
3
|
description: 'Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint. You are the conductor: import cases into the file ledger, spawn a blind answerer, then run eval-answer, eval-diagnose, and eval-improve. Persistence is plain files: the set in the model package''s evals/ directory, runs in the set''s workdir; checkpoints are git commits of the model repo. `eval.py` runs each step from the set''s eval.toml. Use to score a model, diagnose failures, improve behind an acceptance check, or roll back a bad direction.'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# The Evaluation Loop
|
|
@@ -37,8 +37,8 @@ Both, and the split is not stylistic.
|
|
|
37
37
|
**`notebooks/eval_run.malloy`** holds the analytical tables: pass rate, cost, effort,
|
|
38
38
|
where the failures are, retrieval, the backlog. Publisher renders it natively, so
|
|
39
39
|
these are Malloy reading the model directly with no JavaScript in between and
|
|
40
|
-
nothing to drift. It is
|
|
41
|
-
|
|
40
|
+
nothing to drift. It is a linear report of text and query tiles, so it opens in
|
|
41
|
+
the Console's builder like any other report.
|
|
42
42
|
|
|
43
43
|
**`public/index.html`** leads with the score and where every question landed,
|
|
44
44
|
then the diagnosis backlog (failures grouped by shared cause), an effort strip
|
|
@@ -1,10 +1,38 @@
|
|
|
1
1
|
##" One or more scored runs over one evaluation set: the analytical tables.
|
|
2
|
-
|
|
2
|
+
##| artifact { layout=linear title="Eval run"
|
|
3
|
+
tiles=[
|
|
4
|
+
intro { kind=text },
|
|
5
|
+
the_headline { kind=text },
|
|
6
|
+
"run_summary",
|
|
7
|
+
is_the_headline_trustworthy { kind=text },
|
|
8
|
+
"attempts_tiles -> run_health",
|
|
9
|
+
what_it_cost { kind=text },
|
|
10
|
+
"cost",
|
|
11
|
+
"effort",
|
|
12
|
+
where_the_failures_are { kind=text },
|
|
13
|
+
"failures_by_where_to_fix",
|
|
14
|
+
pass_rate_by_coverage { kind=text },
|
|
15
|
+
"pass_by_coverage",
|
|
16
|
+
retrieval_intro { kind=text },
|
|
17
|
+
"retrieval_as_used",
|
|
18
|
+
most_missed_intro { kind=text },
|
|
19
|
+
"most_missed_entities",
|
|
20
|
+
what_to_do_next { kind=text },
|
|
21
|
+
"backlog",
|
|
22
|
+
coverage_gaps_in_the_set_itself { kind=text },
|
|
23
|
+
"coverage_gaps",
|
|
24
|
+
refusal_intro { kind=text },
|
|
25
|
+
"refusal_behaviour",
|
|
26
|
+
comparing_arms { kind=text },
|
|
27
|
+
"flips",
|
|
28
|
+
cases_every_arm_failed { kind=text },
|
|
29
|
+
"failed_everywhere"
|
|
30
|
+
]
|
|
31
|
+
}
|
|
32
|
+
|##
|
|
3
33
|
import "../eval_run.malloy"
|
|
4
34
|
|
|
5
|
-
##|(markdown)
|
|
6
|
-
# Eval run
|
|
7
|
-
|
|
35
|
+
##|(markdown) intro
|
|
8
36
|
One or more scored runs over one evaluation set. Publisher renders this natively,
|
|
9
37
|
so every table below is Malloy reading the same model the app reads -- there is no
|
|
10
38
|
second copy of the logic to drift.
|
|
@@ -18,7 +46,7 @@ Read it top to bottom: the headline, then whether the headline is trustworthy,
|
|
|
18
46
|
then where the failures are, then what to do about them.
|
|
19
47
|
|##
|
|
20
48
|
|
|
21
|
-
##|(markdown)
|
|
49
|
+
##|(markdown) the_headline
|
|
22
50
|
## 1. The headline
|
|
23
51
|
|
|
24
52
|
Pass rate over **decided** verdicts -- `match` against `no_match`. Two things are
|
|
@@ -34,9 +62,7 @@ answered the same cases. Read `near_matches` too. It is the leading indicator of
|
|
|
34
62
|
rubrics going vague, and it moves before the pass rate does.
|
|
35
63
|
|##
|
|
36
64
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
##|(markdown)
|
|
65
|
+
##|(markdown) is_the_headline_trustworthy
|
|
40
66
|
## 2. Is the headline trustworthy?
|
|
41
67
|
|
|
42
68
|
A pass rate means nothing if the attempts behind it were contaminated, if the
|
|
@@ -47,18 +73,7 @@ not evidence about the model. Harness errors are excluded from the rate rather
|
|
|
47
73
|
than counted as failures: a timeout is the eval failing, not the answerer.
|
|
48
74
|
|##
|
|
49
75
|
|
|
50
|
-
|
|
51
|
-
group_by: arm is runs.label
|
|
52
|
-
aggregate:
|
|
53
|
-
attempt_count
|
|
54
|
-
submitted_count
|
|
55
|
-
contaminated_count
|
|
56
|
-
harness_errors
|
|
57
|
-
judge_version is runs.judge_version.max()
|
|
58
|
-
set_version is runs.set_version.max()
|
|
59
|
-
}
|
|
60
|
-
|
|
61
|
-
##|(markdown)
|
|
76
|
+
##|(markdown) what_it_cost
|
|
62
77
|
## 3. What it cost
|
|
63
78
|
|
|
64
79
|
The claim a semantic model makes is not only that the agent gets the right
|
|
@@ -67,11 +82,7 @@ An arm that wins on accuracy while costing three times as much has not obviously
|
|
|
67
82
|
won.
|
|
68
83
|
|##
|
|
69
84
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
run: effort
|
|
73
|
-
|
|
74
|
-
##|(markdown)
|
|
85
|
+
##|(markdown) where_the_failures_are
|
|
75
86
|
## 4. Where the failures are
|
|
76
87
|
|
|
77
88
|
Every failure carries exactly one **where to fix**, so these counts sum to the
|
|
@@ -94,9 +105,7 @@ tables below disagree with each other.
|
|
|
94
105
|
neither.
|
|
95
106
|
|##
|
|
96
107
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
##|(markdown)
|
|
108
|
+
##|(markdown) pass_rate_by_coverage
|
|
100
109
|
### Pass rate by coverage
|
|
101
110
|
|
|
102
111
|
Coverage is the ceiling on what any agent can do. A model-shaped problem shows up
|
|
@@ -107,9 +116,7 @@ only by assembling it, and the agent could not.
|
|
|
107
116
|
declined.
|
|
108
117
|
|##
|
|
109
118
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
##|(markdown)
|
|
119
|
+
##|(markdown) retrieval_intro
|
|
113
120
|
## 5. Retrieval
|
|
114
121
|
|
|
115
122
|
**Per-question entity recall**: of the entities each golden answer depends on, how
|
|
@@ -119,18 +126,14 @@ terms the answerer chose, so it moves when the answerer changes; use it to
|
|
|
119
126
|
attribute a failure *within* one arm, not to compare retrieval across arms.
|
|
120
127
|
|##
|
|
121
128
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
##|(markdown)
|
|
129
|
+
##|(markdown) most_missed_intro
|
|
125
130
|
### Entities most often needed and not returned
|
|
126
131
|
|
|
127
132
|
The ranked retrieval backlog. An entity high on this list is one fix that several
|
|
128
133
|
cases are waiting on.
|
|
129
134
|
|##
|
|
130
135
|
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
##|(markdown)
|
|
136
|
+
##|(markdown) what_to_do_next
|
|
134
137
|
## 6. What to do next
|
|
135
138
|
|
|
136
139
|
Failures grouped by shared root cause, ordered by how many cases one edit would
|
|
@@ -142,9 +145,7 @@ that nobody stands behind is worse than a blank, because a reviewer has to
|
|
|
142
145
|
disprove it.
|
|
143
146
|
|##
|
|
144
147
|
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
##|(markdown)
|
|
148
|
+
##|(markdown) coverage_gaps_in_the_set_itself
|
|
148
149
|
### Coverage gaps in the set itself
|
|
149
150
|
|
|
150
151
|
Independent of any run: cases the model cannot answer outright, with the entity
|
|
@@ -152,9 +153,7 @@ that would close the gap. This is the model-improvement backlog even before an
|
|
|
152
153
|
agent has attempted anything, and it is the part a customer can act on directly.
|
|
153
154
|
|##
|
|
154
155
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
##|(markdown)
|
|
156
|
+
##|(markdown) refusal_intro
|
|
158
157
|
## 7. Refusal behaviour
|
|
159
158
|
|
|
160
159
|
Refusal is scored in both directions, and getting one right while getting the
|
|
@@ -167,9 +166,7 @@ other wrong is a common and misleading result.
|
|
|
167
166
|
looking well-calibrated on the ones above.
|
|
168
167
|
|##
|
|
169
168
|
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
##|(markdown)
|
|
169
|
+
##|(markdown) comparing_arms
|
|
173
170
|
## 8. Comparing arms
|
|
174
171
|
|
|
175
172
|
Only meaningful with two or more arms loaded.
|
|
@@ -182,13 +179,24 @@ one case. What matters is which cases actually moved, and in which direction.
|
|
|
182
179
|
verdicts to see who won each one.
|
|
183
180
|
|##
|
|
184
181
|
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
##|(markdown)
|
|
182
|
+
##|(markdown) cases_every_arm_failed
|
|
188
183
|
### Cases every arm failed
|
|
189
184
|
|
|
190
185
|
Changing the answerer will not move these. They are where model and retrieval
|
|
191
186
|
work pays off, and they are the most durable finding in the whole run.
|
|
192
187
|
|##
|
|
193
188
|
|
|
194
|
-
|
|
189
|
+
// The one table the model does not name as a query: the run's health, by arm.
|
|
190
|
+
source: attempts_tiles is attempts extend {
|
|
191
|
+
# label="Run health"
|
|
192
|
+
view: run_health is {
|
|
193
|
+
group_by: arm is runs.label
|
|
194
|
+
aggregate:
|
|
195
|
+
attempt_count
|
|
196
|
+
submitted_count
|
|
197
|
+
contaminated_count
|
|
198
|
+
harness_errors
|
|
199
|
+
judge_version is runs.judge_version.max()
|
|
200
|
+
set_version is runs.set_version.max()
|
|
201
|
+
}
|
|
202
|
+
}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: eval-report
|
|
3
3
|
description: 'Write the summary a person reads after an evaluation run: what ran, what it scored, which failures were the MODEL and which were the EVAL itself (out of turns, contaminated, unestablished golden, unreadable judge), what was learned, and what to do next. Every claim links to the artifact behind it. Use at the end of any run, before quoting a number to anyone. Never scores an answer (eval-answer) or decides who owns a failure (eval-diagnose).'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Report a Run
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-analysis
|
|
3
3
|
description: "Read before your first get_context call on any business or data question. The required workflow, not background: which source, which rows, how to verify. Skipping it causes confident wrong answers."
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-analysis-report
|
|
3
3
|
description: Combine validated Malloy queries into a report (a linear report, read top to bottom). Use when the user asks to "create a report", "combine these into a report", or wants a persistent multi-query artifact.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
|
+
|
|
6
7
|
<!--
|
|
7
8
|
Copyright (c) Credible Data Inc.
|
|
8
9
|
SPDX-License-Identifier: MIT
|
|
@@ -10,9 +11,7 @@ SPDX-License-Identifier: MIT
|
|
|
10
11
|
|
|
11
12
|
# Creating Reports
|
|
12
13
|
|
|
13
|
-
An ad-hoc report is a Publisher **report** with `layout=linear`: a `.malloy` file at the package root, `<slug>.malloy` (unless the user asks for another location), that combines markdown narrative with live Malloy queries
|
|
14
|
-
|
|
15
|
-
> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
|
|
14
|
+
An ad-hoc report is a Publisher **report** with `layout=linear`: a `.malloy` file at the package root, `<slug>.malloy` (unless the user asks for another location), that combines markdown narrative with live Malloy queries, as text tiles and query tiles read top to bottom. Load `skill:malloy-reports` for the full format and authoring rules; this skill covers when to build one and how to design good report content (tiles, chart annotations, narrative structure). Never write a new `.malloynb`, never write a report as bare `run:` statements (that older form is no longer read), and do not create a `notebooks/` or `dashboards/` directory for it.
|
|
16
15
|
|
|
17
16
|
## Before building a report
|
|
18
17
|
|
|
@@ -22,80 +21,107 @@ An ad-hoc report is a Publisher **report** with `layout=linear`: a `.malloy` fil
|
|
|
22
21
|
|
|
23
22
|
Do NOT build the report in the same turn as `execute_query`. Explain first, then build.
|
|
24
23
|
|
|
25
|
-
## Filters
|
|
24
|
+
## Filters belong to the report, never to the source
|
|
25
|
+
|
|
26
|
+
An ad-hoc report usually needs no filters: it records an analysis already run. If the user wants one a reader can change (a manufacturer, a date range), declare the given in the report and bind it on its tiles, as `skill:malloy-reports` describes. Do not add a `given:` to the source for it: a model-level `where:` filters every read of the source, chat and every other report included, and the report builder can show such a filter but not change or remove it. A given the source already reads (a gate, a value every query must supply) shows in the report with no work, and stays in the model.
|
|
26
27
|
|
|
27
|
-
|
|
28
|
+
Never add a `#(filter)` annotation: it is deprecated, and every use has a `given:` form (`malloy-model` covers the migration).
|
|
28
29
|
|
|
29
30
|
## What goes in the report
|
|
30
31
|
|
|
31
|
-
Do NOT add an H1 heading in any markdown (use H2 and below for sections); the `title` in the `## artifact` tag serves as the title. To redo the structure rather than tweak one
|
|
32
|
+
Do NOT add an H1 heading in any markdown (use H2 and below for sections); the `title` in the `## artifact` tag serves as the title. To redo the structure rather than tweak one tile, rewrite the report file end-to-end.
|
|
32
33
|
|
|
33
|
-
|
|
34
|
+
Text tiles own narrative; query tiles own a single Malloy query whose chart annotation tells the renderer how to display the result. Markdown supports H2 headings, lists, bold, and inline code. Keep text tiles short, one idea per tile, so the rendered output reads as a story instead of a wall of text.
|
|
34
35
|
|
|
35
|
-
The file
|
|
36
|
+
The file has four parts, in this order:
|
|
36
37
|
|
|
37
|
-
|
|
38
|
+
1. **The tag**, `##| artifact { layout=linear title="..." tiles=[…] }`, listing every tile in reading order. A text tile is `name { kind=text }`; a query tile is the quoted string `"<extension> -> <view>"`.
|
|
39
|
+
2. **The `import`** of the model file, relative to the report's own location (`"order_analysis.malloy"` from the package root).
|
|
40
|
+
3. **The text tiles**, each a `##|(markdown) name` block: the name on the opener line, the body on the lines below, `|##` closing it.
|
|
41
|
+
4. **One extension of the source**, `source: <source>_tiles is <source> extend { … }`, holding a `view:` per query tile. The view's chart tag and `# label="…"` (the tile's heading) go directly above it.
|
|
38
42
|
|
|
39
|
-
|
|
43
|
+
Compile the file with `/compile` (`"scope": "file"`, at the path `<slug>.malloy`) before saving; a report compiles as a model, so its errors come back there. After saving and reloading, read the package warnings: a tile that names a missing view is reported there, not by the compile. A complete report:
|
|
40
44
|
|
|
41
45
|
```malloy
|
|
42
|
-
|
|
46
|
+
##| artifact { layout=linear title="Sales report"
|
|
47
|
+
tiles=[
|
|
48
|
+
overview { kind=text },
|
|
49
|
+
"order_analysis_tiles -> headline",
|
|
50
|
+
trend { kind=text },
|
|
51
|
+
"order_analysis_tiles -> revenue_by_month",
|
|
52
|
+
breakdown { kind=text },
|
|
53
|
+
"order_analysis_tiles -> top_categories",
|
|
54
|
+
takeaways { kind=text }
|
|
55
|
+
]
|
|
56
|
+
}
|
|
57
|
+
|##
|
|
43
58
|
import "order_analysis.malloy"
|
|
44
59
|
|
|
45
|
-
##|(markdown)
|
|
60
|
+
##|(markdown) overview
|
|
46
61
|
## Overview
|
|
47
|
-
What is driving sales, and which categories carry it? The
|
|
62
|
+
What is driving sales, and which categories carry it? The tiles below cover every order in the model.
|
|
48
63
|
|##
|
|
49
64
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
# currency
|
|
55
|
-
total_revenue
|
|
56
|
-
|
|
57
|
-
# label="Orders"
|
|
58
|
-
# number=auto
|
|
59
|
-
order_count
|
|
60
|
-
}
|
|
61
|
-
|
|
62
|
-
##(markdown) ## Trend: how does revenue move over time?
|
|
63
|
-
|
|
64
|
-
#" Revenue by month
|
|
65
|
-
# line_chart
|
|
66
|
-
run: order_analysis -> {
|
|
67
|
-
group_by: order_date.month
|
|
68
|
-
aggregate: total_revenue
|
|
69
|
-
order_by: 1
|
|
70
|
-
}
|
|
65
|
+
##|(markdown) trend
|
|
66
|
+
## Trend
|
|
67
|
+
How does revenue move over time?
|
|
68
|
+
|##
|
|
71
69
|
|
|
72
|
-
##|(markdown)
|
|
70
|
+
##|(markdown) breakdown
|
|
73
71
|
## Breakdown
|
|
74
72
|
Which categories account for the most revenue?
|
|
75
73
|
|##
|
|
76
74
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
aggregate: total_revenue
|
|
82
|
-
order_by: total_revenue desc
|
|
83
|
-
limit: 10
|
|
84
|
-
}
|
|
75
|
+
##|(markdown) takeaways
|
|
76
|
+
## Key takeaways
|
|
77
|
+
What to look at next.
|
|
78
|
+
|##
|
|
85
79
|
|
|
86
|
-
|
|
80
|
+
source: order_analysis_tiles is order_analysis extend {
|
|
81
|
+
# big_value
|
|
82
|
+
# label="Headline"
|
|
83
|
+
view: headline is {
|
|
84
|
+
aggregate:
|
|
85
|
+
# label="Revenue"
|
|
86
|
+
# currency
|
|
87
|
+
total_revenue
|
|
88
|
+
|
|
89
|
+
# label="Orders"
|
|
90
|
+
# number=auto
|
|
91
|
+
order_count
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
# line_chart
|
|
95
|
+
# label="Revenue by month"
|
|
96
|
+
view: revenue_by_month is {
|
|
97
|
+
group_by: order_date.month
|
|
98
|
+
aggregate: total_revenue
|
|
99
|
+
order_by: 1
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
# bar_chart
|
|
103
|
+
# label="Top ten categories by revenue"
|
|
104
|
+
view: top_categories is {
|
|
105
|
+
group_by: category
|
|
106
|
+
aggregate: total_revenue
|
|
107
|
+
order_by: total_revenue desc
|
|
108
|
+
limit: 10
|
|
109
|
+
}
|
|
110
|
+
}
|
|
87
111
|
```
|
|
88
112
|
|
|
113
|
+
When the model already has the view you want, the tile view refines it rather than repeating it: `view: top_categories is by_category + { limit: 10 }`.
|
|
114
|
+
|
|
89
115
|
A well-structured report typically follows this pattern:
|
|
90
116
|
|
|
91
117
|
```
|
|
92
|
-
[
|
|
93
|
-
[
|
|
94
|
-
[
|
|
95
|
-
[
|
|
96
|
-
[
|
|
97
|
-
[
|
|
98
|
-
[
|
|
118
|
+
[Text] ## Overview: what question are we answering, what data is in scope (date range, entity count)
|
|
119
|
+
[Query] KPI tile: headline numbers (e.g., # big_value, or # dashboard with nested # big_value views)
|
|
120
|
+
[Text] ## Trend: describe what we should look for over time
|
|
121
|
+
[Query] Time-series tile (e.g., # line_chart on a date dimension)
|
|
122
|
+
[Text] ## Breakdown: where the signal is
|
|
123
|
+
[Query] Categorical tile (e.g., # bar_chart on a categorical dimension)
|
|
124
|
+
[Text] ## Key takeaways: what the user should walk away with
|
|
99
125
|
```
|
|
100
126
|
|
|
101
127
|
Use this as a default; deviate when the analysis warrants. A grounded report names the time range and entity count up front so every number that follows has context.
|
|
@@ -105,17 +131,18 @@ Use this as a default; deviate when the analysis warrants. A grounded report nam
|
|
|
105
131
|
Read `skill:malloy-charts` before picking visualizations: it owns chart-type selection, properties, and the placement rules for chart annotations. `skill:malloy-queries` covers Malloy query patterns and the critical placement rules for chart-annotation tags.
|
|
106
132
|
|
|
107
133
|
When in doubt:
|
|
134
|
+
|
|
108
135
|
- KPIs / single numbers -> `# big_value`, often nested inside `# dashboard`.
|
|
109
136
|
- Trend over time -> `# line_chart`, usually on the primary date dimension.
|
|
110
137
|
- Category comparisons -> `# bar_chart`, ordered by the metric.
|
|
111
|
-
- Tabular data with many columns -> a plain table
|
|
138
|
+
- Tabular data with many columns -> a plain table tile with `# table.size=fill`.
|
|
112
139
|
- Multiple coordinated charts -> `# dashboard` with `nest:` blocks.
|
|
113
140
|
|
|
114
|
-
Annotations go **
|
|
141
|
+
Annotations go **directly above** the tile's `view:`, never inside its curly braces:
|
|
115
142
|
|
|
116
143
|
```malloy
|
|
117
144
|
# bar_chart
|
|
118
|
-
|
|
145
|
+
view: top_categories is {
|
|
119
146
|
group_by: category
|
|
120
147
|
aggregate: revenue
|
|
121
148
|
order_by: revenue desc
|
|
@@ -123,11 +150,11 @@ run: source -> {
|
|
|
123
150
|
}
|
|
124
151
|
```
|
|
125
152
|
|
|
126
|
-
A `# dashboard`
|
|
153
|
+
A `# dashboard` tile composes nested views, useful for KPIs alongside a trend in a single tile. Each `nest:` is a tile; any top-level `aggregate:` measures render as KPI cards. For a fixed grid, use `# dashboard { columns=N }` with `# colspan` on each tile (see `skill:malloy-charts`):
|
|
127
154
|
|
|
128
155
|
```malloy
|
|
129
156
|
# dashboard { columns=2 }
|
|
130
|
-
|
|
157
|
+
view: summary is {
|
|
131
158
|
nest:
|
|
132
159
|
# colspan=2
|
|
133
160
|
# big_value
|
|
@@ -151,7 +178,8 @@ run: source -> {
|
|
|
151
178
|
}
|
|
152
179
|
```
|
|
153
180
|
|
|
154
|
-
Key rendering rules to keep in mind when shaping a
|
|
181
|
+
Key rendering rules to keep in mind when shaping a tile:
|
|
182
|
+
|
|
155
183
|
- FIRST `group_by` = x-axis, FIRST `aggregate` = y-axis.
|
|
156
184
|
- Override field roles with `# x`, `# y`, `# series` on individual fields.
|
|
157
185
|
- For multiple measure series, place `# y` above the `aggregate:` keyword.
|
|
@@ -160,8 +188,8 @@ Key rendering rules to keep in mind when shaping a cell:
|
|
|
160
188
|
|
|
161
189
|
## Editing an existing report
|
|
162
190
|
|
|
163
|
-
For small targeted changes (fix one
|
|
191
|
+
For small targeted changes (fix one view, insert one tile), edit that view or text block in place and add its entry to `tiles=[…]`, rather than recreating the whole report. For structural rewrites (reordering many tiles, changing the narrative arc), rewrite the report file. An existing `.malloynb` is read, not edited: to change its story, write a `.malloy` report. A tag that sets `kind=notebook` is no longer read, so the file is not a report (`report-kind-removed`): replace it with `layout=linear` and list the tiles. A linear file with no `tiles=[…]`, written as bare `run:` statements, is no longer a report: rewrite it as tiles (`skill:malloy-reports`, `reference/linear.md`).
|
|
164
192
|
|
|
165
193
|
## IMPORTANT
|
|
166
194
|
|
|
167
|
-
You CANNOT see the rendered output of report
|
|
195
|
+
You CANNOT see the rendered output of report tiles. Do not claim to see charts, values, or patterns from report tiles you haven't explicitly executed via `execute_query`. If you need to analyze results, run the query via `execute_query` first.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-charts
|
|
3
3
|
description: Read before choosing a chart or adding a chart annotation. Chart types, tag syntax and scale rules, KPI cards, dashboards, sparklines, and the renderer mistakes that fail silently.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-dashboards
|
|
3
3
|
description: "Renamed to malloy-reports. A dashboard is now a report with layout=grid: a .malloy file with an ## artifact tag, written at the package root. Load malloy-reports instead when the user asks for a dashboard, a filterable view, or drill-through between views."
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!--
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-define
|
|
3
3
|
description: Propose scope, a source plan and field definitions for a Malloy model. Which tables and questions, which sources at what grain, then renames, dimensions and measures, each backed by querying data.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -178,7 +178,7 @@ The user will:
|
|
|
178
178
|
- **Remove** fields they don't need.
|
|
179
179
|
- **Change** priorities.
|
|
180
180
|
|
|
181
|
-
Once the definitions are confirmed, write them into the `.malloy` model (see your modeling workflow). Use `#(doc)` annotations to document sources and fields, and `given:`
|
|
181
|
+
Once the definitions are confirmed, write them into the `.malloy` model (see your modeling workflow). Use `#(doc)` annotations to document sources and fields, and a `given:` only where the data's meaning or access depends on a runtime value (a gate, a value every query must supply). A filter a reader picks on a dashboard belongs to the report, not the model. `#(filter)` is deprecated. Never add a `#(filter)` annotation: every use, including `required`, `implicit`, and date/number ranges, has a `given:` form. See `skill:malloy-model` § Parameterizing sources with `given:`. Keep the confirmed definitions in the conversation; there is no separate plan-file store.
|
|
182
182
|
|
|
183
183
|
## Data-driven proposals
|
|
184
184
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-discover
|
|
3
3
|
description: Silent data discovery for Malloy modeling. Used at Step 1 of the modeling workflow. Scans tables, columns, distributions, and relationships without user interaction. The agent builds an internal picture before presenting anything.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.42
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|