@malloy-publisher/skills 0.1.20 → 0.1.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/skills/eval-answer/SKILL.md +45 -5
- package/skills/eval-answer/reference/definition-ledger.md +305 -0
- package/skills/eval-answer/reference/ledger-schema.md +36 -7
- package/skills/eval-answer/scripts/check_coverage.py +12 -21
- package/skills/eval-answer/scripts/check_coverage_test.py +22 -0
- package/skills/eval-answer/scripts/check_findable.py +386 -0
- package/skills/eval-answer/scripts/check_findable_test.py +326 -0
- package/skills/eval-answer/scripts/json_scan.py +43 -0
- package/skills/eval-answer/scripts/ledger.py +63 -7
- package/skills/eval-answer/scripts/ledger_test.py +7 -0
- package/skills/eval-answer/scripts/mcp_payload.py +111 -2
- package/skills/eval-answer/scripts/mcp_payload_test.py +73 -2
- package/skills/eval-answer/scripts/publisher_rest.py +10 -4
- package/skills/eval-answer/scripts/publisher_rest_test.py +56 -0
- package/skills/eval-answer/scripts/score_retrieval.py +326 -41
- package/skills/eval-answer/scripts/score_retrieval_test.py +537 -15
- package/skills/eval-answer/scripts/verify_definitions.py +752 -0
- package/skills/eval-answer/scripts/verify_definitions_test.py +820 -0
- package/skills/eval-answer/scripts/verify_goldens.py +336 -22
- package/skills/eval-answer/scripts/verify_goldens_test.py +261 -2
- package/skills/eval-diagnose/SKILL.md +47 -2
- package/skills/eval-diagnose/scripts/cluster_failures.py +57 -14
- package/skills/eval-diagnose/scripts/cluster_failures_test.py +127 -0
- package/skills/eval-diagnose/scripts/diagnose.py +471 -42
- package/skills/eval-diagnose/scripts/diagnose_test.py +421 -0
- package/skills/eval-import/SKILL.md +180 -1
- package/skills/eval-import/reference/case-format.md +1 -1
- package/skills/eval-improve/SKILL.md +1 -1
- package/skills/eval-improve/scripts/improve.py +29 -3
- package/skills/eval-improve/scripts/improve_test.py +51 -0
- package/skills/eval-judge/SKILL.md +1 -1
- package/skills/eval-loop/SKILL.md +112 -3
- package/skills/eval-loop/reference/measurement.md +44 -4
- package/skills/eval-loop/reference/running-a-run.md +5 -1
- package/skills/eval-loop/scripts/agent_harness.py +21 -20
- package/skills/eval-loop/scripts/agent_harness_test.py +136 -0
- package/skills/eval-loop/scripts/build_run_package.py +9 -3
- package/skills/eval-loop/scripts/flip_table.py +107 -2
- package/skills/eval-loop/scripts/flip_table_test.py +165 -0
- package/skills/eval-loop/scripts/run_baseline.py +956 -97
- package/skills/eval-loop/scripts/run_baseline_test.py +799 -1
- package/skills/eval-loop/templates/eval-run-package/README.md +19 -5
- package/skills/eval-loop/templates/eval-run-package/eval_run.malloy +48 -19
- package/skills/eval-loop/templates/eval-run-package/eval_run.malloynb +11 -4
- package/skills/eval-loop/templates/eval-run-package/public/app.js +12 -4
- package/skills/eval-report/SKILL.md +362 -0
- package/skills/malloy/SKILL.md +2 -1
- package/skills/malloy-analysis/SKILL.md +1 -1
- package/skills/malloy-analysis-pitfalls/SKILL.md +13 -1
- package/skills/malloy-analysis-report/SKILL.md +1 -1
- package/skills/malloy-analyze/SKILL.md +1 -1
- package/skills/malloy-charts/SKILL.md +1 -1
- package/skills/malloy-dashboards/SKILL.md +1 -1
- package/skills/malloy-debug/SKILL.md +1 -1
- package/skills/malloy-define/SKILL.md +1 -1
- package/skills/malloy-discover/SKILL.md +1 -1
- package/skills/malloy-document/SKILL.md +1 -1
- package/skills/malloy-getting-started/SKILL.md +1 -1
- package/skills/malloy-gotchas-modeling/SKILL.md +1 -1
- package/skills/malloy-gotchas-queries/SKILL.md +1 -1
- package/skills/malloy-gotchas-rendering/SKILL.md +1 -1
- package/skills/malloy-html-data-app-embedding/SKILL.md +1 -1
- package/skills/malloy-html-data-app-runtime/SKILL.md +1 -1
- package/skills/malloy-html-data-apps/SKILL.md +1 -1
- package/skills/malloy-lookml-review/SKILL.md +1 -1
- package/skills/malloy-materialization/SKILL.md +1 -1
- package/skills/malloy-materialization-tuning/SKILL.md +1 -1
- package/skills/malloy-model/SKILL.md +1 -1
- package/skills/malloy-model-as-you-go/SKILL.md +1 -1
- package/skills/malloy-modeling/SKILL.md +1 -1
- package/skills/malloy-notebook-chat/SKILL.md +1 -1
- package/skills/malloy-notebooks/SKILL.md +1 -1
- package/skills/malloy-patterns/SKILL.md +1 -1
- package/skills/malloy-phrase-detection/SKILL.md +3 -2
- package/skills/malloy-publish/SKILL.md +1 -1
- package/skills/malloy-queries/SKILL.md +1 -1
- package/skills/malloy-review/SKILL.md +1 -1
- package/skills/malloy-scope/SKILL.md +1 -1
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-analyze
|
|
3
3
|
description: Open-ended exploration with no specific question to answer, and building views/dashboards/notebooks on an existing model. Use when the user asks "what's interesting?", "explore this data", "find insights", "look for patterns", "build a dashboard", or "create views". Not for a specific data question - answering one is `malloy-analysis`, and writing down what the answer assumed is `malloy-model-as-you-go`. For exploration start at Step 1; for views on an existing model jump to View Patterns.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-charts
|
|
3
3
|
description: Chart selection for a Malloy view, plus the renderer reference for bar_chart, line_chart, scatter_chart, maps and layouts. For choosing a visualization or adding a chart annotation.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-dashboards
|
|
3
3
|
description: "Build or modify a Malloy Publisher dashboard, a tagged .malloy file in a package's dashboards/ directory, with auto-rendered filter controls, a grid layout, and # drill click-through. Use when the user asks for a dashboard, a filterable operational view, or drill-through between views, and no code is wanted."
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!--
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-debug
|
|
3
3
|
description: Fix Malloy compile errors and understand error messages. Use when encountering errors in .malloy files, user says "fix this error", "malloy error", "compile error", "syntax error", or sees 20+ cascading errors.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-define
|
|
3
3
|
description: Propose a source plan and field definitions for a Malloy semantic model. Covers picking which sources to model and at what grain, then proposing the specific renames, dimensions, and measures per source, every proposal backed by querying the data.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-discover
|
|
3
3
|
description: Silent data discovery for Malloy modeling. Used at Step 1 of the modeling workflow. Scans tables, columns, distributions, and relationships without user interaction. The agent builds an internal picture before presenting anything.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-document
|
|
3
3
|
description: 'Add documentation with #(doc) tags to Malloy models so fields and sources are described in plain language. Use when user asks to "add documentation", "add doc tags", "document the model", or wants fields and sources described for natural-language search and discovery. For declaring parameterizable filters with #(filter), see the malloy-model skill. Filters are a runtime/modeling construct (governance, latency, correctness), not a documentation tag.'
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-getting-started
|
|
3
3
|
description: First steps for using a Malloy Publisher deployment through its MCP tools. Use when connecting to Publisher for the first time, when you do not yet know the available environments, packages, or models, or when a user asks what data they can explore. Covers verifying the server, discovering data with get_context, and running a first grounded query.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-gotchas-modeling
|
|
3
3
|
description: Common Malloy modeling mistakes, for writing source definitions, dimensions, measures or joins - reserved words, NULL checks, date functions, type casts, field management (extend except/accept/rename vs include public/internal/private), query-based source gotchas.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-gotchas-queries
|
|
3
3
|
description: Common Malloy query and view mistakes, for writing views, queries or notebooks - chart constraints, aggregate filters, joined field aliasing, method syntax, time truncation vs extraction.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-gotchas-rendering
|
|
3
3
|
description: Common Malloy renderer mistakes, for adding chart annotations and formatting tags or building dashboards - tag syntax, scale rules, sparkline setup, big_value patterns.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-html-data-app-embedding
|
|
3
3
|
description: Embed an in-package HTML data app into a host page or another application, including auto-sizing and auth. Read when embedding a Publisher page via Publisher.embed.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-html-data-app-runtime
|
|
3
3
|
description: The JavaScript that drives an in-package HTML data app - calling Publisher.query, building queries from filter state, handling results and errors. For writing the page's data code.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-html-data-apps
|
|
3
3
|
description: Build or modify an in-package HTML data app for a Malloy Publisher package (a public/ directory the package serves). Use when the user wants a hand-authored HTML dashboard or web page backed by a package's Malloy models, with no build step.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-lookml-review
|
|
3
3
|
description: Analyze LookML files as prior art for Malloy modeling. Used during Step 1 (DISCOVER) when .lkml files are present. Coordinates reference files that extract business logic, relationships, and curation decisions. Works with or without a database connection.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-materialization
|
|
3
3
|
description: Add and debug Malloy Persistence materializations in a package - persist an expensive source so queries read a pre-built table. Read this whenever the user wants to materialize a source, add a persist annotation, speed up a slow source, or asks why a persist source isn't building.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-materialization-tuning
|
|
3
3
|
description: Optimize a package's Malloy Persistence materializations for cost and performance using the malloy-pub CLI and the materialization history. Recommend what to persist, what to stop persisting, and how to schedule/scope it. Use when the user asks to make a package cheaper or faster, tune persistence, decide what to materialize, or review persist/schedule choices.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!--
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-model
|
|
3
3
|
description: Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to "create a malloy model", "add dimensions", "add measures", "create a source", or any Malloy model authoring task.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-model-as-you-go
|
|
3
3
|
description: After answering a data question, write down what the answer assumed so the next reader can trust the number. A field with a #(doc) in the model when you can edit it, an extend in the notebook when you can only author reports, or a stated assumption plus a Malloy snippet when you can only chat. Use after every answered question that rested on a judgment call, and whenever a question is asked against tables that have no model yet.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!--
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-modeling
|
|
3
3
|
description: Build semantic models with Malloy for the Malloy Publisher. Read this skill whenever the user asks about modeling data or specifically mentions Malloy.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-notebook-chat
|
|
3
3
|
description: Steps to follow when the chat is bound to a notebook or saved report. The notebook's cells are the agent's primary context, answer from it, run its queries, and only reach for get_context when the user asks about something outside it.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-notebooks
|
|
3
3
|
description: Create Malloy notebooks (.malloynb) for interactive dashboards and data stories. Use when user asks to "create a notebook", "make a dashboard notebook", "write a malloynb", "data story", or needs to build reports/visualizations.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-patterns
|
|
3
3
|
description: An index of what search_malloy_docs covers - language reference, common patterns (YoY, cohorts, percent of total), rendering - for picking a topic worth searching before querying the docs.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-phrase-detection
|
|
3
3
|
description: How to phrase search_text on a get_context call so retrieval returns the fields you need instead of a truncated catalog. Covers target-type classification and decomposition patterns.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!-- Copyright (c) Credible Data Inc. SPDX-License-Identifier: MIT -->
|
|
7
7
|
|
|
@@ -50,7 +50,8 @@ These are the rules you won't apply correctly by default:
|
|
|
50
50
|
5. **Categorical strings that look numeric are still dimensions.** "18-30", "<5 days", "tier 2" are stored as literal strings on a dimension. Target that dimension, then confirm the exact string with `execute_query`.
|
|
51
51
|
6. **"Top N" without a named measure, add a ranking measure.** "top 6 products" becomes a measure for the ranking concept (`"the performance metric for a product"`) plus a dimension for the entity. If the measure is explicit ("top products by total sales"), use it directly and skip the generic ranking measure.
|
|
52
52
|
7. **Multiple values for one concept, one dimension target.** Several values ("premium and basic") still map to a single dimension target for the parent field; enumerate the exact stored values with `execute_query`.
|
|
53
|
-
8. **
|
|
53
|
+
8. **The quantity asked for is a measure, even when nothing names it.** "How many titles were released in 2019?" is asking for a count, but no noun in it is the count: the visible phrases are the subject ("titles") and the filter ("2019"), and mapping only those yields a `source` and a `dimension` target with no way to answer. Always add a measure target for the quantity itself (`"the number of titles"`). This holds for every interrogative that IS the aggregation -- "how many", "how much", "how often" -- and it is the pattern most easily lost when a filter or grouping is the loud part of the sentence. Measured: an answerer got this right on "How many titles are in the dataset?" and dropped it on "How many titles were released in 2019?", where the year took the attention.
|
|
54
|
+
9. **A population qualifier is a target, and so is the one the question omits.** Words like "real", "actual", "genuine", "live" or "production" are not filler: they name rows the model marks for exclusion. Target the flag itself (`"the flag marking synthetic, test or monitoring traffic"`), not just the noun they modify. Add one such target even when the question carries no qualifier at all, because a table of events, requests, sessions or logs usually holds test, internal or cancelled rows and nothing in the wording will say so. Resolve it to the model's own flag rather than inventing a filter on an id or a name; a hand-rolled exclusion and the documented one rarely select the same rows.
|
|
54
55
|
|
|
55
56
|
## Worked example
|
|
56
57
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-publish
|
|
3
3
|
description: Package Malloy models for serving by Malloy Publisher. Use when user asks to "publish", "package", "deploy", or wants to share models with others.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-queries
|
|
3
3
|
description: Malloy query patterns and syntax, for writing or debugging a query - dates, aggregates vs dimensions, join paths, filters, string matching, chart annotations, and the errors a SQL habit produces.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-review
|
|
3
3
|
description: Malloy semantic-model code review. Invoke when the user asks to review, audit, or critique a `.malloy` file, a folder of Malloy models, or a GitHub PR that touches Malloy. Enforces project modeling standards and emits a navigable review file.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
<!--
|
|
7
7
|
Copyright (c) Credible Data Inc.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: malloy-scope
|
|
3
3
|
description: Present discovery findings and propose an analytical scope before modeling. Use after inspecting a package's model and data, to classify tables and recommend an analytical focus the user can pick from.
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.21
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
<!--
|