tabletalk 0.2.1__tar.gz → 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tabletalk-0.5.0/.github/workflows/db_provider_tests.yml +84 -0
- tabletalk-0.5.0/.github/workflows/publish.yml +96 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/.gitignore +26 -0
- tabletalk-0.5.0/PKG-INFO +104 -0
- tabletalk-0.5.0/README.md +75 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/docs/Makefile +1 -1
- tabletalk-0.5.0/docs/architecture.md +28 -0
- tabletalk-0.5.0/docs/commands.md +16 -0
- tabletalk-0.5.0/docs/conf.py +48 -0
- tabletalk-0.5.0/docs/configuration.md +48 -0
- tabletalk-0.5.0/docs/contributing.md +29 -0
- tabletalk-0.5.0/docs/evals.md +49 -0
- tabletalk-0.5.0/docs/getting-started.md +38 -0
- tabletalk-0.5.0/docs/index.md +14 -0
- tabletalk-0.5.0/examples/dbt-analytics/README.md +24 -0
- tabletalk-0.5.0/examples/dbt-analytics/agents/revenue.yaml +10 -0
- tabletalk-0.5.0/examples/dbt-analytics/dbt_project.yml +10 -0
- tabletalk-0.5.0/examples/dbt-analytics/evals/revenue.yaml +21 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/marts/fct_orders.sql +2 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/other/duplicate_orders.sql +2 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/schema.yml +44 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/sources.yml +19 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/staging/disabled_orders.sql +2 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/staging/ephemeral_order_keys.sql +2 -0
- tabletalk-0.5.0/examples/dbt-analytics/models/staging/stg_orders.sql +6 -0
- tabletalk-0.5.0/examples/dbt-analytics/package-lock.yml +4 -0
- tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/dbt_project.yml +4 -0
- tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/models/schema.yml +8 -0
- tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/models/shared_calendar.sql +1 -0
- tabletalk-0.5.0/examples/dbt-analytics/packages.yml +2 -0
- tabletalk-0.5.0/examples/dbt-analytics/profiles.yml +8 -0
- tabletalk-0.5.0/examples/dbt-analytics/seeds/raw_orders.csv +4 -0
- tabletalk-0.5.0/examples/dbt-analytics/tabletalk.yaml +11 -0
- tabletalk-0.5.0/examples/dbt-analytics/target/catalog.json +1 -0
- tabletalk-0.5.0/examples/dbt-analytics/target/manifest.json +1 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/pyproject.toml +20 -26
- tabletalk-0.5.0/tabletalk/__init__.py +22 -0
- tabletalk-0.5.0/tabletalk/agents.py +279 -0
- tabletalk-0.5.0/tabletalk/authoring.py +106 -0
- tabletalk-0.5.0/tabletalk/cli.py +832 -0
- tabletalk-0.5.0/tabletalk/connections.py +166 -0
- tabletalk-0.5.0/tabletalk/evals/__init__.py +538 -0
- tabletalk-0.5.0/tabletalk/factories.py +114 -0
- tabletalk-0.5.0/tabletalk/interfaces.py +78 -0
- tabletalk-0.5.0/tabletalk/manifest.py +465 -0
- tabletalk-0.5.0/tabletalk/project.py +172 -0
- tabletalk-0.5.0/tabletalk/providers/duckdb_provider.py +26 -0
- tabletalk-0.5.0/tabletalk/providers/openai_provider.py +196 -0
- tabletalk-0.5.0/tabletalk/providers/snowflake_provider.py +46 -0
- tabletalk-0.5.0/tabletalk/providers/sqlite_provider.py +30 -0
- tabletalk-0.5.0/tabletalk/runtime/__init__.py +434 -0
- tabletalk-0.5.0/tabletalk/tests/test_connections.py +108 -0
- tabletalk-0.5.0/tabletalk/tests/test_dbt_native.py +900 -0
- tabletalk-0.5.0/tabletalk/traces.py +131 -0
- tabletalk-0.5.0/tabletalk/validation.py +199 -0
- tabletalk-0.5.0/uv.lock +2741 -0
- tabletalk-0.2.1/.github/workflows/db_provider_tests.yml +0 -57
- tabletalk-0.2.1/.github/workflows/publish.yml +0 -31
- tabletalk-0.2.1/PKG-INFO +0 -494
- tabletalk-0.2.1/README.md +0 -451
- tabletalk-0.2.1/contexts/default_context.yaml +0 -13
- tabletalk-0.2.1/docs/conf.py +0 -27
- tabletalk-0.2.1/docs/dev/contexts.rst +0 -45
- tabletalk-0.2.1/docs/dev/local_development.rst +0 -127
- tabletalk-0.2.1/docs/dev/providers.rst +0 -164
- tabletalk-0.2.1/docs/dev/querying.rst +0 -201
- tabletalk-0.2.1/docs/index.rst +0 -26
- tabletalk-0.2.1/examples/ecommerce/README.md +0 -281
- tabletalk-0.2.1/examples/ecommerce/agents/customer_analyst.yaml +0 -44
- tabletalk-0.2.1/examples/ecommerce/agents/inventory_manager.yaml +0 -34
- tabletalk-0.2.1/examples/ecommerce/agents/marketing_analyst.yaml +0 -35
- tabletalk-0.2.1/examples/ecommerce/agents/revenue_analyst.yaml +0 -39
- tabletalk-0.2.1/examples/ecommerce/contexts/customers.yaml +0 -30
- tabletalk-0.2.1/examples/ecommerce/contexts/inventory.yaml +0 -41
- tabletalk-0.2.1/examples/ecommerce/contexts/marketing.yaml +0 -39
- tabletalk-0.2.1/examples/ecommerce/contexts/sales.yaml +0 -53
- tabletalk-0.2.1/examples/ecommerce/dbt_project/dbt_project.yml +0 -16
- tabletalk-0.2.1/examples/ecommerce/dbt_project/models/marts/fct_orders.sql +0 -15
- tabletalk-0.2.1/examples/ecommerce/dbt_project/models/marts/fct_orders.yml +0 -17
- tabletalk-0.2.1/examples/ecommerce/dbt_project/models/sources.yml +0 -22
- tabletalk-0.2.1/examples/ecommerce/dbt_project/models/staging/stg_customers.sql +0 -10
- tabletalk-0.2.1/examples/ecommerce/dbt_project/models/staging/stg_orders.sql +0 -10
- tabletalk-0.2.1/examples/ecommerce/dbt_project/profiles.yml +0 -13
- tabletalk-0.2.1/examples/ecommerce/ecommerce.duckdb +0 -0
- tabletalk-0.2.1/examples/ecommerce/manifest/customers.txt +0 -5
- tabletalk-0.2.1/examples/ecommerce/manifest/inventory.txt +0 -7
- tabletalk-0.2.1/examples/ecommerce/manifest/marketing.txt +0 -7
- tabletalk-0.2.1/examples/ecommerce/manifest/sales.txt +0 -9
- tabletalk-0.2.1/examples/ecommerce/seed.py +0 -317
- tabletalk-0.2.1/examples/ecommerce/tabletalk.yaml +0 -51
- tabletalk-0.2.1/tabletalk/__init__.py +0 -27
- tabletalk-0.2.1/tabletalk/app.py +0 -385
- tabletalk-0.2.1/tabletalk/cli.py +0 -535
- tabletalk-0.2.1/tabletalk/factories.py +0 -155
- tabletalk-0.2.1/tabletalk/interfaces.py +0 -507
- tabletalk-0.2.1/tabletalk/profiles.py +0 -127
- tabletalk-0.2.1/tabletalk/providers/anthropic_provider.py +0 -59
- tabletalk-0.2.1/tabletalk/providers/azuresql_provider.py +0 -179
- tabletalk-0.2.1/tabletalk/providers/bigquery_provider.py +0 -98
- tabletalk-0.2.1/tabletalk/providers/duckdb_provider.py +0 -139
- tabletalk-0.2.1/tabletalk/providers/mysql_provider.py +0 -186
- tabletalk-0.2.1/tabletalk/providers/openai_provider.py +0 -51
- tabletalk-0.2.1/tabletalk/providers/postgres_provider.py +0 -164
- tabletalk-0.2.1/tabletalk/providers/snowflake_provider.py +0 -180
- tabletalk-0.2.1/tabletalk/providers/sqlite_provider.py +0 -84
- tabletalk-0.2.1/tabletalk/static/index.html +0 -1282
- tabletalk-0.2.1/tabletalk/static/logo.jpg +0 -0
- tabletalk-0.2.1/tabletalk/tests/conftest.py +0 -573
- tabletalk-0.2.1/tabletalk/tests/providers/__init__.py +0 -0
- tabletalk-0.2.1/tabletalk/tests/providers/test_azuresql.py +0 -61
- tabletalk-0.2.1/tabletalk/tests/providers/test_bigquery.py +0 -60
- tabletalk-0.2.1/tabletalk/tests/providers/test_duckdb.py +0 -377
- tabletalk-0.2.1/tabletalk/tests/providers/test_mysql.py +0 -129
- tabletalk-0.2.1/tabletalk/tests/providers/test_postgres.py +0 -114
- tabletalk-0.2.1/tabletalk/tests/providers/test_snowflake.py +0 -71
- tabletalk-0.2.1/tabletalk/tests/providers/test_sqlite.py +0 -364
- tabletalk-0.2.1/tabletalk/tests/test_app.py +0 -505
- tabletalk-0.2.1/tabletalk/tests/test_cli.py +0 -488
- tabletalk-0.2.1/tabletalk/tests/test_factories.py +0 -210
- tabletalk-0.2.1/tabletalk/tests/test_interfaces.py +0 -597
- tabletalk-0.2.1/tabletalk/tests/test_profiles.py +0 -378
- tabletalk-0.2.1/tabletalk/tests/test_utils.py +0 -218
- tabletalk-0.2.1/tabletalk/utils.py +0 -122
- tabletalk-0.2.1/tabletalk.yaml +0 -17
- tabletalk-0.2.1/uv.lock +0 -2416
- {tabletalk-0.2.1 → tabletalk-0.5.0}/.pre-commit-config.yaml +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/.python-version +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/LICENSE +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/docs/make.bat +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/main.py +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/providers/__init__.py +0 -0
- {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/tests/__init__.py +0 -0
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
name: Test supported providers
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
branches: [main]
|
|
8
|
+
workflow_dispatch:
|
|
9
|
+
|
|
10
|
+
permissions:
|
|
11
|
+
contents: read
|
|
12
|
+
|
|
13
|
+
jobs:
|
|
14
|
+
test:
|
|
15
|
+
name: Python ${{ matrix.python-version }}
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
strategy:
|
|
18
|
+
fail-fast: false
|
|
19
|
+
matrix:
|
|
20
|
+
python-version: ["3.10", "3.11", "3.12"]
|
|
21
|
+
|
|
22
|
+
steps:
|
|
23
|
+
- uses: actions/checkout@v6
|
|
24
|
+
|
|
25
|
+
- name: Install uv
|
|
26
|
+
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
|
|
27
|
+
with:
|
|
28
|
+
enable-cache: true
|
|
29
|
+
cache-suffix: py${{ matrix.python-version }}
|
|
30
|
+
|
|
31
|
+
- name: Set up Python
|
|
32
|
+
uses: actions/setup-python@v6
|
|
33
|
+
with:
|
|
34
|
+
python-version: ${{ matrix.python-version }}
|
|
35
|
+
|
|
36
|
+
- name: Install supported database extras
|
|
37
|
+
run: uv sync --all-extras
|
|
38
|
+
|
|
39
|
+
- name: Run tests
|
|
40
|
+
run: uv run pytest -q
|
|
41
|
+
|
|
42
|
+
- name: Run static checks
|
|
43
|
+
run: |
|
|
44
|
+
uv run ruff check tabletalk
|
|
45
|
+
uv run mypy tabletalk
|
|
46
|
+
|
|
47
|
+
package:
|
|
48
|
+
name: Verify PyPI distribution
|
|
49
|
+
runs-on: ubuntu-latest
|
|
50
|
+
|
|
51
|
+
steps:
|
|
52
|
+
- uses: actions/checkout@v6
|
|
53
|
+
|
|
54
|
+
- name: Install uv
|
|
55
|
+
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
|
|
56
|
+
|
|
57
|
+
- name: Set up Python
|
|
58
|
+
uses: actions/setup-python@v6
|
|
59
|
+
with:
|
|
60
|
+
python-version: "3.11"
|
|
61
|
+
|
|
62
|
+
- name: Build distribution
|
|
63
|
+
run: uv build
|
|
64
|
+
|
|
65
|
+
- name: Validate PyPI metadata
|
|
66
|
+
run: uvx twine check --strict dist/*
|
|
67
|
+
|
|
68
|
+
- name: Verify wheel contents and installation
|
|
69
|
+
run: |
|
|
70
|
+
python - <<'PY'
|
|
71
|
+
from pathlib import Path
|
|
72
|
+
from zipfile import ZipFile
|
|
73
|
+
|
|
74
|
+
wheel = next(Path("dist").glob("*.whl"))
|
|
75
|
+
with ZipFile(wheel) as archive:
|
|
76
|
+
names = archive.namelist()
|
|
77
|
+
assert not any("/tests/" in name for name in names)
|
|
78
|
+
assert not any(".data/data/tabletalk/static/" in name for name in names)
|
|
79
|
+
assert "tabletalk/static/index.html" in names
|
|
80
|
+
PY
|
|
81
|
+
uv venv .package-test
|
|
82
|
+
uv pip install --python .package-test/bin/python dist/*.whl
|
|
83
|
+
.package-test/bin/tabletalk --version
|
|
84
|
+
.package-test/bin/python -c "from tabletalk import Agent, EvalSuite, Project"
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
name: Publish to PyPI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
release:
|
|
5
|
+
types: [published]
|
|
6
|
+
|
|
7
|
+
permissions:
|
|
8
|
+
contents: read
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
build:
|
|
12
|
+
name: Build and verify distribution
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v6
|
|
17
|
+
|
|
18
|
+
- name: Install uv
|
|
19
|
+
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
|
|
20
|
+
with:
|
|
21
|
+
enable-cache: true
|
|
22
|
+
|
|
23
|
+
- name: Set up Python
|
|
24
|
+
uses: actions/setup-python@v6
|
|
25
|
+
with:
|
|
26
|
+
python-version: "3.11"
|
|
27
|
+
|
|
28
|
+
- name: Verify release tag matches package version
|
|
29
|
+
run: |
|
|
30
|
+
PACKAGE_VERSION="$(python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
|
|
31
|
+
test "${GITHUB_REF_NAME#v}" = "$PACKAGE_VERSION"
|
|
32
|
+
|
|
33
|
+
- name: Build wheel and source distribution
|
|
34
|
+
run: uv build
|
|
35
|
+
|
|
36
|
+
- name: Validate PyPI metadata
|
|
37
|
+
run: uvx twine check --strict dist/*
|
|
38
|
+
|
|
39
|
+
- name: Inspect wheel contents
|
|
40
|
+
run: |
|
|
41
|
+
python - <<'PY'
|
|
42
|
+
from pathlib import Path
|
|
43
|
+
from zipfile import ZipFile
|
|
44
|
+
|
|
45
|
+
wheel = next(Path("dist").glob("*.whl"))
|
|
46
|
+
with ZipFile(wheel) as archive:
|
|
47
|
+
names = archive.namelist()
|
|
48
|
+
assert not any("/tests/" in name for name in names)
|
|
49
|
+
assert not any("tabletalk/static/" in name for name in names)
|
|
50
|
+
assert "tabletalk/app.py" not in names
|
|
51
|
+
for required in (
|
|
52
|
+
"tabletalk/agents.py",
|
|
53
|
+
"tabletalk/evals/__init__.py",
|
|
54
|
+
"tabletalk/manifest.py",
|
|
55
|
+
"tabletalk/project.py",
|
|
56
|
+
"tabletalk/runtime/__init__.py",
|
|
57
|
+
"tabletalk/validation.py",
|
|
58
|
+
):
|
|
59
|
+
assert required in names, f"missing wheel module: {required}"
|
|
60
|
+
PY
|
|
61
|
+
|
|
62
|
+
- name: Verify wheel installation
|
|
63
|
+
run: |
|
|
64
|
+
uv venv .package-test
|
|
65
|
+
uv pip install --python .package-test/bin/python dist/*.whl
|
|
66
|
+
.package-test/bin/tabletalk --version
|
|
67
|
+
.package-test/bin/python -c "from tabletalk import Agent, EvalSuite, Project"
|
|
68
|
+
|
|
69
|
+
- name: Upload verified distributions
|
|
70
|
+
uses: actions/upload-artifact@v7
|
|
71
|
+
with:
|
|
72
|
+
name: python-package-distributions
|
|
73
|
+
path: dist/
|
|
74
|
+
if-no-files-found: error
|
|
75
|
+
|
|
76
|
+
publish:
|
|
77
|
+
name: Publish verified distribution
|
|
78
|
+
needs: build
|
|
79
|
+
runs-on: ubuntu-latest
|
|
80
|
+
environment:
|
|
81
|
+
name: pypi
|
|
82
|
+
url: https://pypi.org/p/tabletalk
|
|
83
|
+
permissions:
|
|
84
|
+
id-token: write
|
|
85
|
+
|
|
86
|
+
steps:
|
|
87
|
+
- name: Download verified distributions
|
|
88
|
+
uses: actions/download-artifact@v8
|
|
89
|
+
with:
|
|
90
|
+
name: python-package-distributions
|
|
91
|
+
path: dist/
|
|
92
|
+
|
|
93
|
+
- name: Publish to PyPI using trusted publishing
|
|
94
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
95
|
+
with:
|
|
96
|
+
packages-dir: dist/
|
|
@@ -65,8 +65,28 @@ user_settings.json
|
|
|
65
65
|
|
|
66
66
|
# Database
|
|
67
67
|
*.db
|
|
68
|
+
*.duckdb
|
|
69
|
+
*.duckdb.wal
|
|
70
|
+
examples/ecommerce/evals/fixtures/*.duckdb
|
|
68
71
|
__pycache__
|
|
69
72
|
|
|
73
|
+
# TableTalk generated output and per-user runtime state
|
|
74
|
+
.tabletalk/
|
|
75
|
+
/examples/*/manifest/
|
|
76
|
+
.tabletalk_history.jsonl
|
|
77
|
+
.tabletalk_favorites.json
|
|
78
|
+
.tabletalk_audit.jsonl
|
|
79
|
+
.tabletalk_schedules.json
|
|
80
|
+
.tabletalk_history/
|
|
81
|
+
.tabletalk_memory/
|
|
82
|
+
|
|
83
|
+
# dbt generated and per-user state
|
|
84
|
+
.user.yml
|
|
85
|
+
dbt_packages/
|
|
86
|
+
logs/
|
|
87
|
+
/examples/*/dbt_project/dbt_packages/
|
|
88
|
+
/examples/*/dbt_project/logs/
|
|
89
|
+
|
|
70
90
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
|
71
91
|
__pypackages__/
|
|
72
92
|
|
|
@@ -128,6 +148,12 @@ docs/_build/
|
|
|
128
148
|
.pybuilder/
|
|
129
149
|
target/
|
|
130
150
|
|
|
151
|
+
# The reference project's manifest is a dbt-generated acceptance fixture.
|
|
152
|
+
!examples/dbt-analytics/target/
|
|
153
|
+
examples/dbt-analytics/target/*
|
|
154
|
+
!examples/dbt-analytics/target/manifest.json
|
|
155
|
+
!examples/dbt-analytics/target/catalog.json
|
|
156
|
+
|
|
131
157
|
# Jupyter Notebook
|
|
132
158
|
.ipynb_checkpoints
|
|
133
159
|
|
tabletalk-0.5.0/PKG-INFO
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: tabletalk
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Evaluation and observability for natural-language agents over dbt projects
|
|
5
|
+
Project-URL: Homepage, https://github.com/wtbates99/tabletalk
|
|
6
|
+
Project-URL: Repository, https://github.com/wtbates99/tabletalk
|
|
7
|
+
Author-email: william bates <wtbates99@gmail.com>
|
|
8
|
+
License: CC BY-NC 4.0
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: ai,database,dbt,llm,natural-language,sql
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Requires-Python: >=3.10
|
|
16
|
+
Requires-Dist: click>=8.0
|
|
17
|
+
Requires-Dist: openai>=1.0.0
|
|
18
|
+
Requires-Dist: pyyaml>=6.0
|
|
19
|
+
Requires-Dist: rich>=13.0
|
|
20
|
+
Requires-Dist: sqlglot>=25.0
|
|
21
|
+
Provides-Extra: all
|
|
22
|
+
Requires-Dist: duckdb>=0.9; extra == 'all'
|
|
23
|
+
Requires-Dist: snowflake-connector-python>=3.0; extra == 'all'
|
|
24
|
+
Provides-Extra: duckdb
|
|
25
|
+
Requires-Dist: duckdb>=0.9; extra == 'duckdb'
|
|
26
|
+
Provides-Extra: snowflake
|
|
27
|
+
Requires-Dist: snowflake-connector-python>=3.0; extra == 'snowflake'
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# TableTalk
|
|
31
|
+
|
|
32
|
+
TableTalk is an evaluation and observability framework for natural-language agents built on existing
|
|
33
|
+
dbt projects. `manifest.json` is the complete authority for what an agent may query; warehouse
|
|
34
|
+
connections are read-only execution adapters for those dbt relations.
|
|
35
|
+
|
|
36
|
+
## Start from a dbt project
|
|
37
|
+
|
|
38
|
+
```console
|
|
39
|
+
cd my-dbt-project
|
|
40
|
+
dbt parse
|
|
41
|
+
dbt docs generate # recommended: adds physical warehouse types
|
|
42
|
+
tabletalk init
|
|
43
|
+
tabletalk agent create
|
|
44
|
+
tabletalk eval create revenue
|
|
45
|
+
tabletalk eval run revenue
|
|
46
|
+
tabletalk ask revenue "What was recognized revenue last month?"
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`tabletalk init` finds `dbt_project.yml` and `target/manifest.json`, resolves a supported SQLite,
|
|
50
|
+
DuckDB, or Snowflake target from the dbt profile, and writes only non-secret TableTalk settings.
|
|
51
|
+
Agent creation asks the user to choose a displayed dbt group, tag, model, path, package, or source and
|
|
52
|
+
whether to include lineage. It previews the exact resolved models, descriptions, catalog types, tests,
|
|
53
|
+
constraints, and lineage before writing a small selector-based resource. Raw selectors remain available
|
|
54
|
+
for automation.
|
|
55
|
+
|
|
56
|
+
```yaml
|
|
57
|
+
name: revenue
|
|
58
|
+
description: Answers questions about recognized revenue.
|
|
59
|
+
select:
|
|
60
|
+
- group:finance
|
|
61
|
+
- tag:revenue
|
|
62
|
+
exclude:
|
|
63
|
+
- model:customer_sensitive
|
|
64
|
+
instructions:
|
|
65
|
+
- Use recognized revenue unless the user explicitly requests bookings.
|
|
66
|
+
sample_questions:
|
|
67
|
+
- What was recognized revenue last month?
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Descriptions, columns, tests, constraints, ownership, access, tags, and lineage stay in dbt. They are
|
|
71
|
+
never duplicated into an agent file.
|
|
72
|
+
|
|
73
|
+
## Correctness and provenance
|
|
74
|
+
|
|
75
|
+
Live questions and eval cases call the exact same `Runtime.answer` path. Before execution, TableTalk
|
|
76
|
+
parses generated SQL and requires one read-only query, in-scope manifest relations, known columns,
|
|
77
|
+
explicit join conditions, row and timeout limits, and explicit sensitive-data permission. It derives
|
|
78
|
+
used dbt nodes and columns from that parsed SQL—not from model output.
|
|
79
|
+
|
|
80
|
+
Every result includes the answer, interpretation, assumptions, generated and executed SQL, dbt nodes,
|
|
81
|
+
columns, relevant test health, bounded evidence, evidence-linked claims, verification outcomes,
|
|
82
|
+
manifest and agent fingerprints, model and warehouse identity, latency, and token usage. Terminal and
|
|
83
|
+
web views expose “How this answer was formed” without separate flags.
|
|
84
|
+
|
|
85
|
+
Eval correctness is determined by execution, scope, model/column expectations, result equality,
|
|
86
|
+
reference-query matching, shape/count assertions, tolerance, and claim evidence. Eval creation uses the
|
|
87
|
+
reviewed SQL as the default changing-data reference, runs it immediately, and writes a verification
|
|
88
|
+
record. A live answer is labeled `VERIFIED` only when its exact normalized question matches an approved
|
|
89
|
+
eval case and all hard checks pass; otherwise it is visibly `UNVERIFIED` and exits nonzero.
|
|
90
|
+
|
|
91
|
+
See the dbt-generated [reference project](examples/dbt-analytics/README.md) and the
|
|
92
|
+
[workflow guide](docs/getting-started.md).
|
|
93
|
+
|
|
94
|
+
## Commands
|
|
95
|
+
|
|
96
|
+
- `tabletalk init`
|
|
97
|
+
- `tabletalk agent create|list|show`
|
|
98
|
+
- `tabletalk eval create|run`
|
|
99
|
+
- `tabletalk ask`
|
|
100
|
+
- `tabletalk doctor`
|
|
101
|
+
|
|
102
|
+
There is no compile/plan/apply lifecycle or applied-artifact authorization state. Source changes are
|
|
103
|
+
active immediately and reproducibility records live under `.tabletalk/runs` and
|
|
104
|
+
`.tabletalk/eval-results`.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# TableTalk
|
|
2
|
+
|
|
3
|
+
TableTalk is an evaluation and observability framework for natural-language agents built on existing
|
|
4
|
+
dbt projects. `manifest.json` is the complete authority for what an agent may query; warehouse
|
|
5
|
+
connections are read-only execution adapters for those dbt relations.
|
|
6
|
+
|
|
7
|
+
## Start from a dbt project
|
|
8
|
+
|
|
9
|
+
```console
|
|
10
|
+
cd my-dbt-project
|
|
11
|
+
dbt parse
|
|
12
|
+
dbt docs generate # recommended: adds physical warehouse types
|
|
13
|
+
tabletalk init
|
|
14
|
+
tabletalk agent create
|
|
15
|
+
tabletalk eval create revenue
|
|
16
|
+
tabletalk eval run revenue
|
|
17
|
+
tabletalk ask revenue "What was recognized revenue last month?"
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
`tabletalk init` finds `dbt_project.yml` and `target/manifest.json`, resolves a supported SQLite,
|
|
21
|
+
DuckDB, or Snowflake target from the dbt profile, and writes only non-secret TableTalk settings.
|
|
22
|
+
Agent creation asks the user to choose a displayed dbt group, tag, model, path, package, or source and
|
|
23
|
+
whether to include lineage. It previews the exact resolved models, descriptions, catalog types, tests,
|
|
24
|
+
constraints, and lineage before writing a small selector-based resource. Raw selectors remain available
|
|
25
|
+
for automation.
|
|
26
|
+
|
|
27
|
+
```yaml
|
|
28
|
+
name: revenue
|
|
29
|
+
description: Answers questions about recognized revenue.
|
|
30
|
+
select:
|
|
31
|
+
- group:finance
|
|
32
|
+
- tag:revenue
|
|
33
|
+
exclude:
|
|
34
|
+
- model:customer_sensitive
|
|
35
|
+
instructions:
|
|
36
|
+
- Use recognized revenue unless the user explicitly requests bookings.
|
|
37
|
+
sample_questions:
|
|
38
|
+
- What was recognized revenue last month?
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Descriptions, columns, tests, constraints, ownership, access, tags, and lineage stay in dbt. They are
|
|
42
|
+
never duplicated into an agent file.
|
|
43
|
+
|
|
44
|
+
## Correctness and provenance
|
|
45
|
+
|
|
46
|
+
Live questions and eval cases call the exact same `Runtime.answer` path. Before execution, TableTalk
|
|
47
|
+
parses generated SQL and requires one read-only query, in-scope manifest relations, known columns,
|
|
48
|
+
explicit join conditions, row and timeout limits, and explicit sensitive-data permission. It derives
|
|
49
|
+
used dbt nodes and columns from that parsed SQL—not from model output.
|
|
50
|
+
|
|
51
|
+
Every result includes the answer, interpretation, assumptions, generated and executed SQL, dbt nodes,
|
|
52
|
+
columns, relevant test health, bounded evidence, evidence-linked claims, verification outcomes,
|
|
53
|
+
manifest and agent fingerprints, model and warehouse identity, latency, and token usage. Terminal and
|
|
54
|
+
web views expose “How this answer was formed” without separate flags.
|
|
55
|
+
|
|
56
|
+
Eval correctness is determined by execution, scope, model/column expectations, result equality,
|
|
57
|
+
reference-query matching, shape/count assertions, tolerance, and claim evidence. Eval creation uses the
|
|
58
|
+
reviewed SQL as the default changing-data reference, runs it immediately, and writes a verification
|
|
59
|
+
record. A live answer is labeled `VERIFIED` only when its exact normalized question matches an approved
|
|
60
|
+
eval case and all hard checks pass; otherwise it is visibly `UNVERIFIED` and exits nonzero.
|
|
61
|
+
|
|
62
|
+
See the dbt-generated [reference project](examples/dbt-analytics/README.md) and the
|
|
63
|
+
[workflow guide](docs/getting-started.md).
|
|
64
|
+
|
|
65
|
+
## Commands
|
|
66
|
+
|
|
67
|
+
- `tabletalk init`
|
|
68
|
+
- `tabletalk agent create|list|show`
|
|
69
|
+
- `tabletalk eval create|run`
|
|
70
|
+
- `tabletalk ask`
|
|
71
|
+
- `tabletalk doctor`
|
|
72
|
+
|
|
73
|
+
There is no compile/plan/apply lifecycle or applied-artifact authorization state. Source changes are
|
|
74
|
+
active immediately and reproducibility records live under `.tabletalk/runs` and
|
|
75
|
+
`.tabletalk/eval-results`.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Runtime safety and architecture
|
|
2
|
+
|
|
3
|
+
Every live and eval question follows one pipeline:
|
|
4
|
+
|
|
5
|
+
```text
|
|
6
|
+
manifest → resolved agent scope → interpretation/SQL → AST validation
|
|
7
|
+
→ read-only connector → evidence rows → answer/claims → trace verification
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
`manifest.py` owns artifact normalization, selection, graph navigation, and fingerprints. `agents.py`
|
|
11
|
+
contains only selector resources and resolved in-memory scopes. `validation.py` parses SQL and derives
|
|
12
|
+
actual model/column use. `connections.py` resolves the selected dbt target and exposes only read-only
|
|
13
|
+
query execution. `runtime` constructs the answer and shared trace. `evals` calls that exact runtime and
|
|
14
|
+
adds deterministic comparisons. `traces.py` persists one schema for live and eval records.
|
|
15
|
+
|
|
16
|
+
Validation rejects multiple statements, writes/commands, forbidden external-read functions,
|
|
17
|
+
out-of-scope or ambiguous relations, unknown columns, unconditioned joins, excessive limits, and
|
|
18
|
+
unapproved sensitive resources. Connector permissions remain the final security boundary.
|
|
19
|
+
|
|
20
|
+
Lineage proves dependency, not semantic join correctness. TableTalk never labels a join verified merely
|
|
21
|
+
because two nodes share a lineage edge. Safe join evidence belongs in dbt constraints, relationship
|
|
22
|
+
tests, explicit `meta`, and passing eval coverage.
|
|
23
|
+
|
|
24
|
+
Post-execution verification requires successful execution and validates every claim’s row/column
|
|
25
|
+
evidence. Numeric claims must appear in cited result cells. Correctness additionally requires an exact
|
|
26
|
+
approved eval-question match and passing deterministic comparisons. Reported models and columns always
|
|
27
|
+
come from the validated AST. Trace records hold fingerprints and identities, not credentials or
|
|
28
|
+
deployment state.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Commands
|
|
2
|
+
|
|
3
|
+
- `tabletalk init`: initialize from `dbt_project.yml`, a parsed manifest, and a dbt target.
|
|
4
|
+
- `tabletalk agent create`: guided selector preview and agent authoring.
|
|
5
|
+
- `tabletalk agent list`: list source agents and resolved model counts.
|
|
6
|
+
- `tabletalk agent show NAME`: show source, resolved node IDs, fingerprint, and warnings.
|
|
7
|
+
- `tabletalk eval create NAME`: execute and approve a question/reference case.
|
|
8
|
+
- `tabletalk eval run [NAME] [--case CASE] [--trials N]`: run deterministic hard-gate evals and
|
|
9
|
+
optionally override the suite's independent-trial count.
|
|
10
|
+
- `tabletalk ask NAME QUESTION`: answer (quoted or unquoted), show provenance, and require passing exact eval coverage for a
|
|
11
|
+
`VERIFIED` status.
|
|
12
|
+
- `tabletalk doctor`: fail on artifact, target, connectivity, selector, or eval-coverage blockers and
|
|
13
|
+
report incomplete dbt descriptions as non-blocking metadata warnings.
|
|
14
|
+
|
|
15
|
+
`compile`, `plan`, `apply`, `connect`, `discover`, `connections`, and registry-style `agents` commands
|
|
16
|
+
were removed. A source agent is active immediately and asking never depends on applied state.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Configuration file for the Sphinx documentation builder.
|
|
2
|
+
# https://www.sphinx-doc.org/en/master/usage/configuration.html
|
|
3
|
+
|
|
4
|
+
# -- Project information -------------------------------------------------------
|
|
5
|
+
|
|
6
|
+
project = "tabletalk"
|
|
7
|
+
copyright = "2025, william bates"
|
|
8
|
+
author = "william bates"
|
|
9
|
+
release = "0.5.0"
|
|
10
|
+
|
|
11
|
+
# -- General configuration -----------------------------------------------------
|
|
12
|
+
|
|
13
|
+
extensions = [
|
|
14
|
+
"myst_parser",
|
|
15
|
+
]
|
|
16
|
+
|
|
17
|
+
# MyST parser settings — enable common markdown extensions
|
|
18
|
+
myst_enable_extensions = [
|
|
19
|
+
"colon_fence", # ::: fenced directives
|
|
20
|
+
"deflist", # definition lists
|
|
21
|
+
"tasklist", # - [ ] checkboxes
|
|
22
|
+
]
|
|
23
|
+
|
|
24
|
+
# Auto-generate anchors for headings up to level 3 (enables #anchor-name links)
|
|
25
|
+
myst_heading_anchors = 3
|
|
26
|
+
|
|
27
|
+
# Use index.md as the root document
|
|
28
|
+
root_doc = "index"
|
|
29
|
+
|
|
30
|
+
source_suffix = {
|
|
31
|
+
".md": "markdown",
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
templates_path = ["_templates"]
|
|
35
|
+
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
|
|
36
|
+
|
|
37
|
+
# -- Options for HTML output ---------------------------------------------------
|
|
38
|
+
|
|
39
|
+
html_theme = "alabaster"
|
|
40
|
+
html_static_path = []
|
|
41
|
+
|
|
42
|
+
html_theme_options = {
|
|
43
|
+
"description": "dbt-native evaluation and observability for natural-language agents",
|
|
44
|
+
"github_user": "wtbates99",
|
|
45
|
+
"github_repo": "tabletalk",
|
|
46
|
+
"github_button": True,
|
|
47
|
+
"fixed_sidebar": True,
|
|
48
|
+
}
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Configuration and resources
|
|
2
|
+
|
|
3
|
+
`tabletalk.yaml` is intentionally small:
|
|
4
|
+
|
|
5
|
+
```yaml
|
|
6
|
+
dbt:
|
|
7
|
+
project_dir: .
|
|
8
|
+
manifest: target/manifest.json
|
|
9
|
+
catalog: target/catalog.json
|
|
10
|
+
run_results: target/run_results.json
|
|
11
|
+
target: dev
|
|
12
|
+
|
|
13
|
+
llm:
|
|
14
|
+
provider: ollama
|
|
15
|
+
model: gemma4:31b-cloud
|
|
16
|
+
base_url: http://localhost:11434/v1
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Commands first look for `tabletalk.yaml` in the current directory, then for
|
|
20
|
+
`tabletalk/tabletalk.yaml`. This keeps TableTalk in its own repository folder without requiring a
|
|
21
|
+
`--project-folder` option on every command.
|
|
22
|
+
|
|
23
|
+
`project_dir`, `manifest`, and optional `profiles_dir` paths are relative to `tabletalk.yaml`. The dbt
|
|
24
|
+
target remains in `profiles.yml`; environment variables are resolved only at runtime. Supported
|
|
25
|
+
adapters are SQLite, DuckDB, and Snowflake, all opened or governed as read-only.
|
|
26
|
+
|
|
27
|
+
Agents support only `group:`, `tag:`, `model:`, `source:`, `path:`, and `package:` selectors plus exclusions.
|
|
28
|
+
`include_parents` and `include_children` are explicit booleans. Ephemeral and disabled models are not
|
|
29
|
+
queryable. Sources require an explicit `source:SOURCE.TABLE` selector; lineage expansion never grants
|
|
30
|
+
source access. Physical relation names must resolve to one dbt unique ID in the agent scope.
|
|
31
|
+
|
|
32
|
+
`instructions` documents business rules the manifest cannot express, while `reject_if_contains`
|
|
33
|
+
provides a small deterministic guardrail for known concepts that the selected dbt resources cannot
|
|
34
|
+
answer. Rejections happen before an LLM call and should be used sparingly for unambiguous missing-data
|
|
35
|
+
boundaries, not as a general intent router.
|
|
36
|
+
|
|
37
|
+
Optional `catalog.json` enriches physical types and statistics; optional `run_results.json` adds recent
|
|
38
|
+
dbt-test health. Neither can add queryable resources. The manifest fingerprint changes with any
|
|
39
|
+
manifest content change and is included in every run; the catalog has its own fingerprint. `tabletalk
|
|
40
|
+
init` detects both artifacts beside the manifest automatically.
|
|
41
|
+
|
|
42
|
+
The model prompt receives each selected resource's relation, description, physical catalog types,
|
|
43
|
+
column descriptions, tests, constraints, owner, access, materialization, package, explicit join
|
|
44
|
+
metadata, and upstream/downstream lineage. Lineage is provenance, never automatic join permission.
|
|
45
|
+
|
|
46
|
+
Sensitive models use `meta: {sensitive: true}`; sensitive columns use column-level
|
|
47
|
+
`meta: {sensitive: true}` or model `meta.sensitive_columns`. The agent must explicitly list allowed dbt
|
|
48
|
+
unique IDs or column names under `allow_sensitive`.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
TableTalk requires Python 3.10+ and uses `uv`:
|
|
4
|
+
|
|
5
|
+
```console
|
|
6
|
+
uv sync --all-extras
|
|
7
|
+
uv run pytest
|
|
8
|
+
uv run ruff check .
|
|
9
|
+
uv run mypy tabletalk
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
The acceptance fixture is a real dbt project under `examples/dbt-analytics`. When its dbt resources
|
|
13
|
+
change, regenerate the checked-in artifact instead of hand-editing it:
|
|
14
|
+
|
|
15
|
+
```console
|
|
16
|
+
cd examples/dbt-analytics
|
|
17
|
+
uv run dbt deps
|
|
18
|
+
uv run dbt parse --profiles-dir . --no-partial-parse
|
|
19
|
+
uv run dbt docs generate --profiles-dir .
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Keep these boundaries intact: manifest selection is the only source of query scope; database
|
|
23
|
+
introspection cannot add resources; all live and eval questions use `Runtime.answer`; model and column
|
|
24
|
+
usage comes from parsed SQL; result comparison is the hard correctness gate; and persisted records are
|
|
25
|
+
observability evidence, never deployment authorization.
|
|
26
|
+
|
|
27
|
+
Tests should cover manifest selection, SQL safety/scope, connectors, deterministic comparisons,
|
|
28
|
+
evidence-linked traces, and the complete dbt → agent → eval → ask journey. Snowflake behavior should use
|
|
29
|
+
a mocked connector contract unless a test is explicitly marked for live credentials.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Execution-based evals
|
|
2
|
+
|
|
3
|
+
Suites may declare `kind: regression|capability`, a human-readable `description`, and `trials: 1..20`.
|
|
4
|
+
Regression suites are expected to remain nearly perfect; capability suites hold harder tasks with room
|
|
5
|
+
to improve. Repeated trials are persisted separately, all must pass for the command to succeed, and the
|
|
6
|
+
terminal reports aggregate trial pass rate.
|
|
7
|
+
|
|
8
|
+
Create evals interactively with `tabletalk eval create AGENT`. The default is the generated SQL the
|
|
9
|
+
user just reviewed, so changing warehouse data is compared by executing candidate and golden queries
|
|
10
|
+
against the same snapshot. The proposed case runs immediately and automated authoring refuses to save
|
|
11
|
+
an already-failing case. Literal rows remain available for stable fixtures and CI.
|
|
12
|
+
|
|
13
|
+
```yaml
|
|
14
|
+
name: revenue-regression
|
|
15
|
+
agent: revenue
|
|
16
|
+
cases:
|
|
17
|
+
- name: july-revenue
|
|
18
|
+
question: What was recognized revenue in July 2026?
|
|
19
|
+
expect:
|
|
20
|
+
reference_sql: |
|
|
21
|
+
select sum(recognized_revenue) as recognized_revenue
|
|
22
|
+
from {{ ref('fct_orders') }}
|
|
23
|
+
where order_date >= '2026-07-01'
|
|
24
|
+
and order_date < '2026-08-01'
|
|
25
|
+
result:
|
|
26
|
+
comparison: scalar
|
|
27
|
+
tolerance: 0.01
|
|
28
|
+
models:
|
|
29
|
+
required: [model.analytics.fct_orders]
|
|
30
|
+
columns:
|
|
31
|
+
required: [recognized_revenue, order_date]
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Result modes are `scalar`, `ordered`, `ordered_values`, `unordered`, and `keyed`; keyed comparisons
|
|
35
|
+
require `keys`. `ordered_values` deliberately ignores presentation-only aliases while preserving row
|
|
36
|
+
and value order.
|
|
37
|
+
Expectations may also include literal `value` or `rows`, `row_count`, exact result `columns`, forbidden
|
|
38
|
+
models/columns, `allow_extra_columns: true` for harmless additional evidence fields, and
|
|
39
|
+
`outcome: ambiguity|rejection`. Reference SQL supports only dbt `ref()` templating,
|
|
40
|
+
is parsed with the same read-only scope validator, and executes against the same warehouse snapshot.
|
|
41
|
+
|
|
42
|
+
Hard gates are execution, SQL safety/scope, expected models/columns, expected rejection, result
|
|
43
|
+
comparison and tolerance, shape/count, and evidence-linked claims. Latency and token usage are recorded
|
|
44
|
+
but informational. Empty literal row sets are enforced rather than treated as a missing expectation.
|
|
45
|
+
Results are written to `.tabletalk/eval-results/AGENT` and failures exit with code 3.
|
|
46
|
+
|
|
47
|
+
Live correctness uses exact normalized question matching, not semantic guesswork. A covered question is
|
|
48
|
+
`VERIFIED` only when every hard gate passes. Uncovered questions still return their inspectable trace,
|
|
49
|
+
but are labeled `UNVERIFIED` and exit with code 4.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Get started
|
|
2
|
+
|
|
3
|
+
TableTalk starts with a parsed, runnable dbt project. It does not discover arbitrary database tables.
|
|
4
|
+
|
|
5
|
+
```console
|
|
6
|
+
cd my-dbt-project
|
|
7
|
+
dbt parse
|
|
8
|
+
dbt docs generate
|
|
9
|
+
tabletalk init
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Initialization finds the dbt project and manifest, reads the project profile, asks for a target, checks
|
|
13
|
+
that its adapter is SQLite, DuckDB, or Snowflake, summarizes dbt version/models/groups/tags, and writes
|
|
14
|
+
`tabletalk.yaml`. Credentials stay in environment variables and `profiles.yml`.
|
|
15
|
+
|
|
16
|
+
Create an agent from dbt selectors:
|
|
17
|
+
|
|
18
|
+
```console
|
|
19
|
+
tabletalk agent create
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Choose a displayed dbt group, tag, model, path, package, or source, then choose whether to include its
|
|
23
|
+
lineage. The guided preview shows exact resources, catalog types, tests, constraints, and metadata gaps.
|
|
24
|
+
Create the first eval when prompted. TableTalk shows the interpretation and SQL before read-only
|
|
25
|
+
execution, uses that reviewed SQL as the default golden query, verifies it immediately, and saves both
|
|
26
|
+
the case and result record.
|
|
27
|
+
|
|
28
|
+
```console
|
|
29
|
+
tabletalk eval run revenue
|
|
30
|
+
tabletalk ask revenue "What was recognized revenue last month?"
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Both commands use the same runtime. `ask` prints `VERIFIED` only for an exact normalized question match
|
|
34
|
+
against an approved, passing eval; new or changed questions print `UNVERIFIED` and exit nonzero. Run
|
|
35
|
+
`tabletalk doctor` in CI or locally to detect missing/stale
|
|
36
|
+
artifacts, unsupported targets, connectivity failures, broken selectors, metadata gaps, and uncovered
|
|
37
|
+
agents. Metadata gaps are warnings; conditions that prevent safe or verified execution fail the command.
|
|
38
|
+
The complete runnable journey is in `examples/dbt-analytics`.
|